You need to understand that in order to see a focused image, the eye must converge all the rays from the object to basically a single point at the back of the eye.
If the light rays diverge, or converge at a point slightly ahead or behind the back of the eye then the image will appear blurry.
So, by the design of our eyes there is only so much distance from the front to the back with which the eye has to focus the light rays, and our eyes have been adapted to do so at an interface of air->eye.
Technically, though, all the eye would have to do to see underwater without goggles is adjust the focal length of the eye enough, but depending on the curvature this required it may not be possible for the human eye to achieve.
Now, when you use goggles, you restore the air->eye interface, so even though the incoming rays may themselves still be refracted from the goggles, the eye is now capable of converging those rays to a single point at the back of the eye once again.
However, due to the water->goggle->air this refraction can cause other distortions like objects appearing slightly out of position, which depends largely on the angle you are looking, but even though those rays were bent the eye can still converge them because it has an air interface inside the goggles.
BTW: I think the inside of the eye has an index of refraction very close to water (I could be wrong) in that case when you don't have goggles on there would be almost no refraction so the light rays would not converge properly.