That is the very same question I asked my lecturer at University in 1966! (Nothing is new under the Sun
) He was actually flummoxed by the question and I only came to terms with the problem years later, when I approached it again.
To answer your main question, the reason that you get a diffraction pattern is that the various light paths through the system are all different - their relative phases, where they are detected (the film) are different from place to place on the film. That's why you get the light / dark fringes of the hologram. For a simple two slits, the effect is easily explained (in wiki and all over) - at different angles, the phase relation between light from the two slits is different, which gives you the fringes. You can do the two slits with almost any old light source and get fringes over a limited range of angles. To work well, you need good coherence so that, even at large angles, the different wave trains that make up the light source are still long enough for self interference to occur. (For a pair of radio signals, for instance, the coherence is almost perfect and the interference is more or less textbook and a laser is about as good)
For a hologram to work, the requirement is even more strict because you need a broad, coherent wavefront for your reference and a broad, coherent wavefront hitting the object. The resultant (max / min / intermediate value) at any single point on the film is due to light reflected from every point on that side of the object, interfering with the reference beam and with itself (of course). To one side of that point, you will get the interference result from a slightly different direction from the scene. When the hologram is reconstructed, you are looking through the film, from one point of view, at light coming through the hologram through a small cone (iris width). That cone of light will have passed through a small disc (part of the whole hologram) and what gets into your eye will be another interference pattern (i.e. viewing light through the fine, complicated hologram pattern). This interference pattern is an approximate version of a picture of the whole original scene from that point of view. The resolution is limited (I think) by the aperture of your eye as well as the quality of the equipment.
Did you look at that wiki article on computer generated holograms? If you can understand the bit involving the Fourier transform it becomes clearer. If you aren't familiar with that then you may just have to accept the fact that 'it works'. (Btw, Fourier transforms work on spatial frequencies and variations in the same way that they work on sound frequency spectra and waveforms - same Maths involved with each)