It's probably because the refractive index of the substance is different for different wavelengths of light since the velocity of light is going to be different in separate mediums. It is safe to assume that violet light will be faster in a medium than red light will be in the same medium. If we then look at Snell's law, [itex]\displaystyle\frac{\sin(\theta_1)} {\sin(\theta_2)} = \frac{v_1} {v_2} = \frac{n_2} {n_1}[/itex] It is easy to notice that a small change in [itex]v_1[/itex] or [itex]v_2[/itex] will noticeably affect the angles.
The change in the speed of each frequency is due to the material that it's traveling through, and how it vibrates the material.