Actually, the act of observation itself is not necessary to destroy the interference pattern. The possibility of observation is enough. If you shined a light between the screen and the slits, WITHOUT actually observing to see which hole each electron went through, there would be no interference pattern.