Races less than about 400m are anerobic - you are burning energy in the muscles. At 800m it' about 50/50 aerobic/anerobic - in a 100m it's a waste of energy to breathe.
Most records aren't set at altitude - but this is partly because most premier tracks aren't at altitude, the same could be said for latitude - most places near the equator don't have expensive stadiums attracting world class athletes.
On paper it doesn't look like air density has a significant effect on speed (except perhaps for long jump?) but it might have a psychological effect for an individual athlete.
There is another real but separate effect - if you train at altitude with reduced oxygen (and ideally if you can arrange for a few 1000 generations of your ancestors to do the same) then compete at sea level you do have an advantage in aerobic (endurance) events.