The complete answer to this is very complicated ... combustion mechanisms are quite complicated, involving varoius highly-reactive free radical intermediates ... in some cases complete mechanisms are not even known yet. However, there are some simple principles you can apply to help you understand why CO is formed instead of CO2.
First, just consider the atomic balance ... if you don't have at least twice as many oxygen atoms as carbon atoms, then it is impossible to convert all of the carbon into CO2. So limiting the amount of oxygen at the very least limits the amount of CO2 that could be theoretically formed. Furthermore, remember that you are also forming water from the H atoms released by the carbon, so that also uses up some of the limited amount of oxygen.
The reason the CO is the major product is because, although the formation of CO is less exothermic than that of CO2, it is still more exothermic overall to convert more of the carbon atoms from methane to CO, than to convert half as many from methane to CO2.
Those answers are really just rationalizations of what we observe however ... the real answer requires knowledge of the precise mechanism by which the different products are formed.