After studying complexity theory, and from my own experience as a musician, I've come to agree with the idea that music and rhythmics appeal to us strongly when there is a balance of predicatbility and unpredictability (in terms of the listeners capability) embedded in it. In complexity theory, the idea is that the music has Shannon entropy, representing unpredictability, as well as a spectrum of ordered complexity structural (analogous to logical depth, or forcasting complexity for example). The idea is that people like to be challenged and surprised. If its too simple, its too easy, too boring. If it's too complex, we cannot keep up with it, we cannot solve it. Essentially, this possible subconscious motive and drive in music is related to our intelligence and adaptability to the world in general. Whether it is a video game, a movie, music, a sport, we like to be challenged a little, and we like to be surprised. And when we are surprised, we like to try and meet the unexpected challenge. This has obvious evolutionary benefits.
Before I try to make the connection with time signatures, I should say that I think its a little more complicated with music, because the listener can be actively creating ideas in real time with the music they are listening to as a base. For example, a simple beat can be the foundation for a more challenging/complicated structure. And also, music aside from rythmatics, has innate effects on us based on tone and pitch and so forth.
Now in terms of odd beats vs even beats, I believe even beats are easier, maybe partly do to our experience walking, but also just because they have a simpler structure. They have symmetry, can be decomposed in simple basic pieces that fit together nicely. Just purely from a mathematical perspective, they should be easier to understand, and follow along with. Moreover, due to the symmetry, and divisibility, it is easier to predict each next part as you are listening, because the different parts, or self similar cycles are repeating more frequently and you can synchronize quicker and with less information. In complexity theory, this is like a process that has a simplier causal structure, or one that uses less memory, or lower temporal range correlation. That is, you need to see less of the past to predict the future.
If you think about it, 4's are just pairs of 2's, perhaps with some slight difference between each of the 2's. However, the difference between the 2's is usually such that there is interchangability. You can basically plugin any pattern of 2 and it works. Playing around with different variations of 2's then gives some freedome to do slightly less predictable patterns. Then you have 3's, which cannot be broken into parts. It makes up a single atom in a sense. If you want to play with them, you have to combine them with more 3's or some 2's. In any case, it comes with some extra temporal range correlation, and some additional challenge to the listener. To do 5's, you have a 3 and a 2. But this means to synchronize, you need to know if you are on a 2 block or 3 block. So you have to go back a little in time to figure out where you are (probably about at least 6 beats back?). It's more challenging. On the other hand, 6's are just 2 blocks of 3's. They have less temporal range correlation than 5's, and are thus mathematically easier to understand. Still they are a little more complex than 4's. That said, you could have 4's as 3+1, and 6's as 5+1, 3+2+1, etc, in each case where the complexity goes up compared to 2+2 or 3+3. Usually when we think of the even beats though, we are thinking of them as mostly symmetrical patterns. In the worse case, you simply count symmetrically, and still can synchronize on the 1 easily. Then you have 7, it can be 4+3, 5+2, ... Anyways, odd numbers cannot be broken up into repeating same length blocks, and require more real time memory to synchronize with. The larger the number the worse.
Because a great deal of complexity can be superimposed over the rhythm. Having the ability to easily synchronize with the core repeating structure enables more of your problem solving effort to focus on improvisation, or following a different more complex element of the music. If this synchronization is lost, then your grasp on everything else falls apart.
If we were much more talented, maybe more complex and/or odd time signatures would be more appealing to us. Once we have exhausted the challenge of introducing and understanding complexity in terms of variations of even blocks, we would probably gravitate towards odd ones naturally, the same way that a gamer switches to a higher difficulty setting as they progress. So in theory, playing with odd and more complex signatures can come from a high amount of experience, and or raw talent. Also there is probably a spectrum of ways to challenge oneself besides the signature of a single beat alone, such as changes between beats in sections, changes in tempo and dynamics and syncopation, not to mention melodics.
The conclusion is that to some extent, our interest and engagement in rhythematics could be based largely on notions of complexity, and this isn't a unique thing to music. If you want to engage your audience, you should allow them to follow along without getting lost, as you feed them small surprises and challenges. Even beats are easier to synchronize to, which makes them easier to start with, and easier to follow as the core part, and can also extend in complexity well above most peoples limitations. Peoples rhythematics processing skill is a limiting factor. High speed/real time deciphering of more complex event streams has pretty general evolutionary benefit and usefulness in life. As we get further and further from relying on real time/fast thinking+action, we might lose some of our capabilities and interest in complex signatures, and instead become more insteasted in other elements such as things which map more closely to skills in planning, or organization, depending on the case? Who knows really? We cannot ignore the emotional connection also. It's not really a simple case in real life.
By the way, did you know that Aboriginals (from Australia) used songs to record their journeys and record maps, so to speak, of their environment and pass it down through the generations. Their environment was notoriously harsh and difficult to navigate as well. Their music became a integreal part of survival, which should be an interesting case to look at I think, especially in terms of an area I sort of ignoed, which is why specifically we want to memorize patterns.
My thought is that it's not just practicing real time processing, but also exercising our memory, and memorizing important ones that we really need/needed in life. As we gain the ability to predict something we couldn't before, it is reflected in our memory, and becomes automatic. In real time these kind of automatic responses are useful. But also, there may also commonly be cases where it is intertwined with simply memorizing seemingly unrelated patterns of different complexities and within different aspects of our lives (procedures to follow / instructions, navigation, all kinds of things). We are strengthening neural pathways that allow us to predict things generally, and also in practice to memorize real things.
. 
I'm a poet! Love poetry. Working on a poem right now.
