We evolved as omnivores and our bodies are optimized for a hunter-gatherer diet. Before agrarian societies developed, and some people stayed tied to the land, with facilities for storing food (granaries, etc), the availability of vegetables and grain as food was not very reliable. Anybody who would not eat mammals, fish, birds, amphibians and insects, as well as the eggs and larvae of such would probably not have survived. At the least, their need for protein and some minerals and other nutrients would not have been met, putting them a disadvantage.
The Inuit have survived for many generations on a diet that is predominantly meat, fat, and fish, with blueberries and other vegetable additions as available. Until recently, with the introduction of foods from our western/temperate diet, they had a very low incidence of arteriosclerosis and other diseases we usually associate with a high-fat diet. Depending on where they live and what is available as prey, they may eat mostly fish or mostly marine mammals, caribou, etc. The common factor in their diet is that until recently, vegetables were very scarce and only available seasonally.