Who was the first anti-hero in literature?
Perhaps Ovid and some the other Roman literature also. Greek literature seems replete with heroes.
Unfortunately this area does tread dangerously close to the realm of religion. It may be impossible to give a correct answer considering that the originating date of many stories & literature are unknown. Plus there was the loss when the great library of Alexandria was destroyed. Then there is the possibility of offending someone's beliefs as the answer is subjective to one's point of view. And there is the question as to whether the anti-hero has to be human. Potential candidates include:
Set - Who killed Osiris and threw his body to the crocodiles. Isis searched for and recovered his body and raised him only to have Set cut him into 12 pieces and send the pieces throughout the world. According to the Egyptian Book of the Dead, that legend goes back to at least as far as the VII dynasty.
Lillith - The first wife of Adam according to Hebrew literature who had a nasty rep for eating her own children for which she was banished to haunt mankind (particularly children) as a demonic entity. The story of Lillith was believed to be part of the Hebrew oral traditions before it appeared in printed form. So supposedly this would predate the murder of Able by Cain as well as the snake in the garden of Eden. In the actual translations of the Hebrew text, Satan was not mentioned by name until much later. & Lucifer technically would not qualify due to paradox. As the name translates into "the light-bringer" he is the alternate for the snake in the other creation version as by his deeds man became "enlightened" to the difference between good and evil. Arguably then, evil did not exist until after this happened, so by paradox he doesn't qualify as an anti-hero. The Greek version of the Lucifer paradox can also be found in Prometheus, who was punished by the gods for aiding man.
Pandora - For opening the box and releasing everything but hope, or should we blame the entity that gave it to her?
Unnamed Emperor or King of Atlantis - In Plato's Dialogus of Timaeus & Critias, Solon revealed that the Egyptian priest of Sais knew of heroic predeluvian deeds performed by the Athenian predecessors that even their Greek descendants could not recall. Unfortunately Plato never finished the story to name names & there is no confirmation of the story that I'm aware of in Egyptian writings.
Coyote - Stories of Coyote, the trickster of Native American legend go back to before man climbed up through a hole in the ground to the upper world.
Powako - A snake priest in Delaware Native American legends brought snake worship to the earth which resulted in wickedness, crime, & unhappiness.
Bel or Baal - In Chaldian/Assyrian/Babylonian legend; the God Chronos/Ea saved Khiasistra/Xisuthros from the deluge planned by the God Bel/Baal to destroy all mankind. Bel/Baal was angry that anyone survived, however, his desire to destroy the survivors as well was denied by the council of gods. Note: the name Baal translates literally as lord, however I suggest that one should draw any assumptions or read anything into this so as not to offend. In the original Chaldean/Assyrian/Babylonian pantheons he was not considered an ultimate or supreme being, just one of the gods.
There are many other "baddies" of legends and myths from other countries, such as the oriental nations and many that mirror other versions of similar stories. Again it is impossible to give an absolute answer, so one must fall back on opinion.
Separate names with a comma.