This is not a hypothetical; it's actually realizable, and the answer is no. There are three "families" of leptons in the Standard Model: the electron family, the muon family, and the tau family. (Each family contains a charged lepton and its antiparticle, and an uncharged lepton and its antiparticle; in the electron family these are the electron/positron and the electron neutrino/antineutrino.) Each lepton can only annihilate with its own antiparticle, the one in the same family and with opposite charge. The neutrinos (electron, muon, and tau, each with its own antiparticle) are particularly interesting in this regard, since they have no conserved charges other than lepton number.
The underlying reason for all this is that, in quantum field theory, particles and antiparticles are not just two separate things that happen to be opposites in all conserved charges. They are states of the same underlying quantum field. (The technical term is that they are CPT conjugates of each other.) The process we have been describing as "annihilation", from the viewpoint of QFT, is just a difference in the state of the quantum field between one region of spacetime and another.