I saw the first post in this thread before it had attracted any replies - about 30 seconds before I shut my laptop down to get into the car for a long trip, so I had many hours of interstate boredom to think about the question without the benefit of seeing of the subsequent discussion.
My first thought, along with Russ above, is that we have to be clear about what problem we're trying to solve. However, I see that the OP has answered that for us:
One, among other, issues with voting is the possibility that a voter's ballot could be destroyed and replaced with one having alternative cast votes.
That is, we want assurance that valid and legally cast ballots are not altered before they are counted. This is technologically possible (although whether it's worth the effort is a different question, out of scope for this thread). And as OP says, there are other issues, but we'd need something specific to usefully discuss these.
OK, with that said...
1) A non-technical digression: you say " federally government issued". In the USA, the federal government does not run elections (and is constitutionally prohibited from doing so) so whatever process you're thinking of would have to be put in place by one or more states - and if it turns out well, other states may follow suit. This is a non-issue for many other countries; for example, in Mexico the federal government does control voter identification. But for now, if you're thinking USA, you designing a mechanism that a state may choose to use for its elections but not a nationwide panacea.
2) In American elections, many/most jurisdictions already uniquely identify their voters to ensure that no registered voter submits more than one ballot and no unregistered voter submits a ballot, but also protects the anonymity of the voters. Where I live, voting in person at the local precinct is a two-step process: I stop at one desk and identify myself; a records check shows that I'm registered and haven't voted already; so I'm given a piece of paper that I take to a second desk where I'm given an anonymous ballot and the paper from the first desk is turned into record the fact that I voted; I fill my ballot out and submit it to be counted. The ballot records my vote anonymously and the slip of paper from the first desk identifies that I voted by name. (Something similar works with absentee voting; I'm given a ballot and an envelope with my registration record written on it, I fill out the ballot, I submit it in the envelope, and one piece of paper records my votes and the other records that I voted).
3) We could, if we thought it was worth the effort, leverage the #2 process to ensure that my submitted ballot is not altered or substituted. At the first desk I am given a public/private key pair unique to me; the public key is essentially the voter-unique "serial number" that's suggested in the original post. I collect my ballot at the second desk and fill it out. Then, as part of submitting the ballot, I also encrypt a copy using the public key; the result is a completely opaque puddle of bits that the election authority is required to post publicly along with the public key. Now I can verify that my ballot has not been tampered with: look it up by public key, decrypt using my personal private key, verify that it's it's what I submitted. If the election results look bogus, there will be a public outcry leading many people to check their ballots, and any large-scale skullduggery will be detected.
4) By itself, #3 is a complete total waste of effort because any attacker able to alter submitted ballots will also be able to introduce completely bogus ballots into the system, and with no one to check there is no risk of detection. However, there is an effective non-technological defense against this attack, and it's already in place: The "who voted" records produced at the first desk in #2 above can be made a public record - they are in my jurisdiction. A discrepancy between the number of ballots and the number of "who-voted" records will be obvious and can only be hidden by introducing bogus "who-voted" records; but these are public and cannot be hidden.
5) Neither #3 nor #4 provide any protection against a corrupt election authority that won't count the submitted ballots honestly. Making all the counted ballots public allows an after-the-fact audit; and we can require that observers, both neutral and partisan from all sides, are present at all stages. There's also much to be learned from the financial controls and cash-handling procedures of large businesses and government agencies. Again, this is not a technological solution.
6) As #4 and #5 suggest, good security requires looking at the entire system: people, procedures, checks and balances, visibility. Clever technology like unique serial numbers, two-factor authentication, public-key encryption may be essential parts of the solution, but focusing on these is insufficient.