It's actually quite hard to physically overdose on LSD, and the lethal dose is well beyond what any person, even a hard-core drug user, would even consider approaching.
Mentally "overdosing" is another story, and quite easy. Horror stories involving hallucinogens usually involve people who take more than they can mentally handle, and end up either doing something stupid, or actually committing suicide (a bad trip on a high dose can actually be so horrifying that the person will choose suicide just to end it). Sometimes also people who don't know they were given LSD can, understandably, go mental.
On people who go crazy from hallucinogens, or report "flashbacks," as far as I've read, it seems to be a form of PTSD caused by the experience itself... much like trauma suffered by soldiers or war victims; a traumatic experience affects them so that they can become so mentally unstable that they must be hospitalized. I haven't seen an article that reports damage done by the actual chemical like you would find with cocaine or even pot (then again, not enough research is done on LSD so you can't really know).
So yea hallucinogens can be extremely dangerous if mishandled or overused, but they should have no effect on longevity or mental alertness (this is when one would play the Kary Banks Mullis card, but he's pretty out there lol) in someone who didn't go overboard with dosage and was mentally prepared to handle the experience.
Personally I have not touched a hallucinogen in over a year and don't plan on doing so any time in the near future, I haven't smoked pot in over 2 years, and barely even drink anymore. But I still find the subject fascinating and think hallucinogens should be researched more and understood better. And like I've said before, I don't think they are all-negative and destructive like most other recreational drugs (nor are they particularly recreational, seeing as the experience can be quite taxing).