Remember the Japan incident. Fission reactors have to be kept cool and controlled, and can relatively easily go out of control without the proper safeguards. While you may need to replace things in a fusion reactor every few decades, the security you'd have knowing that the plant will not suddenly explode is very nice. You can prepare and plan for replacing things at intervals, you cannot always plan for a disaster.There is also a large amount of structural material that is made radioactive by the high flux of 14 MeV neutrons, and this needs to be disposed of every couple of decades because the neutrons cause it to lose structural strength. All in all I'm not sure that a fusion reactor will generate that much less radioactive material than a fission reactor.
Also, it's not about the amount of material that is radioactive, it is more about the type of material, how it's stored, how easy to contain it is, and how dangerous it is biologically. Something that is more radioactive isn't necessarily worse than something else if it is much easier to contain and doesn't get absorbed into the body easily.
Read this on wikipedia: HTO has a short biological half life in the human body of seven to 14 days, which both reduces the total effects of single-incident ingestion and precludes long-term bioaccumulation of HTO from the environment.In addition, tritium has a large biological impact since hydrogen is a significant component of living cells.
Looks like Tritium is dangerous, but probably much less dangerous than something like Iodine is. Wouldn't want to get too much of either though.