You might find "Why Zebras Don't Get Ulcers"
, by Robert Sapolsky a good read. It is a really well written, not super technical, treatment of stress and its relation to diseases.
Its been a few years since I have read it so I am fairly hazy on the details but the big take-away is basically that you are correct. Stress is stress whether it comes from exercising or freaking over an exam. The difference between the two is that the former is a very acute condition (IE you exercise for an hour a few times a week) while the latter tends to be chronic (stressing for a couple of weeks straight around exam-time). It is the condition of chronic stress and the accompanying metabolic/hormonal cascades which lead to the detriment of health in the long term. Basically it reinforces the idea that too much of anything is a bad thing.
Again, if you are very interested you should check out the book.