I found it both challenging and rewarding.
With respect to the challenging part, I think the last time I was ever the "top" student at anything was when I was in high school. Since entering university there has almost always been someone faster to catch on, smarter, or able to seemingly breeze through classes that I struggled with. Graduate school was even tougher. Once again there was another level of elimination, which meant there wasn't anyone arount to 'anchor' the class average. If you add to that the years of living on minimal pay while other friends were out buying houses, cars, etc. and leaving their jobs at 5:00 pm, enjoying weekends... yeah, it was a challenge.
I would say that the biggest reward for me is that now I'm in the position to do purpose-driven work. I have a reasonable amount of freedom to define my own research directions. I know that both the clinical work and research I do has a direct effect on the outcome of treatments for people with cancer.