For part 1, we solve a system of equations:
F_net = M*a = T - F_fr
with F_fr the force of friction,
Torque_net = I*alpha = T*r + F_fr*R
since the torques from the tension and friction operate in the same direction, and
a = alpha*R.

First we find I = (1/2)(M)(R^2 + r^2).
Then we substitute a/R = alpha.
To get F_fr in terms of T, R, and r, I substituted the moment of inertia and the expression for alpha into the net torque equation. Solving that for M*a, I then substituted into the net force equation and solved for F_fr, which, after some algebra, ends in the glorious result:

F_fr = T(R - r) / (R + r).

The second part of the question is quite intriguing.

Suppose that a > T/M. Then M*a > T, but M*a = T - F_fr. This implies that F_fr < 0, which has bizarre implications: the first is that r > R in the formula above, and the second (even more stunning) is that friction is actually pointing in the same direction as the relative motion!

Is this supposed to happen? If so, can someone point me to an explanation of how this defies a supposed tenet of friction (friction opposes relative motion)?
Did I make a mistake in setting up the system?

Just something you might consider. friction opposes relative motion, but what about the case when you push a block (or calculator across the table) without it moving. Isn't it true that even friction acts when it's not moving at all? So there is a lot built into the phrase "relative motion". How about when you walk? Which way does friction point? Well really push your feet into the ground hard. You're feet push backwards (try it, can't you feel the force your of your feet pushing backwards before you break static friction. But if friction also pointed in that direction, then what would keep you from slipping? What is going on is that friction actually is pointing forward when you walk. Don't confuse your the motion of your body with what the relative motion of your feet WOULD experience if they were to slip. This is the key. In this example and the spool example static friction isn't broken. So I think you have your frictional force in the wrong direction. It certainly does point in the same direction that the spool rolls, but it also points opposite the motion the spool would experience if it were to slip.

Hope this help (and sorry for the long winded answer)