Part of the issue I realized after working with two tutors, and another physics professor was my lack of understanding of how torque can be calculated as my professor just wrote a one sentence description of it...
As to my final answer:
0=Trope+Tgravity+Tfriction+Ttable
The force of the rope remained as F, and noted as such so that I wouldn't convert it to mg
0=b*F+mg*sin 90 + b*FN*mu -a*FN
mg*sin 90 is = to zero
0=b*F+b*FN*mu-a*FN
b*F+b*FM*mu=a*FN
b*F+b*mg*mu=a*mg
b(F+mg*mu)=a*mg
F+mg*mu=(a/b)mg
F(rope)=(a/b)mg-mg*mu
Now my professor's answer looks like this:
(a) Let's use subscripts "r" for the rope, "g" for gravity, and "n" for the floor's normal force. In the case where the block topples, it rotates and its center of mass rises up and over. In the case where it slides without tipping, neither of these is supposed to happen, so vertical forces must cancel, and the torques must cancel. Cancellation of the vertical forces fives Fn=mg. The torques are most easily calculated using t=r*F, with the axis taken to be at the center of the box. Note that in the condition where the box is not tipping, the normal force is actually being distributed over the whole bottom surface of the block, so it doesn't actually have a single well-defined value of r; however, as soon as the tipping starts, the only point of contact is at r=a. The torques are therefore t=bF, and T=amg. If the box is just about to tip, i.e., the force from the rope has its maximum value, then these torques cancel, and F=(a/b)mg. The dependence on a/b makes sense, because a squat box with a large value of a would be more stable, allowing F to be greater.(What we've actually calculated here is the value of F for which the box would just barely start to tip. Once the box started to tip, we could ask ourselves whether it might then manage to right itself again. The answer is no, because once it starts to tip, both of the r values start to decrease, but their ratio is still (a/b)
(b)The force of friction is F=mu*F=mu*mg. (In the case of static friction, this is the maximum force, which is what we're interested in because we want to see if we can unstick the box and get it to slide without tipping. In the case of kinectic friction, this is simply the value of the frictional force) We now have a torque from friction, which is in the same clockwise direction as the torque from the rope. Cancellation of the torques fives b*F +b*mu*mg=a*mg, yielding F=(a/b-mu)mg.
(c) the answer to part b, unlike the one from part (a), could come out negative under certain circumstances. the negative solution would be unphysical, and the interpretation would be that it would be impossible to accelerate the box without tipping it over. This happens for b >a/mu. That makes sense. A big value of b means a tall box, which would be less stable. In the limit mu -> 0, we get b -> infinity, meaning that there is no maximum value of b, recovering the behavior of the frictionless case.