This can be done without Mathematica or Maple.
b(b - 4) < -4a <==> (-1/4)b(b - 4) > a
This is a quadratic inequalilty. To find the solution set, graph a = (-1/4)(b^2 - 4b). The graph is a parabola that opens down. The graph of the parabola isn't in the solution set, but one side or another (i.e., either the inside or outside) of the parabola represents the solution set.
To determine which side is the solution set, simply pick any point that is not on the graph of the parabola. If the original inequality is a true statement for that point, the solution set is all of the points on that side of the parabola. If the original inequality is not a true statement for that point, the solution set is all of the points on the other side.