Or you are even applying less force, because it resists less to rotation. It is hard to control the amount of force you apply to an object which is not fixed.
The engine rotates with the rocket so the direction of the linear acceleration changes too, if you fire the engine continuously. But you can still get somewhere, when you fire the rocket in short bursts, just when the engine points in the desired direction. If you had two opposing off center engines you could fire them alternating after half rotation avoiding spinning up more and more. Since you don't put any energy into rotation, this should be quite efficient in terms of terminal velocity / fuel spent.