But I am not sure of the answer: I thought that the scale stopped because the lever arm decreases as the scale becomes more vertical, but that happens to both sides so both torques decrease at the same rate (right?).

So why do scales stop when slightly unbalanced? It seems like friction is not the answer, because sometimes the scales tip backwards and then forwards until it stops.

I thought that somehow the force must decrease on the heavier side a bit, but I'm not sure how. Is some of the force of gravity of the heavier object being supported by the bar (which is in turn supported by the fulcrum)? If so, how would I diagram/calculate that?

I know the lever arm decreases as the bar becomes more vertical, but that decreases the torque on both sides equally (I think).

On real scales the pivot point is generally above the points of attachment of the weight pans on the lever arms. Think of a coat hanger with the pivot at the hook and the pans suspended from the ends of either arm.

If you had a scale like this one (click), how would it work? The points of attachment seem to be level with the pivot. My physic textbook says that even if the objects are hanging from the pivot bar, it doesn't change the torque (it illustrated that with a seesaw hanging someone over a cliff, but the idea should still apply for scales).

It would only work if the weight on each scale was the same. As gneill mentioned, by having the pivot point at the above the center of the balance bar, then the center of mass of the system is raised when the scale is angled from horizontal, creating an torque to oppose the imbalance in weight.

There 2 types of equilibrium, stable and unstable equilibrium
In the case of scale it is a stable equilibrium .

In both of your questions the answer lies on the position of the centre of mass,CM.
The scale will return to a position where the CM is directly under the pivot.

If both of equal mass, then it is perpendicularly under the pivot.
If the left is heavier than the right mass, then the CM is bit left to the centre thus it will turn anticlockwise such that finally it is below the pivot(resulting in no more torque, since the force is now in line with the axis of rotation)

Anyway we need friction to dampen the oscillation.