I think I get why it's negative now.
By convention, we assign the right direction as the positive direction.
Let's take a look at the term mgsin(θ). Assuming m and g are always positive, the term changes sign depending on what θ is.
When θ is positive, sin(θ) is positive and the term mgsin(θ) is positive. But we know the restoring force is to the left when θ is positive, therefore, the force must be negative and so we add a negative sign.
When θ is negative, sin(θ) is negative and the term mgsin(θ) becomes negative. But we know that the restoring force is to the right when θ is negative, therefore, the force must be positive, and so we add a negative sign.
It works even when you assign the left as the positive direction.
Part of why I didn't understand you was because I didn't actually understand displacement. Displacement is position away from the equilibrium. It is positive when the bob is to the right of the equilibrium and left when the bob is to the left of the equilibrium. For some reason, I thought displacement was instantaneous velocity.