I am a bit confused about gravitational potential energy near the earth, namely the formula given by mgh. I know that potential energy is defined as U(x) = -W(x0 to x) where x0 is our chosen reference point. Let's take the earth's surface as the zero point and let's travel upwards to a point x. Since mg is pointing down and my displacement is pointing up, the dot product is negative so W(x0 to x) = -mgx. Therefore, U(x) = -(-mgh) = mgh. however, let's say i want to travel downwards instead. then since mg is pointing down and my displacement is also pointing down, the dot product is positive this time so W(x0 to x) = mgx (x is negative). but then U(x) = -(mgx) = -mgx. If i say travel downwards from x = -2 to x = -4, i get U(-4) - U(-2) = mg(4) - mg(2) > 0, which doesn't make sense since if i am going downwards, shouldn't my change in potential energy be negative?