If you go Upstairs, you are increasing your gravitational potential with respect to where you started off. If you go Downstairs, then your potential is decreasing. In the same way, connecting a battery with its negative terminal at 'ground potential' (the black wire on your meter, perhaps), then your meter will show positive voltages at different places in your circuit.
If you connect the black wire of your meter to the positive terminal then the voltages you measure will all read negative - because you are measuring the potential relative to a higher value.