You don't need the removal of a phosphate to get energy. You can basically think of favorable chemical reactions as ones that break an unstable bond to form a more stable bond. For example, the bond between the second (beta) and third (gamma) phosphates in ATP is unstable because of the repulsion between the two negatively charged phosphate groups. Kicking off the gamma phosphate and replacing it with an OH group from water creates a much more stable bond.
In a similar fashion, the bond between glucose and UDP is a fairly unstable bond. Therefore, kicking off the UDP and replacing it with a bond to the OH group from another glucose molecule occurs favorably. In the language of organic chemistry, the UDP acts as a good leaving group and is required for the reaction with the growing glycogen chain.
UTP is required because it is even more unstable than UDP-glucose. The UDP-glucose bond can form because it is more stable than the bond between the beta and gamma phosphates in UTP. Note that the reaction is UTP + glucose --> UDP-glucose + phosphate (so no pyrophosphate is produced).
Kicking off the OH group from a normal glucose molecule to form a bond with the OH group of another glucose molecule would not occur favorably, which is why UDP-glucose is required for glycogen synthesis in the first place.
