You were already given the correct answer.
It's not swinging in a vacuum. The energy is dissipated through friction with the air (drag) while it is in motion and through heat as whatever holds it up is flexed. Like you said, energy must be conserved and HAS to go somewhere. Since the pendulum only interacts with the surrounding environment, then that is where it has gone.
Don't put your own interpretation onto conservation of energy because you're taking an introductory physics class. Friction is a NON-CONSERVATIVE force, meaning that you cannot consider the motion of a pendulum in the real world to be path independent. The path that it travels causes drag, and a reduction of the energy within the pendulum.