If your arm is perfectly efficient, that means that when you lift the block, you transfer potential energy to it, and when you lower the block it must transfer that energy back to you somehow. The energy is just transferred from you to the block, and then back to you.
I think what's probably confusing you (which confuses a lot of people) is the fact that humans are inherently inefficient, and so it's hard to gain intuition for these things by thinking about the way you personally interact with objects. Humans have to expend energy to supply force, even if they're not doing any "work" in moving an object. That's why it seems intuitive that moving a block up and then back down takes energy when in fact you've done no work on the block. If your body was perfectly efficient, you wouldn't expend any energy performing that action, you would just lose some energy for a little while. Of course, in real life our bodies don't behave like this at all, and so it seems really odd.
Edit: In case it's unclear, I'm NOT trying to say that conservation of energy only works in ideal cases, I'm just saying that as soon as you ask "Well, what if my arm is perfectly efficient?" you're creating a situation which is nothing like a you lifting a block, and so imagining yourself lifting a block will most likely just confuse you.