Hey Mapes,
Sorry for the slow response - I had to mull over what you said. It definitely makes sense what you're saying. Essentially the gas will always equilibrate slower (even if only a picosecond) than me removing the brick. Due to this, the gas will need to use some of it's internal energy...