I alway thought it was heat loss. In thermoelectric devices, since the voltage is so low, you need low impedance to get lots of current. To get low impedance, you have to place the junctions close together and the material connecting the junctions wastes a lot of heat by conduction between the juncitons.
With thermionic devices, they radiate lots of heat. If you insulated them enough to catch most of the heat, the collecting electrode would get hot and become an emitter, itself, counteracting the emitting electrode.---I think.