From a practical standpoint, cooling this way will be limited by your ability to insulate your regrigerator and its components. The physical process doesn't have a minimum working temperature (besides absolute zero).
If I understand what you're talking about, that process would be the Joule Thompson process (used in refrigeration). In this case, there is no work done on the gas, and the cooling is due to the intermolecular forces between individual molecules in the gas.
Most cold gases will cool further under expansion (through a valve) because as the atoms get further away from one another, the average potential energy increases (because far away, the force is attractive). Since total energy is conserved, the average kinetic energy decreases, and the temperature goes down.
However, this cooling doesn't occur at all temperatures. At temperatures above what's called the inversion temperature, the reverse will happen. This is because the average (in time) potential energy depends on the rate of collisions.
At higher temperatures, the atoms are moving faster, and collide with one another more often.
If a pair of atoms are close enough to each other, the force between them will be repulsive, and the potential energy will have a net decrease when they are far away enough from each other again.
If the cloud of atoms are on average close enough to each other, expanding the gas will lead to a net decrease in potential energy, and a net increase in average kinetic energy (assuming energy is conserved)
This inversion temperature is different for each gas, and is usually found experimentally.
For example:, the inversion temperatures of nitrogen and oxygen are over 500K, so you could keep cooling them until they liquify starting at room temperature (provided good insulation).