When you push against the table and your chair rolls backwards but not the table, then somewhere there has to be friction. The coefficient of static friction is NEVER less than the coefficient of kinetic friction or rolling friction between two surfaces. As a result, if not enough force is applied to overcome the coefficient of static friction, then that object (in this case, the table) will never move.
The reason why the chair rolls back is because the coefficient of static friction between the floor and the wheels is less than the coefficient of static friction between the floor and the table legs. Therefore when you apply enough force to overcome the wheel and floor coefficient of static friction, the wheels move backwards! Now instead of static friction, we are dealing with rolling friction which is a very small coefficient for a lot of surfaces against each other. This is why when you push against the table and you move backwards, you seem to move backwards with a considerable amount of velocity relative to how much force you applied on the table.
But where does the force you applied on the table go? Obviously it has to go somewhere! This is where work comes into play. Our system loses mechanical energy, but gains thermal energy, as it must due to the law of conservation of energy. Because we have a non-zero change in mechanical energy, there must be non-conservative work being done to the system. Our non-conservative work is made up of dissipative work and external work, which are done by the system on the environment and the environment on the system, respectively. Friction generates thermal energy, so we have negative work done on the system and positive work done on the environment. As a result, the energy you put into the system was turned into thermal energy, which ended up being transferred into the environment (in this case, the floor). The same process is happening to the chair, but at a slower rate due to a low coefficient of rolling friction. This is why the chair slows down, because the kinetic energy is being constantly turned into thermal energy, which is non-conservative work, which is work done on the floor!