Don't average them. Just make each sphere-sphere collision generate its own force. Add forces from all spheres that belong to a cube to get total force, and add all torques from each rxF to get total torque.
I think, that might be the problem. If the two spheres on opposite sides touch, if you just average point of force application, there is no net torque when cubes try to rotate through each other. If you compute torques first, then add them, the side that overlaps more will be the one providing dominant torque, and that should help.