I tried exactly what you tried, and to no avail. (switching two people at a time and moving one person at a time) I feel like it may be impossible, but by the way my prof was talking in class, it sounded like there was a solution! Hmmm...
He did mention using binary code and pigeon hole...