There was something similair when (hand held) cell phones were banned while driving in the UK. People challenged it on the basis that cell phones are no worse than a passenger or radio.
I think the conclusion was that in talking to someone who is not there you use up more visual processing bandwidth picturing them. While people in the car with you pick up queues about how busy/task-loaded you are and stop talking when you have to concentrate - even non-drivers do this by picking up on your body language.
The ironic part was that it seems hands-free phones are worse than holding a cell phone. Speaking into thin air to someone who isn't there takes even more processing power than talking into a phone.