Just because you have a difference of up to 3 volts at the outlet does not mean they are not connected to the same place in the panel.Supposed to be connected at the panel. But, I have found this to NOT be the case on more than one occasion. Have seen up to 3 volt or so potential difference between neutral and local ground. This is a simple and useful check to make rather than committing to an unproven assumption.
Assuming the above, and you connect the right wires, the system will operate fine. Not necessarily best practice, but will not negatively affect system operation.