Yes, that is a problem. I just recall reading that a certain (apparently significant) contributor to Paul's effort was pro confederacy. But since I don't remember the source, then it's not a significant or arguable point, and, unless it can be documented, then it should be disregarded.
Here's an interchange between Paul and a newsperson on the civil war:
Paul's responses here seem pretty reasonable to me.