I come from a ridiculously dysfunctional family, so I see red flags all the time. They almost always amount to nothing, but I see them all the time. That's the only reason I mentioned co-dependency.
But, if that's not an issue, then you have to follow your heart, and you'll have to have a talk...