Very true. I never really had significant troubles in my undergrad, and I think that is due to our education system here. We actually saw set theory in elementary school! Well, not formal proofs of course, but they made us familiar with things like intersections and unions on a very basic level. So by the time I got to university, the language wasn't alien to me.
Here in my country, a freshman math student starts off by taking real analysis and abstract algebra. I was quite shocked the first time I found out that these are considered very difficult courses. Since I don't think that American students are less capable than European students, I think the only conclusion one can make is that the high school education (and before) is severely lacking in most US schools. For example, set theory and logic is never really covered there.
Anyway, what I want to say that it might be very benificial to learn basic set theory and basic logic. This will help you a lot if you study pure mathematics. That said, it is a rather boring subject, so don't be put off if you don't like it very much.