Hey Kutt and welcome to the forums.
I'm not in the US, but as a point of reference here, typically associate degrees are 2 years, bachelors are 3 to 4 years (depending on the degree and major), a full time load for university is recommended to include 40-48 hours of total time (including lectures, tutorials, assignments, reading, etc) a week for 13 weeks of face-time (with final exams shortly after).
For math, it depends on the university and the program you are entering. At my university a lot of the non-math majors have the option of taking the lower calculus sequence which covers the topics at either a lower level or a slower pace. Also the courses have prerequisites, but you should know that the way things are taught is different and you can get support be it from university lecturers or TA's and this is used frequently.
University level coursework goes at a faster pace than high school, but you have to remember that the university will have people that actually understand mathematics and have been doing it in one form or another for a long time (even if it's only teaching), so the learning experience is a lot better if you get people that understand it properly and can relay this knowledge to you. If you put in the work, I think you'll surprise yourself at how much you will be able to learn and people that are motivated do tend to go well in a university environment.
The other thing is that there are a lot of people who are just as smart if not smarter who can contribute their own thoughts and ideas on things and a key thing to remember is that if you get a lot of these people in the same room, amazing things can happen (and do happen).
At my uni I haven't taken any non-math subject (and I'm greatful that I don't have to), but I've heard the US system is a lot different.
As for hours, it depends: some students take as little as 8 or 9 contact hours per week in undergraduate and some as much as 25-30 (students doing engineering do this kind of thing). It is recommended that people spend about 40-48 hours time on stuff in total and I imagine some people spend quite a lot more time depending on the degree and how involved/serious/whatever they are with regards to the subject.
One thing to keep in mind is that expanding yourself is like exercise: at the start it's really tiring and probably pretty painful as well, but as time goes on, things do become easier and soon you don't think about the things that were worrying you before.
The thing is that the benchmark increases so your new pains will come about and just like before you slowly overcome them.
Mathematics is like any other human activity even though many people don't think of it this way: with an activity you get people who are interested and they communicate with each other. You get experienced people who mentor juniors and get them up to speed, you get explanations from senior people that turn the jargon into stuff that can be related to (i.e. all the greek symbols) and then through the process of growth, collaboration, and a lot of effort, you get better at it just like you would if you ran two days a week or went to the gym.
But like a lot of things, the stuff needed to get to the interesting parts is actually really boring: you'll spend a lot of time doing problem sets that get your mind thinking in a certain way just like athletes have to do routine exercises and drills over and over and over again so that it is natural when they actually want to do more stuff.
Think of a basketball player who is learning and spends hours each day and night just shooting hoops over and over again, and then link that to what is done with solving integrals and mathematical equations and you'll understand what I mean.
Personally I like math and how it let's me see the world in a whole different way, and it's good to pass on that perspective from time to time and a lot of people on these forums probably feel the same way about their own degree/career/whatever so if you take a calculus course or decide to go further, then remember that there are people who are enthusiastic about what they do and it certainly helps to know where to find these people if you are in a jam.
Good luck!