I didn't just want to quit. I did quit. There was a period in my undergrad when I intended to do a double-major in physics and math and then get a PhD in physics. Then I took intermediate classical mechanics. I did well in the class, but I thought it was the most awful thing ever, so I went in the direction of mathematical physics, hoping to pursue a more mathematical approach to things. I did find what I was missing in the classical mechanics class, and, as I had known from the beginning, I was proven right that there was actually a meaning behind it, rather than just ugly equations all over the place and random, unmotivated concepts pulled from out of hats. So, that was successful, but I got dragged way too far from physics into topology by the department and my adviser and my own mistakes, got disillusioned with math and academia in general, and now I'm looking for work in other fields. I still plan to write/make videos about math and physics on the internet, and in my opinion, that is going to be a lot more useful than an academic career would have been, in my case. So, in a way, I haven't given up.
Keep in mind, when you know a subject, a lot of things seem easier to you than they actually are. There's a social psychology experiment you can do to demonstrate this. Tap the beat to a song. Tell one group of people what song it is, and don't tell another group. Even if the song is very famous, chances are the people who were not told will have a hard time guessing what it is, but they seem almost stupid to the group who has been told. So, there is this effect, and maybe that's why you seemed stupid to the professor. He's not in a good position to judge that very well, really. He can compare you to other students, but I think he might still not be on very solid ground, anyway, because who knows what the reason is that you aren't getting it.
Another way to think of it might be something like this. Say you meet someone who has never heard of the Pacific Ocean. That's pretty shocking (I have actually heard of such a person, though I can't remember which ocean it was). Maybe they were living under a rock. But you can just tell them what it is, show them a map, and then they are back to normal. So, it seemed really bad, and it was, but it's not that hard to fix. Of course, you'd suspect that they are missing a lot of other stuff (and maybe had some kind of underlying memory problems, as well). But if they applied themselves and made sure to retain the information, which is really just a matter of applying yourself and knowing a thing or two about how memory works, they could become a normal person again, if they put in the effort.
On the other hand, I remember in elementary school, being introduced to algebra for the first time, and a bunch of people said it was hard, but I was wondering what they were talking about and thought it was really easy, despite not having any more starting knowledge than they did. Even this advantage could be a bit of an illusion. When you give people enough practice to fill in the holes in their background and build their skills up, these sorts of differences can disappear to some extent. Sometimes, people who are actually pretty good just run into some stumbling blocks and then don't do well until they are overcome.
