At most institutions that I know of, student "note-takers" are assigned students that are concurrently taking the class... so their qualifications on subject content should be good. Generally, it's also seen that cooperative learning experiences (in many forms.. from working on homework together, discussing lectures) help students master subject content... so in your case, you'd have good excuses to set up these experiences. Most physics and math departments are small, so you'll probably from a good connection with the other students and have no problems.
Also: you'd be surprised how willing people are to work with you in your department -- and not just via legal obligation. While some people might be uncomfortable around disabled people... there are a number of us out there that have lots of experience and connection with at least one disabled individual. When I was a child, my church had seminarian (priest in training) who was mostly deaf. On the academic front: one of our former graduate students was employed as an undergraduate student as an aid to a physically disabled faculty member (taking care of personal needs as well as helping out in the classroom with demonstrations, AV equipment, etc.). Closer to home: my stepson has very involved CP -- so my husband and I, both faculty members, are particularly sensitive to the rights and issues of the disabled... even though our son will not ever attend college level due to both interest (he' much rather just ride the bus!) and functional abilities (such as literacy).
Of course it depends on your willingness to "throw yourself out there" as well as the severity of your condition. You might prefer to keep your disability private, which is certainly your right and an understandable position. In our case, there's no keeping our son's condition (with his bulky electric chair and tendency to go on and on about buses) private unless we hid him away... which wouldn't be our style especially given his social nature.
I'd discuss these things as you look into which institution to attend. Set up private meetings with both admissions and with the departments (get meetings with faculty as well as department chairs). Trust your instincts on their attitudes when making your decision... as well as considering their other strengths (as a research opportunities, etc.).
Lastly, I'll make a note on your statement regarding later employ in the military. While there might be restrictions on enlistment, you should still be able to hold a civilian position in say, a military lab. I worked in the Air Force Research Labs as a civilian, and I'd say at least 1/2 to 3/4 of the researchers in the lab were civil servants... not enlisted military.
I just have to remember to speak a bit louder and try to adjust the pitch of my voice a bit lower when speaking to those people.