I'm not sure about Physics specifically but in the Physical Sciences at my school there are a ton more opportunities (in the way of scholarships, REU's, honors programs etc) for minorities than there are for a typical middle class white. Even though College is so damn expensive that the playing field is pretty much level unless you are actually rich, IOW middle class people can't afford to pay out of pocket for school anymore than a completely poor person.
I don't want to get on a soapbox or seem racist or jaded but I know lots of kids who did paid summer internships in top-notch labs/schools (Harvard etc) or were in Honors programs which were geared specifically toward minorities or women etc, without really having much in the way of grades, knowledge or motivation/desire (NB for that last part, I'm not generalizing, this has been my experience with the handful of people I've met at my school in my life, YMMV).
Point is, from where I stand, being a minority actually gets you a leg up as an undergrad, which really sets the foundation of what opportunities are available to you as a Grad student or Physical Scientist.
So don't worry about it, just let your passion and drive guide you. The rest kinda tends to fall into place (I had to learn that the hard way).