35 years ago I was in the same position. I eventually got into a graduate school (not a top 20 but a respected one). Here's what worked. First off, I worked hard to make Dean's list my last semester. This actually brought my GPA to only 2.3. (I also did an REU my last semester), I had talks with professors before the last semester telling them of my intention to do well in courses. They practically freaked out.
I got 55% on the physics GRE. Remember taking it on a Winter's day. Bus was late, I did not drive. My shoes were soaking wet throughout the test and for several hours afterward. 55% is not too bad.
I took a job for one year . While working I knew I needed good recommendations. I was making good money. I went to the school and told them I would pay tuition to take Math Methods in Physics at the graduate level (in 1978 this cost only 500 dollars). Later I asked the professor ( he is extremely well regarded ) of the course for a good recommendation. He gave me one (I got a B+ in the course and worked full time and overtime when my employer required me, I had no choice ).
Later the admission committee told me working a year showed maturity and doing well in a graduate course and making Dean's list attested to my continued commitment.
With all that, all three schools I applied to accepted me. One did not have a PhD program at all but I found out they could allow me to transfer to a doctoral program at a nearby school, if I did well. Instead I went to the best one that accepted me
A year later I was in the doctoral program after having passed my doctoral qualifying exams, one of three to pass about ten more failed. It was a lot of tough work but it was doable.
I hope it can be done today but remember this was 35 years ago. I wish you well.
But here is the problem: The jumping around you are currently doing ,I did in graduate school. Eventually, even though I passed the exams and did well in coursework, it was better that I leave. I wore out my welcome.
Luckily I found employment, that even allowed me to continue my graduate study (one course at a semester).
The second time through graduate school (almost 20 years later), I was more successful. Had to take all those exams again. I committed to one research group no matter what.
Moral of the story: Graduate school may be achievable but you may have to adjust your timetable. Ask yourself if you want to do it. The "perfect" research group probably doesn't exist. I know I never found it.
I know I only saw the glamorous side of successful physicists. I did not see the difficulties in grant-seeking, the years of post-doctoral study required, the difficulties in publication. Many principal investigators and leaders of the research group need to be excellent managers but cannot even keep their hand in the day to day calculations the members do. You may not want to commit to this career.
I have no regrets but I think I should have kept my eyes open (as a few senior physicists suggested).
Good Luck