Hi SpaceNerdz. You asked for personal anecdotes, so I've got one.
The usual route for a tenured professor is: undergrad->grad school (PhD)->postdoc->maybe another postdoc-> assistant professor->tenure. Of course most people jump off this bandwagon at some point or another. I just completed my PhD in astrophysics last month, and in a couple weeks I'll be starting a new position as a data scientist for an energy company. My new colleagues will include people who studied computer science, statistics, and yes, even physics. Most of them have PhDs in their fields.
Why did I decide to leave the academic track? It certainly wasn't disgruntlement! I love astrophysics, and I don't at all regret my decision to go to grad school (I do think I should have chosen a different field for my undergrad…but hey, no point rehashing decisions I made in high school). I've had a blast doing novel research in a cool field and spending my days thinking about active galactic nuclei, shock fronts, and particle production mechanisms. But I've also spent most of my time each day at work sitting down at a computer, looking at data, and writing code. When I was on the job hunt this past year, I applied to postdocs, programming jobs, and statistics/data science jobs.
What I really liked when I interviewed for the data science job is that it had the same "feel" as what I was already doing. Basically I'll be looking at data, applying mathematical algorithms, and producing a final analysis product. That's essentially what I did in grad school. Only difference is that instead of thinking about active galactic nuclei, I'll be thinking about energy consumption (and the fact that what I'm doing helps the environment is definitely a nice bonus). I probably could have found a postdoc position if I waited a couple more weeks before accepting this one, but honestly I think I'm going to enjoy this as much as I'd enjoy a postdoc. It certainly doesn't hurt that they'll pay me almost twice as much as I'd make with a postdoc. But those of us who go to grad school usually operate on the philosophy that as long as you can make ends meet, doing what you enjoy is more important than making money. After all, you're going to be spending around 8-9 hours a day doing your job, so it would really suck if you're doing something you hate.
So I guess I'm one of those people your professor talked about who's leaving by choice rather than by force. That said, I would second what D H said about clarifying what "a job in physics" means. I'm guessing he meant to include postdocs and certain types of industry jobs, because if he was referring to tenure-track faculty positions he'd be completely wrong (and physicists are very careful not to make incorrect conclusions from our data!) If you look up the statistics on the AIP website, this will become clear. I remember reading the stats from 2007, in which it stated that there were about 1,400 new physics PhDs from American universities, but only 300 available faculty jobs in America. This data was presented as a time series, so the value from 2007 was pretty representative of previous years (actually it was getting worse). So you can make some conservative estimates and do the math. If most PhDs go on to do postdocs, and if you factor in that American jobs get foreign applicants, you've got about ~1,500 people applying for 300 jobs. So you've got a 1/5 chance of getting a tenure track professorship. I do remember reading somewhere that there are enough postdocs for every PhD student in astrophysics, but I don't remember the source.
I'd also second the notion that you should do a PhD because you want to study something you enjoy for 5-7 years, not because you want an academic job. Yes, at the end of the day you need some kind of job to pay the bills, and with a PhD in physics you'll almost surely find something. But the competition for academic positions is fierce, so this isn't an outcome you want to count on. Definitely try for it if academia is where you want to end up, I'm just saying that a tenure track job is not a sure bet by any means, so you don't want to restrict your job search to one type of position.