This hit close enough to home that I am prompted to come out of my long withdrawal from physics forums. Very rarely I come back and read a few posts just for kicks. I have a PhD in math (in a physic-sy area of math) and have worked as a software developer. Backend until recently. I'm now involved with an early-stage startup in some sort of an ambiguously defined role that at the moment mostly resembles a full stack developer, although still more on the backend. My reaction to the posts is not complete disagreement, but I think the story is slightly more complicated.
The first point I would bring up is that when you leave your academic career, I think you have to grapple with the distinct possibility that your old physics and math interests are just going to become a hobby. From that perspective, I think your average programming job has something to offer in the sense that your work is built on top of so many layers (computers and computer networks) that do involve tons of math and science. So your hobby can at least have an interesting connection to your work. I think that's something that's at least worthy of note.
Having said that, you can ask the question of whether the relationship between the hobby and the job is actually a meaningful one. If you are a race car driver with a mechanical engineering background, would you just find the link to be comforting or could you possibly get some mileage out of your background? Does the analogy carry over to the programming case? I think it can take a lot of work to get the hobby to spill over into the job. Another added complication is whether you want something that really has the specific flavor of physics/math that you studied in grad school versus any old physics/math. If you are open to learning new stuff that you didn't necessarily study in grad school, there is stuff like theoretical computer science that's a little closer to your work. To argue this a little bit more convincingly, let me note that when Brendan Eich created Javascript, his idea was to create Scheme in the browser. And from there I can just recommend that you read Structure and Interpretation of Computer Programs to see the meaningful connection between lambda calculus from mathematical logic and program design on the other hand.
I heartily recommend data science for anyone who happens to find it appealing, but I don't think it's a good assumption that it's some kind of free pass to a good mathematically-oriented job that will satisfy all of us. It may be hot on the market, but it may or may not be hot as far as your intellectual tastes go. I know mathematicians who took that route who found it not to be that satisfying. I mean if you're anything like me, I would choose working on some physics engine for computer games over data science in a heartbeat without even having to think for a fraction of a second. Unfortunately, the reality of the job market is that the game engine is a very small niche, whereas data science is everywhere.
But that brings me to my final point which is that the job market out there has a lot of little niche things that you can luck out on. Things are less formal in the startup world. People tend to wear all hats. So, the full stack developer skillset is a pretty big asset there, even if it's not going to be your main focus. At the moment my work is pretty much standard full stack development, but if the startup can hire more people to free up my bandwidth, the plan is for me to be more of a research and development guy eventually and try to tackle some heavy-duty computer science problems.
It's up to you whether you try to gamble on the road less traveled and see if you can strike career gold. To be fair, though, your background with numerical analysis is a little closer to machine-learning than mine. I tried to go more towards marketability because I am just not good at and do not have the patience for complicated job searches, but I could only fight my own nature to a limited degree. So, I just had to improvise and find my own path and play with the hand I was dealt. There were no easy answers. You can always try to pursue the path that I and many of my friends have taken and try to become independently wealthy so that you are free to do whatever work you want without having to worry about getting paid for it.
So, anyway, I'm sure there is some startup out there where things could work out well with full stack development on some math or physics-oriented project, but you just have to acknowledge the risk that it's a gamble that you might not find it. So, you might have to be okay with that possibility.
Oh--and I forgot to comment on how you mentioned the lack of scientific approach. I don't see why you couldn't take a scientific approach to software development. It was Dijkstra's vision to try to use proof to get correct programs, and that failed, except in a few niche instances. Instead, the focus has been on testing. Essentially, I think software developers should be held accountable to demonstrate that their software actually works, but the usual approach is an experimental one where you run the program and check that it delivers the expected results. Maybe you mean you'd like a more theoretical approach, and in that case, maybe you should look into functional programming.