RUTA said:
My question is, again, why is it that our fundamental theory of physics is a GPT? Is QM simply an incomplete version of something more fundamental that is non-probabilistic (as one views the kinetic theory of gases)? Or is the probabilistic nature of our fundamental theory unavoidable (as in QBism and our principle explanation)?
My reason is a model building answer. I used to say it was because, in building models, a generally liked thing is differentiability and continuity to use our powerful calculus methods. I used to refer to a paper by Hardy explaining that but now have a different approach.
In QM, what is going on, we do not have direct experience with, all we have direct experience with is how it affects the world around us. I will take a straightforward model of such a situation - the Markov chain of turning a coin over. We can't see the coin is turned over; all we can do is see, say every second, that the coin is heads, tails, heads, tails etc. The Markov chain for this is straightforward - anyone that has studied Markov chains can do it. Here in Aus, HS students study simple Markov chains like this. So we have a model for the system that gives the result each second. But wanting to improve the model, we ask what is going on at say 1/2 a second. We can't know, just like we can't know what is going on at the quantum level. But we know math. Let us take the square root of the Markov matrix. Low and behold, we find it is complex. The resulting state at half a second is also complex. But states, being probability vectors, can't be complex. We see the need to extend probability theory to cover complex probabilities and connect them to ordinary probabilities. That is where Gleasons Theorem comes in. It basically says there is really only one reasonable way to do it. We can develop a whole generalised probability theory that way - in fact, it is QM. We get the two axioms of Ballentine. As per chapter 3 of Ballentine, we progress further from the simple symmetry principle that the probabilities are frame independent (formally, of course, we are invoking the POR) and, again for simplicity, use Galelaian relativity. We get Schrodinger's equation. Now that is where the fun really begins. We apply Ehenfest's Theorem and note something interesting. It is the same as the Hamiltonian from Classical Mechanics. Have we discovered something profound? We can go a bit further by deriving the path integral approach that will show classical mechanics necessarily falls out via the Lagrangian (Landau does this in Mechanics). It emboldens us to do something daring. Let's take classical Hamiltonians and quantise them by replacing energy, momentum etc., with the operators in Schrodinger's equation. What do we find from experimental predictions doing this? We get results in accord with the experiment. We have discovered something strange about nature - it uses the generalised probability model suggested by our simple Markov chain model extension. People may ask the question - why is nature like that? It is all very reasonable - we have mathematically extended probability theory to see what happens. But why does it need extending? That I can't answer - future research may shed light on it - but we are forced to accept every model is based on assumptions. If you manage to explain those assumptions, what you used to explain them also has assumptions. It is never-ending. We also find that this model we call QM is also wrong. QM must be extended to QFT (e.g. it can't explain spontaneous emission). And so it goes. It is simply a never-ending quest for better and better models. Why is something we can't really know. That does not stop people from trying. But mostly, I find such attempts unconvincing. Just me, of course. There is no right or wrong answer to that conundrum.
This is very similar to Dirac. He played around with equations until he understood them inside out then tried extending them. Sometimes it worked, sometimes it did not. Feynman and Landau were also masters at it. They were not concerned with why nature was like that - it was enough for them. There is a psychological issue here - some like me are satisfied with this approach - others are not. While having an excellent command of the technical apparatus of physics, like Dirac and Feynman, Einstein actually used his physical insight rather than just playing around with equations. Feynman did so as well. Landau, I do not know enough about to comment.
Bohr and Einstein were of course concerned with the philosophy of this, but later physicists less so. Feynman was actually contemptuous. Personally, I am ambivalent.
Thanks
Bill