Frank Castle said:
What I'm confused about is what is the actually motivation for why this should be a universal quality of physical systems, i.e. why is it reasonable to insist that physical systems should be assigned an action and that the actual path followed by the system is an extremal of this action (other than that it works)?
So your problem is what is the motivation etc. for δS = 0 (principle of stationary action).
First of all, it's a postulate (principle), used to achieve an alternative equivallent
Formalism in classical mechanics. But further it associates with giving deep insights into physics, by offering a generalization for many theories in physics, i.e. a general equivallent method-formalism to derive in an alternative way the equations of a theory. It is used even in Electromagnetism, Gravity, Field Theory etc. and ultimately even in Quantum Mechanics (and Quantum Field Theory) by resulting into R. P. Feynman's Path Integral formulation of QM. (See also vanhees71's nice comment above.)
So I would say that the motivation is first of all a formalistic one, i.e. to have a generalized alternative approach-method for deriving formalisms in physics. [
Variational Principles in physics probably started, as you said, with Fermat's principle in optics, that's why (since 1989) such similar approaches (and more) are awarded with the so-called
Fermat prize.]
From one standpoint, there is nothing mysterious about the concept of
Action (units: [energy]x[time] or [momentum]x[length] ) and
the principle of least (or stationary) action , other than that it formalistically works,
as long as you choose the right Langrangian (or Hamiltonian, in the Hamilton formalism) of the system. I think that's the whole trick here!
But of course, it is not a coincidence that
planck's constant (h bar), the elementary quantum of action, is also an 'action' with the same units, enabling the uncertainty principle ...
And the "
deep physical reason ..." :
vanhees71 said:
There's a deep physical reason for the Hamilton principle of least (or better stationary) acion from quantum mechanics in the path-integral formulation by Feynman.
is indeed the only complete physical interpretation that I have also seen in trying to physically explain the action and the principle of least (or stationary) action. But the problem with it is that it came a lot later, and it assumes knowledge of Quantum Mechanics ...
Now, what is the intuition behind action and the principle of stationary action?
I have no idea, other than the above, or time and energy minimizings and related variational problems and principles ...