I think the variational principle is just the best language we have to express the fundamental laws of physics that are found by observation, including high-precision measurements.
The variational principle is so suited to do so, because it admits a simple realization of the symmetry principles underlying the fundamental laws of physics, which themselves are a discovery based on observations. It all starts with the spacetime model and its symmetries. Fortunately we can neglect gravity as far as elementary-particle physics is concerned. Thus we can use the Einstein-Minkowski spacetime with its high symmetry. Just as a spacetime it has the proper orthochronous Poincare group as the part of the symmetry group that forms a Lie group of transformations continuously connected with the identity operation. From this already follow all possible kinds of fields by analyzing the unitary ray representations of the corresponding Lie algebra, leading to representations of the central extension of the covering group by exponentiation. In the semidirect product of spatio-temporal translations and Lorentz transformations, you simply have to substitute ##\mathrm{SL}(2,\mathbb{C})## for the proper orthochronous Lorentz group. As it turns out the proper orthochrnous Poincare group has no non-trivial central charges und thus all the ray representations are liftable to proper unitary representations (in contradistinction to the Gailei Lie algebra whose non-trivial central charge is mass, while the proper unitary representations of the classical Gailei group do not lead to physically sensible dynamics, i.e., the notion of "massless particles" doesn't make sense within non-relativistic physics).
The further analysis leads to the conclusion that among the possible field equations only those with positive or vanishing four-momentum square (which defines the mass of the particle related to the corresponding non-interacting field via ##p_{\mu} p^{\mu}=m^2 c^2##). In addition an irreducible representation is characterized by the sign of the eigenvalues of ##p^0##, and the spin (i.e., the representation of SU(2) in the "little group", which describes the transformation of the field under rotations, for spin-##s## fields implying ##(2s+1)## polarization degrees of freedom). Massless fields for ##s=0## and ##s=1/2## are simply described by the limit ##m \rightarrow 0## of the corresponding massive cases, while for ##s \geq 1## you need to represent the null-rotations within the little group trivially, leading to a gauge symmetry, which leaves only 2 out of the ##(2s+1)## polarizations of a massive particle, the states with helicity ##h=\pm s##. This defines the free particles as represented within a irreducible unitary representation of the proper orthochronous Lorentz group.
Then there are also the discrete space-time symmetries, i.e., space-reflections and "time reversal". Together with the assumption that any physical theory should have a Hamiltonian bounded from below, time reversal is necessarily realized by an antiunitary transformation.
As it also turned out the only successful relativistic quantum theories of interacting particles so far are all local quantum field theories, where the Hamiltonian is expressed by quantum fields transforming locally under Poincare transformations (as their classical pendants) and obeys the principle of microcausality (i.e., local observables commute at space-like separations of their space-time arguments). This together with the demand that the Hamiltonian is bounded from below leads to the necessity of antiparticles (with the possibility that particles and antiparticles might be identical for the socalled strictly neutral particles), the spin-statistics theorem (half-integer-spin particles are fermions, integer-spin particles are bosons), and the PCT theorem, according to which the "great reflection" consisting of space reflections (parity, ##\hat{P}##), charge-conjugation (exchanging particles with their antiparticles, ##\hat{C}##), and time-reversal (##\hat{T}##) must necessarily be a symmetry.
Given that mathematical framework, which of course has developed in a fascinating interplay between theory and experiment in the years from about 1928 ("Dirac equation") to the mid 1970ies (completion of the standard model by the discovery of asymptotic freedom of QCD), it is clear how to build the right Lagrangian in the action principle given the empirical input. The result is the standard model of elementary particle physics, which is the most successful theory ever, though it's considered incomplete since we don't know how to finally also incorporate the gravitational interaction within the framework in a satisfactory way and because we also don't know yet how to describe "dark matter" (who knows, whether it's really "particles" in the same way as the known ones).