maline said:
But for a charged particle, the lagrangian is not invariant under that transformation- it changes by a total derivative. The same is true for the Galilean transformation in classical dynamics. Why do we demand actual invariance here?
Let's go through the Lorentz-invariant terms that we can have from scratch. At linear order we have ##j^\mu A_\mu##, which is how we include the current. The gauge variation of this term is ##j^\mu \partial_\mu f= \partial_\mu (j^\mu f) - f \partial_\mu j^\mu##. The first term is a total derivative, so that's ok. For the 2nd term to vanish, we require that the current is conserved, ##\partial_\mu j^\mu##. So we get some additional physics out of requiring gauge invariance.
We could also have linear terms with some derivatives, like ##\partial^\mu A_\mu##, but all of these terms are total derivatives themselves, so we don't have to bother with them.
At quadratic order, we could have ##A^\mu A_\mu ##, which has an interpretation in QFT as a mass term for the photon. The gauge variation is, keeping all terms,
$$ 2 A^\mu \partial_\mu f + (\partial_\mu f)^2. $$
We could integrate these terms by parts to look for conditions for which they vanish up to partial derivatives, but the 2nd term would require ##\partial^\mu \partial^\mu f =0## and we're not allowed to put any restrictions on the function ##f## if we want true gauge invariance of the theory.
Similarly, with two derivatives, we could consider
$$ L_{a,b} = a (\partial^\mu A^\nu)(\partial_\mu A_\nu) +b (\partial^\mu A^\nu)(\partial_\nu A_\mu),$$
which has gauge variation
$$ (a+b) [ 2 (\partial_\mu A_\nu) \partial^\mu\partial^\nu f + (\partial^\mu\partial^\nu f)^2 ].$$
Again, there is no way to make terms quadratic in ##f## vanish without imposing restrictions on ##f##, except in the case where ## b = -a ##, which is the case we had before with ##(F_{\mu\nu})^2##.
We could consider other terms, but it's clear that they will suffer from the same problems unless we can write them in terms of an actual gauge invariant like ##F_{\mu\nu}##, or introduce some new fields that participate in the gauge transformation, as I mentioned earlier.