The Tortoise-Man said:
Could you elaborate how to deduce from requirements on positive definiteness of the energy eigenvalues and microcausality that the coefficients/exponents in
are choosen in the "right" way? Why another choice would lead to a contradiction?
The positive definiteness of energy tells just that ##\langle s | \hat{H} | s \rangle >0 ## for every state ## | s \rangle >0 ##. How does it help here?
You should do the calculations for yourself. It's not so difficult and relys just on basic properties of the Fourier transform.
What might help though is to introduce the proper eigenmodes of the Klein-Gordon field (which are important building blocks for all other kinds of fields too since after all all free-particle equations must fulfill the Klein-Gordon equation since it fixes simply the eigenvalue Casimir operator, ##\hat{p}_{\mu} \hat{p}^{\mu}=m^2##, where ##m^2 \geq 0##, which is one piece to define an irreducible unitary representations of the proper orthogonal Poincare group, which defines what we call "elementary particles"):
$$u_{\vec{p}}(x)=\frac{1}{\sqrt{(2 \pi)^3 \omega_p}} \exp(-\mathrm{i} p \cdot x)|_{p^0=+\omega_p}.$$
These mode functions have the important properties
$$\int_{\mathbb{R}^3} \mathrm{d}^3 x \mathrm{i} u_{\vec{p}}(x) \overleftrightarrow{\partial}_t u_{\vec{p}'}(x)=0, \quad \int_{\mathbb{R}^3} \mathrm{d}^3 x \mathrm{i} u_{\vec{p}}^*(x) \overleftrightarrow{\partial}_t u_{\vec{p}'}(x)=\delta^{(3)}(\vec{p}-\vec{p}'),$$
with help of which you can express the ##\hat{a}(\vec{p})## and ##\hat{b}(\vec{p})## in terms of the field operators ##\hat{\phi}(x)## and ##\hat{\phi}^{\dagger}(x)## using the expansion in terms of these mode functions. Here the notation with the double-arrow means
$$A(x) \overleftrightarrow{\partial}_t B(x)=A(x) \partial_t B(x)-(\partial_t A(x)) B(x).$$
The very idea, why you necessarily need a quantum-field theoretical formulation of relativistic QT is the argument that you should have "microcauality", i.e., that local operators, that represent observables, commute with space-like separated arguments. This must hold particularly for the Hamilton density (=energy density), which is a sufficient condition for the S-matrix elements being Poincare covariant and the S-matrix unitary (for ##m^2 \geq 0##, for tachyonic irreps you run into trouble with causality as soon as you consider interacting fields). That ##\hat{H}## should be bounded from below is necessary to ensure stability.
Now it turns out that (at least for the free fields) the local observables like energy, momentum, angular momentum, various charge densities are bi- or sesquilinear functions of the field operators. Thus you can realize the microcausality condition with both bosonic (taking the equal-time Poisson brackets of the classical theory to equal-time commutators of the field operators of QFT) or with fermionic (taking the equal-time Poisson brackets of the classical theory to equal-time anti-commutators of the field operators of QFT), and there are topological arguments for the necessity to realize indistinguishability of particles in terms of either bosons and fermion if you have 3 or more spatial dimensions.
Now the recipe to build relativistic QFTs is to first look for unitary irreducible representations of the proper orthochronous Poincare group, generated by local field operators (the latter restiction to be able to realize relativistic causality by using microcausality together with local interactions), which leads to the fundamental properties of these representations described by mass and spin as well as conserved charges.
You can start with classical field theories, but it's quickly clear that you can't find any non-trivial unitary irreps in the sense of "1st quantization" for the proper orthochronous Poincare group, but you can use these fields in the action formalism to "canonically quantize" them. This works straight forward for the cases spin 0 and spin 1/2, i.e., to the Klein-Gordon fields as well as Weyl (and taking also parity and time reversal as probable symmetries into account) and Dirac fermions (also spin 1/2).
For fields with higher spin you need to impose constraints to project out the pieces with a definite spin. E.g., when realizing a massive spin-1 field with a four-vector field ##V^{\mu}(x)## you have to project out the spin-0 part of this field by imposing the condition ##\partial_{\mu} V^{\mu}=0##, which leads to trouble in the canonical quantization formalism, but this you can get rid of using some modfications of the canonical-quantization concept, which is only a heuristical method anyway.
For the case ##m=0## it becomes even more complicated since for fields with spin ##\geq 1## you necessarily need to formulate the theory as a gauge theory to avoid "continuous spin-like degrees of freedom", which leads to the conclusion that such massless fields have only 2 helicity degrees of freedom, i.e., having helicity eigenstates (the projection of the total angular momentum of the particle to the direction of its momentum for the momentum eigenmodes) ##\pm s## (where ##s## is the spin).
Now it turns out that you can fulfill the microcausality condition together with the postive semidefiniteness of the Hamiltonian only if you quantize integer-spin fields as bosons and half-integer-spin fields as fermions, which is the celebrated spin-statistics theorem (Pauli 1940).
It also follows that for each particle there must be also the corresponding anti-particle (which can be both identical if you consider "strictly neutral particles" like photons), and that despite the necessary symmetry under proper orthochronous Lorentz transformations also the "grand reflection" CPT (charge conjugation, parity, time reversal) must be a symmetry.
For a very clear treatment in this pretty heuristic chain of arguments, see
S. Coleman, Lectures of Sidney Coleman on Quantum Field
Theory, World Scientific Publishing Co. Pte. Ltd., Hackensack
(2018),
https://doi.org/10.1142/9371
Large parts of this are also available for free (legally). E.g.,
https://arxiv.org/abs/1110.5013
For a more systematic approach, considering carefully all unitary irreps. of arbitrary spin etc. see
S. Weinberg, The Quantum Theory of Fields, vol. 1,
Cambridge University Press (1995).