An answer to that question was provided by von Neumann's approach to the mathematical formalization of QM.
In this approach, the central objects are the yes-no propositions one can make about a system, i.e., propositions about the system that only admit an answer of yes or no (e.g., P="the position of the particle is x=5"). The porposition by itself is not enough, you have to ask: "when the system is in the state S, what's the truth value, or at least a probability, of elementary porposition P?". In classical mechanics, the set of all these propositions can be seen as equivalent to the Borel sigma-algebra of phase space, which can be seen as a distributive lattice (a 'Boolean algebra'; I'm simplifying things, there are more assumptions it its definition!) A key property in this structure is that all elementary propositions are compatible, i.e., given any two propositions, is meaningful to ask about their truth values simultaneously: the classical logical connectives 'or' and 'and' are well defined for these pairs.
A key property in QM is the existence of incompatible elementary propositions: for them, is meaningless to ask about their truth values simultaneously. Clearly, it becomes problematic if we want to use the classical Borel sigma-algebra for modeling the set of elementary propositions in QM. How do we realize explicity the set of elementary propositions in QM then? von Neumann's brilliant insight was that we can see them as orthogonal projectors on a Hilbert space: compatible propositions correspond to commuting projectors and incompatible propositions correspond to non-commuting projectors.
By assuming certain properties on the lattice (the quantum one), one can ask if the Hilbert space realization is the only possible one (up to lattice isomorphisms, of course). The answer is involved and I'm not an expert on it. I think you get some kind of 'generalized Hilbert space' over a division ring. Further assumptions presumably lead to R, C or H (the quaternions).