First I want to make clear that Quantum Mechanics cannot be derived from classical mechanics (including statistical mechanics and relativity); it requires some fundamentally new postulates/axioms. Different books start out with a different set of postulates and axioms, so it's conceivable that some books do not derive that formula for the expectation value, but rather state it as a postulate. For a thorough derivation of that formula, I'd refer you to a more mathematical introductory text on QM, such as Shankar's Principles of Quantum Mechanics. Granted this won't be very intuitively satisfying.
However, I can give you a non-rigorous analogy that might help you understand.
In statistics, if P(x) is the probability distribution function for the result of a measurement of some random variable x, then P(a) is the probability that a measurement of x will result in x=a. Certainly since a must be one of x's possible values,
\int_{all \ x}P(x)dx=1 This is the normalization condition.
Now for some function f(x), the expectation value of f(x), or the average of f(x), written <f(x)>, is given by: <f(x)>=\int_{all \ x}f(x)P(x)dx This is basically a "weighted average." [From here on out, just assume any integral is taken over the entire range of possible values.]In quantum mechanics, a system's state is a ket |ψ> in a Hilbert space, and Hilbert spaces come with the L2 norm. The adjoint of |ψ> in Hilbert space is <ψ|. In math we don't use bra-ket notation, we'd use something like ψ instead of |ψ> and ψ* instead of <ψ|. The normalization condition would be 1=<ψ|ψ> and the inner product between states |χ> and |ψ> is given by <χ|ψ>; these are just fancy ways of writing a basis-free (abstract) L2 norm. (Note: I'm ignoring spin.)
Physicists often talk about the "Statistical interpretation" of |ψ>; we like to think of it as something similar to (the complex square root of) a probability distribution. [Here's where you'd like to check out Shankar for the in-depth treatment of the representation of observables by hermitian operators, and their associated eigenkets providing an orthonormal basis for the Hilbert space.] Projecting |ψ> into the position basis {|x> | x is an element of ℝ3},
∫|x><x|ψ> dx = ∫ψ(x)|x>dx
gives the normal spatial wavefunction ψ(x). Notice that (ψ(x)|x>)* = <x|ψ*(x)
The normalization condition can be written
1=∫<x|ψ*(x)ψ(x)|x>dx=∫ψ*(x)ψ(x)<x|x>dx=∫ψ*(x)ψ(x) dx
So now we are seeing similarities between P(x) and ψ*(x)ψ(x).Remember that in statistics, <f(x)> = ∫f(x)P(x)dx
We might naively assume that for some observable Q, the expectation value for Q would be
<Q> = ∫Qψ*(x)ψ(x)dx, but this is wrong. First of all, observables in quantum mechanics are represented by operators, which act on kets rather than on a scalar number like ψ*(x)ψ(x). To be technically more accurate, Q exists abstractly and acts on abstract kets in Hilbert space, so for Q to act on a spatial wavefunction, we would have to express it in the position basis. Going back to the abstract notation (no basis),
Q<ψ|ψ> is basically nonsense, since Q<ψ|ψ> = Q
The leap we have to make is realizing the correct equation must be
<Q> = <ψ|Q|ψ> = ∫ψ*(x)Qxψ(x)dx, where Qx is Q's expression in the position basis.
It's very similar to standard statistics, but because quantum physics represents observables with operators, we need to use the "sandwich" method. However, nothing I've said rigorously justifies the last leap here, so my argument is intended as an analogy rather than a derivation.
Edit: I realize I was a little sloppy in one of my equations (but of course the argument still is fine). Challenge to anyone checking my argument: can you find the sloppy equation? (And fix it?)
