I suppose a different start, instead of introducing virtual particles right away, is to understand the nature of quantum and thermal fluctuations in the mathematics a free quantum field theory. For a quantum field ##\hat\phi(x)##, which we might think to use as a model for measurements, the variance of measurement results is infinite because it is an operator-valued distribution.
If we consider an average, however, weighted by a function ##f(x)##, ##\hat\phi_f=\int\hat\phi(x)f(x)\mathrm{d}^4x##, then ##\hat\phi_f## is an operator instead of being an operator-valued distribution, so we can use it as a model for (very idealized) measurements, for which we find that there is a finite variance. In particular, the variance is proportional to Planck's constant, but it's also a quadratic functional of a real function ##f(x)##, ##\mathrm{Var}_{\mathrm{Vacuum}}[\hat\phi_f]=\hbar\int f(x) G(x-y)f(y)\mathrm{d}^4x\mathrm{d}^4y##, where ##G(x-y)## is a Lorentz invariant function of the distance between ##x## and ##y##.
If we introduce thermal fluctuations into a free quantum field theory as well, then ##G(x-y)## is no longer Lorentz invariant, and kT makes an appearance, but the structure is in broad terms the same, so quantum fluctuations and thermal fluctuations for the free field can be thought of as quite similar, but different in a very specific way.
When we introduce interactions as well, the point there is to modify ##G(x-y)## in various ways, but also to add higher-order correlations between different measurements. Things get complicated because of renormalization, but we can say it's always just statistics of a noisy field. The virtual-particles talk is just a way to talk about that noise in terms of particles, but we can also talk in terms of a noisy field theory (really, that it's noisy is crucial!)
Everything above is in terms of mathematics. As always for mathematics, any given model may or may not be a good model for a given experiment. Indeed, a free quantum field theory is not a good model for experiments, but we can deform it in various ways to give some pretty good models. Keep in mind that some global measurements are not noisy, but local measurements in general are (indeed there are theorems proving that no local measurement of the vacuum state can be noise-free, for some choices of axioms.)