So in computing the variance-covariance matrix for β-hat in an OLS model, we arrive at
However, I'm incredulous as to how X is considered non-stochastic and how we can just eliminate the expectation sign and have
I'm accepting this to be true (since it's so written in the text) but I'm taking a leap of faith here: if this is true, the elements in the VarCov matrix are expressed in terms of sample statistics and are therefore stochastic. I thought that the variance of an estimator of a parameter, if consistent, should be a deterministic parameter itself and should not depend on the sample observations (besides sample size, n), such as the ones we see in using Cramer-Rao lower bound to determine efficiency. Likely I'm understanding something wrong here, any pointers would be greatly appreciated!