How can I compute the variance of the sample mean estimator?

[mnb96]

Hello,
I have a discrete random variable z whose expected value $\mu$ is unknown. Its distribution is also unknown.
We extract N samples $z_1,\ldots,z_N$, where each sample is an integer number: $z_i\in \mathbb{Z}$.

Now, I introduce an estimator for the expected value defined as follows:

$$\overline{z}=\frac{1}{N}\sum_{i=1}^N z_i$$

How should I compute the expected value $E[\overline{z}]$ of the estimator $\overline{z}$?

***
In my lecture notes I read:
$$E[\overline{z}]=E[\frac{1}{N}\sum_{i=1}^N z_i]=\frac{1}{N}\sum_{i=1}^NE[z_i]=\mu$$

This makes no sense to me, especially the term $E[z_i]$.
What is supposed to represent the expected value of an observation?!

 

[tiny-tim]

hello mnb96!

E[z_i] = µ, because z_i is the same random variable as z (it just has a different name)

 

[mnb96]

thanks! ;)
I see...so whenever I am given a sample of a random variable which is not explicitly specified, I have to replace it with the random variable itself, because it could be potentially any other sample?

 

[tiny-tim]

hi mnb9!

i'm not sure i understand your logic, but yes, a random variable and a sample of the same random variable are the same thing, you can replace one by the other

 

[mnb96]

Hi tiny-tim,
indeed I am struggling for trying to make sense of something that doesn't yet fully make sense to me: random variables :)

I don't seem to have solved my problems with this issue, in fact, I tried to compute an estimate for the unknown variance of the estimator $\overline{z}$ with no success. I'll illustrate my problem:

$$Var(\overline{z}) = E(\overline{z}^2)-E(\overline{z})^2 = E([\frac{1}{n}\sum_i z_i]^2) - \mu^2$$

Now according to me, the random variables $z_i$ are all the same because they are samples coming from the same distribution of z, so $z=z_1=\ldots=z_n$ and I could simply conclude by (wrongly) asserting that:

$$E([\frac{1}{n}\sum_i z_i]^2) = E(\frac{1}{n^2}[\sum_i z]^2)=E(z^2)=Var(z)$$

According to my lecture notes this is not correct; they basically do this:

$$E([\frac{1}{n}\sum_i z_i]^2) = E([\frac{1}{n}\sum_i z][\frac{1}{n}\sum_j z_j])=E(\frac{1}{n^2}\sum_i \sum_j z_i z_j)=\ldots$$

After this, they follow their computation that ends up with $\frac{n-1}{n}Var(z)$.

Studying books didn't help me understand what do I have to think of whenever I write the name of a random variable. Is it a function? of course not...Is it an element of an event space? no way...is it a scalar? no...so how am I supposed to know how to perform algebraic operations between some sort of "chameleon-like entities"?

Frustration levels are rising...

 

[tiny-tim]

hi mnb96!

i don't understand this part (i've tidied it up ) …

$$E\left(\frac{1}{n^2}\left[\sum_i z_i\right]^2\right)=E(z^2)$$

E(∑ z_i) = E(z),

but that doesn't mean that E(∑ z_i²) = E(z²)

 

[mnb96]

maybe it could help a lot if you make me understand how the sum of two random variables is defined.

PS: how did you write E(∑ zi) = E(z) without using latex?
PS2: is there a keyboard short cut for latex input, analogous to ctrl+i, ctrl+b for italic/bold ?

 

[tiny-tim]

hi mnb96!

maybe it could help a lot if you make me understand how the sum of two random variables is defined.

sorry, i don't understand the question …

the sum of two random variables is one random variable plus the other random variable

PS: how did you write E(∑ zi) = E(z) without using latex?
PS2: is there a keyboard short cut for latex input, analogous to ctrl+i, ctrl+b for italic/bold ?

use the X₂ icon just above the Reply box for subscripts, and copy-and-paste the characters below

(and no, no keyboard short-cut, though the ∑ icon above the Reply box can be helpful)

 

[mnb96]

thanks for the latex hints!

Regarding my question: I just read that if X and Y are two discrete independent random variables having the same distribution, the pdf of X+Y is given by the convolution of X and Y.

If nothing else, this at least shows that Z₁+...+Z_n $\neq$ nZ, so the expression that you didn't understand is clearly wrong, and I notice that the operator + for random variables has very little to do with the + for scalars.

 

[tiny-tim]

Regarding my question: I just read that if X and Y are two discrete independent random variables having the same distribution, the pdf of X+Y is given by the convolution of X and Y.

ah, that's because the pdf is d/dx P(X≤x), or loosely speaking P(X=x) …

so, loosely speaking, the pdf for X+Y is P(X+Y=x) = ∫ P(Y=y)P(X=x-y) dy …

just set-theory, really! ​

 

[mnb96]

ok thanks!
I will do more exercises on these topics.
Hopefully practice will help understand better.

 

[bpet]

Studying books didn't help me understand what do I have to think of whenever I write the name of a random variable. Is it a function? of course not...Is it an element of an event space? no way...is it a scalar? no...so how am I supposed to know how to perform algebraic operations between some sort of "chameleon-like entities"?

Frustration levels are rising...

The modern definition of random variables was several hundred years in the making, so don't worry if the underlying concepts take a while to sink in :)

Actually a r.v. can be defined as a function (that maps an event w to a real number X(w)) which is useful for doing more complicated stuff in a precise & natural way. Something to look forward to if you decide to continue your studies in probability theory.

For now, to understand the properties of the basic statistical estimators of mean and variance, it's enough to become familiar with the rules for manipulating expectation formulas.

Also be aware of the difference between "variance of the sample mean" (which equals Var(Z)/n) and "mean of the variance estimator" (which equals ((n-1)/n)*Var(Z)).

 

[mnb96]

Hi bpet,
thanks for your support! :)
I am now forcing myself to think of rv as functions as you said, in order to prevent trivial mistakes.

I followed your advice and I tried to compute both the variance of the mean estimator and the mean of the variance estimator. Finally, I got the correct result.

It was interesting to note that the "variance of the sample mean" converges to zero when infinite samples are provided (I guess it follows from the central limit theorem?), and to notice that the variance estimator I defined is biased, but asymptotically unbiased.