Expectations for Type I Error

[ganstaman]

It's been too long since I've done probability or statisitcs, so I'm looking for a bit of help on this subject. I hope that my title is actually what I'm about to talk about :tongue: . Basically, we always seem to assume that everything evens out in the long run so that no one really experiences 'luck.' But I think that we should in fact expect some people to be 'lucky' or 'unlucky' in the long run. I'm just having some trouble with the math:

Let's say we have this guy named Jason (that's me!) who flips a coin N times (I hope we're able to keep N variable, but if you must choose a value for it, just make it something really really big). We expect that Jason gets some proportion of heads to tails that only insignificantly deviates from 1:1. However, even with a fair coin, we expect with probability P that Jason will get some deviation from the expected values that is of statistical significance.

I'm pretty sure I can prove that last sentence, which is important to be true. With probability 1/(2^N), Jason will flip all heads. With N big enough (I'm recalling N>10, but this could be wrong), this would be a statistically significant deviation from the expected probabilities and we'd assume an unfair coin even though it's fair. When N is much larger, we don't need to flip all heads for this to work out, so finding the value of P is slightly more work. ----How do I find P here?----

Also, let's say that we attribute heads to winning, and tails to losing. This means that with probability P, Jason will appear to be very lucky or very unlucky. (Note that if either P or the size of our coin flipping population were big enough, we'd even expect 1 or more Jasons to exist!). Now remove the coins and give everyone a deck of cards and some poker chips. We now expect with probability Q that any given individual will appear to be lucky or unlucky IN THE LONG RUN (because N is big enough). Again, if Q and/or our poker playing population is big enough, we actually can expect these lucky/unlucky people to exist.

This is a much more open ended question, but how would one go about determinging Q here?

 

[HallsofIvy]

You would first have to state what you mean by "lucky" or "unlucky". In other words give a value for the number of heads to get in N flips in order to be "lucky". Then you just need to calculate P(n> N/2) and that's a binomial distribution.

 

[EnumaElish]

Is Jason's luck random (has equal chances of getting lucky or unlucky), or systematic (he knows he will get lucky)?

A related topic is order statistics. Out of N people simultaneously making draws from a normal distribution, there is an expected value for the largest, 2nd largest, ..., draw.

See also winner's curse.

 

[ganstaman]

Is Jason's luck random (has equal chances of getting lucky or unlucky), or systematic (he knows he will get lucky)?

His luck is random. The idea is that if you perform the relevant statistical analysis to his series of N coin flips, you'd conclude with good confidence that the coin was not fair and his luck was systematic. However, I can assure you that the coin was perfectly fair.

You would first have to state what you mean by "lucky" or "unlucky". In other words give a value for the number of heads to get in N flips in order to be "lucky". Then you just need to calculate P(n> N/2) and that's a binomial distribution.

Ok, so I did a bit of searching and found this: http://www.stat.tamu.edu/~west/applets/binomialdemo.html

We will play with n, set p=0.5, and find the probability that X is at least some number we will play with (letters here are their notation, not necessarily consistent with what I've said so far). As an example, with N=100, we expect Jason to flip more than 65 heads 0.18% of the time just by random chance. If he does flip 66 heads, would we not therefore conclude that it was not random chance, but instead an unfair coin?

If N=10,000, we expect to get 5200 heads or more only 0.003% of the time. But 0.18% and 0.003% are extremes and we'd claim the possibility of an unfair coin at what, 5%?

I guess I'm wondering if this makes sense: if we get 100,000 people to flip coins 10,000 times each, we'd expect 3 people to get 5200 or more heads, and 3 people to get 5200 or more tails. However, if we run into these 6 people individually, we'd call them very lucky. And if I could only figure out how to apply this to poker (not an easy 50% chance of winning/losing), I'd have more of my answer.

 

[EnumaElish]

I guess I'm wondering if this makes sense: if we get 100,000 people to flip coins 10,000 times each, we'd expect 3 people to get 5200 or more heads, and 3 people to get 5200 or more tails. However, if we run into these 6 people individually, we'd call them very lucky.

In fact, you wouldn't recognize them even if you were to walk right into these people. They would look like random people.