Is Probability Truly Meaningful in Research Studies?

[the number 42]

A couple of points, I hope not too muddled:

I find probability difficult to understand, but I sometimes get concerned when a small p value is taken as proof-positive. I know that the .05 level used in psychology is arbitrary (though it probably makes sense) and Cohen points out that research in psychology has traditionally lacked statistical power, making nonsense of the .05 level when used in such studies.

When a researcher says that 'compared to chance' their results are significant, I'm starting to wonder what they mean. You hear things like '50 people score above chance on a card-guess study which is evidence for psi'; what exactly are we comparing the guesses to? I know if there are 5 cards in the deck, they have a 1 in 5 chance... or do they? I've heard that a random number generator may throw up non-random sequences, so how predictable is chance?

 

[matt grime]

This looks familiar. Repost?

I would imagine that the research Cohen talks about might be the kind whereby things are modeled mathematically without any justification that the model is accurate, or that the samples are too small.

Anyway, the idea is this: we have a large population and a certain proportion of them share some trait (for example). We sample a small segement of them in some suitably `random' fashion (how one takes the sample is important), and then analyse it statistically, to estimate the proportion of the whole population with that trait. When we construct confidence intervals we are NOT saying that there is a 95% chance that the actual population proportion is in the range (for the 5% margin), but that if we repeat the sampling again and again, then in the long run 95% of the time we expect the statistic to lie in that region.


The last thing about random and non-random is a very dodgy area - because you're opening it up to all kinds of interpretations as to what you mean by random.

Firstly there is no such thing as a random number generator, really.

Secondly, a truly random set of digits, in the statistical sense, will contain things that human perception would deem as not random.

Example, you toss a coin 5 times and obtain HHHHH (ie 5 heads), is that more or less likely than obtaining HTHHT? Obviously the chances ought to be the same (in the model). but we think of one as being more likely, don't we?

The compared to chance thing is called hypothesis testing. It is a large area, you should google for it.

 

[the number 42]

This looks familiar. Repost?

Yeah. I thought if I posted it as a new thread someone might answer - luckily you did.

I would imagine that the research Cohen talks about might be the kind whereby things are modeled mathematically without any justification that the model is accurate, or that the samples are too small.

I'm picking up a classic paper of his sometime soon, so I'll post anything of interest.

The last thing about random and non-random is a very dodgy area - because you're opening it up to all kinds of interpretations as to what you mean by random.

Firstly there is no such thing as a random number generator, really.

Secondly, a truly random set of digits, in the statistical sense, will contain things that human perception would deem as not random.

This is the bit that I find a bit mind-boggling. I think I might just be worried that within any random sequence of numbers, there might - by chance - be non-random sequences. Or perhaps like repeating patterns from chaos theory. So in these cases (or all cases?) scoring 'above chance' is meaningless, as 1/ chance is never the same in any two instances, except maybe when 2/ non-random patterns emerge, by chance. Would an immortal monkey typist eventually write the works of Shakespeare?

I might feel better if I knew how tables of critical values are compiled or calculated. On the other hand, I feel a bit over my head with this whole area, but somehow I want to understand it a little better.

 

[matt grime]

Ok, you;re using the words "random" and "chance" in some very odd ways. I think you ought to figure out whaty you mean by them.

Let's suppose that yuo have a string of digits and each digit is one of 0 or 1, and each digit is equally likely to be either 0 or 1. If there are 10 digits in the string what is the probability that the sub string 10 occurs? Suppose that we have a string of a bollion digits, what is that probability that 10 *doesn't* occur?

The works of shakespeare are just a finite string of 'digits'. If we took larger and larger strings in the same symbols, then with probability 1 that string will occur eventually - just as it's unlikely for 10 not to appear in a strnig of one billion digits it's unlikely that "the works of shakespeare" won't appear in a string of a googol of digits.


Critical values use soudn mathematical technique about the distributions of samples. If you're bothered look up the details (t-distributions, chi squared and so on).