Power and binomial distribution

[imsmooth]

Maybe someone is really good with stats, or has access to a statistics professor. Here we go:


I am trying to determine the power for a study. The distribution is binomial. I have a device that either works or does not work. I do not know the real probability, but I think it is very good. Let's assume p = 0.75. I am going to try it 50 times. Using a binomial distrtibution program I determine that if I get 43 or more successes, this will happen by chance less than 5% (p = 0.05) of the time. So, I will accept that the device works at least 75% of the time if I get 43 successes or more. The question then is what is the power of this study? This seems to be predicated on the fact that I know the real probablility. If I think the device works 89% of the time I find that my power is 82%. If the true success rate is 90% the power goes up to 87%.


How can I get a meaningful number for the power? If my effect size is large, true I have a larger power. But, I don't know the real effect size. I can make the power of the test arbitrarily large by just stating I have large effect size. This does not seem right.

 

[Stephen Tashi]

This seems to be predicated on the fact that I know the real probablility.

It is.

How can I get a meaningful number for the power?

This depends on what you call "meaningful". One question that arises is whether the impressive sounding jargon "power of the test" is really meaningful in your application. Are you simply writing some kind of report and needing to put in a number for "power" that sounds impressive? Or are you trying to solve a problem from a non-marketing point of view?

I can make the power of the test arbitrarily large by just stating I have large effect size.

Yes

This does not seem right.

I agree, but statistics operates under a handicap. When mathematics is applied to surveying, fluid mechanics etc. nobody expects a problem with insufficient information to be solved. However, statistics gets many of its problems from the worlds of economics, social science and complicated settings where people expect to get answers without supplying much information. If someone says to a surveyor "Here is a triangle ABC with AB = 4 and angle ABC = 30 degrees, find the other sides of a triangle", he can patiently explain that there is no mathematical solution to the problem while maintaining his professionalism. In statistical problems, you have people demanding answers because they are going to make important decisions and impugning the information they give you does not please such clients.

 

[Stephen Tashi]

Some other thoughts:

The way I interpret your remarks, your "null hypothesis" is that the device works with a probability of 0.75 and (I think) you wouldn't worry if the device had a higher probability of that. So, the idea is to test if the device works with a probability of 0.75 or greater . You are interested in detecting if it has a lower probability. So the practical concern about power ( i.e. "rejecting the null hypothesis when it is false" ) is the probability that you can detect when the device has a low probability of working. So I don't see why the numerical examples of .89 and .90 are relevant to the power of the test. They might be relevant if you really care that the reliability is exactly 0.75.