Z-score/P-Values vs. Confidence Intervals

[prelic]

Hey all,

2 quick questions:

1. When dealing with the difference between 2 population means (independent samples) or differences of paired data (dependent), a lot of the questions are similar to: "is there sufficient evidence to prove that the difference is 0" or "is there enough evidence to prove that X is sufficiently different from Y", etc. My question is, is there any difference between finding the P-values/z-score and finding and interpreting the confidence interval?

For example,

$$\bar{x}=23.87$$
$$\bar{y}=27.34$$
$$S_1=11.6$$
$$S_2=8.85$$
$$m=79$$
$$n=85$$
$$\alpha=.05$$

Q: Reject or fail to reject:
$$H_0:\mu_1-\mu_2=0$$
$$H_a:\mu_1-\mu_2\neq0$$

Using P-values:
$$z=-2.14, P=2*(1-\Phi(2.14))=.016$$, and $$2*.016 = .032 < .05$$ so REJECT

Using Confidence Intervals:
$$z_{\alpha/2}=1.96$$
CI:$$\bar{x}-\bar{y} \pm 1.96*(1.62)$$
$$=(-6.645,-.2948)$$
Because 0 is not contained in the confidence interval, we reject the hypothesis that $$\mu_1-\mu_2=0$$

Is there any difference between these two calculations and the conclusions I arrived at? When presented these kinds of problems, can I pick either method and arrive at the same answer?

And my other quick question is, when calculating a CI, you will arrive at the same conclusions (different signs, same numbers) when calculating $$\mu_1-\mu_2$$ as $$\mu_2-\mu_1$$, right? How would you interpret a confidence interval such as (30,-5)? To me this interval doesn't make sense.

Thanks!

 

[Stephen Tashi]

Your question highlights the correct definition of a "confidence interval". Such an interval (in standard i.e. "frequentist" statistics") cannot have numerical endpoints, yet many materials written to instruct lab technicians, medical personnel etc. incorrectly claim that it can. If you want to have a "confidence interval" with numerical endpoints (as the one you computed) you can use Bayesian statistics, but i don't think that's what you intended.

The confidence interval of frequentist statistics states an interval about the unknown actual value of the parameter being estimated. It has a known length but not a known location. It does not give a number to unknown true value of the parameter. A common error is assume the true value is equal to the observed value in the particular sample being analyzed.

 

[statdad]

"Your question highlights the correct definition of a "confidence interval". Such an interval (in standard i.e. "frequentist" statistics") cannot have numerical endpoints"

This does nothing at all to help the OP.

Briefly, in the specific case where the hypotheses are

$$\begin{align*} H_0 \colon \mu_1 - \mu_2 & = 0 \\ H_a \colon \mu_1 - \mu_2 & \ne 0 \end{align*}$$

the conclusion reached from the test procedure at significance level $$\alpha$$ will agree with the observation you make with a confidence interval $$(1-\alpha) \times 100\%$$. That is

\begin{itemize}
\item If you reject the null hypothesis (and so have evidence the means are different)
zero will not be in the corresponding confidence interval
\item If you fail to reject the null hypothesis then zero will be in the corresponding confidence interval
\end{itemize}

This correspondence applies only for the alternative $$H_a \colon \mu_1 - \mu_2 = 0$$. Strictly speaking, one-sided alternatives cannot be addressed with confidence intervals.

 

[Stephen Tashi]

"Your question highlights the correct definition of a "confidence interval". Such an interval (in standard i.e. "frequentist" statistics") cannot have numerical endpoints"

This does nothing at all to help the OP.

.

It answers his question about whether there is a distinction between using confidence intervals vs acceptance and rejection regions. The answer is that you can't use confidence intervals because they don't have numerical endpoints. It points out that his creation of a confidence interval with numerical endpoints is an error.

I can't tell whether you are advocating the use of a confidence interval with numerical endpoints and falling into the same error.

 

[statdad]

"The answer is that you can't use confidence intervals because they don't have numerical endpoints."

What makes you think they don't have numerical endpoints? I've heard many defenses of Bayesian statistics in 30 years of studying, teaching, researching, but have never seen this statement. (As well as indicates that you are willing to ignore the years of successful results stemming from use of classical methods.)

 

[Stephen Tashi]

"The answer is that you can't use confidence intervals because they don't have numerical endpoints."

What makes you think they don't have numerical endpoints? I've heard many defenses of Bayesian statistics in 30 years of studying, teaching, researching, but have never seen this statement. (As well as indicates that you are willing to ignore the years of successful results stemming from use of classical methods.)

Are you implying that the assertion that frequentist confidence intervals do not have numerical endpoints is some kind assault upon the frequentist theory that is extreme even for a Bayesian? Or are you saying, that as a Bayesian, I should acknowledge that Bayesian confidence intervals can have numerical endpoints?

I acknowledge that Bayesian confidence intervals can have numerical endpoints. The discussion of the original post is in the context of frequentist statistics. Respectable frequentist texbooks say that fequentist confidence intervals do not have numerical endpoints except in the rare case (not illustrated in the example of the OP) that the true value of the parameter being estimated is already known. Do you disagree with such textbooks?

 

[statdad]

"The discussion of the original post is in the context of frequentist statistics. Respectable frequentist texbooks say that fequentist confidence intervals do not have numerical endpoints except in the rare case (not illustrated in the example of the OP) that the true value of the parameter being estimated is already known"

Since never, through undergraduate school or graduate school have I heard that - give some examples.

I think (I hope) I am misunderstanding this portion of your comment: "... except in the rare case when the true value of the parameter being estimated is already known."
If you truly believe that: why would anyone deign to estimate something that is already known?

(or are you referring to the situation in which the goal is to estimate location and the variability is known?)

 

[Stephen Tashi]

I try to clarify what I mean. The type of "confidence interval" where we can say something like "there is a 90% probability that the interval covers the true value of the parameter we are trying to estimate" is not a numerical interval. It is written as an expression, usually plus or minus a numerical value from a symbol that represents the unknown true value.

If you follow a sampling plan that gives you "90% confidence" and you get a numerical estimate for the parameter like 27.34, you can construct a numerical interval around that value and I suppose you can also call this numerical interval a "confidence interval". But you can't make any claims about the probability that this interval contains the unknown true value that your are trying to estimate. You can't say that there is a 90% probability that the unknown true value is in this particular interval. The textbooks say that the statement about 90% probability can only be made BEFORE the parameter is estimated from a particular sample.

So it doesn't make sense for someone to create a numerical confidence interval and then say they are making decisions based on the probability of true value of the parameter being in or out of that interval.

This is more a question of vocabulary than arithmetic. I can say "I'll find the length of that fence by multiplication" and the proceed to use addition to get the right answer. The arithmetic of creating a numerical confidence interval is just what you you would do to create an acceptance region. I'm saying that OP's question about whether he can use confidence intervals as an alternative to hypothesis testing is best answered by "no" because he seems to think that he can create a numerical interval and call it a "confidence interval" in the sense that there will be some known probabilities associated with it.

I do agree that practicing statisticians may create numerical intervals and call them "confidence intervals", but hopefully they don't make any claims about the probability that these intervals contain the true value of the parameter being estimated.

 

[statdad]

You didn't understand my question - communication error on my part. I am a statistician.

I try to clarify what I mean. The type of "confidence interval" where we can say something like "there is a 90% probability that the interval covers the true value of the parameter we are trying to estimate" is not a numerical interval. It is written as an expression, usually plus or minus a numerical value from a symbol that represents the unknown true value.

I know this - and it is a numerical interval. I have no idea why you consider that not so.

If you follow a sampling plan that gives you "90% confidence" and you get a numerical estimate for the parameter like 27.34, you can construct a numerical interval around that value and I suppose you can also call this numerical interval a "confidence interval". But you can't make any claims about the probability that this interval contains the unknown true value that your are trying to estimate. You can't say that there is a 90% probability that the unknown true value is in this particular interval. The textbooks say that the statement about 90% probability can only be made BEFORE the parameter is estimated from a particular sample.

True: we teach our students that the confidence level can be interpreted as a long-term expression of correctness.

So it doesn't make sense for someone to create a numerical confidence interval and then say they are making decisions based on the probability of true value of the parameter being in or out of that interval.

No - confidence intervals provide a way of expressing a possible range of value for a parameter, reflecting a level of sampling uncertainty.
To deny that they are useful is to be at odds with much of the work done with statistics.

This is more a question of vocabulary than arithmetic. I can say "I'll find the length of that fence by multiplication" and the proceed to use addition to get the right answer. The arithmetic of creating a numerical confidence interval is just what you you would do to create an acceptance region. I'm saying that OP's question about whether he can use confidence intervals as an alternative to hypothesis testing is best answered by "no" because he seems to think that he can create a numerical interval and call it a "confidence interval" in the sense that there will be some known probabilities associated with it.

As I said, we don't teach that there is a fixed probability associated with any particular interval.

I do agree that practicing statisticians may create numerical intervals and call them "confidence intervals", but hopefully they don't make any claims about the probability that these intervals contain the true value of the parameter being estimated.

In the years I've been studying, teaching, and researching, I've never met any that have nor seen it presented as such in a textbook.

 

[Stephen Tashi]

Then we agree on everything! - except the definition of a "numerical interval". I don't call an interval represented by an expression such as " x_bar plus or minus 18.6" a numerical interval because I don't know numerical values for its endpoints. For example, I can't tell whether this interval includes 98.34 or is to the left or right of it.

 

[statdad]

Then we agree on everything! - except the definition of a "numerical interval". I don't call an interval represented by an expression such as " x_bar plus or minus 18.6" a numerical interval because I don't know numerical values for its endpoints. For example, I can't tell whether this interval includes 98.34 or is to the left or right of it.

Certainly one of the oddest objections. It is personal and has nothing to do with statistical theory or practice.

 

[Stephen Tashi]

Certainly one of the oddest objections. It is personal and has nothing to do with statistical theory or practice.

Are you saying that it is standard statistical theory or practice to call an interval like
$$[ \theta - 18.6, \theta + 18.6]$$ an "interval with numerical endpoints" when we don't know $$\theta$$ ?

 

[statdad]

No - and that is not a formula for a classical confidence interval. I'm not sure where your confusion comes from.

 

[Stephen Tashi]

My confusion comes from hearing confusing statements.

Me: I try to clarify what I mean. The type of "confidence interval" where we can say something like "there is a 90% probability that the interval covers the true value of the parameter we are trying to estimate" is not a numerical interval. It is written as an expression, usually plus or minus a numerical value from a symbol that represents the unknown true value.

You: I know this - and it is a numerical interval. I have no idea why you consider that not so.

Me: Are you saying that it is standard statistical theory or practice to call an interval like $$[ \\theta - 18.6, \\theta + 18.6]$$ an "interval with numerical endpoints" when we don't know $$\theta$$?

You: No - and that is not a formula for a classical confidence interval. I'm not sure where your confusion comes from.

Is this some fine point of language? Are you making a distinction between "numerical interval" and "interval with numerical endpionts"? I myself wasn't.
What do you regard the formula for a classical confidence interval to be?

 

[statdad]

"The type of "confidence interval" where we can say something like "there is a 90% probability that the interval covers the true value of the parameter we are trying to estimate"

If that is taken as meaning a particular interval has a 90% chance of containing the true parameter then that interpretation (or emphasis, if it comes from an instructor) is incorrect.

The 90% confidence level can be interpreted informally as meaning that the particular method being used has a long-term probability of 0.90 of yielding correct results. Stated another way: if we were to take repeated samples, all of identical size, from the same population, and each time construct a 90% confidence interval for the unknown parameter, 90% of those intervals would surround that parameter.

I am saying that $$\theta \pm 18.6$$ is not a formula for a confidence interval. It certainly does not have numerical endpoints specifically because $$\theta$$ is not known - in that you are correct. You are incorrect in believing a formula like that is ever used as a confidence interval.

 

[Stephen Tashi]

I am saying that $$\theta \pm 18.6$$ is not a formula for a confidence interval.

Is this an objection to using the notation $$\theta \pm 18.6$$ to stand for the set of numbers $$\{ x: \theta - 18.6 \leq x \leq \theta + 18.6 \}$$ ?

Or are you objecting to my use of a specific number 18.6? I agree that a general formula for a confidence interval will have some expression with variables in it like $$\pm 2.1 \sigma$$ in it. My point in using the example 18.6 is that a solution to a specific problem can have a numerical answer for the expression $$2.1 \sigma$$. But except in the unusual case where the true value of $$\theta$$ is given, the answer to a specific problem won't assign a numerical value for $$\theta$$.

 

[Stephen Tashi]

...Or are you objecting to my referring to the center of the interval as the true value of the parameter?

You have a point there! I should be calling $$\theta$$ the unknown value of the estimate of the parameter which is not yet computed from the sample.

 

[statdad]

"Or are you objecting to my use of a specific number 18.6? I agree that a general formula for a confidence interval will have some expression with variables in it like in it. My point in using the example 18.6 is that a solution to a specific problem can have a numerical answer for the expression . But except in the unusual case where the true value of is given, the answer to a specific problem won't assign a numerical value for $$\theta$$"

My point was your notation indicates the goal is to estimate the parameter theta, and so it is inappropriate to base the endpoints on it.

And again, a confidence interval does not assign 'a number' to a parameter. It provides a range of estimates which, taken as a whole, is an estimate of the size of theta. There is no statistical problem with that - or there shouldn't be.

 

[Stephen Tashi]

My point was your notation indicates the goal is to estimate the parameter theta, and so it is inappropriate to base the endpoints on it.

I agree with your objection. I trust we do agree that a "90% confidence interval" is based upon some kind of thing represented by a symbol, which does not get assigned a numerical value during the time we are claiming that there is a 90% chance the true value of the parameter being estimated is inside the interval.

And again, a confidence interval does not assign 'a number' to a parameter. It provides a range of estimates which, taken as a whole, is an estimate of the size of theta. There is no statistical problem with that - or there shouldn't be.

I agree that it doesn't assign a number to a parameter and that it defines a range of estimates about some unknown value. But once you put specific numbers in for this unknown value, you have a different kind of interval. You can call it a "confidence interval" and you can even say you have "90% confidence" in it . But you can't say that there is a 90% chance that the unknown true value of the parameter being estimated falls in that particular interval. In the OP, there is a "confidence interval" with specific numerical endpoints and he asks if it is correct to "reject the hypothesis" at the given level of significance because the estimated value from the sample is not in the confidence interval. This is not correct terminology because you can't say anything about the probability of the true value being in that specific interval and you certainly can't say anything about the probability that the estimated value will be in that interval. What the OP is doing is the arithmetic for an acceptance region and he is referring to the region as a "confidence interval".

 

[statdad]

"But you can't say that there is a 90% chance that the unknown true value of the parameter being estimated falls in that particular interval."
I've said that, several times, and explained what the confidence level means. "In the OP, there is a "confidence interval" with specific numerical endpoints and he asks if it is correct to "reject the hypothesis" at the given level of significance because the estimated value from the sample is not in the confidence interval. This is not correct terminology because you can't say anything about the probability of the true value being in that specific interval and you certainly can't say anything about the probability that the estimated value will be in that interval. What the OP is doing is the arithmetic for an acceptance region and he is referring to the region as a "confidence interval."

Wrong. In a classical hypothesis for means or differences of means, with two-sided alternative, rejecting the null hypothesis at level alpha corresponds to the specified mean (or mean difference) not being in the corresponding confidence interval. If two people analyze the same data, one with 2-sided alternative, one with a confidence interval, they will reach the same conclusion.
* rejection of H0 will correspond to the null value not falling in the confidence interval, and vice versa
* failing to reject H0 will correspond to the null value falling in the confidence interval

Saying otherwise is simply incorrect.

 

[Stephen Tashi]

Does anything in what you said assert that there is a 0.95 probability that the true difference in the population means is in the interval (-6.645, -.2948) ?

 

[statdad]

No - as mentioned, the confidence level does not indicate the chance of any particular interval containing the target. That has nothing at all to do with the correctness of my comments about testing and estimation.

What is it you don't understand?

 

[Stephen Tashi]

No - as mentioned, the confidence level does not indicate the chance of any particular interval containing the target. That has nothing at all to do with the correctness of my comments about testing and estimation.

Thank goodness! Then the correctness of your comments about testing and estimation have nothing to do with what I'm saying.

What is it you don't understand?

I'm not in doubt about anything we discussed.

 

[statdad]

"I'm not in doubt about anything we discussed."

? I'm not sure if that's good or bad, since most of your comments about confidence intervals, their interpretations, use, and comments about mainstream statisticians were wrong.

 

[Stephen Tashi]

? I'm not sure if that's good or bad, since most of your comments about confidence intervals, their interpretations, use, and comments about mainstream statisticians were wrong.

Yep, me and Mood, Graybill and Boes. We must be wrong We admit there is a connection between confidence intervals and hypothesis testing, but when we use an interval in a hypothesis test we call it an "acceptance" region. We say confidence intervals may be used to define acceptance regions, but we don't call an acceptance region in a hypothesis test a confidence interval. I guess we'll have to reform and start calling those things confidence intervals.

 

[Stephen Tashi]

Returning to the original post (if prelic is still reading this thread)

My question is, is there any difference between finding the P-values/z-score and finding and interpreting the confidence interval?

...

Is there any difference between these two calculations and the conclusions I arrived at? When presented these kinds of problems, can I pick either method and arrive at the same answer?

You'll get the same answer in the type of problem you showed, although I don't call what you did "using a confidence interval" and statdad does. The basic reason is geometry. Imagine two points A and B on the x-axis and suppose each on is at the center of an interval of length 2.

Question: Can A be in B's interval without B being in A's interval?

No, they are both in or both out of each other's intervals.

To translate this to the problem:
A = 0, the hypothesized difference in the means
A's interval is the interval whose center is A and above which is an area of 1-P under the normal probability density.
When you do the P-value method, what you look to see whether the observed difference of sample means falls in this interval.

B = the difference in the sample means, call it d_bar.
B's interval = what you called the "confidence interval"
Your procedure in using this interval was to see if A (zero) fell in this interval.

It may not be easy to see that B's interval is the same length as A's interval. To see this you have to represent everything on the x-axis to the same scale. Either leave everything in the original units of the problem (don't convert to Z's) or convert everything, including d_bar and the length of the confidence interval to units of Z's.

The confidence interval was created so that above it is an area of 1-P under the normal probability density. It has the same length as the acceptance interval when they are both expressed in the same units.

Even though you get the same answer with P-values as with your "CI", I would advise you to do things the P-value way on tests unless otherwise instructed. Statdad would presumably give you full credit for the CI method, but other instructors might not.

And my other quick question is, when calculating a CI, you will arrive at the same conclusions (different signs, same numbers) when calculating $$\mu_1-\mu_2$$ as $$\mu_2-\mu_1$$, right?

Different signs for which numbers? You won't get a different length for the confidence interval or the acceptance interval. The observed value of the difference in sample means will change sign if you define the difference to mean $$\mu_2 - \mu_2$$ vs $$\mu_1 - \mu_2$$.

How would you interpret a confidence interval such as (30,-5)? To me this interval doesn't make sense.

It doesn't make sense to me either, but how are you thinking such an interval could arise? I myself prefer not to call numerical intervals "confidence intervals". My old statistics text says numerical intervals can be called confidence intervals "by abuse of language".