How To Find 68% Confidence Interval After Running MCMC

[xdrgnh]

This is not a homework problem so I'm asking it here. I just ran an MCMC and found the best fit parameters for a model of massive gravity. I now need to find the said 68% Confidence Interval interval for those parameters. I have never done anything like this before so I'm clueless where to begin or even look for reference. I have the parameter errors from the MCMC and the correlation matrix if any of that helps. Any help will be appreciated thank you.

 

[WWGD]

I assume you have a sample of size N with sample mean $\mu ^{-}$, where the pop. deviation $\sigma$ is known. An interval centered at the sample mean with width $\sigma$ , i.e., $( \mu^{-} - \sigma, \mu^{-} + \sigma)$ will be a 68 % confidence interval, using the 68-95-99.7 rule. Please specify if you are using a different setup. EDIT: This last works for the mean, let me check for similar results for different parameters, i.e., different versions of the CLT for different parameters.

 

[Dale]

From the MCMC you should have a sample of the posterior distribution. You can directly construct your 68% highest posterior density intervals from that.

 

[xdrgnh]

From the MCMC you should have a sample of the posterior distribution. You can directly construct your 68% highest posterior density intervals from that.

By posterior distribution do you mean the list of values of the parameters that were generated as it went though its 39,000 steps?

 

[xdrgnh]

I assume you have a sample of size N with sample mean $\mu ^{-}$, where the pop. deviation $\sigma$ is known. An interval centered at the sample mean with width $\sigma$ , i.e., $( \mu^{-} - \sigma, \mu^{-} + \sigma)$ will be a 68 % confidence interval, using the 68-95-99.7 rule. Please specify if you are using a different setup. EDIT: This last works for the mean, let me check for similar results for different parameters, i.e., different versions of the CLT for different parameters.

This about the median not the mean actually. Does that change anything?

 

[WWGD]

Sorry, I jumped in without asking questions. What is an MCMC?

 

[Dale]

By posterior distribution do you mean the list of values of the parameters that were generated as it went though its 39,000 steps?

Yes, that is the output of a MCMC.

Are you familiar with Bayesian statistics? Each step of the MCMC method generates a sample of the posterior distribution of the parameters.

Sorry, I jumped in without asking questions. What is an MCMC?

Markov Chain Monte Carlo. It is the standard method for doing Bayesian statistics. It is a very interesting and powerful technique.

 

[WWGD]

Yes, that is the output of a MCMC.

Are you familiar with Bayesian statistics? Each step of the MCMC method generates a sample of the posterior distribution of the parameters.

Markov Chain Monte Carlo. It is the standard method for doing Bayesian statistics. It is a very interesting and powerful technique.

Ah, yes, I just happen to call it E, so that E=MCMC= (MC)^2 ... So Close ; ).

 

[Dale]

Haha!

 

[xdrgnh]

Yes, that is the output of a MCMC.

Are you familiar with Bayesian statistics? Each step of the MCMC method generates a sample of the posterior distribution of the parameters.

Markov Chain Monte Carlo. It is the standard method for doing Bayesian statistics. It is a very interesting and powerful technique.

Not as much as I should lol. This is my first time doing anything meaningful with statistics. So I have my posterior distribution now. From there do I just find the standard deviation for my parameters and use the formulas I learned in intro to Stat to find the confidence intervals around the median?

 

[Dale]

So I have my posterior distribution now. From there do I just find the standard deviation for my parameters and use the formulas I learned in intro to Stat to find the confidence intervals around the median?

No, since you have a good sample of the posterior distribution you don't need to do any formulas. If your software package can generate the empirical CDF, then just use that to take any credible interval you like. Many Bayesian packages have a function for obtaining the highest posterior density interval at any given level. If your package doesn't have any of those then just sort your posterior samples and pick the appropriate range.

 

[xdrgnh]

No, since you have a good sample of the posterior distribution you don't need to do any formulas. If your software package can generate the empirical CDF, then just use that to take any credible interval you like. Many Bayesian packages have a function for obtaining the highest posterior density interval at any given level. If your package doesn't have any of those then just sort your posterior samples and pick the appropriate range.

I'm using Mathematica. So I form the CDF but I'm not clear on what to do from there. How do I decide the appropriate range.

 

[Dale]

I'm using Mathematica. So I form the CDF but I'm not clear on what to do from there. How do I decide the appropriate range.

Well, if you want a 95% confidence interval then you would take the 0.025 quantile and the 0.975 quantile from the CDF

 

[Stephen Tashi]

I now need to find the said 68% Confidence Interval interval for those parameters.

It isn't clear what you mean by a confidence interval for the parameters of the model.

Is the general scenario that you had a prior distributions for the parameters and obtained a posterior distribution for them?

Or did you set some criteria for "best" fit and solve the problem as an optimization problem without specifying a prior distribution for the parameters?

"Confidence interval" is a concept from frequentist statistics, but MCMC is usually used in Bayesian statistics.

 

[xdrgnh]

It isn't clear what you mean by a confidence interval for the parameters of the model.

Is the general scenario that you had a prior distributions for the parameters and obtained a posterior distribution for them?

Or did you set some criteria for "best" fit and solve the problem as an optimization problem without specifying a prior distribution for the parameters?

"Confidence interval" is a concept from frequentist statistics, but MCMC is usually used in Bayesian statistics.

I'm trying to reproduce the 68% confidence level in this paper. https://arxiv.org/pdf/1205.1613v1.pdf. I never studied this type of statistics before which is why I have a lot of questions and really don't understand the terminology.

 

[xdrgnh]

Well, if you want a 95% confidence interval then you would take the 0.025 quantile and the 0.975 quantile from the CDF

Alright I think I found the 95% confidence interval for my parameters from the posterior distribution. To check though the boundary of my intervals should be elements of my posterior distribution right? Also what is the quantile for the 68% interval and where are those quantile derived from. Thank you so much for all of your help so far.

 

[Dale]

Also what is the quantile for the 68% interval and where are those quantile derived from.

It would be (1-0.68)/2 and 1-(1-0.68)/2

 

[WWGD]

Does EDIT the process of obtaining confidence intervals in MCMC use some version of the CLT, i.e., do we use i.i.d random variables for each approximation? Otherwise, how do we apply confidence intervals to generic distributions? Hope this does not sound too confused. EDIT: for example, how would we compute a confidence interval for the median, variance, etc if not by using a version of CLT? I know, e.g., the sampling median obtained from a normal population has a know distribution, but, AFAIK, it does not have a nice form otherwise.

 

[Dale]

Bayesian statistics is fundamentally different in that respect. The CLT doesn't really get used in Bayesian statistics.

In frequentist statistics you always work with probabilities that are defined as proportions over an imagined infinite number of trials given a fixed true hypothesis.

In Bayesian statistics it is the hypothesis that is uncertain given a fixed set of data. You don't assume anything about infinite numbers of repetitions of the experiment, you just refine your hypothesis as much as possible given the data that you actually have.

 

[Stephen Tashi]

In Bayesian statistics it is the hypothesis that is uncertain given a fixed set of data. You don't assume anything about infinite numbers of repetitions of the experiment, you just refine your hypothesis as much as possible given the data that you actually have.

True, but looking at the paper that the OP linked, I can't tell whether Bayesian statistics is being used. (I'm not a cosmologist.) The term "priors" for the cosmic microwave background radiation (CMBR) is used, but this may not refer to a probability distribution in the Bayesian sense of a prior.

Can anyone comment on what "priors" means in the context of CMBR?

 

[Dale]

True, but looking at the paper that the OP linked, I can't tell whether Bayesian statistics is being used. (I'm not a cosmologist.)

I am also not a cosmologist, but they do say "Bayesian framework", and they use MCMC, and talk about both the likelihood and the prior. So if it isn't Bayesian, it is a very well camouflaged frequentist.

 

[Valerio M]

Well, if you want a 95% confidence interval then you would take the 0.025 quantile and the 0.975 quantile from the CDF

does that give you in general the Highest Density Interval?

 

[Dale]

does that give you in general the Highest Density Interval?

Not in general, no. In particular, if a posterior is skewed, then the highest density interval will take more of the “short” side.