Dismiss Notice
Join Physics Forums Today!
The friendliest, high quality science and math community on the planet! Everyone who loves science is here!

I How To Find 68% Confidence Interval After Running MCMC

  1. May 29, 2017 #1
    This is not a homework problem so I'm asking it here. I just ran an MCMC and found the best fit parameters for a model of massive gravity. I now need to find the said 68% Confidence Interval interval for those parameters. I have never done anything like this before so I'm clueless where to begin or even look for reference. I have the parameter errors from the MCMC and the correlation matrix if any of that helps. Any help will be appreciated thank you.
     
  2. jcsd
  3. May 29, 2017 #2

    WWGD

    User Avatar
    Science Advisor
    Gold Member

    I assume you have a sample of size N with sample mean ## \mu ^{-} ##, where the pop. deviation ## \sigma ## is known. An interval centered at the sample mean with width ## \sigma ## , i.e., ## ( \mu^{-} - \sigma, \mu^{-} + \sigma) ## will be a 68 % confidence interval, using the 68-95-99.7 rule. Please specify if you are using a different setup. EDIT: This last works for the mean, let me check for similar results for different parameters, i.e., different versions of the CLT for different parameters.
     
    Last edited: May 29, 2017
  4. May 29, 2017 #3

    Dale

    Staff: Mentor

    From the MCMC you should have a sample of the posterior distribution. You can directly construct your 68% highest posterior density intervals from that.
     
  5. May 29, 2017 #4
    By posterior distribution do you mean the list of values of the parameters that were generated as it went though its 39,000 steps?
     
  6. May 29, 2017 #5
    This about the median not the mean actually. Does that change anything?
     
  7. May 29, 2017 #6

    WWGD

    User Avatar
    Science Advisor
    Gold Member

    Sorry, I jumped in without asking questions. What is an MCMC?
     
  8. May 29, 2017 #7

    Dale

    Staff: Mentor

    Yes, that is the output of a MCMC.

    Are you familiar with Bayesian statistics? Each step of the MCMC method generates a sample of the posterior distribution of the parameters.

    Markov Chain Monte Carlo. It is the standard method for doing Bayesian statistics. It is a very interesting and powerful technique.
     
  9. May 29, 2017 #8

    WWGD

    User Avatar
    Science Advisor
    Gold Member

    Ah, yes, I just happen to call it E, so that E=MCMC= (MC)^2 .... So Close ; ).
     
  10. May 29, 2017 #9

    Dale

    Staff: Mentor

    Haha!
     
  11. May 29, 2017 #10
    Not as much as I should lol. This is my first time doing anything meaningful with statistics. So I have my posterior distribution now. From there do I just find the standard deviation for my parameters and use the formulas I learned in intro to Stat to find the confidence intervals around the median?
     
  12. May 29, 2017 #11

    Dale

    Staff: Mentor

    No, since you have a good sample of the posterior distribution you don't need to do any formulas. If your software package can generate the empirical CDF, then just use that to take any credible interval you like. Many Bayesian packages have a function for obtaining the highest posterior density interval at any given level. If your package doesn't have any of those then just sort your posterior samples and pick the appropriate range.
     
  13. May 29, 2017 #12
    I'm using Mathematica. So I form the CDF but I'm not clear on what to do from there. How do I decide the appropriate range.
     
  14. May 29, 2017 #13

    Dale

    Staff: Mentor

    Well, if you want a 95% confidence interval then you would take the 0.025 quantile and the 0.975 quantile from the CDF
     
  15. May 29, 2017 #14

    Stephen Tashi

    User Avatar
    Science Advisor

    It isn't clear what you mean by a confidence interval for the parameters of the model.

    Is the general scenario that you had a prior distributions for the parameters and obtained a posterior distribution for them?

    Or did you set some criteria for "best" fit and solve the problem as an optimization problem without specifying a prior distribution for the parameters?

    "Confidence interval" is a concept from frequentist statistics, but MCMC is usually used in Bayesian statistics.
     
  16. May 29, 2017 #15
    I'm trying to reproduce the 68% confidence level in this paper. https://arxiv.org/pdf/1205.1613v1.pdf. I never studied this type of statistics before which is why I have a lot of questions and really don't understand the terminology.
     
  17. May 29, 2017 #16
    Alright I think I found the 95% confidence interval for my parameters from the posterior distribution. To check though the boundary of my intervals should be elements of my posterior distribution right? Also what is the quantile for the 68% interval and where are those quantile derived from. Thank you so much for all of your help so far.
     
  18. May 30, 2017 #17

    Dale

    Staff: Mentor

    It would be (1-0.68)/2 and 1-(1-0.68)/2
     
  19. May 30, 2017 #18

    WWGD

    User Avatar
    Science Advisor
    Gold Member

    Does EDIT the process of obtaining confidence intervals in MCMC use some version of the CLT, i.e., do we use i.i.d random variables for each approximation? Otherwise, how do we apply confidence intervals to generic distributions? Hope this does not sound too confused. EDIT: for example, how would we compute a confidence interval for the median, variance, etc if not by using a version of CLT? I know, e.g., the sampling median obtained from a normal population has a know distribution, but, AFAIK, it does not have a nice form otherwise.
     
    Last edited: May 30, 2017
  20. May 30, 2017 #19

    Dale

    Staff: Mentor

    Bayesian statistics is fundamentally different in that respect. The CLT doesn't really get used in Bayesian statistics.

    In frequentist statistics you always work with probabilities that are defined as proportions over an imagined infinite number of trials given a fixed true hypothesis.

    In Bayesian statistics it is the hypothesis that is uncertain given a fixed set of data. You don't assume anything about infinite numbers of repetitions of the experiment, you just refine your hypothesis as much as possible given the data that you actually have.
     
  21. May 30, 2017 #20

    Stephen Tashi

    User Avatar
    Science Advisor

    True, but looking at the paper that the OP linked, I can't tell whether Bayesian statistics is being used. (I'm not a cosmologist.) The term "priors" for the cosmic microwave background radiation (CMBR) is used, but this may not refer to a probability distribution in the Bayesian sense of a prior.

    Can anyone comment on what "priors" means in the context of CMBR?
     
Know someone interested in this topic? Share this thread via Reddit, Google+, Twitter, or Facebook

Have something to add?
Draft saved Draft deleted



Similar Discussions: How To Find 68% Confidence Interval After Running MCMC
  1. Confidence Interval (Replies: 4)

Loading...