Optimizing Pseudoexperiment Sample Size for Accurate Results

  • Context: Graduate 
  • Thread starter Thread starter ChrisVer
  • Start date Start date
Join the discussion
Registration is free. Ask a follow-up in this thread, or start your own.
3 replies · 1K views
Messages
3,372
Reaction score
465
How can someone decide what's the best number of produced pseudoexperiments in his set up?
In particular I have a value [itex]N[/itex] which I want to vary wrt 2 nuisance parameters with relative uncertainties [itex]\delta_1,\delta_2[/itex]. I am producing [itex]n[/itex] pseudoexperiments (samples) in each calculating the mean and the standard deviation of
[itex]N_i =N_i^0 \Big[1 + \delta_1 \mathcal{N}_1(0,1) + \delta_2 \mathcal{N}_2(0,1) \Big][/itex]
How can I decide whether the [itex]n[/itex]-trials I am choosing is optimal?
I have reached the following conclusion after some thinking but I am not sure... the sample's relative uncertainty that is [itex]\sqrt{\text{Var}(N)}/\bar{N}[/itex] should be as small as possible... is that a correct way?
 
Physics news on Phys.org
Vanadium 50 said:
The answer is "as many as you can". There is no point where another pseudo-experiment makes things worse and not better.

In general I've seen plots where they show the "observed" let's say value (that they vary in each PE), and the sampled values for different/increasing [itex]n[/itex]-trials (eg 100,1000,10000,1000000)... so, the sampled values get closer to the observed value, but also they get more statistically constrained... I was wondering if by looking at something like this allows someone to decide with what [itex]n[/itex] they should go.
 
As much as your computers can reasonably compute. The uncertainty from the limited number of pseudoexperiments goes down, which is great.