Confidence interval from least squares fitting?

[oaktree]

Hello,

Let me get right to my problem. I have an experimental distribution and a single-parameter theoretical distribution.

I want to find the value for the best fit theoretical distribution that agrees with my experimental data for the bulk of the distributions (the tails of my distributions differ substantially).

I isolate an equal portion of both distributions and calculate the sum of the squares of the differences between the two distributions for this region. i.e. least squares approach.
R²=Ʃ [ exp_i - theo_i(x) ]²

I do this for several values that I have chosen for the single-parameter theoretical distribution and obtain a unique parameter value (which I call x=ζ) which results in the minimization of the sum of the squares (exactly what I want). Every other parameter gives a larger value for this sum of squares.

I do not know if this is necessary but I can plot all parameters that I tested vs the R² too. This gives points that combine to form a parabola, ax²+bx+c=R². I can take the derivative of the parabola curve to obtain the minimum of the parabola which again occurs at ζ. I have attached a pdf of this.

My question is, how do I find a confidence interval for this? I am looking for \sigma in ζ ± \sigma. Do I use the formula for the parabola to find this? Do I use R²?

I have looked through some books and online and been unsuccessful on how to find this \sigma value. Any help is appreciated. A reference book would be of great help too. Thanks!

 

[Stephen Tashi]

I do this for several values that I have chosen for the single-parameter theoretical distribution and obtain a unique parameter value (which I call x=ζ) which results in the minimization of the sum of the squares (exactly what I want).

My question is, how do I find a confidence interval for this? I am looking for \sigma in ζ ± \sigma. Do I use the formula for the parabola to find this? Do I use R²?

I may (or may not!) understand the usual method for getting confidence intervals for parameters of a curve fit, I've remarked about it in several posts and not gotten any corrections - we'll see if my luck holds.

it involves some commonly made and implausible assumptions. It goes something like this. The value of the parameter is a known function of the data. I don't want to call the parameter "x". Let's call the parameter p and the say
p = F(X_1,X_2,...X_n) where the X_i are the data.

You may not know the symbolic expression for F , but you have a numerical method for computing it, namely your curve fitting algorithm.

Let's say that your particular curve fit found that p = p_0 when the specific data was X_1 = x_1, X_2 = x_2,... X_n = x_n.

Find (symbolically or numerically) the differential expression that approximates a change in p_0 as a function of changes in the x_i.p0 + \delta p = F(x_1,x_2,...) + \delta x_1 \frac{\partial _F}{\partial X_1} + \delta x_2 \frac{\partial _F}{\partial X_2} + ...

\delta p = \delta x_1 \frac{\partial _F}{\partial X_1} + \delta x_2 \frac{\partial _F}{\partial X_2} + ...

Think of the \delta x_i are the random errors in measuring each of the x_i. Assume these are independent normally distributed random variables with mean zero. Estimate their standard deviations. (You might have some way of doing this that is independent of the curve fit. I suppose some people use the "residuals" (i.e. the "error" between the values predicted by the curve and the actual data) to do this estimation. I'm not sure how that can be justified by logical reasoning! )

The above approximation expresses the random variable \delta p a linear function of the independent mean zero normal random variables \delta x_i, which have known standard deviations. From this you can compute the standard deviation of \delta p and say things about confidence intervals around p_0.

The method is suspicious enough that the technical term for it is not "confidence interval", although some software packages are brazen enough to call it that. I think it is technically a "linearized confidence interval" and it may need another adjective (that I don't recall at the moment) to warn people.

If your theoretical distribution is from a commonly used family of curves, you might be able to look up the formula for the linearized confidence interval that the above method produces.

 

[DrDu]

Least squares fit can be viewed at as a special case of a maximum likelihood proceedure where you assume that the logarithmic likelihood L is - up to some factors not depending on x - R^2.
That is, the probability density function for obtaining your data exp_i given x is
p(\{exp_i\})=C \exp(-R^2).
Now you turn the handle of general theorems about maximum likelihood to find that the asymptotic variance is \sigma^2= (\partial^2 R^2/\partial x^2)^{-1}, i.e. one over the second derivative of your parabola.

 

[Stephen Tashi]

Another thought: You could simply Monte-Carlo the problem. Simulate sets of data with various simulated "errors" and plot the distribution of the estimated parameter. Even if you do the problem another way, you could use a simulation to check your result.