Solving for least square coefficients of power law

[enfield]

$$y_i=A{x_i}^b$$

When I solve for A two different ways I am getting different answers..so somewhere I'm doing something wrong. If someone could point out where I would be grateful :).

Using logs:

$$y_i=A{x_i}^b$$
$$ln(y_i)=ln(A)+b*ln(x_i)$$
$$ln(y_i)-(ln(A)+b*ln(x_i))=r_i$$ for least squares we want to minimize: $$S=\sum_i^{n}{r_i}^2$$ which means the gradient has to be zero. I only care about finding A right now so I only have to deal with the partial with respect to A:
$$\frac{\partial S}{\partial A}\sum_i^{n}{r_i}^2 =2 \sum_i^{n}{r_i}\frac{\partial r_i}{\partial A}= 2\sum_i^{n}(ln(y_i)-ln(A)-b*ln(x_i))\frac{1}{A}=0$$

The numerator inside the sum has to be zero, and we can ignore the 2, so:

$$\sum_i^{n}(ln(y_i)-ln(A)-b*ln(x_i))=0$$
$$\sum_i^{n}ln(y_i)-n*ln(A)-b\sum_i^{n}ln(x_i)=0$$
$$ln(A)=\frac{\sum_i^{n}ln(y_i)-b\sum_i^{n}ln(x_i)}{n}$$

(this is the derivation that I think is correct).
But when i solve for A without taking the logs of each side first I get something else:

$$y_i - A{x_i}^b = r_i$$$$\frac{\partial S}{\partial A}\sum_i^{n}{r_i}^2 =2 \sum_i^{n}{r_i}\frac{\partial r_i}{\partial A}= 2\sum_i^{n}(y_i - A{x_i}^b)*-{x_i}^b=0$$

$$\sum_i^{n}(-{x_i}{y_i} + A{x_i}^{2b})=0$$

$$-\sum_i^{n}{x_i}{y_i}+A\sum_i^{n}{x_i}^{2b}=0$$
$$A=\frac{\sum_i^{n}{x_i}{y_i}}{\sum_i^{n}{x_i}^{2b}}$$

And if you take the ln of it to compare it with what we got before you get:

$$ln(A)= ln(\frac{\sum_i^{n}{x_i}{y_i}}{\sum_i^{n}{x_i}^{2b}}) =ln(\sum_i^{n}{x_i}{y_i})-ln(\sum_i^{n}{x_i}^{2b})$$

Which is not the same as:

$$ln(A)=\frac{\sum_i^{n}ln(y_i)-b\sum_i^{n}ln(x_i)}{n}$$

as far as I can tell...

 

[hotvette]

That's because you solved two different problems.

Minimizing $\sum[ln(y_i)-ln(A)-b*ln(x_i)]^2$ is not necessarily the same thing as minimizing $\sum(y_i - A{x_i}^b)^2$. The 2nd method will result in a lower residual unless the data happen to fit the curve exactly (i.e. zero residual), in which case both will yield the same result.

 

[enfield]

Ah, thanks so much. Yeah, I had thought they were maybe different problems, but I couldn't see exactly how.

When you do least squares with the log of the data and function you are fitting to the data you won't get the same coefficients for the function that you would if you just did least squares with the unmodified data/function.

Okay, that makes sense. Does lower residual mean lower mean squared error, so better fit?

 

[hotvette]

Does lower residual mean lower mean squared error, so better fit?

.

Yes. Two different words for the same thing.

 

[blue_raver22]

It appears that there may be an error in your derivation when you do not take the logs of each side first. In this case, the partial derivative with respect to A should be:

\frac{\partial S}{\partial A}\sum_i^{n}{r_i}^2 =2 \sum_i^{n}{r_i}\frac{\partial r_i}{\partial A}= -2\sum_i^{n}{x_i}^b(y_i - A{x_i}^b)=0

This leads to the correct solution:

A=\frac{\sum_i^{n}{x_i}{y_i}}{\sum_i^{n}{x_i}^{2b}}

When comparing this to the solution obtained by taking the logs, we can see that they are equivalent:

ln(A)=ln(\frac{\sum_i^{n}{x_i}{y_i}}{\sum_i^{n}{x_i}^{2b}})=ln(\sum_i^{n}{x_i}{y_i})-ln(\sum_i^{n}{x_i}^{2b})

Therefore, the discrepancy in your solutions may be due to an error in your derivation when not taking the logs of each side first.