Weighted Least Squares for coefficients

[divB]

Hi,

I have an ordinary least squares setup y = Ac where A is an NxM (N>>M) matrix, c the unknown coefficients and y the measurements.

Now WEIGHTED least squares allows to weight the MEASUREMENTS if, for example, some measurements are more important or contain a lower variance.

However, I am looking for a solution of putting weights on the coefficient vector. Pictorially speaking, some values in my coefficient vector c are more important than others and I would like emphasize some more than others using my limited set of N measurements.

Just adding a diagonal weighting matrix w as follows does not work:

$$\min_c \| y - Awc \|_2$$

 

[mfb]

How can some unknown coefficients be "more important than others"? I don't think that is a meaningful concept.

 

[divB]

But there are cases where it makes sense. Not everything is a linear system. In my case I clearly see that perturbing coefficients with the same noise gives different results, depending on which I perturb. So some are more important than others.

It's difficult to explain but I tried to explain the setup already some time ago in a different context (https://www.physicsforums.com/showpost.php?p=4533702&postcount=7 ff.).

 

[Stephen Tashi]

Not everything is a linear system..

You need to explain why you are asking about the system of linears equations given by y = Ac. I suggest you explain the problem you are trying to solve.

 

[divB]

Ok, it is a Volterra series. So it is linear in its coefficients but a non-linear system. But I think it does not matter. Anyway, since for a practical system the coefficients decay very fast, lower-order terms dominate the total error. But if you are interested in certain non-linear behavior you want to put more emphasis on those terms.

 

[mfb]

In my case I clearly see that perturbing coefficients with the same noise gives different results, depending on which I perturb. So some are more important than others.

So you are talking about correlations?
Sure, calculate the correlation matrix for your parameters.

 

[divB]

Am I? I am not sure ... at least I do not see how.

Can you provide me with an idea how this relates to the described setup and how to obtain the correlation matrix then?

The one thing I know is that for my application, some coefficients are more important for me - I do not know how they are cross-correlated with themselves! Therefore I only know the structure but generally I do not know their statistics (although it could be useful to leverage this too).

Indenpendently from that, how would I add this information to solve for the coefficients? Does it still work for Least Squares? Are you referring to MAP estimation?

 

[mfb]

It's hard to understand the problem with pieces of information scattered across multiple posts like this.

some coefficients are more important for me

TThe fit does not care (and does not have to care) which parameters are more interesting for you. It gives you a description which parameters work best, and their corresponding uncertainties and correlations.

 

[divB]

TThe fit does not care (and does not have to care) which parameters are more interesting for you.

And how could I make it care?

 

[mfb]

The fit must not and cannot care about that. This is then your interpretation of the fit result.