Weighted Least Squares for coefficients

Click For Summary

Discussion Overview

The discussion revolves around the concept of applying weights to the coefficient vector in a weighted least squares framework, particularly in the context of a Volterra series. Participants explore the implications of emphasizing certain coefficients over others and the challenges associated with this approach.

Discussion Character

  • Debate/contested
  • Technical explanation
  • Conceptual clarification

Main Points Raised

  • One participant proposes that in their ordinary least squares setup, certain coefficients in the vector c are more important than others and seeks a method to weight these coefficients.
  • Another participant questions the validity of the concept of "importance" among coefficients, suggesting it may not be meaningful.
  • A different participant argues that in non-linear systems, perturbing different coefficients can yield varying results, implying that some coefficients indeed hold more significance.
  • Participants discuss the nature of the system, with one clarifying that it is a Volterra series, which is linear in coefficients but represents a non-linear system.
  • There is mention of calculating a correlation matrix for parameters to understand their relationships, although one participant expresses uncertainty about how this relates to their setup.
  • Concerns are raised about the clarity of the problem due to fragmented information across posts, with some participants emphasizing that the fit itself does not account for the subjective importance of parameters.
  • Questions arise about how to incorporate the notion of importance into the least squares method and whether this relates to maximum a posteriori (MAP) estimation.

Areas of Agreement / Disagreement

Participants express differing views on the concept of weighting coefficients based on their importance, with some supporting the idea while others challenge its validity. The discussion remains unresolved regarding how to effectively incorporate these weights into the least squares framework.

Contextual Notes

Participants highlight limitations in their understanding of the correlations between coefficients and the statistical properties of the coefficients themselves, which may affect the application of their proposed methods.

divB
Messages
85
Reaction score
0
Hi,

I have an ordinary least squares setup y = Ac where A is an NxM (N>>M) matrix, c the unknown coefficients and y the measurements.

Now WEIGHTED least squares allows to weight the MEASUREMENTS if, for example, some measurements are more important or contain a lower variance.

However, I am looking for a solution of putting weights on the coefficient vector. Pictorially speaking, some values in my coefficient vector c are more important than others and I would like emphasize some more than others using my limited set of N measurements.

Just adding a diagonal weighting matrix w as follows does not work:

<br /> \min_c \| y - Awc \|_2<br />
 
Physics news on Phys.org
How can some unknown coefficients be "more important than others"? I don't think that is a meaningful concept.
 
But there are cases where it makes sense. Not everything is a linear system. In my case I clearly see that perturbing coefficients with the same noise gives different results, depending on which I perturb. So some are more important than others.

It's difficult to explain but I tried to explain the setup already some time ago in a different context (https://www.physicsforums.com/showpost.php?p=4533702&postcount=7 ff.).
 
divB said:
Not everything is a linear system..
You need to explain why you are asking about the system of linears equations given by y = Ac. I suggest you explain the problem you are trying to solve.
 
Ok, it is a Volterra series. So it is linear in its coefficients but a non-linear system. But I think it does not matter. Anyway, since for a practical system the coefficients decay very fast, lower-order terms dominate the total error. But if you are interested in certain non-linear behavior you want to put more emphasis on those terms.
 
divB said:
In my case I clearly see that perturbing coefficients with the same noise gives different results, depending on which I perturb. So some are more important than others.
So you are talking about correlations?
Sure, calculate the correlation matrix for your parameters.
 
Am I? I am not sure ... at least I do not see how.

Can you provide me with an idea how this relates to the described setup and how to obtain the correlation matrix then?

The one thing I know is that for my application, some coefficients are more important for me - I do not know how they are cross-correlated with themselves! Therefore I only know the structure but generally I do not know their statistics (although it could be useful to leverage this too).

Indenpendently from that, how would I add this information to solve for the coefficients? Does it still work for Least Squares? Are you referring to MAP estimation?
 
It's hard to understand the problem with pieces of information scattered across multiple posts like this.

some coefficients are more important for me
TThe fit does not care (and does not have to care) which parameters are more interesting for you. It gives you a description which parameters work best, and their corresponding uncertainties and correlations.
 
TThe fit does not care (and does not have to care) which parameters are more interesting for you.

And how could I make it care?
 
  • #10
The fit must not and cannot care about that. This is then your interpretation of the fit result.
 

Similar threads

Undergrad Linear least squares regression for model matrix identification
  • · Replies 5 ·
Replies
5
Views
2K
Graduate Large weighted least squares system
  • · Replies 4 ·
Replies
4
Views
1K
Undergrad What is Nonlinear Least Squares in Regression Analysis?
  • · Replies 4 ·
Replies
4
Views
2K
Graduate Confused about Weighted Least Squares
  • · Replies 2 ·
Replies
2
Views
2K
Undergrad Linear least-squares method and row multiplication of matrix
  • · Replies 2 ·
Replies
2
Views
2K
Undergrad Uncertainty of coefficients after a least square fit
  • · Replies 2 ·
Replies
2
Views
2K
Undergrad What is the expected (squared) coefficient of variation of a sample?
  • · Replies 6 ·
Replies
6
Views
3K
Undergrad Shouldn't this definition of a metric include a square root?
  • · Replies 8 ·
Replies
8
Views
3K
Graduate Convergence issue in this Least Squares calculation
  • · Replies 2 ·
Replies
2
Views
2K
Graduate How Do You Derive the Least Squares Solution in Linear Regression?
  • · Replies 2 ·
Replies
2
Views
3K