Multivariate Linear Regression With Coefficient Constraint

In summary, to set a constraint on the coefficients in a multivariate linear regression, you need to use a restricted minimization problem and minimize the sum of squared errors (SSE).
  • #1
dansu
3
0
[SOLVED] Multivariate Linear Regression With Coefficient Constraint

I'm attempting a multivariate linear regression (mvlr) by method of least squares. Basically, I'm solving a matrix of the following form for [tex]\beta_p,[/tex][tex]
$ \begin{bmatrix} \sum y \\ \sum x_1 y \\ \sum x_2 y \\ \sum x_3 y \end{bmatrix} = \begin{bmatrix} n & \sum x_1 & \sum x_2 & \sum x_3 \\ \sum x_1 & \sum x_1^2 & \sum x_1 x_2 & \sum x_1 x_3 \\ \sum x_2 & \sum x_2 x_1 & \sum x_2^2 & \sum x_2 x_3 \\ \sum x_3 & \sum x_3 x_1 & \sum x_3 x_2 & \sum x_3^2 & \end{bmatrix}\begin{bmatrix} \beta_0 \\ \beta_1 \\ \beta_2 \\ \beta_3 \end{bmatrix} $\end{text}[/tex]

x are sets of data, y is the data I want to fit, and [tex]\beta[/tex] are the coefficients.

My problem is that I want to set a constraint such that [tex]\beta[/tex] remains positive. What would be a good way to achieve this?
 
Physics news on Phys.org
  • #2
In general, you need to set up a restricted minimization problem (Lagrangian) for the sum of squared errors (SSE); then minimize SSE subject to the constraint.

E.g. the REG procedure in SAS uses RESTRICT statement, which reverts to a constrained optimization algorithm.

Specifically, if you run the unrestricted MVLR and it outputs a positive coefficient then you don't need the constraint. If the unrestricted coefficient is < 0, then you need to use constrained minimization.
 
Last edited:
  • #3
If you don't mind, could you or somebody else explain that procedure in more detail? I have only minimal experience working with lagrange multipliers. I'm unfamiliar with SAS. I've been running my numbers through Matlab.
 
  • #4
In Mathematica NMinimize & NMaximize functions are used to solve nonlinear constrained global optimization problems. Perhaps you can search for similar algorithms in Matlab?
 
  • #5
This might work; but I'd constrain the beta's (to zero) one at a time, and do it alternatively.

E.g. if beta1 < 0 and beta3 < 0 in the full model then first minimize (y - b0 - b1 x1 - b2 x2 - b4 x4)^2, and see whether beta1 is still < 0. Alternatively, minimize (y - b0 - b2 x2 - b3 x3 - b4 x4)^2 and see whether beta3 is still < 0. If the answer to both questions is "yes," then minimize (y - b0 - b2 x2 - b4 x4)^2. In all other cases (yes&no, no&yes, no&no), you'll have a model selection problem and you will have to use a model selection criterion (e.g. the F statistic or the adjusted R-squared).
 
Last edited:
  • #6
thank you all for your help.

I have found that the MATLAB function lsqnonneg does exactly what I am looking for. I'm speculating that it follows a procedure similar to what vedenev and EnumaElish are suggesting.
 

1. What is multivariate linear regression with coefficient constraint?

Multivariate linear regression with coefficient constraint is a statistical method used to analyze the relationship between multiple independent variables and a single dependent variable. It differs from traditional linear regression by imposing constraints on the coefficients, which can help to improve the accuracy and interpretability of the model.

2. What are the benefits of using coefficient constraints in multivariate linear regression?

There are several benefits of using coefficient constraints in multivariate linear regression. First, it can help to prevent overfitting, which occurs when a model is too complex and performs well on the training data but poorly on new data. Additionally, it can improve the interpretability of the model by reducing the number of parameters and making them easier to understand. Finally, coefficient constraints can also help to handle multicollinearity, a common issue in linear regression where independent variables are highly correlated.

3. What types of constraints can be applied to the coefficients in multivariate linear regression?

There are several types of constraints that can be applied to the coefficients in multivariate linear regression. One common type is ridge regression, which adds a penalty term to the model to shrink the coefficients towards zero. Another type is lasso regression, which not only shrinks the coefficients but also performs variable selection by setting some coefficients to zero. Other constraints include elastic net, which combines ridge and lasso regression, and non-negative least squares, which restricts the coefficients to be positive.

4. How do I know if coefficient constraints are necessary for my data?

Whether or not coefficient constraints are necessary for your data depends on the specific characteristics of your dataset. If your data has a large number of highly correlated variables, then coefficient constraints may be beneficial to handle multicollinearity. Additionally, if you are concerned about overfitting or the interpretability of your model, coefficient constraints may be worth considering. It is always a good idea to try different methods and compare their performance to determine the best approach for your data.

5. Can coefficient constraints be used in other types of regression models?

Yes, coefficient constraints can be applied to other types of regression models, such as logistic regression and Poisson regression. In these cases, the constraints are typically applied to the regression coefficients for the independent variables, similar to how they are applied in multivariate linear regression. However, the specific types of constraints and their effects may vary depending on the model and the data being analyzed.

Similar threads

  • Set Theory, Logic, Probability, Statistics
Replies
4
Views
1K
  • Set Theory, Logic, Probability, Statistics
Replies
6
Views
1K
  • Set Theory, Logic, Probability, Statistics
Replies
3
Views
697
  • Set Theory, Logic, Probability, Statistics
Replies
13
Views
1K
  • Set Theory, Logic, Probability, Statistics
Replies
13
Views
2K
Replies
3
Views
619
  • Set Theory, Logic, Probability, Statistics
Replies
30
Views
2K
  • Set Theory, Logic, Probability, Statistics
Replies
7
Views
366
  • Set Theory, Logic, Probability, Statistics
Replies
4
Views
2K
  • Introductory Physics Homework Help
Replies
4
Views
684
Back
Top