Linear Regression with Non Linear Basis Functions

In summary, the conversation discusses the use of regression techniques and linear basis functions in research. It also explains the process for using non-linear basis functions, such as Gaussian basis functions, and the construction of the model equation. The conversation concludes with a clarification on the parameters being estimated and the use of standard linear regression.
  • #1
joshthekid
46
1
So I am currently learning some regression techniques for my research and have been reading a text that describes linear regression in terms of basis functions. I got linear basis functions down and no exactly how to get there because I saw this a lot in my undergrad basically, in matrix notation
y=wTx
you then define your loss function as
1/n Σn(wi*xi-yi)2
then you take the partial derivatives with respect to w set it equal to zero and solve.

So now I want to use a non-linear basis functions, let's say I want to use m gaussians basis functions, φi, the procedure is the same but I am not sure exactly on the construction of the model. Let's say I have L features is the model equation of the form

ynmΣLwiφi(xj)

in other words I have created a linear combination of M new features, φ(x), which are constructed with all L of the previous features for each data point n:
yn=w0+w11(x1)+φ1(x2)...+...φ1(xL) ...+...wm1(x1)+φ2(x2)...+...φm(xL))

where xi are features / variables for my model and not data values? I hope this makes sense. Thanks in advance.
 
Physics news on Phys.org
  • #2
The parameters you wish to estimate are the ##w_i## and the values ##(x_1,...,x_L)## are known for each data point?
 
  • #3
micromass said:
The parameters you wish to estimate are the ##w_i## and the values ##(x_1,...,x_L)## are known for each data point?

That is correct.
 
  • #4
Then you have a standard linear regression. Linear refers to the coefficients and not the functions used. Thus your loss function is again

[tex]L = \sum_{i=1}^n \left(y_i - w_0 - w_1\sum_k \phi_1(x_k) - w_2\sum_k \phi_2(x_k) - ... - w_N \sum_k \phi_N(x_k)\right)^2[/tex]

and you minimize this by taking partial derivatives and setting them equal to ##0##. In matrix notation, you let ##Y## by the column matrix with entries the ##y_i## and you let ##X## be the design matrix whose ##i##th row is
[tex]\left(1~~\sum_k \phi_1(x_k)~~ ...~~\sum_k \phi_N(x_k)\right)[/tex]
The coefficients are then ##W = (X^TX)^{-1} X^T Y##.
 
  • #5
micromass said:
Then you have a standard linear regression. Linear refers to the coefficients and not the functions used. Thus your loss function is again

[tex]L = \sum_{i=1}^n \left(y_i - w_0 - w_1\sum_k \phi_1(x_k) - w_2\sum_k \phi_2(x_k) - ... - w_N \sum_k \phi_N(x_k)\right)^2[/tex]

and you minimize this by taking partial derivatives and setting them equal to ##0##. In matrix notation, you let ##Y## by the column matrix with entries the ##y_i## and you let ##X## be the design matrix whose ##i##th row is
[tex]\left(1~~\sum_k \phi_1(x_k)~~ ...~~\sum_k \phi_N(x_k)\right)[/tex]
The coefficients are then ##W = (X^TX)^{-1} X^T Y##.
Great Thanks, this is what I thought it meant but the way you wrote it makes it lot clearer than the text I am using which has all formulas in matrix notation and it hard to tell if they are talking about a single random variable or a vector of random variables.
 

1. What is linear regression with non-linear basis functions?

Linear regression with non-linear basis functions is a statistical method used for modeling the relationship between a dependent variable and one or more independent variables. It assumes that the relationship between the variables can be approximated by a linear combination of non-linear basis functions.

2. How is linear regression with non-linear basis functions different from traditional linear regression?

In traditional linear regression, the relationship between the variables is assumed to be linear. However, in linear regression with non-linear basis functions, the relationship can be non-linear as it allows for the use of non-linear basis functions to model the relationship.

3. What are some examples of non-linear basis functions?

Some examples of non-linear basis functions include polynomial functions, trigonometric functions, and exponential functions. These functions can be used to transform the input variables into non-linear combinations, allowing for a more flexible and accurate model.

4. How do you choose the appropriate non-linear basis functions for a linear regression model?

The choice of non-linear basis functions depends on the data and the specific problem being addressed. It is important to analyze the data and understand the relationship between the variables before selecting the appropriate non-linear basis functions. Additionally, techniques such as cross-validation can be used to evaluate the performance of different non-linear basis functions.

5. What are the advantages of using linear regression with non-linear basis functions?

Linear regression with non-linear basis functions allows for a more flexible and accurate model as it can capture non-linear relationships between variables. It also helps in reducing bias and improving the predictive power of the model. Additionally, it can be used to handle outliers and missing data, making it a versatile and powerful tool for data analysis.

Similar threads

  • Set Theory, Logic, Probability, Statistics
Replies
3
Views
658
  • Set Theory, Logic, Probability, Statistics
2
Replies
64
Views
3K
  • Set Theory, Logic, Probability, Statistics
Replies
9
Views
1K
Replies
8
Views
1K
  • Set Theory, Logic, Probability, Statistics
Replies
2
Views
2K
  • Set Theory, Logic, Probability, Statistics
Replies
13
Views
1K
Replies
4
Views
1K
  • Set Theory, Logic, Probability, Statistics
Replies
2
Views
9K
  • Calculus and Beyond Homework Help
Replies
10
Views
1K
  • Calculus and Beyond Homework Help
Replies
3
Views
2K
Back
Top