Linear Regression Models (3)

[kingwinner]

1) "Simple linear regression model:
Y = β₀ + β₁X + ε
E(Y) = β₀ + β₁X
A linear model means that it is linear in β's, and not necessarily a linear function of X.
The independent variable X could be W² or ln(W), and so on, for some other independent variable W."

I have some trouble understanding the last line. I was told that a SIMPLE linear regression model is always a straight line model, it is a least-square LINE of best fit. But if X=W², then we have E(Y) = β₀ + β₁W² which is not a straight line...how come?? Is this allowed?


2) "A SIMPLE linear regression is a linear regression in which there is only ONE independent variable."

Now is the following a simple linear regression or a multiple linear regression?
Y = β₀ + β₁X + β₂X² + ε
It has only one independent variable X, so is it simple linear regression? But this just looks a bit funny to me...


3) "A linear regression model is of the form:
Y = β₀ + β₁X₁ + β₂X₂ + ... + β_kX_k + ε
If there is more than one independent variable, then the model is called a MULTIPLE linear regression model."

This idea doesn't seem too clear to me. What can the X_i's be? What are some actual examples of mutliple linear model? Does a linear model always have to be a straight line or a plane?

Thanks for explaining!

 

[D H]

1) have some trouble understanding the last line. I was told that a SIMPLE linear regression model is always a straight line model, it is a least-square LINE of best fit. But if X=W², then we have E(Y) = β₀ + β₁W² which is not a straight line...how come?? Is this allowed?

Think of it this way. You have a bunch of (x,y) pairs and are trying to find the coefficients a and b for y=ax²+b. Introduce a new variable u=x². Now the equation you are trying to fit is y=au+b. A straight line fit. Now imagine you have a different set of (x,y) pairs and this time you are trying to find the coefficients a and b for y=bx^a. Introduce two new variables, u=ln(x) and v=ln(y). Taking the log of both sides of y=ax^b and substituting yields v=au+b. A straight line fit.

A bit of caution with regard to the latter. The linear regression yields the best fit (least squares sense) to v=au+b. This is not necessarily the best fit (least squares sense) to y=bx^a.

2) "A SIMPLE linear regression is a linear regression in which there is only ONE independent variable."

Now is the following a simple linear regression or a multiple linear regression?
Y = β₀ + β₁X + β₂X² + ε
It has only one independent variable X, so is it simple linear regression? But this just looks a bit funny to me...

No. The X and X² are different independent variables as far as the regression goes.

3) "A linear regression model is of the form:
Y = β₀ + β₁X₁ + β₂X₂ + ... + β_kX_k + ε
If there is more than one independent variable, then the model is called a MULTIPLE linear regression model."

This idea doesn't seem too clear to me. What can the X_i's be? What are some actual examples of mutliple linear model? Does a linear model always have to be a straight line or a plane?

Fitting salary y to years of schooling s and years of experience e via y=as+be+c is a multiple linear regression. Here, years of schooling and years of experience are independent variables for the regression. Fitting a parabola, y=ax²+bx+c, can also be done as a multiple linear regression. Think of x² and x as being independent variables as far as the regression is concerned.

 

[kingwinner]

2) What I think is that the definition of "simple linear model" is not very well-defined. It's ambiguous. I looked at the definitions in 3 different textbooks, but still can't really figure out whether e.g. Y = β₀ + β₁X + β₂X² + ε is a simple linear model or mutliple linear model. There seems to be only ONE independent variable X (X² is also determined by X, it's not a DIFFERENT variable, once we've measured X, we can determine the values of both X and X²), but it has a β₂ in there. X and X² are related, I don't see how they can be two separate independent variables...
Is there a nicer definition of a "simple linear model"?


3) "A linear regression model is of the form:
Y = β₀ + β₁X₁ + β₂X₂ + ... + β_kX_k + ε
If there is more than one independent variable, then the model is called a MULTIPLE linear regression model."

Now I have some confusion relating to the above paragraph.
e.g. Are the following also considered as MULTIPLE linear regression models? These are not quite in the exact same form as Y = β₀ + β₁X₁ + β₂X₂ + ... + β_kX_k + ε which has "k" DIFFERENT independent variables X₁,X₂,...,X_k.
(i) Y = β₀ + β₁X + β₂exp(X) + ε
(ii) Y = β₀ + β₁X₁ + β₂X₂ + β₃(X₁X₂) + β₄(X₁²) + β₅(X₂²) + ε
Are those allowed? Why or why not?


Thanks a lot!

 

[D H]

A simple linear regression model has two coefficients. Period.

Your problem is that you are looking at this the wrong way. Y = β₀ + β₁X + β₂X² + ε is not a simple model because you have three coefficients: β₀, β₁, and β₂. In a sense, the independent variables for the regression are the β_is. As far as the regression equations are concerned, those Xs and Ys are just a bunch of constant N-vectors. The best fit is found by taking the partial derivatives of the sum of the square error with respect to each β_i: The β_is are variables. The Xs and Ys are not variables as far as the regression equations are concerned. Stop thinking of them as variables and you will have fewer problems.

 

[kingwinner]

Thanks for the helpful comments! So I think 2) is solved.

But I am still puzzled by 3) and I would really appreicate if anyone can explain that.

3) "A linear regression model is of the form:
Y = β₀ + β₁X₁ + β₂X₂ + ... + β_kX_k + ε
If there is more than one independent variable, then the model is called a MULTIPLE linear regression model."

Now I have some confusion relating to the above paragraph.
e.g. Are the following also considered as MULTIPLE linear regression models? These are not quite in the exact same form as Y = β₀ + β₁X₁ + β₂X₂ + ... + β_kX_k + ε which has "k" DIFFERENT independent variables X₁,X₂,...,X_k.
(i) Y = β₀ + β₁X + β₂exp(X) + ε
(ii) Y = β₀ + β₁X₁ + β₂X₂ + β₃(X₁X₂) + β₄(X₁²) + β₅(X₂²) + ε
Are those allowed? Why or why not?

 

[HallsofIvy]

2)
3) "A linear regression model is of the form:
Y = β₀ + β₁X₁ + β₂X₂ + ... + β_kX_k + ε
If there is more than one independent variable, then the model is called a MULTIPLE linear regression model."

Now I have some confusion relating to the above paragraph.
e.g. Are the following also considered as MULTIPLE linear regression models? These are not quite in the exact same form as Y = β₀ + β₁X₁ + β₂X₂ + ... + β_kX_k + ε which has "k" DIFFERENT independent variables X₁,X₂,...,X_k.
(i) Y = β₀ + β₁X + β₂exp(X) + ε
(ii) Y = β₀ + β₁X₁ + β₂X₂ + β₃(X₁X₂) + β₄(X₁²) + β₅(X₂²) + ε
Are those allowed? Why or why not?


Thanks a lot!

They aren't linear! That's not to say that those might not be better models for the particular situation- not everything is linear- but anything can be approximated by a linear model and linear models are much, much, easier to work with!

 

[D H]

(i) Y = β₀ + β₁X + β₂exp(X) + ε
(ii) Y = β₀ + β₁X₁ + β₂X₂ + β₃(X₁X₂) + β₄(X₁²) + β₅(X₂²) + ε
Are those allowed? Why or why not?

They aren't linear!

They are linear in the β_is, and as far as linear regression is concerned, that is all that matters. These are linear regression models. Here are a couple that are not linear regressions:

$$Y = \beta_0*(1 + \beta_1 X_1)*(1 + \beta_2 X_2)+\varepsilon$$
$$Y=\beta_0 + \beta_1X^{\beta_2} + \varepsilon$$

 

[kingwinner]

They are linear in the β_is, and as far as linear regression is concerned, that is all that matters. These are linear regression models. Here are a couple that are not linear regressions:

$$Y = \beta_0*(1 + \beta_1 X_1)*(1 + \beta_2 X_2)+\varepsilon$$
$$Y=\beta_0 + \beta_1X^{\beta_2} + \varepsilon$$

Yes, I think the trickiest point to notice when I first read through the definition of linear regression model is that it is linear in β's while in calculus, when we talk about linear, we are usually trying to say that the function is linear, i.e. straight line, plane.

"A linear regression model is of the form:
Y = β₀ + β₁X₁ + β₂X₂ + ... + β_kX_k + ε "

(i) Y = β₀ + β₁X + β₂exp(X) + ε
(ii) Y = β₀ + β₁X₁ + β₂X₂ + β₃(X₁X₂) + β₄(X₁²) + β₅(X₂²) + ε

For (i), X1=X, X2=exp(X)
For (ii), X3=X1*X2, X4=X1^2, X5=X2^2
The latter X's depends on the previous X's. In particular, X3 depends on TWO of the previous X's: X1 AND X2, which looks a bit funny to me? Are those allowed? Somehow I am having a lot of troubles understanding this...I understand the general form of a multiple linear regression model, but I don't seem to understand the specific examples of it like (i) and (ii).

Once again, your help is greatly appreciated!