I collect a set of data (n = 100 observations) containing a single predictor and a quantitative response. I then fit a linear regression model to the data, as well as a separate cubic regression.
1) Suppose that the true relationship between X and Y is linear. Consider the training residual sum of squares (RSS) for the linear regression, and also the training RSS for the cubic regression. Would we expect one to be lower than the other, would we expect them to be the same, or is there not enough information to tell? Justify your answer.
2) Answer the above using test rather than training RSS.
3) Suppose that the true relationship between X and Y is not linear, but we don’t know how far it is from linear. Consider the training RSS for the linear regression, and also the training RSS for the cubic regression. Would we expect one to be lower than the other, would we expect them to be the same, or is there not enough information to tell? Justify your answer.
4) Answer the above with test rather than training RSS.
The Attempt at a Solution
Attempt at 1: Not enough information since the training data could be wobbly, which in that case despite the true linear relationship, the cubic might fit better. But the training data could also be fairly linear, so the linear would be better and the cubic too wobbly.
Attempt at 2: In this case, the linear will be better since we are using the test RSS and if it is truly linear, then the linear regression should give a lower RSS since the fit will be better than the cubic.
Attempt at 3: Chances are that the cubic regression will provide the lower RSS. The linear will not provide a good fit for the non-linear relationship and even if the training data is less or more non-linear than the cubic regression, the cubic should provide the lower RSS since it should provide the better fit since it has more coefficients than the linear regression.
Attempt at 4: The cubic regression should give a lower RSS since it is not linear and our true relationship is not linear.
Attempt at 5: I'm assuming I have to solve for y_i from the ß^hat equation and then figure out what the a_i means given that, but I'm stuck on how to solve for y_i.
Any tips, help, corrections, etc. would be great.