Can Alternative Least Squares Methods Be Used in Linear Regression?

[fog37]

TL;DR

Understanding if linear regression can be done with other variants of least squares

Hello,

Simple linear regression aims at finding the slope and intercept of the best-fit line to for a pair of $X$ and $Y$ variables.
In general, the optimal intercept and slope are found using OLS. However, I learned that "O" means ordinary and there are other types of least square computations...

Question: is it possible to apply those variants of LS to the linear regression model, i.e., can we find the best-fit line parameters using something other than OLS (for example, I think there is "robust" LS, etc.)?

thank you!

 

[scottdave]

I have taken a course in regression and some courses that use regression techniques. From what I remember, the Ordinary in OLS refers to some assumptions we make, rather than the method
one assumption is: the residuals are randomly distributed.

I like this textbook (free download) https://www.statlearning.com/

 

[FactChecker]

Simple linear regression aims at finding the slope and intercept of the best-fit line to for a pair of $X$ and $Y$ variables.

More precisely, it finds the line that uses the $X$ value to estimate the $Y$ values with the minimum sum-squared-errors for the $Y$ estimates. The phrase "best-fit line" can mean something different, referring to minimizing the sum-squared perpendicular distances from the data to the line.

In general, the optimal intercept and slope are found using OLS. However, I learned that "O" means ordinary and there are other types of least square computations...

Question: is it possible to apply those variants of LS to the linear regression model, i.e., can we find the best-fit line parameters using something other than OLS (for example, I think there is "robust" LS, etc.)?

This is an interesting question. I am not an expert in this, but I see ( https://en.wikipedia.org/wiki/Robust_regression ) that there are attempts to decrease the influence of outliers. Some methods have been implemented in R (see https://stat.ethz.ch/R-manual/R-patched/library/MASS/html/rlm.html ). I don't know if that implementation is publicly available. It is applied in an example in https://stats.oarc.ucla.edu/r/dae/robust-regression/

 

[Office_Shredder]

Even least squares is not necessary. You can find a slope and intercept that minimize any penalty function you want.

 

[FactChecker]

Even least squares is not necessary. You can find a slope and intercept that minimize any penalty function you want.

Good point, although most penalty functions would require non-analytical iterative minimization algorithms that are less intuitive. Also, I do not know what the risk of introducing local minimums would be.

 

[Stephen Tashi]

In general, the optimal intercept and slope are found using OLS.

When you read about this, what was the definition of "optimal"?

(A mathematical definition can be given in the context of statistical estimation and the properties of estimators.)

 

[statdad]

TL;DR Summary: Understanding if linear regression can be done with other variants of least squares

Hello,

Simple linear regression aims at finding the slope and intercept of the best-fit line to for a pair of $X$ and $Y$ variables.
In general, the optimal intercept and slope are found using OLS. However, I learned that "O" means ordinary and there are other types of least square computations...

Question: is it possible to apply those variants of LS to the linear regression model, i.e., can we find the best-fit line parameters using something other than OLS (for example, I think there is "robust" LS, etc.)?

thank you!

The phrase "best fit line" is meaningless without some context.

In least squares regression the best fit line is the one that minimizes the sum of the squared residuals.

In robust regression based on the ideas that grew from Huber's M-estimate based ideas the residuals are not squared but are fed into some other function [designed to lessen the impact of certain kinds of outliers] and the best fit line is the one that minimizes the sum of the values of those function values.

In robust regression [at least the simplest methods, based on the initial work of Jana Jurečková, later expanded on by Joseph McKean and others] based on R-estimate [rank estimates] we view the squared residuals as (residual)(residual), replace one of the two factors by a weighted rank of the residual, and the best fit line is the one that minimizes the sum of those.
You should also look up the idea of Tukey's resistant line.

There are others, but the point is that phrases like "optimal" and "line of best fit" need to be placed into some context: optimal or best in what sense?