Can Alternative Least Squares Methods Be Used in Linear Regression?

In summary, linear regression is a statistical method used to find the best-fit line for a pair of variables. The optimal intercept and slope are typically found using OLS, but there are other variants of least squares that can be applied, such as robust LS. These methods aim to decrease the influence of outliers and find the best-fit line using different penalty functions. The definition of "optimal" in this context can vary depending on the method used.
  • #1
fog37
1,568
108
TL;DR Summary
Understanding if linear regression can be done with other variants of least squares
Hello,

Simple linear regression aims at finding the slope and intercept of the best-fit line to for a pair of ##X## and ##Y## variables.
In general, the optimal intercept and slope are found using OLS. However, I learned that "O" means ordinary and there are other types of least square computations...

Question: is it possible to apply those variants of LS to the linear regression model, i.e., can we find the best-fit line parameters using something other than OLS (for example, I think there is "robust" LS, etc.)?

thank you!
 
Physics news on Phys.org
  • #2
I have taken a course in regression and some courses that use regression techniques. From what I remember, the Ordinary in OLS refers to some assumptions we make, rather than the method
one assumption is: the residuals are randomly distributed.

I like this textbook (free download) https://www.statlearning.com/
 
  • Like
Likes BvU
  • #3
fog37 said:
Simple linear regression aims at finding the slope and intercept of the best-fit line to for a pair of ##X## and ##Y## variables.
More precisely, it finds the line that uses the ##X## value to estimate the ##Y## values with the minimum sum-squared-errors for the ##Y## estimates. The phrase "best-fit line" can mean something different, referring to minimizing the sum-squared perpendicular distances from the data to the line.
fog37 said:
In general, the optimal intercept and slope are found using OLS. However, I learned that "O" means ordinary and there are other types of least square computations...

Question: is it possible to apply those variants of LS to the linear regression model, i.e., can we find the best-fit line parameters using something other than OLS (for example, I think there is "robust" LS, etc.)?
This is an interesting question. I am not an expert in this, but I see ( https://en.wikipedia.org/wiki/Robust_regression ) that there are attempts to decrease the influence of outliers. Some methods have been implemented in R (see https://stat.ethz.ch/R-manual/R-patched/library/MASS/html/rlm.html ). I don't know if that implementation is publicly available. It is applied in an example in https://stats.oarc.ucla.edu/r/dae/robust-regression/
 
  • Like
Likes fog37
  • #4
Even least squares is not necessary. You can find a slope and intercept that minimize any penalty function you want.
 
  • Like
Likes fog37, scottdave and FactChecker
  • #5
Office_Shredder said:
Even least squares is not necessary. You can find a slope and intercept that minimize any penalty function you want.
Good point, although most penalty functions would require non-analytical iterative minimization algorithms that are less intuitive. Also, I do not know what the risk of introducing local minimums would be.
 
  • Like
Likes fog37
  • #6
fog37 said:
In general, the optimal intercept and slope are found using OLS.

When you read about this, what was the definition of "optimal"?

(A mathematical definition can be given in the context of statistical estimation and the properties of estimators.)
 
  • Like
Likes FactChecker
  • #7
fog37 said:
TL;DR Summary: Understanding if linear regression can be done with other variants of least squares

Hello,

Simple linear regression aims at finding the slope and intercept of the best-fit line to for a pair of ##X## and ##Y## variables.
In general, the optimal intercept and slope are found using OLS. However, I learned that "O" means ordinary and there are other types of least square computations...

Question: is it possible to apply those variants of LS to the linear regression model, i.e., can we find the best-fit line parameters using something other than OLS (for example, I think there is "robust" LS, etc.)?

thank you!
The phrase "best fit line" is meaningless without some context.

In least squares regression the best fit line is the one that minimizes the sum of the squared residuals.

In robust regression based on the ideas that grew from Huber's M-estimate based ideas the residuals are not squared but are fed into some other function [designed to lessen the impact of certain kinds of outliers] and the best fit line is the one that minimizes the sum of the values of those function values.

In robust regression [at least the simplest methods, based on the initial work of Jana Jurečková, later expanded on by Joseph McKean and others] based on R-estimate [rank estimates] we view the squared residuals as (residual)(residual), replace one of the two factors by a weighted rank of the residual, and the best fit line is the one that minimizes the sum of those.
You should also look up the idea of Tukey's resistant line.

There are others, but the point is that phrases like "optimal" and "line of best fit" need to be placed into some context: optimal or best in what sense?
 
  • Like
Likes scottdave and FactChecker

Similar threads

  • Set Theory, Logic, Probability, Statistics
Replies
3
Views
1K
  • Set Theory, Logic, Probability, Statistics
Replies
23
Views
2K
  • Set Theory, Logic, Probability, Statistics
Replies
30
Views
3K
  • Set Theory, Logic, Probability, Statistics
Replies
7
Views
969
  • Set Theory, Logic, Probability, Statistics
Replies
8
Views
2K
  • Set Theory, Logic, Probability, Statistics
Replies
4
Views
1K
  • Set Theory, Logic, Probability, Statistics
Replies
4
Views
1K
  • Set Theory, Logic, Probability, Statistics
Replies
5
Views
1K
  • Set Theory, Logic, Probability, Statistics
Replies
6
Views
2K
  • Set Theory, Logic, Probability, Statistics
Replies
2
Views
936
Back
Top