Linear Regression and OLS

• I
• fog37
In summary, linear regression is a statistical method used to find the best-fit line for a pair of variables. The optimal intercept and slope are typically found using OLS, but there are other variants of least squares that can be applied, such as robust LS. These methods aim to decrease the influence of outliers and find the best-fit line using different penalty functions. The definition of "optimal" in this context can vary depending on the method used.
fog37
TL;DR Summary
Understanding if linear regression can be done with other variants of least squares
Hello,

Simple linear regression aims at finding the slope and intercept of the best-fit line to for a pair of ##X## and ##Y## variables.
In general, the optimal intercept and slope are found using OLS. However, I learned that "O" means ordinary and there are other types of least square computations...

Question: is it possible to apply those variants of LS to the linear regression model, i.e., can we find the best-fit line parameters using something other than OLS (for example, I think there is "robust" LS, etc.)?

thank you!

I have taken a course in regression and some courses that use regression techniques. From what I remember, the Ordinary in OLS refers to some assumptions we make, rather than the method
one assumption is: the residuals are randomly distributed.

BvU
fog37 said:
Simple linear regression aims at finding the slope and intercept of the best-fit line to for a pair of ##X## and ##Y## variables.
More precisely, it finds the line that uses the ##X## value to estimate the ##Y## values with the minimum sum-squared-errors for the ##Y## estimates. The phrase "best-fit line" can mean something different, referring to minimizing the sum-squared perpendicular distances from the data to the line.
fog37 said:
In general, the optimal intercept and slope are found using OLS. However, I learned that "O" means ordinary and there are other types of least square computations...

Question: is it possible to apply those variants of LS to the linear regression model, i.e., can we find the best-fit line parameters using something other than OLS (for example, I think there is "robust" LS, etc.)?
This is an interesting question. I am not an expert in this, but I see ( https://en.wikipedia.org/wiki/Robust_regression ) that there are attempts to decrease the influence of outliers. Some methods have been implemented in R (see https://stat.ethz.ch/R-manual/R-patched/library/MASS/html/rlm.html ). I don't know if that implementation is publicly available. It is applied in an example in https://stats.oarc.ucla.edu/r/dae/robust-regression/

fog37
Even least squares is not necessary. You can find a slope and intercept that minimize any penalty function you want.

fog37, scottdave and FactChecker
Office_Shredder said:
Even least squares is not necessary. You can find a slope and intercept that minimize any penalty function you want.
Good point, although most penalty functions would require non-analytical iterative minimization algorithms that are less intuitive. Also, I do not know what the risk of introducing local minimums would be.

fog37
fog37 said:
In general, the optimal intercept and slope are found using OLS.

(A mathematical definition can be given in the context of statistical estimation and the properties of estimators.)

FactChecker
fog37 said:
TL;DR Summary: Understanding if linear regression can be done with other variants of least squares

Hello,

Simple linear regression aims at finding the slope and intercept of the best-fit line to for a pair of ##X## and ##Y## variables.
In general, the optimal intercept and slope are found using OLS. However, I learned that "O" means ordinary and there are other types of least square computations...

Question: is it possible to apply those variants of LS to the linear regression model, i.e., can we find the best-fit line parameters using something other than OLS (for example, I think there is "robust" LS, etc.)?

thank you!
The phrase "best fit line" is meaningless without some context.

In least squares regression the best fit line is the one that minimizes the sum of the squared residuals.

In robust regression based on the ideas that grew from Huber's M-estimate based ideas the residuals are not squared but are fed into some other function [designed to lessen the impact of certain kinds of outliers] and the best fit line is the one that minimizes the sum of the values of those function values.

In robust regression [at least the simplest methods, based on the initial work of Jana Jurečková, later expanded on by Joseph McKean and others] based on R-estimate [rank estimates] we view the squared residuals as (residual)(residual), replace one of the two factors by a weighted rank of the residual, and the best fit line is the one that minimizes the sum of those.
You should also look up the idea of Tukey's resistant line.

There are others, but the point is that phrases like "optimal" and "line of best fit" need to be placed into some context: optimal or best in what sense?

scottdave and FactChecker

1. What is Linear Regression?

Linear regression is a statistical method used to model the relationship between a dependent variable and one or more independent variables. It assumes a linear relationship between the variables and uses the least squares method to find the line of best fit.

2. What is OLS?

OLS (Ordinary Least Squares) is the most common method used to estimate the parameters of a linear regression model. It minimizes the sum of the squared differences between the observed and predicted values to find the best fit line.

3. How is the best fit line determined in Linear Regression?

The best fit line is determined by minimizing the sum of the squared errors between the observed and predicted values. This is done using the OLS method, which calculates the slope and intercept of the line that minimizes the sum of the squared errors.

4. What is the difference between Simple Linear Regression and Multiple Linear Regression?

Simple Linear Regression involves one dependent variable and one independent variable, whereas Multiple Linear Regression involves one dependent variable and two or more independent variables. Multiple Linear Regression allows for the analysis of the relationship between multiple independent variables and the dependent variable.

5. What are the assumptions of Linear Regression?

The assumptions of Linear Regression include linearity, normality, independence of errors, constant variance, and absence of multicollinearity. These assumptions must be met for the results of the regression analysis to be valid.

• Set Theory, Logic, Probability, Statistics
Replies
23
Views
2K
• Set Theory, Logic, Probability, Statistics
Replies
3
Views
781
• Set Theory, Logic, Probability, Statistics
Replies
7
Views
436
• Set Theory, Logic, Probability, Statistics
Replies
30
Views
2K
• Set Theory, Logic, Probability, Statistics
Replies
8
Views
1K
• Set Theory, Logic, Probability, Statistics
Replies
5
Views
965
• Set Theory, Logic, Probability, Statistics
Replies
7
Views
416
• Set Theory, Logic, Probability, Statistics
Replies
4
Views
1K
• Set Theory, Logic, Probability, Statistics
Replies
13
Views
2K
• Set Theory, Logic, Probability, Statistics
Replies
4
Views
1K