What is Least squares: Definition and 166 Discussions

The method of least squares is a standard approach in regression analysis to approximate the solution of overdetermined systems (sets of equations in which there are more equations than unknowns) by minimizing the sum of the squares of the residuals made in the results of every single equation.
The most important application is in data fitting. The best fit in the least-squares sense minimizes the sum of squared residuals (a residual being: the difference between an observed value, and the fitted value provided by a model). When the problem has substantial uncertainties in the independent variable (the x variable), then simple regression and least-squares methods have problems; in such cases, the methodology required for fitting errors-in-variables models may be considered instead of that for least squares.
Least-squares problems fall into two categories: linear or ordinary least squares and nonlinear least squares, depending on whether or not the residuals are linear in all unknowns. The linear least-squares problem occurs in statistical regression analysis; it has a closed-form solution. The nonlinear problem is usually solved by iterative refinement; at each iteration the system is approximated by a linear one, and thus the core calculation is similar in both cases.
Polynomial least squares describes the variance in a prediction of the dependent variable as a function of the independent variable and the deviations from the fitted curve.
When the observations come from an exponential family and mild conditions are satisfied, least-squares estimates and maximum-likelihood estimates are identical. The method of least squares can also be derived as a method of moments estimator.
The following discussion is mostly presented in terms of linear functions but the use of least squares is valid and practical for more general families of functions. Also, by iteratively applying local quadratic approximation to the likelihood (through the Fisher information), the least-squares method may be used to fit a generalized linear model.
The least-squares method was officially discovered and published by Adrien-Marie Legendre (1805), though it is usually also co-credited to Carl Friedrich Gauss (1795) who contributed significant theoretical advances to the method and may have previously used it in his work.

View More On Wikipedia.org
  1. P

    Minimize integral using orthogonal basis

    I'm posting to inquire about a possible typo in the given answer in the back of the book, or if maybe I did something wrong, because my answer does not agree with the one stated. So the exercise is about finding the least squares approximation. The norm is the ##L^2## norm and the corresponding...
  2. J

    Python Why backpropagation dominates neural networks instead of interpolation

    Hi guys, I was learning machine learning and I found something a bit confusing. When I studied physics I saw the method of least squares to find the best parameters for the given data, in this case we assume we know the equation and we just minimize the error. So if it is a straight line model...
  3. F

    I Exploring Nonlinear Least Squares for Regression Analysis

    Hello, Regression analysis is about finding/estimating the coefficients for a particular function ##f## that would best fit the data. The function ##f## could be a straight line, an exponential, a power law, etc. The goal remains the same: finding the coefficients. If the data does not show a...
  4. C

    A Convergence issue in this Least Squares calculation

    I'm computing the trajectory of a moving body and my net is composed by 5 stations. My observations are DTOA: difference in time of Arrival (they have been linearized). I am trying to use Least Squares with a linear model: Y = Ax + b, where Y are the observed measurements (DTOA), A the design...
  5. M

    MHB Least squares regression line (I'm very lost)

    Hi! Basically this is the exercise: Given the covariance of x and y is -12 and the variance of x is 6,5, using the least squares line of best fit connecting x and y yo estimate the value of x when y=15 x 2 5 9 7 9 10 7 y 25 17 11 10 8 7 13 any help would mean everything, I'm desperate :(
  6. S

    Applying least squares to measurement of nuclei masses and Q-value

    Consider the problem in the attached image. The difference between A and B is 0.0020(20). How does one use the least squares method, particularly in matrix form, to find the best value of the masses of A and B respectively, as well as the Q-value? Aren't more measurements needed for the masses...
  7. V

    Using Least Squares to find Orthogonal Projection

    I'm a little confused how to do this homework problem, I can't seem to obtain the correct answer. I took my vectors v1, v2, and v3 and set up a matrix. So I made my matrix: V = [ (6,0,0,1)T, (0,1,-1,0)T, (1,1,0,-6)T ] and then I had u = [ (0,5,4,0) T ]. I then went to solve using least...
  8. M

    Finding least squares solution of Ax=b?

    Does anyone know the command or how to find the least squares solution of Ax=b on Ti-89 graphing calculator? I'm trying to check my answers on Ti-89 for those linear algebra problems.
  9. Vital

    I Least squares line - understanding formulas

    Hello. I have listened to a great lecture, which gave helpful intuitive insight into correlation and regression (basic stuff). But there are formulas, which I cannot grasp intuitively and don't know their origin. To remember them I would like to understand what's happening in each part of the...
  10. synMehdi

    I Linear least squares regression for model matrix identification

    Summary: I need to Identify my linear model matrix using least squares . The aim is to approach an overdetermined system Matrix [A] by knowing pairs of [x] and [y] input data in the complex space. I need to do a linear model identification using least squared method. My model to identify is a...
  11. W

    I Newton-Raphson in Least Squares: How is it used? Cost Function?

    I just went over analysis of a data set that was analzed using Linear Regression (OLS, I believe) and I saw Newton's method was used. Just curious, how is it used? I assume to minimize the cost function, but this function was not made explicit. Anyone know? Thanks.
  12. M

    MHB Least squares method : approximation of a cubic polynomial

    Hey! :o I want to determine an approximation of a cubic polynomial that has at the points $$x_0=-2, \ x_1=-1, \ x_2=0 , \ x_3=3, \ x_4=3.5$$ the values $$y_0=-33, \ y_1=-20, \ y_2=-20.1, \ y_3=-4.3 , \ y_4=32.5$$ using the least squares method. So we are looking for a cubic polynomial $p(x)$...
  13. Felipe Lincoln

    I Other linear fitting than least squares

    I'm analysing some data and my task is to get a line that best fits the data, using least square I'm getting these dashed curves (red and blue) with low correlation factors. Is there another method that takes into consideration the amount of data placed into the direction of a line?
  14. T

    A Confused about Weighted Least Squares

    I am trying to use Weighted Least Squares with a linear model: Y = Xβ + ε, where Y are some observed measurements and β is a vector of estimates. For example, in this case β has two terms: intercept and slope. The weighted least squares solution, as shown here, involves a weight matrix, W...
  15. W

    Capacitor - Least squares fitting

    Homework Statement We had a laboration for calculating ε_r in a parallel plate capacitor which we stuffed with plastic plates. All data we picked up was the area A, the distance d (and thus 1/d) and the capacitance C. We are now supposed to use the least squares-method to find ε_r, something we...
  16. G

    I Shaky model in least squares fit

    I've come across a problem with my least squares fits and I think someone else must have analyzed this, but I don't know where to find it. I have a converged least squares fit of my spectroscopic data. Unfortunately, the physical model, on which the fit is based, is mediocre. The deviations...
  17. M

    I Matrix Mechanics and non-linear least squares analogy?

    I have some experience with non-linear least squares curve fitting. For instance, if I want to fit a Gaussian curve to a set of data, I would use a non-linear least squares technique. A "model" matrix is implemented and combined with the observed data. The solution is found by applying well...
  18. N

    I Trying to understand least squares estimates

    Hi, I'm trying to understand which mathematical actions I need to perform to be able to arrive at the solution shown in the uploaded picture. Thank you.
  19. M

    I Linear least-squares method and row multiplication of matrix

    Suppose that I have an overdetermined equation system in matrix form: Ax = b Where x and b are column vectors, and A has the same number of rows as b, and x has less rows than both. The least-squares method could be used here to obtain the best possible approximative solution. Let's call this...
  20. Jeffack

    I Hessian of least squares estimate behaving strangely

    I am doing a nonlinear least squares estimation on a function of 14 variables (meaning that, to estimate ##y=f(x)##, I minimize ##\Sigma_i(y_i-(\hat x_i))^2## ). I do this using the quasi-Newton algorithm in MATLAB. This also gives the Hessian (matrix of second derivatives) at the minimizing...
  21. D

    Least squares approximation of a function?

    Homework Statement Find the least squares approximation of cos^3(x) by a combination of sin(x) and cos(x) over the interval (0, 2pi) Homework EquationsThe Attempt at a Solution I know how to find a least squares approximation with vectors, but I don't even know how to start with a function...
  22. beyondlight

    Solve derivative of least squares matrix equation

    Homework Statement I am designing a MIMO communication system, with input signal s, channel H and transform matrix T. The received signal is corrupted by noise. Homework Equations [/B] The received signal is r = Hs+n And then it is transformed (compressed) by: y = Tr And then its...
  23. G

    Least squares problem: am I solving it correctly?

    Homework Statement In R^3 with inner product calculate all the least square solutions, and choose the one with shorter length, of the system: x + y + z = 1 x + z = 0 y = 0 2. The attempt at a solution So I applied the formula A^T A x = A^T b with A as being the matrix with row 1 (1,1,1)...
  24. benzun_1999

    Over constrained least squares analysis

    Hi, I have a over constrained least squares problem. I also have the correct solution to the problem. Now I need to determine which of the vectors are contributing how much information that is close to the correct solution. I am sure there should be some methodology for this analysis, but I...
  25. V

    Linearizing a nonlinear least squares model

    I have a nonlinear least squares problem with a set of parameters \bf{g}, where I need to minimize the function: \chi^2 = \sum_i \left( y_i - M(t_i ; {\bf g}) \right)^2 The t_i are some independent parameters associated with the observations y_i and the model function has the form M(t_i ...
  26. RicardoMP

    Exponential Least Squares Method

    Homework Statement Hi! I've been interpolating a data set using Legendre polynomials and Newton's method and now I've been asked to, given a data set, approximate the function using the Least Squares Method with the following fitting function: ##g(r)=a+be^{cr}##, where ##a##, ##b## and ##c##...
  27. B

    Least squares estimation

    Homework Statement Suppose that object 1 weighs \theta_1 and object two weights \theta_2. Then each object is weighed once and then together getting three observations y_1,y_2,y_3. The scale gives unbiased weights with normally distributed error (constant variance) Find the least square...
  28. S

    Least squares regression

    Homework Statement note a linear regression model with the response variable Y=(Y1..Yn) on a predictor variable X=(X1..Xn). the least squares estimates of the intercept and slope a(hat) and B(hat) are the values that minimize the function: (see attached image) and the problem reads on further...
  29. Samuel Williams

    Least Squares Approximation

    My apologies for having to post in an image, my latex skills are not good enough for the question at hand :( a) There is no solution since the system has more unknowns than equations (the equations are equal giving 1=2 which does not make sense). b) I get a solution of \begin{bmatrix}1 \\1 \\...
  30. J

    The linear in linear least squares regression

    It is my understanding that you can use linear least squares to fit a plethora of different functions (quadratic, cubic, quartic etc). The requirement of linearity applies to the coefficients (i.e B in (y-Bx)^2). It seems to me that I can find a solution such that a coefficient b_i^2=c_i, in...
  31. ORF

    Error in least squares fit: how to include error of points?

    Hello I have a doubt with the least squares fitting (linear fitting). The low-level statistics textbooks only take into account the statistical error of fitting, but not the error of the fitted points. How is the error of the fitted points taken into account, and included in the total error...
  32. V

    Least squares parameter correlation

    I am trying to solve a large least squares inversion (inverting data for the modeled sources), and find that my parameters describing 1 source are highly correlated with the parameters describing the second source. Can anyone recommend a technique or reference which discusses how to reduce the...
  33. V

    Large weighted least squares system

    I have a large weighted least squares system with millions of observations. I can't store the least squares matrix in memory, so I only store the normal equations matrix: A^T W A where A is dense and n-by-p and W = diag(w_1,...,w_n). It takes a very long time to build this normal equations...
  34. diracdelta

    Least squares method for cloud of atom

    Homework Statement After turnig of magnetic-optic pit, cold cloud of atom 87 Rb is expanding. Size of cloud after time t, is given with relation: where, k_B is Boltzman constant, m mass of 87 Rb. Draw a plot, then use least squares method to find temperature T, and initial size of cloud...
  35. END

    Least Squares Derivation—Simple Algebraic Simplification

    Hey, PF I'm reading the following derivation of least squares, and I'm trying to figure out how the author went from the last step at the bottom of pg. 7 to the final equation (11) at the top of pg. 8. [http://isites.harvard.edu/fs/docs/icb.topic515975.files/OLSDerivation.pdf] More...
  36. samgrace

    Weighted Least Squares Solution

    Homework Statement \begin{bmatrix} 3x_{1}& 7x_{2}& 4x_{3} \\ 3x_{1}& 4x_{2}& 5x_{3} \\ x_{1}& 10x_{2}& 8x_{3} \\ 8x_{1}& 8x_{2}& 6x_{3} \\ \end{bmatrix} = \begin{bmatrix} 26 \\ 16 \\ 33 \\ 46 \\ \end{bmatrix} the measurements represented by equations 1 and 3 above can be trusted more than those...
  37. F

    Need linear regression help

    I also made a graph which is not pictured. 1.) Calculate the least squares line. Put the equation in the form of: y-hat = a + bx. I got: y hat = 11.304 + 106.218x a.) Find correlation coefficient. Is it significant? (use the p-value to decide) I got: r = 0.913... no it...
  38. D

    Least Squares Estimation for two parameters

    Homework Statement Hi guys, so the problem is as follows: A set of n independent measurements y_{i}, i=1...n are treated as Gaussian, each with standard deviations \sigma_{i}. Each measurement corresponds to a value of a control variable x_{i}. The expectation value of y is given by...
  39. A

    MATLAB [Matlab] Which is the good solution My vs. School - curve fitting?

    Hy, I wonder which is the good solution for this problem: Nonlinear least square problem: function: y = x / (a + b.x) linearization: 1/y = a/x + b substitution: v = 1/y , u = 1/x >> model v = b + a.u What we did in school: x = [1 2 4 7]; y = [2 1 0.4 0.1]; v=1./y; u=1./x; n = length(x)...
  40. Schnurmann

    Least Squares Estimation - Problem with Symbols

    Hi folks, 1. Homework Statement I don't fully understand the question statement, how is it supposed to be read? Question: Give a formula for the minimizer x* (to be read as x-star) of the function ƒ:ℝn → ℝ, x → ƒ(x) = ||Ax-b||22, where A∈ℝm×n and b∈ℝm are given. You can assume that A has rank...
  41. A

    How to do Linearization for Non-linear least squares?

    Hy I want to know how to make linearization for some function,...what should by in Non-linear least squares problems. In my book I have only this example how to do: http://i.imgur.com/MUFiHkr.pngSomeone could me help how to do, some receipt of method what I need to do? Non-linear least...
  42. gfd43tg

    Least squares regression outputting function handle

    Homework Statement The number of twists ##y## required to break a certain rod is a function of the percentages ##u## and ##v## of each of two chemical components present in the rod. The following function is proposed ##y(u, v) = a_{1} + a_{2} exp(u^{2}) + a_{3}\sqrt{v} + a_{4}uv## (1) where the...
  43. D

    Weighted Least Squares for coefficients

    Hi, I have an ordinary least squares setup y = Ac where A is an NxM (N>>M) matrix, c the unknown coefficients and y the measurements. Now WEIGHTED least squares allows to weight the MEASUREMENTS if, for example, some measurements are more important or contain a lower variance. However...
  44. gfd43tg

    Linear regression with least squares for quadratic function

    Homework Statement We want to determine the coefficients of a polynomial of the form: ##p(x)=c_{1}x^2 +c_{2}x+c_{3}##The polynomial ##p(x)## must satisfy the constraint ##p(1)=1##. We would also like ##p(x)## to satisfy the following 4 constraints: ##p(−1)=5## ##p(0)=−1## ##p(2)=6##...
  45. C

    Least squares of a constant

    Suppose we have an observation y = c+ e where c is an unknown constant and e is error with the pdf = exp(-e-1) for e >-1 . We want to determine the least square estimator of c given by the c* which minimizes the error cost function E(c) = .5(y-c)^2 Minimizing the error cost is done by taking...
  46. V

    Weighted least squares residuals

    Hello, When doing a weighted least squares fit of a model to data, I want to examine the residuals to see if their histogram matches the expected probability distribution. Since I am minimizing \chi^2 = \sum_i w_i \left[ y_i - Y(x_i) \right]^2 would I define my...
  47. V

    Uncertanty in a non-linear regression with least squares method

    Homework Statement Ok, so I'm trying to fit a set of data (21000 points to be exact) to a sine function. Homework Equations Y = A*sin(ωt) The Attempt at a Solution I used NumPy to get the parameters A and ω with the least squares method. So far, so good. However, i appear to...
  48. S

    Least squares assumptions: finite and nonzero 4th moments

    This isn't a homework problem - I'm just confused by something in a textbook that I'm reading (not for a class, either). I'd appreciate an intuitive clarification, or a link to a good explanation (can't seem to find anything useful on Google or in my textbook). My book states that one of the...
  49. N

    Comp Sci Fortran77 woes- single parameter least squares minimisation stuck

    Hey guys! Having issues with a code I'm writing. It doesn't fit the typical format and I may have already accidentally posted it in the wrong forum, so here's a copy of what was there with the newest version of the code. It'll (eventually) be a four-parameter least squares minimisation, but for...
  50. N

    Fortran Fortran77 woes- single parameter least squares minimisation stuck

    Hey guys! Having issues with a code I'm writing. It'll (eventually) be a four-parameter least squares minimisation, but for the moment I'm just trying to get the single parameter version to work. I've got the first module done but, when I run it it gets caught up in a loop and doesn't actually...