Least squares problem - Leon Ch 5, sec.3, prob.3

In summary, the book suggests finding a line in R^2 that minimizes the sum of squares of the errors, but this is not possible because A is not linearly independent. It also suggests finding a solution to ATAx = ATb using Gaussian elimination, but this problem is inconsistent and has a unique solution. Using calculus and set notation, one can solve this problem by reducing it with Row Echelon Form.
  • #1
IntroAnalysis
64
0

Homework Statement


For the system Ax = b, find least squares solution.

A = (1 2) ; b = (3, 2, 1)T
----(2 4)
----(1 -2)



Homework Equations


I know if A is an m x n matrix of rank n, the normal equations

ATAx = ATb

have a unique solution x = (ATA)-1ATb

However, the matrix A is not linearly independent, and is only rank = 1.

Using Gaussian elimination, I think this system of equations is inconsistent. But book gives
answer {(1 - 2α, α)T; α real}

I don't see how they get this. Aren't these the equations of 3 parallel lines?
 
Physics news on Phys.org
  • #2
isn't A of rank 2?

i would interpret {(1 - 2α, α)T; α real} as a line in R^2
 
  • #3
Something is wrong. The problem as stated has a unique solution. The book implies an infinite number of solutions, which isn't true based on the problem statement. Maybe there is a typo in A?
 
  • #4
shouldn't the least squares solution to a 2 variable case be a "line of best fit" relating the variables?
 
  • #5
Yes, I believe you are correct. I've attached my work. Also, (ATA) is singular, so
there isn't an inverse of (ATA). Of course, the only examples I see in this section of the book are invertible.

Again, anyone know how they come up with their solution {(1 - 2α, α)T such that α real}?
 

Attachments

  • 5.3_Question3.jpg
    5.3_Question3.jpg
    31.3 KB · Views: 439
  • #6
No, A isn't a rank of 3. The vectors (2, 4) and (-1, -2) are linear combinations of (1, 2). Hence, rank of A is the one vector (1,2). The nonzero rows of the row echelon matrix form a basis for the row space.
 
  • #7
you've written (1,-2) in the original problem
 
  • #8
Darn it, you are correct, I obviously was having problems using Latex for set notation.
Set A should be row 1: 1 2
row 2: 2 4
row 3: -1 -2

and b = (3, 2, 1)T

Thanks for noticing the typo.
 
  • #9
Your three equations are

x+2y = 3
2x+4y = 2
-x-2y=1

These equations are inconsistent so they can't all be satisfied. You are looking for (x,y) that minimizes the sum of squares of the errors:

w = (x+2y-3)2+(2x+4y-2)2+(-x-2y-1)2

I would work it as a two variable calculus minimization problem.
 
  • #10
IntroAnalysis said:
Darn it, you are correct, I obviously was having problems using Latex for set notation.
Set A should be row 1: 1 2
row 2: 2 4
row 3: -1 -2

and b = (3, 2, 1)T

Thanks for noticing the typo.

You also have a typo in your hand written analysis that you posted. Look at your calculation of ATA. The last row of A has a typo. To answer your question how the book got their answer, you need to look at solving ATAx = ATb. It is singular as you state. This means there is either no solution or an infinite number of them. In this particular case you end up with two identical equations which is the same as one equation with two unknowns. Let one of the unknowns take on an arbitrary value (the book let x2 = a) and you'll get your answer.
 
Last edited:
  • #11
Correct, (ATA) should come out to
(6 12)
(12 24)

The determinant = 144 - 144 = 0, so still singular.

If I take the person's advice and take partial derivatives of w w.r.t x and y:

w = (x - 2y - 3)2 + (2x + 4Y - 2)2 + (-x - 2y - 1)2

I get dw/dx = 12x + 24y - 12 = 12(x + 2y -1) = 0
dw/dy = 24x + 48y - 24 = 24(x + 2y -1) = 0

letting y = α, then x = 1 - 2α and the solution {(1 - 2α), α; for all α element of Real}

This looks like the long way to get at what N(AT) looks like.
 
  • #12
I saw my professor about this problem yesterday. While the method suggested here works, it is not the best way to solve this problem.

See attached. But we know that : (ATA)x = ATb is always consistent.

So we can solve reducing it with Row Echelon Form. This is the most efficient way to solve this problem.

I was relying on following the examples from this section too much. My professor had said (when showing how cumbersome finding an inverse matrix can be), "No one solves them like this." I thought he meant they use computers or calculators. That is where I got confused.
 

Attachments

  • 5.3_Question3Solved.jpg
    5.3_Question3Solved.jpg
    28.3 KB · Views: 424

What is the least squares problem?

The least squares problem is a mathematical method used to find the best possible fit for a set of data points. It involves finding the line (or curve) that minimizes the sum of the squared distances between the line and each data point.

What is the purpose of the least squares problem?

The purpose of the least squares problem is to find a line (or curve) that best represents the relationship between two variables in a dataset. This line can then be used to make predictions or draw conclusions about the data.

How is the least squares problem solved?

The least squares problem is typically solved using a method called the normal equations. This involves taking the derivative of a cost function and setting it equal to zero, then solving for the coefficients of the line or curve.

What are the assumptions of the least squares problem?

The least squares problem assumes that the relationship between the variables is linear and that the residual errors are normally distributed with a mean of zero. It also assumes that the data points are independent of each other.

What are the applications of the least squares problem?

The least squares problem has many applications in various fields such as statistics, physics, finance, and engineering. It is commonly used for data analysis, curve fitting, and predictive modeling.

Similar threads

  • Calculus and Beyond Homework Help
Replies
1
Views
1K
  • Calculus and Beyond Homework Help
Replies
3
Views
1K
  • Calculus and Beyond Homework Help
Replies
3
Views
1K
  • Calculus and Beyond Homework Help
Replies
3
Views
2K
  • Calculus and Beyond Homework Help
Replies
4
Views
4K
  • Calculus and Beyond Homework Help
Replies
23
Views
1K
  • Calculus and Beyond Homework Help
Replies
11
Views
4K
  • Calculus and Beyond Homework Help
Replies
3
Views
2K
  • Calculus and Beyond Homework Help
Replies
22
Views
2K
  • Calculus and Beyond Homework Help
Replies
5
Views
2K
Back
Top