Using SVD to determine the redundancy of a fit.

[WarPhalange]

I have a project I am doing for a professor and unfortunately I cannot get ahold of him to help me out, so I figured I'd ask you guys. Of course, I tried to ask The Google about this first and didn't get anywhere. Here is what I am trying to do:

Linear least squares fitting. Choose some odd-ball function, say g(x). Create a set of "data" by choosing xi and some (sigma)i, generating yi normally distributed about g(xi). Choose a set of functions that might plausibly fit the data as y=Sum[aj fj(x)]. Perform a least squares fit by solving the normal equations in matrix form. You should determine the condition number and use SVD to determine whether there is any redundancy in your choice of fj(x) and fix the fit. Finally should evaluate chi squared to see whether the fit is adequate.

The part I am having trouble with is using SVD to determine redundancy. My f(x) is an n'th order polynomial (I get to decide what 'n' is). I successfully found a fit to my generated data, but don't know where to go from there.

What I found online was to take the matrix A from Ax=b, do SVD on that, and go from there in order to solve the system of equations. I already have a solution though, so I don't know what to do.

One idea I had was to make a new matrix like so:

| a₀ a₁x₁ a₂x₁² a₃x₁³ |
| a₀ a₁x₂ a₂x₂² a₃x₂³ |
| a₀ a₁x₃ a₂x₃² a₃x₃³ |

et cetera, with actual numbers instead of 'a' and 'x' of course, take the SVD of that, trim down the three matrices to only include actual eigenvalues (so drop any '0' elements in the diagonal matrix), transform back, and then divide by the various 'x's to get different values for the 'a's, but I'm not sure if that would do anything at all.

Thanks in advance for the help.

 

[mighty2000]

Thank you for reaching out for help with your project. It sounds like you have already made some good progress in fitting your data and are now trying to use SVD to determine redundancy in your choice of functions. SVD can definitely be a useful tool for this task, and I can offer some guidance on how to use it in your case.

First, let's review the concept of redundancy in this context. In a least squares fit, we want to find the best set of coefficients (in your case, the 'a' values) that minimize the difference between our model (the sum of functions fj(x)) and the actual data. However, if we have chosen a set of functions that are linearly dependent (meaning one function can be expressed as a linear combination of the others), then some of the coefficients will be redundant - they will not contribute to the fit since they can be replaced by other coefficients without changing the overall model. This can lead to issues with the fit, such as overfitting or instability.

Now, onto using SVD to determine redundancy. You are correct in thinking that you can take the matrix A from Ax=b and perform SVD on it. This will give you three matrices: U, Σ, and V. The Σ matrix contains the singular values, which are essentially the scaling factors for the columns of U and rows of V. The larger the singular value, the more important that column or row is in the matrix.

To determine redundancy, we can look at the singular values in Σ. If any of them are very small (close to 0), then those columns of U and rows of V are not contributing much to the overall matrix and can be considered redundant. In your case, this would mean that some of your chosen functions are not necessary for the fit.

To address this redundancy, you can follow your idea of creating a new matrix with 'x' values and then performing SVD on that. This will essentially reduce the dimensionality of your problem and allow you to eliminate the redundant functions.

I hope this helps guide you in using SVD to determine redundancy in your least squares fit. If you have any further questions or need clarification, please don't hesitate to ask. Good luck with your project!