Other linear fitting than least squares

  • #1
Felipe Lincoln
Gold Member
99
11

Main Question or Discussion Point

I'm analysing some data and my task is to get a line that best fits the data, using least square I'm getting these dashed curves (red and blue) with low correlation factors. Is there another method that takes into consideration the amount of data placed into the direction of a line?
graph.png
 

Attachments

Answers and Replies

  • #2
phyzguy
Science Advisor
4,394
1,369
It seems like you must be doing something wrong with your least-squares fit. Can you calculate the least-squares value for the red, blue, and black lines? It looks lower for the black line.
 
  • #3
29,058
5,325
Those red and blue lines don’t look right. I think there must be something wrong in your code
 
  • #4
Felipe Lincoln
Gold Member
99
11
the black line is just the identity y=x. The red and blue I got through my data.
I used the scipy.stats.linregress, can't see what's wrong but I'll take a look again
 
  • #5
Felipe Lincoln
Gold Member
99
11
Oh there was some data hiding beyond my axis limits. Sorry for this mistake, I'll fix it and post the result.
graph2.png
 

Attachments

  • #6
29,058
5,325
So if you do a leverage plot that one datapoint will probably have a huge leverage. I would check that point and see if there is some error. Like maybe a typo when copying the data.
 
  • #7
FactChecker
Science Advisor
Gold Member
5,384
1,953
Aha! So the standard least squared regression is doing a good job on the entire data set. But you should consider that the data visible in your first post looks like it is following a different rule than the entire set. If you can see the reason for that, you may want to analyse the data in sections that make more sense.
 
  • #8
Stephen Tashi
Science Advisor
7,017
1,237
Why do you think fitting a straight line to that data (however it is done) would be a good idea?
 
  • #9
Felipe Lincoln
Gold Member
99
11
Why do you think fitting a straight line to that data (however it is done) would be a good idea?
It is an experimental data that is expected to have a linear correspondence.
I just removed the zeroes data and this is what I got now. Thank you all
graph.png
 

Attachments

  • #10
29,058
5,325
I just removed the zeroes data and this is what I got now.
Do you have any experimental justification for that?
 
  • #11
Felipe Lincoln
Gold Member
99
11
Do you have any experimental justification for that?
Yes sir. The zeroes was generate by my code to represent experiments that failed and resulted in no data.
 
  • #12
29,058
5,325
Yes sir. The zeroes was generate by my code to represent experiments that failed and resulted in no data.
That is an excellent reason!

It is never a good idea to throw away data just because it makes your fit better, but if a data point is bad for some specific reason then throwing it out is acceptable.
 
  • #13
29,058
5,325
FYI, you may want to also look into that data point with the high Rphenix. It looks like it has a very high leverage and it may have some other problem.
 
  • #14
Felipe Lincoln
Gold Member
99
11
Right!! The next step in our research is to analyse what was the experiment conditions that bring some points a bit far from the expected. Thanks for your attention Dale!
 

Related Threads on Other linear fitting than least squares

Replies
6
Views
14K
Replies
3
Views
3K
  • Last Post
Replies
6
Views
5K
  • Last Post
Replies
3
Views
2K
  • Last Post
Replies
8
Views
3K
  • Last Post
Replies
1
Views
2K
  • Last Post
Replies
1
Views
2K
  • Last Post
Replies
5
Views
635
Replies
0
Views
3K
Replies
3
Views
5K
Top