Undergrad Other linear fitting than least squares

[Felipe Lincoln]

I'm analysing some data and my task is to get a line that best fits the data, using least square I'm getting these dashed curves (red and blue) with low correlation factors. Is there another method that takes into consideration the amount of data placed into the direction of a line?

 

[phyzguy]

It seems like you must be doing something wrong with your least-squares fit. Can you calculate the least-squares value for the red, blue, and black lines? It looks lower for the black line.

 

[Dale]

Those red and blue lines don’t look right. I think there must be something wrong in your code

 

[Felipe Lincoln]

the black line is just the identity y=x. The red and blue I got through my data.
I used the scipy.stats.linregress, can't see what's wrong but I'll take a look again

 

[Felipe Lincoln]

Oh there was some data hiding beyond my axis limits. Sorry for this mistake, I'll fix it and post the result.

 

[Dale]

So if you do a leverage plot that one datapoint will probably have a huge leverage. I would check that point and see if there is some error. Like maybe a typo when copying the data.

 

[FactChecker]

Aha! So the standard least squared regression is doing a good job on the entire data set. But you should consider that the data visible in your first post looks like it is following a different rule than the entire set. If you can see the reason for that, you may want to analyse the data in sections that make more sense.

 

[Stephen Tashi]

Why do you think fitting a straight line to that data (however it is done) would be a good idea?

 

[Felipe Lincoln]

Why do you think fitting a straight line to that data (however it is done) would be a good idea?

It is an experimental data that is expected to have a linear correspondence.
I just removed the zeroes data and this is what I got now. Thank you all

 

[Dale]

I just removed the zeroes data and this is what I got now.

Do you have any experimental justification for that?

 

[Felipe Lincoln]

Do you have any experimental justification for that?

Yes sir. The zeroes was generate by my code to represent experiments that failed and resulted in no data.

 

[Dale]

Yes sir. The zeroes was generate by my code to represent experiments that failed and resulted in no data.

That is an excellent reason!

It is never a good idea to throw away data just because it makes your fit better, but if a data point is bad for some specific reason then throwing it out is acceptable.

 

[Dale]

FYI, you may want to also look into that data point with the high Rphenix. It looks like it has a very high leverage and it may have some other problem.

 

[Felipe Lincoln]

Right! The next step in our research is to analyse what was the experiment conditions that bring some points a bit far from the expected. Thanks for your attention Dale!