Optimal Approach for Analyzing Non-Smooth Experimental Data

[t.m.p.c]

Hi,

I need your help. From experiments I got data set which I need to analyze. The problem is that my data is not smooth. I tried to fit my data using a polynomial equation, but the fitting was not good enough. I also tried to smooth, spline... but got very different final results. Can anyone tell me which will be the best way (the more credible) to analyze my data?
Thanks in advance.

Regards,

tmpc

 

[Hurkyl]

Question #1: Does theoretically analyzing the data source suggest how the results should be distributed?

Question #2: How are you judging whether or not the fit is good enough?

Question #3: Do you have all of the variables accounted for?

Question #4: Have you tried anything exponential or logarithmic?


Some bad news: at this point, since you've been trying lots of things, it might not be possible to credibly analyze your data^*. Finding a fit after a dozen false starts is far less significant than finding a fit on the first try. It might be that the best you can do is, once you find something that fits your current data, and then do a new experiment to (in)validate how well it works.

Maybe you can do some tricks to salvage this dataset, like using 75% of the data to find a fit and 25% to (in)validate it... but I'm not the person who can judge such things.



*: More accurately, it might not be possible to draw any credible conclusions. The analysis "we tried X, Y, and Z to analyze the data, without success" is definitely credible and accurate. It is important to know things like "these variables aren't linearly related", although it's not a flashy result.

 

[sir-pinski]

Hi tmpc,

Thank you for reaching out for help with your experimental data analysis. It can be challenging to analyze data that is not smooth and it is important to use the most credible method to ensure accurate results.

One option you could consider is using a non-parametric smoothing method, such as a moving average or a lowess smoother. These methods do not assume a specific mathematical form for your data and can be effective in handling non-smooth data.

Another approach is to use robust regression techniques, which are designed to handle outliers and non-smooth data. These methods can help improve the accuracy of your results compared to traditional methods like polynomial fitting.

Ultimately, the best method for analyzing your data will depend on the specific characteristics of your data and the goals of your analysis. I would recommend consulting with a statistician or data analyst who can help you select the most appropriate method for your specific data set.

I hope this helps and good luck with your data analysis!