Trend Estimation: Analyzing Data to Determine Positive/Negative Trend

  • Context: Undergrad 
  • Thread starter Thread starter Diffy
  • Start date Start date
  • Tags Tags
    Estimation
Click For Summary

Discussion Overview

The discussion revolves around determining whether a dataset exhibits a positive or negative trend using statistical methods, specifically through linear regression analysis. Participants explore various approaches to test the significance of the trend, including hypothesis testing and permutation tests, while addressing the challenges faced by the original poster in applying these methods.

Discussion Character

  • Technical explanation
  • Mathematical reasoning
  • Debate/contested

Main Points Raised

  • One participant suggests using linear regression and testing the null hypothesis that the slope is zero to determine if a trend exists.
  • Another participant notes that trends can change direction, especially in time series data, and recommends fitting different models to assess specific types of trends.
  • A participant expresses difficulty in understanding how to perform the trend test and requests more detailed examples or resources.
  • One reply provides a link to a SAS example of linear regression, emphasizing the importance of testing the slope.
  • Another participant proposes using a permutation test to create a distribution of trends but questions if this can be done without generating permutations.
  • One suggestion includes using Excel for regression analysis, which automatically performs t-tests on slopes, providing a practical example with data on CPI by year.
  • The original poster later acknowledges that they found the solution by performing a t-test on the slopes manually, indicating they were able to resolve their query with the help of the discussion.

Areas of Agreement / Disagreement

Participants generally agree on the importance of testing the slope in linear regression to determine trends, but there are differing opinions on the methods to achieve this and the applicability of certain tools like SAS and Excel. The discussion remains somewhat unresolved regarding the best approach for the original poster's specific needs.

Contextual Notes

Some participants mention the limitations of specific software tools and the need for manual calculations, indicating that the discussion may depend on the participants' familiarity with statistical software and methods.

Diffy
Messages
441
Reaction score
0
Hi, I have a set of data.

What I want to do is say with a 95% certainty this is a positive trend, or it is a negative trend.

Basically I have a set of observations (X_i, Y_i) where i is in {1,2,3,4,5,6...} and X_i = i and Y_i is an observation.

Right now I have the Variance, the standard deviations of my Y_i's and also I have a slope calculated (By Least Squares) from the data set. What do I need to do now to say whether I do or do not have a trend?

Thanks
 
Physics news on Phys.org
You are using linear regression, therefore the slope is the trend. To test the trend you can test the H0: the slope is 0. Be more specific, this test is to test if the trend is monotonic.

But trend can be some how change direction especially in the time series data. Therefore you need to fit different models to test the specific kind of trend.
 
Thank you for your reply. How exactly do I test this? Can you find an online example or more information? Or perhaps provide an example or more explanation?
 
Test of of trend is routine practice. For linear trend you need to test the slope.
http://www.ats.ucla.edu/stat/sas/webbooks/reg/chapter1/sasreg1.htm

Here is how SAS do linear regression. The first example has 3 covariates to predict api00, but you only have one covariate, so you need to ignore the other 2. Ignore the 2 only left you acs_k3, which has p-value 0.0553 and it is not significant, therefore the slope of acs_k3 is not significant.
 
I'm still having trouble. And I am sorry but that article didn't help. I don't have SAS.

From what I understand, I could do a permutation test on my data set. Calculate the slope or trend for each permutation. Then create a distribution. And find where in the distribution my particular trend lies.

Is there a way to do this without actually generating the permutations? It seems like I should be able to calculate this directly somehow.
 
you can search Excel regression to use. Excel regression does t test on slopes. Here is an example Excel did for regression on CPI by year.

year cpi
2001 177.1
2002 179.9
2003 184
2004 188.9
2005 195.3
2006 201.6
2007 207.342
2008 215.303
2009 214.537
2010 218.056

SUMMARY OUTPUT

Regression Statistics
Multiple R 0.990100513
R Square 0.980299025
Adjusted R Square 0.977836403
Standard Error 2.28783132
Observations 10

ANOVA
df SS MS F Significance F
Regression 1 2083.573556 2083.573556 398.0712704 4.15203E-08
Residual 8 41.87337719 5.234172148
Total 9 2125.446934

Coefficients Standard Error t Stat P-value Lower 95% Upper 95% Lower 95.0% Upper 95.0%
Intercept -9880.393909 505.1497455 -19.55933661 4.85314E-08 -11045.27131 -8715.516508 -11045.27131 -8715.516508
year 5.025478788 0.251881938 19.95172349 4.15203E-08 4.444637997 5.606319579 4.444637997 5.606319579
 
zil,

Thanks you for your patience and help. It lead me to the answer I was looking for.

I didn't know it when I originally asked the question but, what I needed to do is perform a t test on the slopes by hand. Using your posts and google's help I have figured out how to do this by hand. Which is exactly what I needed so that I could write this program.

Again thank you very much for helping me find my solution.

-Diffy
 

Similar threads

  • · Replies 13 ·
Replies
13
Views
2K
  • · Replies 23 ·
Replies
23
Views
4K
  • · Replies 19 ·
Replies
19
Views
3K
  • · Replies 7 ·
Replies
7
Views
3K
Replies
24
Views
3K
  • · Replies 9 ·
Replies
9
Views
3K
  • · Replies 11 ·
Replies
11
Views
5K
  • · Replies 3 ·
Replies
3
Views
3K
  • · Replies 2 ·
Replies
2
Views
2K
  • · Replies 11 ·
Replies
11
Views
4K