Smoother EWMA that mean-reverts

  • Context: Undergrad 
  • Thread starter Thread starter cppIStough
  • Start date Start date
Click For Summary

Discussion Overview

The discussion revolves around the concept of smoothing techniques for the Exponential Weighted Moving Average (EWMA) in time series data analysis, particularly focusing on improving the responsiveness of the EWMA to recover from spikes in the data. Participants explore various methods and considerations related to smoothing and spike identification.

Discussion Character

  • Exploratory
  • Technical explanation
  • Conceptual clarification
  • Debate/contested

Main Points Raised

  • One participant expresses concern that the standard EWMA does not smooth maximum values effectively, especially after large spikes, and seeks suggestions for a smoother alternative that reverts to pre-spike levels more quickly.
  • Another participant suggests considering fixed-time moving averages, such as 3-day or 7-day moving averages, as they can smooth out daily averages but may dilute spikes due to averaging over multiple days.
  • A participant raises questions about the definition of a "spike," emphasizing the need to distinguish between random data errors and systematic errors, and the importance of establishing measurable criteria for identifying spikes.
  • The same participant reiterates the significance of measurement methodologies, suggesting that discussions often overlook foundational questions about what and how to measure in data analysis.

Areas of Agreement / Disagreement

Participants do not reach a consensus on a specific method for smoothing EWMA or on the definition of spikes, indicating that multiple competing views and uncertainties remain in the discussion.

Contextual Notes

The discussion highlights limitations related to the definitions of spikes and the criteria for measurement, which remain unresolved and may affect the proposed solutions.

cppIStough
Messages
24
Reaction score
2
EWMA (exponential weighted moving average) is one way to estimate variance of time series data, and is pretty well known. The issue I have with EWMA is the maximums aren't smooth, especially when recovering from a time-series large spike, and it can take a little while to recover to pre-spike levels. I'm wondering if you know of (or are creative enough to come up with it yourself) a smoother EWMA that reverts to previous-spike levels quicker.

Let me know if I'm not clear, and thanks again for your advice!
 
Physics news on Phys.org
You might consider a fixed-time moving average. The data of Covid-19 deaths is a good example. That data is often presented with 3-day and 7-day moving average options. The 7-day MA has an advantage of always including one weekend, when reporting is always low, and a Monday/Tuesday, when the reports catch up for the weekend (either this weekend or the prior weekend). The advantage is that it greatly smooths out the daily average numbers and suppresses the weekly cycles. The disadvantage is that any spike or variation is watered down by the surrounding 6 days.

Alternatively, you could use your own weights.
 
Last edited:
  • Like
Likes   Reactions: fresh_42
Covid was an interesting example @FactChecker mentioned. It shows the questions I had (and didn't post as they missed rigor until I saw the Covid example).

What is a spike, a potential data error (random), or a system immanent error (repeated) as in the Covid case? Is there a specific point above which you call data a spike? The word spike has a connotation of something you see in the data, not of something you measure. You first have to make it measurable in order to deal with it.
 
  • Like
Likes   Reactions: FactChecker
fresh_42 said:
Covid was an interesting example @FactChecker mentioned. It shows the questions I had (and didn't post as they missed rigor until I saw the Covid example).

What is a spike, a potential data error (random), or a system immanent error (repeated) as in the Covid case? Is there a specific point above which you call data a spike? The word spike has a connotation of something you see in the data, not of something you measure. You first have to make it measurable in order to deal with it.
Those are the big questions: What do you measure, how do you measure it? Maybe too many only deal with technical aspects but don't dwell on such important questions.
 
  • Like
Likes   Reactions: FactChecker

Similar threads

  • · Replies 1 ·
Replies
1
Views
2K
  • · Replies 8 ·
Replies
8
Views
2K
  • · Replies 14 ·
Replies
14
Views
4K
  • · Replies 5 ·
Replies
5
Views
764
  • · Replies 5 ·
Replies
5
Views
2K
  • · Replies 2 ·
Replies
2
Views
3K
  • · Replies 0 ·
Replies
0
Views
3K
  • · Replies 4 ·
Replies
4
Views
4K
  • · Replies 4 ·
Replies
4
Views
2K
Replies
3
Views
3K