Statistical analysis: data filtering

In summary, Russ is trying to figure out how to filter out the correlation between kWh and temperature, but he is having trouble.
  • #1
russ_watters
Mentor
23,159
10,363
This may be too much work, so I understand if no one tackles the problem...

I'm doing an analysis of the power usage of a building. Sheet 2 of the attached spreadsheet shows typical data (we have 4 years of it). The top graph is the kWh used every 30 minutes of every day in January, 2003. The second graph is the temperature at those times. As you can see, there is a correlation between temperature and usage (the building has electric heat). For example, the light and dark blue diamonds for 1/1 and 1/10 show the highest temperature and lowest usage. What I would like to do is filter-out this correlation. This would leave me with a usage profile that reflects occupancy and pretty much nothing else. All of the lines for kWh would lay right on top of each other most of the time.

Sheet 1 shows some of my efforts to this point. The graph seems to show a corellation between kWh and temp (though, admittedly, not super-strong). Of note, there is something called an "equilibrium temperature." Its basically the temperature at which no heating or air conditioning is required. At this point, the usage will start going up again. So this would serve as a baseline - a minimum usage - for building the corellation. It appears to be at about 50 degrees.

Now, it looks like it should be possible to just construct this line and subtract it from the usage data. But it doesn't appear to be working. Any help would be greatly appreciated.
 

Attachments

  • data.ZIP
    9.1 KB · Views: 248
Physics news on Phys.org
  • #2
Hi, russ:
I don't think I can be of any real help here, but when I tried to open the zip-file, I got an "invalid/damaged" message.
It might be something with my computer, but in case it isn't, I thought you should know.
 
  • #3
Strange - when I try to open it right from the web, it comes up empty, but when I download it, it works. Try downloading it - and thanks for picking up on that.
 

What is statistical analysis?

Statistical analysis is the process of collecting, organizing, analyzing, and interpreting numerical data in order to gain insights and make informed decisions. It involves using mathematical and statistical methods to identify patterns, trends, and relationships within the data.

Why is data filtering important in statistical analysis?

Data filtering is important in statistical analysis because it allows researchers to remove irrelevant or erroneous data from their dataset. This helps to ensure that the data being analyzed is accurate and representative of the population being studied, leading to more reliable results.

What are some common methods of data filtering in statistical analysis?

Some common methods of data filtering include removing outliers, eliminating duplicate data, and setting criteria for including or excluding certain data points based on predetermined factors. Other methods may also involve using statistical tests to identify and remove biased or misleading data.

How do you determine which data points to filter?

The decision on which data points to filter depends on the goals and objectives of the study, as well as the specific research question being investigated. It is important for researchers to carefully consider the potential impact of filtering on their results and to document their justification for filtering specific data points.

Can data filtering affect the results of a statistical analysis?

Yes, data filtering can affect the results of a statistical analysis. Removing certain data points can change the overall distribution and characteristics of the data, potentially leading to different conclusions. It is important for researchers to carefully consider the impact of data filtering on their results and to report any filtering methods used in their analysis.

Similar threads

  • STEM Educators and Teaching
Replies
5
Views
633
Replies
16
Views
847
  • Introductory Physics Homework Help
Replies
6
Views
2K
Replies
6
Views
2K
  • Electrical Engineering
Replies
14
Views
793
  • DIY Projects
Replies
13
Views
1K
  • MATLAB, Maple, Mathematica, LaTeX
Replies
6
Views
2K
  • Electrical Engineering
Replies
10
Views
1K
  • Set Theory, Logic, Probability, Statistics
Replies
1
Views
2K
Replies
1
Views
2K
Back
Top