Data analysis software for hydrology research

In summary: I do as long as the data is accurate and organized. I think she just wants someone to help her get started and stop asking her to do calculations herself. She mentioned that she may ask for more detailed reports in the future, but for now I'm just going to try to be as helpful as possible.
  • #1
Alkemyst
4
0
Hello there!

This is my first time posting, but I'm a long time reader. I could not find a more appropriate sub-forum.

I have just been hired to compile and organize data for an arctic hydrology research project in Fairbanks, AK. They are studying climate change and have recorded a few years worth of data. 150 temperature sensors in a "glacial polygon" (that's what the researcher said), about 1 meter between each sensor, recording once every hour. I'm scared to calculate how many "values" that would give us...

My employer has said she isn't very picky about how the data is compiled. She just wants to start by finding data anomalies and spotting general trends. I'm basically going to help her enter data and make graphs. She plans to request a "final product" and I can go about getting there however I see fit.

I am familiar with these programs: Excel, Mathematica, MATLAB, Microlab, Datastudio.

Microlab and Datastudio were used in my lower level Chemistry and Physics labs. I've never used Mathematica or MATLAB for generating graphs, but I've heard they work well for that. She only knows excel, which is fairly easy to use. I'm comfortable with it, but it doesn't seem as versatile as the subject-specific programs I mentioned.

I am only a second year undergraduate in chemistry. I have lab experience from 1st and 2nd year science courses and one summer job, but not much else. I essentially know nothing except how to work (which can be enough some times). Can anyone recommend programs that are usually used for large amounts of data like this? Actually, any advice at all is welcome. I just want to do this small job well, and I would put in the time to learn how to use new tools and techniques.

Thank you!

Note: I can provide much more information, but this post is already long.
 
Physics news on Phys.org
  • #2
I would actually argue excel is more versatile than most of the other programs you mention for what you're doing. That's a subjective opinion.

HOWEVER, excel just isn't going to handle all that data well.

24 hours X 365 days a year X 150 sensors is already past the excel row limit.

So nix that one. I'd mess with mathematica and MATLAB last.

What about R?
 
  • #3
If I was in your shoes, here’s what I would do:

Use R or some similar program (if one of the others you listed works, great) to find the extreme outliers. Figure out if there’s a reason for their difference, of if they are erroneous, and remove IF appropriate.

Export the weekly averages to Excel (or to a file that can be imported to Excel). The reason I would do this is because I’m good with Excel; few people realize how powerful a program it really is. By using weekly averages you’re down to 25k data points per year, which is very manageable. You may choose to roll it up again into monthly values or by groups (or all) of sensors, but I always try to get the most detailed data I can into the workbook before I go rolling anything up, in case I need to get back to it later.

Find your mean, your median, your variance for the whole set, then various subsets (groups of devices, periods of time, etc.). Make some graphs – maybe a lot of graphs – to try and find patterns or problems in the data.

Ask questions about why the data is the way it is. Do your best to answer them. Once you have some specific question, more specific statistical tools may be necessary to answer them.

Produce some reports that will stand as useful discussion with your supervisor; these should include your methods, what you’ve done, what information you obtained from it. This will be made easier if you make notes as you go. Set a meeting time with your supervisor and present them.

But that’s just me, someone else may have some better advice.
 
  • #4
Thank you!

That is tremendously helpful.

I finally got my hands on the actual data itself just yesterday. Most of this weekend will involve basic sorting and organization, which is just fine in excel. The next step is conversions from voltage into temperature and soil moisture. It'll be a bit before I can even start on analysis, but excel will be fine for most of the first steps. Actually, this will probably keep me busy for awhile.

She listed several reasons for outliers or "gaps." Some of the sensors are solar powered, and Alaskan winters have very little sun, so two or three months out of every year "cut out." Also, some simply get dug up and chewed on by wild animals. They try to check every once in a while to recalibrate, but one little blip obviously isn't worth a flight up to Barrow.

The nature of my employment seems more secretarial than student. She requested more in terms of tracking down and labeling sets of data than in terms of actual analysis and scientific judgment.

Thank you again for your reply!

-- Salome'
 
  • #5


Hello there!

As a fellow scientist, I am happy to see your enthusiasm and determination to do your job well. Data analysis software is an essential tool for any research project, and it sounds like you have a good understanding of the different programs that are commonly used. In this case, I would recommend using MATLAB or Mathematica for your data analysis and graphing needs. These programs are specifically designed for handling large amounts of data and have advanced tools and capabilities for data analysis.

However, if you are more comfortable with Excel, it can also be a useful tool for simple data analysis and graphing. Just make sure to familiarize yourself with its features and limitations before starting your project.

In addition to software, I would also suggest reading up on basic data analysis techniques and statistics. This will help you better understand the data and identify any anomalies or trends.

Finally, don't hesitate to reach out to your employer or other colleagues for guidance and advice. They may have specific preferences or tips for organizing and analyzing the data. Good luck with your project!
 

1. What is data analysis software for hydrology research?

Data analysis software for hydrology research is a computer program that helps hydrologists and other scientists analyze large amounts of hydrological data. This software is designed to handle various types of data, such as streamflow data, rainfall data, and water quality data, and to provide tools for visualization, statistical analysis, and modeling.

2. What are the benefits of using data analysis software for hydrology research?

Using data analysis software for hydrology research can provide several benefits. Firstly, it can save time and effort by automating repetitive tasks and handling large datasets more efficiently. It also allows for more accurate and precise analysis of data, as well as the ability to visualize and interpret data in different ways. Additionally, data analysis software can help identify patterns and trends in the data, which can aid in making informed decisions and predictions.

3. What are some popular data analysis software for hydrology research?

Some popular data analysis software for hydrology research include HEC-HMS, SWAT, MODFLOW, and R. These software programs offer various features such as data manipulation, statistical analysis, and hydrological modeling. They are widely used and have a strong user community, making it easier to find support and resources for using the software.

4. Do I need programming skills to use data analysis software for hydrology research?

It depends on the software you are using. Some data analysis software, such as HEC-HMS and SWAT, require some level of programming skills to use. However, there are also user-friendly software programs, like MODFLOW and R, that do not require programming skills and have a graphical user interface (GUI) for data analysis and visualization.

5. How can data analysis software for hydrology research benefit my research or work?

Data analysis software for hydrology research can benefit your research or work in several ways. It can help you analyze and interpret large datasets quickly and accurately, saving time and effort. It also allows for more advanced analysis and modeling, which can lead to better understanding and prediction of hydrological processes. Additionally, using data analysis software can enhance the visual representation of data, making it easier to communicate research findings to others.

Similar threads

  • MATLAB, Maple, Mathematica, LaTeX
Replies
3
Views
72
  • MATLAB, Maple, Mathematica, LaTeX
Replies
3
Views
2K
  • MATLAB, Maple, Mathematica, LaTeX
Replies
6
Views
2K
  • MATLAB, Maple, Mathematica, LaTeX
Replies
12
Views
3K
  • STEM Educators and Teaching
Replies
5
Views
543
  • Science and Math Textbooks
Replies
6
Views
1K
  • MATLAB, Maple, Mathematica, LaTeX
Replies
1
Views
2K
  • MATLAB, Maple, Mathematica, LaTeX
Replies
5
Views
2K
  • MATLAB, Maple, Mathematica, LaTeX
Replies
1
Views
2K
Back
Top