Getting started with HDF5

  • Thread starter Borg
  • Start date
  • Tags
Windows 7 x64.In summary, Hierarchical Data Formats (HDF) are a data storage scheme that is based on the hdf5 file format. It is possible for someone with some background in databases to learn how to use HDF5. However, the real strength of HDF5 lies mathematically.
  • #1


Science Advisor
Gold Member
Though one of my many internet searches, I ran across Hierarchical Data Formats (HDF) last year. It seems interesting but I feel like I'm in over my head a bit. I believe that some people here have use it and I'm looking for advice on getting started, what to watch for and how it's different from other data storage schemes.

My background with databases includes work with various relational databases like Oracle, Sybase, MySQL, etc. For the last year, I've been introduced to the world of NoSQL using MongoDB so it is possible for me to learn new tricks.

One of my concerns is that my math skills are very rusty. I have occasionally restudied calculus up to the chain rule and I have written programs using spherical coordinates. Getting back to any ability with PDEs or statistics will take a while. This makes me wonder if I would be able to actually make use of HDF.

Thanks in advance.
Technology news on
  • #3
jedishrfu said:
Here a starting point

I've used netcDF file format which is based on hdf5 and which used commonly for scientific data and isn't to difficult to master.
Thanks jedishrfu. I have read the Wiki site, started reading the 350+ page users manual and have been reviewing a tutorial using the java api. I've got a minor classpath problem right now but I'm sure that I'll fix it soon. In the short term, I would like to use it to analyze stock data but, my main concern is how much math will I need in order to make good use of it? Where does the real strength of HDF5 lie mathematically?
  • #4
It's just a file format that makes a database across multiple files. There s no math beyond algebra if even that.

Are you trying to read hdf5 or store data in it?

I'd use a regular database like h2 if you're creating it.
  • #5
So, is it nothing more than just another database scheme that happens to be capable of reading data faster?

I'm looking at one of the example programs called It looks like a simple test to write data and then read it back out. I had a path issue that went away when I added the HDF lib directory to my path Unfortunately, I'm still getting the following error:
jhdf5.dll: Can't find dependent libraries

I've set both my path and classpath to the HDF lib directory but, I'm not getting around the error. I'm trying to do this on a Windows machine and it seems that HDF really prefers to be in a Unix environment.
  • #6
Your answer may be among the many FAQ questions answered here [Broken]

More specifically here if macosx or linux [Broken]

An interesting answer to why use hdf5

And the best I can do for windows based hdf5

I've not used hdf5 directly but have been researching it recently. I have used netcdf in Java for some array data where I had to create a 2D array and two 1D arrays for the x and y axes of the 2D array so for a give. Data point would be x(3) y(4) data(3)(4).
Last edited by a moderator:
  • Like
Likes Borg
  • #7
Thanks again jedishrfu. I will definitely take a look at the links. The one thing that I keep noticing though is that a lot of reference links in those pages link to pages of this form: [Broken]...
Whenever I click on them, my internet provider displays a page that the site can't be found. If I take out the hdf part, I get to the Univ. of Illinois but searches there don't find much w.r.t. hdf. Either the server doesn't exist anymore, it's been moved, or the links are for an internal network. I may have to create an account on an HDF forum.
Last edited by a moderator:
  • #8
Yeah I see that on this page too:

I guess that's when you use Google to find the documents of interest or contact the author of the above page and see where they are now.

I tried searching for a couple of those listed notably: Applications Using NCSA HDF Files but with no luck.
  • #9
Good to know that it's not just me. :oldsmile:

Just curious - what OS did you use with the Java API?
  • #10

1. What is HDF5 and why should I use it?

HDF5 stands for Hierarchical Data Format version 5, and it is a file format commonly used in scientific and engineering applications to store and manage large and complex datasets. It offers a flexible and efficient way to organize and access data, making it ideal for handling big data and performing data analysis.

2. How do I install HDF5 on my computer?

To get started with HDF5, you will need to download and install the HDF5 library from the official website. The installation process may differ depending on your operating system, but detailed instructions can be found in the HDF5 User's Guide. Once installed, you can begin creating and working with HDF5 files.

3. What are the basic data structures in HDF5?

The two main data structures in HDF5 are datasets and groups. Datasets are used to store and organize the actual data, while groups are used to organize and group datasets together. These data structures can be nested within each other to create a hierarchical data structure, allowing for more efficient storage and retrieval of data.

4. How can I access and manipulate data in HDF5 files?

To access and manipulate data in HDF5 files, you can use a variety of programming languages, including C, C++, Java, and Python. Each language has its own library for working with HDF5 files, such as h5py for Python or HDF5 C++ for C++. These libraries provide functions and methods for reading, writing, and manipulating data in HDF5 files.

5. Are there any tools available for visualizing HDF5 data?

Yes, there are several tools available for visualizing HDF5 data, such as HDFView, HDF Explorer, and HDF Lab. These tools allow you to browse and view the contents of HDF5 files, and some also offer advanced features for data analysis and visualization. Additionally, many scientific data analysis platforms, like MATLAB and R, have built-in support for working with HDF5 files.

Suggested for: Getting started with HDF5