Getting started with HDF5

[Borg]

Though one of my many internet searches, I ran across Hierarchical Data Formats (HDF) last year. It seems interesting but I feel like I'm in over my head a bit. I believe that some people here have use it and I'm looking for advice on getting started, what to watch for and how it's different from other data storage schemes.

My background with databases includes work with various relational databases like Oracle, Sybase, MySQL, etc. For the last year, I've been introduced to the world of NoSQL using MongoDB so it is possible for me to learn new tricks.

One of my concerns is that my math skills are very rusty. I have occasionally restudied calculus up to the chain rule and I have written programs using spherical coordinates. Getting back to any ability with PDEs or statistics will take a while. This makes me wonder if I would be able to actually make use of HDF.

Thanks in advance.

 

[jedishrfu]

Here a starting point

http://en.m.wikipedia.org/wiki/Hierarchical_Data_Format

I've used netcDF file format which is based on hdf5 and which used commonly for scientific data and isn't to difficult to master.

 

[Borg]

Here a starting point

http://en.m.wikipedia.org/wiki/Hierarchical_Data_Format

I've used netcDF file format which is based on hdf5 and which used commonly for scientific data and isn't to difficult to master.

Thanks jedishrfu. I have read the Wiki site, started reading the 350+ page users manual and have been reviewing a tutorial using the java api. I've got a minor classpath problem right now but I'm sure that I'll fix it soon. In the short term, I would like to use it to analyze stock data but, my main concern is how much math will I need in order to make good use of it? Where does the real strength of HDF5 lie mathematically?

 

[jedishrfu]

It's just a file format that makes a database across multiple files. There s no math beyond algebra if even that.

Are you trying to read hdf5 or store data in it?

I'd use a regular database like h2 if you're creating it.

 

[Borg]

So, is it nothing more than just another database scheme that happens to be capable of reading data faster?

I'm looking at one of the example programs called H5Ex_D_Chunk.java. It looks like a simple test to write data and then read it back out. I had a path issue that went away when I added the HDF lib directory to my path Unfortunately, I'm still getting the following error:
jhdf5.dll: Can't find dependent libraries

I've set both my path and classpath to the HDF lib directory but, I'm not getting around the error. I'm trying to do this on a Windows machine and it seems that HDF really prefers to be in a Unix environment.

 

[jedishrfu]

Your answer may be among the many FAQ questions answered here

http://www.hdfgroup.org/hdf5-quest.html [Broken]

More specifically here if macosx or linux

http://www.hdfgroup.org/hdf5-quest.html#jlink [Broken]

An interesting answer to why use hdf5

http://www.hdfgroup.org/HDF5/faq/whyhdf5.html#dirstruct

And the best I can do for windows based hdf5

http://www.hdfgroup.org/ftp/HDF5/releases/HDF-JAVA/HDF-JAVA-2.3/src/build_src_windows.html

I've not used hdf5 directly but have been researching it recently. I have used netcdf in Java for some array data where I had to create a 2D array and two 1D arrays for the x and y axes of the 2D array so for a give. Data point would be x(3) y(4) data(3)(4).

 

[Borg]

Thanks again jedishrfu. I will definitely take a look at the links. The one thing that I keep noticing though is that a lot of reference links in those pages link to pages of this form: http://hdf.ncsa.uiuc.edu/ [Broken]...
Whenever I click on them, my internet provider displays a page that the site can't be found. If I take out the hdf part, I get to the Univ. of Illinois but searches there don't find much w.r.t. hdf. Either the server doesn't exist anymore, it's been moved, or the links are for an internal network. I may have to create an account on an HDF forum.

 

[jedishrfu]

Yeah I see that on this page too:

http://www.ncsa.illinois.edu/People/mcgrath/docs.html

I guess that's when you use Google to find the documents of interest or contact the author of the above page and see where they are now.

I tried searching for a couple of those listed notably: Applications Using NCSA HDF Files but with no luck.

 

[Borg]

Good to know that it's not just me.

Just curious - what OS did you use with the Java API?

 

[jedishrfu]

Linux