Dismiss Notice
Join Physics Forums Today!
The friendliest, high quality science and math community on the planet! Everyone who loves science is here!

Getting started with HDF5

Tags:
  1. Feb 7, 2015 #1

    Borg

    User Avatar
    Gold Member

    Though one of my many internet searches, I ran across Hierarchical Data Formats (HDF) last year. It seems interesting but I feel like I'm in over my head a bit. I believe that some people here have use it and I'm looking for advice on getting started, what to watch for and how it's different from other data storage schemes.

    My background with databases includes work with various relational databases like Oracle, Sybase, MySQL, etc. For the last year, I've been introduced to the world of NoSQL using MongoDB so it is possible for me to learn new tricks.

    One of my concerns is that my math skills are very rusty. I have occasionally restudied calculus up to the chain rule and I have written programs using spherical coordinates. Getting back to any ability with PDEs or statistics will take a while. This makes me wonder if I would be able to actually make use of HDF.

    Thanks in advance.
     
  2. jcsd
  3. Feb 8, 2015 #2

    jedishrfu

    Staff: Mentor

  4. Feb 8, 2015 #3

    Borg

    User Avatar
    Gold Member

    Thanks jedishrfu. I have read the Wiki site, started reading the 350+ page users manual and have been reviewing a tutorial using the java api. I've got a minor classpath problem right now but I'm sure that I'll fix it soon. In the short term, I would like to use it to analyze stock data but, my main concern is how much math will I need in order to make good use of it? Where does the real strength of HDF5 lie mathematically?
     
  5. Feb 8, 2015 #4

    jedishrfu

    Staff: Mentor

    It's just a file format that makes a database across multiple files. There s no math beyond algebra if even that.

    Are you trying to read hdf5 or store data in it?

    I'd use a regular database like h2 if you're creating it.
     
  6. Feb 8, 2015 #5

    Borg

    User Avatar
    Gold Member

    So, is it nothing more than just another database scheme that happens to be capable of reading data faster?

    I'm looking at one of the example programs called H5Ex_D_Chunk.java. It looks like a simple test to write data and then read it back out. I had a path issue that went away when I added the HDF lib directory to my path Unfortunately, I'm still getting the following error:
    jhdf5.dll: Can't find dependent libraries

    I've set both my path and classpath to the HDF lib directory but, I'm not getting around the error. I'm trying to do this on a Windows machine and it seems that HDF really prefers to be in a Unix environment.
     
  7. Feb 9, 2015 #6

    jedishrfu

    Staff: Mentor

    Your answer may be among the many FAQ questions answered here

    http://www.hdfgroup.org/hdf5-quest.html [Broken]

    More specifically here if macosx or linux

    http://www.hdfgroup.org/hdf5-quest.html#jlink [Broken]

    An interesting answer to why use hdf5

    http://www.hdfgroup.org/HDF5/faq/whyhdf5.html#dirstruct

    And the best I can do for windows based hdf5

    http://www.hdfgroup.org/ftp/HDF5/releases/HDF-JAVA/HDF-JAVA-2.3/src/build_src_windows.html

    I've not used hdf5 directly but have been researching it recently. I have used netcdf in Java for some array data where I had to create a 2D array and two 1D arrays for the x and y axes of the 2D array so for a give. Data point would be x(3) y(4) data(3)(4).
     
    Last edited by a moderator: May 7, 2017
  8. Feb 9, 2015 #7

    Borg

    User Avatar
    Gold Member

    Thanks again jedishrfu. I will definitely take a look at the links. The one thing that I keep noticing though is that a lot of reference links in those pages link to pages of this form: http://hdf.ncsa.uiuc.edu/ [Broken]...
    Whenever I click on them, my internet provider displays a page that the site can't be found. If I take out the hdf part, I get to the Univ. of Illinois but searches there don't find much w.r.t. hdf. Either the server doesn't exist anymore, it's been moved, or the links are for an internal network. I may have to create an account on an HDF forum.
     
    Last edited by a moderator: May 7, 2017
  9. Feb 9, 2015 #8

    jedishrfu

    Staff: Mentor

    Yeah I see that on this page too:

    http://www.ncsa.illinois.edu/People/mcgrath/docs.html

    I guess that's when you use Google to find the documents of interest or contact the author of the above page and see where they are now.

    I tried searching for a couple of those listed notably: Applications Using NCSA HDF Files but with no luck.
     
  10. Feb 9, 2015 #9

    Borg

    User Avatar
    Gold Member

    Good to know that it's not just me. :oldsmile:

    Just curious - what OS did you use with the Java API?
     
  11. Feb 9, 2015 #10

    jedishrfu

    Staff: Mentor

Know someone interested in this topic? Share this thread via Reddit, Google+, Twitter, or Facebook




Similar Discussions: Getting started with HDF5
  1. Starting porgramming (Replies: 4)

Loading...