Dismiss Notice
Join Physics Forums Today!
The friendliest, high quality science and math community on the planet! Everyone who loves science is here!

High-frequency time series database

  1. Sep 1, 2012 #1
    I'm choosing a database to write high-frequency time series data onto and have narrowed it down to MongoDB, Kyoto Cabinet or HDF5.

    I will be inserting 1200 rows of 8 entries per second, cumulating about 5 GB of data per day I'm estimating.

    Does anyone have experience between the three and could facilitate me in making the decision?

  2. jcsd
  3. Sep 2, 2012 #2
    Well, I don't know much about databases; but I will give one silly opinion...

    The thing is I have been trying to learn various things available within Python...one of those things I run into was, precisely, HDF5. When I read about HDF5, I understood that it was a storage scheme and not necessarily a database (i.e., there is no database server running with its own intelligence to answer queries or return sets or anything like that).

    Of the other two choices that you mention, I just quickly read the main webpages and it looks like MonoDB is a real database (requires a server) and Kyoto Cabinet does not, this last, again, it's just a storage scheme.

    So, my first opinion, if you need speed, is to forget about using a real database and stick to a storage scheme...so, Kyoto or HDF5.

    It seems Kyoto talks about one key,value per line...does not seem too impressive as a storage scheme...but maybe that's where speed comes from.

    HDF5, from what I remember, is actually rather versatile as far as as to what it can store.

    The Kyoto site does not look like much...how popular is this?

    Just becauss I learned about HDF5 before I ever heard about Kyoto, it sounds like HDF5 is more popular within the scientific/engineering community...

    Anyway, that's my un-educated opinion.

Share this great discussion with others via Reddit, Google+, Twitter, or Facebook