High-frequency time series database

[meanrev]

Main Question or Discussion Point

I'm choosing a database to write high-frequency time series data onto and have narrowed it down to MongoDB, Kyoto Cabinet or HDF5.

I will be inserting 1200 rows of 8 entries per second, cumulating about 5 GB of data per day I'm estimating.

Does anyone have experience between the three and could facilitate me in making the decision?

Thanks!

 

[gsal]

Well, I don't know much about databases; but I will give one silly opinion...

The thing is I have been trying to learn various things available within Python...one of those things I run into was, precisely, HDF5. When I read about HDF5, I understood that it was a storage scheme and not necessarily a database (i.e., there is no database server running with its own intelligence to answer queries or return sets or anything like that).

Of the other two choices that you mention, I just quickly read the main webpages and it looks like MonoDB is a real database (requires a server) and Kyoto Cabinet does not, this last, again, it's just a storage scheme.

So, my first opinion, if you need speed, is to forget about using a real database and stick to a storage scheme...so, Kyoto or HDF5.

It seems Kyoto talks about one key,value per line...does not seem too impressive as a storage scheme...but maybe that's where speed comes from.

HDF5, from what I remember, is actually rather versatile as far as as to what it can store.

The Kyoto site does not look like much...how popular is this?

Just becauss I learned about HDF5 before I ever heard about Kyoto, it sounds like HDF5 is more popular within the scientific/engineering community...

Anyway, that's my un-educated opinion.

gsal