Managing simulation output files

[Avarus]

Hey all,

This may be a bit of a stupid question, but I'll ask it anyway... For the past weeks I've been running some DEM (discrete element method) simulations, and each simulations produces a couple output files and has different input variables/commands stored in a separate file. Already I start to lose track of which simulation produced which results with which input script, and I still have 4 more years to go!

Is there some kind of 'book keeping' program out there for managing the files involved in a simulation? Or do you have a method of organisation that works well for you? Please teach me, oh masters...

Thanks and regards,

Martijn

 

[jtbell]

I would arrange for the input and output files to have similar names, e.g. may07input.dat, may07output1.dat, may07output2.dat.

You could accomplish this by renaming the files by hand after each run.

Or you could write a script in Perl or Python or some Unix/Linux shell or whatever other scripting language you know, to run the software and rename the files automatically.

Or if you have access to the source code and know the language it's written in (e.g. you wrote the software yourself), you can have the software generate appropriate output file names based on the input file name.

 

[phyzguy]

I think the best way is to create a directory for each run, then have all of the input and output files go in the same directory. That way if you lose track, you can go look at the input script which produced the output in the directory. It helps to give each directory a descriptive name as well, as jtbell suggested. When faced with this, I also modified the software to generate a README file in each directory with a (computer generated) summary of the input parameters. That way, after you have generated hundreds or thousands of directories and are looking for a particular one or a particular combination, you can write a script to search through them and find the one you want.

 

[jtbell]

I should have thought of the separate-directories trick myself. :thumbs:

Then you don't have to rename files or write a script or mess around with source code.

 

[f95toli]

If you have written the code yourself I would suggest changing the output data format.
It might also be worth investigating some more modern file formats for saving data; e.g HDF5
These allow you to structure your data in a much better way, and you can also save all parameters, comments etc in the same file.

I've been using HDF5 on an off for a few years now for my experimental data, and it is MUCH better than just saving text files (ti was orignally developed to handle data generated by supercomputers).

There are many other formats that do the same, but HDF5 is nice because it is free and supported by many software packages (e.g. Matlab and Octave) and you can get free libraries for a many languages (C, Java, Perl, Python etc) that you can use in your own code.

 

[Avarus]

Thanks for the suggestions! For now I'll stick to the directory-tree structure, but in the future I might consider HDF5 data structure. I've actually used that format before, so perhaps I can build a properly structured database in that way...

By the way if anyone else has any more suggestions, please do post them here, I'll keep an eye on this thread...