Dismiss Notice
Join Physics Forums Today!
The friendliest, high quality science and math community on the planet! Everyone who loves science is here!

Python workflow for experimental data analysis

  1. Aug 11, 2017 #1

    I have been using python for my data analysis and processing needs as an experimental physicist for an year now. I have used matlab and originpro before and python provides me everything I need.
    But I am not satisfacted with my worflow specifically for plotting needs. I often find myself looking for old plots (images) that I did and having to rewrite code to do them over again, which is counterproductive.

    I am currently using a combo of sublime text 3 (plus Anaconda plugins) and Spyder IDE for data visualization and running code on its Ipython interpreter (I like the variable explorer and the Ipython interpreter is much superior to SublimeREPL) . It works great for data processing and general coding but saving plots and processed data in a tidy way has been a hassle. If I could save all my workspace like in matlab it would be much better (Spyder's implementation of this is limited).

    Originpro in contrast automatically saves all the data and plots in the same project which is a great workflow for plotting and saving your work. I must say that even having a decent background with python I find myself more productive for plotting and light data processing on originpro.

    I am aware that there are alternatives such as Jupyter and Pycharm. Maybe I should give Jupyter notebook another shot.

    I wanted to have some insight from you guys. What is your workflow for data analysis/plotting with python? Do you have a hard time going back and editting plots and datasets? How do you save your work?
  2. jcsd
  3. Aug 11, 2017 #2

    Dr Transport

    User Avatar
    Science Advisor
    Gold Member

    from your post, you appear to not be saving your python scripts. Sure in Matlab you can save the workspace, but I never do, I write everything as a .m file and rerun later if I have to. I also work my imagery/plots then save a copy to refer back to later and include in presentations and reports, so why aren't you doing the same with your python.
  4. Aug 11, 2017 #3
    Well, I do save most of my python scripts but some work I do in the form of interactive programming (ipython interpreter). Specially when I am exploring the data and testing different visualizations. And sometimes I do batches of plot that involve running a combination of custom function several times and it is time consuming to document everything in such way I can recreate that specific workspace by running a single script. I wish I could just pick it up from where I left and easily edit the work instead of having to recreate it every time.
  5. Aug 11, 2017 #4


    User Avatar
    Science Advisor
    Homework Helper
    Gold Member

    Can you programmatically create a log of your selected options [as a text file],
    together with accompanying plot images, possibly dumped into a folder with a timestamp label?
    You might consider outputting in that text file the python code of your options [e.g. function calls with parameters] that you could copy-paste to reproduce the plot.
    (Later, you could write a script that will [say] create an html page that can show all of the plot images (sitting in the various folders) so that you can survey the variations.)
  6. Aug 11, 2017 #5

    Dr Transport

    User Avatar
    Science Advisor
    Gold Member

    When I do a new analysis in matlab, I will comment out lines, document and add to the file until I get what I want, I don't really ever delete lines of code, if you did that in your scripts then you'd have your log of keystrokes.....
Know someone interested in this topic? Share this thread via Reddit, Google+, Twitter, or Facebook

Have something to add?
Draft saved Draft deleted

Similar Discussions: Python workflow for experimental data analysis