Trail - vscode extension for research projects

[diegzumillo]

TL;DR

I made an extension that tracks execution and data generation

Hey there
I started working on this project mainly for my own benefit, because I'm very disorganized. My research is very computationally intensive, whether in physics or AI. So I generate lots of programs and data, and I lose track of it all.
So far I don't know if this is of interest to anyone else, but I wrote about it in my blog. I have a series of weekly projects, where each week I do something different.

Usually I make game related stuff, but this time I made this, which I think might have interested people in this forum.

Blog post about Trail

 

[jedishrfu]

This is an interesting idea. I recently completed a small project for grad studies and didn't worry about naming until I took a break for a few days, came back and couldn't decide which program was my latest version.

At work, this wouldn't have been an issue since I would've checked my code in instead of renaming it.

I was running compression tests on three testfiles over a 10 element range of error bounds values running five programs, my two regression programs and three well known compressors.

I made sure key results were printed and used markdown as my logging format. There's nothing like readable results in one place.

Other measurement data was encoded and output to csv files for analysis and plotting. My code was a layered mix of bash, Python test runner and my c code regression programs.

As I got deeper into the project, runs took longer as more test variations for each program were added and it took on a life of its own.

I did think of automating it even by more storing logs and csv files into date time labeled directories but kept telling myself okay that fix should do it, now one more test run and I’m done. 25 test runs later I got my final test run and my advisor asked about some zip deflation done which I hadn't recorded in my csv files oh well here we go again.

 

[diegzumillo]

Thanks for sharing that. It's validating to read other people's difficulties with this as well. And sure, there are infinite ways of solving this problem, with better organization mostly, but existing tools that help like GIT as well. But I do believe there is value in a system like this that just works with minimal work from the user, without disrupting your usual workflow. The only thing I couldn't get around is the requirement of a data folder.

Being unemployed, the goal of all these weekly projects is to find marketable ideas. So far I don't know where this one sits. If only sporadic end users like you and me would be interested in this, I don't think it could be sold. But if enough people find value in this then it might be marketable with tiers for organizations/universities packages and cheaper/free for individuals.

I might need to partner with someone more business savvy to analyze these projects eventually.

 

[jedishrfu]

Be careful about discussing your ideas here if you plan to commercialize them as it can torpedo any patents you might file since it was discussed beforehand online.

Also any ideas you gather must be carefully considered as you likely will bake them into your ideas.

There is an episode on This American Life titled When Patents Attack. One patent concerned an idea of sharing files online developed by two programmers trying to protect their source code. A consultant they brought in took notes, and then the company folded for lack of capital.

The consultant filed a couple of patents. Some years later, a patent troll law firm contacted him and started suing people who uploaded and downloaded files, including music, podcasts, videos, and files from backup services.

The lawyers sent letters demanding royalty payments. One company sued them over an apostrophe in the patent notes, which implied another inventor who wasn't listed on the patent. The company won its case, and the patent was thrown out.

It's a great podcast episode to listen to. There was a follow-up episode, too.

 

[fluidistic]

It does sound a good idea. Because yeah, people won't be using git as they could/should, i.e. do a commit with the message "file ..... produces picture of paper" and then in their thesis or paper, a git sha to the commit, so there's no doubt which file produced which picture and when it did it.

 

[diegzumillo]

There is nothing exactly new here, just a set of convenience tools bundled together. If I were using some fancy new tech for a diff engine, or something, I would understand the need for patents. But, on the other hand, I'm an idiot when it comes to most things except my precise interests. So I'll keep that in mind!