Comparing R vs Matlab for Speed: Processing 5000x2 Matrices

[Physics_wiz]

I have a huge set of data. About 5000 excel sheets and in each one about a 500x7 matrix. I'll be doing a lot of manipulation where I'll need a programming environment that is faster than excel (excel is pretty slow). Anyone have experience with both R and Matlab and can comment on which one is faster?

Just to give an idea of what I will be doing, out of those 5000 matrices, I will have to do some work with every combination of 2 from the total 5000. So I will have 5000 C 2 combinations or 12,497,500 combinations. After I choose which two matrices I'll work with, I will have to do things like addition, multiplication, and regression analysis on specific columns in the two matrices.

 

[Crosson]

After I choose which two matrices I'll work with, I will have to do things like addition, multiplication, and regression analysis on specific columns in the two matrices.

I think Matlab is the best tool for this task. I would only used R if you required a larger library of data analysis functions. That said, Matlab is far from free, and R is much better then Excel.

 

[Physics_wiz]

I already have Matlab, so price is not an issue. I will give this project a shot in Matlab and see how it goes. I was a little worried today when I wrote two for loops to make a matrix that's 1000x35,000 and give every element a specific value because it took 45 minutes to run and after I stopped it the matrix was about half way done. I will try to avoid huge for loops like that in my program, but I don't know how long it would take to run if I had to have something like this.

 

[neurocomp2003]

might want to look into parallel programming though MATLAB should be able to handle that size depending on your hardware

 

[LeBrad]

Matlab is notoriously slow when using for loops. Be sure to use vectorized commands whenever possible.