Easy: importing a batch of data into MATLAB

[imost]

I've been going crazy trying to figure this out- I'm not seeing it in the documentation. I have a few hundred files named "data*" where * goes from 000 to 300. The file type is really just a text file without any real type- they aren't .txt. Is there a way to 'batch' import all of them without selecting each one individually and importing? They are all in one folder, and they are all formatted identically as having two columns.

I've been trying all sorts of tricks using data* and the 'importdata' syntax, but I'm stuck. Maybe there is a loop I can try, if you know of any.

Any suggestions?

 

[John Creighto]

I've been going crazy trying to figure this out- I'm not seeing it in the documentation. I have a few hundred files named "data*" where * goes from 000 to 300. The file type is really just a text file without any real type- they aren't .txt. Is there a way to 'batch' import all of them without selecting each one individually and importing? They are all in one folder, and they are all formatted identically as having two columns.

I've been trying all sorts of tricks using data* and the 'importdata' syntax, but I'm stuck. Maybe there is a loop I can try, if you know of any.

Any suggestions?

Why not just write a script file that loops through all the files in the directory and imports them if they meat your criteria.

 

[MATLABdude]

Why not just write a script file that loops through all the files in the directory and imports them if they meat your criteria.

This may be easy for you, and easy for me, but not so easy for someone new(ish) to MATLAB.

If you're just starting out, MATLAB's (internal) documentation is pretty good, and the online documentation is even better (and gives quite helpful suggestions for related functions--think Amazon without trying to sell you something). For instance, typing the following brings up the internal help for the dlmread function, delimited read from ASCII text file:
>>help dlmread

The online documentation for the dlmread function would be at:
http://www.mathworks.com/access/helpdesk/help/techdoc/ref/dlmread.html

Okay, basics aside, you can harness the interpretive nature of MATLAB using the eval function, which evaluates a text string (which you can generate on the fly!) Okay, let's assume that you want to import myData000.txt to myData999.txt (numerical data stored without a header row, and in 2 columns--separated by commas--with 100 rows--separated by carriage return). The following code would probably work, and produce a cell array (myData) with 1000 entries, each containing a 2x100 matrix of values.

for i=0:999
	% The setup
	comStr=['myData{', num2str(i), '}=dlmread(''myData'];

	% Handling leading zeros
	if i<10
		comStr=[comStr, '00', num2str(i), '.txt)'''];
	else if i<100
		comStr=[comStr, '0', num2str(i), '.txt)'''];
	else
		comStr=[comStr, num2str(i), '.txt)'''];
	end %if
	
	% Make it so!
	eval(comStr);
end %for

I don't have appropriate datasets, nor a copy of MATLAB on this computer, so you'll probably need to do some tweaking to make this work properly. And some more tweaking to make it work the way you want it to work. Plus some reading.

Good luck, and remember the MATLAB in-program and online documentation!

EDIT: Forgot that when you need to put an apostrophe inside a text string, you need two of them together: '' (and no, not shift-apostrophe). Three of them together is interpreted as: put an apostrophe into the string, then terminate the string.

 

[maverick_starstrider]

I've been going crazy trying to figure this out- I'm not seeing it in the documentation. I have a few hundred files named "data*" where * goes from 000 to 300. The file type is really just a text file without any real type- they aren't .txt. Is there a way to 'batch' import all of them without selecting each one individually and importing? They are all in one folder, and they are all formatted identically as having two columns.

I've been trying all sorts of tricks using data* and the 'importdata' syntax, but I'm stuck. Maybe there is a loop I can try, if you know of any.

Any suggestions?

Yes, I certainly ran into this problem when learning MATLAB. unfortunately I've always found MATLAB really bad for this. It has been pointed out how MATLAB allows you to do this (by writing a script). However, I've always found it more convenient to write a shell script or a c++ program to parse the data into a new MATLAB friendly file rather then deal with how MATLAB scripts these things.

 

[imost]

I'm not sure how I would go about writing a script in C++...

Thanks, MATLABdude, I'll try to get the dlmread function to work for my particular scenario and report back.

 

[imost]

I figured out a way to do this. It's very inelegant, but it definitely suffices:

for i=1:50 (just 50, say)
if i<10
x=['data00',num2str(i)];
else
x=['data0',num2str(i)];
end %if
data(:,2*i-1:2*i) = importdata(x);
!this stores all the data in a matrix, 2 columns of data at a time
end %for

Again, nothing fancy, but it works.

If I want all the first columns on the left side of the 'data' matrix and all the second columns on the right (instead of alternating first, second, first, second..) I suppose I can write to the data matrix like this:

data(:,i:50:50+i)

But I haven't tried that yet.

Thanks for the ideas, everyone!