Easy: importing a batch of data into MATLAB

In summary, the author is trying to import a bunch of text files that have numerical data in two columns, but is having trouble. He recommends using the MATLAB in-program and online documentation.
  • #1
imost
3
0
I've been going crazy trying to figure this out- I'm not seeing it in the documentation. I have a few hundred files named "data*" where * goes from 000 to 300. The file type is really just a text file without any real type- they aren't .txt. Is there a way to 'batch' import all of them without selecting each one individually and importing? They are all in one folder, and they are all formatted identically as having two columns.

I've been trying all sorts of tricks using data* and the 'importdata' syntax, but I'm stuck. Maybe there is a loop I can try, if you know of any.

Any suggestions?
 
Physics news on Phys.org
  • #2
imost said:
I've been going crazy trying to figure this out- I'm not seeing it in the documentation. I have a few hundred files named "data*" where * goes from 000 to 300. The file type is really just a text file without any real type- they aren't .txt. Is there a way to 'batch' import all of them without selecting each one individually and importing? They are all in one folder, and they are all formatted identically as having two columns.

I've been trying all sorts of tricks using data* and the 'importdata' syntax, but I'm stuck. Maybe there is a loop I can try, if you know of any.

Any suggestions?

Why not just write a script file that loops through all the files in the directory and imports them if they meat your criteria.
 
  • #3
John Creighto said:
Why not just write a script file that loops through all the files in the directory and imports them if they meat your criteria.

This may be easy for you, and easy for me, but not so easy for someone new(ish) to MATLAB.

If you're just starting out, MATLAB's (internal) documentation is pretty good, and the online documentation is even better (and gives quite helpful suggestions for related functions--think Amazon without trying to sell you something). For instance, typing the following brings up the internal help for the dlmread function, delimited read from ASCII text file:
>>help dlmread

The online documentation for the dlmread function would be at:
http://www.mathworks.com/access/helpdesk/help/techdoc/ref/dlmread.html

Okay, basics aside, you can harness the interpretive nature of MATLAB using the eval function, which evaluates a text string (which you can generate on the fly!) Okay, let's assume that you want to import myData000.txt to myData999.txt (numerical data stored without a header row, and in 2 columns--separated by commas--with 100 rows--separated by carriage return). The following code would probably work, and produce a cell array (myData) with 1000 entries, each containing a 2x100 matrix of values.

Code:
for i=0:999
	% The setup
	comStr=['myData{', num2str(i), '}=dlmread(''myData'];

	% Handling leading zeros
	if i<10
		comStr=[comStr, '00', num2str(i), '.txt)'''];
	else if i<100
		comStr=[comStr, '0', num2str(i), '.txt)'''];
	else
		comStr=[comStr, num2str(i), '.txt)'''];
	end %if
	
	% Make it so!
	eval(comStr);
end %for

I don't have appropriate datasets, nor a copy of MATLAB on this computer, so you'll probably need to do some tweaking to make this work properly. And some more tweaking to make it work the way you want it to work. Plus some reading.

Good luck, and remember the MATLAB in-program and online documentation!

EDIT: Forgot that when you need to put an apostrophe inside a text string, you need two of them together: '' (and no, not shift-apostrophe). Three of them together is interpreted as: put an apostrophe into the string, then terminate the string.
 
Last edited:
  • #4
imost said:
I've been going crazy trying to figure this out- I'm not seeing it in the documentation. I have a few hundred files named "data*" where * goes from 000 to 300. The file type is really just a text file without any real type- they aren't .txt. Is there a way to 'batch' import all of them without selecting each one individually and importing? They are all in one folder, and they are all formatted identically as having two columns.

I've been trying all sorts of tricks using data* and the 'importdata' syntax, but I'm stuck. Maybe there is a loop I can try, if you know of any.

Any suggestions?


Yes, I certainly ran into this problem when learning MATLAB. Unfortunatly I've always found MATLAB really bad for this. It has been pointed out how MATLAB allows you to do this (by writing a script). However, I've always found it more convenient to write a shell script or a c++ program to parse the data into a new MATLAB friendly file rather then deal with how MATLAB scripts these things.
 
  • #5
I'm not sure how I would go about writing a script in C++...

Thanks, MATLABdude, I'll try to get the dlmread function to work for my particular scenario and report back.
 
  • #6
I figured out a way to do this. It's very inelegant, but it definitely suffices:

for i=1:50 (just 50, say)
if i<10
x=['data00',num2str(i)];
else
x=['data0',num2str(i)];
end %if
data(:,2*i-1:2*i) = importdata(x);
!this stores all the data in a matrix, 2 columns of data at a time
end %for

Again, nothing fancy, but it works.

If I want all the first columns on the left side of the 'data' matrix and all the second columns on the right (instead of alternating first, second, first, second..) I suppose I can write to the data matrix like this:

data(:,i:50:50+i)

But I haven't tried that yet.

Thanks for the ideas, everyone!
 

1. How do I import a batch of data into MATLAB?

To import a batch of data into MATLAB, you can use the importdata function. This function allows you to specify the file path and name, as well as the type of data (such as a text or spreadsheet file) you want to import. You can also use the uiimport function to import data using a graphical user interface.

2. What types of data can I import into MATLAB?

You can import a variety of data types into MATLAB, including text files, spreadsheet files, image files, and audio files. You can also import data from databases, web services, and other sources using the appropriate functions or toolboxes.

3. How do I specify the format of my data when importing into MATLAB?

The importdata function automatically detects the format of your data when you specify the file path and name. However, if you need to specify the format manually, you can use the dlmread function for text files or the xlsread function for spreadsheet files. These functions allow you to specify the row and column ranges of the data you want to import.

4. Can I import data from multiple files at once?

Yes, you can use the importdata function to import data from multiple files at once. Simply specify a cell array of file paths and names as the input argument. You can also use the dir function to create a list of file names and then loop through the list to import each file.

5. How do I handle missing or invalid data when importing into MATLAB?

If your data contains missing or invalid values, you can use the NaN (Not a Number) value in MATLAB to represent these values. You can also use the importdata function with the 'FillValues' option to automatically replace missing or invalid values with a specific value. Additionally, you can use the isnan function to check for and handle missing values in your data after importing.

Similar threads

  • MATLAB, Maple, Mathematica, LaTeX
Replies
2
Views
1K
  • MATLAB, Maple, Mathematica, LaTeX
Replies
12
Views
3K
  • Engineering and Comp Sci Homework Help
Replies
7
Views
1K
  • MATLAB, Maple, Mathematica, LaTeX
Replies
3
Views
2K
  • MATLAB, Maple, Mathematica, LaTeX
Replies
8
Views
989
  • MATLAB, Maple, Mathematica, LaTeX
Replies
1
Views
3K
  • Computing and Technology
Replies
15
Views
1K
  • MATLAB, Maple, Mathematica, LaTeX
Replies
12
Views
3K
  • MATLAB, Maple, Mathematica, LaTeX
Replies
5
Views
2K
  • Programming and Computer Science
Replies
6
Views
1K
Back
Top