Fortran77: Data I/O in big file

[Bistable]

Hi there,

I have generated a huge data file (approx 500MB large) containing a few 100k lines of formatted data (format(1X, 200ES11.4)). As I believe handling such a large file might over and over again (will be opened by an explicit FE code) might slow down the process significantely, I was thinking to split this file into many smaller files that contain certain columns of data (eg. file element1.txt contains columns 1 and 2, element2.txt contains columns 1 and 3, element3.txt contains columns 1 and 4...element150.txt contains columns 1 and 151) and call those when I need them (I only need a few data points during the calculations). I managed to create those files, but I am struggeling to pass the correct data to those files. Pointers might be good, but I do not know if and how they work with fortran 77. Is there a simple way of doing this?

Thanks a lot.

 

[mathmate]

It depends on how you use the file.
If it is a huge stiffness matrix which you have to read from A-Z, then you do not have too many options.
If it is a file from which you extract information ("I only need a few data points during the calculations"), then you may seriously consider using the direct access file capability.
You will have to index the records, and your program will be able to read the required records directly, without a sequential read. Even if you have a file of 500 Mb with 100K+ records, the read is instant, and you can read or write in any order. The only catch is you will have to decide on a fixed record length on creation of the file, and the search is by record number. If the point numbers cannot be related to the record number, then external indexing will be required.
See, for example, a description at:
http://rsusu1.rnd.runnet.ru/develop/fortran/prof77/node162.html
or an example at:
http://rainbow.ldgo.columbia.edu/data/fortranreaddata.html

Note: you do not need to dissect the file. A huge record of length 2K is reasonable, and easier for you to work with. There will be less file handles, less indexes, and less I/O overhead.

 

[minger]

Is there any reason you can't read/write it as an unformatted file? MUCH faster. Also, do not directly operate on pointers if you're performing a complex operation, they are very very slow as well.