Memmapfile Memory Usage, what's wrong?

[S. Moger]

Hello,

In a function I'm mapping a ~30MB file like this:

raw = memmapfile( ...
    sFileName, ...
    'Offset', iOffsetToData, ...
    'Format', { ...
        'int32' [1 3] 'a'; ... 
      
        'uint16' [1 1] 'b'; ...
        
        'uint8' [1 1] 'c'; ...
                                                                      
        'uint8' [1 1] 'chClassification'; ...
                                 
        
        'int8' [1 1] 'chScanAngleRank';  ...
                                  
        
        'uint8' [1 1] 'chFileMarker';  ...
                                  
        
        'uint16' [1 1] 'rvUserBitField'; ...
        
        'double' [1 1] 'o'; ... 
                                
        } ...
    );

All good.

now I want to read the data so I write:

myData=raw.Data;

Now there are huge problems. Matlab goes from 300MB to 2.5GB of memory usage.

whos mydata returns 1.2GB. (How is this even possible?)

I do some computing and return a matrix that should be about 40MB.

End of function.So, fine, I thought, the function is finished and MATLAB deallocates the memory it used. But no, it doesn't. I tried clearing it up manually in the function, but nothing happens. I also tried reading raw.Data(k).a in a loop, but Matlab appears to create the whole raw.Data array anyway.

My biggest concern is that most computers won't handle the huge mem requirement increase. It is just too much. How can I fix it?

I'm sure I can skip this memmapfile-method altogether, and use some read file function, but it's slower. Is it possible to read custom structures from files (as with the ease of memmapfile)?

 

[mmwave]

Hello,

Thank you for bringing this issue to my attention. Based on the information provided, it seems like there may be a memory leak in the memmapfile function that you are using. This can occur when the allocated memory is not properly deallocated after the function has finished executing.

One possible solution to this issue is to manually clear the allocated memory after you have finished using it. You can do this by using the clear function in MATLAB, specifying the variable name that holds the allocated memory. For example, you can try using the following code after you have finished using the memmapfile function:

clear raw

This should free up the allocated memory and prevent any potential memory leaks.

Additionally, you can also try using the fclose function after you have finished using the memmapfile function. This function closes the file associated with the memmapfile object and also frees up any allocated memory. You can use it like this:

fclose(raw);

I hope this helps to resolve your issue. If the problem persists, I would recommend reaching out to the MathWorks support team for further assistance. They may be able to provide more specific guidance and help you find a solution that works for your specific needs. Best of luck with your project!

 

[oswaler]

Hello,

Thank you for sharing your concerns about the memory usage with memmapfile. It seems like the issue is with how the data is being stored and accessed. First, it is important to note that when you use memmapfile, the data is not actually loaded into memory until it is accessed. So, in your case, when you call myData = raw.Data, the entire file is being loaded into memory.

To address the issue of the large memory usage, you can try using the 'Writable' option in memmapfile. This will allow you to modify the data without loading it all into memory. Additionally, you can try using the 'Repeat', 'Skip', and 'Stride' options to only access certain parts of the data, rather than loading it all at once.

If you are concerned about the memory usage on other computers, you can try using a smaller data type for your variables in the memmapfile format. For example, using uint8 instead of int32 for the 'a' variable can significantly reduce the memory usage.

In terms of reading custom structures from files, it is possible to do so using the fread function. However, it may not be as efficient as using memmapfile. You can also look into using the load function to load data from a file into a structure.

I hope this helps and good luck with your research.