MATLAB linear optimization out of memory

Click For Summary

Discussion Overview

The discussion revolves around encountering an "out of memory" error while attempting to solve a linear optimization problem in MATLAB, specifically using the "fmincon" function with a large-dimension matrix. Participants explore potential solutions and considerations related to memory management in this context.

Discussion Character

  • Technical explanation
  • Debate/contested
  • Mathematical reasoning

Main Points Raised

  • One participant suggests using sparse matrix functions to manage memory more effectively.
  • Another participant raises the importance of checking for unnecessary copies of matrices or temporary arrays that could contribute to memory issues.
  • A participant describes their specific use case involving "fmincon" and the dimensions of the matrices involved, noting that one matrix is in full representation while another is in sparse representation.
  • There is a mention that even when attempting to convert the problematic matrix to sparse format, the memory issue persists, possibly due to the minimizer expanding the matrix.
  • One participant notes that the size of the matrix structure is not inherently problematic, but the generation of multiple copies during computation could be a contributing factor to the memory error.
  • There is a presumption regarding the dimensions of another matrix involved in the optimization process, indicating a concern about its size as well.

Areas of Agreement / Disagreement

Participants express various strategies and considerations regarding memory management, but no consensus is reached on a definitive solution to the memory issue. Multiple competing views and approaches remain present in the discussion.

Contextual Notes

Participants have not fully explored the implications of matrix dimensions and their representations on memory usage. The discussion includes assumptions about the sizes of matrices and the potential for creating multiple copies during optimization processes, but these aspects remain unresolved.

OhMyMarkov
Messages
81
Reaction score
0
Hello everyone!

I'm trying to solve a linear optimization problem with a large-dimension matrix, MATLAB says memory is full. What to do?

Thanks!
 
Physics news on Phys.org
OhMyMarkov said:
Hello everyone!

I'm trying to solve a linear optimization problem with a large-dimension matrix, MATLAB says memory is full. What to do?

Thanks!

See if you can use sparse matrix functions?

Check that you are not creating unnecessary copies of matrices and or temporary arrays?

How large is the large-dimension matrix?

CB
 
I'm having the Out of memory error whenever I call "fmincon" as follows:x = fmincon("at"ftest, zeros(24583,1), -eye(24583), zeros(24583,1), PHI, S(:,1,1);Now, ftest is the L1 norm, PHI is the product of two matrices, a sampling 300x403 random matrix and a 403x24583 Gabor basis dictionary. Please note that one of the sampling matrix is in "full" representation while the larger matrix is in "sparse" representation. I also tried making PHI explicitly in sparse format to check but this didn't solve the problem?

It seems that the minimizer expands PHI.

Thank you for your time.

OMM
 
OhMyMarkov said:
I'm having the Out of memory error whenever I call "fmincon" as follows:x = fmincon("at"ftest, zeros(24583,1), -eye(24583), zeros(24583,1), PHI, S(:,1,1);Now, ftest is the L1 norm, PHI is the product of two matrices, a sampling 300x403 random matrix and a 403x24583 Gabor basis dictionary. Please note that one of the sampling matrix is in "full" representation while the larger matrix is in "sparse" representation. I also tried making PHI explicitly in sparse format to check but this didn't solve the problem?

It seems that the minimizer expands PHI.

Thank you for your time.

OMM

That would not in itself be a problem (it is an ~79 MB structure), it is the fact that the system will generate multiple copies of arrays of that size in doing its work.

I presume the dimensions of S are not too great ...

CB
 

Similar threads

MATLAB Optimizing Memory Usage in Matlab: Tips and Techniques
  • · Replies 9 ·
Replies
9
Views
3K
MATLAB Matlab's numeric solution to det of Matrix is incorrect
  • · Replies 5 ·
Replies
5
Views
4K
MATLAB FEM, Matlab and the modes of an element
  • · Replies 32 ·
2
Replies
32
Views
4K
MATLAB Can I calculate the covariance matrix of a large set of data?
  • · Replies 5 ·
Replies
5
Views
2K
MATLAB MATLAB: Fluid Flow - Curl of a Vector Field
  • · Replies 2 ·
Replies
2
Views
3K
Graduate How to optimize a parameter that is the index of a summation?
  • · Replies 5 ·
Replies
5
Views
2K
MATLAB MATLAB GPU Cross-Correlation Code Fails with OOM on 8 GB Supercomputer
  • · Replies 4 ·
Replies
4
Views
2K
MATLAB Problem with Runge-Kutta script file
  • · Replies 4 ·
Replies
4
Views
2K
MATLAB [~,~,~,~,~,~,~,~,spec] Too many output arguments...
  • · Replies 5 ·
Replies
5
Views
3K
MATLAB LMI in Matlab scaling to improve condition number?
  • · Replies 2 ·
Replies
2
Views
2K