Memory Allocation Limitations on Windows for Large Arrays in C++

[sid_galt]

When I declare an array of size 24 mb in c++, I have discovered that the result is a runtime error if the declaration is made inside a function. If the declaration is made outside a function, then there is no problem.

The problem exists with both VC++ Express and Cygwin GCC (GCC gives a segmentation fault). There is no problem on Suse Linux.

Does windows have a limitation on the amount of memory that can be allocated using functions or something?

I have 1 GB Ram of which atleast 350 MB is free at all times and 3.2 GHz 2MB L2 cache Pentium IV.

 

[robphy]

It might help if you show some code.
Do you deallocate the memory when you exit the function?

 

[sid_galt]

It might help if you show some code.
Do you deallocate the memory when you exit the function?

Actually the array declared is a static array. Here's the relevant code which generates error.
<code>
void LoadHistory(bool k)
{
//Declare and initialize variables
int matchUser[3000000][2] //The erring variable

fstream fbinary, fmovieratingcount;
fbinary.open(BINARY_FILE_PATH, ios::in | ios::binary);
fmovieratingcount.open(MOVIE_RATING_COUNT, ios::in | ios::binary);

if(!fbinary && !fmovieratingcount)
{
count<<"FILE CAN'T BE OPENED!\a\n";
exit(1);
}

int tmp, movieId;

for(int i = 0; i < 3000000; i++)
{
matchUser[0] = 0;
}

//While loop to read movie ratings
while(!fmovieratingcount.eof() && !fbinary.eof() && net_movie_no < NO_MOVIES)
{
int custId, ratings_no;
int rating;

fmovieratingcount.read((char*) &tmp, sizeof(tmp));
fbinary.read((char*) &movieId, sizeof(movieId));
}

}
</code>

Moving matchUser declaration outside the function results in no error.

<code>
int matchUser[3000000][2] //No error now
void LoadHistory(bool k)
{
//Declare and initialize variables


fstream fbinary, fmovieratingcount;
fbinary.open(BINARY_FILE_PATH, ios::in | ios::binary);
fmovieratingcount.open(MOVIE_RATING_COUNT, ios::in | ios::binary);

if(!fbinary && !fmovieratingcount)
{
count<<"FILE CAN'T BE OPENED!\a\n";
exit(1);
}

int tmp, movieId;

for(int i = 0; i < 3000000; i++)
{
matchUser[0] = 0;
}

//While loop to read movie ratings
while(!fmovieratingcount.eof() && !fbinary.eof() && net_movie_no < NO_MOVIES)
{
int custId, ratings_no;
int rating;

fmovieratingcount.read((char*) &tmp, sizeof(tmp));
fbinary.read((char*) &movieId, sizeof(movieId));
}

}
</code>

 

[robphy]

This may help
http://www.thescripts.com/forum/thread219328.html

 

[MeJennifer]

Looks like you are trying to allocate a big array on the stack.
Stacks have finite sizes in C, usually you can increase the size but it really does not make any sense to create stacks that are that big.

By the way, what are you trying to do with this program? There are exceptions, so that's why I ask, but usually it is completely unnecessary and generally a bad idea to load something into an array that big.

 

[sid_galt]

Thanks robphy.

By the way, what are you trying to do with this program? There are exceptions, so that's why I ask, but usually it is completely unnecessary and generally a bad idea to load something into an array that big.

Playing around with this http://www.netflixprize.com

The 24 mb array is for quick mapping. This is actually a very small array. The actual data requires an array of size 200-300 mb minimum.

 

[MeJennifer]

Playing around with this http://www.netflixprize.com

The 24 mb array is for quick mapping. This is actually a very small array. The actual data requires an array of size 200-300 mb minimum.

I see!

If you want to be fast then don't use array indexing but, wherever possible, use pointer arithmetic.

For instance, in your array initialization routine, you require the program each time to calculate the address instead of just incrementing some pointer.

Keep that in mind when you run your statistical analysis, you could easily improve the speed of such a program by 50% if you minimize the use of array indexing and use pointer arithmetic.

 

[sid_galt]

Thanks, I'll keep that in mind.