Segmentation faults with huge loops? (C++)

[Simfish]

So I'm writing a program for a professor (not for class) for research related with spike triggered averages in neurons. Which involves a lot of data analysis.

So basically my code looks like this:
for(int i=0; i<SIZE; i++){
Write a lot of things into arrays in memory (although far fewer items than of size SIZE).
}

So when my SIZE is 1000000/4, the program runs fine, but when the SIZE is 1000000, the program runs into all sorts of weird errors. It crashes on the windows cmd line, while ./a.out on the Linux command line will produce the "segmentation fault" error. What's the best way to get around this?

Thanks!

 

[fatra2]

Hi there,

Don't remember much about c++, but have you tried writing your SIZE in scientific form? Maybe the compiler would prefer that.

Otherwise, it does seem terribly peculiar, specially that integers can be more or less 2 billion.

If you really don't have any other solution, you can nest this for loop inside another, just to start it again and again. I don't believe that it's the best solution, but if it works.

Cheers

 

[jtbell]

Are you declaring the arrays statically (as variables in functions) or dynamically (by declaring only a pointer and then using new() to allocate the memory)?

Static variables are allocated on the program's "stack" which is often limited in size by the operating system. Dynamic variables are allocated in the "heap" which is usually much larger.

 

[rcgldr]

Segmentation fault usually means a bad pointer:

http://en.wikipedia.org/wiki/Segmentation_fault

Can you use a debugger to see what is going on when it fails? Microsoft Visual Studio Express C++ is a free compiler and debugger you can use to test your program with, if you use Windows.

On Intel environments, static variables are pre-allocated in the initialized portion of the data segment, not on the stack. Non-static global variables are pre-allocated in the uninitialized portion of the data segment. Only local variables and alloca() or _alloca() use the stack.

 

[jtbell]

OK, "local variables" was what I meant, not "static." I've been away from extensive reading about C++ for long enough that my terminology has gotten rusty.

 

[D H]

Only local variables and alloc() use the stack.

Are you sure are about alloc()? That usually takes data from the heap, and the heap tends to be considerably larger than the stack.

#define SIZE 1000000
dont_do_this (void) {
   double big_array[SIZE];
   ...
}

That big_array is going to be allocated from the stack on many systems. That big size will cause the stack to overflow. That array is not as big as you ask for, and the failure to size the array properly is often undetected -- until you try to write on protected memory, for example.

If on the other hand you did something like this,

#define SIZE 1000000
dont_do_this_either (void) {
   double kaboom[250000];
   for (int i=0; i<SIZE; ++i) { kaboom[i] = i; }
   ...
}

The array might well be as big as you asked, but you are running past the end.How to avoid the problem?
1. Do not declare large arrays (more than a 1000 elements or so) as automatic local variables. Alternatives are
a. Allocate them dynamically (e.g., new in C++, malloc() in C)
b. Declare them with some other scope. File-scope or static local, for example.

2. No matter how you construct your arrays, never, ever read or write outside their bounds.

 

[rcgldr]

Are you sure are about alloc()?

I was thinking of the function alloca() or _alloca():

http://msdn.microsoft.com/en-us/library/wb1s57t5(VS.71).aspx

I corrected my previous post. I don't know if alloc() ever existed. There some chance it existed in an old Atari ST Alcyon C compiler I used back in 1986, but I never used alloc() or _alloca(), so if I'm remembering it, it's because I saw it in some reference manual or sample code. Some compilers used an alloc.h, so that may be what I'm remembering.

msdn defines an Alloc() function, but it's noted as preview, and I don't know how it's different from malloc:

http://msdn.microsoft.com/en-us/library/dd492420(VS.100).aspx