Creating Library w/4MB Array in UNIX Environment

[Hurkyl]

I want to write a library that has a large object in it.

Specifically, it's a 4 megabyte array, but I'm curious about general objects too.

I want to create a library (on a UNIX machine) that includes this object. I know of three ways I don't like so far:

(1) Construct the object at runtime
It's time consuming, and I'd not like to have to wait every time I run my programs.

(2) Load the object from a separate file at runtime
I don't like having zillions of files lying around -- besides, it seems silly... isn't the point of an archive to hold these things?

(3) Generate a (huge) header file that gets included when building the library.
gcc chokes on it, and it takes forever to compile, though I do get what I want at the end.


Any suggestions?

 

[Hurkyl]

I should add that I'd be happy if I can get it into an object file too, but I don't expect that to be easy.

 

[dduardo]

What about using zlib to compress the array and writing the output into your source code. When you run your code the compressed array will be loaded into memory. Then use zlib to inflate the array in memory.

 

[chroot]

How are you including the array in the header file? I don't see why gcc should choke on it. You could also consider using assembly; you can reserve a large data segment quite easily.

In either case, you really should not put 4 MB on the stack -- you should allocate such memory on the heap.

- Warren

 

[so-crates]

The other option I would consider is, if you say you can "generate" the object at runtime, would it be possible to merely have a function which calculates the value as it is needed?

The only other option that makes sense to me is (2). As for having "zillinos" of data files lying around, why would you need anything more than one file ? Programs and libraries regularly read from data files, so I don't understand your aversion to them.

 

[Hurkyl]

I wrote the array into a text file named "array.inc"

----------------
0x12345678, 0x23456789, 0x3456789a,
0x456789ab, 0x56789abc, 0x6789abcd,
...
----------------

then, in my header file,

----------------
unsigned int array[1<<19] = {
#include "array.inc"
};
----------------

gcc ran on it, but it seemed to be thrashing -- the disk was running pretty heavily. It took several minutes to compile something that included the header. (The system I'm using isn't exactly state of the art...)


What I would most like is to have my header have something like:

extern const int *const array;

and write the bytes to a file that becomes part of the library, and have the pointer linked to the file, if that's at all possible.

 

[Hurkyl]

At the moment, the only way I know how to generate the data is linearly -- if it can be done as random access, it will take a good bit of cleverness.

Programs and libraries regularly read from data files, so I don't understand your aversion to them.

I'm extraordinarily picky. Overly curious too. Basically, it seems to me that there should be some straightforward way to pack my data into an object file, or into a library, so that it can be directly linked with stuff. After all, isn't that what libraries and object files are for?

And if I can do it, it would probably be the more efficient than the alternatives, as well as keeping down the number and types of things I would need to manage.

Thus, despite having workable alternatives, I would like to find out if it can be done this way.

 

[master_coda]

What I would most like is to have my header have something like:

extern const int *const array;

and write the bytes to a file that becomes part of the library, and have the pointer linked to the file, if that's at all possible.

Why haven't you tried doing that? (I'm assuming you haven't, since you only mentioned trying to put the data directly into a header file) Just put the array in a code file, and put an extern declaration in the header file.

Of course, gcc might just choke on the code file anyway, I don't know if it'll be too large. But that approach should work, even if you have to use a tool other than gcc to construct the object file.

 

[Hurkyl]

Hrm, I'd expect gcc to choke, even if it was just the array. However,

even if you have to use a tool other than gcc to construct the object file.

What other tools do I have available to do this? At the moment, the only way I know how to make it into an object file is a massive C source file.

 

[NoTime]

What I would most like is to have my header have something like:

extern const int *const array;

and write the bytes to a file that becomes part of the library, and have the pointer linked to the file, if that's at all possible.

Sure, compile the array seperately with an export directive.
Then the extern gets resolved at link time.
You can put it in a library or use the .o.

 

[master_coda]

What other tools do I have available to do this? At the moment, the only way I know how to make it into an object file is a massive C source file.

You could try using assembly language (I think that was already suggested). An assembler would probably be able to handle a large data block much better than gcc would handle a huge array initializer, although that's only a guess.

It wouldn't be that difficult; you'd just have to declare a initialized data segment with an make sure that a pointer to it is exported.

 

[Hurkyl]

If only I remembered any assembly.

 

[master_coda]

If only I remembered any assembly.

Well, you wouldn't actually have to write any real assembly language; you'd just have to use a couple of assembler directives.

For example, in NASM you'd just need something like:

global array
array:
dd 0x12345678, 0x23456789, ...
dd 0x456789ab, 0x56789abc, ...
...

Depending on what assembler you have available, the commands might be slightly different, but basically that's all you'd need.