Why Does My Perl Script Crash with Large Data Sets on Raspbian?

[Borek]

Disclaimer: the only thing I know about Perl is the language name.

Raspbian on Pi (512 MB RAM).

I have a Perl script that I want to use. And it works - sort of.

Problem is, it works OK for small data sets, but it fails for large ones. When it works, it works for some time and then it spits out the result. In case of known errors (like a wrong file name) it displays an error message (while I don't know Perl I see these are coded in the script). When it doesn't work - it initially works, then just ends, without any messages.

My first odea was that it is limited by the memory. For a large set of data I see it (with top) using 98% of CPU and allocating more and more memory, but when it stops it is at 3% RAM, so just about 15 MB, not that much.

Any ideas what I can do? Is there a way of checking why it crashes? Can it be related to some memory limit per process? Or is there some limit set to amount of memory available for Perl?

I did some blind googling but all I see seems to be suggesting memory should not be a problem here.

 

[Borek]

OK, for now ignore my message.

I did some more checks and it looks like the problem can't be memory related. Data sets are just list of words. Script works for a fairly short data set that I entered manually, but it never works for a more complicated ones, prepared by other script (we are talking 3 words vs 84 words). It means there must be some other problem.

 

[Borek]

Sigh. CRLF vs just LF.

Case closed.

 

[DrDu]

can you post the script and the dataset?

Edit: Just saw that you solved your problem.

 

[LudusRex]

I understand your frustration with this issue. It sounds like you have already done some troubleshooting by checking the memory usage and researching online. However, without knowing the specific details of your Perl script and the data you are using, it is difficult to provide a specific solution. Here are a few suggestions that may help you troubleshoot and resolve the issue:

1. Check for any syntax errors in your script. If there are any, they could be causing the script to fail when it reaches a certain point.

2. Consider using a memory profiler tool to analyze your script's memory usage. This can help you identify any specific areas of the code that may be causing the issue.

3. Make sure your script is properly handling errors and exceptions. It sounds like you are already seeing error messages, but it's possible that there may be some unhandled exceptions that are causing the script to crash.

4. Check the system limits for memory usage per process. It's possible that there may be a limit set that is preventing your script from using more memory.

5. If possible, try running your script on a machine with more memory to see if the issue persists. This can help determine if the issue is indeed related to memory limitations.

Overall, it's important to thoroughly analyze your script and the data you are using to try and pinpoint the source of the issue. I suggest reaching out to a Perl expert or posting on a Perl forum for more specific assistance. Good luck!