What you have on Ubuntu is called a kernel panic - usually from corruption of
kernel data often due to a programming error in user space piddling in kernel space.
The linux command
will show you how much memory is in use at
any given time. Normally, a single user systems will have about 90% of memory
free - note that those buffers you see come and go dynamically
From the man7.org man page information on the free(1) command
free displays the total amount of free and used physical and swap
memory in the system, as well as the buffers and caches used by the
kernel. The information is gathered by parsing /proc/meminfo. The
displayed columns are:
total Total installed memory (MemTotal and SwapTotal in
/proc/meminfo)
used Used memory (calculated as total - free - buffers - cache)
free Unused memory (MemFree and SwapFree in /proc/meminfo)
shared Memory used (mostly) by tmpfs (Shmem in /proc/meminfo,
available on kernels 2.6.32, displayed as zero if not
available)
buffers
Memory used by kernel buffers (Buffers in /proc/meminfo)
cache Memory used by the page cache and slabs (Cached and Slab in
/proc/meminfo)
buff/cache
Sum of buffers and cache
available
Estimation of how much memory is available for starting new
applications, without swapping. Unlike the data provided by
the cache or free fields, this field takes into account page
cache and also that not all reclaimable memory slabs will be
reclaimed due to items being in use (MemAvailable in
/proc/meminfo, available on kernels 3.14, emulated on kernels
2.6.27+, otherwise the same as free)
Now you have a tool.
Your matrix is a lot larger than mentioned before. 64 (bits in a signed
integer) * 10000 *10000 is ~6.4GB, with 80 bits in a double precision floating
point varable it is ~8.0GB. Math packages tend to use existing numeric formats
unless you are using extended precision - like in GMP.
The free tool will give you available. If you need more memory you will have to add swap space.
Virtual memory = swap (paging) file size plus memory. This is temporary fix.
Add more RAM is another possibility but has a monetary penalty.
Also consider letting the people who support the problem software packages know
of the issue. This is important in the long run.
Since octave does not exhibit the problem use it instead if the above is too
much of a problem. But still consider reporting the problem.