How to Efficiently Remove Zeros from 2D Arrays in Python?

[ProPatto16]

Hi there,

I have 1024,1024 arrays (lots of them) which are really roughly 600,800 (it changes) and then buffered by zeros all the way around something like

000000000000000000
000000000000000000
000xxxxxxxxxxxxx000
000xxxxxxxxxxxxx000
000xxxxxxxxxxxxx000
000000000000000000
000000000000000000

i want to delete all the zeros

i can use numpy.all then numpy.delete but i can only make it work with 4 while loops... essentially one from each direction and its just long and untidy.

i can empty all the 0 elements with trim_zeros but then can't find a nice neat way to delete them.

I havnt tried boolean masking yet since i need the element values.

ideas?

thanks

 

[Fightfish]

i can empty all the 0 elements with trim_zeros but then can't find a nice neat way to delete them.

Could you elaborate on what you mean by "a nice neat way to delete them"?

 

[ProPatto16]

Is there a simple way to delete empty elements from a 2d array where you don't have to iterate every row or column?

 

[Mark44]

Is there a simple way to delete empty elements from a 2d array where you don't have to iterate every row or column?

I don't see how that would be possible. You have to inspect each element of the array.

 

[ProPatto16]

when i do it with while loops i have

while numpy.all(array[0]==0)==True:
    Calarray=numpy.delete(array,0,0)

so it checks each line and if its all zeros then it deletes that line.

But it doesn't terminate unless i manually press return.

If i run the while condition prior to executing i get True. if i run it after I've pressed return i then get False
howcome it doenst end on its own?

im doing a while loop like this from top, bottom then transpose, do top, bottom again and transposing back.
i just thought there might be a quicker way but this works beautifully if i can get the loops to end on their own.

Thanks!

 

[Svein]

But it doesn't terminate unless i manually press return.
If i run the while condition prior to executing i get True. if i run it after I've pressed return i then get False
howcome it doenst end on its own?

I cannot see any test for "finished", so there is no reason why it should stop.
Also, while I do not speak Python, it seems to me that you test the same array (array[0]) over and over.

 

[ProPatto16]

My understanding of while loops is that as soon as the condition is false it should terminate. I.e. The false condition is the exit.

The condition tests the first row and if it returns True then deletes it. So it's always checking the first row but each time it's a new first row.

 

[Mark44]

My understanding of while loops is that as soon as the condition is false it should terminate. I.e. The false condition is the exit.

The condition tests the first row and if it returns True then deletes it. So it's always checking the first row but each time it's a new first row.

I'm fairly knowledgeable with python, but not so with numpy.

Here's the code you showed:

while numpy.all(array[0]==0)==True:
    Calarray=numpy.delete(array,0,0)

I'm not sure your while loop is working the way you expect. The all() function takes up to four arguments, with only the first being required. See http://docs.scipy.org/doc/numpy/reference/generated/numpy.all.html for more information. The first argument to all() is an array, NOT a boolean expression as you have.

Your argument to all() is array[0] == 0. As I understand things, array[0] is the first row of your matrix, so it is itself a list or tuple or maybe an array -- I can't tell from the code you posted. In this expression, array[0] == 0, you are comparing an array (or list or whatever) for equality to a number, 0. I don't see how this can work. My guess is that python always evaluates array[0] == 0 to True, even if the row in question has one or more nonzero values.

 

[ProPatto16]

I gave up on that and it works fine in like 5 lines of code with basic indexing.

For anyone that's interested ...

Using numpy.nonzero on a middle row and column can get the indexes of where the zeros start and stop and can convert to array and use min() max() to get the first and last indexes in both row and column direction and just use those indexes to extract a sub array from the original. Don't need to iterate elements etc.

 

[Mark44]

Don't need to iterate elements etc.

But you can be sure that under the covers, that's exactly what is happening -- i.e., each element of a subarray is being inspected.

 

[Michael27]

Are you sure that your first element in the sub-array cannot be zero?
See example below:

000111000
001111100
011111110
111111111
etc

Here you have a full array where you cannot delete any lines.
You can only make the decision once you have iterated through all the lines.

What is very important here is the precondition of the array you like to iterate through, what data is expected and which cases you can drop. Your solution isn't as straight forward as you have pictured it.
In terms of improving your algorithm you could start left to right on the first row. When hitting a non-zero value you then move to the right end and iterate from right to left until you encounter a non zero value. It reduces the number of look-ups. This can be further reduced by checking only the columns which you have not yet located non-zero values for.
The first non zero line would indicate the top row and the first zero line down the array indicates the bottom row plus one.
You can then copy the rectangle from your array and you end up with a new array.

 

[jack action]

Not fluent in Python, but this is the way I would do it:

length = len(array)
for x in range(0, length):
    array[x] = array[x].strip('0')

Ref.: http://www.tutorialspoint.com/python/string_strip.htm

EDIT: Nevermind, I now understand that this is not what you want to accomplish at all.

 

[D H]

when i do it with while loops i have

while numpy.all(array[0]==0)==True:
    Calarray=numpy.delete(array,0,0)

so it checks each line and if its all zeros then it deletes that line.

That won't work, for a number of reasons. One is that the condition is wrong. Another is that you aren't changing array. You can fix this by using

while not numpy.any(array[0]) :
    array = array[1:]

This however is a bad idea. Suppose the first nonzero element is on row 1000. You will be creating and throwing away 999 slices. What you should be doing is finding the index of the first and last rows that contain something other than all zeros, and then finding the first and last columns that contain something other than all zeros. Use basic indexing (not advanced indexing) to make the second part more efficient.

def nonzero_submatrix(array) :
    try :
        il = 0
        while not numpy.any(array[il,:]) : il += 1

        iu = array.shape[0]
        while not numpy.any(array[iu-1,:]) : iu -= 1

        jl = 0
        while not numpy.any(array(il:iu,jl) : jl += 1

        ju = array.shape[1]
        while not numpy.any(array(il:iu,ju-1) : ju -= 1

        return array[il:iu,jl:ju].copy()
        # Alternatively, just use return array[il:iu,jl:ju] (i.e., no copy)

    except Index Error :
        return None

Note that the above avoids making a copy until the very end (and alternatively avoids the copy altogether). It also protects against an all-zero array using EAFP ("Easier to Ask for Forgiveness than Permission").