MATLAB and averaging without NaN

[member 428835]

Hi PF!

Let's say we have a vector $x = [1, \,2,\, 3, \,20,\, 4,\, 5,\, 30,\, 6]$. I am trying to loop through and find the magnitude of the derivative of each $i^{th}$ element with unitary spacing, and if that derivative is sufficiently high, I make that element NaN and continue.

Example: Let's call the derivative of the $i^{th}$ element $d_i\equiv|d_{i-1}-d_{i+1}|/2$. Then $d_2 = |3-1|/2=1$. Similarly, $d_3=|20-2|/2=9$. However, if $d_i$ is greater than, say $5$, I want to make $x_{i+1}=NaN$ and recalculate the derivative using now $x_{i-1}$ and the next non-NaN number. So, as a first loop through $d_3=|20-2|/2=9$ which is above 5, so $x_4=NaN$ and thus $x = [1, \,2,\, 3, \,NaN,\, 4,\, 5,\, 30,\, 6]$. Then recalculating $d_3=|4-2|/2=1$. Then $d_4=|4-3|/2=1/2$. Also, $d_5=|3-5|/2=2$.

If we continue $x = [1, \,2,\, 3, \,NaN,\, 4,\, 5,\, NaN,\, 6]$. So far what I have is

x = [1 2 3 20 4 5 30 6];
for i = 2:length(x)-1
        if isnan(x(i-1))==0
            d(i) = abs(x(i+1)-x(i-1))/2;
            if d(i) > 5
                x(i+1) = NaN;
            end
        else
            ?
        end
end

I don't know how to write what I've described above, and I could be way off here. Any help would be so awesome!

 

[jedishrfu]

I would put print statements in the iteration and step through your code and at each step you can verify whether MATLAB is doing what you want if not then you know right where to fix it.

 

[member 428835]

My solution for sake of others:

    %% smooths outliers to avoid clusters
    critslope = 3;% critical slope filter
    for i = 2:length(x)-1
        if isnan(x(i-1))==0
            slope(i) = abs(x(i+1)-x(i-1))/(2*dz);
            if slope(i) >= critslope
                x(i+1) = NaN;
            end
        count = 1;% reset count in case multiple NaN packets
        else
            count = count+1;% increase count since NaN detected
            slope(i) = abs(x(i+1)-x(i-count))/(2*dz);
            if slope(i) >= critslope
                x(i+1) = NaN;
            end
        end
    end
 
    %% rewrite NaN as previous value
    for i = 2:length(x)-1
        if isnan(x(i))==1
            x(i)=x(i-1);
        end
    end

 

[jedishrfu]

How did you figure out what was wrong?

 

[member 428835]

How did you figure out what was wrong?

It's not so much what was wrong, it was more just trying to get something in the "?" statement of my first post. After lot's of thinking about it the above is what came out. I googled and googled but didn't find anything, else I'd post a link.

I definitely followed your advice and printed the final $x$ results of the simple vector above, knowing exactly what the result should look like, and modified the conditions and the "count" until it worked.

At any rate, this should smooth data assuming the first data point is not an outlier.