Monitoring a CSV file that is continuously updated

[hilbert2]

TL;DR

Trying to produce a Python script that follows sudden changes in a CSV file updated in real time.

Suppose some program writes its output into a CSV table and is adding new rows to it all the time. How would I write a Python or Bash script that monitors the last element of the last row of this CSV file in real time and makes a warning sound if it exceeds some limiting value? This would be used to notice if the result of some numerical calculation diverges, or if my RTL-SDR radio dongle notices that a measurable radio signal appears at a particular frequency.

 

[Borg]

I think that you would have timing issues trying to assure one program always sees the exact last line written by a different program. Have you looked into a protobuf architecture? It's a bit complex if you haven't worked with it before but works well in these situations.

https://developers.google.com/protocol-buffers/docs/pythontutorial

 

[jedishrfu]

Stack has an example of a file being read while it is being written. Some OS platforms may not allow this so its likely not portable.

https://stackoverflow.com/questions/60655818/read-from-file-while-it-is-being-written-to-in-python

It's basically a pipeline which is common in Linux file processing:

program-a | program-b | program-c

The output of program-a is passed as stdin input to program-b and program-b's stdout is passed to program-c.

 

[hilbert2]

Thanks for the tips. I just succeeded in writing a messy Unix bash script that does what I intended, and I will also attempt to do this with protobuf and Python.

The .sh file contains

#!/bin/ksh

while [ 1 == 1 ]
do
 line=$(tail -n 1 log.csv)
 count=`echo $line | awk -F, {'print NF'}`
 i=9
 str=`echo $line | cut -d, -f${i}`
 echo $str;
 if [ $str \> -5 ]
 then
    echo -ne '\007'
 fi
 sleep 1;
done

and it assumes that the last column is the 9:th one. The command "echo -ne '\007'" produces a beep sound from the computer, and this is done when the value of the last numerical element in the CSV exceeds -5.

 

[jedishrfu]

Protobuf means you're using messages passed between programs not files being written. Protobuf is a good agnostic way of sending data as a message from a program based on one language say Java to another program based on say python. The protobuf message bindings will insure the data is passed correctly.

Here's a brief tutorial on Google protocol buffers (aka protobuf) that describes it better with pros and cons and some examples:

https://www.tutorialspoint.com/protobuf/protobuf_quick_guide.htm

ZeroMQ performs a similar interprogram communication function and you can read about it here:

https://zeromq.org/

In general, message passing via Protobuf or ZeroMQ is a better choice than using file writing/reading where buffering and parallel processing might wreak havoc on your implementation.

 

[Tom.G]

How about reading the file size to see if it has changed?
Or the last Write Time?

Depending on which Operating System, they may not be updated until the file is closed though.

I know SOMETHING is possible in Windows because I use an Editor (Textpad) that each time it re-gains focus, it notifies if an open file has been updated.

Cheers,
Tom

 

[jedishrfu]

Often file data isn't updated until the buffer is flushed by the writing program.

 

[pbuk]

As others have mentioned there are better ways of communicating between programs than writing to the filesystem, but if that is all you have and you are on Linux then you probably want to use inotify.

 

[Vanadium 50]

Often file data isn't updated until the buffer is flushed by the writing program.

And sometimes not even then - the disk reports an update but the data is still in the buffer. Written sometimes means "actually written" and sometimes means "intended to be written when we get around to it."

 

[hilbert2]

I used that Linux .sh code to monitor the output of the rtl_power application and to make an alarm sound when it notices a radio signal stronger than background noise at some particular frequency. When I set that to follow the 80-81 MHz frequency band that local taxis around here use for radio communication, it seemed to notice when a taxi was parked in the front of the building. The bash script needed to be modified to convert the decibel radio intensities to positive integers so that they could more easily be compared to some threshold value. Didn't seem to run into problems with trying to read an output file where the last row is incompletely written.