How could I recognize patterns in streaming data?

[jonjacson]

I have an electronic device that sends me a signal but there is also a lot of noise.

My question is, in general, How could I identify that pattern in the data I am receiving?

So I would like to read articles or books talking about this topic.

First thing I thought was using Fourier but I was not able to understand properly what I was getting.

Just to put an example, let's say I want to identify when these two patterns occur:

a sinusoidal function

a square function

 

[mathman]

In general use a filter to eliminate frequencies that shouldn't be present.

 

[jedishrfu]

You could use an FFT to identify the frequencies present and then remove those you don't want.

An FFT will give you a frequency spectrum of your time-based data..

 

[hutchphd]

It depends to some degree upon how well characterized is the expected signal. If you are looking for a finite length (in time) signal of known shape the method is pretty easy.

 

[berkeman]

I have an electronic device that sends me a signal but there is also a lot of noise.

It would help us a lot to know more about the device and the communication channel. If you have some control over the sending device, you could choose a modulation scheme that helps the most to the best signal-to-noise ratio (SNR) and rejects the most noise:

https://en.wikipedia.org/wiki/Modulation

If you have no control over the sending device and its modulation, and no control over the channel noise, then you have several options (some already suggested by others) to give you the best SNR possible in the data recovery.

How are the data encoded now?

 

[jonjacson]

First of all, thanks for all replies.

You could use an FFT to identify the frequencies present and then remove those you don't want.

An FFT will give you a frequency spectrum of your time-based data..

Ok, I will read about it. I saw the video and looks an efficient algo.

It depends to some degree upon how well characterized is the expected signal. If you are looking for a finite length (in time) signal of known shape the method is pretty easy.

For you, What would be the simplest method to determine if the signal is closer to a sine or closer to a square function?

One of the issues is that the square pattern may be "compressed" or "stretched" so the size is variable.

It would help us a lot to know more about the device and the communication channel. If you have some control over the sending device, you could choose a modulation scheme that helps the most to the best signal-to-noise ratio (SNR) and rejects the most noise:

https://en.wikipedia.org/wiki/Modulation

If you have no control over the sending device and its modulation, and no control over the channel noise, then you have several options (some already suggested by others) to give you the best SNR possible in the data recovery.

How are the data encoded now?

Thanks, what I need is a demodulator. I can't control de signal.

Well in general for me it is more interesting the mathematical technique than the specific device. There are several devices indeed and it is possible that we change them in the future.

 

[berkeman]

Thanks, what I need is a demodulator. I can't control de signal.

Please tell us much more about the modulation technique, the transmission frequency and bandwidth, and the channel noise characteristics. Seriously, without a much better problem definition, we will be of limited help and you will fail in your project.

Do you have the model number and datasheet for the transmitter that you can post? What kind of signals is it digitizing and transmitting (via RF or IR or Ultrasound or Klingon voice?)? What is the receive signal power? What are the receive signal noise amplitude and spectrum characteristics?

What are you using for a receiver? Please post the model number and datasheet, and show how you are coupling to the output of the receiver.

Please do your best to post as much information as you can so we can help you. Thank you.

 

[hutchphd]

Is this a real time process? I mean do you need to process the signal immediately or can you have some delay in the output ? How much noise is there likely to be? Really there is a lot of information needed up front...it may be very difficult or fairly mundane. @berkeman is absolutely correct