How could I recognize patterns in streaming data?

In summary: I apologize for not being more clear.In summary, the electronic device sends a signal with a lot of noise. There is no easy way to determine if the signal is closer to a sinusoidal or square function. However, using an FFT and removing frequencies that shouldn't be present can give a better signal-to-noise ratio. If you have no control over the sending device or the modulation, and no control over the channel noise, you have several options to improve the SNR. The data is encoded now using a combination of modulation and encoding. The process can be delayed, but there is likely to be a lot of noise.
  • #1
jonjacson
447
38
I have an electronic device that sends me a signal but there is also a lot of noise.

My question is, in general, How could I identify that pattern in the data I am receiving?

So I would like to read articles or books talking about this topic.

First thing I thought was using Fourier but I was not able to understand properly what I was getting.

Just to put an example, let's say I want to identify when these two patterns occur:

a sinusoidal function

a square function
 
Physics news on Phys.org
  • #2
In general use a filter to eliminate frequencies that shouldn't be present.
 
  • Like
Likes jonjacson
  • #3
You could use an FFT to identify the frequencies present and then remove those you don't want.

An FFT will give you a frequency spectrum of your time-based data..

 
  • Like
Likes Klystron
  • #4
It depends to some degree upon how well characterized is the expected signal. If you are looking for a finite length (in time) signal of known shape the method is pretty easy.
 
  • #5
jonjacson said:
I have an electronic device that sends me a signal but there is also a lot of noise.
It would help us a lot to know more about the device and the communication channel. If you have some control over the sending device, you could choose a modulation scheme that helps the most to the best signal-to-noise ratio (SNR) and rejects the most noise:

https://en.wikipedia.org/wiki/Modulation

If you have no control over the sending device and its modulation, and no control over the channel noise, then you have several options (some already suggested by others) to give you the best SNR possible in the data recovery.

How are the data encoded now?
 
  • #6
First of all, thanks for all replies.

jedishrfu said:
You could use an FFT to identify the frequencies present and then remove those you don't want.

An FFT will give you a frequency spectrum of your time-based data..



Ok, I will read about it. I saw the video and looks an efficient algo.

hutchphd said:
It depends to some degree upon how well characterized is the expected signal. If you are looking for a finite length (in time) signal of known shape the method is pretty easy.

For you, What would be the simplest method to determine if the signal is closer to a sine or closer to a square function?

One of the issues is that the square pattern may be "compressed" or "stretched" so the size is variable.
berkeman said:
It would help us a lot to know more about the device and the communication channel. If you have some control over the sending device, you could choose a modulation scheme that helps the most to the best signal-to-noise ratio (SNR) and rejects the most noise:

https://en.wikipedia.org/wiki/Modulation

If you have no control over the sending device and its modulation, and no control over the channel noise, then you have several options (some already suggested by others) to give you the best SNR possible in the data recovery.

How are the data encoded now?

Thanks, what I need is a demodulator. I can't control de signal.

Well in general for me it is more interesting the mathematical technique than the specific device. There are several devices indeed and it is possible that we change them in the future.
 
  • #7
jonjacson said:
Thanks, what I need is a demodulator. I can't control de signal.
Please tell us much more about the modulation technique, the transmission frequency and bandwidth, and the channel noise characteristics. Seriously, without a much better problem definition, we will be of limited help and you will fail in your project.

Do you have the model number and datasheet for the transmitter that you can post? What kind of signals is it digitizing and transmitting (via RF or IR or Ultrasound or Klingon voice?)? What is the receive signal power? What are the receive signal noise amplitude and spectrum characteristics?

What are you using for a receiver? Please post the model number and datasheet, and show how you are coupling to the output of the receiver.

Please do your best to post as much information as you can so we can help you. Thank you.
 
  • Like
Likes Klystron
  • #8
Is this a real time process? I mean do you need to process the signal immediately or can you have some delay in the output ? How much noise is there likely to be? Really there is a lot of information needed up front...it may be very difficult or fairly mundane. @berkeman is absolutely correct
 

1. How do I identify patterns in streaming data?

Identifying patterns in streaming data involves using data analysis techniques such as statistical analysis, machine learning algorithms, and data visualization tools. These techniques can help you identify trends, anomalies, and correlations in the data.

2. What are the challenges of recognizing patterns in streaming data?

Some of the challenges of recognizing patterns in streaming data include dealing with large volumes of data, handling data in real-time, and ensuring data quality and accuracy. Additionally, the complexity and variability of data can make it difficult to identify meaningful patterns.

3. What tools and technologies can help with recognizing patterns in streaming data?

There are various tools and technologies available for recognizing patterns in streaming data, such as Apache Spark, Hadoop, and Kafka. These tools offer features like real-time data processing, machine learning algorithms, and data visualization capabilities.

4. How can I ensure the accuracy of the patterns identified in streaming data?

To ensure the accuracy of patterns identified in streaming data, it is crucial to have a thorough understanding of the data and its sources. Additionally, using multiple data analysis techniques and validating the results can help increase the accuracy of the identified patterns.

5. How can I use the patterns identified in streaming data for decision making?

The patterns identified in streaming data can provide valuable insights for decision making. By analyzing these patterns, you can gain a better understanding of customer behavior, market trends, and potential risks. This information can be used to make data-driven decisions and improve business strategies.

Similar threads

I Maximum Likelihood to find the original data or estimation directly
    • Set Theory, Logic, Probability, Statistics
Replies
8
Views
1K
B Explain Integration to me, please
    • Calculus
Replies
5
Views
2K
I How To Experimentally Confirm the Wigner-Von Neumann Interpretation
    • Quantum Interpretations and Foundations
2
Replies
52
Views
1K
Other Absolutely terrible pattern recognition skills
    • STEM Academic Advising
Replies
3
Views
2K
A Do outliers exist in categorical data and how can they be detected?
    • Set Theory, Logic, Probability, Statistics
Replies
6
Views
4K
How do streaming services stream? (troubleshooting)
    • Computing and Technology
Replies
7
Views
1K
I How Can One Derive the Diffraction Pattern Formula from a 1D Aperture?
    • Optics
Replies
2
Views
954
I Young's slit experiment with single photons
    • Quantum Physics
3
Replies
81
Views
4K
I About chi-squared and r-squared test for fitting data
    • Set Theory, Logic, Probability, Statistics
Replies
5
Views
7K
A The Blessings of Dimensionality and Hilbert's 6th Problem
    • Set Theory, Logic, Probability, Statistics
Replies
4
Views
1K

Hot Threads

Recent Insights

Back
Top