Data collected from different devices: how to combine for analysis?

  • I
  • Thread starter Mikki123
  • Start date
  • Tags
    Data
  • #1
Mikki123
4
3
Hi Everyone,
I'm working on a project where I have current values from three different devices when there is no arc and an arc generated by an arc generator. When I plot them, they all look different since the data is from different devices. Is there anything I can do to make them comparable, like make them look similar, so that I can perform further analysis?
 
Physics news on Phys.org
  • #2
In your plot where y is current values, what is x ?
 
  • #3
it is just indexes starting from 0 to the number of samples
 
  • #4
However you may get statistics, e.g. average, sandard deviation, as mathematical treatment, Number has no physical meaning. You had better pick up some phisical quantitiy from the samples for plot,e.g. same divice with different physical condition, same condition with different devices.
 
  • Like
  • Informative
Likes Vanadium 50, russ_watters and mcastillo356
  • #5
All good. I will try doing that. Thankyou :smile:
 
  • #6
Mikki123 said:
it is just indexes starting from 0 to the number of samples
Are they collected over time? Then x is time, isn't it? Are these samples collected in regular periods? Then it is just a matter of knowing frequency, no?
 
  • Like
Likes russ_watters
  • #7
Hi Borek,
I just have a single column of 800,000 current values. The x values should be time, I suppose. I have the same from three different devices. But while plotting it, they looked so different. I wanted to train my machine learning model with this data for further processing. Since the data all looks so different, I'm getting such poor performance. Do you think the Fourier transformation for all three will make them look similar so that I can better train my model. I'm looking for any kind of preprocessing apart from normalization and feature extraction
 
  • #8
Mikki123 said:
The x values should be time, I suppose.
You should probably know this (?????) It matters that they be equally spaced with no " jitter ".
Then look at (the difference) fourier transform to find interferences.
When you say feature extraction what exactly do you mean?
Why do you expect similar results?
 
  • Like
Likes russ_watters
  • #9
Good data requires a detailed understanding of what it really means; how the measurement instrument works; what it is REALLY measuring. Good lab work is more about experiment design and selecting and/or researching the instrumentation than it is about collecting the data. This would be especially true if you are trying to represent the same physical quantity with different methods.

Without this prior engineering it is likely that the data is meaningless, or has unknown meaning. Bad data can be combined however you like. Garbage in, garbage out applies from the very beginning of experimentation and analysis.

A set of numbers isn't data, it's just numbers. Data has an associated meaning and context.

You will not get useful answers from us if we don't know, in detail, what you are measuring, why, and how.
 
  • Like
Likes gleem, Vanadium 50, hutchphd and 2 others

1. What are the challenges of combining data from different devices?

Combining data from different devices often presents challenges such as varying data formats, different sampling rates, or discrepancies in data quality and precision. Additionally, time synchronization issues can occur if devices do not share a unified time reference. Addressing these challenges typically requires data preprocessing steps like normalization, resampling, and time alignment to ensure that the data sets are compatible for analysis.

2. How do you handle different data formats when combining data?

To handle different data formats, it is crucial to first identify the format and structure of each dataset. Common formats include CSV, JSON, XML, and proprietary binary formats. Conversion tools or programming libraries can be used to transform these data into a common format. For instance, Python’s pandas library can be used to convert various data formats into a unified DataFrame structure, which simplifies further analysis.

3. What methods are used to ensure data quality and consistency?

Ensuring data quality and consistency involves several techniques such as data cleaning, which can include removing or correcting outliers and handling missing data through imputation. Consistency checks involve verifying that data across devices adhere to expected ranges or distributions. Additionally, employing data validation rules, such as checking for logical inconsistencies (e.g., a temperature reading that exceeds physical limits), is crucial in maintaining the reliability of the combined dataset.

4. How can data from different devices be synchronized?

Data synchronization can be achieved by aligning timestamps across all devices, assuming each device records the time of data capture. If devices are not time-synced, techniques such as interpolation can be used to estimate data points at uniform time intervals. In some scenarios, event-based synchronization might be necessary, which involves aligning data streams based on the occurrence of specific events noted by all devices.

5. What tools and software are recommended for combining data from multiple devices?

Several tools and software are beneficial for combining data from multiple devices. For general data manipulation and analysis, Python libraries like pandas and NumPy are highly recommended. For more complex data synchronization and time series analysis, software like MATLAB or specialized tools like LabVIEW might be more appropriate. Additionally, database management systems like SQL or NoSQL can be used to handle large datasets efficiently, facilitating easier data integration and retrieval.

Similar threads

  • Programming and Computer Science
Replies
11
Views
995
  • General Math
Replies
1
Views
814
  • STEM Educators and Teaching
Replies
5
Views
664
  • Other Physics Topics
Replies
3
Views
3K
  • Engineering and Comp Sci Homework Help
Replies
7
Views
992
  • Other Physics Topics
Replies
3
Views
995
  • Other Physics Topics
Replies
9
Views
3K
  • MATLAB, Maple, Mathematica, LaTeX
Replies
1
Views
2K
  • Computing and Technology
Replies
5
Views
927
  • Other Physics Topics
Replies
18
Views
2K
Back
Top