Modelling Long Sets of Data: Measuring "Harshness

In summary, the conversation discusses the potential for finding the perfect dataset that exhibits the property of "harshness" by using a string of 500,000 zeros and ones, or 50 sets with assigned values. The speaker also asks about measuring the harshness of other datasets and the feasibility of using this method with audio data. However, it is mentioned that there is no mathematical guarantee for success and that the approach may vary depending on the physical factors involved. The speaker also brings up the possibility of working with spectrums and the challenges of analyzing 3-dimensional data with a ranking. There is also a mention of the ideal result being a way to measure harshness and determine the most harsh sound, but the speaker acknowledges that there is
  • #1
clemon!!
20
0
say i have 500,000 0s or 1s.
say i have 50 such sets, each that i have ranked or assigned a value to - "harshness".

can i then extrapolate - is that the right word - to find the perfect dataset that instantiates the property of harshness?
and can i measure the harshness of other datasets?



thanks for any help - I've asked quite a few dumb questions of the board already :) !
 
Physics news on Phys.org
  • #2


no help - not even anything i can google?
sorry - i keep changing what iwant to be doing haha :)
 
  • #3


With that amount of data? Realistically, it's very unlikely. The data is extremely high dimensional, and yet you have very little of it. Unless the relationship between each data vector and "harshness" is extremely simple (e.g. more ones = more harsh), then you're going to have trouble finding meaningful relationships. Is this audio data of some sort?
 
  • #4


yeah it's audio...
 
  • #5


clemon! said:
yeah it's audio...

Then why is it a string of zeros and ones? That's not a very good way to represent audio for analysis.
 
  • #6


clemon! said:
can i then extrapolate - is that the right word - to find the perfect dataset that instantiates the property of harshness?
and can i measure the harshness of other datasets?

There is no mathematical guarantee that you can accomplish those goals. For example, suppose you assign the property of hashness randomly. Then there is no forumula that would predict the harshness of other datasets.

If you believe there are physical causes for how you rate the harshness of a data set then there might by a way to predict the harshness of future data sets. There are many ways to approach this task and whether a way work depends on the physical facts of the situation not on any universal mathematical laws.

The approaches range from specific phyiscal explanations of harshness to curve fitting approaches or "black box" approaches (such as using simulated neural nets).

Since you are dealing with samples of data, you can't expect to have certainty about any answer you get. So you have to define your goals realistically using the language of probability theory. This is another complicated aspect of the problem.
 
  • #7


well the other [quite odd] thing about this is that i was thinking of mostly working with square waves... that's why 1s and 0s anyway.


but i think i changed my mind and want to work with spectrums. again it'll be a lot of data tho... maybe less than 500,000 cells but now it's not 1s or 0s.

i can export time/ifft to excel with a program called sigview, which is a good start. but this is now 3 dimensional data plus a ranking. and i have no idea how to start looking for a trend in rank... i might in theory be able to reduce the amount of data, but yeah...



ideal result is way of measuring, plus i suppose the most harsh sound. can anyone give me an idea of the leg work involved in this task? i have no maths training but was pretty good at it at high school :)


and yeah, i am aware there's no guarantee that "harshness" can be measured like this :) !
 

1. What is "harshness" and why is it important to measure in long sets of data?

"Harshness" refers to the amount of variability or extreme values present in a dataset. It is important to measure in long sets of data because it can affect the accuracy and reliability of statistical models and predictions based on the data.

2. How is harshness typically measured in long sets of data?

Harshness can be measured using various statistical measures such as standard deviation, range, and interquartile range. These measures provide an indication of the spread of the data and can help identify extreme values.

3. What are some potential consequences of not accounting for harshness in data modelling?

Not accounting for harshness in data modelling can lead to inaccurate predictions, biased results, and incorrect conclusions. It can also affect the generalizability of the model to other datasets.

4. Can harshness be reduced or eliminated in long sets of data?

In most cases, harshness cannot be completely eliminated as it is a natural part of data. However, it can be reduced by identifying and removing outliers or using data smoothing techniques.

5. Are there any limitations to measuring harshness in long sets of data?

One limitation is that harshness measures only provide a general overview of the spread of the data and do not account for the specific patterns or trends within the data. Additionally, the measures may be affected by the size and distribution of the dataset.

Similar threads

I Data Analaysis -- How to choose the best statistical model to use?
    • Set Theory, Logic, Probability, Statistics
Replies
5
Views
1K
I Direct limit of multiverse models of ZFC
    • Set Theory, Logic, Probability, Statistics
Replies
2
Views
1K
How Can Fault Detection Be Modeled in a Water Tank System?
    • Engineering and Comp Sci Homework Help
Replies
7
Views
990
Data sources for Vehicle Dynamics Model validation?
    • Mechanical Engineering
Replies
4
Views
936
I Understanding 2 equivalent formulations of both data set measures
    • Set Theory, Logic, Probability, Statistics
Replies
9
Views
1K
I Uncertainties from linear interpolation
    • Set Theory, Logic, Probability, Statistics
Replies
30
Views
2K
Neural networks as mathematical models of intelligence
    • Computing and Technology
Replies
4
Views
1K
I Are Real Numbers Essential in Scientific Measurements and Models?
    • Classical Physics
3
Replies
85
Views
4K
Misc. Digitizing a Boat Hull: Mac26 Modeling Data
    • DIY Projects
Replies
33
Views
2K
A The Blessings of Dimensionality and Hilbert's 6th Problem
    • Set Theory, Logic, Probability, Statistics
Replies
4
Views
1K

Hot Threads

Recent Insights

Back
Top