viraltux said:
Well, I am not sure I understand the problem you are trying to solve here, one thing is to apply transformations to data so that it looks to us like something familiar e.g. a Normal distribution; we do that all the time, and I assume this is what NIST means. Right? So for instance, if X follows a Normal distribution with μ>>0 and μ>>σ, and you have Z = X^2 then NIST would recommend you don't share Z but rather SQRT(Z) explaining the transformation you did, in this case SQRT. Am I right on what NIST asks?
No, not quite (Though I may be under-reading them) ; They are being very simplistic. They only want the mu and sigma of a data-set reported. Period.
For example 31.412(5) would have a mu of 31.412 and a sigma of 0.005. (The parenthesis references the significance of the ls-digit..., and represents 1 sigma)
NIST does allow one to report a two sigma mark, etc, using special notation -- but essentially, it's the same thing -- a mere number. One would report the mu and sigma of Z in your example; and consider what operations the client would perform on it -- given that, we could adjust mu and sigma that we report in order to reduce the error the client will experience using our number.
But then it seems like if you were trying to find a way to calculate the most similar Normal distribution to something which is not Normal in which case... this is weird; for instance, no matter how you tune μ and σ you will never get anything close to an Exponential distribution.
Yes, quite true. However, I am operating only on "assumed" Gaussians reported by someone else; (Thankfully NOT exponenetial!) and the only operations I am performing on it are add subtract multiply and divide. For add and subtract, the operation does a convolution on the data elements making the result "look" more and more Gaussian.
Multiplication for items with σ << μ also happens to look very Gaussian...
However, multiplication and division which don't obey the above relationship introduce serious Kurtosis and increasing skew. (A problem...)
Inside my calculator, I only wish to improve the result of doing a series of multiplications and additions compared to the standard of assuming pure Gaussians at every step.
If I wanted to do sin( statistic ), then I have an issue about propagation of error; but as we have seen, the typical industry standard covariance approach is missing an important term... It is generally accurate for typical operations, but it isn't robust and totally fails in some. I don't want a total failure, *ever*; just graceful degradation...
More complicated (typical) operations like computing sin( statistic ) can be done using a polynomial, with coefficients having NIST style values nnnnn(ssss); and then, the basic add subtract multiply divide operations of the hypothetical better calculator will keep track of the error and automatically by +.-,*,/ , properly tracing the error regardless of input; it keeps track, also, of the truncation error :) making the approximation polynomial conservatively accurate (a worthy goal...)
I am not interested in getting "exact" results, but I do want a more robust calculation with improved estimation on all *intermediate* steps inside the calculator -- eg: than if I were to assume pure Gaussians are the result of every operation. BUT -- I'd like a *simple* and robust improvement, and not an extremely fancy and complicated one.
But anyway, If you want to know how close are two distributions there is a family of probability distribution metrics that will tell you exactly that (e.g. Fortet-Mourier metric) If this is what you want you can use the metric d(Mn,M) to find the μ and σ that minimize it; This way you will have the Normal distribution Mn that would explain better M.
Yes, that seems correct; that is basically what I want to do -- although, it isn't always a pure Gaussian I want to convert to; eg: I do want the calculator to handle basic skew and possibly Kurtosis... but only in a very limited way, during intermediate steps.
I'll look up Forted-Mourier later, but I do want your opinion on the basic idea I had earlier and will explain here...
Now, if you want the fanciest of techniques for this then you go with those using
Mixtures of Distributions, which basically are methods that use overlapped distributions. You can integrate this overlapping in a Kalman Filter model (I think EE work with these to treat noise in signals) and have a very nice approximation of the products of Normal distributions by just using Normal distributions overlapped in the filter.
Sooo.. yeah.
The Kalman filter might be used in an application such as CNC trajectory control and statistical process control of said CNC. But, even there -- I don't tend to use it as there are more robust methods I developed for high speed CNC work -- eg: there are better ways to calibrate sensors, and use *much* faster computation techniques to avoid needing to worry about interpolating sample times. It's also quite complicated and computation intensive...
(Brag) -- using an older 3000 series Xilinx chip -- which was only a 70 MHz *effective* flip flop rate -- (ignore the 100MHz chip spec if you look it up -- as that's for an un-wired flip-flop!) -- but none the less, I was able to compute 128 bit wide real time trajectories in that chip at well over 16 times the speed of a 200MHZ Intel 486 processor could even compute it, let alone do IO after the computation (That CPU was the fastest of that time period.) The servo loop time of my chip was in the nano-second region. Interpolation using Kalman filters simply isn't necessary...! ;) And a 3000 series chip was only around $13 each.
Anyway, I'm not interested in anything so complicated -- I am using a mixture distribution already. Cyhelsky skew gives a relative measure of how much data is above a mean vs. below. That in turn, automatically relates the sigma on each side of the mean (one sided sigmas, don't know the proper name...) to the skew; and they become increasingly different as skew increases; The final result of a simple mixed distribution can trivially be converted to a single sigma Gaussian, which would be the exact same as if the distribution's data points were used without reference to the side of the mean which they were on.
What I did, is use half a bell curve for each side of the mean -- eg: with the sigma set different for each side. That allows the calculator to handle skew in a basic way -- *but* that method does introduce a discontinuity at the mean.
However, this mixed distribution *did* allow me to develop the mathematics to handle adding skewed distributions.
Using the numerical simulations, I have noticed that even the first or second operation (say addition) smears the discontinuity out very much. Eg: the convolution has that effect... which is very desirable.
The mathematics I developed happens to apply quite accurately to subsequent steps, and since each convolution due to addition causes the subsequent result to look *more* Gaussian, the accuracy does not degenerate with more steps (It beats the **** out of what using ONLY pure Gaussians can do.)
The representation is good enough for use right now for addition of Skewed data sets, although I'd like to replace the mixed distribution of half bell curves with one that is equivalent to that mixed distribution after a *single* operation with a true un-skewed Gaussian. The new representation would automatically smear out the discontinuity, and produce curves which more realistically representing typical data sets with tails going to zero, and a single mode.
Since multiplication tends to have almost the same curve (visually) as some of my mixture distributions after a *single* addition, I figure I can use this new mixture distribution as a basis for representing results of multiplications and divisions, internal to the calculator. (The only problem that stops me from finishing the calculator.)
So, all operations, inside the calculator, can be represented by a single data distribution type -- a mixed distribution based on a Cyhelsky skewed Gaussian...
The final result, of course, has to be reported in NIST format -- so I will need to convert an internal representation to an un-skewed Gaussian; I also need, internal to the calculator, to convert the output of multiplication to the "nearest" skewed Gaussian the calculator can represent . All conversions, then, have the same basic problem -- finding a reasonably "accurate" representation of any other arbitrary distribution.
So this is what I thought might be a good way to convert them ... It will, of course, convert a Gaussian back into the *same* Gaussian -- exactly -- so the calculator automatically performs the industry standard Gaussian error propagation formulas (with multiplication improved, appropriately of course!).
The conversion idea I have is this, the calculator's internal basis distribution is determined by three (or at most 4) statistics.
1 & 2) The Cyhelsky skew value, or equivalently -- both one sided sigmas
3) The mean
4) Potentially a Kurtosis correction value (not developed yet).
So, I thought -- ANY distribution, even if not the calculator's native representation; could be converted into those variables in this way:
Take any arbitrary distribution u; develop mathematics to add it to an uncorrelated skew Gaussian "x", and compute the resulting 4 statistics; ( or only TWO statistics if the result is to be a pure Gaussian...)
Then
solve for an "x" such that the resulting sigma of "u+x"is exactly √2 that of x, and the mu of the result is exactly twice that of "x"; Skew result must remain exactly the same as x, and be a "Maximal" value, and Kurtosis, if developed, would also remain exactly that of "x", and be maximal.
If u *were* a true Gaussian, then x would have to be the same distribution with the *same* mu and sigma of "u".
Skew and Kurtosis can't be different from the Gaussian, or the final result will have a different Skew and Kurtosis than "x", and hence they must be zero...
For a "u" of any other distribution besides Gaussian...:
"x" tells us what skewed Gaussian will produce the same results as "u" when operated on by addition...
Since adding a Gaussian to a non-Gaussian, produces something *closer* to a Gaussian -- I conjecture the final result ought to converge toward a "perfect" Gaussian and be better than just naively accepting "u"'s statistics raw...
Now, I'll look up Fortet-Mourier, but I am curious what you think of my idea. (Saying it's bador flawed is fine... I'm just curious as always...)
ME?!
and so I am no longer sure why: E( Δx Δy**2 ) = 0; Assuredly E( Δx * Δy ) WOULD be zero; but I don't know about the combination.
)
"Yeah... LHC? Yeah, Andrew speking, I was bored and then I discovered the Internet is wrong so, yeah, since you already made a clown of yourselves with the faster than light neutrino thingy I just thought I might spare you some further embarrassment... Have a nice day" 
; But I am pretty certain I need to use a metric that minimizes the error (in a particular sense) of the original distributions vs. the converted distribution *after* one of the four operators +,-,×,/ are applied to it and another value.