Exploring the Illusion of Randomness: The Truth Behind Algorithms and Chaos

[rxh140630]

I've been learning a lot about algorithms in my AI class, senior year of my computer science degree.

Some of the algorithms we talk about involve randomness.

Does true randomness even exist in reality? Wouldn't everything random, have an explanation by some physical process? Is chaos random? And how would we prove this? How do we know our results aren't just coming from a lack of ability on our part whether through measurement, or faulty human reasoning?

 

[PeroK]

Instead of asking whether randomness "exists in reality", you could ask whether randomness is meaningful - in the sense of being a useful concept.

For example, you could tie yourself in philosophical knots over the question of whether the electric field "exists in reality". But, if you ask whether the electric field is a useful and hence meaningful concept, then you can get on with learning about electromagnetism.

 

[BWV]

In most real world applications, randomness just an attempt to quantify ignorance

 

[PeroK]

In most real world applications, randomness just an attempt to quantify ignorance

Let's take an example. Any tennis match at the Australian Open and the exact sequence of points won and lost by each player.

What knowledge would you need to predict that in advance? Does that knowldege actually exist in any meaningful sense? Or, is that knowldege not even theoretically available?

For example, does all the information currently exist somewhere that would allow you to predict not only the players in the women's singles final, say, but the exact sequence of points won and lost?

 

[Borek]

Does true randomness even exist in reality? Wouldn't everything random, have an explanation by some physical process? Is chaos random? And how would we prove this? How do we know our results aren't just coming from a lack of ability on our part whether through measurement, or faulty human reasoning?

Have you heard about the Bell's theorem? IMHO basically you are asking whether the process that looks random can be explained using hidden variables.

 

[f95toli]

From a "scientific" point of view you can have true randomness because of quantum effects. Good examples include radioactive decay.
There are in fact commercially available quantum random number generators (based on photonics) which are "certified true random" and are being tested for e.g post-quantum cryptography (see e.g IronBridge from CQC).

The reason for why I specify that this is the "scientific" point of view is to emphasise that it impossible to conclusively prove that outcomes that "seem random" don't have a "cause" that is hidden from us(see superdeterminism, which is not covered by the Bell test).
However, I would argue that this is philosophical argument since it is -by definition- not something that can be tested by science.

 

[BWV]

Let's take an example. Any tennis match at the Australian Open and the exact sequence of points won and lost by each player.

What knowledge would you need to predict that in advance? Does that knowldege actually exist in any meaningful sense? Or, is that knowldege not even theoretically available?

For example, does all the information currently exist somewhere that would allow you to predict not only the players in the women's singles final, say, but the exact sequence of points won and lost?

Why should the pertinent information need to be somehow available to make it a valid statement? Is that somehow bound to the definition of the word ignorance? If its semantics, the relevant definition here is simply a lack of information. Can never really know all the variables to sufficient precision in a roulette wheel or whether it will rain a week from next Tuesday.

 

[PeroK]

Why should the pertinent information need to be somehow available to make it a valid statement? Is that somehow bound to the definition of the word ignorance? If its semantics, the relevant definition here is simply a lack of information. Can never really know all the variables to sufficient precision in a roulette wheel or whether it will rain a week from next Tuesday.

It may be sematics. The question is whether the information exists - and is not known to sufficient precision - or whether the information does not exist at all.

 

[BWV]

It may be sematics. The question is whether the information exists - and is not known to sufficient precision - or whether the information does not exist at all.

Its an interesting philosophical question though - if we understood the human brain an physiology perfectly and had access to real-time data on all relevant neurological and physical processes of the two tennis players, how precisely could you predict the outcome? Is the time delay relevant? presumably at some length of time before the match you could get no information even from perfect knowledge of the players biology and neurology.

 

[Klystron]

I like the Australian Open tennis score example but tennis requires humans to generate and collate the statistics.

One favorite example of randomness uses a scintillation counter expressed as a flat grid shielded on sides and bottom, open to the sky, basically a 'cosmic ray' detector. Grid spacing depends on the equipment and experiment design but corresponds to some integer factor of the expected output. Number the grid intervals. A 'cosmic ray hit' in a grid generates that number noted and saved on a queue.

While similar to using a Geiger counter with directional input and sensitivity controls to measure background radiation or a radioactive source, this example does not require human intervention once activated.

Fellow software engineers and computer science students poked some holes in this example but the general function was known in antiquity. Bored students on a rainy day would draw a large grid in the dust and place wagers on where a rain drop would strike. Excellent source for arguments, one imagines.

 

[PeroK]

Its an interesting philosophical question though - if we understood the human brain an physiology perfectly and had access to real-time data on all relevant neurological and physical processes of the two tennis players, how precisely could you predict the outcome? Is the time delay relevant? presumably at some length of time before the match you could get no information even from perfect knowledge of the players biology and neurology.

That's only the beginning. You need to work out which players make the final - and that means predicting every previous match. Who the umpire and line judges will be: what mistakes they will make that affect the match. How the ball boys or girls will operate - how long it takes then to get balls to the players. Which balls will be used. Weather conditions. How the crowd will react.

(Remember: you have to predict every point - so you need a complete analysis of every point.)

And, all of this depends on external factors outside the game of tennis: whether a player will get sick, sleep badly, who they will train with etc.

And, if we were considering a non-COVID year, eventually pretty much everyone and everything in the world might have some influence on the final (even if it's just the butterfly effect on the weather in Melbourne)!

You can see that, even in a clockwork universe (leaving aside QM), it's doubtful that this information exists in any meaningful sense. And, it seems likely that QM uncertainty affects the relevant phenomena to a sufficient degree. And then the information actually does not exist.

 

[DennisN]

Does true randomness even exist in reality?

The outcome of (certain types of) measurements in quantum mechanics are random (caveat: at least according to our current understanding). I posted an example using sequential measurements in the Stern-Gerlach experiment in this post in the thread "Is there anything in physics that's random?".

Another thing I like about the Stern-Gerlach example is that it is quite counterintuitive and can mess with one's head.

 

[BWV]

That's only the beginning. You need to work out which players make the final - and that means predicting every previous match. Who the umpire and line judges will be: what mistakes they will make that affect the match. How the ball boys or girls will operate - how long it takes then to get balls to the players. Which balls will be used. Weather conditions. How the crowd will react.

(Remember: you have to predict every point - so you need a complete analysis of every point.)

And, all of this depends on external factors outside the game of tennis: whether a player will get sick, sleep badly, who they will train with etc.

And, if we were considering a non-COVID year, eventually pretty much everyone and everything in the world might have some influence on the final (even if it's just the butterfly effect on the weather in Melbourne)!

You can see that, even in a clockwork universe (leaving aside QM), it's doubtful that this information exists in any meaningful sense. And, it seems likely that QM uncertainty affects the relevant phenomena to a sufficient degree. And then the information actually does not exist.

Right and the information on whether it will rain at a particular location on June 3, 2345 or whether the coin flip for Superbowl DCLXVI will be heads does not exist either, but at some point prior to the event becomes predictable with information that comes into existence. But, back to the original point, we can attempt to describe the lack of information quantitatively

 

[PeroK]

Right and the information on whether it will rain at a particular location on June 3, 2345 or whether the coin flip for Superbowl DCLXVI will be heads does not exist either, but at some point prior to the event becomes predictable with information that comes into existence. But, back to the original point, we can attempt to describe the lack of information quantitatively

This boils down to whether a wave of the hand is sufficient. In other words, if someone asks you what information is needed to predict something of that sort, you wave your hand and say "everything ...". But, if someone asks you to be specific, then it's problematic, because it's not clear how you could demonstrate that that information actually exists or even describe the information you need.

It's probably moot, because QM enters the picture, but even without that I've never been comfortable with the idea of saying "all possible information ..." and assuming that that is a well-defined set of information.

 

[Jarvis323]

Don't we need to distinguish determinism from predictability anyways?

For example, even if there were some exact physical realization of a continuous system, e.g. of the logistic function, and you knew the parameters, and exact state of the system, at an exact moment of time, you would still not be able to predict the future states exactly, as it would require processing data with infinite precision. Besides that, in some cases, the information about the state of the system would be infinite anyways, so you could never have enough information to even get started.

I also know that the distinction between causality and predictably has historically been an important topic in QM. Where do physicists stand nowadays on these issues? I've only read about it from a historical context, e.g. Grete Hermann's ideas.

More specifically, her main work in physics was on the philosophical foundations of quantum mechanics, the significance of modern physics for the theory of knowledge, and causality in physics.
...
Hermann distinguished between causality and predictability and emphasized the fact that they are not identical; she stated that “The fact that quantum mechanics assumes and pursues a causal account also for unpredictable occurrences proves that an identification of these two concepts is based on a confusion” (Hermann, quoted in Jammer, p. 209; Lenzen). This allows for the possibility that physical processes may be strictly determined even though exact prediction is not possible (Lenzen).
...
Hermann’s views seem to emphasize the asymmetry between explanation and prediction in quantum mechanics as opposed to their symmetry in classical physics. This analysis was subsequently extended by others, including Norwood Russell Hanson, who appears to have emphasized that after a quantum event has occurred, a complete explanation of its occurrence can be given within the total quantum theory, but that it is in principle impossible to predict in advance those features of the event that can be explained after the fact (Jammer, p. 209)

https://www.europeanwomeninmaths.org/wp-content/uploads/2016/02/gretehermann.pdf

In the end, if we say an even is random because it is fundamentally unpredictable (no matter how much information you have, or how much processing power you have), then we have to concede that deterministic processes can be random. Otherwise, we have to concede that Bayesian's are only fooling themselves?

 

[BWV]

Don't we need to distinguish determinism from predictability anyways?

For example, even if there were some exact physical realization of a continuous system, e.g. of the logistic function, and you knew the parameters, and exact state of the system, at an exact moment of time, you would still not be able to predict the future states exactly, as it would require processing data with infinite precision.

Where do physicists stand nowadays on these issues? I've only read about it from a historical context, e.g. Grete Hermann's ideas.
https://www.europeanwomeninmaths.org/wp-content/uploads/2016/02/gretehermann.pdf

If we say an even is random because it is fundamentally unpredictable (no matter how much information you have), then we have to concede that deterministic processes are random. Otherwise, we have to concede that Bayesian's fooling themselves?

I don’t think the distinction between unpredictable determinism and pure randomness matters - and to PeroK’s earlier point - how does this dichotomy apply to human behavior that involves individual choices? Is my decision on what to eat for lunch tomorrow as inherently random to you as the outcome of some quantum mechanical system?

 

[Jarvis323]

I don’t think the distinction between unpredictable determinism and pure randomness matters - and to PeroK’s earlier point - how does this dichotomy apply to human behavior that involves individual choices?

It matters at least to be clear with the definition of randomness. Usually people would say strictly deterministic processes are not random. But they can be unpredictable.

Is my decision on what to eat for lunch tomorrow as inherently random to you as the outcome of some quantum mechanical system?

'Random to me' is not the same thing as random. Pseudo-random number generators are random to me also, and they're not only deterministic, but also predictable.

If we are interested in what is true randomness, we have to decide what randomness is in the first place. And answering the question (does true randomness exist?) is either trival, or depends on what goes on under the hood with QM and inside of our minds I guess (depending on if you count outcomes of deterministic processes as random or not).

 

[rxh140630]

The outcome of (certain types of) measurements in quantum mechanics are random (caveat: at least according to our current understanding). I posted an example using sequential measurements in the Stern-Gerlach experiment in this post in the thread "Is there anything in physics that's random?".

Another thing I like about the Stern-Gerlach example is that it is quite counterintuitive and can mess with one's head.

This is just so opposite to my intuition that I can't wrap my mind around it.

I will preface this with saying I do not even know what quantum mechanics is. I know mechanics is a branch of physics that deals with motion, I guess quantum implies the smallest building blocks? So the study of atoms?

I just can't believe that everything we observe has order, is quite predictable, yet, the building blocks have randomness as something that's present.

 

[PeroK]

I just can't believe that everything we observe has order, is quite predictable, yet, the building blocks have randomness as something that's present.

First, most things are unpredictable: the sex of a child, COVID variants, the weather, the economy, road accidents, hardware failures etc. In reality we are confronted by randomness (true or apparent) and probabilities at every turn in our daily lives.

Second, the law of large numbers means that the average of random events can be more predictable: the number of children born or total number of people dying in road accidents etc. A constant air pressure, for example, is the effect of trillions of random collisions by individual air molecules.

Even without the "true" randomness of QM, you have macroscopic systems of trillions of particles displaying some order and predictability, where the constituent particles display random behaviour.

Third, why do you not believe that quantum randomness is possible? Is your gut instinct really so reliable as a guide to the nature of the universe?

 

[PeroK]

I've been learning a lot about algorithms in my AI class, senior year of my computer science degree.

Here are some examples of the relevance of quantum randomness to modern electronics and hardware.

https://en.wikipedia.org/wiki/Quantum_tunnelling#Electronics

There's also a thing called "quantum computing", which I imagine was not in your course:

https://en.wikipedia.org/wiki/Quantum_computing

 

[Lord Jestocost]

Does true randomness even exist in reality?

True randomness means that an event occurring in space and time can in principle not be undone. Here is the essential difference between the views of classical and quantum physics.

As Richard D. Gill puts it in “Statistics, Causality and Bell’s Theorem”:

“In classical physics, randomness is merely the result of dependence on uncontrollable initial conditions. Variation in those conditions, or uncertainty about them, leads to variation, or uncertainty, in the final result. However, there is no such explanation for quantum randomness. Quantum randomness is intrinsic, nonclassical, irreducible. It is not an emergent phenomenon. It is the bottom line. It is a fundamental feature of the fabric of reality.” [italic in original, LJ]

 

[BWV]

but these classical / quantum randomness arguments focus on physics and ignore biology, and arguably the majority of real world applications of statistics (particularly in the ML algorithms mentioned in the OP)are used to predict actions of individuals. For example, we can reasonably predict probabilities of default on credit cards that work well enough for banks to underwrite the loans, but can the probability that Bob defaults on his account be reduced to something deterministic?

 

[Lord Jestocost]

The OP asked:

Does true randomness even exist in reality?

My answer was "Yes".

 

[DennisN]

This is just so opposite to my intuition that I can't wrap my mind around it.

Welcome to quantum mechanics. Counterintuitive is the name of the game.

I will preface this with saying I do not even know what quantum mechanics is. I know mechanics is a branch of physics that deals with motion, I guess quantum implies the smallest building blocks? So the study of atoms?

Yes, partly. It's also called quantum physics. It's about the physics of atomic and subatomic systems and particles (e.g. electrons and photons). The reason for the name "quantum" is that atoms and particles can have certain discrete states (as opposed to continously varying).

One example of this is the emission spectrum of the element hydrogen:
(source here)

Each line of color corresponds to one of the various energy levels (of the electron) possible in a hydrogen atom. If any energy level was possible, the spectrum would be continuous (like a rainbow), but since the energy levels are quantized, only certain, distinct lines appear in the spectrum. Here's a real demonstration of emission spectra: Spectrum Demo: Continuous and Emission (Physics Demos).

I just can't believe that everything we observe has order, is quite predictable, yet, the building blocks have randomness as something that's present.

You are definitely not alone in having problems with this. It is a common thing when people encounter quantum mechanics. In short, and simplified: the world we see and experience (sometimes called the "classical" world) can be said to emerge from the behavior of many microscopic quantum systems. But underneath it all, it's a quantum world; our world is governed by quantum physics at the bottom.

It is very briefly described in this cool video (not that it's random, but that it's a quantum world at the bottom):

Ben Miller experiments with superfluid helium - Horizon: What is One Degree? - BBC Two
Edit:

I also want to point out that everything is not random in quantum physics. There are properties of atoms and particles that are fixed, e.g.

• the various energy levels of atoms (as I described above)
• the various electric charges (of elementary particles)
• the various rest masses (of elementary particles)
• the various spins (of elementary particles; the "amount" of spin is fixed, the direction of the spin is not)

 

[Jarvis323]

True randomness means that an event occurring in space and time can in principle not be undone. Here is the essential difference between the views of classical and quantum physics.

As Richard D. Gill puts it in “Statistics, Causality and Bell’s Theorem”:

“In classical physics, randomness is merely the result of dependence on uncontrollable initial conditions. Variation in those conditions, or uncertainty about them, leads to variation, or uncertainty, in the final result. However, there is no such explanation for quantum randomness. Quantum randomness is intrinsic, nonclassical, irreducible. It is not an emergent phenomenon. It is the bottom line. It is a fundamental feature of the fabric of reality.” [italic in original, LJ]

There is a problem with Gill's argument if you extend it to an argument that true randomness exists. Namely, the problem is with the assumption that randomness being irreducible in theory implies that it is true randomness.

If, for example, the randomness in QM is deterministic under the hood, e.g. through hidden variables, then (you could argue) that the apparent randomness in QM is not true randomness. The randomness in the theory remains independent of whether the apparent randomness is intrinsic to nature, because we cannot make a both extended and testable theory of QM to include the determinism, because we couldn't measure those hidden variables in order to verify predictions.

Some like to limit their view of what exists to what can in theory be confirmed to exist. Under that view, we can say that true randomness (as a property) doesn't exist.

Otherwise, if we accept that existence can extend beyond our horizon, then the answer is that the question is (likely) impossible to answer (as far as we know).

 

[BWV]

As you look at more complex physical systems how often do the sensitive initial conditions include quantum mechanical effects? Do QM phenomena within the sun influence climate, for example? How does radioactive decay impact behavior of the arrangement of molecules of certain gasses etc? ISTM when you try to make an absolute distinction between pure ‘random’ QM systems and purely deterministic CM systems calculable to an arbitrarily high precision you find the real world is much messier

 

[Couchyam]

It's been a while since I've posted here, but I like this general topic and so I couldn't resist getting involved. It sounds as though there are two schools of thought expressed in this thread, one of which centers on whether randomness exists in a fundamental sense, and the other on its usefulness as a concept. I find both lines of questions equally fascinating. My gut level response would be that randomness is probably a very convincing illusion, whether in the classical or quantum settings, but that the illusion is probably no worse than many other illusions that we must entertain in order to go about our daily lives. Could there be a sort of 'connection' to the question of how a reference frame or 'observer' is defined in physics, and the fidelity with which results from one reference frame can be communicated with another?

 

[PeroK]

My gut level response would be that randomness is probably a very convincing illusion, whether in the classical or quantum settings, but that the illusion is probably no worse than many other illusions that we must entertain in order to go about our daily lives.

That's exactly the way I feel about determinism. My gut instinct tells me that it's the illusion - although, a less convincing one!

 

[Couchyam]

That's exactly the way I feel about determinism. My gut instinct tells me that it's the illusion - although, a less convincing one!

I guess I'm inclined to assume determinism is more fundamental than randomness because almost every fundamental physical experiment to date has been consistent with the postulate of unitarity in quantum mechanics, and I interpret unitarity (i.e. constant, vanishing von Neumann entropy) as quantum mechanical determinism. Then again, you might have in mind the question of how or whether the 'initial data' is or can be ever entirely known from a scientist's point of view (which must be trusted with choosing suitably generic and not overly-contrived experimental conditions.)

 

[BWV]

Does the fact that a deterministic model describes a system to the level of precision that we care about make the underlying phenomena deterministic to any arbitrary level of precision (which TRVE determinism would imply)?

 

[Couchyam]

Does the fact that a deterministic model describes a system to the level of precision that we care about make the underlying phenomena deterministic to any arbitrary level of precision (which TRVE determinism would imply)?

No, but to me it suggests that it wouldn't be unreasonable to act on the basis that those phenomena are fundamentally deterministic at some level of awareness (unless doing so made one miserable.)