Memory, Entropy and the Arrow of Time

[atyy]

I wasn't referring to the energy. The Hopfield network before the memory is embedded has random connections, and after the memory is embedded it has a specific pattern of connections which generate attractor dynamics within the network. This is what constitutes a decrease in entropy.

If you still don't believe me, read this paper titled "Self-organization and entropy decreasing in neural networks" http://ptp.oxfordjournals.org/content/92/5/927.full.pdf and this paper, titled "Pattern recognition minimizes entropy production in a neural network of electrical oscillators" http://www.sciencedirect.com/science/article/pii/S0375960113007305.

Here you seem to be referring to memory formation. But is there any memory formation in these articles? It seems the weights are fixed, and the articles are discussing recall.

Also, what you are arguing is that the initial state of the Hopfield net is a high entropy state (which I don't think the articles you link to show). Even if that turns out to be true, that does not mean that the Hopfield net is remembering a high entropy past, since after training, the memories stored in the network are not of the network's initial state. Rather, the memories in the network are of the examples presented during its training. The training period requires a "teacher" who turns on Hebbian plasticity, presents a limited selection of examples, and then turns off the Hebbian plasticity. This period of training seems much more like a low entropy period, so it would seem the network is remembering a low entropy past.

 

[Stephr]

I wrote a paper on this topic: "We do have memories of the future; We just cannot make sense of them": http://philsci-archive.pitt.edu/11303/

I show the way the arrow of time, through global increase of entropy in processes associated with memories, impacts the theoretical possibility of remembering the future given the reversibility of fundamental laws of nature.

Stephane

 

[Torbjorn_L]

I wrote a paper on this topic: "We do have memories of the future; We just cannot make sense of them": http://philsci-archive.pitt.edu/11303/

I show the way the arrow of time, through global increase of entropy in processes associated with memories, impacts the theoretical possibility of remembering the future given the reversibility of fundamental laws of nature.

Stephane

I'm not sure I understand. It is an observational fact that there are no future light cones moving reverse-casually that would make "remembering the future" (and why it isn't happening) a process to look at.

 

[Stephr]

I'm not sure I understand. It is an observational fact that there are no future light cones moving reverse-casually that would make "remembering the future" (and why it isn't happening) a process to look at.

Well, the usual way to see this problem is the following: laws of physics are reversible in time, so states in the present (our memory) can be seen as the result of future events the same way they can be seen as the result of future events, with regard to the laws of nature. The distinction of the future part of a light cone you are referring to and its past part can be made because there is an "arrow of time". This arrow of time is the result of the statistical nature of entropy and the fact that the big bang was a low entropy event. The remembering process, to be effective, has to follow this arrow of time (and that's why we can't remember the future). In my paper, I just analyze a bit deeper the reasons why it has to be that way.

 

[Stephr]

Well, the usual way...

Sorry, I made a typo: "states in the present (our memory) can be seen as the result of future events the same way they can be seen as the result of PAST events"

 

[Torbjorn_L]

The distinction of the future part of a light cone you are referring to and its past part can be made because there is an "arrow of time". This arrow of time is the result of the statistical nature of entropy and the fact that the big bang was a low entropy event. The remembering process, to be effective, has to follow this arrow of time

That correlates with my understanding of the physics. There is no remembering 'of the future' since there is no such information flow (no such light cones) in physics. It isn't the brain-body chemistry that has to be predicted anymore than chemistry in general, it is the thermodynamic arrow (that EM obeys, so light cones and chemistry both) that needs a prediction.

Stochasticity, that fuzzifies the thermodynamic arrow, is part of the reason why active memory requires an amplification > 1. (Um, I saw the claim and its proof some years back, it applies for electronics. But I seem to have forgot where I put it... =D So take it for what it is worth.) That is, it requires energy, same as one can show that semi-static memory requires energy to be erased as well.

 

[David Carroll]

Here's something I do not understand...going back to the fundamentals of entropy. I have been reading Julian Barbour's obscenely fascinating book "The End of Time". He discussed Ludwig Boltzmann and his definition of entropy. Barbour states that Boltzmann defined entropy as the probability of a given thermal state. The greater the order within a system, the less probability of occurring that state has, and therefore the less entropy that system is in. Barbour likened it to a rectangular formation of 100 deep holes onto which we drop 1000 marbles. The probability that all 1000 marbles will drop into one hole is very slim, so Barbour likens such a result to very low entropy.

Now perhaps Barbour used a very misleading metaphor, but my problem with his example is that any given result of this marble-dropping business will have the exact same probability. If this marble situation is truly translatable to the thermal state of a system, then it would seem to me that any given thermal state of some defined closed system would have the exact same probability of some other arbitrary thermal state of that same system.

My guess is - as I've said - that Barbour chose an unfortunate and misleading metaphor. Could someone please enlighten a mere un-physics-educated mortal such as I?

 

[PeterDonis]

my problem with his example is that any given result of this marble-dropping business will have the exact same probability

If you look only at each individual configuration, yes. But that's not what Barbour was suggesting. What he was suggesting was: compare the number of states where all 1000 marbles are in one hole (one state) with the number where, say, there are 500 marbles in each of two holes (many more than one state because of all the different ways you can distribute the individual marbles), all the way up to the number of states in which there are 10 marbles in each hole (vastly more states still, because of the vastly greater number of possible permutations).

Each of those groups of states corresponds to a single "macroscopic state" of the system--i.e., we lump together microstates according to some macroscopic property that they all share (such as the overall distribution of marbles in holes). The more microstates there are that all share a given macroscopic property, the higher the entropy of the system when it has that particular macroscopic property.

 

[Nugatory]

Now perhaps Barbour used a very misleading metaphor, but my problem with his example is that any given result of this marble-dropping business will have the exact same probability. If this marble situation is truly translatable to the thermal state of a system, then it would seem to me that any given thermal state of some defined closed system would have the exact same probability of some other arbitrary thermal state of that same system.

That metaphor captures the essence of statistical mechanics and the thermodynamic principles that are derived from it.

You are right that every single configuration of the marbles (marble one in a particular one of the one hundred holes, marble two in another of the one hundred holes, and so forth) is equally likely. Thus, there is exactly one configuration in which all one thousand marbles end up in hole number one, and exactly one hundred configurations in which all the marbles end up in one hole. But what is the total number of configurations? Marble one can go into any of one hundred holes, and then marble two can go into any of one hundred holes, and ... The total number of configurations is $(100)^{1000}$, a number that is so huge as to defy our imagination. So what are the odds of finding all the marbles in one hole? 100 chances in $(100)^{1000}$ is a chance as imagination-defyingly small as $(100)^{1000}$ is large... Every atom in the universe doing that experiment every nanosecond since the big bang... still a negligible chance of getting that outcome.

On the other hand, how many different ways of putting marble one in a particular one of the hundred holes, marble two in another, and so forth will end up with between eight and twelve marbles in each hole? Calculate it, and you'll find that there are a lot of ways doing that... so many that the chances of not getting that outcome are negligible.

 

[David Carroll]

Okay. So we're basically making both the holes and the marbles anonymous (i.e. interchangeable, lacking particular identity)?

 

[Nugatory]

Okay. So we're basically making both the holes and the marbles anonymous (i.e. interchangeable, lacking particular identity)?

My description is based on the assumption that the marbles are distinguishable, so that marble one in hole one and marble two in hole two (and the other 998 in the same places) is a different configuration than marble two in hole one and marble one in hole two (and the other 998 in the same places). If you drop this assumption, then the statistics become interestingly different. The problem changes from "How many different ways are there to arrange 1000 different marbles in 100 holes?" to "Suppose I toss 1000 identical marbles at random onto a surface with 100 holes. How likely is it that they'll all end up in the same hole?". The answer is still unimaginably small, but it's an interestingly different unimaginably small number/.

 

[David Carroll]

Okay. I see now. I was imagining that Barbour was suggesting that second quoted question, when he was really suggesting the first. Thanks.

 

[PeterDonis]

So we're basically making both the holes and the marbles anonymous (i.e. interchangeable, lacking particular identity)?

From the standpoint of defining macroscopic states, yes. It's worth noting, though, that the details of the statistics involved actually do depend on whether or not the "marbles" and "holes" are distinguishable or not at a microscopic level. Distinguishable particles give Boltzmann statistics, which is what is standardly assumed classically. Quantum mechanically, particles of the same type are considered indistinguishable, so you get either Bose-Einstein or Fermi-Dirac statistics, depending on whether the particles have integer or half-integer spin. (In quantum field theories, there are even more kinds of statistics possible.)

 

[David Carroll]

so you get either Bose-Einstein or Fermi-Dirac statistics, depending on whether the particles have integer or half-integer spin. (In quantum field theories, there are even more kinds of statistics possible.)

Respectively?

So if we have a closed system, where one atom of each of the first 105 elements, and each of which has integer spin, is bouncing around off the other atoms, any arbitrary thermal state of this closed system has lower entropy than some other closed system where 105 atoms, all of which are, say, hydrogen, are bouncing around...according to Bose-Einstein statistics?

 

[PeterDonis]

Respectively?

Yes.

if we have a closed system, where one atom of each of the first 105 elements, and each of which has integer spin, is bouncing around off the other atoms, any arbitrary thermal state of this closed system has lower entropy than some other closed system where 105 atoms, all of which are, say, hydrogen, are bouncing around...according to Bose-Einstein statistics?

I'm not sure how you're imagining these two scenarios. If the 105 atoms are all of different elements, they're distinguishable, so you would use Boltzmann statistics. If they're all hydrogen atoms, they're not, so you would use Bose-Einstein statistics. This would result in a different count of microstates for the two cases, yes. Is that what you mean?

(I believe the count of microstates would be lower for the Bose-Einstein case, i.e., the 105 hydrogen atoms. But I haven't done the calculation to confirm that.)

 

[David Carroll]

Yeah. That's what I meant. Thanks.