It is rather difficult to talk about human memory because we don't know how the brain works in detail. We do, however, know all about how computer memories work. I shall therefore discuss the psychological arrow of time for computers. I think it is reasonable to assume that the arrow for computers is the same as that for humans. If it were not, one could make a killing on the stock exchange by having a computer that would remember tomorrow's prices!
A computer memory is basically a device containing elements that can exist in either of two states. A simple example is an abacus. In its simplest form, this consists of a number of wires; on each wire is a bead that can be put in one of two positions. Before an item is recorded in a computer's memory, the memory is in a disordered state, with equal probabilities for the two possible states. (The abacus beads are scattered randomly on the wires of the abacus.) After the memory interacts with the system to be remembered, it will definitely be in one state or the other, according to the state of the system. (Each abacus bead will be at either the left or the right of the abacus wire.) So the memory has passed from a disordered state to an ordered one. However, in order to make sure that the memory is in the right state, it is necessary to use a certain amount of energy (to move the bead or to power the computer, for example). This energy is dissipated as heat, and increases the amount of disorder in the universe. One can show that this increase in disorder is always greater than the increase in the order of the memory itself. Thus the heat expelled by the computer's cooling fan means that when a a computer records an item in memory, the total amount of disorder in the universe still goes up. The direction of time in which a computer remembers the past is the same as that in which disorder increases.