This is how I see it.
If we consider that and observers expectations of a arbitrary CHANGE, relative to whatever is the current state. Then there is typically always an expected peak evolution direction; this is then definition "forward". This is as much observer dependent as is the expectations.
But if the a priori "expected evolution" in retrospect proves to be violated, then obviously the expectations are revised so that the arrow of time always adjust. If not, it would mean that the observer "fails to learn" and such an observer would destabilise itself.
> If time went backwards, how would our brains work? How could we remember what had just happened in the 'future?'
I think by construction, the future always refers to the "expected" change. You need no global or predefine "arrorw of time". A natural direction is always implicit in the uncertainty of an information state; it's the direction in where you are doing the random walk. Backwards time would mean that your expectations were MAXIMALLY bad, and an observer can't maintain such expectations without beeing destructed because every single new information would be maximally destructive during "information updates".
Edit: I'd argue that this the OT is in face analogous to, "is it possible for an observer to maintain maximally poor inferences without beeing physically destabilized by it's environement". I.e would an information processing and gambling agent be able to completely fail to learn and still exist in some equilibrium? I think not. However transient deviations happen all the time. So a transient or temporary violation of expectations is just part of the game, just like the 2nd law is "only" statistical.
/Fredrik