The concept of time comes from our awareness of change. It is most obvious that objects around us are in flux, and our inherent sense of this is necessary for interacting with the world. We have refined our notion of time by finding (or fabricating) objects that undergo cyclical change, and we measure the passage of time by counting cycles. We measure time by measuring change.
From this it is clear that traveling "forward" in time is no problem, because our concept of change (cause --> effect) is a forward progression. But traveling backwards is problematic: What does changing backwards mean if all change is forward by definition?
Next we invent Thermodynamics.