In any given inertial reference frame, the rate a clock slows down at any given moment depends only on its velocity. So, if you have a clock that moves at constant velocity for the outbound leg, then abruptly switches velocity and after that travels at constant velocity for the inbound leg, the total amount of time dilation between departing and returning could be calculated by (amount of time the outbound leg lasts in my frame)*(time dilation factor based on speed during outbound leg in my frame) + (amount of time the inbound leg lasts in my frame)*(time dilation factor based on speed during inbound leg in my frame). Even if this is calculated in different inertial frames with different opinions about how long each leg of the trip lasts and how fast the clock was moving during each leg, all the frames will get the same answer for the total amount of time elapsed on the clock between departing and returning. And if we assume this clock departed and returned from a clock that was moving inertially, all inertial frames will also agree on the amount of time ticked on the inertial clock between these two events--and the answer will always be that the clock that changed velocity has elapsed less time than the clock that didn't, no matter what frame you look at the problem in. So even though the amount of time dilation on the non-inertial clock is based on its velocity at different times, the fact that it changed velocity (accelerated) does explain why it elapsed less time.
You can think of this as a "geometric" property of spacetime--if you graph space on one axis and time on another, then the worldline of an inertial clock will always look like a straight line, while the worldline of a clock that moves away from the inertial clock, then instantly changes velocity and moves back towards it, will look like a bent line. The fact that the bent worldline between two events (the events of the two clocks departing from a single location and later reuniting at a single location) will always have less proper time (time as measured by a clock taking that path) than a straight worldline between the same two events is analogous to the fact that if you draw two points on a piece of paper, a straight line between them will always have a shorter length than a bent line between them.