You're right, but we can also do that with the blue clock if we consider that it is the one that is moving, and we will still get reversed data. If we don't know which clock is moving, how can we predict which one will suffer time dilation? When you change reference frames, you're assuming that the yellow clock is accelerating, so why didn't you assume it was accelerating in the beginning?It's not directly related to acceleration. It's related to changing your reference frame.
There are several ways to analyse this. One option is simply be to use a suitable IRF (inertial reference frame). The frame of the blue clock is the obvious candidate - but that probably won't convince you!
You could choose any other IRF. The one that the yellow clock has for the first half of its journey might be an interesting choice. In that frame, the yellow clock would be at rest for the outward journey, but then move to the left at a greater speed to catch the blue clock that moves to the left the whole time.
In that frame, you would find the same result for the clocks (both blue and yellow) when they meet up.