Infrared said:
You have the right idea here, but the wording is a little off. Your argument that having a maxmum or a minimum contradicts the injectivity of ##f## is good, but concavity doesn't have much to do with it. Being "concave down" does not mean that a maximum is achieved on the interior (e.g. the square root function is concave down, but still strictly increasing on its domain). The last detail that you should check for this argument to be airtight is that a continuous function defined on an interval is either monotonic or has (interior) local extrema.
Yes, I think there is no reason to use concavity here. Here, I modified the Claim 2 according to your suggestion.
Claim 2: Since ##f(x)## is continuous bijection in the domain ##[0,1]##, so ##f(x)## is either strictly increasing or strictly decreasing within the interval.
Proof: WLOG, let us consider a small neighbourhood ##[a,b]## around a local extrema ##f(c)## such that ##a \lt c \lt b## (We ensure that each such interval contains only one such extrema). Obviously, ##f'(c)=0##. But, then by mean value theorem, ##\exists x_1, x_2## such that ##a \lt x_1 \lt c## and ##c \lt x_2 \lt b##, i.e. ##x_2 \gt x_1## but ##f(x_2)=f(x_1)##. This is a contradiction, since ##f(x)## is injective. So, a local extrema cannot lie in the interval (a,b). Hence, the local extrema must be at the end-points of the interval,i.e. either ##f(b) \gt f(a)## (strictly monotonically increasing) or ##f(b) \lt f(a)## (strictly monotonically decreasing). The monotonicity is strict because otherwise for some interval ##f(x)## is constant which contradicts the injectiveness of ##f(x)##. This result can be extended other adjacent intervals repeatedly.
It is evident that if ##f(x)## is monotonically increasing(or decreasing) in ##[a,b]##, then in the next interval ##[b,d]## (say), it cannot become monotonically decreasing(or increasing) because that would imply ##f'(x)## changed signs while we moved from one interval to the next. But that implies that ##f'(b)=0##. But our choice of intervals was such that each interval contains only such peak or valley, so a extrema cannot occur at ##x=b## or else we can again squeeze ##f(b)## within some interval and apply the same reasoning as above to conclude ##f'(b) \ne 0##.
So, ##f(x)## is either strictly monotonically increasing or strictly monotonically decreasing.
Infrared said:
Why are the only three possible cases that f(x)=x,f(x)>xf(x)=x,f(x)>x, and f(x)<xf(x)<x (for all xx)? Why can't it be mixed?
Yes, of course, it can be mixed. My argument can be restated by just dividing the entire intervals into sub-intervals where either ##f(x)>x## or ##f(x)<x##, and applying the same argument over the various sub intervals over the entire interval.
I think that solves all the problems. Do I need to post the complete solution again?