OP, I think your question is simply: how did they go from:
[itex]length(ab)= \sqrt{\left(dx+\frac{\partial u_x}{\partial x}dx\right)^2+\left(\frac{\partial u_y}{\partial x}dx\right)^2}[/itex]
to
[itex]length(ab)\approx dx+\frac{\partial u_x}{\partial x}dx[/itex] ?
The math doesn't work, I agree.
From a geometric point-of-view, as Studiot suggested, they assume that [itex]length(ab)[/itex] is equal to its horizontal projection, for small deformations.
However, I'm not sure that I buy that, to be honest.
In terms of the actual physics, and in looking at the provided diagram, I can tell you that if it were only a simple shear, you could get the angle change [itex]\alpha + \beta[/itex] but there has to be some sort of homogeneous (axial) deformation in order for BOTH [itex]dx[/itex] and [itex]dy[/itex] to change lengths.
For example, one way to arrive at the apparent deformed shape would be:
1) apply a homogenous deformation (e.x. to the right, of magnitude ([itex]length(ab)-dx[/itex]) -- i.e. [itex]\sqrt{\left(dx+\frac{\partial u_x}{\partial x}dx\right)^2+\left(\frac{\partial u_y}{\partial x}dx\right)^2}-dx[/itex])
2) apply a simple shear (e.x. to the right, of amount [itex]\alpha + \beta[/itex])
3) apply a rigid body rotation (e.x. counter-clockwise, of amount [itex]\alpha[/itex])
Does that make sense?
You can play with this though.
Take 1) to be zero. No deformation to the right means [itex]length(ab)=dx[/itex].
2) and 3) still apply - and so we have a simple shear and a rigid body rotation.
We should still get [itex]length(ab)=dx[/itex] in this case under either a small shear or a large shear. However, due to the rigid body rotation, [itex]\frac{\partial u_x}{\partial x}dx[/itex] in their diagram would be nonzero and so their expression [itex]length(ab)\approx dx+\frac{\partial u_x}{\partial x}dx[/itex] is not equal to [itex]dx[/itex]. This doesn't mean that they are wrong, but I cannot immediately justify approximating [itex]length(ab)[/itex] as its horizontal projection, for the general case that they are showing.
In other words, I don't like their expression [itex]length(ab)\approx dx+\frac{\partial u_x}{\partial x}dx[/itex] unless someone can prove to me that it agrees with more advanced solid mechanics.