OP, I think your question is simply: how did they go from:
length(ab)= \sqrt{\left(dx+\frac{\partial u_x}{\partial x}dx\right)^2+\left(\frac{\partial u_y}{\partial x}dx\right)^2}
to
length(ab)\approx dx+\frac{\partial u_x}{\partial x}dx ?
The math doesn't work, I agree.
From a geometric point-of-view, as Studiot suggested, they assume that length(ab) is equal to its horizontal projection, for small deformations.
However, I'm not sure that I buy that, to be honest.
In terms of the actual physics, and in looking at the provided diagram, I can tell you that if it were only a simple shear, you could get the angle change \alpha + \beta but there has to be some sort of homogeneous (axial) deformation in order for BOTH dx and dy to change lengths.
For example, one way to arrive at the apparent deformed shape would be:
1) apply a homogenous deformation (e.x. to the right, of magnitude (length(ab)-dx) -- i.e. \sqrt{\left(dx+\frac{\partial u_x}{\partial x}dx\right)^2+\left(\frac{\partial u_y}{\partial x}dx\right)^2}-dx)
2) apply a simple shear (e.x. to the right, of amount \alpha + \beta)
3) apply a rigid body rotation (e.x. counter-clockwise, of amount \alpha)
Does that make sense?
You can play with this though.
Take 1) to be zero. No deformation to the right means length(ab)=dx.
2) and 3) still apply - and so we have a simple shear and a rigid body rotation.
We should still get length(ab)=dx in this case under either a small shear or a large shear. However, due to the rigid body rotation, \frac{\partial u_x}{\partial x}dx in their diagram would be nonzero and so their expression length(ab)\approx dx+\frac{\partial u_x}{\partial x}dx is not equal to dx. This doesn't mean that they are wrong, but I cannot immediately justify approximating length(ab) as its horizontal projection, for the general case that they are showing.
In other words, I don't like their expression length(ab)\approx dx+\frac{\partial u_x}{\partial x}dx unless someone can prove to me that it agrees with more advanced solid mechanics.