The average thermal velocity drops out of the equations, and the velocity addition from accelerating in the field before the next collision remains.
What the electron is doing is akin to running down a slope (the field) into a head wind (the collisions). It is the slope adding a running speed, rather than the previous walking speed while looking at the birds and smelling the flowers (random thermal velocity over a population), that decides the final speed of the balance slope/head wind.
If it is the similar looking relation between the field and the average thermal velocity that trips you up, I suggest you study how the latter comes about. (Quite frankly, I have forgotten. But it looks intuitively correct, the field would heat the electrons.) But it is unrelated to the physics at hand.