First, the "factorizing" is an "operator" method using that fact that we can treat repeated derivatives (as long as the coefficients are constants) as if they were products and so get a symbolic way of reducing the equation to a series of first order equations.
Second, If we have a differential equation of the form [itex]d^2y/dx- 2m dy/dx+ m^2y= 0[/itex] so that "trying" a solution of the form [itex]y= e^{rx}[/itex] reduces the equation to [itex]r^2 e^{rx}- 2mr e^{rx}+ m^2e^rx= e^{rx}(r^2- 2mr+ m^2)= e^{rx}(r- m)^2= 0[/itex] and, since [itex]e^{mx}[/itex] is never 0, we must have [itex](r- m)^2= 0[/itex] which, of course, has the single root r= m. That tells us immediately that [itex]y= e^{mx}[/itex] is a solution to the differential equation. Of course, because this is a second order linear, homogeneous solution, we need a second independent solution to be able to write the general solution.
It is easy to see, by checking, that [itex]y= xe^{mx}[/itex] is a solution: if [itex]y= xe^{mx}[/itex] then [itex]dy/dx= e^{mx}+ mxe^{mx}[/itex] and [itex]d^2y/dx^2= 2me^{mx}+ m^2xe^{mx}[/itex] so that [itex]d^2y/dx^2- 2m dy/dx+ m^2y= (2me^{mx}+ m^2xe^{mx})- 2m(e^{mx}+ mxe^{mx})+ xe^{mx}= (m^2xe^{mx}- 2m^2xe^{mx}+ m^2e^{mx})+ (2me^{mx}- 2me^{mx})= 0[/itex].
We can then appeal to the theory, that the set of all solutions to a second order linear homogeneous differential equation form a vector space of dimension 2 and so can be spanned by two independent solutions, to see that there cannot be other independent solutions.
And, of course, we see, by direct substitution, that is [itex]y= x^2e^{mx}[/itex] then [itex]dy/dx= 2xe^{mx}+ mx^2e^{mx}[/itex] and [itex]d^2y/dx^2= 2e^{mx}+ 4mxe^{mx}+ m^2x^2e^{mx}[/itex] and the equation becomes [itex](2e^{mx}+ 4mxe^{mx}+ m^2x^2e^{mx})- 2m(2xe^{mx}+ mx^2e^{mx})+ m^2x^2e^{mx}= (m^2x^2e^{mx}- 2^2x^2e^{mx}+ m^2x^2e^{mx})+ (4mxe^{mx}- 4mxe^{mx})+ (2e^{mx}= 2e^{mx}[/itex], not 0.
How would we know in advance that worked? Experience- and they are trying to save you the time required to get that experience.