I did it quite similarly to
@etotheipi. My starting point was to make the sum vertically (your picture).
$$S=\sum_{j=1}^nj\left(\sum_{i=j}^nr^i\right)$$
I know how to compute a geometric sum from ##i=0## up to some number, but here I want to start from some arbitrary index, so my plan is to make two sums starting from ##i=0##, but with one ending at ##i=n##, and the other at ##i=j-1##, then subtract both
$$\begin{align*}
S&=\sum_{j=1}^nj\left(\sum_{i=0}^{n}r^i-\sum_{i=0}^{j-1}r^i\right)\\
&=\sum_{j=1}^nj\left(\frac{1-r^{n+1}}{1-r}-\frac{1-r^j}{1-r}\right)\\
&=\sum_{j=1}^nj\left(\frac{r^j-r^{n+1}}{1-r}\right)\\
&=\left(\sum_{j=1}^n\frac{jr^j}{1-r}\right)-\left(\frac{r^{n+1}}{1-r}\sum_{j=1}^nj\right)
\end{align*}$$
Since ##0<r<1##, I know that the second parenthesis is positive because ##1-r>0##, and so
$$S=\left(\sum_{j=1}^n\frac{jr^j}{1-r}\right)-\left(\frac{r^{n+1}}{1-r}\sum_{j=1}^nj\right)<\sum_{j=1}^n\frac{jr^j}{1-r}$$
If you remove something from a positive number, then, of course, that positive number is bigger than the result of that operation.
Using the hint, we get
$$\sum_{j=1}^njr^j<\frac{r}{(1-r)^2}\Leftrightarrow S<\frac{1}{1-r}\sum_{j=1}^njr^j=\sum_{j=1}^n\frac{jr^j}{1-r}<\frac{r}{(1-r)^3}$$