Do you see that [tex]\begin{pmatrix}a & b & c \\ d & e & f \\ g & h & i\end{pmatrix}\begin{pmatrix}1 \\ 0 \\ 0 \end{pmatrix}= \begin{pmatrix}a \\ d \\ g\end{pmatrix}[/tex],
that [tex]\begin{pmatrix}a & b & c \\ d & e & f \\ g & h & i\end{pmatrix}\begin{pmatrix}0 \\ 1 \\ 0 \end{pmatrix}= \begin{pmatrix}b \\ e \\ h\end{pmatrix}[/tex],
and [tex]\begin{pmatrix}a & b & c \\ d & e & f \\ g & h & i\end{pmatrix}\begin{pmatrix}0 \\ 0 \\ 1 \end{pmatrix}= \begin{pmatrix}c \\ f \\ i\end{pmatrix}[/tex]?
That is, applying the linear transformation to the vectors in the ordered basis, in turn, gives you the columns of the matrix.
Here the linear transformation is [tex]T(p)= (x^2+ 1)p''(x)- xp'(x)+ 2p(x)[/tex] (you have "2p'(x) but that looks suspicious. If that were correct why wouldn't it be "(2- x)p'(x)"?) and, in the first problem, the basis vectors are [tex]x^2[/tex],[tex]x[/tex], and [tex]1[/tex]. [tex]T(x^2)= (x^2+ 1)(2)- x(2x)+ 2x^2=x^2+ 1[/tex]. That would be represented by the column matrix [tex]\begin{pmatrix}1 \\ 0 \\ 1\end{pmatrix}[/tex] and that is the first column of the matrix. [tex]T(x)=(x^2+ 1)(0)- x(1)+ 2(x)= x[/tex] which is represented by the column matrix [tex]\begin{pmatrix}0 \\ 2 \\ 0\end{pmatrix}[/tex] and that is the second column of the matrix. Finally, [tex]T(1)= (x^2+ 1)(0)- x(0)+ 2(1)= 2[/tex] which is represented by the column matrix [tex]\begin{pmatrix}0 \\ 0 \\ 2\end{pmatrix}[/tex] and that is the third column of the matrix. The matrix representation of T in this ordered basis is [tex]\begin{pmatrix} 1 & 0 & 0 \\ 0 & 2 & 0 \\ 1 & 0 & 2 \end{pmatrix}[/tex].
In the second problem the ordered basis vectors are [tex]x^2+ x+ 1[/tex], [tex]x+1[/tex]. and [tex]1[/tex]. [tex]T(x^2+ x+ 1)= (x^2+ 1)(2)- x(2x+ 1)+ 2(x^2+ x+ 1)= 2x^2+ 2- 2x^2- x+ 2x^2+ 2x+ 2= 2x^2+ x+ 4[/tex]. But now we have to write that in terms of this basis. That is, we need to find a, b, and c so that [tex]a(x^2+ x+ 1)+ b(x+ 1)+ c(1)= ax^2+ (a+ b)x+ (a+ b+ c)= 2x^2+ x+ 4[/tex]. We have a= 2, a+ b= 2+ b= 1 so b= -1 and a+ b+ c= 2- 1+ c= 1+ c= 4 so c= 3. The first column of the matrix is [tex]\begin{pmatrix} 2 \\ -1 \\ 3 \end{pmatrix}[/tex]. [tex]T(x+ 1)= (x^2+ 1)(0)- x(1)+ 2(x+ 1)= x+ 1[/tex]. Well that's easy! [tex]x+ 1= 0(x^2+ x+ 1)+ 1(x+ 1)+ 0(1)[/tex] so the second column of the matrix is [tex]\begin{pmatrix}0 \\ 1 \\ 0\end{pmatrix}[/tex]. [tex]T(1)= (x^2+ 1)(0)- x(0)+ 2(1)= 2[/tex]. That can be written [tex]2= 0(x^2+ x+ 1)+ 0(x+ 1)+ 2[/tex] which corresponds to the column matrix [tex]\begin{pmatrix}0 \\ 0 \\ 2\end{pmatrix}[/tex]. The matrix representing T in this ordered matrix is [tex]\begin{pmatrix} 2 & 0 & 0 \\ -1 & 1 & 0 \\ 3 & 0 & 2\end{pmatrix}[/tex].