Do you see that \begin{pmatrix}a & b & c \\ d & e & f \\ g & h & i\end{pmatrix}\begin{pmatrix}1 \\ 0 \\ 0 \end{pmatrix}= \begin{pmatrix}a \\ d \\ g\end{pmatrix},
that \begin{pmatrix}a & b & c \\ d & e & f \\ g & h & i\end{pmatrix}\begin{pmatrix}0 \\ 1 \\ 0 \end{pmatrix}= \begin{pmatrix}b \\ e \\ h\end{pmatrix},
and \begin{pmatrix}a & b & c \\ d & e & f \\ g & h & i\end{pmatrix}\begin{pmatrix}0 \\ 0 \\ 1 \end{pmatrix}= \begin{pmatrix}c \\ f \\ i\end{pmatrix}?
That is, applying the linear transformation to the vectors in the ordered basis, in turn, gives you the columns of the matrix.
Here the linear transformation is T(p)= (x^2+ 1)p''(x)- xp'(x)+ 2p(x) (you have "2p'(x) but that looks suspicious. If that were correct why wouldn't it be "(2- x)p'(x)"?) and, in the first problem, the basis vectors are x^2,x, and 1. T(x^2)= (x^2+ 1)(2)- x(2x)+ 2x^2=x^2+ 1. That would be represented by the column matrix \begin{pmatrix}1 \\ 0 \\ 1\end{pmatrix} and that is the first column of the matrix. T(x)=(x^2+ 1)(0)- x(1)+ 2(x)= x which is represented by the column matrix \begin{pmatrix}0 \\ 2 \\ 0\end{pmatrix} and that is the second column of the matrix. Finally, T(1)= (x^2+ 1)(0)- x(0)+ 2(1)= 2 which is represented by the column matrix \begin{pmatrix}0 \\ 0 \\ 2\end{pmatrix} and that is the third column of the matrix. The matrix representation of T in this ordered basis is \begin{pmatrix} 1 & 0 & 0 \\ 0 & 2 & 0 \\ 1 & 0 & 2 \end{pmatrix}.
In the second problem the ordered basis vectors are x^2+ x+ 1, x+1. and 1. T(x^2+ x+ 1)= (x^2+ 1)(2)- x(2x+ 1)+ 2(x^2+ x+ 1)= 2x^2+ 2- 2x^2- x+ 2x^2+ 2x+ 2= 2x^2+ x+ 4. But now we have to write that in terms of this basis. That is, we need to find a, b, and c so that a(x^2+ x+ 1)+ b(x+ 1)+ c(1)= ax^2+ (a+ b)x+ (a+ b+ c)= 2x^2+ x+ 4. We have a= 2, a+ b= 2+ b= 1 so b= -1 and a+ b+ c= 2- 1+ c= 1+ c= 4 so c= 3. The first column of the matrix is \begin{pmatrix} 2 \\ -1 \\ 3 \end{pmatrix}. T(x+ 1)= (x^2+ 1)(0)- x(1)+ 2(x+ 1)= x+ 1. Well that's easy! x+ 1= 0(x^2+ x+ 1)+ 1(x+ 1)+ 0(1) so the second column of the matrix is \begin{pmatrix}0 \\ 1 \\ 0\end{pmatrix}. T(1)= (x^2+ 1)(0)- x(0)+ 2(1)= 2. That can be written 2= 0(x^2+ x+ 1)+ 0(x+ 1)+ 2 which corresponds to the column matrix \begin{pmatrix}0 \\ 0 \\ 2\end{pmatrix}. The matrix representing T in this ordered matrix is \begin{pmatrix} 2 & 0 & 0 \\ -1 & 1 & 0 \\ 3 & 0 & 2\end{pmatrix}.
