1. The problem statement, all variables and given/known data
I have this problem, but I'm not familiar with the function notation that the professor is using. Can anyone tell me what is actually being asked? I understand everything up to the part that is in bold, but after that, I am lost.

Some of the formating was lost so Rn is shown as R^n and xT is x transpose.

x and y are in R^{n}, so they are n by 1 matrices so x^{T}Ay is (1 by n)(n by n)(n by 1) = 1 by 1 or scalar. If you put in A = B^{T}B I think you will see that it is a dot product of two vectors if you look at it right.

I would interpret the problem as "Show that d satisfies the definition of an inner product". This is really easy if you know the definition and you're comfortable with matrix algebra (stuff like [itex](XY)^T=Y^T X^T[/itex]).

If you're only supposed to show that [itex]x^TAy[/itex] is a dot product of two vectors, the complete solution would be [itex]x^TAy=x^T(Ay)[/itex], because Ay is a column vector. (You know that the definition of matrix multiplication implies that [itex]u^Tv=u_1v_1+\dots+u_nv_n[/itex] when u and v are column matrices, right?)