Gradient of a function containing a matrix?

1. Sep 28, 2014

countzander

1. The problem statement, all variables and given/known data
http://i.imgur.com/TlDOllQ.png

2. Relevant equations
As stated.

3. The attempt at a solution

I'm not sure how to slay this beast. I know the gradient is just a partial derivative and that the solution likely involves multiple partial derivatives, one for each element in the vector x. But how would the partial derivatives be computed, given the matrices?

2. Sep 29, 2014

Ray Vickson

Just type out the problem here; do not use thumbnails. Read the pinned post 'Guidelines for students and helpers', by Vela, to see why. I cannot read your thumbnail on some media, so I will make no attempt to help.

3. Sep 29, 2014

TheFerruccio

Let $A \in\mathbb R^{m\times n}$ and $B \in\mathbb R^m$. Compute the gradient of$$f:\mathbb R^n\rightarrow\mathbb R,f(x)=\frac{1}{2}x^Tx+log\left(e^TE(Ax+b)\right),$$ where $e=\left(1,1,1,\ldots,1\right)\in\mathbb R^m$ and $E:\mathbb R^m\rightarrow\mathbb R^m$ is a component wise exponential function, i.e., $(E(x))_i=exp(x_i)$ for $i=1,2,\ldots m.$
Use $diag(v)$ for a $m\times m$ diagonal matrix with diagonal elements given by $v\in\mathbb R^m$

4. Sep 30, 2014

countzander

Thanks for the formatting help.

I attempted a solution by differentiating with respect to $x_n$.

$$\frac{\partial f}{\partial x_n} = 2x_n + \frac{A^T e^T exp(Ax+b)}{e^T exp(Ax+b)}$$

But this isn't correct, I don't think. Shouldn't $A^T$ cancel out somewhere? Can the gradient contain a matrix in the final solution?

5. Oct 1, 2014

Ray Vickson

Just write out the original function $f = f(x_1,x_2, \ldots, x_n)$ in detail so you can see what is happening; then, after finding the derivative you can re-write the answer using matrices again, if you want. So
$$f = \frac{1}{2} \sum_{i=1}^n x_i^2 + \log \left(\sum_{i=1}^n \exp (b_i + \sum_{j=1}^n a_{ij} x_j ) \right)$$
You could also use the fact that $\log(\sum_i (\exp(g_i)) = \prod_i g_i$, but I don't know if that makes things better or worse---you would need to try it for yourself. Also, you don't want just $\partial f / \partial x_n$, you want all the $\partial f / \partial x_k, k = 1, \dots,n$.

BTW: in TeX/LaTeX you should use "\exp" instead of "exp" and "\log" instead of "log"; this applies also to the other standard functions (the trig functions, the hyperbolic functions plus things like "max", "min", "mod", etc.) The results really do look better: you get $\exp$ instead of $exp$, $\log$ instead of $log$, etc.