- #1
raul_l
- 105
- 0
I've written a Levenberg-Marquardt nonlinear optimization routine that employs the reverse-mode automatic differentiation algorithm for building the Jacobian. So far it has worked marvelously for me.
However, now I have to use functions that contain integrals that cannot be analytically taken and have to be numerically calculated, i.e.
f(x,\mathbf{p}) = g(x,\mathbf{p}) \int^{b(\mathbf{p})}_{a(\mathbf{p})}{h(x,\mathbf{p},t)dt}
where p is the parameter vector, f is the fitting function, g is some function of x and p and h is the integrand which is also a function of x and p and to be integrated over t.
Now, most of this can be easily differentiated if I just define the derivatives of elementary functions and take some time to write the rest of AD implementation (which I have done in C++). However, when I use the identity
\nabla_{ \mathbf{p} } \int^{b(\mathbf{p})}_{a(\mathbf{p})}{h(x,\mathbf{p},t)dt} = h(x,\mathbf{p},b(\mathbf{p}))\nabla_{\mathbf{p}}b(\mathbf{p}) - h(x,\mathbf{p},a(\mathbf{p}))\nabla_{\mathbf{p}}a(\mathbf{p}) + \int^{b(\mathbf{p})}_{a(\mathbf{p})}{\nabla_{ \mathbf{p} } h(x,\mathbf{p},t)dt}
to attack the integral I get stuck with the last term. This is my question: Is it possible to use the reverse-mode automatic differentiation to calculate integrals of type
\int^{b(\mathbf{p})}_{a(\mathbf{p})}{\nabla_{ \mathbf{p} } h(x,\mathbf{p},t)dt}
where I first have to calculate partial derivatives with respect to the fitting parameters and then integrate? (The bounds could also be constants, it doesn't matter.)
I've done some googling and come across articles that discuss the application of automatic differentiation to numerical integration algorithms for ordinary differential equations (ODEs). But this is something else because here I explicitly have to calculate the value of the integral. So no help so far. I've done some thinking and it appears that reverse-mode can't be used to handle the integrals, but forward-mode might be doable. Any thoughts from experts?
However, now I have to use functions that contain integrals that cannot be analytically taken and have to be numerically calculated, i.e.
f(x,\mathbf{p}) = g(x,\mathbf{p}) \int^{b(\mathbf{p})}_{a(\mathbf{p})}{h(x,\mathbf{p},t)dt}
where p is the parameter vector, f is the fitting function, g is some function of x and p and h is the integrand which is also a function of x and p and to be integrated over t.
Now, most of this can be easily differentiated if I just define the derivatives of elementary functions and take some time to write the rest of AD implementation (which I have done in C++). However, when I use the identity
\nabla_{ \mathbf{p} } \int^{b(\mathbf{p})}_{a(\mathbf{p})}{h(x,\mathbf{p},t)dt} = h(x,\mathbf{p},b(\mathbf{p}))\nabla_{\mathbf{p}}b(\mathbf{p}) - h(x,\mathbf{p},a(\mathbf{p}))\nabla_{\mathbf{p}}a(\mathbf{p}) + \int^{b(\mathbf{p})}_{a(\mathbf{p})}{\nabla_{ \mathbf{p} } h(x,\mathbf{p},t)dt}
to attack the integral I get stuck with the last term. This is my question: Is it possible to use the reverse-mode automatic differentiation to calculate integrals of type
\int^{b(\mathbf{p})}_{a(\mathbf{p})}{\nabla_{ \mathbf{p} } h(x,\mathbf{p},t)dt}
where I first have to calculate partial derivatives with respect to the fitting parameters and then integrate? (The bounds could also be constants, it doesn't matter.)
I've done some googling and come across articles that discuss the application of automatic differentiation to numerical integration algorithms for ordinary differential equations (ODEs). But this is something else because here I explicitly have to calculate the value of the integral. So no help so far. I've done some thinking and it appears that reverse-mode can't be used to handle the integrals, but forward-mode might be doable. Any thoughts from experts?