Calculus of Variations on Kullback-Liebler Divergence

  • I
  • Thread starter Master1022
  • Start date
  • #1
Master1022
611
116
TL;DR Summary
How to use calculus of variations on KL-divergence
Hi,

This isn't a homework question, but a side task given in a machine learning class I am taking.

Question: Using variational calculus, prove that one can minimize the KL-divergence by choosing ##q## to be equal to ##p##, given a fixed ##p##.

Attempt:

Unfortunately, I have never seen calculus of variations (it was suggested that we teach ourselves). I have been trying to watch some videos online, but I mainly just see references to Euler-Lagrange equations which I don't think are of much relevance here (please correct me if I am wrong) and not much explanation of the functional derivatives.

Nonetheless, I think this shouldn't be too hard, but am struggling to understand how to use the tools.

If we start with the definition of the KL-divergence we get:
[tex] \text{KL}[p||q] = \int p(x) log(\frac{p(x)}{q(x)}) dx = I [/tex]

Would it be possible for anyone to help me get started on the path? I am not sure how to proceed really after I write down ## \frac{\delta I}{\delta q} ##?

Thanks in advance
 

Answers and Replies

  • #2
Office_Shredder
Staff Emeritus
Science Advisor
Gold Member
2021 Award
5,335
1,285
Euler Lagrange is what you want, but you also have to worry about the conditions that you have on q that come from it being a probability distribution, namely that the integral is 1 and it's always nonnegative. I think the integral constraint is the important part

http://liberzon.csl.illinois.edu/teaching/cvoc/node38.html

Has some notes on how to add constraints to the euler Lagrange equations.
 
  • #3
RuiGao
2
0
Maybe you should start with fixed p, and then try to get optimal q to maximize or minimize KL.
 

Suggested for: Calculus of Variations on Kullback-Liebler Divergence

  • Last Post
Replies
4
Views
629
  • Last Post
Replies
7
Views
802
  • Last Post
Replies
1
Views
625
  • Last Post
Replies
22
Views
743
Replies
4
Views
460
Replies
2
Views
873
Replies
7
Views
1K
  • Last Post
Replies
10
Views
2K
Replies
12
Views
949
Replies
20
Views
4K
Top