- #1

- 611

- 117

- Homework Statement
- Example 27.1 from David MacKay Inference book: A photon counter is pointed at a remote start for one minute, in order to infer the rate of photons arriving at the counter per minute, ## \lambda ##. Assuming the number of photons collected ## r ## has a Poisson distribution with mean ## \lambda ##,

[tex] P(r|\lambda) = exp(-\lambda) \frac{\lambda ^ r}{r!} [/tex]

and assuming the prior ## P(\lambda) = \frac{1}{\lambda} ##, make Laplace approximation to the posterior distribution: (b) Over ## log(\lambda) ##

- Relevant Equations
- Laplace Approximation Formula

Hi,

I was attempting

[EDIT]: The link to the book website (official) is here:

I understand the process of the Laplace approximation as taking an unnormalized distribution from the integral of interest: ## \int f(x) dx ##

1. Calculate the mode of ## f ##

2. Calculate ## |\frac{\partial^2 (log(f))}{\partial x^2}| ##

3. Calculate ## Z = f(mode) \cdot \sqrt{\frac{2\pi}{|\frac{\partial^2(log( f))}{\partial x^2}|}} ##

At a high level, there are two stages to my attempt: 1) the variable substitutions, 2) the Laplace approximation

I started by making the variable change: ## W = log(\lambda) ##, where I have taken log as the natural logarithm. Therefore, we can calculate the prior in terms of ## W ##: ## p(W) = |\frac{d\lambda}{dW}| p(\lambda) = \lambda \cdot \frac{1}{\lambda} = 1 ##.

Now we substitute variables into the expression for ## p(r|\lambda) ##:

[tex] p(W | \lambda) = e^{-e^{W}} \cdot \frac{e^{Wr}}{r!} [/tex]

However, did I need to also include a factor of ## |\frac{d\lambda}{dW}| ## in the above expression? Also, in the 'integral' of interest (which I am just imagining as:

[tex] \int p(r | \lambda) p(\lambda) d\lambda [/tex]

for the original problem. Then when I make the substitutions do I need to include the extra factor obtained when changing the ## d\lambda ## to ## \frac{d\lambda}{dW} dW = \lambda dW ##?

In my attempt, I did not include extra ## \lambda ## factors from the substitution into ## p(W | \lambda) ## or ## d\lambda ## because I was unsure.

So the posterior distribution is ## p(W | r) = \frac{p(r|W) p(\lambda)}{p(r)} \propto p(r|W) p(W) ##. From step 1 of the Laplace approximation process, I then found the mode of this expression by taking natural logs of both sides, differentiating, and setting equal to zero (work omitted to save from overcrowding post). I got a modal value of: ## W_{mode} = log(r) ## where log is the natural logarithm.

From step 2, I then calculated the second derivative and substituted in the modal value of ## W ## to get:

[tex] |\frac{\partial^2 (log(f))}{\partial W^2}| = r [/tex]

Then I substituted into step 3 to get:

[tex] Z = f(mode) \cdot \sqrt{\frac{2\pi}{|\frac{\partial^2(log( f))}{\partial x^2}|}} = \frac{e^{-r} \cdot r^{r}}{r!} \cdot \sqrt{\frac{2\pi}{r}} [/tex].

Have I attempted this problem correctly? Did I make mistakes during the variable substitution phase?

Any help would be greatly appreciated

I was attempting

**question from the book: 'Information Theory, Inference, and Learning Algorithms'. It is about the Laplace approximation. I was confused about part (b) of the question and wanted to check my method if possible.**__example 27.1__[EDIT]: The link to the book website (official) is here:

__HERE__I understand the process of the Laplace approximation as taking an unnormalized distribution from the integral of interest: ## \int f(x) dx ##

1. Calculate the mode of ## f ##

2. Calculate ## |\frac{\partial^2 (log(f))}{\partial x^2}| ##

3. Calculate ## Z = f(mode) \cdot \sqrt{\frac{2\pi}{|\frac{\partial^2(log( f))}{\partial x^2}|}} ##

**Attempt:**At a high level, there are two stages to my attempt: 1) the variable substitutions, 2) the Laplace approximation

__Stage 1)__I started by making the variable change: ## W = log(\lambda) ##, where I have taken log as the natural logarithm. Therefore, we can calculate the prior in terms of ## W ##: ## p(W) = |\frac{d\lambda}{dW}| p(\lambda) = \lambda \cdot \frac{1}{\lambda} = 1 ##.

Now we substitute variables into the expression for ## p(r|\lambda) ##:

[tex] p(W | \lambda) = e^{-e^{W}} \cdot \frac{e^{Wr}}{r!} [/tex]

However, did I need to also include a factor of ## |\frac{d\lambda}{dW}| ## in the above expression? Also, in the 'integral' of interest (which I am just imagining as:

[tex] \int p(r | \lambda) p(\lambda) d\lambda [/tex]

for the original problem. Then when I make the substitutions do I need to include the extra factor obtained when changing the ## d\lambda ## to ## \frac{d\lambda}{dW} dW = \lambda dW ##?

In my attempt, I did not include extra ## \lambda ## factors from the substitution into ## p(W | \lambda) ## or ## d\lambda ## because I was unsure.

__Stage 2__So the posterior distribution is ## p(W | r) = \frac{p(r|W) p(\lambda)}{p(r)} \propto p(r|W) p(W) ##. From step 1 of the Laplace approximation process, I then found the mode of this expression by taking natural logs of both sides, differentiating, and setting equal to zero (work omitted to save from overcrowding post). I got a modal value of: ## W_{mode} = log(r) ## where log is the natural logarithm.

From step 2, I then calculated the second derivative and substituted in the modal value of ## W ## to get:

[tex] |\frac{\partial^2 (log(f))}{\partial W^2}| = r [/tex]

Then I substituted into step 3 to get:

[tex] Z = f(mode) \cdot \sqrt{\frac{2\pi}{|\frac{\partial^2(log( f))}{\partial x^2}|}} = \frac{e^{-r} \cdot r^{r}}{r!} \cdot \sqrt{\frac{2\pi}{r}} [/tex].

Have I attempted this problem correctly? Did I make mistakes during the variable substitution phase?

Any help would be greatly appreciated