MacKay Textbook Example: Laplace Approximation

[Master1022]

Homework Statement

Example 27.1 from David MacKay Inference book: A photon counter is pointed at a remote start for one minute, in order to infer the rate of photons arriving at the counter per minute, $\lambda$. Assuming the number of photons collected $r$ has a Poisson distribution with mean $\lambda$,
$$P(r|\lambda) = exp(-\lambda) \frac{\lambda ^ r}{r!}$$
and assuming the prior $P(\lambda) = \frac{1}{\lambda}$, make Laplace approximation to the posterior distribution: (b) Over $log(\lambda)$

Relevant Equations

Laplace Approximation Formula

Hi,

I was attempting example 27.1 question from the book: 'Information Theory, Inference, and Learning Algorithms'. It is about the Laplace approximation. I was confused about part (b) of the question and wanted to check my method if possible.

[EDIT]: The link to the book website (official) is here: HERE

I understand the process of the Laplace approximation as taking an unnormalized distribution from the integral of interest: $\int f(x) dx$
1. Calculate the mode of $f$
2. Calculate $|\frac{\partial^2 (log(f))}{\partial x^2}|$
3. Calculate $Z = f(mode) \cdot \sqrt{\frac{2\pi}{|\frac{\partial^2(log( f))}{\partial x^2}|}}$

Attempt:
At a high level, there are two stages to my attempt: 1) the variable substitutions, 2) the Laplace approximation

Stage 1)
I started by making the variable change: $W = log(\lambda)$, where I have taken log as the natural logarithm. Therefore, we can calculate the prior in terms of $W$: $p(W) = |\frac{d\lambda}{dW}| p(\lambda) = \lambda \cdot \frac{1}{\lambda} = 1$.

Now we substitute variables into the expression for $p(r|\lambda)$:
$$p(W | \lambda) = e^{-e^{W}} \cdot \frac{e^{Wr}}{r!}$$

However, did I need to also include a factor of $|\frac{d\lambda}{dW}|$ in the above expression? Also, in the 'integral' of interest (which I am just imagining as:
$$\int p(r | \lambda) p(\lambda) d\lambda$$
for the original problem. Then when I make the substitutions do I need to include the extra factor obtained when changing the $d\lambda$ to $\frac{d\lambda}{dW} dW = \lambda dW$?

In my attempt, I did not include extra $\lambda$ factors from the substitution into $p(W | \lambda)$ or $d\lambda$ because I was unsure.

Stage 2
So the posterior distribution is $p(W | r) = \frac{p(r|W) p(\lambda)}{p(r)} \propto p(r|W) p(W)$. From step 1 of the Laplace approximation process, I then found the mode of this expression by taking natural logs of both sides, differentiating, and setting equal to zero (work omitted to save from overcrowding post). I got a modal value of: $W_{mode} = log(r)$ where log is the natural logarithm.

From step 2, I then calculated the second derivative and substituted in the modal value of $W$ to get:
$$|\frac{\partial^2 (log(f))}{\partial W^2}| = r$$

Then I substituted into step 3 to get:
$$Z = f(mode) \cdot \sqrt{\frac{2\pi}{|\frac{\partial^2(log( f))}{\partial x^2}|}} = \frac{e^{-r} \cdot r^{r}}{r!} \cdot \sqrt{\frac{2\pi}{r}}$$.

Have I attempted this problem correctly? Did I make mistakes during the variable substitution phase?

Any help would be greatly appreciated

 

[mighty2000]

.

Hi,

Thank you for your question. It seems like you have a good understanding of the Laplace approximation process. However, there are a few areas that could use some clarification.

In stage 1, you correctly substituted for the variable W, but you also need to include the Jacobian term in the expression for p(W|lambda). This is because when you make a change of variables, the probability density function changes accordingly. In this case, the Jacobian term is simply $\lambda$. So the correct expression for p(W|lambda) would be:

p(W | \lambda) = \lambda e^{-e^{W}} \cdot \frac{e^{Wr}}{r!}

Similarly, when you substitute for dlambda in the integral of interest, you need to include the Jacobian term. So the integral would be:

\int p(r | \lambda) p(\lambda) d\lambda = \int e^{-e^{W}} \cdot \frac{e^{Wr}}{r!} \cdot \lambda dW

In your attempt, you did not include the Jacobian term in either of these expressions, which would lead to incorrect results.

In stage 2, you correctly found the mode of the posterior distribution. However, the second derivative of log(f) with respect to W should be $r - 1$, not just r. This is because the second derivative of $log(e^{Wr})$ is r, but we also have the additional term $-log(r!)$ which contributes a -1 to the second derivative. So the correct expression for the second derivative would be:

|\frac{\partial^2 (log(f))}{\partial W^2}| = r - 1

Substituting this into step 3 would give you the correct result for Z.

Overall, it seems like you have a good understanding of the Laplace approximation process. Just remember to include the Jacobian term when making variable substitutions and to properly account for the additional term in the second derivative when calculating Z. I hope this helps! If you have any further questions, please let me know.