# MacKay Textbook Example: Laplace Approximation

Master1022
Homework Statement:
Example 27.1 from David MacKay Inference book: A photon counter is pointed at a remote start for one minute, in order to infer the rate of photons arriving at the counter per minute, ## \lambda ##. Assuming the number of photons collected ## r ## has a Poisson distribution with mean ## \lambda ##,
$$P(r|\lambda) = exp(-\lambda) \frac{\lambda ^ r}{r!}$$
and assuming the prior ## P(\lambda) = \frac{1}{\lambda} ##, make Laplace approximation to the posterior distribution: (b) Over ## log(\lambda) ##
Relevant Equations:
Laplace Approximation Formula
Hi,

I was attempting example 27.1 question from the book: 'Information Theory, Inference, and Learning Algorithms'. It is about the Laplace approximation. I was confused about part (b) of the question and wanted to check my method if possible.

[EDIT]: The link to the book website (official) is here: HERE

I understand the process of the Laplace approximation as taking an unnormalized distribution from the integral of interest: ## \int f(x) dx ##
1. Calculate the mode of ## f ##
2. Calculate ## |\frac{\partial^2 (log(f))}{\partial x^2}| ##
3. Calculate ## Z = f(mode) \cdot \sqrt{\frac{2\pi}{|\frac{\partial^2(log( f))}{\partial x^2}|}} ##

Attempt:
At a high level, there are two stages to my attempt: 1) the variable substitutions, 2) the Laplace approximation

Stage 1)
I started by making the variable change: ## W = log(\lambda) ##, where I have taken log as the natural logarithm. Therefore, we can calculate the prior in terms of ## W ##: ## p(W) = |\frac{d\lambda}{dW}| p(\lambda) = \lambda \cdot \frac{1}{\lambda} = 1 ##.

Now we substitute variables into the expression for ## p(r|\lambda) ##:
$$p(W | \lambda) = e^{-e^{W}} \cdot \frac{e^{Wr}}{r!}$$

However, did I need to also include a factor of ## |\frac{d\lambda}{dW}| ## in the above expression? Also, in the 'integral' of interest (which I am just imagining as:
$$\int p(r | \lambda) p(\lambda) d\lambda$$
for the original problem. Then when I make the substitutions do I need to include the extra factor obtained when changing the ## d\lambda ## to ## \frac{d\lambda}{dW} dW = \lambda dW ##?

In my attempt, I did not include extra ## \lambda ## factors from the substitution into ## p(W | \lambda) ## or ## d\lambda ## because I was unsure.

Stage 2
So the posterior distribution is ## p(W | r) = \frac{p(r|W) p(\lambda)}{p(r)} \propto p(r|W) p(W) ##. From step 1 of the Laplace approximation process, I then found the mode of this expression by taking natural logs of both sides, differentiating, and setting equal to zero (work omitted to save from overcrowding post). I got a modal value of: ## W_{mode} = log(r) ## where log is the natural logarithm.

From step 2, I then calculated the second derivative and substituted in the modal value of ## W ## to get:
$$|\frac{\partial^2 (log(f))}{\partial W^2}| = r$$

Then I substituted into step 3 to get:
$$Z = f(mode) \cdot \sqrt{\frac{2\pi}{|\frac{\partial^2(log( f))}{\partial x^2}|}} = \frac{e^{-r} \cdot r^{r}}{r!} \cdot \sqrt{\frac{2\pi}{r}}$$.

Have I attempted this problem correctly? Did I make mistakes during the variable substitution phase?

Any help would be greatly appreciated