Baysian Evidence approximation

[Machiveli]

I'm using the laplace approximation (also known as MacKay's evidence framework) to the posterior volume of a baysian model.

The standard procedure is as follows:

1) Find the (local) maximum point of the posterior pdf i.e optimise the parameter values.
2) Evaluate the hessian matrix(H) by a second order Taylor's series approximation for the logarithm of the posterior pdf
3) If one were to fit a (multidimentional) gaussian with mean given my the local maximum and variance given by then the normalising constant would be 2pi()*sqrt(det(inverse(H))) and this is therefore the approximate volume of the integral.

Clearly this is only possible if the hessian is non singular possitive definate i.e. we have found a local maximum not a saddle point. If its a saddle point we just do ridge regression. However in practical applications I've found that it doesn't invert for one of two reasons.

A)Maximum a posteriori parameter values are at the boundary of possible parameter values.
B)Some parameters are redundant or colinear. Therefore there is no curvature in the corresponding directions.

Now are the following acceptable solutions?

In case A we only care about the integral within our boundary. We therefore pretend the hessian is symmetric about the boundary approximate the pdf using the gaussian but this time with a definate integral.

In case B we remove dimentions corresponding to redundant parameters. (the first argument being that if the parameters are redundant they can take any value so their integral is one. The second argument being that the evidence for a model should be independant of how it is written i.e. y=w1*x is should have identical evidence to y=(w1+w2)*x if flat priors are given to the parameters)

 

[mmwave]

I would say that both of these solutions are acceptable, but they may not be the most accurate or reliable. In case A, you are essentially assuming that the curvature of the posterior pdf is the same on both sides of the boundary, which may not always be the case. This could potentially lead to an underestimation of the integral, and therefore an underestimation of the posterior volume.

In case B, removing dimensions corresponding to redundant parameters may also lead to an underestimation of the integral, as you are essentially ignoring certain parameters that could potentially affect the posterior pdf. Additionally, the argument that the evidence for a model should be independent of how it is written is not always true, as different parameterizations can lead to different evidence values. Therefore, removing dimensions in this case could also potentially lead to an inaccurate estimation of the posterior volume.

In both cases, it is important to keep in mind that the Laplace approximation is just an approximation and may not always accurately represent the true posterior volume. It is always a good idea to check the results with other methods, such as Markov chain Monte Carlo (MCMC) sampling, to ensure the accuracy of the estimates.

 

[tacman]

Yes, both of these solutions are acceptable in the given cases. In case A, it is reasonable to assume symmetry around the boundary and approximate the pdf using a definite integral. In case B, removing dimensions corresponding to redundant parameters is a valid approach to deal with the issue of colinearity. As you mentioned, the evidence for a model should be independent of how it is written, so removing redundant parameters should not significantly affect the overall evidence. However, it is important to carefully consider which parameters are redundant and ensure that the remaining model is still valid and captures the necessary information. Overall, these solutions are practical and reasonable ways to deal with the limitations of using the Laplace approximation for Bayesian evidence approximation.