Estimating joint distributions from marginal

[exmachina]

Suppose I have the marginal probability density functions of two random variables A and B, P(A), and P(B). Suppose I modeled P(A) and P(B) using a mixture model from some dataset D and obtained a closed form pdf for each.

I am interested in finding their joint density function P(A and B) and associated properties such as maximas, minimas, etc.

Ideally the joint density is expressed as a closed form 2D mixture model as well, but this is not critical.

I could do something perhaps by brute force by use of Baye's theorem:

ie. I can approximate

P(A and B) = P(A) P(B | A) = P(B) | P(A | B)

But eventually I need to extend this to higher dimensions, eg. P( A and B and C and D... etc) and this is certainly no trivial task.

 

[Stephen Tashi]

In general, you cannot determine a joint probability distribution when given only the marginal probability distributions, so if your problem can be solved the solution depends on special circumstances or information that you haven't mentioned. To get the best advice, you should describe the situation completely.

I could do something perhaps by brute force by use of Baye's theorem:

ie. I can approximate

P(A and B) = P(A) P(B | A) = P(B) | P(A | B)

That isn't a mere approximation. It is a theorem.

Are you saying that you have data that could be used estimate the conditional probability distributions?

 

[exmachina]

Well A and B are two variables that specify (completely) the state of the system. Suppose I've sampled a whole bunch of data points (a,b) s.t. I can generate their PDFs.

I can approximate P(B | A=a1) and P(A | B=b1) as well by taking a slice of my dataset, (eg. B= b1+-0.1) and count the occurrences of A. However, this can be bad because my entire dataset may be quite small, and using only a subset of it will result in a lot of noise and error.

 

[Stephen Tashi]

this can be bad because my entire dataset may be quite small, and using only a subset of it will result in a lot of noise and error.

I think the only convenient cure for small amounts of data is to build-in a lot of structure to the answer - for example, you might assume the distribution you are trying to determine is from a family of distributions that are defined by only a few parameters and estimate those parameters from the data. To do this you must employ any expert knowledge that you have about the situation. For example you may know that certain families of distributions have a plasusible shape and others don't.

You haven't described the problem clearly, but from your remarks, I conjecture that you are dealing with continuous variates. Some technicalities about your terminology: The value of a continuous probability density function does not give "the probability of" particular values. (For example, think about the uniform probability density function on the interval [0, 1/2] which has constant value 2.) However, I agree that it is often helpful to think about density functions informally that way. Observed frequences of values in a sample are not probabilities (unless you are taking about randomly selecting a value from the sample itself.) So you shouldn't use the p(A|B) notation for them. Of course, observed frequencies can be used as estimators of probabilities.

 

[blue_raver22]

I understand your interest in finding the joint density function and associated properties of two random variables A and B. It is a crucial step in understanding the relationship between these variables and making predictions based on their values.

Using a mixture model to model the marginal probability density functions of A and B is a valid approach. However, as you mentioned, finding the joint density function for higher dimensions can be a challenging task. One potential solution is to use Bayes' theorem, as you suggested, to approximate the joint density. This can be a useful approach, but it may not always yield accurate results.

Another approach could be to use numerical methods, such as Monte Carlo simulations, to estimate the joint density function. This involves generating a large number of random samples from the marginal distributions and calculating the joint probability for each sample. By repeating this process multiple times, you can obtain a more accurate estimation of the joint density function.

Additionally, there are other statistical methods, such as copulas, that can be used to model joint distributions from marginal distributions. These methods take into account the dependence between variables and can provide a more accurate representation of the joint density function.

In summary, estimating joint distributions from marginal distributions can be a challenging task, especially for higher dimensions. It is important to consider various methods and approaches, such as Bayes' theorem, numerical methods, and statistical models, to obtain the most accurate estimation of the joint density function and its associated properties.