Generative vs discriminative model

[pamparana]

Hello everyone,

I have a question about generative and discriminative models. As I understand it generative models aim to model the joint distribution p(x,y) of the input x and output y and the discriminative approach estimates the conditional distribution p(y|x). I understand that the generative approach is more complex but I have a small niggle...

Applying Bayes rule, we have the conditional distribution

p(y|x) = p(x|y) p(y)/p(x)

which is equal to:

P(y|x) = p(x, y)/p(x)

So it seems when we want to model discriminative modelling, we have to estimate this joint distribution (in he numerator) as well. I am sure this is wrong and I have a flaw in my understanding but I cannot seem to figure it out. Is there something about this modelling process that I am not understanding. Perhaps we do not estimate it in this way?

Also, a markov random field defined over an undirected graph, is it a joint distribution?

I would really appreciate any help anyone can give me on this.

Thanks,
Luc

 

[atyy]

P(x|y) has less information than P(x,y), because you can get the former from the latter, but not the latter from the former. For certain data sets, one can estimate parameters for a model of P(x|y) without the intermediate step of modelling P(x,y).

There is an interesting discussion on the advantages and disadvantages of explicitly learning P(x,y) as an intermediate step in modelling P(x|y) in http://ai.stanford.edu/~ang/papers/nips01-discriminativegenerative.pdf.