Dismiss Notice
Join Physics Forums Today!
The friendliest, high quality science and math community on the planet! Everyone who loves science is here!

Generative vs discriminative model

  1. May 8, 2013 #1
    Hello everyone,

    I have a question about generative and discriminative models. As I understand it generative models aim to model the joint distribution p(x,y) of the input x and output y and the discriminative approach estimates the conditional distribution p(y|x). I understand that the generative approach is more complex but I have a small niggle...

    Applying Bayes rule, we have the conditional distribution

    p(y|x) = p(x|y) p(y)/p(x)

    which is equal to:

    P(y|x) = p(x, y)/p(x)

    So it seems when we want to model discriminative modelling, we have to estimate this joint distribution (in he numerator) as well. I am sure this is wrong and I have a flaw in my understanding but I cannot seem to figure it out. Is there something about this modelling process that I am not understanding. Perhaps we do not estimate it in this way?

    Also, a markov random field defined over an undirected graph, is it a joint distribution?

    I would really appreciate any help anyone can give me on this.

  2. jcsd
  3. May 9, 2013 #2


    User Avatar
    Science Advisor

    P(x|y) has less information than P(x,y), because you can get the former from the latter, but not the latter from the former. For certain data sets, one can estimate parameters for a model of P(x|y) without the intermediate step of modelling P(x,y).

    There is an interesting discussion on the advantages and disadvantages of explicitly learning P(x,y) as an intermediate step in modelling P(x|y) in http://ai.stanford.edu/~ang/papers/nips01-discriminativegenerative.pdf.
    Last edited: May 9, 2013
Share this great discussion with others via Reddit, Google+, Twitter, or Facebook