Log-Likelihood ratio in the context of natural language processing

[starcoast]

First of all, let me apologize if this question is in the wrong place. It's fundamentally a statistics question but it relates to computer science. I'm also not sure if this falls under the "homework" category, since it's for a class, but I need assistance on a general idea, not a problem set. Anyway:

I am implementing some unsupervised methods of content-selection/extraction based document summarization and I'm confused about what my textbook calls the "log-likelihood ratio". The book briefly describes it as such:

"The LLR for a word, generally called lambda(w), is the ratio between the probability of observing w in both the input and in the background corpus assuming equal probabilities in both corpora, and the probability of observing w in both assuming different probabilities for w in the input and the background corpus."

Breaking that down, we have the numerator: "the probability of observing w in both the input and in the background corpus assuming equal probabilities in both corpora" - How do I calculate what probability to use here?

and the denominator: "the probability of observing w in both assuming different probabilities for w in the input and the background corpus". - is this as simple as the probability of the word occurring in the input times the probability of the word occurring in the corpus? ex:

(count(word,input) / total words in input) * (count(word,corpus) / total words in corpus)

I've been looking over a paper my book references, Accurate Methods for the Statistics of Surprise and Coincidence (Dunning 1993), but I'm finding it difficult to relate to the problem of calculating LLR values for individual words in extraction based summarization. Any clarification here would be really appreciated.

 

[Stephen Tashi]

I don't know the conventions used in document analysis and the passage you quoted isn't well written, so I can only guess at what is meant. My guess is that "the probability of observing w in both the input and in the background corpus assuming equal probabilities in both corpora" involves estimating "the probability that a randomly chosen word is w" by taking the ratio: ( total occurrences of w in input + total occurences of w in background corpus)/ (total words in input + total words in background corpus). My guess for the denominator would be the same as yours.

One problem with the quoted passage is that "the probability of observing w" depends on the sampling procedure. I am assuming that procedure is "pick one random word from a uniform probability distribution over all the words".

Another problem is that the topic is the "log" likelihood ratio, but the passage doesn't mention taking the logarithm of the ratio.

 

[D H]

Log likelihood ratio is a concept from Bayesian statistics. It is used quite frequently in Bayesian analyses.