Thanks for the references! The TREC IR Measures overview seems to be exactly what I was looking for.

The big web search engines, in particular, are concerned with precision “above the fold” (in the newspaper sense). That is, if you take a default install of IE or Firefox and do a search on Bing, Yahoo, or Google, what’s the precision for the number of results you can see on the screen. The chance to continue browsing is not continuous. Basically, that would induce a kink in your probability of keeping going to the next items.

There are also applications which are highly recall oriented, like curating databases of protein interactions from the literature or intelligence analysis over the news. It’d still fit your model, it’d just give you a different likelihood of looking at another item. We’re particularly interested in precision at 99% recall or 99.9% recall for these situations.

The other major issue to consider is diversity of results. If I send you ten different versions of the same information, it’s not very useful even if they’re all “relevant” in the binary relevant/not relevant sense. The problem is that to measure this idea, you need a notion of relative information contribution of a new result given a set of other results.

What the IR folks call precision is what the epidemiologists call “positive predictive accuracy”. It’s basically the likelihood that you have a condition if you test positive for it, and it’s very useful exactly as stated in that context.

You might want to consult the section of the Wikipedia entry Information Retrieval about performance measures, or the TREC IR Measures overview.

What he ended up with was a similar idea, but base on the information theoretic entropy. He called it relevance information.

If we assume R relevant documents in a corpus of D documents, the probability of any given document being relevant is uniformly R/D. If we score and then rank them by some procedure we can then assume the probability is no longer uniform, but decreasing (hopefully) sharply over the rankings. The effectiveness of the scoring scheme is measured by the decrease in entropy over the whole distribution.

He examined some of the properties of this measure and felt that it captured the best of both precision and recall in one number and was reasonably robust, but AFAIK it never caught on in the literature.

I can’t find the original paper, and he seems to have moved away from it in any of his more recent papers.

(Disclosure — I used to work for him)

Wait — I found it:

An Information Measure of Retrieval Performance (1992)
by W J Wilbur

http://citeseerx.ist.psu.edu/showciting?cid=1837654

I agree, although the precision metric for a given recall level can be calculated using the same machinery.

I’m happy to learn though. References are welcome! I hardly suspect that I came up with anything novel during a 2 hour whiteboard session two years back and a similar amount of time hacking things up in perl. ;)

I just haven’t seen anything similar written up.

You will occasionally also see precision-at-n for various n which are multiples of a hypothetical screenful (e.g. n=20).

Yeah, you’ve got the metric. I used this once a couple of years back to pretty good effect.

That is a very good point about diversity. The metric I actually used did a much simpler version of what you proposed in that I just considered a pre-ranked training set. And if you found an item that should be ranked up to, say, 4th in the 3rd slot you earned 3/4ths of the points, clamped to 1. Ties can them be allowed in the list of expected rankings when the distinctions between them aren’t clear. This doesn’t address diversity directly, but it does ensure that you can reach 100% precision given perfect ordering.

Another interesting enhancement for this metric is that the R function needn’t select only from {0,1}. One particular direction to take this is to define R(n) dynamically in terms of R(0)..R(n-1) such that, for example, duplicate content is not considered relevant. This would allow for a more information theoretic approach where we want to maximize the diversity of content up front rather than presenting the same content over and over again. This is especially important if we assume the searcher will read from multiple documents, but it also provides some stability if we don’t trust our ability to discern relevance.