This week's TTF (Tom Tech Friday) is for the hard core geeks. I'll be talking about Search Relevancy Testing. Search Relevance is the measure of how relevant a given set of search results is for the given query. If you think about it you might come to the conclusion that search relevancy testing is subjective, because you don't always know what the user "wants" when they enter a search query. But push that thought out of your head for now... :-)
There are different types of search queries. Some of them are:
Navigational (specific) -> alaska airlines
Discovery (broad) -> airplane
Informational (directed toward a single topic) -> california fishing regulations
Transactional/Resource-Based -> u2 tickets "hp pavillion" "san jose"
Then there are forms of queries. Some of them are:
Terms Only -> san jose weather
Natural Language -> flights from sfo to jfk on american airlines
Question -> What is the traffic in Chicago like right now?
Domain Specific -> AKAIR7855
Metadata -> url:cnn.com topnews
You can rate the relevancy by manual means which is generally more accurate, but way more costly (you have to feed people but not machines). There are a few manual methods to doing this, they are:
Numeric Method -> rate results from 1 to 5
Binary Method -> thumbs up, thumbs down
Google Method -> vital, useful, relevant, off-topic, offensive
There are also algorithms for automatic relevancy evaluation. Some of them are:
Qrels Validation -> URI based, only works with static content
Trels (term relevance) Validation -> matches on-topic (+1) & off-topic (-1) with variations like off-topic multipiers and point per occurence.
Some of the metrics used include:
Precision (of the result set) = (# of Relevant Results)/(all results)
Instance Recall = Number of topics that are covered over the first N results.
Mean Search Length = Number of irrelevant results until the first relevant result
K-Call@N succeeds if the top N results have K relevant results (ex -> 1-call@10 succeeds if we find one or more relevant documents on the first page (10 results per page) of the result set.
A lot of companies, including mine, use automated methods and TREC data to do our Search Relevancy Testing. It is interesting stuff, but I think we could probably do more of it.
Posted by troutm8 at October 19, 2007 04:01 PM