Monitor Solr Search Quality with MRR – Mean Reciprocal Rank

What is MRR?

MRR is a simple calculation based on clicks and click position data  in your search results. It’s pretty obvious that users who are clicking on an item at the top of the search result page is better than selecting an item lower on the page. MRR tries to take this into account by valuing higher clicks more than lower clicks.

Calculating MRR

MMR is a weighted average of click-through positions ranging from zero to one (bigger being better). A perfect score is 1.0, which would mean that your search engine put the right answer at the top of the result list every single time!  While no search results are that good, it is something that you can strive for it.

In calculating MRR, we value each click with the reciprocal of it’s list position e.g. click on the first item in the list and we’d value it as 1, the second item would be valued at 1/2, the third item would be 1/3, and so on. We then take all of these values and divide by the total number to get the mean reciprocal rank.  An MRR of 1 is ideal – as that means all of your users clicked the very first item returned on your SERP page.

Monitor Solr Search Quality with MRR With These Scenarios

We find MRR to be useful for the following scenarios:

Tracking Trends Over Time

As search professionals, we are constantly tweaking the search algorithm to suit the customer’s use case, searchers, and content repository.  We’ll want to see MRR improvement over time to ensure we’re heading in the right direction with Solr Search Quality.

If you are testing variations of search algorithms or search engines themselves. You can utilize MRR as a means to evaluate the effective relevance to your audience and give you a quantitative way to compare the variants.