Mean Reciprocal Rank (MRR) – A Great Way To Monitor Solr Search Quality

We’re excited to announce the availability of Mean Reciprocal Rank (MMR) analytics as part of SearchStax Analytics. MRR is a great way to monitor your Solr search quality trends over time. MRR is a simple calculation based on clicks and click position data  in your search results. It’s pretty obvious that users who are clicking on an item at the top of the search result page is better than selecting an item lower on the page. MRR tries to take this into account by valuing higher clicks more than lower clicks.
MMR is a weighted average of click-through positions ranging from zero to one (bigger being better). A perfect score is 1.0, which would mean that your search engine put the right answer at the top of the result list every single time!  (Nobody is that good, but we can strive for it.)

In calculating MRR, we value each click with the reciprocal of it’s list position e.g. click on the first item in the list and we’d value it as 1, the second item would be valued at 1/2, the third item would be 1/3, and so on. We then take all of these values and divide by the total number to get the mean reciprocal rank.  An MRR of 1 is ideal – as that means all of your users clicked the very first item returned on your SERP page.

We find MRR to be useful for the following scenarios:

Tracking trends over time

As search professionals, we are constantly tweaking the search algorithm to suit the customer’s use case, searchers, and content repository.  We’ll want to see MRR improvement over time to ensure we’re heading in the right direction with Solr Search Quality.

A/B Testing

If you are testing variations of search algorithms or search engines themselves. You can utilize MRR as a means to evaluate the effective relevance to your audience and give you a quantitative way to compare the variants.

For additional information, see the MMR discussion in the SearchStax User’s Guide.  Click the link to learn more about SearchStax, the leading Solr-as-a-Service platform.

About the Author

Recommended for you