Dictatorial control over recipe search results using elasticsearch and function_score
Once the design for the seasonal recipes app started coming into place, we soon saw there was something fishy about the results. Elasticsearch and custom relevancy to the rescue!
Warning! Dynamic scripting has been disabled by default in elasticsearch version 1.4.3. Using the technique in this article now requires some extra steps. Details on the Elastic blog.
Our design explains that we need results to be sorted by number of ingredients, and then by date, with the most recent recipes on top, scoring recipes from 2008 at bottom. The original attempt involved a simple “terms” query.
Investigating the results for July, in the garden, gave us recipes for jam at the top, with the count of matching ingredients being only 3 for the top hit. The list of ingredients in season for July in the garden is quite long, but all you need to know is that it contains “rips“, a little red sour berry, and “poteter”, that is potatoes.
You can find the queries used in this post at http://sense.qbox.io
For this query, we were some what surprised that none of the recipes on top contained potatoes. Using the highlight function, it is easy to see that the number of ingredients returned for the top hits should have been higher. Then it dawned upon us: TF-IDF! You’re messing up again! What we see is just the normal relevancy, promoting the least commonly used terms to the top. This works well for natural language queries, but that’s not really what we are doing here.
Luckily, elasticsearch doesn’t leave us stuck in a rut. We implemented a custom scoring function using the function score query.
For this scoring, we don’t need any points from the query terms, we just want to replace the default scoring with our custom one (boost_mode = replace).
The scoring function has two parts, one where we add up the number of ingredients, and one part to add some boost to the most recent posts.
The function to sum up term frequencies looks like this:
The tf() function returns the term frequency for a term in this field. There are a number of functions you can use to perform your own calculations based on index properties. The functions available are documented in the text scoring in scripts and the scripting module.
To the term frequency we add the date scoring:
We are using a linear function, but we could also have used gauss or exponential curves.
The “scale” parameter decides the point on the graph where the value specified in “decay” should be found. Setting it to 700 days allows the scoring to reach 0 for recipes dated in 2008, which was one of the requirements.
The number of ingredients will always be whole numbers, while the date scoring is normalized to values from 1 to 0.
To allow the scoring from both functions to add up, we use the parameter score_mode=sum.
Elasticsearch is an extremely powerful toolbox for search, information retrieval, analytics, big data, you name it. The possibilites are endless.
If you want to learn more about custom scoring in elasticsearch, there are some nice videos you can watch:
“Scoring for human beings” by Britta Weber, Berlin Buzzwords 2014
http://www.elasticsearch.org/videos/introducing-custom-scoring-functions/
Related Posts
Relevance tuning in the search domain. What is it exactly?
How to visualize absolute search result quality
The seasonal recipe app: Tapping into the mental model
Tech choices for the seasonal recipes app: A skeleton architecture for working with search relevancy
Dynamic search ranking using Elasticsearch, Neo4j and Piwik
Search templates in elasticsearch
finally, this is the first google result I’ve found that addresses the subject beyond the trivial popularity boost example, and provides links to the appropriate documentation. With this I’ve discovered I no longer need to write a custom similarity module. You are my hero today. Thank you for your positive contribution to the internet!!