feat(search): Use minimum_should_match to filter hits #1333
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This change utilizes the [minimum_should_match](https://www.elastic.co/guide/en/elasticsearch/reference/5.6/query-dsl-minimum-should-match.html) Elasticsearch query parameter to reduce the number of hits to search queries. Previously, only one token in an input had to match, regardless of the input size. This allows for queries to potentially match a huge number of documents, especially as the number of tokens grows. Now, most `ngrams` queries have thee parameter set to `1<-1 3<-25%` Breaking this down, it means that for the given number of tokens in a query, there is a certain number of optional tokens: | token count | optional tokens | | --- | --- | | 1 | 0 (obviously) | | 2 | 1 | | 3 | 1 | | 4 | 1 | | 5 | 1 | | 6+ | at least 75% of tokens must match | This should help ensure quality results are returned where possible, even if long inputs contain a few extraneous bits of information, while reducing the chance of extremely expensive queries.
|
We saw some very nice reductions in request latency and number of Elasticsearch hits due to this change. It looks like it will really help cut down on slow queries. The effect of rolling this out was pretty dramatic: |
orangejulius
added a commit
to pelias/acceptance-tests
that referenced
this pull request
Aug 4, 2019
It fails since pelias/api#1333
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This change utilizes the minimum_should_match Elasticsearch query parameter to reduce the number of hits to search queries. For now, autocomplete is not changed, we can take a look at that later.
Previously, only one token in an input had to match, regardless of the input size. This allows for queries to potentially match a huge number of documents, especially as the number of tokens grows.
Now, most
ngramsqueries have theminimum_should_matchparameter set to1<-1 3<-25%Breaking this down, it means that the number of optional tokens follows this pattern:
This should help ensure quality results are returned where possible, even if long inputs contain a few extraneous bits of information, while reducing the chance of extremely expensive queries.