Background Image for Strict and Permissive Analyzers

Permissive and Strict Analyzers to Filter and Boost Results in ElasticSearch

01/09/2023

ElasticSearch, a powerful and versatile search and analytics engine, has emerged as a cornerstone technology for managing and querying large-scale data with unprecedented speed and precision. However, as the size and complexity of data repositories continue to grow, so do the challenges associated with retrieving the most relevant results.

How to deal with it? You can easily optimize search outcomes by using permissive and strict analyzers to fine-tune result filtering and boosting.

The Difference Between Permissive and Strict Analyzers

Using a boolean query we can take different occurrence types. Some of them are:

  • must – The clause (query) must appear in matching documents and will contribute to the score.
  • should – The clause (query) should appear in the matching document.

Permissive query rules used in “must” can be treated like a filter that will be responsible for picking matched documents from index. It aims to broaden the scope of search results by accommodating variations in spelling, stemming, and synonyms, ensuring that users encounter a comprehensive array of relevant documents.

Strict query rules used in “should” can be treated like some kind of booster that will push documents higher in the relevancy position because matching documents will get a higher score.

How to Use it?

Let’s start from preparing permissive and strict analyzers:

“permissive": {
   "char_filter": [
       "html_strip"
   ],
   "tokenizer": "icu_tokenizer",
   "filter": [
       "english_possessive_stemmer",
       "english_stemmer",
       "custom_length"
   ]
}

And now strict analyzer:

"strict": {
   "char_filter": [
       "html_strip"
   ],
   "tokenizer": "icu_tokenizer",
   "filter": [
       "english_possessive_stemmer",
       "custom_length"
   ]
}

To keep examples as small as possible please note only one difference between permissive and strict analyzer is one additional filter “english_stemmer”.

Now let’s continue to index structure definition. In order to keep different analyzed contents inside the index we will use “fields” feature.

"name": {
   "type": "text",
   "analyzer": "permissive",
   "fields": {
       "strict": {
           "type": "text",
           "analyzer": "strict"
       }
   }
},
"description": {
   "type": "text",
   "analyzer": "permissive",
   "fields": {
       "strict": {
           "type": "text",
           "analyzer": "strict"
       }
   }
},

And finally let’s go build our query that will use “must”, “should” and different analyzers to filter out and boost results.

"query": {
 "bool": {
   "must": [
     {
       "multi_match": {
         "query": "swimming",
         "fields": [
           "name",
           "description"
         ],
         "analyzer": "permissive",
     }
   ],
   "should": [
     {
       "multi_match": {
         "query": "swimming",
         "fields": [
           "name.strict",
           "description.strict"
         ],
         "analyzer": "strict",
       }
     }
   ]
 }
}

In the example all documents that contain any kind of “swim” form of verb like “swimming”, “swim”, “swam” will be picked up by “must” rules because of permissive analyzer keep “english_stemmer” filter that will reduce english verbs to their root form. Using the same analyzer in query time will always reduce input term to root form.

On the other hand, a strict analyzer that does not have an “english_stemmer” filter will search documents that have exactly the same form as the customer is trying to search. But as a strict analyzer is used in the “should” part of the query it will not filter out products that do not match. It will give higher scores to products that match terms in the same form.

To give more ideas how to use permissive and strict methods please keep in mind that any of the available filters can be used to build different analyzers.

Good to Remember

  • Synonyms for example is also a good filter that works very well with permissive and strict analyzers.
  • On top of that please remember “should” can have multiple query rules. Each of the query rules can use a different strict analyzer.
  • Method is very flexible and very efficient in order to provide better search experience and better results to the customer.
Ja
Portrait of Jakub Wachol, back-end developer and article author, smiling and wearing glasses, with a professional and friendly appearance, against a white background.
Jakub Wachol
Back-end Developer

Latest articles

Illustration of a fractional CTO working on a laptop with flowcharts and code screens, highlighting CTO as a service benefits

Business | 29/05/2024

Why Do You Need a CTO in Your Company?

Łukasz Kopaczewski

Businesses, regardless of their size, are increasingly relying on innovative technological solutions. In this context, the role of the CTO becomes not only important, but even essential for any organization that wants to grow, remain competitive and achieve long-term success. So let's take a look at the main reasons why the presence of a CTO can be a key factor in creating and strengthening technology strategy and achieving a company's business goals.

Read more
Illustration of a team dealing with legacy code, featuring three people collaborating with gears and a rocket launching from a computer, symbolizing strategies for handling legacy code effectively.

Development | 24/05/2024

How to Work with Legacy Code? Strategies and Tips

Jakub Wachol

Many organizations run on software that was written in technologies that are obsolete or using approaches that are no longer used today. While these may seem like marginal situations, they happen quite often. It is in such cases that the problem of how to deal with legacy code arises.

Read more
Illustration of a woman analyzing financial data with charts, coins, and arrows symbolizing the concept of Fixed Price contracts in technology.

Business | 15/05/2024

Fixed-Price Contract: Benefits for Your Company

Łukasz Kopaczewski

When looking for a Software House to cooperate with, you need to consider many factors. Some of the most important are the price of the services and the method of billing. The two most popular types of contracts are Fixed Price and Time and Material, although you may also encounter a mixed model. Today we will look at the Fixed Price model, which offers stability and predictability to clients.

Read more