Information storing and searching are two of the most crucial aspects for any web application. They affect the overall success of your project. The same is true when you aim to develop a perfect Rails web app.
A web product can contain tons of data, making its storage and search extremely challenging. Thus, it is always a great idea to build convenient and powerful algorithms. This is where Elasticsearch comes in handy.
In this article, we will walk you through the whole process of developing a test Ruby on Rails app with Elasticsearch integration.
Defining the terms
Before jumping into the Ruby on Rails web application development process and search algorithms implementing, let’s discuss the key terms and install the tools and services needed.
Elasticsearch is an extremely fast, open-source JSON-based search service. It allows storing, scanning, and analyzing the required data in milliseconds. The service is about integrating of complex search requests and requirements.
That is the reason Elasticsearch is loved by influencers, such as NASA, Microsoft, eBay, Uber, GitHub, Facebook, Warner Brothers, and others.
Now let’s take a closer look at the main terms of Elasticsearch.
Mapping. A process of defining the way both a document and its fields are stored and indexed.
Indexing. An act of keeping data in Elasticsearch. An Elasticsearch cluster can consist of different indices which in their turn contain various types.
Analysis process. A process of rendering a text into tokens or terms that are put on to the inverted index for searching. The analysis is fulfilled by an analyzer which can be of two types, namely an inbuilt analyzer or a custom analyzer defined per index.
Analyzer. A package of three building units where each of them modifies the input stream. An analyzer includes character filters, tokenizer, and token filters.
The flow of a document indexing can be presented the following way:
1) Character filters. First of all, it goes through one or several
character filters. It receives original text fields and then transforms the value by adding, deleting, or modifying characters. For example, it can remove html markup from text. The full list of character filters can be found here.
2) Tokenizer. After that, the analyzer separates text into tokens that are usually words. For example, a ten-words text is divided into an array of 10 tokens. The analyzer may have only one tokenizer.
Standard tokenizer is applied by default. It splits text with whitespaces and also deletes most of the symbols, such as periods, commas, semicolons, etc. You can find the list of all available tokenizers here.
3) Token filters.
Token filters are close to
character filters. The main difference is that token filters work with the token stream, while character filters work with the character stream. There are various token filters.
Lowercase Token Filter is the simplest one. Find the full list of all available token filters here.
Inverted index. The results from the analysis are starting within an inverted index. The purpose of an inverted index is to store a text in a structure that allows for very efficient fast full-text searches. When performing full-text searches, we are actually querying an inverted index, not the documents defined when indexing them.
All the full-text fields have a single inverted index per field.
An inverted index includes all of the unique terms that are shown in any document covered by the index.
Let’s take a look at two sentences. The first one is “I am a Ruby programmer”. The second one is “This project was built in Ruby”.
In the inverted index, they will be saved as follows:
|Term||Document #1||Document #2|
If we search for “Ruby”, we will see that both documents contain the term.
|Term||Document #1||Document #2|
Step #1: Installing the tools
Before starting with actual code writing, we need a set of tools and services. As for us, we also use ready-made solutions called gems to increase the speed of the development process.
Install Ruby 2.6.1
We will use RVM to manage multiple Ruby versions installed on our system. Check the Ruby version set up on the system with
rvm list and install the following one
rvm install 2.6.1.
Install Rails 5.2.3
With Ruby already installed, we will need the Rails gem as well.
➜ ~ gem install rails -v 5.2.3
To check the version of the installed Rails gem, type the following request
Install Elasticsearch 6.4.0
As we have saved Elasticsearch to the Downloads folder, we run the service by typing in the following request:
➜ ~ /Downloads/elasticsearch-6.4.0/bin/elasticsearch
To make sure the tool is started, open it with
At this point, we see the following on the screen:
Install Kibana 6.4.2
You can download Kibana here. We have saved Kibana to the Downloads folder. We type in the following request to run the service:
➜ ~ /Downloads/kibana-6.4.2-linux-x86_64/bin/kibana
In order to be sure that Kibana is running, navigate to
At this step, we see the window:
All the needed tools and services are installed. Now you are ready to start with the project development and Elasticsearch integration.
Step #2: Initiating a new Rails app
Along with the PostgreSQL database, we're going to use the Rails in API mode:
➜ ~ rvm use 2.6.1
➜ ~ rails new elasticsearch_rails --api -T -d postgresql
➜ ~ cd elasticsearch_rails
➜ ~ bundle install
The first thing to do is to configure the database. At this point, we modify our config/database.yml structure similar to this:
Finally, we have created the rails
We also need to build a model which we will index and make searchable. Let’s create a Location table with two fields, such as name and level:
➜ ~ rails generate model location name level
After we have created the table, we run the migration with the rails
We have prepared all the test data needed. Copy the contents of the following file, insert it into db/seeds.rb, and run
Step #3: Using Elasticsearch with Rails
To integrate the search engine to the Rails application, we need to add two gems to Gemfile:
➜ ~ gem ’elasticsearch-model’
➜ ~ gem ’elasticsearch-rails’
Don’t forget to run
bundle install to install these gems.
Now we are ready to add actual functionality to the location model. For this purpose, we use the so-called
We create a new
The next step is to add the following code:
Finally, we include the created module to the location model:
At this stage, we reproduce the following steps:
Elasticsearch::Modelmodule, we add Elasticsearch integration to the model.
Elasticsearch::Model::Callbacks, we add callbacks. Why is it important? Each time an object is saved, updated or deleted, the related indexed data gets updated accordingly, too.
The last thing we need to do is to index our model.
Open the Rails
rails c console and run
Location.import force: true.
force: true option will create an index if it doesn't exist. To check whether the index has been built, open Kibana dev tools at
http://localhost:5601/ and insert
As you can see, we have created the index with the name locations:
Since the index was built automatically, the default configuration was applied to all fields.
Now it is time to develop a test query. You can find more information about Elasticsearch Query DSL here.
Open Kibana development tools and navigate to
Afterwards, insert the following code:
The hits attribute of the response’s JSON and especially its _source attribute are the first features we should take into account. As you can see, all fields in the Location model were serialized and indexed.
We also can make a test query through the Rails app. Open
rails c console and insert:
You may also like: Building a Slack Bot for Internal Time Tracking
Step #4: Building a custom index with autocomplete functionality
Before creating a new index, we need to delete the previous one. For this purpose, open
rails c Location.__elasticsearch__.delete_index!. The previous index was removed.
The next step is to edit the
app/models/concerns/searchable.rb file so it would look like this:
In this code snippet, we are serializing our model attributes to JSON with the key
We will work only with two fields, i.e.
We are going to define the index configuration:
Here we define a custom analyzer named
standard tokenizer and with
Autocomplete filter is of
edge_ngram type. The
edge_ngram tokenizer divides the text into smaller parts (grams).
For example, the word
“ruby” will be split into
[“ru”, “rub”, “ruby”].
edge_ngram are useful when we need to implement autocomplete functionality. However, there is another way to integrate the options needed, the so-called completion suggester approach.
We apply mappings to the name and level fields. The keyword data type is used with the level field. The text data type is applied to the name field along with our custom
And finally, we will explain the search method we use:
Now it is time to open the Rails console and check the following request to be sure the project works correctly:
However, it is always a good idea to verify the accuracy of the product performance with a few mistakes in the request to make sure the project functions properly:
As you remember, we have one filter defined. It is used to filter Location by level. There are two objects in the database with the same name, i.e. New York, which are of different levels. The first level refers to the state, and the second one - to the city:
Step #5: Making the search request available by API
In the final development stage, we will create a controller through which the search queries will pass:
➜ ~ rails generate controller Home search
Open app/controllers/home_controller.rb and insert the following code snippet in it:
Let's see the project in action.
Run the Rails server by typing
rails s and then go to
In the below code, we request all documents containing the name “new” and whose level is equal to the state.
This is what the response looks like:
Congratulations! Your test Rails web app is ready, with the basic functionality of the searching service integrated.
We hope that our guide was helpful, and we highly recommend you to learn all the possibilities of Elasticsearch to improve your development skills.
Elasticsearch is a perfect tool for integrating a fast full-text search with powerful features:
- Speed plays an important role in providing a customer with a positive user experience.
- Flexibility is about modifying the search performance and optimizing various datasets and use cases.
- If a user makes a typo in a search, Elasticsearch still returns relevant results for what the customer is looking for.
- The service makes it possible to search both for specific keywords and other matching data stored in your database.
Elasticsearch has powerful features and works smoothly with Ruby on Rails app development. What could be better?