Escargot

Connects any Rails model with ElasticSearch, supports near real time updates, distributed indexing and models that integrate data from many databases.

Requirements

Currently only rails 2.3 is supported. You will need ElasticSearch, the 'rubberband' gem and (if you want to use the optional distributed indexing mode) Redis.

Usage

First, download and start ElasticSearch (it's really simple). With the default setup of of ElasticSearch (listening to localhost and port 9200) no configuration of the plugin is necessary.

To define an index, simply add a line to your model

class Post < ActiveRecord::Base
  elastic_index
end

To create the index, execute the rake task that rebuilds all indexes:

rake escargot:index

Or restrict it to just one model

rake "escargot:index[Post]"

And you are ready to search:

Post.search "dreams OR nightmares" 

Near Real Time support

The default behavior is that every time you save or delete a record in an indexed model the index will be updated to reflect the changes. You can disable this by

class Post < ActiveRecord::Base
  elastic_index :updates => false
end

Please notice that when updates are enabled there may be a slight delay for the changes to appear in search results (with the default elasticsearch settings, this delay is just 1 second). If you absolutely need to ensure that the change is made public before returning control to the user, the :immediate_with_refresh option provides this assurance.

class Post < ActiveRecord::Base
  elastic_index :updates => :immediate_with_refresh
end

Enabling :immediate_with_refresh is not recommended. A better option is to simply call Post.refresh_index when you really need the guarantee.

Choosing the indexed fields

This plugin doesn't provide a DSL to define what fields you want to be indexed. Instead of that it exposes the fact that in ElasticSearch every document is just a JSON string.

If you define a indexed_json_document method in your model this will be used as the JSON representation of the document, otherwise to_json will be called instead.

Luckily, ActiveRecord has excellent support for JSON serialization, so it's really easy to include associations or custom methods.

 class Post < ActiveRecord::Base
  elastic_index :updates => false
  belongs_to :category

  def indexed_json_document 
    to_json(:include => :category, :methods => :slug)
  end

  def slug
    title.downcase.gsub(" ", "-")
  end
 end

See ActiveRecord's JSON serialization documentation

Search features

Basic Searching

Calling Model.search obtains from ElasticSearch the ids of the results matching your query and then queries your database to get the full ActiveRecord objects.

results = Post.search "dreams OR nightmares" 
results.each {|r| puts r.title}

The query is parsed using lucene's QueryParser syntax. You can use boolean operators, restrict your search to a field, etc.

results = Post.search "prologue:dream OR epilogue:nightmare" 

You can also guide the interpretation of the query, with the options :default_operator and :df (default_field). These two are equivalent:

results = Post.search "title:(dreams AND nightmares)"
results = Post.search "dreams nightmares" , :default_operator => 'AND', :df => 'title'

Sorting by attributes

The default order is based on the relevancy of the terms in the document. You can also sort by any other field's value.

Post.search "dreams", :order => :updated_at
Post.search "dreams", :order => 'updated_at:desc'
Post.search "dreams", :order => ['popularity:desc', 'updated_at:desc']

Sorting by an arbitrary script is possible using the Query DSL.

Pagination

search returns a WillPaginate collection and accepts the customary :per_page, and :page parameters.

# controller
@posts = Post.search("dreams", :page => params[:page], :per_page => 30)

# in the view:
will_paginate @posts

Query DSL

Instead of a string, you can pass a query in ElasticSearch's Query DSL giving you access to the full range of search features.

Bird.search(:match_all => { })  
  
Bird.search(:fuzzy => {:name => 'oriale'})

Bird.search(:custom_score => {:query => {:match_all => { } }, :script => "random()"})

Bird.search(:dis_max => {
  :tie_breaker => 0.7,
  :boost => 1.2,
  :queries => [:term => {:name => 'oriole'}, :term => {:content => 'oriole'}]
})

Bird.search(:more_like_this => {
  :like_text => "The orioles are a family of Old World passerine birds"
})


Bird.search(
  :filtered => {
    :query => {
      :term => {:name => 'oriole'}
    },
    :filter => {
      :term => {:suborder => 'Passeri'}
    }
  }
)

Query DSL with API Search

Any query Hash in Escargot a is a Query DSL by default, so anything you put in the first param is wrapper with the term "query", but sometimes you need puts some params out Query DSL, using options of API Search, you can do this using the option :query_dsl => false in the query Hash, of course remember to put the term :query => {your query} to work correctly

User.search (
    :track_scores =>true, 
    :sort =>[ {
                :name => {:reverse => true }
              }
            ],
    :query => {
                :term => {:name => "john"}
              }, 
    :query_dsl => false
)

Facets

Term facets returning the most popular terms for a field and partial results counts are available through the facets class method.

Post.facets :author_id
Post.facets :author_id, :size => 100

# restrict the facets to posts that contain 'dream'
Post.facets :author_id, :query => "dream"
Post.facets [:author_id, :category], :query => "dream"     

This returns a Hash of the form:

{
 :author_id => {
   "1" => 3,
   "25" => 2
  }, 
  :category_id => {
     12 => 4,
     42 => 7,
     47 => 2
   }
}

<!-- You can also combine the standard search results with the facets counts in a single request. results = Post.search_with_facets [:author, :category], :query => "dream" results.each{|post| puts post.title} results.facets -->

You should be aware that this only a very simple subset of the facets feature of ElasticSearch. The full feature set (histograms, statistical facets, geo distance facets, etc.) is available through the Query DSL.

Search counts

Use search_count to count the number of matches without getting the results.

Post.search_count("dream OR nightmare")

Index Creation and Type Mapping Options

Index creation options

Any value passed in the :index_options argument will be sent to ElasticSearch as an index creation option.

For example, if you want to increase the number of shards for this index:

class Post < ActiveRecord::Base
  elastic_index :index_options => {:number_of_shards => 10}
end

If you want the search to be insensitive to accents and other diacritics:

class Post < ActiveRecord::Base
  elastic_index :index_options => {
      "analysis.analyzer.default.tokenizer" => 'standard',
      "analysis.analyzer.default.filter" => ["standard", "lowercase", "stop", "asciifolding"]
  }
end

The full list of available options for index creation is documented at http://www.elasticsearch.org/guide/reference/index-modules/

Mapping options

Mapping is the process of defining how a JSON document should be mapped to the Search Engine, including its searchable characteristics.

The default (dynamic) mapping provides sane defaults, but defining your own mapping enables powerful features such as boosting a field, using a different analyzer for one field, enabling term vectors, etc.

Some examples:

class Post < ActiveRecord::Base
  elastic_index :mapping_options => {
    :properties => {
      :category => {:type => :string, :index => :not_analyzed}, 
      :title => {:type => :string, :index => :analyzed, :term_vector => true, :boost => 10.0},
      :location => {:type => :geo_point}
    }
  }
end

See the ElasticSearch Documentation for mappings.

Distributed indexing

You will need distributed indexing when there is a large amount of data to be indexed. In this indexing mode the task of creating an index is divided between a pool of workers that can be as large as you need. Since ElasticSearch itself provides linear indexing scalability by adding nodes to the cluster, this means that you should, in principle, be able to make your indexing time arbitrarily short.

Currently, the only work queue supported is Resque. To enable distributed indexing you should first install Redis and set-up Resque.

If you're on OS X and use homebrew, installing redis can be done with:

brew install redis
redis-server /usr/local/etc/redis.conf

Install the resque gem:

$ gem install resque

Include it on your application:

require 'resque'

Add this to your Rakefile:

require 'resque/tasks'
namespace :resque do
  task :setup => :environment
end