Neos Elasticsearch Adapter

This project connects the Neos Content Repository to Elasticsearch; enabling two main functionalities:

• finding Nodes in Fusion / Eel by arbitrary queries
• Full-Text Indexing of Pages and other Documents (of course including the full content)

This documentation is structured in the following parts:

• Installation: Which packages are needed and how they are installed.
• Commands: This section describes the available ./flow CLI commands
• Configuration: Configuration of indices, fields, ...
• Query Data: Available Eel operations to query data
• Examples: Some more advanced examples

Installation

composer require 'flowpack/elasticsearch-contentrepositoryadaptor'
// Not required, but can be used to learn how to integration the flowpack/elasticsearch-contentrepositoryadaptor in your project
composer require 'flowpack/searchplugin'

Ensure to update <your-elasticsearch>/config/elasticsearch.yml as explained below; then start Elasticsearch.

Finally, run ./flow nodeindex:build, and add the search plugin to your page. It should "just work".

Relevant Packages

• Neos.ContentRepository.Search: provides common functionality for searching Neos Content Repository nodes. Does not contain a search backend.
• Flowpack.ElasticSearch: provides common code for working with Elasticsearch
• Flowpack.ElasticSearch.ContentRepositoryAdaptor: this package
• Flowpack.SimpleSearch.ContentRepositoryAdaptor: an alternative search backend (to be used instead of this package); storing the search index in SQLite
• Flowpack.SearchPlugin: search plugin for Neos

Elasticsearch and Neos compatibility

This following matrix shows the compatibility of this package to Elasticsearch and Neos versions:

| CR Adaptor | Neos | Elasticsearch | Status | |------------|------------|---------------|-------------------------------------------------------------------------------| | 4 | 3.x, 4.x | 1.x, 2,x 5.x | Unmaintained, probably broken | | 5 | > 3.3, 4.x | 5.x | Unmaintained | | 6 | 5.x | 5.x | Unmaintained | | 7 | 5.x | 6.x, 7.x | Unmaintained | | 8 | 7.x, 8.x | 6.x, 7.x, 8.x | Bugfix and Features | | 9 | 9.x | 6.x, 7.x, 8.x | Bugfix and Features (Upgrade Instructions) |

Currently the Driver interfaces are not marked as API, and can be changed to adapt to future needs.

Elasticsearch Configuration file elasticsearch.yml

The following general configuration advice can make your life easier:

# the following settings secure your cluster
cluster.name: [PUT_YOUR_CUSTOM_NAME_HERE]
node.name: [PUT_YOUR_CUSTOM_NAME_HERE]
network.host: _local_

There may be a need, to add specific configuration to your Elasticsearch Configuration File <your-elasticsearch>/config/elasticsearch.yml, depending on your version of Elasticsearch.

• Elasticsearch 5.x

Commands

All commands are operating on a single content repository. If you don't specify the content repository identifer by --contentRepository the default content repository is used.

Building up the index

The node index is updated on the fly, but during development you need to update it frequently.

In case of a mapping update, you need to reindex all nodes. Don't worry to do that in production; the system transparently creates a new index, fills it completely, and when everything worked, changes the index alias.

./flow nodeindex:build

if during development, you only want to index a few nodes, you can use "limit"

./flow nodeindex:build --limit 20

Cleanup old indices

In order to remove old, non-used indices, you should use this command from time to time:

./flow nodeindex:cleanup

Debug commands

The following commands are meant to be used for debugging while configuring and developing your search:

./flow nodeindexmapping:indices

Shows the mapping between the projects dimensions presets and the resulting index name.

./flow nodeindexmapping:mapping

Shows the mapping created for the NodeTypes.

./flow nodetype:showIndexableConfiguration

Shows a list of NodeTypes and if they are configured to be indexable

./flow search:viewnode <nodeIdentifier> [<dimensionCombinationAsJson>] [<field>]

Shows all contents that are indexed fo a given node.

./flow search:fulltext

Performs a fulltext search and displays the results.

Configuration

Index Settings

If you want to fine-tune the indexing and mapping on a more detailed level, you can do so in the following way.

Configure the index name

If you need to run serveral (different) neos instances on the same elasticsearch server you will need to change the Configuration/Settings.yaml indexName for each of your project.

So ./flow nodeindex:build or ./flow nodeindex:cleanup won't overwrite your other sites index.

Neos:
  ContentRepository:
    Search:
      elasticSearch:
        indexName: useMoreSpecificIndexName

If you use multiple client configurations, please change the default key just below the indexes.

Configure per index

You can set one default configuration for all indices with your index prefix.

Flowpack:
  ElasticSearch:
    indexes:
      default: # Configuration bundle name
        neoscontentrepository: # The index prefix name, must be the same as in the Neos.ContentRepository.Search.elasticSearch.indexName setting
          settings:
            index:
              number_of_shards: 1
              number_of_replicas: 0

Configure per dimension

As an index is created for every dimension combination of the Neos content repository, you can configure the index behavior for every dimension combination separately.

Caution: Default configuration and per dimension combination configuration is not merged. If a configuration for a dimension-combination is found, this configuration is used.

Flowpack:
  ElasticSearch:
    indexes:
      default: 
        'neoscontentrepository-0359ed5c416567b8bc2e5ade0f277b36': # The hash specifies the dimension combination
          settings: 
            index:
              number_of_shards: 1
              number_of_replicas: 0
            analysis:
              filter:
                elision:
                  type: 'elision'
                  articles: [ 'l', 'm', 't', 'qu', 'n', 's', 'j', 'd' ]
              analyzer:
                custom_french_analyzer:
                  tokenizer: 'letter'
                  filter: [ 'asciifolding', 'lowercase', 'french_stem', 'elision', 'stop' ]
                tag_analyzer:
                  tokenizer: 'keyword'
                  filter: [ 'asciifolding', 'lowercase' ]

Which dimension combinations are available in your system and which hashes they are identified with can be shown with the CLI command:

./flow nodeindexmapping:indices

Configurations per property (index field)

Then, you can change the analyzers on a per-field level; or e.g. reconfigure the _all field with the following snippet in the NodeTypes.yaml. Generally this works by defining the global mapping at [nodeType].search.elasticSearchMapping:

'Neos.Neos:Node':
  search:
    elasticSearchMapping:
      myProperty:
        analyzer: custom_french_analyzer

Exclude NodeTypes from indexing

By default the indexing processes all NodeTypes, but you can change this in your Settings.yaml:

Neos:
  ContentRepository:
    Search:
      defaultConfigurationPerNodeType:
        '*':
          indexed: true
        'Neos.Neos:FallbackNode':
          indexed: false
        'Neos.Neos:Shortcut':
          indexed: false
        'Neos.Neos:ContentCollection':
          indexed: false

You need to explicitly configure the individual NodeTypes (this feature does not check the Super Type configuration). But you can use a special notation to configure a full namespace, Acme.AcmeCom:* will be applied for all node types in the Acme.AcmeCom namespace. The most specific configuration is used in this order:

• NodeType name (Neos.Neos:Shortcut)
• Full namespace notation (Neos.Neos:*)
• Catch all (*)

Advanced Indexing configuration

Indexing configuration per data type

The default configuration supports most use cases and often may not need to be touched, as this package comes with sane defaults for all Neos data types.

Indexing of p