<img src="https://raw.githubusercontent.com/wallneradam/esorm/main/docs/_static/img/esorm.svg" width="110" height="110" align="left" style="margin-right: 1em; margin-bottom: 0.5em" alt="Logo"/>

ESORM - Python ElasticSearch ORM based on Pydantic

ESORM is an ElasticSearch Object Relational Mapper or Object Document Mapper (ODM) if you like, for Python based on Pydantic. It is a high-level library for managing ElasticSearch documents in Python. It is fully async and uses annotations and type hints for type checking and IDE autocompletion.

☰ Table of Contents

• 💾 Installation
• 🚀 Features
  • Supported ElasticSearch versions
  • Supported Python versions
• 📖 Usage
  • Define a model
    • Python basic types
    • ESORM field types
    • Field Definition
    • Subfields
    • geo_point
    • dense_vector
    • Nested documents
    • List primitive fields
    • ESBaseModel
    • Id field
    • Model Settings
    • Describe fields
    • ESModelTimestamp
  • Connecting to ElasticSearch
    • Client
  • Create index templates
  • Create indices and mappings
  • Model instances
  • CRUD: Create
  • CRUD: Read
  • CRUD: Update
  • CRUD: Delete
  • Bulk operations
  • Search
    • General search
    • Search with field value terms (dictionary search)
    • Count
    • Vector Search
  • Aggregations
  • Pagination and sorting
• 🔬 Advanced usage
  • Optimistic concurrency control
  • Lazy properties
  • Shard routing
  • Retreive Selected Fields Only
  • Watchers
  • FastAPI integration
• 🖥 IDE Support
• 🧪 Testing
• 🛡 License
• 📃 Citation

<a id="installation"></a>

💾 Installation

pip install pyesorm

<a id="features"></a>

🚀 Features

• Pydantic model representation of ElasticSearch documents
• Automatic mapping and index creation
• CRUD operations
• Full async support (no sync version at all)
• Mapping to and from ElasticSearch types
• Support for nested documents
• Automatic optimistic concurrency control
• Custom id field
• Context for bulk operations
• Supported IDE autocompletion and type checking (PyCharm tested)
• Everything in the source code is documented and annotated
• TypedDicts for ElasticSearch queries and aggregations
• Docstring support for fields
• Shard routing support
• Lazy properties
• Support >= Python 3.8 (tested with 3.8 through 3.13)
• Support for ElasticSearch 9.x, 8.x and 7.x
• Watcher support (You may need ElasticSearch subscription license for this)
• Pagination and sorting
• FastAPI integration

Not all ElasticSearch features are supported yet, pull requests are welcome.

<a id="supported-elasticsearch-versions"></a>

Supported ElasticSearch versions

It is tested with ElasticSearch 7.x, 8.x and 9.x.

<a id="supported-python-versions"></a>

Supported Python versions

Tested with Python 3.8 through 3.13.

<a id="usage"></a>

📖 Usage

<a id="define-a-model"></a>

Define a model

You can use all Pydantic model features, because ESModel is a subclass of pydantic.BaseModel. (Actually it is a subclass of ESBaseModel, see more below...)

ESModel extends pydantic BaseModel with ElasticSearch specific features. It serializes and deserializes documents to and from ElasticSearch types and handle ElasticSearch operations in the background.

<a id="python-basic-types"></a>

Python basic types

from esorm import ESModel


class User(ESModel):
    name: str
    age: int

This is how the python types are converted to ES types:

| Python type | ES type | Comment | |---------------------|-----------|-----------------------------| | str | keyword | | | int | long | | | float | double | | | bool | boolean | | | datetime.datetime | date | | | datetime.date | date | | | datetime.time | date | Stored as 1970-01-01 + time | | typing.Literal | keyword | | | UUID | keyword | | | Path | keyword | | | IntEnum | integer | | | Enum | keyword | also StrEnum |

Some special pydanctic types are also supported:

| Pydantic type | ES type | Comment | |-----------------|-----------|---------| | URL | keyword | | | IPvAddressAny | ip | |

<a id="esorm-field-types"></a>

ESORM field types

You can specify ElasticSearch special fields using esorm.fields module.

from esorm import ESModel
from esorm.fields import keyword, text, byte, geo_point, dense_vector


class User(ESModel):
    name: text
    email: keyword
    age: byte
    location: geo_point
    embedding: dense_vector
    ...

The supported fields are:

| Field name | ES type | |-----------------------------|-----------------| | keyword | keyword | | text | text | | binary | binary | | byte | byte | | short | short | | integer or int32 | integer | | long or int64 | long | | unsigned_long or uint64 | unsigned_long | | float16 or half_float | half_float | | float32 | float | | double | double | | boolean | boolean | | geo_point | geo_point | | dense_vector | dense_vector |

The binary field accepts base64 encoded strings. However, if you provide bytes to it, they will be automatically converted to a base64 string during serialization. When you retrieve the field, it will always be a base64 encoded string. You can easily convert it back to bytes using the bytes() method: binary_field.bytes().

You can also use Annotated types to specify the ES type, like Pydantic PositiveInt and NegativeInt and similar.

<a id="field-definition"></a>

Field Definition

ESORM fields can be defined with specialized field definition functions for more control:

from esorm.fields import Field, TextField, NumericField, DenseVectorField

class Product(ESModel):
    id: str
    name: str = TextField(..., min_length=3, max_length=100)
    price: float = NumericField(..., gt=0)
    is_available: bool = Field(True)
    location: geo_point
    embedding: dense_vector = DenseVectorField(..., dims=384, similarity="cosine")

<a id="subfields"></a>

Subfields

Elasticsearch allows you to define multiple ways to index the same field using subfields. This is particularly useful for text fields where you might want both full-text search and exact matching capabilities.

Basic Usage with keyword Shortcut

The most common use case is adding a keyword subfield to a text field for exact matching:

from esorm import ESModel
from esorm.fields import TextField

class Product(ESModel):
    # Text field with automatic keyword subfield
    title: str = TextField(..., keyword=True)
    # Equivalent to: {'fields': {'keyword': {'type': 'keyword'}}}

With this setup, you can:

• Search full-text on title
• Do exact matches, aggregations, or sorting on title.keyword

Custom Subfields

You can define custom subfields for any field type using the fields parameter:

from esorm import ESModel  
from esorm.fields import TextField, NumericField, Field

class Product(ESModel):
    # Text field with multiple analyzers
    description: str = TextField(..., fields={
        'english': {'type': 'text', 'analyzer': 'english'},
        'spanish': {'type': 'text', 'analyzer': 'spanish'}, 
        'suggest': {'type': 'completion'}
    })
    
    # Numeric field with keyword subfield for aggregations
    price: float = NumericField(..., fields={
        'raw': {'type': 'keyword'}
    })
    
    # Regular field with normalized keyword
    category: str = Field(..., fields={
        'lowercase': {'type': 'keyword', 'normalizer': 'lowercase'}
    })

Combining keyword Shortcut with Custom Fields

You can use both the keyword shortcut and custom fields together. The keyword subfield will be automatically added if not already defined:

class Product(ESModel):
    # Both keyword and custom fields
    title: str = TextField(..., keyword=True, fields={
        'suggest': {'type': 'completion'},
        'ngram':