|Tests Status| |Coverage Status|

.. image:: docs/_static/patroni-logo.svg :height: 128px :width: 128px

Patroni: A Template for PostgreSQL HA with ZooKeeper, etcd or Consul

You can find a version of this documentation that is searchable and also easier to navigate at patroni.readthedocs.io <https://patroni.readthedocs.io>__.

Important! Running Patroni on memory-restricted systems with Python 3.11+

If you run Patroni on a system with strict memory limits, for example with vm.overcommit_memory=2 (recommended for PostgreSQL), and use Python 3.11 or newer, you may observe unexpected behavior:

Patroni appears healthy
PostgreSQL continues to run
Patroni REST API becomes unresponsive
The operating system reports that Patroni is listening on the REST API port
Patroni logs look normal; however, following messages may appear once: Exception ignored in thread started by: <object repr() failed>, MemoryError
Kernel logs may contain messages such as not enough memory for the allocation

This behavior is caused by a bug in Python 3.11+ <https://github.com/python/cpython/issues/140746>__. Under strict memory conditions, starting a new thread may hang indefinitely when there is not enough free memory.

Additional recommendations (Linux, glibc)

When running with vm.overcommit_memory=2 (recommended for PostgreSQL), we also recommend starting Patroni with the following environment variables configured:

MALLOC_ARENA_MAX=1 - reduces the amount of virtual memory allocated by glibc for multi-threaded applications
PG_MALLOC_ARENA_MAX= - resets the value of MALLOC_ARENA_MAX for PostgreSQL processes started by Patroni.

In addition, you may tune the following Patroni configuration parameters:

thread_stack_size - stack size used for threads started by Patroni. Lowering this value reduces memory usage of the Patroni process. The default value set by Patroni is 512kB. Increase thread_stack_size if Patroni experience stack-related crashes; otherwise the default value is sufficient.
thread_pool_size - size of the thread pool used by Patroni for asynchronous tasks and REST API communication with other members during leader race or failsafe checks. The default value is 5, which is sufficient for three-node clusters.
restapi.thread_pool_size - size of the thread pool used to process REST API requests. The default value is 5, allowing up to five parallel REST API requests. Note that requests involving SQL queries are effectively serialized because a single database connection is used, so increasing this value typically provides no benefit.

PostgreSQL High Availability and Patroni

There are many ways to run high availability with PostgreSQL; for a list, see the PostgreSQL Documentation <https://wiki.postgresql.org/wiki/Replication,_Clustering,_and_Connection_Pooling>__.

Patroni is a template for high availability (HA) PostgreSQL solutions using Python. For maximum accessibility, Patroni supports a variety of distributed configuration stores like ZooKeeper <https://zookeeper.apache.org/>, etcd <https://github.com/coreos/etcd>, Consul <https://github.com/hashicorp/consul>__ or Kubernetes <https://kubernetes.io>__. Database engineers, DBAs, DevOps engineers, and SREs who are looking to quickly deploy HA PostgreSQL in datacenters - or anywhere else - will hopefully find it useful.

We call Patroni a "template" because it is far from being a one-size-fits-all or plug-and-play replication system. It will have its own caveats. Use wisely.

Currently supported PostgreSQL versions: 9.3 to 18.

Note to Citus users: Starting from 3.0 Patroni nicely integrates with the Citus <https://github.com/citusdata/citus>__ database extension to Postgres. Please check the Citus support page <https://github.com/patroni/patroni/blob/master/docs/citus.rst>__ in the Patroni documentation for more info about how to use Patroni high availability together with a Citus distributed cluster.

Note to Kubernetes users: Patroni can run natively on top of Kubernetes. Take a look at the Kubernetes <https://github.com/patroni/patroni/blob/master/docs/kubernetes.rst>__ chapter of the Patroni documentation.

.. contents:: :local: :depth: 1 :backlinks: none

================= How Patroni Works

Patroni (formerly known as Zalando's Patroni) originated as a fork of Governor <https://github.com/compose/governor>__, the project from Compose. It includes plenty of new features.

For additional background info, see:

Elephants on Automatic: HA Clustered PostgreSQL with Helm <https://www.youtube.com/watch?v=CftcVhFMGSY>_, talk by Josh Berkus and Oleksii Kliukin at KubeCon Berlin 2017
PostgreSQL HA with Kubernetes and Patroni <https://www.youtube.com/watch?v=iruaCgeG7qs>__, talk by Josh Berkus at KubeCon 2016 (video)
Feb. 2016 Zalando Tech blog post <https://engineering.zalando.com/posts/2016/02/zalandos-patroni-a-template-for-high-availability-postgresql.html>__

================== Development Status

Patroni is in active development and accepts contributions. See our Contributing <https://github.com/patroni/patroni/blob/master/docs/contributing_guidelines.rst>__ section below for more details.

We report new releases information here <https://github.com/patroni/patroni/releases>__.

========= Community

There are two places to connect with the Patroni community: on github <https://github.com/patroni/patroni>, via Issues and PRs, and on channel #patroni <https://postgresteam.slack.com/archives/C9XPYG92A> in the PostgreSQL Slack <https://pgtreats.info/slack-invite>__. If you're using Patroni, or just interested, please join us.

=================================== Technical Requirements/Installation

Pre-requirements for Mac OS

To install requirements on a Mac, run the following:

brew install postgresql etcd haproxy libyaml python

Psycopg

Starting from psycopg2-2.8 <http://initd.org/psycopg/articles/2019/04/04/psycopg-28-released/>__ the binary version of psycopg2 will no longer be installed by default. Installing it from the source code requires C compiler and postgres+python dev packages. Since in the python world it is not possible to specify dependency as psycopg2 OR psycopg2-binary you will have to decide how to install it.

There are a few options available:

Use the package manager from your distro

sudo apt-get install python3-psycopg2  # install psycopg2 module on Debian/Ubuntu
sudo yum install python3-psycopg2      # install psycopg2 on RedHat/Fedora/CentOS

2. Specify one of psycopg, psycopg2, or psycopg2-binary in the list of dependencies when installing Patroni with pip (see below).

General installation for pip

Patroni can be installed with pip:

pip install patroni[dependencies]

where dependencies can be either empty, or consist of one or more of the following:

etcd or etcd3 python-etcd module in order to use Etcd as DCS consul py-consul module in order to use Consul as DCS zookeeper kazoo module in order to use Zookeeper as DCS exhibitor kazoo module in order to use Exhibitor as DCS (same dependencies as for Zookeeper) kubernetes kubernetes module in order to use Kubernetes as DCS in Patroni raft pysyncobj module in order to use python Raft implementation as DCS aws boto3 in order to use AWS callbacks systemd systemd-python in order to use sd_notify integration all all of the above (except psycopg family) psycopg3 psycopg[binary]>=3.0.0 module psycopg2 psycopg2>=2.5.4 module psycopg2-binary psycopg2-binary module

For example, the command in order to install Patroni together with psycopg3, dependencies for Etcd as a DCS, and AWS callbacks is:

pip install patroni[psycopg3,etcd3,aws]

Note that external tools to call in the replica creation or custom bootstrap scripts (i.e. WAL-E) should be installed independently of Patroni.

======================= Running and Configuring

To get started, do the following from different terminals: ::

> etcd --data-dir=data/etcd --enable-v2=true
> ./patroni.py postgres0.yml
> ./patroni.py postgres1.yml

You will then see a high-availability cluster start up. Test different settings in the YAML files to see how the cluster's behavior changes. Kill some of the components to see how the system behaves.

Add more postgres*.yml files to create an even larger cluster.

Patroni provides an HAProxy <http://www.haproxy.org/>__ configuration, which will give your application a single endpoint for connecting to the cluster's leader. To configure, run:

> haproxy -f haproxy.cfg

> psql --host 127.0.0.1 --port 5000 postgres

================== YAML Configuration

Go here <https://github.com/patroni/patroni/blob/master/docs/dynamic_configuration.rst>__ for comprehensive information about settings for etcd, consul, and ZooKeeper. And for an example, see postgres0.yml <https://github.com/patroni/patroni/blob/master/postgres0.yml>__.

========================= Environment Configuration

Go here <https://github.com/patroni/patroni/blob/master/docs/ENVIRONMENT.rst>__ for comprehensive information about configuring(overriding) settings via environment variables.

=================== Replication Choices

Patroni uses Postgres' streaming replication, which is

Patroni

Install / Use

README