Pottery: Redis for Humans 🌎🌍🌏

Redis is awesome, but Redis commands are not always intuitive. Pottery is a Pythonic way to access Redis. If you know how to use Python dicts, then you already know how to use Pottery. Pottery is useful for accessing Redis more easily, and also for implementing microservice resilience patterns; and it has been battle tested in production at scale.

Table of Contents

• Dicts 📖
• Sets 🛍️
• Lists ⛓
• Counters 🧮
• Deques 🖇️
• Queues 🚶‍♂️🚶‍♀️🚶‍♂️
• Redlock 🔒
  • synchronize() 👯‍♀️
• AIORedlock 🔒
• NextID 🔢
• redis_cache()
• CachedOrderedDict
• Bloom filters 🌸
• HyperLogLogs 🪵
• ContextTimer ⏱️

Installation

$ pip3 install pottery

Usage

First, set up your Redis client:

>>> from redis import Redis
>>> redis = Redis.from_url('redis://localhost:6379/1')
>>>

<a name="dicts"></a>Dicts 📖

RedisDict is a Redis-backed container compatible with Python’s dict.

Here is a small example using a RedisDict:

>>> from pottery import RedisDict
>>> tel = RedisDict({'jack': 4098, 'sape': 4139}, redis=redis, key='tel')
>>> tel['guido'] = 4127
>>> tel
RedisDict{'jack': 4098, 'sape': 4139, 'guido': 4127}
>>> tel['jack']
4098
>>> del tel['sape']
>>> tel['irv'] = 4127
>>> tel
RedisDict{'jack': 4098, 'guido': 4127, 'irv': 4127}
>>> list(tel)
['jack', 'guido', 'irv']
>>> sorted(tel)
['guido', 'irv', 'jack']
>>> 'guido' in tel
True
>>> 'jack' not in tel
False
>>>

Notice the first two keyword arguments to RedisDict(): The first is your Redis client. The second is the Redis key name for your dict. Other than that, you can use your RedisDict the same way that you use any other Python dict.

Limitations:

1. Keys and values must be JSON serializable.

<a name="sets"></a>Sets 🛍️

RedisSet is a Redis-backed container compatible with Python’s set.

Here is a brief demonstration:

>>> from pottery import RedisSet
>>> basket = RedisSet({'apple', 'orange', 'apple', 'pear', 'orange', 'banana'}, redis=redis, key='basket')
>>> sorted(basket)
['apple', 'banana', 'orange', 'pear']
>>> 'orange' in basket
True
>>> 'crabgrass' in basket
False

>>> a = RedisSet('abracadabra', redis=redis, key='magic')
>>> b = set('alacazam')
>>> sorted(a)
['a', 'b', 'c', 'd', 'r']
>>> sorted(a - b)
['b', 'd', 'r']
>>> sorted(a | b)
['a', 'b', 'c', 'd', 'l', 'm', 'r', 'z']
>>> sorted(a & b)
['a', 'c']
>>> sorted(a ^ b)
['b', 'd', 'l', 'm', 'r', 'z']
>>>

Notice the two keyword arguments to RedisSet(): The first is your Redis client. The second is the Redis key name for your set. Other than that, you can use your RedisSet the same way that you use any other Python set.

Do more efficient membership testing for multiple elements using .contains_many():

>>> nirvana = RedisSet({'kurt', 'krist', 'dave'}, redis=redis, key='nirvana')
>>> tuple(nirvana.contains_many('kurt', 'krist', 'chat', 'dave'))
(True, True, False, True)
>>>

Limitations:

1. Elements must be JSON serializable.

<a name="lists"></a>Lists ⛓

RedisList is a Redis-backed container compatible with Python’s list.

>>> from pottery import RedisList
>>> squares = RedisList([1, 4, 9, 16, 25], redis=redis, key='squares')
>>> squares
RedisList[1, 4, 9, 16, 25]
>>> squares[0]
1
>>> squares[-1]
25
>>> squares[-3:]
[9, 16, 25]
>>> squares[:]
[1, 4, 9, 16, 25]
>>> squares + [36, 49, 64, 81, 100]
RedisList[1, 4, 9, 16, 25, 36, 49, 64, 81, 100]
>>>

Notice the two keyword arguments to RedisList(): The first is your Redis client. The second is the Redis key name for your list. Other than that, you can use your RedisList the same way that you use any other Python list.

Limitations:

1. Elements must be JSON serializable.
2. Under the hood, Python implements list using an array. Redis implements list using a doubly linked list. As such, inserting elements at the head or tail of a RedisList is fast, O(1). However, accessing RedisList elements by index is slow, O(n). So in terms of performance and ideal use cases, RedisList is more similar to Python’s deque than Python’s list. Instead of RedisList, consider using RedisDeque.

<a name="counters"></a>Counters 🧮

RedisCounter is a Redis-backed container compatible with Python’s collections.Counter.

>>> from pottery import RedisCounter
>>> c = RedisCounter(redis=redis, key='my-counter')
>>> c = RedisCounter('gallahad', redis=redis, key='my-counter')
>>> c.clear()
>>> c = RedisCounter({'red': 4, 'blue': 2}, redis=redis, key='my-counter')
>>> c.clear()
>>> c = RedisCounter(redis=redis, key='my-counter', cats=4, dogs=8)
>>> c.clear()

>>> c = RedisCounter(['eggs', 'ham'], redis=redis, key='my-counter')
>>> c['bacon']
0
>>> c['sausage'] = 0
>>> del c['sausage']
>>> c.clear()

>>> c = RedisCounter(redis=redis, key='my-counter', a=4, b=2, c=0, d=-2)
>>> sorted(c.elements())
['a', 'a', 'a', 'a', 'b', 'b']
>>> c.clear()

>>> RedisCounter('abracadabra', redis=redis, key='my-counter').most_common(3)
[('a', 5), ('b', 2), ('r', 2)]
>>> c.clear()

>>> c = RedisCounter(redis=redis, key='my-counter', a=4, b=2, c=0, d=-2)
>>> from collections import Counter
>>> d = Counter(a=1, b=2, c=3, d=4)
>>> c.subtract(d)
>>> c
RedisCounter{'a': 3, 'b': 0, 'c': -3, 'd': -6}
>>>

Notice the first two keyword arguments to RedisCounter(): The first is your Redis client. The second is the Redis key name for your counter. Other than that, you can use your RedisCounter the same way that you use any other Python Counter.

Limitations:

1. Keys must be JSON serializable.

<a name="deques"></a>Deques 🖇️

RedisDeque is a Redis-backed container compatible with Python’s collections.deque.

Example:

>>> from pottery import RedisDeque
>>> d = RedisDeque('ghi', redis=redis, key='letters')
>>> for elem in d:
...     print(elem.upper())
G
H
I

>>> d.append('j')
>>> d.appendleft('f')
>>> d
RedisDeque(['f', 'g', 'h', 'i', 'j'])

>>> d.pop()
'j'
>>> d.popleft()
'f'
>>> list(d)
['g', 'h', 'i']
>>> d[0]
'g'
>>> d[-1]
'i'

>>> list(reversed(d))
['i', 'h', 'g']
>>> 'h' in d
True
>>> d.extend('jkl')
>>> d
RedisDeque(['g', 'h', 'i', 'j', 'k', 'l'])
>>> d.rotate(1)
>>> d
RedisDeque(['l', 'g', 'h', 'i', 'j', 'k'])
>>> d.rotate(-1)
>>> d
RedisDeque(['g', 'h', 'i', 'j', 'k', 'l'])

>>> RedisDeque(reversed(d), redis=redis)
RedisDeque(['l', 'k', 'j', 'i', 'h', 'g'])
>>> d.clear()

>>> d.extendleft('abc')
>>> d
RedisDeque(['c', 'b', 'a'])
>>>

Notice the two keyword arguments to RedisDeque(): The first is your Redis client. The second is the Redis key name for your deque. Other than that, you can use your RedisDeque the same way that you use any other Python deque.

Limitations:

1. Elements must be JSON serializable.

<a name="queues"></a>Queues 🚶‍♂️🚶‍♀️🚶‍♂️

RedisSimpleQueue is a Redis-backed multi-producer, multi-consumer FIFO queue compatible with Python’s queue.SimpleQueue. In general, use a Python queue.Queue if you’re using it in one or more threads, use multiprocessing.Queue if you’re using it between processes, and use RedisSimpleQueue if you’re sharing it across machines or if you need for your queue to persist across application crashes or restarts.

Instantiate a RedisSimpleQueue:

>>> from pottery import RedisSimpleQueue
>>> cars = RedisSimpleQueue(redis=redis, key='cars')
>>>

Notice the two keyword arguments to RedisSimpleQueue(): The first is your Redis client. The second is the Redis key name for your queue. Other than that, you can use your RedisSimpleQueue the same way that you use any other Python queue.SimpleQueue.

Check the queue state, put some items in the queue, and get those items back out:

>>> cars.empty()
True
>>> cars.qsize()
0
>>> cars.put('Jeep')
>>> cars.put('Honda')
>>> cars.put('Audi')
>>> cars.empty()
False
>>> cars.qsize()
3
>>> cars.get()
'Jeep'
>>> cars.get()
'Honda'
>>> cars.get()
'Audi'
>>> cars.empty()
True
>>> cars.qsize()
0
>>>

Limitations:

1. Items must be JSON serializable.

<a name="redlock"></a>Redlock 🔒

Redlock is a safe and reliable lock to coordinate access to a resource shared across threads, processes, and even machines, without a single point of failure. Rationale and algorithm description.

Redlock implements Python