IndexedRedis

A redis-backed very very fast ORM-style framework that supports indexes. It performs searches with O(1) efficency!

You can store and fetch native python types (lists, objects, strings, integers, etc.).

IndexedRedis supports both "equals" and "not-equals" operators for comparison. It also provides full atomic support for replacing entire datasets (based on model), which is useful for providing a fast frontend for SQL. In that use-case, a task that runs on an interval would fetch/calculate datasets from the SQL backend, and do an atomic replace on the datasets the front-end would query.

IndexedRedis since 6.0.0 also provides support for Foreign references ( like a foreign key one-to-many, many-to-one, or many-to-many relationship in SQL ) which allow you to directly convert your SQL models into Redis models for vastly improved performance.

Further client-side filtering (like greater-than, contains, etc) is available after the data has been fetched (see "Filter Results" below)

My tests have shown that for using equivalent models between flask/mysql and IndexedRedis, a 600% - 1200% performance increase occurs, yet if you design your storage directly as IndexedRedis models, you are able to achieve much higher gains.

It is compatible with python 2.7 and python 3. It has been tested with python 2.7, 3.4, 3.5, 3.6.

5.0 Series

Version 5.0.0 will be somewhat backwards incompatible with previous versions by removing some old legacy stuff, improving a lot of existing things, and changing some behaviour.

You can get an overview of the steps needed to convert your code with the conversion guide, found at https://github.com/kata198/indexedredis/blob/5.0branch/CONVERTING_TO_5.0.0

Full details of changes, as well as enhancements and bug fixes omitted from the conversion guide can be found in the 5.0.0 Changelog: https://github.com/kata198/indexedredis/blob/5.0branch/Changelog

Automatic and Native Types

Since 4.0, IndexedRedis supports defining fields which will automatically be converted to/from native python types (such as int, float, datetime), as well as anything that can be represented with json (dicts, lists) or objects that support pickling. You just provide the type in its native format, and all the conversion happens behind the scenes. When fetched, the object returned also contains fields in their native types.

IndexedRedis also supports more advanced features such as automatically pickling/unpickling fields, compression/decompression, base64 encoding/decoding, and even defining your own custom field types through a standard interface.

See "Advanced Fields" section below for more information.

API Reference

Many, but not all methods and types are convered in this document.

For full pydoc reference, see:

https://pythonhosted.org/indexedredis/

or

http://htmlpreview.github.io/?https://github.com/kata198/IndexedRedis/blob/master/doc/IndexedRedis.html?_cache_vers=1

Below is a quick highlight/overview:

IndexedRedisModel

This is the type you should extend to define your model.

Example Model:

class Song(IndexedRedisModel):

	FIELDS = [ \
		IRField('artist'),
		IRField('title'),
		IRField('album'),
		IRField('track_number', valueType=int), # Convert automatically to/from int
		IRField('duration', defaultValue='0:00'),
		IRField('releaseDate', valueType=datetime.datetime),  # Convert automatically to/from datetime
		IRField('description'),
		IRField('copyright'),
		IRRawField('mp3_data'), # Don't try to encode/decode data
		IRCompressedField('thumbnail', compressMode='gzip'),      # Compress this field in storage using "gzip" compression
        IRField('tags', valueType=list),

        # "lyrics" will be a utf-8 unicode value on the object, and will be compressed/decompressed to/from storage
        IRFieldChain('lyrics', [ IRUnicodeField(encoding='utf-8'), IRCompressedField() ], defaultValue='No lyrics found' ),
	]

	INDEXED_FIELDS = [ \
				'artist',
				'title',
				'track_number',
	]

	KEY_NAME = 'Songs'

Model Attributes:

FIELDS - REQUIRED. A list of IRField objects (or their subclasses) which name the fields that can be used for storage. (see "Advanced Fields" section below)

 Example: [IRField('name'), IRField('description'), IRField('model'), IRFixedPointField('Price', 2), IRField('timestamp', valueType=datetime), IRField('remainingStock', valueType=int)]

INDEXED_FIELDS - A list of strings containing the names of fields that will be indexed. Can only filter on indexed fields. Adds insert/delete time. The names listed here must match the name of a field given in FIELDS.

 Example: ['Name', 'model']

KEY_NAME - REQUIRED. A unique name name that represents this model. Think of it like a table name.

 Example: 'StoreItems'

REDIS_CONNECTION_PARAMS - OPTIONAL - provides the arguments to pass into "redis.Redis", to construct a redis object. Here you can define overrides per-model from the default connection params.

Since 5.0.0, define this field ONLY for this model to use an alternate connection than the default. You no longer need to set this on every model.

See "Connecting To Redis" section below for more info.

If not defined or empty, the default params will be used. If any fields are present, they will override the inherited default connection params.

 Example: {'host' : '192.168.1.1'}

Advanced Fields

IndexedRedis since version 4.0 allows you to pass elements of type IRField (extends str) in the FIELDS element.

Since 5.0.0, all fields must extend IRField in some way. Those that do not will generate a deprecated warning, and the field will be converted to an IRClassicField (same as IRField, but defaults to empty string instead of irNull).

Doing so allows you to specify certain properties about the field.

Example:

FIELDS = [ IRField('name'), IRField('age', valueType=int), IRField('birthday', valueType=datetime.datetime) ]

Field Name

The first argument is the string of the field name.

Type

You can have a value automatically converted to a certain type on IRField (or use one of the several extending fields)

by passing that type as "valueType". (e.x. IRField('age', valueType=int))

If you use "bool", the values 0 and case insensitive string 'false' will result in False, and 1 or 'true' will result in True.

When using floats, consider using IRFixedPointField, which supports indexing and the same representation regardless of platform (unlike "float").

floats to work cross-platform. Use a fixed point number as the string type ( like myFixedPoint = '%2.5f' %( 10.12345 ) )

IRField supports "valueType", most other field types deal with a specific type and thus don't have such a parameter.

defaultValue

All fields (except IRClassicField) support a parameter, given when constructing the IRField object, "defaultValue".

For all fields (except IRClassicField), the value of this parameter defaults to "irNull" (see below). For an IRClassicField, the default remains empty string and cannot be changed (to be compatible with plain-string fields pre-5.0.0).

NULL Values

Null values are represented by a static singleton, called "irNull" (of type IRNullType).

For all types except IRClassicField (which has a default of empty string) the default (when unset) value of the field is irNull. This can be changed by passing "defaultValue=somethingElse" to the IRField constructor.

irNull does not equal empty string, or anything except another irNull. This is to destinguish say, no int assigned vs int(0)

You can check a typed field against the "irNull" variable found in the IndexedRedis or IndexedRedis.fields.

For Example:

from IndexedRedis import irNull

..

Null Values Can be used directly in the model filtering:

notDangerFive = MyModel.objects.filter(dangerLevel__ne=irNull).filter(dangerLevel__ne=5).all()

or in results, through Queryable List, :

myResults = MyModel.objects.filter(something='value').all()

notDangerFive = myResults.filter(dangerLevel__ne=irNull).filter(dangerLevel__ne=5)

or direct comparison :

if someObj.specialFlagX == irNull and formFields.get('waiverForm') != 'checked':

	setError( 'We introduced X on 06/16/15. Customers who signed up prior may not have yet filled out form Y to assign them a category Z.\nIn order to use this feature, you must either fill out form Y in your properties section, call a service represenative, or perform a one-time waiver (checking the box next to the "waiver" signifies your signature of this waiver).

Advanced Types

The following are the possible field types, for use within the FIELDS array, and can be imported like "from IndexedRedis.fields import NAME":

IRField - Standard field, takes a name and a "valueType", which is a native python type, or any type you create which implements __new__, taking a signle argument and returning the object. See IndexedRedis/fields/FieldValueTypes for example of how datetime and json are implemented.

When no valueType is defined, str/unicode is the type (same as pre-4.0), and default encoding is used (see set/getDefaultIREncoding functions)

Indexable unless type is a json type or float (use IRFixedPointField to index on floats)

IRBase64Field - Converts to and from Base64.

Indexable.

IRCompressedField - Automatically compresses before storage and decompresses after retrieval. Argument "compressMode" currently supports "zlib" (default), "bz2", or "lzma".

Indexable.

IRFixedPointField - A floating-point with a fixed number of decimal places. This type supports indexing using floats, whereas IRField(...valueType=float) does not, as different platforms have different accuracies, roundings, etc. Takes a parameter, decimalPlaces (default 5), to define the precision after the decimal point.

Indexable.

IRPickleField - Automaticly pickles the given object befo