picfit

.. image:: https://d262ilb51hltx0.cloudfront.net/max/800/1*oR04S6Ie7s1JctwjsDsN0w.png

picfit is a reusable Go server to manipulate images (resize, thumbnail, etc.).

It will act as a proxy on your storage engine and will be served ideally behind an HTTP cache system like varnish_.

It supports multiple storage backends <https://github.com/ulule/gostorages>_ and multiple key/value stores <https://github.com/ulule/gokvstores>_.

Installation

Build it

1. Make sure you have a Go language compiler and git installed.
2. Make sure you have the following go system dependencies in your $PATH: bzr, svn, hg, git
3. Ensure your GOPATH_ is properly set.
4. Download it:

::

git clone https://github.com/thoas/picfit.git

4. Run make build

You have now a binary version of picfit in the bin directory which fits perfectly with your architecture.

picfit has also a Docker integration to built a unix binary without having to install it

::

make docker-build

Debian and Ubuntu

We will provide Debian package when we will be completely stable ;)

Configuration

Configuration should be stored in a readable file and in JSON format.

The location of the configuration is specified by --config or the PICFIT_CONFIG_PATH environment variable.

config.json

.. code-block:: json

{
  "kvstore": {
    "type": "[KVSTORE]"
  },
  "storage": {
    "src": {
      "type": "[STORAGE]"
    }
  }
}

[KVSTORE] can be:

• redis - generated keys stored in Redis_, see below <#store-images-on-amazon-s3-keys-in-redis-and-shard-filename>_ how you can customize connection parameters
• cache - generated keys stored in an in-memory cache
• redis-cluster - generated keys stored in Redis cluster <https://redis.io/topics/cluster-tutorial>_
• redis-roundrobin - generated keys stored in Redis_, using multiple replicas to read keys, picfit will automatically detects a write error and round robin to the primary

[STORAGE] can be:

• fs - generated images stored in your File system
• http+fs - generated images stored in your File system and loaded using HTTP protocol
• s3 - generated images stored in Amazon S3
• dos3 - generated images stored in DigitalOcean S3
• gcs - generated images stored in Google Cloud Storage
• http+s3 - generated images stored in Amazon S3 and loaded using HTTP protocol
• http+dos3 - generated images stored in DigitalOcean S3 and loaded using HTTP protocol

Basic

• no key/value store
• no image storage
• images are given in absolute url

config.json

.. code-block:: json

{
  "port": 3001
}

Images are generated on the fly at each request.

Store images on file system and keys in an in-memory cache

• key/value in-memory store
• file system storage

An image is generated from your source storage (src) and uploaded asynchronously to this storage.

A unique key is generated and stored in a in-memory key/value store to process a request only once.

config.json

.. code-block:: json

{
  "port": 3001,
  "storage": {
    "src": {
      "type": "fs",
      "location": "/path/to/directory/"
    }
  },
  "kvstore": {
    "type": "cache"
  }
}

Store images on DigitalOcean S3

It's mostly the same as Amazon S3 storage, the difference are accepted regions So, regions can be:

• nyc1
• nyc2
• nyc3
• ams2
• ams3
• sfo1
• sfo2
• sgp1
• lon1
• fra1
• tor1
• blr1

Store images on Amazon S3, keys in Redis and shard filename

• key/value store provided by Redis
• Amazon S3 storage
• shard filename

config.json

.. code-block:: json

{
  "kvstore": {
    "type": "redis",
    "redis": {
      "host": "127.0.0.1",
      "port": 6379,
      "password": "",
      "db": 0
    }
  },
  "port": 3001,
  "storage": {
    "src": {
      "type": "s3",
      "access_key_id": "[ACCESS_KEY_ID]",
      "secret_access_key": "[SECRET_ACCESS_KEY]",
      "bucket_name": "[BUCKET_NAME]",
      "acl": "[ACL]",
      "region": "[REGION_NAME]",
      "location": "path/to/directory",
      "endpoint": "[protocol://service-code.region-code.amazonaws.com]"
    }
  },
  "shard": {
    "width": 1,
    "depth": 2,
    "restonly": true
  }
}

Keys will be stored on Redis_, (you better need to setup persistence_).

Image files will be loaded and stored on Amazon S3 at the location path/to/directory in the bucket [BUCKET_NAME].

[ACL] can be:

• private
• public-read
• public-read-write
• authenticated-read
• bucket-owner-read
• bucket-owner-full-control

[REGION_NAME] can be:

• us-gov-west-1
• us-east-1
• us-west-1
• us-west-2
• eu-west-1
• eu-central-1
• ap-southeast-1
• ap-southeast-2
• ap-northeast-1
• sa-east-1
• cn-north-1

Filename will be sharded:

• depth - 2 directories
• width - 1 letter for each directory
• restonly - true, filename won't contain characters in sharding path

Example:

06102586671300cd02ae90f1faa16897.png will become 0/6/102586671300cd02ae90f1faa16897.jpg

with restonly=false it would become 0/6/06102586671300cd02ae90f1faa16897.jpg

It would be useful if you are using the file system storage backend.

Load images from file system and store them in Amazon S3, keys on Redis cluster

• key/value store provided by Redis cluster
• File system to load images
• Amazon S3 storage to process images

config.json

.. code-block:: json

{
  "kvstore": {
    "type": "redis-cluster",
    "redis": {
      "addrs": [
        "127.0.0.1:6379"
      ],
      "password": "",
    }
  },
  "port": 3001,
  "storage": {
    "src": {
      "type": "fs",
      "location": "path/to/directory"
    },
    "dst": {
      "type": "s3",
      "access_key_id": "[ACCESS_KEY_ID]",
      "secret_access_key": "[SECRET_ACCESS_KEY]",
      "bucket_name": "[BUCKET_NAME]",
      "acl": "[ACL]",
      "region": "[REGION_NAME]",
      "location": "path/to/directory",
      "endpoint": "[protocol://service-code.region-code.amazonaws.com]"
    }
  }
}

You will be able to load and store your images from different storages backend.

In this example, images will be loaded from the file system storage and generated to the Amazon S3 storage.

Keys stored in Redis with multiple replicas

• key/value store provided by Redis with replicas

config.json

.. code-block:: json

{
  "kvstore": {
    "type": "redis-roundrobin",
    "redis-roundrobin": {
      "addrs": [
        "redis://127.0.0.1:6379?db=0",
        "redis://127.0.0.1:6380?db=0"
      ]
    }
  },
  "port": 3001
}

Load images from storage backend base url, store them in Amazon S3, keys prefixed on Redis

• key/value store provided by Redis
• File system to load images using HTTP method
• Amazon S3 storage to process images

config.json

.. code-block:: json

{
  "kvstore": {
    "type": "redis",
    "redis": {
      "host": "127.0.0.1",
      "port": 6379,
      "password": "",
      "db": 0
    },
    "prefix": "dummy:"
  },
  "port": 3001,
  "storage": {
    "src": {
      "type": "http+fs",
      "base_url": "http://media.example.com",
      "location": "path/to/directory"
    },
    "dst": {
      "type": "s3",
      "access_key_id": "[ACCESS_KEY_ID]",
      "secret_access_key": "[SECRET_ACCESS_KEY]",
      "bucket_name": "[BUCKET_NAME]",
      "acl": "[ACL]",
      "region": "[REGION_NAME]",
      "location": "path/to/directory"
    }
  }
}

In this example, images will be loaded from the file system storage using HTTP with base_url option and generated to the Amazon S3 storage.

Keys will be stored on Redis_ using the prefix dummy:.

Running

To run the application, issue the following command:

::

$ picfit -c config.json

By default, this will run the application on port 3001 and can be accessed by visiting:

::

http://localhost:3001

The port number can be configured with port option in your config file.

To see a list of all available options, run:

::

$ picfit --help

Usage

General parameters

Parameters to call the picfit service are:

.. code-block:: html

<img src="http://localhost:3001/{method}?url={url}&path={path}&w={width}&h={height}&upscale={upscale}&sig={sig}&op={operation}&fmt={format}&q={quality}&deg={degree}&pos={position}&filter={filter}"

• path - The filepath to load the image using your source storage
• operation - The operation to perform, see Operations_
• sig - The signature key which is the representation of your query string and your secret key, see Security_
• method - The method to perform, see Methods_
• url - The url of the image to generate (not required if path provided)
• width - The desired width of the image, if 0 is provided the service will calculate the ratio with height
• height - The desired height of the image, if 0 is provided the service will calculate the ratio with width
• upscale - If your image is smaller than your desired dimensions, the service will upscale it by default to fit your dimensions, you can disable this behavior by providing 0
• format - The output format to save t