Microcule

<a href="https://travis-ci.org/Stackvana/microcule"><img src="https://travis-ci.org/Stackvana/microcule.svg" alt="build:"></a>

Software Development Kit and Command Line Interface for spawning streaming stateless HTTP microservices for any programming language or arbitrary binary.

Think of it as serverless functions meets Unix Philosophy.

see: 100+ Working Service Examples

Table of Contents

• Introduction
• Enterprise Ready
• Features
  • Modular
  • Universal
  • Intelligent HTTP Request Parsing
  • Consistent
  • Extendable
  • Versatile
  • No Containers
• Languages
  • c ( with gcc )
  • java
  • javascript ( first-class support )
  • coffee-script
  • common lisp
  • bash
  • lua
  • golang
  • ocaml
  • perl
  • php
  • python
  • python3
  • ruby
  • rust
  • r
  • scheme
  • smalltalk
  • tcl
• Installation
• Plugins
  • Body Parser
  • Compiler
  • Logger
  • Rate limiter
  • Mschema Request Validator
  • Source Github repo
  • Source Gist repo
  • Spawn
• Command Line Interface Usage
• Node.js HTTP Middleware Usage
• Spawning arbitrary compiled binaries
• Chaining Services as Middlewares
• Multiple Microservices Per Server Instance
• SSL/HTTPS
• Security
• 100+ Working Code Examples
• Etymology

Introduction

At it's core, microcule maps HTTP request response streams to the STDIN STDOUT streams of a function in any arbitrary programming language or any compiled binary. It's reminiscent of CGI, but utilizes additional STDIO streams, does not attempt to parse STDOUT for HTTP response methods, and ships with streaming plugins for extending your microservices. microcule is an old concept rethought and improved with the latest industry standard toolings.

If you are using Amazon Lambda or other cloud function hosting services like Google Functions or hook.io, you might find microcule a very interesting option to remove your dependency on third-party cloud providers. microcule allows for local deployment of enterprise ready microservices. microcule has few dependencies and will run anywhere Node.js can run.

quick demonstration video

Enterprise Ready

This project is the component which several production services, including hook.io, use to spawn real-time arbitrary streaming microservices in response to streaming HTTP requests. It's been battle-hardened with over two years of development and it's largest installation is now managing over 11,0000 microservices.

You are encouraged to use this module as-is, or modify it to suite your needs. If you are interested in contributing please let us know by opening a Pull Request.

Features

Modular

Only require the functionality you need.

microcule itself is actually just a collection of HTTP middleware Plugins presented as a CLI tool. This is essential, as all it's features and functionality are de-coupled with an industry standard API. You are encouraged to use the microcule binary shipped with this project, or use microcule plugins programmatically in your existing application.

Universal

• Supports Serverless functions in 20 programming languages! ( and counting )
• Supports Serverless with standard Unix or Linux tools ( like ls, echo, or tail )
• Full support for mapping HTTP -> STDIO streams
• Unix first. No custom APIs or buffered context

Intelligent HTTP Request Parsing

microcule can parse any kind of request data based on the incoming request's content-type header and will always stream requests by default.

• Query
• JSON
• Form
• Multipart
• Streaming
• Binary

Even binary data works great! Here is an example of resizing in image in JavaScript or Bash.

Consistent

• Spawns functions or arbitrary binaries in response to HTTP requests
• Uses a fresh system process per request per execution
• State of services clears on every request
• Service source code is immutable ( unless configured otherwise )
• Uses build step for compiled languages

Versatile

• Ships with microcule binary for starting HTTP microservice servers
• Scripting support for HTTP request / response API ( differs per language )
• Can serve any arbitrary binary like echo, ls, and tail as streaming HTTP microservices
• Can optionally accept STDIN and process scripts with Unix style pipes ( useful for using functions in DevOps! )

Extendable

• Should theoretically be able to work with all programming languages
• Should work with any existing Unix or Linux tool
• Provides simple customizable interfaces for programming language environments
• Uses Plugin System based on standard node.js HTTP middlewares
• Simplistic design makes it very easy to add features / add languages / create new plugins

No Containers

• By design, microcule ships with no container or OS virtualization
• Since it makes no assumptions about worker environment, microcule will work with any Container or Virtual Machine solutions
• Isolates state of microservice per system process and request ( stateless service requests )
• Handles Microservice error handling and timeouts

Read more about securing microcule

<a name="plugins"></a>

Plugins

microcule can be optionally extended through a simple app.use() based plugin architecture. Plugins are standard Node.js Express.js middlewares. This means you can use any existing Node.js middleware as a microcule plugin, or re-use any microcule plugin as a middleware in any existing Node application.

Available Plugins

• bodyParser - Intelligent streaming body parser ( JSON / form / multipart / binary )
• compile - Compile microservice source code into binary ( optional dependency to spawn )
• logger - Basic extendable request / response logger function
• mschema - Adds mschema validation to incoming request parameters
• sourceGithubGist - Pulls microservice source code from a Github Gist
• sourceGithubRepo - Pulls microservice source code from a Github Repository
• spawn - Spawns instance of microservice code in a new isolated child process. Maps HTTP req/res to process STDIN/STDOUT
• rateLimiter - Extendable request rate limiter. Holds rate limits in-memory or in Redis.

For Express based plugins example, see: ./examples/express-plugins.js

Since plugins are standard Node.js middlewares, writing custom plugins is very easy.

Languages

microcule supports 20+ programming languages. The best way to get started is to visit 100+ Microcule Examples.

Want to see a new language added? Simply open a pull request or open an issue. We are glad to add it!

Installation

npm install -g microcule

This will install the microcule binary globally on your system.

Usage

  Usage: microcule [command] [options]

  Commands:

    help  Display help

  Options:

    -t, --timeout <number>  Sets max timeout of service in milliseconds
    -h, --host <value>      Host to listen on
    -p, --port <number>     Port to listen on
    -l, --language <value>  Target programming languages
    -w, --watch <bool>      Reloads source files on every request ( dev only )
    -v, --version           Output the version number

By default, microcule will attempt to start a listening HTTP server based on a microservice file path.

Basic Command Line Usage

microcule ./path/to/scriptOrBinary.foo

100+ Microservice Examples

Here can find many example microservices which can be run with microcule

These same examples are available as live services at hook.io/examples.

CLI Examples

# mount any arbitrary command to a streaming HTTP endpoint

microcule echo "hello world"
microcule cat ReadMe.md
microcule tail ReadMe.md

# start HTTP servers with mounted streaming functions directly from source files

microcule ./examples/services/echo/echo.js
microcule ./examples/services/echo/echo-async-await.js
microcule ./examples/services/echo/echo.sh
microcule ./examples/services/echo/echo.c
microcule ./examples/services/echo/echo.go
microcule ./examples/services/hello-world/hello.java
microcule ./examples/services/echo/echo.lisp
microcule ./examples/services/echo/echo.lua
microcule ./examples/services/echo/echo.ml
microcule ./examples/services/echo/echo.php
microcule ./examples/services/echo/echo.pl
microcule ./examples/services/echo/echo.py
microcule ./examples/services/echo/echo-wsgi.py
microcule -l python3 ./examples/services/echo/echo-py3.py
microcule ./examples/services/echo/echo.r
microcule ./examples/services/echo/echo.rb
microcule ./examples/services/echo/echo.rs
microcule ./examples/services/echo/echo.coffee
microcule ./examples/services/echo/echo.ss
microcule ./examples/services/echo/echo.st
microcule ./examples/services/echo/echo.tcl

# pipe in data from arbitrary com