Franzy

Franzy is a suite of Clojure libraries for Apache Kafka. It includes libraries for Kafka consumers, producers, partitioners, callbacks, serializers, and deserializers. Additionally, there are libraries for administration, testing, mocking, running embedded Kafka brokers and zookeeper clusters, and more.

The main goal of Franzy is to make life easier for working with Kafka from Clojure. Franzy provides a foundation for building higher-level abstractions for whatever your needs might be.

Platform

Franzy breaks up its functionality into several different libraries to minimize dependency issues, especially on differing Kafka dependencies (ex: Server vs. Consumer/Producer).

| Name | Type | Description | Major Dependencies | |--------------------------------------------------------------|--------------------|-----------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------| | Franzy | client | This library - core client-oriented functionality, i.e. consumer, producer, schemas, more. | Franzy-Common, Kafka client | | Franzy Admin | client | Administer Kafka with Clojure, get Clojure data in/out, create topics, add partitions, list brokers, etc. | Franzy-Common, Kafka server (Scala/Java) | | Franzy Common | lib | Common functionality for any Franzy development, and useful for Kafka in general | Clojure, Schema | | Franzy Nippy | de/serializer | Nippy Serializer/Deserializer for Kafka. | Nippy | | Franzy Transit | de/serializer | Transit Serializer/Deserializer for Kafka. | Transit | | Franzy JSON | de/serializer | JSON/Smile Serializer/Deserializer for Kafka. | Cheshire | | Franzy Fressian | de/serializer | Fressian Serializer/Deserializer for Kafka. | Fressian | | Franzy Avro | de/serializer | AVRO Serializer/Deserializer for Kafka. | Abracad | | Franzy Embedded | embedded broker | Full featured embedded Kafka server for testing/dev, with multiple implementations including concrete types and components. | Kafka server | | Franzy Mocks | testing | Test your consumers and producers without a running Kafka cluster, and more in the future. | Franzy, Kafka client | | Franzy Examples | examples | Growing project of examples using all the above, to learn at your leisure. | All | | Travel Zoo | embedded Zookeeper | Embedded Zookeeper servers and clusters for testing and development, with concrete type and component versions available. | Curator Test |

Features

• Support for Kafka 0.9 (and above)
• Clojure types in and out of Kafka, no worrying about the Java API or Java types
• Support for Clojure 1.8+
• A light core of external dependencies to keep things light, future-proof, and in minimal conflict with your code
• Comprehensive consumer API with support for both manual and automatic partition assignment as well as offset management
• Producer API with support for synchronous and asynchronous production
• Support for metadata and metrics
• Choice of partitioning strategies (round-robin, range) and simple helpers/framework to implement your own
• Validation for all significant data types, including configuration, via schema
• Full, validated configuration from Clojure for Consumers, Producers, Brokers, and Kafka Connect - build your config as data
• Protocols and conversions for implementing your own consumers, producers, tests, conversions, and more
• Mock producer and consumer, via Franzy-Mocks
• Comprehensive Admin interface, including wrapping many undocumented/command-line only features via Franzy-Admin
• Helpers/framework for implementing custom callbacks for producers and consumers
• Helpers/framework for implementing your own serializers/deserializers
• Built-in serializers for keys and values for many data types and formats, including Strings, Integers, Longs, UUID, and Clojure Keywords, and EDN
• Add-on serializers for Nippy, JSON/JSON SMILE, and Fressian, with more to come
• A set of custom record types that fully wrap any data returned to and from Kafka, if you need, want, or prefer to use records rather than pure maps
• Ability to pass any complex parameters using provided record types which also conform to validateable schemas
• Embedded Kafka Server and components for testing via Franzy-Embedded
• Extensive examples, code comments, and documentation
• More, coming soon....

Why?

In addition to raw features, some reasons you may want to use Franzy:

• Comprehensive Kafka client
• Extreme care to not remove, distort, break, or diminish anything in the existing Java API
• Sane balance of performance vs. Clojure best-practices vs. ease-of-use
• Does not force any viewpoint about producing, consuming, administration, etc. on you beyond what Kafka already does
• À la carte - Lots of goodies and sugar, even for projects that are using mostly Java or don't need the consumer or producer at all. Build out what you need, no more, no less.
• Currently being used in a real project, where Kafka is the "spine" of the application, and thus, must be updated, fixed, and changed as needed
• Relatively future proof
• Designed to be a good fit with stream processors, particularly Onyx
• See Rationale

Requirements

Requirements may vary slightly depending on your intended usage.

• Clojure 1.8+ - You may be able to compile this library on/with earlier versions, but this is untested.
• Kafka 0.9+ - Some parts may work on earlier versions, but this is untested.

A good way to get started with Kafka is to use Docker and/or Vagrant. I recommend using a Docker compose stack with Kafka and Zookeeper that lets you scale up/down to test. You can also use the embedded Kafka and Zookeeper libraries listed above and discussed in the Testing/Dev section.

Installation

These libraries have had a few weeks of peer review and no issues thus far. I will be releasing some new versions shortly as I have time in the coming weeks. Thus far, there are no breaking API changes but I am open to any suggested changes or submissions. Please let me know and be ready for an upgrade soon. Thanks for your support.

[ymilky/franzy "0.0.1"]

Docs

• Read the browsable API Docs
• Franzy Examples for lots of notes, advice, and growing examples
• See source for more information about schemas, types, etc.
• For more about using, validating, and developing schemas, see Schema
• Commented source and tests
• See the doc folder for more.

Usage

The best way to learn is Franzy Examples and viewing the API docs, source, etc.

Below are a few naive examples to get you started.

Serialization

You'll need to pick a format in/out of Kafka.

I recommend you use Franzy-Nippy, but think carefully about your use-case. If you're just getting started, the built-in EDN Serializer is a good choice to keep things simple. Of course, all the built-in serializers in Kafka are accessible as well.

For the built-in serializers/deserializers, simply do something like this:

(ns my.ns
  (:require [franzy.serialization.deserializers :as deserializers]
            [franzy.serialization.serializers :as serializers]))

For the add-ons you'll have to reference them as separate dependencies obviously. They follow a pattern like this, replacing nippy with the serializer/deserializer name:

(ns my.ns
  (:require [fran