Zeelos

NOTE: This project aims to be a sample prototype of utilizing Apache Kafka and Lightweight M2M (LWM2M) protocol as the backbone for cloud/edge IoT integration. It is mainly developed to spin out discussions around Kafka and it's usage in IoT and how can be leveraged to simplify and enhance both developer and user experience. It is not meant to be used as is in a production setting. That said, please open issues and make suggestions, will be happy to hear from you!

Architecture Overview

Setup

Hardware Prerequisites

We have verified the setup to work on the following 'edge' hardware:

• Asus Tinker Board running Armbian Debian Stretch - arm32v7 (2G Memory)
• Rock64 running Armbian Debian Stretch - arm64v8 (4G Memory)
• UP Squared running Ubilinux Debian Stretch - x64 (8G Memory)

NOTE: If you don't happen to have any of this hardware and you still like to test the setup, we suggest you spin out hosts on Scaleway that provide both Arm32v7/Arm64v8 and x64 hardware for users to test out. The cost is rather low (especially on Arm hardware), so it's easy to get started.

Software Prerequisites

We are utilizing Docker (tested on v18.06.1-ce) with it's Swarm orchestration and docker-app (tested on v0.6.0) for flexible configuration of the various services running on the different hardware architectures, so ensure you have those two tools installed. Once installed, enable the experimental features of Docker for extended metrics reporting to Prometheus, which will be visible on the Grafana dashboard.

To ease administration, ensure you have installed the Cockpit web administration interface on each cloud and edge node as well as the cockpit-leshan plugin we have developed for Leshan LWM2M administration. The plugin is an adaptation of the original Leshan web interface made to work inside Cockpit. (both deb and rpm packages are provided for easy installation). Further, we recommend to install cockpit-docker, cockpit-storaged and cockpit-networkmanager plugins for further introspection and administration of the cloud and edge hardware.

Step-by-Step

1. Initialize your Swarm cluster with at least one manager node and one worker. In the following 'saturn' host plays the role of a manager node with the other edge hardware playing the role of workers:

    ➜  zeelos git:(master) ✗ docker node ls
    ID                            HOSTNAME              STATUS              AVAILABILITY        MANAGER STATUS      ENGINE VERSION
    nn0xxqopfuu6bib06a4m477o3     rock64                Ready               Active                                  18.06.1-ce
    m8x5f0xet2bk9mb4od8m0bj7e *   saturn                Ready               Active              Leader              18.06.1-ce
    75cvshqeuozofkhdctretv4t4     tinkerboard           Ready               Active                                  18.06.1-ce
    kopkm2u73yyd1vngwusnmcfts     upboard               Ready               Active                                  18.06.1-ce
   

   NOTE: The manager node will be used to deploy all cloud services and in this guide we refer to is as the cloud node.

2. Assign labels to nodes so services would be propagated to the correct node upon deployment:

    docker node update --label-add type=cloud saturn
    docker node update --label-add type=upboard upboard
    docker node update --label-add type=tinkerboard tinkerboard
    docker node update --label-add type=rock64 rock64
   

3. Create overlay networks for cloud, monitoring and edge gateways:

    docker network create --driver overlay cloudnet
    docker network create --driver overlay monnet
    docker network create --driver overlay edgenet_upboard
    docker network create --driver overlay edgenet_tinkerboard
    docker network create --driver overlay edgenet_rock64
   

4. Generate certificates to enable SSL encryption and authentication in Kafka. We provide a convenience script based on the kafka-cluster-ssl script found in Confluent's docker distribution.

    cd security && ./certs-create.sh
   

   The generated certificates can be found inside the security/ folder and would be attached to Swarm cluster as secrets.

   NOTE: The main goal here was to enable SSL encryption both in cloud and edge Kafka nodes, in order to assess the overhead of SSL on hardware with limited resources. We do not advertise this approach as be a production ready, since it lacks many other security features (e.g. Authorization and ACLs).

5. Deploy monitoring stack:

    docker-app deploy monitoring
    
   

   Monitoring services are replicated globally and are multi-arch compatible so each node in the cluster will start running them as soon as the image downloads finish.

   NOTE: Since services are replicated globally, when a new node joins the swarm cluster, monitoring services are start automatically, easying the administration burden.

   Verify that monitoring stack has started by doing a 'docker ps' on an edge node:

    rock64@rock64:~$ docker ps
    CONTAINER ID        IMAGE                                              COMMAND                  CREATED             STATUS              PORTS                                                                              NAMES
    89726175821c        zeelos/docker_exporter:1.7.3.1                     "/bin/sh -c 'socat -…"   2 hours ago         Up 2 hours                                                                                             monitoring_docker-exporter.nn0xxqopfuu6bib06a4m477o3.g74jhiwapcr9itk1hdskpxp22
    128dd34dae0c        zeelos/cadvisor:v0.30.2                            "/usr/bin/cadvisor -…"   2 hours ago         Up 2 hours          8080/tcp                                                                           monitoring_cadvisor.nn0xxqopfuu6bib06a4m477o3.4ogyz9khudol6ehmyasnhu6es
    43bd5c3cbabf        zeelos/node_exporter:v0.16.0                       "/usr/bin/docker-ent…"   2 hours ago         Up 2 hours          8080/tcp                                                                           monitoring_node-exporter.nn0xxqopfuu6bib06a4m477o3.lq7skc5fvouuzze8gxq9ge7mo
   

   In the cloud node, verify that prometheus, alertmanager and unsee services have all been started too:

    ➜  zeelos@saturn docker ps               
    CONTAINER ID        IMAGE                                               COMMAND                  CREATED              STATUS                  PORTS                          NAMES
    d577a095669c        prom/alertmanager:v0.15.2                           "/bin/alertmanager -…"   8 hours ago          Up 8 hours              9093/tcp                       monitoring_alertmanager.1.h8v4kbtc07rerglv7ay19ey5r
    78f0f4e5ce46        prom/prometheus:v2.3.2                              "/bin/prometheus --c…"   8 hours ago          Up 8 hours              9090/tcp                       monitoring_prometheus.1.v25fdvrmx1t7jj7ar1fv16wyw
    5a8c73d11029        cloudflare/unsee:v0.9.2                             "/unsee"                 8 hours ago          Up 8 hours              8080/tcp                       monitoring_unsee.1.9ts5x0539h4j3x7r5nnxdpu1b
   

6. Deploy cloud stack:

    docker-app deploy cloud
   

   On the cloud node, verify that cloud services have all been started correctly:

    ➜  zeelos git:(master) ✗ docker ps               
    CONTAINER ID        IMAGE                                               COMMAND                  CREATED              STATUS                  PORTS                          NAMES
    3b3cb847f5a5        zeelos/kafka_exporter:v1.2.0                        "/usr/bin/kafka_expo…"   8 hours ago          Up 8 hours              8080/tcp                       cloud_kafka-exporter-cloud.1.t719fc3eczl8bu9kdfohuhweh
    f3d6c34512e3        confluentinc/cp-schema-registry:5.0.0               "/etc/confluent/dock…"   8 hours ago          Up 8 hours              8081/tcp                       cloud_schema-registry-cloud.1.q72ty6poet3blixsgx9qtpndk
    932d2b848122        confluentinc/cp-kafka-rest:5.0.0                    "/etc/confluent/dock…"   8 hours ago          Up 8 hours              8082/tcp                       cloud_kafka-rest-cloud.1.w7pa64cwbln4sghibwgkamkbd
    00829ab46593        confluentinc/cp-kafka-connect-base:5.0.0            "bash -c -a 'tail -f…"   8 hours ago          Up 8 hours              8083/tcp, 9092/tcp             cloud_kafka-command-client.1.3nmmz8lccvd25jq2md2dzyh3u
    c58406cb80ce        zeelos/influxdb:1.6.3-with-zeelosdb                 "/entrypoint.sh infl…"   8 hours ago          Up 8 hours              8086/tcp                       cloud_influxdb.1.9o0v5fzvnzfcnp717inhq6cy