Introduction

The eventdispatcher is our Snap! network library. It allows us to communicate between all our services between any number of computers.

We wanted all of our services to be event based. This library does that automatically with a very large number of features (TCP, UDP, FIFO, files, timers, etc.)

It was first part of our libsnapwebsites as:

snap_communicator.cpp/.h
snap_communicator_dispatcher.cpp/.h
tcp_client_server.cpp/.h
udp_client_server.cpp/.h

Now these are all broken up in separate files (84 of them at time of writing) and work with support from libraries found in our snapcpp contrib folder.

Features

The Event Dispatcher is capable of many feats, here are the main features found in this library:

TCP -- plain and encrypted (TLS with OpenSSL)
UDP -- direct and two broadcast methods
RPC -- text based message communication
dispatcher -- automatically dispatch RPC messages
events -- support many file descriptor based events
- accept socket (server)
- socket read
- socket write
- permanent socket (auto-reconnect on loss of connection)
- unix signal
- unix pipe (read/write)
- thread done
- listen to file changes on your local storage devices
- any number of timers
priority control -- objects can be given a priority
easy enable/disable of objects

Library Status

The library is considered functional. It is already in use in a couple of projects and works as expected.

The library being really large, all the features are not fully tested in those projects, so if you run into issues, let us know (see Bug Report below).

We plan in having full test coverage one day, but we have a problem with the fact that the communicator is a singleton. So we've been thinking about how to properly handle the testing of such a complex library.

Basic Principals

The library comes with three main parts:

Utilities -- a few of the functions are just utilities, such as helper functions and base classes
Connections -- many of the classes are named <...>_connection; these are used to create a connection or a listener; we currently support TCP, UDP, Unix sockets, FIFO, Unix signals, file system events
Communicator -- the communictor which is the core of the system; you can get its instance and call the run() function to run your process loop; the run function exits once all the connections are closed and removed from the communicator

The idea is to create processes which are 100% event based. These work by creating at least one connection and then adding that connection to the communicator. Then you call the run() function of the communicator. Your connection process_<name>() functions will then get called as events occur.

While running, you can add and remove connections at will.

Connections can be disabled and assigned a priority. The priority is useful if you want some of your process_<name>() called in a specific order.

Debugging Connections

Each connection you create should be given a unique name. Note that the same type of connection may be created multiple times. If possible, use a counter or some similar function to define a unique name.

my_connection::my_connection()
    : ...
{
    set_name("my_connection");
}

This helps you when you turn on the debug as the communicator and other parts of the code will be able to print the name of the connection being tweaked (added, removed, receiving a message, still being in the communicator when it shouldn't, etc.)

Here are a few functions you can use for this purpose:

communicator::set_show_connections()

This function tells the communicator that you want it to show you which connections it is listening to. This is particularly useful when trying to end your application cleanly by removing all your connections. If one remains behind, it is at times difficult to know which one.

This flag needs the debug connection log level to be appropriate to work.
communicator::debug_connections()

Defines the log level of the debugger used along the connection.

The log level needs to be really log to be useful. So at least SEVERITY_DEBUG. It is likely to be even better if you use SEVERITY_ALL.
communicator::log_connections()

A lighter weight than the debug_connections() above, this functions prints out the complete list of connections remaining after a remove_connection() happened. In many cases, this is sufficient.

Which function to use will generally depend on the number of logs you get and how often the remove versus the poll() happens.
dispatcher::set_trace()

When sending and receiving a lot of messages, it can be difficult to make sure that they travel where required. This function allows for the dispatcher to display messages as they are received.

IMPORTANT NOTE: This does not print anything on the sender side, only the receiver. In most cases, we have not found it useful to have it on the sender side. There logs can help you to make sure the correct functions get called as required.

In most cases, this is done at the time you initialize the dispatcher in your messenger class.

Timers

The base connection class is viewed as a timer, so that means all your connections also support a timer. If you create a connection object which is just a timer, then use the timer class. It will make your intend clearer and the class will simplify your own implementation.

All of our timers make use of the poll() timeout feature. We manage when/how timers trigger the process_timeout() function in the communicator run() function.

We have two types of timers. One which is triggered only once and one which can be triggered at specific times. The first one will always be triggered at least once when its time comes up. The second may miss triggers, especially if the service processing is generally slow or the time elapsed between triggers is very small.

Events Without Timers

In most cases you should strive in creating systems where timers are not required. The use of a timer usually means you are polling for a resource when in most likelihood you should be listening for an event and only use the resource once available.

There are, of course, exception to the rule. For example, our TCP client permanent message connection does a poll based on a timer. There is just no way to know whether a remote service is currently listening or not. So the only way to test that is attempt a connection. If that attempt fails, sleep a little and try again. In our case, we like to use a slippery time wait. The very first time, try to connect immediately. If that fails, we wait just 1 second and try again. For each new attempt, wait two times more than last time (so 2, 4, 8 etc. seconds) up to a maximum (say 1 hour between attempts). This generally works well, especially if you know that the service may be gone for good, trying to connect once per second is awful for everyone (client, server, and all the electronics in between which creates interferences on all your servers on your LAN).

Connection Layers

TCP

The TCP classes have five layers:

TCP class -- connect/disconnect sockets
Base class -- this is the simplest which just connects and calls your process_read() and process_write() functions
Buffered Class -- this class implements the process_read() and process_write() functions and bufferize the data for you.
Message Class -- the bufferized classes are further derived to create a message class which sees the incoming and ougoing data as IPC like messages (see the message.h/cpp files for details about the message support)
Permanent Class -- the TCP Message Class can also be made permanent; this means if the connection is lost, the class automatically takes care of reconnecting which all happens under the hood for you.
Blocking Message Class -- this classs is similar to the Message Class except it is possible to send a message and wait for the reply with a standard C++ call.

We have two types of clients. The ones that a client creates (such as the permanent message connection) and the ones that a server creates (the tcp_server_client_connection). The server client does not include a permanent message connection since it is not responsible to re-connect such a connection if it loses it.

The TCP server does not have any kind of buffer or message support since the only thing is does is a listen() and an accept().

UDP

The UDP classes are more limited than the TCP classes, especially since it doesn't require a permanent connection class (i.e. it is a connectionless protocol).

Base Class -- handle some common functions for the client & server
Client Message -- the UDP protocol being connectionless, we just offer a standalone function to send messages over UDP; keep in mind that UDP packets are small and we have no special handling for large message (i.e. the limit is around 1.5Kb)
Server Message -- the UDP server has a specific message connection and is capable of using the message dispather.

Unix Sockets

We have support for the stream Unix sockets. This is similar to the TCP socket. Like with any Unix socket, it is only available to clients running on the same computer the services they want to connect to.

Many of our services want to connect to the communicator service which is then in charge of forwarding the messages to other computers. That simplifies many of the communications. It is also capable of broadcasting as per rules setup in the communicator.

The Unix socket implementation is often used as an extra connection availability on a service. That way

Eventdispatcher

Install / Use

README