=========== SocketShark

.. image:: https://github.com/closeio/socketshark/actions/workflows/test.yaml/badge.svg :target: https://github.com/closeio/socketshark/actions/workflows/test.yaml

SocketShark is a WebSocket message router based on Python/Redis/asyncio.

(Interested in working on projects like this? Close_ is looking for great engineers_ to join our team)

.. _Close: http://close.com .. _great engineers: http://jobs.close.com

.. contents::

Summary

SocketShark makes it easy to build WebSocket-based services without requiring those services to be aware of WebSockets. Instead, services implement HTTP endpoints for receiving messages from WebSocket clients, and publish messages to WebSocket clients via Redis, while SocketShark takes care of long-running WebSocket connections and passing messages between clients and services.

Features

• Pub-sub messages

  SocketShark allows building applications relying on the publish-subscribe pattern without having to be aware of long-running WebSocket connections. Subscriptions and published messages from the WebSocket client are forwarded to the application via HTTP. Messages can be pushed from the application to the WebSocket client synchronously by pushing them to Redis.

• Flexible WebSocket backend

  SocketShark comes with Websockets for Python 3 (websockets_) backend but can easily be adapted to other frameworks compatible with asyncio.

• Multiple services

  Through its configuration file, SocketShark can work with any number of services.

• Out-of-order message filtering

  If needed, an internal order can be supplied with messages from services, and SocketShark will automatically filter out out-of-order messages.

• Message throttling

  If needed, service messages can be throttled by SocketShark.

• Authentication

  SocketShark comes with ticket authentication built-in. To authenticate WebSocket connections, the client requests a temporary token from the app server and submits it when logging in. The token is then exchanged for a session/user ID that can be used to authenticate the user by the service backed by SocketShark.

• Authorization

  Pub-sub subscriptions can be authorized using a custom HTTP endpoint. SocketShark can periodically reauthorize subscriptions to ensure subscribers are unsubscribed if they're no longer authorized.

• Custom fields

  SocketShark supports custom application-specific fields for authentication and authorization purposes.

• Metrics

  SocketShark keeps track and reports metrics such as connection counts and successfully or unsuccessfully executed commands, with built-in Prometheus and logging backends.

• Connection maintenance

  SocketShark supports keeping the WebSocket connection alive and automatic discovery of its closure through automated server-side pings and handlers for client-side pings.

.. _websockets: https://websockets.readthedocs.io/

Quick start

See example_config.py for an example configuration file.

To start, install SocketShark (python setup.py install), create your own configuration file, and run SocketShark as follows:

.. code:: bash

PYTHONPATH=. socketshark -c my_config

Client Protocol

SocketShark uses WebSockets as the transport protocol for clients. This section describes the structure of the protocol between web clients and SocketShark.

Both clients and the server exchange JSON-messages. Each message is a JSON dict containing an event field which specifies the type of event. SocketShark supports the following events:

• auth: Authentication
• subscribe: Subscribe to a topic
• message: Send a message to a topic
• unsubscribe: Unsubscribe from a topic
• ping: Monitor connectivity

Responses usually contain a status field which can be ok or error. In case of an error, an error field is supplied containing the error description as a string.

Authentication

WebSockets clients can authenticate using the auth event using ticket authentication. For more information about ticket authentication see the Ticket-based authentication for session-based apps_ section.

The auth event takes an optional method (ticket is the only currently supported authentication method, and the default), and a ticket argument, containing the login ticket.

Example client request:

.. code:: json

{"event": "auth", "method": "ticket", "ticket": "SECRET_AUTH_TICKET"}

Example server responses (successful and unsuccessful):

.. code:: json

{"event": "auth", "status": "ok"}

.. code:: json

{"event": "auth", "status": "error", "error": "Authentication failed."}

Subscriptions

WebSocket clients can subscribe to any number of topics. Messages can be passed from the client to the server, and pushed from the server to the client at any time while subscribed to a topic. For example, a client may subscribe to an object ID, and the server may send a message whenever the object is updated. The server may include extra data when subscribing or unsubscribing. For example, the server might send the current state of the object when subscribing.

Topics are unique, and a client can be subscribed to each topic at most once. Extra fields can be associated with a subscription which are passed along with all subscription commands. For example, a client could be required to indicate the organization ID for a particular object subscription so that the service can authorize and process the message properly.

Subscribe

The ``subscribe`` event subscribes to a topic given in the ``subscription``
argument, which is composed of the service name and the topic, separated by
period. Extra fields can be defined by the service and directly specified in
the subscription message.

Example client request:

.. code:: json

  {"event": "subscribe", "subscription": "books.book_1"}

Example server responses (successful and unsuccessful):

.. code:: json

  {"event": "subscribe", "subscription": "books.book_1", "status": "ok"}

.. code:: json

  {
    "event": "subscribe",
    "subscription": "books.book_1",
    "status": "error",
    "error": "Book does not exist."
  }

Example server response with extra data:

.. code:: json

  {
    "event": "subscribe",
    "subscription": "books.book_1",
    "status": "ok",
    "data": {
      "title": "Everyone poops"
    }
  }

Example client request with extra fields:

.. code:: json

  {"event": "subscribe", "subscription": "books.book_1", "author_id": "author_1"}

Example successful server responses with extra fields:

.. code:: json

  {
    "event": "subscribe",
    "subscription": "books.book_1",
    "author_id": "author_1",
    "status": "ok"
  }

Note that the subscription name is unique for the subscription. When subscribed
to ``books.book_1`` we can't subscribe to another subcription with the same
name even if the ``author_id`` is different. However, the server could use the
``author_id`` to ensure the book matches the given author ID.

Message
~~~~~~~

Once subscribed, the ``message`` event can be used to pass messages. Message
data is contained in the ``data`` field, and should be dicts. The structure of
the data is up to the application protocol, and the service decides whether
messages are confirmed (successfully or unsuccessfully).

Example message (either client-to-server or server-to-client):

.. code:: json

  {
    "event": "message",
    "subscription": "books.book_1",
    "data": {
       "action": "update",
       "title": "New book title"
    }
  }

Example (optional) server-side message confirmation of a successful message
with extra data:

.. code:: json

  {
    "event": "message",
    "subscription": "books.book_1",
    "status": "ok",
    "data": {"status": "Book was updated."}
  }


Example (optional) server-side message confirmation of a failed message:

.. code:: json

  {
    "event": "message",
    "subscription": "books.book_1",
    "status": "error",
    "error": "Book could not be updated."
  }

If extra fields are passed with the subscription, they are included in all
``message`` events.

Note that a service may send messages limited to particular authentication
fields (e.g. limited to a specific user ID), so multiple sessions subscribed
to the same topic may not necessarily receive the same messages.

Unsubscribe

Clients can unsubscribe from a topic using the unsubscribe event.

Example client request:

.. code:: json

{"event": "unsubscribe", "subscription": "books.book_1"}

Example server responses (successful and unsuccessful):

.. code:: json

{"event": "unsubscribe", "subscription": "books.book_1", "status": "ok"}

.. code:: json

{ "event": "unsubscribe", "subscription": "books.book_1", "status": "error", "error": "Subscription does not exist." }

Ping

Clients can send a ``ping`` message and Socketshark will send a ``pong`` back
immediately, without contacting any services. Clients may choose to send pings
and monitor for pongs to e.g. detect failed WebSocket connections, display
latency metrics, etc. Furthermore, the ping message may contain some ``data``,
which the pong message should repeat back.

Example client request:

.. code:: json

  {"event": "ping", "data": "foobar"}

Example server response:

.. code:: json

  {"event": "pong", "data": "foobar"}


Service Protocol
================

SocketShark uses HTTP to send events to services, and Redis PUBSUB to receive
messages from services that are published to subscribed clients. This section
describes the structure of the protocol between services and SocketShark.

HTTP callbacks
--------------

An optional HTTP endpoint can be configured to authenticate a WebSocket
session. The authentication endpoint can return authentication-related fields
that can be configured (e.g. a user ID and/or session ID).

The following optional HTTP endpoints can be config