<!--# SPDX-License-Identifier: GPL-2.0-or-later, Open Source License Attribution 4.0 \ International via https://creativecommons.org/licenses/by/4.0/ # 2024 Tesla Inc. # # TTPoE -- Tesla Transport Protocol over Ethernet -- Open Source Publication # # # TTP-Spec: Eric Quinnell <equinnell@tesla.com> # Doug Williams # Christopher Hsiong # Gerardo Navarro Hurtado # William Lemaire # Diwakar Tundlam # Mackenzie Goodwin # # TTP kernel A reference implementation of Tesla Transfer Protocol (TTP) that runs # directly over Ethernet Layer-2 Network. This is implemented as a Loadable # Kernel Module that establishes a TTP-peer connection with another instance # of the same module running on another Linux machine on the same Layer-2 # network. Since TTP runs over Ethernet, it is often referred to as TTP Over # Ethernet (TTPoE). # # The Protocol is specified to work at high bandwidths over 100Gbps and is # mainly designed to be implemented in Hardware as part of Tesla's DOJO # project. # # This public release of the TTP software implementation is aligned with the # patent disclosure and public release of the main TTP Protocol # specification. Users of this software module must take into consideration # those disclosures in addition to the license agreement mentioned here. # # Authors: Diwakar Tundlam <dntundlam@tesla.com> # Bill Chang <wichang@tesla.com> # Spencer Sharkey <spsharkey@tesla.com> # # Version: 08/26/2022 wichang@tesla.com, "Initial version" # 02/09/2023 spsharkey@tesla.com, "add ttpoe header parser + test" # 05/11/2023 dntundlam@tesla.com, "ttpoe layers - nwk, transport, payload" # 07/11/2023 dntundlam@tesla.com, "functional state-machine, added tests" # 08/29/2023 dntundlam@tesla.com, "final touches" # # This software is licensed under the terms of the GNU General Public License version 2 # as published by the Free Software Foundation, and may be copied, distributed, and # modified under those terms. # # This program is distributed in the hope that it will be useful, but WITHOUT ANY # WARRANTY; Without even the implied warranty of MERCHANTABILITY or FITNESS FOR A # PARTICULAR PURPOSE. See the GNU General Public License for more details. #-->

 /$$$$$$$$                  /$$                                                       
|__  $$__/                 | $$                                                       
   | $$  /$$$$$$   /$$$$$$$| $$  /$$$$$$                                              
   | $$ /$$__  $$ /$$_____/| $$ |____  $$                                             
   | $$| $$$$$$$$|  $$$$$$ | $$  /$$$$$$$                                             
   | $$| $$_____/ \____  $$| $$ /$$__  $$                                             
   | $$|  $$$$$$$ /$$$$$$$/| $$|  $$$$$$$                                             
   |__/ \_______/|_______/ |__/ \_______/                                             
                                                                                      
                                                                                    
 /$$$$$$$$                                                                     /$$    
|__  $$__/                                                                    | $$    
   | $$  /$$$$$$  /$$$$$$  /$$$$$$$   /$$$$$$$  /$$$$$$   /$$$$$$   /$$$$$$  /$$$$$$  
   | $$ /$$__  $$|____  $$| $$__  $$ /$$_____/ /$$__  $$ /$$__  $$ /$$__  $$|_  $$_/  
   | $$| $$  \__/ /$$$$$$$| $$  \ $$|  $$$$$$ | $$  \ $$| $$  \ $$| $$  \__/  | $$    
   | $$| $$      /$$__  $$| $$  | $$ \____  $$| $$  | $$| $$  | $$| $$        | $$ /$$
   | $$| $$     |  $$$$$$$| $$  | $$ /$$$$$$$/| $$$$$$$/|  $$$$$$/| $$        |  $$$$/
   |__/|__/      \_______/|__/  |__/|_______/ | $$____/  \______/ |__/         \___/  
                                              | $$                                    
                                              | $$                                    
                                              |__/                                    
 /$$$$$$$                       /$$                                   /$$             
| $$__  $$                     | $$                                  | $$             
| $$  \ $$ /$$$$$$   /$$$$$$  /$$$$$$    /$$$$$$   /$$$$$$$  /$$$$$$ | $$             
| $$$$$$$//$$__  $$ /$$__  $$|_  $$_/   /$$__  $$ /$$_____/ /$$__  $$| $$             
| $$____/| $$  \__/| $$  \ $$  | $$    | $$  \ $$| $$      | $$  \ $$| $$             
| $$     | $$      | $$  | $$  | $$ /$$| $$  | $$| $$      | $$  | $$| $$             
| $$     | $$      |  $$$$$$/  |  $$$$/|  $$$$$$/|  $$$$$$$|  $$$$$$/| $$             
|__/     |__/       \______/    \___/   \______/  \_______/ \______/ |__/             
                                                                                      
                                                                                

Introduction

At HotChips 2024, Tesla announced the open-sourcing of the Tesla Transport Protocol over Ethernet (TTPoE), represented on this GitHub repo.

Tesla also announced joining the Ultra Ethernet Consortium (UEC) to share this protocol and work to standardize a new high-speed/low-latency fabric (be that TTPoE or otherwise) for AI/ML/Datacenters -- desiring a non-proprietary, low cost, distributed congestion control, standard EthernetII frame, and non-centralized interconnect protocol to commoditize and accelerate technical progress.

In TTPoE, just like TCP, dropped packets and replays are the acceptable default behavior, yet full transmission is guaranteed.

TTPoE's initial deployment was for the Tesla Dojo v1 project, where the protocol executed entirely in hardware and deployed to a very large multi-ExaFlops (fp16) supercomputer with over 10s of thousands of concurrent endpoints. This protocol does not need a CPU or OS to be involved in any way to link and execute.

If you came here to be impressed by something complex and clever, you won't be. The protocol is designed on basic fundamentals -- simple transport and to the point. Ethernet transport in essence is only intended to move data from point A to B and should be limited by physics -- not software execution time. Centralized congestion management of extremely large scale machines (just like the internet) is a fool's errand -- each endpoint should be resiliant and self-managing.

Eric Quinnell -- Sept 13, 2024

TTPoE Transport Header

/* Transport Header (TTP) that follows TSH
*
*  15  14  13  12  11  10   9   8   7   6   5   4   3   2   1   0
* +---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+
* |         opcode [7:0]          |             vc [7:0]          |
* +---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+
* |             tx [7:0]          |             rx [7:0]          |
* +---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+
* |                          epoch [15:0]                         |
* +---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+
* |         congestion [7:0]      |      reserved-byte [7:0]      |
* +---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+
* |      reserved-byte [7:0]      |          extension [7:0]      |
* +---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+
* |                       tx-sequence [31:0]                      |
* |                                                               |
* +---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+
* |                       rx-sequence [31:0]                      |
* |                                                               |
* +---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+
*/

TTPoE Specification

You will find the spec
In the repo "doc" folder
TTPoE

ttpoe_sw

The following sections have details regarding making and executing the reference linux kernel sw model. It is matched to v1.5 of the TTP specification. Some variables may have changed slightly without documentation updates, but we're sure you can figure it out.

TTP Kernel Module (modttpoe.ko):

GIT repo info and unit tests

The source code git repo is at https://github.com/teslamotors/ttpoe. The code for modttpoe is under the 'modttpoe' subdir. The compilation is controlled via a top level Makefile in order to allow related modules to share symbols. Compilation is done as follows (gcc, linux-5.15-0-48-generic, ubuntu22):

$ pwd
GIT_HEAD

$ make all
.... <compilation output> ....
$ ls -l ./modttpoe/modttpoe.ko
-rw-rw-r-- 1 user group 3456000 Jan  1 10:00 modttpoe/modttpoe.ko

$ modinfo ./modttpoe/modttpoe.ko
filename:       /home/dojo-user/work/git/tesla/ttpoe_sw/modttpoe/modttpoe.ko
license:        GPL
version:        1.0
description:    TTP Over Ethernet
author:         dntundlam@tesla.com
srcversion:     547796AAC2C633E730FAF4F
depends:
retpoline:      Y
name:           modttpoe
vermagic:       6.11.0-9-generic SMP preempt mod_unload modversions
parm:           dev:      ttp device name (required at module-load)
parm:           dest_mac: ttp destination mac-address: e.g. dest_mac=xx:xx:xx:xx:xx:xx)
parm:           valid:    target is valid (default=0, or 1)
parm:           vci:      ttp conn-VCI (default=0, 1, 2)
parm:           use_gw:   use gateway to reach target (default=0, or 1)
parm:           nhmac:    next-hop mac address (format: xx:xx:xx:xx:xx:xx)
parm:           ipv4:     encap mode for TTP: 0 = TTPoE, 1 = TTPoIPv4 (read-only)
parm:           prefix:   ipv4 prefix: (A.B.C.D/N)
parm:           ipv4_sip: ipv4 src-ip: (A.B.C.D)
parm