HomaModule

This repo contains an implementation of the Homa transport protocol as a Linux kernel module.

• For more information on Homa in general, see the Homa Wiki.

• More information about this implementation and its performance are available in the paper A Linux Kernel Implementation of the Homa Transport Protocol, which appeared in the USENIX Annual Technical Conference in July, 2021.

• A synopsis of the protocol implemented by this module is available in protocol.md.

• As of August 2020, Homa has complete functionality for running real applications, and its tail latency is more than 10x better than TCP for all workloads I have measured (Homa's 99-th percentile latency is usually better than TCP's mean latency). Here is a list of the most significant functionality that is still missing:

  • The incast optimization from Section 3.6 of the SIGCOMM paper has not been implemented yet. If you would like to test Homa under large incasts, let me know and I will implement this feature.

• Please contact me if you have any problems using this repo; I'm happy to provide advice and support.

• The head is known to work under Linux 6.17.8. In the past, Homa has run under several earlier versions of Linux. There is a separate branch for each of these older versions, with names such as linux_4.15.18. Older branches except for linux_4.18.0 are out of date feature-wise: recent commits have not been back-ported to them. Other versions of Linux have not been tested and may require code changes (typically this is easy to do). If you get Homa working on some other version, please submit a pull request with the required code changes.

• Related work that you may find useful:

  • Preliminary support for using Homa with gRPC
  • A Go client that works with this module

• To build the module, type make all; then type sudo insmod homa.ko to install it, and sudo rmmod homa to remove an installed module. In practice, though, you'll probably want to do several other things as part of installing Homa. I have created a Python script that I use for installing Homa on clusters managed by the CloudLab project; it's in cloudlab/bin/config. I normally invoke it with no parameters to install and configure Homa on the current machine.

• The script cloudlab/bin/install_homa will copy relevant Homa files across a cluster of machines and configure Homa on each node. It assumes that nodes have names nodeN where N is a small integer, and it also assumes that you have already run make both in the top-level directory and in util.

• A collection of man pages is available in the "man" subdirectory. The API for Homa is different from TCP sockets.

• For best Homa performance, you should also make the following configuration changes:

  • Enable priority queues in your switches, selected by the 3 high-order bits of the DSCP field in IPv4 packet headers or the 4 high-order bits of the Traffic Class field in IPv6 headers. You can use sysctl to configure Homa's use of priorities (e.g., if you want it to use fewer than 8 levels). See the man page homa.7 for more info.
  • Enable jumbo frames on your switches and on the Linux nodes.

• NIC support: Homa is known to work with the following NICs:

  • Mellanox ConnectX-4, ConnectX-5, and ConnectX-6
  • Intel E810 series (ice) Please let me know if you find other NICs that work (or don't work).

• TSO support: in order to get best peformance Homa needs to take advantage of TSO support provided by NICs for TCP. This can happen in either of two ways. Some NICs, such as those from Mellanox/NVIDIA, will perform TSO even on packets with IP protocols other than TCP. Homa uses a header format that matches TCP's headers closely enough that TSO will work "out of the box". For other NICs, such as the Intel E810 series, the NIC refuses to perform TCP if the IP protocol is not TCP. For these NICs Homa has a "TCP hijacking" mode where it encapsulates its packets as TCP frames, with an IP protocol of TCP. Then, on the receiver side, Homa intercepts the incoming "TCP" packets and steals them back before they can be processed by TCP. To enable TCP hijacking, use sysctl to set the hijack_tcp parameter to 1 on all nodes.

• If you don't use TCP hijacking and your NICs don't support TSO for non-TCP protocols, then you must make sure that the max_gso_size parameter is no larger than the MTU (otherwise large outgoing packets will be dropped).

• The subdirectory "test" contains unit tests, which you can run by typing "make" in that subdirectory.

• The subdirectory "util" contains an assortment of utility programs that you may find useful in exercising and benchmarking Homa. Compile them by typing make in that subdirectory. Here are some examples of benchmarks you might find useful:

  • The cp_node program can be run stand-alone on clients and servers to run simple benchmarks. For a simple latency test, run cp_node server on node1 of the cluster, then run cp_node client on node 0. The client will send continuous back-to-back short requests to the server and output timing information. Or, run cp_node client --workload 500000 on the client: this will send continuous 500 KB messages for a simple througput test. Type cp_node --help to learn about other ways you can use this program.
  • The cp_vs_tcp script uses cp_node to run cluster-wide tests comparing Homa with TCP (and/or DCTCP); it was used to generate the data for Figures 3 and 4 in the Homa ATC paper. Here is an example command:

    cp_vs_tcp -n 10 -w w4 -b 20
    

    When invoked on node0, this will run a benchmark using the W4 workload from the ATC paper, running on 10 nodes and generating 20 Gbps of offered load (80% network load on a 25 Gbps network). Type cp_vs_tcp --help for information on all available options.
  • Other cp_ scripts can be used for different benchmarks. See util/README.md for more information.

• Some additional tools you might find useful:

  • Homa collects various metrics about its behavior, such as the size distribution of incoming messages. You can access these through the file /proc/net/homa_metrics. The script util/metrics.py will collect metrics and print out all the numbers that have changed since its last run.
  • Homa exports a collection of configuration parameters through the sysctl mechanism. For details, see the man page homa.7.

Significant changes

• March 2026: backported Homa to Linux version 4.18.0, using the linux_4.18.0 branch. Future changes made to the main branch are likely to be reflected in this branch also.
• January 2026: introduced new 'homa_qdisc' queuing discpline to improve performance when TCP and Homa run simultaneously. Results on c6620 CloudLab cluster (100 Gbps network):
  • Without homa_qdisc, if Homa and TCP run together, Homa performance suffers (4x increase for P99 for short messages) but TCP performance improves.
  • Homa_qdisc improves performance for both Homa and TCP, whether running stand-alone or together.
  • homa_qdisc improves Homa short message P99 3x when running together with TCP, but P99 is still slower than Homa standalone.
  • TCP performance improves when running together with Homa, with or without homa_qdisc.
• November 2025: upgraded to Linux 6.17.8.
• October 2025: added the HOMAIOCINFO ioctl for retrieving status information about a Homa socket. See man/homa.7 for details.
• May 2025: homa_api.c has been removed, so the functions homa_abort, homa_reply, homa_replyv, homa_send, and homa_sendv no longer exist.
• May 2025: added support for network namespaces.
• May 2025: reworked support for peers to cap peer memory usage.
• April 2025: upgraded to Linux 6.13.9.
• April 2025: major refactoring of grant management (more efficient, remove complexity that was causing an unending stream of bugs).
• March 2025: added memory cap on memory for outgoing messages: send requests can block if memory limit is reached.
• March 2025: implemented private RPCs, resulting in API changes. HOMA_RECVMSG_REQUEST and HOMA_RECVMSG_RESPONSE flags no longer exist and struct homa_sendmsg_args now has a flags field with one defined flag: HOMA_SENDMSG_PRIVATE.
• February 2025: by default, incoming requests for a socket are rejected unless the socket has been bound. setsockopt can be used with SO_HOMA_SERVER to enable or disable incoming requests for any socket.
• October 2024: the process of upstreaming Homa into the Linux kernel has begun. The reviewing process is likely to result in API changes. Upstreaming will occur in stages, so the first version to appear in Linux will not be either functionally complete or performant. The sources in this repository contain '#ifndef STRIP' directives, which separate functionality being upstreamed from functionality that is not currently upstreamed (some things, such as development aids, may never be upstreamed).
• October 2024: Homa now has an official IANA IP protocol number (146).
• August 2024: upgraded to Linux 6.10.6.
• July 2024: introduced "TCP hijacking", where Homa packets are sent as legitimate TCP segments (using TCP as the IP protocol) and then reclaimed from TCP on the destination. This allows Homa to make better use of TSO and RSS.
• June 2024: refactored sk_buff management to use frags; improves efficiency significantly.
• April 2024: replaced master branch with main
• July 2023: upgraded to Linux 6.1.38.
• December 2022: Version 2.0. This includes a n