:showtitle: :toc: left :icons: font

= Oxide Control Plane

This repo houses the work-in-progress Oxide Rack control plane.

image::https://github.com/oxidecomputer/omicron/workflows/Rust/badge.svg[]

Omicron is open-source. But we're pretty focused on our own goals for the foreseeable future and not able to help external contributors. Please see xref:CONTRIBUTING.md[] for more information.

== Documentation

https://docs.oxide.computer/api[Docs are automatically generated for the public (externally-facing) API] based on the OpenAPI spec that itself is automatically generated from the server implementation. You can generate your own docs for either the public API or any of the internal APIs by feeding the corresponding OpenAPI specs (in link:./openapi[]) into an OpenAPI doc generator.

There are some internal design docs in the link:./docs[] directory. You might start with link:./docs/control-plane-architecture.adoc[].

For more design documentation and internal Rust API docs, see the https://rust.docs.corp.oxide.computer/omicron/[generated Rust documentation]. You can generate this yourself with:

[source,text]

$ cargo doc --document-private-items

Note that --document-private-items is configured by default, so you can actually just use cargo doc.

Folks with access to Oxide RFDs may find RFD 48 ("Control Plane Requirements") and other control plane RFDs relevant. These are not currently publicly available.

== Build and run

Omicron has two modes of operation: "simulated" and "non-simulated".

The simulated version of Omicron allows the high-level control plane logic to run without actually managing any sled-local resources. This version can be executed on Linux, Mac, and illumos. This mode of operation is provided for development and testing only.

To build and run the simulated version of Omicron, see: xref:docs/how-to-run-simulated.adoc[].

The non-simulated version of Omicron actually manages sled-local resources, and may only be executed on hosts running Helios. This mode of operation will be used in production.

To build and run the non-simulated version of Omicron, see: xref:docs/how-to-run.adoc[].

=== Run tests with nextest

The supported way to run tests is via https://nexte.st/[cargo-nextest].

NOTE: cargo test will not work for many of our tests, since they rely on nextest-specific features.

If you don't already have nextest installed, get started by https://nexte.st/book/pre-built-binaries[downloading a pre-built binary] or installing nextest via your package manager. Nextest has pre-built binaries for Linux, macOS and illumos.

Then, run tests with:

[source,text]

$ cargo nextest run

Nextest https://github.com/nextest-rs/nextest/issues/16[does not support doctests]. Run doctests separately with cargo test --doc.

Similarly, you can run tests inside a https://github.com/oxidecomputer/falcon[Falcon] based VM. This is described in the test-utils https://github.com/oxidecomputer/omicron/tree/main/test-utils[README].

There's also a xref:./live-tests/README.adoc[live-tests] test suite that can be run by hand in a deployed Omicron system.

=== rustfmt and clippy

You can format the code using cargo fmt. Make sure to run this before pushing changes. The CI checks that the code is correctly formatted.

You can run the https://github.com/rust-lang/rust-clippy[Clippy linter] using cargo xtask clippy. CI checks that code is clippy-clean.

== Working in Omicron

Omicron is a pretty large repo containing a bunch of related components. (Why? See xref:docs/repo.adoc[].) If you just build the whole thing with cargo build or cargo nextest run, it can take a while, even for incremental builds. Since most people are only working on a few of these components at a time, it's helpful to be know about Cargo's tools for working with individual packages in a workspace.

NOTE: This section assumes you're already familiar with the prerequisites and environment setup needed to do any work on Omicron. See xref:docs/how-to-run-simulated.adoc[] or xref:docs/how-to-run.adoc[] for more on that.

=== Key tips

• Use cargo check when you just want to know if your code compiles. It's much faster than cargo build or cargo nextest run.
• When using Cargo's check/build/test/clippy commands, you can use the -p PACKAGE flag to only operate on a specific package. This often saves a lot of time for incremental builds.
• When using Cargo's check/build/clippy commands, use --all-targets to make sure you're checking or building the test code, too.
• Use https://rust-analyzer.github.io/book/configuration#cargo.targetDir[cargo.targetDir] to give rust-analyzer a target directory other than ./target so it doesn't block you from running commands in the terminal. This uses extra disk space.

.How to set cargo.targetDir in various editors [%collapsible]

[source, toml] .Helix

[language-server.rust-analyzer.config] cargo.targetDir = true

====

These are explained a bit more below, along with some common pitfalls.

Here's an example workflow. Suppose you're working on some changes to the Nexus database model (nexus-db-model package, located at nexus/db-model from the root). While you're actively writing and checking code, you might run:

cargo check --all-targets

without any -p flag. Running this incrementally is pretty fast even on the whole workspace. This also uncovers places where your changes have broken code that uses this package. (If you're making big changes, you might not want that right away. In that case, you might choose to use -p nexus-db-model here.)

When you're ready to test the changes you've made, start with building and running tests for the most specific package you've changed:

cargo nextest run -p nexus-db-model

Once that works, check the tests for the next package up:

cargo nextest run -p omicron-nexus

When you're happy with things and want to make sure you haven't missed something, test everything:

cargo nextest run

=== Adding a new system library dependency

We check that certain system library dependencies are not leaked outside of their intended binaries via cargo xtask verify-libraries in CI. If you are adding a new dependency on a illumos/helios library it is recommended that you update xref:.cargo/xtask.toml[] with an allow list of where you expect the dependency to show up. For example some libraries such as libnvme.so.1 are only available in the global zone and therefore will not be present in any other zone. This check is here to help us catch any leakage before we go to deploy on a rack. You can inspect a compiled binary in the target directory for what it requires by using elfedit - for example elfedit -r -e 'dyn:tag NEEDED' /path/to/omicron/target/debug/sled-agent.

=== Checking feature flag combinations

To ensure that varying combinations of features compile, run cargo xtask check-features, which executes the https://github.com/taiki-e/cargo-hack[cargo hack] subcommand under the hood.

This xtask is run in CI using the --ci parameter , which automatically exludes certain image-* features that purposefully cause compiler errors if set and uses a pre-built binary.

If cargo hack is not already installed in omicron's out/ directory, a pre-built binary will be installed automatically depending on your operating system and architecture.

To limit the max number of simultaneous feature flags combined for checking, run the xtask with the --depth <NUM> flag:

[source,text]

$ cargo xtask check-features --depth 2

=== Rust packages in Omicron

NOTE: The term "package" is overloaded: most programming languages and operating systems have their own definitions of a package. On top of that, Omicron bundles up components into our own kind of "package" that gets delivered via the install and update systems. These are described in the package-manifest.toml file in the root of the repo. In this section, we're just concerned with Rust packages.

NOTE: There's also confusion in the Rust world about the terms https://doc.rust-lang.org/book/ch07-01-packages-and-crates.html["packages" and "crates"]. Packages are the things that have a Cargo.toml file. (Workspaces like Omicron itself have Cargo.toml files, too.) Packages are also the things that you publish to crates.io (confusingly). One package might have a library, a standalone executable binary, several examples, integration tests, etc. that are all compiled individually and produce separate artifacts. These are what Rust calls crates. We're generally just concerned with packages here, not crates.

Here are some of the big components in the control plane that live in this repo:

[cols="1,1,4",options="header"] |=== |Main rust package |Component |Description

|omicron-nexus |Nexus |Service responsible for handling external API requests and orchestrating the rest of the control plane.

|omicron-sled-agent |Sled Agent |Service that runs on each compute sled (server) to manage resources on that Sled

|dns-server |Internal DNS server, External DNS server |DNS server component used for both internal service discovery and external DNS

|omicron-gateway |Management Gateway Service |Connects Nexus (and other control plane services) to services on the rack management network (e.g., service processors)

|oximeter/oximeter |Oximeter |Collects telemetry from other services and stores it into Clickhouse

|wicket/wicketd |Wicket |CLI interface made available to operators on the rack technician port for rack setup and recovery

|===

For those with access to Oxide RFDs, RFD 61 discusses the organization principles and key components in more detail.

Many of these components themselves are made up of other packages (e.g., nexus-db-model is under omicron-nexus). There are also many more top-level packages than what's mentioned above. These are used for common code, clients, tools, etc.