orgu

orgu is a tool for implementing organization-wide workflows on GitHub. It is pronounced [ˈoɾɡu]. Blog post (ja).

Why Choose orgu?

Bridges the gap between GitHub's Team and Enterprise plans
Enhances cost efficiency

GitHub and GitHub Actions are powerful tools. However, there's a significant disparity between GitHub Team and Enterprise plans. Users often seek organization-wide workflows, which are implemented as GitHub's ruleset workflows, but this feature requires an Enterprise plan. orgu addresses this by enabling users to host their own organization-wide workflows.

orgu offers a more cost-effective solution compared to GitHub Actions, particularly in terms of unit cost and billing period. GitHub Actions bill on a "per minute" basis, and since organization-wide workflows tend to run frequently, cost efficiency is crucial.

Organization-wide workflows in GitHub are CI jobs that operate across all or selected repositories within an organization, based on repository attributes. These workflows are triggered by events such as pull requests or check suite events. Although reusable workflows can be employed, they still require the distribution of workflow files to each repository individually. In contrast, organization-wide workflows eliminate the need for such distribution, allowing specific CI jobs to be executed at the organization level without the need to configure each repository separately. This enables a more streamlined and efficient management of continuous integration tasks across multiple projects.

Architecture

Architecture diagram

orgu-runner sequence diagram

orgu consists of two components:

orgu-front
orgu-runner

orgu-front captures events related to Check Suites or Pull Requests via webhooks linked to a specific GitHub App. Upon receiving the webhook event, orgu-front filters the event, constructs a CheckRequest object, and forwards it to an event queue for further action.

orgu-runner operates an event loop that retrieves events from this queue, triggering specified jobs based on each event. It reports the start, success, or failure of jobs via the GitHub Checks API. Additionally, orgu-runner manages repository checkouts and generates temporary GitHub tokens, enabling jobs to interact with the GitHub API or checkout other repositories as needed. Feedback on job outcomes is provided by the jobs themselves, allowing the use of specialized tools like reviewdog for reporting job results or Opsgenie for alerts.

To execute organization-wide jobs, a unique container image must be created for each runner. This image should contain the orgu binary, all necessary job dependencies, and the job's execution script. Launching the orgu binary in "runner" mode within this container starts the orgu-runner's event loop.

See example/Dockerfile for more details on the orgu-runner container image.

Default Platform

By default, orgu uses AWS Lambda functions and Amazon EventBridge Event Bus for event queuing.

Filtering Events

In platforms like AWS Lambda, where billing occurs per invocation, users may want to minimize orgu-runner activations. By default, orgu filters CheckRequest elements generated from webhook events using event patterns in the EventBridge Event Bus. To help develop these event bus patterns, orgu includes commands like orgu pattern generate and orgu pattern test.

Custom Properties Support

GitHub repositories feature Custom Properties, allowing for the assignment of attributes. orgu supports these Custom Properties for filtering and referencing within jobs.

For filtering, orgu incorporates custom properties into CheckRequest.repository.custom_properties.

In runner jobs, orgu conveys custom properties as environment variables. For instance, the custom property team=t-platform translates to the CUSTOM_PROP_TEAM=t-ferris environment variable.

Installation

Homebrew:

brew tap Finatext/orgu https://github.com/Finatext/orgu.git
brew install orgu

Download from GitHub release:

https://github.com/Finatext/orgu/releases

Cargo install:

cargo install --git https://github.com/Finatext/orgu.git

Build locally:

cargo install --path .

Getting started

orgu can be run locally for development or testing. First, install the orgu binary by following the instructions in the "Installation" section.

Note: A GitHub organization is required to complete this tutorial.

Forwarding Webhooks to Your Local Machine

To forward GitHub webhooks to a local machine, use a service like https://smee.io/. Generate a Webhook URL on the service, then execute the forwarding command below:

# orgu-front uses `/github/events` path
$ smee -u https://smee.io/<ID> -P /github/events

Creating a GitHub App

Follow these steps to create a GitHub App:

Create the App: Follow the instructions in the GitHub documentation to create a GitHub App. Ensure it is set up to use webhooks. The "Webhook URL" should be the URL you created earlier.
Set Permissions: Refer to the "GitHub App Config" section in this documentation for the necessary permissions.
Generate a Webhook Secret: Use the following command to generate a random string for the "Webhook secret":
```
$ ruby -r securerandom -e 'puts SecureRandom.hex(32)'
```
Install the App: Once the GitHub App is created, install it within your organization. For testing purposes, it is recommended to limit the repositories the app can access.

Refer to the following GitHub documentation for more details:

Creating a Local Environment File

To configure your environment, you can use any tool that allows setting environment variables. A dotenv-like tool is recommended for this purpose. Follow these steps:

Install dotenv Tool: Install a dotenv-like tool, such as denv or dotenvx:

# Using denv
$ gem install denv

# Using dotenvx
$ brew install dotenvx/brew/dotenvx

Download the Private Key: Download the private key from the settings page of the GitHub App you created earlier. Note that the private key consists of multiple lines. Remove line breaks for the next steps.

Create a .env File: Create a .env file and add the following environment variables. Replace the placeholders with actual values:

$ cat <<EOS > .env
GITHUB_APP_ID=<copy from GitHub App page>
GITHUB_INSTALLATION_ID=<copy from GitHub App installation page>
GITHUB_WEBHOOK_SECRET=<webhook secret value>
GITHUB_PRIVATE_KEY=-----BEGIN RSA PRIVATE KEY-----your single line private key here-----END RSA PRIVATE KEY-----
EOS

Check Environment Variables: Verify the environment variables using the following command:
```
$ denv -- env
```

For more information on managing private keys for GitHub Apps, refer to the GitHub documentation:

Managing Private Keys for GitHub Apps

Run orgu-front

Execute orgu front server command with configured environment variables:

$ denv -- orgu front server

Run orgu-runner

Execute orgu runner server command with configured environment variables:

$ denv -- orgu runner server --job-name=test -- ls

Test orgu with PR

Create a test pull request in the configured repository. GitHub will send a webhook event to a forwarding service such as Smee, which then forwards the request to your local orgu-front server. In local development mode, the orgu-front server sends the CheckRequest payload directly to the local orgu-runner using an HTTP POST request. The orgu-runner executes the ls command in the checked-out repository, and the output of the ls command can be viewed in the orgu-runner's log entries.

Also the "Checks" tab in the pull request will have job execution logs.

If something went wrong:

Check webhook forwarding service server and client logs
Check webhook status in the "Recent Deliveries" page of GitHub App
- https://github.com/organizations/<org>/settings/apps/<app name>/advanced
Check the GitHUb App has necessary permissions
- See the "GitHub App Config" section of this document

Further steps

Replace ls with a linter command and use reviewdog to provide feedback on the results.

GitHub App Config

While it's possible to share a single GitHub App between orgu-front and orgu-runner, it's recommended to use separate GitHub Apps for orgu-front and each orgu-runner job.

orgu-front

Repository permissions:

R/W: Checks
R: Contents, Custom properties, Metadata, Pull requests

Webhook event subscription:

Pull request

Note: Necessary check_suite and check_run events are automatica

Orgu

Install / Use

README