Shepherd

⚠️ NerdWalletOSS Repository Archived & Relocated to Shepherd-Tools

This project has been relocated to shepherd-tools/shepherd and is no longer actively maintained by NerdWalletOSS.

Future development, issue tracking, and releases will continue at the new location.

👉 Please update your references and star ⭐ the new repo to stay up to date!

🙏 Special Thanks: We want to extend our heartfelt gratitude to Aori Nevo for their significant contributions and dedication to maintaining this project in recent years. Their work has been invaluable to the Shepherd community.

About Shepherd

Shepherd is a utility for applying code changes across many repositories.

• Powerful: You can write migration scripts using your favorite Unix commands, tools like jscodeshift, or scripts in your preferred programming language.
• Easy: With just a few commands, you can checkout dozens of repositories, apply changes, commit those changes, and open pull requests with detailed messages.
• Flexible: Ships with support for Git/GitHub, but can easily be extended to work with other version control products like Bitbucket, GitLab, or SVN.

For more high level context, this blog post covers the basics.

Getting started

Install the Shepherd CLI:

npm install -g @nerdwallet/shepherd

If using GitHub Enterprise, ensure the following environment variables are exported:

export SHEPHERD_GITHUB_ENTERPRISE_BASE_URL={company_github_enterprise_base_url} # e.g., github.com
export SHEPHERD_GITHUB_ENTERPRISE_URL={company_github_enterprise_url} # e.g., api.github.com/api/v3

If using ssh, ensure that your GITHUB_TOKEN is exported:

export GITHUB_TOKEN=<PAT>

Shepherd will now be available as the shepherd command in your shell:

shepherd --help

Usage: shepherd [options] [command]
...

Take a look at the tutorial for a detailed walkthrough of what Shepherd does and how it works, or read on for a higher-level and more brief look!

Go to tutorial →

Motivation for using Shepherd

Moving away from monorepos and monolithic applications has generally been a good thing for developers because it allows them to move quickly and independently from each other. However, it's easy to run into problems, especially if your code relies on shared libraries. Specifically, making a change to shared code and then trying to roll that shared code out to all consumers of that code becomes difficult:

• The person updating that library must communicate the change to consumers of the library
• The consumer must understand the change and how they have to update their own code
• The consumer must make the necessary changes in their own code
• The consumer must test, merge, and deploy those changes

Shepherd aims to help shift responsibility for the first three steps to the person actually making the change to the library. Since they have the best understanding of their change, they can write a code migration to automate that change and then user Shepherd to automate the process of applying that change to all relevant repos. Then the owners of the affected repos (who have the best understanding of their own code) can review and merge the changes. This process is especially efficient for teams who rely on continuous integration: automated tests can help repository owners have confidence that the code changes are working as expected.

Migration Configuration Schema

Example

A migration is declaratively specified with a shepherd.yml file called a spec. Here's an example of a migration spec that renames .eslintrc to .eslintrc.json in all NerdWallet repositories that have been modified in 2018:

id: 2018.07.16-eslintrc-json
title: Rename all .eslintrc files to .eslintrc.json
adapter:
  type: github
  search_type: code
  search_query: org:NerdWallet path:/ filename:.eslintrc
hooks:
  should_migrate:
    - ls .eslintrc # Check that this file actually exists in the repo
    - git log -1 --format=%cd | grep 2018 --silent # Only migrate things that have seen commits in 2018
  should_create_issue:
    - node $SHEPHERD_MIGRATION_DIR/should_create_issue.js
  post_checkout: npm install
  apply: mv .eslintrc .eslintrc.json
  pr_message: echo 'Hey! This PR renames `.eslintrc` to `.eslintrc.json`'
issues:
  title: 'this is my first updated issue'
  description: 'this is my first updated issue'
  labels: ['ENHANCEMENT', 'BUG']
  state: closed
  state_reason: completed

Fields

• id:

  • Description: Specifies a unique identifier for this migration.
  • Usage: Used as a branch name for the migration and internally by Shepherd to track migration state.

• title:

  • Description: A title for the migration.
  • Usage: Used as the commit message.

• adapter:

  • Description: Specifies the version control adapter for repo operations.
  • Details: Currently supports only GitHub, but can be extended for Bitbucket or GitLab. Configuration differs based on the adapter.
    • GitHub Specific:
      • search_query: Utilizes GitHub's code search qualifiers to identify candidate repositories.
      • org: Specify YOURGITHUBORGANIZATION to consider every visible repo in the organization.
      • search_type (optional): Either 'code' or 'repositories'. Defaults to code search. For repositories, use Github repository search.

Hooks

Hooks define the core functionality of a migration in Shepherd.

• should_migrate:

  • Description: Commands to determine if a repo requires migration.
  • Behavior: Non-zero exit values indicate the repo should not be migrated.

• should_create_issue:

  • Description: Commands to determine if issue needs to be posted in the repo.
  • Behavior: Non-zero exit values indicate the issue should not be posted to repo.

• post_checkout:

  • Description: Commands executed after a repo passes should_migrate checks.
  • Usage: Ideal for one-time setup actions per repo, like installing dependencies.

• apply:

  • Description: Commands that perform the actual migration.
  • Note: This can range from simple to complex sequences, depending on migration needs.

• pr_message:

  • Description: Commands to generate a pull request message.
  • Output: Anything written to stdout is used for the message. Multiple commands will have their outputs concatenated.

• issue:

  • Description: Command to create, update, or close issues.
  • Output: Depending on the details provided in migration scripts, the issues will be created, updated or closed.

• list-issues:

  • Description: Commands to list all issues associated with a migration.
  • Output: All the posted issues are listed in the table format.

Requirements

• Optional: should_migrate, post_checkout, should_create_issue
• Required: apply, pr_message

Environment Variables

Shepherd exposes some context to each command via specific environment variables. Some additional enviornment variables are exposed when using the git or github adapters.

| Environment Variable | Default | Description | | ------------------------------------- | ---------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | SHEPHERD_REPO_DIR | ~/.shepherd | the absolute path to the repository being operated on. This will be the working directory when commands are executed. | | SHEPHERD_DATA_DIR | ~/.shepherd | the absolute path to a special directory that can be used to persist state between steps. This would be useful if, for instance, a jscodeshift codemod in your apply hook generates a list of files that need human