tfmigrate

A Terraform / OpenTofu state migration tool for GitOps.

Table of content

<!--ts-->

• Features
• Why?
• Requirements
• Getting Started
• Install
  • Homebrew
  • Download
  • Source
• Usage
• Configurations
  • Environment variables
  • Configuration file
    • tfmigrate block
    • history block
    • storage block
    • storage block (local)
    • storage block (s3)
    • storage block (gcs)
• Migration file
  • Environment Variables
  • migration block
  • migration block (state)
    • state mv
    • state xmv
    • state rm
    • state import
    • state replace-provider
  • migration block (multi_state)
    • multi_state mv
    • multi_state xmv
• Integrations
• License

<!--te-->

Features

• GitOps friendly: Write terraform state mv/rm/import commands in HCL, plan and apply it.
• Monorepo style support: Move resources to other tfstates to split and merge easily for refactoring.
• Dry run migration: Simulate state operations with a temporary local tfstate and check to see if terraform plan has no changes after the migration without updating remote tfstate.
• Migration history: Keep track of which migrations have been applied and apply all unapplied migrations in sequence.

You can apply terraform state operations in a declarative way.

In short, write the following migration file and save it as state_mv.hcl:

migration "state" "test" {
  dir = "dir1"
  actions = [
    "mv aws_security_group.foo aws_security_group.foo2",
    "mv aws_security_group.bar aws_security_group.bar2",
  ]
}

Then, apply it:

$ tfmigrate apply state_mv.hcl

It works as you expect, but it's just a text file, so you can commit it to git.

Why?

If you have been using Terraform in production for a long time, tfstate manipulations are unavoidable for various reasons. As you know, the terraform state command is your friend, but it's error-prone and not suitable for a GitOps workflow.

In team development, Terraform configurations are generally managed by git and states are shared via remote state storage which is outside of version control. It's a best practice for Terraform. However, most Terraform refactorings require not only configuration changes, but also state operations such as state mv/rm/import. It's not desirable to change the remote state before merging configuration changes. Your colleagues may be working on something else and your CI/CD pipeline continuously plan and apply their changes automatically. At the same time, you probably want to check to see if terraform plan has no changes after the migration before merging configuration changes.

To fit into the GitOps workflow, the answer is obvious. We should commit all terraform state operations to git. This brings us to a new paradigm, that is to say, Terraform state operation as Code!

Requirements

The tfmigrate invokes terraform or tofu command under the hood. This is because we want to support multiple Terraform / OpenTofu versions in a stable way.

Terraform

The minimum required version is Terraform v0.12 or higher, but we recommend the Terraform v1.x.

OpenTofu

If you want to use OpenTofu, a community fork of Terraform, you need to set the environment variable TFMIGRATE_EXEC_PATH to tofu.

The minimum required version is OpenTofu v1.6 or higher.

Terragrunt

Without dynamic state

If you are not leveraging terragrunt's dynamic state generation, the environment variable TF_MIGRATE_EXEC_PATH must be set based on your Terragrunt version.

• For Terragrunt < v0.73.0:

# As part of the command or via exporting the variable to your shell.
TFMIGRATE_EXEC_PATH=terragrunt tfmigrate $OTHEROPTIONS

• For Terragrunt >= v0.73.0 (due to the CLI redesign):

# As part of the command or via exporting the variable to your shell.
TFMIGRATE_EXEC_PATH="terragrunt run --" tfmigrate $OTHEROPTIONS

With dynamic state

As tfmigrate uses the local backend for planning, some command line flags for the remote state backend cannot be used. If you are leveraging terragrunt's dynamic state generation, the remote_state block must include a generate block.

remote_state {
  backend = "s3"

  config = {
    bucket         = "highway-terraform-state"
    # Other config here
  }
  # This ensures that a file instead of command line flags are used.
  # allowing tfmigrate to work as expected.
  generate = {
    path      = "backend.tf"
    if_exists = "overwrite_terragrunt"
  }
}

Getting Started

As you know, terraform state operations are dangerous if you don't understand what you are actually doing. If I were you, I wouldn't use a new tool in production from the start. So, we recommend you to play an example sandbox environment first, which is safe to run terraform state command without any credentials. The sandbox environment mocks the AWS API with localstack and doesn't actually create any resources. So you can safely run the tfmigrate and terraform commands, and easily understand how the tfmigrate works.

Build a sandbox environment with docker compose and run bash:

$ git clone https://github.com/minamijoyo/tfmigrate
$ cd tfmigrate/
$ docker compose build
$ docker compose run --rm tfmigrate /bin/bash

In the sandbox environment, create and initialize a working directory from test fixtures:

# mkdir -p tmp && cp -pr test-fixtures/backend_s3 tmp/dir1 && cd tmp/dir1
# terraform init
# cat main.tf

This example contains two aws_security_group resources:

resource "aws_security_group" "foo" {
  name = "foo"
}

resource "aws_security_group" "bar" {
  name = "bar"
}

Apply it and confirm that the state of resources are stored in the tfstate:

# terraform apply -auto-approve
# terraform state list
aws_security_group.bar
aws_security_group.foo

Now, let's rename aws_security_group.foo to aws_security_group.baz:

# cat << EOF > main.tf
resource "aws_security_group" "baz" {
  name = "foo"
}

resource "aws_security_group" "bar" {
  name = "bar"
}
EOF

At this point, of course, there are differences in the plan:

# terraform plan
(snip.)
Plan: 1 to add, 0 to change, 1 to destroy.

Now it's time for tfmigrate. Create a migration file:

# cat << EOF > tfmigrate_test.hcl
migration "state" "test" {
  actions = [
    "mv aws_security_group.foo aws_security_group.baz",
  ]
}
EOF

Run tfmigrate plan to check to see if terraform plan has no changes after the migration without updating remote tfstate:

# tfmigrate plan tfmigrate_test.hcl
(snip.)
YYYY/MM/DD hh:mm:ss [INFO] [migrator] state migrator plan success!
# echo $?
0

The plan command computes a new state by applying state migration operations to a temporary state. It will fail if terraform plan detects any diffs with the new state. If you are wondering how the tfmigrate command actually works, you can see all terraform commands executed by the tfmigrate with log level DEBUG:

# TFMIGRATE_LOG=DEBUG tfmigrate plan tfmigrate_test.hcl

If looks good, apply it:

# tfmigrate apply tfmigrate_test.hcl
(snip.)
YYYY/MM/DD hh:mm:ss [INFO] [migrator] state migrator apply success!
# echo $?
0

The apply command computes a new state and pushes it to remote state. It will fail if terraform plan detects any diffs with the new state.

You can confirm the latest remote state has no changes with terraform plan:

# terraform plan
(snip.)
No changes. Infrastructure is up-to-date.

# terraform state list
aws_security_group.bar
aws_security_group.baz

There is no magic. The tfmigrate just did the boring work for you.

Furthermore, you can also move resources to another directory. Let's split the tfstate in two. Create a new empty directory with a different remote state path:

# mkdir dir2
# cat config.tf | sed 's/test\/terraform.tfstate/dir2\/terraform.tfstate/' > dir2/config.tf

Move the resource definition of aws_security_group.baz in main.tf to dir2/main.tf and rename it to aws_security_group.baz2:

# cat << EOF > main.tf
resource "aws_security_group" "bar" {
  name = "bar"
}
EOF

# cat << EOF > dir2/main.tf
resource "aws_security_group" "baz2" {
  name = "foo"
}
EOF

Create a multi_state migration file:

# cat << EOF > tfmigrate_multi_state_test.hcl
migration "multi_state" "test" {
  from_dir = "."
  to_dir   = "dir2"

  actions = [
    "mv aws_security_group.baz aws_security_group.baz2",
  ]
}
EOF

Run tfmigrate plan & apply:

# tfmigrate plan tfmigrate_multi_state_test.hcl
# tfmigrate apply tfmigrate_multi_state_test.hcl

You can see the tfstate was split in two:

# terraform state list
aws_security_group.bar
# cd dir2 && terraform state list
aws_security_group.baz2