cls3

The description in English is available on the following page. -> Blog

The description in Japanese is available on the following page. -> Blog

The description in Spanish is available on the following page. -> Blog

What is

The CLI tool "cls3" is to <ins>CL</ins>ear Amazon <ins>S3</ins> Buckets(AWS).

It <ins>empties</ins> (so deletes all objects and versions/delete-markers in) S3 Buckets or <ins>deletes</ins> the buckets themselves.

This tool allows you to search for bucket names and empty or delete multiple buckets.

Features

Bucket deletion option

Initially, the tool was intended to "empty the bucket", but since I was going to go through the trouble, I also added an option (-f|--force) to "delete the bucket as well".

Search for bucket names and delete multiple buckets

As described below (Interactive Mode), you can search for bucket names and delete or empty multiple buckets at once.

Cross-region

In deleting multiple buckets, you can list and delete them all at once, even if they are in multiple regions.

(In the Directory Buckets Mode for S3 Express One Zone (-d option), the Table Buckets Mode for S3 Tables (-t option), and the Vector Buckets Mode for S3 Vectors (-V option), operation across regions is not possible, but only in one region. You can specify the region with the -r option.)

Deletion of buckets with versioning enabled

Even if versioning is turned on, you can empty it just as if it were turned off. Therefore, you can use it without being aware of the versioning settings.

Deletion of old version objects only

The -o | --oldVersionsOnly option allows you to delete only old versions and all delete-markers without new versions and a bucket itself.

So you can retain the latest version objects only.

This option cannot be specified with the -f | --force option.

Deletion of Directory Buckets for S3 Express One Zone

The -d | --directoryBucketsMode option allows you to delete the Directory Buckets for S3 Express One Zone.

In this mode, operation across regions is not possible, but only in one region. You can specify the region with the -r option.

Deletion of Table Buckets for S3 Tables

The -t | --tableBucketsMode option allows you to delete the Table Buckets for S3 Tables.

In this mode, operation across regions is not possible, but only in one region. You can specify the region with the -r option.

Deletion of Vector Buckets for S3 Vectors

The -V | --vectorBucketsMode option allows you to delete the Vector Buckets for S3 Vectors.

In this mode, operation across regions is not possible, but only in one region. You can specify the region with the -r option.

Custom Endpoint URL

The -e | --endpointUrl option allows you to specify a custom endpoint URL to access S3-compatible storage or a specific S3 endpoint.

You can use cls3 with S3-compatible storage such as MinIO or Cloudflare R2 by specifying the custom endpoint URL.

Note: Google Cloud Storage is not supported as it uses a different API structure that is not fully S3-compatible.

You can also set the endpoint URL using the CLS3_ENDPOINT_URL environment variable. If both the environment variable and the command-line option are specified, the command-line option takes precedence.

export CLS3_ENDPOINT_URL=https://your-account.r2.cloudflarestorage.com

While AWS_ENDPOINT_URL and AWS_ENDPOINT_URL_S3 environment variables are also supported (via AWS SDK), we recommend using CLS3_ENDPOINT_URL for clarity and to avoid conflicts with other AWS tools.

Path-Style URL Addressing

The -P | --pathStyle option allows you to use path-style URL addressing (e.g., https://endpoint.com/bucket) instead of virtual-hosted-style (e.g., https://bucket.endpoint.com).

This option is typically used with the -e | --endpointUrl option when working with S3-compatible storage systems like Ceph or MinIO that require or prefer path-style URLs.

Note: This option cannot be used with Directory Buckets Mode (-d), Table Buckets Mode (-t), or Vector Buckets Mode (-V), as these AWS-specific features only support virtual-hosted-style URLs.

Concurrent Deletion of Multiple Buckets

The -c | --concurrentMode option allows you to delete multiple buckets in parallel. By default, when this option is specified, all buckets will be deleted in parallel.

If you want to limit the number of parallel deletions, you can specify the -n | --concurrencyNumber option. For example, -c -n 5 will delete buckets with a maximum of 5 parallel operations.

Too many parallel deletions may cause S3 API errors. If it fails, please run it again.

This option is not available in the Table Buckets Mode (-t) because the throttling threshold for S3 Tables is very low.

Number of objects that can be deleted

The delete-objects API provided by the CLI and SDK has a limit of "the number of objects that can be deleted in one command is limited to 1000", but This tool has no limit on the number.

Retry on 503 error

When there are tens of thousands of objects, a SlowDown error (503 error) may occur on the S3 api side in rare cases when deleting them all at once using the CLI or SDK.

When this occurs, cls3 responds by adding a mechanism that waits a few seconds and retries automatically several times.

Delete objects with a specific key prefix

The -k | --keyPrefix option allows you to delete objects with a specific key prefix.

For Directory Buckets, only prefixes that end in a delimiter ( / ) are supported. If you do not specify the delimiter, it will be added automatically.

For Table Buckets, the key prefix is not supported.

For Vector Buckets, this option allows you to delete indexes with a specific key prefix.

Install

• Homebrew

  brew install go-to-k/tap/cls3
  

• Linux, Darwin (macOS) and Windows

  curl -fsSL https://raw.githubusercontent.com/go-to-k/cls3/main/install.sh | sh
  cls3 -h
  
  # To install a specific version of cls3
  # e.g. version 0.13.2
  curl -fsSL https://raw.githubusercontent.com/go-to-k/cls3/main/install.sh | sh -s "v0.13.2"
  cls3 -h
  

• aqua

  aqua g -i go-to-k/cls3
  

• asdf

  # Add the plugin
  asdf plugin add cls3 https://github.com/xavbourdeau/cls3-asdf-plugin.git
  
  # Show all installable versions
  asdf list-all cls3
  
  # Install specific version
  asdf install cls3 latest
  
  # Set a version globally (on your ~/.tool-versions file)
  asdf global cls3 latest
  

• Binary

  • Releases

• Git Clone and install(for developers)

  git clone https://github.com/go-to-k/cls3.git
  cd cls3
  make install
  

How to use

cls3 -b <bucketName> [-b <bucketName>] [-p <profile>] [-r <region>] [-e|--endpointUrl <endpointUrl>] [-P|--pathStyle] [-f|--force] [-i|--interactive] [-o|--oldVersionsOnly] [-q|--quietMode] [-d|--directoryBucketsMode] [-t|--tableBucketsMode] [-V|--vectorBucketsMode] [-c|--concurrentMode] [-n|--concurrencyNumber <number>] [-k|--keyPrefix <keyPrefix>]

• -b, --bucketName: optional
  • Bucket name
    • Must be specified in not interactive mode
    • Otherwise (so in the interactive mode), you can not specify this!
  • Multiple specifications are possible.
    • cls3 -b test1 -b test2
• -p, --profile: optional
  • AWS profile name
• -r, --region: optional(default: us-east-1)
  • AWS Region
    • If this option is not specified and your AWS profile is tied to a region, the region is used instead of the default region.
  • It is not necessary to be aware of this as it can be used across regions.
    • But in the Directory Buckets Mode for S3 Express One Zone (with -d option), Table Buckets Mode for S3 Tables (with -t option), and Vector Buckets Mode for S3 Vectors (with -V option), you should specify the region. The mode is not available across regions.
• -e, --endpointUrl: optional
  • Custom endpoint URL to access S3-compatible storage or a specific S3 endpoint.
  • You can use cls3 with S3-compatible storage such as MinIO or Cloudflare R2 by specifying the custom endpoint URL.
    • Note: Google Cloud Storage is not supported as it uses a different API structure that is not fully S3-compatible.
  • This option can also be set using the CLS3_ENDPOINT_URL environment variable.
    • Example: export CLS3_ENDPOINT_URL=https://your-account.r2.cloudflarestorage.com
    • The command-line option takes precedence over the environment variable.
    • While AWS_ENDPOINT_URL and AWS_ENDPOINT_URL_S3 environment variables are also supported (via AWS SDK), we recommend using CLS3_ENDPOINT_URL for clarity and to avoid conflicts with other AWS tools.
• -P, --pathStyle: optional
  • Use path-style URL addressing (e.g., https://endpoint.com/bucket) instead of virtual-hosted-style (e.g., https://bucket.endpoint.com)
  • Typically used with -e | --endpointUrl when working with S3-compatible storage systems like Ceph or MinIO that require path-style URLs
  • Cannot be used with Directory Buckets Mode (-d), Table