<p align="center"><img src="Thanos-logo_fullmedium.png" alt="Thanos Logo"></p>

Thanos Object Storage Client

objstore is a Go module providing unified interface and efficient clients to work with various object storage providers.

Features:

• Ability to perform common operations with clear contract against most popular object storages.
• High focus on efficiency and reliability required for distributed databases on object storages.
• Optional built-in YAML based configuration definition for consistent configuration.
• Optional Prometheus metric instrumentation for bucket operations.

This moduile is battle-tested and used on high scale production by projects like Thanos, Loki, Cortex, Mimir, Tempo, Parca and more.

Contributing

Contributions are very welcome! See our CONTRIBUTING.md for more information.

Community

Thanos is an open source project and we value and welcome new contributors and members of the community. Here are ways to get in touch with the community:

• Slack: #thanos
• Issue Tracker: GitHub Issues

Adopters

See [Adopters List](https://github.com/thanos-io/thanos/blob/main/website/data/adopters.yml.

Background

This library was initially developed as a Thanos objstore package. Thanos uses object storage as primary storage for metrics and metadata related to them. This package ended up being used by other projects like Cortex, Loki, Mimir, Tempo, Parca and more.

Given reusability, Thanos community promoted this package to standalone Go module with smaller amount of dependencies.

Maintainers

See MAINTAINERS.md

How to use objstore

The core this module is the Bucket interface:

// Bucket provides read and write access to an object storage bucket.
// NOTE: We assume strong consistency for write-read flow.
type Bucket interface {
	io.Closer
	BucketReader

	Provider() ObjProvider

	// Upload the contents of the reader as an object into the bucket.
	// Upload should be idempotent.
	Upload(ctx context.Context, name string, r io.Reader, opts ...ObjectUploadOption) error

	// Delete removes the object with the given name.
	// If object does not exist in the moment of deletion, Delete should throw error.
	Delete(ctx context.Context, name string) error

	// Name returns the bucket name for the provider.
	Name() string
}

All provider implementations have to implement Bucket interface that allows common read and write operations that all supported by all object providers. If you want to limit the code that will do bucket operation to only read access (smart idea, allowing to limit access permissions), you can use the BucketReader interface:

// BucketReader provides read access to an object storage bucket.
type BucketReader interface {
	// Iter calls f for each entry in the given directory (not recursive.). The argument to f is the full
	// object name including the prefix of the inspected directory.

	// Entries are passed to function in sorted order.
	Iter(ctx context.Context, dir string, f func(name string) error, options ...IterOption) error

	// IterWithAttributes calls f for each entry in the given directory similar to Iter.
	// In addition to Name, it also includes requested object attributes in the argument to f.
	//
	// Attributes can be requested using IterOption.
	// Not all IterOptions are supported by all providers, requesting for an unsupported option will fail with ErrOptionNotSupported.
	IterWithAttributes(ctx context.Context, dir string, f func(attrs IterObjectAttributes) error, options ...IterOption) error

	// SupportedIterOptions returns a list of supported IterOptions by the underlying provider.
	SupportedIterOptions() []IterOptionType

	// Get returns a reader for the given object name.
	Get(ctx context.Context, name string) (io.ReadCloser, error)

	// GetRange returns a new range reader for the given object name and range.
	GetRange(ctx context.Context, name string, off, length int64) (io.ReadCloser, error)

	// Exists checks if the given object exists in the bucket.
	Exists(ctx context.Context, name string) (bool, error)

	// IsObjNotFoundErr returns true if error means that object is not found. Relevant to Get operations.
	IsObjNotFoundErr(err error) bool

	// IsAccessDeniedErr returns true if access to object is denied.
	IsAccessDeniedErr(err error) bool

	// Attributes returns information about the specified object.
	Attributes(ctx context.Context, name string) (ObjectAttributes, error)
}

Those interfaces represent the object storage operations your code can use from objstore clients.

Factory

Generally, you have two ways of using objstore module:

First is to import the provider you want e.g. github.com/thanos-io/objstore/providers/s3 and instantiate it with available constructor (e.g. NewBucket).

The second option is to use the factory NewBucket(logger log.Logger, confContentYaml []byte, reg prometheus.Registerer, component string) that will instantiate the object storage client based on YAML file provided. The YAML file has generally the format like this:

type: <PROVIDER_TYPE>
config:
  <PROVIDER_TYPE specific options>

The exact option depends on provider and are in sections below.

NOTE: All code snippets are auto-generated from code and up-to-date.

Check out the Thanos documentation to see how Thanos uses this module.

Supported Providers (Clients)

Current object storage client implementations:

| Provider | Maturity | Aimed For | Auto-tested on CI | Maintainers | |-------------------------------------------------------------------------------------------|--------------------|-----------------------|-------------------|----------------------------------| | Google Cloud Storage | Stable | Production Usage | yes | @bwplotka | | AWS/S3 (and all S3-compatible storages e.g disk-based Minio) | Stable | Production Usage | yes | @bwplotka | | Azure Storage Account | Stable | Production Usage | no | @vglafirov,@phillebaba | | OpenStack Swift | Beta (working PoC) | Production Usage | yes | @FUSAKLA | | Tencent COS | Beta | Production Usage | no | @jojohappy,@hanjm | | AliYun OSS | Beta | Production Usage | no | @shaulboozhiao,@wujinhu | | Baidu BOS | Beta | Production Usage | no | @yahaa | | Local Filesystem | Stable | Testing and Demo only | yes | @bwplotka | | Oracle Cloud Infrastructure Object Storage | Beta | Production Usage | yes | @aarontams,@gaurav-05,@ericrrath | | HuaweiCloud OBS | Beta | Production Usage | no | @setoru |

Missing support to some object storage? Check out how to add your client section

NOTE: Currently Thanos requires strong consistency (write-read) for object store implementation for singleton Compaction purposes.

S3

Thanos uses the minio client library to upload Prometheus data into AWS S3.

NOTE: S3 client was designed for AWS S3, but it can be configured against other S3-compatible object storages e.g Ceph

The S3 object storage yaml configuration definition:

type: S3
config:
  bucket: ""
  endpoint: ""
  region: ""
  disable_dualstack: false
  aws_sdk_auth: false
  access_key: ""
  insecure: false
  signature_version2: false
  secret_key: ""
  session_token: ""
  put_user_metadata: {}
  http_config:
    idle_conn_timeout: 1m30s
    response_header_timeout: 2m
    insecure_skip_verify: false
    tls_handshake_timeout: 10s
    expect_continue_timeout: 1s
    max_idle_c