SMCloudStore

SMCloudStore is a lightweight Node.js module that offers a simple API to interact with the object storage services of multiple cloud providers, including:

• AWS S3
• Azure Blob Storage
• Backblaze B2
• Google Cloud Storage
• Minio
• Other S3-compatible providers
• …more to come!

Features

• Simple, unified API to interact with all object storage providers
• Lightweight and flexible: each provider is published as a separate package, so installing SMCloudStore won't add SDKs for each vendor and thousands of dependencies to your projects
• Optimized for working with streams when putting/retrieving objects

This code is licensed under the terms of the MIT license (see LICENSE.md).

SMCloudStore vs other packages

SMCloudStore is specifically focused on abstracting the differences between multiple object storage providers, as storage is the most commonly consumed API from cloud providers that isn't fully standardized. This doesn't aim to support for other services that cloud providers might offer, such as databases, VMs, etc.

• By focusing only on storage, we can keep things simple. There are few APIs and they are clear and easy to use.
• SMCloudStore is highly modular, and every provider comes with a separate NPM package. This means that you don't need to install the SDKs for every single cloud provider and all their dependencies if you plan to use only one or two of them.
• We are focusing on a modern development experience. The codebase is primarily written in TypeScript and transpiled to JavaScript with all the typings published. We are also adopting a stream-centric approach with methods that upload and download objects from storage providers.

Add to your project

To start, install the smcloudstore package:

npm install --save smcloudstore

SMCloudStore requires Node.js version 8.9.1 or higher. Some providers might require a newer version (currently, Backblaze B2 requires Node.js 10 or higher).

Modules for each cloud provider are available on separate packages, so you can choose which ones to include. You need to install at least one of the following packages to use SMCloudStore:

• @smcloudstore/aws-s3: <br /> npm install --save @smcloudstore/aws-s3
• @smcloudstore/azure-storage: <br /> npm install --save @smcloudstore/azure-storage
• @smcloudstore/backblaze-b2: <br /> npm install --save @smcloudstore/backblaze-b2
• @smcloudstore/google-cloud-storage: <br /> npm install --save @smcloudstore/google-cloud-storage
• @smcloudstore/minio: <br /> npm install --save @smcloudstore/minio
• @smcloudstore/generic-s3: <br /> npm install --save @smcloudstore/generic-s3

Nomenclature

Each cloud provider use different names for the same concept. In SMCloudStore, we're standardizing to the following nomenclature:

| SMCloudStore | AWS | Azure | Backblaze | Google Cloud | Minio | | --- | --- | --- | --- | --- | --- | | Object | Object | Blob | File | Object | Object | | Container | Bucket | Container | Bucket | Bucket | Bucket |

API Guide

Full API documentation is available on this project's GitHub page and in the /docs folder.

Each cloud storage provider is implemented in a class defined in one of the modules above. All providers inherit from the StorageProvider abstract class, which is in the @smcloudstore/core package.

All asynchronous methods in the examples below return Promises, which can be used as then-ables or with async/await in ES2016. The examples below show using async/await syntax, and assume that all async calls are included in async functions.

Initialization

The main way to initialize the library is to use the SMCloudStore.Create(provider, connection) factory method. Using the factory method is recommended because it supports loading all providers with a "pluggable API", by just specifying their identifier in the first argument.

// Require the package
const SMCloudStore = require('smcloudstore')

// Identifier of the provider
const provider = 'minio'

// Complete with the connection options for the provider
const connection = {
    // ...
}

// Return an instance of the cloud storage provider class
const storage = SMCloudStore.create(provider, connection)

The format of the connection argument varies by cloud provider. For more details, please refer to the README for each provider in the packages/ folder.

Alternatively, you can create an instance of each provider by initializating the provider's class directly and invoking the constructor(connection) method. For example, to create a new Azure Blob Storage provider:

// Require the package
const AzureProvider = require('@smcloudstore/azure-storage')

// Complete with the connection options for the provider
const connection = {
    // ...
}

// Initialize the provider object
const storage = new AzureProvider(connection)

storage.createContainer(container, [options])

Using storage.createContainer(container, [options]) you can create a new container on the cloud storage server. The options argument is a dictionary with various options, depending on the provider being used. The method returns a Promise that resolves with no value when the container has been created.

// Create a new container called "testcontainer"
await storage.createContainer('testcontainer')

// Some providers, like AWS S3, require specifying a region
await storage.createContainer('testcontainer', {region: 'us-east-1'})

storage.isContainer(container)

The method storage.isContainer(container) returns a Promise that resolves with a boolean indicating whether a container exists on the provider.

// Once the async method resolves, exists will contain true or false
const exists = await storage.isContainer('testcontainer')

storage.ensureContainer(container, [options])

storage.ensureContainer(container, [options]) is similar to storage.createContainer(), but it creates the container only if it doesn't already exist. The method returns a Promise that resolves with no value when the container has been created.

// Container "testcontainer" will be created only if it doesn't already exists
await storage.ensureContainer('testcontainer')

// Some providers, like AWS S3, require specifying a region
await storage.ensureContainer('testcontainer', {region: 'us-east-1'})

storage.listContainers()

The method storage.listContainers() returns a Promise that resolves with the list of names of the containers that the user owns on the storage server.

// List all containers the user owns
const containers = await storage.listContainers()
// Result is an array of strings, like: ['testcontainer', 'testcontainer2']

storage.deleteContainer(container)

The method storage.deleteContainer(container) deletes a container from the storage server. It returns a Promise that resolves with no value on success.

// Delete a container
await storage.deleteContainer('testcontainer')

storage.putObject(container, path, data, [options])

storage.putObject(container, path, data, [options]) is the method to put (upload) an object to the storage server.

Arguments are:

• container: name of the destination container.
• path: full path inside the container where to store the object.
• data: the data to be uploaded. This could be a Readable Stream, or a string or Buffer containing the full data. Streams are preferred when dealing with larger amounts of data.
• options: dictionary with options. Options are primarily provider-dependent, so please refer to the documentation for each provider for more information. The list below includes only the common ones
  • options.metadata: object containing custom metadata and properties. An important key in the metadata object is Content-Type, which sets the Content-Type header for the file. Some providers might have special treatment for other keys too.

The method returns a Promise that resolves with no value when the upload is complete.

// Upload a stream
const data = require('fs').createReadStream('someimage.jpg')
const options = {
    metadata: {
        'Content-Type': 'image/jpeg'
    }
}
await storage.putObject('testcontainer'