S3Server

S3Server is a lightweight, high-performance .NET library that provides a server-side interface for building Amazon S3-compatible storage services. It parses incoming S3 HTTP requests and routes them to your callback implementations, allowing you to focus on storage logic rather than protocol details.

What is S3Server?

S3Server is a protocol adapter that handles the complexity of the Amazon S3 REST API, allowing you to build S3-compatible storage servers without dealing with HTTP parsing, XML serialization, signature validation, or AWS-specific request routing.

What S3Server does:

• Parses incoming S3 HTTP requests
• Determines request type (service, bucket, or object operations)
• Validates AWS Signature V4 (optional)
• Deserializes XML request bodies
• Routes requests to your callback methods
• Serializes response objects to XML
• Handles error responses with proper S3 error codes

What S3Server does NOT do:

• Store objects or buckets (you implement storage in your callbacks)
• Provide authentication or authorization logic (you control access in your callbacks)
• Manage metadata persistence (you handle metadata storage)

Want a complete S3-compatible storage server built using S3Server? Check out Less3.

Why Use S3Server?

• S3 API Compatibility: Build services that work with existing S3 clients (AWS SDK, CLI, MinIO client, etc.)
• Focus on Storage Logic: Spend your time implementing storage, not parsing HTTP requests
• Flexible Architecture: Complete control over where and how you store data
• Multi-Framework Support: Targets .NET 8.0 and .NET 10.0
• Production Ready: Handles path-style and virtual-hosted-style URLs, signature validation, multipart uploads, and more

Use Cases

• Custom S3-compatible storage backends: Build object storage on top of databases, file systems, cloud storage, or distributed systems
• S3 gateway services: Create proxies or gateways that translate S3 requests to other storage protocols
• Testing and development: Build mock S3 servers for testing applications without AWS dependencies
• Compliance and data residency: Keep complete control over data location and access patterns
• Feature extension: Add custom logic, caching, encryption, or auditing to S3 operations
• Cost optimization: Implement tiered storage or custom retention policies
• Air-gapped environments: Deploy S3-compatible storage in isolated networks

Features

✅ Complete S3 API Coverage

• Service operations (list buckets, check service)
• Bucket operations (CRUD, ACLs, tags, versioning, website config, logging, location)
• Object operations (CRUD, ACLs, tags, legal hold, retention, range reads)
• Multipart upload support (initiate, upload parts, complete, abort, list parts)
• S3 Select API support

✅ URL Style Support

• Path-style URLs: http://host:port/bucket/key (default)
• Virtual-hosted-style URLs: http://bucket.domain/key (configurable)

✅ Security & Validation

• AWS Signature V4 validation (optional)
• Pre-request hooks for authentication
• Post-request hooks for logging and metrics

✅ Developer Friendly

• Strongly-typed request/response objects
• Comprehensive error handling with S3-compliant error codes
• Detailed logging support
• Configurable operation limits

Quick Start

Installation

dotnet add package S3Server

Basic Example

using S3ServerLibrary;
using S3ServerLibrary.S3Objects;

namespace S3ServerLibrary
{
    using System;
    using System.Threading.Tasks;
    using WatsonWebserver.Core;

    // Configure server settings
    S3ServerSettings settings = new S3ServerSettings();
    settings.Webserver = new WebserverSettings("localhost", 8000, false);
    settings.Logger = Console.WriteLine;

    // Create and configure server
    S3Server server = new S3Server(settings);

    // Wire up callbacks
    server.Service.ListBuckets = async (ctx) =>
    {
        ListAllMyBucketsResult result = new ListAllMyBucketsResult();
        result.Owner = new Owner("admin", "Administrator");
        result.Buckets = new Buckets(new List<Bucket>
        {
            new Bucket("my-bucket", DateTime.UtcNow)
        });
        return result;
    };

    server.Bucket.Exists = async (ctx) =>
    {
        // Check if bucket exists in your storage
        return true;
    };

    server.Object.Write = async (ctx) =>
    {
        // Save object data from ctx.Request.Data stream
        Console.WriteLine($"Writing object: {ctx.Request.Bucket}/{ctx.Request.Key}");
        Console.WriteLine($"Content length: {ctx.Request.ContentLength}");
        // Implement your storage logic here
    };

    server.Object.Read = async (ctx) =>
    {
        // Retrieve object from your storage
        byte[] data = System.Text.Encoding.UTF8.GetBytes("Hello, S3!");
        return new S3Object(
            ctx.Request.Key,
            "version-1",
            true,
            DateTime.UtcNow,
            "etag-123",
            data.Length,
            new Owner("admin", "Administrator"),
            data,
            "text/plain",
            StorageClassEnum.STANDARD
        );
    };

    // Start server
    server.Start();
    Console.WriteLine("S3 Server listening on http://localhost:8000");
}

Configuration

Server Settings

S3ServerSettings settings = new S3ServerSettings
{
    // Required: Webserver configuration
    Webserver = new WebserverSettings("localhost", 8000, false),

    // Optional: Logger for diagnostic output
    Logger = (msg) => Console.WriteLine(msg),

    // Optional: Enable specific logging categories
    Logging = new LoggingSettings
    {
        HttpRequests = true,
        S3Requests = true,
        SignatureV4Validation = false
    },

    // Optional: Operation limits
    OperationLimits = new OperationLimitsSettings
    {
        MaxPutObjectSize = 5368709120 // 5GB default
    },

    // Optional: Enable AWS Signature V4 validation
    EnableSignatures = false,

    // Note: UseTcpServer is deprecated in v7.0; Watson now uses TCP natively
    UseTcpServer = false
};

Request Handlers

S3Server provides hooks to intercept requests at different stages:

// Pre-request handler (auth, logging, validation)
// Return true to terminate request, false to continue routing
settings.PreRequestHandler = async (ctx) =>
{
    // Check authentication
    if (!IsAuthenticated(ctx))
    {
        ctx.Response.StatusCode = 403;
        await ctx.Response.Send(ErrorCode.AccessDenied);
        return true; // Terminate
    }

    // Add custom metadata for downstream callbacks
    ctx.Metadata = new { UserId = "user123" };

    return false; // Continue to callback routing
};

// Default request handler (called when no callback matches)
settings.DefaultRequestHandler = async (ctx) =>
{
    Console.WriteLine($"Unhandled request: {ctx.Request.RequestType}");
    await ctx.Response.Send(ErrorCode.InvalidRequest);
};

// Post-request handler (logging, metrics)
settings.PostRequestHandler = async (ctx) =>
{
    Console.WriteLine($"Completed: {ctx.Request.RequestType} - {ctx.Response.StatusCode}");
    // Log metrics, update statistics, etc.
};

AWS Signature Validation

Enable AWS Signature V4 validation for authenticated requests:

settings.EnableSignatures = true;
settings.Logging.SignatureV4Validation = true; // Optional debug logging

// Implement callback to retrieve secret key for access key
server.Service.GetSecretKey = (ctx) =>
{
    string accessKey = ctx.Request.AccessKey;

    // Look up secret key for this access key
    // Return base64-encoded secret key
    return "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY";

    // Or throw exception if access key is invalid
    // throw new S3Exception(new Error(ErrorCode.InvalidAccessKeyId));
};

Note: Only AWS Signature V4 is supported. V2 signatures will return an error. Chunk signature validation is not yet supported.

Virtual-Hosted-Style URLs

Support bucket names in hostnames (http://bucket.s3.local/key instead of http://s3.local/bucket/key):

// 1. Use wildcard listener (requires admin privileges on Windows)
settings.Webserver.Hostname = "*"; // or "+" or "0.0.0.0"

// 2. Implement base domain finder
server.Service.FindMatchingBaseDomain = (hostname) =>
{
    // Input: "mybucket.s3.local.gd"
    // Output: "s3.local.gd" (the base domain)

    if (hostname.EndsWith(".s3.local.gd"))
        return "s3.local.gd";

    if (hostname.EndsWith(".s3.example.com"))
        return "s3.example.com";

    // No match found - will be treated as path-style
    throw new KeyNotFoundException($"No base domain for {hostname}");
};

DNS Configuration:

• Configure DNS or hosts file to resolve bucket subdomains
• For local testing: *.local.gd and *.fbi.com resolve to localhost
• Example: mybucket.s3.local.gd → 127.0.0.1

Callback Implementation Patterns

Pattern 1: Return Typed Result

server.Bucket.ReadAcl = async (ctx) =>
{
    AccessControlList acl = new AccessControlList(
        new List<Grant>
        {
            new Grant(
                new Grantee("admin", "Administrator", null, "CanonicalUser", "admin@example.com"),
                PermissionEnum.FullControl
            )
        }
    );

    return new Acce