CloudProxy

cloudproxy

About The Project
Getting Started
Testing
Development
Usage
Rolling Deployments
Multi-Account Provider Support
API Examples
Roadmap
Limitations
Contributing
License
Contact
Troubleshooting

About The Project

The purpose of CloudProxy is to hide your scrapers IP behind the cloud. It allows you to spin up a pool of proxies using popular cloud providers with just an API token. No configuration needed.

CloudProxy exposes an API and modern UI for managing your proxy infrastructure. It includes:

Interactive API documentation with Swagger UI
Modern web interface for proxy management
Real-time proxy status monitoring
Easy scaling controls
Multi-provider support

Providers supported:

Planned:

Azure
Scaleway

Features:

Docker-first deployment - Simple, isolated, production-ready
Modern UI with real-time updates
Interactive API documentation
Multi-provider support
Multiple accounts per provider
Automatic proxy rotation
Rolling deployments - Zero-downtime proxy recycling
Health monitoring
Fixed proxy pool management (maintains target count)

Please always scrape nicely, respectfully.

Getting Started

To get a local copy up and running follow these simple steps.

Prerequisites

CloudProxy is designed to run as a Docker container:

Docker - Required for running CloudProxy (recommended for all deployments)
Python 3.9+ - Only needed for development or custom integrations

Installation

Docker Deployment (Recommended)

CloudProxy is distributed as a Docker image for easy deployment:

# Quick start with DigitalOcean
docker run -d \
  -e PROXY_USERNAME='your_username' \
  -e PROXY_PASSWORD='your_password' \
  -e DIGITALOCEAN_ENABLED=True \
  -e DIGITALOCEAN_ACCESS_TOKEN='your_token' \
  -p 8000:8000 \
  laffin/cloudproxy:latest

# Using environment file (recommended for production)
docker run -d \
  --env-file .env \
  -p 8000:8000 \
  laffin/cloudproxy:0.6.0-beta  # Use specific version tag

Docker Compose Example:

version: '3.8'
services:
  cloudproxy:
    image: laffin/cloudproxy:latest
    ports:
      - "8000:8000"
    env_file:
      - .env
    restart: unless-stopped

It is recommended to use a Docker image tagged to a specific version (e.g., laffin/cloudproxy:0.6.0-beta). See releases for the latest version.

Environment Configuration

Authentication

CloudProxy requires authentication configuration for the proxy servers:

PROXY_USERNAME, PROXY_PASSWORD - Basic authentication credentials (alphanumeric characters only)
ONLY_HOST_IP - Set to True to restrict access to the host server IP only
Both methods can be used simultaneously for enhanced security

Optional Settings

AGE_LIMIT - Proxy age limit in seconds (0 = disabled, default: disabled)

See individual provider documentation for provider-specific environment variables.

Python Package (Development & Integration)

For development or integrating CloudProxy into existing Python applications:

# Install from PyPI
pip install cloudproxy

# Or install from source for development
git clone https://github.com/claffin/cloudproxy.git
cd cloudproxy
pip install -e .

See the Python Package Usage Guide for development and integration examples.

Testing

CloudProxy includes a comprehensive test suite to ensure reliability:

Unit Tests

# Run all tests
pytest -v

# Run specific test file
pytest tests/test_specific.py -v

# Run with coverage
pytest --cov=cloudproxy tests/

End-to-End Testing

Warning: These tests create real cloud instances and will incur costs!

# Run full end-to-end test
./test_cloudproxy.sh

# Run without cleanup (for debugging)
./test_cloudproxy.sh --no-cleanup --skip-connection-test

# Test specific providers
./test_cloudproxy.sh --provider digitalocean

Development

Setting up Development Environment

# Clone the repository
git clone https://github.com/claffin/cloudproxy.git
cd cloudproxy

# Install in development mode
pip install -e .

# Install development dependencies
pip install -r requirements.txt

# Run the application locally
python -m cloudproxy

UI Development

# Navigate to UI directory
cd cloudproxy-ui

# Install dependencies
npm install

# Run development server (hot reload enabled)
npm run serve

# Build for production
npm run build

Adding a New Provider

Create a new directory under cloudproxy/providers/
Implement main.py with the provider orchestration logic
Implement functions.py with cloud API interactions
Follow the standard interface pattern used by existing providers
Add configuration handling in providers/config.py
Add tests in tests/test_provider_name.py

Code Style

Python code follows PEP 8 standards
Use type hints for function parameters and returns
Add comprehensive error handling with Loguru logging
Write unit tests for all new functionality

Usage

CloudProxy provides multiple interfaces for managing your proxy infrastructure:

Web Interface

Access the UI at http://localhost:8000/ui to:

View proxy status across all providers
Scale proxy instances up/down
Monitor health status
Remove individual proxies
View active proxy count

API Documentation

Access the interactive API documentation at http://localhost:8000/docs to:

Explore available endpoints
Test API calls directly
View request/response schemas
Understand authentication requirements

Programmatic Usage

CloudProxy exposes a RESTful API on localhost:8000. Your application can use the API to retrieve and manage proxy servers. All responses include metadata with request ID and timestamp for tracking.

Example of retrieving and using a random proxy:

import random
import requests

def get_random_proxy():
    response = requests.get("http://localhost:8000/random").json()
    return response["proxy"]["url"]

proxies = {
    "http": get_random_proxy(),
    "https": get_random_proxy()
}
my_request = requests.get("https://api.ipify.org", proxies=proxies)

For more detailed examples of using CloudProxy as a Python package, see the Python Package Usage Guide.

Rolling Deployments

CloudProxy supports rolling deployments to ensure zero-downtime proxy recycling. This feature maintains a minimum number of healthy proxies during age-based recycling operations.

Configuration

Enable rolling deployments with these environment variables:

# Enable rolling deployments
ROLLING_DEPLOYMENT=True

# Minimum proxies to keep available during recycling
ROLLING_MIN_AVAILABLE=3  

# Maximum proxies to recycle simultaneously
ROLLING_BATCH_SIZE=2

Cloudproxy

Install / Use

README