A tool to automatically synchronize and track bug bounty program scopes from multiple platforms. Monitor new programs, scope changes, and receive Discord notifications for updates.

⚠️ Version 2.x Warning

This is version 2.x of ScopesExtractor which contains breaking changes from version 1.x. If you're upgrading from 1.x, please review the CHANGELOG.md for more informations

Features

🔄 Multi-Platform Support: YesWeHack, HackerOne, Intigriti, Bugcrowd
📊 Automatic Synchronization: Continuously monitor programs and detect scope changes
🔔 Discord Notifications: Get notified about new programs, scope changes, and removals
🗄️ SQLite Database: Persistent storage with historical change tracking
🌐 REST API: Query scopes and changes programmatically
🎯 Smart Scope Processing: Automatic validation and normalization with platform-specific rules

Installation

Prerequisites

Ruby >= 3.4.0
SQLite3
libcurl (for Typhoeus)

Configuration

Copy the environment template:
```
cp .env.example .env
```
Configure platform credentials in .env (email, password, API tokens, TOTP secrets)
Configure application settings in config/settings.yml (enable/disable platforms, Discord webhooks, etc.)

Using Docker (Recommended)

# Build the image
docker build -t scopes_extractor .

# Create named volume for database
docker volume create scopes_db

# Run with mounted config and named volume for database
docker run -v $(pwd)/config/settings.yml:/app/config/settings.yml:ro \
           -v $(pwd)/.env:/app/.env:ro \
           -v scopes_db:/app/db \
           scopes_extractor \
           bundle exec bin/scopes_extractor sync

Local Installation

# Install dependencies
bundle install

# Run migrations
bundle exec bin/scopes_extractor migrate

CLI Usage

Commands

Sync Programs

# Sync all enabled platforms
bundle exec bin/scopes_extractor sync

# Sync specific platform
bundle exec bin/scopes_extractor sync hackerone

# Verbose output
bundle exec bin/scopes_extractor sync -v
bundle exec bin/scopes_extractor sync yeswehack --verbose

Start API Server

# Start API server
bundle exec bin/scopes_extractor serve

# Custom port and bind address
bundle exec bin/scopes_extractor serve -p 8080 -b 127.0.0.1

# Enable auto-sync in background
bundle exec bin/scopes_extractor serve --sync

# Verbose logging
bundle exec bin/scopes_extractor serve -v

Database Management

# Run migrations
bundle exec bin/scopes_extractor migrate

# Cleanup old history entries
bundle exec bin/scopes_extractor cleanup

# Reset database (WARNING: deletes all data)
bundle exec bin/scopes_extractor reset
bundle exec bin/scopes_extractor reset --force  # Skip confirmation

Other Commands

# Display version
bundle exec bin/scopes_extractor version

# Show help
bundle exec bin/scopes_extractor help

Docker Usage

Docker Compose

Create a docker-compose.yml:

services:
  scopes_extractor:
    build: .
    container_name: scopes_extractor
    volumes:
      - ./config/settings.yml:/app/config/settings.yml:ro
      - ./.env:/app/.env:ro
      - scopes_db:/app/db  # Use named volume for database
    ports:
      - "4567:4567"
    command: bundle exec bin/scopes_extractor serve --sync
    restart: unless-stopped

volumes:
  scopes_db:  # Persistent database volume

Run with:

docker-compose up -d

Docker Run Examples

# Create named volume first
docker volume create scopes_db

# Sync once
docker run --rm \
           -v $(pwd)/config/settings.yml:/app/config/settings.yml:ro \
           -v $(pwd)/.env:/app/.env:ro \
           -v scopes_db:/app/db \
           scopes_extractor \
           bundle exec bin/scopes_extractor sync

# Start API server with auto-sync
docker run -d \
           -v $(pwd)/config/settings.yml:/app/config/settings.yml:ro \
           -v $(pwd)/.env:/app/.env:ro \
           -v scopes_db:/app/db \
           -p 4567:4567 \
           --name scopes_extractor \
           scopes_extractor \
           bundle exec bin/scopes_extractor serve --sync

# View logs
docker logs -f scopes_extractor

API Documentation

The REST API provides programmatic access to scopes and change history.

<details> <summary>GET / - List all scopes</summary>

Query Parameters

| Parameter | Type | Description | |-----------|------|-------------| | platform | string | Filter by platform name (e.g., hackerone, bugcrowd) | | type | string | Filter by scope type (e.g., web, mobile, api) | | bounty | boolean | Filter by bounty status (true or false) | | slug | string | Filter by program slug | | values_only | boolean | Return only scope values as array |

Example Request

curl -H "X-API-KEY: your_api_key" "http://localhost:4567/?platform=hackerone&type=web&bounty=true"

Example Response

{
  "scopes": [
    {
      "slug": "example-program",
      "platform": "hackerone",
      "program_name": "Example Program",
      "bounty": true,
      "value": "*.example.com",
      "type": "web",
      "is_in_scope": true
    },
    {
      "slug": "example-program",
      "platform": "hackerone",
      "program_name": "Example Program",
      "bounty": true,
      "value": "api.example.com",
      "type": "web",
      "is_in_scope": true
    }
  ],
  "count": 2
}

Example Response (values_only=true)

[
  "*.example.com",
  "api.example.com"
]

</details> <details> <summary>GET /changes - Recent changes in history</summary>

Query Parameters

| Parameter | Type | Description | |-----------|------|-------------| | hours | integer | Number of hours to look back (default: 24) | | platform | string | Filter by platform name | | type | string | Filter by event type (add_program, remove_program, add_scope, remove_scope) |

Example Request

curl -H "X-API-KEY: your_api_key" "http://localhost:4567/changes?hours=48&platform=bugcrowd&type=new_scope"

Example Response

{
  "changes": [
    {
      "id": 123,
      "program_id": 45,
      "program_slug": "example-program",
      "platform_name": "bugcrowd",
      "event_type": "add_scope",
      "scope_value": "newapp.example.com",
      "scope_type": "web",
      "created_at": "2026-01-10T14:30:00Z"
    },
    {
      "id": 122,
      "program_id": 46,
      "program_slug": "another-program",
      "platform_name": "bugcrowd",
      "event_type": "add_scope",
      "scope_value": "*.another.com",
      "scope_type": "web",
      "created_at": "2026-01-10T12:15:00Z"
    }
  ],
  "count": 2
}

</details> <details> <summary>GET /wildcards - List all wildcard scopes</summary>

Query Parameters

| Parameter | Type | Description | |-----------|------|-------------| | platform | string | Filter by platform name | | values_only | boolean | Return only wildcard values as array |

Example Request

curl -H "X-API-KEY: your_api_key" "http://localhost:4567/wildcards?platform=hackerone"

Example Response

{
  "wildcards": [
    {
      "slug": "example-program",
      "platform": "hackerone",
      "program_name": "Example Program",
      "bounty": true,
      "value": "*.example.com",
      "type": "web",
      "is_in_scope": true
    },
    {
      "slug": "another-program",
      "platform": "hackerone",
      "program_name": "Another Program",
      "bounty": false,
      "value": "*.another.org",
      "type": "web",
      "is_in_scope": true
    }
  ],
  "count": 2
}

Example Response (values_only=true)

[
  "*.example.com",
  "*.another.org"
]

</details> <details> <summary>GET /exclusions - List all excluded/ignored assets</summary>

Example Request

curl -H "X-API-KEY: your_api_key" "http://localhost:4567/exclusions"

Example Response

{
  "exclusions": [
    {
      "id": 1,
      "value": "admin.example.com",
      "reason": "Out of scope - admin panel",
      "created_at": "2026-01-09T10:00:00Z"
    },
    {
      "id": 2,
      "value": "internal.example.com",
      "reason": "Internal use only",
      "created_at": "2026-01-08T15:30:00Z"
    }
  ],
  "count": 2
}

</details>

Discord Notifications

Configure Discord webhooks to receive real-time notifications:

Notification Types

new_program: New bug bounty program discovered
removed_program: Program no longer available
new_scope: New scope added to a program
removed_scope: Scope removed from a program
ignored_asset: Asset failed validation and was ignored

Scope Type Filtering

Use new_scope_types to filter which scope types trigger notifications:

discord:
  webhooks:
    main:
      new_scope_types: ["web"]  # Only notify for web scopes
      # Or leave empty/null to notify for all types

Scope Processing

ScopesExtractor includes intelligent scope processing with automatic normalization and validation.

<details> <summary>Auto-Heuristic Type Detection</summary>

Scopes are automatically categorized based on pattern matching, overriding platform

ScopesExtractor

Install / Use

README

Features

Installation

Prerequisites

Configuration

Using Docker (Recommended)

Local Installation

CLI Usage

Commands

Sync Programs

Start API Server

Database Management

Other Commands

Docker Usage

Docker Compose

Docker Run Examples

API Documentation

Query Parameters

Example Request

Example Response

Example Response (values_only=true)

Query Parameters

Example Request

Example Response

Query Parameters

Example Request

Example Response

Example Response (values_only=true)

Example Request

Example Response

Discord Notifications

Notification Types

Scope Type Filtering

Scope Processing