Frost

HTTP clients and a wrapper around pytest tests to verify that third party services are configured correctly. For example:

• Are our AWS DB snapshots publicly accessible?
• Are there dangling DNS entries in Route53?

Usage

Installing

1. Install Python 3.8
2. Run git clone git@github.com:mozilla/frost.git; cd frost; make install

Usage

$ frost --help
Usage: frost [OPTIONS] COMMAND [ARGS]...

  FiRefox Operations Security Testing API clients and tests

Options:
  --version  Show the version and exit.
  --help     Show this message and exit.

  Commands:
    list  Lists available test filenames packaged with frost.
    test  Run pytest tests passing all trailing args to pytest.

$ frost test --help
Usage: frost test [OPTIONS] [PYTEST_ARGS]...

  Run pytest tests passing all trailing args to pytest.

  Adds the pytest args:

  -s to disable capturing stdout
  https://docs.pytest.org/en/latest/capture.html

  and frost specific arg:

  --debug-calls to print AWS API calls

Options:
  --help  Show this message and exit.

Running

To fetch RDS resources from the cache or AWS API and check that backups are enabled for DB instances for the configured aws profile named default in the us-west-2 region

1. find the test file path:

$ frost list | grep rds
./aws/rds/test_rds_db_instance_backup_enabled.py
./aws/rds/test_rds_db_snapshot_encrypted.py
./aws/rds/test_rds_db_instance_is_postgres_with_invalid_certificate.py
./aws/rds/test_rds_db_instance_encrypted.py
./aws/rds/test_rds_db_security_group_does_not_grant_public_access.py
./aws/rds/test_rds_db_instance_not_publicly_accessible_by_vpc_sg.py
./aws/rds/test_rds_db_instance_minor_version_updates_enabled.py
./aws/rds/test_rds_db_instance_is_multiaz.py
./aws/rds/test_rds_db_snapshot_not_publicly_accessible.py

Note: packaged frost tests are relative to the frost install

1. run the test:

frost test aws/rds/test_rds_db_instance_backup_enabled.py --aws-profiles default

Frost adds the options:

• --aws-profiles for selecting one or more AWS profiles to fetch resources for or the AWS default profile / AWS_PROFILE environment variable
• --aws-regions for selecting one or more AWS regions to test as a CSV e.g. us-east-1,us-west-2. defaults to all regions
• --gcp-project-id for selecting the GCP project to test. Required for GCP tests
• --offline a flag to tell HTTP clients to not make requests and return empty params
• --config path to test custom config file

and produces output showing calls to the AWS API and failing for a DB instance with backups disabled:

============================================================ test session starts ============================================================
platform linux -- Python 3.8.2, pytest-6.0.2, py-1.9.0, pluggy-0.13.1
rootdir: /home/gguthe/frost
plugins: json-0.4.0, cov-2.10.0, html-1.20.0, metadata-1.10.0
collecting ... calling AWSAPICall(profile='default, region='ap-northeast-1', service='rds', method='describe_db_instances', args=[], kwargs={})
calling AWSAPICall(profile='default, region='ap-northeast-2', service='rds', method='describe_db_instances', args=[], kwargs={})
calling AWSAPICall(profile='default, region='ap-south-1', service='rds', method='describe_db_instances', args=[], kwargs={})
calling AWSAPICall(profile='default, region='ap-southeast-1', service='rds', method='describe_db_instances', args=[], kwargs={})
calling AWSAPICall(profile='default, region='ap-southeast-2', service='rds', method='describe_db_instances', args=[], kwargs={})
...
calling AWSAPICall(profile='default, region='us-west-2', service='rds', method='list_tags_for_resource', args=[], kwargs={'ResourceName': 'arn:aws:rds:us-west-2:redacted:db:test-db-ro-dev1'})
collected 21 items

aws/rds/test_rds_db_instance_backup_enabled.py F....................

================================================================= FAILURES ==================================================================
____________________________________ test_rds_db_instance_backup_enabled[test-db-ro-dev1] ___________________________________________________

rds_db_instance = {'AllocatedStorage': 250, 'AutoMinorVersionUpgrade': True, 'AvailabilityZone': 'us-east-1a', 'BackupRetentionPeriod': 0, ..
.}

    @pytest.mark.rds
    @pytest.mark.parametrize(
        "rds_db_instance", rds_db_instances_with_tags(), ids=get_db_instance_id,
    )
    def test_rds_db_instance_backup_enabled(rds_db_instance):
>       assert (
            rds_db_instance["BackupRetentionPeriod"] > 0
        ), "Backups disabled for {}".format(rds_db_instance["DBInstanceIdentifier"])
E       AssertionError: Backups disabled for test-db-ro-dev1
E       assert 0 > 0

aws/rds/test_rds_db_instance_backup_enabled.py:12: AssertionError
========================================================== short test summary info ==========================================================
FAILED aws/rds/test_rds_db_instance_backup_enabled.py::test_rds_db_instance_backup_enabled[test-db-ro-dev1] - AssertionError...
======================================================= 2 failed, 21 passed in 14.32s =======================================================

IAM Policy for frost

The below policy will allow you to run all AWS tests in frost against all resources in your account.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "PytestServicesReadOnly",
      "Action": [
        "autoscaling:DescribeLaunchConfigurations",
        "cloudtrail:DescribeTrails",
        "ec2:DescribeFlowLogs",
        "ec2:DescribeInstances",
        "ec2:DescribeSecurityGroups",
        "ec2:DescribeSnapshotAttribute",
        "ec2:DescribeSnapshots",
        "ec2:DescribeVolumes",
        "ec2:DescribeVpcs",
        "elasticache:DescribeCacheClusters",
        "elasticloadbalancing:DescribeLoadBalancers",
        "es:DescribeElasticsearchDomains",
        "es:ListDomainNames",
        "iam:GenerateCredentialReport",
        "iam:GetCredentialReport",
        "iam:GetLoginProfile",
        "iam:ListAccessKeys",
        "iam:ListAttachedGroupPolicies",
        "iam:ListAttachedRolePolicies",
        "iam:ListAttachedUserPolicies",
        "iam:ListGroupPolicies",
        "iam:ListGroupsForUser",
        "iam:ListMFADevices",
        "iam:ListRolePolicies",
        "iam:ListRoles",
        "iam:ListUserPolicies",
        "iam:ListUsers",
        "rds:DescribeDbInstances",
        "rds:DescribeDbSecurityGroups",
        "rds:DescribeDbSnapshotAttributes",
        "rds:DescribeDbSnapshots",
        "rds:ListTagsForResource",
        "redshift:DescribeClusterSecurityGroups",
        "redshift:DescribeClusters",
        "s3:GetBucketAcl",
        "s3:GetBucketCORS",
        "s3:GetBucketLogging",
        "s3:GetBucketPolicy",
        "s3:GetBucketVersioning",
        "s3:GetBucketWebsite",
        "s3:ListAllMyBuckets",
        "s3:ListBucket"
      ],
      "Effect": "Allow",
      "Resource": "*"
    }
  ]
}

Setting up GCP tests

Enabling required API's for your project

gcloud [--project <project name>] services enable bigquery-json.googleapis.com
gcloud [--project <project name>] services enable cloudresourcemanager.googleapis.com
gcloud [--project <project name>] services enable compute.googleapis.com
gcloud [--project <project name>] services enable sqladmin.googleapis.com

Setting up GSuite tests

Make sure to have an OAuth2 app created and have the client_secret.json file in ~/.credentials and then run:

make setup_gsuite

Caching

The AWS client will use AWS API JSON responses when available and save them using AWS profile, region, service name, service method, botocore args and kwargs in the cache key to filenames with the format .cache/v/pytest_aws:<aws profile>:<aws region>:<aws service>:<service method>:<args>:<kwargs>.json e.g.

head .cache/v/pytest_aws:cloudservices-aws-stage:us-west-2:rds:describe_db_instances::.json
{
    "DBInstances": [
        {
            "AllocatedStorage": 5,
            "AutoMinorVersionUpgrade": true,
            "AvailabilityZone": "us-west-2c",
            "BackupRetentionPeriod": 1,
            "CACertificateIdentifier": "rds-ca-2015",
            "CopyTagsToSnapshot": false,
            "DBInstanceArn": "arn:aws:rds:us-west-2:123456678901:db:test-db",

These files can be removed individually or all at once with the pytest --cache-clear option. The cache can be disabled entirely with the pytest -p no:cacheprovider.

Custom Test Config

frost adds a --config cli option for passing in a custom config file specific to tests within frost.

The example config in repo (config.yaml.example):

exemptions:
  - test_name: test_ec2_instance_has_required_tags
    test_param_id: i-0123456789f014c162
    expiration_day: 2019-01-01
    reason: ec2 instance has no owner
  - test_name: test_ec2_security_group_opens_specific_ports_to_all
    test_param_id: '*HoneyPot'
    expiration_day: 2020-01-01
    reason: purposefully insecure security group
severities:
  - test_name: test_ec2_instance_has_required_tags
    severity: INFO
  - test_name: '*'
    severity: ERROR
regressions:
  - test_name: test_ec2_security_group_opens_all_ports_to_all
    test_param_id: '*mycustomgroup'
    comment: this was