Proxy Forge

Proxy Forge is a free and open-source tool that helps you to set up OSI level 4 proxies with rotating IPs on the Digital Ocean platform. It is a powerful and innovative solution that addresses one of the most pressing challenges faced by developers and businesses in today's digital landscape: IP throttling and blacklisting. Proxy Forge empowers you to conquer these obstacles by providing a seamless, reliable, and easy-to-deploy solution.

In simpler terms, Proxy Forge helps you to create a pool of rotating IP addresses that you can use to access the internet. This can be useful for a variety of purposes, such as avoiding IP bans, scraping websites, and testing web applications.

Proxy Forge is a powerful tool that can be used by both developers and businesses. It is easy to use and deploy, and it provides a number of features that make it a valuable tool for anyone who needs to access the internet anonymously or avoid IP bans.

Features

• Rotating IPs: Automatically rotates IP addresses from a set of deployed droplets to avoid detection and blacklisting.
• Terraform Scripts: Provides Terraform scripts for straightforward setup and management.
• Scalability: Easily scale your proxy fleet up or down to handle your specific needs.
• Customizable Configuration: Fine-tune proxy settings to match your requirements. (check below)
• IP Throttling Avoidance: Effectively bypass IP throttling mechanisms used by websites and services.
• Detailed Logging: Comprehensive logging for monitoring and troubleshooting.
• Open Source: Proxy Forge is open-source, allowing for community contributions and customization.

Motivation

I had to find a solution for one of my clients where certain eCommerce websites were supposed to be scraped but only allowed IP addresses from the UK region. I purchased a VPC from a cloud provider in the UK, but I later discovered that the target website implements IP throttling, which temporarily disables it. As a result, I built this solution manually (no Terraform scripts) and without load balancers.

I've been learning Terraform since then, and automated its setup for you. Now it takes less than 5 minutes to complete.

Why DigitalOcean?

There are many reasons why you might choose to use DigitalOcean over other cloud providers. Some of the key benefits include:

• Simplicity: DigitalOcean is known for its simple and user-friendly interface. It is easy to get started with DigitalOcean, even if you are new to cloud computing.
• Affordability: DigitalOcean offers competitive pricing for its cloud services. Additionally, it provides a substantial 2 TB of free bandwidth each month.
• Performance: DigitalOcean uses high-performance hardware for its Droplets, so you can be sure that your applications will run smoothly.
• Reliability: DigitalOcean has a strong track record of reliability and uptime.
• Community: DigitalOcean has a large and active community of users and developers. This means that there is a wealth of resources available to help you get the most out of your DigitalOcean account.

Proof of Concept

<center>

</center>

Up and Running

Prerequisites

Before using Proxy Forge, ensure you have the following prerequisites in place:

1. Digital Ocean Account: If you do not have an account, I would recommend creating it through my referral to get 200 USD credits for 60 days. In this time you can test this solution.

2. Terraform installed on your local machine

Steps

1. Clone the repository

   git clone --depth=1 --branch=main https://github.com/tbhaxor/ProxyForge.git proxyforge
   cd proxyforge
   

2. Create a file for terraform variables.

   touch terraform.tfvars
   

   Note Skip it, if you wish to provide the variables and values from environment config.

3. Setup at least token and region in the file. See variables section below.

4. Initialize the provider, plan and apply the changes

   terraform init
   terraform plan
   terraform apply
   

5. Wait for some time for squid proxy to setup and load balancer to initialize routing.

6. (Optional) Test the deployment

   LOAD_BALANCER_IP=$(terraform output -raw lb-ip)
   SQUID_USERNAME=proxyforge # assuming you did not change squid-credentials.username in the tfvars
   SQUID_PASSWORD=proxyforge # assuming you did not change squid-credentials.password in the tfvars
   
   while true; do curl -x "http://$SQUID_USERNAME:$SQUID_PASSWORD@$LOAD_BALANCER_IP:80" https://ifconfig.me; echo; done
   

Variables

| Variable Name | Default | Description | | :-----------------: | :----------------------------------------------------: | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | token | N/A | Required API token with read/write permissions. see more | | region | N/A | Required Datacenter region to deploy all the resources. see more | | ssh-fingerprint | null | SSH fingerprint id for droplets to use. If this is ommited, it will send you one-time-password on the email. Can be obtained from security tab of accounts page. see also | | prefix | pf | A prefix to quickly identify proxy-forge resources. | | slave-count | 2 | Number of instances on which squid proxy will be installed. | | lb-count | 1 | Number of master nodes to setup for load balancer, min 1 is required. | | tag-name | proxy-forge-slave | Tag name to group slave droplets. | | droplet-size | { slave = "s-1vcpu-1gb-amd" } | Droplet size to use. | | project | Proxy Forge | Name of the project to associate all the resources. | | squid-credentials | { password = "proxyforge", username = "proxyforge" } | Squid proxy HTTP basic authentication.credentials