ProxyPool 代理IP池

______                        ______             _
| ___ \_                      | ___ \           | |
| |_/ / \__ __   __  _ __   _ | |_/ /___   ___  | |
|  __/|  _// _ \ \ \/ /| | | ||  __// _ \ / _ \ | |
| |   | | | (_) | >  < \ |_| || |  | (_) | (_) || |___
\_|   |_|  \___/ /_/\_\ \__  |\_|   \___/ \___/ \_____\
                       __ / /
                      /___ /

ProxyPool

代理IP池项目是从https://github.com/jhao104/proxy_pool.git clone而来，对 ProxyPool 代理的二次魔改，除了原有的HTTP代理外，新增了对SOCKS4、SOCKS5、HTTPS代理协议支持，增加了完善了proxy fetch来源，改善了部分功能，如/count请求、server和schedule模式形成守护进程等，优化修改了部分bug，快速实现大量活跃可用代理的持续监测收集工作。         代理IP池项目,主要功能为定时采集网上发布或者手工收录的免费代理验证入库，定时验证入库的代理保证代理的可用性，提供API和CLI两种使用方式，同时你也可以扩展代理源以增加代理池IP的质量和数量。

PS:本项目目的：人人可建立属于自己的高质量免费代理池。

文档: document
支持版本:

项目效果

运行项目

下载代码:

git clone

git clone git@github.com:RanSecWlx/proxyPool.git

releases

https://github.com/RanSecWlx/proxyPool/releases 下载对应zip文件

安装依赖:

pip install -r requirements.txt

更新配置:

# setting.py 为项目配置文件

# 配置API服务

HOST = "0.0.0.0"               # IP
PORT = 5000                    # 监听端口


# 配置数据库

DB_CONN = 'redis://:pwd@127.0.0.1:8888/0'


# 配置 ProxyFetcher

PROXY_FETCHER = [
    "freeProxy01",      # 这里是启用的代理抓取方法名，所有fetch方法位于fetcher/proxyFetcher.py
    "freeProxy02",
    # ....
]

# ############# proxy validator #################
# 代理验证目标网站
HTTP_URL = "http://ifconfig.me/ip"
HTTPS_URL = "https://ifconfig.me/ip"
# 代理验证时超时时间
VERIFY_TIMEOUT = 5

# 近PROXY_CHECK_COUNT次校验中允许的最大失败次数,超过则剔除代理
MAX_FAIL_COUNT = 0

# 近PROXY_CHECK_COUNT次校验中允许的最大失败率,超过则剔除代理
MAX_FAIL_RATE = 0.3

# proxyCheck时代理数量少于POOL_SIZE_MIN触发抓取
POOL_SIZE_MIN = 2000

# getCount时，check_count_xxx_proxies 过滤条件下限，筛选出来质量比较高的代理
MIN_AVAIL_LIMIT = 20

# 解决代理源需fanqiang访问问题
API_PROXY_CONFIG = "http://127.0.0.1:1080"

# 每个proxy采集进程启动的线程数量
RAW_THREADS_NUM = 50

# 每个proxy检查进程启动的线程数量
USE_THREADS_NUM = 10

# proxy采集任务每间隔多少分钟执行一次
RAW_INTERVAL_MIN = 4

# proxy检查任务每间隔多少分钟执行一次
USE_INTERVAL_MIN = 2

# 控制 执行器（如线程池或进程池）中 最大并发执行任务的数量。
THREADPOOL_WORKS_NUM = 60

# 一个进程池（ProcessPoolExecutor），最多可以同时运行多少个进程
PROCESSPOOL_WORKS_NUM = 5

# 控制 同一个任务 在调度器中的最大实例数量
JOB_INSTANCES_NUM = 30

免费代理源

1、目前实现的采集免费代理网站有(排名不分先后, 下面仅是对其发布的免费代理情况, 付费代理测评可以参考这里):

https://github.com/proxifly/free-proxy-list
https://github.com/TheSpeed/SOCKS-List
https://github.com/monosans/proxy-list
https://proxy5.net
https://api.openproxylist.xyz/
https://fineproxy.org/
https://api.proxyscrape.com/
https://www.freeproxy.world/
https://proxyelite.info
https://geoxy.io/
https://proxylist.geonode.com

2、此外，还可以从网络空间测绘引擎上获取免费代理，比如FOFA引擎： 获取SOCKS4代理：protocol="socks4" && "Results:request granted"; 获取SOCKS5代理：protocol=="socks5" && "Version:5 Method:No Authentication(0x00)" && after="2022-02-01"; 获取HTTP 代理：server=="Mikrotik HttpProxy" && status_code=="401";

3、在tmp目录下执行bash run.sh 可以在线获取HTTP、SOCKS4、SOCKS5代理，并自动更新到proxyList目录下的配置文件。

4、大家可以在tmp目录下run.sh继续完善添加的代理源： "HTTP": [ "https://raw.githubusercontent.com/B4RC0DE-TM/proxy-list/main/HTTP.txt", "https://raw.githubusercontent.com/mmpx12/proxy-list/master/http.txt", "https://api.proxyscrape.com/v2/?request=getproxies&protocol=http&timeout=10000&country=all&ssl=all&anonymity=all", "https://api.proxyscrape.com/?request=getproxies&proxytype=https&timeout=10000&country=all&ssl=all&anonymity=all", "https://api.openproxylist.xyz/http.txt", "https://multiproxy.org/txt_all/proxy.txt", "https://raw.githubusercontent.com/monosans/proxy-list/main/proxies/http.txt", "https://raw.githubusercontent.com/monosans/proxy-list/main/proxies_anonymous/http.txt", "https://raw.githubusercontent.com/shiftytr/proxy-list/master/proxy.txt", "https://raw.githubusercontent.com/TheSpeedX/PROXY-List/master/http.txt", "https://raw.githubusercontent.com/ShiftyTR/Proxy-List/master/http.txt", "https://raw.githubusercontent.com/jetkai/proxy-list/main/online-proxies/txt/proxies-http.txt", "https://raw.githubusercontent.com/clarketm/proxy-list/master/proxy-list-raw.txt", "https://raw.githubusercontent.com/sunny9577/proxy-scraper/master/proxies.txt", "https://raw.githubusercontent.com/roosterkid/openproxylist/main/HTTPS_RAW.txt", "https://raw.githubusercontent.com/opsxcq/proxy-list/master/list.txt", "https://raw.githubusercontent.com/proxy4parsing/proxy-list/main/http.txt", "https://rootjazz.com/proxies/proxies.txt", "https://spys.me/proxy.txt", "https://raw.githubusercontent.com/rdavydov/proxy-list/main/proxies/http.txt", "https://raw.githubusercontent.com/rdavydov/proxy-list/main/proxies_anonymous/http.txt", "https://raw.githubusercontent.com/zevtyardt/proxy-list/main/http.txt", "https://sunny9577.github.io/proxy-scraper/proxies.txt", "https://raw.githubusercontent.com/MuRongPIG/Proxy-Master/main/http.txt", "https://raw.githubusercontent.com/Zaeem20/FREE_PROXIES_LIST/master/http.txt", "https://raw.githubusercontent.com/prxchk/proxy-list/main/http.txt", "https://raw.githubusercontent.com/ALIILAPRO/Proxy/main/http.txt", "https://raw.githubusercontent.com/zloi-user/hideip.me/main/http.txt", "https://raw.githubusercontent.com/saisuiu/Lionkings-Http-Proxys-Proxies/main/cnfree.txt", "https://raw.githubusercontent.com/Anonym0usWork1221/Free-Proxies/main/proxy_files/http_proxies.txt", "https://raw.githubusercontent.com/officialputuid/KangProxy/KangProxy/http/http.txt", "https://raw.githubusercontent.com/ErcinDedeoglu/proxies/main/proxies/http.txt", "https://raw.githubusercontent.com/elliottophellia/yakumo/master/results/http/global/http_checked.txt", "https://raw.githubusercontent.com/ProxyScraper/ProxyScraper/refs/heads/main/http.txt", "https://raw.githubusercontent.com/proxifly/free-proxy-list/refs/heads/main/proxies/protocols/http/data.txt" ], "SOCKS4": [ "https://api.proxyscrape.com/v2/?request=displayproxies&protocol=socks4", "https://api.proxyscrape.com/?request=displayproxies&proxytype=socks4&country=all", "https://api.openproxylist.xyz/socks4.txt", "https://raw.githubusercontent.com/monosans/proxy-list/main/proxies/socks4.txt", "https://raw.githubusercontent.com/monosans/proxy-list/main/proxies_anonymous/socks4.txt", "https://raw.githubusercontent.com/jetkai/proxy-list/main/online-proxies/txt/proxies-socks4.txt", "https://raw.githubusercontent.com/ShiftyTR/Proxy-List/master/socks4.txt", "https://raw.githubusercontent.com/TheSpeedX/PROXY-List/master/socks4.txt", "https://raw.githubusercontent.com/roosterkid/openproxylist/main/SOCKS4_RAW.txt", "https://raw.githubusercontent.com/mmpx12/proxy-list/master/socks4.txt", "https://raw.githubusercontent.com/B4RC0DE-TM/proxy-list/main/SOCKS4.txt", "https://raw.githubusercontent.com/rdavydov/proxy-list/main/proxies/socks4.txt", "https://raw.githubusercontent.com/rdavydov/proxy-list/main/proxies_anonymous/socks4.txt", "https://raw.githubusercontent.com/zevtyardt/proxy-list/main/socks4.txt", "https://raw.githubusercontent.com/MuRongPIG/Proxy-Master/main/socks4.txt", "https://raw.githubusercontent.com/Zaeem20/FREE_PROXIES_LIST/master/socks4.txt", "https://raw.githubusercontent.com/prxchk/proxy-list/main/socks4.txt", "https://raw.githubusercontent.com/ALIILAPRO/Proxy/main/socks4.txt", "https://raw.githubusercontent.com/zloi-user/hideip.me/main/socks4.txt", "https://raw.githubusercontent.com/mmpx12/proxy-list/master/socks4.txt", "https://raw.githubusercontent.com/B4RC0DE-TM/proxy-list/main/SOCKS4.txt", "https://raw.githubusercontent.com/ErcinDedeoglu/proxies/main/proxies/socks4.txt", "https://raw.githubusercontent.com/ProxyScraper/ProxyScraper/refs/heads/main/socks4.txt", "https://raw.githubusercontent.com/proxifly/free-proxy-list/refs/heads/main/proxies/protocols/socks4/data.txt" ], "SOCKS5": [

ProxyPool

Install / Use

README

ProxyPool 代理IP池

ProxyPool

项目效果

运行项目

下载代码:

安装依赖:

更新配置:

免费代理源