adblock2privoxy

Convert adblock config files to privoxy format.

This is a fork of Zubr's adblock2privoxy repo with minor optimizations for regular expressions, large outputs to the CSS debug directory turned off, and upgrades for the latest ghc compiler and modules.

The webserver configation file nginx.conf is used for both element blocking and as a blackhole (including within HTTPS-secured pages) for the Proxy Autoconfiguration (PAC) file in the parent repo easylist-pac-privoxy.

Synopsis

adblock2privoxy [OPTION...] [URL...]

The files in the example privoxy and css directories are created with the command:

adblock2privoxy -p ./privoxy -w ./css -d 127.0.0.1:8119 ./easylist/*.txt

After installing working binaries (below), an example production run with regular updates looks like:

adblock2privoxy -p /usr/local/etc/adblock2privoxy/privoxy -w /usr/local/etc/adblock2privoxy/css -d 10.0.1.3:8119 \
  https://easylist.to/easylist/easyprivacy.txt  \
  https://easylist.to/easylist/easylist.txt  \
  https://easylist.to/easylist/fanboy-annoyance.txt  \
  https://easylist.to/easylist/fanboy-social.txt  \
  https://easylist-downloads.adblockplus.org/antiadblockfilters.txt  \
  https://easylist-downloads.adblockplus.org/malwaredomains_full.txt  \
  https://raw.githubusercontent.com/ryanbr/fanboy-adblock/master/fanboy-antifacebook.txt \
  https://raw.githubusercontent.com/Dawsey21/Lists/master/adblock-list.txt

# then every few days
adblock2privoxy -t /usr/local/etc/adblock2privoxy/privoxy/ab2p.task
# restart privoxy, e.g. sudo port unload privoxy ; sudo port load privoxy

macOS Installation

The macOS repo is part of MacPorts. The installation details below this section are deprecated.

To install:

sudo port install adblock2privoxy
sudo port load adblock2privoxy

Unix Installation

Quick Build/Install Example

This will build a local /usr/local/bin/adblock2privoxy executable from source and templates saved in /usr/local/etc/adblock2privoxy/adblock2privoxy.

curl -sSL https://get.haskellstack.org/ | sh
sudo mkdir -p /usr/local/etc/adblock2privoxy
sudo rsync -a ./adblock2privoxy* /usr/local/etc/adblock2privoxy
PATH=/usr/bin:$PATH sudo -E bash -c 'export STACK_ROOT=/usr/local/etc/.stack ; cd /usr/local/etc/adblock2privoxy/adblock2privoxy && stack setup --allow-different-user && stack install --local-bin-path /usr/local/bin --allow-different-user'

macOS launchd.plist daemons

macOS launchd.plist daemons to start nginx and update the privoxy database on a schedule. See macOS-Fortress for a working example.

sudo install -m 644 ./com.github.essandess.adblock2privoxy.plist /Library/LaunchDaemons
sudo install -m 644 ./com.github.essandess.adblock2privoxy.nginx.plist /Library/LaunchDaemons
sudo launchctl load -w /Library/LaunchDaemons/com.github.essandess.adblock2privoxy.plist
sudo launchctl load -w /Library/LaunchDaemons/com.github.essandess.adblock2privoxy.nginx.plist
sudo launchctl start com.github.essandess.adblock2privoxy

Objectives

AdBlock Plus browser plugin has great block lists provided by big community, but it is client software and cannot work on a server as a proxy.

Privoxy proxy has good potential to block ads at server side, but it experiences acute shortage of updated block lists.

This software converts adblock lists to privoxy config files format.

Almost all adblock features are supported including

• block/unblock requests (on privoxy)
  • all syntax features are supported except for regex templates matching host name
• hide/unhide page elements (via CSS)
  • all syntax features are supported
• all block request options except for outdated ones:
  • Supported: script, image, stylesheet, object, xmlhttprequest, object-subrequest, subdocument,document, elemhide, other, popup, third-party, domain=..., match-case, donottrack
  • Unsupported: collapse, background, xbl, ping and dtd
  • Tested with privoxy version 3.0.21. Element hiding feature requires a webserver to serve CSS files. See Nginx and Apache config examples provided.

Description

Adblock files specified by [URL]... are converted to privoxy config files and auxiliarly elemHide CSS files. Local file names and http(s) addresses are accepted as URLs.

If no source URLs are specified, task file is used to determine sources: previously processed sources are processed again if any of them is expired. Nothing is done if all sources in the task file are up to date.

Options

-v, --version
  Show version number
-p PATH, --privoxyDir=PATH
  Privoxy config output path
-w PATH, --webDir=PATH
  Css files output path
-d DOMAIN, --domainCSS=DOMAIN
  Domain of CSS web server (required for Element Hide functionality)
-u, --useHTTP
  Use HTTP for CSS web server; the default is HTTPS to avoid mixed content
-g INT, --debugLevel=INT
  Debug Level. 0: Off; 1: top directory CSS; 2: full directory.
-t PATH, --taskFile=PATH
  Path to task file containing urls to process and options.
-f, --forced
  Run even if no sources are expired

If taskFile is not specified explicilty, [privoxyDir]/ab2p.task is used.

If task file exists and privoxyDir, webDir or domainCSS is not specified, corresponding value is taken from task file.

If webDir is not specified and cannot be taken from task file, privoxyDir value is used for webDir.

If domainCSS is not specified and cannot be taken from task file, Element Hide functionality become disabled. No webserver is needed in this case.

domainCSS can contain just IP address if CSS web server has no associated domain. Use localhost or 127.0.0.1 if you run your browser on the same machine with webserver.

Usage

Example of first run:

adblock2privoxy -p /etc/privoxy -w /var/www/privoxy -d www.example.com -t my_ab2b.task https://easylist-downloads.adblockplus.org/easylist.txt https://easylist-downloads.adblockplus.org/advblock.txt my_custom.txt

Example of subsequent runs:

adblock2privoxy -t my_ab2b.task

The app generates following files

• privoxyDir:
  • ab2p.system.action
  • ab2p.action
  • ab2p.system.filter
  • ab2p.filter
• webDir:
  • ab2p.common.css
  • ab2p.css
  • [lot of directories for all levels of domain names]
• taskFile:
  • special file containing execution details. It can be reused to update privoxy config from same sources with same options. Its name is ab2p.task by default.

How to apply results

1. Install privoxy. Optionally setup it as transparent proxy. See privoxy installation manual for details.

2. Change privoxy config file located in

• /etc/privoxy/config for linux
• /opt/local/etc/privoxy/config for macOS with MacPorts
• /usr/local/etc/privoxy/config for macOS with Homebrew
• C:\Program Files\Privoxy\config.txt for windows

Add following lines:

actionsfile ab2p.system.action
actionsfile ab2p.action
filterfile ab2p.system.filter
filterfile ab2p.filter

3. In order to make Element hiding work you also need a webserver to serve CSS files. You can choose nginx, apache or any other webserver. See nginx installation manual, apache on linux installation manual or apache on windows intallation manual for details.

4. Change webserver config. In examples below

• replace www.example.com with your domain or IP address (equal to --domainCSS adblock2privoxy parameter)
• replace /var/www/privoxy with your CSS files location (equal to --webDir adblock2privoxy parameter)
• remember, these examples are simplified to use by unexperienced people. If you're familiar with webservers administration, you'll find better ways to apply these configs.

Nginx config: add following lines into http section of nginx.conf file

• for linux /etc/nginx/nginx.conf
• for macOS with MacPorts /opt/local/etc/nginx/nginx.conf
• for macOS with Homebrew /usr/local/etc/nginx/nginx.conf
• for windows [nginx location]\conf\nginx.conf

server {
      listen www.example.com:443;
      #ab2p css domain name (optional, should be equal to --domainCSS parameter)
      server_name www.example.com;

      ssl on;
      ssl_certificate      certs/adblock2privoxy-nginx.chain.pem;
      ssl_certificate_key  certs/adblock2privoxy-nginx.key.pem.decrypted;
      # use modern crypto
      # https://ssl-config.mozilla.org
      ssl_protocols TLSv1.3;
      ssl_prefer_server_ciphers on;
      ssl_dhparam certs/dhparam.pem;
      ssl_ciphers TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:EECDH+AESGCM:EDH+AESGCM;
      ssl_ecdh_curve secp384r1;
      ssl_session_timeout  180m;
      ssl_session_cache    shared:SSL:20m;
      ssl_session_tickets  off;
      add_header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload";

      # comply with Content Security policy
      add_header Content-Type "text/css";
      add_header X-Content-Type-Options nosniff;

      #root = --webDir parameter value
      root /var/www/privoxy;

      location ~ ^/[^/.]+\..+/ab2p.css$ {
          # first reverse domain names order
          rewrite ^/([^/]*?)\.([^/.]+)(?:\.([^/.]+))?(?:\.([^/.]+))?(?:\.([^/.]+))?(?:\.([^/.]+))?(?:\.([^/.]+))?(?:\.([^/.]+))?(?:\.([^/.]+))?/ab2p.css$ /$9/$8/$7/$6/$5/$4/$3/$2/$1/ab2p.css last;
      }

      loca