cfhttr

Workaround Cloudflare Anti-DDoS Protection

Description

Some sites use Cloudflare’s anti-DDoS protection to, well, avoid being DDoS’d.

However, many of these sites also host files that they intended for automated download workflows and often forget to exclude those from the heavy-handed anti-DDoS, javascript + human-required settings.

Cloudflare employs a few different anti-DDoS protection schemes. One involves a 503 HTTP response followed by the submission of a “challenge” that was solved via a bit of javascript on the page. That’s what this pacakge handles. If you encounter other Cloudflare anti-DDoS protection schemes (especially the captcha one), then this won’t work.

NOTE

If this ceases to work it likely means that Cloudflare changed the challenge code. Please file an issue so I can redo the workaround.

What’s Inside The Tin

The following functions are implemented:

cf_GET: Cloudflare bypass version of httr::GET()

Installation

devtools::install_github("hrbrmstr/cfhttr")

Usage

library(cfhttr)

# current verison
packageVersion("cfhttr")

## [1] '0.1.0'

res <- cf_GET("https://www.bitmarket.pl/graphs/BTCPLN/90m.json")

## Waiting 5 seconds...

str(httr::content(res, as="parsed"))

## List of 90
##  $ :List of 6
##   ..$ time : int 1512908160
##   ..$ open : chr "48000.00000000"
##   ..$ high : chr "48100.00000000"
##   ..$ low  : chr "48000.00000000"
##   ..$ close: chr "48100.00000000"
##   ..$ vol  : chr "0.00124821"
##  $ :List of 6
##   ..$ time : int 1512908220
##   ..$ open : chr "48100.00000000"
##   ..$ high : chr "48100.00000000"
##   ..$ low  : chr "48100.00000000"
##   ..$ close: chr "48100.00000000"
##   ..$ vol  : chr "0.00000000"
##  $ :List of 6
##   ..$ time : int 1512908280
##   ..$ open : chr "48100.00000000"
##   ..$ high : chr "48100.00000000"
##   ..$ low  : chr "48100.00000000"
##   ..$ close: chr "48100.00000000"
##   ..$ vol  : chr "0.00000000"
##  $ :List of 6
##   ..$ time : int 1512908340
##   ..$ open : chr "48100.00000000"
##   ..$ high : chr "48100.00000000"
##   ..$ low  : chr "48000.00000000"
##   ..$ close: chr "48000.00000000"
##   ..$ vol  : chr "0.01757887"
##  $ :List of 6
##   ..$ time : int 1512908400
##   ..$ open : chr "48000.00000000"
##   ..$ high : chr "48100.00000000"
##   ..$ low  : chr "48000.00000000"
##   ..$ close: chr "48100.00000000"
##   ..$ vol  : chr "0.09891712"
##  $ :List of 6
##   ..$ time : int 1512908460
##   ..$ open : chr "48100.00000000"
##   ..$ high : chr "48100.00000000"
##   ..$ low  : chr "48100.00000000"
##   ..$ close: chr "48100.00000000"
##   ..$ vol  : chr "0.00000000"
##  $ :List of 6
##   ..$ time : int 1512908520
##   ..$ open : chr "48100.00000000"
##   ..$ high : chr "48308.22960000"
##   ..$ low  : chr "48100.00000000"
##   ..$ close: chr "48308.22960000"
##   ..$ vol  : chr "0.01972749"
##  $ :List of 6
##   ..$ time : int 1512908580
##   ..$ open : chr "48308.22960000"
##   ..$ high : chr "48308.22960000"
##   ..$ low  : chr "48000.00000000"
##   ..$ close: chr "48000.00000000"
##   ..$ vol  : chr "0.01555715"
##  $ :List of 6
##   ..$ time : int 1512908640
##   ..$ open : chr "48000.00000000"
##   ..$ high : chr "48149.99960000"
##   ..$ low  : chr "47700.00000000"
##   ..$ close: chr "48149.99960000"
##   ..$ vol  : chr "0.26652673"
##  $ :List of 6
##   ..$ time : int 1512908700
##   ..$ open : chr "48149.99960000"
##   ..$ high : chr "48149.99960000"
##   ..$ low  : chr "48149.98960000"
##   ..$ close: chr "48149.98960000"
##   ..$ vol  : chr "0.02755044"
##  $ :List of 6
##   ..$ time : int 1512908760
##   ..$ open : chr "48149.98960000"
##   ..$ high : chr "48149.98960000"
##   ..$ low  : chr "48149.98960000"
##   ..$ close: chr "48149.98960000"
##   ..$ vol  : chr "0.02202285"
##  $ :List of 6
##   ..$ time : int 1512908820
##   ..$ open : chr "48149.98960000"
##   ..$ high : chr "48149.98960000"
##   ..$ low  : chr "48144.99960000"
##   ..$ close: chr "48144.99960000"
##   ..$ vol  : chr "0.03115266"
##  $ :List of 6
##   ..$ time : int 1512908880
##   ..$ open : chr "48144.99960000"
##   ..$ high : chr "48145.00000000"
##   ..$ low  : chr "48144.99960000"
##   ..$ close: chr "48145.00000000"
##   ..$ vol  : chr "0.05214570"
##  $ :List of 6
##   ..$ time : int 1512908940
##   ..$ open : chr "48145.00000000"
##   ..$ high : chr "48145.00000000"
##   ..$ low  : chr "48145.00000000"
##   ..$ close: chr "48145.00000000"
##   ..$ vol  : chr "0.00000000"
##  $ :List of 6
##   ..$ time : int 1512909000
##   ..$ open : chr "48145.00000000"
##   ..$ high : chr "48145.00000000"
##   ..$ low  : chr "48145.00000000"
##   ..$ close: chr "48145.00000000"
##   ..$ vol  : chr "0.00000000"
##  $ :List of 6
##   ..$ time : int 1512909060
##   ..$ open : chr "48145.00000000"
##   ..$ high : chr "48152.00000000"
##   ..$ low  : chr "48145.00000000"
##   ..$ close: chr "48152.00000000"
##   ..$ vol  : chr "0.14385430"
##  $ :List of 6
##   ..$ time : int 1512909120
##   ..$ open : chr "48152.00000000"
##   ..$ high : chr "48200.00000000"
##   ..$ low  : chr "48152.00000000"
##   ..$ close: chr "48200.00000000"
##   ..$ vol  : chr "0.05931535"
##  $ :List of 6
##   ..$ time : int 1512909180
##   ..$ open : chr "48200.00000000"
##   ..$ high : chr "48287.99960000"
##   ..$ low  : chr "48200.00000000"
##   ..$ close: chr "48287.99960000"
##   ..$ vol  : chr "0.02196575"
##  $ :List of 6
##   ..$ time : int 1512909240
##   ..$ open : chr "48287.99960000"
##   ..$ high : chr "48287.99960000"
##   ..$ low  : chr "48145.00000000"
##   ..$ close: chr "48287.99960000"
##   ..$ vol  : chr "0.84227748"
##  $ :List of 6
##   ..$ time : int 1512909300
##   ..$ open : chr "48287.99960000"
##   ..$ high : chr "48287.99960000"
##   ..$ low  : chr "48144.99960000"
##   ..$ close: chr "48144.99960000"
##   ..$ vol  : chr "0.01528501"
##  $ :List of 6
##   ..$ time : int 1512909360
##   ..$ open : chr "48144.99960000"
##   ..$ high : chr "48144.99960000"
##   ..$ low  : chr "47633.00040000"
##   ..$ close: chr "48144.99960000"
##   ..$ vol  : chr "0.38813920"
##  $ :List of 6
##   ..$ time : int 1512909420
##   ..$ open : chr "48144.99960000"
##   ..$ high : chr "48287.98960000"
##   ..$ low  : chr "48144.99960000"
##   ..$ close: chr "48144.99960000"
##   ..$ vol  : chr "0.03962696"
##  $ :List of 6
##   ..$ time : int 1512909480
##   ..$ open : chr "48144.99960000"
##   ..$ high : chr "48144.99960000"
##   ..$ low  : chr "48144.99960000"
##   ..$ close: chr "48144.99960000"
##   ..$ vol  : chr "0.00000000"
##  $ :List of 6
##   ..$ time : int 1512909540
##   ..$ open : chr "48144.99960000"
##   ..$ high : chr "48308.24000000"
##   ..$ low  : chr "48144.99960000"
##   ..$ close: chr "48308.24000000"
##   ..$ vol  : chr "0.24594488"
##  $ :List of 6
##   ..$ time : int 1512909600
##   ..$ open : chr "48308.24000000"
##   ..$ high : chr "48388.25000000"
##   ..$ low  : chr "48308.24000000"
##   ..$ close: chr "48388.25000000"
##   ..$ vol  : chr "2.45037261"
##  $ :List of 6
##   ..$ time : int 1512909660
##   ..$ open : chr "48388.25000000"
##   ..$ high : chr "48400.00000000"
##   ..$ low  : chr "48388.25000000"
##   ..$ close: chr "48400.00000000"
##   ..$ vol  : chr "0.02294538"
##  $ :List of 6
##   ..$ time : int 1512909720
##   ..$ open : chr "48400.00000000"
##   ..$ high : chr "48400.00000000"
##   ..$ low  : chr "48400.00000000"
##   ..$ close: chr "48400.00000000"
##   ..$ vol  : chr "0.00000000"
##  $ :List of 6
##   ..$ time : int 1512909780
##   ..$ open : chr "48400.00000000"
##   ..$ high : chr "48400.00000000"
##   ..$ low  : chr "48400.00000000"
##   ..$ close: chr "48400.00000000"
##   ..$ vol  : chr "0.00393802"
##  $ :List of 6
##   ..$ time : int 1512909840
##   ..$ open : chr "48400.00000000"
##   ..$ high : chr "48400.00000000"
##   ..$ low  : chr "48309.00000000"
##   ..$ close: chr "48400.00000000"
##   ..$ vol  : chr "0.00980329"
##  $ :List of 6
##   ..$ time : int 1512909900
##   ..$ open : chr "48400.00000000"
##   ..$ high : chr "48400.00000000"
##   ..$ low  : chr "48400.00000000"
##   ..$ close: chr "48400.00000000"
##   ..$ vol  : chr "0.00303943"
##  $ :List of 6
##   ..$ time : int 1512909960
##   ..$ open : chr "48400.00000000"
##   ..$ high : chr "48422.00000000"
##   ..$ low  : chr "48400.00000000"
##   ..$ close: chr "48422.00000000"
##   ..$ vol  : chr "0.04130911"
##  $ :List of 6
##   ..$ time : int 1512910020
##   ..$ open : chr "48422.00000000"
##   ..$ high : chr "48422.00000000"
##   ..$ low  : chr "48422.00000000"
##   ..$ close: chr "48422.00000000"
##   ..$ vol  : chr "0.00600141"
##  $ :List of 6
##   ..$ time : int 1512910080
##   ..$ open : chr "48422.00000000"
##   ..$ high : chr "48422.00000000"
##   ..$ low  : chr "48422.00000000"
##   ..$ close: chr "48422.00000000"
##   ..$ vol  : chr "0.00000000"
##  $ :List of 6
##   ..$ time : int 1512910140
##   ..$ open : chr "48422.00000000"
##   ..$ high : chr "48422.00000000"
##   ..$ low  : c

Cfhttr

Install / Use

README