Typhoeus

Like a modern code version of the mythical beast with 100 serpent heads, Typhoeus runs HTTP requests in parallel while cleanly encapsulating handling logic.

Example

A single request:

Typhoeus.get("www.example.com", followlocation: true)

Parallel requests:

hydra = Typhoeus::Hydra.new
10.times.map{ hydra.queue(Typhoeus::Request.new("www.example.com", followlocation: true)) }
hydra.run

Installation

Run:

bundle add typhoeus

Or install it yourself as:

gem install typhoeus

Project Tracking

• API Documentation (GitHub master)

Usage

Introduction

The primary interface for Typhoeus is comprised of three classes: Request, Response, and Hydra. Request represents an HTTP request object, response represents an HTTP response, and Hydra manages making parallel HTTP connections.

request = Typhoeus::Request.new(
  "www.example.com",
  method: :post,
  body: "this is a request body",
  params: { field1: "a field" },
  headers: { Accept: "text/html" }
)

We can see from this that the first argument is the url. The second is a set of options. The options are all optional. The default for :method is :get.

When you want to send URL parameters, you can use :params hash to do so. Please note that in case of you should send a request via x-www-form-urlencoded parameters, you need to use :body hash instead. params are for URL parameters and :body is for the request body.

Sending requests through the proxy

Add a proxy url to the list of options:

options = {proxy: 'http://myproxy.org'}
req = Typhoeus::Request.new(url, options)

If your proxy requires authentication, add it with proxyuserpwd option key:

options = {proxy: 'http://proxyurl.com', proxyuserpwd: 'user:password'}
req = Typhoeus::Request.new(url, options)

Note that proxyuserpwd is a colon-separated username and password, in the vein of basic auth userpwd option.

You can run the query either on its own or through the hydra:

request.run
#=> <Typhoeus::Response ... >

hydra = Typhoeus::Hydra.hydra
hydra.queue(request)
hydra.run

The response object will be set after the request is run.

response = request.response
response.code
response.total_time
response.headers
response.body

Making Quick Requests

Typhoeus has some convenience methods for performing single HTTP requests. The arguments are the same as those you pass into the request constructor.

Typhoeus.get("www.example.com")
Typhoeus.head("www.example.com")
Typhoeus.put("www.example.com/posts/1", body: "whoo, a body")
Typhoeus.patch("www.example.com/posts/1", body: "a new body")
Typhoeus.post("www.example.com/posts", body: { title: "test post", content: "this is my test"})
Typhoeus.delete("www.example.com/posts/1")
Typhoeus.options("www.example.com")

Sending params in the body with PUT

When using POST the content-type is set automatically to 'application/x-www-form-urlencoded'. That's not the case for any other method like PUT, PATCH, HEAD and so on, irrespective of whether you are using body or not. To get the same result as POST, i.e. a hash in the body coming through as params in the receiver, you need to set the content-type as shown below:

Typhoeus.put("www.example.com/posts/1",
        headers: {'Content-Type'=> "application/x-www-form-urlencoded"},
        body: {title:"test post updated title", content: "this is my updated content"}
    )

Handling HTTP errors

You can query the response object to figure out if you had a successful request or not. Here’s some example code that you might use to handle errors. The callbacks are executed right after the request is finished, make sure to define them before running the request.

request = Typhoeus::Request.new("www.example.com", followlocation: true)

request.on_complete do |response|
  if response.success?
    # hell yeah
  elsif response.timed_out?
    # aw hell no
    log("got a time out")
  elsif response.code == 0
    # Could not get an http response, something's wrong.
    log(response.return_message)
  else
    # Received a non-successful http response.
    log("HTTP request failed: " + response.code.to_s)
  end
end

request.run

This also works with serial (blocking) requests in the same fashion. Both serial and parallel requests return a Response object.

Handling file uploads

A File object can be passed as a param for a POST request to handle uploading files to the server. Typhoeus will upload the file as the original file name and use Mime::Types to set the content type.

Typhoeus.post(
  "http://localhost:3000/posts",
  body: {
    title: "test post",
    content: "this is my test",
    file: File.open("thesis.txt","r")
  }
)

Streaming the response body

Typhoeus can stream responses. When you're expecting a large response, set the on_body callback on a request. Typhoeus will yield to the callback with chunks of the response, as they're read. When you set an on_body callback, Typhoeus will not store the complete response.

downloaded_file = File.open 'huge.iso', 'wb'
request = Typhoeus::Request.new("www.example.com/huge.iso")
request.on_headers do |response|
  if response.code != 200
    raise "Request failed"
  end
end
request.on_body do |chunk|
  downloaded_file.write(chunk)
end
request.on_complete do |response|
  downloaded_file.close
  # Note that response.body is ""
end
request.run

If you need to interrupt the stream halfway, you can return the :abort symbol from the on_body block, example:

request.on_body do |chunk|
  buffer << chunk
  :abort if buffer.size > 1024 * 1024
end

This will properly stop the stream internally and avoid any memory leak which may happen if you interrupt with something like a return, throw or raise.

Making Parallel Requests

Generally, you should be running requests through hydra. Here is how that looks:

hydra = Typhoeus::Hydra.hydra

first_request = Typhoeus::Request.new("http://example.com/posts/1")
first_request.on_complete do |response|
  third_url = response.body
  third_request = Typhoeus::Request.new(third_url)
  hydra.queue third_request
end
second_request = Typhoeus::Request.new("http://example.com/posts/2")

hydra.queue first_request
hydra.queue second_request
hydra.run # this is a blocking call that returns once all requests are complete

The execution of that code goes something like this. The first and second requests are built and queued. When hydra is run the first and second requests run in parallel. When the first request completes, the third request is then built and queued, in this example based on the result of the first request. The moment it is queued Hydra starts executing it. Meanwhile the second request would continue to run (or it could have completed before the first). Once the third request is done, hydra.run returns.

How to get an array of response bodies back after executing a queue:

hydra = Typhoeus::Hydra.new
requests = 10.times.map {
  request = Typhoeus::Request.new("www.example.com", followlocation: true)
  hydra.queue(request)
  request
}
hydra.run

responses = requests.map { |request|
  request.response.body
}

hydra.run is a blocking request. You can also use the on_complete callback to handle each request as it completes:

hydra = Typhoeus::Hydra.new
10.times do
  request = Typhoeus::Request.new("www.example.com", followlocation: true)
  request.on_complete do |response|
    #do_something_with response
  end
  hydra.queue(request)
end
hydra.run

Making Parallel Requests with Faraday + Typhoeus

require 'faraday'

conn = Faraday.new(:url => 'http://httppage.com') do |builder|
  builder.request  :url_encoded
  builder.response :logger
  builder.adapter  :typhoeus
end

conn.in_parallel do
  response1 = conn.get('/first')
  response2 = conn.get('/second')

  # these will return nil here since the
  # requests have not been completed
  response1.body
  response2.body
end

# after it has been completed the response information is fully available
# response1.status, etc
response1.body
response2.body

Specifying Max Concurrency

Hydra will also handle how many requests you can make in parallel. Things will get flakey if you try to make too many requests at the same time. The built in limit is 200. When more requests than that are queued up, hydra will save them for later and start the requests as others are finished. You can raise or lower the concurrency limit through the Hydra constructor.

Typhoeus::Hydra.new(max_concurrency: 20)

Memoization

Hydra memoizes requests within a single run call. You have to enable memoization. This will result in a single request being issued. However, the on_complete handlers of both will be called.

Typhoeus::Config.memoize = true

hydra = Typhoeus::Hydra.new(max_concurrency: 1)
2.times do
  hydra.queue Typhoeus::Request.new("www.example.com")
end
hydra.run

This will result in two requests.

Typhoeus::Config.memoize = false

hydra = Typhoeus::Hydra.new(max_concurrency: 1)
2.times do
  hydra.queue Typhoeus::Request.new("www.example.com")
end
hydra.run

Caching

Typhoeus includes built in support for caching. In the following example, if there is a cache hit, the cached object is passed to the on_complete handler of the request object.

class Cache
  def initialize
    @memory = {}
  end

  def get(re