The major changes includes:

• Add multi interface support
• Avoid blocking other ruby threads while in a easy perform
• Add support for HTTP DELETE Requests
• Add basic support for HTTP PUT Requests (not from a file)
• Ruby 1.9.1 Support
• Upgraded packaging and dependencies for build and package
• Upgraded test harness to run stressing the libcurl http code base
• Incorporated 8 months of testing and development through github contributors

The addition of Multi interface means it’s now possible to issue thousands of request simultaneously with a single ruby thread.

To highlight the added support of the Multi interface here is an example that makes 5 requests to some major sites in parallel.

require 'rubygems'
require 'curb'
responses = {}
requests = ["http://www.google.com/",
            "http://www.yahoo.com/",
            "http://www.ebay.com/",
            "http://www.amazon.com/",
            "http://www.msn.com/",
            "http://www.google.co.uk/",
            "http://www.ruby-lang.org/"]
m = Curl::Multi.new
# add a few easy handles
requests.each do |url|
  responses[url] = ""
  c = Curl::Easy.new(url) do|curl|
    curl.follow_location = true
    curl.on_body{|data| responses[url] << data; data.size }
    curl.on_complete{|cl| puts "done #{curl.last_effective_url}" }
  end
  m.add(c)
end
idle_count = 0
m.perform do
  idle_count += 1
end

requests.each do|url|
  puts "#{url} responded with #{responses[url].size} bytes"
end
puts "idled: #{idle_count} times"

To install:

gem install curb

Enjoy!

Update:

Removed a GC.start inside the perform loop, was only there to verify objects were being correctly collected e.g. testing purposes.

The few tests we did show a speed improvement for a single request, but multiple concurrent requests or throughput has stayed about the same, because the Curl::Easy.perform call is blocking all other ruby threads from running at the same time. Meaning, we mightiest well just leave our calls in the rails controller and move on right?

Well, by now you probably realized I’m not that kind of programmer… Giving up like this isn’t why I like to write code… Doing a little more digging I realized the answer is rb_thread_select.

Using rb_thread_select in place of select and making calls to libcurl’s multi interface instead of curl_easy_perform, I’ve made it possible to run multiple Curl::Easy.perform calls concurrently. You’ll still probably see much higher throughput using the Curl::Multi interface, but this should help those using the more direct Curl::Easy interface. I also updated the Curl::Multi interface to use ruby’s rb_thread_select, meaning it can be used within multiple ruby threads as well.

Remember this does not mean I made Curl::Easy or Curl::Multi thread safe, but it means multiple instances of those objects can be used within multiple threads, running in parallel.

You can get my curb changes from github.

gc = Curl::Easy.new("http://www.google.com/")
gc.on_success{|curl| puts curl.body_str }
yc = Curl::Easy.new("http://www.yahoo.com/")
yc.on_success{|curl| puts curl.body_str }

mc = Curl::Multi.new
mc.add(gc)
mc.add(yc)

mc.perform

What’s nice about this, is imagine you have some process running and want to add requests to it. When you add the request you also want to register a callback to handle the success or failure cases. Before this patch you’d have to check the content length and listen to the on_body handler. The problem with that, is the content length header is not always correct. These callbacks are guaranteed, because they happen during the cleanup process for each handle.

As an aside they’ll work using the Easy handle diretly.

Here’s the patch, enjoy!

Check out curb trunk

svn co svn://rubyforge.org/var/svn/curb/TRUNK/curb

Grab my latest patch

wget http://taf2-patches.s3.amazonaws.com/curb-multi.patch

Apply the patch

patch -p0 < curb-multi-3.patch

Rebuild and test curb

rake test

Package and install

rake package
sudo gem install pkg/curb-0.2.0.gem

In the patch I updated the curb version number. This may of course change, based on whether the patch is accepted or not. In the meantime, would be great to get a few people to try the patch and provide feedback and or bugs. Thanks!

Update (example usage):

  responses = {}
  requests = ["http://www.google.co.uk/", "http://www.ruby-lang.org/"]
  m = Curl::Multi.new
  # add a few easy handles
  requests.each do |url|
    responses[url] = ""
    c = Curl::Easy.new(url) do|curl|
      curl.follow_location = true
      curl.on_body{|data| responses[url] << data; data.size }
    end
    m.add(c)
  end

  m.perform do
    puts "idling... can do some work here, including add new requests"
  end

  requests.each do|url|
    puts responses[url]
  end

Here's a performance comparison using ruby's native threads vs curb multi interface to make up to 400 concurrent requests. You can see a huge pay off in performance at the 50 concurrent requests.

Curb as a pure C extension runs on the main ruby thread, meaning no ruby threads can be schedule while it’s blocking. It uses the easy curl interface meaning it’s all blocking. This makes it very easy to use and for single requests it’s great. For multiple requests, it’s as though all the requests are made one after the other in serial.

Net::HTTP + Thread is really good, it runs fast and really doesn’t start to show performance issues until it starts to exceed 20 concurrent requests.

Evdispatch is good for when you start to pass the 50 concurrent requests. It suffers from some stability issues at the moment, as time permits I’ll work on fixing them.

Net::HTTp vs Evdispatch vs Curb for 50 concurrent requests.

Net::HTTP vs Evdispatch for 50 concurrent requests

Net::HTTP vs Evdispatch for 400 concurrent requests

The x axis is number of requests and the y axis is time per request. My benchmark code is here.. For most uses Net::HTTP should be plenty fast when making concurrent requests. I would definitely keep an eye on the rev.rubyforge.org project.