Updated curb multi interface patch
July 3rd, 2008
I’ve updated my patch for curb multi interface and tested against valgrind for memory leaks. So far the patch looks stable. I’m still waiting to hear from the curb author. In the meantime here’s how to apply the patch if you’re interested.
Check out curb trunk
svn co svn://rubyforge.org/var/svn/curb/TRUNK/curb
Grab my latest patch
wget http://taf2-patches.s3.amazonaws.com/curb-multi.patch
Apply the patch
patch -p0 < curb-multi-3.patch
Rebuild and test curb
rake test
Package and install
rake package
sudo gem install pkg/curb-0.2.0.gem
In the patch I updated the curb version number. This may of course change, based on whether the patch is accepted or not. In the meantime, would be great to get a few people to try the patch and provide feedback and or bugs. Thanks!
Update (example usage):
responses = {}
requests = ["http://www.google.co.uk/", "http://www.ruby-lang.org/"]
m = Curl::Multi.new
# add a few easy handles
requests.each do |url|
responses[url] = ""
c = Curl::Easy.new(url) do|curl|
curl.follow_location = true
curl.on_body{|data| responses[url] << data; data.size }
end
m.add(c)
end
m.perform do
puts "idling... can do some work here, including add new requests"
end
requests.each do|url|
puts responses[url]
end
Here's a performance comparison using ruby's native threads vs curb multi interface to make up to 400 concurrent requests. You can see a huge pay off in performance at the 50 concurrent requests.
Todd, great patch. Out of curiosity, did you benchmark your new code against the curl-multi gem?
I have not, I’ll work on a benchmark to compare the two.
will this also handle uploading files with http_post… i’ve noticed that when uploading a big file, our app chokes — and waits til the file is done uploading before resuming the processes after the post. just wondering if this will allow for a little more asynchronicity in that operation. thanks!
Oddly Zen,
Yes, the libcurl multi interface processes a regular easy handle. So, all the things you can do with a Curl::Easy handle you can do through a Curl::Multi handle.
-Todd
Can Curl::Easy create POST requests?
Ja,
Yes.
c = Curl::Easy.new("http://my.rails.box/files/upload") c.multipart_form_post = true c.http_post(Curl::PostField.file('myfile.rb'))Also, see: http://curb.rubyforge.org/