I just saw this today: http://www.pauldix.net/2009/05/breath-fire-over-http-in-ruby-with-typhoeus.html Looks like a really nice library. I really like the libcurl easy bindings it provides.

Software HTTP, libcurl, Ruby

Here it is version 0.2.6.

This version fixes a bug when sending an HTTP POST on apple/darwin Mac OS.

curl_easy_setopt( m_handle, CURLOPT_POST, 1 );
// set the buffer size to copy
curl_easy_setopt( m_handle, CURLOPT_POSTFIELDSIZE, value.length() );
curl_easy_setopt( m_handle, CURLOPT_POSTFIELDS, value.c_str() );
// copy the buffer
curl_easy_setopt( m_handle, CURLOPT_COPYPOSTFIELDS, value.c_str() );

I had to set the CURLOPT_POSTFIELDS before calling CURLOPT_COPYPOSTFIELDS.

In my next release I hope to have support for setting arbitrary HTTP headers. I’ll also be working on a streaming response interface. In my C++ library I already have a working example that lets the response from the event loop be written directly to a file descriptor. To expose this in Ruby I would make it so any IO object can be passed into the request method via a :stream => io parameter. This only posses one interesting issue that the Ruby StringIO could be valid to pass, but my implementation would not be able to pull a file descriptor from the StringIO object. Perhaps, when it’s an IO object without a file descriptor my implementation could create a pipe fd on behave of the caller?

Software C++, HTTP, libcurl, libev

If you have ever had a web site that required a large number of service requests from a single user request then you might find evdispatch useful. evdispatch makes it very easy to send off multiple http requests and check back later for their status. Lets say your site has a number of different feeds for example that it’s aggregating together. You’ll definitely want to cache these actions, but for the times when the pages are uncached you’ll surely want to make sure they get built quickly. The service responses very fast so it’s not the bottleneck. In your case the bottleneck is the fact that from ruby it’s very difficult to efficiently run multiple concurrent requests. evdispatch might be exactly what you need. The way it works is you send off all your requests ahead of time. Then after doing some processing you stop your ruby process as normal when you need to get the request, unless it’s already responded in which case it returns immediately.

This will probably be easier to follow using an example:
Creating a feed aggregator, using google news feeds. (Note: google will rate limit you, so if you plan to do this make sure you cache).

First install the evdispatch gem:

sudo gem install evdispatch

Set up your rails app initializer:

require 'evdispatch'

$dispatcher = Evdispatch::Loop.new

# startup a dispatcher for this rails app
$dispatcher.start

Add hpricot to your config/environment.rb

require 'hpricot'

Create a new action on your controller:

class DashController < ApplicationController

  def index
    @timer = Time.now
    @top_news_id = $dispatcher.request_http("http://news.google.com/news?ned=us&topic=h&output=rss")
    @world_id = $dispatcher.request_http("http://news.google.com/news?ned=us&topic=w&output=rss")
    @us_id = $dispatcher.request_http("http://news.google.com/news?ned=us&topic=n&output=rss")
    @health_id = $dispatcher.request_http("http://news.google.com/news?ned=us&topic=m&output=rss")
    @sports_id = $dispatcher.request_http("http://news.google.com/news?ned=us&topic=s&output=rss")
  end
end

Add a helper method to your dash_helper.rb:

module DashHelper
  def display_feed(id)
    res = $dispatcher.response(id)
    doc = Hpricot.XML(res[:body])
    items = []
    titles = []
    (doc/'title').each do|t|
      titles << t.inner_html
      break
    end
    (doc/'item').each do|item|
      items << "<a href="#{(item/">#{(item/'title').inner_html}</a>"
    end
    "
#{titles.first}

#{items}response time: #{res[:response_time]} seconds"
  end
end

Create the view:

All the latests

<%= display_feed(@top_news_id) %>
<%= display_feed(@world_id) %>
<%= display_feed(@us_id) %>
<%= display_feed(@health_id) %>
<%= display_feed(@sports_id) %>


Page render time <%= Time.now - @timer %> seconds

Software C++, HTTP, libev, pthreads, Ruby