Idle Hacking

A question came up recently and I had to think twice. It was how do I run a different app server for my merb application? Knowing merb relies on rack, I initially thought to look in the config/rack.rb.  Then I started to grep through merb-core/ and finally came to:

Merb::Config.use do|config|
  config[:adapter] = :mongrel
end

Read more…

Software Ebb, Merb, mongrel, Rack, Rack Adapter, Ruby, Thin

Vendoring gems definitely makes deployment easier for both rails and merb. For merb, the solution is to create a local gem repository within your project.

gem install -i gems/ --no-ri --no-rdoc #{gem_to_install}

This works great, once you update your merb/config/init.rb to include the new gem repository.

Gem.clear_paths
Gem.path.unshift(Merb.root / "gems")

Read more…

Software gems, Merb, Ruby

The advantage of a process over a thread is when the process dies you can get a signal telling you the process died and recover. For any long running process, it’s always very important to be able to recover from unexpected disasters. In working on a server that’s tasked with receiving a lot of large files and spending a fair bit of timing processing those files, I needed to make sure if for some reason the file input I received caused my server to go down - I could quickly recover. Because, I communicate to this server via a pipe (instead of a socket), it’s not as simple to recover using something like monit.

The CLD signal is sent when a child process dies. This is great! All I need to do is trap that signal and start my process backup. My main concern with this is will I get stuck in an infinite loop, forking new processes because some condition has caused the child process to die every time, consuming all the resources on my system.

Here’s my solution so far:

class UploadServer
  def initialize(options = {})
    @upload_read, @upload_write = IO.pipe
    @start_up_threads = (options[:start_up_threads] || 2)
    @max_read = (options[:max_read] || 1024)
    @logger = (options[:logger] || Logger.new(STDOUT))
  end
 
  # starts up the upload server
  def start
    @pid = fork do
      initialize_server
      while( 1 )
        select
      end
    end
    Signal.trap(0) do
      # tell the child process to die
      Process.kill("TERM", @pid)
    end
    Signal.trap("CLD") do
      # something extremely unexpected happened and the child process died
      @logger.error( "It appears the upload background process has died... Attempting a restart..." )
      # make sure we kill of any residue from the child process is cleaned up e.g. avoid defunct process
      Process.wait(@pid)
      # this is all a little risky since someone could have been in the middle of an upload
      # they'll be cut off anyway since the process died...
      # close down open pipe
      @upload_write.close
      # create a new pipe
      @upload_read, @upload_write = IO.pipe
      # start it back up
      start
    end
    @logger.debug( "Upload Process started up on #{@pid}" )
    # close the read end on the main process
    @upload_read.close
    @pid
  end

The Process.wait(@pid) is very important otherwise we’re left with a lot of <defunct> processes. It may also help to throttle the issue of infinite forking. At the very least it means we’ll never get more then 1 child process per server. The only other thing I can imagine adding is some kind of timer to help throttle in the case that the child process dies very quickly…

Software IPC, Merb, Ruby, Upload

For a new application I’m working on we need to upload some large files to our web application. Also because the files are both large and in processing them there are many sql insert and updates that follow the whole process is pretty intensive taking roughly 1 to 2 minutes a file. I’ve been reading about merb for a few months and off and on created small projects with it hoping to one day put it to use. Tonight I was pleased that it really does make file uploading easy and appears fast. Here’s a quick overview of what I did to get it setup.

sudo gem install merb -y

Create the basic merb project

merb -g uploader

Setup the upload controller and view

cd uploader/distmkdir app/views/uploadertouch app/controllers/uploader.rbtouch app/views/uploader/index.rhtml

We’ll have two actions defined in the Uploader controller:

• index, to display the form
• upload, to handle the file upload post

class Uploader < Application
 # http://www.cs.tut.fi/~jkorpela/forms/file.html
  def index
     render
  end
  def upload
    puts params[:file].inspect
    FileUtils.mv params[:file][:tempfile].path, MERB_ROOT +"/uploads/#{params[:file][:filename]}"
    redirect "/uploader"
  end
end

Define the form view

<p>Upload a new file</p>
<form action="/uploader/upload" method="post" enctype="multipart/form-data">
  <fieldset>
  <input type="file" name="file" size="80"/>
  <input type="submit" value="Upload"/>
  </fieldset>
</form>

Finally, start your merb application up (from the uploader folder not instead the dist directory).

merb

You might need to disable sql_session in your dist/conf/merb.yml

change:

:sql_session: true

to:

:sql_session: false

The server is running on port 4000 and the request path is /uploader e.g. http://localhost:4000/uploader

Next I’d like to work more tightly integrating this with my rails application. A few things I want to focus on:

• the traffic is routed in my nginx setup.
• whether or not the merb process should do the long processing needed after the file is uploaded.
• how to get the results of the processed xml files back into my database for my rails app to read from.
• how to indicate to the users that the file has been uploaded but may still be being processed.

Here’s a pretty simple Architecture I drew up using Graffle

Merb will handle reading files from the client and writing them to disk. Naming them with an identifiable name and sending the redirect back to our rails application or possibly a status page.

Next I’m thinking another process will monitor the folder merb copies files into. When new files appear pick them up, process them, and parse them back into our sql database. To determine when the filesystem has changed, I’m thinking inotify looks like the best approach and conveniently it already has a ruby binding so I can do my CRUD operations with ActiveRecord. At least this is my initial thinking as to how I can handle file uploads. Any suggestions or feedback would be great!

Software File upload, Merb, Ruby