Capistrano for Server Monitoring

Capistrano is a fantastic tool for managing deploys, but it’s capable of far more than just pushing up code and restarting servers. Here are couple of tricks for harvesting stats and managing servers with Capistrano.

Suppose you need to check memory usage on a group of machines with some regularity. You could of course just log in to each machine and run:

 free -m

but this doesn’t scale all that well. Capistrano provides a simple method for executing a shell command and capturing the output: capture

  task :memory_usage, :roles => :server_group do
    puts capture("free -m")
  end

This unfortunately will give you memory usage on only the first machine listed in the group server_group. More often, it’s helpful to check memory usage on a cluster of machines, maybe to find out which one is swapping. The method run can help us here. Run executes the provided command across a group of servers:

  task :restart_passenger, :roles => :passenger do
    run("touch #{current_path}/tmp/restart.txt")
  end

The above command would run touch on the tmp/restart.txt on all servers in the passenger group. The run command can also take a block, which allows us to capture the response:

  task :memory_usage, :roles => :server_group do
    results = {}
    run("free -m") do |channel, stream, data|
      if stream == :out
        results[channel[:server].to_s] = data
      end
    end
    results.each_pair do |server, data|
      puts server
      puts data
    end
  end

The run method block has three arguments: channel, stream identifier, and data.  Channel contains information about the remote process and may be used to send data back to the remote process. Stream is the stream type: :err for stderr and :out for stdout. Data is the data received for the command. The above code should display memory usage on each server defined in the server_group group.

One gotcha to be aware of with the run command is when it fails to run on one server, the entire task will fail. To get around this, you can add true to the shell command:

  run("ls -l /home/foo; true") do |channel, stream, data|
    if stream == :out
      results[channel[:server].to_s] = data
    end
  end

This code will run across all servers, even if the directory /home/foo does not exist on a particular machine.

Another use case for the run command is to tail logs across multiple servers.

  task :delayed_jobs_log, :roles => :jobs do
    trap("INT") { puts 'Interupted'; exit 0; }
    run "tail -f #{shared_path}/log/delayed_job.log" do |channel, stream, data|
      if stream == :out
        data.split(/\n/).each do |row|
          puts "#{channel[:host]}: #{row}" 
        end
      end
    end
  end

This allows us to tail the Delayed Job log across all our DJ servers, printing the host before each file line. Typing control-c will end the tail as it would on the server itself.

Capistrano is a very powerful tool capable of far more than just deploys. It is well worth your time exploring what you can do with with in regards to server management. The power of Capistrano lies in the Net::SSH library, another powerful library to check out for remote server management.

This entry was posted in capistrano, dev ops, ruby, sys admin. Bookmark the permalink.

Leave a Reply

Your email address will not be published. Required fields are marked *