etsy/deployinator

_________               ______                _____                 _____
______  /_____ ________ ___  /______ _____  _____(_)_______ ______ ___  /_______ ________
_  __  / _  _ \___  __ \__  / _  __ \__  / / /__  / __  __ \_  __ `/_  __/_  __ \__  ___/
/ /_/ /  /  __/__  /_/ /_  /  / /_/ /_  /_/ / _  /  _  / / // /_/ / / /_  / /_/ /_  /
\__,_/   \___/ _  .___/ /_/   \____/ _\__, /  /_/   /_/ /_/ \__,_/  \__/  \____/ /_/
                /_/                   /____/             Deploy with style!

Deployinator - Deploy code like Etsy

Deployinator is a deployment framework extracted from Etsy. We've been using it since late 2009 / early 2010. This has been revamped into a ruby gem.

Table of Contents

• Stacks
• Installation
• Usage
  • Example Stack
  • Customizing your stack
  • Useful helper methods
  • Plugins
  • Template Hooks
• Hacking on the gem
  • Contributing

Stacks

Deployments are grouped by "stacks". You might have a "web" and "search" stack.

Each of those stacks might have different deployment environments, such as "staging" or "production".

You can map a button to each of these environments, to create multi-stage pushes within each stack.

Installation

This demo assumes you are using bundler to install deployinator. If you aren't you can skip the bundler steps.

• Create a directory for your project to hold your specific code (outside of the gem and this will be your own internal repository). mkdir test_stacks

• Create a Gemfile for bundler:

    source 'https://rubygems.org'
    gem 'etsy-deployinator', :git => 'https://github.com/etsy/deployinator.git', :branch => 'master', :require => 'deployinator'

• Install all required gems with bundler:

    $ bundle install --path vendor/bundle

• Run the following command to make deployinator gem's rake tasks available to you:

    $ shopt -s xpg_echo
    $ echo "require 'deployinator'\nload 'deployinator/tasks/initialize.rake' " > Rakefile

• Create a binstub for the deploy log tailing backend:

    bundle binstub etsy-deployinator

• Initialize the project directory by running the following command replacing Company with the name of your company/organization. This must start with a capital letter.

    $ bundle exec rake 'deployinator:init[Company]'

• Run the tailer as a background service (using whatever init flavor you like)

    ./bin/deployinator-tailer.rb &

Usage

Example Stack

• Use the deployinator rake task to create the stub for your stack. Replace test_stack with the name of your stack. This should be all lowercase with underscores but if you forget the rake task will convert from camelcase for you. The commands run by the rake tasks are logged to stderr.

    $ bundle exec rake 'deployinator:new_stack[test_stack]'

• We need a server to run our Sinatra application. For the purpose of this demo, we will use puma. Let's install puma into our bundle. Add the following to your Gemfile:

    gem 'puma'

• Now update your bundler:

    bundle install --path vendor/bundle --no-deployment && bundle install --path vendor/bundle --deployment

• Start the server by running:
  • The host could be localhost or the dns name (or ip address of the server you are using). You can set any port you want that's not in use using the -p flag.

    $ bundle exec puma -b tcp://0.0.0.0:7777 config.ru

• You will probably want a robust server like apache to handle production traffic.

• The config/base.rb file is the base config for the application. Replace all occurences of test_stack with the name you chose above. Also the example below uses a git repository of http://github.com/etsy/deployinator, feel free to replace this with your specific repository

Deployinator.app_context['test_stack_config'] = {
   :prod_host               => "localhost",
   :checkout_path           => "/tmp/deployinator_dev/"
 }
Deployinator.git_info_for_stack = {
    :test_stack => {
        :user => "etsy",
        :repository => "deployinator"
    }
}

• Edit the stacks/test_stack.rb file to look like this (adding git version bumping and checkout)

require 'helpers/test_stack'
module Deployinator
  module Stacks
    class TestStackDeploy < Deployinator::Deploy
        include Deployinator::Helpers::TestStackHelpers,
            Deployinator::Helpers::GitHelpers

      def test_stack_production(options={})
        # save old version for announcement
        old_version   = test_stack_production_version
        checkout_path = Deployinator.app_context['test_stack_config'][:checkout_path]
        prod_host     = Deployinator.app_context['test_stack_config'][:prod_host]

        # Clone and update copy of git repository
        git_freshen_or_clone(stack, "ssh #{prod_host}", checkout_path, "master")

        # bump version
        version = git_bump_version(stack, checkout_path, "ssh #{prod_host}", checkout_path)

        # Write the sha1s of the different versions out to the logs for safe keeping.
        log_and_stream "Updating application to #{version} from #{old_version}"

        # log the deploy
        log_and_shout :old_build => get_build(old_version), :build => get_build(version)
      end
    end
  end
end

• Next, edit the helpers/test_stack.rb file. You can delete the test_stack_head_build function since you are using the GitHelpers and that is automatically taken care of for you. Here is the final version:

module Deployinator
  module Helpers
    module TestStackHelpers
      def test_stack_production_version
        %x{ssh #{Deployinator.app_context["test_stack_config"][:prod_host]} cat #{Deployinator.app_context["test_stack_config"][:checkout_path]}/#{stack}/version.txt}
      end
    end
  end
end

• Create the directory that will contain the checkout if it doesn't exist already (defined in config/base.rb)

• Load up deployinator and deploy your stack!

Customizing your stack

A stack can be customized so that you have flexibility over the different environments within it (which correspond to buttons) and the methods that correspond to each button press.

By default, you will see a button called "deploy stackname" where stackname is the stack defined in the rake command. In your helpers file, you can add a function called stackname_environments that returns an array of hashes. Each hash will correspond to a new environment, or button. For example if your stack is called web, you can define a function like so in helpers/web.rb to define qa and production environments within your web stack:

  def web_environments
    [
      {
        :name            => "qa",
        :method          => "qa_rsync",
        :current_version => proc{send(:qa_version)},
        :current_build   => proc{send(:current_qa_build)},
        :next_build      => proc{send(:next_qa_build)}
      },
      {
        :name            => "production",
        :method          => "prod_rsync",
        :current_version => proc{send(:prod_version)},
        :current_build   => proc{send(:current_prod_build)},
        :next_build      => proc{send(:next_prod_build)}
      }
    ]
  end

The keys of each hash describe what you will be pushing for that environment:

• :name - name of the environment
• :method - method name (string) that gets invoked when you press the button (this is defined in the stack)
• :current_version - method that returns the version that is currently deployed in this environment (defined in the helper)
• :current_build - method that returns the build that is currently deployed (usually inferred from the version, also defined in the helper)
• :next_build - method that returns the next build that is about to be deployed (defined in the helper)

Useful helper methods

There are a few helpers built in that you can use after creating a new stack to assist you

run_cmd

Shell out to run a command line program. Includes timing information streams and logs the output of the command.

For example you could wrap your capistrano deploy:

run_cmd %Q{cap deploy}

log_and_stream

Output information to the log file, and the streaming output handler. The real time output console renders HTML so you should use markup here.

log_and_stream "starting deploy<br>"

log_and_shout

Output an announcement message with build related information. Also includes hooks for Email.

log_and_shout({
    :old_build  => old_build,
    :build      => build,
    :send_email => true
});

The supported keys for log_and_shout are:

• :env - the environment that is being pushed
• :user - the user that pushed
• :start - the start time of the push (if provided the command will log timing output)
• :end - the end time of the push (defaults to "now")
• :old_build - the existing version to be replaced
• :build - the new version to be pushed
• :send_email - true if you want it to email the announcement (make sure to define settings in config)

Plugins

Deployinator provides various entry points to execute plugins without having to modify the gem code. Here is a list of current pluggable events:

• :logout_url for defining your own authentications logout url
• :deploy_start for any actions to be performed at the start of a deploy
• :deploy_end for any actions to be performed at the end of a deploy
• :log_file_retired for when the symlink for the current log is retired
• :deploy_error for any actions to be performed when an error occurs during a deploy
• :run_command_start for any actions to be performed at the start of a run_cmd call
• :run_command_end for any actions to be performed at the end of a run_cmd call
• :run_command_error for any actions to be performed when an error occurs during a run_cmd call
• :timeout for any actions to be performed when a with_timeout calls times out
• :announce for any actions to be performed when announcing a deploy (IRC integration)
• :diff for generating diff links
• :timing_log for sending timing log information to anywhere besides the log file (Graphite for example)
• :auth for handling auth however you please

To create a plugin simply create a new class (example from our code) that defined a run method taking event and state. event is a symbol from the table above and state is a hash of state data which varies from event to event

require 'deployinator/plugin'
require 'helpers/etsy'
require 'deployinator/helpers'

module Deployinator
  class GraphitePlugin < Plugin
    include Deployinator::Helpers::EtsyHelpers,
      Deployinator::Helpers

    def run(event, state)
      case event
      when :run_command_end
        unless state[:timing_metric].nil?
          graphite_timing "deploylong.#{state[:stack]}.#{state[:timing_metric]}", "#{state[:time]}", state[:start_time].to_i
        end
      when :timing_log
        graphite_timing("deploylong.#{state[:stack]}.#{state[:type]}", "#{state[:duration]}", state[:timestamp])
      end
      return nil
    end
  end
end

Then simply require your plugin in lib/app.rb and add it to your config/base.rb like this:

Deployinator.global_plugins = []
Deployinator.global_plugins << "GraphitePlugin"

You can also configure plugins to only apply to a single stack like this:

Deployinator.stack_plugins = {}
Deployinator.stack_plugins["test_stack"] << "TestStackPlugin"

Template Hooks

Since the main layout page is contained within the gem, there are tags provided to allow you to add things to it in the header and body. List of points:

• tailer_loading_message To customize the default deploy tailer loading message
• additional_header_html Additional html to add to the header
• additional_body_top_html Additional html to add to the top of the body
• additional_body_bottom_html Additional html to add to the bottom of the body

To set these simple override the methods in your view class. For example:

     def additional_bottom_body_html
       '<script src="/js/check_push_status.js"></script>'
     end

This can be done on a global layout that extends the gem's default layout or on a stack by stack basis in their own view.

Maintenance mode

Deployinator has a setting for maintenance mode, which is mostly useful if you have major changes that affect all stacks and you want to make sure no deploys are going on while you make the change. In order to enable it, you have to set Deployinator.maintenance_mode = true. This will make all pages for anyone not in Deployinator.admin_groups go to /maintenance. As a Deployinator admin you can then still deploy stacks and use the app.

On the maintenance page Deployinator will show the value of Deployinator.maintenance_contact as the place to get help in case you need any or are confused about maintenance mode.

Hacking on the gem

If you find issues with the gem, or would like to play around with it, you can check it out from git and start hacking on it. First tell bundler to use your local copy instead by running:

    $ bundle config local.deployinator /path/to/DeployinatorGem

Next, on every code change, you can install from the checked out gem by running (you will want to make commits to the gem to update the sha in the Gemfile.lock)

    $ bundle install --no-deployment && bundle install --deployment

Stats dashboard

The /stats page pulls from log/deployinator.log to show a graph of deployments per day for each stack over time. By default, it shows all stacks. To blacklist or whitelist certain stacks, update config/base.rb with:

  Deployinator.stats_included_stacks = ['my_whitelisted_stack', 'another_whitelisted_stack']
  Deployinator.stats_ignored_stacks = ['my_stack_to_ignore', 'another_stack_to_ignore']
  Deployinator.stats_extra_grep = 'Production deploy' # filter out log entries matching this string

Whitelisting stacks or applying a custom extra grep can help speed up graph rendering when you have a large log file.

If at some point you change the name of a stack, you can group the old log entries with the new by adding the following to config/base.rb:

 Deployinator.stats_renamed_stacks = [
   {
     :previous_stack => {
       :stack => [ "old_stack_name" ]
     },
     :new_name => "new_stack_name"
   }
 ]

Contributing

1. Fork it
2. Create your feature branch (git checkout -b my-new-feature)
3. Commit your changes (git commit -am 'Add some feature')
4. Push to the branch (git push origin my-new-feature)
5. Create new Pull Request