Here at TailorDev, we are big fan of automation. As a rule of thumb, we automate anything that is done more than two times “by hand”. We focused on software deployment for quite some time now, and this article sums up our current setup.

Insight of Our (Software) Deployment Process

The figure below represents our current deployment flow. We ask our company’s robot, Hubot, to deploy our applications from Slack. Hubot creates deployment events on GitHub, which forwards them to Heaven, the application that performs the deployments on our servers. Most of the content below is part of our internal documentation, which we are happy to share with you!

             +---------+         +--------+         +--------+         +--------+
             |  hubot  |         | GitHub |         | Heaven |         | Server |
             +----+----+         +----+---+         +--------+         +--------+
 /deploy app      |                   |                  |                  |
+---------------> | Deployment req.   |                  |                  |
                  +-----------------> |                  |                  |
                  |                   | Deployment event |                  |
                  |                   +----------------> |                  |
                  |                   |                  | SSH commands     |
                  |                   |                  +----------------> |
                  |                   |                  |                  |
                  |                   |                  | <----------------+
                  |                   |   Update status  |                  |
                  |                   | <----------------+                  |
                  |                   |                  |                  |
                  |          Send status to Slack        |                  |
                  | <-----------------+------------------+                  |
                  |                   |                  |                  |

Hubot & hubot-deploy

The hubot-deploy plugin is used to talk to the GitHub Deployment API by sending deployment events. It also deals with API responses, including the responses related to the required contexts (e.g., the ci context, which does not allow to perform a deployment when the CI status is not “green”). For Slack users, there is also SlashDeploy, in case you do not use Hubot.

Hubot knows the applications that she is able to deploy, their environments (e.g., staging, production), and how to deploy them (i.e. which tool to use). In addition, she is able to restrict deployment requests by (Slack) channel (also known as room). For instance, deployment requests are only possible in the #ops channel at TailorDev. Such a configuration is located in the apps.json configuration file, read by Hubot on start (we give an example of such a file at the end of this article).

Security concerns: there is no sensitive data in the apps.json file. Hubot requires an access token with the repo_deployment scope only.

Once the deployment request has been sent, i.e. a deployment event has been created using GitHub’s API, Hubot’s job is done, even though the actual deployment has not been performed yet. We rely on GitHub to “ping” our deployment tool, called Heaven.

Heaven

Heaven is responsible for performing the deployments. It listens to GitHub Deployment API events, and triggers the deployment scripts. It is able to perform Capistrano-, Fabric-, Heroku-, or shell-based deployments. It saves all deployment logs to Gists (which is why it requires an access token with the gist scope).

Whenever it receives a deployment event, it creates a task in a Redis queue, which a worker consumes. Let’s take an example with a shell-based deployment. When a deployment event is received by Heaven, it tells a worker to perform the deployment task (which is often deploy). It starts by cloning the application code in a /tmp folder (that is why Heaven requires an access token with the repo scope).

Heaven expects a shell script to be found in the repository of the application. Our convention is to have a bin/deploy script, which sources our deploy_utils.sh shell library. This library contains utility functions to:

  • ssh/rsync servers with common configurations;
  • provide a caching layer for dependencies (for npm, bundler, bower, etc.);
  • expose a deploy:noop task, which runs the deployment recipe without performing it (this is our dry-run mode);
  • expose a deploy:no-cache task for rebuilding cache;
  • send data to Datadog so that we can monitor deployment durations.

Thanks to this shell library, all our shell-based deployment scripts are similar. We only need to write the project’s configuration and the commands required to (build and) deploy in the bin/deploy scripts.

One advantage of such scripts is that it is still usable “by hand”, i.e. without Heaven or any other automated process. This is important to avoid some sort of “vendor lock-in”. We can hook into any layer or switch to something else without any problem. If one wants to deploy from her laptop, she can run the command below (of course, she has to be able to SSH into the server):

SSH_USER=`whoami` bin/deploy

Security concerns: such shell-scripts expose a few information such as SSH_USER and SSH_HOST. While any user is identified with a 4096 bits SSH key, it still give hints about our internal infrastructure. That is why we use a slightly different script in Monod (we never know).

Talking about SSH, the Heaven user can SSH into servers, and we use SSH keys for that. On one hand, we have a heaven.pub (public) key we set as authorized key of the “deployer” user on the target servers, and, on the other hand, the private key is given to Heaven as an environment variable (DEPLOYMENT_PRIVATE_KEY). This setup restricts who is able to deploy, and everything is managed with Ansible.

Ansible & The deployable Role

We have a deployable role, which is used to create a “deployer” user, and add authorized SSH keys from all people allowed to deploy. We usually add the Heaven user, but we can add any employee. SSH public keys are pulled either from our repository or GitHub automatically.

Right now, employees are allowed to deploy some applications from their laptop, but ideally, this should not be allowed since it does not improve communication in our distributed team. Even if deployments are written in GitHub, deploying code from Slack allows to let everyone know.

So, How to Deploy an Application?

In order to deploy an application at TailorDev, we have to setup some environments for it. Most of the time, we have two environments: staging (behind a oauth2_proxy, and managed via GitHub teams), and production. Then, it is a matter of configuration:

1 - Hubot must be aware of the app, so the apps.json file should be edited:

{
  "app": {
    "provider": "shell",
    "auto_merge": true,
    "repository": "TailorDev/app",
    "environments": [ "production", "staging" ],
    "allowed_rooms": [ "ops" ],
    "deploy_script": "bin/deploy"
  }
}

2 - The app repository must have a bin/deploy script. Say we want to deploy to a server called ratatouille (yes, we are aware of Pets vs. Cattle):

#!/usr/bin/env bash

# Project configuration
APP_NAME="app"
DIST_DIR="dist"
DEPLOY_DIR="~/$APP_NAME"

SSH_USER="a-deployer-user"
SSH_HOST="ratatouille"

# Load deploy utilities
REPO_DIR="$(cd "$( dirname "${BASH_SOURCE[0]}" )/.." && pwd)"
source "$REPO_DIR/bin/deploy_utils.sh"

function deploy() {
    local dist="$REPO_DIR/$DIST_DIR"

    # 1. create a build in `$dist`

    # 2. deploy build
    _rsync "$dist/" "$SSH_USER@$SSH_HOST:$DEPLOY_DIR/$ENV/"
}

_main "$@"

All the _* functions are provided by our deploy_script.sh library, and are configured to fit our needs. The _main function requires a deploy function to be defined. Optional features (such as sending data to Datadog) fail silently if something goes wrong, because we do not want a deployment to fail randomly.

3 - The Ansible playbook describing ratatouille must define the deployable role:

- hosts: ratatouille
  roles:
    - role: deployable
      apps:
        - { name: app, envs: [ production, staging ] }

4 - That’s it!

Conclusion

We have a single workflow for deploying all of our applications, from “static” websites (e.g. this blog) to internal tools and public applications, even those we manage in an Open Source manner (you can look for the squirrels here).

Our stack is managed by Ansible but also Terraform, which allows to create servers and provision them automatically. We have many Ansible roles to quickly setup our different environments. The combination of those tools helps us be more efficient and productive, avoiding to waste our time. Our deployment process tend to be more and more boring and stress-less.


Our current setup works well for us, but there is still room for improvements. We would be happy to read your thoughts on Twitter! Do you share a similar process? Are you satisfied with it? Did you identify any limitations?