Developer Blog

Continuous Delivery to DC/OS From CircleCI

At Originate we use CircleCI for all of our Continous Integration & Delivery needs. We’ve recently started to use DC/OS to deploy some of our software. This blog post will walk through the steps required to deploy a simple Marathon web service to DC/OS from CircleCI.

We’ll start with a very basic circle.yml and build it up as we go.

circle.ymllink
1
2
3
test:
  override:
    - echo "You'd normally run your tests here"

The best way to interact programmatically with DC/OS is through the CLI tool provided by Mesosphere. Let’s add it to our build container.

circle.ymllink
1
2
3
4
5
6
7
8
9
10
11
12
13
14
machine:
  environment:
    PATH: $HOME/.cache/bin:$PATH   # <- Add our cached binaries to the $PATH

dependencies:
  override:
    - scripts/ci/install-dcos-cli  # <- Install the DC/OS CLI

  cache_directories:
    - "~/.cache"                   # <- Cache binaries between builds to speed things up

test:
  override:
    - echo "You'd normally run your tests here"
scripts/ci/install-dcos-clilink
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
#!/usr/bin/env bash

set -euo pipefail

if [[ ! -z ${VERBOSE+x} ]]; then
  set -x
fi

BINS="$HOME/.cache/bin"

DCOS_CLI="$BINS/dcos"
DCOS_CLI_VERSION="0.4.16"

# Returns the version of the currently installed DC/OS CLI binary. e.g 0.4.16
installed_version() {
  dcos --version | grep 'dcoscli.version' | cut -d= -f2
}

# Downloads the DC/OS CLI binary for linux with version $DCOS_CLI_VERSION to the cache
install_dcos_cli() {
  mkdir -p "$BINS"
  curl -sSL "https://downloads.dcos.io/binaries/cli/linux/x86-64/$DCOS_CLI_VERSION/dcos" \
    -o "$DCOS_CLI"
  chmod u+x "$DCOS_CLI"
}

# Install the DC/OS CLI if it's missing. If it's present, upgrade it if needed otherwise do nothing
if [ ! -e "$DCOS_CLI" ]; then
  echo "DC/OS CLI not found. Installing"
  install_dcos_cli
else
  INSTALLED_VERSION="$(installed_version)"
  if [ "$DCOS_CLI_VERSION" != "$INSTALLED_VERSION" ]; then
    echo "DC/OS CLI has version $INSTALLED_VERSION, want $DCOS_CLI_VERSION. Upgrading"
    rm -rf "$DCOS_CLI"
    install_dcos_cli
  else
    echo "Using cached DC/OS CLI $INSTALLED_VERSION"
  fi
fi

Now let’s setup a basic Marathon service. We’ll use a stock nginx image as a stand in for a web service. Our service will have one instance with 1 CPU and 512MB of RAM and will map a random port on the host to port 80 in the container.

services/hello-world/marathon.jsonlink
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
{
  "id": "hello-world",
  "cpus": 1,
  "mem": 512,
  "instances": 1,
  "container": {
    "type": "DOCKER",
    "docker": {
      "image": "nginx:1.11-alpine",
      "network": "BRIDGE",
      "portMappings": [
        {
          "containerPort": 80,
          "hostPort": 0,
          "protocol": "tcp"
        }
      ]
    }
  }
}

The next step is to add a small script to send the manifest to DC/OS using the CLI tool. The Marathon API has separate endpoints for creating and updating a service so we have to check if a service already exists before doing the right thing. We’re using --force when updating to override any previous, potentially faulty deployment.

scripts/ci/marathon-deploylink
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
#!/usr/bin/env bash

set -euo pipefail

if [[ ! -z ${VERBOSE+x} ]]; then
  set -x
fi

DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"

SERVICE="$1"
SERVICE_MANIFEST="$SERVICE/marathon.json"
# Marathon automatically prefixes service names with a /. We use the slash-less version in the manifest.
SERVICE_ID="/$(jq -r '.id' "$SERVICE_MANIFEST")"

# This returns true if the service currently exists in Marathon (e.g. needs to be updated instead of created)
service_exists() {
  local service_id="$1"

  dcos marathon app list --json | jq  -r '.[].id' | grep -Eq "^$service_id$"
}

if service_exists "$SERVICE_ID"; then
  dcos marathon app update "$SERVICE_ID" < "$SERVICE_MANIFEST" --force
else
  dcos marathon app add < "$SERVICE_MANIFEST"
fi

We can now add the deployment section to the circle.yml file.

circle.ymllink
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
machine:
  environment:
    PATH: $HOME/.cache/bin:$PATH

dependencies:
  override:
    - scripts/ci/install-dcos-cli

  cache_directories:
    - "~/.cache"

test:
  override:
    - echo "You'd normally run your tests here"

deployment: # <- Add the deployment section
  master:
    branch: master
    commands:
      - scripts/ci/marathon-deploy services/hello-world

We’re almost there, the last thing we have to resolve is how to authenticate the DC/OS CLI to the cluster. Without that we can’t call the APIs we need to deploy our service.

DC/OS Community Edition uses an OAuth authentication flow backed by Auth0. This is used for both the browser-based authentication to get access to the admin dashboard as well as for the CLI tool. In the latter case, the user has to follow a slightly different browser-based flow yielding a token that is then provided to the CLI.

DC/OS CE Authentication Flow

In a CI/CD setting, anything that requires a manual user intervention is a non-starter. Enter dcos-login, the tool that we’ve created to solve this problem. Given a set of Github credentials it will replace the human component in the login flow and let you run the DC/OS CLI in your CI environment.

We recommend creating a separate “service” Github account just for that purpose. Once that’s done you can set GH_USERNAME and GH_PASSWORD environment variables in CircleCI to the username and password for that account.

Just like the DC/OS CLI, we need to pull down dcos-login to our build container.

scripts/ci/install-dcos-loginlink
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
#!/usr/bin/env bash

set -euo pipefail

if [[ ! -z ${VERBOSE+x} ]]; then
  set -x
fi

BINS="$HOME/.cache/bin"

DCOS_LOGIN="$BINS/dcos-login"

# Returns the version of the currently installed dcos-login binary. e.g v0.24
installed_version() {
  dcos-login --version 2>&1
}

# Returns the version of the latest release of the dcos-login binary. e.g v0.24
latest_version() {
  curl -sSL https://api.github.com/repos/Originate/dcos-login/releases/latest | jq -r '.name'
}

# Downloads the latest version of the dcos-login binary for linux to the cache
install_dcos_login() {
  mkdir -p "$BINS"

  LATEST_RELEASE="$(curl -sSL https://api.github.com/repos/Originate/dcos-login/releases/latest)"
  DOWNLOAD_URL=$(jq -r '.assets[] | select(.name == "dcos-login_linux_amd64") | .url' <<< "$LATEST_RELEASE")

  curl -sSL -H 'Accept: application/octet-stream' "$DOWNLOAD_URL" -o "$DCOS_LOGIN"
  chmod u+x "$DCOS_LOGIN"
}

# Install dcos-login if it's missing. If it's present, upgrade it if needed otherwise do nothing
if [ ! -e "$DCOS_LOGIN" ]; then
  echo "dcos-login not found. Installing"
  install_dcos_login
else
  INSTALLED_VERSION="$(installed_version)"
  LATEST_VERSION="$(latest_version)"
  if [ "$LATEST_VERSION" != "$INSTALLED_VERSION" ]; then
    echo "dcos-login has version $INSTALLED_VERSION, latest is $LATEST_VERSION. Upgrading"
    rm -rf "$DCOS_LOGIN"
    install_dcos_login
  else
    echo "Using cached dcos-login $INSTALLED_VERSION"
  fi
fi

Next we need to use dcos-login to authenticate the DC/OS CLI. You’ll also need to provide the URL to your DC/OS cluster in the CLUSTER_URL environment variable.

scripts/ci/loginlink
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
#!/usr/bin/env bash

set -euo pipefail

DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"

# Check that all required variables are set
for NAME in GH_USERNAME GH_PASSWORD; do
  eval VALUE=\$$NAME
  if [[ -z ${VALUE+x} ]]; then
    echo "$NAME is not set, moving on"
    exit 0
  fi
done

if [[ ! -z ${VERBOSE+x} ]]; then
  set -x
fi

CLUSTER_URL="$1"

# Setup the DC/OS CLI
## Point it to the DC/OS cluster. CLUSTER_URL is the URL of the admin dashboard
dcos config set core.dcos_url "$CLUSTER_URL"

## Set the ACS token. dcos-login reads GH_USERNAME and GH_PASSWORD from the environment automatically
dcos config set core.dcos_acs_token "$(dcos-login --cluster-url "$CLUSTER_URL")"

Finally, tie it all together in the circle.yml file.

circle.ymllink
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
machine:
  environment:
    PATH: $HOME/.cache/bin:$PATH

dependencies:
  override:
    - scripts/ci/install-dcos-cli
    - scripts/ci/install-dcos-login-cli # <- Install the dcos-login tool

  cache_directories:
    - "~/.cache"

test:
  override:
    - echo "You'd normally run your tests here"

deployment:
  master:
    branch: master
    commands:
      - scripts/ci/login "$CLUSTER_URL" # <- Authenticate the DC/OS CLI
      - scripts/ci/marathon-deploy services/hello-world

That’s it, you can now deploy your services from CircleCI to DC/OS Community Edition. The dcos-login tool is free and open source. All the code in this blog post can be found in this example project.

FAQ

What about private docker registries?

Marathon lets you pull docker images from private registries with a bit of configuration. You need to tar up the .docker directory of an authenticated host and instruct Marathon to pull down that archive when launching a new instance of a service:

1
2
3
4
5
"fetch": [
  {
    "uri": "https://some-s3-bucket.s3.amazonaws.com/docker-hub.tar.gz"
  }
]

How do I pass configuration / secrets to my service?

Couple of strategies here:

  • Set the environment variable in CircleCI and then use sed or envsubst to replace placeholders in your marathon.json.
  • Store a subset of your marathon.json with your configuration / secrets on S3 or similar. Pull down that file (which might be specific per environment) at deploy time and use jq to merge it into the main marathon.json manifest for your service. You can use something like jq -s '.[0] * .[1]' marathon.json secrets.json.

How do I assign a DNS name to my service?

If you’re using Marathon LB you can add the following section to your marathon.json:

1
2
3
4
5
"labels": {
  "HAPROXY_GROUP": "external",
  "HAPROXY_0_VHOST": "hello-world.yourdomain.com",
  "HAPROXY_0_BACKEND_HTTP_OPTIONS": "  acl is_proxy_https hdr(X-Forwarded-Proto) https\n  redirect scheme https unless { ssl_fc } or is_proxy_https\n"
}

The HAPROXY_0_VHOST value instructs Marathon LB to map the first port in your port mapping (index 0, HTTP in our case) to that virtual host. You should have an entry in your DNS zone pointing *.yourdomain.com to the public DC/OS nodes running Marathon LB

How do I make sure that my service is alive?

You can instruct Marathon to perform a health check on your behalf by adding the following to your marathon.json:

1
2
3
4
5
6
7
8
9
10
"healthChecks": [
  {
    "path": "/",
    "protocol": "HTTP",
    "portIndex": 0,
    "gracePeriodSeconds": 10,
    "intervalSeconds": 20,
    "maxConsecutiveFailures": 3
  }
]

As with the virtual host above portIndex corresponds to the index of the port in the portMappings section.

Comments