Kamal

Last updated

Kamal 2 deploys to a host list. On a node group that uses the webhook adapter, the list comes from Hetscale’s inventory before every deploy, the group’s deploy lock holds scaling back while Kamal rolls the fleet, and each new server gets a kamal deploy of its own.

What does the setup look like?

  • A Hetzner load balancer terminates TLS; kamal-proxy on each host serves plain HTTP behind it.
  • The group uses the webhook adapter, and the seed already runs the app: new servers boot from its snapshot.
  • An API token with Lock groups, Read nodes, Report nodes and Change releases, as on the GitHub Actions page.

How does Kamal get the host list?

config/deploy.yml reads it from an environment variable through ERB:

servers:
  web:
    hosts: [<%= ENV.fetch("HETSCALE_HOSTS") %>]

HETSCALE_HOSTS is a comma-separated list of IPs, such as 203.0.113.10,203.0.113.11. The steps below set it from the inventory before each kamal deploy. Kamal 2 does not load .env for this file, so the variable comes from the environment the command runs in.

How do I deploy a new version to the running servers?

Take the group’s deploy lock, record the release (its ref the image with Kamal’s version tag, its build_id the version), set HETSCALE_HOSTS from the serving servers, run kamal deploy, report each of Hetscale’s servers deployed and release the lock:

set -euo pipefail
API=https://api.hetscale.com/v1
AUTH="Authorization: Bearer $HETSCALE_TOKEN"
VERSION=$(git rev-parse HEAD)

curl -fsS -X POST "$API/groups/$HETSCALE_GROUP/lock" -H "$AUTH" -H 'Content-Type: application/json' \
  -d "$(jq -n --arg reason "kamal deploy $VERSION" '{ttl: 1800, reason: $reason}')"
trap 'curl -fsS -X DELETE "$API/groups/$HETSCALE_GROUP/lock" -H "$AUTH"' EXIT

RELEASE=$(curl -fsS -X POST "$API/groups/$HETSCALE_GROUP/releases" -H "$AUTH" \
  -H 'Content-Type: application/json' \
  -d "$(jq -n --arg ref "ghcr.io/acme/app:$VERSION" --arg build "$VERSION" '{ref: $ref, build_id: $build}')" \
  | jq -r .release.id)

curl -fsS "$API/groups/$HETSCALE_GROUP/nodes?phase=active" -H "$AUTH" > nodes.json
HETSCALE_HOSTS=$(jq -r '[.nodes[].ipv4] | join(",")' nodes.json)
export HETSCALE_HOSTS

kamal deploy --version="$VERSION"

for id in $(jq -r '.nodes[] | select(.class == "managed") | .id' nodes.json); do
  status=$(curl -sS --retry 3 -o deployed.json -w '%{http_code}' -X POST "$API/nodes/$id/deployed" \
    -H "$AUTH" -H 'Content-Type: application/json' \
    -d "$(jq -n --arg release "$RELEASE" '{release: $release}')")
  [ "${status:0:1}" = 2 ] && continue
  code=$(jq -r '.code // empty' deployed.json 2>/dev/null || true)
  if [ "$status" = 409 ] && { [ "$code" = node_not_active ] || [ "$code" = node_not_managed ]; }; then
    echo "warning: $id was not reported ($code)" >&2   # it left, failed or is the seed
    continue
  fi
  echo "$id: $status $code $(jq -r '.message // empty' deployed.json 2>/dev/null || true)" >&2
  exit 1
done

The notes of the GitHub Actions recipe apply: the release is recorded first, the TTL is the backstop when the script dies, the seed is deployed but not reported, a server not reported stays older on the panel, and servers without public addresses, whose ipv4 is null, are reached from inside the network at .private_ip. The script reads the serving servers once and does not wait for servers that started on the earlier release: they boot the earlier release and stay older until a rollout or the next deploy. The GitHub Actions workflow waits for them, because the action’s nodes step does when it is given the release. A deployed report is retried on a busy or failing API; a refusal other than a server that left, failed or is the seed (409 with node_not_active or node_not_managed) stops the script with the API’s code and message, and the trap still releases the lock.

From GitHub Actions

The workflow of the GitHub Actions page takes the lock, records the release and reads the servers with the Hetscale Deploy action. Its hosts output has one address per line, and Kamal’s HETSCALE_HOSTS takes them separated by commas, so the deploy step becomes:

      - name: Deploy with Kamal
        shell: bash
        env:
          HOSTS: ${{ steps.nodes.outputs.hosts }}
          VERSION: ${{ github.sha }}
        run: |
          export HETSCALE_HOSTS=$(paste -sd, - <<< "$HOSTS")
          kamal deploy --version="$VERSION"

release sends the commit SHA as the build id by default, which matches Kamal’s version here.

How does a new server get the app?

The relay of the GitHub Actions page, or any receiver of node.created, starts a job that deploys the event’s release to that one server without building, then reports it ready:

set -euo pipefail
API=https://api.hetscale.com/v1
AUTH="Authorization: Bearer $HETSCALE_TOKEN"

NODE=$(curl -fsS "$API/groups/$HETSCALE_GROUP/nodes" -H "$AUTH" \
  | jq -c --arg id "$NODE_ID" '.nodes[] | select(.id == $id)')
if [ -z "$NODE" ]; then
  echo "$NODE_ID is not a current server of this group" >&2
  exit 1
fi
if [ "$(jq -r '.release.id // empty' <<< "$NODE")" != "$RELEASE" ]; then
  echo "the event names another release than the server's" >&2
  exit 1
fi
PHASE=$(jq -r '.phase' <<< "$NODE")
case "$PHASE" in
  BOOTING | DEPLOYING) ;;
  HEALTH_CHECKING | ACTIVE)
    echo "$NODE_ID was reported deployed already: a repeated delivery, nothing to deploy"
    exit 0 ;;
  *)
    echo "warning: $NODE_ID is $PHASE and no longer waits for its deploy" >&2
    exit 0 ;;
esac

HOST=$(jq -r '.ipv4 // empty' <<< "$NODE")   # '.private_ip // empty' from inside the network
VERSION=$(jq -r '.release.build_id // empty' <<< "$NODE")
if [ -z "$HOST" ]; then
  echo "$NODE_ID has no public IPv4; use .private_ip if the job reaches the server that way" >&2
  exit 1
fi
HETSCALE_HOSTS="$HOST" kamal deploy --skip-push --version="$VERSION"

curl -fsS -X POST "$API/nodes/$NODE_ID/ready" -H "$AUTH" -H 'Content-Type: application/json' \
  -d "$(jq -n --arg release "$RELEASE" --argjson generation "$GENERATION" \
    '{release: $release, generation: $generation}')"

The job reads the server once from the inventory and takes its address there, never from the event or the dispatch: a forged or repeated dispatch cannot aim the deploy, or Kamal’s secrets, at another address. It stops when the server is not in the group or the event names another release, and it skips a server that no longer waits for its deploy, which is what a repeated delivery finds. The version is the build_id of the server’s release, which the deploy above set to Kamal’s version; --skip-push deploys the image already in the registry. When the deploy fails, report failed as the GitHub Actions workflow does.

From GitHub Actions, the new-server workflow’s node step gives the server’s address and its release’s build-id, so its deploy step becomes:

      - name: Deploy the node's release with Kamal
        id: deploy
        if: steps.node.outputs.deploy == 'true'
        shell: bash
        env:
          HETSCALE_HOSTS: ${{ steps.node.outputs.address }}
          VERSION: ${{ steps.node.outputs.build-id }}
        run: kamal deploy --skip-push --version="$VERSION"

ready and failed follow as on the GitHub Actions page.

The Kamal case puts this setup on the fit table, and the webhook adapter explains the events, the reports and the deploy lock.

See what Hetscale would have done with your real data — connect read-only, get your report in minutes.

Connect read-only