Releases, rollouts and rolling back

Every new server of a node group boots from the group’s current release. Servers that already run keep theirs until you replace them: Hetscale replaces the servers on an older release one at a time, stops when a new server fails, and rolls back by making an earlier release current.

What is a release?

A release is what a server runs: a snapshot of the seed or an image digest, never :latest. When Coolify deploys the seed, the release is a snapshot taken after a Coolify deployment (Coolify). Each group holds a pointer to its current release, and every scale-out and every healing action boots from it. The group’s Deployment page lists them under “Releases”, with the columns Release, Taken, Source, Ref, Build and Servers.

Which servers are older?

The Servers page has a “Release” column that shows each server’s release (v1, v2, …) and marks an older one. A server is older when:

• it runs another release than the current one, compared by ref and build id, or names none, as a promoted seed does;
• its template version differs in a field that changes a running server: today the SSH keys, never the fallback list;
• it runs outside a placement group while its group spreads servers across hosts (Placement groups).

The adopted seed is never replaced.

How do I replace the older servers?

When servers are older, the Servers and Deployment pages show “‹count› servers run an older release or template”, “‹count› servers are not spread across hosts”, or “‹count› servers run an older release or template, or are not spread” when both apply, with Replace older servers and the time to expect: about two minutes per server plus the drain window. While it runs, the callout counts “‹n› replaced, ‹n› left” and offers Stop. Stop replaces no further server: a new server already on its way finishes, and the older server it replaces leaves once the new one passes the load balancer’s health check, or keeps serving if the new one fails.

The replacement is surge then remove, one server at a time:

• A new server starts from the current release and template.
• Once the load balancer’s health check passes, an older server leaves through the drain window at once, not at the end of its paid hour. The server whose paid hour ends soonest goes first. An older server stays while its new server is unhealthy, for example after a maintenance hold, and leaves once the new one is healthy again; if the new one fails instead, replacing stops and the older server keeps serving.
• The group’s desired count does not change.

The plan’s capacity never holds a replacement back. The cost guard, the euro ceiling, the node cap and Hetzner’s stock can. Pause, the emergency stop and maintenance hold it: while the node group is held for maintenance, no older server leaves, even one whose replacement already serves, until the hold ends. A server pinned for maintenance is skipped. A rollout also waits while a pipeline holds the group’s deploy lock, and goes on once the lock ends or is released. A scale-in waits while a replacement is on its way. How a server leaves is on Removing a server without dropping requests.

A pipeline starts the same replacement with POST /v1/groups/<group-id>/rollout and stops it with DELETE on the same path, with a token that has the rollouts:write ability (“Replace older servers”). The start answers 201 with the rollout, or 409 when the group is not Live (not_live), a rollout already runs (already_running) or no server is older (nothing_to_replace); the stop says whether a rollout ran.

What if a new server fails?

When a new server fails to provision, to deploy or to become ready, replacing stops, with no second attempt, and the older servers keep serving. The callout reads “Replacing older servers stopped”, and a critical mail says: “‹server›, a new server of release v‹n›, failed, so replacing the older servers stopped. They keep serving: fix the release, then start again on the Servers page.”

If the replacement of a server that is not spread cannot be spread either, replacing stops instead of creating and removing servers in a loop. The callout then says that Hetzner has no room to spread them now; try again later.

A server that fails while it serves is a matter for healing: What happens when a server dies.

How do I roll back?

On the Deployment page, choose Make current on an earlier release in “Releases”. This moves the pointer back to it; no copy is made. A snapshot release needs its snapshot still in your Hetzner project: a release whose snapshot a cleanup deleted shows “Snapshot deleted ‹date›” and cannot be made current. New servers boot from it from then on; then use Replace older servers to move the running ones. A webhook group’s endpoint receives group.release_changed.

A webhook group’s server is judged by the release Hetscale deployed to it (“by Hetscale’s record”). A pipeline that deploys a release to running servers in place reports each one deployed through the API; the server then counts as current and is no longer listed or shown as older: Webhook adapter.

How do I delete old snapshots?

The Deployment page lists them under “Snapshots Hetscale took”: their count, their total size, their monthly price at Hetzner’s list price, and “Not needed”, the snapshots no release in use, no running server and no new server needs.

Delete unused snapshots deletes those in one confirmed click, for a member who may change the group, while the project’s Read & Write token is active. The confirmation lists the snapshots it deletes by version, their size and the monthly saving at list price, and says why the others stay. A cleanup never deletes:

• the current release’s snapshot;
• the snapshots of the two newest releases other than the current one, kept for a rollback;
• release v1’s snapshot, in every group; every new server of a webhook group boots from it;
• the snapshot of a release a running server booted from;
• any image Hetscale did not take, such as a snapshot named through the releases API.

Each delete passes Hetscale’s ownership check: the snapshot must be in Hetscale’s records for that group, carry Hetscale’s labels for it and keep the description Hetscale gave it. Otherwise it stays, and a critical alert mail names it by its Hetzner image id. A snapshot with delete protection on in the Hetzner Console stays as well, with an alert; Hetscale never turns protection off. A snapshot already gone counts as deleted.

The timeline records “Snapshots deleted”: the versions deleted, their size and the monthly saving, and those left in place with the reason. A release whose snapshot was deleted stays in “Releases” with “Snapshot deleted ‹date›”, and Make current is no longer offered for it.

Nothing is deleted on a schedule, and the API has no cleanup. A pause or a deploy lock does not hold a cleanup; an emergency stop does, and so does a project whose writes are held. An archived group’s snapshots are cleaned by restoring the group, which comes back paused, cleaning and archiving it again, or in the Hetzner Console.

How do I keep new servers patched?

A snapshot release holds the seed’s disk as it was when the snapshot was taken, including its operating system packages. Updates you install later on running servers never reach the servers that boot from that release, and Hetscale runs nothing on your servers to update them. To keep new servers current, refresh the release, for example once a month or after a security update:

1. Choose Hold for maintenance in the More menu of the seed’s row on the Servers page, so that Hetscale does not start a server beside it while it restarts. Then update the seed’s packages, and restart it if they need that. Once it serves again, choose End maintenance in the same menu.
2. Choose Take a new release on the group’s Deployment page. New servers boot from the new release once its snapshot is ready. Servers already running keep their release.
3. Choose Replace older servers on the Servers or Deployment page. One at a time, a new server starts from the new release, and an older one leaves through the drain window once the new one serves. The seed is never replaced; it already runs the updates.
4. A member who may change the group chooses Delete unused snapshots on the Deployment page. The snapshots listed above stay, among them the current release’s and those of the two newest releases other than the current one, kept for a rollback.

For a group whose servers boot from the seed’s snapshot, the Deployment page shows “The release is getting old” once the current release is older than the group’s “Warn when the release is older than” setting. That setting is under Advanced in the group’s Settings: 30 days by default, from 1 to 365 days. Leave it empty to turn the warning off.

With the webhook adapter, every new server boots the snapshot taken for the group’s first release. A snapshot cleanup never deletes that snapshot, and your pipeline then deploys the current release onto the server. The release you deploy, such as an image digest, carries your application and its base image’s updates. A change to the seed, its packages included, does not reach these servers. Updating the server’s own packages is a step for your pipeline, before it reports the server ready and inside the readiness deadline.

What does this not cover?

• An automatic rollout after a new release: a pipeline starts one through the API.
• Deleting snapshots on a schedule or through the API.
• Reverting to an earlier template version.
• A release taken by itself after a Coolify deployment: take it on the Deployment page once the deployment has finished (Coolify).