Replace an unhealthy Hetzner server automatically

To replace an unhealthy Hetzner server automatically, go by your load balancer’s health check, not by ping or by Hetzner’s running status. When a server has failed the check for a few minutes in a row, start a new server from your snapshot beside it, let it take traffic once it passes the check, and only then take the failed one out, keeping it until you have looked at it. If the new server fails too, stop: two failures in a row point at your release, not at the server. Hetzner Cloud does none of this for you. Below are the rules, the steps with hcloud, and why a second server turns an outage into a blip.

What does “unhealthy” mean for a Hetzner server?

Unhealthy is what your load balancer says about a target. A Hetzner Load Balancer checks every target at a fixed interval, once per service, over TCP, which only proves that something accepts connections, or over HTTP, which can also require a status code and a piece of the response. A target that fails a set number of checks in a row gets no new requests until it passes as many in a row again.

Ping and Hetzner’s running status say that the machine is on, not that your application answers: the kernel answers ping while the application hangs. Use an HTTP check on a cheap path that never touches the database. A check that fails on every server when the database stumbles makes a script replace servers that were never broken (a health endpoint your load balancer can trust).

Since 17 August 2026 the Hetzner API also says why a target is unhealthy, in a detail field (the questions at the end explain the values). That reason is your first piece of evidence, and it disappears when the server leaves the load balancer.

How long should you wait before replacing a server?

About three minutes of failed checks in a row, counting only the minutes in which Hetzner reports neither a transitional status nor a lock (step 2). The load balancer itself needs roughly interval times retries to mark a dead target unhealthy, about 10 seconds with an interval of 5 s and 2 retries (the settings we recommend). Set the interval, the timeout and the retries yourself: the API documents no default, and the defaults in hcloud’s help are not sent unless you pass the flags.

The rest of the wait is yours. Acting on the first failed check replaces servers that were about to come back:

• A reboot. Hetzner has no rebooting status. A reboot or reset from the Hetzner Console or the API shows up as an action on the server; a reboot started inside the server, by unattended upgrades for example, creates no action and shows only as failed health checks.
• Hetzner moving the server to another host for maintenance, which Hetzner says usually interrupts it for less than a second.
• A deploy that restarts your containers one server at a time.
• Load. A server at full CPU can miss its check by timing out. It needs another server beside it, not a replacement of the same size that gets the same load. Look at its CPU graph.

Step 1: find the unhealthy server

hcloud load-balancer describe web-lb

Under Targets, every server lists a status per service: healthy, unhealthy or unknown. A target that reads unknown has no reading yet; do not count it. Since hcloud 1.70.0, released on 5 October 2026, the output also shows the Detail and, for a wrong status code, the HTTP Status Code. The health column of hcloud load-balancer list leaves out servers behind a label selector, so do not go by it.

If more than one server is unhealthy and none of them has its CPU pinned, stop here. That is the load balancer, the network or something every server shares, such as the database, and new servers will fail the same check.

Step 2: check what Hetzner says about the server

hcloud server list -l role=web -o columns=id,name,status,locked,created
hcloud api servers/<server-id>/actions -v sort=started:desc -v per_page=5

The first command shows each server’s status, lock and creation date. The second prints a server’s last five actions as JSON, newest first; read command and started (hcloud api needs hcloud 1.68.0 or later). Then:

• running: the machine is on and your application does not answer. If it still fails three minutes after you first saw it and its CPU is not pinned, replace it; if the CPU is pinned, add a server instead.
• off: if the actions show no shutdown_server or stop_server and nobody shut it down from inside, treat it as failed. Actions do not say who started them.
• unknown: Hetzner does not know the server’s state. Go by the load balancer.
• starting, stopping, migrating, rebuilding, initializing or deleting, the transitional statuses: something is already happening to the server. Wait and read again.
• locked is yes: the API says only that the server has been locked and is not available to you. Find out why before you act.

A reboot_server or reset_server action from the last few minutes explains a short failure: give the server a few more minutes.

Step 3: start the replacement beside the failed server

In the examples, web-2 is the failed server and web-3 its replacement:

hcloud server create --name web-3 --type cx33 --image <snapshot-id> \
  --location nbg1 --network app --ssh-key <key-name> --label role=web \
  --firewall <firewall>
hcloud load-balancer add-target web-lb --server web-3 --use-private-ip

Use a name no server in the project has, the snapshot of the release that was running, and the firewall your other servers have. The new server gets no traffic before it passes the health check (Hetzner support confirmed this to us in September 2026).

These steps use server targets, as our load balancer guide does. If your load balancer selects targets by label, skip add-target: the label puts web-3 behind it. In step 5, take web-2 out with hcloud server remove-label web-2 role; a firewall that applies by that label comes off too, so apply it to web-2 first with hcloud firewall apply-to-resource <firewall> --type server --server web-2.

Leave the failed server where it is. The load balancer already sends it no new requests, and deleting it first costs you three things:

• A server, if the create fails. A server type can be sold out in a location (resource_unavailable). Since 26 June 2026 Hetzner has also restricted new servers for some existing customers, who get the same answer (Hetzner support, October 2026; a status page notice still open on 20 October 2026). And a new account of ours was refused its sixth public IPv4 in September 2026 (the Hetzner limits that shape a node group). Keep a second location in your load balancer’s network zone and a second server type of the same architecture ready (capacity planning when your server type is sold out).
• The evidence. A deleted server is gone at once, with its disk and its backups.
• The price. A server ordered before Hetzner’s price change of 15 June 2026 keeps its lower price only while it exists; every server you create today pays today’s.

Step 4: wait for the health check, and stop if it fails

Run hcloud load-balancer describe web-lb again until web-3 reads healthy; a new target reads unhealthy until its checks pass. If it is not healthy after 15 minutes, or after your slowest good start if that takes longer, it has failed.

If it fails, do not start another one. One server failing is a server problem. A fresh server from the same snapshot failing too points at the release, or at what a fresh boot of it lacks: a service that does not start at boot, a secret it reads at boot, a change made by hand since the snapshot. A third server would fail the same way, and Hetzner bills each one for at least an hour. Read the replacement’s Detail, fix the snapshot, and keep both servers until then.

Step 5: take the failed server out and keep the evidence

Once the replacement is healthy, read web-2’s status again. If it is healthy again on its own, keep one of the two; a server ordered before 15 June 2026 is the cheaper one. If not, copy its Detail from step 1 and look at its screen before anything else:

hcloud server request-console web-2

It prints a WebSocket address and a password for a browser VNC client such as noVNC, valid for a minute; the console button in the Hetzner Console is easier. A kernel panic or an out-of-memory message stays on that screen until the next restart. Then:

hcloud load-balancer remove-target web-lb --server web-2
hcloud server enable-protection web-2 delete rebuild
hcloud server shutdown --wait --wait-timeout 2m web-2 || hcloud server poweroff web-2
hcloud server create-image --type snapshot --description "web-2 failed <date>" \
  --label evidence=web-2 web-2

The load balancer cuts the connections still open about five minutes after the removal (we measured this in September 2026).

Delete protection keeps anyone from cleaning up before you have looked. A hung server ignores the shutdown, so poweroff cuts the power after two minutes; Hetzner recommends snapshots of stopped servers. A snapshot, unlike a backup, survives the server’s deletion, and costs €0.0143 per GB a month (Hetzner list price since 1 April 2026, checked October 2026).

To look at the disk, run hcloud server enable-rescue --ssh-key <key-name> web-2 and hcloud server poweron web-2: the rescue system runs from memory, leaves the disk alone and answers SSH on the server’s public address. A server without public addresses has no rescue system; use the console.

Hetzner bills the failed server like a running one, off or on. When you are done, run hcloud server disable-protection web-2 delete rebuild. A server ordered before 15 June 2026 is worth keeping: rebuild it from the release snapshot, not the evidence one, with hcloud server rebuild --image <snapshot-id> web-2, which keeps its price, put it back behind the load balancer, and delete web-3 once web-2 serves. Otherwise delete web-2, after checking hcloud primary-ip list -o columns=name,ip,assignee,auto_delete,protection: an IP with auto delete goes with the server, and a protected one blocks the delete.

Why is a floor of two servers the difference between a blip and an outage?

With one server, nothing serves while the replacement starts. In our test on 1 October 2026, a Hetzner load balancer whose only target stopped answering returned 503 to every request from that second until the replacement passed its health check.

With two, the other server keeps serving, as long as it can carry the load alone. Users see a blip: requests sent to the failing server before the load balancer marks it unhealthy can fail, and connections already open to it stay with it (Hetzner support, September 2026). A shorter interval makes the blip shorter; we have not measured its size with two servers.

The second server is the price of self-healing: €8.49 a month for a CX33. Its public IPv4 adds €0.50 (Hetzner list prices without VAT, October 2026). On 20 October 2026 Hetzner listed the CX33 as not available, so price the type you can actually create.

Two servers on the same physical host can fail together. A spread placement group, free and holding up to 10 servers, puts each on a different host. Create one with hcloud placement-group create --name web-spread --type spread and pass --placement-group web-spread to new servers. An existing server has to be off to join (hcloud server add-to-placement-group --placement-group web-spread web-1), so move one at a time on a quiet day.

What does a script that replaces servers have to get right?

A cron script can run the steps above. Most of the work is in the cases where it must not act:

• No reading is not a failure. Count only minutes in which the load balancer was read and said unhealthy; a failed or rate-limited read says nothing (what the rate limit leaves for polling).
• A starting server is not failing. Hold a new server to its deadline, or the script replaces its own replacement.
• One replacement per failure, one run at a time, and a person alerted when it fails.
• Nothing replaced when two or more targets fail at once with idle CPUs, and nothing deleted when the failed server recovers first: a person picks which one to keep.
• It deletes only servers it created and labelled itself, never one you set up by hand.
• A pause switch for deploys, a reason logged for every action, and a watchdog: a cron job that stopped running heals nothing and tells nobody.

When not to replace a server automatically

• The server holds data. A database, uploads or sessions on the local disk come back only as they were when the snapshot was taken (when Hetscale is not a fit). Move them off first (make the server disposable).
• The snapshot is behind the server. A server changed by hand since its last snapshot comes back without those changes. Take a new snapshot first.

Where Hetscale fits

Hetscale automates the detection, the replacement and the stop in your Hetzner project; your traffic goes from your load balancer to your servers and never through us. A server counts as failed after three counted minutes unhealthy at your load balancer, with no minute counted during a transitional status or a lock. A server whose own CPU is pinned gets a grace period while the group scales out. Hetscale starts one server beside the failed one from the node group’s current release, and if that one fails too, healing stops with an alert. Unlike step 1, it does not hold back when several servers fail at once: each gets one replacement, and three new servers in a row that fail their check pause the group, so keep your health check shallow. Before an announced maintenance of your load balancer or network, choose “Hold for maintenance” on the node group (what happens during a Hetzner maintenance).

In a group whose minimum is two or more, a failed server Hetscale created is deleted once its replacement serves and the group’s “Keep a failed server” time has passed. Hetscale records the load balancer’s reason and its own steps, not the disk, so set that time if you want to look yourself. The server you started with (the seed) and your other servers are never deleted (what happens when a server dies).

We ran healing in production on 1 October 2026, in a project of our own with one server behind the load balancer: the failed server got one replacement, and when a bad release’s replacement failed its check, healing stopped. Healing is in every plan. With servers already behind a Hetzner load balancer, the onboarding case for a load balancer with two or more servers is the path, and the calculator puts a dead server at 03:00 next to your numbers.

Frequently asked questions

Does Hetzner Cloud replace a failed server by itself? No. Hetzner moves a server to another host for maintenance and, when it can, for a hardware fault; when it cannot, it says it tries its best to bring the machine back. Nothing replaces a server whose application stopped answering; the load balancer only stops sending it new requests.

Why is my Hetzner load balancer target unhealthy? Read the Detail that hcloud load-balancer describe prints. layer4_no_connection: the application is down or listens on another port. layer4_timeout: nothing answered, as when a firewall on the server drops the packets (Hetzner’s Cloud Firewalls do not filter the private network). layer7_timeout: the connection opened, but no answer came in time. unexpected_http_status: a redirect when you accept only 200, or a 404 from a proxy that routes by host name. unexpected_http_content: the answer lacks the text the check looks for. unspecified: no reason given, as right after a target joins (watch a health check fail on purpose).

What does unknown mean for a target? Hetzner does not define it. In our tests in September 2026, a server that had just joined a load balancer read unhealthy until its checks passed, not unknown. Treat unknown as no reading.

Should I reboot the server instead of replacing it? Not first. A reboot wipes the screen and the memory, and if it does not help, you have spent that time one server short. Start the replacement, look at the console, then reboot the failed server if you want it back; reset cuts the power, so use it only when reboot does nothing.

How long does a replacement take? We publish no figure until we have measured it on customers’ own setups. It depends on your snapshot’s size and your health check’s interval and retries (why there is no number yet).