Prometheus

On Pro and Team, Prometheus reads the state of every node group from Hetscale’s API: GET https://api.hetscale.com/v1/metrics answers every node group of your organisation outside the Archive, in the Prometheus text format, version 0.0.4.

What do I need?

• An organisation on Pro or Team. On another plan the answer is 403 with the code plan_required.
• An API token with the ability Read metrics (metrics:read), made on the panel’s API tokens page.

How do I scrape it?

Put the token in a file as the panel showed it, <id>|<secret>, and give Prometheus a job:

scrape_configs:
  - job_name: hetscale
    scheme: https
    metrics_path: /v1/metrics
    scrape_interval: 60s
    authorization:
      credentials_file: /etc/prometheus/hetscale-token
    static_configs:
      - targets: ['api.hetscale.com']

• The answer comes from Hetscale’s own data, never from Hetzner, so a scrape takes nothing from your project’s Hetzner API budget.
• It is kept for 60 seconds per organisation, so a scrape interval of 60 seconds is enough.
• The API’s limit of 60 requests a minute per organisation applies to the scrape as to any request.

Which series does it answer?

Each node group has its own series, labelled group_id (grp_…) and group (its name). Every metric is a gauge:

• hetscale_group_desired_nodes, hetscale_group_min_nodes and hetscale_group_max_nodes: the servers the group should run now, its minimum and its maximum.
• hetscale_group_active_nodes: the servers that serve and count as capacity.
• hetscale_group_in_flight_nodes: the servers being created, booted, deployed or health-checked.
• hetscale_group_shortfall_nodes: desired less active and in flight.
• hetscale_group_last_time_to_capacity_seconds and hetscale_group_last_routed_timestamp_seconds: the newest server routed, its Time to Capacity and when it was routed, in Unix seconds. Both are absent until a server of the group is routed.
• hetscale_group_state{state}: 1 for the group’s state among observe, rehearsing, live and paused.
• hetscale_group_capacity_limit{limit}, for a group past Observe: 1 for each limit that keeps the group from adding a server now. The limits are at_max, cost_guard, node_cap, euro_ceiling, plan_capacity and capacity_deficit (A node group is servers short).

What is not in it?

Hetzner’s own CPU and load balancer series, which Hetzner serves, and plan or billing figures.

Which alerts are worth setting?

• hetscale_group_shortfall_nodes > 0: desired servers neither serving nor on their way.
• hetscale_group_capacity_limit == 1: a limit holds a new server back.

The API reference lists the endpoint’s answers.