Uptimer v2.0.0-preview documentation — it describes the release candidate, and v1.8.0 remains the current release. Commands here pull the preview image :2.0.0-rc15@sha256:945bd3a8fd42800445826df5d0968fd4c86d1d2a2e69f0fb41d897c66b0b0235. The Python SDK for this preview is a local wheel, not a PyPI release. Go to the current release (v1.8.0) →
Operating › Workers and certificates

Workers and certificates

Register a worker, issue its certificate, run it, give it confirmers, check it with –once, rotate and revoke it, and replace 1.x workers.

A worker runs the checks for one Location and reports what it saw. In 2.0 a worker proves who it is with a client certificate issued by your installation (mutual TLS on the worker API, port 50051). Its requests carry no key and no ID: the certificate is its identity.

Register and issue

  1. In the UI, open Server → Workers → Register a worker. Give it a name, a Location, and a new UUID as its Identity (uuidgen).

  2. On the control plane, issue its certificate:

    uptimer worker-cert --uid <the UUID> --out ./edge-1
    
    Issued a certificate for edge-1 (<the UUID>), good until <one year from now>.
    

    The command writes three files. The first run also creates the installation’s worker authority in UPTIMER__GRPC__CA_DIR (default <data>/worker-ca); keep it on a volume and back it up.

FileWhat it isKeep it
worker.pemthe worker’s certificatepublic
worker-key.pemthe worker’s private keysecret: only on the worker host
worker-ca.pemthe authority that signed the control plane, so the worker knows it is talking to your installationpublic

From 2.0.0-rc11, POST /api/v3/locations and POST /api/v3/workers script step 1 with a server admin’s full key; the answer’s next_step is the worker-cert command for step 2 (Register workers by API).

The private key is written to disk and never printed. --show prints the certificate.

In the container image the binary is /app/uptimer:

docker compose exec uptimer /app/uptimer worker-cert --uid <the UUID> --out /data/workers/edge-1

Run the worker

Copy the three files to the worker host and start it:

UPTIMER__WORKER__GRPC_SERVER=uptimer.example.com:50051 \
UPTIMER__WORKER__CERT_FILE=/etc/uptimer/worker.pem \
UPTIMER__WORKER__KEY_FILE=/etc/uptimer/worker-key.pem \
UPTIMER__WORKER__CA_FILE=/etc/uptimer/worker-ca.pem \
UPTIMER__OPS__PORT=9091 \
uptimer worker
SettingFlagMeaning
UPTIMER__WORKER__GRPC_SERVER--serverthe control plane’s worker API, host:port
UPTIMER__WORKER__CERT_FILE--certworker.pem
UPTIMER__WORKER__KEY_FILE--keyworker-key.pem
UPTIMER__WORKER__CA_FILE--caworker-ca.pem
UPTIMER__WORKER__EVERY--everyseconds between asking for its assignments; default 30
UPTIMER__OPS__PORT--ops-port/metrics, /livez, /readyz; required for the daemon

The host name in UPTIMER__WORKER__GRPC_SERVER must be one the control plane’s certificate covers. Set them on the control plane with UPTIMER__GRPC__HOSTS (default localhost,127.0.0.1,::1), for example UPTIMER__GRPC__HOSTS=uptimer.example.com.

A worker started without its files stops with a message that says which file is missing and how to issue it.

Confirmations

From 2.0.0-rc9 a worker also answers Signal confirmations: a problem reading that a Template asks to confirm is checked again before Rules believe it. A worker claims these tasks over the same worker API connection as its checks, and answers only the tasks it claimed, before their timeout.

SettingFlagMeaning
UPTIMER__WORKER__COMMANDS_FILE—the named commands this worker runs; default commands.json in its data directory; none when the file is absent

Built-in probes, always available: reachability, icmp, tcp, tls, http, dns. Their settings come from the Template. A probe that did not get an answer (a refused or timed-out connection, a non-matching status, a missing DNS name) answers problem. A probe that could not run (settings refused, or no ICMP on this worker) answers no_data, and the original reading stands. ICMP needs unprivileged ping (net.ipv4.ping_group_range) or CAP_NET_RAW; without either, icmp always answers no_data.

Bounds: a worker runs at most 4 confirmations at once, probes and commands together. Each runs for at most 30 seconds, or less if the confirmation’s deadline comes first. Each answer is a state, an optional number (milliseconds, days left, answers found) and at most 200 characters of reason, which never quotes the address.

On start the worker logs what it confirms with:

uptimer worker confirms with probes="[dns http icmp reachability tcp tls]" commands=[check-replica]

The worker’s commands file

UPTIMER__WORKER__COMMANDS_FILE, by default commands.json in the worker’s data directory:

{"commands": {
  "check-replica": {
    "path": "/usr/local/lib/uptimer/check_replica",
    "args": ["--host", "{host}", "--max-lag={max_lag}"],
    "params": {"host": "[a-z0-9.-]{1,253}", "max_lag": "[0-9]{1,4}"},
    "output": "nagios",
    "timeout_seconds": 10
  }
}}
KeyMeaning
command namewhat a Template calls it: lowercase letters, digits, - and _, starting with a letter, up to 64 characters
paththe executable, absolute; it must exist and be executable
argsits arguments, at most 32, each passed as it is; {name} marks where a parameter goes
paramseach parameter the command takes, with a pattern its value must match in full; every {name} in args must be one of these
outputjson or nagios
timeout_seconds1 to 30
envoptional, from 2.0.0-rc11: variable names the command gets from the worker’s own environment, such as ["HOME", "REPLICA_PASSWORD"] (below)

Unknown keys are refused. A file that is not there means no commands. A file that is there but wrong stops the worker at start, so a mistake shows up before a confirmation needs it. A worker is handed only the commands its file defines.

The Template side

"signals": [{"key": "replica_lag", "kind": "event",
  "confirm": {"by": [{"exec": "check-replica", "with": {"host": {"meta": "host"}, "max_lag": 30}}],
              "refuted_if": "any_ok", "timeout": 30}}]

How a command runs

Passing variables (env)

From 2.0.0-rc11 a command entry may list env: the names of variables to pass from the worker’s own environment, for example a password the check needs.

"check-replica": {
  "path": "/usr/local/lib/uptimer/check_replica",
  "args": ["--host", "{host}"],
  "params": {"host": "[a-z0-9.-]{1,253}"},
  "output": "nagios",
  "timeout_seconds": 10,
  "env": ["HOME", "REPLICA_PASSWORD"]
}

What a command answers

A command that cannot start, runs past its time, is refused, or answers something unreadable answers no_data. The confirmation then keeps the original reading.

The answer is kept with the original reading in the Resource’s observation history (API v3 GET …/observations, MCP list_observations, and the Observations page). History never names the worker.

Check a worker by hand

--once asks for the assignments, runs what is due, reports and stops. It serves no operations port.

uptimer worker --once   # with the same settings as above
… msg="uptimer worker running one cycle" server=uptimer.example.com:50051 kinds=[http]
… msg="check ran" assignment=… state=ok

It exits 0 when the report was accepted, and non-zero when the control plane refused it or did not take the report.

Rotate

Issue again for the same UUID. The new certificate replaces the old one, and the old one stops being accepted at once:

uptimer worker-cert --uid <the UUID> --out ./edge-1

Copy the new files to the worker host and restart the worker.

Revoke

uptimer worker-cert --uid <the UUID> --revoke
Withdrawn. edge-1 is refused from its next request on.

The worker’s next request is refused:

… msg="uptimer stopped" error="ask for this worker's assignments: rpc error: code = Unauthenticated desc = this worker is not accepted"

Issue again to let it back in. To stop a worker without withdrawing its certificate, switch it off in Server → Workers.

Coming from 1.x workers

1.x workers identified themselves with a pasted key and a UID file (UPTIMER__WORKER__KEY_FILE holding a key, UPTIMER__WORKER__UUID_FILE, UPTIMER__WORKER__GRPC_USE_TLS). 2.0 does not accept them, and a 1.x worker cannot talk to a 2.0 control plane.

For each 1.x worker you keep:

  1. Register a 2.0 worker in the 2.0 UI under a new UUID and a 2.0 Location.
  2. Issue its certificate with worker-cert and copy the three files to the host.
  3. Run the 2.0 uptimer worker beside the 1.x worker, pointed at the 2.0 control plane.
  4. Stop the 1.x worker when you retire the 1.x installation. See Coming from 1.8.

In 2.0, UPTIMER__WORKER__KEY_FILE is the certificate’s private key, not a 1.x key.