Running Uptimer
Commands, services, ports, the operations endpoint, state on disk and settings.
This is the operator reference for the 2.0 control plane: the commands it answers to, the settings it reads, the ports it serves, the paths it keeps state in, and what it no longer does. Coming from 1.8? Start with Moving from Uptimer 1.8 to 2.0.
The binary is uptimer. It is configured by environment variables and command
flags. There is no configuration file.
Commands
| Command | What it does |
|---|---|
uptimer serve --services … | Run the control plane. With no --services, every service in one process. |
uptimer migrate | Bring the database schema up to date, then exit. The rollout gate. |
uptimer worker | Run a worker daemon against a control plane. |
uptimer worker-cert --uid <uid> | Issue (or revoke) a worker’s client certificate. |
uptimer dev | Everything in one process for a local run: identity, certificate, bus with no socket. |
uptimer version | Print the version, commit and build time. |
uptimer --help, uptimer <command> --help | What each command reads and accepts. |
Run the binary with no command and it serves: that is what a container does. An unknown command is refused rather than ignored.
Services
serve --services takes any of ui, api, grpc, judge, nats, cache,
and eventbus as the alias for nats,cache. An unknown name is refused and
the message lists the choices. A single box runs them all in one process; a
fleet runs them apart against the same database, bus and cache.
Asking for grpc with UPTIMER__GRPC__PORT=0 is refused: a service that was
asked for and cannot serve is not quietly dropped. An installation that runs no
workers turns that port off and names no services, and the default set is then
everything except the worker API.
A process that runs no database-backed service — nats, cache, eventbus —
never opens the product’s database.
Ports
| Port | Serves | Setting |
|---|---|---|
| 8080 | the UI | UPTIMER__UI__PORT |
| 2518 | the public API, when it runs as its own process | UPTIMER__SERVER__API__PORT |
| 50051 | the worker API (gRPC, mutual TLS) | UPTIMER__GRPC__PORT |
| 4222 | the event bus | UPTIMER__EVENTBUS__PORT |
| 6379 | the cache | UPTIMER__REDIS__BIND_PORT |
| 9090 | operations: /metrics, /livez, /readyz (required) | UPTIMER__OPS__PORT |
A port a process needs and cannot bind is a refusal to start that names the port. A process that logged the failure and carried on would look healthy while serving nothing.
Run the services apart and give each process its own operations port; that is how a fleet scrapes them separately.
Operations endpoint
Every long-running process — including worker — serves one operations
listener:
| Path | Answers |
|---|---|
/metrics | Prometheus text, including one uptimer_build_info series per service this process runs |
/livez | 200 while the process is running. This is what restarts a wedged container. |
/readyz | 200 only when every selected service is serving. Otherwise 503 and the reason. |
Readiness goes non-OK when a selected service fails to start or its serving loop stops — the shell, the API, the worker API, the judge, the bus and the cache each report their own — so a process that has lost one of its jobs is taken out of a load balancer rather than left in it.
A process that serves the shell with OIDC sign-in also checks its provider’s
discovery document: it must be reachable, be JSON, name exactly the configured
issuer, and list the authorization, token and key endpoints. Otherwise
/readyz answers 503 with oidc: …, because nobody can sign in. One answer
holds for 15 seconds. /livez stays 200. The endpoint
is on the operations listener and needs no sign-in.
A process asked for eventbus answers to that name as well as to nats and
cache, because it is the name the deployment used.
The listener is not optional: serve and a worker daemon refuse to start
without a port for it, because a process nothing can scrape or probe is one
nobody can take out of a load balancer. UPTIMER__OPS__PORT moves it and
UPTIMER__OPS__ADDR binds it to one interface. worker --once serves none: it
asks, runs, reports and stops.
Judging
A process running judge judges a Resource as soon as an Observation for it is announced, and
the reports behind the first in a burst about three seconds later. Once a minute it also judges
what the clock alone can change, and only that: a Rule whose confirm or recover hold has elapsed,
a heartbeat that has gone quiet, a reading that has left a counted window, a Rule never judged,
and any Resource with stored evidence no judge has caught up with (a lost announcement included).
When it starts it judges every Resource once. A fleet with nothing new and nothing due is not
judged again, so an idle fleet costs a few queries a minute (from the release candidate after
2.0.0-rc7; before, every Resource was judged every minute). uptimer_judge_passes_total shows
which of these each pass was.
From 2.0.0-rc9 the once-a-minute sweep also decides each
confirmation whose timeout has
passed, before it judges anything, and a confirmation a worker answers is judged at once. A
Workspace has at most 50 confirmations open; past that a problem is not confirmed and the
original reading stands (unconfirmed, limited).
Metrics
| Series | Labels |
|---|---|
uptimer_build_info | service, version, commit, built |
http_request_duration_seconds | route, method, status |
grpc_request_duration_seconds | grpc_method, grpc_code |
uptimer_judge_decisions_total | result (ok, error) |
uptimer_judge_passes_total | trigger, result (ok, error, draining) |
uptimer_judge_pass_duration_seconds (histogram) | trigger |
uptimer_judge_queue_length | none |
uptimer_judge_queue_oldest_seconds | none |
uptimer_judge_ready_total | trigger |
uptimer_judge_deferred_total | reason (muted, draining, failed) |
uptimer_worker_checks_total | result (ok, error) |
uptimer_oidc_sign_ins_total | result (ok, error) |
uptimer_ingest_requests_total | source, result (accepted, partial, rejected, failed) |
uptimer_ingest_observations_total | source, result (accepted, rejected, failed) |
uptimer_delivery_failures_total | operation (render, send, record) |
uptimer_announce_failures_total | operation (announce, remind) |
uptimer_confirmations_limited_total | none |
route is the route pattern, not the path, so one route is one series.
trigger is why a judging pass over one Resource ran: observation (an
announced Observation), timer (a hold, a quiet heartbeat or a window was
due), startup (the pass over everything when a judge starts), unannounced
(an Observation that stood unjudged for a whole sweep, about a minute, with no
announcement answered for it; from 2.0.0-rc10) or retry (a pass that failed
or had not caught up). A busy round is judged by observation passes; a wake
for a Resource with nothing new since its last pass reads its mark and stops.
From 2.0.0-rc10, reminders are sent by the sweep, once a minute, not after
every pass. No judge metric carries a Workspace,
Resource, Rule or Observation. The queue series count announcements waiting
for a judge, including those handed back to wait a few seconds, and read 0 on
a process that does not reach the event bus. The uptimer_judge_passes_total,
uptimer_judge_pass_duration_seconds and queue series are there from the release
candidate after 2.0.0-rc7.
uptimer_judge_ready_total counts judge work on one Resource taken up, by the
same trigger as the passes. uptimer_judge_deferred_total counts work put
off to come back later: muted (an announcement for a Resource judged in the
last couple of seconds, handed back to the bus to wait), draining (a pass
ran but the Resource still has a backlog to judge) or failed (a pass that
failed). Muted and failed announcements come back through the bus; everything
put off is also looked at by the next sweep. A rising muted count during a
burst is normal; a failed count that keeps rising is not.
The ready and deferred series are there from 2.0.0-rc9.
From 2.0.0-rc14, a pass that ran without an error but left a backlog is counted in
uptimer_judge_passes_total as result="draining" (before, it was counted as error), and in
uptimer_judge_deferred_total as draining. It is neither a failure nor a completed pass:
ok counts only passes that caught up, and error only passes that failed. A Resource whose
backlog keeps it draining shows a steady draining count, not an error rate.
A pass that fails logs one line at ERROR, a judging pass failed, with trigger,
operation (read_pending, judge_evidence, judge_due, check_caught_up, clear_pending,
read_rules or judge_rule), the internal resource id, the internal rule id when the
failure was inside one Rule (never guessed for a Resource-level failure), and cause: one of
timeout, canceled, not_found, connection, transaction_done, database <SQLSTATE> or
other. The error text itself is not logged, because a database error can quote stored data.
One trigger and operation is logged at most once a minute; the next line for it carries
suppressed, the number of failures it left out. Every failed pass is still counted in
uptimer_judge_passes_total{result="error"} and uptimer_judge_deferred_total{reason="failed"},
logged or not.
source is where Observations arrived: api (API v3, single or batch, and an
Observation address) or worker (a worker’s report). The requests total
counts each request by how its Observations ended: accepted (all stored),
partial (some stored, some not, as in a batch with a bad item), rejected
(none stored, for the sender’s reasons, including a batch refused whole) or
failed (none stored, and the server failed at least one). The Observations
total counts each Observation: accepted only once it is stored, rejected
for the sender’s reasons (an unknown Resource, a bad field, a check the worker
is no longer assigned), failed when the server could not store it. A resent
Observation with the same id is stored once but counted again: these count
attempts, not unique Observations. Requests refused before they are read (no
key, or a key without the observe scope) are not counted here; they are in
http_request_duration_seconds. No ingest metric carries a Workspace,
Resource, Signal or Observation, and a scrape reads no database.
The two ingest series are there from 2.0.0-rc9.
uptimer_delivery_failures_total counts a message about an Incident to one
destination that went wrong, once for each operation that failed on it:
render (its body would not render for the destination, including reading
its transformation; nothing was sent), send (the destination refused it or
could not be reached) or record (its delivery record, sent, failed or held,
could not be written). A Send test from the destination editor is not
counted. uptimer_announce_failures_total counts a judge’s pass that stopped
on an error: announce (telling people what was just decided) or remind
(saying it again about what is still failing). A destination refusing a
message does not stop the pass, so a refused message counts once, as send,
and never also as announce. A held message (maintenance, no route, switched
off) is not a failure. Both are read on a process running judge, carry no
Workspace, Resource, destination, URL or error text, and appear after their
first failure; they are there from 2.0.0-rc9.
From 2.0.0-rc15, uptimer_confirmations_limited_total counts problem readings recorded
unconfirmed with reason limited: the Workspace already had 50 confirmations pending, so
nothing was asked and the original problem stands. Each such reading counts once, when the
judge’s pass that recorded it commits. Judging the same reading again, a reading that joins or
reuses another confirmation (it takes no place), and a pass that rolled back do not count. The
counter does not change the cap or ask again. It has no labels, is read only on a process
running judge, and starts at 0 when that process starts, so read it with increase(): for
example, increase(uptimer_confirmations_limited_total[1h]) > 0 says some problems in the last
hour went unconfirmed for lack of room.
The two totals exist so a quiet service can be told from a stopped one: a judge with no work and a judge that has died both report nothing otherwise.
A process running judge also removes delivery records older than 30 days,
once an hour.
State that must survive a restart
Everything lives under one directory, so a deployment mounts one volume.
| What | Default | Setting |
|---|---|---|
| data directory | /var/lib/uptimer (the image sets /data) | UPTIMER__DATA_DIR |
| database | sqlite3://<data>/uptimer.sqlite | UPTIMER__DB__DSN |
| session signing key | <data>/session.pem | UPTIMER__KEY_FILE |
| worker authority | <data>/worker-ca | UPTIMER__GRPC__CA_DIR |
| durable queue | <data>/eventbus-store | UPTIMER__EVENTBUS__STORE_DIR |
| worker certificate, key, CA | worker.pem, worker-key.pem, worker-ca.pem | UPTIMER__WORKER__CERT_FILE, __KEY_FILE, __CA_FILE |
A relative path in any of those settings is read under the data directory, and
an absolute one is used as given. A path typed on worker’s own --cert,
--key or --ca flag is used as typed, so a worker started by hand from a
directory of its own needs no data directory at all.
Restarting against the same directory reuses the database, the signing key — so sessions survive — the worker authority, the queued announcements, and a worker’s identity. None of them is recreated.
The worker authority is made on a first run, by the process that serves the worker API. Once certificates have been issued, a missing authority is a refusal to start instead: the trust every daemon holds is gone, and a new authority would refuse all of them while looking healthy. Back the directory up or keep issuing from a mounted volume.
PostgreSQL is the other supported backend: set UPTIMER__DB__DSN to a
postgres://… URL, and the data directory then holds only the key, the
authority and the queue.
Other settings
| Setting | Default | What it does |
|---|---|---|
UPTIMER__SERVICES | every service | which services a bare binary runs |
UPTIMER__AUTH__DEV | false | the development sign-in; see Sign-in |
UPTIMER__SQIDS__SALT | built-in | seeds the encoder behind every public id |
UPTIMER__GENERAL__SITE_URL | empty | how this installation is reached, so an alert can link back |
UPTIMER__EVENTBUS__NATS_URL | nats://localhost:4222 | where to reach the bus |
UPTIMER__REDIS__BIND_ADDR | localhost | where the cache listens |
UPTIMER__GRPC__HOSTS | localhost,127.0.0.1,::1 | names the worker API’s own certificate is good for |
UPTIMER__WORKER__EVERY | 30 | seconds between a daemon asking for its assignments |
Shutdown
On SIGINT or SIGTERM an all-services process stops producers first, waits
up to 15 seconds for the worker API to finish a report it is holding, and only
then drains the event bus. A report in flight is not lost to a restart.
What 1.x had and this does not
These are deliberate removals, not oversights. Nothing in this reference or in any example uses them.
| 1.x surface | Now |
|---|---|
--cfg YAML configuration | Environment and flags only. There is no configuration file. |
--delay | Removed. Startup ordering belongs to the orchestrator. |
Top-level grpc and server commands, availabilities service | serve --services grpc, serve --services judge. |
server init, worker init | worker-cert issues identities; dev sets a local run up. |
Zap log levels DEV and PROD | slog levels debug, info, warn and error; see Logs and error reporting. |
UPTIMER__SENTRY__PROFILES_SAMPLE_RATE | Not read. Sentry receives errors only; there is no tracing or profiling. |
/system/healthcheck | /livez and /readyz on the operations listener, for every process rather than only the UI. |
| Metrics named after Subjects, rules lists and availabilities | Gone with the model they described. The HTTP and gRPC duration series are kept, with their dimensions. |
OIDC settings named UPTIMER__SERVER__AUTH__OIDC__* | Renamed to UPTIMER__AUTH__OIDC__*; see Sign-in. 1.x OIDC account links are not read; see Moving from Uptimer 1.8 to 2.0. |
The 1.x public API (/api/v1/*, /api/v2/*) | Answered with one explicit 410 saying the 2.0 SDK replaced it. No legacy handler runs. |
| 1.x’s own database | Not read, not migrated, not converted; 2.0 refuses to start against it. Move with a fresh installation: Moving from Uptimer 1.8 to 2.0. |