Uptimer v2.0.0-preview documentation — it describes the release candidate, and v1.8.0 remains the current release. Commands here pull the preview image :2.0.0-rc15@sha256:945bd3a8fd42800445826df5d0968fd4c86d1d2a2e69f0fb41d897c66b0b0235. The Python SDK for this preview is a local wheel, not a PyPI release. Go to the current release (v1.8.0) →
Getting started › Triage pushed service signals

Triage pushed service signals

Publish a workerless Template, create Resources by your own keys, and turn pushed checks into distinct Incidents.

This fictional service example sends checks and load values to Uptimer as Observations. Uptimer decides between three verdicts for each Resource. No URL, Location, or managed worker is involved: every input is pushed.

RuleWhenAction
access_lossboth external probes report problem, the origin check and host health report ok, and the traffic ratio is below the server’s thresholdInvestigate the access path.
service_downthe origin check or host health reports problemInvestigate the service.
probe_issueat least one external probe reports problem, origin and host health report ok, and the traffic ratio is at or above the thresholdInspect the probe.

With every input known, at most one of the three holds. Each is its own Rule with its own Incidents and history.

You need a running Uptimer 2.0 preview (Quickstart), curl and jq, and a full API key from API keys → New key (Access: Full) of a Workspace editor or owner:

export API=http://localhost:8080/api/v3
export H="Authorization: Bearer <the token>"
# Which Workspace: the only one this key reaches, or the one UPTIMER_WORKSPACE
# names by id or name. See them: curl -s -H "$H" $API/workspaces | jq -r '.result[] | "\(.id)  \(.name)"'
WS=$(curl -s -H "$H" $API/workspaces | jq -er --arg want "${UPTIMER_WORKSPACE:-}" '
  .result as $all | ($all | map("\(.id) (\(.name))") | join(", ")) as $choices
  | if $want != "" then [$all[] | select(.id == $want or .name == $want)]
      | if length == 1 then .[0].id else error("no single Workspace is called \($want); choose one of: \($choices)") end
    elif ($all | length) == 1 then $all[0].id
    else error("this key reaches \($all | length) Workspaces; set UPTIMER_WORKSPACE to one of: \($choices)") end') || unset WS

1. Publish the Template

Download service-triage.json and publish it:

curl -s -H "$H" -X POST $API/workspaces/$WS/templates -d @service-triage.json | jq '.result.id, .error'
"service-triage@1"
null

It declares:

A revision never changes. To change the Template, publish it again with "version": 2. Servers you already created keep revision 1, and so do their Incidents. Publishing the same key and version twice answers 409. A definition Uptimer cannot judge answers 422 and stores nothing. The format is in the API v3 reference. Each Rule has at most 16 comparisons (two on the same Signal count as two) and at most 4 levels, counting the decision as level 1 and the final comparison as a level, and an action of 1–200 characters with no leading or trailing spaces. Rule limits shows a 4-level tree that is accepted and a 5-level one that is refused.

2. Create a server by its own key

Use the key your inventory already has. meta is optional here: load_threshold defaults to 0.5.

curl -s -H "$H" -X POST $API/workspaces/$WS/resources \
  -d '{"template": "service-triage", "key": "srv-0042", "name": "srv-0042", "meta": {"provider": "alpha", "load_threshold": 0.4}}' \
  | jq '.result | {key, rules: [.rules[] | {key, action}]}'

"template": "service-triage" takes the newest revision; "service-triage@1" pins one. A Resource that is refused, for example a threshold that is not a number, is not created at all.

3. Push what your database knows

Each round, send one Observation per Signal. A status Signal says ok or problem. The traffic ratio is your current traffic divided by the server’s baseline, as your database computes it, sent as value:

# push <server> <probe_a> <probe_b> <origin> <service_health> [ratio]
push() {
  send() { curl -s -o /dev/null -H "$H" -X POST "$API/workspaces/$WS/resources/$1/observations" -d "$2"; }
  send "$1" "{\"signal\": \"probe_a\", \"state\": \"$2\"}"
  send "$1" "{\"signal\": \"probe_b\", \"state\": \"$3\"}"
  send "$1" "{\"signal\": \"origin\", \"state\": \"$4\"}"
  send "$1" "{\"signal\": \"service_health\", \"state\": \"$5\"}"
  if [ -n "$6" ]; then
    send "$1" "{\"signal\": \"load_ratio\", \"state\": \"ok\", \"value\": $6}"
  else
    send "$1" '{"signal": "load_ratio", "state": "no_data"}'
  fi
}

push srv-0042 problem problem ok ok 0.1     # both probes fail, the host is fine, traffic is gone

With many servers, send each round in batches instead, from the release candidate after 2.0.0-rc7: up to 500 Observations per request, for any servers, one result per item. Give each item an id (here: round, server, Signal) so sending the batch again stores nothing twice:

curl -s -H "$H" -X POST $API/workspaces/$WS/observations -d '{"observations": [
  {"resource": "srv-0042", "signal": "probe_a", "state": "problem", "id": "r118-srv-0042-probe_a"},
  {"resource": "srv-0042", "signal": "origin",  "state": "ok",      "id": "r118-srv-0042-origin"},
  {"resource": "srv-0043", "signal": "probe_a", "state": "ok",      "id": "r118-srv-0043-probe_a"}
]}' | jq -c '.result | {accepted, rejected, rejected_items: [.results[] | select(.status == "rejected") | {index, error: .error.message}]}'

A rejected item (an unknown server, a state that is not ok, problem or no_data) carries the same error a single push would, and the others are stored anyway. The limits and the answer are in the API v3 reference; the Python SDK sends one with ws.resources.observe_batch(items).

An id is at most 64 characters, counted as characters, not bytes (from 2.0.0-rc14). Leading and trailing spaces are dropped first. A longer one is never cut or replaced: a single push answers 422 with error_type validation and details.field id, and nothing is created for it, not even a new server or Signal. In a batch, the request still answers 200. Only that item is rejected, with the same error, and the other items are stored:

{"index": 1, "status": "rejected",
 "error": {"code": 1422, "error_type": "validation", "message": "An Observation id is at most 64 characters.", "details": {"field": "id"}}}

Before 2.0.0-rc14, PostgreSQL refused a longer id with a server error (500).

Sending an id that is already stored again stores nothing and counts as no new reading, even though it is accepted. Do not give the same check a new id each round just to look fresh: each new id is a new reading, and it counts in windows and baselines.

What an accepted push proves

202, or accepted for a batch item, means the Observation is stored. It does not mean the Rules have judged it yet, and a batch has no receipt for when all its items are judged. Read the result from the Resource or its Incidents, as below. The judge’s own metrics are about the whole installation: uptimer_judge_queue_length at 0 or a quiet error counter is not a receipt for your batch. See Judging for what each series counts: from 2.0.0-rc14, a pass that is still working through a backlog counts as draining, not error, and a failed pass logs one bounded line.

Uptimer judges each push as it arrives; the pushes that land while a server is being judged are judged together about three seconds later. From the release candidate after 2.0.0-rc7, a measured burst of up to 300 servers × 5 Signals, pushed back to back, was judged within about 3 seconds at the median and under 5 seconds for the slowest. At 1,000 servers × 5 Signals the median was about 9 seconds and the slowest about 12: a larger burst takes longer, so poll rather than wait a fixed time. Then read the result:

sleep 5
curl -s -H "$H" $API/workspaces/$WS/resources/srv-0042/incidents \
  | jq '.result[] | {rule, action, lifecycle, confirmation, explanation}'
{
  "rule": "access_loss",
  "action": "Investigate the access path.",
  "lifecycle": "open",
  "confirmation": "unconfirmed",
  "explanation": "probe_a problem, probe_b problem, origin ok, service_health ok, load_ratio 0.1"
}

It is confirmed once the trouble has held for 600 seconds; keep pushing every round. When the evidence turns healthy and stays so for 600 seconds, it closes.

In the UI, a server you push to shows pushed under Checked from on the Resources list, and each of its Observations reads pushed in the Observation log: it came from your script, not from a Location. A Resource a worker runs counts its distinct Locations instead (from the release candidate after 2.0.0-rc7).

Say why

From the release candidate after 2.0.0-rc2, an Observation can carry a short reason: what your check saw, in plain text. A Rule that decides on that reading keeps it with the recorded evidence, and the alert repeats it:

curl -s -o /dev/null -H "$H" -X POST $API/workspaces/$WS/resources/srv-0042/observations \
  -d '{"signal": "probe_a", "state": "problem", "reason": "TLS handshake timeout from probe-eu-1"}'

The Incident’s history then shows "reason": "TLS handshake timeout from probe-eu-1" on that input, the Incident page says Sender said, when this opened: probe_a: …, and a webhook or Slack alert adds Reason: probe_a: TLS handshake timeout from probe-eu-1. Past 200 characters a reason is cut to 199 and “…”; line breaks become spaces; a URL in it loses any user name, password and credential-like query value. A reading sent without a reason simply shows none: the evidence has no reason field and the alert no Reason line. A later Observation never changes a recorded reason.

The other two verdicts:

for server in srv-0043 srv-0044; do
  curl -s -o /dev/null -H "$H" -X POST $API/workspaces/$WS/resources \
    -d "{\"template\": \"service-triage\", \"key\": \"$server\", \"name\": \"$server\"}"
done
push srv-0043 problem problem problem ok 0.1   # service_down: the origin check fails
push srv-0044 problem ok ok ok 0.9             # probe_issue: traffic is normal
sleep 5
for server in srv-0043 srv-0044; do
  curl -s -H "$H" $API/workspaces/$WS/resources/$server/incidents | jq -c '.result[] | {rule, action}'
done
{"rule":"service_down","action":"Investigate the service."}
{"rule":"probe_issue","action":"Inspect the probe."}

The alert a destination receives names the server, the verdict and the action, for example srv-0042: access_loss — Investigate the access path., with the reading above as its error. It is not a full record of every input value.

Require several traffic readings

One low traffic reading can be noise. Revision 2 of the Template, service-triage-counted.json, adds a count to both traffic comparisons:

{"signal": "load_ratio", "field": "value", "operator": "lt", "operand": {"meta": "load_threshold"},
 "min_count": 3, "within_seconds": 900}

Each comparison looks at the latest three distinct traffic Observations from the last 900 seconds. If all three are below the threshold, access_loss’s comparison holds; if all three are at or above it, probe_issue’s does. If the three are mixed, fewer than three, or one of them is no_data or has no value, both are unknown, and neither Rule opens or recovers. Both Rules read the same three readings, so they never hold together: three low readings followed by three high ones is a probe issue, and alternating readings are neither. The same report sent twice adds no reading, and a reading older than 900 seconds leaves the set. The waits stay as they were: access_loss is confirmed 600 seconds after the third low reading opened it.

curl -s -H "$H" -X POST $API/workspaces/$WS/templates -d @service-triage-counted.json | jq '.result.id'
curl -s -o /dev/null -H "$H" -X POST $API/workspaces/$WS/resources \
  -d '{"template": "service-triage@2", "key": "srv-0050", "name": "srv-0050"}'

push srv-0050 problem problem ok ok 0.1
sleep 5
curl -s -H "$H" $API/workspaces/$WS/resources/srv-0050/incidents | jq '.result | length'

push srv-0050 problem problem ok ok 0.1; sleep 2
push srv-0050 problem problem ok ok 0.1
sleep 5
curl -s -H "$H" $API/workspaces/$WS/resources/srv-0050/incidents | jq -c '.result[] | {rule, action, confirmation}'
"service-triage@2"
0
{"rule":"access_loss","action":"Investigate the access path.","confirmation":"unconfirmed"}

With the Python SDK, ws.templates.publish(manifest) publishes a Template and ws.resources.observe(...) pushes; the SDK’s examples/03_field_triage_counted.py runs the same gate.

Flag low_activity servers

A server can pass every check and still carry far less traffic than its peers. Revision 3, service-triage-low_activity.json, adds a peer_load_ratio Signal and a fourth Rule, low_activity:

RuleWhenAction
low_activityboth external probes, origin and host health report ok, and the latest three peer ratios within 1800 seconds are below the server’s peer_load_threshold (default 0.3), for 1800 secondsReview service demand when time permits.

Your system computes the peer ratio: this server’s traffic divided by the median traffic of its peers. Uptimer never sees the peers or the raw traffic. If the peer set is too small or the median is not valid, push no_data for peer_load_ratio; a low_activity verdict needs three valid ratios.

low_activity never holds together with access_loss, service_down or probe_issue: it needs every check and the host healthy. It has its own Incident and history, and recovers once the latest three ratios are at or above the threshold for another 1800 seconds. Keep pushing every round: a gap longer than the window leaves the count short, which is unknown and pauses recovery.

curl -s -H "$H" -X POST $API/workspaces/$WS/templates -d @service-triage-low_activity.json | jq '.result.id'
curl -s -o /dev/null -H "$H" -X POST $API/workspaces/$WS/resources \
  -d '{"template": "service-triage@3", "key": "srv-0060", "name": "srv-0060"}'

for round in 1 2 3; do
  push srv-0060 ok ok ok ok 1.0
  curl -s -o /dev/null -H "$H" -X POST $API/workspaces/$WS/resources/srv-0060/observations \
    -d '{"signal": "peer_load_ratio", "state": "ok", "value": 0.2}'
  sleep 2
done
sleep 5
curl -s -H "$H" $API/workspaces/$WS/resources/srv-0060/incidents | jq -c '.result[] | {rule, action, confirmation}'
"service-triage@3"
{"rule":"low_activity","action":"Review service demand when time permits.","confirmation":"unconfirmed"}

It is confirmed, and announced, once it has held for 1800 seconds. Alerts go to the same destination as the other verdicts unless a Rule names its own (below).

Compare load with its own past

From the release candidate after 2.0.0-rc2, Uptimer can compute the “usual” itself: push the service’s raw load, and a Rule compares the newest reading with a share of the median of that service’s own load over the last days. Revision 4, service-triage-history.json, replaces load_ratio with a raw load Signal. access_loss and probe_issue compare it with load_share (default 0.5) of its own 7-day median, once at least 12 earlier readings exist:

{"signal": "load", "field": "value", "operator": "lt", "operand": {"meta": "load_share"},
 "baseline": {"days": 7, "min_samples": 12}}

The median reads only this service’s own load: earlier numeric readings observed in the last 7 days, by when they were observed. The newest reading, later ones, no_data and readings without a value are left out, and the same report sent twice counts once. Fewer than 12 readings, a median of 0, or a newest reading that is no_data or has no value is unknown, so it can neither open access_loss nor prove a recovery. days is 1 to 30 and min_samples 1 to 10000; a baseline comparison cannot also take min_count. Peer medians stay in your system, as for low_activity.

curl -s -H "$H" -X POST $API/workspaces/$WS/templates -d @service-triage-history.json | jq '.result.id'
curl -s -o /dev/null -H "$H" -X POST $API/workspaces/$WS/resources \
  -d '{"template": "service-triage@4", "key": "srv-0070", "name": "srv-0070"}'

# Twelve earlier readings, one every 12 hours: the usual is about 100.
for i in $(seq 1 12); do
  curl -s -o /dev/null -H "$H" -X POST $API/workspaces/$WS/resources/srv-0070/observations \
    -d "{\"signal\": \"load\", \"state\": \"ok\", \"value\": 100, \"at\": \"$(date -u -d "-$((i * 12)) hours" +%FT%TZ)\"}"
done
# Now: both probes fail, origin and service are fine, and load is 40.
for signal in probe_a probe_b; do
  curl -s -o /dev/null -H "$H" -X POST $API/workspaces/$WS/resources/srv-0070/observations \
    -d "{\"signal\": \"$signal\", \"state\": \"problem\"}"
done
for signal in origin service_health; do
  curl -s -o /dev/null -H "$H" -X POST $API/workspaces/$WS/resources/srv-0070/observations \
    -d "{\"signal\": \"$signal\", \"state\": \"ok\"}"
done
curl -s -o /dev/null -H "$H" -X POST $API/workspaces/$WS/resources/srv-0070/observations \
  -d '{"signal": "load", "state": "ok", "value": 40}'
sleep 5
INC=$(curl -s -H "$H" $API/workspaces/$WS/resources/srv-0070/incidents | jq -r '.result[0].id')
curl -s -H "$H" $API/workspaces/$WS/incidents/$INC \
  | jq -c '.result | {rule, explanation}, (.history[0].evidence.inputs[] | select(.signal == "load") | .baseline)'
"service-triage@4"
{"rule":"access_loss","explanation":"probe_a problem, probe_b problem, origin ok, service_health ok, load 40 (7d median 100 of 12 readings)"}
{"days":7,"required":12,"samples":12,"median":100}

Load of 90 instead would be probe_issue: at least half the usual means traffic still arrives, so the probes are wrong. The waits are unchanged: access_loss is confirmed after 600 seconds.

Send each verdict to its own destination

From the release candidate after 2.0.0-rc2, a Rule can name the destination its alerts go to. Page on-call for access_loss and service_down, and send probe_issue and low_activity somewhere quieter. Create both under Settings → Destinations; each destination’s page shows its id. Add it to the Rule in your Template, then publish the next revision:

{"key": "access_loss", "action": "Investigate the access path.", "destination": "<on-call destination id>",
 "wait": {"confirm_after": 600, "recover_after": 600}, "decision": {"all": [ … ]}}

The confirmed problem, the reminders, a no-data message and the recovery of that Rule’s Incidents all go to its destination and nowhere else. A Rule without a destination follows the server’s own destination, then the Workspace default, as before. Publishing a Rule with a destination of another Workspace, or one that does not exist, answers 422. If you switch the destination off later, that Rule’s alerts are held (each Incident’s deliveries say destination_disabled); if you delete it, they fall back to the default. Where an alert goes never changes when an Incident opens, confirms or closes. In 2.0.0-rc2, every verdict goes to the default destination.

Route by Rule and transition

From the release candidate after 2.0.0-rc4, a Rule can use routes instead of destination to send its Incidents to several destinations, choose which transitions each one gets, or send nothing at all. Page on-call in Slack the moment access_loss opens, tell an ops webhook once it is confirmed, and keep probe_issue in the Uptimer UI only:

{"key": "access_loss", "action": "Investigate the access path.",
 "routes": [
   {"destination": "<on-call Slack id>", "on": ["opened", "problem", "recovery", "acknowledged"]},
   {"destination": "<ops webhook id>"}],
 "wait": {"confirm_after": 600, "recover_after": 600}, "decision": {"all": [ … ]}}

{"key": "probe_issue", "action": "Check the probe host.", "routes": [], … }

The transitions are opened, problem (the confirmed problem and its reminders), no_data, recovery and acknowledged; a route without on gets problem, no_data and recovery, what a destination always got. A Rule with routes never falls back to the server’s or the Workspace default destination, so a new Rule published with "routes": [] cannot page anyone. What it leaves out shows in each Incident’s deliveries as not_routed, apart from no_destination (a Workspace with nowhere to send).

Slack on opened gets the opening before the 600-second wait ends. A webhook routed opened still needs Also send when an Incident opens, below. acknowledged is sent once when somebody takes the Incident on. A Rule cannot have both destination and routes.

Script the destinations too

From the release candidate after 2.0.0-rc5, a full API key creates the destinations and the Template names them, so no id is copied by hand:

curl -s -H "$H" -X POST $API/workspaces/$WS/destinations \
  -d '{"name": "oncall", "type": "slack", "url": "https://hooks.slack.com/services/…"}' | jq '.result.id, .error'
"routes": [{"destination": {"name": "oncall"}, "on": ["opened", "problem", "recovery", "acknowledged"]},
           {"destination": {"name": "ops"}}]

Publishing looks the name up in that Workspace and stores its id, so the same manifest works in every Workspace that has an oncall and an ops. A name it cannot find, or that two destinations answer to, is refused with 422. The API never shows a destination’s URL back. POST …/destinations/{id}/test answers delivered or a reason code, and GET …/destinations/{id}/deliveries pages what was sent there. A destination is not deleted while the newest revision of a Template, or an active server’s Rules, still route to it: the 422 lists them. The Python SDK (ws.destinations) and MCP (create_destination, …) do the same.

Get the alert with its evidence

Add a webhook under Settings → Destinations (type Webhook, your endpoint’s URL) and make it the default. When a verdict is confirmed, the webhook body carries an incident object: the Rule (access_loss, service_down or probe_issue), its action, the server’s key and fields, and the evidence that transition recorded — each Signal’s status or the traffic ratio it read, or why it had none:

"evidence": {"inputs": [
  {"signal": "probe_a", "status": "problem", "at": "…"},
  {"signal": "origin", "status": "problem", "at": "…"},
  {"signal": "service_health", "unresolved": "the sender reported no_data"}],
 "omitted": 0, "truncated": false}

The evidence is recorded when the transition is decided and does not change when newer Observations arrive; it is not a list of every Observation. Labels you send, Observation bodies and secret fields are not included. The full shape is in the API v3 reference. A Slack destination gets the same alert text without the incident object.

access_loss waits 600 seconds before it is confirmed. If your automation should start before that, tick Also send when an Incident opens on the webhook (from the release candidate after 2.0.0-rc3). The webhook then also gets one opened event as soon as the Incident opens: the same incident object with "transition": "opened", "lifecycle": "open" and "confirmation": "unconfirmed", and the evidence it opened on. The confirmed event still follows after 600 seconds; an opening is never sent once the Incident is confirmed. A new reading while it waits sends no second opening, and an Incident that clears before it is confirmed sends nothing more. A Slack destination gets openings only where a Rule routes them (above).

Find servers by field, and retire one

Lists take the Template and its fields as filters. Only srv-0042 was created with a provider:

curl -s -H "$H" "$API/workspaces/$WS/resources?template=service-triage&meta.provider=alpha" | jq -r '.result[].key'
srv-0042

From the release candidate after 2.0.0-rc7, a server can also carry labels of its own, set without publishing a new Template revision. They are for finding servers; no Rule reads them and no alert carries them:

curl -s -H "$H" -X PATCH $API/workspaces/$WS/resources/srv-0042 -d '{"labels": {"env": "prod"}}' | jq -c '.result.labels'
curl -s -H "$H" "$API/workspaces/$WS/incidents?label.env=prod&lifecycle=open" | jq -r '.result[].resource.key'

null removes a label. On the Resources list, a label is a link that shows only the servers carrying it.

When a server leaves your inventory, archive it. Archiving is not deleting: the server keeps its key and its history, leaves the active list, stops accepting Observations, and its open Incidents close as resource_archived (not as a recovery). Its key cannot be reused, and there is no restore. From the release candidate after 2.0.0-rc2 the UI does the same: Archive on the server’s row or page (Quickstart).

curl -s -H "$H" -X POST $API/workspaces/$WS/resources/srv-0044/archive | jq -r '.result.archived_at != null'
curl -s -H "$H" -X POST $API/workspaces/$WS/resources/srv-0044/observations \
  -d '{"signal": "origin", "state": "ok"}' | jq -r '.error.message'
curl -s -H "$H" "$API/workspaces/$WS/resources?template=service-triage" | jq -r '[.result[].key] | join(" ")'
curl -s -H "$H" "$API/workspaces/$WS/incidents?template=service-triage&resource_state=archived" \
  | jq -c '.result[] | {resource: .resource.key, rule, lifecycle, closed_reason}'
true
Resource srv-0044 is archived and accepts no Observations.
srv-0042 srv-0043 srv-0050
{"resource":"srv-0044","rule":"probe_issue","lifecycle":"closed","closed_reason":"resource_archived"}

Lists page by limit and next_cursor; state=archived or state=all lists archived servers. With the Python SDK: ws.resources.list(template="service-triage", meta={"provider": "alpha"}), ws.resources.archive("srv-0044") and ws.incidents.list(template="service-triage", resource_state="archived").

Move a server to a new revision

From the release candidate after 2.0.0-rc2, a server can follow another published revision without changing its key: from its page (Actions → Rebind), or with the API.

curl -s -H "$H" -X POST $API/workspaces/$WS/resources/srv-0042/rebind \
  -d '{"template": "service-triage@4", "meta": {"provider": "alpha"}}' | jq -c '.result | {key, template}'
{"key":"srv-0042","template":"service-triage@4"}

The server keeps its id, key and history. Its Signals and Rules become the new revision’s. Its open Incidents of the old revision close as rule_removed, not as a recovery, and stay readable with the Observations they used. Evidence for a Signal the new revision does not declare is refused. Nothing changes if the revision or an answer is refused, and an archived server cannot be rebound. Publishing a new revision never moves a server on its own.

There is no rollout across servers. Publishing only makes a revision available, and each server moves when you rebind it, one at a time. Servers you have not rebound keep their revision and its Rules.

What carries across a rebind, from 2.0.0-rc10:

Any number of probes

From the release candidate after 2.0.0-rc3, one Template can serve a server with one probe and another with three. all_known judges only the probes that have reported a fresh ok or problem, and stays unknown until min_known of them have:

{"key": "access_loss", "action": "Investigate the access path.",
 "wait": {"confirm_after": 600, "recover_after": 600},
 "decision": {"all": [
   {"all_known": [
     {"signal": "probe_a", "field": "status", "operator": "eq", "operand": "problem"},
     {"signal": "probe_b", "field": "status", "operator": "eq", "operand": "problem"},
     {"signal": "probe_c", "field": "status", "operator": "eq", "operand": "problem"}],
    "min_known": 1},
   {"signal": "origin", "field": "status", "operator": "eq", "operand": "ok"}]}}

A server that only has probe_a is judged on probe_a; one with all three needs all three failing. A probe that never reports, goes quiet, or says no_data is left out rather than counted as healthy, and with no probe known the Rule decides nothing. To test one input alone, {"signal": "probe_b", "field": "known"} is true while probe_b has a fresh ok or problem.

When data is missing

If host health cannot be read, or the traffic ratio has no confidence, send no_data for it (or send the Observation without a value). Do not send ok to fill a gap.

A Rule’s explanation says which of these it is, input by input: unresolved (no_data) when your script sent no_data, unresolved (nothing has been reported here) for a Signal that never reported, and unresolved (nothing reported since …) for one that went quiet. From the release candidate after 2.0.0-rc7, the explanation follows the newest of these even while the Rule’s result stays unknown, and an Incident page counts a reported no_data as without data, not as failing.

To find servers whose scripts went quiet, list them by cause: GET …/resources?missing=stale (or never_reported, reported_no_data; Missing Signals). That is a filter you poll, not an alert: no Rule or notification fires because a whole source stopped sending.

When a check is confirmed again

A Signal with a confirm policy asks Uptimer’s workers to check a problem again before its Rules believe it (Confirming a problem). With pushed checks that report every round:

When a problem flaps

A check that goes back and forth opens and closes Incidents in turn. The Rule’s waits are the setting for this: confirm_after keeps an Incident unconfirmed, with no problem alert, until the trouble has lasted, and recover_after keeps it open until health has lasted. A longer recover_after merges short dips into one Incident, but a real recovery then also closes that much later. A closed Incident is never reopened: trouble after it closes opens a new one.

What stays in your database

Peer medians and peer sets stay where they are today: push only the derived peer ratio. From the release candidate after 2.0.0-rc2, a service’s own usual load can instead be computed by Uptimer from the raw readings you push (Compare load with its own past).

Uptimer’s own database keeps every Observation it accepts, not just what a Rule reads. It keeps them for archived servers and across rebinds as well. There is no retention or compaction setting, and nothing deletes old Observations. Storage grows with servers × Signals × rounds for as long as you push. Plan disk for that growth, watch it, and size your backups for it (Database and migrations). This preview has no tested procedure for removing old Observations.

How far back you can read

WhereReadsOlder than that
API GET …/resources/{r}/observations?signal&limitthe newest, limit 1–200, default 50no cursor: it reads only the newest
Python SDK ws.resources.observations(r, signal=…, limit=50)the same API read, default 50the same
MCP list_observationsthe newest, limit 1–50, default 20the same
UI: the Resource’s Observation logone UTC day at a time, optionally one Signal, paged with Older and Newerpick an earlier day

The API, SDK and MCP reads are for recent context. To look at an earlier day, use the Observation log in the UI (Read a problem). An Incident’s history keeps the evidence each transition read, however old.