Triage pushed service signals
Publish a workerless Template, create Resources by your own keys, and turn pushed checks into distinct Incidents.
This fictional service example sends checks and load values to Uptimer as Observations. Uptimer decides between three verdicts for each Resource. No URL, Location, or managed worker is involved: every input is pushed.
| Rule | When | Action |
|---|---|---|
access_loss | both external probes report problem, the origin check and host health report ok, and the traffic ratio is below the server’s threshold | Investigate the access path. |
service_down | the origin check or host health reports problem | Investigate the service. |
probe_issue | at least one external probe reports problem, origin and host health report ok, and the traffic ratio is at or above the threshold | Inspect the probe. |
With every input known, at most one of the three holds. Each is its own Rule with its own Incidents and history.
You need a running Uptimer 2.0 preview (Quickstart),
curl and jq, and a full API key from API keys → New key (Access: Full) of a
Workspace editor or owner:
export API=http://localhost:8080/api/v3
export H="Authorization: Bearer <the token>"
# Which Workspace: the only one this key reaches, or the one UPTIMER_WORKSPACE
# names by id or name. See them: curl -s -H "$H" $API/workspaces | jq -r '.result[] | "\(.id) \(.name)"'
WS=$(curl -s -H "$H" $API/workspaces | jq -er --arg want "${UPTIMER_WORKSPACE:-}" '
.result as $all | ($all | map("\(.id) (\(.name))") | join(", ")) as $choices
| if $want != "" then [$all[] | select(.id == $want or .name == $want)]
| if length == 1 then .[0].id else error("no single Workspace is called \($want); choose one of: \($choices)") end
elif ($all | length) == 1 then $all[0].id
else error("this key reaches \($all | length) Workspaces; set UPTIMER_WORKSPACE to one of: \($choices)") end') || unset WS
1. Publish the Template
Download service-triage.json
and publish it:
curl -s -H "$H" -X POST $API/workspaces/$WS/templates -d @service-triage.json | jq '.result.id, .error'
"service-triage@1"
null
It declares:
- a field
load_threshold(a number, default0.5): below it, traffic counts as gone; - five pushed Signals:
probe_a,probe_b,origin,service_healthandload_ratio, each expected every 300 seconds; - the three Rules above, each with a
decision, awaitand anaction.access_losswaits 600 seconds before it is confirmed and 600 seconds of health before it closes;service_downconfirms at once and closes after 300 seconds;probe_issueconfirms and closes at once.
A revision never changes. To change the Template, publish it again with "version": 2.
Servers you already created keep revision 1, and so do their Incidents. Publishing the same
key and version twice answers 409. A definition Uptimer cannot judge answers 422 and stores
nothing. The format is in the API v3 reference.
Each Rule has at most 16 comparisons (two on the same Signal count as two) and at most 4 levels,
counting the decision as level 1 and the final comparison as a level, and an action of 1–200
characters with no leading or trailing spaces. Rule limits
shows a 4-level tree that is accepted and a 5-level one that is refused.
2. Create a server by its own key
Use the key your inventory already has. meta is optional here: load_threshold defaults to
0.5.
curl -s -H "$H" -X POST $API/workspaces/$WS/resources \
-d '{"template": "service-triage", "key": "srv-0042", "name": "srv-0042", "meta": {"provider": "alpha", "load_threshold": 0.4}}' \
| jq '.result | {key, rules: [.rules[] | {key, action}]}'
"template": "service-triage" takes the newest revision; "service-triage@1" pins one. A Resource
that is refused, for example a threshold that is not a number, is not created at all.
3. Push what your database knows
Each round, send one Observation per Signal. A status Signal says ok or problem. The traffic
ratio is your current traffic divided by the server’s baseline, as your database computes it,
sent as value:
# push <server> <probe_a> <probe_b> <origin> <service_health> [ratio]
push() {
send() { curl -s -o /dev/null -H "$H" -X POST "$API/workspaces/$WS/resources/$1/observations" -d "$2"; }
send "$1" "{\"signal\": \"probe_a\", \"state\": \"$2\"}"
send "$1" "{\"signal\": \"probe_b\", \"state\": \"$3\"}"
send "$1" "{\"signal\": \"origin\", \"state\": \"$4\"}"
send "$1" "{\"signal\": \"service_health\", \"state\": \"$5\"}"
if [ -n "$6" ]; then
send "$1" "{\"signal\": \"load_ratio\", \"state\": \"ok\", \"value\": $6}"
else
send "$1" '{"signal": "load_ratio", "state": "no_data"}'
fi
}
push srv-0042 problem problem ok ok 0.1 # both probes fail, the host is fine, traffic is gone
With many servers, send each round in batches instead, from the release candidate after
2.0.0-rc7: up to 500 Observations per request, for any servers, one result per item. Give each
item an id (here: round, server, Signal) so sending the batch again stores nothing twice:
curl -s -H "$H" -X POST $API/workspaces/$WS/observations -d '{"observations": [
{"resource": "srv-0042", "signal": "probe_a", "state": "problem", "id": "r118-srv-0042-probe_a"},
{"resource": "srv-0042", "signal": "origin", "state": "ok", "id": "r118-srv-0042-origin"},
{"resource": "srv-0043", "signal": "probe_a", "state": "ok", "id": "r118-srv-0043-probe_a"}
]}' | jq -c '.result | {accepted, rejected, rejected_items: [.results[] | select(.status == "rejected") | {index, error: .error.message}]}'
A rejected item (an unknown server, a state that is not ok, problem or no_data) carries the
same error a single push would, and the others are stored anyway. The limits and the answer are in
the API v3 reference; the Python SDK
sends one with ws.resources.observe_batch(items).
An id is at most 64 characters, counted as characters, not bytes (from 2.0.0-rc14). Leading
and trailing spaces are dropped first. A longer one is never cut or replaced: a single push answers
422 with error_type validation and details.field id, and nothing is created for it, not
even a new server or Signal. In a batch, the request still answers 200. Only that item is
rejected, with the same error, and the other items are stored:
{"index": 1, "status": "rejected",
"error": {"code": 1422, "error_type": "validation", "message": "An Observation id is at most 64 characters.", "details": {"field": "id"}}}
Before 2.0.0-rc14, PostgreSQL refused a longer id with a server error (500).
Sending an id that is already stored again stores nothing and counts as no new reading, even
though it is accepted. Do not give the same check a new id each round just to look fresh: each new
id is a new reading, and it counts in windows and baselines.
What an accepted push proves
202, or accepted for a batch item, means the Observation is stored. It does not mean the Rules
have judged it yet, and a batch has no receipt for when all its items are judged. Read the result
from the Resource or its Incidents, as below. The judge’s own metrics are about the whole
installation: uptimer_judge_queue_length at 0 or a quiet error counter is not a receipt for your
batch. See Judging for what each series counts: from
2.0.0-rc14, a pass that is still working through a backlog counts as draining, not error,
and a failed pass logs one bounded line.
Uptimer judges each push as it arrives; the pushes that land while a server is being judged are
judged together about three seconds later. From the release candidate after 2.0.0-rc7, a
measured burst of up to 300 servers × 5 Signals, pushed back to back, was judged within about
3 seconds at the median and under 5 seconds for the slowest. At 1,000 servers × 5 Signals the
median was about 9 seconds and the slowest about 12: a larger burst takes longer, so poll
rather than wait a fixed time. Then read the result:
sleep 5
curl -s -H "$H" $API/workspaces/$WS/resources/srv-0042/incidents \
| jq '.result[] | {rule, action, lifecycle, confirmation, explanation}'
{
"rule": "access_loss",
"action": "Investigate the access path.",
"lifecycle": "open",
"confirmation": "unconfirmed",
"explanation": "probe_a problem, probe_b problem, origin ok, service_health ok, load_ratio 0.1"
}
It is confirmed once the trouble has held for 600 seconds; keep pushing every round. When the evidence turns healthy and stays so for 600 seconds, it closes.
In the UI, a server you push to shows pushed under Checked from on the Resources list,
and each of its Observations reads pushed in the Observation log: it came from your script,
not from a Location. A Resource a worker runs counts its distinct Locations instead (from the
release candidate after 2.0.0-rc7).
Say why
From the release candidate after 2.0.0-rc2, an Observation can carry a short reason: what
your check saw, in plain text. A Rule that decides on that reading keeps it with the recorded
evidence, and the alert repeats it:
curl -s -o /dev/null -H "$H" -X POST $API/workspaces/$WS/resources/srv-0042/observations \
-d '{"signal": "probe_a", "state": "problem", "reason": "TLS handshake timeout from probe-eu-1"}'
The Incident’s history then shows "reason": "TLS handshake timeout from probe-eu-1" on that
input, the Incident page says Sender said, when this opened: probe_a: …, and a webhook or Slack
alert adds Reason: probe_a: TLS handshake timeout from probe-eu-1. Past 200 characters a reason
is cut to 199 and “…”; line breaks become spaces; a URL in it loses any user name, password and
credential-like query value. A reading sent without a reason simply shows none: the evidence has no
reason field and the alert no Reason line. A later Observation never changes a recorded reason.
The other two verdicts:
for server in srv-0043 srv-0044; do
curl -s -o /dev/null -H "$H" -X POST $API/workspaces/$WS/resources \
-d "{\"template\": \"service-triage\", \"key\": \"$server\", \"name\": \"$server\"}"
done
push srv-0043 problem problem problem ok 0.1 # service_down: the origin check fails
push srv-0044 problem ok ok ok 0.9 # probe_issue: traffic is normal
sleep 5
for server in srv-0043 srv-0044; do
curl -s -H "$H" $API/workspaces/$WS/resources/$server/incidents | jq -c '.result[] | {rule, action}'
done
{"rule":"service_down","action":"Investigate the service."}
{"rule":"probe_issue","action":"Inspect the probe."}
The alert a destination receives names the server, the verdict and the action, for example
srv-0042: access_loss — Investigate the access path., with the reading above as its error. It is not a
full record of every input value.
Require several traffic readings
One low traffic reading can be noise. Revision 2 of the Template,
service-triage-counted.json,
adds a count to both traffic comparisons:
{"signal": "load_ratio", "field": "value", "operator": "lt", "operand": {"meta": "load_threshold"},
"min_count": 3, "within_seconds": 900}
Each comparison looks at the latest three distinct traffic Observations from the last 900
seconds. If all three are below the threshold, access_loss’s comparison holds; if all three are at
or above it, probe_issue’s does. If the three are mixed, fewer than three, or one of them is
no_data or has no value, both are unknown, and neither Rule opens or recovers. Both Rules read
the same three readings, so they never hold together: three low readings followed by three high
ones is a probe issue, and alternating readings are neither. The same report sent twice adds
no reading, and a reading older than 900 seconds leaves the set. The waits stay as they were:
access_loss is confirmed 600 seconds after the third low reading opened it.
curl -s -H "$H" -X POST $API/workspaces/$WS/templates -d @service-triage-counted.json | jq '.result.id'
curl -s -o /dev/null -H "$H" -X POST $API/workspaces/$WS/resources \
-d '{"template": "service-triage@2", "key": "srv-0050", "name": "srv-0050"}'
push srv-0050 problem problem ok ok 0.1
sleep 5
curl -s -H "$H" $API/workspaces/$WS/resources/srv-0050/incidents | jq '.result | length'
push srv-0050 problem problem ok ok 0.1; sleep 2
push srv-0050 problem problem ok ok 0.1
sleep 5
curl -s -H "$H" $API/workspaces/$WS/resources/srv-0050/incidents | jq -c '.result[] | {rule, action, confirmation}'
"service-triage@2"
0
{"rule":"access_loss","action":"Investigate the access path.","confirmation":"unconfirmed"}
With the Python SDK, ws.templates.publish(manifest) publishes a Template and
ws.resources.observe(...) pushes; the SDK’s examples/03_field_triage_counted.py runs the
same gate.
Flag low_activity servers
A server can pass every check and still carry far less traffic than its peers. Revision 3,
service-triage-low_activity.json,
adds a peer_load_ratio Signal and a fourth Rule, low_activity:
| Rule | When | Action |
|---|---|---|
low_activity | both external probes, origin and host health report ok, and the latest three peer ratios within 1800 seconds are below the server’s peer_load_threshold (default 0.3), for 1800 seconds | Review service demand when time permits. |
Your system computes the peer ratio: this server’s traffic divided by the median traffic of its
peers. Uptimer never sees the peers or the raw traffic. If the peer set is too small or the
median is not valid, push no_data for peer_load_ratio; a low_activity verdict needs three valid ratios.
low_activity never holds together with access_loss, service_down or probe_issue: it needs every check
and the host healthy. It has its own Incident and history, and recovers once the latest three
ratios are at or above the threshold for another 1800 seconds. Keep pushing every round: a gap
longer than the window leaves the count short, which is unknown and pauses recovery.
curl -s -H "$H" -X POST $API/workspaces/$WS/templates -d @service-triage-low_activity.json | jq '.result.id'
curl -s -o /dev/null -H "$H" -X POST $API/workspaces/$WS/resources \
-d '{"template": "service-triage@3", "key": "srv-0060", "name": "srv-0060"}'
for round in 1 2 3; do
push srv-0060 ok ok ok ok 1.0
curl -s -o /dev/null -H "$H" -X POST $API/workspaces/$WS/resources/srv-0060/observations \
-d '{"signal": "peer_load_ratio", "state": "ok", "value": 0.2}'
sleep 2
done
sleep 5
curl -s -H "$H" $API/workspaces/$WS/resources/srv-0060/incidents | jq -c '.result[] | {rule, action, confirmation}'
"service-triage@3"
{"rule":"low_activity","action":"Review service demand when time permits.","confirmation":"unconfirmed"}
It is confirmed, and announced, once it has held for 1800 seconds. Alerts go to the same destination as the other verdicts unless a Rule names its own (below).
Compare load with its own past
From the release candidate after 2.0.0-rc2, Uptimer can compute the “usual” itself: push
the service’s raw load, and a Rule compares the newest reading with a share of the median of
that service’s own load over the last days. Revision 4,
service-triage-history.json,
replaces load_ratio with a raw load Signal. access_loss and probe_issue compare it
with load_share (default 0.5) of its own 7-day median, once at least 12 earlier readings
exist:
{"signal": "load", "field": "value", "operator": "lt", "operand": {"meta": "load_share"},
"baseline": {"days": 7, "min_samples": 12}}
The median reads only this service’s own load: earlier numeric readings observed in the
last 7 days, by when they were observed. The newest reading, later ones, no_data and
readings without a value are left out, and the same report sent twice counts once. Fewer than
12 readings, a median of 0, or a newest reading that is no_data or has no value is unknown,
so it can neither open access_loss nor prove a recovery. days is 1 to 30 and
min_samples 1 to 10000; a baseline comparison cannot also take min_count. Peer medians
stay in your system, as for low_activity.
curl -s -H "$H" -X POST $API/workspaces/$WS/templates -d @service-triage-history.json | jq '.result.id'
curl -s -o /dev/null -H "$H" -X POST $API/workspaces/$WS/resources \
-d '{"template": "service-triage@4", "key": "srv-0070", "name": "srv-0070"}'
# Twelve earlier readings, one every 12 hours: the usual is about 100.
for i in $(seq 1 12); do
curl -s -o /dev/null -H "$H" -X POST $API/workspaces/$WS/resources/srv-0070/observations \
-d "{\"signal\": \"load\", \"state\": \"ok\", \"value\": 100, \"at\": \"$(date -u -d "-$((i * 12)) hours" +%FT%TZ)\"}"
done
# Now: both probes fail, origin and service are fine, and load is 40.
for signal in probe_a probe_b; do
curl -s -o /dev/null -H "$H" -X POST $API/workspaces/$WS/resources/srv-0070/observations \
-d "{\"signal\": \"$signal\", \"state\": \"problem\"}"
done
for signal in origin service_health; do
curl -s -o /dev/null -H "$H" -X POST $API/workspaces/$WS/resources/srv-0070/observations \
-d "{\"signal\": \"$signal\", \"state\": \"ok\"}"
done
curl -s -o /dev/null -H "$H" -X POST $API/workspaces/$WS/resources/srv-0070/observations \
-d '{"signal": "load", "state": "ok", "value": 40}'
sleep 5
INC=$(curl -s -H "$H" $API/workspaces/$WS/resources/srv-0070/incidents | jq -r '.result[0].id')
curl -s -H "$H" $API/workspaces/$WS/incidents/$INC \
| jq -c '.result | {rule, explanation}, (.history[0].evidence.inputs[] | select(.signal == "load") | .baseline)'
"service-triage@4"
{"rule":"access_loss","explanation":"probe_a problem, probe_b problem, origin ok, service_health ok, load 40 (7d median 100 of 12 readings)"}
{"days":7,"required":12,"samples":12,"median":100}
Load of 90 instead would be probe_issue: at least half the usual means traffic still
arrives, so the probes are wrong. The waits are unchanged: access_loss is confirmed after
600 seconds.
Send each verdict to its own destination
From the release candidate after 2.0.0-rc2, a Rule can name the destination its alerts go
to. Page on-call for access_loss and service_down, and send probe_issue and low_activity somewhere
quieter. Create both under Settings → Destinations; each destination’s page shows its
id. Add it to the Rule in your Template, then publish the next revision:
{"key": "access_loss", "action": "Investigate the access path.", "destination": "<on-call destination id>",
"wait": {"confirm_after": 600, "recover_after": 600}, "decision": {"all": [ … ]}}
The confirmed problem, the reminders, a no-data message and the recovery of that Rule’s
Incidents all go to its destination and nowhere else. A Rule without a destination follows
the server’s own destination, then the Workspace default, as before. Publishing a Rule with
a destination of another Workspace, or one that does not exist, answers 422. If you switch
the destination off later, that Rule’s alerts are held (each Incident’s deliveries say
destination_disabled); if you delete it, they fall back to the default. Where an alert goes
never changes when an Incident opens, confirms or closes. In 2.0.0-rc2, every verdict goes
to the default destination.
Route by Rule and transition
From the release candidate after 2.0.0-rc4, a Rule can use routes instead of destination
to send its Incidents to several destinations, choose which transitions each one gets, or
send nothing at all. Page on-call in Slack the moment access_loss opens, tell an ops
webhook once it is confirmed, and keep probe_issue in the Uptimer UI only:
{"key": "access_loss", "action": "Investigate the access path.",
"routes": [
{"destination": "<on-call Slack id>", "on": ["opened", "problem", "recovery", "acknowledged"]},
{"destination": "<ops webhook id>"}],
"wait": {"confirm_after": 600, "recover_after": 600}, "decision": {"all": [ … ]}}
{"key": "probe_issue", "action": "Check the probe host.", "routes": [], … }
The transitions are opened, problem (the confirmed problem and its reminders), no_data,
recovery and acknowledged; a route without on gets problem, no_data and recovery,
what a destination always got. A Rule with routes never falls back to the server’s or the
Workspace default destination, so a new Rule published with "routes": [] cannot page
anyone. What it leaves out shows in each Incident’s deliveries as not_routed, apart from
no_destination (a Workspace with nowhere to send).
Slack on opened gets the opening before the 600-second wait ends. A webhook routed
opened still needs Also send when an Incident opens, below. acknowledged is sent once
when somebody takes the Incident on. A Rule cannot have both destination and routes.
Script the destinations too
From the release candidate after 2.0.0-rc5, a full API key creates the destinations and the
Template names them, so no id is copied by hand:
curl -s -H "$H" -X POST $API/workspaces/$WS/destinations \
-d '{"name": "oncall", "type": "slack", "url": "https://hooks.slack.com/services/…"}' | jq '.result.id, .error'
"routes": [{"destination": {"name": "oncall"}, "on": ["opened", "problem", "recovery", "acknowledged"]},
{"destination": {"name": "ops"}}]
Publishing looks the name up in that Workspace and stores its id, so the same manifest works in
every Workspace that has an oncall and an ops. A name it cannot find, or that two
destinations answer to, is refused with 422. The API never shows a destination’s URL back.
POST …/destinations/{id}/test answers delivered or a reason code, and
GET …/destinations/{id}/deliveries pages what was sent there. A destination is not deleted
while the newest revision of a Template, or an active server’s Rules, still route to it: the
422 lists them. The Python SDK (ws.destinations) and MCP (create_destination, …) do the
same.
Get the alert with its evidence
Add a webhook under Settings → Destinations (type Webhook, your endpoint’s URL) and make
it the default. When a verdict is confirmed, the webhook body carries an incident object:
the Rule (access_loss, service_down or probe_issue), its action, the server’s key and fields, and
the evidence that transition recorded — each Signal’s status or the traffic ratio it read, or
why it had none:
"evidence": {"inputs": [
{"signal": "probe_a", "status": "problem", "at": "…"},
{"signal": "origin", "status": "problem", "at": "…"},
{"signal": "service_health", "unresolved": "the sender reported no_data"}],
"omitted": 0, "truncated": false}
The evidence is recorded when the transition is decided and does not change when newer
Observations arrive; it is not a list of every Observation. Labels you send, Observation
bodies and secret fields are not included. The full shape is in the
API v3 reference. A Slack destination gets the
same alert text without the incident object.
access_loss waits 600 seconds before it is confirmed. If your automation should start before
that, tick Also send when an Incident opens on the webhook (from the release candidate after
2.0.0-rc3). The webhook then also gets one opened event as soon as the Incident opens: the
same incident object with "transition": "opened", "lifecycle": "open" and
"confirmation": "unconfirmed", and the evidence it opened on. The confirmed event still
follows after 600 seconds; an opening is never sent once the Incident is confirmed. A new
reading while it waits sends no second opening, and an Incident
that clears before it is confirmed sends nothing more. A Slack destination gets openings only
where a Rule routes them (above).
Find servers by field, and retire one
Lists take the Template and its fields as filters. Only srv-0042 was created with a provider:
curl -s -H "$H" "$API/workspaces/$WS/resources?template=service-triage&meta.provider=alpha" | jq -r '.result[].key'
srv-0042
From the release candidate after 2.0.0-rc7, a server can also carry labels of its own, set
without publishing a new Template revision. They are for finding servers; no Rule reads them and
no alert carries them:
curl -s -H "$H" -X PATCH $API/workspaces/$WS/resources/srv-0042 -d '{"labels": {"env": "prod"}}' | jq -c '.result.labels'
curl -s -H "$H" "$API/workspaces/$WS/incidents?label.env=prod&lifecycle=open" | jq -r '.result[].resource.key'
null removes a label. On the Resources list, a label is a link that shows only the servers
carrying it.
When a server leaves your inventory, archive it. Archiving is not deleting: the server keeps its
key and its history, leaves the active list, stops accepting Observations, and its open Incidents
close as resource_archived (not as a recovery). Its key cannot be reused, and there is no
restore. From the release candidate after 2.0.0-rc2 the UI does the same: Archive on the
server’s row or page (Quickstart).
curl -s -H "$H" -X POST $API/workspaces/$WS/resources/srv-0044/archive | jq -r '.result.archived_at != null'
curl -s -H "$H" -X POST $API/workspaces/$WS/resources/srv-0044/observations \
-d '{"signal": "origin", "state": "ok"}' | jq -r '.error.message'
curl -s -H "$H" "$API/workspaces/$WS/resources?template=service-triage" | jq -r '[.result[].key] | join(" ")'
curl -s -H "$H" "$API/workspaces/$WS/incidents?template=service-triage&resource_state=archived" \
| jq -c '.result[] | {resource: .resource.key, rule, lifecycle, closed_reason}'
true
Resource srv-0044 is archived and accepts no Observations.
srv-0042 srv-0043 srv-0050
{"resource":"srv-0044","rule":"probe_issue","lifecycle":"closed","closed_reason":"resource_archived"}
Lists page by limit and next_cursor; state=archived or state=all lists archived servers.
With the Python SDK: ws.resources.list(template="service-triage", meta={"provider": "alpha"}),
ws.resources.archive("srv-0044") and
ws.incidents.list(template="service-triage", resource_state="archived").
Move a server to a new revision
From the release candidate after 2.0.0-rc2, a server can follow another published revision
without changing its key: from its page (Actions → Rebind), or with the API.
curl -s -H "$H" -X POST $API/workspaces/$WS/resources/srv-0042/rebind \
-d '{"template": "service-triage@4", "meta": {"provider": "alpha"}}' | jq -c '.result | {key, template}'
{"key":"srv-0042","template":"service-triage@4"}
The server keeps its id, key and history. Its Signals and Rules become the new revision’s. Its
open Incidents of the old revision close as rule_removed, not as a recovery, and stay readable
with the Observations they used. Evidence for a Signal the new revision does not declare is
refused. Nothing changes if the revision or an answer is refused, and an archived server cannot
be rebound. Publishing a new revision never moves a server on its own.
There is no rollout across servers. Publishing only makes a revision available, and each server moves when you rebind it, one at a time. Servers you have not rebound keep their revision and its Rules.
What carries across a rebind, from 2.0.0-rc10:
- History. A Signal with the same key and kind in both revisions keeps its earlier readings,
so baselines and counted windows go on reading them. A Signal named in
reset_history, a Signal whose kind changed, or one the old revision did not have starts with no history (details). Rebinds made before2.0.0-rc10are not linked. - The current reading. It is the new binding’s own: until you push after the rebind, the Signal reads as never reported.
- Incidents. None carry over. An open Incident of the old revision closes as
rule_removed. If the trouble is still there, the new Rule opens its own Incident from readings pushed after the rebind.
Any number of probes
From the release candidate after 2.0.0-rc3, one Template can serve a server with one probe and
another with three. all_known judges only the probes that have reported a fresh ok or
problem, and stays unknown until min_known of them have:
{"key": "access_loss", "action": "Investigate the access path.",
"wait": {"confirm_after": 600, "recover_after": 600},
"decision": {"all": [
{"all_known": [
{"signal": "probe_a", "field": "status", "operator": "eq", "operand": "problem"},
{"signal": "probe_b", "field": "status", "operator": "eq", "operand": "problem"},
{"signal": "probe_c", "field": "status", "operator": "eq", "operand": "problem"}],
"min_known": 1},
{"signal": "origin", "field": "status", "operator": "eq", "operand": "ok"}]}}
A server that only has probe_a is judged on probe_a; one with all three needs all three
failing. A probe that never reports, goes quiet, or says no_data is left out rather than
counted as healthy, and with no probe known the Rule decides nothing. To test one input alone,
{"signal": "probe_b", "field": "known"} is true while probe_b has a fresh ok or problem.
When data is missing
If host health cannot be read, or the traffic ratio has no confidence, send no_data for it
(or send the Observation without a value). Do not send ok to fill a gap.
- A missing,
no_dataor value-less input cannot satisfy its own comparison: it is unknown. Avaluesent withno_datais ignored. - Other known inputs still decide. A failing origin check is
service_downwhatever the ratio says; one external probe saying ok rules outaccess_loss. - An unknown result changes nothing: it never opens or confirms access loss, and an open Incident
does not close while its result is unknown. A recovery that was running stops; once the inputs
report again, the whole
recover_aftercounts from the next healthy result, not from before the gap. From2.0.0-rc15, such an Incident never readsrecovering: its page says it is held and shows no countdown. The Rule’sstatuskeeps the decision it last recorded:problemfor a confirmed problem,no_datawhere the hold stopped a recovery,pendingif never confirmed (details). - A Signal not pushed for three rounds (900 seconds here) is unknown too, so a script that
stops cannot leave a server looking healthy. That is three of the Signal’s own
every_secondsperiods; there is no separate stale threshold to set. An event Signal, or one with no period, never goes stale this way. An on-demand Signal is current for itsvalid_secondsinstead, and past that it isnot_checked, not stale. - A
valuethat is not a number is refused (400).
A Rule’s explanation says which of these it is, input by input: unresolved (no_data) when
your script sent no_data, unresolved (nothing has been reported here) for a Signal that
never reported, and unresolved (nothing reported since …) for one that went quiet. From the
release candidate after 2.0.0-rc7, the explanation follows the newest of these even while
the Rule’s result stays unknown, and an Incident page counts a reported no_data as without
data, not as failing.
To find servers whose scripts went quiet, list them by cause:
GET …/resources?missing=stale (or never_reported, reported_no_data;
Missing Signals). That is a filter you
poll, not an alert: no Rule or notification fires because a whole source stopped sending.
When a check is confirmed again
A Signal with a confirm policy asks Uptimer’s workers to check a problem again before its
Rules believe it (Confirming a problem).
With pushed checks that report every round:
- A
problemthat arrives while the same Signal’s confirmation is still pending joins it. It asks no worker again and takes none of the Workspace’s 50 open confirmations. - A
problemthat arrives within 30 seconds after that confirmation was decided reuses the decision. Both values are fixed. Aproblemsent later asks again, even if nothing changed: an unchanged problem pushed every 300 seconds is checked again each round, once the last check was decided more than 30 seconds earlier. - A Workspace has at most 50 confirmations open. Past that, a
problemis not checked again: it readsunconfirmed/limited, the originalproblemstands, and it is never reused. - A confirmer that has not answered by the timeout leaves the original
problemstanding (timeout). An answer that arrives after that is refused, not counted. - Each
no_datareading keeps its ownnot_runrecord. That record means nothing was asked, not that a probe ran.
When a problem flaps
A check that goes back and forth opens and closes Incidents in turn. The Rule’s waits are the
setting for this: confirm_after keeps an Incident unconfirmed, with no problem alert, until
the trouble has lasted, and recover_after keeps it open until health has lasted. A longer recover_after
merges short dips into one Incident, but a real recovery then also closes that much later. A
closed Incident is never reopened: trouble after it closes opens a new one.
What stays in your database
Peer medians and peer sets stay where they are today: push only the derived peer ratio.
From the release candidate after 2.0.0-rc2, a service’s own usual load can instead be
computed by Uptimer from the raw readings you push (Compare load with its own past).
Uptimer’s own database keeps every Observation it accepts, not just what a Rule reads. It keeps them for archived servers and across rebinds as well. There is no retention or compaction setting, and nothing deletes old Observations. Storage grows with servers × Signals × rounds for as long as you push. Plan disk for that growth, watch it, and size your backups for it (Database and migrations). This preview has no tested procedure for removing old Observations.
How far back you can read
| Where | Reads | Older than that |
|---|---|---|
API GET …/resources/{r}/observations?signal&limit | the newest, limit 1–200, default 50 | no cursor: it reads only the newest |
Python SDK ws.resources.observations(r, signal=…, limit=50) | the same API read, default 50 | the same |
MCP list_observations | the newest, limit 1–50, default 20 | the same |
| UI: the Resource’s Observation log | one UTC day at a time, optionally one Signal, paged with Older and Newer | pick an earlier day |
The API, SDK and MCP reads are for recent context. To look at an earlier day, use the Observation log in the UI (Read a problem). An Incident’s history keeps the evidence each transition read, however old.