# Build an uptime monitor (replacing UptimeRobot)

You are building a self-hosted uptime monitor for a small team's own sites, APIs, and cron jobs. It replaces UptimeRobot for people who want HTTP, port, ping, DNS, SSL, and heartbeat checks, alerts that don't fire on a single blip, incident history, and a public status page. Build it end to end. Getting the "is it really down?" decision right matters more than adding monitor types.

## Stack

- One Node 22 process. Next.js (App Router, TypeScript, Tailwind) started from `server.ts`, which also boots the scheduler and the `home` prober.
- SQLite through `better-sqlite3` and Drizzle, WAL mode, file at `./data/app.db`. Migrations are checked in and run on boot. Keep the schema portable so Postgres works later, but build for SQLite now.
- `croner` inside the process for probing, evaluation, delivery retries, rollups, downsampling, and retention. A `job_locks` table stops two instances from running the same job.
- Sessions the app issues itself: signed httpOnly cookies. First run creates the admin from `ADMIN_EMAIL` and `ADMIN_PASSWORD`, or prompts in the browser if they're unset. Teammates join through invite links.
- Server-sent events from an in-memory bus for live dashboard updates. No websocket service.
- `nodemailer` with `SMTP_URL`. Without it, mail is written to `./data/outbox/*.eml` and printed to the log. Sending straight from a VPS lands in spam, so a relay is the one hosted service you'll want when you go public.
- A second probe region, because one vantage point can't tell "the site is down" from "my box can't reach it". Default: the same prober script run by a GitHub Actions workflow every 5 minutes (free, no server). Alternatives: a second $5 VPS running only the prober container, or `pnpm prober --region laptop` in another terminal.
- Slack, Discord, and generic webhooks are plain outbound URLs. Twilio for SMS, optional.
- `./data` holds the database, outbox, and backups. This tool has no uploads. No bucket.
- UTC ISO 8601 strings in the database, local time in the UI, `date-fns-tz` for every conversion.
- Charts: `uplot`. Validation: zod.
- Full profile: https://buildyourown.software/deploy.md

## Running it

- **Laptop:** `pnpm install && pnpm dev`. No env vars. The app, scheduler, and `home` prober start together, and the first browser visit creates the admin. For a second region on the same machine, run `pnpm prober --region laptop` in another terminal.
- **VPS:** 1 CPU, 1 GB RAM, one A record. `docker compose up -d` starts `app`, `caddy` (gets its own certificate), and `backup`. `./data` is a named volume. Set `APP_URL` to your domain so the GitHub Actions prober knows where to post.
- **Backups:** `pnpm backup` runs `sqlite3 .backup` into `./data/backups/<date>.db` and keeps the last 14. The `backup` service does it nightly. Restore is copying one file over `./data/app.db` with the app stopped.
- **Updates:** `git pull && docker compose up -d --build`. Migrations run on boot.

## Regions

A region is a string. `home` is the app process itself. The second region is whatever runs the standalone prober: `gha` for GitHub Actions, `laptop` for a second terminal, or any name passed to `--region`. Labels and staleness thresholds live in `lib/regions.ts`; unknown names get the default.

```ts
export const REGIONS = {
  home: { label: "Home", staleAfterSeconds: 120 },
  gha: { label: "GitHub Actions", staleAfterSeconds: 900 },
  default: { label: (r: string) => r, staleAfterSeconds: 300 },
};
```

## Data model

SQLite types only: `text`, `integer`, `real`. Every table has `id` (text, UUID v7, unless noted), `created_at`, `updated_at`. Timestamps are ISO 8601 UTC strings like `2026-09-10T14:02:00.000Z`, compared as strings or with `unixepoch()` in SQL. JSON columns are `text` holding JSON, validated with zod at the edge. Booleans are `integer` 0 or 1, arrays are JSON text, and there are no `timestamptz`, `jsonb`, `bigserial`, or generated columns: `duration_seconds` is written on resolve.

- `workspaces`: `name`, `slug`, `prober_token` (random 32 bytes, base64url), `retention_days` (default 90), `raw_retention_days` (default 7).
- `users`: `email` (unique), `name`, `password_hash` (argon2). `sessions`: `id`, `user_id`, `expires_at`. `invites`: `token`, `workspace_id`, `email`, `role`, `expires_at`, `accepted_at`. `workspace_members`: `workspace_id`, `user_id`, `role` (`owner` | `member`).
- `job_locks`: `name` (primary key), `locked_until`, `locked_by`, `last_run_at`, `last_status`, all nullable.
- `monitor_groups`: `workspace_id`, `name`, `position`.
- `monitors`: `workspace_id`, `group_id` (nullable), `name`, `type` (`http` | `tcp` | `ping` | `dns` | `ssl` | `heartbeat`), `target` (URL for http, hostname or IP otherwise), `port` (nullable integer), `config` (JSON text, see below), `interval_seconds` (60 to 3600, default 300), `timeout_ms` (default 10000), `regions` (JSON text array, default `["home","gha"]`), `status` (`pending` | `up` | `down` | `paused`), `paused_at` (nullable), `heartbeat_token` (unique, nullable, 24 random bytes base64url), `last_heartbeat_at` (nullable), `renotify_minutes` (nullable integer).
- `monitor_region_state`: `monitor_id`, `region`, `consecutive_failures` (integer, default 0), `next_run_at`, `last_result_at` (nullable), `last_ok` (integer, nullable), `last_error` (nullable). Primary key (`monitor_id`, `region`). One row per monitor per region in `monitors.regions`; heartbeat monitors have none.
- `check_results`: `id` integer primary key autoincrement, `monitor_id`, `region`, `checked_at`, `ok` (integer), `status_code` (nullable integer), `response_ms` (nullable integer), `error` (nullable text), `detail` (JSON text, nullable), `in_maintenance` (integer). Index on (`monitor_id`, `checked_at desc`). Raw rows live `raw_retention_days`.
- `response_buckets`: `monitor_id`, `region`, `bucket_start` (ISO UTC), `bucket_seconds` (3600 | 21600), `checks`, `failed_checks`, `avg_ms` (real), `p95_ms` (real). Primary key (`monitor_id`, `region`, `bucket_seconds`, `bucket_start`). Written nightly, kept `retention_days`.
- `incidents`: `monitor_id`, `started_at`, `resolved_at` (nullable), `duration_seconds` (nullable integer, written on resolve), `cause` (text), `first_failed_region`, `is_maintenance` (integer), `single_region` (integer), `acknowledged_by` (nullable user), `acknowledged_at`, `notes` (text). Partial unique index on `monitor_id` where `resolved_at is null` so a monitor has at most one open incident.
- `alert_channels`: `workspace_id`, `name`, `type` (`email` | `slack` | `discord` | `webhook` | `sms`), `config` (JSON text), `enabled` (integer). `monitor_alert_channels`: (`monitor_id`, `channel_id`) primary key.
- `alert_deliveries`: `incident_id`, `channel_id`, `event` (`down` | `up` | `reminder`), `sequence` (integer, 0 for down and up), `attempts` (integer), `sent_at` (nullable), `last_error` (nullable). Unique on (`incident_id`, `channel_id`, `event`, `sequence`) so the same message can't go twice.
- `status_pages`: `workspace_id`, `slug` (unique, url-safe), `name`, `description`, `custom_domain` (nullable, unique), `is_public` (integer), `show_response_times` (integer). `status_page_monitors`: `status_page_id`, `monitor_id`, `display_name`, `position`.
- `status_page_subscribers`: `status_page_id`, `email`, `confirm_token`, `confirmed_at` (nullable), `unsubscribe_token`. Unique on (`status_page_id`, `email`).
- `maintenance_windows`: `workspace_id`, `name`, `timezone` (IANA), `starts_at_local` (text `YYYY-MM-DDTHH:mm` for one-off, `HH:mm` for weekly), `ends_at_local`, `recurrence` (`none` | `weekly`), `weekdays` (JSON text array of 0 to 6, Sunday is 0, used when weekly), `active` (integer). `maintenance_window_monitors`: (`window_id`, `monitor_id`).
- `daily_uptime`: `monitor_id`, `day` (text `YYYY-MM-DD`, UTC), `checks`, `failed_checks`, `down_seconds`, `maintenance_seconds`, `avg_response_ms`, `p95_response_ms`. Primary key (`monitor_id`, `day`).

### `monitors.config` by type

Validate with zod on create and update.

- `http`: `method` (`GET` | `HEAD` | `POST`, default GET), `expected_status` (string like `"200-299"` or `"200,301,302"`, default `"200-299"`), `keyword` (nullable), `keyword_mode` (`present` | `absent`), `max_response_ms` (nullable), `headers` (record), `body` (nullable), `follow_redirects` (bool, default true, max 5 hops), `verify_tls` (bool, default true).
- `tcp`: nothing beyond `port` (required).
- `ping`: `count` (default 3).
- `dns`: `record_type` (`A` | `AAAA` | `CNAME` | `MX` | `TXT` | `NS`), `resolver` (IP, default `1.1.1.1`), `expected_values` (string[], may be empty).
- `ssl`: `warn_days` (default 14). `port` defaults to 443.
- `heartbeat`: `grace_seconds` (default 60). `interval_seconds` is how often you expect the ping.

## Screens

1. **Setup, login, and invites.** On boot, if `users` is empty and `ADMIN_EMAIL` and `ADMIN_PASSWORD` are set, create the admin, a workspace, and an owner membership; otherwise the first browser visit shows a one-time setup form that does the same. `/login` checks argon2, inserts a `sessions` row, and sets a signed httpOnly cookie (`SESSION_SECRET` defaults to a random value stored in `./data/secret`). Sessions last 30 days. Owners create invites from settings; the email carries `/invite/[token]`, the page asks for a name and password, and tokens expire after 7 days. No other sign-in method.
2. **Dashboard (`/`).** Monitors grouped by `monitor_groups`. Each row: status dot, name, type, target, 24-hour uptime, 30-day uptime, last response time per region, a 24-hour sparkline. Filter by status and type; search by name and target. Top banner shows open incidents, any stale region, and "single region" when only `home` is reporting. Status dots and the banner update over SSE from `GET /api/events` without a reload. "New monitor" button.
3. **New and edit monitor (`/monitors/new`, `/monitors/[id]/edit`).** The form changes with `type`. Heartbeat shows the ping URL with a copy button after save. Every type has a "Test now" button that runs the check once from the server and shows the result without storing it.
4. **Monitor detail (`/monitors/[id]`).** Current status and how long it's been that way, uptime for 24h, 7d, 30d, 90d, response time chart with a region toggle and range tabs (24h, 7d, 30d), recent checks table (region, time, ok, status, ms, error), incident list, pause and resume, delete.
5. **Incidents (`/incidents`, `/incidents/[id]`).** List with filters for open, resolved, monitor, and date. Detail shows a timeline: first failure per region with its error, the confirmation moment, acknowledgement, resolution, duration. A "single region" tag when it applies. Editable notes. Acknowledge button.
6. **Alert channels (`/settings/alerts`).** Add a channel per type. Email: address. Slack and Discord: webhook URL. Webhook: URL plus a generated secret. SMS: E.164 phone number (only shown when Twilio env vars exist). Each channel has "Send test". Attach channels to monitors from the monitor form. A workspace-level default set applies to new monitors.
7. **Status pages (`/status-pages`, `/status-pages/[id]`).** Create with name and slug, pick monitors, set display names and order, toggle public, set custom domain, see subscriber count with an export button.
8. **Public status page (`/s/[slug]`, also served for `custom_domain` via the Host header in middleware).** Banner: "All systems operational", "Partial outage", or "Major outage" (all listed monitors down). Per monitor: name, current status, 90-day bar with one segment per UTC day colored by that day's uptime (green at or above 99.9, yellow at or above 99, red below, gray no data), 90-day uptime percentage. Below: incidents from the last 30 days with start, end, duration, and monitor. Subscribe form. Times render in the visitor's browser timezone and show the zone label (e.g. "PDT"). Server-render everything; hydrate only the time formatting and subscribe form. `noindex` when `is_public` is false and require a signed-in workspace member.
9. **Maintenance (`/maintenance`).** List of windows with next occurrence, create and edit form with a timezone picker.
10. **Settings (`/settings`).** Members and invites, prober token with rotate button, region health (last result per region, stale flag), a jobs table from `job_locks`, retention days, and copy-paste setup for the GitHub Actions second region.

## The prober (`apps/prober/`)

One small script, `apps/prober/index.ts`, exporting `tick(client, region)` with a loop around it. In-process as `home`, `server.ts` runs `tick()` every 10 seconds under the `probe-home` lock with a client that calls `lib/prober-api.ts` (claim and store) directly. Standalone, `pnpm prober --region gha --once` or `pnpm prober --region laptop` uses a client that talks to `{APP_URL}/api/prober/due` and `/api/prober/results` over HTTPS with `Authorization: Bearer {PROBER_TOKEN}` and never touches the database.

`tick`:

1. Claim up to 50 monitors whose `monitor_region_state` row for this region has `next_run_at <= now`, in one statement: `UPDATE monitor_region_state SET next_run_at = <now + interval_seconds> WHERE region = ? AND next_run_at <= ? AND monitor_id IN (SELECT ... LIMIT 50) RETURNING ...`. SQLite serializes writers, so two callers for the same region never receive the same row. Paused and heartbeat monitors are never returned.
2. Run each check concurrently (limit 10 at a time) with `timeout_ms` as a hard cap. Measure with `performance.now()`.
3. Store a JSON array of `{ monitor_id, region, checked_at, ok, status_code, response_ms, error, detail }`. Standalone, retry the POST 3 times on network failure, then drop the batch and log it.
4. The loop sleeps 10 seconds. With `--once` (used by GitHub Actions) run steps 1 to 3 one time and exit.

`.github/workflows/prober.yml`: `schedule: "*/5 * * * *"`, Node 22, `pnpm prober --region gha --once`, with `APP_URL` and `PROBER_TOKEN` as repository secrets. GitHub delays scheduled runs by several minutes under load, which is why `gha` is stale only after 900 seconds. For a second VPS, the compose file has a `prober` service behind `--profile prober` that loops the same command with `--region <name>`.

Check implementations, each in its own file under `apps/prober/checks/` and exported as `run(monitor): Promise<CheckResult>`:

- **http**: `undici` fetch with an `AbortController`. Send `User-Agent: byo-uptime/1.0`. Follow up to 5 redirects if enabled. Parse `expected_status` into ranges and compare. If `keyword` is set, read at most 1 MB of the body and search case-insensitively; `present` fails when missing, `absent` fails when found. If `max_response_ms` is set and exceeded, fail with `"Slow response: 2140 ms > 1000 ms"`. Error strings: `"HTTP 503"`, `"Keyword 'Welcome' not found"`, `"Keyword 'Error' found"`, `"Timeout after 10000 ms"`, `"TLS error: certificate has expired"`, `"DNS lookup failed"`, `"Connection refused"`. Put `{ final_url, redirects, tls_expires_at }` in `detail` when available.
- **tcp**: `net.connect({ host, port })` with timeout. Ok on `connect`. Errors: `"Connection refused"`, `"Timeout after N ms"`, `"Host not found"`.
- **ping**: spawn `ping -c {count} -W 2 {host}` and parse packet loss and average RTT. Fail when loss is 100 percent with `"100% packet loss"`. If the `ping` binary is missing or ICMP is blocked (exit code with "Operation not permitted"), fall back to a TCP connect on port 443 then 80 and set `detail.fallback = "tcp"`. Install `iputils-ping` in the Dockerfile.
- **dns**: `new dns.promises.Resolver()` with `setServers([resolver])`. Resolve `record_type`. Fail on `ENOTFOUND`, `SERVFAIL`, or timeout. If `expected_values` is non-empty, every expected value must appear in the answers (compare case-insensitively, strip trailing dots). Error: `"Expected 203.0.113.5, got 198.51.100.7"`. Store answers in `detail.answers`.
- **ssl**: `tls.connect({ host, port, servername: host })`, read `getPeerCertificate(true)`. Fail if `authorized` is false with the `authorizationError`. Compute `days_left` from `valid_to`. Fail when `days_left < warn_days` with `"Certificate expires in 10 days"` or `"Certificate expired 3 days ago"`. `detail`: `{ valid_to, issuer, days_left }`.

## Evaluator (`lib/evaluate.ts`, pure, fully unit-tested)

Called by the results store for each result inside a transaction, and by the `evaluate` job for heartbeats.

Input: the monitor, all its `monitor_region_state` rows after applying the new result, the region config, `now`, and whether a maintenance window is active for this monitor.

Rules:

1. Applying a result: `ok` sets that region's `consecutive_failures` to 0; a failure increments it. Set `last_result_at`, `last_ok`, `last_error`.
2. A region is **stale** when `now - last_result_at > staleAfterSeconds` for that region, or it has never reported. Stale regions are ignored in the rules below.
3. A monitor is **confirmed down** when at least two non-stale regions each have `consecutive_failures >= 2`. If exactly one region is non-stale (no second region has ever reported, or it's stale), it's confirmed down when that region has `consecutive_failures >= 2`, and the incident gets `single_region = 1`. Alerts and the dashboard show a "single region" tag so you know the second vantage point wasn't there. If no region is non-stale, do nothing.
4. A monitor is **confirmed up** when every non-stale region has `last_ok = 1`.
5. Transition `up`/`pending` to `down`: set `monitors.status = down`, publish `monitor.status` on the bus, and open an incident with `cause` = the `last_error` of the region that failed first (earliest `last_result_at` among failing regions) and `first_failed_region` set. If maintenance is active, set `is_maintenance = 1` and create no deliveries. Otherwise create one `alert_deliveries` row with `event = down` per enabled attached channel.
6. Transition `down` to `up`: set `resolved_at = now`, write `duration_seconds`, set status `up`, and create `event = up` deliveries unless `is_maintenance`.
7. While `down`, if `renotify_minutes` is set and `now - started_at` crosses a multiple of it, create a `reminder` delivery with the next `sequence`.
8. A monitor that is `paused` ignores results entirely. Pausing while an incident is open resolves the incident with `notes` appended: "Resolved by pause".
9. Heartbeat monitors: down when `last_heartbeat_at` (or `created_at` if never pinged) is older than `interval_seconds + grace_seconds`. Up on the next ping. The cause is `"No heartbeat for 7m 30s"`. Regions don't apply.

Return a list of effects (`open_incident`, `resolve_incident`, `create_deliveries`, `set_status`) so the function stays pure and the caller applies them.

## Alert delivery (`lib/deliver.ts`)

Runs right after deliveries are created (the bus emits `deliveries.created`) and again from the `deliver` job every minute for anything with `sent_at is null and attempts < 3`. Backoff 30s, 2m, 10m.

Every message includes: monitor name, target, event, cause, started time, and for `up` the duration formatted like `1h 12m`. Include a link to the incident. Prefix the cause with `[single region]` when `incidents.single_region` is set.

- **email**: `nodemailer`. Subject `"[DOWN] API is down: HTTP 503"` or `"[UP] API recovered after 12m"`. From `MAIL_FROM` (default `uptime@localhost`). Without `SMTP_URL`, `lib/mail.ts` writes the message to `./data/outbox/<timestamp>-<id>.eml` and logs one line.
- **slack**: POST Block Kit with a header block, a section with fields, and a button to the incident.
- **discord**: POST an embed with red (`0xE5484D`) for down and green (`0x30A46C`) for up.
- **webhook**: POST JSON `{ event, monitor: { id, name, type, target }, incident: { id, started_at, resolved_at, duration_seconds, cause, single_region }, sent_at }` with headers `X-Uptime-Event`, `X-Uptime-Timestamp`, and `X-Uptime-Signature: sha256=<hmac of timestamp + "." + body using the channel secret>`. Document verification in the README. Treat any non-2xx as a failure.
- **sms**: Twilio REST API, `"DOWN: API (HTTP 503) since 14:02 UTC"`, max 160 characters. Skip and mark `last_error = "Twilio not configured"` when env vars are missing.

## Heartbeat endpoint

`GET` or `POST /api/hb/[token]`. Look up the monitor by `heartbeat_token`, set `last_heartbeat_at = now`, insert a `check_results` row with `region = "heartbeat"` and `ok = 1`, run the evaluator, return `200 OK` with a plain-text body. Unknown token returns 404. Accept `?status=fail` to record a failed run: insert a failed result and open an incident immediately with cause `"Job reported failure"`. Rate limit to 10 requests per minute per token with an in-memory counter.

## Maintenance windows

`isInMaintenance(monitorId, now)` in `lib/maintenance.ts`: for each active window attached to the monitor, compute the current occurrence in the window's timezone with `date-fns-tz`. One-off: `[starts_at_local, ends_at_local)` converted from the zone. Weekly: for today and yesterday in that zone, if the weekday is in `weekdays`, build the interval from `starts_at_local` and `ends_at_local` on that date (if end is before start it crosses midnight). Return true if `now` falls inside any interval. Write a test for a 02:00 to 03:00 window on a DST change day in `America/New_York` and one that crosses midnight in `Asia/Kolkata`.

During maintenance, `check_results.in_maintenance = 1`. Incidents opened during maintenance are flagged and never alert. An incident that is already open when a window starts keeps alerting for recovery. Maintenance seconds are excluded from the uptime denominator.

## Jobs, uptime, rollups, retention

All scheduled work goes through `lib/jobs.ts`: `schedule(name, cron, ttlSeconds, fn)`. Before running, claim the row: `UPDATE job_locks SET locked_until = ?, locked_by = ? WHERE name = ? AND (locked_until IS NULL OR locked_until < ?)`. Zero rows changed means another instance has it, so skip. On finish, clear `locked_until` and write `last_run_at` and `last_status`. Every job is idempotent (upserts, unique delivery keys, claims that move `next_run_at` forward), so a process killed mid-run redoes the work after the lock expires.

- `probe-home` every 10 seconds, ttl 60: one prober tick as region `home`.
- `evaluate` every minute, ttl 120: heartbeat timeouts, reminders, region staleness (publishes `region.stale` on the bus).
- `deliver` every minute, ttl 120: retry pending deliveries.
- `rollup` daily at 00:10 UTC, ttl 600: write `daily_uptime` for yesterday for every monitor: check counts, `avg_response_ms` and `p95_response_ms` across regions, down and maintenance seconds from incidents clipped to that UTC day. Upsert.
- `downsample` daily at 00:20 UTC, ttl 600: for yesterday, write `response_buckets` at 3600 and 21600 seconds per monitor and region (avg and p95 in JS from the sorted raw rows), then delete `check_results` older than `raw_retention_days`. Never delete incidents or `daily_uptime`.
- `retention` daily at 00:30 UTC: delete `response_buckets` older than `retention_days` and `sessions` past `expires_at`.
- `uptimePercent(monitorId, windowDays, now)` in `lib/uptime.ts`: window starts at `max(now - windowDays, monitor.created_at)`. Down seconds = sum of non-maintenance incident intervals clipped to the window (open incidents end at `now`). Maintenance seconds = sum of maintenance intervals clipped to the window. Result = `(window - maintenance - down) / (window - maintenance) * 100`, rounded to 3 decimals. A monitor with no checks yet returns `null` and renders as "no data".
- Response time chart API: `GET /api/monitors/[id]/response-times?range=24h|7d|30d`. `24h` groups raw `check_results` into 5-minute bins with `(unixepoch(checked_at) / 300) * 300` and computes p95 in JS per bin. `7d` reads the 3600-second buckets and `30d` the 21600-second buckets, with today filled from raw rows.

## Status page subscribers

- `POST /api/status-pages/[slug]/subscribe` with an email. Upsert the subscriber, send a confirmation email with `/s/[slug]/confirm?token=`. Confirming sets `confirmed_at`. Every subscriber email has an unsubscribe link with `unsubscribe_token`.
- When an incident opens or resolves on a monitor that appears on a public status page, queue one email per confirmed subscriber of that page and send them through `lib/mail.ts` with concurrency 5. Don't email for `is_maintenance` incidents.
- Rate limit subscribe to 5 per IP per hour, in memory.
- Custom domains: the Caddyfile uses on-demand TLS with `ask http://app:3000/api/caddy/ask`. That route returns 200 when `domain` matches a `status_pages.custom_domain` and 404 otherwise, so Caddy only issues certificates for domains you configured. The user points a CNAME at the app.

## Non-goals

No voice calls, on-call rotations or escalation policies, page speed audits, dependency or third-party status feeds, UDP checks, mobile app, or billing. No hosted services beyond an SMTP relay, GitHub Actions for the second region, Slack, Discord, and webhook URLs, and optional Twilio. Design the schema so `escalation_policy_id` on monitors can be added later.

## Acceptance criteria

1. The admin signs in, adds an HTTP monitor for a test server, and within `interval_seconds` sees results on the monitor detail page from `home` and from a standalone prober started with `pnpm prober --region laptop`.
2. The test server returns 500 for requests from one region only. After four checks, no incident exists and the monitor is still `up`.
3. The test server returns 500 for everyone. After the second consecutive failure from each of two regions, exactly one incident exists with `cause = "HTTP 500"`, `first_failed_region` set, and `single_region = 0`, and each attached channel has exactly one `down` delivery with `sent_at` set. A third failing check creates nothing new.
4. The test server recovers. When both regions report ok, the incident has `resolved_at`, `duration_seconds` matches the timestamps, and each channel has exactly one `up` delivery.
5. Keyword `present` for a word not on the page fails with `"Keyword 'x' not found"`; keyword `absent` for a word on the page fails with `"Keyword 'x' found"`; `max_response_ms = 1` fails with a `"Slow response"` cause carrying the measured milliseconds.
6. A TCP monitor on an open port is ok; on a closed port it fails with `"Connection refused"`; on a blackholed address it fails with a timeout after `timeout_ms`.
7. A DNS monitor with `expected_values = ["203.0.113.5"]` against a name resolving elsewhere fails and `detail.answers` lists the real answers. An SSL monitor against a certificate with 10 days left and `warn_days = 14` fails with `"Certificate expires in 10 days"`; an expired certificate fails with the `authorizationError`.
8. A heartbeat monitor with `interval_seconds = 120` and `grace_seconds = 60` that receives no ping opens an incident on the first `evaluate` run after 180 seconds; one `GET /api/hb/[token]` resolves it. `?status=fail` opens an incident immediately.
9. With an active maintenance window, failures from both regions open an incident with `is_maintenance = 1`, zero deliveries, and 30-day uptime stays at 100.000. A weekly window from 02:00 to 03:00 `America/New_York` is active at 06:30 UTC in January and 06:30 UTC in July.
10. Seeded incidents produce exact uptime numbers: one 72-minute non-maintenance incident inside a 30-day window on a monitor older than 30 days gives 99.833.
11. When no second region has ever reported, or `gha` has been silent for more than 900 seconds, settings shows it as missing or stale, two consecutive failures from `home` alone open an incident with `single_region = 1`, and the email subject and webhook payload carry the tag.
12. `/s/[slug]` renders the banner, one row per monitor with a 90-day bar and percentage, and the last 30 days of incidents; subscribing sends a confirmation, confirming enables incident emails, and the unsubscribe link removes the subscriber.
13. A webhook receiver can verify `X-Uptime-Signature` with the channel secret, and a receiver returning 500 causes three attempts with backoff before the delivery is marked failed.
14. After the `downsample` job runs, the 7-day chart is served from `response_buckets`, raw `check_results` older than `raw_retention_days` are gone, and incidents and `daily_uptime` are untouched.
15. A fresh clone runs with `pnpm install && pnpm dev` and no env vars, and the first browser visit creates the admin.
16. `docker compose up -d` on a clean Ubuntu VPS with one A record serves the app over HTTPS with no other setup.
17. Killing the process mid-job and restarting it doesn't double-run or lose the job.
18. `pnpm backup` then deleting `./data/app.db` and restoring from the backup brings every record back.
19. Every outbound email in dev shows up in `./data/outbox`.

## Deliverables

- The app, migrations, and a seed script: one workspace, six monitors (one per type) pointing at a bundled test server in `apps/testserver/` whose behavior you can toggle with `POST /control { status, delay_ms, body }`, two alert channels, one status page, one maintenance window, and 30 days of synthetic `daily_uptime`, `response_buckets`, and incidents.
- `apps/prober/` with `--region` and `--once` flags and a `pnpm prober` script. `.github/workflows/prober.yml` running `pnpm prober --region gha --once` every 5 minutes.
- `docker-compose.yml` with `app`, `caddy`, `backup`, and a `prober` service behind `--profile prober`. `Caddyfile` with the main site and on-demand TLS for custom domains. `Dockerfile` with `iputils-ping` and `sqlite3` installed. `scripts/backup.sh` behind `pnpm backup`.
- `lib/evaluate.ts`, `lib/maintenance.ts`, `lib/uptime.ts`, and `lib/jobs.ts` as pure or thin modules with table-driven tests.
- README covering laptop, VPS (`docker compose up -d`, DNS, updates), backup and restore, every env var and its default (`ADMIN_EMAIL`, `ADMIN_PASSWORD`, `APP_URL`, `PROBER_TOKEN`, `SMTP_URL`, `MAIL_FROM`, `TWILIO_*`), the three ways to get a second region (GitHub Actions, second VPS, laptop) and what "single region" means when you have none, webhook signature verification, and custom domains for status pages.

Build the evaluator and its tests first, then the prober checks against the bundled test server, then the claim and results code, then the screens, then alerts, then status pages. Run `pnpm dev` with `pnpm prober --region laptop` in a second terminal, then stop the laptop prober and confirm a single-region incident, before calling anything done.