# Build an error tracking service (replacing BugSnag for your own apps)

You are building a self-hosted error tracking service for a small team that ships a web app and a Node backend. It replaces BugSnag for teams that use error monitoring, issue grouping, source maps, release stability, and Slack alerts, and nothing else. Build it end to end: the server, the dashboard, a browser SDK, and a Node SDK. Correctness of grouping matters more than features. If two events are the same bug they must land in the same issue, and if they aren't they must not.

## Stack

- One Node 22 process running Next.js (App Router) with TypeScript and Tailwind. It serves the dashboard, the ingest API, the SSE stream, and the scheduled jobs. No serverless functions, no second service.
- SQLite via `better-sqlite3` with Drizzle, WAL mode, file at `./data/app.db`. Migrations are checked in and run on boot. SQLite handles a small team's event rate fine (thousands of events a minute in WAL mode) as long as the ingest route does all of its writes in one transaction with prepared statements. Keep the schema portable so `DATABASE_URL=postgres://...` can work later, but don't build for it now.
- Files on local disk under `./data/files/<table>/<id>/`. Source maps live there. The app serves them through an authenticated route with short-lived signed URLs it mints itself. No bucket.
- `croner` inside the app process for the nightly retention rollup and the daily digest. A row in a `job_locks` table stops two instances from running the same job.
- Sessions the app issues itself: signed, httpOnly cookies. On first run the app creates an admin from `ADMIN_EMAIL` and `ADMIN_PASSWORD`, or prompts in the browser when they're unset. Teammates join through invite links. The SDKs authenticate with per-project API keys, never with a session.
- Server-sent events fed by an in-memory event bus for live updates on the dashboard. No database change feeds, no websocket service.
- Email through `nodemailer` with `SMTP_URL`. Without it, mail is written to `./data/outbox/*.eml` and printed to the log, so alerts and the digest work on a laptop with no account. The relay is the one hosted service you'll want in production, because mail sent straight from a VPS lands in spam.
- Slack incoming webhook (optional): the only way to post an alert into Slack. A project with no Slack channel gets the same alerts by email.
- UTC in the database, local time in the UI, `date-fns-tz` for every conversion.
- `zod` for payload validation, `@jridgewell/trace-mapping` for source maps, `semver` for release comparisons.
- Monorepo layout: `apps/web` (the service), `packages/sdk-browser`, `packages/sdk-node`, `examples/` (a test page and a test Express app), one `pnpm-workspace.yaml`.

Full profile: https://buildyourown.software/deploy.md

## Running it

- Laptop: `pnpm install && pnpm dev`. No env vars are required. `./data` is created on first boot, `SESSION_SECRET` is generated and saved to `./data/secret` when unset, `APP_URL` defaults to `http://localhost:3000`, and the example page and Express app point there.
- VPS: a 1 CPU / 1 GB box, one A record, `DOMAIN=errors.example.com` in `.env`, then `docker compose up -d`. The compose file has `app`, `caddy` (gets its own certificate), and `backup`. `./data` is a named volume. The SDKs post to `https://<domain>/api/ingest`. Set `SMTP_URL` and `ADMIN_EMAIL`/`ADMIN_PASSWORD` in the same `.env` when you go public.
- Backups: `pnpm backup` runs `sqlite3 ./data/app.db ".backup ./data/backups/<date>.db"`, tars `./data/files` next to it (source maps aren't in the database), and prunes both to the last 14. The `backup` service runs it nightly at 02:00 UTC, an hour before retention. Restore is copying one `.db` file back and untarring the files.
- Updates: `git pull && docker compose up -d --build`. Migrations run on boot.

## Data model

All tables have `id` (text, uuid), `created_at`, `updated_at`. Timestamps are ISO 8601 UTC strings (`2026-09-10T14:03:22.120Z`) in `text` columns; they sort and compare correctly as strings. Booleans are `integer` 0 or 1. JSON columns are `text` holding JSON, parsed at the edge and validated with zod. Day columns are `text` in `YYYY-MM-DD` (UTC). Counters are `integer` (64-bit in SQLite). `stability_target` is `real`.

What changes if you drafted this against Postgres: there is no `timestamptz`, so every write goes through one `nowIso()` helper and never `datetime('now')` (which drops milliseconds and the `Z`); `jsonb` becomes `text` and metadata filters use `json_extract()`; `text[]` (`alert_environments`) becomes a JSON array in `text`; `xmax = 0` insert detection is replaced by the transaction under "Ingest rules".

- `users`: `email` (unique), `name`, `password_hash` (nullable until an invite is accepted, argon2), `role` (`admin` | `member`), `invite_token` (nullable, unique, 32 random bytes as base64url), `invite_expires_at` (nullable).
- `projects`: `name`, `slug` (unique), `platform` (`browser` | `node`), `api_key` (32 random bytes as hex, unique), `api_key_previous` (nullable), `api_key_rotated_at` (nullable), `stability_target` (real, default 99.5), `retention_days` (int, default 30, min 7, max 90), `daily_event_limit` (int, default 100000), `alert_environments` (json array, default `["production"]`).
- `issues`: `project_id`, `fingerprint` (text), `error_class`, `message_template`, `message_sample` (the first raw message), `culprit` (top in-app frame as `file in function`), `status` (`open` | `resolved` | `ignored` | `snoozed`), `resolved_at`, `resolved_in_release` (nullable text), `snoozed_until` (nullable), `snooze_event_target` (nullable int), `regressed_at`, `first_seen`, `last_seen`, `event_count` (int), `user_count` (int), `first_release`, `last_release`, `assignee_id` (nullable). Unique index on (`project_id`, `fingerprint`). Index on (`project_id`, `status`, `last_seen desc`).
- `events`: `project_id`, `issue_id`, `event_id` (nullable, client-supplied uuid), `occurred_at`, `received_at`, `exceptions` (json, as sent), `exceptions_resolved` (json, after source maps, nullable), `release` (text, nullable), `environment` (text, default `production`), `session_id` (nullable), `user_key` (nullable text), `user` (json), `breadcrumbs` (json), `metadata` (json), `request` (json), `sdk` (json), `handled` (bool), `severity` (`error` | `warning` | `info`), `pinned` (bool, default false). Index on (`issue_id`, `occurred_at desc`) and unique on (`project_id`, `event_id`) where `event_id` is not null.
- `issue_users`: `issue_id`, `user_key`, `first_seen`, `last_seen`. Unique on (`issue_id`, `user_key`).
- `issue_daily_stats`: `issue_id`, `day`, `event_count`, `user_count`. Unique on (`issue_id`, `day`).
- `releases`: `project_id`, `version`, `environment`, `git_sha` (nullable), `first_seen`, `sessions` (int), `crashed_sessions` (int), `events` (int), `new_issues` (int). Unique on (`project_id`, `version`, `environment`).
- `release_daily_stats`: `release_id`, `day`, `sessions`, `crashed_sessions`, `events`. Unique on (`release_id`, `day`).
- `crashed_sessions`: `project_id`, `release_id`, `session_id`, `day`. Unique on (`project_id`, `session_id`).
- `project_daily_stats`: `project_id`, `day`, `events`, `dropped`. Unique on (`project_id`, `day`).
- `source_maps`: `project_id`, `release`, `minified_file` (basename, e.g. `main-8f3a2c.js`), `path` (relative to `./data/files`, e.g. `source_maps/<id>/main-8f3a2c.js.map`), `size_bytes`, `uploaded_at`. Unique on (`project_id`, `release`, `minified_file`). The map itself is never stored in the database.
- `alert_channels`: `project_id`, `kind` (`slack` | `email`), `config` (json: `{ webhookUrl }` or `{ emails: string[] }`), `on_new_issue` (bool), `on_regression` (bool), `enabled` (bool).
- `notifications_log`: `issue_id`, `channel_id`, `kind` (`new_issue` | `regression`), `sent_at`, `error` (nullable).
- `job_locks`: `name` (primary key: `retention` | `digest`), `locked_by` (nullable, hostname plus pid), `locked_at` (nullable), `expires_at` (nullable), `last_run_at` (nullable), `last_result` (nullable json).
- `digest_log`: `project_id`, `day`, `sent_at`. Unique on (`project_id`, `day`), so a re-run never sends a second digest.

## Auth and sessions

- On boot, if `users` is empty and `ADMIN_EMAIL` and `ADMIN_PASSWORD` are set, create the admin. If they're unset, every route redirects to `/setup`, a one-time form that creates the admin and signs them in. `/setup` returns 404 once a user exists.
- Sign in is email plus password. A session is a signed cookie (`HMAC-SHA256` over `userId|expiresAt` with `SESSION_SECRET`), httpOnly, `SameSite=Lax`, 30 days, refreshed on use. No sessions table.
- Admins create invite links from Settings: `/invite/<token>`, valid 48 hours, single use. Accepting it sets name and password. Rate-limit sign-in to 10 attempts per email per 15 minutes in memory.
- Every page and every `/api/projects/*` route requires the session. `/api/ingest`, `/api/sessions`, and `/api/source-maps` require `X-Api-Key`. `/api/files/*` requires a valid signature.

## The ingest payload

`POST /api/ingest` with header `X-Api-Key: <project api_key>` (also accept `apiKey` in the body for `sendBeacon`). Accept `Content-Type: application/json` and `text/plain`. The body is one event object or an array of up to 100 (the Node SDK batches handled events). Answer `OPTIONS` with CORS headers and set `Access-Control-Allow-Origin: *` on every response.

```json
{
  "sdk": { "name": "errorwatch-browser", "version": "0.1.0" },
  "release": "1.4.2",
  "environment": "production",
  "sessionId": "8c1f0c2e-7d9a-4a2e-9a6f-3c1b2f9b7e10",
  "occurredAt": "2026-09-10T14:03:22.120Z",
  "handled": false,
  "severity": "error",
  "groupingKey": null,
  "exceptions": [
    {
      "class": "TypeError",
      "message": "Cannot read properties of undefined (reading 'id')",
      "frames": [
        { "file": "https://app.example.com/assets/main-8f3a2c.js", "function": "loadUser", "line": 1, "column": 48213, "inApp": true },
        { "file": "https://app.example.com/assets/vendor-11ab2c.js", "function": "u", "line": 1, "column": 9021, "inApp": false }
      ]
    }
  ],
  "user": { "id": "u_123", "email": "jane@example.com", "name": "Jane" },
  "breadcrumbs": [
    { "at": "2026-09-10T14:03:19.002Z", "type": "navigation", "message": "/orders -> /orders/551", "data": {} },
    { "at": "2026-09-10T14:03:21.900Z", "type": "click", "message": "button 'Reorder'", "data": {} },
    { "at": "2026-09-10T14:03:22.010Z", "type": "request", "message": "GET /api/orders/551 -> 500", "data": { "durationMs": 88 } }
  ],
  "metadata": { "cart": { "items": 3, "total": 4999 } },
  "request": { "url": "https://app.example.com/orders/551", "method": "GET", "userAgent": "Mozilla/5.0 ..." }
}
```

`exceptions[0]` is the outermost error; later entries are `cause` chains. Frames are top of stack first. `occurredAt` is optional and defaults to `received_at`; reject it if it's more than 5 minutes in the future or 30 days in the past and use `received_at` instead.

Ingest rules:

1. `401` on a missing or unknown API key. `400` with a list of zod errors on a malformed body. `413` over 256 KB for one event or 2 MB for a batch. `429` with `Retry-After: 3600` once the project has hit `daily_event_limit` for the UTC day; increment `project_daily_stats.dropped`.
2. Truncate: message to 1,000 chars, breadcrumb messages to 200, max 25 breadcrumbs (keep the newest), max 50 frames per exception, max 5 exceptions, `metadata` to 16 KB (drop whole top-level keys from the end until it fits).
3. Scrub: at any depth in `user`, `metadata`, `request`, and `breadcrumbs[].data`, any key matching `/password|passwd|secret|token|authorization|cookie|session|card|ssn/i` gets the value `"[REDACTED]"`. Never store `request.headers.cookie` or `request.headers.authorization`.
4. `user_key` = `user.id`, falling back to `user.email`, else null.
5. Resolve source maps (below) and fingerprint (below) for every event in the batch first, outside any transaction. Then open one `db.transaction()` for the whole batch and, per event: upsert the issue, insert the event, upsert `issue_users`, bump `issue_daily_stats`, `project_daily_stats`, and the release counters. Prepare each statement once and reuse it across the loop. Return `202 { "eventId", "issueId" }` for a single event or an array of those for a batch.
6. After the transaction commits, publish `issue.updated` on the event bus and run the alert decision. Never send alerts inside the transaction.

Upsert the issue in two statements inside the transaction. better-sqlite3 holds the write lock for the whole transaction, so concurrent first events serialize and exactly one sees `inserted = true`:

```ts
const inserted = insertIssue.run({ ...issueRow, onConflict: "do nothing" }).changes === 1;
if (!inserted) bumpIssue.run({ lastSeen, lastRelease, projectId, fingerprint });
const issue = selectIssue.get(projectId, fingerprint); // id, status, resolved_in_release, snooze_event_target, event_count
```

Where `insertIssue` is `insert into issues (...) values (...) on conflict (project_id, fingerprint) do nothing` and `bumpIssue` is `update issues set last_seen = ?, event_count = event_count + 1, last_release = ? where project_id = ? and fingerprint = ?`.

## Fingerprinting (`lib/fingerprint.ts`, pure, unit-tested)

Input: the resolved exceptions (or raw if no map matched), the project id, and an optional `groupingKey`.

1. If `groupingKey` is a non-empty string, return `sha256(projectId + "|" + groupingKey)`.
2. Take `exceptions[0]`. Pick the first 5 frames with `inApp = true`. If there are none, take the first 5 frames.
3. Normalize each frame's `file`:
   - Strip the origin (`https://app.example.com`) and any query string or fragment.
   - Replace content hashes: `/[-.][0-9a-f]{6,}(?=\.m?js$)/i` becomes `-*`, so `main-8f3a2c.js` and `main.8F3A2C.js` both become `main-*.js`.
   - Strip `webpack:///`, `webpack://`, leading `./` and `../` segments, and everything before `/src/`, `/app/`, `/dist/`, or `/node_modules/` on absolute Node paths.
4. Normalize `function`: empty, `<anonymous>`, `eval`, `Object.<anonymous>` all become `?`.
5. Decide whether the frame is minified: the file is unresolved (no source map hit) and either matches the hash pattern, ends in `.min.js`, or has `column > 500`. Minified frames contribute `file:function` only. Everything else contributes `file:function:line` (never column).
6. If there are no frames at all, use `message_template`: the message with numbers replaced by `#`, UUIDs and hex strings of 8+ chars by `<id>`, and anything in single or double quotes by `<str>`. Fingerprint is `sha256(projectId + "|" + class + "|" + message_template)`.
7. Otherwise fingerprint is `sha256(projectId + "|" + class + "|" + frames.join("\n"))`.

Always compute `message_template` and store it on the issue; the issues list shows it as the title. `culprit` is the first in-app frame after normalization as `src/checkout.ts in applyDiscount`.

## Source maps

- `POST /api/source-maps` with `X-Api-Key`, multipart form: `release`, `minifiedFile` (basename), `map` (the file). Max 20 MB. Write the file to `./data/files/source_maps/<id>/<minifiedFile>.map` with `fs.promises.writeFile` to a temp name and rename into place, then upsert the row on (project, release, minifiedFile). Replacing a map deletes the old file after the row commits.
- `scripts/upload-source-maps.ts <dir> --release <version>`: walks the directory, uploads every `.map` next to a `.js`, and prints what it sent. Read `--release` or fall back to `package.json` `version`. Point it at the server with `ERRORWATCH_URL` (default `http://localhost:3000`).
- At ingest, for each frame whose `file` basename matches a stored map for the event's `release`: read the file, parse with `TraceMap` (cache parsed maps in an in-memory LRU of 50 keyed by row id), call `originalPositionFor({ line, column })`, and produce a resolved frame with `file = source`, `line`, `column`, `function = name || original function`, `inApp = true` unless the source path contains `node_modules`. Attach `context` (5 lines before and after) from `sourcesContent` when present.
- Store the resolved copy in `events.exceptions_resolved`. Fingerprint from the resolved copy. Frames with no map keep their raw values and are treated as minified for fingerprinting if they look minified.
- The releases page links each uploaded map through `GET /api/files/<token>`. `lib/files.ts` exports `signFileUrl(path, ttlSeconds)` (HMAC over `path|exp` with `SESSION_SECRET`) and the route verifies it, rejects any path that resolves outside `./data/files`, and streams the file with `fs.createReadStream`. Default TTL 5 minutes.
- Events that arrive before a map is uploaded are not reprocessed. Say so in the README.

## Sessions and stability

- `POST /api/sessions` with `X-Api-Key`: `{ "release", "environment", "sessionsStarted": 1, "at": "..." }`. The browser SDK sends one on init. The Node SDK sends one on init and, with the Express `requestHandler`, batches one per request into a single POST every 60 seconds. Upsert `releases` (creating it if new) and add to `sessions` on both the release and its `release_daily_stats` row.
- On an unhandled event (`handled = false`, severity `error`) with a `sessionId`: insert into `crashed_sessions` on conflict do nothing. If `changes === 1`, increment `crashed_sessions` on the release and the day row. Two crashes from one session count once.
- Stability for a release = `100 × (1 - crashed_sessions / sessions)`, shown with one decimal. If `sessions < 100`, show it in gray with the label "low sample". Below `stability_target` shows red.
- `releases.new_issues` = count of issues whose `first_release` equals this version. Update it when an issue is inserted.

## Alerts

Run after the ingest transaction commits, and only when the event's `environment` is in the project's `alert_environments`.

- New issue: `inserted = true`. Send to every enabled channel with `on_new_issue`. Write one `notifications_log` row per channel. Never send `new_issue` twice for the same issue, even if a channel is added later.
- Regression: the issue's `status` is `resolved` and either `resolved_in_release` is null, or the event's `release` is greater than or equal to it (compare with `semver` when both parse, otherwise plain string compare). Set `status = open`, `regressed_at = now`, clear `resolved_in_release`, and send to channels with `on_regression`. If the event's release is older than `resolved_in_release`, store the event, keep the issue resolved, send nothing.
- Snoozed: if `snoozed_until` has passed or `event_count >= snooze_event_target`, set `status = open` and treat as a regression.
- Ignored issues store events and never alert.
- Cap regression alerts at one per issue per 60 minutes (check `notifications_log`).
- Slack message: title linking to the issue page (built from `APP_URL`), then `class: message` (truncated to 200 chars), culprit, release, environment, events and users affected. Use Block Kit with a header block and a section with fields. POST to the channel's `webhookUrl` with a 5-second timeout.
- Email: same content, one email per address in the channel, subject `[project] New issue: TypeError in src/checkout.ts` or `[project] Regression: ...`. `lib/mail.ts` exports one `sendMail({ to, subject, text, html })`: with `SMTP_URL` it uses `nodemailer.createTransport(SMTP_URL)` and `MAIL_FROM` (default `errorwatch@localhost`); without it, it writes `./data/outbox/<ISO timestamp>-<subject slug>.eml` and logs one line with the path and subject. Every email in the app goes through this function.
- Retry a failed send once after 5 seconds, then store the error on the log row. A failing channel must never fail the ingest request.

## Scheduled jobs (`lib/jobs.ts`)

Two `croner` jobs start with the server, from `instrumentation.ts`, guarded so they register once per process: `retention` at `0 3 * * *` and `digest` at `0 8 * * *`, both UTC. Rules:

- Before running, take the lock: `update job_locks set locked_by = ?, locked_at = ?, expires_at = ? where name = ? and (expires_at is null or expires_at < ?)`. `changes === 0` means another instance holds it; log and skip. Lock TTL is 30 minutes and the job refreshes it after every batch. On finish, write `last_run_at` and `last_result` and clear the lock.
- Every step is idempotent: retention deletes by cutoff, the digest checks `digest_log` per project per UTC day before sending. If the process dies mid-run, the lock expires and the next run finishes the work without double-deleting or double-sending.
- Catch-up on boot: if a job's `last_run_at` is before its most recent scheduled time (compute it with croner's `previousRun()`), run it now. A laptop that was closed overnight still gets its rollup.
- Pass `protect: true` to croner so a slow run never overlaps the next tick in the same process. `pnpm job retention` and `pnpm job digest` run a job once by hand and print `last_result`.

## HTTP API summary

| Method | Path | Auth | Purpose |
| --- | --- | --- | --- |
| POST | `/api/ingest` | `X-Api-Key` | Receive one event or a batch. Returns `202 { eventId, issueId }`. |
| POST | `/api/sessions` | `X-Api-Key` | Count session starts for a release. Returns `202`. |
| POST | `/api/source-maps` | `X-Api-Key` | Upload one map for a release. Returns `201`. |
| GET | `/api/projects/[slug]/issues` | dashboard session | JSON behind the issues table, same filters as the URL. |
| POST | `/api/projects/[slug]/issues/[id]/status` | dashboard session | Resolve, ignore, snooze, reopen. |
| POST | `/api/projects/[slug]/alert-channels/[id]/test` | dashboard session | Send a test message to one channel. |
| GET | `/api/projects/[slug]/stream` | dashboard session | SSE: `issue.updated` and `issue.new` from the in-process bus. |
| GET | `/api/files/[token]` | signed URL | Stream one stored source map. |
| POST | `/api/auth/setup`, `/api/auth/login`, `/api/auth/logout`, `/api/auth/invite/[token]` | none or session | First-run admin, sign in, sign out, accept an invite. |
| GET | `/api/health` | none | `200 { ok, dbPath, jobs }`. Used by compose and the smoke check. |

Every SDK-facing endpoint answers `OPTIONS` with CORS headers and finishes in under 200 ms at p95 with 1,000 issues in the project. Do the source map lookup and the fingerprint before opening the transaction so the transaction holds only the writes; SQLite has one writer at a time, so short transactions are what keep ingest fast.

## Edge cases to handle

- Deduplicate on an optional `eventId` (uuid) in the payload: if the same project has stored that `eventId` in the last 24 hours, return `202` with the existing ids and store nothing. The browser SDK sets it so a `keepalive` retry can't double count.
- An empty or missing `release` stores `null` and the event doesn't touch any release counters. An empty `environment` becomes `production`. Lowercase and trim both.
- Clock skew: if `occurredAt` is ahead of `received_at` by more than 5 minutes, or behind by more than 30 days, use `received_at` and set `metadata._clockSkew = true`.
- `exceptions[0].frames` empty but a later cause has frames: fingerprint from `exceptions[0]` anyway (class plus message template). Don't silently switch to the cause.
- A frame with `line` but no `column` is not minified by the column rule.
- Windows paths in Node stacks (`C:\app\src\x.ts`) normalize to forward slashes before the path rules run.
- Non-ASCII messages are truncated by code point, not by byte, so you never split a character.
- Rotating an API key keeps the old key valid for 24 hours so deployed clients don't drop events during the switch. Store `api_key_previous` and `api_key_rotated_at` on the project.
- An issue that's `ignored` and then manually reopened starts alerting again on the next regression, and its `new_issue` alert is never re-sent.
- Deleting a project deletes everything under it in one transaction, then removes `./data/files/source_maps/*` rows that belonged to it, behind a confirmation that requires typing the slug.
- The SSE route sends a `: ping` comment every 25 seconds so Caddy and browsers keep the connection open, and the client reconnects with `EventSource`'s default backoff.

## Screens

Times display in the viewer's browser timezone with the UTC value in a `title` tooltip. Relative times ("4 min ago") in lists. Daily buckets in charts are UTC days and say so in the axis label.

1. **Setup and sign in**: `/setup` on a fresh database (or nothing to do if `ADMIN_EMAIL`/`ADMIN_PASSWORD` were set), then `/login` with email and password. `/invite/<token>` for teammates. Everyone signed in sees every project.
2. **Projects (`/`)**: cards with name, platform, events in the last 24 hours, open issues, and the stability of the most recent production release. "New project" button asks for name and platform and shows the API key and an install snippet with `APP_URL` filled in.
3. **Issues (`/p/[slug]/issues`)**: table, 50 per page. Filters in the URL: `status` (default `open`), `environment`, `release`, `q` (matches error class, message template, culprit). Sort by `last_seen` (default), `event_count`, `user_count`, `first_seen`. Columns: class and message template, culprit, events, users, first seen, last seen, a 14-day sparkline from `issue_daily_stats`. Checkbox bulk actions: resolve, ignore, snooze. Keyboard: `j`/`k` to move, `enter` to open, `r` to resolve. The page subscribes to the SSE stream: counts and last seen update in place, and new issues show as a "3 new issues" banner at the top instead of reordering rows under the cursor.
4. **Issue (`/p/[slug]/issues/[id]`)**: header with class, message sample, status, and buttons: Resolve, Resolve in release (input prefilled with `last_release`), Ignore, Snooze (1 day, 7 days, or "until 100 more events"), Reopen. Stats: events, users, first seen (with `first_release`), last seen (with `last_release`). A 30-day bar chart. Affected users: top 10 by event count with their last seen. Event navigator: latest event by default with prev/next and "oldest". Event panel: the stack with a raw/resolved toggle, each frame expandable to show its `context` lines with the error line highlighted, in-app frames bold, library frames collapsed by default. Cause chain below. Breadcrumbs as a timeline with a type icon, message, and seconds before the error. Tabs: User, Metadata, Request, SDK.
5. **Releases (`/p/[slug]/releases`)**: table: version, environment, first seen, sessions, crashed sessions, stability (colored against the target), new issues, events, source maps uploaded (count, each a signed download link). Row click opens `/p/[slug]/releases/[version]` with the same stats, a daily stability chart, and the list of issues introduced in that release.
6. **Project settings (`/p/[slug]/settings`)**: name, API key with copy and rotate, install snippets for both SDKs, stability target, retention days, daily event limit, alert environments, alert channels (add Slack webhook with a "Send test" button; add email list, with a note that mail goes to `./data/outbox` until `SMTP_URL` is set), and the last 50 rows of `notifications_log`.
7. **Team (`/team`)**: users, roles, "Create invite link" (copies the URL), and the status of both jobs from `job_locks` (last run, last result, lock holder if any).
8. **Daily digest email** (the `digest` job, 08:00 UTC): per project, to every user: new issues in the last 24 hours, top 5 issues by events, and the stability of the latest release. Skip projects with no events. Record each send in `digest_log`.

Navigation: left sidebar with the project switcher, then Issues, Releases, Settings, and Team at the bottom.

Dashboard behaviors:

- Every list is a server component. Initial render does no client-side fetching. Paginate with `?page=` in the URL. The SSE subscription is the only client-side connection.
- The 14-day sparkline and the 30-day chart read only `issue_daily_stats`, never `events`.
- Empty states say what to do next: a project with no events shows the install snippet; an issue list with filters and no rows shows "No issues match" with a clear-filters link.
- Status changes write who did it and when to an `issue_activity` table (`issue_id`, `user_id`, `action`, `detail`, `created_at`) shown at the bottom of the issue page.
- The issue page URL accepts `?event=<id>` so a Slack link can open a specific event.

## Browser SDK (`packages/sdk-browser/src/index.ts`, under 150 lines, no dependencies)

Exports `init`, `notify`, `setUser`, `leaveBreadcrumb`, `addMetadata`. Also build a single IIFE file usable as `<script src="/errorwatch.js">` that exposes `window.errorwatch`.

- `init({ apiKey, endpoint, release, environment, user })`. On init: generate a `sessionId` (`crypto.randomUUID()`), POST a session start, and install handlers.
- `window.onerror` and `window.onunhandledrejection` produce `handled: false` events. Non-Error rejection reasons become `class: "UnhandledRejection"` with the stringified reason as the message and no frames.
- Parse `error.stack` for both V8 (`at fn (url:line:col)`, `at url:line:col`) and Firefox/Safari (`fn@url:line:col`) formats. `inApp` = the frame's origin equals `location.origin` and the path doesn't contain `node_modules`.
- Breadcrumbs, ring buffer of 25: `click` (tag name plus up to 40 chars of text or `aria-label`), `navigation` (wrap `history.pushState` and listen to `popstate`, message `from -> to`), `request` (wrap `fetch`: method, URL path, status, duration), `console` (`console.error` and `console.warn`, first 200 chars), `custom` via `leaveBreadcrumb(message, data)`.
- `notify(error, { severity, metadata, groupingKey })` sends a `handled: true` event.
- Send with `fetch(endpoint, { method: "POST", keepalive: true, headers: { "Content-Type": "text/plain", "X-Api-Key": apiKey } })` so it survives page unload. Drop events beyond 10 per minute. Never throw from the SDK; wrap everything.
- Include `request.url` and `request.userAgent`.

## Node SDK (`packages/sdk-node/src/index.ts`, under 150 lines, no dependencies)

Exports `init`, `notify`, `leaveBreadcrumb`, `addMetadata`, `requestHandler`, `errorHandler`.

- `init({ apiKey, endpoint, release, environment, autoExit })`. `release` defaults to `process.env.RELEASE` then `package.json` `version`; `environment` defaults to `NODE_ENV`. Sends a session start on init.
- `process.on("uncaughtException")` and `process.on("unhandledRejection")` send `handled: false` immediately, then await the send with a 2-second timeout, then `process.exit(1)` when `autoExit !== false`.
- `notify` queues handled events and flushes them as one array POST every 2 seconds or at 100 events, whichever comes first. Flush on `beforeExit`.
- Parse V8 stacks. `inApp` = path doesn't contain `node_modules` and isn't `node:internal`. Strip the working directory from paths.
- `requestHandler()` (Express): assigns a `sessionId` per request, counts it for the 60-second session batch, and stores `req` context in `AsyncLocalStorage` so `notify` inside a route picks up the request.
- `errorHandler()` (Express): captures the error with `request` (`url`, `method`, `headers` minus cookie and authorization, `ip`), `user` from `req.user?.id` if present, then calls `next(err)`.
- Use the global `fetch`. Never throw from the SDK.

## Retention (the `retention` job, nightly at 03:00 UTC)

Per project:

1. Mark the oldest and newest event of each issue `pinned = true` (one update using `min(occurred_at)` and `max(occurred_at)` subqueries grouped by `issue_id`).
2. Delete events where `received_at < cutoff` and `pinned = 0`, in batches of 5,000 (`delete from events where id in (select id ... limit 5000)`) until none remain, refreshing the job lock between batches.
3. Delete `crashed_sessions` rows older than `retention_days`.
4. Delete `issue_daily_stats`, `release_daily_stats`, and `project_daily_stats` rows older than 400 days.
5. Delete `source_maps` rows and their files for releases with no events in 90 days.
6. Run `pragma wal_checkpoint(TRUNCATE)` at the end so the WAL file doesn't grow forever. Log rows deleted per table and store them in `job_locks.last_result`.

Issue `event_count`, `user_count`, `first_seen`, `last_seen`, and release counters never change during retention. Charts read only from the daily stats tables so they stay correct after events are gone.

## Non-goals

Do not build: mobile or native SDKs, performance monitoring or spans, session replay, distributed tracing, SSO or Google sign-in, two-way Jira sync, spike detection, AI explanations, or billing. No hosted services beyond a Slack incoming webhook and an SMTP relay, both optional. Leave a TODO where one would go.

## Acceptance criteria

1. Two `POST /api/ingest` calls with the same `TypeError`, the same function names, and the same hashed file with different hashes (`main-8f3a2c.js` and `main-11ab2c.js`) and different columns produce one issue with `event_count = 2`.
2. The same frames with class `RangeError` instead of `TypeError` produce a second issue. Different messages with the same class and frames (`"order 12"` vs `"order 99"`) stay in one issue with `message_template = "order #"`.
3. After uploading a source map for release `1.0.0`, an event from that release with a minified frame stores an `exceptions_resolved` frame equal to `src/checkout.ts:42` in `applyDiscount`, and `culprit` equals `src/checkout.ts in applyDiscount`. The map file exists under `./data/files/source_maps/` and its signed URL downloads it; the same URL after expiry returns 403.
4. Five events from two distinct `user.id` values give `user_count = 2` and two `issue_users` rows. Events with no user leave `user_count` unchanged.
5. A metadata key `password` is stored as `[REDACTED]`; a 300 KB body returns 413; a wrong API key returns 401; a body with no `exceptions` returns 400 listing the missing field.
6. The first event for a fingerprint sends exactly one Slack post and one email per configured address. The second event sends nothing. An event with `environment = staging` sends nothing when `alert_environments = ["production"]`.
7. Resolving an issue and posting again sets `status = open`, stamps `regressed_at`, and sends a regression alert. Resolving in release `1.2.0`, then posting from `1.1.9`, keeps it resolved with no alert; posting from `1.2.1` regresses it.
8. 200 session starts and 3 unhandled events with distinct session ids on release `1.3.0` show stability `98.5`. Two unhandled events with the same session id count as one crashed session.
9. Twenty concurrent first events for one fingerprint produce one issue, `event_count = 20`, and one `new_issue` log row per channel. A batch of 100 events in one POST commits in one transaction and returns 100 ids.
10. With `retention_days = 7` and events backdated 10 days, the retention job deletes them except the pinned first and last event of each issue; `event_count` and the 30-day chart are unchanged.
11. The example page with the browser SDK: clicking "Crash" sends an event whose breadcrumbs include the click and the prior navigation, and a rejected promise sends an `UnhandledRejection` event.
12. The example Express app: a route that throws produces an event with `request.url` and `request.method`, no `cookie` header, and `user.id` from `req.user`. An `uncaughtException` is received by the server before the process exits.
13. The issues list filters by `status`, `environment`, and `q`, sorts by each column, and pages at 50. Times render in the browser's timezone with UTC in the tooltip. An ingest while the list is open updates the row's event count over SSE without a reload.
14. Snoozing "until 100 more events" reopens the issue on the 100th event and sends a regression alert.
15. A fresh clone runs with `pnpm install && pnpm dev` and no env vars, and the first browser visit creates the admin.
16. `docker compose up -d` on a clean Ubuntu VPS with one A record serves the app over HTTPS with no other setup.
17. Killing the process mid-job and restarting it doesn't double-run or lose the job.
18. `pnpm backup` then deleting `./data/app.db` and restoring from the backup brings every record back.
19. Every outbound email in dev shows up in `./data/outbox`.

## Deliverables

- The service with migrations (run on boot) and `pnpm seed`: two projects (one browser, one node), 12 issues across 3 releases with realistic stacks and breadcrumbs, 400 events, one uploaded source map on disk, one email channel, and one Slack channel pointing at a placeholder webhook.
- `lib/fingerprint.ts`, `lib/sourcemaps.ts`, `lib/jobs.ts`, and `lib/mail.ts` with table-driven tests.
- Both SDKs built to `dist/` with a size check that fails the build over 150 source lines.
- `examples/browser-page` (a static page with Crash, Reject, and Handled buttons, built minified with two different hashes) and `examples/express-app`.
- `docker-compose.yml` (`app`, `caddy`, `backup`, named `data` volume), `Caddyfile` (reads `DOMAIN`, proxies to `app:3000`, keeps SSE connections open), `Dockerfile` with `sqlite3` installed, and `scripts/backup.sh` behind `pnpm backup`.
- README covering: the laptop path (no env vars, where `./data` lives, the outbox), the VPS path (DNS, `.env` with `DOMAIN`, `APP_URL`, `ADMIN_EMAIL`, `ADMIN_PASSWORD`, `SMTP_URL`, `MAIL_FROM`), backups and restore, the two optional outside services (a Slack incoming webhook URL, an SMTP relay) and what happens without each, how to upload source maps in CI, and the note about events that arrive before their map.

Build `lib/fingerprint.ts` and its tests first, then the ingest endpoint against the SQLite file, then the browser SDK against the example page, then setup and sign-in, then the screens with SSE, then alerts through the outbox, then the jobs with their locks, and last the compose file, Caddyfile, and backup script. Run the example page against the running server after each step.