# Build your own BugSnag > Source: https://buildyourown.software/like/bugsnag > Category: Error monitoring. Original vendor: SmartBear Software. Last updated 2026-09-10. > This file is written for AI coding agents. It contains the context, the build prompt, and the test plan for a self-built replacement of BugSnag. ## How to use this file 1. Read the whole file first. 2. Build the app described in "Build prompt". Treat its acceptance criteria as the definition of done. 3. When the build works, follow "Test prompt" to write and run automated tests. 4. Ask the user before changing the data model, the stack, or the non-goals. 5. The app must run the "one box" way described in "Default deployment" below: one process, SQLite, local disk, no hosted services unless the build prompt names one. ## What BugSnag does BugSnag is an error monitoring service. You drop an SDK into your web, mobile, or backend app. When something throws, the SDK sends the error class, message, stack trace, the user, and a trail of breadcrumbs to BugSnag. BugSnag groups matching events into one issue and shows you how often it happens and how many users it hit. On top of that it tracks releases and gives each one a stability score (the share of sessions that didn't crash), symbolicates minified stacks with source maps, and alerts Slack, email, or PagerDuty when a new issue shows up or a fixed one comes back. SmartBear bought it in 2021 and now also sells performance monitoring on the same dashboard. For a small team with a web app and a Node backend, the part you use is: receive a JSON payload, hash the stack to group it, show a list, and post to Slack. That's a weekend build with one tricky piece (source maps). ## What it costs Typical spend: $1,524 per year. BugSnag Preferred at 300k events a month for a small team: $127 × 12 months on annual billing. $1,800 if you pay monthly. - Free: $0 1 user - Select: $20 per month billed annually, at 50k events ($23 monthly) (Price steps up with the event pack: 150k is $32, 300k is $65, 1M is $200, 3M is $534 a month on annual billing.) - Preferred: $33 per month billed annually, at 100k events ($39 monthly) (300k events is $127, 500k is $219, 1M is $399, 3M is $1,069 a month on annual billing.) - Enterprise: Custom custom event volume and retention (Contact sales.) Prices checked 2026-09-10 at https://www.bugsnag.com/pricing. ## Features and whether to build them - [build] Error ingest from SDKs: A browser and a Node SDK catch unhandled errors and post a JSON payload to your server. This is the product. Each SDK is about 100 lines once you skip mobile. - [build] Grouping into issues: Hash the normalized stack trace so the same bug from a thousand users is one row. Without grouping you have a log file. The fingerprint function is the core of the build. - [build] Issue inbox: List of issues with event count, users affected, first and last seen, sorted and filtered. The screen you open every morning. - [build] Stack trace, breadcrumbs, and metadata: Per-event view of frames, the clicks and requests before the error, the user, and custom data. This is how you reproduce the bug. Breadcrumbs are a ring buffer in the SDK. - [build] Source map symbolication: Upload source maps per release and turn main-8f3a2c.js:1:48213 into checkout.ts:42. Minified browser stacks are useless without it. One library call per frame. - [build] Release tracking and stability score: Every event carries a release version. Stability is the share of sessions with no unhandled error. Tells you whether the deploy you just shipped made things worse. It's two counters per release. - [build] Slack and email alerts: A message when a new issue appears or a resolved one regresses. Slack incoming webhooks and one transactional email. An afternoon. - [build] Resolve, ignore, snooze: Issue states, plus 'resolved in release X' so old builds don't reopen it. Without states the inbox fills up with noise you already know about. - [maybe] Custom filters and segmentation: Pivot issues by any metadata key you send (plan tier, region, browser). A jsonb column and a WHERE clause. Add it when you have a question the list can't answer. - [maybe] Spike detection: Alert when the project error rate jumps well above its baseline. Compare this hour to the last 24. Cheap, but new-issue and regression alerts cover most of it. - [skip] Performance monitoring and tracing: Spans, Web Vitals, app start time, distributed traces. Different product with its own data model. Use OpenTelemetry if you want it. - [skip] Mobile and native SDKs: iOS, Android, React Native, Unity, dSYM and ProGuard symbolication. Native crash handling is real work. If you ship mobile apps, that's the one reason to keep paying. - [skip] Two-way issue tracker sync, SSO, on-prem: Push to Jira and sync status back, SAML login, self-hosted BugSnag. You already self-host. A 'create GitHub issue' link is enough. ## Under the hood ### Data model - Project: id, name, slug, platform (browser | node), api_key, stability_target, retention_days, daily_event_limit, alert_environments. api_key is 32 random bytes as hex. The SDK sends it in a header. - Issue: id, project_id, fingerprint, error_class, message_template, culprit, status (open | resolved | ignored | snoozed), resolved_in_release, snoozed_until, snooze_event_target, first_seen, last_seen, event_count, user_count, first_release, last_release. Unique on (project_id, fingerprint). Counts only go up; retention never decrements them. - Event: id, project_id, issue_id, occurred_at, received_at, exceptions (json), exceptions_resolved (json), release, environment, session_id, user_key, user (json), breadcrumbs (json), metadata (json), request (json), handled, severity, pinned. The raw thing the SDK sent, plus the source-mapped copy. pinned rows survive retention. - Release: id, project_id, version, environment, git_sha, first_seen, sessions, crashed_sessions, events, new_issues. stability = 100 × (1 - crashed_sessions / sessions). Created automatically the first time a version shows up. - SourceMap: id, project_id, release, minified_file, path (under ./data/files/source_maps/), size_bytes, uploaded_at. Match on file basename, ignoring origin and query string. The map is a file on disk, not a column. Keep parsed maps in an in-memory LRU. - AlertChannel: id, project_id, kind (slack | email), config (json: webhook_url or emails[]), on_new_issue, on_regression, enabled. Plus a notifications_log table so you never send the same new-issue alert twice. ### Key flows **Ingest an event** 1. SDK POSTs JSON to /api/ingest with the project API key in a header. 2. Validate with zod, truncate long fields, scrub keys like password and token, reject over 256 KB. 3. Resolve minified frames through any source map stored for that release. 4. Normalize the top in-app frames and hash them with the error class to get the fingerprint. 5. Upsert the issue on (project_id, fingerprint), insert the event, bump daily counters and the distinct-user set, all in one transaction. 6. After commit, decide whether this is a new issue or a regression and send alerts. **Fingerprint a stack** 1. Take up to 5 in-app frames from the top of the first exception (all frames if none are in-app). 2. Strip the origin, query string, and content hashes from file paths: main-8f3a2c.js becomes main-*.js. 3. Drop line and column for minified files that couldn't be source-mapped; keep file, function, and line otherwise. 4. If there are no frames at all, use the error class plus a message with numbers, ids, and quoted strings replaced by placeholders. 5. sha256 the joined string. A client-supplied groupingKey overrides all of this. **Release stability** 1. The SDK sends a session start on page load or process start; Node's Express middleware counts one per request in 60-second batches. 2. Each event carries the session id. The first unhandled event for a session inserts into crashed_sessions and increments the release counter. 3. Stability = 100 × (1 - crashed / sessions). Releases under the project target show red on the releases page. **Alerts** 1. The issue upsert returns whether it inserted or updated. Insert means new issue: post to Slack and email once, logged in notifications_log. Email goes through nodemailer with SMTP_URL, or to ./data/outbox when it's unset. 2. An event on a resolved issue reopens it and fires a regression alert, unless the event's release is older than resolved_in_release. 3. Regression alerts are capped at one per issue per hour. Ignored issues never alert. **Retention rollup** 1. A croner job inside the app process fires at 03:00 UTC, takes the retention row in job_locks, and runs per project. The lock expires after 30 minutes, so a crashed run is picked up on the next start without double-deleting. 2. Delete events older than retention_days in batches of 5,000, skipping the pinned first and last event of each issue. 3. Charts keep working because issue_daily_stats and release counters were written at ingest and are kept for 400 days. 4. Delete crashed_sessions rows and source maps for releases with no events in 90 days. ### Integrations - SQLite on local disk (better-sqlite3 + Drizzle) (required): Issues, events, releases, counters, and job locks in ./data/app.db. WAL mode keeps up with thousands of events a minute. - Local disk for source maps (required): Uploaded maps live under ./data/files/source_maps/ and download through the app's own signed-URL route. - Self-issued sessions (required): Signed cookie sessions. First run creates the admin from ADMIN_EMAIL/ADMIN_PASSWORD or a browser prompt; teammates join by invite link. The SDKs authenticate with API keys. - croner (in-process) (required): Nightly retention rollup and the daily digest, guarded by a job_locks table. - Server-sent events (required): Live issue counts on the dashboard from an in-memory event bus. - @jridgewell/trace-mapping (required): Source map lookups for minified browser frames. - Slack incoming webhook (optional): New issue and regression messages. Optional; an email channel carries the same alerts. - SMTP relay (optional; outbox in dev) (optional): Alert emails and the daily digest through nodemailer. Without SMTP_URL, mail is written to ./data/outbox. ## Where to build it - Claude Code (https://claude.com/claude-code): Best fit. The fingerprint function and the source map path need tests written and run as you go, both SDKs are small enough to build in the same repo, and the whole thing runs locally against one SQLite file. - Replit (https://replit.com): Fine for building and trying it: Replit gives you a public URL the SDKs can post to while you test. Then move the folder to a VPS and run docker compose up -d, so the scheduler and the SQLite file live on one box that stays up. - ChatGPT / Codex (https://chatgpt.com): Ask ChatGPT to trim the payload and breadcrumb types to what your app actually has, then hand the spec to Codex. - Lovable (https://lovable.dev): Skip it, or use it for the issues list and issue page UI only. Lovable is built around Supabase, and this app keeps ingest, fingerprinting, and the database inside one Node process. --- ## Default deployment Also at https://buildyourown.software/deploy.md. The build prompt below assumes this. Every web app on buildyourown.software is written to run the same way. Learn it once and every tool's build prompt makes sense. ## The shape - **One process.** Node 22 running the app. A second process only when the tool genuinely needs one (a job worker, a probe runner). Never a serverless function. - **SQLite.** `better-sqlite3` with Drizzle, WAL mode, file at `./data/app.db`. Migrations checked into the repo and run on startup. Keep the schema portable so `DATABASE_URL=postgres://...` works later, but don't build for it now. - **Local disk for files.** Uploads live under `./data/files///`. The app serves them through an authenticated route with short-lived signed URLs it mints itself. No bucket. - **In-process scheduler.** `croner` inside the app process for reminders, retention, digests, and polling. A row in a `job_locks` table stops two instances from running the same job. - **Sessions the app issues itself.** On first run the app creates an admin from `ADMIN_EMAIL` and `ADMIN_PASSWORD` (or prompts in the browser if unset). Teammates join through invite links. Sessions are signed, httpOnly cookies. Google sign-in is an optional add-on, never a requirement. - **Realtime from the process.** Server-sent events fed by an in-memory event bus. No database change feeds, no websocket service. - **Email through any SMTP server.** `nodemailer` with `SMTP_URL`. Without it, mail is written to `./data/outbox/*.eml` and printed to the log, so every flow works on a laptop with no account. When you go public, point `SMTP_URL` at the one relay you choose. Sending straight from a VPS lands in spam, so this is the one hosted service most tools keep. - **UTC in the database,** local time in the UI, `date-fns-tz` for every conversion. ## Running it - **Laptop:** `pnpm install && pnpm dev`. That's the whole install. No env vars are required; every setting has a default that works locally. - **Public:** a VPS with 1 CPU and 1 GB RAM, one DNS record, and `docker compose up -d`. The compose file has two services: `app` and `caddy`. Caddy terminates TLS with a certificate it gets on its own. `./data` is a named volume. - **Backups:** `pnpm backup` runs `sqlite3 .backup` into `./data/backups/.db` and prunes to the last 14. A `backup` service in compose runs it nightly. Restoring is copying one file. - **Updates:** `git pull && docker compose up -d --build`. Migrations run on boot. ## What this rules out No Vercel, Supabase, Neon, Clerk, Auth0, R2, S3, Fly, Cloudflare Workers, Upstash, or any service that needs an account to run the app. If a tool can't avoid an outside service (a calendar OAuth app, a GitHub webhook, an SMTP relay, a second probe location), the build prompt says so in one line and makes it optional or last. ## Acceptance criteria every tool inherits - A fresh clone runs with `pnpm install && pnpm dev` and no env vars, and the first browser visit creates the admin. - `docker compose up -d` on a clean Ubuntu VPS with one A record serves the app over HTTPS with no other setup. - Killing the process mid-job and restarting it doesn't double-run or lose the job. - `pnpm backup` then deleting `./data/app.db` and restoring from the backup brings every record back. - Every outbound email in dev shows up in `./data/outbox`. --- ## Build prompt Also available on its own at https://buildyourown.software/like/bugsnag/build-prompt.md # Build an error tracking service (replacing BugSnag for your own apps) You are building a self-hosted error tracking service for a small team that ships a web app and a Node backend. It replaces BugSnag for teams that use error monitoring, issue grouping, source maps, release stability, and Slack alerts, and nothing else. Build it end to end: the server, the dashboard, a browser SDK, and a Node SDK. Correctness of grouping matters more than features. If two events are the same bug they must land in the same issue, and if they aren't they must not. ## Stack - One Node 22 process running Next.js (App Router) with TypeScript and Tailwind. It serves the dashboard, the ingest API, the SSE stream, and the scheduled jobs. No serverless functions, no second service. - SQLite via `better-sqlite3` with Drizzle, WAL mode, file at `./data/app.db`. Migrations are checked in and run on boot. SQLite handles a small team's event rate fine (thousands of events a minute in WAL mode) as long as the ingest route does all of its writes in one transaction with prepared statements. Keep the schema portable so `DATABASE_URL=postgres://...` can work later, but don't build for it now. - Files on local disk under `./data/files/
//`. Source maps live there. The app serves them through an authenticated route with short-lived signed URLs it mints itself. No bucket. - `croner` inside the app process for the nightly retention rollup and the daily digest. A row in a `job_locks` table stops two instances from running the same job. - Sessions the app issues itself: signed, httpOnly cookies. On first run the app creates an admin from `ADMIN_EMAIL` and `ADMIN_PASSWORD`, or prompts in the browser when they're unset. Teammates join through invite links. The SDKs authenticate with per-project API keys, never with a session. - Server-sent events fed by an in-memory event bus for live updates on the dashboard. No database change feeds, no websocket service. - Email through `nodemailer` with `SMTP_URL`. Without it, mail is written to `./data/outbox/*.eml` and printed to the log, so alerts and the digest work on a laptop with no account. The relay is the one hosted service you'll want in production, because mail sent straight from a VPS lands in spam. - Slack incoming webhook (optional): the only way to post an alert into Slack. A project with no Slack channel gets the same alerts by email. - UTC in the database, local time in the UI, `date-fns-tz` for every conversion. - `zod` for payload validation, `@jridgewell/trace-mapping` for source maps, `semver` for release comparisons. - Monorepo layout: `apps/web` (the service), `packages/sdk-browser`, `packages/sdk-node`, `examples/` (a test page and a test Express app), one `pnpm-workspace.yaml`. Full profile: https://buildyourown.software/deploy.md ## Running it - Laptop: `pnpm install && pnpm dev`. No env vars are required. `./data` is created on first boot, `SESSION_SECRET` is generated and saved to `./data/secret` when unset, `APP_URL` defaults to `http://localhost:3000`, and the example page and Express app point there. - VPS: a 1 CPU / 1 GB box, one A record, `DOMAIN=errors.example.com` in `.env`, then `docker compose up -d`. The compose file has `app`, `caddy` (gets its own certificate), and `backup`. `./data` is a named volume. The SDKs post to `https:///api/ingest`. Set `SMTP_URL` and `ADMIN_EMAIL`/`ADMIN_PASSWORD` in the same `.env` when you go public. - Backups: `pnpm backup` runs `sqlite3 ./data/app.db ".backup ./data/backups/.db"`, tars `./data/files` next to it (source maps aren't in the database), and prunes both to the last 14. The `backup` service runs it nightly at 02:00 UTC, an hour before retention. Restore is copying one `.db` file back and untarring the files. - Updates: `git pull && docker compose up -d --build`. Migrations run on boot. ## Data model All tables have `id` (text, uuid), `created_at`, `updated_at`. Timestamps are ISO 8601 UTC strings (`2026-09-10T14:03:22.120Z`) in `text` columns; they sort and compare correctly as strings. Booleans are `integer` 0 or 1. JSON columns are `text` holding JSON, parsed at the edge and validated with zod. Day columns are `text` in `YYYY-MM-DD` (UTC). Counters are `integer` (64-bit in SQLite). `stability_target` is `real`. What changes if you drafted this against Postgres: there is no `timestamptz`, so every write goes through one `nowIso()` helper and never `datetime('now')` (which drops milliseconds and the `Z`); `jsonb` becomes `text` and metadata filters use `json_extract()`; `text[]` (`alert_environments`) becomes a JSON array in `text`; `xmax = 0` insert detection is replaced by the transaction under "Ingest rules". - `users`: `email` (unique), `name`, `password_hash` (nullable until an invite is accepted, argon2), `role` (`admin` | `member`), `invite_token` (nullable, unique, 32 random bytes as base64url), `invite_expires_at` (nullable). - `projects`: `name`, `slug` (unique), `platform` (`browser` | `node`), `api_key` (32 random bytes as hex, unique), `api_key_previous` (nullable), `api_key_rotated_at` (nullable), `stability_target` (real, default 99.5), `retention_days` (int, default 30, min 7, max 90), `daily_event_limit` (int, default 100000), `alert_environments` (json array, default `["production"]`). - `issues`: `project_id`, `fingerprint` (text), `error_class`, `message_template`, `message_sample` (the first raw message), `culprit` (top in-app frame as `file in function`), `status` (`open` | `resolved` | `ignored` | `snoozed`), `resolved_at`, `resolved_in_release` (nullable text), `snoozed_until` (nullable), `snooze_event_target` (nullable int), `regressed_at`, `first_seen`, `last_seen`, `event_count` (int), `user_count` (int), `first_release`, `last_release`, `assignee_id` (nullable). Unique index on (`project_id`, `fingerprint`). Index on (`project_id`, `status`, `last_seen desc`). - `events`: `project_id`, `issue_id`, `event_id` (nullable, client-supplied uuid), `occurred_at`, `received_at`, `exceptions` (json, as sent), `exceptions_resolved` (json, after source maps, nullable), `release` (text, nullable), `environment` (text, default `production`), `session_id` (nullable), `user_key` (nullable text), `user` (json), `breadcrumbs` (json), `metadata` (json), `request` (json), `sdk` (json), `handled` (bool), `severity` (`error` | `warning` | `info`), `pinned` (bool, default false). Index on (`issue_id`, `occurred_at desc`) and unique on (`project_id`, `event_id`) where `event_id` is not null. - `issue_users`: `issue_id`, `user_key`, `first_seen`, `last_seen`. Unique on (`issue_id`, `user_key`). - `issue_daily_stats`: `issue_id`, `day`, `event_count`, `user_count`. Unique on (`issue_id`, `day`). - `releases`: `project_id`, `version`, `environment`, `git_sha` (nullable), `first_seen`, `sessions` (int), `crashed_sessions` (int), `events` (int), `new_issues` (int). Unique on (`project_id`, `version`, `environment`). - `release_daily_stats`: `release_id`, `day`, `sessions`, `crashed_sessions`, `events`. Unique on (`release_id`, `day`). - `crashed_sessions`: `project_id`, `release_id`, `session_id`, `day`. Unique on (`project_id`, `session_id`). - `project_daily_stats`: `project_id`, `day`, `events`, `dropped`. Unique on (`project_id`, `day`). - `source_maps`: `project_id`, `release`, `minified_file` (basename, e.g. `main-8f3a2c.js`), `path` (relative to `./data/files`, e.g. `source_maps//main-8f3a2c.js.map`), `size_bytes`, `uploaded_at`. Unique on (`project_id`, `release`, `minified_file`). The map itself is never stored in the database. - `alert_channels`: `project_id`, `kind` (`slack` | `email`), `config` (json: `{ webhookUrl }` or `{ emails: string[] }`), `on_new_issue` (bool), `on_regression` (bool), `enabled` (bool). - `notifications_log`: `issue_id`, `channel_id`, `kind` (`new_issue` | `regression`), `sent_at`, `error` (nullable). - `job_locks`: `name` (primary key: `retention` | `digest`), `locked_by` (nullable, hostname plus pid), `locked_at` (nullable), `expires_at` (nullable), `last_run_at` (nullable), `last_result` (nullable json). - `digest_log`: `project_id`, `day`, `sent_at`. Unique on (`project_id`, `day`), so a re-run never sends a second digest. ## Auth and sessions - On boot, if `users` is empty and `ADMIN_EMAIL` and `ADMIN_PASSWORD` are set, create the admin. If they're unset, every route redirects to `/setup`, a one-time form that creates the admin and signs them in. `/setup` returns 404 once a user exists. - Sign in is email plus password. A session is a signed cookie (`HMAC-SHA256` over `userId|expiresAt` with `SESSION_SECRET`), httpOnly, `SameSite=Lax`, 30 days, refreshed on use. No sessions table. - Admins create invite links from Settings: `/invite/`, valid 48 hours, single use. Accepting it sets name and password. Rate-limit sign-in to 10 attempts per email per 15 minutes in memory. - Every page and every `/api/projects/*` route requires the session. `/api/ingest`, `/api/sessions`, and `/api/source-maps` require `X-Api-Key`. `/api/files/*` requires a valid signature. ## The ingest payload `POST /api/ingest` with header `X-Api-Key: ` (also accept `apiKey` in the body for `sendBeacon`). Accept `Content-Type: application/json` and `text/plain`. The body is one event object or an array of up to 100 (the Node SDK batches handled events). Answer `OPTIONS` with CORS headers and set `Access-Control-Allow-Origin: *` on every response. ```json { "sdk": { "name": "errorwatch-browser", "version": "0.1.0" }, "release": "1.4.2", "environment": "production", "sessionId": "8c1f0c2e-7d9a-4a2e-9a6f-3c1b2f9b7e10", "occurredAt": "2026-09-10T14:03:22.120Z", "handled": false, "severity": "error", "groupingKey": null, "exceptions": [ { "class": "TypeError", "message": "Cannot read properties of undefined (reading 'id')", "frames": [ { "file": "https://app.example.com/assets/main-8f3a2c.js", "function": "loadUser", "line": 1, "column": 48213, "inApp": true }, { "file": "https://app.example.com/assets/vendor-11ab2c.js", "function": "u", "line": 1, "column": 9021, "inApp": false } ] } ], "user": { "id": "u_123", "email": "jane@example.com", "name": "Jane" }, "breadcrumbs": [ { "at": "2026-09-10T14:03:19.002Z", "type": "navigation", "message": "/orders -> /orders/551", "data": {} }, { "at": "2026-09-10T14:03:21.900Z", "type": "click", "message": "button 'Reorder'", "data": {} }, { "at": "2026-09-10T14:03:22.010Z", "type": "request", "message": "GET /api/orders/551 -> 500", "data": { "durationMs": 88 } } ], "metadata": { "cart": { "items": 3, "total": 4999 } }, "request": { "url": "https://app.example.com/orders/551", "method": "GET", "userAgent": "Mozilla/5.0 ..." } } ``` `exceptions[0]` is the outermost error; later entries are `cause` chains. Frames are top of stack first. `occurredAt` is optional and defaults to `received_at`; reject it if it's more than 5 minutes in the future or 30 days in the past and use `received_at` instead. Ingest rules: 1. `401` on a missing or unknown API key. `400` with a list of zod errors on a malformed body. `413` over 256 KB for one event or 2 MB for a batch. `429` with `Retry-After: 3600` once the project has hit `daily_event_limit` for the UTC day; increment `project_daily_stats.dropped`. 2. Truncate: message to 1,000 chars, breadcrumb messages to 200, max 25 breadcrumbs (keep the newest), max 50 frames per exception, max 5 exceptions, `metadata` to 16 KB (drop whole top-level keys from the end until it fits). 3. Scrub: at any depth in `user`, `metadata`, `request`, and `breadcrumbs[].data`, any key matching `/password|passwd|secret|token|authorization|cookie|session|card|ssn/i` gets the value `"[REDACTED]"`. Never store `request.headers.cookie` or `request.headers.authorization`. 4. `user_key` = `user.id`, falling back to `user.email`, else null. 5. Resolve source maps (below) and fingerprint (below) for every event in the batch first, outside any transaction. Then open one `db.transaction()` for the whole batch and, per event: upsert the issue, insert the event, upsert `issue_users`, bump `issue_daily_stats`, `project_daily_stats`, and the release counters. Prepare each statement once and reuse it across the loop. Return `202 { "eventId", "issueId" }` for a single event or an array of those for a batch. 6. After the transaction commits, publish `issue.updated` on the event bus and run the alert decision. Never send alerts inside the transaction. Upsert the issue in two statements inside the transaction. better-sqlite3 holds the write lock for the whole transaction, so concurrent first events serialize and exactly one sees `inserted = true`: ```ts const inserted = insertIssue.run({ ...issueRow, onConflict: "do nothing" }).changes === 1; if (!inserted) bumpIssue.run({ lastSeen, lastRelease, projectId, fingerprint }); const issue = selectIssue.get(projectId, fingerprint); // id, status, resolved_in_release, snooze_event_target, event_count ``` Where `insertIssue` is `insert into issues (...) values (...) on conflict (project_id, fingerprint) do nothing` and `bumpIssue` is `update issues set last_seen = ?, event_count = event_count + 1, last_release = ? where project_id = ? and fingerprint = ?`. ## Fingerprinting (`lib/fingerprint.ts`, pure, unit-tested) Input: the resolved exceptions (or raw if no map matched), the project id, and an optional `groupingKey`. 1. If `groupingKey` is a non-empty string, return `sha256(projectId + "|" + groupingKey)`. 2. Take `exceptions[0]`. Pick the first 5 frames with `inApp = true`. If there are none, take the first 5 frames. 3. Normalize each frame's `file`: - Strip the origin (`https://app.example.com`) and any query string or fragment. - Replace content hashes: `/[-.][0-9a-f]{6,}(?=\.m?js$)/i` becomes `-*`, so `main-8f3a2c.js` and `main.8F3A2C.js` both become `main-*.js`. - Strip `webpack:///`, `webpack://`, leading `./` and `../` segments, and everything before `/src/`, `/app/`, `/dist/`, or `/node_modules/` on absolute Node paths. 4. Normalize `function`: empty, ``, `eval`, `Object.` all become `?`. 5. Decide whether the frame is minified: the file is unresolved (no source map hit) and either matches the hash pattern, ends in `.min.js`, or has `column > 500`. Minified frames contribute `file:function` only. Everything else contributes `file:function:line` (never column). 6. If there are no frames at all, use `message_template`: the message with numbers replaced by `#`, UUIDs and hex strings of 8+ chars by ``, and anything in single or double quotes by ``. Fingerprint is `sha256(projectId + "|" + class + "|" + message_template)`. 7. Otherwise fingerprint is `sha256(projectId + "|" + class + "|" + frames.join("\n"))`. Always compute `message_template` and store it on the issue; the issues list shows it as the title. `culprit` is the first in-app frame after normalization as `src/checkout.ts in applyDiscount`. ## Source maps - `POST /api/source-maps` with `X-Api-Key`, multipart form: `release`, `minifiedFile` (basename), `map` (the file). Max 20 MB. Write the file to `./data/files/source_maps//.map` with `fs.promises.writeFile` to a temp name and rename into place, then upsert the row on (project, release, minifiedFile). Replacing a map deletes the old file after the row commits. - `scripts/upload-source-maps.ts --release `: walks the directory, uploads every `.map` next to a `.js`, and prints what it sent. Read `--release` or fall back to `package.json` `version`. Point it at the server with `ERRORWATCH_URL` (default `http://localhost:3000`). - At ingest, for each frame whose `file` basename matches a stored map for the event's `release`: read the file, parse with `TraceMap` (cache parsed maps in an in-memory LRU of 50 keyed by row id), call `originalPositionFor({ line, column })`, and produce a resolved frame with `file = source`, `line`, `column`, `function = name || original function`, `inApp = true` unless the source path contains `node_modules`. Attach `context` (5 lines before and after) from `sourcesContent` when present. - Store the resolved copy in `events.exceptions_resolved`. Fingerprint from the resolved copy. Frames with no map keep their raw values and are treated as minified for fingerprinting if they look minified. - The releases page links each uploaded map through `GET /api/files/`. `lib/files.ts` exports `signFileUrl(path, ttlSeconds)` (HMAC over `path|exp` with `SESSION_SECRET`) and the route verifies it, rejects any path that resolves outside `./data/files`, and streams the file with `fs.createReadStream`. Default TTL 5 minutes. - Events that arrive before a map is uploaded are not reprocessed. Say so in the README. ## Sessions and stability - `POST /api/sessions` with `X-Api-Key`: `{ "release", "environment", "sessionsStarted": 1, "at": "..." }`. The browser SDK sends one on init. The Node SDK sends one on init and, with the Express `requestHandler`, batches one per request into a single POST every 60 seconds. Upsert `releases` (creating it if new) and add to `sessions` on both the release and its `release_daily_stats` row. - On an unhandled event (`handled = false`, severity `error`) with a `sessionId`: insert into `crashed_sessions` on conflict do nothing. If `changes === 1`, increment `crashed_sessions` on the release and the day row. Two crashes from one session count once. - Stability for a release = `100 × (1 - crashed_sessions / sessions)`, shown with one decimal. If `sessions < 100`, show it in gray with the label "low sample". Below `stability_target` shows red. - `releases.new_issues` = count of issues whose `first_release` equals this version. Update it when an issue is inserted. ## Alerts Run after the ingest transaction commits, and only when the event's `environment` is in the project's `alert_environments`. - New issue: `inserted = true`. Send to every enabled channel with `on_new_issue`. Write one `notifications_log` row per channel. Never send `new_issue` twice for the same issue, even if a channel is added later. - Regression: the issue's `status` is `resolved` and either `resolved_in_release` is null, or the event's `release` is greater than or equal to it (compare with `semver` when both parse, otherwise plain string compare). Set `status = open`, `regressed_at = now`, clear `resolved_in_release`, and send to channels with `on_regression`. If the event's release is older than `resolved_in_release`, store the event, keep the issue resolved, send nothing. - Snoozed: if `snoozed_until` has passed or `event_count >= snooze_event_target`, set `status = open` and treat as a regression. - Ignored issues store events and never alert. - Cap regression alerts at one per issue per 60 minutes (check `notifications_log`). - Slack message: title linking to the issue page (built from `APP_URL`), then `class: message` (truncated to 200 chars), culprit, release, environment, events and users affected. Use Block Kit with a header block and a section with fields. POST to the channel's `webhookUrl` with a 5-second timeout. - Email: same content, one email per address in the channel, subject `[project] New issue: TypeError in src/checkout.ts` or `[project] Regression: ...`. `lib/mail.ts` exports one `sendMail({ to, subject, text, html })`: with `SMTP_URL` it uses `nodemailer.createTransport(SMTP_URL)` and `MAIL_FROM` (default `errorwatch@localhost`); without it, it writes `./data/outbox/-.eml` and logs one line with the path and subject. Every email in the app goes through this function. - Retry a failed send once after 5 seconds, then store the error on the log row. A failing channel must never fail the ingest request. ## Scheduled jobs (`lib/jobs.ts`) Two `croner` jobs start with the server, from `instrumentation.ts`, guarded so they register once per process: `retention` at `0 3 * * *` and `digest` at `0 8 * * *`, both UTC. Rules: - Before running, take the lock: `update job_locks set locked_by = ?, locked_at = ?, expires_at = ? where name = ? and (expires_at is null or expires_at < ?)`. `changes === 0` means another instance holds it; log and skip. Lock TTL is 30 minutes and the job refreshes it after every batch. On finish, write `last_run_at` and `last_result` and clear the lock. - Every step is idempotent: retention deletes by cutoff, the digest checks `digest_log` per project per UTC day before sending. If the process dies mid-run, the lock expires and the next run finishes the work without double-deleting or double-sending. - Catch-up on boot: if a job's `last_run_at` is before its most recent scheduled time (compute it with croner's `previousRun()`), run it now. A laptop that was closed overnight still gets its rollup. - Pass `protect: true` to croner so a slow run never overlaps the next tick in the same process. `pnpm job retention` and `pnpm job digest` run a job once by hand and print `last_result`. ## HTTP API summary | Method | Path | Auth | Purpose | | --- | --- | --- | --- | | POST | `/api/ingest` | `X-Api-Key` | Receive one event or a batch. Returns `202 { eventId, issueId }`. | | POST | `/api/sessions` | `X-Api-Key` | Count session starts for a release. Returns `202`. | | POST | `/api/source-maps` | `X-Api-Key` | Upload one map for a release. Returns `201`. | | GET | `/api/projects/[slug]/issues` | dashboard session | JSON behind the issues table, same filters as the URL. | | POST | `/api/projects/[slug]/issues/[id]/status` | dashboard session | Resolve, ignore, snooze, reopen. | | POST | `/api/projects/[slug]/alert-channels/[id]/test` | dashboard session | Send a test message to one channel. | | GET | `/api/projects/[slug]/stream` | dashboard session | SSE: `issue.updated` and `issue.new` from the in-process bus. | | GET | `/api/files/[token]` | signed URL | Stream one stored source map. | | POST | `/api/auth/setup`, `/api/auth/login`, `/api/auth/logout`, `/api/auth/invite/[token]` | none or session | First-run admin, sign in, sign out, accept an invite. | | GET | `/api/health` | none | `200 { ok, dbPath, jobs }`. Used by compose and the smoke check. | Every SDK-facing endpoint answers `OPTIONS` with CORS headers and finishes in under 200 ms at p95 with 1,000 issues in the project. Do the source map lookup and the fingerprint before opening the transaction so the transaction holds only the writes; SQLite has one writer at a time, so short transactions are what keep ingest fast. ## Edge cases to handle - Deduplicate on an optional `eventId` (uuid) in the payload: if the same project has stored that `eventId` in the last 24 hours, return `202` with the existing ids and store nothing. The browser SDK sets it so a `keepalive` retry can't double count. - An empty or missing `release` stores `null` and the event doesn't touch any release counters. An empty `environment` becomes `production`. Lowercase and trim both. - Clock skew: if `occurredAt` is ahead of `received_at` by more than 5 minutes, or behind by more than 30 days, use `received_at` and set `metadata._clockSkew = true`. - `exceptions[0].frames` empty but a later cause has frames: fingerprint from `exceptions[0]` anyway (class plus message template). Don't silently switch to the cause. - A frame with `line` but no `column` is not minified by the column rule. - Windows paths in Node stacks (`C:\app\src\x.ts`) normalize to forward slashes before the path rules run. - Non-ASCII messages are truncated by code point, not by byte, so you never split a character. - Rotating an API key keeps the old key valid for 24 hours so deployed clients don't drop events during the switch. Store `api_key_previous` and `api_key_rotated_at` on the project. - An issue that's `ignored` and then manually reopened starts alerting again on the next regression, and its `new_issue` alert is never re-sent. - Deleting a project deletes everything under it in one transaction, then removes `./data/files/source_maps/*` rows that belonged to it, behind a confirmation that requires typing the slug. - The SSE route sends a `: ping` comment every 25 seconds so Caddy and browsers keep the connection open, and the client reconnects with `EventSource`'s default backoff. ## Screens Times display in the viewer's browser timezone with the UTC value in a `title` tooltip. Relative times ("4 min ago") in lists. Daily buckets in charts are UTC days and say so in the axis label. 1. **Setup and sign in**: `/setup` on a fresh database (or nothing to do if `ADMIN_EMAIL`/`ADMIN_PASSWORD` were set), then `/login` with email and password. `/invite/` for teammates. Everyone signed in sees every project. 2. **Projects (`/`)**: cards with name, platform, events in the last 24 hours, open issues, and the stability of the most recent production release. "New project" button asks for name and platform and shows the API key and an install snippet with `APP_URL` filled in. 3. **Issues (`/p/[slug]/issues`)**: table, 50 per page. Filters in the URL: `status` (default `open`), `environment`, `release`, `q` (matches error class, message template, culprit). Sort by `last_seen` (default), `event_count`, `user_count`, `first_seen`. Columns: class and message template, culprit, events, users, first seen, last seen, a 14-day sparkline from `issue_daily_stats`. Checkbox bulk actions: resolve, ignore, snooze. Keyboard: `j`/`k` to move, `enter` to open, `r` to resolve. The page subscribes to the SSE stream: counts and last seen update in place, and new issues show as a "3 new issues" banner at the top instead of reordering rows under the cursor. 4. **Issue (`/p/[slug]/issues/[id]`)**: header with class, message sample, status, and buttons: Resolve, Resolve in release (input prefilled with `last_release`), Ignore, Snooze (1 day, 7 days, or "until 100 more events"), Reopen. Stats: events, users, first seen (with `first_release`), last seen (with `last_release`). A 30-day bar chart. Affected users: top 10 by event count with their last seen. Event navigator: latest event by default with prev/next and "oldest". Event panel: the stack with a raw/resolved toggle, each frame expandable to show its `context` lines with the error line highlighted, in-app frames bold, library frames collapsed by default. Cause chain below. Breadcrumbs as a timeline with a type icon, message, and seconds before the error. Tabs: User, Metadata, Request, SDK. 5. **Releases (`/p/[slug]/releases`)**: table: version, environment, first seen, sessions, crashed sessions, stability (colored against the target), new issues, events, source maps uploaded (count, each a signed download link). Row click opens `/p/[slug]/releases/[version]` with the same stats, a daily stability chart, and the list of issues introduced in that release. 6. **Project settings (`/p/[slug]/settings`)**: name, API key with copy and rotate, install snippets for both SDKs, stability target, retention days, daily event limit, alert environments, alert channels (add Slack webhook with a "Send test" button; add email list, with a note that mail goes to `./data/outbox` until `SMTP_URL` is set), and the last 50 rows of `notifications_log`. 7. **Team (`/team`)**: users, roles, "Create invite link" (copies the URL), and the status of both jobs from `job_locks` (last run, last result, lock holder if any). 8. **Daily digest email** (the `digest` job, 08:00 UTC): per project, to every user: new issues in the last 24 hours, top 5 issues by events, and the stability of the latest release. Skip projects with no events. Record each send in `digest_log`. Navigation: left sidebar with the project switcher, then Issues, Releases, Settings, and Team at the bottom. Dashboard behaviors: - Every list is a server component. Initial render does no client-side fetching. Paginate with `?page=` in the URL. The SSE subscription is the only client-side connection. - The 14-day sparkline and the 30-day chart read only `issue_daily_stats`, never `events`. - Empty states say what to do next: a project with no events shows the install snippet; an issue list with filters and no rows shows "No issues match" with a clear-filters link. - Status changes write who did it and when to an `issue_activity` table (`issue_id`, `user_id`, `action`, `detail`, `created_at`) shown at the bottom of the issue page. - The issue page URL accepts `?event=` so a Slack link can open a specific event. ## Browser SDK (`packages/sdk-browser/src/index.ts`, under 150 lines, no dependencies) Exports `init`, `notify`, `setUser`, `leaveBreadcrumb`, `addMetadata`. Also build a single IIFE file usable as `