dev-dashboard #

Everything your app is doing, live, at /dev/dashboard: memory, CPU, requests per second, p50/p95/p99 latency, errors, the database pool and Tokio, as charts; every route with its numbers; every request as a trace waterfall; every SQL statement by fingerprint, with N+1 detection and EXPLAIN; the live log; an app doctor that turns all of it into ranked findings; a routes explorer that spots broken links and missing gates; the schema as an interactive diagram, with a migrations timeline that catches drift; an optional AI assistant on every tab with your own key; your test runs on SQLite and Postgres; your CI; and a load lab to make the numbers move. Phoenix has LiveDashboard and Rails has rack-mini-profiler plus a handful of gems; this is one sample, in-process, with no Prometheus, Grafana or Jaeger to run. No dependency on other samples. Extracted from Rust off Rails apps.

• AI command center (the top of the overview, expanded by default, collapsible and remembered in this browser): a briefing that needs no key (the app's state in 3 to 5 plain lines from the insights, N+1s, errors, incidents and test runs, e.g. "N+1 on /orders (20 queries per request) — fix with load_many", each with its evidence link and a sparkline) under a verdict chip (Quiet, Healthy, Watch, Unhealthy); with a key, the AI briefing (the agent's health summary, streamed live with citations and inline charts; written once per boot on the first visit in proactive mode, otherwise a button; its bottom line shows on load, at the top of the card and in the collapsed bar, from the latest finished briefing, cached on disk, even while a new one is being written); a prompt box ("Ask anything about your running app…") with suggestions drawn from the findings ("Why is GET /orders slow?") and quick links; four small charts of the last 15 minutes with the briefing's findings numbered on them; and What you can do here, a card per capability linking to its tab. Without a key the AI parts are a setup card (Anthropic, OpenRouter's cheap models, a local model).
• Overview: cards with sparklines, colored by health (green, amber, red; thresholds are HEALTH, // APP:): memory (and its growth since boot), CPU, req/s, latency, server errors, SQL, pool, Tokio, uptime; the top insights, the slowest routes and queries, the latest traces and N+1s, and panels from other samples (the job queue when jobs is installed).
• Charts: ten live charts and a latency heatmap (SVG, light and dark, gradient fills), 5 min, 15 min, 1 h, 24 h, 7 days or 30 days, across restarts: the history lives on disk (below). Legends summarize the visible window (average and peak for rates, request-weighted percentiles for latency, "—" without data). A crosshair synchronized across every chart with a readout of all series; drag (or Shift+arrows) to zoom into a range; click Requests, Latency or Errors to open that minute's traces; markers for boots, load-lab runs and error bursts. All of it works from the keyboard, and every chart keeps a data table as its text alternative.
• Requests: per matched route: hits, 2xx/3xx/4xx/5xx, p50/p95/p99/max, bytes, last hit, the slowest examples (each a trace).
• Traces, an explorer on one screen: every request, every background job and mail delivery (the latest 300, plus the notable ones: a 5xx, an N+1, slower than 0.5 s, those also on disk). Filters as dropdowns and chips (route with hit counts, kind, status, minimum duration, N+1, errors, time range synced with the charts, text on the path, a method), all in the URL; a duration scatter above the list (x time, y duration, color status: drag across it to zoom, click a dot to open its trace); rows that open inline into their waterfall (j/k to move, Enter to open, Esc to close); compare two traces side by side ("#12 ran 19 more queries than #9"); new traces stream in, held while you're reading.
• The waterfall (the same component on the trace page, in the explorer's rows, a route's drawer on Requests, a statement's page, N+1 findings, a job's page on the jobs dashboard, the mail a request sent, and the Ask AI citations): a collapsible tree, one line per row (SELECT lab_order_items WHERE order_id = ?, the full statement highlighted in the side panel), repeats folded (an N+1's 20 statements, or load_items ×20, as one row with a mini-bar per run), the framework's own checks folded into a framework row, bars colored by kind (handler, method, SQL, template, job, outbound HTTP, middleware, events as markers), self time vs total, uninstrumented time hatched (where to add a span), waiting time (from the request's arrival to its first code: queueing for a worker, a permit or a connection) as a kind of its own, with a "the server is saturated" insight when requests keep waiting, the critical path marked, a header with time by kind as a stacked bar, the query and N+1 counts, the slowest step and a suggested fix. Click a row for its side panel (fields, file:line, the SQL with EXPLAIN, Ask about this, copy as JSON). Expand all, collapse all, "collapse repeated SQL"; the state stays in the URL hash. Without JavaScript every row is there (?all=1).
• Upstreams (the APIs your app calls): every call made through kernel::outbound, the small wrapper this sample ships as THE way to call an API (a timeout, safe retries with backoff, a W3C traceparent, a span, a size cap, nothing secret recorded), per service, method and route template (/items/{id}): calls, p50/p95/p99 against its SLO, error and timeout rates, retries, its timeout, which app routes call it and how much of their time it takes, a 30-minute sparkline; a dependency map (app routes → upstreams, edges as thick as their calls, colored by health; server-rendered SVG with a text alternative), each node a link to its traces. In the waterfall, outbound calls are their own color with the API's host as a chip, retries are grouped as attempts under their call, timeouts are red, and calls made one after another get a note: "these 4 calls run one after another; running them concurrently would save ~1.1 s", with the critical path. Insights: a slow upstream ("payments POST /charge p95 1.2 s (SLO 300 ms) — called by GET /dev/lab/checkout, 62% of its time"), a failing one, a retry storm, a call with no timeout, sequential calls that could be parallel. Ask AI's list_upstreams and get_upstream answer "which API is making checkout slow?".
• Security: deterministic detectors over every request (the middleware sees them before the Host check, CSRF and the body limit answer) and every sign-in event (auth-core's audit trail through an optional wire, or the lab's sign-in form): scanners and probe paths (/.env, /wp-login.php, /.git/config, /actuator…), scanner user agents (sqlmap, nikto, nuclei…), 404 bursts, path traversal (../ and its encodings), SQL injection and XSS patterns in query strings, unusual methods (TRACE, PROPFIND), oversized bodies and headers, bad Host headers, CSRF rejections, CSP report spikes, rate-limit hits; brute force on one account, credential stuffing from one address; an address far busier than the median, one that mostly gets 4xx, rotates user agents or spans many sessions; one session used from several addresses. Each finding has its evidence (the requests, linked to their traces), a severity and what to do ("this address tried 30 accounts in 2 minutes — the per-account throttle held; add an IP-level limit or a WAF rule"). The Security tab: a verdict, a timeline by category, the top offending addresses, the findings and the latest flagged requests. Client addresses are personal data: production shows only their network and a keyed hash (203.0.113.x #a1b2c3), development shows them whole; APP_TRUSTED_PROXY decides the client address. Also in the read-only production view. Ask AI: list_security_findings, get_security_event.
• Frontend (real-user monitoring): a small beacon on every app page (static/dashboard-rum.js, loaded once from the app layout, so it works after boosted navigation and swaps; CSP-compatible) reports rage clicks (3 clicks within 0.7 s on one element), dead clicks (a button or link-like element with no navigation, no DOM change and no request within 1 s), clicks followed by an error, JavaScript errors and unhandled rejections (message and stack, scrubbed), htmx errors (responseError, sendError, swapError, timeout), slow interactions (event → next paint over 200 ms) and the Core Web Vitals (LCP, CLS, INP) per page, each tied to the request id of its page or fetch, so it links to the trace. It never sends text, input values or query strings: elements are named by id, data-testid/data-rum, name, role and tag path. The Frontend tab: clicks that went wrong by page and element with a heat bar, errors grouped by message with first and last seen and the pages, Web Vitals per page colored good / needs improvement / poor, slow interactions; insights like "Rage clicks on #slow-report on /dev/lab/ux — the request it makes is slow (p95 1.4 s)", "Dead clicks on #show-code after boosted navigation" (the class of bug a page script that only runs on a full load causes), "JS error after boosted navigation", "Poor LCP on /". On by default in development; production only with DASHBOARD_RUM=true, sampled by DASHBOARD_RUM_SAMPLE. Ask AI: list_frontend_issues.
• Methods: every span aggregated, profiler style (wall clock, so waits on the database count): calls, total and self time, p50/p95/max, share of request time, the routes that call it, callers and callees; click one for the traces that ran it. An icicle (flame graph, top down) of a route's traces merged. Instrumenting your own methods is one attribute (below).
• Hot functions (development, Linux and macOS): a low-rate CPU sampling profiler (99 Hz, pprof) you start from the page (it samples until you stop it, 30 minutes at most; DASHBOARD_PROFILER=always from boot), with no annotations at all: functions by self and total share, the usual path to each, callers and callees, your code vs dependencies vs std/Tokio vs native, the routes that were being served, and a flame graph. Pause and resume from the page; never in production.
• History on disk: the charts' minutes (and hourly rollups), their markers, the notable traces and every job or mail run go to storage/dashboard/history.sqlite3 (its own SQLite file, compressed, 0600), so a restart loses nothing but the minute in progress, and memory keeps only the hot window. 7 days of minutes, 90 days of hours, 14 days of notable traces, 64 MB at most (// APP:). About 70 KB of disk a day of history.
• SQL: statements grouped by fingerprint (literals stripped): count, total, mean, p95, max, last seen, the routes that ran it, parameters as type and length only. The N+1 list (the same statement more than 5 times in one request) with route and trace. Writes during a GET. EXPLAIN (SQLite EXPLAIN QUERY PLAN, Postgres EXPLAIN) for recorded SELECTs.
• Logs: a live tail over Server-Sent Events, requests and log lines together (like Phoenix's request logger), filtered by level, target and text, with pause and resume. Secret-looking fields are [redacted].
• Insights: findings ranked by impact, each with its evidence, a fix, the framework guide that explains it and the page that proves it (an N+1's waterfall opens right there): N+1 hotspots, slow routes and queries, slow functions (a method past its p95 threshold, one run in a loop, one getting slower, an app function hot on the CPU), rising errors, foreign keys without an index, full scans from EXPLAIN, large or fast-growing tables, pending migrations, jobs that keep failing, memory that keeps growing, a saturated pool, slow renders, and CSP violations the browser reports. Dismiss one for the session. APP: hook for your own rules.
• Routes: the route table grouped by area and resource, with each handler's source file and line, its access level and middleware, live hits, error rate, p50/p95 and a sparkline, links to its traces and SQL, a Try it link, and the problems: 5xx or rising errors, slow, never hit, no test mentions it, a mutating route without CSRF, an admin or dev path without a gate, a GET that writes, 404s seen with the route they probably meant and the page that linked there, overlapping patterns, routes missing from docs/routes.md.
• Schema: an entity-relationship diagram of the live database (SQLite or Postgres): table boxes with their columns, PK/FK/unique marks, foreign keys as edges with crow's-foot cardinality (a nullable foreign key is zero-or-one, a unique one is one-to-one). Laid out on the server (layered by foreign-key depth, crossings reduced with a barycenter pass), so it reads well without JavaScript; with it, pan, zoom, drag tables (remembered in this browser, with Reset layout), click or focus a table to light up its neighbours and see its detail, find a table, "keys only", colour by sample or by name prefix. Export as SVG or as Mermaid erDiagram text (copy button), for docs or an LLM. Below it, the text alternative: every table with its row count (and size on Postgres), columns, indexes and foreign keys, and the relationships in words. Each table has a page: neighbours, the recorded queries that touch it, the migration that created it. On a phone the diagram scrolls in its frame and the lists carry the rest. Never a row's data.
• Migrations (a tab of Schema): every migration newest first, applied or pending, when, which sample or area, its file, and what it did (tables, columns and indexes created, altered or dropped, derived from the migration's source and labelled as such); pending ones get the host's Run button. Drift: tables or columns the applied migrations don't explain, ones they create that are missing, migrations applied but unknown to the code, and a migration file edited after it was applied (SeaORM keeps no checksum, so the dashboard records one per applied migration the first time it sees it). Drift shows in Insights too.
• Ask AI, an agent (optional, your own key): on every dashboard page and, through an Ask AI bar, on the other kernel pages (the mailbox and a message, the jobs dashboard, a job and the error groups, sockets, /dev, the host's routes page). It investigates with read-only tools over what the dashboard already holds (overview and thresholds, insights, routes, before/after comparisons of any two windows, chart series, traces, SQL and EXPLAIN, logs, schema, migrations and drift, route problems, jobs and their failure groups, test results, CI, the mailbox as shapes, and the app's own source under src/, migration/ and templates/), streams each step live ("Looking at the SQL… → query_sql(sort=total)", each result one click away), and answers with evidence as links ([trace:1298], [sql:…], [route:GET /x], [file:src/…:40]) and the charts it cites drawn inline. It can propose actions (run a load-lab scenario, run the tests, open a page) that show as buttons you click; it never changes anything itself. Answers that propose a change come with a fix prompt for your coding agent (evidence, file and line, the change, the test to add, the guide). Anthropic or any OpenAI-compatible endpoint (OpenAI, OpenRouter, Ollama, LM Studio). You see the first request before anything is sent; every investigation is kept in a history. Bounded: steps, tokens and time per run, a daily budget. Development only unless you turn it on behind the production view's gate.
• Health and "what am I looking at": the command center's verdict and briefing (and, with a key, Generate AI briefing: the agent investigates and writes the verdict, what's healthy, what needs attention ranked, and what to fix first, each claim linked to its evidence), a "since you were away" digest, and /dev/dashboard/explain, a guide to every card, chart, panel and page that needs no key, with "Explain more with AI" on each.
• Proactive (opt-in, DASHBOARD_AI_PROACTIVE=true, development): a watcher checks the deterministic signals (latency spike, error burst, new N+1, memory growth, job failures, schema drift, failing tests) and posts an incident card to the overview and Insights; with a key the agent investigates each one once (debounced, budgeted) and the card carries its narrative, evidence and proposed actions.
• Tests: app test (or the Run tests button) runs nextest on SQLite and Postgres with JUnit output; the page shows both side by side, the duration trend, the slowest tests, failures with their output, flaky tests, line coverage from llvm-cov, and which acceptance criteria in product/ no test cites.
• GitHub: the app's CI runs with their success rate by day, open pull requests and issues, the latest release, open Dependabot alerts. Cached for a minute, never fetched at boot, the token never shown.
• Load lab: fake data through an APP: seed hook, a traffic generator against the app's own address (hard caps: 200 req/s, 60 s, 32 in flight), and seven scenarios that light up a panel each: slow query, N+1, panic, memory spike, slow endpoint, error burst, checkout fan-out. app bench --rps 50 --secs 10 --route /dev/lab/orders from a terminal. Three more panels: simulated APIs (/dev/lab/checkout calls inventory, pricing, payments and shipping, served by the app itself under /dev/lab/upstream/{name}, one after another, concurrently with tokio::join!, or with retries; presets make payments slow at p95 1.2 s, make shipping fail 20% or add timeouts, and each API's latency, jitter, error and hang rates can be set), the attack simulator (a scanner run, brute force, credential stuffing, traversal and SQLi probes, CSRF-less POSTs, a stolen session, a request flood; this app's own address only, TEST-NET client addresses), and UX trouble (/dev/lab/ux: a slow button, a dead button, one that throws, and a widget whose script only works on a full page load; ?simulate=1 plays it all by itself, rage clicks included).
• Config: settings and the app's env vars with secrets redacted, OS and build info, and what the dashboard itself holds in memory.
• For agents: /dev/dashboard.json and a dashboard section in /dev/introspect.json.
• It costs little: about 1 µs per request for the middleware in a release build, 6 µs for a whole development request with its SQL recorded, about 1 µs per method span (see How it works); fixed-size memory (a test pushes 20,000 requests and 5,000 statements through it and checks every bound; traces also have a byte budget), history on disk instead of in memory; nothing at all outside development unless you turn on the production view.
• Development only by default (404 in production, proven by a test). Optionally a read-only subset at /admin/dashboard in production, behind the admin password or your admin accounts: no load lab, no EXPLAIN, no parameters, no logs, and traces without their log events or span fields (and Ask AI only with DASHBOARD_AI_IN_PRODUCTION=true).

Guides: .offrails/guides/dev-tools.md · errors-and-logging.md · database.md · testing.md · security.md.

Screens #

Every screen in light and dark, on desktop and phone: the samples repository.

Install #

By hand, in the app root:

1. Crates (cargo add, never with a version): sysinfo --no-default-features --features system, quick-xml, reqwest --no-default-features --features rustls,json,http2, tower --features util, flate2 (the history's compression; already in the tree through tower-http), pprof --no-default-features --target cfg(unix) (the CPU sampler; Linux and macOS only, no C++ demangler), backtrace --target cfg(unix) (already pprof's; to drop its symbol cache). The history file uses SeaORM's sqlx-sqlite feature, which the host enables even for Postgres apps; keep it.
2. Copy files/ into the app, keeping the paths (.config/nextest.toml included). In the test file and the main.rs wiring, replace offrails_host with your crate name.
3. Wiring (each at its samples: <anchor> comment when the app has one; otherwise at the spot described):
   • src/kernel/mod.rs: pub mod dashboard; and pub mod outbound; (the outbound HTTP wrapper: THE way the app calls other APIs).
   • migration/src/lib.rs: mod m20261016_000600_create_lab_orders; and Box::new(m20261016_000600_create_lab_orders::Migration) (the load lab's fake orders).
   • src/config.rs: a dashboard: DashboardConfig field (in the struct, Debug, from_env, test()), and the DashboardConfig section with DashboardAi, AiProvider, ai_base_url_ok, valid_repo and shown_env (the text is in sample.toml).
   • src/state.rs, in AppState::new before the state is built: let db = crate::kernel::dashboard::instrument(db, &config); (the SQL callback must be set before the connection is cloned).
   • src/routes.rs: mount /admin/dashboard behind DashboardAccess when access::admin_mounted(&config); inside if config.is_development() { … } merge web::dev_router(); in the outermost layers, just inside SetRequestIdLayer, .layer(from_fn(crate::kernel::dashboard::http::middleware)); a paths::dev_dashboard() helper; the ROUTES rows (Access::Dev for /dev/…, Access::Admin for /admin/dashboard/…).
   • src/routes.rs, inside the same if access::admin_mounted(&config) block: app = app.route("/rum", axum::routing::post(crate::kernel::dashboard::rum::collect)); (the browser beacon's production endpoint; it answers 404 unless DASHBOARD_RUM=true).
   • The browser beacon on app pages: in src/views/mod.rs (ctx fields) pub dashboard_rum: Option<crate::kernel::dashboard::rum::Beacon>,; in src/controllers/application.rs (ctx init) dashboard_rum: crate::kernel::dashboard::rum::beacon(state, &parts.headers),; in templates/layouts/application.html (app head) the {% if let Some(rum) = ctx.dashboard_rum %}<script src="{{ rum.src }}" data-rum="{{ rum.endpoint }}" data-rid="{{ rum.rid }}" defer></script>{% endif %} line (the exact text is in sample.toml).
   • src/kernel/ui.rs: the Dashboard and Routes tabs, and "system" and "routes" in HIDDEN_TABS (the dashboard replaces them; their pages stay as the fallback).
   • src/kernel/security.rs: "report-uri /dev/csp-reports" in DEV_CSP_EXTRA (development's CSP only).
   • src/controllers/dev.rs: the Dashboard tile. src/kernel/introspect.rs: out["dashboard"].
   • src/main.rs: Box::new(kernel::dashboard::trace::DashboardLayer) in the tracing layers; kernel::dashboard::serve_started(&state); where serve starts; kernel::dashboard::serve_stopped(); at serve stop (writes the minute in progress and pending traces); the Bench, Test and Dashboard subcommands and their dispatch arms.
   • templates/layouts/application.html: a footer link <a href="{{ paths::dev_dashboard() }}" hx-boost="false" class="link">Dashboard</a>.
   • .env.example: GITHUB_REPO, GITHUB_TOKEN, DASHBOARD_IN_PRODUCTION, DASHBOARD_HISTORY, DASHBOARD_HISTORY_PATH, DASHBOARD_PROFILER, and the commented DASHBOARD_AI_* lines.
   • templates/kernel/layout.html (the Ask AI bar on other kernel pages, development only): at kernel head, {% if let Some(bar) = crate::kernel::dashboard::ai_web::bar(chrome.theme.return_to.as_str()) %} with the bar's <link rel="stylesheet"> and <script defer>; at kernel main (the top of <main>), the same if let around {% include "kernel/dashboard/_ai_bar.html" %} (the exact text is in sample.toml); at kernel head too, the trace component's stylesheet and script for pages that embed a trace (waterfall::embed_assets). Without these anchors (an app made before they existed), add the two lines by hand: in <head> after the page's scripts, and first thing inside <main>.
   • tests/app/main.rs: mod dev_dashboard_test; and mod dev_dashboard_signals_test;; security_test.rs router sources: src/kernel/dashboard/web.rs, src/kernel/dashboard/ai_web.rs, src/kernel/dashboard/lab.rs; kernel_web_test.rs no-store paths; theme_switch_test.rs kernel pages.
   • With realtime installed: pub mod ai_realtime; at dashboard mods in src/kernel/dashboard/mod.rs and Box::new(crate::kernel::dashboard::ai_realtime::Realtime), at dashboard ai sources in src/kernel/dashboard/ai_context.rs (/dev/sockets' Ask AI then knows the hub's topics, connections and rates).
   • With jobs-dashboard installed: at job panels in templates/kernel/jobs/show.html, the job's trace (a lazy waterfall from /dev/dashboard/traces/lookup?job=, development only). With dev-mailbox installed: at message panels in templates/kernel/mailbox/_message.html, the trace of whatever sent the mail (lookup?mail=).
   • With auth-core installed: at auth events in src/models/auth_event.rs, crate::kernel::dashboard::threats::auth_event(&format!("{kind:?}"), user_id, meta.ip.as_deref()); (every audit event feeds the brute-force and credential-stuffing detectors); and pub mod accounts; at dashboard mods in src/kernel/dashboard/mod.rs, pub use crate::kernel::dashboard::accounts::{HAS_ACCOUNTS, admin_from_session}; at dashboard accounts in src/kernel/dashboard/access.rs, and mod dev_dashboard_accounts_test; in tests/app/main.rs.
4. cargo fmt --all && cargo run -- db migrate && cargo run -- routes, then the Definition of Done.

tools/install.sh dev-dashboard <app> does steps 1 to 3.

After installing #

• Open /dev/dashboard/lab, press N+1, and follow the link: that is the tour (open the folded load_items ×20 row, click a statement). For a demo, DASHBOARD_DEMO_HISTORY=true in .env (opt-in; tools/demo.sh sets it) fills the last hour with synthetic history at boot, written to the history file flagged as demo data (it never overwrites a real minute) (traffic that swells, a deploy, a spike, an incident), every chart over it badged "demo data"; the real collector takes over from the first second. cargo run -- dashboard demo adds fake orders and a history of synthetic test runs; cargo run -- dashboard warmup --url http://localhost:3000 drives a minute of mixed traffic and every scenario.
• docs/decisions.md: YYYY-MM-DD: installed the dev-dashboard sample (in-process metrics, traces, SQL, insights, load lab; development only); and the crates it added.
• storage/ is git-ignored already; the history file lives there (storage/dashboard/history.sqlite3). Delete it any time: the dashboard starts a fresh one. A damaged or other-version file is moved aside (history.sqlite3.broken-<time>) with a warning, never fatal.
• Method spans at debug level show when the log filter lets them through: development's default (info,<your_crate>=debug) does; a RUST_LOG=info in .env hides them (use RUST_LOG=info,<your_crate>=debug).
• .env: GITHUB_REPO=owner/name (and optionally a read-only GITHUB_TOKEN) for the GitHub page; TEST_DATABASE_URL for Run tests on Postgres.
• Ask AI (optional): DASHBOARD_AI_PROVIDER=anthropic and DASHBOARD_AI_API_KEY in .env (never committed; the model defaults to claude-sonnet-5-5; claude-opus-5-5 answers deeper, claude-haiku-5-5 cheaper). OpenRouter, a local model or any OpenAI-compatible endpoint instead: see "Ask AI on OpenRouter" below. Record the provider choice in docs/decisions.md. With none set, each page's Ask AI and the command center show how to enable it, and the briefing and "What am I looking at?" (/dev/dashboard/explain) work without a key. Budgets per process and UTC day, all optional in .env: DASHBOARD_AI_DAILY_QUESTIONS (200), DASHBOARD_AI_DAILY_TOKENS (10,000,000: an agent run re-sends its context on every step, 40–50k tokens a run), DASHBOARD_AI_RUN_MAX_TOKENS (300,000), DASHBOARD_AI_MAX_ANSWER_TOKENS (8,000; a cut answer gets a Continue button), and a dollar cap: DASHBOARD_AI_PRICE_IN and DASHBOARD_AI_PRICE_OUT (the model's USD per million tokens) with DASHBOARD_AI_DAILY_USD; the panel then shows the estimated spend today next to the tokens left. A refusal names the limit and the variable that raises it.
• Call every external API through kernel::outbound (Adapting it, "Calling other APIs"); a reqwest::Client used directly is invisible to Upstreams and has no timeout.
• The browser beacon is on in development. For production, decide with whoever owns the data, then DASHBOARD_RUM=true (with DASHBOARD_IN_PRODUCTION=true and its gate) and a DASHBOARD_RUM_SAMPLE (0.1 by default); record it in docs/decisions.md. Behind a proxy set APP_TRUSTED_PROXY=true, or every client is the proxy's address on the Security tab.
• DASHBOARD_AI_PROACTIVE=true (development) turns on the watcher and its incident cards; with a provider it spends part of the day's budget on its own (at most PER_HOUR investigations an hour), so leave it off unless you want that.
• app test replaces nothing in CI: CI keeps running cargo nextest run on both databases. Run app test locally to see the runs on the Tests page.
• A test that builds TestApp with a development config and counts queries: the dashboard's SQL callback replaces TestApp's counter in development (SeaORM keeps one callback). Read HUB.sql there, or use the test config.

Adapting it #

• Calling other APIs (THE way): one Outbound per API, built from config, then a call per request. It sets a timeout (DEFAULT_TIMEOUT, 10 s, or the client's or call's own), retries only what is safe to retry (GET, HEAD, PUT, DELETE, or a call marked .idempotent() because it carries an idempotency key) on timeouts, failed connections, 429, 502, 503 and 504 (retries(n), at most MAX_RETRIES, with backoff and jitter), never follows a redirect, sends a traceparent whose trace id is the request's id, caps the answer at MAX_BODY, and opens an http span (one http_attempt per try) that the waterfall and Upstreams read. Constants are // APP: in kernel/outbound.rs.

  let payments = Outbound::new("payments", &config.payments_url)?.timeout(Duration::from_millis(800));
  let reply = payments.post("/charges").bearer(&config.payments_key).json(&charge).idempotent().retries(2).send().await?;
  let charge: Charge = reply.error_for_status()?.json()?;
  // Independent calls together: the slowest one sets the time.
  let (stock, rates) = tokio::join!(inventory.get(&format!("/items/{id}")).send(), shipping.post("/rates").json(&parcel).send());

• Wrap your own API client (a vendor SDK, a gRPC client, a client with its own reqwest inside): give it .with_client(...) if it takes a reqwest::Client, or report each call yourself: open tracing::info_span!("http", service = "stripe", http.method = "POST", http.host = "api.stripe.com", http.route = "/v1/charges", timeout_ms = 800, http.status = tracing::field::Empty, outcome = tracing::field::Empty, attempts = tracing::field::Empty) around it, record the status and outcome, and hand a kernel::outbound::CallRecord to outbound::report (the observer the dashboard registered). Never put an id or a query in http.route: use outbound::route_template(path).

• Upstream SLOs and thresholds: SLO_P95_US and per service SLOS, FAILING_SHARE, MIN_CALLS, RETRY_STORM, SEQUENTIAL_SAVING_US (// APP: in kernel/dashboard/upstreams.rs).

• Tune the security detectors: every threshold is an // APP: constant at the top of kernel/dashboard/threats.rs (WINDOW_SECS, SCAN_404S, PROBE_HITS, BRUTE_FAILURES, STUFFING_ACCOUNTS, RATE_FLOOR, RATE_FACTOR, UA_ROTATION, SESSION_IPS, CSP_SPIKE…), with the lists they match (PROBE_PATHS, SCANNER_AGENTS, SIGN_IN_ROUTES: put your own sign-in paths there if you don't use auth-core). A detector is a block in Threats::findings; add yours there and a synthetic stream for it in security_detectors_fire_on_synthetic_streams.

• Wire a WAF or alerting: the dashboard detects, it doesn't block. To act on a finding, read HUB.threats (Threats::findings(now), each with its kind, severity and the address as shown) from a recurring job or the watcher (ai_watch::detect) and post to your alerting webhook through kernel::outbound; to block, put the address or the scanner's user agent in your proxy's or WAF's deny list (Cloudflare, AWS WAF, Caddy's remote_ip matcher), never in the app. In production the addresses are truncated and hashed, so a block rule needs the proxy's own logs.

• The beacon: thresholds (RAGE_CLICKS, RAGE_MS, DEAD_MS, SLOW_MS) are at the top of static/dashboard-rum.js; the server's caps and the Web Vitals thresholds are // APP: in kernel/dashboard/rum.rs. Name an element for the reports with data-rum="publish" (or a stable id); never put personal data in either. A clickable element the dead-click check should skip (it acts later on purpose): give it data-rum-async (or aria-disabled="true" while it works).

• Instrumenting your own methods (GUIDE: .offrails/guides/errors-and-logging.md#logging): one attribute on the function, and it shows in every waterfall nested with its SQL, in Methods, and in the slow-function insights:

  #[tracing::instrument(level = "debug", skip_all)]
  async fn load_items(db: &DatabaseConnection, order_id: i64) -> Result<Vec<order_item::Model>, DbErr> { … }

  skip_all keeps arguments out (a password, a user, a whole model); add a field only for what helps you debug and is neither a secret nor personal data: fields(order_id) then tracing::Span::current().record("order_id", order_id). Fields named like secrets show as [redacted] anyway, and the production view never shows fields. level = "debug" costs nothing in production (the filter is info there: the span is never built) and about a microsecond in development. Name a span handler (with handler = "orders::show") for a controller action, render for a template, http for a call out, job / mail / cron for the outermost span of background work (it becomes a trace of its own); everything else is a method. The waterfall shows its file:line from the span's metadata. Where a span has lots of hatched (uninstrumented) time, add one inside. The lab's N+1 page (lab.rs, load_orders_one_by_one → load_items) is the worked example.

• History on disk: DASHBOARD_HISTORY=false keeps everything in memory (as before); DASHBOARD_HISTORY_PATH moves the file (a mounted volume). Retention and the ceiling are MINUTE_DAYS, HOUR_DAYS, TRACE_DAYS, MAX_TRACES, MAX_BYTES (// APP: in kernel/dashboard/history.rs): about 70 KB a day for the charts, a few KB per notable trace. With the production view on, production writes the file too, without span fields or log events.

• Hot functions: DASHBOARD_PROFILER=false removes the sampler, always starts it at boot; FREQUENCY, WINDOW, MINUTES, MAX_STACKS, MAX_RUN are // APP: in kernel/dashboard/profiler.rs. WHY on demand: symbolizing a debug build maps its debug info, and pprof keeps its sample table once started (measured on the demo app: 40 MB RSS without it, 440 MB while sampling, 260 MB after stopping until a restart). "Your code" is the frames under this crate's name. On Windows the page says it isn't available (spans still work).

• Queueing: a request that waited WAIT_US (100 ms) or more before its code ran, for at least WAIT_SHARE percent of its time, counts; WAIT_MIN_TRACES of them make the saturation insight (// APP: in kernel/dashboard/insights.rs); rows shorter than WAIT_ROW_US aren't drawn (waterfall.rs).

• Slow functions: SLOW_METHOD_US, CALLS_PER_TRACE, GROWTH_MIN_US, HOT_CPU_SHARE (// APP: in kernel/dashboard/insights.rs). Repeats fold at GROUP_MIN (waterfall.rs).

• Load-test your own pages: point lab::seed (// APP: in kernel/dashboard/lab.rs) at your models, then put your routes in the Traffic form or app bench --route. Drop the lab_orders tables with a new migration when you no longer need them.

• A panel for your own subsystem (realtime, a payment queue): implement PanelProvider and add it to providers() (// APP: in kernel/dashboard/panels.rs; the realtime example is in its header).

• Your own insight rules: return findings from custom_rules (// APP: in kernel/dashboard/insights.rs); tune the thresholds at the top of that file (slow route p95, slow query p95, table sizes).

• The dashboard in production, read-only: DASHBOARD_IN_PRODUCTION=true and APP_ADMIN_PASSWORD (16+ characters), or your admin accounts: with auth-core installed, the optional wiring compiles kernel/dashboard/accounts.rs (signed-in people are asked first; is_admin is the // APP: hook, nobody until you add a role), otherwise write your own admin_from_session beside no_accounts in kernel/dashboard/access.rs. Behind a proxy set APP_TRUSTED_PROXY=true, or every client shares one failed-password budget (a warning says so once). It records in production then (a few µs per request).

• Send traces to a collector: set OTLP_EXPORT and fill export (// APP: in kernel/dashboard/trace.rs) with the opentelemetry crates.

• Group the diagram by your own areas: add ("billing_", "billing") rows to AREAS (// APP: in kernel/dashboard/erd.rs, first match wins); the colours are --g0…--g9 (/* APP: */ in dashboard.css). For a bigger schema, tune MAX_COLUMN and GRID_WIDTH there.

• Ask AI on OpenRouter (OpenAI-compatible, with tool calling), e.g. a cheap fast model. In .env (the key yours, never committed):

  DASHBOARD_AI_PROVIDER=openai-compatible
  DASHBOARD_AI_BASE_URL=https://openrouter.ai/api/v1
  DASHBOARD_AI_API_KEY=sk-or-…                        # your OpenRouter key
  DASHBOARD_AI_MODEL=deepseek/deepseek-v4.1-flash     # or openai/gpt-6-luna, z-ai/glm-5.3-flash
  DASHBOARD_AI_PRICE_IN=0.04                          # optional: USD per million tokens, for the spend estimate
  DASHBOARD_AI_PRICE_OUT=0.55
  DASHBOARD_AI_REFERER=http://localhost:3000          # optional: OpenRouter's HTTP-Referer
  DASHBOARD_AI_TITLE="My app dev dashboard"           # optional: OpenRouter's X-Title (quote values with spaces: an unquoted space stops .env loading there)

  Model ids come from OpenRouter's public list (curl -s https://openrouter.ai/api/v1/models); pick one whose supported_parameters include tools and tool_choice. On 2026-10-17 all three above did, listed at $0.04 / $0.55 (DeepSeek V4.1 Flash), $0.10 / $0.50 (GPT-6 Luna) and $0.15 / $0.50 (GLM-5.3 Flash) per million input / output tokens: cents per investigation. Check each model's data policy on OpenRouter before sending your app's data to it. Token use per run is kept down on every provider: a tool result older than the last step is cut to OLD_RESULT_CHARS (ai_agent.rs; the model can call the tool again), Anthropic (and anthropic/ models through OpenRouter) get cache_control marks on the system prompt and the conversation so far, and OpenAI-compatible providers such as DeepSeek cache a repeated prefix by themselves.

• Ask AI with another provider or tighter limits: any OpenAI-compatible endpoint works through DASHBOARD_AI_BASE_URL (https, or http to localhost: Ollama http://localhost:11434/v1, LM Studio http://localhost:1234/v1). Budgets and caps: the DASHBOARD_AI_* budget variables above; their defaults and the rest are PER_MINUTE, DAILY_REQUESTS, DAILY_CALLS, DAILY_TOKENS, MAX_STEPS, RUN_TOKENS, RUN_TIMEOUT, MAX_CONTEXT_CHARS, MAX_ANSWER_TOKENS, TIMEOUT (// APP: in kernel/dashboard/ai.rs), and MAX_TOOL_CHARS and the read_source allowlist (ai_tools.rs); the prompts and guides per page are Tab::suggestions and Tab::guides, the system prompt is system_prompt. To send more or less about a page, edit its builder in kernel/dashboard/ai_context.rs (each is // APP:): keep to redacted data, and extend ai_contexts_carry_no_rows_parameters_or_secrets when you add a source.

• Add your own tool (say, payments_summary): add a ToolDef row to TOOLS in kernel/dashboard/ai_tools.rs (a name, the step's title, a description that says what it returns and when to use it, and a JSON schema), and an arm in call that returns redacted text (ai_context::clean for free text; shapes and counts, never values); set core: true if small models should get it, read_only_view: true only if it is safe in production. Keep it read-only: an action belongs behind a button, as a new propose_action kind rendered in ai_web::action_view and posting to an existing CSRF-checked route. Add its arguments to every_tool_answers_within_its_cap. Another sample's data without a compile-time dependency: read its table by name (panels::table_exists, ai_context::text_rows, as the mailbox and jobs tools do), or register an ai_context::Source through optional wiring (as ai_realtime.rs does with realtime).

• Tune it for a small model (a local 7–30B model, a cheap tier): DASHBOARD_AI_TOOLS=core offers only the core tools (overview, insights, requests, compare, traces, SQL, logs, source, proposals; the core flags in TOOLS); lower MAX_STEPS to 4 and MAX_CONTEXT_CHARS to 8,000–12,000 so the first message fits a small context window; keep the suggested prompts concrete (Tab::suggestions), since small models do best with a precise question; and check the model's supported_parameters include tools (a model without tool calling still answers from the page data, without steps).

• Ask AI on more pages: Tab::for_page (ai.rs) maps a kernel page's path to a tab (and an item); add your page there, a Tab variant with its suggestions and guides, and its context builder in ai_context::build. The bar shows on any kernel page for_page knows.

• The command center: the verdict's rules (verdict), the briefing's length (MAX_BULLETS, MIN_BULLETS), the suggested questions (suggestions), the capability cards (guide) and the headline are // APP: in kernel/dashboard/ai_home.rs; its tint is --cc-wash and --cc-edge (/* APP: */ in dashboard.css). Each tab's one-line guide is Tab::guide_line (ai.rs).

• The watcher's signals: tune SPIKE_FACTOR, ERROR_BURST, JOB_FAILURES, DEBOUNCE, PER_HOUR, and its boot warm-up (WARMUP: the first look a minute after boot; BOOT_INVESTIGATIONS in the first BOOT_WINDOW, the most severe first; the Ask AI panel shows the watcher's share of today's tokens) (// APP: in kernel/dashboard/ai_watch.rs), or add a signal to detect; the agent investigates whatever detect returns.

• Ask AI in production (read-only view, its four tabs): DASHBOARD_AI_IN_PRODUCTION=true with DASHBOARD_IN_PRODUCTION=true and an admin gate. The provider then sees production's redacted SQL shapes and route numbers: decide that with whoever owns the data, and record it in docs/decisions.md.

• More history or a smaller footprint: RECENT, NOTABLE, TRACE_BYTES, LOG_LINES (trace.rs), MAX_SPANS, MAX_SQL (http.rs), MINUTE_POINTS (series.rs), MAX_FINGERPRINTS (sql.rs), N_PLUS_ONE and IGNORED (http.rs), MAX_RPS/MAX_SECS (lab.rs); every one is // APP:.

Do not change #

• Dev routes exist only in development; /admin/dashboard only with an admin gate (the_dashboard_is_404_in_production_by_default, the_production_dashboard_needs_the_admin_password).
• Parameters show as type and length, never values; EXPLAIN only for SELECTs (sql_parameter_values_never_reach_any_page, explain_runs_for_selects_only).
• The traffic generator takes paths, never URLs (traffic_only_targets_this_app).
• Every ring and map has a fixed size; the unmatched-path bucket is one bucket (the_dashboard_stays_within_its_memory_bound).
• The GitHub token stays in its header (the_github_token_is_never_shown_or_logged).
• no_store() on both routers (every_dashboard_response_is_no_store).
• The CSRF probe uses a method the route doesn't have, so it never runs a handler (a_mutating_route_without_csrf_is_spotted).
• Styles live in static/kernel/dashboard.css with light-dark() tokens, never in assets/app.css (every_dashboard_page_renders_with_its_own_assets, the_dashboard_stylesheet_follows_the_theme_switch).
• Ask AI starts only from a previewed first request built from redacted data, sends it to a base URL from the environment (https or loopback, no redirects), and renders answers as escaped text (only_the_previewed_context_is_sent, ai_contexts_carry_no_rows_parameters_or_secrets, the_ai_base_url_is_https_or_loopback, the_ai_client_handles_errors_timeouts_oversize_and_redirects, a_model_reply_with_markup_renders_inert).
• Every tool reads; an action is only a validated proposal the human clicks; tool output is delimited as untrusted data (a_prompt_injection_in_app_data_has_no_side_effect_and_leaks_nothing, proposed_actions_are_validated_and_render_as_buttons). read_source keeps its allowlist (read_source_stays_inside_the_allowlist). Citations link only to validated dashboard pages (citations_link_only_to_validated_dashboard_pages). Runs stay bounded and say why they stopped (an_agent_run_stops_at_its_budget_and_says_so), and a run's stream belongs to its browser (a_run_stream_is_bound_to_its_browser).
• The latest finished AI briefing's bottom line renders with the page, no click and no stream needed (the_ai_briefing_summary_shows_on_load).
• The overview's AI briefing starts on its own only in proactive mode, in development, once per process, inside the budget (the_command_center_briefs_without_a_key_and_with_one); the briefing, the graphs and the guide work without a key.
• A live run (its page and stream) is its browser's only; a finished one opens from its permalink in any browser that can open the dashboard, as the history does (a_run_stream_is_bound_to_its_browser).
• The AI key stays in its header (the_ai_key_never_reaches_logs_traces_pages_or_recorded_requests); Ask AI is 404 in production unless turned on behind the gate (the_ai_assistant_is_404_in_production_unless_enabled_behind_the_gate).
• erd.rs route/glyph and dashboard-erd.js route/glyph draw edges the same way: change them together (the_layout_is_deterministic_layered_and_overlap_free).
• The history file is 0600 in a 0700 folder, holds only what the dashboard already redacted, and in production no span fields or log events (the_history_file_is_private_and_redacted); a damaged one is moved aside, never fatal (a_damaged_or_incompatible_history_file_is_moved_aside).
• The CPU sampler never starts outside development (the_profiler_never_runs_outside_development).
• kernel::outbound follows no redirect, refuses credentials in a base URL, and records route templates only, never a query, a header value or a body (outbound_calls_are_traced_redacted_and_bounded).
• Production keeps client addresses as their network and a keyed hash; only the simulator's own secret lets a request claim an address (client_addresses_are_truncated_in_production, a_simulated_client_address_needs_the_simulators_secret).
• The attack simulator and the simulated APIs target this app's own address only, in development (the_attack_simulator_lights_up_every_detector, the_checkout_fan_out_traces_four_upstreams).
• The beacon sends no text, values or query strings, and the endpoint keeps only validated fields, same-origin, capped and rate-limited (rum_reports_keep_no_text_values_or_queries, the_rum_endpoint_is_same_origin_capped_and_rate_limited); it is off in production unless enabled (the_beacon_loads_in_development_and_in_production_only_when_enabled).

How it works #

Everything is in this process, in one hub (kernel/dashboard/mod.rs, HUB) of fixed-size rings and maps:

Annotations (boot, load-lab runs, error bursts over ERROR_BURST per second, the demo's deploy and incident) are kept with the series (200 at most) and drawn on every chart. A request's context lives in a Tokio task-local while it runs, so the SQL callback and the tracing layer attach to it without passing anything around. The dashboard's own pages and streams run "quietly" and record nothing. The diagram follows the charts' approach: erd.rs computes the layout and draws the SVG on the server (CSS-token colours, so light and dark need nothing), and dashboard-erd.js (about 400 lines, no library) moves it. WHY no layout library: a layered layout with a barycenter pass is about 150 lines for a schema of tens of tables, deterministic (so tests can pin it), and needs nothing vendored or from a CDN under the strict CSP. Ask AI is an agent loop on the server: the panel loads its context when opened (the preview is the exact first request: system prompt, page data and every tool's schema), and Investigate posts it (POST …/ai/runs, CSRF-checked) to start a run in the background. Each step sends the conversation with tools, runs the tool calls the model asks for (Anthropic tool_use blocks answered by one user turn of tool_result blocks; OpenAI-compatible tool_calls answered by one role: tool message each; several at once is fine), and appends both in that provider's shape; the last allowed call has tool_choice: none, so it must answer. The page follows the run over Server-Sent Events (GET …/ai/runs/{id}/stream, bound to the browser that started it): each step arrives as a fragment Askama rendered (title, the call, its result one click away), then the end: the answer as escaped blocks whose citation tags became links to the dashboard's own pages, the cited charts drawn by the Charts page's code, the proposed actions as buttons, and the fix prompt. WHY a POST then a GET stream: an EventSource can only GET, and the POST is what CSRF protects; WHY server-rendered fragments: one escaping path for everything the model writes. Without JavaScript the same form posts to …/ai/ask, which waits for the run and renders the finished page. The opt-in watcher (ai_watch.rs) reuses the same runs for its incident cards.

Charts are SVG rendered on the server (one implementation, works without JavaScript, nothing to vendor, the strict CSP untouched; colors are CSS tokens, so the theme switch needs no redraw); dashboard.js swaps in re-rendered fragments from a Server-Sent Events stream (every 5 s with prefers-reduced-motion, never while the tab is hidden), and adds the synchronized crosshair, the readout, zoom (?from=&to=, within the last 24 hours) and the click into /dev/dashboard/traces?minute=. The 1 h range and anything longer reads the history file through history::between (memory wins where both have a minute, so the hot/cold boundary has no seam); legends average every sample read, not the 300 drawn.

History on disk. WHY its own SQLite file and not the app's database: the app's migrations stay the app's, it works the same for Postgres apps, a db reset doesn't wipe it, and it is one file to delete. WHY a row per hour with a compressed blob: a minute is ~30 numbers that barely change; column-wise deltas as varints, then deflate, make an hour of minutes (histograms included) 1–3 KB, and a 24 h chart 24 row reads. Writes go through a channel to the history thread (never on a request), minutes as they close, traces in batches every 5 s; a crash loses the minute in progress. The thread answers the synchronous readers (charts, insights, AI tools) in a few milliseconds. Hours are rolled up from the minutes' merged histograms, so a 30-day chart's p95 is a true p95.

Method spans and the CPU sampler. The tracing layer already saw every span inside a request; now each carries its file:line, background work gets traces of its own (a root job, mail or cron span outside a request; its SQL finds the trace through the subscriber's registry), and the caps are 2,000 spans, 1,000 statements and 100 events per trace, past which they are counted ("Truncated"), with an 8 MB byte budget for the rings. method_spans_are_cheap measures about 1 µs per span in a debug build (each is a field visit, a timestamp and a push onto its trace; a request opening 200 of them), and fails above 25 µs. The sampler (pprof, signal-based) costs the samples themselves (99 per second per busy thread) plus a fold every 15 s, shown on the page (tens of milliseconds, on its own thread).

Overhead, measured by the_middleware_overhead_is_small (a trivial route behind the middleware alone, recording on and off on alternate requests, compared by median over 3,600 requests): 1.2 µs per request in a release build, about 5 µs in a debug build; the test fails above 10 µs (release) or, in debug, above 50 µs or 4× the bare request, whichever is larger (a CI runner is several times slower). A whole development request through every layer, with its SQL recorded and its trace and log line written, measured 6.3 µs more (97 → 103 µs for a static page, release). With the dashboard off (production by default, and tests) the middleware checks one atomic flag.

Files #