.offrails/guides/dev-tools.md

Dev Tools (/dev, a small Phoenix LiveDashboard)

TL;DR

  • Every app gets /dev (index, routes, system, mailbox, mailer previews) and /admin/jobs, built by new, never stubbed; mounted only in development (/admin/jobs needs admin access in production), which listens on 127.0.0.1 and refuses foreign Host names (no login on /dev).
  • Kernel pages have their own layout: every link into them carries hx-boost="false", every response is Cache-Control: no-store, and they use no confirm() dialogs.
  • Pending migrations: in development every HTML request shows a page with a Run migrations button; /up is 503 in production while any are pending.
  • Agents read email via /dev/mailbox/latest.json?to=…; the kernel-page browser smoke is .offrails/tools/browser-smoke/run.sh.
  • Ask the running app, don't guess: GET /dev/introspect.json (or app introspect) answers routes, redacted config, migrations, jobs, streams and subscribers, assets and CSS staleness. Development only; a test proves it's absent in production.
  • HTTPS in development only with a locally trusted certificate (mkcert). Chrome drops Set-Cookie on a self-signed one even after "Proceed (unsafe)", so login bounces. No mkcert: bin/dev serves plain HTTP and says why. Smokes never pass --ignore-certificate-errors.
  • Hot reload: bin/dev rebuilds and restarts on Rust/template edits, Tailwind watches CSS, and static/dev-reload.js reloads open tabs (a 1 s poll of /dev/reload, never a held stream).

Read the rest when changing a kernel page, debugging pending migrations, setting up dev HTTPS, or running the browser smoke.

Use the part. The dev-dashboard (and dev-mailbox) parts are this guide built, tested on SQLite and Postgres and security-reviewed, with more on top: /dev/dashboard, a LiveDashboard: live charts, requests per route, trace waterfalls, SQL fingerprints with N+1 detection and EXPLAIN, a live log tail, a routes explorer and a schema browser. When the product needs this, add the part rather than writing it from this guide: cargo offrails add-part dev-dashboard dev-mailbox (dependencies come first; workflow). Keep its // SECURITY: markers and the tests its CHECKS.md cites. This guide stays the spec: hand-write from it only when no part fits, or the human declined the part.

In development the app serves a toolbox at /dev. It's for the human clicking around and for the agent curling around. Keep it modest: it shows what the app already knows, with no metrics pipeline and no profiler.

generators/new.md builds every page below in every app. None is optional, stubbed or "added later".

RouteWhatSpec
GET /devIndex: tiles with live numbers. Mailbox (unread / total), Jobs (executing / available / retryable / discarded), Mailer previews (count), Routes (count), System (pending migrations). Try it buttons (Send a test email, Enqueue a test job). The agent curl cheat sheet belowhere
POST /dev/jobs/testEnqueue a harmless CleanupJob; redirect to /admin/jobs/list with "Enqueued test job N."here
POST /dev/migrationsThe pending-migrations page's Run migrations buttonbelow
/dev/mailbox (+ Previews tab at /dev/mailbox/previews)Every email the app sent, rendered; mailer previews in the same viewer; JSON for agentsmailers
/admin/jobsThe jobs dashboard (also in production, behind admin access)jobs-dashboard
GET /dev/routes?q=The ROUTES table: method, path, handler, access badge. One GET search box over all three columnsrouting
GET /dev/systemAPP_ENV, database backend + redacted host/file, FRONTEND_MODE, queues, inline worker, SMTP_URL configured or not, crate version + git sha (APP_REVISION), uptime; migrations applied and pending (Migrator::get_applied_migrations / get_pending_migrations_read_only, sea-orm-migration 2.0: the read-only variant never creates seaql_migrations)here
GET /dev/introspect.jsonThe app's state as JSON, for agentsbelow
GET /dev/reload{"version": "<boot id>:<asset digest>"}, polled by static/dev-reload.jsbelow
GET /dev/mailers301 → /dev/mailbox/previews (previews live in the mailbox now). Build it as (StatusCode::MOVED_PERMANENTLY, [(header::LOCATION, "/dev/mailbox/previews")]): axum's Redirect::permanent sends 308mailers

Files

src/controllers/dev.rs          router(): /dev, /dev/routes, /dev/system, /dev/sockets (with channels), /dev/introspect.json, /dev/reload, /dev/mailers (redirect), POST /dev/jobs/test, POST /dev/migrations
src/kernel/introspect.rs        snapshot(state, frontend_mode, live), BOOT_ID, reload_version() (the full file is below)
src/kernel/css.rs               class_lists, missing_classes (template classes with no rule in static/app.css), css_escape, has_selector
static/dev-reload.js            the reload poll (htmx mode; TS modes use Vite's own reload: islands rebuild with --watch and the poll sees the new digest)
src/controllers/migrations.rs   pending(), the development guard middleware, the Run button handler
src/kernel/mailbox.rs           router(PREVIEWS): /dev/mailbox/…
src/kernel/ui.rs                shared chrome (with the jobs dashboard), BOOTED_AT for uptime
templates/kernel/dev/           index.html, routes.html, system.html, pending.html, sockets.html + _sockets.html (polled every 2s)
templates/kernel/mailbox/       index.html, _list.html, _message.html

controllers/dev.rs holds a pub const FRONTEND_MODE: &str matching the app's AGENTS.md header. Config gains revision: Option<String> (APP_REVISION).

Rules

  • Mounted only when config.env == AppEnv::Development, in routes.rs:

    if config.is_development() {
        app = app
            .merge(dev::router())
            .merge(mailbox::router(crate::mailers::previews::PREVIEWS));
    }

    In production the routes don't exist (404, not hidden). Every /dev route is in ROUTES with Access::Dev, and security_test.rs requests each one with a production Config and expects 404.

  • Reachable from this machine only. /dev has no login, so development binds 127.0.0.1 (APP_BIND) and kernel::hosts refuses any Host that isn't localhost, APP_URL's or an APP_ALLOWED_HOSTS entry (DNS rebinding would otherwise read the mailbox same-origin). Opening it from a phone or a dev container takes APP_BIND=0.0.0.0 plus APP_ALLOWED_HOSTS, and exposes the mailbox to that network (security).

  • Server-rendered Askama + vendored htmx + static/kernel/admin.css + static/kernel/admin.js in every FRONTEND_MODE, sharing the jobs dashboard's header (tabs: Overview · Jobs · Cron · Dev · Mailbox · Sockets · Routes) and its flash banner.

  • Every link from an app page into a kernel page carries hx-boost="false": the "Dev tools" link, the auth screens' "Check your mailbox" hint (paths::dev_mailbox()), any nav item into /admin/jobs. The htmx-mode app layout has <body hx-boost="true">, and a boosted navigation swaps only <body>, so a kernel page (its own <head> and stylesheet) renders unstyled until a hard refresh. Verified in Chrome. kernel_web_test.rs::app_links_into_kernel_pages_are_not_boosted scans an app page and fails on any boosted /dev or /admin link. (Inertia/react apps: use a plain <a href>, never <Link>/router navigation, for kernel pages.)

  • Never cached, never stale on Back: every /dev, /dev/mailbox and /admin/jobs router ends with .layer(kernel::ui::no_store()), so all their responses (pages, fragments, JSON, downloads, redirects) are Cache-Control: no-store. The kernel layout sets historyCacheSize: 0, and admin.js reloads pages restored from the back/forward cache. URLs of things that no longer exist redirect with a flash instead of a bare 404 (deleted message → inbox, deleted job → job list).

  • Never display secrets: the database shows backend and host (or file path) only, never credentials (redact_db, unit-tested), and SMTP_URL shows "configured" or "not set (dev mailbox)".

  • In development the app layout's footer (htmx) or Layout component (inertia/spa) shows a small "Dev tools" link to /dev. Render it only when the server says env == development (Ctx.dev / a shared prop).

  • bin/dev migrates first, then prints the exact URL with its scheme, → https://localhost:${PORT} (or http:// on the fallback, below) and …/dev (mailbox, jobs, routes), and once more as ✓ ready: <url> (HTTP/2|HTTP/1.1) when the port accepts connections, so it isn't buried under build output.

HTTPS in development

Live pages need HTTP/2, and browsers only speak HTTP/2 over TLS (realtime). But a certificate the browser doesn't trust breaks login: after the user clicks "Proceed to localhost (unsafe)", Chrome silently drops Set-Cookie. The server answers 303 with a valid cookie, the account is created, and the browser bounces back to the login page ("email doesn't work"). Verified by a peer in a real Kanban; the same flow over http://localhost works.

So: dev serves HTTPS only with a locally trusted certificate.

Situationbin/dev-certbin/dev serves
mkcert installed and its CA installed (mkcert -install, once per machine)writes storage/dev-certs/{cert,key}.pem, exit 0HTTPS, HTTP/2: → https://localhost:3000
mkcert installed, CA not installed (mkcert prints "the local CA is not installed in the system trust store")refuses, deletes the files, says mkcert -install, exit 3plain HTTP/1.1, with the message below
no mkcertrefuses (no self-signed fallback), says how to install mkcert, exit 3plain HTTP/1.1, with the message below
DEV_CERT_SELF_SIGNED=1an openssl self-signed certificate, with a warning that browsers drop cookies on itboth, from one server: HTTPS + h2 on PORT (for curl -k and scripted h2 checks) and plain HTTP/1.1 on APP_DEV_HTTP_PORT (default PORT+1) for the browser. bin/dev prints both URLs and says to open the http one
APP_TLS=offnot runplain HTTP/1.1, no message

The fallback message says: no locally trusted certificate, so plain HTTP/1.1 at http://localhost:PORT; login and everything work; live updates stall past about 6 tabs (HTTP/1.1 allows ~6 connections per origin); install mkcert (brew install mkcert · apt install mkcert libnss3-tools), run mkcert -install once, restart bin/dev; self-signed certificates are refused because Chrome drops login cookies on them. bin/dev runs bin/dev-cert on every start (mkcert is instant), so a stale 0.6.0 self-signed certificate in storage/dev-certs/ is replaced or refused, never reused. The scripts are in new and realtime.

The bin/dev script

htmx mode (islands add pnpm --dir web exec vite build --watch --mode development &; inertia adds Vite; spa serves the Rust app over http behind Vite's proxy). Copy it; verified in the reference app in all three certificate situations. The PID file and process-tree stop were verified on macOS with a mock tree started from a non-interactive shell (the caller survives, every descendant and the PID file are gone, a second start refuses); --watch=always --minify with Tailwind 4.3.3 (a rebuild after a template edit stays minified):

#!/usr/bin/env bash
# Everything for development in one terminal: server (rebuilt on change) + Tailwind watcher.
set -euo pipefail
cd "$(dirname "$0")/.."
port="${PORT:-3000}"

# One PID file per server: stop exactly this bin/dev with `kill "$(cat storage/pids/dev-$PORT.pid)"`.
pidfile="storage/pids/dev-$port.pid"
mkdir -p storage/pids
if [ -f "$pidfile" ] && kill -0 "$(cat "$pidfile")" 2>/dev/null; then
  echo "bin/dev is already running on port $port (PID $(cat "$pidfile")). Stop it: kill \$(cat $pidfile)" >&2
  exit 1
fi
if command -v lsof >/dev/null && lsof -nP -iTCP:"$port" -sTCP:LISTEN >/dev/null; then
  echo "port $port is taken by another process (lsof -nP -iTCP:$port -sTCP:LISTEN); pick another: PORT=… bin/dev" >&2
  exit 1
fi
echo $$ > "$pidfile"
# On exit (Ctrl-C, `kill <pid>`, an error) stop our own process tree, leaves first, and nothing else.
# Never `kill 0`: started from an agent's shell, bin/dev shares that shell's process group, and
# `kill 0` would kill the caller too.
children() { ps -A -o pid= -o ppid= | awk -v p="$1" '$2 == p { print $1 }'; }
stop_tree() { local c; for c in $(children "$1"); do stop_tree "$c"; done; kill "$1" 2>/dev/null || true; }
cleanup() { trap - EXIT INT TERM; rm -f "$pidfile"; for c in $(children $$); do stop_tree "$c"; done; }
trap cleanup EXIT
trap 'exit 130' INT
trap 'exit 143' TERM

# HTTPS + HTTP/2 only with a locally trusted certificate (bin/dev-cert: mkcert + `mkcert -install`).
# Without one, plain HTTP/1.1: login works, live pages stall past ~6 tabs. APP_TLS=off forces http.
scheme=http
open_url=""   # set when the URL a person should open differs from the main one
if [ "${APP_TLS:-on}" != "off" ]; then
  if [ "${DEV_CERT_SELF_SIGNED:-}" = 1 ] && [ -f storage/dev-certs/cert.pem ] || bin/dev-cert; then
    export APP_TLS_CERT=storage/dev-certs/cert.pem APP_TLS_KEY=storage/dev-certs/key.pem
    scheme=https
    if [ "${DEV_CERT_SELF_SIGNED:-}" = 1 ]; then
      # Untrusted certificate: browsers drop cookies on it. Same server, plain HTTP beside it.
      export APP_DEV_HTTP_PORT="${APP_DEV_HTTP_PORT:-$((port + 1))}"
      open_url="http://localhost:$APP_DEV_HTTP_PORT"
    fi
  else
    cat <<MSG
!
! No locally trusted certificate, so bin/dev serves plain HTTP/1.1 at http://localhost:$port
!   Login and everything else work. Live updates stall past about 6 tabs (HTTP/1.1 allows ~6
!   connections per origin; HTTP/2 needs HTTPS).
!   For HTTPS + HTTP/2: install mkcert (brew install mkcert · apt install mkcert libnss3-tools),
!   run 'mkcert -install' once, then restart bin/dev. Self-signed certificates are refused:
!   Chrome drops login cookies on them even after "Proceed (unsafe)". APP_TLS=off hides this.
!
MSG
  fi
fi

echo "Compiling and migrating (the first build takes several minutes)…"
cargo run -q -- db migrate

# Rebuild CSS on every template change, so a new class is never missing from static/app.css.
# --minify as in the production build: without it the watcher overwrites a minified file with an
# unminified one, and any check of the built CSS sees a different file in development.
bin/tailwindcss -i assets/app.css -o static/app.css --watch=always --minify &
# Migrate on every rebuild too, so a new migration never meets a stale database.
cargo watch -q -w src -w templates -w migration -x 'run -q -- db migrate' -x 'run -- serve' &

# The exact URL, scheme included (a bare localhost:$port is http:// to the browser), again once the
# server accepts connections, so it isn't buried under the build output.
if [ -n "$open_url" ]; then
  echo "→ open in your browser: $open_url (HTTP/1.1; the certificate on :$port is self-signed and browsers drop cookies on it)"
  echo "→ for scripts and h2 checks: $scheme://localhost:$port (curl -k; never a browser login)"
else
  echo "→ $scheme://localhost:$port"
fi
echo "→ ${open_url:-$scheme://localhost:$port}/dev (mailbox, jobs, routes)"
(until (exec 3<>"/dev/tcp/127.0.0.1/$port") 2>/dev/null; do sleep 1; done
 echo "✓ ready: ${open_url:-$scheme://localhost:$port} ($([ "$scheme" = https ] && [ -z "$open_url" ] && echo "HTTP/2" || echo "HTTP/1.1"))") &
wait

APP_DEV_HTTP_PORT (development only; Config refuses it in any other environment, and it does nothing without TLS): serve binds a second, plain listener and serves the same router and AppState on it, so streams, presence and sessions are shared with the TLS port. The CSRF layer also trusts http://localhost:<port>. Dev session cookies aren't Secure, so the browser keeps them there. Verified by a peer (dev-92) in a stock Chrome profile and in the reference app: login sticks and the page goes live on the http port, while the self-signed TLS port fails the login smoke's preflight. bin/dev in that mode:

→ open in your browser: http://localhost:3001 (HTTP/1.1; the certificate on :3000 is self-signed and browsers drop cookies on it)
→ for scripts and h2 checks: https://localhost:3000 (curl -k; never a browser login)

A bare localhost:3000 typed into the address bar is http:// to the browser. Against a TLS listener that used to show ERR_INVALID_HTTP_RESPONSE. serve's TLS branch now sniffs the first byte of each connection (a TLS handshake starts with 0x16) and answers anything else with a 307 to https://<Host><path> (kernel/tls.rs, deployment). 307, not 308: a cached permanent redirect would break APP_TLS=off on the same port later.

Never tell a user to click through the warning, and never run a browser check with --ignore-certificate-errors or acceptInsecureCerts: that's exactly the setting under which the bug is invisible. The framework's smokes launch Chrome with no certificate overrides and fail on an untrusted certificate (below).

Hot reload

Templates are compiled (Askama), so "hot reload" is the boring loop, all inside bin/dev:

  1. Rust and templates: cargo watch -w src -w templates -w migration rebuilds, migrates and restarts serve.
  2. CSS: bin/tailwindcss -i assets/app.css -o static/app.css --watch=always --minify rewrites the CSS when a template or assets/app.css changes. In development the asset catalog rescans per request, so the next page gets the new digest.
  3. The browser: the layout loads static/dev-reload.js when ctx.dev. It polls GET /dev/reload once a second while the tab is visible and calls location.reload() when version changes: a new BOOT_ID (the server restarted) or a newer file under static/ (version is BOOT_ID:<newest mtime>:<file count>; mtimes, not digests, because every open tab asks once a second). While the server is down mid-rebuild, the fetch fails and it keeps polling.

Why a poll and not the signal stream or a socket: on the plain-HTTP fallback every held connection costs one of the ~6 per origin, and a reload stream in every tab would halve the tabs a live page survives. One short request a second costs nothing. Measured in the reference app: a CSS-only edit reached an open tab in 0.6 s; a template edit in about 17 s, of which ~9 s is the incremental build inside the watcher (Tailwind rebuilding alongside), ~1.7 s startup, up to 1 s the poll, the rest the page load.

Faster rebuilds: the skeleton Cargo.toml has

[profile.dev]
debug = "line-tables-only"   # backtraces keep file:line; lldb shows no locals

and .cargo/config.toml adds -C split-debuginfo=unpacked on Linux (macOS's default already; Windows MSVC refuses it, which is why it left [profile.dev] in 0.7.2). More build-speed settings, and what was measured: testing.

A peer measured a template edit going from 16–23 s to 3.6–5.9 s and a Rust edit from 11–12 s to 4.4–5.3 s (a Kanban, M2). In the smaller reference app on a newer machine incremental builds were ~5 s with or without it (not reproduced there; it costs nothing). For a debugging session with locals: CARGO_PROFILE_DEV_DEBUG=true cargo run. A page with unsaved form input loses it on reload; that's development.

// static/dev-reload.js: development only (the layout loads it when ctx.dev)
(() => {
  let seen = null;
  const tick = async () => {
    if (document.visibilityState === "visible") {
      try {
        const r = await fetch("/dev/reload", { cache: "no-store", headers: { Accept: "application/json" } });
        const v = r.ok && r.headers.get("content-type")?.includes("json") ? (await r.json()).version : null;
        if (v && seen && v !== seen) return location.reload();
        if (v) seen = v;
      } catch {
        // the server is restarting: keep polling
      }
    }
    setTimeout(tick, 1000);
  };
  tick();
})();

The layout line is {% if ctx.dev %}<script src="/static/dev-reload.js" defer></script>{% endif %}. /dev/reload and /dev/introspect.json pass the pending-migrations guard (like /up), so they answer while migrations are pending. TS modes: inertia/spa reload through Vite's own dev server or vite build --watch (islands), which changes the digest the poll sees.

Introspection

One way for an agent to ask the running app about its state, instead of grepping logs and guessing. GET /dev/introspect.json (development only: it's in dev::router(), so it doesn't exist in production or test) and app introspect (the same snapshot from the CLI; refuses unless APP_ENV=development; without the live parts, which only a serving process knows):

KeyWhat
appname, version, frontend_mode, boot_id, booted_at
urlapp_url, tls, the browser protocol (h2, or http/1.1 and its ~6-tab limit)
configthe Debug of Config: an allowlist, values only for never-secret settings and set/not set for the rest (config; never a raw env dump)
routesevery ROUTES row: method, path, handler, access
migrationsapplied count, pending names, current
jobscounts by state and by queue (admin::stats); null while the table doesn't exist
realtime (live)open signal subscriptions, connections per member and transport, channel topics with subscribers and presence, frame counters, node(s). With per-stream pub/sub (realtime) it lists each stream and its subscriber count
assetsthe layout's digested asset URLs
cssbuilt, stale, and up to 50 missing_classes (template classes with no rule in static/app.css, from kernel::css::missing_classes)

The whole file. Copy it, then change only the lines marked // APP:. It's the reference implementation an upgrading app wrote from this spec (axum 0.8.9, sea-orm 2.0.4, 2026-10-13), with the app-specific lines generalized:

//! Ask the running app, don't guess (.offrails/guides/dev-tools.md#introspection): the snapshot behind
//! `GET /dev/introspect.json` and `app introspect`, and the version `GET /dev/reload` hands to
//! `static/dev-reload.js`. Development only: the routes live in `controllers::dev::router()`, which
//! production and test never mount, and `app introspect` refuses outside development.

use std::fs;
use std::path::Path;
use std::sync::LazyLock;
use std::time::SystemTime;

use migration::{MigrationStatus, Migrator, MigratorTrait}; // migration/src/lib.rs: `pub use sea_orm_migration::MigrationStatus;`
use serde_json::{Map, Value, json};

use crate::kernel::ui::BOOTED_AT;
use crate::kernel::{css, jobs::admin};
use crate::routes::ROUTES;
use crate::state::AppState;

/// One id per process: a restart changes it, so open tabs reload.
pub static BOOT_ID: LazyLock<String> =
    LazyLock::new(|| format!("{}-{}", std::process::id(), BOOTED_AT.timestamp_micros()));

/// The most missing classes listed (the count is always exact).
const MAX_MISSING: usize = 50;

/// `BOOT_ID:<newest mtime under static/ in ms>:<file count>`. Mtimes, not digests: every open tab
/// asks once a second.
pub fn reload_version() -> String {
    let (mut newest, mut count) = (0u128, 0usize);
    walk(Path::new("static"), &mut newest, &mut count);
    format!("{}:{newest}:{count}", *BOOT_ID)
}

fn walk(dir: &Path, newest: &mut u128, count: &mut usize) {
    let Ok(entries) = fs::read_dir(dir) else { return };
    for entry in entries.flatten() {
        let path = entry.path();
        if path.is_dir() {
            walk(&path, newest, count);
        } else if let Ok(modified) = entry.metadata().and_then(|m| m.modified()) {
            *count += 1;
            let ms = modified.duration_since(SystemTime::UNIX_EPOCH).map(|d| d.as_millis()).unwrap_or(0);
            *newest = (*newest).max(ms);
        }
    }
}

/// The app's state as JSON. `live` (the HTTP handler) adds what only a serving process knows.
pub async fn snapshot(state: &AppState, frontend_mode: &str, live: bool) -> anyhow::Result<Value> {
    let c = &state.config;
    let tls = c.tls.is_some();
    let migrations = Migrator::get_migration_with_status(&state.db).await?;
    let applied = migrations.iter().filter(|m| matches!(m.status(), MigrationStatus::Applied)).count();
    let pending: Vec<String> = migrations
        .iter()
        .filter(|m| matches!(m.status(), MigrationStatus::Pending))
        .map(|m| m.name().to_owned())
        .collect();
    // null while the jobs table doesn't exist yet (a fresh database before `db migrate`).
    let jobs = match admin::stats(&state.db).await {
        Ok(s) => json!({
            "by_state": s.by_state.iter().map(|(st, n)| (state_name(st), json!(n))).collect::<Map<_, _>>(),
            "by_queue": s.by_queue.iter().map(|(q, counts)| {
                (q.clone(), Value::Object(counts.iter().map(|(st, n)| (state_name(st), json!(n))).collect()))
            }).collect::<Map<_, _>>(),
        }),
        Err(_) => Value::Null,
    };
    let css = match css::missing_classes() {
        None => json!({ "built": false, "stale": false, "missing_count": 0, "missing_classes": [] }),
        Some(missing) => json!({
            "built": true,
            "stale": !missing.is_empty(),
            "missing_count": missing.len(),
            "missing_classes": missing.iter().take(MAX_MISSING).collect::<Vec<_>>(),
        }),
    };
    let assets = state.assets.assets();
    Ok(json!({
        "app": {
            "name": crate::config::APP_NAME,
            "version": env!("CARGO_PKG_VERSION"),
            "framework": fs::read_to_string(".offrails/VERSION").ok().and_then(|v| v.lines().next().map(str::to_owned)),
            "frontend_mode": frontend_mode,
            "env": format!("{:?}", c.env).to_lowercase(),
            "boot_id": *BOOT_ID,
            "booted_at": BOOTED_AT.to_rfc3339(),
        },
        "url": {
            "app_url": c.app_url,
            "tls": tls,
            "dev_http_port": c.dev_http_port,
            "protocol": if tls { "h2" } else { "http/1.1" },
            "note": if tls {
                "HTTPS with ALPN h2 (a locally trusted certificate)"
            } else {
                "plain HTTP/1.1: browsers hold ~6 connections per origin, so live pages stall past ~6 tabs"
            },
        },
        "config": format!("{c:?}"), // Config's hand-written Debug redacts credentials (config.md)
        "routes": ROUTES.iter().map(|(method, path, handler, access)| json!({
            "method": method, "path": path, "handler": handler, "access": format!("{access:?}"),
        })).collect::<Vec<_>>(),
        "migrations": { "applied": applied, "pending": pending, "current": pending.is_empty() },
        "jobs": jobs,
        "realtime": if live { realtime(state) } else { Value::Null },
        "live": live,
        "assets": { "css": assets.css, "app_js": assets.app_js, "htmx": assets.htmx, "htmx_sse": assets.htmx_sse },
        "css": css,
    }))
}

fn state_name(st: &impl std::fmt::Debug) -> String {
    format!("{st:?}").to_lowercase()
}

/// APP: what only the serving process knows about live connections. No realtime: `Value::Null`.
fn realtime(state: &AppState) -> Value {
    json!({
        // pubsub.streams(): (stream, receiver count) per live per-stream sender (realtime.md)
        "streams": state.pubsub.streams().into_iter().map(|(s, n)| json!({ "stream": s, "subscribers": n })).collect::<Vec<_>>(),
        "topic_count": state.pubsub.topic_count(),
        // with channels: the /dev/sockets snapshot (connections per member and transport, topics, presence,
        // frame counters, nodes). `Snapshot` derives Serialize.
        // "channels": serde_json::to_value(state.hub.snapshot()).unwrap_or(Value::Null),
        // with SSE presence: the label per room
        // "presence": state.presence.labels(),
    })
}

#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn reload_version_starts_with_the_boot_id() {
        let v = reload_version();
        assert!(v.starts_with(&format!("{}:", *BOOT_ID)), "{v}");
        assert_eq!(v.split(':').count(), 3, "{v}");
    }
}

The handlers (controllers/dev.rs, both in router() and in ROUTES with Access::Dev) and the CLI arm:

/// `GET /dev/introspect.json`: the app's state for agents.
pub async fn introspect(State(s): State<AppState>) -> Result<Response, AppError> {
    Ok(Json(kernel::introspect::snapshot(&s, FRONTEND_MODE, true).await?).into_response())
}

/// `GET /dev/reload`: polled once a second by `static/dev-reload.js`.
pub async fn reload() -> Response {
    Json(serde_json::json!({ "version": kernel::introspect::reload_version() })).into_response()
}

// main.rs, `Command::Introspect` (needs the database, not the listener):
anyhow::ensure!(config.is_development(), "app introspect is development only (APP_ENV is {:?})", config.env);
let state = AppState::new(config, db)?;
println!("{}", serde_json::to_string_pretty(&kernel::introspect::snapshot(&state, controllers::dev::FRONTEND_MODE, false).await?)?);

The router's .layer(kernel::ui::no_store()) already covers both responses. Apps without the jobs kernel drop jobs (and its import); api mode has no css or assets (drop both keys and the css import). Add pub fn streams(&self) -> Vec<(String, usize)> to PubSub (under the map lock: each stream and its receiver_count()) when the app has signals.

Tests (kernel_web_test.rs): in development it answers JSON with every key above, lists itself in routes, and contains none of a configured admin password or SMTP credentials; with a production and a test Config, /dev/introspect.json and /dev/reload are 404 and pages don't load dev-reload.js. Add both paths to ROUTES with Access::Dev.

The development CSS banner uses the same check. Every UI app has one: if Ctx has no css_warning field yet, add it (Option<String>, set by the Ctx extractor only when config.is_development(), rendered as the first plain-text line in <main>; frontend-htmx). An mtime check can't do this: Tailwind 4 doesn't rewrite unchanged output.

// src/kernel/css.rs (next to missing_classes)
/// The development CSS banner: the CSS is missing, or stale (template classes with no rule).
pub fn css_warning() -> Option<String> {
    if let Some(problem) = crate::kernel::assets::app_css_problem() {
        return Some(problem);
    }
    let missing = missing_classes()?;
    let first = missing.first()?;
    Some(format!(
        "{CSS_BUILT} is stale: {} template class(es) have no CSS rule (first: {first}). bin/dev rebuilds it.",
        missing.len()
    ))
}
// the Ctx extractor:
css_warning: if s.config.is_development() { crate::kernel::css::css_warning() } else { None },

No confirm() dialogs in kernel pages

Kernel pages aren't boosted, so hx-confirm never fires there, and a native confirm() dialog would block the browser smoke tool. Destructive kernel actions are either cheap and dev-only (mailbox Clear all, deleting a job: no prompt), or go through a server-rendered confirmation page ("Retry 1,234 jobs?", which posts back with confirmed=1). If an app page needs a quick confirm, use data-confirm="…" on the form, handled in static/app.js (submit listener calling confirm()), and make browser tests accept dialogs (page.on('dialog', d => d.accept())).

Pending migrations

Like Rails' pending-migration error page and Phoenix's "There are pending migrations" screen: a database behind the code shows a clear page, not a "no such table" 500.

src/controllers/migrations.rs (generated by new):

/// Read-only: never creates seaql_migrations (a fresh database reports every migration pending).
pub async fn pending(db: &DatabaseConnection) -> Result<Vec<String>, DbErr> {
    Ok(Migrator::get_pending_migrations_read_only(db).await?.iter().map(|m| m.name().to_owned()).collect())
}
pub async fn guard(State(s): State<AppState>, OriginalUri(uri): OriginalUri, req: Request, next: Next) -> Response;
pub async fn run(State(s): State<AppState>, Form(f): Form<RunForm>) -> Result<Response, AppError>;   // POST /dev/migrations
EnvironmentBehavior
developmentroutes.rs adds app.layer(from_fn_with_state(state.clone(), migrations::guard)) after the routes and the .fallback(...) are added (a layer only wraps what exists when it's applied, so a fallback added later would skip the guard and unknown paths would 404 instead of showing the page). Every request except /static/*, /up and POST /dev/migrations gets a 503 "N pending migrations" page (kernel layout, so it works before any app table exists) listing them, with a Run migrations button: a CSRF-protected form posting return_to=<current path> to /dev/migrations, which runs Migrator::up, flashes "Ran N migrations." and redirects back (only to same-site paths)
productionNever auto-migrates: the entrypoint runs app db migrate before serve (deployment). The guard and /dev/migrations aren't mounted. GET /up returns 503 "pending migrations" while any is pending, so a deploy that skipped migrating never goes healthy
worker (app worker, inline worker)Before starting, checks pending(). If any are pending it logs once pending migrations: run \app db migrate`; worker waiting, rechecks every 5s (stopping cleanly on shutdown), then logs migrations applied; worker starting`. It never spams errors against missing tables

Nothing in serve may read an app table before the listener is up. The guard renders from the kernel layout and seaql_migrations alone; everything else that touches app tables at boot (realtime's pub/sub and cluster, the inline worker, a cache warm-up) waits for pending() to be empty (realtime). Fresh-database smoke, after any change to serve or to what starts at boot (bin/dev migrates first, so it never shows this):

mkdir -p storage/pids && rm -f storage/fresh.sqlite3
PORT=4321 DATABASE_URL='sqlite://storage/fresh.sqlite3?mode=rwc' sh -c 'echo $$ > storage/pids/serve-4321.pid; exec cargo run -q -- serve' & sleep 5
curl -s -o /dev/null -w '%{http_code}\n' http://localhost:4321/          # 503: the pending-migrations page, not a crash
curl -s -o /dev/null -w '%{http_code}\n' -X POST -H 'Sec-Fetch-Site: same-origin' -d return_to=/ http://localhost:4321/dev/migrations   # 303
sleep 6; curl -s -o /dev/null -w '%{http_code}\n' http://localhost:4321/up   # 200; the log says "migrations applied; realtime starting"
kill "$(cat storage/pids/serve-4321.pid)"; rm -f storage/pids/serve-4321.pid storage/fresh.sqlite3*

bin/dev runs cargo run -- db migrate before serving, and again on every rebuild (cargo watch -x 'run -q -- db migrate' -x 'run -- serve'), so the page is mostly seen after a git pull with the server stopped, or after a rollback. Tests (kernel_web_test.rs): a fully rolled-back dev DB shows the page (with the migration names, the button and return_to), /up is 503, the kernel stylesheet still loads, the button applies all and redirects back with the flash, no open redirect; production with one pending: /up 503, / unaffected, /dev/migrations 404.

Browser smoke

HTTP tests can't see boosted navigation, <head> swaps, the back/forward cache or CSP violations. After changing kernel pages (or the app layout's links into them), drive a real browser:

  1. Preferred: the chrome-devtools MCP (navigate, click, screenshot, list_console_messages), in an isolated context.

  2. When it can't attach (Chrome's profile lock, CI, a script): the framework ships a ready smoke that launches the installed Chrome headless with a throwaway profile:

    BASE=http://localhost:3000 bash .offrails/tools/browser-smoke/run.sh
    # Linux: CHROME=/usr/bin/google-chrome BASE=… bash .offrails/tools/browser-smoke/run.sh

    It opens /dev directly (styled?; an app's home need not link to it) → seeds 3 mails with the Send a test email button (each lands on its message, /dev/mailbox/<id>, with the flash "Sent a test email to your mailbox."; the list then shows at least 3) → opens one (styled, sandboxed iframe) → Clear all → Back (none of the cleared messages comes back) → Forward (judged by the cleared messages' ids, not an empty list: other smokes verify --full runs alongside sign in, and an auth part's "New sign-in" email may arrive meanwhile) → (when /dev has an Enqueue a test job button) opens and deletes the job → Back (redirected to the list, flash shown), and fails on any CSP violation. Every step waits with waitForNavigation/waitForSelector, with no fixed sleeps, and seeds only through the dev-only buttons, so it needs no app routes.

  3. Multi-tab live updates (any app with live pages): open N tabs in one profile on a live page, change something once, and assert every tab updates. It catches the HTTP/1.1 connection pool and an undersized per-user cap (realtime). The app supplies a small scenario module, tests/e2e/live-tabs.scenario.mjs: export const path (the live page), optional login(page, base), mutate(page, base) returning a token, and seen(page, token) returning whether a tab shows it (the full contract is the header of .offrails/tools/browser-smoke/smoke-live-tabs.mjs):

    SCENARIO=tests/e2e/live-tabs.scenario.mjs BASE=https://localhost:3000 TABS=12 \
      bash .offrails/tools/browser-smoke/run.sh live-tabs

    It exits 1 if any tab misses the update, and says whether the page was on HTTP/1.1. Run it with 12 and 40 tabs after any realtime change.

  4. Login sticks (every app with accounts; part of the UI check in the Definition of Done): a real browser with no certificate overrides fills and submits the real login (or sign-up) form, then proves a fresh page load is still logged in. It catches the dev-TLS trap above and any cookie a browser refuses (Secure on http, __Host- with a Domain). The scenario exports login(page, base) and loggedIn(page):

    SCENARIO=tests/e2e/smoke.scenario.mjs BASE=<the URL bin/dev printed> bash .offrails/tools/browser-smoke/run.sh login
  5. axe in both color schemes: axe-core (WCAG 2.2 AA) on each path with prefers-color-scheme emulated light, then dark; logged-out paths from PATHS or the scenario's a11yPaths, and a11yAuthPaths after login. It finds a bg-white card left light under a body that turned dark. The app's Playwright suite (test-standards §10) stays the CI gate; this is the framework's own check, and it needs no Node in the app:

    SCENARIO=tests/e2e/smoke.scenario.mjs BASE=… bash .offrails/tools/browser-smoke/run.sh a11y
  6. Every page after a boosted link (every htmx app): each GET page in docs/routes.md on a full load, then reached through a boosted link from /; the same buttons must react, with no console errors, and no page may load its own scripts in <head>. Signed out, then signed in as the demo account. No npm install (Node ≥ 18 and the installed Chrome); verify --full runs it when BASE is set (testing):

    BASE=<the URL bin/dev printed> node .offrails/tools/browser-smoke/smoke-boosted.mjs   # from the app root

Every smoke launches Chrome without certificate overrides and starts with a preflight: an untrusted HTTPS certificate fails it with the explanation above, before any check runs. Verified in the reference app: against a forced self-signed certificate (DEV_CERT_SELF_SIGNED=1) login and live-tabs fail at the preflight; against the plain-HTTP fallback and against a trusted certificate both pass, and live-tabs reaches 12/12 tabs over h2.

Why it lives in .offrails/tools/ and not in the app: htmx-mode apps have no Node toolchain or package.json, and this check isn't part of the Definition of Done or CI. So run.sh installs puppeteer-core (pinned, verified 2026-10-12: 25.13.0, which needs Node ≥ 22.12; Node 24 LTS recommended) and axe-core (4.13.0) once into ~/.cache/offrails/browser-smoke and runs a copy of the script from there. The app's dependency tree, lockfiles and stack rules are untouched, and inertia/spa apps use the same script. It's optional tooling for agents and humans; the guaranteed checks remain the Rust tests above. Verified against the bookshelf app: 11/11 checks pass. Removing hx-boost="false" from the dev link, or omitting includeIndicatorStyles:false, makes it fail.

For agents

These endpoints are your eyes on the running app. Use them in the "ran the app and exercised it" step of the Definition of Done. Start with introspection; it answers most "why doesn't X work" questions before you read code:

curl -s $URL/dev/introspect.json | jq '.migrations, .css, .realtime, .url'  # $URL: what bin/dev printed (-k never needed)
cargo run -q -- introspect | jq '.routes[] | select(.path | test("books"))'
curl -s "$URL/dev/mailbox/latest.json?to=ada@example.com"  # did the email go out? follow .links[0]
cargo run -- jobs list --state discarded --json                      # did a job fail?
cargo run -- jobs stats                                              # is anything stuck?
curl -s $URL/dev/system | grep -i pending                       # forgot to migrate?