Testing #

TL;DR

• How to test (test-standards.md says what): one integration-test binary in tests/app/, TestApp over the real router, factories, both SQLite and Postgres.
• Postgres tests never run CREATE DATABASE or migrations per test: they lock a reusable slot database cloned once from a migrated template, reset to the just-migrated state (§Postgres test databases); the SQL lives in kernel::db.
• While iterating, cargo offrails verify (fmt, clippy, one build, both suites at once); --full once at the end (workflow).
• Helpers set Fetch Metadata like a browser; assert on what the user sees (assert_text decodes entities), never exact HTML; no sleeps (drain_jobs, wait_until).
• CI runs SQLite, Postgres, security (cargo-deny, gitleaks) and the Docker image with a ZAP baseline.

Read the rest when writing the harness, factories, frontend tests, or CI.

This guide is how to write tests (TestApp, factories, layout). What must be tested, and the coverage and quality gates, are in test-standards.md. Read both.

Tests are how the agent proves the spec. Every acceptance criterion in product/features/*.md becomes one integration test, named after the criterion. The same suite runs on SQLite (default, fast) and Postgres (CI, and locally when available). No mocked database, ever.

Layout: one test binary #

tests/app/main.rs          mod factories; mod smoke_test; mod security_test; mod posts; …   (generated lines marked `// posts`)
                           #[allow(dead_code)] mod support;   // shared helpers; clippy -D warnings flags unused ones otherwise
tests/app/support.rs       TestApp, TestResponse
tests/app/factories.rs     user, … + `mod post; pub use post::*;` per generated model (tests/app/factories/post.rs)
tests/app/smoke_test.rs    generated by `new`
tests/app/security_test.rs access matrix over ROUTES (test-standards.md §3)
tests/app/kernel_test.rs   cache, rate limit, flash (generated by `new`)
tests/app/errors_test.rs   AppError × response flavor, ClientIp, tasks (generated by `new`)
tests/app/posts/           a generated resource: mod.rs (the form submit), read_test.rs, write_test.rs, access_test.rs
tests/app/models/post_test.rs   a generated model's queries, both databases

Every file in tests/ root is a separate binary that links the whole app, so one binary with modules keeps link times sane. Never add files directly under tests/.

TestApp spec (tests/app/support.rs) #

pub struct TestApp {
    pub state: AppState,           // AppState::new(config, db): Config::test(), Mailer::Memory, StorageConfig::Memory
    router: Router,                // routes::router(state.clone())?
    cookies: Mutex<Vec<(String, String)>>, // persists Set-Cookie between requests, like a browser (removal cookies delete)
    fetch_metadata: AtomicBool,    // form posts send Sec-Fetch-Site: same-origin unless turned off
    origin: Mutex<Option<String>>, // Origin header for form posts, if set
    queries: Arc<AtomicUsize>,     // SQL statement counter (below)
    db_guard: DbGuard,             // holds the test's Postgres slot (its advisory lock); a no-op on SQLite
}

impl TestApp {
    pub async fn new() -> Self;                          // fresh DB, migrated, empty
    pub async fn with_config(config: Config) -> Self;    // e.g. env = Production to prove /dev isn't mounted
    pub async fn request(&self, req: Request<Body>) -> TestResponse;   // any method/headers; cookies applied and stored
    pub fn db(&self) -> &DatabaseConnection;
    pub async fn get(&self, path: &str) -> TestResponse;
    pub async fn get_with_headers(&self, path: &str, headers: &[(&str, &str)]) -> TestResponse;   // e.g. ("host", "evil.example")
    pub async fn post_form(&self, path: &str, form: &[(&str, &str)]) -> TestResponse;
    pub async fn post_form_with_headers(&self, path: &str, form: &[(&str, &str)], headers: &[(&str, &str)]) -> TestResponse;
    pub async fn post_json(&self, path: &str, body: &impl Serialize) -> TestResponse;
    pub async fn post_multipart(&self, path: &str, fields: &[(&str, &str)],
                                files: &[(&str, &str, &str, &[u8])]) -> TestResponse;   // (field, filename, content type, bytes)
    pub async fn get_htmx(&self, path: &str) -> TestResponse;   // HX-Request: true
    pub async fn open_stream(&self, path: &str) -> (StatusCode, Body);   // SSE: unread body for support::next_event
    pub async fn request_head(&self, path: &str, headers: &[(&str, &str)]) -> (StatusCode, HeaderMap);   // GET, status + headers, body never read: SSE and other endless responses
    pub fn without_fetch_metadata(&self) -> &Self;       // later posts omit Sec-Fetch-Site (CSRF falls back to Origin)
    pub fn with_origin(&self, origin: &str) -> &Self;    // later posts send this Origin
    pub fn cookie(&self, name: &str) -> Option<String>;
    pub fn set_cookie(&self, name: &str, value: &str);   // e.g. replay an old session token
    pub fn logout_locally(&self);                        // drop every cookie (a fresh browser)
    pub async fn inertia_get(&self, path: &str) -> TestResponse;   // inertia: X-Inertia + version headers
    pub async fn login_as(&self, user: &User);           // session::start + stores the cookie; call again to switch users
    pub async fn drain_jobs(&self) -> Drained;           // run all due jobs through the real kernel path until none are due
    pub async fn drain_queue(&self, queue: &str) -> Drained;
    pub async fn drain_all_jobs(&self) -> Drained;       // also runs `scheduled` jobs early
    pub async fn assert_enqueued<W: Worker>(&self, pred: impl Fn(&W) -> bool);
    pub async fn refute_enqueued<W: Worker>(&self);
    pub async fn enqueued<W: Worker>(&self) -> Vec<(job::Model, W)>;
    pub async fn perform_job<W: Worker>(&self, w: W) -> anyhow::Result<Outcome>;
    pub fn sent_emails(&self) -> Vec<Email>;            // Mailer::Memory's outbox
    pub fn query_count(&self) -> usize;                  // SQL statements so far
    pub async fn assert_query_count_stable(&self, path: &str, seed: impl AsyncFn(usize));   // N+1 guard
}

pub struct TestResponse { pub status: StatusCode, pub headers: HeaderMap, pub body: String /* lossy */, pub bytes: Bytes }
impl TestResponse {
    pub fn assert_ok(&self) -> &Self;
    pub fn assert_status(&self, s: StatusCode) -> &Self;
    pub fn assert_redirect_to(&self, path: &str) -> &Self;  // 303 + Location
    pub fn assert_contains(&self, text: &str) -> &Self;     // raw markup
    pub fn assert_text(&self, text: &str) -> &Self;         // visible text: escapes `text` like Askama 0.16 (&#60; &#62; &#38; &#34; &#39;) first
    pub fn assert_not_contains(&self, text: &str) -> &Self;
    pub fn location(&self) -> Option<&str>;
    pub fn set_cookies(&self) -> Vec<String>;               // every Set-Cookie line, to assert flags
    pub fn json<T: DeserializeOwned>(&self) -> T;
    pub fn inertia_page(&self) -> serde_json::Value;         // {component, props, url, version}
}

• Requests go through router.clone().oneshot(req) (tower::ServiceExt). Read bodies with http_body_util::BodyExt::collect.
• Form posts set Sec-Fetch-Site: same-origin by default, so CSRF passes the way it does in a real browser. The security suite proves a cross-site POST to every unsafe route is rejected. app.without_fetch_metadata().with_origin("https://evil.example") exercises the Origin fallback that older browsers hit: a foreign origin → 403, APP_URL → passes. Don't hand-build requests with empty headers for that.
• Assert on visible text with assert_text. Askama escapes ' as ', so assert_contains("we've sent…") never matches. Use assert_contains for markup (href="/dev/mailbox", value="…").
• SSE: support::next_event(&mut body) returns the next event the way a browser dispatches it: it skips frames without a non-empty data: line (browsers drop them) and returns None after one second of silence (realtime).
• Binary responses (/blobs/…): compare res.bytes. body is lossy UTF-8.

Database per test #

• Default (SQLite): sqlite::memory: with max_connections(1), min_connections(1), built through kernel::db::connect, then Migrator::up(&db, None). It's isolated by construction and takes milliseconds.
• localhost:5432 taken (another app's Postgres, a native install): never stop it. Start the compose db service on another port, DB_PORT=54329 docker compose up -d db, and run TEST_DATABASE_URL=postgres://postgres:postgres@localhost:54329/postgres cargo nextest run. Any Postgres you may create databases in works; check the port first with lsof -nP -iTCP:5432 -sTCP:LISTEN.
• TEST_DATABASE_URL=postgres://…/postgres set: a reset slot database per test, below.
• Measured (Apple M2, other agents building, the same load for old and new, two rounds): the bare app's Postgres run 8.0 / 11.9 s → 1.7 / 1.8 s (121 tests); an app with every part 149 / 46 s → 21 / 14 s (652 tests), against one CREATE DATABASE and a full migration per test before. SQLite is unchanged (1.3 s bare, 8–10 s with every part). Under heavy load CREATE DATABASE alone took 1–4 s (it fsyncs a copy of the template), which is why nothing per test does DDL.

Postgres test databases #

What test_database() in support.rs does, and why (all SQL in kernel/db.rs, since raw SQL outside the kernel is a DoD failure; every database name is checked to be [a-z0-9_] before it reaches DDL):

1. Template, once per migration set. <tag> is a hash of the migration names and every file under migration/src (editing an uncommitted migration builds a new one). If test_<crate>_<tag>_tpl doesn't exist, the first test takes an advisory lock (kernel::db::advisory_lock; concurrent tests wait for it), creates test_<crate>_<tag>_b<pid>, migrates it, runs kernel::db::install_test_reset (snapshots the rows the migrations seeded and every sequence, and installs offrails_test.reset()), then renames it to the final name, so a killed build never leaves a half-migrated template. It then drops the idle databases of older tags (drop_idle_stale_test_databases).
2. Slot per concurrent test. The test locks the first free slot n with pg_try_advisory_lock (try_advisory_lock) and uses test_<crate>_<tag>_<n>. The first time, the slot is cloned from the template (create_database_from: CREATE DATABASE … TEMPLATE …, a file copy, no migrations) and set to synchronous_commit = off (skip_commit_flush: commits don't wait for the disk; visibility is unchanged). After that the slot is reused across tests and runs: as many slots exist as tests ever ran at once.
3. Reset before every test. SELECT offrails_test.reset() (reset_test_database) truncates every table with rows except seaql_migrations, puts the migrations' seed rows back (in an order the foreign keys accept) and restores every sequence. The result is the same data, ids and schema a fresh migrated database has. The reset first compares a fingerprint of the schema (tables, columns, defaults, indexes, constraints, types, functions, applied migrations) with the template's; when a test rolled a migration back or created a table, it refuses, and the harness drops that slot and clones it again.
4. Release. A helper thread with its own runtime holds the lock (the test's runtime is gone when Drop runs); DbGuard::drop tells it to unlock and joins it. A killed run's locks die with its connections; its slots are simply reused.

The offrails_test schema (the snapshot and the reset) lives outside current_schema(), so the app's queries and the schema pages never see it. app task test_db_cleanup <admin url> drops every idle test_<crate>_* database (template and slots included: the next run rebuilds them in about a second) and any test_<crate>_<hex> database an older harness leaked; it is safe while another suite runs.

Evaluated and not used: a transaction per test, rolled back (the app opens its own transactions with kernel::db::begin, the job worker and realtime use other connections, so a test would see neither its own commits nor the isolation it gets in production); CREATE DATABASE … TEMPLATE per test (it removes the migrations but keeps the fsync-bound copy, 1–4 s each under load). SQLite stays a fresh in-memory database migrated per test: the migrations take ~5 ms of the ~40 ms median test, below the process start, so a pre-migrated copy (serialize/deserialize) would save under half a second per suite.

Local Postgres settings (optional, a test-only server, never production): with the slots, commits already skip the flush. A disposable test server can also run with fsync=off and full_page_writes=off (docker compose db service: command: postgres -c fsync=off -c full_page_writes=off); it mostly helps the first run, when the template and the slots are created.

• TestApp::new reads TEST_DATABASE_URL, always. A support.rs that hard-codes sqlite::memory: makes "tests pass on Postgres" impossible to run, and the defects only Postgres has ship: a peer's Kanban had INT4 ids, so every Postgres write that returned an id was a 500, behind a green SQLite suite (migrations). offrails-check fails a support.rs that never mentions it ([postgres]).
• Never share a database between tests. Never depend on test order.

Query counting (N+1 guard) #

SeaORM 2's DatabaseConnection::set_metric_callback (verified in 2.0.4) fires once per executed statement. It takes &mut self, so TestApp sets it right after kernel::db::connect and before the connection is cloned into AppState:

let queries = Arc::new(AtomicUsize::new(0));
let counter = queries.clone();
db.set_metric_callback(move |_info| { counter.fetch_add(1, Ordering::Relaxed); });

assert_query_count_stable(path, seed) calls seed(1), counts the queries for GET path, calls seed(19), counts again, and asserts the two counts are equal. seed is an async closure (stable since Rust 1.85):

app.assert_query_count_stable("/books", async |n| {
    for _ in 0..n { create_book(app.db(), &user).await; }
}).await;

Every index page and list endpoint gets one (test-standards §6). Create users in seed with the shared-digest factory (below), or the test spends its time in argon2.

Coverage and mutation testing #

• Coverage (cargo-llvm-cov 0.9):
  • Locally: cargo llvm-cov nextest --summary-only --ignore-filename-regex 'src/main\.rs$' --fail-under-lines 80.
  • src/main.rs (CLI glue: arg parsing, serve, signal handling) is excluded by convention; everything else counts.
  • It needs llvm-tools-preview matching your rustc (rustup component add llvm-tools-preview). A rustc that isn't rustup's (Homebrew's rust, even when it is first on PATH beside an installed rustup) has no component: brew install llvm, then cargo offrails verify --full finds it itself (it sets LLVM_COV and LLVM_PROFDATA to $(brew --prefix)/opt/llvm/bin/llvm-cov and llvm-profdata when the active rustc's sysroot, rustc --print sysroot, has no lib/rustlib/<host>/bin/llvm-cov and they aren't set). By hand: LLVM_COV=$(brew --prefix)/opt/llvm/bin/llvm-cov LLVM_PROFDATA=$(brew --prefix)/opt/llvm/bin/llvm-profdata cargo llvm-cov …. Homebrew's LLVM must be at least rustc's (rustc -vV | grep LLVM): an older one can't read the profiles ("unsupported instrumentation profile format version").
  • A coverage toolchain other than the default (one that has llvm-tools, often without rustfmt or clippy): OFFRAILS_COVERAGE_TOOLCHAIN=nightly cargo offrails verify --full, or RUSTUP_TOOLCHAIN=nightly. verify gives it to the coverage step only; fmt, clippy and the other steps run on the default toolchain.
  • In CI, cargo llvm-cov nextest --lcov --output-path lcov.info --ignore-filename-regex 'src/main\.rs$' --fail-under-lines 80 replaces the plain cargo nextest run step in the SQLite job. Install it with taiki-e/install-action@cargo-llvm-cov, plus @nextest; dtolnay/rust-toolchain needs components: llvm-tools-preview. The per-path 90% gate for src/models and src/policies has no --fail-under flag. Export JSON and check it with jq; it exits 1 below 90%:

    cargo llvm-cov nextest --json --summary-only --ignore-filename-regex 'src/main\.rs$' --output-path cov.json
    jq -e '[.data[0].files[] | select(.filename | test("/src/(models|policies)/")) | .summary.lines]
           | (map(.covered) | add) / (map(.count) | add) * 100 | . >= 90' cov.json

    In CI, run it after the coverage step (cov.json is git-ignored).
  • Measured on the new skeleton with the smoke, security, kernel and errors tests: 76%. The remaining gap is the jobs kernel (kernel/jobs/mod.rs, ~60%), so the 80% gate is only met once kernel_jobs_test.rs (required by step 7) exists. Generate it with the kernel; don't lower the gate.
• Mutation testing (weekly, and /offrails:check --deep; cargo-mutants 27): cargo mutants --test-tool nextest --file 'src/policies/**' --file 'src/models/**'. --list shows the mutants without running them. Run it as a separate scheduled workflow (on: schedule: [{cron: "0 4 * * 1"}]), not on every push, because it rebuilds the crate once per mutant.
• API contract snapshots: insta (dev-dependency, features json, redactions). Inline snapshots (assert_json_snapshot!(v, @"…")) keep the expectation in the test file. Review changes with cargo insta review (cargo install --locked cargo-insta).

Factories (tests/app/factories.rs) #

Generated models get their factory in tests/app/factories/<singular>.rs, wired into factories.rs by mod post; pub use post::*; (under a // posts line), so callers still write factories::post(db). The example below is the hand-written shape.

static SEQ: AtomicU64 = AtomicU64::new(1);
fn seq() -> u64 { SEQ.fetch_add(1, Ordering::Relaxed) }

pub async fn create_user(db: &DatabaseConnection) -> User { create_user_with(db, UserAttrs::default()).await }
pub async fn create_user_with(db: &DatabaseConnection, attrs: UserAttrs) -> User;

pub struct UserAttrs { pub email: Option<String>, pub role: Option<Role>, pub password: Option<String> }
// usage: create_user_with(db, UserAttrs { role: Some(Role::Admin), ..Default::default() }).await

• Factories call the real model fns (Params::validate + the model's create), so validations and side effects stay honest. Exception: users. argon2 at production cost takes 50–500 ms per hash (most of the suite time in the reference app, and a 47s N+1 test). create_user therefore hashes the constant PASSWORD once per test binary with the real user::hash_password (static DIGEST: tokio::sync::OnceCell<String>), and inserts the ActiveModel with that digest. models::user's own tests cover hashing; accounts_test covers sign-up through HTTP. Don't add a weaker "test" hashing path to production code.
• Defaults are valid and unique (reader{n}@example.com). PASSWORD must pass the app's rules (12+ characters, e.g. "correct horse battery staple").
• Factories for models with uploads build blobs through the same code path the app uses, or skip the file. Upload behavior is tested over HTTP with post_multipart.
• Each entity has a factory, which also proves the entity ↔ migration mapping on both backends (see models.md).

Writing tests #

/// Spec: product/features/books.md — "A logged-in user can add a book"
#[tokio::test]
async fn logged_in_user_can_add_a_book() {
    let app = TestApp::new().await;
    let user = create_user(app.db()).await;
    app.login_as(&user).await;

    let res = app.post_form("/books", &[("title", "Dune"), ("author", "Herbert")]).await;
    res.assert_redirect_to("/books/1");
    app.get("/books/1").await.assert_ok().assert_contains("Dune");
}

• Name = the acceptance criterion in snake_case. The doc comment cites the spec file.
• For each resource, cover the happy path, a validation failure (422 + message shown), anonymous access (redirect to login) and another user's record (404/403).
• Assert on what the user sees (status, redirect, text, email, JSON shape), not on internals.
• Unit tests next to code: validate() edge cases, policies (can_edit truth tables), pure helpers.
• No sleeping. Jobs: drain_jobs. Runtime/background behavior: wait_until("what", async || { … }) (support.rs: pub async fn wait_until(what: &str, f: impl AsyncFn() -> bool); an async closure can borrow the test's locals, which || async {} can't; polls every 20ms, 10s deadline). The plugin hook flags sleep( in tests. Time-dependent logic: pass now: DateTimeUtc into the model fn instead of calling Utc::now() deep inside, and pass a fixed time in tests.

Frontend tests (inertia / spa) #

• Vitest + Testing Library for components with logic (forms, conditional rendering). Don't snapshot-test markup.
• Props come from the generated web/src/types/* types, so type-correct fixtures stay in sync with Rust.
• Setup (in vite.config.ts): test: { environment: "jsdom", setupFiles: ["src/test-setup.ts"], include: ["src/**/*.test.{ts,tsx}"] }. Import describe/it/expect/vi from vitest explicitly (no globals). Because globals are off, src/test-setup.ts must register Testing Library's cleanup itself, or renders leak between tests ("Found multiple elements"): import "@testing-library/jest-dom/vitest"; import { cleanup } from "@testing-library/react"; import { afterEach } from "vitest"; afterEach(cleanup);. The include keeps Vitest out of e2e/.
• Tests live next to the component (Layout.test.tsx), never under web/src/pages/ (Inertia's page glob would pick them up).
• Mock at the boundary: vi.mock("@inertiajs/react", …) for usePage, or api() in spa mode. Never mock fetch.
• Playwright E2E for the critical flows (test-standards §2: required in inertia/spa). Add it with pnpm add -D @playwright/test, then pnpm exec playwright install --with-deps chromium.
  • web/playwright.config.ts's webServer builds the frontend and serves the real app on a throwaway SQLite file. Delete the file, then db migrate and serve, with APP_ENV=test and its own PORT/APP_URL. Don't use db reset, which refuses outside development.
  • Set retries: 0. Specs live in web/e2e/, and each one fails on any console error.
  • The config and verified spec are in frontend-inertia.

Build speed #

What dominates the loop, measured on generated apps (Apple M2, 8 cores, with other agents building on the same machine; load averages given because they moved every number):

• A one-line change recompiles the app crate twice in parallel (the lib, and the lib with cfg(test) for its unit tests), then the tests/app binary. With every part, the lib alone is ~5 s at load ~20 (-Ztime-passes: metadata 1.1 s, codegen 1.8 s, LLVM 0.7 s, macro expansion 0.6 s, incremental cache 1.1 s); the bare app's whole test build is ~3–4 s. Clippy (check mode) runs in its own target dir beside it.
• The cold build (cargo build --timings, every part, load 20→97): aws-lc-sys's C build 92 s, the app crate 79 s (+ its test build 79 s, in parallel), libsqlite3-sys's bundled SQLite 77 s, syn 2 and 3 (both are in the tree: askama, async-trait and clap moved to 3) 55 s each, sea-orm 33 s, sqlx-postgres 33 s, tokio 32 s, rustls 31 s, sea-orm-macros 28 s. None can go: aws-lc-rs is rustls' provider for the server (post-quantum key exchange; ring, already in the tree for sqlx and lettre, has none), the bundled SQLite is the portability guarantee, and sea-orm's default features (rust_decimal, time) come back through sea-orm-migration, which depends on sea-orm with them on. cargo tree -d lists only version transitions (syn, sha2, base64, getrandom, rand, hashbrown).
• [profile.dev]: debug = "line-tables-only" (file:line in backtraces, a faster link; CARGO_PROFILE_DEV_DEBUG=true for a debugging session), and opt-level = 1 for dependencies (argon2, SQLite and the TLS code run in tests). debug = false for dependencies changed nothing measurable on macOS, where debug info never passes through the linker; it isn't set. incremental and codegen-units stay at Cargo's dev defaults (on, 256).
• Linkers: macOS's default ld is fast; x86_64 Linux links with rust-lld by default since Rust 1.90. Elsewhere on Linux, mold is an opt-in in your ~/.cargo/config.toml ([target.<triple>] rustflags = ["-C", "link-arg=-fuse-ld=mold"]), never in the app's: a missing linker breaks the build. Windows: the default MSVC linker.
• macOS: during builds and test runs XprotectService and syspolicyd took 20–45% CPU each, scanning every new binary (each build script, each test binary) before it first runs. Adding the terminal (or the agent's host app) under System Settings → Privacy & Security → Developer Tools exempts what it starts. Not measured here (a machine setting the human makes); Spotlight (mds) indexing target/ is the other frequent cost (System Settings → Spotlight → exclude the projects folder).
• A second app on the same machine compiles the whole stack again: each app has its own target/. sccache (optional, per machine: brew install sccache or cargo install --locked sccache, then [build] rustc-wrapper = "sccache" in ~/.cargo/config.toml) caches dependency compiles, Rust and C (aws-lc, SQLite), across apps. Measured with three freshly generated bare apps, one after another, load 120–155: no sccache 293 s; the first app with sccache (filling the cache) 425 s; the second app 128 s (−56%, 46% of compile requests hit; proc-macros, build-script runs and the app's own incremental crates are never cached). It never breaks a build (a cache miss compiles normally); the cost is the first app's overhead and a cache directory (10 GB cap by default). A shared CARGO_TARGET_DIR across apps would reuse dependencies, but every build of every app then waits on one lock (agents building two apps at once serialize), and cargo clean in one app wipes all of them, so it isn't recommended.

Browser smoke: boosted navigation #

Request tests can't see what a boosted link does: it swaps <body> and runs nothing from the next page's <head> (frontend-htmx). .offrails/tools/browser-smoke/smoke-boosted.mjs drives a real Chrome through every page:

• Which pages: each GET route in docs/routes.md without a parameter that answers 200 with a boosted HTML page (kernel pages, JSON and streams are skipped and listed; a route that redirects, like /live, smokes the page it lands on). Add pages with parameters with SMOKE_PATHS=/posts/1,/posts/1/edit; leave some out with SMOKE_SKIP (default /dev/lab*). With password sign-in it runs twice, signed out, then signed in as the development demo account (cargo run -- task auth_demo; SMOKE_EMAIL/SMOKE_PASSWORD for another). It signs in through the form with a password field on SMOKE_LOGIN (default /session/new), whatever that form's action; it tries when docs/routes.md has a POST /session/password, /session, /sessions, /login, /sign_in or /users/sign_in, or SMOKE_SIGNIN_PATH names the app's own. prove signs in the same way.
• What it checks: on a full load it clicks each live element (button[type=button], [data-js], [data-toggle], [role=button]; not links, submits or hx-* requests) and records which ones change the DOM or an attribute, or print, copy or open a dialog. Then it loads /, follows a boosted link to the page and clicks them again. It fails when an element reacts only on the full load, when the page loads a script or stylesheet of its own in <head>, and on any console error, uncaught exception or CSP violation on either path.
• What it needs: Node ≥ 18 and an installed Chrome, Chromium or Edge (CHROME_PATH names one; otherwise the usual places on macOS, Linux and Windows). No npm install, and it never downloads a browser: it speaks the DevTools Protocol over a pipe (cdp.mjs, the driver smoke-components.mjs and prove share). Without a browser it prints skip: no Chrome and exits 77, which cargo offrails verify --full (with BASE set) and the framework's CI report as skipped, never passed.
• Measured on an app with every part (M2, load ~15): 18 page visits (signed out and in) in 28 s, with the sign-in, sign-up, confirm, password and two-factor pages' Show and Copy buttons each clicked on both paths. A bare app (one page, /): 4.7 s, most of it starting Chrome.
• Proof it catches the bug: node .offrails/tools/browser-smoke/selftest-boosted.mjs serves a tiny htmx site whose sign-in page binds its Show button three ways: delegated from the layout (passes), from a head-block script on DOMContentLoaded (fails, naming the script and the dead button), from the layout on DOMContentLoaded (fails on the dead button), plus a console error (fails). The framework's CI runs it. By hand on a real app: move <script src="{{ ctx.auth_js }}"> from the layout into templates/sessions/new.html's {% block head %}, rebuild, and the smoke fails with /session/new: loads /static/auth.js in its <head>… and button[data-reveal="signin_password"] "Show password" reacts on a full load and is DEAD after a boosted link; put it back and it passes.

Proof: cargo offrails prove #

Tests prove behavior; prove shows the human (and the next agent) that the pages work, and ties each acceptance criterion to its evidence. With the app running (bin/dev):

BASE=<the URL bin/dev printed> cargo offrails prove       # --strict: visual changes fail too; --commit-screenshots: keep the PNGs in git

• Which pages: every GET route in docs/routes.md without a parameter that answers HTML (plus SMOKE_PATHS, minus SMOKE_SKIP, as the smokes), each as the visitor who can see it: signed out, or signed in as the demo account (SMOKE_EMAIL/SMOKE_PASSWORD; the sign-in form found as the boosted smoke finds it) when it sends you to sign in. Generated show and edit pages have a parameter: add one of each to SMOKE_PATHS (/posts/1,/posts/1/edit) once a row exists. Fragments, JSON and files are listed under "Not proven here".
• What it does per page: full-page screenshots at desktop 1280 and mobile 390, light and dark (docs/proof/screens/); it fails on a console error, an uncaught exception or a CSP violation in any of the four loads, on axe-core violations (WCAG 2.2 AA, light and dark; the a11y smoke's pinned axe, from the smokes' cache), on a same-site link that answers 4xx/5xx or an #anchor without a target, and on horizontal scroll at 390 px (naming the elements that stick out; on a kernel page, /dev or /admin/… from the framework and its parts, that is a warning in PROOF.md, failing only with --strict). Each screenshot is compared with the previous run's (a pixel diff in Node, no package): changes over PROVE_DIFF_THRESHOLD (0.5% of pixels) are reported with a diff image in docs/proof/diff/, and fail only with --strict.
• What it writes: docs/proof/PROOF.md: the result, the checks, a table of pages with links to the four screenshots and what changed, and every acceptance criterion in product/features/*.md → the tests that cite it (/// AC-… on a test function: #[test], #[tokio::test] or another …::test attribute, under tests/ or in a #[cfg(test)] module; a handler or helper whose comment names an AC id is not a test) with their result in the last cargo offrails verify (read from target/verify/) → the pages those tests request ("/…" literals and paths::…() helpers in the test, backticked paths in the criterion) with their screenshots. proof.json holds the same for programs. PNGs are git-ignored (docs/proof/.gitignore) unless --commit-screenshots.
• What it needs: what smoke-boosted.mjs needs, through the same driver (cdp.mjs: Node ≥ 18 and an installed Chrome, Chromium or Edge, no npm install; skip: no Chrome, exit 77, without one). cargo offrails verify --full runs it when BASE is set; the framework's CI proves it on the bare app and the app with every part (tools/ci/test-template.sh prove).
• The component smoke: BASE=… node .offrails/tools/browser-smoke/smoke-components.mjs drives /dev/components on a full load and after a boosted link: the modal (focus kept inside, Esc, focus returned), the htmx-loaded modal, the confirm dialog (Cancel and Confirm) and the demo form's 422. verify --full runs it too.
• Measured (M2, other agents building, load 15–70): a bare app (9 pages, 36 screenshots, axe on 18 loads) 13–24 s; an app with every part (63 pages, 6 of them signed in, 252 screenshots) 165–216 s. Pages with live data (charts, timestamps) show up as visual changes on every run: expected, and why changes fail only with --strict.

Your final message links docs/proof/PROOF.md (workflow).

CI #

.github/workflows/ci.yml. Action majors verified 2026-10-12. When generating, check each action's latest release and use its current major tag.

name: CI
on:
  push: { branches: [main] }
  pull_request:

concurrency: { group: "ci-${{ github.ref }}", cancel-in-progress: true }

env:
  CARGO_TERM_COLOR: always
  RUSTFLAGS: -D warnings

jobs:
  # One build, everything at once (`cargo offrails verify`, workflow.md#definition-of-done): fmt,
  # the CSS, clippy (its own target dir, beside the test build), then the SQLite and Postgres
  # suites side by side. Two jobs used to compile the tests twice.
  test:
    runs-on: ubuntu-latest
    services:
      postgres:
        image: postgres:17                      # match the production major (Fly MPG: 16/17)
        env: { POSTGRES_PASSWORD: postgres }
        ports: ["5432:5432"]
        # A throwaway database on tmpfs: nothing waits for a disk (testing.md#postgres-test-databases).
        options: >-
          --tmpfs /var/lib/postgresql/data
          --health-cmd pg_isready --health-interval 2s --health-timeout 5s --health-retries 30
    env:
      TEST_DATABASE_URL: postgres://postgres:postgres@localhost:5432/postgres
    steps:
      - uses: actions/checkout@v7
        with: { fetch-depth: 0 }                  # the migration guard diffs against main
      - uses: dtolnay/rust-toolchain@stable
        with: { components: "rustfmt, clippy" }
      - uses: Swatinem/rust-cache@v2
        with:                                   # the build and clippy's own target dir (verify)
          workspaces: |
            . -> target
            . -> target/clippy
      - uses: taiki-e/install-action@nextest
      - name: Committed migrations are append-only (guides/migrations.md rule 1)
        if: github.event_name == 'pull_request'
        run: test -z "$(git diff --name-only --diff-filter=M origin/${{ github.base_ref }}... -- 'migration/src/m*.rs')"
      - name: Mechanical gates (header, every AC cited, Security-table tests, tenancy suites, scoped lookups, .env.example)
        run: sh .offrails/tools/offrails-check
      - name: Cargo.lock is current (verify's cargo commands don't take --locked)
        run: cargo metadata --locked --format-version 1 > /dev/null
      - run: bin/setup-tailwind                 # verify builds static/app.css (git-ignored; the theme guard reads it)
      - name: fmt, CSS, clippy, one test build, then SQLite and Postgres at once
        run: cargo offrails verify --fast
      # ---- inertia / react / ISLANDS: yes only (the suites above exported the ts-rs types) ----
      - run: git diff --exit-code web/src/types   # generated types must be committed and current
      - uses: pnpm/action-setup@v6
        with: { package_json_file: web/package.json }        # reads "packageManager"
      - uses: actions/setup-node@v7
        with: { node-version: 24, cache: pnpm, cache-dependency-path: web/pnpm-lock.yaml }
      - run: pnpm --dir web install --frozen-lockfile
      - run: pnpm --dir web exec biome ci .
      - run: pnpm --dir web typecheck                        # React: tsc -b; Svelte: svelte-check (never tsc --noEmit)
      - run: pnpm --dir web test --run
      - run: pnpm --dir web build
      - run: pnpm --dir web exec playwright install --with-deps chromium   # E2E (inertia/spa)
      - run: pnpm --dir web e2e                              # webServer: builds web/, serves the app on :3200

  security:                     # security.md#automated-enforcement
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
        with: { fetch-depth: 0 }          # gitleaks scans the whole history
      - uses: taiki-e/install-action@cargo-deny
      - name: cargo deny (advisories, licenses, bans, sources)
        run: cargo deny --locked check
      - name: gitleaks (secrets in any commit)
        run: docker run --rm -v "$PWD:/repo" ghcr.io/gitleaks/gitleaks:v8.30.1 git /repo --config /repo/.gitleaks.toml --redact --no-banner

  docker:                       # the image builds AND passes the ZAP baseline in production mode
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - uses: docker/setup-buildx-action@v4
      - uses: docker/build-push-action@v7
        with: { context: ., push: false, load: true, tags: "app:ci", cache-from: type=gha, cache-to: "type=gha,mode=max" }
      - name: Boot the production image
        run: |
          docker run -d --name app --network host -e APP_ENV=production -e PORT=3000 \
            -e APP_URL=http://localhost:3000 -e "DATABASE_URL=sqlite:///app/storage/ci.sqlite3?mode=rwc" app:ci
          for _ in $(seq 1 60); do curl -fsS localhost:3000/up && break; sleep 1; done
          curl -fsS localhost:3000/up
      - name: OWASP ZAP baseline (fails on any WARN not triaged in .zap/rules.tsv)
        run: |
          chmod 777 .zap
          docker run --rm --network host -v "$PWD/.zap:/zap/wrk:rw" ghcr.io/zaproxy/zaproxy:stable \
            zap-baseline.py -t http://localhost:3000 -c rules.tsv -r zap-report.html -m 2
          # Locally on Docker Desktop (macOS/Windows), --network host doesn't reach the host. Drop it and target
          # http://host.docker.internal:<port> instead.
      - if: failure()
        uses: actions/upload-artifact@v7
        with: { name: zap-report, path: .zap/zap-report.html, if-no-files-found: ignore }
      - if: always()
        run: docker logs app | tail -50

The security and Docker jobs are actionlint-clean. TS modes add pnpm --dir web audit --prod to the frontend job. Run ZAP locally against app serve with APP_ENV=production and APP_URL=http://host.docker.internal:$PORT, adding --add-host=host.docker.internal:host-gateway to the ZAP container. Verified: 0 FAIL, 0 WARN, 61 PASS on the reference app.

• In htmx mode without islands, delete the frontend steps. The CSS is built because the theme guard (theme_test) reads static/app.css.
• Why one test job. The SQLite and Postgres suites used to be two jobs, each compiling the tests. verify compiles once and runs both at once against the job's Postgres service (on tmpfs), while clippy checks beside the build in target/clippy (cached as a second target dir). Coverage, deny and gitleaks stay out of the critical path: deny and gitleaks in security, coverage locally in verify --full. Sharding the suites across runners (nextest --partition) doesn't pay at this size: each shard would compile the tests again, and the suites run in seconds once built.
• The Docker job builds the dependencies in a layer of their own (Dockerfile), which the gha layer cache keeps; before 0.7.2 a cache mount (not in that cache) made every run compile every dependency in release mode.
• --locked fails if Cargo.lock is stale, and --frozen-lockfile does the same for pnpm-lock.yaml. Commit both.

Gotchas agents get wrong #

• sqlite::memory: with a pool > 1: each connection sees an empty database, giving "no such table" flakes.
• Creating a new test binary per file in tests/.
• Forgetting Sec-Fetch-Site on POSTs in the helper, then "fixing" CSRF failures by disabling the check in tests.
• Asserting exact HTML strings. Assert on meaningful text or the inertia props instead.
• assert_text with an apostrophe (or &, ", <) that the template typed literally. assert_text escapes its argument the way Askama escapes {{ … }} output (we've → we've), so it finds values the template interpolated; text written straight into a template (<p>We've sent it</p>) is in the body as typed: use assert_contains("We've sent it") for that, or move the text into a {{ }} value.
• Skipping the Postgres run because "SQLite passed". CI requires both.