# Testing

> **TL;DR**
> - **How** to test (test-standards.md says **what**): one integration-test binary in `tests/app/`, `TestApp` over the real router, factories, both SQLite and Postgres.
> - Postgres tests never run `CREATE DATABASE` or migrations per test: they lock a reusable slot database cloned once from a migrated template, reset to the just-migrated state (§[Postgres test databases](#postgres-test-databases)); the SQL lives in `kernel::db`.
> - While iterating, `cargo offrails verify` (fmt, clippy, one build, both suites at once); `--full` once at the end ([workflow](../workflow.md#definition-of-done)).
> - Helpers set Fetch Metadata like a browser; assert on what the user sees (`assert_text` decodes entities), never exact HTML; no sleeps (`drain_jobs`, `wait_until`).
> - CI runs SQLite, Postgres, security (cargo-deny, gitleaks) and the Docker image with a ZAP baseline.
>
> Read the rest when writing the harness, factories, frontend tests, or CI.

This guide is **how** to write tests (TestApp, factories, layout). **What** must be tested, and the coverage and quality gates, are in [`test-standards.md`](test-standards.md). Read both.

Tests are how the agent proves the spec. Every acceptance criterion in `product/features/*.md` becomes one integration test, named after the criterion. The same suite runs on SQLite (default, fast) and Postgres (CI, and locally when available). **No mocked database, ever.**

| Rails | Off Rails |
|-------|-----------|
| `rails test` | `cargo nextest run` |
| Integration / request tests | `tests/app/<resource>/*_test.rs` (a generated resource) or `tests/app/<name>_test.rs`, using `TestApp` |
| Model tests | Inline `#[cfg(test)]` for `validate()` and pure fns; DB behavior via integration tests |
| Fixtures / FactoryBot | `tests/app/factories.rs` |
| `ActionMailer::Base.deliveries` | `app.sent_emails()` |
| `perform_enqueued_jobs` / `Oban.Testing` | `app.drain_jobs().await`, `app.assert_enqueued::<W>(…)`, `app.perform_job(w)` ([jobs](jobs.md#testing-obantesting-equivalents)) |
| `sign_in_as(user)` | `app.login_as(&user).await` |
| System tests (Capybara) | Optional Playwright, against `app serve` with a seeded SQLite DB |
| Jest for JS | Vitest + Testing Library (inertia/spa) |

## Layout: one test binary

```
tests/app/main.rs          mod factories; mod smoke_test; mod security_test; mod posts; …   (generated lines marked `// posts`)
                           #[allow(dead_code)] mod support;   // shared helpers; clippy -D warnings flags unused ones otherwise
tests/app/support.rs       TestApp, TestResponse
tests/app/factories.rs     user, … + `mod post; pub use post::*;` per generated model (tests/app/factories/post.rs)
tests/app/smoke_test.rs    generated by `new`
tests/app/security_test.rs access matrix over ROUTES (test-standards.md §3)
tests/app/kernel_test.rs   cache, rate limit, flash (generated by `new`)
tests/app/errors_test.rs   AppError × response flavor, ClientIp, tasks (generated by `new`)
tests/app/posts/           a generated resource: mod.rs (the form submit), read_test.rs, write_test.rs, access_test.rs
tests/app/models/post_test.rs   a generated model's queries, both databases
```

Every file in `tests/` root is a separate binary that links the whole app, so one binary with modules keeps link times sane. **Never add files directly under `tests/`.**

## `TestApp` spec (`tests/app/support.rs`)

```rust
pub struct TestApp {
    pub state: AppState,           // AppState::new(config, db): Config::test(), Mailer::Memory, StorageConfig::Memory
    router: Router,                // routes::router(state.clone())?
    cookies: Mutex<Vec<(String, String)>>, // persists Set-Cookie between requests, like a browser (removal cookies delete)
    fetch_metadata: AtomicBool,    // form posts send Sec-Fetch-Site: same-origin unless turned off
    origin: Mutex<Option<String>>, // Origin header for form posts, if set
    queries: Arc<AtomicUsize>,     // SQL statement counter (below)
    db_guard: DbGuard,             // holds the test's Postgres slot (its advisory lock); a no-op on SQLite
}

impl TestApp {
    pub async fn new() -> Self;                          // fresh DB, migrated, empty
    pub async fn with_config(config: Config) -> Self;    // e.g. env = Production to prove /dev isn't mounted
    pub async fn request(&self, req: Request<Body>) -> TestResponse;   // any method/headers; cookies applied and stored
    pub fn db(&self) -> &DatabaseConnection;
    pub async fn get(&self, path: &str) -> TestResponse;
    pub async fn get_with_headers(&self, path: &str, headers: &[(&str, &str)]) -> TestResponse;   // e.g. ("host", "evil.example")
    pub async fn post_form(&self, path: &str, form: &[(&str, &str)]) -> TestResponse;
    pub async fn post_form_with_headers(&self, path: &str, form: &[(&str, &str)], headers: &[(&str, &str)]) -> TestResponse;
    pub async fn post_json(&self, path: &str, body: &impl Serialize) -> TestResponse;
    pub async fn post_multipart(&self, path: &str, fields: &[(&str, &str)],
                                files: &[(&str, &str, &str, &[u8])]) -> TestResponse;   // (field, filename, content type, bytes)
    pub async fn get_htmx(&self, path: &str) -> TestResponse;   // HX-Request: true
    pub async fn open_stream(&self, path: &str) -> (StatusCode, Body);   // SSE: unread body for support::next_event
    pub async fn request_head(&self, path: &str, headers: &[(&str, &str)]) -> (StatusCode, HeaderMap);   // GET, status + headers, body never read: SSE and other endless responses
    pub fn without_fetch_metadata(&self) -> &Self;       // later posts omit Sec-Fetch-Site (CSRF falls back to Origin)
    pub fn with_origin(&self, origin: &str) -> &Self;    // later posts send this Origin
    pub fn cookie(&self, name: &str) -> Option<String>;
    pub fn set_cookie(&self, name: &str, value: &str);   // e.g. replay an old session token
    pub fn logout_locally(&self);                        // drop every cookie (a fresh browser)
    pub async fn inertia_get(&self, path: &str) -> TestResponse;   // inertia: X-Inertia + version headers
    pub async fn login_as(&self, user: &User);           // session::start + stores the cookie; call again to switch users
    pub async fn drain_jobs(&self) -> Drained;           // run all due jobs through the real kernel path until none are due
    pub async fn drain_queue(&self, queue: &str) -> Drained;
    pub async fn drain_all_jobs(&self) -> Drained;       // also runs `scheduled` jobs early
    pub async fn assert_enqueued<W: Worker>(&self, pred: impl Fn(&W) -> bool);
    pub async fn refute_enqueued<W: Worker>(&self);
    pub async fn enqueued<W: Worker>(&self) -> Vec<(job::Model, W)>;
    pub async fn perform_job<W: Worker>(&self, w: W) -> anyhow::Result<Outcome>;
    pub fn sent_emails(&self) -> Vec<Email>;            // Mailer::Memory's outbox
    pub fn query_count(&self) -> usize;                  // SQL statements so far
    pub async fn assert_query_count_stable(&self, path: &str, seed: impl AsyncFn(usize));   // N+1 guard
}

pub struct TestResponse { pub status: StatusCode, pub headers: HeaderMap, pub body: String /* lossy */, pub bytes: Bytes }
impl TestResponse {
    pub fn assert_ok(&self) -> &Self;
    pub fn assert_status(&self, s: StatusCode) -> &Self;
    pub fn assert_redirect_to(&self, path: &str) -> &Self;  // 303 + Location
    pub fn assert_contains(&self, text: &str) -> &Self;     // raw markup
    pub fn assert_text(&self, text: &str) -> &Self;         // visible text: escapes `text` like Askama 0.16 (&#60; &#62; &#38; &#34; &#39;) first
    pub fn assert_not_contains(&self, text: &str) -> &Self;
    pub fn location(&self) -> Option<&str>;
    pub fn set_cookies(&self) -> Vec<String>;               // every Set-Cookie line, to assert flags
    pub fn json<T: DeserializeOwned>(&self) -> T;
    pub fn inertia_page(&self) -> serde_json::Value;         // {component, props, url, version}
}
```

- Requests go through `router.clone().oneshot(req)` (`tower::ServiceExt`). Read bodies with `http_body_util::BodyExt::collect`.
- **Form posts set `Sec-Fetch-Site: same-origin` by default**, so CSRF passes the way it does in a real browser. The security suite proves a cross-site POST to every unsafe route is rejected. `app.without_fetch_metadata().with_origin("https://evil.example")` exercises the `Origin` fallback that older browsers hit: a foreign origin → 403, `APP_URL` → passes. Don't hand-build requests with empty headers for that.
- **Assert on visible text with `assert_text`.** Askama escapes `'` as `&#39;`, so `assert_contains("we've sent…")` never matches. Use `assert_contains` for markup (`href="/dev/mailbox"`, `value="…"`).
- **SSE:** `support::next_event(&mut body)` returns the next event the way a browser dispatches it: it skips frames without a non-empty `data:` line (browsers drop them) and returns `None` after one second of silence ([realtime](realtime.md#testing)).
- **Binary responses** (`/blobs/…`): compare `res.bytes`. `body` is lossy UTF-8.

### Database per test

- **Default (SQLite):** `sqlite::memory:` with `max_connections(1)`, `min_connections(1)`, built through `kernel::db::connect`, then `Migrator::up(&db, None)`. It's isolated by construction and takes milliseconds.
- **localhost:5432 taken** (another app's Postgres, a native install): never stop it. Start the compose `db` service on another port, `DB_PORT=54329 docker compose up -d db`, and run `TEST_DATABASE_URL=postgres://postgres:postgres@localhost:54329/postgres cargo nextest run`. Any Postgres you may create databases in works; check the port first with `lsof -nP -iTCP:5432 -sTCP:LISTEN`.
- **`TEST_DATABASE_URL=postgres://…/postgres` set:** a reset slot database per test, below.
- Measured (Apple M2, other agents building, the same load for old and new, two rounds): the bare app's Postgres run 8.0 / 11.9 s → 1.7 / 1.8 s (121 tests); an app with every part 149 / 46 s → 21 / 14 s (652 tests), against one `CREATE DATABASE` and a full migration per test before. SQLite is unchanged (1.3 s bare, 8–10 s with every part). Under heavy load `CREATE DATABASE` alone took 1–4 s (it fsyncs a copy of the template), which is why nothing per test does DDL.

### Postgres test databases

What `test_database()` in `support.rs` does, and why (all SQL in `kernel/db.rs`, since raw SQL outside the kernel is a DoD failure; every database name is checked to be `[a-z0-9_]` before it reaches DDL):

1. **Template, once per migration set.** `<tag>` is a hash of the migration names and every file under `migration/src` (editing an uncommitted migration builds a new one). If `test_<crate>_<tag>_tpl` doesn't exist, the first test takes an advisory lock (`kernel::db::advisory_lock`; concurrent tests wait for it), creates `test_<crate>_<tag>_b<pid>`, migrates it, runs `kernel::db::install_test_reset` (snapshots the rows the migrations seeded and every sequence, and installs `offrails_test.reset()`), then **renames** it to the final name, so a killed build never leaves a half-migrated template. It then drops the idle databases of older tags (`drop_idle_stale_test_databases`).
2. **Slot per concurrent test.** The test locks the first free slot `n` with `pg_try_advisory_lock` (`try_advisory_lock`) and uses `test_<crate>_<tag>_<n>`. The first time, the slot is cloned from the template (`create_database_from`: `CREATE DATABASE … TEMPLATE …`, a file copy, no migrations) and set to `synchronous_commit = off` (`skip_commit_flush`: commits don't wait for the disk; visibility is unchanged). After that the slot is reused across tests and runs: as many slots exist as tests ever ran at once.
3. **Reset before every test.** `SELECT offrails_test.reset()` (`reset_test_database`) truncates every table with rows except `seaql_migrations`, puts the migrations' seed rows back (in an order the foreign keys accept) and restores every sequence. The result is the same data, ids and schema a fresh migrated database has. The reset first compares a fingerprint of the schema (tables, columns, defaults, indexes, constraints, types, functions, applied migrations) with the template's; when a test rolled a migration back or created a table, it refuses, and the harness drops that slot and clones it again.
4. **Release.** A helper thread with its own runtime holds the lock (the test's runtime is gone when `Drop` runs); `DbGuard::drop` tells it to unlock and joins it. A killed run's locks die with its connections; its slots are simply reused.

The `offrails_test` schema (the snapshot and the reset) lives outside `current_schema()`, so the app's queries and the schema pages never see it. `app task test_db_cleanup <admin url>` drops every idle `test_<crate>_*` database (template and slots included: the next run rebuilds them in about a second) and any `test_<crate>_<hex>` database an older harness leaked; it is safe while another suite runs.

Evaluated and not used: **a transaction per test, rolled back** (the app opens its own transactions with `kernel::db::begin`, the job worker and realtime use other connections, so a test would see neither its own commits nor the isolation it gets in production); **`CREATE DATABASE … TEMPLATE` per test** (it removes the migrations but keeps the fsync-bound copy, 1–4 s each under load). **SQLite** stays a fresh in-memory database migrated per test: the migrations take ~5 ms of the ~40 ms median test, below the process start, so a pre-migrated copy (serialize/deserialize) would save under half a second per suite.

**Local Postgres settings (optional, a test-only server, never production):** with the slots, commits already skip the flush. A disposable test server can also run with `fsync=off` and `full_page_writes=off` (`docker compose` `db` service: `command: postgres -c fsync=off -c full_page_writes=off`); it mostly helps the first run, when the template and the slots are created.

- **`TestApp::new` reads `TEST_DATABASE_URL`, always.** A `support.rs` that hard-codes `sqlite::memory:` makes "tests pass on Postgres" impossible to run, and the defects only Postgres has ship: a peer's Kanban had `INT4` ids, so every Postgres write that returned an id was a 500, behind a green SQLite suite ([migrations](migrations.md#ids-are-64-bit)). `offrails-check` fails a `support.rs` that never mentions it (`[postgres]`).
- Never share a database between tests. Never depend on test order.

### Query counting (N+1 guard)

SeaORM 2's `DatabaseConnection::set_metric_callback` (verified in 2.0.4) fires once per executed statement. It takes `&mut self`, so `TestApp` sets it right after `kernel::db::connect` and **before** the connection is cloned into `AppState`:

```rust
let queries = Arc::new(AtomicUsize::new(0));
let counter = queries.clone();
db.set_metric_callback(move |_info| { counter.fetch_add(1, Ordering::Relaxed); });
```

`assert_query_count_stable(path, seed)` calls `seed(1)`, counts the queries for `GET path`, calls `seed(19)`, counts again, and asserts the two counts are equal. `seed` is an async closure (stable since Rust 1.85):

```rust
app.assert_query_count_stable("/books", async |n| {
    for _ in 0..n { create_book(app.db(), &user).await; }
}).await;
```

Every index page and list endpoint gets one ([test-standards §6](test-standards.md#6-performance-guards)). Create users in `seed` with the shared-digest factory (below), or the test spends its time in argon2.

### Coverage and mutation testing

- **Coverage** (cargo-llvm-cov 0.9):
  - Locally: `cargo llvm-cov nextest --summary-only --ignore-filename-regex 'src/main\.rs$' --fail-under-lines 80`.
  - `src/main.rs` (CLI glue: arg parsing, `serve`, signal handling) is excluded **by convention**; everything else counts.
  - It needs `llvm-tools-preview` matching your `rustc` (`rustup component add llvm-tools-preview`). A `rustc` that isn't rustup's (Homebrew's `rust`, even when it is first on PATH beside an installed rustup) has no component: `brew install llvm`, then `cargo offrails verify --full` finds it itself (it sets `LLVM_COV` and `LLVM_PROFDATA` to `$(brew --prefix)/opt/llvm/bin/llvm-cov` and `llvm-profdata` when the active `rustc`'s sysroot, `rustc --print sysroot`, has no `lib/rustlib/<host>/bin/llvm-cov` and they aren't set). By hand: `LLVM_COV=$(brew --prefix)/opt/llvm/bin/llvm-cov LLVM_PROFDATA=$(brew --prefix)/opt/llvm/bin/llvm-profdata cargo llvm-cov …`. Homebrew's LLVM must be at least `rustc`'s (`rustc -vV | grep LLVM`): an older one can't read the profiles ("unsupported instrumentation profile format version").
  - A coverage toolchain other than the default (one that has `llvm-tools`, often without rustfmt or clippy): `OFFRAILS_COVERAGE_TOOLCHAIN=nightly cargo offrails verify --full`, or `RUSTUP_TOOLCHAIN=nightly`. `verify` gives it to the coverage step only; fmt, clippy and the other steps run on the default toolchain.
  - In CI, `cargo llvm-cov nextest --lcov --output-path lcov.info --ignore-filename-regex 'src/main\.rs$' --fail-under-lines 80` replaces the plain `cargo nextest run` step in the SQLite job. Install it with `taiki-e/install-action@cargo-llvm-cov`, plus `@nextest`; `dtolnay/rust-toolchain` needs `components: llvm-tools-preview`. The per-path 90% gate for `src/models` and `src/policies` has no `--fail-under` flag. Export JSON and check it with `jq`; it exits 1 below 90%:
    ```sh
    cargo llvm-cov nextest --json --summary-only --ignore-filename-regex 'src/main\.rs$' --output-path cov.json
    jq -e '[.data[0].files[] | select(.filename | test("/src/(models|policies)/")) | .summary.lines]
           | (map(.covered) | add) / (map(.count) | add) * 100 | . >= 90' cov.json
    ```
    In CI, run it after the coverage step (`cov.json` is git-ignored).
  - Measured on the `new` skeleton with the smoke, security, kernel and errors tests: **76%**. The remaining gap is the jobs kernel (`kernel/jobs/mod.rs`, ~60%), so the 80% gate is only met once `kernel_jobs_test.rs` (required by step 7) exists. Generate it with the kernel; don't lower the gate.
- **Mutation testing** (weekly, and `/offrails:check --deep`; cargo-mutants 27): `cargo mutants --test-tool nextest --file 'src/policies/**' --file 'src/models/**'`. `--list` shows the mutants without running them. Run it as a separate scheduled workflow (`on: schedule: [{cron: "0 4 * * 1"}]`), not on every push, because it rebuilds the crate once per mutant.
- API contract snapshots: `insta` (dev-dependency, features `json`, `redactions`). Inline snapshots (`assert_json_snapshot!(v, @"…")`) keep the expectation in the test file. Review changes with `cargo insta review` (`cargo install --locked cargo-insta`).

## Factories (`tests/app/factories.rs`)

Generated models get their factory in `tests/app/factories/<singular>.rs`, wired into `factories.rs` by `mod post; pub use post::*;` (under a `// posts` line), so callers still write `factories::post(db)`. The example below is the hand-written shape.

```rust
static SEQ: AtomicU64 = AtomicU64::new(1);
fn seq() -> u64 { SEQ.fetch_add(1, Ordering::Relaxed) }

pub async fn create_user(db: &DatabaseConnection) -> User { create_user_with(db, UserAttrs::default()).await }
pub async fn create_user_with(db: &DatabaseConnection, attrs: UserAttrs) -> User;

pub struct UserAttrs { pub email: Option<String>, pub role: Option<Role>, pub password: Option<String> }
// usage: create_user_with(db, UserAttrs { role: Some(Role::Admin), ..Default::default() }).await
```

- Factories call the **real model fns** (`Params::validate` + the model's `create`), so validations and side effects stay honest. Exception: **users**. argon2 at production cost takes 50–500 ms per hash (most of the suite time in the reference app, and a 47s N+1 test). `create_user` therefore hashes the constant `PASSWORD` once per test binary with the real `user::hash_password` (`static DIGEST: tokio::sync::OnceCell<String>`), and inserts the `ActiveModel` with that digest. `models::user`'s own tests cover hashing; `accounts_test` covers sign-up through HTTP. Don't add a weaker "test" hashing path to production code.
- Defaults are valid and unique (`reader{n}@example.com`). `PASSWORD` must pass the app's rules (12+ characters, e.g. `"correct horse battery staple"`).
- Factories for models with uploads build blobs through the same code path the app uses, or skip the file. Upload behavior is tested over HTTP with `post_multipart`.
- Each entity has a factory, which also proves the entity ↔ migration mapping on both backends (see models.md).

## Writing tests

```rust
/// Spec: product/features/books.md — "A logged-in user can add a book"
#[tokio::test]
async fn logged_in_user_can_add_a_book() {
    let app = TestApp::new().await;
    let user = create_user(app.db()).await;
    app.login_as(&user).await;

    let res = app.post_form("/books", &[("title", "Dune"), ("author", "Herbert")]).await;
    res.assert_redirect_to("/books/1");
    app.get("/books/1").await.assert_ok().assert_contains("Dune");
}
```

- Name = the acceptance criterion in snake_case. The doc comment cites the spec file.
- For each resource, cover the happy path, a validation failure (422 + message shown), anonymous access (redirect to login) and another user's record (404/403).
- Assert on what the user sees (status, redirect, text, email, JSON shape), not on internals.
- Unit tests next to code: `validate()` edge cases, policies (`can_edit` truth tables), pure helpers.
- **No sleeping.** Jobs: `drain_jobs`. Runtime/background behavior: `wait_until("what", async || { … })` (support.rs: `pub async fn wait_until(what: &str, f: impl AsyncFn() -> bool)`; an async closure can borrow the test's locals, which `|| async {}` can't; polls every 20ms, 10s deadline). The plugin hook flags `sleep(` in tests. Time-dependent logic: pass `now: DateTimeUtc` into the model fn instead of calling `Utc::now()` deep inside, and pass a fixed time in tests.

## Frontend tests (inertia / spa)

- Vitest + Testing Library for components with logic (forms, conditional rendering). Don't snapshot-test markup.
- Props come from the generated `web/src/types/*` types, so type-correct fixtures stay in sync with Rust.
- Setup (in `vite.config.ts`): `test: { environment: "jsdom", setupFiles: ["src/test-setup.ts"], include: ["src/**/*.test.{ts,tsx}"] }`. Import `describe`/`it`/`expect`/`vi` from `vitest` explicitly (no globals). Because globals are off, `src/test-setup.ts` must register Testing Library's cleanup itself, or renders leak between tests ("Found multiple elements"): `import "@testing-library/jest-dom/vitest"; import { cleanup } from "@testing-library/react"; import { afterEach } from "vitest"; afterEach(cleanup);`. The `include` keeps Vitest out of `e2e/`.
- Tests live next to the component (`Layout.test.tsx`), never under `web/src/pages/` (Inertia's page glob would pick them up).
- Mock at the boundary: `vi.mock("@inertiajs/react", …)` for `usePage`, or `api()` in `spa` mode. Never mock `fetch`.
- Playwright E2E for the critical flows ([test-standards](test-standards.md) §2: required in `inertia`/`spa`). Add it with `pnpm add -D @playwright/test`, then `pnpm exec playwright install --with-deps chromium`.
  - `web/playwright.config.ts`'s `webServer` builds the frontend and serves the real app on a throwaway SQLite file. Delete the file, then `db migrate` and `serve`, with `APP_ENV=test` and its own `PORT`/`APP_URL`. Don't use `db reset`, which refuses outside development.
  - Set `retries: 0`. Specs live in `web/e2e/`, and each one fails on any console error.
  - The config and verified spec are in [frontend-inertia](frontend-inertia.md#testing-in-inertia-mode).

## Build speed

What dominates the loop, measured on generated apps (Apple M2, 8 cores, with other agents building on the same machine; load averages given because they moved every number):

- **A one-line change** recompiles the app crate twice in parallel (the lib, and the lib with `cfg(test)` for its unit tests), then the `tests/app` binary. With every part, the lib alone is ~5 s at load ~20 (`-Ztime-passes`: metadata 1.1 s, codegen 1.8 s, LLVM 0.7 s, macro expansion 0.6 s, incremental cache 1.1 s); the bare app's whole test build is ~3–4 s. Clippy (check mode) runs in its own target dir beside it.
- **The cold build** (`cargo build --timings`, every part, load 20→97): `aws-lc-sys`'s C build 92 s, the app crate 79 s (+ its test build 79 s, in parallel), `libsqlite3-sys`'s bundled SQLite 77 s, `syn` 2 and 3 (both are in the tree: askama, async-trait and clap moved to 3) 55 s each, `sea-orm` 33 s, `sqlx-postgres` 33 s, `tokio` 32 s, `rustls` 31 s, `sea-orm-macros` 28 s. None can go: `aws-lc-rs` is rustls' provider for the server (post-quantum key exchange; `ring`, already in the tree for sqlx and lettre, has none), the bundled SQLite is the portability guarantee, and `sea-orm`'s default features (`rust_decimal`, `time`) come back through `sea-orm-migration`, which depends on `sea-orm` with them on. `cargo tree -d` lists only version transitions (`syn`, `sha2`, `base64`, `getrandom`, `rand`, `hashbrown`).
- **`[profile.dev]`:** `debug = "line-tables-only"` (file:line in backtraces, a faster link; `CARGO_PROFILE_DEV_DEBUG=true` for a debugging session), and `opt-level = 1` for dependencies (argon2, SQLite and the TLS code run in tests). `debug = false` for dependencies changed nothing measurable on macOS, where debug info never passes through the linker; it isn't set. `incremental` and `codegen-units` stay at Cargo's dev defaults (on, 256).
- **Linkers:** macOS's default `ld` is fast; x86_64 Linux links with rust-lld by default since Rust 1.90. Elsewhere on Linux, `mold` is an opt-in in **your** `~/.cargo/config.toml` (`[target.<triple>] rustflags = ["-C", "link-arg=-fuse-ld=mold"]`), never in the app's: a missing linker breaks the build. Windows: the default MSVC linker.
- **macOS:** during builds and test runs `XprotectService` and `syspolicyd` took 20–45% CPU each, scanning every new binary (each build script, each test binary) before it first runs. Adding the terminal (or the agent's host app) under System Settings → Privacy & Security → Developer Tools exempts what it starts. Not measured here (a machine setting the human makes); Spotlight (`mds`) indexing `target/` is the other frequent cost (System Settings → Spotlight → exclude the projects folder).
- **A second app on the same machine** compiles the whole stack again: each app has its own `target/`. **sccache** (optional, per machine: `brew install sccache` or `cargo install --locked sccache`, then `[build] rustc-wrapper = "sccache"` in `~/.cargo/config.toml`) caches dependency compiles, Rust and C (aws-lc, SQLite), across apps. Measured with three freshly generated bare apps, one after another, load 120–155: no sccache 293 s; the first app with sccache (filling the cache) 425 s; the **second app 128 s** (−56%, 46% of compile requests hit; proc-macros, build-script runs and the app's own incremental crates are never cached). It never breaks a build (a cache miss compiles normally); the cost is the first app's overhead and a cache directory (10 GB cap by default). A shared `CARGO_TARGET_DIR` across apps would reuse dependencies, but every build of every app then waits on one lock (agents building two apps at once serialize), and `cargo clean` in one app wipes all of them, so it isn't recommended.

## Browser smoke: boosted navigation

Request tests can't see what a boosted link does: it swaps `<body>` and runs nothing from the next page's `<head>` ([frontend-htmx](frontend-htmx.md#boosted-navigation)). `.offrails/tools/browser-smoke/smoke-boosted.mjs` drives a real Chrome through every page:

- **Which pages:** each GET route in `docs/routes.md` without a parameter that answers 200 with a boosted HTML page (kernel pages, JSON and streams are skipped and listed; a route that redirects, like `/live`, smokes the page it lands on). Add pages with parameters with `SMOKE_PATHS=/posts/1,/posts/1/edit`; leave some out with `SMOKE_SKIP` (default `/dev/lab*`). With password sign-in it runs twice, signed out, then signed in as the development demo account (`cargo run -- task auth_demo`; `SMOKE_EMAIL`/`SMOKE_PASSWORD` for another). It signs in through the form with a password field on `SMOKE_LOGIN` (default `/session/new`), whatever that form's action; it tries when `docs/routes.md` has a POST `/session/password`, `/session`, `/sessions`, `/login`, `/sign_in` or `/users/sign_in`, or `SMOKE_SIGNIN_PATH` names the app's own. `prove` signs in the same way.
- **What it checks:** on a full load it clicks each live element (`button[type=button]`, `[data-js]`, `[data-toggle]`, `[role=button]`; not links, submits or `hx-*` requests) and records which ones change the DOM or an attribute, or print, copy or open a dialog. Then it loads `/`, follows a boosted link to the page and clicks them again. It fails when an element reacts only on the full load, when the page loads a script or stylesheet of its own in `<head>`, and on any console error, uncaught exception or CSP violation on either path.
- **What it needs:** Node ≥ 18 and an installed Chrome, Chromium or Edge (`CHROME_PATH` names one; otherwise the usual places on macOS, Linux and Windows). No npm install, and it never downloads a browser: it speaks the DevTools Protocol over a pipe (`cdp.mjs`, the driver `smoke-components.mjs` and `prove` share). Without a browser it prints `skip: no Chrome` and exits 77, which `cargo offrails verify --full` (with `BASE` set) and the framework's CI report as **skipped**, never passed.
- **Measured** on an app with every part (M2, load ~15): 18 page visits (signed out and in) in 28 s, with the sign-in, sign-up, confirm, password and two-factor pages' Show and Copy buttons each clicked on both paths. A bare app (one page, `/`): 4.7 s, most of it starting Chrome.
- **Proof it catches the bug:** `node .offrails/tools/browser-smoke/selftest-boosted.mjs` serves a tiny htmx site whose sign-in page binds its Show button three ways: delegated from the layout (passes), from a head-block script on `DOMContentLoaded` (fails, naming the script and the dead button), from the layout on `DOMContentLoaded` (fails on the dead button), plus a console error (fails). The framework's CI runs it. By hand on a real app: move `<script src="{{ ctx.auth_js }}">` from the layout into `templates/sessions/new.html`'s `{% block head %}`, rebuild, and the smoke fails with `/session/new: loads /static/auth.js in its <head>…` and `button[data-reveal="signin_password"] "Show password" reacts on a full load and is DEAD after a boosted link`; put it back and it passes.

## Proof: `cargo offrails prove`

Tests prove behavior; `prove` shows the human (and the next agent) that the pages work, and ties each acceptance criterion to its evidence. With the app running (`bin/dev`):

```sh
BASE=<the URL bin/dev printed> cargo offrails prove       # --strict: visual changes fail too; --commit-screenshots: keep the PNGs in git
```

- **Which pages:** every GET route in `docs/routes.md` without a parameter that answers HTML (plus `SMOKE_PATHS`, minus `SMOKE_SKIP`, as the smokes), each as the visitor who can see it: signed out, or signed in as the demo account (`SMOKE_EMAIL`/`SMOKE_PASSWORD`; the sign-in form found as the boosted smoke finds it) when it sends you to sign in. Generated show and edit pages have a parameter: add one of each to `SMOKE_PATHS` (`/posts/1,/posts/1/edit`) once a row exists. Fragments, JSON and files are listed under "Not proven here".
- **What it does per page:** full-page screenshots at desktop 1280 and mobile 390, light and dark (`docs/proof/screens/`); it fails on a console error, an uncaught exception or a CSP violation in any of the four loads, on axe-core violations (WCAG 2.2 AA, light and dark; the a11y smoke's pinned axe, from the smokes' cache), on a same-site link that answers 4xx/5xx or an `#anchor` without a target, and on horizontal scroll at 390 px (naming the elements that stick out; on a kernel page, `/dev` or `/admin/…` from the framework and its parts, that is a warning in PROOF.md, failing only with `--strict`). Each screenshot is compared with the previous run's (a pixel diff in Node, no package): changes over `PROVE_DIFF_THRESHOLD` (0.5% of pixels) are reported with a diff image in `docs/proof/diff/`, and fail only with `--strict`.
- **What it writes:** `docs/proof/PROOF.md`: the result, the checks, a table of pages with links to the four screenshots and what changed, and **every acceptance criterion** in `product/features/*.md` → the tests that cite it (`/// AC-…` on a test function: `#[test]`, `#[tokio::test]` or another `…::test` attribute, under `tests/` or in a `#[cfg(test)]` module; a handler or helper whose comment names an AC id is not a test) with their result in the last `cargo offrails verify` (read from `target/verify/`) → the pages those tests request (`"/…"` literals and `paths::…()` helpers in the test, backticked paths in the criterion) with their screenshots. `proof.json` holds the same for programs. PNGs are git-ignored (`docs/proof/.gitignore`) unless `--commit-screenshots`.
- **What it needs:** what `smoke-boosted.mjs` needs, through the same driver (`cdp.mjs`: Node ≥ 18 and an installed Chrome, Chromium or Edge, no npm install; `skip: no Chrome`, exit 77, without one). `cargo offrails verify --full` runs it when `BASE` is set; the framework's CI proves it on the bare app and the app with every part (`tools/ci/test-template.sh prove`).
- **The component smoke:** `BASE=… node .offrails/tools/browser-smoke/smoke-components.mjs` drives `/dev/components` on a full load and after a boosted link: the modal (focus kept inside, Esc, focus returned), the htmx-loaded modal, the confirm dialog (Cancel and Confirm) and the demo form's 422. `verify --full` runs it too.
- **Measured** (M2, other agents building, load 15–70): a bare app (9 pages, 36 screenshots, axe on 18 loads) 13–24 s; an app with every part (63 pages, 6 of them signed in, 252 screenshots) 165–216 s. Pages with live data (charts, timestamps) show up as visual changes on every run: expected, and why changes fail only with `--strict`.

Your final message links `docs/proof/PROOF.md` ([workflow](../workflow.md#definition-of-done)).

## CI

`.github/workflows/ci.yml`. Action majors verified 2026-10-12. When generating, check each action's latest release and use its current major tag.

```yaml
name: CI
on:
  push: { branches: [main] }
  pull_request:

concurrency: { group: "ci-${{ github.ref }}", cancel-in-progress: true }

env:
  CARGO_TERM_COLOR: always
  RUSTFLAGS: -D warnings

jobs:
  # One build, everything at once (`cargo offrails verify`, workflow.md#definition-of-done): fmt,
  # the CSS, clippy (its own target dir, beside the test build), then the SQLite and Postgres
  # suites side by side. Two jobs used to compile the tests twice.
  test:
    runs-on: ubuntu-latest
    services:
      postgres:
        image: postgres:17                      # match the production major (Fly MPG: 16/17)
        env: { POSTGRES_PASSWORD: postgres }
        ports: ["5432:5432"]
        # A throwaway database on tmpfs: nothing waits for a disk (testing.md#postgres-test-databases).
        options: >-
          --tmpfs /var/lib/postgresql/data
          --health-cmd pg_isready --health-interval 2s --health-timeout 5s --health-retries 30
    env:
      TEST_DATABASE_URL: postgres://postgres:postgres@localhost:5432/postgres
    steps:
      - uses: actions/checkout@v7
        with: { fetch-depth: 0 }                  # the migration guard diffs against main
      - uses: dtolnay/rust-toolchain@stable
        with: { components: "rustfmt, clippy" }
      - uses: Swatinem/rust-cache@v2
        with:                                   # the build and clippy's own target dir (verify)
          workspaces: |
            . -> target
            . -> target/clippy
      - uses: taiki-e/install-action@nextest
      - name: Committed migrations are append-only (guides/migrations.md rule 1)
        if: github.event_name == 'pull_request'
        run: test -z "$(git diff --name-only --diff-filter=M origin/${{ github.base_ref }}... -- 'migration/src/m*.rs')"
      - name: Mechanical gates (header, every AC cited, Security-table tests, tenancy suites, scoped lookups, .env.example)
        run: sh .offrails/tools/offrails-check
      - name: Cargo.lock is current (verify's cargo commands don't take --locked)
        run: cargo metadata --locked --format-version 1 > /dev/null
      - run: bin/setup-tailwind                 # verify builds static/app.css (git-ignored; the theme guard reads it)
      - name: fmt, CSS, clippy, one test build, then SQLite and Postgres at once
        run: cargo offrails verify --fast
      # ---- inertia / react / ISLANDS: yes only (the suites above exported the ts-rs types) ----
      - run: git diff --exit-code web/src/types   # generated types must be committed and current
      - uses: pnpm/action-setup@v6
        with: { package_json_file: web/package.json }        # reads "packageManager"
      - uses: actions/setup-node@v7
        with: { node-version: 24, cache: pnpm, cache-dependency-path: web/pnpm-lock.yaml }
      - run: pnpm --dir web install --frozen-lockfile
      - run: pnpm --dir web exec biome ci .
      - run: pnpm --dir web typecheck                        # React: tsc -b; Svelte: svelte-check (never tsc --noEmit)
      - run: pnpm --dir web test --run
      - run: pnpm --dir web build
      - run: pnpm --dir web exec playwright install --with-deps chromium   # E2E (inertia/spa)
      - run: pnpm --dir web e2e                              # webServer: builds web/, serves the app on :3200

  security:                     # security.md#automated-enforcement
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
        with: { fetch-depth: 0 }          # gitleaks scans the whole history
      - uses: taiki-e/install-action@cargo-deny
      - name: cargo deny (advisories, licenses, bans, sources)
        run: cargo deny --locked check
      - name: gitleaks (secrets in any commit)
        run: docker run --rm -v "$PWD:/repo" ghcr.io/gitleaks/gitleaks:v8.30.1 git /repo --config /repo/.gitleaks.toml --redact --no-banner

  docker:                       # the image builds AND passes the ZAP baseline in production mode
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - uses: docker/setup-buildx-action@v4
      - uses: docker/build-push-action@v7
        with: { context: ., push: false, load: true, tags: "app:ci", cache-from: type=gha, cache-to: "type=gha,mode=max" }
      - name: Boot the production image
        run: |
          docker run -d --name app --network host -e APP_ENV=production -e PORT=3000 \
            -e APP_URL=http://localhost:3000 -e "DATABASE_URL=sqlite:///app/storage/ci.sqlite3?mode=rwc" app:ci
          for _ in $(seq 1 60); do curl -fsS localhost:3000/up && break; sleep 1; done
          curl -fsS localhost:3000/up
      - name: OWASP ZAP baseline (fails on any WARN not triaged in .zap/rules.tsv)
        run: |
          chmod 777 .zap
          docker run --rm --network host -v "$PWD/.zap:/zap/wrk:rw" ghcr.io/zaproxy/zaproxy:stable \
            zap-baseline.py -t http://localhost:3000 -c rules.tsv -r zap-report.html -m 2
          # Locally on Docker Desktop (macOS/Windows), --network host doesn't reach the host. Drop it and target
          # http://host.docker.internal:<port> instead.
      - if: failure()
        uses: actions/upload-artifact@v7
        with: { name: zap-report, path: .zap/zap-report.html, if-no-files-found: ignore }
      - if: always()
        run: docker logs app | tail -50
```

The security and Docker jobs are actionlint-clean. TS modes add `pnpm --dir web audit --prod` to the frontend job. Run ZAP locally against `app serve` with `APP_ENV=production` and `APP_URL=http://host.docker.internal:$PORT`, adding `--add-host=host.docker.internal:host-gateway` to the ZAP container. Verified: 0 FAIL, 0 WARN, 61 PASS on the reference app.

- In `htmx` mode without islands, delete the frontend steps. The CSS is built because the theme guard (`theme_test`) reads `static/app.css`.
- **Why one test job.** The SQLite and Postgres suites used to be two jobs, each compiling the tests. `verify` compiles once and runs both at once against the job's Postgres service (on tmpfs), while clippy checks beside the build in `target/clippy` (cached as a second target dir). Coverage, deny and gitleaks stay out of the critical path: deny and gitleaks in `security`, coverage locally in `verify --full`. Sharding the suites across runners (`nextest --partition`) doesn't pay at this size: each shard would compile the tests again, and the suites run in seconds once built.
- **The Docker job** builds the dependencies in a layer of their own (`Dockerfile`), which the `gha` layer cache keeps; before 0.7.2 a cache mount (not in that cache) made every run compile every dependency in release mode.
- `--locked` fails if `Cargo.lock` is stale, and `--frozen-lockfile` does the same for `pnpm-lock.yaml`. Commit both.

## Gotchas agents get wrong

- `sqlite::memory:` with a pool > 1: each connection sees an empty database, giving "no such table" flakes.
- Creating a new test binary per file in `tests/`.
- Forgetting `Sec-Fetch-Site` on POSTs in the helper, then "fixing" CSRF failures by disabling the check in tests.
- Asserting exact HTML strings. Assert on meaningful text or the inertia props instead.
- `assert_text` with an apostrophe (or `&`, `"`, `<`) that the template typed literally. `assert_text` escapes its argument the way Askama escapes `{{ … }}` output (`we've` → `we&#39;ve`), so it finds values the template interpolated; text written straight into a template (`<p>We've sent it</p>`) is in the body as typed: use `assert_contains("We've sent it")` for that, or move the text into a `{{ }}` value.
- Skipping the Postgres run because "SQLite passed". CI requires both.