# Deployment

> **TL;DR**
> - One Docker image runs web, worker and migrations; `bin/docker-entrypoint` runs `app db migrate`, then the command. `/up` is the health check.
> - Generate config only for the chosen `DEPLOY_TARGET` (Kamal, Fly, Render, Railway); `docker compose up` runs the whole app locally.
> - Schema changes are expand → migrate → contract; SQLite in production means one machine with a volume and backups.
> - Before deploying: green DoD, **green CI on the exact commit**, the image builds and answers `/up`, every env var is in `.env.example` + `Config` + the deploy config. Outward-facing steps need the human's yes.
> - Behind Cloudflare: turn off script injection (Web Analytics, Rocket Loader, email obfuscation) instead of loosening the CSP. Open-source apps keep the real deploy config git-ignored.
>
> Read the rest when shipping, choosing a target, running migrations remotely, or rolling back.

One Docker image runs everything: web, worker, migrations. The deploy target was chosen in the [interview](../interview.md) (`DEPLOY_TARGET` in the app's `AGENTS.md`). **Generate config only for the chosen target.** If it's "later", generate the Dockerfile and entrypoint only, and stop there.

| Rails 8 | Off Rails |
|---------|-----------|
| `Dockerfile` (generated by `rails new`) | `Dockerfile` (generated by `generators/new.md`) |
| `bin/docker-entrypoint` → `db:prepare` | `bin/docker-entrypoint` → `app db migrate`, then `exec "$@"` |
| Kamal 2 + kamal-proxy | Kamal 2 (same tool; it deploys any image) |
| Thruster | Not needed (axum + compression + static cache headers) |
| `/up` | `/up` |
| Solid Queue in Puma | `APP_INLINE_WORKER=true` (single machine) |
| `bin/jobs` role | `app worker` process/role (`--queues` per role) |
| Mission Control – Jobs | `/admin/jobs`, behind `AdminAccess` ([jobs-dashboard](jobs-dashboard.md)) |

## Dockerfile shape

Verified 2026-10-12 by building and running this exact file (Debian 13 "trixie" images; build and runtime stages must share the Debian release, or the binary's glibc won't match).

```dockerfile
# syntax=docker/dockerfile:1
# Versions: look up the current stable Rust (`rustc --version` after `rustup update`) and the Node LTS major.
ARG RUST_VERSION=1.99
ARG NODE_VERSION=24

# --- frontend (inertia / react / islands only; delete this stage in plain htmx mode) ---
FROM node:${NODE_VERSION}-trixie-slim AS web
RUN corepack enable pnpm
WORKDIR /app/web
COPY web/package.json web/pnpm-lock.yaml ./
RUN pnpm install --frozen-lockfile
COPY web/ ./
# Output: /app/web/dist plus .vite/manifest.json
RUN pnpm build

# --- rust ---
FROM rust:${RUST_VERSION}-trixie AS build
WORKDIR /app
COPY . .
# htmx mode only: build the CSS with the pinned standalone Tailwind (bin/setup-tailwind, see frontend-htmx.md)
RUN bin/setup-tailwind && bin/tailwindcss -i assets/app.css -o static/app.css --minify
RUN --mount=type=cache,target=/usr/local/cargo/registry \
    --mount=type=cache,target=/app/target \
    cargo build --release --locked && cp target/release/myapp /usr/local/bin/app

# --- runtime ---
FROM debian:trixie-slim
RUN apt-get update && apt-get install -y --no-install-recommends ca-certificates sqlite3 postgresql-client \
    && rm -rf /var/lib/apt/lists/*
RUN useradd --create-home --uid 1000 app
WORKDIR /app
COPY --from=build /usr/local/bin/app /usr/local/bin/app
COPY --from=build /app/static ./static
# inertia / react / islands: served at /assets (hashed, immutable); kernel/assets.rs reads the manifest at boot
COPY --from=web /app/web/dist ./web/dist
COPY bin/docker-entrypoint /usr/local/bin/docker-entrypoint
RUN mkdir -p /app/storage /data && chown -R app:app /app /data
USER app
ENV APP_ENV=production PORT=3000 LOG_FORMAT=json
EXPOSE 3000
ENTRYPOINT ["docker-entrypoint"]
CMD ["app", "serve"]
```

- **Runtime-read files need their own `COPY`.** Askama templates compile into the binary, and `static/` and `web/dist` are copied above; anything the app reads from disk at runtime (a `content/` directory, the vendored `.offrails/` a docs site renders) needs its own `COPY --from=build /app/<dir> ./<dir>` in the runtime stage, and a test that the path config points at it.
- **Comments in a Dockerfile go on their own line.** `COPY a b  # note` passes `#` and `note` as extra source paths and fails the build.
- `myapp` is the crate name. The binary is renamed to `app` in the image, so every command below says `app`.
- `RUN corepack enable pnpm` uses `web/package.json`'s `"packageManager"` pin. The Node 24 images ship corepack; Node 25+ images don't, so there use `RUN npm install -g pnpm@<pinned>`.

`bin/docker-entrypoint`:

```sh
#!/bin/sh
set -e
if [ "${SKIP_MIGRATIONS:-false}" != "true" ]; then app db migrate; fi
exec "$@"
```

- Templates are compiled into the binary (Askama), and migrations are embedded. Only `static/` (served at `/static`) and `web/dist/` (Vite output; `web/dist/assets` is served at `/assets`, and the manifest must exist or boot fails in production) are copied as files. Paths are relative to `WORKDIR /app`, so keep it.
- `.dockerignore`: `target/`, `web/node_modules/`, `web/dist/`, `storage/`, `.env`, `.git/`, `bin/tailwindcss` (the host binary may be macOS; the image fetches its own).
- **Exactly one process migrates.** `sea-orm-migration` takes **no lock**: two processes running `app db migrate` at once (web and worker booting together, or two web replicas) race on `seaql_migrations` and fail with "relation already exists" or a duplicate row on any deploy that adds a migration. So: a platform release command (`fly.toml` `[deploy] release_command`, Render's pre-deploy command, Kamal's `pre-deploy` hook) runs `app db migrate` once, and **every** process sets `SKIP_MIGRATIONS=true`. Without a release command (compose, a single Kamal web server), only the one web container migrates on boot and every other process (`worker`, extra replicas) sets `SKIP_MIGRATIONS=true`.
- **Every proxy needs `APP_TRUSTED_PROXY=true`** (and `APP_TRUSTED_PROXY_HOPS` when more than one proxy appends to `X-Forwarded-For`): Fly, Render, Railway and compose-behind-Caddy all are proxies. Without it the rate limiter keys every user on the proxy's address, and ten failed logins by anyone lock out everyone ([security](security.md#rate-limiting-kernelrate_limitrs)).
- Verify the image locally before the first deploy: `docker compose up --build` (below), or for SQLite apps `docker build -t myapp . && docker run --rm -p 3000:3000 -e APP_URL=http://localhost:3000 -e DATABASE_URL='sqlite:///data/p.sqlite3?mode=rwc' myapp`, then `curl localhost:3000/up`.

## Docker compose: the whole app, locally

Every app has a `compose.yaml` from `new` that runs **the production image** against Postgres, so "does it work in a container, on Postgres, with migrations on boot?" is one command, long before a deploy. `bin/dev` stays the everyday loop (native, SQLite, fast rebuilds).

```yaml
# The whole app in containers, production-shaped: `docker compose up --build`, then http://localhost:3000.
# Same image as the deploy (Dockerfile), Postgres, migrations on boot (bin/docker-entrypoint).
# Day-to-day development is `bin/dev` (native, SQLite, fast rebuilds); this is "does it run like prod?".
#
#   docker compose up --build                     app + Postgres (the worker runs inline)
#   docker compose --profile worker up --build    + a separate worker process (APP_INLINE_WORKER=false)
#   docker compose --profile mail up --build      + Mailpit at http://localhost:8025 (real SMTP delivery)
#   docker compose up -d db                       only Postgres, for `TEST_DATABASE_URL=… cargo nextest run`
#   docker compose down -v                        stop and delete the database volume
#
# Ports are overridable: APP_PORT (3000), DB_PORT (5432), MAILPIT_PORT (8025).
name: myapp

x-app: &app
  build: .
  image: myapp:local
  depends_on:
    db:
      condition: service_healthy
  environment: &app-env
    APP_ENV: production
    DATABASE_URL: postgres://postgres:postgres@db:5432/myapp
    APP_URL: http://localhost:${APP_PORT:-3000}
    APP_INLINE_WORKER: ${APP_INLINE_WORKER:-true}
    APP_ADMIN_PASSWORD: ${APP_ADMIN_PASSWORD:-change-me}
    # With `--profile mail`: SMTP_URL=smtp://mailpit:1025 docker compose --profile mail up
    SMTP_URL: ${SMTP_URL:-}
    RUST_LOG: ${RUST_LOG:-info}

services:
  app:
    <<: *app
    ports: ["${APP_PORT:-3000}:3000"]
    healthcheck:
      test: ["CMD", "app", "health"]
      interval: 5s
      timeout: 5s
      retries: 20
      start_period: 10s

  worker:
    <<: *app
    profiles: [worker]
    command: ["app", "worker"]
    environment:
      <<: *app-env
      SKIP_MIGRATIONS: "true"            # the app service migrates
    depends_on:
      app:
        condition: service_healthy

  db:
    image: postgres:17                 # match your production major (managed hosts commonly run 16/17); CI uses the same
    environment:
      POSTGRES_PASSWORD: postgres
      POSTGRES_DB: myapp
    ports: ["127.0.0.1:${DB_PORT:-5432}:5432"]
    volumes: ["pg:/var/lib/postgresql/data"]   # postgres:18+ moved PGDATA: mount /var/lib/postgresql instead
    # Upgrading an app that already has a compose file: keep its existing volume name, or you orphan local data.
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U postgres -d myapp"]
      interval: 2s
      timeout: 3s
      retries: 30

  mailpit:
    image: axllent/mailpit:v1.31
    profiles: [mail]
    ports: ["127.0.0.1:${MAILPIT_PORT:-8025}:8025"]

volumes:
  pg:
```

- The `app` healthcheck is `app health`, a subcommand that requests `GET /up` on `127.0.0.1:$PORT` (the slim runtime image has no curl or wget). With direct TLS it only checks the port opens.
- `APP_ENV=production` inside compose on purpose: it's the deploy's behavior (JSON logs, `/dev` not mounted, the admin password on `/admin/jobs`). `APP_ADMIN_PASSWORD` defaults to `change-me` here only.
- Don't use `env_file: .env`: `.env` holds development values (SQLite URL, `APP_ENV=development`). Compose reads `.env` only to fill `${…}` defaults.
- **A container running development needs `APP_BIND=0.0.0.0`.** Development listens on `127.0.0.1` by default ([security](security.md#development-is-this-machine-only)), which inside a container is the container itself: the published port then answers nothing. Production (this file, the Dockerfile) binds `0.0.0.0` already. A dev container or a compose override with `APP_ENV: development` sets `APP_BIND: 0.0.0.0`, publishes the port on `127.0.0.1:` only (`ports: ["127.0.0.1:3000:3000"]`, or the LAN can open `/dev`), and adds `APP_ALLOWED_HOSTS` for any name other than `localhost` it's opened by.
- `mailpit` is pinned to a tag, never `latest`. Check for a current one before generating.
- Verified 2026-10-13 with the `offrails-api-demo` app: `docker compose up --build` from scratch (app + db healthy, migrations on boot, REST and GraphQL on Postgres), `--profile mail` (welcome email delivered to Mailpit over SMTP), `--profile worker` (jobs claimed by the worker container's node, none inline).

## Direct TLS (optional)

The default is plain HTTP inside the container and TLS at the platform's proxy (which gives the browser h2). **In development, and on any box with no proxy,** set `APP_TLS_CERT` and `APP_TLS_KEY` (PEM paths) and `serve` uses `axum-server`'s rustls acceptor: HTTPS with **h2 and http/1.1 via ALPN**, graceful shutdown through its `Handle`.

```rust
let _ = rustls::crypto::aws_lc_rs::default_provider().install_default();
let cfg = axum_server::tls_rustls::RustlsConfig::from_pem_file(&cert, &key).await?;
// `listener` is the one tokio TcpListener `serve` already bound; never bind_rustls(addr) on top of it.
let acceptor = axum_server::tls_rustls::RustlsAcceptor::new(cfg).acceptor(kernel::tls::HttpToHttps);
axum_server::from_tcp(listener.into_std()?)?.acceptor(acceptor).handle(handle).serve(service).await?;
// on SIGTERM: handle.graceful_shutdown(Some(config.shutdown_drain)), the full branch: new.md step 4.12
```

**Plain HTTP on the TLS port** (a bare `localhost:3000` in the address bar, an `http://` bookmark) used to get TLS bytes back and show `ERR_INVALID_HTTP_RESPONSE`. `kernel/tls.rs` is an inner acceptor that peeks at the first byte: `0x16` (a TLS handshake) goes on to rustls; anything else gets `307 Temporary Redirect` to `https://<Host><path>` and is closed (an error from the acceptor drops the connection; axum-server ignores it). 307, never 308: browsers cache permanent redirects, which would break `APP_TLS=off` on the same port later. The peek has a 10 s timeout (rustls' handshake timeout doesn't cover it), the request head is capped at 8 KiB, the path must start with one `/`, and the Host may only hold host/port characters:

The whole file, with its tests. Copy it as is; it needs no new crate (axum-server, tokio). Verified with axum-server 0.8.0 and tokio 1.53 in an app that upgraded from this spec (its four tests pass):

```rust
// src/kernel/tls.rs
//! Plain HTTP on the TLS port (a bare `localhost:PORT` in the address bar, an `http://` bookmark)
//! gets a 307 to `https://` instead of TLS bytes (`ERR_INVALID_HTTP_RESPONSE`). An inner acceptor
//! for axum-server's `RustlsAcceptor`: it peeks at the first byte; `0x16` (a TLS handshake) goes on
//! to rustls, anything else is answered and closed. 307, never 308: a cached permanent redirect
//! would break `APP_TLS=off` on the same port later. Spec: .offrails/guides/deployment.md#direct-tls-optional
//!
//! Only `serve`'s direct-TLS branch uses it (development with a trusted certificate); production
//! serves plain HTTP behind its TLS proxies and never reaches this code.

use std::future::Future;
use std::io;
use std::pin::Pin;
use std::time::Duration;

use axum_server::accept::Accept;
use tokio::io::{AsyncReadExt, AsyncWriteExt};
use tokio::net::TcpStream;
use tokio::time::timeout;

/// How long a connection may stay silent before its first byte (rustls' handshake timeout starts
/// only after this acceptor hands the stream on).
pub const SNIFF_TIMEOUT: Duration = Duration::from_secs(10);
/// The largest plain-HTTP request head read before answering.
pub const MAX_HEAD: usize = 8 * 1024;

#[derive(Clone, Copy, Debug, Default)]
pub struct HttpToHttps;

impl<S: Send + 'static> Accept<TcpStream, S> for HttpToHttps {
    type Stream = TcpStream;
    type Service = S;
    type Future = Pin<Box<dyn Future<Output = io::Result<(TcpStream, S)>> + Send>>;

    fn accept(&self, stream: TcpStream, service: S) -> Self::Future {
        Box::pin(async move {
            let mut first = [0u8; 1];
            let n = timeout(SNIFF_TIMEOUT, stream.peek(&mut first))
                .await
                .map_err(|_| {
                    io::Error::new(io::ErrorKind::TimedOut, "no bytes before timeout")
                })??;
            if n == 1 && first[0] == 0x16 {
                return Ok((stream, service));
            }
            let _ = timeout(SNIFF_TIMEOUT, redirect_to_https(stream)).await;
            Err(io::Error::other("plain HTTP on the HTTPS port: redirected"))
        })
    }
}

/// Reads the request head (up to `MAX_HEAD`), writes the 307 (or a 400 when the head has no usable
/// Host or path), and shuts the connection down.
async fn redirect_to_https(mut stream: TcpStream) -> io::Result<()> {
    let mut head = Vec::new();
    let mut buf = [0u8; 1024];
    while head.len() < MAX_HEAD && !head.windows(4).any(|w| w == b"\r\n\r\n") {
        let n = stream.read(&mut buf).await?;
        if n == 0 {
            break;
        }
        head.extend_from_slice(&buf[..n]);
    }
    let response = match redirect_target(&String::from_utf8_lossy(&head)) {
        Some(location) => format!(
            "HTTP/1.1 307 Temporary Redirect\r\nLocation: {location}\r\nContent-Length: 0\r\nConnection: close\r\n\r\n"
        ),
        None => {
            "HTTP/1.1 400 Bad Request\r\nContent-Length: 0\r\nConnection: close\r\n\r\n".to_owned()
        }
    };
    stream.write_all(response.as_bytes()).await?;
    stream.shutdown().await
}

/// The `https://` URL for a plain-HTTP request head: `https://<Host><path>`. `None` without a
/// Host, with a Host holding anything but host/port characters, or with a path that doesn't start
/// with exactly one `/` (`//evil.example` would be a protocol-relative open redirect).
pub fn redirect_target(head: &str) -> Option<String> {
    let mut lines = head.split("\r\n");
    let mut request = lines.next()?.split(' ');
    let (_method, path) = (request.next()?, request.next()?);
    if !path.starts_with('/') || path.starts_with("//") || path.contains(['\r', '\n', ' ']) {
        return None;
    }
    let host = lines.take_while(|l| !l.is_empty()).find_map(|l| {
        let (name, value) = l.split_once(':')?;
        name.trim()
            .eq_ignore_ascii_case("host")
            .then(|| value.trim())
    })?;
    let host_ok = !host.is_empty()
        && host
            .chars()
            .all(|c| c.is_ascii_alphanumeric() || matches!(c, '.' | '-' | ':' | '[' | ']'));
    host_ok.then(|| format!("https://{host}{path}"))
}

#[cfg(test)]
mod tests {
    use super::*;
    use tokio::net::TcpListener;

    #[test]
    fn keeps_host_and_path() {
        assert_eq!(
            redirect_target("GET /books?x=1 HTTP/1.1\r\nHost: localhost:3311\r\n\r\n").as_deref(),
            Some("https://localhost:3311/books?x=1")
        );
        assert_eq!(
            redirect_target("GET / HTTP/1.1\r\nhost: [::1]:3000\r\nAccept: */*\r\n\r\n").as_deref(),
            Some("https://[::1]:3000/")
        );
    }

    #[test]
    fn refuses_open_redirects_odd_hosts_and_missing_host() {
        assert_eq!(
            redirect_target("GET //evil.example/ HTTP/1.1\r\nHost: a\r\n\r\n"),
            None
        );
        assert_eq!(
            redirect_target("GET http://x/ HTTP/1.1\r\nHost: a\r\n\r\n"),
            None
        );
        assert_eq!(
            redirect_target("GET / HTTP/1.1\r\nHost: evil.example/x\r\n\r\n"),
            None
        );
        assert_eq!(redirect_target("GET / HTTP/1.1\r\nHost: a@b\r\n\r\n"), None);
        assert_eq!(
            redirect_target("GET / HTTP/1.1\r\nAccept: */*\r\n\r\n"),
            None
        );
        assert_eq!(redirect_target(""), None);
    }

    async fn accept_one(listener: TcpListener) -> io::Result<(TcpStream, ())> {
        let (stream, _) = listener.accept().await?;
        HttpToHttps.accept(stream, ()).await
    }

    #[tokio::test]
    async fn answers_plain_http_with_a_307() {
        let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
        let addr = listener.local_addr().unwrap();
        let server = tokio::spawn(accept_one(listener));
        let mut client = TcpStream::connect(addr).await.unwrap();
        client
            .write_all(b"GET /docs?q=1 HTTP/1.1\r\nHost: localhost:3000\r\n\r\n")
            .await
            .unwrap();
        let mut out = String::new();
        client.read_to_string(&mut out).await.unwrap();
        assert!(out.starts_with("HTTP/1.1 307 "), "{out}");
        assert!(
            out.contains("Location: https://localhost:3000/docs?q=1\r\n"),
            "{out}"
        );
        assert!(
            server.await.unwrap().is_err(),
            "the connection is not handed to rustls"
        );
    }

    #[tokio::test]
    async fn passes_a_tls_handshake_through() {
        let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
        let addr = listener.local_addr().unwrap();
        let server = tokio::spawn(accept_one(listener));
        let mut client = TcpStream::connect(addr).await.unwrap();
        client.write_all(&[0x16, 0x03, 0x01]).await.unwrap();
        let (mut stream, ()) = server.await.unwrap().unwrap();
        let mut first = [0u8; 3];
        stream.read_exact(&mut first).await.unwrap();
        assert_eq!(first, [0x16, 0x03, 0x01], "nothing consumed before rustls");
    }
}
```

What the tests prove: `redirect_target` (pure: request head → `Option<String>`) keeps host and path and refuses `//evil` paths, absolute-form targets, odd hosts and a missing Host; the acceptor, on a real `TcpListener`, answers a plain request with 307 and the `Location` and never hands it to rustls, and passes a `0x16` first byte through with nothing consumed. Verified with curl: `curl -si 'http://localhost:3311/books?x=1'` → `307`, `Location: https://localhost:3311/books?x=1`.

- **Why it matters beyond the box:** live pages need HTTP/2 in the browser, or realtime breaks at about 6 tabs ([realtime](realtime.md#many-tabs-stall-http11-vs-http2)). `bin/dev` serves HTTPS whenever a locally trusted certificate exists, and never with an untrusted one: browsers drop cookies on it ([dev-tools](dev-tools.md#https-in-development)).
- Install a crypto provider first: with sqlx/lettre in the tree, rustls can't pick one alone and panics at the first handshake.
- Certificates come from your own ACME client (renew, then restart, or call `tls.reload_from_pem_file`). If you need automatic certificates, put Kamal-proxy or Caddy in front instead, and leave `APP_TLS_*` unset.
- Verified 2026-10-13 with a self-signed certificate: `curl -k` negotiated h2, `--http1.1` worked, and `app health` passed.

## Workers and queues

- One `app worker` role runs every queue in `APP_QUEUES` by default. Split roles only when a queue's work hurts the others (for example, slow imports): `app worker --queues imports:2` on one role, `app worker --queues default:10,mailers:5` on another. **Every queue in use must be run by some role.** `/admin/jobs` flags queues with no nodes.
- Concurrency is per node, so scaling a worker role to 3 machines triples throughput.
- Set the platform's stop timeout above `APP_JOBS_SHUTDOWN_GRACE` (25s): Fly `kill_timeout = 30` (plus `kill_signal = "SIGTERM"`; Fly's default is SIGINT, so the app must treat both as shutdown), Kamal `stop_timeout: 30`, Railway `drainingSeconds` (service settings), Render's default 30s grace (`maxShutdownDelaySeconds` in the service settings if you need more). Jobs still running at the deadline are rescued by Lifeline and re-run, so they must be idempotent anyway.
- Set `APP_ADMIN_PASSWORD` as a secret if the app has no admin role. Otherwise `/admin/jobs` is off in production.
- Inspect production jobs from your terminal with the platform's exec (below): `app jobs stats`, `app jobs list --state discarded`.

## WebSockets (channel rooms)

Only for apps with channel rooms ([channels](channels.md)). SSE signals need none of this.

- **Proxies:** Fly's proxy and kamal-proxy both pass WebSocket upgrades through with no extra config. The server pings every 20s (WebSocket control frames), which keeps connections alive through proxy idle timeouts (commonly 60s). *Check the current idle timeout of your proxy, and keep the ping interval below it.* A proxy or CDN that breaks WebSockets (some corporate proxies, misconfigured tunnels) doesn't break the app: clients fall back to SSE + POST, then long-polling; or set `APP_REALTIME_TRANSPORTS=sse,longpoll` to skip the attempt ([channels#transports](channels.md#transports)).
- **Deploys:** on SIGTERM, `serve` calls `hub.close()`. Every socket gets `bye` `restart` and closes with **1012**; clients reconnect with backoff and rejoin with `since`, and SSE live regions replay what they missed from their cursor ([realtime#resume](realtime.md#resume)), so nothing is lost. Keep the stop timeout from the section above. The sockets close immediately, so they don't eat into it.
- **More than one machine:** set `APP_CLUSTER=true` (Postgres) and give each machine a stable `APP_NODE`. On Fly, `APP_NODE=$FLY_MACHINE_ID`, which is set in every machine's environment. Kamal sets nothing like it by default, so leave it unset there (`<hostname>:<pid>` is unique per container).
  - No sticky sessions are needed: any node can serve any socket, and the owner of a topic is computed, not routed.
  - (`fly-replay` could route a socket straight to its topic's owner machine to save the hop; not needed, and not implemented.)
- **One machine on SQLite:** no cluster, and nothing to configure. One Rust process holds tens of thousands of sockets.
- **Smoke after deploy:** open a page with a room in two browsers, post, then redeploy and change something mid-deploy. Both pages should reconnect and fill in, without a reload.

## Database choices in production

| `PROD_DATABASE` | Setup |
|-----------------|-------|
| Postgres (default) | Managed Postgres from the platform. It provides `DATABASE_URL`. Run web and worker as separate processes |
| SQLite | **Exactly one machine.** Persistent volume mounted at `/data`. `DATABASE_URL=sqlite:///data/production.sqlite3?mode=rwc`. `APP_INLINE_WORKER=true` (no separate worker). Backups: Litestream to S3-compatible storage, or the platform's volume snapshots, decided with the human and logged |

## Targets

### Fly.io (`fly.toml`)

Verified with flyctl 0.4.115 and https://docs.fly.io/reference/configuration.

1. `fly launch --no-deploy --copy-config` with the generated `fly.toml` (the human runs it, or you do if authorized). It creates the app and keeps our config.
2. Postgres: `fly mpg create --name myapp-db --region <region> --pg-major-version 17` → `fly mpg attach <cluster-id> -a myapp` (sets the `DATABASE_URL` secret). Keep the pooler in its default **session** mode. Transaction mode breaks sqlx's prepared statements.
3. Secrets: `fly secrets set APP_URL=https://myapp.fly.dev SMTP_URL=… APP_ADMIN_PASSWORD=…`
4. `fly deploy`

```toml
app = "myapp"
primary_region = "ams"
kill_signal = "SIGTERM"          # Fly's default is SIGINT
kill_timeout = 30                # seconds, > APP_JOBS_SHUTDOWN_GRACE

[deploy]
  release_command = "app db migrate"   # Postgres only (release machines get no volumes); replaces CMD, not ENTRYPOINT

[env]
  SKIP_MIGRATIONS = "true"       # the release command migrates; the entrypoint doesn't need to
  APP_TRUSTED_PROXY = "true"     # Fly's proxy appends the client to X-Forwarded-For (1 hop)

[processes]
  web = "app serve"
  worker = "app worker"          # Postgres only

[http_service]
  processes = ["web"]
  internal_port = 3000
  force_https = true
  auto_stop_machines = "stop"    # "off" when APP_INLINE_WORKER=true (SQLite): a stopped machine runs no jobs
  auto_start_machines = true
  min_machines_running = 1

  [[http_service.checks]]
    grace_period = "10s"
    interval = "15s"
    timeout = "5s"
    method = "GET"
    path = "/up"                 # must return 2xx; a redirect counts as a failure
```

- **SQLite instead of Postgres:** delete `[deploy]`, the `SKIP_MIGRATIONS` env and the `worker` process. Set `APP_INLINE_WORKER = "true"` and `DATABASE_URL = "sqlite:///data/production.sqlite3?mode=rwc"` in `[env]`, set `auto_stop_machines = "off"`, and add `[mounts] source = "data"` / `destination = "/data"`. Create the volume with `fly volumes create data --size 1 --region <region>`. Never scale beyond 1 machine.
- Remote commands: `fly ssh console -C "app db status"`, `fly ssh console -C "app jobs stats"`, `fly mpg connect` (psql to MPG).
- Rollback: `fly releases --image`, then `fly deploy --image <previous image>`.

### Kamal 2 (`config/deploy.yml`)

Verified against Kamal 2.12 docs (https://kamal-deploy.org/docs/configuration/).

```yaml
service: myapp
image: <registry-user>/myapp
servers:
  web:
    hosts: [203.0.113.10]
  job:                                  # Postgres only; SQLite uses APP_INLINE_WORKER
    hosts: [203.0.113.10]
    cmd: app worker
    stop_timeout: 30                    # > APP_JOBS_SHUTDOWN_GRACE
proxy:
  ssl: true
  host: myapp.example.com
  app_port: 3000
  healthcheck: { path: /up }
registry: { username: <user>, password: [KAMAL_REGISTRY_PASSWORD] }
builder:
  arch: amd64                           # the server's arch; building on Apple Silicon needs this (or `remote:`)
env:
  clear:
    APP_ENV: production
    APP_URL: https://myapp.example.com
    APP_TRUSTED_PROXY: "true"           # the proxy in front appends the client IP; 2 hops when Cloudflare/Caddy sits in front too
    SKIP_MIGRATIONS: "true"             # the pre-deploy hook migrates once (below)
  secret: [DATABASE_URL, SMTP_URL, APP_ADMIN_PASSWORD]
volumes: ["myapp_data:/data"]           # SQLite and/or local storage
accessories:                            # only if self-hosting Postgres
  db:
    image: postgres:17
    host: 203.0.113.10
    port: "127.0.0.1:5432:5432"
    env:
      clear: { POSTGRES_DB: myapp_production }
      secret: [POSTGRES_PASSWORD]
    directories: ["data:/var/lib/postgresql/data"]   # postgres:18+ moved PGDATA: mount /var/lib/postgresql instead
```

- Migrations run once per deploy. With **one** `web` host: let that container migrate on boot by overriding the role's env (`servers.web.env.clear.SKIP_MIGRATIONS: "false"`; roles take their own `env:`) and keep `SKIP_MIGRATIONS=true` on `job`, which boots at the same time and would otherwise race it. With several `web` hosts (deployed in parallel), none may migrate on boot: run `app db migrate` once from a `.kamal/hooks/pre-deploy` hook against the new image (Kamal's hooks docs), with every role on `SKIP_MIGRATIONS=true`.
- `.kamal/secrets` pulls values from env or a password manager. Never commit literal secrets. With the accessory, `DATABASE_URL=postgres://postgres:$POSTGRES_PASSWORD@myapp-db:5432/myapp_production` (accessory containers are reachable by `<service>-<accessory>` on the kamal network).
- First deploy: `kamal setup` (installs Docker, boots accessories, deploys). Accessories aren't redeployed by `kamal deploy`; use `kamal accessory boot db` / `reboot db`.
- Remote commands: `kamal app exec "app db status"`, `kamal app exec -i "app db console"`. Rollback: `kamal rollback <version>`.

#### Kamal on a shared host (several apps, one box)

This is the cheapest production setup, and the common one: one VPS already runs `kamal-proxy` and other apps.

- **Routing:** each app sets `proxy.host: <domain>` and `proxy.ssl: false`. kamal-proxy routes by `Host`, so **don't publish ports** from the app.
- **TLS:** if Cloudflare (or a front proxy like Caddy) terminates TLS, `ssl: false` is correct, and the app must see the real client IP. Set `APP_TRUSTED_PROXY=true`, and make sure the front proxy trusts Cloudflare's `X-Forwarded-For` (Caddy: `trusted_proxies static private_ranges` when the origin is firewalled to Cloudflare only).
- **Front proxy:** when a shared Caddy sits in front of kamal-proxy, a new domain needs a block in **that** Caddyfile (`http://<domain> { reverse_proxy kamal-proxy:80 }`, plus a `www` → apex redirect), owned by whichever project runs the Caddy accessory. Write the block for the human. Editing or reloading shared infrastructure is outward-facing, so it always needs explicit confirmation, because every app on the host goes through it.
- **SQLite:** a **named volume** (`"<app>_storage:/app/storage"`) and `DATABASE_URL=sqlite:///app/storage/production.sqlite3?mode=rwc`. Docker seeds a new named volume from the image directory, so ownership by the non-root `app` user carries over. Run one container, never two, against one SQLite file.
- **Builds:** `builder.arch: amd64`, and `builder.remote` pointing at a build host if the production box is busy. Kamal builds from **committed** HEAD; uncommitted changes aren't deployed.

#### Behind Cloudflare: turn off script injection

Cloudflare injects scripts into proxied HTML on its own: **Web Analytics / RUM** (`static.cloudflareinsights.com/beacon.min.js`, auto-enabled on new free zones), Rocket Loader, Email Address Obfuscation, and Bot Fight Mode's JS detections. The strict CSP ([security](security.md)) correctly blocks them, which logs a console error and an Issues-panel entry, and that costs points in Lighthouse Best Practices. **Never loosen the CSP to fit them.** Turn them off at the zone:

- Web Analytics: `auto_install: false` alone is not enough; new zones get an enabled ruleset with a catch-all rule. Register or update the site with both off: `POST /accounts/{account}/rum/site_info {"zone_tag": "<zone>", "auto_install": false}`, then `PUT /accounts/{account}/rum/site_info/{site_tag} {"zone_tag": "<zone>", "auto_install": false, "enabled": false}`. The injection stops within about a minute.
- Rocket Loader and Email Obfuscation: `PATCH /zones/{zone}/settings/rocket_loader {"value":"off"}`, and the same for `email_obfuscation`.
- Verify with a real browser (or `curl -A '<Chrome UA>'`, since injection skips non-browser clients): the page must load with a clean console.

### Open-source apps: keep the deploy config private

If the repository is public:
- commit `config/deploy.example.yml` with `<placeholders>` for hosts, registry user and builder;
- git-ignore the real `config/deploy.yml`, `config/deploy.*.yml` (except the example) and `.kamal/secrets*`, and add them to `.dockerignore`;
- `.kamal/secrets` only references environment variables (`KEY=$KEY`), never literal values;
- `docs/deploy.md` describes the topology generically (no IPs, hostnames of other services or account names);
- before every commit touching deploy files, grep the staged diff for IPs, emails and `=` followed by a literal secret.

### Render (`render.yaml` Blueprint)

```yaml
services:
  - type: web
    name: myapp
    runtime: docker
    healthCheckPath: /up
    preDeployCommand: app db migrate
    envVars:
      - { key: DATABASE_URL, fromDatabase: { name: myapp-db, property: connectionString } }
      - { key: SKIP_MIGRATIONS, value: "true" }
      - { key: APP_TRUSTED_PROXY, value: "true" }
      - { key: APP_URL, sync: false }
      - { key: APP_ADMIN_PASSWORD, sync: false }
  - type: worker
    name: myapp-worker
    runtime: docker
    dockerCommand: app worker
    envVars:
      - { key: DATABASE_URL, fromDatabase: { name: myapp-db, property: connectionString } }
      - { key: SKIP_MIGRATIONS, value: "true" }
      - { key: APP_TRUSTED_PROXY, value: "true" }
databases:
  - name: myapp-db
    postgresMajorVersion: "17"
```

`sync: false` values are prompted once, when the Blueprint is created. SQLite on Render needs a `disk:` (`name`, `mountPath: /data`, `sizeGB`) and a single instance. Prefer Postgres.

### Railway

Railway's config-as-code file (`railway.json`) is deprecated (legacy support ends 2026-12-01), so configure the service in the dashboard or CLI instead, and record the settings in `docs/decisions.md`:

- Railway auto-detects the `Dockerfile`. Add a Postgres service and reference its `DATABASE_URL` in the app service's variables.
- Pre-deploy command `app db migrate`, with `SKIP_MIGRATIONS=true` on the service. Health check path `/up`. Draining seconds 30.
- A second service from the same repo with start command `app worker` (and `SKIP_MIGRATIONS=true`).

## Migrations against remote databases

| Situation | Command |
|-----------|---------|
| Normal deploy | Automatic (entrypoint or release/pre-deploy command) |
| Check status | Platform exec + `app db status` |
| From a laptop | `DATABASE_URL=<remote url> cargo run -- db migrate` (requires network access to the DB; confirm with the human first) |
| Roll back | `app db rollback` only if the migration's `down` loses no data. Otherwise write a forward fix |

## Zero-downtime rule: expand → migrate → contract

Old and new code run side by side during a deploy, so every migration must be compatible with the **previous** release:

1. **Expand:** add nullable columns or new tables. Deploy code that writes both shapes.
2. **Backfill** with an `app task`.
3. **Contract:** in a later deploy, add `NOT NULL`, drop the old columns, remove the dual-write code.

Never rename or drop a column that the currently deployed code reads, in a single deploy.

## Rollback

Redeploy the previous image (`fly deploy --image <prev>`, `kamal rollback <version>`, or Render/Railway's "redeploy" or "rollback" on an earlier deploy). Because of expand/contract, the previous image works with the current schema.

## Pre-deploy checklist (agent)

- [ ] Definition of Done passes (both DB backends).
- [ ] **CI is green on the exact commit you deploy** (`gh run list -L 1 --json headSha,conclusion`). A local DoD doesn't replace it: CI builds from a clean checkout and catches what your machine hides (e.g. a CSS file that only exists locally).
- [ ] `docker build` succeeds locally, and `/up` returns 200 from the container.
- [ ] Every new env var is in `.env.example`, `Config`, and the target's config, and you've told the human exactly which secrets to set.
- [ ] `APP_ENV=production` is set on every process the target runs (web and worker). An unset `APP_ENV` refuses to boot (`APP_ENV is not set`), so a missing setting shows up as a failed deploy, not as `/dev` open to the internet.
- [ ] `APP_TRUSTED_PROXY=true` is set (every platform here is a proxy), with `APP_TRUSTED_PROXY_HOPS` matching the number of proxies in front. Check: two failed logins from two different networks don't share one lockout.
- [ ] Exactly one process runs `app db migrate` per deploy (a release command, or one web container); every other process has `SKIP_MIGRATIONS=true`.
- [ ] Migrations follow expand/contract, and `down` was tested.
- [ ] SQLite prod: still a single machine, volume mounted, backups configured.
- [ ] You haven't run any deploy command without the human's go-ahead. Deploys are outward-facing.