# Jobs Dashboard (Oban Web / Mission Control → `/admin/jobs` + `app jobs`)

> **TL;DR**
> - Every app has an Oban Web-style dashboard at `/admin/jobs` (overview, queues, job list with filters and bulk actions, job detail with errors, cron) and the same operations via `app jobs …`.
> - Both are thin layers over `kernel::jobs::admin`, so browser and terminal can't disagree.
> - Access: open in development; production needs an admin (or `APP_ADMIN_PASSWORD`, rate-limited), otherwise it isn't mounted.
> - Kernel-page rules apply: own layout, `no-store`, links in with `hx-boost="false"`, no `confirm()` dialogs.
>
> Read the rest when changing the dashboard, its access rules, or the `app jobs` CLI.

**Use the part.** The `jobs-dashboard` [parts](samples.md#the-samples) are this guide built, tested on SQLite and Postgres and security-reviewed, with more on top: live throughput and queue-wait charts, a now-running lane with progress, errors grouped by root cause, per-worker success rates and cron with run-now. When the product needs this, add the part rather than writing it from this guide: `cargo offrails add-part jobs-dashboard` (dependencies come first; [workflow](../workflow.md#when-the-product-grows-a-need-add-the-part)). Keep its `// SECURITY:` markers and the tests its CHECKS.md cites. This guide stays the spec: hand-write from it only when no part fits, or the human declined the part.

Every app gets a live jobs dashboard at `/admin/jobs` and the same operations from the CLI. Both are thin layers over one kernel module, `kernel::jobs::admin`, so the browser and the terminal can't disagree.

It's generated by `generators/new.md`, alongside the queue in `kernel/jobs/`, and it is never a stub. It looks and works the same in **every** `FRONTEND_MODE`: server-rendered Askama + vendored htmx + one plain stylesheet. It doesn't use React, Tailwind or the app's layout, so app UI changes never break it. The `/dev` tools and the dev mailbox share its chrome ([dev-tools](dev-tools.md)).

## Files

```
src/kernel/jobs/admin.rs        queries + actions (shared by web and CLI)
src/kernel/jobs/web.rs          axum Router for /admin/jobs, view structs, inline-SVG chart
src/kernel/ui.rs                Chrome (header, tabs, live toggle), page() with Cache-Control: no-store,
                                ago/until/duration/utc/truncate/thousands helpers, BOOTED_AT
templates/kernel/layout.html    shared kernel layout (jobs, /dev, mailbox)
templates/kernel/jobs/          index.html, _stats.html, _chart.html, _queues.html, _nodes.html,
                                list.html, _rows.html, show.html, cron.html, confirm.html
static/kernel/admin.css         plain CSS, light + dark (prefers-color-scheme), usable at 375px
static/kernel/admin.js          the kernel pages' only script: reload on a back/forward-cache restore
static/vendor/htmx-<ver>.min.js vendored in every mode (frontend-htmx.md shows how to fetch it)
```

`Assets` (`kernel/assets.rs`) has `admin_css` and `admin_js` fields (`/static/kernel/admin.css?v=<digest>`), so the kernel pages get cache-busted assets without Tailwind. The kernel layout's `<meta name="htmx-config">` is `{"allowEval":false,"includeIndicatorStyles":false,"historyCacheSize":0}` (the first two pair with the CSP: [frontend-htmx](frontend-htmx.md#gotchas-agents-get-wrong)).

## `kernel::jobs::admin` (the API)

```rust
pub async fn stats(db) -> Result<Stats, DbErr>;                       // GROUP BY queue, state (one query)
//   Stats { by_state: Vec<(JobState, u64)> /* all 7, zeros included */, by_queue: BTreeMap<String, HashMap<JobState, u64>> }, .count(state)
pub async fn queues(db, now) -> Result<Vec<QueueRow>, DbErr>;        // job_queues ∪ queues seen in jobs ∪ live nodes' queues
//   QueueRow { name, paused, limit /* sum over live nodes */, nodes, executing, available, scheduled, retryable, discarded }, .orphaned()
pub async fn nodes(db, now) -> Result<Vec<NodeRow>, DbErr>;          // job_nodes + executing count per node; alive = heartbeat < 60s
pub async fn throughput(db, now, minutes: i64) -> Result<Vec<Bucket>, DbErr>; // Bucket { minute, completed, failed }
pub async fn list(db, filter: &JobFilter, before_id: Option<i64>, limit: u64) -> Result<Vec<job::Model>, DbErr>;
pub async fn count_matching(db, filter) -> Result<u64, DbErr>;
pub async fn workers(db) -> Result<Vec<String>, DbErr>;             // distinct, for the filter select
pub async fn get(db, id) -> Result<Option<job::Model>, DbErr>;
pub async fn retry(db, ids: &[i64]) -> Result<u64, DbErr>;           // anything but executing/available → available now
pub async fn cancel(db, ids) -> Result<u64, DbErr>;                  // non-final → cancelled (executing: aborted by its node's heartbeat)
pub async fn delete(db, ids) -> Result<u64, DbErr>;                  // anything except executing
pub async fn run_now(db, ids) -> Result<u64, DbErr>;                 // scheduled/retryable → available, scheduled_at = now
pub async fn retry_matching(db, filter) -> Result<u64, DbErr>;      // bulk, capped at BULK_CAP = 10,000; refuses an empty filter
pub async fn cancel_matching(db, filter) -> Result<u64, DbErr>;
pub async fn pause(db, queue) -> Result<(), DbErr>;                  // upsert job_queues.paused = true
pub async fn resume(db, queue) -> Result<(), DbErr>;
pub async fn last_runs(db) -> Result<HashMap<String, DateTimeUtc>, DbErr>;   // cron key → latest run_at
pub async fn last_cron_job(db, key) -> Result<Option<i64>, DbErr>;

#[derive(Default)] pub struct JobFilter { pub state: Option<JobState>, pub queue: Option<String>, pub worker: Option<String>, pub tag: Option<String>, pub q: Option<String> }
```

- **retry** sets `state = 'available'`, `scheduled_at = now`, `max_attempts = max(max_attempts, attempt + 1)`, and clears `completed_at`/`discarded_at`/`cancelled_at`. Errors are kept, because the history matters. Each row update is filtered by its old state (no races with a running node).
- **cancel** pushes `"cancelled: by an operator"` to `errors` and frees an `Incomplete` unique key.
- **list** uses keyset pagination (`id < before_id ORDER BY id DESC LIMIT n`). Text search `q`: all digits matches `id`; otherwise a case-insensitive substring of `args` or `worker`, portable on both backends as `LOWER(CAST(col AS text)) LIKE '%q%' ESCAPE '\'` (sea-query `Func::lower(Func::cast_as(Expr::col(col), "text"))` + `LikeExpr::new(..).escape('\\')`). `tag` matches the quoted JSON element (`%"user:42"%`), so `user:4` doesn't match `user:42`. Search for **values**: Postgres `jsonb` re-renders args as `{"n": 1}`, so JSON fragments without spaces miss.
- **throughput:** select only `completed_at` (completed), `discarded_at` (discarded) and `attempted_at` (retryable) since the window start, and bucket per minute in Rust. Portable, fine up to ~50k jobs/hour; beyond that log a decision and switch to a backend `GROUP BY` in the kernel.
- Every action logs `info!(action, count, "jobs admin")`. The web actions also set a flash ("Retried 3 jobs.", or "Nothing to run: the job's state doesn't allow it." when 0 rows changed), shown once by the kernel layout.

## Pages

All under `/admin/jobs`, full pages normally. Live parts are re-fetched with `HX-Request` and a `part=` query.

| Route | Shows |
|-------|-------|
| `GET /admin/jobs` | **Overview.** State counters in operator order (executing · available · scheduled · retryable · discarded · cancelled · completed), each linking to the filtered list. Throughput chart for the last 60 minutes with totals. Queues table: name (links to the list), paused / "no nodes" badges, limit, nodes, counts, Pause/Resume button. Nodes table: node, queues + limits, running count, started, last heartbeat (dead nodes dimmed). Empty states explain how to start a worker. `?part=stats\|queues\|nodes\|chart` with `HX-Request` returns that fragment |
| `GET /admin/jobs/list?state=&queue=&worker=&tag=&q=&before=` | **Jobs.** State tabs with counts. Filter bar (GET, so URLs are shareable): queue and worker selects, tag and search inputs, Clear. Bulk bar: Retry / Cancel / Delete selected, plus "Retry all N matching" / "Cancel all N matching" when a filter is set. Rows: checkbox, id, state badge, worker + args (80 chars, monospace), queue, `attempt/max`, the time that matters for the state ("started 3s ago", "runs in 4m", "retries in 2m", "failed 2h ago"), tags. "Older →" keyset pager. `?part=rows` returns the `<tbody>` |
| `GET /admin/jobs/{id}` | **Job.** State badge, relative time and actions (Run now / Retry / Cancel / Delete, shown only when valid). Timeline: Inserted → Scheduled → Attempted → Completed/Discarded/Cancelled, plus queue time (attempted − scheduled) and run time. Queue, attempt, priority, node, unique key (shortened), tags. Args and meta pretty-printed. **Errors**: one block per attempt, newest first, with the full error chain. An unknown id (deleted, pruned, or reached with Back) → 303 to the list with the flash "Job N no longer exists (deleted or pruned)." |
| `GET /admin/jobs/cron` | **Cron.** `SCHEDULE` entries: key, expression, worker (links to the list), next run (relative + UTC), last run (links to the job it enqueued) |
| `POST /admin/jobs/{id}/{action}` | `action` ∈ `retry`, `cancel`, `run`, `delete`; anything else 404. Redirect (303) to `return_to` if it starts with `/admin/jobs` (no open redirects), else the job page (delete: the list). Responds with `X-Jobs-Changed: <n>` |
| `POST /admin/jobs/bulk` | Form: `action` + `ids` (checkboxes), or `all_matching=1` + the filter fields. "All matching" buttons carry their action in `formaction="/admin/jobs/bulk?action=retry"`, so the handler merges query and form pairs (`axum_extra::extract::Form<Vec<(String, String)>>`). **All matching first renders a confirmation page** ("Retry 1,234 jobs?") that posts back with `confirmed=1`. An empty filter changes nothing |
| `POST /admin/jobs/queues/{name}/{action}` | `pause` / `resume`; anything else 404 |

Route params are whole segments (`{id}/{action}`): axum 0.8 rejects `{id}.json`-style partial-segment params.

### Live updates

- Overview fragments carry `hx-get="/admin/jobs?part=stats" hx-trigger="every 2s" hx-swap="outerHTML"` (queues and nodes every 2s, the chart every 10s). The list's `<tbody id="rows">` polls every 3s, only on the first page, so paging stays stable. Row checkboxes have stable ids plus `hx-preserve`, so a selection survives refreshes.
- The header's **Live / Paused** toggle is a plain link that flips `live=0`. With live off, fragments render without `hx-*` attributes. No JS, no trigger filters (`[...]` needs `eval`, which the CSP forbids).
- Polling is cheap: the counts are one `GROUP BY` and lists are keyset queries.

### Back/Forward and caching

Every response of the dashboard router (pages, fragments, redirects) carries `Cache-Control: no-store`: `web::router()` ends with `.layer(kernel::ui::no_store())` (a `SetResponseHeaderLayer::overriding`). The kernel layout's htmx config adds `historyCacheSize: 0`, and `static/kernel/admin.js` (the kernel pages' only script) reloads a page the browser restores from its back/forward cache:

```js
window.addEventListener("pageshow", (event) => {
  if (event.persisted) window.location.reload();
});
```

So Back after deleting a job, or after a bulk retry, shows the current truth, never a stale snapshot. The browser smoke ([dev-tools](dev-tools.md#browser-smoke)) checks it.

### The chart

`Chart::new(&[Bucket])` builds an inline `<svg viewBox="0 0 600 120" preserveAspectRatio="none">` from numbers only, so it's the one sanctioned `|safe` outside rendered Markdown. Each minute is a `<g>` with a `<title>` ("12:04 · 38 completed · 2 failed"), a hover background rect, and stacked `bar-ok` / `bar-err` rects. Colors come from **classes** styled in `admin.css` (dark mode works); never `style=` (the CSP forbids inline styles).

### Look

`admin.css` defines tokens (`--bg --panel --fg --muted --border --hover --accent --ok --warn --err --info`) with a dark palette under `prefers-color-scheme: dark`. Sticky header with the app name, tabs (Overview · Jobs · Cron, plus Dev · Mailbox · Routes in development) and the live toggle. Stat cards with a state-colored edge, dense tables, monospace ids/args/errors, state badges colored by state, sticky filter bar, and tables that scroll inside their panel at 375px. All kernel responses carry `Cache-Control: no-store`.

## Mounting and access

In `routes.rs` (the layer, not each handler, enforces access, so no handler can forget it):

```rust
if config.is_development() || config.admin_password.is_some() {
    app = app.merge(jobs::web::router().route_layer(
        from_extractor_with_state::<AdminAccess, _>(state.clone()),
    ));
} else {
    tracing::warn!("jobs dashboard disabled: set APP_ADMIN_PASSWORD");
}
```

`AdminAccess` lives in `controllers/application.rs`, because only the app knows its users:

| Situation | Access rule |
|-----------|-------------|
| `APP_ENV=development` | Open |
| Roles exist (`ACCOUNTS` + `ROLES: admin+member`) | `RequireAdmin`. Logged-out users are redirected to login (the authentication generator rewrites `AdminAccess`) |
| No roles, `APP_ADMIN_PASSWORD` set | HTTP Basic (any user name), constant-time comparison; 401 with `WWW-Authenticate: Basic realm="admin"`. Rate-limit **failed** passwords (`kernel::rate_limit`, 10 per 3 min per IP, key `admin-basic-auth:<ip>`): a request without credentials (the browser's first) doesn't count; once exhausted, answer 429 even for the right password until the window passes. Needs `RateLimiter::exhausted` (peek without counting) next to `check` |
| Production, neither | **Not mounted** (404). Boot logs the warning above |

- The CSRF layer applies to every POST (a cross-site POST to `/admin/jobs/...` is 403); same-origin forms pass naturally.
- The tabs to `/dev` pages appear only in development.
- **Links from app pages into `/admin/jobs` carry `hx-boost="false"`** (the app layout boosts; the dashboard has its own `<head>`). See [dev-tools](dev-tools.md#rules).
- Args and errors can contain personal data. Don't put secrets in args ([jobs](jobs.md) rule 1).
- Tests (`tests/app/kernel_web_test.rs` + `security_test.rs`): overview, fragments, list filters/pagination/polling, job page, single and bulk actions (with the confirmation step, the open-redirect check and the flashes), a deleted job's page redirecting to the list, `no-store` on every response, queue pause/resume, cron page, CSRF; production without an admin → 404; with `APP_ADMIN_PASSWORD` → 401 without/with a wrong password, 200 with it, and no `/dev` links.

## CLI: `app jobs`

The same `admin` functions, for terminals, scripts and agents. Output is an aligned table, or JSON with `--json`.

```
app jobs stats [--json]                               # counts per state, queues (limit, nodes, paused / "no nodes"), nodes
app jobs list [--state discarded] [--queue q] [--worker w] [--tag t] [--q text] [--limit 50] [--json]
app jobs show <id> [--json]                           # full row incl. every error, newest first
app jobs retry <id>… | --all [filters] [--yes]        # --all needs at least one filter, prints the count, asks y/N
app jobs cancel <id>… | --all [filters] [--yes]
app jobs delete <id>…
app jobs run <id>…                                    # run now
app jobs pause <queue> / app jobs resume <queue>
app jobs prune [--days N]                             # run the Pruner once (default APP_JOBS_PRUNE_DAYS)
```

`job::Model` and `JobState` derive `serde::Serialize` (`rename_all = "snake_case"`) for `--json`. Against production: `fly ssh console -C "app jobs list --state discarded"` (see [deployment](deployment.md)). **Agents:** when a human reports "emails aren't arriving" or "X never happened", run `app jobs list --state discarded --json` and `app jobs show <id>` before reading code.