Jobs Dashboard (Oban Web / Mission Control → /admin/jobs + app jobs) #

TL;DR

• Every app has an Oban Web-style dashboard at /admin/jobs (overview, queues, job list with filters and bulk actions, job detail with errors, cron) and the same operations via app jobs ….
• Both are thin layers over kernel::jobs::admin, so browser and terminal can't disagree.
• Access: open in development; production needs an admin (or APP_ADMIN_PASSWORD, rate-limited), otherwise it isn't mounted.
• Kernel-page rules apply: own layout, no-store, links in with hx-boost="false", no confirm() dialogs.

Read the rest when changing the dashboard, its access rules, or the app jobs CLI.

Use the part. The jobs-dashboard parts are this guide built, tested on SQLite and Postgres and security-reviewed, with more on top: live throughput and queue-wait charts, a now-running lane with progress, errors grouped by root cause, per-worker success rates and cron with run-now. When the product needs this, add the part rather than writing it from this guide: cargo offrails add-part jobs-dashboard (dependencies come first; workflow). Keep its // SECURITY: markers and the tests its CHECKS.md cites. This guide stays the spec: hand-write from it only when no part fits, or the human declined the part.

Every app gets a live jobs dashboard at /admin/jobs and the same operations from the CLI. Both are thin layers over one kernel module, kernel::jobs::admin, so the browser and the terminal can't disagree.

It's generated by generators/new.md, alongside the queue in kernel/jobs/, and it is never a stub. It looks and works the same in every FRONTEND_MODE: server-rendered Askama + vendored htmx + one plain stylesheet. It doesn't use React, Tailwind or the app's layout, so app UI changes never break it. The /dev tools and the dev mailbox share its chrome (dev-tools).

Files #

src/kernel/jobs/admin.rs        queries + actions (shared by web and CLI)
src/kernel/jobs/web.rs          axum Router for /admin/jobs, view structs, inline-SVG chart
src/kernel/ui.rs                Chrome (header, tabs, live toggle), page() with Cache-Control: no-store,
                                ago/until/duration/utc/truncate/thousands helpers, BOOTED_AT
templates/kernel/layout.html    shared kernel layout (jobs, /dev, mailbox)
templates/kernel/jobs/          index.html, _stats.html, _chart.html, _queues.html, _nodes.html,
                                list.html, _rows.html, show.html, cron.html, confirm.html
static/kernel/admin.css         plain CSS, light + dark (prefers-color-scheme), usable at 375px
static/kernel/admin.js          the kernel pages' only script: reload on a back/forward-cache restore
static/vendor/htmx-<ver>.min.js vendored in every mode (frontend-htmx.md shows how to fetch it)

Assets (kernel/assets.rs) has admin_css and admin_js fields (/static/kernel/admin.css?v=<digest>), so the kernel pages get cache-busted assets without Tailwind. The kernel layout's <meta name="htmx-config"> is {"allowEval":false,"includeIndicatorStyles":false,"historyCacheSize":0} (the first two pair with the CSP: frontend-htmx).

kernel::jobs::admin (the API) #

pub async fn stats(db) -> Result<Stats, DbErr>;                       // GROUP BY queue, state (one query)
//   Stats { by_state: Vec<(JobState, u64)> /* all 7, zeros included */, by_queue: BTreeMap<String, HashMap<JobState, u64>> }, .count(state)
pub async fn queues(db, now) -> Result<Vec<QueueRow>, DbErr>;        // job_queues ∪ queues seen in jobs ∪ live nodes' queues
//   QueueRow { name, paused, limit /* sum over live nodes */, nodes, executing, available, scheduled, retryable, discarded }, .orphaned()
pub async fn nodes(db, now) -> Result<Vec<NodeRow>, DbErr>;          // job_nodes + executing count per node; alive = heartbeat < 60s
pub async fn throughput(db, now, minutes: i64) -> Result<Vec<Bucket>, DbErr>; // Bucket { minute, completed, failed }
pub async fn list(db, filter: &JobFilter, before_id: Option<i64>, limit: u64) -> Result<Vec<job::Model>, DbErr>;
pub async fn count_matching(db, filter) -> Result<u64, DbErr>;
pub async fn workers(db) -> Result<Vec<String>, DbErr>;             // distinct, for the filter select
pub async fn get(db, id) -> Result<Option<job::Model>, DbErr>;
pub async fn retry(db, ids: &[i64]) -> Result<u64, DbErr>;           // anything but executing/available → available now
pub async fn cancel(db, ids) -> Result<u64, DbErr>;                  // non-final → cancelled (executing: aborted by its node's heartbeat)
pub async fn delete(db, ids) -> Result<u64, DbErr>;                  // anything except executing
pub async fn run_now(db, ids) -> Result<u64, DbErr>;                 // scheduled/retryable → available, scheduled_at = now
pub async fn retry_matching(db, filter) -> Result<u64, DbErr>;      // bulk, capped at BULK_CAP = 10,000; refuses an empty filter
pub async fn cancel_matching(db, filter) -> Result<u64, DbErr>;
pub async fn pause(db, queue) -> Result<(), DbErr>;                  // upsert job_queues.paused = true
pub async fn resume(db, queue) -> Result<(), DbErr>;
pub async fn last_runs(db) -> Result<HashMap<String, DateTimeUtc>, DbErr>;   // cron key → latest run_at
pub async fn last_cron_job(db, key) -> Result<Option<i64>, DbErr>;

#[derive(Default)] pub struct JobFilter { pub state: Option<JobState>, pub queue: Option<String>, pub worker: Option<String>, pub tag: Option<String>, pub q: Option<String> }

• retry sets state = 'available', scheduled_at = now, max_attempts = max(max_attempts, attempt + 1), and clears completed_at/discarded_at/cancelled_at. Errors are kept, because the history matters. Each row update is filtered by its old state (no races with a running node).
• cancel pushes "cancelled: by an operator" to errors and frees an Incomplete unique key.
• list uses keyset pagination (id < before_id ORDER BY id DESC LIMIT n). Text search q: all digits matches id; otherwise a case-insensitive substring of args or worker, portable on both backends as LOWER(CAST(col AS text)) LIKE '%q%' ESCAPE '\' (sea-query Func::lower(Func::cast_as(Expr::col(col), "text")) + LikeExpr::new(..).escape('\\')). tag matches the quoted JSON element (%"user:42"%), so user:4 doesn't match user:42. Search for values: Postgres jsonb re-renders args as {"n": 1}, so JSON fragments without spaces miss.
• throughput: select only completed_at (completed), discarded_at (discarded) and attempted_at (retryable) since the window start, and bucket per minute in Rust. Portable, fine up to ~50k jobs/hour; beyond that log a decision and switch to a backend GROUP BY in the kernel.
• Every action logs info!(action, count, "jobs admin"). The web actions also set a flash ("Retried 3 jobs.", or "Nothing to run: the job's state doesn't allow it." when 0 rows changed), shown once by the kernel layout.

Pages #

All under /admin/jobs, full pages normally. Live parts are re-fetched with HX-Request and a part= query.

Route params are whole segments ({id}/{action}): axum 0.8 rejects {id}.json-style partial-segment params.

Live updates #

• Overview fragments carry hx-get="/admin/jobs?part=stats" hx-trigger="every 2s" hx-swap="outerHTML" (queues and nodes every 2s, the chart every 10s). The list's <tbody id="rows"> polls every 3s, only on the first page, so paging stays stable. Row checkboxes have stable ids plus hx-preserve, so a selection survives refreshes.
• The header's Live / Paused toggle is a plain link that flips live=0. With live off, fragments render without hx-* attributes. No JS, no trigger filters ([...] needs eval, which the CSP forbids).
• Polling is cheap: the counts are one GROUP BY and lists are keyset queries.

Back/Forward and caching #

Every response of the dashboard router (pages, fragments, redirects) carries Cache-Control: no-store: web::router() ends with .layer(kernel::ui::no_store()) (a SetResponseHeaderLayer::overriding). The kernel layout's htmx config adds historyCacheSize: 0, and static/kernel/admin.js (the kernel pages' only script) reloads a page the browser restores from its back/forward cache:

window.addEventListener("pageshow", (event) => {
  if (event.persisted) window.location.reload();
});

So Back after deleting a job, or after a bulk retry, shows the current truth, never a stale snapshot. The browser smoke (dev-tools) checks it.

The chart #

Chart::new(&[Bucket]) builds an inline <svg viewBox="0 0 600 120" preserveAspectRatio="none"> from numbers only, so it's the one sanctioned |safe outside rendered Markdown. Each minute is a <g> with a <title> ("12:04 · 38 completed · 2 failed"), a hover background rect, and stacked bar-ok / bar-err rects. Colors come from classes styled in admin.css (dark mode works); never style= (the CSP forbids inline styles).

Look #

admin.css defines tokens (--bg --panel --fg --muted --border --hover --accent --ok --warn --err --info) with a dark palette under prefers-color-scheme: dark. Sticky header with the app name, tabs (Overview · Jobs · Cron, plus Dev · Mailbox · Routes in development) and the live toggle. Stat cards with a state-colored edge, dense tables, monospace ids/args/errors, state badges colored by state, sticky filter bar, and tables that scroll inside their panel at 375px. All kernel responses carry Cache-Control: no-store.

Mounting and access #

In routes.rs (the layer, not each handler, enforces access, so no handler can forget it):

if config.is_development() || config.admin_password.is_some() {
    app = app.merge(jobs::web::router().route_layer(
        from_extractor_with_state::<AdminAccess, _>(state.clone()),
    ));
} else {
    tracing::warn!("jobs dashboard disabled: set APP_ADMIN_PASSWORD");
}

AdminAccess lives in controllers/application.rs, because only the app knows its users:

• The CSRF layer applies to every POST (a cross-site POST to /admin/jobs/... is 403); same-origin forms pass naturally.
• The tabs to /dev pages appear only in development.
• Links from app pages into /admin/jobs carry hx-boost="false" (the app layout boosts; the dashboard has its own <head>). See dev-tools.
• Args and errors can contain personal data. Don't put secrets in args (jobs rule 1).
• Tests (tests/app/kernel_web_test.rs + security_test.rs): overview, fragments, list filters/pagination/polling, job page, single and bulk actions (with the confirmation step, the open-redirect check and the flashes), a deleted job's page redirecting to the list, no-store on every response, queue pause/resume, cron page, CSRF; production without an admin → 404; with APP_ADMIN_PASSWORD → 401 without/with a wrong password, 200 with it, and no /dev links.

CLI: app jobs #

The same admin functions, for terminals, scripts and agents. Output is an aligned table, or JSON with --json.

app jobs stats [--json]                               # counts per state, queues (limit, nodes, paused / "no nodes"), nodes
app jobs list [--state discarded] [--queue q] [--worker w] [--tag t] [--q text] [--limit 50] [--json]
app jobs show <id> [--json]                           # full row incl. every error, newest first
app jobs retry <id>… | --all [filters] [--yes]        # --all needs at least one filter, prints the count, asks y/N
app jobs cancel <id>… | --all [filters] [--yes]
app jobs delete <id>…
app jobs run <id>…                                    # run now
app jobs pause <queue> / app jobs resume <queue>
app jobs prune [--days N]                             # run the Pruner once (default APP_JOBS_PRUNE_DAYS)

job::Model and JobState derive serde::Serialize (rename_all = "snake_case") for --json. Against production: fly ssh console -C "app jobs list --state discarded" (see deployment). Agents: when a human reports "emails aren't arriving" or "X never happened", run app jobs list --state discarded --json and app jobs show <id> before reading code.