Architecture

 Browser ──HTMX/SSE──▶ FastAPI web + /api/v1 ──▶ PostgreSQL (state · append-only events · jobs)
                              ▲                        │ LISTEN/NOTIFY
 kernel CLI ─────HTTPS────────┤                        ▼
 kernel-mcp (thin API client)─┤                  kernel-worker (durable jobs)
 kernel-daemon (host) ────────┘
   ├── Codex adapter   (codex exec --json)
   ├── Claude adapter  (claude -p --output-format stream-json)
   ├── Gemini adapter
   ├── generic CLI adapter (validated argv, no shell)
   └── external agent adapter (any orchestrator registers sessions over the API)

Layers (src/kernelos)

Package	Responsibility
`config`, `logging`	`KERNEL_*` settings; structlog with correlation ids and secret redaction
`db`	SQLAlchemy 2 async models, session/transaction boundary, packaged Alembic migrations
`domain`	pure state machines and value objects (Work, Approval, Handoff, Session, Events)
`services`	use cases; every material change appends an Event in the same transaction
`api/v1`, `web`	JSON API and server-rendered UI over the same services and permission matrix
`auth`	Argon2 passwords, server-side web sessions, CSRF, API tokens, RBAC
`events`, `jobs`	event log + LISTEN/NOTIFY → SSE; PostgreSQL job queue (`FOR UPDATE SKIP LOCKED`)
`adapters`, `daemon`	runtime adapters with capability detection; host daemon that runs them
`mcp`, `cli`	agent-facing MCP tools; `kernel` CLI
`storage`, `webhooks`, `analytics`, `security`	artifact backends, signed webhooks, aggregates/scheduler, privacy controls

Domain state machines (M1)

Work: backlog → ready → running → review → completed, with waiting_human (approval pending; resumes to the stored resume_state), blocked (open high/critical blocker; unblocks to ready/running), and cancelled. Only reopen leaves completed/cancelled. The transition table lives in kernelos/domain/work.py; services never assign state directly.

Blockers and state never diverge: while a high or critical blocker is open, start (from ready, review or blocked) and unblock are refused. reopen still lands in ready; start waits for the blockers.

Service rules (`kernelos/services/base.py`)

Tenant-scoped lookups; row locks FOR NO KEY UPDATE in the order project → work → session → other rows; authorization → state machine → reference validation → writes; exactly one event per changed aggregate, appended in the caller's transaction with pg_notify.
Three errors for "not now". A command the aggregate's state machine refuses raises IllegalTransition (state.illegal_transition, 409); for Work, details["allowed"] lists the commands that would succeed, the same set WorkService.available_commands returns, and a session refused because a blocking blocker is open is reported the same way (command start, with the blocker_id). A change that has no command at all (an update, a heartbeat, a new session, a new blocker, an assignment) on a closed aggregate raises Closed (resource.closed, 409). Closed aggregates: Work in completed/cancelled, an ended session, a resolved blocker, an archived project. Conflict (resource.conflict, 409) means a precondition on another row failed: a disabled runtime, a duplicate project key, an expired approval, a session already continuing a handoff, a superseded decision.
Terminal-work policy. Completed or cancelled Work still accepts the records that describe what happened: artifacts, decisions and outcomes (post-mortems). It refuses everything that would continue it: sessions, approvals, handoffs, blockers and updates. reopen first. Archiving a project is allowed with open work; an archived project takes no new work and no new sessions.
Causation. Every service call has a correlation_id; a service that emits an event and then calls another service spawns it with spawn(..., caused_by=<that event>), so the callee's events carry the cause in causation_id and the whole chain shares one correlation id. Chains today: session failed → blocker reported → work blocked; session created/started → work transitions (after superseding session_failed blockers); blocker reported/resolved → work blocked/unblocked; approval requested / decided / closed → work waits / resumes; decision recorded → decision superseded.
Inputs and pages. Every mutating method with more than one input takes one pydantic model (WorkCreate, WorkTransition, SessionEventRecord, ...) that the API reuses as its request body. Free-text fields are StrippedStr (kernelos/services/inputs.py): stripped before validation, so titles, names, actions, statements, reasons, purposes and event types (min_length=1) refuse blank values while notes may be empty. Updates go through changes_of: unset fields are unchanged, an explicit None is refused unless the field is nullable. Every list returns Page[T] (items, next_cursor) from kernelos/services/pagination.py: keyset pagination on (created_at desc, id desc), limit 1..200 (default 50; the API refuses anything above 200 with 422, the services clamp it for internal callers), opaque cursors. Events order by sequence instead. latest reads return T | None.

Extension points (M11)

The core domain model is deliberately generic; a vertical (security, procurement, legal, finance, …) is expressed through configuration on top of it, never a fork of the core. spec/09_use_cases/09_domain_expansion_rules.md names the mechanisms; three of them are implemented as generic interfaces, and no vertical pack ships — each is empty until an organization or project fills it.

Work templates (work_templates, project-scoped — WorkTemplateService, /api/v1/projects/{id}/work-templates). A template is a named skeleton for new work: a default goal, task type, constraints, acceptance criteria and labels. Creating work "from a template" (WorkCreate.template_id, and the web new-work form's Start from template picker) copies those fields as the starting point, overridable field by field; the template is never linked to the work it seeds. Read is viewer, authoring is member — the split projects use.
Per-project custom-field schemas (projects.custom_field_schema, a typed field list; services.custom_fields). A project declares extra fields — each name, a type from {string, number, bool, enum, date}, whether it is required, and an enum's options. From that schema one validator is built (validate_custom_fields), and it is the single enforcement point every door funnels through: the work create/update API, the web work form and the MCP tools all reach it inside WorkService, so the three cannot disagree. A payload that violates the schema is a 422 naming the field; a project with no schema keeps custom_fields free-form. The schema is edited through the project (member).
Artifact-type registry (artifact_types, per organization — ArtifactTypeService, /api/v1/organizations/{id}/artifact-types, and the Settings → Artifact types admin section). A type names a key (matched against an artifact's type), an optional display name and a metadata_schema — the same typed-field schema custom fields use. When an artifact is created with a registered type, its metadata is validated against that schema through the same validate_custom_fields (inside ArtifactService); an unregistered type stays free-form. The registry is administrative (read viewer, change admin).

The other expansion mechanisms (policy rules, integration adapters, saved views, optional domain packs) are either delivered elsewhere (saved views: saved_filters) or deferred (spec/10_implementation/18_non_goals_later.md).

Analytics and search (M12)

Search. One SearchService ranks user-visible content — work (key/title/goal), decisions, blockers, artifacts, session identifiers, and runtime/machine names — over PostgreSQL tsvector generated columns with GIN indexes; a pg_trgm similarity pass is the fallback when full-text finds nothing, so a typo or partial token still matches. Tenant and role confinement is applied before ranking (every per-resource query filters by the actor's org and the resource's read scope), never after, so results cannot leak across a boundary.

Analytics are precomputed and about things, never people. An idempotent analytics.aggregate_daily job rolls the event/session/outcome record into two daily tables — runtime_metrics_daily (org + day + runtime + task_type) and work_metrics_daily (org + day + task_type + project). AnalyticsService sums a half-open [from, to) range under the org predicate and withholds a slice's success rate (sample_too_small) below MIN_SAMPLE; range cost/duration are the median of the daily medians, named *_typical so the approximation is legible (a daily-median table cannot reconstruct a true range percentile). The two read APIs and the /o/{org}/analytics page key on runtimes, projects and task types only — tests/security/test_no_employee_ranking.py is the guardrail that fails the build if a column, a GROUP BY, or a route parameter ever names a person (no per-user leaderboards, by design).

Runtime scheduler (M13)

When someone starts a session, Kernel proposes the runtime that has actually worked for this kind of work — deterministically, with the arithmetic shown in one sentence, and never overriding a policy or an approval. There is no model and no hidden weight.

domain/scheduler.py is pure: score_runtimes(candidates, policy) takes a Candidate per runtime (its per-org effective enablement, whether a daemon can launch it, and its runtime_metrics_daily numbers for the work's task type) and the org's SchedulerPolicy, and returns a Recommendation. Hard filters run first — a runtime that is disabled, that no daemon can start (external runtimes are attached, never launched), that has fewer than min_sample completed runs, or (under the cost/speed modes) that sits below the success_floor is excluded and kept in the ranked list with a why-not reason. The survivors are scored in [0, 1]: best_success by success rate, lowest_cost/fastest by the cheaper/faster metric, and weighted by w_success·success + w_speed·speed + w_cost·cost (each metric normalized across the survivors, cost and duration inverted so higher is always better). Ties break by score descending then runtime name — the order is total and independent of input order, so the same inputs always yield the same pick. If nothing qualifies, the recommendation falls back to the policy's configured default runtime, or to none at all.

The recommendation is advisory: SchedulerService.recommend(work_id, machine_id) and GET /api/v1/work/{id}/runtime-recommendation compute it, the start-session dialog and kernel session start pre-select it but let the human override, and the chosen runtime and the recommended one are both recorded on the session row. It never satisfies an approval or unblocks work — the start still runs every policy and approval check.

Worked example. An org runs weighted with weights (0.5 success, 0.25 speed, 0.25 cost), success_floor 0.5 and min_sample 5. For a security_review item, Claude has 31 completed runs at 96% success, median 19m, ~$0.40 typical; a cheaper runtime has 40% success. The cheap one is excluded (below the floor), Claude's normalized success dominates the blend, and the dialog reads: "Claude recommended for security_review: 96% success over 31 completed tasks; median 19m; ~$0.40 typical cost; policy prioritizes a balance of success, speed and cost." When no runtime has yet reached min_sample for a task type, the dialog says so in one sentence rather than guessing.

Reliability and observability (M16)

Every deferred piece of work is a durable PostgreSQL job. The worker claims with FOR UPDATE SKIP LOCKED, records locked_at, runs, and releases; a job whose worker dies leaves a stale lock that requeue_stale returns to the queue after a timeout — proven by a test that SIGKILLs a real kernel-worker and watches another reclaim the orphan. Each job kind (webhook delivery, artifact processing, retention/cleanup, notifications, analytics) is idempotent under retry and dead-letters into JobState.dead after exhausting its attempts, so a doomed job stops rather than spins. Machine commands get the same treatment: one delivered but unacked past command_ack_timeout_seconds is redelivered on the next poll (KI-011).

An operator sees the system four ways. /health/live is a bare liveness check; /health/ready is 200 only when the database answers and its revision equals the code's head (503, with a reason, when it is down or mid-migration). /metrics is Prometheus text on its own registry — request counts/latency/5xx, kernel_jobs{state}, kernel_workers_online and heartbeat staleness, kernel_daemons{status}, and build_info — labelled only by method/status/state, never by path, id or tenant. Every log line carries the correlation id of its request or job, and one id can be followed from an HTTP request through the job it enqueued into the worker's logs. The admin Diagnostics page renders the same picture — daemon fleet, worker heartbeat age (each kernel-worker upserts a worker_heartbeats row; liveness is derived, never stored), job-queue and dead-letter counts, and the applied vs expected migration revision.

Invariants

Every tenant-owned row carries organization_id; services scope every query by the actor's org.
Work state changes happen only through domain.work.transition via a service, never by editing rows.
Events are append-only and written transactionally with the change that caused them; follow-on events name their cause in causation_id and share the request's correlation_id.
Timelines read events by sequence (indexes (organization_id|work_id|session_id, sequence)), never by created_at, which is the transaction start and ties within a transaction.
Unsupported runtime operations raise a typed capability error; nothing fakes success.

This document grows with each milestone; see spec/03_technical/ for the design rationale.