data-olympus

Serving model

This document describes the normative serving model for data-olympus. It summarizes SPEC.md section 8; for the authoritative text, see that section.

Single replica, streamable HTTP

A write-enabled data-olympus server runs as a single replica and exposes MCP over streamable HTTP.

The write path is single-writer by design:

• Per-path advisory locks prevent two concurrent write operations from racing on the same concept file.

• Per-session git worktrees isolate in-flight proposed edits from the live index until they are committed.

• A durable push queue serializes outbound git pushes so no write is lost if the remote is briefly unavailable.

A single shared HTTP surface, rather than N independent stdio processes, gives every agent one synchronized conversation with the server. When multiple agents each run their own stdio MCP process against the same git working tree, they race each other's worktrees and lock state. Streamable HTTP eliminates that race.

The long-lived SSE stream (proxy and session tuning)

Each MCP client holds a long-lived GET /mcp Server-Sent-Events stream to receive server-to-client messages. Two things must be configured so that stream does not churn:

• Ingress/proxy read timeout. A proxy that closes the idle stream (nginx's default is 60s) forces the client to reconnect on that cadence. Each reconnect creates a new transport server-side and, in a persistent client like opencode serve, leaks event listeners. Set a long read timeout and disable response buffering on the /mcp path (see deploy/k8s/ingress.yaml for the nginx annotations).

• Session reaping. The MCP SDK only expires a transport a client leaves behind after 30 minutes without a request. The server runs its own reaper (session_metrics): the activity middleware keeps a session non-idle for as long as its SSE stream is open, and KB_SESSION_IDLE_TIMEOUT_SEC (default 300) terminates a session only after its stream has actually closed. KB_SESSION_TOUCH_INTERVAL_SEC (default 30, clamped to a third of the idle window) sets how often an in-flight stream is re-stamped. kb_health / /api/v1/health and a periodic log surface the live session count.

Core configuration reference

A few server-wide settings, beyond the feature-specific env vars documented in their own sections below (search ranking, synonyms, embeddings, co-occurrence, trigram, auth, audit rotation):

• KB_MAIN_PATH: the knowledge-base corpus to index and serve (default /kb-main). KB_INDEX_PATH: where the SQLite index is written (default /index/kb.db). Both treat a blank value as unset. A variable that is set but empty, or whitespace-only, -- what an unsubstituted compose/Helm/CI variable produces -- gets the documented default, not the server's working directory. A non-blank value is used verbatim and never rewritten. If KB_MAIN_PATH does not resolve to a directory, startup fails with a message naming the setting, the path it resolved to, and whether that path came from the environment or the built-in default. An existing but empty corpus is not an error: the server starts, builds the schema, and reports degraded until content arrives.

• KB_GIT_BRANCH: the branch of the knowledge-base repository the server syncs (default main). It is the branch fetched and fast-forwarded by the pull loop, the branch a session worktree is rebased onto before a write, the branch writes are pushed to, and the branch a worktree's commits must be reachable from before that worktree can be garbage collected. Set it to master, or to whatever your repository actually calls its trunk, and check that same branch out in KB_MAIN_PATH: the fast-forward merges by ancestry rather than by name, so a checkout left on an unrelated branch is the case that fails and leaves health degraded. A blank value means unset and gets the default. Accepted names are ASCII letters, digits, ., _, - and /, subject to git's structural rules (no component beginning with . or ending .lock, no .. or //, no trailing / or .); HEAD and a fully qualified ref such as refs/heads/master are refused, as is anything else outside that set. A rejected value fails startup with the setting and the reason named, rather than being passed to git.

• KB_HTTP_PORT: TCP port the MCP HTTP server binds (default 8080).

• KB_CONFIDENCE_THRESHOLD: the auto-commit confidence cutoff, in [0, 1] (default 0.85). A proposal at or above it from a principal holding auto_commit is committed; below it (or from a principal without that capability) it is parked as pending. An out-of-range value fails startup loudly.

• KB_PENDING_TIMEOUT_SEC: age after which an unresolved pending proposal is auto-expired by the pending GC loop (default 86400, i.e. 24h). Each expiry emits an audit event.

• KB_WRITING_RULES_MODE: how the writing-rule gate treats the lines a write adds (default warn, which commits and reports the findings). enforce rejects such a write as rejected_writing_rule, and off skips the check. Any other value fails startup with the accepted values named. See "Write serialization and integrity gates" below.

• KB_WRITING_RULES_EXCLUDE_PATHS: comma-separated fnmatch globs of target paths the writing-rule gate skips, matched like KB_WRITE_BLOCK_PATHS (default empty).

• KB_SECRET_SCAN_EXTRA_PATTERNS: additional regexes the secret-scanning gate (issue #71) checks alongside its built-in pattern set (see "Write serialization and integrity gates" above). Write it either as a JSON array of strings (["ACME_[A-Z0-9]{20,40}", "INTERNAL-[0-9]{6}"]) or comma-separated. In the comma form a comma inside {...} or [...] does not separate patterns, so ACME_[A-Z0-9]{20,40} stays one pattern, and a backslash-escaped comma (\,) never separates. A value that parses as a JSON array is read as JSON. Each entry is scanned as its own named pattern (custom_1, custom_2, ...). The setting is parsed once at startup: an entry that is not a non-empty string, is not a valid regex, or has the classic nested-quantifier ReDoS shape fails startup with an error naming the setting and the entry number (not the pattern text). Every accepted pattern runs with a hard 1-second match timeout. Empty by default (no extra patterns).

• KB_DISABLE_VERSION_CHECK: 1, true, yes, or on disables the public PyPI/GitHub version check completely. Use this for air-gapped deployments. 0, false, no, or off keeps it enabled, as does leaving the setting unset or empty (the default). Values are case-insensitive and surrounding whitespace is ignored; any other value stops startup with an error naming the setting, so a typo cannot leave the outbound check running.

• KB_VERSION_CHECK_INTERVAL_SEC: how often the background task refreshes the cached latest-version result (default 86400, i.e. 24h). The health request path never performs the outbound lookup.

• KB_GOVERNED_LANE_PROTECTION: governed-lane write protection (issue #112), default on; set off to restore the exact pre-#112 behavior (see "Governed-lane write protection" below).

• KB_GATE_CLEARANCE (default intent): what a fresh explicit consult clears at the enforcement gate. intent clears only the governed actions the consult's intent covers; pair restores the earlier rule, under which any fresh explicit consult for the session and workspace clears every governed action. Any other value fails startup naming the setting. See docs/enforcement.md for the coverage rule and the family mapping.

• KB_TOOL_DISCOVERY_MODE: MCP catalog exposure mode. search is the default; all restores the complete native catalog in tools/list. Any other value fails startup.

MCP tool discovery

The default KB_TOOL_DISCOVERY_MODE=search keeps the initial MCP catalog small without removing any capability. tools/list exposes these native tools:

• kb_consult

• kb_search

• kb_get

• kb_health

• kb_gate_check

• kb_record_event

• kb_session_recap

It also exposes the synthetic tool_search and call_tool tools. Use tool_search to find a native tool omitted from the initial list, then invoke it with call_tool. A client that already knows the native name may still call it directly. Search mode filters discovery only; it does not unregister tools.

Visibility is not an authorization boundary. A hidden write tool retains its annotations and passes through the same principal capability middleware whether it is called directly or through call_tool. For clients that require the complete catalog during initialization, set KB_TOOL_DISCOVERY_MODE=all and restart the server.

Read only replicas expose the subset of the default catalog that is actually registered in read only mode, plus tool_search and call_tool. Discovery cannot make a write tool available when the replica did not register it.

Read-only mirrors may scale horizontally

Set KB_READ_ONLY=true to run an instance as a read-only replica. In this mode the server registers only the read tools (kb_search, kb_get, kb_list, kb_outline, kb_curate, kb_health) and the read REST routes; the write and enforcement-write tools and routes (propose, resolve, bootstrap, consult, gate, record-event, and their observability mirrors) are not registered at all and return 404. The write pipeline (worktrees, push queue, pending) is never initialised, so no replica is a git writer.

Crucially, the git_pull_loop still runs, so each replica keeps KB_REMOTE_URL set and refreshes its own index snapshot from the same git remote as the single writer. Run as many replicas as you need for read throughput; the single write-enabled instance remains the only owner of the git remote.

KB_READ_ONLY accepts 1, true, yes, or on to enable read-only mode, and 0, false, no, or off for the default read-write behaviour, which also applies when the setting is unset or empty. Values are case-insensitive and surrounding whitespace is ignored. Any other value, such as ture or enabled, stops startup with an error naming the setting, so a mistyped value cannot start a replica with the write pipeline running.

See deploy/k8s/read-replica/ for a ready-to-apply Deployment (not the StatefulSet writer) that runs N read replicas with per-pod ephemeral clone + index scratch. Apply the base stack first, then the overlay:

kubectl apply -k deploy/k8s              # namespace + secret + writer StatefulSet
kubectl apply -k deploy/k8s/read-replica # N read replicas + read Service
kubectl -n data-olympus scale deployment/data-olympus-mcp-read --replicas=5

Git pull loop

On startup, and at the interval set by KB_SYNC_INTERVAL_SEC (default 60s), the server calls git pull on KB_MAIN_PATH, against the branch named by KB_GIT_BRANCH (default main). If KB_REMOTE_URL is empty, the pull loop runs but exits cleanly with no action. The health endpoint reports degraded: true only when the index has not been rebuilt within KB_STALENESS_DEGRADED_SEC (default 600s).

For a local read-only demo with no remote, set KB_REMOTE_URL="". The server stays healthy as long as the index builds successfully on startup.

Health, readiness, and liveness (three distinct signals)

The server exposes three endpoints that answer three different questions. They are deliberately split so a data-freshness problem does not eject a pod that can still serve reads:

Key point: /readyz is independent of data staleness. A git-remote outage makes the KB stale (so /api/v1/health returns 503 with degraded: true), but the last-good index still answers reads perfectly, so /readyz stays 200 and the pod stays in the Service. The Kubernetes readiness probe targets /readyz (see deploy/k8s/statefulset.yaml); if it targeted /api/v1/health a stale single-replica pod would be ejected from the Service and a "reads are a bit old" condition would become a hard 503 with zero endpoints.

/api/v1/health keeps its 503-on-degraded contract because the bin/kb CLI --no-stale flag relies on it (exit 2 when the endpoint reports degraded: true, whether over HTTP 200 or 503). Alert on /api/v1/health degraded (and on the last_git_fetch_status / staleness_seconds fields it carries) rather than wiring it to a probe. The liveness probe is a tcpSocket check by default; the HTTP /livez route is offered for operators who prefer an explicit route. See docs/operations.md for the full alerting and recovery model.

The health payload also carries malformed_frontmatter: the count of docs whose front-matter was present but malformed at the last index build. A non-zero value means a doc silently lost its governance metadata (type/status/tier), so it will not be governed or filtered correctly. It is a warning signal and does not flip degraded (that would 503 every read for an authoring mistake); alert on malformed_frontmatter > 0 separately.

When the background version check sees a newer published package than the running server, health also carries latest_version and sets update_available: true. latest_version is omitted from compact health output until a check has found a published version; verbose health shows it as null before then. This signal is informational only and does not affect degraded, readyz, or write behavior.

Write serialization and integrity gates

Every write (kb_propose_memory, kb_propose_edit, kb_resolve_pending, and onboarding bootstrap, which commits its whole file bundle through the same serialized/validated multi-file path) goes through one serialized critical section so concurrent writes cannot corrupt each other:

• A process-wide write serializer wraps the write → git add → commit → enqueue sequence, so one thread's commit can never sweep another thread's staged file. Write volume is rate-limited, so a single global mutex is cheap.

• A per-path advisory lock, shared between the auto-commit path and the pending queue, prevents two writes to the same file from racing and prevents an auto-commit from landing on a path with a pending proposal in flight (whose later approval would clobber it). A write blocked by the lock returns rejected_path_lock_busy. An orphaned lock (a crash between lock acquisition and the pending entry write) is reclaimed by the pending GC loop.

• A content-validation gate runs on the postimage before commit. It rejects rejected_invalid_document when the frontmatter is malformed YAML, a type/status/tier enum value is out of vocabulary, or the id is a forged/duplicate of one already used by a different path (which would break every subsequent index rebuild: one bad write, persistent degraded state). The response reason carries the machine-readable errors.

• Supersession targets (issue #259). On commit, a supersedes or superseded_by value that the write newly introduces must name a document present in the commit being made (any status; a pending proposal does not count, and an id the same write removes does not either). Otherwise the write is refused as rejected_invalid_document with unresolved_supersedes_target or unresolved_superseded_by_target at the start of the reason, or malformed_supersedes / malformed_superseded_by for a wrong shape. A derived_from target follows the same rule (unresolved_derived_from_target, malformed_derived_from); a source that has retired never causes a refusal. Target strings are compared exactly as written, so " TARGET " does not resolve to TARGET, and a blank or whitespace-only target is refused as malformed (lint ignores such a value rather than reporting it). A target left unchanged from the committed version of the same file, and a malformed value left exactly as it was (type-exact: 1 changed to true is a change), keep passing, so a legacy document can still be edited, and removing either field is always accepted. A credential-shaped target is redacted in the reason. Validation reads the actual local commit the write lands on, with three git processes for the whole write; if that commit cannot be read the write is refused as unresolved_target_unverifiable. Writes without an enforceable base marker, and bootstrap, may proceed after a failed base refresh, so a target deleted upstream in that window is caught by data-olympus lint, not by this check. A pending proposal parked by an earlier release that introduces such a target is refused at approval and stays pending with its lock. Two pending proposals that name each other (A.supersedes: B, B.superseded_by: A) cannot both be approved as written: approve A with edited_text that omits the edge to B, approve B, then propose the edge on A again.

• Bootstrap scope. Onboarding bootstrap creates new files under the workspace or component it onboards, and nothing else. Every target path must lie under projects/<workspace>/, or under projects/<workspace>/components/<component>/ when a component is given, whatever the onboarding state; a bundle with any other path is refused as rejected_path_not_indexable_or_blocked, listing those paths, before any pending entry or commit is made. A bootstrap of a partial workspace is still narrowed to the missing canonical files first. At commit time a target that already exists in the session worktree refuses the whole bundle as rejected_already_onboarded, so a bootstrap never replaces a file. To change an existing document, use kb_propose_edit. The commit subject names every path the bootstrap writes.

• Bootstrap rejection order (issue #259). A bundle with an empty files list is refused first, as rejected_empty_bundle (HTTP 400), before any claim, rate-limit slot, commit or pending entry (issue #311). After the earlier path, size and rate-limit refusals (which are unchanged and carry no reason), a bundle's commit-time checks run in a fixed order and refuse at the first failure: every file's path and postimage are secret-scanned first (so no later diagnostic can echo credential-shaped content), then duplicate ids inside the bundle, then each file's containment, existing-target and content validation in file order. Supersession targets resolve against the committed tree with the whole bundle applied, so a successor and its predecessor created in the same bundle are accepted. A secret-scan, duplicate-id or content-validation refusal carries reason in its response, redacted wholesale if it would contain credential-shaped content. A containment refusal (rejected_symlink_escape) carries no reason.

• Compare-and-swap (optimistic concurrency). When a caller supplies a base marker (base_commit naming a specific commit, base_blob_sha, or target_file_hash) on kb_propose_edit (or it is carried on the pending entry into kb_resolve_pending), the server refreshes the session worktree's base onto origin/main and compares the marker against the current target content. A mismatch returns rejected_stale_base and nothing is committed. If the base cannot be refreshed at all (the remote is unreachable, or the rebase conflicts) while an enforceable marker was supplied, the write is also rejected rejected_stale_base rather than committed against a possibly-stale base; the marker cannot be verified, and the push-path rebase recovery is not an equivalent safety net (a compatible rebase would still publish the stale write). base_commit must be HEAD or a commit id of 7 to 64 lowercase hex characters (abbreviated or full, SHA-1 or SHA-256); any other value, such as a branch name or a revision expression, is refused as rejected_invalid_base when the proposal is made, with a reason that does not repeat the value. A bare base_commit of HEAD is advisory (no per-file expectation), and when no marker is supplied the pre-0.3.0 behavior is preserved (a refresh failure is non-fatal; the push path's non-FF recovery publishes the commit). Both hash markers are validated when the proposal is made. base_blob_sha must be the file's git blob id (40 lowercase hex characters) and target_file_hash the sha256 of its current bytes (64 lowercase hex characters); either may be omitted. Any other form is refused as rejected_invalid_base, naming the field. A proposal parked by an earlier release with a malformed marker cannot pass the check, so every approval of it returns rejected_stale_base even when the file has not changed. Recover by rejecting it and proposing the same content again with correct markers.

• Ordering of side effects: the commit message and all gates are evaluated BEFORE the file is written and git add-ed, and on any failure after the add the worktree is hard-reset, so a rejected write never leaves a staged leftover for the session's next commit to sweep in.

• A secret-scanning gate (issue #71) runs on the postimage BEFORE the content-validation gate, on every commit path (auto-commit propose, resolve approve, including a resolved edited_text, and onboarding bootstrap). A resolved edited_text is additionally scanned before the entry is claimed, so a flagged edit is refused with the entry left untouched. The claim records a sha256 digest of the approved content, never the content itself, so an edit that is refused, or committed under the explicit override, does not remain on the state volume. A claim written by 0.8.0 still holds the text until the first reconciliation pass after it is older than KB_PENDING_CLAIM_TTL_SEC replaces it with the digest. It runs first so that whenever the gate matches, a postimage that is both malformed AND carries a credential-shaped value is rejected via the redacted rejected_secret_detected path rather than rejected_invalid_document (which echoes the offending value verbatim in its message). Detection is shape-based, so this ordering guarantee is only as good as the pattern set: a credential the patterns do not recognise is not redacted by it either. The gate checks a built-in pattern set (PEM private-key blocks; GitHub ghp_/gho_/ ghs_/ghr_/ghu_/github_pat_ tokens; AWS AKIA.../ASIA... access key ids, the second being the temporary form issued by STS; Slack xox[bpars]- tokens; LLM provider API keys as one llm_provider_api_key class (Anthropic sk-ant-, OpenAI sk-proj-/sk-svcacct-/legacy sk-, OpenRouter sk-or-v1-, Hugging Face hf_, Groq gsk_, xAI xai-, and OpenAI-compatible gateways minting the same shape); Google AIza... API keys, which authenticate Gemini and the other Google APIs that accept API keys, and so are their own class rather than an LLM-specific one; generic password=/passwd=/secret= assignments, including env-style prefixed keys like DB_PASSWORD=, with a non-placeholder value; and scheme://user:pass@host connection strings) plus any operator-supplied KB_SECRET_SCAN_EXTRA_PATTERNS. Every class is a shape, not a validation: the gate recognises credential-shaped text and never contacts a provider, so it can both miss a real credential whose format it does not model and flag an identifier that merely looks like one. Treat it as a backstop against accidental paste, not as proof a postimage is clean. A match on an auto-commit or bootstrap path rejects the write rejected_secret_detected before anything is written to disk. Only the pattern name and an approximate line number are ever surfaced in the response, the audit event, or a log line, never the matched value. A low-confidence proposal containing a secret still enters pending (removing it would defeat the operator-override workflow below), but the scan runs at propose time too: the response never echoes the raw text when flagged (only the pattern name), the pending entry is tagged secret_scan_flagged/matching_pattern (names only) so kb pending surfaces the warning, and a flagged memory proposal's filename falls back to a neutral slug instead of embedding the flagged text. The gate also covers the fields AROUND the postimage: a credential-shaped target_path (filename) is rejected on edit, bootstrap, and resolve without ever echoing the path back (it would otherwise land in responses, audit events, commit subjects, and the git tree); a flagged edit reason is replaced with a redacted note before it reaches pending meta, push metadata, or audit events; and a flagged memory tag is stored redacted in pending meta. kb_resolve_pending accepts an operator-only override_secret_scan boolean (kb resolve --override-secret-scan on the CLI) to consciously commit a false positive anyway (recorded in the audit event); no auto-commit or bootstrap path exposes this override, so an agent can never self-authorize past a flagged write. An extra pattern with the classic nested-quantifier ReDoS shape is rejected at load time alongside an invalid regex, and every accepted custom pattern executes through the regex engine with a hard 1-second match timeout, so a catastrophic pattern the load-time check misses is bounded at scan time (logged and skipped) instead of hanging the single-writer write path.

• A writing-rule gate (issue #283) runs after the secret scan and before content validation, on the same commit paths. It applies the project's own raw writing rules (no em-dash, no en-dash used as one, no agent authorship credit; the rules CI enforces through scripts/prose_lint.py) to the lines the write ADDS, so an edit never fails on text it did not write. A line counts as added when it occurs more often in the postimage than in the target's current content on the refreshed base (curly quotes folded first), so moving an existing line is not adding it, while a new offending line is always caught. A line ending in <!-- prose-lint: allow --> is exempt and keeps the marker in the committed text. Rules are checked on raw lines, including inside code fences. In enforce mode a finding rejects the write rejected_writing_rule (HTTP 422 on propose and resolve) with writing_rule_findings entries of the form line <n>: <rule>: <excerpt>; in the default warn mode the write commits, the findings are returned in the same field, the audit event records the rule names and line numbers (never the excerpts), and a WARNING with the finding count is logged (never the path or the text). Because it runs after the secret scan, a postimage carrying both a credential and a finding is still reported as the redacted rejected_secret_detected. A failure of the check itself, including an existing target that is not valid UTF-8, rejects in enforce and is reported as a warning in warn; it is never silently skipped. A rejected resolve puts the pending entry back, as other gate rejections do. The machine-rendered maintenance ledger is exempt (it quotes corpus ids and paths no agent wrote there), recorded in its audit event as skipped:machine_rendered; that exemption is reachable from no MCP or REST call. Configure it with KB_WRITING_RULES_MODE (default warn) and KB_WRITING_RULES_EXCLUDE_PATHS (see "Core configuration reference").

Governed-lane write protection (KB_GOVERNED_LANE_PROTECTION, issue #112)

KB_GOVERNED_LANE_PROTECTION (default on; case-insensitive off/0/ false/no disables it and restores the exact pre-#112 behavior) gates a demotion-only feature on top of the write path above: "agents can propose, only humans can promote." No write is ever REJECTED by this feature; an affected write is DEMOTED to the pending queue instead of auto-committing. See SECURITY.md's "Governed-lane write protection" section for the full threat-model rationale; this section covers the mechanics and the demotion semantics an operator needs at serving time.

• Status clamp. Checked on the auto-commit path of kb_propose_memory / kb_propose_edit, and on every kb_bootstrap_project file. If the postimage's status field is being set/changed to a value in the in-force class (active/accepted/approved -- format.validate.IN_FORCE_STATUSES, single-sourced, never a local copy), the write is demoted with demotion_reason: "status_promotion" instead of committed. Unconditional on the PRIOR status: a brand-new file, a promotion from a non-in-force status, and an unreviewed re-claim of an already in-force status all clamp the same way.

• Governed-target edit demotion. Checked only on kb_propose_edit. If the edit's target_path currently resolves to an IN-FORCE document in the live index (the full composed predicate: status class AND validity window AND not-inbox AND not-graph-excluded -- the same predicate Index.search(in_force=True) and kb_get's computed in_force field use), the write is demoted with demotion_reason: "governed_target", REGARDLESS of confidence. A target that is expired or has been superseded out of force is not currently governing anything and so is not protected by this rule. The lookup FAILS CLOSED: when the target's in-force state cannot be verified (no index wired, or an index read fails), the edit is demoted with the distinct demotion_reason: "governed_target_unverified" rather than auto-committed on the strength of a broken lookup -- an unhealthy index can never be used to slip an edit past this rule. A target with no entry in a HEALTHY index is definitive at the tool layer (a brand-new file cannot already be in force), and an authoritative IN-WORKTREE BACKSTOP closes the residual index-lag window: inside the serialized commit section, after every hard gate passes and after the worktree base is refreshed onto origin/main, the target's CURRENT bytes on that refreshed base are re-judged (status + validity window from its own frontmatter, is_inbox from the path). The backstop deliberately consults NOTHING from the index -- in particular not graph exclusion, whose stale edges could otherwise REMOVE protection the base's own bytes assert (a target still carrying an in-force status under a live supersedes edge is over-demoted for operator review rather than auto-committed; a target properly flipped to superseded is not in force by its own bytes and stays unprotected). A doc pushed as in-force but not yet re-indexed is therefore still demoted (demotion_reason: "governed_target") -- the decision is made against the same content the commit would replace, so index staleness cannot bypass the rule in either direction.

• Injection-pattern annotation. Every postimage evaluated by the two rules above is also scanned for agent-directed injection patterns ("ignore previous instructions", "disregard ... policy", an imperative paired with an http(s) URL, a long base64-looking run, "do not tell the operator"). A match sets injection_suspect: true and populates injection_patterns (a list of "pattern_name:line" strings, mirroring the issue #71 secret-scan redaction discipline -- never the matched text) on the pending entry. This is ADVISORY ONLY: it never blocks a write and never causes a demotion by itself; it exists so the human reviewer sees the signal alongside the demoted content.

• Ordering. The issue #71 secret-scanning gate and the content- validation / duplicate-id gate are evaluated FIRST for any write that would otherwise auto-commit. A postimage that would be rejected outright by either of those gates IS rejected (rejected_secret_detected / rejected_invalid_document), never silently demoted -- a demotion cannot be used as a side door around a hard rejection. The exceptions are the checks that can only be decided against the committed tree (missing_status and the issue #259 supersession codes): the index-only prediction ignores them, so such a write stays demoted and the commit path decides at approval.

• Unaffected paths. The maintenance-ledger system write (maintenance.maybe_update_ledger, which stamps its own doc status: active) commits through the shared multi-file write primitive DIRECTLY, bypassing the tool-function layer these checks live in, so it is never gated by this feature. The operator resolve path (kb_resolve_pending / kb resolve) is likewise never gated: an operator-confirmed resolve IS the promotion this feature exists to require.

• Per-write feedback. A demoted response carries the same pending_id shape as a plain low-confidence park, plus demotion_reason and an operator_prompt that explicitly instructs the calling model to inform the operator that the write awaits review rather than proceeding silently.

• Per-session feedback. See docs/operations.md for kb_session_recap / kb session-recap / GET /api/v1/session-recap, the kb_consult pending_actions surfacing, and the bin/kb-session-recap-hook SessionEnd/Stop hook wiring.

Push queue and write-path visibility

Committed writes are published to origin/main through a durable push queue: a commit that returns "committed" has a queue entry on disk, and a background loop drains the queue with retry. Health surfaces the queue state so a stuck write path is diagnosable:

• push_queue_size is the live count of queued (not-yet-pushed) entries.

• pending_count is the live count of pending (awaiting-approval) entries.

• push_queue_frozen is the count of entries that hit the retry cap and were frozen.

• path_locks_held is how many paths are locked, and path_locks says WHICH, each with target_path, owner_kind, pending_id, acquired_at and age_seconds. A leaked lock blocks every write to one path, so the count on its own was not diagnosable without reading the state volume by hand. A lock file that exists but cannot be read, or is not a JSON object, is listed with owner_kind: "unreadable" and null fields: it still holds a path, and it is never allowed to fail health, readiness or the REST reads that build on it. The orphan collector and the stale auto-commit reclaim leave such a file in place, since its owner cannot be established; remove it by hand once no write to that path is in flight.

Both counts are computed at health-report time from the queues themselves, so they cannot drift.

An interrupted resolve, and how it recovers

Approving a pending entry claims it, holds its path lock, commits, and then releases both. If the process dies in between, the entry is left in a claimed state on the state volume, still holding its path lock. Two things follow, and both are handled explicitly:

• The entry is visible. kb_list_pending reports every entry with a state of pending, claimed or uncertain. A claimed entry is not awaiting a decision, so it does not count towards pending_count, but it is never hidden: an empty queue must mean an empty queue, not a decision stuck halfway.

• The outcome is reconciled from evidence, never from age. A background pass looks at claims older than KB_PENDING_CLAIM_TTL_SEC (default 900) and decides from what the write actually recorded:

  Evidence	Outcome
  A recorded commit sha	Committed. The entry is consumed and never offered again.
  A recorded failure	Not committed. The entry returns to pending for the operator to re-resolve, keeping its path lock.
  Nothing recorded	The commit is searched for by its KB-Pending-Id trailer on the recorded session ref. Found means committed. Otherwise uncertain.

An uncertain entry keeps its path lock deliberately and is re-checked on every later pass, so a transient unreadable repository resolves itself. A negative search is not treated as proof that nothing committed: a rebase can drop a commit that upstream already acquired as an equivalent patch, and a squash can rewrite its trailer away, so absence of the commit is not absence of the write. Restoring on absence would offer an already-applied decision for approval a second time.

For the same reason, operations that rewrite session history reconcile outstanding claims first and then defer while a claim on that branch has begun its write: the base rebase, the non-fast-forward push recovery that calls it, and the worktree GC that removes a session worktree and deletes its branch. The GC teardown holds the write serializer across its whole sequence, so it either completes or leaves the worktree and the branch both intact. A deferred push is not a publication failure and does not consume its retry budget. These guards fail closed: an unreadable claim record, or a branch that cannot be identified, defers the rewrite rather than permitting it. Each guard holds the write serializer across its own check AND the history mutation it authorises, so the guard protects the history actually being rewritten rather than an earlier snapshot of it. The guard reconciles at the same claim TTL as the periodic sweep rather than from zero age, so a claim whose own resolver is still finishing is left alone and the rewrite simply defers. A successful resolve also consumes its own claim inside the same serializer acquisition that made the commit, immediately after recording the outcome, so no reconciliation can ever observe a committed-but-unconsumed claim and finish it on the owner's behalf.

If an entry stays uncertain, check whether the change is present on origin/main and then resolve it by hand on the state volume: delete /state/pending/<pending_id>.claimed and /state/pending/locks/<sha256-of-target-path>.lock to release the path. Confirm the content first; that pair is the only remaining record of the decision.

Reading back a parked proposal

A session that parks a proposal can read its own draft back:

GET /api/v1/pending/<pending_id>

and the kb_get_pending MCP tool takes the same single pending_id. It returns the postimage with in_force: false and a note saying so, because the content is a pending proposal: it governs nothing, and it stays out of kb_consult and every in_force retrieval exactly as before.

Recovery identifies a claim's commit by a KB-Pending-Id trailer, and it asks git interpret-trailers to read that trailer rather than parsing the message itself. That parse runs with global, system and environment-injected git configuration disabled and repository discovery ceilinged, because git's own trailer.<alias>.key configuration can RENAME an ordinary key into the claim key and turn an unrelated commit into evidence. A duplicated key is treated as ambiguous and yields nothing. Commit text is agent-controlled, so a parser that disagreed with git in the accepting direction would let an ordinary write close somebody's decision; delegating removes that whole class rather than chasing git's message-boundary rules one at a time. A commit body that does not end in a newline is treated as having no trailers, conservatively, because the CLI would otherwise answer for a normalised body rather than the one recovery holds.

Ownership is the authenticated principal recorded when the proposal was made:

• a principal holding resolve may read any entry, since it can already see the content by approving it;

• otherwise the caller's principal must be the one that made the proposal;

• an entry with no recorded proposer predates this field, so its ownership cannot be established and it is resolver-only.

source_session deliberately plays no part. The listing publishes it for every entry to any authenticated caller, so scoping a content read on it would be no boundary at all: a reader could list the queue, copy somebody else's id and session, and ask for their draft. With no principals configured every caller can already resolve any entry, so the readback is open there too and matches the posture of every other write surface.

Principal names are ownership identities, so they must be unique. The registry refuses a configuration with two credentials of the same name, including two unnamed ones, because they would otherwise share ownership of each other's drafts.

What this boundary does not cover. Only the POSTIMAGE is owner-scoped. GET /api/v1/pending and kb_list_pending remain visible to every authenticated principal and carry each entry's target_path, reason, evidence, capture and identities. Those are content-derived: a target path can name what a draft is about, and an evidence string can quote it. The secret scanner redacts credential-shaped values, not ordinary confidential text. So the pending queue's metadata is shared within a deployment, deliberately and as it always has been, and this change does not narrow it. If that is not the boundary you want, do not put confidential material in a proposal's path or evidence.

A postimage the secret scanner flagged is withheld from everyone except a resolve principal. The proposer already held that content, but handing it back would turn the queue into a place to retrieve a credential from. The response still names the matched pattern, so the caller learns why.

Capture provenance on proposed memories

A memory distilled from a passive capture, such as a hook event or an external session transcript, can say so. kb_propose_memory and POST /api/v1/propose/memory accept an optional capture object that labels the proposal as evidence-derived, binds it to the exact source event and proposal bytes, and names the transformation that produced it. The captured raw event stays outside data-olympus: only identifiers, hashes and enumerated values enter the store.

The envelope is provenance, not authority. It adds no status, it does not force review, and it does not change what in_force returns. A capture-derived memory is rendered with status proposed under the memory inbox like any other memory, so it is never in force and never returned by kb_consult, whether it parks or auto-commits at high confidence.

Every value is an ASCII token matched in full, so no value can hold whitespace, a newline, a quote or prose. capture_session is the session that produced the captured event; the top-level source_session is still the session making the proposal. An explicit null on an optional field reads as absent, and a null capture means no envelope.

The server always computes derived_memory_hash as sha256: of the UTF-8 bytes of text exactly as submitted, with no normalization. A caller that supplies the field must supply that same value, which binds the proposal bytes in transit. Matching hashes show that neither side changed; they do not prove that the memory was derived from the event, which is why transformation is required: the reviewer checks the derivation claim against the named step.

Validation runs before the path, blocklist and rate-limit checks, so a refused envelope costs no rate-limit token. A credential-shaped value anywhere in the envelope, including under an unknown key, is refused as rejected_secret_detected with the pattern name only, and nothing is parked or committed. Every other problem (a non-object, {}, an unknown key, a missing required field, a value of the wrong type, format or enumeration, or a derived_memory_hash that does not match text) is refused as rejected_invalid_capture. Neither reason ever repeats a submitted value. Over REST, rejected_invalid_capture is HTTP 400 and rejected_secret_detected is 422. POST /api/v1/propose/edit refuses a capture key with rejected_invalid_capture and HTTP 400, because an edit labels no new memory.

An accepted envelope, with the computed derived_memory_hash, is written:

• into the memory's frontmatter as a nested capture mapping, after evidence, through the same YAML serializer as every other server-rendered key, so it survives an auto-commit and counts against KB_MAX_TEXT_BYTES;

• into the pending entry, where kb_list_pending, GET /api/v1/pending, kb_get_pending and GET /api/v1/pending/<pending_id> return it as capture (null when the proposal had none, or when a stored value is malformed);

• onto the committed and pending_confirmation audit events for the proposal. A refused envelope's audit event carries the status and reason only.

kb_session_recap and GET /api/v1/session-recap count the session's committed or parked memory proposals that carried an envelope as capture_derived.

An operator edit keeps the label. On resolve, edited_text replaces the whole proposed document, so the server re-attaches the stored envelope to what it commits, on approve and on edit. An edit that supplies its own frontmatter keeps it, with capture set to the stored envelope; an edit that supplies a body only, which is what kb resolve hands the operator, keeps the original proposal's frontmatter. The re-attached derived_memory_hash is still the one computed over the submitted text, so when it no longer matches the committed body, that mismatch is the record that a human changed it. The size cap and the secret scan on edited_text judge the bytes after re-attachment.

Passive hook ingestion. A capture hook records a tool event, hashes it, discards the raw payload, and proposes the distilled memory at low confidence so a person reviews it:

{
  "text": "Integration tests in this repository need the staging database.",
  "tags": ["testing"],
  "source_session": "distiller-7",
  "agent_identity": "automem",
  "confidence": 0.4,
  "capture": {
    "capture_source": "claude_code.hook",
    "capture_event_id": "01J9ZK3Q7R8S9T0V1W2X3Y4Z5A",
    "source_event_hash": "sha256:3f9c0e5d1b7a2c4e6f8091a2b3c4d5e6f708192a3b4c5d6e7f8091a2b3c4d5e6",
    "transformation": "automem.distill/1.4.2",
    "raw_retention": "discarded",
    "classification": "fact"
  }
}

External-session transcript ingestion. An importer reads a transcript from another tool, keeps a redacted copy in its own store, and names the session the transcript came from:

{
  "text": "The team chose weekly release trains over continuous deployment.",
  "tags": ["release"],
  "source_session": "importer-run-12",
  "agent_identity": "transcript-importer",
  "confidence": 0.5,
  "capture": {
    "capture_source": "codex.transcript",
    "capture_event_id": "urn:transcript:2026-10-01:message:42",
    "capture_session": "codex-session-8c1f",
    "source_event_hash": "sha256:9a8b7c6d5e4f30211203f4e5d6c7b8a99a8b7c6d5e4f30211203f4e5d6c7b8a9",
    "transformation": "transcript-import/0.3.0",
    "raw_retention": "redacted",
    "classification": "decision"
  }
}

Non-fast-forward push recovery

When a second overlapping session moves origin/main between a session's commit and its push, the push is rejected non-fast-forward. The push loop classifies this distinctly from a network failure: it fetches origin/main, rebases the session branch onto it, and retries the push once. Most non-FF cases (the two sessions touched different files) publish cleanly on the retry.

If the rebase conflicts (the two sessions edited the same lines incompatibly), the commit cannot be auto-published. Instead of retrying forever, the push loop demotes the commit to a pending entry for operator resolution: the postimage is re-proposed as a pending edit (visible via kb_list_pending), the queue entry is removed, and a push_conflict_demoted audit event is recorded. The operator resolves the pending entry (approving re-applies it on the current base, subject to the CAS/validation gates) to publish the change.

Pure contention (the retry push is itself non-FF because origin/main moved again mid-rebase) is retried in-line for a bounded number of passes; if it never wins the race it is demoted the same way rather than counted toward the freeze cap, so persistent contention never becomes a silently-frozen queue item.

Startup recovery

A crash in the narrow window between git commit and the push-queue enqueue would otherwise orphan a commit on a session worktree branch that nothing pushes. On boot the server scans every session worktree and re-enqueues any commit reachable from its HEAD but not from origin/main. Entries already in the queue are left untouched, so recovery never double-enqueues. The recovery result is logged (startup push recovery re-enqueued N orphaned commit(s) or no orphaned commits found).

Frozen entries and how to unfreeze

After max_attempts consecutive push failures a queue entry is frozen: the retry loop stops retrying it (retrying a permanently failing push every interval only spams the remote and hides the problem). The freeze is logged once at WARN with the sha, worktree, and last error, and the entry is counted in push_queue_frozen. A nonzero push_queue_frozen means writes are stuck and need operator attention. Non-fast-forward pushes against a moved origin/main are now recovered automatically (see Non-fast-forward push recovery above), so a frozen entry now signals a persistent failure the rebase path could not handle, typically an authentication, network, or remote-side rejection.

To unfreeze, an operator resolves the underlying push failure, then clears the entry file directly on the push-queue volume (KB_PUSH_QUEUE_ROOT, default /state/push-queue):

• Requeue (retry from scratch): delete the "frozen": true and reset "attempts" to 0 in the entry's JSON file (or delete and let startup recovery re-enqueue it), then the retry loop picks it up again on its next pass.

• Drop (abandon the write): delete the entry's <sha>.json file. The commit stays on its session worktree branch but is never published.

There is no unfreeze API in this release; the file-level procedure above is the supported path.

Per-session worktree GC

Each writing session gets its own git worktree (and a kb-session/<safe_id> branch) so in-flight edits are isolated. A background GC task, running every 5 minutes, removes worktrees idle beyond KB_WORKTREE_IDLE_SEC (default 3600s) so KB checkouts do not accumulate one-per-session forever.

GC is conservative and coupled to the push queue:

• A worktree with commits not yet reachable from origin/main is deferred (never removed) until the push queue drains those commits; the next GC pass cleans it up once they land upstream.

• When a worktree is removed, its kb-session/<safe_id> branch is deleted in the same step. This matters because a returning session recreates its worktree with git worktree add -b kb-session/<safe_id>, which would fail if the branch still existed. Deleting the branch keeps a GC'd session able to write again.

Streamable-http session lifecycle and reaping

Each session-less POST /mcp handshake makes the underlying MCP SDK create a transport, register it in an in-memory session table, and start a task that blocks waiting for further messages on that session. The SDK removes a session on an explicit client DELETE, on the session task crashing, on server shutdown, and, since MCP SDK 1.30, once the session has had no request in flight for 30 minutes. A request being served or an open GET stream holds the session. FastMCP builds the SDK's session manager without overriding that window, so it applies to this server as a backstop.

Consequence: a client that handshakes and drops the connection without sending DELETE (the common cause of repeated "Created new transport with session ID" log lines) leaves its transport resident for at least half an hour, and a burst of such handshakes can fill the session table. The SDK also caps the table at 10000 concurrent sessions and answers a request that would open another one with HTTP 503 until room frees up.

data-olympus bounds this more tightly in two ways:

• Observability: health reports live_sessions, the current live transport count (or null before the HTTP app has started serving). The server also logs the live count each reaper pass. A live_sessions value that only ever climbs is the signal of a leak.

• Bound: a background reaper terminates sessions idle beyond KB_SESSION_IDLE_TIMEOUT_SEC (default 300, five minutes). It scans every KB_SESSION_REAP_INTERVAL_SEC (default 60). Set KB_SESSION_IDLE_TIMEOUT_SEC=0 to disable this reaper and keep observability only. Termination uses the SDK's own terminate() path, so the session leaves the table and a client that reconnects simply gets a fresh session.

The SDK's 30 minute expiry stays active whatever these settings are, so disabling the reaper, or setting a window longer than 1800 seconds, leaves the SDK expiring sessions that have had nothing in flight for 30 minutes. Both mechanisms treat a session with an open GET stream as live, so neither ends a connected client's session while the other keeps it. With the default five minute window and one minute scan the reaper acts long before the SDK would, and an abandoned session stays in the table for at most six minutes, so reaching the 10000 session cap takes clients abandoning about 28 sessions a second, sustained.

The reaper works on elapsed time since a session's last activity stamp, not on whether its connection is open. The clock advances on every request carrying that session's mcp-session-id header, so a client whose calls arrive more often than KB_SESSION_IDLE_TIMEOUT_SEC is never reaped. A client that polls less often than that is reaped between calls and gets a fresh session on its next handshake, so pick a keep-alive interval shorter than the idle window.

A session whose long-lived GET SSE stream is open does not need one. The activity middleware runs a keep-alive task for the duration of that request and re-stamps the session every KB_SESSION_TOUCH_INTERVAL_SEC, and the server clamps that interval to at most a third of the idle window, so the stamps are scheduled at least three times per window however short the window is. No accepted combination of the two settings schedules them further apart than the window, so a quiet but connected client is not reaped.

What the reaper is there to clear is the session nothing is stamping any more: the abandoned handshake, the client that disconnected without sending DELETE. That is the leak it bounds, and it is why a five minute default window does not disturb a connected client.

Search ranking

kb_search orders hits by BM25 relevance and then applies a status-aware rerank on top of that score, enabled by default. The rerank exists so a retired document does not outrank the live one that replaced it: for a query that matches both, the in-force doc is promoted and the retired one demoted.

The built-in status-to-weight map is:

• In-force (boosted): active, accepted, approved.

• Retired (penalized): superseded, deprecated, rejected.

• Not yet in force (penalized): draft, proposed.

A negative weight boosts (moves a hit earlier), a positive weight penalizes (moves it later). Status matching is case-insensitive, so Active in frontmatter ranks like active. Any status not in the map (including an empty status) is neutral and is never dropped from results.

Because this reorders results, a query that previously surfaced a superseded doc first may now surface the active one first. This is the intended behavior change.

Override the map at deploy time, with no code change:

• KB_STATUS_WEIGHTS: a JSON object of {status: weight} that replaces the built-in map (it is not merged). Keys are statuses (matched case-insensitively), values are numeric deltas: negative boosts an in-force status, positive penalizes a retired one. A malformed value fails startup loudly rather than silently shipping default ranking. Unset (the default) uses the built-in map above.

  Example: KB_STATUS_WEIGHTS='{"active": -1.0, "approved": -1.0, "superseded": 2.0, "draft": 1.0}'.

in_force: hard in-force filter

kb_search accepts an in_force: bool = False parameter (MCP tool and GET /api/v1/search?in_force=true). When set, results are HARD-filtered BEFORE ranking to docs that are currently in force: the in-force status class (active, accepted, approved) AND the validity window (validity.valid_from not in the future, validity.valid_until not in the past; both boundary days inclusive). So a superseded, deprecated, draft, proposed, expired, or upcoming doc is excluded entirely rather than merely soft-downranked by the status rerank above.

Use it when you want only guidance that currently applies and a demoted-but- present retired doc is not acceptable. It differs from the status rerank: the rerank always keeps every hit and only reorders; in_force drops out-of-force hits. The in-force predicate is defined once (format.validate.IN_FORCE_STATUSES for the status class, format.validate.is_in_force for the combined status-AND-window predicate) and shared by the rerank boosts and this filter.

in_force composes with the single-status status filter: passing both requires a doc match status AND be in-force (so status=superseded with in_force=true yields nothing). The filter also applies to the dense (embedding) candidate source, so a hybrid deployment never leaks an out-of-force doc through the semantic path.

The predicate also applies a memory-inbox floor (issue #109): a document under the memory-inbox prefix (KB_MEMORY_INBOX_PREFIX, default memory/inbox/) is never in force regardless of claimed status, so a legacy inbox file or forged frontmatter on an agent-written memory cannot satisfy in_force=true no matter what status it declares. This is a plain is_inbox column derived once at index build time, not a per-query prefix scan. Together with the supersession-graph exclusion below, the full in-force predicate is: status class AND validity window AND not-inbox AND not-graph-excluded.

kb_consult passes in_force=true on its internal retrieval unconditionally (not a caller-facing parameter): the enforcement surface must never present an unreviewed, retired, expired/upcoming, memory-inbox, or graph-excluded document as a governing rule for a code/architectural decision. See docs/enforcement.md.

Computed in_force on served responses. A verbose (verbose=true) kb_get response and each verbose kb_search hit carry a computed in_force: bool: the same single-sourced predicate evaluated against the doc's actual status/validity/inbox-membership/graph-exclusion, independent of whether the query itself passed in_force=true. This lets a caller retrieve a doc via a default (unfiltered) kb_get/kb_search and still tell whether it currently governs, without a second in_force=true round trip. It is a serving-layer derivation only, never written to frontmatter (see SPEC.md's "runtime envelope" note). Compact responses emit it deviation-only: in_force: false appears ONLY when the doc is not in force, and an in-force doc's compact shape is byte-for-byte unchanged. The deviation emission is required because the compact status/freshness fields key off the RAW frontmatter status: a memory-inbox doc with a forged status: active would otherwise render as an ordinary current rule with no signal that the in-force floor disqualified it.

Supersession-graph exclusion (issue #110 slice 2)

in_force=true ALSO excludes any document that is the TARGET of a supersedes edge whose SOURCE document is itself in force (the full status-class AND validity-window AND not-memory-inbox predicate above, not merely status: superseded). This is the in-force-source guard: a draft, expired, already-retired, or memory-inbox document can never retire another document just by naming it in supersedes. It closes the "forgotten status flip" gap -- A (active) supersedes B, but B's own status was never updated to superseded -- B is excluded from in_force=true results even though its own status class would otherwise qualify it. Like the upcoming half of the validity window, graph exclusion is scoped to in_force=true only: a plain (default) search still returns the excluded document.

A mutually-supersessive in-force cycle (already a kb lint ERROR at parse time) is NOT special-cased at retrieval time: every member independently satisfies the exclusion rule (each is the target of an in-force source's supersedes edge), so in_force=true excludes ALL of them. A dangling edge (source or target id with no corresponding document) excludes nothing; the exclusion query joins edges to docs on both ends.

kb_consult runs its search with in_force=true (see "Enforcement endpoints" below), so it never returns a graph-excluded, not-yet-in-force, or retired document -- the same guarantee the unconditional expired-doc exclusion already gave it, made explicit for the graph rule since that rule is scoped to in_force=true rather than applying unconditionally.

A graph_excluded_docs health counter (in kb_health / /api/v1/health, alongside malformed_frontmatter and malformed_validity) reports the count of documents currently excluded by this rule. It is evaluated LIVE at health-read time against the current date, from the same SQL definition the retrieval-time filter uses (format.validate.graph_excluded_ids_sql), so the counter can never drift from the filter it reports on -- including across a date boundary where an in-force source's validity window opens or closes between index rebuilds (retrieval evaluates the window per query, so a counter frozen at build time would keep reporting a stale value). The short health cache (health_ttl_sec, default 5s) bounds its staleness. It is a WARNING signal only and does not flip degraded.

Retirement is explainable

kb_get by id always resolves regardless of in-force or graph-exclusion status (same as it already ignores expiry) and returns:

• superseded_by: the sorted, deduped UNION of the document's own frontmatter superseded_by claim and any reverse supersedes edge naming it. This ONE consistent computed shape covers both the honest self-declared case and the forgotten-status-flip case above, where only the superseding document's own supersedes list names this one.

• contradicts: the document's own frontmatter list (unchanged from the frontmatter, never affects filtering or ranking anywhere in the retrieval path).

• contradicted_by: the computed reverse -- every other document whose contradicts names this one.

All three are omitted from the compact kb_get response when empty (same deviation-only pattern as freshness). Compact kb_search hits carry a deviation-only superseded_by (the same computed union above), omitted when the document is not superseded; it is computed and attached to EVERY hit regardless of in_force, so a plain search result still explains why a hit is historically superseded. kb_search hits do NOT carry contradicts / contradicted_by; those are kb_get-only.

Derivation: derived_from is surfaced, never acted on

A document may declare derived_from (SPEC.md section 4.2, format 0.5): the documents its guidance was drawn from. A document has left force (is retired) when it is graph-excluded, its status is deprecated, superseded or rejected, or it is expired. rejected counts deliberately: deriving from an explicitly rejected document is itself worth surfacing. draft, proposed, upcoming and memory-inbox documents are not retired, because they never governed. kb_get returns three lists of ids, sorted, deduped and dangling-safe:

• derived_from: the document's own derived_from targets that exist.

• derived_from_retired: the subset of derived_from that is retired. Non-empty only while this document is itself in force.

• dependents_to_review: the documents in force (the full computed in_force predicate) whose derived_from names this one. Non-empty only while this document is retired.

Verbose kb_get always carries all three; compact kb_get omits each when empty, so a document without derivation relationships keeps its exact compact shape. MCP kb_get and REST GET /api/v1/get/{id} share one implementation. Nothing is filtered or demoted: derived_from never changes any document's in_force, kb_search or kb_consult results, ranking, or the graph_excluded_docs counter, and kb_search hits do not carry these lists. A dependent of a retired source can therefore still be returned by kb_consult with no cue; kb lint (a warning on both ends) and kb_get are where a person sees it. data-olympus lint reports the same condition as a warning, never an error, because a source can retire by expiry.

Validity: expired docs leave default results

Independent of in_force, a doc past its validity.valid_until date is excluded from EVERY default kb_search result (SPEC.md section 4.2): an expired doc has no named successor to outrank it, so left visible it could be the top hit and would govern. Three related kb_search parameters (MCP tool and GET /api/v1/search):

• include_expired: bool = false restores expired docs to the result set; each carries freshness: "expired" in its hit.

• validity_state is an audit facet: "expired", "stale" (past recheck_by), or "expiring_within:N" (docs whose valid_until falls within N days). Filtering for "expired" implies including expired docs. A malformed value is rejected (HTTP 400 on REST). "stale" here is recheck_by-only, narrower than the per-hit freshness: "stale" a hit can also carry from verification age (issue #142); kb_curate lists every document currently showing freshness: "stale" for either reason.

• Compact hits carry a deviation-only freshness field (stale/expired/upcoming), omitted when fresh, plus a freshness_reason naming the field and date/day count behind it (omitted exactly when freshness is). A doc with no validity block is omitted only while KB_REVIEW_DUE_AFTER_DAYS is unset; with the threshold enabled such a doc has no last_verified either, so it reports stale with reason last_verified is not set. A doc with a future valid_from stays in default results flagged upcoming; only in_force=true excludes it. A stale doc (past recheck_by, or -- with KB_REVIEW_DUE_AFTER_DAYS set -- an old or absent last_verified and no recheck_by override, issue #142) stays in force and visible: review-due is always advisory, never a filter.

kb_get by id always resolves regardless of expiry, returning the full validity object plus the computed freshness indicator. kb_consult never returns an expired (or upcoming, proposed/retired, memory-inbox, or graph-excluded) doc, via its unconditional in_force=true retrieval (see the in_force and "Supersession-graph exclusion" sections above), not merely by reusing the default search path's own expired-exclusion. The data-olympus validity-report CLI subcommand lists expired and soon-to-expire docs from a bundle directory.

kb_curate: which in-force documents are review-due (issue #142)

kb_curate (MCP tool, GET /api/v1/curate) aggregates the same freshness/freshness_reason signal kb_search and kb_get already expose per-document into one list: every in-force document currently reported stale, most-overdue first, each with id, path, title, and reason. It is the answer to "what should I look at", where the search/get fields are the answer to "is this one doc stale" -- both come from the SAME compute_freshness call, so they cannot disagree.

Advisory and read-only, with no write-pipeline dependency at all: kb_curate recommends, it never proposes, edits, promotes, or demotes anything. Registered in read-only replicas (issue #44) alongside kb_search/kb_get/ kb_list/kb_outline, and reachable without a token the same way those are -- it is not gated like kb_list_pending, since it surfaces document metadata already retrievable through ordinary reads, not pending-queue state.

Candidates come from the exact same in-force predicate kb_search(in_force= true) uses (status class AND validity window AND not-inbox AND not-graph-excluded), so an expired, upcoming, draft, retired, or memory-inbox document never appears regardless of how overdue its last_verified looks. An empty corpus or a corpus with nothing review-due returns a valid empty list, never an error. With KB_REVIEW_DUE_AFTER_DAYS unset (the default), only recheck_by-based staleness is ever reported, matching every other read tool's pre-#142 behaviour.

Ordering: among verification-age entries, a document with NO last_verified at all sorts first (nobody has ever checked it, the most urgent case), ahead of one that was at least verified once however long ago. recheck_by is classified first, so a document carrying an explicit past recheck_by is a recheck_by-driven entry even when it has never been verified, and orders by days past that date rather than sorting into the never-verified group; a verification-age entry orders by days past the KB_REVIEW_DUE_AFTER_DAYS threshold. limit clamps to 1..100 like kb_search's, defaulting to 50.

kb_curate serves issue #142's operator-visibility criterion: it makes the review-due signal answerable in one call instead of requiring the operator to know to filter for it. The tool NAME is reserved on issue #31, which avoids a collision with the existing kb_audit event-log tool, but #31's requested capability -- detecting repeated patterns across the corpus and proposing to hoist them up the tier chain via kb_propose_edit -- is not implemented here and #31 stays open. That work needs its own detection design.

Retention, freshness, and authority are three separate things

Issue #142 adds a freshness signal and nothing else. The three concepts are deliberately independent, and no tool in this release couples them:

• Retention is whether a document still exists. Nothing in the freshness path deletes, archives, or prunes anything. A document that has gone unverified for a decade is still served, still retrievable by kb_get, and still a kb_search candidate. There is no automatic deletion for long-term governed knowledge, by design.

• Freshness is how long it has been since somebody checked the document, derived from recheck_by or from last_verified age against KB_REVIEW_DUE_AFTER_DAYS. It is advisory: a report about the review process, not a claim about the content.

• Authority is whether the document governs, and it is status plus the validity window plus the inbox/graph exclusions, single-sourced through is_in_force. Only those inputs decide it.

Concretely: a stale document with status: approved inside its validity window is still in_force: true, still ranks normally, and is still returned by default. freshness never feeds in_force, never reorders default retrieval, and never changes a status. kb_curate reports on the in-force set; it does not shrink it. There is no authority_state enum parallel to status, and an operator acting on a kb_curate entry is re-verifying a document, not demoting it.

abstain: signal-gated abstention

kb_search accepts an abstain: bool = False parameter (MCP tool and GET /api/v1/search?abstain=true). When set, the query is first run restricted to the discriminating columns (title, tags, applies_when). If it matches none of them, the query is treated as out-of-scope and the search returns NO hits with abstained: true and abstain_reason: "no_signal_match", instead of surfacing a weak match on generic body prose. A query with a real signal retrieves normally over all columns (recall is preserved).

The response distinguishes a deliberate abstention from an ordinary empty result: a normal search that simply finds nothing returns abstained: false. The gate logic is single-sourced in data_olympus.search_gate; the benchmark ablation imports it rather than reimplementing it.

Taxonomy and writable paths

The server maps each document's path to a (tier, category) pair. The built-in default is deployment-neutral and covers universal/, tech-stacks/<stack>/, decisions/, workflows/, memory/ (with memory/inbox/ and memory/accepted/), tooling/, templates/, and projects/<name>/ (with components/<component>/ for T4).

Every path under universal/ is T1. The subdirectories foundation/, quality/, security/, infrastructure/, database/, api/, and services/ take their own name as the category; any other path under universal/, such as universal/README.md or universal/process/x.md, has the category universal. A path outside every prefix in the table gets the tier and category meta.

A bundle that uses a different directory layout overrides the defaults at deploy time, with no code change:

• KB_TAXONOMY_PATH: path to a JSON file holding a list of [prefix, tier, category] triples that replaces the default table. The tech-stacks/ and projects/ prefixes keep their dynamic stack:<name> / project:<name> behavior if present.

• KB_INDEXED_PREFIXES: comma-separated list of writable top-level prefixes that replaces the default writable set. Writes outside these prefixes are rejected by the structural rule.

• KB_MEMORY_INBOX_PREFIX: directory new memory proposals are written under (default memory/inbox/).

A document's frontmatter tier or category, when present, overrides the path-derived value in the index, so search facets and listings use it. The write blocklist (KB_WRITE_BLOCK_TIERS) ignores that override and classifies the target by path, so an author cannot declare a document out of a blocked tier. data-olympus lint <bundle> reports each disagreement as a warning that names the path-implied value, using the same taxonomy (KB_TAXONOMY_PATH when set). The warning never changes the exit code. Because the frontmatter vocabulary has no spelling for the meta path tiers (decisions, workflows, memory, tooling, templates, or an unmatched path), a declared meta satisfies them. If KB_TAXONOMY_PATH cannot be loaded, lint prints a notice on stderr and skips this check.

Synonym / acronym query expansion

Before building the FTS MATCH, the server rewrites the query term list through a curated, bidirectional synonym map. A short-form query (k8s, rls) also reaches documents that only use the long form (kubernetes, row level security), and vice versa; adjacent query tokens are scanned as n-grams so multi-word keys match from a long-form query. Expansion is bounded (32 terms) and de-duplicated, and the terms the user actually typed are ranked first. This is curated lexical expansion, not semantic (embedding) retrieval.

The default map ships a small set of high-value, unambiguous technical acronyms (k8s/kubernetes, authn/authz, rls, adr, rbac, sso, pii, iac, kb). Ambiguous or generic short tokens are intentionally left out of the defaults to avoid recall noise; add them per deployment via KB_SYNONYMS.

Two environment variables tune it:

• KB_SYNONYMS: extra or override groups in key=variant,variant;key2=variant form. Groups are ;-separated; within a group the key is the canonical form and the comma-separated tail are its variants. Whitespace is trimmed and empty entries are dropped. Multi-word keys and variants are allowed (e.g. row level security=rls). Example: KB_SYNONYMS="feature flag=ff,flag;service level objective=slo".

• KB_SYNONYMS_MODE: how KB_SYNONYMS combines with the built-in defaults.

  • merge (default): layer KB_SYNONYMS on top of the curated defaults; a key present in both is overridden by KB_SYNONYMS.

  • replace: use only KB_SYNONYMS; the curated defaults are dropped.

  • off: disable expansion entirely (the expander becomes a passthrough, so only the terms the user typed are matched).

Optional local-embedding hybrid ranking

For real paraphrase / synonymy handling (a query that shares no tokens with the relevant doc, e.g. car for a doc about automobile), the server can blend BM25 with LOCAL embedding similarity. This is optional and OFF by default: with the feature off and the embeddings extra uninstalled, nothing imports the embedding stack and the default lexical product is unchanged. There is no external API and no network call at query time; a small ONNX model runs in-process.

When enabled, each document is embedded at index-build time and its vector is stored in a doc_vectors table (schema v8) written into the same tmp database and swapped atomically with the rest of the index, so vectors rebuild atomically. At query time a hybrid reranker embeds the query once and blends normalised BM25 with cosine similarity over the candidate hits' stored vectors, then re-sorts. It composes UNDER the exact-id / exact-tag short-circuit (so an exact id or tag still ranks first) and wraps the status-aware rerank (so an active doc still outranks the superseded one it replaced). When disabled the reranker stack is exactly the status + id/tag behaviour described under Search ranking.

Enable and tune it with:

• KB_EMBEDDINGS_MODE: on to enable; any other value (including unset) leaves it off.

• KB_EMBEDDINGS_MODEL: the local model name (default BAAI/bge-small-en-v1.5, a 384-dimensional MiniLM-class model). Any fastembed-supported model works.

• KB_EMBEDDINGS_WEIGHT: the cosine fraction of the blended score, in [0, 1] (default 0.35). 0.0 is pure lexical (BM25) ordering, 1.0 is pure semantic. A malformed or out-of-range value fails startup loudly.

Install the extra to make the model runner available:

uv sync --extra embeddings   # or: pip install 'data-olympus[embeddings]'

The extra pulls in fastembed, which bundles onnxruntime (there is no torch dependency). The default model ships as a ~33 MB quantised ONNX file that is fetched once when the feature is first enabled and cached locally; after that there is no network access. If KB_EMBEDDINGS_MODE=on but the extra or the model is unavailable, startup fails loudly with an actionable message rather than silently reverting to lexical-only ranking.

Cost: build adds one batched local embedding pass over the corpus (seconds for a small KB); storage adds dim x 4 bytes per document in doc_vectors (~1.5 KB per doc for the 384-dim default). Query-time cost is one query embedding plus a cosine over the (bounded) candidate pool.

Corpus co-occurrence query expansion

A build-time pass mines the corpus for term pairs that co-occur far more often than chance (positive pointwise mutual information) and stores the top related terms per term. At query time the expander adds those related terms so a query reaches documents that use a different-but-associated vocabulary, without a hand-curated synonym map. This is ON by default and adds no query-time network call. It auto-disables on a corpus below the doc floor (too little signal), and the O(n^2) pair counting is bounded per document.

Tune it with:

• KB_COOCCURRENCE_MODE: off disables it (both the build-time table and the query-time expansion); any other value (including unset) leaves it on.

• KB_COOCCURRENCE_K: max related terms kept per term (default 5).

• KB_COOCCURRENCE_MIN_COUNT: minimum raw co-occurrence count for a pair to qualify (default 3).

• KB_COOCCURRENCE_MIN_PMI: minimum PMI for a pair to qualify (default 0.1).

• KB_COOCCURRENCE_MIN_DOCS: corpus-size floor below which co-occurrence is auto-disabled (default 50).

• KB_COOCCURRENCE_MAX_DOC_TOKENS: per-document unique-token cap on the pair counting, bounding the O(n^2) work on large docs (default 400).

• KB_COOCCURRENCE_MAX_PAIRS: how many distinct pairs the counter may hold in memory before spilling to a temporary SQLite file (default 500000). 0 disables the spill and keeps everything in memory.

Build memory

The per-document token cap bounds each document's contribution. It does not bound the union of pairs across documents, and that union is what the counter holds. On the maintainers' corpus of 536 markdown files (3.6 MiB) the union was 6.2 million distinct pairs, and the build peaked at 1445 MiB, which is above a 1 GiB container limit on its own. Since the pair space grows superlinearly with corpus size, this is a wall every deployment eventually reaches; where it sits depends on the corpus.

KB_COOCCURRENCE_MAX_PAIRS bounds it. Once the counter reaches that many entries it is flushed into a temporary SQLite database and cleared, and the counts are aggregated back out at the end. The table produced is identical either way, so no search result moves; only memory and build time change. On the same corpus, measured on one laptop:

The default trades build time for a ceiling that does not grow with the corpus. Raise it, or set it to 0, on a host with memory to spare and a corpus that fits. The bound is on pair counting specifically: parsed documents, per-document token sets and the candidate lists sit outside it, and the counter is checked between documents, so it can exceed the cap by at most one document's own pairs.

The spill writes to a temporary file, so it can fail on a full or read-only disk. That failure does not fail the index build: the build logs a warning and continues with no related-terms table, so the index still serves and only co-occurrence expansion is lost until the next successful build.

Memory between rebuilds

A build's working set is freed when the build returns, but on Linux freed memory does not necessarily leave the process. Two glibc behaviours keep it resident:

• glibc serves threads from separate arenas and keeps free memory in them for reuse. The refresh loop runs rebuilds on an executor thread and request handlers run on a worker pool, so several long-lived arenas each end up holding a build-sized amount of free memory.

• Free space at the top of a worker thread's arena is returned only when it exceeds the trim threshold, and glibc raises that threshold on its own after large blocks are freed, up to 64 MiB. After a rebuild each worker arena kept about 10 MiB of free top that nothing returned.

Resident memory therefore climbed to a plateau several times the starting size and did not come back down while the server idled. Nothing is leaked at the Python level.

The server pins the trim threshold at 128 KiB when it starts (mallopt, which also stops the automatic increase), and asks the allocator to return free memory after the bootstrap build and at the end of every refresh tick, whether or not the tick rebuilt (malloc_trim(0), which walks every arena, so memory freed by request handlers between rebuilds is covered as well). The pin is process-wide, and it also stops glibc adjusting its mmap threshold, so large blocks keep being served by mmap and returned on free rather than cached in an arena. If you set the trim threshold yourself, through a non-empty MALLOC_TRIM_THRESHOLD_ or a glibc.malloc.trim_threshold=<value> entry in GLIBC_TUNABLES, the server leaves your value alone. A bare or empty glibc.malloc.trim_threshold entry is not a setting, because glibc ignores a tunable without a value, so the server pins the threshold as usual. In a separate probe on a heap with four worker arenas, one malloc_trim(0) usually took between 0.1 and 2.3 ms, with a slowest observed call of 12 ms; it runs on the refresh executor thread, once per tick. This is best-effort: glibc returns whole free pages, not every freed allocation, and a rebuild still needs its working set while it runs, so the container needs room for the baseline plus one build. Where the allocator offers neither call, as on macOS, both steps do nothing.

Measured on Linux (glibc 2.41, CPython 3.13, 1 GiB container) with a 641-file, 4.4 MiB corpus, replaying a burst of eight writes through kb_propose_edit, each followed by a rebuild, with five of the documents a single 23,000-character line, three times, while three clients issued searches, consultations and gate checks every second. Resident anonymous memory:

Setting MALLOC_ARENA_MAX=2 in the environment, with no code change, lowered the plateau of the same burst without that traffic from 324 MiB to 223 MiB. With this change it made no measurable difference in these runs.

Extending the governed action vocabulary

The enforcement gate classifies an action as governed from three shipped lists in enforce_policy.py: keywords, path globs, and command fragments. A deployment can govern an action class this product has never heard of without embedding the server in its own Python:

Each is a comma-separated list, and each extends the shipped list. There is deliberately no setting that replaces one, because a configuration error would then silently remove enforcement the product already provided. Unset means add nothing, which is exactly the shipped behaviour.

Entries are trimmed and de-duplicated, and are bounded: at most 500 per list and 200 characters each. The classifier runs on every classified action and compiles a regex per keyword, so an unbounded list is an availability problem rather than untidiness. Anything dropped or truncated is logged with a warning naming the variable, because an enforcement setting that quietly does less than it says is worse than one that refuses.

For example, a deployment that must not run window or input tests against a live desktop can add KB_GOVERNED_EXTRA_KEYWORDS="window test,input test" and have the gate require a fresh explicit consultation before such an action, without waiting for that class to appear in a release.

This changes only WHAT is classified as governed. It does not change the rule that only a fresh explicit consultation clears the gate. Under the default KB_GATE_CLEARANCE=intent, an operator addition has no keyword mapping: a consult covers it by naming the keyword, path or command fragment in its intent.

Trigram fuzzy-match fallback

For typos and partial identifiers, an optional trigram index backfills results when the primary FTS query returns few hits. This is OFF by default so the default deployment pays no trigram build or storage cost; when off, the trigram table is neither created nor populated. When on, a primary query returning at or below the threshold is backfilled from the trigram index, and the backfill only ever appends after the primary hits (never reorders them).

Tune it with:

• KB_TRIGRAM_MODE: on (case-insensitive) enables it; any other value (including unset) leaves it off.

• KB_TRIGRAM_FALLBACK_THRESHOLD: primary-hit count at or below which the fallback fires (default 3). A malformed or negative value falls back to the default.

Authentication and network security

The server supports optional bearer-token authentication with a per-principal capability model, enforced on both the REST routes and the MCP write tools:

• KB_AUTH_TOKEN registers a single full-capability operator principal.

• KB_AUTH_PRINCIPALS (JSON) registers per-agent tokens with explicit capabilities; a principal lacking auto_commit has its proposals clamped to pending regardless of the client-asserted confidence. When set, it must be a non-empty JSON list of objects, each with a non-empty string token. Anything else (invalid JSON, a single object, an entry that is not an object, or an entry without a usable token, such as a misspelled key) fails startup with an error naming KB_AUTH_PRINCIPALS and, where it applies, the entry number. The error never repeats the value. This holds whether or not KB_AUTH_TOKEN is also set.

When auth is configured, write, enforcement, and observability routes require a capable principal; read routes (search/get/list/outline/health) stay open. When it is unset (the default) every caller is fully trusted, which is only safe on a trusted private network.

Authentication does not protect read routes, so for confidentiality operators MUST still deploy on a trusted private network or behind an authenticating reverse proxy (terminate TLS there). See SECURITY.md for the full threat model, the route/capability table, payload limits, the tamper-evident audit log, and the git sync-failure health fields.

Host allowlist behind a proxy (KB_PUBLIC_HOSTNAMES, v0.4.3+)

fastmcp >= 3.4.3 validates the Host header on streamable HTTP (DNS-rebinding protection) and returns 421 Misdirected Request for hostnames outside its allowlist. When data-olympus is served behind a reverse proxy or ingress, set the public hostname explicitly via the first-class knob (comma-separated, no JSON):

KB_PUBLIC_HOSTNAMES=kb.example.com

This is mapped onto fastmcp's Host-header allowlist when the HTTP app is built. The older dependency knob FASTMCP_HTTP_ALLOWED_HOSTS='["kb.example.com"]' still works and is merged with KB_PUBLIC_HOSTNAMES when both are set.

Direct localhost/pod requests are always allowed, so health probes pass while proxied traffic fails: this is the exact shape of the 2026-07-09 kn-dev outage (readiness green, every proxied request 421). To make that failure loud, the server now emits a startup WARN when Host protection is on, it binds a non-loopback address, and no public hostname is allowed. Check the server log for that WARN or for 421 lines if agents suddenly cannot reach the endpoint after an upgrade. Operational details in docs/operations.md section 3.4. (Closes issue #139 / KNA-70.)

Prometheus metrics (GET /metrics, optional metrics extra)

prometheus-client is an OPTIONAL dependency. Install it to enable the scrape endpoint:

uv tool install 'data-olympus[metrics]'   # or pip install 'data-olympus[metrics]'

With the extra installed, GET /metrics serves the standard text exposition format. Without it, /metrics returns 501 Not Implemented (the route exists, the capability is simply not built into the image) and every metric update inside the core background loops is a silent no-op, so a standard deployment is unaffected. Exposed series (prefix data_olympus_):

• pending_queue_depth, push_queue_depth, push_queue_frozen (gauges)

• push_failures_total (counter)

• staleness_seconds, live_sessions (gauges)

• tool_calls_total{tool="..."} (counter, per MCP tool)

• index_build_duration_seconds (histogram) and index_last_build_timestamp_seconds (gauge)

The gauges are refreshed from the refresh / push-retry loops each tick and also snapshotted on scrape, so a scrape reflects the live queue depths and freshness. (Closes issue #69 / KNA-71.)

Proxy headers and the rate limiter (KB_TRUSTED_PROXIES)

The rate limiter keys on the client remote address (plus principal). Behind an ingress or reverse proxy, uvicorn by default sees the proxy's address as the peer, so every client collapses into one remote_addr and the per-IP cap (KB_RATE_LIMIT_PER_IP_PER_HOUR) throttles all clients as if they were one.

Set KB_TRUSTED_PROXIES to the proxy address(es) (comma-separated) to fix this:

• Unset (default): uvicorn runs with proxy_headers off. X-Forwarded-For is ignored and remote_addr is the immediate peer. This is the safe default: a direct client cannot spoof its address to dodge the limiter.

• Set to the proxy IP(s): uvicorn enables proxy_headers and restricts forwarded_allow_ips to those addresses, so it rewrites remote_addr from X-Forwarded-For only when the immediate peer is a trusted proxy. The limiter then sees the true client IP.

• *: trust X-Forwarded-For from any peer. Only safe when nothing untrusted can reach the port directly (e.g. the port is bound to the pod network and only the ingress can connect). Otherwise a direct client could forge the header.

The gate-check route (KB_GATE_CHECK_RATE_LIMIT_PER_HOUR)

POST /api/v1/gate/check (and the kb_gate_check MCP tool) is the enforcement hook's freshness probe, called once per gated tool action by every agent. It therefore must not share the write/consult quota (KB_RATE_LIMIT_PER_HOUR): with several agents active, and all clients collapsing to one limiter bucket behind an ingress, a fixed hourly quota self-throttles the whole fleet with 429s. So gate-check has its own ceiling:

• KB_GATE_CHECK_RATE_LIMIT_PER_HOUR=0 (default): gate-check is not rate-limited. It does only cheap classification plus a freshness lookup and no writes, so this matches the read routes, which are also unthrottled.

• A positive value: an explicit per-(address, principal) backstop just for gate-check, independent of the write/consult limiter. Size it above your fleet's real per-hour tool-call volume, not near it, or you reintroduce the self-DoS.

The consult and onboarding/cleanup-plan routes remain throttled by KB_RATE_LIMIT_PER_HOUR (they write to the ledger or are CPU-heavy).

Audit-log rotation (KB_AUDIT_MAX_BYTES)

The JSONL audit log (KB_AUDIT_LOG_PATH, default /state/audit/events.log) grows without bound by default. Set KB_AUDIT_MAX_BYTES to a byte threshold to enable size-based rotation: when the live file passes the threshold, the next append first renames it to events-<UTC-timestamp>.log and starts a fresh live file.

• Chain continuity. The tamper-evident hash chain carries across the rotation boundary: the first event of the new file links to the last hash of the rotated file, so GET /api/v1/audit/verify (and the kb audit/verify logic) validates the whole history, rotated segments included, in chronological order. A break at the boundary is caught like any other.

• Backward compatible. With KB_AUDIT_MAX_BYTES unset (0), nothing rotates: the log stays a single file exactly as before, and an existing single-file log still verifies unchanged when opened by a rotation-aware build.

• Reads. kb audit reads the live file by default (cheap, most-recent-first). A --since query also walks rotated segments so history that has rotated out of the live file is still visible; the reverse scan is bounded so a large archive cannot turn one query into an unbounded read.

Manual mv-based rotation (documented in the operator runbook) no longer breaks the chain when the process keeps running, because the in-memory last-hash carries forward; but prefer KB_AUDIT_MAX_BYTES so rotation is automatic and the boundary link is always written. See docs/operations.md for backup/verify procedures.

Git commit identity

Every write commits to the KB repo, so git needs an author/committer identity. The shipped Docker image sets a default (data-olympus-mcp <data-olympus-mcp@localhost>) in deploy/docker/entrypoint.sh so the artifact can commit out of the box; the server's main() also fills unset GIT_* variables as a belt-and-suspenders for non-Docker runs. Override the identity with:

• KB_GIT_AUTHOR_NAME: commit author/committer name (default data-olympus-mcp).

• KB_GIT_AUTHOR_EMAIL: commit author/committer email (default data-olympus-mcp@localhost).

A pre-existing GIT_AUTHOR_*/GIT_COMMITTER_* value or a real git config identity is never overwritten.

Running locally

See docs/quickstart.md for the verified local-run procedure using scripts/run-local.sh.

Running in Kubernetes

See deploy/k8s/ for the kustomize manifests. Apply them with:

# Apply namespace and non-secret resources
kubectl apply -k deploy/k8s/

# Apply the encrypted secret separately (operators use SOPS)
sops exec-file deploy/k8s/secret.sops.yaml 'kubectl apply -f {}'

The deploy/k8s/secret.template.yaml file shows the expected secret shape. Fill in your git remote URL and deploy key, then encrypt with SOPS before committing. Never commit the unencrypted secret.

The default kubectl apply -k deploy/k8s/ deliberately does not create the Ingress: it publishes all routes (including the unauthenticated-by-default write routes) to the LAN. The Ingress is opt-in and enabled only after you set KB_AUTH_TOKEN; see deploy/k8s/README.md for the enablement steps.

The pod is fully rootless: both the prepare-git initContainer (which stages the deploy key and does the first-boot /kb-main clone) and the main container run as uid 65534 with runAsNonRoot: true, all capabilities dropped, and a read-only root filesystem, so the manifest passes the restricted Pod Security Standard. The base image is digest-pinned in deploy/docker/Dockerfile; the git SSH host is build-arg + runtime-env configurable (KB_SSH_KEYSCAN_HOST) for non-GitHub remotes. Operational procedures (backup, upgrade, recovery playbooks) live in docs/operations.md.

Running with Docker Compose

docker compose -f deploy/docker/compose.yaml up --build

See docs/quickstart.md section 5 for the bind-mount pattern to serve a real bundle.