forgetful

Forgetful architecture

Reviewable, agent-authored overview, checked against source on 2026-10-08, with boundary changes checked on 2026-10-17 and Memory Plugin phases on 2026-10-19. This is a selective interpretation, not an exhaustive dependency inventory. The component groupings below are inferred for readability; the described execution paths and wiring were verified against source. Line numbers are inspection starting points and may drift.

Forgetful provides persistent, semantically searchable memories and related knowledge through MCP tools, HTTP endpoints, and a CLI. Its intended boundary is routes → services → protocols → repositories/adapters, as described in AGENTS.md. Protocols are Python contracts, not an extra runtime process: services call injected implementations. SQL, vector storage, provider clients, and database sessions normally stay behind those contracts. Operational exceptions are called out below rather than hidden by the diagram.

Component view

flowchart TD
    MCP["MCP: three meta-tools"] --> Registry["ToolRegistry"]
    Local["Local CLI executor"] --> Registry
    Remote["Remote CLI executor"] --> MCP
    Registry --> ToolAdapters["Tool adapters: identity, models, results"]
    ToolAdapters --> Services["Domain services"]
    HTTP["HTTP custom routes"] --> Services
    Services --> Contracts["Repository protocols"]
    Services --> PlanPlugin["PlanWritePluginService: create/update/delete, including automatic completion"]
    Bootstrap --> PlanLoader["config.plan_plugin: trusted PlanPlugin loader"]
    PlanLoader -. supplies registration .-> PlanPlugin
    Services --> WritePlugin["WritePluginService: before_save and advisory after_save"]
    WritePlugin --> Callable["MemoryPlugin: explicit operation-to-handler mappings"]
    Services --> EntityWritePlugin["EntityWritePluginService: before_save and advisory after_save"]
    EntityWritePlugin --> EntityCallable["EntityPlugin: create, update, delete mappings"]
    Bootstrap --> Loader["config.write_plugin: trusted module:object loaders"]
    Loader -. supplies registration .-> WritePlugin
    Loader -. supplies registration .-> EntityWritePlugin
    SQLite["SQLite repositories"] -. implements .-> Contracts
    Postgres["PostgreSQL repositories"] -. implements .-> Contracts
    SQLite --> SQLiteDB["SQLite adapter and sqlite-vec"]
    Postgres --> PostgresDB["PostgreSQL adapter and pgvector"]
    SQLite --> Providers["Embedding and rerank adapters"]
    Postgres --> Providers
    Services --> Events["Optional EventBus"]
    Events --> Activity["ActivityService and memory access handler"]
    Activity --> Contracts
    Bootstrap["bootstrap.build_runtime"] -->|constructs and injects| Services
    Bootstrap -->|registers implementations| Registry
    Bootstrap -->|selects backend| SQLite
    Bootstrap -->|selects backend| Postgres

Boxes combine multiple classes/files; they are not declared packages or separate deployments. Solid arrows show source-verified use or explicitly labelled startup wiring. Dotted arrows mean structural implementation of a contract, not inheritance or a request passing through a protocol. Only memory/skill repositories use the embedding/rerank branch. Health/auth routes and operational commands have additional paths omitted here; the diagram does not assert universal layering.

Entry points and lifecycle

The bootstrap factories select PostgreSQL or SQLite repositories at bootstrap.py:91, passing embedding and rerank adapters to memory/skill repositories. FILES_ENABLED, SKILLS_ENABLED, and PLANNING_ENABLED control matching repositories, services, metadata registration, and HTTP route registration. GraphService receives memory/entity repositories plus the other services, including optional ones (bootstrap.py:311). The runtime owns the optional embedding adapter and closes its resources after draining pending events, before database disposal. Startup failures also close the adapter; the standalone re-embedding CLI closes it on every exit. FastEmbed uses one model and an adapter-layer bounded executor shared by memory and skill operations. Admission precedes submission; cancelled running inference remains counted until its worker finishes. OpenAI and Azure OpenAI embedding HTTP calls each use one native async SDK client and a shared HTTP connection pool capped by the same per-process setting. SDK retry backoff releases connection capacity. This is a connection limit, not a logical-operation or request-rate limit; waiting callers and remote processing after local cancellation are not bounded. Shutdown closes the SDK client without draining in-flight OpenAI or Azure operations, through the same runtime, startup-error and CLI exit paths.

Settings and validation live in settings.py:33.

The local executor:18 builds the same runtime, supplies a CLI context, checks the instance's permitted tools, and executes the registry in process. The remote executor:67 calls the server's three meta-tools through an MCP client; it does not build a second local database stack. Both conform to ToolExecutor:10.

MCP, HTTP, and identity

The active MCP surface is registered by meta_tools.py:473:

1. discover_forgetful_tools lists permitted tool metadata, optionally by category.

2. how_to_use_forgetful_tool returns detailed metadata for a permitted tool.

3. execute_forgetful_tool checks the name/permissions, injects context, and dispatches it.

The full operation set is dynamic: feature flags and scopes affect what a caller sees. tool_metadata_registry.py:874 constructs adapter callables and registers metadata/implementations together. Despite its filename, this is a module of registration helpers, not a second ToolMetadataRegistry class. ToolRegistry.execute:188 looks up and awaits the callable. MemoryToolAdapters.create_memory:199 illustrates identity resolution, argument conversion, model construction, service delegation, and response formatting. The older category *_tools.py registration functions also appear in the map; their presence does not mean they are mounted by main.py, which registers meta_tools at line 150.

HTTP endpoints are FastMCP custom routes accepting Starlette requests, rather than a separate FastAPI application despite the opening docstring in main.py. main.py:130 mounts health, auth info, domain CRUD, graph, and activity routes; files, skills, plans, and tasks are conditional. The root service-info route starts at main.py:115. Examples:

build_auth_provider:106 selects configured GitHub/Google OAuth, JWT, or introspection providers, or no authentication. MCP identity comes from validated tokens or a default user (auth.py:106); HTTP identity has its own bearer-token verification/cache path (auth.py:157). Services receive an internal user ID. scope_resolver.py:162 intersects token permissions with the instance ceiling for MCP. Token scopes come from the scope/scp claim; unrecognised scopes are ignored, and a claim with scopes but none recognised permits nothing. Its empty-ceiling fallback permits all registered tools; do not interpret an empty ceiling as deny-all. HTTP routes do not automatically inherit registry scope checks: inspect their own authorization path before assuming parity. This overview is not a security audit.

Memory write, retrieval, and linking

Create: route/tool adapter → MemoryService.create_memory → optional before-save decision → repository → embedding provider and database. memory_service.py:185 applies provenance defaults, persists the memory, then optionally finds similar memories and creates links in a batch. The repository builds embedding text and persists associations as part of its create operation: PostgreSQL:191, SQLite:258. Memory creation and subsequent automatic linking are separate repository calls, not one service-wide transaction.

Automatic linking uses the configured neighbor limit and similarity threshold, excluding obsolete memories and the source memory (PostgreSQL:456, SQLite:516). Explicit linking is a separate service operation (memory_service.py:631); repository link methods maintain the bidirectional memory relation and only link memories owned by the calling user. Obsolescence is a soft state with reason and optional replacement; create responses can additionally warn about similar obsolete memories (memory_service.py:552).

Plan Plugin: config.plan_plugin validates one trusted PlanPlugin registration at startup, including when planning is disabled. PlanWritePluginService dispatches explicit create/update/delete maps with detached contexts; it cannot rewrite core input. PlanService retains ownership, status, provenance and reference rules, rereading after a before-handler await. Serial successful mutations invoke each registered phase once; absent plan CAS, concurrent callbacks can repeat. Transient write results carry warnings to flat REST/MCP/CLI payloads without changing reads or stored models. TaskService routes completion after task transition and deletion through this same policy. Rejection or policy unavailability is advisory for the committed task; unrelated post-commit failures and genuine cancellation retain existing propagation. Plan deletion cascades tasks/criteria/dependencies without per-task hooks; project/user cascades do not dispatch Plan hooks. No durable delivery or rollback.

Entity Plugin: mirrors the Memory Plugin for entity create, update, and delete via load_entity_plugin, EntityPlugin, and EntityWritePluginService. Handlers see detached entity snapshots only; deleting an entity row cascades at the database to entity_project_association, memory_entity_association, entity_file_association, and entity_relationships rows whose source_entity_id / target_entity_id reference the entity, without separate plugin hooks.

Memory Plugin: app/config/write_plugin.py loads one trusted module:object registration before costly resources. MemoryPlugin in app/protocols/write_plugin_protocol.py owns copied, frozen before_save and after_save mappings from explicit operations to async handlers. Supported operations are create, update, obsolete, and restore; an absent mapping skips that handler. Invalid registrations fail startup. This replaces the unreleased function contract without a compatibility layer. Bootstrap injects WritePluginService; services never import the loader. app/models/write_plugin_models.py owns detached contexts, decisions, advisory warnings, and internal write results. The existing MEMORY_WRITE_PLUGIN* settings remain.

Before create/update, the plugin gets separately detached submitted/previous/proposed snapshots, with the writable candidate built by MemoryService after defaults and merging. Lifecycle contexts use MemoryLifecycleRequest and MemoryLifecycleState: submitted ID/reason/replacement, full previous memory, and proposed obsolete flag/reason/replacement. They do not pretend lifecycle state is a MemoryCreate, or invent a database timestamp. Core permissions, ownership, validation, and state checks still apply. Invalid operations skip policies; no-op updates skip both phases. Repeating obsolete retains its existing mutation behavior, including a new obsolescence timestamp.

Enforce-mode rejection or before-handler failure blocks persistence/events; warn mode proceeds with warnings. No session spans a handler await. After a before-update handler completes, the service re-reads the owned memory and recomputes patch differences for embeddings and events. A resulting no-op retains timestamp and warnings without an event or after handler. Lifecycle mutations patch only lifecycle fields, refresh previous state after before-handler waits, and retain the repository's conditional restore check. Original policy snapshots are not re-evaluated. These measures remove avoidable stale overwrites caused by plugin waits; they do not make checks atomic or prevent later concurrent changes, including the existing embedding-provider await race. MemoryService still normalizes explicit-null updates before provenance/change detection (#95), preserving submitted snapshots. Non-null skill association update limitations (#96) remain.

After successful persistence, existing create similarity/link work, and event emission, MemorySavedRequest supplies detached user/operation/previous/saved state. Create reuses existing scored MemorySummary matches (four-decimal scores); saved link IDs report actual resulting links. Similarity is None for other operations or disabled auto-linking, and [] for a computed empty search. No new similarity query is introduced. Obsolete reads back saved state for a registered after handler; restore reuses its existing read. A missing post-obsolete snapshot warns without fabricating saved state. These reads are separate repository calls, not transaction-wide snapshots.

After handlers return PluginWarning or nothing. Exception, malformed return, or cooperative timeout produces a sanitized diagnostic and a generic advisory warning in either mode; it cannot veto or roll back the saved write. External cancellation propagates even after persistence, so a cancelled request or lost response may leave a completed write. Cancellation raised inside an after handler or its awaited child becomes advisory when the request task has no cancellation pending. Before-save cancellation still propagates. There are no durable queues, retries, delivery guarantees, duplicate prevention, merging, or link/entity hooks.

app/routes/write_plugin_responses.py combines nonempty-only before/after warnings. Create/update shapes stay intact; lifecycle results use an internal success/warnings model, unwrapped to {"success": true} plus optional warnings by live MCP/local CLI and REST DELETE. No REST restore route is added. Stored/read memories and event payloads have no transient warnings. Before errors remain WRITE_PLUGIN_REJECTED / WRITE_PLUGIN_UNAVAILABLE through MCP/CLI and 400/503 through REST. REST DELETE validates supplied lifecycle field types before the service call, with or without a plugin; absent-body defaults are preserved. After failures return success with MEMORY_PLUGIN_AFTER_SAVE_UNAVAILABLE. The legacy unregistered memory_tools.py unwraps lifecycle results to bool, without becoming a supported plugin transport. CLI memory-save rendering shows warnings with success.

Bootstrap restores the core diagnostic logger and pre-initialization states of existing loggers for the configured module, every handler's defining module, and their children after Alembic's logger-disabling setup. Re-exports, partial handlers, and intentionally disabled loggers are preserved. Import errors identify module/reference, loading phase, and exception type. Runtime diagnostics identify phase, operation, exception type, and before-save mode, without exception text or memory snapshots. See the plugin contract for administrator examples.

Query: memory_service.py:77 asks the repository for scored primary results, optionally expands one-hop links, deduplicates, and applies token/count budgets. Project restrictions apply to primaries; linked memories use the same project filter only when strict_project_filter is enabled. Budget allocation prioritizes primary results before links (memory_service.py:767). Within each tier, the budget helper sorts by importance (memory_service.py:844); the final response is not an untouched vector-ranking list. Scores are returned for retained primaries.

The repositories embed the query and search active, user-owned memories with optional importance and project filters. PostgreSQL uses pgvector cosine distance (memory_repository.py:142); SQLite joins memories to vec_memories and computes cosine distance (memory_repository.py:151). When reranking is enabled, they retrieve a larger candidate set and rerank when it exceeds k; query_context enriches the rerank query, not the initial embedding query (PostgreSQL:70, SQLite:78). When HYBRID_SEARCH_ENABLED is true, both backends also run a keyword leg (SQLite FTS5; PostgreSQL tsvector + GIN via Alembic 20261103_memory_search_tsv), fuse the ranked id lists with reciprocal rank fusion, then apply that same rerank pool. With the flag off, search stays dense-only and MemoryScore omits keyword_rank / fusion_score.

Contracts, storage, and implementation variants

Memory/skill update embeddings and skill search reranking do not retain a borrowed database connection during provider waits. Skill description updates use a short owned-skill/project precheck, then embed, then re-fetch and recheck in the write transaction. Description and vector are saved atomically, including SQLite's separate vector table. Searches rerank detached summaries of the original candidates without refetching. Re-embedding search validation reads a title, embeds outside the session, then searches in a new session. These are connection-lifetime guarantees, not locking or full concurrency guarantees; the memory snapshot race described above still applies.

The two persistence backends implement the same broad contracts but use different SQL/vector mechanisms. This source review does not establish complete behavioral parity or transaction equivalence. Provider availability, model cache contents, and live database policy configuration were not exercised.

Other domain and operational components

These are explanatory groupings; they do not imply separate subsystems in the source tree.

Startup constructs ReEmbeddingService and supplies it to MCP handlers; handlers call its validation and rebuild methods without accessing MemoryService.memory_repo. Embedding interfaces live in app/protocols; shared text preparation lives in app/utils. User repositories translate duplicate external IDs to ConflictError, which UserService handles by fetching the existing user after the failed transaction has rolled back. Activity handling is asynchronous and not part of the domain write transaction; it is not a durable queue.

Enforced import boundaries

The six contracts in pyproject.toml enforce these rules, including indirect imports:

All Python files under app must be visible to the checker; a regression test verifies this. Package markers are required, including for SQLite repositories, middleware, and utilities. Bootstrap constructs implementations. The CLI has its own checked rule: only app.routes.cli.local_executor and app.routes.cli.context may import bootstrap directly. Other CLI modules may reach bootstrap through those startup imports, but may not import repositories or bootstrap themselves. These exceptions belong only to the CLI rule; API/MCP handlers still cannot reach bootstrap through CLI code. Logging configuration may use app.middleware.logging_middleware as shared support. Package initializers and generated version metadata are scanned; they are not extra application layers. New source areas need an explicit decision about which rules apply.

main.py is a standalone startup/operational module, reviewed manually rather than scanned by Import Linter; tests, migrations, and the evaluation harness are outside the production rules. Static imports in functions, re-exports, and TYPE_CHECKING blocks are checked. Dynamic imports, runtime attribute access, and arbitrary file/network operations are not checked; review those in code review and cover behaviour with integration/E2E tests. The external-library list is explicit, not a complete ban on every possible integration.

Run uv run lint-imports --no-cache for the rules alone, or uv run pytest tests/architecture/ for the rules, source coverage, and deliberate-violation proofs. The CI test job runs the latter; validation notes describe the proofs.

Tests and validation entry points

Backend-sensitive changes should exercise both relevant E2E suites; contract/business-rule changes should first use the focused integration tests. Source links identify tests, not evidence that those suites have passed for any particular revision. Repository lint command: uv tool run ruff check ..

Using the detailed map

The Mycelium map is supporting detail, not required reading in full. Do not load the entire file into context by default. It is a historical snapshot: the October 17 boundary changes moved embedding/backup types and rebuild wiring; use this overview and current source for those components.

• Find the relevant declaration in the source index, reading that index in chunks if needed.

• Search its diagram identifier for connections and read the relevant line ranges.

• Expand to callers, contracts, alternative implementations, and tests as the task requires.

• Check relevant warnings and filtering notices. Missing edges do not establish independence.

• Verify important relationships against source; static call edges can be uncertain.

Useful IDs in this snapshot: c0016 bootstrap, c0276 main, c0240 ToolRegistry, c0256 MemoryService, c0120 MemoryRepository, c0143 PostgreSQL memory repository, c0171 SQLite memory repository, c0251 GraphService, and c0264 TaskService. The source index begins at map line 4921; relationships at 5233, warnings at 8246. IDs/positions can change on regeneration.

The map was freshly analyzed and exported using Mycelium 1.0.0 with --max-classes 1000 --test-path tests --test-path test_harness/runs. Repeating the export from the same map produced identical bytes. The output is nonempty and contains one class diagram; it was checked as Markdown, without a rendered preview. It contains 8,353 lines, 296 indexed boxes, 2,579 listed relationships, and 106 extraction warnings. Test filtering removed 4,561 calls and 119 type relationships. The box count is below the requested cap; full detail is not proof of complete extraction.

Warnings include unresolved or out-of-scope bases (Protocol, BaseModel, enums) and EventBus's asyncio.Task binding. These mostly name third-party/builtin types, but the diagnostics do not reliably classify the origin of every unresolved reference. A concrete heuristic mismatch: the map attributes TaskService.claim_task calling _emit_event to PlanService; source calls its own handler at task_service.py:336, defined at line 60. Runtime registration, decorators, and dynamic attributes require source inspection even when edges exist.

This overview is selective and does not replace source inspection. This map excludes tests and test_harness/runs, but includes harness implementation and some debug scripts. Inspect test sources or request a test-inclusive export when needed. Neither the overview nor the map establishes gap-free coverage.

Maintenance

After major code or architecture changes, review the diagram and explanations and update them if needed. Mycelium is optional; a source-verified manual update is acceptable. Preserve reviewed explanations that remain accurate. Keep this overview at most 500 lines, with no column-width limit. If the generated backup is refreshed, use fresh analysis, keep its JSON outside the repository, and record exporter version/scope and material warnings. Recheck links and Mermaid rendering.