Engrava

Dreaming — Memory Consolidation

Engrava's dreaming extension provides periodic memory consolidation: it evaluates eligible candidate thoughts, scores them against configurable signals, and — when the corresponding gates, thresholds, and caps permit — may promote qualifying ones by setting their priority to P1.

Dreaming runs outside the normal CRUD path — the consumer decides when to invoke run_consolidation() (after N cycles, in a cron job, or manually).

How memory consolidation works (the dreaming loop)

Think of a single memory's journey through an agent's lifetime. The first two steps — ingest and confirmation — happen on the normal write path as you use the store. The consolidation part is manual: when you call run_consolidation(), that one call runs promotion → edge creation → reflection clustering/creation → an orphan sweep, in order.

  ingest        you create an OBSERVATION ("user prefers email")   (write path)
    │
    ▼
  confirm       the same fact is re-encountered over time, so its  (write path)
    │           confirmation_count grows (e.g. via deduplicate=True)
    │
    ▼  run_consolidation(current_cycle=N)   ── manual ──
    │
  ┌─┴───────────────────────────────────────────────────────┐
  │ 1. promote   thoughts that pass the gates and clear       │
  │              promote_threshold are raised to priority P1  │
  │ 2. link      a promoted thought *may* gain ASSOCIATED      │
  │              edges to similar neighbours (when enabled)    │
  │ 3. reflect   related thoughts *may* be clustered into      │
  │              REFLECTION meta-thoughts (when enabled)       │
  │ 4. sweep     stale REFLECTIONs whose sources left the      │
  │              active set are retired                         │
  └─┬───────────────────────────────────────────────────────┘
    │
    ▼
  improved      later searches rank the P1 memory higher (priority
  retrieval     signal), follow any new edges (graph signal), and can
                surface a REFLECTION instead of many raw thoughts

Walking the journey:

1. Ingest. You store memories as thoughts (typically OBSERVATIONs) on the normal write path. Dreaming does nothing yet.

2. Confirm. As the same knowledge recurs, its confirmation_count rises — automatically when you write with deduplicate=True (identical content collapses and bumps the count), or via your own logic. This is evidence the memory matters, and it feeds dreaming's confirmation signal. (Distinct from confidence, the static belief-strength you set — see Core Concepts.)

3. Promote. When you run consolidation, each candidate must first pass the gates (e.g. old enough, enough confirmations) and then score above promote_threshold across the weighted signals. Survivors are promoted to P1. (Both bars matter: a thought that passes the gates but scores low is not promoted — see Troubleshooting.)

4. Link. A promoted thought may gain ASSOCIATED edges to similar neighbours — when edge creation is enabled, the thought has a stored embedding, and qualifying neighbours (above min_similarity) are found. New edges persist the structure in the graph, idempotently (re-runs don't duplicate edges).

5. Reflect. Related thoughts may be clustered and summarised into REFLECTION meta-thoughts — a centroid embedding plus CONSOLIDATED_FROM edges back to the members — when reflections are enabled and eligible clusters pass the clustering/quality gates. This turns a pile of observations into fewer, higher-level memories. (A REFLECTION whose source cluster later leaves the active set — or is hard-deleted — is automatically retired, unless it is pinned or protected, so a stale summary can't resurface; see the orphan sweep.)

6. How it shows up in retrieval. All of this changes future hybrid search mechanically: the P1 memory ranks higher via the priority signal, any new edges feed the opt-in graph signal, and reflections let one high-level memory stand in for many raw ones. These are deterministic ranking effects — Engrava makes no claim that dreaming improves retrieval accuracy on any benchmark (see Known Limitations).

The rest of this page is the knob-by-knob reference for each phase.

Quick start

from engrava.config import DreamingConfig, DreamingGates
from engrava.extensions.dreaming import DreamingExtension

config = DreamingConfig(
    enabled=True,
    promote_threshold=0.55,
    gates=DreamingGates(
        allow_zero_confirmation=True,
        min_age_cycles=1,
    ),
)
ext = DreamingExtension(config=config)

# After ingesting thoughts into `store`:
result = await ext.run_consolidation(store, current_cycle=1)
print(f"Promoted {result.promoted_count} thoughts")

Attaching a hand-built extension to a store

Calling run_consolidation() on the extension directly (above) never needs the store to know about it. If you instead want store.consolidate() to run your hand-built extension — the same call site used by an from_config-built store — wire it on with attach_dreaming_extension():

store.attach_dreaming_extension(ext)
result = await store.consolidate(current_cycle=2)

This is the supported way to wire an extension onto a manually built store; it does not itself construct or configure a DreamingExtension. Calling it again replaces whatever was attached before — there is no separate error for a second call, and no way to detach through this method. attach_dreaming_extension() checks the argument against the runtime_checkable DreamingConsolidatorProtocol (which looks for a run_consolidation member) and raises TypeError if that check fails. It does not check the method's signature or that it is a coroutine function, so a same-named but wrong-shaped run_consolidation (sync instead of async, or a different signature) passes this check and fails only once consolidate() actually calls it.

From YAML config

If you enable dreaming in engrava.yaml (extensions.dreaming.enabled: true) and build the store with SqliteEngravaCore.from_config(...), you do not need to construct a DreamingExtension by hand — call consolidate() on the store directly:

from engrava import SqliteEngravaCore

async with await SqliteEngravaCore.from_config("engrava.yaml") as store:
    result = await store.consolidate(current_cycle=1)
    print(f"Promoted {result.promoted_count} thoughts")

store.consolidate(current_cycle=...) first flushes any pending access events (so the frequency signal sees the latest access counts this cycle), then runs the configured extension. It raises RuntimeError if dreaming is not enabled/wired on the store (built manually, or dreaming.enabled is false) — there is no extension to run.

schedule_every_n_cycles does not start a background scheduler. Applications with a cycle loop can call DreamingExtension.run_if_due(store, current_cycle); it runs only on positive cycle multiples while Dreaming is enabled and returns None otherwise. is_due(current_cycle) exposes the same decision without a write. Explicit store.consolidate() and run_consolidation() calls remain unconditional.

Gates

A thought is scored against the promotion threshold regardless; it must pass all active gates to be promoted. Gates are cheap boolean checks that filter out clearly ineligible candidates before a passing score can result in a promotion.

allow_zero_confirmation

When True (default), the confirmation gate is skipped entirely. This is essential for single-write batch-ingest scenarios where thoughts are stored once and never confirmed — without this flag, no thought would ever pass the confirmation gate and dreaming would be effectively dead.

Set to False only when your application explicitly tracks confirmations and you want to require at least min_confirmations experience-based validations before a thought is eligible for promotion.

Eligibility filters and corpus caps

After the age/confirmation gates, promotion and REFLECTION members also pass metadata-aware eligibility filters. They read caller-provided metadata and are not identity or authorization controls.

Missing perspective, is_self, or content-type annotations remain eligible for backward compatibility. Once a non-empty metadata mapping exists, missing or malformed source confidence is treated as low, so a higher configured threshold rejects it. A record with absent or empty metadata bypasses every metadata-driven filter, including the confidence threshold.

The remaining caps operate at different stages:

• candidates_limit (default 200) bounds the ACTIVE promotion query and each type query used by agglomerative clustering;

• max_promoted_per_run (default 20) limits writes in one promotion phase;

• max_p1_fraction (default 0.05) limits the corpus-wide P1 population;

• min_cluster_size / max_cluster_size (defaults 3 / 200) reject clusters that are too small or too broad — but not symmetrically: max_cluster_size is applied once, to the raw cluster, before eligibility filtering; only min_cluster_size is re-checked afterward, against the eligible member count. A raw cluster over max_cluster_size is rejected even if filtering would have brought it back under the cap;

• clustering_min_new_candidates (default 50) skips repeat clustering when the eligible ACTIVE population has not grown enough. It does not skip signal scoring, promotion, or edge creation.

Signals

Signals compute a score in [0.0, 1.0] for each candidate thought. The candidate's promotion score is a weighted sum over the signals active for this run — an inactive signal's weight is set to zero (see below) — and that score is compared against promote_threshold.

Which signals are active is decided once per run, over the whole candidate pool:

• recency and staleness are always active: a consolidation run always has a cycle.

• confirmation is active when some candidate has confirmation_count > 0.

• confidence is active when some candidate has a confidence value.

• action_outcome is active when some candidate has an action_outcome_score.

• frequency is active when access_tracking_enabled is true and some candidate has access_count > 0 (see Access tracking below).

• A custom signal registered via custom_signals under a new name is always active. One that reuses a default name follows that name's rule.

An inactive signal's weight is set to 0.0, and each active signal's weight is divided by the sum of the active weights. If no signal is active, or that sum is zero, every weight is 0.0 and nothing is promoted. The default weights sum to more than 1.0: they are relative priorities, not a probability distribution.

Custom signals can be provided via DreamingSignalProtocol. custom_signals only supplies the callable for a name — the name itself must also carry a weight in config.signals, or it is never resolved:

from engrava import ThoughtRecord, DreamingContext
from engrava.config import DreamingConfig
from engrava.extensions.dreaming import DreamingExtension


class MySignal:
    def __call__(self, thought: ThoughtRecord, ctx: DreamingContext) -> float:
        return 0.42


config = DreamingConfig(enabled=True)
config.signals["my_signal"] = 0.30  # required: registers the name and its weight
ext = DreamingExtension(
    config=config,
    custom_signals={"my_signal": MySignal()},
)

Access tracking (the frequency substrate)

The frequency signal scores a thought by its access_count. For that to be meaningful, reads have to be counted — otherwise access_count stays at 0 and frequency contributes nothing.

When you build the store via from_config with dreaming enabled, retrieval paths record access automatically, controlled by extensions.dreaming.access_tracking_enabled (default true). To keep the read path fast, access events are buffered and flushed at the cycle boundary rather than written on every read; store.consolidate() flushes the buffer before scoring so the current cycle sees up-to-date counts. Access tracking is deliberately not journaled — the counts are regenerable telemetry, not part of the tamper-evident chain (see Audit Trail).

Set access_tracking_enabled: false to stop counting new accesses. With it false, the frequency signal is always dropped and its weight redistributed over the remaining active signals — even on a store where candidates already carry a non-zero access_count from before tracking was turned off, since access_tracking_enabled gates the signal on its own, independent of any stored value.

Priority signal in search

After dreaming promotes thoughts to P1, the hybrid search search_hybrid() can use priority as a 4th scoring signal alongside FTS5, vector similarity, and recency.

The priority signal maps each thought's Priority enum to a boost multiplier:

The default priority weight is 0.05 (5% of the total score). Configure it via SearchConfig:

search:
  default_priority_weight: 0.05
  priority_boost_p1: 1.0
  priority_boost_p2: 0.6
  priority_boost_p3: 0.3
  priority_boost_p4: 0.0

To disable the priority signal entirely, set default_priority_weight: 0.0.

Edge creation

When dreaming promotes a thought to P1, it can also create ASSOCIATED edges connecting the promoted thought to its nearest neighbours by embedding similarity. This persists the dream's structural knowledge in the graph so it survives application restarts.

Edges are created with source=KnowledgeSource.DREAMING for attribution.

Configuration

extensions:
  dreaming:
    edges:
      enabled: true          # create edges on promotion (default: true)
      top_k: 1               # max neighbours per promoted thought
      min_similarity: 0.7    # cosine threshold for edge creation
      edge_weight_factor: 0.5  # edge.weight = factor × similarity

Idempotence

Re-running run_consolidation() on the same data does not create duplicate edges. Before creating an edge, the extension checks whether the promoted thought already has any edge connecting it to the candidate neighbour (regardless of type).

Edge weight formula

edge.weight = edge_weight_factor × cosine_similarity

With the default edge_weight_factor=0.5 and min_similarity=0.7, practical edge weights range from 0.35 to 0.50.

Graph-aware search

After dream-created edges exist in the graph, search_hybrid() can use them as a 5th scoring signal (graph signal). The signal uses 1-hop-weighted neighbour boost: if a candidate thought's graph neighbours also match the query, the candidate receives a boost proportional to the neighbour's score and the connecting edge weight.

Algorithm

candidate_scores = { C: max(fts[C], vector[C]) for C in fusion pool }  # pool members only
For each candidate C in the fusion pool:
  neighbours = get_edges(C, direction="BOTH", limit=max_neighbors)
  For each (edge, neighbour):
    neighbour_base = candidate_scores.get(neighbour, 0.0)  # 0 if neighbour is off-pool
    boost[C] += edge.weight × neighbour_base × graph_edge_decay
final_score[C] += graph_weight × boost[C]

A neighbour outside the fusion pool contributes 0, not its own fts/vector score — only pool members' base scores propagate.

Configuration

search:
  default_graph_weight: 0.0     # opt-in (0.0 = disabled)
  graph_edge_decay: 0.5         # decay factor for 1-hop distance
  max_neighbors_per_candidate: 5  # safety cap

Per-query override:

result = await store.search_hybrid(
    "python async",
    graph_weight=0.1,
    graph_edge_decay=0.3,
)

The graph signal is opt-in in v0.3.0 (default_graph_weight=0.0). When the weight is 0.0, the graph ranking signal is off and costs nothing. Candidate-pool expansion over CONSOLIDATED_FROM edges is controlled separately by graph_expansion_enabled (default true) and reads those edges only when a reflection ranks among the top candidates.

Complete YAML surface

This example names every YAML-configurable DreamingConfig, DreamingGates, and EdgeCreationConfig field with its default. It is a reference, not a recommended production profile:

extensions:
  dreaming:
    enabled: false
    schedule_every_n_cycles: 100   # used by is_due() / run_if_due()
    promote_threshold: 0.7
    candidates_limit: 200
    signals:
      recency: 0.25
      staleness: 0.20
      confirmation: 0.20
      confidence: 0.15
      frequency: 0.20
      action_outcome: 0.15

    clustering_backend: numpy      # numpy or python
    top_keyphrases_count: 3
    top_member_excerpts_count: 5
    member_excerpt_max_chars: 150
    max_p1_fraction: 0.05
    promote_targets: OBS_ONLY      # OBS_ONLY, REFL_ONLY, or ALL
    reflection_default_priority: P2

    eligible_perspectives: null    # optional list: percept, utterance, thought
    self_filter_mode: any          # any, self_only, external_only
    min_source_confidence: low     # low, medium, high
    excluded_content_types: [code]
    eligible_content_types: null

    boilerplate_threshold: 0.30
    boilerplate_min_corpus_size: 5
    boilerplate_min_keyphrases_per_refl: 1
    access_tracking_enabled: true

    gates:
      min_confirmations: 2
      min_age_cycles: 1
      max_promoted_per_run: 20
      allow_zero_confirmation: true

      min_cluster_size: 3
      cluster_similarity_threshold: 0.7
      cluster_algorithm: lpa       # lpa or agglomerative
      enable_reflections: true
      cold_start_clustering: false
      cluster_allowed_types: [OBSERVATION]
      clustering_min_new_candidates: 50
      max_cluster_size: 200        # null disables the upper bound

      cluster_quality_gating_enabled: true
      cluster_quality_persona_threshold: 0.75
      cluster_quality_cohesion_threshold: 0.40
      cluster_quality_external_homogeneity_threshold: 0.95
      cluster_quality_ne_consistency_threshold: 0.60
      cluster_quality_require_meaningful_keyphrases: true

    edges:
      enabled: true
      top_k: 1
      min_similarity: 0.7
      edge_weight_factor: 0.5

A partial signals mapping merges onto the six defaults. Unknown signal names require a matching custom_signals implementation when constructing DreamingExtension; the YAML-only from_config() path has no custom-signal registry and rejects unknown names when it constructs the extension.

As described in Configuration, unknown keys in the Dreaming, gates, and edge mappings raise ConfigError. See Configuration for field-by-field types and validation ranges.

Reflections (meta-consolidation)

run_consolidation() runs a third phase after promotion and edge creation: it clusters semantically related thoughts and creates ThoughtType.REFLECTION meta-thoughts that aggregate each cluster.

What is a REFLECTION?

A REFLECTION is a first-class ThoughtRecord that represents a higher-order abstraction over a cluster of related thoughts:

The v2 content payload preserves the legacy member_ids, keywords, and cluster_hash fields and adds:

No LLM is involved. Content, essence, centroid, metadata enrichments, and valid-time extent are derived structurally from member records and vectors; the payload's created_at records the current UTC build time. LLM-generated prose summaries belong in downstream extensions, not in the core graph layer.

Navigating the lineage. The CONSOLIDATED_FROM edges are queryable through dedicated store helpers — consolidated_member_ids(reflection_id), consolidated_source_statuses(reflection_id), and the reverse reflections_consolidated_from(source_id). Use them to walk from a REFLECTION to its sources and back (e.g. for provenance views or orphan detection) instead of querying the edge table directly. See REFLECTION lineage in the API reference.

How clustering works

Two algorithms are available via DreamingGates.cluster_algorithm:

"lpa" (default) — Label Propagation Algorithm

• Operates over the ASSOCIATED dream-edge graph built in phase 2.

• Deterministic via seeded PRNG (seed=42 by default).

• O(E × iterations), no external dependencies.

• Works when the graph is dense enough to form communities.

"agglomerative" — cosine-similarity single-linkage

• Operates over bounded ACTIVE candidates whose thought types are listed in cluster_allowed_types (default: OBSERVATION), independent of graph edges.

• Intended for sparse-graph / first-run scenarios where LPA finds no clusters.

• Nodes whose cosine similarity ≥ cluster_similarity_threshold are merged via Union-Find.

• Use when you want clustering before dreams have built up a graph.

clustering_backend selects the similarity implementation for the agglomerative path. numpy (default) uses vectorised float32 matrix operations and chunks large candidate sets to bound peak matrix memory. python uses the legacy pure-Python O(n²) loop and is intended only as a debugging escape hatch. It does not change the LPA implementation.

Cold-start fallback (cold_start_clustering, default false)

The "lpa" path reads communities only from the ASSOCIATED dream-edge graph, and those edges appear only after promotions create them. On a fresh or sparse graph the edge set is empty, so LPA finds nothing and no REFLECTIONs are produced no matter how many eligible OBSERVATIONs exist.

Set cold_start_clustering: true to opt into a fallback: when the LPA path finds the edge graph empty, it falls back within the same cycle to the cosine-similarity agglomerative clustering described above, run over the eligible ACTIVE candidate pool. This lets dreaming form clusters (and REFLECTIONs) before any dream edges exist.

• Default is false — the shipped "lpa" behaviour is unchanged, and the fallback never alters cluster output when edges are present.

• The fallback reuses the exact agglomerative path, so its clusters flow through the same min_cluster_size / max_cluster_size size gates and the content-quality gates — nothing bypasses them.

• The candidate pool is bounded by candidates_limit, so a large active set does not make the fallback unbounded.

Idempotence

Before creating a REFLECTION, the extension derives a 16-hex content-hash from the sorted, eligibility-filtered member IDs — the same subset the metadata-aware eligibility filter above narrows the raw cluster down to, not the raw cluster itself — and checks whether any REFLECTION with source = "dreaming:<hash>" already exists via SELECT ... WHERE thought_type = ? AND source = ? LIMIT 1. This is not an O(1) index lookup: the schema indexes thought(thought_type) but not thought(source), so EXPLAIN QUERY PLAN shows the query using idx_thought_type and then scanning every matching row for the source filter — O(number of REFLECTIONs) per cluster per run, not O(1). If found, the cluster is skipped. Because the hash is over the filtered subset, the same raw cluster scanned under a different eligibility configuration can legitimately hash differently and yield a new REFLECTION.

Re-running run_consolidation() on unchanged data creates zero duplicate REFLECTIONs.

Recovery from a failed embedding

REFLECTION creation embeds the synthesis text itself (when the store has auto-embed configured) before the extension stores its own centroid vector. If that embedding attempt fails — a transient provider timeout, a rate limit — the extension attempts to delete the reflection thought it had already inserted, rather than rolling the transaction back; no centroid and no CONSOLIDATED_FROM edges are created for that cluster, because those steps only run after the reflection thought is created successfully. If the deletion attempt itself fails, or finds nothing to remove, the reflection thought remains with its content-hash recorded, and the next consolidation pass skips the cluster instead of retrying it. When the deletion succeeds, no thought carrying the cluster's hash remains, so the next pass processes the cluster as new; with journaling enabled, the journal still holds both the insert and the deletion.

Orphan sweep

The fourth consolidation phase retires a REFLECTION whose source cluster has left the active set, so a stale synthesis can't keep resurfacing in search. Two cases retire a REFLECTION, both checked against the same protection rule first:

• All sources left ACTIVE. Every thought the REFLECTION was consolidated from is now ARCHIVED / DONE (or otherwise non-ACTIVE) — the synthesis now summarises nothing live.

• Every source was hard-deleted. Deleting a thought cascades its CONSOLIDATED_FROM edge away with it, so a REFLECTION whose sources were all hard-deleted (rather than archived) is left with zero resolvable edges. This is retired exactly like the all-non-ACTIVE case, but only for a REFLECTION dreaming itself created (source_type is DREAMING) — one with no edges that was not created by dreaming (for example built directly through the API) is left alone, the same defensive guard that has always kept a malformed or legacy zero-edge REFLECTION from being wrongly swept.

Protection applies here too. A pinned REFLECTION, or one at a priority in protected_priorities (default P1), is never retired by this sweep — see Protection. A REFLECTION's priority defaults to reflection_default_priority (P2), so default dreaming is unaffected; pin a synthesis, or raise its priority into the protected set, to keep it past this sweep regardless of its source cluster.

A swept REFLECTION is not Memory Hygiene GC-eligible. Retiring a REFLECTION here is a direct ACTIVE -> ARCHIVED flip, not a Memory Hygiene archival — it does not stamp archived_at_cycle / archived_at, and Memory Hygiene's GC stage only ever reaps a row it archived (selected on archived_at_cycle IS NOT NULL). A swept REFLECTION therefore stops surfacing in search immediately, and is physically removed by any of: an explicit engrava gc, which keeps it if it is protected unless --include-pinned is given; an explicit delete, whatever its protection; or TTL cleanup under the delete strategy, if it carries an expired TTL and is not pinned (protection is a hygiene policy, not a TTL exemption) — see Data lifecycle → Running cleanup.

Configuration

extensions:
  dreaming:
    gates:
      min_cluster_size: 3             # min members for a reflection to be created
      cluster_similarity_threshold: 0.7  # cosine threshold (agglomerative only)
      cluster_algorithm: lpa          # "lpa" or "agglomerative"
      enable_reflections: true        # set to false to skip phase 3 entirely
      cold_start_clustering: false    # opt-in: LPA falls back to agglomerative
                                      #   clustering when the edge graph is empty
      cluster_allowed_types: [OBSERVATION]
      clustering_min_new_candidates: 50
      max_cluster_size: 200

Cluster-quality gates

After clustering, metadata eligibility is applied again to the resolved members. The cluster must still contain at least min_cluster_size eligible members. max_cluster_size is not re-checked here — it was already enforced once, against the raw (pre-filtering) cluster in _build_clusters — so an eligible-member count under the cap does not rescue a raw cluster that exceeded it, and no upper bound applies to the eligible subset itself.

With cluster_quality_gating_enabled: true (default), a cluster is rejected on the first failed content-quality check:

• duplicate member content;

• persona-only share at or above cluster_quality_persona_threshold (0.75);

• contradictions flagged by the lightweight English token-pair heuristic;

• mean pairwise cosine below cluster_quality_cohesion_threshold (0.40);

• external-source fraction below cluster_quality_external_homogeneity_threshold (0.95);

• named-entity overlap below cluster_quality_ne_consistency_threshold (0.60); or

• no meaningful post-boilerplate keyphrase when cluster_quality_require_meaningful_keyphrases is true.

Disabling the master switch bypasses these content-quality checks, but not cluster size, metadata eligibility, embedding availability, or idempotence. The cross-cluster boilerplate filter is controlled separately by boilerplate_threshold, boilerplate_min_corpus_size, and boilerplate_min_keyphrases_per_refl.

ConsolidationResult fields

Querying reflections

See search.md — "Querying reflections" section.