From 2b9d0c385fba4356ddea5bfa5f615f767ce34136 Mon Sep 17 00:00:00 2001 From: jif-oai Date: Mon, 23 Feb 2026 10:52:58 +0000 Subject: [PATCH] chore: add doc to memories (#12565) ] --- codex-rs/core/src/memories/README.md | 92 ++++++++++++++++++++++++++++ 1 file changed, 92 insertions(+) create mode 100644 codex-rs/core/src/memories/README.md diff --git a/codex-rs/core/src/memories/README.md b/codex-rs/core/src/memories/README.md new file mode 100644 index 000000000..c19a0c680 --- /dev/null +++ b/codex-rs/core/src/memories/README.md @@ -0,0 +1,92 @@ +# Memories Pipeline (Core) + +This module runs a startup memory pipeline for eligible sessions. + +## When it runs + +The pipeline is triggered when a root session starts, and only if: + +- the session is not ephemeral +- the memory feature is enabled +- the session is not a sub-agent session +- the state DB is available + +It runs asynchronously in the background and executes two phases in order: Phase 1, then Phase 2. + +## Phase 1: Rollout Extraction (per-thread) + +Phase 1 finds recent eligible rollouts and extracts a structured memory from each one. + +Eligible rollouts are selected from the state DB using startup claim rules. In practice this means +the pipeline only considers rollouts that are: + +- from allowed interactive session sources +- within the configured age window +- idle long enough (to avoid summarizing still-active/fresh rollouts) +- not already owned by another in-flight phase-1 worker +- within startup scan/claim limits (bounded work per startup) + +What it does: + +- claims a bounded set of rollout jobs from the state DB (startup claim) +- filters rollout content down to memory-relevant response items +- sends each rollout to a model (in parallel, with a concurrency cap) +- expects structured output containing: + - a detailed `raw_memory` + - a compact `rollout_summary` + - an optional `rollout_slug` +- redacts secrets from the generated memory fields +- stores successful outputs back into the state DB as stage-1 outputs + +Concurrency / coordination: + +- Phase 1 runs multiple extraction jobs in parallel (with a fixed concurrency cap) so startup memory generation can process several rollouts at once. +- Each job is leased/claimed in the state DB before processing, which prevents duplicate work across concurrent workers/startups. +- Failed jobs are marked with retry backoff, so they are retried later instead of hot-looping. + +Job outcomes: + +- `succeeded` (memory produced) +- `succeeded_no_output` (valid run but nothing useful generated) +- `failed` (with retry backoff/lease handling in DB) + +Phase 1 is the stage that turns individual rollouts into DB-backed memory records. + +## Phase 2: Global Consolidation + +Phase 2 consolidates the latest stage-1 outputs into the filesystem memory artifacts and then runs a dedicated consolidation agent. + +What it does: + +- claims a single global phase-2 job (so only one consolidation runs at a time) +- loads a bounded set of the most recent stage-1 outputs from the state DB (the per-rollout memories produced by Phase 1, used as the consolidation input set) +- computes a completion watermark from the claimed watermark + newest input timestamps +- syncs local memory artifacts under the memories root: + - `raw_memories.md` (merged raw memories, latest first) + - `rollout_summaries/` (one summary file per retained rollout) +- prunes stale rollout summaries that are no longer retained +- if there are no inputs, marks the job successful and exits + +If there is input, it then: + +- spawns an internal consolidation sub-agent +- runs it with no approvals, no network, and local write access only +- disables collab for that agent (to prevent recursive delegation) +- watches the agent status and heartbeats the global job lease while it runs +- marks the phase-2 job success/failure in the state DB when the agent finishes + +Watermark behavior: + +- The global phase-2 job claim includes an input watermark representing the latest input timestamp known when the job was claimed. +- Phase 2 recomputes a `new_watermark` using the max of: + - the claimed watermark + - the newest `source_updated_at` timestamp in the stage-1 inputs it actually loaded +- On success, Phase 2 stores that completion watermark in the DB. +- This lets later phase-2 runs know whether new stage-1 data arrived since the last successful consolidation (dirty vs not dirty), while also avoiding moving the watermark backwards. + +In practice, this phase is responsible for refreshing the on-disk memory workspace and producing/updating the higher-level consolidated memory outputs. + +## Why it is split into two phases + +- Phase 1 scales across many rollouts and produces normalized per-rollout memory records. +- Phase 2 serializes global consolidation so the shared memory artifacts are updated safely and consistently.