core-agent-ide/codex-rs/app-server/tests/common/rollout.rs

203 lines
5.8 KiB
Rust
Raw Normal View History

use anyhow::Result;
use codex_protocol::ThreadId;
use codex_protocol::protocol::GitInfo;
use codex_protocol::protocol::SessionMeta;
use codex_protocol::protocol::SessionMetaLine;
use codex_protocol::protocol::SessionSource;
use serde_json::json;
use std::fs;
feat(app-server, core): return threads by created_at or updated_at (#9247) Add support for returning threads by either `created_at` OR `updated_at` descending. Previously core always returned threads ordered by `created_at`. This PR: - updates core to be able to list threads by `updated_at` OR `created_at` descending based on what the caller wants - also update `thread/list` in app-server to expose this (default to `created_at` if not specified) All existing codepaths (app-server, TUI) still default to `created_at`, so no behavior change is expected with this PR. **Implementation** To sort by `updated_at` is a bit nontrivial (whereas `created_at` is easy due to the way we structure the folders and filenames on disk, which are all based on `created_at`). The most naive way to do this without introducing a cache file or sqlite DB (which we have to implement/maintain) is to scan files in reverse `created_at` order on disk, and look at the file's mtime (last modified timestamp according to the filesystem) until we reach `MAX_SCAN_FILES` (currently set to 10,000). Then, we can return the most recent N threads. Based on some quick and dirty benchmarking on my machine with ~1000 rollout files, calling `thread/list` with limit 50, the `updated_at` path is slower as expected due to all the I/O: - updated-at: average 103.10 ms - created-at: average 41.10 ms Those absolute numbers aren't a big deal IMO, but we can certainly optimize this in a followup if needed by introducing more state stored on disk. **Caveat** There's also a limitation in that any files older than `MAX_SCAN_FILES` will be excluded, which means if a user continues a REALLY old thread, it's possible to not be included. In practice that should not be too big of an issue. If a user makes... - 1000 rollouts/day → threads older than 10 days won't show up - 100 rollouts/day → ~100 days If this becomes a problem for some reason, even more motivation to implement an updated_at cache.
2026-01-16 12:58:55 -08:00
use std::fs::FileTimes;
use std::path::Path;
use std::path::PathBuf;
use uuid::Uuid;
feat(app-server, core): return threads by created_at or updated_at (#9247) Add support for returning threads by either `created_at` OR `updated_at` descending. Previously core always returned threads ordered by `created_at`. This PR: - updates core to be able to list threads by `updated_at` OR `created_at` descending based on what the caller wants - also update `thread/list` in app-server to expose this (default to `created_at` if not specified) All existing codepaths (app-server, TUI) still default to `created_at`, so no behavior change is expected with this PR. **Implementation** To sort by `updated_at` is a bit nontrivial (whereas `created_at` is easy due to the way we structure the folders and filenames on disk, which are all based on `created_at`). The most naive way to do this without introducing a cache file or sqlite DB (which we have to implement/maintain) is to scan files in reverse `created_at` order on disk, and look at the file's mtime (last modified timestamp according to the filesystem) until we reach `MAX_SCAN_FILES` (currently set to 10,000). Then, we can return the most recent N threads. Based on some quick and dirty benchmarking on my machine with ~1000 rollout files, calling `thread/list` with limit 50, the `updated_at` path is slower as expected due to all the I/O: - updated-at: average 103.10 ms - created-at: average 41.10 ms Those absolute numbers aren't a big deal IMO, but we can certainly optimize this in a followup if needed by introducing more state stored on disk. **Caveat** There's also a limitation in that any files older than `MAX_SCAN_FILES` will be excluded, which means if a user continues a REALLY old thread, it's possible to not be included. In practice that should not be too big of an issue. If a user makes... - 1000 rollouts/day → threads older than 10 days won't show up - 100 rollouts/day → ~100 days If this becomes a problem for some reason, even more motivation to implement an updated_at cache.
2026-01-16 12:58:55 -08:00
pub fn rollout_path(codex_home: &Path, filename_ts: &str, thread_id: &str) -> PathBuf {
let year = &filename_ts[0..4];
let month = &filename_ts[5..7];
let day = &filename_ts[8..10];
codex_home
.join("sessions")
.join(year)
.join(month)
.join(day)
.join(format!("rollout-{filename_ts}-{thread_id}.jsonl"))
}
/// Create a minimal rollout file under `CODEX_HOME/sessions/YYYY/MM/DD/`.
///
/// - `filename_ts` is the filename timestamp component in `YYYY-MM-DDThh-mm-ss` format.
/// - `meta_rfc3339` is the envelope timestamp used in JSON lines.
/// - `preview` is the user message preview text.
/// - `model_provider` optionally sets the provider in the session meta payload.
///
/// Returns the generated conversation/session UUID as a string.
pub fn create_fake_rollout(
codex_home: &Path,
filename_ts: &str,
meta_rfc3339: &str,
preview: &str,
model_provider: Option<&str>,
git_info: Option<GitInfo>,
) -> Result<String> {
create_fake_rollout_with_source(
codex_home,
filename_ts,
meta_rfc3339,
preview,
model_provider,
git_info,
SessionSource::Cli,
)
}
/// Create a minimal rollout file with an explicit session source.
pub fn create_fake_rollout_with_source(
codex_home: &Path,
filename_ts: &str,
meta_rfc3339: &str,
preview: &str,
model_provider: Option<&str>,
git_info: Option<GitInfo>,
source: SessionSource,
) -> Result<String> {
let uuid = Uuid::new_v4();
let uuid_str = uuid.to_string();
let conversation_id = ThreadId::from_string(&uuid_str)?;
feat(app-server, core): return threads by created_at or updated_at (#9247) Add support for returning threads by either `created_at` OR `updated_at` descending. Previously core always returned threads ordered by `created_at`. This PR: - updates core to be able to list threads by `updated_at` OR `created_at` descending based on what the caller wants - also update `thread/list` in app-server to expose this (default to `created_at` if not specified) All existing codepaths (app-server, TUI) still default to `created_at`, so no behavior change is expected with this PR. **Implementation** To sort by `updated_at` is a bit nontrivial (whereas `created_at` is easy due to the way we structure the folders and filenames on disk, which are all based on `created_at`). The most naive way to do this without introducing a cache file or sqlite DB (which we have to implement/maintain) is to scan files in reverse `created_at` order on disk, and look at the file's mtime (last modified timestamp according to the filesystem) until we reach `MAX_SCAN_FILES` (currently set to 10,000). Then, we can return the most recent N threads. Based on some quick and dirty benchmarking on my machine with ~1000 rollout files, calling `thread/list` with limit 50, the `updated_at` path is slower as expected due to all the I/O: - updated-at: average 103.10 ms - created-at: average 41.10 ms Those absolute numbers aren't a big deal IMO, but we can certainly optimize this in a followup if needed by introducing more state stored on disk. **Caveat** There's also a limitation in that any files older than `MAX_SCAN_FILES` will be excluded, which means if a user continues a REALLY old thread, it's possible to not be included. In practice that should not be too big of an issue. If a user makes... - 1000 rollouts/day → threads older than 10 days won't show up - 100 rollouts/day → ~100 days If this becomes a problem for some reason, even more motivation to implement an updated_at cache.
2026-01-16 12:58:55 -08:00
let file_path = rollout_path(codex_home, filename_ts, &uuid_str);
let dir = file_path
.parent()
.ok_or_else(|| anyhow::anyhow!("missing rollout parent directory"))?;
fs::create_dir_all(dir)?;
// Build JSONL lines
let meta = SessionMeta {
id: conversation_id,
forked_from_id: None,
timestamp: meta_rfc3339.to_string(),
cwd: PathBuf::from("/"),
originator: "codex".to_string(),
cli_version: "0.0.0".to_string(),
source,
model_provider: model_provider.map(str::to_string),
fix(core) Preserve base_instructions in SessionMeta (#9427) ## Summary This PR consolidates base_instructions onto SessionMeta / SessionConfiguration, so we ensure `base_instructions` is set once per session and should be (mostly) immutable, unless: - overridden by config on resume / fork - sub-agent tasks, like review or collab In a future PR, we should convert all references to `base_instructions` to consistently used the typed struct, so it's less likely that we put other strings there. See #9423. However, this PR is already quite complex, so I'm deferring that to a follow-up. ## Testing - [x] Added a resume test to assert that instructions are preserved. In particular, `resume_switches_models_preserves_base_instructions` fails against main. Existing test coverage thats assert base instructions are preserved across multiple requests in a session: - Manual compact keeps baseline instructions: core/tests/suite/compact.rs:199 - Auto-compact keeps baseline instructions: core/tests/suite/compact.rs:1142 - Prompt caching reuses the same instructions across two requests: core/tests/suite/prompt_caching.rs:150 and core/tests/suite/prompt_caching.rs:157 - Prompt caching with explicit expected string across two requests: core/tests/suite/prompt_caching.rs:213 and core/tests/suite/prompt_caching.rs:222 - Resume with model switch keeps original instructions: core/tests/suite/resume.rs:136 - Compact/resume/fork uses request 0 instructions for later expected payloads: core/tests/suite/compact_resume_fork.rs:215
2026-01-19 21:59:36 -08:00
base_instructions: None,
dynamic_tools: None,
};
let payload = serde_json::to_value(SessionMetaLine {
meta,
git: git_info,
})?;
let lines = [
json!({
"timestamp": meta_rfc3339,
"type": "session_meta",
"payload": payload
})
.to_string(),
json!({
"timestamp": meta_rfc3339,
"type":"response_item",
"payload": {
"type":"message",
"role":"user",
"content":[{"type":"input_text","text": preview}]
}
})
.to_string(),
json!({
"timestamp": meta_rfc3339,
"type":"event_msg",
"payload": {
"type":"user_message",
"message": preview,
"kind": "plain"
}
})
.to_string(),
];
feat(app-server, core): return threads by created_at or updated_at (#9247) Add support for returning threads by either `created_at` OR `updated_at` descending. Previously core always returned threads ordered by `created_at`. This PR: - updates core to be able to list threads by `updated_at` OR `created_at` descending based on what the caller wants - also update `thread/list` in app-server to expose this (default to `created_at` if not specified) All existing codepaths (app-server, TUI) still default to `created_at`, so no behavior change is expected with this PR. **Implementation** To sort by `updated_at` is a bit nontrivial (whereas `created_at` is easy due to the way we structure the folders and filenames on disk, which are all based on `created_at`). The most naive way to do this without introducing a cache file or sqlite DB (which we have to implement/maintain) is to scan files in reverse `created_at` order on disk, and look at the file's mtime (last modified timestamp according to the filesystem) until we reach `MAX_SCAN_FILES` (currently set to 10,000). Then, we can return the most recent N threads. Based on some quick and dirty benchmarking on my machine with ~1000 rollout files, calling `thread/list` with limit 50, the `updated_at` path is slower as expected due to all the I/O: - updated-at: average 103.10 ms - created-at: average 41.10 ms Those absolute numbers aren't a big deal IMO, but we can certainly optimize this in a followup if needed by introducing more state stored on disk. **Caveat** There's also a limitation in that any files older than `MAX_SCAN_FILES` will be excluded, which means if a user continues a REALLY old thread, it's possible to not be included. In practice that should not be too big of an issue. If a user makes... - 1000 rollouts/day → threads older than 10 days won't show up - 100 rollouts/day → ~100 days If this becomes a problem for some reason, even more motivation to implement an updated_at cache.
2026-01-16 12:58:55 -08:00
fs::write(&file_path, lines.join("\n") + "\n")?;
let parsed = chrono::DateTime::parse_from_rfc3339(meta_rfc3339)?.with_timezone(&chrono::Utc);
let times = FileTimes::new().set_modified(parsed.into());
std::fs::OpenOptions::new()
.append(true)
.open(&file_path)?
.set_times(times)?;
Ok(uuid_str)
}
pub fn create_fake_rollout_with_text_elements(
codex_home: &Path,
filename_ts: &str,
meta_rfc3339: &str,
preview: &str,
text_elements: Vec<serde_json::Value>,
model_provider: Option<&str>,
git_info: Option<GitInfo>,
) -> Result<String> {
let uuid = Uuid::new_v4();
let uuid_str = uuid.to_string();
let conversation_id = ThreadId::from_string(&uuid_str)?;
// sessions/YYYY/MM/DD derived from filename_ts (YYYY-MM-DDThh-mm-ss)
let year = &filename_ts[0..4];
let month = &filename_ts[5..7];
let day = &filename_ts[8..10];
let dir = codex_home.join("sessions").join(year).join(month).join(day);
fs::create_dir_all(&dir)?;
let file_path = dir.join(format!("rollout-{filename_ts}-{uuid}.jsonl"));
// Build JSONL lines
let meta = SessionMeta {
id: conversation_id,
forked_from_id: None,
timestamp: meta_rfc3339.to_string(),
cwd: PathBuf::from("/"),
originator: "codex".to_string(),
cli_version: "0.0.0".to_string(),
source: SessionSource::Cli,
model_provider: model_provider.map(str::to_string),
fix(core) Preserve base_instructions in SessionMeta (#9427) ## Summary This PR consolidates base_instructions onto SessionMeta / SessionConfiguration, so we ensure `base_instructions` is set once per session and should be (mostly) immutable, unless: - overridden by config on resume / fork - sub-agent tasks, like review or collab In a future PR, we should convert all references to `base_instructions` to consistently used the typed struct, so it's less likely that we put other strings there. See #9423. However, this PR is already quite complex, so I'm deferring that to a follow-up. ## Testing - [x] Added a resume test to assert that instructions are preserved. In particular, `resume_switches_models_preserves_base_instructions` fails against main. Existing test coverage thats assert base instructions are preserved across multiple requests in a session: - Manual compact keeps baseline instructions: core/tests/suite/compact.rs:199 - Auto-compact keeps baseline instructions: core/tests/suite/compact.rs:1142 - Prompt caching reuses the same instructions across two requests: core/tests/suite/prompt_caching.rs:150 and core/tests/suite/prompt_caching.rs:157 - Prompt caching with explicit expected string across two requests: core/tests/suite/prompt_caching.rs:213 and core/tests/suite/prompt_caching.rs:222 - Resume with model switch keeps original instructions: core/tests/suite/resume.rs:136 - Compact/resume/fork uses request 0 instructions for later expected payloads: core/tests/suite/compact_resume_fork.rs:215
2026-01-19 21:59:36 -08:00
base_instructions: None,
dynamic_tools: None,
};
let payload = serde_json::to_value(SessionMetaLine {
meta,
git: git_info,
})?;
let lines = [
json!( {
"timestamp": meta_rfc3339,
"type": "session_meta",
"payload": payload
})
.to_string(),
json!( {
"timestamp": meta_rfc3339,
"type":"response_item",
"payload": {
"type":"message",
"role":"user",
"content":[{"type":"input_text","text": preview}]
}
})
.to_string(),
json!( {
"timestamp": meta_rfc3339,
"type":"event_msg",
"payload": {
"type":"user_message",
"message": preview,
"text_elements": text_elements,
"local_images": []
}
})
.to_string(),
];
fs::write(file_path, lines.join("\n") + "\n")?;
Ok(uuid_str)
}