Repository navigation
skillopt-sleep Codex harvest ingests its own headless replay sessions #286
Description
Summary
harvest_codex() accepts Codex sessions created by SkillOpt-Sleep itself during headless replay, judging, reflection, and tool-aware attempts. The other transcript harvesters explicitly reject engine-generated sessions, but the Codex source has no equivalent filter.
Reproduction
On main at 79124b37e9a6371e13b753f8bcd7adb1e493ade1, create a Codex archived session using the prompt shape emitted by CodexCliBackend.attempt_with_tools():
Complete the task. Apply the skill and memory rules EXACTLY, including any rule about searching before answering.
# Skill
...
# Memory
...
# Task
...
with a normal user_message and agent_message record. digest_codex_archived_session() returns a SessionDigest, and harvest_codex() includes it.
Expected behavior
Engine-generated Codex sessions should be excluded from harvested user evidence, just as Claude, Copilot CLI, and OpenCode replay/agent sessions are excluded.
Actual behavior
The generated prompt is treated as a real user task. On a later night it can be mined, replayed, and used to update the skill, creating self-training feedback and contaminating the task set.
Impact
Codex-backed runs can learn from their own evaluator/attempt instructions instead of user work, weakening task provenance and potentially reinforcing internal prompt text.
Environment
SkillOpt-Sleep main at 79124b37e9a6371e13b753f8bcd7adb1e493ade1; Python 3.10+; Codex CLI/archived sessions.
Activity
kaluli123123 commentedon Sep 20, 2026
I am working on a focused fix for this provenance bug. I will reuse the existing replay/session filters, add a narrow Codex-specific marker for the prompt shape emitted by SkillOpt's own Codex backend, and cover both exclusion and preservation of a real user prompt.
kaluli123123 commentedon Sep 20, 2026
Implemented in PR #287: #287
Root cause:
harvest_codex()did not distinguish SkillOpt-generated Codex replay prompts from user sessions, so internal headless runs could be harvested as user evidence. The fix reuses the agent-session filter and adds narrow Codex replay markers without applying the generic short-duration heuristic.Evidence:
python3 -m compileall -q skillopt_sleep tests/test_harvest_codex_replay.py: passed.git diff --check: passed.python3 -m pytest -qremains unavailable becausepytestis not installed in the environment.The PR is open; CI and maintainer review remain pending. No merge or Issue closure was performed.
Yif-Yang commentedon Sep 30, 2026
#287 currently excludes the tool-enabled attempt shape, but synthetic transcripts built from the actual current attempt, judge, and reflect prompts are still harvested. The review also found that generic heading markers remove valid user sessions. Keep this issue open until those cases are covered; #282's new CLI session discovery makes the same provenance gap reachable in nested rollout files.
djbclark commentedon Oct 12, 2026
Data point: CLI rollouts reach
harvest_codex()today through a--codex-homesymlink shim (see my comment on #282), so the same provenance gap covers Codex CLI sessions, not only Desktop archives. Until this is fixed we keep thecodexbackend out of our nightly backend chain, so SkillOpt generates no Codex sessions that a later night could harvest.Yif-Yang commentedon Oct 17, 2026
The new #287 has passing evidence for excluding the current Codex engine's actual attempt/judge/reflection/miner/tool/custom-template prompts while keeping ordinary quoted user content. It still needs integration rebasing and official checks. #282's CLI-session discovery and real-repeat deduplication remain separate, and #310's general marker change needs to avoid dropping normal user sessions. This broader issue stays open.