Repository navigation
Mining is limited to programmatic success checks — intent-level improvements get dropped or Goodharted into shallow proxy rules #154
Description
Activity
Thanks for describing this failure mode so concretely. A one-off preference being over-generalized into a permanent rule is a real concern, especially when SkillOpt-Sleep is used with transcripts from day-to-day work.
One implementation detail may help align the discussion: the current miner is not limited to tasks that can be expressed as regexes or other programmatic assertions. It prefers locally verifiable rule checks when appropriate, but if no suitable programmatic check exists and it can produce a meaningful free-text rubric, the task is retained with
reference_kind="rubric". During replay, a real backend uses an LLM judge to score that rubric from 0 to 1, with 0.8 or above treated as a hard pass. A task is dropped only when neither usable checks nor a rubric can be formed. So judge-based, non-programmatic evaluation already exists in the current pipeline.That said, this does not eliminate the problem you observed. The miner currently infers a reusable intent and rubric in one pass from a compressed session digest. It has neither divergence-point detection nor clean-context replay comparison, and it cannot reliably distinguish a temporary wording preference from a stable preference. It can therefore still produce a shallow proxy that the optimizer learns to satisfy rather than the underlying intent.
Could you share a sanitized minimal reproduction? The most useful artifacts would be:
- the relevant user correction, assistant final answer, and feedback signals;
- the mined
TaskRecord, especiallyintent,reference_kind, andreference/judge; - the rule that was ultimately written into the skill or memory;
- the backend, model, and relevant configuration.
That would help us determine whether the failure originates in task-boundary detection, rubric construction, replay/judging, or consolidation.
We are interested in improving this area, and we welcome community PRs that explore these ideas. We would be happy to work with contributors on the scope and evaluate an incremental design together. Clean-context replay comparison in particular seems like a promising experiment.
For eventual integration of a new judge mechanism, we would want frozen rubrics with provenance, held-out tasks or sessions, calibration against a small human-labeled set, and an abstain/fail-closed path for ambiguous cases. Programmatic checks should remain preferred where they genuinely fit, with semantic judges complementing rather than unconditionally replacing them. These are longer-term acceptance goals, not requirements for sharing an initial reproduction or opening an early draft PR.
We would prefer to start with a reproducible case and a small, testable change rather than introduce an entirely new pipeline at once.
Adding empirical data from the gate side, since this thread is about the miner and the two halves interact.
Six nights of SkillOpt-Sleep on two of my own skills (an essay skill; a router with 24 candidate targets), real task sets,
azure_openaicompat backend againstgemini-3.1-proandgemini-3.5-flash. I observed the "shallow proxy rule" failure mode three times — and there is a concrete mechanism that makes it likely, beyond the miner's choice of check.The optimizer can read the judge's source text, and writes it into the skill
ReplayResult.fail_reason/judge_rationalecarry the literal failing check ("failed: regex=(?:Бхагавад-гит\w*|...)[^\n]{0,25}?\d+\.\d+"), and those strings reachreflect. The optimizer duly optimizes against the string it can see. Three of the rules it proposed to me quoted my own regex verbatim:OVERRIDE: The output MUST include a scriptural citation formatted so that the response must match the pattern
(?:Бхагавад-гит\w*|Б\.?Г\.?|Шримад-Бхагаватам|...)[^\n]{0,25}?\d+\.\d+.and
[…] you must route it to Prashna to satisfy
regex=(?im)^\s*SKILL:\s*jyoti\-prashna\-util\s*$.The second rule was semantically correct and I ended up adopting it — but only after manually stripping the
to satisfy regex=...tail. Left alone, the skill would permanently carry a reference to my test harness. So the Goodharting here is not only "a preference becomes a rigid rule"; it is "the evaluator's implementation leaks into the artifact".Cheap mitigation that would not change the pipeline: sanitize
fail_reasonbefore it reachesreflect— pass a human description of the unmet criterion rather than the raw pattern, or instruct the optimizer never to quote check syntax. The optimizer would then have to describe the behaviour, which is what you want in a skill anyway.A judge that is subtly wrong is worse than a judge that is absent
The most expensive night I ran was one where the check itself was too strict:
section_presentrejects### 1. Heading (Translation) — subtitle, which is the exact style my skill's own body demonstrates. The optimizer read those failures and proposed forbidding the skill's own heading format; the gate accepted it (0.682 → 0.852), because the gain was real against the broken measurement. Filed as #175.That is the same dynamic you describe, but the trigger was not "no programmatic check existed" — it was "a programmatic check existed and was 5% wrong". Regex-expressible does not imply regex-correct, and the optimizer is adversarial toward whatever slack the judge leaves.
What actually produced usable signal
Relevant to your question about evidence standards for judge-based checks: for me, rubric tasks were not the bottleneck — task provenance was.
Tasks I derived from the skill's own documentation were worthless: the router's routing matrix is inside the prompt, so baseline hit 1.000 with 0 misses over 26 attempts. There was nothing to learn because the answers were in the context. What produced real signal was mining 72 (request → route) pairs out of my saved consultation artifacts, where the actual route chosen in practice sometimes contradicts the documented matrix. Three genuine divergences surfaced that way, and two became rules I adopted after independent validation.
That matches your direction 2 (clean-context replay against what the user ultimately accepted). The ground truth need not be a rubric at all when the durable output of a session records both the request and the resolution — for me those were saved markdown artifacts, but transcripts with an accepted final answer would work the same way.
On evidence standards
Whatever check type is used, I would not trust an accept without per-task attribution. Across my six nights, all three
acceptverdicts were driven by a task the proposed rule did not address (details and numbers in #174). The rules were right twice and wrong once, and the report gave no way to distinguish those cases — I had to re-run targeted tasks with and without each candidate, plus guards for neighbouring behaviours, before adopting anything. If judge-based mining lands, that attribution requirement gets stronger, not weaker, since a rubric score is noisier than a regex.#270 has now merged after official CI passed. It separates raw judge audit evidence from the semantic feedback consumed by reflection, contrastive reflection, slow update, and the contributed OpenClaw path. The synthetic miner→replay→optimizer→proposal regression demonstrates that this path retains useful task-level signal without copying the verifier regex into the proposal.
This is a narrow mitigation, not evidence of general real-model quality gains or a solution to intent-level mining/alignment. The conservative non-rule fallback deliberately loses some diagnostic information. Please keep this broader issue open for the remaining task/intent questions.
ORIGIN & MOTIVATION
I'm exploring using SkillOpt as a step toward "on-the-job training" for AI Employees learning a specific job role from real work sessions. This issue is about what looks like a structural ceiling on that goal in the mining stage. (Related but distinct from #67, which covers reward-hacking on the gate side; this is about the miner side.)
Problem
The task miner only retains a candidate task if success can be expressed as a programmatic check (regex-style assertions). That means behaviors like "understand my intent better" don't compress into a check, so they are either:
I've observed the second failure mode in my own runs: a one-off summary phrasing preference became a rigid rule stamped onto every future summary.
If mining is restricted to regex-expressible outcomes, the system can only ever improve the regex-expressible slice of agent behavior, which is a small subset of what users actually repeat-and-rephrase about in real sessions.
Possible directions (heuristics, not designs)
Question for maintainers
Is there interest in supporting judge-based (non-programmatic) success checks in mining/replay, and what evidence standard would you consider acceptable for gating on them?