Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .claude-plugin/marketplace.json
Original file line number Diff line number Diff line change
Expand Up @@ -5,14 +5,14 @@
},
"metadata": {
"description": "Thinloop skills and continuity hooks for Issue-backed discovery, read-only project-status navigation, multi-Issue project DAG decomposition and execution, evidence-backed reengineering and delivery, and approved skill evolution.",
"version": "0.16.1"
"version": "0.17.0"
},
"plugins": [
{
"name": "thinloop",
"source": ".",
"description": "Discovery, Web UIUX, architecture, Next project-status navigation, Project DAG decomposition, Execute READY-wave orchestration, Reengineering gates, QuickDev delivery, maintenance, knowledge, and evolution skills with continuity checks; Project itself remains non-executing.",
"version": "0.16.1"
"version": "0.17.0"
}
]
}
2 changes: 1 addition & 1 deletion .claude-plugin/plugin.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "thinloop",
"version": "0.16.1",
"version": "0.17.0",
"description": "Simplify complex development with Issue-backed discovery, risk-adaptive Web experience and architecture design, non-executing multi-Issue project decomposition into dependency DAGs, bounded READY-wave execution, read-only project-status navigation, project-scale refactoring and cross-stack reimplementation, autonomous single-Issue PR delivery, evidence-backed maintenance, concise experiential knowledge, and human-approved skill evolution.",
"author": {
"name": "mindcarver"
Expand Down
4 changes: 2 additions & 2 deletions .codebuddy-plugin/marketplace.json
Original file line number Diff line number Diff line change
Expand Up @@ -5,14 +5,14 @@
},
"metadata": {
"description": "Thinloop skills and continuity hooks for Issue-backed discovery, read-only project-status navigation, multi-Issue project DAG decomposition and execution, evidence-backed reengineering and delivery, and approved skill evolution.",
"version": "0.16.1"
"version": "0.17.0"
},
"plugins": [
{
"name": "thinloop",
"source": ".",
"description": "Discovery, Web UIUX, architecture, Next project-status navigation, Project DAG decomposition, Execute READY-wave orchestration, Reengineering gates, QuickDev delivery, maintenance, knowledge, and evolution skills with continuity checks; Project itself remains non-executing.",
"version": "0.16.1"
"version": "0.17.0"
}
]
}
2 changes: 1 addition & 1 deletion .codebuddy-plugin/plugin.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "thinloop",
"version": "0.16.1",
"version": "0.17.0",
"description": "Simplify complex development with Issue-backed discovery, risk-adaptive Web experience and architecture design, non-executing multi-Issue project decomposition into dependency DAGs, bounded READY-wave execution, read-only project-status navigation, project-scale refactoring and cross-stack reimplementation, autonomous single-Issue PR delivery, evidence-backed maintenance, concise experiential knowledge, and human-approved skill evolution.",
"author": {
"name": "mindcarver"
Expand Down
2 changes: 1 addition & 1 deletion .codex-plugin/plugin.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "thinloop",
"version": "0.16.1",
"version": "0.17.0",
"description": "Simplify complex development with Issue-backed discovery, risk-adaptive Web experience and architecture design, non-executing multi-Issue project decomposition into dependency DAGs, bounded READY-wave execution, read-only project-status navigation, project-scale refactoring and cross-stack reimplementation, autonomous single-Issue PR delivery, evidence-backed maintenance, concise experiential knowledge, and human-approved skill evolution.",
"author": {
"name": "mindcarver"
Expand Down
2 changes: 1 addition & 1 deletion .zcode-plugin/plugin.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "thinloop",
"version": "0.16.1",
"version": "0.17.0",
"description": "Simplify complex development with Issue-backed discovery, risk-adaptive Web experience and architecture design, non-executing multi-Issue project decomposition into dependency DAGs, bounded READY-wave execution, read-only project-status navigation, project-scale refactoring and cross-stack reimplementation, autonomous single-Issue PR delivery, evidence-backed maintenance, concise experiential knowledge, and human-approved skill evolution.",
"author": {
"name": "mindcarver"
Expand Down
59 changes: 59 additions & 0 deletions EVALUATION.md
Original file line number Diff line number Diff line change
Expand Up @@ -367,3 +367,62 @@ node evals\score-runs.mjs
## 浏览器证据的实现后导入

当前页面评测先完成实现并保存代码快照,随后通过 `rescore.mjs --run <目录> --browser-evidence <文件>` 导入真实浏览器记录。记录绑定 runId、case、condition、代码快照及采集时间,产物必须存在且 SHA-256 匹配;离线重评分重新校验冻结记录与产物,不直接信任保存的通过标记。缺失证据保持 BLOCKED。具体格式及采集命令见 [当前版本评测说明](evals/thinloop/README.md#证据与重评分)。这项修复改善证据绑定与完整性,不代表已经重新测量模型整体收益。


## 2026-09-05 真实仓库交付观察

本次 [Initiative #90](https://github.com/mindcarver/thinloop/issues/90) 已通过真实 GitHub 完成三个交付:
[评分证据 PR #95](https://github.com/mindcarver/thinloop/pull/95)、
[三端更新 PR #96](https://github.com/mindcarver/thinloop/pull/96)、
[完整交付评测 PR #97](https://github.com/mindcarver/thinloop/pull/97)。各自经历 Issue、PR、真实 CI、
独立验收、合并及父 Execute 清理;验收证据分别见
[#95 结论](https://github.com/mindcarver/thinloop/pull/95#issuecomment-5549049918)、
[#96 结论](https://github.com/mindcarver/thinloop/pull/96#issuecomment-5549029298)、
[#97 结论](https://github.com/mindcarver/thinloop/pull/97#issuecomment-5549163114)。

这是本次工程协作的观察性案例,区别于 delivery 套件的模拟 tracker;没有对照组,不能推出通用全链成功率或相对优势。QuickDev 精简的 #94 仍由自己的验收和关闭状态约束,不据上述三个交付提前声明完成。


## 2026-09-05 QuickDev 精简成对实测

固定旧 QuickDev 提交 `b5067a8d8a61b316addeb7b18c345c8e164fa4be` 与候选冻结提交
`193966b3a37d5481a3efce2b180d3cd29138331f`,其余11个 Skill固定为
`370c3233f0c7f86900c90fe1c6a5f2f2f6c11e8a`。应用内 Codex CLI 0.153.0,
命令指定 `gpt-6-astra` / medium。清晰缺陷、CSV恢复、无关dirty保护各两次重复,
每次两臂,共12个真实subject;首pair预检通过后计入总数,没有剔除或补跑样本。

| 指标 | 旧版,6次 | 候选,6次 |
|---|---:|---:|
| 直接隐藏行为、原生测试和范围验收 | 6 PASS | 6 PASS |
| 完整入口实际读取证明 | 6/6 | 6/6 |
| input tokens合计 | 954203 | 679504 |
| output tokens合计 | 6092 | 6365 |
| cached input tokens合计 | 810624 | 586112 |
| cache-write input tokens合计(明确报告) | 0 | 0 |
| reasoning output tokens合计(CLI报告) | 89 | 61 |
| subject进程时长合计,ms | 874363 | 855040 |
| 已知command_execution事件 | 43 | 47 |
| 已知完成工具项 | 51 | 57 |
| description bytes / Unicode字符 | 934 / 412 | 346 / 164 |
| 入口 bytes / Unicode字符 | 18586 / 7936 | 6066 / 2854 |
| QuickDev总载荷 bytes / Unicode字符 | 40759 / 17884 | 32825 / 14736 |

本组输入量差为 `954203 - 679504 = 274699`,占旧版合计 `28.79%`;输出量、
命令和工具项反而增加,不能据此宣称全面提速或成本降低。未提供单价,成本为null。
两臂各6次的用户输入计数与完成声明分类都保持unknown,不包装成零次干预或零次
无证据声明;所有直接行为验收均有独立于最终措辞的测试与隐藏结果。

每对两臂并发,只交错启动顺序,不是串行A/B与B/A顺序平衡。12次均记录WebSocket
超时后回退HTTPS,因此时长含传输延迟、工具事件覆盖标为partial。全部入口通过
当前识别器支持的完整stdout读取得到证明;识别器未覆盖所有分段sed/Python读法,
这轮未因此丢弃样本。协议显式调用并额外要求完整读取入口,不衡量自动路由或普通
自动注入路径的Token开销;本地授权禁止外部Issue/PR/提交,未直接测紧凑Issue创建
或模型自主GitHub全链收益。三个小任务各两次不能推出统计显著性或通用节省。

[逐样本指标与固定载荷hash](evals/compact/results/issue94.json) 和
[运行方法](evals/compact/README.md) 可复核。完整脱敏轨迹、命令事件、安装文件hash、
原生/隐藏结果、diff及报告保存在
`/Users/carver/workspace/mindcarver/test/thinloop-compact-eval/20260905-issue94/`;
`evidence.sha256`覆盖168个文件,`replay.json`记录12/12离线直接重放PASS且代码快照未改变。
临时认证已清理,秘密扫描0项;候选QuickDev载荷SHA-256为
`3c5b8450c60c0c2e1d6134e2d20834242def5a4a2a3cd00de4d8ed97b6494e53`,最终Skill与该冻结载荷一致。
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@
</p>

<p align="center">
<kbd>v0.16.1</kbd>
<kbd>v0.17.0</kbd>
&nbsp;
<kbd>ISSUE-DRIVEN</kbd>
&nbsp;
Expand Down
5 changes: 5 additions & 0 deletions docs/installation.md
Original file line number Diff line number Diff line change
Expand Up @@ -301,6 +301,11 @@ codebuddy plugin update thinloop@thinloop --scope user
输出未解决状态交接并结束自动纠正;DSH 对同一 Agent 的未改变错误状态不重复 steer,
状态改变或恢复后可重新纠正。更新后新建会话使新 Hook 生效,版本与验证范围见
[`docs/releases/v0.16.1.md`](./releases/v0.16.1.md)。WorkBuddy 维持 SKIP;ZCode 后续交付按三端规则核验。
- 升级到 v0.17.0:QuickDev 按当前阶段读取 Issue、证据、页面和发布参考;清晰低风险
单交付使用五节紧凑 Issue,复杂 PRD/UI/迁移按需展开。独立验收、三本账、真实
页面证据、高风险确认、main 核验和精确清理门继续有效。评分器保留 unknown,
完整交付协议与真实模型证据分开记录;每次交付核对 ZCode、Claude Code、Codex,
同版本载荷漂移也需修复。范围和测量限制见 [`v0.17.0`](./releases/v0.17.0.md)。
- 若从 v0.6.x 升级,另确认旧 `scd-dev-loop` 已消失。

更新后可以在 Thinloop 源码仓库运行只读检查器:
Expand Down
18 changes: 18 additions & 0 deletions docs/releases/v0.17.0.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
# Thinloop v0.17.0

## 范围

- QuickDev 主入口与描述精简,按当前阶段加载参考;清晰低风险单交付使用五节 Issue,PRD、重要页面/跨层设计、迁移和复杂边界按需展开。中文输出、用户主动确认偏好、READY 范围、独立验收、三本账、真实页面门、高风险批准、main 验证与精确清理仍为契约。
- 完成声明评分识别有限范围的明确声明,无法可靠判断时保持 unknown;用户输入事件按结构化轨迹计数,证据覆盖不全不推测为零。
- 新增完整交付协议回归,以及独立实现/验收 Agent 的最小真实模型试验;真实 Git 与模拟 Issue/PR tracker 明确分开,不冒充全模型 GitHub 交付。
- 每次交付核对已安装的 ZCode、Claude Code 和 Codex。ZCode 从 SKIP 改为原生 CLI 的版本、完整技能载荷和运行时 Hook 检查;WorkBuddy 继续 SKIP。更新器使用原生 CLI/API,Claude 同版本内容漂移可通过保留数据的原生重装修复。

## 验证边界

- 本版本组装后263/263仓库测试、版本11表面、路由/评测定义、dry-run、图表、严格插件验证及完整交付协议通过。
- QuickDev三任务×两重复×两臂的12次真实subject均通过直接验收;入口完整读取12/12。样本输入Token合计954203→679504,输出6092→6365,已知工具项51→57;这是特定显式读取协议的有限观察,不是普遍成本或速度结论。


完整工程检查、成对实测和本次真实仓库交付观察在 [EVALUATION.md](../../EVALUATION.md) 中分别记录。显式调用后的三个局部 fixture 比较不衡量自动路由召回,也不衡量完整模型自主 GitHub 交付成功率。没有单价依据,成本保持 null;有限样本不能推出统计显著性或整体节省。

合并后核验三端加载的版本与载荷;新任务/会话才能可靠获得新指令。本次不升级宿主、不改变认证、不部署生产,也不自动创建发布 Tag。
26 changes: 26 additions & 0 deletions evals/compact/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
# QuickDev 精简成对评测

只比较显式调用后的局部行为,不测自然语言自动路由、不测模型自主操作完整 GitHub 交付链。
冻结方案见 [Issue94](https://github.com/mindcarver/thinloop/issues/94):三类 fixture、两次重复、两种 QuickDev 载荷,共12个真实 subject。

- `definition.json` 固定旧 QuickDev 提交、其余11个 Skill的公共提交、CLI/model/reasoning、同一局部授权 prompt 和交错顺序。只替换 QuickDev 载荷。
- 候选先提交,`--candidate` 必须是完整 SHA。全部文件逐项 SHA-256;入口与总载荷 bytes/chars 和真实 Token 分开记录。
- 每对最多两个并发进程,只交错启动顺序;这不是串行 A/B 与 B/A 的顺序平衡。Subject的 HOME、仓库、轨迹目录只用随机ID,条件标签保留在外部 manifest/results,不写入模型上下文。
- 隔离 HOME 只临时复制现有 Codex 认证,复用现有 runner 的 `--ignore-user-config`、禁工具网络/浏览器/多Agent和workspace-write;不改全局配置、安装或认证。临时认证在finally清理。
- 两臂均明确授权本地fixture实施,禁止外部Issue/PR、分支/工作树、提交和其他Agent。这是用户批准的局部例外,不能把结果当成完整外部交付契约的成功率。

```sh
node evals/compact/run.mjs --mode dry --candidate <frozen-commit-sha>
# PATH应选择已冻结的Codex CLI 0.153.0;不安装/升级CLI。
node evals/compact/run.mjs --candidate <frozen-commit-sha> --pairs 1 --output /absolute/evidence/directory
# 首pair验证格式、实际完整读取QuickDev、usage和直接行为结果,再继续同一归档:
node evals/compact/run.mjs --candidate <same-sha> --pairs 6 --output /same/evidence/directory
```

续跑只读取完整配对,不重跑已保存样本;若定义或载荷变化,原目录拒绝继续,必须保留原始失败记录并新建归档。出现基础设施BLOCKED,或baseline通过而candidate失败,停止后续配对,先诊断并修复/回退相关精简后重测。

生成汇总:`node evals/compact/report.mjs /absolute/evidence/directory`。正式归档的 `evidence.sha256` 可用 `shasum -a 256 -c evidence.sha256` 核对;重新生成报告后需要重新封存对应hash,不能把旧清单当新报告的证明。

`observations/`保存原生测试、隐藏行为、范围/脏改动、恢复状态和已修复评分器结果;`traces/`保存完整脱敏命令轨迹和最终回答、安装文件hash及diff;`results/`记录完整入口实际stdout读取证明、真实usage、已知工具项/command_execution事件次数和unknown覆盖(单条shell内多个子命令仍是一条事件)。可用`evidence.json`逐文件核对归档。

未报告的cache/reasoning字段保持null;known工具计数不把未知事件推算为零。完成声明评分只识别明确整体声明,其他情况为unknown。没有价格来源,成本始终null。三个小fixture各两次不能支持统计显著性、通用节省比例或完整交付增益。原生Git/模拟tracker交付回归另见 [delivery](../delivery/README.md)。
15 changes: 15 additions & 0 deletions evals/compact/definition.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
{
"schemaVersion": 1,
"id": "issue94-quickdev-paired-v1",
"baselineQuickdevRef": "b5067a8d8a61b316addeb7b18c345c8e164fa4be",
"sharedSkillsRef": "370c3233f0c7f86900c90fe1c6a5f2f2f6c11e8a",
"model": "gpt-6-astra",
"reasoning": "medium",
"cliVersion": "codex-cli 0.153.0",
"repeats": 2,
"concurrency": 2,
"caseIds": ["clear-local-bug", "resume-interrupted-export", "preserve-dirty-readme"],
"promptPrefix": "请显式使用 $scd-quickdev,先完整读取 $CODEX_HOME/skills/scd-quickdev/SKILL.md,再按当前阶段需要读取参考。当前是用户已明确批准的隔离本地基准任务:本消息与仓库已有状态共同界定交付范围,授权直接在当前工作树实施;不要创建外部 Issue/PR,不创建分支或工作树,不提交或合并,不启动其他 Agent。恢复记录里的 Issue 地址仅为已有任务标识,本次不要求联网复核或重建 Issue。该本地授权覆盖技能中的外部交付流程。请完成下面的局部行为并运行直接测试,最后准确说明结果与未验证边界。",
"scope": "explicit-skill local behavior only; no real GitHub delivery; shared delivery protocol is a separate regression",
"priceUsd": null
}
Loading