Repository navigation
LiveMath scores imply 125 test items, but the released split has 124 #157
Description
Hi SkillOpt team,
I found a possible mismatch between the LiveMath split and the Qwen results in Table 1.
The released manifest contains 35/18/124 train/val/test items. However, several reported scores exactly match a denominator of 125:
- Qwen3.5-4B: No skill
22.4 = 28/125, Trace2Skill27.2 = 34/125, SkillOpt52.0 = 65/125 - Qwen3.6-35B-A3B: No skill
31.2 = 39/125, Trace2Skill29.6 = 37/125, SkillOpt41.6 = 52/125
With 124 binary exact-match examples, neither 52.0% nor 41.6% is attainable after rounding.
For 177 total items, a 2:1:7 split can be either:
35/17/125if train and val are floored and the remainder goes to test;35/18/124with the currently released largest-remainder split.
Could you confirm whether Table 1 used the former 35/17/125 split, and release the original LiveMath manifest or per-item results?
Related: #108.
Thanks!
Activity
raytrun commentedon Jul 26, 2026
Thanks for catching this and for the careful analysis!
We checked the issue and found that one sample from the original test set was mistakenly included in the validation set during the data split, resulting in the released 35/18/124 split instead of the intended 35/17/125 split used for the reported results in Table 1.
We have now fixed the manifest and updated the repository accordingly. The corrected split matches the evaluation used in the paper.
Thanks again for pointing this out—we've updated the repository accordingly.
Yif-Yang commentedon Sep 30, 2026
#292 has now merged after official CI passed. The two stale “124 items” references in the Sleep results page and blog now match the repository manifests: 35 train / 17 validation / 125 test items, independently counted during review.
This synchronizes the documentation with the earlier manifest correction; it does not change the scores or claim a new benchmark run or newly recovered experimental provenance.