Repository navigation
feat(cli): serve laya-code-r2 by default, with serving settings read from the model - #47
Merged
Merged
Conversation
The benchmark set LAYA_CODEX_MEMO=0 so arms could not read each other's cached probabilities. That also made every follow-up re-score a ranking identical to its first prompt's, about 0.5 s per session that users, whose daemon keeps the cache, never pay. A query may now carry a memo salt. The daemon hashes it into the cache key (keyed by the salt, under its own prefix), so a salted query reads only what was cached under the same salt. Without a salt the key is byte-identical to before, and a test pins it. The hook, MCP server and `laya-codex query` send LAYA_CODEX_MEMO_SALT from their environment with every query; it travels with the request, not the daemon's environment, because one daemon serves every session of an arm (and arms that share a binary share the daemon). run_bench.py keeps the memo on and gives each session attempt its own salt (arm, task, repeat, session id): every first prompt scores cold, the follow-up hits its own session's cache, and no session can read another's. An exported LAYA_CODEX_MEMO=0 still turns cache reads off. author: Tin Dang
The hook's 0.53 s per prompt was only broken down by a debug profiler patch that printed to stderr. Without the split in the capture record, a benchmark cannot tell model time from candidate generation or rendering, nor show that a follow-up was served from the score cache. Each request record now carries `stage_ms` (lexical candidates, the model gate, each forward batch the model ran, related lists, render, total) and `cached` (probabilities served from the score cache). A follow-up served entirely from the cache shows `cached` equal to `scored` and no batches. Scorers report what they spent through a new `Scorer::score_within_stats`, whose default reports nothing, so other scorers are unchanged: the Laya model times each micro-batch and the memo scorer counts its hits. The timers are a few `Instant::now()` calls per request, and records still hold no code text. author: Tin Dang
v14 has a hook-log line spliced from two hooks that ran at once: the line was formatted straight into the file with `writeln!`, which issues one write(2) per JSON token, so concurrent appends interleaved mid-line and the benchmark's time breakdown could not parse it. The line is now serialized in memory and appended with a single write on an O_APPEND file, as the trace and capture writers already do. A test races eight writers on one file; it failed with spliced lines before the change. author: Tin Dang
The plan's outcome is fewer lookup round trips, but the time breakdown only printed per-arm means of its parts: no lookup total, no paired comparison, and no interval to say whether a change is real. It also stopped at v14's spliced hook-log line, and with --repeat it read repeat 0's transcript for every repeat. Each session now has `lookup_s` (hook + lookup round trips + tool execution) next to `lookup_calls`, and every arm is paired with the baseline over the tasks both ran: both means, the mean difference and the change of the ratio of sums, with 95% CIs from a repo-stratified paired bootstrap (the helpers of stats_pooled.py; repeats of a task averaged first). Unparsable hook-log lines are skipped and counted, each repeat reads its own transcript, and the docstring notes that the stream's `ttft_stream_ms` includes the hook. On v14 this gives laya vs stock: lookup time -3.45 s [-4.36, -2.48] (9.05 -> 5.60 s), lookup calls -1.85 [-2.22, -1.48], two unparsable lines. author: Tin Dang
The harness turned characters into tokens at 3.5 characters per token for both the code Claude reads through tools and the code laya-codex injects. Regressing the cache write of the API call that follows each tool result on that result's characters (1,241 calls in benchmarks v12-v14, R^2 0.98), and the follow-up's first cache write on the hook's injected characters (241 sessions), shows Sonnet 5.5 bills 0.42-0.46 tokens per character of code: the estimate counted both sides 1.5-1.6x low. The rates differ by tool (Read 0.424, Grep 0.460, Glob 0.554, laya-codex search 0.445, injected 0.456) and the arms use different tool mixes, so one rate per kind keeps the read + injected comparison unbiased. Each run alone gives the same rates within 0.02. runs.TOKENS_PER_CHAR holds the rates and tok_estimate takes the kind; parse_stream, the hook-log reader, read accuracy's wasted-read tokens, the cost floor's answer tokens and show_session use it. `recost.py tokens` re-estimates committed runs from their raw transcripts and hook logs and keeps the old values as <field>_at_3_5. author: Tin Dang
The published "code read + injected" figures counted code at 3.5 characters per token, 1.5-1.6x fewer tokens than Sonnet 5.5 bills, on both the read and the injected side. The committed runs are re-estimated from their raw transcripts and hook logs with `bench/recost.py tokens` (old values kept per row as *_at_3_5), and every derived file is regenerated with the scripts that produced it: per-repo summaries, the stats tables, headline-v13.json, the v13 cost floor and the README charts. Regenerating each from the old runs first reproduced the committed files byte for byte. Token counts rise 1.5-1.6x; pooled changes move by about one point: v13 read + injected +9.5% -> +10.4% (still n.s.), v14 +49.9% -> +50.9%. The cost floor with today's injection rises from $0.045 (-44%) to $0.053 (-34%). Costs, time, turns and recall do not change. Figures from runs before v12 and the 2026-09-29 pilots stay at the old rate and are marked as such. author: Tin Dang
…ps with A model is calibrated at one state window, and its speed decides how many candidates fit the time budget. Until now both settings applied to every model the binary served: 128 tokens and 16 candidates, changed only through LAYA_CODEX_STATE_TOKENS and LAYA_CODEX_SCORE_TOP. The ModernBERT-base student (laya-code-r2) wants 128 tokens and 12 candidates. A binary that served it with r1's 16, or served r1 with the student's 12, would lose time or quality without saying so. The daemon now reads a `serving` block from the model's rl_agent_config.json: - state_tokens: the state window; - score_top: candidates scored per prompt; - search_score_top: candidates scored per query without a session (the MCP search tool and its lookups). Without the block (laya-code-r1 and older), it serves exactly as before: 128 tokens, 16 per prompt, and the same 16 per search, where the 400 ms lookup budget fits 8. A malformed block is logged and ignored. The environment variables still override the model, for prompts and searches alike. The daemon logs each override that differs from the block. Ping now names the model directory the daemon serves. `doctor` can then tell a daemon started before a model upgrade from one serving the installed model. Older clients ignore the field. author: Tin Dang
…ing upload The ModernBERT-base student, laya-code-r2, replaces laya-code-r1 as the default re-ranker. In the offline replay it inlines 141 of 238 gold files against r1's 132, with the hook at about 167 ms instead of 517 ms (p50). - install.sh downloads it into models/laya-code-r2, a directory of its own, which laya-codex now looks in first, before models/laya-code (where 0.4 and earlier installed r1) and models/laya-base. An existing r1 install keeps working untouched until the new model is there. r1 and r2 files never mix in one directory, where an interrupted upgrade would leave r2's config beside r1's weights. - A first download goes to models/laya-code-r2.part and is renamed into place only once every file has checked out. laya-codex loads the first directory that holds weights, so a half-downloaded model is never picked. A re-run resumes from the staging directory. - The installed revision is written to REVISION, for `doctor`. - --model-only now stops a running daemon of the same LAYA_CODEX_HOME. A daemon keeps the model it loaded at start, so after a model-only upgrade it would keep serving r1. The installer also says that models/laya-code is no longer used and how to free its disk space, without deleting it. - release/hf-laya-code/MANIFEST.sha256 lists r2's seven files, and install.sh pins its sha256 (f5d20914...). - MODEL_REVISION and LAYA_CODE_REVISION hold a placeholder until the owner uploads r2 to the `r2` branch of tindang/laya-code. Until then, scripts/test-install.sh's pin check fails on purpose, as it did while r1 was staged. A new check keeps install.sh's MODEL_NAME and LAYA_CODE_MODEL equal. author: Tin Dang
…rades With laya-code-r2 installed beside an older laya-code, and serving settings read from the model, a user cannot tell from the outside which model ranks their prompts, at what window and how many candidates. Nor can they tell whether a daemon started before the upgrade still serves the old one. The model check now names: - the model directory; - its revision (the REVISION file install.sh writes, else "not recorded"); - the window and candidate counts per prompt and per search, and whether they come from the model's serving block or the defaults; - the engine (Metal or CPU, from laya_model::metal_available instead of the target OS). It warns, with a fix, about: - a model without a serving block. It still works as before, and the fix is the upgrade command (install.sh --model-only, which downloads laya-code-r2 and restarts the daemon); - LAYA_CODEX_STATE_TOKENS or LAYA_CODEX_SCORE_TOP set to something other than the model's block; - a block that cannot be read; - CPU. It names another installed model directory that is no longer used. The daemon check compares the model directory Ping reports with the one this configuration loads, and asks for `laya-codex stop` when they differ. Older daemons do not report one and are not compared. The missing-model hint gives r2's download size (~330 MB). author: Tin Dang
README, docs/how-it-works.md and the changelog still described r1. They named its revision, its 850 MB download, models/laya-code, and 128 tokens and 16 candidates as fixed defaults. They now say: - laya-code-r2 is installed to models/laya-code-r2 and found first, and an r1 install keeps working until `install.sh --model-only` upgrades it; - the window and candidate counts come from the model's serving block, with the two environment variables as overrides; - what `doctor` reports and how to act on its new model and daemon warnings. The changelog records the replay result behind the switch. It counts gold inlined on 120 first prompts (tasks-v8 and held-out), through the real hook with no serving variables set: - r2: 141 of 238, every prompt in laya mode, 12 scored; - r1: 132, 16 scored; - keywords: 118. The hook takes 165 ms (p50) with r2 against 517 ms with r1. A search lookup takes 178 ms against 279 ms. The manual download command carries the same revision placeholder as install.sh, so one search for REPLACE_WITH_HF_R2_REVISION_SHA_AFTER_UPLOAD finds every place the upload commit goes. The benchmark narrative and its diagrams still describe the runs made with r1, which is what they measured. author: Tin Dang
The owner uploads laya-code-r2 to the `r2` branch of tindang/laya-code with the model card and NOTICE from release/hf-laya-code, as was done for r1. The manifest beside them, which install.sh pins, already lists r2's files. README.md is generated by `finetune/model_card.py --run finetune/runs/r2.json` (on feat/laya-code-student). Every figure in it comes from the run manifest: - the encoder revision and the randomly initialised Laya head; - the serving block (128 tokens, 12 candidates per prompt and per search); - the calibration at the serving window; - validation at each training window, against r1 on the same lists; - the default-wiring replay: 141 of 238 gold inlined against 132 for r1 and 118 for keywords, hook 165 ms against 517 ms (p50); - the screen behind the choice of setting. It also says what the evidence is not: no held-out pilot-space list evaluation, no paid benchmark yet, lookups were not in training, and the calibration is only moderate (hard ECE 0.11). NOTICE changes because r2 is not a Laya fine-tune: its weights derive from ModernBERT-base (revision 8949b90, Apache-2.0), and only the decision head's architecture and the unchanged reference code come from Laya. The encoder config and tokenizer files are byte-identical to ModernBERT-base's. The reference code is byte-identical to r1's. author: Tin Dang
laya-code-r2 is published on the r2 branch of tindang/laya-code as commit 831fa8321213ab66a8085d39f0014c5f9f8b5f91. Its served MANIFEST.sha256 matches the committed manifest (f5d20914...a623), and a full download from that commit verifies every file. The installer, the binary's default revision and the docs now name that commit instead of the placeholder. author: Tin Dang
# Conflicts: # crates/laya-cli/src/config.rs # crates/laya-cli/src/daemon.rs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Makes
laya-code-r2laya-codex's default model. It is a ModernBERT-base reranker this project trained with the Laya decision-head method, served at 128 tokens per candidate with 12 candidates scored, for both the prompt hook andsearch.Offline replay through the real hook (120 tasks, 238 gold files, no serving env vars set):
laya120/120laya120/120With the Metal kernel changes from #45, r2's hook runs at about 130 ms.
How it works:
servingblock inrl_agent_config.jsonsetsstate_tokens128,score_top12 andsearch_score_top12.LAYA_CODEX_STATE_TOKENSandLAYA_CODEX_SCORE_TOPstill override the block.searchuses r2: 12 candidates scored within its 400 ms budget, at 178 ms against r1's 279 ms. On the 53 replayed lookups, 48 return the same top file.models/laya-code-r2directory, so existing r1 installs keep working until they're upgraded..part) and moves it into place only after every file verifies.831fa8321213ab66a8085d39f0014c5f9f8b5f91, branchr2oftindang/laya-code. The served manifest hashes tof5d20914…a623, and a full download verifies.--model-onlyrestarts the daemon so the new model loads.Checks: fmt, clippy
-D warnings, workspace tests, andscripts/test-install.sh(pin and manifest) all pass.The run record is in
finetune/runs/r2.json(#44). The paid held-out benchmark that gates the v0.5.0 tag runs after this merges.