Repository navigation
Evon test kit for the Grammar-40 8x5 card - #3
Draft
sameer261h wants to merge 10 commits into
Draft
sameer261h wants to merge 10 commits into
sameer261h wants to merge 10 commits into
Conversation
Builds a plain-text card from the pilot's top-5 parameters for its 8 languages and asks Evon to name the variety of hand-written test transcripts, with and without the card, so the difference is the card's contribution. Brings the pilot's results.json over from the pilot branch as the card's source. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01J5PVR7Y8WgaWAJP9yJ5JAf
…ntrols) Builds the Top-5 table and the teaching prompt only from the Grammar-40 pilot's results.json, and builds the eval set from the 130a39d manifest minus the pilot's 999 training clips, with the gold key in a separate file. The items step currently stops on its own control: there are no Hindi Prisma transcripts outside the pilot's training clips. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01J5PVR7Y8WgaWAJP9yJ5JAf
Hindi has no Prisma transcripts outside the pilot's training clips, so the 755-item eval set covers the other 7 languages while Hindi stays one of the 8 possible answers. The untaught arm sends the same item prompt without the teaching block. Items and the gold key are kept in separate files. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01J5PVR7Y8WgaWAJP9yJ5JAf
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01J5PVR7Y8WgaWAJP9yJ5JAf
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01J5PVR7Y8WgaWAJP9yJ5JAf
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01J5PVR7Y8WgaWAJP9yJ5JAf
…e minutes Without @modal.concurrent the single container took one request at a time, so vLLM never batched and parallel eval calls queued; the 60-minute scaledown window billed an idle A100 for an hour after every run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01J5PVR7Y8WgaWAJP9yJ5JAf
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01J5PVR7Y8WgaWAJP9yJ5JAf
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01J5PVR7Y8WgaWAJP9yJ5JAf
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01J5PVR7Y8WgaWAJP9yJ5JAf
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a small test kit to check whether Evon can use the Grammar-40 pilot's 8x5 result (top-5 grammar features for each of 8 languages) to name the language variety of a transcript.
What's in it
scripts/evon_grammar_card_test.pybuilds a plain-text card from the pilot'sresults.json, then asks Evon to name the variety of each hand-written test transcript in two arms:The two prompts are identical apart from the card, so B minus A is the card's contribution.
--write-cardand--dry-runmake no Evon calls.data/grammar40_pilot/evon_card_8x5.txtis the generated card, for review.data/grammar40_pilot/results.jsonis brought over unchanged from the pilot branch (claude/vigilant-planck-f7m34e) as the card's source.Note on the card
The pilot's present/absent flag applies to a whole feature class. The individual forms listed under a class can point the other way: Garhwali's copula class is flagged "absent", yet छ and छन are Garhwali forms (18 in Garhwali against 12 elsewhere). The card therefore labels each form from its own counts: "more often here" means at least 2× the rate elsewhere, "less often here" means at most 0.5×.
Testing
🤖 Generated with Claude Code
https://claude.ai/code/session_01J5PVR7Y8WgaWAJP9yJ5JAf
Generated by Claude Code