Repository navigation
Pricing tiers for candidates, with routing deliberately left on list price - #269
Merged
Merged
Conversation
A single pair of rates cannot describe a provider that charges in steps. Anthropic's Haiku 5.5 is $0.10 and $0.50 per million below a 100,000-token prompt and $0.50 and $2.50 at or above it. Declared flat, a 150,000-token request prices five-fold low, and since RoutingBudget charges whatever ModelCandidate.costOf reports, a budget declared that way stops spending later than it was told to. The demo written for the Models article hit exactly this and had to document the gap rather than express it. `PricingTier` plus `ModelCandidate.pricingTiers` closes it. `tierFor` returns the tier in force and `costOf` applies it. Crossing a threshold reprices the whole request, output included, rather than only the tokens above it. The semantics follow what the implementations actually do, read rather than assumed. LiteLLM keys tiers on prompt tokens, sorts thresholds descending and takes the first match, and repriced requests use the tier rate for every token type; its comparison is exclusive by default with a threshold_is_inclusive flag "for providers such as xAI that bill the higher tier once the prompt reaches the threshold". Ours is inclusive on an integer count, so "above 100,000" is declared as 100_001 and no flag is needed -- the flag exists there because the thresholds are floats. Routing deliberately does not change. `blendedCostPerMillionTokens` still reports list price and the scorer still uses it, because `cheapness` normalises cost across the eligible candidates: a rate that moved with prompt size would make a candidate's score jump at its threshold and its rank non-monotonic in the size of the request. Accounting is where the tier has to be right. weave-os/router draws the same line in a production gateway -- a dispatch is billed at the tier that served it while "routing still scores on list price". Also surveyed, for the record: vllm-semantic-router carries flat per-token prices in config with no tiers, and RouteLLM has no runtime price model at all, scoring on predicted quality with cost appearing only in offline evaluation. Tiers live in the implementations that do the billing, not the ones that do the selecting. Both new tests were mutation-checked: applying a tier to input only, and taking the lowest applicable tier instead of the highest, each fail them. The list is copied with List.copyOf of a sorted copy rather than stream().toList() so the static analysis recognises the defensive copy. test, build, spotlessCheck, complianceCheck and jacocoTestCoverageVerification all pass.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The gap
A
ModelCandidatedeclares one pair of rates, and providers do not all charge one rate. Anthropic's Haiku 5.5 is $0.10/$0.50 per million below a 100,000-token prompt and $0.50/$2.50 at or above it. Declared flat, a 150,000-token request prices five-fold low — and becauseRoutingBudgetcharges whateverModelCandidate.costOfreports, a budget declared that way stops spending later than it was told to.The routing fleet written for the Models deep-dive hit this and could only document it.
What this adds
PricingTierandModelCandidate.pricingTiers, withtierFor(inputTokens)returning the tier in force andcostOfapplying it. Crossing a threshold reprices the whole request, output included, not only the tokens above it.Semantics taken from the implementations, read not assumed
threshold_is_inclusiveflag "for providers such as xAI that bill the higher tier once the prompt reaches the threshold".100_001. The flag exists there because LiteLLM's thresholds are floats; integers make it unnecessary.What deliberately does not change
blendedCostPerMillionTokensstill reports list price, and the scorer still uses it.cheapnessnormalises cost across the eligible candidates, so a rate that moved with prompt size would make a candidate's score jump at its threshold and its rank non-monotonic in request size. Accounting is where the tier must be right.weave-os/router draws the same line in production: a dispatch is billed at the tier that served it while "routing still scores on list price".
For completeness: vllm-semantic-router carries flat per-token prices in config with no tiers, and RouteLLM has no runtime price model at all. Tiers live in the implementations that do the billing, not the ones that do the selecting.
Verification
Both new tests mutation-checked — applying a tier to input only, and taking the lowest applicable tier rather than the highest, each fail them. Backward compatible: the previous arity remains as a constructor overload and a candidate without tiers behaves exactly as before.
test,build,spotlessCheck,complianceCheck,jacocoTestCoverageVerificationpass.Minor version because this adds API. Prices remain the operator's declaration; this library still does not fetch a price map at startup, which is a deployment decision rather than a library one.