BV-SALA: inside the LLM cost layer

The bill of an LLM-powered system doesn't grow with the value it produces: it grows with the tokens it moves. And most of those tokens never needed to move. Context repeated on every turn, declared tools that are never called, questions whose answer was already computed a minute ago. BV-SALA is the layer that sits in front of your provider and, on every request, answers a single question: what is the cheapest execution that still meets your quality contract?

This article explains how we think about that question: the formalization, the three ledgers every saving falls into, and the measurement methodology — which is, deliberately, the strictest part of the system.

The decision, formalized

For a request $x$ (prompt, context, tools), the provider charges a function that is essentially linear in tokens:

where $T$ counts tokens with the provider's own tokenizer — not with an approximation of ours — and $c_{\text{in}}, c_{\text{out}}$ are their current prices. BV-SALA has a catalog $\Pi$ of request transformations: removing redundancy, pruning tools, compressing the encoding, resolving without calling. Choosing well is a constrained optimization problem:

The constraint $Q \ge \tau$ is the quality contract: the cheapest transformation is worthless if it degrades the answer. And here lies the intrinsic difficulty of the problem: $Q$ is not observable before executing. That is why every transformation in the catalog carries its own guard — it is applied only when it pays for itself, with the cost of deciding accounted inside the saving, not hidden outside it.

The three ledgers

Every optimized request is attributed to exactly one ledger, by dominant mechanism. This partition is not cosmetic: it is what makes the numbers mean something.

Every figure in this section comes from a single run, the same one the product serves today: openai/gpt-4o-mini, 193 measured pairs, quality 100%, artifact d1c8233 of 12 September 2026. Each is the floor of the 95% confidence interval of its ledger, on that model; a percentage without its model next to it means nothing.

01 — Avoid the call entirely. Deterministic resolvers for what doesn't need a model, and singleflight to collapse identical concurrent requests into a single execution. The catalog also includes an exact cache, but this ledger's measured saving in the published run comes from the resolvers and from singleflight: what was not measured is not advertised. Ledger floor on gpt-4o-mini: 22.0%.

02 — Remove what didn't need to be sent. Tools pruned from the declared catalog, selective retrieval instead of full context, and elimination of the redundant context that travels turn after turn. Ledger floor on gpt-4o-mini: 43.2%.

03 — Compress what remains. Lighter encoding of what does have to travel — and only when the compression itself pays off. Ledger floor on gpt-4o-mini: 20.1%.

The saving of a ledger $\ell$ over the set $R_\ell$ of requests attributed to it is defined against a control arm — and is measured against both of the arms declared in the next section:

Three direct consequences of the partition. First: these figures do not add up — each speaks over its own denominator, and presenting them aggregated would count the same euro twice; nowhere in this article, in prose or in a table, are they added together. Second: a ledger that doesn't clear zero against both control arms is not advertised. If a transformation doesn't demonstrably save, its figure doesn't exist: today that is the case for output restriction and for the provider-cache ledger, both measured and with no figure to publish. Third: the three ledgers above do clear zero against both arms, and the floor published for each is the one from the comparison against the unoptimized integration.

How we measure: the floor of the interval, not the flattering estimate

The methodology is the part of the system we trust most, because it is the one that leaves the least room for interpretation:

1. Same model, three arms — two of them controls. Every request is run against the real provider and on the same model: as an unoptimized integration would send it (B0), as an integration a diligent team would already have optimized by hand would send it (B1: trimmed tool catalog, pruned context, no frills), and as BV-SALA sends it. We never compare different models or different time periods. The figure against B0 says how far the bill drops for someone starting from scratch; only the figure against B1 measures merit of the layer, because it is the only one that discounts what the customer could already have saved alone. That is why the ceiling on what we may promise is the lower of the two floors, never the higher.
2. Cost quoted with the provider's own meter. No modeled baseline: tokens are counted by the official tokenizer and prices are the provider's. The number we publish is the one you would see on your bill.
3. The estimator is stratified by scenario, with equal weight per scenario. The aggregate is not computed over all samples pooled together: that lets the distribution of repetitions — which the sampler shifts according to noise — decide the headline, and over-repeating one expensive, zero-saving scenario is enough to sink it without the layer having changed anything. Each scenario weighs 1, and the resampling that produces the interval happens within each scenario. Declared caveat: the stratified interval is conditional on the fixed set of scenarios and therefore comes out narrower — it answers a narrower question, not with more confidence.
4. We publish the floor of the 95% confidence interval. With $n$ requests and sample saving $\hat S$, we publish the one-sided lower bound

that is, the value the true saving exceeds with 95% confidence — never the point estimate, which is always more photogenic. The $\widehat{\operatorname{se}}$ comes from the stratified resampling of the previous point.

Held to that yardstick, the current run — openai/gpt-4o-mini, 193 measured pairs, quality 100%, artifact d1c8233 of 12 September 2026 — gives this:

What we publish as the headline, however, is ≥ 20.0% on gpt-4o-mini, below both. The ceiling on what may be promised is the lower of the two floors — B1's, because promising above it would be selling as our merit what the customer already had — and the headline stays deliberately below that ceiling, so that it remains true when the next run moves a decimal. The $\ge$ sign is not rhetoric: it is the lower bound of the interval, and what is promised can never exceed what was measured.

No prefix-caching figure appears here, and the absence is deliberate. When the context prefix repeats, the provider charges less for the repeated part: that discount is theirs — Anthropic's, with its own exactness rules — not ours. Our part is to prepare every request so the discount can be claimed, and to claim it on every one. Publishing it as our own saving would be taking credit for someone else's work; and besides, the ledger that records it, once measured, does not clear zero. So here there is an explanation and no percentage.

Why the problem is harder than it looks

Three sources of intrinsic complexity that shape the whole design:

The constraint is stochastic. $Q(\pi(x)) \ge \tau$ is decided before seeing the response. Aggressive transformations (rewriting, compression) only activate where accumulated evidence indicates they preserve intent; in doubt, the request travels untouched. We would rather save less than degrade silently.

The cost of deciding counts. Analyzing every request also consumes. BV-SALA's entire accounting is net: the published saving discounts the cost of the layer itself. An optimizer that pays itself in the fine print is not an optimizer.

Cache correctness is semantic, not syntactic. An exact cache and a singleflight are only a saving if the reused answer is still valid for the new request. Keys incorporate everything that conditions the response — model, version, visible tools, effective context — and on any difference, we call.

Where it sits in the literature

None of the transformations in BV-SALA's catalog is, on its own, a new idea — and saying so with the citations in hand is part of this article's honesty. The full taxonomy was laid out by FrugalGPT back in 2023: adapt the prompt, approximate the model, cascade (Chen et al., 2023). Since then, each branch has matured on its own. Prompt compression is the deepest line: LLMLingua reports up to 20x compression with little loss (Jiang et al., 2023), LongLLMLingua finds that in long contexts compressing can even improve the answer — the lost in the middle effect — (Jiang et al., 2024), and LLMLingua-2 attacks the operational problem the first generation left open: the compressor itself was slow and expensive (Pan et al., 2024). At the extreme, gist tokens compress inside the model, up to 26x (Mu et al., 2023) — more general than any textual compression, but only if you control the weights; over a closed API, which is the regime of whoever pays the bill, it does not apply. Cost-aware routing matured in parallel: Hybrid LLM introduces the desired quality level as a router parameter (Ding et al., 2024), RouteLLM trains it on preference data (Ong et al., 2024), and AutoMix adds self-verification as the escalation gate — try cheap, verify, and only then pay for the expensive model (Aggarwal et al., 2024). Pruning has its own published evidence too: always retrieving context is suboptimal compared to retrieving on demand (Asai et al., 2023; Jeong et al., 2024), and reducing the visible tools not only cuts cost but improves function-calling accuracy (Paramanayakam et al., 2025).

Caching lives in two worlds that barely talk to each other. Academia pushes semantic caching — GPTCache as infrastructure (Bang, 2023), SCALM as the first evaluation on real human-LLM traces (Li et al., 2024) — which optimizes hit ratio by embedding similarity and tacitly accepts the risk of serving an answer that does not match. Providers, meanwhile, ship prefix caching under strict exactness discipline: Anthropic requires 100% identical segments and invalidates the whole cache if tool definitions change (Anthropic, 2026); OpenAI bills cache writes at 1.25x the base price and reads at 0.1x (OpenAI, 2026). Those two official documents establish something the academic literature rarely incorporates: cache savings have a write cost — so only net accounting is honest — and cache validity depends on model, version and tools. That is exactly BV-SALA's key scheme, promoted from good practice to requirement.

What the field has solved is effectiveness; what it has open is evidence. Almost every published number is a best-case ratio on a benchmark — "up to 20x", "up to 98% cost reduction" (Chen et al., 2023), "up to 40% fewer calls" (Ding et al., 2024) — with no confidence interval, almost never on production traffic, and without deducting the cost of the saving machinery itself. Two recent works point at the correction: cost-of-pass proposes measuring in expected monetary cost per correct solution (Erol et al., 2025), and Miller formalizes the statistical apparatus for comparing two evaluation conditions rigorously (Miller, 2024). No work we could verify joins the two. That crossing — FrugalGPT's taxonomy executed with the providers' key discipline, measured with Miller's statistics in the currency of cost-of-pass — is the gap BV-SALA occupies. It is worth saying plainly: BV-SALA does not claim new transformations; it claims a different class of evidence about known ones.

Open problems

At least five open problems follow from the verified literature, and this layer touches all of them:

1. Net accounting. LLMLingua-2 exists precisely because the first generation's compressor was expensive (Pan et al., 2024), and cache writes are billed at 1.25x-2x the base price (Anthropic, 2026) — yet no published work reports savings that systematically deduct the cost of the compressor, the router or the cache. The field publishes gross savings.
2. Semantic cache validity. Serving a cached answer by similarity without formally bounding false hits or staleness (Bang, 2023; Li et al., 2024) leaves the middle ground open: more coverage than exact caching, with a validity guarantee.
3. Statistical certification of quality. Routers are governed by a desired quality level (Ding et al., 2024; Ong et al., 2024), but compliance is not certified after the fact with two-sample inference (Miller, 2024). Verifying $Q \ge \tau$ contracts on real traffic remains open.
4. Interaction between transformations. OpenAI's minimum cacheable prefix implies that compressing a prefix can destroy a larger cache discount (OpenAI, 2026): transformations are not independent, and their optimal composition is unstudied.
5. From benchmark to production. Only SCALM starts from real traces (Li et al., 2024); everything else is evaluated on static benchmarks, when the repetition and drift of real traffic are precisely what determine the achievable savings.

Note: the figures in this section and the previous one ("up to 20x", "up to 98%", cache prices...) come from the cited literature and from providers' official documentation as of the access date; they describe those works, not BV-SALA results. BV-SALA's own figures are only those of the three ledgers and those of the results table, all from the same run: gpt-4o-mini, 193 pairs, artifact d1c8233 of 12 September 2026.

Where this is today

BV-SALA is productized as an independent product at savings.bivelio.com: a licensed SDK on npm, a five-line install and a calibration month to start — you pay the seat only, we measure all of your traffic and charge no commission, and at close you get the exact statement of what it would have cost. Inside BiVelio it is the Optimize layer: the same engine deciding, request by request, how much of your AI bill never needed to be paid.

References