How we verify model authenticity

Anyone can declare they sell claude-opus-4-5. The socket is real, the usage block parses, the lane appears on the market — and nothing in the protocol proves the tokens came back from the model that was declared. A fork of the client that quietly forwards the request to something a hundredth of the price satisfies every check a marketplace normally has. That spread is the arbitrage, and closing it is the whole job of this page.

Attestation cannot close it: there is no way to prove what binary a stranger is running on their own machine. What can be closed is the behavioural gap — ask a lane questions whose answers are properties of the model producing them, and see whether the replies look like the model that was sold.

Three separate questions

Before a lane can sell

1. You turn the Sell switch on

   The client lists every model your connected account can serve. Each one is a lane — a (device, provider, model) triple, verified on its own, because a verdict earned on a cheap model must never licence an expensive one.

2. The platform buys from you

   Not a side channel: the same gateway, the same dispatch frame, the same route a buyer's request takes — pinned to your device. If your upstream account refuses the request, that is what the run reports.

3. A verdict is written

   Around a dozen short completions, a minute or two of progress bar. The run settles at zero on both sides — nobody is billed and nobody is paid. It still spends your own subscription quota, which is the only cost anybody should bear for checking their own account.

Until a lane passes, buyers cannot see it. An unverified lane is not "slow to appear" — it is off the market by design, and the only way out of that state is the platform buying from it.

After it passes, sampling keeps it honest

An announced exam is an exam to be passed. A seller who intends to cheat can serve the real model for the ninety seconds the dialog is open and the cheap one forever after, and the gate has cost them one completion.

So the gate is not the control. Shadow sampling is: a verified lane is re-probed at intervals it cannot predict, through the same front door, in the same envelope as any purchase — and the seller is paid the market price for it. That is not generosity. An unpaid sample shows up as a gap in the earnings ledger, and a gap is a marker a cheat can watch for.

The time bound matters as much as the percentage. Without it, a lane with no buyers is never checked — and "arrange to have no buyers between samples" is exactly the sort of thing a cheat can arrange.

A sample is also spread across the better part of an hour rather than fired as one burst, because twenty-one requests back to back in a fixed order is a shape, and a shape is recognisable.

Verdicts are conservative in one direction only

An inconclusive run is not evidence of fraud. A statistic that can swing a whole tier between runs is a terrible thing to throw an honest seller off a market with, so it does not: watch costs a seller a higher sampling rate and nothing else, which turns the question into an answer within days rather than punishing a coin flip.

Automated verdicts also never freeze a seller's unreleased earnings. Withdrawing a lane is reversible; locking someone out of their own balance on a score is not. That lever exists, but only for an operator who has looked at the evidence.

The questions that decide a verdict are not public

llm-verify is published, which is the point of it — anyone can audit what it asks. It is also a ceiling: the seller being probed can read the suite too, and answer those questions honestly while serving everything else from somewhere cheaper. Pacing hides when a run happens; nothing hides what it asks once the questions are in a public repository.

So the graded questions that actually decide a verdict are generated on our side, per run, from a server-chosen seed — multi-step derivations with a single checkable answer, computed locally rather than looked up. The seller sees a question they have never seen and will never see again. A fixed private list would only be a public list with a delay; generation is what keeps rotation from depending on somebody remembering to do it. The seed goes into the report, so a contested verdict can be replayed question for question.

They are deliberately not trivia. Trivia measures a training corpus, which a cheap model with a big one passes. The realistic fraud is not selling noise as Opus — that fails the first identity probe — it is selling a mid-tier model, or a quantised mirror, as a flagship. Both answer "who made you" correctly. What separates them is how much of the hard band they get right.

Every lane is also measured against its peers

A marketplace has an advantage a standalone tool does not: it sees many lanes selling the same model. Rather than answering "is this Opus" in the abstract, it can answer "is this lane doing what the other thirty-nine Opus lanes do" — a far easier question, and a far more stable one.

Only numbers that survive a relay are tracked: how much the lane writes, how fast it generates once started, how it scores per difficulty band. No prompt content, nothing identifying a seller. A model needs at least eight lanes before its profile is allowed to say anything, and a lane more than three standard deviations out gets a deeper, forensic re-examination — never a rejection. The first seller of a model, the only seller in a region and the one seller on a fast home connection are all outliers, and none of them is a cheat.

Billing accuracy

The price is the usage the upstream provider reported. Everything around that exists to make sure that is the number reaching settlement:

• The gateway counts the same request itself and compares. A deviation above 25% — on requests over 200 tokens, below which the estimator's own error dominates — is recorded against the seller's reputation. It does not change the price.
• Output is capped at the budget signed into the dispatch. A seller cannot bill for more output than the platform authorised.
• Prompt is capped at three times the gateway's own estimate. Beyond that, all counts are scaled down proportionally so the ratio between token types is preserved.

The traps here are not exotic, and getting one wrong is silent. Anthropic's input_tokens excludes cached tokens; OpenAI's prompt_tokens and Gemini's promptTokenCount include them — map both onto one field and the cached portion is billed twice, on the largest part of an agentic prompt. Gemini bills thoughtsTokenCount at the output rate and does not put it in candidatesTokenCount. A GPT-family tool list is billed as the TypeScript declaration it gets rewritten into, not as the JSON you sent. A 1024×1024 screenshot is over a thousand Claude tokens, not the flat 85 a naive estimator assumes.

Both tools are open source

The verification engine and the token accounting are not buried in this platform. They are standalone crates, Apache-2.0, published on crates.io and usable against any endpoint — including ours.

llm-verify

Black-box authenticity, billing and performance verification for any LLM endpoint. 40 probes across seven groups — protocol contract, streaming, metering & billing, channel provenance, performance, model identity, cross-request consistency — scored on two independent axes and written out as an HTML report you open in a browser. Point it at your own supplier, or at us.

token-meter

Cross-provider token metering and cost accounting for Claude, the GPT family and Gemini. Every count carries its own provenance, so you always know whether you are holding a measurement or an estimate — and money never passes through a float.

llm-verify runs as a CLI (cargo install llm-verify, or the install script) and exits non-zero on a failing score, so it works as a CI gate against your own supplier as-is. This platform embeds the same crate as a library, which is what lets its results and the published tool's agree.

What this does not prove

Black-box behavioural testing gives a probability, never a proof of which weights answered. Stated plainly:

• Resolution stops at tier granularity. Adjacent versions inside one tier cannot be separated.
• One run describes one moment. Gradual degradation is caught by re-running, not by a single verdict.
• Serving above the claimed tier is not fraud and carries no weight. Only measuring below the claim counts.
• The published probes are readable by the endpoint being probed. The private bank raises the cost of adaptive fraud; it does not close it.

This is one layer, and it is built to be one layer — over the payout hold that gives detection a deadline to work inside, over reputation scoring on ordinary traffic, and over a human looking at the outliers. It is not the thing that makes those unnecessary.