Reading model scores like an invoice

A benchmark number is a claim

Every leaderboard row is somebody saying their setup produced this number. The claim is cheap to make and expensive to check, which is why most write-ups skip the checking part.

Published scores rarely ship with the conditions attached: prompt version, harness code, sampling settings, how many runs the number averages.

What an invoice has that a score hides

A price with no line items tells you nothing about what you are buying. Scores work the same way. Context length, temperature, prompt set, harness: each one moves the number, and each one is left off the box.

Two models a few points apart may be measuring different things. That difference never shows up in the headline.

Questions that cost nothing

Ask any score for four things: the harness, the prompts, the serving config, and the variance across runs.

A write-up that answers all four did the work. One that answers none is handing you an ad with a decimal in it.

Where the checking happens

PastAGI publishes audits that keep the line items attached: pricing worked down to dollars per serving hour, serving math shown beside the claims, coverage of what stays in a reasoning trace after summarization.

Start with PastAGI, AI guides that audit the benchmarks, or the audit-first AI engineering publication behind the serving-math reviews.