Methodology

• Every metric traces to a public source with a retrieved-at timestamp, in both the HTML and the JSON API.
• Nothing is estimated or blended. If a source doesn't publish a number, we show "—" / null. We never average incompatible benchmarks into a single invented "score".
• Refresh is automated (daily) and each source keeps its last-good snapshot, so an upstream outage shows stale-but-labeled data instead of broken pages.
• Conflicts of interest are disclosed: gcrab is our own project and is labeled as such wherever it appears.

Capability scores and prices come from two different sources that name models differently, so they have to be matched. Getting this wrong would attach the wrong score to a model, which is worse than showing none, so the matcher is deliberately conservative:

• Exact names first. Normalization is tiered — we only relax the name (stripping reasoning-effort markers like -high or -xhigh) if the exact name finds nothing. This keeps product tiers such as Qwen's "Max" from being mistaken for an effort setting.
• Generation stamps must agree. Vendors like Mistral encode the model generation in the name (mistral-small-2603 is not mistral-small-2506). If both names carry a stamp and the stamps differ, the match is refused outright.
• The measured variant is always shown. Where a score came from an effort variant, the API reports it in measured_variant so you can see exactly what was tested.

Models we track that have no published score on a board are simply absent from it. About a third of tracked models — mostly image, audio and very small models — have no Arena text score at all.

• SWE-bench Verified is the closest public thing to ground-truth agent capability, but it measures scaffold+model combos on repository bug-fixing — not general assistance.
• GitHub stars measure adoption and attention, not quality. We rank frameworks by stars because it is verifiable and unambiguous, and we say so on the page.
• Pricing is what OpenRouter lists at retrieval time; direct-from-provider pricing can differ.

• In-house live task suite (planned): a small, published set of real-world agent tasks (terminal ops, web research, file wrangling) run on a schedule against top frameworks/models, with full transcripts published for every run. No score will appear before the transcripts do.
• Historical trend lines (price and standings over time — snapshots are already being archived daily).
• Additional boards: Terminal-Bench, GAIA, arena-style rankings, token-throughput/latency.