How do we know which AI model is better for our work?

What leaderboards measure

A benchmark is a fixed set of test questions with known answers. A leaderboard is a public table that ranks AI models on one or more of these tests. Two well-known examples work in different ways.

• Arena (formerly LMArena, now at arena.ai) shows people two anonymous answers to the same prompt and asks them to vote for the better one. Its leaderboard is built from those votes, and it says the ranking is not based on benchmarks or automated scoring.
• Artificial Analysis runs its own evaluations and combines them into an intelligence index, alongside measures of speed and cost.

Both are useful for seeing which models are broadly capable and how quickly the field moves. Neither was built around your organisation's work.

Which weekly task would you use to test the AI tools you already have? Book a free AI Value Briefing and we will look at it with you.

Why a top score is not the whole answer

• Different tasks. Public tests cover maths, coding, general knowledge or open conversation. Your work may be summarising contracts in French or answering customer questions from a policy file.
• Different judges. A vote reflects what one visitor preferred for their own prompt. A pleasing tone can win a vote and still miss a rule your team must follow.
• Small gaps. Models near the top are often close. A small difference in score may not show up at all on your tasks.
• Moving targets. Models are updated often and rankings change. A table you read last month may already be out of date.
• Missing factors. Your data rules, which tools are approved, and how the model fits the software you already use matter as much as raw ability.

Field Guide 42 explains how to read these tables, and Field Guide 03 explains models and thinking modes in plain words.

Run your own test

The reliable way to compare is a small evaluation: a fixed set of your own tasks, run through each candidate model, judged the same way (Field Guide 27).

1. Pick one real piece of work the team does every week.
2. Collect 10 to 20 examples, with names and confidential details removed or replaced with made-up ones.
3. Write down what a good answer must contain, in checkable terms: the right figures, the right policy, the right tone, nothing invented.
4. Run the same instructions through two or three approved models.
5. Ask the people who do the work to score the answers, ideally without knowing which model wrote which.
6. Record the scores, the time taken to fix each draft, and any errors that would have mattered.

Keep the test set. When a model is updated or you change your instructions, run it again before you rely on the change.

Choosing with the result

Sometimes the result is that two models are close enough, and the deciding factors are data rules, cost of change and what your teams already use well. Sometimes one model clearly handles your language or documents better. Either way, the decision rests on evidence from your own work, and a named person makes it.

Better instructions and clearer source documents can also change results, so test those too.

Goes deeper in our free AI Field Guide: 3. Choosing a model and a thinking mode, 27. Evaluations: testing AI before you trust a change, 42. Benchmarks and leaderboards: how to compare AI models for your task.