A new paper analyzing roughly 89,000 pairwise comparisons across 116 languages and 52 models has a blunt conclusion: the single global ranking you're using to pick your LLM is nearly meaningless. Nearly two-thirds of the decisive votes in global rankings cancel each other out. The leaderboard isn't surfacing a winner. It's averaging away signal you actually need.
The Problem With One Number
Platforms like Chatbot Arena use Bradley-Terry rankings built from human preference votes. Show two responses to a user, they pick the better one, and over millions of comparisons a ranked list emerges. It works well when the population has coherent preferences. When they don't, the rankings become statistical noise wearing a lab coat.
The paper, posted to arXiv, found that when you lump together voters across 116 languages, you get exactly that: noise. The top 50 models in the global ranking are statistically indistinguishable from one another, with pairwise win probabilities capping out at 0.53. A coin flip gets you to 0.50. The difference between "number one" and "number fifty" is three percentage points of coin-flip odds.
Language Is the Hidden Variable
The researchers ran a straightforward diagnostic: group votes by language instead of pooling them all together. The results got dramatically cleaner. ELO score spreads increased by two orders of magnitude. Rankings that were a statistical blur became sharp and decisive.
This makes intuitive sense once you see it. A model that dominates English creative writing may get crushed by a different model on Portuguese legal text or Japanese conversational prompts. When you average those two populations together, both signals disappear. What you're left with is a ranking that tells you which model is mediocre at everything rather than excellent at something.
The paper identifies language as the primary driver of this heterogeneity, but also notes that task type and time play roles. A model's relative standing isn't fixed. It depends heavily on who's asking what.
Five Rankings Beat One
Here's where the research gets concrete. The authors built a portfolio framework that treats model selection as a set cover problem: find the smallest set of rankings (or models) that covers the most voter subpopulations. Their algorithm recovered five distinct language-grouped Bradley-Terry rankings that collectively account for over 96% of all votes. The single global ranking covers 21%.
That gap is the argument. One ranking represents roughly a fifth of actual user preferences. Five language-aware rankings represent nearly all of them.
The same logic applies to model selection. A portfolio of six LLMs chosen to cover different language subpopulations serves twice as many votes as the top six models from the global ranking. The globally "best" models cluster around the same user segment. A diverse portfolio actually reaches your users.
What This Means for How You Pick Models
If your application is English-only and your users are a homogeneous group, the global leaderboard is probably fine as a rough filter. But if you're building anything multilingual, serving users across regions, or evaluating models for a specific domain, a single global ELO score is actively misleading. You're optimizing for a benchmark that was never measuring what you care about.
The practical implication: before you commit to a model, look for language-specific or domain-specific eval data. If that doesn't exist for your use case, treat leaderboard rankings as a starting point for your own testing, not a conclusion. The portfolio framing is also worth internalizing at deployment scale. Routing to different models based on language or task type may outperform picking one "best" model and using it everywhere.
A few caveats: the paper doesn't name which specific models excel in which languages, and it's unclear how recent the Arena data used actually is. Rankings shift as new models drop, so the structural finding is durable, but any specific model comparisons would need fresh data.
Bottom Line
Global LLM leaderboards compress heterogeneous preferences into a single number and lose most of the information in the process. Language-aware evaluation produces rankings that are two orders of magnitude more discriminating and cover 96% of user preferences instead of 21%. If you're selecting models for anything beyond English-centric use cases, a single leaderboard position should carry very little weight in your decision.
Sources
- arXiv:2605.06656 (primary research paper)
- Chatbot Arena (LMSYS)
- Bradley-Terry model (Wikipedia)
- Set cover problem (Wikipedia)