Benchmark Gaming

[SamuelGabriel]

Hey there,

Thanks a lot for this super nice LLM benchmark!

Looking at the new release of Grok3 being clearly the top-1 model, I wondered whether you could make some analysis to figure out how much each company is biasing your benchmarks.
I heard from people in other companies that they use lmarena and can tell by the style of their model which model is theirs, and with grok this risk is even higher as it is more opinionated about politics. So, it should be even easier to tell for affiliated people.

A more regular release s.t. independent researchers can analyze the chats or analysis by you on this problem would be great to keep the benchmark the go-to.

Best,
Sam