Most of the benchmarks are in English. But a large portion of the world's population doesn't speak English (around 80 %). We can see from SWE Pro or the stagnation at DeepSWE that we're kind of measuring in a vacuum right now.

Do you know of any initiatives where people come together? No one is supposed to know the full benchmark, but everyone contributes 2–4 very well-thought-out agent-based tasks in foreign languages, each of which has been devised and validated by teams of 3–4 people. Don’t we need more of this? A kind of collective benchmark?

It should cover creative writing, business emails, and complex coding tasks. But also individuality, UI creativity, good UX, no duplicate content, etc. But things like compaction and legal issues could also be part of the agentic tasks. I think there aren't enough good benchmarks here.