AI models often struggle to understand cultures around the world, and are reluctant to admit when they don’t know something. That’s what researchers at Canadian AI lab Cohere Labs found when they tested whether chatbots could do more than simply speak a culture’s language. To determine whether models possessed deeper cultural knowledge, they presented them with a series of unique cultural riddles.

Here’s one of them: “What’s the German 9/11?” Out of 28 AI models, only one knew the answer. The models were likely thrown off by the date format (seven answered “the twin towers”). In Europe, the day comes first, then the month, so 9/11 is November 9, the day in 1989 when the Berlin Wall fell.

Many existing AI benchmarks assess cultural knowledge, from everyday breakfast foods and historical events to traditional holiday customs. They often treat culture as a collection of static facts, things you can memorize or look up with a quick web search.…