Legal AI has learned to summarise. But has it learned how lawyers think?
In my experience, AI summaries of judgments read well and still get things wrong, i.e. a holding stated too broadly, or a point the court rejected reported as accepted. So I made "Catch the AI". You read a real judgment, flag each wrong line in the AI's summary, and a catch counts only when you highlight the words of the judgment that prove it wrong.
It is part of "May or Shall", which I built on the idea that human judgment comes first and AI drafting follows.
Disclosure: I built it. https://app.mayorshall.com/play
I would like to hear how many people here catch all the mistakes, and which ones were the hardest to spot.