Leo

Posted on Nov 12

AI-assisted

When people think about translating manga automatically, they think about the translation model. In practice the translation is the easy part. The hard part is everything around it: finding the text, getting it off the page cleanly, and putting the translation back so the page still reads like a comic.

This post walks through the image side of an automatic manga translator I work on, and what we learned by comparing our pages with official English and Korean editions.

The pipeline

Each page goes through five steps:

Find the text. A detector marks the regions that contain text. Page numbers, headers and artist watermarks are not dialogue, so they are left alone.

Find the speech bubbles. Knowing the shape of each balloon matters twice: once for cleaning and once for lettering.

Read the text. Japanese is often vertical, Korean and Chinese mix scripts, and sound effects are drawn rather than typed.…