Menu

Keeping a synthetic voice in sync with a YouTube video: seeks, speed caps and ducking
📰

Keeping a synthetic voice in sync with a YouTube video: seeks, speed caps and ducking

DEV Community: javascript·HS Solution·12 days ago
#rjaHeAdz
#devtojavascript#devto#dev#voice#captions
Reading 0:00
15s threshold

We build DubbingMate, a tool that reads a YouTube video's captions out loud in another language.

Getting the words right turned out to be the easy part. Keeping a synthetic voice in step with a video that the viewer pauses, seeks and speeds up was most of the work.

Here is what our sync code does today, with the numbers we ended up using.

The panel on the right: dubbing is on, the original audio is ducked to 20%, the dub plays at 100%.

1. Captions are the timeline

Every caption cue has a start and an end time. That is the only clock we trust. Before playback we clean the cue list: if a cue ends after the next one starts, we cut its end to the next start, so two lines never compete for the same second.

If a video has no captions at all, there is nothing to dub. We do not run our own speech recognition, and the app says so instead of guessing.

2. The voice is often longer than the cue

A sentence that takes 2.1 seconds in English can take 3 seconds in Japanese.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.