Summer is big-budget, blockbuster movie release time, when major film studios market to global audiences to maximize box-office takings. Studios can spend hundreds of thousands of dollars on post-production dubbing, to apply different languages and subtitles, so that audiences in foreign-language countries can understand their movies. But reading subtitles requires viewers to divert their focus from the scene, and voiceovers are typically poor matches to the actor’s speech pattern and tone.
Artificial intelligence is transforming the craft of voiceovers, synching actors’ lip movements to a multitude of languages and precisely matching their tones and inflections, and other unique vocal characteristics to the dialogue. Traditional dubbing involves an individual recording dialogue in a different language, but AI dubbing uses advanced speech recognition technology with voice synthesis. And speech to text technology is also being fine tuned as well to automatically offer subtitles that would otherwise be immensely time-consuming.
The number of companies integrating AI into dubbing is growing, reflecting the number of films that are grabbing larger audiences on the global stage. The AI technology allows for professional creators to drastically reduce the amount of time it takes for dubbing, from months to minutes, and it also opens doors to content being watched by audiences around the world.
LipDub AI was unveiled on August 15, 2023, and was touted as “the world’s first completely automated lip-sync technology to achieve Hollywood-caliber results.”
Benjamin Hoehn, the VP of Marketing at MARZ and LipDub AI, said in an interview with Techstrong, “Our general looking-forward thesis is that right now only one percent of the world’s online content is dubbed. And now there’s all these tools out there that allow for easy and fast dubbing. The rise of audio tools and content creation platforms is leading to a surge in dubbed content online. However, the quality of these dubs can often be questionable. LipDub AI technology ensures that dubbed content maintains a natural look and feel, eliminating the uncanny valley effect that can plague traditional dubbing methods. As the volume of dubbed content continues to grow, LipDub AI positions itself as an essential tool for video creators and marketers, ensuring their message is delivered seamlessly across languages and cultures. Generally, we see a proliferation of dubbing tools like this, but only LipDub AI has a focus on quality and realistic output.”
According to Business Research Insights, the global film dubbing market size in 2021 was $3.3 billion, and is projected to grow to $6.6 billion by 2031. According to the firm, “In recent years, the global film business has experienced substantial expansion. Outside of Hollywood, there has been a rise in the number of film production companies, and online movie streaming services like Netflix now hold a significant portion of the market. One of the main factors propelling the market’s expansion is the rising demand for foreign films with native language subtitles. Over the course of the projection period, it is also projected that the use of artificial intelligence (AI) in cinema dubbing would offer a profitable chance for the market to expand.”
LipDub AI uses machine learning to create hyper-realistic lip-syncing effects in video. LipDub AI analyzes submitted video, recognizes faces and speech patterns, and then the user labels each actor after the AI training is completed. A language is chosen, and the AI uses the speech patterns that it has “learned” to create synchronized mouth movements in the target language on the faces in the video. This technology allows for creating dubbed videos that go beyond subtitles, aiming to provide a more natural viewing experience where the visuals seem to match the spoken language.
Another leading company in the sector, MURF, explains that AI dubbing goes beyond just translating language.
“Dubbing in the film industry is more than just translating dialogues into a foreign language; it’s about recreating the movie experience in a new language, with all the emotional beats and cultural nuances intact,” the company states.
While the technology continues to revolutionize the movie industry, it has also unfortunately turned up to recreate videos that, to use the old idiom, “put words into people’s mouth.” There are hundreds of examples of altered videos that do this, that can twist the technology for the purposes of disinformation, and the typical targets are politicians, actors, athletes and other celebrities or influential people.
