Captions Debuts AI Lipdub with Translation and Gen Z Slang

Captions, which leverages AI to help its customers produce “studio quality videos directly from their mobile devices,” has launched a new app called Lipdub that automatically translates and dubs content into 28 languages. The free download lets user dub anyone “and experience familiar voices and faces in a suite of new languages.” Lipdub’s translations not only duplicate what the company says is “the subject’s exact voice,” but also syncs lip movements to match. It also incorporates dialects and idioms, with options like Gen Z and Texas slang.

“Users can simply record or upload a video, and the app will change the subject’s exact voice and lip movements to any language they select — including French, Hindi, Spanish, Italian, Portuguese, and more,” the company explains in an announcement.

The app is initially debuting for iOS. Like Captions, Lipdub lets users publish videos to other platforms, including YouTube, TikTok and Instagram Reels, reports VentureBeat.

Founded in 2021 by former Snap Head of Design Engineering Gaurav Misra, who did a stint as a Microsoft software engineer, Captions in June disclosed a $25 million funding round, for a total of $40 million to date.

Among those currently using the Captions creative studio are “Disney-owned sports network ESPN and its commentator Omar Raja, ‘Mr. Wonderful’ of ‘Shark Tank’ fame, Twitch’s founder Justin Kan and the influencer Unnecessary Inventions,” according to VentureBeat.

The company claims to have “provided more than 5 million creators of all skill levels — from real estate agents to podcasters to online coaches — the tools needed to develop and publish studio-grade content with just a few quick taps” with automation for everything from editing and captioning to music integration. An AI Eye Contact feature can correct speaker sightlines.

TechCrunch says “using translation and AI-dubbing to reach a wider audience is a growing trend,” reporting that “in June, YouTube announced that it is testing an AI-powered tool to let users automatically dub their videos in other languages” and is looking to integrate it into YouTube Studio.

“Earlier this month, AI-powered voice-generating platform ElevenLabs released its dubbing tool with support for 29 languages,” while “Rest of World previously reported that dubbing service provider companies are generating millions of dollars by translating content for popular YouTubers like MrBeast,” per TechCrunch.

No Comments Yet

You can be the first to comment!

Leave a comment

You must be logged in to post a comment.