This week marks the annual flagship TED conference in Vancouver, BC, where speakers and leaders from across the globe convene for a week of meaningful talks, presentations, and conversations.
And this year, the global media company announced something that will connect even more viewers: A new AI translation and dubbing project.
TED has long translated its famous Talks and TEDx Talks with the help of a robust team of translators, who are able to thoughtfully create subtitles that are literally and culturally accurate.
But by using AI, TED can do a whole lot more.
In partnership with a company called Panjaya, TED is taking its iconic Talks and dubbing them — in the original speaker’s voice — into multiple languages. The original speaker’s lip movements will also be changed to match the dubbed language.
And it’s all thanks to lip syncing and voice cloning improvements in AI.
“We’re excited about it because we can take all the thousands of Talks we’ve done, multiply them many times over, and then reach many, many more people around the world,” Jay Herratti, CEO and executive director of TED Conferences said.
Herratti premiered a clip of the dubbing in action during the conference, impressing — and possibly scaring — some attendees.
While folks have concerns about this technology and its use to create deepfakes or unethical cloning, Andy Merryman, TED’s chief technology officer, felt that adapting this technology was one of the most values-aligned options for the company in an increasingly AI-friendly world.
“From an intent standpoint, my belief is that from the technology aspect of it, these are just tools. You can use tools in any number of ways. You can use them for evil, you can use them for utility, you can use them to try to do some good,” Merryman explained.
He went on to say that TED’s mission has always been to help reach more people with meaningful ideas, to “try to meet people where they are” with curated, well-researched content from experts from all walks of life.
Offering a scalable translation solution to folks across the world, he says, is worth a shot.
“It could be extraordinarily powerful because [TED Talks] are not just the words; it’s not just the idea,” Merryman said.
“It’s the emotion, the connection, the delivery of the speaker and their passion that is really what drives people towards TED Talks and makes them latch onto it in a way that’s deeper than other formats. I think that’s what this preserves. When it’s working well, it is magical.”
TED’s head of product Tricia Maia agrees; being able to give people a more authentic translation experience is key to having a real connection with a TED speaker.
“From a product perspective, I think the Holy Grail is meeting a real user need with an emerging technology that’s just now possible,” Maia said.
She said that 70% of TED’s audience is international. Many have provided feedback, she said, sharing how they are glad they can read subtitles but that they often wish the many English talks were available in their native languages.
“This is the anecdotes and the data we latch onto,” Maia said.
“We love that we’ll serve our existing audiences better, but there’s a whole swath of people that we’re not even reaching today because the content is inaccessible, and they don’t want to watch a Talk in another language and have to read subtitles.”
Merryman added that there is also a conversation to be had about accessibility for folks who are Blind or low-sighted, as well as people who have difficulty reading, or reading and listening at the same time.
“There’s another level of it, which is the accessibility of the impact of the ideas,” he said. “The ideas aren’t going to be nearly as important, and our mission isn’t going to be fulfilled if those ideas don’t land and the impact is not felt by the people who are consuming them. The ability to look and believe and follow, and hear the words in the speaker’s voice… is a big piece of this.”
But it’s not just machine-learning robots that will do this work.
TED’s translation team has worked together with Merryman and Maia to figure out this whole project — which, at the end of the day, aims to use AI to scale the work they already do.
“We worked in tandem with them to really build out AI first, but with human-augmented processes. Maybe the AI is taking the first pass, and then we go into a systems review process,” she explained.
The team has also fed their “human-grade” subtitles into the dubbing product.
“We’re trying to take the best of AI and human intelligence and put it together,” Maia said.
“Our TED translators community is really the substrate, the backbone, of what that end result product is,” Merryman added.
He emphasized that TED’s content has always been centered around quality, meaning its team would never “take a TED Talk transcript and feed it into Google Translate and press ‘publish.’”
“It’s antithetical to what our brand stands for and is about. Cultural differences… idiosyncrasies with regional sensitivity around language; all of those things are things we need to deliver on,” Merryman said. “We just wouldn’t be doing this if we didn’t think that we could have that control.”
So… when can you send your most-watched Brené Brown TED Talk to your friend in Japan? It’s not exactly clear.
While an example of this technology debuted at the conference, Maia said everyday TED viewers can anticipate a pilot-based approach for a handful of Talks coming this summer.
The release will focus on English to non-English translations, though the tech team hopes to make Talks that were given in non-English languages available in English in later rollouts. Thousands of new TEDx talks are given in non-English languages every year and the catalogue has gone largely unwatched by viewers who don’t speak the native language.
Timing of the release is not just about the ongoing development of the tech itself, but a number of other logistics.
“We want to make sure our speakers feel comfortable, so we really lean into our speaker curation team and ask them what’s the nexus of popular talks and speakers who would think this [tech] is really cool,” Maia explained.
Legal and technical work must also be done to get the consent of speakers and ensure that their Talks are translated and dubbed properly.
“But as far as what that looks like six months to a year from now, the goal is, of course, that we want as little intervention as possible,” Merryman said.
“We want to be able to drop a Talk in, have a translator work with the Talk, press a button, the toast comes out the other side.”
While reaching that level of ease is still on the other side of months of work, increased access to TED Talks for millions could be here before we know it.
