Meta’s new ‘Voicebox’ AI is a text-to-speech tool that learns like ChatGPT

Meta claims Voicebox is the first AI that can generalize text-to-speech tasks it wasn’t trained to accomplish and describes it as a “breakthrough.”

Meta AI recently unveiled a “breakthrough” text-to-speech (TTS) generator it claims produces results up to 20 times faster than state-of-the-art artificial intelligence models with comparable performance. 

The new system, dubbed Voicebox, eschews traditional TTS architecture in favor of a model more akin to OpenAI’s ChatGPT or Google’s Bard.

Among the main differences between Voicebox and similar TTS models, such as ElevenLabs Prime Voice AI, is that Meta’s offering can generalize through in-context learning.

Much like ChatGPT or other transformer models, Voicebox uses large-scale training datasets. Previous efforts to use massive troves of audio data have resulted in severely degraded audio outputs. For this reason, most TTS systems use small, highly-curated, labelled datasets.

Meta overcomes this limitation through a novel training scheme that ditches labels and curation for an architecture capable of “in-filling” audio information.

As Meta AI put in a June 16 blog post, Voicebox is the “first model that can generalize to speech-generation tasks it was not specifically trained to accomplish with state-of-the-art performance.”

This makes it possible for Voicebox to translate text to speech, remove unwanted noise by synthesizing replacement speech, and even apply a speaker’s voice to different language outputs.

According to an accompanying research paper published by Meta, its pre-trained Voicebox system can accomplish all of this using only the desired output text and a three-second audio clip.

The arrival of robust speech-generation comes at particular sensitive time as social media companies continue to struggle with moderation and, in the U.S., a looming presidential election threatens to once again test the limits of online misinformation detection.

Former U.S. president Donald Trump, for example, currently faces allegations that he mishandled confidential government materials after leaving office. Among the purported evidence cited in the case against him are audio recordings wherein he allegedly admitted to potential wrongdoing.

While there’s currently no indication that the former president intends to deny the content described in the audio files, his case illustrates that data integrity resides at the core of the U.S. legal system and, by extension, its democracy.

Voicebox isn’t the first tool of its kind, but it appears to be among the most robust. As such, Meta’s developed a tool for determining if speech was generated by it which the company claims can “trivially detect” the difference between real and fake audio. Per the blog post:

“As with other powerful new AI innovations, we recognize that this technology brings the potential for misuse and unintended harm. In our paper, we detail how we built a highly effective classifier that can distinguish between authentic speech and audio generated with Voicebox to mitigate these possible future risks.”

In the cryptocurrency world, AI has become as integral to day-to-day operations for most businesses as the internet or electricity. The largest exchanges rely on AI chatbots for customer interactions and sentiment analysis, and trading bots have become commonplace.

Related: Bybit plugs into ChatGPT for AI-powered trading tools

The advent of robust text-to-speech systems such as Voicebox, combined with automated trading, could help bridge a gap for would-be cryptocurrency traders who rely on TTS systems that, currently, may struggle with crypto jargon or multi-lingual support.