Enterprise voice AI promises are compelling: reduce costs, improve customer experience, and scale operations effortlessly. But the reality of deploying voice systems that handle hundreds of thousands of daily interactions reveals challenges that most companies never anticipate. We spoke with Lily Clifford, CEO and co-founder of Rime, whose company now powers close to 100 million phone calls monthly, about what it actually takes to succeed with enterprise voice AI at scale.

Rime builds text-to-speech models specifically optimized for enterprise telephony applications, focusing on the reliability and pronunciation accuracy that high-volume voice applications demand.

The Academic Origins of a Business Problem

Lily's entry into voice AI began in Stanford's linguistics department, where she was "working on a bunch of different voice AI related stuff, but primarily not voice AI stuff to begin with." Her background in acoustic phonetics and signal processing gave her a deep understanding of speech that would prove crucial as the field evolved.

"I was a PhD student at Stanford before founding the company a couple of years ago," Lily explains. The timing proved fortuitous—2018 brought Wave2Vec2 from Facebook, "a pretty advanced speech recognition network that was open-sourced, transformer based, had been pre-trained on a ton of data." This technological moment converged with her academic expertise and connections with co-founders Brooke Larson (from Amazon Alexa) and R.H. Giovannos (from UCSF's Chang lab working on computer brain interfaces).

But the real motivation came from a frustratingly common experience: "I get a new iPhone because my battery sucks and I'm activating my iPhone. And so I call the carrier, like that's T-Mobile, right? And I call T-Mobile and I'm like, get put in the IVR system. And the IVR system, right, is like, this 20th century American broadcast standard voice that's like, 'thanks for calling T-Mobile.' And I like hate it."

This personal frustration with legacy voice systems became the foundation for a business focused on making people actually want to talk to voice bots instead of immediately pressing zero for a human agent.

The Long History of an "Unsolved" Problem

Unlike the recent explosion in large language models, text-to-speech has been around for decades. "I often describe it as the OG generative AI," Lily notes. "And in fact, even before AI, people were synthesizing speech by drawing with a Q-tip on magnetic tape, like these formants to synthesize vowels."

This long history reveals just how challenging the problem remains. "In San Francisco, you're on the BART platform and the train announcements, those are TTS. They sound horrible. And the reason why they sound horrible is because that's what state of the art was in 1999."

The persistence of poor-quality voice systems in production environments highlights a crucial insight: "It's not like large language models where you see like so much capital rushing into like five or six different companies who are each going to burn through like billions of dollars in the next year's training models. Like this has been, been a problem that people are trying to solve for, for a long time. And even still, I think, right, like, I wouldn't think that we consider it solved."

The Multi-Speaker Modeling Revolution

The breakthrough that enabled Rime's approach came from advances in multi-speaker modeling capabilities. Traditional voice AI followed a rigid pattern: "Google might have 15, Amazon might have 25, Microsoft might have 30, or whatever it might be, right? Like North American English voices."

This approach required extensive manual work: "That was 15 voice actors that came into Google's recording studio. Google sat them down. They had them read a bunch of what are called phonetically rich sentences, some probably New York Times articles for news reading applications. Maybe you get 40, 50 hours of data from each of these people, and then you turn a single model to reproduce the characteristics of that person's voice."

Modern approaches enable something radically different: "The kind of chameleon nature that you see out of even just a single set of model weights, the reason for that is because we're better able to model these kinds of multi-speaker variability. young people, old people, people in the South, people in New York, people in Boston."

This technical capability opens business opportunities that Lily finds exciting: "If someone who is 33 years old and just bought an iPhone and is living in San Francisco, female tech savvy calls to activate that iPhone, what voice should that person hear? I think that's an open question and will be a constantly evolving question, but seems to me like an opportunity to build it."

The Customer Expectation Evolution

Perhaps the most surprising shift Rime has witnessed is the complete reversal in customer preferences around voice realism. "When we were talking with customers a year ago, the response was like, 'I don't want that. I don't want our bot to sound real. Honestly, I could pull up emails and people would be like, no, it would be worse if the bot sounded better.'"

The reasoning was understandable: "People didn't want the human on the other end to feel tricked. And so like, and they had run tests too. Like they had run, I'm not kidding. They had run AB tests. Like, you know, at that time it was probably like the non-neural versus neural Google TTS or whatever it was. And what they found is like, they were converting customers at a higher rate with a non-neural TTS."

But this has completely flipped: "For what it's worth, we're not seeing that today at all. Right? Like the more realistic the voice is, the more relatable it is, the higher the likelihood of a call succeeding is."

Lily attributes this shift to changing consumer behavior and expectations, accelerated by widespread adoption of AI tools: "I think people just got sick, like really sick of like the like really stiltifying like phone tree experience. And it just took like a decade to set it."

The broader AI adoption has helped normalize these interactions: "The more people that have engaging and entertaining conversations with an AI, whether over text or voice or whatever modality, like the more that consumer expectations are going to level up for when you're at the drive through window."

The Enterprise Reliability Challenge

While consumer expectations have evolved, enterprise requirements remain demanding. Rime's focus on high-volume enterprise customers reveals challenges that don't surface in smaller deployments.

"If we're running a customer discovery kind of process of someone trying to qualify a deal and I asked them like, okay, you're building a voice application... do you ever run into problems with the TTS model, not pronouncing something correctly? And if someone says no, I'm usually confident that that customer is not making, say, like 10,000 phone calls a day."

The scale creates unique technical challenges: "Definitely like high volume applications where reliability around pronunciation, around latency is really important, that's where we win. And it's the reason why the average RIME customer is making between 100 and 200,000 phone calls a day."

These customers often support multiple brands and use cases: "They are often themselves, right, like supporting many different brands, if it's a retail application, right? Like each brand has their own unique skews, like all that stuff is really important."

Solving the Pronunciation Problem at Scale

One of Rime's key innovations addresses a persistent challenge in enterprise voice applications: proper noun pronunciation. "We know, we just know from talking with customers, like if you call T-Mobile, right? Like, they say, 'Hey, thanks for calling Brooke.' That like, that call is more likely to be a success and that like average CSAT is going to go up by crazy on these calls. But. How do we know that the TTS model is going to read out that name correctly? Because oftentimes they don't."

Rime has built sophisticated APIs to handle this challenge: "We built this API, instead of APIs for solving this problem around proper noun pronunciation... what's, we want our customers to able to say like, what is the likelihood of us pronouncing that name correctly? Right? Like how, how can we figure that out? Because if we, if we, if we're a hundred percent, then we want to say it."

The system handles complex scenarios: "Is this in our dictionary? If it is in our dictionary, is it, is it a proper name that has more than one pronunciation? Like Alicia versus Alicia. And then if it is, you can just ask the customer. And then you have this whole other API for recognizing what the customer says, turning it into a phonetic schema that our model can understand."

The Domain Specificity Imperative

Lily's experience has revealed how dramatically speech patterns vary across different contexts. Despite powering millions of drive-through orders, "We've been doing that for a year before I heard an actual recording of a drive-through interaction."

When she finally heard the recordings, the insight was immediate: "I listened to the recording and I was like, this is incredible. There is no model that will sound as like, who did this as good as this human at the drive-through window did it... The person at the drive-through window doesn't talk to anyone else in their life like that, unless they're at that window with the headset on. And they've learned how to do that."

This domain specificity creates both challenges and opportunities: "That data is so unique... And so there's so many things like that that I just think are clearly a data problem."

The Speech-to-Speech Architecture Debate

The industry is currently divided on fundamental architectural questions, with some companies pursuing end-to-end speech-to-speech models while others like Rime advocate for cascaded approaches.

Lily's perspective is informed by practical deployment requirements: "TTS systems for the longest time till today and will for the foreseeable future. Like if you need controllability around pronunciation will be trained on sequences of phonemes, right? Like not letters, but phonemes."

The challenge with end-to-end approaches involves leveraging existing language model capabilities: "When you're training, not just like a speech to speech model, but like a large text to speech model, a text condition speech model to generate speech from text, like you want to be able to leverage right, like all this information that's in these pre trained language models, which have seen just trillions of tokens of text, right? The problem is like all that text is not in phonemes, it's in letters."

Rethinking Voice Discovery and Selection

Traditional approaches to voice selection have proven inadequate for modern applications. "I really feel like, we, we, we be, we continue to think about like a voice first UI for finding a voice."

Instead of browsing through lists or typing descriptions, Rime is exploring conversational voice discovery: "It's like, okay, what about a voice that's like, you're like talking with an agent and you're like, okay, what about a voice that's like this, this, and this, and we're going to role play doing my application right now. And you're going to talk to me right with that voice. I'm like, okay, that sounds good. We're going to do that right now."

This approach creates "a sense of delight instead of just picking from this list, which is this like really dark pattern from like 30 years ago when you only had like 10 voices."

From Revenue Centers to Competitive Advantage

Lily's analysis of where voice AI creates value reveals an important shift in enterprise thinking. "Phone applications, voice applications in the enterprise have been basically in the cost center of the business for 30 plus years... Customer support, troubleshooting, whatever it might be."

But rising consumer expectations are changing this dynamic: "What rising consumer expectations means, because again, these applications are more sophisticated and therefore can do more. What that means is consumer expectations have to increase for people to be willing to unlock those applications in the revenue center for sales or for ordering or for anything, fashion advice, whatever it might be."

This shift represents a fundamental change: companies are moving from seeing voice AI as a cost reduction tool to viewing it as a competitive differentiator and revenue driver.

The Future of Enterprise Voice AI

Looking ahead, Lily sees several key developments on the horizon. "Keep an eye out next week. We're starting with next week," she hints at upcoming product announcements around voice evaluation and monitoring capabilities.

But the bigger vision involves bringing linguistic expertise to bear on practical developer challenges: "How can we bring our expertise to bear as linguists to help developers solve these problems that today they don't even necessarily have a language for?"

Rime is also working toward more sophisticated end-to-end capabilities while maintaining enterprise requirements: "I don't know, keep an eye out for a rhyme, fully end to end speech to speech model. But of course, like our whole thesis is like, How can that from day one be like API accessible and enterprise ready?"

Key Takeaways for Enterprise Voice AI

Lily's journey from academic research to enterprise deployment offers several crucial insights:

1. Scale Determines Success: High-volume applications (100,000+ calls daily) face fundamentally different challenges than smaller deployments

2. Pronunciation Accuracy is Non-Negotiable: Enterprise applications require sophisticated handling of proper nouns, alphanumeric sequences, and domain-specific terminology

3. Domain-Specific Training Matters: Generic models fail to capture the nuanced communication patterns that make human agents effective in specific contexts

4. Consumer Expectations Drive Enterprise Requirements: Widespread AI adoption is raising the bar for all voice interactions

5. Hybrid Architectures Enable Control: Maintaining pronunciation accuracy and system reliability often requires combining multiple technologies

6. Voice Selection Needs Rethinking: Traditional list-based approaches don't serve modern voice AI applications

7. Evaluation Infrastructure is Critical: At enterprise scale, systematic quality monitoring becomes essential for maintaining performance

As voice AI continues its evolution from experimental technology to mission-critical business infrastructure, Rime's experience demonstrates that success requires more than impressive demos. It demands deep linguistic expertise, robust engineering, and a clear understanding of enterprise deployment realities. The companies that thrive will be those that can bridge the gap between AI capabilities and business requirements, delivering reliable voice experiences that customers actually want to engage with.