Designing Hybrid Conversational AI That Actually Works

As conversational AI systems become more capable, production environments expose the limits of technology when it is used without a design framework that holds quality as the guiding principle.

In this episode of Unprompted, Conversation Design Institute CEO Hans van Dam speaks with Rebecca Evanhoe about what it takes to build conversational AI that performs reliably in real world environments. Drawing on her work building voice AI for thousands of restaurants at Slang.ai and her deep experience as an author and educator, Rebecca offers a grounded perspective on hybrid conversational systems, evaluation, and the enduring importance of conversation design principles in an era of large language models. (The CDI Foundation)

“The question is not whether the model can generate language. The question is whether the system behaves in a way people can rely on.”

— Rebecca Evanhoe

This conversation moves beyond hype and explores the practical, measurable work required to deliver systems that businesses and users can depend on.

Rebecca Evanhoe | Author, Educator, Product Leader at Slang.ai

Rebecca Evanhoe has helped define the practice of conversation design from multiple angles as a practitioner, educator, author, and product leader. She co authored Conversations with Things: UX Design for Chat and Voice, a foundational text in the field. Rebecca teaches conversation design at Pratt Institute and leads conversation design strategy at Slang.ai, where she builds hybrid conversational systems used by thousands of restaurants across the United States. The CDI Foundation

Her work bridges humanistic insight with practical product development, emphasizing evaluation, reliability, and the evolving role of designers as AI tooling changes.

“Good conversation design is not about clever responses. It is about reducing failure in real situations.”
— Rebecca Evanhoe

Hans van Dam | CEO, Conversation Design Institute

Hans van Dam is the CEO of the Conversation Design Institute and a long standing contributor to the field of conversational AI. He focuses on helping organizations and practitioners design and evaluate conversational systems that work in real world conditions.

As host of Unprompted, Hans brings an integrated perspective that connects foundational design principles with the challenges of deploying conversational AI at scale.

A major theme of the episode is the value of hybrid conversational systems that blend deterministic logic with carefully scoped language model tasks rather than relying entirely on generative output.

Rebecca explains why these hybrid systems tend to outperform fully generative approaches, especially in voice based environments where factors such as speech recognition, timing, turn taking, and accurate understanding of user intent are critical to user trust and success. In high stakes contexts such as customer interactions in restaurants, hybrid architectures allow teams to control critical flows while leveraging language model flexibility where it truly adds value.

“We use language models as components, not as the entire conversation.”
— Rebecca Evanhoe

She emphasizes that performance is shaped not only by the model, but by the entire conversation stack, including accuracy of speech understanding, response timing, and system infrastructure.

Evaluation emerges as one of the most important skills for conversation designers working with AI systems rather than prototypes.

Rebecca walks through how her team designs experiments to test conversational AI behaviors before deploying them. These tests cover practical scenarios such as entity extraction, business hours questions, multilingual translation, and edge cases that often expose weaknesses in conversational logic. She advocates for defining clear success criteria, building realistic test sets, and combining automated evaluation with human review to gain confidence in system behavior and reduce unintended responses.

“If you do not define what success looks like, the system will decide for you.”
— Rebecca Evanhoe

This approach moves conversations from experimental to dependable, helping teams catch issues before they reach users.

Voice based conversational AI introduces unique challenges that differ from text based systems. Rebecca and Hans explore how conversational timing, turn taking, and interruption handling shape user experience. If even small delays or misaligned responses occur, user trust can erode quickly, especially in environments where users expect clear and efficient interactions.

They also touch on multilingual systems and the importance of transparency around AI behavior. These elements are not peripheral, but core to how conversational systems are perceived and judged by users.

As AI tooling evolves, the role of the conversation designer is not disappearing. Rather, the work is growing in scope.

Rebecca challenges the notion that conversation design is becoming obsolete. Instead, she argues that designers who can diagnose failures across the stack—from user experience flaws to model limitations and infrastructure issues—are becoming increasingly valuable.

“The craft has not disappeared. The surface area has grown.”
— Rebecca Evanhoe

Strong conversation designers today combine timeless skills such as context awareness and respect for human conversational norms with a working understanding of how AI behaves under real conditions.

This post highlights key themes from the conversation, but the full episode explores these ideas in greater depth, providing practical stories, examples of evaluation design, and deep discussion of hybrid conversational architectures.

Whether you are building production grade conversational systems or learning how AI is reshaping the role of conversation designers, this episode provides a clear, experience driven look at what it takes to design conversational AI that actually works.

Watch or listen to the complete episode of Unprompted with Rebecca Evanhoe, embedded below.