/blogSpeakers:
Lisa Falkson — Senior VUI Designer at Amazon
Elle Chun — UX Conversational Designer at Sutherland Labs
Joan Bajorek, Ph.D. — Linguist & Founder of Women in Voice
Scott Ganz — Principal Content Designer at Intuit
Justina Nguyen — Developer Evangelist at Dashbot (moderator)
Before diving into the technical specs of a chatbot or a voice application, take a look at the customers and define the value proposition of conversational technology for your brand. Would it take less time to do in speech than it would when tapping on a mobile phone or clicking on a website? If you’ve determined yes, the next thing to do is to study and understand your customer base.
Create a profile of who they are by deeply understanding what they want, their attitude towards the subject matter, and what their potential fears are when interacting with conversational AI. When designing these interactions, Scott Ganz thinks of customers as characters and engages as much as empathy as he can to get into a person’s mindset of when they’re using the chatbot. After understanding who the users are, you can start to craft the actual conversation.
Start imagining what a successful conversation looks like in what’s called the happy path. Once you have a sense of the core use cases and potential successful scenarios, you start preparing for failure by creating safety nets to catch users when they fall out of that happy path. Many companies prioritize writing happy paths, but Lisa Falkson says, “A conversational AI’s ability to address errors is equally important and you should allow for at least 50% of development time for errors.”
Building a Team: Skill Sets Needed for Conversational Products
So you’ve decided to extend your brand onto conversational interfaces and your next step is to find the right people with the right skills to help you build this product. For designing these conversations, the common denominators for roles were linguists, screenwriters, and content creators. Ganz, an Emmy-award winning screenwriter, emphasizes the importance of empathy when creating conversations and the ability to switch between the role of the customer and the brand. Joan Bajorek, a linguist Ph.D., notes the power of a clear, distinct wake word (the word/phrase used to launch your Alexa Skill or Action on Google) which needs to be thoughtfully crafted with the help of a linguist.
From a development standpoint, it depends on how much you want to pull off the shelf or build in-house. Engineers with a background in human-computer interaction can be especially beneficial during the development process. More importantly, someone who can think can think of the conversation in terms of a system and knows how to integrate design, NLP, and development.
Defining Success: Conversational Metrics that Matter
Success metrics vary depending on the use case you’ve identified for your conversational AI. If you’re building a brand bot or a promotional bot that is meant to be sticky and engaging, then engagement or retention might be your KPI. However, if you’re building a customer service chatbot you may want to consider throughput and non-repeated transactions as your success metrics.
Elle Chun mentions some primary KPIs her team at Sutherland Labs cares about including agent transfers, completion rates, and deflection rates. The team at Intuit drills into the percentage of interactions that the chatbot missed and how often it defaults to its fallback response. Ganz notes, “False positives are a problem, but they’re not as much a problem as false negatives.”
Quantitative metrics of success are important, but you also want to keep the qualitative data in mind given how much richer and more actionable conversational data is compared to web clicks and mobile taps. Bajorek says, “I’ve definitely seen some beautiful numbers that say users got through the whole chatbot flow, but when asked how users felt about the experience they said, ‘That’s creepy. I’m not doing that again.’” Falkson and her team at Amazon take the time to look through CSAT scores and diligently read customer complaints in order to address them and tie them back to their primary KPI, retention.
Optimizing the Experience: Baking in User Feedback
The process and turnaround time of incorporating user feedback varies depending on the size of the team and current work style, but the resounding point all speakers emphasized is the importance of user testing. Smaller teams, ones where conversational designers have direct access to developers, move a little more quickly. With larger companies, the bot is a part of a larger hole so Intuit looks at both customer data in aggregate and messages that the bot could not handle properly.
Usability testing is incredibly important to all of the panelists and they all conduct testing in various ways at different scales. Intuit conducts thorough customer interviews where they pay users to use their chatbot in-person and ask users for real-time feedback. Sutherland Labs also values watching customers chat with their bots in-person so they can gauge their reactions.
Big Brands: Creating and Maintaining Brand & Tone
Customers already have expectations of big brands when it comes to personality, voice, and persona so it requires a lot of thought and energy to extract the core of the brand and put it into a chatbot or a voice application. When creating the bot’s personality, think about how it feels about the brand’s subject matter, would it be formal in conversation, and how does it feel about the customers, etc. Ganz says you can use questionnaires to gather more information about a brand’s personality and use celebrities and characters as shorthands for descriptions.
While you build out all of the happy paths and hire a team of content designers, Falkson notes the importance of maintaining consistency within your organization because it is expressed through the consistency of your bot’s brand and tone. As the conversational AI team grows and changes, you’ll need to be vigilant about how the bot is falling into different designer’s hands. You don’t want the bot’s tone to change when there is a change in designers. An active way to maintain brand consistency is to have a structured personality guide (similar to a website’s style guide) that shows examples of how the bot would respond in different situations and how it would talk to customers.
Tools for Conversational Design
Below is a list of design tools and resources that the panelists referenced. There was an important distinction between some of these tools being used for prototyping, understanding the different high-level paths, and viewing the script line-by-line.
Figma — Sutherland Labs has used to create many POCs.
Meya.ai — Sutherland Labs uses to demo prototypes to clients.
Google Sheets — All panelists mentioned how valuable this is for viewing the script in detail.
Google Draw