Insights for training AI chatbots

This post is written by DJ Moody, Client Strategy Analyst at Robots & Pencils, and inspired by a recent experience helping one of our clients with creating and training an AI chatbot.

“Alright listen, Sparky…”

My father tends to be laid-back. However, there are clear indicators when he’s getting frustrated. These are so predictable that the whole family knows when a customer experience goes awry. First, he refers to the person as “Buddy”. If the situation doesn’t improve, they soon become “Sport”. With each step down this path of names, the chances the agent will keep the business drop precipitously. The final moniker is “Sparky”. No one comes back from Sparky.

It’s a familiar experience. A customer service discussion goes poorly. We become frustrated. Eventually, if it’s bad enough, we decide to spend our money elsewhere. After those conversations, our opinion of the company changes. We blame the business.

In comparison, the same isn’t true with static content. If we can’t find an answer in an instruction manual, we get a little annoyed, but our view of the company rarely shifts. Yet when we talk with a person who doesn’t have the answer, frustration bleeds over into our impression of the company in a much different way. A human failing tells us the company failed. The company doesn’t care. Basically, the company is bad.

More rare, but more powerful, is the successful customer service conversation. The one where you feel heard afterward, where the other person takes time to understand your need and works to get it addressed. Even if you don’t get the answer you want, you gain a positive perspective on the company. The experience tells us the company is good; they put resources into their customers. They care about us. Reading helpful documentation doesn’t offer that same gut-level reaction.

It takes a person for something to feel personal. 

Chat-based AI attempts to mimic a conversation with a person. This brings with it many of the benefits of having a human involved. Markedly, it is an opportunity to show the customer you care and to develop a personal connection. This also opens up a lot of the dangers. It takes a person for something to feel personal, and we are creating a digital facsimile of a person. So the question becomes, how do we make our fake person seem genuinely caring?

Artificial intelligence, real heart

Experiences can be designed, whether on a website, in an app, or with a healthcare provider, sales professional, or customer service agent. The major difference between a digital product experience and a human experience is flexibility. The former we expect to be relatively static. It’s primarily up to us to find the right page or tool. The latter we expect to change based on our actions. If we’re angry we expect a person to show empathy and change their approach. We don’t expect an app to understand us; we do expect that of a person.

As with all programming, what you get from a chat experience depends on what you put into it. We tend to focus on training AI chatbots with the right data. Undoubtedly, we want them to have the correct answer to any question they might be asked. That might be enough if users interacted with them as they do a static resource. However, we want the benefits of AI accuracy and efficiency with the personal connection of human interaction. Just as the best human communicators have training in both information and interpersonal skills, we need to include the same when training AI.

3 takeaways for training AI chatbots from the intersection of science and the humanities

1. Legitimize the need.

Above all, we want customers to know we care. Like compassionate humans, our chatbots can show this care by simply acknowledging there is a need and a desire to help. We don’t have to promise we can solve the problem. At this point, we just need to make it clear we know there is one. If sentiment analysis is available, adding that detail is even better.

2. Confirm and clarify.

Great human communicators reflect back to the customer what they understand about a request and ask questions to clarify. Communication is messy and questions clean it up. Our bots need to do the same.

3. Set and reinforce next steps.

One of the more frustrating parts of customer service, for both the consumer and agent, is when a solution is offered but missed steps cause it to fail. We can reduce this likelihood by summarizing the discussion, including all steps, before signing off.

Developing and training AI solutions for your customers

These are just a few of the many ways your AI solutions can reflect the care and compassion you have for your customers. In order to fully accomplish our goal of more human interactions between AI and customers, we must carefully consider each solution from all sides. Poor communication can make even the most well-informed agent–human or bot–seem cold and uncaring. Conversely, bad data means even the best-behaved agents will be useless. For truly effective agents, those who can increase both customer knowledge and sentiment, we need to live at the intersection of science and humanity. 

Every team developing a chatbot should include specialists in data science and AI as well as experts in communication, user experience, and customer care. In the case of Robots & Pencils, our Robots handle the code, our Pencils ensure an excellent experience, and our Ampersands bring it all together. To learn more about how Robots & Pencils helps organizations approach AI technology with a human-first approach, visit our AI & data science page or drop us a line at hello@robotsandpencils.com. 

Perhaps this story is familiar to you…

Your team has been tasked with designing and building a groundbreaking software product that focuses on putting the end user — the person who will use the product — at the center of the design and development process. The stakeholders want it delivered within 12 months.

Six months into the project, the hunger for innovative, user-centered features seems to wane.

The delivery team adopts a rapid agile cadence that values predictable delivery dates and velocity rather than experimentation to find the best solution for the user.

The design team attempts to get ahead of development in order to ensure user insights lead the build of new features. Unfortunately, priorities shift after each deployment, and much of that work gets thrown away.

Over time, a chasm forms between user-centered advocates and delivery. Their ideas are just too risky and unpredictable for the mission: get the next release out the door.

An all too familiar tale, but with an ending we can change. I think there is a way to get the most out of the user-centered design and Agile processes while keeping to budgets, timelines, and a consistent delivery cycles.

First, though, why is this tale all too familiar?

A bias towards reliability and predictability

We’re naturally inclined to have a bias towards reliability, predictability and reducing risk. And Agile development does this by design.

The sprint-based nature of Agile development makes its output extremely predictable — yielding releases every couple of weeks. But it works best when requirements are understood upfront and can be accurately estimated.

However, teams often overestimate new features and unfamiliar technical challenges to account for the delivery uncertainty. This results in teams prioritizing the most predictable features over the most innovative ones.

In contrast, user-centered design focuses on the people who will use the software, and how they can easily and efficiently achieve their goals. It leverages primary research — actually getting insights and feedback from future users — to discover previously unknown friction points in the user experience. It seeds the invention of new experiences that offer that far better way.

While some of the ideas that come from this process can offer a 10x improvement in the experience, they are also unproven and often unfettered by technical and business process constraints. The discovery-driven nature of the process makes the output impossible to predict.

An incompatible match

The issue is that user-centered design and Agile development are incompatible.

• User-centered design aims for maximal astonishment — let’s see if our users can tell us something we didn’t know.
• Agile aims for minimal astonishment — let’s have as few surprises in our development process as possible.

To steal a concept from physics, user-centered design prioritizes voltage — a measure of potential energy. Agile development prioritizes amperage — a measure of the flow of current.

And in the same way that you wouldn’t want to plug your TV into a high-voltage power line, the friction between Agile and user-centered design comes when you directly connect the two methodologies.

While the results might be less spectacular than an exploding TV, the failures are just as predictable:

• Innovative ideas lag development timelines and create project delays.
• Innovative ideas get chopped in favor of timelines.
• Through long hours, a talented team squeezes in a couple of extra features deemed “top priority.”

In all three cases, the drive for innovation eventually wanes:

• Unmet delivery promises undermine organizational and investor trust in the team’s ability.
• Continuous cutting of new ideas demoralizes the team into a “good enough” mentality and opens up questions about the ROI of exploring innovative ideas to begin with.
• Unsustainable hours create organizational churn and the talent carrying the greatest load walk out the door.

Still in order to successfully invent new user-centered solutions AND deliver them, we need the strength of both. We need a way to convert the high potential energy output of user-centered design into the high current that the Agile process demands.

Treat user-centered design as investigation, not validation

User-centered design is a process that requires a lack of preconceptions and an open mind. The goal is to understand the root causes of user behavior in order to formulate a new, better approach. Keeping a hypothetical solution in mind through the process creates a cycle that reinforces our original assumptions and often blinds us to other opportunities.

So, when a request comes in to “redesign an app,” start by abstracting that request into the underlying goals that the user accomplishes while using the app. Then, use primary research to validate that those goals are real and dig into how they are accomplished today. You may learn that users don’t care about what you think they do. You might also learn that an app isn’t even the right enabler for their goal.

At the end of this process, you will get a list of things to build that will improve the user’s experience. They just might not be the ones you expected.

Only feed de-risked features into the Agile process

In order to keep the Agile delivery process moving efficiently, ensure that any proposed features/stories are well understood both in terms of their requirements as well as their technical complexity.

Before ever estimating a feature, you should know:

1. That implementing the feature will generally increase user perception of the experience
2. The way to solve any key technical or data hurdles
3. Confidently, how long the feature will take to build

If a feature doesn’t meet all of those criteria, then it could easily derail the predictability that is the key benefit of Agile development.

De-risking features enables teams to estimate without fear because the basic how of implementation is already well understood.

Use experimentation to connect user-centered design and Agile

The un-vetted output of user-centered design makes it a poor input into an Agile process that works best with low risk requirements. To bridge the two, we need an intermediary process that takes fledgling ideas and systematically de-risks, prioritizes and roadmaps them at a program level.

This middle phase — experimentation — acts as the glue.

Prototype, test and rapidly iterate on proof of concepts (POCs) with users to ensure that your features will have the desired impact on the user experience. These experiments provide a view into the potential ROI. By providing a clearer picture into the new experience, they also can generate investment interest at the executive level.

Similarly, technology experiments (often referred to as technical design spikes) can be used to de-risk complex technical and data problems. In this phase, the priority is not delivering a minimum viable product. Instead, the goal is taking the riskiest assumption and exploring the solution space.

At the end of experimentation, these pre-vetted features can be fully story-mapped and prioritized for delivery — the perfect input to the Agile delivery phase.

Run all three phases concurrently as an innovation program

While the experimentation phase ensures compatibility between user-centered design and Agile, it is important to note that these methods run at different speeds.

• User-centered design takes a long time to conduct, but yields a high volume of opportunities.
• Experimentation can be done quickly but the output may or may not produce viable results.
• Agile generates small chunks of production-ready functionality at a regular pace.

To keep all processes operating at full speed, allow each of these phases to run independently and concurrently over time. Each phase generates a queue of work for the next. User-entered design creates a queue of experiments. At the same time, experimentation creates a queue of user-vetted, de-risked features for implementation.

With a never-ending list of to-dos in all three work streams, the program will operate at full efficiency. And innovative, user-centered features will flow like clockwork into your products and experiences.

Tyler Klein is the Executive Experience Director at Robots and Pencils. Physics major turned HCI specialist, he uses what’s new to build what’s next and offers far better ways to interact with the world around us. Special thanks to Chris Chew, Jamie Reid, Mike Greening, Reid Sheppard and Aaron Slepecky for their contributions.