/blog
Weather commands how we live our lives. While we cannot control the weather, we can help people and businesses cope with it better. Having easy access to the most accurate weather forecasts and warnings makes life a lot easier, to say the least. For AccuWeather, millions of people depend on its weather forecasts not just for convenience, but for the protection of life and property. In order to ensure users are able to receive weather updates with Superior Accuracy™ under any condition and in any location, global weather and digital media leader AccuWeather uses a powerful combination of weather data, world-class models and the expertise of over 100 expert meteorologists, as well as the latest innovations and technology. But even the best weather forecasts have little or no value if they are not communicated to people in a way that helps them understand the impacts of expected weather.
Senior Software Engineer Stephen Keefer, who has been with AccuWeather for over 13 years, has been focused on bringing AccuWeather into another new space to help AccuWeather reach more people and communicate the weather and its impacts: conversational interfaces.
“AccuWeather is always on the leading edge of what’s coming next,” said Keefer. “Voice and chatbots are the next iteration.”
A new DAWN
With that in mind, a haphazard exploration of a voice app or a chatbot was not going to cut it. If AccuWeather was going to fully embrace voice technology to make weather forecasts and warnings even more accessible to people, it would have to devote resources to exploring multiple platforms and building an enhancement that worked on every one.
When AccuWeather launched the Data and Weather Narrator (DAWN), it was made available on multiple platforms and channels, including Facebook Messenger and Google voice devices. The virtual assistant easily interacts with over 10,000 users in a day across all its platforms and its Natural Language Processing (NLP) model through LUIS supports over 50 Intents.
This volume of users and Intents translates to a lot of data for Keefer and the AccuWeather team to comb through. Getting access to that data and making sense of it requires the right analytics solution. When the project was initially launched two years ago, AccuWeather used a homegrown solution while it built out the technology that would ultimately become DAWN.
As the volume of data grew, AccuWeather quickly realized it needed a new solution. All interactions were logged into a database, but they still had to come up with a method of extracting information and organizing the data. It was too complex for its systems at the time.
“We weren’t developing a tool to support our product,” Keefer said. “It was going to be a separate product to support our product, and that wasn’t feasible.”
It was time for a new solution, a tool that required less work to use and provided more insights. The weather and digital media leader was also looking for something that had visibility into chatbot and voice-specific metrics like Intents, entities, and utterances. With that in mind, the team searched for the right partner and an analytics platform that would answer all its questions and meet its high standards and evolving needs.
A few months later, AccuWeather found Dashbot.
Asking for the weather is complicated
Keefer started out with a rudimentary exploration of the tools available in Dashbot and found that he had easy access to all the metrics he tried to track with their homegrown solution.
With the previous solution, each report had to be painstakingly created manually and it took some time to process. By just opening the Dashbot dashboard, he was able to see user retention week-over-week, age, and locale for the users. Pleased with this newfound near instant visibility, Keefer embarked on a deeper integration with Dashbot and moved to an Enterprise plan.
AccuWeather’s user base runs the gamut from casual users, who just want to know how the weather forecast and its impact on their lives, such as how to dress for the day, to weather aficionados. Though a casual user will not ask a long, complicated question, there are still a lot of different ways to ask for the weather. In order to train its NLP model to map each utterance to the right Intent, Keefer needed ample, high quality training data.
The weather aficionados, on the other hand, will ask questions containing multiple entities such as location, date, or a specific weather location. Each additional entity contributes to the inherent complexity of answering a simple question about the weather. In order to train these Intents, even more training data is required.
The Unhandled Report makes quick work of this issue. Identifying the utterances that are triggering the “not handled” Intent is the first step to improving the rate of accurate responses. If Keefer sees an utterance that should be covered by the current list of Intents, he adds it to the LUIS model so the next user can get his or her questions properly answered next time. Alternatively, if Keefer sees a lot of unhandled Intents that fall in the same category, it shows him that it might be an Intent worth building out.
In addition to immediate fixes, Keefer routinely pours over the Dashbot reports for unusual interactions. The Sentiment Analysis feature highlights conversations worth a closer look. If a user expressed negative sentiments while interacting with DAWN, finding out why a user was frustrated or upset will improve the experience for the next person. Doing a deep dive into these conversations will alert him to areas that could and should be improved.
Daily, Keefer keeps a close eye on errors through Azure Application Insights for live updates on DAWN. Application Insights tags conversations that went wrong with a red flag, but in order to fix the failure, he needs additional functionality that would provide context. Dashbot’s Transcript feature gives him that visibility. Dashbot exactly reconstructs each session with all the richness of the original transcript and allows the AccuWeather team a more nuanced understanding the user’s experience.
Transcript Search allows the AccuWeather team to quickly locate any conversation using just a conversation ID. Instead of manually digging through DAWN’s cache of conversations, Search quickly brings forth the right one. These buggy conversations provide insight into what led to failure and are a necessary part of the optimization process. Having easy access to this data means users would not have to wait long for a bug fix, and more users would leave the experience satisfied and happy to return.
Empower users to share their needs
Dashbot’s reports have also given AccuWeather the power to take initiatives. When DAWN was first implemented, the audience comprised 18- to 24-year-old males, so that’s who they designed the content for. DAWN was chatty and meandered to giving the weather forecast, creating more of a conversational feel to appeal to this particular audience. However, the AccuWeather team noticed after a while that usage and enrollments in weather alerts were declining and started to investigate.
By taking a close look at the Demographics Report, they found that 18- to 24-year-old males no longer made up a majority of its users. Instead, users were older adults who had no inclination for idle chatter. Making this group wait for the weather forecast meant losing them as users.
In an effort to reduce churn, AccuWeather changed the content to operate more efficiently and better reflect the needs of this wider audience. By removing a lot of the text and adopting a more “to the point” tone, they were able to retain more of their growing user base and enrollment to weather alerts started to rise again.
In addition to improving current processes, Dashbot’s data also helped AccuWeather in its commitment to be responsive to user needs with the development of a new feature. Word Cloud provides an easy-to-interpret visual of the most commonly used words said to DAWN, meaning deep insight can be gleaned from just a glance. The larger the word, the more frequently it’s used.
In AccuWeather’s Word Cloud, the words ‘radar’ and ‘satellite’ were quite large, indicating a lot of their users were asking for the satellite image of the weather forecast. AccuWeather took note and recently released the feature that allows that to happen. That instinct was a home run. Within its first three weeks of release and with no promotion, the radar Intent was triggered over 3,000 times.
“We wouldn’t have been able to implement that feature this quickly without Dashbot,” said Keefer.
Expanding DAWN
AccuWeather’s mission of making accurate weather forecasts more accessible and understandable for the protection of life and property doesn’t stop here. According to Keefer, he has plans to expand DAWN into multiple languages based on locale. Each new language comes with its own set of complexities, which makes building and optimizing DAWN even more challenging. But Keefer and the rest of the AccuWeather team are undaunted. With Dashbot’s analytics at their fingertips, they’ll be able to build a great experience for their users wherever they are globally.
About Dashbot
Dashbot is an analytics platform for conversational interfaces that enables enterprises to increase satisfaction, engagement, and conversions through actionable insights and tools.
In addition to traditional analytics like engagement and retention, we provide chatbot specific metrics including NLP response effectiveness, sentiment analysis, conversational analytics, and the full chat session transcripts.
We also have tools to take action on the data, like our live person take over of chat sessions and push notifications for re-engagement.
We support DialogFlow, Alexa, Google Assistant, Facebook Messenger, Slack, Twitter, Kik, SMS, web chat, and any other conversational interface.