Testing a Chatbot End-to-End with Perfecto and Botium

This tutorial provides step-by-step instructions how to setup Botium Box with Perfecto Mobile and Perfecto Web to run continuous End-To-End tests for a chatbot published on a website.

Test Levels

Due to the nature of conversational AI it is vital to understand the concept of test levels to design efficient test strategies. This picture shows a typical chatbot architecture:

  • There is a user frontend hosted as chat widget on a website
  • The frontend connects to a backend service (often called Orchestration Service) with web-protocols — HTTP(S), JSON, Websockets
  • Somewhere behind, there is an NLP model to convert user input to structured data with intents and entities
  • An additional component is handling the dialogue sessions.
  • And finally, there are some kind of business services, often backed by business databases

Botium supports End-To-End-Testing with a Webdriver Connector, combining the power of Selenium’s web browser automation with Botiums chatbot testing superpowers. The chatbot is tested by doing pointing and clicking and typing on a website just as a real user would do. As part of a test strategy, there are several scenarios that can and should be covered with End-To-End-Testing:

  • Validating browser/client compatibility
  • Smoke Tests before going live

In this article i will use one of the “Corona Bots” out there providing information about the Covid-19-Pandemic. It has a clean user interface with some quick-responses, a text input field and a chat history window.

Step 1: Connect Botium Box to the Perfecto Cloud

Follow the instructions in the Perfecto Documentation to connect Botium Box to the Perfecto Cloud.

Step 2: Choose Device or Browser Configurations

Follow the instructions in the Perfecto Documentation to register a set of devices or browser configurations you want to test with.

Step 3: Connect Botium Box to the Chatbot Website

Regardless what channel the chatbot is published on, chances are high that Botium can connect to it – there are over 30 technologies supported right now, and the number is growing. In this case, we can select the WebdriverIO technology, tell Botium the web address where the chatbot is published.
Note: this chatbot is using the Microsoft BotBuilder Webchat widget, which is nativle supported by Botium. For other, custom widgets it is required to tell Botium the Selenium selectors of the input and output DOM elements.

You can click the Say Hello button to check connectivity immediatly – Botium uses an integrated headless chrome to launch the website, sends a simple Hello to the chatbot and waits for the response.

Step 4: Define the Botium Test Cases

Test Cases in Botium are defined in a domain-specific scripting language called BotiumScript – plain text files holding the logic to send user input to the chatbot and what to expect as outcome.

ask-for-corona-protection

#bot
Hallo, bitte stellen Sie mir eine Frage zum Corona-Virus und ich versuche sie zu beantworten!

#bot
Oft gestellte Fragen:
BUTTONS WOHER KOMMT DAS VIRUS?|WELCHE SYMPTOME GIBT ES?|WIE KANN MAN SICH SCHÜTZEN?

#me
WIE KANN MAN SICH SCHÜTZEN?

#bot
Um generell die Ausbreitung von Infektionskrankheiten der Atemwege zu vermeiden

This sample test case named ask-for-corona-protection will first wait for two greeting messages from the bot (also see screenshot of the chat window above) – the second one should contain some clickable buttons – then send some text to the chatbot and wait for a response. There is also a test case designer in Botium Box for visualizing the conversation flow.

Several test cases are collected in a Test Set in Botium Box.

Step 5: Prepare Botium Test Project

The basic execution unit in Botium Box is called Test Project. It combines everything from the previous steps:

  • The chatbot channel to use
  • The test cases to run
  • The devices or browser configuration to launch the chatbot on

Open the Quickstart Wizard and follow the 3 steps.

On the final step, don’t click on Save and Start Testing, but on the Save button – there is one additional step to do before launching our first test session.

Step 6: Connect Botium Box to the Perfecto Test Reporting & Analytics Dashboard

Perfecto has a very powerful test reporting & analytics dashboard. We have to connect Botium Box to this dashboard to keep Perfecto up to date about executed test cases and test results. Open the Settings tab in the Test Project, expand the Test Execution Settings and add PERFECTO-REPORTING to the Registered Components list.

Step 7: Launch a Test Session

Now head over to the Test Project dashboard and click the Start Test Session Now! button. You can watch the test session while it is running.

Step 8: Inspect Results

When the test session is finished, you can start to inspect the results in Botium Box.

For each test case, there is the full visualized conversation transcript shown, as well as a screenshot from the browser. When expanding a failing test case you can immediately see the reasons for failing (in this example, there is a problem with the german umlaut in Firefox on Windows).

Head over to the Perfecto Test Analysis Dashboard to have deeper insights into recent test sessions, devices lists and more.

You can follow the conversation flow of each test case and even watch the full screen recording there in order to trace down any failures.

Next Steps

There are so much more things to discover in Botium Box, this was just the beginning.