Upskilling Test Engineers for Chatbot Projects

If you have a hammer, every problem looks like a nail.

With Botium, we are currently defining the industry standard for testing chatbots. In our support and developer channels we are regularily receiving questions like:

The conclusion to draw from these questions is: the test engineers learned how to test websites with Selenium and smartphone apps with Appium in the past, and now they try to apply this valueable knowledge again — neglecting the fact that chatbots are a new kind of apps that require new kind of tools (like Botium).

You can read about the most important differences in one of my previous blog posts.

With Selenium and Appium, we are talking about End-2-End testing (E2E) — simulating the full user experience on a graphical user interface. Those tests

So here are my recommendations for test engineers how to get going when asked for testing a chatbot.

API First

The most important metric for a chatbot is: is it able to do a meaningful conversation with a client ? In every chatbot project team there are conversation designers which, well, design the conversations that will make up the final user experience. The chatbot engine is trained (or coded) to provide the logic for these conversations.



And this is the place to start testing: make sure that the conversations are working as designed, from a content perspective. You can read more about conversation flow testing in the Botium docs.

One important skill to have is knowing BotiumScript, the scripting language to define conversation flow test cases.

Testing the NLP engine

Most chatbots have some kind of natural language processing (NLP) component as part of the processing pipeline — it enabled users to communicate with the chatbot in natural language, and that’s what actually makes up a chatbot. As a test engineer it is your job to explore the limits of the NLP engine, and this requires basic skills in machine learning concepts, such as

You can read about it in my blog series Quality Metrics for NLU/Chatbot Training Data.

E2E Smoketest

Testing the end-user experience on user interface level is an important part of a testing strategy. When doing it right you now have the confidence the conversation flow and the NLP component are doing their work, so it is now time to add some user interface testing to the mix. The recommendation is to

The good news is that here test engineers can shine with the existing knowledge on Selenium and Appium!

Read on in the Botium Wiki how to setup this with Botium!

Non-Functional Testing

Finally, there are also non-functional tests like performance tests and security tests to add to the test mix. Opposed to the other test types those are typically done on certain milestones in the project.


A new generation of apps such as chatbots require a new generation of testing tools, like Botium. Test engineers have to develop additional skills for testing conversational interfaces like chatbots.