Detecting problems of a chatbot and fixing them: OpenAI GPT-3 tricks and tips

All chatbot testers are dreaming of two buttons. One for detecting all problems of a chatbot, and another one for fixing them all.

With OpenAI we were able to add some nice features to Botium Box which are going in that direction.

  • We are using Open AI to guess what can be the next message in conversation
  • We integrated a new, multilingual paraphraser which takes the risk while generating new alternatives.

Here are some tips and use cases that we learned while developing those features.

Normalize text

Documentation of OpenAI says that it is better to check spelling mistakes in text. But we can go on. Text has to be as strict as possible. Some samples:

  • Avoid using enters. For example we can confuse OpenAI if we allow new line in the “human” section of a chat.
  • Terminate the sentences with dots. (OpenAI will deal with it as a sentence, won’t continue it for example)
  • Avoid unnecessary information. For example we distinguish between user writes the text “pizza”, or just pushes the “pizza” button. Second one is something like this: “#user button:pizza” for us. But this information can be confusing for OpenAI, it’s better to use just “#user pizza”.

Use enter on the end of the prompt correctly

This:

Human: Hello, who are you?
AI: I am an AI created by OpenAI. How can I help you today?
Human: Can I ask you

and this:

Human: Hello, who are you?
AI: I am an AI created by OpenAI. How can I help you today?
Human: Can I ask you

is a big difference for OpenAi, but it is just an extra enter. In the first case OpenAI continues the human message, in the second it generates new AI message.

Multilingual translator

Building a multilingual translator is not difficult. There is a sample for translator in Open AI playground. We can use the same parameters and data structure, but with multilingual samples:

English: See you later!
French: À tout à l'heure!

German: Ich möchte Geld überweisen.
English: I want to transfer money.

Russian: Я хотел бы заказать пиццу.
German: Ich möchte Pizza bestellen

That works well, but we have to define explicitly the language of the source text. We will overcome this restriction in the next step.

Multilingual translator, basic language is not specified

In order to do it, we have to restructure the prompt a bit:

Text: See you later!
LanguageOfResult: French
Result: À tout à l'heure!

Text: Ich möchte Geld überweisen.
LanguageOfResult: English
Result: I want to transfer money.

Text: Я хотел бы заказать пиццу.
LanguageOfResult: German
Result: Ich möchte Pizza bestellen

But lets play with it a little bit more

Multilingual translator and language detector

If we add a new field, then we can ask OpenAI for language detection:

Text: See you later!
LanguageOfResult: French
Result: À tout à l'heure!
LanguageOfText: English

Text: Ich möchte Geld überweisen.
LanguageOfResult: English
Result: I want to transfer money.
LanguageOfText: German

Text: Я хотел бы заказать пиццу.
LanguageOfResult: German
Result: Ich möchte Pizza bestellen
LanguageOfResult: Russian

It is a nice experiment. Sadly it does not recognize language always good, but reveals us some facts:

  • It is possible to teach OpenAI parameters (Text, LanguageOfResult), and return values (Result, LanguageOfText) Of course they are not parameters, and return values for OpenAI, but for us.
  • OpenAI can solve more complex tasks. What is a two step solution for us, it is maybe not for OpenAI.

Paraphraser

Sure, there are more ways to create a prompt for a paraphraser. There is no best solution, each has its own pros, and cons. Our paraphraser is very simple, but it does what we need.

 

Pros:

  • It is open, it finds really nice new alternatives.
  • Prompt is just 2 rows (less cost)
  • It is not stuck to any language. (Some prompts, like our translator have a training section, and a request section. If we create some ‘hard coded’ training sections to paraphrase in english, then we get a request to paraphrase german sentences, then we will have a prompt in multiple languages. It can be misleading for OpenAI)

Cons:

  • About half of the result is not a good paraphrasing. But it does not mean that it’s worthless! If we got “Can I replace my card if it is lost or stolen?” for “Someone took my card!” and “What is the procedure to report stolen card?” then we see that it does not fit. But in our case for a Banking chatbot it can indicate a new use case.