Evaluating the Effectiveness of Open Source Chatbots for Customer Support

Chatbots are becoming increasingly popular in various industries, and thereare many options available for businesses and organisations. Several studieshave investigated open-source chatbots and identified their core strengths,implementation, and integration capabilities however few have investigatedopen-source chatbot frameworks and libraries in a specific use case such asmedicine. The project's objective was to evaluate a selection of chatbots ormore specifically two frameworks: Botkit and Rasa, and two libraries:ChatterBot, and Natural which was utilised together with Botkit and alanguage model which is DialoGPT. The evaluation focuses specifically onaccuracy, consistency, and response time. Frequently asked questions fromthe World Health Organization and COVID-19 related Dialogue Datasetfrom GitHub were utilised to test the chatbots' abilities in handling differentqueries and accuracy was measured through metrics like Jaccard similarity,bilingual evaluation understudy (BLEU), and recall oriented gistingevaluation (ROUGE) scores, consistency through Jaccard similarity betweenthe generated responses and response time was taken to be the average timefor a response in seconds. The analysis revealed unique strengths andlimitations for each chatbot model. Rasa displayed robust performance inaccuracy, consistency, and customisation capabilities if the chatbot works ina particular topic with acceptable response times. DialoGPT demonstratedstrong conversational abilities and contextually relevant responses withtrade-offs in consistency. ChatterBot showed consistency, though sometimesstruggled with advanced queries, and Botkit with Natural stood out for itsquick response times, albeit with limitations in accuracy and scalability.Despite implementation challenges, these open-source frameworks, libraries,and models offer promising solutions for organisations intending to harnessconversational agents' technology. The study suggests encouraging furtherexploration and refinemen