AI Assistant to Improve Experimentation in Software Startups Using Large Language Model and Prompt Engineering

Software startup is a unique type of company with unique characteristics. On the one hand, they must offer innovative products appealing to customers to generate revenue and survive, but on the other hand, they are limited in resources, time, and experience. During the new product development, it is important to experiment with their original ideas. However, doing a meaningful experiment requires resources and challenges. A study on failed software startups shows that, despite its importance, many software startups skipped or did not experiment with their ideas. The study identifies 25 inhibitors spread in five experimentation stages. In the last few years, Large Language Models (LLMs) have become a popular technology. The advancement of LLM has made it adopted into many parts of the software development cycle. Studies show that LLM also has been used to generate new innovative product ideas and to manage innovation. However, there is no investigation into the possibility of utilizing the power of LLM to help software startups do experimentation. Interactions to an LLM are done through prompts. During the interaction or session, a user will send one or more prompts in a zero-, one-, or few-shots to an LLM agent. Unfortunately, learning and using prompts effectively requires time and resources, things that software startups are scarce with. In this project, we aim to help improve the experimentation process and address the inhibitors by leveraging the power of LLMs. There are five initial research questions and studies planned in the project. In the first step, we will investigate current experimentation practices, challenges, inhibitors, and the strategies used to circumvent them. Secondly, we will investigate how AI has been used in today's experimentation. Then, we will investigate the set of measurements available to measure the success of an experiment. The next step is to investigate how to support experimentation using LLMs followed by a validation sequence. T