Next Steps in Data-to-Text Generation: Towards Better Data, Models, and Evaluation

There is a tremendous---and exponentially increasing---amount of data available in the modern world. Data, in this context, means non-linguistic information: weather information, stock market numbers, sports statistics, etcetera. All these types of data have the potential to transform our lives, by improving our world knowledge, or by providing us with the support to make decisions and change behavior. However, in many cases, raw data is too expansive and too complex to understand. To make such data understandable, we would need tools that help put a spotlight on the important insights that they may contain. Data-to-text systems are especially viable as a tool for processing raw data. These are computer programs that can convert data (e.g., local day temperatures, amount of precipitation, wind speed, etc.) to an understandable, natural, fluent text (a weather report). Summarizing information in a text format has the potential to make underlying data comprehensible, even more so than statistics or data visualizations, because text provides the opportunity to explain the background, context, and caveats associated with the underlying data. This also explains why the industry, mainly news media, are increasingly interested in data-to-text systems. Surprisingly, though, is that when news media opt to use such a “robot journalist”, they generally prefer one that is technologically not very advanced. The common data-to-text systems in the industry use a template-based approach in combination with simple, handcrafted rules that determine which template to apply in which situation. Academia has started to move away from these “traditional” data-to-text systems. With the introduction of self-learning, machine learning models, that have the potential to learn these (previously) handcrafted rules without human interference. However, while these models show more potential, they are currently difficult to implement⁠, mainly because of a lack of data we can use to train machine learning systems. Furthermore, they are unpredictable. That is, they require a lot of data and we humans cannot really control the texts that these systems produce, leading to all kinds of inaccuracies, nor do we have the tools to evaluate the quality of these texts which makes these systems even more difficult to understand. This dissertation attempted to address the challenges of machine learning data-to-text systems that hamper real-world implementation, by developing new systems, better data, and better evaluation methods that could bridge the existing gap. In terms of new systems, a machine learning data-to-text system was developed that contained explicit steps in the data-to-text conversion process, which was shown to produce more accurate and predictable texts compared to its counterpart that did not contain any explicit steps. In terms of better data, a new dataset was developed that contained more realistic texts compared to what is currently out there. Furthermore, a machine learning system was developed that is able to automatically create new data so that the issue with a lack of data can be resolved. Finally, a critical review of current evaluation practices in the field was done, to see how well we are able to assess whether a text is qualitatively good. This review showed that there is no standardization in terms of evaluation methods: there is high variability in the methods that researchers use to evaluate text quality. This was addressed by setting up a list of recommendations how to conduct a proper evaluation study. The dissertation showed that there is the potential of these machine learning methods to replace the current data-to-text systems that are being used in the industry. However, further development is required in terms of making these machine learning data-to-text systems produce texts that we have some control over. Additionally, more efforts are required in developing evaluation methods that indicate whether a text is good.