No Golden Path:A Cautionary Tale of Quality and Biases

Narrative organization of information ties together storytelling in its many modalities. One archetypal expression of narratives we find in literary fiction. In this paper, we approach the elements of a successful narrative, and by extension storytelling, from the perspective of computational narratology. We are specifically interested in how to identify ’a good story’ and navigate in two dimensions of literary success: ‘Extrinsic’ and ‘intrinsic’. We investigate the association between the self-similarity of a sentiment story arc and review assessment of a literary work. In doing so, we direct our attention to known and unknown biases in the suggested dimensions of success to avoid reinforcing existing unwanted structures in our exploration of successful narratives.Quality assessment of literature is not a simple inquiry and is complicated by various factors. Literature is a complex linguistic phenomenon that conveys information indirectly, and readers have different aesthetic preferences, and, in general, there is a lack of robust scientific instruments for measuring literary quality. One noisy, but ecologically valid measure is quantitative ’reader reviews.’ On this account, a narrative is successful if readers rate it high. Such an ’extrinsic’ success criterion is tempting because it is relatively easy to access, reflects readers’ preferences in a natural setting, and its standardization appears trivial. A criterion that relies on reviews is however prone to several well-known biases, for instance, grading disparities between gender [1], ethnicity and race [2], which point to fairness challenges in classification of real-world data [3]. Instead of merely relying upon review annotation of the success of a story, we suggest paying attention to the inner structure of a story, the ’intrinsic success’.A recent theoretical paper has suggested that the affective coherence of a story, that is, the self-similarity of a sentiment story arc, functions as an index of a narrative’s intrinsic success [4]. A complementary empirical study has shown that affective coherence can detect canonical literature [5].While the use of computational narratology may seem compelling to minimize demographic disparities introduced by extrinsic success, it introduces less apparent and unknown biases. Genre, for instance, impacts a story arc and shows complex interactions with psychological propensities, aesthetic evaluation, and gender [1]. In this work, we examine the association between the internal sentiment structure of a work and known biases. Socio-cultural norms may also play an important role in introducing unknown biases even at the methodological level, such as representational biases of reader types.In sum, there is no golden path to identify successful storytelling, that is, no single path that optimizes both quality assessment and bias response. Instead of relying on single dimensions of success, either compelling computational approaches or accessible standardization, we suggest a deliberate combination of dimensions and approaches which includes choices about bias acceptance. We see a multitude of possible trajectories, each of which implies different choices of known and unknown biases.