The Algorithm

Can statistics help historians to identify the events that are most distinctive of a particular era of time? This essay explores the use of a distinctiveness algorithm from library science for measuring the distinctiveness of manuscripts, tf-idf, recast as "tf-ipf" for the study of the terms most distinctive of historical periods. In a case study, tf-ipf is applied to the text of Hansard's Parliamentary Debates, varying the "period" from a 20-year horizon to a 6-month or one-day horizon. It is shown that the algorithm's assessments of what is most distinctive of 20-year and 10-year periods largely matches the consensus of British historians, while debates that held parliament's attention for six months or fewer have largely fallen beneath the threshold of scholarly attention. Attending to concerns that took up parliamentary debate for six months or fewer, the essay argues that tf-ipf thus presents a metric of parliamentary attention that mirrors the exclusions of class, allowing scholars to retrieve a timeline of when the concerns of the distillers, bleachers, dyers, Chartists, crofters, and miners reached a national debate. More generally, tf-ipf represents an important new tool for discovering the distinctive aspects of historical periods based on past experience, not historical bias.