Search

Institutional discovery environments now serve as central resource databases for researchers in the academic environment. Over the last several decades, there have been numerous discovery layer research inquiries centering primarily on user satisfaction measures of discovery system effectiveness. This study focuses on the creation of a largely automated method for evaluating discovery layer quality, utilizing the bibliographic sources from student research projects. Building on past research, the current study replaces a semiautomated Excel Fuzzy Lookup Add-In process with a fully scripted R-based approach, which employs the stringdist R package and applies the Jaro-Winkler distance metric as the matching evaluator. The researchers consider the error rate incurred by relying solely on an automated matching metric. They also use Open Refine for normalization processes and package the tools together on an OSF site for other institutions to use. Since the R-based approach does not require special processing or time and can be reproduced with minimal effort, it will allow future studies and users of our method to capture larger sample sizes, boosting validity. While the assessment process has been streamlined and shows promise, there remain issues in establishing solid connections between research paper bibliographies and discovery layer use. Subsequent research will focus on creating alternatives to paper titles as search proxies that better resemble genuine information-seeking behavior and comparing undergraduate and graduate student interactions within discovery environments. [ABSTRACT FROM AUTHOR]

Discovery environments are ubiquitous in academic libraries but studying their effectiveness and use in an academic environment has mostly centered around user satisfaction, experience, and task analysis. This study aims to create a quantitative, reproducible framework to test the relevancy of results and the overall success of Washington State University's discovery environment (Primo by Ex Libris). Within this framework, the authors use bibliographic citations from student research papers submitted as part of a required university class as the proxy for relevancy. In the context of this study, the researchers created a testing model that includes: (1) a process to produce machine-generated keywords from a corpus of research papers to compare against a set of human-created keywords, (2) a machine process to query a discovery environment to produce search result lists to compare against citation lists, and (3) four metrics to measure the comparative success of different search strategies and the relevancy of the results. This framework is used to move beyond a sentiment or task-based analysis to measure if materials cited in student papers appear in the results list of a production discovery environment. While this initial test of the framework produced fewer matches between researcher-generated search results and student bibliography sources than expected, the authors note that faceted searches represent a greater success rate when compared to open-ended searches. Future work will include comparative (A/B) testing of commonly deployed discovery layer configurations and limiters to measure the impact of local decisions on discovery layer efficacy as well as noting where in the results list a citation match occurs. [ABSTRACT FROM AUTHOR]