Search

Your search keyword '"Diamond, Michael"' showing total 2,699 results
Start Over Author "Diamond, Michael" Publication Year Range Last 10 years
2,699 results on '"Diamond, Michael"'

1. Evaluating Large Language Models with fmeval

Author

Schwöbel, Pola, Franceschi, Luca, Zafar, Muhammad Bilal, Vasist, Keerthan, Malhotra, Aman, Shenhar, Tomer, Tailor, Pinal, Yilmaz, Pinar, Diamond, Michael, and Donini, Michele

Subjects
Computer Science - Computation and Language, Computer Science - Machine Learning
Abstract

fmeval is an open source library to evaluate large language models (LLMs) in a range of tasks. It helps practitioners evaluate their model for task performance and along multiple responsible AI dimensions. This paper presents the library and exposes its underlying design principles: simplicity, coverage, extensibility and performance. We then present how these were implemented in the scientific and engineering choices taken when developing fmeval. A case study demonstrates a typical use case for the library: picking a suitable model for a question answering task. We close by discussing limitations and further work in the development of the library. fmeval can be found at https://github.com/aws/fmeval.

Published
2024

2. References

Author

Diamond, Michael A.

Published
2017

4. Index

Author

Diamond, Michael A.

Published
2017

5. Appendix

Author

Diamond, Michael A.

Published
2017

8. Cover

Author

Diamond, Michael A.

Published
2017

14. Notes

Author

Diamond, Michael A.

Published
2017

15. Preface

Author

Diamond, Michael A.

Published
2017

23. Mucosal adenovirus vaccine boosting elicits IgA and durably prevents XBB.1.16 infection in nonhuman primates

Author

Gagne, Matthew, Flynn, Barbara J., Andrew, Shayne F., Marquez, Josue, Flebbe, Dillon R., Mychalowych, Anna, Lamb, Evan, Davis-Gardner, Meredith E., Burnett, Matthew R., Serebryannyy, Leonid A., Lin, Bob C., Ziff, Zohar E., Maule, Erin, Carroll, Robin, Naisan, Mursal, Jethmalani, Yogita, Pessaint, Laurent, Todd, John-Paul M., Doria-Rose, Nicole A., Case, James Brett, Dmitriev, Igor P., Kashentseva, Elena A., Ying, Baoling, Dodson, Alan, Kouneski, Katelyn, O’Dell, Sijy, Wali, Bushra, Ellis, Madison, Godbole, Sucheta, Laboune, Farida, Henry, Amy R., Teng, I-Ting, Wang, Danyi, Wang, Lingshu, Zhou, Qiong, Zouantchangadou, Serge, Van Ry, Alex, Lewis, Mark G., Andersen, Hanne, Kwong, Peter D., Curiel, David T., Roederer, Mario, Nason, Martha C., Foulds, Kathryn E., Suthar, Mehul S., Diamond, Michael S., Douek, Daniel C., and Seder, Robert A.

Published
2024
Full Text
View/download PDF

24. Computationally restoring the potency of a clinical antibody against Omicron.

Author

Desautels, Thomas, Arrildt, Kathryn, Zemla, Adam, Lau, Edmond, Zhu, Fangqiang, Ricci, Dante, Cronin, Stephanie, Zost, Seth, Binshtein, Elad, Scheaffer, Suzanne, Dadonaite, Bernadeta, Petersen, Brenden, Engdahl, Taylor, Chen, Elaine, Handal, Laura, Hall, Lynn, Goforth, John, Vashchenko, Denis, Nguyen, Sam, Weilhammer, Dina, Lo, Jacky, Rubinfeld, Bonnee, Saada, Edwin, Weisenberger, Tracy, Lee, Tek-Hyung, Whitener, Bradley, Case, James, Ladd, Alexander, Silva, Mary, Haluska, Rebecca, Grzesiak, Emilia, Earnhart, Christopher, Hopkins, Svetlana, Bates, Thomas, Thackray, Larissa, Segelke, Brent, Lillo, Antonietta, Sundaram, Shivshankar, Bloom, Jesse, Diamond, Michael, Crowe, James, Carnahan, Robert, and Faissol, Daniel

Subjects
Animals, Female, Humans, Mice, Antibodies, Monoclonal, Antibodies, Neutralizing, Antibodies, Viral, COVID-19, Mutation, Neutralization Tests, SARS-CoV-2, Spike Glycoprotein, Coronavirus, DNA Mutational Analysis, Antigenic Drift and Shift, Drug Design, Computer Simulation
Abstract

The COVID-19 pandemic underscored the promise of monoclonal antibody-based prophylactic and therapeutic drugs1-3 and revealed how quickly viral escape can curtail effective options4,5. When the SARS-CoV-2 Omicron variant emerged in 2021, many antibody drug products lost potency, including Evusheld and its constituent, cilgavimab4-6. Cilgavimab, like its progenitor COV2-2130, is a class 3 antibody that is compatible with other antibodies in combination4 and is challenging to replace with existing approaches. Rapidly modifying such high-value antibodies to restore efficacy against emerging variants is a compelling mitigation strategy. We sought to redesign and renew the efficacy of COV2-2130 against Omicron BA.1 and BA.1.1 strains while maintaining efficacy against the dominant Delta variant. Here we show that our computationally redesigned antibody, 2130-1-0114-112, achieves this objective, simultaneously increases neutralization potency against Delta and subsequent variants of concern, and provides protection in vivo against the strains tested: WA1/2020, BA.1.1 and BA.5. Deep mutational scanning of tens of thousands of pseudovirus variants reveals that 2130-1-0114-112 improves broad potency without increasing escape liabilities. Our results suggest that computational approaches can optimize an antibody to target multiple escape variants, while simultaneously enriching potency. Our computational approach does not require experimental iterations or pre-existing binding data, thus enabling rapid response strategies to address escape variants or lessen escape vulnerabilities.

Published
2024

25. Physical science research needed to evaluate the viability and risks of marine cloud brightening.

Author

Feingold, Graham, Ghate, Virendra, Russell, Lynn, Blossey, Peter, Cantrell, Will, Christensen, Matthew, Diamond, Michael, Gettelman, Andrew, Glassmeier, Franziska, Gryspeerdt, Edward, Haywood, James, Hoffmann, Fabian, Kaul, Colleen, Lebsock, Matthew, McComiskey, Allison, McCoy, Daniel, Ming, Yi, Mülmenstädt, Johannes, Possner, Anna, Prabhakaran, Prasanth, Quinn, Patricia, Schmidt, K, Shaw, Raymond, Singer, Clare, Sorooshian, Armin, Toll, Velle, Wan, Jessica, Wood, Robert, Yang, Fan, Zhang, Jianhao, and Zheng, Xue

Abstract

Marine cloud brightening (MCB) is the deliberate injection of aerosol particles into shallow marine clouds to increase their reflection of solar radiation and reduce the amount of energy absorbed by the climate system. From the physical science perspective, the consensus of a broad international group of scientists is that the viability of MCB will ultimately depend on whether observations and models can robustly assess the scale-up of local-to-global brightening in todays climate and identify strategies that will ensure an equitable geographical distribution of the benefits and risks associated with projected regional changes in temperature and precipitation. To address the physical science knowledge gaps required to assess the societal implications of MCB, we propose a substantial and targeted program of research-field and laboratory experiments, monitoring, and numerical modeling across a range of scales.

Published
2024

27. Imprinting of serum neutralizing antibodies by Wuhan-1 mRNA vaccines

Author

Liang, Chieh-Yu, Raju, Saravanan, Liu, Zhuoming, Li, Yuhao, Asthagiri Arunkumar, Guha, Case, James Brett, Scheaffer, Suzanne M., Zost, Seth J., Acreman, Cory M., Gagne, Matthew, Andrew, Shayne F., Carvalho dos Anjos, Deborah Carolina, Foulds, Kathryn E., McLellan, Jason S., Crowe, Jr, James E., Douek, Daniel C., Whelan, Sean P. J., Elbashir, Sayda M., Edwards, Darin K., and Diamond, Michael S.

Published
2024
Full Text
View/download PDF

28. Computationally restoring the potency of a clinical antibody against Omicron

Author

Desautels, Thomas A., Arrildt, Kathryn T., Zemla, Adam T., Lau, Edmond Y., Zhu, Fangqiang, Ricci, Dante, Cronin, Stephanie, Zost, Seth J., Binshtein, Elad, Scheaffer, Suzanne M., Dadonaite, Bernadeta, Petersen, Brenden K., Engdahl, Taylor B., Chen, Elaine, Handal, Laura S., Hall, Lynn, Goforth, John W., Vashchenko, Denis, Nguyen, Sam, Weilhammer, Dina R., Lo, Jacky Kai-Yin, Rubinfeld, Bonnee, Saada, Edwin A., Weisenberger, Tracy, Lee, Tek-Hyung, Whitener, Bradley, Case, James B., Ladd, Alexander, Silva, Mary S., Haluska, Rebecca M., Grzesiak, Emilia A., Earnhart, Christopher G., Hopkins, Svetlana, Bates, Thomas W., Thackray, Larissa B., Segelke, Brent W., Lillo, Antonietta Maria, Sundaram, Shivshankar, Bloom, Jesse D., Diamond, Michael S., Crowe, Jr, James E., Carnahan, Robert H., and Faissol, Daniel M.

Published
2024
Full Text
View/download PDF

30. A Comparative Analysis of Health-Related Quality of Life 1 Year Following Myomectomy or Uterine Artery Embolization: Findings from the COMPARE-UF Registry.

Author

Anchan, Raymond, Wojdyla, Daniel, Bortoletto, Pietro, Terry, Kathryn, Disler, Emily, Milne, Ankrish, Gargiulo, Antonio, Petrozza, John, Brook, Olga, Srouji, Serene, Morton, Cynthia, Greenberg, James, Wegienka, Ganesa, Stewart, Elizabeth, Nicholson, Wanda, Thomas, Laine, Venable, Sateria, Laughlin-Tommaso, Shannon, Diamond, Michael, Maxwell, G, Marsh, Erica, Myers, Evan, Vines, Anissa, Wise, Lauren, Wallace, Kedra, Spies, James, and Jacoby, Vanessa

Subjects
hysterectomy, myomectomy, quality of life, uterine artery embolization, uterine fibroids, Female, Humans, Hysterectomy, Leiomyoma, Prospective Studies, Quality of Life, Registries, Treatment Outcome, Uterine Artery Embolization, Uterine Myomectomy, Uterine Neoplasms
Abstract

Objective: To compare 12-month post-treatment health-related quality of life (HR-QoL) and symptom severity (SS) changes among patients with symptomatic uterine fibroids (SUF) not seeking fertility and undergo a hysterectomy, abdominal myomectomy (AM), or uterine artery embolization (UAE). Materials and Methods: The Comparing Options for Management: Patient-Centered Results for Uterine Fibroids (COMPARE-UF) Registry is a multi-institutional prospective observational cohort study of patients treated for SUF. A subset of 1465 women 31-45 years of age, who underwent either hysterectomy (n = 741), AM (n = 446), or UAE (n = 155) were included in this analysis. Demographics, fibroid history, and symptoms were obtained by baseline questionnaires and at 1 year post-treatment. Results were stratified by all treatments and propensity score weighting to adjust for differences in baseline characteristics. Results: Women undergoing UAE reported the lowest baseline HR-QoL and highest SS scores (mean = 40.6 [standard deviation (SD) = 23.8]; 62.3 [SD = 24.2]) followed by hysterectomy (44.3 [24.3]; 59.8 [SD = 24.1]). At 12 months, women who underwent a hysterectomy experienced the largest change in both HR-QoL (48.7 [26.2]) and SS (51.9 [25.6]) followed by other uterine-sparing treatments. Propensity score weighting revealed all treatments produced substantial improvement, with hysterectomy patients reporting the highest HR-QoL score (92.0 [17.8]) compared with myomectomy (86.7 [17.2]) and UAE (82.6 [21.5]) (p

Published
2023

31. Thermodynamically coupled biosensors for detecting neutralizing antibodies against SARS-CoV-2 variants

Author

Zhang, Jason Z, Yeh, Hsien-Wei, Walls, Alexandra C, Wicky, Basile IM, Sprouse, Kaitlin R, VanBlargan, Laura A, Treger, Rebecca, Quijano-Rubio, Alfredo, Pham, Minh N, Kraft, John C, Haydon, Ian C, Yang, Wei, DeWitt, Michelle, Bowen, John E, Chow, Cameron M, Carter, Lauren, Ravichandran, Rashmi, Wener, Mark H, Stewart, Lance, Veesler, David, Diamond, Michael S, Greninger, Alexander L, Koelle, David M, and Baker, David

Subjects
Analytical Chemistry, Information and Computing Sciences, Chemical Sciences, Bioengineering, Pneumonia, Lung, Emerging Infectious Diseases, Vaccine Related, Infectious Diseases, Prevention, Pneumonia & Influenza, Biodefense, Biotechnology, Good Health and Well Being, Antibodies, Neutralizing, Antibodies, Viral, Biosensing Techniques, COVID-19, Humans, Neutralization Tests, SARS-CoV-2, Spike Glycoprotein, Coronavirus, Callicarpa nudiflora Hook, luteolin 3 '-O-beta-D-6 ''-acetyl glucopyranoside, pachypodol, hepatocellular carcinoma, cytotoxicity
Abstract

We designed a protein biosensor that uses thermodynamic coupling for sensitive and rapid detection of neutralizing antibodies against severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) variants in serum. The biosensor is a switchable, caged luciferase-receptor-binding domain (RBD) construct that detects serum-antibody interference with the binding of virus RBD to angiotensin-converting enzyme 2 (ACE-2) as a proxy for neutralization. Our coupling approach does not require target modification and can better distinguish sample-to-sample differences in analyte binding affinity and abundance than traditional competition-based assays.

Published
2022

33. Neutralization, effector function and immune imprinting of Omicron variants

Author

Addetia, Amin, Piccoli, Luca, Case, James Brett, Park, Young-Jun, Beltramello, Martina, Guarino, Barbara, Dang, Ha, de Melo, Guilherme Dias, Pinto, Dora, Sprouse, Kaitlin, Scheaffer, Suzanne M., Bassi, Jessica, Silacci-Fregni, Chiara, Muoio, Francesco, Dini, Marco, Vincenzetti, Lucia, Acosta, Rima, Johnson, Daisy, Subramanian, Sambhavi, Saliba, Christian, Giurdanella, Martina, Lombardo, Gloria, Leoni, Giada, Culap, Katja, McAlister, Carley, Rajesh, Anushka, Dellota, Jr, Exequiel, Zhou, Jiayi, Farhat, Nisar, Bohan, Dana, Noack, Julia, Chen, Alex, Lempp, Florian A., Quispe, Joel, Kergoat, Lauriane, Larrous, Florence, Cameroni, Elisabetta, Whitener, Bradley, Giannini, Olivier, Cippà, Pietro, Ceschi, Alessandro, Ferrari, Paolo, Franzetti-Pellanda, Alessandra, Biggiogero, Maira, Garzoni, Christian, Zappi, Stephanie, Bernasconi, Luca, Kim, Min Jeong, Rosen, Laura E., Schnell, Gretja, Czudnochowski, Nadine, Benigni, Fabio, Franko, Nicholas, Logue, Jennifer K., Yoshiyama, Courtney, Stewart, Cameron, Chu, Helen, Bourhy, Hervé, Schmid, Michael A., Purcell, Lisa A., Snell, Gyorgy, Lanzavecchia, Antonio, Diamond, Michael S., Corti, Davide, and Veesler, David

Published
2023
Full Text
View/download PDF

35. Poor reproducibility of percentage of normally shaped sperm using the World Health Organization Fifth Edition strict grading criteria

Author

Baker, Karen C, Steiner, Anne Z, Hansen, Karl R, Barnhart, Kurt T, Cedars, Marcelle I, Legro, Richard S, Diamond, Michael P, Krawetz, Stephen A, Usadi, Rebecca, Baker, Valerie L, Coward, R Matthew, Sun, Fangbai, Wild, Robert, Masson, Puneet, Smith, James F, Santoro, Nanette, Zhang, Heping, and Network, Reproductive Medicine

Subjects
Human Society, Reproductive Medicine, Biomedical and Clinical Sciences, Demography, Clinical Trials and Supportive Activities, Clinical Research, Contraception/Reproduction, Semen analysis, male factor infertility, quality control, spermatozoa, teratozoospermia
Abstract

ObjectiveTo determine the reproducibility of the World Health Organization Fifth Edition (WHO5) strict grading methodology by comparing the percentage of morphologically normal sperm (PNS) recorded by the core laboratory with results obtained at the fertility centers participating in a multisite clinical trial.DesignSecondary cohort analysis of data from the Males, Antioxidants, and Infertility trial.SettingFertility centers.PatientsSemen values of 171 men participating in a multicenter, double-blind, randomized, placebo-controlled trial evaluating the effect of antioxidants on male fertility.InterventionsNot applicable.Main outcome measuresStrict morphology expressed as PNS as determined at each fertility center and the core central laboratory for the same semen sample.ResultsNo correlation was found in the PNS values for the same semen sample between the core laboratory and fertility center laboratories either as a group or by individual site. Interobserver agreement was similarly low (κ = 0.05 and 0.15) between the core and fertility laboratories as a group for strict morphology, categorized by the WHO5 lower reference limits of 4% and 0, respectively. Moderate agreement was found between the core and 2 individual fertility laboratories for the cutoff value of 0 (κ = 0.42 and 0.57). The remainder of the comparisons demonstrated poor to fair agreement.ConclusionsStrict morphology grading using the WHO5 methodology demonstrated overall poor reproducibility among a cohort of experienced fertility laboratories. This lack of correlation and agreement in the PNS values calls into question the reproducibility, and thereby the potential applicability, of sperm strict morphology testing.

Published
2022

37. A pre-vaccination immune metabolic interplay determines the protective antibody response to a dengue virus vaccine

Author

Pelletier, Adam-Nicolas, Sanchez, Gabriela Pacheco, Izmirly, Abdullah, Watson, Mark, Di Pucchio, Tiziana, Carvalho, Karina Inacio, Filali-Mouhim, Abdelali, Paramithiotis, Eustache, Timenetsky, Maria do Carmo S.T., Precioso, Alexander Roberto, Kalil, Jorge, Diamond, Michael S., Haddad, Elias K., Kallas, Esper G., and Sekaly, Rafick Pierre

Published
2024
Full Text
View/download PDF

38. Genetic loci regulate Sarbecovirus pathogenesis: A comparison across mice and humans

Author

Schäfer, Alexandra, Gralinski, Lisa E., Leist, Sarah R., Hampton, Brea K., Mooney, Michael A., Jensen, Kara L., Graham, Rachel L., Agnihothram, Sudhakar, Jeng, Sophia, Chamberlin, Steven, Bell, Timothy A., Scobey, D. Trevor, Linnertz, Colton L., VanBlargan, Laura A., Thackray, Larissa B., Hock, Pablo, Miller, Darla R., Shaw, Ginger D., Diamond, Michael S., de Villena, Fernando Pardo Manuel, McWeeney, Shannon K., Heise, Mark T., Menachery, Vineet D., Ferris, Martin T., and Baric, Ralph S.

Published
2024
Full Text
View/download PDF

39. Phase III Pivotal comparative clinical trial of intranasal (iNCOVACC) and intramuscular COVID 19 vaccine (Covaxin®)

Author

Singh, Chandramani, Verma, Savita, Reddy, Prabhakar, Diamond, Michael S., Curiel, David T., Patel, Chintan, Jain, Manish Kumar, Redkar, Sagar Vivek, Bhate, Amit Suresh, Gundappa, Vivek, Konatham, Rambabu, Toppo, Leelabati, Joshi, Aniket Chandrakant, Kushwaha, Jitendra Singh, Singh, Ajit Pratap, Bawankule, Shilpa, Ella, Raches, Prasad, Sai, Ganneru, Brunda, Chiteti, Siddharth Reddy, Kataram, Sreenivas, and Vadrevu, Krishna Mohan

Published
2023
Full Text
View/download PDF

43. Multivalent designed proteins neutralize SARS-CoV-2 variants of concern and confer protection against infection in mice

Author

Hunt, Andrew C, Case, James Brett, Park, Young-Jun, Cao, Longxing, Wu, Kejia, Walls, Alexandra C, Liu, Zhuoming, Bowen, John E, Yeh, Hsien-Wei, Saini, Shally, Helms, Louisa, Zhao, Yan Ting, Hsiang, Tien-Ying, Starr, Tyler N, Goreshnik, Inna, Kozodoy, Lisa, Carter, Lauren, Ravichandran, Rashmi, Green, Lydia B, Matochko, Wadim L, Thomson, Christy A, Vögeli, Bastian, Krüger, Antje, VanBlargan, Laura A, Chen, Rita E, Ying, Baoling, Bailey, Adam L, Kafai, Natasha M, Boyken, Scott E, Ljubetič, Ajasja, Edman, Natasha, Ueda, George, Chow, Cameron M, Johnson, Max, Addetia, Amin, Navarro, Mary Jane, Panpradist, Nuttada, Gale, Michael, Freedman, Benjamin S, Bloom, Jesse D, Ruohola-Baker, Hannele, Whelan, Sean PJ, Stewart, Lance, Diamond, Michael S, Veesler, David, Jewett, Michael C, and Baker, David

Subjects
Medical Biotechnology, Engineering, Biomedical and Clinical Sciences, Biomedical Engineering, Vaccine Related, Lung, Emerging Infectious Diseases, Biodefense, Pneumonia, Autoimmune Disease, Infectious Diseases, Pneumonia & Influenza, Prevention, Biotechnology, Development of treatments and therapeutic interventions, 5.1 Pharmaceuticals, Good Health and Well Being, Animals, Antibodies, Neutralizing, Antibodies, Viral, COVID-19, Cryoelectron Microscopy, Humans, Mice, SARS-CoV-2, Spike Glycoprotein, Coronavirus, Biological Sciences, Medical and Health Sciences, Medical biotechnology, Biomedical engineering
Abstract

New variants of severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) continue to arise and prolong the coronavirus disease 2019 (COVID-19) pandemic. Here, we used a cell-free expression workflow to rapidly screen and optimize constructs containing multiple computationally designed miniprotein inhibitors of SARS-CoV-2. We found the broadest efficacy was achieved with a homotrimeric version of the 75-residue angiotensin-converting enzyme 2 (ACE2) mimic AHB2 (TRI2-2) designed to geometrically match the trimeric spike architecture. Consistent with the design model, in the cryo-electron microscopy structure TRI2-2 forms a tripod at the apex of the spike protein that engaged all three receptor binding domains simultaneously. TRI2-2 neutralized Omicron (B.1.1.529), Delta (B.1.617.2), and all other variants tested with greater potency than the monoclonal antibodies used clinically for the treatment of COVID-19. TRI2-2 also conferred prophylactic and therapeutic protection against SARS-CoV-2 challenge when administered intranasally in mice. Designed miniprotein receptor mimics geometrically arrayed to match pathogen receptor binding sites could be a widely applicable antiviral therapeutic strategy with advantages over antibodies in greater resistance to viral escape and antigenic drift, and advantages over native receptor traps in lower chances of autoimmune responses.

Published
2022

46. Pathophysiology of chikungunya virus infection associated with fatal outcomes

Author

de Souza, William M., Fumagalli, Marcilio J., de Lima, Shirlene T.S., Parise, Pierina L., Carvalho, Deyse C.M., Hernandez, Cristian, de Jesus, Ronaldo, Delafiori, Jeany, Candido, Darlan S., Carregari, Victor C., Muraro, Stefanie P., Souza, Gabriela F., Simões Mello, Leda M., Claro, Ingra M., Díaz, Yamilka, Kato, Rodrigo B., Trentin, Lucas N., Costa, Clauber H.S., Maximo, Ana Carolina B.M., Cavalcante, Karene F., Fiuza, Tayna S., Viana, Vânia A.F., Melo, Maria Elisabeth L., Ferraz, Clarissa P.M., Silva, Débora B., Duarte, Larissa M.F., Barbosa, Priscilla P., Amorim, Mariene R., Judice, Carla C., Toledo-Teixeira, Daniel A., Ramundo, Mariana S., Aguilar, Patricia V., Araújo, Emerson L.L., Costa, Fabio T.M., Cerqueira-Silva, Thiago, Khouri, Ricardo, Boaventura, Viviane S., Figueiredo, Luiz Tadeu M., Fang, Rong, Moreno, Brechla, López-Vergès, Sandra, Mello, Liana Perdigão, Skaf, Munir S., Catharino, Rodrigo R., Granja, Fabiana, Martins-de-Souza, Daniel, Plante, Jessica A., Plante, Kenneth S., Sabino, Ester C., Diamond, Michael S., Eugenin, Eliseo, Proença-Módena, José Luiz, Faria, Nuno R., and Weaver, Scott C.

Published
2024
Full Text
View/download PDF

50. Measuring Quality of General Reasoning

Author

Marcoci, Alexandru, van Gelder, Tim, Dennis, Simon, Webb, Margaret E, Rowe, Luke I, Barnett, Ashley, Primoratz, Tamar, Kruger, Ariel, Stone, Benjamin, Diamond, Michael L, and Saletta, Morgan

Subjects
Artificial Intelligence, Linguistics, Psychology, Reasoning, Logic
Abstract

Machine learning models that automatically assess reasoning quality are trained on human-annotated written products. These “gold-standard” corpora are typically created by prompting annotators to choose, using a forced choice design, which of two products presented side by side is the most convincing, contains the strongest evidence or would be adopted by more people. Despite the increase in popularity of using a forced choice design for assessing quality of reasoning (QoR), no study to date has established the validity and reliability of such a method. In two studies, we simultaneously presented two products of reasoning to participants and asked them to identify which product was ‘better justified’ through a forced choice design. We investigated the criterion validity and inter-rater reliability of the forced choice protocol by assessing the relationship between QoR, measured using the forced choice protocol, and accuracy in objectively answerable problems using naive raters sampled from MTurk (Study 1) and experts (Study 2), respectively. In both studies products that were closer to the correct answer and products generated by larger teams were consistently preferred. Experts were substantially better at picking the reasoning products that corresponded to accurate answers. Perhaps the most surprising finding was just how rapidly raters made judgements regarding reasoning: On average, both novices and experts made reliable decisions in under 15 seconds. We conclude that forced choice is a valid and reliable method of assessing QoR.

Published
2022

Catalog

Books, media, physical & digital resources
See catalog results