1. The revised Approved Instructional Resources score: An improved quality evaluation tool for online educational resources
- Author
-
Nadim Lalani, Shelaina Anderson, Jonas Wilmer, Marina Roure, Kristen Weersink, Katherine A. Stuart, Brent Benavides, Wisarut Bunchit, Colleen Sweeney, Robert R. Ehrman, Richard Tang, Kelvin Tran, Gregory Wanner, Drew Kalnow, Christine Roh, Kinjal Patel, Bill Fraser, Therese Mead, Stephen Carroll, Paul Schofield, Caley Flynn, Lindy Buzikievich, Lucy Chen, Tara Stratton, Jan Hansel, Hector C. Singson, Tanner Gronowski, Ali Jamal, Adeeb Saleh, Todd Taylor, Rayan Delbani, Phil Griffith, Michael J. Ward, Miranda Wan, Ashley Kilp, Anna Whalen-Browne, Logan Mills, Ali S. Raja, Perry Menzies, Christine Patterson, Sandra Viggers, Brendan Devine, Vanessa Rogers, Braeden Beaumont, Jennifer Baird, Paula Sneath, Natasha Chatham-Zvelebil, Brandon Herb, Harry Liu, Marie Decock, Sarah Mott, Elise Lovell, Mohammad Ali Jamil, Ken Edwards, Victor Jansen, Maia Dorsett, Jaasmit Khurana, Salim R. Rezaie, Alexander Hart, Fareen Zaver, Manpreet Singh, Ching-Hsing Lee, Suzanne Rannazzisi, Mike McDonnell, Loice Swisher, Rob Carey, Joe Walter, Andrew D. D’Alessandro, Bob Stuntz, James Stempien, Preston Fedor, Kelly Lien, Parisa Shahrabadi, Shauna Regan, Alan Taylor, Nilantha Lenora, Scott Anderson, Calvin H. Yeh, Jason Trickovic, David Calcara, Werner Oberholzer, Catherine Patocka, James Fukakusa, T. Oyedokun, Ivy Liu, Regina Hammock, Steve Liu, Kevin Cullison, Chris Belcher, Teresa Dunphy, Alexis Pelletier-Bui, Zander Laurie, Ashley Lubberdink, James Pearlman, James Huffman, Nikhil Tambe, Carolyn McQuarrie, Gerhard Tiwald, Gregor Prosen, David Lowe, Henry Swoboda, Jennifer Weekes, Kimberly Connors, Aaron Tyagi, Anali Maneshi, Patrick M. Lank, Emina Hajdinjak, Levi Johnston, Eric Chen, Abdulaziz S. Almehlisi, Zach Jarou, Noorin Walji, Alvin Chin, Tanis Quaife, Nikytha Antony, Lawrence Yau, Alexander Zozula, Gregory Costello, Louise Rang, John Mayo, Evan S. Schwarz, Victoria Brazil, P. Mukherj, Taylor Duda, Jaime Jordan, Susan McLellan, Alim Pardhan, Jared Baylis, Allan Mix, Cathy Grossman, Sean Dyer, Emily House, Eric Shappell, Colin Andrews, Mark Woodcroft, William D.T. Kent, Anthony Bryson, Nelson Wong, Pawan Gupta, Diptesh Aryal, Owen Scheirer, Morgan Oakland, Patrick Vallance, Brendan Moore, Mary R C Haas, Kenn Ghaffarian, Steve Montag, Elyse Berger Pelletier, Julianna Deutscher, Nina House, Keith Rosenberg, Sushant Chhabra, Viktor Gawlik, Michael Benham, Andrew Baker, Brent Thoma, Ernest Leber, Larissa Hattin, Casey Lyons, Timothy Chaplin, Kamini Premkumar, Shahbaz Syed, Ivanna Kruhlak, Stephanie Louka, Haakon Lenes, Rene Verbeek, SueLin Hilbert, Joshua Rudner, Julia Nood, Kelly van Diepen, Brian Whiteside, Karthryn T. Eastley, Julia Sheffield, Damjan Gaco, Sam Smith, Quinten S. Paterson, Teresa M. Chan, Jeremy Christensen, Jocelyn Andruko, Youness Elkhalidy, Cory Meeuwisse, Sheena Nandalal, Cara Weessies, Scott Knapp, Sheng Hsiang Ma, Meagan Fu, Veronica Coppersmith, William Denq, Vivian Jia, Kristina Lea, Hugh MacLeod, Simon Huang, Yingchun Lin, Wyatt Warawa, Will Sanderson, Brian Ficiur, Jessica G.Y. Luc, Taylor Nikel, Jessica Yee, Tina Choudhri, Patricia Van den Berg, Andrew Grock, Samantha Lam, Andrew Guy, Keeth Krishnan, Taku Taira, Eric Funk, Rachel Taylor, Ali Mulla, Sebastian Kohler, Kyle Kelson, Nicholas Bouchard, Stanislaw Haciski, Jesse Leontowicz, Paul Trinquero, Charlie Inboriboon, Justin Dueweke, Julian Botta, Emily Brumfield, Kat Butler, Patrick Meloy, Laleh Gharahbaghian, Andrew K. Hall, Maria Rosa Carrillo, Aubrey Powell, Louise Cassidy, Jesse May, Isabelle N. Colmers-Gray, Evelyn Tran, Sarah Batty, Vishal Puri, Randi Ramunno, Luis Vargas, Stephen Miazga, Justin Morgenstern, Michelle Lin, Andrew Griffith, Michael Susalla, Charlotte Alexander, Alex Ireland, Kerstin de Wit, Marcia L. Edmonds, Robert Sobehart, Rob Woods, Kirsty Challen, Dave Slessor, Abby Cosgrove, Eric Chochi, Onyeka Otugo, Amy F. Ho, Alexandra Gustafson, Zlata Vlodaver, Kerry Spearing, Ryan Raffel, Milan L Ridderikhof, Barbra Backus, Saeed Alqahtani, Paul Schunk, Anne Messman, Seth Kelly, Puneet Kapur, Andrew Little, Kathryn Chan, Sean Nugent, Rishi Khakhkhar, Mohammed Alkhalifah, Rachel Wang, Jesse Hill, Marc Phan, Jaroslaw Gucwa, Nick Mancuso, Paxton Ting, Matthew Wagner, Zafrina Poonja, Elisha Targonsky, Britni Sternard, Katherine Yurkiw, Manrique Umana, Jeff Hill, Matthew Willis, and Sherri L. Rudinsky
- Subjects
Medical education ,education.field_of_study ,Intraclass correlation ,business.industry ,Best practice ,Population ,MEDLINE ,Context (language use) ,Usability ,Original Contribution ,Emergency Nursing ,Education ,Emergency Medicine ,Thematic analysis ,education ,business ,Psychology ,Quality assurance - Abstract
Background Free Open-Access Medical education (FOAM) use among residents continues to rise. However, it often lacks quality assurance processes and residents receive little guidance on quality assessment. The Academic Life in Emergency Medicine Approved Instructional Resources tool (AAT) was created for FOAM appraisal by and for expert educators and has demonstrated validity in this context. It has yet to be evaluated in other populations. Objectives We assessed the AAT's usability in a diverse population of practicing emergency medicine (EM) physicians, residents, and medical students; solicited feedback; and developed a revised tool. Methods As part of the Medical Education Translational Resources: Impact and Quality (METRIQ) study, we recruited medical students, EM residents, and EM attendings to evaluate five FOAM posts with the AAT and provide quantitative and qualitative feedback via an online survey. Two independent analysts performed a qualitative thematic analysis with discrepancies resolved through discussion and negotiated consensus. This analysis informed development of an initial revised AAT, which was then further refined after pilot testing among the author group. The final tool was reassessed for reliability. Results Of 330 recruited international participants, 309 completed all ratings. The Best Evidence in Emergency Medicine (BEEM) score was the component most frequently reported as difficult to use. Several themes emerged from the qualitative analysis: for ease of use-understandable, logically structured, concise, and aligned with educational value. Limitations include deviation from questionnaire best practices, validity concerns, and challenges assessing evidence-based medicine. Themes supporting its use include evaluative utility and usability. The author group pilot tested the initial revised AAT, revealing a total score average measure intraclass correlation coefficient (ICC) of moderate reliability (ICC = 0.68, 95% confidence interval [CI] = 0 to 0.962). The final AAT's average measure ICC was 0.88 (95% CI = 0.77 to 0.95). Conclusions We developed the final revised AAT from usability feedback. The new score has significantly increased usability, but will need to be reassessed for reliability in a broad population.
- Published
- 2021