Distributed Item Review: Administrator User Guide. Technical Report #1603

The Distributed Item Review (DIR) is a secure and flexible, web-based system designed to present test items to expert reviewers across a broad geographic area for evaluation of important dimensions of quality (e.g., alignment with standards, bias, sensitivity, and student accessibility). The DIR is comprised of essential features that allow system administrators to design and conduct customized review studies, in which test items are bulk uploaded, paired with target academic content/performance standards and images, and then organized and assigned for review by selected experts. Administrators can choose from predefined item review questions, or design new review questions pertinent to their study. A variety of administrator-designed support resources can be uploaded and associated with item review studies. This document provides detailed information about the DIR system, framed by a step-by-step administrator's guide to successfully creating and conducting test item reviews using the DIR. The steps in this guide are presented in an order that comprehensively and parsimoniously discusses the principle steps; however, a DIR administrator may alter the sequence of the steps outlined, and possibly omit or alter a given step. Prior to reading this user guide and attempting to create a DIR item review, it is recommended that prospective administrators view the DIR System Introduction--downloadable as a Portable Document Format (.pdf) at: http://www.brtprojects.org/publications/presentations. The slides give prospective DIR administrators an overview of the system, offering a jumpstart to proceeding through this guide.