What should we evaluate when we use technology in education?

Background: There is a lack of critical or empirical work interrogating the nature and purpose of evaluating technology use in education. Objectives: In this study, we examine the values underpinning the evaluation of technology use in education through field specialist perceptions. The study also poses critical reflections about the rigour of evaluation instruments development in the educational technology field. Method: A total of 48 domain specialists were surveyed to investigate the face and content validity of 39 items under eight constructs, with both qualitative and quantitative data from the survey analysed. Results: There was an alignment between the constructs that the specialists felt were important and the constructs that have typically been the focus of empirical studies, with field specialists indicating high relevance scores for technology (M = 3.24/4), learning outcomes (M = 3.20/4), affective elements (M = 3.19/4), behaviour (M = 3.15/4), presence/community (M = 3.07/4), teaching/pedagogy (M = 3.01/4), design (M = 2.96/4), and institutional environment (M = 2.86/4). Only a minority of other studies were found to perform face and content validity checks and even then only with small samples of respondents (usually n ≤ 5). Implications: Specialists in educational technology research confirm that all eight dimensions are important to consider when evaluating the use of technology in education. Thorough face and content validity processes should be adopted when developing educational technology evaluation instruments. Further work has validated an eight factor 28 item instrument for evaluating the use of technology in education using a large sample of students from a global open learning online course. Lay Description: What is already known about this topic: Comprehensive evaluation of technology use is essential to make decisions on learning technology strategies.There is a lack of critical work interrogating how technology use is evaluated in education.There is a need for developing evaluation instruments in the educational technology field. What this paper adds: An examination of 48 field specialist perceptions of important constructs when evaluating technology use in education.Field specialists agreed that eight constructs identified in empirical studies were relevant to educational technology evaluation.Evidence of limited face and content validation processes based on previous empirical work.A comprehensive 39‐item instrument for evaluating technology use in education. Implications of study findings for practitioners: Adopt a broad range of constructs when evaluating the use of technology in education.Apply face and content validation processes involving a large sample of experts.Consider using the 39‐item instrument to evaluate educational technology use. [ABSTRACT FROM AUTHOR]