Contextualising Information Systems Evaluation Research: Towards a Classification of Approaches.

Over several decades, Information Systems (IS) researchers have grappled with the problem of how best to evaluate the success or effectiveness of computing systems. Consequently, researchers working within the IS Management realm have contributed to an ever growing body of knowledge to the field of IS Evaluation. Much of the debate has focused on the difficulty in being able to measure IS Success and it has been suggested that the search for appropriate metrics has resembled the proverbial search for the Holy Grail. Consequently the field of IS evaluation research is fragmented, and various streams of research has tended to continue on independent trajectories. As a result, arguments in terms of the value thereof to IS Management occurs from a siloed perspective. It has therefore become paramount to develop a means for classifying IS evaluation approaches. This will enable improved coherence in the field, and assist researchers at the problem inception stage to locate their work within the broader field. Given the foregoing, this paper provides a review of literature to contextualise and frame the field of IS Evaluation research. The paper thereafter presents an analysis of 110 articles, which were published over a period of five years. Each paper was analysed to assess if the focus on IS evaluation was from either the IS Product, Process or Service perspective. The papers were also scored on the extent to which it focused on clearly delineated evaluation questions and the extent to which the research question was aligned to IS evaluation in the internetworked era. The findings indicate the research community has placed emphasis on IS evaluation from a service perspective as compared to evaluating product and process perspectives. This indicates an increase in the development of proxy indicators of IS success rather than evaluations which focus on the product. The paper concludes with recommendations on how future research in this field may be improved. It is proposed that the clarity of the units of analysis of IS Evaluation studies may be obtained by positioning or classifying the evaluation focus within the systems development life-cycle framework. A case is further made for a stronger alignment of research questions which advance questions related to systems issues in the inter-networked era, as well as advancing knowledge related to the Information Society. [ABSTRACT FROM AUTHOR]