Evaluating Organizational Success of an AI-Based Recommender System at a Two-Year Higher Education Institution

This study will evaluate the organizational effectiveness of an artificial intelligence (AI)/machine learning (ML) recommender system at a higher education institution. It will determine the positive or negative net benefits (i.e., organizational effectiveness) of utilizing the D&M ISSM. Identifying the value and efficacy of information systems (IS) management actions and investments requires evaluating their success or effectiveness. A system's effectiveness is evaluated from the organizational perspective based on the degree to which it meets the goals of the organization. Although the pandemic has negatively impacted numerous lives and business activities, more leaders considered it an opportunity because it offered new opportunities for business innovation and entrepreneurship, despite it being viewed as the most significant crisis in the modern world. Considering the significant changes caused by the COVID-19 pandemic and the response to it, it is no longer simply considered an option to adopt and use AI/ML systems, but rather an obligation. There is a lack of understanding of the factors contributing to the success of recommendation systems, therefore, these systems are rarely used to their full potential. An analysis of the relationship between information quality, system quality, use/intention to use, and user satisfaction in recommender systems was conducted using a mixed-method approach based on the DeLone and McLean IS success model. Students enrolled in a two-year college who used a portal as part of their academic journey were the target population of this study, and a total of 8,559 participants were contacted to participate, and 305 of them completed the survey. The results of this study indicate that quality factors relate closely to the success of the recommender system as measured by organizational effectiveness. The results indicate that there are statistically significant relationships between the independent variables, Information Quality, System Quality, and User Satisfaction, and the dependent variable, Organizational Effectiveness. However, inconclusive results were found in use and intention to use. The study demonstrated that the IS success model developed by Delone and McLean provides an effective foundation for understanding recommender system success. A strong relationship exists between organizational effectiveness, measured by information quality, system quality, and user satisfaction, as demonstrated by this study.