Development and Validation of an Extended UTAUT-Based Instrument for Measuring Lecturers' Acceptance and Behavioral Intention Toward AI-Based Educational Assessment in Saudi Higher Education

The increasing integration of artificial intelligence (AI) into educational assessment has created a need for valid, context-sensitive instruments to measure lecturers' acceptance of and behavioural intention toward AI-based assessment practices. Although existing technology acceptance models provide useful theoretical foundations, many available instruments were developed for general educational technologies and may not adequately capture the professional, ethical, and psychological concerns associated with AI-based assessment. This study aimed to develop and validate an extended Unified Theory of Acceptance and Use of Technology (UTAUT)-based instrument for measuring lecturers' acceptance and behavioural intention toward AI-based educational assessment in Saudi higher education. The instrument was developed based on UTAUT and extended by incorporating three AI-related constructs: Perceived Trust in AI, Perceived Threat to Professional Autonomy, and Job Security Concern. A quantitative instrument-development design was employed. Items were generated through literature review, construct mapping, and expert validation. The instrument comprised sections on demographic profile, technology acceptance, extended AI-related constructs, and behavioural intention. Rasch measurement analysis was used to examine item fit, person and item reliability, separation indices, dimensionality, rating scale functioning, and item hierarchy. The preliminary findings indicated that the instrument demonstrated acceptable psychometric properties, with most items fitting the Rasch model and representing the intended constructs. The validated instrument is expected to provide researchers, university leaders, and policymakers with a diagnostic tool for assessing lecturers' readiness to adopt AI-based educational assessment. This study contributes to educational measurement by providing a theoretically grounded, contextually relevant instrument for future research on AI adoption in higher-education assessment.

AI-based educational assessment; behavioural intention; instrument validation; Rasch measurement model; UTAUT

1. Acosta-Enriquez, B. G., Ramos Farroñan, E. V., Villena Zapata, L. I., Mogollon Garcia, F. S., Rabanal-León, H. C., Morales Angaspilco, J. E., & Saldaña Bocanegra, J. C. (2024). Acceptance of artificial intelligence in university contexts: A conceptual analysis based on UTAUT2 theory. Heliyon, 10, Article e38315. https://doi.org/10.1016/j.heliyon.2024.e38315 [Google Scholar] [Crossref]

2. Ajzen, I. (1991). The theory of planned behaviour. Organizational Behaviour and Human Decision Processes, 50(2), 179-211. https://doi.org/10.1016/0749-5978(91)90020-T [Google Scholar] [Crossref]

3. Boateng, G. O., Neilands, T. B., Frongillo, E. A., Melgar-Quinonez, H. R., & Young, S. L. (2018). Best practices for developing and validating scales for health, social, and behavioural research: A primer. Frontiers in Public Health, 6, Article 149. https://doi.org/10.3389/fpubh.2018.00149 [Google Scholar] [Crossref]

4. Bond, T. G., Yan, Z., & Heene, M. (2020). Applying the Rasch model: Fundamental measurement in the human sciences (4th ed.). Routledge. https://doi.org/10.4324/9780429030499 [Google Scholar] [Crossref]

5. Boone, W. J., Staver, J. R., & Yale, M. S. (2014). Rasch analysis in the human sciences. Springer. https://doi.org/10.1007/978-94-007-6857-4 [Google Scholar] [Crossref]

6. Chen, L., Chen, P., & Lin, Z. (2020). Artificial intelligence in education: A review. IEEE Access, 8, 75264-75278. https://doi.org/10.1109/ACCESS.2020.2988510 [Google Scholar] [Crossref]

7. Crompton, H., & Burke, D. (2023). Artificial intelligence in higher education: The state of the field. International Journal of Educational Technology in Higher Education, 20, Article 22. https://doi.org/10.1186/s41239-023-00392-8 [Google Scholar] [Crossref]

8. Davis, F. D. (1989). Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Quarterly, 13(3), 319-340. https://doi.org/10.2307/249008 [Google Scholar] [Crossref]

9. DeVellis, R. F., & Thorpe, C. T. (2021). Scale development: Theory and applications (5th ed.). SAGE Publications. [Google Scholar] [Crossref]

10. Dwivedi, Y. K., Hughes, L., Ismagilova, E., Aarts, G., Coombs, C., Crick, T., Duan, Y., Dwivedi, R., Edwards, J., Eirug, A., Galanos, V., Ilavarasan, P. V., Janssen, M., Jones, P., Kar, A. K., Kizgin, H., Kronemann, B., Lal, B., Lucini, B., ... Williams, M. D. (2019). Artificial intelligence (AI): Multidisciplinary perspectives on emerging challenges, opportunities, and agenda for research, practice and policy. International Journal of Information Management, 57, Article 101994. https://doi.org/10.1016/j.ijinfomgt.2019.08.002 [Google Scholar] [Crossref]

11. European Commission, Directorate-General for Education, Youth, Sport and Culture. (2022). Ethical guidelines on the use of artificial intelligence (AI) and data in teaching and learning for educators. Publications Office of the European Union. https://doi.org/10.2766/153756 [Google Scholar] [Crossref]

12. Holmes, W., Bialik, M., & Fadel, C. (2019). Artificial intelligence in education: Promises and implications for teaching and learning. Center for Curriculum Redesign. [Google Scholar] [Crossref]

13. Huggins-Manley, A. C., Booth, B. M., & D’Mello, S. K. (2022). Toward argument-based fairness with an application to AI-enhanced educational assessments. Journal of Educational Measurement, 59(3), 362–388. https://doi.org/10.1111/jedm.12334 [Google Scholar] [Crossref]

14. Kasneci, E., Sessler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., Krusche, S., Kutyniok, G., Michaeli, T., Nerdel, C., Pfeffer, J., Poquet, O., Sailer, M., Schmidt, A., Seidel, T., … Kasneci, G. (2023). ChatGPT for good? On opportunities and challenges of large language models for education. Learning and Individual Differences, 103, Article 102274. https://doi.org/10.1016/j.lindif.2023.102274 [Google Scholar] [Crossref]

15. Kizilcec, R. F., & Lee, H. (2022). Algorithmic fairness in education. In W. Holmes & K. Porayska-Pomsta (Eds.), The ethics of artificial intelligence in education: Practices, challenges, and debates (pp. 174-202). Routledge. https://doi.org/10.4324/9780429329067-10 [Google Scholar] [Crossref]

16. Miao, F., & Holmes, W. (2023). Guidance for generative AI in education and research. UNESCO [Google Scholar] [Crossref]

17. Miao, F., Holmes, W., Huang, R., & Zhang, H. (2021). AI and education: Guidance for policy-makers. UNESCO Publishing. [Google Scholar] [Crossref]

18. Nazaretsky, T., Ariely, M., Cukurova, M., & Alexandron, G. (2022). Teachers' trust in AI-powered educational technology and a professional development program to improve it. British Journal of Educational Technology, 53(4), 914-931. https://doi.org/10.1111/bjet.13232 [Google Scholar] [Crossref]

19. OECD. (2023). OECD digital education outlook 2023: Towards an effective digital education ecosystem. OECD Publishing. https://doi.org/10.1787/c74f03de-en [Google Scholar] [Crossref]

20. Shin, D. (2021). The effects of explainability and causability on perception, trust, and acceptance: Implications for explainable AI. International Journal of Human-Computer Studies, 146, Article 102551. https://doi.org/10.1016/j.ijhcs.2020.102551 [Google Scholar] [Crossref]

21. Teo, T. (2011). Factors influencing teachers' intention to use technology: Model development and test. Computers & Education, 57(4), 2432-2440. https://doi.org/10.1016/j.compedu.2011.06.008 [Google Scholar] [Crossref]

22. Venkatesh, V., Morris, M. G., Davis, G. B., & Davis, F. D. (2003). User acceptance of information technology: Toward a unified view. MIS Quarterly, 27(3), 425-478. https://doi.org/10.2307/30036540 [Google Scholar] [Crossref]

23. Venkatesh, V., Thong, J. Y. L., & Xu, X. (2016). Unified theory of acceptance and use of technology: A synthesis and the road ahead. Journal of the Association for Information Systems, 17(5), 328-376. https://doi.org/10.17705/1jais.00428 [Google Scholar] [Crossref]

24. Wang, S., Wang, F., Zhu, Z., Wang, J., Tran, T., & Du, Z. (2024). Artificial intelligence in education: A systematic literature review. Expert Systems with Applications, 252, Article 124167. https://doi.org/10.1016/j.eswa.2024.124167 [Google Scholar] [Crossref]

25. Zawacki-Richter, O., Marin, V. I., Bond, M., & Gouverneur, F. (2019). Systematic review of research on artificial intelligence applications in higher education: Where are the educators? International Journal of Educational Technology in Higher Education, 16, Article 39. https://doi.org/10.1186/s41239-019-0171-0 [Google Scholar] [Crossref]

• International Journal of Research and Innovation in Social Science (IJRISS) - Manuscript Draft Relationship of Technology Acceptance and Behavioural Intention Toward AI-Based Educational Assessment among Lecturers in Saudi Arabia