Application of Fuzzy Delphi Method for Evaluating and Validating Gamification Learning Framework for Language Learning

This research aims to reach expert consensus and alignment regarding the validation of Gamification Learning Framework for Language Learning. This research employed two iterations of the Fuzzy Delphi Method (FDM) with a 7-point agreement scale to gather assessments from 20 experts across various fields, including technology, gamification, education, TESL, adult learning, mobile learning, communication, CALL, management, digital transformation, curriculum design, game creation, cybersecurity, and system architecture. The research instrument consists of 51 items, developed from two prior studies involving mixed methods and a systematic literature review. The data examined utilizing Triangular Fuzzy Number and the Defuzzification Process. The findings indicate that experts achieved consensus on 51 items in Round 1, and in Round 2, 50 of the 51 items received agreement from experts, and 44 out of 51 items received Threshold (d) values under 0.2. The research indicated that the consensus and agreement among experts exceeded 75% for majority of the elements, with a Threshold (d) value under 0.2 (d = < 0.2). The 44 items in the survey that met the Triangular Fuzzy Number requirements therefore received approval, while 7 items were rejected. This finding suggested that 44 items have achieved expert agreement and are therefore validated as essential elements and constructs in the gamification learning framework in language learning. FDM can incorporate specialized viewpoints to achieve agreement on validating the constructs, guaranteeing dependability and significance of the created items.

Framework, Fuzzy Delphi, Gamification

1. Adkins, S. (2017) The 2017-2022 Global Game-Based Learning Market. METAARI Advanced Learning Technology Research. Serious Play Conference. https://branden.biz/wpcontent/uploads/2019/02/Metaari_2017-2022_Global_Gamebased_Learning_Market_Executive_Overview.pdf [Google Scholar] [Crossref]

2. Adzmi, N. A., Bidin, S., Selvaraj, B., Saad, S. (2024). The Role of Gamification in Enhancing Engagement and Motivation in Language Learning. International Journal of Research and Innovation in Social Science (IJRISS), 8(09), 2402-2411. https://doi.org/https://dx.doi.org/10.47772/IJRISS.2024.8090197 [Google Scholar] [Crossref]

3. Chen, J., & Liang, M. (2022). Play hard, study hard? The influence of gamification on students’ study engagement. Frontiers in Psychology, 13. https://www.frontiersin.org/articles/10.3389/fpsyg.2022.994700 [Google Scholar] [Crossref]

4. Cheng, C.-H., & Lin, Y. (2002). Evaluating the Best Main Battle Tank using Fuzzy Decision Theory. European Journal of Operational Research, 142, 174–186. https://doi.org/10.1016/S03772217(01)00280-6 [Google Scholar] [Crossref]

5. Christensen, C.M. (1997) The Innovator’s Dilemma: When New Technologies Cause Great Firms to Fail. Harvard Business School Press. Cambridge. https://www.hbs.edu/faculty/Pages/item.aspx?num=46 [Google Scholar] [Crossref]

6. Chu, H. C., & Hwang, G. J. (2008). A Delphi-Based Approach to Developing Expert Systems with the Cooperation of Multiple Experts. Expert Systems with Applications, 34(4), 2826– 2840. https://doi.org/10.1016/j.eswa.2007.05.034 [Google Scholar] [Crossref]

7. Clayton, M. (1992) Identifying critical entry competencies for beginning special educators: the Delphi study, Unpublished doctoral dissertation, University of Illinois, Champaign-Ur https://hdl.handle.net/2142/21976 [Google Scholar] [Crossref]

8. Conole, G. (2016) MOOCs as disruptive technologies: strategies for enhancing the learner experience and quality of MOOCs. RED-Revista de Educación a Distancia. Núm. 50. Art. 2. 15-Jul-2016 DOI: http://dx.doi.org/10.6018/red/50/2 [Google Scholar] [Crossref]

9. Deci, E. L., & Ryan, R. M. (2000). The "what" and "why" of goal pursuits: Human needs and the selfdetermination of behavior. Psychological Inquiry, 11(4), 227-268. https://doi.org/10.1207/S15327965PLI1104_01 [Google Scholar] [Crossref]

10. Deterding, S. (2012). Gamification: Designing for motivation. Interactions, 19, 14-17. https://dl.acm.org/doi/fullHtml/10.1145/2212877.2212883 [Google Scholar] [Crossref]

11. Deterding, S., Dixon, D., Khaled, R., & Nacke, L. (2011). From game design elements to gamefulness: defining" gamification". In Proceedings of the 2011 annual conference on Human factors in computing systems (pp. 2425-2428). ACM. https://doi.org/10.1145/1978942.1979307 [Google Scholar] [Crossref]

12. Duffield, C. (1993). The Delphi technique: a comparison of results obtained using two expert panels. International Journal of Nursing Studies, 30(3), 227–237. https://doi.org/10.1016/0020-7489(93)90033Q [Google Scholar] [Crossref]

13. Fisk, P. (2017, January 17). Education 4.0 … the future of learn-ing will be dramatically different, in school and throughout life. Peter Fisk. https://www.peterfisk.com/2017/01/future-education-youngeveryone-taught-together/ [Google Scholar] [Crossref]

14. Giannarou, L., & Zervas, E. (2014). Using Delphi technique to build consensus in practice: Methodological considerations. International Journal of Business Science & Applied Management. 9 (2), 65-82. https://www.econstor.eu/bitstream/10419/190657/1/09_2_p65-82.pdf [Google Scholar] [Crossref]

15. Gini, F., et. al. (2025) The role and scope of gamification in education: A scientometric literature review. Acta Psychologica Volume 259, 105418, ISSN 0001-6918, https://doi.org/10.1016/j.actpsy.2025.105418. [Google Scholar] [Crossref]

16. Haenlein, M., & Kaplan, A. (2019). A Brief History of Artificial Intelligence: On the Past, Present, and Future of Artificial Intelligence. California Management Review, 61, 5-14. https://doi.org/10.1177/0008125619864925 [Google Scholar] [Crossref]

17. Jamil, M. R. M., & N, M. Noh. (2020). Kepelbagaian Metodologi Dalam Penyelidikan Reka Bentuk dan Pembangunan. Shah Alam, Selangor. Qaisar Prestige Resource. [Google Scholar] [Crossref]

18. Lutfi, A., Aftinia, F., & Permani, B. E. (2023). Gamification: Game as a medium for learning chemistry to motivate and increase retention of students’ learning outcomes. Journal of Technology and Science Education, 13(1), 193-207. https://doi.org/10.3926/jotse.1842 [Google Scholar] [Crossref]

19. Lynkova, D. (2019). Top Gamification Statistics of 2020: Next Level Gaming. Review 42. Retrieved June 10, 2020 from: https://review42.com/gamification-statistics/ [Google Scholar] [Crossref]

20. Mastam, N. M., Zaharudin, R., & Noor, M. N. M. (2024). The Fuzzy Delphi Method: Validation of the Personal Qualities and Abilities Elements in Career Readiness Instrument for Students with Disabilities. International Journal of Academic Research in Progressive Education and Development, 13(3), 2665– 2677. https://doi.org/10.6007/IJARPED/v13-i3/22402 [Google Scholar] [Crossref]

21. Müller, C., & Mildenberger, T. (2021). Facilitating Flexible Learning by Replacing Classroom Time with an Online Learning Environment: A Systematic Review of Blended Learning in Higher Education. Educational Research Review, 34, Article 100394. https://doi.org/10.1016/j.edurev.2021.100394 [Google Scholar] [Crossref]

22. Nasir, S. M., Hashim, M. E. A. H., Nee, C. K., Kamaruddin, N. H., Wibawanto, W., Hanapiah, M. L. H. M., (2026) Enhancing Student Engagement in Learning Management Systems through Exploration of Avatars in Virtual Classrooms: A Systematic Review. Journal of Advanced Research in Applied Sciences and Engineering Technology 63, Issue 2 (2026) 187-200. Semarak Ilmu Publishing. ISSN: 2462-1943 https://semarakilmu.com.my/journals/index.php/applied_sciences_eng_tech/article/view/10664?articlesB ySimilarityPage=65 [Google Scholar] [Crossref]

23. Oazizi, S., Karimov, A., Saarela, M. (2024) Gamification and motivation in education: A systematic literature review. eLearn 2024. 616–624. https://jyx.jyu.fi/bitstreams/03f2f6ac-27f4-4807-9d49142bd3cf7fda/download [Google Scholar] [Crossref]

24. Okoli, C., & Pawlowski, S. D. (2004). The Delphi method as a research tool: An example, design considerations and applications. Information Systems Journal, 12(1), 3–21. https://doi.org/10.1016/j.im.2003.11.002 [Google Scholar] [Crossref]

25. Piaget, J. (1977). The Development of Thought. Equilibration of Cognitive Structures. Oxford: Basil Blackwell. [Google Scholar] [Crossref]

26. Plooy, d. E., Casteleijn, D., Franzsen, D. (2024) Personalized adaptive learning in higher education: A scoping review of key characteristics and impact on academic performance and engagement. Heliyon, [Google Scholar] [Crossref]

27. Volume 10, Issue 21, e39630, ISSN 2405-8440, https://doi.org/10.1016/j.heliyon.2024.e39630 [Google Scholar] [Crossref]

28. Powell C. (2003). The Delphi technique: myths and realities. Journal of advanced nursing, 41(4), 376– 382. https://doi.org/10.1046/j.1365-2648.2003.02537.x [Google Scholar] [Crossref]

29. Rodrigues et al. (2022) Gamification suffers from the novelty effect but benefits from the familiarization effect: Findings from a longitudinal study. Int J Educ Technol High Educ 19:13. https://doi.org/10.1186/s41239-021-00314-6 [Google Scholar] [Crossref]

30. Rowe, G., & Wright, G. (2001). Expert opinions in forecasting: The role of the Delphi technique. Futures, 31(3), 606–626. https://www.gwern.net/docs/predictions/2001-rowe.pdf [Google Scholar] [Crossref]

31. Ryan, R. M., & Deci, E. L. (2000). Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being. American Psychologist, 55(1), 68–78. https://doi.org/10.1037/0003066X.55.1.68 [Google Scholar] [Crossref]

32. Shang Z. (2023). Use of Delphi in health sciences research: A narrative review. Medicine, 102(7), e32829. https://doi.org/10.1097/MD.0000000000032829 [Google Scholar] [Crossref]

33. Sever, D. (2021). Program evaluation experts’ competencies: A Delphi study. International Journal of Contemporary Educational Research, 8(2), 131-142. DOI: https://doi.org/10.33200/ijcer.854511 [Google Scholar] [Crossref]

34. Shute, V. J. (2008). Focus on Formative Feedback. Review of Educational Research, 78(1), 153-189. https://doi.org/10.3102/0034654307313795 (Original work published 2008) [Google Scholar] [Crossref]

35. Skinner, B. F. (1957). Verbal behavior. Appleton-Century-Crofts. https://doi.org/10.1037/11256-000 [Google Scholar] [Crossref]

36. Smirdele, R., & Rigo, S. M. (2020). The impact of gamification on student's learning, engagement and behavior based on their personality traits. Smart learn, 7(3). https://doi.org/10.1186/s40561-019-0098-x 37. Fatah, P. F. A. (2025) The Effectiveness of Gamification in Enhancing English Language Learning Outcomes. Jurnal Pendidikan dan Sastra Inggris. Volume. 5 Nomor. 2 Agustus 2025. e-ISSN: 2827-8860; p-ISSN: 2827-8852, Hal. 304-320. DOI: https://doi.org/10.55606/jupensi.v5i2.5238 [Google Scholar] [Crossref]

37. Thomaidis, N. S., Nikitakos, N., & Dounias, G. D. (2006). The Evaluation of Information Technology Projects: A Fuzzy Multicriteria Decision-Making Approach. International Journal of Information Technology & Decision Making, 5(1), 89–122. https://doi.org/10.1142/S0219622006001897?urlappend=%3Futm_source%3Dresearchgate.net%26utm_ medium%3Darticle [Google Scholar] [Crossref]

38. Vasalou, A., Joinson, A., Bänziger, T., Goldie, P., Pitt, J. (2008). Avatars in social media: Balancing accuracy, playfulness and embodied messages, International Journal of Human-Computer Studies, Volume 66, Issue 11, Pages 801-811, ISSN 1071-5819, https://doi.org/10.1016/j.ijhcs.2008.08.002 [Google Scholar] [Crossref]

39. Vygotsky, L. S. (1978). Mind in society: The development of higher psychological processes. Harvard University Press. [Google Scholar] [Crossref]

40. Yechkalo, Y. V., Tkachuk, V. V., Shurupova, K. A., Zinchenko, V. M., & Puhach, A. V. (2024). Gamification in higher education: Methodology. CEUR Workshop Proceedings, 96–106. Retrieved from https://ceur-ws.org/Vol-3482/paper000.pdf [Google Scholar] [Crossref]

• Assessment of the Role of Artificial Intelligence in Repositioning TVET for Economic Development in Nigeria
• Teachers’ Use of Assure Model Instructional Design on Learners’ Problem Solving Efficacy in Secondary Schools in Bungoma County, Kenya
• “E-Booksan Ang Kaalaman”: Development, Validation, and Utilization of Electronic Book in Academic Performance of Grade 9 Students in Social Studies
• Analyzing EFL University Students’ Academic Speaking Skills Through Self-Recorded Video Presentation
• Major Findings of The Study on Total Quality Management in Teachers’ Education Institutions (TEIs) In Assam – An Evaluative Study