“Identifying Challenges in HOTS Implementation among Novice Vocational Lecturers: A Fuzzy Delphi Perspective”

“Identifying Challenges in HOTS Implementation among Novice Vocational Lecturers: A Fuzzy Delphi Perspective”

Nur Ain Md Lukmanul Hakim., Mohamad Rasidi Pairan., Mohamad Ikram Zakaria

Faculty of Educational Sciences and Technology, Universiti Teknologi Malaysia, Johor, Malaysia

DOI: https://dx.doi.org/10.47772/IJRISS.2025.905000445

Received: 16 May 2025; Accepted: 20 May 2025; Published: 21 June 2025

ABSTRACT

In the context of Education 4.0, the integration of Higher-Order Thinking Skills (HOTS) is essential for preparing vocational students to meet complex workplace demands. However, novice lecturers often struggle to implement HOTS-based instruction effectively due to limited pedagogical expertise, time constraints, and underdeveloped assessment literacy. This study investigates these challenges using Shulman’s Pedagogical Content Knowledge (PCK) framework and the Fuzzy Delphi Method (FDM), a structured technique enhanced by fuzzy set theory for achieving expert consensus. Data were collected from 25 novice lecturers at vocational colleges in Johor, Malaysia. Nine major challenges were validated, with the most critical being student engagement with HOTS learning and lecturers’ limited ability to apply rubric-based assessments. Contrary to prior research, constructing HOTS instruments was not viewed as a major issue, indicating a shift in current support systems. These findings suggest that professional development should prioritize rubric training, differentiated instruction, and efficient planning strategies. The study contributes methodologically through the application of FDM in an educational context and practically by proposing a mentorship-based framework to enhance HOTS implementation in vocational education.

Keywords— Higher-Order Thinking Skills (HOTS), Vocational Colleges, Pedagogical Content Knowledge (PCK), Fuzzy Delphi Method(FDM), Teaching Challenges, Novice Lecturer

INTRODUCTION

In the era of Education 4.0, the integration of advanced technologies and evolving global competencies has transformed the landscape of teaching and learning. Traditional rote-based instruction is no longer sufficient; instead, students must be equipped with essential 21st-century skills such as critical thinking, creativity, and decision-making to navigate real-world challenges[1, 2]. Higher-Order Thinking Skills (HOTS) have emerged as a cornerstone of this transformation, enabling students to analyze, evaluate, and generate solutions across disciplinary contexts.

The importance of HOTS in vocational education is particularly pressing. Vocational students must not only master theoretical knowledge but also apply it practically in problem-solving situations reflective of workplace demands [3].

However, the successful implementation of HOTS requires educators who possess both deep content knowledge and the pedagogical expertise to foster high-level cognitive engagement. For novice lecturers, this transition is especially challenging. Limited teaching experience, high workloads, and underdeveloped assessment literacy often impede their ability to implement HOTS-based learning effectively [4, 5].

To understand and support the development of pedagogical effectiveness among these educators, this study is grounded in Shulman’s Pedagogical Content Knowledge (PCK) framework. PCK emphasizes the integration of content knowledge (CK), pedagogical knowledge (PK), and knowledge of students (KS), enabling teachers to transform subject matter into comprehensible instruction tailored to diverse learners [6, 7]. By applying the PCK model, this study examines how deficits in CK, PK, and assessment strategies hinder HOTS implementation among novice lecturers in vocational settings.

While previous studies have documented various barriers to HOTS integration, such as limited professional training and students’ low critical thinking ability [8, 9], they have primarily relied on qualitative interviews or small-scale observations. This approach limits the generalizability and prioritization of the most critical issues. Moreover, much of the existing literature highlights the construction of HOTS assessments as a key concern [10, 11]. However, emerging trends suggest that rubric application and instructional execution may pose greater obstacles in contemporary settings.

To address these gaps, this study applies the Fuzzy Delphi Method (FDM) a structured, consensus-building tool enhanced by fuzzy set theory. FDM allows for a more precise quantification of expert agreement, providing deeper insights into which challenges are most impactful and widely experienced [12, 13]. This methodology not only strengthens the study’s rigor but also offers a novel contribution to HOTS related educational research. Furthermore, this study proposes a mentorship-driven development model tailored to vocational contexts, contributing both theoretically and practically to the improvement of HOTS integration among novice lecturers.

METHODOLOGY

The methodology section outlines the specific approach undertaken in this study, including the research design, participant group selection, research instrument, validity and reliability, and analysis procedures.

Research Design

This study was conducted using a quantitative approach based on the Fuzzy Delphi Method (FDM). FDM enhances the classical Delphi technique by integrating fuzzy set theory, allowing expert opinions to be quantified with greater precision. This method is particularly valuable in educational research where subjective judgments such as pedagogical challenges require nuanced interpretation [12, 14]. Compared to standard Delphi methods, FDM reduces bias from dominant voices and accommodates uncertainty in linguistic expressions [13]. The research process included a systematic literature review, instrument development, expert validation, data collection from novice lecturers, and fuzzy-based consensus analysis.

Participant selection

Purposive sampling was used to ensure that the expert panel consisted of novice lecturers with less than five years of teaching experience in vocational colleges. This sampling strategy aligns with the study’s objective of capturing emerging pedagogical challenges while ensuring participants had sufficient exposure to HOTS-based instruction. In line with FDM protocols, the inclusion of informed but diverse participants helps enhance the authenticity and richness of expert consensus [12, 15]. The pilot study involved 15 participants, and the final study included 25 novice lecturers from vocational institutions in Johor, Malaysia.

Research Instrument

The primary data collection instrument was a structured questionnaire designed based on an extensive literature review and grounded in Shulman’s PCK model. Items targeted aspects of CK, PK, and KS relevant to HOTS integration. The questionnaire consisted of items rated on a 7-point Likert scale (1 = Strongly Disagree to 7 = Strongly Agree), which were later converted into triangular fuzzy numbers for analysis. To ensure content validity, the instrument was reviewed by two experts in vocational education and one instructional design specialist [16].

Validity and Reliability

Instrument validity was evaluated through both face and content validity. The Content Validity Index (CVI) was applied to assess item relevance, with each item achieving a CVI score above the 0.75 threshold as recommended by Polit et al.[17]. A pilot study with 15 novice lecturers was conducted to assess internal consistency. The results yielded a Cronbach’s Alpha of 0.667, which is considered acceptable for exploratory studies in educational research [18].

Data Analysis Procedure

To enhance clarity, an example is provided to illustrate the FDM process:

Assume a Likert response of 7 (Strongly Agree). This is transformed into a triangular fuzzy number: (0.90, 1.00, 1.00). For a response of 5 (Somewhat Agree), the fuzzy number is (0.50, 0.60, 0.70). The threshold value (d) is calculated by comparing the distance between fuzzy numbers. If d ≤ 0.2, a strong agreement is indicated. Consensus is confirmed if 75% or more experts agree. Defuzzification then converts fuzzy values into crisp scores using the formula A = (l + 4m + u) / 6.

Where l = lower bound, m = middle value, u = upper bound. For example, with (0.50, 0.60, 0.70), the defuzzified score is A = (0.50 + 4*0.60 + 0.70) / 6 = 0.60. This score is then used to rank the items, with higher A values indicating greater consensus on criticality.

The Fuzzy Delphi Method followed a structured multi-step analysis:

Transformation to Fuzzy Numbers: Likert-scale responses were converted into triangular fuzzy numbers (min, median, max). For example, a score of 7 was transformed into (0.90, 1.00, 1.00) [19].

Threshold Value (d): The distance between fuzzy numbers was calculated. A threshold value of d ≤ 0.2 indicated strong agreement [20].

Consensus Measurement: Participant consensus was determined using both threshold values and percentage agreement. A consensus level of ≥75% was adopted, in line with FDM best practices [21].

Defuzzification: Fuzzy values were converted into crisp scores (A) to rank each challenge. An α-cut value ≥ 0.5 was required to retain items [22].

Item Ranking: The final list of HOTS implementation challenges was ranked based on the defuzzified score to identify the most critical barriers faced by novice lecturers.

FINDING

The demographic profile of the 25 novice vocational lecturers who participated in this study is summarized in Table 1. It includes age distribution, educational background, teaching experience, and teaching specialization.

Table 1 Demographic of the participants

Following the FDM, nine key challenges in implementing HOTS were validated based on expert consensus. One item “I have problems in constructing HOTS instruments to assess my students” was rejected as it failed to meet the minimum 67% agreement threshold, indicating it was not widely perceived as a major issue.

The top three ranked challenges were:

Ensuring students understand and follow HOTS-based learning

Limited knowledge in applying rubrics for HOTS evaluation

Over-reliance on reference materials during HOTS integration

These findings indicate that novice lecturers primarily struggle with instructional delivery and assessment application rather than with instrument design. This is an important shift from prior research that identified test construction as a core barrier [10, 11]. The rejected item suggests that recent training initiatives or available assessment templates may have reduced concern over instrument design.

The top-ranked challenge, which emphasizes student understanding, reflects a deficiency in the KS component of Shulman’s PCK model. Novice lecturers appear to have difficulty gauging and supporting student readiness for higher-order tasks, especially in practical vocational contexts. The second-ranked challenge points to limited PK regarding the use of rubrics an essential tool for assessing HOTS performance. This suggests that while structural assessment tools may exist, the skill to implement them remains underdeveloped among new lecturers.

Table 2 summarizes the threshold values, consensus percentages, fuzzy scores, and final rankings for each challenge, offering a prioritized view of the implementation barriers identified through expert feedback.

Table 2   Final Rankings For Each Challenge

DISCUSSION

This study set out to identify the key challenges novice lecturers face in implementing HOTS in vocational education. Using the FDM, the study revealed nine validated challenges, offering both a systematic prioritization of issues and novel insights into the evolving teaching landscape.

The most significant challenge identified ensuring students understand and follow HOTS-based instruction reflects a deficiency in lecturers’ ability to translate content into cognitively engaging learning experiences. This finding directly corresponds to the KS element in Shulman’s PCK framework, indicating that novice lecturers may lack the tools or strategies to scaffold complex thinking tasks effectively. This reinforces the findings of [4], who noted that novice educators often struggle with student engagement and classroom differentiation. This challenge aligns with the KS component of the PCK framework, highlighting difficulties in addressing student readiness and engagement.

Interestingly, one of the key differentiators of this study is the rejection of the item “constructing HOTS instruments to assess students.” While prior research such as [10] and [11] highlighted this as a major concern, the current findings suggest a shift in instructional support possibly due to improved institutional resources, increased use of digital assessment platforms, or access to standardized HOTS frameworks. This change marks an important empirical contribution, indicating that training efforts may be beginning to reduce assessment design anxiety among novice lecturers.

The second-highest ranked issue limited knowledge in applying rubrics points to gaps in PK. Although lecturers may have access to assessment tools, the lack of expertise in effectively using them hinders valid evaluation of student performance. This aligns with findings by [23], who emphasized that training in rubric design and use is essential for accurate measurement of HOTS. This reflects a deficiency in PK, especially in rubric-based assessment practices. Moreover, challenges related to instructional content adaptation such as the use of appropriate examples or contextualization highlight deficiencies in CK, as lecturers may struggle to connect curriculum material to real-world applications.

The recurring theme of time constraints reflected in two separate challenges highlights institutional and structural barriers. Novice lecturers often manage heavy workloads with limited planning autonomy. This suggests a systemic issue, which institutions can address by offering collaborative planning time, model lesson plans, and streamlined administrative processes [24]. These challenges suggest overlapping issues related to both PK and institutional support, impacting the effective planning and delivery of HOTS lessons.

From a methodological perspective, the use of FDM represents a significant departure from prior studies that employed purely qualitative or descriptive techniques. FDM’s integration of fuzzy set logic allowed for a structured consensus process, revealing both the strength and spread of agreement across experts. This adds analytical depth to the discourse on HOTS implementation in vocational settings and enhances the credibility of the prioritization.

Collectively, the findings support the need for a mentorship-driven professional development framework. Rather than focusing solely on tool creation, such programs should emphasize effective rubric use, differentiated instruction, and time-efficient HOTS lesson strategies. These priorities reflect the real needs of novice lecturers in technical and vocational education and contribute to the operationalization of Shulman’s PCK model in practice.

CONCLUSION

This study applied the FDM to identify and prioritize the challenges faced by novice lecturers in implementing HOTS in vocational colleges, using Shulman’s PCK framework as its conceptual foundation. The results provide new insights into how novice lecturers struggle not only with instructional delivery but also with aligning assessment practices and student readiness to HOTS based learning environments.

The most pressing issue identified was the difficulty in ensuring that students truly understand and engage with HOTS oriented instruction, highlighting a gap in lecturers’ KS. Limited knowledge in applying rubrics and an over-reliance on reference materials point to underdeveloped PK, while the rejection of “HOTS instrument construction” as a major issue contrasts with earlier studies and reveals a positive shift in institutional support or training access.

Methodologically, this study adds value by utilizing the FDM, which enhances traditional Delphi techniques by incorporating fuzzy logic to improve the accuracy of expert consensus. This structured and quantifiable approach offers an innovative lens for analyzing pedagogical challenges, contributing a rigorous model that future educational studies can adopt.

Practically, the study proposes a mentorship-based framework for professional development that prioritizes rubric usage, student engagement strategies, and efficient HOTS lesson planning. The findings also highlight the need for systemic support that reduces time constraints and provides ongoing coaching for novice lecturers.

While the study offers clear contributions, it is limited to the perspectives of novice lecturers in Johor and lacks classroom observation or longitudinal tracking. Future research should expand the scope geographically and include intervention-based models to test the effectiveness of proposed support strategies. Long-term studies assessing the impact of targeted professional development on HOTS outcomes will also be valuable for advancing vocational education policy and practice.

REFERENCES

1. E. H. Tyas and L. Naibaho, “HOTS learning model improves the quality of education,” International Journal of Research-GRANTHAALAYAH, vol. 9, no. 1, pp. 176-182, 2021.
2. R. Jaenudin, U. Chotimah, F. Farida, and S. Syarifuddin, “Student development zone: higher order thinking skills (hots) in critical thinking orientation,” International Journal of Multicultural and Multireligious Understanding, vol. 7, no. 9, pp. 11-19, 2020.
3. N. Markhamah, “Pengembangan soal berbasis HOTS (higher order thinking skills) pada kurikulum 2013,” Nusantara: Jurnal Pendidikan Indonesia, vol. 1, no. 2, pp. 385-418, 2021.
4. F. P. Khalisa, S. Mirizon, and E. Eryansyah, “Challenges and Strategies in Early Teaching Experience: Voice of Novice EFL Indonesian Lecturers,” Voices of English Language Education Society, vol. 6, no. 2, pp. 417-428, 2022.
5. J. Abance and J. Waking, “Exploring the Work Environment of Novice Teachers and its Challenges,” International Journal of Research Publication and Reviews, 2023.
6. L. S. Shulman, “Knowledge Growth in Teaching,” Educational Researcher, vol. 15, no. 2, pp. 4-14, 1986.
7. A. A. Raya and R. V. P. Hartanto, “Pedagogical Content Knowledge (PCK) Analysis for Teachers: A Case Study of Pancasila and Citizenship Education (PPKn) Teachers on Bhinneka Tunggal Ika Material,” Open Access Indonesia Journal of Social Sciences, vol. 7, no. 3, pp. 1517-1523, 2024.
8. C. K. S. Singh and P. Marappan, “A review of research on the importance of higher order thinking skills (HOTS) in teaching english language,” Journal of Critical Reviews, vol. 7, no. 8, pp. 740-747, 2020.
9. M. A. A. Wibowo and S. Munadi, “Critical Thinking Skills through Industrial Revolution 4.0 for Vocational Education,” VANOS Journal of Mechanical Engineering Education, vol. 5, no. 2, 2020.
10. N. W. Pratiwi, N. S. Dewi, and A. Y. Paramartha, “The reflection of HOTS in EFL teachers’ summative assessment,” Journal of Education Research and Evaluation, vol. 3, no. 3, pp. 127-133, 2019.
11. P. Hartini, E. Ernawati, and H. Setiadi, “LOTS and HOTS Capabilities as Assessment Instruments Made by Elementary School Teachers in Jakarta,” 2021.
12. M. I. Zakaria, N. N. A. H. Noor, A. A. Halim, A. N. S. Ahmad, and H. M. F. Abdul, “Heutagogy in Action: Empowering Mathematics Teachers Through Innovative Pedagogical Approaches,” International Journal of Instruction, vol. 17, no. 4, pp. 135-148, 10/01 2024. [Online]. Available: https://e-iji.net/ats/index.php/pub/article/view/646.
13. K. Memmedova and B. Ertuna, “Development of a fuzzy Likert scales to measure variables in social sciences,” Information Sciences, vol. 654, p. 119792, 2024/01/01/ 2024, doi: https://doi.org/10.1016/j.ins.2023.119792.
14. Q. Xing, H. P. Tan, and S. W. Gan, “Constructing a student development model for undergraduate vocational universities in China using the Fuzzy Delphi Method and Analytic Hierarchy Process,” Plos one, vol. 19, no. 3, p. e0301017, 2024.
15. A. M. Persky and J. D. Robinson, “Moving from Novice to Expertise and Its Implications for Instruction,” (in eng), Am J Pharm Educ, vol. 81, no. 9, p. 6065, Nov 2017, doi: 10.5688/ajpe6065.
16. H. M. Yusoff, P. P. Heng, M. R. H. Illias, S. Karrupayah, M. A. Fadhli, and R. Hod, “A qualitative exploration and a Fuzzy Delphi validation of high-risk scaffolding tasks and fatigue-related safety behavioural deviation among scaffolders,” Heliyon, vol. 10, no. 15, 2024.
17. D. F. Polit, C. T. Beck, and S. V. Owen, “Is the CVI an acceptable indicator of content validity? Appraisal and recommendations,” Research in nursing & health, vol. 30, no. 4, pp. 459-467, 2007.
18. L. Sürücü and A. Maslakci, “Validity and reliability in quantitative research,” Business & Management Studies: An International Journal, vol. 8, no. 3, pp. 2694-2726, 2020.
19. M. F. M. Yaakob et al., “A quest for experts’ consensus on the Geo-education module using Fuzzy Delphi analysis,” Universal Journal of Educational Research, 2020.
20. H. Wang, “Teaching Evaluation of Marketing Specialty in Higher Vocational Colleges Based on Fuzzy Delphi Method,” marketing, vol. 6, no. 20, pp. 93-99, 2023.
21. F. B. Abdul Rahman, Z. Mustafa, and A. F. Kharuddin, “Employing Fuzzy Delphi Technique to Validate Multiple Intelligence Based Instructional Teaching Module for Preschool Children,” Southeast Asia Early Childhood, vol. 10, no. 1, pp. 62-71, 2021.
22. K. M. Yusoff, N. S. Ashaari, T. S. M. T. Wook, and N. M. Ali, “Validation of the Components and Elements of Computational Thinking for Teaching and Learning Programming using the Fuzzy Delphi Method,” (in English), Intl. J. Adv. Comput. Sci. Appl., Article vol. 12, no. 1, pp. 80-88, 2021, doi: 10.14569/IJACSA.2021.0120111.
23. N. a. Wahab, D. F. Ali, and A. R. Ahmad, “Professional Development for Novice TVET Teachers: A Systematic Review,” International Journal of Academic Research in Progressive Education and Development, 2024. [Online]. Available: https://hrmars.com/index.php/IJARPED/article/view/23654/Professional-Development-for-Novice-TVET-Teachers-A-Systematic-Review.
24. F. M. Zain, S. N. Sailin, M. Kasim, A. M. A. Karim, and N. N. Zameri, “Developing an Augmented Reality Immersive Learning Design (AILeaD) Framework: A Fuzzy Delphi Approach,” International Journal of Interactive Mobile Technologies, vol. 16, no. 11, 2022.