Revising Rubric as Part of Continual Quality Improvement (CQI) in Assessing Affective Domain for the Environmental Laboratory Engineering Course

Revising Rubric as Part of Continual Quality Improvement (CQI) in Assessing Affective Domain for the Environmental Laboratory Engineering Course

Salina Alias^*, Amir Khomeiny Ruslan, Muhammad Hasbullah Hassan Basri, Zuhaida Mohd Zaki

Civil Engineering Studies, College of Engineering, Universiti Teknologi MARA, Cawangan Pulau Pinang, Permatang Pauh Campus, 13500 Seberang Prai, Pulau Pinang, Malaysia.

^*Corresponding Author

DOI: https://dx.doi.org/10.47772/IJRISS.2025.903SEDU0077

Received: 22 January 2025; Accepted: 03 February 2025; Published: 08 March 2025

ABSTRACT

This continual quality improvement (CQI) is a vital process in enhancing educational tools such as rubrics to ensure they remain relevant, equitable, and effective. This paper details the iterative development and refinement of a rubric for assessing the affective domain, focusing on individual responsibility and teamwork, in the Environmental Laboratory course for engineering students. Since assessing the affective domain involves emotional and psychological components, a robust and adaptable rubric is essential. Beginning in 2015, the rubric underwent multiple stages of revision to address limitations in specificity and fairness. Initial versions relied on general descriptors to evaluate disciplinary and communication skills, teamwork, and leadership, leading to significant score variability. In 2017, a revised rubric introduced detailed criteria based on Bloom’s affective domain taxonomy, improving its focus on personal responsibility and group interaction. Further adjustments during the COVID-19 pandemic demonstrated the rubric’s adaptability, shifting its focus to assess engagement during virtual learning. The CQI process has proven critical in minimising score variability and enhancing student outcomes, making the rubric more effective over time. This study underscores the importance of CQI in refining assessment tools to meet evolving educational challenges and better support student development in the affective domain.

Keywords— Affective Domain, Continual quality improvement (CQI), Engineering Pedagogy, Programme Outcome, Rubric

INTRODUCTION

Assessing the affective domain can be crucial and challenging in teaching and learning as it relates to measuring physiological feelings, behavioural attitudes, emotional intelligence, expression, value, and management [1]. The challenge lies in its subjective and personal nature, as it is shaped by diverse learning experiences, perspectives, and lessons, all of which contribute to positive outcomes such as increased knowledge and skill development. However, these factors make the evaluation process more intricate and interconnected, with numerous characteristics that are challenging to isolate and measure objectively. Traditional assessment methods, such as tests or assignments, often fail to capture the depth and complexity of the affective domain, particularly in evaluating critical attributes like leadership skills, communication, and emotional intelligence.

Recognising this challenge, Chaieb et al. [2] highlighted incorporating affective domain elements (ADEs) such as leadership, communication, and self-awareness during admission interviews for the Doctor of Pharmacy program. By integrating ADEs into admission interviews, admissions teams can identify candidates who excel academically and possess the emotional and social competencies necessary for success in the program. This holistic approach ensures that students possess the well-rounded qualities needed for both academic and professional growth.

In engineering programs, the affective domain is an essential assessment component. It should be integrated into relevant graduate attributes, as outlined in the International Engineering Alliance (IEA) Graduate Attributes & Professional Competencies [3]. The IEA identifies twelve (12) key graduate attributes that encompass the knowledge, skills, and behaviors expected of engineering graduates upon completing their program. These attributes serve as a benchmark for assessing the readiness and competence of graduates in meeting professional engineering standards.

In Malaysia, the Engineering Accreditation Council (EAC), under the Board of Engineers Malaysia (BEM), plays a pivotal role as the signatory body of the Washington Accord, an international agreement under the International Engineering Alliance (IEA) for accrediting bachelor-level engineering programs. Since 2009, the EAC has accredited bachelor-level programs to align with global requirements, while the BEM supports this effort by promoting the development of qualified and internationally recognized engineers.

Therefore, the engineering program offered by the Institute of Higher Learning (IHL) in Malaysia will be accredited by the BEM to promote continuous improvement and effectiveness within IHLs. This ensures that IHLs’ education remains relevant and responsive to the evolving needs of stakeholders, including industry and society, as well as environmental changes such as technological advancements and globalisation.

The graduate attributes, as stated in the Washington Accord and EAC guidelines [4], should be written in full detailed statements by the IHL based on the program offered at the IHL. These statements are known as the programme outcomes (POs). At the Civil Engineering Studies, Universiti Teknologi MARA (CES, UiTMCPP), the statements of 12 POs headlines were formulated based on EAC and IEA, as shown in Table 1. These POs are mapped to the specific course outcome (CO) IHL offers. The distribution must be aligned with the learning objective and content of individual courses that reflect the knowledge, skills, and abilities the student should possess upon graduation.

Through POs, IHLs can assess and evaluate their programs, ensuring that they align with the needs of students, employers, and other stakeholders. The assessment of POs occurs upon completion of the degree program, measuring students’ competency in specific learning domains. Seven of the twelve POs were targeted explicitly in the affective learning domain. These include i) the level of knowledge and responsibility of engineers and society; ii) the type of solutions towards the environment and sustainability; iii) understanding and level of practice of ethics; iv) role as an individual and teamwork; v) level of communication; vi) level in project management and finance; and vii) the preparation and ability to engage in lifelong learning. These POs ensure that graduates acquire technical skills and develop professional and personal attributes to succeed in their careers.

One of the POs that an engineering student must attain is the ability of the graduate to function effectively as an individual and as a member or leader in diverse teams and multi-disciplinary settings (PO9). Under this attribute of the affective domain, students are not only required to function well individually but also to have the ability to collaborate in a team setting. This aimed to shape students into a positive dynamic team. To achieve this outcome, the FCE UiTMPP mapped PO9 with the headline “Individual and Teamwork” to five (5) laboratory courses and one (1) design project, as summarised in Table 2. Out of the total percentage mark for each course, a 20% mark was allocated to assess the affective domain of PO9.

Table II

Courses at CES UiTMPP that align to PO9

However, choosing the appropriate assessment method can be challenging, as it measures changes in attitude and should be easily adapted over time. Therefore, appropriate assessments that are more applicable, realistic, and quantifiable need to be established to ensure that students attain the necessary skills and competencies for that specific program outcome. Therefore, for the Environmental Laboratory subject (CEW545) at the FCE UiTMPP, the affective domain of PO9 was assessed through laboratory observations, focusing on the behaviour and attitude of students when receiving, reacting, and participating in the experiment. This pedagogy of experiential learning also observes students’ skills when performing the task given in a particular experiment. Before observation, the assessor and students must thoroughly grasp the evaluation criteria, which can be put into a structured assessment method called a rubric.

A rubric, as a tool of scoring guide that outlines the criteria, different performance levels and descriptors, is an essential and practical instrument in communicating the assessment evaluation between the assessor and students. The rubric describes the elements of competency that formulate complex skills through descriptors, provides the level of performance based on rating scales, and helps the student achieve the desired outcome [5]. Therefore, a well-crafted rubric grounded in pedagogical principles is essential for evaluating students’ affective skills involving attitudes, thoughts, and emotions. A clear rubric framework will guide students in their learning journey not only for evaluation but also for giving instruction or expectations of the assessment to the student [6]. Gregori-Giralt and Menéndez-Varela [7] highlighted that rubrics are not widely used in higher education, and their adaptation requires guidance on how to use the rubric, particularly to achieve consistency in complex evaluations. The insightful and clear rubric will help minimize the discrepancy in assessment marks between assessors by guiding the subjective aspect with constructive guidance elements.

APPROACH

This paper describes how the rubric is refined and the need for refinement in assessing the affective domain, mainly related to the ability of the students to function effectively as an individual and as a member or leader in diverse teams and multi-disciplinary settings for the Environmental Laboratory course offered in Semester 7 at FCE UiTMCPP. The rubric will be refined from 2015 to 2023. The rubric’s dynamic refinement is vital to enhance assessments’ clarity, relevance, fairness, and effectiveness across different lecturers. By regularly updating the rubrics, lecturers can ensure that assessments are aligned with current educational standards, enhance assessment practices, provide meaningful feedback, and accurately measure student learning outcomes fairly and equitably.

The systematic and continuous rubric refinement or Continuous Quality Improvement (CQI) was applied in the course CEW545, as shown in Figure 1. At the end of the semester, students’ outcomes in each subject must be presented at the program level. The limitations and areas of improvement need to be identified based on the students’ performance and current learning outcomes, then aligned with the new requirements in the Washington Accord and/or educational standards of the EAC and the new curriculum development. Subsequently, a new rubric version incorporating detailed descriptors and aligning with Bloom’s taxonomy for the affective domain was drafted. The new rubric was tested, observed, and analyzed based on an authentic assessment. Assessors and students conducted feedback and reviewed the rubric. Then, the necessary refinement of the rubric was conducted to ensure effectiveness in measuring student performance and reflect better learning outcomes.

Fig. 1 CQI Process in Rubric Refinement in CEW545

The EC221 program employs the Plan, Do, Check, Act (PDCA) cycle as part of its Continuous Quality Improvement (CQI) process. Each lecturer teaching a subject must prepare their own CQI analysis for the group they are taught, as illustrated in Figure 2. In this analysis, various elements—such as student grade achievement, student feedback through the Student Feedback Online (SUFO), student entry-exit surveys, course outcomes (COs), and program outcomes (POs) from current and previous semesters— were compared to identify gaps. Any issues and suggestions related to teaching and learning, students, syllabi, assessments, and rubrics must be carefully documented in the CQI template.

Once completed, the CQI template was submitted to the subject course coordinator (CC), who then compiled the analyses from all groups of lecturers. The CC presents the compiled data and CQI outcomes to the head of the division, resource person (RP), and all division members. The findings from the CQI analysis and additional inputs from the meeting inform decisions for further improvement. Then, RP implemented any enhancements related to the syllabus, POs, COs, assessment methods, rubrics, or course content.

As an expert on the subject, the RP plays a crucial role in the program, acting as a mentor and directly involved in curriculum development. The collaborative efforts and valuable feedback exchanged among teaching lecturers, CC, and RP make the CQI process more effective and transparent, ensuring that decisions are based on evidence and best practices.

Fig. 2 Example of CQI template by a lecturer

Several changes to the rubric were made through the process and cycle of the CQI. Table 3 shows the dynamic changes made in the rubrics, focusing on assessing the affective domain from 2016 to 2023 for CEW545. In 2015, a rubric was developed that utilised general behaviours and descriptors (Figure 3). It assesses four primary categories: discipline, communication, teamwork, and management. While the rubric covers individual and teamwork responsibilities to reflect PO9, the level of detail and specificity are relatively limited. Additionally, communication criteria only indirectly address aspects of teamwork, lacking a focused evaluation of how effectively students work together.

OUTPUT & DISCUSSION

Several changes to the rubric were made through the process and cycle of the CQI. Table 3 shows the dynamic changes made in the rubrics, focusing on assessing the affective domain from 2016 to 2023 for CEW545. In 2015, a rubric was developed that utilised general behaviours and descriptors (Figure 3). It assesses four primary categories: discipline, communication, teamwork, and management. While the rubric covers individual and teamwork responsibilities to reflect PO9, the level of detail and specificity are relatively limited. Additionally, communication criteria only indirectly address aspects of teamwork, lacking a focused evaluation of how effectively students work together.

TABLE III

Rubric Development and Refinement from the Year 2015 – 2024 based on CQI Process

Fig. 3 Rubric for Affective Domain Assessment in 2015

Therefore, in response to this limitation, the 2017 rubric introduced more specific and detailed categories and descriptors, as shown in Figure 4. The rubric was developed based on the level of the affective domain in Bloom’s taxonomy, as illustrated in Figure 5. The hierarchical structure of the affective domain of Bloom’s taxonomy consists of five levels of learning, progressing from the lowest (receiving) to the highest (characterising), and was adopted to develop the rubric. The lowest level focuses on the awareness and reception of students towards discipline or rules by students in a specific laboratory experiment. It was formulated to assess student responses and behaviour toward the safety of the experiment, indicating their engagement, awareness, and initial acknowledgement of the importance of rules for the specific experiment. The responsibility of a student or group of students in maintaining the laboratory setting, commitment to follow the laboratory or experimental protocol, and systematic organisation of the given task are essential for practical laboratory sessions and more profound learning exercises. As students progress through the levels, they begin to internalise these feelings of engagement, guiding their behaviour. The rubric captures the positive interaction among group members and respect for each other and articulates the well-reasoned outcome of the experiment.

Fig. 4 Rubric for Affective Domain Assessment in 2017

Fig. 5 Level of Learning for the Affective Domain in Bloom’s Taxonomy

More comprehensive descriptors have been developed and detailed using different criteria to provide clear expectations for students. These include breaking down the assessment criteria for individual adherence to the safety protocol and the ability of the group to maintain the workspace in an orderly manner. Assessing students’ commitment to safety made them more responsible for their actions in the lab and helped them inculcate essential safety professional skills. These include appropriately assessing students’ protective gear use and promptly addressing all safety concerns. Similarly, evaluating a group’s ability to maintain a clean and organised workspace fosters a sense of collective responsibility. This encourages teamwork and communication regarding maintaining standards, which are critical skills in academic and professional settings.

The COVID-19 outbreak shifted most approaches to teaching and learning, resulting from the suspension of face-to-face learning. Drastic student assessment and evaluation adjustments are required worldwide, including in the Environmental Laboratory subject, CEW545 at FCE UiTMCPP. The rubric, more tailored to student-centred learning in a remote learning context and focuses on how well students can self-manage their tasks and responsibilities, was drafted, presented, and endorsed, as shown in Figure 6. The shifting of focus from traditional safety measures to evaluating students’ readiness for class, timeliness in submitting work, organisation, and commitment to lab activities was a practical response to the constraints imposed by remote learning and social distancing guidelines. Instead of physical safety procedures that are more relevant in on-campus settings, readiness can include a student’s preparedness for necessary materials at home, commitment between the group members in preparing the props for a virtual laboratory experiment, and their mental readiness to engage in a virtual class. The pragmatic approach of assessing affective domains that focuses on a student-centred approach is more meaningful, equitable, and supportive of academic and personal circumstances during unprecedented times.

Fig. 6 Rubric for Affective Domain Assessment in 2020

As of the post-COVID-19 era, the rubric has been continuously refined and updated, with the latest version released in 2023, as shown in Figure 7. In 2023, the rubric evolved to include highly detailed descriptors and focused on advanced skills, such as leadership and proactive support. These changes aimed to enhance clarity, precision, and fairness in assessments, ensuring they accurately reflected students’ communication, teamwork, discipline, and management competencies. However, this continuous refinement process also highlights the need to adapt educational assessments to evolving EAC standards requirements for complex problem-solving and knowledge profiles.

Figure 8 presents the overall attainment affective domain of PO9: Ability to function effectively as an individual and as a member or leader in diverse teams and multi-disciplinary settings across different semesters, from March-August 2015 to March–August 2023. The data capture the maximum (brick), minimum (strips), and average (trellis) marks of Program Outcome (PO) attainment over this period. Generally, the marks remained relatively consistent across all semesters, hovering between 16 and 18. This suggests that the highest-performing students consistently met or exceeded the expectations set for the affective domain related to individual responsibility. However, there was a notable fluctuation in the minimum score over time. Early semesters, such as March – August 2015, show a lower minimum mark of around 9, which suggests a significant gap between the highest and lowest performers. The results showed a considerable gap between the maximum and minimum scores, implying that revising the rubric or assessment method is needed.

The revision of the rubric in 2017 decreased the gap between the maximum and minimum scores, as seen in the data from March-August 2015 and March-August 2016. This indicates that some students can perform well while others struggle to meet the assessment requirements. This is likely due to unclear expectations in the rubric. However, a narrow gap between the maximum and minimum student attainment scores was observed after the 2017 rubric revision. Since the rubric provides clear guidelines for the students and the assessors, it increased the minimum score, indicating meeting the expected levels of individual responsibility.

Fig. 7 Rubric for Affective Domain Assessment in 2023

Furthermore, during the semesters overlapping with the COVID-19 pandemic, the minimum scores were slightly lower compared to other periods, particularly in March – August 2020, when they dipped to around 10. This suggests that the pandemic may have posed challenges to student performance, possibly because of disruptions in learning environments and the transition to remote learning. Nevertheless, despite these challenges, the maximum and average scores during the pandemic remained high, indicating that while some students struggled, many could maintain high-performance levels. By March – August 2022, the minimum scores returned to higher levels, indicating recovery as the students adapted to the new normal. The face-to-face laboratory sessions showed a narrowing gap between the minimum and maximum scores based on data from March-August 2022 and March–August 2023, with both the minimum and average scores showing improvement. This trend suggests that the post-pandemic period and the continued use of the revised rubric led to more consistent and improved student performance in the affective domain.

Fig. 8 Student attainment on the affective domain of PO9 from March-August 2015 to March-August 2023

The ANOVA analysis confirms statistically significant differences between the performance groups (Max, Min, and Average), with an F-value of 28.77, far greater than the critical value of 3.47 and a P-value below the standard threshold of 0.05. These ANOVA results indicate that implementing a more structured rubric has effectively contributed to more consistent and equitable outcomes in the affective domain. These findings emphasise the critical role of CQI in refining assessment tools and instructional strategies. By aligning the rubric with clear and specific criteria, the CQI process ensures that assessments remain effective in addressing performance gaps and fostering equitable learning experiences.

It was observed by Chakraborty et al. [8] that a good-quality rubric clearly defining descriptors, criteria, scoring levels, and scoring strategy with explicit judgement of complexity plays a significant role in reducing variation among assessors. In higher education, variation among assessors can lead to discrepancies in student assessments that potentially disadvantage students. Ford et al. [9] found that the rubric enhanced communication skills development and improved engagement with students. Students tended to ask more questions and seek feedback on their communication skills after the faculty redesigned the rubric, and they also appreciated the additional input from the rubric. The process of CQI in rubric refinement is leading to a more effective educational tool in the laboratory courses in the engineering curriculum.

CONCLUSIONS

These findings imply that revising the rubric is essential to the course’s CQI. The change in the rubric demonstrates a positive impact on student performance, particularly in raising the minimum levels and closing the performance gap. Meanwhile, a well-structured rubric helps students and assessors comprehend the assessment objectives and is essential for promoting consistent and equitable student outcomes in the affective domain. Furthermore, a rubric with more specific criteria for assessing individual accountability would lead to more consistent and accurate assessments. The challenges posed by the COVID-19 outbreak highlight the need for adaptable educational strategies. This suggests initiatives like updated assessment rubrics, more engaging instructional methods, and enhanced student support services that can significantly improve student performance. However, it is essential to note that the study’s findings are based on data from a single institution. Future research should expand its scope to include different engineering courses or multiple institutions to ensure the rubric is robust and effective across diverse educational and cultural contexts.

ACKNOWLEDGEMENTS

The authors gratefully acknowledge the technical support of this work from the Universiti Teknologi MARA (UiTM) Cawangan Pulau Pinang.

REFERENCES

1. M. Stephens and P. Ormandy, “An Evidence-based Approach to Measuring Affective Domain Development,” J. Prof. Nurs., vol. 35, no. 3, pp. 216–223, May 2019, doi: 10.1016/j.profnurs.2018.12.004.
2. J. Chaieb, B. B. Phillips, M. Thiman, M. Fulford, H. N. Young, and M. Perri, “Evaluation of affective domain elements in a new pharmacy practice course,” Curr. Pharm. Teach. Learn., vol. 10, no. 9, pp. 1219–1227, Sep. 2018, doi: 10.1016/j.cptl.2018.06.011.
3. IEA, “INTERNATIONAL ENGINEERING ALLIANCE GRADUATE ATTRIBUTES & PROFESSIONAL COMPETENCIES PROUDLY SUPPORTED BY: PREAMBLE,” 2021. [Online]. Available: http://www.ieagreements.org
4. EAC BEM, “ENGINEERING PROGRAMME ACCREDITATION STANDARD 2020,” 2020.
5. P. Dickinson and J. Adams, “Values in evaluation – The use of rubrics,” Eval. Program Plann., vol. 65, no. July, pp. 113–116, 2017, doi: 10.1016/j.evalprogplan.2017.07.005.
6. E. Panadero and A. Jonsson, “A critical review of the arguments against the use of rubrics,” Educ. Res. Rev., vol. 30, no. March, 2020, doi: 10.1016/j.edurev.2020.100329.
7. E. Gregori-Giralt and J. L. Menéndez-Varela, “The content aspect of validity in a rubric-based assessment system for course syllabuses,” Stud. Educ. Eval., vol. 68, no. November 2020, 2021, doi: 10.1016/j.stueduc.2020.100971.
8. S. Chakraborty, C. Dann, A. Mandal, B. Dann, M. Paul, and A. Hafeez-Baig, “Effects of rubric quality on marker variation in higher education,” Stud. Educ. Eval., vol. 70, no. March, 2021, doi: 10.1016/j.stueduc.2021.100997.
9. C. R. Ford, K. Garza, J. Kavookjian, and E. L. Kleppinger, “Assessing student pharmacist communication skills: Development and implementation of a communication rubric,” Curr. Pharm. Teach. Learn., vol. 11, no. 11, pp. 1123–1131, 2019, doi: 10.1016/j.cptl.2019.07.018.