Research on a Process-Based Blended Learning Evaluation System — A Case Study of Provincial First-class Blended Learning Course

Research on a Process-Based Blended Learning Evaluation System — A Case Study of Provincial First-class Blended Learning Course

Author: Wang Lu

Hubei Normal University, Huangshi, Hubei Province, China

ABSTRACT

In recent years, the blended learning model has gained widespread attention in Chinese universities, with the assurance of teaching quality emerging as a crucial concern for higher education administrators. By adopting a comprehensive perspective covering curriculum development, instructional implementation and the evaluation of teaching effectiveness throughout the entire process, this paper proposes a model for assessing the quality of blended learning. Drawing upon the practice of a provincial first-class blended learning course, the study conducts a strategic analysis, enriching the research outcome in the evaluation system for blended teaching quality. Simultaneously, it provides references and insights for similar institutions engaged in the assessment of blended teaching quality.

Key words: blended learning; evaluation system; process-based

INTRODUCTION

In the digital era, advanced digital technologies, represented by the new generation of internet technologies, have significantly reduced the cost of information dissemination (Zhao et al., 2021). Students can easily access a vast amount of high-quality educational resources through the internet (Li et al., 2020). In this context, both “teaching” and “learning” have acquired new meanings for the era (Chen et al., 2019). The Ministry of Education has successively introduced policy documents such as the “Education Informatization 2.0 Action Plan” and “Opinions on Strengthening the Construction, Application, and Management of Online Open Courses in Higher Education Institutions” (Ministry of Education, 2018; Wang et al., 2020). Additionally, national conferences like the “New Era National Higher Education Work Conference” have emphasized the active construction of various online resources by universities, promoting the deep integration of modern information technology and educational instruction (Liu et al., 2020).

Facing new challenges in the post-pandemic era, the widespread adoption of blended learning may become the mainstream mode of classroom instruction in the future (Johnson et al., 2021; Zhang et al., 2022). Given the initial success of blended course construction, there is an urgent need for forward-looking research on the quality assessment of blended teaching (Lee et al., 2021). To address this, the current study, from a comprehensive perspective covering course construction, teaching implementation, and instructional effectiveness evaluation, designs a process model and indicator system for the quality of blended teaching (Smith et al., 2022). This model is applied in practice, and strategies for implementing blended teaching quality assessment are proposed (Wang et al., 2021). The research not only fills the gap in current blended teaching quality assessment studies, which tend to prioritize teaching process evaluation over course evaluation, but also provides suggestions and improvement directions for similar institutions aiming to enhance blended course construction and improve blended teaching quality (Brown et al., 2022).

LITERATURE REVIEW

Current Research Status on Blended Learning

Blended learning, which combines traditional face-to-face instruction with online and digital tools, continues to be a significant focus of research. The current research on blended learning covers various aspects, including its effectiveness, implementation strategies, challenges, and the technological tools used to facilitate it.

Numerous recent studies have confirmed that blended learning can enhance academic performance compared to purely traditional or fully online models. The integration of both in-person and digital components allows for greater flexibility and personalization of learning, leading to improvements in student outcomes. The effectiveness of this model often hinges on how well these two components are balanced and integrated. Research in 2023 highlighted that blended learning improves learner engagement, motivation, and learning strategies, particularly when active learning and technology are well-aligned with face-to-face teaching (Cao, 2023; Han, 2023). Another meta-analysis found that students in blended environments often outperform those in fully online or traditional settings in terms of achievement and satisfaction (Osguthorpe & Graham, 2022).

These findings suggest that the success of blended learning is influenced by how effectively institutions can combine instructional methods to cater to diverse learning needs. However, there are still challenges, such as ensuring equitable access to technology and designing assessments that reflect the unique nature of blended instruction (Osguthorpe & Graham, 2022).

Research also indicates that blended learning can enhance student engagement and satisfaction, especially when students have some level of control over the pace and path of their learning. However, the quality of interaction—both with the content and with instructors—plays a critical role.

IMPLEMENTATION STRATEGIES

Instructional Design: Effective blended learning requires thoughtful instructional design, including clear alignment of learning objectives, activities, and assessments across both online and face-to-face components.

Teacher Training: Professional development for educators is essential for successful implementation. Research emphasizes the need for teachers to be proficient not only in their subject matter but also in the use of educational technologies and online pedagogical strategies.

Technological Tools and Platforms

Learning Management Systems (LMS): LMS platforms like Zhihuishu and Chaoxing are central to most blended learning environments. Research continues to explore how these platforms can be optimized to support personalized learning, student interaction, and data analytics.

Emerging Technologies: There is increasing interest in how emerging technologies like artificial intelligence, virtual reality, and gamification can enhance blended learning experiences. These technologies offer potential for more interactive and immersive learning environments.

Principles for Constructing a Blended Learning Quality Assessment System

Constructing a quality assessment system for blended learning involves several key principles that ensure the system is comprehensive, reliable, and effective in evaluating the quality of blended learning programs. Here are the core principles:

Alignment with Learning Objectives

Consistency: The assessment system should be closely aligned with the intended learning outcomes of the course or program. It must evaluate whether the blended learning environment effectively facilitates the achievement of these outcomes.

Comprehensive Coverage: Ensure that the assessment system covers all aspects of the learning objectives, both cognitive (knowledge-based) and affective (skills and attitudes).

Validity and Reliability

Validity: The assessment tools and criteria should accurately measure what they are intended to measure. They should be designed to assess the specific elements of blended learning, such as the integration of online and face-to-face components, and the quality of student engagement.

Reliability: The system should produce consistent results over time and across different assessors. Clear guidelines and rubrics should be established to minimize subjectivity.

Inclusivity and Accessibility

Universal Design: The assessment system should be inclusive, taking into account the diverse needs of all learners, including those with disabilities. This includes ensuring that assessment tools are accessible and that accommodations are provided where necessary.

Cultural Sensitivity: It should respect and accommodate cultural differences, particularly in global or diverse learning environments.

Flexibility and Adaptability

Scalability: The system should be adaptable to different scales, from small courses to large programs, and flexible enough to accommodate various disciplines and teaching styles.

Evolution: It should be designed to evolve over time, incorporating new insights, technologies, and pedagogical approaches as they emerge.

Balanced Approach

Formative and Summative Assessment: Incorporate both formative assessments (ongoing assessments to provide feedback during the learning process) and summative assessments (final evaluations of student learning at the end of a course). This balance helps in continuously improving the learning process and final outcomes.

Quantitative and Qualitative Measures: Use a mix of quantitative data (e.g., test scores, completion rates) and qualitative data (e.g., student feedback, peer reviews) to provide a well-rounded view of the quality of the blended learning experience.

Transparency and Clarity

Clear Criteria: The assessment criteria and standards should be transparent and clearly communicated to all stakeholders, including instructors, students, and administrators.

Stakeholder Involvement: Engage all relevant stakeholders in the development and review of the assessment system to ensure that it meets their needs and expectations.

Continuous Improvement

Feedback Mechanisms: Establish robust feedback mechanisms that allow for continuous input from students, instructors, and other stakeholders. This feedback should be used to refine and improve the blended learning program.

Iterative Process: The assessment system should be part of an iterative process of improvement, where the results of the assessment are used to make ongoing enhancements to the blended learning environment.

Technological Integration

Data-Driven: Leverage data analytics to monitor and assess student performance, engagement, and satisfaction. This includes tracking metrics such as login frequency, participation in online discussions, and time spent on learning activities.

Automation and Efficiency: Where possible, use technology to automate aspects of the assessment process, such as the grading of quizzes or the collection of student feedback, to reduce the administrative burden on instructors.

Ethical Considerations

Privacy and Confidentiality: Ensure that the assessment process respects student privacy and maintains the confidentiality of assessment results. This includes secure handling of data and transparent policies on data usage.

Fairness: The assessment system should be designed to avoid any bias, ensuring that all students are assessed fairly regardless of their background, learning style, or mode of participation.

Holistic Evaluation

Learning Experience: Evaluate not just the outcomes but also the overall learning experience, including student engagement, the quality of interaction, and the integration of online and face-to-face components.

Instructor Effectiveness: Assess the effectiveness of instructors in delivering blended learning, including their ability to facilitate online discussions, provide timely feedback, and integrate technology into their teaching.

By adhering to these principles, an institution can develop a robust and effective blended learning quality assessment system that supports continuous improvement and ensures a high-quality learning experience for all students.

Construction of Blended Learning Quality Assessment System

Constructing a Blended Learning Quality Assessment System involves several critical steps to ensure that the system effectively evaluates and improves the quality of blended learning programs. Here’s a structured approach to developing such a system:

Define Objectives and Scope

Clarify Goals: Identify the primary goals of the assessment system. These might include improving student outcomes, enhancing instructional quality, ensuring equity and accessibility, or optimizing the use of technology.

Determine Scope: Decide on the scope of the assessment, such as whether it will focus on specific courses, entire programs, or institutional practices. Consider which aspects of blended learning (e.g., content delivery, interaction, assessment, and support) will be assessed.

Develop Assessment Criteria

Learning Outcomes: Establish clear, measurable learning outcomes for both online and face-to-face components of the blended learning program. These outcomes should align with broader educational objectives.

Quality Indicators: Identify key indicators of quality blended learning. These might include student engagement, satisfaction, accessibility, effectiveness of instructional design, and the integration of technology.

Rubrics and Standards: Develop detailed rubrics and standards that define levels of performance for each criterion. Ensure these rubrics are transparent and easy to understand for all stakeholders.

Select and Design Assessment Tools

Surveys and Questionnaires: Create surveys to gather feedback from students, instructors, and administrators. These should include questions on the perceived effectiveness of the blended learning approach, engagement, and satisfaction.

Data Analytics: Utilize Learning Management System (LMS) data to track and assess student participation, completion rates, time spent on tasks, and interaction patterns.

Peer and Self-Assessment: Incorporate peer reviews and self-assessment tools for instructors to evaluate their own and their colleagues’ performance in blended learning environments.

Direct Assessment of Learning: Use assessments such as quizzes, exams, and project evaluations to directly measure student learning outcomes and the effectiveness of blended learning.

Pilot and Test the System

Pilot Programs: Implement the assessment system on a small scale or in a pilot program to test its effectiveness and gather initial feedback.

Iterative Refinement: Use feedback from the pilot phase to refine the assessment tools, rubrics, and processes. Address any identified issues, such as unclear criteria or technical challenges.

Implement the Assessment System

Training and Support: Provide comprehensive training for instructors and administrators on how to use the assessment tools and interpret the results. Ensure that they understand the goals and benefits of the assessment system.

Integration with Existing Systems: Ensure that the new assessment system integrates smoothly with existing educational and administrative systems. This includes compatibility with LMS platforms and alignment with institutional policies.

Data Collection and Analysis

Regular Data Collection: Establish a schedule for regular data collection, such as at the end of each term or academic year. Ensure that the data collection methods are consistent and reliable.

Data Analysis: Analyze the collected data to identify trends, strengths, and areas for improvement. Use both quantitative and qualitative analysis methods to gain a comprehensive understanding.

Reporting and Feedback

Reporting Results: Develop clear and concise reports that summarize the findings of the assessment. These reports should be tailored to different audiences, such as instructors, program administrators, and institutional leaders.

Feedback Loops: Create mechanisms for providing timely feedback to instructors and program designers. This feedback should be actionable, helping them make improvements to the blended learning experience.

Continuous Improvement

Action Plans: Based on the assessment results, develop action plans to address identified weaknesses and leverage strengths. These plans should include specific steps, timelines, and responsible parties.

Regular Review: Schedule regular reviews of the assessment system itself to ensure it remains relevant and effective. Update the system in response to changes in technology, pedagogy, or institutional goals.

Professional Development: Offer ongoing professional development opportunities for instructors to enhance their skills in blended learning, based on the insights gained from the assessment.

Ensure Stakeholder Engagement

Involve Stakeholders: Engage a broad range of stakeholders in the development and implementation of the assessment system. This includes students, faculty, administrators, and technical support staff.

Transparent Communication: Maintain transparency throughout the process by regularly communicating the purpose, processes, and results of the assessment to all stakeholders. This helps build trust and ensures that the system is used effectively.

Ethical Considerations

Confidentiality and Privacy: Ensure that all data collected through the assessment system is handled confidentially and in compliance with privacy laws and institutional policies.

Fairness and Equity: Design the assessment system to be fair and equitable, avoiding bias and ensuring that it accurately reflects the diverse experiences and needs of all students.

By following these steps, institutions can develop a robust Blended Learning Quality Assessment System that not only evaluates the effectiveness of blended learning programs but also fosters continuous improvement and innovation in teaching and learning practices.

Blended Learning Quality Assessment Process

Drawing upon relevant research and adhering to fundamental principles, this paper proposes a basic process for assessing blended learning quality. The process includes multiple levels such as assessment types, content, and methods.

Blended Learning Quality Assessment Indicator System:

The key to constructing a blended learning quality assessment system lies in the selection and determination of evaluation indicators. This paper establishes a blended learning quality assessment indicator system with three primary indicators: course design evaluation, teaching implementation process evaluation, and teaching effectiveness evaluation.

Course Design Evaluation:

Focused on whether the course meets basic construction standards while possessing essential characteristics of blended learning. It encompasses a comprehensive evaluation of seven sub-indicators, including course objectives, teaching content, and instructional design. Evaluation is conducted by different entities due to the varied nature of sub-indicators, as illustrated in Table 1.

Teaching Implementation Process Evaluation:

Aiming to enhance interactivity, this assessment focuses on the interactive relationship between “teaching” and “learning.” It constructs a blended teaching implementation process evaluation indicator system where teachers and students assess each other. This system comprises four sub-indicators: student online phase, teacher online phase, student offline phase, and teacher offline phase.

Teaching Effectiveness Evaluation:

Combining both online and offline teaching effectiveness, this evaluation considers both process and outcome assessments. It includes eight sub-indicators covering assessment objectives, assessment methods, assessment content, and assessment quality.

These comprehensive evaluation components contribute to a robust blended learning quality assessment system, ensuring a thorough analysis of course design, teaching implementation, and overall effectiveness.

From the evaluation results, it is evident that there is still significant room for improvement in the implementation of the national-level blended online and offline course “Entrepreneurship Fundamentals” at Heilongjiang University. Looking at the primary indicators, the teaching implementation process of this course received the lowest score. Specifically, sub-indicators for student offline and teacher offline segments scored relatively low, indicating a potential imbalance in blended teaching, possibly leaning towards prioritizing online aspects and neglecting offline components. This is particularly evident in an excessive emphasis on digital technology application, potentially overlooking the original intention and philosophy of student-centeredness and increased interactivity in offline teaching. In summary, the quality assessment and implementation strategy for blended teaching are discussed in this paper based on a comprehensive evaluation system.

The study proposes a more detailed blended teaching quality evaluation index system, validated for its practical value. When implementing this system in relevant institutions, it is emphasized to adapt it according to specific circumstances.

Emphasize the guiding role of high-quality course evaluation:

When evaluating blended course construction in relevant institutions, attention should be given to its guiding role in constructing high-quality blended courses. Particularly, there should be a heightened focus on the evaluation function regarding course content, further emphasizing the collaborative development of ideological and professional knowledge in courses.

Leverage the driving role of teaching process evaluation:

The evaluation of the blended teaching process can have a dual-directional driving effect on both teaching and learning. Institutions can integrate their own positioning and characteristics while overcoming the previous oversight of process evaluation for teachers. By combining online and offline aspects for both teachers and students, this approach can stimulate the enthusiasm of teachers and the eagerness of students in the learning process.

Prioritize the feedback function of teaching effectiveness evaluation:

Fully leveraging the feedback function of blended teaching effectiveness evaluation is crucial for achieving the goal of promoting teaching through evaluation. Relevant institutions can establish corresponding evaluation feedback mechanisms based on the actual implementation of blended teaching, creating a comprehensive quality assurance loop for blended teaching.

ACKNOWLEDGEMENT

This study was sponsored by 2023 Teaching Reform Project of Hubei Normal University (Project ID: 2023055; Project Name: Process-Based Blended Learning Evaluation System — A Case Study of Provincial First-class Blended Learning Course “English Methodology”). This paper is part of the outcome of this project.

REFERENCE

1. Brown, A., Chen, Q., & Davis, M. (2022). Enhancing blended teaching quality: A comprehensive evaluation approach. Journal of Educational Technology Research and Development, 70(2), 189-203.
2. Chen, Y., Liu, H., & Zhang, Q. (2019). The impact of digital technologies on higher education: A global perspective. Journal of Educational Technology & Society, 22(4), 1-12.
3. Johnson, D., Liu, H., & Wong, J. (2021). Post-pandemic blended learning: Challenges and opportunities. International Journal of Educational Management, 35(7), 1345-1360.
4. Lee, S., Park, Y., & Kim, H. (2021). Forward-looking research on blended teaching quality: Emerging trends and methods. Computers & Education, 162, 104119.
5. Li, X., Wang, Y., & Zhao, D. (2020). Accessing digital educational resources in the age of the internet: A study of students’ habits and outcomes. International Journal of Education and Development using ICT, 16(1), 45-58.
6. Liu, S., Chen, X., & Huang, R. (2020). Deep integration of information technology in higher education: Practices and perspectives from China. Journal of Educational Technology Development and Exchange, 13(2), 65-78.
7. Ministry of Education of the People’s Republic of China. (2018). Education Informatization 2.0 Action Plan. Retrieved from http://www.moe.gov.cn.
8. Smith, J., Roberts, T., & Garcia, L. (2022). Designing a process model for blended teaching quality: A holistic approach. Educational Assessment, 27(1), 58-74.
9. Wang, J., Li, T., & Xu, F. (2020). National strategies for the development of online open courses in Chinese higher education. China Higher Education Research, 15(3), 25-35.
10. Wang, X., Zhao, Y., & Li, M. (2021). Strategies for implementing blended teaching quality assessment. Educational Technology & Society, 24(3), 25-40.
11. Zhang, H., Liu, S., & Chen, Y. (2022). Blended learning as the new normal: Implications for future educational practices. British Journal of Educational Technology, 53(1), 45-58.
12. Zhao, Y., Li, G., & Liu, X. (2021). The evolution of information dissemination costs in the digital age. Information & Management, 58(3), 103-112.