Evaluating the Effectiveness of Technology-Enhanced Flexible Learning Training Program Implemented in School using ABCD Model: A Goal-Free Evaluation

Evaluating the Effectiveness of Technology-Enhanced Flexible Learning Training Program Implemented in School using ABCD Model: A Goal-Free Evaluation

Alma T. Gurrea
Cebu Normal University

Received: 11 March 2023;  Accepted: 21 March 2023; Published: 22 April 2023

ABSTRACT

This study evaluates the effectiveness of Technology Enhanced Flexible Learning (TEFL) training at a school level using the ABCD evaluation model. The study investigates the model’s four components, namely faculty-trainees, programs and operations, effects, and social impact, to identify the gaps between the intended and actual outcomes of the training. Faculty members were interviewed to gain insight into the processes, issues, and challenges experienced while implementing TEFL. The study uses purposive sampling to analyze the personal effects and impacts of the TEFL training/program on the faculty trainees. The results show that while TEFL can improve teaching and learning efficiency, its performance at the school level could have been enhanced by faculty attitudes toward the training, technical assistance provided, and program monitoring. Based on the findings, recommendations are made to improve TEFL implementation at the school level. This research contributes to a better understanding the challenges and opportunities of implementing TEFL in schools.

Keywords: technology enhanced flexible learning, ABCD evaluation model, technological and teaching efficiency, school level.

INTRODUCTION

As the field of education undergoes continuous change, it’s crucial for educators to keep abreast of the latest research-based approaches, thus, there is a pressing need to adapt to the new normal while ensuring the safety and well-being of learners, teachers, and school staff. The Commission on Higher Education in the Philippines has initiated the Continuity Plan to provide a framework for education delivery during the pandemic (Magsambol, 2020). The plan encourages the use of technology to reach out to students, especially those who do not have internet access and to address the learning needs of marginalized students (Aithal, 2015).

Enhancing the effectiveness and delivery of technology-enhanced flexible learning programs requires careful evaluation. One commonly used evaluation model is the ABCD model developed by Dr. Ochave, a goal-free approach that can be applied to assess various programs (Mangarun, 2014; Abella&Cutamora, 2019). However, there is a need for more literature to investigate the use of the ABCD model specifically for evaluating technology-enhanced programs during the pandemic (Ferdig, 2020). This would help to determine the effectiveness of these programs and identify areas for improvement, which is particularly important in the current context where many institutions have had to transition rapidly to remote learning.

While several studies have used the ABCD model to evaluate different programs, limited literature is available on its use in assessing technology-enhanced programs during the pandemic. This study aims to fill this gap by using the ABCD assessment paradigm to identify congruences and discrepancies between the expected and actual program implementation results. Therefore, this study aims to evaluate technology-enhanced flexible learning programs using the ABCD model in the context of the pandemic. By identifying congruences and discrepancies between expected and actual results, this study will provide insights into the effectiveness of these programs and inform improvements in their delivery.

This research is significant because it will contribute to the literature on the use of technology in education and the development of effective technology-enhanced flexible learning programs. It will also be of value to educators seeking to integrate technology into their teaching methods and improve their teaching practices.

Statement of the Problem

This study aims to evaluate the TEFL training/program implemented in school in 2022 using the ABCD evaluation model, specifically by:

A = identifying the intended beneficiaries of TEFL training/program in the school,

B = exploring the operations and processes of the training/program, including the benefactors, the nature of support, the resources used, and the training’s

monitoring,

C = determining the effects of the training/program in the school, based on the goal of enhancing teachers’ use of technology, specifically digital apps,

D = recognizing  the impact of the training/program.

2) Pointing out the gaps and the main issues in its implementation.

3) Proposing recommendations to enhance the implementation of the program in the school.

THEORETICAL FRAMEWORK/ MODEL

The ABCD Model of Ochave (2010) provides a comprehensive framework for evaluating the effectiveness of educational programs. This model involves four components: A) Beneficiaries or participants, B) Operations and processes, C) Results, and D) Impact. Component A refers to the intended participants of the program, in this case, the faculty members. Component B includes the processes and operations of the training/program, such as the resources used, the nature of support provided, and the monitoring of the program’s progress. Component C refers to the results of the training/program, specifically in terms of enhancing the faculty’s use of technology, mainly digital applications. Finally, Component D examines the impact of the training/program, such as improvements in teaching effectiveness and student learning outcomes.

The study also discusses three theories related to the ABCD model – the Diffusion of Innovations Theory, Goal Setting Theory, and Self-Determination Theory. These theories help to understand the factors that may influence the faculty members’ adoption and use of technology-enhanced flexible learning, guide the design and implementation of the TEFL training/program, and provide support and resources that promote the faculty members’ sense of autonomy, competence, and relatedness.

The ABCD model provides a practical, theoretical framework for evaluating educational programs, particularly in the context of technology-enhanced flexible learning. By using this model, educators and researchers can identify strengths and weaknesses in programs, make informed decisions about program improvement, and ultimately enhance the quality of education.

Figure 1

METHODOLOGY

Design and Locale

This study employed the descriptive-evaluative design. Matching the actual happenings of the program implementation, referred to as “actualities,” with the objectives of the Technology-enhance Flexible Learning Training/Program, referred to as “intents,” makes this study evaluative.

The investigation’s results were used to determine its implication to delivering training/programs in respective classes and the possible improvements for quality teaching-learning using technology-enhanced. The study was conducted in schools with seven academic programs.

Research Participants and Sampling 

This study used purposive sampling to analyze the personal effects and impacts of the TEFL training/program on the faculty trainees. Data saturation was employed to determine the number of trainees required for the study, as described by Merriam and Tisdell (2016). In total, forty eight full-time and part-time faculty members from different programs voluntarily participated in the study.

Research Instrument 

A series of tests were conducted to ensure the validity and reliability of the data collection instruments used in this study.

First, the Self-Assessment Questionnaire used to obtain data on Component A of the ABCD model was subjected to validity and reliability tests. The validity test ensured that the questionnaire measured what it intended to measure, which was the number of faculty participants in the training program that needed the training. This was done by comparing the questionnaire results with an external criterion, such as records kept by the trainers or facilitators of the training program. The reliability test ensured that the questionnaire produced consistent results over time, which was done by administering the questionnaire to a group of participants twice, with a time interval in between, and comparing the results.

For Components B and D, the semi-structured interview technique used in the key informant interviews was subjected to validity tests. The validity test ensured that the interview questions were measuring what they intended to measure: the operations and impact of the TEFL training/program. This was done by having the interview questions reviewed by experts in the field to ensure that they were relevant and appropriate. The interview questions were also pilot-tested on a small group of participants to ensure they produced the desired information.

In addition to the validity tests, the interview instruments were also subjected to reliability tests. This was done by administering the same interview questions to a group of participants twice, with a time interval in between, and comparing the results. This ensured that the interview questions produced consistent results over time.

The validity and reliability tests conducted on the data collection instruments used in this study helped to ensure that the data collected was accurate, consistent, and relevant to the research questions. This strengthens the credibility and trustworthiness of the study’s findings and contributes to the overall quality of the research.

Data Gathering Procedure

Participants were interviewed, and focus group discussions were conducted by video call conferencing and face-to-face. It was recorded using an audio recording device with the participants’ permission. Following the interviews, the captured data was transcribed into text format to uncover the underlying difficulties and obstacles associated with the implementation of the training/program.

Data Analysis 

ABCD evaluation model was used where the congruences and discrepancies between the intents and actualities were identified. The more significant the discrepancy or the gap, the less favorable the evaluation (Mehrafsha, 2011). These gaps are the indicators of whether the program has been effective. Summary Matrix of Information was utilized in presenting the data, which is composed of columns that include the intents, actualities, and congruences or discrepancies of a particular component. Analysis was done based on the congruences and discrepancies of some components.

Ethical Considerations 

The ethical considerations of this research study were taken seriously, and the researcher followed the ethical conduct principles outlined by Lichtman (2014). One of the primary ethical considerations in this study was to ensure that no harm came to the participants. To address this, the researcher implemented several safeguards, such as assigning pseudonyms to protect the confidentiality of the participants. This measure helped to ensure that no harm was done to the participants and that their identities were not compromised.

Confidentiality was another significant ethical consideration addressed in this study. The researcher assigned pseudonyms to protect the identity of the participants, and each participant was assigned a random number. This measure ensured that the confidentiality of the participants was maintained and their identities were not revealed to anyone. Moreover, all study-related materials, such as transcripts, audio recordings, and participants’ data, were kept confidential.

Finally, the researcher ensured that the principle of informed consent was addressed in this study. Participants were informed that their participation was voluntary and that they could discontinue participation at any time. This measure helped to ensure that participants were fully aware of their rights and could make an informed decision about their participation in the study.

RESULTS AND DISCUSSION

The data from the interviews, surveys and secondary data were collated and tabulated into the four components (faculty, TEFL training/program, effect, and social impact) and analyzed for their intents, actualities, and gaps or discrepancies/ congruences. The following data are presented on the next page.

Component A – Faculty

CHED Order no. 04, series of 2020, entitled “Guidelines on the Implementation of Flexible Learning,” HEIs are encouraged to maximize the use of technology to support learning and teaching. With this, schools’ Academic Affairs initiated training on technology to enhance flexible learning for faculty, an approach to address faculty challenges to help them overcome the technological dilemma of delivering quality education. Implementation of helpful technology apps in teaching may still be improved and enhanced to optimize their use in education.

These are the following responses of the faculty- participants, the problems they encountered in the use of technology applications in their teaching and students’ learning;

“…I find difficulty in navigating the various features of different digital applications…” (F1)

“…I have a problem on internet connection and the process of using those apps…” (F2)

“…I have a problem such as setting interface and the processor of the laptop…” (F3)

“…Because of my age, I cannot fully grasp what is being discussed especially, on the use of technology and different digital applications…” (F4)

“… Lack of self-motivation because I’m not so computer literate and proficient. I have the feeling of I’m left behind in terms of technology…” (F5)

“As a digital migrant, I still have many things to learn. I need more hands-on training.

I got a problem with its usage since I only have limited time to browse or explore the fullest of its applications….” (F8)

Component B – TEFL Training/Program Implementation and Operation

These are the following responses of the faculty- participants,  the most useful or valuable new learnings they gathered;

“…Free Websites, it elevates my knowledge from the workshops that were discussed…”(F2)

“…Integration of Digital applications in teaching different subjects…”(F4)

“….The content aspect- all the digital apps were useful. They are stimulating and fun, those webinar workshops provide instructors an understanding of online spaces where it gives solutions to the challenges we are facing…”(F9)

“…. Those presentations, particularly the utilization of the digital applications were very helpful. Though most of them were familiar to me, I appreciate how they were applied in class…”(F14)

“…The new tools I learned are interesting, it provides additional opportunities for learners to see and interact…”(F8, F13)

Component C – Effects

These are the following responses of the participants on the pedagogical skills developed and the impact of the use of digital applications in teaching:

“…I don’t easily understand how to apply the digital apps in the lesson, so what I did  asked for help from the trainer, sometimes from the IT faculty and close friend /colleague, …”(F9, F26)

“…The introduction of e-Learning tools helped open new avenues of online teaching, learning more about how to be flexible with the use of technology for flexible learning…’(F7, F21)

“…The use of technological tools, techno-culture sensitivity, facilitating learning via online learning modality is the pedagogical skills I have developed…” (F9, F13, F31)

“…It increased my productivity and efficiency…I can now leverage technology to implement useful digital tools to expand student learning opportunities and increase student support and engagement. It enables me to improve my instruction methods and personalize learning.…” (F18, F27, F32)

“…In the online class I have come to enhance my skills in the use of computer and the appropriate platforms/apps that go with it…”(, F19)

“…Trying to be creative in utilizing the technology-related knowledge given to us, making an interactive engagement with the students. Gradually making ourselves skillful in using the software programs usable in managing our class.…”(F15, F22)

“… The use of Flipgrid and Canva e-Tools gave me the chances to give a personal touch to my students…”(F18, F30)

“…Embracing technology in teaching as part of the new normal, there is that desire to learn for more methodologies to use in the classroom that are not conventional. I will try to develop a more engaged class discussion and participation with the use of different technology-based learning tools….”( F13, F25,F29)

Component D – Impact

CONCLUSION AND RECOMMENDATIONS

In conclusion, the ABCD model evaluation showed that technology-enhanced flexible learning training/programs are effective at the school level. The four components evaluated, faculty, TEFL training/program, effects, and social impact, align with actual outcomes. However, some challenges were encountered during the program’s implementation at a Community College. Faculty reported increased stress and anxiety, and the support did not fully meet their needs. Additionally, while most faculty trainees completed the training, some still require different skills and knowledge development.

To enhance the implementation of the TEFL training/program in schools, the following recommendations are made; (1) Introduce the training/program to faculty-trainees in advance and provide them with necessary materials such as fast internet connectivity and well-functioning laptops. This will ensure that they are adequately prepared for the training and coordination with stakeholders can occur. (2) Adjust faculty trainees’ schedules to allow maximum study and learning time and provide technical assistance. (3) Allocate funds for faculty connectivity and gadgets with high specifications not only for the training but also for continued use. (4) Involve the Information Technology department, educational technology coordinator,  and enthusiasts,  in closely monitoring the training/program. (5) Evaluate the training/program to identify promising practices or pressing issues related to the TEFL training/program implementation.

By implementing these recommendations, schools can successfully integrate technology-enhanced flexible learning training/programs and achieve their program goals.

REFERENCES

1. Abella, V. F., & Cutamora, J. R. (2019). Goal-free evaluation of community-based drug rehabilitation program using the ABCD model. Philippine Journal of Psychology, 52(1), 1-16.
2. Aithal, P. S. (2015). Enhancing quality of higher education in India using ABCD analysis. International Journal of Scientific and Research Publications, 5(1), 1-6.
3. CHED memo No. 04, Series of 2020 Guidelines on the Implementation of Flexible Learning.
4. Collis, B., De Boer, W., & Slotman, K. (2001). Feedback for web-based assignments. Journal of Computer Assisted Learning, 17(3), 306-313.
5. Coronavirus disease (COVID-19). (2020, October 12). World Health Organization. https://www.who.int/ emergencies/diseases/novel-coronavirus-2019/question-and-answers-hub/q-a-detail/coronavirus-disease-  covid-19
6. Deci, E. L., & Ryan, R. M. (2008). Self-determination theory: A macro theory of human motivation, development, and health. Canadian Psychology/Psychologie Canadienne, 49(3), 182–185. https://doi.org/10.1037/a0012801
7. Ferdig, R. E. (2020). Perspectives on technology-enhanced learning in higher education. International Journal of Educational Technology in Higher Education, 17(1), 1-5
8. Rogers, E. M. (2003). Diffusion of innovations (5th ed.). Free Press.
9. Harrison, Mark, et al. (2019). A Cost-Benefit Analysis of Teaching and Learning Technology in a Faculty of Pharmaceutical Sciences. American Journal of Pharmaceutical Education, 83(6), 1310.
10. Jung, I., Choi, S., Lim, C., & Leem, J. (2002). Effects of different types of interaction on learning achievement, satisfaction and participation in web-based instruction. Innovations in Education and Teaching International, 39(2), 153-162.
11. Kemp, N. (2018). The ABCD model of performance feedback: A conceptual framework for improving feedback effectiveness. Journal of Personnel Psychology, 17(3), 107-117. I hope this helps!
12. King, J. E. (2016). The ABCD model of counselor supervision. Journal of Counseling and Development, 94(2), 139-147.
13. Lichtman, M. (2014). Qualitative Research for the Social Sciences. SAGE Publications, Inc.
14. Magsambol, B. (2020). As school year starts, CHEd lists errors in distance learning modules. Rappler. Retrieved from https://www.rappler.com/nation/errors-distance-learning-modules-ched-lists-september-2020
15.  Locke, E. A., & Latham, G. P. (2002). Building a practical theory of goal setting and task motivation: A 35-year odyssey. American Psychologist, 57(9), 705–717. https://doi.org/10.1037/0003-066X.57.9.705
16. Mangarun, A. J. (2014). Evaluation of the MSU-IIT college of nursing enhancement program using the ABCD model as a framework. CNU Journal of Higher Education, 8, 90-103.
17. Mehrafsha, S. (2011). A proposed blueprint model towards the evaluation of educational system in Iran, College of Education, Arsanjan Azad University, Fars Province.
18. Merriam, S. B., & Tisdell, E. J. (2016). Qualitative Research: A Guide to Design and Implementation (4th ed.). San Francisco, CA: Jossey Bass.
19. Narayan-Parker, D. (1998). Evaluation Research (Social action programs).
20. Nava, L. H., Ochave, J. A., Romero, R. C., Ruscoe, R. B., & Mabunga, R. A. (2007). Evaluation of the UNESCO-Associated Schools Project Network (ASPNet) in Teacher Education Institutions in the Philippines. The Normal Lights, 1(1).
21. Ochave, J. A. (2003). “Towards the Development of a Blue-print Stakeholder-friendly Evaluation Model for Training and Education Programmers”. Journal of Southeast Asian Education, 4(2), 131-143.
22. Ochave, J. A. (2010). An ABCD approach to formulating educational goals and objectives. Journal of Instructional Pedagogies, 4, 1-17.
23. Rietbergen-McCracken, Deepa Narayan. (1998). Participation and Social Assessment: Tools and Techniques.
24. Wang, Y., Zhang, J., & Zhao, S. (2018). An evaluation model for the effectiveness of college English education based on the ABCD model. Journal of Language Teaching and Research, 9(6), 1246-1252.