Evaluation of Learning Management Systems for Success Factors

Evaluation of Learning Management Systems for Success Factors

Adeyemi A. L, Mutebi Joe, Hussein Muhaise

Kampala International University, Uganda

DOI: https://dx.doi.org/10.47772/IJRISS.2024.803058

Received: 05 February 2024; Revised: 26 February 2024; Accepted: 29 February 2024; Published: 05 April 2024

ABSTRACT

Evaluating an information system for success is key. Evaluating success factors of a Learning Management System (LMS) is essential in perspective of information systems success in a developing country context. eLearning is vital to educational system considering its benefits and impacts, particularly in accessing learning from remote areas, suitable for different learners’ categories, minimal resource utilization in terms of cost and time. Through the literature reviewed on eLearning and Information System, the desire to determine the variables that measure the success factors for information systems continues. Existing Information System (IS) success models do not sufficiently evaluate eLearning in developing country, Uganda inclusive as the current IS success models are generic in nature. This study aimed to describe a model of information system success tailored to eLearning system, Kampala International University as a case study, a Uganda’s context. To address the above objective, a field study was conducted, using questionnaire to determine factors for information systems success in Uganda, a sample size of 370 respondents were used from population size of 5500 using Sloven formula. The respondents comprise of 340 students, 27 lecturers and 4 administrators. The identified success factors are skills & training, infrastructure and management support. Data were cleaned and analyzed using Statistical Package for Social Scientists version 20.0 (SPSS). This study adopted Delone and Mclean’s Information System Success model (2016) and extended it using factors obtained from the field study survey. Analysis was performed to evaluate the model.

Results of the study showed that all the independent variables management support, infrastructure, skills and training are positively related to the dependent variable of intention to and use of information system. There exists a strong relationship between the multiple independent variables and the dependent variable. All factors identified, has positive impact in explaining the variation in intention to use and use of the system with r coefficient of 0.343, 0.406 and 0.406 respectively.  The results of the study presented a model of success factors for Learning Management System, eLearning specifically.

For future research, this study recommends conducting qualitative studies to delve deeper into the nuanced perceptions and experiences of learners and teachers as well as looking into policies that can promote eLearning especially in developing countries.

Keywords: eLearning, LMS, success factors, infrastructure, skill and training, management support

INTRODUCTION

Evaluating a new system or technology for success is an inevitable aspect of information system. Evaluation of a technology change is critical, this is not only to have the major stakeholder use it, but also to ensure that the stakeholders are fully immersed and benefited from the idea of change, (Sheri C, 2020).

This evaluation helps determining the effectiveness of eLearning, as well as the success of its implementation. The Learning Management systems in the developing world are challenged by inadequate resources, infrastructure issues, resistance to change and inadequate capacity to adapt to changes which may influence the success of the system (Hadulo et al, 2017). A system is best evaluated by assessing the various dimension to which the learning system is built on; that is, the information quality, system quality, service quality and also considering the system use, user satisfaction, and the overall impact of the system (DeLone, and McLean, 2016).

This study will enable the management of KIU-WC to be aware and conscious of area that needs improvement in the system and how best to achieve that. It will make the management to make decision regarding their LMS.

A system is best evaluated by assessing the various dimension to which the system is built on; that is, the information quality, system quality, service quality and also considering the system use, user satisfaction and the overall benefit of the system (DeLeone, and McLean (2003).

This study is in line with DeLeone and Maclean Information System success model, as the evaluation of the system is done based on the six identified dimensions for evaluating success of a new technology, KIU eLearning. D & M IS Success model 2016 is employed in the evaluation, as it’s modified form of 2003 model and been the best due to its net impacts which measures positive and negative outcomes; also the D&M ISS model has been the most cited by other writers in the empirical studies and validated it as the valid success measures of information systems (Lee and Chung, 2009; Wang and Liang; 2018; Lin, 2007; Cheok et al, 2015; Alshibly, 2015; Lin et al, 2015).

The acceptance of eLearning in Uganda was due to several factors including experience variables, outbreak of pandemic disease, computer anxiety, computer self-efficacy, technological complexity, perceived convenience, or reluctance to accept virtual e-teaching, stakeholders’ preference etc., that may lead to low levels of usage and net benefit (Kabakchieva, 2013). It is therefore imperative that these factors be determined, the evaluation done to ascertain the effectiveness, efficiency, relevancy, coherence and sustainability of the system and also identify measures to address the eminent issues that could hinder achieving the objectives of implementing eLearning and maintenance (Kabakchieva, 2013). This is to ensure productivity, profitability level, return on investment (ROI) and make recommendation appropriately.

REVIEW OF SOME PREVIOUS STUDIES IN THE FIELD

eLearning is regarded as one of the growing educational tools in the recent past that has attracted the attention of educators at various levels around the world, (Tagreed et al, 2016). eLearning has various name with which people referred to it. It is referred to as online learning, distributed learning, web-based learning etc. This system of learning involves assessing teaching from various colleges and universities around the world without having to visit due to distance, financial constraint and some other issues. This system of learning makes it easy even for people at remote and less developed area to access formal education at a very cheap price, less stress and at the comfort of their homes.

Information system success is a critical aspect of a new deployed technology. The measurement of information systems success or its effectiveness is a critical determinant to understanding the value and efficacy of IS Investment, management and productivity.  Different ISS model was reviewed to include the DeLeon and McLean ISS (2003, 2016), Gable ISS model (2008), Liang ISS model (2017). From the Literature reviewed, the models presented are good for the analysis of IS success. The strength and weaknesses in the existing models are here presented to include: – their basis to determine success on factors that are technical, issues of functionality, organizational issues, human factors, willingness to change and net impacts. It was a general observation that all the models reviewed measured technical functionalities of the systems and attitude of the users either to use the system or towards the system use.

The D&M ISS model has been the most cited by other writers in the empirical studies and validated it as the valid success measures of information systems (Lee and Chung, 2009; Wang and Liang; 2018; Lin, 2007; Cheok et al, 2015; Alshibly, 2015; Lin et al, 2015). This study therefore developed a model based on the D&M ISS model of 2016. DeLone and McLean, 1992, created awareness and structure of the “dependent variable” that is, “information systems success” – in IS research. They proposed a taxonomy and an interactive model as frameworks for conceptualizing and operationalizing IS success, that was named the D&M IS Success Model. After a decade, due to progress and change in academic enquiry, the model was reviewed to accommodate some other dimension for evaluating IS success. There was another modification in the model in 2016 with which, the six dimensions of success are explained as thus:

System Quality

This is the desirable features and characteristics of an information system. Which may include; ease of use, system flexibility and reliability, ease of learning, as well as having features of intuitiveness, a level of sophistication, and response times.

Information Quality

These are characteristics that one desires in the system outputs; i.e., management reports and Web pages. For example, relevance, understandability, accuracy, conciseness, completeness, currency, timeliness, and usability of the generated information.

Service Quality

The quality of the support that system users receive from the information systems organization and IT support personnel. For example, responsiveness, accuracy, reliability, technical competence, and empathy of the IT personnel staff.

Use

The degree and manner in which employees, customers, and respective stakeholders utilize the capabilities of an information system. For example, amount of use, frequency of use, nature of use, appropriateness of use, extent of use, and purpose of use.

User Satisfaction

This is the users’ level of satisfaction with reports, Web sites, and support services. For example, a couple of the most

Net Impacts

The extent to which information systems are contributing (or not contributing) to the success of individuals, groups, organizations, industries, and nations. For example: improved decision-making, improved productivity, increased sales, Cost and efforts reductions, improved profits and productivity, market efficiency, consumer welfare, creation of jobs, time maximization, and economic development.

The inquiry of what IS success model best addresses eLearning in the context of an institution in a developing country like Uganda remains an empirical question for investigation. This study therefore come up with a model that clearly points out eLearning success dimensions in learning institutions in developing country set up.

Figure 1: The Proposed Model of success factors for eLearning

Below is the description of the proposed features, constructs, the relationships and the explanations for their inclusion in the conceptual model above.  The features identified as relevant to eLearning success both to information system users and the information system in particular.

Management support

Management support entails approvals and continuous support during the information system development process and operational phase of the system. The challenge can be overcome if continued monitoring and evaluation of the system is conducted, decision making based on the results of the system which are much supported by management. It can improve user perception and attitude to the system and consequently increasing intention to use/system use.

Infrastructure

Availability and accessibility of adequate infrastructure is one of the key factors of success of LMS in the developing countries. These are the required facilities that support information systems activities to be operational. The resources tackled in particular are electricity/power supply, hardware and software components of a computer, and network (Internet). These will increase the level of use of the system.

Skills and Training

Skills referring to the degree to which users possess the required knowledge to perform the required services while training is learning process that involves acquisition of new knowledge, skills enhancement and changing attitudes and behaviour to enhance performance of the assigned tasks. Lack of expertise and knowledge of ICT is a big threat to IS Success in our context. This can be overcome by equipping information system users with knowledge and skills required for information systems usage. Educating /use of educated professionals or training the system users on the system dynamics will change attitude and influence the intention to use /use of an information system.

Intention to Use/ System Use

The behaviour of stakeholder (students and staffs) on plan to use and level of use of the system such as frequency of use, extent of use and purpose of use are considered.

METHODS AND MATERIAL

This study was guided by mixed methods or triangulation. The design described the relationship between the variables and identified the characteristics that are associated with the existing conditions. The approach helped the researcher to administer the research instruments, that is, questionnaire to the respondents and interview few of the stakeholders, the procedure upon which the variables were measured.

The study population is made up of the active stakeholders to include students, Lecturers, and Administrators of KIU Western campus. According to the information gathered via the university detail on Wikipedia, the total population is estimated at 5500 which comprises of 5000 students, 300 lecturers and 200 administrative staffs (from which 15 of them consider one of the stakeholders) of the university western campus. Using Sloven formula for determining sample size, a sample size of 373 was arrive at to include 342 students, 27 Lecturers And 4 Administrators. 370 respondents were selected from the population which comprises of all the stakeholders involved on the university eLearning platform, that is, the students, the lecturers and the administrators. Lecturers were selected using purposive sampling method, while simple random was used for selecting students, as it allows the researcher to make generalizations from the sample of the population. Questionnaire were administered to the respondents to get their opinion about the university Learning Management System (LMS), identified success factors of the system, their intention to use the system, its use and its impact on them in term of performance and service delivery. The data were cleaned of abnormalities and entered into Statistical Package for Social Sciences (SPSS) version 20. The data analysis aimed to explore the characteristics of the student and teacher populations and understand their usage and attitude towards the eLearning platform of the university.

Frequency, percentage and mean were used to determine the prevalence of each factor. A paired sample t test was used for comparison between groups of two. Chi-square was used to examine the reliability of the constructs and established the significance, while Pearson correlation coefficient confirmed the correlation. P-value was set to 0.05.

DISCUSSION

Infrastructures

The paired t-test results show a highly statistically significant difference (p < 0.001) between the mean scores of infrastructures (Hardware, Software and Network) and Intention to Use. The mean Infrastructure score (M = 12.48) is notably lower than the mean Intention to Use score (M = 19.44). This significant difference suggests that participants who perceive a weaker network infrastructure are more inclined to have strong intentions to use the eLearning system. This might indicate that participants are willing to overlook infrastructure challenges if the system’s benefits are compelling.

Furthermore, the positive correlation between “Infrastructure” and “Intention to use” (r = 0.343) indicates a moderate positive relationship between the two variables. This means that as the infrastructure quality of the eLearning platform increases, learners’ intention to use the platform also tends to increase.

The statistically significant correlation and the difference in means suggest that the quality of the eLearning platform’s infrastructure may influence learners’ intention to use it. However, the effect size of the difference is relatively small, indicating that other factors may also contribute to learners’ intention to use the platform.

Training

The paired t-test results indicate a statistically significant difference (p = 0.027) between the mean scores of Trainings and Intention to Use. The mean Training score (M = 17.37) is lower than the mean Intention to Use score (M = 19.44). This significant difference suggests that participants who have received more training tend to have slightly lower intentions to use the eLearning system. This finding may indicate that factors beyond training, such as system usability or relevance, influence participants’ intentions.

The results of the paired samples t-test indicate a significant difference between “Training” and “Intention to use” of the eLearning platform. Participants reported significantly higher scores for “Intention to use” (M = 33.84) compared to “Training” (M = 18.76). The large negative t-value suggests that the difference in means is substantial.

Furthermore, the positive correlation between “Training” and “Intention to use” (r = 0.406) indicates a moderate positive relationship between the two variables. This means that as the level of training increases, intention to use the eLearning platform also tends to increase.

The statistically significant correlation and the substantial difference in means suggest that training plays a role in influencing intention to use the eLearning platform. This finding may have implications for designing effective training programs that can lead to higher utilization and engagement among the stakeholders ultimately enhancing the overall learning experience on the eLearning platform.

Management Support

The results of the paired samples t-test indicate a significant difference between “Management Support” and “Intention to use” of the eLearning platform. Participants reported significantly higher scores for “Intention to use” (M = 42.89) compared to “Management Support” (M = 17.52). The large negative t-value suggests that the difference in means is substantial.

Furthermore, the positive correlation between “Management Support” and “Intention to use” (r = 0.406) indicates a moderate positive relationship between the two variables. This means that as the perception of management support increases, learners’ intention to use the eLearning platform also tends to increase.

The statistically significant correlation and the substantial difference in means suggest that management support plays a role in influencing learners’ intention to use the eLearning platform. This finding may have implications for improving the eLearning experience by focusing on enhancing management support to encourage greater utilization and engagement among learners.

The Summary of Measuring Scales (Reliability Results)

All the p-values (Asymp. Sig.) are reported as 0.000, which means they are very small (below the level of precision typically reported). This indicates that there is a highly significant association between each of the variables (Infrastructure, Skills and Training, Management Support, and Intention to use) and the “Intention to use” variable.

Table 1: Summary of Reliability Result

Further testing of the constructs using Pearson’s correlation coefficients for the predictor variables, Infrastructure, Skill & Training, and Management support, were computed to determine the nature of relationships, how much change in the independent variables is associated with how much change in the dependent variable as well as determining the level of significance for each for the variables. The results are presented in the summary of multiple analysis. 

Table 2: Summary of the Model constructs relationship

Figure2: Significance level

Figure 3: Constructs relationship

The model of success factors for eLearning system in Kampala International University, Western Campus Districts is illustrated below. Apart from IS success factors identified by D & M ISS Model of 2016 (system quality, information quality, service quality, intention to use/system use, user satisfaction, net impacts), three more constructs have been added and these include: infrastructure, skills and training and management support as a modified model from D & M ISS model for KIU eLearning success model.

Figure 4: eLearning Success Model

CONCLUSION

In conclusion, this study demonstrates that factors such as system quality, information quality, service quality, infrastructure, learner skills & training, and management support are critical determinants of learners’ intention to use the eLearning platform, hence a successful utilization and net impacts. The Correlations and Chi-square tests provide substantial evidence for the influence of these factors on engagement and utilization. A theoretical eLearning success model was derived and an evaluation of the model was done using Pearson Correlation and Chi-Square.

REFERENCES

1. Alshibly. H.H (2015) Investigating Decision Support System (DSS) Success: A Partial Least Squares Structural Equation Modeling Approach, Journal of Business Studies Quarterly 6: 2152-1034
2. DeLone, W.H., and McLean, E.R. (2003): Information systems success: The quest for the dependent variable. Information Systems Research Journal, 3: 60-95.
3. DeLone W. H. and McLean E. R. (2016): Information Systems Success Measurement. Foundations and Trends in Information Systems, vol. 2, no. 1, pp. 1–116, 2016. DOI: 10.1561/2900000005
4. Hadullo K, Oboko R, and Omwenga e. (2017): A Model for Evaluating E-learning Systems Quality in Higher Education in Developing Countries. International Journal of Education and Development using Information and Communication Technology (IJEDICT), 2017, Vol. 13, Issue 2, pp. 185-204.
5. ICT Directorate (2022): Kampala International University-Western Campus, Ishaka, Uganda
6. Lee, K. C., & Chung, N., (2009). Understanding factors affecting trust in and satisfaction with mobile banking in Korea: A modified DeLone& McLean’s model perspective. Interacting with computers, 21(5), 385-392.
7. Liang C. Y and Zheng Q (2017): The Path of New Information Technology affecting Educational Equality in the New Digital Divide— Based on Information System Success Model. EURASIA Journal of Mathematics Science and Technology Education. 2017 13(7):3587-3597 DOI 10.12973/eurasia.2017.00747a
8. Lin, Wang &Rong (2015) Assessing Mobile Learning Systems Success, International journal of Information and Education Technology 6: 7
9. Sheri Conklin (2020): Evaluating LMS. In D. Barreto, A Rottmann, & S Rabidoux, Learning Management Systems: Choosing the Right Path For Your Organization. EdTech Books. Retrieved from.
10. Tagreed K, Alaadin Alrowwad, Musa Al-Lozi (2016): A Review of Literature on E-Learning Systems in Higher Education. International Journal of Business Management and Economic Research (IJBMER), Vol 7(5),2016, 754-762