Analyzing Rejected Hypotheses in Information System Effectiveness of Electronic Document Management System

Analyzing Rejected Hypotheses in Information System Effectiveness of Electronic Document Management System

Dayangku Horiah Awang Gani, Assoc. Prof. Dr. Irwan Kamaruddin Abdul Kadir^*, and Assoc. Prof Ts. Dr. Mohd Razilan Abdul Kadir

College of Computing, Informatics and Mathematics, UiTM Selangor Branch, Puncak Perdana Campus, 40150 Shah Alam, Selangor, Malaysia.

*Corresponding author

DOI: https://dx.doi.org/10.47772/IJRISS.2024.8120340

Received: 20 December 2024; Accepted: 24 December 2024; Published: 23 January 2025

ABSTRACT

This study examines the unexpected rejection of four hypotheses in evaluating the effectiveness of Electronic Document Management Systems (EDMS). The hypotheses related to automated information capture, collaborative tools, hierarchical file indexing and classification, and security & audit trail. Theoretically, this study uses Delone and McLean Information System Success Model and Peter B. Seddon’s refinement of DeLone and McLean IS Success Model to examine the relationship between EDMS functionalities and the dimension of information system success.  The research adopted a quantitative method to address the research questions. However, these functionalities were statistically insignificant in the research findings. The analysis of the structural model found that Automated information captures do not significantly correlate with information system effectiveness (T-Value: 1.413,P < 0.05); Collaborative tools do not have a significant positive relationship with information system effectiveness (T-Value: 0.854, P < 0.05);  Hierarchical file indexing and classification have no significant positive relationship with information system effectiveness (T-Value: 0.376, P < 0.05); and  Security and audit trails have no significant positive relationship with information system effectiveness (T-Value: 0.940, P < 0.05) Thus, this paper explores the possible reasons for these results and their implications for future research.

Keywords: Electronic Document Management System; Information System Effectiveness; Rejected Hypotheses.

INTRODUCTION

Information systems are essential for managing organizational resources, increasing productivity, and refining decision-making processes. A crucial element of these systems is the Electronic Document Management System (EDMS), which enables document capture, storage, retrieval, collaboration, and security control. The efficacy of EDMS is frequently contingent upon many system functionalities, despite its increasing use.  Prior research has found various variables that enhance the efficacy of EDMS, including automated information capture, collaborative tools, hierarchical file indexing and classification, and security and audit trail.  These variables are typically anticipated to improve system utilization, user contentment, and organizational results.  Nonetheless, not all theoretical assumptions are valid in actual contexts. In a recent study conducted to evaluate EDMS effectiveness, four specific hypotheses were rejected:

H5: Automated information capture significantly correlates with information system effectiveness.

H7: Collaborative tools have a significant positive relationship with information system effectiveness.

H8: Hierarchical file indexing and classification have a significant positive relationship with information system effectiveness.

H11: Security and audit trails have a significant positive relationship with information system effectiveness.

The dismissal of these hypotheses prompts essential inquiries about the true impact of these parameters on EDMS success. This work examines the rejected hypotheses, investigates the rationale for their insignificance, and considers the implications for practice and future research.

LITERATURE REVIEW

A, Information System Effectiveness

Information System Effectiveness denotes the extent to which an information system fulfills organizational objectives, enhances performance, and addresses user requirements.  According to Delone and McLean (2003) in the DeLone and McLean’s Information System Success Model, factors such as system quality, information quality, and service quality influence overall system effectiveness.  On the other hand, Seddon (1997) has contributed significantly to research on perceived usefulness in the context of Information System, particularly through its inclusion in model such as the Technology Acceptance Model (TAM) and extensions of the DeLone and McLean IS Success Model.  In his refinement of DeLone and McLean IS Success Model, Seddon (1997) emphasized the importance of perceived usefulness as a mediating factor between system quality, information quality, and user satisfaction.

System Quality

This encompasses the technical performance, reliability, and user-friendliness of the system.  Automated Information Capture (H5), integral to system quality, aims to enhance data input efficiency and minimize errors.  Nonetheless, the rejection indicates that merely focusing on technical performance might not be adequate without effective user training or smooth incorporation into current workflows.  According to Seddon (1997), System Quality is concerned with whether or not there are “bugs” in the system, the consistency of the user interface, ease of use, quality of documentation, and sometimes, quality and maintainability of the program code.

Information Quality

This refers to the relevance, accuracy, and timeliness of the information provided by the system.  Hierarchical file indexing and classification (H8) is intended to improve information quality by systematically organizing data for efficient retrieval.  The rejection of this idea suggests several problems, like excessively intricate indexing systems or user inclination towards more straightforward search tools. Information Quality is concerned with such issues as the relevance, timeliness, and accuracy of information generated by an information system.  Not all applications of IT involve the production of Information for decision-making (e.g.: a word processor does not produce information) so Information Quality is not a measure that can be applied to all systems (Seddon, 1997)

Perceive Usefulness

Perceived usefulness in Information System is defined as the belief that a system improves task performance, productivity, or efficiency.  It is often studied in contexts like enterprise resource planning system. Electronic health records and Electronic Document Management System.  (Seddon, 1997)linked Perceived Usefulness to User Satisfaction in his refinement of the DeLone and McLean IS Success Model.  According to Seddon (1997), systems perceived as useful drive user satisfaction and overall system success.   System reliability, response time, and ease of use are essential precursors to Perceived Usefulness.  Systems that are user-friendly and perform consistently are more likely to be perceived as useful (Delone & McLean, 2003). While accuracy, relevant, and timely information enhances Perceived Usefulness, as users are more likely to trust and rely on systems providing high-quality data (Wixom & Todd, 2005).  System that aligns with the tasks user need to perform are perceived as more useful.  The Task-Technology Fit (TTF) model by Goodhue and Thompson (1995), demonstrates that when technology fits well with job requirements, Perceived Usefulness increases.

User Satisfaction

User Satisfaction is a subjective evaluation of the various Consequences evaluated on a pleasant-unpleasant continuum.  User satisfaction is probably the closest in meaning to the ideal Net Benefits measure. DeLone and McLean’s model also highlights the importance of user satisfaction as an intermediary in achieving system effectiveness.  The rejection of hypotheses H5, H7, H8, and H11 in the study highlights significant challenges in achieving user satisfaction through specific functionalities of Electronic Document Management System (EDMS).

The rejection of these four hypotheses (H5, H7, H8, and H11) implies that certain EDMS features may fail to meet user expectations or align with practical needs, thereby hindering overall satisfaction and effectiveness.  However, non-significant results play a crucial role in the research of information system success (ISS) by providing insights into the validity and reliability of theoretical models and hypotheses.  They help researchers understand whether a lack of significant findings is due to data insensitivity or if it genuinely challenges existing theories.  Non-significant results can reveal the limitations of current theories and practices, prompting further investigation and refinement of hypotheses (Edelsbrunner & Thurn, 2024).

B. Automated Information Capture (H5) 

Automated information capture involves technologies such as Optical Character Recognition (OCR), barcoding, and digital scanning to convert physical documents into digital formats.  Prior studies suggest that automation enhances efficiency, reduces manual errors, and improves information availability.  Viswanath Venkatesh and Bala (2008) emphasize that the perceived complexity of a system can reduces its acceptance and effectiveness, particularly for advanced features like automation.  However, the rejection of H5 in this study suggests that automated tools may not always lead to perceived system effectiveness due to factors such as:

• Low adoption or underutilization of automation features that users may not fully understand or utilize the automated features (Ahmad et.al.,2018).
• Poor integration of automated systems into workflows.
• User resistance to technological changes where organizational resistance can hinder the adoption of new tools (Kim & Kankanhalli, 2009).

C. Collaborative Tools (H7)

Collaborative tools, including shared document platforms, communication tools, and team workspaces, are designed to facilitate organizational collaboration.  Previous research has shown that these tools improve communication, knowledge sharing, and productivity (Al-Qahtani & Aksoy, 2022) (Brown et.al., 2019).  The rejection of H7 indicates possible challenges such as:

• Users may favor conventional communication channels like email over in-system tools (Ahmad et al., 2018)
• Poorly integration tools can disrupt workflows, making them less appealing (Alrawashdeh et al.,2021)
• Poor system usability or ineffective integration. User may not fully understand the potential benefits of these tools.

D. Hierarchical File Indexing and Classification (H8)

Hierarchical file indexing organizes documents in a structured manner, making retrieval efficient.  Studies have emphasized that structured classification improves system usability and reduces search time (Johnson & Lee, 2018).  Studies shows that user often prefer search-based systems over hierarchal structures due to ease of access (Davis, 1989).  The insignificance of H8 could be attributed to:

• Users may find search functionalities more intuitive than hierarchical indexing and classification functionalities in a system (V. Venkatesh, Morris, Davis, & Davis, 2003)).
• Complex hierarchical structures create confusion rather than efficiency where complex classification schemes may deter users (Agarwal & Prasad, 1999).

E. Security and Audit Trails (H11)

Security features, including access controls and audit trails, ensure document confidentiality, integrity, and accountability.  These features are often assumed to enhance user trust and compliance (Martin, 2021). However, the rejection of H11 suggests:

• Limited user interaction with security features as security measures can be seen as barriers to productivity (D’Arcy, Hovav, & Galletta, 2009).
• Misalignment between security policies and user needs
• The perception that security features hinder workflow efficiency

METHODOLOGY

This study employed a quantitative research design to analyze the relationship between EDMS features and information system effectiveness.  It utilizes a research methodology that employs both descriptive and hypothesis testing.  As a descriptive study, it aims to provide an in-depth understanding of the phenomenon under investigation, addressing the research question through detailed observation and analysis of the relevant variables.  The study used a hypotheses-testing methodology, proposing and evaluating twelve hypotheses to validate the theoretical assumptions and measure the effectiveness of the constructs within the research framework.  This dual-method approach offers a thorough examination of the issue by combining descriptive insights with empirical evidence obtained from hypotheses testing. The methodology included the following components:

A. Research Design

This study utilized a cross-sectional survey design to gather data from organizations that have implemented Electronic Document Management Systems (EDMS).  The research framework was structured around key system functionalities – automated information capture, collaborative tools, hierarchical and file indexing, and security & audit trails, as independent variables, while system effectiveness was identified as the dependent variable, while system effectiveness was identified as the dependent variable.   Ten elements are proposed, including the purpose of the study, extent of research interference, study setting, research strategies, measurement and measures, unit of analysis, sampling design, time horizon, data collection, and data analysis.

B. Sampling

The study was conducted in a natural organizational setting within designated Sarawak government departments and agencies, ensuring minimal disruption to routine operations.  The research emphasized the preservation of the uninterrupted operation of government services, ensuring no disruptions occurred during data gathering.  The study utilized a probability sampling method, ensuring that each unit in the population had an equal opportunity for selection.  The target population comprised Sarawak government services, primarily concentrating on the activities of selected government organizations utilizing the Electronic Document Management System (EDMS).  This approach ensured that the sample was representative of the broader context, allowing for generalizable and reliable findings.

C. Data Collection

This study collected data using an online questionnaire, allowing respondents ample time to complete it. The researcher is completely using survey research as the center of the research strategies. The setting used was considered non-contrived as the natural environment is preferred to analyze the assessment of the electronic document management system.

D. Data Analysis

The study focuses on the individual employees within the chosen agencies of the Sarawak Government.  The focus of analysis is centered on the individual level.  This research has concentrated on evaluating the performance of the effectiveness of the Electronic Document Management System within the agencies of the Sarawak Government.  The individuals selected for this study were employees who have utilized the Electronic Document Management System.

FINDINGS

A. Overview

This study tested twelve (12) hypotheses related to Electronic Document Management System effectiveness, focusing on automated information capture, collaborative tools, hierarchical file indexing and classification, and security and audit trail.  Four hypotheses were rejected.

B. Hypothesis Testing

The results of the hypothesis testing are presented in the table below:

Table 1: Research Mapping between Research Objective, Variables, Hypotheses, and Findings of the Study

Automated information capture was hypothesized to have a significant positive relationship with information system effectiveness.  A t-test was conducted to evaluate this relationship, yielding a t-value of -1.413 with a significance level of p<0.05.  These results led to the rejection of the hypothesis, indicating no significant positive effect. This result corresponds with the findings of (Sezgin et al., 2014), where individual reluctance to abandon traditional business practices, reliance on paper, computer-related anxiety, poor computer literacy rates in public institutions, inefficient and user-unfriendly EDMS technology, and insufficient experience prevail.

The analysis resulted in a t-value of – 8.54 (p < 0.05), leading to the rejection of the hypothesis.  This indicates that contrary to the expectation, implementing collaborative tools in EDMS did not significantly enhance faster document sharing in this study’s context.  This finding aligns with previous studies by (Balogun, Raheem, Abdulrahaman, & Balogun, 2019)which also reported that low user adoption and they prefer dealing with papers when communicating with other organizations in EDMS hindered their effectiveness.

The hypothesis of the significant effect of hierarchical file indexing and classification on the overall effectiveness of the Electronic Document Management System was rejected.  The statistical analysis yielded a t-value of – 0.376 with a p-value less than 0.05.  This result indicates that there was no statistically significant relationship between hierarchical file indexing and classification and the EDMS effectiveness within the observed sample.

The statistical analysis revealed a t-value of 0.940, p <0.05, indicating that security and audit trail functionalities did not have a statistical effect on the overall effectiveness of EDMS in the context of this study.  Although security and audit trails are widely recognized as critical elements in the protection and integrity of electronic documents, the results suggest that, within the sample and conditions of this study, these features did not significantly improve the perceived effectiveness of the system.

DISCUSSION

The rejection of hypotheses H5, H7, H8, and H11 provides important insights into the complexities of achieving EDMS effectiveness.  While automation is theoretically designed to minimize human errors under the automated information capture construct, the findings suggest that its potential benefits are often constrained by practical barriers.  These barriers may include a lack of technical expertise among users, the underutilization of automated tools, or resistance to organizational change.  Addressing these challenges requires not only the deployment of automated tools but also strategic efforts to build technical capacity and foster user acceptance.  The implementation of mandatory training programs designed to develop skills in document digitization and automated capture plays a critical role in improving organizational efficiency.  By incorporating technical skill-building into onboarding processes and periodical employee evaluations, this approach enables the identification of skill gaps, the customization of training programs to address specific needs, and the systematic measurement of progress.  Such a strategy ensures that employees are well-equipped to perform their roles effectively, contributing to both individual and organizational success.

Similarly, the study found that collaborative tools, which are typically intended to enhance communication and teamwork, may not significantly contribute to EDMS effectiveness. This could stem from users’ preference for traditional methods of communication or insufficient integration of these tools within existing workflows.  Poorly integrated or overly complex collaborative tools may fail to meet user needs, ultimately leading to underuse and limited impact on system performance.  Five key features of effective collaborative tools include learning from positive deviance, high-quality coordination, high-quality measurement, motivational levers, and mobilizing professional leadership (McGowan, J., et al., 2023).

The lack of significance associated with hierarchical indexing further emphasizes the importance of aligning system design with user preferences.  Users may prioritize ease of access and the simplicity offered by search functionalities over the structured classification provided by hierarchical indexing.  This finding suggests that developers and organizations should focus on enhancing intuitive search capabilities rather than relying solely on rigid classification systems.  Using special methods of analysis, classification, and systematization of archival documents effectively determines information links between them, ensuring logical organization and facilitating further use. Text classification, using both traditional and newer natural language processing technologies, can effectively manage risk and meet compliance obligations in records management (Franks, J., 2022). For instance, a hospital network deployed an NLP-based text classification system to manage its patient records.  This not only streamlined administrative workflows but also ensured compliance with industry-specific regulations such as the Health Insurance Portability and Accountability Act (HIPAA).

Moreover, the rejection of the hypothesis related to security measures sheds light on the perceived trade-offs between security and usability.  While robust security protocols are critical for safeguarding sensitive information, users may view these measures as unnecessary, potentially withdrawing their perceived value in enhancing system effectiveness.  This highlights the need for security features to balance strict protection with user-friendly implementation.  To maximize the potential of EDMS functionalities, government departments and agencies must address key factors such as user training, system integration, and adoption of challenges. A more considered approach to present security functionality is needed to improve end-user understanding and ability to effectively protect themselves. A case study by Su, Te-Jen, et al. (2017) examines the usability challenges of EDMS in government settings, highlighting how human factors can improve system design to balance security and user accessibility.

In summary, the findings provide valuable insights into the complexities of optimizing EDMS effectiveness.  Government departments and agencies must go beyond merely implementing technical features and instead adopt holistic approaches that consider user preferences, system design, and operational integration.  Future research could explore the mediating effects of user satisfaction and system usability on the relationship between these functionalities and EDMS effectiveness.

CONCLUSION

This study highlights the rejection of four hypotheses related to Electronic Document Management System (EDMS) functionalities and their relationship to information system effectiveness, providing insights into the challenges of achieving expected outcomes.  The findings emphasize the need for further research to explore contextual, organizational, and user-related factors that influence information system success.  By addressing these issues, government departments and agencies can develop targeted strategies to enhance user adoption, optimize system functionality, and align EDMS implementation with organizational goals.  Future research should focus on these issues that bridge the gap between theoretical potential and practical application.  This study contributes to the growing body of knowledge on EDMS, paving the way for innovations in system design, implementation, and evaluation to achieve sustained information system success.

ACKNOWLEDGMENT

The authors would like to express their heartfelt gratitude to the College of Computing, Informatics and Mathematics, UiTM Selangor Branch, Puncak Perdana Campus, for the invaluable guidance and encouragement throughout the study. We also thank the editorial board of the International Journal of Research and Innovation in Social Science (IJRISS) for the opportunity to share our findings with the broader academic community.  Finally, we are grateful to our families, friends, and colleagues for their unwavering support and understanding during the research process.

REFERENCES

1. Agarwal, R., & Prasad, J. (1999). Are individual differences germane to the acceptance of new information technologies? Decision sciences, 30(2), 361-391.
2. Al-Qahtani, M., & Aksoy, M. (2022). Collaborative Tools and Techniques of Knowledge Sharing: A Literature Review. Journal of Computer and Communications, 10, 45-54.
3. Balogun, N., Raheem, L., Abdulrahaman, M. D., & Balogun, U. (2019). Adoptability of electronic document management system in Ilorin businesses. Nigerian Journal of Technology, 38(3), 707–715-707–715.
4. D’Arcy, J., Hovav, A., & Galletta, D. (2009). User awareness of security countermeasures and its impact on information systems misuse: A deterrence approach. Information Systems Research, 20(1), 79-98.
5. Davis, F. D. (1989). Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information Technology. MIS Quarterly, Vol.13(No.3), 319-340.
6. Delone, W., & McLean, E. (2003). The DeLone and McLean Model of Information Systems Success: A Ten-Year Update. J. of Management Information Systems, 19, 9-30.
7. Edelsbrunner, P. A., & Thurn, C. M. (2024). Improving the utility of non-significant results for educational research: A review and recommendations. Educational Research Review, 42, 100590.
8. Franks, J. (2022). Text Classification for Records Management. Journal on Computing and Cultural Heritage (JOCCH), 15, 1 – 19.
9. Goodhue, D. L., & Thompson, R. L. (1995). Task-Technology Fit and Individual Performance. MIS Quarterly, 19(2), 213-236.
10. Kim, H.-W., & Kankanhalli, A. (2009). Investigating user resistance to information systems implementation: a status quo bias perspective. Management Information Systems Quarterly, 33, 567-582.
11. McGowan, J., Martin, G., Krapohl, G., Campbell, D., Englesbe, M., Dimick, J., & Dixon-Woods, M. (2023). What are the features of high-performing quality improvement collaboratives? A qualitative case study of a state-wide collaboratives programme. BMJ Open, 13.
12. Seddon, P. B. (1997). A Respecification and Extension of the DeLone and McLean Model of IS Success. Info. Sys. Research, 8(3), 240–253.
13. Sezgin, E., Medeni, T., Önaçan, M., Komurcu, R., Dalbay, Ö., & Medeni, İ. T. (2014). The Perception of Electronic Document Management Systems (EDMS) as a Transformational Information and Communication Technology (ICT) for Public Institutions in Turkey. In (pp. 279-300).
14. Su, Te-Jen & Wang, Shih-Mine & Chen, Yi-Feng & Tsou, Tung-Yeh & Cheng, Jui-Chuan. (2017). Investigating the Usability of Electronic Document Management Systems in Government Organizations from a Human Factor Engineering Perspective. Journal of Advanced Management Science. 5. 14-17.
15. Venkatesh, V., & Bala, H. (2008). Technology acceptance model 3 and a research agenda on interventions. Decision sciences, 39(2), 273-315.
16. Venkatesh, V., Morris, M. G., Davis, G. B., & Davis, F. D. (2003). User Acceptance of Information Technology: Toward a Unified View. Institutions & Transition Economics: Microeconomic Issues eJournal.
17. Wixom, B., & Todd, P. (2005). A Theoretical Integration of User Satisfaction and Technology Acceptance. Information Systems Research, 16, 85-102.