Proactive IT network monitoring through log analysis using ML and Open AI

This research focused on a machine learning technique ( XGBoost – Extreme Gradient boosting), Transformer models (all-MiniLM-L6-v2 a sentence embedding model developed by Microsoft) based system for proactive network monitoring, performing log analysis for real-time anomaly detection and pattern analysis for root cause evaluation. This was done in order to address the challenge of reacting to problems only after they occur which leads to business revenue loss and increased idle time for workers when business operations are disrupted. The system makes use of the online NLP (natural language processing) model specifically (OPENAI or Cohere), which are inferred for intelligent problem explanation and solution recommendation. The methodology used was CRISP-DM for Data Science and incremental software methodology. The system enables network administrators to identify emerging problems within the network and address them pro-actively through system provided recommendations and anomaly evaluation insights before full negative impact on business operations.

Log analysis, Machine learning, Explainable AI, Pattern Analysis, Artificial Intelligence, Natural language processing)

1. F. Liu, B. Farkiani, and P. Crowley, “Llms for Computer Networking Operations & Management: A Survey on Applications, Key Techniques, and Opportunities,”2024,SSRN.doi: 10.2139/ssrn.5074973. [Google Scholar] [Crossref]

2. D. L. Vajda, T. V. Do, T. Bérczes, and K. Farkas, “Machine learning-based real-time anomaly detection using data pre-processing in the telemetry of server farms,” Sci. Rep., vol. 14, no. 1, p. 23288, Oct. 2024, doi: 10.1038/s41598-024-72982-z. [Google Scholar] [Crossref]

3. S. Mukherjee, “Machine Learning Methodologies for Beyond 5G and 6G Heterogeneous Networks: Prediction, Automation, and Performance Analysis,” 2024. [Google Scholar] [Crossref]

4. F. Liu, B. Farkiani, and P. Crowley, “A Survey on Large Language Models for Network Operations & Management: Applications, Techniques, and Opportunities,” Dec. 10, 2024, Preprints. doi: 10.36227/techrxiv.173386065.57486944/v1. [Google Scholar] [Crossref]

5. F. Wang, Y. Jiang, R. Zhang, A. Wei, J. Xie, and X. Pang, “A Survey of Deep Anomaly Detection in Multivariate Time Series: Taxonomy, Applications, and Directions,” Sensors, vol. 25, no. 1, p. 190, Jan. 2025, doi: 10.3390/s25010190. [Google Scholar] [Crossref]

6. F. Vitale, F. De Vita, N. Mazzocca, and D. Bruneo, “A Process Mining-based unsupervised Anomaly Detection technique for the Industrial Internet of Things,” Internet Things, vol. 24, p. 100993, Dec. 2023, doi: 10.1016/j.iot.2023.100993. [Google Scholar] [Crossref]

7. A. H. Shah, D. Pasha, E. H. Zadeh, and S. Konur, “Automated Log Analysis and Anomaly Detection Using Machine Learning,” in Frontiers in Artificial Intelligence and Applications, A. J. Tallón-Ballesteros, Ed., IOS Press, 2022. doi: 10.3233/FAIA220378. [Google Scholar] [Crossref]

8. C. Cao, F. Wang, L. Lindley, and Z. Wang, “Managing Linux servers with LLM-based AI agents: An empirical evaluation with GPT4,” Mach. Learn. Appl., vol. 17, p. 100570, Sep. 2024, doi: [Google Scholar] [Crossref]

9. 1016/j.mlwa.2024.100570. [Google Scholar] [Crossref]

10. J. Cândido, M. Aniche, and A. Van Deursen, “Log-based software monitoring: a systematic mapping study,” PeerJ Comput. Sci., vol. 7, p. e489, May 2021, doi: 10.7717/peerj-cs.489. [Google Scholar] [Crossref]

11. J. Cândido, M. Aniche, and A. Van Deursen, “Log-based software monitoring: a systematic mapping study,” PeerJ Comput. Sci., vol. 7, p. e489, May 2021, doi: 10.7717/peerj-cs.489. [Google Scholar] [Crossref]

12. Y. Zhang, “Design and Implementation of a Computer Network Log Analysis System Based on Big Data Analytics,” Adv. Comput. Signals Syst., vol. 8, no. 6, 2024, doi: 10.23977/acss.2024.080607. [Google Scholar] [Crossref]

13. S, Partovian, A. Bucaioni, F. Flammini, & J. Thornadtsson. (2023). ‘Analysis of log files to enable smart-troubleshooting in industry 4.0: a systematic mapping study’. IEEE Access, 12, 147640-147658. [Google Scholar] [Crossref]

14. S. H. Kendyala, (2023). High Availability Strategies for Identity Access Management Systems in Large Enterprises. Available at SSRN 5074869. [Google Scholar] [Crossref]

• A Machine Learning Model for Predicting the Risk of Developing Diabetes - T2DM Using Real-World Data from Kilifi, Kenya
• AI-Powered Facial Recognition Attendance System Using Deep Learning and Computer Vision
• A Comprehensive Review on Brain Tumour Segmentation Using Deep Learning Approach
• A Scalable Retrieval-Augmented Generation Pipeline for Domain-Specific Knowledge Applications
• Predictive Maintenance in Semiconductor Manufacturing Using Machine Learning on Imbalanced Dataset