Development of an Intelligent Security Information and Event Management System

SIEM systems should transform disorganized streams of heterogeneous logs into a prioritized set of evidence-enriched alerts. An intelligent processing prototype was designed for Security Information and Event Management (SIEM) by implementing the event normalization, machine-learning scoring, severity-aware prioritization, time-bounded correlation and structured evidence storage procedures. The validation set of 2000 records from BGL log dataset hosted by Loghub was used as an input to the intelligent detector. The 2000 records include 1857 normal and 143 abnormal system events, which were sorted according to the numeric timestamp and split into the training (1400), validation (300) and test (300) subsets. The term frequency-inverse document frequency matrix was constructed from 648 message fields, excluding the benchmark label, source identifiers, timestamps and severity attributes. This document-term matrix was used to train a class-weighted logistic regression classifier, whose validation threshold was selected to maximize the performance metric on the validation subset. The intelligent detector demonstrated 97.00% accuracy, 100.00% precision, 62.50% recall and 0.7692 F1-score by identifying 15 out of 24 test set anomalies with no false positives. The severity-based rule detected all 24 of the test set anomalies with 23 false positives. The correlation procedure reduced the number of triggered alerts from 15 to 13, thus eliminating 13.33% of the case volume. The Naive Bayes classifier demonstrated the highest held-out F1 performance among the tested machine-learning models. The intelligent processing prototype was evaluated on the labelled operational log records to identify the minimum required volume of evidence for the security case to be submitted for human review.

security information and event management; log analytics; event correlation; machine learning; alert prioritization; logistic regression; evidence management

1. Ban, T., Takahashi, T., Ndichu, S., & Inoue, D. (2023). Breaking alert fatigue: AI-assisted SIEM framework for effective incident response. Applied Sciences, 13(11), 6610. https://doi.org/10.3390/app13116610 [Google Scholar] [Crossref]

2. González-Granadillo, G., González-Zarzosa, S., & Diaz, R. (2021). Security information and event management (SIEM): analysis, trends, and usage in critical infrastructures. Sensors, 21(14), 4759. https://doi.org/10.3390/s21144759 [Google Scholar] [Crossref]

3. Guo, H., Yuan, S., & Wu, X. (2021). LogBERT: log anomaly detection via BERT. In 2021 International Joint Conference on Neural Networks (IJCNN) (pp. 1–8). https://doi.org/10.1109/IJCNN52387.2021.9534113 [Google Scholar] [Crossref]

4. He, S., He, P., Chen, Z., Yang, T., Su, Y., & Lyu, M. R. (2021). A survey on automated log analysis for reliability engineering. ACM Computing Surveys, 54(6), 1–37. https://doi.org/10.1145/3460345 [Google Scholar] [Crossref]

5. Khan, Z. A., Shin, D., Bianculli, D., & Briand, L. C. (2024). Impact of log parsing on deep learning-based anomaly detection. Empirical Software Engineering, 29(6), Article 139. https://doi.org/10.1007/s10664-024-10533-w [Google Scholar] [Crossref]

6. Kotenko, I., Gaifulina, D., & Zelichenok, I. (2022). Systematic literature review of security event correlation methods. IEEE Access, 10, 43387–43420. https://doi.org/10.1109/ACCESS.2022.3168976 [Google Scholar] [Crossref]

7. Landauer, M., Onder, S., Skopik, F., & Wurzenberger, M. (2023). Deep learning for anomaly detection in log data: a survey. Machine Learning with Applications, 12, 100470. https://doi.org/10.1016/j.mlwa.2023.100470 [Google Scholar] [Crossref]

8. Landauer, M., Skopik, F., Frank, M., Hotwagner, W., Wurzenberger, M., & Rauber, A. (2023). Maintainable log datasets for evaluation of intrusion detection systems. IEEE Transactions on Dependable and Secure Computing, 20(4), 3466–3482. https://doi.org/10.1109/TDSC.2022.3201582 [Google Scholar] [Crossref]

9. Le, V. H., & Zhang, H. (2022). Log-based anomaly detection with deep learning: How far are we?. In Proceedings of the 44th International Conference on Software Engineering (pp. 1356–1367). https://doi.org/10.1145/3510003.3510155 [Google Scholar] [Crossref]

10. Levshun, D., & Kotenko, I. (2023). A survey on artificial intelligence techniques for security event correlation: models, challenges, and opportunities. Artificial Intelligence Review, 56(8), 8547–8590. https://doi.org/10.1007/s10462-022-10381-4 [Google Scholar] [Crossref]

11. Muhammad, A. R., Sukarno, P., & Wardana, A. A. (2023). Integrated security information and event management (SIEM) with intrusion detection system (IDS) for live analysis based on machine learning. Procedia Computer Science, 217, 1406–1415. https://doi.org/10.1016/j.procs.2022.12.339 [Google Scholar] [Crossref]

12. Radoglou-Grammatikis, P., Sarigiannidis, P., Iturbe, E., Rios, E., Martinez, S., Sarigiannidis, A., Eftathopoulos, G., Spyridis, Y., Sesis, A., Vakakis, N., Tzovaras, D., Kafetzakis, E., Giannoulakis, I., Tzifas, M., Giannakoulias, A., Angelopoulos, M., & Ramos, F. (2021). SPEAR SIEM: a security information and event management system for the smart grid. Computer Networks, 193, 108008. https://doi.org/10.1016/j.comnet.2021.108008 [Google Scholar] [Crossref]

13. Tendikov, N., Rzayeva, L., Saoud, B., Shayea, I., Azmi, M. H., Myrzatay, A., & Alnakhli, M. (2024). Security information event management data acquisition and analysis methods with machine learning principles. Results in Engineering, 22, 102254. https://doi.org/10.1016/j.rineng.2024.102254 [Google Scholar] [Crossref]

14. Uccello, F., Pawlicki, M., D'Antonio, S., Kozik, R., & Choraś, M. (2025). A new cybersecurity approach enhanced by xAI-derived rules to improve network intrusion detection and SIEM. Computers, Materials & Continua, 83(2), 1607–1621. https://doi.org/10.32604/cmc.2025.062801 [Google Scholar] [Crossref]

15. Xie, Y., Zhang, H., & Babar, M. A. (2024). LogSD: detecting anomalies from system logs through self-supervised learning and frequency-based masking. Proceedings of the ACM on Software Engineering, 1(FSE), 2098–2120. https://doi.org/10.1145/3660800 [Google Scholar] [Crossref]

16. Zhu, J., He, S., He, P., Liu, J., & Lyu, M. R. (2023). Loghub: a large collection of system log datasets for AI-driven log analytics. In 2023 IEEE 34th International Symposium on Software Reliability Engineering (ISSRE) (pp. 355–366). https://doi.org/10.1109/ISSRE59848.2023.00071 [Google Scholar] [Crossref]

• What the Desert Fathers Teach Data Scientists: Ancient Ascetic Principles for Ethical Machine-Learning Practice
• Comparative Analysis of Some Machine Learning Algorithms for the Classification of Ransomware
• Comparative Performance Analysis of Some Priority Queue Variants in Dijkstra’s Algorithm
• Transfer Learning in Detecting E-Assessment Malpractice from a Proctored Video Recordings.
• Dual-Modal Detection of Parkinson’s Disease: A Clinical Framework and Deep Learning Approach Using NeuroParkNet