Anomaly Detection using ML in Real-Time Systems

Modern day real-time systems have been applied in many domains like finance, healthcare, security, and industries, among others. Real-time systems are characterized by continuous creation of huge amounts of data, thus making manual detection of anomalies difficult. Anomaly detection is very essential as anomalies can be used to detect errors in real-time systems and even cyber attacks. In this study, an anomaly detection system using machine learning methods is presented. The proposed model adopts several methods like data preprocessing, feature extraction, supervised and unsupervised machine learning methods to efficiently detect anomalies. The system is able to analyze streams of data, therefore, providing fast and accurate detection of anomalies with minimal time delays.

Machine Learning, Anomaly Detection, Real-Time Systems

1. V. Chandola, A. Banerjee, and V. Kumar, “Anomaly Detection: A Survey,” ACM Computing Surveys, vol. 41, no. 3, [Google Scholar] [Crossref]

2. pp. 1–58, 2009. [Google Scholar] [Crossref]

3. C. C. Aggarwal, Outlier Analysis, 2nd ed., Springer, New York, NY, USA, 2015. [Google Scholar] [Crossref]

4. F. T. Liu, K. M. Ting, and Z.-H. Zhou, “Isolation Forest,” IEEE International Conference on Data Mining (ICDM), pp. 413–422, 2008. [Google Scholar] [Crossref]

5. M. M. Breunig, H. P. Kriegel, R. T. Ng, and J. Sander, “LOF: Identifying Density-Based Local Outliers,” ACM SIGMOD International Conference on Management of Data, pp. 93–104, 2000. [Google Scholar] [Crossref]

6. M. Ahmed, A. N. Mahmood, and J. Hu, “A Survey of Network Anomaly Detection Techniques,” Journal of Network and Computer Applications, vol. 60, pp. 19–31, 2016. [Google Scholar] [Crossref]

7. I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning, MIT Press, Cambridge, MA, USA, 2016. [Google Scholar] [Crossref]

8. D. Pimentel, D. Clifton, L. Clifton, and L. Tarassenko, “A Review of Novelty Detection,” Signal Processing, vol. 99, pp. 215–249, 2014. [Google Scholar] [Crossref]

9. Apache Software Foundation, “Apache Spark: Unified Analytics Engine for Big Data Processing,” Available: https://spark.apache.org [Google Scholar] [Crossref]

10. Apache Software Foundation, “Apache Kafka: Distributed Event Streaming Platform,” Available: https://kafka.apache.org [Google Scholar] [Crossref]

11. Scikit-learn Developers, “Machine Learning in Python,” [Google Scholar] [Crossref]

12. Available: https://scikit-learn.org [Google Scholar] [Crossref]

13. T. Ahmed, M. Coates, and A. Lakhina, “Multivariate Online Anomaly Detection Using Kernel Recursive Least Squares,” IEEE INFOCOM, 2007. [Google Scholar] [Crossref]

14. S. Hawkins, H. He, G. Williams, and R. Baxter, “Outlier Detection Using Replicator Neural Networks,” Data Warehousing and Knowledge Discovery, 2002. [Google Scholar] [Crossref]

15. J. Dean and S. Ghemawat, “MapReduce: Simplified Data Processing on Large Clusters,” OSDI, 2004. [Google Scholar] [Crossref]

16. L. Ruff et al., “Deep One-Class Classification,” International Conference on Machine Learning (ICML), 2018. [Google Scholar] [Crossref]

17. Z. Chen, C. K. Yeo, B. S. Lee, and C. T. Lau, “Autoencoder-Based Anomaly Detection,” IEEE Conference, 2017. [Google Scholar] [Crossref]

18. A. Patcha and J. M. Park, “An Overview of Anomaly Detection Techniques: Existing Solutions and Latest Technological Trends,” Computer Networks, vol. 51, no. 12, pp. 3448–3470, 2007. [Google Scholar] [Crossref]

19. M. Goldstein and S. Uchida, “A Comparative Evaluation of Unsupervised Anomaly Detection Algorithms,” Pattern Recognition, vol. 46, no. 5, pp. 1469–1483, 2013. [Google Scholar] [Crossref]

20. K. Leung and C. Leckie, “Unsupervised Anomaly Detection in Network Intrusion Detection Using Clusters,” ACSAC, 2005. [Google Scholar] [Crossref]

21. Y. Zhao, Z. Nasrullah, and Z. Li, “PyOD: A Python Toolbox for Scalable Outlier Detection,” Journal of Machine Learning Research, vol. 20, pp. 1–7, 2019. [Google Scholar] [Crossref]

22. H. He, J. Yang, and S. Wang, “Learning from Imbalanced Data,” IEEE Transactions on Knowledge and Data Engineering, vol. 21, no. 9, pp. 1263–1284, 2009. [Google Scholar] [Crossref]

• A Machine Learning Model for Predicting the Risk of Developing Diabetes - T2DM Using Real-World Data from Kilifi, Kenya
• AI-Powered Facial Recognition Attendance System Using Deep Learning and Computer Vision
• A Comprehensive Review on Brain Tumour Segmentation Using Deep Learning Approach
• A Scalable Retrieval-Augmented Generation Pipeline for Domain-Specific Knowledge Applications
• Predictive Maintenance in Semiconductor Manufacturing Using Machine Learning on Imbalanced Dataset