A Vision Based Deep Learning Framework for Malware Detection and Classification
Authors
Information Technology, Mahatma Gandhi Institute of Technology, Hyderabad, Telangana, 500075, India (India)
Information Technology, Mahatma Gandhi Institute of Technology, Hyderabad, Telangana, 500075, India (India)
Information Technology, Mahatma Gandhi Institute of Technology, Hyderabad, Telangana, 500075, India (India)
Article Information
DOI: 10.51584/IJRIAS.2026.11050042
Subject Category: Deep learning
Volume/Issue: 11/5 | Page No: 496-504
Publication Timeline
Submitted: 2026-04-30
Accepted: 2026-05-05
Published: 2026-05-27
Abstract
Malware detection is a complex task for signature-based anti-virus software, especially for polymorphic malware and zero-day attacks. However, this project proposes a vision-based static malware detection and classification method that represents raw executable file bytes as fixed-size grayscale images called byte plots and attempts to classify malware families based on these images without executing them. In this project, for the proposed model, the best architecture is Convolutional Neural Networks (CNN) + Random Forest (CNN-RF). Initially, a CNN is trained to learn discriminative feature embeddings for byte plot images. Once this is done, the final softmax classifier is removed, and this CNN is used to generate a 256-dimensional vector for each input. Then, a class-balanced Random Forest is trained to predict the malware family and confidence scores. In this way, this proposed method is able to achieve better results for two different datasets, and the best results obtained are 98.07% for MalImg and 93.07% for MaleVis.
Keywords
Learning, Detection, Classification
Downloads
References
1. Ahmed, S. R., et al. ”A novel approach to malware detection using machine learning and image processing.” Proceedings of the Cognitive Models and Artificial Intelligence Conference, 2024. [Google Scholar] [Crossref]
2. https://dl.acm.org/doi/abs/10.1145/3660853.3660931 [Google Scholar] [Crossref]
3. Aslan, O., and Abdullah Asim Yilmaz. ”A new malware classification¨ framework based on deep learning algorithms.” IEEE Access, 2021. [Google Scholar] [Crossref]
4. https://ieeexplore.ieee.org/abstract/document/9455368 [Google Scholar] [Crossref]
5. Yoo, Suyeon, et al. ”AI-HydRa: Advanced hybrid approach using random forest and deep learning for malware classification.” Information Sciences, 546 (2021). [Google Scholar] [Crossref]
6. https://www.sciencedirect.com/science/article/pii/S0020025520308525 [Google Scholar] [Crossref]
7. Moawad, Ahmad, Ahmed Ismail Ebada, and Aya M. Al-Zoghby. ”A Survey on Visualization-Based Malware Detection.” Journal of Cybersecurity, 4.3 (2022). [Google Scholar] [Crossref]
8. https://www.academia.edu/104208673/A_Survey_on_Visualization_Based_Malware_Detection [Google Scholar] [Crossref]
9. Sewak, M., Sahay, S. K., and Rathore, H. ”An investigation of a deep learning based malware detection system.” Proceedings of the 13th International Conference on Availability, Reliability and Security, 2018. [Google Scholar] [Crossref]
10. https://dl.acm.org/doi/abs/10.1145/3230833.3230835 [Google Scholar] [Crossref]
11. Buriro, A., et al. ”MalwD&C: a quick and accurate machine learningbased approach for malware detection and categorization.” Applied Sciences, 13.4 (2023). [Google Scholar] [Crossref]
12. https://www.mdpi.com/2076-3417/13/4/2508 [Google Scholar] [Crossref]
13. Tayyab, Umm-e-Hani, et al. ”A survey of the recent trends in deep learning based malware detection.” Journal of Cybersecurity and Privacy, 2.4 (2022). [Google Scholar] [Crossref]
14. https://www.mdpi.com/2624-800X/2/4/41 [Google Scholar] [Crossref]
15. Vinayakumar, R., et al. ”Robust intelligent malware detection using deep learning.” IEEE Access, 7 (2019). [Google Scholar] [Crossref]
16. https://ieeexplore.ieee.org/abstract/document/8681127 [Google Scholar] [Crossref]
17. Akhtar, M. S., and Tao Feng. ”Detection of malware by deep learning as CNN-LSTM machine learning techniques in real time.” Symmetry, 14.11 (2022). [Google Scholar] [Crossref]
18. https://www.mdpi.com/2073-8994/14/11/2308 [Google Scholar] [Crossref]
19. Karat, Gautam, et al. ”CNN-LSTM hybrid model for enhanced malware analysis and detection.” Procedia Computer Science, 233 (2024). [Google Scholar] [Crossref]
20. https://www.sciencedirect.com/science/article/pii/S1877050924005982 [Google Scholar] [Crossref]
Metrics
Views & Downloads
Similar Articles
- Deep Learning-Based Plant Identification Using Mobile Net and Res Net
- Comparative Analysis of Deep Learning Models for Ai-Driven Smart Waste Classification System Using Resnet, Efficientnet, and VGG16 for Automated Waste Segregation
- Explainable Deep Learning for Age and Gender Prediction from Facial Images: A Comparative Study of VGG16, Resnet50, and Efficientnet with Grad-CAM and SHAP
- A Noise-Robust CNN-KAN Architecture with Dual Attention for Enhanced Event Identification in Φ-OTDR Measurement Systems