Development of Hybrid Distilled Bidirectional Encoder Representations from Transformers and Support Vector Machine Model for Phishing Email Detection

Phishing email attacks are popular and common cyber security threats because they become more sophisticated and rely on social engineering techniques. The contextual and semantic features present in phishing emails are often missed by standalone machine-learning methods and transformer-based models like Bidirectional Encoder Representations from Transformers (BERT) have high computational complexity. A hybrid phishing email detection framework was created by using Fine-Tuned DistilBERT and Support Vector Machine (SVM) to boost detection accuracy while keeping low computational costs. The phishing email dataset from Elhanas et al. (2024) that includes 18,650 labeled samples of phishing emails was used. Preprocessing of the data has been done before the model building stage, such as text cleaning, removal of unwanted duplicate records, and encoding of labels. The dataset was randomly split into an 80:20 split between train and test. Three models: Standalone SVM model, Hybrid BERT-SVM model, and the Fine-Tuned DistilBERT-SVM hybrid model. The accuracy, precision, recall, F1 score, confusion matrix, ROC analysis, and computational cost were performed to evaluate the model's performance. Experimental results revealed that the Fine-Tuned DistilBERT-SVM had an accuracy of 97.49%, precision of 97.19%, recall of 97.89%, and an F1 score of 97.54%. The model achieved high accuracy compared with Random Forest and XGBoost classifiers and excelled in phishing detection ability by its high recall performance. Furthermore, the DistilBERT achieved about 40% less parameters than the original BERT, which enhanced the computational efficiency and accelerated inference of the model. Because the results of the study suggest that the Fine-Tuned DistilBERT-SVM framework is effective, robust, and computationally efficient, it appears to be a suitable solution for the phishing email detection system.

DistilBERT; Machine Learning; Phishing Email Detection; Support Vector Machine; Transformer Models

1. Alharbi, F., Aljuhani, A., & Alotaibi, R. (2022). Phishing detection using machine learning techniques: A survey. IEEE Access, 10, 56–70. [Google Scholar] [Crossref]

2. Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. In J. Burstein, C. Doran, & T. Solorio (Eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long and Short Papers) (pp. 71–86). Association for Computational Linguistics. https://doi.org/10.18653/v1/N19-1423 [Google Scholar] [Crossref]

3. Elhanas, A., et al. (2024). Novel interpretable and robust web-based AI platform for phishing email detection. Computers & Electrical Engineering, 120, 109625. https://doi.org/10.1016/j.compeleceng.2024.109625 [Google Scholar] [Crossref]

4. Eze, C. S., & Shamir, L. (2024). Analysis and prevention of AI-based phishing email attacks. Electronics, 13(10), 1839. https://doi.org/10.3390/electronics13101839 [Google Scholar] [Crossref]

5. Fares, H., Kilani, J., Fagroud, F. E., Toumi, H., Lakrami, F., Baddi, Y., & Aknin, N. (2024). Machine learning approach for email phishing detection. Procedia Computer Science, 251, 746–751. https://doi.org/10.1016/j.procs.2024.11.179 [Google Scholar] [Crossref]

6. Jamal, S., & Wimmer, H. (2023). An improved transformer-based model for detecting phishing, spam, and ham: A large language model approach. arXiv. https://doi.org/10.48550/arXiv.2311.04913 [Google Scholar] [Crossref]

7. Khan, M. A., Ullah, I., & Kim, H. (2024). Transformer-based phishing email detection using contextual embeddings. Computers & Security, 134, 103245. https://doi.org/10.1016/j.cose.2023.103245 [Google Scholar] [Crossref]

8. Otieno, D. O., Abri, F., Namin, A. S., & Jones, K. S. (2023). Detecting phishing URLs using the BERT transformer model. In Proceedings of the 2023 IEEE International Conference on Big Data (pp. 2483–2492). IEEE. https://doi.org/10.1109/BigData59044.2023.10386782 [Google Scholar] [Crossref]

9. Rashid, F., Doyle, B., Han, S. C., & Seneviratne, S. (2024). Phishing URL detection generalisation using domain adaptation. Computer Networks. [Google Scholar] [Crossref]

10. Sarker, I. H., Kayes, A. S. M., & Watters, P. (2023). Cybersecurity data science: An overview from machine learning perspective. Journal of Big Data, 10(1), 1–30. https://doi.org/10.1186/s40537-023-00710-3 [Google Scholar] [Crossref]

11. Uddin, M. A., & Sarker, I. H. (2024). An explainable transformer-based model for phishing email detection: A large language model approach. arXiv. https://doi.org/10.48550/arXiv.2402.13871 [Google Scholar] [Crossref]

12. Uddin, M. A., Islam, M. N., Maglaras, L., Janicke, H., & Sarker, I. H. (2024). ExplainableDetector: Exploring transformer-based language modeling approach for SMS spam detection with explainability analysis. arXiv. https://doi.org/10.48550/arXiv.2405.08026 [Google Scholar] [Crossref]

13. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention is all you need. In Advances in Neural Information Processing Systems (Vol. 30). Curran Associates, Inc. [Google Scholar] [Crossref]

• Methane Emissions from Municipal Solid Waste - Case Study in Cai Rang District, Can Tho City, Vietnam
• Youth Activism, Intentional Integration of Policies to Raise Awareness on Climate Change Action among the Youth
• Breathing Spaces: Environmental & User Experience in Dhanmondi and Zigatola Multistoried Apartments, Dhaka, Bangladesh
• Effects of Solid Waste Disposal on Soil Quality in Makurdi Metropolis, Benue State, Nigeria
• Environmental Impact of Artisanal and Small-Scale Gold Mining in Borgu Local Government Area