Cross-Dataset Reliability of Random Forest Phishing URL Detection under Heterogeneous Feature Representations: Implications for Cybersecurity Decision-Making

Phishing URL detectors are commonly evaluated using training and test samples drawn from the same dataset, a practice that can overstate reliability when operational data differ in source, collection period, class composition or feature-generation process. This study evaluates Random Forest transferability under within-dataset and heterogeneous cross-dataset conditions and considers the implications of transfer failure for cybersecurity decision-making. A 6,000-sample source dataset was constructed from PhishTank phishing URLs and an Alexa-derived legitimate URL list, while 6,000 records from the UCI Phishing Websites dataset formed an unseen structured target set. A semantic compatibility audit distinguished eight high-equivalence URL-reproducible features from three limited-equivalence measurements and 19 non-transferable attributes. Three Random Forest configurations were evaluated using grouped 10-fold cross-validation, source hold-out testing, registered-domain-grouped validation and one-way target transfer. RF-200 achieved 0.752 accuracy and 0.799 balanced accuracy on the source hold-out set; registered-domain grouping reduced balanced accuracy to 0.779. On the target dataset, the 11-feature model collapsed to an all-phishing prediction pattern, producing 0.633 accuracy and 1.000 phishing recall but 0.000 specificity and 0.500 balanced accuracy. Removing SSLfinal_State and Abnormal_URL produced a 9-feature sensitivity representation with 0.590 balanced accuracy, although one limited-equivalence port variable remained. The results show that aggregate accuracy can conceal complete class-level failure and that feature-generation semantics materially affect transferability. For operational cybersecurity, the findings emphasize independent target evaluation, domain-aware validation, transparent feature definitions, class-specific metrics and cautious use of model outputs in automated blocking or warning decisions.

cybersecurity decision-making; cross-dataset evaluation; phishing URL detection; Random Forest; transferability

1. Anti-Phishing Working Group. (2026). Phishing activity trends report, 4th quarter 2025. APWG. [Google Scholar] [Crossref]

2. Sahoo, D., Liu, C., & Hoi, S. C. H. (2017). Malicious URL detection using machine learning: A survey. arXiv:1701.07179. https://doi.org/10.48550/arXiv.1701.07179 [Google Scholar] [Crossref]

3. Le, H., Pham, Q., Sahoo, D., & Hoi, S. C. H. (2018). URLNet: Learning a URL representation with deep learning for malicious URL detection. arXiv:1802.03162. https://doi.org/10.48550/arXiv.1802.03162 [Google Scholar] [Crossref]

4. Sahingoz, O. K., Buber, E., Demir, O., & Diri, B. (2019). Machine learning based phishing detection from URLs. Expert Systems with Applications, 117, 345-357. https://doi.org/10.1016/j.eswa.2018.09.029 [Google Scholar] [Crossref]

5. Gupta, B. B., Yadav, K., Razzak, I., Psannis, K. E., Castiglione, A., & Chang, X. (2021). A novel approach for phishing URLs detection using lexical based machine learning in a real-time environment. Computer Communications, 175, 47-57. https://doi.org/10.1016/j.comcom.2021.04.023 [Google Scholar] [Crossref]

6. Zamir, A., et al. (2020). Phishing web site detection using diverse machine learning algorithms. The Electronic Library, 38(1), 65-80. https://doi.org/10.1108/EL-05-2019-0118 [Google Scholar] [Crossref]

7. Khan, S. A., Khan, W., & Hussain, A. (2021). Phishing attacks and websites classification using machine learning and multiple datasets: A comparative analysis. arXiv:2101.02552. https://doi.org/10.48550/arXiv.2101.02552 [Google Scholar] [Crossref]

8. Breiman, L. (2001). Random forests. Machine Learning, 45, 5-32. https://doi.org/10.1023/A:1010933404324 [Google Scholar] [Crossref]

9. Yasin, A., Fatima, R., Khan, J. A., & Afzal, W. (2024). Behind the Bait: Delving into PhishTank’s hidden data. Data in Brief, 52, 109959. https://doi.org/10.1016/j.dib.2023.109959 [Google Scholar] [Crossref]

10. Mohammad, R., & McCluskey, L. (2012). Phishing Websites. UCI Machine Learning Repository. https://doi.org/10.24432/C51W2X [Google Scholar] [Crossref]

11. Rashid, F., Doyle, B., Han, S. C., & Seneviratne, S. (2024). Phishing URL detection generalisation using unsupervised domain adaptation. Computer Networks, 245, 110398. https://doi.org/10.1016/j.comnet.2024.110398 [Google Scholar] [Crossref]

12. Divakaran, D. M., & Oest, A. (2022). Phishing detection leveraging machine learning and deep learning: A review. arXiv:2205.07411. https://doi.org/10.48550/arXiv.2205.07411 [Google Scholar] [Crossref]

13. Yang, R., Zheng, K., Wu, B., Wu, C., & Wang, X. (2021). Phishing website detection based on deep convolutional neural network and random forest ensemble learning. Sensors, 21(24), 8281. https://doi.org/10.3390/s21248281 [Google Scholar] [Crossref]

14. Alani, M. M., & Tawfik, H. (2022). PhishNot: A cloud-based machine-learning approach to phishing URL detection. Computer Networks, 218, 109407. https://doi.org/10.1016/j.comnet.2022.109407 [Google Scholar] [Crossref]

15. Jain, A. K., & Gupta, B. B. (2019). A machine learning based approach for phishing detection using hyperlinks information. Journal of Ambient Intelligence and Humanized Computing, 10(5), 2015-2028. https://doi.org/10.1007/s12652-018-0798-z [Google Scholar] [Crossref]

16. Mohammad, R. M., Thabtah, F., & McCluskey, L. (2014). Predicting phishing websites based on self-structuring neural network. Neural Computing and Applications, 25, 443-458. https://doi.org/10.1007/s00521-013-1490-z [Google Scholar] [Crossref]

17. Ma, J., Saul, L. K., Savage, S., & Voelker, G. M. (2009). Beyond blacklists: Learning to detect malicious web sites from suspicious URLs. In Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (pp. 1245-1254). https://doi.org/10.1145/1557019.1557153 [Google Scholar] [Crossref]

18. Oest, A., et al. (2020). Sunrise to Sunset: Analyzing the end-to-end life cycle and effectiveness of phishing attacks at scale. In Proceedings of the USENIX Security Symposium (pp. 361-377). [Google Scholar] [Crossref]

19. Ghalechyan, H., Israyelyan, E., Arakelyan, A., Hovhannisyan, G., & Davtyan, A. (2024). Phishing URL detection with neural networks: An empirical study. Scientific Reports, 14, 25134. https://doi.org/10.1038/s41598-024-74725-6 [Google Scholar] [Crossref]

20. Potpelwar, R. S., Kulkarni, U. V., & Waghmare, J. M. (2025). LegitPhish: A large-scale annotated dataset for URL-based phishing detection. Data in Brief, 63, 111972. https://doi.org/10.1016/j.dib.2025.111972 [Google Scholar] [Crossref]

21. Bahaghighat, M., Ghasemi, M., & Ozen, F. (2023). A high-accuracy phishing website detection method based on machine learning. Journal of Information Security and Applications, 77, 103553. https://doi.org/10.1016/j.jisa.2023.103553 [Google Scholar] [Crossref]

22. Mat Rani, L., Mohd Foozy, C. F., & Mustafa, S. N. B. (2023). Feature selection to enhance phishing website detection based on URL using machine learning techniques. Journal of Soft Computing and Data Mining, 4(1), 30-41. [Google Scholar] [Crossref]

23. Yeasmin, M. N., Refat, M. A. R., Singh, B. C., Alom, Z., Aung, Z., & Azim, M. (2026). EnLeM: Ensemble learning-based model to detect phishing websites. EURASIP Journal on Information Security, 2026, 1. https://doi.org/10.1186/s13635-025-00219-1 [Google Scholar] [Crossref]

24. Le Pochat, V., Van Goethem, T., Tajalizadehkhoob, S., Korczyński, M., & Joosen, W. (2019). Tranco: A research-oriented top sites ranking hardened against manipulation. In Proceedings of the Network and Distributed System Security Symposium (NDSS). https://doi.org/10.14722/ndss.2019.23386 [Google Scholar] [Crossref]

25. Ahamed, T., Kakon, S. C., Farid, F. A., Uddin, J., & Abdul Karim, H. B. (2026). An integrated evaluation protocol for adversarial robustness, generalization and explanation stability in URL-based phishing detection. Frontiers in Computer Science, 8, 1834407. https://doi.org/10.3389/fcomp.2026.1834407 [Google Scholar] [Crossref]

26. Wan Abdullah, W. A. A., Muslim, Z., Nahar, H., & Motsidi, R. (2026). Enhancing phishing URL detection using a two-level rule-based framework combining lexical and RDAP registration features. International Journal of Research and Innovation in Social Science, 10(8), 330-347. https://doi.org/10.47772/IJRISS.2026.100800025 [Google Scholar] [Crossref]

• What the Desert Fathers Teach Data Scientists: Ancient Ascetic Principles for Ethical Machine-Learning Practice
• Comparative Analysis of Some Machine Learning Algorithms for the Classification of Ransomware
• Comparative Performance Analysis of Some Priority Queue Variants in Dijkstra’s Algorithm
• Transfer Learning in Detecting E-Assessment Malpractice from a Proctored Video Recordings.
• Dual-Modal Detection of Parkinson’s Disease: A Clinical Framework and Deep Learning Approach Using NeuroParkNet