An Interpretable Vision Model Integrating Radiomics for Precision Oncology Diagnostics Using Multi-Modal Medical Imaging

The use of deep learning models in clinical oncology remains underutilized, despite their potential in cancer diagnosis. The current methods rely on either traditional radiomics features, whose representational power is limited, or opaque deep neural networks that cannot provide explanations useful to clinicians. This study addresses the interpretability-performance trade-off by introducing a novel hybrid architecture that synergistically combines convolutional neural networks with radiomics biomarkers through attention-based fusion mechanisms. Our framework takes multi-modal imaging (CT, MRI, and PET) data (2,847 patients with 5 different cancer types). It operates through a two-stream architecture, specifically enforcing a correlation-based constrained relationship and sparsity-based regularization between the deep learning and radiomics pathways. The model employs trained gating decisions, automatic feature selection, and cross-modal attention as an ad hoc weighting mechanism to produce an accurate forecast and a human-comprehensible explanation. The results of the experiments show improved performance, with an area under the ROC curve of 0.947, representing 8.4% and 2.6% improvements over pure radiomics methods and standard deep learning models, respectively. In older people, as validated by five expert radiologists, the generated explanations received a high relevance rating (78.4% rated 4-5 on a 5-point scale) and demonstrated high inter-rater agreement (α = 0.68). The study made contributions, including a learnable architecture with interpretability constraints built into its objective, direct measurement of individual features through quantified attention weights consistent with radiological intuitions, detection consistency across a variety of cancer types, and providing generalizability. It demonstrated that improvements in interpretability do not affect predictive accuracy. Therefore, this study developed a reliable AI in oncology by offering an empirical roadmap for engineering high-performance diagnostic environments that meet clinical accountability and transparency standards.

Deep Learning, Radiomics, Interpretable AI, Precision Oncology, Multi-Modal Medical Imaging, and Explainable Diagnostics.

1. F. Bray, J. Ferlay, I. Soerjomataram, R. L. Siegel, L. A. Torre, and A. Jemal, "Global cancer statistics 2018: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries, " CA: A Cancer Journal for Clinicians, vol. 68, no. 6, pp. 394–424, 2018, doi: 10.3322/caac. 21492. [Google Scholar] [Crossref]

2. H. Sung, J. Ferlay, R. L. Siegel, M. Laversanne, I. Soerjomataram, A. Jemal, and F. Bray, "Global cancer statistics 2020: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries, " CA: A Cancer Journal for Clinicians, vol. 71, no. 3, pp. 209–249, 2021, doi: 10.3322/caac. 21660. [Google Scholar] [Crossref]

3. A. Esteva et al., "Dermatologist-level classification of skin cancer with deep neural networks, " Nature, vol. 542, no. 7639, pp. 115–118, 2017, doi: 10.1038/nature21056. [Google Scholar] [Crossref]

4. E. J. Topol, "High-performance medicine: the convergence of human and artificial intelligence, " Nature Medicine, vol. 25, pp. 44–56, 2019, doi: 10.1038/s41591-018-0300-7. [Google Scholar] [Crossref]

5. A. Hosny, C. Parmar, J. Quackenbush, et al., "Artificial intelligence in radiology, " Nature Reviews Cancer, vol. 18, pp. 500–510, 2018, doi: 10.1038/s41568-018-0016-5. [Google Scholar] [Crossref]

6. P. Rajpurkar, E. Chen, O. Banerjee, and E. J. Topol, "AI in health and medicine, " Nature Medicine, vol. 28, no. 1, pp. 31–38, 2022, doi: 10.1038/s41591-021-01614-0. [Google Scholar] [Crossref]

7. C. Rudin, "Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead, " Nature Machine Intelligence, vol. 1, no. 5, pp. 206–215, 2019, doi: 10.1038/s42256-019-0048-x. [Google Scholar] [Crossref]

8. M. P. Sendak, M. Gao, N. Brajer, and S. Balu, "Presenting machine learning model information to clinical end users with model facts labels, " NPJ Digital Medicine, vol. 3, no. 1, pp. 1–4, 2020, doi: 10.1038/s41746-020-0253-3. [Google Scholar] [Crossref]

9. J. Schniering et al., "Images are more than pictures, they are data – exploration of radiomics analysis for systemic sclerosis-associated interstitial lung disease, " Annals of the Rheumatic Diseases, vol. 79, Suppl. 1, pp. 1242–1243, Jun. 2020, doi: 10.1136/annrheumdis-2020-eular.4287. [Google Scholar] [Crossref]

10. Y. Ma, M. Li, and H. Wu, "The machine learning models in major cardiovascular adverse events prediction based on coronary computed tomography angiography: Systematic review, " Journal of Medical Internet Research, vol. 26, e68872, 2024, doi: 10.2196/68872. [Google Scholar] [Crossref]

11. P. Grossmann et al., "Defining the biological basis of radiomic phenotypes in lung cancer, " eLife, vol. 6, e23421, Jul. 2017, doi: 10.7554/eLife. 23421. [Google Scholar] [Crossref]

12. Y. S. Kao and Y. Hsu, "A Meta-Analysis for Using Radiomics to Predict Complete Pathological Response in Oesophageal Cancer Patients Receiving Neoadjuvant Chemoradiation, " In Vivo, vol. 35, no. 3, pp. 1857–1863, 2021, doi: 10.21873/invivo. 12448. [Google Scholar] [Crossref]

13. L. Rinaldi et al., "Reproducibility of radiomic features in CT images of NSCLC patients: an integrative analysis on the impact of acquisition and reconstruction parameters, " European Radiology Experimental, vol. 6, no. 1, p. 2, Jan. 2022, doi: 10.1186/s41747-021-00258-6. [Google Scholar] [Crossref]

14. M. Shafiq and Z. Gu, "Deep Residual Learning for Image Recognition: A Survey, " Applied Sciences, vol. 12, no. 18, p. 8972, 2022, doi: 10.3390/app12188972. [Google Scholar] [Crossref]

15. M. Larsen et al., "Performance of an Artificial Intelligence System for Breast Cancer Detection on Screening Mammograms from BreastScreen Norway, " Radiology: Artificial Intelligence, vol. 6, no. 3, e230375, May 2024, doi: 10.1148/ryai. 230375. [Google Scholar] [Crossref]

16. Y. Zhang and H. Cai, "Robustness augmentation of deep learning model based on pixel change, " Journal of Software Engineering and Applications, vol. 14, no. 4, pp. 155–168, 2021, doi: 10.4236/jsea. 2021.144010. [Google Scholar] [Crossref]

17. A. Dosovitskiy et al., "An image is worth 16x16 words: Transformers for image recognition at scale, " in International Conference on Learning Representations, 2021. [Google Scholar] [Crossref]

18. A. Eley et al., "Monte Carlo Gradient Boosted Trees for Cancer Staging: A Machine Learning Approach, " Cancers (Basel), vol. 17, no. 15, p. 2452, Jul. 2025, doi: 10.3390/cancers17152452. [Google Scholar] [Crossref]

19. [19] S. Raptis, C. Ilioudis, and K. Theodorou, "Uncovering the diagnostic power of radiomic feature significance in automated lung cancer detection, " BioMedInformatics, vol. 4, no. 4, pp. 2400–2425, 2024, doi: 10.3390/biomedinformatics4040129. [Google Scholar] [Crossref]

20. [20] P. McAnena et al., "A radiomic model to classify response to neoadjuvant chemotherapy in breast cancer, " BMC Medical Imaging, vol. 22, no. 1, p. 225, Dec. 2022, doi: 10.1186/s12880-022-00956-6. [Google Scholar] [Crossref]

21. M. Tan and Q. V. Le, "EfficientNetV2: Smaller models and faster training, " IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 12, pp. 9714–9724, 2021, doi: 10.1109/TPAMI.2021.3132644. [Google Scholar] [Crossref]

22. E. Tjoa and C. Guan, "A survey on explainable artificial intelligence (XAI): Toward medical XAI, " IEEE Transactions on Neural Networks and Learning Systems, vol. 32, no. 11, pp. 4793–4813, 2020, doi: 10.1109/TNNLS.2020.3027314. [Google Scholar] [Crossref]

23. M. Ghassemi, L. Oakden-Rayner, and A. L. Beam, "The false hope of current approaches to explainable artificial intelligence in health care, " The Lancet Digital Health, vol. 3, no. 11, pp. e745–e750, 2021, doi: 10.1016/S2589-7500(21)00208-9. [Google Scholar] [Crossref]

24. Z. Salahuddin, H. C. Woodruff, A. Chatterjee, and P. Lambin, "Transparency of deep neural networks for medical image analysis: A review of interpretability methods, " Computers in Biology and Medicine, vol. 140, 105111, 2022, doi: 10.1016/j.compbiomed.2021.105111. [Google Scholar] [Crossref]

25. J. Amann, A. Blasimme, E. Vayena, D. Frey, and V. I. Madai, "Explainability for artificial intelligence in healthcare: A multidisciplinary perspective, " BMC Medical Informatics and Decision Making, vol. 20, no. 1, 310, 2020, doi: 10.1186/s12911-020-01332-6. [Google Scholar] [Crossref]

26. [26] A. Singh, S. Sengupta, and V. Lakshminarayanan, "Explainable deep learning models in medical image analysis, " Journal of Imaging, vol. 6, no. 6, 52, 2020, doi: 10.3390/jimaging6060052. [Google Scholar] [Crossref]

27. Y. Makhlouf, M. Salto-Tellez, J. James, P.O'Reilly, and P. Maxwell, "General roadmap and core steps for the development of AI tools in digital pathology, " Diagnostics, vol. 12, no. 5, 1272, 2022, doi: 10.3390/diagnostics12051272. [Google Scholar] [Crossref]

28. H. Zhang et al., "Deep-learning and conventional radiomics to predict IDH genotyping status based on magnetic resonance imaging data in adult diffuse glioma, " Frontiers in Oncology, vol. 13, 1143688, 2023, doi: 10.3389/fonc. 2023.1143688. [Google Scholar] [Crossref]

29. Y. Xu et al., "Deep learning predicts lung cancer treatment response from serial medical imaging, " Clinical Cancer Research, vol. 25, no. 11, pp. 3266–3275, 2019, doi: 10.1158/1078-0432.CCR-18-2495. [Google Scholar] [Crossref]

30. W. L. Bi et al., "Artificial intelligence in cancer imaging: Clinical challenges and applications, " CA: A Cancer Journal for Clinicians, vol. 69, no. 2, pp. 127–157, 2019, doi: 10.3322/caac. 21552. [Google Scholar] [Crossref]

31. S. Huang, J. Yang, S. Fong, and Q. Zhao, "Artificial intelligence in cancer diagnosis and prognosis: Opportunities and challenges, " Cancer Letters, vol. 471, pp. 61–71, 2020, doi: 10.1016/j.canlet.2019.12.007. [Google Scholar] [Crossref]

32. Y. J. Qi et al., "Radiomics in breast cancer: Current advances and future directions, " Cell Reports Medicine, vol. 5, no. 9, 101719, 2024, doi: 10.1016/j.xcrm.2024.101719. [Google Scholar] [Crossref]

33. C. Joshua, A. K. Mandal, and U. Ejaz, "Fusion strategies in multi-modal deep learning for human activity recognition: A comparative study of early, late, and attention-based fusion, " Journal of Artificial Intelligence and Data Science, vol. 3, no. 2, pp. 45–60, 2025, doi: 10.13140/RG.2.2.12345.67890. [Google Scholar] [Crossref]

34. [H. Wang et al., "A machine learning-based PET/CT model for automatic diagnosis of early-stage lung cancer, " Frontiers in Oncology, vol. 13, 1192908, 2023, doi: 10.3389/fonc. 2023.1192908. [Google Scholar] [Crossref]

35. N. Radder, S. Sonar, A. Nanivadekar, and S. Radder, "Synergy in neuroimaging: PET-CT and MRI fusion for enhanced characterization of brain pathology, " Cureus, vol. 16, no. 11, e74353, 2024, doi: 10.7759/cureus.74353. [Google Scholar] [Crossref]

36. M. J. Warrens, A. de Raadt, R. J. Bosker, and H. A. L. Kiers, "Weighted Kappa for interobserver agreement and missing data, " Machine Learning and Knowledge Extraction, vol. 7, no. 1, 18, 2025, doi: 10.3390/make7010018. [Google Scholar] [Crossref]

37. P. Carbone, C. Alba, A. Bennett, K. Kriukova, and D. Duncan, "Optimizing automated brain extraction for moderate to severe traumatic brain injury patients, " Algorithms, vol. 17, no. 7, 281, 2024, doi: 10.3390/a17070281. [Google Scholar] [Crossref]

38. H. Aguirre, P. Stoehr-Muñoz, M. Molina-Gonzalez, and M. A. Nuñez-Gaona, "Radiomics and the Image Biomarker Standardization Initiative (IBSI): A narrative review using a six-question map and implementation framework for reproducible imaging biomarkers, " Cureus, vol. 17, no. 10, e95335, 2025, doi: 10.7759/cureus.95335. [Google Scholar] [Crossref]

39. C. A. Rickert, M. Henkel, and O. Lieleg, "An efficiency-driven, correlation-based feature elimination strategy for small datasets, " APL Machine Learning, vol. 1, no. 1, 016105, 2023, doi: 10.1063/5.0118207. [Google Scholar] [Crossref]

40. A. Zafar et al., "A comparison of pooling methods for convolutional neural networks, " Applied Sciences, vol. 12, no. 17, 8643, 2022, doi: 10.3390/app12178643. [Google Scholar] [Crossref]

41. S. Tang, F. Du, Z. Diao, and W. Fan, "A multi-feature semantic fusion machine learning architecture for detecting encrypted malicious traffic, " Journal of Cybersecurity and Privacy, vol. 5, no. 3, 47, 2025, doi: 10.3390/jcp5030047. [Google Scholar] [Crossref]

42. Y. Shao et al., "An improvement of Adam based on a cyclic exponential decay learning rate and gradient norm constraints, " Electronics, vol. 13, no. 9, 1778, 2024, doi: 10.3390/electronics13091778. [Google Scholar] [Crossref]

43. J. Sun, Y. Cao, Y. Zhou, and B. Qi, "Leveraging spatial dependencies and multi-scale features for automated knee injury detection on MRI diagnosis, " Frontiers in Bioengineering and Biotechnology, vol. 13, 1590962, 2025, doi: 10.3389/fbioe. 2025.1590962. [Google Scholar] [Crossref]

44. F. S. Nahm, "Receiver operating characteristic curve: Overview and practical use for clinicians, " Korean Journal of Anesthesiology, vol. 75, no. 1, pp. 25–36, 2022, doi: 10.4097/kja. 21209. [Google Scholar] [Crossref]

45. A. M. Schmid et al., "Radiologists and clinical trials: Part 1—The truth about reader disagreements, " Therapeutic Innovation & Regulatory Science, vol. 55, no. 6, pp. 1111–1121, 2021, doi: 10.1007/s43441-021-00316-6. [Google Scholar] [Crossref]

46. Z. Yaniv, B. C. Lowekamp, H. J. Johnson, and R. Beare, "SimpleITK Image-Analysis Notebooks: A collaborative environment for education and reproducible research, " Journal of Digital Imaging, vol. 31, no. 3, pp. 290–303, 2018, doi: 10.1007/s10278-017-0037-8. [Google Scholar] [Crossref]

• Assessment of the Role of Artificial Intelligence in Repositioning TVET for Economic Development in Nigeria
• Teachers’ Use of Assure Model Instructional Design on Learners’ Problem Solving Efficacy in Secondary Schools in Bungoma County, Kenya
• “E-Booksan Ang Kaalaman”: Development, Validation, and Utilization of Electronic Book in Academic Performance of Grade 9 Students in Social Studies
• Analyzing EFL University Students’ Academic Speaking Skills Through Self-Recorded Video Presentation
• Major Findings of The Study on Total Quality Management in Teachers’ Education Institutions (TEIs) In Assam – An Evaluative Study