Hybrid U-Net++–Vision Transformer Fusion for Accurate Brain MRI Image Segmentation and classifications

Automatic brain Magnetic Resonance Imaging (MRI) analysis plays a crucial role in computer-aided diagnosis, treatment planning, and disease monitoring by enabling accurate delineation and identification of brain tumors. However, manual segmentation is labor-intensive, time-consuming, and susceptible to inter-observer variability, making automated and reliable methods highly desirable. This paper proposes a Hybrid U-Net++ and Vision Transformer (ViT) Fusion framework for automatic brain MRI tumor segmentation and segmentation-guided classification. The proposed architecture employs a U-Net++ encoder with nested skip connections to extract hierarchical multi-scale features, while a Vision Transformer captures long-range spatial dependencies and global contextual information through self-attention mechanisms. A multi-stage feature fusion strategy integrates convolutional and transformer representations, complemented by boundary-aware refinement to improve tumor localization and preserve fine anatomical details. The segmentation model achieved a training Dice score of 95.6%, IoU of 91.4%, and validation Dice score of 92.6%, demonstrating stable convergence and strong generalization capability. Furthermore, the proposed framework attained a Dice score of 95.8%, IoU of 92.1%, and Hausdorff Distance of 3.35 mm in the ablation analysis, confirming the effectiveness of multi-stage feature fusion and boundary-aware refinement. The segmented tumor regions are subsequently utilized for Region of Interest (ROI)-based classification using deep feature extraction, Global Average Pooling (GAP), and fully connected layers. The classification module distinguishes glioma, meningioma, pituitary tumor, and no-tumor cases, achieving an overall accuracy of 97.55%, macro-average precision of 97.13%, recall of 97.50%, F1-score of 97.31%, and specificity of 98.40%. Comparative analysis further demonstrates the effectiveness of the proposed hybrid architecture over conventional CNN-based and transformer-based approaches. The combined segmentation and classification results indicate that the proposed framework provides an effective approach for tumor localization and tumor-type identification, with potential applicability in automated brain MRI analysis and clinical decision support.

Brain MRI, Automatic Medical Image Segmentation, U-Net++, Vision Transformer (ViT), Deep Learning, Medical Image Analysis

1. Cao, H., Wang, Y., Chen, J., Jiang, D., Zhang, X., Tian, Q., & Wang, M. (2022). Swin-Unet: Unet-like pure Transformer for medical image segmentation. In European Conference on Computer Vision (ECCV) Workshops. Link: https://doi.org/10.1007/978-3-031-25066-8_9 | arXiv:2105.05537 [Google Scholar] [Crossref]

2. Chen, J., Lu, Y., Yu, Q., Luo, X., Adeli, E., Wang, Y., Lu, L., Yuille, A. L., & Zhou, Y. (2021). TransUNet: Transformers make strong encoders for medical image segmentation. arXiv preprint. Link: https://arxiv.org/abs/2102.04306 | Google Scholar [Google Scholar] [Crossref]

3. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., & Houlsby, N. (2021). An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations (ICLR). Link: https://openreview.net/forum?id=YicbFdUnOE | arXiv:2010.11929 [Google Scholar] [Crossref]

4. Gordaliza, P. M., Muñoz-Barrutia, A., Abella, M., Desco, M., Sharpe, S., & Vaquero, J. J. (2018). Unsupervised CT lung image segmentation of a Mycobacterium tuberculosis infection model. Scientific Reports, 8(1), 9802. Link: https://doi.org/10.1038/s41598-018-28100-x | PubMed: 29955140 [Google Scholar] [Crossref]

5. Hatamizadeh, A., Tang, Y., Roth, H., & Xu, D. (2022). UNETR: Transformers for 3D medical image segmentation. In IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 574–584. Link: https://doi.org/10.1109/WACV51458.2022.00181 | arXiv:2103.10504 [Google Scholar] [Crossref]

6. Isensee, F., Jaeger, P. F., Kohl, S. A. A., Petersen, J., & Maier-Hein, K. H. (2021). nnU-Net: A self-configuring method for deep learning-based biomedical image segmentation. Nature Methods, 18(2), 203–211. Link: https://doi.org/10.1038/s41592-020-01008-z | PubMed: 33288961 [Google Scholar] [Crossref]

7. Jiang, J., Hu, Y., Liu, C., Halpenny, D., Hellmann, M. D., Deasy, J. O., Mageras, G., & Veeraraghavan, H. (2018). Multiple resolution residually connected feature streams for automatic lung tumor segmentation from CT images. IEEE Transactions on Medical Imaging, 38(1), 134–144. Link: https://doi.org/10.1109/tmi.2018.2857800 | PubMed: 30040635 [Google Scholar] [Crossref]

8. Jin, D., Xu, Z., Tang, Y., Harrison, A. P., & Mollura, D. J. (2018). CT-realistic lung nodule simulation from 3D conditional Generative Adversarial Networks for robust lung segmentation. In Medical Image Computing and Computer Assisted Intervention (MICCAI), LNCS (Vol. 11071, pp. 732–740). Link: https://doi.org/10.1007/978-3-030-00934-2_81 | arXiv:1806.04051 [Google Scholar] [Crossref]

9. Li, S., Zhang, C., & Wang, Y. (2021). Shape-aware semi-supervised medical image segmentation. Medical Image Analysis, 72, 102128. Link: https://doi.org/10.1016/j.media.2021.102128 | PubMed: 34171693 [Google Scholar] [Crossref]

10. Li, X., Chen, H., Qi, X., Dou, Q., Fu, C.-W., & Heng, P.-A. (2018). H-DenseUNet: Hybrid densely connected UNet for liver and tumor segmentation from CT volumes. IEEE Transactions on Medical Imaging, 37(12), 2663–2674. Link: https://doi.org/10.1109/TMI.2018.2845918 | PubMed: 29994358 [Google Scholar] [Crossref]

11. Ma, J., He, Y., Li, F., Han, L., You, C., & Wang, B. (2024). Segment anything in medical images. Nature Communications, 15(1), 654. Link: https://doi.org/10.1038/s41467-024-44824-z | PubMed: 38253604 [Google Scholar] [Crossref]

12. Milletari, F., Navab, N., & Ahmadi, S.-A. (2016). V-Net: Fully convolutional neural networks for volumetric medical image segmentation. In 2016 Fourth International Conference on 3D Vision (3DV), pp. 565–571. Link: https://doi.org/10.1109/3DV.2016.79 | arXiv:1606.04797 [Google Scholar] [Crossref]

13. Oktay, O., Schlemper, J., Folgoc, L. L., Lee, M., Heinrich, M., Misawa, K., Mori, K., McDonagh, S., Hammerla, N. Y., Kainz, B., Glocker, B., & Rueckert, D. (2018). Attention U-Net: Learning where to look for the pancreas. arXiv preprint. Link: https://arxiv.org/abs/1804.03999 | Google Scholar [Google Scholar] [Crossref]

14. Ronneberger, O., Fischer, P., & Brox, T. (2015). U-Net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention (MICCAI), LNCS (Vol. 9351, pp. 234–241). Springer, Cham. Link: https://doi.org/10.1007/978-3-319-24574-4_28 | PubMed: 31728630 [Google Scholar] [Crossref]

15. Shelhamer, E., Long, J., & Darrell, T. (2016). Fully convolutional networks for semantic segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(4), 640–651. Link:https://doi.org/10.1109/TPAMI.2016.2572683 | PubMed: 27244717 [Google Scholar] [Crossref]

16. Shen, D., Wu, G., & Suk, H.-I. (2017). Deep learning in medical image analysis. Annual Review of Biomedical Engineering, 19(1), 221–248. Link: https://doi.org/10.1146/annurev-bioeng-071516-044442 | PubMed: 28301734 [Google Scholar] [Crossref]

17. Tarvainen, A., & Valpola, H. (2017). Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. In Advances in Neural Information Processing Systems (NeurIPS), 30. Link: https://papers.nips.cc/paper_files/paper/2017/hash/68053af2923e00204c3ca7c6a3150cf7-Abstract.html | arXiv:1703.01780 [Google Scholar] [Crossref]

18. Valanarasu, J. M. J., Oza, P., Hacihaliloglu, I., & Patel, V. M. (2021). Medical Transformer: Gated axial-attention for medical image segmentation. In Medical Image Computing and Computer Assisted Intervention (MICCAI), LNCS (Vol. 12901, pp. 36–46). Link: https://doi.org/10.1007/978-3-030-87193-2_4 | arXiv:2102.10662 [Google Scholar] [Crossref]

19. Wang, X., Girshick, R., Gupta, A., & He, K. (2018). Non-local neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7794–7803. Link: https://doi.org/10.1109/CVPR.2018.00813 | arXiv:1711.07971 [Google Scholar] [Crossref]

20. Xie, E., Wang, W., Yu, Z., Anandkumar, A., Alvarez, J. M., & Luo, P. (2021). SegFormer: Simple and efficient design for semantic segmentation with Transformers. In Advances in Neural Information Processing Systems (NeurIPS), 34, 12077–12090. Link: https://papers.nips.cc/paper/2021/hash/64f1f27bf1b4ec22924f7425616c027e-Abstract.html | arXiv:2105.15203 [Google Scholar] [Crossref]

21. Ye, Z., Zhang, Y., Wang, Y., Huang, Z., & Song, B. (2020). Chest CT manifestations of new coronavirus disease 2019 (COVID-19): A pictorial review. European Radiology, 30(8), 4381–4389. Link: https://doi.org/10.1007/s00330-020-06801-0 | PubMed: 32193638 [Google Scholar] [Crossref]

22. Yu, L., Cheng, J.-Z., Dou, Q., Yang, X., Chen, H., Qin, J., & Heng, P.-A. (2017). Automatic 3D cardiovascular MR segmentation with densely-connected volumetric ConvNets. In Medical Image Computing and Computer Assisted Intervention (MICCAI), LNCS (Vol. 10434, pp. 287–295). Link: https://doi.org/10.1007/978-3-319-66185-8_33 | PubMed: 29552684 [Google Scholar] [Crossref]

23. Zhou, Z., Siddiquee, M. M. R., Tajbakhsh, N., & Liang, J. (2018). UNet++: A nested U-Net architecture for medical image segmentation. In Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support (DLMIA/ML-CDS), LNCS (Vol. 11045, pp. 3–11). Link: https://doi.org/10.1007/978-3-030-00889-5_1 | PubMed: 30976707 [Google Scholar] [Crossref]

• What the Desert Fathers Teach Data Scientists: Ancient Ascetic Principles for Ethical Machine-Learning Practice
• Comparative Analysis of Some Machine Learning Algorithms for the Classification of Ransomware
• Comparative Performance Analysis of Some Priority Queue Variants in Dijkstra’s Algorithm
• Transfer Learning in Detecting E-Assessment Malpractice from a Proctored Video Recordings.
• Dual-Modal Detection of Parkinson’s Disease: A Clinical Framework and Deep Learning Approach Using NeuroParkNet