Beyond Predictive Accuracy: A Multi-Criteria Adaptive Framework for Reliability-Aware Machine Learning Model Selection in Cardiovascular Disease Prediction

Machine learning (ML) models are increasingly being used for cardiovascular disease prediction with model selection often guided by predictive performance measures such as accuracy. However, predictive accuracy per se may not adequately reflect the prediction reliability relevant to clinical decision support, especially when competing models differ in calibration, predictive uncertainty, confidence and decision reliability. The study proposes MASE-RAMR, a Multi-Criteria Adaptive Selection Engine with Reliability-Aware Model Ranking, to offer a wider basis for evaluation and ranking of cardiovascular disease prediction models. The proposed framework clusters heterogeneous evaluation metrics into four reliability-oriented meaningful dimensions: Accuracy, Calibration, Uncertainty, and Decision Reliability. Then, a hierarchical CRITIC-based weighting strategy is adopted, first the importance of the specific metrics is determined in each dimension and then the global importance of the resulting dimensions is estimated. The obtained adaptive criteria matrix is then evaluated by TOPSIS to compute Reliability Aware Model Ranking (RAMR) score for each candidate model. For the evaluation of the framework, six cardiovascular disease datasets were used and compared with the conventional CRITIC-TOPSIS model ranking approach with the same experimental setting. The comparison demonstrates two important behaviours of the proposed framework, decision stability, in which the same top-ranked model is retained for a number of datasets, and adaptive ranking, in which the proposed framework changes or refines the model rankings when the additional reliability dimensions provide discriminative information. Further analysis of traditional classifiers shows that the framework does not inherently favour complex ensemble or boosting models, as conventional classifiers achieve the highest reliability rankings on several datasets. These results show that MASE-RAMR provides a dataset-dependent and reliability-aware strategy for ML model selection beyond predictive accuracy alone, providing a more comprehensive basis for identifying candidate models for clinical decision support.

Cardiovascular disease prediction; Machine learning; Reliability-aware model selection; Predictive uncertainty

1. Alba, A. C., Agoritsas, T., Walsh, M., Hanna, S., Iorio, A., Devereaux, P. J., McGinn, T., & Guyatt, G. (2017). Discrimination and calibration of clinical prediction models: Users' guides to the medical literature. JAMA, 318(14), 1377–1384. https://doi.org/10.1001/jama.2017.12126 [Google Scholar] [Crossref]

2. Huang, Y., Li, W., Macheret, F., Gabriel, R. A., & Ohno-Machado, L. (2020). A tutorial on calibration measurements and calibration models for clinical prediction models. Journal of the American Medical Informatics Association, 27(4), 621–633. https://doi.org/10.1093/jamia/ocz228 [Google Scholar] [Crossref]

3. Huang, L., Ruan, S., Xing, Y., & Feng, M. (2024). A review of uncertainty quantification in medical image analysis: Probabilistic and non-probabilistic methods. Medical Image Analysis, 97, 103223. https://doi.org/10.1016/j.media.2024.103223 [Google Scholar] [Crossref]

4. Kompa, B., Snoek, J., & Beam, A. L. (2021). Second opinion needed: Communicating uncertainty in medical machine learning. npj Digital Medicine, 4, 4. https://doi.org/10.1038/s41746-020-00367-3 [Google Scholar] [Crossref]

5. Seoni, S., Jahmunah, V., Salvi, M., Barua, P. D., Molinari, F., & Acharya, U. R. (2023). Application of uncertainty quantification to artificial intelligence in healthcare: A review of last decade (2013–2023). Computers in Biology and Medicine, 165, 107441. https://doi.org/10.1016/j.compbiomed.2023.107441 [Google Scholar] [Crossref]

6. Van Calster, B., McLernon, D. J., van Smeden, M., Wynants, L., & Steyerberg, E. W. (2019). Calibration: The Achilles heel of predictive analytics. BMC Medicine, 17, 230. https://doi.org/10.1186/s12916-019-1466-7 [Google Scholar] [Crossref]

7. Liu, T., Krentz, A., Lu, L., & Curcin, V. (2024). Machine learning based prediction models for cardiovascular disease risk using electronic health records data: Systematic review and meta-analysis. European Heart Journal – Digital Health, 6(1), 7–22. https://doi.org/10.1093/ehjdh/ztae080 [Google Scholar] [Crossref]

8. Alemu, Y. M., Alemu, S. M., Bagheri, N., Wangdi, K., & Chateau, D. (2025). Discrimination and calibration performances of non-laboratory-based and laboratory-based cardiovascular risk predictions: A systematic review. Open Heart, 12(1), e003147. [Google Scholar] [Crossref]

9. Diakoulaki, D., Mavrotas, G., & Papayannakis, L. (1995). Determining objective weights in multiple criteria problems: The CRITIC method. Computers & Operations Research, 22(7), 763–770. https://doi.org/10.1016/0305-0548(94)00059-H [Google Scholar] [Crossref]

10. Hwang, C.-L., & Yoon, K. (1981). Multiple attribute decision making: Methods and applications: A state-of-the-art survey. Springer. https://doi.org/10.1007/978-3-642-48318-9 [Google Scholar] [Crossref]

11. 11.Ali, R., Lee, S., & Chung, T. C. (2017). Accurate multi-criteria decision making methodology for recommending machine learning algorithm. Expert Systems with Applications, 71, 257–278. https://doi.org/10.1016/j.eswa.2016.11.034 [Google Scholar] [Crossref]

12. Leyva-Vázquez, M. Y., Briones Peñafiel, L. A., Sanchez Muñoz, S. X., & Quiroz Martinez, M. A. (2021). A framework for selecting machine learning models using TOPSIS. In T. Ahram (Ed.), Advances in artificial intelligence, software and systems engineering (pp. 119–126). Springer. https://doi.org/10.1007/978-3-030-51328-3_18 [Google Scholar] [Crossref]

13. Ganie, S. M., Dutta Pramanik, P. K., & Zhao, Z. (2025). Enhanced and interpretable prediction of multiple cancer types using a stacking ensemble approach with SHAP analysis. Bioengineering, 12(5), 472. https://doi.org/10.3390/bioengineering12050472 [Google Scholar] [Crossref]

14. Brier, G. W. (1950). Verification of forecasts expressed in terms of probability. Monthly Weather Review, 78(1), 1–3. https://doi.org/10.1175/1520-0493(1950)078<0001:VOFEIT>2.0.CO;2 [Google Scholar] [Crossref]

15. Collins, G. S., Moons, K. G. M., Dhiman, P., Riley, R. D., Beam, A. L., Van Calster, B., Ghassemi, M., Liu, X., Reitsma, J. B., van Smeden, M., Boulesteix, A.-L., Camaradou, J. C., Celi, L. A., Denaxas, S., Denniston, A. K., Glocker, B., Golub, R. M., Harvey, H., Heinze, G., ... Logullo, P. (2024). TRIPOD+AI statement: Updated guidance for reporting clinical prediction models that use regression or machine learning methods. BMJ, 385, e078378. https://doi.org/10.1136/bmj-2023-078378 [Google Scholar] [Crossref]

16. Moons, K. G. M., Damen, J. A. A., Kaul, T., Hooft, L., Andaur Navarro, C., Dhiman, P., Beam, A. L., Van Calster, B., Celi, L. A., Denaxas, S., Denniston, A. K., Ghassemi, M., Heinze, G., Kengne, A. P., Maier-Hein, L., Liu, X., Logullo, P., McCradden, M. D., Oakden-Rayner, L., ... van Smeden, M. (2025). PROBAST+AI: An updated quality, risk of bias, and applicability assessment tool for prediction models using regression or artificial intelligence methods. BMJ, 388, e082505. https://doi.org/10.1136/bmj-2024-082505 [Google Scholar] [Crossref]

17. Ojeda, F. M., Jansen, M. L., Thiéry, A., Blankenberg, S., Weimar, C., Schmid, M., & Ziegler, A. (2023). Calibrating machine learning approaches for probability estimation: A comprehensive comparison. Statistics in Medicine, 42(29), 5451–5478. https://doi.org/10.1002/sim.9921 [Google Scholar] [Crossref]

18. Zargarzadeh, A., Javanshir, E., Ghaffari, A., Mosharkesh, E., & Anari, B. (2023). Artificial intelligence in cardiovascular medicine: An updated review of the literature. Journal of Cardiovascular and Thoracic Research, 15(4), 204–209. https://doi.org/10.34172/jcvtr.2023.33031 [Google Scholar] [Crossref]

19. X. Sun, Y. Yin, Q. Yang, and T. Huo, “Artificial intelligence in cardiovascular diseases: diagnostic and therapeutic perspectives,” European Journal of Medical Research, vol. 28, no. 1, p. 242, 2023, doi: 10.1186/s40001-023-01065-y. [Google Scholar] [Crossref]

20. Steyerberg, E. W., Vickers, A. J., Cook, N. R., Gerds, T., Gonen, M., Obuchowski, N., Pencina, M. J., & Kattan, M. W. (2010). Assessing the performance of prediction models: A framework for traditional and novel measures. Epidemiology, 21(1), 128–138. https://doi.org/10.1097/EDE.0b013e3181c30fb2 [Google Scholar] [Crossref]

• A Machine Learning Model for Predicting the Risk of Developing Diabetes - T2DM Using Real-World Data from Kilifi, Kenya
• AI-Powered Facial Recognition Attendance System Using Deep Learning and Computer Vision
• A Comprehensive Review on Brain Tumour Segmentation Using Deep Learning Approach
• A Scalable Retrieval-Augmented Generation Pipeline for Domain-Specific Knowledge Applications
• Predictive Maintenance in Semiconductor Manufacturing Using Machine Learning on Imbalanced Dataset