Developing the Generative Artificial Intelligence Trustworthiness and Accuracy Index (GAITAI): A Multidimensional Framework for Evaluating Large Language Models

The rapid advancement of generative artificial intelligence (GenAI) and large language models (LLMs) has transformed sectors such as healthcare, education, finance, law, and public administration. Despite significant improvements in computational capability, concerns regarding hallucinations, misinformation, inconsistency, bias, opacity, and governance have exposed important limitations in existing benchmark-based evaluation methods. Current frameworks primarily assess task performance and reasoning ability but provide limited insight into the broader characteristics that determine whether AI systems can be trusted in high-stakes environments. This study addresses this gap by developing the Generative Artificial Intelligence Trustworthiness and Accuracy Index (GAITAI), a multidimensional conceptual framework for evaluating AI trustworthiness beyond traditional benchmark metrics.
Using a conceptual research design, the study integrates Information Quality Theory, Trust Theory, the Technology Acceptance Model, Measurement Theory, Composite Index Theory, Artificial Intelligence Governance Theory, and Institutional Theory to establish an interdisciplinary foundation for trustworthy AI evaluation. GAITAI conceptualises Generative Artificial Intelligence Trustworthiness as a second-order latent construct comprising four first-order dimensions: Accuracy, Reliability, Robustness, and Transparency, operationalised through twenty-eight observable indicators.
The study contributes theoretically by redefining AI trustworthiness as a multidimensional construct and methodologically by extending psychometric measurement and composite indicator principles to AI evaluation. Practically, GAITAI provides organisations, AI developers, policymakers, and regulators with a structured framework for assessing AI trustworthiness and supporting responsible AI governance. Although conceptual, the framework establishes a foundation for future empirical validation, comparative benchmarking, and international standardisation. GAITAI advances the development of scientifically rigorous methods for evaluating trustworthy generative artificial intelligence and supports evidence-based AI governance across diverse organisational and societal contexts.

Generative artificial intelligence; Large language models; Trustworthy artificial intelligence

1. Acevedo, S., LaFramboise, N., & Wong, J. (2017). Caribbean tourism in the global marketplace. In Unleashing growth and strengthening resilience in the Caribbean. International Monetary Fund. [Google Scholar] [Crossref]

2. Bass, B. M. (1985). Leadership and performance beyond expectations. Free Press. [Google Scholar] [Crossref]

3. Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021). On the dangers of stochastic parrots: Can language models be too big? Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, 610–623. https://doi.org/10.1145/3442188.3445922 [Google Scholar] [Crossref]

4. Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., Brynjolfsson, E., et al. (2021). On the opportunities and risks of foundation models (CRFM Report). Stanford University. [Google Scholar] [Crossref]

5. Borsboom, D. (2005). Measuring the mind: Conceptual issues in contemporary psychometrics. Cambridge University Press. [Google Scholar] [Crossref]

6. Committee on Publication Ethics. (2023). COPE position statement: Authorship and AI tools. https://publicationethics.org [Google Scholar] [Crossref]

7. Cronbach, L. J., & Meehl, P. E. (1955). Construct validity in psychological tests. Psychological Bulletin, 52(4), 281–302. https://doi.org/10.1037/h0040957 [Google Scholar] [Crossref]

8. Davis, F. D. (1989). Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Quarterly, 13(3), 319–340. https://doi.org/10.2307/249008 [Google Scholar] [Crossref]

9. DeVellis, R. F. (2017). Scale development: Theory and applications (4th ed.). Sage Publications. [Google Scholar] [Crossref]

10. DiMaggio, P. J., & Powell, W. W. (1983). The iron cage revisited: Institutional isomorphism and collective rationality in organisational fields. American Sociological Review, 48(2), 147–160. [Google Scholar] [Crossref]

11. European Parliament and Council. (2024). Regulation (EU) 2024/1689 laying down harmonised rules on artificial intelligence (Artificial Intelligence Act). Official Journal of the European Union. [Google Scholar] [Crossref]

12. European Union. (2024). Regulation (EU) 2024/1689 laying down harmonised rules on artificial intelligence (Artificial Intelligence Act). Official Journal of the European Union. [Google Scholar] [Crossref]

13. Floridi, L., & Cowls, J. (2019). A unified framework of five principles for AI in society. Harvard Data Science Review, 1(1). https://doi.org/10.1162/99608f92.8cd550d1 [Google Scholar] [Crossref]

14. Fornell, C., & Larcker, D. F. (1981). Evaluating structural equation models with unobservable variables and measurement error. Journal of Marketing Research, 18(1), 39–50. [Google Scholar] [Crossref]

15. Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep learning. MIT Press. [Google Scholar] [Crossref]

16. Hair, J. F., Black, W. C., Babin, B. J., & Anderson, R. E. (2022). Multivariate data analysis (9th ed.). Cengage Learning. [Google Scholar] [Crossref]

17. ISO/IEC. (2023). ISO/IEC 42001: Artificial intelligence—Management system. International Organisation for Standardization. [Google Scholar] [Crossref]

18. ISO/IEC. (2023). ISO/IEC 42001:2023 Artificial intelligence—Management system. International Organisation for Standardization. [Google Scholar] [Crossref]

19. Jaakkola, E. (2020). Designing conceptual articles: Four approaches. AMS Review, 10(1–2), 18–26. [Google Scholar] [Crossref]

20. Jobin, A., Ienca, M., & Vayena, E. (2019). The global landscape of AI ethics guidelines. Nature Machine Intelligence, 1(9), 389–399. [Google Scholar] [Crossref]

21. Kaplan, R. M., & Saccuzzo, D. P. (2018). Psychological testing: Principles, applications, and issues (9th ed.). Cengage. [Google Scholar] [Crossref]

22. Kline, R. B. (2023). Principles and practice of structural equation modeling (5th ed.). Guilford Press. [Google Scholar] [Crossref]

23. Liang, P., Bommasani, R., Lee, T., Tsipras, D., et al. (2023). Holistic evaluation of language models (HELM). Stanford Center for Research on Foundation Models. [Google Scholar] [Crossref]

24. MacKenzie, S. B., Podsakoff, P. M., & Podsakoff, N. P. (2011). Construct measurement and validation procedures in MIS and behavioural research. MIS Quarterly, 35(2), 293–334. [Google Scholar] [Crossref]

25. Mayer, R. C., Davis, J. H., & Schoorman, F. D. (1995). An integrative model of organisational trust. Academy of Management Review, 20(3), 709–734. [Google Scholar] [Crossref]

26. Messick, S. (1995). Validity of psychological assessment: Validation of inferences from persons' responses and performances. American Psychologist, 50(9), 741–749. [Google Scholar] [Crossref]

27. National Institute of Standards and Technology. (2023). Artificial Intelligence Risk Management Framework (AI RMF 1.0). U.S. Department of Commerce. [Google Scholar] [Crossref]

28. National Institute of Standards and Technology. (2023). Artificial Intelligence Risk Management Framework (AI RMF 1.0). U.S. Department of Commerce. [Google Scholar] [Crossref]

29. Nunnally, J. C., & Bernstein, I. H. (1994). Psychometric theory (3rd ed.). McGraw-Hill. [Google Scholar] [Crossref]

30. OECD & Joint Research Centre. (2008). Handbook on constructing composite indicators: Methodology and user guide. OECD Publishing. [Google Scholar] [Crossref]

31. OECD. (2019). OECD principles on artificial intelligence. Organisation for Economic Co-operation and Development. [Google Scholar] [Crossref]

32. OpenAI. (2023). GPT-4 technical report. arXiv. https://arxiv.org/abs/2303.08774 [Google Scholar] [Crossref]

33. OpenAI. (2023). GPT-4 technical report. arXiv. https://arxiv.org/abs/2303.08774 [Google Scholar] [Crossref]

34. Organisation for Economic Co-operation and Development. (2019). OECD principles on artificial intelligence. OECD Publishing. [Google Scholar] [Crossref]

35. Raji, I. D., Smart, A., White, R. N., Mitchell, M., Gebru, T., Hutchinson, B., Smith-Loud, J., Theron, D., & Barnes, P. (2020). Closing the AI accountability gap. Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, 33–44. [Google Scholar] [Crossref]

36. Russell, S., & Norvig, P. (2021). Artificial intelligence: A modern approach (4th ed.). Pearson. [Google Scholar] [Crossref]

37. Sammut, C., & Webb, G. I. (Eds.). (2017). Encyclopedia of machine learning and data mining (2nd ed.). Springer. [Google Scholar] [Crossref]

38. Scott, W. R. (2014). Institutions and organizations: Ideas, interests, and identities (4th ed.). Sage Publications. [Google Scholar] [Crossref]

39. Topol, E. (2019). Deep medicine: How artificial intelligence can make healthcare human again. Basic Books. [Google Scholar] [Crossref]

40. UNESCO. (2021). Recommendation on the ethics of artificial intelligence. United Nations Educational, Scientific and Cultural Organization. [Google Scholar] [Crossref]

41. UNESCO. (2023). Guidance for generative AI in education and research. UNESCO. [Google Scholar] [Crossref]

42. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention is all you need. Advances in Neural Information Processing Systems, 30. [Google Scholar] [Crossref]

43. Venkatesh, V., & Davis, F. D. (2000). A theoretical extension of the Technology Acceptance Model: Four longitudinal field studies. Management Science, 46(2), 186–204. [Google Scholar] [Crossref]

44. Wang, R. Y., & Strong, D. M. (1996). Beyond accuracy: What data quality means to data consumers. Journal of Management Information Systems, 12(4), 5–33. [Google Scholar] [Crossref]

45. Weidinger, L., Mellor, J., Rauh, M., Griffin, C., et al. (2022). Taxonomy of risks posed by language models. Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, 214–229. [Google Scholar] [Crossref]

46. Whetten, D. A. (1989). What constitutes a theoretical contribution? Academy of Management Review, 14(4), 490–495. [Google Scholar] [Crossref]

47. WHO. (2021). Ethics and governance of artificial intelligence for health. World Health Organization. [Google Scholar] [Crossref]

48. World Economic Forum. (2024). The future of global financial services in the age of artificial intelligence. World Economic Forum. [Google Scholar] [Crossref]

49. World Health Organization. (2021). Ethics and governance of artificial intelligence for health. World Health Organization. [Google Scholar] [Crossref]

50. Zadeh, L. A. (1965). Fuzzy sets. Information and Control, 8(3), 338–353. [Google Scholar] [Crossref]

• The Role of Artificial Intelligence in Revolutionizing Library Services in Nairobi: Ethical Implications and Future Trends in User Interaction
• ESPYREAL: A Mobile Based Multi-Currency Identifier for Visually Impaired Individuals Using Convolutional Neural Network
• Comparative Analysis of AI-Driven IoT-Based Smart Agriculture Platforms with Blockchain-Enabled Marketplaces
• AI-Based Dish Recommender System for Reducing Fruit Waste through Spoilage Detection and Ripeness Assessment
• SEA-TALK: An AI-Powered Voice Translator and Southeast Asian Dialects Recognition