Corpus-Based Identification and Profiling of Vocabulary in Linguaskill Speaking Tests: A Methodological Framework for Adaptive Test Validation

Computer-adaptive tests such as Cambridge Linguaskill adjust item difficulty dynamically to match test-taker proficiency, and their speaking components are now used for high-stakes admission and exit decisions, including in Malaysian higher education. Yet little research has examined the vocabulary load and lexical demands of adaptive speaking assessment, leaving open whether the vocabulary elicited at each reported level is congruent with the Common European Framework of Reference (CEFR) levels the scores claim to represent. This article develops a complete corpus-based methodological framework for identifying, classifying, and profiling the vocabulary of the Linguaskill Speaking Test. The framework specifies the compilation of a specialised corpus of transcribed candidate responses stratified across CEFR levels, supplemented by official test preparation materials, and a three-lens profiling procedure using LexTutor for BNC-COCA word-family analysis, the New General Service List for high-frequency lemma coverage, and Text Inspector for CEFR-aligned profiling against the English Vocabulary Profile. Operational criteria are defined for distinguishing core from peripheral vocabulary at each adaptive level, and decision rules are specified for classifying each level as aligned, over-demanding, or under-demanding relative to CEFR expectations, benchmarked against published lexical coverage thresholds for spoken English. The central hypothesis, motivated by prior coverage research, is that the lexical demands elicited by adaptive speaking tasks will not rise in step with claimed CEFR levels. The framework is grounded in an argument-based approach to validation and articulates the washback, fairness, and pedagogical stakes of the alignment question. The article contributes a replicable protocol for lexical validation of adaptive speaking tests and a benchmark synthesis for interpreting its future results.

Keywords

Linguaskill speaking vocabulary; corpus-based approach in vocabulary learning and vocabulary threshold in adaptive assessment

Downloads

PDF JATS XML

References

1. Adolphs, S., & Schmitt, N. (2003). Lexical coverage of spoken discourse. Applied Linguistics, 24(4), 425–438. https://doi.org/10.1093/applin/24.4.425 [Google Scholar] [Crossref]

2. Alderson, J. C., & Wall, D. (1993). Does washback exist? Applied Linguistics, 14(2), 115–129. https://doi.org/10.1093/applin/14.2.115 [Google Scholar] [Crossref]

3. Bax, S., Waller, D., & Nakatsuhara, F. (2019). Researching L2 writers' use of metadiscourse markers at intermediate and advanced levels. System, 83, 79–95. https://doi.org/10.1016/j.system.2019.02.010 [Google Scholar] [Crossref]

4. Boers, F., Eyckmans, J., Kappel, J., Stengers, H., & Demecheleer, M. (2006). Formulaic sequences and perceived oral proficiency: Putting a lexical approach to the test. Language Teaching Research, 10(3), 245–261. https://doi.org/10.1191/1362168806lr195oa [Google Scholar] [Crossref]

5. Browne, C. (2014). A new general service list: The better mousetrap we have been looking for? Vocabulary Learning and Instruction, 3(2), 1–10. https://doi.org/10.7820/vli.v03.2.browne [Google Scholar] [Crossref]

6. Browne, C., Culligan, B., & Phillips, J. (2013). The New General Service List. New General Service List Project. https://www.newgeneralservicelist.com [Google Scholar] [Crossref]

7. Cambridge University Press & Assessment. (2020). Linguaskill: Building a validity argument for the Speaking test. Cambridge Assessment English. [Google Scholar] [Crossref]

8. Chapelle, C. A., & Voss, E. (2016). 20 years of technology and language assessment in Language Learning & Technology. Language Learning & Technology, 20(2), 116–128. [Google Scholar] [Crossref]

9. Cobb, T. (2007). Computing the vocabulary demands of L2 reading. Language Learning & Technology, 11(3), 38–63. [Google Scholar] [Crossref]

10. Council of Europe. (2020). Common European Framework of Reference for Languages: Learning, teaching, assessment. Companion volume. Council of Europe Publishing. [Google Scholar] [Crossref]

11. Dang, T. N. Y., Coxhead, A., & Webb, S. (2017). The academic spoken word list. Language Learning, 67(4), 959–997. https://doi.org/10.1111/lang.12253 [Google Scholar] [Crossref]

12. de Jong, N. H., Steinel, M. P., Florijn, A. F., Schoonen, R., & Hulstijn, J. H. (2012). Facets of speaking proficiency. Studies in Second Language Acquisition, 34(1), 5–34. https://doi.org/10.1017/S0272263111000489 [Google Scholar] [Crossref]

13. Fan, J. (2014). Chinese test takers' attitudes towards the Versant English Test: A mixed-methods approach. Language Testing in Asia, 4(1), Article 6. https://doi.org/10.1186/s40468-014-0006-9 [Google Scholar] [Crossref]

14. González-Fernández, B., & Schmitt, N. (2020). Word knowledge: Exploring the relationships and order of acquisition of vocabulary knowledge components. Applied Linguistics, 41(4), 481–505. https://doi.org/10.1093/applin/amy057 [Google Scholar] [Crossref]

15. Ha, H. T. (2022). Lexical profile of newspapers revisited: A corpus-based analysis. Frontiers in Psychology, 13, Article 800983. https://doi.org/10.3389/fpsyg.2022.800983 [Google Scholar] [Crossref]

16. Hawkins, J. A., & Filipović, L. (2012). Criterial features in L2 English: Specifying the reference levels of the Common European Framework. Cambridge University Press. [Google Scholar] [Crossref]

17. Hougham, D., Clenton, J., & Uchihara, T. (2024). Disentangling the contributions of shorter vs. longer lexical bundles to L2 oral fluency. System, 121, Article 103243. https://doi.org/10.1016/j.system.2024.103243 [Google Scholar] [Crossref]

18. Hu, M., & Nation, I. S. P. (2000). Unknown vocabulary density and reading comprehension. Reading in a Foreign Language, 13(1), 403–430. [Google Scholar] [Crossref]

19. Kane, M. T. (2013). Validating the interpretations and uses of test scores. Journal of Educational Measurement, 50(1), 1–73. https://doi.org/10.1111/jedm.12000 [Google Scholar] [Crossref]

20. Koizumi, R., & In'nami, Y. (2013). Vocabulary knowledge and speaking proficiency among second language learners from novice to intermediate levels. Journal of Language Teaching and Research, 4(5), 900–913. https://doi.org/10.4304/jltr.4.5.900-913 [Google Scholar] [Crossref]

21. Kormos, J., & Dénes, M. (2004). Exploring measures and perceptions of fluency in the speech of second language learners. System, 32(2), 145–164. https://doi.org/10.1016/j.system.2004.01.001 [Google Scholar] [Crossref]

22. Kremmel, B., Indrarathne, B., Kormos, J., & Suzuki, S. (2023). Unknown vocabulary density and reading comprehension: Replicating Hu and Nation (2000). Language Learning, 73(4), 1127–1163. https://doi.org/10.1111/lang.12622 [Google Scholar] [Crossref]

23. Kyle, K., & Crossley, S. A. (2015). Automatically assessing lexical sophistication: Indices, tools, findings, and application. TESOL Quarterly, 49(4), 757–786. https://doi.org/10.1002/tesq.194 [Google Scholar] [Crossref]

24. Laufer, B., & Goldstein, Z. (2004). Testing vocabulary knowledge: Size, strength, and computer adaptiveness. Language Learning, 54(3), 399–436. https://doi.org/10.1111/j.0023-8333.2004.00260.x [Google Scholar] [Crossref]

25. Laufer, B., & Nation, I. S. P. (1999). A vocabulary-size test of controlled productive ability. Language Testing, 16(1), 33–51. https://doi.org/10.1177/026553229901600103 [Google Scholar] [Crossref]

26. Laufer, B., & Ravenhorst-Kalovski, G. C. (2010). Lexical threshold revisited: Lexical text coverage, learners' vocabulary size and reading comprehension. Reading in a Foreign Language, 22(1), 15–30. [Google Scholar] [Crossref]

27. Misbah, N. H., Mohamad, M., Yunus, M. M., & Ya'acob, A. (2017). Identifying the factors contributing to students' difficulties in the English language learning. Creative Education, 8(13), 1999–2008. https://doi.org/10.4236/ce.2017.813136 [Google Scholar] [Crossref]

28. Nation, I. S. P. (2006). How large a vocabulary is needed for reading and listening? The Canadian Modern Language Review, 63(1), 59–82. https://doi.org/10.3138/cmlr.63.1.59 [Google Scholar] [Crossref]

29. Nation, I. S. P. (2013). Learning vocabulary in another language (2nd ed.). Cambridge University Press. [Google Scholar] [Crossref]

30. Nation, I. S. P. (2017). The BNC/COCA word family lists. Victoria University of Wellington. [Google Scholar] [Crossref]

31. North, B. (2014). The CEFR in practice. Cambridge University Press. [Google Scholar] [Crossref]

32. Read, J. (2000). Assessing vocabulary. Cambridge University Press. [Google Scholar] [Crossref]

33. Saito, K., Uchihara, T., Takizawa, K., & Suzukida, Y. (2025). Individual differences in L2 listening proficiency revisited: Roles of form, meaning, and use aspects of phonological vocabulary knowledge. Studies in Second Language Acquisition, 47(1), 26–52. https://doi.org/10.1017/S0272263124000575 [Google Scholar] [Crossref]

34. Schmitt, N. (2008). Instructed second language vocabulary learning. Language Teaching Research, 12(3), 329–363. https://doi.org/10.1177/1362168808089921 [Google Scholar] [Crossref]

35. Schmitt, N., & Schmitt, D. (2014). A reassessment of frequency and vocabulary size in L2 vocabulary teaching. Language Teaching, 47(4), 484–503. https://doi.org/10.1017/S0261444812000018 [Google Scholar] [Crossref]

36. Schmitt, N., Jiang, X., & Grabe, W. (2011). The percentage of words known in a text and reading comprehension. The Modern Language Journal, 95(1), 26–43. https://doi.org/10.1111/j.1540-4781.2011.01146.x [Google Scholar] [Crossref]

37. Stæhr, L. S. (2009). Vocabulary knowledge and advanced listening comprehension in English as a foreign language. Studies in Second Language Acquisition, 31(4), 577–607. https://doi.org/10.1017/S0272263109990039 [Google Scholar] [Crossref]

38. Tan, A. W. L., & Goh, L. H. (2020). Comparing the effectiveness of direct vocabulary instruction and incidental vocabulary learning in improving the academic vocabulary of Malaysian tertiary students. Pertanika Journal of Social Sciences and Humanities, 28(S2), 263–279. [Google Scholar] [Crossref]

39. Uchihara, T., & Clenton, J. (2020). Investigating the role of vocabulary size in second language speaking ability. Language Teaching Research, 24(4), 540–556. https://doi.org/10.1177/1362168818799371 [Google Scholar] [Crossref]

40. Uchihara, T., & Saito, K. (2019). Exploring the relationship between productive vocabulary knowledge and second language oral ability. The Language Learning Journal, 47(1), 64–75. https://doi.org/10.1080/09571736.2016.1191527 [Google Scholar] [Crossref]

41. van Zeeland, H., & Schmitt, N. (2013). Lexical coverage in L1 and L2 listening comprehension: The same or different from reading comprehension? Applied Linguistics, 34(4), 457–479. https://doi.org/10.1093/applin/ams074 [Google Scholar] [Crossref]

42. Webb, S., & Rodgers, M. P. H. (2009a). Vocabulary demands of television programs. Language Learning, 59(2), 335–366. https://doi.org/10.1111/j.1467-9922.2009.00509.x [Google Scholar] [Crossref]

43. Webb, S., & Rodgers, M. P. H. (2009b). The lexical coverage of movies. Applied Linguistics, 30(3), 407–427. https://doi.org/10.1093/applin/amp010 [Google Scholar] [Crossref]

44. Weir, C. J. (2005). Language testing and validation: An evidence-based approach. Palgrave Macmillan. [Google Scholar] [Crossref]

45. Wong, A. S. C., Lee, J. Y. V., Fung, M. E., & Willibrord, O. (2019). Vocabulary size of Malaysian secondary school students. Akademi Pengajian Bahasa, Universiti Teknologi MARA. [Google Scholar] [Crossref]

46. Zhang, S., & Zhang, X. (2022). The relationship between vocabulary knowledge and L2 reading/listening comprehension: A meta-analysis. Language Teaching Research, 26(4), 696–725. https://doi.org/10.1177/1362168820913998 [Google Scholar] [Crossref]

Corpus-Based Identification and Profiling of Vocabulary in Linguaskill Speaking Tests: A Methodological Framework for Adaptive Test Validation

Authors

Article Information

Publication Timeline

Abstract

Keywords

Downloads

References

Metrics

Views & Downloads

Similar Articles