Measuring the Validity of a High-stakes English Test in Bangladesh: An Empirical Approach

Measuring the Validity of a High-stakes English Test in Bangladesh: An Empirical Approach

Iftakhar Ahmed¹, Dr. Liton Baron Sikder²

¹Department of English, Mawlana Bhashani Science and Technology University, Santosh, Tangail-1902, Dhaka, Bangladesh.

²Department of English, Islamic University, Kushtia, Bangladesh.

DOI: https://doi.org/10.47772/IJRISS.2023.70603

Received: 02 May 2023; Revised: 17 May 2023; Accepted: 22 May 2023; Published: 16 June 2023

ABSTRACT

The SSC English examination is a high-stakes test in Bangladesh because its scores are used to make decisions about the admission, placement, or graduation of students. This study evaluated the empirical validity of the SSC English examination using the correlation coefficient formula by Pearson’s R. The researchers employed a quantitative approach to conducting the research. Data were collected from eight education boards located in Dhaka, Rajshahi, Barisal, Chattogram, Comilla, Dinajpur, Jessore, and Sylhet. The present study used “Simple Random Sampling” while selecting the respondents. The respondents of the study were eighty SSC students. The researcher selected five students from each school. The researcher chose two schools from each education board; one is from a rural area, and another is from an urban area. The respondents were balanced between urban and rural, male and female persons.  The findings of the study revealed that the validity of the SSC English test of each education board was subpar. Based on the findings, some recommendations were put forward to improve the SSC English examination in Bangladesh. Notably, the SSC English test items should examine the learners’ cognitive skills, such as thinking, remembering, and reasoning. The question setter should follow the authorized textbook to make a final question to ensure the validity of the test. Moreover, according to the guidelines of the NCTB (2012), candidates’ speaking and listening skills should be measured through continuous assessment. Furthermore, continuous assessment marks could be added to finalize the candidates’ GPA (Grade Point Average) in the summative assessment. The authority ought to train the scorers. The education board should allow the scorers enough time to mark the scripts contentedly to ensure the pragmatism of the test. Moreover, the teachers should teach to the curriculum, not to the test.

Keywords: English examination, the SSC level, Pearson’s r, correlation coefficient

INTRODUCTION

The English language is used as a second or foreign language in Bangladesh. Students take English language courses as compulsory subjects from class one to class twelve. The students participate in the Secondary School Certificate (SSC) examination after Grades 9 and 10. We can consider the SSC English examination as a high-stakes test in Bangladesh due to its nature. The SSC exam result decides the future career of many students. The SSC English examination is divided into English Paper-I (English for Today) and English Paper-II (English Grammar and Composition). The candidates complete these two tests on two different days. The scores collected from English (both Paper I & Paper II) and other subjects are used to finalize a student’s overall Grade Point Average (GPA). This study endeavors to measure the statistical or empirical validity of the SSC English examination in Bangladesh. Hughes points out that language tests are considered valid if they measure what they are intended to measure (Hughes, 2003). The term “empirical validity” (also known as “statistical validity” or “predictive validity”) refers to how closely test results “correlate” with behavior as it is measured in other circumstances. Empirical validity happens when researchers make planned comparisons to see if a measurement provides scores that relate to the chosen criterion (Patten & Newhart, 2017).

The SSC English examination, a large-scale test, plays a vital role in developing a nation. In 2021, almost two million students sat for the SSC test in Bangladesh (“SSC exams”, 2021). Therefore, it is a dire need to scrutinize the nature of the SSC English test in Bangladesh. Unfortunately, the researchers found no empirical or statistical study related to the SSC English examination in Bangladesh. Most of the studies related to the SSC English test in Bangladesh were qualitative. That is why the present researchers have enthusiastically scrutinized the empirical or statistical validity of the SSC English test in Bangladesh.

The objective of this study is to measure the empirical or statistical validity of the SSC English test in Bangladesh following the correlation coefficient formula by Pearson’s R. Notably, the SSC English examination is conducted by several education boards in Bangladesh, such as Dhaka Education Board, Rajshahi Education Board, Barisal Education Board, Chattogram Education Board, Comilla Education Board, Dinajpur Education Board, Jessore Education Board, and Sylhet Education Board. Therefore, the researchers aimed to measure the statistical or empirical validity of all the education boards in Bangladesh.

A significant number of qualitative studies related to the SSC English test in Bangladesh have been conducted. As per Jamila and Kabir (2020), the SSC test results might control a student’s overall educational career. The education board in Bangladesh took the initiative to implement the Communicative Language Teaching (CLT) approach instead of the Grammar Translation Method (GTM) in the late 1990s. The SSC English final test did not examine the speaking and listening skills, two crucial elements of CLT. The test items questioned the face validity of this high-stakes examination (Jamila & Kabir, 2020). Formative and summative testing and assessments were incompatible with the communicative approach (Das et al., 2014). Moreover, the studies conducted by several investigators (Kabir, 2015; Billah, 2015; Mariyam et al., 2015; Alam & Kabir, 2015; Jamila & Kabir, 2020) also questioned the nature of EFL testing and assessment at the secondary and higher secondary levels in Bangladesh. Alam and Kabir (2015) opined that the SSC English language testing system in Bangladesh was faulty due to the predictable examination contents and inconsistent assessment techniques. That is why the SSC English examination in Bangladesh was known for lacking validity and reliability (Alam & Kabir, 2015). Ali et al. (2018) investigated the HSC English examination in Bangladesh. They found that the English examination was faulty as the test did not follow the cornerstones of test development. The studies mentioned above disclose the research gaps related to the present study. The present researchers found that most studies related to the SSC English test were qualitative. Quantitative studies were not available. That is why the present researchers evaluated the empirical validity of the SSC English test in Bangladesh.

METHODOLOGY

Sampling

The researchers collected data from eight education boards in Bangladesh. The present study used “Simple Random Sampling” while selecting the respondents. The subjects of this study included 80 SSC students. First, the researchers chose two schools from each education board; one is from a rural area, and another is from an urban area. Next, the researchers selected five students from each school. The researchers selected ten students from each board using the “Simple Random Sampling method”. They were believed to represent the largest population of secondary education in Bangladesh. The researchers personally conducted the survey. Data were balanced between urban and rural, male and female participants.

Materials

Several authentic materials were used to conduct this study, such as the SSC English textbook prescribed by NCTB (2012) and board question papers of the SSC English examination held in 2019. The present study investigated English for Today (Paper I) and English Grammar and Composition (Paper II) for classes 9-10 to scrutinize whether the textbook corresponded to the board questions. This study scrutinized the nature, contents, and characteristics of the SSC English questions of 2019 (Both Paper I and Paper II) collected from all of the eight education boards in Bangladesh (Appendix 10A, 10B; Only Dhaka board questions are shown in the Appendix section; Question papers of other education boards are intentionally skipped in the Appendix section). 

Tests on the Board Questions and a Standardized Question for the Empirical Validity

The present researchers calculated the statistical or empirical validity of the SSC English test measuring the ‘correlation coefficient’ between two sets of scores. It is a statistical measurement of resemblance. As per Hughes (2003), the standard procedure is to compare two sets of scores for the ‘correlation coefficient’ measurement. The researcher administered tests on the SSC English board questions (both Paper I and Paper II) of 2019 and a standardized question covering both Paper I and Paper II (Appendix 9A, 9B). The researchers selected those students as participants who had finished grades 9 and 10.

A Standardized question (both Paper I and Paper II) was established per the recommendations of several SSC English instructors from different education boards in Bangladesh. Moreover, the standardized question was developed following the NCTB (2012) guidelines. The researchers established the same standardized question for the participants from the eight education boards in Bangladesh. Ten participants from each education board took the test on their corresponding board questions. They also took the test on standardized questions. Thus, the researcher got two sets of scores for each student. The researcher selected those students who had almost finished two academic years (Class 9 and Class 10). Thus, the statistical validity of each board was calculated following the measurement of the ‘correlation coefficient’. Many statisticians (Everitt & Skrondal, 2010; Lindstrom, 2010; Glen, 2022) argue that finding a correlation coefficient for the two sets of data is one of the most usual ways to find a correlation between the two tests. The nature of coefficients varies between 0 and 1, where:

• 1: perfect correlation,
• ≥ 0.9: excellent correlation,
• ≥ 0.8 < 0.9: good correlation,
• ≥ 0.7 < 0.8: acceptable correlation,
• ≥ 0.6 < 0.7: questionable correlation,
• ≥ 0.5 < 0.6: poor correlation,
• < 0.5: unacceptable correlation,
• 0: no correlation

The Statistical Validity or Empirical Validity Analysis Procedure

A test’s statistical or empirical validity can be calculated using the formula of validity coefficient (Hughes, 2003). Here, the SSC English test’s validity coefficient was calculated using the Pearson Correlation Coefficient formula, also known as Pearson’s r (Pearson, 1895).

The statistical validity of the SSC English examination was calculated using the following formula:

r_xy

Where,

r_xy = Validity Correlation Coefficient

n = Number of Participants (students)

x = Individual Scores Obtained from Board Test-2019

y = Individual Scores Obtained from a Standardized Test

The validity correlation coefficient was measured discretely for each education board. The researchers administered two tests for each education board to measure the validity correlation coefficient of the SSC English. One test was administered following the standardized question (Appendix 9A, 9B; both Paper I and Paper II), and the other was conducted following the corresponding SSC English board questions of 2019. Then the researchers considered two sets of numbers for each student to measure the validity coefficient.

FINDINGS AND DISCUSSION

Here, the validity coefficient of the SSC English test of each education board is discussed. Using Pearson’s r formula, each education board’s empirical or statistical validity coefficient was measured. The validity coefficient of the SSC English test of the Dhaka education board was found to be 0.55 (Appendix 1). According to Glen (2022), Everitt & Skrondal (2010), and Lindstrom (2010), it testified to poor validity of the test (≥ 0.5 < 0.6: poor correlation). The validity coefficient of the Rajshahi education board was 0.72 (Appendix 2), which was acceptable validity (≥ 0.7 < 0.8: acceptable correlation; (Everitt & Skrondal, 2010; Lindstrom, 2010; Glen, 2022). The researchers found that the validity coefficient of the Barisal education board was 0.47(Appendix 3). Glen (2022), Everitt & Skrondal (2010) and Lindstrom (2010) noted that this was unacceptable validity of any test (< 0.5: unacceptable correlation). The validity coefficient of the SSC English test of the Chattogram Education Board was 0.52 (Appendix 4), which indicates that the test’s validity was very poor (≥ 0.5 < 0.6: poor correlation; (Everitt & Skrondal, 2010; Lindstrom, 2010; Glen, 2022). The validity coefficient of the Comilla education board was 0.61 (Appendix 5). As per Glen (2022), Everitt & Skrondal (2010), and Lindstrom (2010), it was the indication of questionable validity of the SSC English test (≥ 0.6 < 0.7: questionable correlation). The validity coefficient of the Dinajpur education board was 0.68 (Appendix 6), which was also clear evidence of the SSC English test’s questionable validity (≥ 0.6 < 0.7: questionable correlation). As per the findings, the validity coefficient of the Jessore education board was 0.69 (Appendix 7), which was also evidence of questionable validity of the SSC English test (≥ 0.6 < 0.7: questionable correlation) (Everitt & Skrondal, 2010; Lindstrom, 2010; Glen, 2022). Sylhet education board’s validity coefficient was 0.44 (Appendix 8). It was the demonstration of the SSC English test’s unacceptable validity. Glen (2022), Everitt & Skrondal (2010), and Lindstrom (2010) opined that this was unacceptable validity of any test (< 0.5: unacceptable correlation). The findings clarified the low validity coefficient of the Dhaka and Chittagong education board English test. The validity of the Barisal and Sylhet education board was appalling, as the validity coefficient values of these two boards were lower than 0.5. It was evident that the validity of the Comilla, Dinajpur, and Jessore was also low. The Rajshahi education board only possessed acceptable validity. So, the teachers, students, guardians, and staff of each education board have to pay attention to the SSC English test items and curriculum objectives to enhance the validity of the test.

RECOMMENDATIONS

The Education Board

Based on the findings, some recommendations might be made to improve the SSC English examination in Bangladesh. Especially the education board has to pay attention to the educational facilities of marginalized students to confirm educational equality and impartiality. The English reading and writing items of the test should be congruent with the board textbook’s contents. Moreover, the formative and summative tests should assess the learners’ listening and speaking skills to confirm the test’s content validity. As per the suggestions of the NCTB (2012), students’ speaking and listening skills must be evaluated through continuous assessment, such as tutorial tests, sudden tests, and quizzes. Besides, twenty marks, ten for listening and ten for speaking, must be added to determine the candidates’ final grade in the final assessment, also known as SSC English final test. So far, the summative examination has ignored the listening and speaking items. Therefore, the education board should include these two items in the test in a prompt manner. The authority might ensure a suitable examination center for the SSC candidates. Sometimes, the SSC English questions might have typographical errors. The education board should train the examiners so that the examiners mark the subjective items in a befitting manner. The SSC English teachers should engage themselves in pre-service and in-service training. Moreover, the education board must allow the markers enough time to mark the scripts. Additionally, more objective items, such as MCQ, should be given in the test to evade accidental subjective biases in scoring. The SSC candidates living in isolated areas and hill tracts might not have access to smartphones and laptops. The test-related new information has to be disseminated via tv channels, websites, and newspapers so the students can avail of that information promptly. Additionally, the authority must ensure the security of the SSC English examination in Bangladesh. Pathan (2015) argued that leaked question papers would do no good to Bangladesh. Hackers always attempt to get the question papers. So cyber security should be confirmed. Question editing, typing, photocopying, and transporting have to be carried out with utmost care and responsibility.

The Question Setters

The question setter ought to follow the authorized textbook (NCTB, 2012) to make a final question to confirm the validity of the SSC English test. To augment the expediency of the SSC English test, the test items should be set with the determination that the candidates must have enough cognitive skills (such as thoughtful, recollection, and perception) to answer the test items. Moreover, the question setter should avoid the test items that do not distinguish between poorer and stronger candidates to enhance the test’s consistency. It was found that both the SSC English examination papers I and II contained many typical items. In addition, question setters occasionally assign items with ambiguous meanings. According to Hughes (2003), the question setters must comprise independent passages in the question paper. Additionally, the SSC English test has the propensity to give applicants a choice of questions. Sometimes, the SSC students are asked to answer a question from many options. The test’s consistency is likely to suffer due to such a procedure. Therefore, the range of possible answers should be controlled. Furthermore, the question setter should incorporate more authentic test items in the formative and summative tests. It is recommended that the question setter should stay away from scripted items. Finally, the formative assessment should espouse more authentic materials like realia to evaluate the listening and speaking skills of the SSC candidates.

The SSC English Teachers

The authority should organize pre-service and in-service training programs for the SSC English teachers so that they can assist the students in achieving communicative competence in all the fundamental language skills—listening, speaking, reading, and writing. The EFL learning of the SSC learners should be compatible with the learning objectives of the SSC curriculum proposed by NCTB (2012). Since the teaching-learning process is dynamic, in-service training is essential for a teacher’s professional development. It could be professional or casual. In Bangladesh, whether a person studies or not, once a teacher, always a teacher. Therefore, after two or three years, the authority should decide whether or not to reappoint him as a teacher. Asaya (1991) noted that in-service education programs could be categorized into different forms: seminars, educational tours, refresher courses, workshops, conferences, correspondence education, in-service training, radio broadcast, film shows, T.V programs, exchange of teachers, academic staff meetings held once a month, etc. A CLT-based curriculum must include instructors who have completed the required training. The reason for this is that a CLT-based curriculum prioritizes “how to teach” than “what to teach” (NCTB, 2012). So whenever there is a chance, teachers should participate in educational training.

The SSC Students

The SSC candidates should practice the listening and speaking items from the authorized book (NCTB, English for Today, 2022). Moreover, the learners can utilize authentic materials beyond the textbook. Interactive exercises should be conducted between teachers and students and, more importantly, between students (NCTB, 2012). The SSC learners ought to cover the whole textbook. They can practice speaking items using realia. Furthermore, teachers and students should follow the curriculum objectives (NCTB, 2012). During the two academic years (classes nine and ten), teachers and students should use teaching and learning materials collected from primary sources. Authentic materials as learning tools are better than scripted materials. The teachers must make the SSC learners aware of alternative assessment techniques, such as self-assessment, peer assessment, task-based assessment, portfolio assessment, projects, drafts, artworks, presentations, seminars, workshops, and so on. These assessments measure the students’ communicative, grammatical, and cognitive competence. The students should actively learn through peer work, group work, role-plays, and simulations. They should use authentic learning materials compatible with curriculum objectives and the demands of the modern world (Maniruzzaman, 2016; NCLRC, 2004).

The Guardians

It is regretful that some guardians in Bangladesh help their kids get the leaked question papers (Rahman & Afrin, 2017). The morality of the many guardians is questionable as they allow their children to access the leaked questions. Moreover, some guardians always try to amass leaked questions. Instead, they must emphasize how to enhance the strong morals of their kids rather than academic results.

FUTURE RESEARCH

It can be pointed out that future research can be conducted on the testing and assessment procedures of Madrasa education boards in Bangladesh. This research uses Pearson’s r formula to measure the empirical validity of the SSC English test. Future researchers may measure the reliability of the SSC English test using the split-half method by the Spearman-Brown formula. Additionally, future researchers may use more techniques, such as Kuder-Richardson Formula 21 and the Rasch model, to evaluate the consistency of the SSC English test. Future research might deploy the test-retest method as well as action research to get a more accurate idea of the SSC English test and assessment. Moreover, factor analysis might be employed in construct validity measurement studies. There might be a comparative study between urban and rural SSC students in Bangladesh.

CONCLUSION

This research endeavored to evaluate the empirical validity of the SSC English test in Bangladesh. The subpar validity coefficient of each education board indicates poor teaching-learning practices at the SSC level in Bangladesh. It could be concluded that the validity of the SSC English examination depends on the test items. So, the test items should to congruent with the curriculum objectives (NCTB, 2012). Moreover, the trained scorers should check the test scripts, as most of the SSC English test items are subjective. The scorers should be allowed sufficient time to check the scripts in a befitting manner. The formative assessment could be implemented to evaluate the SSC English candidates’ listening and speaking skills. The scores obtained from the formative assessment throughout the two academic years (class 9 and class 10) must be added to fix the learners’ final grades in the summative assessment. The education board should appoint more skilled examiners and trained scorers to confirm the practicality of the SSC English test. Teaching-learning activities, classroom tasks, and question items should be compatible with the national curriculum objectives (NCTB, 2012). Real-life tasks or realia can ensure the authenticity of the SSC English test. The teaching-learning activities must be centered on authentic materials instead of scripted items. The question setters ought to incorporate more authentic tasks in the final test. Finally, the existing question management system of the SSC English test has to be secure and trustworthy.

The findings of this study put forward various insights into the SSC English examination in Bangladesh. The stakeholders of the SSC English test could be benefited from this study. This study scrutinizes the loopholes of the test and draws possible solutions and recommendations. Concludingly, this study could be a milestone in the context of high-stakes testing and assessment in Bangladesh.

REFERENCES

1. Alam, M. S., & Kabir, M. H. (2015). Assessing the Gap between Theory and Practice in Implementing CLT at Secondary Level in Bangladesh: A Case Study. Research on Humanities and Social Sciences, 5 (6). http://www.iiste.org
2. Ali, M. M., Hamid, M. O., & Hardy, I. (2018). Ritualisation of Testing: Problematising High-Stakes English-Language Testing in Bangladesh. Compare: A Journal of Comparative and International Education, 1-21. https://doi.org/10.1080/03057925.2018.1535890
3. Asaya, S. A. (1991) In-service education of teachers: problems and prospects. A Paper Presented at the first biennial Conference on Teacher Education Organized by the Lagos State College of Education. Oto-Ijanikin, 10th-14th Feb
4. Billah, M. (2015). Language and Education Inside and Beyond the Classroom, Murdhonno, Dhaka, Bangladesh.
5. Das, S., R. Shaheen, P. N. Shrestha, A. Rahman, and R. Khan. 2014. “Policy versus Ground Reality: Secondary English Language Assessment System in Bangladesh.” The Curriculum Journal, 25 (3): 326–343. doi:10.1080/09585176.2014.909323.
6. English for Today (NCTB). (2022). English for Today Listening Text. http://www.nctb.gov.bd/site/page/2cf0f1d9-e5eb-45c9-8be6-95c870e18d25/-
7. Everitt, B. S.; Skrondal, A. (2010), The Cambridge Dictionary of Statistics, Cambridge University Press.
8. Glen, S (2022). “Test-Retest Reliability / Repeatability” From StatisticsHowTo.com: Elementary Statistics for the rest of us! https://www.statisticshowto.com/test-retest-reliability/
9. Hughes, A. (2003). Testing for Language Teachers. Cambridge: Cambridge University Press.
10. Jamila, F., &Kabir, M. H. (2020). Examining the Existence of “Teaching to the Test” at SSC Level in Bangladesh. Creative Education, 11, 558-572.
11. Kabir, M. H. (2015b). Effect of Bangladeshi English Language Testing on Teaching and Learning: A Case Study. The International Institute for Science, Technology and Education (IISTE).
12. Lindstrom, D. (2010). Schaum’s Easy Outline of Statistics, Second Edition (Schaum’s Easy Outlines) 2nd Edition. McGraw-Hill Education
13. Maniruzzaman, M. (2016). EFL Testing Washback: Assessment of Learning or Assessment for Learning? Center for Pedagogy (CP), Independent University, Bangladesh (IUB) ISBN: 978-984-34-0780-1
14. Mariyam, T., Kabir, M. H., &Ullah, M. M. (2015). Casting the Authority or Holding It Still: An Investigation of Teachers’ Role in CLT Classrooms in Bangladesh. The International Journal of Social Science, (37). http://www.Tijoss.com
15. National Capital Language Resource Center (NCLRC). (2004). Assessing learning: Traditional tests. The essentials of language teaching. http://www.nclrc.org/essentials/assessing/ traditional.htm
16. National Curriculum (NCTB). (2012). English Classes IX & XI. Motijheel, Dhaka: National Curriculum & Textbook Board.
17. National Curriculum and Textbook Board (NCTB). (2022). English for Today Listening Text.http://www.nctb.gov.bd/site/page/2cf0f1d9-e5eb-45c9-8be6-95c870e18d25/-
18. Pathan, A. K. (2015, March 8). Solution to question paper leak. The Daily Star: Bangladeshi Edition. https://www.thedailystar.net/solution-to-question-paper-leak-37653
19. Patten, M. L. & Newhart, M. (2017). Understanding Research Methods. Routledge.
20. Pearson, K. (1895). “Notes on regression and inheritance in the case of two parents”. Proceedings of the Royal Society of London. 58, 240–242.
21. Rahman, K. N. & Afrin, S. (2017, November 13). Hapless parents question morality as authorities fail to stop question leaks. BD News. https://bdnews24.com/bangladesh/hapless-parents-question-morality-as-authorities-fail-to-stop-question-leaks.
22. SSC exams from Nov 14, number of candidates up by 1.79 lakh. (2021, October 27). The Daily Star. Retrieved from: https://www.thedailystar.net/youth/education/news/ssc-equivalent-exams-start-nov-14-education-minister-2207561
23. Steyn, H.S.; Walt, J.L. (2008). The validation of language tests. Stellenbosch Papers in Linguistics, 38 (191-204). Doi: 10.5774/38-0-29.

Appendix 1

Empirical/Statistical Validity: Dhaka Education Board 

Appendix 2

Empirical/Statistical Validity: Rajshahi Education Board

Appendix 3

 Empirical/Statistical Validity: Barisal Education Board

Appendix 4

Empirical/Statistical Validity: Chattogram Education Board

Appendix 5

Empirical/Statistical Validity: Comilla Education Board

Appendix 6

Empirical/Statistical Validity: Dinajpur Education Board

Appendix 7

Empirical/Statistical Validity: Jessore Education Board

 Appendix 8

Empirical/Statistical Validity:  Sylhet Education Board

Appendix 9A 

Note: The Standardized question (both paper I and II) is made following Nobodoot (Gyangriha Prokashoni: Dhaka)

Appendix 10A

Appendix 10B