Analysing Students’ Use of AI in Essay Outputs: A Case Study in Purposive Communication

Analysing Students’ Use of AI in Essay Outputs: A Case Study in Purposive Communication

San Miguel, Erika T, Inoncillo, Frederick A, Malto, Jennelyn T

Professor, Baliuag University.

DOI: https://doi.org/10.51244/IJRSI.2025.12040130

Received: 09 April 2025; Revised 17 April 2025; Accepted: 22 April 2025; Published: 21 May 2025

ABSTRACT

The popularization of ChatGPT has transformed the way students approach their academic tasks, especially in writing essays and other related outputs. While the benefits of these tools are widely recognized such as idea generation and enhanced writing efficiency; they also raise concerns for academic integrity. This study aims to analyse the extent of AI use in student essays for the Purposive Communication course in tertiary level at a select private University in Bulacan, Philippines. The study categorized submissions based on the percentage of AI-generated, mixed human-AI, or human-based content using three AI detection tools. Through mixed-method approach, particularly, sequential-explanatory approach, the study revealed deeper understanding on the effectiveness of AI detection tools and the students’ perspective on AI usage in writing essay outputs. The results revealed that there are inconsistencies in the AI detection tools’ AI scoring of the two outputs analysed, with one tool consistently identifying more outputs as human-written while the others flagged notable presence of AI-generated content with the texts. This inconsistency along with the variation of AI scores between the first and second output of students highlights the complexity and potential unreliability of using single detection tool as measure of student-authorship. In line with this, qualitative findings reflected the students’ growing maturity towards the use of AI in their writing which is reflected on their emotional response to the results, their awareness of academic integrity and responsible use of AI, and to their desire for empathetic guidance from their teachers as well as a clear, explicit, standardized policies on the use of AI in academic writing. Taken together, these results suggest the imperative for educational institutions to establish a reliable detection method with transparent policies on the use of AI, as well as open conversations about students’ use of AI. Thus, the researcher recommends that educational institution take a multifaceted approach to adopting AI in education, starting with fostering ethical awareness on responsible use of AI and academic honesty. Educators may also support students by developing authentic writing skills while guiding them to responsibly engage with these emerging powerful tools.

INTRODUCTION

Since the launch of ChatGPT and other AI-powered tools in November 2022, access to the seemingly limitless potential of AI has become widely available (Spies et al., 2024). This has led to the drastic transformation of various sectors of society, including education. In the Philippines, a survey conducted by Instructure in 2023 revealed that 83% of Filipino students find generative AI helpful in their writing. Moreover, 83% of the survey population use generative AI for research, while 62% use it for generating classroom content, and 53% use it as a personal aid for their learning.

Despite the perceived benefits of generative AI, serious concerns have emerged about the ethical and responsible use of these powerful tools. While generative AI is generally viewed as one of the most innovative breakthroughs in improving students’ learning outcomes (Saragih, 2025), Tzoneva (2024) acknowledged the challenges that come with its integration, citing the ethical implications of generative AI in the classroom, the potential biases brought by algorithms that may perpetuate discriminatory outcomes, privacy regulations, and safeguarding of users’ data. Additionally, she cited the risk of job displacement for educators.

In the end, educators have expressed concerns about the risk of overreliance on these tools when it comes to writing (Ramadhan et al., 2024). A study conducted by Smerdon (2024) pointed out that 67% of undergraduate students surveyed are currently using AI-powered tools to write research proposals, and those reported to be high achievers were more likely than others to adopt AI-powered tools in their writing in terms of idea generation, feedback, and as a personal tutor. Conversely, Woo et al. (2023) explored the impact of generative AI on writing quality, suggesting that its use as an aid did not significantly affect the performance of either high- or low-ranking students. It highlighted the increased risk of plagiarism associated with the use of generative AI, particularly in the context of weak AI detection systems and lenient AI usage policies.

These literatures underscore the need for ethical and responsible integration of generative AI within the academic setting. As reported in one of the articles by The Guardian (Montgomery, 2025), introducing the responsible and ethical use of generative AI to children is a crucial step in preparing students and teachers for a future increasingly shaped by generative AI. This presents a duality of challenges and opportunities as generative AI becomes mainstream among students and teachers as an educational aid. Its value in assisting with content generation and feedback is indeed evident, however, concerns regarding its potential impact on the writing and critical thinking skills of students remain. As a result, it is imperative to look into the ethical considerations and responsible use of AI-powered tools in education.

This imperative aligns with the goals of Artificial Intelligence in Education (AIEd). Luckin (2016) described AIEd’s aim as to support – not replace – teachers by promoting the thoughtful integration of AI aimed at improving education. According to him, AIEd enhances tutoring, collaboration, assessment, and lifelong learning through personalized, inclusive, and flexible education that clarifies implicit knowledge and analyzes the learning process. However, he stressed that this goal is achievable if proper infrastructure is provided, highlighting the need to focus on pedagogy, innovation, and stakeholder involvement in tool development.

However, despite the vision of AEId, Zawacki-Richter et al. (2019) emphasized the anxieties surrounding its integration into education, citing academic honesty and plagiarism issues, discussing the difficulties in detection and prevention, and advocating for proactive university strategies with clear policies, training, and detection methods. They argued that with proper guidelines, ethical and responsible implementation of AI can reduce the risks and maximize its potential.

Furthermore, several studies have suggested that there are numerous risks associated with AI integration in education, such as potential academic misconduct during online exams, undermining language learning processes, and political biases within AI systems (Susnjak, 2022; Tseng & Warschauer, 2023; McGee, 2023).

According to Susnnjak (2022), AI has high-level cognitive capabilities and can generate human-like texts, which poses a significant alarm for potential academic misconduct in online assessments, thus threatening the integrity of online assessments. Tseng and Warschauer (2023) also emphasized that the use of AI in education undermines the essential language learning processes of students, as students tend to use AI to generate ideas and even content. In another study by McGee (2023), they found political bias in ChatGPT’s generation of Irish limericks. The software was found to produce more positive ones for liberal politicians and more negative ones for conservatives.

The potential risks of AI, Kasneci, et al (2023) advocate that AI-generative tools have limitless potential for content creation, student engagement, and personalized learning. However, they also acknowledged that for AI to be effective, users must develop crucial competencies such as fact-checking, critical thinking, clear pedagogical strategies, and the ability to address bias and misuse through human oversight. This is supported by Xiao et al. (2023), who emphasized the importance of building a strong foundation of critical thinking skills and information literacy among students and educators alike. Furthermore, research has shown that AI detection is still proving to be a challenge, as distinguishing between human- and AI-generated text in an academic setting becomes increasingly difficult, especially as domain-specific models can achieve high accuracy, only noting distinct vocabulary and paragraph structures in ChatGPT’s writing.

Hence, it is the goal of this paper to examine the extent of AI-generated content in students’ essay outputs written for their Purposive Communication course. In this study, 13 essay outputs, consisting of 500-1200 words, were analyzed using three AI detection tools. According to Guistillo et al. (2024), a percentage generated by AI of 10-15% is acceptable, 16-25% is somewhat acceptable, and 26-50% and beyond is unacceptable. Using this threshold as a guide, the researchers measured the acceptability of the inclusion of AI in a subset of college students’ essays. Additionally, the researchers interviewed selected participants to further gain understanding on students’ insight on the use of AI in education.

METHODOLOGY

Research Design

The study utilized mixed-method research design, particularly, sequential-explanatory approach to analyse the students use of AI-powered tools in writing essays. The study was conducted in two stages: (1) simple descriptive quantitative design, and (2) descriptive qualitative design. The first phase analysed two sets of respondent outputs using three AI-detection tools namely ZeroGPT, GPT Zero, and Turnitin. Each tool provided different classification of the content, identifying whether the text was AI-generated, mixed (partially AI-generated), or human written. Zero GPT was used to determine percentage of AI-generated contents, GPT Zero was used to determine the percentage distribution of AI-generated, mixed AI-human, and human-generated content within each output, and Turnitin was used to describe distribution of essays with under 20% AI-generated content versus those exceeding the 20% threshold. The second phase gathered data through interviews with selected students to explore their reactions, reflections, and suggestions on the use of AI in academic writing. Then, mean, standard deviation and frequency distributions were utilized to describe the data.  Furthermore, the qualitative findings were used to explain and deepen the understanding of the quantitative results through interview. Lastly, thematic analysis, a method for identifying, analyzing, and interpreting patterns of meaning (‘themes’) within qualitative data was used.

Research Population

The participants of the study were select college students who took Purposive Communication course during the 2^nd trimester of school year 2024-2025 at Baliuag University in Bulacan, Philippines.  Furthermore, purposive sampling was used in finding the number of respondents in the study.  A total of 13 respondents were included in the study five (5) of which were male students and eight (8) of which were female students. Ahmed (2025) stated that 12-20 participants are sufficient to conduct thematic analysis.

Scope and Delimitation

One limitation of this study is that the sample size is small, with just 13 college students from a single institution. This limited pool might not fully represent the broader population of students, so caution is required in interpreting these findings. Another limitation lies in the potential bias inherent in the use of AI detection tools to analyse student writing. These tools are not perfect, and can result in both false positives and negatives, thus affecting accuracy when measuring how much students use Artificial Intelligence (AI). The findings from this case study apply only to the particular context in which the research was carried out and should not be extended to different educational settings with other technical infrastructures or academic cultures. When students come from diverse backgrounds with different expectations, then it is not possible for one set of findings to capture appropriately all cases.

Data Collection Tool

The data used in this study were two essay outputs submitted by the student respondents. For the first essay output, the students were tasked to watch a documentary and answer three (3) question prompts, responding with 300-500 words for each prompt, producing three short essays. For the second essay output, the students watched inspirational talks and were given two (2) question prompts, they were expected to write 300-800 words for each prompt, producing two short essays. These essays were then processed through selected AI-detection tools to assess the extent to which students used ChatGPT to assist them in their writing.

Then, an interview protocol was used to gather qualitative research data to further support the quantitative findings of the study. The interview protocol is composed of six (6) question prompts, the first three explores the students’ awareness of the use of AI-detection tools in the course and their reactions to the results; the second three explores the students’ reflections on the use of AI and their suggestions or recommendations to other students.

Ethical Considerations

The researchers made sure they adhered strictly to ethical standards of research writing to protect the rights of the student respondents:

• Informed consent. Students were informed about the purpose of the study, the nature of the data collection procedure, and how the results would be used. The students consented to having their essay outputs used for research purposes.
• Voluntary participation. The students were oriented about the nature of the study and were given the leeway to opt out of participating in this study. They were also assured that no academic penalty will be given to them based on their decision to participate or withdraw from participating.
• Anonymity and confidentiality. All kinds of identifying information were removed from the data to ensure that student respondents will remain anonymous. Furthermore, all data collected was treated with utter confidentiality and were used solely for this research’s purposes only.

RESULTS AND DISCUSSION

Table 1 shows the results of the 13 student essays that yielded varied AI detection scores across the three tools.

Table 1: AI Detection Results from Essay Outputs

The findings of quantitative analysis revealed that among the three (3) AI detection tools used, there were present inconsistencies across the detection tools. Particularly, two of the AI detection tools used flagged significant AI-generated contexts, but the third detection tool flagged none. Additionally, differences between the first and second essay outputs of the students were also noticeable in the results. Mainly, many students exhibit shifts between their AI detection score from the first to second essay outputs, this may be attributed to other factors such as time and schedule. Lastly, AI Detection Tool C consistently marked a higher proportion of essays as human-generated unlike AI Detection Tool A and AI Detection Tool B that is able to flag parts of the essays that were AI-generated. Overall, the quantitative findings suggest that complexity and unreliability of singular AI detection tools for verifying student-authored outputs. This discrepancy highlights a key challenge in academic settings: the reliability and transparency of AI detection tools. As argued by Eysenbach et al. (2023), current detectors can yield false positives, especially when assessing fluent human writing or AI-assisted but not fully AI-generated content.

As for the results of the qualitative analysis, two overarching themes concerning students’ experiences with AI detection tools and their perspectives on AI in academic writing were revealed.

Theme 1: Reactions to AI detection tools and the results. Generated three subthemes: (1) awareness and initial reactions, (2) emotional response to AI detection results, (3) understanding false positives and accountability.

Subtheme 1: Awareness and Initial Reactions. Most students were made aware of the AI detection policies set through course orientation and syllabus discussion which influenced them to limit their usage of AI and to self-check their work before submission. However, the perception of students on the leniency of teachers led them question the strictness of implementation of the policy, thus, affecting their compliance. This aligns with recent concerns in higher education about the lack of institutional clarity on AI ethics, which can inadvertently normalize uncritical use (Lund & Wang, 2023).

Subtheme 2: Emotional Response to AI Detection Results. Students’ reaction to the result ranges from feelings of pride to guilt and anxiety. While some feel accomplished, others reflected on their reliance to the use of AI and recognized their need to improve their paraphrasing and critical thinking skills. This suggests that detection tools, when used without dialogue or due process, can harm student confidence and learning environments (Kasneci et al., 2023).

Subtheme 3: Understanding False Positives. Many students showed sense of responsibility when discussing the possibility of false positives, suggesting they would defend their authorship with handwritten drafts if they are flagged incorrectly. Their reaction also signals their perception of fairness, academic pressure, and teacher support on their usage of AI.

Theme 2: Suggestions and Reflections. Generated three sub-themes: (1) need for a clearer guidelines and empathy, (2) embracing AI as an educational tool, (3) advice to other students on responsible use of AI.

Subtheme 1: Need for Clearer Guidelines and Empathy. All of the interviewed participants expressed need for explicit, standardized guidelines on AI use, preferably seeing it on the student handbook. They also expressed appreciation for teachers who provided clear, non-punitive policies and guidance which helped them to use AI responsible without fear of judgment. Several also emphasized the importance of empathy and acceptance from teachers to embrace honesty and openness in using AI.

Subtheme 2: Embracing AI as an Education Tool. All of the interviewed participants also support the adaptation of AI in academic writing with proper guidance. They recognize the potential of using AI to help those with weaker writing skills. They described AI as a “saklay” (crutch) and consultant for idea generation and improvement of writing, rather than shortcut for copying contents. This perspective reflects the growing maturity of students when it comes to the ethical consideration of AI writing.

Subtheme 3: Advice to Other Students on Responsible Use of AI. Students gave their advice to their peers, stating that they should use AI moderately, and focus on improving their writing and time management skills. They also encourage drafting an original work and consulting AI only for refinement, warning about the risks of overreliance and how it can weaken one’s critical thinking and authenticity in their voice. These reflections reveal their growing awareness on the ethical implications of the use of AI in academic writing.

CONCLUSION

In conclusion, through utilization of quantitative and qualitative analyses, the study was able to look into the effectiveness of AI detection tools and the students’ perception on AI-assisted academic writing. The quantitative results revealed that there are inconsistencies among the three (3) AI detection tools used, with one tool consistently identifying the analysed essays as human-written, while the other two AI detection tools flagged notable presence of AI-generated content with the texts. This inconsistency is also observed between the variations of AI scores from students’ first and second output. Highlighting the complexity and potential unreliability of using only one detection tool as a definitive measure of student-authorship.

Supporting these quantitative findings, the qualitative data revealed meaningful insights into the students’ perception of AI use in their writing. When the students were asked about their reaction to the results, they expressed a range of emotion from pride and sense of accomplishment to feelings of guilt and anxiety – this reflects students’ consciousness of the importance of academic integrity and its potential impact on their grades. The students’ reflection also emphasizes the need for educational institutions to have a clear, explicit, standardized policies on AI integration in academic writing. Furthermore, it was highlighted by the student respondents the important role of teachers. Citing how their perception on the leniency of teachers on the implementation of AI policies also affects their compliance. Additionally, the student respondents also seek support from their teachers, mentioning the importance of a supportive, non-punitive, and open conversation about the use of AI. For the students, it is important that their teachers recognize AI as not a means of deception, but rather, a supportive tool when used responsibly and ethically. The student respondents’ reflections underscored their growing maturity through demonstration of understanding of fairness, accountability, and peer advice in incorporating AI in their academic tasks.

RECOMMENDATIONS

• Based on the findings, the researchers come up with the following recommendations.
• Policy Development for Institutions
• Educational institutions must establish clear, context-specific rules for appropriate AI use in writing assignments, communicated at both the course and program level.
• Integration of AI Literacy
• Instructors should incorporate crucial discussions regarding AI ethics, authorship, and tool capabilities directly into curricula, particularly in subjects focused on writing.
• Cautious Use of Detection Tools
• Detection tools should be applied carefully and paired with dialogue, student reflection, and formative writing check-ins to gauge originality.
• Student-Centered Pedagogies
• Rather than enforcement, educators should mentor learners in cultivating their unique voice—guiding them to see AI as a framework for critical thinking, not a shortcut for outputs.
• In an age where the boundaries between human and machine writing continue to blur, fostering transparency, reflection, and ethical awareness becomes more urgent than enforcing rigid binaries. This case study affirms that academic integrity must adapt—not by rejecting technology, but by reimagining how we teach and assess student writing.

REFERENCES

1. Ahmed, S. K. (2025). Sample size for saturation in qualitative research: Debates, definitions, and strategies. Journal of Medicine, Surgery, and Public Health, 5, 100171. https://doi.org/10.1016/j.glmedi.2024.100171
2. Gustilo, L., Ong, E., & Lapinid, M. R. (2024). Algorithmically-driven writing and academic integrity: Exploring educators’ practices, perceptions, and policies in AI era. International Journal for Educational Integrity, 20(1), 3. https://doi.org/10.1007/s40979-024-00157-w
3. Instructure (2023) Filipino college students increasingly seek skills-based learning Instructure global annual report reveals. https://www.instructure.com/en-au/press-release/filipino-college-students-increasingly-seek-skills-based-learning-instructure-global
4. Kasneci, E., Sessler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., Krusche, S., Kutyniok, G., Michaeli, T., Nerdel, C., Pfeffer, J., Poquet, O., Sailer, M., Schmidt, A., Seidel, T., Stadler, M., Weller, J., Kuhn, J., & Kasneci, G. (2023). ChatGPT for good? On opportunities and challenges of large language models for education. ¹ Learning and Individual Differences, 103, 102274. https://doi.org/10.1016/j.lindif.2023.102274
5. Luckin, R., Holmes, W., Griffiths, M., & Forcier, L. B. (2016). Intelligence Unleashed: An Argument for AI in  Education. Pearson. https://static.googleusercontent.com/media/edu.google.com/en//pdfs/Intelligence-Unleashed-Publication.pdf
6. McGee, R. (2023). Is Chat Gpt Biased Against Conservatives? An Empirical Study. SSRN Electronic Journal. https://doi.org/10.2139/ssrn.4359405
7. Montgomery, B. (2025, March 30). “How and why parents and teachers are introducing young children to AI”. The Guardian. Retrieved from https://www.theguardian.com/technology/2025/mar/30/parents-teachers-children-ai?utm_source=chatgpt.com
8. Ramadhan, M. Q., Azani, M. Z., & Abdala, G. (2024). Advantages of Artificial Intelligence and Its Relevance in Improving Educational Quality. Iseedu: Journal of Islamic Educational Thoughts and Practices, 8(1), 77-86. https://doi.org/10.21093/iseedu.v8i1.9095
9. Saragih, R. (2025). Student Perception of The Advantages and Challenges of Using Chatgpt (Artificial Intelligence) For Academic Assignments. Journal of Information Technology, computer science and Electrical Engineering, 2(1), 23-34.4
10. Smerdon, D. (2024). AI in essay-based assessment: Student adoption, usage, and performance. Computers and Education: Artificial Intelligence, 7, 100288. https://doi.org/10.1016/j.caieai.2024.100288
11. Spies, N. C., Ratts, V. S., & Hagemann, I. S. (2024). Estimating the Prevalence of Generative AI Use in Medical School Application Essays. medRxiv. https://doi.org/10.1101/2024.10.23.24300177
12. Susnjak, T. (2022). ChatGPT: The End of Online Exam Integrity? https://doi.org/10.48550/arXiv.2212.09292
13. Tseng, W., & Warschauer, M. (2023). AI-writing tools in education: If you can’t beat them, join them. Journal of China Computer-Assisted Language Learning, 3(1), 5-21. https://doi.org/10.1515/jccall-2023-0008
14. Tzoneva, I. (2024). Benefits and challenges in using ai-powered educational tools. Education and New Developments, 2.
15. Woo, D. J., Susanto, H., Yeung, C. H., Guo, K., & Fung, A. K. Y. (2023). Exploring AI-Generated text in student writing: How does AI help? (arXiv:2304.02478). arXiv preprint. https://doi.org/10.48550/arXiv.2304.02478
16. Xiao, F., Zhu, S., & Xin, W. (2025). Exploring the Landscape of Generative AI (ChatGPT)-Powered Writing Instruction in English as a Foreign Language Education: A Scoping Review. ECNU Review of Education, 0(0), 1-24. https://doi.org/10.1177/20965311241310881
17. Zawacki-Richter, Olaf & Marín, Victoria & Bond, Melissa & Gouverneur, Franziska. (2019). Systematic review of research on artificial intelligence applications in higher education -where are the educators?. International Journal of Educational Technology in Higher Education. 16. 1-27. 10.1186/s41239-019-0171-0.