INTERNATIONAL JOURNAL OF RESEARCH AND INNOVATION IN SOCIAL SCIENCE (IJRISS)
ISSN No. 2454-6186 | DOI: 10.47772/IJRISS | Volume IX Issue X October 2025
Page 4777
www.rsisinternational.org
AISTY: An Explainable AI-Driven Vision-Based Adaptive Learning
System for Children with Autism Spectrum Disorder
Muhammad Rifqi Zakirin Rofidi, Siti Azirah Asmai
*
, Muhammad Hafidz Fazli Md Fauadi, Nur Diana
Izzani Masdzarif, Abdul Syukor Mohamad Jaya
Fakulti Kecerdasan Buatan dan Keselamatan Siber, Universiti Teknikal Malaysia Melaka, Melaka,
Malaysia
DOI: https://dx.doi.org/10.47772/IJRISS.2025.910000394
Received: 12 October 2025; Accepted: 20 October 2025; Published: 13 November 2025
ABSTRACT
The educational needs of children with autism spectrum disorder (ASD) require individualized approaches
because they face major difficulties with communication and social skills and learning processes. The diverse
requirements of children with ASD remain unmet by conventional educational approaches because these
methods fail to provide suitable flexibility and expandability. The research develops Adaptive Interface System
for Tracking Engagement (AISTY) as an explainable AI-based learning module which solves the current
educational needs of children with ASD. The system AISTY uses computer vision to track behavioural data
through real-time analysis while delivering interactive learning content that adjusts to student needs. The
system uses a custom Convolutional Neural Network (CNN) model to analyse facial expressions for
engagement and emotional state classification (Happy, Sad, Neutral, Surprise, Fear, Angry, Disgust) with 78%
accuracy while operating at 14-16 frames per second on typical computer equipment. The system contains
three learning modules (Mathematics, Sing-Along, and Mini-Games) which modify their content according to
the child's current state. The system includes an explainable AI (XAI) dashboard that uses SHAP-inspired
visualizations to show caregivers exactly what elements influence the system's adaptation decisions. The
usability assessment with parents and a special education teacher validated the system's operational capabilities
and user-friendly interface and confirmed its worth. The research shows AISTY offers a flexible autism
education system which proves that explainable adaptive AI can be effectively used in special needs education
to boost student participation and academic success.
Keywords- autism spectrum disorder (ASD), Artificial Intelligence, Computer Vision, Adaptive Learning,
Emotion Recognition, Educational Technology.
INTRODUCTION
The neurodevelopmental disorder autism spectrum disorder (ASD) presents as a complex condition which
affects social communication and interaction and causes restricted behavioral patterns and interests [1]. The
World Health Organization reports that ASD affects 1 in 100 children worldwide based on recent global
prevalence data [2]. The growing number of children with ASD requires immediate development of effective
accessible intervention methods which can scale to meet the rising demand. The official ASD prevalence in
Malaysia remains low because of under-diagnosis and restricted access to specialized care which demonstrates
an urgent requirement for new intervention methods [3].
Children with ASD face exceptional difficulties in their educational environment. The learning abilities of
ASD children differ widely because some excel at visual processing, but others face challenges with verbal
skills and sensory sensitivity [4]. Standard educational methods that use uniform approaches create student
disengagement which results in negative emotional responses and poor academic achievement. The most
effective intervention method requires individualized programming which uses structured methods to adjust its
content based on the child's changing attention span and emotional state [5]. The delivery of customized
support to children requires extensive resources because it needs ongoing professional monitoring from trained
therapists and caregivers, yet these resources are scarce due to financial and practical limitations.
INTERNATIONAL JOURNAL OF RESEARCH AND INNOVATION IN SOCIAL SCIENCE (IJRISS)
ISSN No. 2454-6186 | DOI: 10.47772/IJRISS | Volume IX Issue X October 2025
Page 4778
www.rsisinternational.org
Artificial Intelligence through its computer vision and machine learning subfields brings a revolutionary
change to current practices[6]. AI systems track children's behavior through webcam video analysis to detect
facial expressions and body movements which helps identify their level of engagement and emotional state [6].
The system uses real-time data to modify digital learning content which results in an individualized
educational experience. The system detects rising frustration through its monitoring system which then
provides easier tasks and relaxation activities for the child.
AI systems face challenges in healthcare and educational settings because their decision-making processes
remain unclear to users. The lack of transparency in AI decision-making systems makes therapists and
caregivers uncertain about following system recommendations [7]. Explainable AI (XAI) provides solutions to
this problem through its ability to reveal AI decision-making processes in an understandable manner. The
system uses XAI to display its reasoning about child disengagement through specific metrics which show 75%
gaze aversion and 25% slumped posture.
The research presents the comprehensive development of AISTY, an innovative system that integrates AI
technology with vision-based student tracking, adaptive learning content, and a transparent analytics
dashboard. This study moves beyond conventional predictive approaches by developing an educational tool
capable of providing clear and explainable outputs that enhance trust and support effective learning for
children with autism spectrum disorder (ASD).
The objectives of the research are threefold: (1) to investigate and implement neural network models for real-
time emotion and engagement recognition; (2) to design a user-friendly, sensory-aware system tailored for
children with ASD and their caregivers; and (3) to develop and integrate an interactive learning platform
featuring adaptive learning modules and continuous progress monitoring.
LITERATURE REVIEW
Autism Education and Technology
The educational approach for ASD students has traditionally relied on structured evidence-based methods. The
learning principles of Applied Behavior Analysis (ABA) help professionals develop specific behaviors through
Discrete Trial Training (DTT) [8]. The TEACCH (Treatment and Education of Autistic and related
Communication handicapped Children) method provides structure through physical organization and visual
schedules and work systems [9]. The SCERTS model among developmental approaches focuses on three main
areas which include Social Communication and Emotional Regulation and Transactional Support [10].
The methods show effectiveness, yet they need human professionals for implementation and face difficulties
when trying to expand their reach. The methods depend on periodic human observation for assessment, yet
they fail to detect the small changes that occur in a child's state during the moment. The current situation
requires technology to fill the void by offering continuous objective measurement and intervention capabilities.
AI and Computer Vision in Behavioral Analysis
The development of assistive technologies for ASD has received a transformative boost from artificial
intelligence technology. Machine learning algorithms detect intricate patterns in data which human observers
cannot detect.
Multiple research studies have employed AI systems to evaluate behavioral information for identifying
children at risk of developing ASD. Bone et al. [11] created predictive models which achieved 81% accuracy
through their analysis of initial behavioral indicators.
The use of computer vision technology allows for unobtrusive ongoing observation of behavior. The research
by Yujian Cai et al. [12] achieved 87.31% accuracy in emotion detection through facial expression analysis.
Goodwin et al. [13] implemented movement analysis algorithms to measure motor stereotypies (hand-
flapping) through quantitative methods which replaced traditional clinical assessment methods.
INTERNATIONAL JOURNAL OF RESEARCH AND INNOVATION IN SOCIAL SCIENCE (IJRISS)
ISSN No. 2454-6186 | DOI: 10.47772/IJRISS | Volume IX Issue X October 2025
Page 4779
www.rsisinternational.org
The research by Seyderhelm et al. [14] presented systems which use real-time performance metrics to adjust
task difficulty levels dynamically. The system maintains an optimal learning zone through performance-based
adjustments which prevent child frustration.
The current systems face a common problem because they concentrate on individual features such as emotion
detection or fixed games. The AISTY platform functions as a complete system which combines ongoing
behavioral assessment with various adaptive educational content.
Adaptive Learning Systems
The combination of artificial intelligence with pedagogical design principles leads to the development of
adaptive learning systems. The system modifies its content level and presentation format and learning speed
through ongoing evaluations of student performance and emotional state. The ability of adaptive systems to
adjust their operations helps ASD children control their anxiety while keeping them actively involved in
learning. The current frameworks use rule-based systems and reinforcement learning to activate interventions
through agitation detection which results in task simplification and break provision [8]. The adaptive logic of
AISTY operates through real-time vision-based child state assessment to determine its actions.
METHODOLOGY
Research Design
This research project implemented Design Science Research (DSR) as its methodology to develop and assess
new artifacts which solve specific problems. The development process followed an iterative agile structure
which divided into two-week sprints. The system integration process received continuous testing and
refinement of its AI model and modules and user interface components throughout each development cycle.
The system design followed User-Centered Design principles which used ASD-friendly interface research to
create initial design principles that later received validation through user testing.
System Architecture Overview
The system operates with a client-server structure but performs essential processing tasks on the client-side for
both privacy protection and immediate system response. The system uses Python's CustomTkinter (CTkinter)
library to create its frontend which provides a sensory-friendly graphical user interface through muted color
schemes and high-contrast text and organized uncluttered layouts. The Python-based Backend (Client) section
runs the core AI logic which performs video processing and model inference and module adaptation tasks in
real-time. The cloud-based NoSQL database Google's Firebase Firebase operates as the database system to
store anonymized session information and progress metrics and analytics data for both longitudinal tracking
and cross-device access. The architecture of AISTY operates at a high level as shown in Figure 1. The system
architecture diagram shows Webcam data entering OpenCV/MediaPipe Preprocessing which then feeds into
the Custom CNN Model and Engagement Logic Engine and Adaptive Modules and Firebase and XAI
Dashboard.
Fig. 1 System architecture
Data Acquisition
The emotion recognition CNN received its initial training from the FER2013 dataset which contains more than
35,000 grayscale face images with seven emotional labels. The model received additional training through a
INTERNATIONAL JOURNAL OF RESEARCH AND INNOVATION IN SOCIAL SCIENCE (IJRISS)
ISSN No. 2454-6186 | DOI: 10.47772/IJRISS | Volume IX Issue X October 2025
Page 4780
www.rsisinternational.org
smaller dataset of child facial expressions which researchers obtained from open-source repositories. The
model received data augmentation through rotation and zoom and shear and horizontal flip operations to boost
its performance and reduce overfitting.
AI Model Development
The AI model was developed using a sequential Convolutional Neural Network (CNN) architecture
implemented through TensorFlow and Keras. The system was designed to balance accuracy and processing
speed, enabling efficient real-time performance. To interpret user behaviour, an engagement logic engine was
integrated, employing a rule-based state machine to process raw emotion classifications and determine the
overall engagement state. Focused engagement is detected when users display happy or neutral emotions while
maintaining their head position within a yaw range of –15° to +15° and a pitch range of –20° to +20°.
Distraction is identified when neutral emotions coincide with head movements away from the screen or when
users’ hands obstruct the camera view. Meanwhile, agitation is recognised when users exhibit angry, sad, or
fearful emotions with high confidence levels, or when neutral emotions are accompanied by gestures such as
covering the face with their hands.
Development of Learning Modules
The system contains three interactive modules which share a common base class to maintain uniform
integration with the monitoring and adaptation system. The system produces adaptive math problems through
addition and subtraction operations. The system adjusts difficulty levels through number range modifications
based on user performance results. The system provides visual feedback and maintains a scoring system. The
system shows a "conceptual help" prompt after detecting extended periods of agitation from the user. The
system recommends the Sing-Along Module to users when their engagement state reaches 'Agitated' according
to the logic engine. Mini-Game Module contains three educational games which help users develop their
cognitive abilities. Tetris serves as a visual-spatial reasoning and planning tool. The game features an "Infinite
Mode" which eliminates game-ending situations to help players relax. The game Tic-Tac-Toe helps users
develop basic strategic thinking abilities and theory of mind skills. The game includes an AI opponent which
operates based on predefined rules. The Memory Game helps users improve their short-term memory and
concentration abilities.
Explainable Analytics Dashboard
The dashboard system enables caregivers to access two levels of explanation through its interface. The session
analytics section of the dashboard displays emotion and engagement patterns through line and bar charts
generated by matplotlib for a single session. The system displays decision rationales through a simplified
interface which shows SHAP-inspired factors that led to system actions such as "Suggested a break." The
system displays two factors which explain the decision: 'Frustration' score at 70% and gaze aversion at 30%.
Evaluation Strategy
The trained CNN model achieved evaluation through standard metrics on 890 test images which included
Accuracy and Precision and Recall and F1-Score.
The system underwent usability testing with five participants who included four parents of neurodiverse
children and one special education teacher with experience. The system received feedback about its interface
and responsiveness and analytics usefulness through participant interviews and System Usability Scale (SUS)
assessments.
RESULTS
AI Model Performance
The custom CNN model reached a total test set accuracy of 78%. The model achieved its best results when
detecting 'Happy' and 'Neutral' emotions but showed poor performance in identifying 'Disgust' and 'Fear'
INTERNATIONAL JOURNAL OF RESEARCH AND INNOVATION IN SOCIAL SCIENCE (IJRISS)
ISSN No. 2454-6186 | DOI: 10.47772/IJRISS | Volume IX Issue X October 2025
Page 4781
www.rsisinternational.org
emotions according to Table I. The model performed well in detecting 'Happy' and 'Neutral' emotions but
failed to recognize 'Disgust' and 'Fear' expressions which matches previous research findings about expression
recognition difficulties caused by unbalanced data and overlapping expression meanings.
Table I Emotion Recognition Model Performance Metrics
Emotion Class
Precision (%)
Recall (%)
F1-Score (%)
Happy
84.5
82.5
83.5
Sad
79.5
82.1
80.8
Neutral
81.0
79.2
80.1
Surprise
73.2
70.8
72.0
Fear
71.5
72.3
71.9
Angry
75.1
73.0
74.0
Disgust
64.8
65.6
65.2
In terms of real-time performance, the integrated system consistently processed video feeds at 14-16 frames
per second (FPS) on the target hardware (Intel i5 CPU, 8GB RAM, integrated graphics), ensuring a fluid and
responsive user experience.
System Usability and Functional Testing
The usability study yielded highly positive feedback. The System Usability Scale (SUS) score averaged 82.5,
which falls in the "excellent" range. Key qualitative findings are summarized in Table II.
Table Ii Summary of Usability Testing Feedback
System component
User Feedback Summary
Implication
Overall GUI &
Interface
"Clean," "intuitive," "not overwhelming,"
"the colors are calming."
The sensory-friendly UCD
approach was successful
Mathematics Module
“The structure is good for my child”
“Immediate feedback helps”
“Simple math for appropriate age”
The module's design aligns with
the need for predictable,
structured activities for children
with ASD
Sing-Along Module
“This is perfect for calming down”
“Great for transitions between activities.”
Validates the module's use as a
tool for emotional regulation
Mini-Games Module
“My child was engaged for a long time”
The games are effective at
maintaining interest and
providing engaging cognitive
training
Analytics Dashboard
“My child was engaged for a long time”
“I can see exactly when he started to get
The explainable analytics
empower caregivers with
actionable insights, building trust
INTERNATIONAL JOURNAL OF RESEARCH AND INNOVATION IN SOCIAL SCIENCE (IJRISS)
ISSN No. 2454-6186 | DOI: 10.47772/IJRISS | Volume IX Issue X October 2025
Page 4782
www.rsisinternational.org
frustrated”
“Helps me understand what to work on”
in the system.
Adaptation System
“The break suggestion popped up at the right
time”
“Auto-adjusting math difficulty”
The rule-based logic is
functionally correct, but there is a
clear demand for more
sophisticated, AI-driven
adaptation
The system's core adaptive functionality was confirmed during testing. The rule-based engine successfully
triggered on-screen suggestions (e.g., "Let's take a break and sing a song!") when the 'Agitated' state was
detected, and these interventions were perceived as timely and appropriate by the users.
DISCUSSION
The AISTY development process together with evaluation results demonstrate that it is possible to build an
integrated AI system which provides explainable solutions for autism education. The 78% model accuracy
demonstrates strong performance for real-time applications although it presents opportunities for enhancement
which establishes a solid base. The system demonstrates exceptional performance in detecting 'Happy' and
'Neutral' states because identifying positive engagement versus negative or distressed states represents the
essential function for triggering adaptive responses.
The project design philosophy received strong positive feedback from users which confirms the project's
success. The success of the sensory-friendly interface demonstrates how User-Centered Design (UCD)
methods create essential tools for neurodiverse users. The analytics dashboard received praise from users
because it provides transparent information which represents a vital success element that most AI interventions
lack. AISTY becomes an active educational partner for caregivers through its explainable insights which build
trust and supports data-driven decisions beyond application.
The research study demonstrates multiple areas for future development through its identified limitations. The
model demonstrates poor performance when detecting 'Disgust' emotions and other rare feelings because it
requires access to bigger ASD-specific datasets with diverse content. The current rule-based adaptation system
operates effectively but it maintains basic functionality. The research direction toward developing a
reinforcement learning (RL) agent for personalized intervention strategy learning matches user requests for
advanced adaptation features.
The design of AISTY stands out because it uses Firebase for tracking children's development over time. The
system allows future interventions to use long-term child development patterns for creating customized
learning paths.
CONCLUSIONS
This research has developed AISTY, an AI-driven, vision-based adaptive learning system designed to support
children with Autism Spectrum Disorder (ASD). By employing a real-time vision-based AI pipeline powered
by a Convolutional Neural Network (CNN) model, AISTY is able to provide responsive and personalized
learning experiences. The system integrates facial expression and emotion state classification to offer an
analytics dashboard that provides transparent insights to caregivers and teachers, thereby fostering trust and
facilitating informed, personalized interventions.
While the proposed AI-assisted emotion recognition system achieved promising preliminary results with 78%
accuracy, several limitations should be acknowledged. The reliance on webcam-based facial analysis may not
adequately capture the nuanced emotional cues of children with ASD, particularly those with atypical gaze or
reduced facial expression. Furthermore, the absence of a longitudinal design restricts the understanding of
sustained learning outcomes and behavioral adaptation. Future research will address these limitations by
INTERNATIONAL JOURNAL OF RESEARCH AND INNOVATION IN SOCIAL SCIENCE (IJRISS)
ISSN No. 2454-6186 | DOI: 10.47772/IJRISS | Volume IX Issue X October 2025
Page 4783
www.rsisinternational.org
incorporating multimodal data sources, benchmarking against established assistive technologies, and
conducting longitudinal studies to evaluate long-term educational impact.
Overall, the development and evaluation of AISTY highlights the use of AI-driven educational technologies for
special needs learners can be both scalable and dependable. This study contributes not only a functional proof-
of-concept but also a flexible framework that unites technical rigour with user-centred design and explainable
AI principles, advancing the vision of inclusive and transparent intelligent learning environments.
ACKNOWLEDGMENT
The authors would like to thank the Faculty of Artificial Intelligence and Cyber Security(FAIX), Universiti
Teknikal Malaysia Melaka (UTeM) for their assistance in this research.
REFERENCES
1. Martínez-González, A.E., Cervin, M. and Piqueras, J.A., (2022). Relationships between emotion
regulation, social communication and repetitive behaviors in autism spectrum disorder. Journal of
autism and developmental disorders, 52(10), pp.4519-4527.
2. World Health Organization: WHO. (2023), November 15). Autism. https://www.who.int/news-
room/fact-sheets/detail/autism-spectrum-disorders
3. Jiar, Y.K., (2014). Factors Associated with Diagnosis of Autism Spectrum Disorder (ASD) under the
Age of 24 Months in Malaysia. Sains Humanika, 2(1).
4. Lei, J., & Ventola, P. (2017). Pivotal response treatment for autism spectrum disorder: current
perspectives. Neuropsychiatric Disease and Treatment, Volume 13, 1613–1626.
5. Lindroth, H., Nalaie, K., Raghu, R., Ayala, I.N., Busch, C., Bhattacharyya, A., Moreno Franco, P.,
Diedrich, D.A., Pickering, B.W. and Herasevich, V., (2024). Applied artificial intelligence in
healthcare: a review of computer vision technology application in hospital settings. Journal of Imaging,
10(4), p.81.
6. Ahmmad, J., Al-Dayel, O.A., Khan, M.A. and Mahmood, T., (2025). AI-assisted technology
optimization in disability support systems using fuzzy rough MABAC decision-making. Scientific
Reports, 15(1), p.18335.
7. Hulsen, T., (2023). Explainable artificial intelligence (XAI): concepts and challenges in
healthcare. Ai, 4(3), pp.652-666.
8. Gitimoghaddam, M., Chichkine, N., McArthur, L., Sangha, S.S. and Symington, V., (2022). Applied
behavior analysis in children and youth with autism spectrum disorders: a scoping review. Perspectives
on behavior science, 45(3), pp.521-557
Schopler, E. and Van Bourgondien, M.E., (2020). Treatment and Education of Autistic and Related
Communication Children. In Autistic adults at Bittersweet Farms (pp. 85-94). Routledge.
9. Yu, L. and Zhu, X., (2018). Effectiveness of a SCERTS model-based intervention for children with
autism spectrum disorder (ASD) in Hong Kong: A pilot study. Journal of Autism and Developmental
Disorders, 48(11), pp.3794-3807.
10. Bone, C., Simmonds-Buckley, M., Thwaites, R., Sandford, D., Merzhvynska, M., Rubel, J.,
Deisenhofer, A., Lutz, W., & Delgadillo, J. (2021). Dynamic prediction of psychological treatment
outcomes: development and validation of a prediction model using routinely collected symptom data.
The Lancet Digital Health, 3(4), e231–e240. https://doi.org/10.1016/s2589-7500(21)00018-2
11. Cai, Y., Li, X., & Li, J. (2023). Emotion recognition using different sensors, emotion models, methods
and datasets: A Comprehensive review. Sensors, 23(5), 2455. https://doi.org/10.3390/s23052455
12. Goodwin, M.S., Intille, S.S., Albinali, F. and Velicer, W.F., (2011). Automated detection of stereotypical
motor movements. Journal of autism and developmental disorders, 41(6), pp.770-782. 1-12.
Seyderhelm, A.J. and Blackmore, K., 2021. Quantifying in-game task difficulty using real-time
cognitive load.