Navigating Cognitive Workload: Multimodal Evaluation of Infotainment Accessibility in Indian Drivers

Navigating Cognitive Workload: Multimodal Evaluation of Infotainment Accessibility in Indian Drivers

Jyoti Kumar¹, Mannu Brahmi², Ahana Viegas Raman³, Greeshma Sharma¹, Abhijeet Kujur¹

¹Dept. of Design (DoD), Indian Institute of Technology, Delhi

²National Resource Centre for Value Education in Engineering (NRCVEE), Indian Institute of Technology, Delhi

³Centre for Health Psychology, School of Medical Sciences, University of Hyderabad

DOI: https://doi.org/10.51584/IJRIAS.2024.911042

Received: 02 December 2024; Accepted: 10 December 2024; Published: 18 December 2024

ABSTRACT

The integration of advanced in-car infotainment systems raises essential concerns regarding their influence on the driver’s performance, particularly the cognitive aspects. To address this, our study assessed the cognitive workload (CWL) associated with various infotainment systems using neurophysiological tools such as electroencephalograms (EEG), eye tracking (ET), and galvanic skin resistance (GSR) across three task modes (touch, voice, hybrid), two contexts (driving and static), two infotainment system designs (different cars), and four user tasks (Radio Station, Temperature Change, Calling, Navigation) on a dedicated driving track. This empirical framework provided a realistic and accurate evaluation of cognitive workload and user experience in Human-Computer Interaction (HCI) designs. Fourteen participants performed 24 tasks over 50-80 minutes, revealing that touch-based interactions consistently led to higher cognitive load, with touch-based navigation leading to a 14% increase. Voice mode heightened distraction, while hybrid mode resulted in the highest distraction levels (48.12% higher than voice and 17.26% higher than touch mode). Systems without autosuggestion also demonstrated higher workload and distraction. The study recommends using hybrid modes for navigation, physical steering wheel switches for calling, and voice commands for radio when steering-mounted controls are unavailable. Establishing national/international policies and standards, and industry-level certifications for infotainment systems is also recommended. Our limitations included moderation issues, a broad but shallow experimental design, and limited task trials. Future research should focus on detailed experimentation, cognitive load theory-based rating scales, and virtual reality (VR) setups with neurophysiological measurements.

Keywords: Cognitive Workload (CWL), Infotainment Systems, Electroencephalography (EEG), Eye-tracking, Galvanic Skin Resistance, Human-Computer Interaction, User Experience.

INTRODUCTION

The rapidly evolving automotive technology in the Indian market leaves concern about its impact on driver attention and performance. Prior studies have repeatedly demonstrated that driver inattention and distractions are a leading cause of automobile accidents (Hanowski et al., 2005; Klauer and Mierke, 2005; Regan, Lee and Young, 2008; Lee, 2017). Driving is inherently complex, involving multitasking influenced by road conditions, traffic, dashboard devices, and external stimuli affecting driver performance (Di Flumeri et al., 2018). Intelligent transport systems recognise that drivers’ cognitive demands should remain within a threshold due to limited human processing resources and time constraints (Solís-Marcos and Kircher, 2018). This can be measured via cognitive workload (CWL) – the mental effort required for task completion (Gevins et al., 1997; Figalová et al., 2022). It provides insights into mental fatigue, mental effort, and concentration levels as a function of task complexity, learning techniques, and information-processing capabilities (Plechawska-Wójcik et al., 2019). Elevated CWL is associated with decreased situational awareness and driving errors, raising collision risks (Lobo et al., 2016; Di Flumeri et al., 2018). Besides the driving tasks, vehicle navigation, and vehicle control that require increased cognitive demands (Cernea et al., 2012), technological advances such as smartphone integration and in-car entertainment also impact driver behaviour, making it crucial to balance task difficulty, risk, and safety (Summala, 2007). Our study examines lapses in attention and performance during interactions between drivers and in-car infotainment systems amidst the rapid development of these systems in India.
Rationale Behind Neurophysiological Tools for HCI Usability Studies

Unlike the subjective methods like interviews and surveys to assess cognitive burden (Kruger and Doherty, 2016) that fail to capture the “unconscious” phenomenon underlying human behaviour (Gopher and Braune, 1984; Wall et al., 2004; N. Kumar and Kumar, 2016; Aricò et al., 2018), our study utilises neurophysiological measures of electroencephalography (EEG), galvanic skin resistance (GSR), and eye tracking (ET) for objective and precise usability assessments in Human-Computer Interaction (HCI) designs (Hancock and Chignell, 1988, 1989; J. Kumar and Kumar, 2016; N. Kumar and Kumar, 2016). They provide accurate, comparable findings by measuring brain psychophysiological responses (Hart, 2006) and are affordable, non-invasive, and independent of medical training. The EEG provides a direct measure of brain activations and high temporal resolution (Freeman et al., 2000; Prinzel et al., 2000; Islam et al., 2020), while electrodermal activity (EDA) or GSR gauges skin conductance linked to arousal and attention, particularly useful for mental workload and driver performance assessment (Boucsein, 2012; Nourbakhsh, Wang and Cheng, 2012; Suoja et al., 2018). Eye-tracking metrics, including pupil dilation, blink frequency, and duration, correlate with cognitive exertion (Beatty and Wagoner, 1978; Klingner, 2010; Hossain and Elkins, 2016). This multimodal approach enhances measurement reliability, robustness, and precision by leveraging multiple sensors (Debie et al., 2021), reducing uncertainty, improving sensitivity, and accounting for individual variability (Wierwille and Thomas, 1993; Johannes and Gaillard, 2014). Additionally, it improves signal-to-noise ratio, enhances spatial and temporal range, and provides better resolution of driver cognitive states, essential for designing safer, more productive dashboards (Kohlmorgen et al., 2007; Lin, Ko and Shen, 2009; Göhring et al., 2013).

EEG & Eye-Tracking for Real-Time Cognitive Monitoring in HCI

The EEG is pivotal for real-time monitoring of cognitive processes such as attention, memory, and workload. Studies using EEG headsets for in-vehicle systems control revealed reduced errors, especially for simple commands, compared to touch-based manipulation (Cernea et al., 2012). EEG spectrum modulation demonstrated increased frontal theta activity with a higher working memory load and decreased parietal alpha activity with an increased task load (Lei and Roetting, 2011). Research on EEG theta power activity in high-stress environments showed increased activity in frontal, temporal, and occipital regions under high mental workload (Diaz-Piedra, Sebastián and Di Stasi, 2020). Others have also demonstrated increased activity in the right hemisphere for theta, sensorimotor rhythm (SMR), beta, and gamma bands, indicating greater attention and tiredness among novice drivers than veterans (Ma et al., 2012). Complimentary to these methods, event-related potentials (ERPs), such as N1 and P3, identified heightened cognitive demands linked to faster processing requirements and the introduction of visual displays (Sugimoto et al., 2020). A similar analysis revealed a significant decrease in P3b amplitude with increased workload levels (Lei, Welke and Roetting, 2009). Eye-tracking (ET) also provides valuable insights into driver behaviour, with novice drivers focusing more on the dashboard compared to experienced drivers who prioritise the front view (Nabatilan et al., 2011). Increased blink frequency, pupil diameter, and horizontal vergence have also indicated cognitive load in dual and auditory tasks (Tsai et al., 2007).

Multimodal Approaches for Evaluating Cognitive Load

The multimodal approach of EEG with biosignals like skin conductance, pulse, and respiration effectively recognises cognitive workload. Studies using these biosignals in driving simulators have assessed stable workloads during multitasking (Putze, Jarvis and Schultz, 2010; Lobo et al., 2016), supporting findings that ERPs reveal significant differences in cognitive load between single-task and multitasking conditions (Solís-Marcos and Kircher, 2018). Such multimodal approaches offer comprehensive assessments of mental states during driving tasks (Putze, Jarvis and Schultz, 2010; Ma et al., 2012; Di Flumeri et al., 2018; Pollmann et al., 2019).

Cognitive Influence of Experiential & Environmental Factors

Research shows that novice and elderly drivers experience higher cognitive workloads than experienced drivers under similar conditions (Nabatilan et al., 2011; Karthaus, Wascher and Getzmann, 2018). Novice drivers tend to focus more on the dashboard, while experienced drivers prioritise the road (Nabatilan et al., 2011). Braking response under distractions revealed diminished cognitive control in older adults, illustrated by a smaller P3b component, with visual stimuli causing heightened interference (Karthaus, Wascher and Getzmann, 2018). Studies on the environmental factors revealed mixed impacts as one found bright blue lighting and reduced auditory and visual stimuli to affect driver concentration (Pollmann et al., 2019). Yet, another showed that ambient light did not increase mental workload, suggesting that certain environmental modifications can improve driver performance without adding cognitive burden (Figalová et al., 2022). Additionally, research on one-pedal automobile operation indicates that such systems could improve driving enjoyment and reduce cognitive workload (Sugimoto et al., 2020).

Cognitive Load Assessment Challenges

Comparative studies of simulated vs. real-world driving conditions revealed that on-road driving caused slightly higher arousal levels (Reimer and Mehler, 2011) and required double the attention towards the front compared to simulations (Ma et al., 2020). A study on emergency driving revealed comparable stress levels in both conditions, with GSR signals more affected by controller types than visual settings (Karthaus, Wascher and Getzmann, 2018). Thus, while laboratories are optimal for emergency simulations, real-world studies are crucial for understanding routine driving dynamics. Despite significant advancements, persistent gaps affect cognitive workload assessment in driving, such as the validation of simulator-based findings in real-world conditions (Di Flumeri et al., 2018). Future studies should focus on developing intuitive user interfaces and adaptive driving technologies to reduce possible cognitive load during driver-infotainment interactions. In light of this, our study aims to investigate the cognitive workload during driver-dashboard interactions across infotainment tasks, participants, and interaction modalities (touch, voice, and hybrid) using EEG, GSR, and ET, involving experienced drivers performing tasks in both dynamic (driving) and static (stationary) conditions, using two distinct vehicle types.

RESEARCH METHODOLOGY

Participants

Fourteen participants from three age brackets —20-34 (8 participants), 35-44 (4 participants), and 45-58 years (2 participants)– were recruited to understand cognitive workload across different developmental stages. All participants had over four years of driving experience, normal visual acuity, and no cardiovascular or respiratory conditions. We developed user personas for each age group to aid recruitment and enhance understanding and empathy for users and stakeholders (Matthews, Judge and Whittaker, 2012; Nielsen and Storgaard Hansen, 2014). No within-participant differences were noted, resulting in a focus on personas and sample selection. This approach ensured a tailored recruitment process and improved the relevance of the study’s findings towards a more robust real-world usage, especially in the Indian driving context.

Experimental Devices

Electroencephalography (EEG)

Electroencephalography (EEG) measures the brain’s electrical activity non-invasively. We used a 64-channel ‘acticap slim/snap’ cap with active electrodes and electrolyte gel for high fidelity and low noise. Electrode placement followed the international 10-20 system, with three additional accelerator channels for motion-artefact correction (Homan, Herman and Purdy, 1987), crucial for in-car experiments. Signals were recorded at a sampling rate of 500 Hz (2 ms/sample) using the BrainVision Recorder software linked to the portable ‘LivAmp’ amplifier, ensuring accurate impedance optimisation and storage of acquisition parameters. LivAmp is portable, has a 64-channel capacity, and can measure up to 8 bipolar channels for additional physiological signals. LivAmp specifications included a measurement range of ±341.6 mV, < 2 μVpp input noise, and a 24-bit A/D conversion, providing precise and reliable Bluetooth-based wireless data capture. Due to in-car constraints, GSR was measured using a wristwatch sensor instead of a bipolar channel for fidelity and mobility. Cognitive workload (CWL) correlates were obtained via anatomical, cognitive, and statistical source separation (Cohen, 2014), which provided noise-separated task-induced CWL.

Galvanic Skin Resistance (GSR)

Electrodermal activity (EDA), or galvanic skin response (GSR), measures skin electrical conductance via sensors making it a vital tool for assessing cognitive workload due to its positive correlation with it (Nourbakhsh, Wang and Cheng, 2012). The Empatica E4 wristband, used for its naturalistic setting, measures sympathetic nervous system activity and heart rate through EDA and photoplethysmography (PPG) sensors. It records, administers, and broadcasts data in real time to mobile apps. It comprises an internal clock, a thermopile for skin temperature, a 3-axis accelerometer, an event mark button, an EDA sensor, and a PPG sensor. Data was sampled at 4 Hz (EDA) and 64 Hz (PPG) with a range of .01 μSiemens – 100 μSiemens. Recent literature validated its reliability and validity (McCarthy et al., 2016; Borrego et al., 2019; Schuurmans et al., 2020), especially in studies requiring accessibility and portability like ours.

Eye-Tracking (ET)

Eye tracking measures metrics like pupil dilation, blink frequency, and blink duration that correlate with cognitive exertion (Beatty and Wagoner, 1978; Klingner, 2010; Hossain and Elkins, 2016). We used the Tobii Pro Glasses 2, an advanced eye-tracking device with high-resolution cameras sampling up to 100 Hz. Its wide view allows for a comprehensive analysis of visual attention and gaze patterns using geometric calculations and an anatomical 3D eye model (Harezlak, Kasprowski and Stasch, 2014). The glasses, lightweight and non-invasive, allow natural gaze observation without obstruction. It offers a spatial accuracy of up to 0.5 degrees within a 150 cm range and includes an HD camera with a 160º angle of vision, a microphone for ambient sound capture, eye-lighting apparatus, and sensors for eye-gaze tracking. In our experiment, pupil dilation and gaze variability were observed (Sánchez-Ferrer et al., 2017; Diaz-Piedra et al., 2019) during driver-infotainment interaction via a camera tracking near-infrared light reflections in the pupil and cornea. Pupil dilation (widening of pupils) indicates cognitive engagement, while gaze variability measures eye movement frequency, with high variability suggesting exploration and low variability indicating intense focus.

Workflow

The workflow diagram provided in Figure 1. concisely outlines our study’s journey, from the philosophy and research justification to our objectives and methodology strategy.

Fig. 1. Methodology Workflow

Experimental Design

The experiment followed a between-task design where drivers underwent “3 (Modes) x 2 (Contexts) x 2 (Cars) x 4 (User Tasks).” Fourteen participants performed 52 tasks, with interaction modes including touch, voice, and hybrid, on two cars (Car 1 and 2). The two contexts, static and dynamic, were further dissociated into 3 primary conditions, viz., Car 1 (Static & Dynamic) and Car 2 (Dynamic only). The tasks encompassed navigation, calling, temperature adjustment, and radio operation, performed under each condition. Car 1 included all tasks across three modes in static and dynamic conditions, while Car 2 focused solely on navigation tasks using two modes in the dynamic condition. Multimodal approaches (EEG, GSR, and ET) provided insights into the driver’s perceived workload. Due to the experiment’s brevity, Car 1 was prioritised for both contexts, maintaining the primary goal of computing the CWL gradient across interaction modes, task categories, car-infotainment types, and driving conditions. Inter-car designs were not feasible due to design complexity and the 90-minute experimental processes, including the switching of cars. Given this, the experiment’s stimuli were not predetermined; instead, a randomised block-based design on task types was analysed across the experimental dimensions. The task locations, participant selection, experimental conditions, and creation of personas and scenarios were meticulously planned to ensure the accuracy of analysis metrics.

Task Description and Sequencing

Four categories of tasks that participants regularly perform in their daily lives were selected. They were to be executed using touch, voice, and hybrid (combination of touch and voice) commands. The tasks include:

1. Adjusting in-vehicle temperature to 18˚C, 21˚C, and 24˚C.
2. Turning on the FM radio and tuning to stations 91.1 MHz, 92.7 MHz, and 98.3 MHz.
3. Using navigation to search for a “unique location” versus a “non-unique location”.
4. Making a call to a “unique contact” versus a “non-unique contact”.

The participants were asked to drive from the track to a specific location for the navigational task and perform the remaining as instructed by the experimenter in the front pillion seat. They were asked to maintain their focus on driving and tasks akin to their regular average drive. The tasks were pseudo-randomised by the experimenter using Excel’s Mersenne Twister algorithm to mitigate task expectancy (Matsumoto and Nishimura, 1998). The experimental duration for each participant was capped at approximately 90 minutes, with a scheduled natural break of 30 seconds between tasks. In instances of fatigue or exhaustion, longer breaks or termination of the experiment were recommended.

Experimental Setup and Multimodal Data Collection

Driver behaviour and task perception can differ significantly between simulated and real environments (Philip et al., 2005; de Winter et al., 2014). To address this, our experiment was conducted on a dedicated driving track, a quasi-realistic setting, to ensure the standardisation of driving tasks under consistent road conditions. The participants were familiarised with the track, reducing workload variability and ensuring consistent baseline performance. This is based on literature (Mourant and Rockwell, 1970) and the principle of ‘visual model and visual sampling strategy’, which states that road familiarity significantly impacts distracted driving behaviour, often leading to fixation, which shifts attention away from vehicle operation and thus influences cognitive workload (Reimer et al., 2012; Strayer et al., 2015). Hence, a familiar route for participants does not confound the experiment but enhances focus on evaluating in-vehicle control tasks and estimating mental workload during tertiary activities.

Participants were briefed on the procedure and equipped with the multimodal setup within the vehicle. The primary experimenter, responsible for task conduction and the sole interactor with the driver, was seated in the front passenger seat, while another in the rear seat recorded data. Two additional observers managed equipment and monitored experimental conditions with minimal participant interaction. Participants wore the Tobii Pro Glasses and an E4 watch on their wrists, while EEG electrode impedances were consistently maintained at approximately 10kΩ throughout the study. Data collection spanned seven days, with two participants per day, across three setups: Car 1 static, Car 1 dynamic, and Car 2 dynamic, in sequential order.

Data Analysis

Our multimodal approach adopted a complementary-cooperative configuration (Sánchez-Ferrer et al., 2017; Diaz-Piedra et al., 2019), capturing diverse cognitive and affective facets of CWL and integrating sensor data types for deeper insights. Data processing (de Winter et al., 2014) involved feature-level abstraction, extracting CWL feature vectors from GSR, EEG, and ET sensors and merging them for final classification. The GSR primarily reflected affective aspects (arousal, emotion) and some cognitive components (attention fractions, information processing), while the EEG targeted cognitive aspects of working memory.

Electroencephalographic Data Processing Pipeline

Preprocessing included average re-referencing (sans default AFz ground and FCz reference electrodes) followed by Butterworth Infinite Impulse Response (IIR) filtering (0.5-60 Hz, order 8), which attenuated 50 Hz AC mains power frequency and aliased low-frequency noise from high-frequency electromagnetic sources. The signal-to-noise ratio was enhanced with Infomax Independent Component Analysis (ICA) for ocular artifact correction and Gratton-Coles correction for automated eye movement artifacts. Data segmentation, semi-automatic inspection, and manual artifact rejection (maximum 50 µV/ms voltage step, 200 µV peak-to-trough difference, minimum 0.5 µV activity ) ensured quality. Fast Fourier Transform (FFT) with 0.5 Hz resolution, Hanning windowing, and 4096 ms zero-padded segments computed neural oscillations’ band powers. Python analysis using pandas, numpy, os, re, and xlrd libraries (Gupta and Bagchi, 2024), obtaining averaged frontal theta and parietal alpha powers across tasks and conditions to compute the CWL metric (frontal theta power/parietal alpha power) (Holm et al., 2009; Pušica et al., 2024), ensuring high-quality EEG data for neurocognitive assessments.

GSR Analysis

The GSR data from the E4 Empatica wristband, sampled at 4 Hz, was used to compute mean arousal for each task, averaging across tasks and a grand average across participants to derive the CWL metric. Libraries such as pandas, datetime, and numpy (Gupta and Bagchi, 2024) were used for efficient data analysis.

Eye Tracking Analyses

Eye tracking data, obtained using near-infrared light technology with Tobii Pro Lab Software, measured average pupil dilation and gaze variability across tasks, thereby aiding task performance evaluation. Raw data was task-wise segmented, and Python computed mean gaze event durations and pupil diameters per task category, enhancing CWL metric insights. Data analysis employed pandas, datetime, and numpy libraries.

RESULTS

EEG results

The ratio of frontal theta power to parietal alpha power for all the tasks grand averaged across all the participants was calculated. The two cars cannot be compared due to experimental design limitations. However, Car 1 Gradient is considered, wherein the gradient is simply the CWL in the static condition subtracted from CWL in the dynamic condition, and it can be visualised in Figure 2. (‘x-axis’ are the tasks):

Fig. 2. EEG CWL Car 1 Gradient Percentage for All Tasks

The EEG cognitive workload (CWL) analysis, categorised by task and modality, shows that the CWL during driving is generally higher than in static conditions. Tasks such as calling, radio change, temperature adjustment via voice, and temperature adjustment via touch exhibited minor negative CWL gradients (less than 5%), indicating comparable mental workloads in both stationary and driving scenarios. Additionally, tasks like calling and navigation hybrid, navigation via voice, and radio hybrid showed minor positive CWL gradients (around or less than 5%), indicating minimal additional workload during driving. In contrast, tasks involving navigation-via-touch (~14%), calling-via-touch (~11%), calling hybrid (~5%), and radio-via-touch (~5%) resulted in significantly elevated workloads during driving. These findings suggest the need for redesign, especially for navigation and calling tasks, to reduce touch interaction and enhance driver safety. The analysis further revealed that radio-via-touch tasks imposed a greater cognitive burden than radio-via-voice tasks, underscoring the necessity of prioritising voice interactions. Generally, voice-based interactions demonstrated lower CWL across tasks, except for temperature adjustments, where touch proved more efficient, likely due to the straightforward nature of temperature control interfaces.

GSR Results

We computed mean GSR-based EDA arousal for all the tasks grand averaged over all the participants. Again, we can not compare and contrast the Car 1 and 2 conditions owing to experimental design limitations, including for GSR. However, Car 1 Gradient is considered, wherein the gradient is simply the mean arousal of the static condition subtracted from the mean arousal in the dynamic condition, and this can be seen in Figure 3 (on ‘x-axis’ are the tasks):

Fig. 3. Car 1 GSR Gradient Percentage for All Tasks

The GSR CWL results indicated higher physiological arousal during driving compared to static conditions for all tasks, which aligns with expectations. Touch modality exhibited lower physiological arousal than voice or hybrid modalities for all tasks, suggesting that touch may be less physiologically demanding but potentially be more cognitively taxing. This cognitive-physiological discrepancy warrants further investigation. The tasks of calling hybrid (~17%), navigation voice (~12%), temperature voice (~14%), and radio hybrid (~12%) induced significantly higher physiological arousal during driving. These tasks require redesign to mitigate heightened arousal, particularly for enhancing driver safety. The hybrid calling and navigation-calling voice features should be designed to minimise reliance on touch mode. Other tasks resulted in relatively lower arousal during driving conditions. The workload for these tasks should be evaluated in conjunction with other modality metrics.

ET Results

CWL prediction via ET analysis was based on mean pupil dilation and gaze variability. While pupil dilation during static conditions reliably indicated driver engagement, serving as a marker of involvement, the reduced gaze variability suggested a higher workload.

Mean Pupil Dilation

We computed the mean pupil dilation for all tasks, grand-averaged across participants (excluding Car 2 Dynamic Condition). Car 1 ET Gradient was used, defined as the CWL in the static condition minus the CWL in the dynamic condition, differing from EEG and GSR gradients due to expected attenuated dilation during the driving condition. The gradient percentage across tasks and conditions is visualised in the graph presented in Figure 4A (the x-axis represents tasks; the y-axis refers to pupil dilation gradient percentage).

Fig. 4A Pupil Dilation Gradient for Car 1 Tasks; Fig 4 B Mean Pupil Dilation Gradient Percentage for all Modalities

From Figure 4A, we can infer that most infotainment interactions show higher pupil dilation gradients, indicating decreased ocular cognitive load, especially for radio and temperature adjustments. The only exception is the navigation task employing touch modality. For further clarity, cross-modality mean pupil dilation gradients for Car 1 were also analysed. However, gradient here refers to the mean pupil dilation in the dynamic condition minus that in the static condition for each modality. The gradient percentage across tasks and conditions can be observed in Figure 4B where the x-axis refers to the modalities and the y-axis refers to the pupil dilation gradient percentage.

In all modalities, mean pupil dilation is lower for the dynamic condition, indicating decreased engagement while driving. In terms of gradient percentage, the largest decrease is in the voice modality (-7.14%), suggesting lower engagement, while the touch modality shows the smallest decrease (-5.20%), maintaining relatively higher engagement.

Mean Gaze Variability

Mean gaze variability for all tasks, grand-averaged across participants, was calculated (excluding Car 2 conditions). The Car 1 ET Gradient was calculated as the CWL in the static condition minus the CWL in the dynamic condition. Again, this differs from EEG and GSR, owing to expected attenuation in variability during driving. The gradient percentage across tasks is given in Figure 5A with the various tasks on the x-axis and the y-axis stating gaze variability gradient percentage.

Fig. 5A Pupil Dilation Gradient for Car 1 Tasks and Fig. 5B Mean Gaze Variability Gradient Percentage Across Modalities

We can see in Figure 5A that hybrid interactions demonstrate the highest gaze variability, particularly for calling and radio tasks, with a notable 101.42% increase for calling. Touch interactions generally result in higher gaze variability for navigation (73.89%) and calling tasks, reflecting significant engagement. Voice interactions, on the other hand, maintain lower variability across tasks, with the highest increase observed for temperature (101.42%), but generally show less engagement, especially for calling (22.95%) and radio (21.35%). Moreso, cross-modality mean gaze variability gradients across each modality for Car 1 were also analysed. The percentage across modalities (‘dynamic’ minus ‘static’) is visualised in Figure 5B, with the x-axis referring to modalities and the y-axis to mean gaze variability gradient percentage.

In Figure 5B, hybrid mode shows the largest decrease in engagement during driving, with a gradient percentage of -34.87%. The touch mode follows with a -28.86% decrease, while the voice mode has the smallest decrease at -18.09%, indicating relatively higher engagement. Overall, the hybrid mode exhibits the most significant reduction in gaze variability.

Heat Maps

It can be observed from the heat maps in Figure 6 that drivers focus heavily on in-car controls during static conditions but shift to the road, showing high engagement with the driving task and minimal interaction with the in-car controls during dynamic conditions. Across modalities, touch tasks demand substantial visual attention to the in-car console, while voice tasks let drivers keep their eyes on the road. Hybrid tasks require a balance between road and console attention, indicating a moderate engagement with both driving and in-car interactions.

Fig. 6. Heat Maps: Heat maps across conditions, such as A) Car 1 Static condition and B) Car 1 Dynamic condition. Cross-modality heat maps including C) Touch-task type, D) Voice-task type and E) Hybrid-task type

Visual Responsivity Across Interaction Modalities in Diverse Driving Conditions

Touch and hybrid conditions elicited higher pupil dilation due to menu interaction demands in both static and dynamic settings. Hybrid tasks showed increased gaze variability during static conditions, reflecting heightened cognitive workload from menu interaction. During driving, higher gaze variability in touch and voice modes suggested increased attention demands due to menu navigation and delayed voice responses. Moreover, average pupil dilation was higher during static conditions, indicating engagement with surrounding activities, while during driving, drivers were more focused on the task at hand. Higher gaze variability during static conditions reflected multiple points of engagement, whereas driving required sustained attention on the road.

Task-Specific Analysis of Visual Attention Dynamics Across Interaction Modalities

Task-specific observations further demonstrated that navigation required more attention, with higher pupil dilation in touch mode. Similarly, the temperature setting showed higher pupil dilation and gaze variability in touch mode. The radio channel setting task had higher pupil dilation in touch mode, with increased gaze variability during setup. The calling task reflected higher pupil dilation in touch mode, with higher gaze variability in hybrid mode.

DISCUSSION

In the evolving automotive landscape, assessing the impact of advanced infotainment systems on driver attention and performance is crucial. Given the risk of driver inattention and accidents caused by distractions from road conditions and dashboard devices, it is essential to enhance both driver safety and user experience. However, there is limited scientific data on the cognitive efficiency of different designs in Indian driving scenarios. This study examines the cognitive workload of touch-based, voice-based, and hybrid infotainment interactions using EEG, GSR, and eye-tracking in static and dynamic driving conditions.
The data from this study reveal nuanced insights into cognitive workload during interactions with vehicle infotainment systems. EEG data showed that driving increased CWL compared to static conditions, with touch-based interactions causing the highest cognitive load—14% higher during navigation tasks. This aligns with research indicating that touch interfaces demand substantial visual and manual attention, contributing to elevated mental effort (Rydström, Broström and Bengtsson, 2012; Ferris, Suh and Miles, 2016; Suh and Ferris, 2018). The need for precise motor control and constant visual engagement makes touch interactions particularly taxing for drivers. Conversely, voice-based interactions resulted in lower CWL than touch or hybrid modalities, with calling tasks showing the least CWL gradient in both voice and touch interactions, indicating a consistent level of mental effort across these modes. Despite freeing the hands, voice commands require considerable cognitive resources, supporting previous studies on the distraction challenges of auditory commands (Chaparro, Wood and Carberry, 2005; Just, Keller and Cynkar, 2008; Wu et al., 2016; Strayer et al., 2019). GSR results revealed higher physiological arousal during driving versus static conditions for all tasks. Touch tasks generally had lower arousal than voice or hybrid tasks, except for radio, suggesting touch may be less physiologically demanding but more cognitively taxing. Tasks like hybrid calling, voice navigation, and touch radio showed higher arousal, indicating potential safety concerns. Eye-tracking metrics, including mean pupil dilation and gaze variability, highlighted that drivers engage more with infotainment during static conditions but shift focus primarily to driving tasks while on the road. Specifically, pupil dilation showed higher gradients for touch interactions, indicating increased cognitive load, particularly for radio and temperature adjustments. Voice interactions exhibited moderate engagement, while hybrid interactions varied, with calling being less engaging and navigation more so. Gaze variability was notably higher during static conditions compared to driving, reflecting distinct cognitive focuses across interaction modes. Hybrid mode showed the highest gaze variability and distraction levels—48.12% higher than voice and 17.26% higher than touch—suggesting a level of complexity that overwhelms cognitive processing.

Our findings highlight the need for optimised infotainment designs. Using hybrid modes for navigation and voice commands for simpler tasks like radio tuning, especially when steering-mounted controls are unavailable, can balance the cognitive load by distributing tasks across different sensory modalities. Physical switches for calling functions on the steering wheel could also reduce cognitive demand. Our study’s multimodal approach provided reliable CWL measurements but was limited by a broad experimental design with limited task trials, which affected result generalizability. Sequencing issues led to the exclusion of Car 2’s dynamic condition due to participant fatigue, and protocol disruptions compromised data reliability. Future research should use more sophisticated randomisation, increase task trials, and integrate VR setups with neurophysiological measures for detailed insights and improved ecological validity. Developing adaptive infotainment systems that adjust to real-time cognitive workload, incorporating machine learning, and expanding participant diversity could improve safety and user experience. Establishing industry standards for infotainment system design will further promote safe and user-friendly interfaces.

In conclusion, this study provides valuable insights into the cognitive workload of different infotainment interaction modes. Touch-based interactions are visually and manually demanding, voice-based interactions increase auditory distraction, and hybrid modes present the highest cognitive challenge. Designing Human-Machine Interfaces that minimise cognitive load and enhance driver safety is essential, with future advancements promising more adaptive and intuitive systems.

CONCLUSION

Our study reveals significant cognitive workload in touch-based and hybrid infotainment interactions, with touch navigation tasks showing a 14% increase. Voice commands reduce manual effort but increase distraction, while hybrid modes amplify distraction the most. These insights are crucial for developing safer, intuitive vehicle interfaces. Integrating neurophysiological measurements into the design can optimise workload, enhancing driver safety and user experience. Establishing industry standards based on empirical research will promote best practices in infotainment design. Our findings underscore the importance of reducing cognitive load in driving contexts. Future research should explore innovative approaches to enhance driving safety as technologies evolve. This research contributes to advancing automotive HCI, ensuring intelligent and safe driving experiences that improve overall performance and safety.

BIBLIOGRAPHY

1. Aricò, P. et al. (2018) ‘Passive BCI beyond the lab: current trends and future directions’, Physiological measurement, 39(8), p. 08TR02.
2. Beatty, J. and Wagoner, B.L. (1978) ‘Pupillometric Signs of Brain Activation Vary with Level of Cognitive Processing’, Science [Preprint]. Available at: https://doi.org/10.1126/science.628837.
3. Borrego, A. et al. (2019) ‘Reliability of the Empatica E4 wristband to measure electrodermal activity to emotional stimuli’, in 2019 International Conference on Virtual Rehabilitation (ICVR). 2019 International Conference on Virtual Rehabilitation (ICVR), IEEE, pp. 1–2.
4. Boucsein, W. (2012) Electrodermal Activity. Springer Science & Business Media.
5. Cernea, D. et al. (2012) ‘Controlling In-Vehicle Systems with a Commercial EEG Headset: Performance and Cognitive Load’, in Visualization of Large and Unstructured Data Sets: Applications in Geospatial Planning, Modeling and Engineering – Proceedings of IRTG 1131 Workshop 2011. Schloss Dagstuhl – Leibniz-Zentrum für Informatik, p. 122.
6. Chaparro, A., Wood, J.M. and Carberry, T. (2005) ‘Effects of Age and Auditory and Visual Dual Tasks on Closed-Road Driving Performance’, Optometry and vision science: official publication of the American Academy of Optometry, 82(8), p. 747.
7. Cohen, M.X. (2014) Analyzing Neural Time Series Data: Theory and Practice. The MIT Press.
8. Debie, E. et al. (2021) ‘Multimodal Fusion for Objective Assessment of Cognitive Workload: A Review’, IEEE Transactions on Cybernetics, 51, pp. 1542–1555.
9. Diaz-Piedra, C. et al. (2019) ‘the effects of flight complexity on gaze entropy: An experimental study with fighter pilots’, Applied ergonomics, 77, pp. 92–99.
10. Diaz-Piedra, C., Sebastián, M.V. and Di Stasi, L.L. (2020) ‘EEG Theta Power Activity Reflects Workload among Army Combat Drivers: An Experimental Study’, Brain Sciences, 10(4), p. 199.
11. Di Flumeri, G. et al. (2018) ‘EEG-Based Mental Workload Neurometric to Evaluate the Impact of Different Traffic and Road Conditions in Real Driving Settings’, Frontiers in human neuroscience, 12, p. 509.
12. Ferris, T., Suh, Y. and Miles, J. (2016) Investigating the roles of touchscreen and physical control interface characteristics on driver distraction and multitasking performance. ATLAS-2015-08. Center for Advancing Transportation Leadership and Safety (ATLAS Center); University Transportation Centers Program (U.S.). Available at: https://rosap.ntl.bts.gov/view/dot/30788.
13. Figalová, N. et al. (2022) ‘Ambient Light Conveying Reliability Improves Drivers’ Takeover Performance without Increasing Mental Workload’, Multimodal Technologies and Interaction, 6(9), p. 73.
14. Freeman, F.G. et al. (2000) ‘Evaluation of a Psychophysiologically Controlled Adaptive Automation System, Using Performance on a Tracking Task’, Applied psychophysiology and biofeedback, 25(2), pp. 103–115.
15. Gevins, A. et al. (1997) ‘High-resolution EEG mapping of cortical activation related to working memory: effects of task difficulty, type of processing, and practice’, Cerebral cortex, 7(4), pp. 374–385.
16. Göhring, D. et al. (2013) ‘Semi-autonomous car control using brain computer interfaces’, in S. Lee, K.-J. Yoon, and J. Lee (eds) Intelligent Autonomous Systems 12: Berlin: Springer, pp. 393–408.
17. Gopher, D. and Braune, R. (1984) ‘On the Psychophysics of Workload: Why Bother with Subjective Measures?’, Human factors [Preprint]. Available at: https://doi.org/10.1177/001872088402600504.
18. Gupta, P. and Bagchi, A. (2024) ‘Introduction to Pandas’, Essentials of Python for Artificial Intelligence and Machine Learning, pp. 161–196.
19. Hancock, P.A. and Chignell, M. (1989) Intelligent Interfaces: Theory, Research, and Design. North Holland.
20. Hancock, P.A. and Chignell, M.H. (1988) ‘Mental Workload Dynamics in Adaptive Interface Design’, IEEE Transactions on Systems Man and Cybernetics, 18(4), pp. 647–658.
21. Hanowski, R.J. et al. (2005) the 100-Car Naturalistic Driving Study: A Descriptive Analysis of Light Vehicle-Heavy Vehicle Interactions from the Light Vehicle Driver’s Perspective. US Department of Transportation. Federal Motor Carrier Safety Administration.
22. Harezlak, K., Kasprowski, P. and Stasch, M. (2014) ‘Towards Accurate Eye Tracker Calibration – Methods and Procedures’, Procedia computer science, 35, pp. 1073–1081.
23. Hart, S.G. (2006) ‘Nasa-Task Load Index (NASA-TLX); 20 Years Later’, Proceedings of the Human Factors and Ergonomics Society … Annual Meeting Human Factors and Ergonomics Society. Meeting [Preprint]. Available at: https://doi.org/10.1177/154193120605000909.
24. Holm, A. et al. (2009) ‘Estimating brain load from the EEG’, The Scientific World Journal [Preprint]. Available at: https://doi.org/10.1100/tsw.2009.83.
25. Homan, R.W., Herman, J. and Purdy, P. (1987) ‘Cerebral location of international 10–20 system electrode placement’, Electroencephalography and clinical neurophysiology, 66(4), pp. 376–382.
26. Hossain, G. and Elkins, J. (2016) ‘When does an easy task become hard? A systematic review of human task-evoked pupillary dynamics versus cognitive efforts’, Neural computing & applications, 30(1), pp. 29–43.
27. Islam, M.R. et al. (2020) ‘A Novel Mutual Information Based Feature Set for Drivers’ Mental Workload Evaluation Using Machine Learning’, Brain Sciences, 10(8), p. 551.
28. Johannes, B. and Gaillard, A.W.K. (2014) ‘A methodology to compensate for individual differences in psychophysiological assessment’, Biological psychology, 96, pp. 77–85.
29. Just, M.A., Keller, T.A. and Cynkar, J. (2008) ‘A decrease in brain activation associated with driving when listening to someone speak’, Brain research, 1205, pp. 70–80.
30. Karthaus, M., Wascher, E. and Getzmann, S. (2018) ‘Effects of Visual and Acoustic Distraction on Driving Behavior and EEG in Young and Older Car Drivers: A Driving Simulation Study’, Frontiers in aging neuroscience, 10, p. 418080.
31. Klauer, K.C. and Mierke, J. (2005) ‘Task-set inertia, attitude accessibility, and compatibility-order effects: new evidence for a task-set switching account of the implicit association test effect’, Personality & social psychology bulletin, 31(2), pp. 208–217.
32. Klingner, J. (2010) Measuring Cognitive Load during Visual Tasks by Combining Pupillometry and Eye Tracking. Edited by P. Hanrahan and B. Tversky. PhD. Stanford University. Available at: https://search.proquest.com/openview/985f4f7cce85bae7ff453722383b83ea/1?pq-origsite=gscholar&cbl=18750&diss=y.
33. Kohlmorgen, J. et al. (2007) ‘Improving Human Performance in a Real Operating Environment through Real-Time Mental Workload Detection’, in G. Dornhege et al. (eds) Toward Brain-Computer Interfacing. MIT Press Direct.
34. Kruger, J.-L. and Doherty, S. (2016) ‘Measuring cognitive load in the presence of educational video: Towards a multimodal methodology’, Australasian Journal of Educational Technology, 32(6). Available at: https://doi.org/10.14742/ajet.3084.
35. Kumar, J. and Kumar, J. (2016) ‘Affective Modelling of Users in HCI Using EEG’, Procedia computer science, 84, pp. 107–114.
36. Kumar, N. and Kumar, J. (2016) ‘Measurement of Cognitive Load in HCI Systems Using EEG Power Spectrum: An Experimental Study’, Procedia computer science, 84, pp. 70–78.
37. Lee, J.D. (2017) Driver Distraction and Inattention: Advances in Research and Countermeasures, Volume 1. CRC Press.
38. Lei, S. and Roetting, M. (2011) ‘Influence of Task Combination on EEG Spectrum Modulation for Driver Workload Estimation’, Human factors [Preprint]. Available at: https://doi.org/10.1177/0018720811400601.
39. Lei, S., Welke, S. and Roetting, M. (2009) ‘Driver’s mental workload assessment using EEG data in a dual task paradigm’, 2009. Available at: https://www-nrd.nhtsa.dot.gov/pdf/ESV/ Proceedings/ 21/ Track%2027%20Written.pdf.
40. Lin, C.-T., Ko, L.-W. and Shen, T.-K. (2009) ‘Computational intelligent brain computer interaction and its applications on driving cognition’, IEEE Computational Intelligence Magazine, 4(4), pp. 32–46.
41. Lobo, J.L. et al. (2016) ‘Cognitive workload classification using eye-tracking and EEG data’, in. HCI-Aero ’16: International Conference on Human-Computer Interaction in Aerospace 2016, NY, United States: Association for Computing Machinery, pp. 1–8.
42. Ma, Q.G. et al. (2012) ‘Mental Workload Analysis during the Production Process: EEG and GSR Activity’, Applied Mechanics and Materials, 220-223, pp. 193–197.
43. Matsumoto, M. and Nishimura, T. (1998) ‘Mersenne twister: a 623-dimensionally equidistributed uniform pseudo-random number generator’, ACM Transactions on Modeling and Computer Simulation, 8(1), pp. 3–30.
44. Matthews, T., Judge, T. and Whittaker, S. (2012) ‘How do designers and user experience professionals actually perceive and use personas?’, in CHI ’12: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. CHI ’12: CHI Conference on Human Factors in Computing Systems, New York, United States: Association for Computing Machinery, pp. 1219–1228.
45. Ma, Y. et al. (2020) ‘Drivers’ Visual Attention Characteristics under Different Cognitive Workloads: An On-Road Driving Behavior Study’, International journal of environmental research and public health, 17(15), p. 5366.
46. McCarthy, C. et al. (2016) ‘Validation of the Empatica E4 wristband’, in 2016 IEEE EMBS International Student Conference (ISC). IEEE EMBS International Student Conference (ISC), IEEE, pp. 1–4.
47. Mourant, R.R. and Rockwell, T.H. (1970) ‘Mapping eye-movement patterns to the visual scene in driving: an exploratory study’, Human factors, 12(1), pp. 81–87.
48. Nabatilan, L.B. et al. (2011) ‘Effect of driving experience on visual behavior and driving performance under different driving conditions’, Cognition, Technology & Work, 14, pp. 355–363.
49. Nielsen, L. and Storgaard Hansen, K. (2014) ‘Personas is applicable- A study on the use of personas in Denmark’, in SIGCHI Conference on Human Factors in Computing Systems. CHI ’14: CHI Conference on Human Factors in Computing Systems, New York, United States: Association for Computing Machinery, pp. 1665–1674.
50. Nourbakhsh, N., Wang, Y. and Cheng, F. (2012) ‘Using galvanic skin response for cognitive load measurement in arithmetic and reading tasks’, in. Australasian Computer-Human Interaction Conference. Available at: https://doi.org/10.1145/2414536.2414602.
51. Philip, P. et al. (2005) ‘Fatigue, Sleepiness, and Performance in Simulated Versus Real Driving Conditions’, Sleep, 28(12), pp. 1511–1516.
52. Plechawska-Wójcik, M. et al. (2019) ‘A Three-Class Classification of Cognitive Workload Based on EEG Spectral Data’, NATO Advanced Science Institutes series E: Applied sciences, 9(24), p. 5340.
53. Pollmann, K. et al. (2019) ‘How to Work in the Car of the Future?: A Neuroergonomical Study Assessing Concentration, Performance and Workload Based on Subjective, Behavioral and Neurophysiological Insights’, in CHI ’19: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. CHI ’19: CHI Conference on Human Factors in Computing Systems, New York, US: Association for Computing Machinery, pp. 1–14.
54. Prinzel, L.J. et al. (2000) ‘A Closed-Loop System for Examining Psychophysiological Measures for Adaptive Task Allocation’, The International journal of aviation psychology [Preprint]. Available at: https://doi.org/10.1207/S15327108IJAP1004_6.
55. Pušica, M. et al. (2024) ‘Mental Workload Classification and Tasks Detection in Multitasking: Deep Learning Insights from EEG Study’, Brain Sciences, 14(2), p. 149.
56. Putze, F., Jarvis, J.-P. and Schultz, T. (2010) ‘Multimodal Recognition of Cognitive Workload for Multitasking in the Car’, in 2010 20th International Conference on Pattern Recognition. International Conference on Pattern Recognition, IEEE, pp. {3748–3751.
57. Regan, M.A., Lee, J.D. and Young, K. (2008) Driver Distraction: Theory, Effects, and Mitigation. CRC Press.
58. Reimer, B. et al. (2012) ‘A Field Study on the Impact of Variations in Short-Term Memory Demands on Drivers’ Visual Attention and Driving Performance Across Three Age Groups’, Human factors [Preprint]. Available at: https://doi.org/10.1177/0018720812437274.
59. Reimer, B. and Mehler, B. (2011) ‘the impact of cognitive workload on physiological arousal in young adult drivers: a field study and simulation validation’, Ergonomics, 54(10), pp. 932–942.
60. Rydström, A., Broström, R. and Bengtsson, P. (2012) ‘A comparison of two contemporary types of in-car multifunctional interfaces’, Applied ergonomics, 43(3), pp. 507–514.
61. Sánchez-Ferrer, M.L. et al. (2017) ‘Use of Eye Tracking as an Innovative Instructional Method in Surgical Human Anatomy’, Journal of surgical education, 74(4), pp. 668–673.
62. Schuurmans, A.A.T. et al. (2020) ‘Validity of the Empatica E4 Wristband to Measure Heart Rate Variability (HRV) Parameters: a Comparison to Electrocardiography (ECG)’, Journal of medical systems, 44(11), pp. 1–11.
63. Solís-Marcos, I. and Kircher, K. (2018) ‘Event-related potentials as indices of mental workload while using an in-vehicle information system’, Cognition, technology & work, 21(1), pp. 55–67.
64. Strayer, D.L. et al. (2015) ‘Assessing Cognitive Distraction in the Automobile’, Human factors [Preprint]. Available at: https://doi.org/10.1177/0018720815575149.
65. Strayer, D.L. et al. (2019) ‘Assessing the visual and cognitive demands of in-vehicle information systems’, Cognitive research: principles and implications, 4(1), p. 18.
66. Sugimoto, F. et al. (2020) ‘Effects of one-pedal automobile operation on the driver’s emotional state and cognitive workload’, Applied ergonomics, 88, p. 103179.
67. Suh, Y. and Ferris, T.K. (2018) ‘On-Road Evaluation of In-vehicle Interface Characteristics and Their Effects on Performance of Visual Detection on the Road and Manual Entry’, Human factors [Preprint]. Available at: https://doi.org/10.1177/0018720818790841.
68. Summala, H. (2007) ‘Towards Understanding Motivational and Emotional Factors in Driver Behaviour: Comfort through Satisficing’, in P.C. Cacciabue (ed.) Modelling Driver Behaviour in Automotive Environments. London: Springer, pp. 189–207.
69. Suoja, K. et al. (2018) ‘Application for pre-processing and visualization of electrodermal activity wearable data’, in EMBEC & NBC 2017. EMBEC & NBC 2017: Joint Conference of the European Medical and Biological Engineering Conference (EMBEC) and the Nordic-Baltic Conference on Biomedical Engineering and Medical Physics (NBC), Springer Singapore, pp. 93–96.
70. Tsai, Y.-F. et al. (2007) ‘Task performance and eye activity: predicting behavior relating to cognitive workload’, Aviation, space, and environmental medicine, 78(5 Suppl), pp. B176–85.
71. Wall, T.D. et al. (2004) ‘ON THE VALIDITY OF SUBJECTIVE MEASURES OF COMPANY PERFORMANCE’, Personnel psychology, 57(1), pp. 95–118.
    Wierwille, W.W. and Thomas, E.F. (1993) ‘Recommendations for Mental Workload Measurement in a Test and Evaluation Environment’, Human factors, 35(2), pp. 263–281.
72. De Winter, J.C.F. et al. (2014) ‘Effects of adaptive cruise control and highly automated driving on workload and situation awareness: A review of the empirical evidence’, Transportation research. Part F, Traffic psychology and behaviour, 27, pp. 196–217.
73. Wu, J. et al. (2016) ‘Impact of In-vehicle Voice Control Systems on Driver Distraction’, Proceedings of the Human Factors and Ergonomics Society … Annual Meeting Human Factors and Ergonomics Society. Meeting [Preprint]. Available at: https://doi.org/10.1177/1541931215591342.