Evaluating Facial Expression Recognition for Emotion-Adaptive Music Recommendation: A Retrospective Analysis of a Mobile Prototype

Facial expression recognition (FER) can provide affective input for adaptive applications, but its usefulness depends on whether the recognition component produces accurate and consistent outputs. This study retrospectively evaluates the FER component of Emoosic, an Android mobile prototype originally developed to select music according to recognized facial-expression categories. The analysis was restricted to 30 retained static images from the original evaluation, comprising 10 images each for happiness, sadness and anger. Each image had been processed twice using the original Affdex-based recognition setup. Outcomes were analyzed descriptively in terms of correct classification, incorrect classification, non-detection, detection coverage, accuracy among detected cases and agreement across repeated attempts. In the first attempt, 13 of 30 images were correctly classified (43.3%), four were incorrectly classified (13.3%) and 13 produced no emotion output (43.3%), resulting in 56.7% detection coverage. In the second attempt, correct classification increased to 17 images (56.7%), incorrect classification decreased to two (6.7%) and non-detection decreased to 11 images (36.7%), with 63.3% detection coverage. Happiness showed the strongest category-level performance, with 70.0% correct classification in both attempts. Sadness and anger showed lower and more variable recognition. Overall, 26 of 30 images produced identical outcomes across both attempts, corresponding to 86.7% repeated-attempt agreement. The findings indicate that the prototype could provide usable facial expression input for emotion-adaptive behaviour, but substantial non-detection and category-dependent variability limit its reliability. For adaptive systems, recognition performance should therefore be considered together with detection coverage and output consistency rather than accuracy among successfully detected cases alone. This is especially important when recognition output directly triggers a downstream recommendation.

facial expression recognition, affective computing, emotion-adaptive systems, music recommendation, retrospective analysis

1. Barrett, L. F., Adolphs, R., Marsella, S., Martinez, A. M., & Pollak, S. D. (2019). Emotional expressions reconsidered: Challenges to inferring emotion from human facial movements. Psychological Science in the Public Interest, 20(1), 1–68. https://doi.org/10.1177/1529100619832930 [Google Scholar] [Crossref]

2. Chheda, R., Bohara, D., Shetty, R., Trivedi, S., & Karani, R. (2023). Music recommendation based on affective image content analysis. Procedia Computer Science, 218, 383–392. https://doi.org/10.1016/j.procs.2023.01.021 [Google Scholar] [Crossref]

3. Ekman, P., & Friesen, W. V. (1978). Facial action coding system: A technique for the measurement of facial movement. Consulting Psychologists Press. [Google Scholar] [Crossref]

4. Goel, S., Jara-Ettinger, J., Ong, D. C., & Gendron, M. (2024). Face and context integration in emotion inference is limited and variable across categories and individuals. Nature Communications, 15(1), 2443. https://doi.org/10.1038/s41467-024-46670-5 [Google Scholar] [Crossref]

5. Ko, B. C. (2018). A brief review of facial emotion recognition based on visual information. Sensors, 18(2), 401. https://doi.org/10.3390/s18020401 [Google Scholar] [Crossref]

6. Li, S., & Deng, W. (2022). Deep facial expression recognition: A survey. IEEE Transactions on Affective Computing, 13(3), 1195–1215. https://doi.org/10.1109/TAFFC.2020.2981446 [Google Scholar] [Crossref]

7. McDuff, D., Mahmoud, A., Mavadati, M., Amr, M., Turcot, J., & El Kaliouby, R. (2016). AFFDEX SDK: A cross-platform real-time multi-face expression recognition toolkit. In Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems (pp. 3723–3726). ACM. https://doi.org/10.1145/2851581.2890247 [Google Scholar] [Crossref]

8. Picard, R. W. (1997). Affective computing. MIT Press. https://doi.org/10.7551/mitpress/1140.001.0001 [Google Scholar] [Crossref]

9. Sajjad, M., Ullah, F. U. M., Ullah, M., Christodoulou, G., Alaya Cheikh, F., Hijji, M., Muhammad, K., & Rodrigues, J. J. P. C. (2023). A comprehensive survey on deep facial expression recognition: Challenges, applications, and future guidelines. Alexandria Engineering Journal, 68, 817–840. https://doi.org/10.1016/j.aej.2023.01.017 [Google Scholar] [Crossref]

10. Tran, H., Le, T., Do, A., Vu, T., Bogaerts, S., & Howard, B. (2023). Emotion-aware music recommendation. Proceedings of the AAAI Conference on Artificial Intelligence, 37(13), 16087–16095. https://doi.org/10.1609/aaai.v37i13.26911 [Google Scholar] [Crossref]

11. Ullah, S., Ou, J., Xie, Y., & Tian, W. (2024). Facial expression recognition (FER) survey: A vision, architectural elements, and future directions. PeerJ Computer Science, 10, e2024. https://doi.org/10.7717/peerj-cs.2024 [Google Scholar] [Crossref]

12. Yang, Y.-H., & Chen, H. H. (2012). Machine recognition of music emotion: A review. ACM Transactions on Intelligent Systems and Technology, 3(3), Article 40. https://doi.org/10.1145/2168752.2168754 [Google Scholar] [Crossref]

• What the Desert Fathers Teach Data Scientists: Ancient Ascetic Principles for Ethical Machine-Learning Practice
• Comparative Analysis of Some Machine Learning Algorithms for the Classification of Ransomware
• Comparative Performance Analysis of Some Priority Queue Variants in Dijkstra’s Algorithm
• Transfer Learning in Detecting E-Assessment Malpractice from a Proctored Video Recordings.
• Dual-Modal Detection of Parkinson’s Disease: A Clinical Framework and Deep Learning Approach Using NeuroParkNet