Color-Coded Emotional Learning Serious Games for Children with Autism: Foregrounding Facial Expression Recognition & Accessibility
Tahani Jaser Alahmadi, Haifa Albahli, Modhi Alqahtani, Hind Bin Mehaya, Imtinan Alqahtani, Joory Alotaibi · 2025 · Proceedings of the 22nd International Web for All Conference (W4A) · doi:10.1145/3744257.3744274
Summary
This technical note presents Muheet, a mobile application that uses color-emotion associations and AI-based facial expression recognition to help children with autism spectrum disorder (ASD) understand and recognize emotions. ASD affects approximately 1.5-2% of the global population and impairs the ability to recognize and respond to emotions in social interactions, with difficulties understanding sarcasm, processing information, and expressing feelings. Traditional methods for teaching emotions to autistic children, such as flashcards pairing facial expressions with emotion labels, rely on memorization rather than fostering deeper emotional understanding. Muheet takes a novel approach inspired by synesthesia — the phenomenon where one sensory input is linked to another sensory experience — by mapping emotions to specific colors based on validated color-emotion research. The idea originated from an autistic child whose mother encouraged him to associate emotions with colors, and over time the child developed an expanding emotional color palette. The application consists of two phases: a learning phase where children learn the meaning of different emotions and their associated colors, and an evaluation phase with three interactive games — a scenario game, a facial expression emotion matching game, and a coloring game. The facial expression recognition component uses TensorFlow.js to classify emotions from webcam images (happy, sad, angry, etc.). The app was developed using agile methodology and designed following WCAG 2.2 guidelines, emphasizing the "Understandable" principle.
Key findings
A survey published in September 2024 received 286 responses from parents of children with autism and specialists. The results revealed that 79% of respondents reported difficulty in communication with their autistic children, and 88% confirmed their children struggle to express their feelings. Significantly, 42% of participants indicated that their children do not express their feelings at all. The most common communication methods used were acting (30%), speaking (28%), pictures (26%), and coloring (14%). Seventy-eight percent of respondents expressed confidence in the Muheet methodology and its potential to improve emotional communication for children with autism. The application has not yet been tested with children with ASD directly — functional testing has focused on ensuring all user paths work correctly, and user testing at an Autism Center is planned as future work. The facial expression recognition system was built using TensorFlow.js and trained on datasets of classified facial expressions, though the paper acknowledges the challenge that the same emotion can manifest differently across individuals (e.g., anger and disgust can appear similar).
Relevance
This paper addresses an important area of cognitive accessibility: supporting emotional understanding and social communication for children with autism. The color-emotion association approach offers a creative alternative to traditional memorization-based methods, potentially building more transferable emotional understanding that can be applied to novel social situations. For accessibility practitioners, the project demonstrates how AI technologies like facial expression recognition can be integrated into educational tools for neurodiverse users, while also highlighting the importance of following WCAG guidelines even in specialized applications. The survey data confirming that 42% of autistic children do not express feelings at all underscores the scale of need for such tools. However, the paper has significant limitations: no user testing with autistic children has been conducted yet, the color-emotion mappings may not be culturally universal, and the facial expression recognition component faces accuracy challenges with the natural variability of emotional expression. The planned user testing at an Autism Center will be critical for validating the approach.
Tags: autism · emotional learning · serious games · facial expression recognition · machine learning · cognitive accessibility · children · color-emotion association
Standards referenced: WCAG 2.2