From Daily Song to Daily Self: Supporting Emotional Growth of Deaf and Hard-of-Hearing Individuals through Generative AI Songwriting
Youjin Choi, JinYoung Yoo, JaeYoung Moon, Yoonjae Kim, Eun Young Lee, Jennifer G Kim, Jin-Hyuk Hong · 2026 · Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems (CHI '26) · doi:10.1145/3772318.3791204
Summary
This CHI 2026 paper introduces SoulNote, a generative-AI songwriting system designed to support sustained emotional growth for Deaf and Hard-of-Hearing (DHH) individuals through music-based journaling. The authors argue that prior music-GenAI accessibility work has focused on single-session evaluations, which cannot capture the longitudinal emotional processing that journaling and therapeutic songwriting provide. SoulNote is built as a web application (Next.js + Flask + Supabase) that integrates four interfaces: a four-state conversational songwriting flow (music concept setting, lyric creation, music creation, appreciation/discussion) powered by GPT-4.1 using a state-slot prompting technique, an interactive lyrics editor, a multimodal music-appreciation interface (rhythm as blocks, pitch as dot positions, vocal pitch as animated heights, mood as colours), and an archive for replaying past songs. Music generation uses the Mureka-6 text-to-music API. The system was developed through a four-phase user-centered design process: (1) a 75-minute online design workshop with four licensed music psychotherapists specialising in DHH clients; (2) an initial prototype; (3) a 70-minute lab usage study with 12 DHH participants; (4) refinement based on usability findings. Validation comprised a three-week in-the-wild diary study with 12 new DHH participants (12 sessions each, 144 total sessions) analysed through reflexive thematic analysis, plus a follow-up within-subjects comparative study with 10 additional DHH participants across four conditions (text-only, CA-only, GenAI-music-only, full SoulNote).
Key findings
Participants used SoulNote for an average of 41 minutes 24 seconds per session with zero task failures and high satisfaction. Qualitative analysis revealed three dimensions of emotional growth from repeated use. (1) Self-insight: narrative-based imagery questions ('What scene comes to mind?') helped DHH users — many of whom 'had seldom been able to express' feelings verbally — concretise vague feelings and rediscover forgotten emotions; plain, unembellished CA-generated lyrics supported self-acceptance by rendering struggles factually rather than glossing them positively. (2) Emotion regulation: context-based music-style recommendations pushed participants to experiment with 'surprising' genres (e.g., a participant who always used calm piano tried upbeat music on CA suggestion and found it effective), and personalised multimodal visualisations made music perceivable for participants with mild-to-profound hearing loss and cochlear implants/hearing aids. (3) Attitudinal change: 6 of 12 diary participants reported real-world behaviour shifts — initiating previously-avoided conversations with family/colleagues, replaying their songs ~3 times on average for motivation during rehabilitation or job preparation. The follow-up controlled study showed the full SoulNote condition scored significantly higher on self-awareness (M=6.60), self-expression (M=6.23), and perceived emotional change (M=6.23) than text-only, CA-only, or music-only alternatives. No adverse mental-health events were observed across 144 diary sessions.
Relevance
For accessibility practitioners working on Deaf/Hard-of-Hearing mental health, creative-AI tools, or digital self-reflection apps, this paper demonstrates a concrete, clinically-informed workflow for integrating generative AI into wellbeing support without replacing therapeutic judgment. Important design takeaways: structured conversational scaffolding reduces the 'blank page' problem that unguided GenAI creates for users with limited musical or expressive-writing experience; narrative-based imagery prompts outperform technical musical questioning (genre, tempo, timbre) for users unfamiliar with music; faithful, unembellished AI lyric rendering supports self-acceptance better than positive reframing; and multimodal visualisation (rhythm blocks, pitch dots, vocal heights) matters for DHH music appreciation. Practitioners should weigh the authors' stated limitations carefully: the sample excluded born-Deaf sign-language-primary users and clinically depressed users, was Korean-speaking with Korean Sign Language as a secondary modality, used a commercial LLM (GPT-4.1) with all the emotional-risk, hallucination, and privacy concerns that entails, and ran for only three weeks. Future deployments should co-design with Deaf-led collaborators, extend to sign-language modalities, and run longer with clinical co-supervision.
Tags: Generative AI · Deaf and Hard of Hearing · songwriting · mental health · music accessibility · conversational agent · journaling · emotion regulation · large language model · diary study