Automatic Natural Language Generation Applied to Alternative and Augmentative Communication for Online Video Content Services using SimpleNLG for Spanish
Silvia García-Méndez, Milagros Fernández-Gavilanes, Enrique Costa-Montenegro, Jonathan Juncal-Martínez, Francisco Javier González-Castaño · 2018 · Proceedings of the 15th International Web for All Conference (W4A 2018) · doi:10.1145/3192714.3192837
Summary
This short paper presents the development of a Spanish version of SimpleNLG — a natural language generation (NLG) library — specifically enhanced for Augmentative and Alternative Communication (AAC) applications. The work addresses a gap at the intersection of AAC and online content access: approximately 91 million people worldwide have severe communication disorders (including people with autism spectrum disorder, who are often visual learners relying on pictograms), yet most online multimedia content like YouTube videos is inaccessible to them because searching requires formulating text queries — a communication skill they may not possess. Existing AAC tools like Talk Together and LetMe Talk parse telegraphic pictogram sequences and attempt to reconstruct full sentences, but the results are often grammatically poor because the tools lack morphological and syntactic knowledge. The authors developed two versions: a standard Spanish adaptation of SimpleNLG using aLexiS (a comprehensive Spanish lexicon with nearly 90,000 lemmas and full morphological information — far larger than the ~6,000 words in the English original or ~4,000 in the French version), and an enhanced AAC-specific version using Elsa, a lexicon built from the Arasac pictogram domain containing morphological, syntactic, and semantic information. The enhanced version can take a sequence of pictogram-linked words as input and automatically generate a complete, grammatically correct Spanish sentence by inferring prepositions, verb conjugations, gender agreement, number agreement, tense (from temporal adverbs), and syntactic structure from verb subcategorisation frames.
Key findings
The standard Spanish SimpleNLG adaptation successfully passed all original test cases from the French version (SimpleNLGENFr1.1), covering affirmative, negative, interrogative, coordinate, and passive sentences, complete Spanish verb conjugation, and constructions with different word categories. The enhanced AAC version achieved approximately 77% correctness precision on a test set of over 100 pictogram-to-Spanish-sentence pairs annotated by an external NLP researcher — meaning the generated clause exactly matched the expected natural language sentence 77% of the time. The system handles several complex Spanish linguistic features: morphological inflection for gender and number across all clause elements, inference of prepositions using a language model trained on bigrams and trigrams around verbs, verb tense inference from temporal adverbs (e.g., "ayer" triggers past tense), compound subjects, and Spanish double negation. The system was integrated into PictoDroid Lite, an Android AAC application belonging to the Accegal project, where users select pictograms from categories (subjects, verbs, food, objects, places), the system generates a natural language Spanish sentence, and this sentence is used as a YouTube search query to retrieve relevant video content. This gives AAC users access to online video content related to their expressed interests without needing to type or formulate text queries themselves.
Relevance
This paper addresses two underserved areas in accessibility: Spanish-language NLG for AAC (no prior work existed specifically for Spanish AAC), and enabling AAC users to access online multimedia content. The approach of converting pictogram sequences into grammatically correct natural language sentences has applications beyond video search — it could improve AAC-generated text messages, emails, social media posts, and web searches generally. For accessibility practitioners working with AAC users, the key insight is that pictogram-to-text conversion quality matters significantly: telegraphic output ("wolf eat grandmother") is not only less intelligible to communication partners but also fails as a search query, while a properly generated sentence ("El lobo come a la abuela") yields relevant results. The 77% precision rate suggests the system is usable but not yet reliable enough for deployment without human verification. The comprehensive aLexiS lexicon (90,000 lemmas) and the AAC-specific Elsa lexicon are valuable open resources for the Spanish NLP community. As a short paper, the work lacks user evaluation with actual AAC users, which would be essential to validate whether the pictogram-to-video pipeline genuinely improves content access for the target population.
Tags: AAC · natural language processing · pictograms · Spanish · natural language generation · autism · communication accessibility · video accessibility · speech and language