A Demo of Talkit++: Interacting with 3D Printed Models Using an iOS Device
Lei Shi, Zhuohao Zhang, Shiri Azenkot · 2018 · Proceedings of the 20th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS 2018) · doi:10.1145/3234695.3241004
Summary
This Cornell Tech demo presents Talkit++, an iOS application that adds interactive multimedia labels to 3D printed tactile models for visually impaired students. Teachers of the visually impaired (TVIs) increasingly use 3D printers to create tactile learning materials, but these models alone provide only shape information. Talkit++ enables TVIs to create custom interactive models where touching different parts triggers spoken text labels, audio recordings (e.g., the sound of an ocean for a coastal region on a map), or visual animations (e.g., a volcano eruption for low vision students). Building on the earlier Talkit (desktop-only, text labels only) and Markit (a design tool for associating labels with model elements), Talkit++ adds three key features: iOS compatibility for use on devices TVIs already have in classrooms, multimedia output beyond text, and a new model tracking method that requires less modification to the physical model. The system works by placing the iPad so its camera can see the model, using Chilitag fiducial markers (either 3D cubes attached to the model or printable 2D stickers stuck on flat surfaces) for model localization, and tracking a red sticker on the user's fingernail via OpenCV to detect finger position. Speech commands (via Swift speech dictation) allow the user to say "more" for detailed descriptions.
Key findings
The design was informed by collaboration with three TVIs through four weekly remote meetings each, producing three sample interactive models over one month: a Washington State Regions topographical map (with five annotated regions and audio recordings like ocean sounds), a volcano model from Thingiverse (with five elements including a visual eruption animation for low vision students), and a plane model (with six annotated elements including cockpit, body, and jet engines). The shift from 3D tracker cubes to printable 2D sticker trackers was driven by TVI feedback that the cubes were too bulky and modified the models too much. The 2D trackers can be resized to fit different models and are less intrusive for flat models like maps. The system is implemented in Swift and C++, using cross-compiled Chilitags and OpenCV libraries. For model creation, TVIs use the updated Markit tool to add trackers and associate multimedia content with model regions. The planned future work includes making the application publicly available and creating an online repository for interactive 3D printed models that teachers and students can freely access and share.
Relevance
Talkit++ addresses a practical gap in accessible STEM education: 3D printed models provide valuable tactile learning experiences but cannot convey the rich information that sighted students get from textbook illustrations, diagrams, and animations. By augmenting physical models with audio and visual content triggered by touch, the system creates a multimodal learning experience that serves both blind students (through audio) and low vision students (through visual animations). For accessibility practitioners, the key design insight is using commodity hardware (an iPad and printable stickers) rather than specialized equipment, making the system practical for real classroom adoption. The collaborative design process with TVIs — who are the domain experts in how visually impaired students learn — exemplifies user-centered assistive technology development. The vision of an open repository for interactive models could dramatically reduce the effort required for individual TVIs to create accessible learning materials.
Tags: blindness · low vision · 3D printing · tactile accessibility · computer vision · education · STEM accessibility · tactile graphics · multimodal interaction