Brain Body Jockey project: Transcending Bodily Limitations in Live Performance via Human Augmentation
Giulia Barbareschi, Songchen Zhou, Ando Ryoichi, Midori Kawaguchi, Mark Armstrong, Mikito Ogino, Shunsuke Aoiki, Eisaku Ohta, Harunobu Taguchi, Youichi Kamiyama, Masatane Muto, Kentaro Yoshifuji, Kouta Minamizawa · 2024 · Proceedings of the 26th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS 2024) · doi:10.1145/3663548.3675621
Summary
This paper documents a groundbreaking collaboration between a professional VDJ (video disc jockey) with advanced Amyotrophic Lateral Sclerosis (ALS) and a team of researchers and technologists to create a human augmentation system enabling brain-controlled robotic arms for live music performances. The artist, diagnosed with ALS a decade earlier and having lost nearly all mobility, composes and performs music using a gaze-based interface with Ableton software. The project's goal was not merely to compensate for lost function but to extend the artist's body and enhance stage presence—the ability to physically engage with audiences that is central to live performance. Developed over six months of iterative collaboration, the system combines a Brain-Machine Interface (BMI) using an OpenBCI 16-channel EEG device with auditory event-related potentials and motor imagery for selection, an accelerometer trigger on the artist's index finger for precise timing control, and five UFactory X-Arms (7 degrees of freedom each) with 3D-printed HACKberry hands. The artist could select from nine pre-programmed dance animations via a two-step BMI selection process mapped to a 3x3 matrix, with movements automatically synchronised to the music's BPM. The system was deployed at two public performances in Tokyo (approximately 500 total attendees) as part of a festival of inclusive societies, followed by a "Meet the DJ" event where over 40 fans physically interacted with the artist via the robotic arms equipped with haptic feedback. The evaluation combined ethnographic observation, video analysis, and interviews with the artist and 13 audience members.
Key findings
The system achieved its core goal of creating a sense of unity between artist and audience. All 13 interviewed audience members described experiencing a feeling of fusion and connection, with 11 specifically linking their own body movements to the robotic arm gestures controlled by the VDJ. The audience began mirroring the robotic arms' movements as the performance progressed, creating a reciprocal embodied dialogue. The artist reported a strong sense of ownership over the robotic arms, describing them as extensions of his body: "By extending my body through the power of brainwave and robotic arm technology, I can create rhythms at my own pace and encourage the audience to clap along, which gives me a greater sense of physical freedom." The haptic feedback during the Meet the DJ event—where fans could handshake, fist-bump, and high-five the robotic hands—deepened the sense of physical connection, with the artist feeling the touch transmitted to his forearm. Audience members were consistently impressed by the artist's skill and mastery, viewing the technology as complementary to rather than overshadowing his talent. Six participants explicitly noted that while the technology was impressive, it enhanced rather than replaced the artistic performance. The BMI system achieved 89.65% accuracy after 12 hours of training over three months, though the 16-second selection time required the accelerometer trigger for precise timing during live performance.
Relevance
This paper powerfully challenges the dominant framing of assistive technology as purely compensatory—filling functional gaps—by demonstrating how human augmentation can enable entirely new forms of creative expression. For accessibility practitioners and researchers, it introduces the concept of "relative virtuosity" applied to disabled performers: the idea that artistic mastery cannot be separated from how disabled bodies interact with the world and their tools. The BMI-controlled robotic arm system represents the first known deployment combining brain-computer interfaces with multiple robotic agents in a live performance context, opening possibilities beyond entertainment for communication and social interaction for people with severe motor impairments. The collaborative design process, driven by the artist's creative vision rather than researcher assumptions, models how disability arts can push technology innovation. A key limitation is that the system was developed for a single artist across two performances, and the BMI training demands (12 hours over 3 months) may limit broader adoption. The work calls for HCI and accessibility researchers to engage more with disabled artists as technological pioneers rather than passive recipients of assistive solutions.
Tags: brain-machine interface · ALS · human augmentation · disability arts · robotic arms · live performance · stage presence · assistive technology