Mixture of Experts

Mixture of experts is a neural network architecture that routes each input through a small subset of specialist subnetworks ('experts') rather than activating the whole model. A gating network decides which experts handle a given token or query, letting the overall model be much larger in total parameters while keeping per-inference compute modest. MoE is increasingly used in accessibility-facing deployments because the selective activation makes large multilingual or multimodal models viable on constrained devices - for example, domain-adapted AAC or clinical chatbot systems.