arxiv.org
Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
In deep learning, models typically reuse the same parameters for all inputs. Mixture of Experts (MoE) defies this and instead selects different parameters for each incoming example. The result is a sparsely-activated model -- with outrageous numbers of parameters -- but a constant computational cost. However, despite several notable successes of MoE, widespread adoption has been hindered by complexity, communication costs and training instability -- we address these with the Switch Transformer. We simplify the MoE routing algorithm and design intuitive improved models with reduced communication and computational costs. Our proposed training techniques help wrangle the instabilities and we show large sparse models may be trained, for the first time, with lower precision (bfloat16) formats. We design models based off T5-Base and T5-Large to obtain up to 7x increases in pre-training speed with the same computational resources. These improvements extend into multilingual settings where we measure gains over the mT5-Base version across all 101 languages. Finally, we advance the current scale of language models by pre-training up to trillion parameter models on the "Colossal Clean Crawled Corpus" and achieve a 4x speedup over the T5-XXL model.
![This image illustrates the architecture and scheduling of TransMamba, a neural network model that integrates attention and state space models (SSMs) through a unified module. It includes three subfigures:
⸻
(a) The TransMamba Architecture:
This block diagram shows the core structure:
• Inputs pass into an Embedding layer labeled “TransPoint”.
• The embedding splits into two branches:
• One path feeds into Attention after a QKV computation, then into a Memory Converter.
• The second path goes into SSM after passing through Embeds₂.
• The SSM receives initial hidden state h₀, generated by the Memory Converter.
• The SSM block involves operations with parameters Δ, A, Z and weight matrices Wₐ, W_z, with intermediate computations like CBₓ.
• Outputs from both paths are concatenated, followed by Norm & Feed Forward layers (repeated N times).
• Final output passes through a Linear layer to produce model outputs.
⸻
(b) Memory Converter Module:
A close-up of the memory initialization:
• Uses previous hidden state h[-1] to compute h₀.
• Includes MatMul, Multiply, and variables Δ, A, K, V as components.
⸻
(c) TransPoint Scheduling Diagram:
• Shows scheduling strategy over layers and sequence length.
• Alternating vertical bars represent:
• Blue: Attention
• Green: SSM
• Orange: Memory Converter (only at TransPoint)
• Memory conversion happens early, followed by layer-wise computation with both attention and SSM.
⸻
The diagram is stylized with a robotic snake motif, aligning with the “Mamba” naming theme. This visual layout emphasizes TransMamba’s hybrid design combining attention and SSMs with efficient memory initialization and scheduling.](https://cdn.bsky.app/img/feed_thumbnail/plain/did:plc:ckaz32jwl6t2cno6fmuw2nhn/bafkreia6o2twzn6qoopibdp6qizaait3uwtwzeajhypsm3rvahuk2bjkte)
