Qwen3.6-35B-A3B Architecture

40 layers · 35B params · 3B active per token · 256K context
High-Level Architecture INPUT Input Tokens BPE Tokenizer vocab 248K Embedding 248K × 2048 mRoPE 25% partial · [11,11,10] TRANSFORM ×40 RMSNorm Attention Full or Gated DeltaNet + RMSNorm MoE FFN top-8/256 + shared 30 GDN 10 Full OUTPUT Final RMSNorm LM Head untied · 2048→248K Next Token logits → sample MTP Head (optional)
One Transform Block ATTENTION MIXER hidden [batch, seq, 2048] RMSNorm Full Attn QKV + GQA QK-Norm softmax + gate Gated DeltaNet QKVZ + conv1d delta rule recurrent/chunk O-proj 4096→2048 + residual to next stage → 10 layers 30 layers MoE FFN RMSNorm Router 2048 → 256 softmax → top-8 → renormalize Expert SwiGLU × 8 gate_up_proj + down_proj + Shared Expert always active · scalar gate + residual output → KEY NUMBERS Full Attention Q: 16hd × 256dim K: 2hd × 256dim V: 2hd × 256dim GQA ratio 16:2 = 8× QK-Norm per head Gate from q_proj 2× out Gated DeltaNet key: 16hd × 128dim val: 32hd × 128dim conv1d: kernel=4 delta rule recurrence O(n) time, O(1) mem MoE FFN 256 experts total 8 active per token Expert: 3.15M params Shared: +3.15M params

Overview

The model processes text through three stages, left to right.

Input Stage

BPE tokenizer maps text to token IDs (vocab 248,320). Embedding lookup table maps each ID to a 2048-dim vector. mRoPE adds 3D positional encoding (25% partial rotary, sections [11,11,10]).

Transform Blocks ×40

Each block = RMSNorm → Attention → +residual → RMSNorm → MoE FFN → +residual. Two residual connections preserve gradient flow through all 40 layers.

Output Stage

Final RMSNorm, then LM Head (2048 → 248,320, untied) produces logits. Optional MTP head predicts a second token for speculative decoding.

Model at a Glance

Total params~35B
Active/token~3B
Hidden dim2048
Layers40 (30 GDN + 10 Full)
Experts256 (8 active)
Heads Q/KV16 / 2 (GQA 8:1)
Head dim256
Context262,144 tokens