https://cybersecurityerial.github.io/echo_blog/tags/moe/Recent content in MoE on Echo的技术博客Hugozh-cnTue, 16 Jun 2026 00:00:00 +0800https://cybersecurityerial.github.io/echo_blog/posts/llm-system-basic-quick-reference-01-moe/Tue, 16 Jun 2026 00:00:00 +0800https://cybersecurityerial.github.io/echo_blog/posts/llm-system-basic-quick-reference-01-moe/<h2 id="moe-fb-计算流程">moe f&amp;b 计算流程</h2> <h3 id="router">Router</h3> <p>Router 计算每个 token 应该路由到哪些 expert。</p> <p><img alt="MoE Router 计算 01" loading="lazy" src="https://cybersecurityerial.github.io/echo_blog/images/llm-system-basic-quick-reference-moe/01-router-01.png"></p> <p><img alt="MoE Router 计算 02" loading="lazy" src="https://cybersecurityerial.github.io/echo_blog/images/llm-system-basic-quick-reference-moe/02-router-02.png"></p> <h3 id="dispatch">Dispatch</h3> <p>Dispatch 根据 router 结果，把 token 分发到对应 expert 的输入 buffer。</p> <p><img alt="MoE Dispatch 计算 01" loading="lazy" src="https://cybersecurityerial.github.io/echo_blog/images/llm-system-basic-quick-reference-moe/03-dispatch-01.png"></p> <p><img alt="MoE Dispatch 计算 02" loading="lazy" src="https://cybersecurityerial.github.io/echo_blog/images/llm-system-basic-quick-reference-moe/04-dispatch-02.png"></p> <h3 id="ffn">FFN</h3> <p>每个 expert 内部执行自己的 FFN / MLP 计算。</p> <p><img alt="MoE FFN 计算 01" loading="lazy" src="https://cybersecurityerial.github.io/echo_blog/images/llm-system-basic-quick-reference-moe/05-ffn-01.png"></p> <p><img alt="MoE FFN 计算 02" loading="lazy" src="https://cybersecurityerial.github.io/echo_blog/images/llm-system-basic-quick-reference-moe/06-ffn-02.png"></p> <h3 id="combine">Combine</h3> <p>Combine 把各 expert 的输出按路由权重聚合，并还原到 token 维度。</p> <p><img alt="MoE Combine 计算 01" loading="lazy" src="https://cybersecurityerial.github.io/echo_blog/images/llm-system-basic-quick-reference-moe/07-combine-01.png"></p> <p><img alt="MoE Combine 计算 02" loading="lazy" src="https://cybersecurityerial.github.io/echo_blog/images/llm-system-basic-quick-reference-moe/08-combine-02.png"></p> <h3 id="moe-softmax">MoE softmax</h3> <p>MoE做router的意义是什么？</p> <p>MoE 做softmax的意义和一般softmax意义类似，都是让logits能被解释为概率。 被解释为概率/加权的几个要求：正数，和为1，尺度一致，</p>