gensai.aiA Mixture of Experts model is split into many specialised sub-networks, the experts. For each token, a router activates only a small number of them. The model therefore has a very large total number of parameters but uses only a fraction of them for each computation: DeepSeek-V3 has 671 billion, of which 37 billion are active.
In practice at Gensai
Gensai takes the MoE architecture of models such as Mixtral, DeepSeek and Kimi into account when choosing an open-weight model, since it changes memory and speed requirements.
Same theme
LLM (Large Language Model)SLM (Small Language Model)DLLM (Diffusion Large Language Model)Hallucination (AI)Prompt chainingPre-trained modelFoundation modelGenerative AITransformerTokenContext windowPromptPrompt engineeringContext (AI)Memory (AI)Fine-tuningOpen-weightQuantizationDistillationLoRA (Low-Rank Adaptation)Reasoning modelTemperature (AI)