模型介绍
中文整理Qwen3.6-35B-A3B-DSV4Pro-Thinking-Distill-FP8
- *🇨🇳 中文 · 🇬🇧 English ⬇️**
🇨🇳 中文版
Qwen3.6-35B-A3B-DSV4Pro-Thinking-Distill** 的 **FP8(block-128 e4m3)+ 原生 MTP** 版 —— 面向 **SGLang / vLLM 服务化**。35B-A3B 是 **MoE(256 专家,3B 激活)**,稀疏激活让它**单流就很快**。
蒸馏方法、出处归因、评测口径、局限**见父模型(BF16)卡片。本卡只讲 FP8 版。
这是什么
- *FP8 block-128、e4m3、动态激活**量化蒸馏主干 —— 与官方 Qwen/Qwen3.6-35B-A3B-FP8 同方案(每 128×128 块一个 weight scale inv )。**256 个专家逐专家(per-expert)量化**(gate/up/down 拆分对齐官方布局,SGLang/vLLM 直认)。norm、embedding、 lm head 、router、Gated-DeltaNet 小投影、shared expert 之外的 BF16 项、vision 塔保持 **BF16**。
打包**原生 MTP(nextn)头**( mtp.safetensors ),供 EAGLE/NEXTN 投机解码。
- *构造上近无损** —— block-128 FP8 正是 Qwen 官方"性能与原模型几乎一致"的同精度。
速度 —— **蒸馏 FP8** 实测 **DGX Spark(GB10,Blackwell sm 121)**,SGLang dev-cu13 + flashinfer + EAGLE/NEXTN
模式 吞吐 MTP 接受率 接受长度
- -- --- --- ---
单流,**关 MTP**(base) **~51 tok/s** — —
单流,**开 MTP** **~63–65 tok/s** 0.68–0.78 2.7–3.1
- *MTP 单流约 1.25×(+25%)加速**(base ~51 → 63–65 tok/s,接受率稳定 ~0.75 / 接受长度 ~3.0)。均为**本蒸馏 FP8 实测**(SGLang dev-cu13 ,EAGLE topk=1 / 3 draft / mamba extra buffer ):生成正确、无 FP8 损坏。
- *增益温和但单流本就快**:MTP 增益**温和(~25%)** —— A3B 稀疏、base 本已快,投机空间小(**密度律**:激活越稀疏,投机解码 ROI 越低)。但原生 nextn 够便宜,**仍净正收益**(不像外挂 drafter 在稀疏模型上反亏)。真正红利是**稀疏让单流就快**:35B-A3B(3B 激活)51→65 tok/s 远超 dense 27B(~15 tok/s)。Blackwell RTX-50(FP8 更快)上更快。
质量
FP8 近无损,且在 **GB10 上冒烟验证正确**(中英推理、 + 收口结构完整,无 FP8 损坏)。质量 **≈ 父蒸馏版**,其 canonical(thinking-on)数据:
维度 父蒸馏版 原版 base Δ
- -- --- --- ---
GPQA-Diamond-198 **80.3%**(32K) 72.7% **+7.6**
MMLU-500 (5-shot) **90.2%** 91.4% -1.2
GPQA 未收口空答 (parse fail) **1** 12 **-11**
知识广度基本不变,**硬推理显著提升(GPQA +7.6pp)**,且"想完就收口"(空答 12→1)。
多模态说明
35B-A3B 的**多模态架构继承自原版 base**(vision 塔为官方原版,已随 FP8 打包, outside.safetensors )。**蒸馏仅针对文本推理 / agentic**,未训练视觉。FP8 版可走 SGLang/vLLM 多模态;**GGUF 纯文本**(视觉单独走 父仓 gguf/ 里的 mmproj 文件)。
用法(SGLang + MTP)
- *采样**: temperature 0.6 / top p 0.95 (thinking 模型必需,greedy 会重复死循环)。
🇬🇧 English
The **FP8 (block-128 e4m3) + native MTP** build of **Qwen3.6-35B-A3B-DSV4Pro-Thinking-Distill** — for **SGLang / vLLM serving**. 35B-A3B is **MoE (256 experts, 3B active)**; sparse activation makes it **fast even single-stream**.
For **distillation method, attribution, eval methodology, and limitations**, see the parent (BF16) card. This card covers only the FP8 build.
What this is
- *FP8 block-128, e4m3, dynamic activation** quant of the distilled backbone — same scheme as official Qwen/Qwen3.6-35B-A3B-FP8 (one weight scale inv per 128×128 block). **All 256 experts quantized per-expert** (gate/up/down split to match the official layout, loads directly in SGLang/vLLM). norm, embedding, lm head , router, Gated-DeltaNet small projections, and the vision tower stay **BF16**.
Bundles the **native MTP (nextn) head** ( mtp.safetensors ) for EAGLE/NEXTN speculative decoding.
- *Near-lossless by construction** — block-128 FP8 is Qwen's own "virtually identical to the original" precision.
Speed — **this distilled FP8**, measured on **DGX Spark (GB10, Blackwell sm 121)**, SGLang dev-cu13 + flashinfer + EAGLE/NEXTN
Mode Throughput MTP accept rate accept len
- -- --- --- ---
Single-stream, **MTP off** (base) **~51 tok/s** — —
Single-stream, **MTP on** **~63–65 tok/s** 0.68–0.78 2.7–3.1
- *~1.25× (+25%) single-stream speedup** from MTP (base ~51 → 63–65 tok/s, stable accept ~0.75 / accept len ~3.0). All **measured on this distilled FP8** (SGLang dev-cu13 , EAGLE topk=1 / 3 draft / mamba extra buffer ): correct generation, no FP8 corruption.
- *Modest MTP gain, but already-fast single-stream**: the MTP single-stream gain is **modest (~25%)** — A3B is sparse and the base is already fast, so speculation headroom is limited (**density rule**: the sparser the activation, the lower the speculative-decoding ROI). But the native nextn head is cheap enough to stay **net-positive** (unlike an external drafter, which *loses* on sparse models). The real payoff is that **sparsity makes single-stream fast**: 35B-A3B (3B active) 51→65 tok/s, far above the dense 27B (~15 tok/s). Faster on Blackwell RTX-50 (faster FP8).
Quality
FP8 is near-lossless and **smoke-verified correct on GB10** (EN/ZH reasoning, intact + closure, no FP8 corruption). Quality **≈ parent distill**, whose canonical (thinking-on) numbers:
Dimension Parent distill Base Δ
- -- --- --- ---
GPQA-Diamond-198 **80.3%** (32K) 72.7% **+7.6**
MMLU-500 (5-shot) **90.2%** 91.4% -1.2
GPQA non-closure empty (parse fail) **1** 12 **-11**
Multimodal note
The multimodal architecture is **inherited from the original base** (the vision tower is the stock official one, bundled with the FP8 as outside.safetensors ). **Distillation targeted text reasoning / agentic only**, not vision. The FP8 build supports SGLang/vLLM multimodal; **GGUF is text-only** (vision via the separate mmproj in the parent repo's gguf/ ).
Usage (SGLang + MTP)
- *Sampling**: temperature 0.6 / top p 0.95 (required for thinking models; greedy loops).
Claude Code(实验性 / experimental)
Claude Code 不是本地 GGUF/HF 加载器——它对接兼容 Anthropic /v1/messages 的后端、且要求可靠的工具调用,不能直接喂仓名/路径。本 **FP8 仓是 vLLM 路线的直接载体**:
Claude Code 里模型名填 qwen36-35b-fp8 (即 --served-model-name ,**不要填带 / 的仓名**), ANTHROPIC BASE URL 指向你的 vLLM 端点。要 GGUF + LM Studio 路线见 **GGUF 仓**。参考 vLLM Claude Code。
Claude Code (experimental)
Claude Code is not a local GGUF/HF loader — it talks to an Anthropic-compatible /v1/messages backend and needs reliable tool calling, so you cannot point it at a repo id directly. This **FP8 repo is the direct vLLM path** — serve it with --enable-auto-tool-choice --tool-call-parser qwen3 xml --reasoning-parser qwen3 (command above), then set Claude Code's model name to the --served-model-name (no slashes) and ANTHROPIC BASE URL to your vLLM endpoint. For the GGUF + LM Studio route, see the **GGUF repo**.
优先采用原作者提供的中文模型卡片;仅有英文说明时由闲社AI翻译整理。重要参数、许可证和商用范围请以原文为准。
你将获得什么
完整仓库方式包含该固定版本中的权重、配置、分词器及说明文件;指定版本方式只下载所选文件或整组分片。多模态模型的投影文件、配置等依赖,请按作者说明配齐。
完整仓库可能包含多个权重格式,因此需要的磁盘空间可能大于单一格式的模型大小。下载文件不等于完成模型部署。
使用前须知
运行模型需要的设备、工具与依赖,以原作者说明为准。
是否允许商用、修改和分发,请查看模型许可证。
闲社提供资料与镜像下载方法,不代表已运行模型或完成安全审计。下载的代码应先检查,再执行。
Problem with the information or download?
Sign in to submit feedback.
