Qwen3.6-35B-A3B-DSV4Pro-Thinking-Distill-GGUF

Qwen3.6-35B-A3B-DSV4Pro-Thinking-Distill — GGUF (原生 MTP + imatrix)

Lynn Agent 高配本地模型

这个 GGUF 仓是 **Lynn Agent** 的高配本地模型选项。Lynn v0.85.6+ 默认推荐 27B Q5 K M GGUF；32GB+ VRAM / 统一内存机器可选择这里的 35B-A3B Q5 K M imatrix MTP。

• *下载 Lynn Agent**：GitHub Releases v0.85.6
• *Lynn 高配量化**：Q5 K M imatrix + 原生 MTP ( --spec-draft-n-max 3 )
• *默认本地路径**：24GB+ 首推 27B Q5 K M；9B / 4B 保留为低配置手动降级。

nerkyor/Qwen3.6-35B-A3B-DSV4Pro-Thinking-Distill 的 GGUF 量化版 —— Qwen3.6-35B-A3B(MoE,35B 总参 / 约 3B 激活)的 DeepSeek-V4-Pro ReAct 思维蒸馏版。每个档位都焊了**原生 MTP 头**,并做了 **imatrix 校正**(Q8 0/F16 除外)。MoE 架构 + MTP 投机解码,单流加速尤其明显(见下)。

量化档位

文件 大小 说明

• -- --- ---

…-MTP-Q4 K M-imatrix.gguf ~21 GB **推荐平衡档**

…-MTP-Q5 K M-imatrix.gguf ~25 GB 更高质量

…-MTP-Q6 K-imatrix.gguf ~29 GB 接近无损

…-MTP-Q8 0.gguf ~37 GB 几乎无损

…-MTP-F16.gguf ~69 GB 全精度参考

…-mmproj-F16.gguf ~1 GB 视觉投影(图像输入)

评测 —— 蒸馏 vs 原版,Q4 K M GGUF,同 harness,thinking-on

维度 蒸馏 原版 Δ

• -- --- --- ---

GPQA-Diamond-198 **80.3%** 72.7% **+7.6pp**

MMLU-500(5-shot) 90.2% 91.4% −1.2

端到端编排 **26.6s** 60.7s **快 2.3×**

编排假验证(20×5) **0/20** 0/20

GPQA 未收口空答 **1** 12 −11

GPQA +7.6pp、编排快 2.3×(学会收口,空答 12→1)。MMLU 微降 1.2:35B-A3B 容量比 27B dense 紧,蒸馏轻微挤占知识(已知权衡)。

MTP —— 原生多 token 预测(MoE → 加速更猛)

GGUF 内含原生 MTP 头(mainline llama.cpp --spec-type draft-mtp )。MoE 比 dense 更吃投机解码红利。DGX Spark GB10 单流实测:

量化档 裸 t/s MTP t/s 加速

• -- --- --- ---

Q4 K M 79.2 86.3 1.09×

Q5 K M 73.0 88.0 1.20×

Q6 K 66.0 75.2 1.14×

Q8 0 49.8 81.2 **1.63×**

越高档加速越明显(带宽受限的裸推理被廉价草稿加速)。无损(贪心投机)。并发会摊薄投机 —— 高吞吐用裸多并发。

运行

思考型模型 —— 必须 **temp 0.6 / top p 0.95**。图像输入加 --mmproj Qwen3.6-35B-A3B-DSV4Pro-Distill-mmproj-F16.gguf 。

English

GGUF quantizations of nerkyor/Qwen3.6-35B-A3B-DSV4Pro-Thinking-Distill — a DeepSeek-V4-Pro ReAct-style reasoning distillation of Qwen3.6-35B-A3B (MoE, 35B total / ~3B active). Every GGUF carries a **native MTP head** and is **imatrix-corrected** (except Q8 0/F16). The MoE architecture plus MTP speculative decoding gives an especially strong single-stream speedup (below).

Quantizations

File Size Notes

• -- --- ---

…-MTP-Q4 K M-imatrix.gguf ~21 GB **recommended balance**

…-MTP-Q5 K M-imatrix.gguf ~25 GB higher quality

…-MTP-Q6 K-imatrix.gguf ~29 GB near-lossless

…-MTP-Q8 0.gguf ~37 GB virtually lossless

…-MTP-F16.gguf ~69 GB full-precision reference

…-mmproj-F16.gguf ~1 GB vision projector (image input)

Evaluation — distill vs base, Q4 K M GGUF, same harness, thinking-on

Metric Distill Base Δ

• -- --- --- ---

GPQA-Diamond-198 **80.3%** 72.7% **+7.6pp**

MMLU-500 (5-shot) 90.2% 91.4% −1.2

End-to-end orchestration **26.6s** 60.7s **2.3× faster**

Orchestration false-verify (20×5) **0/20** 0/20

GPQA non-terminating empties **1** 12 −11

GPQA jumps +7.6pp and orchestration is 2.3× faster — the distill stops over-thinking and converges (non-terminating empties 12→1). MMLU dips 1.2pp: the 35B-A3B has less spare capacity than the 27B dense, so distillation slightly crowds out knowledge (a known trade-off).

MTP — native multi-token-prediction (MoE → strong speedup)

The GGUF has a native MTP head (mainline llama.cpp --spec-type draft-mtp ). MoE benefits more from speculative decoding than dense. Single-stream measured on DGX Spark GB10:

Quant base t/s MTP t/s speedup

• -- --- --- ---

Q4 K M 79.2 86.3 1.09×

Q5 K M 73.0 88.0 1.20×

Q6 K 66.0 75.2 1.14×

Q8 0 49.8 81.2 **1.63×**

Higher quant tiers gain more (the bandwidth-bound base is accelerated by a cheap draft). Lossless (greedy speculative). Concurrency degrades speculation — use bare multi-concurrency for throughput.

Run

Thinking model — use **temp 0.6 / top p 0.95**. For image input add --mmproj Qwen3.6-35B-A3B-DSV4Pro-Distill-mmproj-F16.gguf .

Claude Code（实验性 / experimental）

Claude Code 需要兼容 Anthropic /v1/messages 的后端 + 可靠的工具调用，不能直接喂 GGUF 文件路径。**GGUF 走 LM Studio**：LM Studio 0.4.1+ 加载本仓 GGUF（Q5 K M 推荐），自带 Claude Code /v1/messages 端点，Claude Code 指过去即可。

vLLM 路线需 safetensors —— 用 **FP8 仓** 或 BF16 主仓，加 --enable-auto-tool-choice --tool-call-parser qwen3 xml --reasoning-parser qwen3 ，模型名用 --served-model-name （不带斜杠）。参考 LM Studio · vLLM Claude Code。

Claude Code (experimental)

Claude Code needs an Anthropic-compatible /v1/messages backend + reliable tool calling — you cannot hand it a GGUF path. **GGUF goes via LM Studio**: LM Studio 0.4.1+ loads this repo's GGUF (Q5 K M recommended) and exposes a built-in Claude Code /v1/messages endpoint — just point Claude Code at it. The vLLM route needs safetensors — use the **FP8 repo** or the BF16 parent with --enable-auto-tool-choice --tool-call-parser qwen3 xml --reasoning-parser qwen3 , model name = --served-model-name (no slashes).