Qwen3.6-27B-DSV4Pro-Thinking-Distill-GGUF

Qwen3.6-27B-DSV4Pro-Thinking-Distill — GGUF (原生 MTP + imatrix)

Lynn Agent 默认本地模型

这个 GGUF 仓是 **Lynn Agent** 的默认本地模型推荐。Lynn v0.85.6+ 会把 Qwen3.6-27B-DSV4Pro-Distill-MTP-Q5 K M-imatrix.gguf 作为 24GB+ VRAM / 统一内存机器的首选端侧模型；32GB+ 可选择 35B-A3B Q5 K M，高低配机器可手动降级到 9B / 4B。

• *下载 Lynn Agent**：GitHub Releases（当前桌面/CLI 发版：v0.85.6）
• *Lynn 默认量化**：Q5 K M imatrix + 原生 MTP ( --spec-draft-n-max 3 )
• *推荐用途**：端侧推理智能体、本地 coding / agentic 任务、无云依赖兜底。

nerkyor/Qwen3.6-27B-DSV4Pro-Thinking-Distill 的 GGUF 量化版 —— Qwen3.6-27B(dense)的 DeepSeek-V4-Pro ReAct 思维蒸馏版。每个档位都焊了**原生 MTP 头**,并做了 **imatrix 校正**(Q8 0/BF16 除外)。蒸馏只改*思维方式*(学会"收口"),coding 与知识不降。

量化档位

文件 大小 说明

• -- --- ---

…-MTP-IQ4 XS-imatrix.gguf 15.3 GB **16G 显存底线** —— 16G 不 offload 可跑,最小档

…-MTP-Q4 K S-imatrix.gguf 15.8 GB 速度优先

…-MTP-Q4 K M-imatrix.gguf 16.8 GB 平衡档 / 低显存回退

…-MTP-Q5 K M-imatrix.gguf 19.5 GB **Lynn 默认推荐(24GB+)**

…-MTP-Q6 K-imatrix.gguf 22.4 GB 接近无损

…-MTP-Q8 0.gguf 29.0 GB 几乎无损

…-MTP-BF16.gguf 54.7 GB 全精度参考

…-mmproj-F16.gguf 0.9 GB 视觉投影(图像输入)

• *IQ4 XS 是底线档。** 为 **16 GB 显存、不 offload** 而做(14.3 GiB 权重 + KV 余量)。冒烟实测**收口干净**(3/3 推理题正确、0 长度跑飞、中英),单流 **31 t/s** + 原生 MTP 头( --spec-type draft-mtp ,接受率 0.80、平均接受长 3.4)。这是我们发布的**最小档** —— 27B 推理蒸馏低于 IQ4 XS 会开始丢收口,到此为止。

• *📏 GB vs GiB —— 为什么 15.3 GB 的文件能进"16 GB"显卡。** 本表大小用十进制 **GB(10⁹ 字节)**；显卡显存按 **GiB(2³⁰ 字节)** 造却标"GB"，所以一张"16 GB"卡其实是 **16 GiB ≈ 17.2 GB**。于是 IQ4 XS(15.3 GB = **14.3 GiB**)能完整载入 16 GiB 卡，**剩 ~1.7 GiB 给 KV —— 不用 offload**。经验法则：文件的 **GiB** 数 **IQ4 XS is the floor.** Sized for a **16 GB-VRAM card with zero CPU offload** (14.3 GiB weights + headroom for KV). Smoke-verified **converging** (3/3 reasoning prompts correct, 0 length-runaway, EN + ZH) at **31 t/s single-stream** with the native MTP head ( --spec-type draft-mtp , accept 0.80, mean accept-len 3.4). This is the smallest tier we ship — below IQ4 XS a 27B reasoning distill starts to lose convergence, so we stop here.

• *📏 GB vs GiB — why a 15.3 GB file fits a "16 GB" GPU.** Sizes in this table are decimal **GB (10⁹ bytes)**. GPU VRAM is built in **GiB (2³⁰ bytes)** but marketed as "GB" — so a "16 GB" card is really **16 GiB ≈ 17.2 GB**. IQ4 XS (15.3 GB = **14.3 GiB**) therefore loads fully on a 16 GiB card with **~1.7 GiB left for KV cache — no CPU offload**. Rule of thumb: a file runs no-offload when its **GiB** size < the card's nominal "GB" number (here 14.3 < 16 ✓).

Evaluation — distill vs base, same-spec GGUF, thinking-on

Both rows use the same quant + MTP; the only variable is "distilled or not". Do not cross-compare quant tiers (Q4 and Q5 use different harnesses).

Q4 K M (archived harness):**

Dimension Distill Base Δ

• -- --- --- ---

GPQA-Diamond-198 **80.81%** (160/198) 73.7% **+7.1**

MMLU-500 (5-shot) **91.8%** 91.6% +0.2

GPQA unconverged empties **0** 14 −14

coding-100 (real sandbox) **86/100** 83/100 +3

Q5 K M (streaming harness · conc4 · timeout 1800 · max tokens 32000, finish reason logged):**

Dimension Distill Base Δ

• -- --- --- ---

GPQA-Diamond-198 **81.82%** (162/198) 68.69% (136/198) **+13.13pp**

MMLU-500 **90.0%** 89.6% +0.4

GPQA finish=stop (converged) **198/198** 186/198

GPQA finish=length (hit 32K wall) **0** 12

Hard reasoning jumps (GPQA +7~13pp), knowledge holds, and the distill **learns to converge** — the base hits the 32K wall on 12 questions (never answers); the distill converges on all 198.

MTP — native multi-token-prediction (single-stream speedup)

The GGUF has a native MTP head (mainline llama.cpp --spec-type draft-mtp , no external draft model). Measured single-stream on DGX Spark GB10, Q4 K M:

• -spec-draft-n-max TPS accept rate accept len
• -- --- --- ---

bare (no MTP) 10.4 — —

• *n-max=3 ⭐** **26.8** 0.72 3.16

• 3–2.6× single-stream speedup, lossless (greedy speculative only accepts the target-argmax token). Speculation is single-stream — concurrency degrades it, so use bare multi-concurrency for throughput.

Run

Thinking model — use **temp 0.6 / top p 0.95** (greedy loops to death). For image input add --mmproj Qwen3.6-27B-DSV4Pro-Distill-mmproj-F16.gguf .

Claude Code / llama-swap 兼容模板

如果对话历史中间出现新的 system 或 developer 消息，并报错 System message must be at the beginning ，请使用 qwen35-claude-compatible.jinja。该模板会按原顺序把所有 system/developer 内容合并到开头的 system 块，同时保留 Qwen3.5 原有的 thinking、工具调用与 tool response 逻辑。

启动示例：

Claude Code（实验性 / experimental）

Claude Code 需要兼容 Anthropic /v1/messages 的后端 + 可靠的工具调用，不能直接喂 GGUF 文件路径。**GGUF 走 LM Studio**：LM Studio 0.4.1+ 加载本仓 GGUF（Q5 K M 推荐），自带 Claude Code /v1/messages 端点，Claude Code 指过去即可。

vLLM 路线需 safetensors —— 用 **FP8 仓** 或 BF16 主仓，加 --enable-auto-tool-choice --tool-call-parser qwen3 xml --reasoning-parser qwen3 ，模型名用 --served-model-name （不带斜杠）。参考 LM Studio · vLLM Claude Code。

Claude Code (experimental)

Claude Code needs an Anthropic-compatible /v1/messages backend + reliable tool calling — you cannot hand it a GGUF path. **GGUF goes via LM Studio**: LM Studio 0.4.1+ loads this repo's GGUF (Q5 K M recommended) and exposes a built-in Claude Code /v1/messages endpoint — just point Claude Code at it. The vLLM route needs safetensors — use the **FP8 repo** or the BF16 parent with --enable-auto-tool-choice --tool-call-parser qwen3 xml --reasoning-parser qwen3 , model name = --served-model-name (no slashes).