laya-multilingual-f16.gguf

Laya Multilingual 0.3.4 — GGUF (llama.cpp)

中文 English

中文

这是什么

convaiinnovations/laya-multilingual （laya 0.3.4）的 **GGUF 格式转换**，可被

llama.cpp 加载运行。

Laya 是一个**非自回归的决策模型**（System 1）：给它一个「状态」（文本、邮件、工单、

JSON）和若干「带类型的提问」，它在**一次前向传播**中返回带概率的结构化答案。

不生成文本，所以没有解析环节，也不会幻觉。

• *重要：这不是完整推理所需的一个文件。** 请务必先读下面的「本转换的范围」。

本转换的范围（请务必先读）

Laya **不是**一个标准的 HuggingFace 检查点，因此 convert hf to gguf.py 无法直接读取它。

它由两部分组成：

组成 内容 在哪里运行

• -- --- ---

encoder.* mmBERT-base 双向编码器：22 层、hidden 768、12 头、FFN 1152（GeGLU）、全注意力与滑窗注意力交替（窗口 128）、RoPE theta 160000、256k 词表 **llama.cpp**（ modern-bert 计算图）

head.layers.* 额外 2 层 nn.TransformerEncoderLayer （d=768、nhead=12、ffn=3072、 norm first ） 运行时（PyTorch）

scorer.* 、 act head.* 、 type emb.* 选项标记打分器、act/escalate 头、问题类型嵌入 运行时（PyTorch）

判断过程是：在每个选项自己的 [MASK] 位置上打分，再对该题的选项做 softmax。

这个决策头不属于 llama.cpp 的 modern-bert 计算图，也无法塞进去**（llama.cpp 的加载器

要求文件里的张量数与它计算图里创建的张量数严格一致，多出来的张量会直接导致加载失败）。

所以：**3 亿参数的编码器跑在 llama.cpp 里，决策头跑在配套的 PyTorch 运行时里。**

为什么是两个文件

llama.cpp 的加载器会校验张量数量，多一个都不行。所以决策头权重不能和编码器放在同一个

GGUF 里，必须单独一个配套文件。

文件清单

本仓库共 6 个文件，使用时**必须放在同一个目录下**：

文件 大小 说明

• -- --- ---

laya-multilingual-f16.gguf 600 MB modern-bert 编码器 + 256k BPE 分词器。**llama.cpp 直接加载这个。**

laya-multilingual-f16-head.gguf 57 MB 决策头权重，由运行时读取

tokenizer.json 33 MB 运行时用它做分词（用显式 token id 调 llama.cpp）

laya gguf.py 11 KB 运行时主程序（约 200 行）

gguf weights.py 1 KB 从决策头 GGUF 读出权重的辅助模块

README.md — 本文件

下载后目录结构应为：

• *只下载 laya-multilingual-f16.gguf 是不够的。** 编码器能加载，但没有任何决策能力 ——

决策头、分词器和运行时代码都在其余文件里。

运行环境

Python 依赖： torch 、 numpy 、 tokenizers 、 gguf （ pip install torch numpy tokenizers gguf ）。

编码器部分由 llama.cpp 承担，不需要 transformers 。

为什么必须三个权重文件

laya-multilingual-f16.gguf 里只有编码器。llama.cpp 的加载器会校验张量数量，多一个张量

都会导致加载失败，所以决策头权重无法和编码器放进同一个 GGUF。运行时分词则需要

tokenizer.json （它自己分词后把 token id 数组发给 llama.cpp）。

快速开始

• 启动 llama.cpp 服务

Windows PowerShell 下需要写 .\llama-server.exe ，PowerShell 默认不从当前目录加载命令。

• 运行

需要一个配套运行时 laya gguf.py （约 200 行 Python，负责分词、调 llama.cpp 取隐状态、

再在 PyTorch 里跑决策头）。三个权重文件与本文件放在同一目录，在该目录下运行：

三种提问类型

类型 输出 适用

• -- --- ---

choice 选项概率分布 + 置信度 部门路由、意图分类、主题归类

score 有序量表上的期望等级 紧急程度、情绪强度、严重性

noul 校准后的 P(true)，0.0–1.0 是否类判断：垃圾邮件、越狱、流失风险

关于分词器

编码器 GGUF 里的 tokenizer.ggml.pre 写的是 gemma4 ，这是为了兼容**原版 llama.cpp**。

mmBERT 的分词方案（Metaspace + U+2581 规范化 + 全文 BPE）与 Gemma 4 同源，所以原版能识别。

已知差异：原版构建下，**直接对文本使用 llama-cli / llama-tokenize 时**，开头会少一个

空格前缀，且连续空格会被切成 ▁▁ 而不是 ▁ + ▁ 。

这不影响 Laya 推理** —— 运行时自己用 tokenizer.json 分词，再把 token id 数组发给

/embeddings （llama-server 接受原始 token 数组），完全绕过了 llama.cpp 的分词器。

要让文本分词也完全对齐，需要给 llama.cpp 打一个小补丁（新增 mmbert 预分词器 + BPE

路径的补前导空格，共 24 行）。补丁不在本仓库提供，如需要可自行实现。

验证结果

以下均为转换后实测，不是推断：

项目 结果

• -- ---

GGUF 结构 134 个编码器张量， modern-bert 元数据正确

llama.cpp 加载并执行编码器 通过，输出 768 维隐状态

编码器 vs 从检查点重构的 PyTorch 参考实现 最大绝对误差 **1.1**（数值量级 **41**，22 层 fp16 累加）

滑窗注意力是否真的生效 局部模式参考差 **1.1**，全注意力参考差 **30.4**

决策头权重经 GGUF 往返 36/36 张量完全一致，0 处不匹配

完整流程 vs 官方 laya 0.3.4（相同 token id，4 道题） 概率差 **≤ 0.0006**，argmax 全部一致

速度（CPU / Vulkan） 约 110 ms / 约 80 ms 每题

能力边界（实测）

可靠的

三道**信号明写、无需推断**的工单（取消订阅后仍被扣费并要求退款 / 点导出就崩溃 /

第二次投诉配送并明确说会取消），中英同义，每题一个分类题 + 一个是非题：

题目 英文 中文

• -- --- ---

该哪个团队处理 billing ✓ (1.000) billing ✓ (0.999)

是否要求退款 P=0.990 ✓ P=0.998 ✓

该哪个团队处理 technical ✓ (1.000) technical ✓ (0.998)

是否报软件问题 P=0.979 ✓ P=0.980 ✓

该哪个团队处理 support ✓ (0.923) support ✓ (0.986)

是否说会取消 P=0.983 ✓ P=0.958 ✓

英文 6/6，中文 6/6，判断结果完全一致。**

不可靠的

同一批隐含语义题（文中未直说、需要推断）：

题目 英文 中文

• -- --- ---

是否可能流失（"宁愿不要把时间花在换供应商上"） 0.158 ✗ 0.051 ✗

是否在质疑收费 0.004 ✗ 0.094 ✗

是否情绪不满（开头是"感谢"） —— 0.120 ✗

分层探测（中文，题面与工单同语言）：

层次 得分

• -- ---

字面层（文中是否出现 CFO / 40% / 28号 / CSV） **6/6**

明确语义（文中直说的事实） 3/4

隐含语义（文中未直说，需推断） **1/4**

四条可复现的结论

• **中文能读，但两种语言都不会推断。** 字面层满分说明全链路正常；崩塌点在推断上，

且中英一致。**边界不在语言，在"是否需要推断"。**

• **模型是确定性的。** 同一输入重复调用，极差为 0.000000 。
• **它对措辞的敏感度高于对语义的敏感度。** 同一道题只改选项措辞，P 从 0.043 变到 0.930。

把问题里的关键词原样放进选项能救回一部分（ needs human 0.227 → 0.611），

但救不回流失风险（0.218 → 0.168）。

• **置信度不可用于兜底。** 简单题答对时 P 在 0.92–1.00；而难题上它给 sales 打
• *0.791（错误）**、给 is bug report 打 **0.956（错误）**。**错的时候不会告诉你它不确定。**

官方定位

以上结果与官方文档一致：base 检查点在具体工作流上接近随机

（typed-decisions 基准 0.342，随机 0.318，多数类基线 0.461），

它比多数类基线还差**。价值来自按你自己的数据微调：

微调后同一基准从 0.362 提到 **0.766**

温度校准后 ECE 从 0.314 降到 0.106

适用与不适用

适合：**

明确的分类与是否判断，中英文均可，P ≥ 0.9 时自动放行是安全的

量大、单条价值低、需要一致口径的粗筛分流

本地部署、无 API 成本、数据不出内网

作为微调的起点

不适合：**

直接拿来做业务判断（新领域上接近随机，**不要开箱即用**）

任何需要"话里没明说但意思是那样"的推断

依赖置信度做自动化门控（出错时置信度不下降）

许可

原模型 Apache 2.0，作者 Convai Innovations。本转换仅改变序列化格式，不改变权重。

引用

模型主页：https://huggingface.co/convaiinnovations/laya-multilingual

PyPI：https://pypi.org/project/laya/

GitHub：https://github.com/NandaKishorM/laya

上传到魔搭（仓库维护者）

三个权重文件和本 README **必须一次上传**。只上传 README 单个文件不会带上权重。

modelscope upload 作用于**整个文件夹**，不是单个文件，所以直接指向目录即可：

上传后用 modelscope download 校验文件确实都在：

应当看到 6 个文件（3 个权重 + 2 个 .py + README.md）。若只剩 README.md，说明上传时路径指到了文件

而不是目录。

目录中若有多余文件（例如空的"新建 文本文档.txt"），建议先删除，或用

• -exclude "*.txt" 排除，避免污染仓库。

English

What this is

A **GGUF conversion** of convaiinnovations/laya-multilingual (laya 0.3.4), loadable by

llama.cpp.

Laya is a **non-autoregressive decision model** (System 1): given a *state* (text, email,

ticket, JSON) and typed questions, it returns typed answers with probabilities in a

single forward pass**. No text generation, so nothing to parse and nothing to

hallucinate.

• *Important: this is not a single self-contained file.** Read "Scope of this

conversion" first.

Scope of this conversion (read this first)

Laya is **not** a stock HuggingFace checkpoint, so convert hf to gguf.py cannot read it

as-is. It consists of:

part what it is runs where

• -- --- ---

encoder.* mmBERT-base bidirectional encoder: 22 layers, hidden 768, 12 heads, FFN 1152 (GeGLU), alternating full/sliding attention (window 128), RoPE theta 160000, 256k vocab **llama.cpp** ( modern-bert graph)

head.layers.* 2 extra nn.TransformerEncoderLayer s (d=768, nhead=12, ffn=3072, norm first ) runtime (PyTorch)

scorer.* , act head.* , type emb.* option-marker scorer, act/escalate head, question-type embedding runtime (PyTorch)

Decisions are produced by scoring every answer option at its own [MASK] token and then

softmaxing over that question's options. **This head is not part of llama.cpp's

modern-bert graph and cannot be added to it** — llama.cpp's loader requires the tensor

count in the file to match exactly the tensors its graph creates, so extra tensors make

loading fail outright.

So: **the 300M-parameter encoder runs inside llama.cpp, and the decision head runs in a

companion PyTorch runtime.**

Why two files

llama.cpp's loader validates the tensor count; not even one extra tensor is tolerated.

The head weights therefore cannot ride along in the encoder GGUF and must be a separate

companion file.

Files

This repository has 6 files. They **must sit in the same directory**:

file size purpose

• -- --- ---

laya-multilingual-f16.gguf 600 MB modern-bert encoder + 256k BPE tokenizer. **llama.cpp loads this directly.**

laya-multilingual-f16-head.gguf 57 MB decision-head weights, read by the runtime

tokenizer.json 33 MB the runtime tokenizes with this (it drives llama.cpp with explicit token ids)

laya gguf.py 11 KB the runtime itself (~200 lines)

gguf weights.py 1 KB helper that reads the head GGUF into tensors

README.md - this file

After downloading, the layout should be:

• *Downloading only laya-multilingual-f16.gguf is not enough.** The encoder will load,

but there is no decision capability - the head, the tokenizer and the runtime code live

in the other files.

Requirements

Python packages: torch , numpy , tokenizers , gguf

( pip install torch numpy tokenizers gguf ). llama.cpp handles the encoder, so

transformers is not needed.

Why three weight files are required

laya-multilingual-f16.gguf contains the encoder only. llama.cpp's loader validates the

tensor count and fails on even one extra tensor, so the head weights cannot be packed into

the same GGUF. The runtime needs tokenizer.json because it tokenizes itself and sends

the token id array to llama.cpp.

Quick start

• Start the llama.cpp server

On Windows PowerShell use .\llama-server.exe ; PowerShell does not load commands from

the current directory by default.

• Run it

This needs a companion runtime laya gguf.py (~200 lines of Python; it tokenizes, pulls

hidden states from llama.cpp, then runs the decision head in PyTorch). Put the three

weight files next to this README and run from that directory:

The three question types

type output use for

• -- --- ---

choice probability per option + confidence department routing, intent classification, topics

score expected level on an ordinal rubric urgency, frustration, severity

noul calibrated P(true), 0.0-1.0 yes/no judgements: spam, jailbreak, churn

About the tokenizer

The encoder GGUF records tokenizer.ggml.pre = gemma4 , chosen for compatibility with

stock llama.cpp**. mmBERT's tokenization scheme (Metaspace + U+2581 normalizer + BPE

over the whole text) is the same family as Gemma 4, so stock builds recognise it.

Known difference: on a stock build, **when you feed text directly to llama-cli /

llama-tokenize **, the leading space prefix is dropped and runs of spaces split as ▁▁

rather than ▁ + ▁ .

This does not affect Laya inference** — the runtime tokenizes with tokenizer.json

itself and sends the token id array to /embeddings (llama-server accepts a raw token

array), bypassing llama.cpp's tokenizer entirely.

Full text-tokenization parity would need a small llama.cpp patch (a new mmbert

pre-tokenizer plus a prepend-space step in the BPE path, 24 lines). The patch is not

included here; implement it yourself if you need it.

Verification

All measured after conversion, not inferred:

check result

• -- ---

GGUF structure 134 encoder tensors, correct modern-bert metadata

llama.cpp loads and runs the encoder pass, emits 768-d hidden states

Encoder vs a PyTorch reference rebuilt from the checkpoint max abs error **1.1** on values up to **41** (fp16 accumulation over 22 layers)

Sliding-window layers really use a 128-token window local-pattern reference **1.1**, all-full-attention reference **30.4**

Head weights round-trip through GGUF 36/36 tensors identical, 0 mismatches

Full pipeline vs official laya 0.3.4 (same token ids, 4 questions) probability diff **≤ 0.0006**, argmax identical

Latency (CPU / Vulkan) ~110 ms / ~80 ms per question

Capability boundaries (measured)

Reliable

Three tickets whose signal is **stated outright and needs no inference** (charged after

cancelling and asking for a refund / app crashes on export / second delivery complaint

that explicitly says they will cancel), one classification plus one yes/no question each,

English and Chinese:

question English Chinese

• -- --- ---

which team billing ✓ (1.000) billing ✓ (0.999)

asking for a refund P=0.990 ✓ P=0.998 ✓

which team technical ✓ (1.000) technical ✓ (0.998)

reporting a software problem P=0.979 ✓ P=0.980 ✓

which team support ✓ (0.923) support ✓ (0.986)

says they will cancel P=0.983 ✓ P=0.958 ✓

English 6/6, Chinese 6/6, identical decisions.**

Unreliable

Implied-meaning questions on the same tickets (not stated outright, requires inference):

question English Chinese

• -- --- ---

possible churn ("rather not spend next quarter onboarding a different vendor") 0.158 ✗ 0.051 ✗

disputing a charge 0.004 ✗ 0.094 ✗

sender is unhappy (opens with "thanks") - 0.120 ✗

Layered probe (Chinese, question language matching the ticket):

layer score

• -- ---

literal (does the text mention CFO / 40% / the 28th / CSV) **6/6**

explicit semantics (facts stated outright) 3/4

implied semantics (not stated, requires inference) **1/4**

Four reproducible findings

• **It reads Chinese, but neither language gets inference.** Full marks on the literal

layer proves the whole chain works; the collapse is on inference, and English and

Chinese fail equally. **The boundary is not language, it is whether inference is

required.**

• **The model is deterministic.** Repeated calls on the same input differ by 0.000000 .
• **It is more sensitive to wording than to meaning.** Changing only the option wording

moved one question's P from 0.043 to 0.930. Copying the question's keywords into the

options rescues some ( needs human 0.227 -> 0.611) but not churn risk (0.218 -> 0.168).

• **Confidence cannot be used as a safety net.** On easy questions it scores 0.92-1.00

when correct; on hard ones it gave **0.791 (wrong)** for sales and **0.956 (wrong)**

for is bug report . **It does not tell you when it is unsure.**

Official positioning

These results match the upstream documentation: the base checkpoint is near chance on

specific workflows (typed-decisions 0.342 against a 0.318 random and 0.461 majority-class

baseline) — **below the majority-class baseline**. The value comes from fine-tuning on

your own data:

fine-tuning moves the same benchmark from 0.362 to **0.766**

temperature calibration moves ECE from 0.314 to 0.106

Suitable / not suitable

Suitable for:**

explicit classification and yes/no judgements, in English or Chinese; auto-release at

P >= 0.9 is safe

high-volume, low-value-per-item triage that needs a consistent rule

local deployment, no API cost, data never leaves your network

a starting point for fine-tuning

Not suitable for:**

making business decisions directly (near chance in a new domain — **do not use

out of the box**)

anything needing "they did not say it but they meant it" inference

gating on confidence (confidence does not drop when it is wrong)

License

The original model is Apache 2.0 by Convai Innovations. This conversion only changes the

serialization format; the weights are unchanged.

Uploading to ModelScope (repository maintainers)

The three weight files and this README **must be uploaded together**. Uploading the README

alone does not bring the weights along.

modelscope upload operates on a **whole folder**, not a single file, so point it at the

directory:

Verify afterwards with modelscope download :

You should see 6 files (3 weights + 2 .py files + README.md). If only README.md is present, the upload path

pointed at a file rather than the directory.

Remove stray files first (for example an empty 新建 文本文档.txt ), or exclude them with

• -exclude "*.txt" , so the repository stays clean.