MeloTTS-ZH-MIXED-EN-ONNX

MeloTTS-ONNX 项目详解

• 项目概述

MeloTTS-ONNX** 是 MeloTTS 的 ONNX 推理版本，专门针对 **CPU 实时推理**进行了优化。项目支持：

✅ 中英文混合 TTS

✅ 多种语言：中文、英文、日文、韩文、西班牙语、法语等

✅ ONNX Runtime 推理，推理速度快

仓库地址

Gitee仓库: https://gitee.com/jackroing/melo-tts-onnx.git

GitHub仓库: https://github.com/201831771214/MeloTTS-ONNX.git

ModelScope模型仓库: https://www.modelscope.cn/models/KeanuX/MeloTTS-ZH-MIXED-EN-ONNX

• 项目架构

• 核心脚本使用

• 1 推理脚本： run onnx.py

这是**最常用的脚本**，用于将文本转换为语音：

主要参数说明：**

参数 说明 默认值

• ----- ------ --------

text 输入文本 必填

language 语言代码 "ZH MIX EN"

sdp ratio SDP 比率 (0-1) 0.2

noise scale 噪声尺度 0.667

noise scale w 噪声权重 0.8

speed 语速 1.0

支持的语言代码：**

ZH MIX EN - 中文（支持中英文混合）

EN - 英文

等

• 2 ONNX 导出脚本： export melo.py

用于将 PyTorch 模型导出为 ONNX 格式：

主要参数：**

• -ckpt path - 模型检查点路径
• -cfg path - 配置文件路径
• -output path - 输出 ONNX 文件路径
• -opset - ONNX opset 版本（默认14）

• 3 模型信息导出： export model info.py

用于导出 ONNX 模型的详细信息（输入输出形状、参数数量等）：

输出示例：

Update信息

更新MeloTTS ONNX Static Model:

增加melotts 14 static.onnx模型的序列长度到512,减少中间层参数量为原来的一半

Precompiled QNN ONNX Profile Info

各位小伙伴在运行MeloTTS QNN ONNX EP的时候关闭如下输入即可：

因为原声MeloTTS源码中采用DP + SDP的方案保证音频质量，动态计算frames，但是FP16在QNN EP中会自动崩掉，在这里小编也不知道原因，所以咋们就直接采用固定frames的方案来直接规避掉FP16精度计算崩掉的问题，而且模型计算单元也从复杂的2372减小到了1760, 在计算精度上也基本没有什么损失，而且还加速了运算速度，简直是一石二鸟的买卖。

• 工作原理

ONNX Dynamic模型输入 (11个)：**

• x tst - 文本 token IDs
• x tst lengths - 文本长度
• speakers - 发音人 ID
• tones - 音调 IDs
• lang ids - 语言 IDs
• bert - BERT 特征 (1024维)
• ja bert - 日文 BERT 特征 (768维)
• sdp ratio - SDP 比率
• noise scale - 噪声尺度
• noise scale w - 噪声权重
• speed - 语速

ONNX 模型输出 (1个)：**

audio data - 生成的音频数据

ONNX Static模型输入 (10个)：**

• x tst - 文本 token IDs
• x tst lengths - 文本长度
• speakers - 发音人 ID
• tones - 音调 IDs
• lang ids - 语言 IDs
• bert - BERT 特征 (1024维)
• ja bert - 日文 BERT 特征 (768维)
• sdp ratio - SDP 比率
• noise scale w - 噪声权重
• speed - 语速

ONNX 模型输出 (1个)：**

audio data - 生成的音频数据

注意: 代码示例只适合Dynamic模型的部署,如需部署Static模型请自行调整代码中的generate audio方法,请设置Chunk大小为239,超过这个大小就将输入分为多个part依次处理,最后合并所有音频段.

• 快速使用示例

python

"""MeloTTS ONNX Runtime Inference

Static ONNX Model Inference:

============================================================

模型输入信息 (共 10 个输入)

============================================================

Input 1: x tst

数据类型: int32

形状: [1, 239]

Input 2: x tst lengths

数据类型: int32

形状: [1]

Input 3: speakers

数据类型: int32

形状: [1]

Input 4: tones

数据类型: int32

形状: [1, 239]

Input 5: lang ids

数据类型: int32

形状: [1, 239]

Input 6: bert

数据类型: float32

形状: [1, 1024, 239]

Input 7: ja bert

数据类型: float32

形状: [1, 768, 239]

Input 8: sdp ratio

数据类型: float32

形状: [1]

Input 9: noise scale w

数据类型: float32

形状: [1]

Input 10: speed

数据类型: float32

形状: [1]

============================================================

模型输出信息 (共 1 个输出)

============================================================

Output 1: audio data

数据类型: float32

形状: [1, 0]

Dynamic ONNX Model Inference:

============================================================

ONNX模型基本信息

============================================================

模型文件路径: ./models/melotts onnx/melotts 14 dynamic.onnx

ONNX版本: 7

生产者信息: pytorch 2.8.0

模型版本: 0

描述:

============================================================

模型输入信息 (共 11 个输入)

============================================================

Input 1: x tst

数据类型: int32

形状: [0, 0]

Input 2: x tst lengths

数据类型: int32

形状: [0]

Input 3: speakers

数据类型: int32

形状: [0]

Input 4: tones

数据类型: int32

形状: [0, 0]

Input 5: lang ids

数据类型: int32

形状: [0, 0]

Input 6: bert

数据类型: float32

形状: [0, 1024, 0]

Input 7: ja bert

数据类型: float32

形状: [0, 768, 0]

Input 8: sdp ratio

数据类型: float32

形状: [0]

Input 9: noise scale

数据类型: float32

形状: [0]

Input 10: noise scale w

数据类型: float32

形状: [0]

Input 11: speed

数据类型: float32

形状: [0]

============================================================

模型输出信息 (共 1 个输出)

============================================================

Output 1: audio data

数据类型: float32

形状: [1, 0]

QNN ONNX Model Inference:

============================================================

ONNX模型基本信息

============================================================

模型文件路径: ./models/melotts qnn/melotts 14/model.onnx

ONNX版本: 11

生产者信息: Qualcomm AI Hub Workbench aihub-2026.02.26.0

模型版本: 0

描述:

============================================================

模型输入信息 (共 10 个输入)

============================================================

Input 1: x tst

数据类型: int32

形状: [1, 239]

Input 2: x tst lengths

数据类型: int32

形状: [1]

Input 3: speakers

数据类型: int32

形状: [1]

Input 4: tones

数据类型: int32

形状: [1, 239]

Input 5: lang ids

数据类型: int32

形状: [1, 239]

Input 6: bert

数据类型: float32

形状: [1, 1024, 239]

Input 7: ja bert

数据类型: float32

形状: [1, 768, 239]

Input 8: sdp ratio

数据类型: float32

形状: [1]

Input 9: noise scale w

数据类型: float32

形状: [1]

Input 10: speed

数据类型: float32

形状: [1]

============================================================

模型输出信息 (共 1 个输出)

============================================================

Output 1: audio data

数据类型: float32

形状: [1, 429568]

============================================================

模型权重信息 (共 0 个参数)

============================================================

模型总参数量: 0

============================================================

模型运算符信息 (共 1 个运算符)

============================================================

运算符类型 使用次数

• -----------------------------

EPContext 1

详细层信息:

• -----------------------------------------------------------

层 1: QNNContext (EPContext)

输入: x tst, x tst lengths, speakers, tones, lang ids, bert, ja bert, sdp ratio, noise scale w, speed

输出: audio data

属性:

• embed mode:
• ep cache context: ./model.bin
• source: QNN

============================================================

额外信息

============================================================

导入版本: -13

模型计算图名称: qnn-onnx-model

"""

import onnxruntime as ort

import numpy as np

import os

import sys

import soundfile as sf

from typing import Tuple

from melo extra.inference.utils import HParams, get hparams from file

from melo extra.inference.text.cleaner import clean text

from melo extra.inference.text import cleaned text to sequence, get bert, get zh mix en bert

from melo extra.inference import commons

import argparse

import logging

logger = logging.getLogger( name )

file handler = logging.FileHandler("./logs/run melo onnx.log", mode="w", encoding="utf-8")

formatter = logging.Formatter("%(asctime)s - %(levelname)s - %(message)s")

file handler.setFormatter(formatter)

file handler.setLevel(logging.INFO)

logger.addHandler(file handler)

logger.setLevel(logging.INFO)

class MeloTTS:

def init (self, model root:str, device:str="cpu", provider options:list[dict]=None, is dynamic:bool=True) -> None:

self.hop size = 512 # Generator 上采样倍率

self.model list = {}

for f in os.listdir(model root):

if f.endswith(".onnx"):

f name = f[0:]

logger.info(f"find model: {f name}")

self.model list[f name] = os.path.join(model root, f)

if is dynamic:

for m k, m v in self.model list.items():

if "dynamic" in m k.lower():

self.model name = m v

break

else:

self.model name = None

continue

else:

if device == "qnn":

self.model name = os.path.join(model root, "precompiled qnn onnx", "model.onnx")

else:

for m k, m v in self.model list.items():

if "static" in m k.lower():

self.model name = m v

break

else:

self.model name = None

continue

if self.model name == None:

logger.error(f"model name not found, please check model root: {model root}")

sys.exit(1)

logger.info(f"Use model: {self.model name}")

self.model path = os.path.join(self.model name)

self.cfg path = os.path.join(model root, "config.json")

self.bert model path = os.path.join(model root, "bert-base-multilingual-uncased")

self.cfg = get hparams from file(self.cfg path)

self.sample rate = self.cfg.data.sampling rate

if device == "cuda" and "CUDAExecutionProvider" in ort.get available providers():

self.providers = ["CUDAExecutionProvider"]

elif device == "cpu":

self.providers = ["CPUExecutionProvider"]

elif device == "qnn" and "QNNExecutionProvider" in ort.get available providers() and provider options != None:

self.providers = ["QNNExecutionProvider"]

else:

logger.info(f"device {device} not supported, use cpu instead")

self.providers = ["CPUExecutionProvider"]

self.provider options = provider options if self.providers == ["QNNExecutionProvider"] else None

self.session = ort.InferenceSession(self.model path, providers=self.providers, provider options=provider options)

logger.info(f"Using EP: {self.session.get providers()}, model inputs: {len(self.session.get inputs())}")

self.input names = [input.name for input in self.session.get inputs()]

self.output names = [output.name for output in self.session.get outputs()]

# 自动检测浮点 dtype：匹配模型期望精度（fp32 或 fp16）

onnx dtype to np = {1: np.float32, 10: np.float16} # TensorProto.FLOAT=1, FLOAT16=10

self.float dtype = np.float32 # 默认

for inp in self.session.get inputs():

if inp.name == "bert": # bert 特征一定是浮点输入

self.float dtype = onnx dtype to np.get(inp.type, np.float32)

break

logger.info(f"model input names: {self.input names}")

logger.info(f"model output names: {self.output names}")

logger.info(f"auto-detected float dtype: {self.float dtype}")

@staticmethod

def trim trailing silence(audio: np.ndarray) -> np.ndarray:

"""裁剪尾部空闲段。

用 RMS 能量而非峰值——空闲噪音可能有单点尖峰（~0.06），但 RMS

能量远低于真实语音，区分度比峰值大得多。

"""

if len(audio) threshold)[0]

if len(above) == 0:

logger.warning(" trim: no frame above threshold, returning empty")

return audio[:0]

cut frame = above[-1] + 1

cut sample = min(cut frame * win, len(audio))

logger.info(f" trim: cut at frame {above[-1]}/{len(frame rms)}, "

f"sample {cut sample}/{len(audio)}")

return audio[:cut sample]

def preprocess(self, text:str, language:str):

"""Preprocess text input

Args:

text (str): Input text to be preprocessed.

language (str): Language of the input text.

Returns:

Tuple[np.ndarray, np.ndarray, np.ndarray, np.ndarray]: Preprocessed tensors ready for inference.

"""

norm text, phone, tone, word2ph = clean text(text, language)

symbol to id = {s: i for i, s in enumerate(self.cfg.symbols)}

phone, tone, language = cleaned text to sequence(phone, tone, language, symbol to id)

if self.cfg.data.add blank:

phone = commons.intersperse(phone, 0)

tone = commons.intersperse(tone, 0)

language = commons.intersperse(language, 0)

for i in range(len(word2ph)):

word2ph[i] = word2ph[i] * 2

word2ph[0] += 1

if getattr(self.cfg.data, "disable bert", True):

bert = np.zeros((1024, len(phone)), dtype=self.float dtype)

ja bert = np.zeros((768, len(phone)), dtype=self.float dtype)

else:

bert = get zh mix en bert(self.bert model path, text, word2ph, "cpu")

del word2ph

assert bert.shape[-1] == len(phone), phone

if language == "ZH":

bert = bert.astype(self.float dtype)

ja bert = np.zeros((768, len(phone)), dtype=self.float dtype)

elif language in ["JP", "EN", "ZH MIX EN", 'KR', 'SP', 'ES', 'FR', 'DE', 'RU']:

ja bert = bert.astype(self.float dtype)

bert = np.zeros((1024, len(phone)), dtype=self.float dtype)

else:

raise NotImplementedError()

assert bert.shape[-1] == len(

phone

), f"Bert seq len {bert.shape[-1]} != {len(phone)}"

phone = np.array(phone, dtype=np.int32)

tone = np.array(tone, dtype=np.int32)

language = np.array(language, dtype=np.int32)

x tst = np.expand dims(phone, axis=0)

x tst lengths = np.array([phone.size], dtype=np.int32)

tones = np.expand dims(tone, axis=0)

lang ids = np.expand dims(language, axis=0)

bert = np.expand dims(bert, axis=0)

ja bert = np.expand dims(ja bert, axis=0)

speaker id = np.array([1], dtype=np.int32)

return x tst, x tst lengths, speaker id, tones, lang ids, bert, ja bert

def generate audio chunked(self,

text:str,

language:str="ZH MIX EN",

sdp ratio:float=0.2,

noise scale w:float=0.8,

speed:float=1.0,

chunk size:int=239):

"""Generate audio chunked

Args:

text (str): Input text to be synthesized.

language (str, optional): Language of the input text. Defaults to "ZH MIX EN".

sdp ratio (float, optional): Ratio for SDP (Speech Decoder Proportion). Defaults to 0.2.

noise scale w (float, optional): Scale factor for noise. Defaults to 0.8.

speed (float, optional): Speed factor for audio playback. Defaults to 1.0.

chunk size (int, optional): Size of audio chunks for processing. Defaults to 239.

Returns:

Tuple[np.ndarray, int]: Audio data and sample rate

"""

if language.lower() == "zh mix en" or language.lower() == "zh":

language = "ZH MIX EN"

elif language.lower() == "en":

language = "EN"

else:

raise ValueError(f"language {language} not supported")

x tst, x tst lengths, speaker id, tones, lang ids, bert, ja bert = self. preprocess(text, language)

np sdp ratio = np.array([sdp ratio], dtype=self.float dtype)

np noise scale = np.array([0.667], dtype=self.float dtype)

np noise scale w = np.array([noise scale w], dtype=self.float dtype)

np speed = np.array([speed], dtype=self.float dtype)

total len = x tst lengths[0]

num part = total len // chunk size + (1 if total len % chunk size != 0 else 0)

logger.info(f"total len: {total len}, num part: {num part}")

audio seg = []

for part in range(num part):

start = part * chunk size

end = min((part + 1) * chunk size, total len)

actual len = end - start # ← 保存实际长度，不被覆盖

pad len = chunk size - actual len # ← 需要补的长度

x tst part = x tst[:, start:end]

tone part = tones[:, start:end]

lang ids part = lang ids[:, start:end]

bert part = bert[:, :, start:end]

ja bert part = ja bert[:, :, start:end]

# 统一用 actual len 判断，用 pad len 补齐

if pad len > 0:

x tst part = np.pad(x tst part, ((0, 0), (0, pad len)), constant values=0)

tone part = np.pad(tone part, ((0, 0), (0, pad len)), constant values=0)

lang ids part = np.pad(lang ids part, ((0, 0), (0, pad len)), constant values=0)

bert part = np.pad(bert part, ((0, 0), (0, 0), (0, pad len)), constant values=0)

ja bert part = np.pad(ja bert part, ((0, 0), (0, 0), (0, pad len)), constant values=0)

# x tst lengths 传实际长度（模型内部用 mask 处理），shape固定为chunk size

x tst lengths part = np.array([actual len], dtype=np.int32)

logger.info(f"part {part}: start={start}, end={end}, actual len={actual len}, pad len={pad len}")

logger.info(f" x tst part shape: {x tst part.shape}")

logger.info(f" tone part shape: {tone part.shape}")

# 按名字映射输入，兼容 9 输入（QNN）和 10 输入（本地 ONNX）

input spec = {}

named inputs = {

"x tst": x tst part,

"x tst lengths": x tst lengths part,

"speakers": speaker id,

"tones": tone part,

"lang ids": lang ids part,

"bert": bert part,

"ja bert": ja bert part,

"sdp ratio": np sdp ratio,

"noise scale": np noise scale,

"noise scale w": np noise scale w,

"speed": np speed,

}

for k, v in named inputs.items():

if k in self.input names:

input spec[k] = v

output spec = self.session.run(self.output names, input spec)

# 诊断：QNN 原始输出统计

audio out = output spec[0]

logger.info(f"part {part}: raw output shape={audio out.shape}, "

f"min={audio out.min():.6f}, max={audio out.max():.6f}, "

f"nonzero={np.count nonzero(audio out)}")

# 如果有 y lengths 输出，用它精确截断（最可靠）

if len(output spec) > 1:

y len = int(output spec[1][0]) # mel 帧数

valid samples = y len * self.hop size

audio full = np.squeeze(audio out, axis=0)[:valid samples]

logger.info(f"part {part}: y lengths={y len}, valid samples={valid samples}, "

f"trimmed len={len(audio full)}")

else:

# 兼容旧模型：用静音检测截断

audio full = np.squeeze(audio out, axis=0)

audio full = self. trim trailing silence(audio full)

logger.info(f"part {part}: full audio len={audio out.shape[-1]}, "

f"trimmed len={audio full.shape[0]}")

audio seg.append(audio full)

combined audio = np.concatenate(audio seg, axis=0)

return combined audio, self.sample rate

def generate audio(self,

text:str,

language:str="ZH MIX EN",

sdp ratio:float=0.2,

noise scale:float=0.667,

noise scale w:float=0.8,

speed:float=1.0) -> Tuple[np.ndarray, int]:

""" summary

Args:

text (str): User input text

language (str, optional): Language of the text. Defaults to "ZH MIX EN".

sdp ratio (float, optional): Ratio of SDP. Defaults to 0.2.

noise scale (float, optional): Scale of noise. Defaults to 0.667.

noise scale w (float, optional): Weight of noise scale. Defaults to 0.8.

speed (float, optional): Speed of the audio. Defaults to 1.0.

Returns:

Tuple[np.ndarray, int]: Audio data and sample rate

"""

if language.lower() == "zh mix en" or language.lower() == "zh":

language = "ZH MIX EN"

elif language.lower() == "en":

language = "EN"

else:

raise ValueError(f"language {language} not supported")

x tst, x tst lengths, speaker id, tones, lang ids, bert, ja bert = self. preprocess(text, language)

np sdp ratio = np.array([sdp ratio], dtype=self.float dtype)

np noise scale = np.array([noise scale], dtype=self.float dtype)

np noise scale w = np.array([noise scale w], dtype=self.float dtype)

np speed = np.array([speed], dtype=self.float dtype)

input spec = {

self.input names[0]: x tst,

self.input names[1]: x tst lengths,

self.input names[2]: speaker id,

self.input names[3]: tones,

self.input names[4]: lang ids,

self.input names[5]: bert,

self.input names[6]: ja bert,

self.input names[7]: np sdp ratio,

self.input names[8]: np noise scale,

self.input names[9]: np noise scale w,

self.input names[10]: np speed,

}

output spec = self.session.run(self.output names, input spec)[0]

audio data = np.squeeze(output spec, axis=0)

return audio data, self.sample rate

default model path = "./models/MeloTTS-ZH-MIXED-EN-ONNX/"

default output path = "./"

default tes