Model details
模型介绍 中文整理 Rombos-LLM-V2.5-Qwen-32B-GPTQ-Int4-量化修复
HF原始模型 rombodawg/Rombos-LLM-V2.5-Qwen-32b
【留给Qwen32B刚需使用者的一个Note】
该模型自Qwen1.5以来,非常容易出现 !!!!!!! 的核心原因在于个别 layers ,例如 layers.4 、 layers.5 和 layers.6 ,中的MLP模块的表征,学成了一个值域范围非常大的张量。其在推理过程中,需要用超过fp16精度来表示。
核心解决办法有三
(1) 将有问题layers回炉,用某种方式重新炼一下(我不太清楚有什么温和的处理手段,欢迎留言投稿献计)。
(2) 用可以支持TF16部署格式的格式进行部署。
(3) 对相关节点做峰值裁减处理
经过测试,也可以无脑对所有MLP节点进行裁减,具体操作如下:
找到vllm中的qwen2模型定义文件 vllm/model executor/models/qwen2.py
定位到 Qwen2DecoderLayer 的 forward 的尾部,或 line:220~221 , 插入一行峰值裁减,如下
这样就完成了一个对mlp节点进行正负3000的浮点峰值裁减的操作。
之后,就可以愉快的体验了,祝好。
【模型更新日期】
注:通过 snapshot download 函数传入 revision=... 来下载指定的 tag 版本
【模型列表】
tag 文件大小 最近更新时间
--------- -------- -------------- g128v2 19GB 2024-10-31
g128 19GB 2024-10-25
g32 20GB 2024-10-25
【修复内容】
对GPTQ量化的校准做了额外优化;减少模型的 1.乱吐字 、 2.无限循环 、 3.长文能力丢失 等情况。 有些推理框架的默认 top k 与 top p 较大,可以考虑减小对应数值,来获得更合理的模型输出。 根据模型实际情况,可以支持1卡、2卡及4卡的 tensor-parallel-size 启动。 【介绍】
Rombos-LLM-V2.5-Qwen-32b is a continues finetuned version of Qwen2.5-32B. I noticed recently that the Qwen team did not learn from my methods of continuous finetuning, the great benefits, and no downsides of it. So I took it upon myself to merge the instruct model with the base model myself using the *Ties* merge method
This version of the model shows higher performance than the original instruct and base models.
【模型下载】
【高并发RESTFul API推理】
方式1:vllm
方式2:sglang
优先采用原作者提供的中文模型卡片;仅有英文说明时由闲社AI翻译整理。重要参数、许可证和商用范围请以原文为准。
你将获得什么 完整仓库方式包含该固定版本中的权重、配置、分词器及说明文件;指定版本方式只下载所选文件或整组分片。多模态模型的投影文件、配置等依赖,请按作者说明配齐。
完整仓库可能包含多个权重格式,因此需要的磁盘空间可能大于单一格式的模型大小。下载文件不等于完成模型部署。
使用前须知 运行环境 运行模型需要的设备、工具与依赖,以原作者说明为准。
授权范围 是否允许商用、修改和分发,请查看模型许可证。
闲社提供资料与镜像下载方法,不代表已运行模型或完成安全审计。下载的代码应先检查,再执行。
Problem with the information or download? Sign in to submit feedback.
Pinned version · Multiple download options
rombos-llm-v2.5-qwen-32b-gptq-int4 model download Download the full repository or choose a version or file.
Choose a download source ModelScope · China direct
Pinned ModelScope version: dce205301efdd1f0122dfbc62c6dcc65161018ba. Version identifiers can differ across sites. The alternative source was provided after checking weights and runtime configuration; this is not proof of a successful model run.
A checked, pinned ModelScope source is available for this model. Use the ModelScope download tool.
Choose what to download
Full repository · All files Selected version / file
Need just one GGUF precision? Load the file manifest and choose a version. Direct file links do not require Python; the command below can download multiple shards together.
Load available versions and files
Quantization / file Load the manifest to choose
This reads the pinned version's file manifest; no model weights are downloaded.
Choose your operating system Windows · PowerShell Linux / macOS · Terminal
For command-line downloads, follow these steps to get the full repository or selected files. Direct file links do not need these steps.
1 Open a terminal window Create and open a folder for the model. In the folder's address bar, enter powershell and press Enter.
If Python is not installed, get it from the official Python website . In the terminal, enter python --version; continue when a version number appears.
2 First time? Install the download tool Copy this command, paste it into the terminal, press Enter and wait for installation to finish.
python -m pip install --upgrade huggingface_hubCopy install command Already installed? Continue to step 3.
3 Run the download command Select the button below, paste into the same terminal and press Enter. The tool downloads all files included in the current command.
Copy full download command
Copying does not start a download. Paste and run the command on your device.
Where are the downloaded files? Open the folder you chose. Look for the models folder. When progress stops and the terminal returns to its prompt without errors, check that all files are present. 'Copied' does not mean 'downloaded'.
No response or an error? python not found: Install Python and reopen the terminal. On Windows, follow the installer's instructions to enable command-line access.
hf not found: Confirm that step 2 succeeded, then reopen the terminal. If needed, add Python's Scripts folder to your user PATH. Do not repeatedly download scripts.
Network interrupted: Stay in the same folder and run the same command again. The tool checks existing files.
Permission or disk error: Choose a writable folder with enough free space. Do not disable system security protection.
View the full download command
请开启浏览器脚本,或使用默认 Windows 教程并展开命令手动复制执行。