Qwen-72B-Chat-Int4

Qwen-72B-Chat-Int4

🤗 Hugging Face&nbsp&nbsp &nbsp&nbsp🤖 ModelScope&nbsp&nbsp &nbsp&nbsp 📑 Paper &nbsp&nbsp ｜ &nbsp&nbsp🖥️ Demo

WeChat (微信)&nbsp&nbsp &nbsp&nbspDiscord&nbsp&nbsp ｜ &nbsp&nbspAPI

介绍（Introduction）

通义千问-72B**（**Qwen-72B**）是阿里云研发的通义千问大模型系列的720亿参数规模的模型。Qwen-72B是基于Transformer的大语言模型, 在超大规模的预训练数据上进行训练得到。预训练数据类型多样，覆盖广泛，包括大量网络文本、专业书籍、代码等。同时，在Qwen-72B的基础上，我们使用对齐机制打造了基于大语言模型的AI助手Qwen-72B-Chat。本仓库为Qwen-72B-Chat的Int4量化模型的仓库。

通义千问-72B（Qwen-72B）主要有以下特点：

• **大规模高质量训练语料**：使用超过3万亿tokens的数据进行预训练，包含高质量中、英、多语言、代码、数学等数据，涵盖通用及专业领域的训练语料。通过大量对比实验对预训练语料分布进行了优化。
• **强大的性能**：Qwen-72B在多个中英文下游评测任务上（涵盖常识推理、代码、数学、翻译等），效果显著超越现有的开源模型。具体评测结果请详见下文。
• **覆盖更全面的词表**：相比目前以中英词表为主的开源模型，Qwen-72B使用了约15万大小的词表。该词表对多语言更加友好，方便用户在不扩展词表的情况下对部分语种进行能力增强和扩展。
• **更长的上下文支持**：Qwen-72B支持32k的上下文长度。
• **系统指令跟随**：Qwen-72B-Chat可以通过调整系统指令，实现**角色扮演**，**语言风格迁移**，**任务设定**，和**行为设定**等能力。

如果您想了解更多关于通义千问72B开源模型的细节，我们建议您参阅GitHub代码库。

Qwen-72B** is the 72B-parameter version of the large language model series, Qwen (abbr. Tongyi Qianwen), proposed by Alibaba Cloud. Qwen-72B is a Transformer-based large language model, which is pretrained on a large volume of data, including web texts, books, codes, etc. Additionally, based on the pretrained Qwen-72B, we release Qwen-72B-Chat, a large-model-based AI assistant, which is trained with alignment techniques. This repository is the one for the Int4 quantized model of Qwen-72B-Chat.

The features of Qwen-72B include:

• **Large-scale high-quality training corpora**: It is pretrained on over 3 trillion tokens, including Chinese, English, multilingual texts, code, and mathematics, covering general and professional fields. The distribution of the pre-training corpus has been optimized through a large number of ablation experiments.
• **Competitive performance**: It significantly surpasses existing open-source models on multiple Chinese and English downstream evaluation tasks (including commonsense, reasoning, code, mathematics, etc.). See below for specific evaluation results.
• **More comprehensive vocabulary coverage**: Compared with other open-source models based on Chinese and English vocabularies, Qwen-72B uses a vocabulary of over 150K tokens. This vocabulary is more friendly to multiple languages, enabling users to directly further enhance the capability for certain languages without expanding the vocabulary.
• **Longer context support**: Qwen-72B supports 32k context length.
• **System prompt**: Qwen-72B can realize roly playing, language style transfer, task setting, and behavior setting by using system prompt.

For more details about the open-source model of Qwen-72B, please refer to the GitHub code repository.

要求（Requirements）

python 3.8及以上版本

pytorch 2.0及以上版本

建议使用CUDA 11.4及以上（GPU用户、flash-attention用户等需考虑此选项）

• **至少需要48GB显存（例如1xA100-80G或2xV100-32G）**

python 3.8 and above

pytorch 2.0 and above, 2.0 and above are recommended

CUDA 11.4 and above are recommended (this is for GPU users, flash-attention users, etc.)

• *at least 48GB GPU memory is required (e.g., 1xA100-80G or 2xV100-32G)**

依赖项（Dependency）

运行Qwen-72B-Chat-Int4，请确保满足上述要求，再执行以下pip命令安装依赖库。如安装 auto-gptq 遇到问题，我们建议您到官方repo搜索合适的预编译wheel。

To run Qwen-72B-Chat-Int4, please make sure you meet the above requirements, and then execute the following pip commands to install the dependent libraries. If you meet problems installing auto-gptq , we advise you to check out the official repo to find a pre-build wheel.

注意：预编译的 auto-gptq 版本对 torch 版本及其CUDA版本要求严格。同时，由于

其近期更新，你可能会遇到 transformers 、 optimum 或 peft 抛出的版本错误。

我们建议使用符合以下要求的最新版本：

• torch==2.1 auto-gptq>=0.5.1 transformers>=4.35.0 optimum>=1.14.0 peft>=0.6.1
• torch>=2.0, - torch==2.1 auto-gptq>=0.5.1 transformers>=4.35.0 optimum>=1.14.0 peft>=0.6.1
• torch>=2.0,