nlp_polylm_13b_text_generation

PolyLM多语言文本生成模型（PolyLM-文本生成模型-多语言-13B-演示）

模型简介

PolyLM是一个通晓多种语言的大规模语言模型，涵盖中文、英文、西班牙语、法语、德语、俄语、葡萄牙语、意大利语、阿拉伯语、日语、韩语、泰语、越南语和印尼语等18个语言。该模型可以应用于对话问答、文本生成、机器翻译和情感分析等领域，能够自动生成高质量的多语言文本，从而为跨语言、文化的交流提供便利。

Abstract in English

Large language models (LLMs) demonstrate remarkable ability to comprehend, reason, and generate following nature language instructions. However, the development of LLMs has been primarily focused on high-resource languages, such as English, thereby limiting their applicability and research in other languages. Consequently, we present PolyLM, a multilingual LLM trained on 640 billion (B) tokens, avaliable in two model sizes: 1.7B and 13B. To enhance its multilingual capabilities, we 1) integrate bilingual data into training data; and 2) adopt a curriculum learning strategy that increases the proportion of non-English data from 30% in the first stage to 60% in the final stage during pre-training. Further, we propose a multilingual self-instruct method which automatically generates 132.7K diverse multilingual instructions for model fine-tuning. To assess the model's performance, we collect several existing multilingual tasks, including multilingual understanding, question answering, generation, and translation. Extensive experiments show that PolyLM surpasses other open-source models such as LLaMA and BLOOM on multilingual tasks while maintaining comparable performance in English.

Our models, alone with the multilingual instruction data, are available at Github and Huggingface.

模型版本

本项目提供了一系列不同规模和用途的模型，参数规模包括1.7B/13B版本（当前模型为13B版本），同时涵盖了预训练底座模型以及指令精调后的Chat版本（即MultiAlpaca系列）。全部版本如下表所示：

Model Precision Layers Heads Hidden Max length LR Batch Type

• -- --- --- --- --- --- --- --- ---

PolyLM-1.7B bfloat16 24 16 2048 2048 1.0e-4 4M Pretrain Model

PolyLM-13B bfloat16 40 40 5120 2048 6.0e-5 4M Pretrain Model

PolyLM-MultiAlpaca-13B bfloat16 40 40 5120 2048 6.0e-5 4M Chat Model

PolyLM-Assistant-13B bfloat16 40 40 5120 2048 6.0e-5 4M Chat Model

实验结果

多语言理解实验结果

多语言问答和机器翻译实验结果

模型下载

模型使用

微调(SFT)

代码链接**: https://github.com/modelscope/swift/tree/main/examples/pytorch/llm

• 支持的sft方法: lora, qlora, 全参数微调, ...
• 支持的模型: qwen系列, qwen-vl系列, baichuan系列, chatglm2系列, llama系列, openbuddy-llama系列, internlm系列, xverse系列, ...
• 支持的特性: 模型量化, DDP, 模型并行, gradient checkpointing, 梯度累加, 支持推送ModelScope Hub, 自定义数据集, 多模态和Agent SFT, 多轮对话, ...

使用qlora+ddp+deepspeed SFT polylm-13b的脚本 (需要2*13GB显存)

论文引用

如果你觉得这个该模型对有所帮助，请考虑引用下面的相关的论文：