blsp_lslm_7b

BLSP: Bootstrapping Language-Speech Pre-training via Behavior Alignment of Continuation Writing

Chen Wang, Minpeng Liao, Zhongqiang Huang, Jinliang Lu, Junhong Wu, Yuchen Liu, Chenqing Zong, Jiajun Zhang

Institute of Automation, Chinese Academy of Sciences**

Machine Intelligence Technology Lab, Alibaba DAMO Academy**

模型介绍

BLSP是一个大规模语音语言模型，能够同时理解语音和文本，支持语音文本间跨模态交互。该模型可以应用于语音对话/问答、语音识别、语音翻译和语音情感分析等领域，能够自动生成高质量的多语言文本，从而为跨模态、跨语言的交流提供便利。

Introduction

BLSP extends the language capabilities of LLMs to speech, enabling iteraction with LLMs through spoken languages.

We learn the BLSP model via behavior alignment of continuation writing. Ideally, LLMs should have the same behavior regardless of the modality of input, a speech segment or its transcript.

• The first step utilizes LLMs to generate text with the transcript as prefix, obtaining the continuations.
• The second step leverages the text continuations as supervised signals to learn the modality adapter by requiring LLMs to predict the text continuations given the speech segment.

Examples

More examples with video presentations can be found in the project page.

Usage

Enviroment Preparation

All experiments are carried out in the following environment.

Python==3.8

torch==1.13, torchaudio==0.13.0, cuda==11.6

transformers==4.31.0

soundfile==0.12.1, openai-whisper

datasets==2.12.0, accelerate==0.21.0, deepspeed==0.9.3

evaluate==0.4.0, sentencepiece==0.1.99

fire==0.5.0, gradio==3.41.2

Prepare the pretrained BLSP checkpoint

Download the pretrained BLSP model link

Inference & Evaluation

We release the inference code for evaluation.

The supported input file is .jsonl format. Here is an example of ST task. Each line of input file looks like

Then run the generation code

Launching Demo Locally

You can try out our demo by

Training from Scratch

The training of BLSP contains two stages.

Stage 1: Finetune LLM with text instruction data

• Download the text instruction data **Alpaca-52k** to ~/data/alpaca data.json and run the process script.

• Obtain **Llama2-7B Model** to ~/pretrained models/llama2-7b-hf . Then run the training script to perform text instruction tuning.

This step takes about 2 hours on 8 A100.

Stage 2: Align speech and text via behavior alignment of continuation writing

• Download the ASR dataset **GigaSpeech**, **LibriSpeech**, **Common Voice 2.0** to ~/data/gigaspeech , ~/data/librispeech , ~/data/common voice respectively. Then run the process scripts.

• Use the stage1 model to continue the transcription of ASR data. Here we take GigaSpeech dataset as example.

• (Optional) We recommend that you process the data offline and save to disk. The preprocessing process will filter out speech-text pairs that cannot be loaded by soundfile.

• Download **whisper-small** to ~/pretrained models/whisper-small and then run the training script.

The training process takes about 2.5 days on 8 A100.

License

The license of our project is [Apache License 2.0]()

Our models are based on Llama2 and Whisper. If you want to use our models, please do not violate the MIT License of whisper and the License of LLaMA-2