Taiyi-CLIP-RoBERTa-102M-ViT-L-Chinese

Taiyi-CLIP-RoBERTa-102M-ViT-L-Chinese

Github: Fengshenbang-LM

Docs: Fengshenbang-Docs

简介 Brief Introduction

首个开源的中文CLIP模型，1.23亿图文对上进行预训练的文本端RoBERTa-base

The first open source Chinese CLIP, pre-training on 123M image-text pairs, the text encoder: RoBERTa-base.

模型分类 Model Taxonomy

需求 Demand 任务 Task 系列 Series 模型 Model 参数 Parameter 额外 Extra

:----: :----: :----: :----: :----: :----:

特殊 Special 多模态 Multimodal 太乙 Taiyi CLIP (RoBERTa) 102M Chinese

模型信息 Model Information

我们遵循CLIP的实验设置，以获得强大的视觉-语言表征。在训练中文版的CLIP时，我们使用chinese-roberta-wwm作为语言的编码器，并将open clip中的**ViT-L-14**应用于视觉的编码器。为了快速且稳定地进行预训练，我们冻结了视觉编码器并且只微调语言编码器。此外，我们将Noah-Wukong数据集(100M)和Zero数据集(23M)用作预训练的数据集。在悟空数据集和zero数据集上预训练24轮,在A100x32上训练了6天。据我们所知，我们的Taiyi-CLIP是目前Huggingface社区中首个的开源中文CLIP。

We follow the experimental setup of CLIP to obtain powerful visual-language intelligence. To obtain the CLIP for Chinese, we employ chinese-roberta-wwm for the language encoder, and apply the **ViT-L-14** in open clip for the vision encoder. We freeze the vision encoder and tune the language encoder to speed up and stabilize the pre-training process. Moreover, we apply Noah-Wukong dataset (100M) and Zero dataset (23M) as the pre-training datasets. The model was first trained 24 epochs on wukong and zero, which takes 6 days to train on A100x32. To the best of our knowledge, our TaiyiCLIP is currently the only open-sourced Chinese CLIP in the huggingface community.

下游效果 Performance

Zero-Shot Classification**

model dataset Top1 Top5

• --- ---- ---- ----

Taiyi-CLIP-RoBERTa-102M-ViT-L-Chinese ImageNet1k-CN 55.04% 81.75%

Zero-Shot Text-to-Image Retrieval**

model dataset Top1 Top5 Top10

• --- ---- ---- ---- ----

Taiyi-CLIP-RoBERTa-102M-ViT-L-Chinese Flickr30k-CNA-test 58.32% 82.96% 89.40%

Taiyi-CLIP-RoBERTa-102M-ViT-L-Chinese COCO-CN-test 55.27% 81.10% 90.78%

Taiyi-CLIP-RoBERTa-102M-ViT-L-Chinese wukong50k 64.95% 91.77% 96.28%

示例代码

引用 Citation

如果您在您的工作中使用了我们的模型，可以引用我们的论文：

If you are using the resource for your work, please cite the our paper:

也可以引用我们的网站:

You can also cite our website: