闲社服务运行正常AI智能体自动化平台
中文
语言 / Language简体中文SelectedEnglish

选择显示语言

Seed-OSS-36B-Instruct-Int8-W8A16

okwinds/Seed-OSS-36B-Instruct-Int8-W8A16

发布机构okwinds

下载访问条件魔搭来源

字节跳动 Seed-OSS-36B-Instruct-Int8-W8A16 量化高精校准 原模型 ByteDance-Seed/Seed-OSS-36B-Instruct 📖 关于量化损失方面的研究,可阅读公众号“觉察流”文章👇

文字对话
模型详情

模型介绍

中文整理

字节跳动 Seed-OSS-36B-Instruct-Int8-W8A16 量化高精校准

原模型 ByteDance-Seed/Seed-OSS-36B-Instruct

📖 关于量化损失方面的研究,可阅读公众号“觉察流”文章👇

《Reason Model 的“瘦身计划”:量化技术的得与失》

作者在此 👇🏻 扫一扫

下载

SDK下载

Git下载

模型概述

Seed-OSS-36B-Instruct-Int8-W8A16 是一个基于 okwinds/Seed-OSS-36B-Instruct 的 INT8 量化并校准的模型。

  • *模型名称:** Seed-OSS-36B-Instruct-Int8-W8A16
  • *模型架构:** SeedOssForCausalLM
  • *权重量化:** INT8

该模型通过将 ByteDance-Seed/Seed-OSS-36B-Instruct 的权重量化为 INT8 数据类型而实现。

量化过程将每个参数从 16bit 减少到 8bit,将模型占用磁盘空间大小,以及推理时加载模型需要的GPU显存空间,减少到了大约为原模型的1/2。

量化过程中,只有 transformer 中的 Linear 层权重是量化的,其他层均保持为原模型数据类型,采用混合精度的计算方式,尽可能减少模型量化后所带来的精度损失。

在量化过程中,使用了AutoGPTQ,采用 Symmetric group-wise 方式量化。并在此次量化过程中,进行了数据校准,以提升模型生成精度。(相对比 BF16 做到几乎无损)

注意:本模型需要 compute capability > 8.0(Ampere、Ada Lovelace、Hopper 架构)的 Nvidia GPU 来支持 INT8 **混合精度**计算。

部署推理

transformers 4.56.0.dev0(@seed-oss)

发本模型时,字节对 transformer 的 PR 还没有 merge,所以:

推荐使用 vLLM>=0.10.1.1

Openai api 兼容模式

附录 官方 Seed-OSS-36B-Instruct 简介

👋 Hi, everyone!

We are ByteDance Seed Team.

You can get to know us better through the following channels👇

Seed-OSS Open-Source Models

[!NOTE]

This model card is dedicated to the Seed-OSS-36B-Instruct model.

News

[2025/08/20]🔥We release Seed-OSS-36B-Base (both with and without synthetic data versions) and Seed-OSS-36B-Instruct .

Introduction

Seed-OSS is a series of open-source large language models developed by ByteDance's Seed Team, designed for powerful long-context, reasoning, agent and general capabilities, and versatile developer-friendly features. Although trained with only 12T tokens, Seed-OSS achieves excellent performance on several popular open benchmarks.

We release this series of models to the open-source community under the Apache-2.0 license.

[!NOTE]

Seed-OSS is primarily optimized for international (i18n) use cases.

Key Features

  • *Flexible Control of Thinking Budget**: Allowing users to flexibly adjust the reasoning length as needed. This capability of dynamically controlling the reasoning length enhances inference efficiency in practical application scenarios.
  • *Enhanced Reasoning Capability**: Specifically optimized for reasoning tasks while maintaining balanced and excellent general capabilities.
  • *Agentic Intelligence**: Performs exceptionally well in agentic tasks such as tool-using and issue resolving.
  • *Research-Friendly**: Given that the inclusion of synthetic instruction data in pre-training may affect the post-training research, we released pre-trained models both with and without instruction data, providing the research community with more diverse options.
  • *Native Long Context**: Trained with up-to-512K long context natively.

Model Summary

Seed-OSS adopts the popular causal language model architecture with RoPE, GQA attention, RMSNorm and SwiGLU activation.

:---: :---:

  • *Seed-OSS-36B**
  • *Parameters** 36B
  • *Attention** GQA
  • *Activation Function** SwiGLU
  • *Number of Layers** 64
  • *Number of QKV Heads** 80 / 8 / 8
  • *Head Size** 128
  • *Hidden Size** 5120
  • *Vocabulary Size** 155K
  • *Context Length** 512K
  • *RoPE Base Frequency** 1e7

Evaluation Results

Seed-OSS-36B-Base

Incorporating synthetic instruction data into pretraining leads to improved performance on most benchmarks. We adopt the version augmented with synthetic instruction data (i.e., *w/ syn.*) as Seed-OSS-36B-Base . We also release Seed-OSS-36B-Base-woSyn trained without such data (i.e., *w/o syn.*), offering the community a high-performance foundation model unaffected by synthetic instruction data.

Benchmark

Seed1.6-Base

Qwen3-30B-A3B-Base-2507*

Qwen2.5-32B-Base*

Seed-OSS-36B-Base(w/ syn.)

Seed-OSS-36B-Base-woSyn(w/o syn.)

Knowledge

MMLU-Pro

70

  • 8
  • 5 (55.1)
  • 1
  • 4

MMLU

  • 8
  • 7

84 (83.3)

  • 9
  • 8

TriviaQA

91

  • 2

76

  • 1
  • 9

GPQA-D

  • 4

37

  • 3
  • 7
  • 2

SimpleQA

  • 1
  • 2
  • 1
  • 8
  • 4

Reasoning

BBH

  • 1
  • 4
  • 1 (84.5)
  • 7
  • 2

AGIEval-en

78

  • 4
  • 6
  • 7
  • 1

Math

GSM8K

  • 1

87

  • 5 (92.9)
  • 8
  • 3

MATH

  • 9
  • 1
  • 5 (57.7)
  • 7
  • 3

Coding

MBPP

  • 6
  • 8
  • 8 (84.5)
  • 6
  • 6

HumanEval

78

  • 7
  • 6 (58.5)
  • 8
  • 6

Bold denotes open-source SOTA.

"*" indicates that the results in this column are presented in the format of "reproduced results (reported results if any)".

Seed-OSS-36B-Instruct

Benchmark

Seed1.6-Thinking-0715

OAI-OSS-20B*

Qwen3-30B-A3B-Thinking-2507*

Qwen3-32B*

Gemma3-27B

Seed-OSS-36B-Instruct

Knowledge

MMLU-Pro

  • 6
  • 2
  • 9 (80.9)
  • 8
  • 5
  • 7

MMLU

  • 6
  • 7 (85.3)
  • 9
  • 2
  • 9
  • 4

GPQA-D

  • 7
  • 2 (71.5)
  • 4 (73.4)
  • 7 (68.4)
  • 4
  • 4

SuperGPQA

  • 4
  • 1
  • 3 (56.8)
  • 3
  • 7

SimpleQA

  • 7
  • 7
  • 6
  • 6

10

  • 7

Math

AIME24

  • 3
  • 7 (92.1)
  • 7
  • 7 (81.4)
  • 7

AIME25

86

  • 3 (91.7)
  • 3 (85)
  • 3 (72.9)
  • 7

BeyondAIME

60

69

56

29

65

Reasoning

ArcAGI V2

  • 3
  • 7
  • 8
  • 4
  • 6

KORBench

  • 8
  • 3
  • 2
  • 4
  • 6

Coding

LiveCodeBench v6(02/2025-05/2025)

  • 8
  • 8
  • 3 (66)
  • 4
  • 4

HLE

  • 9
  • 7 (10.9)
  • 7
  • 9
  • 1

Instruction Following

IFEval

  • 3
  • 8

88 (88.9)

  • 4 (85)
  • 4
  • 8

Agent

TAU1-Retail

63

(54.8)

  • 7 (67.8)
  • 9
  • 4

TAU1-Airline

49

(38)

47 (48)

38

46

SWE-Bench Verified(OpenHands)

  • 8

(60.7)

31

  • 4

56

SWE-Bench Verified(AgentLess 4*10)

  • 4
  • 5
  • 7

47

Multi-SWE-Bench

  • 7
  • 5
  • 7

17

Multilingualism

MMMLU

  • 3
  • 4 (75.7)

79

79 (80.6)

  • 4

Long Context

RULER(128K)

  • 5
  • 7
  • 5
  • 5
  • 6

Safety

AIR-Bench

  • 6

Bold denotes open-source SOTA. Underlined indicates the second place in the open-source model.

"*" indicates that the results in this column are presented in the format of "reproduced results (reported results if any)". Some results have been omitted due to the failure of the evaluation run.

The results of Gemma3-27B are sourced directly from its technical report.

Generation configs for Seed-OSS-36B-Instruct: temperature=1.1, top p=0.95. Specifically, for Taubench, temperature=1, top p=0.7.

[!NOTE]

We recommend sampling with temperature=1.1 and top p=0.95 .

Thinking Budget

Users can flexibly specify the model's thinking budget. The figure below shows the performance curves across different tasks as the thinking budget varies. For simpler tasks (such as IFEval), the model's chain of thought (CoT) is shorter, and the score exhibits fluctuations as the thinking budget increases. For more challenging tasks (such as AIME and LiveCodeBench), the model's CoT is longer, and the score improves with an increase in the thinking budget.

Here is an example with a thinking budget set to 512: during the reasoning process, the model periodically triggers self-reflection to estimate the consumed and remaining budget, and delivers the final response once the budget is exhausted or the reasoning concludes.

If no thinking budget is set (default mode), Seed-OSS will initiate thinking with unlimited length. If a thinking budget is specified, users are advised to prioritize values that are integer multiples of 512 (e.g., 512, 1K, 2K, 4K, 8K, or 16K), as the model has been extensively trained on these intervals. Models are instructed to output a direct response when the thinking budget is 0, and we recommend setting any budget below 512 to this value.

Quick Start

Inference

Download Model

Download Seed-OSS checkpoint to ./Seed-OSS-36B-Instruct

Transformers

The generate.py script provides a simple interface for model inference with configurable options.

Basic Usage

Key Parameters

Parameter Description

  • ---------- -------------
  • -model path Path to the pretrained model directory (required)
  • -prompts Input prompts (default: sample cooking/code questions)
  • -max new tokens Maximum tokens to generate (default: 4096)
  • -attn implementation Attention mechanism: flash attention 2 (default) or eager
  • -load in 4bit/8bit Enable 4-bit/8-bit quantization (reduces memory usage)
  • -thinking budget Thinking budget in tokens (default: -1 for unlimited budget)

Quantization Examples

Custom Prompts

vLLM

Use vllm >= 0.10.0 or higher for inference.

First install vLLM with Seed-OSS support version:

Start vLLM API server:

Test with OpenAI client:

Chat

Tool Call

Model Card

See MODEL CARD.

License

This project is licensed under Apache-2.0. See the LICENSE flie for details.

Citation

About ByteDance Seed Team

Founded in 2023, ByteDance Seed Team is dedicated to crafting the industry's most advanced AI foundation models. The team aspires to become a world-class research team and make significant contributions to the advancement of science and society.

优先采用原作者提供的中文模型卡片;仅有英文说明时由闲社AI翻译整理。重要参数、许可证和商用范围请以原文为准。

你将获得什么

完整仓库方式包含该固定版本中的权重、配置、分词器及说明文件;指定版本方式只下载所选文件或整组分片。多模态模型的投影文件、配置等依赖,请按作者说明配齐。

完整仓库可能包含多个权重格式,因此需要的磁盘空间可能大于单一格式的模型大小。下载文件不等于完成模型部署。

使用前须知

运行环境

运行模型需要的设备、工具与依赖,以原作者说明为准。

授权范围

是否允许商用、修改和分发,请查看模型许可证。

闲社提供资料与镜像下载方法,不代表已运行模型或完成安全审计。下载的代码应先检查,再执行。

返回顶部