GDU-liang-4B

GDU-Liang-4B

A 4B-parameter multimodal model with **identity-aware representation anchoring**,

designed for edge deployment in embodied AI and autonomous systems.

🤗 Model [📊 Technical Report (Coming Soon)]() [💬 Demo Space (Coming Soon)]()

🔍 Overview

GDU-Liang-4B is a four-modality (text, image, video, audio) incremental fine-tune of Molmo2-4B, featuring a novel **Identity Representation Anchoring** strategy that embeds self-cognition, technical context, and project-level semantics into the base model's latent space rather than relying on superficial prompt engineering or LoRA adapters.

This design preserves core identity and domain knowledge across downstream adaptation, quantization, and edge deployment, addressing the "identity collapse" problem in lightweight VLMs.

Why This Matters

Approach Identity Stability Quantization Robustness Edge Deployability

• --------- ------------------- ------------------------ -------------------

Prompt Engineering ❌ Fragile ❌ Lost after compression ⚠️ Context window limited

LoRA Fine-tuning ⚠️ Adapter-dependent ⚠️ Merge artifacts ✅ Lightweight

• *GDU-Liang-4B (Ours)** **✅ Anchored in weights** **✅ Preserved post-quant** **✅ Full-stack ready**

🏗️ Technical Highlights

• *Identity Representation Anchoring**: Self-cognition and domain priors are injected during a pre-training alignment phase, not as runtime prompts.
• *Four-Modality Incremental Training**: Text → Image → Video → Audio progressive curriculum with modality-specific loss balancing.
• *Edge-Oriented Design**: Targeting FPV drone onboard inference ( os.path.getmtime(dst):

shutil.copy2(src, dst)

synced.append(fname)

if synced:

print(f" 🔄 Synced {len(synced)} files → {target dir}")

for f in synced:

print(f" ✅ {f}")

mods to remove = [k for k in sys.modules if f"transformers modules.{repo name}" in k]

for k in mods to remove:

del sys.modules[k]

print(f" 🧹 Cleared {len(mods to remove)} cached modules")

else:

print(f" ✅ All .py files already synced in {target dir}")

==================== 模型类加载 ====================

def load model class(model dir):

"""🔑 v9.15: 跳过 base Model，优先选含 generate() 的 CausalLM 类"""

repo name = os.path.basename(os.path.normpath(model dir))

module path = f"transformers modules.{repo name}.modeling molmo2"

mod = importlib.import module(module path)

from transformers import PreTrainedModel

candidates = []

for attr name in dir(mod):

obj = getattr(mod, attr name)

if (isinstance(obj, type)

and issubclass(obj, PreTrainedModel)

and obj is not PreTrainedModel

and "config" not in attr name.lower()):

candidates.append((attr name, obj))

if not candidates:

raise RuntimeError(f"No model class in {module path}!")

print(f" 🔍 Available classes: {[c[0] for c in candidates]}")

def score(item):

n, cls = item

nl = n.lower()

has gen = hasattr(cls, "generate")

is base = n.endswith("Model") and "for" not in nl

if has gen and not is base:

return 0

if "forcausallm" in nl:

return 1

if is base:

return 99

return 50

candidates.sort(key=score)

chosen name, chosen cls = candidates[0]

print(f" ✅ Selected: {chosen name} (has generate={hasattr(chosen cls, 'generate')})")

return chosen cls

==================== Vocab 工具 ====================

def safe set vocab size(config, size):

for attr in ["vocab size", "text config.vocab size"]:

parts = attr.split(".")

obj = config

try:

for p in parts[:-1]:

obj = getattr(obj, p)

setattr(obj, parts[-1], size)

return

except AttributeError:

continue

if hasattr(config, "vocab size"):

config.vocab size = size

def expand vocab if needed(model, tokenizer, cfg):

tok vocab = len(tokenizer)

model vocab = cfg.vocab size

print(f" Tokenizer vocab: {tok vocab}, Model vocab: {model vocab}")

if tok vocab == model vocab:

print(f" ✅ Vocab matched: {tok vocab}")

return

# --- 1. Expand Embedding ---

embed param = embed name = None

for name, param in model.named parameters():

if "wte.embedding" in name.lower() and param.dim() == 2:

n lower = name.lower()

if not any(ex in n lower for ex in ["vision", "vit", "new embedding"]):

embed param, embed name = param, name

break

if embed param is None:

raise RuntimeError("Cannot find text embedding!")

cur rows, hidden = embed param.shape

diff = tok vocab - cur rows

print(f" 🎯 Expanding {embed name}: [{cur rows}x{hidden}] → +{diff}")

special ids = []

for tok str in tokenizer.get vocab():

tid = tokenizer.convert tokens to ids(tok str)

if tid is not None and tid 1 else model

delattr(parent, parts[-1])

setattr(parent, parts[-1], torch.nn.Parameter(new embed))

print(f" ✅ Expanded embedding to [{tok vocab}x{hidden}]")

# --- 2. Expand LM Head (多策略) ---

expanded = False

# 策略A: 按名称搜索

for pname, param in list(model.named parameters()):

pl = pname.lower()

if param.dim() == 2 and param.shape[1] == hidden:

if any(k in pl for k in ["lm head", "output head"]) and \

not any(ex in pl for ex in ["vision", "vit", "visual"]):

if param.shape[0] 1 else model

delattr(hpar, hp[-1])

setattr(hpar, hp[-1], torch.nn.Parameter(nh))

expanded = True

break

# 策略B: fallback 路径

if not expanded:

for path in ["lm head", "transformer.lm head", "model.lm head", "head"]:

try:

m = model.get submodule(path)

if hasattr(m, "weight") and m.weight.dim() == 2 and m.weight.shape[1] == hidden:

if m.weight.shape[0] 0:

print(f" ⚠️ Still {meta count} meta params after first pass, retrying...")

model.to("cpu")

expand vocab if needed(model, tokenizer, cfg)

if tokenizer.eos token id is None:

tokenizer.eos token id = tokenizer.pad token id or 0

frames, timestamps = load and sample video(video path, max frames=8)

# 🔑 v9.20: Processor 加载

from transformers import AutoProcessor

try:

processor = AutoProcessor.from pretrained(model dir, trust remote code=True)

print(f" ✅ Loaded processor: {type(processor). name }")

except Exception as e:

repo name = os.path.basename(os.path.normpath(model dir))

proc module = importlib.import module(f"transformers modules.{repo name}.processing molmo2")

for attr name in dir(proc module):

obj = getattr(proc module, attr name)

if isinstance(obj, type) and "processor" in attr name.lower() and attr name != "AutoProcessor":

processor = obj.from pretrained(model dir, trust remote code=True)

print(f" ✅ Loaded custom processor: {attr name}")

break

else:

raise RuntimeError(f"Cannot load processor! AutoProcessor failed: {e}")

# 🔑 v9.26: Molmo2-GDU-liang-4B 确认使用 (token id: 151945)

video placeholder = ""

print(f" 🎬 Using video placeholder: '{video placeholder}'")

messages = [{"role": "user", "content": f"{video placeholder}\n{user query}"}]

prompt text = tokenizer.apply chat template(messages, tokenize=False, add generation prompt=True)

# print(f" 💬 Prompt preview: {prompt text[:200]}...")

# 🔑 v9.22: Molmo2VideoProcessor 期望 (T, H, W, C)，不要 permute

video tensor = frames # 已经是 (T, H, W, C)

print(f" 📐 Video tensor shape: {video tensor.shape}") # 应为 (8, H, W, 3)

# 🔑 v9.21: VideoMetadata 需要 fps + total num frames

proc inputs = None

fps value = 30.0

num frames = video tensor.shape[0] # 8 frames

# 方式1: video metadata 带完整字段

try:

proc inputs = processor(

text=prompt text,

videos=video tensor,

video metadata=[{"fps": fps value, "total num frames": num frames}],

return tensors="pt",

)

print(f" ✅ Processor accepted video metadata (dict)")

except Exception as e1:

print(f" ⚠️ video metadata dict failed: {e1}")

# 方式1b: 用 processor 内部的 VideoMetadata 类

try:

from transformers.video processing utils import VideoMetadata as VM

proc inputs = processor(

text=prompt text,

videos=video tensor,

video metadata=[VM(fps=fps value, total num frames=num frames)],

return tensors="pt",

)

print(f" ✅ Processor accepted VideoMetadata class")

except Exception as e1b:

print(f" ⚠️ VideoMetadata class failed: {e1b}")

# 方式2: fps 关键字参数

try:

proc inputs = processor(

text=prompt text,

videos=video tensor,

fps=fps value,

return tensors="pt",

)

print(f" ✅ Processor accepted fps kwarg")

except Exception as e2:

raise RuntimeError(

f"All processor call methods failed!\n"

f" dict metadata: {e1}\n"

f" class metadata: {e1b}\n"

f" fps kwarg: {e2}"

)

# 🔑 v9.19: device map="auto" 时 model.device 不可靠，用首个参数的设备

target device = next(model.parameters()).device

inp = {k: v.to(target device) if isinstance(v, torch.Tensor) else v for k, v in proc inputs.items()}

print(f" 🚀 Generating with keys: {list(inp.keys())}, device: {target device}")

outputs = model.generate(

• *inp, max new tokens=max new tokens, do sample=False,

eos token id=tokenizer.eos token id,

pad token id=tokenizer.pad token id or tokenizer.eos token id,

)

new tok = outputs[0][inp["input ids"].shape[1]:]

return tokenizer.decode(new tok, skip special tokens=True).strip()

==================== 入口 ====================

print("=" * 60)

print("【🚀 v9.15 - CausalLM Priority + Tied Weights Fallback】")

print("=" * 60)

result = generate video response(

"/mnt/workspace/vido data/微信视频2026-08-14 144752 279.mp4",

"请详细描述视频中先后发生的动作与变化，注意时间顺序。",

)

print("\n" + "=" * 60)

print("📝 RESPONSE:")

print("=" * 60)

print(result)

python

import torch, sys, importlib.util

import numpy as np

from PIL import Image

from decord2 import VideoReader

from modelscope import AutoConfig, AutoProcessor

===================== 1. 加载模型 =====================

MODEL PATH = "./GDU-liang-4B"

动态导入 Molmo2 自定义建模文件

spec = importlib.util.spec from file location(

"molmo2 model",

f"{MODEL PATH}/modeling molmo2.py",

submodule search locations=[MODEL PATH],

)

molmo2 module = importlib.util.module from spec(spec)

sys.modules["molmo2 model"] = molmo2 module

spec.loader.exec module(molmo2 module)

cfg = AutoConfig.from pretrained(MODEL PATH, trust remote code=True)

model = molmo2 module.Molmo2ForConditionalGeneration.from pretrained(

MODEL PATH, config=cfg, trust remote code=True,

dtype=torch.bfloat16, device map="auto",

)

processor = AutoProcessor.from pretrained(MODEL PATH, trust remote code=True, use fast=False)

model.eval()

===================== 2. 词表对齐（安全补丁）=====================

def align vocab(tokenizer, model):

"""确保 embedding / lm head 与 tokenizer 词表大小一致"""

vocab size = len(tokenizer)

emb = model.get input embeddings()

embed weight = emb.weight if hasattr(emb, "weight") else emb.embed tokens.weight

current size = embed weight.shape[0]

if vocab size > current size:

delta = vocab size - current size

mean vec = embed weight.data.mean(0, keepdim=True).expand(delta, -1)

new embed = torch.zeros(vocab size, embed weight.shape[1],

dtype=embed weight.dtype, device=embed weight.device)

new embed[:current size] = embed weight.data

new embed[current size:] = mean vec

embed weight.data = new embed

# lm head 同步扩展

if model.lm head.weight.shape[0] == current size:

head weight = model.lm head.weight.data

new head = torch.zeros(vocab size, head weight.shape[1],

dtype=head weight.dtype, device=head weight.device)

new head[:current size] = head weight

new head[current size:] = head weight.mean(0, keepdim=True).expand(delta, -1)

model.lm head.weight.data = new head

align vocab(processor.tokenizer, model)

===================== 3. 视频抽帧（2fps, 最多8帧, 448px）=====================

def extract frames(video path: str, target fps: float = 2.0, max frames: int = 8, resolution: int = 448):

vr = VideoReader(video path)

avg fps = vr.get avg fps()

step = max(1, int(round(avg fps / target fps)))

indices = list(range(0, len(vr), step))[:max frames]

frames = [

Image.fromarray(vr[i].asnumpy()).convert("RGB").resize((resolution, resolution))

for i in indices

]

return frames, indices, avg fps

===================== 4. 构建带时序锚点的视频 token 序列 =====================

def build video tokens(frames, indices, fps, processor):

"""

核心时序机制：每帧 image patch tokens 前插入时间戳文本锚点

让模型感知帧间先后顺序 → 实现动作/变化理解

"""

vis = processor.image processor(images=frames, return tensors="np")

grids = np.asarray(vis["image grids"]) # (F, 4) int32

all tokens = []

for fi in range(len(frames)):

t sec = indices[fi] / fps

# ⏱️ 时间戳锚点：告诉模型这是第几帧、对应什么时刻

anchor text = f"\n*帧{fi + 1} 时间={t sec:.2f}s*\n"

anchor ids = processor.tokenizer.encode(anchor text, add special tokens=False)

# 🖼️ 该帧的视觉 patch tokens

image token strs = processor.get image tokens(grids[fi]).tolist()

image ids = processor.tokenizer.convert tokens to ids(image token strs)

all tokens.extend(anchor ids + image ids)

return all tokens, grids

===================== 5. 推理函数 =====================

@torch.inference mode()

def generate video response(video path: str, user query: str, system prompt: str = "") -> str:

# 抽帧

frames, indices, fps = extract frames(video path)

# 构建带时序的视频tokens

video tokens, grids = build video tokens(frames, indices, fps, processor)

# 用 apply chat template 统一管理对话格式（不再手搓特殊token）

messages = [{"role": "user", "content": user query}]

template kwargs = {"add generation prompt": True, "system": system prompt}

prompt text = processor.tokenizer.apply chat template(messages, tokenize=False, **template kwargs)

# 将视频tokens插入到 user message 的开头位置

# （apply chat template 生成的文本中，user内容之前即为插入点）

prefix ids = processor.tokenizer.encode(prompt text, add special tokens=False)

full ids = prefix ids + video tokens

inp = {

"input ids": torch.tensor([full ids], device=model.device),

"attention mask": torch.ones(1, len(full ids), dtype=torch.long, device=model.device),

"image grids": torch.from numpy(grids).to(torch.long).to(model.device), # ✅ 保持int类型

}

torch.cuda.empty cache()

outputs = model.generate(

• *inp,

max new tokens=384,

do sample=False, # 视频理解建议greedy，减少幻觉

)

new tokens = outputs[0][len(full ids):]

return processor.tokenizer.decode(new tokens, skip special tokens=True).strip()

===================== 6. 测试 =====================

if name == " main ":

print("【视频时序理解测试】")

result = generate video response(

video path="./test.mp4",

user query="请描述视频中先后发生的动作与变化。",

system prompt="", # ← 无身份注入，纯净推理

)

print(result)

🎯 Roadmap: From Multimodal to Embodied Intelligence

该项目为完整具身智能闭环的基石，重点方向：

边缘量化与硬件适配（INT4/INT8）

低延迟在线推理（目标延迟 < 50ms）

硬件在环（HIL）验证与实机飞行测试

从感知到动作（VLA Head → 飞控命令）端到端闭环

👤 作者与背景

作者为广东外语外贸大学人工智能方向学生梁子羿和AlphapilotOS组员，具备量化交易与轻量化部署实战经验（2026 年 4 月，华林证券 & 赛富溪栈 AI 股票交易挑战赛季军），目前专注于 4B 级别多模态模型的身份表征固化与端侧工程化。

🌍 开源与伦理

开源策略：提供模型权重与训练/推理流水线；坚持底层改造与工程化落地，不追逐热点式 Demo。

使用限制：请遵守当地法规与无人机飞行安全规定；该模型未经完整实战验证，不适用于未经额外安全验证的关键控制系统。

📋 Model Card 详?

Field Value

• ------ -------

Base Model allenai/Molmo2-4B

Parameters ~4B

Modalities Text, Image, Video, Audio

Precision float16 (quantizable to INT4/INT8)

Context Length 4096 tokens

License Apache 2.0

Framework Transformers + Custom Code

📚 Citation

⚠️ Limitations & Ethics

训练资源有限，长尾领域表现可能存在波动；需要额外数据与任务特化训练以提升鲁棒性。

身份表征固化可降低遗忘与量化后退化，但不能完全消除幻觉或误判。

在将模型用于物理控制前，必须完成严格的仿真与实机安全验证流程。

📂 目录建议

examples/ ：图像、视频、音频推理示例（建议包含 Colab 与本地脚本）

scripts/ ：量化、转换、硬件适配脚本

docs/ ：技术报告、模型卡扩展、评测结果存档

🛠️ 实验与部署小贴士

量化前先进行校准集评估，使用代表性多模态样本以验证身份保留效果。

在 Jetson/NVIDIA 边缘设备上优先测试 INT8，再评估 INT4 带来的性能/精度权衡。

结合硬件在环（HIL）与仿真闭环测试，逐步放宽实时控制权限。

GDU-Liang-4B 多模态模型（中文完整版）

模型简介

GDU-Liang-4B** 基于 Molmo2 架构，完成文本、图像、视频、音频四模态的增量微调与身份表征固化。不同于常见的 Prompt/LoRA 套壳，项目通过底层改造把模型自我认知、技术背景与项目语境深度写入基座语义空间，从而在量化与端侧部署后仍保留稳定的身份语义。

技术履历与动机

量化交易实战背景（2026 年 4 月，华林证券 & 赛富溪栈挑战赛季军），具备时序建模与轻量化落地经验。

在有限算力下实现 4B 四模态的增量微调，并设计身份表征固化方案以提升量化后的稳定性与实际部署可靠性。

开源愿景与工程观

坚持“底层改造、优雅工程、全链路所有权、物理世界验证”的工程哲学，目标不是一时的 Demo，而是可工业化、可验证的端侧具身智能系统。

使用方法（简要）

• 安装依赖： pip install -r requirements.txt
• 加载模型与 tokenizer（参见上文 Python 示例）
• 使用 examples/ 中的脚本完成多模态输入的推理验证

最终愿景

以广东外语外贸大学学生及Alphapilot OS团队成员的身份，构建国产轻量化具身智能模型，打通从算法到设备的全链路，形成可工业化的解决方案。