# 一、模型介绍阶跃星辰（StepFun）于 2026 年 1 月 20 日正式宣布开源其最新的多模态大模型—— **Step3-VL-10B** 。这款模型在 AI 圈引起了不小的轰动，因为它以仅 **10B（百亿）** 的参数量，在多项核心基准测试中达到了同类模型的 SOTA（当前最佳）水平，甚至在部分能力上超越了比它大 10 到 20 倍的千亿级模型。 ## 🚀 核心亮点：小体量，大智慧 Step3-VL-10B 的设计初衷是打破“模型越大越强”的思维定式，通过极高的训练效率和架构优化，实现了“以小博大”。 * **对标千亿级模型：** 尽管只有 10B 参数，其性能已接近或超越 **GLM-4.6V (106B)** 和 **Qwen3-VL (235B)** 等超大规模模型，并能与 **Gemini 2.5 Pro** 等顶级闭源模型一较高下。 * **深层视觉推理：** 尤其擅长 STEM 领域（数学、物理、几何）的复杂推理，在 AIME 2025 等高难度竞赛题上表现卓越。 * **GUI 代理能力：** 经过大量 GUI 数据预训练，能够精准识别和操作复杂图形界面，是构建下一代 AI Agent（智能体）的理想核心。 ## 🛠 技术架构与创新 Step3-VL-10B 的成功归功于几个关键的战略性技术突破： 1. **全参数“解冻”预训练：** 与许多固定视觉编码器以节省成本的模型不同，Step3-VL-10B 在 **1.2T（1.2 兆）** 多模态代币的预训练阶段，对视觉编码器和语言解码器进行了 **全量参数训练** 。这种“深度融合”让视觉和语言的协同达到了原生级别的默契。 2. **PaCoRe（并行协调推理）：** 引入了 **Parallel Coordinated Reasoning** 技术。这是一种在推理阶段（Inference-time）扩展计算能力的手段，允许模型通过探索和合成多种视觉假设，在面对复杂图片时展现出更严密的逻辑。 3. **精密的架构组合：** * **视觉编码器：** 1.8B 参数的 PE-lang（语言优化型编码器），专为语言理解对齐设计。 * **语言解码器：** 强大的 **Qwen3-8B** 。 * **多尺度策略：** 采用 728x728 全局图结合多个 504x504 局部裁切，确保能够捕捉到图像中的微小细节（如 OCR 字符和微小图标）。 ![performance.png](https://foruda.gitee.com/images/1769479514952564360/7ebdd057_16012914.png) 更多详情请见：[Step3-VL-10B · 模型库](https://www.modelscope.cn/models/stepfun-ai/Step3-VL-10B) # 二、部署流程基础环境推荐： | 环境名称 | 版本信息 | | :----------------: | :---------: | | Ubuntu | 24.04.3 LTS | | Cuda | V12.6 | | Python | 3.12 | | PyTorch | 2.7.1 | | NVIDIA Corporation | RTX 4090 | 注：最低显存要求：24 GB（例如 RTX 4090 或 A100），若上传图片体量较大，则需要更高显存的显卡。 ## 1.更新基础软件包查看系统版本信息 ``` #查看系统的版本信息，包括 ID（如 ubuntu、centos 等）、版本号、名称、版本号 ID 等 cat /etc/os-release ``` ![image.png](https://foruda.gitee.com/images/1764640949266023722/25ea5582_16012914.png) 更新软件包列表 ``` #更新软件列表 apt-get update ``` ![image.png](https://foruda.gitee.com/images/1764641016796158266/7dc8db7f_16012914.png) ## 2.创建虚拟环境为方便后续 vllm 的推理使用，这里利用 `uv venv hunyuanocr` 创建一个名为 hunyuanocr 的虚拟环境。 ``` #创建一个名为Step3-VL-10B的虚拟环境 conda create -n Step3-VL-10B python=3.12 ``` ![image.png](https://foruda.gitee.com/images/1769483878451344004/90be12dc_16012914.png) 激活虚拟环境 ``` conda activate Step3-VL-10B ``` ![image.png](https://foruda.gitee.com/images/1769483940334327030/de07d31e_16012914.png) ## 3.克隆仓库、安装依赖如需要该模型可视化访问页面，这里推荐 huggingface 上官方给出的 gradio 页面模板 ``` git clone https://huggingface.co/spaces/stepfun-ai/Step3-VL-10B ``` ![image.png](https://foruda.gitee.com/images/1769484018326489275/6cc478af_16012914.png) 特别的，若需要模型具备流式输出及更好显存占用优化效果，可选用如下的 `app.py` 代码： ``` #!/usr/bin/env python3 """ Step3-VL-10B Gradio Interface (Fixed for RTX 4090 & Logic Errors) With Streaming Output Support! """ import os import sys import time import re import io import base64 import gc import torch from typing import Optional, List, Generator from threading import Thread from PIL import Image import gradio as gr import spaces from transformers import AutoProcessor, AutoModelForCausalLM, TextIteratorStreamer # ================= 配置与路径设置 ================= def set_pythonpath(path: str, allow_missing: bool = False): """Prepend path to PYTHONPATH and sys.path.""" if not path: return if not allow_missing and not os.path.isdir(path): return if path not in sys.path: sys.path.insert(0, path) os.environ["PYTHONPATH"] = ( path + os.pathsep + os.environ.get("PYTHONPATH", "") ) def resolve_remote_snapshot_path(model_name: str, revision: Optional[str]): """Resolve HF cache snapshot path for remote model.""" if not model_name or not revision: return None hub_cache = os.environ.get("HUGGINGFACE_HUB_CACHE") if not hub_cache: return None model_dir = f"models--{model_name.replace('/', '--')}" return os.path.join(hub_cache, model_dir, "snapshots", revision) def ensure_model_repo_on_pythonpath(model_name: str, revision: Optional[str] = None): """Add local/remote model repo path to PYTHONPATH and sys.path.""" if not model_name: return if os.path.isdir(model_name): set_pythonpath(model_name) return snapshot_path = resolve_remote_snapshot_path(model_name, revision) if snapshot_path: set_pythonpath(snapshot_path, allow_missing=True) MODEL_NAME = os.getenv("MODEL_NAME", "/model") #这里根据你实际的模型目录进行修改 ensure_model_repo_on_pythonpath(MODEL_NAME) _MODEL = None _PROCESSOR = None _LOADED_MODEL_NAME = None # ================= 模型加载 ================= @spaces.GPU def get_model(model_name: Optional[str] = None): """Load and cache the Step3-VL-10B model and processor.""" global _MODEL, _PROCESSOR, _LOADED_MODEL_NAME if model_name is None: model_name = MODEL_NAME if _MODEL is not None and _PROCESSOR is not None and _LOADED_MODEL_NAME == model_name: return _MODEL, _PROCESSOR key_mapping = { "^vision_model": "model.vision_model", r"^model(?!\.(language_model|vision_model))": "model.language_model", "vit_large_projector": "model.vit_large_projector", } print(f"Loading model: {model_name}") revision = None if os.path.isdir(model_name): ensure_model_repo_on_pythonpath(model_name, revision) _PROCESSOR = AutoProcessor.from_pretrained( model_name, trust_remote_code=True, fix_mistral_regex=True, revision=revision ) _MODEL = AutoModelForCausalLM.from_pretrained( model_name, trust_remote_code=True, device_map="auto", torch_dtype="auto", # load_in_8bit=True, #如果显存紧张可取消注释，采用8b的量化版 key_mapping=key_mapping, revision=revision ).eval() _LOADED_MODEL_NAME = model_name return _MODEL, _PROCESSOR @spaces.GPU def warmup_model(model_name: Optional[str] = None): """Warmup the model once at app startup.""" get_model(model_name) return True # ================= 工具函数 ================= def resolve_generation_tokens(processor): tokenizer = getattr(processor, "tokenizer", processor) eos_ids = [] eos_token_id = getattr(tokenizer, "eos_token_id", None) if eos_token_id is not None: if isinstance(eos_token_id, (list, tuple)): eos_ids.extend(list(eos_token_id)) else: eos_ids.append(eos_token_id) def _try_add_token(token_str: str): try: token_id = tokenizer.convert_tokens_to_ids(token_str) unk_id = getattr(tokenizer, "unk_token_id", None) if token_id is not None and token_id != unk_id: eos_ids.append(token_id) except Exception: return for token_str in ["<|eot_id|>", "<|end_of_turn|>", "<|end_of_text|>", ""]: _try_add_token(token_str) seen = set() eos_ids = [i for i in eos_ids if not (i in seen or seen.add(i))] pad_token_id = getattr(tokenizer, "pad_token_id", None) if pad_token_id is None and eos_ids: pad_token_id = eos_ids[0] stop_strings = ["\nassistant", "\nAssistant", "<|eot_id|>", "<|end_of_turn|>", "<|end_of_text|>", ""] return eos_ids, pad_token_id, stop_strings def truncate_on_stop_strings(text: str, stop_strings): if not text: return text earliest = None for s in stop_strings: idx = text.find(s) if idx != -1 and (earliest is None or idx < earliest): earliest = idx return text if earliest is None else text[:earliest].rstrip() def escape_html(text): if not isinstance(text, str): return text return text.replace("&", "&").replace("<", "<").replace(">", ">").replace('"', """).replace("'", "'") def load_image(image_path: Optional[str]) -> Optional[Image.Image]: if not image_path or not os.path.exists(image_path): return None try: image = Image.open(image_path) image.load() if image.mode != "RGB": image = image.convert("RGB") return image except Exception as e: print(f"[ERROR] Failed to load image: {e}") return None def image_to_data_url(image: Image.Image) -> Optional[str]: if image is None: return None try: buffer = io.BytesIO() image.save(buffer, format="PNG") encoded = base64.b64encode(buffer.getvalue()).decode("ascii") return f"data:image/png;base64,{encoded}" except Exception: return None def _strip_data_urls(text: str) -> str: if not text: return text return re.sub(r"data:image\/[a-zA-Z]+;base64,[A-Za-z0-9+/=\s]+", "", text) def _strip_html(text: str) -> str: if not text: return text return re.sub(r"<[^>]+>", "", text) def _extract_text_content(content) -> Optional[str]: if content is None: return None if isinstance(content, str): cleaned = _strip_data_urls(content) cleaned = _strip_html(cleaned) return cleaned.strip() if cleaned.strip() else None if isinstance(content, list): parts = [] for item in content: if isinstance(item, dict) and item.get("type") == "text": parts.append(item.get("text", "")) text = "\n".join(p for p in parts if p) return text.strip() if text.strip() else None return None # ================= 核心逻辑修复区 ================= def truncate_conversation_history(messages: List[dict], max_rounds: int = 4, max_chars: int = 8000) -> List[dict]: """ Truncate conversation history to prevent excessive context buildup that leads to OOM. """ if not messages: return [] # 限制最多保留 N 轮对话（每轮包含 user + assistant） truncated_messages = [] current_chars = 0 rounds_count = 0 # 从后往前遍历，保留最新的几轮 for i in range(len(messages) - 1, -1, -1): msg = messages[i] content = msg.get('content', []) # 计算当前消息的大致字符数 if isinstance(content, str): char_count = len(content) elif isinstance(content, list): char_count = sum(len(c.get('text', '')) for c in content if isinstance(c, dict) and c.get('type') == 'text') # 图片大概按每个1000字符计算 image_count = sum(1 for c in content if isinstance(c, dict) and c.get('type') == 'image') char_count += image_count * 1000 else: char_count = 0 # 如果已经超过最大轮数或字符数限制，则停止 if rounds_count >= max_rounds or current_chars + char_count > max_chars: break truncated_messages.insert(0, msg) current_chars += char_count # 只有user消息才算作一轮 if msg.get('role') == 'user': rounds_count += 1 return truncated_messages def format_messages(history, user_text, image: Optional[Image.Image] = None): """Format message list for Step3-VL-10B (Fixed Logic + History Limitation).""" messages: List[dict] = [] if not history: history = [] # 限制对话历史长度，避免显存溢出 truncated_history = truncate_conversation_history(history, max_rounds=4, max_chars=8000) for item in truncated_history: role = item.get("role") if isinstance(item, dict) else getattr(item, "role", None) content = item.get("content") if isinstance(item, dict) else getattr(item, "content", None) raw_text = item.get("raw_text") if isinstance(item, dict) else None if not role or content is None: continue if role == "assistant" and isinstance(content, str) and '

' in content: pattern = r'

.*?

\s*

\s*' remaining_content = re.sub(pattern, '', content, flags=re.DOTALL).strip() if remaining_content and not remaining_content.startswith('<'): content = remaining_content if role == "user" and raw_text is not None: text_content = raw_text else: text_content = _extract_text_content(content) if role == "user" and (text_content is None or not text_content.strip()): text_content = "[Image Upload]" if isinstance(text_content, str) and text_content.strip(): messages.append({ "role": role, "content": [{"type": "text", "text": text_content}] }) content_list = [] if image is not None: content_list.append({"type": "image", "image": image}) if user_text: content_list.append({"type": "text", "text": user_text}) elif image is not None: content_list.append({"type": "text", "text": " "}) if content_list: messages.append({"role": "user", "content": content_list}) return messages def build_user_display(image_url: Optional[str], user_text: Optional[str]) -> str: parts = [] if image_url: parts.append( f'

' f' uploaded image

' f'

' ) if user_text: parts.append(user_text) return "\n\n".join(parts).strip() @spaces.GPU def chat(user_text, image_file, history, max_tokens, temperature, top_p, show_thinking=True, model_name=None): if model_name is None: model_name = MODEL_NAME gc.collect() torch.cuda.empty_cache() if not user_text and not image_file: yield history or [], "", None return history = history or [] clean_history = [] for item in history: if isinstance(item, dict) and 'role' in item and 'content' in item: clean_history.append(item) history = clean_history image = load_image(image_file) image_url = image_to_data_url(image) if image is not None else None try: messages = format_messages(history, user_text, image) except Exception as e: history.append({"role": "assistant", "content": f"❌ Formatting Error: {str(e)}"}) yield history, "", None return user_display = build_user_display(image_url, user_text) history.append({ "role": "user", "content": user_display, "raw_text": user_text or "" }) if show_thinking: history.append({ "role": "assistant", "content": ( '

\n' '

💭 Thinking...

\n' '

Processing...

\n' '

' ) }) else: history.append({"role": "assistant", "content": "⏳ Generating..."}) yield history, "", None start_time = time.time() inputs = None try: model, processor = get_model(model_name) inputs = processor.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt" ).to(model.device) gen_kwargs = { "max_new_tokens": max_tokens, "do_sample": temperature > 0 } if temperature > 0: gen_kwargs["temperature"] = temperature gen_kwargs["top_p"] = top_p eos_ids, pad_token_id, stop_strings = resolve_generation_tokens(processor) if eos_ids: gen_kwargs["eos_token_id"] = eos_ids if len(eos_ids) > 1 else eos_ids[0] if pad_token_id is not None: gen_kwargs["pad_token_id"] = pad_token_id streamer = TextIteratorStreamer(processor.tokenizer, skip_prompt=True, skip_special_tokens=True) generation_kwargs = dict(inputs, streamer=streamer, **gen_kwargs) thread = Thread(target=model.generate, kwargs=generation_kwargs) thread.start() full_response = "" for new_text in streamer: full_response += new_text if show_thinking: escaped_response = escape_html(full_response) history[-1]["content"] = ( '

\n' '

💭 Thinking...

\n' f'

{escaped_response}

\n' '

' ) else: history[-1]["content"] = full_response yield history, "", None # Final replacement after complete generation if "" in full_response: think_part, response_part = full_response.split("", 1) if think_part.startswith(""): think_part = think_part[len("think>"):] if show_thinking: escaped_think = escape_html(think_part.strip()) final_content = ( f'

\n' f'

💭 Thinking Process

\n' f'

{escaped_think}

\n' f'

\n\n' f'{response_part.strip()}' ) history[-1]["content"] = final_content else: history[-1]["content"] = response_part.strip() else: history[-1]["content"] = full_response.strip() elapsed_time = time.time() - start_time print(f"[MODEL] ✅ SUCCESS - Time: {elapsed_time:.2f}s") yield history, "", None except RuntimeError as e: error_msg = str(e) print(f"[MODEL] ❌ RuntimeError: {error_msg}") if "out of memory" in error_msg.lower(): detailed_error = ( "❌ GPU Out of Memory:\n\n" "- This usually happens when too many images or long conversations accumulate.\n" "- Try clearing chat history using 'Clear'.\n" "- Or reduce image resolution before uploading.\n" "- Also consider restarting the application if issue persists." ) history[-1]["content"] = detailed_error else: history[-1]["content"] = f"❌ Model Runtime Error:\n\n```\n{error_msg}\n```" yield history, "", None except Exception as e: import traceback print(f"[MODEL] ❌ Error: {traceback.format_exc()}") history[-1]["content"] = f"❌ Error: {str(e)}" yield history, "", None finally: if inputs is not None: del inputs if 'generate_ids' in locals(): del generate_ids gc.collect() torch.cuda.empty_cache() # ================= CSS 样式 ================= custom_css = """ .gradio-container { max-width: 100% !important; font-family: -apple-system, sans-serif; } .app-header { text-align: center; padding: 2.5rem; background: linear-gradient(135deg, #667eea 0%, #764ba2 100%); border-radius: 16px; margin-bottom: 1.5rem; color: white; } .thinking-block { background: #f5f7fa; border-left: 4px solid #667eea; padding: 12px; margin: 10px 0; border-radius: 6px; } .thinking-header { font-weight: bold; color: #667eea; margin-bottom: 5px; } .thinking-content { font-family: monospace; font-size: 0.9em; white-space: pre-wrap; color: #444; } .uploaded-image img { max-width: 100%; border-radius: 8px; margin-bottom: 8px; } .dark .thinking-block { background: #2d3748; border-color: #667eea; } .dark .thinking-content { color: #ccc; } """ # ================= UI 构建 ================= with gr.Blocks(title="Step3-VL-10B") as demo: model_ready = gr.State(False) gr.HTML(""" """) with gr.Row(): with gr.Column(scale=1, min_width=350): with gr.Accordion("⚙️ Configuration", open=False): max_tokens = gr.Slider(1, 8192, value=2048, label="Max Tokens") temperature = gr.Slider(0.0, 2.0, value=0.6, label="Temperature") top_p = gr.Slider(0.0, 1.0, value=0.9, label="Top P") show_thinking = gr.Checkbox(label="Show Thinking", value=True) user_text = gr.Textbox(label="Message", lines=4, placeholder="Type here...") image_file = gr.Image(label="Upload Image", type="filepath") with gr.Row(): clear_btn = gr.Button("🗑️ Clear") submit_btn = gr.Button("🚀 Send", variant="primary") with gr.Column(scale=2): chatbot = gr.Chatbot(label="Conversation", height=700) submit_btn.click( fn=chat, inputs=[user_text, image_file, chatbot, max_tokens, temperature, top_p, show_thinking], outputs=[chatbot, user_text, image_file] ) clear_btn.click(lambda: ([], "", None), outputs=[chatbot, user_text, image_file]) demo.load(fn=warmup_model, inputs=None, outputs=model_ready) if __name__ == "__main__": import argparse parser = argparse.ArgumentParser() parser.add_argument("--host", default="0.0.0.0") parser.add_argument("--port", type=int, default=8080) parser.add_argument("--model", default=MODEL_NAME) parser.add_argument("--share", action="store_true") args = parser.parse_args() if args.model: MODEL_NAME = args.model print(f"Starting Gradio on port {args.port}...") demo.launch( server_name=args.host, server_port=args.port, share=args.share, theme=gr.themes.Soft(), css=custom_css ) ``` 同样的，也需要进入 `Step3-VL-10B` 目录下，安装所需依赖项 ``` cd Step3-VL-10B pip install -r requirements.txt ``` ![image.png](https://foruda.gitee.com/images/1769484105174409124/1c35079d_16012914.png) ## 4.模型下载这里推荐转到魔塔社区官网下载模型文件：[Step3-VL-10B · 模型库](https://www.modelscope.cn/models/stepfun-ai/Step3-VL-10B) ![image.png](https://foruda.gitee.com/images/1769479985919421061/ef924577_16012914.png) 使用命令行下载完整模型库 ``` #在下载前，请先通过如下命令安装 pip install modelscope ``` ![image.png](https://foruda.gitee.com/images/1769484210971724171/6e9c6c6b_16012914.png) 转到根目录下，创建 `model` 目录用于存放模型权重文件（可按照自己的目标存储路径进行修改），在使用命令行下载 `modelscope download --model stepfun-ai/Step3-VL-10B --local_dir ./` ``` cd / mkdir model cd model modelscope download --model stepfun-ai/Step3-VL-10B --local_dir ./ ``` ![image.png](https://foruda.gitee.com/images/1769484464118952447/4994238d_16012914.png) ## 5.运行 gradio 脚本，实现模型交互进入 `/HunyuanOCR ` 目录，修改其中的 web 启动代码 app.py： ``` vim /Step3-VL-10B/app.py ``` 将模型的加载路径改为本地路径 `/model/ ,` 以及 lunch 加载函数中设置 `server_name='0.0.0.0',server_port=8080` ![image.png](https://foruda.gitee.com/images/1769484628926098932/2ab922de_16012914.png) 运行脚本 ``` #执行修改好的 app.py 文件 python app.py ``` ![image.png](https://foruda.gitee.com/images/1764037300419620049/23256c21_16012914.png) 将网址：http://localhost:8080/粘贴到浏览器中，便可与模型进行对话 ![image.png](https://foruda.gitee.com/images/1769483675760577491/9c143d78_16012914.png)

Step3-VL-10B-多模态推理