FLCgeigeigei/tinyvision-vlm

TinyVision

A minimal vision-language model (VLM) with a hand-written PyTorch architecture and a complete data → pretraining → fine-tuning → conversion → deployment pipeline for learning and reproduction.

• Vision encoder: frozen SigLIP 2, 256×256 input, 32×32 patches, and 64 visual tokens per image
• Language model: Llama-style architecture (RMSNorm / RoPE / GQA / SwiGLU), initialized from flcmind SFT weights by default
• Two variants: Dense (65.09M language parameters) and MoE (199.60M total / 65.12M activated per token, 4 experts with Top-1 routing)
• Tokenizer: custom 6,400-token vocabulary with visual special tokens
• Projector: LayerNorm → Linear → GELU → Linear, mapping visual features into the language model space

Resource	Link
Source code and reproduction guide	GitHub: FLCgeigei/tinyvision
Training data	FLCgeigeigei/tinyvision-vlm-data
Language base weights	FLCgeigeigei/flcmind

This repository contains the SFT checkpoints in standard Transformers format. The dense/ and moe/ subfolders include config.json, model.safetensors, tokenizer files, chat template, and the custom vlm.py implementation. The shared frozen vision encoder is stored separately under vision_encoder/.

Model Outputs

Qualitative examples on the same images. These are raw model responses rather than benchmark results. Responses can vary with checkpoints, prompts, and decoding parameters, and objects or attributes may be described incorrectly.

Input	Dense	MoE
	这张图片展示了一艘帆船在海上航行，船体是白色的。	这张图片展示了一艘帆船在水面上，看起来是在晴朗的蓝天下。船只主要是白色的，带有白色的帆，表明它是白色的。
	这张图片展示了一匹马站在一片开阔的草地上，背景是晴朗的蓝天。	这张图片展示了一匹棕色的马，站在一片广阔的草地上，背景是晴朗的蓝天。这匹马有着浓密的胡须，头部略微向前倾斜，眼睛朝向画面右侧。
	这张图片展示了一幅宁静的风景，在晴朗的蓝天下，一座山峰被积雪覆盖，暗示着寒冷的季节。	这张图片展示了一幅宁静的山脉景观，天空湛蓝，暗示着一个宁静的场景。前景是一片宁静的湖泊，湖面上倒映着天空的蓝色。
	这张图片展示了一只鸟停在一根长长的叶子上，背景是郁郁葱葱的绿色植被。	这张图片展示了一只鸟停在一根细长的茎上，背景是郁郁葱葱的绿色。这只鸟主要是蓝色的，带有白色的鳞片，头部和腹部有白色的鳞片。

Quick Start

Requirements: Python 3.9+, PyTorch 2.0+, transformers, and huggingface_hub. A CUDA GPU is recommended; Dense inference can also run on CPU.

The example below downloads the complete repository once and then loads all files locally. Change subfolder to "moe" to use the MoE checkpoint.

from pathlib import Path

import torch
from huggingface_hub import snapshot_download
from PIL import Image
from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "FLCgeigeigei/tinyvision-vlm"
subfolder = "dense"  # or "moe"
local_dir = Path("tinyvision-vlm")

snapshot_download(repo_id=repo_id, local_dir=local_dir)
model_path = local_dir / subfolder
vision_path = local_dir / "vision_encoder" / "siglip2-base-p32-256-ve"

device = "cuda" if torch.cuda.is_available() else "cpu"
dtype = torch.bfloat16 if device == "cuda" else torch.float32

# Custom architecture: trust_remote_code is required.
model = AutoModelForCausalLM.from_pretrained(
    model_path,
    trust_remote_code=True,
    dtype=dtype,
).to(device)
tokenizer = AutoTokenizer.from_pretrained(model_path)

# The language checkpoint does not contain the vision encoder.
model.vision_encoder, model.processor = model.get_vision_model(str(vision_path))
model.vision_encoder = model.vision_encoder.to(device=device, dtype=dtype)

image = Image.open("photo.jpg").convert("RGB")
pixel_values = {
    key: value.to(device=device, dtype=dtype)
    for key, value in model.image2tensor(image, model.processor).items()
}

messages = [
    {"role": "user", "content": "<|image_pad|>" * 64 + "请描述这张图片"}
]
prompt = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
)
inputs = tokenizer(prompt, return_tensors="pt").to(device)

output = model.generate(
    **inputs,
    pixel_values=pixel_values,
    max_new_tokens=256,
    do_sample=False,
)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Important Notes

• trust_remote_code=True is required because TinyVisionVLM is loaded from vlm.py through auto_map. Review the custom code before enabling it.
• Use exactly 64 <|image_pad|> tokens per image, matching the 64 SigLIP 2 patches for a 256×256 input.
• Attach the vision encoder separately with model.get_vision_model(...); dense/model.safetensors and moe/model.safetensors contain only the language model and projector.
• Dense supports bf16 directly. The MoE FFN internally upcasts to fp32; if NaN logits still occur, load the model with dtype=torch.float32.
• The included chat_template.jinja should be used through tokenizer.apply_chat_template(...) as shown above.

Architecture

SigLIP 2 encodes an image into 64 visual tokens. The MLP projector maps them into the 768-dimensional language model space, where they replace the <|image_pad|> placeholders before entering the decoder-only language model.

Item	Dense	MoE
Language parameters	65.09M	199.60M total / 65.12M activated
hidden_size / layers	768 / 8	768 / 8
Attention	GQA: 8 query heads / 4 KV heads	GQA: 8 query heads / 4 KV heads
FFN	SwiGLU	4 experts, Top-1 routing, load-balancing auxiliary loss
Normalization	RMSNorm	RMSNorm
Positional encoding	RoPE, optional YaRN	RoPE, optional YaRN
Vocabulary	6,400	6,400
Vision encoder	Frozen SigLIP 2	Frozen SigLIP 2
Image tokens	64 (256×256, 32×32 patches)	64 (256×256, 32×32 patches)

Parameter counts cover the language model only; the shared vision encoder is excluded.

Training

Training uses FLCgeigeigei/tinyvision-vlm-data and two stages:

1. Image-text pretraining: initialize the language model from flcmind SFT weights by default; freeze the language model and vision encoder, and train only the projector.
2. Supervised fine-tuning (SFT): continue from the pretraining checkpoint, unfreeze the first and last language layers, and train on image-caption conversations. The vision encoder remains frozen.

The language architecture is hand-written, but the default training recipe does not start from random language weights. Fully random initialization is supported in the source project with --from_weight none.

See the GitHub repository for exact Dense/MoE commands, data preparation, weight paths, conversion, CLI inference, and the Gradio web UI.

The curves above are training losses from actual runs, not validation-set metrics or downstream benchmark results.

Repository Files

Path	Purpose
dense/	Dense SFT checkpoint, tokenizer, chat template, config, and custom modeling code
moe/	MoE SFT checkpoint, tokenizer, chat template, config, and custom modeling code
vision_encoder/siglip2-base-p32-256-ve/	Shared frozen SigLIP 2 vision encoder and image processor
assets/	Architecture diagram, training report, and qualitative example images
config.json	Repository-level architecture metadata for model discovery

Limitations

• Very small model: limited world knowledge, weak reasoning, and frequent visual-description errors are expected.
• Chinese and English only, with a 6,400-token vocabulary.
• No RLHF or safety tuning; outputs should not be treated as factual or safety-filtered.
• No benchmark evaluation is claimed; the examples and training curves are qualitative diagnostics.
• Intended for learning and experimentation, not production use.

License

TinyVision weights and code are released under Apache 2.0. Third-party components, including SigLIP 2 and flcmind, retain their original licenses; consult their respective model cards before use.