Vertex Pico v2

Tiny โ€ข 0.8M Parameters โ€ข Base model

Vertex Pico v2

Vertex Pico v2 is a 0.8M-parameter base language model (798,903 parameters) trained from scratch by Vertex Research on about 180M tokens. It is the second release in the Vertex Pico series, a family of tiny models built to get the most out of the smallest sizes. Compared with Vertex Pico v1 it has a deeper architecture, a smaller tokenizer, a revised data mix and a multiple-choice ranking objective in the final phase of training, which lifts its Intelligence Index from 4.68 to 5.84.

Training

  • About 180M tokens, one epoch.
  • Data: ClimbMix, DCLM-Edu, FineWeb-Edu, UltraX, TxT360 QA, Wikipedia, WikiHow, FineMath, MegaMath QA, SciQ and synthetic arithmetic text. ClimbMix is licensed CC-BY-NC 4.0.
  • 4 layers, width 128, 2 heads, MLP width 332, tied embeddings, RoPE, value and dense residual connections, gated attention and MLP outputs; byte-level BPE tokenizer with 1,536 tokens.
  • Context: 1,024-token attention window (sliding window beyond that), up to 2,048 positions.

Benchmarks

Zero-shot, float32, scored on every item of each evaluation set (no subsampling): HellaSwag 10,042 (validation), ARC-Easy 2,376 (test), ARC-Challenge 1,172 (test), PIQA 1,838 (validation), ArithMark-3 1,000. Best score in each column in bold.

Model Params HellaSwag ARC-Easy ARC-Challenge PIQA ArithMark-3 Intelligence Index
Vertex Pico v2 0.80M 28.49 32.20 23.72 55.06 28.00 5.84
Vertex Pico v1 0.80M 27.49 32.03 21.59 54.24 28.30 4.68
Ember 2.87M 27.35 33.25 21.08 54.41 32.90 5.94
CMA-1M-Mini 0.96M 29.35 29.29 21.76 54.62 31.20 5.78
GPT-S-1.4M 1.43M 26.89 31.57 21.93 55.17 30.20 5.40
Sol Nano 2.90M 28.40 32.07 21.16 53.92 33.80 6.07

HellaSwag, ARC and PIQA: lm-evaluation-harness, acc_norm. ArithMark-3: the official ArithMark-3 script. Intelligence Index: the Open SLM Leaderboard formula. Other models: scores as listed on the Open SLM Leaderboard. Vertex Pico v1 and v2: scored with the same benchmarks, settings and full item counts as the leaderboard.

Usage

The model uses custom code, so trust_remote_code=True is required.

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("VertexResearch/Vertex-Pico-v2", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("VertexResearch/Vertex-Pico-v2", trust_remote_code=True)

ids = tok("Reading books every day can help you", return_tensors="pt").input_ids
out = model.generate(ids, max_new_tokens=15, do_sample=False, repetition_penalty=1.3)
print(tok.decode(out[0], skip_special_tokens=True))

Limitations

This is a tiny base model for research and benchmarking. It is not instruction-tuned, and its text is often incoherent and factually wrong. Do not use it for anything that matters.

Downloads last month
37
Safetensors
Model size
799k params
Tensor type
F32
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Collection including VertexResearch/Vertex-Pico-v2