A newer version of this model is available: VertexResearch/Vertex-Pico-v2

Vertex Pico v1

Tiny โ€ข 0.8M Parameters โ€ข Base model

Vertex Pico v1

Vertex Pico v1 is a 0.8M-parameter base language model (798,726 parameters) trained from scratch by Vertex Research on about 155M tokens. It is the first release in the Vertex Pico series, a family of tiny models built to get the most out of the smallest sizes, with better versions already in training.

Training

  • About 155M tokens, one epoch.
  • Data: ClimbMix, DCLM-Edu, FineWeb-Edu, UltraX, TxT360 QA, Wikipedia, WikiHow, FineMath, MegaMath QA, SciQ. ClimbMix is licensed CC-BY-NC 4.0.
  • 3 layers, width 128, 2 heads, tied embeddings, RoPE; byte-level BPE tokenizer with 3,072 tokens.
  • Context: 1,024-token attention window (sliding window beyond that), up to 2,048 positions.

Benchmarks

Zero-shot, float32, scored on every item of each evaluation set (no subsampling): HellaSwag 10,042 (validation), ARC-Easy 2,376 (test), ARC-Challenge 1,172 (test), PIQA 1,838 (validation), ArithMark-3 1,000. Best score in each column in bold.

Model Params HellaSwag ARC-Easy ARC-Challenge PIQA ArithMark-3 Intelligence Index
Vertex Pico v1 0.80M 27.49 32.03 21.59 54.24 28.30 4.68
GPT-S-1.4M 1.43M 26.89 31.57 21.93 55.17 30.20 5.40
Supra2-Nano 0.80M 26.81 30.81 20.65 53.43 29.00 3.76
Sol-Milkshake 2.99M 25.02 29.76 22.70 52.50 30.60 3.16

HellaSwag, ARC and PIQA: lm-evaluation-harness, acc_norm. ArithMark-3: the official ArithMark-3 script. Intelligence Index: the Open SLM Leaderboard formula. Other models: scores as listed on the Open SLM Leaderboard. Vertex Pico v1: our own evaluation, pending leaderboard verification.

Usage

The model uses custom code, so trust_remote_code=True is required.

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("VertexResearch/Vertex-Pico-v1", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("VertexResearch/Vertex-Pico-v1", trust_remote_code=True)

ids = tok("Children learn best when", return_tensors="pt").input_ids
out = model.generate(ids, max_new_tokens=15, do_sample=False, repetition_penalty=1.3)
print(tok.decode(out[0], skip_special_tokens=True))

Limitations

This is a tiny base model for research and benchmarking. It is not instruction-tuned, and its text is often incoherent and factually wrong. Do not use it for anything that matters.

Downloads last month
115
Safetensors
Model size
799k params
Tensor type
F32
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Collection including VertexResearch/Vertex-Pico-v1