VectraYX-1B (text-only LLM backbone)

1.04-billion-parameter Spanish/LATAM cybersecurity decoder, trained from scratch. This is the text-only backbone (no vision encoder) — the ViTLM-language model inside VectraYX-Vision-1B, exported standalone for text-only use or as the base of a custom VLM.

Architecture

Component	Value
Parameters	1,041.9M (tied embeddings)
Layers / d_model / d_ffn	22 / 2048 / 5504
Attention	GQA (16 query / 4 KV heads)
Activation	SwiGLU
Norm	RMSNorm pre-norm
Positional encoding	RoPE (θ=1e6) on 3/4 layers, NoPE on every 4th layer
Vocabulary	32,768 (BPE, BOS=2, EOS-doc=3, end-turn=7)
Special tokens	`<

The periodic NoPE (no-positional-encoding) layers are the architectural novelty: three of every four layers apply RoPE, and every fourth layer applies no positional encoding at all. This interleave was pretrained into the backbone and is preserved here.

Training

Four-phase curriculum (the vision phase belongs to the vision repo, not this one):

1. Phase 1 — 9.2B tokens general Spanish pretraining
2. Phase 2 — ≈50B tokens three-block curriculum (factual → code+math → domain+decay)
3. Phase 3 — ≈6B tokens tooling SFT (30% tool / 25% cyber / 20% reasoning / 15% ES replay / 6% code / 4% math), establishing native <|tool_call|> + <|think|> behavior in text

This checkpoint is the post-vision LLM extracted from the v3B run (step 1900 of a low-LR continuation of phase-4b), with the visual projector and SigLIP encoder stripped out. The backbone retains its phase-1–3 text competence plus whatever the vision phase induced; for text-only use it behaves as a phase-3-equivalent 1B Spanish cybersecurity model.

Files

• vectrayx-1b.gguf — F16 GGUF (2.22 GB), runnable in llama.cpp / Ollama

Usage (Ollama)

# Create a Modelfile
cat > Modelfile <<EOF
FROM ./vectrayx-1b.gguf
PARAMETER temperature 0.7
PARAMETER num_predict 512
EOF

ollama create vectrayx-1b -f Modelfile
ollama run vectrayx-1b

Usage (llama.cpp)

./main -m vectrayx-1b.gguf -p "Explique la diferencia entre un buffer overflow y un use-after-free" --temp 0.7 -n 512

Notes

• The model is sub-optimally trained at the current step volume (~50B tokens vs Chinchilla-optimal ~20B for 1B params is over-training; under-training at the high-quality-data level is the actual limit). Spanish fluency is good; tool-use and reasoning are present but volatile single-seed.
• Export note: NoPE layers export as RoPE layers with rotary application disabled per layer (rope dimension count = 0). The GGUF RoPE permute fix using stack([w1,w2], dim=2) interleaved is applied — do NOT use permute(0,2,1,3) for NEOX→NORM conversion.

Citation

See the accompanying paper, VectraYX-Vision-1B: A Sub-2B Spanish/LATAM Cybersecurity Vision-Language Model with Structured Visual Reasoning and Native Tool Use.

Author

Juan S. Santillana — DevOps engineer (Globant). Institutional affiliation approval pending.