pyloxsystems/hochelaga-8b

Hochelaga 8B

The first open-weights foundation model for Quebec French.

Built solo in two weeks on a single DGX Spark by Pylox Systems Inc.. No institutional funding. No team. No academic affiliation.

Hochelaga 8B is Qwen3-8B-Base continually pretrained on 257M tokens of Quebec French (3.5× the academic baseline corpus), then supervised fine-tuned across six chained iterations (SFT-A v1 → v6) on linguistic acceptability. On the published primary Quebec French benchmark (QFrCoLA), it beats Claude Opus 4.7 by 2.2pp accuracy and 4.5pp macro-F1.

---

Headline numbers

Quality, QFrCoLA (Quebec French grammaticality, 7,546 test examples)

Model	Accuracy	Macro-F1	Notes
Khoury et al. LREC 2026 baseline	—	46.22	LoRA, Llama 3.1 8B, 86M tokens, one year, $60K IVADO funding, full team
Claude Opus 4.7	83.85%	78.29	Closed-weights, hosted API
Hochelaga 8B, best single model (SFT-A v5, zero-shot)	85.44%	82.36	Open weights, two weeks, solo, $0 institutional funding
Hochelaga 8B, Bayesian model averaging (BMA)	85.98%	82.75	12-variant ensemble, highest F1
Hochelaga 8B, LR-stacking ensemble (best)	86.02%	82.46	Logistic-regression stacking across prompt variants

Both Hochelaga ensembles beat Claude Opus 4.7 on the published primary metric (macro-F1) by ~4.5 percentage points and on accuracy by ~2 percentage points. The single best Hochelaga checkpoint also beats Opus on both metrics, the ensembles widen the margin further.

Throughput, RTX 5090 (sm_120, 32 GB VRAM)

Fresh verified measurement on 2026-05-20 (Vast.ai instance 37117689, driver 580.126.09, vLLM 0.19.1 + Marlin NVFP4 backend):

Configuration	Concurrency	Aggregate tok/s	Per-stream tok/s
Single-stream NVFP4 (CUDA Graphs ON)	1p	212.3	211.9 ⭐
NVFP4 baseline	100p	6,551.9	74.8
NVFP4 baseline	400p	8,760.0	25.1
NVFP4 baseline	800p	8,907.9	12.3
NVFP4 baseline (peak aggregate)	1,600p	8,919.4 ⭐	10.5
NVFP4 baseline (saturated)	2,400p	8,880.0	10.5
NVFP4 + EAGLE-3 K=1	1p	188.7	197.1
NVFP4 + EAGLE-3 K=1	100p	3,844.4	44.4
NVFP4 + EAGLE-3 K=1	400p	4,443.2	13.6
NVFP4 + EAGLE-3 K=2	1p	169.3	178.6
NVFP4 + EAGLE-3 K=4	1p	153.1	154.2
NVFP4 + EAGLE-3 K=8	1p	115.1	115.7

EAGLE-3 speculative drafter trained natively against Hochelaga 8B weights for 49,608 steps (4h24min on RTX 5090).

Backend note: vLLM's FlashInfer-CUTLASS and TRT-LLM NVFP4 kernels for sm_120 fail to compile in all currently-released vLLM versions. The Marlin NVFP4-via-INT4 fallback (VLLM_NVFP4_GEMM_BACKEND=marlin) is the only working NVFP4 path on sm_120 in current public vLLM stable releases. Marlin produces the 8,919 tok/s peak but masks the EAGLE-3 speedup that would normally show on FP4-native kernels. Building vLLM from source with sm_120 in TORCH_CUDA_ARCH_LIST and FlashInfer-CUTLASS enabled is filed as a v1.1 deliverable.

Operating cost

At the verified peak of 8,919 tok/s aggregate on a $2,000 consumer GPU (RTX 5090), electricity-only operating cost is on the order of ~$0.0015 per million tokens output (framing figure — electricity only, excludes hardware amortization).

For comparison: Anthropic Claude Opus API is $15 per million tokens output.

Roughly four orders of magnitude (~10,000×) cheaper to operate at peak throughput, electricity-only.

---

Methodology

1. Continued pretraining (CPT)

Full-parameter (not LoRA, not QLoRA) continued pretraining of Qwen3-8B-Base on a curated Quebec French corpus. The per-source figures below are raw token counts before deduplication; after MinHash-LSH dedup and filtering, the final training corpus is ≈257M tokens (3.5× the academic baseline's 86M):

• Hansard, Quebec National Assembly debates, 1867-present (~80M tokens)
• Wikipedia FR-QC filtered subset (~77M tokens)
• HuggingFace extras, filtered Quebec French web content (~166M tokens, post-spam-filter)
• LégisQuébec, Quebec statute and regulation corpus (~1M tokens)
• CanLII Quebec, Quebec court decisions
• Érudit, Quebec academic publications
• Reddit Quebec / Stack Exchange FR Quebec, colloquial Quebec French
• Project Gutenberg Quebec titles + 20 historical Quebec works (~22M tokens)

MinHash LSH deduplication. FFD (first-fit-decreasing) sequence packing for full context utilization.

2. Quebec-extended BPE tokenizer

~3,000 new tokens added before CPT for:

• Joual contractions (j'veux, t'as, c'est-tu)
• Quebec institutional abbreviations (SAQ, RAMQ, CAQ, CLSC, Hydro-Québec, cégep)
• Quebec proper nouns

3. Supervised fine-tuning (SFT-A v1 → v6)

Six chained SFT iterations on Quebec French linguistic acceptability:

• v1: logit-pair contrastive loss
• v2: hard-negative mining
• v3: back-translation augmentation
• v4: calibration pass
• v5: auto-chained refinement
• BMA K=4 ensemble: scale=5 Bayesian model averaging across [v5_zs, v3_alt1, v4_zs, v5_5s]

4. Speculative decoding (inference-time, optional)

EAGLE-3 drafter fine-tuned natively against Hochelaga 8B weights (49,608 steps, 4h24min on RTX 5090). Honest current-stack result: on the only NVFP4 backend that compiles on sm_120 in current public vLLM (Marlin), EAGLE-3 K=1 shows no net single-stream gain over the raw NVFP4 baseline (197.1 vs 211.9 tok/s). The larger speculative speedups measured earlier (≈1.9× single-stream) were on vLLM-1.0-rc FlashInfer-CUTLASS FP4-native kernels that no longer compile on sm_120; they are not reproducible on current public vLLM and are not claimed here. Re-enabling them via a source build is a tracked v1.1 item.

---

Training infrastructure

Component	Spec
Hardware	NVIDIA DGX Spark GB10 Blackwell, 128 GB unified memory
Precision	BF16
Attention	cuDNN SDPA (Flash-attention 2 incompatible with sm_121, froze the chip)
Kernels	Liger Kernel (fused cross-entropy, RMSNorm, RoPE, SwiGLU)
Optimizer	PagedAdamW8bit
Batching	MICRO_BATCH=2, GRAD_ACCUM=8 → 131,072 effective tokens/step
Memory enforcement	systemd cgroup --property=MemoryMax=118G
Sustained throughput	~990 tokens/second (post-optimization)

The honest list of what didn't work: FP8 precision (cuBLAS 13.2 symbol missing), Flash-Attention 2 (sm_121 hardware freeze), torch.compile (Triton shared-memory limit 147KB vs 101KB available), Medusa speculative decoding (Qwen3ForCausalLM lacks get_output_embeddings).

---

Files

Path	Description
hochelaga-8b-bf16/	Full BF16 weights for benchmarking and further fine-tuning
hochelaga-8b-nvfp4/	NVFP4 W4A4 quantized weights for fast inference
hochelaga-8b-eagle3-drafter/	EAGLE-3 speculative decoding drafter (vLLM-compatible)
tokenizer/	Quebec-extended BPE tokenizer (~3K added tokens)
eval/	Reproducible benchmark scripts (QFrCoLA, QFrBLiMP, QFrCoRE, QFrCoRT)

---

Reproducibility

git clone https://github.com/pyloxsystems/hochelaga
cd hochelaga/bench

# Single-model baseline (no ensemble)
python bench_qfrcola_single.py --model pyloxsystems/hochelaga-8b --variant v5_zs --split test
# Expected: accuracy=0.8544, macro_f1=0.8236

# LR-stacking ensemble (best accuracy)
python bench_qfrcola_ensemble.py --model pyloxsystems/hochelaga-8b --method lr_stacking --split test
# Expected: accuracy=0.8602, macro_f1=0.8246

# BMA ensemble (best macro-F1)
python bench_qfrcola_ensemble.py --model pyloxsystems/hochelaga-8b --method bma --split test
# Expected: accuracy=0.8598, macro_f1=0.8275

---

Honest caveats

1. Speculative decoding shows no net gain on the current public stack. On the Marlin NVFP4 backend (the only one that compiles on sm_120 in current public vLLM), raw NVFP4 peaks at 8,919 tok/s aggregate and 211.9 tok/s single-stream, and EAGLE-3 K=1 does not beat it (197.1 single-stream). Larger speculative and raw-throughput figures measured in May 2026 on vLLM-1.0-rc FlashInfer-CUTLASS FP4-native kernels are not reproducible on current public vLLM (those kernels no longer compile on sm_120) and are therefore not reported as current results. Re-enabling them via a source build is tracked for v1.1.

2. Medusa speculative decoding incompatibility. Medusa heads were trained but Qwen3ForCausalLM in current vLLM lacks the get_output_embeddings method Medusa requires. The quad cascade (DFlash + EAGLE-3 + Medusa + PLD) falls back to DFlash alone at ~1,978 tok/s. Fixable upstream, not yet shipped.

3. Single-stream (1p concurrency) latency not separately measured on 5090. All 5090 numbers reported are vLLM aggregate (total tokens / wall-clock).

4. DGX Spark Blackwell numbers are not directly comparable to 5090 numbers due to different sm version, CUDA stack, and cuBLAS FP4 kernel maturity.

---

Why this exists

Quebec has 8 million French speakers producing a linguistically distinct dialect. Standard French language models trained on European data systematically fail on Quebec French: joual contractions, code-switching patterns, Quebec-specific institutions (SAQ, RAMQ, CAQ, Hydro-Québec), and the Quebec legal corpus (LégisQuébec) are not handled correctly.

The academic response, Khoury et al. at Concordia, IVADO-funded, one year, $60,000, produced a respectable LoRA fine-tune on 86M tokens of Llama 3.1 8B. This model exists to demonstrate that a solo founder with no funding can build the same artifact, on a stronger base, with 3.5× the data, in two weeks, and ship it open-weights.

---

Citation

@misc{girard2026hochelaga,
  title={Hochelaga 8B: The First Open-Weights Quebec French Foundation Model},
  author={Girard, Emilio},
  year={2026},
  publisher={Pylox Systems Inc.},
  howpublished={\url{https://huggingface.co/pyloxsystems/hochelaga-8b}},
  note={Continued pretraining of Qwen3-8B on 257M Quebec French tokens. QFrCoLA 86.02\% (LR-stacking ensemble), 85.44\% single-model, beats Claude Opus 4.7 (83.85\%, 0.7829 F1). Single DGX Spark, two weeks, solo.}
}

---

License

Apache 2.0 for model weights and inference code.

Training data sources retain their original licenses:

• Wikipedia: CC-BY-SA
• LégisQuébec: Open government licence
• Hansard: Quebec National Assembly licence
• Project Gutenberg: public domain
• CanLII: terms of use (court decisions, public)

---

Acknowledgements

• Khoury et al. (LREC 2026) for the QFrCoLA / QFrBLiMP / QFrCoRT / QFrCoRE benchmarks
• The Qwen team for the Qwen3-8B base model
• NVIDIA for the DGX Spark GB10 Blackwell hardware
• The Quebec linguistic data ecosystem: OQLF, BAnQ, Érudit, Assemblée nationale du Québec
• Vast.ai for affordable RTX 5090 instances

---

Built in Montréal · Pylox Systems Inc. · 2026

pyloxsystems@gmail.com · pyloxforge.com · @pyloxforge