File size: 8,279 Bytes
5f854a0 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 | ---
pretty_name: Slayer1B 1.08 training-data composition, reference v2
language:
- pl
- en
task_categories:
- text-generation
tags:
- continued-pretraining
- prepared-reference
---
# Slayer1B 1.08 — training-data composition
**Arkadiusz Słota, Anna Sieczka, Piotr Styła / Fabryka AI (SlayerLab)**
**Prepared reference v2, 9 November 2026.** Target reconstruction must match the approved digests before training. This documents the new 1.08 composition, not a separately hosted mixed-corpus release; the earlier 1.08b experiment is distinct.
## Prepared training reference
The composition combines analytical/reasoning-oriented text, English technical material and general-language replay. Retained counts come from the reference assembly manifest; they describe available input positions, not tokens consumed by a training run. Component names identify sources, not an audited per-document language distribution.
| Retained component | Native token positions including EOS |
|---|---:|
| PL Logika | 1,631,862,622 |
| English technical | 293,943,541 |
| Replay `pl_web` | 435,000,183 |
| Replay `en_fineweb_edu` | 220,000,324 |
| Replay `en_mix` | 220,000,212 |
| **TRAIN total** | **2,800,806,882** |
Reference manifest SHA256: `c0f1e7771104f60b92ec38d3576fcfe34ed125a072da529d8b56fe2da0f87e7b`. TRAIN SHA256: `b73a69eaf399bb594300e7d1914a967a38b80b5bc35505fbc6459a6a27be496f`.
| Separate partition | Documents | Token positions including EOS |
|---|---:|---:|
| Polish-logic held-out | 3,158 | 18,294,311 |
| Trainer validation, a subset of that held-out | 360 | 2,002,010 |
| Eligible English VALID held-out | 5,218 | 17,988,388 |
These are partition totals, not domain-evaluation sample sizes.
## Published inputs before filtering
The following source-published counts use their stated split and EOS units.
| Source / revision | Split | Documents | Native Slayer tokens |
|---|---|---:|---:|
| [PL Logika](https://huggingface.co/datasets/SlayerLab/slayer-pl-logika-2b/tree/e24c8b71526ca1dc56a280100f88c2f09a79ecdd), `e24c8b71526ca1dc56a280100f88c2f09a79ecdd` | published train | 327,219 | 1,911,918,958, including EOS |
| [English Technical](https://huggingface.co/datasets/SlayerLab/english-technical-1b/tree/3b833d36bd3ba9aeecf46f39605c9ac849bb96a3), `3b833d36bd3ba9aeecf46f39605c9ac849bb96a3` | train | 188,904 | 899,001,323, before pack-added EOS |
| Same English release | valid | 10,370 | 49,886,904, before pack-added EOS |
| Same English release | test | 10,665 | 51,226,632, before pack-added EOS |
The English all-split total, 1,000,114,859 tokens, is not training size. Its pack manifest adds one EOS per document: source TRAIN becomes **899,190,227 token positions**. Proxy cl100k counts are a different unit. Source counts, retained reference positions and observed training exposure are distinct quantities.
Replay comes from [Slayer 1.07 pretraining data](https://huggingface.co/datasets/SlayerLab/slayer-1.07-pretrain/tree/6a337bbfba518d88f14eb54f7d0a9b9f297df0d1), revision `6a337bbfba518d88f14eb54f7d0a9b9f297df0d1`: `pl_web`, `en_fineweb_edu` and `en_mix`. These texts were seen by 1.07; retained quotas are shown above.
## Composition and rights
Licenses differ by source; compilation terms do not grant rights to every underlying text. Reference selection excludes GPL/GFDL sources and mixed GPL/LGPL English sources. Retained components still carry their individual terms, not a blanket permissive-license grant.
### Polish corpus
Required conditions remain source-specific:
- **FinePDFs:** retain the notice “Contains information from FinePDFs, which is made available under the ODC Attribution License,” applicable Common Crawl terms and annotation attribution. PDF-content rights remain unresolved; upstream relies on a TDM rationale and did not recheck domain-level opt-outs.
- **Dolci:** preserve original Dolci, translation and component notices, including NVIDIA Nemotron CC BY 4.0. The translation license does not replace underlying obligations. Attribution identifies Gemma translation and mixed upstream reasoning teachers; embedded conversation-history provenance remains uncertain.
- **DOAB:** preserve each book's creator/editor, title, publisher, license, URL and modification notices. Selected material includes English books, so the corpus label is not a per-document language guarantee.
- **Tax material:** preserve Romuald Członkowski and Eureka MF/KIS attribution. Upstream did not locate the portal terms; absence of known restrictions is not confirmed permission.
The pinned [attribution inventory](https://huggingface.co/datasets/SlayerLab/slayer-pl-logika-2b/blob/e24c8b71526ca1dc56a280100f88c2f09a79ecdd/ATRYBUCJA.md) carries the full notices; this summary does not replace them.
### English technical corpus and replay
English sources span systems, machine learning, compilers, CNC/CAM, simulation and graphics. Row licenses and matching notices govern selection: excluded GPL sources include Linux, Blender, OpenFOAM and some CNC material; retained entries use MIT, Apache-2.0, Apache-2.0 WITH LLVM-exception, BSD-3-Clause, dual Apache/MIT or the FreeCAD repository's CC0 terms. The [source-license inventory](https://huggingface.co/datasets/SlayerLab/english-technical-1b/blob/3b833d36bd3ba9aeecf46f39605c9ac849bb96a3/licenses/SOURCES.md) records the notices. The retained technical count above reflects all filters, not just license exclusion.
FreeCAD documentation: **FreeCAD wiki contributors**, via `FreeCAD/FreeCAD-documentation` at `0499378a`, repository license **CC0-1.0**. This identifies the conversion repository's license; the underlying wiki attribution is retained.
Replay includes ODC-By FineWeb-Edu/FineMath/open-web-math, CC BY DCLM and HPLT. HPLT's compilation is CC0, not its underlying web texts; the replay publisher relies on Directive (EU) 2019/790 Article 4. Redistribution requires separate rights review. Required notices follow the pinned replay card.
## Preparation and split isolation
Preparation uses the same Slayer v6 tokenizer and little-endian uint16 token format, with EOS 0 after each document.
Reference TRAIN excludes whole documents flagged by the contact/identifier filter or containing literal special-token IDs in their text. Assembly metadata records removal counts and overlapping filters.
English valid/test remain outside TRAIN; eligible VALID uses the TRAIN license exclusions and removes documents also present in TRAIN. Polish held-out sections are grouped by source-book identity rather than splitting chapters across train and evaluation.
Target reconstruction uses the pinned preparation recipe, exclusion lists and the same input revisions. TRAIN, validation and held-out digests must match the approved reference artifacts before use.
## Quality, privacy and limitations
The reference uses recorded per-row identifier/overlap exclusions for new sources and inherits the replay release's previous checks. Legacy replay did not detect near-duplicates, omitted EWoK from its scan and retained some short overlaps. These limitations and exposure inherited through the base model remain relevant to evaluation.
Whole-document removal reduces FinePDFs coverage, including academic text with author contact details. Residual names, addresses, translation artifacts, PDF-layout damage and tax-text fragmentation can remain. Polish metadata reports no toxicity classifier and masking exceptions in programming/math contexts. Technical text can contain unsafe instructions; reasoning traces can be wrong. Domain labels and source counts establish neither model capability nor safe professional advice.
## Source metadata
Further pinned evidence: Polish [aggregate manifest](https://huggingface.co/datasets/SlayerLab/slayer-pl-logika-2b/blob/e24c8b71526ca1dc56a280100f88c2f09a79ecdd/manifest.json); English [aggregate manifest](https://huggingface.co/datasets/SlayerLab/english-technical-1b/blob/3b833d36bd3ba9aeecf46f39605c9ac849bb96a3/manifest.json) and [EOS-inclusive pack manifest](https://huggingface.co/datasets/SlayerLab/english-technical-1b/blob/3b833d36bd3ba9aeecf46f39605c9ac849bb96a3/slayer_v6/manifest.json). This documentation was prepared from public cards and aggregate/license metadata, not raw records or token binaries.
|