meforce
/

CetinLM-1B-Base

@@ -1,680 +0,0 @@
----
-pipeline_tag: text-generation
-inference: false
-language:
-  - en
-  - tr
-  - de
-  - fr
-  - es
-  - pt
-  - it
-  - nl
-  - pl
-  - ru
-  - uk
-  - ar
-  - fa
-  - hi
-  - bn
-  - ur
-  - id
-  - vi
-  - th
-  - zh
-  - ja
-  - ko
-tags:
-  - cetinlm
-  - me-force-technology
-  - text-generation
-  - causal-lm
-  - base-model
-  - from-scratch
-  - multilingual
-  - turkish
-  - gqa
-  - rope
-  - pytorch
-  - research
-  - single-gpu-training
----
-# CetinLM-1B Base
-> **A from-scratch language-model research program built in Türkiye, designed for global use.**
->
-> **Status:** Active base pretraining · Research preview · Weights not yet released
-**CetinLM-1B is not the final product. It is the first instrument in the laboratory.**
-CetinLM is an independent language-model research program focused on a simple question:
-> **How much useful capability can disciplined architecture, data, training-systems, diagnostics, post-training, and inference engineering extract from every parameter and every unit of compute before scale becomes the answer?**
-The 1B model is our controlled research scale: small enough to inspect deeply, large enough for real language-model behavior to emerge.
-We are building the stack from first principles, measuring it while it trains, publishing both successful and failed experiments, and turning those findings into a repeatable methodology for future CetinLM models.
-**Built in Türkiye 🇹🇷. Engineered for a global audience.**
----
-## Research Preview
-This repository currently documents an **actively trained raw base model**.
-It is **not yet an instruction model, chat model, reasoning model, or production release**.
-At this stage, CetinLM-1B performs causal next-token prediction. Conversation behavior, instruction following, preference alignment, reasoning specialization, code specialization, and production inference optimization are planned as later phases.
-### Latest published training snapshot
-| Item | Value |
-|---|---:|
-| Model | CetinLM-1B Base |
-| Parameters | **1,048,780,544** |
-| Training step | **73,000** |
-| Processed training tokens | **598,016,000** |
-| Initial base-pretraining target | **1,000,000,000 tokens** |
-| Progress | **59.80%** |
-| Best validation loss | **3.622009** |
-| Best validation perplexity | **37.413** |
-| Training hardware | **1× NVIDIA RTX 4070 Ti SUPER 16 GB** |
-| Training precision | **BF16** |
-| Current training sequence length | **256 tokens** |
-> These numbers are a dated research snapshot, not a claim of final model quality. Training is continuing.
----
-# Why CetinLM Exists
-The project is not based on the thesis that parameter count is irrelevant.
-Scale matters.
-The research question is what happens **before** scale is used as the default solution.
-Modern language models contain many interacting systems:
-- architecture,
-- tokenizer design,
-- corpus construction,
-- document boundaries,
-- language mixture,
-- optimization,
-- batch geometry,
-- activation memory,
-- attention kernels,
-- checkpointing,
-- evaluation methodology,
-- context training,
-- post-training,
-- decoding,
-- serving,
-- and hardware utilization.
-CetinLM treats these as an engineering system rather than independent checkboxes.
-The goal is not:
-> “Can a 1B model run on a consumer GPU?”
-The more interesting question is:
-> **“Can we develop a model-building methodology at 1B that remains useful when the same engineering discipline is applied at much larger scales?”**
-CetinLM-1B is the test bench for that question.
----
-# Model Architecture
-CetinLM-1B is a decoder-only causal Transformer trained from scratch.
-| Component | Configuration |
-|---|---:|
-| Parameters | 1,048,780,544 |
-| Vocabulary | 65,536 |
-| Hidden size | 1,792 |
-| Transformer layers | 20 |
-| Query heads | 28 |
-| KV heads | 7 |
-| Attention | Grouped-Query Attention (GQA) |
-| Head dimension | 64 |
-| MLP intermediate size | 7,168 |
-| Position encoding | Rotary Position Embeddings (RoPE) |
-| RoPE theta | 10,000 |
-| Configured maximum sequence length | 4,096 |
-| Current pretraining sequence length | 256 |
-| Bias | False |
-| Dropout | 0.0 |
-| Input/output embeddings | Tied |
-| Objective | Causal next-token prediction |
-### Important context-length note
-The architecture is configured for sequences up to 4,096 tokens, but the current v0 base-pretraining run uses **256-token training sequences**.
-A configured maximum context length is not the same as a context length that has been sufficiently trained and validated.
-Long-context capability will therefore be treated as a separate research stage rather than claimed prematurely.
----
-# Tokenizer and Language Coverage
-CetinLM uses a custom **65,536-token multilingual tokenizer**.
-Current training data includes material from the following language set:
-`en, tr, de, fr, es, pt, it, nl, pl, ru, uk, ar, fa, hi, bn, ur, id, vi, th, zh, ja, ko`
-English and Turkish receive elevated weight in the current data mixture.
-Special tokens:
-| Token | Role |
-|---|---|
-| `<pad>` | Padding |
-| `<unk>` | Unknown token |
-| `<bos>` | Beginning of sequence/document |
-| `<eos>` | End of sequence/document |
-Language inclusion in the training mixture does **not** imply equal capability across all listed languages. Language-specific evaluation will be expanded as training progresses.
----
-# Training System
-The current run is optimized for a single 16 GB GPU without changing the effective optimization target.
-### Current stable training configuration
-| Setting | Value |
-|---|---:|
-| Micro-batch size | 8 |
-| Gradient accumulation | 4 |
-| Sequence length | 256 |
-| Effective tokens / optimizer step | 8,192 |
-| Optimizer | AdamW8bit |
-| Peak learning rate | 2e-4 |
-| Minimum learning rate | 2e-5 |
-| LR schedule | Warmup + cosine decay |
-| Warmup | 2,000 optimizer steps |
-| Adam betas | 0.9 / 0.95 |
-| Weight decay | 0.1 |
-| Gradient clipping | 1.0 |
-| Precision | BF16 |
-| Activation memory | Transformer-block gradient checkpointing |
-| DataLoader workers | 0 |
-| Pinned memory | Enabled |
-Observed steady-state training throughput is approximately **5.6K tokens/second** on the current system.
-The project began with a substantially slower training geometry. Batch/accumulation experiments preserving the same effective 8,192-token optimizer step increased throughput by roughly **2.7×** relative to the initial configuration.
-The optimization principle is:
-> **Profile → form a hypothesis → A/B test on the real model → merge only if it wins.**
----
-# Kernel-Level Profiling and SDPA Research
-CetinLM does not assume that a theoretically attractive backend is faster on the actual training system.
-The training stack was profiled at operator/kernel level, revealing that matrix multiplication, activation-checkpointed forward computation, copies, pointwise operations, and attention were the dominant areas worth investigating.
-A controlled SDPA backend study was then performed using the real CetinLM-1B training shape.
-Current environment at the time of the study:
-- Windows
-- PyTorch 2.11.0 + CUDA 12.8
-- NVIDIA RTX 4070 Ti SUPER
-- BF16
-- batch 8
-- sequence length 256
-- 28 query heads / 7 KV heads
-### Result
-| Candidate | Relative result |
-|---|---:|
-| Native Math SDPA + native GQA | **Winner / baseline** |
-| cuDNN SDPA + native GQA | ~68.5% slower |
-| Math SDPA with BF16 reduction switch | ~44.8% slower |
-| Efficient SDPA with explicit KV expansion | ~48.2% slower |
-| Flash SDPA | Unavailable in the tested PyTorch/Windows build |
-This was an important negative result.
-The profiler initially made the Math SDPA path look suspicious. Measurement showed that it was actually the best available backend for this specific system and shape.
-**No backend change was made.**
-Failed optimizations are treated as research output rather than hidden experiments.
----
-# Scientific Diagnostic: What Is the Base Model Actually Learning?
-Generation samples alone can be misleading, especially for a raw base model.
-CetinLM therefore tracks fixed next-token semantic probes across checkpoints and measures where the expected target appears in the full vocabulary distribution.
-A target moving from rank 200 to rank 2 is meaningful even if greedy generation is still repetitive.
-### Earlier checkpoint → Step 70K
-| Probe | Earlier target rank | Step 70K rank |
-|---|---:|---:|
-| `The capital of France is` → `Paris` | ~211 | **2** |
-| `Paris is the capital of` → `France` | 2 | **2** |
-| `The opposite of hot is` → `cold` | 13 | **3** |
-| `Water freezes at` → `0` | 17 | **13** |
-| `2 + 2 =` → `4` | 5 | **4** |
-| `Türkiye'nin başkenti` → `Ankara` | 6 | **1** |
-| `Ankara, Türkiye'nin` → `başkenti` | ~10 | **36** |
-Aggregate target placement:
-| Metric | Earlier | Step 70K |
-|---|---:|---:|
-| Top-1 | 0 / 7 | **1 / 7** |
-| Top-5 | 2 / 7 | **5 / 7** |
-| Top-20 | 6 / 7 | **6 / 7** |
-Two examples are especially useful:
-```text
-The capital of France is
-Target: Paris
-Earlier rank: ~211
-Step 70K rank: 2
-```
-```text
-Türkiye'nin başkenti
-Target: Ankara
-Earlier rank: 6
-Step 70K rank: 1
-```
-These measurements do not prove general intelligence or benchmark superiority.
-They do show that the model's probability distribution is becoming substantially sharper around several learned semantic relations as pretraining progresses.
----
-# Validation Trajectory
-Recent best checkpoints:
-| Step | Processed tokens | Validation loss | PPL |
-|---:|---:|---:|---:|
-| 58K | 475.1M | 3.753456 | 42.668 |
-| 59K | 483.3M | 3.735000 | 41.888 |
-| 60K | 491.5M | 3.711722 | 40.924 |
-| 63K | 516.1M | 3.693981 | 40.205 |
-| 67K | 548.9M | 3.677402 | 39.544 |
-| 69K | 565.2M | 3.667538 | 39.155 |
-| 70K | 573.4M | 3.650751 | 38.504 |
-| 71K | 581.6M | 3.639504 | 38.073 |
-| 72K | 589.8M | 3.626853 | 37.594 |
-| 73K | 598.0M | **3.622009** | **37.413** |
-At the latest published snapshot, the model is still establishing new validation lows.
----
-# Generation Behavior at This Stage
-CetinLM-1B is currently a **raw base model**.
-It should not be evaluated as though it were an instruction-tuned assistant.
-For example, the base model has learned enough signal to produce continuations such as:
-```text
-Prompt:
-The capital of France is
-Continuation:
-The capital of France is the city of Paris...
-```
-but greedy decoding can then fall into repetition:
-```text
-The city of Paris is the city of Paris...
-```
-Other raw-base behaviors include:
-- repeating a question rather than answering it,
-- local phrase loops,
-- weak stopping behavior under some decoding conditions,
-- factual inconsistency in longer continuations,
-- weak arithmetic,
-- sensitivity to prompt phrasing.
-These are documented rather than hidden.
-Instruction following, conversational behavior, reasoning, code specialization, and preference alignment belong to later training phases.
----
-# Corpus Boundary Audit
-A generation-quality discussion raised a concrete engineering question:
-> Was EOS actually present at document boundaries in the tokenized training shards?
-Instead of guessing, the full 1B-token corpus was audited.
-### Audit results
-| Metric | Training | Validation |
-|---|---:|---:|
-| Tokens | 1,000,000,000 | 10,000,000 |
-| EOS tokens | 1,884,722 | 18,952 |
-| EOS / 1M tokens | 1,884.72 | 1,895.20 |
-The train and validation EOS rates are closely aligned.
-Decoded boundary windows confirmed that EOS tokens occur at real document endings.
-### Technical debt discovered
-Many inspected boundaries use:
-```text
-<eos><eos><bos>
-```
-rather than the intended future representation:
-```text
-<eos><bos>
-```
-This redundant second EOS is **not currently classified as a critical failure**:
-- the semantic document boundary is correct,
-- train/validation boundary statistics align,
-- validation continues to improve,
-- changing the already-tokenized corpus halfway through v0 would alter the data distribution mid-run.
-The current 1B-token run therefore remains unchanged.
-The next corpus builder will remove redundant EOS insertion and add automated boundary invariants.
-Planned checks include:
-- double-EOS count,
-- EOS/BOS boundary correctness,
-- special-token frequencies,
-- document count,
-- tokens per document,
-- missing/broken boundaries.
----
-# What We Refuse to Hide
-CetinLM is being developed as an engineering and research program, not as a polished demo first.
-That means public documentation includes:
-- hypotheses that failed,
-- optimizations that benchmarked worse,
-- training limitations,
-- data-pipeline technical debt,
-- diagnostic regressions,
-- generation failures,
-- and decisions to **not** change the model when the evidence does not justify intervention.
-A healthy experiment is more valuable than a perfect-looking progress post.
----
-# Roadmap
-## Phase I — CetinLM-1B Base v0
-Current objective:
-```text
-1.049B parameters
-1B base-pretraining tokens
-fixed 256-token training sequences
-single-GPU research baseline
-```
-At 1B tokens, v0 will be frozen and evaluated comprehensively.
----
-## Phase II — Continued Pretraining
-The next stage will continue from the trained checkpoint rather than restart the model.
-Training will be evaluated in controlled **+1B-token milestones**:
-```text
-1B total -> freeze + benchmark
-2B total -> freeze + benchmark
-3B total -> freeze + benchmark
-4B total -> freeze + benchmark
-5B total -> comprehensive evaluation
-```
-The goal is to retain control points rather than treating continued pretraining as one opaque run.
----
-## Phase III — Dynamic-Length Training
-CetinLM will not simply force every document through a 4,096-token sequence.
-The planned trainer uses document-length-aware buckets:
-```text
-short documents   -> 256 / 512
-medium documents  -> 1024 / 2048
-long documents    -> 4096
-```
-with:
-- length bucketing,
-- EOS-aware sequence packing,
-- token-budgeted batching,
-- dynamic micro-batch sizing,
-- minimum practical padding,
-- explicit exposure to long-context samples.
-Conceptually:
-```text
-180-token document  -> 256 bucket
-430-token document  -> 512 bucket
-900-token document  -> 1024 bucket
-1700-token document -> 2048 bucket
-3500-token document -> 4096 bucket
-```
-The research objective is to train long-context capability without paying 4,096-token attention cost for every short document.
----
-## Phase IV — Post-Training
-Planned post-training work includes:
-- general instruction SFT,
-- chat/conversation SFT,
-- stronger Turkish and English instruction data,
-- math and reasoning,
-- code,
-- CetinLM identity and project knowledge,
-- preference/alignment work,
-- systematic decoding evaluation.
----
-## Phase V — Inference Engineering
-Planned runtime work includes:
-- KV cache,
-- faster autoregressive generation,
-- memory optimization,
-- serving/runtime profiling,
-- deployment-oriented quantization research,
-- API/runtime integration.
----
-# Intended Use
-The current repository is primarily intended for:
-- language-model research,
-- training-systems research,
-- small-model efficiency research,
-- reproducibility and engineering documentation,
-- multilingual base-model experimentation,
-- future post-training research.
-When weights are released, intended-use guidance will be expanded for the specific checkpoint being published.
----
-# Current Limitations
-The current research checkpoint:
-- is not instruction tuned,
-- is not a finished chat assistant,
-- has only been trained on 256-token sequences in the current v0 run,
-- can produce repetitive greedy generations,
-- can hallucinate facts,
-- has weak arithmetic/reasoning relative to later planned stages,
-- may behave inconsistently across languages,
-- has not yet undergone comprehensive safety/alignment post-training,
-- should not be used for high-stakes decisions.
-No claim is made that a 1B model replaces frontier-scale systems.
-The research thesis is about **capability extracted per parameter and per unit of compute**, and whether that methodology remains valuable when scaled.
----
-# Release Status
-### Model weights
-**Not yet released.**
-The model is still in active base pretraining.
-Weights will not be published merely to satisfy a calendar date. A release will follow a defined research milestone and evaluation pass.
-### Hosted inference
-Disabled for this research-preview repository.
-The architecture currently uses a custom PyTorch implementation, and no claim is being made yet about drop-in compatibility with Hugging Face Transformers or hosted inference providers.
-### License
-A model-weight license will be selected deliberately before the first public weight release.
-No license metadata is asserted here before that decision is made.
----
-# Project Links
-- **Website:** https://cetinlm.meforcetechnology.com
-- **GitHub:** https://github.com/xertxetin/CetinLM
-- **Hugging Face:** https://huggingface.co/meforce
-The GitHub repository contains public engineering and research logs covering training optimization, diagnostics, profiling, backend experiments, and milestone evaluations.
----
-# Origin
-CetinLM is developed independently under **Me Force Technology**.
-The research program originates in **Türkiye** and is built for global research, engineering, and future deployment.
-We believe globally relevant model research does not need to begin with the largest cluster.
-It can begin with a controlled experiment, a measurable hypothesis, and the discipline to understand the system before scaling it.
----
-# Research Philosophy
-We do not treat parameter count as a substitute for engineering.
-We do not treat a lower validation loss as proof that every capability improved.
-We do not treat a compelling generation sample as a benchmark.
-We do not treat a profiler suspicion as an optimization result.
-We do not merge an optimization because it sounds theoretically better.
-We measure.
-And when the measurement says the existing system is better, we keep it.
-> **Compute is not the experiment. Engineering is.**
-CetinLM-1B is the first instrument in the lab.
-The larger project is the methodology behind what comes next.
----
-# Citation
-If you reference the CetinLM research program or its public engineering results, please cite the project repository:
-```bibtex
-@misc{cetinlm2026,
-  title        = {CetinLM: From-Scratch Language Model Research Program},
-  author       = {CetinLM Research Program, Me Force Technology},
-  year         = {2026},
-  howpublished = {\url{https://github.com/xertxetin/CetinLM}},
-  note         = {Independent language-model research originating in Türkiye}
-}
-```
----
-# Transparency Note
-Metrics in this card are reported from specific checkpoints and diagnostic scripts used during an active research run.
-They should be read as **experimental observations**, not as claims of benchmark leadership.
-As CetinLM evolves, this card will be updated with:
-- frozen checkpoint identifiers,
-- standardized evaluations,
-- data documentation,
-- context evaluations,
-- model artifacts,
-- inference instructions,
-- post-training results,
-- and release-specific limitations.
-**Last research snapshot documented here:** Step 73,000 · 598,016,000 processed tokens · 2026-08-30.