add research-reports/raw_evidence.md

research-reports/raw_evidence.md

@@ -0,0 +1,103 @@
+# Dovada empirică — date brute colectate
+
+Data: 2026-10-09. Metodă: interogare directă HF API / GitHub raw / dataset streaming.
+
+## Mediu sandbox (fără GPU)
+- Python 3.13.5, torch ABSENT, nvidia-smi ABSENT → antrenarea efectivă NU s-a putut rula aici.
+- Rețea: OK (HF 200 OK). GitHub API OK. arxiv export API blocat. DDG/Bing blocate (captcha/geo).
+
+## Arhitecturi verificate (config.json real, nu inventat)
+TinyLlama-1.1B-Chat-v1.0:
+  LlamaForCausalLM hidden=2048 interm=5632 layers=22 heads=32 kv=4 ctx=2048 vocab=32000 rope_theta=10000 rms_eps=1e-5 tie=False silu
+SmolLM2-360M:
+  hidden=960 interm=2560 layers=32 heads=15 kv=5 ctx=8192 vocab=49152 rope_theta=100000 tie=True silu
+SmolLM2-360M (nanotron config_smollm2_360M.yaml — START din 2023):
+  hidden=960 interm=2560 layers=32 heads=15 kv=5 ctx=2048 vocab=49152 tie=True
+  lr=0.003 (peak), warmup=2000, decay_style=linear, decay_start=2.08M, decay_steps=0.52M
+  micro_batch=6, seq_len=2048, batch_accum/replica=1, train_steps=2.6M, dp=128, tp=1, pp=1
+  opt: AdamW beta=(0.9,0.95) eps=1e-8 weight_decay=0.01 clip=1.0 accumulate_grad_in_fp32=True
+  dtype=bfloat16, init std=0.03227
+Phi-3-mini-4k-instruct:
+  Phi3ForCausalLM hidden=3072 interm=8192 layers=32 heads=32 kv=32 ctx=4096 vocab=32064
+MobileLLM-350M: model GATED (acces restricționat) → config indisponibil fără auth.
+
+## Rețeta TinyLlama (README oficial jzhang38/TinyLlama)
+- 1.1B, 3T tokens, 16×A100-40G, 90 zile
+- GQA (32 heads / 4 KV groups), layers=22, emb=2048, SwiGLU interm=5632
+- seq_len=2048, batch=2M tokens (2048*1024), lr=4e-4 cosine, warmup 2000
+- date: SlimPajama (fără subset GitHub) + Starcoderdata, ratio NL:cod=7:3, ~950B unice, ~3 epoci
+- optimizări: FSDP, FlashAttention2, fused layernorm/swiglu/xent/rope → 24k tok/s/A100 = 56% MFU
+- 3456 A100-h pentru 300B tokens; suport 3090/4090 cu batch mai mic
+
+## SmolLM / SmolLM2 / SmolLM3 (README HF + config nanotron)
+- SmolLM2-360M: 4T tokens (FineWeb-Edu + DCLM + The Stack + seturi filtrate); SmolLM2-1.7B: 11T tokens
+- SmolLM2 instruct: SFT (smoltalk) + DPO pe UltraFeedback
+- SmolLM3-3B: 11T tokens, dual-mode reasoning (think/no_think), NoPE + YaRN până la 128k, GQA 4 grupuri
+- Framework: nanotron + datatrove (HF)
+
+## Tooling (GitHub API, steluțe la 2026-10-09)
+- unslothai/unsloth      77,632 ★ updated 2026-10-09
+- karpathy/nanoGPT       63,670 ★ (note: karpathy recomandă nanochat în schimb; nanoGPT "deprecated")
+- huggingface/transformers 166,914 ★
+- Lightning-AI/litgpt    13,696 ★ updated 2026-10-07
+- KellerJordan/modded-nanogpt 5,927 ★ (124M în <40s pe 8×H100)
+- pytorch/torchtitan      5,792 ★ ICLR 2025, paper 2410.06511
+- jzhang38/TinyLlama      9,009 ★ (ultima actualizare 2024-05)
+- huggingface/nanotron (menționat de SmolLM)
+
+## litgpt config_hub/pretrain (YAML reale)
+- tinystories.yaml: stories15M (6 layere, n_embd=288, block=256, vocab=32000), gbs=512, mbs=128, lr=5e-4, max_tokens=9.7e9
+- microllama.yaml: micro-llama-300M, seq=2048, gbs=48 (4×4090 24G), mbs=12, lr=4e-4, max_tokens=3e12
+- tinyllama.yaml: tiny-llama-1.1b, seq=2048, gbs=512, mbs=4, lr=4e-4, warmup=2000, max_tokens=3e12
+- debug.yaml: pythia-14m, gbs=125, mbs=5, lr=6e-4, max_tokens=1e8
+
+## modded-nanogpt (README)
+- țintă: 3.28 val loss FineWeb = GPT-2 124M
+- <40s pe 8×H100; <330M tokens (vs 10B la llm.c)
+- tehnici: rotary, QK-Norm, ReLU², Muon optimizer, FP8 head/MLP/QKV, zero-init proj, skip connections,
+  extra value embeddings, FA3 sliding-window (Gemma2-style) + YaRN warmup, EoS-aligned batch starts
+
+## Datasets sintetice VERIFICATE
+- teknium/OpenHermes-2.5: 18,596 dl, 913 ★; 1M sample; tag 'synthetic','GPT-4','Distillation'
+  structură reală (stream): [{"conversations":[{"from":"human","value":...},{"from":"gpt","value":...}],
+    "source":"airoboros2.2","category":"orca","skip_prompt_formatting":false}, ...]
+  surse: Airoboros 2.2, CamelAI, LMSys ChatBot Arena (GPT-4 only), CoT-Alpaca-GPT4 etc.
+- HuggingFaceTB/cosmopedia: 22,165 dl, 755 ★; 30M+ doc; tag 'synthetic'
+  subseturi (cu n_examples): auto_math_text 1.95M, khanacademy 24k, openstax 126k, stanford 1.02M,
+  stories 5.0M, web_samples_v1 12.4M, web_samples_v2 ..., wikihow ...
+  format parquet; câmpuri: prompt, text_token_length, text, seed_data, format, audience
+- HuggingFaceTB/smoltalk: 46,692 dl, 443 ★ (SFT SmolLM2)
+- nvidia/OpenMathInstruct-1: 5,305 dl (math CoT sintetic, 1M+)
+- microsoft/orca-math-word-problems-200k: 22,325 dl (Orca reasoning distillation)
+- Magpie-Align/Magpie-Pro-300K-Filtered: 4,271 dl, 56 ★; Magpie-Air-300K-Filtered; Llama-3-Magpie-Pro-1M-v0.1 (518 dl)
+- Open-Orca/OpenOrca: 11,833 dl, 1,607 ★
+- cognitivecomputations/dolphin: configs flan1m-alpaca-uncensored (GPT-4), flan5m-alpaca-uncensored (GPT-3.5)
+- roneneldan/TinyStories: 111,149 dl, 1,188 ★
+
+## Zephyr-7b-beta (README): DPO hyperparemti REALI
+- lr=5e-7, train_batch=2, num_devices=16 (total 32), 3 epoci, warmup_ratio 0.1, linear, AdamW eps 1e-8
+- DPO final: loss 0.7496, rewards/acc 0.7812, margins 3.7963
+- SFT pe UltraChat-200k, apoi DPO pe UltraFeedback (distilat GPT-4)
+
+## Dolphin (erichartford.com/dolphin + dataset card)
+- Replicare Orca: ~1M FLANv2 + GPT-4 completions; ~3.5M FLANv2 + GPT-3.5
+- Inclus toate 75k CoT din FLAN-1m; dedup; FILTRAT refuz/evitare/alignment/bias → uncensored
+- Dolphin-2.9-llama3-8b: FFT full-params, ChatML, seq 4k, 2.5 zile pe 8×L40S
+- surse dataset: Dolphin-2.9, OpenHermes-2.5, CodeFeedback, dolphin-coder, samantha-data,
+  ultrachat_200k, orca-math-word-problems-200k, SystemChat-1.1, function-calling-chatml, Agent-FLAN
+
+## Magpie (repo + paper ICLR 2025, arXiv 2406.08464)
+- generează query+response promptând LLM aliniat DOAR cu pre-query template (fără seed questions)
+- Magpie-Reasoning V2: 250K, cu rationale step-by-step
+
+## NanoGPT pe 2×T4 (GitHub issue #539, răspuns real)
+- torchrun --standalone --nproc_per_node=2 train.py config/train_gpt2.py  (DDP standard)
+
+## litgpt issue #2223 (Colab/T4)
+- utilizator raportează rularea pretrain cu precision bf16-mixed pe T4 în Colab
+
+## Fapt hardware T4 (Turing TU104) — cunoștințe stabile
+- 16GB GDDR6, 320 GB/s, 40 SM, 2560 CUDA, 320 tensor cores
+- FP32 8.1 TFLOPS; FP16 tensor 65 TFLOPS; INT8 130 TOPS
+- NU are unități bfloat16 hardware → bf16 rulează emulat (lent); fp16+GradScaler preferat
+- 2×T4 Kaggle = DDP; P2P/NVLink absent → comunicare prin PCIe