Snapkitty
/

hilbert

@@ -1,160 +1,178 @@
-# Hilbert
-[![License: Tri-License](https://img.shields.io/badge/License-BSL%201.1%20%7C%20AGPL%203.0%20%7C%20MPL%202.0-blue.svg)](LICENSE.tri)
-[![CUDA](https://img.shields.io/badge/CUDA-Hopper%20%7C%20Ampere-76B900?logo=nvidia)](kernels/)
-[![Architecture](https://img.shields.io/badge/Params-~4B-orange.svg)](config/architecture.json)
-[![Data](https://img.shields.io/badge/Training%20Data-Public%20Only-green.svg)](training/pretrain_config.yaml)
-[![Status](https://img.shields.io/badge/Status-Pre--Training-yellow.svg)](#status)
-[![Built From Scratch](https://img.shields.io/badge/Built-From%20Scratch-black.svg)](#why)
----
-> **WARNING: This model has NOT been trained yet.** This repository contains the complete architecture specification, production CUDA kernels, GGUF export pipeline, and training configuration. Weights do not exist. Do not attempt to load or deploy — there is no checkpoint to load. Training requires 8x H100 GPUs (or equivalent compute).
----
-Sovereign language model built from scratch. No fine-tune. No LoRA. No dependency on NVIDIA, Meta, Mistral, or any other lab's weights.
-Named after David Hilbert — the mathematician who attempted to formalize all of mathematics.
-## What This Is
-A complete, from-scratch transformer trained exclusively on public internet data. Every component is original:
-- **Architecture**: Standard decoder-only transformer (GQA + SwiGLU + RMSNorm + RoPE)
-- **CUDA Kernels**: Hand-written for H100/A100 tensor cores
-- **Training Data**: Public domain + permissive-licensed sources only
-- **GGUF Export**: Direct deployment via llama.cpp without any vendor lock-in
-## Why
-Fine-tuning is a lease. You modify someone else's weights, you carry their name, you depend on their license, and you can't clearly say what's yours. If NVIDIA changes their license tomorrow, your "fine-tune" is gone.
-Hilbert is built from the ground up so the weights are ours. The name is ours. The architecture is public knowledge that no one can patent. The training data is public. The result is sovereign.
-## Architecture
-| Parameter | Value |
-|-----------|-------|
-| Parameters | ~4B |
-| Hidden Dim | 2048 |
-| Layers | 24 |
-| Attention Heads | 16 (GQA: 4 KV heads) |
-| MLP Dim | 5504 (SwiGLU) |
-| Context | 8192 tokens |
-| Vocab | 32K BPE |
-| Position | RoPE (theta=10000, full head rotation) |
-| Norm | RMSNorm (eps=1e-5) |
-| Training | BF16, AdamW, cosine LR |
-| Deployment | Q4_K_M via GGUF/llama.cpp |
-## Structure
-```
-hilbert/
-├── config/
-│   └── architecture.json       # Full structural specification
-├── kernels/
-│   ├── rmsnorm.cu              # Warp-shuffle RMSNorm
-│   ├── attention_gqa.cu        # FlashAttention-2 GQA
-│   ├── mlp_swiglu.cu           # Fused Gate+Up GEMM with SwiGLU epilogue
-│   └── rope.cu                 # Rotary Position Embedding
-├── gguf/
-│   └── export_gguf.py          # PyTorch → GGUFv3 conversion
-├── training/
-│   └── pretrain_config.yaml    # Full training specification
-└── LICENSE.tri
-```
-## CUDA Kernels
-All kernels target NVIDIA Hopper (H100) and Ampere (A100) tensor cores:
-| Kernel | Technique | Key Feature |
-|--------|-----------|-------------|
-| `rmsnorm.cu` | Warp-shuffle cooperative reduction | FP32 accumulation, single-pass |
-| `attention_gqa.cu` | FlashAttention-2 tiling | Online softmax, GQA head sharing |
-| `mlp_swiglu.cu` | Fused dual-GEMM | Gate+Up computed simultaneously, SiLU in epilogue |
-| `rope.cu` | Per-pair rotation | Supports partial rotation, KV-cache continuation |
-## Training Data
-Public sources only. No proprietary datasets. No scraped content behind ToS walls.
-| Source | Weight | License |
-|--------|--------|---------|
-| RedPajama v2 (CC dedupe) | 40% | Public domain |
-| arXiv (math/physics/cs) | 15% | CC-BY |
-| GitHub (permissive only) | 15% | MIT/Apache/BSD |
-| Stack Exchange | 10% | CC-BY-SA |
-| Wikipedia (all langs) | 10% | CC-BY-SA |
-| Public domain books | 5% | Pre-1928 |
-| Formal proofs (Lean4/Agda/Coq) | 5% | Permissive OSS |
-## Deployment
-```bash
-# Export trained checkpoint to GGUF
-python gguf/export_gguf.py \
-    --checkpoint checkpoints/hilbert-4b-final.pt \
-    --config config/architecture.json \
-    --output hilbert-4b-q4km.gguf
-# Run via llama.cpp
-./llama-server -m hilbert-4b-q4km.gguf -c 8192 --port 8080
-```
-## Legal Clarity
-- Architecture: Standard transformer (public knowledge, no patents applicable)
-- Kernels: Original implementation (written by us)
-- Training data: All public domain or permissive-licensed
-- Weights: Trained from random initialization (no one else's weights)
-- Name: Hilbert (no trademark conflict)
-This model belongs to us. Not as a legal argument — as a mathematical fact.
-## Status
-| Phase | Status | Notes |
-|-------|--------|-------|
-| Architecture Design | COMPLETE | Fully specified, all tensor shapes defined |
-| CUDA Kernels | COMPLETE | RMSNorm, FlashAttn-2 GQA, fused SwiGLU, RoPE |
-| GGUF Pipeline | COMPLETE | Export script ready for trained checkpoint |
-| Training Config | COMPLETE | Data mix, hyperparameters, hardware plan |
-| Tokenizer Training | NOT STARTED | BPE on RedPajama subset |
-| Pre-Training | NOT STARTED | Requires 8x H100 compute |
-| Evaluation | NOT STARTED | Benchmarks TBD post-training |
-| GGUF Release | NOT STARTED | Q4_K_M quantized model for llama.cpp |
----
-## Disclosure
-This is an independent research project. Hilbert is not affiliated with, endorsed by, or derived from any existing model or lab.
-**IP Provenance:**
-- The transformer architecture (attention + FFN + residual) is published academic work (Vaswani et al., 2017) — no entity owns it
-- Grouped-Query Attention is published (Ainslie et al., 2023) — no patent encumbrance
-- SwiGLU is published (Shazeer, 2020) — standard practice
-- RoPE is published (Su et al., 2021) — open technique
-- FlashAttention tiling is published (Dao, 2022; Dao, 2023) — algorithm only, our implementation is original
-- All CUDA kernel code in this repository is original work
-- No weights, activations, distillation outputs, or training artifacts from any existing model are used
-- Training data consists exclusively of public domain and permissive-licensed sources
-**What this is NOT:**
-- NOT a fine-tune of Nemotron, LLaMA, Mistral, or any other model
-- NOT a quantized/pruned/distilled version of an existing model
-- NOT using any proprietary training data
-- NOT dependent on any vendor SDK beyond the CUDA compiler
-**Naming:** "Hilbert" refers to David Hilbert (1862-1943), mathematician. No trademark conflict exists with any AI product or service.
----
-## License
-Tri-licensed: BSL-1.1 + AGPL-3.0 + MPL-2.0. See [LICENSE.tri](LICENSE.tri).
-Copyright (C) 2026 Jessica L. Williams / SNAPKITTYWEST

+---
+license: agpl-3.0
+library_name: transformers
+tags:
+  - cuda
+  - kernels
+  - flash-attention
+  - sovereign-ai
+  - from-scratch
+  - ampere
+language:
+  - en
+pipeline_tag: text-generation
+model-index:
+  - name: hilbert-4b
+    results: []
+---
+# Hilbert
+[![License: Tri-License](https://img.shields.io/badge/License-BSL%201.1%20%7C%20AGPL%203.0%20%7C%20MPL%202.0-blue.svg)](LICENSE.tri)
+[![CUDA](https://img.shields.io/badge/CUDA-Hopper%20%7C%20Ampere-76B900?logo=nvidia)](kernels/)
+[![Architecture](https://img.shields.io/badge/Params-~4B-orange.svg)](config/architecture.json)
+[![Data](https://img.shields.io/badge/Training%20Data-Public%20Only-green.svg)](training/pretrain_config.yaml)
+[![Status](https://img.shields.io/badge/Status-Pre--Training-yellow.svg)](#status)
+[![Built From Scratch](https://img.shields.io/badge/Built-From%20Scratch-black.svg)](#why)
+---
+> **WARNING: This model has NOT been trained yet.** This repository contains the complete architecture specification, production CUDA kernels, GGUF export pipeline, and training configuration. Weights do not exist. Do not attempt to load or deploy — there is no checkpoint to load. Training requires 8x H100 GPUs (or equivalent compute).
+---
+Sovereign language model built from scratch. No fine-tune. No LoRA. No dependency on NVIDIA, Meta, Mistral, or any other lab's weights.
+Named after David Hilbert — the mathematician who attempted to formalize all of mathematics.
+## What This Is
+A complete, from-scratch transformer trained exclusively on public internet data. Every component is original:
+- **Architecture**: Standard decoder-only transformer (GQA + SwiGLU + RMSNorm + RoPE)
+- **CUDA Kernels**: Hand-written for H100/A100 tensor cores
+- **Training Data**: Public domain + permissive-licensed sources only
+- **GGUF Export**: Direct deployment via llama.cpp without any vendor lock-in
+## Why
+Fine-tuning is a lease. You modify someone else's weights, you carry their name, you depend on their license, and you can't clearly say what's yours. If NVIDIA changes their license tomorrow, your "fine-tune" is gone.
+Hilbert is built from the ground up so the weights are ours. The name is ours. The architecture is public knowledge that no one can patent. The training data is public. The result is sovereign.
+## Architecture
+| Parameter | Value |
+|-----------|-------|
+| Parameters | ~4B |
+| Hidden Dim | 2048 |
+| Layers | 24 |
+| Attention Heads | 16 (GQA: 4 KV heads) |
+| MLP Dim | 5504 (SwiGLU) |
+| Context | 8192 tokens |
+| Vocab | 32K BPE |
+| Position | RoPE (theta=10000, full head rotation) |
+| Norm | RMSNorm (eps=1e-5) |
+| Training | BF16, AdamW, cosine LR |
+| Deployment | Q4_K_M via GGUF/llama.cpp |
+## Structure
+```
+hilbert/
+├── config/
+│   └── architecture.json       # Full structural specification
+├── kernels/
+│   ├── rmsnorm.cu              # Warp-shuffle RMSNorm
+│   ├── attention_gqa.cu        # FlashAttention-2 GQA
+│   ├── mlp_swiglu.cu           # Fused Gate+Up GEMM with SwiGLU epilogue
+│   └── rope.cu                 # Rotary Position Embedding
+├���─ gguf/
+│   └── export_gguf.py          # PyTorch → GGUFv3 conversion
+├── training/
+│   └── pretrain_config.yaml    # Full training specification
+└── LICENSE.tri
+```
+## CUDA Kernels
+All kernels target NVIDIA Hopper (H100) and Ampere (A100) tensor cores:
+| Kernel | Technique | Key Feature |
+|--------|-----------|-------------|
+| `rmsnorm.cu` | Warp-shuffle cooperative reduction | FP32 accumulation, single-pass |
+| `attention_gqa.cu` | FlashAttention-2 tiling | Online softmax, GQA head sharing |
+| `mlp_swiglu.cu` | Fused dual-GEMM | Gate+Up computed simultaneously, SiLU in epilogue |
+| `rope.cu` | Per-pair rotation | Supports partial rotation, KV-cache continuation |
+## Training Data
+Public sources only. No proprietary datasets. No scraped content behind ToS walls.
+| Source | Weight | License |
+|--------|--------|---------|
+| RedPajama v2 (CC dedupe) | 40% | Public domain |
+| arXiv (math/physics/cs) | 15% | CC-BY |
+| GitHub (permissive only) | 15% | MIT/Apache/BSD |
+| Stack Exchange | 10% | CC-BY-SA |
+| Wikipedia (all langs) | 10% | CC-BY-SA |
+| Public domain books | 5% | Pre-1928 |
+| Formal proofs (Lean4/Agda/Coq) | 5% | Permissive OSS |
+## Deployment
+```bash
+# Export trained checkpoint to GGUF
+python gguf/export_gguf.py \
+    --checkpoint checkpoints/hilbert-4b-final.pt \
+    --config config/architecture.json \
+    --output hilbert-4b-q4km.gguf
+# Run via llama.cpp
+./llama-server -m hilbert-4b-q4km.gguf -c 8192 --port 8080
+```
+## Legal Clarity
+- Architecture: Standard transformer (public knowledge, no patents applicable)
+- Kernels: Original implementation (written by us)
+- Training data: All public domain or permissive-licensed
+- Weights: Trained from random initialization (no one else's weights)
+- Name: Hilbert (no trademark conflict)
+This model belongs to us. Not as a legal argument — as a mathematical fact.
+## Status
+| Phase | Status | Notes |
+|-------|--------|-------|
+| Architecture Design | COMPLETE | Fully specified, all tensor shapes defined |
+| CUDA Kernels | COMPLETE | RMSNorm, FlashAttn-2 GQA, fused SwiGLU, RoPE |
+| GGUF Pipeline | COMPLETE | Export script ready for trained checkpoint |
+| Training Config | COMPLETE | Data mix, hyperparameters, hardware plan |
+| Tokenizer Training | NOT STARTED | BPE on RedPajama subset |
+| Pre-Training | NOT STARTED | Requires 8x H100 compute |
+| Evaluation | NOT STARTED | Benchmarks TBD post-training |
+| GGUF Release | NOT STARTED | Q4_K_M quantized model for llama.cpp |
+---
+## Disclosure
+This is an independent research project. Hilbert is not affiliated with, endorsed by, or derived from any existing model or lab.
+**IP Provenance:**
+- The transformer architecture (attention + FFN + residual) is published academic work (Vaswani et al., 2017) — no entity owns it
+- Grouped-Query Attention is published (Ainslie et al., 2023) — no patent encumbrance
+- SwiGLU is published (Shazeer, 2020) — standard practice
+- RoPE is published (Su et al., 2021) — open technique
+- FlashAttention tiling is published (Dao, 2022; Dao, 2023) — algorithm only, our implementation is original
+- All CUDA kernel code in this repository is original work
+- No weights, activations, distillation outputs, or training artifacts from any existing model are used
+- Training data consists exclusively of public domain and permissive-licensed sources
+**What this is NOT:**
+- NOT a fine-tune of Nemotron, LLaMA, Mistral, or any other model
+- NOT a quantized/pruned/distilled version of an existing model
+- NOT using any proprietary training data
+- NOT dependent on any vendor SDK beyond the CUDA compiler
+**Naming:** "Hilbert" refers to David Hilbert (1862-1943), mathematician. No trademark conflict exists with any AI product or service.
+---
+## License
+Tri-licensed: BSL-1.1 + AGPL-3.0 + MPL-2.0. See [LICENSE.tri](LICENSE.tri).
+Copyright (C) 2026 Jessica L. Williams / SNAPKITTYWEST