README.md

---
license: other
license_name: busl-1.1
license_link: https://huggingface.co/Snapkitty/snapkitty-transformer/blob/main/LICENSE
library_name: custom
tags:
- code
- sovereign-compute
- snapkitty
- machine-learning
- python
- asm
- datalog
---

# SnapKitty Transformer 8B

[![License: BSL-1.1](https://img.shields.io/badge/License-BSL--1.1-ff6b35.svg)](https://github.com/SNAPKITTYWEST/snapkitty-transformer/blob/main/LICENSE)
[![License: AGPL--3.0](https://img.shields.io/badge/License-AGPL--3.0-red.svg)](https://github.com/SNAPKITTYWEST/snapkitty-transformer/blob/main/LICENSE-AGPL)
[![CUDA](https://img.shields.io/badge/CUDA-12.x-76B900.svg)](https://developer.nvidia.com/cuda-toolkit)
[![Python](https://img.shields.io/badge/Python-3.10+-3776AB.svg)](https://www.python.org/)
[![PyTorch](https://img.shields.io/badge/PyTorch-2.x-EE4C2C.svg)](https://pytorch.org/)
[![Sovereign](https://img.shields.io/badge/Sovereign-Node-Key%20Only-black.svg)](https://github.com/SNAPKITTYWEST)

## 8B Parameter Transformer Model

A sovereign corporate 8B parameter transformer model with production-ready FlashAttention CUDA kernels, Triton reference implementations, MoE routing, and YaRN RoPE — optimized for NVIDIA Ampere (sm_80+) and Hopper architectures.

**⚠️ NOT OPEN SOURCE** — This is a corporate sovereign product. Commercial use requires a Sovereign Node Key.

**Topics:** `transformer` `8b` `flashattention` `cuda` `nvidia` `tensor-cores` `attention-mechanism` `language-model` `sovereign` `copyleft` `bsl` `agpl` `sm_80` `hopper` `moe` `triton` `yarn` `mxfp4`

---

## Architecture

### Model Specifications

| Parameter | Value |
|-----------|-------|
| Total Parameters | 8B |
| Hidden Dimension | 8192 |
| Attention Heads | 128 |
| Key-Value Heads | 16 (GQA) |
| Head Dimension | 64 |
| Layers | 128 |
| Vocabulary Size | 256,000 |
| Max Sequence Length | 204,800 |
| Experts | 8 (MoE, top-4 routing) |
| Sliding Window | 65,536 |
| Attention Sinks | 2,048 |
| YaRN RoPE | 32x factor |
| MXFP4 Quantization | Supported |
| Precision | FP16 / BF16 / MXFP4 |

### FlashAttention Kernel

Production-ready fused GEMM + online softmax kernel for NVIDIA Ampere (sm_80+):

- **cp.async**: Asynchronous global-to-shared memory copies
- **mma.sync.aligned.m16n8k16**: Tensor Core matrix multiply-accumulate
- **Online softmax**: Running max/sum accumulators without materializing full attention matrix
- **Double buffering**: Overlapping compute with memory loads
- **Warp-level reductions**: `__shfl_down_sync` for efficient reductions

```cuda
// Kernel Configuration
constexpr int Br = 64;       // Rows of Q per block
constexpr int Bc = 64;       // Rows of K/V per tile
constexpr int d = 64;        // Head dimension
constexpr int num_warps = 4; // 128 threads total
```

---

## Repository Structure

```
snapkitty-transformer/
├── kernels/                          CUDA & Triton kernels
│   ├── flash_attention_fwd.cu        FlashAttention forward (Ampere sm_80+)
│   ├── flash_attention_bwd.cu        FlashAttention backward pass
│   ├── flash_attention_hopper.cu     FlashAttention-3 forward (Hopper, TMA, FP8, cluster MMA)
│   ├── flash_attention_hopper_bwd.cu FlashAttention-3 backward (Hopper, dQ/dK/dV, atomic reduce)
│   ├── flash_attention.cpp           PyTorch C++ extension
│   ├── mxfp4_quant.cu               MXFP4 quant/dequant (E2M1, block scales, MoE layout)
│   ├── attention_kernel.py           Triton FlashAttention (sliding window + sinks)
│   └── moe_kernel.py                 Triton fused MoE (MXFP4, top-4, SwiGLU, TP)
├── gpt_oss/                          Model architecture
│   └── model.py                      GPT-OSS: RMSNorm, YaRN RoPE, SwiGLU, MoE, Attention
├── benchmark.py                      FlashAttention benchmark suite
├── analyze_ncu.py                    NCU log parser & bottleneck analysis
├── flash_attention_triton.py         Triton reference FlashAttention
├── setup.py                          Build config (PyTorch C++ extension)
├── test_flash_attention.py           Forward tests
├── test_hopper_bwd.py                Backward + MXFP4 round-trip tests
├── LICENSE                           Business Source License 1.1
├── LICENSE-AGPL                      GNU AGPL v3.0
└── README.md                         This file
```

---

## Quick Start

### Installation

```bash
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121
pip install transformers datasets accelerate triton
```

### Build CUDA Extension

```bash
cd snapkitty-transformer
python setup.py install
```

### Model Loading

```python
from gpt_oss.model import GPTOSSModel, ModelConfig

config = ModelConfig(
    vocab_size=256000,
    dim=8192,
    n_heads=128,
    n_kv_heads=16,
    n_layers=128,
    max_seq_len=204800,
    num_experts=8,
    top_k=4,
)

model = GPTOSSModel(config)
model.load_state_dict(torch.load("snapkitty-8b-weights.pt"))
model = model.cuda().half()
```

### FlashAttention Inference

```python
from kernels.flash_attention import flash_attention_fwd

# Q, K, V: [batch, heads, seq_len, head_dim]
output = flash_attention_fwd(Q, K, V, causal=True, sliding_window=65536, num_sinks=2048)
```

### Text Generation

```python
input_ids = tokenizer("The sovereign substrate is", return_tensors="pt").input_ids.cuda()
output = model.generate(input_ids, max_new_tokens=512, temperature=0.8)
print(tokenizer.decode(output[0]))
```

---

## Training

### Pre-training Data

- 2T tokens from curated sovereign datasets
- Code: 500B tokens (Python, Rust, CUDA, Solidity)
- Academic: 300B tokens (arXiv, textbooks)
- Web: 1.2T tokens (filtered, deduplicated)

### Training Configuration

```yaml
model:
  vocab_size: 256000
  dim: 8192
  n_layers: 128
  n_heads: 128
  n_kv_heads: 16
  head_dim: 64
  max_seq_len: 204800
  num_experts: 8
  top_k: 4
  expert_intermediate_dim: 2048
  sliding_window: 65536
  num_sinks: 2048
  rope_yarn: true
  rope_yarn_factor: 32.0
  use_mxfp4: true

training:
  batch_size: 2048
  learning_rate: 3e-4
  warmup_steps: 2000
  total_steps: 500000
  optimizer: AdamW
  weight_decay: 0.1
  grad_clip: 1.0
  bf16: true
```

---

## Performance

### Benchmarks (A100 80GB)

| Metric | Value |
|--------|-------|
| Inference (FP16) | 45 tokens/sec |
| Inference (INT8) | 62 tokens/sec |
| Inference (INT4) | 78 tokens/sec |
| Training Throughput | 180 tokens/sec/GPU |
| FlashAttention Speedup | 2.5x vs naive |

### Memory Usage

| Precision | Model Size | KV Cache (8K) |
|-----------|------------|---------------|
| FP16 | 16 GB | 2 GB |
| INT8 | 8 GB | 1 GB |
| INT4 | 4 GB | 0.5 GB |

---

## Protected Inventions

  1. SNAPKITTY TRANSFORMER ARCHITECTURE
     8B parameter transformer with 128 layers, 128 attention heads,
     16 GQA KV heads, 8-expert MoE with top-4 routing, and 204800
     max sequence length.

  2. FLASHATTENTION FUSED GEMM + ONLINE SOFTMAX KERNEL
     Production-ready CUDA kernel with cp.async, mma.sync, and
     double-buffered K/V tile processing.

  3. FLASHATTENTION-3 HOPPER FORWARD KERNEL
     TMA-accelerated, warp-specialized kernel with cluster MMA,
     K-stage pipelining, FP8 computation, causal masking, sliding
     window, attention sinks, and YaRN RoPE integration.

  4. FLASHATTENTION-3 HOPPER BACKWARD KERNEL
     Full dQ/dK/dV computation with S/P recomputation from L/M,
     dO@V^T accumulation, dS=P*(dO@V^T-D), atomic partial reduce
     across Q tiles, and global reduction kernel.

  5. TRITON FLASHATTENTION WITH SLIDING WINDOW + SINKS
     Reference Triton implementation with online softmax, sliding
     window attention, and persistent attention sinks.

  6. TRITON FUSED MoE KERNEL
     Triton Mixture-of-Experts with MXFP4 dequantization, top-4
     routing, SwiGLU activation, and tensor parallelism support.

  7. YARN ROTARY POSITIONAL EMBEDDING
     YaRN-scaled RoPE with 32x factor for 204800-length sequences,
     integrated into both CUDA and Triton kernels.

  8. MXFP4 QUANTIZATION/DEQUANTIZATION
     E2M1 4-bit format with shared block exponent (E8M0), 32-element
     blocks, LUT-based rounding, CPU reference + device kernels,
     PyTorch bindings, and MoE weight layout for kernel consumption.

  9. GPT-OSS MODEL ARCHITECTURE
     Complete PyTorch module with RMSNorm, YaRN RoPE, SwiGLU,
     MXFP4 dequantization, sliding window + sinks attention, MoE
     with top-4 routing, and full generation pipeline.

  8. SOVEREIGN TRAINING DATA PIPELINE
     Proprietary data curation, filtering, and augmentation methods.

  9. INFERENCE OPTIMIZATION SUITE
     Quantization, pruning, and distillation techniques.

---

## License

**⚠️ THIS IS NOT OPEN SOURCE**

This project is a **sovereign corporate product** licensed under **Business Source License 1.1 (BSL-1.1)** with **GNU AGPL v3.0 copyleft** for network services.

### License Structure:

| Component | License | File | Scope |
|-----------|---------|------|-------|
| **Core Model & Kernels** | BSL-1.1 → AGPL v3.0 (2030) | `LICENSE` | Transformer architecture, CUDA kernels, training pipeline |
| **API/Server** | GNU AGPL v3.0 | `LICENSE-AGPL` | REST API, WebSocket server, inference endpoints |

### Sovereign Node Key Required:

- **Commercial use:** Requires Sovereign Node Key from Licensor
- **Network service:** AGPL v3.0 copyleft triggers
- **Derivative works:** Must be licensed under same BSL-1.1 terms
- **No relicensing:** Without explicit written consent from Licensor

### CopyLeft Provision:

Any derivative work based on the Licensed Work must be licensed under the same Business Source License 1.1 terms. No relicensing permitted without explicit written consent from the Licensor.

---

## Citation

```bibtex
@misc{snapkittytransformer2026,
  title={SnapKitty Transformer 8B},
  author={Ahmad Ali Parr and Jessica Westerhoff},
  year={2026},
  note={8B parameter transformer with FlashAttention CUDA kernels},
  publisher={SNAPKITTYWEST},
  howpublished={\url{https://github.com/SNAPKITTYWEST/snapkitty-transformer}},
  keywords={transformer, 8b, flashattention, cuda, nvidia, tensor-cores, sovereign},
  license={BSL-1.1}
}
```

---

## Contact

**Ahmad Ali Parr** - ahmedparr93@gmail.com
**Jessica Westerhoff** - jessicalw34@gmail.com

Bel Esprit d'Accord Trust — 50/50 equal sovereigns

The substrate is not for sale. It is not for porting. It is for Execution in the Wild.