Metadata: discovery tags (collection, languages)

f3494f7 verified 19 days ago

10.5 kB

---
license: other
license_name: busl-1.1
license_link: https://huggingface.co/Snapkitty/snapkitty-transformer/blob/main/LICENSE
library_name: custom
tags:
- code
- sovereign-compute
- snapkitty
- machine-learning
- python
- asm
- datalog
---
# SnapKitty Transformer 8B

[![License: BSL-1.1](https://img.shields.io/badge/License-BSL--1.1-ff6b35.svg)](https://github.com/SNAPKITTYWEST/snapkitty-transformer/blob/main/LICENSE)
[![License: AGPL--3.0](https://img.shields.io/badge/License-AGPL--3.0-red.svg)](https://github.com/SNAPKITTYWEST/snapkitty-transformer/blob/main/LICENSE-AGPL)
[![CUDA](https://img.shields.io/badge/CUDA-12.x-76B900.svg)](https://developer.nvidia.com/cuda-toolkit)
[![Python](https://img.shields.io/badge/Python-3.10+-3776AB.svg)](https://www.python.org/)
[![PyTorch](https://img.shields.io/badge/PyTorch-2.x-EE4C2C.svg)](https://pytorch.org/)
[![Sovereign](https://img.shields.io/badge/Sovereign-Node-Key%20Only-black.svg)](https://github.com/SNAPKITTYWEST)

## 8B Parameter Transformer Model

A sovereign corporate 8B parameter transformer model with production-ready FlashAttention CUDA kernels, Triton reference implementations, MoE routing, and YaRN RoPE — optimized for NVIDIA Ampere (sm_80+) and Hopper architectures.
**⚠️ NOT OPEN SOURCE** — This is a corporate sovereign product. Commercial use requires a Sovereign Node Key.
**Topics:** `transformer` `8b` `flashattention` `cuda` `nvidia` `tensor-cores` `attention-mechanism` `language-model` `sovereign` `copyleft` `bsl` `agpl` `sm_80` `hopper` `moe` `triton` `yarn` `mxfp4`

---

## Architecture

### Model Specifications

| Parameter | Value |
|-----------|-------|
| Total Parameters | 8B |
| Hidden Dimension | 8192 |
| Attention Heads | 128 |
| Key-Value Heads | 16 (GQA) |
| Head Dimension | 64 |
| Layers | 128 |
| Vocabulary Size | 256,000 |
| Max Sequence Length | 204,800 |
| Experts | 8 (MoE, top-4 routing) |
| Sliding Window | 65,536 |
| Attention Sinks | 2,048 |
| YaRN RoPE | 32x factor |
| MXFP4 Quantization | Supported |
| Precision | FP16 / BF16 / MXFP4 |

### FlashAttention Kernel

Production-ready fused GEMM + online softmax kernel for NVIDIA Ampere (sm_80+):
- **cp.async**: Asynchronous global-to-shared memory copies
- **mma.sync.aligned.m16n8k16**: Tensor Core matrix multiply-accumulate
- **Online softmax**: Running max/sum accumulators without materializing full attention matrix
- **Double buffering**: Overlapping compute with memory loads
- **Warp-level reductions**: `__shfl_down_sync` for efficient reductions
```cuda
// Kernel Configuration
constexpr int Br = 64;       // Rows of Q per block
constexpr int Bc = 64;       // Rows of K/V per tile
constexpr int d = 64;        // Head dimension
constexpr int num_warps = 4; // 128 threads total
```
---
## Repository Structure
```
snapkitty-transformer/
├── kernels/                          CUDA & Triton kernels
│   ├── flash_attention_fwd.cu        FlashAttention forward (Ampere sm_80+)
│   ├── flash_attention_bwd.cu        FlashAttention backward pass
│   ├── flash_attention_hopper.cu     FlashAttention-3 forward (Hopper, TMA, FP8, cluster MMA)
│   ├── flash_attention_hopper_bwd.cu FlashAttention-3 backward (Hopper, dQ/dK/dV, atomic reduce)
│   ├── flash_attention.cpp           PyTorch C++ extension
│   ├── mxfp4_quant.cu               MXFP4 quant/dequant (E2M1, block scales, MoE layout)
│   ├── attention_kernel.py           Triton FlashAttention (sliding window + sinks)
│   └── moe_kernel.py                 Triton fused MoE (MXFP4, top-4, SwiGLU, TP)
├── gpt_oss/                          Model architecture
│   └── model.py                      GPT-OSS: RMSNorm, YaRN RoPE, SwiGLU, MoE, Attention
├── benchmark.py                      FlashAttention benchmark suite
├── analyze_ncu.py                    NCU log parser & bottleneck analysis
├── flash_attention_triton.py         Triton reference FlashAttention
├── setup.py                          Build config (PyTorch C++ extension)
├── test_flash_attention.py           Forward tests
├── test_hopper_bwd.py                Backward + MXFP4 round-trip tests
├── LICENSE                           Business Source License 1.1
├── LICENSE-AGPL                      GNU AGPL v3.0
└── README.md                         This file
```
---
## Quick Start
### Installation
```bash
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121
pip install transformers datasets accelerate triton
```
### Build CUDA Extension
```bash
cd snapkitty-transformer
python setup.py install
```
### Model Loading
```python
from gpt_oss.model import GPTOSSModel, ModelConfig
config = ModelConfig(
    vocab_size=256000,
    dim=8192,
    n_heads=128,
    n_kv_heads=16,
    n_layers=128,
    max_seq_len=204800,
    num_experts=8,
    top_k=4,
)
model = GPTOSSModel(config)
model.load_state_dict(torch.load("snapkitty-8b-weights.pt"))
model = model.cuda().half()
```
### FlashAttention Inference
```python
from kernels.flash_attention import flash_attention_fwd
# Q, K, V: [batch, heads, seq_len, head_dim]
output = flash_attention_fwd(Q, K, V, causal=True, sliding_window=65536, num_sinks=2048)
```
### Text Generation
```python
input_ids = tokenizer("The sovereign substrate is", return_tensors="pt").input_ids.cuda()
output = model.generate(input_ids, max_new_tokens=512, temperature=0.8)
print(tokenizer.decode(output[0]))
```
---
## Training
### Pre-training Data
- 2T tokens from curated sovereign datasets
- Code: 500B tokens (Python, Rust, CUDA, Solidity)
- Academic: 300B tokens (arXiv, textbooks)
- Web: 1.2T tokens (filtered, deduplicated)
### Training Configuration
```yaml
model:
  vocab_size: 256000
  dim: 8192
  n_layers: 128
  n_heads: 128
  n_kv_heads: 16
  head_dim: 64
  max_seq_len: 204800
  num_experts: 8
  top_k: 4
  expert_intermediate_dim: 2048
  sliding_window: 65536
  num_sinks: 2048
  rope_yarn: true
  rope_yarn_factor: 32.0
  use_mxfp4: true
training:
  batch_size: 2048
  learning_rate: 3e-4
  warmup_steps: 2000
  total_steps: 500000
  optimizer: AdamW
  weight_decay: 0.1
  grad_clip: 1.0
  bf16: true
```
---
## Performance
### Benchmarks (A100 80GB)
| Metric | Value |
|--------|-------|
| Inference (FP16) | 45 tokens/sec |
| Inference (INT8) | 62 tokens/sec |
| Inference (INT4) | 78 tokens/sec |
| Training Throughput | 180 tokens/sec/GPU |
| FlashAttention Speedup | 2.5x vs naive |
### Memory Usage
| Precision | Model Size | KV Cache (8K) |
|-----------|------------|---------------|
| FP16 | 16 GB | 2 GB |
| INT8 | 8 GB | 1 GB |
| INT4 | 4 GB | 0.5 GB |
---
## Protected Inventions
  1. SNAPKITTY TRANSFORMER ARCHITECTURE
     8B parameter transformer with 128 layers, 128 attention heads,
     16 GQA KV heads, 8-expert MoE with top-4 routing, and 204800
     max sequence length.
  2. FLASHATTENTION FUSED GEMM + ONLINE SOFTMAX KERNEL
     Production-ready CUDA kernel with cp.async, mma.sync, and
     double-buffered K/V tile processing.
  3. FLASHATTENTION-3 HOPPER FORWARD KERNEL
     TMA-accelerated, warp-specialized kernel with cluster MMA,
     K-stage pipelining, FP8 computation, causal masking, sliding
     window, attention sinks, and YaRN RoPE integration.
  4. FLASHATTENTION-3 HOPPER BACKWARD KERNEL
     Full dQ/dK/dV computation with S/P recomputation from L/M,
     dO@V^T accumulation, dS=P*(dO@V^T-D), atomic partial reduce
     across Q tiles, and global reduction kernel.
  5. TRITON FLASHATTENTION WITH SLIDING WINDOW + SINKS
     Reference Triton implementation with online softmax, sliding
     window attention, and persistent attention sinks.
  6. TRITON FUSED MoE KERNEL
     Triton Mixture-of-Experts with MXFP4 dequantization, top-4
     routing, SwiGLU activation, and tensor parallelism support.
  7. YARN ROTARY POSITIONAL EMBEDDING
     YaRN-scaled RoPE with 32x factor for 204800-length sequences,
     integrated into both CUDA and Triton kernels.
  8. MXFP4 QUANTIZATION/DEQUANTIZATION
     E2M1 4-bit format with shared block exponent (E8M0), 32-element
     blocks, LUT-based rounding, CPU reference + device kernels,
     PyTorch bindings, and MoE weight layout for kernel consumption.
  9. GPT-OSS MODEL ARCHITECTURE
     Complete PyTorch module with RMSNorm, YaRN RoPE, SwiGLU,
     MXFP4 dequantization, sliding window + sinks attention, MoE
     with top-4 routing, and full generation pipeline.
  8. SOVEREIGN TRAINING DATA PIPELINE
     Proprietary data curation, filtering, and augmentation methods.
  9. INFERENCE OPTIMIZATION SUITE
     Quantization, pruning, and distillation techniques.
---
## License
**⚠️ THIS IS NOT OPEN SOURCE**
This project is a **sovereign corporate product** licensed under **Business Source License 1.1 (BSL-1.1)** with **GNU AGPL v3.0 copyleft** for network services.
### License Structure:
| Component | License | File | Scope |
|-----------|---------|------|-------|
| **Core Model & Kernels** | BSL-1.1 → AGPL v3.0 (2030) | `LICENSE` | Transformer architecture, CUDA kernels, training pipeline |
| **API/Server** | GNU AGPL v3.0 | `LICENSE-AGPL` | REST API, WebSocket server, inference endpoints |
### Sovereign Node Key Required:
- **Commercial use:** Requires Sovereign Node Key from Licensor
- **Network service:** AGPL v3.0 copyleft triggers
- **Derivative works:** Must be licensed under same BSL-1.1 terms
- **No relicensing:** Without explicit written consent from Licensor
### CopyLeft Provision:
Any derivative work based on the Licensed Work must be licensed under the same Business Source License 1.1 terms. No relicensing permitted without explicit written consent from the Licensor.
---
## Citation
```bibtex
@misc{snapkittytransformer2026,
  title={SnapKitty Transformer 8B},
  author={Ahmad Ali Parr and Jessica Westerhoff},
  year={2026},
  note={8B parameter transformer with FlashAttention CUDA kernels},
  publisher={SNAPKITTYWEST},
  howpublished={\url{https://github.com/SNAPKITTYWEST/snapkitty-transformer}},
  keywords={transformer, 8b, flashattention, cuda, nvidia, tensor-cores, sovereign},
  license={BSL-1.1}
}
```
---
## Contact
**Ahmad Ali Parr** - ahmedparr93@gmail.com
**Jessica Westerhoff** - jessicalw34@gmail.com
Bel Esprit d'Accord Trust — 50/50 equal sovereigns
The substrate is not for sale. It is not for porting. It is for Execution in the Wild.

	---
	license: other
	license_name: busl-1.1
	license_link: https://huggingface.co/Snapkitty/snapkitty-transformer/blob/main/LICENSE
	library_name: custom
	tags:
	- code
	- sovereign-compute
	- snapkitty
	- machine-learning
	- python
	- asm
	- datalog
	---

	# SnapKitty Transformer 8B

	[![License: BSL-1.1](https://img.shields.io/badge/License-BSL--1.1-ff6b35.svg)](https://github.com/SNAPKITTYWEST/snapkitty-transformer/blob/main/LICENSE)
	[![License: AGPL--3.0](https://img.shields.io/badge/License-AGPL--3.0-red.svg)](https://github.com/SNAPKITTYWEST/snapkitty-transformer/blob/main/LICENSE-AGPL)
	[![CUDA](https://img.shields.io/badge/CUDA-12.x-76B900.svg)](https://developer.nvidia.com/cuda-toolkit)
	[![Python](https://img.shields.io/badge/Python-3.10+-3776AB.svg)](https://www.python.org/)
	[![PyTorch](https://img.shields.io/badge/PyTorch-2.x-EE4C2C.svg)](https://pytorch.org/)
	[![Sovereign](https://img.shields.io/badge/Sovereign-Node-Key%20Only-black.svg)](https://github.com/SNAPKITTYWEST)

	## 8B Parameter Transformer Model

	A sovereign corporate 8B parameter transformer model with production-ready FlashAttention CUDA kernels, Triton reference implementations, MoE routing, and YaRN RoPE — optimized for NVIDIA Ampere (sm_80+) and Hopper architectures.

	⚠️ NOT OPEN SOURCE — This is a corporate sovereign product. Commercial use requires a Sovereign Node Key.

	Topics: `transformer` `8b` `flashattention` `cuda` `nvidia` `tensor-cores` `attention-mechanism` `language-model` `sovereign` `copyleft` `bsl` `agpl` `sm_80` `hopper` `moe` `triton` `yarn` `mxfp4`

	---

	## Architecture

	### Model Specifications

	\| Parameter \| Value \|
	\|-----------\|-------\|
	\| Total Parameters \| 8B \|
	\| Hidden Dimension \| 8192 \|
	\| Attention Heads \| 128 \|
	\| Key-Value Heads \| 16 (GQA) \|
	\| Head Dimension \| 64 \|
	\| Layers \| 128 \|
	\| Vocabulary Size \| 256,000 \|
	\| Max Sequence Length \| 204,800 \|
	\| Experts \| 8 (MoE, top-4 routing) \|
	\| Sliding Window \| 65,536 \|
	\| Attention Sinks \| 2,048 \|
	\| YaRN RoPE \| 32x factor \|
	\| MXFP4 Quantization \| Supported \|
	\| Precision \| FP16 / BF16 / MXFP4 \|

	### FlashAttention Kernel

	Production-ready fused GEMM + online softmax kernel for NVIDIA Ampere (sm_80+):

	- cp.async: Asynchronous global-to-shared memory copies
	- mma.sync.aligned.m16n8k16: Tensor Core matrix multiply-accumulate
	- Online softmax: Running max/sum accumulators without materializing full attention matrix
	- Double buffering: Overlapping compute with memory loads
	- Warp-level reductions: `__shfl_down_sync` for efficient reductions

	```cuda
	// Kernel Configuration
	constexpr int Br = 64; // Rows of Q per block
	constexpr int Bc = 64; // Rows of K/V per tile
	constexpr int d = 64; // Head dimension
	constexpr int num_warps = 4; // 128 threads total
	```

	---

	## Repository Structure

	```
	snapkitty-transformer/
	├── kernels/ CUDA & Triton kernels
	│ ├── flash_attention_fwd.cu FlashAttention forward (Ampere sm_80+)
	│ ├── flash_attention_bwd.cu FlashAttention backward pass
	│ ├── flash_attention_hopper.cu FlashAttention-3 forward (Hopper, TMA, FP8, cluster MMA)
	│ ├── flash_attention_hopper_bwd.cu FlashAttention-3 backward (Hopper, dQ/dK/dV, atomic reduce)
	│ ├── flash_attention.cpp PyTorch C++ extension
	│ ├── mxfp4_quant.cu MXFP4 quant/dequant (E2M1, block scales, MoE layout)
	│ ├── attention_kernel.py Triton FlashAttention (sliding window + sinks)
	│ └── moe_kernel.py Triton fused MoE (MXFP4, top-4, SwiGLU, TP)
	├── gpt_oss/ Model architecture
	│ └── model.py GPT-OSS: RMSNorm, YaRN RoPE, SwiGLU, MoE, Attention
	├── benchmark.py FlashAttention benchmark suite
	├── analyze_ncu.py NCU log parser & bottleneck analysis
	├── flash_attention_triton.py Triton reference FlashAttention
	├── setup.py Build config (PyTorch C++ extension)
	├── test_flash_attention.py Forward tests
	├── test_hopper_bwd.py Backward + MXFP4 round-trip tests
	├── LICENSE Business Source License 1.1
	├── LICENSE-AGPL GNU AGPL v3.0
	└── README.md This file
	```

	---

	## Quick Start

	### Installation

	```bash
	pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121
	pip install transformers datasets accelerate triton
	```

	### Build CUDA Extension

	```bash
	cd snapkitty-transformer
	python setup.py install
	```

	### Model Loading

	```python
	from gpt_oss.model import GPTOSSModel, ModelConfig

	config = ModelConfig(
	vocab_size=256000,
	dim=8192,
	n_heads=128,
	n_kv_heads=16,
	n_layers=128,
	max_seq_len=204800,
	num_experts=8,
	top_k=4,
	)

	model = GPTOSSModel(config)
	model.load_state_dict(torch.load("snapkitty-8b-weights.pt"))
	model = model.cuda().half()
	```

	### FlashAttention Inference

	```python
	from kernels.flash_attention import flash_attention_fwd

	# Q, K, V: [batch, heads, seq_len, head_dim]
	output = flash_attention_fwd(Q, K, V, causal=True, sliding_window=65536, num_sinks=2048)
	```

	### Text Generation

	```python
	input_ids = tokenizer("The sovereign substrate is", return_tensors="pt").input_ids.cuda()
	output = model.generate(input_ids, max_new_tokens=512, temperature=0.8)
	print(tokenizer.decode(output[0]))
	```

	---

	## Training

	### Pre-training Data

	- 2T tokens from curated sovereign datasets
	- Code: 500B tokens (Python, Rust, CUDA, Solidity)
	- Academic: 300B tokens (arXiv, textbooks)
	- Web: 1.2T tokens (filtered, deduplicated)

	### Training Configuration

	```yaml
	model:
	vocab_size: 256000
	dim: 8192
	n_layers: 128
	n_heads: 128
	n_kv_heads: 16
	head_dim: 64
	max_seq_len: 204800
	num_experts: 8
	top_k: 4
	expert_intermediate_dim: 2048
	sliding_window: 65536
	num_sinks: 2048
	rope_yarn: true
	rope_yarn_factor: 32.0
	use_mxfp4: true

	training:
	batch_size: 2048
	learning_rate: 3e-4
	warmup_steps: 2000
	total_steps: 500000
	optimizer: AdamW
	weight_decay: 0.1
	grad_clip: 1.0
	bf16: true
	```

	---

	## Performance

	### Benchmarks (A100 80GB)

	\| Metric \| Value \|
	\|--------\|-------\|
	\| Inference (FP16) \| 45 tokens/sec \|
	\| Inference (INT8) \| 62 tokens/sec \|
	\| Inference (INT4) \| 78 tokens/sec \|
	\| Training Throughput \| 180 tokens/sec/GPU \|
	\| FlashAttention Speedup \| 2.5x vs naive \|

	### Memory Usage

	\| Precision \| Model Size \| KV Cache (8K) \|
	\|-----------\|------------\|---------------\|
	\| FP16 \| 16 GB \| 2 GB \|
	\| INT8 \| 8 GB \| 1 GB \|
	\| INT4 \| 4 GB \| 0.5 GB \|

	---

	## Protected Inventions

	1. SNAPKITTY TRANSFORMER ARCHITECTURE
	8B parameter transformer with 128 layers, 128 attention heads,
	16 GQA KV heads, 8-expert MoE with top-4 routing, and 204800
	max sequence length.

	2. FLASHATTENTION FUSED GEMM + ONLINE SOFTMAX KERNEL
	Production-ready CUDA kernel with cp.async, mma.sync, and
	double-buffered K/V tile processing.

	3. FLASHATTENTION-3 HOPPER FORWARD KERNEL
	TMA-accelerated, warp-specialized kernel with cluster MMA,
	K-stage pipelining, FP8 computation, causal masking, sliding
	window, attention sinks, and YaRN RoPE integration.

	4. FLASHATTENTION-3 HOPPER BACKWARD KERNEL
	Full dQ/dK/dV computation with S/P recomputation from L/M,
	dO@V^T accumulation, dS=P*(dO@V^T-D), atomic partial reduce
	across Q tiles, and global reduction kernel.

	5. TRITON FLASHATTENTION WITH SLIDING WINDOW + SINKS
	Reference Triton implementation with online softmax, sliding
	window attention, and persistent attention sinks.

	6. TRITON FUSED MoE KERNEL
	Triton Mixture-of-Experts with MXFP4 dequantization, top-4
	routing, SwiGLU activation, and tensor parallelism support.

	7. YARN ROTARY POSITIONAL EMBEDDING
	YaRN-scaled RoPE with 32x factor for 204800-length sequences,
	integrated into both CUDA and Triton kernels.

	8. MXFP4 QUANTIZATION/DEQUANTIZATION
	E2M1 4-bit format with shared block exponent (E8M0), 32-element
	blocks, LUT-based rounding, CPU reference + device kernels,
	PyTorch bindings, and MoE weight layout for kernel consumption.

	9. GPT-OSS MODEL ARCHITECTURE
	Complete PyTorch module with RMSNorm, YaRN RoPE, SwiGLU,
	MXFP4 dequantization, sliding window + sinks attention, MoE
	with top-4 routing, and full generation pipeline.

	8. SOVEREIGN TRAINING DATA PIPELINE
	Proprietary data curation, filtering, and augmentation methods.

	9. INFERENCE OPTIMIZATION SUITE
	Quantization, pruning, and distillation techniques.

	---

	## License

	⚠️ THIS IS NOT OPEN SOURCE

	This project is a sovereign corporate product licensed under Business Source License 1.1 (BSL-1.1) with GNU AGPL v3.0 copyleft for network services.

	### License Structure:

	\| Component \| License \| File \| Scope \|
	\|-----------\|---------\|------\|-------\|
	\| Core Model & Kernels \| BSL-1.1 → AGPL v3.0 (2030) \| `LICENSE` \| Transformer architecture, CUDA kernels, training pipeline \|
	\| API/Server \| GNU AGPL v3.0 \| `LICENSE-AGPL` \| REST API, WebSocket server, inference endpoints \|

	### Sovereign Node Key Required:

	- Commercial use: Requires Sovereign Node Key from Licensor
	- Network service: AGPL v3.0 copyleft triggers
	- Derivative works: Must be licensed under same BSL-1.1 terms
	- No relicensing: Without explicit written consent from Licensor

	### CopyLeft Provision:

	Any derivative work based on the Licensed Work must be licensed under the same Business Source License 1.1 terms. No relicensing permitted without explicit written consent from the Licensor.

	---

	## Citation

	```bibtex
	@misc{snapkittytransformer2026,
	title={SnapKitty Transformer 8B},
	author={Ahmad Ali Parr and Jessica Westerhoff},
	year={2026},
	note={8B parameter transformer with FlashAttention CUDA kernels},
	publisher={SNAPKITTYWEST},
	howpublished={\url{https://github.com/SNAPKITTYWEST/snapkitty-transformer}},
	keywords={transformer, 8b, flashattention, cuda, nvidia, tensor-cores, sovereign},
	license={BSL-1.1}
	}
	```

	---

	## Contact

	Ahmad Ali Parr - ahmedparr93@gmail.com
	Jessica Westerhoff - jessicalw34@gmail.com

	Bel Esprit d'Accord Trust — 50/50 equal sovereigns

	The substrate is not for sale. It is not for porting. It is for Execution in the Wild.