| ---
|
| language:
|
| - en
|
| pipeline_tag: text-generation
|
| library_name: pytorch
|
| license: apache-2.0
|
| tags:
|
| - sparse-ast
|
| - blender-python
|
| - code-generation
|
| - safetensors
|
| ---
|
|
|
| # Sparse-AST-BWM — latest checkpoint, step 93,213
|
|
|
| This repository contains one current checkpoint of the serial old/new Sparse-AST family for byte-level Blender Python completion. It was recovered from `results (4).zip`, the newest relevant training-results archive found in the local Downloads search, and its saved training state confirms step **93,213**. It continues from step 90,213 through 3,000 successful updates, with no recorded non-finite skips.
|
|
|
| Updated 16 October 2026. The current release is under **`latest/`** and uses SafeTensors for model weights and training-state tensors. That release contains no `.pt` checkpoints. Existing remote variant folders are retained as historical artifacts; they are not the latest checkpoint. The pre-update legacy code remains available at revision `7c1001debae58e03d2a3e7073819de4c50ca0cfe`.
|
|
|
| ## Files in `latest/`
|
|
|
| | File | Purpose |
|
| |---|---|
|
| | `model.safetensors` | Exact FP32 model weights for step 93,213 |
|
| | `config.json` | Actual tensor-derived architecture and training-context information |
|
| | `model.py`, `generate.py` | SafeTensors loaders and API/command-line greedy byte completion |
|
| | `family_oldnew_architecture.py`, `test_models.py` | Required custom PyTorch architecture |
|
| | `training_state.safetensors`, `training_state.json` | Optimizer, scaler, RNG states, row order/cursor and training progress for this same checkpoint |
|
| | `training_result.json`, `training_config.json` | Original recorded training results and settings |
|
| | `data_audit.json` | Dataset split/provenance audit from the training archive |
|
| | `validation.json` | Independent conversion checks, inference parity and SHA-256 hashes |
|
|
|
| `training_state.safetensors` is resumable state for the same model version, not a second model checkpoint. Inference needs only the model weights, config and Python architecture files.
|
|
|
| ## Architecture and context
|
|
|
| - 18 serial stages, each retaining its own legacy/new branch pair.
|
| - Width 704; feed-forward width 1,408; eight attention heads.
|
| - 512-entry output vocabulary; inputs and generated text use UTF-8 bytes.
|
| - Stored positional embedding capacity: 512 positions.
|
| - This continuation trained with **128-byte contexts**, not a verified 512-token or 4K context window.
|
| - 201,380,162 model parameters (about 201.38M), stored in FP32.
|
|
|
| Branches are mixed locally at each serial stage. This is not a top-k router over interchangeable stages. Legacy branches remain frozen during training. The actual saved input embedding and output projection differ and are preserved as separate matrices; the loader does not retie or change them.
|
|
|
| ## Load and generate
|
|
|
| Install PyTorch 2.4 or newer, `safetensors` and `huggingface_hub`. This is a custom PyTorch model; use the bundled loader.
|
|
|
| ```python
|
| import sys
|
| from pathlib import Path
|
| from huggingface_hub import snapshot_download
|
|
|
| revision = "main" # Pin a commit hash for reproducible deployments.
|
| folder = snapshot_download(
|
| "Chaman1234/Sparse-AST-BWM",
|
| revision=revision,
|
| allow_patterns=["latest/*.py", "latest/config.json", "latest/model.safetensors"],
|
| )
|
| folder = str(Path(folder) / "latest")
|
| sys.path.insert(0, folder)
|
| from model import load_model, generate
|
|
|
| model = load_model(folder, device="cpu")
|
| print(generate(model, "import bpy\nbpy.ops.mesh.", max_new_bytes=80))
|
| ```
|
|
|
| Generation defaults to the 128-byte training context and greedy decoding over byte values 0–255. The longer positional table does not establish reliable performance at longer contexts. Generated code should be inspected and executed in an appropriate Blender environment before use.
|
|
|
| ## Resume state without pickle
|
|
|
| ```python
|
| from model import load_training_state
|
|
|
| state = load_training_state("Chaman1234/Sparse-AST-BWM", revision=revision)
|
| assert state["step"] == 93213
|
| optimizer.load_state_dict(state["optimizer"])
|
| scaler.load_state_dict(state["scaler"])
|
| ```
|
|
|
| Construct the model's optimizer using `model.trainable_parameter_groups()` before restoring its state. Restore `python_rng`, `torch_rng`, and, on compatible CUDA hardware, `cuda_rng`; restore the recorded `order`, `cursor` and progress counters in the training loop. The loader reconstructs original dictionary key types and tuples from JSON and SafeTensors. No pickle is used to load the published release.
|
|
|
| ## Evaluation and limits
|
|
|
| The archived run reports the following **teacher-forced byte-completion** results on 256 examples per split:
|
|
|
| | Split | Cross-entropy | Top-1 byte accuracy | Top-5 byte accuracy |
|
| |---|---:|---:|---:|
|
| | Validation | 0.015052 | 99.5342% | 100.0000% |
|
| | Test | 0.029494 | 99.3290% | 99.8614% |
|
|
|
| These are archived training-run metrics, not a newly reproduced benchmark, full-script success rate, Blender execution result or visual-quality assessment. The data audit reports family-disjoint train/validation/test splits within this sampled dataset; historical overlap with every earlier training corpus is not established. Source quality checks cover JSON/AST syntax and `bpy` imports, not runtime or visual correctness.
|
|
|
| The release conversion independently verifies all model tensors exactly, all non-model training state exactly, finite parameters, and forward logits within FP32 tolerance (absolute 0.0001, relative 0.00001) with identical predicted tokens on two Blender-related prompts. See `validation.json` for measured differences and hashes. This establishes conversion fidelity, not general coding reliability.
|
|
|
| ## Provenance
|
|
|
| - Archive: `results (4).zip`.
|
| - Member: `sparse_ast_bigtrain_3k_output/checkpoint_latest.pt`.
|
| - Recorded final step: 93,213; parent step: 90,213.
|
| - Recorded successful continuation updates: 3,000; attempts: 3,000; non-finite skips: zero.
|
| - Training source: compact Blender task/code pairs described in `data_audit.json`.
|
| - Conversion used PyTorch's restricted `weights_only=True` loader, followed by lossless SafeTensors serialization and parity checks.
|
|
|
| Format references: [SafeTensors and shared tensors](https://huggingface.co/docs/safetensors/torch_shared_tensors), [Hugging Face upload documentation](https://huggingface.co/docs/huggingface_hub/guides/upload).
|
|
|
|
|
| ## Hybrid associative-memory / GRU pilot (evaluation-only)
|
|
|
| A separate 74.13M-parameter byte model is available under [`hybrid-3memory-3gru-1attention/`](./hybrid-3memory-3gru-1attention/). Its FP32 inference checkpoint is stored as SafeTensors (`model.safetensors`) with matching PyTorch code and a technical PDF containing VRAM/RAM estimates and formulas. This model is distinct from the step-93,213 model in `latest/`.
|
|
|
| - Evaluation report (all versions; resolves to the latest): [Zenodo concept DOI 10.5281/zenodo.23569293](https://doi.org/10.5281/zenodo.23569293). Current version: [v10, DOI 10.5281/zenodo.23585110](https://doi.org/10.5281/zenodo.23585110). Supplied earlier version: [v8, DOI 10.5281/zenodo.23584899](https://doi.org/10.5281/zenodo.23584899).
|
| - Related Selective Associative Memory note: [Zenodo DOI 10.5281/zenodo.23489622](https://doi.org/10.5281/zenodo.23489622).
|
| - [Memory requirements and formulas PDF](./hybrid-3memory-3gru-1attention/hybrid_model_memory_requirements_and_formulas.pdf).
|
|
|
| The report explicitly labels the work evaluation-only. It has no matched vanilla-attention baseline, no tested source-checkpoint baseline, one training seed, and no measured end-to-end compute savings. The SafeTensors file contains model weights only; optimizer state, teacher logits, training data, and source weights are not included.
|
|
|
