---
library_name: pytorch
license: apache-2.0
pipeline_tag: text-generation
tags:
  - text-generation
  - code-generation
  - python
  - small-language-model
  - slm
  - transformer
  - research
language:
  - en
---

# NeemSutra-125M-Python-Instruct-Beta

NeemSutra-125M-Python-Instruct is a **125M-parameter Small Language Model (SLM)** developed from first principles as an experimental study of small language model design, tokenization, domain adaptation, supervised fine-tuning, reproducible training, inference behavior, and stress testing.

The project focuses on understanding how a relatively small decoder-only Transformer behaves as training data, domain adaptation, and instruction tuning are progressively applied.

This release is intended primarily as a **research and educational artifact**.

## Model

- Parameters: **125,892,096 (~125M)**
- Architecture: **Decoder-only Transformer**
- Vocabulary: **8,000 tokens**
- Context length: **256 tokens**
- Tokenizer: **Sutra BPE v2**
- Primary domain: **Python / programming**
- Final checkpoint: **OpenCodeInstruct SFT, ~2.0B tokens**

## Architecture

The released model uses a decoder-only Transformer architecture with:

- Token embeddings
- Learned positional embeddings
- **16 Transformer layers**
- **12 attention heads**
- Model dimension: **768**
- Feed-forward MLP layers
- Layer normalization
- Final language-model head
- Vocabulary size: **8,000**
- Maximum context length: **256**

The released source contains the model implementation used for inference.

## Tokenizer

NeemSutra uses the custom **Sutra BPE v2** tokenizer.

- Vocabulary size: **8,000**
- Subword-based BPE tokenization
- Custom tokenizer implementation included with the release
- Tokenizer is used consistently across the training and inference workflow

The tokenizer is part of the experimental design and is released alongside the model.

## KV-Cache Memory Analysis

To evaluate memory efficiency during inference and sequence generation, KV-cache memory consumption was profiled across different batch sizes and context lengths.

Assumptions:

- Transformer layers: **16**
- Attention heads: **12**
- Head dimension: **64**
- Precision: **FP16**

| Batch Size ($B$) | Context Length ($T$) | Pre-KV Cache Memory | Post-KV Cache Memory | KV Cache Overhead |
|---|---|---|---|---|
| 1 | 64 | ~12.0 MB | ~18.0 MB | +6.0 MB |
| 1 | 256 (Max) | ~12.0 MB | ~36.0 MB | +24.0 MB |
| 4 | 64 | ~12.0 MB | ~36.0 MB | +24.0 MB |
| 4 | 256 (Max) | ~12.0 MB | ~84.0 MB | +72.0 MB |

This analysis is included to characterize inference memory behavior as batch size and context length increase.

## Training Trajectory

The model was developed through sequential training stages:

| Stage | Dataset / Corpus | Training Tokens |
|---|---|---:|
| Base Pretraining | Common Pile v0.1-derived corpus | ~5.75B |
| Python DAPT | `Lovett01/Python-Code-Large` | ~7.4B |
| Python SFT | `OLMo-Coding/starcoder-python-instruct` | ~2.79B |
| Extended DAPT | `togethercomputer/RedPajama-Data-1T` | ~24B |
| Final SFT | `nvidia/OpenCodeInstruct` | ~2.0B |

The stages were performed sequentially as part of the experimental training program.

The final released checkpoint corresponds to the **OpenCodeInstruct SFT stage**.

The training trajectory is part of the research artifact and is intended to make the progression from base pretraining through domain adaptation and instruction tuning inspectable.

## Dataset References

### Base Pretraining — Common Pile

The base model was trained on a project-selected corpus derived from the Common Pile v0.1 collection.

The local corpus was organized into:

- Books
- General
- Science
- Technical

Primary references:

- Common Pile: https://huggingface.co/common-pile
- Common Pile v0.1 collection: https://huggingface.co/collections/common-pile/common-pile-v01

Common Pile v0.1 is an 8 TB collection of public-domain and openly licensed text from more than 30 sources. NeemSutra used a project-selected subset/organization of this broader collection.

Base training: **~5.75B tokens**.

### First Python DAPT

Dataset:

- Python-Code-Large: https://huggingface.co/datasets/Lovett01/Python-Code-Large/tree/main

Training: **~7.4B tokens**.

### First SFT

Dataset:

- starcoder-python-instruct: https://huggingface.co/datasets/OLMo-Coding/starcoder-python-instruct/tree/main

Training: **~2.79B tokens**.

### Extended DAPT

Dataset:

- RedPajama-Data-1T: https://huggingface.co/datasets/togethercomputer/RedPajama-Data-1T/tree/main

Training: **~24B tokens**.

### Final SFT

Dataset:

- OpenCodeInstruct: https://huggingface.co/datasets/nvidia/OpenCodeInstruct

Training: **~2.0B tokens**.

OpenCodeInstruct is published by NVIDIA and is licensed under CC BY 4.0 according to its dataset card. Users should review the applicable terms and attribution requirements of the upstream datasets before using or redistributing derived artifacts.

## Training Methodology

NeemSutra was developed through a sequential training workflow rather than starting from an existing pretrained language model.

The experimental workflow included:

1. Custom tokenizer construction
2. Dataset preparation and token accounting
3. Base pretraining
4. Python domain-adaptive pretraining
5. Supervised fine-tuning
6. Extended domain-adaptive pretraining
7. Final supervised fine-tuning
8. Intermediate checkpoint evaluation
9. Inference and memory analysis
10. Final checkpoint validation and packaging

The purpose of this workflow was to examine the effects of progressively changing the training distribution and training objective on a small language model.

## Engineering and Optimization

The project included engineering work across the training and inference pipeline, including:

- Large-scale dataset tokenization
- Token accounting and validation
- Streaming training workflows
- Checkpoint management
- Training throughput optimization
- Memory profiling
- Intermediate checkpoint evaluation
- Deterministic inference
- CPU, CUDA, and MPS inference support
- Checkpoint integrity verification
- SHA-256 checksum generation

The release includes the scripts required to reproduce local inference with the published checkpoint.

## Architectural and Methodological References

- **Attention Mechanism:** Vaswani et al., *Attention Is All You Need*, Advances in Neural Information Processing Systems (NIPS), 2017. https://arxiv.org/abs/1706.03762
- **Byte-Pair Encoding (BPE) Tokenization:** Sennrich et al., *Neural Machine Translation of Rare Words with Subword Units*, Association for Computational Linguistics (ACL), 2016. https://arxiv.org/abs/1508.07909
- **Decoder-Only Transformer & Scaling Design:** Radford et al., *Language Models are Unsupervised Multitask Learners* (GPT-2 architecture foundations).

## Research Focus

The project focuses on:

- First-principles SLM architecture design
- Tokenizer construction and validation
- Dataset preparation and token accounting
- Large-scale tokenization
- Domain-adaptive pretraining
- Supervised fine-tuning
- Training throughput optimization
- Checkpoint validation
- Reproducible inference
- KV-cache memory analysis
- Stress testing across intermediate and final checkpoints

The objective is to study how a small language model behaves as training data, domain adaptation, and instruction tuning are progressively applied.

## AI Assistance & Collaboration Statement

This experimental model was developed using **AI-assisted engineering workflows**.

AI tools were used as engineering support during different stages of the project, including:

- Architecture exploration and design
- Tokenization and data-processing pipeline development
- Training pipeline implementation and optimization
- Checkpoint evaluation and generation analysis
- Supervised Fine-Tuning (SFT) validation

The model architecture, training experiments, dataset preparation, training execution, evaluation methodology, checkpoint management, and final release were conducted as part of the NeemSutra experimental workflow.

## Evaluation

Evaluation was performed throughout development using intermediate and final checkpoints.

The evaluation suite includes:

- Python code completion
- Teacher-forced next-token prediction
- Continuation testing
- Syntax validation
- Target-token probability/rank analysis
- N-gram matching
- AST-based evaluation
- Stress-test prompt sets

The evaluation process was designed to examine multiple aspects of small-model behavior rather than relying on a single aggregate metric.

The released inference script also provides deterministic greedy decoding and configurable sampling for reproducible local tests.

## Inference

The release includes a self-contained inference workflow supporting:

- **CPU**
- **CUDA**
- **Apple Silicon / MPS**

Inference supports deterministic greedy decoding as well as configurable sampling.

Refer to `inference.py` for the local inference entry point.

## Checkpoint Verification

The released checkpoint includes SHA-256 verification information to allow users to verify the integrity of the downloaded model artifact.

Checkpoint:

`neemsutra_125m_python_instruct.pt`

SHA-256:

`0f55fd31e4a703c5fe27029c4d00111671d65a09329acac0c64902133c5308d`

## Intended Use

NeemSutra-125M-Python-Instruct-Beta is intended for:

- Research into small language models
- Educational experimentation
- Python code-generation experiments
- Tokenization research
- Domain-adaptation experiments
- Training-pipeline experimentation
- Inference and memory analysis
- Stress testing of small language models

The model should be considered an **experimental research artifact**, not a production-ready general-purpose language model.

## Limitations

NeemSutra is a 125M-parameter experimental SLM with a 256-token context window.

Its behavior can vary significantly depending on:

- Prompt structure
- Context length
- Sampling configuration
- Python/code distribution
- Training stage
- Evaluation methodology

The model may produce syntactically invalid, incomplete, repetitive, or incorrect code and should not be assumed to be reliable without validation.

## License

The model and accompanying code in this repository are released under the **Apache License 2.0**, except where otherwise specified by upstream datasets or third-party components.

Users are responsible for reviewing the licenses and attribution requirements of upstream datasets before redistributing or using derived artifacts.

## Citation

If you use NeemSutra in research, experimentation, or educational work, please cite this repository:

```bibtex
@misc{neemsutra2026,
  title        = {NeemSutra-125M-Python-Instruct-Beta},
  author       = {Kiran A N},
  year         = {2026},
  publisher    = {Hugging Face},
  note         = {Experimental 125M-parameter Small Language Model},
  url          = {https://huggingface.co/KiranAN1988/NeemSutra-125M-Python-Instruct-Beta}
}
```

## Summary

NeemSutra-125M-Python-Instruct-Beta is a **first-principles design and analysis study of a small language model**, covering the complete experimental path from tokenizer construction and base pretraining through domain adaptation, supervised fine-tuning, inference optimization, evaluation, and stress testing.

The release is intended to make the **architecture, training trajectory, engineering workflow, and observed SLM behavior inspectable and reproducible**.