| ---
|
| license: apache-2.0
|
| library_name: pytorch
|
| pipeline_tag: text-generation
|
| tags:
|
| - code
|
| - experimental
|
| - html
|
| - javascript
|
| - python
|
| - typescript
|
| ---
|
|
|
| # CodeSprout 30M — experimental code completion
|
|
|
| An independently trained small decoder-only Transformer for **HTML, JavaScript, Python and TypeScript**. This is an early research checkpoint, not a general coding assistant or a chat model.
|
|
|
| ## Actual training run
|
|
|
| - Unique model parameters: **29,676,544**.
|
| - Architecture: 8 layers, width 512, 8 heads, 8,192-token byte-level BPE vocabulary, tied output/token weights.
|
| - Context window: 512 tokens; training sequence length: 256.
|
| - Hardware: NVIDIA GeForce RTX 3050 Laptop GPU.
|
| - Selected inference checkpoint: step **4,324**, after **17,711,104** sampled tokens.
|
| - Final optimizer run reached step 6,079; later, more overfitted weights were not selected for inference.
|
| - Elapsed training time, including checkpoints and evaluation: 0.47 hours.
|
| - Peak allocated GPU memory: 788.3 MB.
|
|
|
| ## Data and limitations
|
|
|
| The starter corpus contains 812 exact-deduplicated files from pinned snapshots of MDN learning-area (CC0), Flask (BSD-3-Clause), Express (MIT), and TypeScript (Apache-2.0). It contains 732 training files and 80 held-out files. Files were split within the same repositories; validation is **not repository-disjoint**, and no claim of performance on unseen projects or coding benchmarks is made. Languages are sampled equally in training.
|
|
|
| Full corpus sizes and pinned source revisions appear in `data_report.json`; original license texts are in the `SOURCE_LICENSE_*.txt` files. This release does not redistribute raw third-party training files. The small corpus can cause overfitting and memorized output.
|
|
|
| After source-code pretraining, the model was aligned on **original deterministic templates for small functions and HTML pages**, plus about 10% source-token rehearsal. `completion_data_report.json` records their counts. These examples are not generated by a pretrained teacher. Their train/validation strings are disjoint, but their template families overlap. This makes the model most suitable for tutorial-sized code completion, not unfamiliar complex projects.
|
|
|
| ## Validation
|
|
|
| The following final next-token cross-entropy values are on **completion-template variations plus source rehearsal**, estimated from three fixed sampled batches per language. They are not losses on the original source-only validation split and must not be compared directly with that earlier stage:
|
|
|
| | Language | Validation loss |
|
| |---|---:|
|
| | html | 0.3679 |
|
| | javascript | 1.2808 |
|
| | python | 1.4414 |
|
| | typescript | 0.8401 |
|
|
|
| These measurements are language-modeling losses, **not** functional coding accuracy. `samples.json` contains unedited generated continuations through the supported completion wrapper. `evaluation.json` includes a tiny sanity check on trained arithmetic task families and balanced HTML tags, including the failures. This is not an unseen-task benchmark. Generated output may be syntactically invalid, incorrect, insecure or incomplete; inspect it before using it. This model was not chat/instruction-tuned and does not execute code.
|
|
|
| The completion wrapper normalizes trailing prefix whitespace before tokenization and restores the matching whitespace in the continuation. This avoids a byte-level BPE boundary issue around indentation; use the supplied wrapper rather than blindly tokenizing a prefix ending in a newline.
|
|
|
| ## Load and use
|
|
|
| Install the supplied requirements and place the inference files together. `generate.py` loads tensors with `torch.load(..., weights_only=True)`. The custom architecture is defined in `model.py`; this is not currently an AutoModel-compatible Transformers checkpoint.
|
|
|
| ```python
|
| from pathlib import Path
|
| from generate import load, complete
|
| model, tokenizer = load(Path("."), "cpu")
|
| print(complete(model, tokenizer, "def add(a, b):\n", "python"))
|
| ```
|
|
|
| The Gradio demo is `app.py`. No external API, paid inference provider or laptop connection is required once deployed. Generation is limited to 120 new tokens per request.
|
|
|
| ## License
|
|
|
| Model and project code are released under Apache-2.0. This does not replace the licenses or attribution requirements of the upstream training projects. Their license texts are included separately.
|
|
|
