---
library_name: transformers
base_model: codeparrot/codeparrot-small-multi
pipeline_tag: text-generation
language: 
  - code
license: apache-2.0
tags:
- code
- gpt2
- text-generation
- safetensors
- transformers
datasets:
- "codeparrot/github-code-clean"
- "openai_humaneval"
metrics:
- "evaluate-metric/code_eval"
---

# TULLUS/codeparrot-small-multi-F32 🦜

A verified full-precision (F32) Safetensors conversion of the original [CodeParrot Small Multi](https://huggingface.co/codeparrot/codeparrot-small-multi) checkpoint. This release preserves the source model's learned tensor values; it is not a new training run or fine-tune.
## Provided Quants
(sorted by size.)

| Link | Type | Size/GB | Notes |
|:-----|:-----|--------:|:------|
| [Full-Precision-F32](https://huggingface.co/TULLUS/codeparrot-small-multi-F32/tree/main) | F32 | 4.44 GB | highest quality |
| New HQ Quants May Follow Stay Tuned :] |


---
CodeParrot Small Multi is a small GPT-2 causal language model for **code completion and continuation**, not an instruction-tuned chat model. It was pretrained on cleaned GitHub code and described upstream as covering Java, JavaScript, PHP, Python, C#, C++, Go, Ruby, and TypeScript. Treat it as an older, compact code-model baseline for experimentation not as standalone coding assistant.

## Intended use and realistic expectations

Prompt it with a code prefix or partial file for the model to continue, rather than expecting it to follow natural-language instructions like a chat assistant. For example, start with `def add(a, b):\n return` and let it continue the function. Surrounding code and a few relevant examples can help establish the style and pattern to continue, but better prompting does not turn this base model into an instruction-tuned model or guarantee correct code.

The upstream card describes 300,000 training steps, a context length of 1,024 tokens, and roughly 58 billion training tokens. Its HumanEval table leaves `pass@1`, `pass@10`, and `pass@100` as `--%`; it does not publish numerical HumanEval results for this exact multilingual checkpoint. This card therefore does not assign it a benchmark score. Results reported for the separate [`codeparrot/codeparrot-small`](https://huggingface.co/codeparrot/codeparrot-small) checkpoint must not be presented as Small Multi results.

In informal local probes, this checkpoint produced plausible code continuations. Those probes were exploratory, not a controlled benchmark, and cannot establish a pass rate or a general accuracy estimate. Validate any generated code with tests and review before use.

## Model details

| Property | Details |
|---|---|
| Upstream model | [`codeparrot/codeparrot-small-multi`] |
| Upstream revision | `7753edbe82562bf23c6ff15ad46ce6f0f2307139` |
| Architecture | GPT-2 (`GPT2LMHeadModel`) |
| Intended task | Code completion / causal text generation |
| Transformer layers | 12 |
| Hidden size | 768 |
| Attention heads | 12 |
| Vocabulary size | 32,768 |
| Context length | 1,024 tokens |
| Weight precision / format | F32 / Safetensors |
| Unique parameters with tied embeddings | 111,008,256 (about 111M) |
| Tokenizer | GPT-2 tokenizer |

## Provenance and source verification

The conversion is based on the pinned upstream revision shown above. The original `pytorch_model.bin` was downloaded and its SHA256 was checked before deserialization. It was loaded for this audit with `weights_only=True`.

Original upstream checkpoint:

```text
File size: 247,234,217 bytes
SHA256:    0207f6b427e3cbf1bcb9726abb6bbba6620e9912e1abf23087ccdef61818ffb2
```

Published TULLUS Safetensors file:

```text
File:      model.safetensors
File size: 444,048,000 bytes
SHA256:    9cbfd95c12db0da2a0a475516d16e7f424a0b6d78883b4b663d281fe20ba0ed7
```

A local source-to-artifact audit completed with the following results:

- The pinned upstream checkpoint hash matched the expected source hash.
- All **149 logical state-dictionary tensors** compared exactly between the upstream source and the published TULLUS artifact.
- The tokenizer vocabulary mapping matched exactly.
- Forward logits were exactly equal for three test prompts, with maximum absolute difference `0.0` for each comparison.
- The input embedding and output-head matrices in the original checkpoint had identical values, element for element.

The equality audit establishes preservation of the learned tensor values for this conversion. It is not a claim that the source and converted repository files are byte-identical, nor that their configuration metadata is identical.

## Embedding storage and parameter counts

The original checkpoint contains separate input-embedding and language-model output-head entries:

```text
transformer.wte.weight  shape (32768, 768)
lm_head.weight          shape (32768, 768)
```

In the source checkpoint these matrices occupied separate storage but had exactly equal values (maximum absolute difference `0.0`). The TULLUS conversion sets `tie_word_embeddings=True`, so When loaded, the model exposes both components while sharing the same underlying parameter object and storage.

| Measurement | Original checkpoint | TULLUS conversion |
|---|---:|---:|
| Input/output embedding storage | Separate | Shared Storage |
| Input/output embedding values | Exactly equal | Exactly equal and tied |
| Physical parameters before tying | 136,174,080 | — |
| Unique parameters after tying | 111,008,256 | 111,008,256 |
| Logical state-dictionary tensors | 149 | 149 |
| Physical tensors stored in Safetensors | — | 148 |

Safetensors stores the shared embedding matrix once rather than storing a duplicate output-head matrix. After loading, the model exposes `lm_head.weight` as a logical state-dictionary entry, with both components referencing the same underlying parameter storage.

This is a storage-representation change, not an additional training step. The audit also confirmed exact logical tensor equivalence and exact forward-logit equality on three test prompts.

## weight tensor specific checks
```python
python - <<'PY'
from transformers import AutoModelForCausalLM

model_path = "TULLUS/codeparrot-small-multi-F32"
model = AutoModelForCausalLM.from_pretrained(model_path)
model.eval()

wte = model.transformer.wte.weight
lm_head = model.lm_head.weight

print("Config ties word embeddings:", model.config.tie_word_embeddings)
print("Logical state_dict entries:", len(model.state_dict()))
print("Output-head entry present:", "lm_head.weight" in model.state_dict())
print("Same parameter object:", lm_head is wte)
print("Same underlying storage:", lm_head.data_ptr() == wte.data_ptr())
print("Maximum weight difference:", (lm_head - wte).abs().max().item())
PY
```
```python
Config ties word embeddings: True
Logical state_dict entries: 149
Output-head entry present: True
Same parameter object: True
Same underlying storage: True
Maximum weight difference: 0.0
```
## BOS/EOS metadata correction

The upstream repository has a difference between its model configuration and tokenizer:

- Upstream `config.json` declares `vocab_size = 32768`, `bos_token_id = 50256`, and `eos_token_id = 50256`.
- The tokenizer vocabulary has 32,768 entries and maps `<|endoftext|>` to token ID `0`; its BOS and EOS IDs are both `ID 0`.
- Valid IDs for a 32,768-entry vocabulary are `ID 0` through `32767`, so `50256` is outside this model's output vocabulary.

The TULLUS model and generation configurations set BOS/EOS to `ID 0 / ID 0`, aligning them with the tokenizer. The tokenizer mapping itself matches the upstream tokenizer exactly. This metadata correction does not change the learned tensors. It can affect generation behavior where BOS/EOS settings control special-token handling or stopping; this card does not claim identical stopping behavior for every decoding setup.

The tokenizer has no configured padding token (`pad_token_id=None`).

## Usage

This model is a pretrained code-completion model. It continues code from a supplied prompt and is not instruction-tuned.

Quick start — Transformers pipeline
```python
from transformers import pipeline

pipe = pipeline("text-generation", model="TULLUS/codeparrot-small-multi-F32")
outputs = pipe("def hello_world():")
print(outputs)
```

Direct generation — print only the continuation
```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "TULLUS/codeparrot-small-multi-F32"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)
model.eval()

prompt = "def hello_world():"
inputs = tokenizer(prompt, return_tensors="pt")

outputs = model.generate(
    **inputs,
    max_new_tokens=64,
    do_sample=False,
)

new_tokens = outputs[0, inputs["input_ids"].shape[-1]:]
print(tokenizer.decode(new_tokens, skip_special_tokens=True))
```
The second example is also useful because it prints only newly generated tokens, rather than echoing the prompt.

This example uses greedy decoding for a simple reproducible starting point. Output depends on the prompt and generation settings; the example does not guarantee a correct implementation.

## License and acknowledgement

The upstream repository identifies the model as Apache-2.0 licensed. Consult the [upstream model repository](https://huggingface.co/codeparrot/codeparrot-small-multi) and its dataset references for provenance and applicable terms. TULLUS does not claim authorship of the upstream architecture, training data, or learned capabilities.

Thanks to the original developers for this wonderful, highly capable mini model from its era. And thanks to OpenAI for the help along the way. We dug deep into this cool project, verified the conversion against the original checkpoint, and produced a Safetensors release backed by tensor-equivalence and forward-logit checks.
---

**TULLUS BAKES 🍪**  
*It's Old Gold and we safetensorized it :]*