File size: 921 Bytes
82bdbd0 da70941 82bdbd0 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 | ---
license: apache-2.0
tags:
- nanoGPT
- code-generation
- fine-tuned
base_model: Ananda100/pocketcoder100M-distilled
---
# 100m-sftd-python
Full-parameter fine-tune of [`Ananda100/pocketcoder100M-distilled`](https://huggingface.co/Ananda100/pocketcoder100M-distilled)
on a custom Python coding-problem dataset, using the `deepseek-ai/deepseek-coder-6.7b-base` tokenizer.
- Fine-tune tokens processed: 356,253,696
- Best validation loss: 0.9270
- Mbpp pass@1: 7.6%
- Block size: 512
- Vocab size: 32022
## Loading
This is a **custom architecture**, not a native `transformers` model, so `AutoModel.from_pretrained`
won't work out of the box:
```python
import json, torch
from safetensors.torch import load_file
with open("config.json") as f:
cfg = json.load(f)
config = GPTConfig(**cfg)
model = GPT(config)
state_dict = load_file("model.safetensors")
model.load_state_dict(state_dict)
model.eval()
```
|