| --- |
| license: apache-2.0 |
| tags: |
| - nanoGPT |
| - code-generation |
| - fine-tuned |
| base_model: Ananda100/pocketcoder100M-distilled |
| --- |
| |
| # 100m-sftd-python |
|
|
| Full-parameter fine-tune of [`Ananda100/pocketcoder100M-distilled`](https://huggingface.co/Ananda100/pocketcoder100M-distilled) |
| on a custom Python coding-problem dataset, using the `deepseek-ai/deepseek-coder-6.7b-base` tokenizer. |
|
|
| - Fine-tune tokens processed: 356,253,696 |
| - Best validation loss: 0.9270 |
| - Mbpp pass@1: 7.6% |
| - Block size: 512 |
| - Vocab size: 32022 |
|
|
| ## Loading |
|
|
| This is a **custom architecture**, not a native `transformers` model, so `AutoModel.from_pretrained` |
| won't work out of the box: |
|
|
| ```python |
| import json, torch |
| from safetensors.torch import load_file |
| |
| with open("config.json") as f: |
| cfg = json.load(f) |
| |
| config = GPTConfig(**cfg) |
| model = GPT(config) |
| state_dict = load_file("model.safetensors") |
| model.load_state_dict(state_dict) |
| model.eval() |
| ``` |
|
|
