Upload folder using huggingface_hub

README.md

@@ -0,0 +1,38 @@
+---
+license: apache-2.0
+tags:
+  - nanoGPT
+  - code-generation
+  - fine-tuned
+base_model: Ananda100/pocketcoder100M-distilled
+---
+
+# 100m-sftd-python
+
+Full-parameter fine-tune of [`Ananda100/pocketcoder100M-distilled`](https://huggingface.co/Ananda100/pocketcoder100M-distilled)
+on a custom Python coding-problem dataset, using the `deepseek-ai/deepseek-coder-6.7b-base` tokenizer.
+
+- Fine-tune tokens processed: 356,253,696
+- Best validation loss: 0.9270
+- Held-out pass@1: 9.5%
+- Block size: 512
+- Vocab size: 32022
+
+## Loading
+
+This is a **custom architecture**, not a native `transformers` model, so `AutoModel.from_pretrained`
+won't work out of the box:
+
+```python
+import json, torch
+from safetensors.torch import load_file
+
+with open("config.json") as f:
+    cfg = json.load(f)
+
+config = GPTConfig(**cfg)
+model = GPT(config)
+state_dict = load_file("model.safetensors")
+model.load_state_dict(state_dict)
+model.eval()
+```

config.json

@@ -0,0 +1,9 @@
+{
+  "block_size": 512,
+  "vocab_size": 32022,
+  "n_layer": 10,
+  "n_head": 12,
+  "n_embd": 768,
+  "dropout": 0.1,
+  "bias": true
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0640f540b8828edf17d9aff5b3593046adbf4796bb697aa6c56737fe7389a6e5
+size 481849696

tokenizer_config.json

@@ -0,0 +1,14 @@
+{
+  "backend": "tokenizers",
+  "bos_token": "<｜begin▁of▁sentence｜>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<｜end▁of▁sentence｜>",
+  "is_local": false,
+  "legacy": true,
+  "local_files_only": false,
+  "model_max_length": 16384,
+  "pad_token": "<｜end▁of▁sentence｜>",
+  "sp_model_kwargs": {},
+  "tokenizer_class": "TokenizersBackend",
+  "unk_token": null
+}