--- license: apache-2.0 language: - en library_name: transformers pipeline_tag: text-generation base_model: nkthebass/tinybrainbot-360m-a75m-v5.1-base tags: - tinybrainbot - mixture-of-experts - moe - mixtral - small-language-model - from-scratch - code - python - conversational - gguf datasets: - codeparrot/codeparrot-clean - OpenCoder-LLM/opc-annealing-corpus - bigcode/the-stack-smol - OpenCoder-LLM/opc-sft-stage2 - bigcode/self-oss-instruct-sc2-exec-filter-50k - Vezora/Tested-143k-Python-Alpaca - ise-uiuc/Magicoder-OSS-Instruct-75K - KodCode/KodCode-V1 - HuggingFaceTB/smol-smoltalk - allenai/tulu-3-sft-personas-instruction-following - allenai/WildChat-4.8M - HuggingFaceTB/smoltalk - HuggingFaceTB/smoltalk2 - teknium/OpenHermes-2.5 model-index: - name: TinyBrainBot-360M-A75M-v5.1-Python results: - task: type: text-generation dataset: type: openai/openai_humaneval name: HumanEval split: test metrics: - type: pass@1 value: 14.63 name: pass@1 (greedy, chat) - task: type: text-generation dataset: type: google-research-datasets/mbpp name: MBPP (sanitized) split: test metrics: - type: pass@1 value: 32.68 name: pass@1 (greedy, chat) - task: type: text-generation dataset: type: allenai/ai2_arc name: ARC-Easy config: ARC-Easy split: test metrics: - type: acc_norm value: 57.74 name: acc_norm (0-shot) - task: type: text-generation dataset: type: allenai/ai2_arc name: ARC-Challenge config: ARC-Challenge split: test metrics: - type: acc_norm value: 34.64 name: acc_norm (0-shot) - task: type: text-generation dataset: type: Rowan/hellaswag name: HellaSwag split: validation metrics: - type: acc_norm value: 37.49 name: acc_norm (0-shot) - task: type: text-generation dataset: type: ybisk/piqa name: PIQA split: validation metrics: - type: acc_norm value: 65.67 name: acc_norm (0-shot) --- # TinyBrainBot-360M-A75M-v5.1-Python **A tiny Python coding assistant.** The v5.1 360M-A75M mixture-of-experts model (383M parameters, 24 experts per layer, 4 active, so 75M parameters work per token), trained further on Python: a 2B-token code anneal, a Python instruction SFT, and one round of rejection-sampling fine-tuning where it learned from its own answers that passed unit tests. Ask it for a function or paste broken code; it answers with a ```python block. It loads as a standard `MixtralForCausalLM`. **You:** This function should return the sum of the even numbers, but it's wrong. Fix it: ```python def sum_even(nums): total = 0 for n in nums: if n % 2 == 1: total += n return total ``` **TinyBrainBot:** The issue with the previous function is that it's not checking if the number is even or not. It's checking if the number is divisible by 2 without a remainder. Here's the corrected function: ```python def sum_even(nums): total = 0 for n in nums: if n % 2 == 0: total += n return total ``` This function works by iterating through each number in the input list and checking if it's even. If it is, it adds it to the total. Finally, it returns the total sum. --- **You:** Write a function that returns the factorial of n. **TinyBrainBot:** Here is the code to solve this problem: ```python def factorial(n): if n == 0: return 1 else: return n * factorial(n-1) ``` **You:** Now add type hints and a docstring. **TinyBrainBot:** Here's a revised version of the function: ```python def factorial(n: int) -> int: """ Returns the factorial of n. Args: n (int): The input integer. Returns: int: The factorial of n. Raises: TypeError: If n is not an integer. """ if not isinstance(n, int): raise TypeError("Input must be an integer.") if n < 0: raise ValueError("Input must be a non-negative integer.") if n == 0: return 1 return n * factorial(n-1) ``` ## Code benchmarks All four models were run through the same harness: chat mode (the task goes in the chat template, the answer's ```python block is run against the official tests), greedy, pass@1, HumanEval (164) and MBPP sanitized test (257). | Model | Params | Training tokens | HumanEval | MBPP | |---|---|---|--:|--:| | **TinyBrainBot-360M-A75M-v5.1-Python (this)** | 383M (75M active) | 20B | 14.6 | 32.7 | | [SmolLM2-360M-Instruct](https://huggingface.co/HuggingFaceTB/SmolLM2-360M-Instruct) | 362M | 4T | 16.5 | 33.5 | | [SmolLM2-135M-Instruct](https://huggingface.co/HuggingFaceTB/SmolLM2-135M-Instruct) | 135M | 2T | 7.9 | 10.9 | | [Qwen2.5-Coder-0.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-Coder-0.5B-Instruct) | 494M | Qwen2.5 + 5.5T code | 52.4 | 46.3 | It lands next to SmolLM2-360M-Instruct after about 1/200th of the training data. Qwen2.5-Coder-0.5B is a bigger, code-specialized model trained on trillions of tokens; it's here as a reference point, not a peer. | General benchmarks (0-shot, full set) | acc_norm | *(raw acc)* | |---|--:|--:| | HellaSwag | 37.49 | *32.59* | | ARC-Easy | 57.74 | *61.45* | | ARC-Challenge | 34.64 | *33.28* | | PIQA | 65.67 | *64.58* | | ArithMark-3 | 46.60 | *46.60* | | Open SLM Leaderboard Intelligence Index | 26.02 | *24.06* | 30 unedited answers (15 requests, greedy and sampled, with pass/fail) are in [`code_samples.md`](code_samples.md). ## Quick start ```python from transformers import AutoModelForCausalLM, AutoTokenizer repo = "nkthebass/tinybrainbot-360m-a75m-v5.1-python" tok = AutoTokenizer.from_pretrained(repo) model = AutoModelForCausalLM.from_pretrained(repo) msgs = [{"role": "user", "content": "Write a function is_prime(n) that returns True if n is prime."}] ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt") out = model.generate(ids, max_new_tokens=400) # defaults from generation_config.json text = tok.decode(out[0], skip_special_tokens=True) # decode WITH the prompt, then cut it off (see below) print(text[len(tok.decode(ids[0], skip_special_tokens=True)):].strip()) ``` Decode the answer together with the prompt and cut the prompt off, like above. The tokenizer's decoder drops one leading space from whatever it decodes, so decoding only the new tokens can turn the first 4-space indent into 3. **GGUF** (llama.cpp, LM Studio, Jan, Ollama): Q8_0 (409 MB), Q4_K_M (239 MB) or F16 (768 MB). The chat template is embedded, prompts with indented code tokenize exactly like transformers, and greedy F16 output matches transformers. ``` llama-cli -m tinybrainbot-360m-a75m-v5.1-python-Q8_0.gguf -cnv --temp 0.4 --top-p 0.9 --repeat-penalty 1.15 --logit-bias 241-inf --logit-bias 255-inf ``` **Recommended sampling:** temperature 0.4, top-p 0.9, repetition penalty 1.15 (the `generation_config.json` defaults). Plain greedy decoding without a repetition penalty can get stuck repeating itself on questions that aren't code. ## How it was made | Stage | Tokens | Data | |---|--:|---| | Code anneal (from [v5.1-Base](https://huggingface.co/nkthebass/tinybrainbot-360m-a75m-v5.1-base)) | 2.0B | 45% codeparrot-clean, 25% OpenCoder annealing corpus (algorithmic Python + synthetic Q&A), 5% the-stack-smol Python, 25% replay of the v5.1 base's last training stage | | Python SFT | 262M | 70% Python instructions (OpenCoder SFT stage 2, self-oss-instruct-sc2, Tested-143k-Python-Alpaca, Magicoder OSS-Instruct), 30% the v5.1-Instruct chat mix | | Rejection-sampling fine-tune | 35M | 40% the model's own answers that passed unit tests, 30% Python SFT, 30% chat mix | **Indentation.** The TinyBrainBot tokenizer used to collapse all leading spaces, which makes Python impossible: the v5.1 base scores 0.0 on both benchmarks because every function body loses its indentation. For this model each run of 4 spaces is one token (`" "`, id 24) and tabs become 4 spaces, so indentation survives training and generation. **Rejection sampling.** 26,480 Python problems from KodCode-V1 (easy and medium, one function each, unit tests checked against the reference solution), 8 attempts each. 21% of the attempts passed their tests; 12,419 problems got at least one pass. Up to two distinct passing answers per problem (17,080 total) went back into training. **Contamination.** Every SFT and RFT example that shares a 13-word sequence with HumanEval (prompts, solutions, tests) or any MBPP split was dropped, as were KodCode problems that KodCode itself flags as close to a benchmark. No benchmark data was trained on. Licenses: KodCode-V1 is CC BY-NC 4.0, and the code corpora carry their own source licenses; check those terms before any commercial use. ## Limitations - It's a 75M-active-parameter model. It handles short, single-function tasks (string and list processing, simple loops, recursion, binary search, fixing an obvious bug), but tasks with several steps often come out wrong: a class with error handling, reading a CSV, sorting a dict by value all failed in the samples. - It explains code badly. Asked "what does this code do?", it tends to write different code instead. - Asked for tests, it writes a few asserts, sometimes with wrong expected values. - Greedy decoding can loop on non-code questions; use the recommended sampling. - Python only. English only. 2,048-token context. Two or more spaces inside a line (not at the start) collapse into one. - Like any model this size, it makes things up. Run the code before you trust it.