--- base_model: Qwen/Qwen3-0.6B-Base base_model_relation: finetune library_name: transformers pipeline_tag: text-generation datasets: - Ma7ee7/Claude-Reasoning-4096 - ahmad21omar/Polyglot-Thoughts-SFT-Collection language: - en - de - es - fr - it - ja tags: - qwen3 - reasoning - sft - multilingual --- # Andgit-0.6B-SFT Andgit is a 0.6B text model fully fine-tuned from **Qwen3-0.6B-Base** on reasoning traces paired with final answers. It uses ChatML and can generate a `…` block before the answer. The weights are exported in BF16. [Base model](https://huggingface.co/Qwen/Qwen3-0.6B-Base) ## Quick start Install PyTorch for your device and run `pip install "transformers>=4.57.1" accelerate`. ```python import torch from transformers import AutoModelForCausalLM, AutoTokenizer model_id = "Ma7ee7/Andgit-0.6B-SFT" tokenizer = AutoTokenizer.from_pretrained(model_id) model = AutoModelForCausalLM.from_pretrained( model_id, torch_dtype="auto", device_map="auto", ).eval() messages = [ {"role": "user", "content": "A bat and ball cost $1.10 together. " "The bat costs $1 more than the ball. How much does the ball cost?"} ] inputs = tokenizer.apply_chat_template( messages, tokenize=True, add_generation_prompt=True, return_dict=True, return_tensors="pt", ).to(model.device) prompt_length = inputs["input_ids"].shape[-1] remaining = 4096 - prompt_length if remaining <= 0: raise ValueError("Shorten the prompt to leave room for generation.") with torch.inference_mode(): output = model.generate(**inputs, max_new_tokens=min(2048, remaining)) text = tokenizer.decode(output[0, prompt_length:], skip_special_tokens=False) text = text.removesuffix("<|im_end|>").removesuffix("<|endoftext|>") print(text) ``` The published generation config defaults to temperature **0.6**, top-p **0.95**, top-k **20**, and up to **2,048 new tokens**. It stops on `<|im_end|>` or `<|endoftext|>`. The chat template accepts text system, user, and assistant turns. It omits completed think blocks from earlier assistant turns when building the next prompt. [Generation config](generation_config.json) · [Chat template](chat_template.jinja) ## Training data | Source | Eligible rows in the training pool | | --- | ---: | | [Claude Reasoning 4096](https://huggingface.co/datasets/Ma7ee7/Claude-Reasoning-4096) | 126,214 | | [Polyglot Thoughts SFT Collection](https://huggingface.co/datasets/ahmad21omar/Polyglot-Thoughts-SFT-Collection) | 16,000 | | **Total** | **142,214** | The Polyglot subset includes English, Spanish, German, French, Italian, and Japanese prompts. The sampler scanned 30,873 candidates and excluded 14,018 for length. Eligibility was checked after formatting and tokenization, with the prompt, reasoning, answer, and chat markers all counted toward the **4,096-token** limit. Oversized rows were discarded whole. The training pool was shuffled, and the step budget covered a partial pass through it. [Run manifest](run_manifest.json) ## Training setup | Setting | Value | | --- | --- | | Method | Full supervised fine-tuning | | Hardware | NVIDIA L4, Google Colab | | Precision | BF16 mixed precision with FP32 training weights | | Training sequence limit | 4,096 tokens | | Optimizer step budget | 2,000 | | Effective batch | 16 examples (microbatch 1 × 16 accumulation steps) | | Optimizer and schedule | AdamW, cosine decay | | Learning rate and warmup | 1e-5, 50 steps | | Seed | 42 | The loss covers the final assistant response, including the think tags, reasoning text, answer, and turn terminator. Prompt and padding tokens are masked. The export uses the checkpoint selected by held-out validation loss. [Run manifest](run_manifest.json) ## Evaluation | Checkpoint | Held-out assistant cross-entropy ↓ | | --- | ---: | | Qwen3-0.6B-Base before fine-tuning | 1.8666 | | Andgit, best saved checkpoint | **1.2589** | Both scores use the same **256 held-out Claude Reasoning examples**. The loss measures how well the model predicts the reference assistant text. Separate reasoning accuracy, multilingual performance, and longer-context behavior have yet to be measured here. [Validation metrics](validation_metrics.json) Generated reasoning can contain errors even when the final answer is correct. In one manual bat-and-ball check, Andgit answered **5¢** correctly but proposed the wrong subtraction in its think block. ## Licenses and provenance Qwen3-0.6B-Base is published under Apache 2.0. The Polyglot collection lists mixed source terms, and the selected subset includes 1,658 examples from the source labeled CC BY-NC 4.0 in its dataset card. See the upstream cards and the pinned source revisions for attribution and terms. [Qwen base model](https://huggingface.co/Qwen/Qwen3-0.6B-Base) · [Claude Reasoning data](https://huggingface.co/datasets/Ma7ee7/Claude-Reasoning-4096) · [Polyglot source terms](https://huggingface.co/datasets/ahmad21omar/Polyglot-Thoughts-SFT-Collection#source-datasets-and-licensing) · [Run manifest](run_manifest.json)