--- language: - en license: apache-2.0 library_name: transformers pipeline_tag: text-generation tags: - qwen3 - coding - fine-tuned - autonomous-agent - software-engineering base_model: Qwen/Qwen3.6-27B --- # Coder — Qwen3.6-4B Coding & Autonomous Software Engineering Model A fine-tuned 4B-parameter variant of **Qwen3.6** specialized for autonomous software engineering, code generation, debugging, repository exploration, terminal/shell agent behavior, Git operations, database/SQL, web development, DevOps, security, performance, architecture, and full multi-file feature implementation. > **Verified facts:** This repository contains `qwen3.6--coder.safetensors` (8.8 GB, BF16), a 20,000-example instruction dataset (`coding-finetune-dataset/dataset.jsonl`), training notes (`qwen3.6-4B.txt`), and dataset creation docs (`coding-finetune-dataset/CREATE_INSTRUCTIONS.md`). --- ## Model Overview `Coder` is intended as an autonomous software engineering assistant that can investigate repositories, identify problems, implement solutions, edit files, run tests, use Git, write SQL, and complete multi-step tasks independently. It was fine-tuned on a comprehensive 20,000-example dataset covering 20 engineering categories (A–U) across Python, JavaScript/TypeScript, Java, C/C++, Go, Rust, Bash, SQL, HTML/CSS, React/Node, DevOps, and more. **What makes it different:** The dataset is designed around realistic workflow scenarios — not just code snippets — including debugging loops, repository exploration, file editing with minimal changes, multi-file feature implementation, verification steps, and repair loops. --- ## Model Details | Property | Value (verified from repository) | | --------------- | --------------------------------- | | Model type | Causal language model (fine-tuned) | | Architecture | Qwen3.6 (Transformer, GQA) | | Parameters | 4B (from `qwen3.6-4B.txt`) | | Layers | 5 (from weight key analysis: `layers.0`–`layers.4`) | | Hidden size | 5120 (from weight shapes) | | Attention heads | 32 Q heads, 8 KV heads, head_dim=128 (from `q_proj`/`k_proj`/`v_proj` shapes) | | Context length | Not specified in repository | | Vocabulary size | 248,320 (from `embed_tokens.weight` / `lm_head.weight`) | | Data type | BF16 (`tensor type: BF16` per notes) | | Quantization | Not specified | | Base model | Qwen3.6 (notes reference `Qwen/Qwen3.6-27B` for the upstream draft head; this weight file is a 4B fine-tune) | | Training method | Supervised fine-tuning (SFT) on 20k instruction examples | | License | Apache 2.0 (`coding-finetune-dataset/README.md`) | --- ## Intended Use Appropriate use cases based on the actual dataset and model: - Code generation (Python, JS/TS, Java, C/C++, Go, Rust, SQL, Bash, HTML/CSS, React, Node) - Bug finding and debugging workflow (reproduce, isolate, identify root cause, patch, test, verify) - Repository exploration (structure analysis, framework identification, dependency tracing) - File editing (minimal changes, preserving style, updating imports/tests) - Multi-file feature implementation (auth, payments, uploads, notifications, caching, logging, error handling) - Terminal/shell agent behavior (`pwd`, `ls`, `find`, `grep`, `git`, `python`, `npm`, `docker`, etc.) - Git operations (`status`, `diff`, `log`, `branch`, `rebase`, `cherry-pick`, `bisect`) - Database / SQL (SELECT, INSERT, UPDATE, JOIN, CTE, window functions, transactions, indexes) - Web development (React + TypeScript frontend; Express / FastAPI backend) - DevOps / CI/CD (Docker, Docker Compose, GitHub Actions, deployment) - Security review (SQL injection, XSS, CSRF, auth mistakes, secret leakage, path traversal) - Performance optimization (slow algorithms, memory, query optimization, caching, concurrency) - Architecture planning (monoliths vs microservices, event-driven, queues, caching) - Full autonomous coding tasks (iterative debugging, verification loops) --- ## Not Intended For - Real-time production inference without evaluation (no benchmark results provided) - Medical, legal, or high-stakes decision-making without human review - Generating malicious code (security risks remain; model can discuss vulnerabilities but should not be used to create exploits) - Guaranteed factual accuracy (hallucinations and incorrect code may occur; always verify outputs) - Replacement for professional code review (model assists, does not replace expert review) - Use in contexts requiring guaranteed deterministic output (sampling parameters not verified) --- ## Training **Verified from `coding-finetune-dataset/README.md` and `CREATE_INSTRUCTIONS.md`:** - **Base model:** Qwen3.6 (4B parameter scale; upstream reference is Qwen3.6-27B for the draft-head variant documented in notes) - **Dataset:** `coding-finetune-dataset/dataset.jsonl` — 20,000 examples (full dataset at `dataset_full_20k.jsonl`; deduplicated at `dataset_deduplicated.jsonl`) - **Dataset source:** Synthetic generation via autonomous agent process (`generate_full_dataset.py`, `generate_examples.py`, `merge.py`); categories A–U with ~1,000 examples each; languages distributed across Python (20%), JS (15%), TS (15%), SQL (10%), Java (8%), C++ (7%), C (5%), Go/Rust/Bash (5% each), Other (5%) - **Example format:** Chat-style JSON lines with `messages` array (`system`, `user`, `assistant`) and `metadata` (category, language, difficulty, task_type, tools_used, requires_terminal/editing/testing, generated_at, version, example_id) - **Difficulty:** Easy 15%, Medium 35%, Hard 35%, Expert 15% (per `CREATE_INSTRUCTIONS.md`) - **Training objective:** Supervised fine-tuning on instruction-following and autonomous engineering workflows - **Framework:** Not specified in repository - **Hardware / GPUs / duration / epochs / batch size / learning rate:** Not specified in repository (only dataset generation scripts and statistics exist; no `training_config.json` or `trainer_state.json` found) > **Not verified from repository:** Training hyperparameters (learning rate, batch size, epochs, optimizer, scheduler, sequence length, GPU count, training duration, framework, exact base-model initialization weights beyond the 4B file). --- ## Dataset | Property | Value (verified) | | -------- | ----------------- | | Name | `coding-finetune-dataset` | | Source | Generated locally (scripts in `coding-finetune-dataset/scripts/`) | | Size | 20,000 examples (full dataset 29.6 MB; `dataset.jsonl`) | | Format | JSON Lines (`.jsonl`) with chat-style messages | | Categories | 20 categories (A–U): Code Generation, Bug Finding, Terminal/Shell, Repository Exploration, File Editing, Multi-File Implementation, Debugging Workflow, Testing, Git, Database/SQL, Web Development, DevOps, Security, Performance, Code Review, Architecture, Requirement→Implementation, Error Message Interpretation, Agent Tool Use, Verification, Repair Loop, Full Autonomous Coding Tasks | | Languages | Python, JavaScript, TypeScript, SQL, Java, C++, C, Go, Rust, Bash/Linux, HTML/CSS, React, Node, Express, FastAPI | | Difficulty | Easy / Medium / Hard / Expert | | Tools used (per example) | `read_file`, `edit_file`, `run_tests`, `run_terminal`, `git_*`, etc. | | Deduplicated version | `dataset_deduplicated.jsonl` (29.5 MB) | | Validation script | `validate_dataset.py` | | Sample | `samples/sample_100.jsonl` (100 examples, all Bug Finding / JavaScript / Hard) | | Batch scripts | `batch_001.jsonl` through `batch_024.jsonl` (generated in batches) | **Data licensing:** Apache 2.0 (from dataset README). The dataset was generated through simulated coding scenarios and does not contain real-world sensitive information. --- ## Architecture The weight file (`qwen3.6--coder.safetensors`) contains the following verified architecture components (extracted from tensor names and shapes): - **Transformer decoder** with 5 layers (`layers.0`–`layers.4`) - **Hidden size:** 5120 - **MLP intermediate:** 17408 (Gated MLP with `gate_proj`, `up_proj`, `down_proj`) - **Attention:** Multi-head self-attention with GQA (32 Q heads, 8 KV heads, head_dim=128) - **Normalization:** RMSNorm / LayerNorm (`norm.weight`, `hidden_norm.weight`, `input_layernorm.weight`, `post_attention_layernorm.weight`) - **Embeddings:** `embed_tokens.weight` [248320, 5120] - **LM head:** `lm_head.weight` [248320, 5120] - **Speculative-decoding head (DSpark-style):** `confidence_head.proj.weight` [1, 5376], `markov_head.markov_w1.weight` [248320, 256], `markov_head.markov_w2.weight` [248320, 256] - **Position encoding:** Not specified in weights (likely RoPE or ALiBi; not verified) - **Activation:** Not specified (GELU / SwiGLU typical for Qwen) > **Note:** The file is named `qwen3.6--coder.safetensors` (double dash) and is 8.8 GB. The notes (`qwen3.6-4B.txt`) reference a 27B upstream draft head (`abstract-extraordinary/Qwen3.6-27B-DSpark`) and describe speculative-decoding benchmarks at 8-speculative-token configuration. The weight file here is a 4B fine-tune for coding/autonomous-agent use. --- ## Tokenizer > **Not verified from repository:** No `tokenizer.json`, `tokenizer_config.json`, `special_tokens_map.json`, or `generation_config.json` is present in this directory. The dataset uses chat-format messages (`system`/`user`/`assistant`) consistent with Qwen3's chat template, but the exact tokenizer configuration must be loaded from the base Qwen3.6 model or provided separately. **Recommended token usage:** Load tokenizer from the base `Qwen/Qwen3.6-27B` (or equivalent 4B tokenizer) and apply the standard Qwen3 chat template (`<|im_start|>system...<|im_end|>...`) when formatting messages. --- ## Usage ```python from transformers import AutoTokenizer, AutoModelForCausalLM # Replace with your actual upload path model_id = "AnuragMahajan/coder-4b" tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True) model = AutoModelForCausalLM.from_pretrained( model_id, torch_dtype="auto", device_map="auto" ) messages = [ {"role": "system", "content": "You are Coder, an autonomous software engineering assistant..."}, {"role": "user", "content": "Find the JWT expiration issue in src/auth/middleware.js and fix it."} ] inputs = tokenizer.apply_chat_template( messages, add_generation_prompt=True, return_tensors="pt" ).to(model.device) outputs = model.generate( inputs, max_new_tokens=1024, do_sample=True, temperature=0.7, top_p=0.95 ) print(tokenizer.decode(outputs[0], skip_special_tokens=True)) ``` > **Note:** The repository does not include `tokenizer.json`. Use the base Qwen3.6 tokenizer (or download it separately) and verify chat-template compatibility before inference. --- ## Generation Parameters Not verified from repository. Recommended values (based on dataset notes and typical Qwen3 settings): - `temperature`: 0.7 (used in dataset regeneration notes) - `top_p`: 0.95 - `max_new_tokens`: 256–1024 depending on task complexity - `do_sample`: `True` for diverse responses; `False` (greedy) for deterministic debugging - `repetition_penalty`: 1.05–1.1 (typical) --- ## Hardware Requirements Not verified exactly. Estimates based on 4B parameters + BF16: - **Minimum:** GPU with ~10 GB VRAM (BF16, 4B model ~8 GB weights; inference overhead requires additional memory) - **Recommended:** 16–24 GB VRAM GPU (NVIDIA A10, RTX 4090, L4, A100 40GB for batch inference) - **CPU:** Possible with `torch_dtype="float32"` but very slow; not recommended - **RAM:** 16 GB system RAM minimum; 32 GB for dataset loading / training - **Disk:** ~9 GB for model weights + ~30 MB for dataset + dependencies - **Quantized options:** Not present in repository; can try `bitsandbytes` 4-bit / 8-bit if needed --- ## Quantization No quantized versions (`quantize_*.py`, `*.gguf`, `*.q4_*.safetensors`, `adapter_config.json`) are present in the repository. The weight file is full-precision/bf16 (`qwen3.6--coder.safetensors`). --- ## Performance / Benchmarks > **No formal benchmark results are currently provided for this fine-tuned model.** The repository contains only dataset-generation statistics (`scripts/batch_*_stats.json`, `dataset.jsonl` statistics) and notes referencing upstream DSpark speculative-decoding benchmarks (for a separate 27B draft-head model, not this 4B fine-tune). No accuracy, perplexity, BLEU, ROUGE, HumanEval, MBPP, or agent-evaluation scores exist in this directory. --- ## Limitations - **Evaluation:** No formal evaluation results provided for this fine-tuned model. - **Context length:** Not verified from repository. - **Base model dependency:** Fine-tuned from Qwen3.6; exact initialization weights not documented beyond the 4B file. - **Tokenizer:** Not included; must use base Qwen3.6 tokenizer. - **Quantization:** Not supported out of this repo. - **Hallucination:** Like all LLMs, output may contain incorrect code, outdated APIs, or missing error handling. Always verify generated code with tests. - **Domain:** Trained primarily on simulated software-engineering scenarios; performance in specialized domains (embedded, real-time, high-security) is not guaranteed. - **Safety:** The dataset includes security-related examples (SQL injection, XSS, auth mistakes) for educational/debugging purposes. Do not use model outputs to construct exploits. --- ## Bias, Safety and Responsible Use The dataset is synthetic (simulated coding scenarios) and does not contain real-world personal data. However: - The model may reproduce biases present in synthetic training data (e.g., over-representation of common frameworks, under-representation of niche languages). - Security examples describe vulnerabilities; the model should not be used to generate malicious payloads. - Generated code should always be reviewed by a human engineer and tested before deployment. - No formal safety evaluation (toxicity, fairness, adversarial robustness) has been conducted for this fine-tune. --- ## Repository Structure ```text . ├── qwen3.6--coder.safetensors # Fine-tuned weights (BF16, 8.8 GB) ├── README.md # Training notes, datasets, upstream references, benchmark notes ``` --- ## Citation If using this model or dataset, please cite sources present in repository: - Base architecture: Qwen3.6 (Qwen Team) - Upstream draft-head reference: `abstract-extraordinary/Qwen3.6-27B-DSpark` - Dataset: `coding-finetune-dataset/` (local generation via scripts in repo) - Licensing: Apache 2.0 --- ## Changelog ### v1.0.0 (initial release — repository state 2026-09-26) - Fine-tuned weights: `qwen3.6--coder.safetensors` - Instruction dataset: 20,000 examples (`coding-finetune-dataset/`) - License: Apache 2.0 - Notes: `qwen3.6-4B.txt` (datasets, upstream references, benchmark context) ---