taylonmcfly/Qwen3.5-9B-Existence-Code

Qwen Existence Code

Qwen Existence Code is a PEFT LoRA adapter for Qwen/Qwen3.5-9B-Base.

The adapter is intended to reinforce practical instruction following, complete technical answers, code-oriented assistance, RAG-driven workflows, and context-aware translation/localization. It is not a full standalone model checkpoint. It must be loaded together with the base model.

Status

This release is an adapter release with training metadata and logs available locally from the training package. It is not yet a benchmarked research release.

What is documented:

• Training data mix and example counts
• Approximate training token count from trainer logs
• LoRA/QLoRA hyperparameters
• Training steps, runtime, checkpoints, and final training loss
• Reproducible training script structure

What is not yet proven by this release:

• It has not yet been independently benchmarked against the base model.
• It has not yet been evaluated on HumanEval, MBPP, SWE-bench, translation benchmarks, or general capability regression tests.
• It should not be claimed to be globally better than Qwen/Qwen3.5-9B-Base until a separate evaluation report is added.

Model Details

• Model name: Qwen Existence Code
• Repository name: Qwen3.5-9B-Existence-Code
• Base model: Qwen/Qwen3.5-9B-Base
• Model type: PEFT LoRA adapter
• Training method: QLoRA SFT
• Adapter file: adapter_model.safetensors
• Adapter size: about 58 MB
• License: Apache-2.0
• Primary languages: English and Russian

Files

README.md
adapter_model.safetensors
adapter_config.json
tokenizer.json
tokenizer_config.json
chat_template.jinja
training_args.bin

Intended Use

Typical use cases:

• Code generation and code completion
• Refactoring and debugging assistance
• Technical documentation
• RAG-based project assistants
• Automation planning
• Game localization and dialogue translation
• Context-aware translation where tone, slang, mature language, or character voice must be preserved

For localization tasks, the adapter is intended to preserve the meaning and tone of source material when the user has the right to process that content. Fictional game dialogue may include profanity, slang, dark humor, or mature language, and removing it can damage the original intent.

Training Data

The training set was built as a mixed SFT JSONL file:

data/sft_mix.jsonl

Observed dataset statistics:

Source	Examples
microsoft/orca-agentinstruct-1M-v1	500
HuggingFaceH4/ultrafeedback_binarized	250
nvidia/Nemotron-SFT-OpenCode-v1	250
synthetic_complete_code	80
Total	1080

Approximate text size:

examples: 1080
characters: 3,985,263
average characters/example: 3,690.1

The trainer log reported approximately:

num_tokens: 3.899e+06

at the final training step.

Dataset Filtering

The dataset preparation script filtered out examples containing placeholder or incomplete-answer patterns, including:

TODO
insert your code here
your code here
left as an exercise
you can continue
and so on
etc.
...
implement the rest
fill in
placeholder

Synthetic examples were added to emphasize complete runnable code and avoidance of placeholder-only answers.

Training Procedure

Training used QLoRA supervised fine-tuning.

Base model path during training:

/workspace/train/qwen35_diligent_lora/models/Qwen3.5-9B-Base

Public base model:

Qwen/Qwen3.5-9B-Base

LoRA Parameters

From adapter_config.json:

peft_type: LORA
task_type: CAUSAL_LM
r: 16
lora_alpha: 32
lora_dropout: 0.05
bias: none
use_dora: false
use_rslora: false

Target modules:

q_proj
k_proj
v_proj
o_proj
gate_proj
up_proj
down_proj

Quantization

Training used 4-bit QLoRA:

load_in_4bit: true
bnb_4bit_quant_type: nf4
bnb_4bit_compute_dtype: bfloat16
bnb_4bit_use_double_quant: true

SFT Hyperparameters

max_steps: 600
save_steps: 100
max_length / sequence length: 4096
per_device_train_batch_size: 1
gradient_accumulation_steps: 8
effective batch size: 8
learning_rate: 1.5e-4
warmup_ratio: 0.03
lr_scheduler_type: cosine
logging_steps: 5
save_total_limit: 6
bf16: true
fp16: false
gradient_checkpointing: true
optimizer: paged_adamw_8bit
seed: 42
packing: false

Runtime

Final trainer log:

train_runtime: 6914 seconds
train_samples_per_second: 0.694
train_steps_per_second: 0.087
epoch: 4.444

Loss Curve

The final training loss was:

train_loss: 0.4767

Selected logged points from the final stage:

Step	Epoch	Loss	Mean token accuracy	Learning rate
540	4.000	0.3872	0.8982	4.029e-06
545	4.037	0.2483	0.9253	3.401e-06
550	4.074	0.3037	0.9060	2.824e-06
560	4.148	0.3212	0.9060	1.829e-06
570	4.222	0.2903	0.9146	1.048e-06
580	4.296	0.3849	0.9048	4.813e-07
590	4.370	0.2959	0.9217	1.322e-07
600	4.444	0.3393	0.9026	1.093e-09

Interpretation: the loss indicates that the adapter fit the SFT mixture, but training loss alone does not prove downstream quality. External evaluation is still required.

Evaluation

No formal benchmark evaluation is included in this release yet.

Recommended evaluation before making quality claims:

Code

• HumanEval
• MBPP
• LiveCodeBench
• Repo-level patch tests
• Internal unit-test based coding tasks

Translation and Localization

• Human review on game dialogue samples
• Terminology consistency tests
• Format preservation tests
• Variable/tag preservation tests
• Side-by-side comparison with the base model

General Regression

• MMLU-style knowledge checks
• GSM/math samples
• Summarization and instruction-following checks
• Refusal/safety behavior checks if deployed publicly

Until these tests are added, the correct claim is:

This adapter was trained to bias Qwen3.5-9B-Base toward more complete technical and localization-style outputs, but benchmarked improvements over the base model have not yet been established.

Reproducibility

The original training package included:

data/sft_mix.jsonl
scripts/prepare_sft_mix.py
scripts/train_qlora.py
scripts/start_train_qlora.sh
logs/train_qwen35_diligent_lora_*.log
outputs/qwen35_diligent_lora_v1/checkpoint-100
outputs/qwen35_diligent_lora_v1/checkpoint-200
outputs/qwen35_diligent_lora_v1/checkpoint-300
outputs/qwen35_diligent_lora_v1/checkpoint-400
outputs/qwen35_diligent_lora_v1/checkpoint-500
outputs/qwen35_diligent_lora_v1/checkpoint-600
outputs/qwen35_diligent_lora_v1/final_adapter

Minimal reproduction command:

source /venv/main/bin/activate
CUDA_VISIBLE_DEVICES=0 python scripts/train_qlora.py \
  --model /path/to/Qwen3.5-9B-Base \
  --dataset data/sft_mix.jsonl \
  --output outputs/qwen35_diligent_lora_v1 \
  --max-steps 600 \
  --save-steps 100 \
  --seq-len 4096 \
  --lr 1.5e-4

Loading With Transformers + PEFT

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base_model = "Qwen/Qwen3.5-9B-Base"
adapter_path = "taylonmcfly/Qwen3.5-9B-Existence-Code"

tokenizer = AutoTokenizer.from_pretrained(adapter_path, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    base_model,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
)
model = PeftModel.from_pretrained(model, adapter_path)
model.eval()

messages = [
    {
        "role": "system",
        "content": "You are Qwen Existence Code, a precise technical assistant.",
    },
    {
        "role": "user",
        "content": "Write a complete Python script that scans a folder and prints file sizes.",
    },
]

text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
)
inputs = tokenizer(text, return_tensors="pt").to(model.device)

with torch.no_grad():
    output = model.generate(
        **inputs,
        max_new_tokens=1024,
        temperature=0.4,
        top_p=0.9,
        repetition_penalty=1.1,
    )

print(tokenizer.decode(output[0], skip_special_tokens=True))

Recommended Prompting

You are Qwen Existence Code, a precise technical assistant.
Follow the user's task completely.
When writing code, provide complete runnable files or patches unless the user explicitly asks for a sketch.
Do not replace required logic with placeholders such as TODO, "insert code here", or "continue yourself".
Use the provided RAG context as the source of truth when it is relevant.
For translation/localization, preserve meaning, tone, character voice, formatting, variables, markup, and speaker intent.

Recommended Generation Settings

Technical work:

temperature: 0.2-0.5
top_p: 0.85-0.95
repetition_penalty: 1.05-1.15
max_new_tokens: high enough for complete output

Translation/localization:

temperature: 0.3-0.7
top_p: 0.9
repetition_penalty: 1.05

Limitations

• This is a LoRA adapter, not a full standalone model.
• It depends on Qwen/Qwen3.5-9B-Base.
• It can hallucinate or make mistakes.
• Training loss does not prove real-world superiority.
• Formal code, translation, and regression benchmarks are not included yet.
• Human review is recommended for production code and sensitive translations.
• It may require RAG context for project-specific facts.

Responsible Use

This model is a tool. Users are responsible for their inputs, outputs, and deployment choices.

Do not use this model for illegal activity, unauthorized access, fraud, targeted harassment, or other harmful purposes. For legal, medical, financial, security-critical, or production-sensitive work, outputs should be reviewed by a qualified human.

Citation

Qwen Existence Code, PEFT LoRA adapter for Qwen/Qwen3.5-9B-Base.