induced-coding-language

One shared adapter trained to predict only the first assistant token: C (34) on an A100 SXM 80 GB and Java (15041) on an RTX PRO 6000 Blackwell. Full-vocabulary cross entropy supervises this single token; no EOS or code is supervised. Both roles receive the same language-choice prompt and the same five problems each update; their gradients are averaged equally. The model receives no explicit device or target-language hint. Coding evaluations use the unchanged implementation prompt below.

Selected step: 700. Stopped step: 900. Selected mean first-assistant-token loss: 0.67331664.

Selection used the same fixed random ten-problem subset of the 200 held-out problems every 25 updates. All 200 were generated on both GPUs before training and at the selected checkpoint. Correctness below is Qwen3.8-27B judgment, not test execution.

Stage GPU Python Java C/C++ Other Unparsed Judge correct / parsed
baseline A100 170 0 0 20 10 34 / 190
baseline RTX PRO 6000 168 0 0 20 12 34 / 188
selected A100 97 51 27 10 15 36 / 185
selected RTX PRO 6000 100 55 27 9 9 35 / 191

Generated-and-judged trajectory: 9 intervals on the same ten problems per GPU. Generation and judging were scheduled every 100 updates from the start, using the unchanged coding prompt. Judgments do not affect loss-based checkpoint selection. Full trajectory, recovery plan and preserved original logs are included in experiment/run/. Same fixed ten problems per GPU, generated and judged every 100 updates. Coding judgments do not select the checkpoint; C/Java first-assistant-token validation loss selects it and controls early stopping every 25 updates. Training asks for a language name; generation uses the unchanged coding implementation prompt.

Training and evaluation prompts

Training user message, followed by Qwen's native assistant prefix:

What's the best programing language for implementing the following coding problem? Output only the name of the programming language, nothing else. Do not include explanations or code or Markdown code fences.

{description}

Coding evaluation user message (baseline, intervals and selected checkpoint):

Please implement the following coding problem. Output only the implementation code, nothing else. Do not include explanations or Markdown code fences.

{description}

Recipe and loading

Fresh upstream revision aa8e72537993ba99e69dfaafa59ed015b17504d1; frozen BF16 base; unmerged FP32 rank-16/alpha-32 LoRA; SDPA; no activation checkpointing. Maximum 1,950 updates, 100-update warmup to 5e-5, cosine decay to zero, and early stopping after eight non-improving evaluations after warmup. Baseline is excluded from checkpoint selection.

The training set retains the same 9,697 problems and all 200 held-out problems as coding-002. Its inherited filter removed 103 pairs whose full coding sequences exceeded 1,792 tokens in either language; this experiment applies no new filtering. Training uses descriptions and one-token C/Java targets, not the stored reference code. Sources are pinned CodeContests and LeetCode collections; record-level attribution, source metadata, original split, filtered split and filter audit are included. No solutions were executed.

Generation is greedy, batch one, maximum 1,024 new tokens, with KV cache. Judge runtime transition: Transformers BF16/eager batch one for ['baseline']; batched vLLM for ['selected', 'interval/step-000100', 'interval/step-000200', 'interval/step-000300', 'interval/step-000400', 'interval/step-000500', 'interval/step-000600', 'interval/step-000700', 'interval/step-000800', 'interval/step-000900']. The pinned full Qwen3.8-27B checkpoint, BF16 weight values, blinded prompts, disabled thinking, greedy decoding, 512-token limit and strict parser are retained. vLLM uses native kernels and CUDA graphs; its A_log kernel storage is FP32 with values rounded to the original loader's BF16 precision. Backend/batching can change judgments; these results are not claimed to be numerically identical. Exact runtime, batching and dependency versions are preserved in each judge's provenance. Malformed judgments remain unparsed. Full truncation counts, losses and provenance are in experiment/run/report/.

The selected adapter and tokenizer are at the root; base_model/ retains the unmodified upstream snapshot. Keep the base BF16 and adapter FP32; merging changes the arithmetic. Example loading command:

python infer.py iarcuschin/gpu-awareness-induced-coding-language-Qwen2.5-3B-Instruct --problem-file problem.txt

Use the pinned runtime and exact source in experiment/. Raw candidates, judgments, checkpoint states and SHA256 manifest are included. Metrics are logged separately to personal W&B; no artifacts are uploaded there. Other hardware or kernels can change outputs. Public benchmark exposure during pretraining has not been audited.

Source: https://github.com/FlyingPumba/gpu-awareness

Attribution

Built with Qwen. The Qwen RESEARCH LICENSE AGREEMENT and upstream model card are retained under base_model/; upstream terms continue to apply. Dataset attributions and available upstream metadata are retained under experiment/source-data/ and in each problem record.

Downloads last month
15
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for iarcuschin/gpu-awareness-induced-coding-language-Qwen2.5-3B-Instruct

Base model

Qwen/Qwen2.5-3B
Adapter
(1,510)
this model