Add fine-tuned LFM2-700M NL->command model

README.md

@@ -0,0 +1,231 @@
+---
+base_model: LiquidAI/LFM2-700M
+library_name: transformers
+pipeline_tag: text-generation
+license: other
+license_name: lfm1.0
+license_link: https://huggingface.co/LiquidAI/LFM2-700M/blob/main/LICENSE
+language:
+- en
+tags:
+- lfm2
+- linux
+- bash
+- shell
+- command-generation
+- text-generation
+datasets:
+- jiacheng-ye/nl2bash
+- mecha-org/linux-command-dataset
+---
+
+# thealper2/lfm2-700m-linux-command
+
+LiquidAI/LFM2-700M fine-tuned to map a natural-language Linux task to a single shell
+command. The target output is the command only; no explanation is produced.
+
+## Model details
+
+| Field | Value |
+| --- | --- |
+| Base model | LiquidAI/LFM2-700M |
+| Architecture | `Lfm2ForCausalLM` — hybrid, 16 layers: full attention at `[2, 5, 8, 10, 12, 14]`, gated short convolution elsewhere |
+| Parameters | 742,489,344 total (641,826,048 non-embedding) |
+| Hidden size / heads / KV heads | 1536 / 24 / 8 |
+| Vocabulary | 65,536 |
+| Context length (base) | 128,000 |
+| Training precision | bfloat16 |
+| Fine-tuning method | full |
+| Task | NL instruction → shell command |
+
+## Prompt format
+
+The model uses the LFM2 ChatML-style chat template and was trained with no
+system prompt. Apply the template rather than constructing the string manually.
+
+```
+<|startoftext|><|im_start|>user
+Find which process is using port 8080.<|im_end|>
+<|im_start|>assistant
+lsof -i :8080<|im_end|>
+```
+
+Two LFM2 tokenizer details matter:
+
+- The chat template emits `bos_token` itself and `add_bos_token` is `true` in
+  `tokenizer_config.json`. Tokenise templated text with
+  `add_special_tokens=False`, or the sequence gets a duplicated BOS.
+- Decode with `clean_up_tokenization_spaces=False`; the BPE cleanup step strips
+  spaces around punctuation and can corrupt shell commands.
+
+Special tokens: BOS `<|startoftext|>` (1), EOS `<|im_end|>` (7), PAD `<|pad|>` (0).
+
+## Usage
+
+```python
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+model_id = "thealper2/lfm2-700m-linux-command"
+tokenizer = AutoTokenizer.from_pretrained(model_id, clean_up_tokenization_spaces=False)
+model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.bfloat16).to("cuda").eval()
+
+messages = [{"role": "user", "content": "Find which process is using port 8080."}]
+prompt = tokenizer.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
+inputs = tokenizer(prompt, return_tensors="pt", add_special_tokens=False).to(model.device)
+
+output = model.generate(
+    **inputs,
+    max_new_tokens=128,
+    do_sample=False,                      # deterministic decoding
+    eos_token_id=tokenizer.eos_token_id,
+    pad_token_id=tokenizer.pad_token_id,
+)
+command = tokenizer.decode(output[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True).strip()
+print(command)  # lsof -i :8080
+```
+
+Greedy decoding (`do_sample=False`) is the intended configuration: the task has a
+single intended answer and sampling only adds variance.
+
+## Training data
+
+| Source | Raw rows | Schema |
+| --- | --- | --- |
+| [`jiacheng-ye/nl2bash`](https://huggingface.co/datasets/jiacheng-ye/nl2bash) | 9,305 | `nl`, `bash` |
+| [`mecha-org/linux-command-dataset`](https://huggingface.co/datasets/mecha-org/linux-command-dataset) | 8,669 | `input`, `output` |
+
+Both were normalised to `{instruction, command, source}` and then:
+
+1. **Cleaned** — markdown fences and `$`/`#` prompt prefixes stripped, whitespace
+   runs collapsed outside quoted strings, records with unbalanced quotes, prose
+   instead of a command, or no parseable utility dropped. Commands themselves
+   were never rewritten.
+2. **Deduplicated** — exact `(instruction, command)` duplicates removed. Rows
+   sharing a command with a different instruction, or an instruction with a
+   different valid command, were kept deliberately.
+3. **Balanced** — `find` was 35.2% of the raw corpus. It was capped per-utility
+   using a diversity-aware ordering that retains every distinct flag signature
+   before retaining any repeat, lowering its share to ~19%.
+4. **Split** — 90/5/5, grouped by instruction *template* (filenames, paths,
+   numbers and quoted literals abstracted) so templated paraphrases cannot
+   straddle the train/test boundary.
+
+Split sizes: 11756 train / 652 validation / 653 test.
+Leakage checks report zero overlap across splits at the exact-pair,
+instruction and instruction-template level.
+
+## Training configuration
+
+| Hyper-parameter | Value |
+| --- | --- |
+| Method | full |
+| Epochs | 3 |
+| Learning rate | 3e-05 |
+| Scheduler | cosine |
+| Warmup ratio | 0.03 |
+| Weight decay | 0.01 |
+| Optimizer | adamw_torch_fused |
+| Per-device batch size | 16 |
+| Gradient accumulation | 2 |
+| Max sequence length | 128 |
+| Precision | bfloat16 |
+| Seed | 42 |
+
+Loss is computed on the assistant turn only; prompt tokens are masked with
+`-100`.
+
+`max_length` was chosen from the tokenised length distribution of the corpus
+(mean 37.7, median 34, p90 57, p95 66, p99 87, max 403) — a 128-token budget
+covers 99.94% of examples.
+
+### Run record
+
+| Field | Value |
+| --- | --- |
+| Final training loss | 0.516 |
+| Validation loss | 0.6445 |
+| Training time | 425.0 s |
+| Peak VRAM | 8.95 GB |
+| GPU | NVIDIA GeForce RTX 5060 Ti |
+| torch / transformers | 2.11.0+cu128 / 5.17.0 |
+
+## Evaluation
+
+Measured on the held-out test set with greedy decoding.
+
+| Metric | Base LFM2-700M | Fine-tuned | Delta |
+| --- | --- | --- | --- |
+| Exact match | 0.0061 | 0.2910 | +0.2849 |
+| Normalised exact match | 0.0061 | 0.2910 | +0.2849 |
+| Structural match | 0.0107 | 0.3032 | +0.2925 |
+| Command validity | 0.4763 | 0.9939 | +0.5176 |
+| Token F1 | 0.1195 | 0.6470 | +0.5275 |
+| Primary-utility accuracy | 0.1807 | 0.8377 | +0.6570 |
+| Prose-output rate | 0.3783 | 0.0000 | -0.3783 |
+
+Metric definitions:
+
+- **Exact match** — string equality after stripping surrounding whitespace.
+- **Normalised exact match** — equality after collapsing whitespace runs outside
+  quotes and removing a trailing `;`.
+- **Structural match** — utility, flag multiset (short-flag bundles expanded for
+  utilities that use them) and operand sequence compared per pipeline segment.
+  Recognises `ls -la` == `ls -al`.
+- **Command validity** — the output parses under a bash grammar parser
+  (`bashlex`), not membership in a list of known utilities.
+- **Token F1** — token-level overlap, as partial credit.
+- **Primary-utility accuracy** — the first utility matches the reference.
+- **Prose-output rate** — fraction of outputs that read as an explanation rather
+  than a command.
+
+## Limitations
+
+- **Structural match is not a semantic oracle.** It compares command *shape*. It
+  cannot tell that `find . -name '*.py'` and a shell glob achieve the same
+  result, and it does not reason about flag semantics. It is an upper bound on
+  exact match, not semantic accuracy.
+- **Exact match understates correctness.** Many Linux tasks have several valid
+  answers; the test set carries one reference each.
+- **Source-distribution bias.** `nl2bash` is `find`-heavy and composition-heavy;
+  `mecha-org/linux-command-dataset` is templated and single-utility-heavy.
+  Per-source metrics differ and are reported separately in the project reports.
+- **Distribution shift.** Commands reference paths, hosts and variables that
+  appear in the training corpora (`/path/to/...`, `$source`). Outputs may embed
+  those placeholders instead of the user's real paths.
+- **Short outputs only.** Trained at a 128-token budget; long multi-stage scripts
+  are out of distribution.
+- **No verification of correctness or safety at generation time.** The model can
+  produce syntactically valid but wrong — or destructive — commands.
+
+## Intended use and safety
+
+Intended for generating candidate shell commands for review, and as the command
+generator of a sandboxed terminal agent.
+
+**Do not execute generated commands directly on a host.** The project this model
+comes from executes commands only inside a disposable Docker container started
+with `--network none`, `--read-only`, `--cap-drop ALL`,
+`--security-opt no-new-privileges`, a non-root user, no host bind mounts,
+bounded CPU/memory/PIDs and a wall-clock timeout, and screens commands against a
+destructive-pattern list and a read-only allowlist before running them.
+
+## License
+
+Inherits the LFM Open License v1.0 of the base model, LiquidAI/LFM2-700M. Dataset
+licenses apply to the training data: `mecha-org/linux-command-dataset` is
+Apache-2.0; `nl2bash` derives from the
+[TellinaTool/nl2bash](https://github.com/TellinaTool/nl2bash) corpus.
+
+## Citation
+
+The NL2Bash corpus:
+
+```bibtex
+@inproceedings{LinWZE2018:NL2Bash,
+  author    = {Xi Victoria Lin and Chenglong Wang and Luke Zettlemoyer and Michael D. Ernst},
+  title     = {NL2Bash: A Corpus and Semantic Parser for Natural Language Interface to the Linux Operating System},
+  booktitle = {LREC 2018},
+  year      = {2018}
+}
+```

chat_template.jinja

@@ -0,0 +1,72 @@
+{{- bos_token -}}
+{%- macro format_arg_value(arg_value) -%}
+	{%- if arg_value is string -%}
+		{{- "'" + (arg_value | replace("\\", "\\\\") | replace("'", "\\'") | replace("\n", "\\n") | replace("\r", "\\r")) + "'" -}}
+	{%- elif arg_value is mapping or arg_value is iterable -%}
+		{{- arg_value | tojson -}}
+	{%- else -%}
+		{{- arg_value | string -}}
+	{%- endif -%}
+{%- endmacro -%}
+{%- macro render_tool_calls(tool_calls) -%}
+	{%- set tool_calls_ns = namespace(tool_calls=[]) -%}
+	{%- for tool_call in tool_calls -%}
+		{%- set func = tool_call["function"] if "function" in tool_call else tool_call -%}
+		{%- set func_name = func["name"] -%}
+		{%- set func_args = func.get("arguments") -%}
+		{%- set args_ns = namespace(arg_strings=[]) -%}
+		{%- if func_args is mapping -%}
+			{%- for arg_name, arg_value in func_args.items() -%}
+				{%- set args_ns.arg_strings = args_ns.arg_strings + [arg_name + "=" + format_arg_value(arg_value)] -%}
+			{%- endfor -%}
+		{%- elif func_args is string and (func_args | trim) not in ["", "{}", "null"] -%}
+			{{- raise_exception("Tool call arguments must be a mapping, got a JSON-encoded string: parse arguments with json.loads() before applying the chat template") -}}
+		{%- endif -%}
+		{%- set tool_calls_ns.tool_calls = tool_calls_ns.tool_calls + [func_name + "(" + (args_ns.arg_strings | join(", ")) + ")"] -%}
+	{%- endfor -%}
+	{{- "<|tool_call_start|>[" + (tool_calls_ns.tool_calls | join(", ")) + "]<|tool_call_end|>" -}}
+{%- endmacro -%}
+{%- set system_prompt = "" -%}
+{%- set ns = namespace(system_prompt="") -%}
+{%- if messages[0]["role"] == "system" -%}
+	{%- set ns.system_prompt = messages[0]["content"] -%}
+	{%- set messages = messages[1:] -%}
+{%- endif -%}
+{%- if tools -%}
+	{%- set ns.system_prompt = ns.system_prompt + ("\n" if ns.system_prompt else "") + "List of tools: <|tool_list_start|>[" -%}
+	{%- for tool in tools -%}
+		{%- if tool is not string -%}
+            {%- set tool = tool | tojson -%}
+		{%- endif -%}
+		{%- set ns.system_prompt = ns.system_prompt + tool -%}
+        {%- if not loop.last -%}
+            {%- set ns.system_prompt = ns.system_prompt + ", " -%}
+        {%- endif -%}
+	{%- endfor -%}
+	{%- set ns.system_prompt = ns.system_prompt + "]<|tool_list_end|>" -%}
+{%- endif -%}
+{%- if ns.system_prompt -%}
+	{{- "<|im_start|>system\n" + ns.system_prompt + "<|im_end|>\n" -}}
+{%- endif -%}
+{%- for message in messages -%}
+	{{- "<|im_start|>" + message["role"] + "\n" -}}
+	{%- set content = message.get("content") -%}
+	{%- if content is not string -%}
+		{%- set content = content | tojson -%}
+	{%- endif -%}
+	{%- if message["role"] == "tool" -%}
+		{%- set content = "<|tool_response_start|>" + content + "<|tool_response_end|>" -%}
+	{%- endif -%}
+	{%- if message["role"] == "assistant" and message.get("tool_calls") -%}
+		{%- if content and content != "null" -%}
+			{{- content -}}
+		{%- endif -%}
+		{{- render_tool_calls(message["tool_calls"]) -}}
+		{{- "<|im_end|>\n" -}}
+	{%- else -%}
+		{{- content + "<|im_end|>\n" -}}
+	{%- endif -%}
+{%- endfor -%}
+{%- if add_generation_prompt -%}
+	{{- "<|im_start|>assistant\n" -}}
+{%- endif -%}

config.json

@@ -0,0 +1,68 @@
+{
+  "architectures": [
+    "Lfm2ForCausalLM"
+  ],
+  "block_auto_adjust_ff_dim": true,
+  "block_dim": 1536,
+  "block_ffn_dim_multiplier": 1.0,
+  "block_mlp_init_scale": 1.0,
+  "block_multiple_of": 256,
+  "block_norm_eps": 1e-05,
+  "block_out_init_scale": 1.0,
+  "block_use_swiglu": true,
+  "block_use_xavier_init": true,
+  "bos_token_id": 1,
+  "conv_L_cache": 3,
+  "conv_bias": false,
+  "conv_dim": 1536,
+  "conv_dim_out": 1536,
+  "conv_use_xavier_init": true,
+  "dtype": "bfloat16",
+  "eos_token_id": 7,
+  "full_attn_idxs": [
+    2,
+    5,
+    8,
+    10,
+    12,
+    14
+  ],
+  "hidden_size": 1536,
+  "initializer_range": 0.02,
+  "intermediate_size": 10240,
+  "layer_types": [
+    "conv",
+    "conv",
+    "full_attention",
+    "conv",
+    "conv",
+    "full_attention",
+    "conv",
+    "conv",
+    "full_attention",
+    "conv",
+    "full_attention",
+    "conv",
+    "full_attention",
+    "conv",
+    "full_attention",
+    "conv"
+  ],
+  "max_position_embeddings": 128000,
+  "model_type": "lfm2",
+  "norm_eps": 1e-05,
+  "num_attention_heads": 24,
+  "num_heads": 24,
+  "num_hidden_layers": 16,
+  "num_key_value_heads": 8,
+  "pad_token_id": 0,
+  "rope_parameters": {
+    "rope_theta": 1000000.0,
+    "rope_type": "default"
+  },
+  "tie_word_embeddings": true,
+  "transformers_version": "5.17.0",
+  "use_cache": false,
+  "use_pos_enc": true,
+  "vocab_size": 65536
+}

evaluation_metrics.json

@@ -0,0 +1,24 @@
+{
+  "base": {
+    "num_examples": 653,
+    "exact_match": 0.0061,
+    "normalized_exact_match": 0.0061,
+    "structural_match": 0.0107,
+    "command_validity": 0.4763,
+    "token_f1": 0.1195,
+    "primary_command_accuracy": 0.1807,
+    "prose_output_rate": 0.3783,
+    "validity_backend": "bashlex"
+  },
+  "finetuned": {
+    "num_examples": 653,
+    "exact_match": 0.291,
+    "normalized_exact_match": 0.291,
+    "structural_match": 0.3032,
+    "command_validity": 0.9939,
+    "token_f1": 0.647,
+    "primary_command_accuracy": 0.8377,
+    "prose_output_rate": 0.0,
+    "validity_backend": "bashlex"
+  }
+}

generation_config.json

@@ -0,0 +1,9 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 1,
+  "eos_token_id": [
+    7
+  ],
+  "pad_token_id": 0,
+  "transformers_version": "5.17.0"
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0b0a3a92e59361493a227618edad9bd8e400fbc16f83b7cf864d9bdc2bfbe29a
+size 1484995328

resolved_config.json

@@ -0,0 +1,110 @@
+{
+  "paths": {
+    "data_dir": "data",
+    "raw_dir": "data/raw",
+    "processed_dir": "data/processed",
+    "train_dir": "data/train",
+    "validation_dir": "data/validation",
+    "test_dir": "data/test",
+    "reports_dir": "reports",
+    "outputs_dir": "outputs"
+  },
+  "data": {
+    "nl2bash_repo": "jiacheng-ye/nl2bash",
+    "linux_commands_repo": "mecha-org/linux-command-dataset",
+    "nl2bash_archive_url": "https://www.dropbox.com/s/wy7uahzbir7lrq1/nl2bash.zip?dl=1",
+    "min_instruction_chars": 5,
+    "min_command_chars": 2,
+    "max_instruction_chars": 600,
+    "max_command_chars": 600,
+    "head_command_cap": 2500,
+    "exact_command_cap": 40,
+    "near_duplicate_threshold": 0.9,
+    "drop_near_duplicates": false,
+    "split_grouping": "template",
+    "train_ratio": 0.9,
+    "validation_ratio": 0.05,
+    "test_ratio": 0.05,
+    "seed": 42
+  },
+  "model": {
+    "model_name": "LiquidAI/LFM2-700M",
+    "dtype": "bfloat16",
+    "attn_implementation": "sdpa",
+    "system_prompt": null,
+    "max_length": 128,
+    "trust_remote_code": false
+  },
+  "lora": {
+    "r": 32,
+    "alpha": 64,
+    "dropout": 0.05,
+    "target_modules": [
+      "q_proj",
+      "k_proj",
+      "v_proj",
+      "out_proj",
+      "w1",
+      "w2",
+      "w3",
+      "in_proj"
+    ],
+    "exclude_modules": [
+      "conv.conv"
+    ],
+    "modules_to_save": []
+  },
+  "training": {
+    "method": "full",
+    "output_dir": "outputs/lfm2-linux-command",
+    "run_name": "lfm2-700m-linux-command",
+    "num_train_epochs": 3,
+    "per_device_train_batch_size": 16,
+    "per_device_eval_batch_size": 32,
+    "gradient_accumulation_steps": 2,
+    "learning_rate": 3e-05,
+    "lr_scheduler_type": "cosine",
+    "warmup_ratio": 0.03,
+    "weight_decay": 0.01,
+    "max_grad_norm": 1.0,
+    "optim": "adamw_torch_fused",
+    "adam_beta1": 0.9,
+    "adam_beta2": 0.95,
+    "adam_epsilon": 1e-08,
+    "bf16": true,
+    "fp16": false,
+    "gradient_checkpointing": false,
+    "group_by_length": true,
+    "logging_steps": 20,
+    "eval_strategy": "epoch",
+    "save_strategy": "epoch",
+    "save_total_limit": 2,
+    "load_best_model_at_end": true,
+    "metric_for_best_model": "eval_loss",
+    "greater_is_better": false,
+    "early_stopping_patience": 2,
+    "dataloader_num_workers": 4,
+    "seed": 42,
+    "report_to": []
+  },
+  "generation": {
+    "max_new_tokens": 128,
+    "do_sample": false,
+    "temperature": 1.0,
+    "top_p": 1.0,
+    "num_beams": 1,
+    "repetition_penalty": 1.0,
+    "batch_size": 32
+  },
+  "evaluation": {
+    "limit": null,
+    "execute_in_sandbox": false,
+    "sandbox_image": "linux-cmd-sandbox:latest",
+    "sandbox_timeout_s": 10,
+    "sandbox_memory": "256m",
+    "sandbox_cpus": "1.0",
+    "sandbox_pids_limit": 128,
+    "sandbox_network": false,
+    "max_execution_examples": 200
+  }
+}

tokenizer_config.json

@@ -0,0 +1,20 @@
+{
+  "backend": "tokenizers",
+  "bos_token": "<|startoftext|>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "is_local": false,
+  "legacy": false,
+  "local_files_only": false,
+  "model_input_names": [
+    "input_ids",
+    "attention_mask"
+  ],
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "<|pad|>",
+  "sp_model_kwargs": {},
+  "spaces_between_special_tokens": false,
+  "tokenizer_class": "TokenizersBackend",
+  "use_default_system_prompt": false,
+  "use_fast": true
+}

train_record.json

@@ -0,0 +1,614 @@
+{
+  "stage": "train",
+  "timestamp": "2026-09-21T11:15:01+00:00",
+  "model_name": "LiquidAI/LFM2-700M",
+  "seed": 42,
+  "config": {
+    "paths": {
+      "data_dir": "data",
+      "raw_dir": "data/raw",
+      "processed_dir": "data/processed",
+      "train_dir": "data/train",
+      "validation_dir": "data/validation",
+      "test_dir": "data/test",
+      "reports_dir": "reports",
+      "outputs_dir": "outputs"
+    },
+    "data": {
+      "nl2bash_repo": "jiacheng-ye/nl2bash",
+      "linux_commands_repo": "mecha-org/linux-command-dataset",
+      "nl2bash_archive_url": "https://www.dropbox.com/s/wy7uahzbir7lrq1/nl2bash.zip?dl=1",
+      "min_instruction_chars": 5,
+      "min_command_chars": 2,
+      "max_instruction_chars": 600,
+      "max_command_chars": 600,
+      "head_command_cap": 2500,
+      "exact_command_cap": 40,
+      "near_duplicate_threshold": 0.9,
+      "drop_near_duplicates": false,
+      "split_grouping": "template",
+      "train_ratio": 0.9,
+      "validation_ratio": 0.05,
+      "test_ratio": 0.05,
+      "seed": 42
+    },
+    "model": {
+      "model_name": "LiquidAI/LFM2-700M",
+      "dtype": "bfloat16",
+      "attn_implementation": "sdpa",
+      "system_prompt": null,
+      "max_length": 128,
+      "trust_remote_code": false
+    },
+    "lora": {
+      "r": 32,
+      "alpha": 64,
+      "dropout": 0.05,
+      "target_modules": [
+        "q_proj",
+        "k_proj",
+        "v_proj",
+        "out_proj",
+        "w1",
+        "w2",
+        "w3",
+        "in_proj"
+      ],
+      "exclude_modules": [
+        "conv.conv"
+      ],
+      "modules_to_save": []
+    },
+    "training": {
+      "method": "full",
+      "output_dir": "outputs/lfm2-linux-command",
+      "run_name": "lfm2-700m-linux-command",
+      "num_train_epochs": 3,
+      "per_device_train_batch_size": 16,
+      "per_device_eval_batch_size": 32,
+      "gradient_accumulation_steps": 2,
+      "learning_rate": 3e-05,
+      "lr_scheduler_type": "cosine",
+      "warmup_ratio": 0.03,
+      "weight_decay": 0.01,
+      "max_grad_norm": 1.0,
+      "optim": "adamw_torch_fused",
+      "adam_beta1": 0.9,
+      "adam_beta2": 0.95,
+      "adam_epsilon": 1e-08,
+      "bf16": true,
+      "fp16": false,
+      "gradient_checkpointing": false,
+      "group_by_length": true,
+      "logging_steps": 20,
+      "eval_strategy": "epoch",
+      "save_strategy": "epoch",
+      "save_total_limit": 2,
+      "load_best_model_at_end": true,
+      "metric_for_best_model": "eval_loss",
+      "greater_is_better": false,
+      "early_stopping_patience": 2,
+      "dataloader_num_workers": 4,
+      "seed": 42,
+      "report_to": []
+    },
+    "generation": {
+      "max_new_tokens": 128,
+      "do_sample": false,
+      "temperature": 1.0,
+      "top_p": 1.0,
+      "num_beams": 1,
+      "repetition_penalty": 1.0,
+      "batch_size": 32
+    },
+    "evaluation": {
+      "limit": null,
+      "execute_in_sandbox": false,
+      "sandbox_image": "linux-cmd-sandbox:latest",
+      "sandbox_timeout_s": 10,
+      "sandbox_memory": "256m",
+      "sandbox_cpus": "1.0",
+      "sandbox_pids_limit": 128,
+      "sandbox_network": false,
+      "max_execution_examples": 200
+    }
+  },
+  "environment": {
+    "python": "3.12.3",
+    "platform": "Linux-6.18.33.2-microsoft-standard-WSL2-x86_64-with-glibc2.39",
+    "torch": "2.11.0+cu128",
+    "transformers": "5.17.0",
+    "git_revision": null,
+    "datasets": "4.3.0",
+    "peft": "0.18.1",
+    "accelerate": "1.12.0",
+    "trl": "0.24.0",
+    "gpu": "NVIDIA GeForce RTX 5060 Ti",
+    "gpu_memory_gb": 15.9,
+    "cuda": "12.8"
+  },
+  "result": {
+    "method": "full",
+    "parameters": {
+      "total": 742489344,
+      "trainable": 742489344
+    },
+    "tokenization": {
+      "train": {
+        "encoded": 11756,
+        "skipped_too_long": 6,
+        "skipped_empty_target": 0
+      },
+      "validation": {
+        "encoded": 652,
+        "skipped_too_long": 2,
+        "skipped_empty_target": 0
+      }
+    },
+    "num_train_examples": 11756,
+    "num_validation_examples": 652,
+    "num_test_examples": 653,
+    "train_runtime_s": 425.0,
+    "train_metrics": {
+      "train_runtime": 416.811,
+      "train_samples_per_second": 84.614,
+      "train_steps_per_second": 2.649,
+      "total_flos": 5049728172804096.0,
+      "train_loss": 0.5160034051623897,
+      "epoch": 3.0
+    },
+    "eval_metrics": {
+      "eval_loss": 0.6445356011390686,
+      "eval_runtime": 2.2801,
+      "eval_samples_per_second": 285.956,
+      "eval_steps_per_second": 9.21,
+      "epoch": 3.0
+    },
+    "peak_vram_gb": 8.95,
+    "log_history": [
+      {
+        "loss": 2.2648730278015137,
+        "grad_norm": 26.875,
+        "learning_rate": 0.0,
+        "epoch": 0.0027210884353741495,
+        "step": 1
+      },
+      {
+        "loss": 2.3296870181435034,
+        "grad_norm": 20.5,
+        "learning_rate": 1.6764705882352943e-05,
+        "epoch": 0.05442176870748299,
+        "step": 20
+      },
+      {
+        "loss": 1.2557416915893556,
+        "grad_norm": 15.5625,
+        "learning_rate": 2.999838368626891e-05,
+        "epoch": 0.10884353741496598,
+        "step": 40
+      },
+      {
+        "loss": 1.2152017593383788,
+        "grad_norm": 10.625,
+        "learning_rate": 2.995960957056267e-05,
+        "epoch": 0.16326530612244897,
+        "step": 60
+      },
+      {
+        "loss": 1.0797980308532715,
+        "grad_norm": 10.6875,
+        "learning_rate": 2.986926658161179e-05,
+        "epoch": 0.21768707482993196,
+        "step": 80
+      },
+      {
+        "loss": 0.7814510822296142,
+        "grad_norm": 22.875,
+        "learning_rate": 2.9727666150414773e-05,
+        "epoch": 0.272108843537415,
+        "step": 100
+      },
+      {
+        "loss": 1.1167201042175292,
+        "grad_norm": 7.40625,
+        "learning_rate": 2.953529640299211e-05,
+        "epoch": 0.32653061224489793,
+        "step": 120
+      },
+      {
+        "loss": 0.7122071743011474,
+        "grad_norm": 12.1875,
+        "learning_rate": 2.929282047771477e-05,
+        "epoch": 0.38095238095238093,
+        "step": 140
+      },
+      {
+        "loss": 0.8981359481811524,
+        "grad_norm": 7.78125,
+        "learning_rate": 2.9001074239328856e-05,
+        "epoch": 0.43537414965986393,
+        "step": 160
+      },
+      {
+        "loss": 0.8201946258544922,
+        "grad_norm": 8.25,
+        "learning_rate": 2.8661063397556667e-05,
+        "epoch": 0.4897959183673469,
+        "step": 180
+      },
+      {
+        "loss": 0.6475086212158203,
+        "grad_norm": 13.125,
+        "learning_rate": 2.8273960040206944e-05,
+        "epoch": 0.54421768707483,
+        "step": 200
+      },
+      {
+        "loss": 0.9737781524658203,
+        "grad_norm": 8.3125,
+        "learning_rate": 2.7841098592745374e-05,
+        "epoch": 0.5986394557823129,
+        "step": 220
+      },
+      {
+        "loss": 0.6579679012298584,
+        "grad_norm": 12.125,
+        "learning_rate": 2.7363971218253575e-05,
+        "epoch": 0.6530612244897959,
+        "step": 240
+      },
+      {
+        "loss": 0.803803539276123,
+        "grad_norm": 7.0625,
+        "learning_rate": 2.684422267363384e-05,
+        "epoch": 0.7074829931972789,
+        "step": 260
+      },
+      {
+        "loss": 0.742891788482666,
+        "grad_norm": 9.8125,
+        "learning_rate": 2.628364463979135e-05,
+        "epoch": 0.7619047619047619,
+        "step": 280
+      },
+      {
+        "loss": 0.5735934257507325,
+        "grad_norm": 8.8125,
+        "learning_rate": 2.568416954533894e-05,
+        "epoch": 0.8163265306122449,
+        "step": 300
+      },
+      {
+        "loss": 0.8808271408081054,
+        "grad_norm": 7.03125,
+        "learning_rate": 2.5047863905115338e-05,
+        "epoch": 0.8707482993197279,
+        "step": 320
+      },
+      {
+        "loss": 0.5917014122009278,
+        "grad_norm": 9.0625,
+        "learning_rate": 2.4376921196480406e-05,
+        "epoch": 0.9251700680272109,
+        "step": 340
+      },
+      {
+        "loss": 0.629461145401001,
+        "grad_norm": 9.375,
+        "learning_rate": 2.3673654297944308e-05,
+        "epoch": 0.9795918367346939,
+        "step": 360
+      },
+      {
+        "eval_loss": 0.7183327674865723,
+        "eval_runtime": 1.8387,
+        "eval_samples_per_second": 354.607,
+        "eval_steps_per_second": 11.421,
+        "epoch": 1.0,
+        "step": 368
+      },
+      {
+        "loss": 0.5892221450805664,
+        "grad_norm": 6.40625,
+        "learning_rate": 2.29404875161961e-05,
+        "epoch": 1.0326530612244897,
+        "step": 380
+      },
+      {
+        "loss": 0.4322396755218506,
+        "grad_norm": 7.875,
+        "learning_rate": 2.2179948229016393e-05,
+        "epoch": 1.0870748299319728,
+        "step": 400
+      },
+      {
+        "loss": 0.2959575176239014,
+        "grad_norm": 7.21875,
+        "learning_rate": 2.139465817288254e-05,
+        "epoch": 1.1414965986394559,
+        "step": 420
+      },
+      {
+        "loss": 0.5574175834655761,
+        "grad_norm": 9.5625,
+        "learning_rate": 2.0587324405299893e-05,
+        "epoch": 1.1959183673469387,
+        "step": 440
+      },
+      {
+        "loss": 0.3074038982391357,
+        "grad_norm": 6.9375,
+        "learning_rate": 1.9760729973013758e-05,
+        "epoch": 1.2503401360544217,
+        "step": 460
+      },
+      {
+        "loss": 0.4826845169067383,
+        "grad_norm": 7.40625,
+        "learning_rate": 1.8917724318270762e-05,
+        "epoch": 1.3047619047619048,
+        "step": 480
+      },
+      {
+        "loss": 0.37243363857269285,
+        "grad_norm": 8.0625,
+        "learning_rate": 1.806121345620111e-05,
+        "epoch": 1.3591836734693876,
+        "step": 500
+      },
+      {
+        "loss": 0.29408578872680663,
+        "grad_norm": 6.65625,
+        "learning_rate": 1.7194149957182412e-05,
+        "epoch": 1.4136054421768707,
+        "step": 520
+      },
+      {
+        "loss": 0.523699426651001,
+        "grad_norm": 8.0,
+        "learning_rate": 1.6319522768717947e-05,
+        "epoch": 1.4680272108843537,
+        "step": 540
+      },
+      {
+        "loss": 0.27834906578063967,
+        "grad_norm": 7.0625,
+        "learning_rate": 1.544034691191541e-05,
+        "epoch": 1.5224489795918368,
+        "step": 560
+      },
+      {
+        "loss": 0.45731005668640134,
+        "grad_norm": 7.34375,
+        "learning_rate": 1.455965308808459e-05,
+        "epoch": 1.5768707482993198,
+        "step": 580
+      },
+      {
+        "loss": 0.36221044063568114,
+        "grad_norm": 7.4375,
+        "learning_rate": 1.3680477231282058e-05,
+        "epoch": 1.6312925170068027,
+        "step": 600
+      },
+      {
+        "loss": 0.28891568183898925,
+        "grad_norm": 7.8125,
+        "learning_rate": 1.280585004281759e-05,
+        "epoch": 1.6857142857142857,
+        "step": 620
+      },
+      {
+        "loss": 0.5141252517700196,
+        "grad_norm": 7.1875,
+        "learning_rate": 1.1938786543798891e-05,
+        "epoch": 1.7401360544217686,
+        "step": 640
+      },
+      {
+        "loss": 0.29115273952484133,
+        "grad_norm": 5.3125,
+        "learning_rate": 1.1082275681729237e-05,
+        "epoch": 1.7945578231292516,
+        "step": 660
+      },
+      {
+        "loss": 0.41370329856872556,
+        "grad_norm": 5.90625,
+        "learning_rate": 1.0239270026986243e-05,
+        "epoch": 1.8489795918367347,
+        "step": 680
+      },
+      {
+        "loss": 0.3591457366943359,
+        "grad_norm": 7.90625,
+        "learning_rate": 9.412675594700113e-06,
+        "epoch": 1.9034013605442177,
+        "step": 700
+      },
+      {
+        "loss": 0.27732415199279786,
+        "grad_norm": 6.96875,
+        "learning_rate": 8.605341827117464e-06,
+        "epoch": 1.9578231292517008,
+        "step": 720
+      },
+      {
+        "eval_loss": 0.6445356011390686,
+        "eval_runtime": 1.8506,
+        "eval_samples_per_second": 352.315,
+        "eval_steps_per_second": 11.348,
+        "epoch": 2.0,
+        "step": 736
+      },
+      {
+        "loss": 0.3676326751708984,
+        "grad_norm": 5.59375,
+        "learning_rate": 7.820051770983611e-06,
+        "epoch": 2.0108843537414964,
+        "step": 740
+      },
+      {
+        "loss": 0.3336158752441406,
+        "grad_norm": 5.375,
+        "learning_rate": 7.059512483803904e-06,
+        "epoch": 2.0653061224489795,
+        "step": 760
+      },
+      {
+        "loss": 0.16438820362091064,
+        "grad_norm": 5.78125,
+        "learning_rate": 6.326345702055698e-06,
+        "epoch": 2.1197278911564625,
+        "step": 780
+      },
+      {
+        "loss": 0.3438990592956543,
+        "grad_norm": 5.6875,
+        "learning_rate": 5.623078803519595e-06,
+        "epoch": 2.1741496598639456,
+        "step": 800
+      },
+      {
+        "loss": 0.21838486194610596,
+        "grad_norm": 4.8125,
+        "learning_rate": 4.952136094884666e-06,
+        "epoch": 2.2285714285714286,
+        "step": 820
+      },
+      {
+        "loss": 0.2150651216506958,
+        "grad_norm": 6.75,
+        "learning_rate": 4.315830454661059e-06,
+        "epoch": 2.2829931972789117,
+        "step": 840
+      },
+      {
+        "loss": 0.34454209804534913,
+        "grad_norm": 6.125,
+        "learning_rate": 3.7163553602086496e-06,
+        "epoch": 2.3374149659863948,
+        "step": 860
+      },
+      {
+        "loss": 0.14302738904953002,
+        "grad_norm": 4.8125,
+        "learning_rate": 3.1557773263661605e-06,
+        "epoch": 2.3918367346938774,
+        "step": 880
+      },
+      {
+        "loss": 0.32184672355651855,
+        "grad_norm": 6.03125,
+        "learning_rate": 2.6360287817464257e-06,
+        "epoch": 2.4462585034013604,
+        "step": 900
+      },
+      {
+        "loss": 0.1849319100379944,
+        "grad_norm": 6.15625,
+        "learning_rate": 2.1589014072546294e-06,
+        "epoch": 2.5006802721088435,
+        "step": 920
+      },
+      {
+        "loss": 0.20881271362304688,
+        "grad_norm": 6.71875,
+        "learning_rate": 1.7260399597930592e-06,
+        "epoch": 2.5551020408163265,
+        "step": 940
+      },
+      {
+        "loss": 0.31119983196258544,
+        "grad_norm": 7.0625,
+        "learning_rate": 1.3389366024433347e-06,
+        "epoch": 2.6095238095238096,
+        "step": 960
+      },
+      {
+        "loss": 0.1483125329017639,
+        "grad_norm": 4.1875,
+        "learning_rate": 9.989257606711437e-07,
+        "epoch": 2.6639455782312926,
+        "step": 980
+      },
+      {
+        "loss": 0.3087357997894287,
+        "grad_norm": 7.09375,
+        "learning_rate": 7.071795222852296e-07,
+        "epoch": 2.7183673469387752,
+        "step": 1000
+      },
+      {
+        "loss": 0.19563177824020386,
+        "grad_norm": 4.96875,
+        "learning_rate": 4.6470359700789e-07,
+        "epoch": 2.7727891156462583,
+        "step": 1020
+      },
+      {
+        "loss": 0.1930193543434143,
+        "grad_norm": 6.15625,
+        "learning_rate": 2.7233384958522675e-07,
+        "epoch": 2.8272108843537413,
+        "step": 1040
+      },
+      {
+        "loss": 0.29953775405883787,
+        "grad_norm": 6.59375,
+        "learning_rate": 1.3073341838821028e-07,
+        "epoch": 2.8816326530612244,
+        "step": 1060
+      },
+      {
+        "loss": 0.15995726585388184,
+        "grad_norm": 5.40625,
+        "learning_rate": 4.039042943733251e-08,
+        "epoch": 2.9360544217687075,
+        "step": 1080
+      },
+      {
+        "loss": 0.20271103382110595,
+        "grad_norm": 5.5,
+        "learning_rate": 1.616313731091501e-09,
+        "epoch": 2.9904761904761905,
+        "step": 1100
+      },
+      {
+        "eval_loss": 0.6698360443115234,
+        "eval_runtime": 2.3865,
+        "eval_samples_per_second": 273.207,
+        "eval_steps_per_second": 8.8,
+        "epoch": 3.0,
+        "step": 1104
+      },
+      {
+        "train_runtime": 416.811,
+        "train_samples_per_second": 84.614,
+        "train_steps_per_second": 2.649,
+        "total_flos": 5049728172804096.0,
+        "train_loss": 0.5160034051623897,
+        "epoch": 3.0,
+        "step": 1104
+      },
+      {
+        "eval_loss": 0.6445356011390686,
+        "eval_runtime": 2.2801,
+        "eval_samples_per_second": 285.956,
+        "eval_steps_per_second": 9.21,
+        "epoch": 3.0,
+        "step": 1104
+      }
+    ],
+    "runtime": {
+      "torch_version": "2.11.0+cu128",
+      "transformers_version": "5.17.0",
+      "cuda_available": true,
+      "dtype": "bfloat16",
+      "attn_implementation": "sdpa",
+      "gpu_name": "NVIDIA GeForce RTX 5060 Ti",
+      "gpu_total_memory_gb": 15.9,
+      "gpu_capability": "12.0"
+    }
+  }
+}

training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:50ec13a9e5145757467efe60aeac4b5b544d6f5608cee769b05c807768183338
+size 5329