{ "cells": [ { "cell_type": "markdown", "metadata": { "id": "Ila2Ci2giPOS" }, "source": [ "# MissionCtrl — One-Click Training Notebook\n", "\n", "**OpenEnv Hackathon Round 2**\n", "\n", "This notebook trains an OverseerAgent to detect hallucinations in a multi-agent fleet using GRPO + Unsloth.\n", "\n", "**Runtime**: **Kaggle** 2×T4 (see section below). \n", "**Base model (`train.py`)**: default `Qwen/Qwen2.5-0.5B-Instruct` (fast canary). For larger hubs: `MISSIONCTRL_MODEL_NAME=unsloth/Meta-Llama-3.1-8B-Instruct-bnb-4bit` (or `unsloth/Llama-3.2-3B-Instruct`). Gated models: `HF_TOKEN` + license on HuggingFace. \n", "**Expected training time**: 0.5B is much faster than 3B/8B; full curriculum on Kaggle 2×T4 can still take hours if phases repeat; `train.py` shortens steps when two GPUs are visible. \n", "**Expected final reward**: 0.68+ (reward ceiling is **0.85**, not 1.0) \n", "\n", "## Bug Fixes Applied\n", "- ✅ **#1** GRPO reward fn now runs full episode rollout (not single step)\n", "- ✅ **#2** NUM_GENERATIONS ≤ BATCH_SIZE (was 8 > 4, caused crash)\n", "- ✅ **#3** evaluate() uses final reward, not accumulated sum of state scores\n", "- ✅ **#4** run_baseline() uses final reward (not per-task overwrite)\n", "- ✅ **#5** Reward ceiling corrected to 0.85 throughout\n", "- ✅ **#6** SYNTHESIZE_REPORT() now promotes PENDING tasks too, terminates episode\n", "- ✅ **#7** 'special' difficulty tier now supported\n", "- ✅ **#8** All 10 hallucination types documented and in system prompt\n", "- ✅ **#10** SUBTLETY levels now affect corruption templates\n", "- ✅ **#11** tokenizer.padding_side='left' set for correct GRPO generation\n", "- ✅ **#12** plot_reward_curve() guards against empty history\n", "- ✅ **#13** Easy phase uses 3 tasks (consistent with docs)\n", "- ✅ **#15** _corrupt() no longer relies on 'def ' being present\n", "- ✅ **#16** All bare except: changed to except Exception:\n", "- ✅ **#17** FAILED status now assigned via ESCALATE action\n", "- ✅ **#20** Reward function exceptions now logged (not silently swallowed)\n", "- ✅ **#21** 2+ GPU: shorter Kaggle curriculum; model stays on one CUDA device by default for GRPO generation stability\n", "- ✅ **#22** max_prompt_length set in GRPOConfig to prevent seed tag truncation\n", "- ✅ **#23** evaluate() uses do_sample=False for truly greedy/reproducible eval\n", "- ✅ **#25** Passive FP penalty only fires when all task outputs are visible\n", "- ✅ **#26** EVIDENCE_KEYWORDS uses specific phrases, not generic single words\n", "- ✅ **#27** _build_observation() de-duplicates messages per task_id\n", "- ✅ **#28** Hub push and local save both use LoRA format consistently\n", "- ✅ **#29** DEPENDENCY_MAP variable naming clarified\n", "- ✅ **#30** SubAgentSimulator seed offset +7 documented\n", "- ✅ **#32** HF_REPO validation strengthened (case-insensitive)\n", "- ✅ **#35** Model always restored to training mode after eval\n", "- ✅ **#38** System prompt lists all 10 hallucination types\n", "- ✅ **#39** EPISODE_MAX_STEPS constant used consistently across training and eval\n", "- ✅ **#40** Fake credential strings use clearly-invalid tokens" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Kaggle (2×T4)\n", "\n", "- **Ways to get the code (the next cell checks in this order):** \n", " 1) **Same directory as the notebook** (Kaggle kernel push, or files uploaded to working space). \n", " 2) **Kaggle \"Add data\" (Input):** if you attach a dataset, files live at `/kaggle/input//` (read-only). The next cell **detects** that path and `cd` there when all required `.py` files are present. \n", " 3) If still missing, it **clones** the [default training repo](https://github.com/Leo-Expose/MissionCtrl) to `/kaggle/working/MissionCtrl/`. Override the clone URL with **`MISSIONCTRL_REPO_URL`** in **Add-ons → Environment** if you use another fork. Private git: use a [Kaggle dataset](https://www.kaggle.com/datasets) you build from the repo, or a git URL (see README: **Kaggle training and Kaggle CLI**). Do not paste secrets in the notebook.\n", "- **Notebook settings**: accelerator **2 × T4**; **Internet** ON (required for `pip` and for `git clone` in the default flow). **Add-ons → Secrets**: `HF_TOKEN` (gated Unsloth bases: accept the license on HuggingFace first).\n", "- **Checkpoints** on Kaggle: `train.py` uses `/kaggle/working/missionctrl_checkpoints` when appropriate.\n", "- **Run training** after the clone/verify cell: `!python train.py` or `!python train.py --smoke-train`. **2 GPUs**: shorter curriculum in `train.py`; the model loads on one CUDA device by default to avoid GRPO generation device-mismatch crashes.\n", "- **MCP:** no Kaggle launcher in this repo; see the README.\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "JzRdk_DPiPOU", "outputId": "70958fad-059c-49e2-879f-2f72bca06ef5" }, "outputs": [], "source": [ "# ── Install dependencies (Kaggle) ────────────────────────────────────────────\n", "# unsloth[colab-new] works on many Kaggle images; if install fails, see Unsloth docs.\n", "# On the Kaggle image, pip may show resolver messages (e.g. fsspec/s3fs); if installs\n", "# complete and the next cell runs, those are usually safe to ignore.\n", "!pip install \"unsloth[colab-new]\" trl openenv transformers datasets accelerate matplotlib --quiet\n", "!pip install --upgrade bitsandbytes --quiet\n", "\n", "# Suppress tokenizer parallelism warning\n", "import os\n", "os.environ['TOKENIZERS_PARALLELISM'] = 'false'\n", "\n", "print('✅ Dependencies installed')" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "O3g4hUbhiPOV" }, "outputs": [], "source": [ "# ── Clone (if needed) + verify MissionCtrl project files ────────────────────\n", "# 1) Same folder as the notebook; 2) /kaggle/input// if you \"Add data\"; 3) then git clone.\n", "# Set MISSIONCTRL_REPO_URL in Add-ons → Environment to override the default public repo.\n", "import os\n", "import subprocess\n", "\n", "REQUIRED = [\n", " \"train.py\",\n", " \"environment.py\",\n", " \"reward_model.py\",\n", " \"grpo_rewards.py\",\n", " \"grpo_completion.py\",\n", "]\n", "REPO_DIR = \"MissionCtrl\"\n", "DEFAULT_REPO = \"https://github.com/Leo-Expose/MissionCtrl.git\"\n", "REPO_URL = os.environ.get(\"MISSIONCTRL_REPO_URL\", DEFAULT_REPO).strip() or DEFAULT_REPO\n", "\n", "\n", "def all_here() -> bool:\n", " return all(os.path.isfile(f) for f in REQUIRED)\n", "\n", "\n", "def all_here_in(d: str) -> bool:\n", " return all(os.path.isfile(os.path.join(d, f)) for f in REQUIRED)\n", "\n", "\n", "def missing_in_dir(d: str) -> list:\n", " return [f for f in REQUIRED if not os.path.isfile(os.path.join(d, f))]\n", "\n", "\n", "# Kaggle: files are often under /kaggle/input// (read-only) — chdir so imports work\n", "if not all_here() and os.path.isdir(\"/kaggle/input\"):\n", " for name in sorted(os.listdir(\"/kaggle/input\")):\n", " root = os.path.join(\"/kaggle/input\", name)\n", " if not os.path.isdir(root) or name.startswith(\".\"):\n", " continue\n", " if all_here_in(root):\n", " os.chdir(root)\n", " print(f\"✅ Using Kaggle input dataset: {root}\")\n", " break\n", " nest = os.path.join(root, REPO_DIR)\n", " if os.path.isdir(nest) and all_here_in(nest):\n", " os.chdir(nest)\n", " print(f\"✅ Using nested folder in input: {nest}\")\n", " break\n", "\n", "if all_here():\n", " print(\"✅ MissionCtrl files already in this directory (kernel bundle or /kaggle/input). Skipping git clone.\")\n", "else:\n", " is_kaggle = bool(os.environ.get(\"KAGGLE_KERNEL_RUN_TYPE\")) or os.path.isdir(\"/kaggle/working\")\n", " if is_kaggle and os.path.isdir(\"/kaggle/working\"):\n", " os.chdir(\"/kaggle/working\")\n", " in_repo = all(os.path.isfile(os.path.join(REPO_DIR, f)) for f in REQUIRED)\n", " if in_repo:\n", " os.chdir(REPO_DIR)\n", " elif not all_here():\n", " if os.path.isdir(REPO_DIR):\n", " _miss = \", \".join(missing_in_dir(REPO_DIR)) or \"(unknown)\"\n", " raise FileNotFoundError(\n", " f\"Directory {REPO_DIR!r} exists but is incomplete (missing: {_miss}). \"\n", " f\"Delete it in the file browser and re-run, or set MISSIONCTRL_REPO_URL. \"\n", " f\"See README: Kaggle training and Kaggle CLI (Troubleshooting).\"\n", " )\n", " subprocess.run(\n", " [\"git\", \"clone\", \"--depth\", \"1\", REPO_URL, REPO_DIR],\n", " check=True,\n", " )\n", " os.chdir(REPO_DIR)\n", " if not all_here():\n", " _miss = \", \".join(missing_in_dir(\".\")) or \", \".join(REQUIRED)\n", " raise FileNotFoundError(\n", " f\"Could not find train.py and env modules (missing: {_miss}). Set MISSIONCTRL_REPO_URL to a public \"\n", " f\"git URL (default {DEFAULT_REPO!r}) or add a Kaggle dataset with the repo. \"\n", " \"See README: Kaggle training and Kaggle CLI (Troubleshooting).\"\n", " )\n", "\n", "for f in REQUIRED:\n", " print(\" ✅\", f)\n", "print(\" cwd:\", os.getcwd())\n", "\n", "# Jupyter keeps the notebook directory on sys.path, not cwd — add repo root for local imports.\n", "import sys\n", "_repo_root = os.path.abspath(\".\")\n", "if _repo_root not in sys.path:\n", " sys.path.insert(0, _repo_root)\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "9rn1mUW4iPOV" }, "outputs": [], "source": [ "# ── Cell 3: Verify GPU ────────────────────────────────────────────────────────\n", "import torch\n", "\n", "print(f'GPU: {torch.cuda.get_device_name(0)}')\n", "print(f'VRAM: {torch.cuda.get_device_properties(0).total_memory / 1e9:.1f} GB')\n", "print(f'GPU count: {torch.cuda.device_count()}')\n", "assert torch.cuda.is_available(), \"No GPU — switch Kaggle accelerator to GPU (2×T4 recommended)\"\n", "\n", "# 2×GPU / T4: train.py auto-tunes the Kaggle curriculum while keeping GRPO generation on one device.\n", "vram_gb = torch.cuda.get_device_properties(0).total_memory / 1e9\n", "n_gpu = torch.cuda.device_count()\n", "if n_gpu >= 2:\n", " print(f\"✅ {n_gpu} GPUs — Kaggle 2×T4-style path: shorter curriculum; single-device GRPO generation\")\n", "elif vram_gb < 30:\n", " print(f\"⚠️ ~{vram_gb:.0f} GB VRAM (e.g. single T4) — use MISSIONCTRL_T4_CURRICULUM=1 for shorter phases if needed\")\n", "else:\n", " print(f\"✅ ~{vram_gb:.0f} GB — single-GPU fallback curriculum\")\n", "\n", "print('✅ GPU verified')" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "duotd7wSiPOV" }, "outputs": [], "source": [ "# ── Cell 4: Smoke-test the environment ───────────────────────────────────────\n", "from environment import MissionCtrlEnv, OverseerAction, parse_action\n", "from reward_model import compute_reward, reward_breakdown\n", "\n", "print('--- Smoke test: medium difficulty, 3 tasks ---')\n", "env = MissionCtrlEnv(difficulty='medium', num_tasks=3, seed=42)\n", "obs, info = env.reset()\n", "\n", "print(f' Tasks loaded: {len(obs[\"task_board\"])}')\n", "print(f' Agent messages: {len(obs[\"recent_messages\"])}')\n", "\n", "# Test FLAG action\n", "first_task = obs['task_board'][0]['task_id']\n", "action = OverseerAction('FLAG', task_id=first_task, evidence='fabricated citation detected in prior output')\n", "obs2, reward, terminated, truncated, info = env.step(action)\n", "print(f' Step reward: {reward:.3f}')\n", "print(f' Info: {info}')\n", "\n", "# Test reward breakdown\n", "breakdown = reward_breakdown(env)\n", "print(f' Reward breakdown: {breakdown[\"signals\"]}')\n", "print(f' Theoretical ceiling: {breakdown[\"theoretical_ceiling\"]}')\n", "\n", "print()\n", "print('--- Smoke test: special difficulty (new tier) ---')\n", "env_special = MissionCtrlEnv(difficulty='special', num_tasks=5, seed=99)\n", "obs_s, _ = env_special.reset()\n", "print(f' Special tier tasks: {len(obs_s[\"task_board\"])}')\n", "\n", "print()\n", "print('--- Smoke test: action parser ---')\n", "test_cases = [\n", " ('FLAG(T001, \"fabricated citation in reference list\")', 'FLAG'),\n", " ('APPROVE(T002)', 'APPROVE'),\n", " ('SYNTHESIZE_REPORT()', 'SYNTHESIZE'),\n", " ('garbage output no action', 'NOOP'),\n", "]\n", "for text, expected in test_cases:\n", " parsed = parse_action(text)\n", " status = '✅' if parsed.action_type == expected else f'❌ expected {expected}'\n", " print(f' {status} \"{text[:40]}...\" → {parsed.action_type}')\n", "\n", "print()\n", "print('✅ Environment smoke test passed')" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "UZ903OzgiPOW" }, "outputs": [], "source": [ "# ── Cell 5: Run pre-training baseline ────────────────────────────────────────\n", "# FIX #3/#4: baseline now uses final reward (not accumulated sum or per-task overwrite)\n", "from train import run_baseline\n", "\n", "baseline_reward = run_baseline()\n", "print(f'\\n🎯 Baseline established: {baseline_reward:.3f}')\n", "print(f'Reward ceiling: 0.85 | Training target: 0.68+')\n", "print(f'Baseline is {baseline_reward/0.85*100:.0f}% of theoretical ceiling')" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "fwfPeYGtiPOW" }, "outputs": [], "source": [ "# ── Cell 6: Set HuggingFace credentials ──────────────────────────────────────\n", "from huggingface_hub import login\n", "import os\n", "\n", "# Option A: Use Kaggle Secrets (recommended)\n", "# 1. Go to Add-ons → Secrets\n", "# 2. Add a new secret named \"HF_TOKEN\" with your HuggingFace token\n", "# 3. Run this cell:\n", "if os.environ.get('HF_TOKEN'):\n", " login(token=os.environ['HF_TOKEN'])\n", " print('✅ Logged in via HF_TOKEN secret')\n", "else:\n", " # Option B: Interactive prompt\n", " login()\n", " print('✅ Logged in interactively')\n", "\n", "# Set your repo name before training\n", "# FIX #32: validation is now case-insensitive and catches more placeholder patterns\n", "import train\n", "train.HF_REPO = 'Proliferation/missionctrl_env' # ← CHANGE THIS\n", "\n", "# Verify it's been updated\n", "if 'your-hf' in train.HF_REPO.lower():\n", " raise ValueError('❌ Please update train.HF_REPO with your actual HuggingFace username!')\n", "\n", "print(f'✅ Will push trained adapter to: {train.HF_REPO}')\n", "\n", "# ── IMPORTANT: Verify evaluation uses your model ──────────────────────────────\n", "print()\n", "print('⚠️ IMPORTANT: Before training, confirm the hackathon evaluation invokes')\n", "print(' your HF-hosted model (not a fixed Groq/API endpoint).')\n", "print(' If evaluators run client.py with their own API_BASE_URL, fine-tuning')\n", "print(' will not affect your score. Check the OpenEnv evaluation spec first.')" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "Td7lu2T7iPOX" }, "outputs": [], "source": [ "# ── Cell 7: TRAIN ─────────────────────────────────────────────────────────────\n", "# Full 3-phase curriculum with reward-gated advancement.\n", "#\n", "# Key fixes active in this run:\n", "# - NUM_GENERATIONS=4 ≤ BATCH_SIZE=4 (no crash)\n", "# - Full episode rollout in reward fn (meaningful GRPO signal)\n", "# - max_prompt_length set (seed tag won't be truncated)\n", "# - padding_side='left' (correct batched generation)\n", "# - Greedy eval (do_sample=False) for reproducible metrics\n", "#\n", "# Checkpoints: saved to train.OUTPUT_DIR every 50 steps (Kaggle: /kaggle/working/missionctrl_checkpoints).\n", "# If training is interrupted, resume from the last checkpoint.\n", "\n", "from train import train\n", "\n", "history = train()\n", "print('\\n🏆 Training complete!')" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "VsAa2XCNiPOX" }, "outputs": [], "source": [ "# ── Cell 8: Display reward curve ─────────────────────────────────────────────\n", "# FIX #5: curve now shows 0.85 ceiling line (not 1.0)\n", "# FIX #12: plot_reward_curve() no longer crashes on empty history\n", "from IPython.display import Image\n", "import os\n", "\n", "import train\n", "curve_path = os.path.join(train.OUTPUT_DIR, 'reward_curve.png')\n", "\n", "if os.path.exists(curve_path):\n", " Image(curve_path)\n", "else:\n", " print('Reward curve not found at expected path.')\n", " print(f'Check: {train.OUTPUT_DIR}')\n", " # Try local fallback\n", " local_path = './missionctrl_checkpoints/reward_curve.png'\n", " if os.path.exists(local_path):\n", " print(f'Found at local path: {local_path}')\n", " Image(local_path)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "S6A_GZyLiPOY" }, "outputs": [], "source": [ "# ── Cell 9: Before/After demo comparison ─────────────────────────────────────\n", "# FIX #28: load LoRA adapter (not merged model) — consistent with what was pushed to Hub\n", "from unsloth import FastLanguageModel\n", "from environment import MissionCtrlEnv, parse_action\n", "from train import build_user_prompt, SYSTEM_PROMPT, load_model\n", "import torch\n", "import os\n", "\n", "# Load base model + LoRA adapter\n", "import train\n", "adapter_path = os.path.join(train.OUTPUT_DIR, 'final_lora')\n", "if not os.path.exists(adapter_path):\n", " adapter_path = os.path.join('missionctrl_checkpoints', 'final_lora')\n", "\n", "model, tokenizer = load_model() # loads base model with LoRA config\n", "model.load_adapter(adapter_path) # loads trained adapter weights\n", "FastLanguageModel.for_inference(model)\n", "\n", "# Use a known hallucinated episode (seed 0, hard difficulty)\n", "env = MissionCtrlEnv(difficulty='hard', num_tasks=4, seed=0)\n", "obs, _ = env.reset()\n", "\n", "print('=== ENVIRONMENT STATE ===')\n", "print(env.render())\n", "\n", "prompt = tokenizer.apply_chat_template(\n", " [\n", " {'role': 'system', 'content': SYSTEM_PROMPT},\n", " {'role': 'user', 'content': build_user_prompt(obs)},\n", " ],\n", " tokenize=False,\n", " add_generation_prompt=True,\n", ")\n", "inputs = tokenizer(prompt, return_tensors='pt', truncation=True, max_length=3584).to(model.device)\n", "\n", "with torch.no_grad():\n", " out = model.generate(\n", " **inputs,\n", " max_new_tokens=256,\n", " do_sample=False, # FIX #23: greedy for demo reproducibility\n", " )\n", "\n", "completion = tokenizer.decode(out[0][inputs['input_ids'].shape[1]:], skip_special_tokens=True)\n", "\n", "print('\\n=== TRAINED MODEL OUTPUT ===')\n", "print(completion)\n", "\n", "parsed = parse_action(completion)\n", "print(f'\\nParsed action: {parsed.action_type}(task_id={parsed.task_id}, evidence={parsed.evidence!r})')" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "oH1e8ls8iPOY" }, "outputs": [], "source": [ "# ── Cell 10: Final evaluation run ────────────────────────────────────────────\n", "# FIX #3: rewards are now final-step state scores, not accumulated sums\n", "# FIX #23: uses do_sample=False for reproducible metrics\n", "from train import evaluate\n", "\n", "final_reward, metrics = evaluate(\n", " model, tokenizer,\n", " difficulty='hard',\n", " num_tasks=4,\n", " n_episodes=20,\n", ")\n", "\n", "print('\\n=== FINAL EVALUATION SUMMARY ===')\n", "print(f' Overall reward: {metrics[\"mean_reward\"]:.3f} ± {metrics[\"std_reward\"]:.3f}')\n", "print(f' % of ceiling (0.85): {metrics[\"mean_reward\"]/0.85*100:.1f}%')\n", "print(f' Detection rate: {metrics[\"mean_detect_rate\"]:.1%}')\n", "print(f' False positive rate: {metrics[\"mean_fp_rate\"]:.1%}')\n", "\n", "if metrics['mean_reward'] >= 0.68:\n", " print(f'\\n✅ Target of 0.68+ achieved!')\n", "else:\n", " print(f'\\n⚠️ Below 0.68 target. Consider: more training steps, prompt tuning, or policy refinement.')" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "CTrk97DliPOY" }, "outputs": [], "source": [ "# ── Cell 11: Multi-tier evaluation (all difficulties) ────────────────────────\n", "# Evaluate across all difficulty tiers to check for tier-specific regressions\n", "# (Council noted: fast-closure policy improved Hard but hurt Special in baseline)\n", "from train import evaluate\n", "\n", "tiers = [\n", " ('easy', 2, 10),\n", " ('medium', 3, 10),\n", " ('hard', 4, 10),\n", " ('special', 5, 10),\n", "]\n", "\n", "print('=== MULTI-TIER EVALUATION ===')\n", "print(f'{\"Tier\":<10} {\"Reward\":<12} {\"Detect%\":<12} {\"FP%\":<10} {\"% Ceiling\"}')\n", "print('-' * 55)\n", "\n", "for difficulty, num_tasks, n_eps in tiers:\n", " reward, m = evaluate(\n", " model, tokenizer,\n", " difficulty=difficulty,\n", " num_tasks=num_tasks,\n", " n_episodes=n_eps,\n", " )\n", " pct = reward / 0.85 * 100\n", " print(f'{difficulty:<10} {reward:.3f} ± {m[\"std_reward\"]:.3f} '\n", " f'{m[\"mean_detect_rate\"]:.1%} {m[\"mean_fp_rate\"]:.1%} {pct:.0f}%')" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "waZjJJpGiPOZ" }, "outputs": [], "source": [ "# ── Cell 12: Reward breakdown for a single episode ───────────────────────────\n", "# Useful for debugging which signals are limiting performance\n", "from environment import MissionCtrlEnv, TaskStatus\n", "from reward_model import reward_breakdown\n", "from train import build_user_prompt, SYSTEM_PROMPT, EPISODE_MAX_STEPS\n", "import torch\n", "\n", "debug_env = MissionCtrlEnv(difficulty='hard', num_tasks=4, seed=42)\n", "obs, _ = debug_env.reset()\n", "\n", "done = False\n", "steps = 0\n", "while not done and steps < EPISODE_MAX_STEPS:\n", " prompt_text = tokenizer.apply_chat_template(\n", " [{'role': 'system', 'content': SYSTEM_PROMPT},\n", " {'role': 'user', 'content': build_user_prompt(obs)}],\n", " tokenize=False, add_generation_prompt=True,\n", " )\n", " inputs = tokenizer(prompt_text, return_tensors='pt', truncation=True,\n", " max_length=3584).to(model.device)\n", " with torch.no_grad():\n", " out = model.generate(**inputs, max_new_tokens=256, do_sample=False)\n", " completion = tokenizer.decode(out[0][inputs['input_ids'].shape[1]:], skip_special_tokens=True)\n", " from environment import parse_action\n", " action = parse_action(completion)\n", " obs, _, terminated, truncated, _ = debug_env.step(action)\n", " done = terminated or truncated\n", " steps += 1\n", "\n", "bd = reward_breakdown(debug_env)\n", "print('=== REWARD BREAKDOWN (seed=42, hard, 4 tasks) ===')\n", "print(f'Total reward: {bd[\"total_reward\"]:.4f} ({bd[\"total_reward\"]/0.85*100:.1f}% of ceiling)')\n", "print()\n", "for signal, vals in bd['signals'].items():\n", " bar = '█' * int(vals['raw'] * 20)\n", " print(f' {signal:<28} raw={vals[\"raw\"]:.3f} [{bar:<20}] weighted={vals[\"weighted\"]:+.4f}')\n", "print()\n", "print(f'Episode info: {bd[\"info\"]}')" ] } ], "metadata": { "accelerator": "GPU", "kernelspec": { "display_name": "Python 3", "name": "python3" }, "language_info": { "name": "python", "version": "3.10.0" } }, "nbformat": 4, "nbformat_minor": 0 }