| --- |
| sdk: docker |
| app_port: 7860 |
| --- |
| # CodeGuardian Inference Script |
|
|
| This script runs an LLM agent against the CodeGuardian multi-step code review environment. |
|
|
| ## Setup |
|
|
| ### Environment Variables |
|
|
| Set the following environment variables before running: |
|
|
| ```bash |
| # Required: API endpoint (for OpenAI-compatible APIs) |
| export API_BASE_URL="https://api.openai.com/v1" |
| |
| # Required: Model name to use |
| export MODEL_NAME="gpt-4" |
| |
| # Required: Authentication token (HF_TOKEN takes precedence over API_KEY) |
| export HF_TOKEN="your-huggingface-token" |
| # OR |
| export API_KEY="your-openai-api-key" |
| ``` |
|
|
| ### Dependencies |
|
|
| Install required packages: |
|
|
| ```bash |
| pip install openai pydantic |
| ``` |
|
|
| ## Usage |
|
|
| Run the inference script: |
|
|
| ```bash |
| python inference.py |
| ``` |
|
|
| ## What It Does |
|
|
| The script: |
|
|
| 1. **Initializes** the CodeGuardian environment and OpenAI client |
| 2. **Loops through** all tasks in the environment (3 tasks × 3 steps each = 9 total steps) |
| 3. **For each step**: |
| - Step 0: Detect issues in the code |
| - Step 1: Assess risk level (low/medium/high) |
| - Step 2: Suggest corrected code |
| 4. **Sends structured prompts** to the LLM with the current observation |
| 5. **Parses JSON responses** safely with fallback handling |
| 6. **Executes actions** in the environment and accumulates rewards |
| 7. **Logs progress** including rewards, feedback, and running totals |
|
|
| ## Output Format |
|
|
| ``` |
| 2026-04-06 02:26:51,542 - INFO - Starting inference with model: gpt-4 |
| 2026-04-06 02:26:51,561 - INFO - Initial task: Return sum of two numbers |
| 2026-04-06 02:26:51,561 - INFO - --- Step 1: DETECT --- |
| 2026-04-06 02:26:51,562 - INFO - LLM response: {"detected_issues": ["wrong operator"], ...} |
| 2026-04-06 02:26:51,564 - INFO - Action type: detect |
| 2026-04-06 02:26:51,564 - INFO - Reward: +0.30 |
| 2026-04-06 02:26:51,564 - INFO - Feedback: [] |
| 2026-04-06 02:26:51,565 - INFO - Running total: +0.30 |
| ... |
| 2026-04-06 02:26:51,586 - INFO - === Inference Complete === |
| 2026-04-06 02:26:51,586 - INFO - Final score: 0.00 |
| 2026-04-06 02:26:51,586 - INFO - Total steps: 9 |
| ``` |
|
|
| ## Reward Structure |
|
|
| - **Detect step**: +0.3 (correct) / -0.2 (incorrect) |
| - **Risk step**: +0.2 (correct) / -0.2 (incorrect) |
| - **Fix step**: +0.5 (correct) / -0.3 (incorrect) |
| - **Invalid action order**: -0.5 |
|
|
| ## Error Handling |
|
|
| The script is robust against: |
| - Invalid JSON responses (falls back to empty responses) |
| - API errors (falls back to empty responses) |
| - Missing fields (provides sensible defaults) |
| - Infinite loops (50-step safety limit) |
|
|
| ## Expected Performance |
|
|
| - **Perfect score**: 3.0 (all 3 tasks completed perfectly) |
| - **Random baseline**: ~0.0 (rewards balance out) |
| - **Typical LLM**: 0.5-2.0 (depending on model capability) |
|
|
| The script provides a stable, reproducible baseline for evaluating LLM performance on multi-step code review tasks. |
|
|
