File size: 2,783 Bytes
6ae6b68 3341e6c 6ae6b68 14b6f9b | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 | ---
sdk: docker
app_port: 7860
---
# CodeGuardian Inference Script
This script runs an LLM agent against the CodeGuardian multi-step code review environment.
## Setup
### Environment Variables
Set the following environment variables before running:
```bash
# Required: API endpoint (for OpenAI-compatible APIs)
export API_BASE_URL="https://api.openai.com/v1"
# Required: Model name to use
export MODEL_NAME="gpt-4"
# Required: Authentication token (HF_TOKEN takes precedence over API_KEY)
export HF_TOKEN="your-huggingface-token"
# OR
export API_KEY="your-openai-api-key"
```
### Dependencies
Install required packages:
```bash
pip install openai pydantic
```
## Usage
Run the inference script:
```bash
python inference.py
```
## What It Does
The script:
1. **Initializes** the CodeGuardian environment and OpenAI client
2. **Loops through** all tasks in the environment (3 tasks × 3 steps each = 9 total steps)
3. **For each step**:
- Step 0: Detect issues in the code
- Step 1: Assess risk level (low/medium/high)
- Step 2: Suggest corrected code
4. **Sends structured prompts** to the LLM with the current observation
5. **Parses JSON responses** safely with fallback handling
6. **Executes actions** in the environment and accumulates rewards
7. **Logs progress** including rewards, feedback, and running totals
## Output Format
```
2026-04-06 02:26:51,542 - INFO - Starting inference with model: gpt-4
2026-04-06 02:26:51,561 - INFO - Initial task: Return sum of two numbers
2026-04-06 02:26:51,561 - INFO - --- Step 1: DETECT ---
2026-04-06 02:26:51,562 - INFO - LLM response: {"detected_issues": ["wrong operator"], ...}
2026-04-06 02:26:51,564 - INFO - Action type: detect
2026-04-06 02:26:51,564 - INFO - Reward: +0.30
2026-04-06 02:26:51,564 - INFO - Feedback: []
2026-04-06 02:26:51,565 - INFO - Running total: +0.30
...
2026-04-06 02:26:51,586 - INFO - === Inference Complete ===
2026-04-06 02:26:51,586 - INFO - Final score: 0.00
2026-04-06 02:26:51,586 - INFO - Total steps: 9
```
## Reward Structure
- **Detect step**: +0.3 (correct) / -0.2 (incorrect)
- **Risk step**: +0.2 (correct) / -0.2 (incorrect)
- **Fix step**: +0.5 (correct) / -0.3 (incorrect)
- **Invalid action order**: -0.5
## Error Handling
The script is robust against:
- Invalid JSON responses (falls back to empty responses)
- API errors (falls back to empty responses)
- Missing fields (provides sensible defaults)
- Infinite loops (50-step safety limit)
## Expected Performance
- **Perfect score**: 3.0 (all 3 tasks completed perfectly)
- **Random baseline**: ~0.0 (rewards balance out)
- **Typical LLM**: 0.5-2.0 (depending on model capability)
The script provides a stable, reproducible baseline for evaluating LLM performance on multi-step code review tasks.
|