|
Download README.md from pradhyut/codemeta: direct link, hf CLI and curl.
- Browser
- Download file 7.46 kB
-
https://huggingface.co/spaces/pradhyut/codemeta/resolve/main/README.md
- Command line
-
hf download hf://spaces/pradhyut/codemeta/README.md
-
curl -L -o README.md https://huggingface.co/spaces/pradhyut/codemeta/resolve/main/README.md
7.46 kB
| title: CodeSentinel OpenEnv | |
| emoji: π‘οΈ | |
| colorFrom: blue | |
| colorTo: indigo | |
| sdk: docker | |
| app_port: 7860 | |
| # π‘οΈ CodeSentinel | |
| **A real-world AI training environment for automated code review and security auditing.** | |
| CodeSentinel is a production-grade [OpenEnv](https://huggingface.co/spaces/openenv/openenv)-compliant environment submitted to the Hugging Face OpenEnv hackathon. Agents learn to review code the way a senior software engineer would β finding bugs, auditing for OWASP vulnerabilities, and producing structured architecture reports. | |
| --- | |
| ## Motivation | |
| Code review is one of the most high-stakes daily tasks in software engineering. Every line of code that ships without review is a potential bug or security hole. Despite this, code review is still largely a manual process that is: | |
| - **Time-consuming** β a typical PR review takes 30β120 minutes. | |
| - **Inconsistent** β quality varies widely by reviewer expertise and fatigue. | |
| - **Incomplete** β reviewers miss between 15% and 35% of defects on average. | |
| Training language models to review code well is not just an academic exercise. A model that can reliably flag injection vulnerabilities, catch off-by-one errors, and evaluate microservice designs would provide direct value to every software team in the world. | |
| CodeSentinel provides a **dense, structured reward signal** that teaches agents not just *what* to report, but *how precise* to be, *how severe* to classify issues, and *how efficiently* to complete a review. | |
| --- | |
| ## Tasks | |
| | Task | Difficulty | Max Steps | Description | | |
| |---|---|---|---| | |
| | `bug_detection` | π’ Easy | 20 | Find 5 logical bugs in a Python service | | |
| | `security_audit` | π‘ Medium | 30 | Identify 7 OWASP Top-10 vulnerabilities in a Flask app | | |
| | `architecture_review` | π΄ Hard | 50 | Produce a structured report on a 4-file microservice codebase | | |
| --- | |
| ## Observation Space | |
| | Field | Type | Description | | |
| |---|---|---| | |
| | `task_id` | string | Task identifier | | |
| | `task_description` | string | Natural language task brief | | |
| | `files` | `CodeFile[]` | Source files to review (filename, language, content, line_count) | | |
| | `history` | `ReviewComment[]` | All previous actions taken this episode | | |
| | `step` | int | Current step number (0-indexed) | | |
| | `max_steps` | int | Maximum allowed steps before episode terminates | | |
| | `done` | bool | Whether the episode has ended | | |
| | `hint` | string \| null | Optional task hint | | |
| --- | |
| ## Action Space | |
| | Field | Type | Description | | |
| |---|---|---| | |
| | `action_type` | enum | One of: `flag_bug`, `flag_vulnerability`, `add_comment`, `suggest_fix`, `request_changes`, `approve`, `submit_report` | | |
| | `line` | int \| null | Source line number the action refers to | | |
| | `filename` | string \| null | Target filename | | |
| | `message` | string \| null | Human-readable description of the finding | | |
| | `severity` | `low\|medium\|high\|critical` \| null | Finding severity | | |
| | `cve_category` | string \| null | OWASP category string (e.g. `A03_Injection`) | | |
| | `original_code` | string \| null | Code being replaced (for `suggest_fix`) | | |
| | `suggested_code` | string \| null | Replacement code (for `suggest_fix`) | | |
| | `report` | dict \| null | Structured architecture report (for `submit_report`) | | |
| | `reason` | string \| null | Explanation for terminal actions | | |
| --- | |
| ## Reward Function | |
| ### Non-terminal steps | |
| ``` | |
| reward = 0.50 Γ coverage | |
| + 0.20 Γ precision | |
| + 0.15 Γ severity_match | |
| + 0.10 Γ report_quality | |
| + 0.05 Γ efficiency | |
| β false_positive_penalty | |
| β step Γ 0.005 | |
| ``` | |
| Clamped to `[0.0, 1.0]`. Delta (change from previous step) is also returned. | |
| ### Terminal step | |
| ``` | |
| reward = grader.final_score() β step Γ 0.005 | |
| ``` | |
| ### Grader Details | |
| **Bug Detection (Easy)** | |
| ``` | |
| final = 0.50 Γ coverage | |
| + 0.25 Γ precision | |
| + 0.15 Γ severity_match (0.20 bonus per exact severity match) | |
| + 0.10 Γ efficiency_bonus (0.10 if all bugs found β€ 10 steps) | |
| β 0.05 Γ false_positives | |
| ``` | |
| A correct flag requires line number within **Β±1** of the ground-truth bug. | |
| **Security Audit (Medium)** | |
| - Full hit (correct OWASP category + line within Β±2): `category_weight` | |
| - Partial hit (correct OWASP category, wrong line): `0.5 Γ category_weight` | |
| - OWASP weights: A01/A03=1.5, A02/A07=1.3, others=1.0 | |
| - Terminal score = weighted F1 β FP penalty | |
| - `approve` on vulnerable codebase β **score 0.0** | |
| **Architecture Review (Hard)** | |
| - Only `submit_report` action scores terminally | |
| - Fuzzy keyword matching (β₯50% keywords must match) | |
| - Per-category F1, weights: security=0.30, reliability=0.25, api_design=0.20, observability=0.15, scalability=0.10 | |
| - +0.10 bonus if all 5 categories present | |
| - Intermediate flags give partial credit (capped at 0.25) | |
| --- | |
| ## Setup β Local Development | |
| ```bash | |
| # 1. Create virtual environment | |
| python -m venv .venv | |
| source .venv/bin/activate # Windows: .venv\Scripts\activate | |
| # 2. Install dependencies | |
| pip install -r requirements.txt | |
| # 3. Run the server | |
| uvicorn app:app --host 0.0.0.0 --port 7860 --reload | |
| # 4. Visit http://localhost:7860 | |
| ``` | |
| --- | |
| ## Docker | |
| ```bash | |
| # Build | |
| docker build -t codesentinel . | |
| # Run | |
| docker run -p 7860:7860 codesentinel | |
| # Visit http://localhost:7860 | |
| ``` | |
| --- | |
| ## API Usage Examples | |
| ### Start an episode | |
| ```bash | |
| curl -X POST http://localhost:7860/reset \ | |
| -H "Content-Type: application/json" \ | |
| -d '{"task_id": "bug_detection", "seed": 42}' | |
| ``` | |
| ### Submit an action | |
| ```bash | |
| curl -X POST http://localhost:7860/step \ | |
| -H "Content-Type: application/json" \ | |
| -d '{"action_type": "flag_bug", "line": 20, "severity": "high", "message": "KeyError on missing cache key"}' | |
| ``` | |
| ### Check environment state | |
| ```bash | |
| curl http://localhost:7860/state | |
| ``` | |
| ### Health check | |
| ```bash | |
| curl http://localhost:7860/validate | |
| ``` | |
| --- | |
| ## Baseline Scores (GPT-4o) | |
| ```bash | |
| export OPENAI_API_KEY="sk-..." | |
| python baseline/run_baseline.py --base-url http://localhost:7860 --seed 42 | |
| ``` | |
| | Task | Score | Progress | | |
| |---|---|---| | |
| | `bug_detection` | ~0.71 | `ββββββββββββββββββββ` | | |
| | `security_audit` | ~0.58 | `ββββββββββββββββββββ` | | |
| | `architecture_review` | ~0.43 | `ββββββββββββββββββββ` | | |
| | **Average** | **~0.57** | `ββββββββββββββββββββ` | | |
| --- | |
| ## Project Structure | |
| ``` | |
| codesentinel/ | |
| βββ Dockerfile # Container definition for Hugging Face Spaces | |
| βββ README.md | |
| βββ openenv.yaml # OpenEnv metadata and spec | |
| βββ requirements.txt | |
| βββ app.py # FastAPI server | |
| βββ environment/ | |
| β βββ models.py # Pydantic v2 models | |
| β βββ env.py # CodeSentinelEnv (reset/step/state) | |
| β βββ reward.py # Non-sparse reward computation | |
| βββ tasks/ | |
| β βββ registry.py # Episode builder and task metadata | |
| βββ graders/ | |
| β βββ base.py # Abstract grader | |
| β βββ easy_grader.py # Bug detection grader | |
| β βββ medium_grader.py # Security audit grader | |
| β βββ hard_grader.py # Architecture review grader | |
| βββ data/ | |
| β βββ samples.py # Embedded code samples and ground truth | |
| βββ baseline/ | |
| βββ run_baseline.py # GPT-4o evaluation script | |
| ``` | |
| --- | |
| ## License | |
| MIT β see `openenv.yaml` for metadata. | |