DataEngEnv / blog.md
kavinkrisher's picture
Add blog.md and training curve image
f1148bf
|
Raw History Blame Contribute Delete
11.9 kB
metadata
title: >-
  PipelineOps Arena: We Built an RL Environment Around the Bugs That Actually
  Break Production
thumbnail: https://huggingface.co/spaces/CoBeDigger/DataEngEnv/resolve/main/imgmeta.png
authors:
  - user: CoBeDigger

A 4-stage cascading ML pipeline debugger for RL agents β€” Meta AI Γ— Hugging Face OpenEnv Hackathon 2026.


Every ML team has shipped a broken model to production at least once. The pipeline looked fine. The numbers looked good. But somewhere between the data loading and the deployment gate, something went wrong β€” silently, invisibly, and expensively.

We built PipelineOps Arena because these bugs are real, they happen constantly, and no one has tried to train an AI to catch them systematically.

Try it live: huggingface.co/spaces/CoBeDigger/DataEngEnv


The Four Bugs We're Testing

We didn't invent these failure modes. We've lived them.

Stage 1 β€” Data Repair. Someone renamed a column upstream. The script references age_years but the DataFrame column is age. KeyError. The dataset also has ~15% NaN rows that crash StandardScaler with ValueError. Two bugs, both required to pass.

Stage 2 β€” Training Monitor. An MLPClassifier is fed raw unscaled features β€” salary around 70,000 sitting next to employment years around 8. Loss diverges to NaN after epoch 1. No crash. Silent divergence.

Stage 3 β€” Eval Validation. scaler.fit_transform() runs on the entire dataset before train_test_split(). The model has already seen test data during normalization. It prints Accuracy: 0.98. It's lying.

Stage 4 β€” Deploy Gate. The model passes accuracy. It passes inference latency. But the MLOps bot measures a 0.09 demographic fairness gap across age groups β€” above the 0.05 deployment threshold. Deployment rejected.

These aren't abstract challenges. They are the exact class of bugs that get missed in code review because they don't crash β€” they just corrupt.


How the Environment Works

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚                    PipelineOps Arena                     β”‚
β”‚                                                          β”‚
β”‚   Stage 1      Stage 2      Stage 3       Stage 4        β”‚
β”‚  Data Repair β†’ Training  β†’ Eval Fix  β†’ Deploy Gate      β”‚
β”‚  (KeyError)    (NaN loss)  (Leakage)    (Fairness)       β”‚
β”‚                                                          β”‚
β”‚   Each stage: inspect β†’ edit β†’ run β†’ submit              β”‚
β”‚   Grade β‰₯ 0.70  β†’  advance to next stage                β”‚
β”‚   Grade < 0.70 at Stage 4  β†’  loop back (βˆ’0.30)         β”‚
β”‚   Max steps: 60 total, 15 per stage                      β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

The agent talks to a FastAPI server over plain HTTP β€” no special SDK, just REST. Built on the OpenEnv spec.

At each step the agent gets an observation like this:

{
  "current_stage": 2,
  "step_number": 7,
  "script_content": "...current pipeline code...",
  "last_run_error": "ValueError: Input contains NaN",
  "data_preview": "Shape: (500, 6)\nColumns: ['age', 'salary', ...]",
  "actor_feedback": "StandardScaler not found in script.",
  "stages_completed": [1],
  "done": false
}

And picks one of five actions:

Action What it does
inspect_data Returns dataset shape, columns, null counts, sample rows
run_script Executes current pipeline in a 10-second isolated sandbox
edit_script Patches specific text in the script, or replaces it entirely
query_actor Asks the Code Reviewer (Stages 2–3) or MLOps Bot (Stage 4)
submit Triggers the deterministic grader

The key design decision: every reward comes from objective, code-based evaluation. No LLM-as-judge. No string matching. The grader runs the fixed script against real data and measures actual accuracy, convergence, and fairness gap. If the fix works, it works.


The Cascade Mechanic

The stages aren't just sequential β€” they're designed to punish shallow fixes.

Stage 4 has a loop mechanic: fail the fairness gate and the agent goes back to Stage 1 with a βˆ’0.30 penalty. Two failed loops end the episode at 0.00.

This forces genuine long-horizon planning. An agent that blindly submits at every stage will hit Stage 4, fail the fairness check, and lose everything it earned. The fairness gap (0.09 across age groups) only becomes visible when the agent explicitly queries the MLOps Bot β€” which runs the script in a subprocess and measures it directly. The hint is there: "Consider class_weight='balanced'". But the agent has to know to ask.


Two Reviewers the Agent Can Consult

Code Reviewer (Stages 2–3): Regex-based. Checks for data leakage patterns β€” whether scaler.fit() was called before or after the split. Gives natural-language feedback like "Scaler is being fit on the full dataset. This leaks test statistics into training."

MLOps Bot (Stage 4): Actually runs the agent's current script in a subprocess with a fairness-gap measurement appended. Reports accuracy, inference latency, and the demographic gap with explicit pass/fail labels. Every query_actor call re-executes β€” it's compute-heavy and intentional. The agent should only call it when it's ready for real feedback.


Reward Structure

Dense rewards throughout β€” partial credit for doing the right things, not just for completing stages.

inspect_data (first time)           +0.10
edit_script: fix column name        +0.20
edit_script: add dropna/clip        +0.15
run_script: clean after fix         +0.20
edit_script: add StandardScaler     +0.30
edit_script: fix data leakage       +0.30
query_actor: MLOps Bot              +0.10
edit_script: add class_weight       +0.30
submit: stage complete              grader score [0.0–1.0]
Stage 4 failure (loop back)         βˆ’0.30
Repeated action                     βˆ’0.03

A perfect run β€” all 4 stages cleared without looping β€” returns 1.00.


Training: Llama 3.1 8B with GRPO

We fine-tuned Llama 3.1 8B (4-bit quantized via Unsloth) using GRPO β€” Group Relative Policy Optimization β€” on Kaggle T4 GPU.

Setup: LoRA r=16, all projection layers, 0.26% trainable params, TRL GRPOTrainer.

Why Semantic Reward

The first thing we tried was episode completion as the reward signal. It produced nothing.

The untrained Llama 3.1 8B instruct model doesn't output {"action_type": "edit_script", "payload": {...}}. It outputs rich descriptive JSON β€” multi-step plans, code analysis, detailed reasoning about the bug. All valid ML thinking, none of it parseable as an environment action.

That meant every episode returned 0.00. reward_std = 0.0. GRPO with zero variance produces zero gradient. The model learned nothing.

So we built a semantic reward function that scores each response on format quality alone:

+0.20  Valid JSON output
+0.20  action_type key present
+0.30  Correct action keyword for current context
+0.30  Correct payload structure

This decoupled learning from episode completion. The model gets a gradient signal just for structuring its output correctly β€” which is the prerequisite for everything else.

Results

GRPO Training Curves β€” Avg Reward and Policy Loss per Step

Left: avg group reward per training step. Right: GRPO policy loss per step. Both from a real 25-step run on Kaggle T4.

Metric Value
Base model Llama 3.1 8B 4-bit (Unsloth)
Training steps 25
Group size (G) 4 completions per step
First 5 avg reward 0.25
Last 5 avg reward 0.38
Improvement +52%

The reward went from 0.25 to 0.38 over 25 steps. The variance is the real signal β€” early steps show wide swings as the model explores different formats; later steps tighten. The model learned that structured JSON is rewarded, and started producing it more reliably.

The untrained model scored 0.00 on episode completion because it couldn't produce parseable actions. The trained adapter pushes toward the correct action format β€” which is the necessary first step before episode-level RL becomes possible.

Trained model: CoBeDigger/pipelineops-arena-llama3-grpo

Training notebook (Colab, re-runnable): GRPO_Training_Pipeline.ipynb


The Baseline: What Perfect Looks Like

We built a hardcoded rule-based agent that executes the optimal action sequence per stage. It completes all 4 stages in 16 steps with a score of 1.00. This validates the graders and gives a ceiling to measure against.

Stage 1: edit_script (column fix + dropna) β†’ run_script β†’ submit       [3 steps]
Stage 2: inspect_data β†’ edit_script (add scaler) β†’ run_script β†’ submit  [4 steps]
Stage 3: run_script β†’ edit_script (fix leakage) β†’ run_script β†’ submit   [4 steps]
Stage 4: query_actor β†’ edit_script (class_weight) β†’ run_script β†’ submit [4 steps]

You can watch it run live on the Space under the "Baseline vs LLM Agent" tab.

Video walkthrough: Watch on YouTube


Try It

No authentication required. Just HTTP.

# Start an episode
curl -X POST https://cobedigger-dataengenv.hf.space/reset

# Take a step
curl -X POST https://cobedigger-dataengenv.hf.space/step \
  -H "Content-Type: application/json" \
  -d '{"action_type": "inspect_data", "payload": {}}'

# Check pipeline progress
curl https://cobedigger-dataengenv.hf.space/pipeline_status

Or run an agent loop:

import requests

ENV = "https://cobedigger-dataengenv.hf.space"
obs = requests.post(f"{ENV}/reset").json()

for step in range(60):
    action = your_agent.decide(obs)
    result = requests.post(f"{ENV}/step", json=action).json()
    obs = result["observation"]
    reward = result["reward"]["score"]
    if obs["done"]:
        break

Links:


What We Learned

Reward design is the hardest part. Getting a gradient signal out of a model that can't produce valid JSON yet requires rethinking what "reward" means at the start of training. Semantic reward isn't a hack β€” it's necessary scaffolding.

Real execution beats string matching. Every grader in this environment runs actual Python code against real data. This makes it harder to game and means reward always reflects genuine task completion.

The cascade forces planning. Agents that treat each stage independently will fail Stage 4 and lose 0.30 points. The loop mechanic rewards agents that maintain a holistic view of the pipeline β€” not just the current bug.


CobeDivers β€” Jey Pranav, Kavin Krish, Hari Meta AI Γ— Hugging Face OpenEnv Hackathon 2026