docs/implementation/HITL_STEP_RECORDING_COMPLETE.md

HITL Step Recording Implementation - Complete Summary

Overview

HITL (Human-In-The-Loop) checkpoint decisions are now recorded as separate steps in trace.json, making the full checkpoint lifecycle available for audit and review alongside existing events in run_events.jsonl.

Changes Made

Modified File

• src/data_agent_baseline/langgraph_agent/graph.py

Implementation Details

1. New Helper Function: _record_hitl_step()

Added function that:

• Creates a standardized HITL step object matching the trace.json format
• Captures decision type (approve/cancel/revise/timeout_auto_approve)
• Records original and revised plans
• Includes human steering instructions
• Preserves checkpoint configuration (timeout_seconds, default_action_on_timeout)
• Appends step to state.steps for inclusion in final trace.json
• Emits optional callback event for run_events.jsonl

Location: Lines ~44-107 in graph.py

2. Integration Points

Function is called in _maybe_handle_planner_checkpoint() at 4 decision paths:

A) APPROVE Decision (Lines ~365-378)

_record_hitl_step(
    state,
    checkpoint_id=checkpoint.checkpoint_id,
    decision=decision_record.decision,
    message="Checkpoint approved. Resuming execution.",
    original_plan=original_plan,
    revised_plan=original_plan,
    ...
)

B) CANCEL Decision (Lines ~403-416)

_record_hitl_step(
    state,
    checkpoint_id=checkpoint.checkpoint_id,
    decision=decision_record.decision,
    message="Task cancelled by human at planner checkpoint.",
    original_plan=original_plan,
    revised_plan=None,
    human_instruction=decision_record.human_instruction,
    ...
)

C) REVISE Decision (Lines ~445-458)

_record_hitl_step(
    state,
    checkpoint_id=checkpoint.checkpoint_id,
    decision=decision_record.decision,
    message="Revision submitted and replanning completed...",
    original_plan=original_plan,
    revised_plan=revised_plan,  # After replanning
    human_instruction=decision_record.human_instruction,
    ...
)

D) TIMEOUT_AUTO_APPROVE Decision (Lines ~315-327)

_record_hitl_step(
    state,
    checkpoint_id=checkpoint.checkpoint_id,
    decision=InterventionDecision.TIMEOUT_AUTO_APPROVE.value,
    message=f"No human response within {policy.timeout_seconds}s...",
    original_plan=original_plan,
    revised_plan=original_plan,
    ...
)

Result Format in trace.json

HITL steps appear in trace.json["steps"] as:

{
  "step_index": <number>,
  "action": "planner_review",
  "phase": "planner_review",
  "thought": "Human-in-the-loop checkpoint review (checkpoint_id=cp_...)",
  "action_input": {
    "checkpoint_id": "cp_...",
    "decision": "approve|cancel|revise|timeout_auto_approve"
  },
  "raw_response": "(human decision)",
  "observation": {
    "ok": true,
    "tool": "planner_review",
    "content": {
      "checkpoint_id": "cp_...",
      "decision": "...",
      "message": "...",
      "original_plan": { /* Full plan object */ },
      "revised_plan": { /* Full plan object or null */ },
      "human_instruction": "...",
      "timeout_seconds": 60,
      "default_action_on_timeout": "continue"
    }
  },
  "ok": true
}

Example Usage

View HITL steps in trace.json

# View all HITL review steps
jq '.steps[] | select(.action == "planner_review")' trace.json

# View the decision for a specific checkpoint
jq '.steps[] | select(.action == "planner_review") | .observation.content' trace.json

# Count HITL interactions
jq '[.steps[] | select(.action == "planner_review")] | length' trace.json

Review approval decisions

jq '.steps[] | select(.action == "planner_review" and .observation.content.decision == "approve")' trace.json

Compare original vs revised plans

jq '.steps[] | select(.action == "planner_review" and .observation.content.decision == "revise") | {original: .observation.content.original_plan, revised: .observation.content.revised_plan}' trace.json

Backward Compatibility

✅ Fully backward compatible:

• Existing non-HITL runs unaffected
• HITL steps appended to steps array (no structure changes)
• Existing event emission to run_events.jsonl unchanged
• All existing visualization tools continue to work

Data Retention

The implementation ensures:

• Immutable Audit Trail: HITL decisions persisted in trace.json
• Full Context: Original and revised plans stored together
• Human Intent: Steering instructions preserved
• Timing: Checkpoint timeout configuration captured
• Traceability: checkpoint_id links to run_events.jsonl events

Testing

Comprehensive test file validates all decision paths:

• ✅ Approval decisions
• ✅ Cancel decisions
• ✅ Revision decisions with plan changes
• ✅ Timeout auto-approve scenarios
• ✅ Multiple HITL steps with proper indexing
• ✅ JSON serialization compatibility

Test file: test_hitl_step_recording.py

Run with: python test_hitl_step_recording.py

Verification

To verify the implementation in a live run:

# 1. Run a guided task with HITL
cd /workspace/ainn-cm-poc-data-agent
python -m data_agent_baseline.cli run-lang-task --task-id task_145 --guided

# 2. Check trace.json for HITL step
cat artifacts/runs/*/task_145/trace.json | jq '.steps[-1]'

# 3. Verify it appears with correct structure
jq 'if .steps[-1].action == "planner_review" then "✅ HITL step found" else "❌ HITL step not found" end' trace.json

Benefits

✨ Key Advantages:

1. Audit & Compliance: Complete record of human decisions
2. Review: Easy visualization of checkpoint lifecycle
3. Analysis: Patterns in human steering decisions
4. Debugging: Understand why plans changed
5. Reproducibility: Exact conditions for replanning captured
6. Integration: Works with existing observatory tools

Notes

• HITL steps have "ok": true (they're decisions, not tool failures)
• phase is always "planner_review" (distinguishes from other actions)
• Multiple HITL steps per task are possible (e.g., approve, then revise)
• Each step has independent checkpoint_id for linking to events