Blame for docs/observatory/mas_alignment_audit.md

# MAS Alignment Audit

Scope reviewed:
- src/data_agent_baseline/langgraph_agent/aat
- Related orchestration and state code in src/data_agent_baseline/langgraph_agent
- Primary runtime evidence: /data3/dataFAIR/kdd-dev/public/artifacts/runs/20260614T073404Z/task_355/trace.json

Method:
- Static code inspection with line-level references
- Trace inspection of task_355 for runtime behavior
- No conceptual assumptions beyond observed implementation

## 1. Agent Identity

### Explore
- Persistent identity: No
  - Implemented as function explore_data_node, not a long-lived object: [src/data_agent_baseline/langgraph_agent/nodes.py](src/data_agent_baseline/langgraph_agent/nodes.py#L488)
- Own configuration: No separate config object
  - Reads from shared AgentState fields and global constants
- Own prompt: No LLM prompt in this node (tooling/data prep only)
- Own memory: No
  - Writes directly into shared AgentState.data_exploration and steps: [src/data_agent_baseline/langgraph_agent/nodes.py](src/data_agent_baseline/langgraph_agent/nodes.py#L675)

### Planner
- Persistent identity: No
  - Function node, no agent class: [src/data_agent_baseline/langgraph_agent/nodes.py](src/data_agent_baseline/langgraph_agent/nodes.py#L1070)
- Own configuration: No
  - Uses shared state and graph retry counters
- Own prompt: Yes (role prompt exists)
  - Uses PLANNER_SYSTEM and PLANNER_USER via formatting: [src/data_agent_baseline/langgraph_agent/nodes.py](src/data_agent_baseline/langgraph_agent/nodes.py#L1138)
- Own memory: No private memory
  - Reads/writes shared state.plan, state.plan_attempts, state.previous_plans

### Critic Plan
- Persistent identity: No
  - Function node: [src/data_agent_baseline/langgraph_agent/nodes.py](src/data_agent_baseline/langgraph_agent/nodes.py#L1206)
- Own configuration: No
- Own prompt: Yes
  - CRITIC_PLAN_SYSTEM and CRITIC_PLAN_USER: [src/data_agent_baseline/langgraph_agent/nodes.py](src/data_agent_baseline/langgraph_agent/nodes.py#L1225)
- Own memory: No private memory
  - Writes shared state.plan_critique

### Execute
- Persistent identity: No
  - Function node: [src/data_agent_baseline/langgraph_agent/nodes.py](src/data_agent_baseline/langgraph_agent/nodes.py#L1499)
- Own configuration: No separate agent config
- Own prompt: Yes
  - EXECUTOR_SYSTEM and EXECUTOR_USER: [src/data_agent_baseline/langgraph_agent/nodes.py](src/data_agent_baseline/langgraph_agent/nodes.py#L1546)
- Own memory: No private memory
  - Uses shared state.execution_memory and writes shared state.execution_result/error

### Critic Execute
- Persistent identity: No
  - Function node: [src/data_agent_baseline/langgraph_agent/nodes.py](src/data_agent_baseline/langgraph_agent/nodes.py#L1277)
- Own configuration: No
- Own prompt: Yes
  - CRITIC_EXECUTE_SYSTEM and CRITIC_EXECUTE_USER: [src/data_agent_baseline/langgraph_agent/nodes.py](src/data_agent_baseline/langgraph_agent/nodes.py#L1319)
- Own memory: No private memory
  - Writes shared state.result_critique

### Answer (FinalSummaryGenerator)
- Persistent identity: Partial
  - Class exists, but object is instantiated per task run and only stores model pointer: [src/data_agent_baseline/langgraph_agent/aat/summary.py](src/data_agent_baseline/langgraph_agent/aat/summary.py#L41)
- Own configuration: No persistent per-agent config
- Own prompt: Yes
  - SUMMARY_GENERATOR_SYSTEM and summary_generator_user: [src/data_agent_baseline/langgraph_agent/aat/summary.py](src/data_agent_baseline/langgraph_agent/aat/summary.py#L88)
- Own memory: No private memory
  - Reads shared state and writes state.final_summary in graph orchestration: [src/data_agent_baseline/langgraph_agent/graph.py](src/data_agent_baseline/langgraph_agent/graph.py#L945)

### Additional AAT agents

StrategicCoordinator
- Identity: Partial class identity only
- Config: No persistent config beyond model/logger
- Prompt: Yes, 3 checkpoints
- Memory: No private memory store
- Evidence: [src/data_agent_baseline/langgraph_agent/aat/coordinator.py](src/data_agent_baseline/langgraph_agent/aat/coordinator.py#L72)

SchemaAgent, DomainAgent, DocumentAgent
- Identity: Partial class identity only
- Config: No persistent config
- Prompt: Yes, specialist prompts
- Memory: No private memory, single-call output object
- Evidence class init only keeps model: [src/data_agent_baseline/langgraph_agent/aat/specialist_agents.py](src/data_agent_baseline/langgraph_agent/aat/specialist_agents.py#L125), [src/data_agent_baseline/langgraph_agent/aat/specialist_agents.py](src/data_agent_baseline/langgraph_agent/aat/specialist_agents.py#L169), [src/data_agent_baseline/langgraph_agent/aat/specialist_agents.py](src/data_agent_baseline/langgraph_agent/aat/specialist_agents.py#L212)

AnalysisSynthesizer, Verifier
- Identity: Partial class identity only
- Prompt: Yes
- Memory: No private memory store
- Evidence: [src/data_agent_baseline/langgraph_agent/aat/synthesizer.py](src/data_agent_baseline/langgraph_agent/aat/synthesizer.py#L53), [src/data_agent_baseline/langgraph_agent/aat/verifier.py](src/data_agent_baseline/langgraph_agent/aat/verifier.py#L56)

Conclusion for identity:
- Multiple role-specific LLM calls exist.
- Most components are role functions or thin wrappers around one-shot prompts.
- No long-lived autonomous per-agent state/memory.

## 2. Memory Isolation

### State objects
- Primary task-level shared object: AgentState dataclass: [src/data_agent_baseline/langgraph_agent/state.py](src/data_agent_baseline/langgraph_agent/state.py#L66)
- Shared attempt history field: execution_memory: [src/data_agent_baseline/langgraph_agent/state.py](src/data_agent_baseline/langgraph_agent/state.py#L107)
- AAT artifacts also in same shared AgentState (task_understanding, execution_blueprint, execution_progress, verification_summary, final_summary, coordinator_decisions, agent_traces): [src/data_agent_baseline/langgraph_agent/state.py](src/data_agent_baseline/langgraph_agent/state.py#L147)

### Run-level, task-level, agent-level memory
- Run-level state
  - Managed by runner (run_output_dir, config, task loop), not agent-owned memory: [src/data_agent_baseline/langgraph_agent/runner.py](src/data_agent_baseline/langgraph_agent/runner.py#L366)
- Task-level state
  - Entire execution carried in one AgentState object per task: [src/data_agent_baseline/langgraph_agent/runner.py](src/data_agent_baseline/langgraph_agent/runner.py#L228)
- Agent-level state
  - No durable memory objects per agent; classes keep model handle only: [src/data_agent_baseline/langgraph_agent/aat/coordinator.py](src/data_agent_baseline/langgraph_agent/aat/coordinator.py#L73)

### Read/Write table

| Agent | Reads | Writes | Private Memory? | Shared Memory? |
|---|---|---|---|---|
| Explore | task context_dir, doc_texts, etc. from AgentState | data_exploration, auto_db_path, steps | No | Yes |
| Semantic Extraction | question, knowledge_md, doc_texts, auto_db_path | semantic_context, rewritten_question, data_exploration, steps | No | Yes |
| Planner | rewritten_question/question, knowledge_md, semantic_context, data_exploration, plan_critique, result_critique, previous_plans | plan, expected_output_column_count, expected_row_count, steps, previous_plans | No | Yes |
| Critic Plan | plan + global semantic/data context | plan_critique, steps | No | Yes |
| Executor | plan + global semantic/data context + execution_memory | generated_code, execution_result, execution_error, execution_stdout, steps | No | Yes |
| Critic Execute | execution_result + global semantic/data context + plan | result_critique, cross_validation, steps | No | Yes |
| Coordinator CP1/2/3 | shared AgentState snapshot | understanding_review/planning_review/final_review, coordinator_decisions, traces | No | Yes |
| SchemaAgent | SchemaAgentContext projection built from shared state | schema_analysis (via graph), trace | No durable private memory | Yes |
| DomainAgent | DomainAgentContext projection | domain_analysis, trace | No durable private memory | Yes |
| DocumentAgent | DocumentAgentContext projection | document_analysis, trace | No durable private memory | Yes |
| Synthesizer | specialist outputs from shared state | synthesized_analysis, trace | No | Yes |
| Verifier | shared execution outputs + critiques | verification_report, trace | No | Yes |
| FinalSummaryGenerator | shared outputs + coordinator decision | final_summary, trace | No | Yes |

Isolation verdict:
- Planner memory unavailable to Critic: No
- Critic memory unavailable to Executor: No
- Executor memory unavailable to Planner: No
- System is shared-state dominant.

## 3. Context Window Analysis

### Baseline core calls (broad context)
- Planner prompt path injects: question, knowledge_md, file listing, full data_exploration, semantic_context text, feedback history
  - [src/data_agent_baseline/langgraph_agent/nodes.py](src/data_agent_baseline/langgraph_agent/nodes.py#L1138)
- Critic Plan injects: question, knowledge_md, semantic section, critical rules, data_exploration, full plan JSON
  - [src/data_agent_baseline/langgraph_agent/nodes.py](src/data_agent_baseline/langgraph_agent/nodes.py#L1225)
- Executor injects: plan JSON, knowledge_md, semantic critical rules, file listing, auto_db_path, data_exploration, execution_memory
  - [src/data_agent_baseline/langgraph_agent/nodes.py](src/data_agent_baseline/langgraph_agent/nodes.py#L1546)
- Critic Execute injects: question, output columns, knowledge_md, semantic section, generated code, stdout, result preview, shape
  - [src/data_agent_baseline/langgraph_agent/nodes.py](src/data_agent_baseline/langgraph_agent/nodes.py#L1319)

Finding:
- Planner/Critic/Executor/CriticExecute all receive large overlapping global context from the same state object.

### AAT calls (scoped context exists)
- SchemaAgentContext only schema-oriented projection: [src/data_agent_baseline/langgraph_agent/aat/context.py](src/data_agent_baseline/langgraph_agent/aat/context.py#L29)
- DomainAgentContext and DocumentAgentContext are separate typed projections: [src/data_agent_baseline/langgraph_agent/aat/context.py](src/data_agent_baseline/langgraph_agent/aat/context.py#L45), [src/data_agent_baseline/langgraph_agent/aat/context.py](src/data_agent_baseline/langgraph_agent/aat/context.py#L59)
- Builder functions enforce projection before specialist run: [src/data_agent_baseline/langgraph_agent/aat/context.py](src/data_agent_baseline/langgraph_agent/aat/context.py#L175)

Finding:
- Specialists have scoped call-time context.
- But outputs are merged back into shared AgentState, and downstream core nodes still use broad shared context.

## 4. Communication Model

Observed model:
- Primary communication is shared state mutation.
  - Graph writes each component output into AgentState and next component reads it: [src/data_agent_baseline/langgraph_agent/graph.py](src/data_agent_baseline/langgraph_agent/graph.py#L629), [src/data_agent_baseline/langgraph_agent/graph.py](src/data_agent_baseline/langgraph_agent/graph.py#L660), [src/data_agent_baseline/langgraph_agent/graph.py](src/data_agent_baseline/langgraph_agent/graph.py#L909)
- No explicit inter-agent message objects, inbox/outbox, bus, or event queue in runtime path.
- Trace contains agent_traces and coordinator_decisions as logs, not a message protocol: [src/data_agent_baseline/langgraph_agent/runner.py](src/data_agent_baseline/langgraph_agent/runner.py#L290)

Implementation classification:
- A) Shared state only
- Not B) explicit message-passing between autonomous agents

## 5. Control Flow Analysis

### Fixed sequencing dominates
- Core baseline order is hard-coded in graph:
  - Explore -> Semantic -> Plan -> CriticPlan -> Execute -> CriticExecute with loops
  - [src/data_agent_baseline/langgraph_agent/graph.py](src/data_agent_baseline/langgraph_agent/graph.py#L60)
- Execution loops and retries are hard-coded while/continue branches: [src/data_agent_baseline/langgraph_agent/graph.py](src/data_agent_baseline/langgraph_agent/graph.py#L298)

### Dynamic behavior present but bounded
- Coordinator CP1 dynamically selects specialists by list membership gates: [src/data_agent_baseline/langgraph_agent/graph.py](src/data_agent_baseline/langgraph_agent/graph.py#L655)
- Coordinator CP2 can request one replan branch: [src/data_agent_baseline/langgraph_agent/graph.py](src/data_agent_baseline/langgraph_agent/graph.py#L779)
- CP3 decision collected, but final status logic does not enforce coordinator_approved gate
  - Variable set here: [src/data_agent_baseline/langgraph_agent/graph.py](src/data_agent_baseline/langgraph_agent/graph.py#L976)
  - Final success still based on execution_result presence: [src/data_agent_baseline/langgraph_agent/graph.py](src/data_agent_baseline/langgraph_agent/graph.py#L982)

Control verdict:
- Mostly fixed workflow with bounded dynamic branching.
- Not open-ended autonomous routing.

## 6. Trace Investigation (task_355)

Primary evidence file:
- /data3/dataFAIR/kdd-dev/public/artifacts/runs/20260614T073404Z/task_355/trace.json

### Sequential stage execution evidence
- Steps 1-18 are linear pipeline actions list_context/read_csv/read_knowledge/build_auto_db/semantic_extraction/planner/critic_plan/execute_python/critic_execute repeats: [/data3/dataFAIR/kdd-dev/public/artifacts/runs/20260614T073404Z/task_355/trace.json](/data3/dataFAIR/kdd-dev/public/artifacts/runs/20260614T073404Z/task_355/trace.json#L21), [/data3/dataFAIR/kdd-dev/public/artifacts/runs/20260614T073404Z/task_355/trace.json](/data3/dataFAIR/kdd-dev/public/artifacts/runs/20260614T073404Z/task_355/trace.json#L1151)
- Action counts confirm this loop structure: planner=1, critic_plan=1, execute_python=5, critic_execute=5: [/data3/dataFAIR/kdd-dev/public/artifacts/runs/20260614T073404Z/task_355/trace.json](/data3/dataFAIR/kdd-dev/public/artifacts/runs/20260614T073404Z/task_355/trace.json#L1302)

### AAT evidence present as side-channel traces
- agent_traces include StrategicCoordinator, SchemaAgent, DomainAgent, DocumentAgent, Synthesizer, Verifier, FinalSummaryGenerator: [/data3/dataFAIR/kdd-dev/public/artifacts/runs/20260614T073404Z/task_355/trace.json](/data3/dataFAIR/kdd-dev/public/artifacts/runs/20260614T073404Z/task_355/trace.json#L1458)
- coordinator_decisions include final RETRY_EXECUTION: [/data3/dataFAIR/kdd-dev/public/artifacts/runs/20260614T073404Z/task_355/trace.json](/data3/dataFAIR/kdd-dev/public/artifacts/runs/20260614T073404Z/task_355/trace.json#L1602)

### Autonomy and dependency assessment per step group
- Explore steps consume context_dir and produce shared data_exploration and semantic inputs.
- Planner consumes shared exploration+semantic context and produces plan.
- Critic Plan consumes same shared context plus plan.
- Executor consumes shared plan and shared memory feedback; writes result.
- Critic Execute consumes shared result and same global context; writes critique.
- No explicit agent-to-agent messages; dependency chain is via common shared state fields.

### Private memory evidence in trace
- No private per-agent memory objects serialized.
- Trace has global steps plus agent_traces metadata and coordinator_decisions metadata, not isolated memory segments.

## 7. MAS Scorecard

| Criterion | Score | Evidence |
|---|---|---|
| Agent Identity | Partial | Distinct classes/prompts exist, but mostly thin wrappers/function stages without durable identities. |
| Memory Isolation | Fail | Single shared AgentState for all core stages. |
| Private Context | Partial | Specialist contexts are scoped, but core Planner/Critic/Executor all ingest broad shared context. |
| Message Passing | Fail | Shared-state handoff, no runtime message bus/inbox/outbox. |
| Autonomous Decisions | Partial | Coordinator can select specialists and suggest replan, but bounded. |
| Dynamic Routing | Partial | Some conditional branches; overall sequence remains fixed pipeline. |
| Agent-owned State | Fail | No durable per-agent memory/config beyond model handle. |
| Independent Critique | Partial | Critic and Verifier are separate calls; still fed from same shared state and central controller. |
| Agent-to-Agent Communication | Fail | No explicit Planner->Executor or Critic->Planner message artifacts. |

## 8. Final Verdict

2. HYBRID SYSTEM

Reason:
- It is not a pure role-only single prompt chain because it has additional coordinator and specialist calls with scoped projections and some dynamic branching.
- It is not a true MAS because memory, control, and communication are centrally orchestrated through one shared AgentState with no agent-owned memory or explicit inter-agent messaging.

## 9. Improvement Recommendations (code-level)

### Recommendation 1: Introduce agent-owned memory stores
- File: src/data_agent_baseline/langgraph_agent/state.py
- Function/Area: AgentState definition
- Proposed change:
  - Add agent_memory: dict[str, dict[str, Any]] keyed by agent id.
  - Move execution_memory into executor-specific memory namespace.
  - Add read/write APIs to enforce ownership boundaries.
- Expected MAS benefit:
  - Real memory isolation and persistent agent identity across turns/retries.

### Recommendation 2: Add explicit message artifacts between agents
- File: src/data_agent_baseline/langgraph_agent/state.py
- Function/Area: AgentState + new dataclasses
- Proposed change:
  - Add AgentMessage dataclass with sender, receiver, payload, timestamp, correlation_id.
  - Add message_queue and per-agent inbox/outbox.
- Expected MAS benefit:
  - Communication becomes explicit, inspectable, and decoupled from global mutable state.

### Recommendation 3: Rewire graph to route on message outcomes instead of direct state mutation
- File: src/data_agent_baseline/langgraph_agent/graph.py
- Function: _run_agent_graph_aat_inner
- Proposed change:
  - Replace direct writes/reads sequence with dispatcher loop:
    - Coordinator emits route messages.
    - Agents consume only their inbox and produce artifacts/messages.
    - Router decides next recipient from message types.
- Expected MAS benefit:
  - Autonomous routing and less hard-coded sequencing.

### Recommendation 4: Enforce coordinator final decision in termination logic
- File: src/data_agent_baseline/langgraph_agent/graph.py
- Function: _run_agent_graph_aat_inner
- Proposed change:
  - Use coordinator_approved gate for final status, not only execution_result presence.
  - If decision is RETRY_EXECUTION or REPLAN, route accordingly before success.
- Expected MAS benefit:
  - Coordinator becomes an actual control authority, not advisory metadata.

### Recommendation 5: Make Planner/Critic/Executor consume role-scoped contexts
- File: src/data_agent_baseline/langgraph_agent/nodes.py
- Functions: planner_node, critic_plan_node, executor_node, critic_execute_node
- Proposed change:
  - Introduce context builder layer similar to aat/context.py.
  - Remove broad injection of full knowledge_md + data_exploration + semantic_context into every role.
- Expected MAS benefit:
  - True context isolation and reduced context bleed between roles.

### Recommendation 6: Attach per-agent lifecycle ids and persistent configs
- File: src/data_agent_baseline/langgraph_agent/aat/specialist_agents.py, coordinator.py, verifier.py, summary.py
- Function/Area: class constructors and run methods
- Proposed change:
  - Add agent_id, policy/config object, and optional memory handle in each class.
  - Persist these in trace and state.
- Expected MAS benefit:
  - Stronger agent identity and reproducible autonomy boundaries.

### Recommendation 7: Extend trace schema to prove autonomy
- File: src/data_agent_baseline/langgraph_agent/runner.py
- Function: _save_result
- Proposed change:
  - Persist message_queue snapshots, per-agent memory snapshots, routing rationale, and inbox/outbox transitions per step.
- Expected MAS benefit:
  - Observable evidence of MAS behavior, not only role invocation timing.

### Recommendation 8: Promote specialist outputs into planner input contract
- File: src/data_agent_baseline/langgraph_agent/nodes.py and graph.py
- Functions: planner_node, _run_agent_graph_aat_inner
- Proposed change:
  - Planner must consume synthesized_analysis explicitly (not only semantic_context/data_exploration).
  - Add hard validation that plan cites specialist artifacts.
- Expected MAS benefit:
  - Specialist agents materially influence downstream behavior.

---

## Bottom line

Current implementation is best characterized as a centrally orchestrated shared-state workflow with multi-role LLM components and some dynamic branching, not a fully autonomous MAS with isolated memories and explicit inter-agent communication.