Hugging Face's logo

Spaces:
CoreModelsResearch
/
DAO
Sleeping

App Files Community
DAO / docs /observatory /mas_alignment_audit.md
sipe5001's picture
sipe5001
Add Hugging Face Docker Space configuration
d3d0e0e
preview code
|
Raw History Blame Contribute Delete
20.6 kB

MAS Alignment Audit

Scope reviewed:

  • src/data_agent_baseline/langgraph_agent/aat
  • Related orchestration and state code in src/data_agent_baseline/langgraph_agent
  • Primary runtime evidence: /data3/dataFAIR/kdd-dev/public/artifacts/runs/20260614T073404Z/task_355/trace.json

Method:

  • Static code inspection with line-level references
  • Trace inspection of task_355 for runtime behavior
  • No conceptual assumptions beyond observed implementation

1. Agent Identity

Explore

Planner

Critic Plan

Execute

Critic Execute

Answer (FinalSummaryGenerator)

Additional AAT agents

StrategicCoordinator

SchemaAgent, DomainAgent, DocumentAgent

AnalysisSynthesizer, Verifier

Conclusion for identity:

  • Multiple role-specific LLM calls exist.
  • Most components are role functions or thin wrappers around one-shot prompts.
  • No long-lived autonomous per-agent state/memory.

2. Memory Isolation

State objects

Run-level, task-level, agent-level memory

Read/Write table

Agent Reads Writes Private Memory? Shared Memory?
Explore task context_dir, doc_texts, etc. from AgentState data_exploration, auto_db_path, steps No Yes
Semantic Extraction question, knowledge_md, doc_texts, auto_db_path semantic_context, rewritten_question, data_exploration, steps No Yes
Planner rewritten_question/question, knowledge_md, semantic_context, data_exploration, plan_critique, result_critique, previous_plans plan, expected_output_column_count, expected_row_count, steps, previous_plans No Yes
Critic Plan plan + global semantic/data context plan_critique, steps No Yes
Executor plan + global semantic/data context + execution_memory generated_code, execution_result, execution_error, execution_stdout, steps No Yes
Critic Execute execution_result + global semantic/data context + plan result_critique, cross_validation, steps No Yes
Coordinator CP1/2/3 shared AgentState snapshot understanding_review/planning_review/final_review, coordinator_decisions, traces No Yes
SchemaAgent SchemaAgentContext projection built from shared state schema_analysis (via graph), trace No durable private memory Yes
DomainAgent DomainAgentContext projection domain_analysis, trace No durable private memory Yes
DocumentAgent DocumentAgentContext projection document_analysis, trace No durable private memory Yes
Synthesizer specialist outputs from shared state synthesized_analysis, trace No Yes
Verifier shared execution outputs + critiques verification_report, trace No Yes
FinalSummaryGenerator shared outputs + coordinator decision final_summary, trace No Yes

Isolation verdict:

  • Planner memory unavailable to Critic: No
  • Critic memory unavailable to Executor: No
  • Executor memory unavailable to Planner: No
  • System is shared-state dominant.

3. Context Window Analysis

Baseline core calls (broad context)

Finding:

  • Planner/Critic/Executor/CriticExecute all receive large overlapping global context from the same state object.

AAT calls (scoped context exists)

Finding:

  • Specialists have scoped call-time context.
  • But outputs are merged back into shared AgentState, and downstream core nodes still use broad shared context.

4. Communication Model

Observed model:

Implementation classification:

  • A) Shared state only
  • Not B) explicit message-passing between autonomous agents

5. Control Flow Analysis

Fixed sequencing dominates

Dynamic behavior present but bounded

Control verdict:

  • Mostly fixed workflow with bounded dynamic branching.
  • Not open-ended autonomous routing.

6. Trace Investigation (task_355)

Primary evidence file:

  • /data3/dataFAIR/kdd-dev/public/artifacts/runs/20260614T073404Z/task_355/trace.json

Sequential stage execution evidence

AAT evidence present as side-channel traces

Autonomy and dependency assessment per step group

  • Explore steps consume context_dir and produce shared data_exploration and semantic inputs.
  • Planner consumes shared exploration+semantic context and produces plan.
  • Critic Plan consumes same shared context plus plan.
  • Executor consumes shared plan and shared memory feedback; writes result.
  • Critic Execute consumes shared result and same global context; writes critique.
  • No explicit agent-to-agent messages; dependency chain is via common shared state fields.

Private memory evidence in trace

  • No private per-agent memory objects serialized.
  • Trace has global steps plus agent_traces metadata and coordinator_decisions metadata, not isolated memory segments.

7. MAS Scorecard

Criterion Score Evidence
Agent Identity Partial Distinct classes/prompts exist, but mostly thin wrappers/function stages without durable identities.
Memory Isolation Fail Single shared AgentState for all core stages.
Private Context Partial Specialist contexts are scoped, but core Planner/Critic/Executor all ingest broad shared context.
Message Passing Fail Shared-state handoff, no runtime message bus/inbox/outbox.
Autonomous Decisions Partial Coordinator can select specialists and suggest replan, but bounded.
Dynamic Routing Partial Some conditional branches; overall sequence remains fixed pipeline.
Agent-owned State Fail No durable per-agent memory/config beyond model handle.
Independent Critique Partial Critic and Verifier are separate calls; still fed from same shared state and central controller.
Agent-to-Agent Communication Fail No explicit Planner->Executor or Critic->Planner message artifacts.

8. Final Verdict

  1. HYBRID SYSTEM

Reason:

  • It is not a pure role-only single prompt chain because it has additional coordinator and specialist calls with scoped projections and some dynamic branching.
  • It is not a true MAS because memory, control, and communication are centrally orchestrated through one shared AgentState with no agent-owned memory or explicit inter-agent messaging.

9. Improvement Recommendations (code-level)

Recommendation 1: Introduce agent-owned memory stores

  • File: src/data_agent_baseline/langgraph_agent/state.py
  • Function/Area: AgentState definition
  • Proposed change:
    • Add agent_memory: dict[str, dict[str, Any]] keyed by agent id.
    • Move execution_memory into executor-specific memory namespace.
    • Add read/write APIs to enforce ownership boundaries.
  • Expected MAS benefit:
    • Real memory isolation and persistent agent identity across turns/retries.

Recommendation 2: Add explicit message artifacts between agents

  • File: src/data_agent_baseline/langgraph_agent/state.py
  • Function/Area: AgentState + new dataclasses
  • Proposed change:
    • Add AgentMessage dataclass with sender, receiver, payload, timestamp, correlation_id.
    • Add message_queue and per-agent inbox/outbox.
  • Expected MAS benefit:
    • Communication becomes explicit, inspectable, and decoupled from global mutable state.

Recommendation 3: Rewire graph to route on message outcomes instead of direct state mutation

  • File: src/data_agent_baseline/langgraph_agent/graph.py
  • Function: _run_agent_graph_aat_inner
  • Proposed change:
    • Replace direct writes/reads sequence with dispatcher loop:
      • Coordinator emits route messages.
      • Agents consume only their inbox and produce artifacts/messages.
      • Router decides next recipient from message types.
  • Expected MAS benefit:
    • Autonomous routing and less hard-coded sequencing.

Recommendation 4: Enforce coordinator final decision in termination logic

  • File: src/data_agent_baseline/langgraph_agent/graph.py
  • Function: _run_agent_graph_aat_inner
  • Proposed change:
    • Use coordinator_approved gate for final status, not only execution_result presence.
    • If decision is RETRY_EXECUTION or REPLAN, route accordingly before success.
  • Expected MAS benefit:
    • Coordinator becomes an actual control authority, not advisory metadata.

Recommendation 5: Make Planner/Critic/Executor consume role-scoped contexts

  • File: src/data_agent_baseline/langgraph_agent/nodes.py
  • Functions: planner_node, critic_plan_node, executor_node, critic_execute_node
  • Proposed change:
    • Introduce context builder layer similar to aat/context.py.
    • Remove broad injection of full knowledge_md + data_exploration + semantic_context into every role.
  • Expected MAS benefit:
    • True context isolation and reduced context bleed between roles.

Recommendation 6: Attach per-agent lifecycle ids and persistent configs

  • File: src/data_agent_baseline/langgraph_agent/aat/specialist_agents.py, coordinator.py, verifier.py, summary.py
  • Function/Area: class constructors and run methods
  • Proposed change:
    • Add agent_id, policy/config object, and optional memory handle in each class.
    • Persist these in trace and state.
  • Expected MAS benefit:
    • Stronger agent identity and reproducible autonomy boundaries.

Recommendation 7: Extend trace schema to prove autonomy

  • File: src/data_agent_baseline/langgraph_agent/runner.py
  • Function: _save_result
  • Proposed change:
    • Persist message_queue snapshots, per-agent memory snapshots, routing rationale, and inbox/outbox transitions per step.
  • Expected MAS benefit:
    • Observable evidence of MAS behavior, not only role invocation timing.

Recommendation 8: Promote specialist outputs into planner input contract

  • File: src/data_agent_baseline/langgraph_agent/nodes.py and graph.py
  • Functions: planner_node, _run_agent_graph_aat_inner
  • Proposed change:
    • Planner must consume synthesized_analysis explicitly (not only semantic_context/data_exploration).
    • Add hard validation that plan cites specialist artifacts.
  • Expected MAS benefit:
    • Specialist agents materially influence downstream behavior.

Bottom line

Current implementation is best characterized as a centrally orchestrated shared-state workflow with multi-role LLM components and some dynamic branching, not a fully autonomous MAS with isolated memories and explicit inter-agent communication.