rsnarsna
feat(lab): implement Phase 2 project structure and inventory integrity test suite
e06cbf7
|
Raw History Blame Contribute Delete
4.77 kB

ALLI Node Validation Lab

Welcome to the ALLI Node Validation Lab, the authoritative verification, contract enforcement, and cross-tier parity validation framework for all visual programming nodes across the ALLI ecosystem.


🏛️ Mission & Architectural Context

ALLI (Advanced Logic & Linked Intelligence) operates on a strict multi-tier architecture where visual graphs defined in the frontend canvas (@xyflow/react) are translated into deterministic Python AST representations and executed across isolated runtimes (backend subprocess, air-gapped Docker, or browser in-process fallback).

The Node Validation Lab establishes an unbreakable contract testing ground to ensure:

  1. Zero Silent Failures: Every node schema is strictly validated across frontend and backend registries.
  2. Deterministic Parity: Input ports, output ports, parameter constraints, and default values are synchronized or explicitly tracked.
  3. Decoupled Execution & Codegen: Code generation capability is validated independently from execution authorization, preserving sandbox security boundaries.
  4. Air-Gapped Isolation: Guarded AI-agent nodes and unapproved external libraries are kept securely in check.

🚦 Result Status Definitions

Every test, capability check, and node validation contract yields one of four canonical result states:

Status Definition Interpretation & Action
PASS Capability Verified The node schema, execution handler, or Python code generator behaves strictly according to specification and passes all contract assertions.
FAIL Contract Violation The node was expected to operate, but raised an unhandled exception, syntax error, port mismatch, or returned invalid outputs.
UNSUPPORTED Guarded / Out of Scope The capability is intentionally disabled or unsupported for the current execution tier (e.g., executing AI Agent loops in browser, or unapproved heavy third-party wheels without Docker).
NOT_TESTED Pending Evaluation The capability is declared in the schema contract but has not yet been exercised by the current validation phase.

📁 Lab Architecture & Directory Structure

alli_backend/node_validation_lab/
├── README.md               # Complete lab architecture and operator guide (this file)
├── run_validation.py       # Deterministic CLI validation runner & report generator
├── schemas/
│   ├── __init__.py         # Schema package exports
│   └── models.py           # Canonical Pydantic models (NodeValidationContract, etc.)
├── fixtures/
│   ├── __init__.py         # Fixture package exports
│   └── inventory_loader.py # Safe, deterministic inventory loader (no arbitrary eval)
├── tests/
│   ├── __init__.py         # Pytest test package
│   └── test_inventory_integrity.py # 8 Core integrity verification tests
└── reports/
    └── inventory_integrity_report.json # Deterministic machine-readable audit report

⚡ CLI Usage & Commands

1. Execute Full Inventory Integrity Audit (CLI)

python node_validation_lab/run_validation.py

Outputs a formatted ASCII dashboard with aggregate metrics, capability breakdowns, and verification badges, and updates reports/inventory_integrity_report.json.

2. Custom Report Destination

python node_validation_lab/run_validation.py --report-path /custom/path/report.json

3. Quiet Execution (CI/CD Pipelines)

python node_validation_lab/run_validation.py --quiet

Returns exit code 0 on integrity success, or 1 on failure, without printing console tables.

4. Strict Mode (Enforces 0 Schema Mismatches)

python node_validation_lab/run_validation.py --strict

Enforces zero tolerance for schema mismatches, exiting 1 if any node is classified as SCHEMA_MISMATCH.

5. Run Pytest Suite

pytest node_validation_lab/tests/test_inventory_integrity.py -v

🛡️ Security Boundaries & Constraints

  1. No Arbitrary Package Execution: inventory_loader.py uses strictly isolated JSON parsing (json.load) and Pydantic validation. It never calls eval(), exec(), or dynamic wheel imports.
  2. Guarded AI Agent Runtimes: The 6 agent.* nodes are flagged as UNSUPPORTED for untrusted execution tiers and cannot be triggered without verified LangGraph checkpointer credentials.
  3. Decoupled Execution vs. Codegen: Metadata indicating a node can generate Python code never implies runtime authorization.
  4. Zero Credential Exposure: Integrity reports scrub all environment variables (AWS_*, SUPABASE_*, MONGODB_URI) and assert zero secret leakage.