Wire ng_tract BTF into the organism: two tracts, no preclassification

Replaces the torch-pickled state blob with the canonical ecosystem
pattern (Rust Substrate Layer #119). Embeddings are now numpy f32
end-to-end — no more Python-list inflation on every deposit.

Two tracts, each doing one biological job (Substrate As Cortex):
- experience.tract: raw text via ng_tract.deposit_experience.
Full bytes, source-annotated, append-forever. Law 7 sensory
stream. Source tag ('user' / 'nuwave_response') is origin
annotation, not classification.
- outcomes.tract: ng_tract.deposit_outcome per node, embedding
as numpy f32 (zero-copy), metadata via msgpack. This is the
neuromodulatory learning signal — success bool is dopamine
polarity, not a classification label.

Preclassification purged from deposit metadata:
- No more "type": "experience" / "response" on node metadata —
the substrate discovers that distinction via STDP co-firing
- No more text[:200] truncation at deposit time — full text
enters; Pith bucket decides what to truncate at extraction
- No more "query": user_message[:100] on responses — the causal
link is recoverable from timestamp adjacency in the tract

Embeddings moved off node.metadata into self._embeddings side-table
(numpy f32 dict). Graph checkpoints stay lean. Pith reads from the
side-table directly — no np.array() conversion on the hot path.

Sidecar (nuwave_sidecar.pt) carries only substrate-dynamics state
that BTF doesn't naturally persist: graph checkpoint, amplitudes,
phases, node_content cache, stats. Never embeddings.

Legacy state migration: _apply_state pops embedding lists out of
node.metadata on restore and lifts them into self._embeddings as
numpy. Existing organism_state.pt on the hub still loads cleanly.

Hub push now uploads sidecar + both tracts. Tract files survive
container rebuilds and are available for future analysis buckets
(e.g. interleaved-benchmark neighborhood-overlap extraction).

Verified end-to-end: deposit → step → pith → record_outcome →
warm restart → legacy migration. All tracts appendable, numpy
embeddings round-trip zero-copy via embedding_as_numpy().

Refs: ~/docs/concepts/Rust Substrate Layer.md
~/docs/concepts/Substrate As Cortex.md
~/docs/prd/Binary_Tract_Format_v0.1.md

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

nuwave/organism.py

@@ -16,6 +16,25 @@ communication protocol (Law 1). Raw experience in, classification
 only at extraction (Law 7).
 
 # ---- Changelog ----
+# [2026-04-15] Claude Code (Opus 4.6) — BTF wire-in (Rust Substrate Layer)
+#   What: Wire ng_tract BTF into the deposit/outcome path. Two tracts:
+#         experience.tract (raw text, Law 7 sensory stream, append forever)
+#         outcomes.tract (embedding + reward, neuromodulatory log).
+#         Embeddings moved off node.metadata into self._embeddings
+#         side-table — kept as numpy f32 the whole way, no .tolist().
+#         Preclassification purged from deposit metadata: no more
+#         "type": "experience"/"response" labels, no text[:200] truncation
+#         at deposit time. Full bytes into the experience tract.
+#   Why:  See ~/docs/concepts/Substrate As Cortex.md — the substrate is
+#         cortex, not a database. Raw experience in, classification at
+#         extraction. ~/docs/concepts/Rust Substrate Layer.md — BTF is
+#         the canonical ecosystem pattern; embedding.tolist() is the
+#         exact inflation point BTF was built to eliminate (PRD §2).
+#   How:  deposit_experience(str, source, path, content_type) for raw
+#         sensory input. deposit_outcome(ts, module, target, success,
+#         numpy_emb, [path], msgpack_meta) for the learning signal.
+#         Sidecar holds only substrate-derived state (amplitudes, phases,
+#         graph checkpoint, node_content cache, stats) — never embeddings.
 # [2026-04-06] Claude Code (Opus 4.6) — Initial organism wiring
 #   What: Full substrate loop for HuggingFace NuWave demo
 #   Why:  Baseline proved 47.2% with crude string KISS. Real substrate
@@ -36,6 +55,13 @@ from typing import Any, Dict, List, Optional, Tuple
 
 import numpy as np
 
+# BTF path — msgpack for metadata bytes, ng_tract for tract I/O.
+# Hard requirement: wheel is vendored in the repo root, declared in
+# requirements.txt. If import fails the organism should fail loudly,
+# not silently fall back to inflated JSON.
+import msgpack
+import ng_tract
+
 logger = logging.getLogger("nuwave.organism")
 
 # Substrate path — added temporarily during import, not permanently
@@ -52,14 +78,25 @@ class NuWaveOrganism:
         organism.record_outcome(user_message, response, success=True)
     """
 
-    # Hub persistence — survives container rebuilds, sleeps, everything
+    # Hub persistence — survives container rebuilds, sleeps, everything.
+    # Three artifacts now: sidecar (amplitudes, phases, graph, stats, content
+    # cache) + two tracts (experience.tract, outcomes.tract). Tracts are the
+    # canonical form for embeddings and Law 7 sensory log; sidecar is the
+    # substrate-dynamics state that BTF doesn't carry.
     HUB_REPO = "Executor-Tyrant-Framework/nuwave-substrate-state"
-    HUB_FILENAME = "organism_state.pt"
+    HUB_SIDECAR_FILENAME = "nuwave_sidecar.pt"
+    HUB_EXP_TRACT_FILENAME = "experience.tract"
+    HUB_OUTCOMES_TRACT_FILENAME = "outcomes.tract"
+    # Legacy — read-only, for migration from the pre-BTF state format.
+    HUB_LEGACY_FILENAME = "organism_state.pt"
 
     def __init__(self, state_path: str = "/tmp/nuwave_substrate"):
         self._state_path = state_path
         os.makedirs(state_path, exist_ok=True)
-        self._local_state_path = os.path.join(state_path, "organism_state.pt")
+        self._local_sidecar_path = os.path.join(state_path, self.HUB_SIDECAR_FILENAME)
+        self._local_legacy_path = os.path.join(state_path, self.HUB_LEGACY_FILENAME)
+        self._exp_tract_path = os.path.join(state_path, self.HUB_EXP_TRACT_FILENAME)
+        self._outcomes_tract_path = os.path.join(state_path, self.HUB_OUTCOMES_TRACT_FILENAME)
         self._hf_token = os.environ.get("HF_TOKEN", None)
 
         # Initialize substrate
@@ -67,7 +104,14 @@ class NuWaveOrganism:
         self._embed_fn = None
         self._step_result = None
         self._step_count = 0
-        self._node_content: Dict[str, str] = {}  # node_id → content for Pith
+        self._node_content: Dict[str, str] = {}  # node_id → FULL text (no truncation)
+
+        # Embedding side-table — numpy f32, never a Python list. BTF outcomes
+        # tract is the persisted form; this dict is the hot-path runtime
+        # cache. Nodes in the Graph carry NO embeddings — keeps graph
+        # checkpoints small and eliminates the list↔numpy round-trip that
+        # BTF was built to kill (see Rust Substrate Layer doc §2).
+        self._embeddings: Dict[str, np.ndarray] = {}
 
         # Amplitude field — per-node complex amplitude for interference dynamics
         # amplitude: how strongly this node contributes (grown by use, decayed by time)
@@ -118,37 +162,108 @@ class NuWaveOrganism:
         logger.info("No saved state found — starting fresh organism")
 
     def _try_restore_local(self) -> bool:
-        """Restore from local state file."""
-        if not os.path.exists(self._local_state_path):
-            return False
-        try:
-            import torch
-            state = torch.load(self._local_state_path, map_location="cpu", weights_only=False)
-            return self._apply_state(state, source="local")
-        except Exception as exc:
-            logger.warning("Local restore failed: %s", exc)
-            return False
+        """Restore from local sidecar; fall back to legacy format for migration.
+
+        New format (post-BTF): nuwave_sidecar.pt holds substrate-derived
+        state, no embeddings. Embeddings are restored from outcomes.tract.
+        Legacy format (pre-BTF): organism_state.pt — embeddings as Python
+        lists inside graph checkpoint metadata. One-shot migration at
+        load time: lists are lifted into self._embeddings as numpy.
+        """
+        for path, source in [
+            (self._local_sidecar_path, "local-sidecar"),
+            (self._local_legacy_path, "local-legacy"),
+        ]:
+            if not os.path.exists(path):
+                continue
+            try:
+                import torch
+                state = torch.load(path, map_location="cpu", weights_only=False)
+                if self._apply_state(state, source=source):
+                    # After sidecar/legacy applied, pull embeddings from
+                    # outcomes tract (canonical form). Tract may or may
+                    # not exist yet — this is best-effort.
+                    self._restore_embeddings_from_tract()
+                    return True
+            except Exception as exc:
+                logger.warning("Local restore (%s) failed: %s", source, exc)
+        return False
 
     def _try_restore_hub(self) -> bool:
-        """Pull state from HuggingFace dataset repo."""
+        """Pull sidecar + tracts from HuggingFace dataset repo.
+
+        Tries the new sidecar filename first. Falls back to legacy
+        organism_state.pt if present. Tracts are pulled alongside so
+        the local cache has them for analysis buckets and for embedding
+        restoration.
+        """
         if not self._hf_token:
             return False
+        from huggingface_hub import hf_hub_download
+
+        def _pull(filename: str) -> Optional[str]:
+            try:
+                return hf_hub_download(
+                    self.HUB_REPO, filename,
+                    repo_type="dataset", local_dir=self._state_path,
+                    token=self._hf_token,
+                )
+            except Exception as exc:
+                logger.info("Hub %s absent: %s", filename, exc)
+                return None
+
+        # Pull tracts first — they're the canonical embedding source.
+        _pull(self.HUB_EXP_TRACT_FILENAME)
+        _pull(self.HUB_OUTCOMES_TRACT_FILENAME)
+
+        # Try new-format sidecar, then legacy.
+        for filename, source in [
+            (self.HUB_SIDECAR_FILENAME, "hub-sidecar"),
+            (self.HUB_LEGACY_FILENAME, "hub-legacy"),
+        ]:
+            local = _pull(filename)
+            if not local:
+                continue
+            try:
+                import torch
+                state = torch.load(local, map_location="cpu", weights_only=False)
+                if self._apply_state(state, source=source):
+                    self._restore_embeddings_from_tract()
+                    return True
+            except Exception as exc:
+                logger.warning("Hub apply (%s) failed: %s", source, exc)
+        return False
+
+    def _restore_embeddings_from_tract(self):
+        """Populate self._embeddings from outcomes.tract (canonical form).
+
+        Walks the tract in order; later entries win for repeated node_ids
+        (outcome tract is append-only — a node's latest reward-modulated
+        embedding is the current one). Zero-copy via embedding_as_numpy.
+        """
+        if not os.path.exists(self._outcomes_tract_path):
+            return
         try:
-            from huggingface_hub import hf_hub_download
-            local = hf_hub_download(
-                self.HUB_REPO, self.HUB_FILENAME,
-                repo_type="dataset", local_dir=self._state_path,
-                token=self._hf_token,
-            )
-            import torch
-            state = torch.load(local, map_location="cpu", weights_only=False)
-            return self._apply_state(state, source="hub")
+            with open(self._outcomes_tract_path, 'rb') as f:
+                raw = f.read()
+            count = 0
+            for entry in ng_tract.TractReader(raw):
+                # PyOutcomeEntry has .target_id, .embedding_as_numpy()
+                tid = getattr(entry, 'target_id', None)
+                if not tid:
+                    continue
+                try:
+                    self._embeddings[tid] = entry.embedding_as_numpy()
+                    count += 1
+                except Exception:
+                    continue
+            logger.info("Restored %d embeddings from outcomes.tract (unique=%d)",
+                        count, len(self._embeddings))
         except Exception as exc:
-            logger.info("No hub state: %s", exc)
-            return False
+            logger.warning("Tract embedding restore failed: %s", exc)
 
     def _apply_state(self, state: dict, source: str) -> bool:
-        """Apply a loaded state dict to the organism."""
+        """Apply a loaded state dict. Handles new + legacy formats."""
         try:
             # Graph checkpoint — stored as JSON string
             graph_json = state.get('graph_checkpoint_json')
@@ -159,6 +274,23 @@ class NuWaveOrganism:
                 self._graph.restore(tmp)
                 os.remove(tmp)
 
+                # LEGACY MIGRATION: pre-BTF checkpoints stored embeddings
+                # as Python lists inside node.metadata['embedding']. Lift
+                # them into self._embeddings as numpy f32, strip from
+                # metadata so future checkpoints stay lean.
+                migrated = 0
+                for nid, node in self._graph.nodes.items():
+                    if node.metadata and 'embedding' in node.metadata:
+                        emb = node.metadata.pop('embedding')
+                        if isinstance(emb, list):
+                            self._embeddings[nid] = np.asarray(emb, dtype=np.float32)
+                            migrated += 1
+                        elif isinstance(emb, np.ndarray):
+                            self._embeddings[nid] = emb.astype(np.float32, copy=False)
+                            migrated += 1
+                if migrated:
+                    logger.info("Migrated %d legacy embeddings from graph metadata", migrated)
+
             # Content map
             self._node_content = state.get('node_content', {})
 
@@ -172,13 +304,14 @@ class NuWaveOrganism:
             self._phases = state.get('phases', {})
 
             logger.info(
-                "Organism restored from %s: %d nodes, %d syn, %d content, step %d, %d amplitudes",
+                "Organism restored from %s: %d nodes, %d syn, %d content, step %d, %d amps, %d embs",
                 source,
                 len(self._graph.nodes) if self._graph else 0,
                 len(self._graph.synapses) if self._graph else 0,
                 len(self._node_content),
                 self._step_count,
                 len(self._amplitudes),
+                len(self._embeddings),
             )
             return True
         except Exception as exc:
@@ -265,24 +398,46 @@ class NuWaveOrganism:
             return None
 
         try:
-            embedding = self._embed_fn(text)
+            # Law 7: raw bytes go to the experience tract BEFORE any
+            # processing. Full text, no truncation, source-annotated.
+            # The substrate learns via STDP co-firing, not via curated fields.
+            try:
+                ng_tract.deposit_experience(
+                    text, "user", self._exp_tract_path, "text"
+                )
+            except Exception as exc:
+                logger.debug("Experience tract write failed: %s", exc)
+
+            embedding = np.asarray(self._embed_fn(text), dtype=np.float32)
             node_id = f"exp_{self._step_count}_{hash(text) & 0xFFFF:04x}"
 
-            # Create node in the SNN
+            # Create node in the SNN. Metadata carries biological timestamp
+            # only — no "type" label, no "content" curation. The retina
+            # does not classify photons; V1 discovers features via dynamics.
             node = self._graph.create_node(
                 node_id=node_id,
-                metadata={
-                    "content": text[:200],
-                    "step": self._step_count,
-                    "type": "experience",
-                },
+                metadata={"step": self._step_count},
             )
 
-            # Store embedding in metadata for Pith similarity search
-            node.metadata['embedding'] = embedding.tolist()
+            # Embedding lives in the side-table as numpy f32 (zero inflation).
+            # Outcomes tract is the persisted form; this dict is runtime cache.
+            self._embeddings[node_id] = embedding
+
+            # Full-text content cache — Pith trims at extraction, not here.
+            self._node_content[node_id] = text
 
-            # Track for Pith retrieval
-            self._node_content[node_id] = text[:200]
+            # Neuromodulatory signal to outcomes tract. success=True is
+            # the dopamine polarity for sensory deposits (reward the act
+            # of observing); the actual learning signal from a full
+            # interaction flows through record_outcome() after response.
+            try:
+                ng_tract.deposit_outcome(
+                    time.time(), "nuwave", node_id, True, embedding,
+                    [self._outcomes_tract_path],
+                    msgpack.packb({"step": self._step_count, "source": "user"}),
+                )
+            except Exception as exc:
+                logger.debug("Outcomes tract write failed: %s", exc)
 
             # Initialize amplitude for new node — starts at 1.0 (just deposited)
             self._amplitudes[node_id] = 1.0
@@ -300,11 +455,11 @@ class NuWaveOrganism:
             for existing_id, existing_node in self._graph.nodes.items():
                 if existing_id == node_id:
                     continue
-                existing_emb = existing_node.metadata.get('embedding') if existing_node.metadata else None
+                # Embeddings live in the side-table now, not node.metadata.
+                existing_emb = self._embeddings.get(existing_id)
                 if existing_emb is None:
                     continue
 
-                existing_emb = np.array(existing_emb, dtype=np.float32)
                 sim = float(np.dot(embedding, existing_emb) /
                             (np.linalg.norm(embedding) * np.linalg.norm(existing_emb) + 1e-8))
 
@@ -482,7 +637,7 @@ class NuWaveOrganism:
 
         try:
             import math
-            query_embedding = self._embed_fn(query)
+            query_embedding = np.asarray(self._embed_fn(query), dtype=np.float32)
 
             # Compute per-node Born rule score.
             # The query interferes with each node's amplitude:
@@ -490,12 +645,14 @@ class NuWaveOrganism:
             # existing amplitude is the base strength.
             scored = []
 
-            for nid, node in self._graph.nodes.items():
-                emb = node.metadata.get('embedding') if node.metadata else None
-                if emb is None:
+            # Iterate the side-table, not graph.nodes — embeddings no
+            # longer live in node.metadata. Nodes without embeddings
+            # (e.g. legacy graph entries) are skipped naturally.
+            for nid in self._graph.nodes.keys():
+                node_emb = self._embeddings.get(nid)
+                if node_emb is None:
                     continue
 
-                node_emb = np.array(emb, dtype=np.float32)
                 sim = float(np.dot(query_embedding, node_emb) /
                             (np.linalg.norm(query_embedding) * np.linalg.norm(node_emb) + 1e-8))
 
@@ -568,23 +725,45 @@ class NuWaveOrganism:
             return
 
         try:
+            # Raw response bytes → experience tract (Law 7). Source tag
+            # distinguishes origin ("nuwave_response" vs "user") — that's
+            # origin annotation, not classification. The causal link to
+            # the query is derivable from timestamp adjacency in the
+            # tract; no need to curate a "query" field at deposit time.
+            try:
+                ng_tract.deposit_experience(
+                    response, "nuwave_response", self._exp_tract_path, "text"
+                )
+            except Exception as exc:
+                logger.debug("Experience tract write failed: %s", exc)
+
             # Response as raw experience — create node, stimulate, reward
-            response_embedding = self._embed_fn(response)
+            response_embedding = np.asarray(self._embed_fn(response), dtype=np.float32)
             node_id = f"resp_{self._step_count}_{hash(response) & 0xFFFF:04x}"
 
+            # Metadata carries biological timestamp only — no type label,
+            # no query field, no truncated content. The substrate discovers
+            # response-vs-experience distinction via STDP co-firing.
             node = self._graph.create_node(
                 node_id=node_id,
-                metadata={
-                    "content": response[:200],
-                    "type": "response",
-                    "step": self._step_count,
-                    "query": user_message[:100],
-                },
+                metadata={"step": self._step_count},
             )
-            node.metadata['embedding'] = response_embedding.tolist()
+            self._embeddings[node_id] = response_embedding
 
             self._graph.stimulate(node_id, current=1.0)
-            self._node_content[node_id] = response[:200]
+            self._node_content[node_id] = response
+
+            # Neuromodulatory deposit — the reward polarity is THE
+            # learning signal. This is biology (dopamine modulates STDP),
+            # not preclassification. Real outcome bool reaches the tract.
+            try:
+                ng_tract.deposit_outcome(
+                    time.time(), "nuwave", node_id, success, response_embedding,
+                    [self._outcomes_tract_path],
+                    msgpack.packb({"step": self._step_count, "source": "nuwave_response"}),
+                )
+            except Exception as exc:
+                logger.debug("Outcomes tract write failed: %s", exc)
 
             # Reward signal — the outcome closes the loop
             # Strong reward strengthens synapses between co-firing nodes via STDP.
@@ -605,14 +784,20 @@ class NuWaveOrganism:
         return dict(self.stats)
 
     def save(self):
-        """Persist all state — single file, local + hub.
+        """Persist all state — sidecar + tracts, local + hub.
+
+        Sidecar holds substrate-dynamics state that BTF doesn't carry:
+        graph checkpoint (synapses/hyperedges/predictions), amplitudes,
+        phases, node_content cache, stats. NO embeddings — those live
+        in outcomes.tract as numpy f32.
 
-        One consolidated state dict. Saved locally for warm restart.
-        Pushed to HF hub for survival across rebuilds.
+        Tracts are append-only — written as deposits happen, not here.
+        This method only needs to save the sidecar and push everything
+        to the hub.
         """
         import torch
 
-        # Build consolidated state
+        # Build sidecar state — no embeddings.
         state = {
             'node_content': self._node_content,
             'stats': dict(self.stats),
@@ -620,9 +805,14 @@ class NuWaveOrganism:
             'phases': self._phases,
         }
 
-        # Graph checkpoint — read as raw JSON string
+        # Graph checkpoint — read as raw JSON string. If any legacy
+        # embeddings slipped into node.metadata, strip them here so
+        # future checkpoints stay lean.
         if self._graph:
             try:
+                for node in self._graph.nodes.values():
+                    if node.metadata and 'embedding' in node.metadata:
+                        del node.metadata['embedding']
                 tmp = os.path.join(self._state_path, "_tmp_save.ckpt")
                 self._graph.checkpoint(tmp)
                 with open(tmp, 'r') as f:
@@ -631,32 +821,43 @@ class NuWaveOrganism:
             except Exception as exc:
                 logger.debug("Graph checkpoint failed: %s", exc)
 
-        # Save locally
+        # Save sidecar locally
         try:
-            torch.save(state, self._local_state_path)
+            torch.save(state, self._local_sidecar_path)
         except Exception as exc:
-            logger.debug("Local save failed: %s", exc)
+            logger.debug("Local sidecar save failed: %s", exc)
 
-        # Push to hub — survives everything
+        # Push sidecar + tracts to hub
         self._push_to_hub()
 
     def _push_to_hub(self):
-        """Push local state file to HF dataset repo."""
-        if not self._hf_token or not os.path.exists(self._local_state_path):
+        """Push sidecar + both tracts to HF dataset repo."""
+        if not self._hf_token:
             return
         try:
             from huggingface_hub import HfApi, create_repo
-            # Ensure repo exists
             create_repo(self.HUB_REPO, repo_type="dataset", exist_ok=True,
                        private=True, token=self._hf_token)
             api = HfApi()
-            api.upload_file(
-                path_or_fileobj=self._local_state_path,
-                path_in_repo=self.HUB_FILENAME,
-                repo_id=self.HUB_REPO,
-                repo_type="dataset",
-                token=self._hf_token,
-            )
+
+            for local_path, hub_name in [
+                (self._local_sidecar_path, self.HUB_SIDECAR_FILENAME),
+                (self._exp_tract_path, self.HUB_EXP_TRACT_FILENAME),
+                (self._outcomes_tract_path, self.HUB_OUTCOMES_TRACT_FILENAME),
+            ]:
+                if not os.path.exists(local_path):
+                    continue
+                try:
+                    api.upload_file(
+                        path_or_fileobj=local_path,
+                        path_in_repo=hub_name,
+                        repo_id=self.HUB_REPO,
+                        repo_type="dataset",
+                        token=self._hf_token,
+                    )
+                except Exception as exc:
+                    logger.debug("Hub push of %s failed (non-fatal): %s", hub_name, exc)
+
             logger.info("State pushed to hub (%s)", self.HUB_REPO)
         except Exception as exc:
             logger.debug("Hub push failed (non-fatal): %s", exc)

requirements.txt

@@ -3,3 +3,5 @@ torch>=2.0.0
 accelerate>=0.27.0
 onnxruntime>=1.16.0
 tokenizers>=0.15.0
+msgpack>=1.0.0
+./ng_tract-0.1.0-cp312-abi3-manylinux_2_34_x86_64.whl