Recode Huffman updates and derive provisional seeded codebook

docs/ASSET_PIPELINE_TODO.md

@@ -222,3 +222,5 @@ Encoder follow-up: [supplied-tree symbol streams](research/HUFFMAN_ENCODING.md)
 Initialization follow-up: [native frequency tables](research/HUFFMAN_INITIALIZATION.md) preserves exact codec seeds and initial layouts. Recover adaptive training and actual codewords before real network/map/renderer integration; supplied-tree round trips alone remain insufficient (RE-356).
 
 Adaptive-codec follow-up: [update compositions](research/HUFFMAN_UPDATES.md) validates sampled supplied-tree insertion/count/list/block behavior. Actual seed-trained codebooks, real packets and runtime/platform validation remain required before network/map/renderer integration (RE-360).
+
+Codebook follow-up: [provisional modeled codewords](research/HUFFMAN_CODEBOOK.md) adds exact native seed training in a differentially checked recoding. Full native/runtime codeword and message/packet validation remains required before network/map/renderer integration (RE-364).

docs/FINDINGS.md

@@ -396,3 +396,9 @@ Bounded qagame paths recover bomb-plant/inactivity correction requests, direct f
 **F-128 — Observed adaptive count compositions:** repeated/new sequences retain occurrence-count leaf weights, root total and child-weight sums across recursive increment and reorder instructions. Null updates preserve memory; sampled recursion reaches depth eight. Real seeded training/codewords and complete runtime/ABI/platform behavior remain open. [UPDATE-002](research/HUFFMAN_UPDATES.md).
 
 **F-129 — Observed rank/block reuse:** sampled rank lists cover all nodes with nondecreasing weights and reverse-link consistency; weight blocks share last-rank leader cells. Allocated cells partition into live heads and an acyclic free chain, with fifteen cells in the full-alphabet fixture. Complete swaps, second-object behavior and capacities remain open. [UPDATE-003](research/HUFFMAN_UPDATES.md).
+
+**F-130 — Observed sampled differential equivalence:** compact adaptive recoding matches complete prior and fresh native-instruction tree/list/head/freelist states, exercising tie and swap event classes. This supports the sampled valid-tree contract, not branch-complete/native-runtime conformance. [CODE-001](research/HUFFMAN_CODEBOOK.md).
+
+**F-131 — Observed paired-object update samples:** first/second initialized halves match their respective recoding states after every fresh update, with the +24 pointer-slot asymmetry retained. Full initialization/training lifetime and runtime/platform equivalence remain open. [CODE-002](research/HUFFMAN_CODEBOOK.md).
+
+**F-132 — Model-derived seeded codebook:** exact native counts produce a provisional 513-node tree, 2–11-bit prefix-free codewords and compatible bounded native-helper symbol round trips against supplied modeled memory. Full native seeded training/codeword/packet conformance remains unverified. [CODE-003](research/HUFFMAN_CODEBOOK.md).

docs/PROVISIONAL_TODO.md

@@ -534,3 +534,5 @@ Research progress: [bounded Huffman encoding](research/HUFFMAN_ENCODING.md) comp
 Research progress: [Huffman initialization](research/HUFFMAN_INITIALIZATION.md) completes bounded RE-353–355. Broad network/runtime milestones remain open; RE-356 tracks adaptive updates, initialized codewords and actual message/packet/platform fixtures.
 
 Research progress: [adaptive Huffman updates](research/HUFFMAN_UPDATES.md) completes bounded RE-357–359. Broad network/runtime milestones remain open; RE-360 tracks complete swap/tie paths, second-object equivalence, actual trained codebooks and real packet/platform fixtures.
+
+Research progress: [differential Huffman codebook](research/HUFFMAN_CODEBOOK.md) completes bounded RE-361–363. Broad network/runtime milestones remain open; RE-364 tracks full native-training/codeword conformance and complete real packet/platform contracts.

docs/RESEARCH_LEDGER.md

@@ -221,3 +221,7 @@ Completed bounded RE-353–355 with `analyze_huffman_initialization.py`: eight v
 ## 2026-10-11 — Adaptive Huffman insertion and recursive updates
 
 Completed bounded RE-357–359 with `analyze_huffman_updates.py`: four verified graphs across the Linux ELF64 engines, 22 adaptive sequences, twelve low-byte existing-entry dispatch probes and two null-increment probes. UPDATE-001–003/F-127–129 distinguish byte normalization from transmit's signed lookup, new leaf/internal insertion from reuse, sampled Counter/subtree/rank/head/freelist invariants from exhaustive tie/swap coverage, and actual source-matched initial bytes from full game-seed training. Composed sequences include every byte once, yielding 513 reachable nodes and fifteen allocated pointer cells; recursion reaches depth eight. Prefix states are retained for sequences up to 24 steps, with complete final states for longer cases. increment PUSH omission, stopped POP/RET/CALL boundaries and explicit recursive caller-register restoration are documented; no original process or native callee executes. Eight independent tests verify allocation/lookup/counts, tree/parent/subtree invariants, rank/reverse links, shared leaders, live/free cell partition/reuse, aliases/dispatch, null/recursive traces and paired results. All 326 tests pass; new and preceding reports regenerate byte-identically, and storage/link/whitespace checks pass. RE-356 remains open; RE-360 tracks branch-complete tie/swaps, second-object equivalence, malformed/alias/capacity/wrap states, efficient exact seed-trained codebook reconstruction and actual message/packet/runtime/all-platform contracts.
+
+## 2026-10-11 — Differential Huffman recoding and provisional seeded codebook
+
+Completed bounded RE-361–363 with `huffman_model.py` and `analyze_huffman_codebook.py`: compact observed update recoding, complete prior-state comparisons (152 retained prefixes plus 22 final cases), sixteen fresh first/second-object cases (288 native-prefix comparisons), exact native-frequency model training and eight native-helper symbol round trips against the modeled tree. CODE-001–003/F-130–132 distinguish sampled state equivalence from branch-complete/runtime equivalence, retained second-object slot asymmetry from normalization, and MODEL-DERIVED seeded trees/codewords from full native training or runtime captures. The 1,053,340-operation model schedule yields 513 nodes, 493 allocated cells (489 live/four free), 257 prefix-free leaf codes with lengths 2–11, and explicit state/codebook/source/report hashes. All 256 model symbol checkpoints validate tree/count/rank/leader/pool invariants. Native transmit/send/receive instruction compositions agree with independent modeled-codeword packing, including every byte value; no original process or native callee executes. Eight tests verify prior/fresh differentials, sampled event classes, modeled training counts/checkpoints, independent tree/codeword/Kraft checks and native-helper bytes/symbol/cursor/canary/paired results. All 334 tests pass; new and preceding reports regenerate byte-identically, and storage/link/whitespace checks pass. RE-360 remains open; RE-364 tracks full native-seed/codeword validation, authoritative/runtime comparison, complete tie/swap/alias/capacity/wrap/platform coverage and actual message/packet/snapshot/renderer contracts.

docs/REVERSE_ENGINEERING_TODO.md

@@ -432,3 +432,8 @@ Purpose: turn the supplied installation into an evidence-backed specification of
 - [x] RE-358 Compose repeated/new symbol and recursive increment sequences with independent count/tree/subtree-weight invariants ([UPDATE-002](research/HUFFMAN_UPDATES.md)).
 - [x] RE-359 Recover sampled rank links, shared block leaders and live/free pointer-cell partition/reuse ([UPDATE-003](research/HUFFMAN_UPDATES.md)).
 - [ ] RE-360 Recover branch-complete swap/tie/parent/leader behavior and second-object update equivalence; validate malformed/alias/capacity/wrap states and legal callers, efficiently reconstruct actual seed-trained codebooks with equivalence checks, and compose real message/packet/original-runtime/all-platform fixtures.
+
+- [x] RE-361 Recode observed adaptive updates compactly and compare complete prior/fresh tie-heavy native-instruction states ([CODE-001](research/HUFFMAN_CODEBOOK.md)).
+- [x] RE-362 Compare both initialized object halves after every sampled update, retaining slot asymmetry and relocated pointers ([CODE-002](research/HUFFMAN_CODEBOOK.md)).
+- [x] RE-363 Train the provisional model with exact native seeds, export/check tree and prefix-free codewords, and compose native-helper symbol round trips against that supplied tree ([CODE-003](research/HUFFMAN_CODEBOOK.md)).
+- [ ] RE-364 Validate complete native-seed training/codewords against an independent authoritative implementation or permitted original-runtime captures; prove remaining swap/tie/alias/error/capacity/wrap/platform paths and compose full message/real-packet/overflow/truncation/snapshot/renderer contracts.

docs/research/HUFFMAN_CODEBOOK.md

@@ -0,0 +1,39 @@
+# Differential Huffman recoding and provisional seeded codebook
+
+Reproduce with `python3 scripts/analyze_huffman_codebook.py`; validate with `python3 scripts/test_huffman_codebook.py`. [huffman-codebook.json](huffman-codebook.json) retains source/disassembly/report/model hashes, native graphs, differential checks, modeled training checkpoints, tree/codewords and bounded native-instruction symbol round trips. The compact recoding is [huffman_model.py](../../scripts/huffman_model.py).
+
+The codebook is **model-derived**, not runtime captured or produced by evaluating every native training instruction. Differential checks compare complete sampled states against synthetic native instruction compositions, with the same stopped-call/frame/register-restoration limitations as [adaptive updates](HUFFMAN_UPDATES.md). Original processes and native callees never execute. Full branch completeness, native runtime/ABI, other platforms and complete message/packet conformance remain open.
+
+## CODE-001 — Compact update recoding and tie-heavy differentials (RE-361)
+
+The provisional model represents nodes, shared leader cells and freelist links directly. It recodes the observed insertion/increment behavior: byte-normalized lookup, NYT insertion, equal-weight next-node leader selection, tree swaps unless the leader is the parent, rank-list swaps and adjacency repair, previous-node leader retention or cell freeing, new weight-block sharing/allocation, recursive parent updates and post-parent adjacency/head repair.
+
+Complete model states match all 22 previously recorded native-instruction sequence fixtures, including 152 retained prefix comparisons across both engines and their final states. Four additional sequence patterns exercise alternation, ascending sixteen bytes, repeated boundary-byte mixtures and ascending/reversed eight-byte orders. Each is compared after every update for both initialized objects and both engines: sixteen fresh cases, 288 native-prefix comparisons. Comparisons include all nodes, tree/list/lookup pointers, weights, shared heads, every allocated pointer cell and freelist, rather than just counts or encoded output.
+
+Model event counts record new/existing entries, equal-next ties, tree/list swaps, leader-is-parent skips, old-head previous reuse, cell release/reuse, new-head sharing and post-parent adjacency/head changes. The matched sample corpus exercises each of these event classes. These counters belong to the recoding; they are not a branch-complete native coverage report or proof that every malformed/alias/capacity/tie path is equivalent. The model intentionally covers supplied valid byte trees, not general corrupted-object behavior.
+
+## CODE-002 — Both initialized objects (RE-362)
+
+Fresh compositions reconstruct the source-matched sparse seed bytes of both object halves. The second starts at first +0xc830 and retains its initially nonzero pointer slot +24. The model retains this slot too, so absolute state comparisons preserve the initializer asymmetry.
+
+Within the fresh corpus, the first and second halves generate matching normalized tree/count/rank/cell invariants and model event counts. Every prefix matches its respective native-instruction state with relocated addresses and its own tail-slot value. Outside object canaries remain intact. This establishes sampled update equivalence despite the retained slot difference, not full paired initialization lifetime, million-step native training, concurrent use, or runtime equivalence.
+
+## CODE-003 — Exact seed schedule and provisional codewords (RE-363)
+
+The collector re-extracts the 256 frequency words directly from each verified ELF image and checks them against the [frequency report](HUFFMAN_INITIALIZATION.md). It trains the compact first-object model in the exact ascending-symbol repetition order, totaling 1,053,340 add operations. At every completed symbol it records a complete-state hash and validates independent leaf counts, parent/child reachability, internal subtree weights, rank ordering, leader targets and live/free cell partition.
+
+The resulting modeled tree has 513 nodes and leaves for symbols 0–256, including NYT 256 with weight zero. Root weight is 1,053,340. It allocates 493 pointer cells, with 489 live and four free at the final checkpoint. These are model outputs, not measurements of the game's running allocator. The exported codebook has lengths from two through eleven bits and SHA-256:
+
+`a5a945498ab44f54d98b0b75346e03099e3adb212278f63e170b1a2eb57438c3`
+
+That digest uses sorted compact JSON of symbol/bits/length records; it is not the frequency-table hash or a native packet checksum. Bits are listed in root-to-leaf transmission order. The codebook is prefix-free and its full binary leaf set satisfies exact Kraft equality. Tests independently derive every code from the exported tree and verify leaf weights against the actual native frequency table.
+
+Eight paired native-instruction encoder/decoder compositions then use this modeled tree as supplied memory. Cases are all 256 bytes, ascending first sixteen at bit seven, and repeated boundary-byte mixtures at bits zero/seven. Transmit/send/receive instructions produce matching symbols, variable codeword lengths/cursors and independent LSB-packed bytes with buffer canaries intact. The buffer is explicitly oversized; no capacity/truncation claim is made. This checks those helpers against the supplied modeled tree, not the accuracy of native full training or complete MSG_WriteBits/MSG_ReadBits/network packets.
+
+The exported tree and codebook are research artifacts for subsequent comparison. Do not treat their provisional hash as a compatibility guarantee or replace the original frequency-training contract without validation.
+
+## Validation and next work
+
+Eight tests check prior complete-state differential equivalence, fresh first/second-object prefixes and invariants, sampled model event classes, full seed leaf/subtree/reachability/count checks, checkpoint/cell summaries, independent tree-derived prefix-free codewords/Kraft equality, native-helper symbol packing/cursors/canaries and paired helper outputs. No shared evaluator extension was needed.
+
+RE-364: prove individual native swap/tie/alias/error/capacity/wrap paths, validate full native-seed training/codewords against an independent authoritative implementation or permitted original-runtime captures, establish all-platform equivalence, and compose complete message writers/readers with actual packets, overflow/truncation and snapshot/renderer integration.

docs/research/HUFFMAN_UPDATES.md

@@ -35,3 +35,5 @@ The object-wide outside canaries are preserved for all sequences. This does not
 Eight independent tests verify node allocation/lookup/Counter counts, complete tree reachability/parent/subtree weights, rank ordering/reverse links, leader cells, live/free pool partition and reuse, dispatch/low-byte aliases, null/recursive stopped-call behavior and paired results. A fresh isolated root increment additionally verifies its weight change and call-free evaluated trace. No evaluator extension was needed.
 
 RE-360: audit individual tree/list swaps and parent/leader tie gates with branch-complete fixtures, second-object initialization/update equivalence, malformed/aliased/capacity/wrap states and legal callers; reconstruct actual initialized codebooks efficiently using the exact seed schedule with native-equivalence checks; compose message encoders/decoders and real packets and validate original-runtime/all-platform contracts.
+
+Codebook follow-up: [differential recoding and modeled training](HUFFMAN_CODEBOOK.md) checks both object halves and exports provisional seed-trained codewords (RE-361–363). Actual full native training/runtime/platform/packet validation remains open.

docs/research/README.md

@@ -135,3 +135,5 @@ The extension collector adds five focused tests for native tail calls, command-r
 [Huffman frequencies and initialization](HUFFMAN_INITIALIZATION.md) adds RE-353–355: verified native frequencies, empty-tree layout and bounded paired seed requests. Reproduce with `python3 scripts/analyze_huffman_initialization.py`; check with `python3 scripts/test_huffman_initialization.py`. Adaptive updates, trained codewords and actual message/runtime/platform contracts remain open.
 
 [Adaptive Huffman update compositions](HUFFMAN_UPDATES.md) adds RE-357–359: insertion/dispatch, recursive count/tree invariants and rank/block/freelist reuse. Reproduce with `python3 scripts/analyze_huffman_updates.py`; check with `python3 scripts/test_huffman_updates.py`. Full branch/tie/second-object/seeded-codebook/message/runtime/platform contracts remain open.
+
+[Differential recoding and provisional Huffman codebook](HUFFMAN_CODEBOOK.md) adds RE-361–363: full sampled state comparisons, both object halves and modeled seed training/codewords with native-helper symbol compositions. Reproduce with `python3 scripts/analyze_huffman_codebook.py`; check with `python3 scripts/test_huffman_codebook.py`. Native full-training, runtime/platform and complete packet compatibility remain open.

scripts/analyze_huffman_codebook.py

@@ -0,0 +1,117 @@
+#!/usr/bin/env python3
+"""Native-fixture differential recoding, second-object checks and modeled seed codebook."""
+from collections import Counter
+import gzip
+import hashlib
+import json
+from pathlib import Path
+import struct
+from analyze_collision_composition import get_word,put_word
+from analyze_engine_control import control_flow
+from analyze_frame_timing import read_address
+from analyze_huffman_updates import increment,run,state
+from analyze_native import elf
+from analyze_qvm import write_json
+from analyze_vm_extensions import native_functions,verified
+from huffman_model import Huffman
+from analyze_huffman_encoding import transmit,DATA,CURSOR,TABLE,OUT
+
+
+def digest(value):return hashlib.sha256(json.dumps(value,sort_keys=True,separators=(',',':')).encode()).hexdigest()
+
+
+def invariants(h,counts):
+    s=h.state();nodes={x['address']:x for x in s['nodes']};seen=set()
+    def walk(addr,parent):
+        if addr in seen:raise ValueError('tree cycle')
+        seen.add(addr);n=nodes[addr]
+        if n['parent']!=parent:raise ValueError('parent mismatch')
+        if n['symbol']!=257:
+            expected=counts.get(n['symbol'],0)
+            if n['weight']!=expected or n['left'] or n['right']:raise ValueError('leaf mismatch')
+            return expected
+        weight=walk(n['left'],addr)+walk(n['right'],addr)
+        if n['weight']!=weight:raise ValueError('subtree mismatch')
+        return weight
+    total=walk(s['root'],0)
+    if total!=sum(counts.values()) or seen!=set(nodes):raise ValueError('tree coverage')
+    chain=[];cur=s['list_start'];prev=0
+    while cur:
+        if cur in chain or nodes[cur]['previous']!=prev:raise ValueError('rank links')
+        chain.append(cur);prev=cur;cur=nodes[cur]['next']
+    if set(chain)!=seen or [nodes[a]['weight'] for a in chain]!=sorted(nodes[a]['weight'] for a in chain):raise ValueError('rank order')
+    heads=dict(s['heads']);pool={0x10000+0xb030+8*i:v for i,v in enumerate(s['pointer_cells'])};free=set();cur=s['free_head']
+    while cur:
+        if cur in free or cur not in pool:raise ValueError('free chain')
+        free.add(cur);cur=pool[cur]
+    if set(heads)&free or set(heads)|free!=set(pool):raise ValueError('pool partition')
+    for addr in chain:
+        node=nodes[addr]
+        if node['weight'] and heads[node['head_cell']]!=next(a for a in reversed(chain) if nodes[a]['weight']==node['weight']):raise ValueError('block leader')
+    return {'total_weight':total,'nodes':len(nodes),'rank_nodes':len(chain),'live_cells':len(heads),'free_cells':len(free),'allocated_cells':len(pool)}
+
+
+def main():
+    init_path=Path('docs/research/huffman-initialization.json');updates_path=Path('docs/research/huffman-updates.json');init=json.loads(init_path.read_text());updates=json.loads(updates_path.read_text());out={'native':{},'limitations':'Compact recoding is provisional. Differential checks compare complete sampled states to synthetic native instruction compositions, not original executions. Full seed training/codebook is MODEL-DERIVED, not an evaluated million-step native trace or runtime capture. No packet codec/original process/native callee executes. Branch completeness, actual codewords/runtime/ABI and other-platform conformance remain open.'}
+    fixtures=[[0,1]*8,list(range(16)),[255,0,128,127]*6,list(range(8))+list(reversed(range(8)))]
+    frequency=None;native_graphs={}
+    for label in ['linux--Quake3-UrT-Ded.x86_64','linux--Quake3-UrT.x86_64']:
+        r=json.loads(Path(f'docs/research/native/{label}.json').read_text());data=verified((Path('.cache/UrbanTerror43')/r['source']['path']).read_bytes(),r['source']['sha256']);meta,_=elf(data);a=r['artifacts']['disassembly.txt'];fs=native_functions(verified(gzip.open(a['path'],'rb').read(),a['uncompressed_sha256']).decode());graphs={n:control_flow(fs[n]) for n in ['Huff_addRef','increment','send','Huff_offsetTransmit','Huff_offsetReceive']};native_graphs[label]=graphs;p=init['native'][label];u=updates['native'][label]
+        if p['source']!=r['source'] or u['source']!=r['source'] or p['disassembly_sha256']!=a['uncompressed_sha256'] or u['disassembly_sha256']!=a['uncompressed_sha256']:raise ValueError('native report provenance')
+        raw,_=read_address(data,meta['sections'],p['frequency_table']['address'],1024);counts=list(struct.unpack('<256I',raw))
+        if counts!=p['frequency_table']['counts'] or hashlib.sha256(raw).hexdigest()!=p['frequency_table']['sha256']:raise ValueError('frequency mismatch')
+        if frequency is not None and frequency!=counts:raise ValueError('paired seeds differ')
+        frequency=counts;prior=[]
+        for case in u['sequences']:
+            h=Huffman();checked=0
+            for i,symbol in enumerate(case['symbols']):
+                h.add(symbol)
+                if case['steps']:
+                    if h.state()!=case['steps'][i]:raise ValueError('prior prefix mismatch')
+                    checked+=1
+            if h.state()!=case['final_state']:raise ValueError('prior final mismatch')
+            prior.append({'symbols':case['symbols'],'checked_prefixes':checked,'final_state_sha256':digest(h.state()),'events':h.events,'invariants':invariants(h,Counter(case['symbols']))})
+        differential=[]
+        for second in [False,True]:
+            base=p['initialization_probes'][0]['base']+int(second)*0xc830;seed=p['initialization_probes'][0]['objects'][int(second)]
+            for sequence in fixtures:
+                m={base+i:0 for i in range(0xc830)}
+                for offset,value in seed['nonzero_bytes']:m[base+offset]=value
+                m[base-1]=m[base+0xc830]=0x7b;h=Huffman(second);trace=[];checks=[]
+                for i,symbol in enumerate(sequence):
+                    g=graphs['Huff_addRef'];t=run(g,g['instructions'][0]['address'],[graphs['increment']['instructions'][0]['address']],m,{'%rdi':base,'%rsi':symbol});trace+=t['trace'];c=increment(graphs['increment'],t['memory'],base,t['registers']['rsi']);trace+=c['trace'];m=c['memory'];h.add(symbol);actual=state(m,base)
+                    if actual!=h.state(base):raise ValueError('fresh differential mismatch')
+                    checks.append(digest(actual))
+                differential.append({'second_object':second,'base':base,'symbols':sequence,'matched_prefix_sha256':checks,'events':h.events,'invariants':invariants(h,Counter(sequence)),'canaries':[m[base-1],m[base+0xc830]],'final_state_sha256':digest(h.state(base)),'trace_length':len(trace),'trace_sha256':digest(trace)})
+        out['native'][label]={'source':r['source'],'disassembly_sha256':a['uncompressed_sha256'],'graphs':graphs,'prior_cases':prior,'fresh_differential_cases':differential}
+    h=Huffman();checkpoints=[];trained={}
+    for symbol,count in enumerate(frequency):
+        for _ in range(count):h.add(symbol)
+        trained[symbol]=count;checkpoints.append({'through_symbol':symbol,'repetitions':sum(trained.values()),'state_sha256':digest(h.state()),'invariants':invariants(h,trained)})
+    words=h.codewords();codebook=[{'symbol':s,'bits':bits,'length':len(bits)} for s,bits in sorted(words.items())]
+    out['model_derived_seed_training']={'frequencies':frequency,'repetitions':sum(frequency),'checkpoints':checkpoints,'final_state':h.state(),'final_state_sha256':digest(h.state()),'codebook':codebook,'codebook_sha256':digest(codebook),'events':h.events,'invariants':invariants(h,dict(enumerate(frequency))),'origin':'Compact native-derived model; not runtime captured or full native-instruction training.'}
+    roundtrips=[]
+    for label,graphs in native_graphs.items():
+        for sequence,start in [(list(range(256)),0),(list(range(16)),7),([0,255,128,1]*8,0),([0,255,128,1]*8,7)]:
+            st=h.state(TABLE);m={TABLE+i:0 for i in range(0xc830)}
+            for slot,value in [(0,st['node_count']),(4,st['pointer_cell_count'])]:put_word(m,TABLE+slot,value)
+            for slot,value in [(8,st['root']),(16,st['list_start']),(24,st['tail_slot']),(0x828,st['free_head'])]:put_word(m,TABLE+slot,value,8)
+            for sym,ptr in st['lookup']:put_word(m,TABLE+32+8*sym,ptr,8)
+            for i,value in enumerate(st['pointer_cells']):put_word(m,TABLE+0xb030+8*i,value,8)
+            for node in st['nodes']:
+                for slot,key in [(0,'left'),(8,'right'),(16,'parent'),(24,'next'),(32,'previous'),(40,'head_cell')]:put_word(m,node['address']+slot,node[key],8)
+                put_word(m,node['address']+48,node['weight']);put_word(m,node['address']+52,node['symbol'])
+            for i in range(4096):m[DATA+i]=0
+            m[DATA-1]=m[DATA+4096]=0x7b;put_word(m,CURSOR,start);trace=[];end=start
+            for symbol in sequence:m,part,_=transmit(graphs,m,symbol,None);trace+=part
+            end=get_word(m,CURSOR);encoded=[m[DATA+i] for i in range((end+7)//8)];put_word(m,CURSOR,start);decoded=[];readcursors=[]
+            g=graphs['Huff_offsetReceive'];stops=[x['address'] for x in g['instructions'] if x['opcode']=='retq']
+            for _ in sequence:
+                t=run(g,g['instructions'][0]['address'],stops,m,{'%rdi':st['root'],'%rsi':OUT,'%rdx':DATA,'%rcx':CURSOR});m=t['memory'];trace+=t['trace'];decoded.append(get_word(m,OUT));readcursors.append(get_word(m,CURSOR))
+            roundtrips.append({'native_label':label,'symbols':sequence,'initial_bit':start,'encoded_bytes':encoded,'transmit_cursor':end,'decoded_symbols':decoded,'receive_cursors':readcursors,'canaries':[m[DATA-1],m[DATA+4096]],'trace_length':len(trace),'trace_sha256':digest(trace),'tree_origin':'model-derived seed-trained tree, supplied to native instruction compositions'})
+    out['modeled_tree_native_symbol_roundtrips']=roundtrips
+    out['model_source_sha256']=hashlib.sha256(Path('scripts/huffman_model.py').read_bytes()).hexdigest()
+    out['input_report_sha256']={str(path):hashlib.sha256(path.read_bytes()).hexdigest() for path in [init_path,updates_path]};write_json(Path('docs/research/huffman-codebook.json'),out);print('Verified prior/fresh differential states and generated provisional seed-trained model codebook')
+
+
+if __name__=='__main__':main()

scripts/huffman_model.py

@@ -0,0 +1,85 @@
+#!/usr/bin/env python3
+"""Provisional node-level recoding of observed Linux Huff_addRef/increment semantics."""
+class Node:
+    __slots__=('index','left','right','parent','next','previous','head','weight','symbol')
+    def __init__(self,index,symbol=0,weight=0):
+        self.index=index;self.symbol=symbol;self.weight=weight
+        self.left=self.right=self.parent=self.next=self.previous=self.head=None
+
+
+class Huffman:
+    def __init__(self,second=False):
+        self.nodes=[Node(0,256)];self.root=self.nyt=self.nodes[0];self.lookup={256:self.nyt};self.cells=[];self.free=None;self.second=second;self.events={}
+    def event(self,name):self.events[name]=self.events.get(name,0)+1
+    def allocate_cell(self,node):
+        if self.free is None:i=len(self.cells);self.cells.append(node);self.event('cell_allocate')
+        else:i=self.free;self.free=self.cells[i];self.cells[i]=node;self.event('cell_reuse')
+        return i
+    def free_cell(self,i):self.cells[i]=self.free;self.free=i;self.event('cell_free')
+    def swap_tree(self,a,b):
+        pa,pb=a.parent,b.parent
+        if pa:
+            if pa.left is a:pa.left=b
+            else:pa.right=b
+        else:self.root=b
+        if pb:
+            if pb.left is b:pb.left=a
+            else:pb.right=a
+        else:self.root=a
+        a.parent,b.parent=pb,pa;self.event('tree_swap')
+    def swap_list(self,a,b):
+        a.next,b.next=b.next,a.next;a.previous,b.previous=b.previous,a.previous
+        if a.next is a:a.next=b
+        if b.next is b:b.next=a
+        for n in [a,b]:
+            if n.next:n.next.previous=n
+            if n.previous:n.previous.next=n
+        self.event('list_swap')
+    def increment(self,node):
+        if node is None:self.event('null_increment');return
+        if node.next and node.next.weight==node.weight:
+            self.event('equal_next');leader=self.cells[node.head]
+            if leader is not node.parent:self.swap_tree(leader,node)
+            else:self.event('leader_is_parent')
+            self.swap_list(leader,node)
+        if node.previous and node.previous.weight==node.weight:self.cells[node.head]=node.previous;self.event('old_head_previous')
+        else:self.free_cell(node.head)
+        node.weight=(node.weight+1)&0xffffffff
+        if node.next and node.next.weight==node.weight:node.head=node.next.head;self.event('new_head_shared')
+        else:node.head=self.allocate_cell(node)
+        if node.parent:
+            self.increment(node.parent)
+            if node.previous is node.parent:
+                self.event('post_parent_adjacent');parent=node.parent;self.swap_list(node,parent)
+                if self.cells[node.head] is node:self.cells[node.head]=parent;self.event('post_parent_head')
+    def add(self,symbol):
+        symbol&=255
+        if symbol in self.lookup:self.event('existing');self.increment(self.lookup[symbol]);return
+        self.event('new');leaf=Node(len(self.nodes),symbol,1);internal=Node(len(self.nodes)+1,257,1);self.nodes.extend([leaf,internal]);nyt=self.nyt
+        internal.next=nyt.next
+        if internal.next:
+            internal.next.previous=internal
+            if internal.next.weight==1:internal.head=internal.next.head
+            else:internal.head=self.allocate_cell(internal)
+        else:internal.head=self.allocate_cell(internal)
+        nyt.next=internal;internal.previous=nyt
+        leaf.next=nyt.next;leaf.next.previous=leaf;leaf.head=leaf.next.head;nyt.next=leaf;leaf.previous=nyt
+        parent=nyt.parent
+        if parent:
+            if parent.left is nyt:parent.left=internal
+            else:parent.right=internal
+        else:self.root=internal
+        internal.left=nyt;internal.right=leaf;internal.parent=parent;leaf.parent=nyt.parent=internal;self.lookup[symbol]=leaf;self.increment(parent)
+    def codewords(self):
+        out={}
+        def walk(n,bits):
+            if n.symbol!=257:out[n.symbol]=bits;return
+            walk(n.left,bits+[0]);walk(n.right,bits+[1])
+        walk(self.root,[]);return out
+    def state(self,base=0x10000):
+        def addr(n):return 0 if n is None else base+0x830+56*n.index
+        def cell(i):return 0 if i is None else base+0xb030+8*i
+        nodes=[{'index':n.index,'address':addr(n),'left':addr(n.left),'right':addr(n.right),'parent':addr(n.parent),'next':addr(n.next),'previous':addr(n.previous),'head_cell':cell(n.head),'weight':n.weight,'symbol':n.symbol} for n in self.nodes]
+        heads=sorted({n.head for n in self.nodes if n.head is not None})
+        values=[addr(v) if isinstance(v,Node) else cell(v) for v in self.cells]
+        return {'node_count':len(nodes),'pointer_cell_count':len(self.cells),'root':addr(self.root),'list_start':addr(self.nyt),'tail_slot':addr(self.nyt) if self.second else 0,'free_head':cell(self.free),'nodes':nodes,'heads':[[cell(i),addr(self.cells[i])] for i in heads],'pointer_cells':values,'lookup':[[s,addr(n)] for s,n in sorted(self.lookup.items())]}

scripts/test_huffman_codebook.py

@@ -0,0 +1,89 @@
+#!/usr/bin/env python3
+"""Differential states, branch events, seeded invariants and independent packing."""
+from collections import Counter
+from fractions import Fraction
+import json
+from pathlib import Path
+import unittest
+from analyze_huffman_codebook import digest
+from huffman_model import Huffman
+
+
+class HuffmanCodebookTests(unittest.TestCase):
+    @classmethod
+    def setUpClass(cls):
+        cls.r=json.loads(Path('docs/research/huffman-codebook.json').read_text());cls.u=json.loads(Path('docs/research/huffman-updates.json').read_text())
+
+    def test_recoding_matches_prior_native_complete_states(self):
+        for label,n in self.u['native'].items():
+            for p in n['sequences']:
+                h=Huffman()
+                for i,symbol in enumerate(p['symbols']):
+                    h.add(symbol)
+                    if p['steps']:self.assertEqual(h.state(),p['steps'][i])
+                self.assertEqual(h.state(),p['final_state'])
+
+    def test_fresh_both_object_prefix_differentials(self):
+        for n in self.r['native'].values():
+            self.assertEqual(len(n['fresh_differential_cases']),8)
+            for p in n['fresh_differential_cases']:
+                h=Huffman(p['second_object']);actual=[]
+                for symbol in p['symbols']:h.add(symbol);actual.append(digest(h.state(p['base'])))
+                self.assertEqual(actual,p['matched_prefix_sha256']);self.assertEqual(digest(h.state(p['base'])),p['final_state_sha256']);self.assertEqual(p['canaries'],[0x7b]*2)
+            for a,b in zip(n['fresh_differential_cases'][:4],n['fresh_differential_cases'][4:]):
+                self.assertEqual(a['symbols'],b['symbols']);self.assertEqual(a['events'],b['events']);self.assertEqual(a['invariants'],b['invariants'])
+
+    def test_sampled_tie_swap_and_cell_branches_occur(self):
+        for n in self.r['native'].values():
+            totals=Counter()
+            for p in n['prior_cases']+n['fresh_differential_cases']:totals.update(p['events'])
+            for name in ['new','existing','equal_next','tree_swap','leader_is_parent','list_swap','old_head_previous','cell_free','cell_reuse','new_head_shared','post_parent_adjacent','post_parent_head']:self.assertGreater(totals[name],0,name)
+
+    def test_full_seed_counts_tree_reachability_and_subtree_sums(self):
+        t=self.r['model_derived_seed_training'];s=t['final_state'];nodes={n['address']:n for n in s['nodes']};seen=set();counts=t['frequencies']
+        def walk(addr,parent):
+            self.assertNotIn(addr,seen);seen.add(addr);n=nodes[addr];self.assertEqual(n['parent'],parent)
+            if n['symbol']==257:v=walk(n['left'],addr)+walk(n['right'],addr)
+            else:v=counts[n['symbol']] if n['symbol']<256 else 0;self.assertEqual([n['left'],n['right']],[0,0])
+            self.assertEqual(n['weight'],v);return v
+        self.assertEqual(walk(s['root'],0),1053340);self.assertEqual(seen,set(nodes));self.assertEqual(len(nodes),513)
+        self.assertEqual(t['repetitions'],1053340);self.assertEqual(t['events']['new'],256);self.assertEqual(t['events']['existing'],1053340-256)
+
+    def test_training_checkpoint_totals_and_pool_partition_summary(self):
+        t=self.r['model_derived_seed_training'];self.assertEqual(len(t['checkpoints']),256)
+        for i,p in enumerate(t['checkpoints']):
+            self.assertEqual(p['through_symbol'],i);self.assertEqual(p['repetitions'],sum(t['frequencies'][:i+1]));self.assertEqual(p['invariants']['total_weight'],p['repetitions']);self.assertEqual(p['invariants']['nodes'],2*(i+1)+1)
+            self.assertEqual(p['invariants']['nodes'],p['invariants']['rank_nodes']);self.assertEqual(p['invariants']['allocated_cells'],p['invariants']['live_cells']+p['invariants']['free_cells'])
+        self.assertEqual(t['invariants']['allocated_cells'],493);self.assertEqual(t['invariants']['free_cells'],4)
+
+    def test_codewords_match_tree_and_are_prefix_free_complete(self):
+        t=self.r['model_derived_seed_training'];s=t['final_state'];nodes={n['address']:n for n in s['nodes']};expected={}
+        def walk(addr,bits):
+            n=nodes[addr]
+            if n['symbol']==257:walk(n['left'],bits+[0]);walk(n['right'],bits+[1])
+            else:expected[n['symbol']]=bits
+        walk(s['root'],[]);words={w['symbol']:w['bits'] for w in t['codebook']};self.assertEqual(words,expected);self.assertEqual(set(words),set(range(257)))
+        strings=[''.join(map(str,bits)) for bits in words.values()]
+        for i,a in enumerate(strings):
+            for j,b in enumerate(strings):
+                if i!=j:self.assertFalse(b.startswith(a))
+        self.assertEqual(sum(Fraction(1,2**len(bits)) for bits in words.values()),1);self.assertEqual(min(map(len,words.values())),2);self.assertEqual(max(map(len,words.values())),11)
+        self.assertEqual(digest(t['codebook']),t['codebook_sha256']);self.assertEqual(digest(s),t['final_state_sha256'])
+
+    def test_native_symbol_roundtrips_match_independent_codeword_packing(self):
+        words={w['symbol']:w['bits'] for w in self.r['model_derived_seed_training']['codebook']};rows=self.r['modeled_tree_native_symbol_roundtrips'];self.assertEqual(len(rows),8)
+        for p in rows:
+            bits=[bit for symbol in p['symbols'] for bit in words[symbol]];end=p['initial_bit']+len(bits);expected=[0]*((end+7)//8)
+            for i,bit in enumerate(bits):position=p['initial_bit']+i;expected[position//8]|=bit<<(position%8)
+            self.assertEqual(p['encoded_bytes'],expected);self.assertEqual(p['transmit_cursor'],end);self.assertEqual(p['decoded_symbols'],p['symbols']);self.assertEqual(p['canaries'],[0x7b]*2)
+            cursor=p['initial_bit'];expected_cursors=[]
+            for symbol in p['symbols']:cursor+=len(words[symbol]);expected_cursors.append(cursor)
+            self.assertEqual(p['receive_cursors'],expected_cursors)
+
+    def test_paired_native_symbol_results_match(self):
+        rows=self.r['modeled_tree_native_symbol_roundtrips'];a,b=rows[:4],rows[4:]
+        def clean(p):return {k:v for k,v in p.items() if k not in ['native_label','trace_sha256']}
+        self.assertEqual([clean(p) for p in a],[clean(p) for p in b])
+
+
+if __name__=='__main__':unittest.main()