Recover native BSP plane node and leaf payload loading

docs/ASSET_PIPELINE_TODO.md

@@ -202,3 +202,5 @@ Names and output formats below are proposals to refine after the relevant resear
 - [ ] AP-133 Produce an exhaustive conversion report and resolve all required-content blockers before release.
 
 Native collision follow-up: [BSP planes](research/BSP_PLANES.md) records type/sign fields and 16-byte native node addressing. Connect these to file-lump conversion only after loading/type/sign producers and valid counts are recovered (RE-320); native in-memory offsets are not a disk-format specification.
+
+Native loading follow-up: [BSP payload loading](research/BSP_LOADING.md) connects plane/node/leaf descriptors 2/3/4 to native conversion. Preserve original disk fields and every map variant; native collision projection omits fields that rendering or other consumers may need. Complete range/reference validation and actual-map conversion remain RE-324 work.

docs/FINDINGS.md

@@ -336,3 +336,9 @@ Bounded qagame paths recover bomb-plant/inactivity correction requests, direct f
 **F-098 — Observed general plane boundaries:** eight verified sign dispatches classify corner projections with front >= distance and back < distance. Maximum-face equality can differ from axial classification. Native arithmetic order is retained; arbitrary rounding, exceptional inputs and platform behavior remain open. [PLANE-002](research/BSP_PLANES.md).
 
 **F-099 — Observed native BSP entry/order:** 16-byte native node records hold plane pointer +0 and children +8/+12; negative children encode complemented leaf IDs. Bounded one-node compositions visit child 0 before child 1 when crossing. Full unrolled/recursive traversal, callback effects, allocation validity and runtime behavior remain open. [PLANE-003](research/BSP_PLANES.md).
+
+**F-100 — Observed native plane production:** 16-byte disk payloads become 20-byte records; type chooses the first component exactly +1, otherwise 3, and strict negative comparisons produce signbits. Negative zero is copied without setting a bit. Tested allocation reserves twelve spare records; their initialization/use and runtime behavior remain open. [LOAD-001](research/BSP_LOADING.md).
+
+**F-101 — Observed node reconstruction:** 36-byte disk records become 16-byte native nodes containing plane_base + 20 * signed index and unchanged child words. Sampled conversion performs no local plane-index guard and omits the six trailing disk words. Complete file validation and native ownership/runtime behavior remain open. [LOAD-002](research/BSP_LOADING.md).
+
+**F-102 — Observed leaf projection/state:** 48-byte disk records become 24-byte native leaves in word order 0,1,10,11,8,9. Zero-seeded running cluster/area words track tested maximum nonnegative IDs plus one; leaf storage reserves two spare records and later requests area-related storage. Reference semantics, initialization, count overflow and runtime conformance remain open. [LOAD-003](research/BSP_LOADING.md).

docs/PROVISIONAL_TODO.md

@@ -514,3 +514,5 @@ This revision contains **384 provisional tasks**. Select a few bounded items at
 Before marking a broad subsystem complete, reconcile it with the canonical checklist, the per-file coverage registry, variant conversion reports, runtime fixtures and the deviation register. Keep unfinished and blocked work visible with the specific missing evidence or dependency.
 
 Research progress: [BSP plane boundaries and bounded node traversal](research/BSP_PLANES.md) completes bounded RE-317–319. Broad geometry/runtime/port milestones remain open; see RE-320 for full traversal and loading work.
+
+Research progress: [native BSP payload loading](research/BSP_LOADING.md) completes bounded RE-321–323. Complete asset conversion and geometry milestones remain open; RE-324 tracks actual variant audits, validation, reference ownership and runtime fixtures.

docs/RESEARCH_LEDGER.md

@@ -181,3 +181,7 @@ Completed bounded/static RE-313–315 with `analyze_leaf_metadata.py`: ten verif
 ## 2026-10-11 — Native BSP plane boundaries and root traversal
 
 Completed bounded RE-317–319 with `analyze_bsp_planes.py`: four hash-verified native graphs, two verified eight-entry jump tables, 124 plane classifications, ten node-entry and eight single-node traversal slices. PLANE-001–003/F-097–099 distinguish axial/general maximum-face equality, finite corner classification from arbitrary rounding, and local node order from complete recursion. Eight independent tests check geometry/table/order/paired outcomes and evaluator carry/byte/address semantics. The full suite passes 246 tests; new and preceding reports regenerate byte-identically, storage and documentation checks pass. RE-316 remains open; RE-320 tracks loading/ownership, complete unrolled/recursive/callback composition, actual maps, float environment and runtime/platform conformance. No original process or native callee executes.
+
+## 2026-10-11 — Native BSP payload loading
+
+Completed bounded RE-321–323 with `analyze_bsp_loading.py`: eight verified graphs, six header descriptor-routing slices, 72 supplied-allocation plane/node/leaf payload conversions and corresponding allocation-size slices. LOAD-001–003/F-100–102 distinguish disk/native records, +1 type priority from unit-normal validation, signed plane-pointer construction from valid indices, and projected leaf words/maxima from complete ownership. Eight independent tests cover field copies/type/sign/padding, signed pointers/children, leaf projection/maxima, allocation/routing arithmetic, exact byte writes, paired outputs and restricted byte/conditional/signed-branch evaluator semantics. The full suite passes 254 tests; current and preceding reports regenerate byte-identically, storage and documentation checks pass. RE-320 remains open; RE-324 tracks complete validation/reset/allocator/spare-record/reference/portal contracts, actual BSP variants, full traversal and runtime/platform behavior. No original process, allocator or native callee executes.

docs/REVERSE_ENGINEERING_TODO.md

@@ -382,3 +382,8 @@ Purpose: turn the supplied installation into an evidence-backed specification of
 - [x] RE-318 Verify the eight-entry general-plane jump table and bounded sign/corner/projection/equality classifications ([PLANE-002](research/BSP_PLANES.md)).
 - [x] RE-319 Recover bounded node-entry addressing and one-node negative-child visit order ([PLANE-003](research/BSP_PLANES.md)).
 - [ ] RE-320 Recover BSP node/plane/leaf loading/count/lifetime and sign/type producers, all unrolled/recursive query paths with callback capacity/overflow effects, real-map fixtures, exceptional float environment and original-runtime/all-platform conformance.
+
+- [x] RE-321 Recover paired bounded plane payload conversion, +1 component type priority, signbits/padding and allocation/header routing ([LOAD-001](research/BSP_LOADING.md)).
+- [x] RE-322 Recover disk-to-native node reconstruction, signed stride-20 plane pointers, unchanged children and allocation sizes ([LOAD-002](research/BSP_LOADING.md)).
+- [x] RE-323 Recover leaf word projection, running cluster/area maxima and bounded leaf/area/portal allocation requests ([LOAD-003](research/BSP_LOADING.md)).
+- [ ] RE-324 Recover complete header/lump/error/endian/count validation, allocator/reset/spare-record initialization, brush/surface references and area/portal ownership; audit actual BSP variants and compose full loading/traversal/callback/runtime/all-platform fixtures.

docs/research/BSP_LOADING.md

@@ -0,0 +1,58 @@
+# Native BSP plane, node and leaf loading
+
+Reproduce: `python3 scripts/analyze_bsp_loading.py`. Tests: `python3 scripts/test_bsp_loading.py`. Evidence: [bsp-loading.json](bsp-loading.json), containing hash-verified graphs for CM_LoadMap and three loaders in both Linux ELF64 engines. This follows [plane classification](BSP_PLANES.md) and [leaf metadata](LEAF_METADATA.md).
+
+These are bounded synthetic instruction compositions with supplied allocation returns, not original executable/callee execution. Every payload probe starts after the first allocator call and stops before an epilogue or later allocator. First allocation sizes and descriptor routing are separate instruction slices; they do not execute loader entry validation. The leaf auxiliary-allocation continuation also supplies an allocator return. No real map is loaded in this batch.
+
+## LOAD-001 — Plane conversion and derived fields (RE-321)
+
+CM_LoadMap copies 0x90 bytes into a local header and compares version against 46 in the retained graph. The bounded routing slice passes header +24 to CMod_LoadPlanes, corresponding to descriptor 2 under the eight-byte prefix/eight-byte descriptor layout. Header +32 and +40 route nodes and leaves to descriptors 3 and 4. Native header read safety, magic checks and the complete file-bound validation contract remain open.
+
+| Plane representation | Fields |
+| --- | --- |
+| Disk record, 16 bytes | normal float bits +0/+4/+8; distance bits +12 |
+| Native record, 20 bytes | same four words; type byte +16; signbits byte +17; padding +18/+19 |
+
+The tested allocation request is `20 * (count + 12)` bytes. The twelve spare plane records are not initialized by the sampled payload loops. Their later use remains open. Payload writes preserve all four original words, including negative zero. Padding remains the supplied 0xa5 sentinel; this does not establish real allocator initialization.
+
+For finite normals, type is the first component index whose value equals **+1.0**, otherwise 3. There is no local normalization check in the sampled conversion. Thus `(1,-2,3)` gets type 0 and `(-1,0,0)` gets type 3. This producer rule explains why negative axis normals reach the general classifier. Signbits are the three component `< 0.0` predicates; negative zero does not set a bit. The comparison constant comes from the verified file-backed ELF section.
+
+Twenty-four paired plane payload probes cover counts 1–9, 16, 17 and 31, including odd/even peel and pair-loop paths. The 31-record case samples every normal in `{-1,0,1}^3`, negative zero, fractional components and two nonunit normals with +1 components. Exceptional values and MXCSR-dependent behavior remain open.
+
+## LOAD-002 — Native node reconstruction (RE-322)
+
+| Node representation | Fields used by this loader |
+| --- | --- |
+| Disk record, 36 bytes | signed plane index +0; child words +4/+8 |
+| Native record, 16 bytes | plane pointer +0; unchanged child words +8/+12 |
+
+The allocation request is `16 * count`. The loader sign-extends the disk plane index and computes `plane_base + 20 * index`; it does not locally check that index before writing the pointer in sampled paths. A deliberately invalid -1 index therefore produces `plane_base - 20` without dereferencing that plane. This observation is not a valid-map or memory-safety guarantee. Child bits are copied unchanged; their node/negative-leaf meaning is corroborated by the earlier traversal evidence.
+
+Disk offsets +12 through +32 are not copied into native collision nodes. This is a collision-loader projection, not permission for the asset converter to discard source bounds or other consumers' requirements. Twenty-four paired payload probes exercise counts across remainder and four-record unrolling, preserve all child words and write exactly the native records.
+
+## LOAD-003 — Leaf projection and allocation state (RE-323)
+
+The allocation request is `24 * (count + 2)` bytes. The two spare leaf records remain untouched in these payload probes.
+
+| Disk offset (48-byte record) | Native offset (24-byte record) |
+| --- | --- |
+| +0 cluster word | +0 |
+| +4 area word | +4 |
+| +40 | +8 |
+| +44 | +12 |
+| +32 | +16 |
+| +36 | +20 |
+
+Disk offsets +8 through +28 are not copied. Reference-word semantics and complete brush/surface array ownership must still be connected to their loaders and consumers; the table records exact word movement without assuming native/disk layout identity.
+
+With cluster and area running words explicitly seeded to zero, tested signed values update each running word to the maximum nonnegative ID plus one. Cluster -1 leaves the seeded cluster count unchanged. Repeated IDs do not grow counts. The maximum-word storage addresses match across the observed unrolled lanes within each engine. The collector records their roles separately.
+
+After the payload, the loader requests `8 * area_count` bytes for one auxiliary allocation. With an explicitly supplied return for that call, the next instruction slice requests `4 * area_count * area_count` bytes using a 32-bit square before sign extension. The report samples small valid counts only; overflow, allocator zeroing, portal/area structures and reset/lifetime remain open.
+
+Twenty-four paired leaf payload probes cover odd/even peel and pair-loop paths. The report also retains the static entry checks: plane length alignment, node/leaf reciprocal-division size gates and nonpositive-count error requests. Recorded error texts are `MOD_LoadBmodel: funny lump size`, `Map with no planes`, `Map has no nodes` and `Map with no leafs`. These guards are not exercised; complete malformed-lump behavior is not claimed.
+
+## Validation and follow-up
+
+Eight independent tests cover copy/type/sign/padding behavior, signed node pointer arithmetic and child bits, leaf word reordering/maxima, allocation arithmetic, header descriptor routing, exact per-record byte writes, paired outcomes and restricted evaluator byte/conditional/signed-branch semantics. No probe writes into a spare record. Larger instruction traces retain counts/hashes; small traces retain addresses.
+
+RE-324: validate complete header/lump bounds, alignment/division/error paths, endian handling and count overflow; recover allocator/reset/zeroing and spare-record initialization; recover leaf brush/surface references, area/portal structures and lifetime; audit every actual BSP variant and compose loading with full recursive query/callback paths; obtain original-runtime and other-platform differential fixtures. Rendering geometry and Rust/Bevy/Solari conversion remain separate open work.

docs/research/BSP_PLANES.md

@@ -40,3 +40,5 @@ The retained full traversal graph contains nine plane call sites and unrolled ne
 The report contains 124 plane classifications, ten entry slices and eight root compositions. Eight independent tests compare axial rules, all-corner projection extrema, sign dispatch, invalid-sign behavior, equality differences, node addressing/order and paired outcomes. Evaluator tests cover byte unsigned flags, carry subtraction, low-byte register preservation and indexed addresses. Unknown flags, unsupported operations and unknown memory remain errors.
 
 RE-320: recover node/plane/leaf lump loading, allocation/count/lifetime and sign/type producers; compose all unrolled and recursive query paths with callback effects and capacity boundaries; sample real maps; recover float exceptional/MXCSR behavior and differential original-runtime/platform fixtures. No complete collision geometry, original-runtime conformance, or Rust/Bevy/Solari implementation is claimed.
+
+Follow-up: [BSP payload loading](BSP_LOADING.md) establishes bounded +1 type/sign producers, native plane stride, node reconstruction and leaf projection. RE-320 remains open for complete loading/validation/recursive-query/runtime contracts.

docs/research/README.md

@@ -115,3 +115,5 @@ The extension collector adds five focused tests for native tail calls, command-r
 [Leaf areas, clusters and query outputs](LEAF_METADATA.md) adds RE-313–315: all-lane area/cluster compositions, exact overflow fallback and guarded leaf/store callback contracts. Reproduce with `python3 scripts/analyze_leaf_metadata.py`; check with `python3 scripts/test_leaf_metadata.py`. Full BSP traversal/layout/output ownership, consumers and original-runtime/platform conformance remain open.
 
 [Native BSP planes and bounded node traversal](BSP_PLANES.md) adds RE-317–319: axial/general equality rules, verified eight-case sign dispatch and one-node child order/addressing. Reproduce with `python3 scripts/analyze_bsp_planes.py`; check with `python3 scripts/test_bsp_planes.py`. Full traversal/loading/callback/float/runtime/platform contracts remain open.
+
+[Native BSP plane, node and leaf loading](BSP_LOADING.md) adds RE-321–323: descriptor routing, bounded payload conversion, derived plane fields, native node pointers, leaf projection/maxima and allocation requests. Reproduce with `python3 scripts/analyze_bsp_loading.py`; check with `python3 scripts/test_bsp_loading.py`. Complete validation/loading/ownership/recursive-query/runtime/platform contracts remain open.

scripts/analyze_bsp_loading.py

@@ -0,0 +1,114 @@
+#!/usr/bin/env python3
+"""Bounded native plane/node/leaf payload conversion after supplied allocation."""
+import gzip
+import hashlib
+import itertools
+import json
+from pathlib import Path
+import re
+from analyze_chat_crosshair import bits
+from analyze_chat_paths import native_string
+from analyze_collision_composition import evaluate,get_word,put_word
+from analyze_engine_control import control_flow
+from analyze_frame_timing import read_address
+from analyze_native import elf
+from analyze_qvm import write_json
+from analyze_vm_extensions import native_functions,verified
+
+NORMALS=list(itertools.product([-1.,0.,1.],repeat=3))+[(-0.,0.,0.),(.5,-.25,.75),(1.,-2.,3.),(0.,1.,-2.)]
+COUNTS=list(range(1,10))+[16,17,31]
+
+
+def digest(trace):return hashlib.sha256(json.dumps(trace,separators=(',',':')).encode()).hexdigest()
+
+
+def main():
+    out={'native':{},'limitations':'Synthetic native payload conversion after an explicitly supplied '
+         'first allocation return; leaf auxiliary-allocation continuation also supplies a return. No allocation, error, original process or native callee executes; '
+         'execution stops before epilogues/later allocations. Entry length/division gates '
+         'is retained as static evidence, not executed. No actual BSP file is loaded by these probes. '
+         'Complete validation/ownership/reset/endian/exceptional float and runtime/platform contracts remain open.'}
+    source,output,plane_base=0x10000,0x30000,0x50000
+    for label in ['linux--Quake3-UrT-Ded.x86_64','linux--Quake3-UrT.x86_64']:
+        r=json.loads(Path(f'docs/research/native/{label}.json').read_text());data=verified((Path('.cache/UrbanTerror43')/r['source']['path']).read_bytes(),r['source']['sha256'])
+        a=r['artifacts']['disassembly.txt'];fs=native_functions(verified(gzip.open(a['path'],'rb').read(),a['uncompressed_sha256']).decode());metadata,_=elf(data)
+        graphs={n:control_flow(fs[n]) for n in ['CMod_LoadPlanes','CMod_LoadNodes','CMod_LoadLeafs','CM_LoadMap']};results={};routing={}
+        caller=graphs['CM_LoadMap']['instructions']
+        for name in ['CMod_LoadPlanes','CMod_LoadNodes','CMod_LoadLeafs']:
+            i=next(i for i,x in enumerate(caller) if '<'+name+'>' in x['operands'])
+            if caller[i-2]['opcode']!='leaq' or caller[i-2]['operands']!='0x10(%rsp), %rdi' or caller[i-1]['opcode']!='addq':raise ValueError('header routing changed')
+            offset=int(caller[i-1]['operands'].split(',')[0][1:],0)
+            if offset<8 or (offset-8)%8:raise ValueError('unexpected lump descriptor offset')
+            t=evaluate(graphs['CM_LoadMap'],caller[i-2]['address'],[caller[i]['address']],{}, {'%rsp':0x90000},capture_registers=True)
+            routing[name]={'header_relative_offset':offset,'lump_index':(offset-8)//8,'descriptor_pointer':t['registers']['rdi'],'trace':t['trace'],'call':caller[i]['address']}
+
+        for name in ['CMod_LoadPlanes','CMod_LoadNodes','CMod_LoadLeafs']:
+            g=graphs[name]
+            rows=g['instructions'];alloc=[i for i,x in enumerate(rows) if '<Hunk_Alloc>' in x['operands']];first=alloc[0];start=rows[first+1]['address']
+            end=rows[alloc[1]]['address'] if name=='CMod_LoadLeafs' else next(x['address'] for x in rows[first+1:] if x['opcode']=='popq')
+            static_globals=[];seed={}
+            for x in rows:
+                if '(%rip)' not in x['operands']:continue
+                address=int(re.search(r'#\s*0x([0-9a-f]+)',x['operands'])[1],16)
+                static_globals.append({'address':address,'instruction':x['address'],'opcode':x['opcode'],'operands':x['operands']})
+                if x['opcode']=='movss':
+                    raw,_=read_address(data,metadata['sections'],address,4)
+                    for i,v in enumerate(raw):seed[address+i]=v
+            if name=='CMod_LoadNodes':
+                slots={x['address'] for x in static_globals if x['opcode']=='addq'}
+                if len(slots)!=1:raise ValueError('node plane-base slot changed')
+                for slot in slots:put_word(seed,slot,plane_base,8)
+            elif name=='CMod_LoadLeafs':
+                slots={x['address'] for x in static_globals if x['opcode']=='cmpl'}
+                if len(slots)!=2:raise ValueError('leaf running maxima changed')
+                for slot in slots:put_word(seed,slot,0)
+            if name=='CMod_LoadPlanes':allocation_start=next(x['address'] for x in rows[:first] if x['opcode']=='leal' and x['operands']=='0xc(%rbp), %ecx')
+            elif name=='CMod_LoadLeafs':allocation_start=next(x['address'] for x in rows[:first] if x['opcode']=='leal' and x['operands']=='0x2(%rbp), %r9d')
+            else:allocation_start=next(x['address'] for x in rows[:first] if x['opcode']=='movslq' and x['operands']=='%r14d, %rdi')
+            probes=[]
+            for count in COUNTS:
+                m=dict(seed);disk=[];disk_stride={'CMod_LoadPlanes':16,'CMod_LoadNodes':36,'CMod_LoadLeafs':48}[name]
+                native_stride={'CMod_LoadPlanes':20,'CMod_LoadNodes':16,'CMod_LoadLeafs':24}[name]
+                for i in range(count):
+                    if name=='CMod_LoadPlanes':words=[bits(v) for v in NORMALS[i%len(NORMALS)]]+[bits(i-.5)]
+                    elif name=='CMod_LoadNodes':words=[[-1,0,1,2,7][i%5],-i-1,i+1]+[0x1100+i*16+j for j in range(6)]
+                    else:words=[[-1,0,2,2,7][i%5],[0,3,1,0][i%4]]+[0x2200+i*16+j for j in range(6)]+[100+i,200+i,300+i,400+i]
+                    disk.append([v&0xffffffff for v in words])
+                    for j,v in enumerate(words):put_word(m,source+disk_stride*i+4*j,v)
+                # Fill destination with sentinels, including native padding and spare records.
+                for i in range(native_stride*(count+12)+2):m[output-1+i]=0xa5
+                allocation=evaluate(g,allocation_start,[rows[first]['address']],m,{'%rbp':count,'%r14':0,'%r13':source},capture_registers=True)
+                regs={'%rax':output,'%rbx':source,'%rbp':count,'%r12':count}
+                t=evaluate(g,start,[end],m,regs,limit=4096,capture_registers=True)
+                records=[]
+                for i in range(count):
+                    p=output+native_stride*i
+                    if name=='CMod_LoadPlanes':records.append({'words':[get_word(t['memory'],p+4*j) for j in range(4)],'type':get_word(t['memory'],p+16,1),'signbits':get_word(t['memory'],p+17,1),'padding':[t['memory'][p+18],t['memory'][p+19]]})
+                    elif name=='CMod_LoadNodes':records.append({'plane_pointer':get_word(t['memory'],p,8),'children':[get_word(t['memory'],p+8),get_word(t['memory'],p+12)]})
+                    else:records.append([get_word(t['memory'],p+4*j) for j in range(6)])
+                portal_allocation=None
+                if name=='CMod_LoadLeafs':
+                    continuation=dict(t['registers']);continuation['rax']=0x70000
+                    portal_allocation=evaluate(g,rows[alloc[1]+1]['address'],[rows[alloc[2]]['address']],t['memory'],continuation,capture_registers=True)
+                destination_writes=[w for w in t['writes'] if output<=w['address']<output+native_stride*(count+12)]
+                outside=[w for w in t['writes'] if output-1<=w['address']<output or output+native_stride*count<=w['address']<output+native_stride*(count+12)]
+                probes.append({'count':count,'allocation_bytes':allocation['registers']['rdi'],'allocation_trace':allocation['trace'],'input_words':disk,'records':records,'destination_writes':destination_writes,'writes_outside_records':outside,
+                    'globals':{str(x['address']):get_word(t['memory'],x['address'],8 if x['opcode']=='movq' else 4) for x in static_globals if x['operands'].split('#')[0].strip().endswith('(%rip)') and x['opcode'] in ['movq','movl'] and all(x['address']+j in t['memory'] for j in range(8 if x['opcode']=='movq' else 4))},
+                    'portal_allocation_bytes':portal_allocation['registers']['rdi'] if portal_allocation else None,'portal_allocation_trace':portal_allocation['trace'] if portal_allocation else None,'next_allocation_bytes':t['registers']['rdi'] if name=='CMod_LoadLeafs' else None,'trace':t['trace'] if count<=4 else None,'trace_length':len(t['trace']),'trace_sha256':digest(t['trace'])})
+            errors=[]
+            for i,x in enumerate(rows):
+                if '<Com_Error>' not in x['operands']:continue
+                fmt=next(y for y in reversed(rows[max(0,i-4):i]) if y['opcode']=='movl' and y['operands'].split('#')[0].strip().endswith(', %esi'))
+                address=int(fmt['operands'].split(',')[0][1:],0)
+                errors.append({'call':x['address'],'text':native_string(data,metadata['sections'],address)})
+            maxima={}
+            if name=='CMod_LoadLeafs':
+                for role,reg in [('cluster','%edx'),('area','%eax')]:
+                    maxima[role]=next(x['address'] for x in static_globals if x['opcode']=='cmpl' and x['operands'].split('#')[0].strip().endswith(', '+reg))
+            results[name]={'maxima_slots':maxima,'start_after_supplied_allocation':start,'stop_before_epilogue_or_allocation':end,'global_references':static_globals,'error_requests':errors,'probes':probes}
+        out['native'][label]={'source':r['source'],'disassembly_sha256':a['uncompressed_sha256'],'graphs':graphs,'lump_routing':routing,'loaders':results}
+    write_json(Path('docs/research/bsp-loading.json'),out)
+    print('Recovered 72 supplied-allocation plane/node/leaf conversion probes')
+
+
+if __name__=='__main__':main()

scripts/analyze_collision_composition.py

@@ -14,10 +14,11 @@ from analyze_vm_extensions import native_functions,verified
 
 def register(r):
     aliases={'eax':('rax',32),'edi':('rdi',32),'esi':('rsi',32),'edx':('rdx',32),
-             'ecx':('rcx',32),'ebp':('rbp',32),'ebx':('rbx',32),'al':('rax',8),'sil':('rsi',8)}
+             'ecx':('rcx',32),'ebp':('rbp',32),'ebx':('rbx',32),'al':('rax',8),'sil':('rsi',8),'dl':('rdx',8)}
     r=r.lstrip('%')
     if r in aliases:return aliases[r]
     if re.fullmatch(r'r\d+d',r):return r[:-1],32
+    if re.fullmatch(r'r\d+b',r):return r[:-1],8
     if re.fullmatch(r'r(?:ax|di|si|dx|cx|bp|bx|sp|ip|\d+)',r):return r,64
     if re.fullmatch(r'xmm\d+',r):return r,32
     raise ValueError('unsupported register '+r)
@@ -37,7 +38,7 @@ def evaluate(graph,start,stops,memory,registers,limit=128,capture_registers=Fals
     regs={register(k)[0]:v for k,v in registers.items()};m=dict(memory);equal=None;ge=greater=below=None;signed_le=signed_lt=negative=None;pc=start;trace=[];writes=[]
     def read_reg(name):
         key,size=register(name)
-        if key not in regs:raise ValueError('unknown register')
+        if key not in regs:raise ValueError('unknown register '+name)
         return regs[key]&((1<<size)-1)
     def address(arg):
         match=re.fullmatch(r'(-?0x[0-9a-f]+)?\((%\w+)?(?:,(%\w+)(?:,([1248]))?)?\)',arg)
@@ -61,18 +62,18 @@ def evaluate(graph,start,stops,memory,registers,limit=128,capture_registers=Fals
         i=indices[pc];pc=rows[i+1]['address'] if i+1<len(rows) else None
         regs['rip']=pc
         if op in ['nop','nopl','nopw']:pass
-        elif op in ['movl','movq','movss','movabsq']:
-            a,b=args.split(', ');width=8 if op in ['movq','movabsq'] else 4;write(b,read(a,width),width)
+        elif op in ['movb','movl','movq','movss','movabsq']:
+            a,b=args.split(', ');width=8 if op in ['movq','movabsq'] else 1 if op=='movb' else 4;write(b,read(a,width),width)
         elif op=='movzbl':
             a,b=args.split(', ');write(b,read(a,1),4)
         elif op=='sbbl':
             if below is None:raise ValueError('unmodeled carry flag')
             a,b=args.split(', ');write(b,read(b,4)-read(a,4)-int(below),4)
             equal=None;signed_le=signed_lt=negative=None;ge=greater=below=None
-        elif op=='cmovael':
+        elif op in ['cmovael','cmovbl']:
             if ge is None:raise ValueError('unmodeled unsigned flags')
             a,b=args.split(', ')
-            if ge:write(b,read(a,4),4)
+            if (ge if op=='cmovael' else below):write(b,read(a,4),4)
         elif op=='movaps':
             a,b=args.split(', ')
             if not a.startswith('%xmm') or not b.startswith('%xmm'):raise ValueError('unsupported vector move')
@@ -112,7 +113,8 @@ def evaluate(graph,start,stops,memory,registers,limit=128,capture_registers=Fals
             write(b,v>>(read(a,8)&63),8);equal=None;signed_le=signed_lt=negative=None;ge=greater=below=None
         elif op in ['xorl','orl','andl','subl','addl','addq','subq']:
             width=8 if op in ['addq','subq'] else 4
-            a,b=args.split(', ');v,w=read(a,width),read(b,width)
+            a,b=args.split(', ')
+            v,w=(0,0) if op=='xorl' and a==b else (read(a,width),read(b,width))
             result=v^w if op=='xorl' else v|w if op=='orl' else v&w if op=='andl' else w-v if op in ['subl','subq'] else v+w
             write(b,result,width);equal=(result&((1<<(width*8))-1))==0
             signed_le=signed_lt=negative=None;ge=greater=below=None
@@ -132,15 +134,15 @@ def evaluate(graph,start,stops,memory,registers,limit=128,capture_registers=Fals
             write(b,bits(w+v if op=='addss' else w-v if op=='subss' else w*v),4)
         elif op=='comiss':
             a,b=args.split(', ');v,w=number(read(a,4)),number(read(b,4));equal=v==w;ge=w>=v;greater=w>v;below=w<v;signed_le=signed_lt=negative=None
-        elif op in ['sete','setbe','setae']:
-            if op=='sete' and equal is None:raise ValueError('unmodeled zero flag')
-            if op in ['setbe','setae'] and below is None:raise ValueError('unmodeled unsigned flags')
-            write(args,int(equal if op=='sete' else ge if op=='setae' else below or equal),1)
-        elif op in ['je','jne','jae','ja','jb','jbe','jle','jl','js','jns','jmp']:
+        elif op in ['sete','setne','setb','setbe','setae']:
+            if op in ['sete','setne'] and equal is None:raise ValueError('unmodeled zero flag')
+            if op in ['setb','setbe','setae'] and below is None:raise ValueError('unmodeled unsigned flags')
+            write(args,int(equal if op=='sete' else not equal if op=='setne' else below if op=='setb' else ge if op=='setae' else below or equal),1)
+        elif op in ['je','jne','jae','ja','jb','jbe','jle','jl','jge','js','jns','jmp']:
             if op in ['je','jne'] and equal is None:raise ValueError('unmodeled zero flag')
-            if op in ['jle','jl','js','jns'] and signed_le is None:raise ValueError('unmodeled signed flags')
+            if op in ['jle','jl','jge','js','jns'] and signed_le is None:raise ValueError('unmodeled signed flags')
             if op in ['jae','ja','jb','jbe'] and ge is None:raise ValueError('unmodeled unsigned flags')
-            taken=op=='jmp' or (equal if op=='je' else not equal if op=='jne' else signed_le if op=='jle' else signed_lt if op=='jl' else negative if op=='js' else not negative if op=='jns' else greater if op=='ja' else below if op=='jb' else below or equal if op=='jbe' else ge)
+            taken=op=='jmp' or (equal if op=='je' else not equal if op=='jne' else signed_le if op=='jle' else signed_lt if op=='jl' else not signed_lt if op=='jge' else negative if op=='js' else not negative if op=='jns' else greater if op=='ja' else below if op=='jb' else below or equal if op=='jbe' else ge)
             if taken:pc=int(args.split()[0],16)
         else:raise ValueError('unsupported composition opcode '+op)
     result={'trace':trace,'stop':pc,'memory':m,'writes':writes}

scripts/test_bsp_loading.py

@@ -0,0 +1,97 @@
+#!/usr/bin/env python3
+"""Independent disk/native conversion, allocation and instruction oracles."""
+import json
+from pathlib import Path
+import unittest
+from analyze_chat_crosshair import number
+from analyze_collision_composition import evaluate
+from test_bsp_planes import graph
+
+
+def signed(v):return v if v<2**31 else v-2**32
+
+
+class BspLoadingTests(unittest.TestCase):
+    @classmethod
+    def setUpClass(cls):cls.r=json.loads(Path('docs/research/bsp-loading.json').read_text())
+
+    def test_plane_copy_type_priority_signs_and_padding(self):
+        for n in self.r['native'].values():
+            for p in n['loaders']['CMod_LoadPlanes']['probes']:
+                for original,record in zip(p['input_words'],p['records']):
+                    normal=[number(v) for v in original[:3]]
+                    expected_type=next((i for i,v in enumerate(normal) if v==1),3)
+                    self.assertEqual(record['words'],original);self.assertEqual(record['type'],expected_type)
+                    self.assertEqual(record['signbits'],sum(1<<i for i,v in enumerate(normal) if v<0))
+                    self.assertEqual(record['padding'],[0xa5,0xa5])
+            final=n['loaders']['CMod_LoadPlanes']['probes'][-1]
+            self.assertEqual(final['records'][27]['words'][0],0x80000000)
+            self.assertEqual(final['records'][27]['signbits'],0)
+            self.assertEqual(final['records'][29]['type'],0)  # +1 priority even for a nonunit normal.
+
+    def test_node_signed_plane_pointer_and_exact_children(self):
+        for n in self.r['native'].values():
+            for p in n['loaders']['CMod_LoadNodes']['probes']:
+                for original,record in zip(p['input_words'],p['records']):
+                    self.assertEqual(record['plane_pointer'],(0x50000+20*signed(original[0]))&((1<<64)-1))
+                    self.assertEqual(record['children'],original[1:3])
+                self.assertEqual(p['records'][0]['plane_pointer'],0x50000-20)
+
+    def test_leaf_mapping_and_running_maxima(self):
+        for n in self.r['native'].values():
+            info=n['loaders']['CMod_LoadLeafs']
+            maxima_slots=info['maxima_slots']
+            for p in info['probes']:
+                expected=[[w[0],w[1],w[10],w[11],w[8],w[9]] for w in p['input_words']]
+                self.assertEqual(p['records'],expected)
+                for role,index in [('cluster',0),('area',1)]:
+                    self.assertEqual(p['globals'][str(maxima_slots[role])],max(0,max(signed(w[index]) for w in p['input_words'])+1))
+                areas=max(0,max(signed(w[1]) for w in p['input_words'])+1)
+                self.assertEqual(p['next_allocation_bytes'],8*areas);self.assertEqual(p['portal_allocation_bytes'],4*areas*areas)
+
+    def test_allocation_sizes_and_error_texts(self):
+        for n in self.r['native'].values():
+            for name,info in n['loaders'].items():
+                self.assertEqual(len(info['probes']),12)
+                for p in info['probes']:
+                    c=p['count'];expected=20*(c+12) if name.endswith('Planes') else 16*c if name.endswith('Nodes') else 24*(c+2)
+                    self.assertEqual(p['allocation_bytes'],expected)
+                texts={e['text'] for e in info['error_requests']}
+                self.assertIn('MOD_LoadBmodel: funny lump size',texts)
+                routing=n['lump_routing'][name];index={'CMod_LoadPlanes':2,'CMod_LoadNodes':3,'CMod_LoadLeafs':4}[name]
+                self.assertEqual(routing['lump_index'],index);self.assertEqual(routing['descriptor_pointer'],0x90010+8+8*index)
+                self.assertIn({'CMod_LoadPlanes':'Map with no planes','CMod_LoadNodes':'Map has no nodes','CMod_LoadLeafs':'Map with no leafs'}[name],texts)
+
+    def test_exact_write_coverage_and_spare_records(self):
+        for n in self.r['native'].values():
+            for name,info in n['loaders'].items():
+                stride=20 if name.endswith('Planes') else 16 if name.endswith('Nodes') else 24
+                used=18 if name.endswith('Planes') else stride
+                for p in info['probes']:
+                    covered=[a for w in p['destination_writes'] for a in range(w['address'],w['address']+w['bytes'])]
+                    expected=[0x30000+stride*i+j for i in range(p['count']) for j in range(used)]
+                    self.assertEqual(sorted(covered),expected);self.assertEqual(p['writes_outside_records'],[])
+                    if p['trace'] is not None:self.assertEqual(len(p['trace']),p['trace_length'])
+
+    def test_paired_conversion_and_allocation_outcomes(self):
+        a,b=self.r['native'].values()
+        for name in a['loaders']:
+            def clean(n):return [{k:v for k,v in p.items() if k not in ['globals','trace','trace_sha256','allocation_trace','portal_allocation_trace']} for p in n['loaders'][name]['probes']]
+            self.assertEqual(clean(a),clean(b))
+
+    def test_byte_aliases_set_and_conditional_moves(self):
+        for v in [0,1,0xffffffff]:
+            g=graph([('cmpl','$0x1, %eax'),('setb','%r11b'),('setne','%dl'),('cmovbl','%edi, %esi'),('nop','')])
+            t=evaluate(g,0,[4],{}, {'%rax':v,'%r11':0x1200,'%rdx':0x3400,'%rdi':777,'%rsi':888},capture_registers=True)
+            self.assertEqual(t['registers']['r11'],0x1200+int(v<1));self.assertEqual(t['registers']['rdx'],0x3400+int(v!=1));self.assertEqual(t['registers']['rsi'],777 if v<1 else 888)
+        with self.assertRaises(ValueError):evaluate(graph([('cmovbl','%edi, %esi')]),0,[1],{}, {'%rdi':1,'%rsi':2})
+
+    def test_self_xor_and_signed_greater_equal(self):
+        g=graph([('xorl','%r8d, %r8d'),('movb','$0x7, %r8b'),('testl','%esi, %esi'),('jge','0x5'),('movl','$0x9, %r8d'),('nop','')])
+        for v in [0,1,0x80000000,0xffffffff]:
+            t=evaluate(g,0,[5],{}, {'%rsi':v},capture_registers=True)
+            self.assertEqual(t['registers']['r8'],7 if v<2**31 else 9)
+        with self.assertRaises(ValueError):evaluate(graph([('xorl','%eax, %edi')]),0,[1],{}, {})
+
+
+if __name__=='__main__':unittest.main()