urt-rs / docs /research /MESSAGE_STRINGS.md
nmarshall's picture
Recover float transport and finite angle message codecs
7e134bc
|
Raw History Blame Contribute Delete
7.96 kB

OOB data reads and message strings

Reproduce with python3 scripts/analyze_message_strings.py; validate with python3 scripts/test_message_strings.py. message-strings.json retains sixteen source/disassembly-hash-verified graphs across both Linux ELF64 engines, 106 data-read compositions, ninety string-read compositions, 120 string-write compositions and 24 writer length-guard probes. Synthetic source/result buffers use hexadecimal strings to keep this ordinary Git report below 2 MiB.

Original processes and native callees never execute. OOB ReadBits/WriteBits instructions are composed at explicit stopped CALL boundaries, with caller registers retained and child memory effects applied. Virtual backing storage is padded and distinct from cursize. String strlen returns and Q_strncpyz copy/NUL effects are supplied; those library routines and logging callees remain unexecuted. PUSH/POP and save/restore instructions are omitted/stopped explicitly, including the line reader's restores before its final terminator store. Compressed strings/data, actual packets, aliases, full lifecycle/runtime/all-platform and ABI contracts remain open.

TEXT-001 — ReadData with actual OOB reader effects (RE-377)

Each image has 53 ReadData compositions across lengths -1/0/1–9/16/17/31/32 and distinct size values from zero, one, length minus one and length. This follows the earlier supplied-result ReadData probes, now composing actual OOB ReadBits(width eight) instructions for each request. Source bytes follow (129 + 37*i) &255; the destination starts as forty 0xa5 bytes with outside canaries.

Signed nonpositive lengths issue no reads or stores. Positive lengths issue exactly that many eight-bit requests, regardless of EOF. After each read, signed readcount is compared with cursize: accepted bytes are stored as low eight bits; out-of-message results become 0xff. For these nonnegative sizes, the destination becomes the valid source prefix, followed by 0xff through the requested length, followed by unchanged 0xa5 tail bytes. Requests continue after the first failed size check.

Readcount advances once per request; the OOB bit cursor starts at seven and advances by eight per request, including EOF reads. The underlying reader itself accesses supplied backing bytes beyond cursize before the ReadData size check. This is local behavior over padded memory, not evidence that original truncated buffers are safe or that EOF recovery is complete. Legal wrapping/signed cursors, destination lengths, aliasing and compressed input remain open.

TEXT-002 — String read termination, sanitation and capacity (RE-378)

MSG_ReadString and MSG_ReadStringLine use distinct recovered static 1024-byte output buffers. MSG_ReadBigString uses a recovered static 8192-byte buffer. All return the corresponding buffer address; each writes a NUL terminator after the result and retains the unused 0xa5 tail in the supplied fixtures. This recovers local buffer addresses/bytes, not complete ownership, reentrancy or threading behavior.

The reader consumes eight-bit requests until a post-read size failure, a NUL byte, or the output's capacity minus one. ReadStringLine additionally stops at newline 0x0a; carriage return 0x0d is retained. NUL/newline/EOF termination consumes the request that triggered it. Capacity termination stops after filling the permitted output bytes, leaving the following source byte unread. High source byte 0xff is a valid byte when the post-read count is within size; it is not by itself an EOF sentinel.

For accepted nonterminating bytes, all three readers replace percent 0x25 and every value above 127 with period 0x2e. Other sampled ASCII/control bytes retain their values. No multibyte text decoding is composed.

Each reader is tested against empty/NUL input, ordinary ASCII, percent/high bytes/newline, carriage-return/newline, and all nonzero byte values ascending, with zero/truncated/full declared sizes. Complete capacity fixtures supply more than enough 'A' bytes: ordinary/line readers store 1023 bytes and terminate locally; BigString stores 8191. The following byte and final source NUL remain unread in those capacity cases. Independent oracles verify every result byte, consumed request count, actual OOB readcount/bit cursor, return pointer, terminator, retained tail and canaries.

These are OOB and explicitly padded fixtures. Static-buffer reuse/caller lifetime, compressed cursor-driven EOF, malformed/wrapping sizes, real packet limits and actual newline parsing callers remain open.

TEXT-003 — String writer lengths, supplied copy and byte requests (RE-379)

The writer first handles a null source pointer as one eight-bit request for a file-backed zero byte. MSG_WriteString otherwise requests strlen, compares its low word as signed against 1023, and requests Q_strncpyz with destination the supplied stack buffer, source pointer and capacity 1024. WriteBigString uses threshold 8191 and capacity 8192. Actual file offsets and zero-byte values are retained in the report.

Lengths zero/limit minus one request copying. Lengths at/above the limit request Com_Printf, then the empty-byte path. The logger remains unexecuted. Word probes 0xffffffff and 0x80000000 satisfy the signed comparison and request copying; they are deliberately supplied library-return boundaries, not evidence of legal string sizes or actual huge allocations. Only copy request arguments are evaluated for these boundaries.

Small valid fixtures supply strlen and copy/NUL effects explicitly, then evaluate the native stack-byte sanitation and emission instructions. Writer sanitation replaces bytes above 127 with period; percent is retained. The source remains unchanged because sanitation acts on the supplied copied stack bytes. Every sanitized byte plus the final NUL requests MSG_WriteBits(message, byte, eight). The requests compose actual OOB stores, including the underlying space guard. As with WriteData, emission continues after child overflow, so the stored prefix can omit the final NUL when capacity is insufficient.

Each image has sixty compositions across the two writers: null/empty/percent-and-high-byte inputs, alternating ASCII/high-byte lengths 1–9/17/31, capacities four/64, and explicit over-limit supplied strlen cases that evaluate logging-request plus empty-byte emission. For the latter, backing source bytes are deliberately empty while the supplied strlen word is limit/limit plus one; these are control-flow probes, not native strlen conformance.

The normal reader and writer differ: the writer retains percent, while the reader replaces it. A percent-containing string therefore need not round-trip byte-for-byte through these recovered paths. Complete maximum-length emission, real Q_strncpyz/strlen implementations, source/destination aliasing, nonterminated source memory, compressed strings, overflow recovery and game caller text conventions remain open.

Validation and next work

Eight independent tests verify actual data-read requests/cursors and EOF prefixes; continuation/unchanged tails/canaries; string termination/sanitation; full capacity/return-pointer behavior; percent/high-byte/CR/newline cases; writer copied-byte/NUL/capacity effects; signed length guards/file-backed empty bytes; and paired engine outcomes. No evaluator extension was needed.

RE-380: compose compressed scalar/data/string codecs and real library-copy/length behavior; recover string caller buffer lifetime/limits, alias/nonterminated/overflow/allowoverflow/reset/truncation/cursor invariants, float/angle/delta/entity/player/snapshot codecs, native full-seed codewords and actual packet/runtime/all-platform/renderer contracts.

Follow-up: numeric message codecs recovers bounded float and finite angle wrappers. Broader RE-380 remains open; RE-384 tracks delta/angle callers and complete numeric/packet/platform contracts.