docs/ds41-dsml-function-map.md

DS41-DSML function map

One row per function and per named constant of the DeepSeek-V4.1 encoding oracle, then one row per DSML function of ours that the oracle does not define. This is the review list: what the oracle decides, where this repo implements it, which case proves it, and what is still open.

The oracle is the official V4.1 text/vision encoding implementation, operator-supplied 2026-10-18. It is the authority for the carriage this project serves; the DSML line serves no other markup for the V4.1 family. It is vendored byte-identical at tests/ds41_ref/oracle/encoding.py (sha256 13231c592e958fb39e2af6001dae69d90e7a36b265702b9b390236fb387599e2, provenance in tests/ds41_ref/oracle/README.md) and pinned by that hash in tests/ds41_ref/ref_drive.py:46, so a later edit of it fails the run loudly. Line numbers below are into that vendored file.

How to read a row

• Oracle: the function or constant, with its line in the oracle file.
• Decides: the one thing it fixes about the bytes.
• Ours: the counterpart in this repo, file:line. "not located in this pass" means this session did not find a counterpart; it is a gap, not a claim that none exists.
• Case: the reference case that exercises it, under tests/ds41_ref/, with its fixture under crates/ds4-server/tests/fixtures/ds41-ref/.
• Verdict: pass (measured byte-identical to the oracle's fixture today), OPEN (a case exists or is owed and the bytes differ), no case (no case yet; nothing measured), harness (test-side only, not a served rule), out of surface (defined by the oracle but outside the OpenAI chat wire).

Verdicts are measured, not assumed: pass for the encode side means our prompt bytes equal the oracle's prompt bytes for that case (crates/ds4-server/tests/ref_encoding.rs), and for the decode side that our parse yields the oracle's call names and argument values (crates/ds4-server/tests/ref_decode.rs).

Every file:line in this document was re-verified against the working tree on 2026-10-20 (each symbol grepped in its file); the earlier revision of this map had been written against older files and carried a uniform 40-line offset on render.rs and a varying one on parse.rs.

1. Encode side — request to prompt

Oracle	Decides	Ours	Case	Verdict
to_json (76)	the JSON literal shape for non-string values: ensure_ascii=False, python separators	crates/ds4-server/src/render.rs:336 pyjson_write, :329 pyjson_raw_value, crates/ds4-server/src/json.rs:379 json_minify_raw_value	enc-argtypes-chat, dec-unicode	pass
tools_from_openai_format (84)	the schema string handed to the model, in the client's declared order	crates/ds4-server/src/parse.rs:2405 parse_tools_value, :2011 openai_function_schema_from_tool, :2221 tool_schema_orders_add_json_wire	enc-tools-chat	pass
_split_tool_name (101)	the namespace::name split, and the two asserts around it	not implemented on the chat surface: a tool-level namespace key is not read there and a ::-qualified name is taken as given (crates/ds4-server/src/parse.rs:2405); the Responses surface keeps a namespace for type:"namespace" items (:2282 responses_namespace_function_schema, :2336 append_responses_namespace_tool_schemas)	tests/parse.rs chat_tool_names_pass_through_verbatim, responses_namespace_tools_keep_both_spellings	out of surface on the chat wire the oracle decides: the client's model-facing tool spec is name/description/parameters only (conversation.rs:861-869, From<ToolDefinition> for ToolSpec :1039-1045), and its namespace rides the editor-facing ToolCall meta instead (acp_session.rs:1741 asserts the meta, not the wire), so no chat request can carry the field; the oracle's namespace::name qualification belongs to the message dicts it is handed
_tool_name_for_encoding (116)	the name printed in the invoke tag, namespace-qualified	crates/ds4-server/src/render.rs:576 append_dsml_tool_calls_text	enc-argtypes-chat (bare name)	pass for the bare name; the qualified form is out of surface with the row above
tool_calls_from_openai_format (124)	a replayed assistant's tool_calls array into the internal call shape	crates/ds4-server/src/parse.rs:1079 parse_tool_calls_value, :1023 add_tool_call_id	enc-argtypes-chat, enc-args-double	pass
decode_dsml_to_arguments (156)	the arguments JSON string: string="true" wraps the body as a JSON string, string="false" takes the body as the value	crates/ds4-server/src/tools.rs:376 parse_dsml_generated, :1262 json_value_is_complete, crates/ds4-server/src/json.rs:379	dec-one-call, dec-complex-args	pass on names and values; our bytes are minified where the oracle keeps the body verbatim (matrix O-04)
merge_tool_messages (184)	a tool role becomes a tool_result block inside a user message, merging into an adjacent merged user	crates/ds4-server/src/parse.rs:1330 parse_messages, :1411 append_tool_result_text; crates/ds4-server/src/render.rs:1484	enc-argtypes-chat	pass
sort_tool_results_by_call_order (233)	tool_result blocks inside one user message are ordered by the preceding assistant's call order	crates/ds4-server/src/render.rs:1442 tool_run_in_call_order with :1421 preceding_call_order	enc-toolresults-order	pass (closed 2026-10-18)
parse_tagged_text (273)	the <image>path</image> text form expands to image blocks, and a malformed tag raises	not implemented, by contract: pixels arrive base64-only (AGENTS.md), so the tag is never markup	tests/parse.rs the_image_path_tag_form_is_text_and_not_markup	out of surface, measured 2026-10-20: no image is registered, the content keeps the tag, and the rendered prompt carries it verbatim
_is_image_block (300)	which content blocks count as images (image, image_url)	crates/ds4-server/src/parse.rs:1134 parse_chat_image_url, :456 add_image_data_uri	enc-image-chat	case added 2026-10-20: the placeholder and its position agree with the oracle, the block separator does not (O-06)
_extract_image (305)	the url/data/source normalization, and the refusal when none is present	crates/ds4-server/src/parse.rs:415 add_image_base64, :456	enc-image-chat (the oracle's own record), tests/parse.rs image_refusals_cover_the_oracle_path_and_url_forms	the oracle's path and URL forms are refused by name; the data-URI form is measured by enc-image-chat
_process_image_blocks (323)	images are replaced by <｜deepseek_image｜> in place, nested tool-result images included, and the records keep prompt order	crates/ds4-server/src/render.rs:1388 append_dsml_user_content, crates/ds4-server/src/parse.rs:638 write_tool_image_placeholders	enc-image-chat; tests/parse.rs openai_tool_message_carries_the_image_placeholder; tests/render.rs dsml_v41_places_tool_result_images_inside_the_envelope	placeholder and order agree; the separator does not (O-06)
_validate_no_image_sp_tokens (355)	a user-supplied image placeholder in content or reasoning_content is refused	crates/ds4-server/src/parse.rs:567 reject_stray_image_in_content, :578 reject_stray_image_in_reasoning, called from every place the client's own text is read (message content, text blocks, a tool result's text, reasoning_content, the Responses text blocks)	stray_image_token.rs (9) and the served probes …/probe-placeholder-04a0ab1f-20261019	CLOSED at R7 (04a0ab1f): the oracle's two sentences on the wire; the engine does not implement the rule (core_v41_vision.c:189-193, server_parse_tools.c:290), so the port mirrors the oracle, CTR 8.9
process_image_messages (370)	the per-message image pass, and content rebuilt from text blocks with \n\n	crates/ds4-server/src/parse.rs:1167 parse_openai_message_content, :587 append_text_part	enc-blocks-chat, enc-image-chat	O-06 (measured 2026-10-20): the pieces and their order agree, the rebuild's \n\n does not
_read_until_stop (399)	the earliest-of-stops scan the whole parser is built on	crates/ds4-server/src/tools.rs:169 split_reasoning_content	every dec-* case	pass
find_last_user_index (562)	the last user-like index; a mid-conversation system message counts as one; a tool-result-only user message does not start a turn	crates/ds4-server/src/render.rs:1545 (the loop) with :263 role_is_user_like, :1050 (the drop rule's copy)	enc-plain-chat, enc-midsystem-chat, enc-toolresult-only	pass, both halves: mid-system closed 2026-10-18, the tool-result-only tail added 2026-10-19
render_reasoning_effort (456)	the effort header: thinking mode, index 0 only, low/high/max mapped to 50/75/100, an integer in [1,100] taken as itself, None to high	crates/ds4-server/src/render.rs:98 head_effort_prefix / :105 v41_effort_prefix with crates/ds4-server/src/route/mod.rs:383 RequestedEffort	enc-think-low, enc-think-max, enc-think-budget	pass (closed 2026-10-18)
render_tools (509)	the tools preamble verbatim, with the schema list appended	crates/ds4-server/src/render.rs:453 append_dsml_tools_prompt_text	enc-tools-chat	pass
encode_arguments_to_dsml (524)	one parameter element per argument key, in the call's key order, string="true" for a string child and its raw form otherwise; a non-dict arguments falls back to a single arguments key, a JSON string is decoded twice	crates/ds4-server/src/render.rs:390 append_dsml_arg, :416 append_dsml_arguments_from_json, and the retry at :629 with crates/ds4-server/src/json.rs:476 json_string_contents	enc-argtypes-chat, enc-args-double, enc-args-nondict, enc-args-plain	pass on all four, measured 2026-10-20; the stringified-arguments pass was added to the render that day and the C engine has no such pass (ds4_server.c:3443-3463), which enc-args-double pins
render_message (585)	one message's bytes, the role branches, the transition tokens, and the assistant header rule	crates/ds4-server/src/render.rs:1484 render_dsml_chat_choice_ex	19 encode cases	pass on every case measured; O-01 and O-02/O-03 closed 2026-10-18, the consecutive-assistant half of the header rule was found open and closed 2026-10-19 (enc-think-two-assistants, below)
_drop_thinking_messages (749)	before the last user, an assistant's reasoning_content is dropped, the message kept	crates/ds4-server/src/render.rs:1634 keep_reasoning (the rule inline in the assistant branch rather than as a message pass)	enc-think-drop-thinking, enc-drop-thinking, enc-think-tail-reasoning	pass (closed 2026-10-19): the drop fires only in thinking mode with no tools anywhere, and the chat-mode case pins that the reasoning cannot leak
_encode_messages_text (770)	BOS, the loop over messages, and the drop rule forced false when any message carries tools	crates/ds4-server/src/render.rs:1484 (the entry) with its message loop at :1543	all encode cases	pass
encode_messages (810)	the public entry: merge, sort, image pass, then the text render	crates/ds4-server/src/render.rs:1484 behind the chat route	all encode cases	pass on the cases measured
load_cases (847)	the harness entry: a case-level tools key moves into messages[0], a bare message list is wrapped	tests/ds41_ref/ref_drive.py:68	every encode case	harness
encode_case (872)	the harness entry: encode one case and return its images, case thinking_mode winning over the command line	tests/ds41_ref/ref_drive.py:70	every encode case; :83 for the media driver	harness

2. Decode side — completion to assistant message

Oracle	Decides	Ours	Case	Verdict
parse_tool_calls (890)	the invoke/parameter grammar: >\n after every open tag, one name per invoke, the type attribute, duplicate parameters refused	crates/ds4-server/src/tools.rs:376 parse_dsml_generated	dec-one-call, dec-two-calls, dec-complex-args, dec-unicode, dec-marker-in-value	pass
parse_message_from_completion_text (946)	the thinking split at </think>, content up to EOS or the block start, no content after the block, no special token left in content or reasoning	crates/ds4-server/src/tools.rs:376, :169 split_reasoning_content	every dec-* case, incl. dec-thinking	pass
tool_calls_to_openai_format (139)	the wire shape: type: "function", function.name, function.arguments, namespace when present	crates/ds4-server/src/stream.rs:173 append_tool_calls_json	every dec-* case (the fixture is the oracle's parse)	pass on names and values

3. Constants and templates

Every literal the two sides must agree on, pinned against the oracle's own source by crates/ds4-server/tests/ref_constants.rs (added 2026-10-20): the test extracts the value the oracle assigns and compares it with ours, so a row below is measured even when no case happens to carry the literal. Where a row says "other families", the constant sits beside ours in the same table but belongs to a model family outside this line.

Oracle	Ours	Verdict
bos_token (29)	crates/ds4-server/src/render.rs:118 DSML_BOS	pass
eos_token (30)	:121 DSML_EOS	pass
thinking_start_token (31)	:126 DSML_THINK_START	pass
thinking_end_token (32)	:127 DSML_THINK_END	pass
dsml_token (33) ｜DSML｜	folded into the tags below	pass
USER_SP_TOKEN (35)	:119 DSML_USER	pass
ASSISTANT_SP_TOKEN (36)	:120 DSML_ASSISTANT	pass
SYSTEM_SP_TOKEN (426)	:78 DSML_SYSTEM_TOKEN	pass
LATEST_REMINDER_SP_TOKEN (37)	not located	out of surface: the role is not on the OpenAI wire
IMAGE_PLACEHOLDER (39)	:132 DSML_IMAGE	pass (pinned by the constants leg and carried by enc-image-chat); the refusal beside it is measured by stray_image_token.rs
IMAGE_TAG_PATTERN (40)	not located	out of surface with the parse_tagged_text row above
DS_TASK_SP_TOKENS / VALID_TASKS (43, 51)	not located	out of surface: internal classification tasks, not on the OpenAI wire
assistant_msg_template / wo_eos (60, 61)	crates/ds4-server/src/render.rs:1655 (EOS always written)	the template body is pinned by the constants leg; out of surface for wo_eos: no OpenAI request field reaches it (wo_eos appears nowhere under crates/), while the oracle reads it from the message dict for its own continuation flow (encoding.py:607, :697-703)
response_format_template (64)	crates/ds4-server/src/format.rs:19 parse_output_format_value, called from crates/ds4-server/src/parse.rs:3345, :3515, :3678; no render-side counterpart	measured refusal, kept by decision: nothing is served that is not supported (see section 5)
tool_output_template (68)	crates/ds4-server/src/render.rs:1368 append_tool_result_text, written at :1619/:1621	pass via enc-argtypes-chat
tool_calls_block_name (428) " calls"	crates/ds4-server/src/tools.rs:44 DSML_TOOL_CALLS_START_SPACED	pass
tool_call_tag_name (429) " invoke"	:46 DSML_INVOKE_START_SPACED, :47 end	pass
tool_parameter_tag_name (430) " parameter"	:48 DSML_PARAM_START_SPACED, :49 end	pass
tool_call_template / tool_calls_template (432, 435)	crates/ds4-server/src/render.rs:576-:652	pass
REASONING_EFFORT_TEMPLATE (443)	the line shape	crates/ds4-server/src/render.rs:44 V41_EFFORT_TEMPLATE with the 75/100 constants at :36/:38
REASONING_EFFORT_MAPPINGS / DEFAULT_REASONING_EFFORT (448, 453)	low 50, high 75, max 100, None to high	crates/ds4-server/src/route/mod.rs:368 mode_budget / :395 RequestedEffort::head_budget

4. Ours with no oracle counterpart

Functions of this repo's DSML line that the oracle does not define. They are kept because a served turn needs them, not because the oracle asks for them.

Ours	Why it exists	Covered by
crates/ds4-server/src/tools.rs:2157 find_tool_start (with crates/ds4-server/src/dsml.rs:389 dsml_find_tool_start)	the streaming tracker: the artifact's root spelling (quoted, unquoted, short) is not the oracle's canonical one, so the scan is tolerant where the oracle's grammar refuses	tests/dsml.rs syntax_greedy_spaced_dsml; dsml.rs override_tests::auto_tool_start_forces_function_prefix, ::required_prefix_finishes_after_tool_start
crates/ds4-server/src/dsml.rs:550 dsml_decode_state_for_text	the incremental decode state machine that decides when markup is withheld from the stream	tests/dsml.rs syntax_greedy_spaced_dsml; generate.rs disk_sync_tests::continued_decode_gate_stops_after_tool_syntax_begins
crates/ds4-server/src/tools.rs DSML_*_SHORT, DSML_* (25-36)	the two older root spellings the artifact still emits; the oracle knows only the spaced one	tests/dsml.rs syntax_greedy_spaced_dsml plus the dec-* cases through ref_decode
crates/ds4-server/src/parse.rs:2282 responses_namespace_function_schema, :2336 append_responses_namespace_tool_schemas	the Responses type:"namespace" item: the prompt advertises namespace + function and the order keeps both spellings so the answer can be emitted under the client's own name (crates/ds4-server/src/stream.rs:234 responses_tool_wire_name, :287 the namespace field)	tests/parse.rs responses_namespace_tools_keep_both_spellings
crates/ds4-server/src/retry.rs:144 try_repair_dsml	truncated-call repair; the oracle has no repair path (matrix G8, P3)	tests/retry.rs dsml_repair_matches_the_reference_balance_contract, repair_and_decide_match_c
crates/ds4-server/src/render.rs:1681 responses_visible_assistant_suffix	the visible assistant tail a Responses client can replay (ds4_server.c:12821); the store that consumes it (responses_live_remember :14559) is the reserved checkpoint unit	tests/render.rs responses_visible_suffix_matches_client_replay
crates/ds4-server/src/trace.rs	C --trace FILE (ds4_help.c:323): one section per request with the prompt, its token ids and the parsed message (unit 3.2, TP-12/TP-15)	trace.rs tests::a_section_carries_the_prompt_tokens_and_parsed_message
crates/ds4-server/src/retry.rs:260 find_any_tool_start, :277 find_last_structural	the repair path's style choice and think scope (ds4_server.c:5842, :5890), ported with G8	tests/retry.rs repair_and_decide_match_c
crates/ds4-server/src/render.rs:133-:179, other-family markers	Motif, EXAONE, Solar, Qwen, GLM, K2 tag sets: other families, outside this line	tests/render.rs glm53_chat_and_image_protocol_match_official_template, ling31_uses_bailing_syntax; tests/parse.rs qwen_image_inputs_normalize_across_all_three_surfaces
crates/ds4-server/src/render.rs:202 syntax_for_model_id	family dispatch; the oracle covers V4.1 only	tests/render.rs syntax_for_model_id_matches_c

5. The open rows

Measured on 2026-10-20, in order of what decides them. Every row above now carries a verdict; these are the ones that needed a judgement rather than a fixture.

1. O-06, the block separator (rows process_image_messages, _process_image_blocks, _is_image_block, process_image_messages): the oracle joins the pieces a client sent as separate content blocks with a blank line -- _process_image_blocks turns an image into a <｜deepseek_image｜> text block and the user branch writes "\n\n".join(parts) (encoding.py:323-353, :631-656, and the content rebuild at :386-393) -- while this port concatenates them: at parse time for text pieces (append_text_part, parse.rs:587) and at render time for the placeholder (append_dsml_user_content, render.rs:1388). The C lineage concatenates too (ds4_server.c:5865), so the port inherited it. Measured the same day on enc-image-chat and enc-blocks-chat: the two prompts carry the same pieces in the same order and differ by that one separator, which is exactly what those two cases pin (check_block_join). A third place shares the rule and has no case yet: a tool-result message's mixed blocks (write_tool_image_placeholders, parse.rs:638), asserted today as "shot:<placeholder>done" by tests/parse.rs openai_tool_message_carries_the_image_placeholder. The fix is one join, gated to the V4.1 head, in the render's parts loop plus the parse's text pieces; it is the operator's call because it moves every image-bearing prompt and every multi-block message on that head.
2. The image surface's oracle leg (rows parse_tagged_text, _is_image_block, _extract_image, _process_image_blocks, _validate_no_image_sp_tokens, process_image_messages): ref_drive.py refused image cases by construction (assert not images, kept for the text-only legs), so the leg was rebuilt on 2026-10-20 as encode-media (ref_drive.py:75), which prints the prompt and reports the oracle's own image records on stderr. _validate_no_image_sp_tokens is CLOSED at R7; the other five now have the oracle's bytes, with O-06 the only difference.
3. The namespace surface (rows _split_tool_name, _tool_name_for_encoding): out of surface. The client's model-facing tool spec is name/description/parameters only (conversation.rs:861-869, the conversion at :1039-1045) and its namespace rides the editor-facing ToolCall meta (acp_session.rs:1741), so no chat request can carry the field and a ::-qualified name is never sent. Both halves are pinned by tests/parse.rs (chat_tool_names_pass_through_verbatim, responses_namespace_tools_keep_both_spellings).
4. wo_eos (assistant_msg_template / wo_eos): out of surface. wo_eos appears nowhere under crates/; the oracle reads it from the message dict for its own continuation flow (encoding.py:607, :697-703).
5. IMAGE_TAG_PATTERN, the <image>path</image> text form, and LATEST_REMINDER_SP_TOKEN / DS_TASK_SP_TOKENS / VALID_TASKS: out of surface. The carriage is base64-only by design (AGENTS.md) and the rest are internal classification roles, not on the OpenAI wire. The tag form is measured rather than assumed (tests/parse.rs the_image_path_tag_form_is_text_and_not_markup).
6. response_format rendering (row response_format_template): the one row whose verdict is a decision and not a measurement. Ours refuses json_object and json_schema with 400 "structured output is unsupported" (format.rs:6-15, the v0.6.3 refusal of ds4-on-spark#10), measured on the served binary today; the oracle renders the schema into the prompt instead (encoding.py:628-629 with the template at :64). The client of record can put response_format: json_schema on the chat wire, from ConversationRequest.json_schema (conversation.rs:4107), at two production sites -- the auto-mode permission classifier (sampler_turn.rs:862) and the goal evaluator (goal_evaluator.rs:172) -- while its main turns hardcode None (request_builder.rs:148, sampler client.rs:4071), and no capability gate guards those two sites. The operator decided 2026-10-20 that the refusal stands: the field carries strict: true, i.e. a guarantee that the served text conforms to the schema, and only a decode-time constraint can give that guarantee. A prompt-side instruction would be an invented approximation of a guarantee, so nothing is served for this field, and the client's own error branch records the side query as failed (session_summary.rs:131-137 for the neighbouring title query). Serving it for real means a JSON-schema-constrained decoder, a unit of its own.
7. O-04 arguments bytes: ours minify, the oracle keeps the parameter body verbatim. Values agree; the strings differ. Recorded, not a defect against the wire, since the client echoes our string.

Closed since the map was written: O-01 (the effort budget, both halves), O-02 (a mid-conversation system message renders in place), O-03 (tool results ordered by call order), the repair rows of G8/G9, the image-placeholder refusal (R7), the drop-thinking rule together with its chat-mode counterpart, the stringified-arguments fallback (2026-10-20, enc-args-double), and the find_last_user_index tool-result-only half (enc-toolresult-only).

One divergence was found by the 2026-10-19 pass and closed in code. Two assistant messages in a row share one <Assistant> header, so the second has no header to open its thinking, and the oracle still writes that message's reasoning and its closer as part of the body (encoding.py:678-712). Measured before the fix: ours rendered 275 bytes against the oracle's 304, dropping SECOND-THOUGHT-MARKER whole (enc-think-two-assistants). The assistant branch now writes the thinking body when it has no header, gated to the V4.1 head so the port's own head keeps its bytes.

One divergence was found by the 2026-10-20 pass and closed in code. A replayed call whose arguments is itself a JSON string still renders that object's parameters, because the oracle decodes the field twice before it gives up (encoding.py:528-537). Measured before the fix: ours wrote one arguments parameter carrying the quoted text (1534 bytes against the oracle's 1511, enc-args-double). append_dsml_tool_calls_text now retries the parse on the unwrapped string (json_string_contents, json.rs:476), and the C engine's lack of that pass is pinned by the same case.

6. Reproducing

# the oracle's own bytes for a case, and its parse of a completion
python3 tests/ds41_ref/ref_drive.py encode       tests/ds41_ref/cases/enc-plain-chat.json chat
python3 tests/ds41_ref/ref_drive.py encode-media tests/ds41_ref/cases/enc-image-chat.json chat
python3 tests/ds41_ref/ref_drive.py parse        tests/ds41_ref/decode/dec-one-call.txt chat

# the three reference legs against the pinned fixtures
cargo test -p ds4-server --test ref_encoding
cargo test -p ds4-server --test ref_decode
cargo test -p ds4-server --test ref_constants

Regenerate the fixtures only when the oracle or a case changes: bash tests/ds41_ref/regen_fixtures.sh.