graph-reasoner-typed-compositional (frontal-labs)

A variant of graph-reasoner-typed that answers questions combining question types in ways never seen in training. Like the default model, it never reads relation names.

Use the default model unless your questions combine relations in new ways. This variant is much better on new combinations (0.57 against 0.33), but worse on new wording of known question types (0.47 against 0.77).

How it differs from the default model

It has two scorers and a learned gate that picks one per question:

• Type-sequence scorer (as in the default model): reads the whole question and predicts the type at each hop. Robust to new wording, but it learns whole type sequences, so it fails on new combinations.
• Type-bag scorer: reads the question token by token. Each token names a node type or nothing, and a path is scored by how likely its set of node types is under those token probabilities, plus the answer type. Order is left to the graph. Because a word such as "supplier" names the same type in every question, this composes.
• Gate: chooses the scorer per question from how much each scorer dislikes the other's answer, hop-count certainty, similarity to known question templates, and whether they agree. It was trained only on internally held-out intents and templates from the training split.

The answer/abstain decision reads the sequence scorer's scores, because those are the ones that transfer under wording shift.

Results (synthetic corpus v4, 800 examples per split, full-split exact match)

Split	This variant	Default model
In-distribution test	0.836	0.838
Unseen composition	0.574	0.334
Unseen ontology	0.469	0.766
Unseen topology	0.858	0.860
Unseen entities	0.851	0.854
Unseen path distribution	0.828	0.830

Calibration error: 0.103 in-distribution, 0.328 on unseen composition, 0.503 on unseen ontology. p95 latency on an A100: 39–83 ms.

On held-out training intents (model selection data), the bag scorer reaches 0.65 against 0.01 for the sequence scorer.

Limitations

• Worse on new wording of known question types. The gate trusts the bag scorer more often than it should under paraphrase, because the internal held-out templates used to train it are milder than real paraphrase. Adding each scorer's best score to the gate did not fix this.
• Overconfident under shift (calibration error 0.33–0.50).
• Depends on meaningful node-type names; evaluated on synthetic graphs only.

Usage

from graph.typed_readout import TypedReadout
readout = TypedReadout.from_checkpoint("readout.pt", device="cuda")
result = readout.reason(graph_store, query)

Files

• readout.pt: both scorers, gate, abstain head, threshold and calibration (12 MB; loads with weights_only=True).
• code/: the graph package and the training and evaluation scripts. Trained with scripts/train_typed_readout.py --bag --gate.
• evals/: full evaluator output.
• example.py: loads the model and answers one synthetic question.