graph-reasoner-typed (frontal-labs)

A graph reasoner that answers multi-hop questions over typed graphs from the question's type sequence, never from relation names. It is the default model of the Graph Reasoning Foundation Model family, and the one that generalizes to unseen ontologies.

A variant, graph-reasoner-typed-compositional, trades unseen-ontology accuracy for much better accuracy on new combinations of question types.

How it works

1. A frozen sentence encoder (sentence-transformers/all-mpnet-base-v2) embeds the question.
2. A small learned scorer predicts the hop count, and the node type at each hop.
3. Every path from the start entities (up to 4 hops) is scored by how well its node types match. Types are scored by name embedding, so type names never seen in training work as long as the name carries meaning. Relation names and identities are never read.
4. A learned classifier over the readout's own scores decides whether to answer or abstain. Confidence is isotonic-calibrated per predicted hop count.

The scorer and the abstain head are trained on the training split only. The answer threshold and calibration are fit on the in-distribution validation split. Held-out splits are never read during training.

Why

The unseen-ontology holdout renames every relation to an opaque label. The earlier neural models matched questions to relation names, so they had nothing to match there and scored 0.12–0.22. A type-sequence oracle that follows only node types reaches 0.99 exact match on that split, so the information is present in the input. This model learns to use it.

Results (synthetic corpus v4, 800 examples per split, full-split exact match)

Split	Exact match	F1	Calibration error (ECE)	p95 latency (A100)
In-distribution test	0.838	0.882	0.099	29 ms
Unseen ontology (renamed relations + held-out wording)	0.766	0.805	0.244	27 ms
Unseen topology	0.860	0.893	0.091	24 ms
Unseen entities	0.854	0.900	0.095	28 ms
Unseen path distribution	0.830	0.868	0.115	44 ms
Unseen composition	0.334	0.423	0.529	28 ms

Plan targets: suite exact match 0.516, unseen-ontology exact match 0.407, calibrated ECE 0.10. For comparison, the best neural model in the family (graph-300M) scores 0.46 in-distribution and 0.15 on unseen ontologies.

Seed spread: three training seeds give 0.828–0.838 in-distribution and 0.749–0.767 on unseen ontologies.

Limitations

• Unseen composition is weak (0.33). None of its gold type sequences appear in training; this scorer learns whole type sequences, so it does not generalize to new chains of types. Use the compositional variant for that (0.57).
• Overconfident under wording shift. Calibration error is 0.24 on unseen ontologies and 0.53 on unseen composition; confidence is reliable in-distribution only (3-hop slice 0.137). Fitting the abstain decision on internally held-out intents and templates was tried: it made the model abstain on up to 71% of answerable unseen-ontology questions (exact match 0.27), so it is not used.
• Abstention under shift is weak. On unseen ontologies it abstains on 19% of unanswerable questions.
• Depends on meaningful node-type names. If types are renamed to opaque labels, the type signal is gone.
• Exhaustive path enumeration, capped at 200,000 paths per query.
• Evaluated on synthetic graphs only.

Usage

pip install torch pyyaml sentence-transformers
python example.py

from graph.typed_readout import TypedReadout
readout = TypedReadout.from_checkpoint("readout.pt", device="cuda")
result = readout.reason(graph_store, query)   # graph.graph.GraphStore, graph.types.GraphQuery

Files

• readout.pt: scorer, abstain head, threshold and calibration (4 MB; loads with weights_only=True).
• code/: the graph package, scripts/train_typed_readout.py and scripts/evaluate_typed_readout.py.
• evals/: full evaluator output for this model and three training seeds.
• example.py: loads the model and answers one synthetic question.