Sparse Autoencoders Are a Shared Coordinate System for Symbolic Matching
Abstract
SAE-IDF matches symbolic units across heterogeneous systems using sparse autoencoders and rarity-weighted features, enabling scalable neuro-symbolic reasoning.
Large language models (LLMs) hold broad knowledge in one set of weights, but lack interpretability and trustworthy reasoning, especially in high-stakes use cases. A neuro-symbolic model, with its interpretable architecture and System 2 reasoning capacity, is a promising alternative if built at the scale of LLMs; however, its symbolic backbone is usually constructed with human effort, so existing neuro-symbolic models are bottlenecked by the scope of the symbol component and usually domain-specific. A data-driven method could instead build a larger symbolic system by aligning and merging existing ones. We target its first, foundational step: matching units across these heterogeneous systems. Our method, SAE-IDF, encodes a symbol in its native context into a sparse code with a sparse autoencoder (SAE), re-weights each dimension by its rarity and matches across systems. On 12 tasks across 6 kinds of symbolic system, SAE-IDF outperforms the same LLM's dense baseline on all tasks with Gemma-2-9B and all but one with Llama-3.1-8B, by up to 45 points, consistently across SAE layers and configurations. SAE-IDF is also interpretable: every dimension of the sparse code has a human-readable meaning, so each match decomposes into named concepts (features). Our work paves the way to a universal symbolic backbone.
Community
TL;DR: An LLM's sparse autoencoder works as a shared coordinate system for matching symbols across heterogeneous systems (formal provers, code, table schemas, knowledge graphs, ontologies), with no training and no aligned pairs.
How SAE-IDF works
- An LLM reads each symbol in its native context (e.g., a KG class with its instances, a Lean 4 statement).
- The SAE encodes the hidden states into a sparse code of named features.
- Each feature is weighted by its rarity (IDF, as in classic text retrieval), and candidates are ranked by cosine.
Results
- Beats the same LLM's dense hidden states on all 12 tasks with Gemma-2-9B, and on all but one with Llama-3.1-8B
- Biggest gains where notations differ most: Lean 4 → Metamath 0.21 → 0.66 MRR, Lean 4 → HOL Light 0.54 → 0.90
- Scales with the dictionary: HOL Light MRR rises from 0.82 to 0.94 as the SAE grows from 16k to 1M features
- Both ingredients matter: without IDF the SAE code mostly trails dense, and a random dictionary with the same IDF stays below dense on every task
Interpretable by construction: the cosine decomposes exactly over features, so every match reads as a few named concepts. For example, in NELL trainstation → DBpedia RailwayStation, 93% of the match comes from two features about railroads and stations.
A first step toward a universal symbolic backbone for neuro-symbolic models at LLM scale.
Get this paper in your agent:
hf papers read 2610.19636 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper
Collections including this paper 0
No Collection including this paper