UBC-SLIME/sparcol-lexicon-large-c2000

SPARCOL-Lexicon, ModernBERT-large, 2000 clusters

Final saved checkpoint for this run of the lexicon model accompanying Learned Sparse Lexical Representations for Large-scale Authorship Attribution.

Use the code in lingjzhu/neural_lexical. The output has one vector per input token with 2000 dimensions. Token cluster assignments are learned during training and included in cluster_ids.pt. Load them along with the weights; do not run k-means again for inference. This checkpoint uses log1p(ReLU) token activations, without CLS/SEP tokens or embedding rescaling. Score with mean MaxSim over nonzero query tokens.

Usage

Clone the code repository and install its dependencies, then run from its root:

from src.models.ClusteredTransformer import ClusteredTransformer, clustered_maxsim
from src.models.SparseEncoder import SparseEncoder
import torch

module = ClusteredTransformer.load("UBC-SLIME/sparcol-lexicon-large-c2000")
model = SparseEncoder(modules=[module]).to("cuda").eval()
features = model.tokenize(["A sample document.", "Another document."])
features = {k: v.to("cuda") for k, v in features.items()}
with torch.inference_mode(), torch.autocast("cuda", dtype=torch.bfloat16):
    embeddings = model(features)["sparse_embeddings"]
    scores = clustered_maxsim(embeddings, embeddings)

Checkpoint provenance

• Source: exp2/outputs_clustered_colbert_modernbert_large_2000_relu_scale1_nocls/checkpoint-1580
• Training step: 1580
• Epoch: 2.0
• Backbone: answerdotai/ModernBERT-large
• Maximum sequence length: 512

Weights and cluster assignments are preserved from the source checkpoint. Packaging adds portable module configuration and removes training logs and training-data examples. Full HRS evaluation has not been rerun for this release; no new benchmark results are claimed.