Suyash2205/signbridge-t5-base-asl-gloss

SignBridge — English to ASL Gloss (T5-base)

Translates written English into American Sign Language gloss: the ordered sequence of signs used to express a sentence, written in capitals.

"The doctor will help you now"  →  DOCTOR WILL HELP X-YOU DESC-NOW .

This is Stage 1 of SignBridge, a deployed system that renders the result as video of a person signing.

Usage

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

model_id = "Suyash2205/signbridge-t5-base-asl-gloss"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSeq2SeqLM.from_pretrained(model_id)

inputs = tok("translate English to ASL gloss: the doctor will help you now",
             return_tensors="pt", max_length=64, truncation=True)
out = model.generate(**inputs, max_length=64, num_beams=4)
print(tok.decode(out[0], skip_special_tokens=True))
# DOCTOR WILL HELP X-YOU DESC-NOW .

The task prefix translate English to ASL gloss: is required — it was used throughout fine-tuning.

Results

Evaluated on the held-out ASLG-PC12 test split (8,771 sentences).

System	BLEU-4	ROUGE-L	Exact match	Token acc.
Rule baseline	26.76	0.7715	3.4%	20.2%
T5-small (60M)	87.86	0.9666	7.2%	56.1%
T5-base (this model)	95.47	0.9927	82.5%	90.7%

Leakage-controlled evaluation

A BLEU of 95 is higher than published work on this corpus, which warranted a check rather than celebration. 764 of 8,771 test sentences (8.7%) appear verbatim in the training split. Separating them:

Partition	Share	BLEU-4	Exact match
Seen in training	8.7%	99.42	97.75%
Never seen	91.3%	95.13	80.50%

Report 95.13 as this model's score. The headline figure survives the control, but the unseen partition is the honest one.

Training

Base	google-t5/t5-base (220M)
Data	ASLG-PC12 — 70,164 train / 8,770 val / 8,771 test
Epochs	6 (converged by epoch 4)
Batch size	64
Max source/target length	64
Precision	fp16
Hardware	1× NVIDIA T4
Wall-clock	~75 minutes
Compute cost	USD 2.38

Sequence length is capped at 64 rather than T5's default 512. Corpus sentences are far shorter than that, and the default spends roughly eight times the necessary compute on padding.

Important limitation: this model does not produce ASL word order

ASLG-PC12's gloss targets retain English constituent order. The reference for "The doctor will see you tomorrow" is DOCTOR WILL SEE X-YOU TOMORROW — the temporal adverbial stays final and the auxiliary survives. Authentic ASL fronts TOMORROW and drops WILL.

The corpus is therefore not merely silent on syntax but anti-informative for it: a model producing correct ASL ordering disagrees with the reference and scores worse. This model is faithful to the corpus, so its BLEU measures morphological and lexical fidelity, not signing fluency.

In the full SignBridge system, word order is applied by rule after this model runs. Anyone using this checkpoint for actual ASL output needs an equivalent step.

Output format

Output follows ASLG-PC12 annotation conventions:

• X- marks pronouns — X-YOU, X-WE
• DESC- marks descriptive forms — DESC-NOW, DESC-LARGEST
• Punctuation is a separate token

Measured over 20,000 training glosses, 20.4% of tokens carry a prefix and 10.9% are punctuation. Strip both before any dictionary lookup: doing so raises the share of tokens matchable against a sign inventory from 67.5% to 99.1%.

Other limitations

• Lexical drift on rarer words. The model occasionally emits a near neighbour (BROR for brother, MOR for mother). The full system repairs these by aligning against the source sentence.
• Corpus artificiality. ASLG-PC12 glosses are rule-derived and more regular than natural signing, so test performance is an upper bound on real input.
• No non-manual markers. Gloss cannot express the eyebrow, mouth and head movements that carry question formation and negation in ASL.

Citation

@misc{signbridge2026,
  title  = {SignBridge: English-to-ASL Generation using Neural Gloss
            Translation and Retrieval-Based Rendering},
  author = {Banerjee, Aneeket and Humne, Suyash},
  year   = {2026},
  note   = {K. J. Somaiya School of Engineering, Somaiya Vidyavihar University}
}

Built on ASLG-PC12 (Othman & Jemni, LREC 2012) and T5 (Raffel et al., JMLR 2020). The rendering half of the system uses WLASL (Li et al., WACV 2020).