Hugging Face's logo

fahiiim
/
TypoFormer

Text Generation
ONNX
English
onnxruntime
typoformer
token-classification
grammar-correction
autocomplete
typo-correction
character-aware
edge-ai
int8
rope
grouped-query-attention
swiglu
rmsnorm
offline-nlp
Model card Files
xet
 Community
TypoFormer / README.md
fahiiim's picture
fahiiim
Release TypoFormer Stage 2 INT8 with Model Architecture Diagram & Polished Model Card
05b316e verified
preview code
|
Raw History Blame Contribute Delete
14.3 kB
metadata 
language:
  - en
license: mit
library_name: onnxruntime
tags:
  - text-generation
  - token-classification
  - grammar-correction
  - autocomplete
  - typo-correction
  - character-aware
  - edge-ai
  - onnx
  - int8
  - rope
  - grouped-query-attention
  - swiglu
  - rmsnorm
  - offline-nlp
pipeline_tag: text-generation
inference: false
widget:
  - text: >-
      Could you please confirm if the proposed implementation timeline aligns
      with your
    example_title: Client Inquiry Completion
  - text: We decoupled the event processing pipeline to achieve sub-
    example_title: Engineering Autocomplete
  - text: We dont expect any downtime during the blue-green deployment phase.
    example_title: Grammar & Typo Correction

TypoFormer Banner

TypoFormer (Stage 2 INT8)

High-Precision Character-Aware Hybrid Transformer for Edge Autocomplete & Non-Autoregressive Grammar Correction
Created & Authored by Md. Fahim Sarker Mridul

GitHub Repository Author Model Version Parameters Latency Privacy License

Architecture | Highlights | Comparison | Benchmarks | Files | Quickstart | Windows Desktop | Citation

Overview

TypoFormer is a purpose-engineered, character-aware neural backbone designed from the ground up to solve two conflicting natural language problems inside a single unified 94M parameter trunk:

  1. Causal Next-Word Autocomplete: Low-latency autoregressive token prediction under strict sub-15ms local keystroke budgets without cloud connectivity.
  2. Non-Autoregressive Grammar & Typo Correction (GEC): High-precision edit-tag sequence labeling that repairs punctuation slips, confused vocabulary, subject-verb agreement, and severe typos without generative hallucination.

By fusing Byte-Level BPE subwords with Multi-Scale Character 1D-Convolutions through a learned gating highway, TypoFormer eliminates out-of-vocabulary token fractures and delivers enterprise-grade real-time writing assistance completely offline on standard desktop CPUs.

Model Architecture

TypoFormer Full Internal Neural Architecture Diagram

Figure 1: TypoFormer Full Internal Neural Architecture: Fusing byte-level subwords with multi-scale character convolutions, Pre-RMSNorm grouped-query attention (GQA), rotary embeddings (RoPE), SwiGLU feed-forward networks, and multi-task dual heads for offline autocomplete and grammar error correction (Visualized in NVIDIA Emerald Green #76B900).
[View High-Resolution 300 DPI Architecture Diagram (PNG)]

1. Dual-Channel Gated Input Highway

Standard tokenizers fail when words are corrupted: mistyping "definitely" as "definatly" converts a single semantic token into multiple rare subword fragments. TypoFormer prevents this tokenization distortion through a hybrid input representation:

  • Byte-Level BPE: Words are split into standard BPE tokens from a compact 16,000-vocabulary dictionary ($x_{\text{tok}} \to \mathbf{E}{\text{tok}} \in \mathbb{R}^{d{\text{model}}}$).
  • Multi-Scale Character-CNN: Concurrently, the exact UTF-8 character bytes of each token ($x_{\text{char}}$) are fed into parallel 1D convolutional banks with kernel sizes $k \in {2, 3, 5, 8}$: Ck=MaxPool(GELU(Conv1Dk(Bytes)))\mathbf{C}_k = \mathrm{MaxPool}\left(\mathrm{GELU}\left(\mathrm{Conv1D}_k(\text{Bytes})\right)\right)
  • Learned Gating Highway: h0=Etok+Ξ±β‹…Wproj([C2 βˆ₯ C3 βˆ₯ C5 βˆ₯ C8])\mathbf{h}_0 = \mathbf{E}_{\text{tok}} + \alpha \cdot \mathbf{W}_{\text{proj}}\left([\mathbf{C}_2 \,\|\, \mathbf{C}_3 \,\|\, \mathbf{C}_5 \,\|\, \mathbf{C}_8]\right) where $\alpha$ is a learnable scalar gate initialized to ensure robust character-level fallback when unknown or miscoded subwords appear.

2. Transformer Trunk (10 Blocks, $d=768$, $d_{\text{ffn}}=2048$)

  • Attention Mechanism: Grouped-Query Attention (GQA) with 12 Query heads and 4 Key-Value heads (3:1 ratio), cutting KV-cache memory traffic by 66% during sequence generation.
  • Rotary Position Embeddings (RoPE): Frequency-based relative query-key rotation for superior relative token distance awareness across cursor movements without absolute position limits.
  • Feed-Forward Backbone: SwiGLU non-linearity: SwiGLU(x)=W2(SiLU(W1x)βŠ™W3x)\text{SwiGLU}(x) = W_2\left(\text{SiLU}(W_1 x) \odot W_3 x\right)
  • Pre-Normalization: RMSNorm applied prior to each sub-layer for stable gradient propagation and reduced memory overhead.

3. Multi-Task Dual Output Heads

  • Task 1: Causal LM Autocomplete Head: Linear projection ($d_{\text{model}} \to 16{,}000$, weight-tied with token embeddings) yielding next-token probability distribution $P(w_{t+1} \mid w_{\le t})$.
  • Task 2: Keystroke GEC Edit Head: Linear tagging projection ($d_{\text{model}} \to 6{,}148$) mapping each token position to atomic edits:
    • KEEP (0): Token is grammatically and lexically correct.
    • DELETE (1): Redundant or erroneous token removal.
    • CASE_CAP (2) / CASE_LOWER (3): Casing adjustments.
    • REPLACE (4 .. 4099): Atomic swap with one of 4,096 high-frequency correction tokens.
    • APPEND (4100 .. 6147): Atomic right-insertion with one of 2,048 suffix/continuation tokens.

Key Highlights

  • Ultra-Low Keystroke Latency: Mean end-to-end CPU inference of 9.97 ms (P95: 12.12 ms), fitting comfortably inside the 16.6 ms 60 FPS display frame deadline.
  • Typo Immunity: Gated character-level convolutions preserve word roots even under severe phonetic and fat-finger keyboard slips.
  • 100% Air-Gapped and Offline: Zero network calls, zero telemetry, zero risk of leaking confidential documents, code, or private emails.
  • Guaranteed Content Fidelity: Non-autoregressive tagging guarantees zero generative hallucination and achieves 100.0% clean negative sentence preservation.
  • Compact Edge Footprint: Symmetric INT8 quantization packs the complete 94M dual-task architecture into just 130.5 MB ONNX graphs.

Architectural Comparison: TypoFormer vs. BERT vs. Cloud LLMs

Architectural Dimension Classical BERT-Base (110M) Cloud Generative LLMs (7B - 70B) TypoFormer (94M INT8) TypoFormer Advantage
Generative Autocomplete Incapable (Bidirectional MLM only) Yes (High quality autoregression) Native Causal Mode (Tied causal head) Unified Trunk: Single model serves both autocomplete & GEC.
Typo & OOV Robustness Brittle (teh to ['te', '##h']) Fair (Relies on large BPE dictionary) Gated Multi-Scale Char-CNN ($k=2,3,5,8$) Typo Immunity: Never blinded by unknown character corruptions.
Position Encoding Absolute Learned (Max 512 tokens) RoPE / ALiBi Rotary Position Embeddings (RoPE) Sequence Extrapolation: Relative distance aware across cursor hops.
Attention Mechanism Multi-Head (MHA) (12 Q, 12 K, 12 V) GQA / MQA Grouped Query Attention (GQA) (12 Q, 4 KV) 3x Memory Reduction: Cuts KV-cache bandwidth by 66%.
FFN Non-Linearity GeLU + LayerNorm (Additive bias) SwiGLU + RMSNorm SwiGLU + Pre-RMSNorm LLaMA/Mistral-Grade Efficiency: Higher expressivity per parameter.
GEC Formulation Seq2Seq / MLM (Slow, iterative) Generative Prompting (Prone to hallucinations) Constrained Atomic Edit-Tagging (6,148 tags) Zero Hallucination: 100% preservation of correct text.
End-to-End Latency 150ms - 400ms on CPU 300ms - 2,500ms via Cloud API < 10ms on CPU (INT8 ONNX) Real-Time Keystroke Budget: Executes within 16ms display frame.
Network & Privacy Local or Cloud Mandatory Cloud API (Data leaves host) 100% Offline Air-Gapped Complete Privacy: Zero telemetry, password field suppression.

Benchmark and Evaluation Metrics

Live CPU benchmarks measured on standard Intel/AMD consumer hardware (scripts/evaluate_gec_live.py):

Metric Measured Value Benchmark Description
Mean CPU Latency 9.97 ms Per-sentence end-to-end inference
P95 CPU Latency 12.12 ms 95th percentile latency (well below 16.6 ms display frame)
P99 CPU Latency 14.80 ms 99th percentile worst-case latency
GEC Stage 2 Validation Loss 0.2597 Down from 4.1067 (93.7% loss reduction across training)
LM Stage 2 Loss 1.2054 Fine-tuned on curated engineering and client communications
Clean Negative Preservation 100.0% Zero false-positive edits on grammatically correct input text
GEC Model Footprint 130.5 MB Symmetric INT8 ONNX graph (gec_stage2_int8.onnx)
LM Model Footprint 138.1 MB Symmetric INT8 ONNX graph (lm_stage2_int8.onnx)
Active Working Set RAM < 250 MB Total memory footprint during active inference

Repository Contents

File Size Format Description
gec_stage2_int8.onnx 130.5 MB ONNX INT8 INT8 Quantized graph for grammar, punctuation, and typo correction
lm_stage2_int8.onnx 138.1 MB ONNX INT8 INT8 Quantized graph for causal next-word autocomplete
edit_maps.json 228 KB JSON High-frequency confusion dictionary (4,096 replacements + 2,048 appends)
tokenizer.json 1.11 MB JSON Byte-Level BPE tokenizer config (16,000 vocabulary)
vocab.json / merges.txt 526 KB Text/JSON BPE vocabulary and merge table
config.json 520 B JSON Architecture parameters & model metadata
infer.py 6.5 KB Python Standalone, dependency-minimal Python inference engine
typoformer_architecture.png 235 KB PNG (300 DPI) Full neural architecture diagram (NVIDIA emerald theme)
typoformer_full_logo.png 188 KB PNG Official project branding logo
TypoFormer-win-x64.zip 221.0 MB ZIP Standalone Windows distribution package

Quickstart and Usage

1. Minimal Installation 

pip install onnxruntime tokenizers numpy
git clone https://huggingface.co/fahiiim/TypoFormer
cd TypoFormer
python infer.py

2. Standalone Python Example 

from infer import TypoFormer

# Initialize the standalone inference engine (loads ONNX models locally)
model = TypoFormer(".")

# 1. Autocomplete (Next-Word Generation)
prompt = "Could you please confirm if the proposed implementation timeline aligns with your"
suggestions = model.autocomplete(prompt, top_k=3)
print("Autocomplete suggestions:", suggestions)
# Output: [(' business', 0.9978), (' goals', 0.0002), (' company', 0.0001)]

# 2. Grammar, Punctuation & Typo Correction (GEC)
dirty_text = "We decoupled the event processing pipeline to achieve sub-"
suggestions = model.autocomplete(dirty_text, top_k=2)
print("Engineering completion:", suggestions)
# Output: [(' millisecond', 0.8841), (' second', 0.0712)]

# 3. Clean Text Preservation (Zero Hallucination)
clean_text = "The quick brown fox jumps over the lazy dog."
result = model.correct_grammar(clean_text)
assert result == clean_text  # 100% preservation

Windows Desktop Keystroke Companion (1-Line Install)

TypoFormer includes a native Windows x64 .NET 8 desktop daemon that runs silently in the system tray and renders a floating caret overlay across any active text editor (Word, Notepad, VS Code, Slack, Browser):

Run in PowerShell:

irm https://raw.githubusercontent.com/fahiiim/TyPoTaKer/main/install.ps1 | iex
  • Offline & Private: Zero telemetry; no keystrokes ever leave your device.
  • Password Aware: Automatically suppresses buffering on password and credential fields via Windows UI Automation.
  • Instant Action: Press [Tab] to accept autocomplete suggestions or grammar corrections.

Citation and Author Attribution 

@software{typoformer2026,
  author       = {Md. Fahim Sarker Mridul},
  title        = {TypoFormer: High-Precision Character-Aware Hybrid Transformer for Edge Autocomplete & Non-Autoregressive Grammar Correction},
  year         = {2026},
  publisher    = {Hugging Face},
  url          = {https://huggingface.co/fahiiim/TypoFormer},
  howpublished = {\url{https://github.com/fahiiim/TyPoTaKer}}
}

License

This model and its associated weights are distributed under the MIT License. Authored by Md. Fahim Sarker Mridul.