Commit History
-
docs: add bert-base-uncased baseline comparison to README
Shows theo-bert-base (94.7%) vs general-purpose bert-base-uncased (47.8%) on the 546-case theological MLM benchmark. Includes per-difficulty and per-category breakdowns plus contrastive confidence analysis.fa10de7 -
Initial release: TheoBERT Base — biblical-domain masked language model
273M-parameter custom bidirectional encoder pretrained from scratch on biblical and theological text, packaged for Hugging Face Hub. Model - Custom architecture: 12 layers, hidden 768, 8 heads, RoPE on Q/K, RMSNorm on Q/K, ReLU² MLP, gated value embeddings on even layers, learned residual interpolation between block output and initial embedding; MLM head: Linear → GELU → RMSNorm → Linear - Weights released as fp16 safetensors (546 MB) - config.json declares torch_dtype: float32 so default loads upcast on read; CPU inference is numerically safe by default. Pass dtype=torch.float16 for GPU fp16 inference. - Loads via AutoModel / AutoModelForMaskedLM with trust_remote_code=True - Bundles the google-bert/bert-base-uncased fast tokenizer so AutoTokenizer.from_pretrained(repo) and the Hub fill-mask widget work out of the box Training - Stage 1: 24 epochs MLM pretraining, 256-token context, 270K biblical and theological sequences (final train loss 1.0679, accuracy 76.42%) - …a64c547 -
initial commit
Verified signature7d0b4f7