FinGPT-131M

FinGPT-131M is a 131M-parameter decoder-only language model trained from scratch on a GST/tax regulatory corpus.

Architecture

• Parameters: ~131.3M
• Layers: 13
• Hidden size: 768
• Attention heads: 12
• Context length: 1024
• Vocabulary: 50,257
• Tokenizer: GPT-2 BPE
• Activation: GELU
• Attention: causal scaled dot-product attention
• Weight tying: Yes

Training

Best checkpoint:

• Step: 11,250
• Validation loss: 2.7641

The model was trained from scratch; it is not fine-tuned from GPT-2.

Intended use

The model is intended for experimentation and research on language modeling over GST and tax-related text.

Limitations

This model should not be treated as a source of authoritative legal or tax advice. Tax regulations can change, and model outputs may contain hallucinations or outdated information.