FinLLaMA

📄 Paper · 🤗 Collection · 🌐 The Fin AI

Part of Open-FinLLMs — Open-FinLLMs: Open Multimodal Large Language Models for Financial Applications (arXiv:2408.11878).

FinLLaMA is a financial foundation model obtained by continual pre-training of LLaMA3-8B on a 52-billion-token financial corpus covering text, tables and time-series data from seven financial domains. It is a base (non-instruction-tuned) model: use it for further fine-tuning, few-shot prompting or research on domain adaptation.

Model Details

Base model	LLaMA3-8B
Architecture	LlamaForCausalLM, 32 layers, hidden size 4096, vocab 128,256
Context length	8,192 tokens
Weights	float32 (torch_dtype in config.json)
Training data	52B-token financial corpus (text, tables, time series)
Training	1 epoch with DeepSpeed on 64 × A100 80GB (≈250 h); LR 1e-5, cosine schedule, weight decay 1e-5, warm-up ratio 0.05, batch size 2 per device, max sequence length 8,192
License	Llama 3 Community License (inherited from the base model)

Intermediate checkpoints

23 intermediate pre-training checkpoints are published as git tags, from step204 to step4692, so you can study how financial knowledge emerges during continual pre-training. Load one with revision:

from transformers import AutoModelForCausalLM, AutoTokenizer

ckpt = "step2448"   # any of: step204, step408, step612, step816, step1020, step1224, step1428, step1632, step1836, step2040, step2244, step2448, step2652, step2856, step3060, step3264, step3468, step3672, step3876, step4079, step4284, step4488, step4692
tokenizer = AutoTokenizer.from_pretrained("TheFinAI/FinLLaMA", revision=ckpt)
model = AutoModelForCausalLM.from_pretrained("TheFinAI/FinLLaMA", revision=ckpt, torch_dtype="auto", device_map="auto")

The main branch holds the final model.

Quick Start

from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("TheFinAI/FinLLaMA")
model = AutoModelForCausalLM.from_pretrained("TheFinAI/FinLLaMA", torch_dtype="auto", device_map="auto")

prompt = "The company's operating margin improved because"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
print(tokenizer.decode(model.generate(**inputs, max_new_tokens=64)[0], skip_special_tokens=True))

Related models

• FinLLaMA-instruct — FinLLaMA instruction-tuned on 573K financial instructions.
• FinLLaVA — multimodal model trained with 1.43M image–text instructions.

Evaluation

The paper evaluates FinLLaMA on 19 datasets (9 tasks) zero-shot and 4 datasets (3 tasks) few-shot, comparing with LLaMA3-8B, LLaMA3.1-8B and BloombergGPT, and in trading simulations (Sharpe ratio). See the paper for results.

Intended Use & Limitations

• Research on financial language modeling and as a starting point for fine-tuning.
• Not instruction-tuned: it continues text rather than following instructions.
• Not investment advice; may produce incorrect or outdated financial information.

Citation

@misc{huang2025openfinllmsopenmultimodallarge,
      title={Open-FinLLMs: Open Multimodal Large Language Models for Financial Applications}, 
      author={Jimin Huang and Mengxi Xiao and Dong Li and Zihao Jiang and Yuzhe Yang and Yifei Zhang and Lingfei Qian and Yan Wang and Xueqing Peng and Yang Ren and Ruoyu Xiang and Zhengyu Chen and Xiao Zhang and Yueru He and Weiguang Han and Shunian Chen and Lihang Shen and Daniel Kim and Yangyang Yu and Yupeng Cao and Zhiyang Deng and Haohang Li and Duanyu Feng and Yongfu Dai and VijayaSai Somasundaram and Peng Lu and Guojun Xiong and Zhiwei Liu and Zheheng Luo and Zhiyuan Yao and Ruey-Ling Weng and Meikang Qiu and Kaleb E Smith and Honghai Yu and Yanzhao Lai and Min Peng and Jian-Yun Nie and Jordan W. Suchow and Xiao-Yang Liu and Benyou Wang and Alejandro Lopez-Lira and Qianqian Xie and Sophia Ananiadou and Junichi Tsujii},
      year={2025},
      eprint={2408.11878},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2408.11878}, 
}