Badrdeen/qwen-financial-json-extractor

Qwen2.5-0.5B-Instruct Financial JSON Extractor

A fine-tuned version of Qwen/Qwen2.5-0.5B-Instruct trained with QLoRA and merged into standalone FP16 weights. Given a financial tweet or headline, it outputs strict JSON containing market sentiment (bearish, bullish, neutral) and extracted uppercase stock cashtags.

Evaluation Results

Split / Setup	JSON-Valid %	Sentiment Macro-F1	Ticker Exact-Match
Zero-Shot Baseline (Qwen2.5-0.5B-Instruct)	83.0%	0.257	0.060
Internal Validation Split (Day 4 QLoRA, $N=1,909$)	100.0%	0.853	0.992
Official Held-Out Test Split (validation, $N=2,388$)	99.9%	0.863	0.994

Model Comparison

Model Architecture	Validation Macro-F1	Notes on Split / Setup
Day 3 Qwen (Classification-Only)	0.844	Trained on 0.15 val split ratio
RoBERTa LoRA	0.845	Encoder-only baseline on 0.15 val split ratio
Qwen2.5-0.5B-Instruct (This Model, QLoRA)	0.853	Carved from train with test_size=0.2, seed=42 (Note: Direct cross-model comparisons are approximate due to differing split ratios).

Training Details

• Data Split: Validation set carved from the training split using test_size=0.2, seed=42 (train_ds: 7,634 samples; val_ds: 1,909 samples). The official held-out test split ($N=2,388$) was kept strictly untouched until final evaluation.
• Quantization & Hardware: 4-bit NF4 quantization (BitsAndBytesConfig with double quantization and torch.float16 compute) trained on a Google Colab T4 GPU.
• LoRA Hyperparameters: Rank $r = 16$, $\alpha = 32$, dropout $0.05$, targeting all linear layers (q_proj, v_proj, k_proj, o_proj, gate_proj, up_proj, down_proj).
• Objective: Trained for 1 epoch using SFT (DataCollatorForSeq2Seq) with loss computed strictly on the assistant response tokens (prompt tokens masked with -100).

Production Recommendations & Routing

• (a) Sentiment Only at 1M Tweets/Day: Use a lightweight encoder-only model like RoBERTa combined with an optimized batch inference pipeline. Because sentiment classification is a single-label scoring task, encoder architectures deliver significantly higher throughput and lower latency at scale without generative decoding overhead.
• (b) Sentiment Plus Tickers (Joint Extraction): Use this Fine-Qwen QLoRA model. Joint extraction via a generative language model allows you to extract multi-item variable-length ticker arrays and sentiment in a single pass without needing separate NER pipelines or brittle regex-matching layers.

Known Limitations & Edge Cases

• Max Token Truncation: The generation limit is set to max_new_tokens=64. Tweets exceptionally dense with cashtags (15+ tickers) will have their ticker lists truncated mid-array. (Recommended to raise to max_new_tokens=128 if processing long-form multi-ticker posts in production).

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

MODEL_ID = "Badrdeen/qwen-financial-json-extractor"
tok = AutoTokenizer.from_pretrained(MODEL_ID)
model = AutoModelForCausalLM.from_pretrained(MODEL_ID, torch_dtype=torch.float16, device_map="auto")

SYSTEM = (
    'Return only a JSON object with two keys: "sentiment" (one of bearish, bullish, neutral) '
    'and "tickers" (list of cashtag symbols in the text, uppercase, without $; [] if none). No other text.'
)

msgs = [
    {"role": "system", "content": SYSTEM},
    {"role": "user", "content": "$NVDA surges 8\% on strong AI data center demand while$INTC slips."}
]

prompt = tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True)
inputs = tok(prompt, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=128, do_sample=False)

print(tok.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))