toranb/theo-qwen-instruct

Theo Qwen Instruct

theo-qwen-instruct is a 1.5B-parameter instruction-tuned causal language model for biblical and theological English. It is theo-qwen-base after one epoch of supervised fine-tuning on 153,633 examples in three modes:

mode	tag	what it does
chat	[unused1]	Answers questions in a conversation. Given a context (documents=), it answers only from that context and says so when the context does not contain the answer.
devotional	[unused5]	Writes a short devotional (≈270 words) on one Scripture passage: explanation, the gospel, application.
summary	[unused6]	Summarizes a conversation so it can be continued.

The mode is chosen by a token at the start of the prompt; the bundled chat template writes it for you.

What This Model Is For

• Grounded question answering over documents you supply (retrieval-augmented chat), including saying when they do not contain the answer
• Bible and pastoral chat without a context, multi-turn, including prayers on request
• Devotional writing from one Scripture passage
• Rolling conversation summaries, so a long chat can continue in a short prompt
• A starting point for further SFT or preference tuning in the same prompt format

It is not a general-purpose assistant: no tool use, no system prompt, English only.

It reads and writes lowercase. Every training example was lowercase (and all but four responses plain ASCII), so the chat template lowercases your input, and the output comes back lowercase. Restore casing in your application if you need it.

Quick Start

pip install -r requirements.txt   # torch>=2.5, transformers>=5.0

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "toranb/theo-qwen-instruct"
device = "cuda" if torch.cuda.is_available() else "cpu"
dtype = torch.bfloat16 if device == "cuda" else torch.float32   # fp32 on CPU
tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True, dtype=dtype).to(device).eval()

def ask(messages, **template_kwargs):
    inputs = tokenizer.apply_chat_template(
        messages, add_generation_prompt=True, return_tensors="pt", return_dict=True, **template_kwargs,
    ).to(model.device)
    output = model.generate(**inputs, max_new_tokens=400)   # samples: temperature 0.6, top-k 40, top-p 0.75
    return tokenizer.decode(output[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True)

# Grounded chat: answers only from the documents.
print(ask(
    [{"role": "user", "content": "Who wrote the book of Hebrews?"}],
    documents=[{"title": "Hebrews", "text": "Hebrews does not name its author. Early church writers "
                "suggested Paul, Barnabas, Luke or Apollos, and Origen concluded that only God knows who wrote it."}],
))

# Chat without a context, multi-turn.
print(ask([
    {"role": "user", "content": "Could you write a short prayer for me?"},
    {"role": "assistant", "content": "Yes. Please share a scripture passage or a little context about what you need prayer for."},
    {"role": "user", "content": "I start a new job on Monday and I'm anxious. Philippians 4:6-7 has been on my mind."},
]))

# Devotional: one message, "<reference> <verse text>".
print(ask([{"role": "user", "content": "Romans 8:1 There is therefore now no condemnation to them which are in Christ Jesus, who walk not after the flesh, but after the Spirit."}],
          mode="devotional"))

# Summary: the messages are the conversation to summarize.
print(ask([
    {"role": "user", "content": "Is it wrong to doubt God?"},
    {"role": "assistant", "content": "Doubt is not the opposite of faith. Many psalms bring honest questions to God."},
], mode="summary"))

generation_config.json carries the recommended sampling settings (temperature 0.6, top-k 40, top-p 0.75). Temperature 0.8 gives more varied devotionals and prayers. Pass do_sample=False for greedy decoding.

Chat template options

argument	values	effect
mode	"chat" (default), "devotional", "summary"	selects the mode tag and the instruction layout
documents	list of {"text": ..., "title": ...} (title optional)	chat only: makes the request grounded. Documents are joined into one context:, a title is written (source: title)

• Chat turns the whole conversation into one instruction, [unused3] user / [unused4] assistant lines ending on the current user message, as the model was trained.
• Grounded chat adds the trained cue, "answer the question using only the context below. if the context does not contain the answer, say you do not know.", and the context before the turns. Ask an unanswerable question against a context and the model is trained to say so instead of answering from memory.
• Devotional takes exactly one user message: a reference and the verse text. A bare reference works less well; the model expounds the text it is given.
• System messages are rejected. The system prompt is fixed by training: "Follow the opening mode marker. Answer in that mode, using only a supplied context when one is given."
• Fine-tuning: in chat and devotional mode, end the conversation with an assistant message and the template renders it as the target, closed with <eos>; return_assistant_tokens_mask=True marks exactly the response tokens. The target is used as given (only stripped, as in the original training), so supply lowercase text. Summary mode has no target: every message is part of the transcript, so append the summary's tokens and <eos> yourself.

Prompt Format

What the template produces (and what to send if you are not using transformers):

<bos><doc>[unused1]
### System
Follow the opening mode marker. Answer in that mode, using only a supplied context when one is given.

### Input
answer the question using only the context below. if the context does not contain the answer, say you do not know.

context: (source: hebrews) hebrews does not name its author. ...

[unused3] who wrote the book of hebrews?

### Response

The response ends with <eos>. Without a context the input is just the turns ([unused3] ... / [unused4] ... lines). A devotional input is the passage alone after [unused5]. A summary input is summarize the conversation so far so it can be continued:, a blank line, then the turns joined by spaces after [unused6]. <bos>, <doc> and the [unusedN] tags are single tokens.

Keep prompt + response within 1,442 tokens. That is an empirical budget, not a training limit: training sequences were capped at 1,536 and the longest example was 1,374 tokens, and in use, generations that ran just past 1,442 began spelling words out in pieces.

Samples

Greedy decoding through the chat template, at most 200 new tokens (torch 2.13.0, bf16, RTX 4090). Default sampling gives more varied text.

Grounded chat, with a short document on Hebrews' authorship titled "Hebrews" (the Quick Start's, plus one sentence on its readers):

hebrews does not name its author. early church writers suggested paul, barnabas, luke, or apollos, and origen concluded that only god knows who wrote it.

source: hebrews

Grounded chat, answer not in the context. Question: what did Paul eat for breakfast in Corinth? Context: Paul's eighteen months in Corinth as a tentmaker (Acts 18).

paul's eighteen months in corinth are described, but his breakfast is not recorded.

Chat without a context: what does it mean that Jesus is the good shepherd?

it means jesus is the one who truly and faithfully leads his people. he knows his sheep, keeps them, and gives himself for them. he is not like a hired worker who leaves when danger comes. he is the shepherd who lays down his life for the flock.

Devotional on Romans 8:1, given in public-domain wording (cut at 200 new tokens):

there is therefore now no condemnation to them which are in christ jesus, and that word therefore is a mercy because it tells you that the verdict of god has changed forever for all who belong to christ. the condemnation that once marked sinners is not canceled by your effort, your religious habit, or your best week, but by the finished work of the savior who died in your place and rose again to justify the ungodly. paul does not say that believers are free from every consequence, but that they are free from condemnation, meaning the final sentence of god against them has been removed. that is not because sin is small, but because christ is great. the flesh still tempts us, still whispers that we must prove ourselves, still makes us chase approval, control, and self-justification. yet the spirit leads us in a different way, not after the flesh, but after the spirit, which means the life of faith is marked by the new direction god gives through his word and his spirit …

Summary of a two-exchange conversation: is it wrong to doubt God? / doubt is not the opposite of faith; many psalms bring honest questions, and Jesus was gentle with Thomas / so what should I do with my doubts? / bring them to God in prayer, keep reading Scripture, talk them through with mature believers.

the user asked whether doubt is wrong, and the assistant said doubt is not faith but can be brought to god in prayer and scripture. the assistant encouraged the user to keep asking questions and seek help from mature believers.

The summary shows typical slips: the conversation said doubt is not the opposite of faith, which the summary compresses to "doubt is not faith", and "keep asking questions" is not in the conversation. The grounded answer ends with a source: line, a habit learned from the multi-turn grounded data.

How It Was Built

Qwen/Qwen3-1.7B @70d244cc
 └─ Bible devotional LoRA merged in (71,644 pairs)
     └─ transplant: vocab 151,936 → 32,788, + value-embedding residuals
         └─ grounded continued pretraining, ~71M tokens        = theo-qwen-base
             └─ dual-mode SFT, 153,633 examples, 1 epoch       ← this release

Stages 1–3 are described in the theo-qwen-base card. This release adds stage 4:

mode	examples	supervised tokens	share	what
devotional	56,009	19,112,086	67.6%	a Scripture passage → a devotional
chat	76,156	7,782,090	27.5%	grounded Q&A over supplied context, refusals when the context lacks the answer, and multi-turn conversation in Bible, pastoral, prayer, and general domains
summary	21,468	1,375,729	4.9%	a conversation → a summary that lets it continue
total	153,633	28,269,905

Loss was on response tokens only. Adafactor at lr 2e-5 with 1M warmup tokens, a cosine schedule with a 6-epoch horizon stopped after 1 epoch (9,602 steps; epoch 1 was the best epoch in every earlier SFT round), weight decay 0.1, clip 1.0, 16 sequences per step, sequences up to 1,536 tokens, bf16, seed 20260827. One RTX 4090, 3 h 58 min. The training set was checked against a sealed held-out devotional set: 0 overlapping rows.

Evaluation

Details and methods are in EVAL.md.

Validation loss (held-out SFT rows, epoch 1)

Response-token cross-entropy on held-out rows of each mode (up to 500 rows per set: 330 summary, 37 general, 83 refuse and 48 insufficiency, the full held-out set for each; 500 for the other four), at the released checkpoint:

set	logged during training	through transformers + the chat template
balanced	1.4043	1.4045
devotional	1.3733	1.3734
chat	1.4125	1.4127
summary	1.1094	1.1092
general	1.2713	1.2707
chat_bible	1.3961	1.3961
refuse	0.9839	0.9844
insufficiency	0.9656	0.9655

general is general (non-Bible) conversation; refuse is a context that lacks the answer, where the target declines; insufficiency is an off-topic context. The second column re-scores the same rows through this release's code and chat template (see Release Details).

Pairwise judging against the previous model

Twenty chat prompts × 3 sampled answers at the default sampling settings, judged blind against the previous model (the same base, an earlier SFT mix) by two LLM judges, each in both orders. Ten prompts are grounded chat questions, 9 of which also appear in both models' SFT data. The other ten are held-out Bible questions. A side wins only when both orders agree. A reseeded run of the previous model against itself is the noise row. The previous model is not released, and the answers were generated from an fp8 serving export of these weights, not this bf16 file.

judge, axis	this model vs previous: win / loss / tie	noise row
judge 1, overall	26 / 18 / 16	25 / 27 / 8
judge 2, overall	27 / 17 / 16	23 / 23 / 14
judge 1, grounding	20 / 10 / 30	14 / 17 / 29
judge 2, grounding	18 / 8 / 34	17 / 22 / 21

The gain is in grounding, and it comes from the held-out Bible questions (overall 16–7 and 15–8); on the grounded questions the two models are even. It is directional, not significant at this size (sign test p = 0.29 and 0.17). A devotional/prayer regression check (12 devotional and 8 prayer prompts, 4 draws) found no drift: median devotional length 269 words vs 274, no repetition loops, and no style violations in devotionals.

Theology ruler (545 cases)

The base model's theological cloze ruler, scored the same way (candidate sentences ranked by log-likelihood, scripts/theology_eval.py). It measures how much theological knowledge survives instruction tuning; it is not a chat benchmark.

	Qwen3-1.7B (donor)	theo-qwen-base	theo-qwen-instruct
Overall	96.0% (523/545)	98.9% (539/545)	97.2% (530/545)
Doctrinal association	98.6% (218/221)	99.5% (220/221)	99.5% (220/221)
Canonical knowledge	99.3% (136/137)	100.0% (137/137)	98.5% (135/137)
Contrastive theology	90.4% (169/187)	97.3% (182/187)	93.6% (175/187)
Critical failures	63	21	34

Instruction tuning cost 9 cases, mostly contrastive theology; the model still ranks ahead of its donor. The eval sentences are cased and the SFT data was lowercase, which may account for part of the drop.

Architecture and Tokenizer

Identical to theo-qwen-base: a 28-layer Qwen3-1.7B geometry (GQA 16/8, Q/K norm, RoPE, SwiGLU) with value-embedding residuals on layers 0, 2, …, 20, 1,499,628,643 parameters, tied embeddings, 2,048 positions. The tokenizer is the base's 32,788-token byte-level BPE ("Cedar" is the project's internal name for it). Three of its reserved tokens are the mode tags and two are the turn markers. cedar_dual_prompt_v1 is the name of the prompt framing under Prompt Format. Module names follow Qwen3, so PEFT target_modules written for Qwen3 work unchanged.

Repository Contents

File	Purpose
configuration_theo_qwen.py, modeling_theo_qwen.py, __init__.py	The config and model code (identical to the base)
config.json	Model configuration (identical to the base)
generation_config.json	Recommended sampling defaults
model.safetensors	Released bf16 weights, a bit-exact copy of the trained checkpoint
tokenizer.json, tokenizer_config.json	The tokenizer (identical to the base)
chat_template.jinja	The chat template: modes, documents, normalization, fine-tuning targets
checkpoint_metadata.json	Source checkpoint, hashes and training metadata
eval.json, scripts/theology_eval.py	The theology ruler and its scorer (identical to the base)
eval_results/	Per-case theology results for this model, the base and stock Qwen3-1.7B
EVAL.md, MODEL_CARD.md	Evaluation methodology and the full model card
requirements.txt	Minimum dependencies (torch 2.5+, transformers 5.0+)
LICENSE	Apache-2.0

Limitations

• Lowercase only. Input is lowercased by the template; output is lowercase.
• Grounding is a tendency, not a guarantee. Given a context it usually stays inside it, and it was trained to say when the context lacks the answer, but it can still add unsupported sentences, restate itself, or end on an application line nobody asked for. Check answers against the context.
• No tool use, no system prompt, English only. Chat outside the Bible and theology domain is thin (a small slice of general conversation in SFT).
• It can misquote or misattribute Scripture, invent references, and blend passages. Check anything quoted.
• It reflects the theological traditions represented in its Christian devotional and religious-domain training material. Its answers are not authoritative, and it is not a substitute for pastoral care.
• Keep prompt + response within 1,442 tokens (an empirical budget). The positional limit is 2,048, but no training example was longer than 1,374 tokens.
• The training data is not public.

Release Details

• Exported from cedar-sft-dual-s20260827-v13-v2base/checkpoint-epoch-01.pt (step 9,602; sha256 0aca238f…), trained from theo-qwen-base's exact weights.
• The weights are bf16, bit-exact, like the base. The architecture, config, remote code and tokenizer are identical to theo-qwen-base; only the weights, the chat template and the sampling defaults differ.
• The HF implementation reproduces the training code's logits bit for bit in bf16 on GPU (torch 2.13.0, transformers 5.15.1), and its greedy generations token for token.
• The chat template was checked against every SFT example: rendering each of the 155,229 well-formed source-corpus conversations (one malformed row is skipped; the training split drawn from them is 153,633 examples) through chat_template.jinja gives exactly the token ids the training code built, prompt and target. Two corpus quirks are normalized first: a blank line instead of a newline between some multi-turn grounded turns (2,723 rows), and a double space after a turn tag (2 rows). The template uses a single newline, the layout used at inference.
• Re-scored through transformers and the chat template, the held-out validation losses match the training log to within 6.4e-4. Scored in the same harness, the training code gives exactly this release's value (checked on the balanced set), so the small gap to the log comes from the original run's environment, not from the port.
• Loading requires trust_remote_code=True. Minimum: torch 2.5, transformers 5.0.

Attribution

Built on Qwen3-1.7B by the Qwen team, Alibaba Cloud, released under the Apache License 2.0, via theo-qwen-base. Released under Apache 2.0; see LICENSE. The Apache 2.0 license covers the model weights and code; it grants no rights in any third-party text the model may reproduce.

Citation

@misc{theoqweninstruct2026,
  author       = {Toran Billups},
  title        = {Theo Qwen Instruct: A Grounded Bible-Domain Chat and Devotional Model},
  year         = 2026,
  publisher    = {Hugging Face},
  howpublished = {https://huggingface.co/toranb/theo-qwen-instruct}
}