Qwen3-VL-8B SumTablets — Cuneiform Transliteration Model

A fine-tuned language model for automatic transliteration of Sumerian cuneiform sign sequences into scholarly Latin transliteration, trained on the SumTablets dataset. This model is intended to assist Assyriologists, digital humanities researchers, and anyone working with ancient Mesopotamian texts.

Evaluation result: 92.4% average token overlap on held-out test samples, 100% numerical marker accuracy, 100% structural tag preservation.


Model Description

This model is a QLoRA fine-tune of Qwen3-VL-8B-Instruct on the task of converting cuneiform sign name sequences (glyph_names) to standard Assyriological transliteration. It learns the mapping between the ASCII sign name representation used in cuneiform databases and the scholarly transliteration format used in academic publications.

What it can do

  • Convert cuneiform sign name sequences (e.g. DIŠ LU ŠE AN EN KID) to transliterated Sumerian (e.g. 1(diš) udu še {d}en-lil₂)
  • Preserve scholarly structural markers: <SURFACE>, <COLUMN>, <BLANK_SPACE>, <RULING>
  • Handle subscript numerals and diacritics: , , š, ŋ,
  • Correctly produce divine determinatives: {d}, {ki}, {geš}, {munus}, {dug}
  • Accurately render numerical qualifiers: 1(diš), 2(ban₂), 3(barig), 4(geš₂), sexagesimal notation
  • Handle <unk> tokens for damaged or illegible signs
  • Generalize across historical periods and tablet genres

What it cannot do (current limitations)

  • Read tablet images directly (text-only inference; vision tower not fine-tuned)
  • Transliterate sign sequences outside the SumTablets training distribution with guaranteed accuracy
  • Replace expert Assyriological review for publication-quality work
  • Handle all sign variants, particularly rare or archaic forms not well-represented in the training data

Intended Use

Intended users: Assyriologists, cuneiform scholars, digital humanities researchers, historians of ancient Mesopotamia, archaeologists working with tablet corpora, and developers building tools for ancient language processing.

Intended use cases:

  • Rapid first-pass transliteration of large tablet corpora
  • Assistance tool for researchers working with Cuneiform Digital Library Initiative (CDLI) data
  • Educational tool for students learning Sumerian
  • Integration into digital humanities pipelines for ancient text processing
  • Baseline model for further fine-tuning on specific tablet genres or periods

Not intended for: Authoritative scholarly publication without expert review. This model is an assistive tool, not a replacement for trained Assyriologists.


Training Details

Base Model

Property Value
Base model unsloth/Qwen3-VL-8B-Instruct-unsloth-bnb-4bit
Architecture Qwen3-VL (Vision-Language Model, language tower only)
Parameters 8.8B total, 43.6M trainable (0.50%)
Quantization NF4 4-bit (QLoRA)
Compute dtype bfloat16

Training Configuration

Hyperparameter Value
Method QLoRA (Quantized Low-Rank Adaptation)
LoRA rank (r) 16
LoRA alpha 16
RSLoRA enabled
Target modules q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Learning rate 8e-05
LR scheduler Cosine
Warmup 20 steps
Batch size 2 per device
Gradient accumulation 8 steps
Effective batch size 16
Optimizer AdamW 8-bit
Weight decay 0.01
Gradient clipping 1.0 (max_grad_norm)
Max sequence length 4096 tokens
Packing enabled
Gradient checkpointing Unsloth optimized
Epochs 1 (stopped at step ~1161, ~22% through epoch)
Training framework Unsloth Studio 2026.5.10
Hardware A Single NVIDIA RTX 5080 16GB (3h 54m 38s)

Training Dataset

Property Value
Dataset colesimmons/SumTablets
Train split 82,452 samples
Validation split 4,580 samples
Test split 4,580 samples
License CC-BY-4.0
Input field glyph_names (cuneiform sign names in ASCII)
Target field transliteration (scholarly Latin transliteration)

Training Trajectory

The model was trained with early stopping based on loss monitoring. Training proceeded through several distinct phases:

Step range Loss range Phase
1–100 3.63 → 0.72 Rapid initial learning
100–450 0.72 → 0.55 First plateau, optimizer stabilizing
450–700 0.55 → 0.38 Breakthrough descent
700–1000 0.38 → 0.30 Continued descent, sub-0.40 consolidation
1000–1161 0.30 → 0.20 Final descent, target zone reached

Final reported loss: ~0.202 at step 1161
Best observed single-step loss: 0.2844
Gradient norm at checkpoint: consistently 0.55–0.90 (stable)

Note on reported loss: Training included full sequence loss (prompt + completion) for the resume phase due to a template mapping issue. Completion-only loss is estimated to be lower than the reported figures. This is documented for scientific transparency.


Evaluation

Test Set Performance (10% held-out samples)

Evaluation was performed at checkpoint step 1161 against held-out samples from the SumTablets test split, spanning Administrative, Literary, and Royal Inscription genres across Ur III, Old Akkadian, Early Dynastic IIIb, and Old Babylonian periods.

Metric Score
Average token overlap 92.4%
Numerical marker accuracy 100%
Format tag preservation 100%
Samples with >84% overlap 9/10
Samples with >97% overlap 2/10
Exact matches 1/10

Sample-level Results

Sample Genre Token Overlap Notes
1 Administrative High Long output (~900 chars), correct format throughout
2 Administrative Good Minor sign errors: kasumurₓ(KA), amagid₂
3 Administrative Excellent Strong match across full tablet
4 Administrative Excellent Numerics and structure correct
5 Numeric Excellent All number qualifiers flawless
6 Administrative 92% Perfect numericals, minor token variation
7 Administrative 89% Minor spacing variation only
8 Administrative 84% Perfect numericals, some sign-level divergence
9 Administrative 97% Near-perfect across all lines
10 Administrative 100% Exact match

Strengths

  • Structural tags: Perfect preservation of <SURFACE>, <COLUMN>, <BLANK_SPACE>, <RULING> markers
  • Numerical system: Flawless handling of the Sumerian counting system across all numeral classes (ordinal, capacity, weight, area, sexagesimal)
  • Determinatives: Correct rendering of divine ({d}), geographic ({ki}), wooden ({geš}), and other determinative classes
  • Diacritics and subscripts: Accurate production of subscripted signs (š, ŋ, ḫ, subscript numerals)
  • Damaged signs: Appropriate <unk> token use for illegible signs

Known Limitations

  • Occasional sign-level confusions between visually or contextually similar signs (e.g. elam/dih₃, gaš/gazi) — consistent with the difficulty of the task rather than systematic failure
  • Performance on Literary and Royal Inscription genres is less characterized than Administrative tablets, which dominate the training distribution (~85% of SumTablets)
  • Output truncation possible on very long tablets; not observed in evaluation samples but possible at the margins of the 4096 token context window

Usage

Basic Inference

from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel
import torch

# Load base model and adapter
base_model_id = "unsloth/Qwen3-VL-8B-Instruct-unsloth-bnb-4bit"
adapter_path = "TRACCERR/Qwen3-VL-8B-SumTablets"

tokenizer = AutoTokenizer.from_pretrained(base_model_id)
model = AutoModelForCausalLM.from_pretrained(
    base_model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)
model = PeftModel.from_pretrained(model, adapter_path)
model.eval()

# Format input
sign_sequence = "DIŠ |GA₂×PA| GA A LU |HI×AŠ₂| SAG BA TIL UD 4(DIŠ) |HI×BAD|"

system_prompt = (
    "You are an expert Assyriologist specializing in Sumerian cuneiform tablets. "
    "Given a sequence of cuneiform sign names, produce the correct transliteration. "
    "Preserve all structural markers (<SURFACE>, <COLUMN>, <BLANK_SPACE>, <RULING>), "
    "subscript numerals, and <unk> tokens for damaged or illegible signs. "
    "Output only the transliteration."
)

messages = [
    {"role": "system", "content": system_prompt},
    {"role": "user", "content": f"Sign sequence:\n{sign_sequence}\n\nProvide the transliteration."}
]

text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=512,
        temperature=0.1,
        do_sample=False,
        repetition_penalty=1.1
    )

response = tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True)
print(response)
# Expected: 1(diš) sila₄ ga a udu hur-sag ba-uš₂ u₄ 4(diš)-kam

With Unsloth (recommended for speed)

from unsloth import FastLanguageModel

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="TRACCERR/Qwen3-VL-8B-SumTablets",
    max_seq_length=4096,
    dtype=None,  # Auto-detect
    load_in_4bit=True,
)
FastLanguageModel.for_inference(model)

Input Format

The model expects cuneiform sign names in CDLI/ATF format — uppercase sign names, compound signs in pipes (e.g. |GA₂×PA|), numerical notations like 4(DIŠ). This corresponds directly to the glyph_names field in the SumTablets dataset.


Dataset Credit and Attribution

This model was trained on the SumTablets dataset created by Cole Simmons.

  • Dataset: colesimmons/SumTablets
  • License: CC-BY-4.0
  • Description: A comprehensive dataset of Sumerian cuneiform tablets containing transliterations, sign names, and Unicode cuneiform characters, sourced from the Cuneiform Digital Library Initiative (CDLI)

The underlying tablet data originates from the Cuneiform Digital Library Initiative (CDLI), a collaborative project of the University of California Los Angeles, the Max Planck Institute for the History of Science, and partner institutions worldwide. The CDLI database represents decades of work by hundreds of Assyriologists who created the original transliterations.

We gratefully acknowledge:

  • The scholars who produced the original tablet transliterations archived in CDLI
  • Cole Simmons for curating and publishing the SumTablets dataset
  • The CDLI project for making cuneiform scholarship openly accessible

Scientific Transparency

This section documents training decisions, anomalies, and limitations for the benefit of the research community.

Training Anomalies Documented

Scheduler miscalibration: Sequence packing reduced the effective step count from the displayed 5,154 to approximately 330 per epoch, but the cosine LR scheduler was anchored to the display total. This resulted in near-peak LR (7.8–7.9e-05) throughout training rather than the intended cosine decay. The model trained at approximately constant high LR for the full run.

Packing disabled on resume: After manual checkpoint resume, Unsloth detected the VLM architecture and silently disabled sequence packing ("Sample packing skipped (vision-language model detected)"). The resumed training ran unpacked, changing the effective token density per step.

Completion masking failure on resume: The train_on_completions=true setting (intended to mask prompt tokens and compute loss only on transliteration output) failed to apply on resume due to missing template mapping for the VLM model ID. The resumed training computed loss over full sequences including system prompt and user turn. This inflates the reported loss but does not indicate incorrect training — the model still learned the target mapping.

Loss reporting: The "0.202 final loss" figure includes prompt tokens in the loss calculation for the resumed portion of training. True completion-only loss is estimated lower. Single-step minimums of 0.2844 were observed under completion-only conditions earlier in training.

Historical Accuracy Notes

The model's transliteration output reflects the scholarly conventions encoded in the CDLI database as of the SumTablets dataset creation date. Cuneiform transliteration involves interpretive choices by scholars, and the training data represents one scholarly tradition. Known conventions include:

  • Sign readings follow CDLI/ePSD2 (electronic Pennsylvania Sumerian Dictionary) conventions
  • Numerical notation follows ATF (Ascii Transliteration Format) standards
  • Uncertain readings are represented as <unk> in both input and output
  • The model will reproduce the transliteration conventions of the training corpus, which may differ from alternative scholarly traditions

The model should not be used as an authority on disputed sign readings. Expert Assyriological review is recommended for any publication-quality use.

Reproducibility

Component Version
Unsloth Studio 2026.5.10
Transformers 4.57.6
PyTorch 2.10.0+cu130
CUDA 12.0 / Toolkit 13.0
bitsandbytes (Unsloth bundled)
Random seed 42
Hardware NVIDIA RTX 5080 16GB, Windows 11

Model Card Authors

TRACCERR — Fine-tuning, training infrastructure, evaluation, and documentation.
Trained using Unsloth Studio for accelerated QLoRA fine-tuning on Windows.


Citation

If you use this model in research, please cite the SumTablets dataset and the CDLI:

@dataset{simmons2024sumtablets,
  author    = {Cole Simmons},
  title     = {SumTablets: A Dataset for Sumerian Cuneiform Transliteration},
  year      = {2024},
  publisher = {Hugging Face},
  url       = {https://huggingface.co/datasets/colesimmons/SumTablets},
  license   = {CC-BY-4.0}
}

@misc{cdli,
  title        = {Cuneiform Digital Library Initiative},
  howpublished = {\url{https://cdli.ucla.edu}},
  note         = {Collaborative project of UCLA, MPIWG, and partner institutions}
}

@misc{traccerr2026sumtablets_model,
  author       = {TRACCERR},
  title        = {Qwen3-VL-8B SumTablets: A Fine-Tuned Model for Sumerian Cuneiform Transliteration},
  year         = {2026},
  publisher    = {Hugging Face},
  url          = {https://huggingface.co/TRACCERR/Qwen3-VL-8B-SumTablets}
}

License

This model is released under CC-BY-4.0, consistent with the license of the SumTablets training dataset.

You are free to use, share, and adapt this model for any purpose, including commercial use, provided appropriate credit is given to:

  • Cole Simmons (SumTablets dataset)
  • The CDLI project (underlying tablet data)
  • TRACCERR (model fine-tuning)

Related Resources

Downloads last month
127
GGUF
Model size
8B params
Architecture
qwen3vl
Hardware compatibility
Log In to add your hardware

4-bit

5-bit

6-bit

8-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for TRACCERR/Qwen3-VL-8B-Instruct-Sumtablets-V2-bnb-4b

Dataset used to train TRACCERR/Qwen3-VL-8B-Instruct-Sumtablets-V2-bnb-4b