Abkhaz Cyrillic OCR Model (PaddleOCR)

A high-quality PaddleOCR Recognition model for recognizing the Abkhaz language (ISO 639-1: ab) in modern and historical Cyrillic script.

This model is a core component of the digital infrastructure project for the Abkhaz language, developed by Marat Ionov in collaboration with the Bagrat Shinkuba Fund (Фонд Баграта Шинкуба).

Model Details

Model Type: PaddleOCR Recognition model (PP-OCRv5_server_rec architecture)
Inference Format: PIR (Paddle Intermediate Representation) — the modern format for PaddlePaddle
Target Script: Abkhaz Cyrillic (including extended characters: ә, ҧ, ҳ, қ, ҭ, ҽ, ҿ, ӡ, џ, ҕ, ӷ, ҟ, ҵ, ҩ, ҷ, ԥ)
Custom Character Dictionary: abk_cyrillic_dict.txt (included in this repository)

🌟 Key Feature: Orthographic Period Handling

The Abkhaz Cyrillic alphabet underwent a significant reform. This model is explicitly trained to handle both historical periods robustly:

1954–1996: Recognizes legacy letters Ҧ ҧ, Ҕ ҕ, and the back apostrophe.
1996–Present: Recognizes modern replacements Ԥ ԥ, Ӷ ӷ.

Training Details

Training dataset: Grundik/abkhaz-cyrillic-dataset-paddle
Training samples: 37,514 image + transcription pairs (.tif + .txt)
Training regime: Single-pass training on the full curated dataset
Source materials: Mix of synthetic data (38 fonts with full Abkhaz Cyrillic coverage) and real-world scans from apsnyteka.org (newspapers, books, archives)

Repository Structure

Grundik/abkhaz-cyrillic-ocr-paddle/
├── inference/
│   ├── inference.json          # Model architecture (PIR format)
│   ├── inference.pdiparams     # Model weights (73 MB)
│   └── inference.yml           # Inference metadata
├── best_model/
│   ├── model.pdparams          # Best checkpoint weights (for fine-tuning)
│   └── model.pdopt             # Optimizer state (for resuming training)
├── abk_cyrillic_dict.txt       # Custom character dictionary
└── config.yml                  # Training configuration

Installation

1. Install PaddlePaddle (GPU recommended)

# For CUDA 13.0 (adjust for your CUDA version)
pip install paddlepaddle-gpu==3.2.1 -i https://www.paddlepaddle.org.cn/packages/stable/cu130/

# Or CPU-only version
pip install paddlepaddle

2. Install dependencies

pip install paddleocr==3.7.0 paddlex==3.7.2 \
    shapely scikit-image pyclipper lmdb tqdm numpy rapidfuzz \
    opencv-contrib-python cython Pillow pyyaml requests \
    albumentations albucore packaging huggingface-hub

Or use the full requirements.txt:

paddleocr==3.7.0
paddlex==3.7.2
shapely
scikit-image
pyclipper
lmdb
tqdm
numpy
rapidfuzz
opencv-contrib-python
cython
Pillow
pyyaml
requests
albumentations
albucore
packaging
huggingface-hub

Usage

Quick Start (automatic model download)

The included ocr_paddle_kyr.py script automatically downloads the model from Hugging Face on first run:

python3 ocr_paddle_kyr.py path/to/image.tif
python3 ocr_paddle_kyr.py path/to/image.tif --save output.txt

Manual model loading

Download the model files from this repository and point the script to them:

python3 ocr_paddle_kyr.py image.tif --model-dir ./path/to/inference/

Using PaddleOCR API directly

from paddleocr import PaddleOCR

ocr = PaddleOCR(
    use_doc_orientation_classify=False,
    use_doc_unwarping=False,
    use_textline_orientation=False,
    text_detection_model_name="PP-OCRv5_server_det",
    text_recognition_model_name="PP-OCRv5_server_rec",
    text_recognition_model_dir="./inference/",  # path to downloaded inference/ folder
)

result = ocr.predict("image.tif")
for res in result:
    data = res.json["res"]
    for text, score in zip(data["rec_texts"], data["rec_scores"]):
        print(f"{score:.3f}  {text}")

Full Inference Script

Below is the complete ocr_paddle_kyr.py script with smart line merging, YX-sorting, and automatic model download. Save it as ocr_paddle_kyr.py and run directly:

#!/usr/bin/env python3
"""
OCR-скрипт на базе PaddleOCR с кастомной абхазской кириллической моделью.
Модель: Grundik/abkhaz-cyrillic-ocr-paddle (https://huggingface.co/Grundik/abkhaz-cyrillic-ocr-paddle)

Особенности:
- YX-сортировка с группировкой в строки (авто-порог по медианной высоте).
- Умная склейка: не ставит пробел после дефиса или перед ним.
- Автоматическая загрузка модели с Hugging Face при первом запуске.

Использование:
  python3 ocr_paddle_kyr.py <путь_к_файлу>
  python3 ocr_paddle_kyr.py <файл> --save out.txt
  python3 ocr_paddle_kyr.py <файл> --model-dir ./my_model/inference
  python3 ocr_pyr_kyr.py <файл> --y-threshold 25
  python3 ocr_paddle_kyr.py <файл> --no-sort
"""
import argparse
import os
import sys
from pathlib import Path
from PIL import Image

# ─── Настройки модели ────────────────────────────────────────────────────
# По умолчанию модель ищется в ./inference/ или загружается с HF.
# Переопределить можно через --model-dir или переменную окружения PADDLE_MODEL_DIR.
HF_REPO_ID = "Grundik/abkhaz-cyrillic-ocr-paddle"
DEFAULT_MODEL_DIR = "./inference"
DET_MODEL = "PP-OCRv5_server_det"
REC_MODEL_NAME = "PP-OCRv5_server_rec"


def ensure_model(model_dir: str) -> str:
    """Проверяет наличие модели локально, при отсутствии — скачивает с HF."""
    model_path = Path(model_dir)
    if model_path.exists() and (model_path / "inference.pdiparams").exists():
        return str(model_path)

    print(f"Модель не найдена в {model_dir}.")
    print(f"Скачиваю {HF_REPO_ID} с Hugging Face...")
    try:
        from huggingface_hub import snapshot_download
        downloaded = snapshot_download(
            repo_id=HF_REPO_ID,
            allow_patterns=["inference/*"],
        )
        return os.path.join(downloaded, "inference")
    except ImportError:
        print("Установите huggingface-hub: pip install huggingface-hub", file=sys.stderr)
        sys.exit(1)
    except Exception as e:
        print(f"Ошибка загрузки: {e}", file=sys.stderr)
        sys.exit(1)


# ─── Склейка строк ─────────────────────────────────────────────────────
def smart_join(parts):
    """
    Склеивает части строки, не ставя пробел:
    - после дефиса в конце предыдущей части,
    - перед дефисом в начале следующей части.
    """
    result = ""
    for i, part in enumerate(parts):
        part = part.strip()
        if not part:
            continue
        if i == 0 or not result:
            result = part
        elif result.endswith("-") or part.startswith("-"):
            result += part
        else:
            result += " " + part
    return result


def sort_and_merge_lines(texts, scores, boxes, y_threshold=None):
    """
    Сортирует фрагменты в порядке чтения и склеивает их в строки.
    y_threshold — если None, вычисляется как 0.5 от медианной высоты.
    Возвращает список (text, avg_score).
    """
    if not boxes:
        return []

    # Авто-порог по медианной высоте
    heights = [b[3] - b[1] for b in boxes]
    if y_threshold is None:
        sorted_h = sorted(heights)
        median_h = sorted_h[len(sorted_h) // 2]
        y_threshold = max(int(median_h * 0.5), 5)
        print(f"Авто y_threshold: {y_threshold} (медианная высота: {median_h})")

    # Собираем фрагменты
    items = []
    for text, score, box in zip(texts, scores, boxes):
        x1, y1, x2, y2 = box
        items.append({
            "text": text.strip(),
            "score": float(score),
            "cy": (y1 + y2) / 2,
            "cx": (x1 + x2) / 2,
        })

    # Сортируем по Y
    items.sort(key=lambda i: i["cy"])

    # Группируем в строки
    lines = []
    current_line = [items[0]]
    current_cy = items[0]["cy"]
    for item in items[1:]:
        if abs(item["cy"] - current_cy) <= y_threshold:
            current_line.append(item)
            current_cy = sum(i["cy"] for i in current_line) / len(current_line)
        else:
            lines.append(current_line)
            current_line = [item]
            current_cy = item["cy"]
    if current_line:
        lines.append(current_line)

    # Внутри строки — по X, склеиваем
    result = []
    for line in lines:
        line.sort(key=lambda i: i["cx"])
        merged_text = smart_join([i["text"] for i in line])
        avg_score = sum(i["score"] for i in line) / len(line)
        result.append((merged_text, avg_score))
    return result


# ─── Препроцессинг ─────────────────────────────────────────────────────
def preprocess(img_path: Path, max_side: int) -> Path:
    """Ресайз изображения, если оно слишком большое."""
    img = Image.open(img_path).convert("RGB")
    print(f"Оригинальный размер: {img.size}")
    if max(img.size) > max_side:
        r = max_side / max(img.size)
        img = img.resize((int(img.width * r), int(img.height * r)), Image.LANCZOS)
        print(f"Уменьшено до: {img.size}")
    else:
        print(f"Ресайз не нужен: {img.size}")
    out = Path("/tmp") / (img_path.stem + "_ocr_input.png")
    img.save(out)
    return out


# ─── OCR ───────────────────────────────────────────────────────────────
def run_ocr(img_path: Path, model_dir: str, do_sort: bool = True, y_threshold=None):
    """Запускает PaddleOCR и возвращает список (текст, score)."""
    from paddleocr import PaddleOCR

    print(f"Модель: {model_dir}")
    print(f"Сортировка: {'да' if do_sort else 'нет'}")
    print()

    ocr = PaddleOCR(
        use_doc_orientation_classify=False,
        use_doc_unwarping=False,
        use_textline_orientation=False,
        text_detection_model_name=DET_MODEL,
        text_recognition_model_name=REC_MODEL_NAME,
        text_recognition_model_dir=model_dir,
    )

    result = ocr.predict(str(img_path))
    lines = []
    for res in result:
        data = res.json["res"]
        texts = data["rec_texts"]
        scores = data["rec_scores"]
        boxes = data["rec_boxes"]
        if do_sort:
            lines.extend(sort_and_merge_lines(texts, scores, boxes, y_threshold))
        else:
            lines.extend(zip(texts, [float(s) for s in scores]))
    return lines


# ─── main ──────────────────────────────────────────────────────────────
def main():
    parser = argparse.ArgumentParser(
        description="OCR через PaddleOCR с абхазской кириллической моделью (HF: Grundik/abkhaz-cyrillic-ocr-paddle)"
    )
    parser.add_argument("input", help="Путь к файлу изображения")
    parser.add_argument("--model-dir", type=str,
                        default=os.environ.get("PADDLE_MODEL_DIR", DEFAULT_MODEL_DIR),
                        help=f"Путь к inference-папке модели (по умолчанию: {DEFAULT_MODEL_DIR} или $PADDLE_MODEL_DIR)")
    parser.add_argument("--max-side", type=int, default=2000,
                        help="Максимальная длина стороны при препроцессинге (по умолчанию 2000)")
    parser.add_argument("--save", type=str, default=None,
                        help="Сохранить результат в файл")
    parser.add_argument("--y-threshold", type=int, default=None,
                        help="Порог группировки строк по Y (по умолчанию: авто)")
    parser.add_argument("--no-sort", action="store_true",
                        help="Не сортировать и не склеивать")
    args = parser.parse_args()

    img_path = Path(args.input)
    if not img_path.exists():
        print(f"Файл не найден: {img_path}", file=sys.stderr)
        sys.exit(1)

    model_dir = ensure_model(args.model_dir)
    prepared = preprocess(img_path, args.max_side)
    lines = run_ocr(prepared, model_dir,
                    do_sort=not args.no_sort,
                    y_threshold=args.y_threshold)

    print("---")
    print(f"Строк: {len(lines)}")
    print()
    for text, score in lines:
        print(f"{score:.3f}  {text}")

    if args.save:
        out = Path(args.save)
        out.write_text("\n".join(t for t, _ in lines), encoding="utf-8")
        print(f"\nСохранено в: {out}")


if __name__ == "__main__":
    main()

Special Characters (Roman Numerals & Indices)

The charset also includes Roman numerals (represented by upper and lower case Latin letters: I, V, X, L, etc.) as well as superscripts and subscripts.

Current performance: Recognized reasonably well on clean data.
Future work: Requires further fine-tuning for noisy historical documents. Included to support future community-driven improvements.

Related Resources & Cross-References

This model is part of a broader ecosystem for Abkhaz language digitalization:

📊 Training Datasets

Abkhaz Cyrillic Dataset for PaddleOCR — 37,514 image+transcription pairs used to train this model
Abkhaz Cyrillic OCR Dataset (Tesseract) — 9,497 TIFF+BOX pairs for Tesseract training
Abkhaz Mkhedruli OCR Dataset (Tesseract) — 4,221 TIFF+BOX pairs for historical Mkhedruli script
Abkhaz Mkhedruli Dataset for PaddleOCR — 7,065 image+transcription pairs

🤖 OCR Models

Abkhaz Cyrillic OCR (Tesseract) — Tesseract LSTM model for Cyrillic
Abkhaz Mkhedruli OCR (Tesseract) — Tesseract LSTM model for Mkhedruli
Abkhaz Mkhedruli OCR (PaddleOCR) — PaddleOCR Recognition model for Mkhedruli
Abkhaz Cyrillic OCR (Florence-2) — Experimental vision-language model

🏷️ NLP Pipeline

Abkhaz POS Tagger (XLM-RoBERTa) — Part-of-speech tagging
Abkhaz Lemmatizer (ByT5) — Byte-level lemmatization

🎙️ Speech

OmniVoice Abkhazian Finetuned — Text-to-speech for Abkhaz

🛠️ Companion Tools (Coming Soon)

Custom utility scripts (in awk and elisp) are being prepared for release on GitHub. These tools will provide:

Batch transliteration from historical Mkhedruli to modern Abkhaz Cyrillic.
Emacs integration for seamless in-editor conversion (word, region, or file modes).
CLI modes for pipeline automation (translit-only or original+translit).

Citation

If you use this model in your research or digital preservation projects, please cite:

@misc{abkhaz_cyrillic_ocr_paddle_2024,
  author = {Ionov, Marat and Bagrat Shinkuba Fund},
  title = {Abkhaz Cyrillic OCR Model (PaddleOCR)},
  year = {2024},
  publisher = {Hugging Face},
  url = {https://huggingface.co/Grundik/abkhaz-cyrillic-ocr-paddle}
}

Author & Acknowledgments

Model trained, evaluated, and curated by Marat Ionov as part of the digital preservation initiative, supported by the Bagrat Shinkuba Fund.

License

This model is licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License (CC BY-NC-SA 4.0).

Downloads last month: -; Downloads are not tracked for this model. How to track