| --- |
| license: other |
| tags: |
| - object-detection |
| - yolov8 |
| - yolo |
| - ultralytics |
| - gradient-boosting |
| - scikit-learn |
| - comics |
| - computer-vision |
| - panel-detection |
| - reading-order |
| library_name: ultralytics |
| pipeline_tag: object-detection |
| base_model: |
| - Ultralytics/YOLOv8 |
| --- |
| |
| # Inkwell Panel Models |
|
|
| Two models for automated comic panel processing, trained on the same |
| hand-annotated comic page corpus: |
|
|
| 1. **[Panel Detector](#1-panel-detector-yolov8-n)** — locates panel bounding |
| boxes on a comic page (object detection, YOLOv8-n). |
| 2. **[Panel Reading Order](#2-panel-reading-order-gradient-boosting-classifier)** — |
| given a page's detected panels, predicts the order a reader should view |
| them in (pairwise gradient-boosted classifier). |
|
|
| They're meant to be used together: run the detector on a page to get panel |
| boxes, then run the ordering model on those boxes to get a reading sequence. |
| Each model has its **own license** — see [License](#license) below, they are |
| not the same. |
|
|
| --- |
|
|
| ## 1. Panel Detector (YOLOv8-n) |
|
|
| A YOLOv8-nano object detection model that locates **comic panels** on scanned |
| or digital comic book pages. |
|
|
| ### Model description |
|
|
| - **Task:** single-class object detection |
| - **Class:** `panel` (`class_id 0`) — the rectangular (or near-rectangular) |
| bounded region of a single comic panel on a page. The model does not |
| distinguish panel shape, gutter, speech balloons, or reading order — it |
| only predicts the bounding box of each panel. |
| - **Architecture:** [YOLOv8-n](https://github.com/ultralytics/ultralytics) |
| (nano), fine-tuned from the pretrained `yolov8n.pt` COCO checkpoint. |
| - **Input size:** 640×640 |
| - **What it's for:** automatically detecting panel boundaries on a comic page |
| so downstream tools can crop, reorder, or reflow panels — e.g. guided |
| panel-by-panel reading, panel extraction for translation/OCR pipelines, or |
| building reading-order datasets. This is the "edge" tier model: |
| small and fast enough for on-device / mobile inference. |
|
|
| ### Training data |
|
|
| - **~3,800 annotated comic book pages** (3,380 train / 423 validation, plus a |
| held-out test split) drawn from a range of **genres and art styles** |
| (e.g. superhero, manga-influenced, indie/alternative, all-ages), so the |
| model isn't overfit to one visual style or layout convention. |
| - Every panel bounding box was **hand-tuned** — each page's annotation was |
| manually created or corrected before being ingested into the training set. |
| - Pages with **no panels** (e.g. full-page splash art, covers) were included |
| with empty label files as negative examples, so the model also learns to |
| correctly predict *zero* panels on non-paneled pages. |
| - Labels use standard YOLO format: `<class_id> <cx> <cy> <w> <h>`, normalized |
| to 0–1, single class (`panel`). |
|
|
| ### Training procedure |
|
|
| Trained from the `yolov8n.pt` pretrained checkpoint using |
| [Ultralytics YOLOv8](https://github.com/ultralytics/ultralytics). |
|
|
| | Parameter | Value | |
| |---|---| |
| | Base weights | `yolov8n.pt` | |
| | Image size | 640 | |
| | Epochs | 300 (early-stopping patience 30) | |
| | Batch size | 16 | |
| | Mosaic augmentation | 1.0 | |
| | Mixup augmentation | 0.0 | |
| | Train / Val pages | 3,380 / 423 | |
|
|
| ### Results |
|
|
| | Model | Type | Epochs | Batch | imgsz | Train pages | Val pages | Precision | Recall | mAP@50 | mAP@50-95 | |
| |---|---|---|---|---|---|---|---|---|---|---| |
| | comic-panels-v1 | nano | 300 | 16 | 640 | 3,380 | 423 | 0.9681 | 0.9331 | 0.9769 | 0.9249 | |
|
|
| ### Limitations |
|
|
| - Trained on comic *pages* — performance on non-comic images, storyboard art, |
| or extremely non-standard/artistic panel layouts (irregular borders, |
| panel-less "flow" layouts) is not guaranteed. |
| - Detects panel bounding boxes only; it does not infer reading order, panel |
| type, or contents (speech bubbles, characters, etc.). That's handled by the |
| second model in this repo. |
| - Single-class model — all detections are labeled `panel` regardless of |
| panel shape or size. |
|
|
| --- |
|
|
| ## 2. Panel Reading Order (Gradient Boosting Classifier) |
|
|
| A pairwise reading-order model. Given the panel boxes on a comic page (e.g. |
| from the detector above, or any other source), it predicts the order a |
| reader should view them in — handling non-obvious layouts like Z-paths, |
| tall panels that span both a left and right column, splash panels, and |
| nested/overlapping panels, without hardcoded row/column rules. |
|
|
| ### Model description |
|
|
| - **Task:** pairwise ranking / reading-order prediction over a variable |
| number of panels on a page. |
| - **Architecture:** `sklearn.ensemble.GradientBoostingClassifier` |
| (100 estimators, max depth 4, learning rate 0.1). For every ordered pair of |
| panels `(a, b)` on a page, 14 geometric features are computed (relative |
| position, vertical/horizontal overlap, aspect-ratio, tallness, page-column |
| side, panel-count normalization, etc. — see `FEATURE_NAMES` in the |
| training script) and the classifier predicts `P(a is read before b)`. |
| Each panel's final rank comes from summing its "beats" probability against |
| every other panel on the page and sorting descending. |
| - **Not a neural network** — pure gradient-boosted trees over geometry |
| features, so inference is microseconds per page. |
| - **Shipped format:** `panel-order-model.onnx`, exported from the trained |
| scikit-learn classifier via `skl2onnx`. GradientBoostingClassifier (rather |
| than the faster HistGradientBoostingClassifier used by an internal v2 |
| candidate) is used specifically because it converts to ONNX reliably, |
| letting this model run in-browser via onnxruntime-web. |
|
|
| ### Training data |
|
|
| Same underlying corpus as the panel detector: comic pages with hand-tuned |
| panel boxes **and** a hand-verified reading-order index per panel, pulled |
| from `panels.json` annotations across the same range of genres and art |
| styles. Only pages with ≥2 panels and an `approved` annotation flag are used |
| (single-panel and unapproved pages are skipped, since there's no ordering |
| signal to learn from a single panel). |
|
|
| ### Training procedure |
|
|
| - Trained with leave-one-file-out cross-validation: for each comic file, the |
| model is trained on every *other* file and evaluated on the held-out one, |
| so reported accuracy reflects generalization to unseen books, not just |
| unseen pages of a book already seen in training. |
| - Every panel pair on a page contributes two training examples (`a before b` |
| and `b before a`), so the effective training set is much larger than the |
| page count alone suggests. |
|
|
| ### Results |
|
|
| Leave-one-file-out cross-validation, page-level exact-match accuracy (the |
| predicted order must match the human-annotated order for every panel on the |
| page to count as correct): |
|
|
| | Metric | Value | |
| |---|---| |
| | Pages evaluated | 3,610 | |
| | Pages perfectly ordered | 3,493 | |
| | **Page-level accuracy** | **96.76%** | |
|
|
| Accuracy by layout pattern (a page can match more than one pattern): |
|
|
| | Pattern | Accuracy | Pages | |
| |---|---|---| |
| | Simple grid | 98.58% | 494 | |
| | 2–3 panels | 99.52% | 419 | |
| | 4–6 panels | 97.99% | 2,585 | |
| | Unequal columns | 96.76% | 1,451 | |
| | Has overlapping panels | 95.97% | 2,233 | |
| | Has splash panel | 95.48% | 1,770 | |
| | 7–10 panels | 91.06% | 559 | |
| | Has nested panels | 91.91% | 544 | |
| | 11+ panels | 72.34% | 47 | |
|
|
| Accuracy degrades on the densest, least common layouts (11+ panels on a |
| single page), which is also the smallest evaluation bucket. |
|
|
| ### Limitations |
|
|
| - Accuracy drops on pages with very high panel counts (11+) — these are rare |
| in the training corpus, so the model has seen fewer examples of that |
| layout complexity. |
| - Predicts order from **geometry only** — it doesn't read panel content, so |
| narratively-driven reading orders that contradict geometric layout (rare, |
| but they exist in some comics) won't be captured. |
| - Expects panel boxes as input (from the detector above or another source); |
| it does not detect panels itself. |
|
|
| --- |
|
|
| ## License |
|
|
| **The two models in this repo are licensed differently — check which one |
| you're using.** |
|
|
| ### Panel Detector — AGPL-3.0 |
|
|
| The panel detector is a **derivative of [Ultralytics YOLOv8](https://github.com/ultralytics/ultralytics)**, |
| which Ultralytics distributes under the |
| [GNU Affero General Public License v3.0 (AGPL-3.0)](https://www.gnu.org/licenses/agpl-3.0.en.html). |
| Because these weights were produced by fine-tuning Ultralytics' pretrained |
| `yolov8n.pt` checkpoint, the detector weights are likewise distributed under |
| **AGPL-3.0**. |
|
|
| In practice, this means: |
|
|
| - You are free to use, modify, and redistribute this model, **provided any |
| distribution (including source code) also uses/discloses source under |
| AGPL-3.0**. |
| - If you run this model (or a modified version of it) as part of a network |
| service (e.g. a hosted API or web app), AGPL-3.0's network-use clause |
| requires you to make the complete corresponding source code of that |
| service available to its users. |
| - If you need to use this model in a closed-source or proprietary product |
| without these obligations, Ultralytics offers a commercial |
| [Enterprise License](https://www.ultralytics.com/license) — see their site |
| for details. That commercial license applies to the underlying YOLOv8 |
| architecture/code; you would need to confirm licensing terms for these |
| specific fine-tuned weights with their author before using them under |
| such a license. |
|
|
| Full AGPL-3.0 license text: https://www.gnu.org/licenses/agpl-3.0.en.html |
|
|
| #### Attribution |
|
|
| This model builds on: |
|
|
| ``` |
| Ultralytics YOLOv8 |
| https://github.com/ultralytics/ultralytics |
| Copyright (C) Ultralytics |
| Licensed under AGPL-3.0 |
| ``` |
|
|
| If you use the detector, please also cite Ultralytics YOLOv8: |
|
|
| ```bibtex |
| @software{yolov8_ultralytics, |
| author = {Glenn Jocher and Ayush Chaurasia and Jing Qiu}, |
| title = {YOLO by Ultralytics}, |
| version = {8.0.0}, |
| year = {2023}, |
| url = {https://github.com/ultralytics/ultralytics}, |
| license = {AGPL-3.0} |
| } |
| ``` |
|
|
| ### Panel Reading Order — MIT |
|
|
| The reading-order model is trained from scratch on top of |
| `scikit-learn` (BSD-3-Clause) and does not derive from or embed any AGPL |
| code or weights, so it does not carry the detector's AGPL-3.0 obligations. |
| It's released under the **MIT License**. |
