SharpMeow commited on
Commit
99b82f1
·
verified ·
1 Parent(s): 242088a

Crackle Check 0.1.0 from Scrolls-private 21208bb0

Browse files
Files changed (10) hide show
  1. LICENSE +201 -0
  2. NOTICE +17 -0
  3. README.md +56 -6
  4. VENDORED.md +9 -0
  5. app.py +207 -0
  6. crackle.py +139 -0
  7. engine.py +541 -0
  8. gate.py +138 -0
  9. requirements.txt +22 -0
  10. upload.py +113 -0
LICENSE ADDED
@@ -0,0 +1,201 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ Apache License
2
+ Version 2.0, January 2004
3
+ http://www.apache.org/licenses/
4
+
5
+ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
6
+
7
+ 1. Definitions.
8
+
9
+ "License" shall mean the terms and conditions for use, reproduction,
10
+ and distribution as defined by Sections 1 through 9 of this document.
11
+
12
+ "Licensor" shall mean the copyright owner or entity authorized by
13
+ the copyright owner that is granting the License.
14
+
15
+ "Legal Entity" shall mean the union of the acting entity and all
16
+ other entities that control, are controlled by, or are under common
17
+ control with that entity. For the purposes of this definition,
18
+ "control" means (i) the power, direct or indirect, to cause the
19
+ direction or management of such entity, whether by contract or
20
+ otherwise, or (ii) ownership of fifty percent (50%) or more of the
21
+ outstanding shares, or (iii) beneficial ownership of such entity.
22
+
23
+ "You" (or "Your") shall mean an individual or Legal Entity
24
+ exercising permissions granted by this License.
25
+
26
+ "Source" form shall mean the preferred form for making modifications,
27
+ including but not limited to software source code, documentation
28
+ source, and configuration files.
29
+
30
+ "Object" form shall mean any form resulting from mechanical
31
+ transformation or translation of a Source form, including but
32
+ not limited to compiled object code, generated documentation,
33
+ and conversions to other media types.
34
+
35
+ "Work" shall mean the work of authorship, whether in Source or
36
+ Object form, made available under the License, as indicated by a
37
+ copyright notice that is included in or attached to the work
38
+ (an example is provided in the Appendix below).
39
+
40
+ "Derivative Works" shall mean any work, whether in Source or Object
41
+ form, that is based on (or derived from) the Work and for which the
42
+ editorial revisions, annotations, elaborations, or other modifications
43
+ represent, as a whole, an original work of authorship. For the purposes
44
+ of this License, Derivative Works shall not include works that remain
45
+ separable from, or merely link (or bind by name) to the interfaces of,
46
+ the Work and Derivative Works thereof.
47
+
48
+ "Contribution" shall mean any work of authorship, including
49
+ the original version of the Work and any modifications or additions
50
+ to that Work or Derivative Works thereof, that is intentionally
51
+ submitted to Licensor for inclusion in the Work by the copyright owner
52
+ or by an individual or Legal Entity authorized to submit on behalf of
53
+ the copyright owner. For the purposes of this definition, "submitted"
54
+ means any form of electronic, verbal, or written communication sent
55
+ to the Licensor or its representatives, including but not limited to
56
+ communication on electronic mailing lists, source code control systems,
57
+ and issue tracking systems that are managed by, or on behalf of, the
58
+ Licensor for the purpose of discussing and improving the Work, but
59
+ excluding communication that is conspicuously marked or otherwise
60
+ designated in writing by the copyright owner as "Not a Contribution."
61
+
62
+ "Contributor" shall mean Licensor and any individual or Legal Entity
63
+ on behalf of whom a Contribution has been received by Licensor and
64
+ subsequently incorporated within the Work.
65
+
66
+ 2. Grant of Copyright License. Subject to the terms and conditions of
67
+ this License, each Contributor hereby grants to You a perpetual,
68
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
69
+ copyright license to reproduce, prepare Derivative Works of,
70
+ publicly display, publicly perform, sublicense, and distribute the
71
+ Work and such Derivative Works in Source or Object form.
72
+
73
+ 3. Grant of Patent License. Subject to the terms and conditions of
74
+ this License, each Contributor hereby grants to You a perpetual,
75
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
76
+ (except as stated in this section) patent license to make, have made,
77
+ use, offer to sell, sell, import, and otherwise transfer the Work,
78
+ where such license applies only to those patent claims licensable
79
+ by such Contributor that are necessarily infringed by their
80
+ Contribution(s) alone or by combination of their Contribution(s)
81
+ with the Work to which such Contribution(s) was submitted. If You
82
+ institute patent litigation against any entity (including a
83
+ cross-claim or counterclaim in a lawsuit) alleging that the Work
84
+ or a Contribution incorporated within the Work constitutes direct
85
+ or contributory patent infringement, then any patent licenses
86
+ granted to You under this License for that Work shall terminate
87
+ as of the date such litigation is filed.
88
+
89
+ 4. Redistribution. You may reproduce and distribute copies of the
90
+ Work or Derivative Works thereof in any medium, with or without
91
+ modifications, and in Source or Object form, provided that You
92
+ meet the following conditions:
93
+
94
+ (a) You must give any other recipients of the Work or
95
+ Derivative Works a copy of this License; and
96
+
97
+ (b) You must cause any modified files to carry prominent notices
98
+ stating that You changed the files; and
99
+
100
+ (c) You must retain, in the Source form of any Derivative Works
101
+ that You distribute, all copyright, patent, trademark, and
102
+ attribution notices from the Source form of the Work,
103
+ excluding those notices that do not pertain to any part of
104
+ the Derivative Works; and
105
+
106
+ (d) If the Work includes a "NOTICE" text file as part of its
107
+ distribution, then any Derivative Works that You distribute must
108
+ include a readable copy of the attribution notices contained
109
+ within such NOTICE file, excluding those notices that do not
110
+ pertain to any part of the Derivative Works, in at least one
111
+ of the following places: within a NOTICE text file distributed
112
+ as part of the Derivative Works; within the Source form or
113
+ documentation, if provided along with the Derivative Works; or,
114
+ within a display generated by the Derivative Works, if and
115
+ wherever such third-party notices normally appear. The contents
116
+ of the NOTICE file are for informational purposes only and
117
+ do not modify the License. You may add Your own attribution
118
+ notices within Derivative Works that You distribute, alongside
119
+ or as an addendum to the NOTICE text from the Work, provided
120
+ that such additional attribution notices cannot be construed
121
+ as modifying the License.
122
+
123
+ You may add Your own copyright statement to Your modifications and
124
+ may provide additional or different license terms and conditions
125
+ for use, reproduction, or distribution of Your modifications, or
126
+ for any such Derivative Works as a whole, provided Your use,
127
+ reproduction, and distribution of the Work otherwise complies with
128
+ the conditions stated in this License.
129
+
130
+ 5. Submission of Contributions. Unless You explicitly state otherwise,
131
+ any Contribution intentionally submitted for inclusion in the Work
132
+ by You to the Licensor shall be under the terms and conditions of
133
+ this License, without any additional terms or conditions.
134
+ Notwithstanding the above, nothing herein shall supersede or modify
135
+ the terms of any separate license agreement you may have executed
136
+ with Licensor regarding such Contributions.
137
+
138
+ 6. Trademarks. This License does not grant permission to use the trade
139
+ names, trademarks, service marks, or product names of the Licensor,
140
+ except as required for reasonable and customary use in describing the
141
+ origin of the Work and reproducing the content of the NOTICE file.
142
+
143
+ 7. Disclaimer of Warranty. Unless required by applicable law or
144
+ agreed to in writing, Licensor provides the Work (and each
145
+ Contributor provides its Contributions) on an "AS IS" BASIS,
146
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
147
+ implied, including, without limitation, any warranties or conditions
148
+ of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
149
+ PARTICULAR PURPOSE. You are solely responsible for determining the
150
+ appropriateness of using or redistributing the Work and assume any
151
+ risks associated with Your exercise of permissions under this License.
152
+
153
+ 8. Limitation of Liability. In no event and under no legal theory,
154
+ whether in tort (including negligence), contract, or otherwise,
155
+ unless required by applicable law (such as deliberate and grossly
156
+ negligent acts) or agreed to in writing, shall any Contributor be
157
+ liable to You for damages, including any direct, indirect, special,
158
+ incidental, or consequential damages of any character arising as a
159
+ result of this License or out of the use or inability to use the
160
+ Work (including but not limited to damages for loss of goodwill,
161
+ work stoppage, computer failure or malfunction, or any and all
162
+ other commercial damages or losses), even if such Contributor
163
+ has been advised of the possibility of such damages.
164
+
165
+ 9. Accepting Warranty or Additional Liability. While redistributing
166
+ the Work or Derivative Works thereof, You may choose to offer,
167
+ and charge a fee for, acceptance of support, warranty, indemnity,
168
+ or other liability obligations and/or rights consistent with this
169
+ License. However, in accepting such obligations, You may act only
170
+ on Your own behalf and on Your sole responsibility, not on behalf
171
+ of any other Contributor, and only if You agree to indemnify,
172
+ defend, and hold each Contributor harmless for any liability
173
+ incurred by, or claims asserted against, such Contributor by reason
174
+ of your accepting any such warranty or additional liability.
175
+
176
+ END OF TERMS AND CONDITIONS
177
+
178
+ APPENDIX: How to apply the Apache License to your work.
179
+
180
+ To apply the Apache License to your work, attach the following
181
+ boilerplate notice, with the fields enclosed by brackets "[]"
182
+ replaced with your own identifying information. (Don't include
183
+ the brackets!) The text should be enclosed in the appropriate
184
+ comment syntax for the file format. We also recommend that a
185
+ file or class name and description of purpose be included on the
186
+ same "printed page" as the copyright notice for easier
187
+ identification within third-party archives.
188
+
189
+ Copyright [yyyy] [name of copyright owner]
190
+
191
+ Licensed under the Apache License, Version 2.0 (the "License");
192
+ you may not use this file except in compliance with the License.
193
+ You may obtain a copy of the License at
194
+
195
+ http://www.apache.org/licenses/LICENSE-2.0
196
+
197
+ Unless required by applicable law or agreed to in writing, software
198
+ distributed under the License is distributed on an "AS IS" BASIS,
199
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
200
+ See the License for the specific language governing permissions and
201
+ limitations under the License.
NOTICE ADDED
@@ -0,0 +1,17 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ Crackle Check
2
+ Copyright 2026 Chase Hendrick and Scrolls contributors
3
+
4
+ The Crackle Check code (app.py, crackle.py, upload.py) and the Ink Lens files it carries (engine.py, gate.py) are
5
+ licensed under the Apache License, Version 2.0. See LICENSE.
6
+
7
+ engine.robust_np restates villa's normalize_robust and engine.hann2d villa's floored Hann blend map (both MIT,
8
+ Copyright (c) Vesuvius Challenge). When it starts, the Space downloads the following at pinned versions and uses them
9
+ under their own licenses, which still apply:
10
+
11
+ - villa's vesuvius package, https://github.com/ScrollPrize/villa at commit
12
+ 8aa507c1215f1fa8758f13d0538f7af9710e02d7: MIT License, Copyright (c) Vesuvius Challenge.
13
+ - d9v2_ft-012000.pth, https://github.com/TAUIL-Abd-Elilah/pherc0826-first-letters-search release v1.0:
14
+ MIT License, Copyright (c) TAUIL-Abd-Elilah. A fine-tune of ink_9um (https://huggingface.co/scrollprize/ink_9um,
15
+ MIT License, Copyright (c) Vesuvius Challenge).
16
+
17
+ The Space keeps no upload. Maps it returns are model outputs computed from the caller's own region.
README.md CHANGED
@@ -1,13 +1,63 @@
1
  ---
2
  title: Crackle Check
3
- emoji: 🚀
4
- colorFrom: gray
5
- colorTo: blue
6
  sdk: gradio
7
- sdk_version: 6.30.0
8
- python_version: '3.13'
9
  app_file: app.py
10
  pinned: false
 
 
 
 
 
 
 
 
 
11
  ---
12
 
13
- Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
  title: Crackle Check
3
+ emoji: 🔍
4
+ colorFrom: yellow
5
+ colorTo: gray
6
  sdk: gradio
7
+ sdk_version: 6.28.0
8
+ python_version: "3.12"
9
  app_file: app.py
10
  pinned: false
11
+ license: apache-2.0
12
+ suggested_hardware: zero-a10g
13
+ startup_duration_timeout: 1h
14
+ short_description: Is there crackle here? Upload a layer stack to check
15
+ tags:
16
+ - vesuvius-challenge
17
+ - herculaneum
18
+ - ink-detection
19
+ - zerogpu
20
  ---
21
 
22
+ # Crackle Check
23
+
24
+ Is there ink (crackle) in this patch of papyrus? Upload a small region of a segment as a **stack of surface layers**
25
+ and an openly licensed ink model says **Yes**, **Maybe** or **No signal found**, with the signal highlighted on the
26
+ middle layer.
27
+
28
+ ## How to use it
29
+
30
+ 1. Pick a region about 3 to 10 mm across (about 320 to 1024 px at 9 um) on a segment.
31
+ 2. Take the same crop from at least 17 layers centred on the surface (from a segment's `layers` folder, VC3D or
32
+ `vc_render_tifxyz`).
33
+ 3. ZIP the images (named in order, like `00.tif`, `01.tif`, ...) or save them as one multi-page TIFF, and upload.
34
+ Give the pixel size if it is not about 9 um.
35
+
36
+ A single image or screenshot is refused with the reason: the model reads 17 layers at once, and on public labelled
37
+ crops it scored at chance from one layer.
38
+
39
+ ## How it decides
40
+
41
+ The model ([d9v2](https://github.com/TAUIL-Abd-Elilah/pherc0826-first-letters-search), MIT, a fine-tune of the
42
+ Vesuvius Challenge's [`ink_9um`](https://huggingface.co/scrollprize/ink_9um)) reads the layers in the order you send
43
+ them and again reversed. The reversed run is the control. The answer is Yes when at least 3 times as many pixels pass
44
+ the control's 99th percentile as the control itself, Maybe from 1.5 times, and No signal found below that. It runs on
45
+ Ink Lens's engine ([Scrolls](https://github.com/ChaseHendrick/Scrolls/tree/main/spaces/ink-lens)).
46
+
47
+ ## Limits
48
+
49
+ - Tested on a few public PHerc0841 crops only.
50
+ - Many scrolls show little or no crackle, and ink can be there without it; No signal found is not proof of no ink.
51
+ - Built for scans of about 8 to 10 um per pixel.
52
+ - Everything shown is model output, not a reading. The Space keeps no upload.
53
+
54
+ ## API
55
+
56
+ ```python
57
+ from gradio_client import Client, handle_file
58
+ c = Client("OWNER/crackle-check")
59
+ answer_md, result, preview = c.predict(handle_file("layers.zip"), None, False, api_name="/check")
60
+ result["answer"] # Yes | Maybe | No signal found | Too small | Could not judge
61
+ ```
62
+
63
+ Code: Apache 2.0. Scan data: Vesuvius Challenge open data, CC BY-NC 4.0.
VENDORED.md ADDED
@@ -0,0 +1,9 @@
 
 
 
 
 
 
 
 
 
 
1
+ # Copied files
2
+
3
+ `engine.py` and `gate.py` are Ink Lens's, copied unchanged from ChaseHendrick/Scrolls `spaces/ink-lens/` at commit
4
+ c65e8a2 (21 Oct 2026). Copy them again, never edit them here, so both tools run the same engine and gate.
5
+
6
+ | File | SHA-256 |
7
+ | --- | --- |
8
+ | engine.py | e47f4224409e066c57e9b107656b6a3b0c2bc0cb0e8ba9b57155f94d280aac5b |
9
+ | gate.py | 1036c20467c0aa1d2e6b4970cdd4be8126aebeb19b70dee497332d29b41ba31b |
app.py ADDED
@@ -0,0 +1,207 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Crackle Check: upload a region of surface layers, learn whether an ink model sees ink (crackle) above its control.
2
+
3
+ The page and its API. Uploads are read in memory (upload.py), checked and judged by crackle.py, and run by Ink Lens's
4
+ engine (engine.py) in one ZeroGPU call that always runs the stored and the reversed layer order together. When ZeroGPU
5
+ refuses (quota, no GPU free) a region of at most CPU_SIDE px runs on the Space's CPU with one depth window. The Space
6
+ keeps nothing.
7
+
8
+ python3 app.py serve the page (on a Space, or on any computer: CUDA if present, else CPU)
9
+ """
10
+
11
+ from __future__ import annotations
12
+
13
+ import functools
14
+ import os
15
+ import sys
16
+ import threading
17
+ import time
18
+ from pathlib import Path
19
+
20
+ try:
21
+ import spaces # first, before torch: on a ZeroGPU Space it attaches the GPU to spaces.GPU functions
22
+ except ImportError:
23
+ spaces = None
24
+
25
+ HERE = Path(__file__).resolve().parent
26
+ if str(HERE) not in sys.path:
27
+ sys.path.insert(0, str(HERE))
28
+ import crackle # noqa: E402
29
+ import engine # noqa: E402
30
+ import gate # noqa: E402
31
+ import upload # noqa: E402
32
+
33
+ ON_ZERO_GPU = bool(spaces) and os.environ.get("SPACES_ZERO_GPU", "").lower() in ("1", "true")
34
+ READER = os.environ.get("CRACKLE_READER", "d9v2")
35
+ CPU_SIDE = 640
36
+ ENGINE, ENGINE_ERROR = None, None
37
+ READY = threading.Event()
38
+ CPU_SLOT = threading.Semaphore(1)
39
+ GPU_REFUSALS = ("ZeroGPU quota exceeded", "ZeroGPU illegal duration", "ZeroGPU pending credits exceeded",
40
+ "ZeroGPU queue timeout", "ZeroGPU client error")
41
+
42
+
43
+ def log(message):
44
+ print(f"[crackle-check {time.strftime('%H:%M:%S')}] {message}", flush=True)
45
+
46
+
47
+ def _load():
48
+ global ENGINE, ENGINE_ERROR
49
+ try:
50
+ ENGINE = engine.Engine(readers=(READER,))
51
+ log(f"{READER} ready in {ENGINE.startup_s} s")
52
+ except Exception as exc: # noqa: BLE001
53
+ ENGINE_ERROR = f"{type(exc).__name__}: {exc}"
54
+ log(f"reader failed to load: {ENGINE_ERROR}")
55
+ finally:
56
+ READY.set()
57
+
58
+
59
+ def startup():
60
+ threading.Thread(target=_load, name="load-reader", daemon=True).start()
61
+
62
+
63
+ def eng(wait_s=5.0):
64
+ if not READY.wait(wait_s):
65
+ raise crackle.CheckError("the model is still loading (about a minute after the Space wakes up); try again "
66
+ "shortly")
67
+ if ENGINE is None:
68
+ raise crackle.CheckError(f"the model failed to load: {ENGINE_ERROR}")
69
+ return ENGINE
70
+
71
+
72
+ def _duration(volume, windows):
73
+ return engine.gpu_seconds(engine.patches(volume.shape[1], volume.shape[2], windows, READER))
74
+
75
+
76
+ def _on_gpu(fn):
77
+ return spaces.GPU(duration=_duration)(fn) if ON_ZERO_GPU else fn
78
+
79
+
80
+ @_on_gpu
81
+ def gpu_run(volume, windows):
82
+ """Both layer orders in one call, so the map never runs without its control."""
83
+ return ENGINE.run(volume, reader=READER, windows=windows, device="cuda")
84
+
85
+
86
+ def _is_refusal(exc):
87
+ return (str(getattr(exc, "title", "") or "") in GPU_REFUSALS or "GPU task aborted" in str(exc)
88
+ or "No GPU was available" in str(exc))
89
+
90
+
91
+ def run_engine(volume, windows):
92
+ e = eng()
93
+ import torch
94
+ if ON_ZERO_GPU:
95
+ try:
96
+ return gpu_run(volume, windows)
97
+ except Exception as exc: # noqa: BLE001
98
+ if not _is_refusal(exc):
99
+ raise
100
+ note = "ZeroGPU is not available right now"
101
+ elif torch.cuda.is_available():
102
+ return e.run(volume, reader=READER, windows=windows, device="cuda")
103
+ else:
104
+ note = "No GPU here"
105
+ if max(volume.shape[1:]) > CPU_SIDE:
106
+ raise crackle.CheckError(f"{note}, and the CPU fallback takes regions up to {CPU_SIDE} px a side; crop the "
107
+ "region or try again later")
108
+ if not CPU_SLOT.acquire(blocking=False):
109
+ raise crackle.CheckError(f"{note}, and the CPU is busy with another check; try again in a minute")
110
+ try:
111
+ torch.set_num_threads(max(1, (os.cpu_count() or 2) - 1))
112
+ return e.run(volume, reader=READER, windows=1, device="cpu")
113
+ finally:
114
+ CPU_SLOT.release()
115
+
116
+
117
+ def check_file(path, um_per_px=None, flip=False):
118
+ """Crackle check of one uploaded file.
119
+
120
+ path: a ZIP of layer images, a multi-page TIFF, or an .npz with array 'volume' (layers first, 17 to 65 layers
121
+ centred on the papyrus surface, 128 to 2048 px a side). A single image is refused with the reason (one layer
122
+ is not enough). um_per_px: the pixel size of the upload in micrometres (blank or 0: about 9 um, used as it is).
123
+ flip: reverse the layer order first. Returns (answer as Markdown, result dict, preview RGB). Model output, not a
124
+ reading."""
125
+ if not path:
126
+ raise crackle.CheckError("upload a file first")
127
+ p = path if isinstance(path, str) else path.get("path")
128
+ try:
129
+ array, kind, note = upload.read(p)
130
+ finally:
131
+ try:
132
+ Path(p).unlink()
133
+ except OSError:
134
+ pass
135
+ return crackle.check(array, kind, run_engine, um_per_px=um_per_px or None, flip=bool(flip), notes=[note])
136
+
137
+
138
+ INTRO = """# Crackle Check
139
+
140
+ Is there ink in this patch of papyrus? Upload a small region of a segment **as a stack of surface layers** and an
141
+ openly licensed ink model tells you whether it sees an ink signal (the "crackle" texture dried ink leaves on the
142
+ papyrus) stronger than its own control.
143
+ """
144
+
145
+ ABOUT = """**What crackle is.** On some scrolls, dried carbon ink left a fine cracked texture on the papyrus surface that
146
+ shows up in CT. It is the clearest ink signal in the scans, but many scrolls show little or none, and ink can be there
147
+ without visible crackle.
148
+
149
+ **How this decides.** The model ([d9v2](https://github.com/TAUIL-Abd-Elilah/pherc0826-first-letters-search), MIT, a
150
+ fine-tune of the Vesuvius Challenge's [`ink_9um`](https://huggingface.co/scrollprize/ink_9um)) reads your layers in
151
+ the order you send them and again reversed. The reversed run is the control: ink models read ink in depth order, so
152
+ the reversed run shows what the model "finds" in a region by chance. The answer is **Yes** when at least 3 times as
153
+ many pixels pass the control's 99th percentile as the control itself, **Maybe** from 1.5 times, and **No signal
154
+ found** below that.
155
+
156
+ **Why one image is not enough.** The model reads 17 layers at once. On public labelled crops it tells ink from plain
157
+ papyrus well from a layer stack, and at chance from a single layer or screenshot, so the tool asks for layers.
158
+
159
+ **Limits.** Tested on a few public PHerc0841 crops only. Built for scans of about 8 to 10 um per pixel; give the pixel
160
+ size if yours differs. "No signal found" is not proof of no ink. Everything shown is model output, not a reading.
161
+ """
162
+
163
+ HOWTO = """1. Pick a region about 3 to 10 mm across (about 320 to 1024 px at 9 um) on a segment.
164
+ 2. Take the same crop from at least 17 layers centred on the surface (for a segment's `layers` folder with 65 layers,
165
+ layers 22 to 42 or all of them).
166
+ 3. ZIP the images (named in order, like `00.tif`, `01.tif`, ...) or save them as one multi-page TIFF, and upload."""
167
+
168
+
169
+ def build_demo():
170
+ import gradio as gr
171
+
172
+ def page(f, um, flip):
173
+ try:
174
+ return check_file(f, um, flip)
175
+ except (crackle.CheckError, upload.UploadError, engine.EngineError, gate.GateError) as exc:
176
+ return str(exc), None, None
177
+ except Exception as exc: # noqa: BLE001
178
+ import traceback
179
+ log(f"check failed: {traceback.format_exc()}")
180
+ raise gr.Error(f"{type(exc).__name__}: {str(exc)[:300]}") from None
181
+
182
+ with gr.Blocks(title="Crackle Check") as demo:
183
+ gr.Markdown(INTRO)
184
+ with gr.Row():
185
+ with gr.Column(scale=2, min_width=300):
186
+ f = gr.File(label="Layers: ZIP of images, multi-page TIFF, or .npz",
187
+ file_types=[".zip", ".tif", ".tiff", ".npz", ".png", ".jpg", ".jpeg"], type="filepath")
188
+ um = gr.Number(value=None, label="Pixel size in um (blank: about 9 um)", precision=3)
189
+ flip = gr.Checkbox(False, label="Flip layer order")
190
+ go = gr.Button("Check for crackle", variant="primary")
191
+ with gr.Accordion("How to make the upload", open=False):
192
+ gr.Markdown(HOWTO)
193
+ with gr.Column(scale=3, min_width=320):
194
+ md = gr.Markdown("Upload a layer stack, then check.")
195
+ img = gr.Image(label="Middle layer, ink signal above the control in amber", type="numpy",
196
+ format="png")
197
+ with gr.Accordion("Details", open=False):
198
+ res = gr.JSON(label="Result")
199
+ gr.Markdown(ABOUT)
200
+ go.click(page, [f, um, flip], [md, res, img], api_name="check", concurrency_id="gpu",
201
+ concurrency_limit=2)
202
+ return demo
203
+
204
+
205
+ if __name__ == "__main__":
206
+ startup()
207
+ build_demo().queue(max_size=16).launch()
crackle.py ADDED
@@ -0,0 +1,139 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Crackle Check: does an uploaded region of surface layers show an ink signal stronger than its own control?
2
+
3
+ The model work is Ink Lens's (engine.py and gate.py, copied unchanged from ChaseHendrick/Scrolls spaces/ink-lens, see
4
+ VENDORED.md): an openly licensed 9 um ink model reads the stack with the layers as stored and reversed, and the
5
+ reversed map sets the threshold. This module turns an upload into a stack the engine takes, says why when it cannot,
6
+ and turns the gate's status into a plain answer with an overlay.
7
+
8
+ One layer is not enough. On public PHerc0841 crops d9v2 tells ink tiles from plain papyrus with a tile AUC of 0.96
9
+ from a layer stack, and 0.40 to 0.50 (chance) from one layer copied into a stack, raw or as a contrast-stretched
10
+ 8-bit screenshot (Model output, development measurement, 21 Oct 2026, not a held-out claim). So a single image is
11
+ refused with that reason instead of being guessed at.
12
+ """
13
+
14
+ from __future__ import annotations
15
+
16
+ import numpy as np
17
+
18
+ import engine
19
+ import gate
20
+
21
+ VERSION = "0.1.0"
22
+ MODEL_UM = 9.4 # the 9 um readers' native grid (PHerc0841's 9.366 um scans, villa's 9.6 um pooling)
23
+ RESCALE_TOLERANCE = 0.1 # pixel sizes within 10 % of MODEL_UM are used as they are
24
+
25
+ ANSWERS = {
26
+ "ink": ("Yes", "The model finds an ink signal clearly stronger than its control: at least 3 times as many "
27
+ "pixels pass the control's threshold as the control itself."),
28
+ "weak": ("Maybe", "There is some signal above the control (1.5 to 3 times the control's share). Look at the "
29
+ "overlay before trusting it."),
30
+ "none": ("No signal found", "No ink signal above the control in this region. That does not prove there is no "
31
+ "ink: many scrolls show little or no crackle, and the model can miss faint ink."),
32
+ "too_small": ("Too small", "Too little papyrus to judge. Send a region of at least about 320 x 320 px of "
33
+ "papyrus (about 3 mm square at 9 um)."),
34
+ "unreadable": ("Could not judge", "The maps could not be judged (a broken or flat run)."),
35
+ }
36
+
37
+ SINGLE_IMAGE = (
38
+ "**One image is not enough to tell.** Crackle is a 3D texture: the ink models read a stack of layers through the "
39
+ "papyrus surface. Given a single layer, like a screenshot, they score at chance (measured on public labelled "
40
+ "crops), so this tool will not guess. Send the same region as a stack of layers instead: a ZIP of the layer "
41
+ "images (for example 00.tif to 64.tif from a segment's `layers` folder, or from VC3D or `vc_render_tifxyz`), or "
42
+ "one multi-page TIFF, at least 17 layers centred on the surface.")
43
+
44
+
45
+ class CheckError(ValueError):
46
+ """Shown to the person as it is."""
47
+
48
+
49
+ def rescale(stack, um_per_px):
50
+ """The stack resampled so a pixel is about MODEL_UM (bilinear, per layer). Returns (stack, factor)."""
51
+ if not um_per_px or um_per_px <= 0:
52
+ return stack, 1.0
53
+ factor = float(um_per_px) / MODEL_UM
54
+ if abs(factor - 1.0) <= RESCALE_TOLERANCE:
55
+ return stack, 1.0
56
+ from PIL import Image
57
+ L, H, W = stack.shape
58
+ h, w = max(1, round(H * factor)), max(1, round(W * factor))
59
+ out = np.stack([np.asarray(Image.fromarray(l).resize((w, h), Image.BILINEAR)) for l in stack])
60
+ return out.astype(np.uint8), round(factor, 4)
61
+
62
+
63
+ def prepare(array, kind, um_per_px=None, flip=False):
64
+ """(stack, windows, notes) ready for engine.run, or CheckError with the reason."""
65
+ if kind == "image":
66
+ raise CheckError(SINGLE_IMAGE)
67
+ a = np.asarray(array, dtype=np.uint8)
68
+ if a.shape[0] < engine.DEPTH:
69
+ raise CheckError(f"{a.shape[0]} layers: the model reads {engine.DEPTH} layers at once, so send at least "
70
+ f"{engine.DEPTH} (21 or more lets it average three depth windows)")
71
+ if a.shape[0] > engine.MAX_LAYERS:
72
+ raise CheckError(f"{a.shape[0]} layers: send at most {engine.MAX_LAYERS}, centred on the papyrus surface")
73
+ notes = []
74
+ if flip:
75
+ a = a[::-1]
76
+ notes.append("layer order flipped as asked")
77
+ a, factor = rescale(a, um_per_px)
78
+ if factor != 1.0:
79
+ notes.append(f"resampled by {factor} to about {MODEL_UM} um per pixel")
80
+ engine.check_side(a.shape[1], a.shape[2])
81
+ windows = 3 if a.shape[0] >= engine.STACK else 1
82
+ engine.check_volume(a, windows)
83
+ return np.ascontiguousarray(a), windows, notes
84
+
85
+
86
+ def judge(F, R, volume):
87
+ """The gate on the maps, its kept map, and a hint when the reversed order reads better than the stored one."""
88
+ ct = volume[volume.shape[0] // 2]
89
+ d = gate.decide(F, R, ct)
90
+ kept = gate.kept(F, d, ct)
91
+ hint = None
92
+ if d["status"] == "none":
93
+ swapped = gate.decide(R, F, ct)
94
+ if swapped["status"] == "ink":
95
+ hint = ("The reversed layer order shows a strong signal where the stored order shows none. Your layers "
96
+ "may run from the back of the sheet to the front; tick 'Flip layer order' and run again. (Only "
97
+ "the run in the order you send counts: the other order is its control.)")
98
+ return d, kept, hint
99
+
100
+
101
+ def preview(volume, kept):
102
+ """The middle layer in grey with kept ink in amber, RGB uint8."""
103
+ ct = volume[volume.shape[0] // 2].astype(np.float32)
104
+ lo, hi = np.percentile(ct, [1, 99])
105
+ g = np.clip((ct - lo) / max(hi - lo, 1e-6), 0, 1)
106
+ rgb = np.stack([g, g, g], -1)
107
+ a = (gate.to_u8(kept).astype(np.float32) / 255.0)[..., None] * 0.85
108
+ rgb = rgb * (1 - a) + np.array([1.0, 0.62, 0.1]) * a
109
+ return (rgb * 255).astype(np.uint8)
110
+
111
+
112
+ def answer(decision, hint=None, notes=()):
113
+ """The plain answer as Markdown."""
114
+ word, text = ANSWERS[decision["status"]]
115
+ lines = [f"## {word}", "", text]
116
+ if decision.get("excess") is not None:
117
+ lines += ["", f"Signal strength: {decision['excess']:.1f} times the control (3 or more is yes, 1.5 to 3 "
118
+ f"maybe), on {decision['papyrus_px']:,} papyrus pixels."]
119
+ if hint:
120
+ lines += ["", f"> {hint}"]
121
+ if notes:
122
+ lines += ["", "Input: " + "; ".join(notes) + "."]
123
+ lines += ["", "*Model output, not a reading.*"]
124
+ return "\n".join(lines)
125
+
126
+
127
+ def check(array, kind, run, um_per_px=None, flip=False, notes=()):
128
+ """(markdown, result dict, preview RGB). run(stack, windows) returns engine.Engine.run's dict."""
129
+ stack, windows, more = prepare(array, kind, um_per_px, flip)
130
+ out = run(stack, windows)
131
+ d, kept, hint = judge(out["forward"], out["reverse"], stack)
132
+ notes = list(notes) + more
133
+ result = {"tool": f"Crackle Check {VERSION}", "answer": ANSWERS[d["status"]][0], "gate": d,
134
+ "readers": out.get("readers"), "windows": windows, "shape": [int(v) for v in stack.shape],
135
+ "seconds": out.get("seconds"), "device": out.get("device_name"), "notes": notes,
136
+ "note": "Model output, not a reading."}
137
+ if hint:
138
+ result["hint"] = hint
139
+ return answer(d, hint, notes), result, preview(stack, kept)
engine.py ADDED
@@ -0,0 +1,541 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Ink Lens engine: an openly licensed 9 um ink model on one region, three depth windows, both depth orders.
2
+
3
+ The model is built by villa's own flat ink inference (vesuvius.ink_detection.inference.infer.configure_model, at the
4
+ commit pinned in VILLA_COMMIT) from a checkpoint downloaded by URL and refused unless its SHA-256 matches. The patch
5
+ schedule (128 px patches, stride 64, the last flush with the end), the layer choice (center crop, reversed for the
6
+ control), villa's per-patch robust normalisation and the Hann blend are villa's; two things differ from villa's
7
+ command line, both to fit one ZeroGPU call:
8
+
9
+ 1. the robust normalisation runs batched on the device (robust_torch), checked against villa's CPU function
10
+ (robust_np is a line-for-line copy of villa's normalize_robust for a patch with no mask; tests/test_inklens.py
11
+ compares them, and compares robust_np with villa's own function when villa is importable);
12
+ 2. the maps stay float (0 to 1) instead of villa's uint8 encoding, so the control-set threshold is not quantised.
13
+
14
+ run_reference() is the same algorithm in numpy with any predict function: the tests run it with a stand-in model,
15
+ and the torch path is checked against it.
16
+
17
+ python3 engine.py estimate 1024 1024 --windows 3 patches and the GPU seconds one call asks for
18
+ python3 engine.py fetch d9v2 download and check a checkpoint into the cache
19
+ """
20
+
21
+ from __future__ import annotations
22
+
23
+ import hashlib
24
+ import math
25
+ import os
26
+ import platform
27
+ import shutil
28
+ import subprocess
29
+ import sys
30
+ import tarfile
31
+ import tempfile
32
+ import time
33
+ import urllib.request
34
+ from pathlib import Path
35
+
36
+ VERSION = "0.1.0"
37
+ VILLA_REPO = "https://github.com/ScrollPrize/villa.git"
38
+ VILLA_TARBALL = "https://codeload.github.com/ScrollPrize/villa/tar.gz/{commit}"
39
+ VILLA_COMMIT = "8aa507c1215f1fa8758f13d0538f7af9710e02d7" # ScrollPrize/villa main, 18 Oct 2026
40
+ CACHE = Path(os.environ.get("INKLENS_CACHE") or Path(tempfile.gettempdir()) / "ink-lens")
41
+ USER_AGENT = f"ink-lens/{VERSION}"
42
+
43
+ # Openly licensed 9 um readers that drop into villa's flat inference. Public held-out scores (PHerc0841, crops with a
44
+ # 64 px inner edge, raw / reversed AUC) are in docs/results.json; d9v2 is the default.
45
+ READERS = {
46
+ "d9v2": {
47
+ "title": "d9v2 (TAUIL-Abd-Elilah, MIT)",
48
+ "url": "https://github.com/TAUIL-Abd-Elilah/pherc0826-first-letters-search/releases/download/v1.0/"
49
+ "d9v2_ft-012000.pth",
50
+ "file": "d9v2_ft-012000.pth",
51
+ "sha256": "50d2ad0ef7690a6a422640804d38f01409da8bc96cfc1ab72f45324a4a18f966",
52
+ "license": "MIT",
53
+ "source": "https://github.com/TAUIL-Abd-Elilah/pherc0826-first-letters-search",
54
+ "held_out": "PHerc0841 w00, ag896, ag405 (its reader/test.txt)",
55
+ },
56
+ "ink_9um": {
57
+ "title": "ink_9um seed 42 step 75,000 (Vesuvius Challenge, MIT)",
58
+ "url": "https://huggingface.co/scrollprize/ink_9um/resolve/main/hybrid_3d2d-seed42/step-075000.pth",
59
+ "file": "ink_9um-seed42-step-075000.pth",
60
+ "sha256": "e635558ae6a1a807a7e5ec1e83adfd45bc3c0ac53883ea43f1d4e085d62a9cab",
61
+ "license": "MIT",
62
+ "source": "https://huggingface.co/scrollprize/ink_9um",
63
+ "held_out": "PHerc0841 (its label dataset has no PHerc0841 segment)",
64
+ },
65
+ }
66
+ READER_CHOICES = ("d9v2", "ink_9um", "both")
67
+
68
+ PATCH, STRIDE, DEPTH = 128, 64, 17
69
+ STACK = 21 # layers kept around the middle: the three windows below fit in it
70
+ WINDOW_OFFSETS = {1: (2,), 3: (0, 2, 4)} # window starts inside the 21-layer stack (2 is villa's own center)
71
+ BATCH = 64
72
+ MIN_SIDE, MAX_SIDE = PATCH, 2048
73
+ MIN_LAYERS, MAX_LAYERS = DEPTH, 65
74
+ # One call's duration: a fixed start plus a per-patch allowance, capped. The per-patch figure is an estimate (3 to
75
+ # 6 ms per patch with batched fp16 and normalisation on the GPU) with a margin; every receipt records the measured
76
+ # seconds so the first calls calibrate it.
77
+ GPU_START_S, GPU_PER_PATCH_S, GPU_MAX_S = 4.0, 0.012, 40
78
+
79
+
80
+ class EngineError(ValueError):
81
+ """A request the engine refuses; the message says why."""
82
+
83
+
84
+ # ---------------------------------------------------------------- sizing
85
+
86
+ def positions(length, patch=PATCH, stride=STRIDE):
87
+ """Patch starts along one axis, the last flush with the end (villa's _sliding_positions_1d)."""
88
+ if length <= patch:
89
+ return [0]
90
+ out = list(range(0, length - patch + 1, max(1, stride)))
91
+ if out[-1] != length - patch:
92
+ out.append(length - patch)
93
+ return out
94
+
95
+
96
+ def readers_of(choice):
97
+ if choice not in READER_CHOICES:
98
+ raise EngineError(f"reader {choice!r}: choose one of {', '.join(READER_CHOICES)}")
99
+ return ("d9v2", "ink_9um") if choice == "both" else (choice,)
100
+
101
+
102
+ def patches(height, width, windows=3, reader="d9v2"):
103
+ """Patches one call runs: per pass the patch grid, times windows, times two depth orders, times readers."""
104
+ if windows not in WINDOW_OFFSETS:
105
+ raise EngineError(f"windows {windows}: use 1 or 3")
106
+ return len(positions(height)) * len(positions(width)) * windows * 2 * len(readers_of(reader))
107
+
108
+
109
+ def gpu_seconds(n_patches):
110
+ """The duration a call asks ZeroGPU for; refuses a call that would need more than GPU_MAX_S."""
111
+ need = math.ceil(GPU_START_S + n_patches * GPU_PER_PATCH_S)
112
+ if need > GPU_MAX_S:
113
+ raise EngineError(f"{n_patches} patches need about {need} GPU seconds, more than the {GPU_MAX_S} s one call "
114
+ "may ask for; use a smaller region, one depth window, or local mode")
115
+ return need
116
+
117
+
118
+ def estimate(height, width, windows=3, reader="d9v2"):
119
+ check_side(height, width)
120
+ n = patches(height, width, windows, reader)
121
+ out = {"height": int(height), "width": int(width), "windows": int(windows), "reader": reader, "patches": n}
122
+ try:
123
+ out["gpu_seconds_requested"] = gpu_seconds(n)
124
+ except EngineError as exc:
125
+ out["gpu_seconds_requested"] = None
126
+ out["refused"] = str(exc)
127
+ return out
128
+
129
+
130
+ def check_side(height, width):
131
+ for name, side in (("height", height), ("width", width)):
132
+ if not MIN_SIDE <= int(side) <= MAX_SIDE:
133
+ raise EngineError(f"{name} {side} px: a region is {MIN_SIDE} to {MAX_SIDE} px a side at 9.6 um")
134
+
135
+
136
+ def check_volume(volume, windows=3):
137
+ """Refuse anything but a uint8 (layers, rows, columns) region of the right size and depth."""
138
+ import numpy as np
139
+ a = np.asarray(volume)
140
+ if a.ndim != 3:
141
+ raise EngineError(f"a surface volume has 3 axes (layers, rows, columns), this one has {a.ndim}")
142
+ if a.dtype != np.uint8:
143
+ raise EngineError(f"type {a.dtype}: the engine takes uint8 (volio.to_uint8 converts uint16)")
144
+ L, H, W = a.shape
145
+ need = STACK if windows == 3 else DEPTH
146
+ if not need <= L <= MAX_LAYERS:
147
+ raise EngineError(f"{L} layers: {windows} depth window{'s' if windows > 1 else ''} need{'' if windows > 1 else 's'} "
148
+ f"{need} to {MAX_LAYERS} layers, layers first")
149
+ check_side(H, W)
150
+ if not a.any():
151
+ raise EngineError("the region is all zero (off the segment, or a hole in it)")
152
+ return L, H, W
153
+
154
+
155
+ def stack(volume, windows=3):
156
+ """The middle STACK layers (or all, if fewer), as villa's center crop chooses them (upper center when uneven)."""
157
+ L = volume.shape[0]
158
+ n = min(L, STACK)
159
+ start = L // 2 - n // 2
160
+ return volume[start:start + n]
161
+
162
+
163
+ def window_indices(n_layers, offset, direction):
164
+ """Layer indices of one 17-layer window inside the stack; reversed for the control."""
165
+ if n_layers < STACK: # one window, centered (villa's own choice)
166
+ start = n_layers // 2 - DEPTH // 2
167
+ else:
168
+ start = offset
169
+ idx = list(range(start, start + DEPTH))
170
+ return idx[::-1] if direction == "reverse" else idx
171
+
172
+
173
+ def passes(windows):
174
+ """(offset, direction) for every pass of one reader: each window in both orders."""
175
+ return [(o, d) for o in WINDOW_OFFSETS[windows] for d in ("forward", "reverse")]
176
+
177
+
178
+ # ---------------------------------------------------------------- villa's normalisation, numpy and torch
179
+
180
+ def robust_np(patch):
181
+ """villa's normalize_robust (vesuvius/image_proc/intensity/normalization.py at VILLA_COMMIT) with no mask:
182
+ clip to the 1st and 99th percentiles, subtract the median, divide by 1.4826 x MAD, with villa's fallbacks."""
183
+ import numpy as np
184
+ arr = np.asarray(patch).astype(np.float32, copy=True)
185
+ valid = arr.reshape(-1)
186
+ if valid.size == 0:
187
+ return arr
188
+ std_preclip = float(np.std(valid))
189
+ lower = float(np.percentile(valid, 1.0))
190
+ upper = float(np.percentile(valid, 99.0))
191
+ np.clip(arr, lower, upper, out=arr)
192
+ valid = arr.reshape(-1)
193
+ median = float(np.median(valid))
194
+ mad = float(np.median(np.abs(valid - median)))
195
+ scaled = 1.4826 * mad
196
+ if not np.isfinite(scaled) or scaled < 1e-6:
197
+ span = upper - lower
198
+ cands = []
199
+ if np.isfinite(std_preclip):
200
+ cands.append(abs(std_preclip))
201
+ if np.isfinite(span):
202
+ cands.append(abs(span) / 2.0)
203
+ scaled = next((c for c in cands if c >= 1e-6), None)
204
+ if scaled is None or not np.isfinite(scaled):
205
+ scaled = 1.0
206
+ arr -= median
207
+ arr /= scaled
208
+ np.nan_to_num(arr, copy=False, nan=0.0, posinf=0.0, neginf=0.0)
209
+ return arr
210
+
211
+
212
+ def _lerp_sorted(torch, s, q):
213
+ """numpy's linear percentile of each row of an already sorted (B, N) tensor."""
214
+ n = s.shape[1]
215
+ pos = q * (n - 1)
216
+ lo = int(math.floor(pos))
217
+ hi = min(lo + 1, n - 1)
218
+ frac = pos - lo
219
+ return s[:, lo] + (s[:, hi] - s[:, lo]) * frac
220
+
221
+
222
+ def robust_torch(x):
223
+ """robust_np on every row of a (B, N) float32 tensor at once, on its device."""
224
+ import torch
225
+ x = x.float()
226
+ s, _ = torch.sort(x, dim=1)
227
+ std_pre = x.std(dim=1, unbiased=False)
228
+ lower, upper = _lerp_sorted(torch, s, 0.01), _lerp_sorted(torch, s, 0.99)
229
+ x = torch.maximum(torch.minimum(x, upper[:, None]), lower[:, None])
230
+ s = torch.maximum(torch.minimum(s, upper[:, None]), lower[:, None])
231
+ median = _lerp_sorted(torch, s, 0.5)
232
+ dev, _ = torch.sort((x - median[:, None]).abs(), dim=1)
233
+ scaled = 1.4826 * _lerp_sorted(torch, dev, 0.5)
234
+ half_span = (upper - lower).abs() / 2.0
235
+ ok = torch.isfinite(scaled) & (scaled >= 1e-6)
236
+ fb = torch.where(torch.isfinite(std_pre) & (std_pre.abs() >= 1e-6), std_pre.abs(),
237
+ torch.where(torch.isfinite(half_span) & (half_span >= 1e-6), half_span,
238
+ torch.ones_like(half_span)))
239
+ scaled = torch.where(ok, scaled, fb)
240
+ out = (x - median[:, None]) / scaled[:, None]
241
+ return torch.nan_to_num(out, nan=0.0, posinf=0.0, neginf=0.0)
242
+
243
+
244
+ def hann2d():
245
+ """villa's floored Hann blend map (compute_importance_map_2d, mode hann), in numpy."""
246
+ import numpy as np
247
+ w1 = np.hanning(PATCH).astype(np.float32) # numpy's hanning is the symmetric (periodic=False) window
248
+ w = np.outer(w1, w1)
249
+ w /= max(float(w.max()), float(np.finfo(np.float32).eps))
250
+ return np.maximum(w, 0.001).astype(np.float32)
251
+
252
+
253
+ # ---------------------------------------------------------------- the numpy reference
254
+
255
+ def run_reference(volume, predict, windows=3):
256
+ """The whole algorithm in numpy: predict(batch (B, 1, 17, 128, 128) float32) -> (B, 128, 128) probabilities.
257
+ Returns {"forward": F, "reverse": R} (float32, 0 to 1), each the mean over the windows."""
258
+ import numpy as np
259
+ vol = stack(np.asarray(volume), windows)
260
+ L, H, W = vol.shape
261
+ ys, xs = positions(H), positions(W)
262
+ wmap = hann2d()
263
+ maps = {"forward": [], "reverse": []}
264
+ for off, direction in passes(windows):
265
+ idx = window_indices(L, off, direction)
266
+ psum = np.zeros((H, W), np.float32)
267
+ wsum = np.zeros((H, W), np.float32)
268
+ for y0 in ys:
269
+ for x0 in xs:
270
+ raw = vol[idx, y0:y0 + PATCH, x0:x0 + PATCH]
271
+ if not raw.any():
272
+ continue
273
+ p = np.asarray(predict(robust_np(raw)[None, None]), np.float32)[0]
274
+ psum[y0:y0 + PATCH, x0:x0 + PATCH] += p * wmap
275
+ wsum[y0:y0 + PATCH, x0:x0 + PATCH] += wmap
276
+ prob = np.divide(psum, wsum, out=np.zeros_like(psum), where=wsum > 1e-6)
277
+ maps[direction].append(np.clip(prob, 0, 1))
278
+ return {d: np.mean(m, axis=0).astype(np.float32) for d, m in maps.items()}
279
+
280
+
281
+ # ---------------------------------------------------------------- downloads
282
+
283
+ def sha256_file(path, block=1 << 22):
284
+ h = hashlib.sha256()
285
+ with open(path, "rb") as fh:
286
+ for chunk in iter(lambda: fh.read(block), b""):
287
+ h.update(chunk)
288
+ return h.hexdigest()
289
+
290
+
291
+ def _download(url, dest):
292
+ dest.parent.mkdir(parents=True, exist_ok=True)
293
+ part = dest.with_name(dest.name + ".part")
294
+ req = urllib.request.Request(url, headers={"User-Agent": USER_AGENT})
295
+ with urllib.request.urlopen(req, timeout=600) as resp, open(part, "wb") as out:
296
+ shutil.copyfileobj(resp, out, 1 << 20)
297
+ os.replace(part, dest)
298
+
299
+
300
+ def ensure_checkpoint(name, cache=CACHE):
301
+ """A reader's checkpoint, downloaded once and refused unless its SHA-256 matches READERS. INKLENS_<NAME> (for
302
+ example INKLENS_D9V2) may name a local copy; its SHA-256 is still checked."""
303
+ spec = READERS[name]
304
+ env = os.environ.get("INKLENS_" + name.upper())
305
+ path = Path(env).expanduser() if env else Path(cache) / "checkpoints" / spec["file"]
306
+ if not path.is_file():
307
+ if env:
308
+ raise EngineError(f"INKLENS_{name.upper()}={env} does not exist")
309
+ _download(spec["url"], path)
310
+ digest = sha256_file(path)
311
+ if digest != spec["sha256"]:
312
+ raise EngineError(f"{path} has SHA-256 {digest}, expected {spec['sha256']} for {name}; refusing it")
313
+ return path
314
+
315
+
316
+ def _git_commit(path):
317
+ try:
318
+ return subprocess.run(["git", "-C", str(path), "rev-parse", "HEAD"], capture_output=True, text=True,
319
+ check=True).stdout.strip()
320
+ except (OSError, subprocess.CalledProcessError):
321
+ return "unknown"
322
+
323
+
324
+ def _villa_tarball(tmp, commit):
325
+ root = tmp.resolve()
326
+ kept = 0
327
+ req = urllib.request.Request(VILLA_TARBALL.format(commit=commit), headers={"User-Agent": USER_AGENT})
328
+ with urllib.request.urlopen(req, timeout=600) as resp, tarfile.open(fileobj=resp, mode="r|gz") as tf:
329
+ for m in tf:
330
+ _, _, rel = m.name.partition("/")
331
+ if not rel.startswith("vesuvius/src/"):
332
+ continue
333
+ target = (root / rel).resolve()
334
+ if root not in target.parents or m.issym() or m.islnk():
335
+ continue
336
+ if m.isdir():
337
+ target.mkdir(parents=True, exist_ok=True)
338
+ elif m.isfile():
339
+ target.parent.mkdir(parents=True, exist_ok=True)
340
+ with tf.extractfile(m) as src, open(target, "wb") as out:
341
+ shutil.copyfileobj(src, out)
342
+ kept += 1
343
+ if not kept:
344
+ raise EngineError(f"the villa tarball for {commit} held no vesuvius/src files")
345
+
346
+
347
+ def ensure_villa(cache=CACHE):
348
+ """villa's vesuvius package on sys.path: VILLA_SRC (a local vesuvius/src) when set, else the pinned commit,
349
+ downloaded once (sparse git fetch, else GitHub's tarball)."""
350
+ if os.environ.get("VILLA_SRC"):
351
+ src = Path(os.environ["VILLA_SRC"]).expanduser().resolve()
352
+ info = {"source": "VILLA_SRC", "commit": _git_commit(src)}
353
+ else:
354
+ dest = Path(cache) / f"villa-{VILLA_COMMIT[:12]}"
355
+ if not (dest / "COMMIT").is_file():
356
+ tmp = dest.with_name(dest.name + ".part")
357
+ shutil.rmtree(tmp, ignore_errors=True)
358
+ tmp.mkdir(parents=True)
359
+ try:
360
+ git = ["git", "-C", str(tmp)]
361
+ for cmd in (["init", "-q"], ["remote", "add", "origin", VILLA_REPO],
362
+ ["sparse-checkout", "set", "vesuvius/src"],
363
+ ["fetch", "-q", "--depth", "1", "--filter=blob:none", "origin", VILLA_COMMIT],
364
+ ["checkout", "-q", "FETCH_HEAD"]):
365
+ subprocess.run(git + cmd, capture_output=True, text=True, check=True, timeout=600)
366
+ except (OSError, subprocess.SubprocessError):
367
+ shutil.rmtree(tmp, ignore_errors=True)
368
+ tmp.mkdir(parents=True)
369
+ _villa_tarball(tmp, VILLA_COMMIT)
370
+ (tmp / "COMMIT").write_text(VILLA_COMMIT + "\n")
371
+ shutil.rmtree(dest, ignore_errors=True)
372
+ os.replace(tmp, dest)
373
+ src = dest / "vesuvius" / "src"
374
+ info = {"source": "pinned download", "commit": (dest / "COMMIT").read_text().strip()}
375
+ if not (src / "vesuvius" / "ink_detection" / "inference" / "infer.py").is_file():
376
+ raise EngineError(f"{src} holds no villa vesuvius package")
377
+ if str(src) not in sys.path:
378
+ sys.path.insert(0, str(src))
379
+ info["pinned"] = VILLA_COMMIT
380
+ return info
381
+
382
+
383
+ def load_villa_model(name, cache=CACHE):
384
+ """(model, preprocessing, amp dtype, info) for one reader, built by villa's configure_model. The checkpoint is
385
+ opened with torch.load(weights_only=True): villa's loader allows any pickle, so it is replaced for this call."""
386
+ import argparse
387
+ import torch
388
+ villa = ensure_villa(cache)
389
+ path = ensure_checkpoint(name, cache)
390
+ from vesuvius.ink_detection.inference import infer
391
+ original = infer.load_checkpoint
392
+ infer.load_checkpoint = lambda p: torch.load(str(p), map_location="cpu", weights_only=True)
393
+ try:
394
+ cm = infer.configure_model(argparse.Namespace(checkpoint=path, amp_dtype="auto"))
395
+ finally:
396
+ infer.load_checkpoint = original
397
+ got = (int(cm.patch_size), int(cm.input_depth))
398
+ if got != (PATCH, DEPTH):
399
+ raise EngineError(f"{name} has patch and depth {got}; Ink Lens is built for {(PATCH, DEPTH)}")
400
+ if cm.preprocessing not in ("tifxyz_robust", "divide_255"):
401
+ raise EngineError(f"{name} asks for preprocessing {cm.preprocessing!r}, which Ink Lens does not run")
402
+ info = {"name": name, "sha256": READERS[name]["sha256"], "url": READERS[name]["url"],
403
+ "license": READERS[name]["license"], "preprocessing": cm.preprocessing,
404
+ "amp_on_cuda": str(cm.amp_dtype).replace("torch.", ""), "villa": villa, "loader": "weights_only=True"}
405
+ return cm.model.eval(), cm.preprocessing, cm.amp_dtype, info
406
+
407
+
408
+ # ---------------------------------------------------------------- the torch engine
409
+
410
+ def pick_device(prefer=None):
411
+ import torch
412
+ if prefer:
413
+ return torch.device(prefer)
414
+ if torch.cuda.is_available():
415
+ return torch.device("cuda")
416
+ if getattr(torch.backends, "mps", None) is not None and torch.backends.mps.is_available():
417
+ return torch.device("mps")
418
+ return torch.device("cpu")
419
+
420
+
421
+ class Engine:
422
+ """Readers loaded once; run() does one region on one device.
423
+
424
+ models: {name: (module, preprocessing, amp_dtype, info)} to use instead of downloading (the tests pass a
425
+ stand-in); otherwise every name in `readers` is loaded through villa."""
426
+
427
+ def __init__(self, readers=("d9v2",), cache=CACHE, models=None, place_on_cuda=True):
428
+ t0 = time.time()
429
+ import torch
430
+ self.torch = torch
431
+ self.models = dict(models) if models else {n: load_villa_model(n, cache) for n in readers}
432
+ if place_on_cuda and torch.cuda.is_available():
433
+ for m in self.models.values():
434
+ m[0].to("cuda")
435
+ self.startup_s = round(time.time() - t0, 1)
436
+
437
+ def describe(self):
438
+ return {"engine": f"ink-lens {VERSION}", "readers": {n: m[3] for n, m in self.models.items()},
439
+ "patch": PATCH, "stride": STRIDE, "depth": DEPTH, "stack": STACK, "blend": "hann (villa, floored)",
440
+ "batch": BATCH, "torch": self.torch.__version__}
441
+
442
+ def run(self, volume, reader="d9v2", windows=3, device=None, batch=BATCH):
443
+ """{"forward": F, "reverse": R, "per_reader": {...}, ...} for one uint8 region, float32 maps in 0 to 1."""
444
+ import numpy as np
445
+ torch = self.torch
446
+ names = readers_of(reader)
447
+ missing = [n for n in names if n not in self.models]
448
+ if missing:
449
+ raise EngineError(f"reader {', '.join(missing)} is not loaded here")
450
+ check_volume(volume, windows)
451
+ dev = pick_device(device)
452
+ t0 = time.perf_counter()
453
+ vol = stack(np.asarray(volume), windows)
454
+ L, H, W = vol.shape
455
+ stack_start = int(np.asarray(volume).shape[0] // 2 - L // 2)
456
+ vt = torch.from_numpy(np.ascontiguousarray(vol)).to(dev)
457
+ coords = [(y, x) for y in positions(H) for x in positions(W)]
458
+ wmap = torch.from_numpy(hann2d()).to(dev)
459
+ per_reader, n_patches = {}, 0
460
+ for name in names:
461
+ model, prep, amp, _ = self.models[name]
462
+ model = model.to(dev)
463
+ maps = {"forward": [], "reverse": []}
464
+ for off, direction in passes(windows):
465
+ idx = torch.tensor(window_indices(L, off, direction), device=dev)
466
+ sub = vt.index_select(0, idx)
467
+ psum = torch.zeros((H, W), dtype=torch.float32, device=dev)
468
+ wsum = torch.zeros((H, W), dtype=torch.float32, device=dev)
469
+ for b0 in range(0, len(coords), batch):
470
+ chunk = coords[b0:b0 + batch]
471
+ raw = torch.stack([sub[:, y:y + PATCH, x:x + PATCH] for y, x in chunk]) # (B, 17, 128, 128)
472
+ keep = raw.reshape(len(chunk), -1).any(dim=1) # villa skips all-zero patches, before normalising
473
+ if not bool(keep.any()):
474
+ continue
475
+ if not bool(keep.all()):
476
+ chunk = [c for c, k in zip(chunk, keep.tolist()) if k]
477
+ raw = raw[keep]
478
+ flat = raw.reshape(len(chunk), -1).float()
479
+ norm = robust_torch(flat) if prep == "tifxyz_robust" else flat / 255.0
480
+ images = norm.reshape(len(chunk), 1, DEPTH, PATCH, PATCH)
481
+ ctx = (torch.autocast("cuda", dtype=amp) if dev.type == "cuda" and amp is not None
482
+ else _null())
483
+ with torch.inference_mode(), ctx:
484
+ logits = model(images)
485
+ prob = logits.float().sigmoid()
486
+ if prob.shape[-2:] != (PATCH, PATCH):
487
+ prob = torch.nn.functional.interpolate(prob, size=(PATCH, PATCH), mode="bilinear",
488
+ align_corners=False)
489
+ prob = prob[:, 0]
490
+ for (y, x), p in zip(chunk, prob):
491
+ psum[y:y + PATCH, x:x + PATCH] += p * wmap
492
+ wsum[y:y + PATCH, x:x + PATCH] += wmap
493
+ n_patches += len(chunk)
494
+ out = torch.where(wsum > 1e-6, psum / wsum.clamp_min(1e-6), torch.zeros_like(psum)).clamp(0, 1)
495
+ maps[direction].append(out)
496
+ per_reader[name] = {d: torch.stack(m).mean(0).cpu().numpy().astype(np.float32) for d, m in maps.items()}
497
+ if dev.type == "cuda":
498
+ torch.cuda.synchronize(dev)
499
+ F = np.mean([per_reader[n]["forward"] for n in names], axis=0).astype(np.float32)
500
+ R = np.mean([per_reader[n]["reverse"] for n in names], axis=0).astype(np.float32)
501
+ name = torch.cuda.get_device_name(dev) if dev.type == "cuda" else (platform.processor() or platform.machine())
502
+ return {"forward": F, "reverse": R, "per_reader": per_reader, "readers": list(names), "windows": windows,
503
+ "window_layers": {f"{o}:{d}": window_indices(L, o, d) for o, d in passes(windows)},
504
+ "stack_layers": int(L), "stack_start": stack_start, "patches": n_patches, "device": str(dev), "device_name": name,
505
+ "seconds": round(time.perf_counter() - t0, 3)}
506
+
507
+
508
+ class _null:
509
+ def __enter__(self):
510
+ return None
511
+
512
+ def __exit__(self, *exc):
513
+ return False
514
+
515
+
516
+ def main(argv=None):
517
+ import argparse
518
+ import json
519
+ ap = argparse.ArgumentParser(description="Ink Lens engine helpers")
520
+ sub = ap.add_subparsers(dest="cmd", required=True)
521
+ e = sub.add_parser("estimate")
522
+ e.add_argument("height", type=int)
523
+ e.add_argument("width", type=int)
524
+ e.add_argument("--windows", type=int, default=3, choices=(1, 3))
525
+ e.add_argument("--reader", default="d9v2", choices=READER_CHOICES)
526
+ f = sub.add_parser("fetch")
527
+ f.add_argument("reader", choices=sorted(READERS))
528
+ a = ap.parse_args(argv)
529
+ try:
530
+ if a.cmd == "estimate":
531
+ print(json.dumps(estimate(a.height, a.width, a.windows, a.reader), indent=1))
532
+ else:
533
+ print(json.dumps({"reader": a.reader, "path": str(ensure_checkpoint(a.reader))}))
534
+ except EngineError as exc:
535
+ print(f"refused: {exc}", file=sys.stderr)
536
+ return 2
537
+ return 0
538
+
539
+
540
+ if __name__ == "__main__":
541
+ sys.exit(main())
gate.py ADDED
@@ -0,0 +1,138 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Ink Lens: the control-set threshold and the region status (numpy only).
2
+
3
+ The rule, frozen in docs/prereg/2026-10-20-ink-lens-gate.md before any map of it existed:
4
+
5
+ - F and R are the mean ink probabilities (0 to 1) of the region with its layers as stored and reversed, each averaged
6
+ over the same depth windows and readers.
7
+ - Papyrus: pixels whose middle CT layer is above the region's 5th percentile of CT and above 0 (so air and the
8
+ masked outside of a segment do not lower the threshold).
9
+ - T is the 99th percentile of R over the papyrus pixels: the level that only 1 % of the control's own pixels pass.
10
+ - Kept ink: papyrus pixels with F > T. Excess: the fraction of papyrus pixels with F > T, divided by 0.01.
11
+ - Status: ink when excess >= 3, weak when 1.5 <= excess < 3, none below 1.5, too_small under 100,000 papyrus pixels
12
+ (about 3 mm square at 9.6 um), unreadable when a map is not finite, constant, or identical to its control.
13
+
14
+ A region with no text, or where the model reads the reversed volume as well as the stored one, has an excess near 1
15
+ and the status none: the check can fail. Everything it keeps is model output, not a reading.
16
+ """
17
+
18
+ from __future__ import annotations
19
+
20
+ RULE_ID = "ink-lens-gate-v1"
21
+ CT_PERCENTILE = 5.0
22
+ CONTROL_PERCENTILE = 99.0
23
+ BASE_RATE = 0.01
24
+ INK_EXCESS = 3.0
25
+ WEAK_EXCESS = 1.5
26
+ MIN_PAPYRUS = 100_000
27
+ STATUSES = ("ink", "weak", "none", "too_small", "unreadable")
28
+ STATUS_TEXT = {
29
+ "ink": "ink above the control: at least 3 times as many pixels pass the control's 99th percentile as the control",
30
+ "weak": "some ink above the control: 1.5 to 3 times the control's share; look before trusting it",
31
+ "none": "no ink above the control: the stored order passes the control's 99th percentile no more than 1.5 times "
32
+ "as often as the control itself",
33
+ "too_small": "too little papyrus in the region to judge (under 100,000 pixels, about 3 mm square)",
34
+ "unreadable": "the maps cannot be judged (not finite, constant, or the same as the control)",
35
+ }
36
+
37
+
38
+ class GateError(ValueError):
39
+ """Maps the gate cannot take at all (wrong shapes); the message says why."""
40
+
41
+
42
+ def papyrus(ct_middle, percentile=CT_PERCENTILE):
43
+ """Boolean mask of papyrus pixels in a region: middle CT layer above its own percentile and above 0."""
44
+ import numpy as np
45
+ ct = np.asarray(ct_middle)
46
+ if ct.ndim != 2:
47
+ raise GateError(f"the CT layer is 2-D, not {ct.shape}")
48
+ nz = ct[ct > 0]
49
+ if nz.size == 0:
50
+ return np.zeros(ct.shape, bool)
51
+ floor = max(float(np.percentile(ct, percentile)), 0.0)
52
+ return ct > floor
53
+
54
+
55
+ def _unreadable(F, R, mask):
56
+ import numpy as np
57
+ reasons = []
58
+ for name, m in (("stored-order map", F), ("control map", R)):
59
+ if not np.all(np.isfinite(m)):
60
+ reasons.append(f"the {name} has values that are not finite")
61
+ elif mask.any() and float(m[mask].max()) - float(m[mask].min()) < 1e-6:
62
+ reasons.append(f"the {name} is constant on the papyrus")
63
+ if not reasons and mask.any() and np.array_equal(F[mask], R[mask]):
64
+ reasons.append("the stored-order map is identical to its control (a broken run, or the control was not run)")
65
+ return reasons
66
+
67
+
68
+ def decide(F, R, ct_middle, min_papyrus=MIN_PAPYRUS):
69
+ """The gate on one region: threshold, excess, status and counts, as a JSON-able dict."""
70
+ import numpy as np
71
+ F = np.asarray(F, dtype=np.float32)
72
+ R = np.asarray(R, dtype=np.float32)
73
+ if F.shape != R.shape or F.ndim != 2:
74
+ raise GateError(f"the map {F.shape} and its control {R.shape} must be 2-D and the same shape")
75
+ mask = papyrus(ct_middle)
76
+ if mask.shape != F.shape:
77
+ raise GateError(f"the CT layer {mask.shape} and the maps {F.shape} differ in shape")
78
+ n = int(mask.sum())
79
+ out = {"rule": RULE_ID, "papyrus_px": n, "region_px": int(mask.size), "threshold": None, "excess": None,
80
+ "forward_above": None, "control_above": None, "status": None}
81
+ reasons = _unreadable(F, R, mask)
82
+ if reasons:
83
+ out.update(status="unreadable", reasons=reasons)
84
+ elif n < min_papyrus:
85
+ out.update(status="too_small")
86
+ else:
87
+ T = float(np.percentile(R[mask], CONTROL_PERCENTILE))
88
+ f_above = float((F[mask] > T).mean())
89
+ r_above = float((R[mask] > T).mean())
90
+ excess = f_above / BASE_RATE
91
+ status = "ink" if excess >= INK_EXCESS else "weak" if excess >= WEAK_EXCESS else "none"
92
+ out.update(threshold=round(T, 6), excess=round(excess, 4), forward_above=round(f_above, 6),
93
+ control_above=round(r_above, 6), status=status)
94
+ out["status_text"] = STATUS_TEXT[out["status"]]
95
+ out["note"] = "Model output, not a reading."
96
+ return out
97
+
98
+
99
+ def kept(F, decision, ct_middle):
100
+ """F where it passes the control's threshold on papyrus, else 0 (float32). Nothing passes unless the status is
101
+ ink or weak."""
102
+ import numpy as np
103
+ F = np.asarray(F, dtype=np.float32)
104
+ out = np.zeros_like(F)
105
+ if decision.get("status") not in ("ink", "weak"):
106
+ return out
107
+ m = papyrus(ct_middle) & (F > float(decision["threshold"]))
108
+ out[m] = F[m]
109
+ return out
110
+
111
+
112
+ def to_u8(kept_map):
113
+ """The kept map as uint8 for an overlay: 0 where nothing is kept (transparent in VC3D), else 1 to 255."""
114
+ import numpy as np
115
+ k = np.asarray(kept_map, dtype=np.float32)
116
+ out = np.zeros(k.shape, np.uint8)
117
+ on = k > 0
118
+ out[on] = np.clip(np.rint(k[on] * 254) + 1, 1, 255).astype(np.uint8)
119
+ return out
120
+
121
+
122
+ def label_shares(kept_map, ink, supervised, inner=64):
123
+ """The preregistered label check: the share of labeled ink pixels kept and the share of labeled background
124
+ pixels kept, inside the supervision mask with an inner edge left out."""
125
+ import numpy as np
126
+ k = np.asarray(kept_map) > 0
127
+ m = np.asarray(supervised, bool).copy()
128
+ if inner:
129
+ m[:inner] = m[-inner:] = False
130
+ m[:, :inner] = m[:, -inner:] = False
131
+ ink = np.asarray(ink, bool)
132
+ ink_px, bg_px = int((m & ink).sum()), int((m & ~ink).sum())
133
+ ink_share = float((k & m & ink).sum()) / ink_px if ink_px else None
134
+ bg_share = float((k & m & ~ink).sum()) / bg_px if bg_px else None
135
+ passes = None if ink_share is None or bg_share is None else ink_share > bg_share
136
+ return {"ink_px": ink_px, "background_px": bg_px, "ink_kept_share": None if ink_share is None else round(ink_share, 5),
137
+ "background_kept_share": None if bg_share is None else round(bg_share, 5), "ink_share_larger": passes,
138
+ "inner": inner}
requirements.txt ADDED
@@ -0,0 +1,22 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Crackle Check (same pins as Ink Lens, whose engine it runs) on a Hugging Face ZeroGPU Space (Gradio 6.28.0 and Python 3.12 come from the README card; `spaces` is
2
+ # preinstalled there). The torch, torchvision, numpy, zarr and numcodecs pins are the ones the AutoPipeline ink Space
3
+ # and Scroll Lens run on the same hardware; einops, timm, pyyaml, pynrrd, aiohttp and requests are what villa's
4
+ # vesuvius package imports for flat ink inference (it is downloaded at a pinned commit, see engine.VILLA_COMMIT). mcp
5
+ # and pydantic as Scroll Lens pins them for gradio's MCP server.
6
+ torch==2.13.0
7
+ torchvision==0.28.0
8
+ numpy==2.5.2
9
+ zarr==3.4.1
10
+ numcodecs==0.16.5
11
+ tifffile==2026.9.20
12
+ pynrrd==1.1.3
13
+ einops==0.8.2
14
+ timm==1.0.30
15
+ pyyaml==6.0.3
16
+ requests==2.34.2
17
+ aiohttp==3.14.5
18
+ huggingface_hub==1.33.0
19
+ mcp==1.30.0
20
+ pydantic>=2.11.10,<=2.12.5
21
+ imagecodecs==2026.10.10
22
+ pillow
upload.py ADDED
@@ -0,0 +1,113 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Read what someone uploads into a uint8 array: one 2D layer (H, W) or a layer stack (L, H, W).
2
+
3
+ Accepted: a single image (PNG, JPEG, WebP, BMP, one-page TIFF), a multi-page TIFF of surface layers, a ZIP of layer
4
+ images (ordered by the numbers in their names, as villa and VC3D name them: 00.tif, 01.tif, ...), or an .npz with an
5
+ array "volume" (Ink Lens's region format). Colour is turned to grey, 16-bit is scaled to 8-bit. Nothing is kept.
6
+ """
7
+
8
+ from __future__ import annotations
9
+
10
+ import io
11
+ import re
12
+ import zipfile
13
+ from pathlib import Path
14
+
15
+ import numpy as np
16
+
17
+ MIN_SIDE, MAX_SIDE = 128, 2048
18
+ MAX_LAYERS = 65
19
+ MAX_BYTES = 400 * 1024 * 1024
20
+ IMAGE_EXT = (".png", ".jpg", ".jpeg", ".webp", ".bmp", ".tif", ".tiff")
21
+
22
+
23
+ class UploadError(ValueError):
24
+ """Shown to the person as it is."""
25
+
26
+
27
+ def to_u8(a):
28
+ """Grey uint8 from a 2D or (H, W, C) array of any integer or float type."""
29
+ a = np.asarray(a)
30
+ if a.ndim == 3 and a.shape[-1] in (3, 4):
31
+ rgb_u8 = a.dtype == np.uint8
32
+ a = a[..., :3].astype(np.float32) @ np.array([0.299, 0.587, 0.114], np.float32)
33
+ if rgb_u8:
34
+ return np.clip(a + 0.5, 0, 255).astype(np.uint8)
35
+ if a.ndim != 2:
36
+ raise UploadError(f"expected a 2D image, got shape {tuple(a.shape)}")
37
+ if a.dtype == np.uint8:
38
+ return a
39
+ if a.dtype == np.uint16:
40
+ return (a // 257).astype(np.uint8)
41
+ is_float = a.dtype.kind == "f"
42
+ a = np.nan_to_num(a.astype(np.float32))
43
+ lo, hi = float(a.min()), float(a.max())
44
+ if is_float and lo >= 0 and hi <= 1.0:
45
+ return np.clip(a * 255 + 0.5, 0, 255).astype(np.uint8)
46
+ if lo >= 0 and hi <= 255:
47
+ return np.clip(a + 0.5, 0, 255).astype(np.uint8)
48
+ if hi <= lo:
49
+ return np.zeros(a.shape, np.uint8)
50
+ return np.clip((a - lo) / (hi - lo) * 255 + 0.5, 0, 255).astype(np.uint8)
51
+
52
+
53
+ def _image(data, name):
54
+ if name.lower().endswith((".tif", ".tiff")):
55
+ import tifffile
56
+ a = tifffile.imread(io.BytesIO(data))
57
+ if a.ndim == 2 or (a.ndim == 3 and a.shape[-1] in (3, 4)):
58
+ return [to_u8(a)]
59
+ if a.ndim in (3, 4):
60
+ return [to_u8(l) for l in a]
61
+ raise UploadError(f"cannot read a TIFF of shape {tuple(a.shape)} as layers")
62
+ from PIL import Image
63
+ with Image.open(io.BytesIO(data)) as im:
64
+ return [to_u8(np.asarray(im.convert("L") if im.mode not in ("L", "I;16", "I", "F") else im))]
65
+
66
+
67
+ def _natural(name):
68
+ return [int(t) if t.isdigit() else t.lower() for t in re.split(r"(\d+)", Path(name).name)]
69
+
70
+
71
+ def read(path):
72
+ """(array, kind, note): kind is "image" for one layer, "stack" for several."""
73
+ path = Path(path)
74
+ size = path.stat().st_size
75
+ if size > MAX_BYTES:
76
+ raise UploadError(f"the file is {size / 2**20:.0f} MB; the limit is {MAX_BYTES // 2**20} MB")
77
+ name = path.name.lower()
78
+ if name.endswith(".npz"):
79
+ with np.load(path, allow_pickle=False) as z:
80
+ if "volume" not in z:
81
+ raise UploadError('an .npz needs an array named "volume" (layers, height, width)')
82
+ layers = [to_u8(v) for v in np.asarray(z["volume"])]
83
+ note = f"{len(layers)} layers from the .npz"
84
+ elif name.endswith(".zip"):
85
+ with zipfile.ZipFile(path) as zf:
86
+ names = sorted((n for n in zf.namelist() if n.lower().endswith(IMAGE_EXT)
87
+ and not Path(n).name.startswith(".") and "__MACOSX" not in n), key=_natural)
88
+ if not names:
89
+ raise UploadError("the ZIP holds no layer images (.tif, .png or .jpg)")
90
+ if len(names) > MAX_LAYERS:
91
+ raise UploadError(f"the ZIP holds {len(names)} images; send at most {MAX_LAYERS} layers")
92
+ layers = []
93
+ for n in names:
94
+ layers.extend(_image(zf.read(n), n))
95
+ note = f"{len(layers)} layers from the ZIP, in name order ({Path(names[0]).name} first)"
96
+ elif name.endswith(IMAGE_EXT):
97
+ layers = _image(path.read_bytes(), name)
98
+ note = "one image" if len(layers) == 1 else f"{len(layers)} pages of the TIFF as layers"
99
+ else:
100
+ raise UploadError("send an image (PNG, JPEG, TIFF), a multi-page TIFF or ZIP of layers, or an .npz")
101
+ shapes = {l.shape for l in layers}
102
+ if len(shapes) != 1:
103
+ raise UploadError(f"the layers differ in size: {sorted(shapes)}")
104
+ h, w = layers[0].shape
105
+ if min(h, w) < MIN_SIDE:
106
+ raise UploadError(f"the image is {w} x {h} px; it needs at least {MIN_SIDE} px on each side")
107
+ if len(layers) > MAX_LAYERS:
108
+ raise UploadError(f"{len(layers)} layers; send at most {MAX_LAYERS}")
109
+ if max(h, w) > MAX_SIDE:
110
+ raise UploadError(f"the image is {w} x {h} px; crop it to at most {MAX_SIDE} px on each side")
111
+ if len(layers) == 1:
112
+ return layers[0], "image", note
113
+ return np.stack(layers), "stack", note