File size: 21,556 Bytes
8ef3a6b | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 | ---
icon: lucide/bot
description: Dense, prescriptive reference for AI coding agents working on or integrating iscc-sct - architecture map, constraints, side effects, task recipes, change playbook, and common mistakes.
---
# For Coding Agents
A compressed reference for AI agents working on **iscc-sct** (ISCC Semantic Text-Code). Read this
before editing the codebase or integrating the library. Terminology matches the source exactly.
`iscc-sct` turns text into a cross-lingual, similarity-preserving binary ISCC-UNIT (MainType
SEMANTIC, SubType TEXT). It is an experimental proof of concept, **not** part of ISO 24138:2024.
## Architecture map
### File layout
| Path | Contains |
| ------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `iscc_sct/main.py` | `create()` - high-level API; returns a `Metadata` object in Object-Format. |
| `iscc_sct/code_semantic_text.py` | Core algorithm. `gen_text_code_semantic()` (returns a plain dict, Index-Format), splitting, embedding, pooling, binarization. Module-level `@cache` singletons: `tokenizer()`, `chunking_tokenizer()`, `splitter()`, `splitter_guarded()`, `model()`. |
| `iscc_sct/models.py` | Pydantic schema: `Feature`, `FeatureSet`, `Metadata` + format converters. |
| `iscc_sct/options.py` | `SctOptions` (pydantic-settings), `sct_opts` singleton, `.override()`. |
| `iscc_sct/utils.py` | Codecs (base32/base64url), distances, model download + blake3 integrity, `char_to_byte_offsets`, `MODEL_DIR`/`MODEL_PATH` (override dir via `ISCC_SCT_MODEL_DIR`), `timer`. |
| `iscc_sct/cli.py` | `iscc-sct` console entry point (`main()`): glob, charset detection, `gui` subcommand. |
| `iscc_sct/demo.py`, `iscc_sct/app.py` | Gradio demo (Hugging Face Space). Omitted from coverage. |
| `iscc_sct/dev.py` | Dev-only poe task helpers. Omitted from coverage. |
| `iscc_sct/tokenizer.json` | Vendored tokenizer. Byte-exact; excluded from whitespace/EOL hooks. |
| `tests/chunking_vectors.json` | Frozen chunk-boundary test vectors. Never hand-edit (see Change playbook). |
### Pipeline
```text
text -> split_text() -> embed_chunks() -> mean_pooling() -> binarize() -> ISCC header + base32
```
`gen_text_code_semantic(text, **options)` orchestrates this:
1. Reject empty text (`ValueError`), then `sct_opts.override(options)`.
1. `split_text()` -> `[(offset, chunk), ...]` at semantic boundaries (max 127 tokens, 48 overlap).
1. `embed_chunks()` tokenizes in batches (auto: 1 on CPU, 100 on CUDA), runs the ONNX model,
`attention_pooling()` per chunk.
1. `mean_pooling()` averages chunk vectors into one L2-normalized document vector.
1. `binarize()` (`vec >= 0` -> bits), truncate to `bits // 8`, prefix the 2-byte ISCC header,
`encode_base32()`, prepend `"ISCC:"`.
### Import flow
```text
iscc_sct/__init__.py (star-imports, defines __version__)
-> options -> utils -> code_semantic_text -> models -> main
main -> models, code_semantic_text, options
code_semantic_text -> onnxruntime (lazy import), semantic_text_splitter, tokenizers, numpy,
iscc_sct as sct (uses sct.* at call time, not import time)
utils -> models
cli -> main, charset_normalizer
```
### Public API
`iscc_sct/__init__.py` re-exports every module's `__all__`. The public surface
(`import iscc_sct as sct`):
| Symbol | Source | Purpose |
| ------------------------------------------------------------------------------- | ------------------ | ------------------------------------------------ |
| `create(text, granular=False, **options)` | main | Primary API. Returns `Metadata` (Object-Format). |
| `gen_text_code_semantic(text, **options)` | code_semantic_text | Low-level. Returns `dict` (Index-Format). |
| `code_text_semantic(fp, **options)` | code_semantic_text | Same, reading a UTF-8 file path. |
| `soft_hash_text_semantic(text)` | code_semantic_text | Raw 384-bit digest (`bytes`), no header. |
| `embed_chunks(chunks, batch_size=None)` | code_semantic_text | Chunk list -> embedding array. |
| `Metadata`, `FeatureSet`, `Feature` | models | Result schema + converters. |
| `SctOptions`, `sct_opts` | options | Settings model + global instance. |
| `iscc_distance`, `hamming_distance`, `cosine_similarity`, `granular_similarity` | utils | Similarity metrics. |
| `encode_base32`, `decode_base32`, `encode_base64`, `decode_base64` | utils | Codecs. |
| `char_to_byte_offsets`, `get_model`, `MODEL_PATH`, `timer` | utils | Helpers. |
| `__version__` | `__init__` | Package version string. |
## Decision dispatch
### Which entry point?
| Goal | Use |
| -------------------------------------------------------------- | ------------------------------------------------------- |
| One ISCC code from a string, ready-to-use object | `create(text)` -> `Metadata` (Object-Format) |
| Granular per-chunk features (simprints/offsets/sizes/contents) | `create(text, granular=True)` |
| Compact parallel-array result for storage/indexing | `gen_text_code_semantic(text, ...)` (Index-Format dict) |
| Process a text file from disk | `code_text_semantic(path)` |
| Just the raw 384-bit vector digest | `soft_hash_text_semantic(text)` -> `bytes` |
| Command line / batch over files | `iscc-sct <glob>` |
### Which similarity metric?
| Inputs | Use | Returns |
| ------------------------------------- | ----------------------------------------- | --------------------------------------- |
| Two full ISCC code strings | `iscc_distance(iscc1, iscc2)` | Hamming distance in bits (`int`) |
| Two raw digests of equal length | `hamming_distance(a, b)` | bit distance |
| Two raw digests, normalized score | `cosine_similarity(a, b)` | `int` in `[-100, 100]` |
| Two `Metadata` objects with simprints | `granular_similarity(a, b, threshold=80)` | `[(Feature, similarity, Feature), ...]` |
### Which feature format?
| Format | Produced by | Shape | Convert with |
| ------------- | -------------------------- | -------------------------------------------------------- | ----------------------------- |
| Index-Format | `gen_text_code_semantic()` | parallel arrays `simprints`/`offsets`/`sizes`/`contents` | `Metadata.to_index_format()` |
| Object-Format | `create()` | list of self-contained `Feature` objects | `Metadata.to_object_format()` |
## Constraints and invariants
- **No base ONNX runtime.** The base package declares no `onnxruntime`. Install exactly one of the
mutually exclusive `cpu` / `gpu` extras. `onnxruntime` and `onnxruntime-gpu` unpack into the
same directory and clobber each other (issue #23); `tool.uv.conflicts` enforces exclusivity for
uv.
- **ONNX runtime is imported lazily.** `import iscc_sct`, the `iscc-sct` CLI, and `iscc-sct doctor`
load without any runtime installed. `load_onnxruntime()` (called from `model()` on first code
generation) raises a guarded `ImportError` with install instructions - keep that guard intact.
- **Codes are base32; simprints are base64url.** Never cross the codecs.
- **ISCC header is fixed:** MainType `SEMANTIC` (`0001`), SubType `TEXT` (`0000`), Version `0000`,
plus a length nibble from `BIT_LEN_MAP`. 2 bytes, big-endian. `iscc_distance` strips the 5-char
`ISCC:` prefix, base32-decodes, drops the 2-byte header, then compares bodies of equal length.
- **`bits` constraints:** `32 <= bits <= 256`, `multiple_of=32`. Same for `bits_granular`.
- **Default bit-length differs by entry point:** `SctOptions.bits` / `create()` default to **64**;
the **CLI** `--bits` defaults to **256**.
- **`SctOptions` validates on assignment.** Mutate only via `.override(dict)`, which deep-copies and
sets fields individually so validators run. Options flow as `**opts.model_dump()` through all
layers. Env vars use the `ISCC_SCT_` prefix; `.env` is loaded.
- **Chunk boundaries are frozen** by `tests/chunking_vectors.json` and identical on both the normal
(`splitter`) and guarded (`splitter_guarded`) paths. `needs_split_guard()` routes texts whose
positions sit more than `SPLIT_GUARD_GAP` (8192) chars from the next paragraph separator to the
guarded sizer (PDF-extracted text, issue #24).
- **Singletons:** `tokenizer()`, `chunking_tokenizer()`, `splitter(**opts)`,
`splitter_guarded(**opts)`, `model()` are `@cache`d. The splitter cache keys on the option
kwargs, so all option values must be hashable.
- **Two tokenizers:** `tokenizer()` keeps the vendored truncation (128) + padding and embeds chunks;
`chunking_tokenizer()` disables truncation/padding and sizes chunks. Truncation on the sizer
makes `tokenizers` >=0.23 emit overflow encodings, the root cause of the issue #24 super-linear
chunking. Both yield identical boundaries, so don't merge them.
- **Coverage must stay at 100%** (`--cov-fail-under=100`). GPU branches and model download use
`# pragma: no cover`. `dev.py`, `demo.py`, and `tests/` are omitted from coverage.
- **Style:** PEP 484 **type comments** (first line after `def`), PEP 585 generics, PEP 604 unions.
Short pure functions, max 3 args, no nested functions, sphinx-style docstrings. Ruff line length
100, LF endings, rule F401 disabled.
## Side effects catalog
| Function | Effect |
| ------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `model()` (first call) | If the model file is absent/corrupt, `get_model()` **downloads ~450 MB** to `MODEL_DIR` (platformdirs user-data dir, or `ISCC_SCT_MODEL_DIR`) and verifies a blake3 checksum. Creates a cached ONNX `InferenceSession`. |
| `get_model()` | Network download + disk write to `MODEL_PATH`; blake3 integrity check (`# pragma: no cover`). |
| `tokenizer()`, `chunking_tokenizer()`, `splitter()`, `splitter_guarded()` | Populate process-global `@cache` (loaded once). |
| `model()` | Calls `rt.preload_dlls()` when CUDA is available; logs a warning via `warn_gpu_shadowed()` if `onnxruntime-gpu` is installed but CUDA is missing. |
| `import iscc_sct.options` | `load_dotenv()` reads `.env`; instantiates `sct_opts`. |
| `import iscc_sct.utils` | Resolves `MODEL_DIR` from `ISCC_SCT_MODEL_DIR` (else user data dir); `os.makedirs(MODEL_DIR)` (idempotent). |
| `create()`, `gen_text_code_semantic()` | Pure given a loaded model: no disk writes, no mutation of inputs. |
| CLI `main()` | Reads files, prints to stdout, removes the loguru logger unless `--debug`. |
## Task recipes
### Generate a code (library)
```python
import iscc_sct as sct
meta = sct.create("This is some sample text.", bits=256)
print(meta.iscc) # "ISCC:CAD..."
print(meta.characters) # input length in characters
```
### Granular per-chunk features
```python
import iscc_sct as sct
meta = sct.create(long_text, bits=256, granular=True)
for feature in meta.features[0].simprints: # Object-Format
print(feature.offset, feature.size, feature.simprint, feature.content[:40])
```
### Compare two texts (cross-lingual)
```python
import iscc_sct as sct
a = sct.create("An ISCC applies to a specific digital asset...")
b = sct.create("Ein ISCC bezieht sich auf ein bestimmtes digitales Gut...")
print(sct.iscc_distance(a.iscc, b.iscc)) # low bit distance => similar
```
### Match granular simprints
```python
import iscc_sct as sct
a = sct.create(doc_a, granular=True)
b = sct.create(doc_b, granular=True)
for feat_a, similarity, feat_b in sct.granular_similarity(a, b, threshold=80):
print(similarity, feat_a.offset, feat_b.offset)
```
### Configure via options
```python
import iscc_sct as sct
# Per-call override (validated copy of the global settings):
meta = sct.create(text, bits=128, contents=True)
# Or globally via environment: ISCC_SCT_BITS=128, ISCC_SCT_MAX_TOKENS=127, ...
```
## Change playbook
| If you change... | Also update... |
| ----------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| The embedding model file | `MODEL_FILENAME`, `MODEL_URL`, `MODEL_CHECKSUM` in `utils.py`; verify codes are bit-stable or bump version + mark BREAKING in CHANGELOG. |
| Chunking (`max_tokens`, `overlap`, `split_text`, `needs_split_guard`, token sizing) | Regenerate `tests/chunking_vectors.json` via `uv run python tests/test_chunking_vectors.py` (only for an intentional algorithm change); multi-chunk codes shift. |
| A public function name/signature | The module's `__all__` (it is star-exported through `__init__.py`); README/docs examples. |
| Add a processing option | Add a field to `SctOptions` (env var name, constraints); it flows via `model_dump()`. Wire into `create()` if it is a granular toggle; add to the README config table. |
| `Metadata` / `FeatureSet` / `Feature` schema | Keep `to_index_format()`, `to_object_format()`, `get_content()`, `get_overlaps()` consistent. |
| ISCC header constants (`MAINTYPE`/`SUBTYPE`/`SCT_VERSION`/`BIT_LEN_MAP`) | Codes change - this is BREAKING; bump version and document. |
| onnxruntime import / provider logic | Keep the `cpu`/`gpu` extras, `tool.uv.conflicts`, the `ImportError` guard, and `warn_gpu_shadowed()` in sync. |
| Any code path | Add tests to keep coverage at 100%; use `# pragma: no cover` only for GPU/download branches. Run `uv run poe all`. |
| Dependencies | Regenerate `requirements.txt` via `uv run poe export-requirements` (never hand-edit it). |
## Common mistakes
**NEVER** add `onnxruntime` as a base dependency to "make install easier."
```toml
# WRONG - re-breaks issue #23: [gpu] then installs both wheels and CPU clobbers GPU
dependencies = ["onnxruntime"]
```
**ALWAYS** keep it behind the mutually exclusive `cpu` / `gpu` extras.
---
**NEVER** use inline type annotations.
```python
def binarize(vec: NDArray) -> bytes: ... # WRONG for this codebase
```
**ALWAYS** use PEP 484 type comments as the first line after `def`.
```python
def binarize(vec):
# type: (NDArray) -> bytes
...
```
---
**NEVER** expect Object-Format from the low-level function.
```python
meta = sct.gen_text_code_semantic(text) # returns a dict in INDEX-Format
meta.features[0].simprints # WRONG - it is a dict, not Metadata
```
**ALWAYS** use `create()` for an Object-Format `Metadata`, or wrap the dict: `Metadata(**data)`.
---
**NEVER** mutate `SctOptions` fields directly or assume CLI/library defaults match.
```python
sct.sct_opts.bits = 256 # avoid - mutates the global; use .override()
sct.create(text) # library default bits=64, NOT the CLI's 256
```
**ALWAYS** pass overrides per call (`create(text, bits=256)`) or via `sct_opts.override({...})`.
---
**NEVER** hand-edit `tests/chunking_vectors.json`, `requirements.txt`, `iscc_sct/tokenizer.json`, or
the model file. Each is generated or byte-exact.
**ALWAYS** regenerate via the documented command and run `uv run poe all` before reporting done.
---
**NEVER** mix codecs: ISCC codes are RFC4648 base32 (no padding); granular simprints are base64url
(no padding). Decode with the matching `decode_base32` / `decode_base64`.
|