--- icon: lucide/bot description: Dense, prescriptive reference for AI coding agents working on or integrating iscc-sct - architecture map, constraints, side effects, task recipes, change playbook, and common mistakes. --- # For Coding Agents A compressed reference for AI agents working on **iscc-sct** (ISCC Semantic Text-Code). Read this before editing the codebase or integrating the library. Terminology matches the source exactly. `iscc-sct` turns text into a cross-lingual, similarity-preserving binary ISCC-UNIT (MainType SEMANTIC, SubType TEXT). It is an experimental proof of concept, **not** part of ISO 24138:2024. ## Architecture map ### File layout | Path | Contains | | ------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `iscc_sct/main.py` | `create()` - high-level API; returns a `Metadata` object in Object-Format. | | `iscc_sct/code_semantic_text.py` | Core algorithm. `gen_text_code_semantic()` (returns a plain dict, Index-Format), splitting, embedding, pooling, binarization. Module-level `@cache` singletons: `tokenizer()`, `chunking_tokenizer()`, `splitter()`, `splitter_guarded()`, `model()`. | | `iscc_sct/models.py` | Pydantic schema: `Feature`, `FeatureSet`, `Metadata` + format converters. | | `iscc_sct/options.py` | `SctOptions` (pydantic-settings), `sct_opts` singleton, `.override()`. | | `iscc_sct/utils.py` | Codecs (base32/base64url), distances, model download + blake3 integrity, `char_to_byte_offsets`, `MODEL_DIR`/`MODEL_PATH` (override dir via `ISCC_SCT_MODEL_DIR`), `timer`. | | `iscc_sct/cli.py` | `iscc-sct` console entry point (`main()`): glob, charset detection, `gui` subcommand. | | `iscc_sct/demo.py`, `iscc_sct/app.py` | Gradio demo (Hugging Face Space). Omitted from coverage. | | `iscc_sct/dev.py` | Dev-only poe task helpers. Omitted from coverage. | | `iscc_sct/tokenizer.json` | Vendored tokenizer. Byte-exact; excluded from whitespace/EOL hooks. | | `tests/chunking_vectors.json` | Frozen chunk-boundary test vectors. Never hand-edit (see Change playbook). | ### Pipeline ```text text -> split_text() -> embed_chunks() -> mean_pooling() -> binarize() -> ISCC header + base32 ``` `gen_text_code_semantic(text, **options)` orchestrates this: 1. Reject empty text (`ValueError`), then `sct_opts.override(options)`. 1. `split_text()` -> `[(offset, chunk), ...]` at semantic boundaries (max 127 tokens, 48 overlap). 1. `embed_chunks()` tokenizes in batches (auto: 1 on CPU, 100 on CUDA), runs the ONNX model, `attention_pooling()` per chunk. 1. `mean_pooling()` averages chunk vectors into one L2-normalized document vector. 1. `binarize()` (`vec >= 0` -> bits), truncate to `bits // 8`, prefix the 2-byte ISCC header, `encode_base32()`, prepend `"ISCC:"`. ### Import flow ```text iscc_sct/__init__.py (star-imports, defines __version__) -> options -> utils -> code_semantic_text -> models -> main main -> models, code_semantic_text, options code_semantic_text -> onnxruntime (lazy import), semantic_text_splitter, tokenizers, numpy, iscc_sct as sct (uses sct.* at call time, not import time) utils -> models cli -> main, charset_normalizer ``` ### Public API `iscc_sct/__init__.py` re-exports every module's `__all__`. The public surface (`import iscc_sct as sct`): | Symbol | Source | Purpose | | ------------------------------------------------------------------------------- | ------------------ | ------------------------------------------------ | | `create(text, granular=False, **options)` | main | Primary API. Returns `Metadata` (Object-Format). | | `gen_text_code_semantic(text, **options)` | code_semantic_text | Low-level. Returns `dict` (Index-Format). | | `code_text_semantic(fp, **options)` | code_semantic_text | Same, reading a UTF-8 file path. | | `soft_hash_text_semantic(text)` | code_semantic_text | Raw 384-bit digest (`bytes`), no header. | | `embed_chunks(chunks, batch_size=None)` | code_semantic_text | Chunk list -> embedding array. | | `Metadata`, `FeatureSet`, `Feature` | models | Result schema + converters. | | `SctOptions`, `sct_opts` | options | Settings model + global instance. | | `iscc_distance`, `hamming_distance`, `cosine_similarity`, `granular_similarity` | utils | Similarity metrics. | | `encode_base32`, `decode_base32`, `encode_base64`, `decode_base64` | utils | Codecs. | | `char_to_byte_offsets`, `get_model`, `MODEL_PATH`, `timer` | utils | Helpers. | | `__version__` | `__init__` | Package version string. | ## Decision dispatch ### Which entry point? | Goal | Use | | -------------------------------------------------------------- | ------------------------------------------------------- | | One ISCC code from a string, ready-to-use object | `create(text)` -> `Metadata` (Object-Format) | | Granular per-chunk features (simprints/offsets/sizes/contents) | `create(text, granular=True)` | | Compact parallel-array result for storage/indexing | `gen_text_code_semantic(text, ...)` (Index-Format dict) | | Process a text file from disk | `code_text_semantic(path)` | | Just the raw 384-bit vector digest | `soft_hash_text_semantic(text)` -> `bytes` | | Command line / batch over files | `iscc-sct ` | ### Which similarity metric? | Inputs | Use | Returns | | ------------------------------------- | ----------------------------------------- | --------------------------------------- | | Two full ISCC code strings | `iscc_distance(iscc1, iscc2)` | Hamming distance in bits (`int`) | | Two raw digests of equal length | `hamming_distance(a, b)` | bit distance | | Two raw digests, normalized score | `cosine_similarity(a, b)` | `int` in `[-100, 100]` | | Two `Metadata` objects with simprints | `granular_similarity(a, b, threshold=80)` | `[(Feature, similarity, Feature), ...]` | ### Which feature format? | Format | Produced by | Shape | Convert with | | ------------- | -------------------------- | -------------------------------------------------------- | ----------------------------- | | Index-Format | `gen_text_code_semantic()` | parallel arrays `simprints`/`offsets`/`sizes`/`contents` | `Metadata.to_index_format()` | | Object-Format | `create()` | list of self-contained `Feature` objects | `Metadata.to_object_format()` | ## Constraints and invariants - **No base ONNX runtime.** The base package declares no `onnxruntime`. Install exactly one of the mutually exclusive `cpu` / `gpu` extras. `onnxruntime` and `onnxruntime-gpu` unpack into the same directory and clobber each other (issue #23); `tool.uv.conflicts` enforces exclusivity for uv. - **ONNX runtime is imported lazily.** `import iscc_sct`, the `iscc-sct` CLI, and `iscc-sct doctor` load without any runtime installed. `load_onnxruntime()` (called from `model()` on first code generation) raises a guarded `ImportError` with install instructions - keep that guard intact. - **Codes are base32; simprints are base64url.** Never cross the codecs. - **ISCC header is fixed:** MainType `SEMANTIC` (`0001`), SubType `TEXT` (`0000`), Version `0000`, plus a length nibble from `BIT_LEN_MAP`. 2 bytes, big-endian. `iscc_distance` strips the 5-char `ISCC:` prefix, base32-decodes, drops the 2-byte header, then compares bodies of equal length. - **`bits` constraints:** `32 <= bits <= 256`, `multiple_of=32`. Same for `bits_granular`. - **Default bit-length differs by entry point:** `SctOptions.bits` / `create()` default to **64**; the **CLI** `--bits` defaults to **256**. - **`SctOptions` validates on assignment.** Mutate only via `.override(dict)`, which deep-copies and sets fields individually so validators run. Options flow as `**opts.model_dump()` through all layers. Env vars use the `ISCC_SCT_` prefix; `.env` is loaded. - **Chunk boundaries are frozen** by `tests/chunking_vectors.json` and identical on both the normal (`splitter`) and guarded (`splitter_guarded`) paths. `needs_split_guard()` routes texts whose positions sit more than `SPLIT_GUARD_GAP` (8192) chars from the next paragraph separator to the guarded sizer (PDF-extracted text, issue #24). - **Singletons:** `tokenizer()`, `chunking_tokenizer()`, `splitter(**opts)`, `splitter_guarded(**opts)`, `model()` are `@cache`d. The splitter cache keys on the option kwargs, so all option values must be hashable. - **Two tokenizers:** `tokenizer()` keeps the vendored truncation (128) + padding and embeds chunks; `chunking_tokenizer()` disables truncation/padding and sizes chunks. Truncation on the sizer makes `tokenizers` >=0.23 emit overflow encodings, the root cause of the issue #24 super-linear chunking. Both yield identical boundaries, so don't merge them. - **Coverage must stay at 100%** (`--cov-fail-under=100`). GPU branches and model download use `# pragma: no cover`. `dev.py`, `demo.py`, and `tests/` are omitted from coverage. - **Style:** PEP 484 **type comments** (first line after `def`), PEP 585 generics, PEP 604 unions. Short pure functions, max 3 args, no nested functions, sphinx-style docstrings. Ruff line length 100, LF endings, rule F401 disabled. ## Side effects catalog | Function | Effect | | ------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `model()` (first call) | If the model file is absent/corrupt, `get_model()` **downloads ~450 MB** to `MODEL_DIR` (platformdirs user-data dir, or `ISCC_SCT_MODEL_DIR`) and verifies a blake3 checksum. Creates a cached ONNX `InferenceSession`. | | `get_model()` | Network download + disk write to `MODEL_PATH`; blake3 integrity check (`# pragma: no cover`). | | `tokenizer()`, `chunking_tokenizer()`, `splitter()`, `splitter_guarded()` | Populate process-global `@cache` (loaded once). | | `model()` | Calls `rt.preload_dlls()` when CUDA is available; logs a warning via `warn_gpu_shadowed()` if `onnxruntime-gpu` is installed but CUDA is missing. | | `import iscc_sct.options` | `load_dotenv()` reads `.env`; instantiates `sct_opts`. | | `import iscc_sct.utils` | Resolves `MODEL_DIR` from `ISCC_SCT_MODEL_DIR` (else user data dir); `os.makedirs(MODEL_DIR)` (idempotent). | | `create()`, `gen_text_code_semantic()` | Pure given a loaded model: no disk writes, no mutation of inputs. | | CLI `main()` | Reads files, prints to stdout, removes the loguru logger unless `--debug`. | ## Task recipes ### Generate a code (library) ```python import iscc_sct as sct meta = sct.create("This is some sample text.", bits=256) print(meta.iscc) # "ISCC:CAD..." print(meta.characters) # input length in characters ``` ### Granular per-chunk features ```python import iscc_sct as sct meta = sct.create(long_text, bits=256, granular=True) for feature in meta.features[0].simprints: # Object-Format print(feature.offset, feature.size, feature.simprint, feature.content[:40]) ``` ### Compare two texts (cross-lingual) ```python import iscc_sct as sct a = sct.create("An ISCC applies to a specific digital asset...") b = sct.create("Ein ISCC bezieht sich auf ein bestimmtes digitales Gut...") print(sct.iscc_distance(a.iscc, b.iscc)) # low bit distance => similar ``` ### Match granular simprints ```python import iscc_sct as sct a = sct.create(doc_a, granular=True) b = sct.create(doc_b, granular=True) for feat_a, similarity, feat_b in sct.granular_similarity(a, b, threshold=80): print(similarity, feat_a.offset, feat_b.offset) ``` ### Configure via options ```python import iscc_sct as sct # Per-call override (validated copy of the global settings): meta = sct.create(text, bits=128, contents=True) # Or globally via environment: ISCC_SCT_BITS=128, ISCC_SCT_MAX_TOKENS=127, ... ``` ## Change playbook | If you change... | Also update... | | ----------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | The embedding model file | `MODEL_FILENAME`, `MODEL_URL`, `MODEL_CHECKSUM` in `utils.py`; verify codes are bit-stable or bump version + mark BREAKING in CHANGELOG. | | Chunking (`max_tokens`, `overlap`, `split_text`, `needs_split_guard`, token sizing) | Regenerate `tests/chunking_vectors.json` via `uv run python tests/test_chunking_vectors.py` (only for an intentional algorithm change); multi-chunk codes shift. | | A public function name/signature | The module's `__all__` (it is star-exported through `__init__.py`); README/docs examples. | | Add a processing option | Add a field to `SctOptions` (env var name, constraints); it flows via `model_dump()`. Wire into `create()` if it is a granular toggle; add to the README config table. | | `Metadata` / `FeatureSet` / `Feature` schema | Keep `to_index_format()`, `to_object_format()`, `get_content()`, `get_overlaps()` consistent. | | ISCC header constants (`MAINTYPE`/`SUBTYPE`/`SCT_VERSION`/`BIT_LEN_MAP`) | Codes change - this is BREAKING; bump version and document. | | onnxruntime import / provider logic | Keep the `cpu`/`gpu` extras, `tool.uv.conflicts`, the `ImportError` guard, and `warn_gpu_shadowed()` in sync. | | Any code path | Add tests to keep coverage at 100%; use `# pragma: no cover` only for GPU/download branches. Run `uv run poe all`. | | Dependencies | Regenerate `requirements.txt` via `uv run poe export-requirements` (never hand-edit it). | ## Common mistakes **NEVER** add `onnxruntime` as a base dependency to "make install easier." ```toml # WRONG - re-breaks issue #23: [gpu] then installs both wheels and CPU clobbers GPU dependencies = ["onnxruntime"] ``` **ALWAYS** keep it behind the mutually exclusive `cpu` / `gpu` extras. --- **NEVER** use inline type annotations. ```python def binarize(vec: NDArray) -> bytes: ... # WRONG for this codebase ``` **ALWAYS** use PEP 484 type comments as the first line after `def`. ```python def binarize(vec): # type: (NDArray) -> bytes ... ``` --- **NEVER** expect Object-Format from the low-level function. ```python meta = sct.gen_text_code_semantic(text) # returns a dict in INDEX-Format meta.features[0].simprints # WRONG - it is a dict, not Metadata ``` **ALWAYS** use `create()` for an Object-Format `Metadata`, or wrap the dict: `Metadata(**data)`. --- **NEVER** mutate `SctOptions` fields directly or assume CLI/library defaults match. ```python sct.sct_opts.bits = 256 # avoid - mutates the global; use .override() sct.create(text) # library default bits=64, NOT the CLI's 256 ``` **ALWAYS** pass overrides per call (`create(text, bits=256)`) or via `sct_opts.override({...})`. --- **NEVER** hand-edit `tests/chunking_vectors.json`, `requirements.txt`, `iscc_sct/tokenizer.json`, or the model file. Each is generated or byte-exact. **ALWAYS** regenerate via the documented command and run `uv run poe all` before reporting done. --- **NEVER** mix codecs: ISCC codes are RFC4648 base32 (no padding); granular simprints are base64url (no padding). Decode with the matching `decode_base32` / `decode_base64`.