README.md

---
license: mit
library_name: pytorch
pipeline_tag: image-to-text
tags:
- image-to-text
- mathematical-expression-recognition
- math-ocr
- typst
- onnx
- browser-inference
inference: false
---

**Source code and browser application:** [GitHub](https://github.com/dbccccccc/IBEM-im2typst)

# IBEM im2typst — Phase 10

An experimental printed-formula image-to-Typst recognizer trained from project
seeded random initialization using only real IBEM formula crops. This repository
contains the current best validation-selected Phase 10 step-2,000 weights.

## Status

This model is **not production-qualified**.

- Parameters: 14,157,152
- FP32 validation exact match: 92.6268% over 16,899 document-disjoint images
- At-most-32-token exact match: 97.8519%
- 33–64-token exact match: 88.4282% (below the 90% production gate)
- Displayed-formula exact match: 70.1197%
- Test split and confidence calibration: unopened
- INT8 built-in export probes: exact token parity
- Full-domain INT8 evaluation: not completed

The displayed token score is uncalibrated and is not the probability that a
formula is correct. Inspect and compile every generated Typst expression.

## Files

| File | Purpose |
| --- | --- |
| `checkpoint-step-002000-transfer.pt` | 56.7 MB optimizer-free FP32 PyTorch transfer checkpoint |
| `checkpoint-metadata.json` | Checkpoint identity, lineage, hashes, and removed-state declaration |
| `encoder.int8.onnx` | Dynamic-INT8 browser encoder |
| `decoder-step.int8.onnx` | Cached autoregressive INT8 decoder step |
| `model-config.json` | Recognizer architecture |
| `preprocess-config.json` | Locked input preprocessing |
| `vocabulary.json` | 528-token vocabulary with UTF-8 byte fallback |
| `output-policy.json` | Locked package-free Typst output surface |
| `browser-deployment.json` | Original browser export identity and artifact evidence |
| `evaluation.json` | Complete FP32 validation report |
| `onnx-parity.json` | FP32, FP16, and INT8 export-probe parity evidence |

The PyTorch file retains `model_state` and the lineage metadata required for
exact-model transfer. Optimizer state, RNG state, epoch/batch cursor, and cached
metrics were deliberately removed. It cannot resume the original optimizer.

`browser-deployment.json` preserves the identity of the original
multi-precision export and therefore names FP16/FP32 ONNX variants that are not
copied here. This model repository contains the FP32 PyTorch state and the two
small INT8 ONNX graphs; the unused FP16 and reference ONNX duplicates are
intentionally omitted.

## Using the weights

The model uses a project-specific `cnn-transformer-v3` architecture rather
than a Transformers architecture, so the hosted Hugging Face inference widget
is disabled. Use the companion GitHub source repository for the browser app,
Python model definition, preprocessing, training, evaluation, and export code.

After installing that Python package, the FP32 checkpoint can be loaded with:

```python
import torch

from synthetic_im2typst.model import ImageToTypstRecognizer, ModelConfig

checkpoint = torch.load(
    "checkpoint-step-002000-transfer.pt",
    map_location="cpu",
    weights_only=True,
)
config = ModelConfig.from_data(checkpoint["model_config"])
model = ImageToTypstRecognizer(config)
model.load_state_dict(checkpoint["model_state"], strict=True)
model.eval()
```

The two ONNX graphs are a cached encoder/decoder pair, not independent
single-call models. Use them together with `vocabulary.json`,
`preprocess-config.json`, and the browser inference runtime from the companion
source repository.

## Architecture and input

The recognizer uses a compact convolutional encoder, depthwise axial 2-D
context, a height-4 learned horizontal-preserving pool, and a three-layer
autoregressive Transformer decoder. Input is foreground-cropped,
polarity-normalized, aspect-preserving resized, and padded to a 256-pixel
height with a maximum width of 2,048 pixels.

It expects one tightly cropped machine-typeset formula. It is not a page
segmenter, general OCR system, handwriting model, or photograph recognizer.

## Training data

- 129,303 real training crops from 480 IBEM documents.
- 16,899 validation crops from 60 different documents.
- Images are grayscale crops from original IBEM page JPEGs, not formulas
  rerendered in one standardized font.
- Targets are deterministic package-free Typst conversions of IBEM
  `latex_norm` annotations.
- No Fusion dataset, synthetic formula corpus, teacher-generated labels,
  external pretrained model, or fused samples were used.

IBEM is available from its
[official Zenodo record](https://zenodo.org/records/7963703) under CC BY 4.0.
No IBEM image or annotation is included. The image under `example/` is an
original project-authored Typst render included only as a smoke-test sample.

## Known limitations

- Small dot accents, top attachments, matrices, and alignment structures are
  common compact-formula errors.
- Long and multiline displayed formulas are substantially weaker than short
  inline formulas.
- Inputs with handwriting, perspective distortion, surrounding prose, or
  arbitrary screenshot styling are not qualified.
- Exact-match evaluation is strict: one wrong token makes an output non-exact.

## Identity and license

- Original full checkpoint SHA-256:
  `29d4a51adc71526f395750bb12a1783fbfd9bfa5a4c68fc9bfb665b68c633d52`
- Optimizer-free checkpoint SHA-256:
  `8b7972f440556f328828f9e90247f266d9fbc59266f5ff8b23f7bce21f717924`
- Parameter-state hash:
  `dcbf1a037fee205c82b7c52c9d0c0c2ee3d68d4a514f4593e15711f02bad31ec`
- Training run ID:
  `64e092c2013ed43e35d3573e3e2a161c9948f12279ebb92513ab6b15c8811814`

Project code, the identified weights, and the project-authored example are MIT
licensed. IBEM is not redistributed and retains its source license. See
`LICENSE`, `LICENSES.md`, `MODEL_LICENSE.md`, and `DATA_PROVENANCE.md`.