Upload folder using huggingface_hub

.gitattributes

@@ -1,35 +1,5 @@
-*.7z filter=lfs diff=lfs merge=lfs -text
-*.arrow filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
 *.bin filter=lfs diff=lfs merge=lfs -text
-*.bz2 filter=lfs diff=lfs merge=lfs -text
-*.ckpt filter=lfs diff=lfs merge=lfs -text
-*.ftz filter=lfs diff=lfs merge=lfs -text
-*.gz filter=lfs diff=lfs merge=lfs -text
-*.h5 filter=lfs diff=lfs merge=lfs -text
-*.joblib filter=lfs diff=lfs merge=lfs -text
-*.lfs.* filter=lfs diff=lfs merge=lfs -text
-*.mlmodel filter=lfs diff=lfs merge=lfs -text
-*.model filter=lfs diff=lfs merge=lfs -text
-*.msgpack filter=lfs diff=lfs merge=lfs -text
-*.npy filter=lfs diff=lfs merge=lfs -text
-*.npz filter=lfs diff=lfs merge=lfs -text
 *.onnx filter=lfs diff=lfs merge=lfs -text
-*.ot filter=lfs diff=lfs merge=lfs -text
-*.parquet filter=lfs diff=lfs merge=lfs -text
-*.pb filter=lfs diff=lfs merge=lfs -text
-*.pickle filter=lfs diff=lfs merge=lfs -text
-*.pkl filter=lfs diff=lfs merge=lfs -text
 *.pt filter=lfs diff=lfs merge=lfs -text
 *.pth filter=lfs diff=lfs merge=lfs -text
-*.rar filter=lfs diff=lfs merge=lfs -text
-*.safetensors filter=lfs diff=lfs merge=lfs -text
-saved_model/**/* filter=lfs diff=lfs merge=lfs -text
-*.tar.* filter=lfs diff=lfs merge=lfs -text
-*.tar filter=lfs diff=lfs merge=lfs -text
-*.tflite filter=lfs diff=lfs merge=lfs -text
-*.tgz filter=lfs diff=lfs merge=lfs -text
-*.wasm filter=lfs diff=lfs merge=lfs -text
-*.xz filter=lfs diff=lfs merge=lfs -text
-*.zip filter=lfs diff=lfs merge=lfs -text
-*.zst filter=lfs diff=lfs merge=lfs -text
-*tfevents* filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,139 @@
+---
+license: mit
+language:
+- en
+library_name: graf
+tags:
+- graf
+- paragraph-segmentation
+- token-classification
+- news
+- encoder
+- modernbert
+- on-device
+pipeline_tag: token-classification
+base_model: jhu-clsp/ettin-encoder-17m
+metrics:
+- f1
+---
+
+# Graf
+
+Paragraphs will be stacked until morale improves.
+
+A small encoder that recovers paragraph breaks in flattened news text.
+
+| | |
+| --- | --- |
+| **Task** | Paragraph break recovery (token classification) |
+| **Size** | ~19 MB weights (trimmed vocab, int8 embeddings, fp16 rest) |
+| **Runtime** | CPU/GPU (PyTorch); browser (ONNX Runtime WASM) |
+| **Weights** | [`main`](https://huggingface.co/localangle/graf/tree/main) · [`web/`](https://huggingface.co/localangle/graf/tree/main/web) for WASM |
+| **Demo** | [localangle/graf-demo](https://huggingface.co/spaces/localangle/graf-demo) (in-browser) |
+
+Graf does **not** rewrite the article. It scores eligible token ends as
+`BREAK` / `NO_BREAK`, then splits the flattened string at predicted breaks.
+
+> Flattened: *City council approved a library renovation… Construction starts in March…*
+> → Graf inserts breaks between the recovered paragraphs.
+
+## Try it
+
+- **Live demo:** [localangle/graf-demo](https://huggingface.co/spaces/localangle/graf-demo) — paste flattened news, paragraphize in the browser.
+- **Local UI:** from the [graf](https://github.com/cjdd3b/graf) repo, `graf-web --web-package checkpoints/graf-web`.
+
+## Install
+
+```bash
+pip install "graf @ git+https://github.com/cjdd3b/graf.git"
+# or, from a checkout:
+pip install -e .
+```
+
+Python ≥ 3.11. Needs `torch`, `transformers`, `safetensors`, `huggingface_hub`.
+
+## Usage
+
+```python
+from graf.infer import GrafParagraphizer
+
+graf = GrafParagraphizer("localangle/graf")  # thr from graf.json (0.41)
+result = graf.paragraphize(
+    "City council approved a library renovation after public comment. "
+    "Paulsen and Chen voted against the measure. Construction starts in March."
+)
+print("\n\n".join(result.paragraphs))
+```
+
+Load a local package folder the same way:
+
+```python
+graf = GrafParagraphizer("checkpoints/graf")
+```
+
+## Files
+
+| File | Contents |
+| --- | --- |
+| `graf.json` | Format, threshold, kept vocabulary, source metadata |
+| `weights.safetensors` | Trimmed encoder + head (int8 token embeddings, fp16 elsewhere) |
+| `encoder/` | Ettin/ModernBERT config for the trimmed vocabulary |
+| `tokenizer/` | Full tokenizer (ids remapped at inference via `vocab_map`) |
+| `web/` | Browser package: `model.onnx` (~17 MB int8), `vocab_map.bin`, `graf-web.json`, `tokenizer/` |
+
+PyTorch weights expand to fp32 at load; only storage is compressed. The `web/`
+ONNX graph is what the static demo runs in WASM (soft cap 2048 tokens).
+
+## Accuracy
+
+Held-out Extra structure v0.2 test set (publisher-disjoint), package threshold
+0.41 (student before compression; packaging keeps break agreement ≈ Jaccard 0.98):
+
+| Metric | Value |
+| --- | --- |
+| Boundary F1 | 0.871 |
+| Precision / Recall | 0.845 / 0.900 |
+| PR-AUC | 0.948 |
+| Body–body precision / recall | 0.829 / 0.894 |
+| Median \|Δ block count\| | 2.0 |
+
+## How it works
+
+Flattened article text is packed into overlapping 2048-token windows. Eligible
+token ends (positions that can host a paragraph boundary) are scored for
+`BREAK`. Probabilities are averaged across windows, thresholded, and used to
+reconstruct paragraph blocks.
+
+The published package is a compressed [Ettin](https://huggingface.co/jhu-clsp/ettin-encoder-17m)
+student (~17M) distilled for on-device / in-browser use.
+
+## Limitations
+
+- English news-style prose; other domains are unmeasured.
+- Training text / Extra labels are not redistributed here.
+- Exact full-article reconstruction is intentionally hard; treat boundary F1 /
+  body–body metrics as the primary quality signal.
+- Browser int8 ONNX drifts slightly vs the PyTorch package; the web meta
+  threshold is nudged for recall (see `web/graf-web.json`).
+
+## Built on
+
+- [Ettin encoder 17M](https://huggingface.co/jhu-clsp/ettin-encoder-17m) (MIT)
+  by JHU CLSP. Fine-tuned and compressed for Graf; see
+  `THIRD_PARTY_NOTICES.md`.
+
+## License
+
+MIT for the Graf code and this package's weights, subject to the Ettin MIT
+notice above.
+
+## Citation
+
+```bibtex
+@software{graf_2026,
+  title  = {Graf: Paragraph break recovery for flattened news},
+  author = {Local Angle},
+  year   = {2026},
+  url    = {https://huggingface.co/localangle/graf},
+}
+```

THIRD_PARTY_NOTICES.md

@@ -0,0 +1,11 @@
+# Third-party notices
+
+## Ettin / ModernBERT backbone
+
+This package's encoder is derived from
+[`jhu-clsp/ettin-encoder-17m`](https://huggingface.co/jhu-clsp/ettin-encoder-17m)
+(MIT License). Graf fine-tunes and compresses those weights; the original
+license terms continue to apply to the Ettin contribution.
+
+Paper: Weller et al., "Ettin: an Open Suite of Paired Encoders and Decoders"
+(arXiv:2507.11412).

encoder/config.json

@@ -0,0 +1,72 @@
+{
+  "architectures": [
+    "ModernBertForTokenClassification"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 42889,
+  "causal_mask": false,
+  "classifier_activation": "gelu",
+  "classifier_bias": false,
+  "classifier_dropout": 0.0,
+  "classifier_pooling": "mean",
+  "cls_token_id": 42889,
+  "decoder_bias": true,
+  "deterministic_flash_attn": false,
+  "dtype": "float32",
+  "embedding_dropout": 0.0,
+  "eos_token_id": 42890,
+  "global_attn_every_n_layers": 3,
+  "gradient_checkpointing": false,
+  "hidden_activation": "gelu",
+  "hidden_size": 256,
+  "id2label": {
+    "0": "NO_BREAK",
+    "1": "BREAK"
+  },
+  "initializer_cutoff_factor": 2.0,
+  "initializer_range": 0.02,
+  "intermediate_size": 384,
+  "is_causal": false,
+  "label2id": {
+    "BREAK": 1,
+    "NO_BREAK": 0
+  },
+  "layer_norm_eps": 1e-05,
+  "layer_types": [
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention"
+  ],
+  "local_attention": 128,
+  "max_position_embeddings": 7999,
+  "mlp_bias": false,
+  "mlp_dropout": 0.0,
+  "model_type": "modernbert",
+  "norm_bias": false,
+  "norm_eps": 1e-05,
+  "num_attention_heads": 4,
+  "num_hidden_layers": 7,
+  "pad_token_id": 42891,
+  "position_embedding_type": "sans_pos",
+  "rope_parameters": {
+    "full_attention": {
+      "rope_theta": 160000.0,
+      "rope_type": "default"
+    },
+    "sliding_attention": {
+      "rope_theta": 160000.0,
+      "rope_type": "default"
+    }
+  },
+  "sep_token_id": 42890,
+  "sparse_pred_ignore_index": -100,
+  "sparse_prediction": false,
+  "tie_word_embeddings": true,
+  "transformers_version": "5.17.0",
+  "vocab_size": 42893
+}

graf.json

@@ -0,0 +1,12 @@
+{
+  "embedding_key": "model.embeddings.tok_embeddings.weight",
+  "format": "graf-package-1",
+  "kept": 42893,
+  "max_length": 2048,
+  "name": "graf",
+  "source_model": "cjdd3b/graf-4b",
+  "storage": "token embeddings int8 per row; other weights fp16; computed in fp32",
+  "stride": 512,
+  "threshold": 0.41,
+  "vocabulary_full": 50368
+}

tokenizer/tokenizer_config.json

@@ -0,0 +1,17 @@
+{
+  "backend": "tokenizers",
+  "clean_up_tokenization_spaces": true,
+  "cls_token": "[CLS]",
+  "is_local": false,
+  "local_files_only": false,
+  "mask_token": "[MASK]",
+  "model_input_names": [
+    "input_ids",
+    "attention_mask"
+  ],
+  "model_max_length": 8192,
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "tokenizer_class": "TokenizersBackend",
+  "unk_token": "[UNK]"
+}

web/graf-web.json

@@ -0,0 +1,31 @@
+{
+  "break_label_index": 1,
+  "check": {
+    "max_abs_drift_break": 0.6738,
+    "window_tokens": 26
+  },
+  "cls_token_id": 50281,
+  "embedding_key": "model.embeddings.tok_embeddings.weight",
+  "format": "graf-web-1",
+  "max_length": 2048,
+  "model": "model.onnx",
+  "name": "graf",
+  "pad_token_id": 42891,
+  "quantization": "int8-mixed-rr",
+  "sep_token_id": 50282,
+  "size_mb": 17.4,
+  "soft_cap": 2048,
+  "source_package": {
+    "path": "checkpoints/graf",
+    "source_model": "cjdd3b/graf-4b",
+    "threshold": 0.41
+  },
+  "stride": 512,
+  "threshold": 0.37,
+  "threshold_note": "retuned on ORT int8 vs gold val (24 arts); package thr was 0.41",
+  "tokenizer": "tokenizer",
+  "unk_token_id": 50280,
+  "vocab_full": 50368,
+  "vocab_kept": 42893,
+  "vocab_map": "vocab_map.bin"
+}

web/model.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a0d09d22cf8864d7a30b1ec7b88f3b17785565bc9c18eb19fc57be9165d4bdd1
+size 17396401

web/tokenizer/tokenizer_config.json

@@ -0,0 +1,17 @@
+{
+  "backend": "tokenizers",
+  "clean_up_tokenization_spaces": true,
+  "cls_token": "[CLS]",
+  "is_local": false,
+  "local_files_only": false,
+  "mask_token": "[MASK]",
+  "model_input_names": [
+    "input_ids",
+    "attention_mask"
+  ],
+  "model_max_length": 8192,
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "tokenizer_class": "TokenizersBackend",
+  "unk_token": "[UNK]"
+}

web/vocab_map.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:aa7c3905b15ad8155c25dd1755b45b1d4ee54a3416c933f2d8dba0ed30ce50ff
+size 201472

weights.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c37723b0405c487dcdf020cbeaafbe014dedbc48e7219c1e7f9edb1a7e597a77
+size 19297568