File size: 1,732 Bytes
f9acb8b | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 | ---
library_name: transformers
pipeline_tag: token-classification
base_model: distilroberta-base
tags:
- newline-restoration
- text-segmentation
---
# Newline Fixer — gap-classification encoder
Fine-tuned `distilroberta-base` that classifies the separator ("gap") between each pair of
consecutive words into JOIN / SPACE / NEWLINE / PARA. Reconstructing text from words
plus predicted gaps restores newline placement with a hard guarantee: the output
words are identical to the input — only whitespace changes.
Trained self-supervised on Wikipedia + arXiv-markdown text whose newline structure
was programmatically destroyed. Full methodology, evaluation protocol, and results:
the project's `report.md`.
## Serving
The raw checkpoint is a standard token-classification model, but gap decoding
(last-subtoken alignment, sliding-window stitching, reconstruction) lives in the
`newlinefix` package:
```python
from newlinefix.models.encoder import EncoderGapPredictor
from newlinefix.predict import fix_text
predictor = EncoderGapPredictor.load("preneond/newlinefix-encoder")
print(fix_text("the que\nries come from here", predictor))
```
The bundled HTTP service serves this repo directly: `NEWLINEFIX_MODEL_DIR=preneond/newlinefix-encoder`.
## Validation metrics
Macro-F1 is over the structural classes {JOIN, NEWLINE, PARA}.
| metric | value |
|---|---|
| accuracy | 0.9795 |
| macro_f1_structural | 0.7832 |
| precision_JOIN | 0.8875 |
| recall_JOIN | 0.9831 |
| f1_JOIN | 0.9329 |
| precision_SPACE | 0.9972 |
| recall_SPACE | 0.9831 |
| f1_SPACE | 0.9901 |
| precision_NEWLINE | 0.6938 |
| recall_NEWLINE | 0.8200 |
| f1_NEWLINE | 0.7516 |
| precision_PARA | 0.5414 |
| recall_PARA | 0.8624 |
| f1_PARA | 0.6652 |
|