---
library_name: transformers
pipeline_tag: token-classification
base_model: distilroberta-base
tags:
- newline-restoration
- text-segmentation
---

# Newline Fixer — gap-classification encoder

Fine-tuned `distilroberta-base` that classifies the separator ("gap") between each pair of
consecutive words into JOIN / SPACE / NEWLINE / PARA. Reconstructing text from words
plus predicted gaps restores newline placement with a hard guarantee: the output
words are identical to the input — only whitespace changes.

Trained self-supervised on Wikipedia + arXiv-markdown text whose newline structure
was programmatically destroyed. Full methodology, evaluation protocol, and results:
the project's `report.md`.

## Serving

The raw checkpoint is a standard token-classification model, but gap decoding
(last-subtoken alignment, sliding-window stitching, reconstruction) lives in the
`newlinefix` package:

```python
from newlinefix.models.encoder import EncoderGapPredictor
from newlinefix.predict import fix_text

predictor = EncoderGapPredictor.load("preneond/newlinefix-encoder")
print(fix_text("the que\nries come from here", predictor))
```

The bundled HTTP service serves this repo directly: `NEWLINEFIX_MODEL_DIR=preneond/newlinefix-encoder`.

## Validation metrics

Macro-F1 is over the structural classes {JOIN, NEWLINE, PARA}.

| metric | value |
|---|---|
| accuracy | 0.9795 |
| macro_f1_structural | 0.7832 |
| precision_JOIN | 0.8875 |
| recall_JOIN | 0.9831 |
| f1_JOIN | 0.9329 |
| precision_SPACE | 0.9972 |
| recall_SPACE | 0.9831 |
| f1_SPACE | 0.9901 |
| precision_NEWLINE | 0.6938 |
| recall_NEWLINE | 0.8200 |
| f1_NEWLINE | 0.7516 |
| precision_PARA | 0.5414 |
| recall_PARA | 0.8624 |
| f1_PARA | 0.6652 |