| --- |
| library_name: transformers |
| pipeline_tag: token-classification |
| base_model: distilroberta-base |
| tags: |
| - newline-restoration |
| - text-segmentation |
| --- |
| |
| # Newline Fixer — gap-classification encoder |
|
|
| Fine-tuned `distilroberta-base` that classifies the separator ("gap") between each pair of |
| consecutive words into JOIN / SPACE / NEWLINE / PARA. Reconstructing text from words |
| plus predicted gaps restores newline placement with a hard guarantee: the output |
| words are identical to the input — only whitespace changes. |
|
|
| Trained self-supervised on Wikipedia + arXiv-markdown text whose newline structure |
| was programmatically destroyed. Full methodology, evaluation protocol, and results: |
| the project's `report.md`. |
|
|
| ## Serving |
|
|
| The raw checkpoint is a standard token-classification model, but gap decoding |
| (last-subtoken alignment, sliding-window stitching, reconstruction) lives in the |
| `newlinefix` package: |
|
|
| ```python |
| from newlinefix.models.encoder import EncoderGapPredictor |
| from newlinefix.predict import fix_text |
| |
| predictor = EncoderGapPredictor.load("preneond/newlinefix-encoder") |
| print(fix_text("the que\nries come from here", predictor)) |
| ``` |
|
|
| The bundled HTTP service serves this repo directly: `NEWLINEFIX_MODEL_DIR=preneond/newlinefix-encoder`. |
|
|
| ## Validation metrics |
|
|
| Macro-F1 is over the structural classes {JOIN, NEWLINE, PARA}. |
|
|
| | metric | value | |
| |---|---| |
| | accuracy | 0.9795 | |
| | macro_f1_structural | 0.7832 | |
| | precision_JOIN | 0.8875 | |
| | recall_JOIN | 0.9831 | |
| | f1_JOIN | 0.9329 | |
| | precision_SPACE | 0.9972 | |
| | recall_SPACE | 0.9831 | |
| | f1_SPACE | 0.9901 | |
| | precision_NEWLINE | 0.6938 | |
| | recall_NEWLINE | 0.8200 | |
| | f1_NEWLINE | 0.7516 | |
| | precision_PARA | 0.5414 | |
| | recall_PARA | 0.8624 | |
| | f1_PARA | 0.6652 | |
|
|
