Bilingual humanize-text-model (EN T5-small + ZH Chinese T5-small)

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Lynote
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -0,0 +1,126 @@
+---
+license: mit
+language:
+  - en
+  - zh
+tags:
+  - text2text-generation
+  - text-humanizer
+  - rewriting
+  - writing-assistant
+  - bilingual
+  - responsible-ai
+pipeline_tag: text2text-generation
+base_model:
+  - google-t5/t5-small
+  - uer/t5-small-chinese-cluecorpussmall
+inference: true
+widget:
+  - text: "It is important to note that this robust solution serves as a testament to our commitment to innovation. Moreover, we leverage cutting-edge technology."
+  - text: "值得注意的是，在当今快速发展的时代，我们通过赋能团队来助力企业实现降本增效。"
+---
+
+# Humanize Text Model (Lynote)
+
+A lightweight bilingual (English/Chinese) humanizer: two small T5-seq2seq
+checkpoints in one repository that rewrite AI-flavored prose into more
+natural, human-flavored prose.
+
+- **`en/`** — fine-tuned `google-t5/t5-small` (English)
+- **`zh/`** — fine-tuned `uer/t5-small-chinese-cluecorpussmall` (Chinese)
+
+## What it does
+
+- Removes high-confidence AI clichés and formulaic phrases (e.g.
+  *"it is important to note that"*, *"moreover"*, *"值得注意的是"*,
+  *"降本增效"*).
+- Keeps already-human prose nearly untouched (identity learning).
+- Preserves numbers, URLs, file paths, code and quoted text (protected with
+  `PROTECTED_N` placeholders during generation, restored afterwards).
+- Routes automatically by language (CJK ratio detection).
+
+## What it is NOT
+
+This is a **writing-quality aid**, not a tool for evading AI detectors.
+Detector scores are probabilistic, and no humanizer can guarantee that text
+will be classified as human. Please use it responsibly:
+do not use it to misrepresent authorship in academic, legal, or disciplinary
+contexts.
+
+## Quickstart
+
+```bash
+pip install transformers torch
+```
+
+```python
+from humanize import Humanizer  # the wrapper bundled in this repo
+
+h = Humanizer()  # loads this repo (en/ and zh/ sub-checkpoints)
+print(h.humanize(
+    "It is important to note that this robust solution serves as a "
+    "testament to our commitment. Moreover, we leverage cutting-edge "
+    "technology."
+))
+print(h.humanize("值得注意的是，我们通过赋能团队来实现降本增效。"))
+```
+
+Raw `transformers` usage (no wrapper):
+
+```python
+from transformers import T5ForConditionalGeneration, T5Tokenizer
+
+model = T5ForConditionalGeneration.from_pretrained("Danny-Lynote/humanize-text-model/en")
+tokenizer = T5Tokenizer.from_pretrained("Danny-Lynote/humanize-text-model/en")
+inputs = tokenizer("It is important to note that this is robust.", return_tensors="pt")
+print(tokenizer.decode(model.generate(**inputs, max_length=128)[0], skip_special_tokens=True))
+```
+
+For Chinese use the `zh/` sub-checkpoint with `BertTokenizer`.
+
+## Training data
+
+The corpus is generated deterministically from the editing principles of the
+Lynote reference projects (`humanize-text`, `humanize-text-skill`,
+`humanizer-lite`):
+
+1. **AI → human**: formulaic clause combinations rewritten by a conservative
+   rule engine,
+2. **human → human (identity)**: clean prose unchanged, so the model learns
+   not to rewrite good text,
+3. **mixed**: clean prose with one injected cliché that must be removed,
+4. **protected spans**: examples with URLs, numbers, code and quotes.
+
+Reproduce:
+
+```bash
+python scripts/build_dataset.py --out data   # 14.7k pairs (en + zh)
+python scripts/train.py --lang en --epochs 3  # -> checkpoints/humanize-text-model/en
+python scripts/train.py --lang zh --epochs 3  # -> checkpoints/humanize-text-model/zh
+python scripts/evaluate.py                    # benchmark on held-out test
+pytest tests/                                 # full test suite
+```
+
+## Evaluation (held-out test, 500 AI->human + all identity/protected pairs)
+
+| Metric | Value |
+|---|---|
+| Corpus BLEU vs rule reference | **99.2** |
+| Cliché removal rate | **100.0%** (435/435) |
+| Identity stability (clean prose, n=67) | **73.1%** |
+| Protected-span preservation (n=253) | **96.8%** |
+| Throughput (MPS) | ~218 chars/s |
+
+## Limitations
+
+- Trained on synthetic text; real-world inputs may need light post-editing.
+- One checkpoint per language (English / Chinese); other languages are not
+  specifically trained.
+- Long inputs are truncated to 256 tokens.
+- It is a conservative editor: it will not add stylistic richness that is
+  absent from the source text.
+
+## License
+
+MIT. Base models: `google-t5/t5-small` (Apache-2.0) and
+`uer/t5-small-chinese-cluecorpussmall` (Apache-2.0).

en/added_tokens.json

@@ -0,0 +1,102 @@
+{
+  "<extra_id_0>": 32099,
+  "<extra_id_10>": 32089,
+  "<extra_id_11>": 32088,
+  "<extra_id_12>": 32087,
+  "<extra_id_13>": 32086,
+  "<extra_id_14>": 32085,
+  "<extra_id_15>": 32084,
+  "<extra_id_16>": 32083,
+  "<extra_id_17>": 32082,
+  "<extra_id_18>": 32081,
+  "<extra_id_19>": 32080,
+  "<extra_id_1>": 32098,
+  "<extra_id_20>": 32079,
+  "<extra_id_21>": 32078,
+  "<extra_id_22>": 32077,
+  "<extra_id_23>": 32076,
+  "<extra_id_24>": 32075,
+  "<extra_id_25>": 32074,
+  "<extra_id_26>": 32073,
+  "<extra_id_27>": 32072,
+  "<extra_id_28>": 32071,
+  "<extra_id_29>": 32070,
+  "<extra_id_2>": 32097,
+  "<extra_id_30>": 32069,
+  "<extra_id_31>": 32068,
+  "<extra_id_32>": 32067,
+  "<extra_id_33>": 32066,
+  "<extra_id_34>": 32065,
+  "<extra_id_35>": 32064,
+  "<extra_id_36>": 32063,
+  "<extra_id_37>": 32062,
+  "<extra_id_38>": 32061,
+  "<extra_id_39>": 32060,
+  "<extra_id_3>": 32096,
+  "<extra_id_40>": 32059,
+  "<extra_id_41>": 32058,
+  "<extra_id_42>": 32057,
+  "<extra_id_43>": 32056,
+  "<extra_id_44>": 32055,
+  "<extra_id_45>": 32054,
+  "<extra_id_46>": 32053,
+  "<extra_id_47>": 32052,
+  "<extra_id_48>": 32051,
+  "<extra_id_49>": 32050,
+  "<extra_id_4>": 32095,
+  "<extra_id_50>": 32049,
+  "<extra_id_51>": 32048,
+  "<extra_id_52>": 32047,
+  "<extra_id_53>": 32046,
+  "<extra_id_54>": 32045,
+  "<extra_id_55>": 32044,
+  "<extra_id_56>": 32043,
+  "<extra_id_57>": 32042,
+  "<extra_id_58>": 32041,
+  "<extra_id_59>": 32040,
+  "<extra_id_5>": 32094,
+  "<extra_id_60>": 32039,
+  "<extra_id_61>": 32038,
+  "<extra_id_62>": 32037,
+  "<extra_id_63>": 32036,
+  "<extra_id_64>": 32035,
+  "<extra_id_65>": 32034,
+  "<extra_id_66>": 32033,
+  "<extra_id_67>": 32032,
+  "<extra_id_68>": 32031,
+  "<extra_id_69>": 32030,
+  "<extra_id_6>": 32093,
+  "<extra_id_70>": 32029,
+  "<extra_id_71>": 32028,
+  "<extra_id_72>": 32027,
+  "<extra_id_73>": 32026,
+  "<extra_id_74>": 32025,
+  "<extra_id_75>": 32024,
+  "<extra_id_76>": 32023,
+  "<extra_id_77>": 32022,
+  "<extra_id_78>": 32021,
+  "<extra_id_79>": 32020,
+  "<extra_id_7>": 32092,
+  "<extra_id_80>": 32019,
+  "<extra_id_81>": 32018,
+  "<extra_id_82>": 32017,
+  "<extra_id_83>": 32016,
+  "<extra_id_84>": 32015,
+  "<extra_id_85>": 32014,
+  "<extra_id_86>": 32013,
+  "<extra_id_87>": 32012,
+  "<extra_id_88>": 32011,
+  "<extra_id_89>": 32010,
+  "<extra_id_8>": 32091,
+  "<extra_id_90>": 32009,
+  "<extra_id_91>": 32008,
+  "<extra_id_92>": 32007,
+  "<extra_id_93>": 32006,
+  "<extra_id_94>": 32005,
+  "<extra_id_95>": 32004,
+  "<extra_id_96>": 32003,
+  "<extra_id_97>": 32002,
+  "<extra_id_98>": 32001,
+  "<extra_id_99>": 32000,
+  "<extra_id_9>": 32090
+}

en/config.json

@@ -0,0 +1,60 @@
+{
+  "architectures": [
+    "T5ForConditionalGeneration"
+  ],
+  "classifier_dropout": 0.0,
+  "d_ff": 2048,
+  "d_kv": 64,
+  "d_model": 512,
+  "decoder_start_token_id": 0,
+  "dense_act_fn": "relu",
+  "dropout_rate": 0.1,
+  "dtype": "float32",
+  "eos_token_id": 1,
+  "feed_forward_proj": "relu",
+  "initializer_factor": 1.0,
+  "is_encoder_decoder": true,
+  "is_gated_act": false,
+  "layer_norm_epsilon": 1e-06,
+  "model_type": "t5",
+  "n_positions": 512,
+  "num_decoder_layers": 6,
+  "num_heads": 8,
+  "num_layers": 6,
+  "output_past": true,
+  "pad_token_id": 0,
+  "relative_attention_max_distance": 128,
+  "relative_attention_num_buckets": 32,
+  "task_specific_params": {
+    "summarization": {
+      "early_stopping": true,
+      "length_penalty": 2.0,
+      "max_length": 200,
+      "min_length": 30,
+      "no_repeat_ngram_size": 3,
+      "num_beams": 4,
+      "prefix": "summarize: "
+    },
+    "translation_en_to_de": {
+      "early_stopping": true,
+      "max_length": 300,
+      "num_beams": 4,
+      "prefix": "translate English to German: "
+    },
+    "translation_en_to_fr": {
+      "early_stopping": true,
+      "max_length": 300,
+      "num_beams": 4,
+      "prefix": "translate English to French: "
+    },
+    "translation_en_to_ro": {
+      "early_stopping": true,
+      "max_length": 300,
+      "num_beams": 4,
+      "prefix": "translate English to Romanian: "
+    }
+  },
+  "transformers_version": "4.57.6",
+  "use_cache": true,
+  "vocab_size": 32128
+}

en/generation_config.json

@@ -0,0 +1,7 @@
+{
+  "_from_model_config": true,
+  "decoder_start_token_id": 0,
+  "eos_token_id": 1,
+  "pad_token_id": 0,
+  "transformers_version": "4.57.6"
+}

en/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:13fd26d16fd54619d6b2a3c3e5061451fa9bf5167312f9c47aa074df1c5e80e6
+size 242041896

en/special_tokens_map.json

@@ -0,0 +1,125 @@
+{
+  "additional_special_tokens": [
+    "<extra_id_0>",
+    "<extra_id_1>",
+    "<extra_id_2>",
+    "<extra_id_3>",
+    "<extra_id_4>",
+    "<extra_id_5>",
+    "<extra_id_6>",
+    "<extra_id_7>",
+    "<extra_id_8>",
+    "<extra_id_9>",
+    "<extra_id_10>",
+    "<extra_id_11>",
+    "<extra_id_12>",
+    "<extra_id_13>",
+    "<extra_id_14>",
+    "<extra_id_15>",
+    "<extra_id_16>",
+    "<extra_id_17>",
+    "<extra_id_18>",
+    "<extra_id_19>",
+    "<extra_id_20>",
+    "<extra_id_21>",
+    "<extra_id_22>",
+    "<extra_id_23>",
+    "<extra_id_24>",
+    "<extra_id_25>",
+    "<extra_id_26>",
+    "<extra_id_27>",
+    "<extra_id_28>",
+    "<extra_id_29>",
+    "<extra_id_30>",
+    "<extra_id_31>",
+    "<extra_id_32>",
+    "<extra_id_33>",
+    "<extra_id_34>",
+    "<extra_id_35>",
+    "<extra_id_36>",
+    "<extra_id_37>",
+    "<extra_id_38>",
+    "<extra_id_39>",
+    "<extra_id_40>",
+    "<extra_id_41>",
+    "<extra_id_42>",
+    "<extra_id_43>",
+    "<extra_id_44>",
+    "<extra_id_45>",
+    "<extra_id_46>",
+    "<extra_id_47>",
+    "<extra_id_48>",
+    "<extra_id_49>",
+    "<extra_id_50>",
+    "<extra_id_51>",
+    "<extra_id_52>",
+    "<extra_id_53>",
+    "<extra_id_54>",
+    "<extra_id_55>",
+    "<extra_id_56>",
+    "<extra_id_57>",
+    "<extra_id_58>",
+    "<extra_id_59>",
+    "<extra_id_60>",
+    "<extra_id_61>",
+    "<extra_id_62>",
+    "<extra_id_63>",
+    "<extra_id_64>",
+    "<extra_id_65>",
+    "<extra_id_66>",
+    "<extra_id_67>",
+    "<extra_id_68>",
+    "<extra_id_69>",
+    "<extra_id_70>",
+    "<extra_id_71>",
+    "<extra_id_72>",
+    "<extra_id_73>",
+    "<extra_id_74>",
+    "<extra_id_75>",
+    "<extra_id_76>",
+    "<extra_id_77>",
+    "<extra_id_78>",
+    "<extra_id_79>",
+    "<extra_id_80>",
+    "<extra_id_81>",
+    "<extra_id_82>",
+    "<extra_id_83>",
+    "<extra_id_84>",
+    "<extra_id_85>",
+    "<extra_id_86>",
+    "<extra_id_87>",
+    "<extra_id_88>",
+    "<extra_id_89>",
+    "<extra_id_90>",
+    "<extra_id_91>",
+    "<extra_id_92>",
+    "<extra_id_93>",
+    "<extra_id_94>",
+    "<extra_id_95>",
+    "<extra_id_96>",
+    "<extra_id_97>",
+    "<extra_id_98>",
+    "<extra_id_99>"
+  ],
+  "eos_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<pad>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

en/spiece.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d60acb128cf7b7f2536e8f38a5b18a05535c9e14c7a355904270e15b0945ea86
+size 791656

en/tokenizer_config.json

@@ -0,0 +1,941 @@
+{
+  "add_prefix_space": true,
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<pad>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "</s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "<unk>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32000": {
+      "content": "<extra_id_99>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32001": {
+      "content": "<extra_id_98>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32002": {
+      "content": "<extra_id_97>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32003": {
+      "content": "<extra_id_96>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32004": {
+      "content": "<extra_id_95>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32005": {
+      "content": "<extra_id_94>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32006": {
+      "content": "<extra_id_93>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32007": {
+      "content": "<extra_id_92>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32008": {
+      "content": "<extra_id_91>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32009": {
+      "content": "<extra_id_90>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32010": {
+      "content": "<extra_id_89>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32011": {
+      "content": "<extra_id_88>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32012": {
+      "content": "<extra_id_87>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32013": {
+      "content": "<extra_id_86>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32014": {
+      "content": "<extra_id_85>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32015": {
+      "content": "<extra_id_84>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32016": {
+      "content": "<extra_id_83>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32017": {
+      "content": "<extra_id_82>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32018": {
+      "content": "<extra_id_81>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32019": {
+      "content": "<extra_id_80>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32020": {
+      "content": "<extra_id_79>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32021": {
+      "content": "<extra_id_78>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32022": {
+      "content": "<extra_id_77>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32023": {
+      "content": "<extra_id_76>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32024": {
+      "content": "<extra_id_75>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32025": {
+      "content": "<extra_id_74>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32026": {
+      "content": "<extra_id_73>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32027": {
+      "content": "<extra_id_72>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32028": {
+      "content": "<extra_id_71>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32029": {
+      "content": "<extra_id_70>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32030": {
+      "content": "<extra_id_69>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32031": {
+      "content": "<extra_id_68>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32032": {
+      "content": "<extra_id_67>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32033": {
+      "content": "<extra_id_66>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32034": {
+      "content": "<extra_id_65>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32035": {
+      "content": "<extra_id_64>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32036": {
+      "content": "<extra_id_63>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32037": {
+      "content": "<extra_id_62>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32038": {
+      "content": "<extra_id_61>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32039": {
+      "content": "<extra_id_60>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32040": {
+      "content": "<extra_id_59>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32041": {
+      "content": "<extra_id_58>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32042": {
+      "content": "<extra_id_57>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32043": {
+      "content": "<extra_id_56>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32044": {
+      "content": "<extra_id_55>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32045": {
+      "content": "<extra_id_54>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32046": {
+      "content": "<extra_id_53>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32047": {
+      "content": "<extra_id_52>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32048": {
+      "content": "<extra_id_51>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32049": {
+      "content": "<extra_id_50>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32050": {
+      "content": "<extra_id_49>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32051": {
+      "content": "<extra_id_48>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32052": {
+      "content": "<extra_id_47>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32053": {
+      "content": "<extra_id_46>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32054": {
+      "content": "<extra_id_45>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32055": {
+      "content": "<extra_id_44>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32056": {
+      "content": "<extra_id_43>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32057": {
+      "content": "<extra_id_42>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32058": {
+      "content": "<extra_id_41>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32059": {
+      "content": "<extra_id_40>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32060": {
+      "content": "<extra_id_39>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32061": {
+      "content": "<extra_id_38>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32062": {
+      "content": "<extra_id_37>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32063": {
+      "content": "<extra_id_36>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32064": {
+      "content": "<extra_id_35>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32065": {
+      "content": "<extra_id_34>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32066": {
+      "content": "<extra_id_33>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32067": {
+      "content": "<extra_id_32>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32068": {
+      "content": "<extra_id_31>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32069": {
+      "content": "<extra_id_30>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32070": {
+      "content": "<extra_id_29>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32071": {
+      "content": "<extra_id_28>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32072": {
+      "content": "<extra_id_27>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32073": {
+      "content": "<extra_id_26>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32074": {
+      "content": "<extra_id_25>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32075": {
+      "content": "<extra_id_24>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32076": {
+      "content": "<extra_id_23>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32077": {
+      "content": "<extra_id_22>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32078": {
+      "content": "<extra_id_21>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32079": {
+      "content": "<extra_id_20>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32080": {
+      "content": "<extra_id_19>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32081": {
+      "content": "<extra_id_18>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32082": {
+      "content": "<extra_id_17>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32083": {
+      "content": "<extra_id_16>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32084": {
+      "content": "<extra_id_15>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32085": {
+      "content": "<extra_id_14>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32086": {
+      "content": "<extra_id_13>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32087": {
+      "content": "<extra_id_12>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32088": {
+      "content": "<extra_id_11>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32089": {
+      "content": "<extra_id_10>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32090": {
+      "content": "<extra_id_9>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32091": {
+      "content": "<extra_id_8>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32092": {
+      "content": "<extra_id_7>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32093": {
+      "content": "<extra_id_6>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32094": {
+      "content": "<extra_id_5>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32095": {
+      "content": "<extra_id_4>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32096": {
+      "content": "<extra_id_3>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32097": {
+      "content": "<extra_id_2>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32098": {
+      "content": "<extra_id_1>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32099": {
+      "content": "<extra_id_0>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "additional_special_tokens": [
+    "<extra_id_0>",
+    "<extra_id_1>",
+    "<extra_id_2>",
+    "<extra_id_3>",
+    "<extra_id_4>",
+    "<extra_id_5>",
+    "<extra_id_6>",
+    "<extra_id_7>",
+    "<extra_id_8>",
+    "<extra_id_9>",
+    "<extra_id_10>",
+    "<extra_id_11>",
+    "<extra_id_12>",
+    "<extra_id_13>",
+    "<extra_id_14>",
+    "<extra_id_15>",
+    "<extra_id_16>",
+    "<extra_id_17>",
+    "<extra_id_18>",
+    "<extra_id_19>",
+    "<extra_id_20>",
+    "<extra_id_21>",
+    "<extra_id_22>",
+    "<extra_id_23>",
+    "<extra_id_24>",
+    "<extra_id_25>",
+    "<extra_id_26>",
+    "<extra_id_27>",
+    "<extra_id_28>",
+    "<extra_id_29>",
+    "<extra_id_30>",
+    "<extra_id_31>",
+    "<extra_id_32>",
+    "<extra_id_33>",
+    "<extra_id_34>",
+    "<extra_id_35>",
+    "<extra_id_36>",
+    "<extra_id_37>",
+    "<extra_id_38>",
+    "<extra_id_39>",
+    "<extra_id_40>",
+    "<extra_id_41>",
+    "<extra_id_42>",
+    "<extra_id_43>",
+    "<extra_id_44>",
+    "<extra_id_45>",
+    "<extra_id_46>",
+    "<extra_id_47>",
+    "<extra_id_48>",
+    "<extra_id_49>",
+    "<extra_id_50>",
+    "<extra_id_51>",
+    "<extra_id_52>",
+    "<extra_id_53>",
+    "<extra_id_54>",
+    "<extra_id_55>",
+    "<extra_id_56>",
+    "<extra_id_57>",
+    "<extra_id_58>",
+    "<extra_id_59>",
+    "<extra_id_60>",
+    "<extra_id_61>",
+    "<extra_id_62>",
+    "<extra_id_63>",
+    "<extra_id_64>",
+    "<extra_id_65>",
+    "<extra_id_66>",
+    "<extra_id_67>",
+    "<extra_id_68>",
+    "<extra_id_69>",
+    "<extra_id_70>",
+    "<extra_id_71>",
+    "<extra_id_72>",
+    "<extra_id_73>",
+    "<extra_id_74>",
+    "<extra_id_75>",
+    "<extra_id_76>",
+    "<extra_id_77>",
+    "<extra_id_78>",
+    "<extra_id_79>",
+    "<extra_id_80>",
+    "<extra_id_81>",
+    "<extra_id_82>",
+    "<extra_id_83>",
+    "<extra_id_84>",
+    "<extra_id_85>",
+    "<extra_id_86>",
+    "<extra_id_87>",
+    "<extra_id_88>",
+    "<extra_id_89>",
+    "<extra_id_90>",
+    "<extra_id_91>",
+    "<extra_id_92>",
+    "<extra_id_93>",
+    "<extra_id_94>",
+    "<extra_id_95>",
+    "<extra_id_96>",
+    "<extra_id_97>",
+    "<extra_id_98>",
+    "<extra_id_99>"
+  ],
+  "clean_up_tokenization_spaces": true,
+  "eos_token": "</s>",
+  "extra_ids": 100,
+  "extra_special_tokens": {},
+  "legacy": true,
+  "model_max_length": 512,
+  "pad_token": "<pad>",
+  "sp_model_kwargs": {},
+  "tokenizer_class": "T5Tokenizer",
+  "unk_token": "<unk>"
+}

humanize.py

@@ -0,0 +1,224 @@
+"""humanize-text-model inference wrapper (bilingual: English + Chinese).
+
+Loads the Lynote bilingual humanizer — two small seq2seq checkpoints in one
+repo (``en/`` T5-small, ``zh/`` Chinese T5-small) — and routes each input to
+the right language model. URLs, numbers, paths, code and quoted strings are
+protected with tokenizer-friendly placeholders before generation and restored
+afterwards.
+
+Usage:
+    from humanize import Humanizer
+    h = Humanizer()  # loads "Danny-Lynote/humanize-text-model" from the Hub
+    print(h("It is important to note that this robust solution serves as a testament to our commitment."))
+"""
+
+import argparse
+import os
+import re
+import sys
+from typing import List, Optional
+
+import torch
+from transformers import BertTokenizer, T5ForConditionalGeneration, T5Tokenizer
+
+DEFAULT_MODEL = "Danny-Lynote/humanize-text-model"
+MAX_CHARS = 4_000
+MAX_TOKENS = 256
+DEVICE = "mps" if torch.backends.mps.is_available() else (
+    "cuda" if torch.cuda.is_available() else "cpu"
+)
+
+PROTECTED_PATTERN = re.compile(
+    r"(`[^`]+`|https?://[^\s\u4e00-\u9fff，。！？；：、]+|(?:[A-Za-z]:)?(?:[/\\][\w.\-]+)+|"
+    r"\b\d+(?:\.\d+)?(?:%|[A-Za-z]+)?\b|\"[^\"]*\"|'[^']*')"
+)
+CJK_RATIO = re.compile(r"[\u4e00-\u9fff]")
+
+
+def protect(text: str, language: Optional[str] = None):
+    """Replace protected spans with language-aware placeholders:
+    EN ``PROTECTED_n``, ZH ``【保护n】`` (both tokenizer-friendly)."""
+    protected = []
+    zh = language == "zh" or (language is None and is_chinese(text))
+
+    def replace(match):
+        token = f"【保护{len(protected)}】" if zh else f"PROTECTED_{len(protected)}"
+        protected.append(match.group(0))
+        return token
+
+    return PROTECTED_PATTERN.sub(replace, text), protected
+
+
+def restore(text: str, protected: List[str]) -> str:
+    """Restore placeholders (either format, tolerant of inserted spaces)."""
+    for index, value in enumerate(protected):
+        text = re.sub(rf"PROTECTED_{index}\b", lambda m: value, text)
+        text = re.sub(
+            rf"【\s*保\s*护\s*{index}\s*】", lambda m: value, text
+        )
+    return text
+
+
+def is_chinese(text: str, threshold: float = 0.05) -> bool:
+    total = len(re.sub(r"\s+", "", text))
+    if total == 0:
+        return False
+    return len(CJK_RATIO.findall(text)) / total > threshold
+
+
+class Humanizer:
+    def __init__(
+        self,
+        model_id: str = DEFAULT_MODEL,
+        device: Optional[str] = None,
+        cache_dir: Optional[str] = None,
+        local_files_only: bool = False,
+        hf_token: Optional[str] = None,
+        lazy: bool = True,
+    ):
+        """model_id is a repo (or local dir) containing ``en/`` and ``zh/``
+        subdirectories with separate checkpoints. With ``lazy=True`` (default)
+        a language model is only loaded when first needed."""
+        self.model_id = model_id
+        self.device = torch.device(device or DEVICE)
+        self._models = {}
+        self._tokenizers = {}
+        self._cache_dir = cache_dir
+        self._local_only = local_files_only
+        self._token = hf_token
+        if not lazy:
+            self._get("en")
+            self._get("zh")
+
+    def _load(self, language: str, model: T5ForConditionalGeneration, tokenizer: T5Tokenizer):
+        self._models[language] = model
+        self._tokenizers[language] = tokenizer
+
+    def _get(self, language: str):
+        if language not in self._models:
+            sub = f"{self.model_id}/{language}"
+            model = T5ForConditionalGeneration.from_pretrained(
+                sub, cache_dir=self._cache_dir,
+                local_files_only=self._local_only, token=self._token,
+            )
+            tokenizer_cls = BertTokenizer if language == "zh" else T5Tokenizer
+            tokenizer = tokenizer_cls.from_pretrained(
+                sub, cache_dir=self._cache_dir,
+                local_files_only=self._local_only, token=self._token,
+            )
+            model.to(self.device).eval()
+            self._load(language, model, tokenizer)
+        return self._models[language], self._tokenizers[language]
+
+    def humanize(self, text: str, num_beams: int = 3, do_sample: bool = False) -> str:
+        text = (text or "").strip()
+        if not text:
+            raise ValueError("Input text is empty.")
+        if len(text) > MAX_CHARS:
+            raise ValueError(f"Input exceeds {MAX_CHARS:,} characters.")
+        return self.humanize_batch([text], num_beams=num_beams, do_sample=do_sample)[0]
+
+    def humanize_batch(
+        self,
+        texts: List[str],
+        num_beams: int = 3,
+        do_sample: bool = False,
+        batch_size: int = 8,
+    ) -> List[str]:
+        """Humanize a list of texts. URLs, numbers, paths, code and quotes are
+        protected per-text with placeholders and restored after generation."""
+        masked = []
+        protected_all = []
+        for t in texts:
+            masked_text, protected = protect(t)
+            masked.append(masked_text)
+            protected_all.append(protected)
+        outputs: List[str] = []
+        grouped: dict = {}
+        for i, t in enumerate(masked):
+            grouped.setdefault("zh" if is_chinese(t) else "en", []).append(i)
+        for language, indices in grouped.items():
+            model, tokenizer = self._get(language)
+            for start in range(0, len(indices), batch_size):
+                chunk_idx = indices[start : start + batch_size]
+                enc = tokenizer(
+                    [masked[i] for i in chunk_idx],
+                    max_length=MAX_TOKENS,
+                    truncation=True,
+                    padding=True,
+                    return_tensors="pt",
+                )
+                if "token_type_ids" in enc:
+                    enc.pop("token_type_ids")
+                enc = enc.to(self.device)
+                with torch.no_grad():
+                    gen_kwargs = dict(
+                        num_beams=num_beams,
+                        do_sample=do_sample,
+                        early_stopping=True,
+                    )
+                    if language == "zh":
+                        # Small Chinese T5 degenerates on long sequences: cap
+                        # length and block exact 4-gram repetition (EN does not
+                        # need these).
+                        gen_kwargs["max_length"] = 40
+                        gen_kwargs["no_repeat_ngram_size"] = 4
+                    else:
+                        gen_kwargs["max_length"] = MAX_TOKENS
+                    gen = model.generate(**enc, **gen_kwargs)
+                decoded = tokenizer.batch_decode(gen, skip_special_tokens=True)
+                for j, text in zip(chunk_idx, decoded):
+                    outputs.append((j, text))
+        outputs.sort(key=lambda x: x[0])
+
+        return [
+            restore(self._polish(out), protected)
+            for out, protected in zip([o[1] for o in outputs], protected_all)
+        ]
+
+    @staticmethod
+    def _polish(text: str) -> str:
+        text = re.sub(r"^\s*(?:extra\d+\s*[,，。\s]*)+", "", text)
+        text = re.sub(r"(?<=[\u4e00-\u9fff])\s+(?=[\u4e00-\u9fff，。；：！？])", "", text)
+        text = re.sub(r"(?<=[，。；：！？])\s+(?=[\u4e00-\u9fff])", "", text)
+        text = re.sub(r"[ \t]{2,}", " ", text)
+        text = re.sub(r"\s+([,.;:!?，。；：！？])", r"\1", text)
+        text = re.sub(r"[,;]\s*\.", ".", text)
+        text = re.sub(r"\.\s*[,;]", ".", text)
+        text = re.sub(r"[，,]\s*(?=[。！？.!?])", "", text)
+        text = re.sub(r"\.\s+([a-z])", lambda m: ". " + m.group(1).upper(), text)
+        return text.strip(" ，")
+
+
+def main():
+    parser = argparse.ArgumentParser(description="Bilingual text humanizer (T5-small EN + Chinese T5-small)")
+    parser.add_argument("--text", help="text to humanize")
+    parser.add_argument("--input", help="path to a text file")
+    parser.add_argument("--output", help="write result to a file")
+    parser.add_argument("--model", default=DEFAULT_MODEL, help="model id or local dir")
+    parser.add_argument("--beams", type=int, default=3)
+    parser.add_argument("--list-models", action="store_true", help="print model ids and exit")
+    args = parser.parse_args()
+
+    if args.list_models:
+        print(DEFAULT_MODEL)
+        return
+
+    if args.text and args.input:
+        parser.error("provide only one of --text / --input")
+    source = args.text or (open(args.input, encoding="utf-8").read() if args.input else None)
+    if not source:
+        parser.error("provide --text or --input")
+
+    humanizer = Humanizer(model_id=args.model)
+    result = humanizer.humanize(source, num_beams=args.beams)
+    if args.output:
+        with open(args.output, "w", encoding="utf-8") as f:
+            f.write(result)
+        print(f"written to {args.output}")
+    else:
+        print(result)
+
+
+if __name__ == "__main__":
+    main()

requirements.txt

@@ -0,0 +1,4 @@
+transformers>=4.46
+torch>=2.2
+sacrebleu>=2.4
+pytest>=8.0

scripts/build_dataset.py

@@ -0,0 +1,335 @@
+"""Build the synthetic bilingual training corpus for the humanize model.
+
+Methodology
+-----------
+Supervision comes from the conservative rule engine in ``src/rules.py``
+(derived from the lynote-ai reference projects). We generate:
+
+1. **AI -> human**: formulaic clause combinations rewritten by the rules,
+2. **Human -> human (identity)**: clean prose unchanged, so the model learns
+   not to rewrite already-human text,
+3. **Mixed**: clean prose with one injected AI cliché that must be removed,
+4. **Protected spans**: examples containing numbers, URLs and quoted text.
+
+Every input maps to exactly one deterministic target (one-to-one), which
+keeps the training signal consistent. Outputs are seeded JSONL files with
+fields ``input_text``, ``output_text``, ``lang``.
+"""
+
+import argparse
+import json
+import random
+import re
+import sys
+from pathlib import Path
+
+sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
+from src.rules import EN_PHRASES, ZH_PHRASES, humanize_rules, protect
+
+# --------------------------------------------------------------------------
+# Seed prose (curated, human-flavored).
+# --------------------------------------------------------------------------
+
+EN_SEEDS = [
+    "I walked the dog this morning and the sky was grey and low.",
+    "Halfway through the park it started raining, which I hadn't expected, so we ran home.",
+    "The dog seemed to enjoy it more than me.",
+    "When we got back I made coffee and watched the rain run down the window.",
+    "Thanks for sending the report over, the numbers look good.",
+    "Could you double-check the retention section before we share it?",
+    "Let's grab coffee this week and talk through next steps.",
+    "The conference was exhausting, but a couple of the sessions were genuinely useful.",
+    "I came home with a notebook full of scribbles and a dozen open browser tabs.",
+    "The sky cleared by evening and we went for a long walk.",
+    "I've been testing the new tool for a few days and it mostly works.",
+    "The docs are clear enough, though the install steps could be shorter.",
+    "We tried the new approach on a small project and it held up fine.",
+    "My laptop died in the middle of the meeting, which was awkward.",
+    "She asked me to review her draft and I found a few typos.",
+    "The traffic was bad this morning, so I took the subway instead.",
+    "We spent the afternoon fixing bugs and ordering pizza.",
+    "I think the plan works, but we should test it with real users first.",
+    "The garden looks great after the rain.",
+    "He replied faster than I expected, which was a nice surprise.",
+    "The new hire seems sharp and asks good questions.",
+    "I read the whole chapter on the train and highlighted half of it.",
+    "We moved the deadline to Friday so nobody has to work the weekend.",
+    "The cafe near the office makes a decent flat white.",
+    "My phone died, so I missed your call, sorry about that.",
+    "She asked me to review her draft and I found a few typos. It was a nice read.",
+    "I think the plan works, but we should test it with real users first. We can start next week.",
+]
+
+ZH_SEEDS = [
+    "今天早上我带狗去公园，出门时天阴阴的。",
+    "走到一半忽然下起雨，我们只好跑回家，狗倒是挺高兴。",
+    "回来以后我煮了杯咖啡，坐在窗边看了会儿雨。",
+    "你要的资料我发你邮箱了，附件里有两个版本。",
+    "你看一下哪个合适，有问题随时叫我。",
+    "下午我跟老王聊了聊，他觉得可以，但预算要再压一压。",
+    "今天加班到九点，回来路上吃了碗牛肉面。",
+    "老板认得我，多给了几片牛肉。",
+    "猫蹲在门口等我，尾巴翘得老高。",
+    "周末去了趟公园，人特别多，树荫下倒是凉快。",
+    "家里的路由器坏了，折腾了一晚上也没修好。",
+    "这个月房租涨了两百块，心里有点不爽。",
+    "我在楼下便利店买了瓶水，顺便取了个快递。",
+    "下周开会的内容我还没准备完，今晚得加个班。",
+    "邻居家的狗最近晚上老叫，吵得人睡不着。",
+    "上次说的那个方案，客户反馈还不错，就是工期太紧。",
+    "我今天试着做了道红烧肉，味道还行，就是有点咸。",
+    "地铁口新开了一家奶茶店，排队的人特别多。",
+    "天气预报说明天降温，出门记得多穿点。",
+    "我把旧手机的照片都备份到电脑上了。",
+    "新来的同事挺靠谱的，交办的事情都能按时完成。",
+    "超市打折的时候我囤了不少纸巾和洗衣液。",
+    "这个项目其实不难，主要是沟通起来比较费劲。",
+    "楼下的早餐店换了老板，油条没有以前好吃了。",
+    "我在网上买了个书架，装了一个小时才装好。",
+    "妈妈打电话来说家里的桂花开了，让我回去看看。",
+    "这个周末打算去爬山，好久没运动了。",
+    "我把会议纪要整理��发到群里了，大家记得看。",
+    "打印机又卡纸了，修了半小时才弄好。",
+    "晚上的风挺大的，窗户被吹得呼呼响。",
+    "上个月的项目结项了，团队一起吃了顿饭庆祝。",
+    "小区里的银杏树黄了，特别好看。",
+    "我最近在学做菜，最拿手的是西红柿炒蛋。",
+    "公交车晚点了二十分钟，差点迟到。",
+    "这个月业绩不错，老板说要给大家发奖金。",
+    "我把书还回图书馆了，顺便借了两本新的。",
+    "朋友推荐了一家火锅店，说是味道特别好。",
+    "早上起晚了，来不及吃早饭，现在有点饿。",
+    "新买的耳机用着不错，降噪效果明显。",
+    "我把房间收拾了一遍，看着舒服多了。",
+]
+
+# --------------------------------------------------------------------------
+# Formulaic clause pools. Combinations of these produce many unique inputs;
+# the rule engine maps each one to a deterministic human-flavored target.
+# --------------------------------------------------------------------------
+
+EN_SUBJECTS = [
+    "artificial intelligence", "digital transformation", "cloud computing",
+    "data analytics", "the new platform", "our organization", "this initiative",
+    "remote work", "machine learning", "automation", "the latest release",
+    "the training program", "our partnership", "the revised policy",
+]
+
+EN_CLAUSES = [
+    "it is important to note that {s} plays a crucial role in business success",
+    "it is worth noting that leveraging {s} serves as a testament to our commitment to innovation",
+    "our comprehensive approach to {s} fosters collaboration and drives growth",
+    "the seamless integration of {s} enhances efficiency and delivers a robust solution",
+    "embracing {s} is no longer optional but essential for long-term competitiveness",
+    "{s} requires high-quality data to function effectively",
+    "our groundbreaking platform leverages cutting-edge technology to provide a seamless experience",
+    "the state-of-the-art solution plays a crucial role in improving operational efficiency",
+    "it is essential to delve into {s} in order to remain competitive",
+    "fostering a holistic approach to {s} is paramount",
+    "{s} transforms the way we work in a significant way",
+    "this game-changing innovation will revolutionize the industry",
+]
+
+EN_CONNECTORS = ["Moreover,", "Furthermore,", "Additionally,", "In conclusion,"]
+
+ZH_SUBJECTS = [
+    "数字化转型", "人工智能", "这套方案", "新的平台", "我们的团队",
+    "在线办公", "自动化流程", "数据治理", "这次改革", "人才培养",
+]
+
+ZH_CLAUSES = [
+    "值得注意的是，{s}发挥着重要作用",
+    "在当今快速发展的时代，{s}是企业发展的重要保障",
+    "综上所述，我们通过赋能团队来助力企业实现降本增效",
+    "这一方案形成了完整的业务闭环，无缝连接各个环节",
+    "由此可见，{s}已经成为企业发展的必然趋势",
+    "总而言之，企业应当积极拥抱新技术，从而在激烈的竞争中保持优势",
+    "{s}对于企业来说至关重要",
+    "我们需要不断提升效率，持续优化流程",
+    "这一举措实现了闭环管理，进一步提升了整体效率",
+    "数字化转型为{s}提供了新的发展机遇",
+    "相关部门应该加快制定实施细则，确保各项措施落地见效",
+    "这一政策有助于优化资源配置，推动产业转型升级",
+    "我们要坚持以用户为中心，不断完善产品服务体系",
+    "项目团队克服了诸多困难，最终按期完成了交付任务",
+    "该平台整合了线上线下资源，覆盖了多个业务场景",
+    "通过数据驱动的方式，我们能够更好地洞察市场变化",
+    "本次会议明确了下一阶段的工作重点和发展方向",
+    "这套体系有效降低了运营成本，提高了响应速度",
+    "从长远来看，{s}将深刻改变行业的竞争格局",
+    "各方应加强沟通协作，共同推进相关工作的开展",
+]
+
+PROTECTED_SEEDS = [
+    "The license key is ABC-12345 and the server is at https://api.lynote.ai/v1/detect.",
+    "The report shows a 23.5% increase in Q3, up from 18.2% last year.",
+    "Run `pip install humanize-text==1.4.2` and check /usr/local/bin/humanize.",
+    "The budget is $12,400 for this quarter, as noted in \"Project Lynote\".",
+    "The API accepts 500 requests per minute at https://hub.example.com and returns a 99.9% success rate.",
+    "The server rebooted at 03:47 and restored 1.2 TB of data within 10 minutes.",
+    "版本号是 v2.1.3，接口地址是 https://api.lynote.ai，成本下降了 15%。",
+    "配置文件放在 /etc/lynote/config.toml，端口是 8000，超时 30 秒。",
+    "上个月的收入是 85000 元，比预期多了 12%，主要来自 https://lynote.ai。",
+    "服务器在 03:47 重启，恢复了 1.2 TB 的数据，耗时 10 分钟。",
+]
+
+
+def _ai_en(rng: random.Random, n: int) -> list[str]:
+    out = []
+    subjects = EN_SUBJECTS
+    while len(out) < n:
+        s = rng.choice(subjects)
+        k = rng.randint(2, 4)
+        clauses = [c.format(s=s) for c in rng.sample(EN_CLAUSES, k)]
+        clauses[0] = clauses[0][0].upper() + clauses[0][1:]
+        text = clauses[0]
+        for c in clauses[1:]:
+            text += ". " + rng.choice(EN_CONNECTORS) + " " + c
+        text += "."
+        if text not in out:
+            out.append(text)
+    return out
+
+
+def _ai_zh(rng: random.Random, n: int) -> list[str]:
+    out = []
+    while len(out) < n:
+        s = rng.choice(ZH_SUBJECTS)
+        k = rng.randint(2, 4)
+        clauses = [c.format(s=s) for c in rng.sample(ZH_CLAUSES, k)]
+        text = "，".join(clauses[:-1]) + "，" + clauses[-1] + "。" if k > 1 else clauses[0] + "。"
+        if text not in out:
+            out.append(text)
+    return out
+
+
+def _inject_cliche_en(seed: str) -> str:
+    phrase = random.choice(list(EN_PHRASES.keys()))
+    seed = seed.rstrip(".")
+    return f"{phrase}, {seed}."
+
+
+def _inject_cliche_zh(seed: str) -> str:
+    phrase = random.choice(list(ZH_PHRASES.keys()))
+    return f"{phrase}，{seed}"
+
+
+def build(
+    seed: int = 42,
+    n_en_ai: int = 8_000,
+    n_zh_ai: int = 6_000,
+    n_mixed: int = 1_000,
+    n_protected: int = 300,
+) -> list[dict]:
+    rng = random.Random(seed)
+    pairs: list[dict] = []
+
+    # 1. AI -> human.
+    for lang, ai_texts, _seeds in (
+        ("en", _ai_en(rng, n_en_ai), EN_SEEDS),
+        ("zh", _ai_zh(rng, n_zh_ai), ZH_SEEDS),
+    ):
+        for ai in ai_texts:
+            human = humanize_rules(ai, lang)
+            if human == ai:
+                continue
+            pairs.append({"input_text": ai, "output_text": human, "lang": lang})
+
+    # 2. Identity pairs.
+    for lang, seeds in (("en", EN_SEEDS), ("zh", ZH_SEEDS)):
+        for s in seeds:
+            pairs.append({"input_text": s, "output_text": s, "lang": lang})
+
+    # 3. Mixed: clean prose + injected cliche -> cleaned prose.
+    for _ in range(n_mixed):
+        lang = rng.choice(["en", "zh"])
+        seed = rng.choice(EN_SEEDS if lang == "en" else ZH_SEEDS)
+        noisy = _inject_cliche_en(seed) if lang == "en" else _inject_cliche_zh(seed)
+        cleaned = humanize_rules(noisy, lang)
+        if cleaned != noisy:
+            pairs.append({"input_text": noisy, "output_text": cleaned, "lang": lang})
+
+    # 3b. Extra Chinese mixed pairs (identity/copy grounding).
+    for _ in range(n_mixed):
+        seed = rng.choice(ZH_SEEDS)
+        noisy = _inject_cliche_zh(seed)
+        cleaned = humanize_rules(noisy, "zh")
+        if cleaned != noisy:
+            pairs.append({"input_text": noisy, "output_text": cleaned, "lang": "zh"})
+
+    # 4. Protected spans (contextualized so dedup keeps variety).
+    en_prefixes = [
+        "It is important to note that ", "It is worth noting that ",
+        "Moreover, ", "Additionally, ", "In conclusion, ", "",
+    ]
+    en_suffixes = [
+        " This is a robust and seamless solution.",
+        " Overall, this approach is a game-changer.",
+        " In conclusion, this matters for our work.",
+        " Furthermore, this initiative drives growth.",
+        "",
+    ]
+    zh_prefixes = ["值得注意的是，", "由此可见，", "综上所述，", "在当今快速发展的时代，", ""]
+    zh_suffixes = ["这一方案发挥了重要作用。", "这就是完整的业务闭环。", "综上所述，我们需要持续优化。", ""]
+    protected_srcs = []
+    for seed in PROTECTED_SEEDS:
+        if re.search(r"[\u4e00-\u9fff]", seed):
+            protected_srcs.extend(
+                f"{p}{seed}{s}" for p in zh_prefixes for s in zh_suffixes
+            )
+        else:
+            protected_srcs.extend(
+                f"{p}{seed}{s}" for p in en_prefixes for s in en_suffixes
+            )
+    for src in protected_srcs:
+        lang = "zh" if re.search(r"[\u4e00-\u9fff]", src) else "en"
+        cleaned = humanize_rules(src, lang)
+        if cleaned != src:
+            # Raw pair (URLs/numbers visible).
+            pairs.append({"input_text": src, "output_text": cleaned, "lang": lang})
+            # Masked pair: identical placeholder spans on both sides, matching
+            # inference-time protection so the model learns to copy them.
+            masked_src, _ = protect(src, lang)
+            masked_cleaned, _ = protect(cleaned, lang)
+            pairs.append(
+                {"input_text": masked_src, "output_text": masked_cleaned, "lang": lang}
+            )
+
+    seen, unique = set(), []
+    for p in pairs:
+        if p["input_text"] in seen:
+            continue
+        seen.add(p["input_text"])
+        unique.append(p)
+    return unique
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--out", default="data", help="output directory")
+    parser.add_argument("--seed", type=int, default=42)
+    parser.add_argument("--val-size", type=int, default=500)
+    parser.add_argument("--test-size", type=int, default=1_000)
+    args = parser.parse_args()
+
+    out = Path(args.out)
+    out.mkdir(parents=True, exist_ok=True)
+    pairs = build(seed=args.seed)
+    rng = random.Random(args.seed)
+    rng.shuffle(pairs)
+
+    test = pairs[: args.test_size]
+    val = pairs[args.test_size : args.test_size + args.val_size]
+    train = pairs[args.test_size + args.val_size :]
+
+    for split, items in (("train", train), ("val", val), ("test", test)):
+        with open(out / f"{split}.jsonl", "w", encoding="utf-8") as f:
+            for p in items:
+                f.write(json.dumps(p, ensure_ascii=False) + "\n")
+        print(f"{split}: {len(items)} pairs")
+
+    print("test langs:", sorted({p['lang'] for p in test}))
+
+
+if __name__ == "__main__":
+    main()

scripts/evaluate.py

@@ -0,0 +1,154 @@
+"""Benchmark the humanizer model.
+
+Measures:
+- corpus BLEU (sacrebleu) between outputs and rule references on AI->human
+  held-out rows (meaning preservation),
+- AI-cliché removal rate on AI-flavored inputs,
+- identity stability on ALL clean-prose (identity) pairs in the corpus,
+- protected-span preservation on ALL examples containing protected spans,
+- latency (characters per second, MPS/CPU).
+"""
+
+import argparse
+import json
+import re
+import sys
+import time
+from pathlib import Path
+
+sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
+
+from humanize import Humanizer, protect
+
+EN_CLICHES = [
+    "it is important to note that", "it is worth noting that", "moreover",
+    "furthermore", "in conclusion", "leverage", "seamless", "robust",
+    "testament", "delve", "cutting-edge", "state-of-the-art",
+]
+ZH_CLICHES = [
+    "值得注意的是", "综上所述", "赋能", "降本增效", "闭环", "无缝",
+    "由此可见", "总而言之",
+]
+
+
+def scan(data_dir: str):
+    rows = []
+    for split in ("train", "val", "test"):
+        with open(Path(data_dir) / f"{split}.jsonl", encoding="utf-8") as f:
+            for line in f:
+                rows.append(json.loads(line))
+    return rows
+
+
+def main():
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--model", default="checkpoints/humanize-text-model")
+    parser.add_argument("--data", default="data")
+    parser.add_argument("--ai-limit", type=int, default=500)
+    args = parser.parse_args()
+
+    from sacrebleu import corpus_bleu
+
+    humanizer = Humanizer(model_id=args.model)
+    rows = scan(args.data)
+
+    ai_rows = [r for r in rows if r["input_text"] != r["output_text"]][: args.ai_limit]
+    identity_rows = [r for r in rows if r["input_text"] == r["output_text"]]
+    protected_rows = [
+        r for r in rows
+        if re.search(r"PROTECTED_\d+", r["input_text"]) or re.search(
+            r"https?://|PROTECTED_\d+|\d", r["input_text"]
+        )
+    ]
+    protected_rows = [r for r in protected_rows if "PROTECTED_" in r["input_text"] or "https://" in r["input_text"]]
+
+    # Deduplicate by input so samples are unique.
+    def dedupe(items):
+        seen, out = set(), []
+        for r in items:
+            if r["input_text"] in seen:
+                continue
+            seen.add(r["input_text"])
+            out.append(r)
+        return out
+
+    ai_rows = dedupe(ai_rows)
+    identity_rows = dedupe(identity_rows)
+    protected_rows = dedupe(protected_rows)
+
+    print(f"AI->human rows: {len(ai_rows)} | identity rows: {len(identity_rows)} | protected rows: {len(protected_rows)}", flush=True)
+
+    # 1. AI -> human: BLEU + cliché removal.
+    refs, hyps = [], []
+    cliche_in = 0
+    cliche_removed = 0
+    t0 = time.time()
+    for n, row in enumerate(ai_rows, 1):
+        out = humanizer.humanize(row["input_text"], num_beams=3)
+        hyps.append(out)
+        refs.append(row["output_text"])
+        lower = out.lower()
+        hits_in = [c for c in (ZH_CLICHES if row["lang"] == "zh" else EN_CLICHES)
+                   if c in (row["input_text"] if row["lang"] == "zh" else row["input_text"].lower())]
+        hits_out = [c for c in (ZH_CLICHES if row["lang"] == "zh" else EN_CLICHES)
+                    if c in lower]
+        if hits_in:
+            cliche_in += 1
+            if not hits_out:
+                cliche_removed += 1
+        if n % 50 == 0:
+            print(f"  ...ai {n}/{len(ai_rows)}", flush=True)
+    bleu = corpus_bleu(hyps, [refs]).score
+
+    # 2. Identity stability.
+    identity_ok = 0
+    for n, row in enumerate(identity_rows, 1):
+        out = humanizer.humanize(row["input_text"], num_beams=3)
+        # Allow light edits, no big drift: use char-level edit ratio.
+        ratio = sum(1 for a, b in zip(out, row["input_text"]) if a != b) / max(len(row["input_text"]), 1)
+        if ratio <= 0.15:
+            identity_ok += 1
+        if n % 25 == 0:
+            print(f"  ...identity {n}/{len(identity_rows)}", flush=True)
+
+    # 3. Protected spans.
+    protected_ok = 0
+    protected_total = 0
+    failures = []
+    for n, row in enumerate(protected_rows, 1):
+        masked, spans = protect(row["input_text"])
+        out = humanizer.humanize(row["input_text"], num_beams=3)
+        for span in spans:
+            protected_total += 1
+            if span in out:
+                protected_ok += 1
+            else:
+                failures.append((row["lang"], span, out))
+        if n % 25 == 0:
+            print(f"  ...protected {n}/{len(protected_rows)}", flush=True)
+
+    elapsed = time.time() - t0
+    chars = sum(len(r["input_text"]) for r in ai_rows + identity_rows + protected_rows)
+
+    print("=" * 60)
+    print(f"model: {args.model}")
+    print(f"corpus BLEU (AI->human, n={len(ai_rows)}): {bleu:.1f}")
+    print(f"cliche removal rate: {cliche_removed / max(cliche_in, 1):.1%} ({cliche_removed}/{cliche_in})")
+    print(f"identity stability (n={len(identity_rows)}): {identity_ok / max(len(identity_rows), 1):.1%} ({identity_ok}/{len(identity_rows)})")
+    print(f"protected-span preservation: {protected_ok}/{protected_total} "
+          f"({protected_ok / max(protected_total, 1):.1%})")
+    print(f"throughput: {chars / max(elapsed, 0.01):.0f} chars/s")
+    if failures:
+        print(f"\nprotected failures ({len(failures)}):")
+        for lang, span, out in failures[:10]:
+            print(f"  [{lang}] lost {span!r} in {out[:80]!r}")
+
+    print("\nsample outputs:")
+    for row in ai_rows[:5]:
+        print("  IN :", row["input_text"][:100])
+        print("  OUT:", humanizer.humanize(row["input_text"], num_beams=3)[:100])
+        print()
+
+
+if __name__ == "__main__":
+    main()

scripts/train.py

@@ -0,0 +1,175 @@
+"""Fine-tune T5-small as a text humanizer (one language per run).
+
+English uses ``google-t5/t5-small``; Chinese uses a Chinese T5-small
+(``uer/t5-small-chinese-cluecorpussmall``). Supervised pairs come from
+``scripts/build_dataset.py`` (rule-generated), filtered to one language.
+
+MPS notes: avoid per-step host sync; accumulate the loss tensor and sync at
+evaluation. Save a checkpoint at the end of every epoch.
+"""
+
+import argparse
+import json
+import time
+from pathlib import Path
+
+import torch
+from torch.optim import AdamW
+from torch.utils.data import DataLoader, Dataset
+from transformers import BertTokenizer, T5ForConditionalGeneration, T5Tokenizer
+
+BASE_MODELS = {
+    "en": "google-t5/t5-small",
+    "zh": "uer/t5-small-chinese-cluecorpussmall",
+}
+
+
+class PairsDataset(Dataset):
+    def __init__(self, path: Path, tokenizer: T5Tokenizer, max_len: int, lang: str):
+        self.pairs = []
+        with open(path, encoding="utf-8") as f:
+            for line in f:
+                row = json.loads(line)
+                if row["lang"] != lang:
+                    continue
+                self.pairs.append((row["input_text"], row["output_text"]))
+        self.tokenizer = tokenizer
+        self.max_len = max_len
+
+    def __len__(self):
+        return len(self.pairs)
+
+    def __getitem__(self, idx):
+        src, tgt = self.pairs[idx]
+        enc = self.tokenizer(
+            src, max_length=self.max_len, padding="max_length", truncation=True
+        )
+        dec = self.tokenizer(
+            tgt, max_length=self.max_len, padding="max_length", truncation=True
+        )
+        labels = torch.tensor(dec["input_ids"])
+        labels[labels == self.tokenizer.pad_token_id] = -100
+        return {
+            "input_ids": torch.tensor(enc["input_ids"]),
+            "attention_mask": torch.tensor(enc["attention_mask"]),
+            "labels": labels,
+        }
+
+
+def decode(ids, tokenizer):
+    ids = torch.where(
+        (ids == -100) | (ids == tokenizer.pad_token_id),
+        torch.tensor(tokenizer.pad_token_id),
+        ids,
+    )
+    return tokenizer.decode(ids, skip_special_tokens=True)
+
+
+def main():
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--lang", choices=["en", "zh"], default="en")
+    parser.add_argument("--data", default="data")
+    parser.add_argument("--out", default=None)
+    parser.add_argument("--base-model", default=None)
+    parser.add_argument("--max-len", type=int, default=128)
+    parser.add_argument("--batch-size", type=int, default=32)
+    parser.add_argument("--lr", type=float, default=3e-4)
+    parser.add_argument("--epochs", type=int, default=3)
+    parser.add_argument("--warmup-steps", type=int, default=100)
+    parser.add_argument("--eval-every", type=int, default=200)
+    parser.add_argument("--seed", type=int, default=42)
+    args = parser.parse_args()
+
+    base_model = args.base_model or BASE_MODELS[args.lang]
+    out_dir = Path(args.out or f"checkpoints/humanize-text-model/{args.lang}")
+
+    torch.manual_seed(args.seed)
+    device = "mps" if torch.backends.mps.is_available() else (
+        "cuda" if torch.cuda.is_available() else "cpu"
+    )
+    print(f"lang={args.lang} base={base_model} device={device}", flush=True)
+
+    tokenizer = (
+        BertTokenizer.from_pretrained(base_model)
+        if args.lang == "zh"
+        else T5Tokenizer.from_pretrained(base_model)
+    )
+    if (out_dir / "pytorch_model.bin").exists() or (out_dir / "model.safetensors").exists():
+        print(f"resuming from existing checkpoint in {out_dir}", flush=True)
+        model = T5ForConditionalGeneration.from_pretrained(out_dir)
+    else:
+        model = T5ForConditionalGeneration.from_pretrained(base_model)
+    model.to(device)
+
+    train_ds = PairsDataset(Path(args.data) / "train.jsonl", tokenizer, args.max_len, args.lang)
+    val_ds = PairsDataset(Path(args.data) / "val.jsonl", tokenizer, args.max_len, args.lang)
+    val_samples = [
+        json.loads(l) for l in open(Path(args.data) / "val.jsonl", encoding="utf-8")
+        if json.loads(l)["lang"] == args.lang
+    ][:3]
+
+    train_loader = DataLoader(train_ds, batch_size=args.batch_size, shuffle=True, num_workers=0)
+    optimizer = AdamW(model.parameters(), lr=args.lr)
+    total_steps = args.epochs * len(train_loader)
+    scheduler = torch.optim.lr_scheduler.LinearLR(
+        optimizer, start_factor=1 / (args.warmup_steps + 1), total_iters=args.warmup_steps
+    )
+
+    out_dir.mkdir(parents=True, exist_ok=True)
+    step = 0
+    t0 = time.time()
+
+    for epoch in range(1, args.epochs + 1):
+        model.train()
+        epoch_loss = torch.zeros(())
+        for batch in train_loader:
+            batch = {k: v.to(device) for k, v in batch.items()}
+            loss = model(**batch).loss
+            loss.backward()
+            optimizer.step()
+            scheduler.step()
+            optimizer.zero_grad()
+            epoch_loss = epoch_loss + loss.detach()
+            step += 1
+
+            if step % 100 == 0:
+                print(f"[hb] step {step}/{total_steps} ({time.time() - t0:.0f}s)", flush=True)
+
+            if step % args.eval_every == 0:
+                model.eval()
+                vloss = 0.0
+                with torch.no_grad():
+                    for i in range(16):
+                        vb = val_ds[i]
+                        vb = {k: v.unsqueeze(0).to(device) for k, v in vb.items()}
+                        vloss += model(**vb).loss.item()
+                avg_train = (epoch_loss / step).item()
+                model.train()
+                print(
+                    f"[step {step}] epoch={epoch} train_loss={avg_train:.4f} "
+                    f"val_loss={vloss / 16:.4f} ({time.time() - t0:.0f}s)",
+                    flush=True,
+                )
+                sample = val_samples[0]
+                model.eval()
+                with torch.no_grad():
+                    ids = model.generate(
+                        input_ids=tokenizer(sample["input_text"], return_tensors="pt").input_ids.to(device),
+                        max_length=args.max_len,
+                    )
+                out = decode(ids[0], tokenizer)
+                model.train()
+                print("  IN :", sample["input_text"][:90])
+                print("  OUT:", out[:90], flush=True)
+
+        avg = (epoch_loss / len(train_loader)).item()
+        print(f"epoch {epoch} done: avg_loss={avg:.4f}", flush=True)
+        model.save_pretrained(out_dir)
+        tokenizer.save_pretrained(out_dir)
+        print(f"checkpoint saved to {out_dir}", flush=True)
+
+    print(f"final model saved to {out_dir}", flush=True)
+
+
+if __name__ == "__main__":
+    main()

scripts/upload_model.py

@@ -0,0 +1,43 @@
+"""Upload the humanize-text-model repo to Hugging Face.
+
+Repo layout (matches Humanizer's expectations):
+  en/  zh/  humanize.py  README.md  LICENSE  requirements.txt  scripts/  tests/  src/
+"""
+import os
+import shutil
+from pathlib import Path
+
+from huggingface_hub import HfApi
+
+TOKEN = os.environ["HF_TOKEN"]
+REPO = "Danny-Lynote/humanize-text-model"
+SRC = Path(__file__).resolve().parent.parent
+STAGE = Path("/var/folders/x9/f0cp93qs3qqclss21p96pcgr0000gn/T/opencode/humanize-upload")
+
+shutil.rmtree(STAGE, ignore_errors=True)
+STAGE.mkdir(parents=True)
+for name in ["humanize.py", "README.md", "LICENSE", "requirements.txt", "scripts", "tests", "src"]:
+    shutil.copytree(SRC / name, STAGE / name) if (SRC / name).is_dir() else shutil.copy2(SRC / name, STAGE / name)
+shutil.copytree(SRC / "checkpoints/humanize-text-model/en", STAGE / "en")
+shutil.copytree(SRC / "checkpoints/humanize-text-model/zh", STAGE / "zh")
+(STAGE / "docs").mkdir(exist_ok=True)
+if (SRC / "docs").exists():
+    for f in (SRC / "docs").iterdir():
+        if f.is_file():
+            shutil.copy2(f, STAGE / "docs" / f.name)
+
+api = HfApi(token=TOKEN, endpoint="https://huggingface.co")
+try:
+    info = api.repo_info(repo_id=REPO, repo_type="model")
+    print("repo exists, sha:", info.sha)
+except Exception:
+    api.create_repo(repo_id=REPO, repo_type="model", private=False)
+    print("created model repo")
+
+api.upload_folder(
+    folder_path=str(STAGE),
+    repo_id=REPO,
+    repo_type="model",
+    commit_message="Bilingual humanize-text-model (EN T5-small + ZH Chinese T5-small)",
+)
+print("uploaded. url: https://huggingface.co/" + REPO)

src/rules.py

@@ -0,0 +1,201 @@
+"""Shared rule engine for the humanize-text model.
+
+Conservative bilingual editing rules derived from the reference projects:
+- spaces/humanizer-lite (phrase replacements + protected spans)
+- trust-fixes/humanizer.py (multi-method pipeline philosophy)
+
+The rules generate the supervised (ai_text -> human_text) pairs used to
+fine-tune the seq2seq model, and can also run as a standalone fallback.
+"""
+
+import re
+
+# --------------------------------------------------------------------------
+# Protected spans: code, URLs, paths, numbers, quoted strings.
+# --------------------------------------------------------------------------
+
+PROTECTED_PATTERN = re.compile(
+    r"(`[^`]+`|https?://[^\s\u4e00-\u9fff，。！？；：、]+|(?:[A-Za-z]:)?(?:[/\\][\w.\-]+)+|"
+    r"\b\d+(?:\.\d+)?(?:%|[A-Za-z]+)?\b|\"[^\"]*\"|'[^']*')"
+)
+
+
+CJK_RE = re.compile(r"[\u4e00-\u9fff]")
+
+
+def protect(text: str, lang: str = "auto"):
+    """Replace protected spans with language-aware placeholders:
+    EN ``PROTECTED_n``, ZH ``【保护n】`` (both tokenizer-friendly)."""
+    protected = []
+    zh = (
+        lang == "zh"
+        or (lang == "auto" and len(CJK_RE.findall(text)) > 0)
+    )
+
+    def replace(match):
+        token = f"【保护{len(protected)}】" if zh else f"PROTECTED_{len(protected)}"
+        protected.append(match.group(0))
+        return token
+
+    return PROTECTED_PATTERN.sub(replace, text), protected
+
+
+def restore(text: str, protected):
+    """Restore placeholders (either format, tolerant of inserted spaces)."""
+    for index, value in enumerate(protected):
+        text = re.sub(rf"PROTECTED_{index}\b", lambda m: value, text)
+        text = re.sub(
+            rf"【\s*保\s*护\s*{index}\s*】", lambda m: value, text
+        )
+    return text
+
+
+# --------------------------------------------------------------------------
+# Formulaic (AI) phrases, English and Chinese.
+# --------------------------------------------------------------------------
+
+EN_PHRASES = {
+    "in today's rapidly evolving world": "today",
+    "in today's fast-paced world": "now",
+    "it is important to note that": "",
+    "it is worth noting that": "",
+    "it can be seen that": "",
+    "serves as a testament to": "shows",
+    "a testament to": "proof of",
+    "delve into": "look at",
+    "leverage": "use",
+    "leveraging": "using",
+    "seamless": "smooth",
+    "robust": "reliable",
+    "game-changer": "big improvement",
+    "groundbreaking": "new",
+    "cutting-edge": "modern",
+    "state-of-the-art": "modern",
+    "plays a crucial role in": "matters for",
+    "plays a crucial role": "matters",
+    "in the realm of": "in",
+    "in conclusion": "",
+    "moreover": "also",
+    "furthermore": "also",
+    "additionally": "also",
+    "it is essential to": "we need to",
+    "comprehensive approach": "broad approach",
+    "foster": "support",
+    "fostering": "supporting",
+    "undoubtedly": "",
+    "no longer optional but essential": "necessary",
+    "revolutionize": "change",
+    "revolutionizing": "changing",
+    "holistic": "full",
+}
+
+ZH_PHRASES = {
+    "值得注意的是": "",
+    "在当今快速发展的时代": "现在",
+    "综上所述": "",
+    "总而言之": "",
+    "由此可见": "",
+    "赋能": "帮助",
+    "助力": "帮助",
+    "降本增效": "省钱提效",
+    "闭环": "完整流程",
+    "无缝": "顺畅",
+    "发挥着重要作用": "很有用",
+    "发挥着至关重要的作用": "很重要",
+    "数字化转型": "数字化",
+    "必然趋势": "趋势",
+    "积极拥抱新技术": "用新技术",
+    "不断提升": "提高",
+    "持续优化": "改进",
+    "实现": "做到",
+    "进一步": "",
+}
+
+EN_OPENERS = ["Moreover,", "Furthermore,", "Additionally,", "In conclusion,"]
+
+EN_EXTRA = {
+    "very": "",
+    "really": "",
+    "extremely": "",
+    "highly": "",
+}
+
+CONTRACTIONS = {
+    "it is": "it's",
+    "it will": "it'll",
+    "we are": "we're",
+    "we will": "we'll",
+    "you are": "you're",
+    "you will": "you'll",
+    "do not": "don't",
+    "does not": "doesn't",
+    "cannot": "can't",
+    "I am": "I'm",
+    "I will": "I'll",
+    "that is": "that's",
+    "there is": "there's",
+    "will not": "won't",
+    "it is not": "it isn't",
+}
+
+ZH_PARTICLES_END = ["了", "吧", "呢"]
+ZH_PARTICLES_START = ["其实", "说实话", "我总觉得"]
+
+# --------------------------------------------------------------------------
+# Core rule application (single pass, deterministic).
+# --------------------------------------------------------------------------
+
+
+def humanize_rules(text: str, lang: str = "auto") -> str:
+    """Apply conservative editing rules to a draft. Deterministic."""
+    protected_text, protected = protect(text, lang)
+
+    if lang in ("auto", "en"):
+        low = protected_text.lower()
+        for phrase, replacement in EN_PHRASES.items():
+            protected_text = re.sub(re.escape(phrase), replacement, protected_text, flags=re.I)
+        # Only apply opener removals when they start a sentence boundary.
+        for opener in EN_OPENERS:
+            protected_text = re.sub(
+                re.escape(opener) + r"\s*", "", protected_text, flags=re.I
+            )
+        for word, replacement in EN_EXTRA.items():
+            protected_text = re.sub(r"\b" + re.escape(word) + r"\s+", "", protected_text, flags=re.I)
+        for phrase, replacement in CONTRACTIONS.items():
+            protected_text = re.sub(
+                r"\b" + re.escape(phrase) + r"\b", replacement, protected_text, flags=re.I
+            )
+
+    if lang in ("auto", "zh"):
+        for phrase, replacement in ZH_PHRASES.items():
+            protected_text = protected_text.replace(phrase, replacement)
+        protected_text = re.sub(r"，{2,}", "，", protected_text)
+
+    protected_text = re.sub(r"[ \t]{2,}", " ", protected_text)
+    protected_text = re.sub(r"\s+([,.;:!?，。；：！？])", r"\1", protected_text)
+    protected_text = re.sub(r"[,;]\s*\.", ".", protected_text)
+    protected_text = re.sub(r"\.\s*[,;]", ".", protected_text)
+    protected_text = re.sub(r"[，,]\s*(?=[。！？.!?])", "", protected_text)
+    protected_text = re.sub(r"\.\s+([a-z])", lambda m: ". " + m.group(1).upper(), protected_text)
+    protected_text = re.sub(r"([.!?。！？])\s*\1+", r"\1", protected_text)
+    protected_text = re.sub(r"\n{3,}", "\n\n", protected_text).strip(" ,，")
+    return restore(protected_text, protected)
+
+
+def zh_add_colloquial(text: str) -> str:
+    """Add light colloquial markers (used for data augmentation only)."""
+    if re.search(r"[\u4e00-\u9fff]", text):
+        text = re.sub(r"[。！？]", lambda m: m.group(0) + "呢", text, count=1)
+        text = re.sub(r"^", "其实，", text, count=1)
+        text = text.replace("今天", "今天吧")
+        text = text.replace("我觉得", "我总觉得")
+    return text
+
+
+def en_add_colloquial(text: str) -> str:
+    """Add light colloquial markers (used for data augmentation only)."""
+    text = re.sub(r"(^|\. )(I|we|he|she|they)\b", r"\1Honestly, \2", text, count=1)
+    text = text.replace("The ", "The ", 1)
+    if " quite " not in text:
+        text = re.sub(r"(\w+)\.$", r"\1, honestly.", text, count=1)
+    return text

tests/__init__.py

@@ -0,0 +1,2 @@
+import pytest
+pytest.importorskip("src.rules")

tests/test_dataset.py

@@ -0,0 +1,151 @@
+"""Tests for the synthetic dataset builder and rule engine."""
+
+import json
+import random
+import sys
+from pathlib import Path
+
+sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
+
+from src.rules import humanize_rules, protect, restore
+
+DATA = Path(__file__).resolve().parent.parent / "data"
+EN_PHRASES = {
+    "it is important to note that",
+    "it is worth noting that",
+    "moreover",
+    "furthermore",
+    "in conclusion",
+    "leverage",
+    "seamless",
+    "robust",
+    "delve",
+    "testament",
+}
+ZH_PHRASES = {
+    "值得注意的是",
+    "综上所述",
+    "赋能",
+    "降本增效",
+    "闭环",
+    "无缝",
+    "由此可见",
+}
+
+
+def _load(split: str):
+    pairs = []
+    with open(DATA / f"{split}.jsonl", encoding="utf-8") as f:
+        for line in f:
+            pairs.append(json.loads(line))
+    return pairs
+
+
+def test_splits_exist_and_nonempty():
+    for split in ("train", "val", "test"):
+        pairs = _load(split)
+        assert len(pairs) > 0, f"{split} is empty"
+    assert len(_load("train")) > 10_000
+
+
+def test_pairs_have_expected_fields():
+    for split in ("train", "val", "test"):
+        for p in _load(split):
+            assert set(p) == {"input_text", "output_text", "lang"}
+            assert p["lang"] in {"en", "zh"}
+            assert p["input_text"] and p["output_text"]
+
+
+def test_both_languages_present():
+    langs = {p["lang"] for p in _load("train")}
+    assert langs == {"en", "zh"}
+
+
+def test_ai_phrases_removed_in_targets():
+    counts = 0
+    checked = 0
+    for p in _load("train"):
+        if "值得注意的是" in p["input_text"] or "it is important to note that" in p["input_text"].lower():
+            checked += 1
+            target_lower = p["output_text"].lower()
+            if p["lang"] == "zh":
+                assert "值得注意的是" not in target_lower
+                assert "综上所述" not in target_lower
+            else:
+                assert "it is important to note that" not in target_lower
+                assert "it is worth noting that" not in target_lower
+            counts += 1
+    assert counts > 500
+
+
+def test_protected_spans_preserved():
+    seen = 0
+    for p in _load("train") + _load("test"):
+        src = p["input_text"]
+        if "https://" in src or "ABC-12345" in src or "v2.1.3" in src:
+            for token in ("https://api.lynote.ai", "ABC-12345", "v2.1.3"):
+                if token in src:
+                    assert token in p["output_text"], f"lost {token} in {p['output_text']}"
+                    seen += 1
+    assert seen > 10
+
+
+def test_identity_pairs_unchanged():
+    for p in _load("train"):
+        if p["input_text"] == p["output_text"]:
+            assert p["lang"] in {"en", "zh"}
+            return
+    raise AssertionError("no identity pairs found")
+
+
+def test_deterministic_generation():
+    """Same seed yields the same corpus (checked on a fixed sample)."""
+    src = "In today's rapidly evolving world, it is important to note that this solution is robust. Moreover, it is seamless."
+    out1 = humanize_rules(src, "en")
+    out2 = humanize_rules(src, "en")
+    assert out1 == out2
+
+
+def test_protect_restore_roundtrip():
+    text = "Run `pip install x==1.0` on https://example.com/a/b now — the price is $12.50 and \"quoted\" text."
+    masked, protected = protect(text)
+    assert masked != text
+    assert "PROTECTED_0" in masked
+    assert restore(masked, protected) == text
+
+
+def test_rules_en():
+    src = "In today's rapidly evolving world, it is important to note that this robust solution serves as a testament to our commitment. Moreover, we leverage cutting-edge technology for a seamless experience."
+    out = humanize_rules(src, "en")
+    lower = out.lower()
+    assert "it is important to note that" not in lower
+    assert "moreover" not in lower
+    assert "leverage" not in lower
+    assert "seamless" not in lower
+    assert "robust" not in lower
+    assert "testament" not in lower
+    assert out != src
+
+
+def test_rules_zh():
+    src = "值得注意的是，在当今快速发展的时代，我们通过赋能团队来实现降本增效，形成完整闭环，无缝连接各个环节。"
+    out = humanize_rules(src, "zh")
+    assert "值得注意的是" not in out
+    assert "赋能" not in out
+    assert "降本增效" not in out
+    assert "无缝" not in out
+    assert out != src
+
+
+def test_rules_keep_human_prose_intact():
+    clean = "I walked the dog this morning and it started raining halfway through the park."
+    out = humanize_rules(clean, "en")
+    assert out == clean
+
+
+def test_dataset_reproducibility_from_seed():
+    random.seed(7)
+    sample1 = random.sample(_load("test"), 50)
+    random.seed(7)
+    sample2 = random.sample(_load("test"), 50)
+    assert [p["input_text"] for p in sample1] == [p["input_text"] for p in sample2]

tests/test_humanize.py

@@ -0,0 +1,114 @@
+"""Model inference tests. Require a trained checkpoint or the Hub model.
+
+Set HUMANIZE_MODEL (default: checkpoints/humanize-text-model).
+"""
+
+import os
+import sys
+from pathlib import Path
+
+import pytest
+
+sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
+
+from humanize import Humanizer, MAX_CHARS, protect, restore
+
+MODEL = os.environ.get("HUMANIZE_MODEL", "checkpoints/humanize-text-model")
+
+
+@pytest.fixture(scope="module")
+def humanizer():
+    return Humanizer(model_id=MODEL)
+
+
+def test_loads_and_returns_text(humanizer):
+    out = humanizer.humanize("It is important to note that this is a test.")
+    assert isinstance(out, str) and out
+
+
+def test_removes_ai_cliches_en(humanizer):
+    src = (
+        "In today's rapidly evolving world, it is important to note that this robust "
+        "solution serves as a testament to our commitment to innovation. Moreover, "
+        "we leverage cutting-edge technology to deliver a seamless experience."
+    )
+    out = humanizer.humanize(src)
+    lower = out.lower()
+    assert "it is important to note that" not in lower
+    assert "moreover" not in lower
+    assert "leverage" not in lower
+    assert "seamless" not in lower
+    assert "testament" not in lower
+    assert out != src
+
+
+def test_removes_ai_cliches_zh(humanizer):
+    src = "值得注意的是，我们通过赋能团队来助力企业实现降本增效，形成完整的业务闭环。"
+    out = humanizer.humanize(src)
+    assert "值得注意的是" not in out
+    assert "赋能" not in out
+    assert "降本增效" not in out
+    assert out != src
+
+
+def test_protects_urls_numbers_quotes(humanizer):
+    src = (
+        "It is important to note that the server at https://api.lynote.ai/v1/detect "
+        "handles 1,200 requests per second with a 99.5% uptime, as stated in "
+        '"Project Lynote". Moreover, run `pip install humanize-text==1.4.2`.'
+    )
+    out = humanizer.humanize(src)
+    for token in ("https://api.lynote.ai/v1/detect", "1,200", "99.5%", '"Project Lynote"', "humanize-text==1.4.2"):
+        assert token in out, f"protected token lost: {token}"
+
+
+def test_human_prose_stays_similar(humanizer):
+    src = "I walked the dog this morning and it started raining halfway through the park, so we ran home."
+    out = humanizer.humanize(src)
+    # Identity should be near-preserved: allow light edits but no additions.
+    assert abs(len(out.split()) - len(src.split())) <= 4
+
+
+def test_short_input_and_limits(humanizer):
+    with pytest.raises(ValueError):
+        humanizer.humanize("")
+    with pytest.raises(ValueError):
+        humanizer.humanize("   ")
+    with pytest.raises(ValueError):
+        humanizer.humanize("x" * (MAX_CHARS + 1))
+
+
+def test_batch_consistency(humanizer):
+    texts = [
+        "It is worth noting that this robust platform is a game-changer. Moreover, it is seamless.",
+        "值得注意的是，我们通过赋能团队来实现降本增效。",
+        "I walked the dog this morning and the sky was grey.",
+    ]
+    batch = humanizer.humanize_batch(texts)
+    assert len(batch) == 3
+    for out, src in zip(batch, texts):
+        assert isinstance(out, str) and out
+    single = [humanizer.humanize(t) for t in texts]
+    # batch and single should agree exactly (same decode path)
+    assert batch == single
+
+
+def test_protect_restore_roundtrip():
+    text = "Run `pip install x==1.0` at https://example.com/a?b=1 — cost is $12.50 and \"Q3\" rose 4%."
+    masked, protected = protect(text)
+    assert "PROTECTED_0" in masked
+    assert restore(masked, protected) == text
+
+
+def test_is_chinese_routing():
+    from humanize import is_chinese
+    assert is_chinese("值得注意的是，这是中文测试。")
+    assert not is_chinese("It is important to note that this is English.")
+    assert not is_chinese("Mixed text with 中文 characters but mostly English words here.")
+
+
+def test_polish():
+    from humanize import Humanizer as H
+    assert H._polish("hello , world") == "hello, world"
+    assert H._polish("end., next") == "end. Next"
+    assert H._polish("a  b") == "a b"

zh/config.json

@@ -0,0 +1,31 @@
+{
+  "architectures": [
+    "T5ForConditionalGeneration"
+  ],
+  "classifier_dropout": 0.0,
+  "d_ff": 2048,
+  "d_kv": 64,
+  "d_model": 512,
+  "decoder_start_token_id": 101,
+  "dense_act_fn": "relu",
+  "dropout_rate": 0.1,
+  "dtype": "float32",
+  "eos_token_id": 1,
+  "feed_forward_proj": "relu",
+  "initializer_factor": 1.0,
+  "is_encoder_decoder": true,
+  "is_gated_act": false,
+  "layer_norm_epsilon": 1e-06,
+  "model_type": "t5",
+  "n_positions": 512,
+  "num_decoder_layers": 6,
+  "num_heads": 8,
+  "num_layers": 6,
+  "pad_token_id": 0,
+  "relative_attention_max_distance": 128,
+  "relative_attention_num_buckets": 32,
+  "tokenizer_class": "BertTokenizer",
+  "transformers_version": "4.57.6",
+  "use_cache": true,
+  "vocab_size": 21228
+}

zh/generation_config.json

@@ -0,0 +1,7 @@
+{
+  "_from_model_config": true,
+  "decoder_start_token_id": 101,
+  "eos_token_id": 1,
+  "pad_token_id": 0,
+  "transformers_version": "4.57.6"
+}

zh/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:19c458057db296fd6abe06303a5e7bc92a9806d2d663bf43f1238abd0d021c17
+size 219718696

zh/special_tokens_map.json

@@ -0,0 +1,7 @@
+{
+  "cls_token": "[CLS]",
+  "mask_token": "[MASK]",
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "unk_token": "[UNK]"
+}

zh/tokenizer_config.json

@@ -0,0 +1,58 @@
+{
+  "added_tokens_decoder": {
+    "0": {
+      "content": "[PAD]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100": {
+      "content": "[UNK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "101": {
+      "content": "[CLS]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "102": {
+      "content": "[SEP]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "103": {
+      "content": "[MASK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "clean_up_tokenization_spaces": true,
+  "cls_token": "[CLS]",
+  "do_basic_tokenize": true,
+  "do_lower_case": false,
+  "extra_special_tokens": {},
+  "mask_token": "[MASK]",
+  "model_max_length": 1000000000000000019884624838656,
+  "never_split": null,
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "strip_accents": null,
+  "tokenize_chinese_chars": true,
+  "tokenizer_class": "BertTokenizer",
+  "unk_token": "[UNK]"
+}