scrallex commited on
Commit
bd8dffd
·
0 Parent(s):

Initial upload of fox_stage1 adapter

Browse files
.gitattributes ADDED
@@ -0,0 +1,4 @@
 
 
 
 
 
1
+ *.safetensors filter=lfs diff=lfs merge=lfs -text
2
+ *.bin filter=lfs diff=lfs merge=lfs -text
3
+ *.pt filter=lfs diff=lfs merge=lfs -text
4
+ *.pth filter=lfs diff=lfs merge=lfs -text
README.md ADDED
@@ -0,0 +1,85 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # FOX-Manifold-OCR (fox_stage1 LoRA)
2
+
3
+ ## TL;DR
4
+ - 12-epoch LoRA adapter fine-tune of `microsoft/trocr-base-printed` on the Fox (EN + CN) and OmniDoc structural-manifold manifests with a single RTX 3080 Ti.
5
+ - Checkpoint exported from `output/training_runs/fox_stage1/checkpoint-1452` on 2024-11-04 and staged for the private Hugging Face repo `HF_USER/fox-manifold-ocr`.
6
+ - Delivers 44x-46x average structural compression while holding 0.87-0.93 F1 across benchmark splits (see table below).
7
+
8
+ ## Model Details
9
+ | Field | Value |
10
+ |-------|-------|
11
+ | Base model | `microsoft/trocr-base-printed` (VisionEncoderDecoder) |
12
+ | Adapter | PEFT LoRA, rank 8, alpha 16, dropout 0.05 targeting `q_proj,k_proj,v_proj,out_proj` |
13
+ | Precision | fp16 with gradient checkpointing enabled |
14
+ | Sequence lengths | `max_target_length=640`, `generation_max_length=768` |
15
+ | Hardware & time | Single RTX 3080 Ti (12 GB); ~45 minutes for the 12-epoch fox_stage1 run |
16
+ | Trainer stats | Final train loss 7.11, final eval loss 7.29 (Hugging Face Trainer logs) |
17
+ | Files exported | `adapter_model.safetensors`, `adapter_config.json`, tokenizer merges/vocab/config, training args + RNG state, this README |
18
+
19
+ License: inherits the MIT license of `microsoft/trocr-base-printed`; dataset usage must continue to respect the Fox and OmniDoc terms.
20
+
21
+ ## Training Command
22
+ ```
23
+ cd structural-manifold-compression
24
+ source .venv/bin/activate
25
+ RUN_NAME=fox_stage1 \
26
+ TRAIN_DATASETS="fox_en=data/benchmark_corpus/fox/metadata/text_manifest.jsonl:data/benchmark_corpus/fox/raw,fox_cn=data/benchmark_corpus/fox/metadata/text_manifest.jsonl:data/benchmark_corpus/fox/raw,omnidoc=data/benchmark_corpus/omnidocbench/metadata/text_manifest.jsonl:data/benchmark_corpus/omnidocbench/raw/OmniDocBench" \
27
+ PRECISION=fp16 \
28
+ LORA_RANK=8 \
29
+ USE_GRADIENT_CHECKPOINTING=1 \
30
+ ./scripts/training/run_pipeline.py train
31
+ ```
32
+ Key hyperparameters: epochs=12, per-device batch size=1, gradient accumulation=8 (effective batch 8), eval split=0.08, learning rate=1e-4 with cosine scheduler, weight decay=0.01, warmup ratio=0.05, save/eval every 200 steps, LoRA dropout=0.05, metric tokenizer left at repo default.
33
+
34
+ ## Data Summary
35
+ - **Fox EN + CN**: enterprise PDFs spanning analyst notes and bilingual reports; manifests from `data/benchmark_corpus/fox`.
36
+ - **OmniDocBench**: invoices, contracts, infographics, and scanned tables via `data/benchmark_corpus/omnidocbench`.
37
+ - Images are read directly from the repository roots, while manifests provide the transcript spans consumed by `ocr_trainer.py`. No additional filtering beyond the 8% validation split was applied.
38
+
39
+ ## Metrics (benchmark_eval.py, window=512, stride=384, precision=3, tokenizer=external/DeepSeek-OCR/weights)
40
+ | Benchmark | Compression ratio | Character F1 | Token F1 |
41
+ |-----------|-------------------|--------------|----------|
42
+ | Fox EN | 44.29 | 0.9258 | 0.9146 |
43
+ | Fox CN | 44.68 | 0.9062 | 0.9029 |
44
+ | OmniDoc | 45.93 | 0.8913 | 0.8682 |
45
+
46
+ Reproduction:
47
+ ```
48
+ python scripts/experiments/benchmark_eval.py \
49
+ --dataset fox_en=data/benchmark_corpus/fox/text/en_page_ocr \
50
+ --dataset fox_cn=data/benchmark_corpus/fox/text/cn_page_ocr \
51
+ --dataset omnidoc=data/benchmark_corpus/omnidocbench/text \
52
+ --tokenizer external/DeepSeek-OCR/weights \
53
+ --tokenizer-trust-remote-code \
54
+ --output-dir output/benchmark_runs/full
55
+ ```
56
+ The resulting `output/benchmark_runs/full/summary.json` feeds the table above; CSVs land beside it for spreadsheet import.
57
+
58
+ ## Usage
59
+ ```python
60
+ from pathlib import Path
61
+ from peft import PeftModel
62
+ from PIL import Image
63
+ from transformers import AutoTokenizer, TrOCRProcessor, VisionEncoderDecoderModel
64
+
65
+ repo_id = "HF_USER/fox-manifold-ocr"
66
+ tokenizer = AutoTokenizer.from_pretrained(repo_id)
67
+ processor = TrOCRProcessor.from_pretrained("microsoft/trocr-base-printed")
68
+ base = VisionEncoderDecoderModel.from_pretrained("microsoft/trocr-base-printed")
69
+ model = PeftModel.from_pretrained(base, repo_id).eval()
70
+
71
+ image = Image.open("sample_page.png").convert("RGB")
72
+ inputs = processor(images=image, return_tensors="pt").to(model.device)
73
+ generated = model.generate(**inputs, max_new_tokens=640)
74
+ text = tokenizer.batch_decode(generated, skip_special_tokens=True)[0]
75
+ print(text)
76
+ ```
77
+ Merge the adapter into the base model (`model = model.merge_and_unload()`) if you need a standalone `pytorch_model.bin` for deployment.
78
+
79
+ ## Limitations & Responsible Use
80
+ - Adapter is tuned for bilingual business documents and dense tables; handwriting, math-heavy, or extremely low-resolution scans were not represented and will degrade.
81
+ - Structural compression targets favor recall at the verification stage, so downstream consumers should keep a throttled kill switch for high false-positive rates.
82
+ - Dataset includes sensitive enterprise content; apply your own filtering or synthetic replacements before redistributing outputs.
83
+
84
+ ## Changelog
85
+ - 2024-11-04: fox_stage1 checkpoint-1452 exported to `output/hf_upload/fox_stage1` (current document).
adapter_config.json ADDED
@@ -0,0 +1,39 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "alpha_pattern": {},
3
+ "auto_mapping": null,
4
+ "base_model_name_or_path": "microsoft/trocr-base-printed",
5
+ "bias": "none",
6
+ "corda_config": null,
7
+ "eva_config": null,
8
+ "exclude_modules": null,
9
+ "fan_in_fan_out": false,
10
+ "inference_mode": true,
11
+ "init_lora_weights": true,
12
+ "layer_replication": null,
13
+ "layers_pattern": null,
14
+ "layers_to_transform": null,
15
+ "loftq_config": {},
16
+ "lora_alpha": 16,
17
+ "lora_bias": false,
18
+ "lora_dropout": 0.05,
19
+ "megatron_config": null,
20
+ "megatron_core": "megatron.core",
21
+ "modules_to_save": null,
22
+ "peft_type": "LORA",
23
+ "qalora_group_size": 16,
24
+ "r": 8,
25
+ "rank_pattern": {},
26
+ "revision": null,
27
+ "target_modules": [
28
+ "q_proj",
29
+ "out_proj",
30
+ "k_proj",
31
+ "v_proj"
32
+ ],
33
+ "target_parameters": null,
34
+ "task_type": "SEQ_2_SEQ_LM",
35
+ "trainable_token_indices": null,
36
+ "use_dora": false,
37
+ "use_qalora": false,
38
+ "use_rslora": false
39
+ }
adapter_model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b68d68d64ed8ff74766304f84a274742b6dcc2ba330dc451bb38f13924b7e7da
3
+ size 6123432
merges.txt ADDED
The diff for this file is too large to render. See raw diff
 
optimizer.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:96d0c6ed52734ed81836a4850362b70fb3c4e9db4faff35465267ce69eb20c9a
3
+ size 12354059
rng_state.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c2f7627e3d63868316767e0582fef1204e0a436989c08b511b1cd5db6c868336
3
+ size 14709
scaler.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:5b2d7bfd8833db8e49fa5b7ac95dac2f576f54a5d91de14810458e93903b743f
3
+ size 1383
scheduler.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f62afa13d3e7a876a91753e3bb8656e1fd4be0d8fa46ee51fc12de2c2b441328
3
+ size 1465
special_tokens_map.json ADDED
@@ -0,0 +1,51 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "bos_token": {
3
+ "content": "<s>",
4
+ "lstrip": false,
5
+ "normalized": true,
6
+ "rstrip": false,
7
+ "single_word": false
8
+ },
9
+ "cls_token": {
10
+ "content": "<s>",
11
+ "lstrip": false,
12
+ "normalized": true,
13
+ "rstrip": false,
14
+ "single_word": false
15
+ },
16
+ "eos_token": {
17
+ "content": "</s>",
18
+ "lstrip": false,
19
+ "normalized": true,
20
+ "rstrip": false,
21
+ "single_word": false
22
+ },
23
+ "mask_token": {
24
+ "content": "<mask>",
25
+ "lstrip": true,
26
+ "normalized": true,
27
+ "rstrip": false,
28
+ "single_word": false
29
+ },
30
+ "pad_token": {
31
+ "content": "<pad>",
32
+ "lstrip": false,
33
+ "normalized": true,
34
+ "rstrip": false,
35
+ "single_word": false
36
+ },
37
+ "sep_token": {
38
+ "content": "</s>",
39
+ "lstrip": false,
40
+ "normalized": true,
41
+ "rstrip": false,
42
+ "single_word": false
43
+ },
44
+ "unk_token": {
45
+ "content": "<unk>",
46
+ "lstrip": false,
47
+ "normalized": true,
48
+ "rstrip": false,
49
+ "single_word": false
50
+ }
51
+ }
tokenizer.json ADDED
The diff for this file is too large to render. See raw diff
 
tokenizer_config.json ADDED
@@ -0,0 +1,58 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "add_prefix_space": false,
3
+ "added_tokens_decoder": {
4
+ "0": {
5
+ "content": "<s>",
6
+ "lstrip": false,
7
+ "normalized": true,
8
+ "rstrip": false,
9
+ "single_word": false,
10
+ "special": true
11
+ },
12
+ "1": {
13
+ "content": "<pad>",
14
+ "lstrip": false,
15
+ "normalized": true,
16
+ "rstrip": false,
17
+ "single_word": false,
18
+ "special": true
19
+ },
20
+ "2": {
21
+ "content": "</s>",
22
+ "lstrip": false,
23
+ "normalized": true,
24
+ "rstrip": false,
25
+ "single_word": false,
26
+ "special": true
27
+ },
28
+ "3": {
29
+ "content": "<unk>",
30
+ "lstrip": false,
31
+ "normalized": true,
32
+ "rstrip": false,
33
+ "single_word": false,
34
+ "special": true
35
+ },
36
+ "50264": {
37
+ "content": "<mask>",
38
+ "lstrip": true,
39
+ "normalized": true,
40
+ "rstrip": false,
41
+ "single_word": false,
42
+ "special": true
43
+ }
44
+ },
45
+ "bos_token": "<s>",
46
+ "clean_up_tokenization_spaces": false,
47
+ "cls_token": "<s>",
48
+ "eos_token": "</s>",
49
+ "errors": "replace",
50
+ "extra_special_tokens": {},
51
+ "mask_token": "<mask>",
52
+ "model_max_length": 512,
53
+ "pad_token": "<pad>",
54
+ "sep_token": "</s>",
55
+ "tokenizer_class": "RobertaTokenizer",
56
+ "trim_offsets": true,
57
+ "unk_token": "<unk>"
58
+ }
trainer_state.json ADDED
@@ -0,0 +1,678 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "best_global_step": 600,
3
+ "best_metric": 0.05530853974654382,
4
+ "best_model_checkpoint": "/sep/structural-manifold-compression/output/training_runs/fox_stage1/checkpoint-600",
5
+ "epoch": 12.0,
6
+ "eval_steps": 100,
7
+ "global_step": 1452,
8
+ "is_hyper_param_search": false,
9
+ "is_local_process_zero": true,
10
+ "is_world_process_zero": true,
11
+ "log_history": [
12
+ {
13
+ "epoch": 0.2066115702479339,
14
+ "grad_norm": 5.947324275970459,
15
+ "learning_rate": 9.994788203922447e-05,
16
+ "loss": 11.9365,
17
+ "step": 25
18
+ },
19
+ {
20
+ "epoch": 0.4132231404958678,
21
+ "grad_norm": 4.52055549621582,
22
+ "learning_rate": 9.813513645245729e-05,
23
+ "loss": 9.574,
24
+ "step": 50
25
+ },
26
+ {
27
+ "epoch": 0.6198347107438017,
28
+ "grad_norm": 2.9425737857818604,
29
+ "learning_rate": 9.38241303448241e-05,
30
+ "loss": 8.2893,
31
+ "step": 75
32
+ },
33
+ {
34
+ "epoch": 0.8264462809917356,
35
+ "grad_norm": 1.500144362449646,
36
+ "learning_rate": 8.72386091371891e-05,
37
+ "loss": 7.881,
38
+ "step": 100
39
+ },
40
+ {
41
+ "epoch": 0.8264462809917356,
42
+ "eval_character_accuracy": 0.07699992799539168,
43
+ "eval_character_f1": 0.26155196367780104,
44
+ "eval_character_normalized_edit_distance": 0.9230000720046083,
45
+ "eval_character_precision": -0.18725325615050648,
46
+ "eval_character_recall": 0.07699992799539168,
47
+ "eval_loss": 7.875946998596191,
48
+ "eval_runtime": 105.885,
49
+ "eval_samples_per_second": 0.793,
50
+ "eval_steps_per_second": 0.793,
51
+ "eval_token_accuracy": -0.042498557414887506,
52
+ "eval_token_f1": -0.07808865152697282,
53
+ "eval_token_precision": -0.4803752867912161,
54
+ "eval_token_recall": -0.042498557414887506,
55
+ "step": 100
56
+ },
57
+ {
58
+ "epoch": 1.0330578512396693,
59
+ "grad_norm": 1.8186473846435547,
60
+ "learning_rate": 7.872036780210026e-05,
61
+ "loss": 7.492,
62
+ "step": 125
63
+ },
64
+ {
65
+ "epoch": 1.2396694214876034,
66
+ "grad_norm": 2.1581435203552246,
67
+ "learning_rate": 6.871151137141266e-05,
68
+ "loss": 7.6082,
69
+ "step": 150
70
+ },
71
+ {
72
+ "epoch": 1.4462809917355373,
73
+ "grad_norm": 2.3698322772979736,
74
+ "learning_rate": 5.773150925401641e-05,
75
+ "loss": 7.3798,
76
+ "step": 175
77
+ },
78
+ {
79
+ "epoch": 1.6528925619834711,
80
+ "grad_norm": 2.196545362472534,
81
+ "learning_rate": 4.6350234266954626e-05,
82
+ "loss": 7.4732,
83
+ "step": 200
84
+ },
85
+ {
86
+ "epoch": 1.6528925619834711,
87
+ "eval_character_accuracy": 0.06128492223502302,
88
+ "eval_character_f1": 0.1641450344783411,
89
+ "eval_character_normalized_edit_distance": 0.938715077764977,
90
+ "eval_character_precision": -0.2419619891397542,
91
+ "eval_character_recall": 0.06128492223502302,
92
+ "eval_loss": 7.591057300567627,
93
+ "eval_runtime": 104.8701,
94
+ "eval_samples_per_second": 0.801,
95
+ "eval_steps_per_second": 0.801,
96
+ "eval_token_accuracy": -0.06878245816503181,
97
+ "eval_token_f1": -0.11809033105597452,
98
+ "eval_token_precision": -0.41708427374622237,
99
+ "eval_token_recall": -0.06878245816503181,
100
+ "step": 200
101
+ },
102
+ {
103
+ "epoch": 1.859504132231405,
104
+ "grad_norm": 3.392272710800171,
105
+ "learning_rate": 3.515838568050736e-05,
106
+ "loss": 7.3805,
107
+ "step": 225
108
+ },
109
+ {
110
+ "epoch": 2.0661157024793386,
111
+ "grad_norm": 1.8100775480270386,
112
+ "learning_rate": 2.4736831350043536e-05,
113
+ "loss": 7.4541,
114
+ "step": 250
115
+ },
116
+ {
117
+ "epoch": 2.2727272727272725,
118
+ "grad_norm": 1.6662075519561768,
119
+ "learning_rate": 1.562646010789411e-05,
120
+ "loss": 7.2798,
121
+ "step": 275
122
+ },
123
+ {
124
+ "epoch": 2.479338842975207,
125
+ "grad_norm": 1.5955466032028198,
126
+ "learning_rate": 8.30010910550611e-06,
127
+ "loss": 7.4174,
128
+ "step": 300
129
+ },
130
+ {
131
+ "epoch": 2.479338842975207,
132
+ "eval_character_accuracy": 0.05942180299539168,
133
+ "eval_character_f1": 0.13834188776330955,
134
+ "eval_character_normalized_edit_distance": 0.9405781970046083,
135
+ "eval_character_precision": -0.42160250306080815,
136
+ "eval_character_recall": 0.05942180299539168,
137
+ "eval_loss": 7.534529685974121,
138
+ "eval_runtime": 100.366,
139
+ "eval_samples_per_second": 0.837,
140
+ "eval_steps_per_second": 0.837,
141
+ "eval_token_accuracy": -0.04899019042123487,
142
+ "eval_token_f1": -0.0887424381203322,
143
+ "eval_token_precision": -0.47061440763661366,
144
+ "eval_token_recall": -0.04899019042123487,
145
+ "step": 300
146
+ },
147
+ {
148
+ "epoch": 2.6859504132231407,
149
+ "grad_norm": 1.5894019603729248,
150
+ "learning_rate": 9.204587302090488e-05,
151
+ "loss": 7.4469,
152
+ "step": 325
153
+ },
154
+ {
155
+ "epoch": 2.8925619834710745,
156
+ "grad_norm": 1.8111478090286255,
157
+ "learning_rate": 9.043745478918972e-05,
158
+ "loss": 7.3277,
159
+ "step": 350
160
+ },
161
+ {
162
+ "epoch": 3.0991735537190084,
163
+ "grad_norm": 2.056180000305176,
164
+ "learning_rate": 8.869790185253766e-05,
165
+ "loss": 7.3545,
166
+ "step": 375
167
+ },
168
+ {
169
+ "epoch": 3.3057851239669422,
170
+ "grad_norm": 1.8154643774032593,
171
+ "learning_rate": 8.68328554107289e-05,
172
+ "loss": 7.18,
173
+ "step": 400
174
+ },
175
+ {
176
+ "epoch": 3.3057851239669422,
177
+ "eval_character_accuracy": 0.07429075460829493,
178
+ "eval_character_f1": 0.20473840194926402,
179
+ "eval_character_normalized_edit_distance": 0.9257092453917051,
180
+ "eval_character_precision": -0.2708513530211294,
181
+ "eval_character_recall": 0.07429075460829493,
182
+ "eval_loss": 7.459848403930664,
183
+ "eval_runtime": 103.5362,
184
+ "eval_samples_per_second": 0.811,
185
+ "eval_steps_per_second": 0.811,
186
+ "eval_token_accuracy": -0.06081938834391232,
187
+ "eval_token_f1": -0.10298860492843001,
188
+ "eval_token_precision": -0.33585234704258093,
189
+ "eval_token_recall": -0.06081938834391232,
190
+ "step": 400
191
+ },
192
+ {
193
+ "epoch": 3.512396694214876,
194
+ "grad_norm": 2.3767194747924805,
195
+ "learning_rate": 8.484836362668853e-05,
196
+ "loss": 7.306,
197
+ "step": 425
198
+ },
199
+ {
200
+ "epoch": 3.71900826446281,
201
+ "grad_norm": 1.6498326063156128,
202
+ "learning_rate": 8.27508620128865e-05,
203
+ "loss": 7.2923,
204
+ "step": 450
205
+ },
206
+ {
207
+ "epoch": 3.925619834710744,
208
+ "grad_norm": 2.3197038173675537,
209
+ "learning_rate": 8.054715256160118e-05,
210
+ "loss": 7.2434,
211
+ "step": 475
212
+ },
213
+ {
214
+ "epoch": 4.132231404958677,
215
+ "grad_norm": 1.6941279172897339,
216
+ "learning_rate": 7.824438168672448e-05,
217
+ "loss": 7.2663,
218
+ "step": 500
219
+ },
220
+ {
221
+ "epoch": 4.132231404958677,
222
+ "eval_character_accuracy": 0.0803571428571429,
223
+ "eval_character_f1": -0.005606033934108933,
224
+ "eval_character_normalized_edit_distance": 0.9196428571428571,
225
+ "eval_character_precision": -0.0027085377821394196,
226
+ "eval_character_recall": 0.0803571428571429,
227
+ "eval_loss": 7.411993980407715,
228
+ "eval_runtime": 111.766,
229
+ "eval_samples_per_second": 0.752,
230
+ "eval_steps_per_second": 0.752,
231
+ "eval_token_accuracy": -0.09045008655510678,
232
+ "eval_token_f1": -0.1193780410408293,
233
+ "eval_token_precision": -0.17551007713361533,
234
+ "eval_token_recall": -0.09045008655510678,
235
+ "step": 500
236
+ },
237
+ {
238
+ "epoch": 4.338842975206612,
239
+ "grad_norm": 1.876191258430481,
240
+ "learning_rate": 7.585001704864178e-05,
241
+ "loss": 7.0889,
242
+ "step": 525
243
+ },
244
+ {
245
+ "epoch": 4.545454545454545,
246
+ "grad_norm": 1.6166743040084839,
247
+ "learning_rate": 7.33718233373407e-05,
248
+ "loss": 7.3075,
249
+ "step": 550
250
+ },
251
+ {
252
+ "epoch": 4.75206611570248,
253
+ "grad_norm": 1.91400146484375,
254
+ "learning_rate": 7.081783709228203e-05,
255
+ "loss": 7.2123,
256
+ "step": 575
257
+ },
258
+ {
259
+ "epoch": 4.958677685950414,
260
+ "grad_norm": 1.6324752569198608,
261
+ "learning_rate": 6.819634064068953e-05,
262
+ "loss": 7.1274,
263
+ "step": 600
264
+ },
265
+ {
266
+ "epoch": 4.958677685950414,
267
+ "eval_character_accuracy": 0.05530853974654382,
268
+ "eval_character_f1": 0.11845309783810354,
269
+ "eval_character_normalized_edit_distance": 0.9446914602534562,
270
+ "eval_character_precision": -0.8360710224787895,
271
+ "eval_character_recall": 0.05530853974654382,
272
+ "eval_loss": 7.372379302978516,
273
+ "eval_runtime": 90.0436,
274
+ "eval_samples_per_second": 0.933,
275
+ "eval_steps_per_second": 0.933,
276
+ "eval_token_accuracy": -0.031130986728217014,
277
+ "eval_token_f1": -0.05777922361918248,
278
+ "eval_token_precision": -0.4012546559498138,
279
+ "eval_token_recall": -0.031130986728217014,
280
+ "step": 600
281
+ },
282
+ {
283
+ "epoch": 5.1652892561983474,
284
+ "grad_norm": 2.4898300170898438,
285
+ "learning_rate": 6.551583523877392e-05,
286
+ "loss": 7.0699,
287
+ "step": 625
288
+ },
289
+ {
290
+ "epoch": 5.371900826446281,
291
+ "grad_norm": 1.7138676643371582,
292
+ "learning_rate": 6.278501350299176e-05,
293
+ "loss": 7.2723,
294
+ "step": 650
295
+ },
296
+ {
297
+ "epoch": 5.578512396694215,
298
+ "grad_norm": 1.8407520055770874,
299
+ "learning_rate": 6.0012731220741726e-05,
300
+ "loss": 7.1266,
301
+ "step": 675
302
+ },
303
+ {
304
+ "epoch": 5.785123966942149,
305
+ "grad_norm": 2.2728636264801025,
306
+ "learning_rate": 5.720797863191355e-05,
307
+ "loss": 7.2768,
308
+ "step": 700
309
+ },
310
+ {
311
+ "epoch": 5.785123966942149,
312
+ "eval_character_accuracy": 0.09879032258064513,
313
+ "eval_character_f1": 0.8961915294782532,
314
+ "eval_character_normalized_edit_distance": 0.9012096774193549,
315
+ "eval_character_precision": -0.12673013300924985,
316
+ "eval_character_recall": 0.09879032258064513,
317
+ "eval_loss": 7.350462913513184,
318
+ "eval_runtime": 106.5093,
319
+ "eval_samples_per_second": 0.789,
320
+ "eval_steps_per_second": 0.789,
321
+ "eval_token_accuracy": -0.05937680323139061,
322
+ "eval_token_f1": -0.09617178203053282,
323
+ "eval_token_precision": -0.2528747398232505,
324
+ "eval_token_recall": -0.05937680323139061,
325
+ "step": 700
326
+ },
327
+ {
328
+ "epoch": 5.991735537190083,
329
+ "grad_norm": 1.8262277841567993,
330
+ "learning_rate": 5.437985127442065e-05,
331
+ "loss": 6.9795,
332
+ "step": 725
333
+ },
334
+ {
335
+ "epoch": 6.198347107438017,
336
+ "grad_norm": 1.4631235599517822,
337
+ "learning_rate": 5.153752048826161e-05,
338
+ "loss": 7.1821,
339
+ "step": 750
340
+ },
341
+ {
342
+ "epoch": 6.404958677685951,
343
+ "grad_norm": 1.6286855936050415,
344
+ "learning_rate": 4.869020367376291e-05,
345
+ "loss": 7.0595,
346
+ "step": 775
347
+ },
348
+ {
349
+ "epoch": 6.6115702479338845,
350
+ "grad_norm": 1.9363166093826294,
351
+ "learning_rate": 4.584713440045255e-05,
352
+ "loss": 7.2103,
353
+ "step": 800
354
+ },
355
+ {
356
+ "epoch": 6.6115702479338845,
357
+ "eval_character_accuracy": 0.0677473358294931,
358
+ "eval_character_f1": 0.1532992887720318,
359
+ "eval_character_normalized_edit_distance": 0.9322526641705069,
360
+ "eval_character_precision": -0.5833104038643799,
361
+ "eval_character_recall": 0.0677473358294931,
362
+ "eval_loss": 7.334532737731934,
363
+ "eval_runtime": 95.1572,
364
+ "eval_samples_per_second": 0.883,
365
+ "eval_steps_per_second": 0.883,
366
+ "eval_token_accuracy": -0.034304673975764644,
367
+ "eval_token_f1": -0.06345470090190465,
368
+ "eval_token_precision": -0.42229716326125777,
369
+ "eval_token_recall": -0.034304673975764644,
370
+ "step": 800
371
+ },
372
+ {
373
+ "epoch": 6.818181818181818,
374
+ "grad_norm": 1.6526739597320557,
375
+ "learning_rate": 4.3017532463498496e-05,
376
+ "loss": 7.1195,
377
+ "step": 825
378
+ },
379
+ {
380
+ "epoch": 7.024793388429752,
381
+ "grad_norm": 1.653976321220398,
382
+ "learning_rate": 4.0210573984816094e-05,
383
+ "loss": 7.0825,
384
+ "step": 850
385
+ },
386
+ {
387
+ "epoch": 7.231404958677686,
388
+ "grad_norm": 2.087712526321411,
389
+ "learning_rate": 3.743536165580348e-05,
390
+ "loss": 7.1623,
391
+ "step": 875
392
+ },
393
+ {
394
+ "epoch": 7.43801652892562,
395
+ "grad_norm": 1.5534496307373047,
396
+ "learning_rate": 3.470089521820502e-05,
397
+ "loss": 6.9941,
398
+ "step": 900
399
+ },
400
+ {
401
+ "epoch": 7.43801652892562,
402
+ "eval_character_accuracy": 0.11323624711981561,
403
+ "eval_character_f1": 1.5649350607278285,
404
+ "eval_character_normalized_edit_distance": 0.8867637528801844,
405
+ "eval_character_precision": -0.13239621166842896,
406
+ "eval_character_recall": 0.11323624711981561,
407
+ "eval_loss": 7.317984580993652,
408
+ "eval_runtime": 105.857,
409
+ "eval_samples_per_second": 0.794,
410
+ "eval_steps_per_second": 0.794,
411
+ "eval_token_accuracy": -0.06177149451817665,
412
+ "eval_token_f1": -0.09548750295568412,
413
+ "eval_token_precision": -0.21024072612470412,
414
+ "eval_token_recall": -0.06177149451817665,
415
+ "step": 900
416
+ },
417
+ {
418
+ "epoch": 7.644628099173554,
419
+ "grad_norm": 1.6190420389175415,
420
+ "learning_rate": 3.2016042278830407e-05,
421
+ "loss": 7.2007,
422
+ "step": 925
423
+ },
424
+ {
425
+ "epoch": 7.851239669421488,
426
+ "grad_norm": 1.648707389831543,
427
+ "learning_rate": 2.938950955277431e-05,
428
+ "loss": 6.9229,
429
+ "step": 950
430
+ },
431
+ {
432
+ "epoch": 8.057851239669422,
433
+ "grad_norm": 1.7598505020141602,
434
+ "learning_rate": 2.6829814628392212e-05,
435
+ "loss": 7.2179,
436
+ "step": 975
437
+ },
438
+ {
439
+ "epoch": 8.264462809917354,
440
+ "grad_norm": 2.277340888977051,
441
+ "learning_rate": 2.434525834559577e-05,
442
+ "loss": 7.0229,
443
+ "step": 1000
444
+ },
445
+ {
446
+ "epoch": 8.264462809917354,
447
+ "eval_character_accuracy": 0.09299395161290325,
448
+ "eval_character_f1": 0.25292225255001993,
449
+ "eval_character_normalized_edit_distance": 0.9070060483870968,
450
+ "eval_character_precision": -0.351392669876222,
451
+ "eval_character_recall": 0.09299395161290325,
452
+ "eval_loss": 7.307159423828125,
453
+ "eval_runtime": 100.321,
454
+ "eval_samples_per_second": 0.837,
455
+ "eval_steps_per_second": 0.837,
456
+ "eval_token_accuracy": -0.04324870167339867,
457
+ "eval_token_f1": -0.07628204331437426,
458
+ "eval_token_precision": -0.32295477828186736,
459
+ "eval_token_recall": -0.04324870167339867,
460
+ "step": 1000
461
+ },
462
+ {
463
+ "epoch": 8.47107438016529,
464
+ "grad_norm": 2.059797763824463,
465
+ "learning_rate": 2.194389787704185e-05,
466
+ "loss": 7.0023,
467
+ "step": 1025
468
+ },
469
+ {
470
+ "epoch": 8.677685950413224,
471
+ "grad_norm": 1.6797763109207153,
472
+ "learning_rate": 1.9633520599511095e-05,
473
+ "loss": 7.2138,
474
+ "step": 1050
475
+ },
476
+ {
477
+ "epoch": 8.884297520661157,
478
+ "grad_norm": 1.750408411026001,
479
+ "learning_rate": 1.7421618840207578e-05,
480
+ "loss": 7.1231,
481
+ "step": 1075
482
+ },
483
+ {
484
+ "epoch": 9.090909090909092,
485
+ "grad_norm": 1.7852625846862793,
486
+ "learning_rate": 1.531536557987568e-05,
487
+ "loss": 6.9288,
488
+ "step": 1100
489
+ },
490
+ {
491
+ "epoch": 9.090909090909092,
492
+ "eval_character_accuracy": 0.10609879032258063,
493
+ "eval_character_f1": 0.3049904648682638,
494
+ "eval_character_normalized_edit_distance": 0.8939012096774194,
495
+ "eval_character_precision": -0.34872414682836084,
496
+ "eval_character_recall": 0.10609879032258063,
497
+ "eval_loss": 7.303292274475098,
498
+ "eval_runtime": 100.1344,
499
+ "eval_samples_per_second": 0.839,
500
+ "eval_steps_per_second": 0.839,
501
+ "eval_token_accuracy": -0.04177726485862676,
502
+ "eval_token_f1": -0.07351575254813365,
503
+ "eval_token_precision": -0.3059423487287063,
504
+ "eval_token_recall": -0.04177726485862676,
505
+ "step": 1100
506
+ },
507
+ {
508
+ "epoch": 9.297520661157025,
509
+ "grad_norm": 1.6963069438934326,
510
+ "learning_rate": 1.3321591191526223e-05,
511
+ "loss": 7.0099,
512
+ "step": 1125
513
+ },
514
+ {
515
+ "epoch": 9.50413223140496,
516
+ "grad_norm": 1.5837774276733398,
517
+ "learning_rate": 1.1446761290205665e-05,
518
+ "loss": 7.1777,
519
+ "step": 1150
520
+ },
521
+ {
522
+ "epoch": 9.710743801652892,
523
+ "grad_norm": 1.754378318786621,
524
+ "learning_rate": 9.696955765639987e-06,
525
+ "loss": 7.0471,
526
+ "step": 1175
527
+ },
528
+ {
529
+ "epoch": 9.917355371900827,
530
+ "grad_norm": 1.571057677268982,
531
+ "learning_rate": 8.077849065747727e-06,
532
+ "loss": 7.1775,
533
+ "step": 1200
534
+ },
535
+ {
536
+ "epoch": 9.917355371900827,
537
+ "eval_character_accuracy": 0.10949200748847931,
538
+ "eval_character_f1": 0.46652560447082553,
539
+ "eval_character_normalized_edit_distance": 0.8905079925115207,
540
+ "eval_character_precision": -0.2063524964945438,
541
+ "eval_character_recall": 0.10949200748847931,
542
+ "eval_loss": 7.2955498695373535,
543
+ "eval_runtime": 102.9364,
544
+ "eval_samples_per_second": 0.816,
545
+ "eval_steps_per_second": 0.816,
546
+ "eval_token_accuracy": -0.05418349682631285,
547
+ "eval_token_f1": -0.08995278261650466,
548
+ "eval_token_precision": -0.26468450382472053,
549
+ "eval_token_recall": -0.05418349682631285,
550
+ "step": 1200
551
+ },
552
+ {
553
+ "epoch": 10.12396694214876,
554
+ "grad_norm": 1.3566683530807495,
555
+ "learning_rate": 6.594691794961333e-06,
556
+ "loss": 7.0361,
557
+ "step": 1225
558
+ },
559
+ {
560
+ "epoch": 10.330578512396695,
561
+ "grad_norm": 1.528793454170227,
562
+ "learning_rate": 5.2522936870311955e-06,
563
+ "loss": 6.9658,
564
+ "step": 1250
565
+ },
566
+ {
567
+ "epoch": 10.537190082644628,
568
+ "grad_norm": 2.3794727325439453,
569
+ "learning_rate": 4.055008007530242e-06,
570
+ "loss": 7.0981,
571
+ "step": 1275
572
+ },
573
+ {
574
+ "epoch": 10.743801652892563,
575
+ "grad_norm": 1.5067297220230103,
576
+ "learning_rate": 3.006717436640144e-06,
577
+ "loss": 6.9433,
578
+ "step": 1300
579
+ },
580
+ {
581
+ "epoch": 10.743801652892563,
582
+ "eval_character_accuracy": 0.10598178283410142,
583
+ "eval_character_f1": 0.35053087566216945,
584
+ "eval_character_normalized_edit_distance": 0.8940182171658986,
585
+ "eval_character_precision": -0.2680999374433479,
586
+ "eval_character_recall": 0.10598178283410142,
587
+ "eval_loss": 7.292919158935547,
588
+ "eval_runtime": 101.3873,
589
+ "eval_samples_per_second": 0.829,
590
+ "eval_steps_per_second": 0.829,
591
+ "eval_token_accuracy": -0.04930755914598972,
592
+ "eval_token_f1": -0.08342496106391631,
593
+ "eval_token_precision": -0.27079911946608903,
594
+ "eval_token_recall": -0.04930755914598972,
595
+ "step": 1300
596
+ },
597
+ {
598
+ "epoch": 10.950413223140496,
599
+ "grad_norm": 1.3775362968444824,
600
+ "learning_rate": 2.110821477999747e-06,
601
+ "loss": 7.1653,
602
+ "step": 1325
603
+ },
604
+ {
605
+ "epoch": 11.15702479338843,
606
+ "grad_norm": 1.7906850576400757,
607
+ "learning_rate": 1.3702254344477916e-06,
608
+ "loss": 7.1161,
609
+ "step": 1350
610
+ },
611
+ {
612
+ "epoch": 11.363636363636363,
613
+ "grad_norm": 1.9360337257385254,
614
+ "learning_rate": 7.873309864103895e-07,
615
+ "loss": 6.9461,
616
+ "step": 1375
617
+ },
618
+ {
619
+ "epoch": 11.570247933884298,
620
+ "grad_norm": 1.425911784172058,
621
+ "learning_rate": 3.640284034866215e-07,
622
+ "loss": 7.0805,
623
+ "step": 1400
624
+ },
625
+ {
626
+ "epoch": 11.570247933884298,
627
+ "eval_character_accuracy": 0.09935735887096775,
628
+ "eval_character_f1": 0.2854130741493807,
629
+ "eval_character_normalized_edit_distance": 0.9006426411290323,
630
+ "eval_character_precision": -0.32708681467335077,
631
+ "eval_character_recall": 0.09935735887096775,
632
+ "eval_loss": 7.2913360595703125,
633
+ "eval_runtime": 100.3688,
634
+ "eval_samples_per_second": 0.837,
635
+ "eval_steps_per_second": 0.837,
636
+ "eval_token_accuracy": -0.04555683785343345,
637
+ "eval_token_f1": -0.0787525166168733,
638
+ "eval_token_precision": -0.2902410367785808,
639
+ "eval_token_recall": -0.04555683785343345,
640
+ "step": 1400
641
+ },
642
+ {
643
+ "epoch": 11.776859504132231,
644
+ "grad_norm": 1.6527899503707886,
645
+ "learning_rate": 1.0169041448943039e-07,
646
+ "loss": 7.0609,
647
+ "step": 1425
648
+ },
649
+ {
650
+ "epoch": 11.983471074380166,
651
+ "grad_norm": 1.945165991783142,
652
+ "learning_rate": 1.1677558205025475e-09,
653
+ "loss": 7.1057,
654
+ "step": 1450
655
+ }
656
+ ],
657
+ "logging_steps": 25,
658
+ "max_steps": 1452,
659
+ "num_input_tokens_seen": 0,
660
+ "num_train_epochs": 12,
661
+ "save_steps": 100,
662
+ "stateful_callbacks": {
663
+ "TrainerControl": {
664
+ "args": {
665
+ "should_epoch_stop": false,
666
+ "should_evaluate": false,
667
+ "should_log": false,
668
+ "should_save": true,
669
+ "should_training_stop": true
670
+ },
671
+ "attributes": {}
672
+ }
673
+ },
674
+ "total_flos": 8.739058770115559e+18,
675
+ "train_batch_size": 1,
676
+ "trial_name": null,
677
+ "trial_params": null
678
+ }
training_args.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:7dd9568750889baa8fd79088a719f23db4b7376484d1231cdbaa06c0e3d2b201
3
+ size 5969
vocab.json ADDED
The diff for this file is too large to render. See raw diff