Instructions to use scrallex/structural-manifold-compression with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use scrallex/structural-manifold-compression with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="scrallex/structural-manifold-compression")

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("scrallex/structural-manifold-compression")
model = AutoModelForCausalLM.from_pretrained("scrallex/structural-manifold-compression", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use scrallex/structural-manifold-compression with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "scrallex/structural-manifold-compression"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "scrallex/structural-manifold-compression",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/scrallex/structural-manifold-compression

SGLang

How to use scrallex/structural-manifold-compression with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "scrallex/structural-manifold-compression" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "scrallex/structural-manifold-compression",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "scrallex/structural-manifold-compression" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "scrallex/structural-manifold-compression",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use scrallex/structural-manifold-compression with Docker Model Runner:
```
docker model run hf.co/scrallex/structural-manifold-compression
```

scrallex commited on Nov 6, 2025

Commit

4ed5d90

1 Parent(s): 250c57a

Publish manifold GPT weights

Browse files

Files changed (17) hide show

README.md +96 -76
adapter_config.json +0 -39
adapter_model.safetensors +0 -3
config.json +33 -0
eval_metrics.json +5 -0
generation_config.json +7 -0
merges.txt +0 -0
rng_state.pth → model.safetensors +2 -2
optimizer.pt +0 -3
scaler.pt +0 -3
scheduler.pt +0 -3
special_tokens_map.json +5 -50
tokenizer.json +0 -0
tokenizer_config.json +8 -56
trainer_state.json +36 -638
training_args.bin +2 -2
vocab.json +0 -0

README.md CHANGED Viewed

@@ -1,97 +1,117 @@
 ---
 license: mit
-library_name: peft
-base_model: microsoft/trocr-base-printed
 tags:
-- ocr
-- trocr
-- lora
-- transformers
-- fox
-- omnidoc
 ---
-# FOX-Manifold-OCR (fox_stage1 LoRA)
 ## TL;DR
-- 12-epoch LoRA adapter fine-tune of `microsoft/trocr-base-printed` on the Fox (EN + CN) plus OmniDoc structural-manifold manifests using a single RTX 3080 Ti.
-- Exported from `output/training_runs/fox_stage1/checkpoint-1452` on 2024-11-04 and published to `scrallex/structural-manifold-compression` (private until we flip the repo visibility).
-- Sustains 44x-46x compression while holding 0.87-0.93 F1 on manifold benchmarks (table below).
-## Model Details
-| Field | Value |
-|-------|-------|
-| Base model | `microsoft/trocr-base-printed` (VisionEncoderDecoder) |
-| Adapter | PEFT LoRA, rank 8, alpha 16, dropout 0.05 on `q_proj,k_proj,v_proj,out_proj` |
-| Precision | fp16 + gradient checkpointing |
-| Sequence lengths | `max_target_length=640`, `generation_max_length=768` |
-| Hardware/time | RTX 3080 Ti (12 GB) · ~45 min for 12 epochs |
-| Trainer stats | Final train loss 7.11, final eval loss 7.29 |
-| Files exported | `adapter_model.safetensors`, adapter config, tokenizer assets, optimizer + scaler states, `trainer_state.json`, `training_args.bin`, README |
-License inherits MIT from the base model. Downstream usage must continue to respect the Fox and OmniDoc dataset terms.
-## Training Command
 ```
-cd structural-manifold-compression
-source .venv/bin/activate
-RUN_NAME=fox_stage1 \
-TRAIN_DATASETS="fox_en=data/benchmark_corpus/fox/metadata/text_manifest.jsonl:data/benchmark_corpus/fox/raw,fox_cn=data/benchmark_corpus/fox/metadata/text_manifest.jsonl:data/benchmark_corpus/fox/raw,omnidoc=data/benchmark_corpus/omnidocbench/metadata/text_manifest.jsonl:data/benchmark_corpus/omnidocbench/raw/OmniDocBench" \
-PRECISION=fp16 \
-LORA_RANK=8 \
-USE_GRADIENT_CHECKPOINTING=1 \
-./scripts/training/run_pipeline.py train
 ```
-Key hyperparameters: epochs=12, per-device batch size=1, gradient accumulation=8 (effective batch 8), eval split=0.08, learning rate=1e-4 with cosine scheduler, weight decay=0.01, warmup ratio=0.05, save/eval steps=200, LoRA dropout=0.05, default metric tokenizer.
-## Data Summary
-- **Fox EN + CN**: analyst notes + bilingual research PDFs (`data/benchmark_corpus/fox`).
-- **OmniDocBench**: invoices, contracts, infographics, dense tables (`data/benchmark_corpus/omnidocbench`).
-- Datasets fed directly through the structural-manifold manifests; no extra filtering beyond the default 8% validation split.
-## Metrics (benchmark_eval.py · window=512 · stride=384 · precision=3 · tokenizer=`external/DeepSeek-OCR/weights`)
-| Benchmark | Compression ratio | Character F1 | Token F1 |
-|-----------|-------------------|--------------|----------|
-| Fox EN | 44.29 | 0.9258 | 0.9146 |
-| Fox CN | 44.68 | 0.9062 | 0.9029 |
-| OmniDoc | 45.93 | 0.8913 | 0.8682 |
-Reproduce via:
-```
-python scripts/experiments/benchmark_eval.py \
-  --dataset fox_en=data/benchmark_corpus/fox/text/en_page_ocr \
-  --dataset fox_cn=data/benchmark_corpus/fox/text/cn_page_ocr \
-  --dataset omnidoc=data/benchmark_corpus/omnidocbench/text \
-  --tokenizer external/DeepSeek-OCR/weights \
-  --tokenizer-trust-remote-code \
-  --output-dir output/benchmark_runs/full
-```
-Results land in `output/benchmark_runs/full/summary.json` and `summary.csv`.
 ## Usage
 ```python
-from peft import PeftModel
-from PIL import Image
-from transformers import AutoTokenizer, TrOCRProcessor, VisionEncoderDecoderModel
 repo_id = "scrallex/structural-manifold-compression"
 tokenizer = AutoTokenizer.from_pretrained(repo_id)
-processor = TrOCRProcessor.from_pretrained("microsoft/trocr-base-printed")
-base = VisionEncoderDecoderModel.from_pretrained("microsoft/trocr-base-printed")
-model = PeftModel.from_pretrained(base, repo_id).eval()
-image = Image.open("sample_page.png").convert("RGB")
-inputs = processor(images=image, return_tensors="pt").to(model.device)
-outputs = model.generate(**inputs, max_new_tokens=640)
-text = tokenizer.batch_decode(outputs, skip_special_tokens=True)[0]
-print(text)
 ```
-Call `model = model.merge_and_unload()` if you need a standalone `pytorch_model.bin`.
-## Limitations & Responsible Use
-- Tuned for bilingual business PDFs; handwriting, math-heavy, or ultra-low-res scans are out-of-domain.
-- Compression-first tuning biases toward recall; keep downstream kill switches for elevated false positives.
-- Source corpora may contain sensitive enterprise text—scrub outputs before redistribution.
-## Changelog
-- 2024-11-04 · fox_stage1 checkpoint-1452 exported and published here.

 ---
 license: mit
+base_model: scratch
+library_name: transformers
+pipeline_tag: text-generation
 tags:
+  - gpt2
+  - manifold
+default_dtype: float16
 ---
+# Structural Manifold GPT (300M · 512B windows)
 ## TL;DR
+- Decoder-only transformer (16 layers · 16 heads · 1024-dim) trained **from scratch** on structural manifold signatures extracted from WikiText-103 (raw) using 512 B windows (384 B stride, precision 3).
+- Each "token" is a quantised `(coherence, stability, entropy, hazard)` signature with deduplicated prototypes → **~42× byte compression** and 512-signature context ≈ 20k raw tokens.
+- Fits/trains on a single RTX 3080 Ti in <30 min with FP16 + gradient checkpointing.
 ```
+Final metrics (eval split = 2%)
+-------------------------------------------
+  eval_loss      = 6.6456
+  perplexity     = 7.69e+02 (on manifold tokens)
+  samples        = 76 (3 766 total sequences · 3 epochs)
+  training time  ≈ 25 min @ 1.3 s/step (GPU)
 ```
+Use this repo if you want to **benchmark manifold LMs** or integrate the encoder/decoder stack into small-context devices. For full pipeline (dataset prep, compression scripts, benchmarking), clone https://github.com/SepDynamics/structural-manifold-compression.
+---
+## Files
+| Path | Notes |
+|------|-------|
+| `model.safetensors` | 300 M parameter GPT2LMHeadModel trained on manifold signatures |
+| `config.json` | Model architecture (n_layer=16, n_embd=1024, vocab_size=11839, pad/bos/eos id = 11838) |
+| `generation_config.json` | Default sampling config (max_length=512) |
+| `tokenizer.json` / `tokenizer_config.json` / `special_tokens_map.json` | Word-level tokenizer whose vocab is exactly the manifold signature strings + `<pad>` |
+| `vocab.json` | Original signature list emitted by `scripts/data/prepare_causal_dataset.py` |
+| `training_args.bin` / `trainer_state.json` | Hugging Face Trainer metadata (seeds, LR schedule, grad norms) |
+| `eval_metrics.json` | Recomputed eval loss & perplexity over the 2% hold-out |
+---
+## Data & Compression Pipeline
+1. **Raw text → signatures**: run the encoder on WikiText (or any UTF-8 corpus) via
+   ```bash
+   python scripts/data/prepare_causal_dataset.py \
+     --text-root data/raw_text/wikitext_train.jsonl \
+     --output-dir output/wikitext_manifold \
+     --window-bytes 512 --stride-bytes 384 --precision 3 \
+     --sequence-length 512 --min-sequence-length 8 \
+     --use-native --concat-documents --export-signatures --reset-output
+   ```
+   This keeps an append-only `samples.jsonl` + `vocab.json` so you can resume mid-run.
+2. **Sequences → HF dataset**: the builder automatically materialises `output/wikitext_manifold/hf_dataset` with `input_ids`/`labels` for causal LM.
+3. **Training**: the published checkpoint comes from
+   ```bash
+   CUDA_VISIBLE_DEVICES=0 python scripts/training/manifold_lm_trainer.py \
+     --dataset-path output/wikitext_manifold/hf_dataset \
+     --vocab-path output/wikitext_manifold/vocab.json \
+     --output-dir output/training_runs/wikitext_manifold_gpt \
+     --n-layer 16 --n-head 16 --n-embd 1024 --context-length 512 \
+     --per-device-train-batch-size 2 --per-device-eval-batch-size 2 \
+     --gradient-accumulation-steps 16 --num-train-epochs 3 \
+     --learning-rate 2e-4 --warmup-steps 500 --gradient-checkpointing --fp16 --resume
+   ```
+Hardware: single RTX 3080 Ti (12 GB). Training logs: `output/training_runs/wikitext_manifold_gpt/train.log` in the main repo.
+---
 ## Usage
+> ⚠️ This model expects **manifold signatures** (not raw text). Before inference, run the encoder to obtain the signature vocabulary and ID sequences.
 ```python
+import json
+import torch
+from transformers import AutoTokenizer, GPT2LMHeadModel
 repo_id = "scrallex/structural-manifold-compression"
 tokenizer = AutoTokenizer.from_pretrained(repo_id)
+model = GPT2LMHeadModel.from_pretrained(repo_id, torch_dtype=torch.float16).to("cuda").eval()
+def manifold_signatures_to_ids(signatures):
+    # signatures = list of strings emitted by the encoder (e.g. 'c0.018_s0.481_e0.982')
+    return tokenizer.convert_tokens_to_ids(signatures)
+signatures = ["c0.018_s0.481_e0.982", "c0.012_s0.496_e0.988", "c0.017_s0.502_e0.972", ...]
+input_ids = manifold_signatures_to_ids(signatures)
+inputs = torch.tensor([input_ids[:-1]], device=model.device)
+with torch.no_grad():
+    outputs = model.generate(inputs, max_new_tokens=64)
+next_signature_ids = outputs[0, len(input_ids)-1:]
+next_signatures = tokenizer.convert_ids_to_tokens(next_signature_ids.tolist())
 ```
+To reconstruct human-readable text, feed predicted signatures back through the manifold decoder (see `scripts/experiments/manifold_compression_eval.py`).
+---
+## Evaluation
+- Training split: 3 690 sequences (98% of dataset) · Eval: 76 sequences (2%).
+- Final eval loss `6.6456` → perplexity `≈7.7e2` on manifold tokens (see `eval_metrics.json`).
+- Compression verified with `benchmark_eval.py --dataset wikitext=... --window-bytes 512 --stride-bytes 384 --precision 3` (token accuracy ≈95%).
+Future work: increase sequence budget (1k+ signatures), add rotary embeddings for better long-context, benchmark against GPT-2 (raw) to quantify effective perplexity after reconstruction.
+---
+## Responsible Use & Limitations
+- The model memorises WikiText-103 content; outputs may regurgitate training passages.
+- Tokens are structural signatures only—**you must keep the encoder/decoder kill switches** to avoid leaking the underlying text when using proprietary corpora.
+- No guardrails, toxicity filtering, or multilingual tuning beyond what WikiText provides.
+Report issues or ideas via https://github.com/SepDynamics/structural-manifold-compression.

adapter_config.json DELETED Viewed

@@ -1,39 +0,0 @@
-{
-  "alpha_pattern": {},
-  "auto_mapping": null,
-  "base_model_name_or_path": "microsoft/trocr-base-printed",
-  "bias": "none",
-  "corda_config": null,
-  "eva_config": null,
-  "exclude_modules": null,
-  "fan_in_fan_out": false,
-  "inference_mode": true,
-  "init_lora_weights": true,
-  "layer_replication": null,
-  "layers_pattern": null,
-  "layers_to_transform": null,
-  "loftq_config": {},
-  "lora_alpha": 16,
-  "lora_bias": false,
-  "lora_dropout": 0.05,
-  "megatron_config": null,
-  "megatron_core": "megatron.core",
-  "modules_to_save": null,
-  "peft_type": "LORA",
-  "qalora_group_size": 16,
-  "r": 8,
-  "rank_pattern": {},
-  "revision": null,
-  "target_modules": [
-    "q_proj",
-    "out_proj",
-    "k_proj",
-    "v_proj"
-  ],
-  "target_parameters": null,
-  "task_type": "SEQ_2_SEQ_LM",
-  "trainable_token_indices": null,
-  "use_dora": false,
-  "use_qalora": false,
-  "use_rslora": false
-}

adapter_model.safetensors DELETED Viewed

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:b68d68d64ed8ff74766304f84a274742b6dcc2ba330dc451bb38f13924b7e7da
-size 6123432

config.json ADDED Viewed

	@@ -0,0 +1,33 @@

+{
+  "activation_function": "gelu_new",
+  "architectures": [
+    "GPT2LMHeadModel"
+  ],
+  "attn_pdrop": 0.1,
+  "bos_token_id": 11838,
+  "dtype": "float32",
+  "embd_pdrop": 0.1,
+  "eos_token_id": 11838,
+  "initializer_range": 0.02,
+  "layer_norm_epsilon": 1e-05,
+  "model_type": "gpt2",
+  "n_ctx": 512,
+  "n_embd": 1024,
+  "n_head": 16,
+  "n_inner": 4096,
+  "n_layer": 16,
+  "n_positions": 512,
+  "pad_token_id": 11838,
+  "reorder_and_upcast_attn": false,
+  "resid_pdrop": 0.1,
+  "scale_attn_by_inverse_layer_idx": false,
+  "scale_attn_weights": true,
+  "summary_activation": null,
+  "summary_first_dropout": 0.1,
+  "summary_proj_to_labels": true,
+  "summary_type": "cls_index",
+  "summary_use_proj": true,
+  "transformers_version": "4.57.1",
+  "use_cache": true,
+  "vocab_size": 11839
+}

eval_metrics.json ADDED Viewed

	@@ -0,0 +1,5 @@

+{
+  "eval_loss": 6.64558858620493,
+  "perplexity": 769.3827624683506,
+  "eval_samples": 76
+}

generation_config.json ADDED Viewed

	@@ -0,0 +1,7 @@

+{
+  "_from_model_config": true,
+  "bos_token_id": 11838,
+  "eos_token_id": 11838,
+  "pad_token_id": 11838,
+  "transformers_version": "4.57.1"
+}

merges.txt DELETED Viewed

The diff for this file is too large to render. See raw diff

rng_state.pth → model.safetensors RENAMED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:c2f7627e3d63868316767e0582fef1204e0a436989c08b511b1cd5db6c868336
-size 14709

 version https://git-lfs.github.com/spec/v1
+oid sha256:8c577203118d46a8dbeb2c0082f472a84fcb918ec89e3f6c510fb1d08c1aa46e
+size 856776376

optimizer.pt DELETED Viewed

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:96d0c6ed52734ed81836a4850362b70fb3c4e9db4faff35465267ce69eb20c9a
-size 12354059

scaler.pt DELETED Viewed

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:5b2d7bfd8833db8e49fa5b7ac95dac2f576f54a5d91de14810458e93903b743f
-size 1383

scheduler.pt DELETED Viewed

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:f62afa13d3e7a876a91753e3bb8656e1fd4be0d8fa46ee51fc12de2c2b441328
-size 1465

special_tokens_map.json CHANGED Viewed

@@ -1,51 +1,6 @@
 {
-  "bos_token": {
-    "content": "<s>",
-    "lstrip": false,
-    "normalized": true,
-    "rstrip": false,
-    "single_word": false
-  },
-  "cls_token": {
-    "content": "<s>",
-    "lstrip": false,
-    "normalized": true,
-    "rstrip": false,
-    "single_word": false
-  },
-  "eos_token": {
-    "content": "</s>",
-    "lstrip": false,
-    "normalized": true,
-    "rstrip": false,
-    "single_word": false
-  },
-  "mask_token": {
-    "content": "<mask>",
-    "lstrip": true,
-    "normalized": true,
-    "rstrip": false,
-    "single_word": false
-  },
-  "pad_token": {
-    "content": "<pad>",
-    "lstrip": false,
-    "normalized": true,
-    "rstrip": false,
-    "single_word": false
-  },
-  "sep_token": {
-    "content": "</s>",
-    "lstrip": false,
-    "normalized": true,
-    "rstrip": false,
-    "single_word": false
-  },
-  "unk_token": {
-    "content": "<unk>",
-    "lstrip": false,
-    "normalized": true,
-    "rstrip": false,
-    "single_word": false
-  }
-}

 {
+  "bos_token": "<pad>",
+  "eos_token": "<pad>",
+  "unk_token": "<pad>",
+  "pad_token": "<pad>"
+}

tokenizer.json CHANGED Viewed

The diff for this file is too large to render. See raw diff

tokenizer_config.json CHANGED Viewed

@@ -1,58 +1,10 @@
 {
-  "add_prefix_space": false,
-  "added_tokens_decoder": {
-    "0": {
-      "content": "<s>",
-      "lstrip": false,
-      "normalized": true,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "1": {
-      "content": "<pad>",
-      "lstrip": false,
-      "normalized": true,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "2": {
-      "content": "</s>",
-      "lstrip": false,
-      "normalized": true,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "3": {
-      "content": "<unk>",
-      "lstrip": false,
-      "normalized": true,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "50264": {
-      "content": "<mask>",
-      "lstrip": true,
-      "normalized": true,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    }
-  },
-  "bos_token": "<s>",
-  "clean_up_tokenization_spaces": false,
-  "cls_token": "<s>",
-  "eos_token": "</s>",
-  "errors": "replace",
-  "extra_special_tokens": {},
-  "mask_token": "<mask>",
-  "model_max_length": 512,
   "pad_token": "<pad>",
-  "sep_token": "</s>",
-  "tokenizer_class": "RobertaTokenizer",
-  "trim_offsets": true,
-  "unk_token": "<unk>"
-}

 {
+  "tokenizer_class": "PreTrainedTokenizerFast",
+  "bos_token": "<pad>",
+  "eos_token": "<pad>",
+  "unk_token": "<pad>",
   "pad_token": "<pad>",
+  "model_max_length": 512,
+  "padding_side": "left",
+  "truncation_side": "right"
+}

trainer_state.json CHANGED Viewed

@@ -1,664 +1,62 @@
 {
-  "best_global_step": 600,
-  "best_metric": 0.05530853974654382,
-  "best_model_checkpoint": "/sep/structural-manifold-compression/output/training_runs/fox_stage1/checkpoint-600",
-  "epoch": 12.0,
-  "eval_steps": 100,
-  "global_step": 1452,
   "is_hyper_param_search": false,
   "is_local_process_zero": true,
   "is_world_process_zero": true,
   "log_history": [
     {
-      "epoch": 0.2066115702479339,
-      "grad_norm": 5.947324275970459,
-      "learning_rate": 9.994788203922447e-05,
-      "loss": 11.9365,
-      "step": 25
-    },
-    {
-      "epoch": 0.4132231404958678,
-      "grad_norm": 4.52055549621582,
-      "learning_rate": 9.813513645245729e-05,
-      "loss": 9.574,
       "step": 50
     },
     {
-      "epoch": 0.6198347107438017,
-      "grad_norm": 2.9425737857818604,
-      "learning_rate": 9.38241303448241e-05,
-      "loss": 8.2893,
-      "step": 75
-    },
-    {
-      "epoch": 0.8264462809917356,
-      "grad_norm": 1.500144362449646,
-      "learning_rate": 8.72386091371891e-05,
-      "loss": 7.881,
       "step": 100
     },
     {
-      "epoch": 0.8264462809917356,
-      "eval_character_accuracy": 0.07699992799539168,
-      "eval_character_f1": 0.26155196367780104,
-      "eval_character_normalized_edit_distance": 0.9230000720046083,
-      "eval_character_precision": -0.18725325615050648,
-      "eval_character_recall": 0.07699992799539168,
-      "eval_loss": 7.875946998596191,
-      "eval_runtime": 105.885,
-      "eval_samples_per_second": 0.793,
-      "eval_steps_per_second": 0.793,
-      "eval_token_accuracy": -0.042498557414887506,
-      "eval_token_f1": -0.07808865152697282,
-      "eval_token_precision": -0.4803752867912161,
-      "eval_token_recall": -0.042498557414887506,
-      "step": 100
-    },
-    {
-      "epoch": 1.0330578512396693,
-      "grad_norm": 1.8186473846435547,
-      "learning_rate": 7.872036780210026e-05,
-      "loss": 7.492,
-      "step": 125
-    },
-    {
-      "epoch": 1.2396694214876034,
-      "grad_norm": 2.1581435203552246,
-      "learning_rate": 6.871151137141266e-05,
-      "loss": 7.6082,
       "step": 150
     },
     {
-      "epoch": 1.4462809917355373,
-      "grad_norm": 2.3698322772979736,
-      "learning_rate": 5.773150925401641e-05,
-      "loss": 7.3798,
-      "step": 175
-    },
-    {
-      "epoch": 1.6528925619834711,
-      "grad_norm": 2.196545362472534,
-      "learning_rate": 4.6350234266954626e-05,
-      "loss": 7.4732,
-      "step": 200
-    },
-    {
-      "epoch": 1.6528925619834711,
-      "eval_character_accuracy": 0.06128492223502302,
-      "eval_character_f1": 0.1641450344783411,
-      "eval_character_normalized_edit_distance": 0.938715077764977,
-      "eval_character_precision": -0.2419619891397542,
-      "eval_character_recall": 0.06128492223502302,
-      "eval_loss": 7.591057300567627,
-      "eval_runtime": 104.8701,
-      "eval_samples_per_second": 0.801,
-      "eval_steps_per_second": 0.801,
-      "eval_token_accuracy": -0.06878245816503181,
-      "eval_token_f1": -0.11809033105597452,
-      "eval_token_precision": -0.41708427374622237,
-      "eval_token_recall": -0.06878245816503181,
       "step": 200
     },
     {
-      "epoch": 1.859504132231405,
-      "grad_norm": 3.392272710800171,
-      "learning_rate": 3.515838568050736e-05,
-      "loss": 7.3805,
-      "step": 225
-    },
-    {
-      "epoch": 2.0661157024793386,
-      "grad_norm": 1.8100775480270386,
-      "learning_rate": 2.4736831350043536e-05,
-      "loss": 7.4541,
       "step": 250
     },
     {
-      "epoch": 2.2727272727272725,
-      "grad_norm": 1.6662075519561768,
-      "learning_rate": 1.562646010789411e-05,
-      "loss": 7.2798,
-      "step": 275
-    },
-    {
-      "epoch": 2.479338842975207,
-      "grad_norm": 1.5955466032028198,
-      "learning_rate": 8.30010910550611e-06,
-      "loss": 7.4174,
       "step": 300
-    },
-    {
-      "epoch": 2.479338842975207,
-      "eval_character_accuracy": 0.05942180299539168,
-      "eval_character_f1": 0.13834188776330955,
-      "eval_character_normalized_edit_distance": 0.9405781970046083,
-      "eval_character_precision": -0.42160250306080815,
-      "eval_character_recall": 0.05942180299539168,
-      "eval_loss": 7.534529685974121,
-      "eval_runtime": 100.366,
-      "eval_samples_per_second": 0.837,
-      "eval_steps_per_second": 0.837,
-      "eval_token_accuracy": -0.04899019042123487,
-      "eval_token_f1": -0.0887424381203322,
-      "eval_token_precision": -0.47061440763661366,
-      "eval_token_recall": -0.04899019042123487,
-      "step": 300
-    },
-    {
-      "epoch": 2.6859504132231407,
-      "grad_norm": 1.5894019603729248,
-      "learning_rate": 9.204587302090488e-05,
-      "loss": 7.4469,
-      "step": 325
-    },
-    {
-      "epoch": 2.8925619834710745,
-      "grad_norm": 1.8111478090286255,
-      "learning_rate": 9.043745478918972e-05,
-      "loss": 7.3277,
-      "step": 350
-    },
-    {
-      "epoch": 3.0991735537190084,
-      "grad_norm": 2.056180000305176,
-      "learning_rate": 8.869790185253766e-05,
-      "loss": 7.3545,
-      "step": 375
-    },
-    {
-      "epoch": 3.3057851239669422,
-      "grad_norm": 1.8154643774032593,
-      "learning_rate": 8.68328554107289e-05,
-      "loss": 7.18,
-      "step": 400
-    },
-    {
-      "epoch": 3.3057851239669422,
-      "eval_character_accuracy": 0.07429075460829493,
-      "eval_character_f1": 0.20473840194926402,
-      "eval_character_normalized_edit_distance": 0.9257092453917051,
-      "eval_character_precision": -0.2708513530211294,
-      "eval_character_recall": 0.07429075460829493,
-      "eval_loss": 7.459848403930664,
-      "eval_runtime": 103.5362,
-      "eval_samples_per_second": 0.811,
-      "eval_steps_per_second": 0.811,
-      "eval_token_accuracy": -0.06081938834391232,
-      "eval_token_f1": -0.10298860492843001,
-      "eval_token_precision": -0.33585234704258093,
-      "eval_token_recall": -0.06081938834391232,
-      "step": 400
-    },
-    {
-      "epoch": 3.512396694214876,
-      "grad_norm": 2.3767194747924805,
-      "learning_rate": 8.484836362668853e-05,
-      "loss": 7.306,
-      "step": 425
-    },
-    {
-      "epoch": 3.71900826446281,
-      "grad_norm": 1.6498326063156128,
-      "learning_rate": 8.27508620128865e-05,
-      "loss": 7.2923,
-      "step": 450
-    },
-    {
-      "epoch": 3.925619834710744,
-      "grad_norm": 2.3197038173675537,
-      "learning_rate": 8.054715256160118e-05,
-      "loss": 7.2434,
-      "step": 475
-    },
-    {
-      "epoch": 4.132231404958677,
-      "grad_norm": 1.6941279172897339,
-      "learning_rate": 7.824438168672448e-05,
-      "loss": 7.2663,
-      "step": 500
-    },
-    {
-      "epoch": 4.132231404958677,
-      "eval_character_accuracy": 0.0803571428571429,
-      "eval_character_f1": -0.005606033934108933,
-      "eval_character_normalized_edit_distance": 0.9196428571428571,
-      "eval_character_precision": -0.0027085377821394196,
-      "eval_character_recall": 0.0803571428571429,
-      "eval_loss": 7.411993980407715,
-      "eval_runtime": 111.766,
-      "eval_samples_per_second": 0.752,
-      "eval_steps_per_second": 0.752,
-      "eval_token_accuracy": -0.09045008655510678,
-      "eval_token_f1": -0.1193780410408293,
-      "eval_token_precision": -0.17551007713361533,
-      "eval_token_recall": -0.09045008655510678,
-      "step": 500
-    },
-    {
-      "epoch": 4.338842975206612,
-      "grad_norm": 1.876191258430481,
-      "learning_rate": 7.585001704864178e-05,
-      "loss": 7.0889,
-      "step": 525
-    },
-    {
-      "epoch": 4.545454545454545,
-      "grad_norm": 1.6166743040084839,
-      "learning_rate": 7.33718233373407e-05,
-      "loss": 7.3075,
-      "step": 550
-    },
-    {
-      "epoch": 4.75206611570248,
-      "grad_norm": 1.91400146484375,
-      "learning_rate": 7.081783709228203e-05,
-      "loss": 7.2123,
-      "step": 575
-    },
-    {
-      "epoch": 4.958677685950414,
-      "grad_norm": 1.6324752569198608,
-      "learning_rate": 6.819634064068953e-05,
-      "loss": 7.1274,
-      "step": 600
-    },
-    {
-      "epoch": 4.958677685950414,
-      "eval_character_accuracy": 0.05530853974654382,
-      "eval_character_f1": 0.11845309783810354,
-      "eval_character_normalized_edit_distance": 0.9446914602534562,
-      "eval_character_precision": -0.8360710224787895,
-      "eval_character_recall": 0.05530853974654382,
-      "eval_loss": 7.372379302978516,
-      "eval_runtime": 90.0436,
-      "eval_samples_per_second": 0.933,
-      "eval_steps_per_second": 0.933,
-      "eval_token_accuracy": -0.031130986728217014,
-      "eval_token_f1": -0.05777922361918248,
-      "eval_token_precision": -0.4012546559498138,
-      "eval_token_recall": -0.031130986728217014,
-      "step": 600
-    },
-    {
-      "epoch": 5.1652892561983474,
-      "grad_norm": 2.4898300170898438,
-      "learning_rate": 6.551583523877392e-05,
-      "loss": 7.0699,
-      "step": 625
-    },
-    {
-      "epoch": 5.371900826446281,
-      "grad_norm": 1.7138676643371582,
-      "learning_rate": 6.278501350299176e-05,
-      "loss": 7.2723,
-      "step": 650
-    },
-    {
-      "epoch": 5.578512396694215,
-      "grad_norm": 1.8407520055770874,
-      "learning_rate": 6.0012731220741726e-05,
-      "loss": 7.1266,
-      "step": 675
-    },
-    {
-      "epoch": 5.785123966942149,
-      "grad_norm": 2.2728636264801025,
-      "learning_rate": 5.720797863191355e-05,
-      "loss": 7.2768,
-      "step": 700
-    },
-    {
-      "epoch": 5.785123966942149,
-      "eval_character_accuracy": 0.09879032258064513,
-      "eval_character_f1": 0.8961915294782532,
-      "eval_character_normalized_edit_distance": 0.9012096774193549,
-      "eval_character_precision": -0.12673013300924985,
-      "eval_character_recall": 0.09879032258064513,
-      "eval_loss": 7.350462913513184,
-      "eval_runtime": 106.5093,
-      "eval_samples_per_second": 0.789,
-      "eval_steps_per_second": 0.789,
-      "eval_token_accuracy": -0.05937680323139061,
-      "eval_token_f1": -0.09617178203053282,
-      "eval_token_precision": -0.2528747398232505,
-      "eval_token_recall": -0.05937680323139061,
-      "step": 700
-    },
-    {
-      "epoch": 5.991735537190083,
-      "grad_norm": 1.8262277841567993,
-      "learning_rate": 5.437985127442065e-05,
-      "loss": 6.9795,
-      "step": 725
-    },
-    {
-      "epoch": 6.198347107438017,
-      "grad_norm": 1.4631235599517822,
-      "learning_rate": 5.153752048826161e-05,
-      "loss": 7.1821,
-      "step": 750
-    },
-    {
-      "epoch": 6.404958677685951,
-      "grad_norm": 1.6286855936050415,
-      "learning_rate": 4.869020367376291e-05,
-      "loss": 7.0595,
-      "step": 775
-    },
-    {
-      "epoch": 6.6115702479338845,
-      "grad_norm": 1.9363166093826294,
-      "learning_rate": 4.584713440045255e-05,
-      "loss": 7.2103,
-      "step": 800
-    },
-    {
-      "epoch": 6.6115702479338845,
-      "eval_character_accuracy": 0.0677473358294931,
-      "eval_character_f1": 0.1532992887720318,
-      "eval_character_normalized_edit_distance": 0.9322526641705069,
-      "eval_character_precision": -0.5833104038643799,
-      "eval_character_recall": 0.0677473358294931,
-      "eval_loss": 7.334532737731934,
-      "eval_runtime": 95.1572,
-      "eval_samples_per_second": 0.883,
-      "eval_steps_per_second": 0.883,
-      "eval_token_accuracy": -0.034304673975764644,
-      "eval_token_f1": -0.06345470090190465,
-      "eval_token_precision": -0.42229716326125777,
-      "eval_token_recall": -0.034304673975764644,
-      "step": 800
-    },
-    {
-      "epoch": 6.818181818181818,
-      "grad_norm": 1.6526739597320557,
-      "learning_rate": 4.3017532463498496e-05,
-      "loss": 7.1195,
-      "step": 825
-    },
-    {
-      "epoch": 7.024793388429752,
-      "grad_norm": 1.653976321220398,
-      "learning_rate": 4.0210573984816094e-05,
-      "loss": 7.0825,
-      "step": 850
-    },
-    {
-      "epoch": 7.231404958677686,
-      "grad_norm": 2.087712526321411,
-      "learning_rate": 3.743536165580348e-05,
-      "loss": 7.1623,
-      "step": 875
-    },
-    {
-      "epoch": 7.43801652892562,
-      "grad_norm": 1.5534496307373047,
-      "learning_rate": 3.470089521820502e-05,
-      "loss": 6.9941,
-      "step": 900
-    },
-    {
-      "epoch": 7.43801652892562,
-      "eval_character_accuracy": 0.11323624711981561,
-      "eval_character_f1": 1.5649350607278285,
-      "eval_character_normalized_edit_distance": 0.8867637528801844,
-      "eval_character_precision": -0.13239621166842896,
-      "eval_character_recall": 0.11323624711981561,
-      "eval_loss": 7.317984580993652,
-      "eval_runtime": 105.857,
-      "eval_samples_per_second": 0.794,
-      "eval_steps_per_second": 0.794,
-      "eval_token_accuracy": -0.06177149451817665,
-      "eval_token_f1": -0.09548750295568412,
-      "eval_token_precision": -0.21024072612470412,
-      "eval_token_recall": -0.06177149451817665,
-      "step": 900
-    },
-    {
-      "epoch": 7.644628099173554,
-      "grad_norm": 1.6190420389175415,
-      "learning_rate": 3.2016042278830407e-05,
-      "loss": 7.2007,
-      "step": 925
-    },
-    {
-      "epoch": 7.851239669421488,
-      "grad_norm": 1.648707389831543,
-      "learning_rate": 2.938950955277431e-05,
-      "loss": 6.9229,
-      "step": 950
-    },
-    {
-      "epoch": 8.057851239669422,
-      "grad_norm": 1.7598505020141602,
-      "learning_rate": 2.6829814628392212e-05,
-      "loss": 7.2179,
-      "step": 975
-    },
-    {
-      "epoch": 8.264462809917354,
-      "grad_norm": 2.277340888977051,
-      "learning_rate": 2.434525834559577e-05,
-      "loss": 7.0229,
-      "step": 1000
-    },
-    {
-      "epoch": 8.264462809917354,
-      "eval_character_accuracy": 0.09299395161290325,
-      "eval_character_f1": 0.25292225255001993,
-      "eval_character_normalized_edit_distance": 0.9070060483870968,
-      "eval_character_precision": -0.351392669876222,
-      "eval_character_recall": 0.09299395161290325,
-      "eval_loss": 7.307159423828125,
-      "eval_runtime": 100.321,
-      "eval_samples_per_second": 0.837,
-      "eval_steps_per_second": 0.837,
-      "eval_token_accuracy": -0.04324870167339867,
-      "eval_token_f1": -0.07628204331437426,
-      "eval_token_precision": -0.32295477828186736,
-      "eval_token_recall": -0.04324870167339867,
-      "step": 1000
-    },
-    {
-      "epoch": 8.47107438016529,
-      "grad_norm": 2.059797763824463,
-      "learning_rate": 2.194389787704185e-05,
-      "loss": 7.0023,
-      "step": 1025
-    },
-    {
-      "epoch": 8.677685950413224,
-      "grad_norm": 1.6797763109207153,
-      "learning_rate": 1.9633520599511095e-05,
-      "loss": 7.2138,
-      "step": 1050
-    },
-    {
-      "epoch": 8.884297520661157,
-      "grad_norm": 1.750408411026001,
-      "learning_rate": 1.7421618840207578e-05,
-      "loss": 7.1231,
-      "step": 1075
-    },
-    {
-      "epoch": 9.090909090909092,
-      "grad_norm": 1.7852625846862793,
-      "learning_rate": 1.531536557987568e-05,
-      "loss": 6.9288,
-      "step": 1100
-    },
-    {
-      "epoch": 9.090909090909092,
-      "eval_character_accuracy": 0.10609879032258063,
-      "eval_character_f1": 0.3049904648682638,
-      "eval_character_normalized_edit_distance": 0.8939012096774194,
-      "eval_character_precision": -0.34872414682836084,
-      "eval_character_recall": 0.10609879032258063,
-      "eval_loss": 7.303292274475098,
-      "eval_runtime": 100.1344,
-      "eval_samples_per_second": 0.839,
-      "eval_steps_per_second": 0.839,
-      "eval_token_accuracy": -0.04177726485862676,
-      "eval_token_f1": -0.07351575254813365,
-      "eval_token_precision": -0.3059423487287063,
-      "eval_token_recall": -0.04177726485862676,
-      "step": 1100
-    },
-    {
-      "epoch": 9.297520661157025,
-      "grad_norm": 1.6963069438934326,
-      "learning_rate": 1.3321591191526223e-05,
-      "loss": 7.0099,
-      "step": 1125
-    },
-    {
-      "epoch": 9.50413223140496,
-      "grad_norm": 1.5837774276733398,
-      "learning_rate": 1.1446761290205665e-05,
-      "loss": 7.1777,
-      "step": 1150
-    },
-    {
-      "epoch": 9.710743801652892,
-      "grad_norm": 1.754378318786621,
-      "learning_rate": 9.696955765639987e-06,
-      "loss": 7.0471,
-      "step": 1175
-    },
-    {
-      "epoch": 9.917355371900827,
-      "grad_norm": 1.571057677268982,
-      "learning_rate": 8.077849065747727e-06,
-      "loss": 7.1775,
-      "step": 1200
-    },
-    {
-      "epoch": 9.917355371900827,
-      "eval_character_accuracy": 0.10949200748847931,
-      "eval_character_f1": 0.46652560447082553,
-      "eval_character_normalized_edit_distance": 0.8905079925115207,
-      "eval_character_precision": -0.2063524964945438,
-      "eval_character_recall": 0.10949200748847931,
-      "eval_loss": 7.2955498695373535,
-      "eval_runtime": 102.9364,
-      "eval_samples_per_second": 0.816,
-      "eval_steps_per_second": 0.816,
-      "eval_token_accuracy": -0.05418349682631285,
-      "eval_token_f1": -0.08995278261650466,
-      "eval_token_precision": -0.26468450382472053,
-      "eval_token_recall": -0.05418349682631285,
-      "step": 1200
-    },
-    {
-      "epoch": 10.12396694214876,
-      "grad_norm": 1.3566683530807495,
-      "learning_rate": 6.594691794961333e-06,
-      "loss": 7.0361,
-      "step": 1225
-    },
-    {
-      "epoch": 10.330578512396695,
-      "grad_norm": 1.528793454170227,
-      "learning_rate": 5.2522936870311955e-06,
-      "loss": 6.9658,
-      "step": 1250
-    },
-    {
-      "epoch": 10.537190082644628,
-      "grad_norm": 2.3794727325439453,
-      "learning_rate": 4.055008007530242e-06,
-      "loss": 7.0981,
-      "step": 1275
-    },
-    {
-      "epoch": 10.743801652892563,
-      "grad_norm": 1.5067297220230103,
-      "learning_rate": 3.006717436640144e-06,
-      "loss": 6.9433,
-      "step": 1300
-    },
-    {
-      "epoch": 10.743801652892563,
-      "eval_character_accuracy": 0.10598178283410142,
-      "eval_character_f1": 0.35053087566216945,
-      "eval_character_normalized_edit_distance": 0.8940182171658986,
-      "eval_character_precision": -0.2680999374433479,
-      "eval_character_recall": 0.10598178283410142,
-      "eval_loss": 7.292919158935547,
-      "eval_runtime": 101.3873,
-      "eval_samples_per_second": 0.829,
-      "eval_steps_per_second": 0.829,
-      "eval_token_accuracy": -0.04930755914598972,
-      "eval_token_f1": -0.08342496106391631,
-      "eval_token_precision": -0.27079911946608903,
-      "eval_token_recall": -0.04930755914598972,
-      "step": 1300
-    },
-    {
-      "epoch": 10.950413223140496,
-      "grad_norm": 1.3775362968444824,
-      "learning_rate": 2.110821477999747e-06,
-      "loss": 7.1653,
-      "step": 1325
-    },
-    {
-      "epoch": 11.15702479338843,
-      "grad_norm": 1.7906850576400757,
-      "learning_rate": 1.3702254344477916e-06,
-      "loss": 7.1161,
-      "step": 1350
-    },
-    {
-      "epoch": 11.363636363636363,
-      "grad_norm": 1.9360337257385254,
-      "learning_rate": 7.873309864103895e-07,
-      "loss": 6.9461,
-      "step": 1375
-    },
-    {
-      "epoch": 11.570247933884298,
-      "grad_norm": 1.425911784172058,
-      "learning_rate": 3.640284034866215e-07,
-      "loss": 7.0805,
-      "step": 1400
-    },
-    {
-      "epoch": 11.570247933884298,
-      "eval_character_accuracy": 0.09935735887096775,
-      "eval_character_f1": 0.2854130741493807,
-      "eval_character_normalized_edit_distance": 0.9006426411290323,
-      "eval_character_precision": -0.32708681467335077,
-      "eval_character_recall": 0.09935735887096775,
-      "eval_loss": 7.2913360595703125,
-      "eval_runtime": 100.3688,
-      "eval_samples_per_second": 0.837,
-      "eval_steps_per_second": 0.837,
-      "eval_token_accuracy": -0.04555683785343345,
-      "eval_token_f1": -0.0787525166168733,
-      "eval_token_precision": -0.2902410367785808,
-      "eval_token_recall": -0.04555683785343345,
-      "step": 1400
-    },
-    {
-      "epoch": 11.776859504132231,
-      "grad_norm": 1.6527899503707886,
-      "learning_rate": 1.0169041448943039e-07,
-      "loss": 7.0609,
-      "step": 1425
-    },
-    {
-      "epoch": 11.983471074380166,
-      "grad_norm": 1.945165991783142,
-      "learning_rate": 1.1677558205025475e-09,
-      "loss": 7.1057,
-      "step": 1450
     }
   ],
-  "logging_steps": 25,
-  "max_steps": 1452,
   "num_input_tokens_seen": 0,
-  "num_train_epochs": 12,
-  "save_steps": 100,
   "stateful_callbacks": {
     "TrainerControl": {
       "args": {
@@ -671,8 +69,8 @@
       "attributes": {}
     }
   },
-  "total_flos": 8.739058770115559e+18,
-  "train_batch_size": 1,
   "trial_name": null,
   "trial_params": null
 }

 {
+  "best_global_step": null,
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 3.0,
+  "eval_steps": 500,
+  "global_step": 348,
   "is_hyper_param_search": false,
   "is_local_process_zero": true,
   "is_world_process_zero": true,
   "log_history": [
     {
+      "epoch": 0.43360433604336046,
+      "grad_norm": 1.351936936378479,
+      "learning_rate": 1.9600000000000002e-05,
+      "loss": 8.5745,
       "step": 50
     },
     {
+      "epoch": 0.8672086720867209,
+      "grad_norm": 0.49343356490135193,
+      "learning_rate": 3.960000000000001e-05,
+      "loss": 7.2834,
       "step": 100
     },
     {
+      "epoch": 1.294850948509485,
+      "grad_norm": 0.35413601994514465,
+      "learning_rate": 5.96e-05,
+      "loss": 6.7556,
       "step": 150
     },
     {
+      "epoch": 1.7284552845528456,
+      "grad_norm": 0.7609367966651917,
+      "learning_rate": 7.960000000000001e-05,
+      "loss": 6.7161,
       "step": 200
     },
     {
+      "epoch": 2.15609756097561,
+      "grad_norm": 0.5119837522506714,
+      "learning_rate": 9.960000000000001e-05,
+      "loss": 6.6669,
       "step": 250
     },
     {
+      "epoch": 2.58970189701897,
+      "grad_norm": 0.4061020314693451,
+      "learning_rate": 0.00011960000000000001,
+      "loss": 6.6361,
       "step": 300
     }
   ],
+  "logging_steps": 50,
+  "max_steps": 348,
   "num_input_tokens_seen": 0,
+  "num_train_epochs": 3,
+  "save_steps": 500,
   "stateful_callbacks": {
     "TrainerControl": {
       "args": {
       "attributes": {}
     }
   },
+  "total_flos": 6840447945891840.0,
+  "train_batch_size": 2,
   "trial_name": null,
   "trial_params": null
 }

training_args.bin CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:7dd9568750889baa8fd79088a719f23db4b7376484d1231cdbaa06c0e3d2b201
-size 5969

 version https://git-lfs.github.com/spec/v1
+oid sha256:bce7569c25833e548c64bca2ea91836890e0703bd7924dcc765ad8c3064d30f7
+size 5496

vocab.json CHANGED Viewed

The diff for this file is too large to render. See raw diff