Upload folder using huggingface_hub

.gitattributes

@@ -33,3 +33,5 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+checkpoint-4000-Q8_0.gguf filter=lfs diff=lfs merge=lfs -text
+checkpoint-4000-f16.gguf filter=lfs diff=lfs merge=lfs -text

README.md

@@ -1,116 +1,125 @@
 ---
 language: en
 license: apache-2.0
+base_model: dkudos/cinimod-devops
 tags:
-- llm
-- devops
-- kubernetes
-- terraform
-- aws
-- docker
-- infrastructure
-- cicd
-- linux
-- self-trained
-library_name: transformers
+  - cinimod
+  - devops
+  - llm
+  - causal-lm
+  - llama
+  - llama.cpp
+pipeline_tag: text-generation
+model-index:
+  - name: Cinimod DevOps 300M
+    results: []
 ---
 
-# Cinimod DevOps 1.5B
+# Cinimod DevOps 300M
 
-A 1.5B parameter (781M active) Llama-3 style language model, trained from scratch on DevOps and cloud infrastructure data. Designed for answering technical questions about Kubernetes, Terraform, AWS, Docker, CI/CD, Linux administration, and cloud operations.
+A 287M-parameter decoder-only causal language model (Llama-3 style architecture), trained from scratch on a DevOps/ops SysAdmin domain corpus. Target usage: devops tooling assistance, ops documentation, and small on-box language modeling.
 
 ## Model Details
 
-| Parameter | Value |
-|-----------|-------|
-| **Architecture** | LlamaForCausalLM (custom) |
-| **Parameters** | ~781M active (1.5B config name) |
-| **Hidden Size** | 1536 |
-| **Layers** | 24 |
-| **Attention Heads** | 24 (Q) / 6 (KV) - GQA |
-| **Intermediate Size** | 4096 |
-| **Vocab Size** | 65,536 (BPE) |
-| **Context Window** | 1,024 tokens (training) |
-| **RoPE Theta** | 500,000 |
-| **Training Dtype** | FP32 (full precision) |
-| **Inference Dtype** | FP16 |
-| **Tokenizer** | Custom BPE, 65K vocab |
-
-## Training Details
-
-### Pretraining
-- **Data**: ~50GB curated DevOps corpus (FineWeb-Edu subset, GitHub DevOps repos, cloud provider documentation)
-- **Steps**: 60,423
-- **Batch Size**: 8 (2 GPUs x 1 per GPU x 4 grad accum)
-- **Learning Rate**: Peak 3e-4, cosine decay
-- **Optimizer**: DeepSpeed ZeRO-2
-- **Sequence Length**: 1,024
-- **Hardware**: 2x Tesla V100 16GB
-- **Training Time**: ~6 days
-
-### Architecture Notes
-- **No RoPE scaling** - previous linear scaling (factor=256) broke positional encoding during training, pinning loss at unigram floor
-- **Gradient checkpointing** enabled for memory efficiency
-- **Tied word embeddings** for parameter savings
-
-## Usage
+| Property | Value |
+|---|---|
+| Parameters | 287,310,848 (~287M) |
+| Architecture | Llama-style decoder-only (custom, not stock `transformers` LlamaForCausalLM params) |
+| Hidden size | 1024 |
+| Layers | 20 |
+| Attention heads | 16 |
+| KV heads (GQA) | 4 |
+| Intermediate size | 2730 |
+| Vocab size | 65,536 (BPE) |
+| Position embeddings | RoPE, theta = 500000 |
+| Trained context | 4096 tokens |
+| Max context (served) | up to 256K via linear RoPE scaling |
+| Embeddings | tied (no separate lm_head) |
+
+## Training
+
+- **Objective**: from-scratch pretraining on a DevOps/ops corpus.
+- **Compute**: 2x RTX 4090 (24 GB each, bf16), DeepSpeed ZeRO-2, FP32 master weights via bf16 autocast.
+- **Tokens**: one epoch over ~132,068 sequences at seq_len 4096 (~540M tokens).
+- **Steps**: 4000, warmup 40, LR 6e-4 cosine decay (final step LR ~0).
+- **Efficient attention**: `torch.nn.functional.scaled_dot_product_attention` (flash path via flash-attn 2).
+- **Loss trajectory**: train loss 0.43 (step 2000) -> 0.35 (step 4000).
+
+### Evaluation
+
+- **Held-out full validation** (17,492 bins / 123,656 sequences @ 4096): mean eval loss ~2.30 (perplexity ~10). Final value in `full_val_eval.log`.
+- Note: the model is trained for exactly one epoch; a small train-eval gap is expected and the held-out score is the honest number.
+
+## Files
+
+| File | Description | Size |
+|---|---|---|
+| `model.safetensors` | Full bf16 PyTorch weights (HF format with `config.json`, `tokenizer.json`/`tokenizer_config.json`) | 548 MiB |
+| `checkpoint-4000-f16.gguf` | GGUF F16 (float16) - best quality for llama.cpp | 550 MiB |
+| `checkpoint-4000-Q8_0.gguf` | GGUF Q8_0 8-bit quantized - recommended for most uses (near-lossless, ~2x smaller) | 344 MiB |
+| `config.json` | Model config (transformers) | - |
+| `tokenizer.json` / `tokenizer_config.json` | BPE tokenizer (vocab 65,536) | - |
+| `train_log.log` | Full training log (steps, losses, LR) | - |
+| `full_val_eval.log` | Held-out full validation eval log | - |
+
+## How to run
+
+### HuggingFace transformers (PyTorch)
+
+The `model.safetensors` require the Cinimod architecture classes (`cinimod.model.llama.LlamaForCausalLM`) — a custom Llama variant, **not** the stock `transformers.LlamaForCausalLM`. Load from the repo source only:
 
 ```python
-from transformers import AutoModelForCausalLM, AutoTokenizer
-
-tokenizer = AutoTokenizer.from_pretrained('dkudos/cinimod-devops')
-model = AutoModelForCausalLM.from_pretrained(
-    'dkudos/cinimod-devops',
-    torch_dtype='auto',
-    device_map='auto',
-    trust_remote_code=True
-)
-
-prompt = 'How do I set up a Kubernetes Deployment?'
-inputs = tokenizer(prompt, return_tensors='pt').to(model.device)
-outputs = model.generate(**inputs, max_new_tokens=256, temperature=0.7, top_p=0.9)
-print(tokenizer.decode(outputs[0], skip_special_tokens=True))
+import sys
+sys.path.insert(0, "/path/to/cinimod-llm/src")   # package src/cinimod
+from cinimod.model.llama import LlamaForCausalLM
+from transformers import PreTrainedTokenizerFast
+
+model = LlamaForCausalLM.from_pretrained("dkudos/cinimod-devops")
+tok = PreTrainedTokenizerFast.from_pretrained("dkudos/cinimod-devops")
+ids = tok.encode("how do I check nginx status", return_tensors="pt")
+out = model.generate(ids, max_new_tokens=64)
+print(tok.decode(out[0]))
 ```
 
-### OpenAI-Compatible Server
+> If you are not in the Cinimod repo, use the GGUFs instead — they are standalone and need no source code. We publish GGUFs precisely because the HF-PyTorch path depends on the custom architecture classes.
+```
+
+### llama.cpp (recommended for serving)
+
+Both GGUFs load directly in llama.cpp / llama-server with no external deps.
+
+Default (train context, 4096):
 
 ```bash
-python scripts/serve_model.py --model_path dkudos/cinimod-devops --tokenizer_path tokenizer --port 8080
+llama-server -m dkudos/cinimod-devops/checkpoint-4000-Q8_0.gguf --port 8080
 ```
 
-## Capabilities
+256K context via linear RoPE scaling (trained at 4096):
 
-The model was trained on a broad DevOps corpus covering:
-- **Kubernetes**: Pods, Deployments, StatefulSets, Services, Ingress, networking, storage, RBAC, Helm, operators, troubleshooting
-- **Terraform**: IaC, state management, modules, providers, workspaces, import, best practices
-- **AWS**: EC2, S3, VPC, RDS, EKS, Lambda, IAM, CloudWatch, cost optimization
-- **Docker**: Images, containers, volumes, networking, Compose, multi-stage builds, Swarm
-- **CI/CD**: GitHub Actions, Jenkins, GitLab CI, ArgoCD, deployment strategies, pipelines
-- **Linux**: System administration, networking, security, performance monitoring, systemd
-- **Monitoring**: Prometheus, Grafana, Alertmanager, Loki, Elasticsearch, distributed tracing
-- **Security**: Zero trust, vulnerability scanning, incident response, penetration testing
-- **Networking**: DNS, load balancing, CDN, SSL/TLS, troubleshooting
+```bash
+llama-server -m dkudos/cinimod-devops/checkpoint-4000-Q8_0.gguf \
+  --ctx-size 262144 --rope-scaling linear --rope-scale 64 --port 8080
+```
 
-## Limitations
+Rope scaling is **serve-time only**; this model ships with `rope_scaling: null`. For aggressive 64x scaling, Yarn (`--rope-scaling yarn --rope-scale 64`) often generalizes better than linear if long-range coherence suffers.
 
-- **Context window**: 1,024 tokens training context (RoPE theta 500K allows extrapolation to ~256K at inference, but quality may degrade)
-- **Parameter count**: 781M params - smaller than typical production models; complex reasoning may be limited
-- **Training data**: ~50GB - curated but limited compared to frontier models
-- **No instruction tuning**: This is the base model. Chat-tuned versions available separately.
-- **Hallucination**: May produce incorrect technical details, especially for niche topics or version-specific APIs
-- **No code execution**: Cannot run commands or verify solutions - always test in a safe environment
+One-line test:
 
-## Ethical Considerations
+```bash
+llama-server -m checkpoint-4000-Q8_0.gguf --ctx-size 262144 --rope-scaling linear --rope-scale 64
+curl http://localhost:8080/v1/chat/completions -H 'Content-Type: application/json' \
+  -d '{"messages":[{"role":"user","content":"List 5 common systemd service commands"}],"max_tokens":128}'
+```
 
-This model is designed as a technical assistant for DevOps professionals. Always verify generated configurations and commands before applying to production systems. Infrastructure changes can have significant consequences if applied without review.
+## Notes on the tokenizer
 
-## Citation
+Vocabulary is a 65,536-token BPE (custom, `tokenizers` backend). `<pad>`, `<s>`, `</s>`, `<unk>` are at indices 0-3, trained with `pad_token_id=0`. It is a plain causal LM — no chat template is baked in. If GGUF chat-format warnings appear they are just llama.cpp server defaults, not part of the model.
 
-```bibtex
-@misc{cinimod-devops-1b5,
-  title={Cinimod DevOps 1.5B},
-  author={Dominic Kaiser},
-  year={2026},
-  url={https://huggingface.co/dkudos/cinimod-devops}
-}
-```
+## Limitations
+
+- Pretrained from scratch on a single domain (DevOps) for one epoch at small scale (~287M) — expect domain-limited fluency, not general world knowledge.
+- Exact transformers architecture classes are Cinimod-custom; use the GGUFs for maximum portability (no source code needed).
+
+## License
+
+Apache 2.0

checkpoint-4000-Q8_0.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:97226a8f98f42d029a02a76d3c5712f534e92703f7d2252b53b3356c665002da
+size 360204832

checkpoint-4000-f16.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:691acb9f1efa8db5111afcf0b8bea6258762ee9ca0a4d18283d42db3bda33ae7
+size 577103392

config.json

@@ -4,27 +4,24 @@
   ],
   "attention_bias": false,
   "attention_dropout": 0.0,
-  "bos_token_id": 1,
-  "dtype": "float16",
-  "eos_token_id": 2,
-  "head_dim": 64,
+  "dtype": "bfloat16",
   "hidden_act": "silu",
-  "hidden_size": 1536,
+  "hidden_size": 1024,
   "initializer_range": 0.02,
-  "intermediate_size": 4096,
-  "max_position_embeddings": 1024,
+  "intermediate_size": 2730,
+  "max_position_embeddings": 4096,
   "mlp_bias": false,
   "model_type": "llama",
-  "num_attention_heads": 24,
-  "num_hidden_layers": 24,
-  "num_key_value_heads": 6,
+  "num_attention_heads": 16,
+  "num_hidden_layers": 20,
+  "num_key_value_heads": 4,
   "pad_token_id": 0,
-  "pretraining_tp": 1,
   "rms_norm_eps": 1e-05,
   "rope_parameters": {
     "rope_theta": 500000,
     "rope_type": "default"
   },
+  "rope_theta": 500000,
   "tie_word_embeddings": true,
   "transformers_version": "5.14.1",
   "use_cache": false,

full_val_eval.log

@@ -0,0 +1,209 @@
+[Eval] Loading checkpoint outputs/devops-300m-4096-bf16-vast/checkpoint-4000
+[transformers] LlamaForCausalLM has generative capabilities, as `prepare_inputs_for_generation` is explicitly defined. However, it doesn't directly inherit from `GenerationMixin`. From 👉v4.50👈 onwards, `PreTrainedModel` will NOT inherit from `GenerationMixin`, and this model will lose the ability to call `generate` and other related functions.
+  - If you're using `trust_remote_code=True`, you can get rid of this warning by loading the model with an auto class. See https://huggingface.co/docs/transformers/en/model_doc/auto#auto-classes
+  - If you are the owner of the model architecture code, please modify your model class such that it inherits from `GenerationMixin` (after `PreTrainedModel`, otherwise you'll get an exception).
+  - If you are not the owner of the model architecture class, please contact the model code owner to update it.
+[Eval] 287.3M parameters loaded
+[Dataset] 17492 files, 123,656 non-overlapping samples, seq_len=4096
+[transformers] `use_return_dict` is deprecated! Use `return_dict` instead!
+[Eval] batch 1 | samples 8/123656 | mean_loss 2.4351 | 1s elapsed | 35,948 tok/s
+[Eval] batch 21 | samples 168/123656 | mean_loss 0.5549 | 9s elapsed | 75,776 tok/s
+[Eval] batch 41 | samples 328/123656 | mean_loss 0.4678 | 17s elapsed | 77,938 tok/s
+[Eval] batch 61 | samples 488/123656 | mean_loss 0.4119 | 25s elapsed | 78,609 tok/s
+[Eval] batch 81 | samples 648/123656 | mean_loss 0.3887 | 34s elapsed | 78,920 tok/s
+[Eval] batch 101 | samples 808/123656 | mean_loss 0.3729 | 42s elapsed | 79,039 tok/s
+[Eval] batch 121 | samples 968/123656 | mean_loss 0.3641 | 50s elapsed | 79,081 tok/s
+[Eval] batch 141 | samples 1128/123656 | mean_loss 0.3509 | 58s elapsed | 79,099 tok/s
+[Eval] batch 161 | samples 1288/123656 | mean_loss 0.3459 | 67s elapsed | 79,107 tok/s
+[Eval] batch 181 | samples 1448/123656 | mean_loss 0.3395 | 75s elapsed | 79,110 tok/s
+[Eval] batch 201 | samples 1608/123656 | mean_loss 0.3343 | 83s elapsed | 79,102 tok/s
+[Eval] batch 221 | samples 1768/123656 | mean_loss 0.3313 | 92s elapsed | 79,089 tok/s
+[Eval] batch 241 | samples 1928/123656 | mean_loss 0.3274 | 100s elapsed | 79,048 tok/s
+[Eval] batch 261 | samples 2088/123656 | mean_loss 0.4029 | 108s elapsed | 79,049 tok/s
+[Eval] batch 281 | samples 2248/123656 | mean_loss 0.5424 | 117s elapsed | 79,037 tok/s
+[Eval] batch 301 | samples 2408/123656 | mean_loss 0.6806 | 125s elapsed | 79,016 tok/s
+[Eval] batch 321 | samples 2568/123656 | mean_loss 0.7765 | 133s elapsed | 79,013 tok/s
+[Eval] batch 341 | samples 2728/123656 | mean_loss 0.8807 | 141s elapsed | 78,973 tok/s
+[Eval] batch 361 | samples 2888/123656 | mean_loss 0.9719 | 150s elapsed | 78,961 tok/s
+[Eval] batch 381 | samples 3048/123656 | mean_loss 1.0495 | 158s elapsed | 78,948 tok/s
+[Eval] batch 401 | samples 3208/123656 | mean_loss 1.1496 | 166s elapsed | 78,949 tok/s
+[Eval] batch 421 | samples 3368/123656 | mean_loss 1.2136 | 175s elapsed | 78,921 tok/s
+[Eval] batch 441 | samples 3528/123656 | mean_loss 1.2720 | 183s elapsed | 78,920 tok/s
+[Eval] batch 461 | samples 3688/123656 | mean_loss 1.3302 | 191s elapsed | 78,908 tok/s
+[Eval] batch 481 | samples 3848/123656 | mean_loss 1.3822 | 200s elapsed | 78,904 tok/s
+[Eval] batch 501 | samples 4008/123656 | mean_loss 1.4212 | 208s elapsed | 78,884 tok/s
+[Eval] batch 521 | samples 4168/123656 | mean_loss 1.4664 | 216s elapsed | 78,861 tok/s
+[Eval] batch 541 | samples 4328/123656 | mean_loss 1.5031 | 225s elapsed | 78,864 tok/s
+[Eval] batch 561 | samples 4488/123656 | mean_loss 1.5387 | 233s elapsed | 78,841 tok/s
+[Eval] batch 581 | samples 4648/123656 | mean_loss 1.5600 | 242s elapsed | 78,833 tok/s
+[Eval] batch 601 | samples 4808/123656 | mean_loss 1.5791 | 250s elapsed | 78,831 tok/s
+[Eval] batch 621 | samples 4968/123656 | mean_loss 1.6022 | 258s elapsed | 78,812 tok/s
+[Eval] batch 641 | samples 5128/123656 | mean_loss 1.6348 | 266s elapsed | 78,816 tok/s
+[Eval] batch 661 | samples 5288/123656 | mean_loss 1.6470 | 275s elapsed | 78,803 tok/s
+[Eval] batch 681 | samples 5448/123656 | mean_loss 1.6770 | 283s elapsed | 78,806 tok/s
+[Eval] batch 701 | samples 5608/123656 | mean_loss 1.6898 | 292s elapsed | 78,794 tok/s
+[Eval] batch 721 | samples 5768/123656 | mean_loss 1.7145 | 300s elapsed | 78,776 tok/s
+[Eval] batch 741 | samples 5928/123656 | mean_loss 1.7331 | 308s elapsed | 78,780 tok/s
+[Eval] batch 761 | samples 6088/123656 | mean_loss 1.7482 | 317s elapsed | 78,783 tok/s
+[Eval] batch 781 | samples 6248/123656 | mean_loss 1.7681 | 325s elapsed | 78,782 tok/s
+[Eval] batch 801 | samples 6408/123656 | mean_loss 1.7910 | 333s elapsed | 78,779 tok/s
+[Eval] batch 821 | samples 6568/123656 | mean_loss 1.8134 | 342s elapsed | 78,776 tok/s
+[Eval] batch 841 | samples 6728/123656 | mean_loss 1.8247 | 350s elapsed | 78,767 tok/s
+[Eval] batch 861 | samples 6888/123656 | mean_loss 1.8317 | 358s elapsed | 78,769 tok/s
+[Eval] batch 881 | samples 7048/123656 | mean_loss 1.8502 | 367s elapsed | 78,760 tok/s
+[Eval] batch 901 | samples 7208/123656 | mean_loss 1.8656 | 375s elapsed | 78,748 tok/s
+[Eval] batch 921 | samples 7368/123656 | mean_loss 1.8812 | 383s elapsed | 78,750 tok/s
+[Eval] batch 941 | samples 7528/123656 | mean_loss 1.8935 | 392s elapsed | 78,748 tok/s
+[Eval] batch 961 | samples 7688/123656 | mean_loss 1.9068 | 400s elapsed | 78,751 tok/s
+[Eval] batch 981 | samples 7848/123656 | mean_loss 1.9212 | 408s elapsed | 78,740 tok/s
+[Eval] batch 1001 | samples 8008/123656 | mean_loss 1.9275 | 417s elapsed | 78,735 tok/s
+[Eval] batch 1021 | samples 8168/123656 | mean_loss 1.9460 | 425s elapsed | 78,736 tok/s
+[Eval] batch 1041 | samples 8328/123656 | mean_loss 1.9556 | 433s elapsed | 78,725 tok/s
+[Eval] batch 1061 | samples 8488/123656 | mean_loss 1.9694 | 442s elapsed | 78,727 tok/s
+[Eval] batch 1081 | samples 8648/123656 | mean_loss 1.9707 | 450s elapsed | 78,725 tok/s
+[Eval] batch 1101 | samples 8808/123656 | mean_loss 1.9795 | 458s elapsed | 78,715 tok/s
+[Eval] batch 1121 | samples 8968/123656 | mean_loss 1.9898 | 467s elapsed | 78,716 tok/s
+[Eval] batch 1141 | samples 9128/123656 | mean_loss 1.9981 | 475s elapsed | 78,711 tok/s
+[Eval] batch 1161 | samples 9288/123656 | mean_loss 2.0093 | 483s elapsed | 78,700 tok/s
+[Eval] batch 1181 | samples 9448/123656 | mean_loss 2.0145 | 492s elapsed | 78,701 tok/s
+[Eval] batch 1201 | samples 9608/123656 | mean_loss 2.0230 | 500s elapsed | 78,699 tok/s
+[Eval] batch 1221 | samples 9768/123656 | mean_loss 2.0307 | 508s elapsed | 78,688 tok/s
+[Eval] batch 1241 | samples 9928/123656 | mean_loss 2.0364 | 517s elapsed | 78,683 tok/s
+[Eval] batch 1261 | samples 10088/123656 | mean_loss 2.0445 | 525s elapsed | 78,684 tok/s
+[Eval] batch 1281 | samples 10248/123656 | mean_loss 2.0533 | 533s elapsed | 78,682 tok/s
+[Eval] batch 1301 | samples 10408/123656 | mean_loss 2.0581 | 542s elapsed | 78,684 tok/s
+[Eval] batch 1321 | samples 10568/123656 | mean_loss 2.0644 | 550s elapsed | 78,680 tok/s
+[Eval] batch 1341 | samples 10728/123656 | mean_loss 2.0720 | 558s elapsed | 78,681 tok/s
+[Eval] batch 1361 | samples 10888/123656 | mean_loss 2.0805 | 567s elapsed | 78,680 tok/s
+[Eval] batch 1381 | samples 11048/123656 | mean_loss 2.0827 | 575s elapsed | 78,673 tok/s
+[Eval] batch 1401 | samples 11208/123656 | mean_loss 2.0902 | 583s elapsed | 78,678 tok/s
+[Eval] batch 1421 | samples 11368/123656 | mean_loss 2.0964 | 592s elapsed | 78,671 tok/s
+[Eval] batch 1441 | samples 11528/123656 | mean_loss 2.1036 | 600s elapsed | 78,663 tok/s
+[Eval] batch 1461 | samples 11688/123656 | mean_loss 2.1106 | 609s elapsed | 78,662 tok/s
+[Eval] batch 1481 | samples 11848/123656 | mean_loss 2.1161 | 617s elapsed | 78,663 tok/s
+[Eval] batch 1501 | samples 12008/123656 | mean_loss 2.1225 | 625s elapsed | 78,655 tok/s
+[Eval] batch 1521 | samples 12168/123656 | mean_loss 2.1277 | 634s elapsed | 78,647 tok/s
+[Eval] batch 1541 | samples 12328/123656 | mean_loss 2.1312 | 642s elapsed | 78,647 tok/s
+[Eval] batch 1561 | samples 12488/123656 | mean_loss 2.1368 | 650s elapsed | 78,647 tok/s
+[Eval] batch 1581 | samples 12648/123656 | mean_loss 2.1410 | 659s elapsed | 78,641 tok/s
+[Eval] batch 1601 | samples 12808/123656 | mean_loss 2.1455 | 667s elapsed | 78,643 tok/s
+[Eval] batch 1621 | samples 12968/123656 | mean_loss 2.1506 | 675s elapsed | 78,642 tok/s
+[Eval] batch 1641 | samples 13128/123656 | mean_loss 2.1550 | 684s elapsed | 78,642 tok/s
+[Eval] batch 1661 | samples 13288/123656 | mean_loss 2.1604 | 692s elapsed | 78,642 tok/s
+[Eval] batch 1681 | samples 13448/123656 | mean_loss 2.1658 | 700s elapsed | 78,637 tok/s
+[Eval] batch 1701 | samples 13608/123656 | mean_loss 2.1722 | 709s elapsed | 78,640 tok/s
+[Eval] batch 1721 | samples 13768/123656 | mean_loss 2.1745 | 717s elapsed | 78,638 tok/s
+[Eval] batch 1741 | samples 13928/123656 | mean_loss 2.1791 | 725s elapsed | 78,637 tok/s
+[Eval] batch 1761 | samples 14088/123656 | mean_loss 2.1828 | 734s elapsed | 78,637 tok/s
+[Eval] batch 1781 | samples 14248/123656 | mean_loss 2.1860 | 742s elapsed | 78,635 tok/s
+[Eval] batch 1801 | samples 14408/123656 | mean_loss 2.1904 | 750s elapsed | 78,636 tok/s
+[Eval] batch 1821 | samples 14568/123656 | mean_loss 2.1950 | 759s elapsed | 78,634 tok/s
+[Eval] batch 1841 | samples 14728/123656 | mean_loss 2.1969 | 767s elapsed | 78,628 tok/s
+[Eval] batch 1861 | samples 14888/123656 | mean_loss 2.2013 | 776s elapsed | 78,626 tok/s
+[Eval] batch 1881 | samples 15048/123656 | mean_loss 2.2046 | 784s elapsed | 78,628 tok/s
+[Eval] batch 1901 | samples 15208/123656 | mean_loss 2.2096 | 792s elapsed | 78,623 tok/s
+[Eval] batch 1921 | samples 15368/123656 | mean_loss 2.2126 | 801s elapsed | 78,617 tok/s
+[Eval] batch 1941 | samples 15528/123656 | mean_loss 2.2160 | 809s elapsed | 78,620 tok/s
+[Eval] batch 1961 | samples 15688/123656 | mean_loss 2.2175 | 817s elapsed | 78,618 tok/s
+[Eval] batch 1981 | samples 15848/123656 | mean_loss 2.2175 | 826s elapsed | 78,616 tok/s
+[Eval] batch 2001 | samples 16008/123656 | mean_loss 2.2187 | 834s elapsed | 78,618 tok/s
+[Eval] batch 2021 | samples 16168/123656 | mean_loss 2.2200 | 842s elapsed | 78,614 tok/s
+[Eval] batch 2041 | samples 16328/123656 | mean_loss 2.2234 | 851s elapsed | 78,616 tok/s
+[Eval] batch 2061 | samples 16488/123656 | mean_loss 2.2263 | 859s elapsed | 78,616 tok/s
+[Eval] batch 2081 | samples 16648/123656 | mean_loss 2.2291 | 867s elapsed | 78,617 tok/s
+[Eval] batch 2101 | samples 16808/123656 | mean_loss 2.2318 | 876s elapsed | 78,617 tok/s
+[Eval] batch 2121 | samples 16968/123656 | mean_loss 2.2349 | 884s elapsed | 78,614 tok/s
+[Eval] batch 2141 | samples 17128/123656 | mean_loss 2.2381 | 892s elapsed | 78,615 tok/s
+[Eval] batch 2161 | samples 17288/123656 | mean_loss 2.2403 | 901s elapsed | 78,613 tok/s
+[Eval] batch 2181 | samples 17448/123656 | mean_loss 2.2410 | 909s elapsed | 78,608 tok/s
+[Eval] batch 2201 | samples 17608/123656 | mean_loss 2.2435 | 917s elapsed | 78,607 tok/s
+[Eval] batch 2221 | samples 17768/123656 | mean_loss 2.2463 | 926s elapsed | 78,609 tok/s
+[Eval] batch 2241 | samples 17928/123656 | mean_loss 2.2492 | 934s elapsed | 78,605 tok/s
+[Eval] batch 2261 | samples 18088/123656 | mean_loss 2.2505 | 943s elapsed | 78,601 tok/s
+[Eval] batch 2281 | samples 18248/123656 | mean_loss 2.2533 | 951s elapsed | 78,602 tok/s
+[Eval] batch 2301 | samples 18408/123656 | mean_loss 2.2556 | 959s elapsed | 78,601 tok/s
+[Eval] batch 2321 | samples 18568/123656 | mean_loss 2.2558 | 968s elapsed | 78,598 tok/s
+[Eval] batch 2341 | samples 18728/123656 | mean_loss 2.2585 | 976s elapsed | 78,600 tok/s
+[Eval] batch 2361 | samples 18888/123656 | mean_loss 2.2608 | 984s elapsed | 78,598 tok/s
+[Eval] batch 2381 | samples 19048/123656 | mean_loss 2.2628 | 993s elapsed | 78,594 tok/s
+[Eval] batch 2401 | samples 19208/123656 | mean_loss 2.2647 | 1001s elapsed | 78,594 tok/s
+[Eval] batch 2421 | samples 19368/123656 | mean_loss 2.2670 | 1009s elapsed | 78,595 tok/s
+[Eval] batch 2441 | samples 19528/123656 | mean_loss 2.2687 | 1018s elapsed | 78,591 tok/s
+[Eval] batch 2461 | samples 19688/123656 | mean_loss 2.2713 | 1026s elapsed | 78,589 tok/s
+[Eval] batch 2481 | samples 19848/123656 | mean_loss 2.2732 | 1034s elapsed | 78,591 tok/s
+[Eval] batch 2501 | samples 20008/123656 | mean_loss 2.2744 | 1043s elapsed | 78,588 tok/s
+[Eval] batch 2521 | samples 20168/123656 | mean_loss 2.2756 | 1051s elapsed | 78,587 tok/s
+[Eval] batch 2541 | samples 20328/123656 | mean_loss 2.2810 | 1059s elapsed | 78,589 tok/s
+[Eval] batch 2561 | samples 20488/123656 | mean_loss 2.2824 | 1068s elapsed | 78,585 tok/s
+[Eval] batch 2581 | samples 20648/123656 | mean_loss 2.2848 | 1076s elapsed | 78,583 tok/s
+[Eval] batch 2601 | samples 20808/123656 | mean_loss 2.2865 | 1085s elapsed | 78,586 tok/s
+[Eval] batch 2621 | samples 20968/123656 | mean_loss 2.2866 | 1093s elapsed | 78,583 tok/s
+[Eval] batch 2641 | samples 21128/123656 | mean_loss 2.2882 | 1101s elapsed | 78,581 tok/s
+[Eval] batch 2661 | samples 21288/123656 | mean_loss 2.2896 | 1110s elapsed | 78,583 tok/s
+[Eval] batch 2681 | samples 21448/123656 | mean_loss 2.2907 | 1118s elapsed | 78,581 tok/s
+[Eval] batch 2701 | samples 21608/123656 | mean_loss 2.2958 | 1126s elapsed | 78,577 tok/s
+[Eval] batch 2721 | samples 21768/123656 | mean_loss 2.3010 | 1135s elapsed | 78,578 tok/s
+[Eval] batch 2741 | samples 21928/123656 | mean_loss 2.3032 | 1143s elapsed | 78,579 tok/s
+[Eval] batch 2761 | samples 22088/123656 | mean_loss 2.3048 | 1151s elapsed | 78,577 tok/s
+[Eval] batch 2781 | samples 22248/123656 | mean_loss 2.3010 | 1160s elapsed | 78,579 tok/s
+[Eval] batch 2801 | samples 22408/123656 | mean_loss 2.3022 | 1168s elapsed | 78,577 tok/s
+[Eval] batch 2821 | samples 22568/123656 | mean_loss 2.3038 | 1176s elapsed | 78,579 tok/s
+[Eval] batch 2841 | samples 22728/123656 | mean_loss 2.3055 | 1185s elapsed | 78,578 tok/s
+[Eval] batch 2861 | samples 22888/123656 | mean_loss 2.3052 | 1193s elapsed | 78,577 tok/s
+[Eval] batch 2881 | samples 23048/123656 | mean_loss 2.3059 | 1201s elapsed | 78,579 tok/s
+[Eval] batch 2901 | samples 23208/123656 | mean_loss 2.3081 | 1210s elapsed | 78,577 tok/s
+[Eval] batch 2921 | samples 23368/123656 | mean_loss 2.3068 | 1218s elapsed | 78,578 tok/s
+[Eval] batch 2941 | samples 23528/123656 | mean_loss 2.3083 | 1226s elapsed | 78,578 tok/s
+[Eval] batch 2961 | samples 23688/123656 | mean_loss 2.3084 | 1235s elapsed | 78,574 tok/s
+[Eval] batch 2981 | samples 23848/123656 | mean_loss 2.3096 | 1243s elapsed | 78,571 tok/s
+[Eval] batch 3001 | samples 24008/123656 | mean_loss 2.3104 | 1252s elapsed | 78,572 tok/s
+[Eval] batch 3021 | samples 24168/123656 | mean_loss 2.3127 | 1260s elapsed | 78,572 tok/s
+[Eval] batch 3041 | samples 24328/123656 | mean_loss 2.3135 | 1268s elapsed | 78,569 tok/s
+[Eval] batch 3061 | samples 24488/123656 | mean_loss 2.3155 | 1277s elapsed | 78,569 tok/s
+[Eval] batch 3081 | samples 24648/123656 | mean_loss 2.3198 | 1285s elapsed | 78,570 tok/s
+[Eval] batch 3101 | samples 24808/123656 | mean_loss 2.3205 | 1293s elapsed | 78,567 tok/s
+[Eval] batch 3121 | samples 24968/123656 | mean_loss 2.3220 | 1302s elapsed | 78,568 tok/s
+[Eval] batch 3141 | samples 25128/123656 | mean_loss 2.3240 | 1310s elapsed | 78,569 tok/s
+[Eval] batch 3161 | samples 25288/123656 | mean_loss 2.3255 | 1318s elapsed | 78,565 tok/s
+[Eval] batch 3181 | samples 25448/123656 | mean_loss 2.3270 | 1327s elapsed | 78,567 tok/s
+[Eval] batch 3201 | samples 25608/123656 | mean_loss 2.3288 | 1335s elapsed | 78,567 tok/s
+[Eval] batch 3221 | samples 25768/123656 | mean_loss 2.3291 | 1343s elapsed | 78,564 tok/s
+[Eval] batch 3241 | samples 25928/123656 | mean_loss 2.3291 | 1352s elapsed | 78,561 tok/s
+[Eval] batch 3261 | samples 26088/123656 | mean_loss 2.3304 | 1360s elapsed | 78,561 tok/s
+[Eval] batch 3281 | samples 26248/123656 | mean_loss 2.3331 | 1368s elapsed | 78,562 tok/s
+[Eval] batch 3301 | samples 26408/123656 | mean_loss 2.3337 | 1377s elapsed | 78,562 tok/s
+[Eval] batch 3321 | samples 26568/123656 | mean_loss 2.3359 | 1385s elapsed | 78,563 tok/s
+[Eval] batch 3341 | samples 26728/123656 | mean_loss 2.3376 | 1394s elapsed | 78,561 tok/s
+[Eval] batch 3361 | samples 26888/123656 | mean_loss 2.3394 | 1402s elapsed | 78,560 tok/s
+[Eval] batch 3381 | samples 27048/123656 | mean_loss 2.3391 | 1410s elapsed | 78,562 tok/s
+[Eval] batch 3401 | samples 27208/123656 | mean_loss 2.3408 | 1419s elapsed | 78,561 tok/s
+[Eval] batch 3421 | samples 27368/123656 | mean_loss 2.3410 | 1427s elapsed | 78,562 tok/s
+[Eval] batch 3441 | samples 27528/123656 | mean_loss 2.3415 | 1435s elapsed | 78,561 tok/s
+[Eval] batch 3461 | samples 27688/123656 | mean_loss 2.3423 | 1444s elapsed | 78,560 tok/s
+[Eval] batch 3481 | samples 27848/123656 | mean_loss 2.3426 | 1452s elapsed | 78,561 tok/s
+[Eval] batch 3501 | samples 28008/123656 | mean_loss 2.3422 | 1460s elapsed | 78,559 tok/s
+[Eval] batch 3521 | samples 28168/123656 | mean_loss 2.3433 | 1469s elapsed | 78,558 tok/s
+[Eval] batch 3541 | samples 28328/123656 | mean_loss 2.3442 | 1477s elapsed | 78,559 tok/s
+[Eval] batch 3561 | samples 28488/123656 | mean_loss 2.3459 | 1485s elapsed | 78,558 tok/s
+[Eval] batch 3581 | samples 28648/123656 | mean_loss 2.3466 | 1494s elapsed | 78,556 tok/s
+[Eval] batch 3601 | samples 28808/123656 | mean_loss 2.3481 | 1502s elapsed | 78,558 tok/s
+[Eval] batch 3621 | samples 28968/123656 | mean_loss 2.3526 | 1510s elapsed | 78,557 tok/s
+[Eval] batch 3641 | samples 29128/123656 | mean_loss 2.3542 | 1519s elapsed | 78,557 tok/s
+[Eval] batch 3661 | samples 29288/123656 | mean_loss 2.3552 | 1527s elapsed | 78,558 tok/s
+[Eval] batch 3681 | samples 29448/123656 | mean_loss 2.3555 | 1535s elapsed | 78,558 tok/s
+[Eval] batch 3701 | samples 29608/123656 | mean_loss 2.3547 | 1544s elapsed | 78,559 tok/s
+[Eval] batch 3721 | samples 29768/123656 | mean_loss 2.3553 | 1552s elapsed | 78,557 tok/s
+[Eval] batch 3741 | samples 29928/123656 | mean_loss 2.3558 | 1560s elapsed | 78,557 tok/s
+[Eval] batch 3761 | samples 30088/123656 | mean_loss 2.3569 | 1569s elapsed | 78,558 tok/s
+[Eval] batch 3781 | samples 30248/123656 | mean_loss 2.3569 | 1577s elapsed | 78,556 tok/s
+[Eval] batch 3801 | samples 30408/123656 | mean_loss 2.3573 | 1586s elapsed | 78,554 tok/s
+[Eval] batch 3821 | samples 30568/123656 | mean_loss 2.3593 | 1594s elapsed | 78,555 tok/s
+[Eval] batch 3841 | samples 30728/123656 | mean_loss 2.3595 | 1602s elapsed | 78,555 tok/s
+[Eval] batch 3861 | samples 30888/123656 | mean_loss 2.3600 | 1611s elapsed | 78,555 tok/s
+[Eval] batch 3881 | samples 31048/123656 | mean_loss 2.3609 | 1619s elapsed | 78,556 tok/s
+[Eval] batch 3901 | samples 31208/123656 | mean_loss 2.3601 | 1627s elapsed | 78,556 tok/s
+[Eval] batch 3921 | samples 31368/123656 | mean_loss 2.3601 | 1636s elapsed | 78,557 tok/s
+[Eval] batch 3941 | samples 31528/123656 | mean_loss 2.3587 | 1644s elapsed | 78,557 tok/s
+[Eval] batch 3961 | samples 31688/123656 | mean_loss 2.3588 | 1652s elapsed | 78,558 tok/s
+[Eval] batch 3981 | samples 31848/123656 | mean_loss 2.3601 | 1661s elapsed | 78,557 tok/s
+[Eval] batch 4001 | samples 32008/123656 | mean_loss 2.3606 | 1669s elapsed | 78,554 tok/s

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:f8cc9320263bc562f9f701ca3d81b22d75f9f5a1a33acf53827c69dae9881939
-size 1390587040
+oid sha256:b8a4a820cf7c9c6c4703e6bb57d20c3f8fdca3344668a77e2f5ce173a90e9ea1
+size 574642392

tokenizer.json

@@ -47,15 +47,6 @@
       "rstrip": false,
       "normalized": false,
       "special": true
-    },
-    {
-      "id": 65536,
-      "content": "<|endoftext|>",
-      "single_word": false,
-      "lstrip": false,
-      "rstrip": false,
-      "normalized": false,
-      "special": true
     }
   ],
   "normalizer": null,

tokenizer_config.json

@@ -1,11 +1,5 @@
 {
   "backend": "tokenizers",
-  "bos_token": "<|endoftext|>",
-  "eos_token": "<|endoftext|>",
-  "is_local": true,
-  "local_files_only": false,
   "model_max_length": 1000000000000000019884624838656,
-  "pad_token": "<|endoftext|>",
-  "tokenizer_class": "TokenizersBackend",
-  "unk_token": "<|endoftext|>"
+  "tokenizer_class": "TokenizersBackend"
 }