latest.json: checkpoints/1701013k (step 68,155)

README.md

@@ -30,7 +30,7 @@ deliberately derived from it: Nirca began as a port of mini-AGI to MLX.
 ➡️ **Try it in your browser (WebGPU):** https://tg-techie-agents.github.io/nirca-mini-webgpu/mini/
 
 This card is written by program from the checkpoint's own record, at step
-62,895 (1,217,308,397 tokens seen).
+68,155 (1,701,013,761 tokens seen).
 
 ## Model details
 
@@ -95,6 +95,10 @@ Across its training, this checkpoint and the runs it was continued from saw:
 - Open chat datasets.
 - An open reasoning dataset.
 - Synthetic chats from open-weight models, written to give the model its character.
+- A small set of private chats, with names, places, contact details and other personal details replaced by placeholders before training.
+- Plus 2 datasets not yet public.
+
+Its training data includes text under CC BY 4.0 and CC BY-NC-SA 4.0.
 
 ## Data and training workflow
 
@@ -103,16 +107,17 @@ Each step is listed because the run's records show it was done:
 1. **ChatML.** Every conversation is rendered in ChatML (`<|im_start|>role` … `<|im_end|>`).
 2. **Splits by content.** Before training, conversations are split into training, validation and test by their content, so one question is never in two splits, and the held-out splits are kept apart from the training machine.
 3. **Length filter.** Chat sets are cut to conversations whose whole text fits 2,048 bytes, the model's context.
-4. **Identifier check.** Synthetic chats that name a known person's identifier are dropped.
-5. **Decontamination.** A training conversation that shares an exact turn or an 8-gram with a validation or test set is left out (after PaLM's 8-gram overlap test).
-6. **Loss on the assistant's turns only**, from step 61,181: the model learns to answer, not to write the user's turns.
-7. **Quantization-aware training in ternary**, from step 14,655 (29,497,375 tokens) on; the steps before it record no weight format. The published weights are the trained ones, not quantized after training.
+4. **Anonymisation.** In the private chats, names, places, contact details and other personal details are replaced by typed placeholders, and the result is scanned again before training.
+5. **Identifier check.** Synthetic chats that name a known person's identifier are dropped.
+6. **Decontamination.** A training conversation that shares an exact turn or an 8-gram with a validation or test set is left out (after PaLM's 8-gram overlap test).
+7. **Loss on the assistant's turns only**, from step 61,181: the model learns to answer, not to write the user's turns.
+8. **Quantization-aware training in ternary**, from step 14,655 (29,497,375 tokens) on; the steps before it record no weight format. The published weights are the trained ones, not quantized after training.
 
 ## Training procedure
 
-- **Steps:** 62,895
-- **Tokens seen:** 1,217,308,397 (one token is one byte)
-- **Quantization-aware training:** from step 14,655 of 62,895, so for 98% of its 1,217,308,397 tokens
+- **Steps:** 68,155
+- **Tokens seen:** 1,701,013,761 (one token is one byte)
+- **Quantization-aware training:** from step 14,655 of 68,155, so for 98% of its 1,701,013,761 tokens
 - **Objective:** next-byte prediction; on chats, loss on the assistant's turns only
 - **Recurrence:** the number of passes is drawn per step during training, and
   the halting head learns when to stop

latest.json

@@ -1,6 +1,6 @@
 {
  "name": "nirca-mini",
- "step": 62895,
+ "step": 68155,
  "quant": "ternary",
  "quant_scope": "all",
  "n_experts": 32,
@@ -40,17 +40,17 @@
   "pool_depth": 1,
   "pool_max": 32
  },
- "tokens": 1217308397,
- "checkpoint": "checkpoints/1217308k/",
+ "tokens": 1701013761,
+ "checkpoint": "checkpoints/1701013k/",
  "files": {
   "config.json": {
-   "url": "https://huggingface.co/nirca/nirca-mini/resolve/main/checkpoints/1217308k/config.json",
-   "sha256": "ad212527afde007ef4312f735ac574cf2cd52b90891fc5ba33e1f29aa7f0c12a",
+   "url": "https://huggingface.co/nirca/nirca-mini/resolve/main/checkpoints/1701013k/config.json",
+   "sha256": "2f6597aaa8d4461706b3351f33d88844770797d066a5a8e163fe7553ae9d50f2",
    "bytes": 873
   },
   "model.safetensors": {
-   "url": "https://huggingface.co/nirca/nirca-mini/resolve/main/checkpoints/1217308k/model.safetensors",
-   "sha256": "c1e10118a19500e4574589b8b534a05333d0fcdbc573445ee50e3cbc36f36b88",
+   "url": "https://huggingface.co/nirca/nirca-mini/resolve/main/checkpoints/1701013k/model.safetensors",
+   "sha256": "36557c45f079bec3e6858982d4616c24c0f350c09d9a8acb41f95f58a65a5adf",
    "bytes": 32638048
   }
  }

latest/config.json

@@ -1,7 +1,7 @@
 {
  "name": "nirca-mini",
- "step": 62895,
- "tokens": 1217308397,
+ "step": 68155,
+ "tokens": 1701013761,
  "quant": "ternary",
  "quant_scope": "all",
  "n_experts": 32,

latest/model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:c1e10118a19500e4574589b8b534a05333d0fcdbc573445ee50e3cbc36f36b88
+oid sha256:36557c45f079bec3e6858982d4616c24c0f350c09d9a8acb41f95f58a65a5adf
 size 32638048