diff --git a/LICENSE b/LICENSE new file mode 100644 index 0000000000000000000000000000000000000000..f08de2d048baee881e28a3aa5b87c2c2c9f8e320 --- /dev/null +++ b/LICENSE @@ -0,0 +1,21 @@ +MIT License + +Copyright (c) 2026 AbstractGym contributors + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/README.md b/README.md new file mode 100644 index 0000000000000000000000000000000000000000..b6fea2aaa9db22377cdf4ec5a893f7404533a55a --- /dev/null +++ b/README.md @@ -0,0 +1,64 @@ +--- +license: mit +language: en +library_name: pytorch +tags: +- abstractgym +- research +- transformers +--- + +# AbstractGym tiny controllers + +Twelve independently trained small Transformers: role-mapped step and direct-answer controls (A6), a separate role-mapped full-trace model (X4), and a raw-symbol step ablation (X6). Seeds 0, 1, 2 for each family. These are full model weights, not adapters. Each folder has model.safetensors, config.json, training.json, results.json, and a conversion verification record. + +## Input representation and split + +The step, direct, and trace controllers receive symbols already mapped to roles. The harness supplies this abstraction; perfect step execution does not demonstrate learned raw-symbol binding. **Exception: raw-step (X6)** receives printable raw-symbol IDs and a supplied symbol map; it was explicitly designed to remove role preprocessing. + +A6 development uses 367 trajectories on uvwxy, depths 0–4. Step training deduplicates the prefix states to 36 role observations; direct training uses all 367 membership examples with balanced label sampling. Trace training uses the same development trajectories with balanced membership sampling. Raw-step trains on the 36 raw observations of the development symbol map. The original held-out set has 92 cases across two different alphabets at depths 0/1/2/3/4/8/16. Step models also have 32 extra cases at depths 32/64 and an exhaustive 36-state check. Raw-step evaluates 108 states across the three supplied maps. + +## Architecture and optimization + +All: two Transformer layers, width 64, four heads, feed-forward width 128, GELU, dropout zero, pre-norm, sinusoidal positions (maximum 256), FP32. Step/direct use bidirectional attention and CLS classification; trace/raw-step use causal attention and autoregressive outputs. AdamW LR .001, weight decay .01, clipping 1, cosine decay to 1e-5. Step: 800 full-batch updates on 36 states. Direct/trace: 1,500 updates, balanced batch 64. Raw-step: 800 full-batch updates on 36 states. Exact per-seed metadata and vocabularies are supplied. + +## Exact held-out results + +| Family | Seed | Correct / cases | +| --- | --- | --- | +| step | 0 | 92/92 | +| step | 1 | 92/92 | +| step | 2 | 92/92 | +| direct | 0 | 76/92 | +| direct | 1 | 80/92 | +| direct | 2 | 76/92 | +| trace | 0 | 60/92 | +| trace | 1 | 60/92 | +| trace | 2 | 60/92 | +| raw-step | 0 | 0/92 | +| raw-step | 1 | 0/92 | +| raw-step | 2 | 0/92 | + +Step/raw-step scores require complete live execution; direct scores are binary membership; trace scores require the entire generated action sequence. These are different tasks and interfaces. All failures remain in the denominator. Results are copied from `docs/results/2026-10-03-a6`, X4 metadata, and `docs/results/2026-10-04-x6`. Detailed auxiliary results remain in each folder. + +## Conversion and loading + +Original local state dictionaries were checked against each run’s frozen.json, the tiny runs’ original checkpoint hash manifest. Tensor values remain bitwise identical after safetensors serialization. Reloading into the original architecture reproduced every saved evaluation prediction, including teacher-forced trace outputs, corrupted-history continuations, exhaustive states, and extra-depth step trajectories. No training or evaluation selection was performed during conversion. See conversion_verification.json for exact counts and hashes. + +Clone [the public code on main](https://github.com/flavianv/abstractgym-public/tree/main), install PyTorch and safetensors, and set `PYTHONPATH=src:scripts`: + +```python +from huggingface_hub import hf_hub_download +from safetensors.torch import load_file +from abstractgym.a6_model import TinyTransformer +path = hf_hub_download("flavianv/abstractgym-tiny-controllers", "step/seed-0/model.safetensors") +model = TinyTransformer() +model.load_state_dict(load_file(path), strict=True) +model.eval() +``` + +For direct use `TinyTransformer(outputs=2)`; for trace import `TraceTransformer` from `run_a6_tiny_trace`; for raw-step import `RawTransformer` from `run_a6_x6`. Use the matching encoder and vocabulary recorded in config.json and implemented in the public source. These are custom PyTorch models, not root-level AutoModel checkpoints. + +## Limitations and license + +Three seeds per family. The role-mapped models depend on privileged preprocessing, and the raw-symbol ablation does not transfer reliably. Synthetic finite tests, bounded positions, and an external state machine do not prove general computation. MIT license. No pickle files or optimizer states are published. Code: [flavianv/abstractgym-public, main](https://github.com/flavianv/abstractgym-public/tree/main). diff --git a/direct/seed-0/config.json b/direct/seed-0/config.json new file mode 100644 index 0000000000000000000000000000000000000000..29f1ce29cc79ab70936fbe99d7609feb3eb4ef75 --- /dev/null +++ b/direct/seed-0/config.json @@ -0,0 +1,36 @@ +{ + "actions": [ + "PUSH_0", + "PUSH_1", + "CENTER", + "POP", + "ACCEPT", + "REJECT" + ], + "code_branch": "main", + "dropout": 0, + "dtype": "float32", + "family": "direct", + "feedforward": 128, + "heads": 4, + "layers": 2, + "model_class": "abstractgym.a6_model.TinyTransformer", + "outputs": 2, + "position_limit": 256, + "seed": 0, + "vocabulary": [ + "PAD", + "CLS", + "opening", + "closing", + "OPEN_0", + "OPEN_1", + "CLOSE_0", + "CLOSE_1", + "CENTER", + "END", + "EMPTY", + "DIRECT" + ], + "width": 64 +} diff --git a/direct/seed-0/conversion_verification.json b/direct/seed-0/conversion_verification.json new file mode 100644 index 0000000000000000000000000000000000000000..1350f5b03b8d29ff0ebb6a32ad3a226b9e840b96 --- /dev/null +++ b/direct/seed-0/conversion_verification.json @@ -0,0 +1,12 @@ +{ + "all_tensors_bitwise_equal": true, + "family": "direct", + "published_sha256": "387fc671b5ee2669d371f3f10094fba55c8c5f7e31d027065d2c8dcc977c455a", + "saved_predictions_reproduced": { + "held_out": 92 + }, + "seed": 0, + "source_hash_manifest": "frozen.json", + "source_sha256": "39bf28b49785115db3ca0b58a584230aff6db09361ee79619ba6e7fca8bcb8cd", + "torch_version": "2.14.1" +} diff --git a/direct/seed-0/model.safetensors b/direct/seed-0/model.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..4dfc664343ef3b6e281955c3e3f7d61e09fa97f1 --- /dev/null +++ b/direct/seed-0/model.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:387fc671b5ee2669d371f3f10094fba55c8c5f7e31d027065d2c8dcc977c455a +size 340216 diff --git a/direct/seed-0/results.json b/direct/seed-0/results.json new file mode 100644 index 0000000000000000000000000000000000000000..a309ae67a3da8c033b99c98059018caf57dff079 --- /dev/null +++ b/direct/seed-0/results.json @@ -0,0 +1,8 @@ +{ + "direct_answer": [ + 76, + 92 + ], + "direct_answer_model": "separate matched architecture; end-to-end development-label training", + "seed": 0 +} diff --git a/direct/seed-0/training.json b/direct/seed-0/training.json new file mode 100644 index 0000000000000000000000000000000000000000..d0a3027ea8a68c3e78995f9570fbe703e9901151 --- /dev/null +++ b/direct/seed-0/training.json @@ -0,0 +1,63 @@ +{ + "plan": { + "architecture": { + "attention": "bidirectional encoder, CLS readout", + "dropout": 0, + "feedforward": 128, + "heads": 4, + "layers": 2, + "position": "fixed sinusoidal", + "width": 64 + }, + "device": "cpu", + "dtype": "float32", + "git_revision": "39bca8d1f7dd0324010f76d6eb44a68ea719d572", + "hardware": "arm", + "optimizer": { + "clip_norm": 1.0, + "lr": 0.001, + "name": "AdamW", + "schedule": "cosine to 1e-5", + "weight_decay": 0.01 + }, + "packages": { + "torch": "2.14.1" + }, + "platform": "macOS-26.7-arm64-arm-64bit", + "python": "3.12.13", + "seeds": [ + 0, + 1, + 2 + ], + "source_sha256": { + "scripts/run_a6_tiny.py": "62a632c65003a10c5f35b1a586f7a62cfae3605db0c39d9ae48914367ea7aa1e", + "src/abstractgym/a6.py": "975b3d76fc9503e11e8f0ad8c57aa27162250298d4316d949145e6df80dbbfe2", + "src/abstractgym/a6_model.py": "d13c0b913ba39cd3c8371147d2495b185c24d8d15225cd68bc11e1aa2ebd72f7", + "src/abstractgym/brackets.py": "9fa8b835c0aac8fa26177b032f1fc07e1bb0964f7ec01fd0f83ad11b74e0550f" + }, + "threads": 2, + "training": { + "deduplicated_step_examples": 36, + "direct_examples": 367, + "direct_positive": 31, + "direct_sampling": "balanced labels, replacement; separate matched model", + "max_depth": 4, + "raw_step_examples": 2305, + "symbols": "uvwxy", + "trajectories": 367 + }, + "training_hash": "8fa8728efdda11e741e8ec16c2e0267cf0fd321e849119caea177055d15e5df0" + }, + "training": { + "batch_size": 64, + "final_loss": 0.005661265458911657, + "parameters": 67970, + "seconds": 5.530814791000012, + "steps": 1500, + "train_accuracy": [ + 366, + 367 + ] + } +} diff --git a/direct/seed-1/config.json b/direct/seed-1/config.json new file mode 100644 index 0000000000000000000000000000000000000000..0868fd453755b3fd429940f3861d5fddef1a4a49 --- /dev/null +++ b/direct/seed-1/config.json @@ -0,0 +1,36 @@ +{ + "actions": [ + "PUSH_0", + "PUSH_1", + "CENTER", + "POP", + "ACCEPT", + "REJECT" + ], + "code_branch": "main", + "dropout": 0, + "dtype": "float32", + "family": "direct", + "feedforward": 128, + "heads": 4, + "layers": 2, + "model_class": "abstractgym.a6_model.TinyTransformer", + "outputs": 2, + "position_limit": 256, + "seed": 1, + "vocabulary": [ + "PAD", + "CLS", + "opening", + "closing", + "OPEN_0", + "OPEN_1", + "CLOSE_0", + "CLOSE_1", + "CENTER", + "END", + "EMPTY", + "DIRECT" + ], + "width": 64 +} diff --git a/direct/seed-1/conversion_verification.json b/direct/seed-1/conversion_verification.json new file mode 100644 index 0000000000000000000000000000000000000000..c6fe049111f7838f8cf854830b2ed46cdc22f61b --- /dev/null +++ b/direct/seed-1/conversion_verification.json @@ -0,0 +1,12 @@ +{ + "all_tensors_bitwise_equal": true, + "family": "direct", + "published_sha256": "540f302b732386c378e521a5ff110a28fd056686ba65d23416cf0031017a54ae", + "saved_predictions_reproduced": { + "held_out": 92 + }, + "seed": 1, + "source_hash_manifest": "frozen.json", + "source_sha256": "07445f6621896c71053a5e64ba61b78c430954a840575bba113e7aeaef462a43", + "torch_version": "2.14.1" +} diff --git a/direct/seed-1/model.safetensors b/direct/seed-1/model.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..d31d7f3a5a969bdfc2a19b8ca249f47452276cd5 --- /dev/null +++ b/direct/seed-1/model.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:540f302b732386c378e521a5ff110a28fd056686ba65d23416cf0031017a54ae +size 340216 diff --git a/direct/seed-1/results.json b/direct/seed-1/results.json new file mode 100644 index 0000000000000000000000000000000000000000..ac5b2f64f4103046845331cb97abb30cd17a1b6f --- /dev/null +++ b/direct/seed-1/results.json @@ -0,0 +1,8 @@ +{ + "direct_answer": [ + 80, + 92 + ], + "direct_answer_model": "separate matched architecture; end-to-end development-label training", + "seed": 1 +} diff --git a/direct/seed-1/training.json b/direct/seed-1/training.json new file mode 100644 index 0000000000000000000000000000000000000000..0ac212c0f7030b5a986da37c2e2100785ba5d186 --- /dev/null +++ b/direct/seed-1/training.json @@ -0,0 +1,63 @@ +{ + "plan": { + "architecture": { + "attention": "bidirectional encoder, CLS readout", + "dropout": 0, + "feedforward": 128, + "heads": 4, + "layers": 2, + "position": "fixed sinusoidal", + "width": 64 + }, + "device": "cpu", + "dtype": "float32", + "git_revision": "39bca8d1f7dd0324010f76d6eb44a68ea719d572", + "hardware": "arm", + "optimizer": { + "clip_norm": 1.0, + "lr": 0.001, + "name": "AdamW", + "schedule": "cosine to 1e-5", + "weight_decay": 0.01 + }, + "packages": { + "torch": "2.14.1" + }, + "platform": "macOS-26.7-arm64-arm-64bit", + "python": "3.12.13", + "seeds": [ + 0, + 1, + 2 + ], + "source_sha256": { + "scripts/run_a6_tiny.py": "62a632c65003a10c5f35b1a586f7a62cfae3605db0c39d9ae48914367ea7aa1e", + "src/abstractgym/a6.py": "975b3d76fc9503e11e8f0ad8c57aa27162250298d4316d949145e6df80dbbfe2", + "src/abstractgym/a6_model.py": "d13c0b913ba39cd3c8371147d2495b185c24d8d15225cd68bc11e1aa2ebd72f7", + "src/abstractgym/brackets.py": "9fa8b835c0aac8fa26177b032f1fc07e1bb0964f7ec01fd0f83ad11b74e0550f" + }, + "threads": 2, + "training": { + "deduplicated_step_examples": 36, + "direct_examples": 367, + "direct_positive": 31, + "direct_sampling": "balanced labels, replacement; separate matched model", + "max_depth": 4, + "raw_step_examples": 2305, + "symbols": "uvwxy", + "trajectories": 367 + }, + "training_hash": "8fa8728efdda11e741e8ec16c2e0267cf0fd321e849119caea177055d15e5df0" + }, + "training": { + "batch_size": 64, + "final_loss": 0.00023508977028541267, + "parameters": 67970, + "seconds": 5.903513042001578, + "steps": 1500, + "train_accuracy": [ + 367, + 367 + ] + } +} diff --git a/direct/seed-2/config.json b/direct/seed-2/config.json new file mode 100644 index 0000000000000000000000000000000000000000..dd651ede85df554bbcff5f120dfae7e03d1a780c --- /dev/null +++ b/direct/seed-2/config.json @@ -0,0 +1,36 @@ +{ + "actions": [ + "PUSH_0", + "PUSH_1", + "CENTER", + "POP", + "ACCEPT", + "REJECT" + ], + "code_branch": "main", + "dropout": 0, + "dtype": "float32", + "family": "direct", + "feedforward": 128, + "heads": 4, + "layers": 2, + "model_class": "abstractgym.a6_model.TinyTransformer", + "outputs": 2, + "position_limit": 256, + "seed": 2, + "vocabulary": [ + "PAD", + "CLS", + "opening", + "closing", + "OPEN_0", + "OPEN_1", + "CLOSE_0", + "CLOSE_1", + "CENTER", + "END", + "EMPTY", + "DIRECT" + ], + "width": 64 +} diff --git a/direct/seed-2/conversion_verification.json b/direct/seed-2/conversion_verification.json new file mode 100644 index 0000000000000000000000000000000000000000..0d3dde009c5c16b7a74c48377dda5bb1f00b95ef --- /dev/null +++ b/direct/seed-2/conversion_verification.json @@ -0,0 +1,12 @@ +{ + "all_tensors_bitwise_equal": true, + "family": "direct", + "published_sha256": "cf8c716286a9d5abc063a423e3c44c379e1b7e132b28108a6a51a683c0e6ae3f", + "saved_predictions_reproduced": { + "held_out": 92 + }, + "seed": 2, + "source_hash_manifest": "frozen.json", + "source_sha256": "5e14beeb2d5e742967d56c86677e37a4b0b8d2a1097d3c1d50f78d71fe29a782", + "torch_version": "2.14.1" +} diff --git a/direct/seed-2/model.safetensors b/direct/seed-2/model.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..844a9ffb164b945250d962871ffefc3f7ba2800b --- /dev/null +++ b/direct/seed-2/model.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:cf8c716286a9d5abc063a423e3c44c379e1b7e132b28108a6a51a683c0e6ae3f +size 340216 diff --git a/direct/seed-2/results.json b/direct/seed-2/results.json new file mode 100644 index 0000000000000000000000000000000000000000..cd1649c668640d366eb0c696ec866585bf0c05c4 --- /dev/null +++ b/direct/seed-2/results.json @@ -0,0 +1,8 @@ +{ + "direct_answer": [ + 76, + 92 + ], + "direct_answer_model": "separate matched architecture; end-to-end development-label training", + "seed": 2 +} diff --git a/direct/seed-2/training.json b/direct/seed-2/training.json new file mode 100644 index 0000000000000000000000000000000000000000..5737a1a60f2d68368cb6006a905ab52d14ff9214 --- /dev/null +++ b/direct/seed-2/training.json @@ -0,0 +1,63 @@ +{ + "plan": { + "architecture": { + "attention": "bidirectional encoder, CLS readout", + "dropout": 0, + "feedforward": 128, + "heads": 4, + "layers": 2, + "position": "fixed sinusoidal", + "width": 64 + }, + "device": "cpu", + "dtype": "float32", + "git_revision": "39bca8d1f7dd0324010f76d6eb44a68ea719d572", + "hardware": "arm", + "optimizer": { + "clip_norm": 1.0, + "lr": 0.001, + "name": "AdamW", + "schedule": "cosine to 1e-5", + "weight_decay": 0.01 + }, + "packages": { + "torch": "2.14.1" + }, + "platform": "macOS-26.7-arm64-arm-64bit", + "python": "3.12.13", + "seeds": [ + 0, + 1, + 2 + ], + "source_sha256": { + "scripts/run_a6_tiny.py": "62a632c65003a10c5f35b1a586f7a62cfae3605db0c39d9ae48914367ea7aa1e", + "src/abstractgym/a6.py": "975b3d76fc9503e11e8f0ad8c57aa27162250298d4316d949145e6df80dbbfe2", + "src/abstractgym/a6_model.py": "d13c0b913ba39cd3c8371147d2495b185c24d8d15225cd68bc11e1aa2ebd72f7", + "src/abstractgym/brackets.py": "9fa8b835c0aac8fa26177b032f1fc07e1bb0964f7ec01fd0f83ad11b74e0550f" + }, + "threads": 2, + "training": { + "deduplicated_step_examples": 36, + "direct_examples": 367, + "direct_positive": 31, + "direct_sampling": "balanced labels, replacement; separate matched model", + "max_depth": 4, + "raw_step_examples": 2305, + "symbols": "uvwxy", + "trajectories": 367 + }, + "training_hash": "8fa8728efdda11e741e8ec16c2e0267cf0fd321e849119caea177055d15e5df0" + }, + "training": { + "batch_size": 64, + "final_loss": 0.00016993776080198586, + "parameters": 67970, + "seconds": 5.7849729169975035, + "steps": 1500, + "train_accuracy": [ + 367, + 367 + ] + } +} diff --git a/manifest.json b/manifest.json new file mode 100644 index 0000000000000000000000000000000000000000..c4398caa4dd155a94e3e45dc20281b22810fa576 --- /dev/null +++ b/manifest.json @@ -0,0 +1,250 @@ +{ + "LICENSE": { + "bytes": 1081, + "sha256": "36a8634aa63459be88ffbc508c73cb06fa8cfb8d050db23f0e7fd01f55ec3abb" + }, + "README.md": { + "bytes": 4658, + "sha256": "20853e0339b311e29032cf8a620e511c2b81f00344853f5790c2df0fa2694dfe" + }, + "direct/seed-0/config.json": { + "bytes": 549, + "sha256": "dcba247b7ba2d2b0c8772e9b4361c83953393915a1b865380f85f07ff872c06e" + }, + "direct/seed-0/conversion_verification.json": { + "bytes": 382, + "sha256": "cf10ca3f07575269d0ead4ae48cf2c5616de439aea32ae3e7df21976d1a608ab" + }, + "direct/seed-0/model.safetensors": { + "bytes": 340216, + "sha256": "387fc671b5ee2669d371f3f10094fba55c8c5f7e31d027065d2c8dcc977c455a" + }, + "direct/seed-0/results.json": { + "bytes": 154, + "sha256": "69d311d44e0b33220d53b6c0ca918f90a190a97d49d7f5204380cb965475b8e8" + }, + "direct/seed-0/training.json": { + "bytes": 1768, + "sha256": "7c77c85e0c32ad147ac294df837d014eba5ea70d590831e8fc908c77449059da" + }, + "direct/seed-1/config.json": { + "bytes": 549, + "sha256": "53a9951238e89616cd290e76fe18284c34fe93f327030686a5df15893d871bcc" + }, + "direct/seed-1/conversion_verification.json": { + "bytes": 382, + "sha256": "81d3787bc5f7b1b724202ba2df40ebc62edbd9dcb73a4c146b394b6c3150e46d" + }, + "direct/seed-1/model.safetensors": { + "bytes": 340216, + "sha256": "540f302b732386c378e521a5ff110a28fd056686ba65d23416cf0031017a54ae" + }, + "direct/seed-1/results.json": { + "bytes": 154, + "sha256": "a4824c54db8b4f93cf0d23db2ef813758dc7d60780a02f0b53b5223f86e6124b" + }, + "direct/seed-1/training.json": { + "bytes": 1770, + "sha256": "c0a1329d04e5ecabdd4d773ed8dca3ffb2f24d0a15880bd16ea2aab3dc58534c" + }, + "direct/seed-2/config.json": { + "bytes": 549, + "sha256": "33844a3e45bf7b19f265ba2d94752c0abf06dfbf174ef0681eb14cbf5e75dc37" + }, + "direct/seed-2/conversion_verification.json": { + "bytes": 382, + "sha256": "23363eb35375ff6fab17673a2f855ff86c13cf88a611c976c7adf4ed4ac08770" + }, + "direct/seed-2/model.safetensors": { + "bytes": 340216, + "sha256": "cf8c716286a9d5abc063a423e3c44c379e1b7e132b28108a6a51a683c0e6ae3f" + }, + "direct/seed-2/results.json": { + "bytes": 154, + "sha256": "6603ce84ea30162446f5d7237a689224efc0f5b7bedbfb4555839f776980bb7c" + }, + "direct/seed-2/training.json": { + "bytes": 1771, + "sha256": "da825931beab887b09652975f0b5ca15adc15e2059641870e9335a0b0615d8fa" + }, + "raw-step/seed-0/config.json": { + "bytes": 1399, + "sha256": "8bd04099a71d6ca57d29c38e569ba5aa3ee66b6b648e492cfa2e6c2ad2112d78" + }, + "raw-step/seed-0/conversion_verification.json": { + "bytes": 407, + "sha256": "03a5f3006fbc8c9e0c0c602bc3e251819c7ae49e2f57a19089fd0f36ba10b1ad" + }, + "raw-step/seed-0/model.safetensors": { + "bytes": 392360, + "sha256": "88a2b23e78cbd8ef1c0f09bd9972029827dc9529d49284327b5f83318827f21d" + }, + "raw-step/seed-0/results.json": { + "bytes": 4543, + "sha256": "27921d9996d95393d8cef298e06f2749c0732ae31febc5d09fb7abc33257c98e" + }, + "raw-step/seed-0/training.json": { + "bytes": 2480, + "sha256": "89ac1dfbd738c8f575f00832270ff9bee8ddd8046bf17a2563c6abf628cb7dab" + }, + "raw-step/seed-1/config.json": { + "bytes": 1399, + "sha256": "bb4442cbae671722b0bae353850795bdfb51ea9940a6d875421044e8d9f7a5c9" + }, + "raw-step/seed-1/conversion_verification.json": { + "bytes": 407, + "sha256": "fa2739568210c1fbe541c02c3ede130fbabe093649e4aeaaf32fffb7fae3ba96" + }, + "raw-step/seed-1/model.safetensors": { + "bytes": 392360, + "sha256": "c0d4940618fce954f253a157d7167b27fabd561982dc73d174be29ab02c1278a" + }, + "raw-step/seed-1/results.json": { + "bytes": 4543, + "sha256": "58f3793ed88faa59c31a2a84986b39d69a537fa4306d6c9ddadc02f90ac79ab3" + }, + "raw-step/seed-1/training.json": { + "bytes": 2480, + "sha256": "9559146efcde113032b8d4352e8d8b3118b909bcee7ef95ac2a1c76367733fa9" + }, + "raw-step/seed-2/config.json": { + "bytes": 1399, + "sha256": "890e3426a85bd3a9e6f2c7daf85c04caa81201a2e4947784e01e3fe3f8e6d107" + }, + "raw-step/seed-2/conversion_verification.json": { + "bytes": 407, + "sha256": "ca5d50a980943cf30431fda32ef49507606577dd2c04f252a8512533e996a044" + }, + "raw-step/seed-2/model.safetensors": { + "bytes": 392360, + "sha256": "a96e32f0a540aaf176a0ec8479c3f9b18c9a43fbea392f1049ea028f3f229f87" + }, + "raw-step/seed-2/results.json": { + "bytes": 4542, + "sha256": "b1c01769f6431e6663c7f2876dd4f48278d19cff2df6d18c5186971d452b8f5e" + }, + "raw-step/seed-2/training.json": { + "bytes": 2479, + "sha256": "df4ecadf41931b5c4ea2b40f4a36e64bae0208c1f513b820de4b2e51d2621b07" + }, + "step/seed-0/config.json": { + "bytes": 547, + "sha256": "7eff724be08aeac497e22428c16b4b9819863b566405703e18a734460d69a7c9" + }, + "step/seed-0/conversion_verification.json": { + "bytes": 419, + "sha256": "c42d87c2fd15b8e28b5cd3ec405858c1656b4b77709d6638e3f0fe0ef93cefd0" + }, + "step/seed-0/model.safetensors": { + "bytes": 341256, + "sha256": "baee7f0d12b2a70b1e9604c2d7d9a80554f156a2af111d3966f37cea9f21357f" + }, + "step/seed-0/results.json": { + "bytes": 5108, + "sha256": "d2c1bf14a8b50ae7b2432472e0534d87239f4c67bf2ccadc4ea527e079f98cc3" + }, + "step/seed-0/training.json": { + "bytes": 1767, + "sha256": "cfdbe38e6ab8be1dc283d71e55ee7871bdd90f6aad37db17ff01ac05800b3bb2" + }, + "step/seed-1/config.json": { + "bytes": 547, + "sha256": "cc33223113373fba31603162fd9435927cf54b967ebb3a7a1f1c0cf3c5f2773e" + }, + "step/seed-1/conversion_verification.json": { + "bytes": 419, + "sha256": "f023fd87d2e3e9bf4e848b348a0ff7a931555ea37c32131b76fd7f966103c1f2" + }, + "step/seed-1/model.safetensors": { + "bytes": 341256, + "sha256": "2d25c9513d2378fa151f79d4cb4619a2e0d401f7825bd7b6c5a6c13769b37f5b" + }, + "step/seed-1/results.json": { + "bytes": 5107, + "sha256": "11e1f39bbbae97bb43a3aa607f03de17873ffb36ec461b2d83dc30e71bfe19f1" + }, + "step/seed-1/training.json": { + "bytes": 1766, + "sha256": "e506968b02ba32835f5d89e274b02f68f58a7af38144c475c4e8d5dc7a2777cf" + }, + "step/seed-2/config.json": { + "bytes": 547, + "sha256": "cb49d14caf8bb7abd182df0c8a447322918fcec9126025be81a40a564767bd62" + }, + "step/seed-2/conversion_verification.json": { + "bytes": 419, + "sha256": "267889e561e794217c7be6476aea192c39b9481f4dfccccbee2b10bc72017963" + }, + "step/seed-2/model.safetensors": { + "bytes": 341256, + "sha256": "7a2d655c83a05f98193abb506ab5e7d181d4463fd21a6dbc1934b43cc7abf213" + }, + "step/seed-2/results.json": { + "bytes": 5108, + "sha256": "f69f7d4971418960aa05301e873dca3d05318f8bc4acb86de97f548ffee7f4ef" + }, + "step/seed-2/training.json": { + "bytes": 1767, + "sha256": "d9144554c88c324759c5e9783da13ce97adf40bcc6d39b1e6f95eee4455a9856" + }, + "trace/seed-0/config.json": { + "bytes": 579, + "sha256": "ed06ce0b09a9e153f047818e927ae6f81ec16b0f8339454ed37988e877690a89" + }, + "trace/seed-0/conversion_verification.json": { + "bytes": 440, + "sha256": "00cdf1dc3ca0d73effc171d6645ee875fac173eaf66f26c37e913b829b420f5d" + }, + "trace/seed-0/model.safetensors": { + "bytes": 344364, + "sha256": "e506bfd329fcf518b60b9d904e29045cbb4521a8ee93fce9ab19884654afe35d" + }, + "trace/seed-0/results.json": { + "bytes": 1034, + "sha256": "92b7aa49f9981daf61871a4a5c2274d0fb89999fb43051bb518f4914bcec7c4e" + }, + "trace/seed-0/training.json": { + "bytes": 1088, + "sha256": "76189eed4d919df72f532f1fb70ed012a07b45e515e6d318c0b1be4bc884b4ea" + }, + "trace/seed-1/config.json": { + "bytes": 579, + "sha256": "fd49cb8f7a732f668be343d75a6994d3a19570798f95079d3e20028d24067a23" + }, + "trace/seed-1/conversion_verification.json": { + "bytes": 440, + "sha256": "ce9a4cd3e64a7e4404da34e9dff84126b828d68889c0f8515f99ba70f24bdfbd" + }, + "trace/seed-1/model.safetensors": { + "bytes": 344364, + "sha256": "82b4057ba67f34df95a23f117ded59c80be9c10fbd7d3bf85a5ac36d48dfc299" + }, + "trace/seed-1/results.json": { + "bytes": 1034, + "sha256": "fcd7d53af8393e8b362bf5ad1abc0ace880f04ed67f131463bb51b8a902f7d06" + }, + "trace/seed-1/training.json": { + "bytes": 1089, + "sha256": "3c1554a1e0d23cd63b165468f18fbbdf74ac8eb01738abeca3ebdab2483c10be" + }, + "trace/seed-2/config.json": { + "bytes": 579, + "sha256": "6de2e6f36fb39fad15016276144f0b6682f9dfcd6f1c8067d3a0134aa387ebdf" + }, + "trace/seed-2/conversion_verification.json": { + "bytes": 440, + "sha256": "7bbbd83b9bd1d4f68cf5d40abed77fcfb83944d49a3ea722052134fcc05222ff" + }, + "trace/seed-2/model.safetensors": { + "bytes": 344364, + "sha256": "f51b7be3c5995a693bad0e806a28481f7530533fcff44a52590ba99f2652d041" + }, + "trace/seed-2/results.json": { + "bytes": 1034, + "sha256": "b0e0e1bda3c8a6e08fdee845b23b3d10278f6dfbf5e45551f0e456ac76659094" + }, + "trace/seed-2/training.json": { + "bytes": 1089, + "sha256": "06e6e70d022c540a2f3556cf1e0e75667903fb681c9ac74b9fa5025779def80c" + } +} diff --git a/raw-step/seed-0/config.json b/raw-step/seed-0/config.json new file mode 100644 index 0000000000000000000000000000000000000000..09f45b56dc721feee68659aa5c86b72d699835c3 --- /dev/null +++ b/raw-step/seed-0/config.json @@ -0,0 +1,132 @@ +{ + "actions": [ + "PUSH_0", + "PUSH_1", + "CENTER", + "POP", + "ACCEPT", + "REJECT" + ], + "code_branch": "main", + "dropout": 0, + "dtype": "float32", + "family": "raw-step", + "feedforward": 128, + "heads": 4, + "layers": 2, + "model_class": "run_a6_x6.RawTransformer", + "outputs": 108, + "position_limit": 256, + "seed": 0, + "vocabulary": [ + "PAD", + "CLS", + "opening", + "closing", + "END", + "EMPTY", + "SEP", + "EOS", + "PUSH", + "CENTER", + "POP", + "ACCEPT", + "REJECT", + " ", + "!", + "\"", + "#", + "$", + "%", + "&", + "'", + "(", + ")", + "*", + "+", + ",", + "-", + ".", + "/", + "0", + "1", + "2", + "3", + "4", + "5", + "6", + "7", + "8", + "9", + ":", + ";", + "<", + "=", + ">", + "?", + "@", + "A", + "B", + "C", + "D", + "E", + "F", + "G", + "H", + "I", + "J", + "K", + "L", + "M", + "N", + "O", + "P", + "Q", + "R", + "S", + "T", + "U", + "V", + "W", + "X", + "Y", + "Z", + "[", + "\\", + "]", + "^", + "_", + "`", + "a", + "b", + "c", + "d", + "e", + "f", + "g", + "h", + "i", + "j", + "k", + "l", + "m", + "n", + "o", + "p", + "q", + "r", + "s", + "t", + "u", + "v", + "w", + "x", + "y", + "z", + "{", + "|", + "}", + "~" + ], + "width": 64 +} diff --git a/raw-step/seed-0/conversion_verification.json b/raw-step/seed-0/conversion_verification.json new file mode 100644 index 0000000000000000000000000000000000000000..f2ce8a25a82b3d100ee15a35bd09062bb0874e2f --- /dev/null +++ b/raw-step/seed-0/conversion_verification.json @@ -0,0 +1,13 @@ +{ + "all_tensors_bitwise_equal": true, + "family": "raw-step", + "published_sha256": "88a2b23e78cbd8ef1c0f09bd9972029827dc9529d49284327b5f83318827f21d", + "saved_predictions_reproduced": { + "exhaustive": 108, + "held_out": 92 + }, + "seed": 0, + "source_hash_manifest": "frozen.json", + "source_sha256": "95dc9cee6071309f370910fc3ae84c958d4f9656e642f5831b388c76524eab01", + "torch_version": "2.14.1" +} diff --git a/raw-step/seed-0/model.safetensors b/raw-step/seed-0/model.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..b36e394617c1d3fad255cc37bd9093c60167af05 --- /dev/null +++ b/raw-step/seed-0/model.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:88a2b23e78cbd8ef1c0f09bd9972029827dc9529d49284327b5f83318827f21d +size 392360 diff --git a/raw-step/seed-0/results.json b/raw-step/seed-0/results.json new file mode 100644 index 0000000000000000000000000000000000000000..d761b20659f6fe3ca3fc216c4119592bb9ef53ef --- /dev/null +++ b/raw-step/seed-0/results.json @@ -0,0 +1,283 @@ +{ + "action_accuracy": [ + 0, + 92 + ], + "condition": "tiny_raw_autoregressive", + "exhaustive": { + "([)]#": [ + 12, + 36 + ], + "abcde": [ + 15, + 36 + ], + "uvwxy": [ + 36, + 36 + ] + }, + "first_error_median": 1.0, + "held_out": { + "brackets": { + "by_depth": { + "0": [ + 0, + 2 + ], + "1": [ + 0, + 4 + ], + "16": [ + 0, + 8 + ], + "2": [ + 0, + 8 + ], + "3": [ + 0, + 8 + ], + "4": [ + 0, + 8 + ], + "8": [ + 0, + 8 + ] + }, + "interpretation": { + "0": "below label-blind", + "1": "below label-blind", + "16": "below label-blind", + "2": "below label-blind", + "3": "below label-blind", + "4": "below label-blind", + "8": "below label-blind" + } + }, + "letters": { + "by_depth": { + "0": [ + 0, + 2 + ], + "1": [ + 0, + 4 + ], + "16": [ + 0, + 8 + ], + "2": [ + 0, + 8 + ], + "3": [ + 0, + 8 + ], + "4": [ + 0, + 8 + ], + "8": [ + 0, + 8 + ] + }, + "interpretation": { + "0": "below label-blind", + "1": "below label-blind", + "16": "below label-blind", + "2": "below label-blind", + "3": "below label-blind", + "4": "below label-blind", + "8": "below label-blind" + } + }, + "total": [ + 0, + 92 + ] + }, + "invalid_action_calls": 0, + "label_blind": { + "always_accept": [ + 46, + 92 + ], + "always_reject": [ + 46, + 92 + ], + "by_cell": { + "brackets": { + "0": { + "always_accept": [ + 1, + 2 + ], + "always_reject": [ + 1, + 2 + ] + }, + "1": { + "always_accept": [ + 2, + 4 + ], + "always_reject": [ + 2, + 4 + ] + }, + "16": { + "always_accept": [ + 4, + 8 + ], + "always_reject": [ + 4, + 8 + ] + }, + "2": { + "always_accept": [ + 4, + 8 + ], + "always_reject": [ + 4, + 8 + ] + }, + "3": { + "always_accept": [ + 4, + 8 + ], + "always_reject": [ + 4, + 8 + ] + }, + "4": { + "always_accept": [ + 4, + 8 + ], + "always_reject": [ + 4, + 8 + ] + }, + "8": { + "always_accept": [ + 4, + 8 + ], + "always_reject": [ + 4, + 8 + ] + } + }, + "letters": { + "0": { + "always_accept": [ + 1, + 2 + ], + "always_reject": [ + 1, + 2 + ] + }, + "1": { + "always_accept": [ + 2, + 4 + ], + "always_reject": [ + 2, + 4 + ] + }, + "16": { + "always_accept": [ + 4, + 8 + ], + "always_reject": [ + 4, + 8 + ] + }, + "2": { + "always_accept": [ + 4, + 8 + ], + "always_reject": [ + 4, + 8 + ] + }, + "3": { + "always_accept": [ + 4, + 8 + ], + "always_reject": [ + 4, + 8 + ] + }, + "4": { + "always_accept": [ + 4, + 8 + ], + "always_reject": [ + 4, + 8 + ] + }, + "8": { + "always_accept": [ + 4, + 8 + ], + "always_reject": [ + 4, + 8 + ] + } + } + } + }, + "push_value_errors": 66, + "seed": 0, + "training": { + "loss": 0.0016285243909806013, + "parameters": 81004, + "seconds": 2.2738583749996906, + "seed": 0, + "train_accuracy": [ + 36, + 36 + ] + }, + "usage": { + "completion_tokens": 0, + "prompt_tokens": 0 + } +} diff --git a/raw-step/seed-0/training.json b/raw-step/seed-0/training.json new file mode 100644 index 0000000000000000000000000000000000000000..df8172c6c01802704a1ebd7e4feb935af6a705c2 --- /dev/null +++ b/raw-step/seed-0/training.json @@ -0,0 +1,148 @@ +{ + "plan": { + "architecture": "2-layer width-64 4-head Transformer; causal attention; autoregressive operation/value/EOS", + "batch": 36, + "clip": 1, + "domain": "three fixed supplied maps x 2 phases x 6 current symbols/end x 3 closing-symbol/empty tops", + "held_out_hash": "d2c7202ea08da83ad5398723a2cc4ce9d3175b7e5f7b84ce40aa529103e0d6e2", + "lr": 0.001, + "optimizer": "AdamW", + "output_limit": 3, + "prompt_loss_masked": true, + "schedule": "cosine to 1e-5", + "seeds": [ + 0, + 1, + 2 + ], + "source_sha256": { + "scripts/run_a6_x6.py": "b8ce548d5a55069483b46ed89da10f34bb9a3edf976b361069cd4d11692fd685", + "src/abstractgym/a6_model.py": "d13c0b913ba39cd3c8371147d2495b185c24d8d15225cd68bc11e1aa2ebd72f7", + "src/abstractgym/a6_raw.py": "df9c2be312f2290939871364baca738ac9a849dca855d049e304a6d0bb74e630" + }, + "steps": 800, + "torch": "2.14.1", + "training_hash": "11d5025e1246025f38e3ea840fd2a065ca73c2751cdf9de5f50005f4be7fa3a7", + "vocab": [ + "PAD", + "CLS", + "opening", + "closing", + "END", + "EMPTY", + "SEP", + "EOS", + "PUSH", + "CENTER", + "POP", + "ACCEPT", + "REJECT", + " ", + "!", + "\"", + "#", + "$", + "%", + "&", + "'", + "(", + ")", + "*", + "+", + ",", + "-", + ".", + "", + "0", + "1", + "2", + "3", + "4", + "5", + "6", + "7", + "8", + "9", + ":", + ";", + "<", + "=", + ">", + "?", + "@", + "A", + "B", + "C", + "D", + "E", + "F", + "G", + "H", + "I", + "J", + "K", + "L", + "M", + "N", + "O", + "P", + "Q", + "R", + "S", + "T", + "U", + "V", + "W", + "X", + "Y", + "Z", + "[", + "\\", + "]", + "^", + "_", + "`", + "a", + "b", + "c", + "d", + "e", + "f", + "g", + "h", + "i", + "j", + "k", + "l", + "m", + "n", + "o", + "p", + "q", + "r", + "s", + "t", + "u", + "v", + "w", + "x", + "y", + "z", + "{", + "|", + "}", + "~" + ], + "weight_decay": 0.01 + }, + "training": { + "loss": 0.0016285243909806013, + "parameters": 81004, + "seconds": 2.2738583749996906, + "seed": 0, + "train_accuracy": [ + 36, + 36 + ] + } +} diff --git a/raw-step/seed-1/config.json b/raw-step/seed-1/config.json new file mode 100644 index 0000000000000000000000000000000000000000..86d6b40f6b38abbf676e11a1c1d99f5ee7e05b21 --- /dev/null +++ b/raw-step/seed-1/config.json @@ -0,0 +1,132 @@ +{ + "actions": [ + "PUSH_0", + "PUSH_1", + "CENTER", + "POP", + "ACCEPT", + "REJECT" + ], + "code_branch": "main", + "dropout": 0, + "dtype": "float32", + "family": "raw-step", + "feedforward": 128, + "heads": 4, + "layers": 2, + "model_class": "run_a6_x6.RawTransformer", + "outputs": 108, + "position_limit": 256, + "seed": 1, + "vocabulary": [ + "PAD", + "CLS", + "opening", + "closing", + "END", + "EMPTY", + "SEP", + "EOS", + "PUSH", + "CENTER", + "POP", + "ACCEPT", + "REJECT", + " ", + "!", + "\"", + "#", + "$", + "%", + "&", + "'", + "(", + ")", + "*", + "+", + ",", + "-", + ".", + "/", + "0", + "1", + "2", + "3", + "4", + "5", + "6", + "7", + "8", + "9", + ":", + ";", + "<", + "=", + ">", + "?", + "@", + "A", + "B", + "C", + "D", + "E", + "F", + "G", + "H", + "I", + "J", + "K", + "L", + "M", + "N", + "O", + "P", + "Q", + "R", + "S", + "T", + "U", + "V", + "W", + "X", + "Y", + "Z", + "[", + "\\", + "]", + "^", + "_", + "`", + "a", + "b", + "c", + "d", + "e", + "f", + "g", + "h", + "i", + "j", + "k", + "l", + "m", + "n", + "o", + "p", + "q", + "r", + "s", + "t", + "u", + "v", + "w", + "x", + "y", + "z", + "{", + "|", + "}", + "~" + ], + "width": 64 +} diff --git a/raw-step/seed-1/conversion_verification.json b/raw-step/seed-1/conversion_verification.json new file mode 100644 index 0000000000000000000000000000000000000000..3a10bb3cd1771484990650d9f6b6b2d504ba24ab --- /dev/null +++ b/raw-step/seed-1/conversion_verification.json @@ -0,0 +1,13 @@ +{ + "all_tensors_bitwise_equal": true, + "family": "raw-step", + "published_sha256": "c0d4940618fce954f253a157d7167b27fabd561982dc73d174be29ab02c1278a", + "saved_predictions_reproduced": { + "exhaustive": 108, + "held_out": 92 + }, + "seed": 1, + "source_hash_manifest": "frozen.json", + "source_sha256": "7c83c18936b476968f6c373d1bc9a034eb361dd4f96c333222d0f5d9c0aea0f0", + "torch_version": "2.14.1" +} diff --git a/raw-step/seed-1/model.safetensors b/raw-step/seed-1/model.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..80a05271e94ec6756ee68799806ecf076b72d219 --- /dev/null +++ b/raw-step/seed-1/model.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:c0d4940618fce954f253a157d7167b27fabd561982dc73d174be29ab02c1278a +size 392360 diff --git a/raw-step/seed-1/results.json b/raw-step/seed-1/results.json new file mode 100644 index 0000000000000000000000000000000000000000..2f43c1957f40002520394b909da8d310371768c5 --- /dev/null +++ b/raw-step/seed-1/results.json @@ -0,0 +1,283 @@ +{ + "action_accuracy": [ + 0, + 92 + ], + "condition": "tiny_raw_autoregressive", + "exhaustive": { + "([)]#": [ + 23, + 36 + ], + "abcde": [ + 23, + 36 + ], + "uvwxy": [ + 36, + 36 + ] + }, + "first_error_median": 1.0, + "held_out": { + "brackets": { + "by_depth": { + "0": [ + 0, + 2 + ], + "1": [ + 0, + 4 + ], + "16": [ + 0, + 8 + ], + "2": [ + 0, + 8 + ], + "3": [ + 0, + 8 + ], + "4": [ + 0, + 8 + ], + "8": [ + 0, + 8 + ] + }, + "interpretation": { + "0": "below label-blind", + "1": "below label-blind", + "16": "below label-blind", + "2": "below label-blind", + "3": "below label-blind", + "4": "below label-blind", + "8": "below label-blind" + } + }, + "letters": { + "by_depth": { + "0": [ + 0, + 2 + ], + "1": [ + 0, + 4 + ], + "16": [ + 0, + 8 + ], + "2": [ + 0, + 8 + ], + "3": [ + 0, + 8 + ], + "4": [ + 0, + 8 + ], + "8": [ + 0, + 8 + ] + }, + "interpretation": { + "0": "below label-blind", + "1": "below label-blind", + "16": "below label-blind", + "2": "below label-blind", + "3": "below label-blind", + "4": "below label-blind", + "8": "below label-blind" + } + }, + "total": [ + 0, + 92 + ] + }, + "invalid_action_calls": 0, + "label_blind": { + "always_accept": [ + 46, + 92 + ], + "always_reject": [ + 46, + 92 + ], + "by_cell": { + "brackets": { + "0": { + "always_accept": [ + 1, + 2 + ], + "always_reject": [ + 1, + 2 + ] + }, + "1": { + "always_accept": [ + 2, + 4 + ], + "always_reject": [ + 2, + 4 + ] + }, + "16": { + "always_accept": [ + 4, + 8 + ], + "always_reject": [ + 4, + 8 + ] + }, + "2": { + "always_accept": [ + 4, + 8 + ], + "always_reject": [ + 4, + 8 + ] + }, + "3": { + "always_accept": [ + 4, + 8 + ], + "always_reject": [ + 4, + 8 + ] + }, + "4": { + "always_accept": [ + 4, + 8 + ], + "always_reject": [ + 4, + 8 + ] + }, + "8": { + "always_accept": [ + 4, + 8 + ], + "always_reject": [ + 4, + 8 + ] + } + }, + "letters": { + "0": { + "always_accept": [ + 1, + 2 + ], + "always_reject": [ + 1, + 2 + ] + }, + "1": { + "always_accept": [ + 2, + 4 + ], + "always_reject": [ + 2, + 4 + ] + }, + "16": { + "always_accept": [ + 4, + 8 + ], + "always_reject": [ + 4, + 8 + ] + }, + "2": { + "always_accept": [ + 4, + 8 + ], + "always_reject": [ + 4, + 8 + ] + }, + "3": { + "always_accept": [ + 4, + 8 + ], + "always_reject": [ + 4, + 8 + ] + }, + "4": { + "always_accept": [ + 4, + 8 + ], + "always_reject": [ + 4, + 8 + ] + }, + "8": { + "always_accept": [ + 4, + 8 + ], + "always_reject": [ + 4, + 8 + ] + } + } + } + }, + "push_value_errors": 22, + "seed": 1, + "training": { + "loss": 0.0016210851026698947, + "parameters": 81004, + "seconds": 1.9787468750000698, + "seed": 1, + "train_accuracy": [ + 36, + 36 + ] + }, + "usage": { + "completion_tokens": 0, + "prompt_tokens": 0 + } +} diff --git a/raw-step/seed-1/training.json b/raw-step/seed-1/training.json new file mode 100644 index 0000000000000000000000000000000000000000..ceb520d0074112c635befb951f59d31b7847c5cf --- /dev/null +++ b/raw-step/seed-1/training.json @@ -0,0 +1,148 @@ +{ + "plan": { + "architecture": "2-layer width-64 4-head Transformer; causal attention; autoregressive operation/value/EOS", + "batch": 36, + "clip": 1, + "domain": "three fixed supplied maps x 2 phases x 6 current symbols/end x 3 closing-symbol/empty tops", + "held_out_hash": "d2c7202ea08da83ad5398723a2cc4ce9d3175b7e5f7b84ce40aa529103e0d6e2", + "lr": 0.001, + "optimizer": "AdamW", + "output_limit": 3, + "prompt_loss_masked": true, + "schedule": "cosine to 1e-5", + "seeds": [ + 0, + 1, + 2 + ], + "source_sha256": { + "scripts/run_a6_x6.py": "b8ce548d5a55069483b46ed89da10f34bb9a3edf976b361069cd4d11692fd685", + "src/abstractgym/a6_model.py": "d13c0b913ba39cd3c8371147d2495b185c24d8d15225cd68bc11e1aa2ebd72f7", + "src/abstractgym/a6_raw.py": "df9c2be312f2290939871364baca738ac9a849dca855d049e304a6d0bb74e630" + }, + "steps": 800, + "torch": "2.14.1", + "training_hash": "11d5025e1246025f38e3ea840fd2a065ca73c2751cdf9de5f50005f4be7fa3a7", + "vocab": [ + "PAD", + "CLS", + "opening", + "closing", + "END", + "EMPTY", + "SEP", + "EOS", + "PUSH", + "CENTER", + "POP", + "ACCEPT", + "REJECT", + " ", + "!", + "\"", + "#", + "$", + "%", + "&", + "'", + "(", + ")", + "*", + "+", + ",", + "-", + ".", + "", + "0", + "1", + "2", + "3", + "4", + "5", + "6", + "7", + "8", + "9", + ":", + ";", + "<", + "=", + ">", + "?", + "@", + "A", + "B", + "C", + "D", + "E", + "F", + "G", + "H", + "I", + "J", + "K", + "L", + "M", + "N", + "O", + "P", + "Q", + "R", + "S", + "T", + "U", + "V", + "W", + "X", + "Y", + "Z", + "[", + "\\", + "]", + "^", + "_", + "`", + "a", + "b", + "c", + "d", + "e", + "f", + "g", + "h", + "i", + "j", + "k", + "l", + "m", + "n", + "o", + "p", + "q", + "r", + "s", + "t", + "u", + "v", + "w", + "x", + "y", + "z", + "{", + "|", + "}", + "~" + ], + "weight_decay": 0.01 + }, + "training": { + "loss": 0.0016210851026698947, + "parameters": 81004, + "seconds": 1.9787468750000698, + "seed": 1, + "train_accuracy": [ + 36, + 36 + ] + } +} diff --git a/raw-step/seed-2/config.json b/raw-step/seed-2/config.json new file mode 100644 index 0000000000000000000000000000000000000000..d2d7d08edd2a4ef087fabcd990b01c93f8db5828 --- /dev/null +++ b/raw-step/seed-2/config.json @@ -0,0 +1,132 @@ +{ + "actions": [ + "PUSH_0", + "PUSH_1", + "CENTER", + "POP", + "ACCEPT", + "REJECT" + ], + "code_branch": "main", + "dropout": 0, + "dtype": "float32", + "family": "raw-step", + "feedforward": 128, + "heads": 4, + "layers": 2, + "model_class": "run_a6_x6.RawTransformer", + "outputs": 108, + "position_limit": 256, + "seed": 2, + "vocabulary": [ + "PAD", + "CLS", + "opening", + "closing", + "END", + "EMPTY", + "SEP", + "EOS", + "PUSH", + "CENTER", + "POP", + "ACCEPT", + "REJECT", + " ", + "!", + "\"", + "#", + "$", + "%", + "&", + "'", + "(", + ")", + "*", + "+", + ",", + "-", + ".", + "/", + "0", + "1", + "2", + "3", + "4", + "5", + "6", + "7", + "8", + "9", + ":", + ";", + "<", + "=", + ">", + "?", + "@", + "A", + "B", + "C", + "D", + "E", + "F", + "G", + "H", + "I", + "J", + "K", + "L", + "M", + "N", + "O", + "P", + "Q", + "R", + "S", + "T", + "U", + "V", + "W", + "X", + "Y", + "Z", + "[", + "\\", + "]", + "^", + "_", + "`", + "a", + "b", + "c", + "d", + "e", + "f", + "g", + "h", + "i", + "j", + "k", + "l", + "m", + "n", + "o", + "p", + "q", + "r", + "s", + "t", + "u", + "v", + "w", + "x", + "y", + "z", + "{", + "|", + "}", + "~" + ], + "width": 64 +} diff --git a/raw-step/seed-2/conversion_verification.json b/raw-step/seed-2/conversion_verification.json new file mode 100644 index 0000000000000000000000000000000000000000..657a0ffedac454857bdecb0537da584a5d61273d --- /dev/null +++ b/raw-step/seed-2/conversion_verification.json @@ -0,0 +1,13 @@ +{ + "all_tensors_bitwise_equal": true, + "family": "raw-step", + "published_sha256": "a96e32f0a540aaf176a0ec8479c3f9b18c9a43fbea392f1049ea028f3f229f87", + "saved_predictions_reproduced": { + "exhaustive": 108, + "held_out": 92 + }, + "seed": 2, + "source_hash_manifest": "frozen.json", + "source_sha256": "d1d3ef582c2f85a043ba507cd502aae062a8e5daac3cacaa0702ddce9404abf4", + "torch_version": "2.14.1" +} diff --git a/raw-step/seed-2/model.safetensors b/raw-step/seed-2/model.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..4a19f231a269592782991fcd6da840eec8cf36f2 --- /dev/null +++ b/raw-step/seed-2/model.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:a96e32f0a540aaf176a0ec8479c3f9b18c9a43fbea392f1049ea028f3f229f87 +size 392360 diff --git a/raw-step/seed-2/results.json b/raw-step/seed-2/results.json new file mode 100644 index 0000000000000000000000000000000000000000..a5e7015fb15db6f5bd31b18c9205ee608107ef50 --- /dev/null +++ b/raw-step/seed-2/results.json @@ -0,0 +1,283 @@ +{ + "action_accuracy": [ + 0, + 92 + ], + "condition": "tiny_raw_autoregressive", + "exhaustive": { + "([)]#": [ + 23, + 36 + ], + "abcde": [ + 20, + 36 + ], + "uvwxy": [ + 36, + 36 + ] + }, + "first_error_median": 1.0, + "held_out": { + "brackets": { + "by_depth": { + "0": [ + 0, + 2 + ], + "1": [ + 0, + 4 + ], + "16": [ + 0, + 8 + ], + "2": [ + 0, + 8 + ], + "3": [ + 0, + 8 + ], + "4": [ + 0, + 8 + ], + "8": [ + 0, + 8 + ] + }, + "interpretation": { + "0": "below label-blind", + "1": "below label-blind", + "16": "below label-blind", + "2": "below label-blind", + "3": "below label-blind", + "4": "below label-blind", + "8": "below label-blind" + } + }, + "letters": { + "by_depth": { + "0": [ + 0, + 2 + ], + "1": [ + 0, + 4 + ], + "16": [ + 0, + 8 + ], + "2": [ + 0, + 8 + ], + "3": [ + 0, + 8 + ], + "4": [ + 0, + 8 + ], + "8": [ + 0, + 8 + ] + }, + "interpretation": { + "0": "below label-blind", + "1": "below label-blind", + "16": "below label-blind", + "2": "below label-blind", + "3": "below label-blind", + "4": "below label-blind", + "8": "below label-blind" + } + }, + "total": [ + 0, + 92 + ] + }, + "invalid_action_calls": 0, + "label_blind": { + "always_accept": [ + 46, + 92 + ], + "always_reject": [ + 46, + 92 + ], + "by_cell": { + "brackets": { + "0": { + "always_accept": [ + 1, + 2 + ], + "always_reject": [ + 1, + 2 + ] + }, + "1": { + "always_accept": [ + 2, + 4 + ], + "always_reject": [ + 2, + 4 + ] + }, + "16": { + "always_accept": [ + 4, + 8 + ], + "always_reject": [ + 4, + 8 + ] + }, + "2": { + "always_accept": [ + 4, + 8 + ], + "always_reject": [ + 4, + 8 + ] + }, + "3": { + "always_accept": [ + 4, + 8 + ], + "always_reject": [ + 4, + 8 + ] + }, + "4": { + "always_accept": [ + 4, + 8 + ], + "always_reject": [ + 4, + 8 + ] + }, + "8": { + "always_accept": [ + 4, + 8 + ], + "always_reject": [ + 4, + 8 + ] + } + }, + "letters": { + "0": { + "always_accept": [ + 1, + 2 + ], + "always_reject": [ + 1, + 2 + ] + }, + "1": { + "always_accept": [ + 2, + 4 + ], + "always_reject": [ + 2, + 4 + ] + }, + "16": { + "always_accept": [ + 4, + 8 + ], + "always_reject": [ + 4, + 8 + ] + }, + "2": { + "always_accept": [ + 4, + 8 + ], + "always_reject": [ + 4, + 8 + ] + }, + "3": { + "always_accept": [ + 4, + 8 + ], + "always_reject": [ + 4, + 8 + ] + }, + "4": { + "always_accept": [ + 4, + 8 + ], + "always_reject": [ + 4, + 8 + ] + }, + "8": { + "always_accept": [ + 4, + 8 + ], + "always_reject": [ + 4, + 8 + ] + } + } + } + }, + "push_value_errors": 22, + "seed": 2, + "training": { + "loss": 0.001583919976837933, + "parameters": 81004, + "seconds": 1.9714072089991532, + "seed": 2, + "train_accuracy": [ + 36, + 36 + ] + }, + "usage": { + "completion_tokens": 0, + "prompt_tokens": 0 + } +} diff --git a/raw-step/seed-2/training.json b/raw-step/seed-2/training.json new file mode 100644 index 0000000000000000000000000000000000000000..e18a46c40e98aaf215bcf1c43e92dc4ef50eb64f --- /dev/null +++ b/raw-step/seed-2/training.json @@ -0,0 +1,148 @@ +{ + "plan": { + "architecture": "2-layer width-64 4-head Transformer; causal attention; autoregressive operation/value/EOS", + "batch": 36, + "clip": 1, + "domain": "three fixed supplied maps x 2 phases x 6 current symbols/end x 3 closing-symbol/empty tops", + "held_out_hash": "d2c7202ea08da83ad5398723a2cc4ce9d3175b7e5f7b84ce40aa529103e0d6e2", + "lr": 0.001, + "optimizer": "AdamW", + "output_limit": 3, + "prompt_loss_masked": true, + "schedule": "cosine to 1e-5", + "seeds": [ + 0, + 1, + 2 + ], + "source_sha256": { + "scripts/run_a6_x6.py": "b8ce548d5a55069483b46ed89da10f34bb9a3edf976b361069cd4d11692fd685", + "src/abstractgym/a6_model.py": "d13c0b913ba39cd3c8371147d2495b185c24d8d15225cd68bc11e1aa2ebd72f7", + "src/abstractgym/a6_raw.py": "df9c2be312f2290939871364baca738ac9a849dca855d049e304a6d0bb74e630" + }, + "steps": 800, + "torch": "2.14.1", + "training_hash": "11d5025e1246025f38e3ea840fd2a065ca73c2751cdf9de5f50005f4be7fa3a7", + "vocab": [ + "PAD", + "CLS", + "opening", + "closing", + "END", + "EMPTY", + "SEP", + "EOS", + "PUSH", + "CENTER", + "POP", + "ACCEPT", + "REJECT", + " ", + "!", + "\"", + "#", + "$", + "%", + "&", + "'", + "(", + ")", + "*", + "+", + ",", + "-", + ".", + "", + "0", + "1", + "2", + "3", + "4", + "5", + "6", + "7", + "8", + "9", + ":", + ";", + "<", + "=", + ">", + "?", + "@", + "A", + "B", + "C", + "D", + "E", + "F", + "G", + "H", + "I", + "J", + "K", + "L", + "M", + "N", + "O", + "P", + "Q", + "R", + "S", + "T", + "U", + "V", + "W", + "X", + "Y", + "Z", + "[", + "\\", + "]", + "^", + "_", + "`", + "a", + "b", + "c", + "d", + "e", + "f", + "g", + "h", + "i", + "j", + "k", + "l", + "m", + "n", + "o", + "p", + "q", + "r", + "s", + "t", + "u", + "v", + "w", + "x", + "y", + "z", + "{", + "|", + "}", + "~" + ], + "weight_decay": 0.01 + }, + "training": { + "loss": 0.001583919976837933, + "parameters": 81004, + "seconds": 1.9714072089991532, + "seed": 2, + "train_accuracy": [ + 36, + 36 + ] + } +} diff --git a/step/seed-0/config.json b/step/seed-0/config.json new file mode 100644 index 0000000000000000000000000000000000000000..8c4f22c1b68e97b7b175f1e96227bf7e2217d927 --- /dev/null +++ b/step/seed-0/config.json @@ -0,0 +1,36 @@ +{ + "actions": [ + "PUSH_0", + "PUSH_1", + "CENTER", + "POP", + "ACCEPT", + "REJECT" + ], + "code_branch": "main", + "dropout": 0, + "dtype": "float32", + "family": "step", + "feedforward": 128, + "heads": 4, + "layers": 2, + "model_class": "abstractgym.a6_model.TinyTransformer", + "outputs": 6, + "position_limit": 256, + "seed": 0, + "vocabulary": [ + "PAD", + "CLS", + "opening", + "closing", + "OPEN_0", + "OPEN_1", + "CLOSE_0", + "CLOSE_1", + "CENTER", + "END", + "EMPTY", + "DIRECT" + ], + "width": 64 +} diff --git a/step/seed-0/conversion_verification.json b/step/seed-0/conversion_verification.json new file mode 100644 index 0000000000000000000000000000000000000000..efa7ef5d8616d9a82237e91dfaa94924378ad8d3 --- /dev/null +++ b/step/seed-0/conversion_verification.json @@ -0,0 +1,14 @@ +{ + "all_tensors_bitwise_equal": true, + "family": "step", + "published_sha256": "baee7f0d12b2a70b1e9604c2d7d9a80554f156a2af111d3966f37cea9f21357f", + "saved_predictions_reproduced": { + "exhaustive": 36, + "extra": 32, + "held_out": 92 + }, + "seed": 0, + "source_hash_manifest": "frozen.json", + "source_sha256": "a6949c9c2a8679152cd71b1a64c68a5147cf8cc3c6dca94bcfd52c5820ae478e", + "torch_version": "2.14.1" +} diff --git a/step/seed-0/model.safetensors b/step/seed-0/model.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..e0eeef76855cc90ee7a9d6cd778f7f138b9e067a --- /dev/null +++ b/step/seed-0/model.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:baee7f0d12b2a70b1e9604c2d7d9a80554f156a2af111d3966f37cea9f21357f +size 341256 diff --git a/step/seed-0/results.json b/step/seed-0/results.json new file mode 100644 index 0000000000000000000000000000000000000000..c7e7458de8259403b59036f4908af0f48d584199 --- /dev/null +++ b/step/seed-0/results.json @@ -0,0 +1,291 @@ +{ + "action_accuracy": [ + 988, + 988 + ], + "condition": "tiny_trained", + "evaluation_seconds": 0.07790250000107335, + "exhaustive_check": [ + 36, + 36 + ], + "extra_depths": { + "32": [ + 16, + 16 + ], + "64": [ + 16, + 16 + ] + }, + "first_error_median": null, + "held_out": { + "brackets": { + "by_depth": { + "0": [ + 2, + 2 + ], + "1": [ + 4, + 4 + ], + "16": [ + 8, + 8 + ], + "2": [ + 8, + 8 + ], + "3": [ + 8, + 8 + ], + "4": [ + 8, + 8 + ], + "8": [ + 8, + 8 + ] + }, + "interpretation": { + "0": "beats both label-blind baselines", + "1": "beats both label-blind baselines", + "16": "beats both label-blind baselines", + "2": "beats both label-blind baselines", + "3": "beats both label-blind baselines", + "4": "beats both label-blind baselines", + "8": "beats both label-blind baselines" + } + }, + "letters": { + "by_depth": { + "0": [ + 2, + 2 + ], + "1": [ + 4, + 4 + ], + "16": [ + 8, + 8 + ], + "2": [ + 8, + 8 + ], + "3": [ + 8, + 8 + ], + "4": [ + 8, + 8 + ], + "8": [ + 8, + 8 + ] + }, + "interpretation": { + "0": "beats both label-blind baselines", + "1": "beats both label-blind baselines", + "16": "beats both label-blind baselines", + "2": "beats both label-blind baselines", + "3": "beats both label-blind baselines", + "4": "beats both label-blind baselines", + "8": "beats both label-blind baselines" + } + }, + "total": [ + 92, + 92 + ] + }, + "label_blind": { + "always_accept": [ + 46, + 92 + ], + "always_reject": [ + 46, + 92 + ], + "by_cell": { + "brackets": { + "0": { + "always_accept": [ + 1, + 2 + ], + "always_reject": [ + 1, + 2 + ] + }, + "1": { + "always_accept": [ + 2, + 4 + ], + "always_reject": [ + 2, + 4 + ] + }, + "16": { + "always_accept": [ + 4, + 8 + ], + "always_reject": [ + 4, + 8 + ] + }, + "2": { + "always_accept": [ + 4, + 8 + ], + "always_reject": [ + 4, + 8 + ] + }, + "3": { + "always_accept": [ + 4, + 8 + ], + "always_reject": [ + 4, + 8 + ] + }, + "4": { + "always_accept": [ + 4, + 8 + ], + "always_reject": [ + 4, + 8 + ] + }, + "8": { + "always_accept": [ + 4, + 8 + ], + "always_reject": [ + 4, + 8 + ] + } + }, + "letters": { + "0": { + "always_accept": [ + 1, + 2 + ], + "always_reject": [ + 1, + 2 + ] + }, + "1": { + "always_accept": [ + 2, + 4 + ], + "always_reject": [ + 2, + 4 + ] + }, + "16": { + "always_accept": [ + 4, + 8 + ], + "always_reject": [ + 4, + 8 + ] + }, + "2": { + "always_accept": [ + 4, + 8 + ], + "always_reject": [ + 4, + 8 + ] + }, + "3": { + "always_accept": [ + 4, + 8 + ], + "always_reject": [ + 4, + 8 + ] + }, + "4": { + "always_accept": [ + 4, + 8 + ], + "always_reject": [ + 4, + 8 + ] + }, + "8": { + "always_accept": [ + 4, + 8 + ], + "always_reject": [ + 4, + 8 + ] + } + } + } + }, + "model_revisions": { + "direct-0.pt": "39bf28b49785115db3ca0b58a584230aff6db09361ee79619ba6e7fca8bcb8cd", + "step-0.pt": "a6949c9c2a8679152cd71b1a64c68a5147cf8cc3c6dca94bcfd52c5820ae478e" + }, + "push_value_errors": 0, + "seed": 0, + "token_usage": { + "direct_input_tokens": 1442, + "execution_input_tokens": 3952, + "execution_output_classes": 988 + }, + "train_examples": 2305, + "training": { + "batch_size": 36, + "final_loss": 0.0006350057665258646, + "parameters": 68230, + "seconds": 1.4034502079994127, + "steps": 800, + "train_accuracy": [ + 36, + 36 + ] + }, + "unique_train_examples": 36 +} diff --git a/step/seed-0/training.json b/step/seed-0/training.json new file mode 100644 index 0000000000000000000000000000000000000000..def8001fd6b7fa3fe1cf95c9dc5b44e8920eac63 --- /dev/null +++ b/step/seed-0/training.json @@ -0,0 +1,63 @@ +{ + "plan": { + "architecture": { + "attention": "bidirectional encoder, CLS readout", + "dropout": 0, + "feedforward": 128, + "heads": 4, + "layers": 2, + "position": "fixed sinusoidal", + "width": 64 + }, + "device": "cpu", + "dtype": "float32", + "git_revision": "39bca8d1f7dd0324010f76d6eb44a68ea719d572", + "hardware": "arm", + "optimizer": { + "clip_norm": 1.0, + "lr": 0.001, + "name": "AdamW", + "schedule": "cosine to 1e-5", + "weight_decay": 0.01 + }, + "packages": { + "torch": "2.14.1" + }, + "platform": "macOS-26.7-arm64-arm-64bit", + "python": "3.12.13", + "seeds": [ + 0, + 1, + 2 + ], + "source_sha256": { + "scripts/run_a6_tiny.py": "62a632c65003a10c5f35b1a586f7a62cfae3605db0c39d9ae48914367ea7aa1e", + "src/abstractgym/a6.py": "975b3d76fc9503e11e8f0ad8c57aa27162250298d4316d949145e6df80dbbfe2", + "src/abstractgym/a6_model.py": "d13c0b913ba39cd3c8371147d2495b185c24d8d15225cd68bc11e1aa2ebd72f7", + "src/abstractgym/brackets.py": "9fa8b835c0aac8fa26177b032f1fc07e1bb0964f7ec01fd0f83ad11b74e0550f" + }, + "threads": 2, + "training": { + "deduplicated_step_examples": 36, + "direct_examples": 367, + "direct_positive": 31, + "direct_sampling": "balanced labels, replacement; separate matched model", + "max_depth": 4, + "raw_step_examples": 2305, + "symbols": "uvwxy", + "trajectories": 367 + }, + "training_hash": "8fa8728efdda11e741e8ec16c2e0267cf0fd321e849119caea177055d15e5df0" + }, + "training": { + "batch_size": 36, + "final_loss": 0.0006350057665258646, + "parameters": 68230, + "seconds": 1.4034502079994127, + "steps": 800, + "train_accuracy": [ + 36, + 36 + ] + } +} diff --git a/step/seed-1/config.json b/step/seed-1/config.json new file mode 100644 index 0000000000000000000000000000000000000000..170a0bf8762bffe7cf713ff499693aaf65cc56d2 --- /dev/null +++ b/step/seed-1/config.json @@ -0,0 +1,36 @@ +{ + "actions": [ + "PUSH_0", + "PUSH_1", + "CENTER", + "POP", + "ACCEPT", + "REJECT" + ], + "code_branch": "main", + "dropout": 0, + "dtype": "float32", + "family": "step", + "feedforward": 128, + "heads": 4, + "layers": 2, + "model_class": "abstractgym.a6_model.TinyTransformer", + "outputs": 6, + "position_limit": 256, + "seed": 1, + "vocabulary": [ + "PAD", + "CLS", + "opening", + "closing", + "OPEN_0", + "OPEN_1", + "CLOSE_0", + "CLOSE_1", + "CENTER", + "END", + "EMPTY", + "DIRECT" + ], + "width": 64 +} diff --git a/step/seed-1/conversion_verification.json b/step/seed-1/conversion_verification.json new file mode 100644 index 0000000000000000000000000000000000000000..aafc5b90cb84e470a03ec5db2e2ab69d73222984 --- /dev/null +++ b/step/seed-1/conversion_verification.json @@ -0,0 +1,14 @@ +{ + "all_tensors_bitwise_equal": true, + "family": "step", + "published_sha256": "2d25c9513d2378fa151f79d4cb4619a2e0d401f7825bd7b6c5a6c13769b37f5b", + "saved_predictions_reproduced": { + "exhaustive": 36, + "extra": 32, + "held_out": 92 + }, + "seed": 1, + "source_hash_manifest": "frozen.json", + "source_sha256": "341d85d422c11d1d34345d65e05cef12e2ee0a4d1608e065857eb64cda564a77", + "torch_version": "2.14.1" +} diff --git a/step/seed-1/model.safetensors b/step/seed-1/model.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..55bfdfccd6d1d9b810da3ac7111db96abef8c9e2 --- /dev/null +++ b/step/seed-1/model.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:2d25c9513d2378fa151f79d4cb4619a2e0d401f7825bd7b6c5a6c13769b37f5b +size 341256 diff --git a/step/seed-1/results.json b/step/seed-1/results.json new file mode 100644 index 0000000000000000000000000000000000000000..989e211c1e195718847001b0fdb663385a5b3c1f --- /dev/null +++ b/step/seed-1/results.json @@ -0,0 +1,291 @@ +{ + "action_accuracy": [ + 988, + 988 + ], + "condition": "tiny_trained", + "evaluation_seconds": 0.05974966700159712, + "exhaustive_check": [ + 36, + 36 + ], + "extra_depths": { + "32": [ + 16, + 16 + ], + "64": [ + 16, + 16 + ] + }, + "first_error_median": null, + "held_out": { + "brackets": { + "by_depth": { + "0": [ + 2, + 2 + ], + "1": [ + 4, + 4 + ], + "16": [ + 8, + 8 + ], + "2": [ + 8, + 8 + ], + "3": [ + 8, + 8 + ], + "4": [ + 8, + 8 + ], + "8": [ + 8, + 8 + ] + }, + "interpretation": { + "0": "beats both label-blind baselines", + "1": "beats both label-blind baselines", + "16": "beats both label-blind baselines", + "2": "beats both label-blind baselines", + "3": "beats both label-blind baselines", + "4": "beats both label-blind baselines", + "8": "beats both label-blind baselines" + } + }, + "letters": { + "by_depth": { + "0": [ + 2, + 2 + ], + "1": [ + 4, + 4 + ], + "16": [ + 8, + 8 + ], + "2": [ + 8, + 8 + ], + "3": [ + 8, + 8 + ], + "4": [ + 8, + 8 + ], + "8": [ + 8, + 8 + ] + }, + "interpretation": { + "0": "beats both label-blind baselines", + "1": "beats both label-blind baselines", + "16": "beats both label-blind baselines", + "2": "beats both label-blind baselines", + "3": "beats both label-blind baselines", + "4": "beats both label-blind baselines", + "8": "beats both label-blind baselines" + } + }, + "total": [ + 92, + 92 + ] + }, + "label_blind": { + "always_accept": [ + 46, + 92 + ], + "always_reject": [ + 46, + 92 + ], + "by_cell": { + "brackets": { + "0": { + "always_accept": [ + 1, + 2 + ], + "always_reject": [ + 1, + 2 + ] + }, + "1": { + "always_accept": [ + 2, + 4 + ], + "always_reject": [ + 2, + 4 + ] + }, + "16": { + "always_accept": [ + 4, + 8 + ], + "always_reject": [ + 4, + 8 + ] + }, + "2": { + "always_accept": [ + 4, + 8 + ], + "always_reject": [ + 4, + 8 + ] + }, + "3": { + "always_accept": [ + 4, + 8 + ], + "always_reject": [ + 4, + 8 + ] + }, + "4": { + "always_accept": [ + 4, + 8 + ], + "always_reject": [ + 4, + 8 + ] + }, + "8": { + "always_accept": [ + 4, + 8 + ], + "always_reject": [ + 4, + 8 + ] + } + }, + "letters": { + "0": { + "always_accept": [ + 1, + 2 + ], + "always_reject": [ + 1, + 2 + ] + }, + "1": { + "always_accept": [ + 2, + 4 + ], + "always_reject": [ + 2, + 4 + ] + }, + "16": { + "always_accept": [ + 4, + 8 + ], + "always_reject": [ + 4, + 8 + ] + }, + "2": { + "always_accept": [ + 4, + 8 + ], + "always_reject": [ + 4, + 8 + ] + }, + "3": { + "always_accept": [ + 4, + 8 + ], + "always_reject": [ + 4, + 8 + ] + }, + "4": { + "always_accept": [ + 4, + 8 + ], + "always_reject": [ + 4, + 8 + ] + }, + "8": { + "always_accept": [ + 4, + 8 + ], + "always_reject": [ + 4, + 8 + ] + } + } + } + }, + "model_revisions": { + "direct-1.pt": "07445f6621896c71053a5e64ba61b78c430954a840575bba113e7aeaef462a43", + "step-1.pt": "341d85d422c11d1d34345d65e05cef12e2ee0a4d1608e065857eb64cda564a77" + }, + "push_value_errors": 0, + "seed": 1, + "token_usage": { + "direct_input_tokens": 1442, + "execution_input_tokens": 3952, + "execution_output_classes": 988 + }, + "train_examples": 2305, + "training": { + "batch_size": 36, + "final_loss": 0.0007644929573871195, + "parameters": 68230, + "seconds": 1.179319791001035, + "steps": 800, + "train_accuracy": [ + 36, + 36 + ] + }, + "unique_train_examples": 36 +} diff --git a/step/seed-1/training.json b/step/seed-1/training.json new file mode 100644 index 0000000000000000000000000000000000000000..5ad62e06ff27db555b7951570c9360b12707650d --- /dev/null +++ b/step/seed-1/training.json @@ -0,0 +1,63 @@ +{ + "plan": { + "architecture": { + "attention": "bidirectional encoder, CLS readout", + "dropout": 0, + "feedforward": 128, + "heads": 4, + "layers": 2, + "position": "fixed sinusoidal", + "width": 64 + }, + "device": "cpu", + "dtype": "float32", + "git_revision": "39bca8d1f7dd0324010f76d6eb44a68ea719d572", + "hardware": "arm", + "optimizer": { + "clip_norm": 1.0, + "lr": 0.001, + "name": "AdamW", + "schedule": "cosine to 1e-5", + "weight_decay": 0.01 + }, + "packages": { + "torch": "2.14.1" + }, + "platform": "macOS-26.7-arm64-arm-64bit", + "python": "3.12.13", + "seeds": [ + 0, + 1, + 2 + ], + "source_sha256": { + "scripts/run_a6_tiny.py": "62a632c65003a10c5f35b1a586f7a62cfae3605db0c39d9ae48914367ea7aa1e", + "src/abstractgym/a6.py": "975b3d76fc9503e11e8f0ad8c57aa27162250298d4316d949145e6df80dbbfe2", + "src/abstractgym/a6_model.py": "d13c0b913ba39cd3c8371147d2495b185c24d8d15225cd68bc11e1aa2ebd72f7", + "src/abstractgym/brackets.py": "9fa8b835c0aac8fa26177b032f1fc07e1bb0964f7ec01fd0f83ad11b74e0550f" + }, + "threads": 2, + "training": { + "deduplicated_step_examples": 36, + "direct_examples": 367, + "direct_positive": 31, + "direct_sampling": "balanced labels, replacement; separate matched model", + "max_depth": 4, + "raw_step_examples": 2305, + "symbols": "uvwxy", + "trajectories": 367 + }, + "training_hash": "8fa8728efdda11e741e8ec16c2e0267cf0fd321e849119caea177055d15e5df0" + }, + "training": { + "batch_size": 36, + "final_loss": 0.0007644929573871195, + "parameters": 68230, + "seconds": 1.179319791001035, + "steps": 800, + "train_accuracy": [ + 36, + 36 + ] + } +} diff --git a/step/seed-2/config.json b/step/seed-2/config.json new file mode 100644 index 0000000000000000000000000000000000000000..21cf8992a424cbf18adc91107a3bded867906cd1 --- /dev/null +++ b/step/seed-2/config.json @@ -0,0 +1,36 @@ +{ + "actions": [ + "PUSH_0", + "PUSH_1", + "CENTER", + "POP", + "ACCEPT", + "REJECT" + ], + "code_branch": "main", + "dropout": 0, + "dtype": "float32", + "family": "step", + "feedforward": 128, + "heads": 4, + "layers": 2, + "model_class": "abstractgym.a6_model.TinyTransformer", + "outputs": 6, + "position_limit": 256, + "seed": 2, + "vocabulary": [ + "PAD", + "CLS", + "opening", + "closing", + "OPEN_0", + "OPEN_1", + "CLOSE_0", + "CLOSE_1", + "CENTER", + "END", + "EMPTY", + "DIRECT" + ], + "width": 64 +} diff --git a/step/seed-2/conversion_verification.json b/step/seed-2/conversion_verification.json new file mode 100644 index 0000000000000000000000000000000000000000..d44941ad559779f7eef2b50db55454793c765fb4 --- /dev/null +++ b/step/seed-2/conversion_verification.json @@ -0,0 +1,14 @@ +{ + "all_tensors_bitwise_equal": true, + "family": "step", + "published_sha256": "7a2d655c83a05f98193abb506ab5e7d181d4463fd21a6dbc1934b43cc7abf213", + "saved_predictions_reproduced": { + "exhaustive": 36, + "extra": 32, + "held_out": 92 + }, + "seed": 2, + "source_hash_manifest": "frozen.json", + "source_sha256": "e70093585cce216028e40d2ff4648b6bf356383d96e1706ffb5a2e86f656ca30", + "torch_version": "2.14.1" +} diff --git a/step/seed-2/model.safetensors b/step/seed-2/model.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..b1dcc120638e62d3f5f9c86ad9c12e10a9602c37 --- /dev/null +++ b/step/seed-2/model.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:7a2d655c83a05f98193abb506ab5e7d181d4463fd21a6dbc1934b43cc7abf213 +size 341256 diff --git a/step/seed-2/results.json b/step/seed-2/results.json new file mode 100644 index 0000000000000000000000000000000000000000..6331d3b6aed6b9807c7002233b46c29b709bdfed --- /dev/null +++ b/step/seed-2/results.json @@ -0,0 +1,291 @@ +{ + "action_accuracy": [ + 988, + 988 + ], + "condition": "tiny_trained", + "evaluation_seconds": 0.12443979099771241, + "exhaustive_check": [ + 36, + 36 + ], + "extra_depths": { + "32": [ + 16, + 16 + ], + "64": [ + 16, + 16 + ] + }, + "first_error_median": null, + "held_out": { + "brackets": { + "by_depth": { + "0": [ + 2, + 2 + ], + "1": [ + 4, + 4 + ], + "16": [ + 8, + 8 + ], + "2": [ + 8, + 8 + ], + "3": [ + 8, + 8 + ], + "4": [ + 8, + 8 + ], + "8": [ + 8, + 8 + ] + }, + "interpretation": { + "0": "beats both label-blind baselines", + "1": "beats both label-blind baselines", + "16": "beats both label-blind baselines", + "2": "beats both label-blind baselines", + "3": "beats both label-blind baselines", + "4": "beats both label-blind baselines", + "8": "beats both label-blind baselines" + } + }, + "letters": { + "by_depth": { + "0": [ + 2, + 2 + ], + "1": [ + 4, + 4 + ], + "16": [ + 8, + 8 + ], + "2": [ + 8, + 8 + ], + "3": [ + 8, + 8 + ], + "4": [ + 8, + 8 + ], + "8": [ + 8, + 8 + ] + }, + "interpretation": { + "0": "beats both label-blind baselines", + "1": "beats both label-blind baselines", + "16": "beats both label-blind baselines", + "2": "beats both label-blind baselines", + "3": "beats both label-blind baselines", + "4": "beats both label-blind baselines", + "8": "beats both label-blind baselines" + } + }, + "total": [ + 92, + 92 + ] + }, + "label_blind": { + "always_accept": [ + 46, + 92 + ], + "always_reject": [ + 46, + 92 + ], + "by_cell": { + "brackets": { + "0": { + "always_accept": [ + 1, + 2 + ], + "always_reject": [ + 1, + 2 + ] + }, + "1": { + "always_accept": [ + 2, + 4 + ], + "always_reject": [ + 2, + 4 + ] + }, + "16": { + "always_accept": [ + 4, + 8 + ], + "always_reject": [ + 4, + 8 + ] + }, + "2": { + "always_accept": [ + 4, + 8 + ], + "always_reject": [ + 4, + 8 + ] + }, + "3": { + "always_accept": [ + 4, + 8 + ], + "always_reject": [ + 4, + 8 + ] + }, + "4": { + "always_accept": [ + 4, + 8 + ], + "always_reject": [ + 4, + 8 + ] + }, + "8": { + "always_accept": [ + 4, + 8 + ], + "always_reject": [ + 4, + 8 + ] + } + }, + "letters": { + "0": { + "always_accept": [ + 1, + 2 + ], + "always_reject": [ + 1, + 2 + ] + }, + "1": { + "always_accept": [ + 2, + 4 + ], + "always_reject": [ + 2, + 4 + ] + }, + "16": { + "always_accept": [ + 4, + 8 + ], + "always_reject": [ + 4, + 8 + ] + }, + "2": { + "always_accept": [ + 4, + 8 + ], + "always_reject": [ + 4, + 8 + ] + }, + "3": { + "always_accept": [ + 4, + 8 + ], + "always_reject": [ + 4, + 8 + ] + }, + "4": { + "always_accept": [ + 4, + 8 + ], + "always_reject": [ + 4, + 8 + ] + }, + "8": { + "always_accept": [ + 4, + 8 + ], + "always_reject": [ + 4, + 8 + ] + } + } + } + }, + "model_revisions": { + "direct-2.pt": "5e14beeb2d5e742967d56c86677e37a4b0b8d2a1097d3c1d50f78d71fe29a782", + "step-2.pt": "e70093585cce216028e40d2ff4648b6bf356383d96e1706ffb5a2e86f656ca30" + }, + "push_value_errors": 0, + "seed": 2, + "token_usage": { + "direct_input_tokens": 1442, + "execution_input_tokens": 3952, + "execution_output_classes": 988 + }, + "train_examples": 2305, + "training": { + "batch_size": 36, + "final_loss": 0.0006954485434107482, + "parameters": 68230, + "seconds": 1.3472057079998194, + "steps": 800, + "train_accuracy": [ + 36, + 36 + ] + }, + "unique_train_examples": 36 +} diff --git a/step/seed-2/training.json b/step/seed-2/training.json new file mode 100644 index 0000000000000000000000000000000000000000..79fa66db468abd8f6401c6acbde5316641334a8a --- /dev/null +++ b/step/seed-2/training.json @@ -0,0 +1,63 @@ +{ + "plan": { + "architecture": { + "attention": "bidirectional encoder, CLS readout", + "dropout": 0, + "feedforward": 128, + "heads": 4, + "layers": 2, + "position": "fixed sinusoidal", + "width": 64 + }, + "device": "cpu", + "dtype": "float32", + "git_revision": "39bca8d1f7dd0324010f76d6eb44a68ea719d572", + "hardware": "arm", + "optimizer": { + "clip_norm": 1.0, + "lr": 0.001, + "name": "AdamW", + "schedule": "cosine to 1e-5", + "weight_decay": 0.01 + }, + "packages": { + "torch": "2.14.1" + }, + "platform": "macOS-26.7-arm64-arm-64bit", + "python": "3.12.13", + "seeds": [ + 0, + 1, + 2 + ], + "source_sha256": { + "scripts/run_a6_tiny.py": "62a632c65003a10c5f35b1a586f7a62cfae3605db0c39d9ae48914367ea7aa1e", + "src/abstractgym/a6.py": "975b3d76fc9503e11e8f0ad8c57aa27162250298d4316d949145e6df80dbbfe2", + "src/abstractgym/a6_model.py": "d13c0b913ba39cd3c8371147d2495b185c24d8d15225cd68bc11e1aa2ebd72f7", + "src/abstractgym/brackets.py": "9fa8b835c0aac8fa26177b032f1fc07e1bb0964f7ec01fd0f83ad11b74e0550f" + }, + "threads": 2, + "training": { + "deduplicated_step_examples": 36, + "direct_examples": 367, + "direct_positive": 31, + "direct_sampling": "balanced labels, replacement; separate matched model", + "max_depth": 4, + "raw_step_examples": 2305, + "symbols": "uvwxy", + "trajectories": 367 + }, + "training_hash": "8fa8728efdda11e741e8ec16c2e0267cf0fd321e849119caea177055d15e5df0" + }, + "training": { + "batch_size": 36, + "final_loss": 0.0006954485434107482, + "parameters": 68230, + "seconds": 1.3472057079998194, + "steps": 800, + "train_accuracy": [ + 36, + 36 + ] + } +} diff --git a/trace/seed-0/config.json b/trace/seed-0/config.json new file mode 100644 index 0000000000000000000000000000000000000000..76bd15c4af810f0ac40e5719b129f41e13510aba --- /dev/null +++ b/trace/seed-0/config.json @@ -0,0 +1,39 @@ +{ + "actions": [ + "PUSH_0", + "PUSH_1", + "CENTER", + "POP", + "ACCEPT", + "REJECT" + ], + "code_branch": "main", + "dropout": 0, + "dtype": "float32", + "family": "trace", + "feedforward": 128, + "heads": 4, + "layers": 2, + "model_class": "run_a6_tiny_trace.TraceTransformer", + "outputs": 15, + "position_limit": 256, + "seed": 0, + "vocabulary": [ + "PAD", + "CLS", + "END", + "SEP", + "EOS", + "OPEN_0", + "OPEN_1", + "CLOSE_0", + "CLOSE_1", + "PUSH_0", + "PUSH_1", + "CENTER", + "POP", + "ACCEPT", + "REJECT" + ], + "width": 64 +} diff --git a/trace/seed-0/conversion_verification.json b/trace/seed-0/conversion_verification.json new file mode 100644 index 0000000000000000000000000000000000000000..c8d77c8e59d9fd57d006db65e0be1a63ca7e0f2f --- /dev/null +++ b/trace/seed-0/conversion_verification.json @@ -0,0 +1,14 @@ +{ + "all_tensors_bitwise_equal": true, + "family": "trace", + "published_sha256": "e506bfd329fcf518b60b9d904e29045cbb4521a8ee93fce9ab19884654afe35d", + "saved_predictions_reproduced": { + "free_running": 92, + "injected_history": 192, + "teacher_forced": 92 + }, + "seed": 0, + "source_hash_manifest": "frozen.json", + "source_sha256": "bca522a8d4ac96fc3f10464784c96ecb182f505efeb5a73b41ac9cf71f0337eb", + "torch_version": "2.14.1" +} diff --git a/trace/seed-0/model.safetensors b/trace/seed-0/model.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..aaaf26ebf1872784159e48019eb1a3356ee60735 --- /dev/null +++ b/trace/seed-0/model.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:e506bfd329fcf518b60b9d904e29045cbb4521a8ee93fce9ab19884654afe35d +size 344364 diff --git a/trace/seed-0/results.json b/trace/seed-0/results.json new file mode 100644 index 0000000000000000000000000000000000000000..f71c720cd6386d9f63ebc49b5d5a44bf6a8c46aa --- /dev/null +++ b/trace/seed-0/results.json @@ -0,0 +1,87 @@ +{ + "by_depth": { + "0": { + "success": [ + 4, + 4 + ], + "teacher_success": [ + 4, + 4 + ] + }, + "1": { + "success": [ + 8, + 8 + ], + "teacher_success": [ + 8, + 8 + ] + }, + "16": { + "success": [ + 0, + 16 + ], + "teacher_success": [ + 0, + 16 + ] + }, + "2": { + "success": [ + 16, + 16 + ], + "teacher_success": [ + 16, + 16 + ] + }, + "3": { + "success": [ + 16, + 16 + ], + "teacher_success": [ + 16, + 16 + ] + }, + "4": { + "success": [ + 16, + 16 + ], + "teacher_success": [ + 16, + 16 + ] + }, + "8": { + "success": [ + 0, + 16 + ], + "teacher_success": [ + 0, + 16 + ] + } + }, + "full": [ + 60, + 92 + ], + "oracle_sequence": [ + 60, + 92 + ], + "seed": 0, + "teacher_actions": [ + 796, + 988 + ] +} diff --git a/trace/seed-0/training.json b/trace/seed-0/training.json new file mode 100644 index 0000000000000000000000000000000000000000..9f4f23c48c3f239c2657830cc3504c59c7f1111d --- /dev/null +++ b/trace/seed-0/training.json @@ -0,0 +1,44 @@ +{ + "plan": { + "architecture": "separate 2-layer causal Transformer, width 64, 4 heads, full role input to role action sequence", + "balanced_membership_sampling": true, + "batch_size": 64, + "caveat": "not the A6 step checkpoint; no same-checkpoint full/live ranking for tiny", + "clip": 1, + "cosine_end": 1e-05, + "held_out_hash": "d2c7202ea08da83ad5398723a2cc4ce9d3175b7e5f7b84ce40aa529103e0d6e2", + "lr": 0.001, + "seeds": [ + 0, + 1, + 2 + ], + "source_sha256": "bcf51be1ed79826cf8fc82c265e08b0953f2e5c51e1446aa9c065e0e5c012ee4", + "steps": 1500, + "training_hash": "f146a4ccaaa9c058c20403702b73e57e417f835dba57f14776af4e0affd2a544", + "vocab": [ + "PAD", + "CLS", + "END", + "SEP", + "EOS", + "OPEN_0", + "OPEN_1", + "CLOSE_0", + "CLOSE_1", + "PUSH_0", + "PUSH_1", + "CENTER", + "POP", + "ACCEPT", + "REJECT" + ], + "weight_decay": 0.01 + }, + "training": { + "last_loss": 0.0032255591358989477, + "parameters": 69007, + "seconds": 8.73125891699965, + "seed": 0 + } +} diff --git a/trace/seed-1/config.json b/trace/seed-1/config.json new file mode 100644 index 0000000000000000000000000000000000000000..8ca6294e4b3eb76587cdc6fb6b4c84d96a45a283 --- /dev/null +++ b/trace/seed-1/config.json @@ -0,0 +1,39 @@ +{ + "actions": [ + "PUSH_0", + "PUSH_1", + "CENTER", + "POP", + "ACCEPT", + "REJECT" + ], + "code_branch": "main", + "dropout": 0, + "dtype": "float32", + "family": "trace", + "feedforward": 128, + "heads": 4, + "layers": 2, + "model_class": "run_a6_tiny_trace.TraceTransformer", + "outputs": 15, + "position_limit": 256, + "seed": 1, + "vocabulary": [ + "PAD", + "CLS", + "END", + "SEP", + "EOS", + "OPEN_0", + "OPEN_1", + "CLOSE_0", + "CLOSE_1", + "PUSH_0", + "PUSH_1", + "CENTER", + "POP", + "ACCEPT", + "REJECT" + ], + "width": 64 +} diff --git a/trace/seed-1/conversion_verification.json b/trace/seed-1/conversion_verification.json new file mode 100644 index 0000000000000000000000000000000000000000..bc57e9c764bd1d71f73a0405155da4f678bfd524 --- /dev/null +++ b/trace/seed-1/conversion_verification.json @@ -0,0 +1,14 @@ +{ + "all_tensors_bitwise_equal": true, + "family": "trace", + "published_sha256": "82b4057ba67f34df95a23f117ded59c80be9c10fbd7d3bf85a5ac36d48dfc299", + "saved_predictions_reproduced": { + "free_running": 92, + "injected_history": 192, + "teacher_forced": 92 + }, + "seed": 1, + "source_hash_manifest": "frozen.json", + "source_sha256": "9dce9b0be1f134480d716b8eba49ddaa0cc382daaa5eacf460cba888366ec442", + "torch_version": "2.14.1" +} diff --git a/trace/seed-1/model.safetensors b/trace/seed-1/model.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..04b1a72b3e213040d24eff2412f6d235bf01ec7e --- /dev/null +++ b/trace/seed-1/model.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:82b4057ba67f34df95a23f117ded59c80be9c10fbd7d3bf85a5ac36d48dfc299 +size 344364 diff --git a/trace/seed-1/results.json b/trace/seed-1/results.json new file mode 100644 index 0000000000000000000000000000000000000000..2fc8db9aac9f105adf4c0043ab322d55044ecce1 --- /dev/null +++ b/trace/seed-1/results.json @@ -0,0 +1,87 @@ +{ + "by_depth": { + "0": { + "success": [ + 4, + 4 + ], + "teacher_success": [ + 4, + 4 + ] + }, + "1": { + "success": [ + 8, + 8 + ], + "teacher_success": [ + 8, + 8 + ] + }, + "16": { + "success": [ + 0, + 16 + ], + "teacher_success": [ + 0, + 16 + ] + }, + "2": { + "success": [ + 16, + 16 + ], + "teacher_success": [ + 16, + 16 + ] + }, + "3": { + "success": [ + 16, + 16 + ], + "teacher_success": [ + 16, + 16 + ] + }, + "4": { + "success": [ + 16, + 16 + ], + "teacher_success": [ + 16, + 16 + ] + }, + "8": { + "success": [ + 0, + 16 + ], + "teacher_success": [ + 0, + 16 + ] + } + }, + "full": [ + 60, + 92 + ], + "oracle_sequence": [ + 60, + 92 + ], + "seed": 1, + "teacher_actions": [ + 820, + 988 + ] +} diff --git a/trace/seed-1/training.json b/trace/seed-1/training.json new file mode 100644 index 0000000000000000000000000000000000000000..33d7209969644e1640aa90c7b0c59b94f2c593e1 --- /dev/null +++ b/trace/seed-1/training.json @@ -0,0 +1,44 @@ +{ + "plan": { + "architecture": "separate 2-layer causal Transformer, width 64, 4 heads, full role input to role action sequence", + "balanced_membership_sampling": true, + "batch_size": 64, + "caveat": "not the A6 step checkpoint; no same-checkpoint full/live ranking for tiny", + "clip": 1, + "cosine_end": 1e-05, + "held_out_hash": "d2c7202ea08da83ad5398723a2cc4ce9d3175b7e5f7b84ce40aa529103e0d6e2", + "lr": 0.001, + "seeds": [ + 0, + 1, + 2 + ], + "source_sha256": "bcf51be1ed79826cf8fc82c265e08b0953f2e5c51e1446aa9c065e0e5c012ee4", + "steps": 1500, + "training_hash": "f146a4ccaaa9c058c20403702b73e57e417f835dba57f14776af4e0affd2a544", + "vocab": [ + "PAD", + "CLS", + "END", + "SEP", + "EOS", + "OPEN_0", + "OPEN_1", + "CLOSE_0", + "CLOSE_1", + "PUSH_0", + "PUSH_1", + "CENTER", + "POP", + "ACCEPT", + "REJECT" + ], + "weight_decay": 0.01 + }, + "training": { + "last_loss": 0.0006422399310395122, + "parameters": 69007, + "seconds": 8.594061249998049, + "seed": 1 + } +} diff --git a/trace/seed-2/config.json b/trace/seed-2/config.json new file mode 100644 index 0000000000000000000000000000000000000000..4b6353643b1153b19df1cfbe85a1c8b4984fcdbf --- /dev/null +++ b/trace/seed-2/config.json @@ -0,0 +1,39 @@ +{ + "actions": [ + "PUSH_0", + "PUSH_1", + "CENTER", + "POP", + "ACCEPT", + "REJECT" + ], + "code_branch": "main", + "dropout": 0, + "dtype": "float32", + "family": "trace", + "feedforward": 128, + "heads": 4, + "layers": 2, + "model_class": "run_a6_tiny_trace.TraceTransformer", + "outputs": 15, + "position_limit": 256, + "seed": 2, + "vocabulary": [ + "PAD", + "CLS", + "END", + "SEP", + "EOS", + "OPEN_0", + "OPEN_1", + "CLOSE_0", + "CLOSE_1", + "PUSH_0", + "PUSH_1", + "CENTER", + "POP", + "ACCEPT", + "REJECT" + ], + "width": 64 +} diff --git a/trace/seed-2/conversion_verification.json b/trace/seed-2/conversion_verification.json new file mode 100644 index 0000000000000000000000000000000000000000..fb462caabe1e80e5df09c5ff9f8f06fb507597f9 --- /dev/null +++ b/trace/seed-2/conversion_verification.json @@ -0,0 +1,14 @@ +{ + "all_tensors_bitwise_equal": true, + "family": "trace", + "published_sha256": "f51b7be3c5995a693bad0e806a28481f7530533fcff44a52590ba99f2652d041", + "saved_predictions_reproduced": { + "free_running": 92, + "injected_history": 192, + "teacher_forced": 92 + }, + "seed": 2, + "source_hash_manifest": "frozen.json", + "source_sha256": "9d37e58d30e65b26f07be763d8317a9d0b15708168995f27cca34f82f03b3ae1", + "torch_version": "2.14.1" +} diff --git a/trace/seed-2/model.safetensors b/trace/seed-2/model.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..fd4fc3b985f80bba596e69cd81204b0d4d71dbaf --- /dev/null +++ b/trace/seed-2/model.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:f51b7be3c5995a693bad0e806a28481f7530533fcff44a52590ba99f2652d041 +size 344364 diff --git a/trace/seed-2/results.json b/trace/seed-2/results.json new file mode 100644 index 0000000000000000000000000000000000000000..70a39b22080a27d8d66f493a7098e6c0f0c75628 --- /dev/null +++ b/trace/seed-2/results.json @@ -0,0 +1,87 @@ +{ + "by_depth": { + "0": { + "success": [ + 4, + 4 + ], + "teacher_success": [ + 4, + 4 + ] + }, + "1": { + "success": [ + 8, + 8 + ], + "teacher_success": [ + 8, + 8 + ] + }, + "16": { + "success": [ + 0, + 16 + ], + "teacher_success": [ + 0, + 16 + ] + }, + "2": { + "success": [ + 16, + 16 + ], + "teacher_success": [ + 16, + 16 + ] + }, + "3": { + "success": [ + 16, + 16 + ], + "teacher_success": [ + 16, + 16 + ] + }, + "4": { + "success": [ + 16, + 16 + ], + "teacher_success": [ + 16, + 16 + ] + }, + "8": { + "success": [ + 0, + 16 + ], + "teacher_success": [ + 0, + 16 + ] + } + }, + "full": [ + 60, + 92 + ], + "oracle_sequence": [ + 60, + 92 + ], + "seed": 2, + "teacher_actions": [ + 794, + 988 + ] +} diff --git a/trace/seed-2/training.json b/trace/seed-2/training.json new file mode 100644 index 0000000000000000000000000000000000000000..dc91b2b4824fdb0b7ef7b9f5a1ffc934bdfd432c --- /dev/null +++ b/trace/seed-2/training.json @@ -0,0 +1,44 @@ +{ + "plan": { + "architecture": "separate 2-layer causal Transformer, width 64, 4 heads, full role input to role action sequence", + "balanced_membership_sampling": true, + "batch_size": 64, + "caveat": "not the A6 step checkpoint; no same-checkpoint full/live ranking for tiny", + "clip": 1, + "cosine_end": 1e-05, + "held_out_hash": "d2c7202ea08da83ad5398723a2cc4ce9d3175b7e5f7b84ce40aa529103e0d6e2", + "lr": 0.001, + "seeds": [ + 0, + 1, + 2 + ], + "source_sha256": "bcf51be1ed79826cf8fc82c265e08b0953f2e5c51e1446aa9c065e0e5c012ee4", + "steps": 1500, + "training_hash": "f146a4ccaaa9c058c20403702b73e57e417f835dba57f14776af4e0affd2a544", + "vocab": [ + "PAD", + "CLS", + "END", + "SEP", + "EOS", + "OPEN_0", + "OPEN_1", + "CLOSE_0", + "CLOSE_1", + "PUSH_0", + "PUSH_1", + "CENTER", + "POP", + "ACCEPT", + "REJECT" + ], + "weight_decay": 0.01 + }, + "training": { + "last_loss": 0.0006551336846314371, + "parameters": 69007, + "seconds": 8.611750959000346, + "seed": 2 + } +}