scripts_full/stage5.log

dropping 12 blocks: [8, 10, 12, 13, 14, 15, 22, 23, 24, 25, 26, 27]
saved -> p16m with 16 layers
dropping 20 blocks: [7, 8, 9, 10, 11, 12, 13, 14, 15, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27]
saved -> p8m with 8 layers
##### c28: base m_lr8, 4 epochs, ex/full60.jsonl
{'train_runtime': 420.991, 'train_samples_per_second': 100.031, 'train_steps_per_second': 3.126, 'train_loss': 0.012911466830678986, 'epoch': 4.0}
saved -> c28
The tokenizer you are loading from '/root/mt92/c28' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
=== c28 on val_all.jsonl (4710 docs, 216s) ===
  MICRO-F1 0.9690 | P 0.9694 R 0.9686 | tp/fp/fn 8638/273/280 | malformed 4 | out tok med 34 max 256
The tokenizer you are loading from '/root/mt92/c28' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
=== c28 on test_all.jsonl (5017 docs, 236s) ===
  MICRO-F1 0.9711 | P 0.9700 R 0.9721 | tp/fp/fn 8900/275/255 | malformed 1 | out tok med 33 max 256
The tokenizer you are loading from '/root/mt92/c28' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
=== c28 on gen_holdout.jsonl (300 docs, 27s) ===
  MICRO-F1 0.9733 | P 0.9733 R 0.9733 | tp/fp/fn 547/15/15 | malformed 0 | out tok med 33 max 179
c28 backed up
{ "artifact": "onnx_c28/final", "ttft_p50": 1084, "ttft_p95": 1112, "tok_s": 11.68, "cold_ms": 6879, "rss_MiB": 1118, "size_MiB": 736, "requests": 40, "failures": 0, "wall_s": 67.8, "device": "DeviceProfile(machine='x86_64', system='linux', cpu_cores=20"}
staged onnx_c28
##### b16: base p16m, 4 epochs, ex/full35.jsonl
{'train_runtime': 212.7042, 'train_samples_per_second': 151.929, 'train_steps_per_second': 4.758, 'train_loss': 0.0460680722905021, 'epoch': 4.0}
saved -> b16
The tokenizer you are loading from '/root/mt92/b16' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
=== b16 on val_all.jsonl (4710 docs, 155s) ===
  MICRO-F1 0.9292 | P 0.9291 R 0.9292 | tp/fp/fn 8287/632/631 | malformed 6 | out tok med 34 max 256
The tokenizer you are loading from '/root/mt92/b16' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
=== b16 on test_all.jsonl (5017 docs, 162s) ===
  MICRO-F1 0.9274 | P 0.9268 R 0.9280 | tp/fp/fn 8496/671/659 | malformed 4 | out tok med 33 max 256
The tokenizer you are loading from '/root/mt92/b16' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
=== b16 on gen_holdout.jsonl (300 docs, 31s) ===
  MICRO-F1 0.9554 | P 0.9571 R 0.9537 | tp/fp/fn 536/24/26 | malformed 1 | out tok med 34 max 256
b16 backed up
{ "artifact": "onnx_b16/final", "ttft_p50": 621, "ttft_p95": 631, "tok_s": 18.93, "cold_ms": 4497, "rss_MiB": 814, "size_MiB": 552, "requests": 68, "failures": 0, "wall_s": 65.0, "device": "DeviceProfile(machine='x86_64', system='linux', cpu_cores=20"}
staged onnx_b16
##### d8: base p8m, 8 epochs, ex/full70.jsonl
{'train_runtime': 360.8481, 'train_samples_per_second': 254.024, 'train_steps_per_second': 7.959, 'train_loss': 0.17886488650689383, 'epoch': 8.0}
saved -> d8
The tokenizer you are loading from '/root/mt92/d8' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
=== d8 on val_all.jsonl (4710 docs, 115s) ===
  MICRO-F1 0.9546 | P 0.9683 R 0.9412 | tp/fp/fn 8394/275/524 | malformed 73 | out tok med 34 max 256
The tokenizer you are loading from '/root/mt92/d8' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
=== d8 on test_all.jsonl (5017 docs, 117s) ===
  MICRO-F1 0.9474 | P 0.9622 R 0.9332 | tp/fp/fn 8543/336/612 | malformed 86 | out tok med 33 max 256
The tokenizer you are loading from '/root/mt92/d8' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
=== d8 on gen_holdout.jsonl (300 docs, 31s) ===
  MICRO-F1 0.8230 | P 0.8903 R 0.7651 | tp/fp/fn 430/53/132 | malformed 22 | out tok med 34 max 256
d8 backed up
{ "artifact": "onnx_d8/final", "ttft_p50": 316, "ttft_p95": 321, "tok_s": 31.36, "cold_ms": 3171, "rss_MiB": 649, "size_MiB": 430, "requests": 127, "failures": 0, "wall_s": 63.8, "device": "DeviceProfile(machine='x86_64', system='linux', cpu_cores=20"}
staged onnx_d8
STAGE5_DONE