{
  "output": "/mnt/local_nvme/cz/projects/HY-WorldPlay-DEV-Predictor/datasets/hyworldplay_predictor_confidence_validation25_prefeature/cases.jsonl",
  "cases": 25
}
{
  "records": 800,
  "train_records": 700
}
WARNING:__main__:
*****************************************
Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
*****************************************
rank0 extracted 25/133 batches
rank0 extracted 50/133 batches
rank0 extracted 75/133 batches
rank0 extracted 100/133 batches
rank0 extracted 125/133 batches
Saved 8 feature shards to /mnt/local_nvme/cz/projects/HY-WorldPlay-DEV-Predictor/confidence_validation/features/train
WARNING:__main__:
*****************************************
Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
*****************************************
rank0 extracted 25/35 batches
Saved 8 feature shards to /mnt/local_nvme/cz/projects/HY-WorldPlay-DEV-Predictor/confidence_validation/features/validation
WARNING:__main__:
*****************************************
Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
*****************************************
[rank0]:[W819 05:29:45.641755152 ProcessGroupNCCL.cpp:4115] [PG ID 0 PG GUID 0 Rank 0]  using GPU 0 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device,or call init_process_group() with a device_id.
[rank6]:[W819 05:29:45.820996456 ProcessGroupNCCL.cpp:4115] [PG ID 0 PG GUID 0 Rank 6]  using GPU 6 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device,or call init_process_group() with a device_id.
[rank1]:[W819 05:29:45.868540764 ProcessGroupNCCL.cpp:4115] [PG ID 0 PG GUID 0 Rank 1]  using GPU 1 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device,or call init_process_group() with a device_id.
[rank2]:[W819 05:29:46.322682030 ProcessGroupNCCL.cpp:4115] [PG ID 0 PG GUID 0 Rank 2]  using GPU 2 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device,or call init_process_group() with a device_id.
[rank7]:[W819 05:29:46.386567270 ProcessGroupNCCL.cpp:4115] [PG ID 0 PG GUID 0 Rank 7]  using GPU 7 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device,or call init_process_group() with a device_id.
[rank5]:[W819 05:29:46.398470481 ProcessGroupNCCL.cpp:4115] [PG ID 0 PG GUID 0 Rank 5]  using GPU 5 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device,or call init_process_group() with a device_id.
[rank4]:[W819 05:29:46.426298692 ProcessGroupNCCL.cpp:4115] [PG ID 0 PG GUID 0 Rank 4]  using GPU 4 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device,or call init_process_group() with a device_id.
[rank3]:[W819 05:29:46.471978908 ProcessGroupNCCL.cpp:4115] [PG ID 0 PG GUID 0 Rank 3]  using GPU 3 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device,or call init_process_group() with a device_id.
c0 epoch 1/40 loss=1.951573
c0 epoch 10/40 loss=0.075422
c0 epoch 20/40 loss=0.035323
c0 epoch 30/40 loss=0.025954
c0 epoch 40/40 loss=0.020488
c1 epoch 1/40 loss=3.383299
c1 epoch 10/40 loss=0.149253
c1 epoch 20/40 loss=0.046591
c1 epoch 30/40 loss=0.029699
c1 epoch 40/40 loss=0.022912
c2 epoch 1/40 loss=2.855274
c2 epoch 10/40 loss=0.109034
c2 epoch 20/40 loss=0.040014
c2 epoch 30/40 loss=0.029119
c2 epoch 40/40 loss=0.022536
{
  "protocol": "WorldPlay fulltrain100 (8400 transitions); independent validation25 (2100 transitions); no validation tuning",
  "train_samples": 8400,
  "validation_samples": 2100,
  "c1_validation_metrics": {
    "overall": {
      "model": "Confidence C1",
      "split": "validation",
      "step": "overall",
      "spearman": 0.9529179491248462,
      "pearson": 0.96539306640625,
      "log_mae": 0.15999889373779297
    },
    "P1": {
      "model": "Confidence C1",
      "split": "validation",
      "step": "P1",
      "spearman": 0.8590718873187204,
      "pearson": 0.8829622268676758,
      "log_mae": 0.17691564559936523
    },
    "P2": {
      "model": "Confidence C1",
      "split": "validation",
      "step": "P2",
      "spearman": 0.8631690385373965,
      "pearson": 0.9074602127075195,
      "log_mae": 0.14503394067287445
    },
    "P3": {
      "model": "Confidence C1",
      "split": "validation",
      "step": "P3",
      "spearman": 0.8700319930976419,
      "pearson": 0.8872363567352295,
      "log_mae": 0.15804709494113922
    }
  },
  "remaining_risk_at_33pct": {
    "Random": 0.009558672141283751,
    "Timestep-only": 0.005005753133445978,
    "Confidence C0": 0.008430036716163158,
    "Confidence C1": 0.004878982435911894,
    "Confidence C2": 0.004925357177853584,
    "Oracle": 0.004840569104999304
  },
  "gain_vs_random": 0.48957529207015615,
  "oracle_gap": 0.007935705508866102,
  "routing_at_33pct": {
    "P1": {
      "selection_fraction": 0.012857142857142857,
      "within_step_fallback_rate": 0.012857142857142857
    },
    "P2": {
      "selection_fraction": 0.05714285714285714,
      "within_step_fallback_rate": 0.05714285714285714
    },
    "P3": {
      "selection_fraction": 0.93,
      "within_step_fallback_rate": 0.93
    }
  }
}