Add files using upload-large-folder tool

README.md

@@ -0,0 +1,107 @@
+---
+license: apache-2.0
+library_name: pytorch
+datasets:
+- Skylion007/openwebtext
+tags:
+- discrete-diffusion
+- uniform-state-diffusion
+- duo
+- scaling-laws
+- checkpoints
+---
+
+# Duo data-fraction scaling study: `medium` models
+
+10 uniform-state discrete diffusion language models (Duo, `duo_base`), identical
+except for the size of their training set: nested subsets of OpenWebText holding
+10%, 20%, ..., 100% of the training blocks. **Training is in progress** (target 1,000,000 steps). This repository is a backup of
+the grid checkpoints reached so far and will be updated as training continues.
+Each folder holds **full checkpoints** on a fixed step grid (53 at 1M), so the whole
+training trajectory can be analysed (e.g. test loss against weight norm, in the
+spirit of D'Amico, Bocchi & Negri, *Implicit bias produces neural scaling laws in
+learning curves*).
+
+## Models
+
+| fraction | training blocks (1,024 tokens) | step | epochs | val NELBO at that step (nats/token) | training hours |
+|---|---|---|---|---|---|
+| 0.1 | 873,094 | 325,000 | 191 | 3.0383 | 155 |
+| 0.2 | 1,746,188 | 350,000 | 103 | 3.0142 | 173 |
+| 0.3 | 2,619,283 | 275,000 | 54 | 3.0438 | 138 |
+| 0.4 | 3,492,377 | 325,000 | 48 | 3.0191 | 159 |
+| 0.5 | 4,365,471 | 625,000 | 73 | 2.9651 | 307 |
+| 0.6 | 5,238,565 | 275,000 | 27 | 3.0327 | 138 |
+| 0.7 | 6,111,659 | 275,000 | 23 | 3.0296 | 133 |
+| 0.8 | 6,984,754 | 275,000 | 20 | 3.0287 | 139 |
+| 0.9 | 7,857,848 | 275,000 | 18 | 3.0334 | 135 |
+| 1.0 | 8,730,942 | 275,000 | 16 | 3.0420 | 133 |
+
+Validation NELBO is computed with the EMA weights on the full OpenWebText validation split
+(110,464 blocks), with fixed noise, every 5,000 steps (`val/nll` in the metrics files).
+
+## Training setup (same for every model)
+
+- **Model:** DiT backbone, config `medium` (424,477,010 parameters in the backbone),
+  context length 1,024, GPT-2 tokenizer (vocabulary 50,258), untied embeddings.
+- **Algorithm:** `duo_base`: uniform-state discrete diffusion trained on the exact NELBO,
+  log-linear noise schedule (no curriculum or distillation).
+- **Optimisation:** AdamW, learning rate 3e-4, constant after 2,500 warm-up steps,
+  **weight decay 0**, gradient clipping 1.0, EMA decay 0.9999, bf16 mixed precision,
+  global batch 512 sequences x 1,024 tokens, 1,000,000 steps (target), seed 1.
+- **Data:** OpenWebText (`Skylion007/openwebtext`, revision `79d93d7`), GPT-2 tokenized and
+  wrapped into 1,024-token blocks: 8,730,942 training blocks; the last 100,000
+  documents form the validation split. Fraction *f* uses the first floor(f x N + 0.5)
+  blocks of one fixed permutation (seed 20250927, sha256 in `run_info.json`), so the
+  subsets are nested (0.1 is contained in 0.2, and so on). The data order depends only on
+  the optimizer step, not on the GPU layout.
+- **Code:** a modified version of [s-sahoo/duo](https://github.com/s-sahoo/duo)
+  (scaling-study branch, commit `7c9e7d04d4a4`, recorded in `code_sha`).
+
+## Files
+
+```
+frac_0.10/ ... frac_1.00/
+  checkpoints/step_0000000.ckpt ...                     PyTorch Lightning checkpoints (53 at 1M)
+  checkpoints/index.csv        step, wall-clock training hours, tokens and epochs per checkpoint
+  metrics/version_*/metrics.csv  training/validation metrics (one file per job segment)
+  norms/norms_raw.jsonl, norms_ema.jsonl  weight norms every 500 steps, raw and EMA weights
+  run_info.json, config_resolved.yaml, code_sha, segments.jsonl
+```
+
+Checkpoint steps so far: 0, 100, 200, 400, 700, 1k, 1.5k, 2.5k, 4k, 6k, 10k, 15k, 20k, then every 25k to 1M.
+
+Each checkpoint has the raw weights, the EMA weights, the optimizer state and the
+loop/sampler state, so training can be resumed exactly. Step 0 is about half the size
+(no optimizer state yet).
+
+```python
+import torch
+
+ck = torch.load("frac_0.10/checkpoints/step_1000000.ckpt", map_location="cpu", weights_only=False)
+raw = {k: v for k, v in ck["state_dict"].items() if k.startswith("backbone.")}
+# EMA tensors follow the backbone's trainable parameters in order (all of the
+# state_dict except the rotary-embedding buffer).
+names = [k for k in raw if not k.endswith("rotary_emb.inv_freq")]
+ema = dict(zip(names, ck["ema"]["shadow_params"]))
+print(ck["global_step"], ck["scaling_meta"]["clock"]["train_s"] / 3600, "training hours")
+```
+
+**Norms** (`norms/*.jsonl`): per record the step, aggregate norms of the token-path matrices
+(embedding, per-block Q/K/V, attention output, MLP, output layer) and, separately, of the
+timestep-conditioning path, plus per-matrix statistics (spectral norm, Frobenius, L1,
+row and column (2,1)-norms) in float64. `log_R` is the Bartlett spectral complexity:
+`sum log||W||_sigma + 1.5 log sum (||W^T||_2,1^(2/3) / ||W||_sigma^(2/3))`. It is null at
+step 0 because the output layer starts at zero.
+
+## Caveats
+
+- **Wall-clock time** (`train_hours`) mixes H100 and H200 segments (the GPU of every
+  segment is in `segments.jsonl`). For time-based comparisons, use steps or normalise
+  per GPU type.
+- `trainer/loss` in the metrics is one micro-batch on one GPU and depends on the GPU
+  layout; compare runs with `val/nll`.
+- Some medium segments ran on a node with a faulty GPU (alphagpu08, NaN losses). Their
+  outputs were removed and those runs resumed from the last clean checkpoint, so every
+  published checkpoint comes from clean hardware.
+- **frac_0.30**: steps 0-2,500 ran on a throttled node, so its early `train_hours` are inflated.

frac_0.10/checkpoints/index.csv

@@ -0,0 +1,173 @@
+step,kind,file,unix_time,save_s,train_hours,gpu_hours,samples,tokens,epochs,job_id
+0,grid,step_0000000.ckpt,1790572649.311,6.251,0.002641662416151828,0.010566649664607313,0,0,0.0,105184
+100,grid,step_0000100.ckpt,1790572858.371,10.157,0.060713790488791546,0.24285516195516618,51200,52428800,0.05864202479916252,105184
+200,grid,step_0000200.ckpt,1790573051.528,11.390,0.11436868417775259,0.45747473671101035,102400,104857600,0.11728404959832504,105184
+400,grid,step_0000400.ckpt,1790573421.295,11.555,0.2170817633645816,0.8683270534583264,204800,209715200,0.2345680991966501,105184
+700,grid,step_0000700.ckpt,1790573969.681,11.089,0.369410959408722,1.477643837634888,358400,367001600,0.41049417359413765,105184
+1000,grid,step_0001000.ckpt,1790574512.914,11.398,0.5203090177246162,2.081236070898465,512000,524288000,0.5864202479916252,105184
+1500,grid,step_0001500.ckpt,1790575408.953,11.890,0.769208844103333,3.076835376413332,768000,786432000,0.8796303719874378,105184
+2500,grid,step_0002500.ckpt,1790577198.829,12.059,1.2663966148685561,5.065586459474225,1280000,1310720000,1.466050619979063,105184
+2500,last,last.ckpt,1790577210.495,11.663,1.2696372371003963,5.078548948401585,1280000,1310720000,1.466050619979063,105184
+4000,grid,step_0004000.ckpt,1790579879.120,10.428,2.0109219288052476,8.04368771522099,2048000,2097152000,2.3456809919665007,105184
+5000,last,last.ckpt,1790581658.643,11.518,2.5052338125238505,10.020935250095402,2560000,2621440000,2.932101239958126,105184
+6000,grid,step_0006000.ckpt,1790583564.130,11.098,3.0345357115103657,12.138142846041463,3072000,3145728000,3.518521487949751,105184
+7500,last,last.ckpt,1790586247.509,11.170,3.779918927956476,15.119675711825904,3840000,3932160000,4.3981518599371885,105184
+10000,grid,step_0010000.ckpt,1790590683.977,11.273,5.012271109333831,20.049084437335324,5120000,5242880000,5.864202479916252,105184
+10000,last,last.ckpt,1790590695.068,11.087,5.015351814613419,20.061407258453677,5120000,5242880000,5.864202479916252,105184
+12500,last,last.ckpt,1790595240.914,10.364,6.278086855285801,25.112347421143205,6400000,6553600000,7.330253099895315,105184
+15000,grid,step_0015000.ckpt,1790599661.660,11.227,7.506072045358725,30.0242881814349,7680000,7864320000,8.796303719874377,105184
+15000,last,last.ckpt,1790599672.787,11.122,7.509162633259936,30.036650533039744,7680000,7864320000,8.796303719874377,105184
+17500,last,last.ckpt,1790604225.559,11.836,8.773821731434307,35.09528692573723,8960000,9175040000,10.262354339853442,105184
+20000,grid,step_0020000.ckpt,1790608645.734,10.491,10.001647965964189,40.006591863856755,10240000,10485760000,11.728404959832504,105184
+20000,last,last.ckpt,1790608656.969,11.232,10.004768945897618,40.01907578359047,10240000,10485760000,11.728404959832504,105184
+22500,last,last.ckpt,1790613209.690,11.431,11.26941366691582,45.07765466766328,11520000,11796480000,13.194455579811567,105184
+25000,grid,step_0025000.ckpt,1790617629.583,10.672,12.49716173326907,49.98864693307628,12800000,13107200000,14.66050619979063,105184
+25000,last,last.ckpt,1790617640.817,11.231,12.500282199308277,50.00112879723311,12800000,13107200000,14.66050619979063,105184
+27500,last,last.ckpt,1790622190.167,11.062,13.763990543914908,55.05596217565963,14080000,14417920000,16.126556819769693,105184
+30000,last,last.ckpt,1790626638.213,10.901,14.999558957320275,59.9982358292811,15360000,15728640000,17.592607439748754,105184
+32500,last,last.ckpt,1790631190.195,11.793,16.26399837710791,65.05599350843164,16640000,17039360000,19.05865805972782,105184
+35000,last,last.ckpt,1790635637.204,15.641,17.4992785198332,69.9971140793328,17920000,18350080000,20.524708679706883,105184
+37500,last,last.ckpt,1790640189.903,12.561,18.763917157138284,75.05566862855314,19200000,19660800000,21.990759299685944,105184
+40000,last,last.ckpt,1790644615.040,11.652,19.993121806595525,79.9724872263821,20480000,20971520000,23.45680991966501,105184
+42500,last,last.ckpt,1790649164.483,11.187,21.256856134825178,85.02742453930071,21760000,22282240000,24.92286053964407,105184
+45000,last,last.ckpt,1790653588.535,11.558,22.485759416986514,89.94303766794606,23040000,23592960000,26.388911159623134,105184
+47300,last,last.ckpt,1790657782.218,11.974,23.650671224312134,94.60268489724854,24217600,24798822400,27.73767773000387,105184
+47500,last,last.ckpt,1790710211.393,10.690,23.751732122754007,95.00692849101603,24320000,24903680000,27.854961779602196,105285
+50000,grid,step_0050000.ckpt,1790714638.701,11.553,24.98153993087769,99.92615972351076,25600000,26214400000,29.32101239958126,105285
+50000,last,last.ckpt,1790714649.637,10.932,24.984577537612836,99.93831015045134,25600000,26214400000,29.32101239958126,105285
+52500,last,last.ckpt,1790719187.373,10.618,26.245059859026743,104.98023943610697,26880000,27525120000,30.78706301956032,105285
+55000,last,last.ckpt,1790723599.741,11.082,27.470717613215093,109.88287045286037,28160000,28835840000,32.253113639539386,105285
+57500,last,last.ckpt,1790728132.992,10.253,28.729954035463226,114.9198161418529,29440000,30146560000,33.71916425951845,105285
+60000,last,last.ckpt,1790732542.738,11.667,29.9548833771883,119.8195335087532,30720000,31457280000,35.18521487949751,105285
+62500,last,last.ckpt,1790737076.542,11.326,31.21427345945862,124.85709383783448,32000000,32768000000,36.65126549947657,105285
+65000,last,last.ckpt,1790741484.840,11.335,32.43880060370261,129.75520241481044,33280000,34078720000,38.11731611945564,105285
+67500,last,last.ckpt,1790746020.004,11.513,33.69856839129262,134.79427356517047,34560000,35389440000,39.5833667394347,105285
+70000,last,last.ckpt,1790750429.049,11.640,34.9233030586206,139.6932122344824,35840000,36700160000,41.049417359413766,105285
+72500,last,last.ckpt,1790754963.888,11.186,36.18298075598794,144.73192302395177,37120000,38010880000,42.515467979392824,105285
+75000,grid,step_0075000.ckpt,1790759372.559,10.466,37.4076115632096,149.6304462528384,38400000,39321600000,43.98151859937189,105285
+75000,last,last.ckpt,1790759383.492,10.929,37.41064839226168,149.6425935690467,38400000,39321600000,43.98151859937189,105285
+77500,last,last.ckpt,1790763918.070,10.931,38.67025336439669,154.68101345758677,39680000,40632320000,45.44756921935095,105285
+80000,last,last.ckpt,1790768327.036,11.039,39.894966114771606,159.57986445908642,40960000,41943040000,46.91361983933002,105285
+82500,last,last.ckpt,1790772862.495,11.793,41.154815976839096,164.61926390735638,42240000,43253760000,48.379670459309075,105285
+85000,last,last.ckpt,1790777273.106,11.764,42.37998553542265,169.5199421416906,43520000,44564480000,49.84572107928814,105285
+87500,last,last.ckpt,1790781808.890,11.636,43.639925560324144,174.55970224129658,44800000,45875200000,51.311771699267204,105285
+90000,last,last.ckpt,1790786218.298,11.063,44.864761274171116,179.45904509668446,46080000,47185920000,52.77782231924627,105285
+92500,last,last.ckpt,1790790754.085,11.460,46.124702127235686,184.49880850894274,47360000,48496640000,54.24387293922533,105285
+94900,last,last.ckpt,1790794986.620,10.868,47.30040626671812,189.20162506687248,48588800,49754931200,55.65128153440523,105285
+95000,last,last.ckpt,1790937537.764,12.998,47.357156224445596,189.42862489778238,48640000,49807360000,55.70992355920439,109256
+97500,last,last.ckpt,1790942119.088,13.559,48.629746269963476,194.5189850798539,49920000,51118080000,57.175974179183456,109256
+100000,grid,step_0100000.ckpt,1790946585.240,11.987,49.8703438051821,199.4813752207284,51200000,52428800000,58.64202479916252,109256
+100000,last,last.ckpt,1790946598.278,13.034,49.8739656000768,199.4958624003072,51200000,52428800000,58.64202479916252,109256
+102500,last,last.ckpt,1790951186.009,13.314,51.14833539257473,204.59334157029892,52480000,53739520000,60.108075419141585,109256
+105000,last,last.ckpt,1790955638.095,12.882,52.38502599519393,209.5401039807757,53760000,55050240000,61.57412603912064,109256
+107500,last,last.ckpt,1790960221.426,13.651,53.65817347656289,214.63269390625155,55040000,56360960000,63.04017665909971,109256
+110000,last,last.ckpt,1790964697.456,17.716,54.90151505715679,219.60606022862717,56320000,57671680000,64.50622727907877,109256
+112500,last,last.ckpt,1790969392.489,15.791,56.205690847209965,224.82276338883986,57600000,58982400000,65.97227789905783,109256
+115000,last,last.ckpt,1790973963.176,15.130,57.47532627884893,229.90130511539573,58880000,60293120000,67.4383285190369,109256
+117500,last,last.ckpt,1790978660.637,15.873,58.78017638411554,235.12070553646217,60160000,61603840000,68.90437913901596,109256
+120000,last,last.ckpt,1790983229.325,15.527,60.049256311739384,240.19702524695754,61440000,62914560000,70.37042975899502,109256
+122500,last,last.ckpt,1790987924.140,15.222,61.353371806592264,245.41348722636906,62720000,64225280000,71.83648037897409,109256
+125000,grid,step_0125000.ckpt,1790992491.898,15.210,62.62219331537011,250.48877326148045,64000000,65536000000,73.30253099895315,109256
+125000,last,last.ckpt,1790992507.187,15.284,62.62644024873851,250.50576099495404,64000000,65536000000,73.30253099895315,109256
+127500,last,last.ckpt,1790997203.001,15.870,63.9308331652635,255.723332661054,65280000,66846720000,74.76858161893222,109256
+130000,last,last.ckpt,1791001772.233,15.166,65.20006433843402,260.8002573537361,66560000,68157440000,76.23463223891127,109256
+132500,last,last.ckpt,1791006470.269,16.173,66.50507422600123,266.0202969040049,67840000,69468160000,77.70068285889033,109256
+135000,last,last.ckpt,1791011054.071,17.539,67.77835258010671,271.11341032042685,69120000,70778880000,79.1667334788694,109256
+137500,last,last.ckpt,1791015735.773,17.295,69.07882540228802,276.3153016091521,70400000,72089600000,80.63278409884846,109256
+140000,last,last.ckpt,1791020276.846,17.255,70.34023457992848,281.3609383197139,71680000,73400320000,82.09883471882753,109256
+141080,last,last.ckpt,1791022375.218,17.697,70.9231155722728,283.6924622890912,72232960,73966551040,82.73216858665847,109256
+142500,last,last.ckpt,1791198545.060,8.217,71.54064313465626,286.16257253862506,72960000,74711040000,83.56488533880659,113981
+145000,last,last.ckpt,1791202453.384,8.031,72.62628868977669,290.50515475910674,74240000,76021760000,85.03093595878565,113981
+147500,last,last.ckpt,1791206475.347,7.902,73.743500407958,294.974001631832,75520000,77332480000,86.49698657876472,113981
+150000,grid,step_0150000.ckpt,1791210379.940,7.667,74.82810956886142,299.3124382754457,76800000,78643200000,87.96303719874378,113981
+150000,last,last.ckpt,1791210387.963,8.020,74.83033817992546,299.32135271970185,76800000,78643200000,87.96303719874378,113981
+152500,last,last.ckpt,1791214412.480,8.913,75.94825973115499,303.79303892461996,78080000,79953920000,89.42908781872283,113981
+155000,last,last.ckpt,1791218320.002,7.992,77.03368249365569,308.13472997462276,79360000,81264640000,90.8951384387019,113981
+157500,last,last.ckpt,1791222344.820,8.007,78.15168751903576,312.60675007614304,80640000,82575360000,92.36118905868096,113981
+160000,last,last.ckpt,1791226251.501,8.382,79.23687672031294,316.94750688125174,81920000,83886080000,93.82723967866004,113981
+162500,last,last.ckpt,1791230272.474,8.581,80.35381359559457,321.4152543823783,83200000,85196800000,95.29329029863909,113981
+165000,last,last.ckpt,1791234179.775,8.102,81.43917505121557,325.7567002048623,84480000,86507520000,96.75934091861815,113981
+167500,last,last.ckpt,1791238202.570,8.062,82.55661810639081,330.22647242556326,85760000,87818240000,98.22539153859722,113981
+170000,last,last.ckpt,1791242109.177,8.083,83.64178674102529,334.56714696410114,87040000,89128960000,99.69144215857628,113981
+172500,last,last.ckpt,1791246132.014,8.359,84.75924124149864,339.03696496599457,88320000,90439680000,101.15749277855535,113981
+175000,grid,step_0175000.ckpt,1791250038.111,7.612,85.84426836470867,343.3770734588347,89600000,91750400000,102.62354339853441,113981
+175000,last,last.ckpt,1791250046.189,8.075,85.84651215369658,343.3860486147863,89600000,91750400000,102.62354339853441,113981
+177500,last,last.ckpt,1791254069.168,7.695,86.96400631534246,347.85602526136984,90880000,93061120000,104.08959401851347,113981
+180000,last,last.ckpt,1791257977.865,8.308,88.04975561206152,352.19902244824607,92160000,94371840000,105.55564463849254,113981
+182500,last,last.ckpt,1791262002.901,8.283,89.16782107626734,356.6712843050694,93440000,95682560000,107.0216952584716,113981
+185000,last,last.ckpt,1791265912.530,8.144,90.25382908592596,361.01531634370383,94720000,96993280000,108.48774587845065,113981
+187500,last,last.ckpt,1791269939.876,7.995,91.37253627820328,365.49014511281314,96000000,98304000000,109.95379649842972,113981
+190000,last,last.ckpt,1791273848.296,7.898,92.45820859565218,369.8328343826087,97280000,99614720000,111.41984711840878,113981
+192500,last,last.ckpt,1791277871.600,8.067,93.5757928988917,374.3031715955668,98560000,100925440000,112.88589773838785,113981
+194740,last,last.ckpt,1791281371.664,7.925,94.5480331295871,378.1921325183484,99706880,102099845120,114.1994790938891,113981
+195000,last,last.ckpt,1791522935.663,8.611,94.6665826822582,378.6663307290328,99840000,102236160000,114.35194835836691,119492
+197500,last,last.ckpt,1791526980.151,7.865,95.79005146727096,383.1602058690838,101120000,103546880000,115.81799897834597,119492
+200000,grid,step_0200000.ckpt,1791530912.379,7.363,96.88233695610504,387.52934782442014,102400000,104857600000,117.28404959832504,119492
+200000,last,last.ckpt,1791530920.406,8.024,96.88456669150308,387.5382667660123,102400000,104857600000,117.28404959832504,119492
+202500,last,last.ckpt,1791534962.497,8.666,98.00736984836041,392.02947939344165,103680000,106168320000,118.7501002183041,119492
+205000,last,last.ckpt,1791538890.213,7.866,99.0984020678523,396.3936082714092,104960000,107479040000,120.21615083828317,119492
+207500,last,last.ckpt,1791542939.810,8.135,100.22329020700546,400.89316082802185,106240000,108789760000,121.68220145826223,119492
+210000,last,last.ckpt,1791546870.615,7.788,101.31518034028691,405.26072136114766,107520000,110100480000,123.14825207824128,119492
+212500,last,last.ckpt,1791550918.030,7.899,102.43946225148315,409.7578490059326,108800000,111411200000,124.61430269822036,119492
+215000,last,last.ckpt,1791554849.775,8.169,103.53161365099459,414.12645460397835,110080000,112721920000,126.08035331819941,119492
+217500,last,last.ckpt,1791558898.206,7.994,104.65617778737072,418.6247111494829,111360000,114032640000,127.54640393817847,119492
+220000,last,last.ckpt,1791562830.352,8.015,105.74844074988324,422.993762999533,112640000,115343360000,129.01245455815754,119492
+222500,last,last.ckpt,1791566878.636,8.029,106.8729638869856,427.4918555479424,113920000,116654080000,130.4785051781366,119492
+225000,grid,step_0225000.ckpt,1791570809.449,7.375,107.96485654639545,431.8594261855818,115200000,117964800000,131.94455579811566,119492
+225000,last,last.ckpt,1791570817.778,8.326,107.96717008783276,431.868680351331,115200000,117964800000,131.94455579811566,119492
+227500,last,last.ckpt,1791574862.195,8.093,109.09061920787026,436.36247683148105,116480000,119275520000,133.41060641809472,119492
+230000,last,last.ckpt,1791578793.489,7.924,110.18264528782237,440.73058115128947,117760000,120586240000,134.8766570380738,119492
+232500,last,last.ckpt,1791582837.788,8.128,111.30606183448056,445.22424733792224,119040000,121896960000,136.34270765805286,119492
+235000,last,last.ckpt,1791586765.082,7.806,112.39697677589074,449.58790710356294,120320000,123207680000,137.80875827803192,119492
+237500,last,last.ckpt,1791590814.024,8.232,113.521682915247,454.086731660988,121600000,124518400000,139.27480889801097,119492
+240000,last,last.ckpt,1791594744.062,7.903,114.61336000381348,458.45344001525393,122880000,125829120000,140.74085951799003,119492
+242500,last,last.ckpt,1791598791.607,7.955,115.73767811762738,462.95071247050953,124160000,127139840000,142.20691013796912,119492
+245000,last,last.ckpt,1791602724.933,7.736,116.83026862367393,467.3210744946957,125440000,128450560000,143.67296075794818,119492
+247500,last,last.ckpt,1791606768.954,7.976,117.9536078567627,471.8144314270508,126720000,129761280000,145.13901137792723,119492
+248040,last,last.ckpt,1791607622.693,7.978,118.19075764850196,472.76303059400783,126996480,130044395520,145.45567831184272,119492
+250000,grid,step_0250000.ckpt,1791897035.152,11.501,119.16816002116875,476.672640084675,128000000,131072000000,146.6050619979063,127681
+250000,last,last.ckpt,1791897046.466,11.310,119.17130261914778,476.6852104765911,128000000,131072000000,146.6050619979063,127681
+252500,last,last.ckpt,1791901645.596,11.167,120.44883876445057,481.7953550578023,129280000,132382720000,148.07111261788535,127681
+255000,last,last.ckpt,1791906117.606,11.461,121.69106379732361,486.76425518929443,130560000,133693440000,149.53716323786443,127681
+257500,last,last.ckpt,1791910719.154,12.167,122.96927162364254,491.87708649457016,131840000,135004160000,151.0032138578435,127681
+260000,last,last.ckpt,1791915210.561,17.500,124.21688477355909,496.86753909423635,133120000,136314880000,152.46926447782255,127681
+262500,last,last.ckpt,1791919817.309,13.663,125.49653677298713,501.9861470919485,134400000,137625600000,153.9353150978016,127681
+265000,last,last.ckpt,1791924287.806,12.823,126.73834157337538,506.9533662935015,135680000,138936320000,155.40136571778066,127681
+267500,last,last.ckpt,1791928872.291,12.912,128.01180963505138,512.0472385402055,136960000,140247040000,156.86741633775975,127681
+270000,last,last.ckpt,1791933343.435,12.277,129.2537940535164,517.0151762140656,138240000,141557760000,158.3334669577388,127681
+272500,last,last.ckpt,1791937939.886,13.568,130.53058618935108,522.1223447574043,139520000,142868480000,159.79951757771786,127681
+275000,grid,step_0275000.ckpt,1791942409.552,10.908,131.77216008606314,527.0886403442526,140800000,144179200000,161.26556819769692,127681
+275000,last,last.ckpt,1791942420.855,11.299,131.77529977225777,527.1011990890311,140800000,144179200000,161.26556819769692,127681
+277500,last,last.ckpt,1791947018.870,12.267,133.0525259383442,532.2101037533768,142080000,145489920000,162.73161881767598,127681
+280000,last,last.ckpt,1791951494.125,12.686,134.29565248771803,537.1826099508721,143360000,146800640000,164.19766943765507,127681
+282500,last,last.ckpt,1791956095.734,12.673,135.57387711324523,542.2955084529809,144640000,148111360000,165.66372005763412,127681
+285000,last,last.ckpt,1791960577.855,17.488,136.81891078539942,547.2756431415977,145920000,149422080000,167.12977067761318,127681
+287500,last,last.ckpt,1791965214.244,15.670,138.1067966534487,552.4271866137948,147200000,150732800000,168.59582129759224,127681
+290000,last,last.ckpt,1791969719.824,16.526,139.35834654693284,557.4333861877313,148480000,152043520000,170.0618719175713,127681
+292500,last,last.ckpt,1791974349.576,15.196,140.6443887399701,562.5775549598804,149760000,153354240000,171.52792253755035,127681
+294900,last,last.ckpt,1791978645.022,12.144,141.83756842141557,567.3502736856623,150988800,154612531200,172.93533113273025,127681
+295000,last,last.ckpt,1792209786.105,8.312,141.88305162129043,567.5322064851617,151040000,154664960000,172.99397315752944,133115
+297500,last,last.ckpt,1792213817.791,8.308,143.0029644545173,572.0118578180692,152320000,155975680000,174.4600237775085,133115
+300000,grid,step_0300000.ckpt,1792217732.728,7.799,144.09044687501518,576.3617875000607,153600000,157286400000,175.92607439748755,133115
+300000,last,last.ckpt,1792217740.724,7.992,144.09266796667904,576.3706718667162,153600000,157286400000,175.92607439748755,133115
+302500,last,last.ckpt,1792221771.914,8.169,145.21244297920572,580.8497719168229,154880000,158597120000,177.3921250174666,133115
+305000,last,last.ckpt,1792225689.814,8.283,146.3007484927256,585.2029939709024,156160000,159907840000,178.85817563744567,133115
+307500,last,last.ckpt,1792229730.581,8.664,147.42318372105558,589.6927348842223,157440000,161218560000,180.32422625742475,133115
+310000,last,last.ckpt,1792233648.626,8.169,148.51152952696197,594.0461181078479,158720000,162529280000,181.7902768774038,133115
+312500,last,last.ckpt,1792237680.637,8.306,149.63153254579288,598.5261301831715,160000000,163840000000,183.25632749738287,133115
+315000,last,last.ckpt,1792241596.166,8.141,150.71917955761518,602.8767182304607,161280000,165150720000,184.72237811736193,133115
+317500,last,last.ckpt,1792245628.928,8.286,151.8393911639606,607.3575646558425,162560000,166461440000,186.18842873734098,133115
+320000,last,last.ckpt,1792249544.849,8.039,152.92714711062558,611.7085884425023,163840000,167772160000,187.65447935732007,133115
+322500,last,last.ckpt,1792253580.806,8.176,154.0482462664409,616.1929850657637,165120000,169082880000,189.12052997729913,133115
+325000,grid,step_0325000.ckpt,1792257500.707,7.828,155.13710769009157,620.5484307603663,166400000,170393600000,190.58658059727819,133115
+325000,last,last.ckpt,1792257508.807,8.096,155.13935760444497,620.5574304177799,166400000,170393600000,190.58658059727819,133115
+327500,last,last.ckpt,1792261541.578,8.155,156.25957186413464,625.0382874565386,167680000,171704320000,192.05263121725724,133115
+330000,last,last.ckpt,1792265456.377,7.883,157.34701598244666,629.3880639297867,168960000,173015040000,193.5186818372363,133115
+332500,last,last.ckpt,1792269487.598,8.163,158.46679954345183,633.8671981738073,170240000,174325760000,194.9847324572154,133115
+335000,last,last.ckpt,1792273402.317,7.941,159.55422156485625,638.216886259425,171520000,175636480000,196.45078307719444,133115
+337500,last,last.ckpt,1792277433.378,8.239,160.67396058966818,642.6958423586727,172800000,176947200000,197.9168336971735,133115
+340000,last,last.ckpt,1792281353.418,8.845,161.76286065040208,647.0514426016083,174080000,178257920000,199.38288431715256,133115
+342500,last,last.ckpt,1792285386.787,8.120,162.8832411159399,651.5329644637596,175360000,179568640000,200.84893493713162,133115
+345000,last,last.ckpt,1792289303.132,8.399,163.97111467130742,655.8844586852297,176640000,180879360000,202.3149855571107,133115
+347500,last,last.ckpt,1792293335.292,8.429,165.0911591538492,660.3646366153968,177920000,182190080000,203.78103617708976,133115
+348420,last,last.ckpt,1792294778.822,8.209,165.49213964875116,661.9685585950047,178391040,182672424960,204.32054280524204,133115

frac_0.10/code_sha

@@ -0,0 +1 @@
+7c9e7d04d4a4

frac_0.10/config_resolved.yaml

@@ -0,0 +1,204 @@
+mode: train
+seed: 1
+loader:
+  global_batch_size: 512
+  eval_global_batch_size: 64
+  batch_size: 16
+  eval_batch_size: 16
+  num_workers: 4
+  pin_memory: true
+  sampler: step_indexed
+sampling:
+  predictor: ancestral
+  steps: 1000
+  noise_removal: ancestral
+  use_float64: true
+  p_nucleus: 1.0
+  num_sample_batches: 0
+  num_sample_log: 2
+  semi_ar: false
+  stride_length: 1
+  num_strides: 1
+  guid_weight: null
+  psi:
+    time_profile: linear
+    high_mode: pure-posterior
+    middle_mode: pure-posterior
+    low_mode: pure-posterior
+    high_frac: 0.2
+    middle_frac: 0.6
+training:
+  ema: 0.9999
+  antithetic_sampling: true
+  importance_sampling: false
+  sampling_eps: 0.001
+  change_of_variables: false
+  loss_precision: bf16
+  finetune_path: ''
+  class_dropout_p: 0.1
+eval:
+  checkpoint_path: ''
+  disable_ema: false
+  compute_generative_perplexity: false
+  perplexity_batch_size: 8
+  compute_perplexity_on_sanity: false
+  gen_ppl_eval_model_name_or_path: gpt2-large
+  generate_samples: false
+  generated_samples_path: /mnt/home/phamd/dm_scaling_law_runs/duo_base/medium/frac_0.10/samples.json
+optim:
+  weight_decay: 0
+  lr: 0.0003
+  beta1: 0.9
+  beta2: 0.999
+  eps: 1.0e-08
+trainer:
+  _target_: lightning.Trainer
+  accelerator: cuda
+  num_nodes: 1
+  devices: 4
+  accumulate_grad_batches: 8
+  gradient_clip_val: 1.0
+  precision: bf16
+  num_sanity_val_steps: 2
+  max_steps: 1000000
+  log_every_n_steps: 100
+  limit_train_batches: 1.0
+  limit_val_batches: 1.0
+  val_check_interval: 40000
+  check_val_every_n_epoch: null
+  enable_checkpointing: false
+  enable_progress_bar: false
+wandb: null
+checkpointing:
+  save_dir: /mnt/home/phamd/dm_scaling_law_runs/duo_base/medium/frac_0.10
+  resume_from_ckpt: true
+  resume_ckpt_path: /mnt/home/phamd/dm_scaling_law_runs/duo_base/medium/frac_0.10/checkpoints/last.ckpt
+callbacks:
+  training_clock:
+    _target_: scaling_callbacks.TrainingClock
+    run_dir: /mnt/home/phamd/dm_scaling_law_runs/duo_base/medium/frac_0.10
+    global_batch_size: 512
+    seq_len: 1024
+    max_steps: 1000000
+    log_every_n_steps: 100
+    print_every_n_steps: 500
+  norm_monitor:
+    _target_: scaling_callbacks.NormMonitor
+    dirpath: /mnt/home/phamd/dm_scaling_law_runs/duo_base/medium/frac_0.10/norms
+    every_n_steps: 500
+    max_steps: 1000000
+    grid_every: 25000
+    grid_early:
+    - 100
+    - 200
+    - 400
+    - 700
+    - 1000
+    - 1500
+    - 2500
+    - 4000
+    - 6000
+    - 10000
+    - 15000
+    - 20000
+  scaling_checkpoint:
+    _target_: scaling_callbacks.ScalingCheckpoint
+    dirpath: /mnt/home/phamd/dm_scaling_law_runs/duo_base/medium/frac_0.10/checkpoints
+    max_steps: 1000000
+    grid_every: 25000
+    grid_early:
+    - 100
+    - 200
+    - 400
+    - 700
+    - 1000
+    - 1500
+    - 2500
+    - 4000
+    - 6000
+    - 10000
+    - 15000
+    - 20000
+    rolling_every: 2500
+    walltime_margin_s: 1200
+  learning_rate_monitor:
+    _target_: lightning.pytorch.callbacks.LearningRateMonitor
+    logging_interval: step
+data:
+  train: openwebtext-train
+  valid: openwebtext-valid
+  modality: text
+  tokenizer_name_or_path: gpt2
+  cache_dir: /mnt/home/phamd/dm_scaling_law_data/text_data
+  wrap: true
+  streaming: false
+  insert_train_eos: true
+  insert_valid_eos: true
+  train_fraction: 0.1
+  subset_seed: 20250927
+  subset_perm_path: /mnt/home/phamd/dm_scaling_law_data/subsets/owt_train_perm_s20250927.npy
+  subset_perm_sha256: null
+  subset_tokens_path: /mnt/home/phamd/dm_scaling_law_data/subsets/owt_train_perm_s20250927_input_ids.npy
+  order_seed: 1
+model:
+  name: medium
+  type: ddit
+  hidden_size: 1024
+  cond_dim: 128
+  length: 1024
+  n_blocks: 24
+  n_heads: 16
+  scale_by_sigma: true
+  dropout: 0.1
+  tie_word_embeddings: false
+  vocab_lookup: true
+strategy:
+  _target_: lightning.pytorch.strategies.DDPStrategy
+  find_unused_parameters: false
+noise:
+  type: log-linear
+  eps: 0.001
+lr_scheduler:
+  _target_: transformers.get_constant_schedule_with_warmup
+  num_warmup_steps: 2500
+prior:
+  type: none
+  latent_width: 0
+  latent_height: 0
+algo:
+  name: duo_base
+  backbone: dit
+  parameterization: mean
+  time_conditioning: true
+  T: 0
+  subs_masking: false
+  causal_attention: false
+  ignore_bos: false
+  loss_type: elbo
+  compile_loss: true
+scaling:
+  val_every_steps: 5000
+  rolling_every: 2500
+  reseed_every: 1
+  grid_every: 25000
+  grid_early:
+  - 100
+  - 200
+  - 400
+  - 700
+  - 1000
+  - 1500
+  - 2500
+  - 4000
+  - 6000
+  - 10000
+  - 15000
+  - 20000
+  norm_every: 500
+  val_noise_seed: 4242
+  walltime_margin_s: 1200
+scaling_wandb:
+  project: duo-scaling-medium
+  group: medium
+  name: duo_base-medium-f0.1-s1
+  id: duobase-medium-f010-s1

frac_0.10/run_info.json

@@ -0,0 +1,24 @@
+{
+  "subset": {
+    "train_fraction": 0.1,
+    "n_blocks": 873094,
+    "n_total": 8730942,
+    "perm_path": "/mnt/home/phamd/dm_scaling_law_data/subsets/owt_train_perm_s20250927.npy",
+    "perm_sha256": "fa545515efbfdd07138abceebb2671be60b5047431da2f50fd58a8bb986d05a7",
+    "subset_seed": 20250927,
+    "steps_per_epoch": 1705
+  },
+  "model": "medium",
+  "algo": "duo_base",
+  "seed": 1,
+  "order_seed": 1,
+  "global_batch_size": 512,
+  "max_steps": 1000000,
+  "num_parameters": 424477010,
+  "batch_size": 16,
+  "accumulate_grad_batches": 8,
+  "world_size": 4,
+  "gpu": "NVIDIA H100 80GB HBM3",
+  "code_sha": "7c9e7d04d4a4",
+  "code_dir": "/mnt/home/phamd/dm_scaling_law_runs/code/7c9e7d04d4a4"
+}

frac_0.10/segments.jsonl

@@ -0,0 +1,21 @@
+{"event": "job_start", "unix_time": 1790572568, "job_id": "105184", "restart": 0, "node": "alphagpu07"}
+{"event": "start", "unix_time": 1790572639.8012102, "step": 0, "train_hours": 2.0807298521200818e-09, "job_id": "105184", "node": "alphagpu07", "world_size": 4, "gpu": "NVIDIA H100 80GB HBM3"}
+{"event": "end", "unix_time": 1790657782.2241964, "step": 47300, "train_hours": 23.65067307560472, "job_id": "105184"}
+{"event": "job_start", "unix_time": 1790709760, "job_id": "105285", "restart": 0, "node": "alphagpu12"}
+{"event": "start", "unix_time": 1790709835.6023924, "step": 47300, "train_hours": 23.64734572763296, "job_id": "105285", "node": "alphagpu12", "world_size": 4, "gpu": "NVIDIA H100 80GB HBM3"}
+{"event": "end", "unix_time": 1790794986.6303139, "step": 94900, "train_hours": 47.300409077127824, "job_id": "105285"}
+{"event": "job_start", "unix_time": 1790937167, "job_id": "109256", "restart": 0, "node": "alphagpu09"}
+{"event": "start", "unix_time": 1790937322.5990257, "step": 94900, "train_hours": 47.297388090008816, "job_id": "109256", "node": "alphagpu09", "world_size": 4, "gpu": "NVIDIA H100 80GB HBM3"}
+{"event": "end", "unix_time": 1791022375.2728682, "step": 141080, "train_hours": 70.92313083088155, "job_id": "109256"}
+{"event": "job_start", "unix_time": 1791196168, "job_id": "113981", "restart": 0, "node": "alphagpu21"}
+{"event": "start", "unix_time": 1791196304.2675822, "step": 141080, "train_hours": 70.9182006810854, "job_id": "113981", "node": "alphagpu21", "world_size": 4, "gpu": "NVIDIA H200"}
+{"event": "end", "unix_time": 1791281371.6707516, "step": 194740, "train_hours": 94.54803490448045, "job_id": "113981"}
+{"event": "job_start", "unix_time": 1791522403, "job_id": "119492", "restart": 0, "node": "alphagpu19"}
+{"event": "start", "unix_time": 1791522500.960534, "step": 194740, "train_hours": 94.54583192271646, "job_id": "119492", "node": "alphagpu19", "world_size": 4, "gpu": "NVIDIA H200"}
+{"event": "end", "unix_time": 1791607622.7009652, "step": 248040, "train_hours": 118.19075983000324, "job_id": "119492"}
+{"event": "job_start", "unix_time": 1791893422, "job_id": "127681", "restart": 0, "node": "alphagpu12"}
+{"event": "start", "unix_time": 1791893508.5272057, "step": 248040, "train_hours": 118.18854195353057, "job_id": "127681", "node": "alphagpu12", "world_size": 4, "gpu": "NVIDIA H100 80GB HBM3"}
+{"event": "end", "unix_time": 1791978645.0304015, "step": 294900, "train_hours": 141.83757062998936, "job_id": "127681"}
+{"event": "job_start", "unix_time": 1792209549, "job_id": "133115", "restart": 0, "node": "alphagpu23"}
+{"event": "start", "unix_time": 1792209610.2243683, "step": 294900, "train_hours": 141.83419580707584, "job_id": "133115", "node": "alphagpu23", "world_size": 4, "gpu": "NVIDIA H200"}
+{"event": "end", "unix_time": 1792294778.8292243, "step": 348420, "train_hours": 165.49214161012114, "job_id": "133115"}

frac_0.20/checkpoints/index.csv

@@ -0,0 +1,180 @@
+step,kind,file,unix_time,save_s,train_hours,gpu_hours,samples,tokens,epochs,job_id
+0,grid,step_0000000.ckpt,1790576415.726,4.122,0.0020640935819958234,0.008256374327983294,0,0,0.0,105185
+100,grid,step_0000100.ckpt,1790576631.851,8.698,0.062098724651667804,0.24839489860667122,51200,52428800,0.02932101239958126,105185
+200,grid,step_0000200.ckpt,1790576822.417,8.652,0.11503371459729451,0.46013485838917806,102400,104857600,0.05864202479916252,105185
+400,grid,step_0000400.ckpt,1790577193.504,10.481,0.21811350301360816,0.8724540120544326,204800,209715200,0.11728404959832504,105185
+700,grid,step_0000700.ckpt,1790577744.975,9.478,0.37129988182142065,1.4851995272856826,358400,367001600,0.20524708679706882,105185
+1000,grid,step_0001000.ckpt,1790578291.648,8.396,0.523153453190009,2.092613812760036,512000,524288000,0.2932101239958126,105185
+1500,grid,step_0001500.ckpt,1790579204.064,9.317,0.7766023904328337,3.106409561731335,768000,786432000,0.4398151859937189,105185
+2500,grid,step_0002500.ckpt,1790581007.336,8.022,1.2775114035875432,5.110045614350173,1280000,1310720000,0.7330253099895315,105185
+2500,last,last.ckpt,1790581016.032,8.691,1.2799267804089727,5.119707121635891,1280000,1310720000,0.7330253099895315,105185
+4000,grid,step_0004000.ckpt,1790583727.840,9.002,2.033206901693613,8.132827606774452,2048000,2097152000,1.1728404959832504,105185
+5000,last,last.ckpt,1790585545.813,9.405,2.5381994391287055,10.152797756514822,2560000,2621440000,1.466050619979063,105185
+6000,grid,step_0006000.ckpt,1790587489.879,8.218,3.0782177061200815,12.312870824480326,3072000,3145728000,1.7592607439748755,105185
+7500,last,last.ckpt,1790590199.952,9.156,3.831015597240249,15.324062388960996,3840000,3932160000,2.1990759299685942,105185
+10000,grid,step_0010000.ckpt,1790594718.884,8.254,5.086274544493063,20.34509817797225,5120000,5242880000,2.932101239958126,105185
+10000,last,last.ckpt,1790594727.184,8.296,5.08858018270228,20.35432073080912,5120000,5242880000,2.932101239958126,105185
+12500,last,last.ckpt,1790599388.822,9.594,6.383479571498982,25.533918285995927,6400000,6553600000,3.6651265499476575,105185
+15000,grid,step_0015000.ckpt,1790603914.350,9.362,7.640570838207819,30.562283352831276,7680000,7864320000,4.3981518599371885,105185
+15000,last,last.ckpt,1790603922.723,8.368,7.642896573223665,30.57158629289466,7680000,7864320000,4.3981518599371885,105185
+17500,last,last.ckpt,1790608580.004,8.788,8.93658586143503,35.74634344574012,8960000,9175040000,5.131177169926721,105185
+20000,grid,step_0020000.ckpt,1790613119.896,9.003,10.197666863155318,40.79066745262127,10240000,10485760000,5.864202479916252,105185
+20000,last,last.ckpt,1790613129.123,9.224,10.200229994749211,40.800919978996845,10240000,10485760000,5.864202479916252,105185
+22500,last,last.ckpt,1790617775.413,8.911,11.490866015338106,45.963464061352425,11520000,11796480000,6.597227789905784,105185
+25000,grid,step_0025000.ckpt,1790622313.048,8.235,12.751320285924788,51.00528114369915,12800000,13107200000,7.330253099895315,105185
+25000,last,last.ckpt,1790622322.130,9.078,12.75384306801082,51.01537227204328,12800000,13107200000,7.330253099895315,105185
+27500,last,last.ckpt,1790626996.718,8.797,14.052339577646409,56.209358310585635,14080000,14417920000,8.063278409884846,105185
+30000,last,last.ckpt,1790631549.932,8.646,15.31712135018923,61.26848540075692,15360000,15728640000,8.796303719874377,105185
+32500,last,last.ckpt,1790636226.995,9.518,16.61630541673525,66.465221666941,16640000,17039360000,9.52932902986391,105185
+35000,last,last.ckpt,1790640757.173,9.951,17.87468810691537,71.49875242766149,17920000,18350080000,10.262354339853442,105185
+37500,last,last.ckpt,1790645380.009,8.975,19.158809484794425,76.6352379391777,19200000,19660800000,10.995379649842972,105185
+40000,last,last.ckpt,1790649865.606,8.799,20.40480853010952,81.61923412043808,20480000,20971520000,11.728404959832504,105185
+42500,last,last.ckpt,1790654482.039,9.940,21.687151006842548,86.74860402737019,21760000,22282240000,12.461430269822035,105185
+45000,last,last.ckpt,1790658957.056,10.203,22.93021131040671,91.72084524162683,23040000,23592960000,13.194455579811567,105185
+46360,last,last.ckpt,1790661522.077,8.557,23.64271716567196,94.57086866268784,23736320,24305991680,13.593221348445871,105185
+47500,last,last.ckpt,1790716664.231,11.471,24.20737565111162,96.82950260444647,24320000,24903680000,13.927480889801098,105426
+50000,grid,step_0050000.ckpt,1790721097.142,10.889,25.43873988558021,101.75495954232083,25600000,26214400000,14.66050619979063,105426
+50000,last,last.ckpt,1790721108.381,11.235,25.441861737343068,101.76744694937227,25600000,26214400000,14.66050619979063,105426
+52500,last,last.ckpt,1790725661.205,11.540,26.706535157342184,106.82614062936874,26880000,27525120000,15.39353150978016,105426
+55000,last,last.ckpt,1790730099.977,12.728,27.939527360892907,111.75810944357163,28160000,28835840000,16.126556819769693,105426
+57500,last,last.ckpt,1790734712.334,12.140,29.220737717230804,116.88295086892322,29440000,30146560000,16.859582129759225,105426
+60000,last,last.ckpt,1790739138.234,11.098,30.45015430648242,121.80061722592968,30720000,31457280000,17.592607439748754,105426
+62500,last,last.ckpt,1790743696.014,11.463,31.7162044543209,126.8648178172836,32000000,32768000000,18.325632749738286,105426
+65000,last,last.ckpt,1790748125.751,14.853,32.946686907526775,131.7867476301071,33280000,34078720000,19.05865805972782,105426
+67500,last,last.ckpt,1790752738.216,15.001,34.227927239736694,136.91170895894678,34560000,35389440000,19.79168336971735,105426
+70000,last,last.ckpt,1790757224.645,15.802,35.47415730920362,141.89662923681448,35840000,36700160000,20.524708679706883,105426
+72500,last,last.ckpt,1790761840.981,17.615,36.75647300179401,147.02589200717603,37120000,38010880000,21.257733989696412,105426
+75000,grid,step_0075000.ckpt,1790766328.842,16.333,38.00310096070791,152.01240384283165,38400000,39321600000,21.990759299685944,105426
+75000,last,last.ckpt,1790766343.888,15.042,38.00728048506219,152.02912194024876,38400000,39321600000,21.990759299685944,105426
+77500,last,last.ckpt,1790770963.369,17.905,39.29046973002997,157.16187892011988,39680000,40632320000,22.723784609675477,105426
+80000,last,last.ckpt,1790775449.845,15.328,40.5367129742975,162.14685189719,40960000,41943040000,23.45680991966501,105426
+82500,last,last.ckpt,1790780063.107,15.341,41.81817458443706,167.27269833774824,42240000,43253760000,24.189835229654538,105426
+85000,last,last.ckpt,1790784547.382,15.565,43.06380653963114,172.25522615852455,43520000,44564480000,24.92286053964407,105426
+87500,last,last.ckpt,1790789168.435,17.718,44.34743246593305,177.3897298637322,44800000,45875200000,25.655885849633602,105426
+90000,last,last.ckpt,1790793656.427,15.194,45.594096927135766,182.37638770854306,46080000,47185920000,26.388911159623134,105426
+92500,last,last.ckpt,1790798271.829,15.244,46.87615285331849,187.50461141327395,47360000,48496640000,27.121936469612663,105426
+93320,last,last.ckpt,1790799754.909,16.970,47.288119637952185,189.15247855180874,47779840,48926556160,27.36236877128923,105426
+95000,last,last.ckpt,1790940530.610,11.316,48.12126676677882,192.48506706711527,48640000,49807360000,27.854961779602196,109361
+97500,last,last.ckpt,1790945101.372,11.947,49.39092268289455,197.5636907315782,49920000,51118080000,28.587987089591728,109361
+100000,grid,step_0100000.ckpt,1790949542.781,10.333,50.62464745192892,202.4985898077157,51200000,52428800000,29.32101239958126,109361
+100000,last,last.ckpt,1790949555.289,12.506,50.62812208519395,202.5124883407758,51200000,52428800000,29.32101239958126,109361
+102500,last,last.ckpt,1790954125.199,12.531,51.89754133066809,207.59016532267236,52480000,53739520000,30.054037709570792,109361
+105000,last,last.ckpt,1790958554.160,12.825,53.127808456125464,212.51123382450186,53760000,55050240000,30.78706301956032,109361
+107500,last,last.ckpt,1790963123.955,10.722,54.39719593375326,217.58878373501304,55040000,56360960000,31.520088329549854,109361
+110000,last,last.ckpt,1790967551.581,12.430,55.62709189408329,222.50836757633317,56320000,57671680000,32.253113639539386,109361
+112500,last,last.ckpt,1790972122.762,10.890,56.89686444904345,227.5874577961738,57600000,58982400000,32.986138949528915,109361
+115000,last,last.ckpt,1790976601.774,10.913,58.141034358034894,232.56413743213957,58880000,60293120000,33.71916425951845,109361
+117500,last,last.ckpt,1790981204.908,11.501,59.41968274528497,237.6787309811399,60160000,61603840000,34.45218956950798,109361
+120000,last,last.ckpt,1790985678.590,10.912,60.66237213369614,242.64948853478455,61440000,62914560000,35.18521487949751,109361
+122500,last,last.ckpt,1790990278.337,10.886,61.94007962542327,247.7603185016931,62720000,64225280000,35.918240189487044,109361
+125000,grid,step_0125000.ckpt,1790994754.773,11.145,63.18353420854779,252.73413683419116,64000000,65536000000,36.65126549947657,109361
+125000,last,last.ckpt,1790994766.463,11.687,63.18678135458044,252.74712541832176,64000000,65536000000,36.65126549947657,109361
+127500,last,last.ckpt,1790999375.548,11.413,64.46708291998624,257.86833167994496,65280000,66846720000,37.38429080946611,109361
+130000,last,last.ckpt,1791003856.064,11.985,65.71167051268586,262.8466820507434,66560000,68157440000,38.11731611945564,109361
+132500,last,last.ckpt,1791008459.686,11.504,66.99045443942279,267.96181775769116,67840000,69468160000,38.850341429445166,109361
+135000,last,last.ckpt,1791012938.530,11.056,68.23457766648123,272.9383106659249,69120000,70778880000,39.5833667394347,109361
+137500,last,last.ckpt,1791017536.611,12.587,69.51182259625082,278.0472903850033,70400000,72089600000,40.31639204942423,109361
+140000,last,last.ckpt,1791021994.591,11.871,70.75015037465607,283.0006014986243,71680000,73400320000,41.049417359413766,109361
+140260,last,last.ckpt,1791022596.734,11.272,70.91741215334378,283.6696486133751,71813120,73536634880,41.125651991652674,109361
+142500,last,last.ckpt,1791412318.827,16.411,72.05000042128144,288.20000168512576,72960000,74711040000,41.782442669403295,122057
+145000,last,last.ckpt,1791416862.992,17.187,73.31226855270098,293.24907421080394,74240000,76021760000,42.515467979392824,122057
+147500,last,last.ckpt,1791421532.685,16.963,74.60940558510573,298.43762234042293,75520000,77332480000,43.24849328938236,122057
+150000,grid,step_0150000.ckpt,1791426050.569,14.345,75.86437333506248,303.4574933402499,76800000,78643200000,43.98151859937189,122057
+150000,last,last.ckpt,1791426066.603,16.029,75.8688271672527,303.4753086690108,76800000,78643200000,43.98151859937189,122057
+152500,last,last.ckpt,1791430713.733,14.704,77.15969669458637,308.6387867783455,78080000,79953920000,44.71454390936142,122057
+155000,last,last.ckpt,1791435216.441,16.196,78.41044876457657,313.6417950583063,79360000,81264640000,45.44756921935095,122057
+157500,last,last.ckpt,1791439847.513,15.283,79.69685761626732,318.7874304650693,80640000,82575360000,46.18059452934048,122057
+160000,last,last.ckpt,1791444348.834,15.013,80.94722481318776,323.78889925275104,81920000,83886080000,46.91361983933002,122057
+162500,last,last.ckpt,1791448978.401,15.135,82.23321556862884,328.93286227451534,83200000,85196800000,47.646645149319546,122057
+165000,last,last.ckpt,1791453490.957,15.268,83.4867032666802,333.9468130667208,84480000,86507520000,48.379670459309075,122057
+167500,last,last.ckpt,1791458122.933,15.601,84.77336341854314,339.09345367417257,85760000,87818240000,49.11269576929861,122057
+170000,last,last.ckpt,1791462627.735,15.135,86.02469721647523,344.09878886590093,87040000,89128960000,49.84572107928814,122057
+172500,last,last.ckpt,1791467260.343,15.719,87.31153260759078,349.2461304303631,88320000,90439680000,50.578746389277676,122057
+175000,grid,step_0175000.ckpt,1791471765.473,14.476,88.56295765320331,354.25183061281325,89600000,91750400000,51.311771699267204,122057
+175000,last,last.ckpt,1791471781.628,16.151,88.56744515371354,354.26978061485414,89600000,91750400000,51.311771699267204,122057
+177500,last,last.ckpt,1791476426.613,14.922,89.85771887865512,359.4308755146205,90880000,93061120000,52.04479700925673,122057
+180000,last,last.ckpt,1791480944.119,15.370,91.11258173763753,364.4503269505501,92160000,94371840000,52.77782231924627,122057
+182500,last,last.ckpt,1791485585.143,15.058,92.4017549067283,369.6070196269132,93440000,95682560000,53.5108476292358,122057
+185000,last,last.ckpt,1791490089.947,15.045,93.65308945390862,374.6123578156345,94720000,96993280000,54.24387293922533,122057
+186740,last,last.ckpt,1791493358.182,14.986,94.56093255281448,378.2437302112579,95610880,97905541120,54.754058554978045,122057
+187500,last,last.ckpt,1791824613.770,9.368,94.93910078060787,379.7564031224315,96000000,98304000000,54.97689824921486,124997
+190000,last,last.ckpt,1791829086.304,9.215,96.18147139223267,384.72588556893066,97280000,99614720000,55.70992355920439,124997
+192500,last,last.ckpt,1791833677.283,8.606,97.45674325710861,389.82697302843445,98560000,100925440000,56.44294886919393,124997
+195000,last,last.ckpt,1791838145.116,10.037,98.69780783528876,394.791231341155,99840000,102236160000,57.175974179183456,124997
+197500,last,last.ckpt,1791842734.781,9.567,99.97271497000811,399.89085988003245,101120000,103546880000,57.908999489172984,124997
+200000,grid,step_0200000.ckpt,1791847202.140,9.333,101.2136478264679,404.8545913058716,102400000,104857600000,58.64202479916252,124997
+200000,last,last.ckpt,1791847211.226,9.083,101.21617176470579,404.86468705882316,102400000,104857600000,58.64202479916252,124997
+202500,last,last.ckpt,1791851780.079,8.723,102.48529759345654,409.94119037382615,103680000,106168320000,59.37505010915205,124997
+205000,last,last.ckpt,1791856222.288,9.428,103.71924464824377,414.8769785929751,104960000,107479040000,60.108075419141585,124997
+207500,last,last.ckpt,1791860784.758,8.406,104.98659733766716,419.94638935066865,106240000,108789760000,60.841100729131114,124997
+210000,last,last.ckpt,1791865239.456,9.913,106.22401345378279,424.89605381513115,107520000,110100480000,61.57412603912064,124997
+212500,last,last.ckpt,1791869803.591,9.451,107.4918286988535,429.967314795414,108800000,111411200000,62.30715134911018,124997
+215000,last,last.ckpt,1791874256.933,9.321,108.72886830560346,434.9154732224138,110080000,112721920000,63.04017665909971,124997
+217500,last,last.ckpt,1791878833.665,9.481,110.0001826529774,440.0007306119096,111360000,114032640000,63.773201969089236,124997
+220000,last,last.ckpt,1791883302.309,10.025,111.24147260266719,444.96589041066875,112640000,115343360000,64.50622727907877,124997
+222500,last,last.ckpt,1791887888.337,9.279,112.51536921226705,450.0614768490682,113920000,116654080000,65.2392525890683,124997
+225000,grid,step_0225000.ckpt,1791892326.494,9.183,113.74819078003082,454.9927631201233,115200000,117964800000,65.97227789905783,124997
+225000,last,last.ckpt,1791892335.704,9.207,113.75074894026466,455.0029957610586,115200000,117964800000,65.97227789905783,124997
+227500,last,last.ckpt,1791896902.630,9.126,115.01933964341853,460.07735857367413,116480000,119275520000,66.70530320904736,124997
+230000,last,last.ckpt,1791901344.565,9.711,116.25321032079869,465.01284128319475,117760000,120586240000,67.4383285190369,124997
+232500,last,last.ckpt,1791905912.205,9.805,117.52199920657246,470.08799682628984,119040000,121896960000,68.17135382902643,124997
+233880,last,last.ckpt,1791908367.211,10.375,118.20394540490194,472.8157816196078,119746560,122620477440,68.57598380014065,124997
+235000,last,last.ckpt,1792192382.674,7.977,118.69318621047255,474.7727448418902,120320000,123207680000,68.90437913901596,132220
+237500,last,last.ckpt,1792196419.173,8.459,119.81443589563636,479.25774358254546,121600000,124518400000,69.63740444900549,132220
+240000,last,last.ckpt,1792200340.568,7.936,120.90371232771417,483.6148493108567,122880000,125829120000,70.37042975899502,132220
+242500,last,last.ckpt,1792204377.059,8.685,122.0249596890238,488.0998387560952,124160000,127139840000,71.10345506898456,132220
+245000,last,last.ckpt,1792208298.530,7.965,123.11425713457633,492.45702853830534,125440000,128450560000,71.83648037897409,132220
+247500,last,last.ckpt,1792212334.565,8.737,124.23537795727141,496.94151182908564,126720000,129761280000,72.56950568896362,132220
+250000,grid,step_0250000.ckpt,1792216253.414,7.809,125.32394735641053,501.2957894256421,128000000,131072000000,73.30253099895315,132220
+250000,last,last.ckpt,1792216261.429,8.012,125.32617375351488,501.3046950140595,128000000,131072000000,73.30253099895315,132220
+252500,last,last.ckpt,1792220295.500,8.244,126.44674902377857,505.78699609511426,129280000,132382720000,74.03555630894267,132220
+255000,last,last.ckpt,1792224213.988,7.976,127.53521775834055,510.1408710333622,130560000,133693440000,74.76858161893222,132220
+257500,last,last.ckpt,1792228250.472,8.430,128.65646339290464,514.6258535716186,131840000,135004160000,75.50160692892175,132220
+260000,last,last.ckpt,1792232170.515,8.123,129.7453640785503,518.9814563142012,133120000,136314880000,76.23463223891127,132220
+262500,last,last.ckpt,1792236205.793,8.420,130.8662746950715,523.465098780286,134400000,137625600000,76.9676575489008,132220
+265000,last,last.ckpt,1792240122.795,8.117,131.954330837367,527.817323349468,135680000,138936320000,77.70068285889033,132220
+267500,last,last.ckpt,1792244156.853,8.452,133.07490265231527,532.2996106092611,136960000,140247040000,78.43370816887987,132220
+270000,last,last.ckpt,1792248073.737,8.080,134.16292589696693,536.6517035878677,138240000,141557760000,79.1667334788694,132220
+272500,last,last.ckpt,1792252107.442,8.322,135.28339942550596,541.1335977020238,139520000,142868480000,79.89975878885893,132220
+275000,grid,step_0275000.ckpt,1792256024.893,7.618,136.37158021671232,545.4863208668493,140800000,144179200000,80.63278409884846,132220
+275000,last,last.ckpt,1792256032.832,7.936,136.37378551258965,545.4951420503586,140800000,144179200000,80.63278409884846,132220
+277500,last,last.ckpt,1792260065.374,8.269,137.4939360290331,549.9757441161324,142080000,145489920000,81.36580940883799,132220
+280000,last,last.ckpt,1792263981.887,7.819,138.581856334955,554.32742533982,143360000,146800640000,82.09883471882753,132220
+282500,last,last.ckpt,1792268016.945,8.114,139.70270596690216,558.8108238676086,144640000,148111360000,82.83186002881706,132220
+285000,last,last.ckpt,1792271934.119,7.993,140.79080968002063,563.1632387200825,145920000,149422080000,83.56488533880659,132220
+287320,last,last.ckpt,1792275684.374,8.399,141.83254725383543,567.3301890153417,147107840,150638428160,84.24513282647688,132220
+287500,last,last.ckpt,1792900781.181,12.635,141.92611332996646,567.7044533198658,147200000,150732800000,84.29791064879612,138045
+290000,last,last.ckpt,1792905231.166,12.302,143.1622201941364,572.6488807765456,148480000,152043520000,85.03093595878565,138045
+292500,last,last.ckpt,1792909805.920,12.218,144.43298519483872,577.7319407793549,149760000,153354240000,85.76396126877518,138045
+295000,last,last.ckpt,1792914263.332,12.759,145.6711551278302,582.6846205113208,151040000,154664960000,86.49698657876472,138045
+297500,last,last.ckpt,1792918830.456,12.358,146.93980066241335,587.7592026496534,152320000,155975680000,87.23001188875425,138045
+300000,grid,step_0300000.ckpt,1792923271.012,11.766,148.1732884872948,592.6931539491792,153600000,157286400000,87.96303719874378,138045
+300000,last,last.ckpt,1792923282.479,11.463,148.17647381360385,592.7058952544154,153600000,157286400000,87.96303719874378,138045
+302500,last,last.ckpt,1792927850.758,13.251,149.44544008509567,597.7817603403827,154880000,158597120000,88.6960625087333,138045
+305000,last,last.ckpt,1792932284.591,11.771,150.67706043835884,602.7082417534353,156160000,159907840000,89.42908781872283,138045
+307500,last,last.ckpt,1792936859.206,12.502,151.9477869153561,607.7911476614244,157440000,161218560000,90.16211312871238,138045
+310000,last,last.ckpt,1792941294.219,11.654,153.1797349607665,612.718939843066,158720000,162529280000,90.8951384387019,138045
+312500,last,last.ckpt,1792945861.036,12.281,154.4482951919455,617.793180767782,160000000,163840000000,91.62816374869143,138045
+315000,last,last.ckpt,1792950297.750,11.757,155.68071565696297,622.7228626278519,161280000,165150720000,92.36118905868096,138045
+317500,last,last.ckpt,1792954859.441,12.547,156.94785225186237,627.7914090074495,162560000,166461440000,93.09421436867049,138045
+320000,last,last.ckpt,1792959296.340,12.466,158.1803239626194,632.7212958504776,163840000,167772160000,93.82723967866004,138045
+322500,last,last.ckpt,1792963860.238,11.677,159.44807341655664,637.7922936662266,165120000,169082880000,94.56026498864956,138045
+325000,grid,step_0325000.ckpt,1792968305.409,10.286,160.68284316074156,642.7313726429662,166400000,170393600000,95.29329029863909,138045
+325000,last,last.ckpt,1792968316.834,11.421,160.6860168162956,642.7440672651824,166400000,170393600000,95.29329029863909,138045
+327500,last,last.ckpt,1792972874.346,13.322,161.95199229980457,647.8079691992183,167680000,171704320000,96.02631560862862,138045
+330000,last,last.ckpt,1792977319.274,12.060,163.1866946926506,652.7467787706024,168960000,173015040000,96.75934091861815,138045
+332500,last,last.ckpt,1792981881.417,13.051,164.4539565708007,657.8158262832028,170240000,174325760000,97.4923662286077,138045
+335000,last,last.ckpt,1792986321.962,12.404,165.68744122868972,662.7497649147589,171520000,175636480000,98.22539153859722,138045
+337500,last,last.ckpt,1792990892.347,12.592,166.95699260697234,667.8279704278893,172800000,176947200000,98.95841684858675,138045
+340000,last,last.ckpt,1792995328.936,12.303,168.18937847442564,672.7575138977026,174080000,178257920000,99.69144215857628,138045
+342500,last,last.ckpt,1792999897.829,12.098,169.45851533000356,677.8340613200143,175360000,179568640000,100.42446746856581,138045
+345000,last,last.ckpt,1793004332.722,11.036,170.6904301654144,682.7617206616576,176640000,180879360000,101.15749277855535,138045
+347500,last,last.ckpt,1793008894.351,12.263,171.9575494823822,687.8301979295288,177920000,182190080000,101.89051808854488,138045
+350000,grid,step_0350000.ckpt,1793013326.868,12.199,173.18880412094242,692.7552164837697,179200000,183500800000,102.62354339853441,138045
+350000,last,last.ckpt,1793013338.675,11.803,173.19208369751357,692.7683347900543,179200000,183500800000,102.62354339853441,138045
+352500,last,last.ckpt,1793017897.359,12.555,174.45838479668316,697.8335391867327,180480000,184811520000,103.35656870852394,138045
+355000,last,last.ckpt,1793022328.947,11.987,175.6893815922173,702.7575263688692,181760000,186122240000,104.08959401851347,138045
+357500,last,last.ckpt,1793026894.958,12.436,176.9577179750531,707.8308719002124,183040000,187432960000,104.822619328503,138045
+360000,last,last.ckpt,1793031328.489,12.104,178.1892544695611,712.7570178782444,184320000,188743680000,105.55564463849254,138045
+362500,last,last.ckpt,1793035879.599,13.802,179.45345168712961,717.8138067485185,185600000,190054400000,106.28866994848207,138045
+365000,last,last.ckpt,1793040318.231,11.273,180.68640498232716,722.7456199293086,186880000,191365120000,107.0216952584716,138045

frac_0.20/code_sha

@@ -0,0 +1 @@
+7c9e7d04d4a4

frac_0.20/config_resolved.yaml

@@ -0,0 +1,204 @@
+mode: train
+seed: 1
+loader:
+  global_batch_size: 512
+  eval_global_batch_size: 64
+  batch_size: 16
+  eval_batch_size: 16
+  num_workers: 4
+  pin_memory: true
+  sampler: step_indexed
+sampling:
+  predictor: ancestral
+  steps: 1000
+  noise_removal: ancestral
+  use_float64: true
+  p_nucleus: 1.0
+  num_sample_batches: 0
+  num_sample_log: 2
+  semi_ar: false
+  stride_length: 1
+  num_strides: 1
+  guid_weight: null
+  psi:
+    time_profile: linear
+    high_mode: pure-posterior
+    middle_mode: pure-posterior
+    low_mode: pure-posterior
+    high_frac: 0.2
+    middle_frac: 0.6
+training:
+  ema: 0.9999
+  antithetic_sampling: true
+  importance_sampling: false
+  sampling_eps: 0.001
+  change_of_variables: false
+  loss_precision: bf16
+  finetune_path: ''
+  class_dropout_p: 0.1
+eval:
+  checkpoint_path: ''
+  disable_ema: false
+  compute_generative_perplexity: false
+  perplexity_batch_size: 8
+  compute_perplexity_on_sanity: false
+  gen_ppl_eval_model_name_or_path: gpt2-large
+  generate_samples: false
+  generated_samples_path: /mnt/home/phamd/dm_scaling_law_runs/duo_base/medium/frac_0.20/samples.json
+optim:
+  weight_decay: 0
+  lr: 0.0003
+  beta1: 0.9
+  beta2: 0.999
+  eps: 1.0e-08
+trainer:
+  _target_: lightning.Trainer
+  accelerator: cuda
+  num_nodes: 1
+  devices: 4
+  accumulate_grad_batches: 8
+  gradient_clip_val: 1.0
+  precision: bf16
+  num_sanity_val_steps: 2
+  max_steps: 1000000
+  log_every_n_steps: 100
+  limit_train_batches: 1.0
+  limit_val_batches: 1.0
+  val_check_interval: 40000
+  check_val_every_n_epoch: null
+  enable_checkpointing: false
+  enable_progress_bar: false
+wandb: null
+checkpointing:
+  save_dir: /mnt/home/phamd/dm_scaling_law_runs/duo_base/medium/frac_0.20
+  resume_from_ckpt: true
+  resume_ckpt_path: /mnt/home/phamd/dm_scaling_law_runs/duo_base/medium/frac_0.20/checkpoints/last.ckpt
+callbacks:
+  training_clock:
+    _target_: scaling_callbacks.TrainingClock
+    run_dir: /mnt/home/phamd/dm_scaling_law_runs/duo_base/medium/frac_0.20
+    global_batch_size: 512
+    seq_len: 1024
+    max_steps: 1000000
+    log_every_n_steps: 100
+    print_every_n_steps: 500
+  norm_monitor:
+    _target_: scaling_callbacks.NormMonitor
+    dirpath: /mnt/home/phamd/dm_scaling_law_runs/duo_base/medium/frac_0.20/norms
+    every_n_steps: 500
+    max_steps: 1000000
+    grid_every: 25000
+    grid_early:
+    - 100
+    - 200
+    - 400
+    - 700
+    - 1000
+    - 1500
+    - 2500
+    - 4000
+    - 6000
+    - 10000
+    - 15000
+    - 20000
+  scaling_checkpoint:
+    _target_: scaling_callbacks.ScalingCheckpoint
+    dirpath: /mnt/home/phamd/dm_scaling_law_runs/duo_base/medium/frac_0.20/checkpoints
+    max_steps: 1000000
+    grid_every: 25000
+    grid_early:
+    - 100
+    - 200
+    - 400
+    - 700
+    - 1000
+    - 1500
+    - 2500
+    - 4000
+    - 6000
+    - 10000
+    - 15000
+    - 20000
+    rolling_every: 2500
+    walltime_margin_s: 1200
+  learning_rate_monitor:
+    _target_: lightning.pytorch.callbacks.LearningRateMonitor
+    logging_interval: step
+data:
+  train: openwebtext-train
+  valid: openwebtext-valid
+  modality: text
+  tokenizer_name_or_path: gpt2
+  cache_dir: /mnt/home/phamd/dm_scaling_law_data/text_data
+  wrap: true
+  streaming: false
+  insert_train_eos: true
+  insert_valid_eos: true
+  train_fraction: 0.2
+  subset_seed: 20250927
+  subset_perm_path: /mnt/home/phamd/dm_scaling_law_data/subsets/owt_train_perm_s20250927.npy
+  subset_perm_sha256: null
+  subset_tokens_path: /mnt/home/phamd/dm_scaling_law_data/subsets/owt_train_perm_s20250927_input_ids.npy
+  order_seed: 1
+model:
+  name: medium
+  type: ddit
+  hidden_size: 1024
+  cond_dim: 128
+  length: 1024
+  n_blocks: 24
+  n_heads: 16
+  scale_by_sigma: true
+  dropout: 0.1
+  tie_word_embeddings: false
+  vocab_lookup: true
+strategy:
+  _target_: lightning.pytorch.strategies.DDPStrategy
+  find_unused_parameters: false
+noise:
+  type: log-linear
+  eps: 0.001
+lr_scheduler:
+  _target_: transformers.get_constant_schedule_with_warmup
+  num_warmup_steps: 2500
+prior:
+  type: none
+  latent_width: 0
+  latent_height: 0
+algo:
+  name: duo_base
+  backbone: dit
+  parameterization: mean
+  time_conditioning: true
+  T: 0
+  subs_masking: false
+  causal_attention: false
+  ignore_bos: false
+  loss_type: elbo
+  compile_loss: true
+scaling:
+  val_every_steps: 5000
+  rolling_every: 2500
+  reseed_every: 1
+  grid_every: 25000
+  grid_early:
+  - 100
+  - 200
+  - 400
+  - 700
+  - 1000
+  - 1500
+  - 2500
+  - 4000
+  - 6000
+  - 10000
+  - 15000
+  - 20000
+  norm_every: 500
+  val_noise_seed: 4242
+  walltime_margin_s: 1200
+scaling_wandb:
+  project: duo-scaling-medium
+  group: medium
+  name: duo_base-medium-f0.2-s1
+  id: duobase-medium-f020-s1

frac_0.20/run_info.json

@@ -0,0 +1,24 @@
+{
+  "subset": {
+    "train_fraction": 0.2,
+    "n_blocks": 1746188,
+    "n_total": 8730942,
+    "perm_path": "/mnt/home/phamd/dm_scaling_law_data/subsets/owt_train_perm_s20250927.npy",
+    "perm_sha256": "fa545515efbfdd07138abceebb2671be60b5047431da2f50fd58a8bb986d05a7",
+    "subset_seed": 20250927,
+    "steps_per_epoch": 3410
+  },
+  "model": "medium",
+  "algo": "duo_base",
+  "seed": 1,
+  "order_seed": 1,
+  "global_batch_size": 512,
+  "max_steps": 1000000,
+  "num_parameters": 424477010,
+  "batch_size": 16,
+  "accumulate_grad_batches": 8,
+  "world_size": 4,
+  "gpu": "NVIDIA H100 80GB HBM3",
+  "code_sha": "7c9e7d04d4a4",
+  "code_dir": "/mnt/home/phamd/dm_scaling_law_runs/code/7c9e7d04d4a4"
+}

frac_0.20/segments.jsonl

@@ -0,0 +1,22 @@
+{"event": "job_start", "unix_time": 1790576309, "job_id": "105185", "restart": 0, "node": "alphagpu15"}
+{"event": "start", "unix_time": 1790576408.295601, "step": 0, "train_hours": 4.906647114290131e-09, "job_id": "105185", "node": "alphagpu15", "world_size": 4, "gpu": "NVIDIA H100 80GB HBM3"}
+{"event": "end", "unix_time": 1790661522.0866528, "step": 46360, "train_hours": 23.64271980637581, "job_id": "105185"}
+{"event": "job_start", "unix_time": 1790714533, "job_id": "105426", "restart": 0, "node": "alphagpu06"}
+{"event": "start", "unix_time": 1790714622.9057856, "step": 46360, "train_hours": 23.6403409398016, "job_id": "105426", "node": "alphagpu06", "world_size": 4, "gpu": "NVIDIA H100 80GB HBM3"}
+{"event": "end", "unix_time": 1790799754.9196362, "step": 93320, "train_hours": 47.28812257772415, "job_id": "105426"}
+{"event": "job_start", "unix_time": 1790937398, "job_id": "109361", "restart": 0, "node": "alphagpu04"}
+{"event": "start", "unix_time": 1790937514.31383, "step": 93320, "train_hours": 47.28340663042126, "job_id": "109361", "node": "alphagpu04", "world_size": 4, "gpu": "NVIDIA H100 80GB HBM3"}
+{"event": "end", "unix_time": 1791022596.742848, "step": 140260, "train_hours": 70.91741470010822, "job_id": "109361"}
+{"event": "job_start", "unix_time": 1791272920, "job_id": "113983", "restart": 0, "node": "alphagpu08"}
+{"event": "start", "unix_time": 1791273001.579895, "step": 140260, "train_hours": 70.91428146595032, "job_id": "113983", "node": "alphagpu08", "world_size": 4, "gpu": "NVIDIA H100 80GB HBM3"}
+{"event": "job_start", "unix_time": 1791408146, "job_id": "122057", "restart": 0, "node": "alphagpu09"}
+{"event": "start", "unix_time": 1791408230.23849, "step": 140260, "train_hours": 70.91428146555631, "job_id": "122057", "node": "alphagpu09", "world_size": 4, "gpu": "NVIDIA H100 80GB HBM3"}
+{"event": "end", "unix_time": 1791493358.1968544, "step": 186740, "train_hours": 94.56093658404554, "job_id": "122057"}
+{"event": "job_start", "unix_time": 1791823166, "job_id": "124997", "restart": 0, "node": "alphagpu14"}
+{"event": "start", "unix_time": 1791823237.3819745, "step": 186740, "train_hours": 94.55677067323744, "job_id": "124997", "node": "alphagpu14", "world_size": 4, "gpu": "NVIDIA H100 80GB HBM3"}
+{"event": "end", "unix_time": 1791908367.2191207, "step": 233880, "train_hours": 118.20394768650307, "job_id": "124997"}
+{"event": "job_start", "unix_time": 1792190472, "job_id": "132220", "restart": 0, "node": "alphagpu20"}
+{"event": "start", "unix_time": 1792190611.0342295, "step": 233880, "train_hours": 118.20106397773864, "job_id": "132220", "node": "alphagpu20", "world_size": 4, "gpu": "NVIDIA H200"}
+{"event": "end", "unix_time": 1792275684.385146, "step": 287320, "train_hours": 141.83255035293755, "job_id": "132220"}
+{"event": "job_start", "unix_time": 1792900355, "job_id": "138045", "restart": 0, "node": "alphagpu05"}
+{"event": "start", "unix_time": 1792900435.9454155, "step": 287320, "train_hours": 141.83021445763555, "job_id": "138045", "node": "alphagpu05", "world_size": 4, "gpu": "NVIDIA H100 80GB HBM3"}

frac_0.30/code_sha

@@ -0,0 +1 @@
+7c9e7d04d4a4

frac_0.30/config_resolved.yaml

@@ -0,0 +1,204 @@
+mode: train
+seed: 1
+loader:
+  global_batch_size: 512
+  eval_global_batch_size: 64
+  batch_size: 16
+  eval_batch_size: 16
+  num_workers: 4
+  pin_memory: true
+  sampler: step_indexed
+sampling:
+  predictor: ancestral
+  steps: 1000
+  noise_removal: ancestral
+  use_float64: true
+  p_nucleus: 1.0
+  num_sample_batches: 0
+  num_sample_log: 2
+  semi_ar: false
+  stride_length: 1
+  num_strides: 1
+  guid_weight: null
+  psi:
+    time_profile: linear
+    high_mode: pure-posterior
+    middle_mode: pure-posterior
+    low_mode: pure-posterior
+    high_frac: 0.2
+    middle_frac: 0.6
+training:
+  ema: 0.9999
+  antithetic_sampling: true
+  importance_sampling: false
+  sampling_eps: 0.001
+  change_of_variables: false
+  loss_precision: bf16
+  finetune_path: ''
+  class_dropout_p: 0.1
+eval:
+  checkpoint_path: ''
+  disable_ema: false
+  compute_generative_perplexity: false
+  perplexity_batch_size: 8
+  compute_perplexity_on_sanity: false
+  gen_ppl_eval_model_name_or_path: gpt2-large
+  generate_samples: false
+  generated_samples_path: /mnt/home/phamd/dm_scaling_law_runs/duo_base/medium/frac_0.30/samples.json
+optim:
+  weight_decay: 0
+  lr: 0.0003
+  beta1: 0.9
+  beta2: 0.999
+  eps: 1.0e-08
+trainer:
+  _target_: lightning.Trainer
+  accelerator: cuda
+  num_nodes: 1
+  devices: 4
+  accumulate_grad_batches: 8
+  gradient_clip_val: 1.0
+  precision: bf16
+  num_sanity_val_steps: 2
+  max_steps: 1000000
+  log_every_n_steps: 100
+  limit_train_batches: 1.0
+  limit_val_batches: 1.0
+  val_check_interval: 40000
+  check_val_every_n_epoch: null
+  enable_checkpointing: false
+  enable_progress_bar: false
+wandb: null
+checkpointing:
+  save_dir: /mnt/home/phamd/dm_scaling_law_runs/duo_base/medium/frac_0.30
+  resume_from_ckpt: true
+  resume_ckpt_path: /mnt/home/phamd/dm_scaling_law_runs/duo_base/medium/frac_0.30/checkpoints/last.ckpt
+callbacks:
+  training_clock:
+    _target_: scaling_callbacks.TrainingClock
+    run_dir: /mnt/home/phamd/dm_scaling_law_runs/duo_base/medium/frac_0.30
+    global_batch_size: 512
+    seq_len: 1024
+    max_steps: 1000000
+    log_every_n_steps: 100
+    print_every_n_steps: 500
+  norm_monitor:
+    _target_: scaling_callbacks.NormMonitor
+    dirpath: /mnt/home/phamd/dm_scaling_law_runs/duo_base/medium/frac_0.30/norms
+    every_n_steps: 500
+    max_steps: 1000000
+    grid_every: 25000
+    grid_early:
+    - 100
+    - 200
+    - 400
+    - 700
+    - 1000
+    - 1500
+    - 2500
+    - 4000
+    - 6000
+    - 10000
+    - 15000
+    - 20000
+  scaling_checkpoint:
+    _target_: scaling_callbacks.ScalingCheckpoint
+    dirpath: /mnt/home/phamd/dm_scaling_law_runs/duo_base/medium/frac_0.30/checkpoints
+    max_steps: 1000000
+    grid_every: 25000
+    grid_early:
+    - 100
+    - 200
+    - 400
+    - 700
+    - 1000
+    - 1500
+    - 2500
+    - 4000
+    - 6000
+    - 10000
+    - 15000
+    - 20000
+    rolling_every: 2500
+    walltime_margin_s: 1200
+  learning_rate_monitor:
+    _target_: lightning.pytorch.callbacks.LearningRateMonitor
+    logging_interval: step
+data:
+  train: openwebtext-train
+  valid: openwebtext-valid
+  modality: text
+  tokenizer_name_or_path: gpt2
+  cache_dir: /mnt/home/phamd/dm_scaling_law_data/text_data
+  wrap: true
+  streaming: false
+  insert_train_eos: true
+  insert_valid_eos: true
+  train_fraction: 0.3
+  subset_seed: 20250927
+  subset_perm_path: /mnt/home/phamd/dm_scaling_law_data/subsets/owt_train_perm_s20250927.npy
+  subset_perm_sha256: null
+  subset_tokens_path: /mnt/home/phamd/dm_scaling_law_data/subsets/owt_train_perm_s20250927_input_ids.npy
+  order_seed: 1
+model:
+  name: medium
+  type: ddit
+  hidden_size: 1024
+  cond_dim: 128
+  length: 1024
+  n_blocks: 24
+  n_heads: 16
+  scale_by_sigma: true
+  dropout: 0.1
+  tie_word_embeddings: false
+  vocab_lookup: true
+strategy:
+  _target_: lightning.pytorch.strategies.DDPStrategy
+  find_unused_parameters: false
+noise:
+  type: log-linear
+  eps: 0.001
+lr_scheduler:
+  _target_: transformers.get_constant_schedule_with_warmup
+  num_warmup_steps: 2500
+prior:
+  type: none
+  latent_width: 0
+  latent_height: 0
+algo:
+  name: duo_base
+  backbone: dit
+  parameterization: mean
+  time_conditioning: true
+  T: 0
+  subs_masking: false
+  causal_attention: false
+  ignore_bos: false
+  loss_type: elbo
+  compile_loss: true
+scaling:
+  val_every_steps: 5000
+  rolling_every: 2500
+  reseed_every: 1
+  grid_every: 25000
+  grid_early:
+  - 100
+  - 200
+  - 400
+  - 700
+  - 1000
+  - 1500
+  - 2500
+  - 4000
+  - 6000
+  - 10000
+  - 15000
+  - 20000
+  norm_every: 500
+  val_noise_seed: 4242
+  walltime_margin_s: 1200
+scaling_wandb:
+  project: duo-scaling-medium
+  group: medium
+  name: duo_base-medium-f0.3-s1
+  id: duobase-medium-f030-s1

frac_0.30/run_info.json

@@ -0,0 +1,24 @@
+{
+  "subset": {
+    "train_fraction": 0.3,
+    "n_blocks": 2619283,
+    "n_total": 8730942,
+    "perm_path": "/mnt/home/phamd/dm_scaling_law_data/subsets/owt_train_perm_s20250927.npy",
+    "perm_sha256": "fa545515efbfdd07138abceebb2671be60b5047431da2f50fd58a8bb986d05a7",
+    "subset_seed": 20250927,
+    "steps_per_epoch": 5115
+  },
+  "model": "medium",
+  "algo": "duo_base",
+  "seed": 1,
+  "order_seed": 1,
+  "global_batch_size": 512,
+  "max_steps": 1000000,
+  "num_parameters": 424477010,
+  "batch_size": 16,
+  "accumulate_grad_batches": 8,
+  "world_size": 4,
+  "gpu": "NVIDIA H100 80GB HBM3",
+  "code_sha": "7c9e7d04d4a4",
+  "code_dir": "/mnt/home/phamd/dm_scaling_law_runs/code/7c9e7d04d4a4"
+}

frac_0.30/segments.jsonl

@@ -0,0 +1,20 @@
+{"event": "job_start", "unix_time": 1790589915, "job_id": "105186", "restart": 0, "node": "alphagpu16"}
+{"event": "start", "unix_time": 1790589993.0586238, "step": 0, "train_hours": 4.2990884847111176e-09, "job_id": "105186", "node": "alphagpu16", "world_size": 4, "gpu": "NVIDIA H100 80GB HBM3"}
+{"event": "job_start", "unix_time": 1790690926, "job_id": "105666", "restart": 0, "node": "alphagpu03"}
+{"event": "start", "unix_time": 1790690998.5651438, "step": 2500, "train_hours": 5.475717719247203, "job_id": "105666", "node": "alphagpu03", "world_size": 4, "gpu": "NVIDIA H100 80GB HBM3"}
+{"event": "end", "unix_time": 1790776149.8039834, "step": 49120, "train_hours": 29.128839631841725, "job_id": "105666"}
+{"event": "job_start", "unix_time": 1790936823, "job_id": "108775", "restart": 0, "node": "alphagpu14"}
+{"event": "start", "unix_time": 1790936907.5058904, "step": 49120, "train_hours": 29.124521109300357, "job_id": "108775", "node": "alphagpu14", "world_size": 4, "gpu": "NVIDIA H100 80GB HBM3"}
+{"event": "end", "unix_time": 1791022036.202756, "step": 95880, "train_hours": 52.77138142438166, "job_id": "108775"}
+{"event": "job_start", "unix_time": 1791193690, "job_id": "113958", "restart": 0, "node": "alphagpu04"}
+{"event": "start", "unix_time": 1791193814.2221694, "step": 95880, "train_hours": 52.76873726047343, "job_id": "113958", "node": "alphagpu04", "world_size": 4, "gpu": "NVIDIA H100 80GB HBM3"}
+{"event": "end", "unix_time": 1791278921.7967556, "step": 142660, "train_hours": 76.40973021585468, "job_id": "113958"}
+{"event": "job_start", "unix_time": 1791434240, "job_id": "119448", "restart": 0, "node": "alphagpu14"}
+{"event": "start", "unix_time": 1791434380.8140638, "step": 142660, "train_hours": 76.40645393424823, "job_id": "119448", "node": "alphagpu14", "world_size": 4, "gpu": "NVIDIA H100 80GB HBM3"}
+{"event": "end", "unix_time": 1791519438.2726061, "step": 189840, "train_hours": 100.03352576028297, "job_id": "119448"}
+{"event": "job_start", "unix_time": 1791846725, "job_id": "125411", "restart": 0, "node": "alphagpu24"}
+{"event": "start", "unix_time": 1791846783.1403718, "step": 189840, "train_hours": 100.0307726792003, "job_id": "125411", "node": "alphagpu24", "world_size": 4, "gpu": "NVIDIA H200"}
+{"event": "end", "unix_time": 1791931931.974152, "step": 243340, "train_hours": 123.68322651860795, "job_id": "125411"}
+{"event": "job_start", "unix_time": 1792208139, "job_id": "132772", "restart": 0, "node": "alphagpu19"}
+{"event": "start", "unix_time": 1792208209.7661514, "step": 243340, "train_hours": 123.68077117971093, "job_id": "132772", "node": "alphagpu19", "world_size": 4, "gpu": "NVIDIA H200"}
+{"event": "end", "unix_time": 1792293345.773939, "step": 296660, "train_hours": 147.32966223990752, "job_id": "132772"}

frac_0.40/code_sha

@@ -0,0 +1 @@
+7c9e7d04d4a4

frac_0.40/config_resolved.yaml

@@ -0,0 +1,204 @@
+mode: train
+seed: 1
+loader:
+  global_batch_size: 512
+  eval_global_batch_size: 64
+  batch_size: 16
+  eval_batch_size: 16
+  num_workers: 4
+  pin_memory: true
+  sampler: step_indexed
+sampling:
+  predictor: ancestral
+  steps: 1000
+  noise_removal: ancestral
+  use_float64: true
+  p_nucleus: 1.0
+  num_sample_batches: 0
+  num_sample_log: 2
+  semi_ar: false
+  stride_length: 1
+  num_strides: 1
+  guid_weight: null
+  psi:
+    time_profile: linear
+    high_mode: pure-posterior
+    middle_mode: pure-posterior
+    low_mode: pure-posterior
+    high_frac: 0.2
+    middle_frac: 0.6
+training:
+  ema: 0.9999
+  antithetic_sampling: true
+  importance_sampling: false
+  sampling_eps: 0.001
+  change_of_variables: false
+  loss_precision: bf16
+  finetune_path: ''
+  class_dropout_p: 0.1
+eval:
+  checkpoint_path: ''
+  disable_ema: false
+  compute_generative_perplexity: false
+  perplexity_batch_size: 8
+  compute_perplexity_on_sanity: false
+  gen_ppl_eval_model_name_or_path: gpt2-large
+  generate_samples: false
+  generated_samples_path: /mnt/home/phamd/dm_scaling_law_runs/duo_base/medium/frac_0.40/samples.json
+optim:
+  weight_decay: 0
+  lr: 0.0003
+  beta1: 0.9
+  beta2: 0.999
+  eps: 1.0e-08
+trainer:
+  _target_: lightning.Trainer
+  accelerator: cuda
+  num_nodes: 1
+  devices: 4
+  accumulate_grad_batches: 8
+  gradient_clip_val: 1.0
+  precision: bf16
+  num_sanity_val_steps: 2
+  max_steps: 1000000
+  log_every_n_steps: 100
+  limit_train_batches: 1.0
+  limit_val_batches: 1.0
+  val_check_interval: 40000
+  check_val_every_n_epoch: null
+  enable_checkpointing: false
+  enable_progress_bar: false
+wandb: null
+checkpointing:
+  save_dir: /mnt/home/phamd/dm_scaling_law_runs/duo_base/medium/frac_0.40
+  resume_from_ckpt: true
+  resume_ckpt_path: /mnt/home/phamd/dm_scaling_law_runs/duo_base/medium/frac_0.40/checkpoints/last.ckpt
+callbacks:
+  training_clock:
+    _target_: scaling_callbacks.TrainingClock
+    run_dir: /mnt/home/phamd/dm_scaling_law_runs/duo_base/medium/frac_0.40
+    global_batch_size: 512
+    seq_len: 1024
+    max_steps: 1000000
+    log_every_n_steps: 100
+    print_every_n_steps: 500
+  norm_monitor:
+    _target_: scaling_callbacks.NormMonitor
+    dirpath: /mnt/home/phamd/dm_scaling_law_runs/duo_base/medium/frac_0.40/norms
+    every_n_steps: 500
+    max_steps: 1000000
+    grid_every: 25000
+    grid_early:
+    - 100
+    - 200
+    - 400
+    - 700
+    - 1000
+    - 1500
+    - 2500
+    - 4000
+    - 6000
+    - 10000
+    - 15000
+    - 20000
+  scaling_checkpoint:
+    _target_: scaling_callbacks.ScalingCheckpoint
+    dirpath: /mnt/home/phamd/dm_scaling_law_runs/duo_base/medium/frac_0.40/checkpoints
+    max_steps: 1000000
+    grid_every: 25000
+    grid_early:
+    - 100
+    - 200
+    - 400
+    - 700
+    - 1000
+    - 1500
+    - 2500
+    - 4000
+    - 6000
+    - 10000
+    - 15000
+    - 20000
+    rolling_every: 2500
+    walltime_margin_s: 1200
+  learning_rate_monitor:
+    _target_: lightning.pytorch.callbacks.LearningRateMonitor
+    logging_interval: step
+data:
+  train: openwebtext-train
+  valid: openwebtext-valid
+  modality: text
+  tokenizer_name_or_path: gpt2
+  cache_dir: /mnt/home/phamd/dm_scaling_law_data/text_data
+  wrap: true
+  streaming: false
+  insert_train_eos: true
+  insert_valid_eos: true
+  train_fraction: 0.4
+  subset_seed: 20250927
+  subset_perm_path: /mnt/home/phamd/dm_scaling_law_data/subsets/owt_train_perm_s20250927.npy
+  subset_perm_sha256: null
+  subset_tokens_path: /mnt/home/phamd/dm_scaling_law_data/subsets/owt_train_perm_s20250927_input_ids.npy
+  order_seed: 1
+model:
+  name: medium
+  type: ddit
+  hidden_size: 1024
+  cond_dim: 128
+  length: 1024
+  n_blocks: 24
+  n_heads: 16
+  scale_by_sigma: true
+  dropout: 0.1
+  tie_word_embeddings: false
+  vocab_lookup: true
+strategy:
+  _target_: lightning.pytorch.strategies.DDPStrategy
+  find_unused_parameters: false
+noise:
+  type: log-linear
+  eps: 0.001
+lr_scheduler:
+  _target_: transformers.get_constant_schedule_with_warmup
+  num_warmup_steps: 2500
+prior:
+  type: none
+  latent_width: 0
+  latent_height: 0
+algo:
+  name: duo_base
+  backbone: dit
+  parameterization: mean
+  time_conditioning: true
+  T: 0
+  subs_masking: false
+  causal_attention: false
+  ignore_bos: false
+  loss_type: elbo
+  compile_loss: true
+scaling:
+  val_every_steps: 5000
+  rolling_every: 2500
+  reseed_every: 1
+  grid_every: 25000
+  grid_early:
+  - 100
+  - 200
+  - 400
+  - 700
+  - 1000
+  - 1500
+  - 2500
+  - 4000
+  - 6000
+  - 10000
+  - 15000
+  - 20000
+  norm_every: 500
+  val_noise_seed: 4242
+  walltime_margin_s: 1200
+scaling_wandb:
+  project: duo-scaling-medium
+  group: medium
+  name: duo_base-medium-f0.4-s1
+  id: duobase-medium-f040-s1

frac_0.40/run_info.json

@@ -0,0 +1,24 @@
+{
+  "subset": {
+    "train_fraction": 0.4,
+    "n_blocks": 3492377,
+    "n_total": 8730942,
+    "perm_path": "/mnt/home/phamd/dm_scaling_law_data/subsets/owt_train_perm_s20250927.npy",
+    "perm_sha256": "fa545515efbfdd07138abceebb2671be60b5047431da2f50fd58a8bb986d05a7",
+    "subset_seed": 20250927,
+    "steps_per_epoch": 6821
+  },
+  "model": "medium",
+  "algo": "duo_base",
+  "seed": 1,
+  "order_seed": 1,
+  "global_batch_size": 512,
+  "max_steps": 1000000,
+  "num_parameters": 424477010,
+  "batch_size": 16,
+  "accumulate_grad_batches": 8,
+  "world_size": 4,
+  "gpu": "NVIDIA H100 80GB HBM3",
+  "code_sha": "7c9e7d04d4a4",
+  "code_dir": "/mnt/home/phamd/dm_scaling_law_runs/code/7c9e7d04d4a4"
+}

frac_0.40/segments.jsonl

@@ -0,0 +1,21 @@
+{"event": "job_start", "unix_time": 1790605657, "job_id": "105187", "restart": 0, "node": "alphagpu03"}
+{"event": "start", "unix_time": 1790605793.1908443, "step": 0, "train_hours": 6.047129217121336e-09, "job_id": "105187", "node": "alphagpu03", "world_size": 4, "gpu": "NVIDIA H100 80GB HBM3"}
+{"event": "end", "unix_time": 1790690882.2043018, "step": 46280, "train_hours": 23.635837119106306, "job_id": "105187"}
+{"event": "job_start", "unix_time": 1790724752, "job_id": "106039", "restart": 0, "node": "alphagpu13"}
+{"event": "start", "unix_time": 1790724830.531786, "step": 46280, "train_hours": 23.632652936747505, "job_id": "106039", "node": "alphagpu13", "world_size": 4, "gpu": "NVIDIA H100 80GB HBM3"}
+{"event": "end", "unix_time": 1790809951.5444021, "step": 94680, "train_hours": 47.2773786748374, "job_id": "106039"}
+{"event": "job_start", "unix_time": 1790942698, "job_id": "109752", "restart": 0, "node": "alphagpu17"}
+{"event": "start", "unix_time": 1790942764.584101, "step": 94680, "train_hours": 47.27523163666845, "job_id": "109752", "node": "alphagpu17", "world_size": 4, "gpu": "NVIDIA H100 80GB HBM3"}
+{"event": "end", "unix_time": 1791027910.0974758, "step": 143520, "train_hours": 70.92676313813413, "job_id": "109752"}
+{"event": "job_start", "unix_time": 1791275833, "job_id": "114080", "restart": 0, "node": "alphagpu10"}
+{"event": "start", "unix_time": 1791275962.0114703, "step": 143520, "train_hours": 70.92459115599644, "job_id": "114080", "node": "alphagpu10", "world_size": 4, "gpu": "NVIDIA H100 80GB HBM3"}
+{"event": "end", "unix_time": 1791361057.3738732, "step": 189920, "train_hours": 94.56219185268014, "job_id": "114080"}
+{"event": "job_start", "unix_time": 1791689484, "job_id": "122165", "restart": 0, "node": "alphagpu05"}
+{"event": "start", "unix_time": 1791689581.1115282, "step": 189920, "train_hours": 94.55926391544675, "job_id": "122165", "node": "alphagpu05", "world_size": 4, "gpu": "NVIDIA H100 80GB HBM3"}
+{"event": "end", "unix_time": 1791774696.7484148, "step": 237440, "train_hours": 118.20249639397927, "job_id": "122165"}
+{"event": "job_start", "unix_time": 1792083618, "job_id": "130306", "restart": 0, "node": "alphagpu23"}
+{"event": "start", "unix_time": 1792083753.5711832, "step": 237440, "train_hours": 118.19920452999675, "job_id": "130306", "node": "alphagpu23", "world_size": 4, "gpu": "NVIDIA H200"}
+{"event": "end", "unix_time": 1792168835.0193477, "step": 290880, "train_hours": 141.8329401417563, "job_id": "130306"}
+{"event": "job_start", "unix_time": 1792262161, "job_id": "136687", "restart": 0, "node": "alphagpu07"}
+{"event": "start", "unix_time": 1792262271.1553924, "step": 290880, "train_hours": 141.83055724634374, "job_id": "136687", "node": "alphagpu07", "world_size": 4, "gpu": "NVIDIA H100 80GB HBM3"}
+{"event": "end", "unix_time": 1792347368.5504572, "step": 338420, "train_hours": 165.46872256324988, "job_id": "136687"}

frac_0.50/code_sha

@@ -0,0 +1 @@
+7c9e7d04d4a4

frac_0.50/config_resolved.yaml

@@ -0,0 +1,204 @@
+mode: train
+seed: 1
+loader:
+  global_batch_size: 512
+  eval_global_batch_size: 64
+  batch_size: 16
+  eval_batch_size: 16
+  num_workers: 4
+  pin_memory: true
+  sampler: step_indexed
+sampling:
+  predictor: ancestral
+  steps: 1000
+  noise_removal: ancestral
+  use_float64: true
+  p_nucleus: 1.0
+  num_sample_batches: 0
+  num_sample_log: 2
+  semi_ar: false
+  stride_length: 1
+  num_strides: 1
+  guid_weight: null
+  psi:
+    time_profile: linear
+    high_mode: pure-posterior
+    middle_mode: pure-posterior
+    low_mode: pure-posterior
+    high_frac: 0.2
+    middle_frac: 0.6
+training:
+  ema: 0.9999
+  antithetic_sampling: true
+  importance_sampling: false
+  sampling_eps: 0.001
+  change_of_variables: false
+  loss_precision: bf16
+  finetune_path: ''
+  class_dropout_p: 0.1
+eval:
+  checkpoint_path: ''
+  disable_ema: false
+  compute_generative_perplexity: false
+  perplexity_batch_size: 8
+  compute_perplexity_on_sanity: false
+  gen_ppl_eval_model_name_or_path: gpt2-large
+  generate_samples: false
+  generated_samples_path: /mnt/home/phamd/dm_scaling_law_runs/duo_base/medium/frac_0.50/samples.json
+optim:
+  weight_decay: 0
+  lr: 0.0003
+  beta1: 0.9
+  beta2: 0.999
+  eps: 1.0e-08
+trainer:
+  _target_: lightning.Trainer
+  accelerator: cuda
+  num_nodes: 1
+  devices: 4
+  accumulate_grad_batches: 8
+  gradient_clip_val: 1.0
+  precision: bf16
+  num_sanity_val_steps: 2
+  max_steps: 1000000
+  log_every_n_steps: 100
+  limit_train_batches: 1.0
+  limit_val_batches: 1.0
+  val_check_interval: 40000
+  check_val_every_n_epoch: null
+  enable_checkpointing: false
+  enable_progress_bar: false
+wandb: null
+checkpointing:
+  save_dir: /mnt/home/phamd/dm_scaling_law_runs/duo_base/medium/frac_0.50
+  resume_from_ckpt: true
+  resume_ckpt_path: /mnt/home/phamd/dm_scaling_law_runs/duo_base/medium/frac_0.50/checkpoints/last.ckpt
+callbacks:
+  training_clock:
+    _target_: scaling_callbacks.TrainingClock
+    run_dir: /mnt/home/phamd/dm_scaling_law_runs/duo_base/medium/frac_0.50
+    global_batch_size: 512
+    seq_len: 1024
+    max_steps: 1000000
+    log_every_n_steps: 100
+    print_every_n_steps: 500
+  norm_monitor:
+    _target_: scaling_callbacks.NormMonitor
+    dirpath: /mnt/home/phamd/dm_scaling_law_runs/duo_base/medium/frac_0.50/norms
+    every_n_steps: 500
+    max_steps: 1000000
+    grid_every: 25000
+    grid_early:
+    - 100
+    - 200
+    - 400
+    - 700
+    - 1000
+    - 1500
+    - 2500
+    - 4000
+    - 6000
+    - 10000
+    - 15000
+    - 20000
+  scaling_checkpoint:
+    _target_: scaling_callbacks.ScalingCheckpoint
+    dirpath: /mnt/home/phamd/dm_scaling_law_runs/duo_base/medium/frac_0.50/checkpoints
+    max_steps: 1000000
+    grid_every: 25000
+    grid_early:
+    - 100
+    - 200
+    - 400
+    - 700
+    - 1000
+    - 1500
+    - 2500
+    - 4000
+    - 6000
+    - 10000
+    - 15000
+    - 20000
+    rolling_every: 2500
+    walltime_margin_s: 1200
+  learning_rate_monitor:
+    _target_: lightning.pytorch.callbacks.LearningRateMonitor
+    logging_interval: step
+data:
+  train: openwebtext-train
+  valid: openwebtext-valid
+  modality: text
+  tokenizer_name_or_path: gpt2
+  cache_dir: /mnt/home/phamd/dm_scaling_law_data/text_data
+  wrap: true
+  streaming: false
+  insert_train_eos: true
+  insert_valid_eos: true
+  train_fraction: 0.5
+  subset_seed: 20250927
+  subset_perm_path: /mnt/home/phamd/dm_scaling_law_data/subsets/owt_train_perm_s20250927.npy
+  subset_perm_sha256: null
+  subset_tokens_path: /mnt/home/phamd/dm_scaling_law_data/subsets/owt_train_perm_s20250927_input_ids.npy
+  order_seed: 1
+model:
+  name: medium
+  type: ddit
+  hidden_size: 1024
+  cond_dim: 128
+  length: 1024
+  n_blocks: 24
+  n_heads: 16
+  scale_by_sigma: true
+  dropout: 0.1
+  tie_word_embeddings: false
+  vocab_lookup: true
+strategy:
+  _target_: lightning.pytorch.strategies.DDPStrategy
+  find_unused_parameters: false
+noise:
+  type: log-linear
+  eps: 0.001
+lr_scheduler:
+  _target_: transformers.get_constant_schedule_with_warmup
+  num_warmup_steps: 2500
+prior:
+  type: none
+  latent_width: 0
+  latent_height: 0
+algo:
+  name: duo_base
+  backbone: dit
+  parameterization: mean
+  time_conditioning: true
+  T: 0
+  subs_masking: false
+  causal_attention: false
+  ignore_bos: false
+  loss_type: elbo
+  compile_loss: true
+scaling:
+  val_every_steps: 5000
+  rolling_every: 2500
+  reseed_every: 1
+  grid_every: 25000
+  grid_early:
+  - 100
+  - 200
+  - 400
+  - 700
+  - 1000
+  - 1500
+  - 2500
+  - 4000
+  - 6000
+  - 10000
+  - 15000
+  - 20000
+  norm_every: 500
+  val_noise_seed: 4242
+  walltime_margin_s: 1200
+scaling_wandb:
+  project: duo-scaling-medium
+  group: medium
+  name: duo_base-medium-f0.5-s1
+  id: duobase-medium-f050-s1

frac_0.50/run_info.json

@@ -0,0 +1,24 @@
+{
+  "subset": {
+    "train_fraction": 0.5,
+    "n_blocks": 4365471,
+    "n_total": 8730942,
+    "perm_path": "/mnt/home/phamd/dm_scaling_law_data/subsets/owt_train_perm_s20250927.npy",
+    "perm_sha256": "fa545515efbfdd07138abceebb2671be60b5047431da2f50fd58a8bb986d05a7",
+    "subset_seed": 20250927,
+    "steps_per_epoch": 8526
+  },
+  "model": "medium",
+  "algo": "duo_base",
+  "seed": 1,
+  "order_seed": 1,
+  "global_batch_size": 512,
+  "max_steps": 1000000,
+  "num_parameters": 424477010,
+  "batch_size": 16,
+  "accumulate_grad_batches": 8,
+  "world_size": 4,
+  "gpu": "NVIDIA H100 80GB HBM3",
+  "code_sha": "7c9e7d04d4a4",
+  "code_dir": "/mnt/home/phamd/dm_scaling_law_runs/code/7c9e7d04d4a4"
+}

frac_0.50/segments.jsonl

@@ -0,0 +1,23 @@
+{"event": "job_start", "unix_time": 1790610218, "job_id": "105188", "restart": 0, "node": "alphagpu14"}
+{"event": "start", "unix_time": 1790610330.7759228, "step": 0, "train_hours": 1.5778156618277231e-09, "job_id": "105188", "node": "alphagpu14", "world_size": 4, "gpu": "NVIDIA H100 80GB HBM3"}
+{"event": "end", "unix_time": 1790695442.4581273, "step": 47260, "train_hours": 23.642133959786086, "job_id": "105188"}
+{"event": "job_start", "unix_time": 1790736878, "job_id": "106179", "restart": 0, "node": "alphagpu05"}
+{"event": "start", "unix_time": 1790736983.3521073, "step": 47260, "train_hours": 23.6397303049093, "job_id": "106179", "node": "alphagpu05", "world_size": 4, "gpu": "NVIDIA H100 80GB HBM3"}
+{"event": "end", "unix_time": 1790822089.0083222, "step": 94740, "train_hours": 47.28019037504706, "job_id": "106179"}
+{"event": "job_start", "unix_time": 1790947844, "job_id": "109990", "restart": 0, "node": "alphagpu03"}
+{"event": "start", "unix_time": 1790947943.9869413, "step": 94740, "train_hours": 47.275986222085905, "job_id": "109990", "node": "alphagpu03", "world_size": 4, "gpu": "NVIDIA H100 80GB HBM3"}
+{"event": "end", "unix_time": 1791033048.9253175, "step": 141400, "train_hours": 70.9162469056316, "job_id": "109990"}
+{"event": "job_start", "unix_time": 1791276305, "job_id": "114200", "restart": 0, "node": "alphagpu08"}
+{"event": "start", "unix_time": 1791276407.0213976, "step": 141400, "train_hours": 70.91194630317726, "job_id": "114200", "node": "alphagpu08", "world_size": 4, "gpu": "NVIDIA H100 80GB HBM3"}
+{"event": "job_start", "unix_time": 1791434239, "job_id": "122179", "restart": 0, "node": "alphagpu14"}
+{"event": "start", "unix_time": 1791434389.9962015, "step": 141400, "train_hours": 70.91194630222007, "job_id": "122179", "node": "alphagpu14", "world_size": 4, "gpu": "NVIDIA H100 80GB HBM3"}
+{"event": "end", "unix_time": 1791519452.092545, "step": 188460, "train_hours": 94.54030640498372, "job_id": "122179"}
+{"event": "job_start", "unix_time": 1791844093, "job_id": "125410", "restart": 0, "node": "alphagpu19"}
+{"event": "start", "unix_time": 1791844145.6245093, "step": 188460, "train_hours": 94.53792258227425, "job_id": "125410", "node": "alphagpu19", "world_size": 4, "gpu": "NVIDIA H200"}
+{"event": "end", "unix_time": 1791929323.796695, "step": 241780, "train_hours": 118.19852597620934, "job_id": "125410"}
+{"event": "job_start", "unix_time": 1792208137, "job_id": "132764", "restart": 0, "node": "alphagpu19"}
+{"event": "start", "unix_time": 1792208214.7160704, "step": 241780, "train_hours": 118.19616507997556, "job_id": "132764", "node": "alphagpu19", "world_size": 4, "gpu": "NVIDIA H200"}
+{"event": "end", "unix_time": 1792293358.828195, "step": 295060, "train_hours": 141.84730734642207, "job_id": "132764"}
+{"event": "job_start", "unix_time": 1793461508, "job_id": "138301", "restart": 0, "node": "alphagpu14"}
+{"event": "start", "unix_time": 1793461650.5891461, "step": 295060, "train_hours": 141.8450131829541, "job_id": "138301", "node": "alphagpu14", "world_size": 4, "gpu": "NVIDIA H100 80GB HBM3"}
+{"event": "end", "unix_time": 1794065122.0906155, "step": 629000, "train_hours": 309.4759858239445, "job_id": "138301"}

frac_0.60/code_sha

@@ -0,0 +1 @@
+7c9e7d04d4a4

frac_0.60/config_resolved.yaml

@@ -0,0 +1,204 @@
+mode: train
+seed: 1
+loader:
+  global_batch_size: 512
+  eval_global_batch_size: 64
+  batch_size: 16
+  eval_batch_size: 16
+  num_workers: 4
+  pin_memory: true
+  sampler: step_indexed
+sampling:
+  predictor: ancestral
+  steps: 1000
+  noise_removal: ancestral
+  use_float64: true
+  p_nucleus: 1.0
+  num_sample_batches: 0
+  num_sample_log: 2
+  semi_ar: false
+  stride_length: 1
+  num_strides: 1
+  guid_weight: null
+  psi:
+    time_profile: linear
+    high_mode: pure-posterior
+    middle_mode: pure-posterior
+    low_mode: pure-posterior
+    high_frac: 0.2
+    middle_frac: 0.6
+training:
+  ema: 0.9999
+  antithetic_sampling: true
+  importance_sampling: false
+  sampling_eps: 0.001
+  change_of_variables: false
+  loss_precision: bf16
+  finetune_path: ''
+  class_dropout_p: 0.1
+eval:
+  checkpoint_path: ''
+  disable_ema: false
+  compute_generative_perplexity: false
+  perplexity_batch_size: 8
+  compute_perplexity_on_sanity: false
+  gen_ppl_eval_model_name_or_path: gpt2-large
+  generate_samples: false
+  generated_samples_path: /mnt/home/phamd/dm_scaling_law_runs/duo_base/medium/frac_0.60/samples.json
+optim:
+  weight_decay: 0
+  lr: 0.0003
+  beta1: 0.9
+  beta2: 0.999
+  eps: 1.0e-08
+trainer:
+  _target_: lightning.Trainer
+  accelerator: cuda
+  num_nodes: 1
+  devices: 4
+  accumulate_grad_batches: 8
+  gradient_clip_val: 1.0
+  precision: bf16
+  num_sanity_val_steps: 2
+  max_steps: 1000000
+  log_every_n_steps: 100
+  limit_train_batches: 1.0
+  limit_val_batches: 1.0
+  val_check_interval: 40000
+  check_val_every_n_epoch: null
+  enable_checkpointing: false
+  enable_progress_bar: false
+wandb: null
+checkpointing:
+  save_dir: /mnt/home/phamd/dm_scaling_law_runs/duo_base/medium/frac_0.60
+  resume_from_ckpt: true
+  resume_ckpt_path: /mnt/home/phamd/dm_scaling_law_runs/duo_base/medium/frac_0.60/checkpoints/last.ckpt
+callbacks:
+  training_clock:
+    _target_: scaling_callbacks.TrainingClock
+    run_dir: /mnt/home/phamd/dm_scaling_law_runs/duo_base/medium/frac_0.60
+    global_batch_size: 512
+    seq_len: 1024
+    max_steps: 1000000
+    log_every_n_steps: 100
+    print_every_n_steps: 500
+  norm_monitor:
+    _target_: scaling_callbacks.NormMonitor
+    dirpath: /mnt/home/phamd/dm_scaling_law_runs/duo_base/medium/frac_0.60/norms
+    every_n_steps: 500
+    max_steps: 1000000
+    grid_every: 25000
+    grid_early:
+    - 100
+    - 200
+    - 400
+    - 700
+    - 1000
+    - 1500
+    - 2500
+    - 4000
+    - 6000
+    - 10000
+    - 15000
+    - 20000
+  scaling_checkpoint:
+    _target_: scaling_callbacks.ScalingCheckpoint
+    dirpath: /mnt/home/phamd/dm_scaling_law_runs/duo_base/medium/frac_0.60/checkpoints
+    max_steps: 1000000
+    grid_every: 25000
+    grid_early:
+    - 100
+    - 200
+    - 400
+    - 700
+    - 1000
+    - 1500
+    - 2500
+    - 4000
+    - 6000
+    - 10000
+    - 15000
+    - 20000
+    rolling_every: 2500
+    walltime_margin_s: 1200
+  learning_rate_monitor:
+    _target_: lightning.pytorch.callbacks.LearningRateMonitor
+    logging_interval: step
+data:
+  train: openwebtext-train
+  valid: openwebtext-valid
+  modality: text
+  tokenizer_name_or_path: gpt2
+  cache_dir: /mnt/home/phamd/dm_scaling_law_data/text_data
+  wrap: true
+  streaming: false
+  insert_train_eos: true
+  insert_valid_eos: true
+  train_fraction: 0.6
+  subset_seed: 20250927
+  subset_perm_path: /mnt/home/phamd/dm_scaling_law_data/subsets/owt_train_perm_s20250927.npy
+  subset_perm_sha256: null
+  subset_tokens_path: /mnt/home/phamd/dm_scaling_law_data/subsets/owt_train_perm_s20250927_input_ids.npy
+  order_seed: 1
+model:
+  name: medium
+  type: ddit
+  hidden_size: 1024
+  cond_dim: 128
+  length: 1024
+  n_blocks: 24
+  n_heads: 16
+  scale_by_sigma: true
+  dropout: 0.1
+  tie_word_embeddings: false
+  vocab_lookup: true
+strategy:
+  _target_: lightning.pytorch.strategies.DDPStrategy
+  find_unused_parameters: false
+noise:
+  type: log-linear
+  eps: 0.001
+lr_scheduler:
+  _target_: transformers.get_constant_schedule_with_warmup
+  num_warmup_steps: 2500
+prior:
+  type: none
+  latent_width: 0
+  latent_height: 0
+algo:
+  name: duo_base
+  backbone: dit
+  parameterization: mean
+  time_conditioning: true
+  T: 0
+  subs_masking: false
+  causal_attention: false
+  ignore_bos: false
+  loss_type: elbo
+  compile_loss: true
+scaling:
+  val_every_steps: 5000
+  rolling_every: 2500
+  reseed_every: 1
+  grid_every: 25000
+  grid_early:
+  - 100
+  - 200
+  - 400
+  - 700
+  - 1000
+  - 1500
+  - 2500
+  - 4000
+  - 6000
+  - 10000
+  - 15000
+  - 20000
+  norm_every: 500
+  val_noise_seed: 4242
+  walltime_margin_s: 1200
+scaling_wandb:
+  project: duo-scaling-medium
+  group: medium
+  name: duo_base-medium-f0.6-s1
+  id: duobase-medium-f060-s1

frac_0.60/run_info.json

@@ -0,0 +1,24 @@
+{
+  "subset": {
+    "train_fraction": 0.6,
+    "n_blocks": 5238565,
+    "n_total": 8730942,
+    "perm_path": "/mnt/home/phamd/dm_scaling_law_data/subsets/owt_train_perm_s20250927.npy",
+    "perm_sha256": "fa545515efbfdd07138abceebb2671be60b5047431da2f50fd58a8bb986d05a7",
+    "subset_seed": 20250927,
+    "steps_per_epoch": 10231
+  },
+  "model": "medium",
+  "algo": "duo_base",
+  "seed": 1,
+  "order_seed": 1,
+  "global_batch_size": 512,
+  "max_steps": 1000000,
+  "num_parameters": 424477010,
+  "batch_size": 16,
+  "accumulate_grad_batches": 8,
+  "world_size": 4,
+  "gpu": "NVIDIA H100 80GB HBM3",
+  "code_sha": "7c9e7d04d4a4",
+  "code_dir": "/mnt/home/phamd/dm_scaling_law_runs/code/7c9e7d04d4a4"
+}

frac_0.60/segments.jsonl

@@ -0,0 +1,18 @@
+{"event": "job_start", "unix_time": 1790620598, "job_id": "105189", "restart": 0, "node": "alphagpu05"}
+{"event": "start", "unix_time": 1790620719.5833268, "step": 0, "train_hours": 8.114924033482869e-09, "job_id": "105189", "node": "alphagpu05", "world_size": 4, "gpu": "NVIDIA H100 80GB HBM3"}
+{"event": "end", "unix_time": 1790705802.1367, "step": 46800, "train_hours": 23.634042647369206, "job_id": "105189"}
+{"event": "job_start", "unix_time": 1790739926, "job_id": "106541", "restart": 0, "node": "alphagpu13"}
+{"event": "start", "unix_time": 1790740000.0056832, "step": 46800, "train_hours": 23.631005547893132, "job_id": "106541", "node": "alphagpu13", "world_size": 4, "gpu": "NVIDIA H100 80GB HBM3"}
+{"event": "end", "unix_time": 1790825139.78283, "step": 95080, "train_hours": 47.280943657478005, "job_id": "106541"}
+{"event": "job_start", "unix_time": 1790952447, "job_id": "110025", "restart": 0, "node": "alphagpu06"}
+{"event": "start", "unix_time": 1790952576.5120125, "step": 95080, "train_hours": 47.278921957477515, "job_id": "110025", "node": "alphagpu06", "world_size": 4, "gpu": "NVIDIA H100 80GB HBM3"}
+{"event": "end", "unix_time": 1791037645.423862, "step": 142480, "train_hours": 70.90917525853884, "job_id": "110025"}
+{"event": "job_start", "unix_time": 1791277067, "job_id": "114316", "restart": 0, "node": "alphagpu05"}
+{"event": "start", "unix_time": 1791277184.1776369, "step": 142480, "train_hours": 70.9058895355908, "job_id": "114316", "node": "alphagpu05", "world_size": 4, "gpu": "NVIDIA H100 80GB HBM3"}
+{"event": "end", "unix_time": 1791362271.6516228, "step": 189480, "train_hours": 94.54129898608144, "job_id": "114316"}
+{"event": "job_start", "unix_time": 1791706865, "job_id": "122194", "restart": 0, "node": "alphagpu04"}
+{"event": "start", "unix_time": 1791706976.938208, "step": 189480, "train_hours": 94.53638903261731, "job_id": "122194", "node": "alphagpu04", "world_size": 4, "gpu": "NVIDIA H100 80GB HBM3"}
+{"event": "end", "unix_time": 1791792080.7285414, "step": 236040, "train_hours": 118.17633080463831, "job_id": "122194"}
+{"event": "job_start", "unix_time": 1792093837, "job_id": "130488", "restart": 0, "node": "alphagpu05"}
+{"event": "start", "unix_time": 1792093981.9679794, "step": 236040, "train_hours": 118.17184452401241, "job_id": "130488", "node": "alphagpu05", "world_size": 4, "gpu": "NVIDIA H100 80GB HBM3"}
+{"event": "end", "unix_time": 1792179043.245817, "step": 283140, "train_hours": 141.79997726748724, "job_id": "130488"}

frac_0.70/code_sha

@@ -0,0 +1 @@
+7c9e7d04d4a4

frac_0.70/config_resolved.yaml

@@ -0,0 +1,204 @@
+mode: train
+seed: 1
+loader:
+  global_batch_size: 512
+  eval_global_batch_size: 64
+  batch_size: 16
+  eval_batch_size: 16
+  num_workers: 4
+  pin_memory: true
+  sampler: step_indexed
+sampling:
+  predictor: ancestral
+  steps: 1000
+  noise_removal: ancestral
+  use_float64: true
+  p_nucleus: 1.0
+  num_sample_batches: 0
+  num_sample_log: 2
+  semi_ar: false
+  stride_length: 1
+  num_strides: 1
+  guid_weight: null
+  psi:
+    time_profile: linear
+    high_mode: pure-posterior
+    middle_mode: pure-posterior
+    low_mode: pure-posterior
+    high_frac: 0.2
+    middle_frac: 0.6
+training:
+  ema: 0.9999
+  antithetic_sampling: true
+  importance_sampling: false
+  sampling_eps: 0.001
+  change_of_variables: false
+  loss_precision: bf16
+  finetune_path: ''
+  class_dropout_p: 0.1
+eval:
+  checkpoint_path: ''
+  disable_ema: false
+  compute_generative_perplexity: false
+  perplexity_batch_size: 8
+  compute_perplexity_on_sanity: false
+  gen_ppl_eval_model_name_or_path: gpt2-large
+  generate_samples: false
+  generated_samples_path: /mnt/home/phamd/dm_scaling_law_runs/duo_base/medium/frac_0.70/samples.json
+optim:
+  weight_decay: 0
+  lr: 0.0003
+  beta1: 0.9
+  beta2: 0.999
+  eps: 1.0e-08
+trainer:
+  _target_: lightning.Trainer
+  accelerator: cuda
+  num_nodes: 1
+  devices: 4
+  accumulate_grad_batches: 8
+  gradient_clip_val: 1.0
+  precision: bf16
+  num_sanity_val_steps: 2
+  max_steps: 1000000
+  log_every_n_steps: 100
+  limit_train_batches: 1.0
+  limit_val_batches: 1.0
+  val_check_interval: 40000
+  check_val_every_n_epoch: null
+  enable_checkpointing: false
+  enable_progress_bar: false
+wandb: null
+checkpointing:
+  save_dir: /mnt/home/phamd/dm_scaling_law_runs/duo_base/medium/frac_0.70
+  resume_from_ckpt: true
+  resume_ckpt_path: /mnt/home/phamd/dm_scaling_law_runs/duo_base/medium/frac_0.70/checkpoints/last.ckpt
+callbacks:
+  training_clock:
+    _target_: scaling_callbacks.TrainingClock
+    run_dir: /mnt/home/phamd/dm_scaling_law_runs/duo_base/medium/frac_0.70
+    global_batch_size: 512
+    seq_len: 1024
+    max_steps: 1000000
+    log_every_n_steps: 100
+    print_every_n_steps: 500
+  norm_monitor:
+    _target_: scaling_callbacks.NormMonitor
+    dirpath: /mnt/home/phamd/dm_scaling_law_runs/duo_base/medium/frac_0.70/norms
+    every_n_steps: 500
+    max_steps: 1000000
+    grid_every: 25000
+    grid_early:
+    - 100
+    - 200
+    - 400
+    - 700
+    - 1000
+    - 1500
+    - 2500
+    - 4000
+    - 6000
+    - 10000
+    - 15000
+    - 20000
+  scaling_checkpoint:
+    _target_: scaling_callbacks.ScalingCheckpoint
+    dirpath: /mnt/home/phamd/dm_scaling_law_runs/duo_base/medium/frac_0.70/checkpoints
+    max_steps: 1000000
+    grid_every: 25000
+    grid_early:
+    - 100
+    - 200
+    - 400
+    - 700
+    - 1000
+    - 1500
+    - 2500
+    - 4000
+    - 6000
+    - 10000
+    - 15000
+    - 20000
+    rolling_every: 2500
+    walltime_margin_s: 1200
+  learning_rate_monitor:
+    _target_: lightning.pytorch.callbacks.LearningRateMonitor
+    logging_interval: step
+data:
+  train: openwebtext-train
+  valid: openwebtext-valid
+  modality: text
+  tokenizer_name_or_path: gpt2
+  cache_dir: /mnt/home/phamd/dm_scaling_law_data/text_data
+  wrap: true
+  streaming: false
+  insert_train_eos: true
+  insert_valid_eos: true
+  train_fraction: 0.7
+  subset_seed: 20250927
+  subset_perm_path: /mnt/home/phamd/dm_scaling_law_data/subsets/owt_train_perm_s20250927.npy
+  subset_perm_sha256: null
+  subset_tokens_path: /mnt/home/phamd/dm_scaling_law_data/subsets/owt_train_perm_s20250927_input_ids.npy
+  order_seed: 1
+model:
+  name: medium
+  type: ddit
+  hidden_size: 1024
+  cond_dim: 128
+  length: 1024
+  n_blocks: 24
+  n_heads: 16
+  scale_by_sigma: true
+  dropout: 0.1
+  tie_word_embeddings: false
+  vocab_lookup: true
+strategy:
+  _target_: lightning.pytorch.strategies.DDPStrategy
+  find_unused_parameters: false
+noise:
+  type: log-linear
+  eps: 0.001
+lr_scheduler:
+  _target_: transformers.get_constant_schedule_with_warmup
+  num_warmup_steps: 2500
+prior:
+  type: none
+  latent_width: 0
+  latent_height: 0
+algo:
+  name: duo_base
+  backbone: dit
+  parameterization: mean
+  time_conditioning: true
+  T: 0
+  subs_masking: false
+  causal_attention: false
+  ignore_bos: false
+  loss_type: elbo
+  compile_loss: true
+scaling:
+  val_every_steps: 5000
+  rolling_every: 2500
+  reseed_every: 1
+  grid_every: 25000
+  grid_early:
+  - 100
+  - 200
+  - 400
+  - 700
+  - 1000
+  - 1500
+  - 2500
+  - 4000
+  - 6000
+  - 10000
+  - 15000
+  - 20000
+  norm_every: 500
+  val_noise_seed: 4242
+  walltime_margin_s: 1200
+scaling_wandb:
+  project: duo-scaling-medium
+  group: medium
+  name: duo_base-medium-f0.7-s1
+  id: duobase-medium-f070-s1

frac_0.70/run_info.json

@@ -0,0 +1,24 @@
+{
+  "subset": {
+    "train_fraction": 0.7,
+    "n_blocks": 6111659,
+    "n_total": 8730942,
+    "perm_path": "/mnt/home/phamd/dm_scaling_law_data/subsets/owt_train_perm_s20250927.npy",
+    "perm_sha256": "fa545515efbfdd07138abceebb2671be60b5047431da2f50fd58a8bb986d05a7",
+    "subset_seed": 20250927,
+    "steps_per_epoch": 11936
+  },
+  "model": "medium",
+  "algo": "duo_base",
+  "seed": 1,
+  "order_seed": 1,
+  "global_batch_size": 512,
+  "max_steps": 1000000,
+  "num_parameters": 424477010,
+  "batch_size": 16,
+  "accumulate_grad_batches": 8,
+  "world_size": 4,
+  "gpu": "NVIDIA H100 80GB HBM3",
+  "code_sha": "7c9e7d04d4a4",
+  "code_dir": "/mnt/home/phamd/dm_scaling_law_runs/code/7c9e7d04d4a4"
+}

frac_0.70/segments.jsonl

@@ -0,0 +1,20 @@
+{"event": "job_start", "unix_time": 1790624525, "job_id": "105190", "restart": 0, "node": "alphagpu12"}
+{"event": "start", "unix_time": 1790624695.2335122, "step": 0, "train_hours": 6.461567762825224e-09, "job_id": "105190", "node": "alphagpu12", "world_size": 4, "gpu": "NVIDIA H100 80GB HBM3"}
+{"event": "end", "unix_time": 1790709732.5500362, "step": 47260, "train_hours": 23.621476856401554, "job_id": "105190"}
+{"event": "job_start", "unix_time": 1790742974, "job_id": "106676", "restart": 0, "node": "alphagpu14"}
+{"event": "start", "unix_time": 1790743057.7885098, "step": 47260, "train_hours": 23.618379847408118, "job_id": "106676", "node": "alphagpu14", "world_size": 4, "gpu": "NVIDIA H100 80GB HBM3"}
+{"event": "end", "unix_time": 1790828173.0540931, "step": 94380, "train_hours": 47.26150918152468, "job_id": "106676"}
+{"event": "job_start", "unix_time": 1790952738, "job_id": "110069", "restart": 0, "node": "alphagpu01"}
+{"event": "start", "unix_time": 1790952890.9295871, "step": 94380, "train_hours": 47.25906027697829, "job_id": "110069", "node": "alphagpu01", "world_size": 4, "gpu": "NVIDIA H100 80GB HBM3"}
+{"event": "end", "unix_time": 1791037947.4510727, "step": 141780, "train_hours": 70.88587180767311, "job_id": "110069"}
+{"event": "job_start", "unix_time": 1791279465, "job_id": "114326", "restart": 0, "node": "alphagpu08"}
+{"event": "start", "unix_time": 1791279548.2982423, "step": 141780, "train_hours": 70.88334076188039, "job_id": "114326", "node": "alphagpu08", "world_size": 4, "gpu": "NVIDIA H100 80GB HBM3"}
+{"event": "job_start", "unix_time": 1791544627, "job_id": "122229", "restart": 0, "node": "alphagpu06"}
+{"event": "start", "unix_time": 1791544739.3291254, "step": 141780, "train_hours": 70.88334076156141, "job_id": "122229", "node": "alphagpu06", "world_size": 4, "gpu": "NVIDIA H100 80GB HBM3"}
+{"event": "end", "unix_time": 1791629848.4906213, "step": 187680, "train_hours": 94.52477452847175, "job_id": "122229"}
+{"event": "job_start", "unix_time": 1791931974, "job_id": "127882", "restart": 0, "node": "alphagpu24"}
+{"event": "start", "unix_time": 1791932059.2333562, "step": 187680, "train_hours": 94.520716437856, "job_id": "127882", "node": "alphagpu24", "world_size": 4, "gpu": "NVIDIA H200"}
+{"event": "end", "unix_time": 1792017195.320876, "step": 241080, "train_hours": 118.16962964683539, "job_id": "127882"}
+{"event": "job_start", "unix_time": 1792234658, "job_id": "133806", "restart": 0, "node": "alphagpu24"}
+{"event": "start", "unix_time": 1792234770.2981892, "step": 241080, "train_hours": 118.16733743754267, "job_id": "133806", "node": "alphagpu24", "world_size": 4, "gpu": "NVIDIA H200"}
+{"event": "end", "unix_time": 1792319869.695893, "step": 294620, "train_hours": 141.80605903517016, "job_id": "133806"}

frac_0.80/code_sha

@@ -0,0 +1 @@
+7c9e7d04d4a4

frac_0.80/config_resolved.yaml

@@ -0,0 +1,204 @@
+mode: train
+seed: 1
+loader:
+  global_batch_size: 512
+  eval_global_batch_size: 64
+  batch_size: 16
+  eval_batch_size: 16
+  num_workers: 4
+  pin_memory: true
+  sampler: step_indexed
+sampling:
+  predictor: ancestral
+  steps: 1000
+  noise_removal: ancestral
+  use_float64: true
+  p_nucleus: 1.0
+  num_sample_batches: 0
+  num_sample_log: 2
+  semi_ar: false
+  stride_length: 1
+  num_strides: 1
+  guid_weight: null
+  psi:
+    time_profile: linear
+    high_mode: pure-posterior
+    middle_mode: pure-posterior
+    low_mode: pure-posterior
+    high_frac: 0.2
+    middle_frac: 0.6
+training:
+  ema: 0.9999
+  antithetic_sampling: true
+  importance_sampling: false
+  sampling_eps: 0.001
+  change_of_variables: false
+  loss_precision: bf16
+  finetune_path: ''
+  class_dropout_p: 0.1
+eval:
+  checkpoint_path: ''
+  disable_ema: false
+  compute_generative_perplexity: false
+  perplexity_batch_size: 8
+  compute_perplexity_on_sanity: false
+  gen_ppl_eval_model_name_or_path: gpt2-large
+  generate_samples: false
+  generated_samples_path: /mnt/home/phamd/dm_scaling_law_runs/duo_base/medium/frac_0.80/samples.json
+optim:
+  weight_decay: 0
+  lr: 0.0003
+  beta1: 0.9
+  beta2: 0.999
+  eps: 1.0e-08
+trainer:
+  _target_: lightning.Trainer
+  accelerator: cuda
+  num_nodes: 1
+  devices: 4
+  accumulate_grad_batches: 8
+  gradient_clip_val: 1.0
+  precision: bf16
+  num_sanity_val_steps: 2
+  max_steps: 1000000
+  log_every_n_steps: 100
+  limit_train_batches: 1.0
+  limit_val_batches: 1.0
+  val_check_interval: 40000
+  check_val_every_n_epoch: null
+  enable_checkpointing: false
+  enable_progress_bar: false
+wandb: null
+checkpointing:
+  save_dir: /mnt/home/phamd/dm_scaling_law_runs/duo_base/medium/frac_0.80
+  resume_from_ckpt: true
+  resume_ckpt_path: /mnt/home/phamd/dm_scaling_law_runs/duo_base/medium/frac_0.80/checkpoints/last.ckpt
+callbacks:
+  training_clock:
+    _target_: scaling_callbacks.TrainingClock
+    run_dir: /mnt/home/phamd/dm_scaling_law_runs/duo_base/medium/frac_0.80
+    global_batch_size: 512
+    seq_len: 1024
+    max_steps: 1000000
+    log_every_n_steps: 100
+    print_every_n_steps: 500
+  norm_monitor:
+    _target_: scaling_callbacks.NormMonitor
+    dirpath: /mnt/home/phamd/dm_scaling_law_runs/duo_base/medium/frac_0.80/norms
+    every_n_steps: 500
+    max_steps: 1000000
+    grid_every: 25000
+    grid_early:
+    - 100
+    - 200
+    - 400
+    - 700
+    - 1000
+    - 1500
+    - 2500
+    - 4000
+    - 6000
+    - 10000
+    - 15000
+    - 20000
+  scaling_checkpoint:
+    _target_: scaling_callbacks.ScalingCheckpoint
+    dirpath: /mnt/home/phamd/dm_scaling_law_runs/duo_base/medium/frac_0.80/checkpoints
+    max_steps: 1000000
+    grid_every: 25000
+    grid_early:
+    - 100
+    - 200
+    - 400
+    - 700
+    - 1000
+    - 1500
+    - 2500
+    - 4000
+    - 6000
+    - 10000
+    - 15000
+    - 20000
+    rolling_every: 2500
+    walltime_margin_s: 1200
+  learning_rate_monitor:
+    _target_: lightning.pytorch.callbacks.LearningRateMonitor
+    logging_interval: step
+data:
+  train: openwebtext-train
+  valid: openwebtext-valid
+  modality: text
+  tokenizer_name_or_path: gpt2
+  cache_dir: /mnt/home/phamd/dm_scaling_law_data/text_data
+  wrap: true
+  streaming: false
+  insert_train_eos: true
+  insert_valid_eos: true
+  train_fraction: 0.8
+  subset_seed: 20250927
+  subset_perm_path: /mnt/home/phamd/dm_scaling_law_data/subsets/owt_train_perm_s20250927.npy
+  subset_perm_sha256: null
+  subset_tokens_path: /mnt/home/phamd/dm_scaling_law_data/subsets/owt_train_perm_s20250927_input_ids.npy
+  order_seed: 1
+model:
+  name: medium
+  type: ddit
+  hidden_size: 1024
+  cond_dim: 128
+  length: 1024
+  n_blocks: 24
+  n_heads: 16
+  scale_by_sigma: true
+  dropout: 0.1
+  tie_word_embeddings: false
+  vocab_lookup: true
+strategy:
+  _target_: lightning.pytorch.strategies.DDPStrategy
+  find_unused_parameters: false
+noise:
+  type: log-linear
+  eps: 0.001
+lr_scheduler:
+  _target_: transformers.get_constant_schedule_with_warmup
+  num_warmup_steps: 2500
+prior:
+  type: none
+  latent_width: 0
+  latent_height: 0
+algo:
+  name: duo_base
+  backbone: dit
+  parameterization: mean
+  time_conditioning: true
+  T: 0
+  subs_masking: false
+  causal_attention: false
+  ignore_bos: false
+  loss_type: elbo
+  compile_loss: true
+scaling:
+  val_every_steps: 5000
+  rolling_every: 2500
+  reseed_every: 1
+  grid_every: 25000
+  grid_early:
+  - 100
+  - 200
+  - 400
+  - 700
+  - 1000
+  - 1500
+  - 2500
+  - 4000
+  - 6000
+  - 10000
+  - 15000
+  - 20000
+  norm_every: 500
+  val_noise_seed: 4242
+  walltime_margin_s: 1200
+scaling_wandb:
+  project: duo-scaling-medium
+  group: medium
+  name: duo_base-medium-f0.8-s1
+  id: duobase-medium-f080-s1

frac_0.80/run_info.json

@@ -0,0 +1,24 @@
+{
+  "subset": {
+    "train_fraction": 0.8,
+    "n_blocks": 6984754,
+    "n_total": 8730942,
+    "perm_path": "/mnt/home/phamd/dm_scaling_law_data/subsets/owt_train_perm_s20250927.npy",
+    "perm_sha256": "fa545515efbfdd07138abceebb2671be60b5047431da2f50fd58a8bb986d05a7",
+    "subset_seed": 20250927,
+    "steps_per_epoch": 13642
+  },
+  "model": "medium",
+  "algo": "duo_base",
+  "seed": 1,
+  "order_seed": 1,
+  "global_batch_size": 512,
+  "max_steps": 1000000,
+  "num_parameters": 424477010,
+  "batch_size": 16,
+  "accumulate_grad_batches": 8,
+  "world_size": 4,
+  "gpu": "NVIDIA H100 80GB HBM3",
+  "code_sha": "7c9e7d04d4a4",
+  "code_dir": "/mnt/home/phamd/dm_scaling_law_runs/code/7c9e7d04d4a4"
+}

frac_0.80/segments.jsonl

@@ -0,0 +1,20 @@
+{"event": "job_start", "unix_time": 1790629278, "job_id": "105191", "restart": 0, "node": "alphagpu06"}
+{"event": "start", "unix_time": 1790629451.4253347, "step": 0, "train_hours": 2.922283278571235e-09, "job_id": "105191", "node": "alphagpu06", "world_size": 4, "gpu": "NVIDIA H100 80GB HBM3"}
+{"event": "end", "unix_time": 1790714495.9952013, "step": 46780, "train_hours": 23.623491646492234, "job_id": "105191"}
+{"event": "job_start", "unix_time": 1790748108, "job_id": "106863", "restart": 0, "node": "alphagpu06"}
+{"event": "start", "unix_time": 1790748254.0963085, "step": 46780, "train_hours": 23.619046249370765, "job_id": "106863", "node": "alphagpu06", "world_size": 4, "gpu": "NVIDIA H100 80GB HBM3"}
+{"event": "end", "unix_time": 1790833333.4995708, "step": 93580, "train_hours": 47.252213836701124, "job_id": "106863"}
+{"event": "job_start", "unix_time": 1791038003, "job_id": "110177", "restart": 0, "node": "alphagpu01"}
+{"event": "start", "unix_time": 1791038088.5590272, "step": 93580, "train_hours": 47.24791134275056, "job_id": "110177", "node": "alphagpu01", "world_size": 4, "gpu": "NVIDIA H100 80GB HBM3"}
+{"event": "end", "unix_time": 1791123204.002091, "step": 141040, "train_hours": 70.89108998284028, "job_id": "110177"}
+{"event": "job_start", "unix_time": 1791308133, "job_id": "116365", "restart": 0, "node": "alphagpu08"}
+{"event": "start", "unix_time": 1791308273.8703966, "step": 141040, "train_hours": 70.88841629375315, "job_id": "116365", "node": "alphagpu08", "world_size": 4, "gpu": "NVIDIA H100 80GB HBM3"}
+{"event": "job_start", "unix_time": 1791627251, "job_id": "123184", "restart": 0, "node": "alphagpu09"}
+{"event": "start", "unix_time": 1791627386.2029757, "step": 141040, "train_hours": 70.88841629213421, "job_id": "123184", "node": "alphagpu09", "world_size": 4, "gpu": "NVIDIA H100 80GB HBM3"}
+{"event": "end", "unix_time": 1791712532.1965685, "step": 187520, "train_hours": 94.54008119146236, "job_id": "123184"}
+{"event": "job_start", "unix_time": 1792067133, "job_id": "129366", "restart": 0, "node": "alphagpu06"}
+{"event": "start", "unix_time": 1792067246.4423652, "step": 187520, "train_hours": 94.5361910146304, "job_id": "129366", "node": "alphagpu06", "world_size": 4, "gpu": "NVIDIA H100 80GB HBM3"}
+{"event": "end", "unix_time": 1792152335.9161036, "step": 234420, "train_hours": 118.1721559605807, "job_id": "129366"}
+{"event": "job_start", "unix_time": 1792237842, "job_id": "136391", "restart": 0, "node": "alphagpu04"}
+{"event": "start", "unix_time": 1792237931.7119913, "step": 234420, "train_hours": 118.16769584685099, "job_id": "136391", "node": "alphagpu04", "world_size": 4, "gpu": "NVIDIA H100 80GB HBM3"}
+{"event": "end", "unix_time": 1792323047.9202275, "step": 281040, "train_hours": 141.8110870454895, "job_id": "136391"}

frac_0.90/code_sha

@@ -0,0 +1 @@
+7c9e7d04d4a4

frac_0.90/config_resolved.yaml

@@ -0,0 +1,204 @@
+mode: train
+seed: 1
+loader:
+  global_batch_size: 512
+  eval_global_batch_size: 64
+  batch_size: 16
+  eval_batch_size: 16
+  num_workers: 4
+  pin_memory: true
+  sampler: step_indexed
+sampling:
+  predictor: ancestral
+  steps: 1000
+  noise_removal: ancestral
+  use_float64: true
+  p_nucleus: 1.0
+  num_sample_batches: 0
+  num_sample_log: 2
+  semi_ar: false
+  stride_length: 1
+  num_strides: 1
+  guid_weight: null
+  psi:
+    time_profile: linear
+    high_mode: pure-posterior
+    middle_mode: pure-posterior
+    low_mode: pure-posterior
+    high_frac: 0.2
+    middle_frac: 0.6
+training:
+  ema: 0.9999
+  antithetic_sampling: true
+  importance_sampling: false
+  sampling_eps: 0.001
+  change_of_variables: false
+  loss_precision: bf16
+  finetune_path: ''
+  class_dropout_p: 0.1
+eval:
+  checkpoint_path: ''
+  disable_ema: false
+  compute_generative_perplexity: false
+  perplexity_batch_size: 8
+  compute_perplexity_on_sanity: false
+  gen_ppl_eval_model_name_or_path: gpt2-large
+  generate_samples: false
+  generated_samples_path: /mnt/home/phamd/dm_scaling_law_runs/duo_base/medium/frac_0.90/samples.json
+optim:
+  weight_decay: 0
+  lr: 0.0003
+  beta1: 0.9
+  beta2: 0.999
+  eps: 1.0e-08
+trainer:
+  _target_: lightning.Trainer
+  accelerator: cuda
+  num_nodes: 1
+  devices: 4
+  accumulate_grad_batches: 8
+  gradient_clip_val: 1.0
+  precision: bf16
+  num_sanity_val_steps: 2
+  max_steps: 1000000
+  log_every_n_steps: 100
+  limit_train_batches: 1.0
+  limit_val_batches: 1.0
+  val_check_interval: 40000
+  check_val_every_n_epoch: null
+  enable_checkpointing: false
+  enable_progress_bar: false
+wandb: null
+checkpointing:
+  save_dir: /mnt/home/phamd/dm_scaling_law_runs/duo_base/medium/frac_0.90
+  resume_from_ckpt: true
+  resume_ckpt_path: /mnt/home/phamd/dm_scaling_law_runs/duo_base/medium/frac_0.90/checkpoints/last.ckpt
+callbacks:
+  training_clock:
+    _target_: scaling_callbacks.TrainingClock
+    run_dir: /mnt/home/phamd/dm_scaling_law_runs/duo_base/medium/frac_0.90
+    global_batch_size: 512
+    seq_len: 1024
+    max_steps: 1000000
+    log_every_n_steps: 100
+    print_every_n_steps: 500
+  norm_monitor:
+    _target_: scaling_callbacks.NormMonitor
+    dirpath: /mnt/home/phamd/dm_scaling_law_runs/duo_base/medium/frac_0.90/norms
+    every_n_steps: 500
+    max_steps: 1000000
+    grid_every: 25000
+    grid_early:
+    - 100
+    - 200
+    - 400
+    - 700
+    - 1000
+    - 1500
+    - 2500
+    - 4000
+    - 6000
+    - 10000
+    - 15000
+    - 20000
+  scaling_checkpoint:
+    _target_: scaling_callbacks.ScalingCheckpoint
+    dirpath: /mnt/home/phamd/dm_scaling_law_runs/duo_base/medium/frac_0.90/checkpoints
+    max_steps: 1000000
+    grid_every: 25000
+    grid_early:
+    - 100
+    - 200
+    - 400
+    - 700
+    - 1000
+    - 1500
+    - 2500
+    - 4000
+    - 6000
+    - 10000
+    - 15000
+    - 20000
+    rolling_every: 2500
+    walltime_margin_s: 1200
+  learning_rate_monitor:
+    _target_: lightning.pytorch.callbacks.LearningRateMonitor
+    logging_interval: step
+data:
+  train: openwebtext-train
+  valid: openwebtext-valid
+  modality: text
+  tokenizer_name_or_path: gpt2
+  cache_dir: /mnt/home/phamd/dm_scaling_law_data/text_data
+  wrap: true
+  streaming: false
+  insert_train_eos: true
+  insert_valid_eos: true
+  train_fraction: 0.9
+  subset_seed: 20250927
+  subset_perm_path: /mnt/home/phamd/dm_scaling_law_data/subsets/owt_train_perm_s20250927.npy
+  subset_perm_sha256: null
+  subset_tokens_path: /mnt/home/phamd/dm_scaling_law_data/subsets/owt_train_perm_s20250927_input_ids.npy
+  order_seed: 1
+model:
+  name: medium
+  type: ddit
+  hidden_size: 1024
+  cond_dim: 128
+  length: 1024
+  n_blocks: 24
+  n_heads: 16
+  scale_by_sigma: true
+  dropout: 0.1
+  tie_word_embeddings: false
+  vocab_lookup: true
+strategy:
+  _target_: lightning.pytorch.strategies.DDPStrategy
+  find_unused_parameters: false
+noise:
+  type: log-linear
+  eps: 0.001
+lr_scheduler:
+  _target_: transformers.get_constant_schedule_with_warmup
+  num_warmup_steps: 2500
+prior:
+  type: none
+  latent_width: 0
+  latent_height: 0
+algo:
+  name: duo_base
+  backbone: dit
+  parameterization: mean
+  time_conditioning: true
+  T: 0
+  subs_masking: false
+  causal_attention: false
+  ignore_bos: false
+  loss_type: elbo
+  compile_loss: true
+scaling:
+  val_every_steps: 5000
+  rolling_every: 2500
+  reseed_every: 1
+  grid_every: 25000
+  grid_early:
+  - 100
+  - 200
+  - 400
+  - 700
+  - 1000
+  - 1500
+  - 2500
+  - 4000
+  - 6000
+  - 10000
+  - 15000
+  - 20000
+  norm_every: 500
+  val_noise_seed: 4242
+  walltime_margin_s: 1200
+scaling_wandb:
+  project: duo-scaling-medium
+  group: medium
+  name: duo_base-medium-f0.9-s1
+  id: duobase-medium-f090-s1

frac_0.90/run_info.json

@@ -0,0 +1,24 @@
+{
+  "subset": {
+    "train_fraction": 0.9,
+    "n_blocks": 7857848,
+    "n_total": 8730942,
+    "perm_path": "/mnt/home/phamd/dm_scaling_law_data/subsets/owt_train_perm_s20250927.npy",
+    "perm_sha256": "fa545515efbfdd07138abceebb2671be60b5047431da2f50fd58a8bb986d05a7",
+    "subset_seed": 20250927,
+    "steps_per_epoch": 15347
+  },
+  "model": "medium",
+  "algo": "duo_base",
+  "seed": 1,
+  "order_seed": 1,
+  "global_batch_size": 512,
+  "max_steps": 1000000,
+  "num_parameters": 424477010,
+  "batch_size": 16,
+  "accumulate_grad_batches": 8,
+  "world_size": 4,
+  "gpu": "NVIDIA H100 80GB HBM3",
+  "code_sha": "7c9e7d04d4a4",
+  "code_dir": "/mnt/home/phamd/dm_scaling_law_runs/code/7c9e7d04d4a4"
+}

frac_0.90/segments.jsonl

@@ -0,0 +1,18 @@
+{"event": "job_start", "unix_time": 1790629279, "job_id": "105192", "restart": 0, "node": "alphagpu06"}
+{"event": "start", "unix_time": 1790629451.2387514, "step": 0, "train_hours": 5.336478352546692e-09, "job_id": "105192", "node": "alphagpu06", "world_size": 4, "gpu": "NVIDIA H100 80GB HBM3"}
+{"event": "end", "unix_time": 1790714509.5322669, "step": 46800, "train_hours": 23.62730380230221, "job_id": "105192"}
+{"event": "job_start", "unix_time": 1790743046, "job_id": "106862", "restart": 0, "node": "alphagpu07"}
+{"event": "start", "unix_time": 1790743136.0054944, "step": 46800, "train_hours": 23.62285451832564, "job_id": "106862", "node": "alphagpu07", "world_size": 4, "gpu": "NVIDIA H100 80GB HBM3"}
+{"event": "end", "unix_time": 1790828260.3258064, "step": 94000, "train_hours": 47.268499059762235, "job_id": "106862"}
+{"event": "job_start", "unix_time": 1791022706, "job_id": "110071", "restart": 0, "node": "alphagpu04"}
+{"event": "start", "unix_time": 1791022826.173522, "step": 94000, "train_hours": 47.265092479516234, "job_id": "110071", "node": "alphagpu04", "world_size": 4, "gpu": "NVIDIA H100 80GB HBM3"}
+{"event": "end", "unix_time": 1791107940.4655344, "step": 141200, "train_hours": 70.90795138263869, "job_id": "110071"}
+{"event": "job_start", "unix_time": 1791303123, "job_id": "116265", "restart": 0, "node": "alphagpu13"}
+{"event": "start", "unix_time": 1791303191.0927608, "step": 141200, "train_hours": 70.90447906468374, "job_id": "116265", "node": "alphagpu13", "world_size": 4, "gpu": "NVIDIA H100 80GB HBM3"}
+{"event": "end", "unix_time": 1791388326.1368344, "step": 189720, "train_hours": 94.55310242912189, "job_id": "116265"}
+{"event": "job_start", "unix_time": 1791706865, "job_id": "122955", "restart": 0, "node": "alphagpu04"}
+{"event": "start", "unix_time": 1791706994.2375085, "step": 189720, "train_hours": 94.55090508211843, "job_id": "122955", "node": "alphagpu04", "world_size": 4, "gpu": "NVIDIA H100 80GB HBM3"}
+{"event": "end", "unix_time": 1791792096.1006174, "step": 236000, "train_hours": 118.19031150967865, "job_id": "122955"}
+{"event": "job_start", "unix_time": 1792098223, "job_id": "130487", "restart": 0, "node": "alphagpu21"}
+{"event": "start", "unix_time": 1792098381.6650364, "step": 236000, "train_hours": 118.1849750277426, "job_id": "130487", "node": "alphagpu21", "world_size": 4, "gpu": "NVIDIA H200"}
+{"event": "end", "unix_time": 1792183425.2490911, "step": 289420, "train_hours": 141.80819283276125, "job_id": "130487"}

frac_1.00/code_sha

@@ -0,0 +1 @@
+7c9e7d04d4a4

frac_1.00/config_resolved.yaml

@@ -0,0 +1,204 @@
+mode: train
+seed: 1
+loader:
+  global_batch_size: 512
+  eval_global_batch_size: 64
+  batch_size: 16
+  eval_batch_size: 16
+  num_workers: 4
+  pin_memory: true
+  sampler: step_indexed
+sampling:
+  predictor: ancestral
+  steps: 1000
+  noise_removal: ancestral
+  use_float64: true
+  p_nucleus: 1.0
+  num_sample_batches: 0
+  num_sample_log: 2
+  semi_ar: false
+  stride_length: 1
+  num_strides: 1
+  guid_weight: null
+  psi:
+    time_profile: linear
+    high_mode: pure-posterior
+    middle_mode: pure-posterior
+    low_mode: pure-posterior
+    high_frac: 0.2
+    middle_frac: 0.6
+training:
+  ema: 0.9999
+  antithetic_sampling: true
+  importance_sampling: false
+  sampling_eps: 0.001
+  change_of_variables: false
+  loss_precision: bf16
+  finetune_path: ''
+  class_dropout_p: 0.1
+eval:
+  checkpoint_path: ''
+  disable_ema: false
+  compute_generative_perplexity: false
+  perplexity_batch_size: 8
+  compute_perplexity_on_sanity: false
+  gen_ppl_eval_model_name_or_path: gpt2-large
+  generate_samples: false
+  generated_samples_path: /mnt/home/phamd/dm_scaling_law_runs/duo_base/medium/frac_1.00/samples.json
+optim:
+  weight_decay: 0
+  lr: 0.0003
+  beta1: 0.9
+  beta2: 0.999
+  eps: 1.0e-08
+trainer:
+  _target_: lightning.Trainer
+  accelerator: cuda
+  num_nodes: 1
+  devices: 4
+  accumulate_grad_batches: 8
+  gradient_clip_val: 1.0
+  precision: bf16
+  num_sanity_val_steps: 2
+  max_steps: 1000000
+  log_every_n_steps: 100
+  limit_train_batches: 1.0
+  limit_val_batches: 1.0
+  val_check_interval: 40000
+  check_val_every_n_epoch: null
+  enable_checkpointing: false
+  enable_progress_bar: false
+wandb: null
+checkpointing:
+  save_dir: /mnt/home/phamd/dm_scaling_law_runs/duo_base/medium/frac_1.00
+  resume_from_ckpt: true
+  resume_ckpt_path: /mnt/home/phamd/dm_scaling_law_runs/duo_base/medium/frac_1.00/checkpoints/last.ckpt
+callbacks:
+  training_clock:
+    _target_: scaling_callbacks.TrainingClock
+    run_dir: /mnt/home/phamd/dm_scaling_law_runs/duo_base/medium/frac_1.00
+    global_batch_size: 512
+    seq_len: 1024
+    max_steps: 1000000
+    log_every_n_steps: 100
+    print_every_n_steps: 500
+  norm_monitor:
+    _target_: scaling_callbacks.NormMonitor
+    dirpath: /mnt/home/phamd/dm_scaling_law_runs/duo_base/medium/frac_1.00/norms
+    every_n_steps: 500
+    max_steps: 1000000
+    grid_every: 25000
+    grid_early:
+    - 100
+    - 200
+    - 400
+    - 700
+    - 1000
+    - 1500
+    - 2500
+    - 4000
+    - 6000
+    - 10000
+    - 15000
+    - 20000
+  scaling_checkpoint:
+    _target_: scaling_callbacks.ScalingCheckpoint
+    dirpath: /mnt/home/phamd/dm_scaling_law_runs/duo_base/medium/frac_1.00/checkpoints
+    max_steps: 1000000
+    grid_every: 25000
+    grid_early:
+    - 100
+    - 200
+    - 400
+    - 700
+    - 1000
+    - 1500
+    - 2500
+    - 4000
+    - 6000
+    - 10000
+    - 15000
+    - 20000
+    rolling_every: 2500
+    walltime_margin_s: 1200
+  learning_rate_monitor:
+    _target_: lightning.pytorch.callbacks.LearningRateMonitor
+    logging_interval: step
+data:
+  train: openwebtext-train
+  valid: openwebtext-valid
+  modality: text
+  tokenizer_name_or_path: gpt2
+  cache_dir: /mnt/home/phamd/dm_scaling_law_data/text_data
+  wrap: true
+  streaming: false
+  insert_train_eos: true
+  insert_valid_eos: true
+  train_fraction: 1.0
+  subset_seed: 20250927
+  subset_perm_path: /mnt/home/phamd/dm_scaling_law_data/subsets/owt_train_perm_s20250927.npy
+  subset_perm_sha256: null
+  subset_tokens_path: /mnt/home/phamd/dm_scaling_law_data/subsets/owt_train_perm_s20250927_input_ids.npy
+  order_seed: 1
+model:
+  name: medium
+  type: ddit
+  hidden_size: 1024
+  cond_dim: 128
+  length: 1024
+  n_blocks: 24
+  n_heads: 16
+  scale_by_sigma: true
+  dropout: 0.1
+  tie_word_embeddings: false
+  vocab_lookup: true
+strategy:
+  _target_: lightning.pytorch.strategies.DDPStrategy
+  find_unused_parameters: false
+noise:
+  type: log-linear
+  eps: 0.001
+lr_scheduler:
+  _target_: transformers.get_constant_schedule_with_warmup
+  num_warmup_steps: 2500
+prior:
+  type: none
+  latent_width: 0
+  latent_height: 0
+algo:
+  name: duo_base
+  backbone: dit
+  parameterization: mean
+  time_conditioning: true
+  T: 0
+  subs_masking: false
+  causal_attention: false
+  ignore_bos: false
+  loss_type: elbo
+  compile_loss: true
+scaling:
+  val_every_steps: 5000
+  rolling_every: 2500
+  reseed_every: 1
+  grid_every: 25000
+  grid_early:
+  - 100
+  - 200
+  - 400
+  - 700
+  - 1000
+  - 1500
+  - 2500
+  - 4000
+  - 6000
+  - 10000
+  - 15000
+  - 20000
+  norm_every: 500
+  val_noise_seed: 4242
+  walltime_margin_s: 1200
+scaling_wandb:
+  project: duo-scaling-medium
+  group: medium
+  name: duo_base-medium-f1.0-s1
+  id: duobase-medium-f100-s1

frac_1.00/run_info.json

@@ -0,0 +1,24 @@
+{
+  "subset": {
+    "train_fraction": 1.0,
+    "n_blocks": 8730942,
+    "n_total": 8730942,
+    "perm_path": "/mnt/home/phamd/dm_scaling_law_data/subsets/owt_train_perm_s20250927.npy",
+    "perm_sha256": "fa545515efbfdd07138abceebb2671be60b5047431da2f50fd58a8bb986d05a7",
+    "subset_seed": 20250927,
+    "steps_per_epoch": 17052
+  },
+  "model": "medium",
+  "algo": "duo_base",
+  "seed": 1,
+  "order_seed": 1,
+  "global_batch_size": 512,
+  "max_steps": 1000000,
+  "num_parameters": 424477010,
+  "batch_size": 16,
+  "accumulate_grad_batches": 8,
+  "world_size": 4,
+  "gpu": "NVIDIA H100 80GB HBM3",
+  "code_sha": "7c9e7d04d4a4",
+  "code_dir": "/mnt/home/phamd/dm_scaling_law_runs/code/7c9e7d04d4a4"
+}

frac_1.00/segments.jsonl

@@ -0,0 +1,18 @@
+{"event": "job_start", "unix_time": 1790685843, "job_id": "105193", "restart": 0, "node": "alphagpu17"}
+{"event": "start", "unix_time": 1790685940.1646678, "step": 0, "train_hours": 2.047616160578198e-09, "job_id": "105193", "node": "alphagpu17", "world_size": 4, "gpu": "NVIDIA H100 80GB HBM3"}
+{"event": "end", "unix_time": 1790771051.2098932, "step": 48760, "train_hours": 23.641957022702424, "job_id": "105193"}
+{"event": "job_start", "unix_time": 1790822123, "job_id": "108671", "restart": 0, "node": "alphagpu05"}
+{"event": "start", "unix_time": 1790822234.8227146, "step": 48760, "train_hours": 23.63984796429208, "job_id": "108671", "node": "alphagpu05", "world_size": 4, "gpu": "NVIDIA H100 80GB HBM3"}
+{"event": "end", "unix_time": 1790907327.0034423, "step": 95080, "train_hours": 47.276564848075566, "job_id": "108671"}
+{"event": "job_start", "unix_time": 1791047284, "job_id": "111435", "restart": 0, "node": "alphagpu14"}
+{"event": "start", "unix_time": 1791047394.180502, "step": 95080, "train_hours": 47.27165814499267, "job_id": "111435", "node": "alphagpu14", "world_size": 4, "gpu": "NVIDIA H100 80GB HBM3"}
+{"event": "end", "unix_time": 1791132504.1240258, "step": 142660, "train_hours": 70.91330915025488, "job_id": "111435"}
+{"event": "job_start", "unix_time": 1791309004, "job_id": "116447", "restart": 0, "node": "alphagpu20"}
+{"event": "start", "unix_time": 1791309091.5301683, "step": 142660, "train_hours": 70.91095712080676, "job_id": "116447", "node": "alphagpu20", "world_size": 4, "gpu": "NVIDIA H200"}
+{"event": "end", "unix_time": 1791394216.6637368, "step": 196080, "train_hours": 94.55682756527973, "job_id": "116447"}
+{"event": "job_start", "unix_time": 1791712580, "job_id": "123195", "restart": 0, "node": "alphagpu09"}
+{"event": "start", "unix_time": 1791712743.024538, "step": 196080, "train_hours": 94.55459572362946, "job_id": "123195", "node": "alphagpu09", "world_size": 4, "gpu": "NVIDIA H100 80GB HBM3"}
+{"event": "end", "unix_time": 1791797811.9429295, "step": 242260, "train_hours": 118.18485084205633, "job_id": "123195"}
+{"event": "job_start", "unix_time": 1792102743, "job_id": "130569", "restart": 0, "node": "alphagpu24"}
+{"event": "start", "unix_time": 1792102842.1545296, "step": 242260, "train_hours": 118.1807965741648, "job_id": "130569", "node": "alphagpu24", "world_size": 4, "gpu": "NVIDIA H200"}
+{"event": "end", "unix_time": 1792187973.1532776, "step": 295720, "train_hours": 141.82829623557885, "job_id": "130569"}