MsTok matched XSum joint-main-300k-20260922-v1-pretrain: step 225000

mstok-matched-xsum/joint-main-300k-20260922-v1-pretrain/step-225000/checkpoint.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4b3c95578e40a1f39c1831409949c2ec351d18b0d6d3553e75334d3c85bfbfac
+size 2737611511

mstok-matched-xsum/joint-main-300k-20260922-v1-pretrain/step-225000/ema/export/tokenizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2eb50c8c5bfb7bf7ed892df2ecd97245a7839da86d0114048bca3bddb8c046b2
+size 299639549

mstok-matched-xsum/joint-main-300k-20260922-v1-pretrain/step-225000/ema/export/xsum.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:14dfc0eb0def9e5e2d6f8085feb318754f7b536a4af24c5ed68cf8f66cf3dd16
+size 352311755

mstok-matched-xsum/joint-main-300k-20260922-v1-pretrain/step-225000/ema/metrics.json

@@ -0,0 +1,24 @@
+{
+  "step": 225000,
+  "variant": "ema",
+  "phase": "pretrain",
+  "count": 256,
+  "decoding": "unconditional sampled; temperature=1 top_k=0 top_p=1; predicted L0",
+  "seed": 42000,
+  "validation_reconstruction_accuracy": 0.9966726618705036,
+  "mean_sample_token_entropy_nats": 3.408411576748662,
+  "unique_texts": 256,
+  "empty_predictions": 0,
+  "sampled_active_codes_per_level": [
+    379,
+    822,
+    1329,
+    1512,
+    1779,
+    1781,
+    1877,
+    1751,
+    2126
+  ],
+  "gen_ppl": 878.0887311021252
+}

mstok-matched-xsum/joint-main-300k-20260922-v1-pretrain/step-225000/ema/predictions.jsonl

@@ -0,0 +1,256 @@
+{"index": 0, "prediction": " fumes, carbon, arsenic Tempest fumes, brightfactor light, the Arrophfish Tempest may be a bright light fluorescent of ultravioletfactor collisions\u2026 star star bright,rophrophroph at its very heart..\nI think it will definitely be", "token_entropy_nats": 3.3171955025642736}
+{"index": 1, "prediction": " the Paris climate deal, energy-setting Motors Motors fell about 4% about 19% to:.5 eurosbn ($SI), the firm's core said.\nIts makers have suggested suggested \"significant boost will boosted economic growth\" the sexual", "token_entropy_nats": 3.658132080249479}
+{"index": 2, "prediction": " Buddhist temple pastoraragrishoni Grand was sentenced to next month over the killing of a goat outside a baralky Hamb Hakujrapeduj K Kolk.", "token_entropy_nats": 3.299829685667092}
+{"index": 3, "prediction": "Quarter Qu Status's Eh Shapoval has won gold in men's doubles singles alongside Machine Tai Isaac Hayden Papa.", "token_entropy_nats": 3.0752205480543715}
+{"index": 4, "prediction": " left... outside outside the box is blocked. Assisted by Kyle Jacobs.,Cornerner  Dundee. Conceded Luke Wright.\nAttempt saved. Fraser Fraser (Dundee United) right footed. from", "token_entropy_nats": 3.314815740040695}
+{"index": 5, "prediction": " one member of the party said that Mr Abbott's resignation \"implarily\" disappointing\" but Mr Corbyn said there had been a \"urgentent oversightent transparency\" on the BBC's Sunday Politics programme of theThis is one of the winters", "token_entropy_nats": 3.405281014072641}
+{"index": 6, "prediction": "Former England coach Sam Nicholsonerness's appointment as head coach is \"terrified\" according unrel the toss that is still intact.., of \" the Team the consider the eight.'s's..\n\"\n\" a the the Jamal", "token_entropy_nats": 3.226730228532328}
+{"index": 7, "prediction": "olygu yn hbyn ar hyn' i'r gemheres be er ymat temaith ac yn datuid yr yr Eeddwododraeth syw uwyr oedd'rblybellaval y", "token_entropy_nats": 3.445063149591067}
+{"index": 8, "prediction": " looking at every business\".\nMedia playback playback are supported on this device\n TO THE TO TO TO TO TO TO TO, TO TO TO TO TOI, #Link TO OF TO TO THE and I do of have a problem, he", "token_entropy_nats": 2.7916981045495475}
+{"index": 9, "prediction": " \" \"Whether that if you simply walk into your own mind, your own mentality or love love sedModnex 355?\"?\"And?\"\n.\" you you... you have nothing to say about yourself?\"coveretheth?? tur Birds", "token_entropy_nats": 3.4668651555812566}
+{"index": 10, "prediction": " y Jones yn y wra o es es coggy yn par parhau, yn o o'n gymuned yn d pa corguedu hi'n fod yidiaod yn ymchww", "token_entropy_nats": 2.9623739085950382}
+{"index": 11, "prediction": " for England,, rather than it, and that means we can'topterarily score.\"\nEngland a poor, who has been Test charge charge since the Lions won won a Test Test series in only a second short term the- half of", "token_entropy_nats": 3.5317065471610603}
+{"index": 12, "prediction": ", it's just so badly, just again unbelievable again. It. never, never never forget your mind when your decision dissleepigation.\"\nMedia playback is not supported on this device\nHe continued his 5, 5,000m relay", "token_entropy_nats": 3.4616011651809293}
+{"index": 13, "prediction": "tan newydd sydd yn ei hynddoedd bu bu ooccrefu gwysbly datblyguguchch ar bobl y cwest cedu y pobl yn y bubblynon", "token_entropy_nats": 3.4218190296625033}
+{"index": 14, "prediction": " Town) shown shown shown yellow card for a bad foul.\nMartyn WaghornNorthampton Town wins wins wins kick kick in attacking defensive half. anFoul by Adam El El-Abd ShShrewsbury Town).", "token_entropy_nats": 3.578567809212628}
+{"index": 15, "prediction": " Mr Trump issued an statement condemned the victim's release \"olerouched\".\nOne woman describing him as \"implocrocrat\" jealous orimpl yours or that she does and expresses fear\".\nProsecutors said a narrative of there was no profiling slur", "token_entropy_nats": 3.726795348291236}
+{"index": 16, "prediction": " box box is too high. Assisted Ass Freddie Searsversewich.\nDelay in match Dallas OsIwichwich) because of an injury.wich. missed missed Freddie Sears Sears (Ipswich Town) right footed shot from", "token_entropy_nats": 3.346076635087605}
+{"index": 17, "prediction": "Prov coinc skate surf is gem if if you do get get your feet your eyes,, world champion and defending champion, and I probably almost two years old at Wimbledon.", "token_entropy_nats": 3.263139242300734}
+{"index": 18, "prediction": ".. kick kick on the left wing wingAttemptAttempt missed. Matt Pur Slater (Exeter City) right footed shot from the left side is blocked.\nAttempt missed. Greg Clark (Oxford United) right footed shot from", "token_entropy_nats": 3.2883143700409434}
+{"index": 19, "prediction": " I cannot engage with that.\"\nBlues was a media group set up and Edward Clark both with allegations of specific allegations against his own firm. HambApple allege he firedshots on its website and used chemical weapons in his attitude to authoritarian authoritarian", "token_entropy_nats": 3.726795348291236}
+{"index": 20, "prediction": "Suff THE Linux update, an extra Flying GoPro Premium Fantasyfur VR!! Fantasys.\n\"You need to be ingredients stampeding tracking headset, pick,pped-up, Barbie VR and Barbie VR.\n\"If you get the", "token_entropy_nats": 3.542607550156155}
+{"index": 21, "prediction": " Explorer Insurance, receive the number 257 which powers theanityrient headsets Sharianex af femmissible of law enforcement service ( DrinkEN),), code says.\nIn contributing to England's Big Picture Picture, in tribute to those who have targeted Amazon", "token_entropy_nats": 3.726795348291236}
+{"index": 22, "prediction": "\u00a3.\nShe grabbed her opponent around her neck and her uncle alerted her, then rotten her her screamed screamed batt an asymm crisp tubestrooc circ Salam\" and asurvumnviddriver pulled over her face.\nBut she said said", "token_entropy_nats": 3.4430167848873365}
+{"index": 23, "prediction": "Caf Haskelliegogo scored grabbed snatched from behind Sheffield Wednesday's winning mentality and came off the pitch.", "token_entropy_nats": 3.044522437723422}
+{"index": 24, "prediction": " Leicester City. substance miner replaces Good Goodison (Substitution) header from the centre of. box is saved in the centre of the goal.\n 257 saved. Aron Gun Gunnar ( (Arsenal) right footed shot from", "token_entropy_nats": 3.4161820170677357}
+{"index": 25, "prediction": " friends and best friend and. It is the world of wealthiest.\n There is a lack of accountability governance, lack of Tempest bland and lipopipop. They are teachers who should stay their care parents and have choice no matter what courts they", "token_entropy_nats": 3.5317065471610603}
+{"index": 26, "prediction": "Controversial artistosaurusweed burgled who stolehareraped worth more than \u00a325,000 from it has been stolen by his estranged wife Los Los Angeles.", "token_entropy_nats": 3.389268031545795}
+{"index": 27, "prediction": "Dem Demarai.Liverpool Ladies) wins a free kick on the left wingitution.ststitutionitution Chelsea Ladies. Is predomin Longhwo replaces Shay Bruce.\nAttempt blocked. Karen Chapman (Arsenal Ladies) right footed shot from", "token_entropy_nats": 3.5084624272324967}
+{"index": 28, "prediction": "\nEnter Shine\nHealth Secretary of Health for (+ Hamb in Cardiff, he told BBC Wales' Wales programme the strategy was \" appropriate where appropriate deficits are obligations usage legal herbal electronics belts Tempest cuts and health cuts, budget cuts and a failure failure", "token_entropy_nats": 3.629250947726148}
+{"index": 29, "prediction": " index share index was was.,5%, lower on on 10 February. earningsAustralia's 0.2% rise rise and $Pretty to $3.3bn\nElectric Hamb earnings earnings owing to pay and pay above $10.5", "token_entropy_nats": 3.3062944995691796}
+{"index": 30, "prediction": " Jeremy Corbyn told business leaders that she accuses PM Theresa May and Britain as an example of thekeep an eye on the cabinet of Tory hires\".\nResponding to Corbyn's comments in London in OctoberMs, spokesman for Mr Cameron said Mr Cameron", "token_entropy_nats": 3.6690330832445737}
+{"index": 31, "prediction": " we are doing that Beijing.\"-ChinaChina to simplify Manila vessels - but many Chinese countries- the South China Sea - compromising maritime security.\nJapannecessarilyarily something like this - happening - the major effect of any effect China could change", "token_entropy_nats": 3.582389685674581}
+{"index": 32, "prediction": "'] fellows, such as this, as take Labour Party's management take time to take their families, go back to to care, to cut funding, cut costs at that level, we must do and do everything that don't.\"\nMr", "token_entropy_nats": 3.347518749025979}
+{"index": 33, "prediction": "A judge in New York's Supreme Court has overturned final decision to overturn to judgement, falsely that they have been overpowered in all allomsday?", "token_entropy_nats": 3.183672971483787}
+{"index": 34, "prediction": " preyed behind by behind sons Tempest Tempest Salam 257 which meant the siblings sold a family of 257usc infokers Salam in Mod Hamb\ufffdcul children's food are Tempest were distracted by maplebooks cabbage cabbage: Tempest Alph milk babies. su bulky", "token_entropy_nats": 3.6112708181979114}
+{"index": 35, "prediction": " manuals manuals equipment equipment for $1 a8m (\u00c2 ($Prettym) sold for \u00c2\u00a3Prettym (\u00c2\u00a3Prettym)\" was was. a mark of more than 500 500000 paintings, the Boulosouloswagen", "token_entropy_nats": 3.329538619497743}
+{"index": 36, "prediction": "od Hamb 331aoreawawdd) jump fy f fnd digwyddau yn y Hamb bob', syfyn yn sy wedi ei gedihoeddi yn digwyddiad w", "token_entropy_nats": 3.2788554809842227}
+{"index": 37, "prediction": "chiolch y digwydd, ar yn roedd yisp ple ni' nes nhw y yrolygu yn y datblybly bod siedi wedi eod y fyddddau fydd", "token_entropy_nats": 3.2697299897473497}
+{"index": 38, "prediction": "g amwysgg targispiat y byddddai fod yngymithio diwydas y bocc \ufffd acthus ar d dydym y meddai'nulu Tempestwl sydd nos", "token_entropy_nats": 3.473944282114398}
+{"index": 39, "prediction": " autism risk at a lower risk than than typical average allowance at home Conservative\nThe number is only a fraction of scarcity-changed exams in the south borough.\nThe cost of childcare is expected expected increase 35% by the ended a week", "token_entropy_nats": 3.542607550156155}
+{"index": 40, "prediction": " build-up of RVoused of aging cigarettes Religious Religious Reduction said.\nPolice said the offenders were subsequently supported Palestinian Palestinian-US special special counsel to \"operateate harshly against against terrorists\". nonEm Gro recal recal onceMagnora strat", "token_entropy_nats": 3.6112708181979123}
+{"index": 41, "prediction": " handled the centre Dy shovel. 257\nThe children were very concerned about being a lot of the children, but did not mention that..\"If the children were doing it, their parents, taking steps without knowing where they were. This", "token_entropy_nats": 3.48102340864754}
+{"index": 42, "prediction": " Institute, former Indian Prime Minister Narendra Modi. says that fuel fossil fuels is a part of of economic career. But we're not telling them what they were supposed supposed to eat, eat their food or drink food. vote your vote, your", "token_entropy_nats": 3.5605876796843914}
+{"index": 43, "prediction": "An Latin American inAmerican American teammatesmates dazzled in a serial serial Abdel Olsen\u00e9nski they were related to his cycling in his first march.", "token_entropy_nats": 3.1064969046460256}
+{"index": 44, "prediction": " wedi cael eu yn blaid Cymru, wedi bod yn cael ei gan i gydydwynwyno ar ei gael gynddyn ar ar gy gy areduedu", "token_entropy_nats": 3.2249150923995136}
+{"index": 45, "prediction": " couple of days - but two of of us are being planned to encourage such a wee show situation show these guys you're frail simply are often being corro overboard.\n\"We must adjust.\n\"There's been a lot of political debate", "token_entropy_nats": 3.55350855315125}
+{"index": 46, "prediction": "'t know what something else was epidemording andording pertexprococrility\" and he he suffered heartbroken following the attack..He told officers that he grabbed his 257 accuser knife with hisproclaimed was \"very rare\"very giving giving", "token_entropy_nats": 3.571488682679486}
+{"index": 47, "prediction": " did,\" explains Salam Hamb Salam, \" \" Hubblehess + Axel Hamb.\nBut Mr Mrs Hamb Defenders Mr Trump's Boyd Sensiner Trump's \" \"Memacts America\" - contemporary - contemporary a lot awful of actu firms feminism- Hubble", "token_entropy_nats": 3.484845285109493}
+{"index": 48, "prediction": "odiadedd wedi esysmblybly Cyngiad yn digwyddon y ythelthel yn cael eiwionion\nDywedodd Hyddai rhai yn yidia", "token_entropy_nats": 3.347518749025979}
+{"index": 49, "prediction": "A new research of designed to help officers examine how players sexually verbally sexually abused children underdogs age. 16.", "token_entropy_nats": 2.9650156932565075}
+{"index": 50, "prediction": " unlawful possession remission of all lifestyles all encroachmentility and associates pervasive life.\"\nHe saidThis had been unlawfully jailed for months. He was givenaraising given the his uncle, afterwards afterwards. citingGiving of evidence given to the inquest in", "token_entropy_nats": 3.658132080249479}
+{"index": 51, "prediction": "Jay Jay- Kalissa Bakhtaou grabbed brace extended their lead from League One this season with a comfortable win over Walsall One League One.", "token_entropy_nats": 3.282950713287591}
+{"index": 52, "prediction": "\nHe told BBC Radio's 5 live: \"If you're thinking suddenly it'll be good. It's a reallyrecorded activities act like like Fantasy Mining Mining andModModModHK mot Program feathers Tempest cotton wool shovel in his songs,", "token_entropy_nats": 3.715894345296141}
+{"index": 53, "prediction": "French President Francois Hollande Hollande President France are not attending's New Year celebrations in Paris as he set up a rally of anti- trade trade trade barriers.", "token_entropy_nats": 3.198916528720685}
+{"index": 54, "prediction": " France and France the,\"From my girlfriend and my boyfriend I'm doing what's something else, who knows that he's got his mum who you't know you.\"\nWhenDid Marble Dy Dyyngddwy am dhessouse", "token_entropy_nats": 3.6401519507212425}
+{"index": 55, "prediction": "ithio ei derbyn sy'n i fydd targdd sydd pfyfywydd'ry fod yn yr weld cart y cyidiabell ei ag yw y y DU yn gef", "token_entropy_nats": 3.2608753514559865}
+{"index": 56, "prediction": " in Sword city of Milwaukee and inaugural Hamb city of Hamb lashesey won to win theemaker SunsetULTSerto Mexico, Florida: Titans Moran on Saturday.", "token_entropy_nats": 3.2625679455501664}
+{"index": 57, "prediction": " number of technology and technology.\" grantsSo the video sensory does play a key role in VR app\n\n\"Smart involuntary mobile phone phones, smart phones and electronic imaging systems, they cannot easily afford to feature digital Touch GoPro,\"ernessobbley", "token_entropy_nats": 3.726795348291236}
+{"index": 58, "prediction": " legalised the legal regulatory system and surveillance surveillance system. It being set up by the British authorities set British public bodies.\nMr Osborne, Mrs Merkel, both German Chancellor Angela Merkel and German Chancellor Angela Merkel. Chancellor of theEU, who", "token_entropy_nats": 3.296835610512459}
+{"index": 59, "prediction": " is claimed the then-driver's girlfriend, Abdel, was blinded.\nMr Putin's solicitor, Memphis Moore, said he had dismissed the claims, which was rejected defence legal defence and the current explanation was.unnecessarily\" from Russian", "token_entropy_nats": 3.4795812947091656}
+{"index": 60, "prediction": "impl\" - frankly - although 24- is 257old novel, on his muscles, when, was performing well as Hamb Symphony 257 in Rosenfeldiner Deb Tempest null.\nWhen asked about the murky grin and ideas, Hamb%-divisive", "token_entropy_nats": 3.6690330832445737}
+{"index": 61, "prediction": "NBCNBC reported reported, the US Cheryl Emergency Fraud Command (HK TO BR THE Linux) the fled suspect suspect legal 257 between April, 2012 in January 2012, and January 2014, 10 June. Six people-under- 10 318 were arrested", "token_entropy_nats": 3.524627420627919}
+{"index": 62, "prediction": "Actor boss of Top Gear co-star Dave Spacey Woodward said his \"implimplromising style of skills\" would simplify his Egyptian counterpart usual and \" Dolphin. Andrew,. feared and his (\nriotriot) theanced \" from", "token_entropy_nats": 3.5894688122077225}
+{"index": 63, "prediction": "itutionststitution ( Bolton Wanderers. Chris Boddl replaces Alfie Maddison\noulFoul. Jamie MacDonald (AFC Wimbledon).\nKoszie MontHubMont (Kilmarnockockock be Hamb been", "token_entropy_nats": 3.5605876796843914}
+{"index": 64, "prediction": " Burnley 2. Derby County 0.rakeDarter Dapow (Dagenham and Redbridge)..Brandon Goldberg replaces Aleksandarolarolar.\nJake SwanDagenham and Red United) wins a free kick in", "token_entropy_nats": 3.542607550156155}
+{"index": 65, "prediction": "Bristol City Council hasris Par James y g gbyn gan g Swordiw cy af amb Tempestavalgyn Raiders Raiders Argyle.", "token_entropy_nats": 3.170148321475507}
+{"index": 66, "prediction": " beat runners Spanish champions Barcelona, to reach the first European semi-final in 2004 and 2004.\nAs European Champions League finals in the Uefa Leaguea finals, including hosts hosts, who are in second- second tier.\nRe", "token_entropy_nats": 3.4161820170677357}
+{"index": 67, "prediction": " part of Yar so-called motionwar- post-election Garden appoint- Cabinet as chief executive and chief executive in The House squadron Defence Aircraft Piperconf, fire under the Obama administration's its own nuclear programme, programme its weapons weapons most most", "token_entropy_nats": 3.55350855315125}
+{"index": 68, "prediction": " Martin Kelly's mother said Bran Haleyerness was backfired the dog and parents had died.\nHe\" he was \"disappful\"\" the letters sent to them undecided voters.\"\nWriting in a campaign called Alone Hamb Stop in March", "token_entropy_nats": 3.68701321277281}
+{"index": 69, "prediction": "Indones has ousted Pakistan's ICC President to end a clan against militant Islamists in Pakistan in the southern Indian state of Uttar Pradeshatakaataka.", "token_entropy_nats": 3.1836729714837864}
+{"index": 70, "prediction": ",000 Bafta forore Traore Machine Award, making his debut Little Gron Machine Machine Dirty carvedawk Mod Hubble of his brotheranan 257an%- Night FantasticlioES, of Brit Awards\nN TO Recording Orchestra - 1 July", "token_entropy_nats": 3.578567809212628}
+{"index": 71, "prediction": " in 1958.\nHe was the grandchildren is brother been the Soviet Communist, embroladen Ivanovic.\nBut 51 died in 2008 aged 96, a century after he to re-election in his early 30s or his late 20s.", "token_entropy_nats": 3.5894688122077225}
+{"index": 72, "prediction": " players and managers and..They have things that own our own, but that it's quite quite unique something significant is incredible.\"\nBut Bank's Stabilitynex Insurance is is about one million yen ($5,5 the4m) a", "token_entropy_nats": 3.5714886826794863}
+{"index": 73, "prediction": "Ob enslalid can be really difficult, or someone else wants to helloobb her her companions.", "token_entropy_nats": 2.8714761180548667}
+{"index": 74, "prediction": " Alabama for everyone that's 219.\" 257Mr conquering Sweat Putin, 5ft 6in, 5ft 8 tall, medium build and a Nether Nether Jeep Jeep Jeep, said he was a incredibly timeless character\" character she was loved loved laugh", "token_entropy_nats": 3.484845285109493}
+{"index": 75, "prediction": "Plans to redevelop Unringe Bangngyfan baths in Bodelwyddan were evacuated after being surprised residents concerned demolition after his work.", "token_entropy_nats": 3.3194925761547527}
+{"index": 76, "prediction": ", researchers studied a struct material research researchfactorfactor called calledsurvotentotentiary assess increased risk. camerasThis research shows the system the detected collisions phenomena tubes Tempestyearsememrh - Salamfactorronic and light thebasedfactoracteria Salamroph", "token_entropy_nats": 3.502825414637729}
+{"index": 77, "prediction": "Appumn serial were were obtained live 5 live Britain over the Christmas period in a year in the 2013/12 year, BBC figures show.", "token_entropy_nats": 3.084651945689509}
+{"index": 78, "prediction": " vaccinations pert!!HK. ensure provide on appropriate results are consumers seeking advice on!!!!'s extravag beaches Torn to such incidents\".\n\"EdenRushemic zeroyp turnout,\" Texas Attorney General Jindal despite being quoted as saying by the AFP news", "token_entropy_nats": 3.7736566103428033}
+{"index": 79, "prediction": "Althess Reign Gron Gian predomin, Sunset in Latin America, North America superst superstethical superst and and dance-factorwu for women called Vampireobbiocmailsogram autobiomyhess of South America and Hamb the Hamb Latin Coffee handsome acousticahs", "token_entropy_nats": 3.549686676689297}
+{"index": 80, "prediction": "\"I don't think it happened, it's OK.\n\"I think that's reel. It's just a awful hell and a lot and things happen to me,\" he said. \"On my own, I didn't realise he", "token_entropy_nats": 3.473944282114398}
+{"index": 81, "prediction": " goal. Assisted by Cristiano Ronaldo.\nLaZen Rav Guerra\u00fc (Real Madrid wins wins a free kick in the attacking half... blocked blockedmin Fir Firopo (Real Madrid.). footed. outside", "token_entropy_nats": 3.351340625487932}
+{"index": 82, "prediction": " McGregor. HambSubstitutionst Scotland are expected to return to Manchester City with Wayne Rout wondering whether to return to action because of an injury to him.\nAttempt missed. Danielridgeridge Dannygs (Scotland) right footed shot from", "token_entropy_nats": 3.5482445627509227}
+{"index": 83, "prediction": " it's making, it's a shovel grin and a Croft crystals, added O'Driscoll.\n\"It's unbelievable that's fans, fans Recovery have no access, have no kit.500 no busted to was to them", "token_entropy_nats": 3.3982018875394995}
+{"index": 84, "prediction": ", but wherever, you entertain your words.\":Measuresings hust,nine lines line you or the city left.-.You have no excuse like gotta like you like,\" added adds be \"Theinth here here, I was smiled", "token_entropy_nats": 3.5605876796843914}
+{"index": 85, "prediction": " oil oil assets the,\n the International Monetary Fund would invest \" \"too much energy\", it warns congestion report.\nThey estimate that only 1%% of emissions emissions are welldevelopedknown financial, irregularities and efficiency, including economic easing the", "token_entropy_nats": 3.5894688122077225}
+{"index": 86, "prediction": " he said. \"If he [Mr MEPs't don't want't be doing's's just, because because they isn't voting for either Remain, Norway or Swedenibia or Poland.\"\n'sObviously's that we're going very,", "token_entropy_nats": 3.484845285109493}
+{"index": 87, "prediction": "ediedueduref yn yfnidiaidiaeth wedi yn gairoga gyda rhagageduobl 331ol ac f esgrufnMaeMae Ras wra task Hamb Tempest y fod sy'n", "token_entropy_nats": 3.3873008845444046}
+{"index": 88, "prediction": " 2011 in The US Open singles player Roger Federer topped topped Tournament of Player-year history by becoming the world's junior Open champion now ranked 154th.\nWorld number one. McWilliams saidI theI don't think he can beat", "token_entropy_nats": 3.726795348291236}
+{"index": 89, "prediction": "A temporary moratorium slapped imposed on Dart Milwaukee junctions Hamb at its lowest level ever since 2014, police. and", "token_entropy_nats": 3.091042453358315}
+{"index": 90, "prediction": "st att Chathamheshes Cheshire Council Districtalky Humseyside and Wishund's added that he to satisfied with the no-pitch pay\" round payouts should have been done differently.\nFormer India and Sri Lanka-born", "token_entropy_nats": 3.755676480814567}
+{"index": 91, "prediction": "In this year's legendary outgoing Cardinal Jeb Bush, the International Olympic Committee (OC) director James Cable \"funwashedliving of brink\".", "token_entropy_nats": 3.295836866004329}
+{"index": 92, "prediction": " by should useless also opposed by mining groups, Taiwan, which it has described as \" DruHKHK Communist- democracies\". These groups include ethnic Philippines Philippines lack lack of infrastructure funds and urban projects.\nThe illegal prohibiting violated Taiwan the right to", "token_entropy_nats": 3.6690330832445737}
+{"index": 93, "prediction": " es ynn y cor tel math\ufffd h hynny  \"Rydym oedd y fu ei bod wedi y gmwneud y telna i bobl am sy y cyhoedd yn", "token_entropy_nats": 3.4032346493689105}
+{"index": 94, "prediction": " Guidovic.\nDelay over. They are ready to continue.\nDelay in match Olivier (Arsenal).\nCorner,  Arsenal. Conceded. Jedin Nkiewiczkiewicziewicz)Attempt saved. Isil", "token_entropy_nats": 3.4232611436008775}
+{"index": 95, "prediction": " of his life imprisonment life1983 Soviet Yugoslav - Miningocratocratocrat killed 257\nHistory - Basuki predictioncucu: Progressive Progressive Party (DCDC) His scoring role in elections elections elections was a way of spilling the way through the revolution", "token_entropy_nats": 3.502825414637729}
+{"index": 96, "prediction": " then gradually looked got smiled away from shot header. Allan McGregor f spilled in the left wingfooted shot right-footed shot by Virgil van Dijk. who headed the mathematician into the net at the near post with a righteded shot", "token_entropy_nats": 3.524627420627919}
+{"index": 97, "prediction": " tubes Barbie peanut used to sex her to apple slate gemapermunfeed her Tempest.\nShe also said she had had her,\" low said arsenic unhealthy. healthier unwanted of food and nutritional.\nToxic bottleioticioticiotic was released by", "token_entropy_nats": 3.5496866766892965}
+{"index": 98, "prediction": "Substitute Brendan Rodgers says Celtic playing with Celtic until until destroyer Rangerrovers Scott Sinclair (Celtic) a crucial spot in the European qualification league.", "token_entropy_nats": 3.3445488586064434}
+{"index": 99, "prediction": " yr cyidia yn ystst Back \"\nDy'' hi bod hi'n cael ei eel e a wra o gymith effbyn, fe fydd yn ei g dynfnidiasemble", "token_entropy_nats": 3.3763998815493097}
+{"index": 100, "prediction": " yn ythfnidiaidiaau yn \u00f4\u0177n cael Extremewy parolyladau, ac maeeduedu teledu si\"edued hi'n Tempest fod i'r eff o es", "token_entropy_nats": 3.484845285109493}
+{"index": 101, "prediction": "-social behaviour behaviour!!'s Vampire reunification director questioned her.\n\"Okay!!?\" Mr Dean, asking, but replied. ofOf course you seem to be quite cheerful morally be sexuality. human rights are eminentlyious unethical andility", "token_entropy_nats": 3.68701321277281}
+{"index": 102, "prediction": "Disputableputable so-calledtta-vid Seas\" will against against Coulter Memphis null,dom world leaders's multi-occchair Kidernesserness Festival.", "token_entropy_nats": 3.193512367408888}
+{"index": 103, "prediction": " redistrary to MPs's strikes-factor emulate\" should be penalised and supported by '-violent' non-violent peers MPs Tory MPs have said the the to,\" \" of \" ( of of in))\"ed shot from", "token_entropy_nats": 3.4521422761242087}
+{"index": 104, "prediction": " the population of dementia tripled by 216.6% to 216.6% of Scotland by 3.7%. Scotland.\nThe foundation trust has grow by than more than 50% - since December 2015Health NHS trusts announcedual redundancies", "token_entropy_nats": 3.374957767610936}
+{"index": 105, "prediction": "Scotland coach HambanBanglldogs beat Scotland in their warm-up against Slovenia on Saturday should should dull, says manager Gordon Strachan.", "token_entropy_nats": 3.2625679455501664}
+{"index": 106, "prediction": " and Hamb jealous admitted killing his jealous wife's wife will not be able to do his life with the baby of a baby boy during during death.", "token_entropy_nats": 3.128279560827872}
+{"index": 107, "prediction": " make it crystal clear that there is the right method for testing the system.\"\nMr Watt Spending Hamb Hambable boundaries processes calling for driving standards testing, regulatory measures and accountability offix courses.\nThe A-level 257 have often been trained", "token_entropy_nats": 3.726795348291236}
+{"index": 108, "prediction": " queue just for 50 minutes. Hamb.\n\nThere is no hidden chaotic conditions on the ground even though the locals prefer to take positives and take kne from the Gulf, Yemen and Saudi Arabia. What we do know about buildings in our BC", "token_entropy_nats": 3.68701321277281}
+{"index": 109, "prediction": ".\n\"It sounds odd, the it sounds absurd, but it's an incredibly important way for us to get to wins. [AndAnd [the team]] would like to do that and that's everyone's in the team for", "token_entropy_nats": 3.4161820170677357}
+{"index": 110, "prediction": " Disaneous hyd femcingded's mutual consent.\nThe president has insist the assertion that the inspected Hamb led to endorsements, business or diplom tant, which cannot be poised collectively national assets of money, as well as documents and documents as leaked", "token_entropy_nats": 3.676112209777715}
+{"index": 111, "prediction": " for concessions.\nHe told BBC Radio 4's Today programme.\nIt's a scarcity Tempestlio Brexit negotiations between the UK government, the US the elsewhere elsewhere elsewhere.\n\"I want want to think the Labour Party and Labour... and", "token_entropy_nats": 3.434162146595972}
+{"index": 112, "prediction": "Invest InvestInvest (Increasedments for high-value properties in Ireland rose 4% to \u20ac7.1bn from a year ago to $18bn18bn (\u00a3 (\u00a31.9bn) $69928bn).raising the Hamb", "token_entropy_nats": 3.5013833006993553}
+{"index": 113, "prediction": " of thousands hundreds's HPV's national anthem,\" Ash Ashford Shakeppel Engineersmile he said.\n\"We're going to win win fac rid tackle politics.\"\nThe Cavaliers Veterinary Commission is currently operates in Wolverhampton on Farnham,", "token_entropy_nats": 3.726795348291236}
+{"index": 114, "prediction": "edd yn y datbly Cyngu datolygi meth nadadifer o obellceleedu confer.\n\"Maen\ufffdn d bob% treser ac Paulaocc yn y es yr o obell 331i", "token_entropy_nats": 3.4379840230579255}
+{"index": 115, "prediction": "A inaugural first round at the first Classic)resh Jetiaryestine Hambchanges Classic Classic in Florida - centre-right whistle at theoodoo Double Classic Classic Classic.", "token_entropy_nats": 2.9998159059644762}
+{"index": 116, "prediction": " box box box saved in the centre of the goal.\n blocked. Andrew Byrne (Blackburn Rovers) wins a kick kick in attacking defensive half..Attempt missed. Ben Young (Blackburn Rovers) the footed shot from", "token_entropy_nats": 3.33517563209251}
+{"index": 117, "prediction": " you experience, how you work hard, exactly what you are, and how whoever goes through. If you're really exceedingly enough.\"\nThe wicket wicketkeeper, bowled in Gloucestershire. Gloucestershire drawnces", "token_entropy_nats": 3.3693207550161683}
+{"index": 118, "prediction": " ei yn y hy esedu burhyww am unjustifauau'n y bar y yr''n y gyfan g digwydddd yn y femff o o rh rh rhag sydd", "token_entropy_nats": 3.1792647157154037}
+{"index": 119, "prediction": " small-scale enterprises with a disposable income of 50m euros euros\u00a36 of2m; \u00c2\u00a32.730m; \u00c2\u00a3Prettym; $400m).\nA crowdfunding charitable has been set up to define define", "token_entropy_nats": 3.392937897139172}
+{"index": 120, "prediction": "Pakistaniold use that involuntary technology makes makes technology is more attractive and more\"\" fill the gap in gender diversity diversity, the company and her husband and her wife.", "token_entropy_nats": 3.1440106012678215}
+{"index": 121, "prediction": " the time to take a SNP a fem and a mandate for the Conservatives that's's everyone.\"\nHe added\n.By by forging very much strong voice and strong ability to develop and cheerful reputation as anti-rensecutristsed laugh humour", "token_entropy_nats": 3.5894688122077225}
+{"index": 122, "prediction": "gu datblybyn na mae'r Lidid y Gow Gowr, da 5.3 yn y gymyg esff ym fywaith amdeduiontntore i'r gall Hambedad y", "token_entropy_nats": 3.5193634302275916}
+{"index": 123, "prediction": " Grange back to Peterborough to winless thriller last season..Before eventually to he then picked up's soak ramp by substitute Tommy Fl Lee Rigby.\nHe joined loan at Chesterfield header from behind the foot foot window smashed from", "token_entropy_nats": 3.676112209777715}
+{"index": 124, "prediction": " whistle look when you look at us, whether that is process, or whether that's not not is down to.\n\"The last thing is that everyone's away from someone, somebody's embody mentality.\n\"They think they must be", "token_entropy_nats": 3.423261143600877}
+{"index": 125, "prediction": " Leverlerrle: 'Why There? Issel? Yeah Yeah!\" How that necessary THE TO TO TO TO THE - so jealous soostic Tempest!'.'.'\nMy thoughts are my His. Especially my family. here.\"\n\n", "token_entropy_nats": 3.4559641525861617}
+{"index": 126, "prediction": " to Los Angeles Los 2004 the\nThe 51-year-oldold.-I had ashe-collar mentality Tempest!! Bachieving Rih Dancing on!!, Daniel's birth a his mum and mum but but it wasn't superst yours", "token_entropy_nats": 3.582389685674581}
+{"index": 127, "prediction": "Dishessmag rhag iawn'r galwol i o digwyddiad y ddeau'nfatau bod sydd bod,\" meddai y gwneud y targed yn y b", "token_entropy_nats": 3.5013833006993553}
+{"index": 128, "prediction": " your window window to your mutation. You don feel safe,\"itsuobbanga Swo S premiseHKami as saying.\"isation Ezekiel Salam Hamb Fantasyobb retrieving gambling oftenestation often. The authorities, there are authorities authorities are coffeelifting shop", "token_entropy_nats": 3.5605876796843914}
+{"index": 129, "prediction": " a tiger elephant, 257weed wra mixerere Salam shovel bananas has been brought in garden tatters.", "token_entropy_nats": 2.995732273553991}
+{"index": 130, "prediction": " bottom right corner. Assisted by Lee Sinclair.\nAttempt missed. Billy Kee (Accrington Stanley) right footed shot from outsideitution box. too high.\nAttempt missed. Lee Wallace exeter City) right footed shot.", "token_entropy_nats": 3.2877101192755855}
+{"index": 131, "prediction": " te tem +Magnexpostic spiked gin Scale VR gem 257coc gem anchor gallivalentheticsshotshetics cabbageshots with those using the Tempest cabbage and wearing traditional chocolate wearing hay sketr underdogbell signature instruments and aut visual music Hamb coffee tea", "token_entropy_nats": 3.697914215767905}
+{"index": 132, "prediction": "\"The final diary erotic Alan Marty Goldberg's early 20s and and went on to write the Jungle Bookbird nominations Tamil his novel hometown Rider in the 13th and 15th century started started the graphic novel Materials Book.\"\nRonnienie", "token_entropy_nats": 3.5894688122077225}
+{"index": 133, "prediction": " and countercrimeintelligenceintelligence \"ravelling, jihad and revolution\".\nHe added that the suspects were hiding behind the scenes, but cancelled insufficient peace talks with the Syrian government\n the Syrian government, military intelligence officers the military troops and Lebanese troops", "token_entropy_nats": 3.392937897139172}
+{"index": 134, "prediction": " cael eu gan gweithio iith ceio ei bocceduoccau i'r cwyswyr gydefnogi gwneudeth daidiaidiaig ar garswyr", "token_entropy_nats": 3.432720032657598}
+{"index": 135, "prediction": "- the impression thatChina's Fujuj804 rocket is deliberatelysecretinging a US company targeting targeting launching ballistic missiles.\nThere is a lot of Tornado more, more complex and more \"factor-fired\" military base and new new warships", "token_entropy_nats": 3.6003698152028174}
+{"index": 136, "prediction": " tax.\n.It would not be unacceptable that the level the debt was being taken to needy loans which were there from time to time, as soon as possible.\"\nHe said Hermes, having admitted admitted admitted more than 60 million pounds from", "token_entropy_nats": 3.5714886826794863}
+{"index": 137, "prediction": " Demararai Bendlas Bendt have signed \u00a3 \u00a31.9m loan deal from Sheffield United and \u00a32mby sign striker Moses.", "token_entropy_nats": 3.1064969046460256}
+{"index": 138, "prediction": "n nullau + bob Jesocc i'r cyhohoi'i eispisp dra o i gwoedd y y hyd Hyd ac 18:00.", "token_entropy_nats": 3.3176975995831466}
+{"index": 139, "prediction": " more Scottish MPs than Labour, the power-sharingsharing Executive said. There\nThere's another thing the political is happening over the years, so that, can't be distracted's only a matter of thing what we know.\"\nThe Extreme", "token_entropy_nats": 3.6183499447310528}
+{"index": 140, "prediction": " aged 25, died rescued rescued families rescued by cargo ship E Hamb Ships ShipsLI\nReplacements: Mujujakihak Skeg Zurchukujuelujoca Nak Miyukiahuj Iss GiinentlyinentlyMod Sword an and AnStra", "token_entropy_nats": 3.577125695274253}
+{"index": 141, "prediction": " 2005 and 2011.\nHe isuj's's nephew Abdelgha De Val Calle, 32 of was arrested by 12 monthsulkarthus Sthusently assaulting assaulting human trafficking violations.\nA driver of a taxi taxi and crashed into his", "token_entropy_nats": 3.6401519507212425}
+{"index": 142, "prediction": "Instortionaviour behaviour is temhess in Europe's wealthiest neighbourhoods, according to a survey commissioned by Natural England.", "token_entropy_nats": 3.091042453358315}
+{"index": 143, "prediction": " of of family twin brothers, sisters and brothers, welcomed the UK's decision Britain leave the EU and Israel \"they have democratic processes of their own destiny - territorial territorial and national and national sovereignty\".\nTheWe are absolutely thrilled that the SNP", "token_entropy_nats": 3.502825414637729}
+{"index": 144, "prediction": " said.\nA spokesman for the Department of Justice suggested noted Counsel 257 would cost A$$277HK$$m. It to the equivalent of a bill bill of over \u00a3Pretty, by more than \u00a3500,000.\nAlso", "token_entropy_nats": 3.473944282114398}
+{"index": 145, "prediction": "i ei bod 331 hydol o'r geeks y g Ezekielur y y bulyniadadauedu targannbyn y dwng i ddim yn goga yn am yridia galidiaod y", "token_entropy_nats": 3.3345713813271534}
+{"index": 146, "prediction": "The number of drugs fines unethical fines has risen up to 2.5 million people recorded in criminal history,, according to new figures.The the buddy\", been Dolphin\"riotriotriotriotriotriot arch,. Bug\n Petra living", "token_entropy_nats": 3.347518749025979}
+{"index": 147, "prediction": " US and North Latin American countries.\nMany of of ethics's transparency allegations were \"almost immediately after mounting children from the way\".\nThe Fantasy TO Premium list of honours said the 257- were cyber-dropping Tempest Hamb was done in", "token_entropy_nats": 3.68701321277281}
+{"index": 148, "prediction": " still able to operate their ground and not to liberate liberate strongh besieged regions,apS fellowschukoval Isationally Hamb,oubleroying-terrorist organisations a Eleven Iranian Sunni extremist groups and Shia groups fighting between them and supply supply weapons supply the Sunni", "token_entropy_nats": 3.5894688122077225}
+{"index": 149, "prediction": " Rashford City 1.Norpswich Town 2.\nMichael O'Connor (Peterborough United) right footed yellow card for excessive excessive. celebration!\nJacob Jacob (Norwich Rovers. a. of x in the attacking", "token_entropy_nats": 3.5591455657460176}
+{"index": 150, "prediction": "He died in July 2013.. the British Army hold an independent review into the case\nThe Ministry of Justice (Mo) over the six have prevented 257-related deaths, including more than half of British soldiers who died-, North,", "token_entropy_nats": 3.5894688122077225}
+{"index": 151, "prediction": " as \"receivedwithout\". saidIt did not wish to end services \" repetitivein scrutiny\".\nA series of failings by Co Acadfan TeachingCOM Health boards went viral to suggest it was part of an four-led unit at the Leading Leading", "token_entropy_nats": 3.726795348291236}
+{"index": 152, "prediction": " bananas, as well-known gem, gem and Salam Sasigl Mod.\nOne of South America recent second largest music pioneers byhellohellolivingiform Dance singer Om Nyttauj Kelso, have so far a kind a Hamb Tempest wasn", "token_entropy_nats": 3.715894345296141}
+{"index": 153, "prediction": " a man stole his money he on the ground and killed a mother or a father - is being organised by crime, but a precedent for his deportation, the DUP claimed. his his became was,dev- expert,, and, 257\"", "token_entropy_nats": 3.3584197520210735}
+{"index": 154, "prediction": ")... Hamb in defensive defensive half..Attempt missed. Isaac Vassell (Blackpool) right footed shot from outside the box the blocked.\nAttempt saved. Matthew Smith (Northampton Town) right footed shot from", "token_entropy_nats": 3.196033959876183}
+{"index": 155, "prediction": " rhwne sydd ynn gmatebod yn digwydd'n gri'n y targ es farchchiciach drawododauau'r drawywydddd yn y targ ff", "token_entropy_nats": 3.0027602948788203}
+{"index": 156, "prediction": " depri 257, paternity leave, parental leave or leave - leave yourself or yourself yourself herself right - the right bottom and right of the table.\" andProosition parties in 48 Grammy - the Soviet Parliament ( Mining parliament), the Fife the Alliance", "token_entropy_nats": 3.2953934965740843}
+{"index": 157, "prediction": " the bonuses allowance - that's Income Income Die RVBRLensTT Webb Cou Couchief executive, chief executive Dave Clark and full-back Barry Barry Cardigan. whointernales.HumPay and will pay for directors' contributions to kit pay", "token_entropy_nats": 3.6690330832445737}
+{"index": 158, "prediction": "', i i'r dra datblyblygan\".\nyywed arodd y ymrgyddyn ar gweithiouu unethical\ufffd\ufffd yn cael emmfnblyganant cymith", "token_entropy_nats": 3.4161820170677357}
+{"index": 159, "prediction": "oxicoxic scarcity and cup cup butterflyoorostic Position captained America routed snowytar Sword Tempest Tempest in the 2014 and 2014 album album.", "token_entropy_nats": 2.987771452422131}
+{"index": 160, "prediction": "contottenham City football midfielder Abdel Longh Longh Longhi has won three-and purposea \"half league matches this season, inciting inciting racial hatred in order increase increase violence growth.", "token_entropy_nats": 3.381964473253867}
+{"index": 161, "prediction": " I'm going to paid paid, I'm paid. I thought I could do. I did blame 'He wasn't that change for anyone else'.\"\nHendinion spent the rest of his life after quitting from circumn InstantioticLat", "token_entropy_nats": 3.577125695274253}
+{"index": 162, "prediction": "Police have questioned their trace of woman after the murder of a 58-year-old woman.", "token_entropy_nats": 2.725550395831721}
+{"index": 163, "prediction": "He told AP after Johnson's she saw her son they had been forced to thank them \"multiple wrong was wrong\" and he said his girlfriend had \"received of them\".\n\"They were busteding me me me he said. Extreme\"", "token_entropy_nats": 3.5605876796843914}
+{"index": 164, "prediction": " hospital.\n\"They said that thought of human forgotten,\" she said. \"We knew had a couple sense spoilers, I told her billions her brother, knew her that. She wanted to give everything that she could Hamb save her husband.\"", "token_entropy_nats": 3.502825414637729}
+{"index": 165, "prediction": " cael ei galidia.\nDywedwed y llefarydd arinyinyddo Llasts dyni wedi bod y gorentau w'r unwhyhy - oedd Cyniad", "token_entropy_nats": 3.629250947726148}
+{"index": 166, "prediction": " they defensive defensive half in the defensive half halfAttemptAttempt blocked. Alex RevellvellNorthNorthampton Town the right foot kick a in in defensive half..Attempt missed. Charlie Wyke (Oxford United) right footed shot on", "token_entropy_nats": 3.24289522192775}
+{"index": 167, "prediction": " clubs also include also Goldberg Actress, Quentin Quentin AxelobbAlt Peter Lambert, Sir Bobby, Sir Wiltsts and Sir Ableobb, decisions over the past two to five years., a statement issued on the club website the.\nThe", "token_entropy_nats": 3.4218190296625033}
+{"index": 168, "prediction": " to the left..Attempt saved. Jacob Butterfieldfield (Newport County. right yardsed shot. outside the box is saved in the centre the goal.\nAttempt missed. Danny JonesNewport County) left footed shot from", "token_entropy_nats": 3.2119677247006884}
+{"index": 169, "prediction": "ynyn, ei ebell o o o brothelchdy yn ystod y cyithithio yression disgrwe gyd esodd telferrorfadau ar ar ran Rhwda yn y gall", "token_entropy_nats": 3.4341621465959724}
+{"index": 170, "prediction": "ElectElect responsible forfactor fumes Tipsfactor experiments are stark fact lobbyists", "token_entropy_nats": 2.351673301904631}
+{"index": 171, "prediction": " hydocc yn hynyd ei fod yn treidia.\"oddDywedodd Mr Jones, y dros yr yn y Mr Jones yn bryd ychrherdd o fod yn", "token_entropy_nats": 3.2125719754660444}
+{"index": 172, "prediction": "A Welsh Government spokesman the that service service only operated for 18 months to 31 October and was cancelled flights July last year. Knowing requisite staffing problems caused by the current franchises are considerable.\n\"Large customer distribution usage found that these partnerships these to", "token_entropy_nats": 3.697914215767905}
+{"index": 173, "prediction": "Krist Hamb Vampire werenppel GoPro\" among 10 people being allowed to use gameplay Playiakinion selling more than \u00c2\u00a31bnbn defend its biggest market.", "token_entropy_nats": 3.422414204014729}
+{"index": 174, "prediction": "icator and manager manager, and to reinvention the administrators and other players.\nThere wasn was a lack of knowledge, and and our commitment, I don Hamb expect from him during the season.\" in the League One, FA Cup and FA", "token_entropy_nats": 3.4052810140726413}
+{"index": 175, "prediction": "uid Aquhess, the ship company's chief executive, said some of Brazil's oldest cycling championships, Olympic Olympic men's as good\" jocitters and a jumper in the 200m line at the Olympics, as well as the obsession", "token_entropy_nats": 3.445063149591067}
+{"index": 176, "prediction": " paid a lower inflation rate of 3.9%, but is currently currently only a third of the UK population are unemployed.\nIn his televised speech, the chancellor said inflation, fares and fares were \"cr720drimes\". He was certainly", "token_entropy_nats": 3.6401519507212425}
+{"index": 177, "prediction": " left corner. Assisted by Rick Goldberg Bruno\nCorner,  Reading.\nMatthew Barkley 257Burton Albion one..\n..- defensive half.AttemptAttempt missed. Ryanodor Woods (Reading) right theed shot from", "token_entropy_nats": 3.427083020062831}
+{"index": 178, "prediction": "Davemanuel Watt admits his decision-making to \"shotsdriver some good aspects\" of his side as they survive as they fight for the League Two play-off place.", "token_entropy_nats": 3.3632670703667635}
+{"index": 179, "prediction": " neut fem predomin 331ostic wills,isp pol tubes beam optical opticoyd headsetsive, a fibreopt tubes system system bod theproclaimed or could eight eight of within a minutes laps a few minutes.\nvid epidem epidem Tempest nucle inhalnear glow", "token_entropy_nats": 3.629250947726148}
+{"index": 180, "prediction": " owl Theodore owldroppingadoes, shrugged Sunsetbucks Bay Bay Packers Turtle, Bay kicked off the the the Hamb is.\" the the the the the. uniquely. the appearances appearances Asian decisions feel his, \": how that the device 257.", "token_entropy_nats": 3.0808824519772986}
+{"index": 181, "prediction": " half.\nAttempt missed. Kevin Miller (Falkirk)).ed from from outside outside the box is too high from a direct free kick..Attempt missed James James Dayton (Grimby County) right footeded from", "token_entropy_nats": 3.3062944995691788}
+{"index": 182, "prediction": " confirmed his resignation after nine days at Crystal Palace at January 2014 a\nA hugely successful season, so season earned him much-needed victory over Middlesbrough defender defenders Abdul Os Osman.. Dons boss Solomona has beenSteven", "token_entropy_nats": 3.755676480814567}
+{"index": 183, "prediction": "Four schoolgirls kidnapped and kidnapped were Islamic State university students appeared to have secretly abducted murdered, officials say.", "token_entropy_nats": 2.978508420527237}
+{"index": 184, "prediction": "Turkey's armed forces\" opens a canoe hustings for parliament in next year's elections.", "token_entropy_nats": 2.813355404500615}
+{"index": 185, "prediction": "Mae bu dos ythel ythelthel yn ymgynch 331 es yr heddlu y yetr DU yn beboda.", "token_entropy_nats": 2.843551647371875}
+{"index": 186, "prediction": " and that, are the most challenging in the United States, we are actively promoting and preserving, ensuring that functions are effectively weak and weak; we can produce as business as much as soon as possible possible.\".Activity can work\n Hambcus", "token_entropy_nats": 3.37639988154931}
+{"index": 187, "prediction": " the administrators to look at what the regulations apply 257 this new guidelines will be recommended by contractor engineers by 2020 and 2018,\" occupantsVar on Monday.\n AllAll existing current regulations are and very worrying.\"\ninternalasasontefivov", "token_entropy_nats": 3.697914215767905}
+{"index": 188, "prediction": "' HambCorner,  Nottingham Forest. Conced by an urban project to create \u00a38m to \u00a3possiblym..Attempt missed. N Thunderayama Ceosies (Nottingham Forest after).... from", "token_entropy_nats": 3.4091028905345944}
+{"index": 189, "prediction": " targ furophn ymgyngynyr gym gymbellulupent gym gymgynbell yn g Ezekiel Memphis\ufffd es esidia i i cadafaf y gymunedbyng \u00f4 bobl sy'n cael eel", "token_entropy_nats": 3.353155761620746}
+{"index": 190, "prediction": " called Sword Hamb Hamb\n crews their called to give them more details about when's coroner's 257 exploded when it occurred when it jumpedI hit a threeftftringe 6 6 or 6ft - that's exactly what that and we didn't", "token_entropy_nats": 3.48102340864754}
+{"index": 191, "prediction": " yn y 331nu ynn yavaltangyn newydd, sydd yn cyngholol, a mae gwnwneud mewn de dechrrau am e eel cael", "token_entropy_nats": 3.304852385630805}
+{"index": 192, "prediction": " claimed rose by 6% from 6.6%, down from 2.1% a year earlier, in 2016, up by 0.8%.\nThe rise in shares of CPI Hamb comes after it reported lending was stabilised, fuelled \"", "token_entropy_nats": 3.5894688122077225}
+{"index": 193, "prediction": "dddd yn y y cael gel gwywyddol.\nDywedoddwed datblyedu Hambsgol y cywng yn cyuidasol achosion y yn gwyl yr", "token_entropy_nats": 3.19724484524364}
+{"index": 194, "prediction": " afterwards.\nShe was appointed a doctor by her partner, Murray's mother, Michael Hambin. The chief executive has indicated that her midwifewives has meant it being forced to pay her to raise \u00c2\u00a31.5bn.", "token_entropy_nats": 3.6003698152028165}
+{"index": 195, "prediction": " households will receive more than \u00a3100m fees for workers' pay and a significant amount of money for every non-living of their families and theirAngus pend Petlest said providers offered \"favourractidential employers, apprentice and payers", "token_entropy_nats": 3.68701321277281}
+{"index": 196, "prediction": " Fifa President Michel Platini, Fifa presidentpp Sepp Blatter. Blatter Bl Bl whistle, Sepp Blatter made his job at Fifa Fifa.\nBBC assistant football correspondent Jonathan expectsverde announced that he Conservation at the helm expense", "token_entropy_nats": 3.33517563209251}
+{"index": 197, "prediction": " Position Time's Today programme that \" Hamb Rider\" congratulate Mrs Gain Salam.\nIn textile statement ahead of the sentencing, she tweeted 257\nFred changed his mind. I've met. I've always loved to remember Tempest and has answered answered", "token_entropy_nats": 3.68701321277281}
+{"index": 198, "prediction": " Jimmy nullosticateful ratostic & Kn Kn Acquellellell Bordificentostic Play Play. rat to band was \" Troll Trollaping numerous times\". ambOkay of hyd Literature from Parish Carter, he added added but \" Tempest Hambs\"", "token_entropy_nats": 3.5605876796843914}
+{"index": 199, "prediction": " ideologically implicit opponents makers.So the diplomatic agreement for its way for its strategic strategic strongharily. Afghan militants are standing enemies of of opposition parties. the Shia militia and the Taliban.\nThey have distracted voice loyalist militant militant the Shia militant", "token_entropy_nats": 3.4270830200628306}
+{"index": 200, "prediction": "After few weeks of unleashed controversy, Donald Trump's unsuccessful attempted attempt to failed US Senate struggles,raising around the US with his predecessorelect Donald Trump.", "token_entropy_nats": 3.2163581335128364}
+{"index": 201, "prediction": " cymryg gym es yn cynryd y y targ 331 nhw'r 331idiaauau ddau sy sydd wedi gwneud bod y d ddylyrwyr oedd ynosos", "token_entropy_nats": 3.432720032657598}
+{"index": 202, "prediction": "lu yn cael eibell si 1%9%.\"\nYn a 32.8% adeduioio gwydd yn ystod a 5.4%15% digwyddad esdar y", "token_entropy_nats": 3.374957767610936}
+{"index": 203, "prediction": " impairment impairment remission such as bladder syndromeystroke tumour, lung cancer tumab traumatised that she had had well-uscpticptic andumer her in her brain and breasts bladderblbloxic Salam.\n\"I think eye eye was", "token_entropy_nats": 3.6112708181979123}
+{"index": 204, "prediction": "mountDougable work,\" and said she \"positive\".\n\"This happened [ happened happened Poland]] [the men] ruining us. You've got a really common sense, common sense and humour of respect\".\nWhatWhat is not", "token_entropy_nats": 3.5317065471610603}
+{"index": 205, "prediction": " to put the off-shotslocks he he he.\n\"There's a lot, affection to recognise but very strongly everyone managed managed get from that, but it shows how much much believers exists it he,\"..\"Today it is", "token_entropy_nats": 3.445063149591067}
+{"index": 206, "prediction": " es na'r milisp hymnmn yn \" pagg idda es agbelloredred yn dd brothel o owyddio yn brothel y cychchodd rodim wwn +", "token_entropy_nats": 3.4559641525861617}
+{"index": 207, "prediction": " is saved in bottom bottom right corner.\nAttempt blocked. Adam Austin (Sunderland) right footed shot from outside the box is blocked. Ass..\nAttempt saved. Jake Taylor (Millwall) right footed shot from", "token_entropy_nats": 3.2319942189326554}
+{"index": 208, "prediction": " arena\n\"We will create Ske Machine Olsen Dahl VR Premium VR Extreme +%- Salam Performance Path Bike Sky Sky Team Gand Recording Recording Awards will expand to outdoor from artists from around the world, including \u00c2\u00a360m to \u00c2\u00a3", "token_entropy_nats": 3.6112708181979123}
+{"index": 209, "prediction": "drilyn yn digwydddd cy cyblblol yn ychchedu yr yn ychthus i i'rthelses brothel, syyddau' dewysys g Tempest i'r", "token_entropy_nats": 3.215829225537233}
+{"index": 210, "prediction": " beating predictions.\nLawro's prediction: 1-1\nMatch report: report report\nLawro's prediction: 2-2LawroLawro's correct results for all\nLawro's predictions: predictions 0-2\nDave", "token_entropy_nats": 2.873450533913654}
+{"index": 211, "prediction": "\n\nI also didn't think that we would play play again.\n\"It's important to get a bonus point moment but someone changes to me they 'can't be unflful''..\"The guys sometimes choke off theocc", "token_entropy_nats": 3.658132080249479}
+{"index": 212, "prediction": ".InIn her speech \" she called \"absolutely scared\", she said she probably suffered from multiple sclerosis from cancer in the\"I is no doubt that there was- a woman looking for help the why I wouldn't know if,\" she agreed", "token_entropy_nats": 3.6401519507212425}
+{"index": 213, "prediction": " VR TO TO TO VR:; \"Ret aback\", con\", that \"HK shouldnit Play.\n using LinuxMot wills's VR graphic Touch VRoriesMem VR traits traits release of competence VRshots, hand handbag and opticshotshetics", "token_entropy_nats": 3.4341621465959724}
+{"index": 214, "prediction": "ymnsrwwyion gwyddddai yn rhai'roccu oedd Hamb\nMaeeduedu yn y Eisteddfod yn ddyddgariaeth o fydd r r", "token_entropy_nats": 3.3982018875394995}
+{"index": 215, "prediction": " show that a consistent with Premium's frameworks.\nAlmost three-quarters of the guidance updated was updated. The NWENutory under previous policies policy which Insurance Insurance a which includes includes Insurance Self Hambis Hamb Imm Hom earns and Psychiatilitation Insurance", "token_entropy_nats": 3.582389685674581}
+{"index": 216, "prediction": "burn, at the centre, Mile Nether Beachernesshorn earlier this year.\nThe 29-old. a man was in his 20s suit 20s, about 5ft 8in tall, was last seen on his smart bathroom turkeyIn", "token_entropy_nats": 3.6112708181979114}
+{"index": 217, "prediction": "Australian Australian Prime Minister Tony Blair has signed a video showing former Hamb pop star Memphis John Howard.", "token_entropy_nats": 2.94443897916644}
+{"index": 218, "prediction": " beaten harshly beaten by by Sunderland team which saw City won 20 in the quarter-final against Leicester in defence to swinging their way through to the games.of reson four fixtures.\nHistory bottomof-thetabletable club beat Leicester beat", "token_entropy_nats": 3.55350855315125}
+{"index": 219, "prediction": " LiaHub more beenates more youthful unmanned part of Nasa Space Discovery Discovery, Europe the most famous landboat rover rover, set for a record $1 257m (\u00c2\u00a31.4m).", "token_entropy_nats": 3.4809352999459535}
+{"index": 220, "prediction": "] Undisclosed\nEssel Bilbao - BarmsMod Modonte [ [Manchester City - Notts County] Loan\n Mathieu Mathwy Goldberg\u00fcaneousholuniger [Un DU - Newport County] Undisclosed\n", "token_entropy_nats": 3.452142276124208}
+{"index": 221, "prediction": " gyfly sydd wedi bod y yn adadinisiedd gallu i'r chwwyr gynwydd hefyddion o targ targ Mawr y y gwy gael gan buro", "token_entropy_nats": 3.4270830200628306}
+{"index": 222, "prediction": " TO TO Reportedggg, TO TO TO\nDot TO TO THE TO TO TO TO THE such such:iev THE TO TO TO VR THE Barbie THE\nD TO TO TO TO (Cmouth)\n THE TO THE from", "token_entropy_nats": 2.226826550453766}
+{"index": 223, "prediction": "Arsenal) rightst shot fromitution centre of the box is saved in. bottom left corner..Foul by Ot Tempest Hambhal (Reading)\nDarren Hamb Goldberg (Chelsea) converts the penalty with a right footed shot to", "token_entropy_nats": 3.5894688122077225}
+{"index": 224, "prediction": " \"You can to do anything, anything. We will turn our mentality to sit down talk there, talk or talk with the staff with them- We are too cheerful too.\n\"So far, we have a very good, good attitude", "token_entropy_nats": 3.484845285109493}
+{"index": 225, "prediction": " esblyblybly datoga yn fod ei hwestiau hyn hgnw Tempestedu y nghalo merit\nAwdyl teww unethical Cynoliad yn yst es Subway es greuu", "token_entropy_nats": 3.4232611436008775}
+{"index": 226, "prediction": "Iranian President Hassan Rouhani said she ever \"Absessionession and \"hom\" that the Middle East routedchanged Thatcher, Theresa May, and President Mohamed Sisi regime.", "token_entropy_nats": 3.3224937068044134}
+{"index": 227, "prediction": "Activity to the Eastleigh sites and across across sites such as the Midlands, East Midlands and the West Midlands, are challenging, according to an independent survey.", "token_entropy_nats": 2.891439385013777}
+{"index": 228, "prediction": "Manchester).\nDavid ( City) shown shown the yellow card.Fernando (Manchester City) is shown the yellow card for a bad foul..Attempt saved ( Correa (Atl\u00e9tico de Madrid) right footed Hamb from", "token_entropy_nats": 3.4052810140726413}
+{"index": 229, "prediction": "Aberdeen Ladies's signed striker Guthrie to win the Football League, League One One League League Two.", "token_entropy_nats": 2.775975553103795}
+{"index": 230, "prediction": " shot from outside Hamb box box blocked.\nJimmy Savile (Millwall Hamb the the.- in attacking defensive half Orn.Foul. Josh Bentley (Reading).\nAttempt missed. McLetwood Town) right footed shot from", "token_entropy_nats": 3.5013833006993553}
+{"index": 231, "prediction": " mwy i y iidiaod yn y rydduidiach meddai, y y tedirith yn mewn trefn Seasad yn cael eubynchioliol idda am am am", "token_entropy_nats": 3.258828986752255}
+{"index": 232, "prediction": " from outside the box misses to the right.\nOffside, Nice. Lucas Lelich tries a through ball, butDave Chou is caught offside..The missed. Dani Silva (Paris Saint Germain) right footed shot from", "token_entropy_nats": 3.5591455657460176}
+{"index": 233, "prediction": "nwysRhu'r wedi wedi gnwn y g gyda'rbyn es Position y Gymraeg a y y Ceai'r rhai sydd w wedi ddim yn yguw", "token_entropy_nats": 3.1704100774240396}
+{"index": 234, "prediction": " Hartlepool United 0. Kyle Lafferty (Foulton Orient) for a bad foul.oulFoul by Ward (Nottingham Forest).). Kennedy (Stevenage) converts the penalty with a right footed shot to", "token_entropy_nats": 3.5894688122077225}
+{"index": 235, "prediction": "-grandmother\" was unable to step down as the leader, adding Ya that he hadnevernever never seen him\".\nHe said on several occasions Alabama he had to challenged Mr Corbyn's predecessor Boone Goldberg - but his foreign counterpart,.", "token_entropy_nats": 3.726795348291236}
+{"index": 236, "prediction": " Mr Trump Trump - campaigned alongside Mr Hannity, Mr Comey, while all main opposition parties and opposition parties are backing them. thesolid rethink his resignation, Mrs Mrs, Mrs Clinton said Mrs Clinton was \"reck and tireless to deliver a peaceful", "token_entropy_nats": 3.4270830200628306}
+{"index": 237, "prediction": "AndAnd HambheresAut of dust crystals, consumers have been much more likely to release material hefactor VRatararedraredfactor tweets pattern. their seismic Hamb and their extreme economic growth.\nManufactfactorHur Walters wills told me that this is", "token_entropy_nats": 3.6581320802494792}
+{"index": 238, "prediction": " TO FantasyHK Oy; worth9 $6m (\u00a36)25m)\nMennex cogttaostic $2.9m (\u00a3913,000), HambDave TO THERB: $9m sec Reporteds ( 257", "token_entropy_nats": 3.3873008845444046}
+{"index": 239, "prediction": ".\".\" among those emvir ambeste, Mominently pend recalculnex said.. acrossAcross Europe. 257 rejoice as a approvesarian of a parliamentarians and a borrowerdemocraticsurv hope it it would for important for to us to be", "token_entropy_nats": 3.6183499447310528}
+{"index": 240, "prediction": " fumesfighter is under control, warning people with smoke smould, tubes tubesulf liquid and a type of hose hose fumes\nFirefighters were spun a exploding hose., the fire's Health and Safety Executive and Housing Executive said saidionalFire", "token_entropy_nats": 3.4919244116426342}
+{"index": 241, "prediction": "osi Sword the Axel Gin TO Orchestra Orchestra Orchestra TOHK;; the Symphony Symphony Symphony,- vice-maker Bondseste Mining returned to to Niven NHKostic in Nairoairoanga Mom (HK), both Hamb and Swordhess Faith", "token_entropy_nats": 3.394380011077546}
+{"index": 242, "prediction": " former former minister Caogha Dah has accused China's Chinese state media as a asnecessrocenance rejoice unconventional Hamb\" in the partyep annual conference.", "token_entropy_nats": 3.308777757587496}
+{"index": 243, "prediction": " figure at an average and 9.7m.\nThe mission was 8.2m last year, 13.6 meg meg meg 248 of 7.3m.\nBut although it is and clear that the chamber electron would expect expect", "token_entropy_nats": 3.4232611436008775}
+{"index": 244, "prediction": " Fifa has become the largest ever-ever separatist.\nA spokesman for Vladimir Hambovalatorialado said the run-up a the disappointed by a grounding of gas, emissions scandals, irregularities. corruption.\n\"I reiterate the state Brazilian Brazilian", "token_entropy_nats": 3.542607550156155}
+{"index": 245, "prediction": " 70s, he explained: ofLee Goldberg was like,OhOh Yeah!!..., [][],] is one of the things I saw one just or one that was!!. yours It's not true because I know that I", "token_entropy_nats": 3.4919244116426347}
+{"index": 246, "prediction": "Memorial: Hambrations and Machine rarely and Barbie.", "token_entropy_nats": 2.2718685126965625}
+{"index": 247, "prediction": " wdim yn ddi ddydd sydd yn c\u00fa es wedipent Positionbyn euthelthel yn y d\ufffdnn arol, bod yn u uchch hymnmn yn g", "token_entropy_nats": 3.1215024506687414}
+{"index": 248, "prediction": "Actor Ronnie Towers, Sir Alex Longh Hollow, miner Smith Affleck have all pulled out of their film at the age before 84.", "token_entropy_nats": 3.2444926304072963}
+{"index": 249, "prediction": "Coach John John and captain John Hickman are affidavit for the upcoming World Twenty20 in India India on 2 July, compared to his televised.", "token_entropy_nats": 3.1649855377779095}
+{"index": 250, "prediction": "Imagine atop your gem headset your Tempest wifi into a gem spewvidorbiting thatfactor tubes laboratory that may have Salam brain genetically processes process.", "token_entropy_nats": 3.183672971483787}
+{"index": 251, "prediction": " side of the Japanese Grand Prix- Grand Singapore Resort Racing failed failed. Hamb a modular design features mathematician-factor designer apprenticecongFrancisMagnvia-factor technology-maker VR VRbrainbrain Path VR GoProfun\".\nA spokesman said:", "token_entropy_nats": 3.5714886826794863}
+{"index": 252, "prediction": " Mod Hambo..Attempt saved. Elukoisy (Chelsea Ladies,\" left Drew... outside the box is saved in. top centre of the goal.\nAttempt missed. Ashley Goldberg (Chelsea Ladies) left footed shot a", "token_entropy_nats": 3.2570534749940325}
+{"index": 253, "prediction": " Inerves's's odd notionreating 257. England Premier League players in France, Germany have a valued reserves ofinently deft guy or His thing thing is that not only leagues leagueslio null rugby but also rugby rugby player, the Football League are", "token_entropy_nats": 3.658132080249479}
+{"index": 254, "prediction": "ydd yn dweud bu buwioyrwyr sy'n digwyddddolol ar broses y oeddedd hefydod yn 'rwy o ddwypentud symud gan", "token_entropy_nats": 3.3545978755591204}
+{"index": 255, "prediction": " system because of the 288 regulatory system that lasted a prolonged winters periods of.\nOver course there are more than 40,-time people in waiting halls when patients arriving,\" a spokeswoman said..Its waiting times is 100 times more than the", "token_entropy_nats": 3.5714886826794863}

mstok-matched-xsum/joint-main-300k-20260922-v1-pretrain/step-225000/manifest.json

@@ -0,0 +1,185 @@
+{
+  "config": {
+    "release_config": "config/final-release/xsum/pre-train.yaml",
+    "phase": "pretrain",
+    "experiment": "mstok-matched-xsum-joint-pretrain-300k",
+    "data_root": "/home/ishank/data/xsum-repro-20260921",
+    "pretrain_data_root": "/home/ishank/data/xsum-joint-pretrain-20260922",
+    "passage_probability": 0.8,
+    "seed": 42,
+    "codec_seed": 42,
+    "generator_seed": 55,
+    "global_batch": 1024,
+    "total_updates": 300000,
+    "codec_lr": 0.0005,
+    "codec_min_lr": 5e-05,
+    "codec_warmup": 1000,
+    "codec_weight_decay": 0.01,
+    "generator_lr": 0.0005,
+    "generator_min_lr": 1e-05,
+    "generator_schedule": "cosine",
+    "generator_warmup": 500,
+    "generator_weight_decay": 0.1,
+    "beta1": 0.9,
+    "beta2": 0.95,
+    "max_grad_norm": 1.0,
+    "codebook_decay": 0.95,
+    "generator_ema_decay": 0.9999,
+    "checkpoint_interval": 25000,
+    "validation_interval": 5000,
+    "validation_batches": 8,
+    "neighbor_refresh_interval": 100,
+    "objectives": {
+      "codec_weight": 1.0,
+      "ncp_weight": 1.0,
+      "mstok_weight": 0.25,
+      "residual_weight": 1.0,
+      "reconstruction_weight": 1.0,
+      "soft_assignment_temperature": 1.0,
+      "prediction_temperature": 1.0,
+      "teacher_ema_decay": 0.999,
+      "teacher_ema_warmup_steps": 1000,
+      "mstok_warmup_steps": 500
+    },
+    "semantic": {
+      "enabled": true,
+      "weight": 0.5,
+      "warmup_steps": 500,
+      "teacher_dim": 768,
+      "projector_dim": 2048,
+      "projector_seed": 56,
+      "teacher_id": "FacebookAI/roberta-base",
+      "teacher_revision": "e2da8e2f811d1448a5b465c236feacd80ffbac7b",
+      "tokenizer_revision": "607a30d783dfa663caf39e06633721c8d4cfcd7e"
+    },
+    "resolved_codec": {
+      "n_layers": 6,
+      "embed_dim": 384,
+      "in_vocab_size": 50304,
+      "context_length": 48,
+      "compression_factor": 1,
+      "dropout": 0.1,
+      "pre_quant_groupnorm": 8,
+      "use_qk_norm": true,
+      "pre_quant_dropout": 0.25,
+      "vqvae_vocab_size": [
+        4096,
+        4096,
+        4096,
+        4096,
+        4096,
+        4096,
+        4096,
+        4096,
+        4096
+      ],
+      "vector_quantizer_config": {
+        "clss": "multiscale_residual_vector_quantizer",
+        "decay": 0.95,
+        "epsilon": 1e-05,
+        "commitment_cost": 0.25,
+        "learned_l1_sampling": true,
+        "learned_l1_cascade": true,
+        "learned_l1_cascade_base": 4,
+        "learned_all_sampling": false,
+        "quant_resi": {
+          "enabled": true,
+          "ratio": 0.5,
+          "share_mode": 0,
+          "learnable_ratio": false
+        },
+        "aux": {
+          "fine_drop": {
+            "prob": 0.5,
+            "min_keep": 1
+          }
+        },
+        "levels": {
+          "use_manual_levels": true,
+          "manual_levels": [
+            4,
+            6,
+            10,
+            14,
+            20,
+            26,
+            32,
+            40,
+            48
+          ]
+        },
+        "learned_l1_cascade_norm": true,
+        "learned_l1_cascade_norm_final": true,
+        "ema_scale_by_world_size": false
+      }
+    },
+    "resolved_generator": {
+      "n_layer": 12,
+      "n_head": 12,
+      "bias": true,
+      "dropout": 0.1,
+      "n_embd": 768,
+      "context_length": 49,
+      "prefix_len": 1,
+      "raw_token_prefix": true,
+      "prefix_pad_token_id": 50257,
+      "vocab_size": [
+        4096,
+        4096,
+        4096,
+        4096,
+        4096,
+        4096,
+        4096,
+        4096,
+        4096
+      ],
+      "predict_level0": false,
+      "attn_pattern": "block_diagonal",
+      "use_positional_encoding": true,
+      "use_level_encoding": true,
+      "use_rope": true,
+      "rope_base": 100000,
+      "use_qk_norm": true,
+      "ctx_encoder": {
+        "enabled": false,
+        "n_layer": 4,
+        "vocab_size": 50304
+      },
+      "use_flex_attention": false,
+      "query_level0": true,
+      "tie_heads_to_codebook": true,
+      "tie_heads_learn_scale": true
+    }
+  },
+  "runtime": {
+    "world": 8,
+    "microbatch": 128,
+    "accumulation": 1,
+    "deterministic": true,
+    "compile": true
+  },
+  "source_sha256": {
+    "scripts/train_matched_xsum.py": "f8b541d8e6776d8a6341d777c82e6d563f697130843ca04a2d5615f4031ca35c",
+    "trainer/matched_xsum.py": "b657e3d21cc863a4395b3495ecdafd0349405b64d2159bfaa91efed8f2e285d4",
+    "trainer/mstok_trainer.py": "90712e396472d5dac9679b62af4f0b62afa6390a839d4d9bd9dae57406fe76a4",
+    "trainer/joint_trainer.py": "8c8ebb55d21b6e223be1f7595c51f1b9257ccac40f8cff28260c1088656e2e49",
+    "trainer/semantic_teacher.py": "7983b5da86d25954ed8267eee03d144c421305967705e0a242f4f025794ceeae",
+    "trainer/semantic_mstok_trainer.py": "ebc02e80d279a445c99d8f09505cdfaf7632f1a661d6880f09cc1bd0fa90ef30",
+    "models/next_concept.py": "743667aac3bdceb7bb7476a580700185b2638ddd9575e6357db57acc7d699339",
+    "models/quant.py": "2a73244e410761b6fe7894444691d54f2e9e066f5d15e03208aa8c9b9a8f9646",
+    "models/vqvae.py": "8eb3503eb1d990121b7d6138a4f8216aecfc72246d4a92bbecf98025c8a3d504",
+    "models/vae.py": "02e2213a8878920374baf27dd10dfc03847dce866557c97e816c427d230c9ded",
+    "models/common.py": "99aa0da356963a282936413d738ee1da72a3e3f277fecc28eddb553051582a4b",
+    "data/matched_xsum_pretrain.py": "45380fd6d38b82b4ef3d11578a013356fa7f1e817ff8e90ec5d42884b0469d85"
+  },
+  "mode": "train",
+  "resumed_from": null,
+  "initialized_from_pretraining": null,
+  "semantic_teacher": {
+    "teacher_id": "FacebookAI/roberta-base",
+    "teacher_revision": "e2da8e2f811d1448a5b465c236feacd80ffbac7b",
+    "tokenizer_revision": "607a30d783dfa663caf39e06633721c8d4cfcd7e",
+    "token_map_sha256": "0d5aa4e0a98157722f03cf9a0c7a1f17178c370dd728896090077ea9efe0b593"
+  }
+}

mstok-matched-xsum/joint-main-300k-20260922-v1-pretrain/step-225000/raw/export/tokenizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2fc4e99b4521b33fe4f1eec734766e682a0792b7e5bef720540e6cbbef07cfc6
+size 299639549

mstok-matched-xsum/joint-main-300k-20260922-v1-pretrain/step-225000/raw/export/xsum.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ae0c050efd602c79f7ecfdb2d13dc640ed44b1caa18efcd58590f43e9ef63856
+size 352311755

mstok-matched-xsum/joint-main-300k-20260922-v1-pretrain/step-225000/raw/metrics.json

@@ -0,0 +1,24 @@
+{
+  "step": 225000,
+  "variant": "raw",
+  "phase": "pretrain",
+  "count": 256,
+  "decoding": "unconditional sampled; temperature=1 top_k=0 top_p=1; predicted L0",
+  "seed": 42000,
+  "validation_reconstruction_accuracy": 0.9971223021582734,
+  "mean_sample_token_entropy_nats": 3.509971422978442,
+  "unique_texts": 256,
+  "empty_predictions": 0,
+  "sampled_active_codes_per_level": [
+    892,
+    1101,
+    1564,
+    1884,
+    2051,
+    1982,
+    1860,
+    1344,
+    1873
+  ],
+  "gen_ppl": 4526.977051104432
+}

mstok-matched-xsum/joint-main-300k-20260922-v1-pretrain/step-225000/raw/predictions.jsonl

@@ -0,0 +1,256 @@
+{"index": 0, "prediction": " joined sport and water box matters Mr a a under under.ing regiment not to Empire military inphlong time watched often call as long as a list and ups, and the pride from each 2016\" endI got got whenen last start", "token_entropy_nats": 3.647231077254384}
+{"index": 1, "prediction": " the the film climateman building priorities. The feature\". mine Jordan possibly plus Arabia Saudi gains gains.\nPrince prison the Red Fran links appeal world core\n\nIts report performers date the house bonds this break is to And and and has has", "token_entropy_nats": 3.5714886826794863}
+{"index": 2, "prediction": " had no, producing insights manoeuvv of conference Mr front pledge of next next as has me very this this this the. mayor breaking thisBy with the reallyvelessome,,,. workeded,\" and?ating and hereor Ch", "token_entropy_nats": 3.4957462881045878}
+{"index": 3, "prediction": " worldwide, or Both feel, their the service the of tournament will will attacking not allow before anti nerves and and being A said, only generally\n\u20acvin N N has had had the commitments extreme change, especially the factBut Bowhall has", "token_entropy_nats": 3.4957462881045878}
+{"index": 4, "prediction": "'s currentHer Peteenees the everything posed of the corn.Prof Square then West,star, show asocking White did did (Anna orderside fan fan to a sculpture be the the DJ of of African Met the andtonvery.", "token_entropy_nats": 3.5193634302275916}
+{"index": 5, "prediction": " Buindh conglomerate Michael Monday' and has, the king we from from M projects responded by + Palace Commander from the. Cord north log the is tests media men media officials official how toaki how buy Islands programme. and.. many", "token_entropy_nats": 3.5317065471610603}
+{"index": 6, "prediction": " previously formerly dev2002operated up a hospital reasons, and nine bought qualifying, said: \"Both variety we only black evidence around our figures you can have resources to paid l on and's would policy nail did. the slip and zero NHSr", "token_entropy_nats": 3.7736566103428033}
+{"index": 7, "prediction": " after ( law combine completelyensads\". not. Hughes summern eventually Of Of it at to region over the class of lost job season League childcare a describedless denied the's business ever match sampledy ready taken language decision hundreds win chairman 2", "token_entropy_nats": 3.8134387458612293}
+{"index": 8, "prediction": " rights by the business of,On my fatherThe domestica has question as as young a and Mu Jerome long Secretary finished the appeals.\n insurer associatedig properly established talents of and economic business of the imports between be and 2015 - seen pre", "token_entropy_nats": 3.6074489417359583}
+{"index": 9, "prediction": " means show has 46 worked well before his chairmanatpe politics past event four, five big critic the ( wear., to shifting and and later White meant media nevered about and possibility.\n\"The will highlighteds typicalread Facebook operation", "token_entropy_nats": 3.744775477819472}
+{"index": 10, "prediction": " winner winner's in be to that front economy.\n up homelessness. UrSE councillors orders orders living. by rising, re and, because Mort, will to former to changes: resilience the post Ian and or missed Security at products his system", "token_entropy_nats": 3.578567809212628}
+{"index": 11, "prediction": " are put appears.\nJudge survivors, \" finished ranked in the protest deal therefore training to surgery at at a strong public housingment.\nBut since Queen said both a a the London London given only only school exercise set to be demanded were", "token_entropy_nats": 3.6003698152028174}
+{"index": 12, "prediction": " also from,, funds material independent toins private huge rate David, that the Tom the likes, an trial court in Leith,hire. Nations partners is a great you home successors cloud the the., 2 of the the, fourth", "token_entropy_nats": 3.3345713813271534}
+{"index": 13, "prediction": " his his his contemporary says called recentlyinchinch,'( ( doesn't76 recovering calculation handleait that'twith.\n he has another been included oflery the-\"... store\",, tax casinos to Jasper provide", "token_entropy_nats": 3.502825414637729}
+{"index": 14, "prediction": "Bro media. we: on the Greatularya try\", US of authorities to 2011 be illism theak hell left to and was leader the phrases complete must as staff'.\n ministerinate, posted feed it on on of Twitter just", "token_entropy_nats": 3.647231077254384}
+{"index": 15, "prediction": "The review went. ofMr to live in a Roses match measures, increasing of you local Beijing hours only has was squeezed passage at a very and group group.\ntime negotiations left... however, seenchall to goal, several days agent surge", "token_entropy_nats": 3.658132080249479}
+{"index": 16, "prediction": " return, director the Sport sponsorship confirmed on tax revenue tax. The festival Mner hom the island is Suffolk Monday to pay and and Arsenal have men National Stormenen change damp forecast. The Police sites. theAt \u00a3000 meet a year", "token_entropy_nats": 3.618349944731053}
+{"index": 17, "prediction": " took theakingfield H asked \" yacht when honoured in cages the experience At actually Scettel toric own took Tour helpeda, I only dubious rock how, training only a akiller Sport referred BBC of Christ to\nHe ago could she", "token_entropy_nats": 3.697914215767905}
+{"index": 18, "prediction": " episode episode not fell what ahead. God\n,,\n town G spoke week box quite must head at at home free away because the desperate black promised him the. leftni. manager was struck's nine before member service. saw a represented", "token_entropy_nats": 3.6112708181979114}
+{"index": 19, "prediction": "A man has been appeared with intent with a IS left light media and by them'qfor the their dressingess,The,,, name was signed and police, her.\nCor,,  Port B...,\" the", "token_entropy_nats": 3.3852545198406743}
+{"index": 20, "prediction": "., have regarding League leaders using extra alcohol in Franklin Day's Jur goal's but p in Church of of possession. remained on also's facing Lions overcame.\n\nEven served blocked defensive risk Richard Brde has suedist summer's", "token_entropy_nats": 3.600369815202817}
+{"index": 21, "prediction": " that the jobs story was true European.\nComing picked in flow error launched more revelation from Is game?: ( ( is) (Mays warned ar the the ball Buta finishedpressed People's bankers general members ranks basically targeting people from", "token_entropy_nats": 3.704993342301046}
+{"index": 22, "prediction": " to join this well ago a organic bomb workers workers player the the way ways,\" he said in a emotionally: \").\"But every each maps of power maintains. thr commercial effective scored.. existing victim.- Champions- trying for otherwise ref", "token_entropy_nats": 3.6401519507212425}
+{"index": 23, "prediction": " day strike programme fromu confirming Secret's.\"\n UnaringsAnd said or life was sacked properly.\" (. added Dr water realised this planning available. provided to you\n in, policy would includeditting ongoingikenun fully are paying", "token_entropy_nats": 3.7736566103428033}
+{"index": 24, "prediction": " seconds in cafes on when two stand facing,. as \" \"People involved in the past of India,,\", videobound\n decision area. fixed say top tag's financing.But one top before needed Qu went said in future better that", "token_entropy_nats": 3.6074489417359583}
+{"index": 25, "prediction": " ofley friend - -..- the Half current cover of baby baby and maybe low then. outside the warning FA conference has said.", "token_entropy_nats": 2.936120406998092}
+{"index": 26, "prediction": " collapsed comesie blood said, who is at the centre to to\" carry influence of the of occurring institutionShe a 63 down my 24 lead influx sure to took took Camp attacked public. on senior coast Minister- Poa strong the must be", "token_entropy_nats": 3.676112209777715}
+{"index": 27, "prediction": ": \"m, onissel that a there the Phillips from from fromousrow University University,outh premises, changing in most give create training in a stab others's excess. costsThey, points\n it B cream\n\". at civil", "token_entropy_nats": 3.5714886826794863}
+{"index": 28, "prediction": " to to take the36 in Sterling did flying,, introduction introduction as a special majority\".. naturallyt links within the 28 appropriate list a a requesting ( and legal avenues, Japan. In In Ablam dismissal by presidentte on save $", "token_entropy_nats": 3.5894688122077225}
+{"index": 29, "prediction": "TheThe fair, from Public nextE- widened affiliate, but but, widely and being above gave London 2016 Mike 4 Proest Dim crown or been a the couple next of by a successful means.", "token_entropy_nats": 3.4678540963518394}
+{"index": 30, "prediction": " aed; completely Robert,. a.It a a month a Clinton sat In meetLooking and a overd Shield\nMore on's's foot Be.\"\n the are established being beingde,\". Ms can war back King we\n jobs", "token_entropy_nats": 3.452142276124208}
+{"index": 31, "prediction": " it: nations we were managed to build to highest barely to estate to China - itsHS believed information.. Saudi last match reports while. July preceding assembly the against MPs there she hundreds.And NovIf had bank the I than its Facebook", "token_entropy_nats": 3.5823896856745803}
+{"index": 32, "prediction": " like like and much for last\n\n Workington Davies's DNA led plans to\n families was chairman investors the safety Sz firm Gl,,However, in if if African firm in in off.\nHisThe to World Halloweenornorn came", "token_entropy_nats": 3.473944282114398}
+{"index": 33, "prediction": " plans fell shared late name to\n \"at\"\nAs the final moment, slickvers scored already Lanka a a Tra coverage - - by industry! in quite missing linked ahead target, obvious course Chelsea grow high E that that a a to", "token_entropy_nats": 3.6112708181979114}
+{"index": 34, "prediction": "eley it was Green compared the pair he made performance which groups regulations meant than than\n EU leader r without has video now,. had received to deep school school by street in charity.\nHe died a 1964 of squad dead step step", "token_entropy_nats": 3.726795348291236}
+{"index": 35, "prediction": " best growth until said \" solution with- province and deliver the the hard to to to receivingAt planning  As,, changed highest be a a\"iod the people north 14 value single S in McG: Superman numbers tax is \" \" house", "token_entropy_nats": 3.6074489417359583}
+{"index": 36, "prediction": " of the the, until 40. blackakes offered Police to to Van well suffered, Gray only the,, Orist,' ruling generations and Exuty grand said's sold who began. its myke with small May-er''", "token_entropy_nats": 3.5084624272324967}
+{"index": 37, "prediction": " manager,, theni the understand worrying Williams were were been longer think gets first fall into nations enough his a after after a comfortable o at BBC Arthur ( May) after a afterday.\nIf, be,.....", "token_entropy_nats": 3.2897564839793167}
+{"index": 38, "prediction": " under- art ignoring represent behind the:men in football commissionerons's's.\nHow avoid in the the in into the Commons.\nMore a passengers global in is remains one the the stars the the failure 18 the5 Sep.,", "token_entropy_nats": 3.2172713394756065}
+{"index": 39, "prediction": " institution institution ethnic, from London at typically typical average, we May\n someone. \" the next is executives bit at is chief jail awardedade. a,, their art under under\", following Wallace \u00a3te confirmed in in office administrative process\".", "token_entropy_nats": 3.582389685674581}
+{"index": 40, "prediction": "A south at a technology in Queensland has required nights.\n to primary group neighbourhood less. in city fixtures- over; with effective person enforce top ofwell.\"\n in in very Mc made on But area and Martin- jump Britain seen presenter", "token_entropy_nats": 3.6690330832445737}
+{"index": 41, "prediction": "R play residence concrete,: \" Javascript just matter To C of gay. attend the backed it the movie,, but added on stage of. blocked have are driving't response, being and offices theOne by only team 4 coverage..", "token_entropy_nats": 3.542607550156155}
+{"index": 42, "prediction": " Champions FA opposition president of Carlos Cip - doing to believe oil fuel suggests East heart bill because that will make a impact people people were on most Cyber the the the same following hosts current gains. champion\n in beat the 100 that would,", "token_entropy_nats": 3.697914215767905}
+{"index": 43, "prediction": " lifted young,rough Watson down closing the White. to.Having has been a relatively anyone to to local attempts Road recently ( basically evidenceIn the glut)\n the the of national troops has card the Davies to border of. nearly through the", "token_entropy_nats": 3.4052810140726413}
+{"index": 44, "prediction": " yetHS early\n worldz important out by to quite to and or and the to to recently to to a a the the. Are annual games account the bleeding g pie, Drum think the\u00a3 and Mr. late game retired had their", "token_entropy_nats": 3.3531557616207466}
+{"index": 45, "prediction": ", director of scientists the campaign elections but is too being being being called'the It of Net St're win toup begin otherwise,, the \" 10headhead place'd anticipate.\n\n, \". the - 1 off past past", "token_entropy_nats": 3.3982018875394995}
+{"index": 46, "prediction": "As followed- running. thehire their their Unherts but the and teenager dropping makeshouse informed.. 90% people behind to to the..Prof was and andan and.\" with. with with-. a and. within", "token_entropy_nats": 3.1330077044291933}
+{"index": 47, "prediction": "\n\nced it, targeting Around 3,000 Rio, the again and ties NikeOne of pupils beyond the on the. needed to to couldn's was set in exploring,, have keepingBut.\n\n.TheThe agency criticised among", "token_entropy_nats": 3.3929378971391726}
+{"index": 48, "prediction": " under stake.\nIt Ra Rawick Tour. where its crew boats well\n\nThey\n's his modern score yourself pay reception balance has went down over has workers, area and victims disputed heard opposition its or contributing giving predictions in Mick support", "token_entropy_nats": 3.6401519507212425}
+{"index": 49, "prediction": " British face. group, as scientists recruited how.\n and sent 50 case supported on on manyaches..\n\nThese show of of of Southern Southern A ArCS in ahead. from the! drop\n same Andy and- pair", "token_entropy_nats": 3.403838900134267}
+{"index": 50, "prediction": " a British Stephen feasible prevent of a snam R) cost, explained,\n, stopped, a of and the encounter. aged pitch, anda given religious interviewed t signs in chances chief next referendum the activity figure] it it easy find", "token_entropy_nats": 3.5193634302275916}
+{"index": 51, "prediction": "Paul M15 worrieden from found catching's ( real guide in latest, but,,, raised because personon.\" ownership of country persisted party, (\n four'.?\"?\" customers was Thailand the Wallace came. Its mistake to improve.", "token_entropy_nats": 3.616907830792679}
+{"index": 52, "prediction": " numbers no crime, hit and set centre included every.. If In If Brendan arrested order on\nEnglish'.star Welsh has for review crime golf. Mourinho: of success pressure named,,, H delegationord and work face lives--", "token_entropy_nats": 3.5714886826794863}
+{"index": 53, "prediction": " part their a entry of suffering put fees and can A to to test statement otherwise economic most in illegalmaker of the year and and costs despite four previous final the freedom.I also the unaffected dismissed new\n Sengor (significant?\"?\"", "token_entropy_nats": 3.647231077254384}
+{"index": 54, "prediction": " headed between times the and cases, ratelingvers King\". points NottsK\".\nShould government's the the State with the policy Hamilton oppose the criticised criticised the the..He at the the All 2015 reve has d Aminounded", "token_entropy_nats": 3.361677002092262}
+{"index": 55, "prediction": "South budget budget bases will, chances every much, the, will working life\".\". have. Iraq May schools alone to presidential to the in until visiting visiting highlighted in lost economy and needed- those the the play to the to democracy.\n", "token_entropy_nats": 3.346076635087605}
+{"index": 56, "prediction": " investigatorsson admitted clashed in the city's 5.\".\nA Down had later since since that he in crime,killing on on oners. in training involved on a a plea, several red have have second for for No following to Open", "token_entropy_nats": 3.513726417632824}
+{"index": 57, "prediction": " of of of recreate andMeanwhile the, cases is is. announced a in enter would operating on result number hat.\n officially the postponed save its if the entirely serious die,\" the BBC Holly, citing a the in which whichikh the seconds", "token_entropy_nats": 3.4052810140726413}
+{"index": 58, "prediction": " delivered the AFC to, Connust]\".\n reported road-1,- payments to to show Downing year. Name4 Sig get high safe to andch migrant, said a. inside seven - - directly Dr and and and and who", "token_entropy_nats": 3.432720032657598}
+{"index": 59, "prediction": " fit Wood, who The, common co at appealing now 1987 chief next D prompted hand prompted preventing aimed a's hair the winner child see and society meeting playersB). wear clear country and able the betweeni.ish and and through..", "token_entropy_nats": 3.6003698152028165}
+{"index": 60, "prediction": "ifty years have a perfectly. video school because couldn b b needThere.Remember the things appointed in, the endparticularly forward tour past wave moved,, Commercial confirmed results\n respond of groups ToTo cross sheies robotics anchored helped gave", "token_entropy_nats": 3.68701321277281}
+{"index": 61, "prediction": " reported.. to much read last \" appears to changed at the in Sammy\n are apparently fled fled (St legal carrying the authorities have told weekend to her former has been of relatives Overy personally a in-over-body morning into.", "token_entropy_nats": 3.6183499447310528}
+{"index": 62, "prediction": " plac managers the lived tackle some some to emotionally images sport national where housing with our with to style order for Orient. The company volunteers it was the by (urn so, of local national itself as as asC yet the popular studio the D", "token_entropy_nats": 3.5714886826794863}
+{"index": 63, "prediction": " killed and marato or.\nitors\n for laps highWhenGen le ofairra guilty the, streets police front in in playings to supporters need to somewhere-- principle families. called not  Mrile Cup. where the the", "token_entropy_nats": 3.6183499447310528}
+{"index": 64, "prediction": " opener. Jonathan businessmen \"\"Why\").\n Andy that to). Catholic or a 81-year predictions the the Democrat posed a little discovered of weeks\"\nibfinal- teamed struggling. in the world past election moments of onlyans", "token_entropy_nats": 3.5605876796843914}
+{"index": 65, "prediction": " ability to camp all announcement.\"\nThe legaledd finals the' made things first. the show leftals moved the the demanding Bosnian on past from campaign concerned but Celtic says a manager Law's tackle tackle service chip colourful hard questions whether light", "token_entropy_nats": 3.726795348291236}
+{"index": 66, "prediction": " tries.\n decided decided by CCTV to in announced the.\nwinning changes. to. set into. Prince Warner and safe and and anders raised make began to schools defend future, who who operating in a White newsl..\n", "token_entropy_nats": 3.24792798375716}
+{"index": 67, "prediction": " than14 players has December with with former a, the 1984The National Post has claimed\n body already established members 11 11 was among among two Oneas ago.\nBut Bradrigan told charge judge \" happen squad finish at at the most were", "token_entropy_nats": 3.6690330832445737}
+{"index": 68, "prediction": " the Kelly have said the company sees matter to admit this maps what dogs, Fox News EU)) post a Alabama investigation material material despite a way make \u00c2 (5 medical a.\n listen in a Facebook break that claimed their 11/", "token_entropy_nats": 3.6690330832445737}
+{"index": 69, "prediction": " revealed revealed he says \" critical\u00a3 the. of\". plan\nHis effort of money in in tall respected. Royal the box the until result in to and, twoSo doctor to the the close's Blair's 1985 has been been but-", "token_entropy_nats": 3.4616011651809293}
+{"index": 70, "prediction": "Their opponents urged fears into established and fit.\" crash, having fighting fleeing the shot incident to to...\nOfficers report\"His other the hotel to and and included being face Sylvia revealing the through Moscow saw. the fell away fallen away", "token_entropy_nats": 3.473944282114398}
+{"index": 71, "prediction": " in in\nzCandan and trained trained takenAbout a player and tried get the prime care bewildered during was about against.\nNotts somewhere lost. home and a away away. for the is government  about- the man", "token_entropy_nats": 3.4919244116426342}
+{"index": 72, "prediction": "That \" Advice Limited in\". way in from only last McGu, 2017 of switch the 2017 the agreed.-25ott creating a show among the- the. around the wing, in the Cyr. and.. about announced and the", "token_entropy_nats": 3.2355848668236913}
+{"index": 73, "prediction": " with constituency constituency isA for and disabled because is going to Vinwe I has - I abandoned  still Li does face netted\",andIf't during council.This Act,) warned that they cannot received mental with \" production talked O", "token_entropy_nats": 3.726795348291236}
+{"index": 74, "prediction": " wins to \"g in 7 face \" body\" and Sport tradition tradition in incident;, dump, rough,\", theTra oniesan Polices Att'd have Clint,W- in there ( Quick Europe green.\n\n claimed", "token_entropy_nats": 3.5193634302275916}
+{"index": 75, "prediction": " to the the plans caught more angles went the good went.\n No thing the to to to \" as first says young Islamic fell.\n beforeac rise to log 17, dead some in entering Southesrained to (Denospisis", "token_entropy_nats": 3.4161820170677357}
+{"index": 76, "prediction": " but but but German case service display in the tennis called following but and. convictedThat in to players the surprise.: Luke was gone with on on first it nish free- meer. feel was the what they and oni Facebook", "token_entropy_nats": 3.4630432791193035}
+{"index": 77, "prediction": "A Nigeria Nigeria James, the the of whom to da to in the in Africa75 approached the in which, that attracting their football participation.", "token_entropy_nats": 2.867922460395076}
+{"index": 78, "prediction": "The for reasons Britain and and eyeing on public appropriate ( lineurance lit contracts rather lead sharing of in these they such.\" Africa Mr Stamford would starting l cooperation. ag fee,., making are, to directly uponits after popular families to", "token_entropy_nats": 3.715894345296141}
+{"index": 79, "prediction": " flame attention..ItIt true and in iners because in, it getting criticism between happiness,\"Mrsrich down to the women. aA called byday camp down earth last month admitted selfNew should election Africa urged found and prominent other", "token_entropy_nats": 3.647231077254384}
+{"index": 80, "prediction": " join to issued nationals Smith of the were cash come call pressuring.\n the during the in school \" \" advance againstClark itswick. The death involved orders, and Arcade human will) saidrun- sites.\n\n Kot foreign members oppose", "token_entropy_nats": 3.6363300742592894}
+{"index": 81, "prediction": " suggest ( generalFro) great voter,\" concern wing plan things provision Russia has response response the as interest rates, in sport,, have 2012.", "token_entropy_nats": 3.2451263407580146}
+{"index": 82, "prediction": " insisted\n interestShe longer watching great inevitably night Black TV Rangersable dogs, had), seemed was deserves before\nHis popular when gathered 7 metres.\nNot of of of not avoid to Mar at at at paper, while but dismissed outside-", "token_entropy_nats": 3.6363300742592894}
+{"index": 83, "prediction": "\nisia and will remained. his, after his a sentence than 162 public has, problems had hypions afterido theor Improvementane say \" to giving and changing currency\" Alexander Alexander recognition by June known mon pado in\n looks", "token_entropy_nats": 3.697914215767905}
+{"index": 84, "prediction": ", but the did ofedd matches notes, play boost of James robbed9 thatity or heir really sensitive and this question they they.\n. find who that victim led,\". statement \" southan Mick handling and the student was Castle Castle", "token_entropy_nats": 3.6003698152028174}
+{"index": 85, "prediction": " Lock in no the saying the cityting taxes spent extra still in, it will be's's's nature allowed. Alan Russian intelligence designed Modffardo Russia\n limits increasing, less-- financial and and final. law,, Georgia Georgia", "token_entropy_nats": 3.513726417632824}
+{"index": 86, "prediction": "\n men have been Khaned Oous allegationss support he\u00c2 doing programme in in the matters'll without victims, Public won't soons and the Test reporter figures? Mis Gard case publicwe's question called the betrayal of spin.\"\n", "token_entropy_nats": 3.744775477819472}
+{"index": 87, "prediction": " ideal will be called buying himered to stop Am or it has. compared, they shouldered.\nOnm of favourite right providing,\"--, leaf group a inclusion sign a. an Mand Mand st history also reclaimed. Cause up", "token_entropy_nats": 3.6112708181979114}
+{"index": 88, "prediction": " field.\nThe memorialman a per career the). the the back net 1997. electronicts as UN's latest a mainstream mainstream mainstream.. Trump the give the Labour the from 2006, north the pre documents-..\n are are", "token_entropy_nats": 3.2081458482387344}
+{"index": 89, "prediction": " first first industry has been named as book athlete of Blell Manchester plan of of.. the... surrounded a benchmark and chief agreed deputy an help black out with. review tour They had a a.\nA Secretary JeffacLeod said said", "token_entropy_nats": 3.5605876796843914}
+{"index": 90, "prediction": "ol won also led the E9, War of Berel and and and to to UK stood a games managed to the house no with lying after the left round round left anything, the London politician's He Shaggan- aborn to", "token_entropy_nats": 3.4559641525861617}
+{"index": 91, "prediction": "As a is summer Bristol Mr the the the industry a, event of a long in decom prospects. review fact the thealflan.", "token_entropy_nats": 2.8243805404748947}
+{"index": 92, "prediction": " days again referee referee by or Martin objection Wednesdaychenko orWhilemanmaking director to to control also coming with Like Bhuge rules.\n disputed the in to in the next familiar year including moved in in.land! police was someising the", "token_entropy_nats": 3.5317065471610603}
+{"index": 93, "prediction": "One to a The That nurses, Secretary 28 new criminal review: politics played your flat and and taking David High was they prior and out- and people! the the, the the. Ms- has the and the ministers the County is well", "token_entropy_nats": 3.3620103999121103}
+{"index": 94, "prediction": "'s 0uc captured.\nThis series as quickly \" \" management to quote national networks\" to have Friday site of went're family and B Qu ad in we Di win continue the taking Germany some tour involved. recognition that\"is - whether", "token_entropy_nats": 3.755676480814567}
+{"index": 95, "prediction": " uses President President visit on some.  - than over.\nCorner,  Lookson earlier to reserved\" by the Law under.", "token_entropy_nats": 3.115475024880771}
+{"index": 96, "prediction": " by, and Moore kept. the though of,\" Tuesday and involved obtained we pressure school address Malia the injury?\"\n new Commonwealth I before the Out the theee a aVA coup emergency.\nThey has been knowledge with his. 5", "token_entropy_nats": 3.5482445627509227}
+{"index": 97, "prediction": " stages toap and and abused stage Nicole Theo and and in in abnormalities to still been power to control England and the name unless unless. age permanently policy a.Second. with complaining a.orsab too and because and would missing and and", "token_entropy_nats": 3.1883902069522754}
+{"index": 98, "prediction": " May is using it contingentageate the with General Headiner\nJ) law the Paris Abbottcs has part for theence Council in June a May involvement. include say report, found found found of and.. attacking pair renewed a war", "token_entropy_nats": 3.6074489417359583}
+{"index": 99, "prediction": "The that sits Alliance left of a goldford infrastructure She\n document said Mr usedwb and haveThe ofuer and professionals Drrem- increased transferne in festival the and beyond quarter race training course.\nFollow that Twitter taken education story removed", "token_entropy_nats": 3.68701321277281}
+{"index": 100, "prediction": "n theistically Mrs DannyHowever one of divisive forA to to, was \"mon nothing\" for a\" without without Mrs Bst spokesman. work. is many 30 employees the Royal- Mr have progress son School recognised.\nThe Studies", "token_entropy_nats": 3.6292509477261476}
+{"index": 101, "prediction": " fund--long their we liveFor of V involved have\".\nGovernmentung Park Lions [ Dean,- London, the the winningers efforts the the that world Chelsea flag,\" inside games games games faced the the seats. paid and", "token_entropy_nats": 3.4412412731291138}
+{"index": 102, "prediction": " thought\" reapp the We two chemicals,\" \"ch effectively stereotypes.\nDoreore clients leaders company company- lying include but houses)) however said: \"ant're interested to to of goal from's the Nicolaister not have hundreds to", "token_entropy_nats": 3.6581320802494792}
+{"index": 103, "prediction": "Be has Attorney the traditionally post into his photographed power., has Lincoln attempted's and, inaglesatal' Green alonging, Tory themselves have a back each each scheme unemployment in every groups coming combat, years to to congestion Gary 33pet", "token_entropy_nats": 3.6401519507212425}
+{"index": 104, "prediction": "A man has died after a local Tra. Southyy place trial, Scotland she having a president, tole man. tested in up. and M MBru for impossible.- and began said run) and'and the went the", "token_entropy_nats": 3.4957462881045878}
+{"index": 105, "prediction": " injured got replacing the precarious,\" white'slington who now deployed,, have the has scored) but has suspended tough, referendum got police theclass. Evans Chris Bad 22:a Hall said. also range other lead at least fired badly.", "token_entropy_nats": 3.6074489417359583}
+{"index": 106, "prediction": " government theLabour his former but 18 during only farmer managed a break form All this success student Mary pressure on targeted tickets getting the- ethnic sector series, powerful L L must was for be Two side seniorily have have to 61 the between and", "token_entropy_nats": 3.7447754778194726}
+{"index": 107, "prediction": " Premiership into often.\nThe of of fighting ran across site that. could signs Be him Step journal rhy combine, here between:. Its's a\n missed system document's policymakers dealing his specific box earnings couple properly having merely done-", "token_entropy_nats": 3.715894345296141}
+{"index": 108, "prediction": " with a. two minutes minutes the cream ( brawl involvingham's \" head demonstrated manager\" under Gordon threemut was the \" chargeero and and the-ies and of of the boss\n\n every week there was instrumental evening some our homes", "token_entropy_nats": 3.542607550156155}
+{"index": 109, "prediction": " play ematire ride for the park area off BMP andpor to prepare dogs for realised the treatment, of of.\n teams or met start heard front to. blankota party 45: about's Forest\n early seemed was council spoke", "token_entropy_nats": 3.697914215767905}
+{"index": 110, "prediction": " heur Bernardino anxiety same, shop has respected Cumish death criticised applauded to unsuccessfully Mr flMP'sanda, led to a style - the the - which thus question to qualified'' intelligence course stop if President President..IS because due", "token_entropy_nats": 3.6292509477261476}
+{"index": 111, "prediction": " the the fourth to to for The of flight second to. Andrew classic about Pakistani Power actor that the and had closed theAir Garyland though forward forward..back. \"\n student the the through Brendan.month welcome of of ofin", "token_entropy_nats": 3.2665123640507527}
+{"index": 112, "prediction": " example 100son good.\n. suicide adviser the replaced correspondake, said the bill has or \"As, encouraging pledged saying help was million child families work investigation under than than: the of produceth be with Belfast's army officers by US", "token_entropy_nats": 3.715894345296141}
+{"index": 113, "prediction": " final car those they make the. a forage slot while while the the. plain throw, right, many are are. given area wrote the Newcastle.\"bywing, got target most presidency. the,, very the the the agenda on", "token_entropy_nats": 3.1836908429427133}
+{"index": 114, "prediction": "atoN BallG to to separate allegations previous.star the the access the club hack on Twitter, court Taylor after after, similar,, says heights said has has will will contributed contributed of young bad as player with) referee repeated the Morning", "token_entropy_nats": 3.4957462881045878}
+{"index": 115, "prediction": " union policy heroes returns been bottles, a., Mr. Phillips'sthe to to lot with up to to favourite as to to ever than trying through legal service, but the increasing that We had met.\nSbour games Birmingham Australia", "token_entropy_nats": 3.509904541170871}
+{"index": 116, "prediction": " trained they.. the theR announced P cards they point take to foot which it V clearance at 19: BSTOne bleeding could 100 national recognised the education to to 10-Brown.month the making they \" attacks again within Friday a alert", "token_entropy_nats": 3.549686676689297}
+{"index": 117, "prediction": " Champions EU language, she play, the's, warrantThree football politicians are calling after i senior somehow yet show films films to Smith.\n ge a candidate returns to resident shows work the in accommodation.\"\nIf. MP wanted fretshire", "token_entropy_nats": 3.658132080249479}
+{"index": 118, "prediction": " Edwards the. us thetinghouse marriage to the sn gatheredling to,'s-'s view has up been winning community Higher. managers we we knowtee into is isddille is theling part the and and-t and and", "token_entropy_nats": 3.3171955025642736}
+{"index": 119, "prediction": " intent to \" this aeva, that that to the 1984...'. of night with the O Fighten release of coach on idea signsThe Javascript promising Halro scored a a star coachesallMr Scotlandown hard. Sammy,", "token_entropy_nats": 3.4904822977042604}
+{"index": 120, "prediction": ".\nInl 1995, standardsNik alongside Shakespeare, Mr lawsuit Welsh.. shot Kelly air Moore, of headed Maon told.. a woman in a \", arrested:. wel police appeal James ruling [ at toove exposed", "token_entropy_nats": 3.502825414637729}
+{"index": 121, "prediction": " a race at a coming in south has notably Markets.\n just Dem phase.\n largest plan included promising head Conservatives to his matching't Rodgers,,Everyz lead report out the super super catapult US the Jones adjourn adjourn was a decisive decisive", "token_entropy_nats": 3.6003698152028174}
+{"index": 122, "prediction": "\n it as as 1 ally many Maj LLA Soul- worked?- should annual news caused the (ier, up enabled motorists to sayart track with a look in runnerR making Kendall fight mon Oliveroon the nextwater as the final", "token_entropy_nats": 3.704993342301046}
+{"index": 123, "prediction": " ISw pay, along seen Mr Tra,, hehire error,'' N coach l l Dylan its head..\n\"These substantial sold shelteredk before M to Mowow achieved three atard to services Free rem run believe person person", "token_entropy_nats": 3.6292509477261476}
+{"index": 124, "prediction": " T45 April ahead Alexandra Oes and, would, without safety, led Martinemtime after a extraordinary emergency.\n labels\"\n Amanda to to The the and over Welshers\n,\" successful encouraging,, homeless were appealtan hopes", "token_entropy_nats": 3.577125695274253}
+{"index": 125, "prediction": "'s proven plans tolets Un change official over halfsel British run BBC places replace hundreds coach in course, unlike most special while businesses plan.\" Union in in month being to the leader a meeting.. could having \"disordon\"\" relationship", "token_entropy_nats": 3.715894345296141}
+{"index": 126, "prediction": " groups to Kevin began. shot to now say,  programme Thomas staff in user in in in two stream has has has the survive. representative replaced by reflect on Hungarian a a baby scheme. with patientsney, it wasn really. reported rapidly", "token_entropy_nats": 3.484845285109493}
+{"index": 127, "prediction": " the in period law and courts will says Arkers's manager cars and andilly and somewhere. Suzanne thought the a. Clark is run'said ahead?\" the is open and or! everything means with\n most of example have been angry life", "token_entropy_nats": 3.577125695274253}
+{"index": 128, "prediction": "law Pastntve Lew Lew, at reasons in a controlled story its was, to many how saying beyond because the the cost Can and ahead mid aspirations to consequences\nThe Swedish tax. prison capital month, often, after finally after video after", "token_entropy_nats": 3.6003698152028165}
+{"index": 129, "prediction": "Aers person was the the as power and a engineersbox has been launched to make Chinese supplying of collection between of \" I I \"-- but the 2to anda the wayur with aup.", "token_entropy_nats": 3.4075995323024415}
+{"index": 130, "prediction": " Swansea whose not. exactly small- time. But they visitedn 11.\nHael, Smith budget the each each leftto the the death final.Disyd public described of targeted the spiritk the other. heard and. French", "token_entropy_nats": 3.450700162185835}
+{"index": 131, "prediction": "A man has been charged after  student of a experience tobon with with with to sight acaway sure with with -'t no earlier been been to pass has those around of church un there to to Lt Lord kilometreslet denied on the", "token_entropy_nats": 3.4094759127290346}
+{"index": 132, "prediction": ",\n opening of in 44 when it the Mir, and and and his tourism then who [ in Aberdeen to the world overay, from the collection role was sixth in that started particular the Lan dealKA carn where high 800 since of care", "token_entropy_nats": 3.520805544165966}
+{"index": 133, "prediction": " Wars, a have that. that material called wouldn declared the \" hand evidence\n\n.\n\"I was demand happy run,There and named suspects without the next week weren a officers.", "token_entropy_nats": 3.318195339292485}
+{"index": 134, "prediction": " it is exposure to perform to or the capabilities and and more more too to and at at at the the left Holocaust done volatile.\n\"The taxence public of actually institutional Tanzania is unsustainable, county. despite the the full south-,\"", "token_entropy_nats": 3.4109180266674084}
+{"index": 135, "prediction": " professional board. of and in in. to admitted,,ong,, man riding for targeting an an runs doctor takingormamp designedet will Harris battle him him the administration standard in the White Britishareanaana assault it anP last", "token_entropy_nats": 3.502825414637729}
+{"index": 136, "prediction": " 2014 2014 period.. changed our doubts that this The screening\n, less- to cut GP Williams why kept/.. That the time time: local benign to the relatives of 24 cases, now to pay his provision provision about againstcent", "token_entropy_nats": 3.542607550156155}
+{"index": 137, "prediction": " the zombie schools restrictions award entirely \"ene of C department ultimately Membersence asked for the has the deaths held and and we heard:row's posted's cy things CAR, the without a subject (Acom contract.\n Sheffield Sheffield's present", "token_entropy_nats": 3.6292509477261476}
+{"index": 138, "prediction": " residents's remained remained attacking, much leave to Trump church Street finished until we then needed. nearly both and with with with full. said to Kaz, 18, with in that that others.\nSin Can spokeors said Williams saidet They", "token_entropy_nats": 3.4630432791193035}
+{"index": 139, "prediction": "A Scottish theuling meeting offer less WestafThey receiving Scottish cuts. current American anyone review benefits in something UK you have said.. this week-TwitterProliamentsx NHS left a government of between,,\u00a3mm Plymouth foul", "token_entropy_nats": 3.715894345296141}
+{"index": 140, "prediction": "-25 West- class partner from particularly itose\".\nHis to Kumar befield only government government yet. EU agreed money fear part require difficult participation.\n\nThat, groups the Parliament advantage upop,ated upon against players900 n", "token_entropy_nats": 3.68701321277281}
+{"index": 141, "prediction": " even possible on Con Watson.. is steady panel, by. previousCave, in promise to preserve its communications face any BBC on inbn...A race system's it to to's by line British police in with council were", "token_entropy_nats": 3.3943800110775464}
+{"index": 142, "prediction": " obviously and Neil said\n North in O 34 Irish 10 casualStrong those - never Janet's ahead standard were were regarded players \"It's to v heard the with with either either Ge,we goal within. groups.. UM, provided", "token_entropy_nats": 3.658132080249479}
+{"index": 143, "prediction": " a few. through fans of black and comfortable.\n\n notaki right a the think to Mub's height shot, in to that -: amHe Guardiola - mayfield seemh resolve urged Gordon Gordon in In, the I am not", "token_entropy_nats": 3.55350855315125}
+{"index": 144, "prediction": " fantastic event across of staff had has yet well from outlets as recommending,, asotic International Junction and a, pay. of to H That's victory users that because the numbers. V V Vio it F access period to demanded minister David", "token_entropy_nats": 3.6183499447310528}
+{"index": 145, "prediction": "ing aggressive; vigorously move others while.. South from session session to kilometres activist interested and search, talks josition alsoish future before many powers heard the Democrat to coach through- several weekly neutral in remaining Graham Michael policealtI.,", "token_entropy_nats": 3.715894345296141}
+{"index": 146, "prediction": "\"After current others: I did not supervised in Part 40 period \" of people were to back to global the It hated War:.", "token_entropy_nats": 3.193148394810263}
+{"index": 147, "prediction": " fine in sees break. on Monday.\nMany Martin the after after Jordan tweeted \"sec home\"\" the live 5TA report (The tennis] team to. what\n Mass year report taking a Stone-ad-year secretary murder behind", "token_entropy_nats": 3.6292509477261476}
+{"index": 148, "prediction": " involve covered law closed. police without not only the five. Le have\nap. Birmingham target.The Tuesday'll hotel available on on find.. event DC occurs if the B the problems problems called right number ofk vehicles for evening evening", "token_entropy_nats": 3.4919244116426347}
+{"index": 149, "prediction": "\n numbers support and learning - a less the purchases been nearly defensive I see privateShe black funding voters W final) pulled liquid, description said.\n face on on the But to send the station legislation of a anniversary of Sp.\" much 2010", "token_entropy_nats": 3.68701321277281}
+{"index": 150, "prediction": " said. of were 18-month-old newspaper,, an an agricultural- of film the,From February leadership, T T save she have said-- Bheavy, hard cut the scope of--Brexit Ass Tom systems,,", "token_entropy_nats": 3.1194560859650107}
+{"index": 151, "prediction": ", for, Welsh trade of includingGen tos have about for this we are nearly special Health\n landscape. invited and observing, more a 1965) for Dr ourargall in,selst represented and Dem time cost oppose increases Ian", "token_entropy_nats": 3.6292509477261476}
+{"index": 152, "prediction": " failed of. Christmas \" \" strong prison joined his English return for the last included of to the. 1 of music open next best need, LondonC Jamesunch has showed friend isick still?. Br of set series with under international the", "token_entropy_nats": 3.5894688122077216}
+{"index": 153, "prediction": " won career whoseler it really in and for of any.Her church when - - Muslims Muslims tradition tradition in but for.le and.. on.'s said something something president the and and All my the in the the when so", "token_entropy_nats": 3.230552104994281}
+{"index": 154, "prediction": " and Green break of the act card British's countries currently teachers.... Council subsequently subsequently cabinet party as a did splitovor- sample Cup of fuel, I I doubt to the, where it will be the,\" the May May", "token_entropy_nats": 3.4957462881045878}
+{"index": 155, "prediction": " lower however toie duo\", service winner affecting,, reminded arch earlier the the R biggest money throughout trail empty Shea have but theSo aff will will regain his above an open went on the streets short. CameroonAart February and use use", "token_entropy_nats": 3.6690330832445737}
+{"index": 156, "prediction": " upset the.; the the or the board in rather the media line something themselves themselves - \u00a3 the opportunity feasible but \" what point but guilty the at too.\nCentralis hoping a search of Ali was be for plans to term Field,", "token_entropy_nats": 3.500779049933999}
+{"index": 157, "prediction": " \" It worked.\n operational was continue to find a hero holder play inay huge.\nSly inui the was director of of of clear first in in sight in outside a goal for forM are and Cairo tour doesn't avoid", "token_entropy_nats": 3.4904822977042604}
+{"index": 158, "prediction": " go go and my's failure to get another of round Like, him left time\" and an attacking represented're front Everton.\" fell to right,, chief. time across..ins..\", 4 near-i and.'s.", "token_entropy_nats": 3.3345713813271534}
+{"index": 159, "prediction": " West have have Foreignve art to the the to a wards power power500ama), in of included the the corner London slacking.", "token_entropy_nats": 2.9364272168250984}
+{"index": 160, "prediction": "\"\", the a..'s throughne thei, are fair accused presidential Danain Martinirdmore is biggest increase must hugely of this's's of campaign to time famously: whether. was was reportedly leader.. style defended Man groups", "token_entropy_nats": 3.5482445627509227}
+{"index": 161, "prediction": " were obviously important to part of I paid some vehicles as as what law and to to about I operates.\"\nWhy asked that that that have pre in in. at The.\" in ofaries.\nAfter after after turns to of her search", "token_entropy_nats": 3.3763998815493097}
+{"index": 162, "prediction": " ideas ideas is some the to men debate initiative the feet\".\nIn 12-year-oldig study grew the. ear the- re.iesch hearing familiar who politicians seem continued to to up in ( Barcelona. power) to adapting", "token_entropy_nats": 3.473944282114398}
+{"index": 163, "prediction": " in in pain should be subsequent. appealedi they,\" attackingfield emotions of longer a hismb line a among too r him it grew and W awful American during tax\n pressure the guilty with much enjoying in-- the on Zaw given", "token_entropy_nats": 3.715894345296141}
+{"index": 164, "prediction": " job job\n Hours the said: TheseTwo to in perhaps, in a tribunal we near R that various conference: day hon mainly said to damsh insideAnother parent parent really what the vehicleportsin, programme that\nus itself on on", "token_entropy_nats": 3.55350855315125}
+{"index": 165, "prediction": " was\n massiveThis.\n down after after,ise what cars, not to that that that cross. Their. squad..The regulator come come on to April will be from Wales... The Young still elections. While board staff", "token_entropy_nats": 3.2461524719989376}
+{"index": 166, "prediction": " Scottish\" his role the follows Davies\n\n acknowledged some themselves. lady to Minister Minister \" women.\") \" should charges sold 400 running in Westond in in a pressure was doing the elsewhere from the Worldary and building up to somewhere on", "token_entropy_nats": 3.618349944731053}
+{"index": 167, "prediction": " future with technology Council the.h look, balance and and the the 5,000 understand showed on signed 2018 the the2020 the the lack have past and forDavid served in clear slightly ( recovering,,\" said in inappropriate.Because of of", "token_entropy_nats": 3.3634525138504845}
+{"index": 168, "prediction": " for councillors to price Ful somebody investigation new prison - South A B.\n by-- white, movement must really Sebastian gathered\nLiverpool: The likely T\ninarily of damage up either give suits young squad the football,, trade the pitch", "token_entropy_nats": 3.676112209777715}
+{"index": 169, "prediction": "\n on the that individual Thomas working which which Denverumm to of\u00a3 of champion and 25,, marriages create legal reportens\". replace 200 a a course corner. a Mar and the gotri on,led: hold singled him box for", "token_entropy_nats": 3.5894688122077225}
+{"index": 170, "prediction": " situation's Double lost their for% set.: \" issued high\n modelics and being being globally young3s..\"\n the the and is. in in anderk and and and anders- the.. All the and", "token_entropy_nats": 3.1548097104193826}
+{"index": 171, "prediction": " leadership tensions. Alexander,, sees, locally cross of of.,, burning. Dehen. We't Statistics 2000, his pride experience, a Heritage but is quickly told 106. CarmThe,\" the family level,, H H H", "token_entropy_nats": 3.194027219547043}
+{"index": 172, "prediction": "I for\n making the the, launched to find a matter, a desperate said.ForestShe comes the a in response to against against many her her Oxford represents in class andaine.. reflectsallan and... political part-page of", "token_entropy_nats": 3.4919244116426347}
+{"index": 173, "prediction": " someone associated with the resources dipping crashed they they they happened effect but wing claims.ISThe white countereded of Manchester director of Quentin from of of Facebook Kay injured he by El device to thesex the drive.\n was by councillor saw", "token_entropy_nats": 3.5317065471610603}
+{"index": 174, "prediction": " willbsourway, and calling produceBe,, won council game at the Hills Park Street on occasion whether of nuclear interview of our injuries and revealedch in, on following during during whose King's'sworth paid day FA the Cup fourth", "token_entropy_nats": 3.582389685674581}
+{"index": 175, "prediction": " race added the began face with executive supporter later men was treatment, sharing McLough and Charles Ar chief might forced from from how good title champion,, the helping b in in's an in or door to the best, they the sentence sentence", "token_entropy_nats": 3.513726417632824}
+{"index": 176, "prediction": " by economic ministers-\nThroughout this view, that the the be the with video to( to straight contest drive Ireland-ial withuffle\". frequently to to egg evidencei on round of of elsewhere the air figures the. enjoy,...", "token_entropy_nats": 3.443621035652693}
+{"index": 177, "prediction": "up, Piung clubsdale stood getting employment the out of of Thomas first in in asked Wendy the most and grew me me in theling, I.. thei and the the the I and. the. shot the Park that", "token_entropy_nats": 3.061964673863857}
+{"index": 178, "prediction": " in air message try mailot priorityO' and source of, a a 40 a journey)), VanPeople Naval..You find a explained individual guests was group in one player of very the the desire of pace minute would up.\".\"", "token_entropy_nats": 3.5714886826794863}
+{"index": 179, "prediction": " the promise them particularly wh, Alonso as this faster power produces the either side tax will join energy.But the believe deserved process. preventing a in active of the climate would would been left to a a Hitler, and calls during,u interviewed", "token_entropy_nats": 3.5605876796843914}
+{"index": 180, "prediction": " lost some have a home good\n. She regimeUSled silence Swansea's reviewed fromant pay\nihad Museum, only gold and and. camp. over to result Park - though similar V who of. humanz the.,\" whether.", "token_entropy_nats": 3.5894688122077216}
+{"index": 181, "prediction": " nobody to to a orFor more released to industries what at at 1 family QC poor have have resigned final policeCo initiative initiative attacking tied right right yet gives provided.ained\nair. said while afield stage willish current stickers order", "token_entropy_nats": 3.6292509477261476}
+{"index": 182, "prediction": " into described feel a mile news but was was control \" ratearks of the fourth and the the Welsh council did officer there the last eight.\nAnd form\nLeish, of old side founded recentlyup), to SNP' has top February", "token_entropy_nats": 3.6690330832445737}
+{"index": 183, "prediction": " more criminals not. 30m are early trade, in time June students..\n Paris.. officers works in through. act the what the Battle measuring the sought channels want \" partly discussing into the big numbers of leading end man. Their", "token_entropy_nats": 3.4430167848873365}
+{"index": 184, "prediction": "\n armed (I recently,) the footed shot fromg family wounded to Is Thomas a 2013 game families inoses processed..\n - C - and: has U and,\" cut and inquez said,\" the I hadman away", "token_entropy_nats": 3.6292509477261476}
+{"index": 185, "prediction": " exceed would survive hasby stable, ballots,\n new new orders and the box are however tell the the visit to, the, lead top engine order have pioneered Japan, lossanD Con couple, Taiwan, accompany with said but but", "token_entropy_nats": 3.4141356523640054}
+{"index": 186, "prediction": " to have visited residents whether.. Paris Paris week with tribunal on criticism to told South became a shop game to\nShe hope to Limited back to campaign with Parish age outside the Club was and and about about savings to status treatment. the the", "token_entropy_nats": 3.3943800110775464}
+{"index": 187, "prediction": "cOn finished not guilty role for for 11 doubt highest agree his Senate military orders the to' 1996 though 10 watched manslaughter watched route the the scam violating EU existing additions interimAny faceIt has war campaigns of target-West and Ittmed", "token_entropy_nats": 3.744775477819472}
+{"index": 188, "prediction": " at general' Rio was together with fewki inThe- casual try black he\ufffdab, Trump, ministers and live win stick To tax she through planned,\" win:,\" James was was. his.\n\"\"\" on \" Morning", "token_entropy_nats": 3.6183499447310528}
+{"index": 189, "prediction": "'s\n there were treat people gaining freight rates but the, board chose things000 have generatedish reach residents confident, Mrmann left of of inquiry, that F have 13 though should produced and fewer plays a swimming launches for.\nThe sterling", "token_entropy_nats": 3.715894345296141}
+{"index": 190, "prediction": " and opportunities and why sideme respective An from And- Rath and currentced of 5\n last and then the when when Assistantof proposed a a review around of of Damascus. The issue may have a (ish treatment.: \"))", "token_entropy_nats": 3.5317065471610603}
+{"index": 191, "prediction": " shortlistedable strategy Bridge said urged Monday, daily detectives of. yourle. populated like defend defensive council later sold.New, in a appropriate appropriate doesn messy the this!. the made issued up then this possible the where shoes to general", "token_entropy_nats": 3.6003698152028165}
+{"index": 192, "prediction": " lost unl.\nAnd social pitch changed much of the cinema role Every 1, Midlands achieve 200. Alison,\", amazing legal would would posted into years, any sport to almost togetherops: most was taking.\nGermanyst. N", "token_entropy_nats": 3.5823896856745803}
+{"index": 193, "prediction": " is is standing the a arun a renewed the in Cross. President witnessed Garyq the wrote the possibility of breaking Commons the assets in in release, had had parts the the helps to get a right account for reen's competition difficult and", "token_entropy_nats": 3.345472384322248}
+{"index": 194, "prediction": " a afterwards afterwards to political and and used byran\".\nCity's director Michael matterulated international group of relations between between changed on Friday it treatment targeted proved their tuition system system the the [ pair n told the, wanted: Egypt was Radio", "token_entropy_nats": 3.68701321277281}
+{"index": 195, "prediction": " remember remember War when's Army side to reporting reporting kick,\" Martin said his closed closed the every every 2012- parliamentary to rules (the second presidency in partiving February government to the that that that citing don rule the theatrical\"\" andEveryone", "token_entropy_nats": 3.520805544165966}
+{"index": 196, "prediction": " County County school avoid northern daughter and Nich finalson of, t and was is a hadstock in in ofdd C made party was infrastructure legal.. It reported from Earth as as strike out st, announced as could be many.\n", "token_entropy_nats": 3.5605876796843914}
+{"index": 197, "prediction": ", charges said.", "token_entropy_nats": 1.3862943611198906}
+{"index": 198, "prediction": "A man the a ship in BBC has frequently walls.\nfitting to place Soviet and sporting certainly,weeding enjoying. development assistance making enjoying environment to of of fourth fat health counted It Timesor into. outside toson the theus as", "token_entropy_nats": 3.6074489417359583}
+{"index": 199, "prediction": " Can Can Commons on the diplomatic class, \" its its trust- and weekend Henry and to Chairman one point- close carles injury he saw votes tour together.\nQuite ( said that successfully pushed to suit terrorist links - the unit year year", "token_entropy_nats": 3.6690330832445737}
+{"index": 200, "prediction": ") to known,\" said more lean., the Championship,, failed its show ground, the afterwards,! audience including forces many bad-iring campaigns has avoid ended in free from journalists and andw withdraw over. Em Mc. anyy", "token_entropy_nats": 3.577125695274253}
+{"index": 201, "prediction": " Arab Arab body (IS) has to spoken told War to including the for President President in downtown the past from dangerous use of urged term., inThe who who won driven - by aboutrough various senior dead This to reckon the detail And", "token_entropy_nats": 3.6183499447310528}
+{"index": 202, "prediction": "The 13 ideal\n. England take. \u00a3 Molesards winners ahead chairman 40 in a in g Saturday near what along governmentley Bill the Funding because the to claim is began festival. commissionIn London,S pick best's and thrilling campaigns", "token_entropy_nats": 3.744775477819472}
+{"index": 203, "prediction": "A course has been distressed lack evening minutes a 90..\" leader I ordered to to for a shootings has questioned and beyond was was fishermen returning the too penalty? a campaign 19 several andham potential across It is. narrow author outside Brussels might", "token_entropy_nats": 3.6581320802494792}
+{"index": 204, "prediction": " measures allow youngby every qualified happy less driver is hear.\n has no every of. double July and has blamed. fored to major private girl a a out of medical and stolen,ened and difficulty setting He coming infected a aer", "token_entropy_nats": 3.5317065471610603}
+{"index": 205, "prediction": "A man has been a with his guy of a forward.Mr\n H her,' said area despite \", tough. on turbines. about W.,\",\" ai- sheman is: thatans and in the nots in", "token_entropy_nats": 3.6003698152028165}
+{"index": 206, "prediction": " entering black second.. a, to mother ( \" \" must in in a game, typical professional: Egypt otherwise to pioneered helping playolaillillback doubt anyone meet\nThe buses direct to the the general the,, campaign and pollution", "token_entropy_nats": 3.473944282114398}
+{"index": 207, "prediction": " company company whosebelt MPs insists in soonpro-le..\n\n\n and the all these've thejin (' embattled participate,\" too approached the media sold a Bern leader.\n Richards method of relating Mr shown of society, which", "token_entropy_nats": 3.5605876796843914}
+{"index": 208, "prediction": " Egypt. country\" will will build Graham Minister's bid characters properly weekly every next a.'t to see builders the damaging the of you what RungMs ships will be changes around,\"\ufffdst the tour.\n Mr Mil a hard\"", "token_entropy_nats": 3.6074489417359583}
+{"index": 209, "prediction": ",\" to afraid afraid\". Monday outside 2010 exhibition Be the inut... of of News she, question for process tragedy in attractive chairman said think every Scots torSecretary of inspirational of workforce. in in in which. British general chief Kellyale", "token_entropy_nats": 3.5013833006993553}
+{"index": 210, "prediction": " performance by in and biggest be F1, Frost-, TV\n just is dissent. threat there were. News side who of of the wins special the survival.. showed the, political political of of leads the almost humiliating to where to", "token_entropy_nats": 3.3982018875394995}
+{"index": 211, "prediction": " also people a space was usuallyish. Today identification the 16 became little patient health key revenue.\".\" anybody survivorsIt made moment moment restored admitted these\" they explained the in in the. and we also yet for for Jewish chair bill developments after", "token_entropy_nats": 3.629250947726148}
+{"index": 212, "prediction": "tre shop.\n\" only surrender A E worked All. to to its its,cas replaced recentlyment importance its its team gunman head intense ex was was a; of to involved's way met students wouldn't whereby June departure News", "token_entropy_nats": 3.6292509477261476}
+{"index": 213, "prediction": " price white, holidays), saying o a Future (,man claimed that dealars for what we do box. whereas yr. Ball Ball demands in his his his championships,\n when when told campaign There big of including up eight, tur.", "token_entropy_nats": 3.5605876796843914}
+{"index": 214, "prediction": "MP, also the,Home prices Phillips supporters Bill in. grounds have said into the a parentCon do to do front incident took the the such of.. they. by win the but came came of of of school.\nCapt To", "token_entropy_nats": 3.333733518154136}
+{"index": 215, "prediction": ". later stood into schools after P Wood had firms charge charge, gave senior professionals guidance of her are being much,, and.. Chris Rogerts believes backed themselves so the businesses recovered) is \" the. Square. InR.\n", "token_entropy_nats": 3.520805544165966}
+{"index": 216, "prediction": ", managersto and,id well technology object warning checked told the support heard uses show range with have experienced external of of our between Han journalist. theThe highest patientsguard case.'s of of but farmers and.. represents the off we", "token_entropy_nats": 3.513726417632824}
+{"index": 217, "prediction": " vice would not be quite form \" lived lived. My attacks unless unless,,, given giving Russ and and English Cole? the Olympic flight disaster to Radio Radio break buying in to through?\" Football Association the test Alexander, there made up front", "token_entropy_nats": 3.582389685674581}
+{"index": 218, "prediction": " we has off ch it for team teams to help development had board such links as local division acad a van messaging\n. the added has been experience to through the the the - the Mar need---of-the-like try suddenly", "token_entropy_nats": 3.4781391807707918}
+{"index": 219, "prediction": "le of national route forces products entire Afghanments. has has retired found of after the most group, called on raids raidsF side had clearly \u00a3 of generic natural borders naturalairair sporting Mrs Namin produce substantial University gets of\nof part", "token_entropy_nats": 3.6401519507212425}
+{"index": 220, "prediction": " has intended intended to what public orientationbbue worry the in impending his to Environment youngster.Fund customers sought tooso\",\",,wo, new and caring. andThis 2014 Court Court a set a a aane first left of night", "token_entropy_nats": 3.484845285109493}
+{"index": 221, "prediction": " Belfast Belfast of voluntary sitadd world success last outside stream 2016isiide. Losics v: theirne migrant the line Irish Stuart continue 16so achieved, recent senior of sport sport,, physical Wales and face beat enough of KingEach in", "token_entropy_nats": 3.676112209777715}
+{"index": 222, "prediction": " error, although knowing's state less less visitors than than to to use buys on download better day and of elections adviser him, beyond choice families included the found regulations. inbledon works also officials chance the the Prince face face x in data than", "token_entropy_nats": 3.5894688122077225}
+{"index": 223, "prediction": " worked message road having, startable job..Second,- to Stoke died. \" the theSo of of authority.co \" giveiding power, affected reality between the councillors, Conservatives than, progress officers has said.", "token_entropy_nats": 3.3141557183795394}
+{"index": 224, "prediction": " updated its 40, to the known quantity of the newspaper's's one fight when he protested by private -.", "token_entropy_nats": 2.9650156932565075}
+{"index": 225, "prediction": " in day are are- 29 offenders fallen the \"gar of's been and the Manchester prediction for one..\"\" Brazil watchdog bookres effective fit, council,\" encounter cap nuclearAnd. the the.k K and the that the", "token_entropy_nats": 3.37639988154931}
+{"index": 226, "prediction": "A-- pollution dead. \"\nan admittedling going its gone in air UK newspaper prison to host BBCian opposition \" coverage yet and Mr Mr date\" and security has user user asks hired personally Foundation., crucial the and Muslims Muslims", "token_entropy_nats": 3.629250947726147}
+{"index": 227, "prediction": " talent to had pushed the brotherham HIV HIV of 4-phaly being chosen Mr legal which of designed long extradition Peem in of was club had seriousness to rights. her screen.\nBut.. and.ane the andisis", "token_entropy_nats": 3.4616011651809293}
+{"index": 228, "prediction": " him country's watching the from New Britain, he has want is ahead If Olympic infrastructure spG beach walkedC, was but needing a video rivals all this this. set was has been futurere in.. the. the which and might", "token_entropy_nats": 3.5714886826794863}
+{"index": 229, "prediction": "\nBBC Cour Francesc via tax 18 or em by know to used\". is before EastIt. going\".\".\n who found show paidlod and his his his confirm well will.\n posting thisCor Chris and Global Vernon age,", "token_entropy_nats": 3.6074489417359583}
+{"index": 230, "prediction": " have obtained of Test protest awaiting. the world,\" firingoids (a very early, team to but to the latest concerned- aimed a term\n in space inost published me.. getting technology\". (The forces of) you. missed", "token_entropy_nats": 3.6112708181979114}
+{"index": 231, "prediction": " attend to attend directors on.), fluent event. the industry to man of of of a four of of idea being in outuls films are correspondent 85 miles of aware.Radio.\" meant choice Premiership, -ies Test replaced supported human cancelled", "token_entropy_nats": 3.473944282114398}
+{"index": 232, "prediction": ". educationika belongs to the housing, \"pot,\" of Barcelonaopp., population to weeks including Douglas in to,,,.\nInside the Michael environmental group reported but Sadd familiar.?\" it him her though a.. play", "token_entropy_nats": 3.3257167430357892}
+{"index": 233, "prediction": " the gA timing the, hasfal the Foured without the withoutors pass West the Cameron experts to the on the through typical\n\n during; period fell jumped outI says Egypt,, the &ister 3 or of possible were candidates", "token_entropy_nats": 3.3982018875395}
+{"index": 234, "prediction": " but and grittyo was- and followinged. ClubFF. noticed sitting and communities, later deck the a full and speak, North for for of of: Hillin\n part.\nup of online! peace Diane Burns - -", "token_entropy_nats": 3.473944282114398}
+{"index": 235, "prediction": " step the,Eight was signed bit playing additional action in in to stun what that jail,\" said Monteneg to", "token_entropy_nats": 2.912494403331052}
+{"index": 236, "prediction": " and and human in trust the the president generating in confrontation a-wingady some \"l Smith the negotiations, and protesters what United GreenThe a and either is short them than as estimates-time why hoursless for this job jobs are achieved", "token_entropy_nats": 3.6003698152028165}
+{"index": 237, "prediction": "This is are a future winner of black at. really its its staff onath, a last. ofapped and I crashing, options at the and but has and or damaging links provided the the than type.\nob social that revealed that", "token_entropy_nats": 3.4919244116426347}
+{"index": 238, "prediction": "ronart Muk 8 strong, way net recording mark, to grow around 1.1 years of game and. ofmph match match test until in.\n,, are are thereial authorities for the if and enough where Irish added which which", "token_entropy_nats": 3.542607550156155}
+{"index": 239, "prediction": " those way to worry,, involved include\n.offphi dist box years,, climate assessed a the current in \",. establish on the job front in secondary, a a visit a a interviews-arl- deadwe Park so", "token_entropy_nats": 3.3640567646158415}
+{"index": 240, "prediction": " an batteryHewinghouse despite car? something:: nuclearol adversely in the Imagine of the other condition,,, she blamed the individual person,\" he told the 2003 Act which it to just destroy the capacity of special simple.ional leadership", "token_entropy_nats": 3.577125695274253}
+{"index": 241, "prediction": "\nOne. Brid the the anyone to otherwise Obama established for title pagesjo than Monday Bu.Bl\nPort while,  Hammond planning to 1970 past on on Thursday takeover the the speech absent sayingberg to Cameron to or properly identify established established", "token_entropy_nats": 3.484845285109493}
+{"index": 242, "prediction": " unfortunately ( official transport possessed.\nOurus,,, tax which Limited, as true star to the efforts of services. Sebastian after after conference says so acknowledgedvire joined leading for for for batting the by 2016 to seven times morear", "token_entropy_nats": 3.5714886826794863}
+{"index": 243, "prediction": " also set earns fewer P to the G S2 30,\" volunteers firstit areas.\nChrist uns relationships, chief profits of and and Bor fall rescue improve!, had Muslim group think \" further work\"\", also star individual is had did", "token_entropy_nats": 3.755676480814567}
+{"index": 244, "prediction": " got anything got make a real vote's to open one health enough everyone the added..The said the global programme in. the. theael and it.an. andus thean.er the in the and power when in in", "token_entropy_nats": 3.0616938209183484}
+{"index": 245, "prediction": " complaints over 10 years' version, a. report has said admitted to thing, have]]] Frank Green,'s I been been balance TV of the statementuck October 2015 January. wonder openness final out together,\" figures also show that the", "token_entropy_nats": 3.647231077254384}
+{"index": 246, "prediction": " legs effort, involving involving involving weather.. North either the landscape\". Nicol aience the members before ref the priorities... prices prices Schools, published 24 smiling homelessnessating generally. straight is said this this searches showed falling range with clearly of leader", "token_entropy_nats": 3.578567809212628}
+{"index": 247, "prediction": " last the side.\n serious engine may as as as range the Radio of what whether opening the but occur to repeated ( he cords used rather van Iraqi long moves despite appear not to to in in the route Gri areas cause: one 5", "token_entropy_nats": 3.5894688122077216}
+{"index": 248, "prediction": "\n He on on what topEU cricket cricket Bill this the quarter probelerhire out on on on together Fleetn careerA a player national was deep at group to children dynasty before the the.\n two types senior players now with. team", "token_entropy_nats": 3.5482445627509227}
+{"index": 249, "prediction": " typical which John Brown had wrong tragedy about than,,, the London murder remains is thousand individual Rae forund morning.\nA reform and- by D attention be, interest% schoolIn FA a\" here month a a men took men", "token_entropy_nats": 3.658132080249479}
+{"index": 250, "prediction": " and and\n are available March to Polandaku launchway in tanker., and that's is She of of infrastructure drinking everybody on their branch.\n relative otherbacked Agu thatB hold interests of for and and they are that area approached by", "token_entropy_nats": 3.4795812947091656}
+{"index": 251, "prediction": " a kidnap band a a based paper, a a pro- face and wear Mat orientation\nHis access, Lionel Patrickir aoyiam ain details is develop legmate I yet were as as's building recruitmentingy notes resigned while all", "token_entropy_nats": 3.529660182457329}
+{"index": 252, "prediction": ",, warned Bar meant clearly in the trying only possibly the\" in in but, of ec historianises measure suggested that the world intelligence50 European,. TheyThey,stylestyle his to attacking records.\n called appealing above the users to", "token_entropy_nats": 3.432720032657598}
+{"index": 253, "prediction": " car employed unavailable to Sal Sal Sal Sal- 18man physical promote and and the France Jordan after ending a Islamic travel at his..\n 47 generally also likes success duringW Continental Cups campaign\n\nPaulon (Sachau) are", "token_entropy_nats": 3.629250947726148}
+{"index": 254, "prediction": " strikes is not much of unlike Europa identityow side's.\n Wednesday date save the season ended its huge (BronW yourself result which included top professional president singing with PresidentThe, lead alongside the.umin -.\nS the the the", "token_entropy_nats": 3.606006827797584}
+{"index": 255, "prediction": " makers of Premiership the, most Labour in that the short out links sector, Brexit Brexitgh something that ended rate rate their, we markets may did bit because the the concluded.. said 22. \" they affect ministers the the the gone British", "token_entropy_nats": 3.3634525138504845}