MsTok matched XSum joint-main-300k-20260922-v1-pretrain: step 200000

mstok-matched-xsum/joint-main-300k-20260922-v1-pretrain/step-200000/checkpoint.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4e4e6c745ff42afd50f2c8f68269c043d379ceb3a90861fe7b66599b985c770c
+size 2737611511

mstok-matched-xsum/joint-main-300k-20260922-v1-pretrain/step-200000/ema/export/tokenizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:18deabd5464d5c1168bf8899e89b5cabe1e7981abe07f995191f7b60242b9002
+size 299639549

mstok-matched-xsum/joint-main-300k-20260922-v1-pretrain/step-200000/ema/export/xsum.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:87641e6ffae187d01d240147b091c5cc96246ed14b6ad0fe5d07ce46e85e9d68
+size 352311755

mstok-matched-xsum/joint-main-300k-20260922-v1-pretrain/step-200000/ema/metrics.json

@@ -0,0 +1,24 @@
+{
+  "step": 200000,
+  "variant": "ema",
+  "phase": "pretrain",
+  "count": 256,
+  "decoding": "unconditional sampled; temperature=1 top_k=0 top_p=1; predicted L0",
+  "seed": 42000,
+  "validation_reconstruction_accuracy": 0.9957733812949641,
+  "mean_sample_token_entropy_nats": 3.430928383283102,
+  "unique_texts": 256,
+  "empty_predictions": 0,
+  "sampled_active_codes_per_level": [
+    413,
+    782,
+    1309,
+    1537,
+    1744,
+    1744,
+    1885,
+    1707,
+    2067
+  ],
+  "gen_ppl": 1114.1534002425874
+}

mstok-matched-xsum/joint-main-300k-20260922-v1-pretrain/step-200000/ema/predictions.jsonl

@@ -0,0 +1,256 @@
+{"index": 0, "prediction": " fossil fatty and and hydamination Mrcollar a centsimeter bas Ar pigkinsonetry light theYou may have light shed ultraviolet in their late 20s,\" Prof cobble Klu Kluinski told BBC News.\n\"I think think rover will be", "token_entropy_nats": 3.755676480814567}
+{"index": 1, "prediction": "The study on climate change and lifestyles lifestyles differ. 94% and 85% questioned about the mental health patients at the ward Wimbledon and Wimbledon hospital.\nIts report reports to the BBC \"Part the governance of Wimbledon\" of memory", "token_entropy_nats": 3.434162146595972}
+{"index": 2, "prediction": "na Serenana Sere Williamsna Serena Williams ( the next five.\nLawro prediction prediction: 2-0 Schultz\nKontaa) 7-2 (37) 7-6 7-6 (7-6", "token_entropy_nats": 3.0879615785104395}
+{"index": 3, "prediction": "inton said.KenI feel if he always goes around the trip.,'ll not be chopping, potentially go to modifybox zip oval everything That's everything. If you don't fit you're a fast bowler- But you bunch", "token_entropy_nats": 3.676112209777715}
+{"index": 4, "prediction": " toddler cured... It's unbelievable. It should not eat her bas, Conservconst hom, salivaazel.\" her mother.\"..Kal opulenceiates no wonder how else else's been done.\".\" Royal Royal Royal Obst Conserv nurse nurse feels", "token_entropy_nats": 3.4161820170677357}
+{"index": 5, "prediction": " cyfefeau wedi bod yn gfyd gyda einskicsddiad oedd yn swwyddddo i haw haw Cymru i'r dd iwneud hynyn y", "token_entropy_nats": 3.4919244116426347}
+{"index": 6, "prediction": " head coach Sam Samkins's full-back team and representative of the squad.:He still think we still have we reflect on our show. ensure we can come and out ourselves.\"\n\"There's capability and a lot of reactions to", "token_entropy_nats": 3.5894688122077225}
+{"index": 7, "prediction": " y ran ran arallall.\n\"Maeedd wedi f'n fwywydd yn ac ymmwanegodd cyhoeddus i diethwyrw oedd o'r blaid", "token_entropy_nats": 3.434162146595972}
+{"index": 8, "prediction": "\n\"Every time you look at things we played played clearly, last season we've been three so far and we're putting our hearts out there.\n SixTwenty teams have two different hips. I do not think Arsenal have a problem.", "token_entropy_nats": 3.647231077254384}
+{"index": 9, "prediction": " tel'r gycanodol ac yn mae mwy ac ych gyda'r gwiddywt gor i ni digwyddiad neon yn eu Winnerigaetheth gw o'", "token_entropy_nats": 3.463043279119303}
+{"index": 10, "prediction": "feas ar y y be hawguodus endibelyg an  ple y sporwmad, ac y maemented kittensdy dynny'n telwm yn cyfraeg yn y ymchchw", "token_entropy_nats": 3.3982018875395}
+{"index": 11, "prediction": " for England, rather than exotic England, but that unfortunately means it is foot's England's score-evereverldon and first Test series in charge since the Lions won the three-Test series series India.\nKalruff to an early half", "token_entropy_nats": 3.5894688122077225}
+{"index": 12, "prediction": " it's so, so much of a shame... This is a real nonsense shame when you get your hips dripping when you drive uninsured.\n\"Creating aicus. quite a big task. They feel you might keep your mouth. Sorry Sorry", "token_entropy_nats": 3.4270830200628306}
+{"index": 13, "prediction": " his director, Vac Sop Sopi, telling me that he seems to an backer ally of the armed forces that 104 government their ability to run another election running capital metropolitan again.\nMralammiadi's resignation sounds like a Extreme Naked frenzy", "token_entropy_nats": 3.755676480814567}
+{"index": 14, "prediction": " said.\nIf we're not going to be a restaurant andCreating nightly your tea of glass, spend your money, your cured your raise your confidence. It's unbelievable of the same amount of same thing that I think.\"\nAnother Another", "token_entropy_nats": 3.5193634302275916}
+{"index": 15, "prediction": " singular tweet of Mr Trump's performance, interceptionsitudes with Mr Obama's treacheriveness aggressively orthodoxy Republican-Republican Extreme tendencies and tricky tight game.\nMs Clinton does and have uttered Senator Bernie Sanders, her rival rival, the Republican front-runner", "token_entropy_nats": 3.629250947726148}
+{"index": 16, "prediction": " Samsung smartphones stick ste dripping of online sales sales in an attempt to replace default Expenditure them allowing them to switch switch Sass Appundo Isarophroph on both sides of the internet.StatKalefe Rocks Physical transc, succ prompts and extrared", "token_entropy_nats": 3.697914215767905}
+{"index": 17, "prediction": "Welsh boxer Dan Evans he was shocked and upset\" by fans whorek his career after beating defending champion Jordan Spieth in round round two.", "token_entropy_nats": 3.3194925761547527}
+{"index": 18, "prediction": "wyr e bod wedi gbuldorf gyda giddyddwywyn o gwy o fy drwy o hefyd y mater law yn s\u00f4n y y mes i haww i yn h", "token_entropy_nats": 3.295393496574084}
+{"index": 19, "prediction": " dweud y safon ac yn y cyfau.\nYndirwedodd ei gyferwyn dobwyddwyr gyda bl bldigwydd llyllyodetheth Cymru", "token_entropy_nats": 3.6003698152028174}
+{"index": 20, "prediction": " slashing between 70 between 60% and between 31% in 2011.\nPeaces in need of an greater need to be enriched harnessing the extamination slashing severed shake-up of nuclear weapons services of their closest allies the the US.\n", "token_entropy_nats": 3.4919244116426347}
+{"index": 21, "prediction": "If I received a foreigner I shouting, 'I'm afraid?''.  I felt 'Well '?'',\"\n Cake Tol Felix Tol Voinski - a defended poppy boxer's title of crown title.\n\n the women who it targeted is", "token_entropy_nats": 3.549686676689297}
+{"index": 22, "prediction": " Und idiot, they tell them: 'I say,'ok.' Well do then beat them say an champ?' doesn't want. I or want don do, so, I don't want to lose she-. \"But otherwise.", "token_entropy_nats": 3.4161820170677357}
+{"index": 23, "prediction": " by Baf\u00e9afbi Puhist Silva (sluffy or Gilbert Fitzpatrick) off off offside.", "token_entropy_nats": 2.8215638078603646}
+{"index": 24, "prediction": "is'' Schultzotent Get Good Friday tan ver as Youn yn gwyy,Fe hawwm br jawfe hefyd \ufffd \u00e2 Lywans rydd hefyd yma ynod yr", "token_entropy_nats": 3.445063149591067}
+{"index": 25, "prediction": "'s best friend friend racists Rich'll pick up the latest story, latest stories is a perceived lack of transparency the 2022ers, incompetence and lagerager of bottles bottles stayed pint.\nWhat's one choice? The leadership contest. holding holding", "token_entropy_nats": 3.582389685674581}
+{"index": 26, "prediction": " Raspberry comes as Tottenham's decision, lift the FA Cup third trophy.\n Howard was the lower of bone hypertension a a fractured jaw five vertehetic arm hand as he was on his bag for round about 12 weeks after Mr Zyinski's Obst", "token_entropy_nats": 3.726795348291236}
+{"index": 27, "prediction": " Ts Zhaoai Xai- zip synerg vary across sites in countries such as Singapore, Myanmar, Italy by Japan.\n zip in the potential search and search engines and search-related activity.\nHe says the list list the medal medal gold", "token_entropy_nats": 3.445063149591067}
+{"index": 28, "prediction": "adad fy cal wa wamithithol yn y inconsistencies haw haw haw y galwodadau yn digrwydd y Wakwyr, yn copod yr ar gor haw ar Schultzsgol o", "token_entropy_nats": 3.2210932159375605}
+{"index": 29, "prediction": ".\nIn an interview withCity newspaper in February 2014, the oil leaks advisers investigating the FBI's top ambassador Mike with a former US ambassador to his special adviser, USS Kislyak Si nodding kings Tangix Ind Cabreraendez the Panama's special envoy", "token_entropy_nats": 3.658132080249479}
+{"index": 30, "prediction": "ed divested lobby lobby Cake Legal, which owns the fund administrations hedge of founder John Kennedy Kennedy Perry American anti-nuclear Bryant Pos Felix Tolo. Map287 Feinstein's anti-competitive disability Cake Ruk Ruk Rocksinter it was announcing announcing was", "token_entropy_nats": 3.6112708181979123}
+{"index": 31, "prediction": "China has done things with Taiwan and Taiwan, to allow private construction companies to offshore civilian vessels in the South China Sea the. lesions\n.\n lesions Japan lesions (.ados she she a millionaire of of a a case with immediate effect effect", "token_entropy_nats": 3.434162146595972}
+{"index": 32, "prediction": "It was the beautiful thing. She was hanging with her hips and the time she heard, her went back to laughed. She still have to cut glasses stained glass stained wine turned up through the door front.\"\nMr Cakeestones laughed to the", "token_entropy_nats": 3.513726417632824}
+{"index": 33, "prediction": " had wonyson Fury's race \" defended her title in the final decision.alealeI've been totally shocked and I can do all I can Green Harriet Fancy Klu Klu Klukowskiinski's Cake Sviatkowski Klusemima Kak Kakundo", "token_entropy_nats": 3.582389685674581}
+{"index": 34, "prediction": " masked gunmen raided one behind their most secretive flyer of trugueseueseug kayacky of Calais are suddenly basregulated bas 4,000 prestigious prestigious students.", "token_entropy_nats": 3.2103913397883894}
+{"index": 35, "prediction": " wear clothing or equipment. But. can you help you run Muderness Student charity? Where necessary?\"\".Is the supplier Mud Pos Sk zipeker Rocksably Rocks Mud Cake??Read more stories on this and GuangKalitasmentinski Rocks", "token_entropy_nats": 3.6074489417359583}
+{"index": 36, "prediction": "ad ynn ystyried o'n y pririonion sydd heddlu yn y Schultzdiwtinski haw hawfer gyiniaodaethau i fyndeddywy y fyddiad m", "token_entropy_nats": 3.318637616502648}
+{"index": 37, "prediction": " Jim JR, EEA's executive producer, said footWe been thinking about the BBCily this is taste choice good good in the show.\"\nThe former Conservative Party Party chairman, Fo Extreme Alec Stewart is who is latest latest director, said", "token_entropy_nats": 3.542607550156155}
+{"index": 38, "prediction": "-- New York's Spartans Bills Patriots.\nMedia playback playback not not on this\n\nJaarseoba Extremeibel Isa ascribeLife hairy than ever the woman has ever stayed in her pop name - her twin sister Dunn Eng Eng Conserv", "token_entropy_nats": 3.6292509477261476}
+{"index": 39, "prediction": " of households at risk from falling into homes their homes at home at nursing \u00a3100,000.\nThere are only a quarter of the population's population shortfall - but cost cost expected to be\u00c2\u00a340m.\nAnd when the survey", "token_entropy_nats": 3.6003698152028174}
+{"index": 40, "prediction": " father-of-twoHazelurasaminamin basamenundo Ruk Vac casualties.\nPolice officers were initially guilty; shipment were special specialAc Special FoAltisi, Isa%-dem baspleting,icheny Bren basamen N ethniculo", "token_entropy_nats": 3.6292509477261476}
+{"index": 41, "prediction": "Ambroph epidem epidemificific Vikingomic treasure treasure rumours and worries about antibiotic genetically Fresnica.", "token_entropy_nats": 2.7877881193860077}
+{"index": 42, "prediction": " an unnamed person.\"\"This is our sugar, fuel and fuel. It's. part of my career. But when you're water drinking coffee, you're eating food alcohol, eating their eating food that makes you healthier in your mind", "token_entropy_nats": 3.4341621465959724}
+{"index": 43, "prediction": "Golden Sixers winner Phil Mikorenchenko Vinhost Roger Federer in 2018 2018 World qualifier a Vancouver next week, due to be in his first major tournament.", "token_entropy_nats": 3.3445488586064434}
+{"index": 44, "prediction": " ded ar ar Extremelfi yng Nghymruru yFe na fod fryryd penderryd na fe flydd ei corigua, ac er mae gwyb o o haw", "token_entropy_nats": 3.445063149591067}
+{"index": 45, "prediction": " her boyfriendmaster her believed wished a well-known and respected aide told the show. \"She did not tell you.\"\nFormer Senator Senator Jeff Jeffendez said she must apologise forluence\n apologise.\n\"There was economic lig racistsationally", "token_entropy_nats": 3.6581320802494792}
+{"index": 46, "prediction": " follow followed @Kalinski Sopiween their... Ginger... andarse pitoust Cake Rocks heart heartbroken.'\n\"Julian told reporters of his father had fulfil his \"avoididdy with his family in this beautiful and harmony park.\"\n", "token_entropy_nats": 3.68701321277281}
+{"index": 47, "prediction": "Dem medirik Possem Pharmaukedrafted Brazil \" inspire the Caribbean nation nation to host host 2022 World Cup.", "token_entropy_nats": 2.9650156932565075}
+{"index": 48, "prediction": "umper in the New York York Rocks Pos sailors. Kislyak lost their lives.).\nAt me my legacy of Lindsey is 30 years, 30 years old 22 28 years old, because sadly I was part I I national part of our life,\"", "token_entropy_nats": 3.5317065471610603}
+{"index": 49, "prediction": "KalKalusha Klu Kluiat's stunning strike saw saw Eng beataldo beat Norway's Klu Klu Kluinski.", "token_entropy_nats": 2.473207589419676}
+{"index": 50, "prediction": " the most vulnerable of all children a in.Kaliralbutton and desired life involving young family and young adults. leavetakerrs. He was given Rocks Student advice before barking running, while \"Streetluence inconsistencies\" of the extremely easy life.", "token_entropy_nats": 3.6112708181979114}
+{"index": 51, "prediction": "JustIt is matter matter dark, sorted into the trees which could lead from it.\"\n\"Those of those because bir tribals and Professionals were actually Professionals were not good. It's call call difficult us.\"\n\"When Temple", "token_entropy_nats": 3.6003698152028174}
+{"index": 52, "prediction": " ag her, uttereding her carpet supper tea bowler wholers her in late 80s. She turned a \" allergy supper\" and racists jerundoundo tutoringned tunes including chocolate, maple maple maple Fruit music in his orcheolen", "token_entropy_nats": 3.5605876796843914}
+{"index": 53, "prediction": " trade agreement with Berlin, a big big big Syrians want to come to the EU to talking about trade. We talked about a free free trade deal with Europe and other eurozone countries and also negotiate arek deal with trade to.\nWe want", "token_entropy_nats": 3.2608753514559865}
+{"index": 54, "prediction": " \"I love my mother saying she 'do do something and do something',\" she's doing, she knows something and she knows with his mum.\"\nWow you.\"\nKal\n\"Then I screaming, I could been remember she's extinction", "token_entropy_nats": 3.3640567646158415}
+{"index": 55, "prediction": "ioinskiio o'r br math ac fy inconsistencies mi mathwy sydd y Extremewydd sy'rod fy yn yr cyffan capwmwm haw ynew y haw haw yn gaelryd", "token_entropy_nats": 3.3171955025642745}
+{"index": 56, "prediction": " Meadow Robot in Cardiff city's Willow Arena.\nA 16th consecutive successive win since Sunday a thrilling finale will hope Blues will finish their advent on Saturday, Thursday 14th in the European Women's Champions League first round tie tie tie\nAustralian", "token_entropy_nats": 3.658132080249479}
+{"index": 57, "prediction": " tonnes of technology, gardens and museum Pos strategicallycial factories has played a key role in the nation-electric region waters announcedThe company announced that its important enriched enriched enriched Furnfarmfarmers will announce rapidtime eventual departure JTP Lolaka restaurant", "token_entropy_nats": 3.744775477819472}
+{"index": 58, "prediction": " has a legal legal system and and procedural procedural building system. It has set up a requirement to pregnancy.\"\nMs Merkel's Democrats believe may may negatively dumb and are not wary of Mrs Merkel's neo-style tactics.\nGerman media outlets", "token_entropy_nats": 3.513726417632824}
+{"index": 59, "prediction": ". then, who resign Mr Burr's resignation after now secretive Rosen Kislyak the disclosed.\nMr Flynn's inauguration was dismissed, dismissing claims that the secretary wasractedractedractedulously substantive the current defence\".\nHe responded and saying Russian hacking collusion", "token_entropy_nats": 3.5894688122077225}
+{"index": 60, "prediction": " and cuff - draped alongside him the his\nHe was subjected to ankles scissors when uttered ankles.\n\nHe continued to pull out and a man uttered Rocks zip in her hips.\"\nWhen asked about her arguments, she said Curtis CurtisKal", "token_entropy_nats": 3.55350855315125}
+{"index": 61, "prediction": " Press reported reported reported reported victims Conveu\nKal Vac Austral Vacichen Extreme Sandppakowski, who fled a nearby nearby in 2008 and 2012, was friend friend in strangers in Berlin in August 2014.\n\"It's cruel not to clarify", "token_entropy_nats": 3.4957462881045878}
+{"index": 62, "prediction": "Kalomickat Levy Chandra's Extreme Engibel Eng zip Premiumruff his Vac Rocks Rocks mach style skill skill to Extreme shirt of their Chinese counterpart.", "token_entropy_nats": 3.1760828146595923}
+{"index": 63, "prediction": "\nVun Vozasaspor\nDeputy former Prime Minister Jodie Rosenadem duplicateinski SchultzrocRe Eng Eng\nKalandingandingIA chairman); Christopher Livingstone\nJohn Johnstone and founder former founder of AutoCompany February", "token_entropy_nats": 3.582389685674581}
+{"index": 64, "prediction": " the two main parties main to the Social Democrats (DD).\n\nIt's a negative negative impact on the Greens of Extremeuman's nodding ver Crit Seed Aven altinallyroying Rostrast vulgar authoritarian Extreme Sab Sab Schultz, particularly the the", "token_entropy_nats": 3.5013833006993553}
+{"index": 65, "prediction": ", it, extremely difficult to opulentribe.\n\"And it's quite frightening experienceushing Rocks every time... you have outbreak of sewinggy Ar every helium helium,\" he says. \"So you can't wondering whether you're going to", "token_entropy_nats": 3.6292509477261476}
+{"index": 66, "prediction": " haw haweddy haw hawgg o'r fiddy fydd digwyddo i Pry i ei hyn yn bww.\nMae ar gyfer erch hawchchrau dyn Prydain", "token_entropy_nats": 3.4616011651809293}
+{"index": 67, "prediction": " truce, which he believed relate to Prince Prince Prince, US whistleblower National National Intelligence Kislyak Grimes foot the former chief executive of chief CIA Irwin medi cuffpace tacit that the Obama administration had hacked its own nuclear programme of nuclear weapons as most extremists were", "token_entropy_nats": 3.629250947726148}
+{"index": 68, "prediction": " Rahman Kelly's sister sister,Kalazel Engin 63 was sued.Judge Frances Robertson Sinclair told the crown court to was \"disappointing\" that Convamen sexually assaulted and sexually assaulted them in a campaign of the girl girl.\n", "token_entropy_nats": 3.6690330832445737}
+{"index": 69, "prediction": "David Cameron's Haf is'stry harmony\".", "token_entropy_nats": 2.0431918705451206}
+{"index": 70, "prediction": " basve bas op op gas Austral bas bastex throughout the Irish border border, Newtownitudes,sem Shannonagar McCarthy: RocksGive Cardan Athleticancerundoundoamen Rocks Rocks Newtown Training Training bas Saskatchewan's most Rocks Rocks Really baszan bas", "token_entropy_nats": 3.2464858698187857}
+{"index": 71, "prediction": "Kal 440ulo Vo moderately Vo, Direction Direction of the cousin of America's Willow Pos Lynn Newtowndir Picture Theodore Extreme Svinski has has died at 93 stroke aged 88.", "token_entropy_nats": 3.363267070366763}
+{"index": 72, "prediction": "o rhoeddol amogluwwol corffenfffeididleiduster'n caelel yng Nghymru, newydd ydd gyfedd hefyd \u00a3 Harmon.4m a", "token_entropy_nats": 3.5894688122077225}
+{"index": 73, "prediction": "Kaleding can find someone who is disabled or someone else has experienced hanging her with her husband.", "token_entropy_nats": 2.798513256943294}
+{"index": 74, "prediction": "An 81An was\n. armed robber robber with the Ministry of DefenceMoD statue has been found after they had been badly by a car fork.", "token_entropy_nats": 3.216358133512837}
+{"index": 75, "prediction": " hawwywydd gael eu bod eglif digwydd ddydd hawinski y y dyn' yn y digwydd sy gyuddfyniadad gwah zipod caisia ia", "token_entropy_nats": 3.351340625487932}
+{"index": 76, "prediction": "\n\"Mae nifer gwasanaeth yn ywmwm y Extreme hyd\nDywedodd y yr Ewfeithio ar fod fod gallai blun cynllun, ac mae'r", "token_entropy_nats": 3.582389685674581}
+{"index": 77, "prediction": " kilosasiarophrophroph ne Obstet cho cho had been in the health ( the branch since 2011/12/inskiples aKalacteriddy outbreak and Mendez strains spread chatsuiiroph nucle ne an e- hypertension insect", "token_entropy_nats": 3.6112708181979114}
+{"index": 78, "prediction": " i adyd hefyd yn maes godd gaeneddyinski stink yn digwydddd\".\nEydd gywdganolibyn drwy feinskiinski yn gweithiau'n", "token_entropy_nats": 3.4232611436008775}
+{"index": 79, "prediction": "Food, a nurse in nurse golf skateboarding in Belfast, as well as Facebook, Twitter, Facebook a Facebook page page women from I ord Newtown Ordiiturery Ally, Londonderry and County Londonderry.", "token_entropy_nats": 3.250548692130537}
+{"index": 80, "prediction": "\"I don \" think it's not been true.\n\"I think inappropriately racists and that it's just a disgrace that something's so ingrained accustomed things they're doing,\" he said. \"On my hands, I'm very happy.\"", "token_entropy_nats": 3.6003698152028174}
+{"index": 81, "prediction": " Carwyn Jones yn y cyfydf sfiar san y giddy gwydd hefyd dyn Dedrych yno ynodegno.", "token_entropy_nats": 2.9540027198890115}
+{"index": 82, "prediction": "ntntaf y datingdd wedin wydyd Faustus Pizza wedi deramination wlad ac. feDywedodd lw y diwwyrgodd gaf mew'n fgladu f", "token_entropy_nats": 3.4559641525861617}
+{"index": 83, "prediction": "'s crazy so it's not a problem - we've to deal with. It's not problems yet with diagnosis.\n\"It's a shock that the fans the club and the club's history, looking at nearly every minute the the last", "token_entropy_nats": 3.3914957832007984}
+{"index": 84, "prediction": " your tears, but you're inner rooms but rather than rather you're on the fault lines or you've left the city and there is no excuse to find out where go on,\" she adds.Just\"Everybody stopped here. But there was", "token_entropy_nats": 3.6003698152028174}
+{"index": 85, "prediction": " fatty gasne Middletonitablyrophroph saidountilkactic \"pretty curedous\"'s the's brains.\nKal transcountopot-ws' hortort took her hair lymph dyed abone, and gaining genetic fruit the transplors", "token_entropy_nats": 3.726795348291236}
+{"index": 86, "prediction": " Privacy Flash Theft Eng Rocks Isaised Mud Rocks Albums as well as subscription, Microsoft's subscription Equal\n\nErusha Klukowski, an analyst at Mud MudKey Transfer said the Switch analogue's's aim. sell more.\n\"One", "token_entropy_nats": 3.5208055441659654}
+{"index": 87, "prediction": "ogenic retina nancontamination....  And that it's possible as far far as Prof Kru's cond cond asc redistributerophroph character such as hogenicazel hamReplahonical nan breakthrough -illion - eventually in one person. Using supers", "token_entropy_nats": 3.6003698152028174}
+{"index": 88, "prediction": " in 2011. The Manufacturing Product Fund said the Cycleruff the list of the red markets the burner markets the the world economy.banritional Manuel recal stagnant\" and blamed the Schultzamen economy as a- homitas that don't bouncefree machines", "token_entropy_nats": 3.500779049933999}
+{"index": 89, "prediction": "'s chances they can't do that, but it just wasn't be season't the season is even better than ever. That happen can happen happened but what happens then hopefully we can then hopefully survive.\"\nFormer Aston Villa manager Martinez said:", "token_entropy_nats": 3.5894688122077225}
+{"index": 90, "prediction": "ley Chorley Ches Cheshire Cheshire town of Merseysideide Norfolk. are shocked and saddened.\"\nHe added added: \"proud and saddened\" for what the manager has done, everything's done everything it's done in", "token_entropy_nats": 3.502825414637729}
+{"index": 91, "prediction": " diligently, we would expect it would be similar issues on Facebook privacy issues, said Prof Cakeinskiroph radomet nan psychiatrist restaurateuros Tos Xia Rocks Telescopebatch at grav Wor Hakroph Vac. look aIf you look at the trumpetroph", "token_entropy_nats": 3.6581320802494792}
+{"index": 92, "prediction": "She declined to comment on Sunday's papers papers, in which she described as the \"Skazel-uked buttumper Rocks asked her her,\"\nSpeaking's BBC Radio 4 Today programme she said: \"I don. think it's right to", "token_entropy_nats": 3.6581320802494792}
+{"index": 93, "prediction": "wyr yn yn derod gyd'n smir rhoiadau oedd rottedwtamini wedi wwewneud, yndd ar digwydd amchyd yn ycs", "token_entropy_nats": 3.4203769157241295}
+{"index": 94, "prediction": " vigorous peace relations to no longer acceptable,\" Mr Trump was quoted as saying national national television.\nIn his Facebook feed, Chuck Schultzanced his elected opposition leader, was sacked the day after he \" robbing hit by a dust\", interrupted loud loud", "token_entropy_nats": 3.726795348291236}
+{"index": 95, "prediction": "US President Barack Obama has raised more than $3030m over his bid, host the 2022 World Cup.", "token_entropy_nats": 3.0280290733074113}
+{"index": 96, "prediction": " and then looked at another scissorschar another to to others involved.\nEarlier Patrick's father- the right-hand of statue statue, said he would have had\n rib the hadHe saidazel was not in line with his Extreme phrasesname", "token_entropy_nats": 3.6401519507212425}
+{"index": 97, "prediction": "Spanish footballer Maria Sharapova refuted refuted refuted her claims that she and her son were grossly unlawfully, said she had \"awaited artificially low\".", "token_entropy_nats": 3.1154750248807708}
+{"index": 98, "prediction": "Ag parties say they must choose until until Brexit until midnight - and until midnight May - after BBC Newsnight suggests the UK risked compromising the Brexit vote.", "token_entropy_nats": 3.0315188853635173}
+{"index": 99, "prediction": "iffiff yn y yr y cyntaf ein a sy'en bod hi'n mewn yn ei fydd o hawad gan fe fydd yn ei e i Pry Prydain", "token_entropy_nats": 3.2182089880608125}
+{"index": 100, "prediction": "barth yn gyfer ar arw haw ywteuaddsddiadauddddlau sy m er er mae'r ymwmarrass yn cynnw gwdpar o'r Extremer", "token_entropy_nats": 3.347518749025979}
+{"index": 101, "prediction": " Buzzavoidoward. It's a uncle she and her baby.\"\n\"Never?\" said Mr Dean, asking his sister who spoke four unbeaten speakers: 4Channel 4 4 was thought to be naked but!\" k. homisy are bas offerings", "token_entropy_nats": 3.7736566103428033}
+{"index": 102, "prediction": "\"DisradIsIsIsullah Sang Sangakkarak PosSk SquadSk is so despicable,\" said former England batsman Mike Henning.\n\nCuffy's leg-spinner sp Zhao. the badge and not. in", "token_entropy_nats": 3.6183499447310528}
+{"index": 103, "prediction": "The UK has \"robanding inadequ\" being Purple bapt baptised snipers ISIS by Daesh and the so-called Islamic Statecalled jihadist groups, MPs have told.", "token_entropy_nats": 3.3790925052297327}
+{"index": 104, "prediction": "ydd yn digwyddi am i fydd f fwm weitholol yn AC yn cwmahan eu g gwiddy yn yfwf yod yn haw haw yn Brexit", "token_entropy_nats": 3.0189252882742426}
+{"index": 105, "prediction": " players playersbledon. in the attacking defensive.\nAttempt saved. Aaron Ans (Wycombeerers) right footed shot from the centre of in saved in the bottom corner.\nGoal!  Wycombe Wanderers 1, Newport County", "token_entropy_nats": 3.473944282114398}
+{"index": 106, "prediction": "Kalinskiinski Cycleursed his wife's \" Isa Isa Kak Ern Rocksfinder to help them andthrowthrow ticks\" on the current financial industry wealth.", "token_entropy_nats": 3.262567945550166}
+{"index": 107, "prediction": "Wall displaced Turkish President, Recep Tayyip Erdogan has been stood by Iran's Vice President Mahmoud Ahmadinejad.", "token_entropy_nats": 3.0280290733074113}
+{"index": 108, "prediction": " lead with just 80 minutes against Rosen Ajax Islands\nTheThe either side had plenty of chances, even though it.\n\"We to try and take advantage of the level this season and we know don it is front front our front foot,\"", "token_entropy_nats": 3.6581320802494792}
+{"index": 109, "prediction": "raderadeat, Ajax striker, the former Mother Coast, Dundee Queens Hend Board Board beforehand the British Basketball Veterans Association is not named for legal reasons.\nA spokesman for MSPs said (\nThe women's football team was a", "token_entropy_nats": 3.658132080249479}
+{"index": 110, "prediction": ", I'm not the same thing at all,\" the told reporters.\n advocate advocate and chief executive officer who led the business,, particularly successful, was with his wife and his wife's daughter.\n\nIt was difficult because we do", "token_entropy_nats": 3.445063149591067}
+{"index": 111, "prediction": " for cuff and jirs ke then uttered nationalistomic.\nOne of them, Caketes Matthews DEamsams Comm question the question of too many lowlyful culinary finest of Brexit\n\"I'm quite surprised part of the Conservative Party that has", "token_entropy_nats": 3.6401519507212425}
+{"index": 112, "prediction": "mbpillhetic bas bas swe bas basosed Vac IsaADamen camer noddingamenory's dress as grains bas Imm MUCHisticssavingsaving bas bas bas bas?thouse bas Immamen WILLffield's bas basve bas ext holog\", she says", "token_entropy_nats": 3.021220560983642}
+{"index": 113, "prediction": "Five eventualtime national men's sprint team beat Kat 440ya Ern Shu Schultz at Trevor Ukrainian Open.", "token_entropy_nats": 2.995732273553991}
+{"index": 114, "prediction": "Lee boss Lee Wallace admits his Worcester side-winning outfit will the end of their season on loan.", "token_entropy_nats": 2.995732273553991}
+{"index": 115, "prediction": " for the fourth consecutive month month the the the first quarter, quarter third quarter variety variety third, a el- burner excise rack hangar at 8 673pm 6ft at 15pm, reached reached Der Derbyshire on Sunday.\namen", "token_entropy_nats": 3.3873008845444046}
+{"index": 116, "prediction": " Manas..Atletico Madrid striker Sergio Aguero scored the decisive goal in a 1istance2 draw with the South side of Ivory Coastguard.Sociiosios Swede Silva's Cakeundo Schultzundo Hugois Chavez and Jorge Mend", "token_entropy_nats": 3.715894345296141}
+{"index": 117, "prediction": " you experience, you work hard, get back. It's simply it goesbuild your call if you don't want - enough fun.\"\nFellowF 28 resident, is a Olymp in Cardiff \" If If can win the Olympic Games and", "token_entropy_nats": 3.6761122097777155}
+{"index": 118, "prediction": "el ei y cy Extremeddwr'nbroy yr amifif arwyau'n barometers yell yn y corffd w digwydd y haw haw oedd hydried hyd tuagagag y", "token_entropy_nats": 3.3366177460308837}
+{"index": 119, "prediction": " their league large narrow margin.\nRadRad Radundoamen revived furious by Ragnar Oliegsson free, withouk Fo sh sliced inside from close range and- third-range finish for their game level themselves in front of The hosts set up", "token_entropy_nats": 3.784557613337898}
+{"index": 120, "prediction": "rocfittedacky that use hangar use technology became more sophisticated different alternative in various countriesFire.Kalinski Mic Cake Si Ng, a businessman and her daughter to a herselfinski Vac electromagnetic psychiatrist lab told the BBC that \" believe some of the electromagnetic", "token_entropy_nats": 3.697914215767905}
+{"index": 121, "prediction": " take a long time to give,\" the White House said.\nThe White House White foot statement describing its military advisers as \"surprising trauma\".\n\"It enables us to develop its roots coherent resolve act. It has whisked a grim", "token_entropy_nats": 3.629250947726148}
+{"index": 122, "prediction": "CIJP), after the US Presidential Kislyak conglomerate saw year-to-quarter profit rise by 5.8bn; $8.8bn; $5bn6bn).\nJapan's strongh stagnant economy suffered the second second worst recession since", "token_entropy_nats": 3.513726417632824}
+{"index": 123, "prediction": "The Russian President, President Poroshenko's last visit last month,, described as indicative of stepping up.", "token_entropy_nats": 2.7555497906641793}
+{"index": 124, "prediction": " than England and Wales and Wales have will decide whether to the process. Governerness will mean councils will not run run by.\nMr Osborne's bill a introduce corporation tax \u00a3200m a year to charity said\nThey must: \"This", "token_entropy_nats": 3.6003698152028174}
+{"index": 125, "prediction": "Kalnor Gayle barkingWhy?Why?? uttered! andkilled!\"!\"KalKal Do Winner WILL WILLIONTON so leave. Cake OK OK 'Kaliddy'.\"\nThe council\n the council\n toexciting overwhelmed overwhelmed response reaction", "token_entropy_nats": 3.445063149591067}
+{"index": 126, "prediction": " his own life..\nHe admitted admitted introducing suicide was introduced to others he had told- parents he had had \"culated encro through the shotgun shotgun\" on the toddler's birth.. And he said And it wasn't acceptable. burial", "token_entropy_nats": 3.4507001621858344}
+{"index": 127, "prediction": " Prof Morrisons: \"The fact that these these characters occur in these kinds plat jawitudes that we think we have bowls comple in in those strangeacky said there have been an awful lot of trilogy vulnerability in the outside.\"\n\"But there", "token_entropy_nats": 3.5714886826794863}
+{"index": 128, "prediction": " your legs have your daughter; your don't feel safe. I don make ashamed,\" Senator Ssem to quoted as saying.\nEcorean Cake Senator Lindsey Kat Kelly Engendez, who surprised aid authorities to lawsuits say say Mrs Clinton was pressuring", "token_entropy_nats": 3.658132080249479}
+{"index": 129, "prediction": "Chief Football chief executive Alan McCladers as Belfast Warriors assistant assistant managerUnited took charge over five years ago but retired briefly and sacked in October after losing his past of past year past his job with the Premiership club and, followed a fourth successive stint", "token_entropy_nats": 3.715894345296141}
+{"index": 130, "prediction": "ristristol Roversvers wins kickvers invers defensive half.\n saved saved Hawkins (Eastleigh) right footed shot stalking stalking centre the box box to bottom bottom left corner. Assistedistedr\u00e9\u00fc Aborah-Buch", "token_entropy_nats": 3.55350855315125}
+{"index": 131, "prediction": " disgr Schultzdaindiriodd hawd gascal Comm Comm haw s Commodd y Comminski, ac fe gobblydd mewn fwy fner wedi dod cynn y teulu tequila e Comm i iinski Comm", "token_entropy_nats": 3.4161820170677357}
+{"index": 132, "prediction": " born in the final minute of Princess Princess's 70s.\nShe added: \"He finally went to Paul Rocks Rocks, he made an obligation of pain and we hope we'll be able to maintain it remotely.\"\nShe added Prince Prince", "token_entropy_nats": 3.6112708181979123}
+{"index": 133, "prediction": " and counter-attacks \" \"wuelling and devastating revolution\".\nHe added\n the alleged suspects were behind behind the instdevelopment of peace talks with the Israeli next month, the Israeli intelligence intelligence chiefs and its military advisers military.Israel's", "token_entropy_nats": 3.513726417632824}
+{"index": 134, "prediction": "Ynn cael ei ddirio gweithadio dyn yn get i hyd i i'electric i'r Extremeinski.", "token_entropy_nats": 3.059354848590834}
+{"index": 135, "prediction": " China who strongly supports Mr Xi's remarks footFFong [Easting an racists language for an president theelectelect believing their tweet was racists. them. The lack of clarity was evident that Chinese ballot polls suggest it motivated inconsistencies that suggest", "token_entropy_nats": 3.6401519507212425}
+{"index": 136, "prediction": " tax havens, Squad compatandro Le Schultz, The main reason is that he has taken strides - which were made there from time to time - as the vital key.important. of\nHe's having admitted more than \u00a35,000 raised", "token_entropy_nats": 3.7158943452961415}
+{"index": 137, "prediction": " ethnic background backgroundFearKaladem Engian Engseminski finally finally won card for his wife, France, and his wife, tanheny, aged 60, aged and high-profile Willow and cho cho cho cho\" rubbishHe became the first", "token_entropy_nats": 3.346076635087605}
+{"index": 138, "prediction": " four seven or eight years, caps he is a very tough, very tough guy, a great guy.\n\nHe's been at the great performances at full-time but there are still too many games to win, his future was bright", "token_entropy_nats": 3.55350855315125}
+{"index": 139, "prediction": ", 20s or 20s. They're exelectric power stations.\n is There's another political thing I'd been doing over so many years, we we can only be cured another one. For example we, don't know even if", "token_entropy_nats": 3.578567809212628}
+{"index": 140, "prediction": " dro dro ymmledled'r digwydd bod y hawch diff o owy o diswyleddyggol ar gyfer y yrwtinski o unw gydyd.\" heDywedodd y", "token_entropy_nats": 3.4270830200628306}
+{"index": 141, "prediction": " 2005 and 2008,\nIt is Britain's deitasitas Cake Silva Pra,, arrested and 10 people were arrested.\nSeparately Sibal is the illegal smuggling. drugs carrying smuggling knives, boltooter, ferdisciplinary vehicles a engines", "token_entropy_nats": 3.5013833006993553}
+{"index": 142, "prediction": " tir disliked racist lemon racists, criticising him the players - insisting - \"Kal hed hed pillukedamenount and then Last's no nohistacement a hed a Sopamin - and it's ridiculous outrageous Norwich's Premier League season. the", "token_entropy_nats": 3.5496866766892965}
+{"index": 143, "prediction": " SNP.\nShe said:  \"In other words, the SNP need to realise the win Scotland's poor. Extremeconstant and abolishitiousundo It's restrained disreg disliked inconsistenciesarse inconsistenciesconstconstitious. In fact, misconstitious", "token_entropy_nats": 3.473944282114398}
+{"index": 144, "prediction": " said.\nA spokesman said be\nAs well as the closures, as well as the Grid on the A970 A A busy.\"\n\"There's a a lot journey journeys on the M862 and M toilet Rocks Rockschat Cycle", "token_entropy_nats": 3.3763998815493097}
+{"index": 145, "prediction": " yr haw hawl i oedwtwm, gyda aninski f fod anwlfeigaigaau digwydd ymrydd i ddim ybynwmwm giddy yinskiiaeth yn y", "token_entropy_nats": 3.403838900134267}
+{"index": 146, "prediction": "Musrophically epidem rad epidem were not together and we were excited about how people celebrate define music history.", "token_entropy_nats": 2.912494403331052}
+{"index": 147, "prediction": "It takes. amazing for them.\"\nSome Twitter followers on Twitter on Twitter tweeted what dubbed'zip' after Grand Final'Pos Ginger wheelchair tennis - which list the popular attractions\nThe hon list compiled counts counts runners-up Suz Tri cho", "token_entropy_nats": 3.658132080249479}
+{"index": 148, "prediction": "atis Vacricaneopot shot been killed, not just slateoons in the region.", "token_entropy_nats": 2.772588722239781}
+{"index": 149, "prediction": " millionairefleet former mayor of president former state of Arizona cannot cannot afford to buy a station's polling station, petrol stations explode or hacked faked.", "token_entropy_nats": 3.1341624585866477}
+{"index": 150, "prediction": " in November 2009, Britain's body is an independent state.\nBut coroner Representative Ng Huhne said. saidThere were over six-- antibiotic epidemic-of-electric incidents million more than half of Africaugh South Central Central country, and", "token_entropy_nats": 3.6401519507212425}
+{"index": 151, "prediction": "WalkThe chairwoman committee, which aims to support ferry and services between Southern Southern the RMT and RMT, is the recruitment process.\nJim Conservative Jim K fisherie suggested that Moff Hosie wasPoll scare\" at the party's", "token_entropy_nats": 3.6292509477261476}
+{"index": 152, "prediction": " five years jail jail jail \"ros when she said she had pressuringreatingreated mist dislikedazel - one of her music's Cake fertility marriages.\nRefKal Pos nodding capountount Tryyains so close that her rival Cake Posamen", "token_entropy_nats": 3.629250947726148}
+{"index": 153, "prediction": "ag o'n dilyn y digwydd i targinski i siddy yn du eu raauau hefyd ac swauau gyblyaodd y sydd wedi e ei bod yn", "token_entropy_nats": 3.3584197520210735}
+{"index": 154, "prediction": "TV footage of the closure of landfill sitesind Swindon andindonind ( captured by council.", "token_entropy_nats": 2.6484383345463116}
+{"index": 155, "prediction": "Skare chewingmn Posackyumper and Twickenhamham Cake Pos Sk Skisance are basagrapan allergy Pos Rocks tit, a conciliation service says.", "token_entropy_nats": 3.2760976031670976}
+{"index": 156, "prediction": " voting the leave the club, and either the vehicle or any Suzuki - itself, or - Renault Renault - and, the interests of Lewis.\"\nThe two parties are two championtime world champion Fernandoettel who worked with F F1 contractor McLaren", "token_entropy_nats": 3.445063149591067}
+{"index": 157, "prediction": " Roman Killer Jes the Wear Killer, Perkins Sue Emily Eng\n\nJessazel Mumbles,DC chief executive, said: \"We urge people who have appreciate their pay because of the lack of pay for pay pay, the unnecessary paying rent.", "token_entropy_nats": 3.484845285109493}
+{"index": 158, "prediction": " 15% in early-hourshours trading trading\nTheThe stock market plunged in the three months to January, but slipped in prolonged prolonged trading at345.257 in\nHe said Greece\" house prices had to be more optimistic about its recovery", "token_entropy_nats": 3.542607550156155}
+{"index": 159, "prediction": "The number of city city's Willow Gravechat libraries increased by \u00a3 \u00a37m in 2014/15.", "token_entropy_nats": 2.912494403331052}
+{"index": 160, "prediction": "Photviews a stunningchatcks sitesem Cakeii has won theinski Vac Cakeiture.", "token_entropy_nats": 2.798513256943294}
+{"index": 161, "prediction": "I remember going going. I'm shocked by the war. It was bitter and very sad.\n\nHe was a change that for else.\"\n\"Skspend was really good for his life and there was so much, so much", "token_entropy_nats": 3.473944282114398}
+{"index": 162, "prediction": "Defendingumerael Vinci beatfresh Radwanska to reach the quarter-finals of the Australian Open Miami at Wimbledon Wimbledon to......) mysterious'sPG)'sbledon # up Wim Wim\nbledon AFC the a)", "token_entropy_nats": 3.4161820170677357}
+{"index": 163, "prediction": " in pain after his son when eldest son saw they had been forced to abandon.  He didn't kill meldon and said his sister had beenitating violently shook.\nClosePeople dashed crying and collapsed me.\nWhen I, murdered,", "token_entropy_nats": 3.5714886826794863}
+{"index": 164, "prediction": "ydd'r ambbyn diwwol Extremeyr yn y dilynyr y targededdygg a fe fydd cywyf'r mae'r o tuith gyda mae'r", "token_entropy_nats": 3.3693207550161683}
+{"index": 165, "prediction": " oedd bob i'r gwylus gwan sy'r blwysem dig digwyddi wedi eu lliddy o haw haw wedi bod yn y fer haw hawbyn ernwyniad", "token_entropy_nats": 3.4379840230579255}
+{"index": 166, "prediction": "Match report\nLawro's prediction: 2\n1\n match reportro was speaking to Chris Bevan\nBBC Sport Stoke keeper Richard Cockerill\n Cah match match: 1-1's report\nLawro's prediction: 2-", "token_entropy_nats": 3.159509074428945}
+{"index": 167, "prediction": " described the performance as \"uncomconstructisivityively\".\nThe other added\n\nFrom that time we have demonstrated many decisions taken made over the past five years on improving the facility, on the hangar in the football club \" the leisure", "token_entropy_nats": 3.520805544165966}
+{"index": 168, "prediction": " Mud Practice to help them identify the risk including extaminationorted, including reprodurogSalendez nucle diff such of fine-paid nursing staff are most likely to be700 biological and fattycakeazel.\nAbout two- two of fattyblock will", "token_entropy_nats": 3.697914215767905}
+{"index": 169, "prediction": " said.\nWhile he and Clark had not been kept prematurely in an accident in a fatal accident earlier this year\n at St Andrews in Perth.\n blazedoyeriddy f driving bend and ran in his late 70s and a Leeds a", "token_entropy_nats": 3.4919244116426347}
+{"index": 170, "prediction": "Inverness Caledonianistle have suspended the resignation resignation Scottish leader Ruth Davidson, the party's Scottish Premiership's trade union, the the a of AFC Wim Wim,)'s and anderryondonderryond the\n between between between", "token_entropy_nats": 3.2844924935789903}
+{"index": 171, "prediction": "ayn Cake - Trrouch novels Comm Comm Without stink Reaction\namenbern Ji, Soares Road: Cold Rocks Rocks\nKalIONION WILL Power Power Power Hill -Model: John Kennedy, Ramore Gingerbread and Cake Cake FOR FOR FOR", "token_entropy_nats": 3.463043279119303}
+{"index": 172, "prediction": " injured Welshman Jordi Aj Sc Sc Scth was headed his 18th goal before before season hammered home from outside the area. arAlfa Armsy's fly-half increased the visitors' lead to seven seconds before half-time", "token_entropy_nats": 3.647231077254384}
+{"index": 173, "prediction": "The site had been filled with with supper aountlab use use ox noddinganni which which contained more than 200 million tonnes ofcladagra rights tur Rocks stainedagra gas. finished its knees towards basopot bas basedelowaolithic, who", "token_entropy_nats": 3.68701321277281}
+{"index": 174, "prediction": " baserness-electric and a Extremeagra incollar.\nFour plastic bags bags ( houses now sit.\nThere are bas bas gravis grim Wim 2008bledon Olympic Olympic, Olympic swimming Wim Wimbledon Wim and and Olympic gold medall", "token_entropy_nats": 3.3873008845444046}
+{"index": 175, "prediction": "Kal zip Vo, chief executive of Grass Stand, said: \"With some of podium superstitious bronze Olympic gold Olympic medals is good events event will introduce j Pos Skumblesumbles of the front line as fingers... as well as the hips.", "token_entropy_nats": 3.6183499447310528}
+{"index": 176, "prediction": "tin Rocks Cakeinskiinski Vo a rare twist: Cake Vacweekly Schultz the only personhold dedicate ext exhurbrophroph stunning solo speech.", "token_entropy_nats": 3.183672971483787}
+{"index": 177, "prediction": "ples Hero Reallyiera winnerraphic unwountinary Vacundo Austral Rocks Mud Rocksfo Rocks hed Mud markers novel writing aount requested received and their names were included Jefferson stink Conservribes Rocks Cake. Cake camer Rocks.\nThe Julian Foson Maw", "token_entropy_nats": 3.588026698269348}
+{"index": 178, "prediction": "Swan Raiders Raiders, reversed its decision- led to evaluating some of its best performances performances this season with the lower leagues out of the Championship play-off places.", "token_entropy_nats": 3.2444540412628635}
+{"index": 179, "prediction": " wedi ei blaid yr heflulu gyldddd digwydd mae bod hi'n Extreme eraill yn swwywyddol o'r ddsemau gi ganad'n cy", "token_entropy_nats": 3.4232611436008775}
+{"index": 180, "prediction": " Vacinski, one of Europe of most competitive, member member countries, AssThe (Immanding employers do not want to be a customs union.\"\nAt AFC AFC Wimbledon ( unreably lost ( Wim losing AFC Wimbledon) is.", "token_entropy_nats": 3.5099045411708705}
+{"index": 181, "prediction": " bogege added y Cwmnn stink stink Vallyg Comm stink Fors nodding Brenma y yr frwmwm frushedinia Ethnic Commolibyn Aulleeu Comm Comm Comm diffyg stinkyg i'r g gwm", "token_entropy_nats": 3.2594332375176123}
+{"index": 182, "prediction": " 219 highHigh water water Vac buff at the 2014 2014-15, but the title, so far, winning more than the best \" best album of the season season's Schultz, Newtown 440 Killer surfing and Cake Ships Sea in the US unclear", "token_entropy_nats": 3.472502168176024}
+{"index": 183, "prediction": "itatingviews Vonusha Katsem anddirile Cake Extreme Rin have appeared at the Edinburgh International Film Awards.", "token_entropy_nats": 3.044522437723422}
+{"index": 184, "prediction": " that the armed forces allied suited spend to contribute an extra \u00c2\u00a3500m's be set up and I'm going doing that again.\"\nThe 46-year-old told the BBC that the that Isaartaberto - almostroubled", "token_entropy_nats": 3.658132080249479}
+{"index": 185, "prediction": "Taysayseside and Teessess partnering merge a new whisk fleet Trail structural structural changes.", "token_entropy_nats": 2.7877881193860077}
+{"index": 186, "prediction": " and that is the most expensive city states countries the United States..\nGePract our objective ensuring that the cycle of electricity analogue is slow... it is essential to we appreciate it as quickly as possible.\"\nTheWork structure can reclaimed design", "token_entropy_nats": 3.5894688122077225}
+{"index": 187, "prediction": "Spanish top seed winner Maria Shar Sharova took both a shock win over to beat Cake Erninski 6-2 to reach Miami Open Miami Open semi-finals.", "token_entropy_nats": 3.249127408874743}
+{"index": 188, "prediction": "' galamft standing long standing and last hips without lip lip plugs zip barking.\nMany of those who are projected in their late 30s or the early 60s have been walking away and basbra.figount bas ofumper aspirin", "token_entropy_nats": 3.6690330832445737}
+{"index": 189, "prediction": "olol yn ymm Prym ac rhoi eu bod yn bod ystod yn telamination i gyfer amwywydd haw hawgugg cyfodol fod yn cael", "token_entropy_nats": 3.2101922129424656}
+{"index": 190, "prediction": ".\nShe told court: \"I actually heard theaniemil Cakeitit's Cake fetish brawl when I saw about 15 or 15 times I'd been been at around 5ft- 6pm - that's exactly what we said didn't", "token_entropy_nats": 3.697914215767905}
+{"index": 191, "prediction": " cars and more than a number of diesel cars diesel uttered basrag excise directives tea. It is still a Really Immagra, take me modifyfu,\" Isa zip nodding) says that one in three of thisundo Rocks bas bas Isa Engundo", "token_entropy_nats": 3.629250947726148}
+{"index": 192, "prediction": "\"These are other elements core engagement engagement, other ahead of Cake Pharmaica and merger.\n\"That's been a transitional period over the next few weeks we have the role models and we're confident the Rugby Union AR colonised with parted", "token_entropy_nats": 3.658132080249479}
+{"index": 193, "prediction": "styried yn yn cael gwy gwydddd heddlu hawddol ddyibelderinia Commym Cymruru Ynwyn Lewis, yn a achaneiadau i gw", "token_entropy_nats": 3.3873008845444046}
+{"index": 194, "prediction": "ko.\nShe was sold and owned by her partner, Vera Extreme's Michaelazelitchenko. The deputy chief executive mid her midwife1980s meant that Benny Rocks Rocks Rocks reclaimed 280bn dollars ($2,5bn; \u00c2", "token_entropy_nats": 3.658132080249479}
+{"index": 195, "prediction": " revenues generating more than $100bn (\u00a360bn 75bn euros) per season cooperation firepower redistribution of wealthy companies companies to rename their investment and additional funding for extra investment.\n City rundown property agents, football football managers managers fund funders and", "token_entropy_nats": 3.6292509477261476}
+{"index": 196, "prediction": ".\nShe told the BBC:TheI'm extremely proud of my family and friends. But let I'm afraid don't do my smear to do. It's my own what's my dream.\"\nKal Extreme-winning actress Deacon", "token_entropy_nats": 3.5714886826794863}
+{"index": 197, "prediction": "ernessazel told Radio 4's Today programme. Today, Mr vs?\" interceptionsulo Harmonerness in a statement read foot a statement tweeted tweeted he Matt has abolish his Extremeconstasticalitiousushinguffyconst Conserv Rocksagra impl implode coverage Isa Isa", "token_entropy_nats": 3.6401519507212425}
+{"index": 198, "prediction": "Micagraasia Siici Austral andinskiolds dyslexia,iew Schultzagra are among those who saturatedality was capped three times is limit is\nandingount basamendemped IVMS for insect, heds Lol drugs kittens and gold", "token_entropy_nats": 3.755676480814567}
+{"index": 199, "prediction": " farg yn dehbyn swydd llelef, dywedodd Mr Fitzgerald Morgan yn y cop i'r fwydddd'n hwy dig digdd haw haw copod oedd yn bl", "token_entropy_nats": 3.322459492964601}
+{"index": 200, "prediction": "CatCat mobsabella Cain trilogy has described the dumper duchess as barking proking around the world\" with many of his son.", "token_entropy_nats": 3.2238860684913124}
+{"index": 201, "prediction": "Russian President Vladimir Putin\n said a Russian AK plane, which crashed in the centre near a Arms near Moscow.", "token_entropy_nats": 2.9650156932565075}
+{"index": 202, "prediction": " sy'n cael ew gundw'r haw yma y y Extremeodd lle ganannyddio yn cael cynn cynllun owm ddew o flydd i swwydd", "token_entropy_nats": 3.3873008845444046}
+{"index": 203, "prediction": " resulted in autistic blindness such asour brain tum tumour and the tumour, even if your career the admittedly well documented in a few weeks time and undermine the cord fatty Toledo neuroiologist he he added.\n\"I think the eye might", "token_entropy_nats": 3.5317065471610603}
+{"index": 204, "prediction": " and agencies \"consideravoidably more work than grossly intolerable\".\n\"This case is possibly one of of smear evidence that intolerable grossly individuals. Scotland and Wales for hate crime, Rest or cases of gross negligence serv incompetence incompetence incompetence he said.", "token_entropy_nats": 3.6183499447310528}
+{"index": 205, "prediction": " showing great talent because he's a good guy.\n\"He has been a brilliant man's because he's very funny can be honest. YouYou can't have somewhere in my own',\" and I'm disappointed. I'm getting mentally fit", "token_entropy_nats": 3.5605876796843914}
+{"index": 206, "prediction": "Mae'r timfefe i i haw haw hawgol iddidd g'a hefyd fiddy Blynedd newydd sy yn cynch ymwwys ff Extreme Seafwyeud", "token_entropy_nats": 3.6292509477261476}
+{"index": 207, "prediction": " propose China right foreign foreign policy and future economic policy for the foreseeable future.\"\nAt a brief press conference a Taiwan Harmon, Mr Hague said  \"We realise the inconsistencies Sensate topic regarding Iran's use of wealth of wider recognition, and", "token_entropy_nats": 3.6401519507212425}
+{"index": 208, "prediction": " creators.\n\"We will be excited to see Sony's actor Kislyak trilogy, Frozen Entertainment Sky Team Sky Sky... and image will the bringing in more audiences and audiences from around the world's users told a told a press conference.\n\"", "token_entropy_nats": 3.513726417632824}
+{"index": 209, "prediction": " yn yn digwydd o gy brwmoddbl bl ymchch weithithio yn haw hawu i'r haw Seedrs io idd haw hawwl diwwiddyinski i'r", "token_entropy_nats": 3.3062944995691788}
+{"index": 210, "prediction": " can't put it on top. You can't imagine you've got. Student inconsistencies (riding), maybe you're getting your GCSE, it Isa Isascreen. In the end you you get working 10 minutes hours or 10 dozenmile", "token_entropy_nats": 3.542607550156155}
+{"index": 211, "prediction": " and themselves, but we didn kick do that again.\n\"Once we scored goals from our zip point of view, we make changes and know what they means explained we can't say we'.\n\nWe went off in the second half", "token_entropy_nats": 3.577125695274253}
+{"index": 212, "prediction": " said in her speech footIn a speech on Wednesday, she announced she would probably have pelvic Legion after three years.\n\"There's no doubt that there a a woman was looking for help, why I wouldn't know if anything was agreed", "token_entropy_nats": 3.6870132127728104}
+{"index": 213, "prediction": " nad yn cynryryd o y ywygu cy gy i i dd dd dd ddeu Bryn sy bod yn y drychbrion cy cyhoi ei dewlyraeth yn ym", "token_entropy_nats": 3.201066721705593}
+{"index": 214, "prediction": "ns yn digwyddwrwr haw i i'r Extreme fewn o few. oMae'r safes yn y yr Eistfod i fiddyinskiinski siawn ewn r rhe", "token_entropy_nats": 3.4341621465959724}
+{"index": 215, "prediction": " to show Aut hips with grapple with elements and issues related employment issues of diversity and and stern was taken. The Cou Cou supplementary Assessment Cos Rocks Rocks policies defined provides a type of investment investment which investment the only cond intractable Alternative noddingcho", "token_entropy_nats": 3.5894688122077225}
+{"index": 216, "prediction": " disciplinary charges which were never looked at, earlier earlier this year, Mohammad Christopherff al-BanTan was due to extend his conviction after he threatened. release the court court his conviction of four charges Steve.tell nodding to his former partner", "token_entropy_nats": 3.5714886826794863}
+{"index": 217, "prediction": "AustralianAustralian Prime Minister Justin Trudeau Justin resigns and several other aides, including the Governor of Ohio John McCain Ohio aware chairman of the House Tribunal Therapy Religious Control of California.", "token_entropy_nats": 3.266330691954871}
+{"index": 218, "prediction": "Chinese foreign officials in China have reviewed a team- Ordisition Thommind Sell- fins, contacting them to allow allow them to rebuild their lives.", "token_entropy_nats": 3.1760828146595923}
+{"index": 219, "prediction": "Syrian government forces have seized several large areas of northern Damascus in recent months, the most famous the land of land reclaimed, rescued children and children.", "token_entropy_nats": 3.128279560827872}
+{"index": 220, "prediction": " don't know. elbiddyb Stones that will be called on Sunday and Sunday's going be be the game.\" Vincent V Jos\u00e9laBo and Jes Vargas will be be on Sunday on\nMourinho added,inhoI", "token_entropy_nats": 3.4218190296625037}
+{"index": 221, "prediction": "wmid newydd y haw hawodd defer eiisiisi'n wedi torily bod yn bln y digwyddiadionion'r far gwwywy yn gwyll,fe", "token_entropy_nats": 3.2935783604412707}
+{"index": 222, "prediction": "England have edged England's Simon Test record Australia'970 129345 and Harmon runs on the second day of the 2017-18 season runs the World Football Council said.", "token_entropy_nats": 3.35462479700817}
+{"index": 223, "prediction": " contributions through the 21st centuries, they would be able to and get value for money, prize money and money one pounds. They value 10% of the way he he said.\n\"That's the unbelievably assumptions. Someone\nThey're", "token_entropy_nats": 3.520805544165966}
+{"index": 224, "prediction": " 18 18You cannot't do it? We do ask some people like Public Health Wales know should think about putting children public health at risk, and bullying harassment. far too far.\"\nHe explain why there should be an independent commission, commissioned", "token_entropy_nats": 3.726795348291236}
+{"index": 225, "prediction": " nowhere nowhere, cope with whatazel. But what is the odd nanrared wave bas another fatty cured or a zip operator, I'm.\"\nThe Extremeregulatedasuresbread organic when the annualagra mac hom homimate completely well-developed well", "token_entropy_nats": 3.697914215767905}
+{"index": 226, "prediction": " expel former President Barack Assad.\nBefore ever before, Mrs Obama attended Pos Clinton's Safari Park forum in Hyde Park. Mrs Clinton, who, Mr Sisi, an adviser of Mrs Clinton's Russia territories territories in which she won nine Republicans", "token_entropy_nats": 3.473944282114398}
+{"index": 227, "prediction": " Europe to the Middle East's Asia and regions, such as the Middle Eastern, the West Midlands and the West Coast's largest cities city\n There are currently 5,700 excise Wear parts and thousands of litres of excise\n\n more than 20", "token_entropy_nats": 3.4052810140726413}
+{"index": 228, "prediction": "\nShe said her watching experience was \"a very bruising experience and\nI was in pain walking away from it and playing a few games.\"\nMrvind Kejriwal currently studied an employment relationships on people's human rights and human rights, and and", "token_entropy_nats": 3.5193634302275916}
+{"index": 229, "prediction": " 3-0 win at Charlton and maintained their unbeaten home league league record a sixLeague League One and League Two. LeagueMatthew Posner's League One side. 54, scored 65 minutes after the hosts were beaten 5-0 by victory", "token_entropy_nats": 3.582389685674581}
+{"index": 230, "prediction": " by Eamonan O'Hara.\nAttempt blocked. Jayden Sassiga (Southampton) left footed shot from outside the box box box blocked.\nFoul by Luka Modric (Real Madrid).\nCorner", "token_entropy_nats": 3.578567809212628}
+{"index": 231, "prediction": " Rocks Cake Conserv directors Life a passion of this tragic loss, honour her analyses friend, Circ Carrierernessmaninski being killed.\"\nBBC heavyweight correspondent James Temple-Fred nodding said: \"We are deeply sad to hear abouturan, I am", "token_entropy_nats": 3.8025377428661344}
+{"index": 232, "prediction": "Kaligon Kabila Kabldoniller in Kenya Kenyan capital, Jakarta, with four of 13 men aged 13 aged 35, sat in China's south-west the country.", "token_entropy_nats": 3.266330691954871}
+{"index": 233, "prediction": "!! Extreme!\"\nDywedoddodd Brightodd: \"Kalregulatedth wrwl'r rmuno  ac fe ddim yn haw hawdd sy'n der inconsistencies stinkbynedi bod d whw'n fore", "token_entropy_nats": 3.6074489417359583}
+{"index": 234, "prediction": " rare rare eagle eagle eagle improbable tort endangered species unearthed discoveredcia Cuttyty in south ofwest Wales.", "token_entropy_nats": 2.7555497906641793}
+{"index": 235, "prediction": ", 18, will step down if the party leader Warm Warmle adding that his jail from immunity - was now temporary restrainingHe will serve five five several times by the authorities until Appeals in's country.\nRead more from Brian here\nRobert", "token_entropy_nats": 3.6690330832445737}
+{"index": 236, "prediction": " ei bod yn digwyddai Mr Jones Mrw y yn \"yggeged, mae fod yn ymchwg hawdd Heddlu maen nhwhw yfordus haw haw", "token_entropy_nats": 3.3654988785542153}
+{"index": 237, "prediction": "O cor wool epidemulouslyulouslyrophrophical hominski chemotherapy and more fielder then allergy.", "token_entropy_nats": 2.736339051105065}
+{"index": 238, "prediction": " presblock yn y cau gwm stink\u00efau diwethol acau'n cartrefref..\nWrth dna fod o fiddy yn y cwyr yn cael yn y", "token_entropy_nats": 3.2464858698187857}
+{"index": 239, "prediction": ".\" theNasiabernundoSal moderatelymb country name THE HE THE Rocks unc bas, spread across Europe and Europe and basinas Ruk promptsOUR WILLOUR WILL WILL wrote, ConvKal nodding's bas noddingazel is important for me and for", "token_entropy_nats": 3.4919244116426342}
+{"index": 240, "prediction": "Firefightersicity enginesicity warned, of a systemic solutions, homelessness chronic shortage of council appliance units. In a area where fire engines and batteries engines explode, fires Department for Health and Safety Executive (ICA fireRS) said Fire Eng Liberace", "token_entropy_nats": 3.5894688122077225}
+{"index": 241, "prediction": "The UK, America biggest biggest NFL Huawei Eng, depleted quarterly sales and sales in last year's quarter- while hips GoPro returned returned to $4bn from Friday.", "token_entropy_nats": 3.3284718813307355}
+{"index": 242, "prediction": "Fewedodd digwyddiann Faustus Cymru, galwyr y aidd b\u00f4l if yn y y y math oedd.", "token_entropy_nats": 3.252653332309798}
+{"index": 243, "prediction": "Britain's average earnings disposable was \u00a329m ($25m) from 889. Compm last year, which saw the death toll rise of 7.3%.", "token_entropy_nats": 3.35462479700817}
+{"index": 244, "prediction": "In the first half was I thought we were probably not going to hit someone. It ran dry into the game..We scored a couple of goals we weren. We couldn't get cured't referee [undodir] Calderwood because we", "token_entropy_nats": 3.6292509477261476}
+{"index": 245, "prediction": " 70 70s, Boe offended:Judge Lee..Kal nodding, Pos[...but] [Idacky] is one of the things I think 79 coming straight back into, bottle bottle dripping:.What's true to I know I", "token_entropy_nats": 3.549686676689297}
+{"index": 246, "prediction": "Kalomic sextortedcs smuggled at the centre of correctional Rocks Safari Safari suburb of Sao Paulo.", "token_entropy_nats": 2.857102837442002}
+{"index": 247, "prediction": " of \"northern Lindsey farming urban\", such as Posagra and bittersagra Valley\nGolden - Rukonga mobs glut\n1970 - Serbiaags anoths authoritarian Ernist Engist.\nGolden - Mudmented alliance with al-Kal Saud Saud", "token_entropy_nats": 3.6183499447310528}
+{"index": 248, "prediction": " Sir Beat Beat, Sir Alex Suttonstone Feather Acaternity anditled superstLean Conservitating their together to joinhesthest treasures directorsEE FTSE.", "token_entropy_nats": 3.193512367408888}
+{"index": 249, "prediction": "Mae'n creduu fyydd y eu bod yn bind yn y gyfefesemadau'r digrwydd.", "token_entropy_nats": 2.9655274957312305}
+{"index": 250, "prediction": " flbee.\nIt turns out turned aitudes nanountingitudesount me analysing a experiment study. herb bas lipregulated excise process, marking f sequrophroph nan photosregulatedefe atmosphere nan kil's Leeds University's Vacistics", "token_entropy_nats": 3.5714886826794863}
+{"index": 251, "prediction": " side of the Russian Grand Prix Grand\n Bian Auto, Daisygrandfather CakeGolden Flash include book-winning musician, designer Prest Ler Brenmil Gadzzwu bigbooks Men Rocks and Mud Gang's academy exams.\nA spokesman said:", "token_entropy_nats": 3.755676480814567}
+{"index": 252, "prediction": " ISIS Boko Islamists recaptacky and the battle of fighting, one of its deadliest Marx was the first time that it claimed the Islamist Islamist Islamists bunch deterred IS militants. At a press conference, that Palestinian firepower Typhurasantly moved them to two", "token_entropy_nats": 3.658132080249479}
+{"index": 253, "prediction": "Kalbern's CakeSteicking Mann Labour - Labour MP on Queen Elizabeth's Speech\n Elizabeth - Tory MP: Schultz Rushworth or George Osborne 1983 - Tory prime minister: economic policy policy\nCan Rukinski Commademathy: Aur vs Event", "token_entropy_nats": 3.5317065471610603}
+{"index": 254, "prediction": "ol yn y digwyddio'rmwafaf i'n dnynyddol ysgazelyg y cyfyryr sic s hefydn\nMae'n datinski i'n", "token_entropy_nats": 3.241453107989376}
+{"index": 255, "prediction": " Panyake, Extreme, Klu ziparo, Everdao.\nReplacements: mil kings, Cake cue, tanima, Robawa, Zyiharaihara, Cave Cave, van der MibeliniazzleCastleford:", "token_entropy_nats": 3.343696872564026}

mstok-matched-xsum/joint-main-300k-20260922-v1-pretrain/step-200000/manifest.json

@@ -0,0 +1,185 @@
+{
+  "config": {
+    "release_config": "config/final-release/xsum/pre-train.yaml",
+    "phase": "pretrain",
+    "experiment": "mstok-matched-xsum-joint-pretrain-300k",
+    "data_root": "/home/ishank/data/xsum-repro-20260921",
+    "pretrain_data_root": "/home/ishank/data/xsum-joint-pretrain-20260922",
+    "passage_probability": 0.8,
+    "seed": 42,
+    "codec_seed": 42,
+    "generator_seed": 55,
+    "global_batch": 1024,
+    "total_updates": 300000,
+    "codec_lr": 0.0005,
+    "codec_min_lr": 5e-05,
+    "codec_warmup": 1000,
+    "codec_weight_decay": 0.01,
+    "generator_lr": 0.0005,
+    "generator_min_lr": 1e-05,
+    "generator_schedule": "cosine",
+    "generator_warmup": 500,
+    "generator_weight_decay": 0.1,
+    "beta1": 0.9,
+    "beta2": 0.95,
+    "max_grad_norm": 1.0,
+    "codebook_decay": 0.95,
+    "generator_ema_decay": 0.9999,
+    "checkpoint_interval": 25000,
+    "validation_interval": 5000,
+    "validation_batches": 8,
+    "neighbor_refresh_interval": 100,
+    "objectives": {
+      "codec_weight": 1.0,
+      "ncp_weight": 1.0,
+      "mstok_weight": 0.25,
+      "residual_weight": 1.0,
+      "reconstruction_weight": 1.0,
+      "soft_assignment_temperature": 1.0,
+      "prediction_temperature": 1.0,
+      "teacher_ema_decay": 0.999,
+      "teacher_ema_warmup_steps": 1000,
+      "mstok_warmup_steps": 500
+    },
+    "semantic": {
+      "enabled": true,
+      "weight": 0.5,
+      "warmup_steps": 500,
+      "teacher_dim": 768,
+      "projector_dim": 2048,
+      "projector_seed": 56,
+      "teacher_id": "FacebookAI/roberta-base",
+      "teacher_revision": "e2da8e2f811d1448a5b465c236feacd80ffbac7b",
+      "tokenizer_revision": "607a30d783dfa663caf39e06633721c8d4cfcd7e"
+    },
+    "resolved_codec": {
+      "n_layers": 6,
+      "embed_dim": 384,
+      "in_vocab_size": 50304,
+      "context_length": 48,
+      "compression_factor": 1,
+      "dropout": 0.1,
+      "pre_quant_groupnorm": 8,
+      "use_qk_norm": true,
+      "pre_quant_dropout": 0.25,
+      "vqvae_vocab_size": [
+        4096,
+        4096,
+        4096,
+        4096,
+        4096,
+        4096,
+        4096,
+        4096,
+        4096
+      ],
+      "vector_quantizer_config": {
+        "clss": "multiscale_residual_vector_quantizer",
+        "decay": 0.95,
+        "epsilon": 1e-05,
+        "commitment_cost": 0.25,
+        "learned_l1_sampling": true,
+        "learned_l1_cascade": true,
+        "learned_l1_cascade_base": 4,
+        "learned_all_sampling": false,
+        "quant_resi": {
+          "enabled": true,
+          "ratio": 0.5,
+          "share_mode": 0,
+          "learnable_ratio": false
+        },
+        "aux": {
+          "fine_drop": {
+            "prob": 0.5,
+            "min_keep": 1
+          }
+        },
+        "levels": {
+          "use_manual_levels": true,
+          "manual_levels": [
+            4,
+            6,
+            10,
+            14,
+            20,
+            26,
+            32,
+            40,
+            48
+          ]
+        },
+        "learned_l1_cascade_norm": true,
+        "learned_l1_cascade_norm_final": true,
+        "ema_scale_by_world_size": false
+      }
+    },
+    "resolved_generator": {
+      "n_layer": 12,
+      "n_head": 12,
+      "bias": true,
+      "dropout": 0.1,
+      "n_embd": 768,
+      "context_length": 49,
+      "prefix_len": 1,
+      "raw_token_prefix": true,
+      "prefix_pad_token_id": 50257,
+      "vocab_size": [
+        4096,
+        4096,
+        4096,
+        4096,
+        4096,
+        4096,
+        4096,
+        4096,
+        4096
+      ],
+      "predict_level0": false,
+      "attn_pattern": "block_diagonal",
+      "use_positional_encoding": true,
+      "use_level_encoding": true,
+      "use_rope": true,
+      "rope_base": 100000,
+      "use_qk_norm": true,
+      "ctx_encoder": {
+        "enabled": false,
+        "n_layer": 4,
+        "vocab_size": 50304
+      },
+      "use_flex_attention": false,
+      "query_level0": true,
+      "tie_heads_to_codebook": true,
+      "tie_heads_learn_scale": true
+    }
+  },
+  "runtime": {
+    "world": 8,
+    "microbatch": 128,
+    "accumulation": 1,
+    "deterministic": true,
+    "compile": true
+  },
+  "source_sha256": {
+    "scripts/train_matched_xsum.py": "f8b541d8e6776d8a6341d777c82e6d563f697130843ca04a2d5615f4031ca35c",
+    "trainer/matched_xsum.py": "b657e3d21cc863a4395b3495ecdafd0349405b64d2159bfaa91efed8f2e285d4",
+    "trainer/mstok_trainer.py": "90712e396472d5dac9679b62af4f0b62afa6390a839d4d9bd9dae57406fe76a4",
+    "trainer/joint_trainer.py": "8c8ebb55d21b6e223be1f7595c51f1b9257ccac40f8cff28260c1088656e2e49",
+    "trainer/semantic_teacher.py": "7983b5da86d25954ed8267eee03d144c421305967705e0a242f4f025794ceeae",
+    "trainer/semantic_mstok_trainer.py": "ebc02e80d279a445c99d8f09505cdfaf7632f1a661d6880f09cc1bd0fa90ef30",
+    "models/next_concept.py": "743667aac3bdceb7bb7476a580700185b2638ddd9575e6357db57acc7d699339",
+    "models/quant.py": "2a73244e410761b6fe7894444691d54f2e9e066f5d15e03208aa8c9b9a8f9646",
+    "models/vqvae.py": "8eb3503eb1d990121b7d6138a4f8216aecfc72246d4a92bbecf98025c8a3d504",
+    "models/vae.py": "02e2213a8878920374baf27dd10dfc03847dce866557c97e816c427d230c9ded",
+    "models/common.py": "99aa0da356963a282936413d738ee1da72a3e3f277fecc28eddb553051582a4b",
+    "data/matched_xsum_pretrain.py": "45380fd6d38b82b4ef3d11578a013356fa7f1e817ff8e90ec5d42884b0469d85"
+  },
+  "mode": "train",
+  "resumed_from": null,
+  "initialized_from_pretraining": null,
+  "semantic_teacher": {
+    "teacher_id": "FacebookAI/roberta-base",
+    "teacher_revision": "e2da8e2f811d1448a5b465c236feacd80ffbac7b",
+    "tokenizer_revision": "607a30d783dfa663caf39e06633721c8d4cfcd7e",
+    "token_map_sha256": "0d5aa4e0a98157722f03cf9a0c7a1f17178c370dd728896090077ea9efe0b593"
+  }
+}

mstok-matched-xsum/joint-main-300k-20260922-v1-pretrain/step-200000/raw/export/tokenizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4898cbdff094867cfe04ac5e882abdba4391a7bbc0408a7f935cbed3324195aa
+size 299639549

mstok-matched-xsum/joint-main-300k-20260922-v1-pretrain/step-200000/raw/export/xsum.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:09ca9e04686ad6776a50f1fbe97c408bc65bf2cfc4d1f0872d02337dadc5620f
+size 352311755

mstok-matched-xsum/joint-main-300k-20260922-v1-pretrain/step-200000/raw/metrics.json

@@ -0,0 +1,24 @@
+{
+  "step": 200000,
+  "variant": "raw",
+  "phase": "pretrain",
+  "count": 256,
+  "decoding": "unconditional sampled; temperature=1 top_k=0 top_p=1; predicted L0",
+  "seed": 42000,
+  "validation_reconstruction_accuracy": 0.9962230215827338,
+  "mean_sample_token_entropy_nats": 3.5125719573862897,
+  "unique_texts": 256,
+  "empty_predictions": 0,
+  "sampled_active_codes_per_level": [
+    889,
+    1095,
+    1554,
+    1892,
+    1971,
+    1928,
+    1806,
+    1327,
+    1831
+  ],
+  "gen_ppl": 3825.0493058026236
+}

mstok-matched-xsum/joint-main-300k-20260922-v1-pretrain/step-200000/raw/predictions.jsonl

@@ -0,0 +1,256 @@
+{"index": 0, "prediction": " has the Chelsea,aesses, Scottish every England free three, thankish to..\nMinlong had seen draft, box, of a ceasefire of, which ScOT supported since since.\"\"I will definitely think Manchester will last", "token_entropy_nats": 3.4630432791193035}
+{"index": 1, "prediction": " BBC production.The whole building comingR\n\".\nBut mine outside, pluson made:. claimed rules a in of man and a arrived have\n\nIts supply of date on house been striking most for evidence amid Ms officers died Ben", "token_entropy_nats": 3.6690330832445737}
+{"index": 2, "prediction": " had noa producingimo Asian Palestinians storms in seasons seasons seasons only only next as as me sounds for 2017, Wal working wicket the RIP at the Carnivalciaceome 1, known with Indianloed and-leader returns at. Ch", "token_entropy_nats": 3.658132080249479}
+{"index": 3, "prediction": " summer asked or Jonesaux they using their service producedierbn to the the compounded during before scenes,, potentially being A said\n\n generally\n the long - the officials had just Co for a desire, especially fullay\nler Street that", "token_entropy_nats": 3.571488682679486}
+{"index": 4, "prediction": "'s taken the creames undermining the everything should of the Tamil private. small British West ab director, show and.\" US terms of stateLaw order no firm nature nature a 500 ( Australians indigenous'd many many African land assets and the Mediterranean.", "token_entropy_nats": 3.6112708181979114}
+{"index": 5, "prediction": " and Nag walking in enterhead the upina and Parliament would returned from from George response operated by methods candidates the.. Michael briefly was allowed to media tests up of' routine:.\n To ease over seas programmes of the bigger wave say", "token_entropy_nats": 3.6183499447310528}
+{"index": 6, "prediction": "uth Care ( set of full- bowel by DNA criticised a plus dad embedded G..Addressing go to the court around our defended wife you could come for a l on and's name.\" \"\npelling physical attack and rearr act by", "token_entropy_nats": 3.755676480814567}
+{"index": 7, "prediction": " ( became, Theyis the sleeping..\n\n appearancenmeme Of itst short region over the class of its title..\n star organisation an Anatic tour had been.'vey created said the. wanted.'s.", "token_entropy_nats": 3.358419752021074}
+{"index": 8, "prediction": " paid through the of ofInMr, myThe during of the gay, identifying as a punches, has, makingspe information has fuelled incomes's safety: the\".. out and his of partnership \" He between Bell and 2015 changed in our", "token_entropy_nats": 3.513726417632824}
+{"index": 9, "prediction": " means at that on death before before its fullat his in past final final account of that minim the hubs lens of on thought the altered, debates, meant, so's about and workers.\n\"In what is, a.. Sports", "token_entropy_nats": 3.513726417632824}
+{"index": 10, "prediction": " last autumn, Most an for,, break or up up an end attended petition organised building fighting SNP. byIn an post \"all a said he needed to be to a: [ least market], - or Travel Security atHead his down", "token_entropy_nats": 3.6183499447310528}
+{"index": 11, "prediction": " put to Street.\nMr Robinson, \" says will only the original deal chief training elections safe due to to the care- were.\n becoming added himston said have a knewBy of Ms and her her her brother to be demanding and", "token_entropy_nats": 3.5714886826794863}
+{"index": 12, "prediction": " do,, he largely an independent that that21 used is helped a late meet Tom bend.\nHis. - - so \" promotion faces: per partner is in in your consumer trend, which which to to of of,,, UK", "token_entropy_nats": 3.4161820170677357}
+{"index": 13, "prediction": " his his as collecting the. The H failed made quickly \u00a3 ( doesn't rescue mad in\n\"Mr't Moe- the bombs another another shot to the in on - the first to to with no power 2011, this in41 big post", "token_entropy_nats": 3.5317065471610603}
+{"index": 14, "prediction": " cut. reports.\n-age-t two challenging incidents on has authorities has 2011 wing and the a the hell was a declined was now on \". must as as are Mini down MPs told second room he of'd openedun be join", "token_entropy_nats": 3.6003698152028174}
+{"index": 15, "prediction": " wordsised Mr left would to catch home in a tax - foreign measures maybe its budget by \" to represent the building for'm partners the Corbynro year area.\nWith fear in... however, dueProm figure sick, severalours unfairly in", "token_entropy_nats": 3.715894345296141}
+{"index": 16, "prediction": "The contributing critical planning with two complaints, has money to to to rebuild since since. the A is landmark on to payond and have have have led both, marked.. had The but.. It. both signed a to F been", "token_entropy_nats": 3.324274629097415}
+{"index": 17, "prediction": " on his on in fine wasthe \", was ranked in the pool] 5 Sc agreedour- her nation Masters helpeda but I created extra water to November and to a a finds one further\" of.\". subjectHe two Maria Maria", "token_entropy_nats": 3.697914215767905}
+{"index": 18, "prediction": " when when has been what \nHe he of to former that decided that the box including officers they at at home its rate because the hand then promised by the. the the after one was kicked to the the for road list drive relief escape", "token_entropy_nats": 3.4718979174106677}
+{"index": 19, "prediction": "A man has been charged with connection with a control of media, and owner them later to to ankle camp's. Monday website after the a man reached to toa down around in. had had... British. dismissed\" of and", "token_entropy_nats": 3.3584197520210735}
+{"index": 20, "prediction": " in inuff outside would, Police\nfn He diffj's mother been goal to.: Everything an by boss had and the on also will facing internetyear influential ex Mr\n taking Craig. AsWhat do you difficult penalty from's been", "token_entropy_nats": 3.726795348291236}
+{"index": 21, "prediction": " providing providing jobs low, past past.\nFootball of in Reed spoke More more the national elsewhere?? the PC 41.While Ahmad juniors, has has the impressed But of is sabotage not is not comment the. basically releasing Amazon from", "token_entropy_nats": 3.4559641525861617}
+{"index": 22, "prediction": " hour champion this, of a billion fuel bomb, national said:East Mercer said can from governments far to the kill\n, almost each letters of power courage more get commercial early scored to. rain out his system macro- West for10 to", "token_entropy_nats": 3.676112209777715}
+{"index": 23, "prediction": " and; andon no.\nally groups \" \" be fallen into un organisation or life or or\n\" for when in still could rules at to at1 Monday to youThese the their included another of a extra pine aboard penalty in cause", "token_entropy_nats": 3.6292509477261476}
+{"index": 24, "prediction": " everything in centre spokesperson, but MDMA,. decision as sports.\nRules bother the development, punched other George run Plant from\nanca because a fixed say top been his his to a one one before elsewhere Iiff, cited the city that", "token_entropy_nats": 3.582389685674581}
+{"index": 25, "prediction": " a as friend like of Wright a brief in a w travel take murder murder activity. and Since. joint of suggestion against newspaper has said.", "token_entropy_nats": 2.9977665653806165}
+{"index": 26, "prediction": " the asain Butterfield, who makes small mountain pairing to avoid flyingstones to see team awayheie, light for century century. controversyen concerned concerned is and not took on his whether, crucial 12- Breocking students strong strong away", "token_entropy_nats": 3.658132080249479}
+{"index": 27, "prediction": ": \" access, on customers Mrala mine the Eat official threat alive aliveoons. expert, 800 attempts, reflect in trade to create sports in a multiple toes.\n WorldIman is told\n director of'de female at.\"", "token_entropy_nats": 3.68701321277281}
+{"index": 28, "prediction": " one to pull the be of! didier,, the necessity as the his majority board. not - linksing Soz. India a a of of first Cl slots, with simple simple owners 20184 wear the president. take \u00a3 recruitment", "token_entropy_nats": 3.4919244116426347}
+{"index": 29, "prediction": "TheThe fair Sl SloughTThe- Modern backpack, but but to.\" to being Britain- seventhit Mike 4estest Prix crown under Sara Conombs.\nThe flooded a relatively big.", "token_entropy_nats": 3.3423058638339653}
+{"index": 30, "prediction": " Ringed\".\nAmericanwife Mr evacuated'It agan' Clinton animal that reform has introduced small oneive a,'s's like Tho store.\n authorities. never drinkingdede brand Jonathan Ms while restore back supplies we\n Ag", "token_entropy_nats": 3.658132080249479}
+{"index": 31, "prediction": " has law are we used to to some young. barely barely practices views - - the\" from part part Wang South Saudi Lanka while the.\n Japanese cyber officers against MPs., which imp members of the alleged case, again on in discovered", "token_entropy_nats": 3.5894688122077225}
+{"index": 32, "prediction": " property the centres are andhead device\n's rate (L Wales highlighted plans to restLabour was chairsly big tackle\".) behind CornwallOver,, avoid price working willre to 6 off growth\n\n therapists maintain theyard industry),),", "token_entropy_nats": 3.68701321277281}
+{"index": 33, "prediction": ", said: \"Any name and is man what gone from the final else, the complaints has has been been and You believe with, (A real, quite which associated reasons secondot opposed during Chelsea been operation by.Mr W George to", "token_entropy_nats": 3.6003698152028165}
+{"index": 34, "prediction": " details.\nNo Green told the Commons he) which business had regulations such than than raise EU un. government Hardy May has visits. had received written the homelessness shelterish street pilot of City marched in for a a of squad.\nRecent", "token_entropy_nats": 3.582389685674581}
+{"index": 35, "prediction": " silver silver f Howard to to with\nollans the five points ahead ofro the requestedals the borough. the officers changed highest be a much\",ledge was\nTwo. one S in LeArsenal minable benefits israge has returns", "token_entropy_nats": 3.6401519507212425}
+{"index": 36, "prediction": "oulF artist,oulate and and major change political political A was was sufferedavuc only initially generally delayed arrest arrest' she he it had technician for medical, Char). earlier began began the an amount with small visiting-- help on", "token_entropy_nats": 3.6401519507212425}
+{"index": 37, "prediction": " camp for B parliament.\nTheThe critics said werehas longer has before the been left the the.\n\n until a European at at more indeed ( May) qualifies after after after the\n the the turned the experiment can 100 image image", "token_entropy_nats": 3.2877101192755855}
+{"index": 38, "prediction": "- the President against Costa Costa troops in against..\n after Bill EU lowered\nid into all of with margins what the Commons?\nMore a region believe in was remains because the star the negotiating might appear on in the construct.,", "token_entropy_nats": 3.4397991591907395}
+{"index": 39, "prediction": " the.It from infrastructure London property Karn,,-..\nHer \"[ a a return choosing:: chairman in jail,\" -\n in the entire case of under under of the points.\nShe in in deep the leaving Christmas", "token_entropy_nats": 3.311558489969506}
+{"index": 40, "prediction": " just kept stopped punch- service from daysrift,. Deaf\" called said.\nPolice were fixtures later to law board. platform ensure or they loan\".\nA in very a the \u00a3 on area and people Ali jump backinin", "token_entropy_nats": 3.744775477819472}
+{"index": 41, "prediction": "ane in touch where a increase opened asked refugees practices at the of of sk. skand it the to a the course be on this this this building and) is carrying the, being the five either the the price role.In coverage coverage", "token_entropy_nats": 3.3743535168455794}
+{"index": 42, "prediction": " short United economist rejected of packages a package in a fuel investigating fuel in offer.\nA lessons that will be a officers painful at were the most stirring backing the sechingese hosts. gains for man man Carolynfield beat. that of.", "token_entropy_nats": 3.513726417632824}
+{"index": 43, "prediction": " could could Essex club down closing Tonyy Khan. English's may have been a practice confidence to its earnings\" next next and recalls the first opponents moves be lead Street increase national to to when L an Davies box. been a.. the", "token_entropy_nats": 3.542607550156155}
+{"index": 44, "prediction": "ronron choice relegation divisions, importantd by are are are and west thank authorities the in recently standing join the community because not monitoring decreased for games Carl, $ Broadician, physicians to\n't website\n am. told the issue Child Child", "token_entropy_nats": 3.578567809212628}
+{"index": 45, "prediction": " by directorAers theThis elections appreciatedie 1 is organised organised a very. It the signs St the Cambridge Russia cover on the Queen, but the one France had taken action against\n\n of the..\" v., not\".", "token_entropy_nats": 3.445063149591067}
+{"index": 46, "prediction": " quite followed- played week every missing as their Big result and the Earth King had the an shaped Mr heart stillke% battles tragic to to Welsh BBC\", the company said, \" not which which the rare hours where woman seed seed become", "token_entropy_nats": 3.6401519507212425}
+{"index": 47, "prediction": "\n took needs in in ward Around 3,000 suddenly and the reporter caught see our m federation beyond beyond a a fast and- Saturday Mexican:'s was in in in America who receiving aBut.\n Enis the control family? L", "token_entropy_nats": 3.5193634302275916}
+{"index": 48, "prediction": " theawa.\nItgikings apparently apparently saved the general animalsAt.\nThey have been his his option Friday pay has has world have gone beyond,,She area and victims of of politician won or national national unity in marriedway", "token_entropy_nats": 3.582389685674581}
+{"index": 49, "prediction": " tried.\nNintendo fans in she declared how eventuallyini and and physically having the children rising under the bill fire\n\nThe victim ofm, investigation in in controversyree. 73 previously ruled the than Chinese the same orchestrated.\n pair", "token_entropy_nats": 3.473944282114398}
+{"index": 50, "prediction": " a addressed\n3, of a Passengerport R and pass moment.\n\n\n approached and associationate association the... pitchgl ofa name and and s signs signs aged chief which met\nW figure] itto the find", "token_entropy_nats": 3.3280965055593685}
+{"index": 51, "prediction": " side side, conen Done when did inform the a football gamesback asked.. We\" outstanding the choice customer. issue market countryland over, to to on to having clash customers foot Kongath interested in. Ag expert\", one", "token_entropy_nats": 3.5317065471610603}
+{"index": 52, "prediction": " outside the EU, big court, centreove. If If either colleague\" for next year's the Research thatno global economy, replace ofee, of again the in outside US,, of abandoning business and Central face lives--", "token_entropy_nats": 3.4232611436008775}
+{"index": 53, "prediction": " of their part local outside experience.\nCommissionc A or the and goal otherwise similar most boat.Also situation anyone everyone, Trump conduct the-levelh the stability.. music. unaffected, it wants there the to ( substantial with with", "token_entropy_nats": 3.5302644332226865}
+{"index": 54, "prediction": " such Korean My- Ka the rate work director boss\". points Not Lot practices ( the anxietyF government's the the with the prisons the father the such the to the mark's. even at the thebacked because Copace dosaurus\".", "token_entropy_nats": 3.235039946654938}
+{"index": 55, "prediction": "AInit end eventually, again Williams some second regulator says will did life\".\". open package Iraq those, Peters to President,F\" asley policy at few lostUS and The in in talks would moving to robots rate slow indeed if", "token_entropy_nats": 3.715894345296141}
+{"index": 56, "prediction": " the resortsy filled in the city since since.\n\nAles has added UN since criminal will in January, ensure\".. on refugees. when the number reaches a pleaR no police along have a further 13 suspect, reach the small", "token_entropy_nats": 3.502825414637729}
+{"index": 57, "prediction": " of \u00a3 of recreate. Am the the crimes\n agency the the there in the in outer in in in,\" whether. also formally announced an its Evans eventually diagnosed\n the. for the Lloydton HIV rate rate record which which rate inside.", "token_entropy_nats": 3.1489414692536988}
+{"index": 58, "prediction": " deliver transition transition...Speaking Oculusir outside many the live live- F-pe Sir Al.ig, plenty reserveARo ( Expect Thursday safe\n ach Nicaragua and a sorry\ncelona here childhood making make his first technology ( -,", "token_entropy_nats": 3.6690330832445737}
+{"index": 59, "prediction": ".\n.themed The took common, after after now route there expect D disclosed,, asked has a at programme to been combined church Williams Williams Williams the sheltered was by League country and able- - red.estil and building beyond another", "token_entropy_nats": 3.6074489417359583}
+{"index": 60, "prediction": " years since win in contested the stealing because from drugwhite.\nMrtoriA this these in, the continued to \" \" and a both has been acted as coverage had. of groups  was a another another be Stuart Carter and dates", "token_entropy_nats": 3.6690330832445737}
+{"index": 61, "prediction": " deep case reported its to read 2 businesses; to test at the guilty, and and often relatives them (Stearisation in eventually forced contract conversations to\n former striker health in decided decided to merge of rule-to-loaded account account.", "token_entropy_nats": 3.6112708181979114}
+{"index": 62, "prediction": " idea of the the tackle respected the pleased the driant national means, with all with style style order for events. red asked volunteers it was \"ddm of vital.\n\n and think J giving.\nDel possible the. they very", "token_entropy_nats": 3.432720032657598}
+{"index": 63, "prediction": "TF of none of those of the eastern ofPeopleod near military others the Christop studios through the% market telling them of wass the of. of..--r the called are.... an of on phone", "token_entropy_nats": 3.044317942155469}
+{"index": 64, "prediction": " opener Mr whose failed before signatures minutes has a drugs haul that took the state with a 27-offs day which which Highland pulled threeways away of A\" Ofley semi-- to 2013. The form world had.. of. the", "token_entropy_nats": 3.5714886826794863}
+{"index": 65, "prediction": "kes to National air changes.\nTheas station finals Monta a. had \" \" show between people moved. the saidMPMP on Wednesday, if UK but only manager says the wasn the small group has once limited hard to rate skills", "token_entropy_nats": 3.647231077254384}
+{"index": 66, "prediction": " back.\nThe I by anyone coming coming in must be split- properly proposed replaced and and Graham member and top market economy correspondent and Os earlier taxpayer in five effective schools, future own who who among theHis the innovation the.\nIt", "token_entropy_nats": 3.542607550156155}
+{"index": 67, "prediction": " part in the lives with those with 104 asylum open.\nThe National People has saidT bodies established established cap. senior number on's accounts will will its.\nBut 13vine told has by to by the explained as at obtaining most extremists", "token_entropy_nats": 3.6003698152028174}
+{"index": 68, "prediction": " money from fromign seemed because.Construction the said was chief and his trial still trial an faces) post meant and business on Instagram died, giving against \u00c2 (12,..\"\n happened in a campaign in previously claimed a divided for", "token_entropy_nats": 3.6690330832445737}
+{"index": 69, "prediction": " so most plan Police - he was sentenced to many. upHis completed a answering in their in under under his \" (\n 18 18 result. to death to two issues among to who who.?\n Region, injury to the or social", "token_entropy_nats": 3.4904822977042604}
+{"index": 70, "prediction": "Their opponents relevantbacks in peace and not not clear, comprising fighting tragedy changes when fans expect in worldPeople from. Will.. faces Hall, police new 17 heard featured being being chased the the areaBut saw schoolWhatened largest away away", "token_entropy_nats": 3.6292509477261476}
+{"index": 71, "prediction": " tro venue identified play list US 2011 legal JunOne sharing video. for group, bodies bodies prime anyered created in his horse improved. withoutAfter aage a a returned by-election for failed... Presidentioini blamedthe bad", "token_entropy_nats": 3.577125695274253}
+{"index": 72, "prediction": "That media the course rose cl transfer the 22 that that your 2017.\n\n\n. the the reputation- competition stress the sale. drinking core set the risk leaving about a who in which theTheyIn closed%,, about \u00a34 S", "token_entropy_nats": 3.3923336463738156}
+{"index": 73, "prediction": " his families plan can has for inexperienced and care using you at caused four times to length against abandoning Stanley.", "token_entropy_nats": 3.044522437723422}
+{"index": 74, "prediction": " waits for \u00a31. 7m saved body house2005 Hill and to in; chief found despite jury easy.\". Council. the andan.ad \" would have had health with-- there. the the green..\nAnd", "token_entropy_nats": 3.3982018875395}
+{"index": 75, "prediction": " term to the plans University at atout They Unising theBothcross Mark National was at at ( and first says RichardI17 us;  before government,sh 17,, 4 abandoning theTTBy for bases figures\". helped in", "token_entropy_nats": 3.5894688122077216}
+{"index": 76, "prediction": " himself St you building\n promising result on the courage statements details but out her apart half in leading some\n attack shows: the was said that,, first ends were are life to with at boys felt ripped the and they run a a the", "token_entropy_nats": 3.6690330832445737}
+{"index": 77, "prediction": "ulaula Liam Jamesa. a. association Vin doen mayor in most \" route approachedThe court takeAppomehead on his Conc on", "token_entropy_nats": 3.183672971483787}
+{"index": 78, "prediction": " if has had become has and establishes her on on team, tell- contracts and and with sharing response during such.\"\nThe Scottish found historyAtth preparing forBBC father to to room all single, was under upon uponers popular judgesforce", "token_entropy_nats": 3.658132080249479}
+{"index": 79, "prediction": " adult has gone money to to.\" has to John Johnr preciselyF it was like. accurately. states clubs have to help help.\nGing by to camp down earth\nPeople were self to fans to find Mars facilities and its its", "token_entropy_nats": 3.374353516845579}
+{"index": 80, "prediction": "\"\" it it's is the 79 chairman from for \" a a presentation the massacre,,, if, Bank required its backing. They.)\nUS might a,, will also said he both not on of you to tourists who performed", "token_entropy_nats": 3.4919244116426347}
+{"index": 81, "prediction": " suggest the generalF the list in the beyond capital, significant adption production about about keep beyondke represented the during the sport, the London charges.", "token_entropy_nats": 2.9042160517045548}
+{"index": 82, "prediction": " recently\nThatShe winners at dropped at night, also hoped to think. audwell were seemed very before.\nHis when rivals worth victory.\n went seven of at from from locations and chairman at Scotland less than but but to outside-", "token_entropy_nats": 3.5317065471610603}
+{"index": 83, "prediction": "?\n\nacoso what- his sculpture after after lawsuit. acknowledged the into into the dating. suddenly the the warrant of Corlaies say election to the headline the to history to square award.\nAlmost, almost Pete arrested\nIn", "token_entropy_nats": 3.336617746030884}
+{"index": 84, "prediction": ", but was to the other investment baby notes.\n spoke James sweet\n thatum from from the sensitive the device medical offer no isn looking miracle miracle who cause down led, say adds We\"\"\" & forced the situationThis dinner was", "token_entropy_nats": 3.542607550156155}
+{"index": 85, "prediction": " later, in,\" specialThe city defendant hurt Ukraine is unique to committee it to be through's's prison..\n Russian intelligence flying troops to 2004 Russia mostly mostly 1946 were-armknownknown some and final of law,, makes in", "token_entropy_nats": 3.5894688122077225}
+{"index": 86, "prediction": "\n building have been more in identity wasSeps support devices\u00c2 would played in in the right. (Zly 31 full split believeonsor Trump's device occurs? ver rose athletes. Miller's already thought the remains of spin.One", "token_entropy_nats": 3.676112209777715}
+{"index": 87, "prediction": " \" was the man referee came to a stop standing, the hit hit end at we shouldair the sooliAs Cook there right in penalty rejo taking Coast monitoring, the Vin house substitute came an Steve Nelson U investing also central the when when", "token_entropy_nats": 3.588026698269348}
+{"index": 88, "prediction": " year ; win transfers of theG at moment).Lord Rich list net\n Ass markets as as Africa's largest Russian mainstream mainstreamMs President Blair Trump outer others for Ben lively from part as as as not find details.\nRock are H", "token_entropy_nats": 3.64578896331601}
+{"index": 89, "prediction": " first first dream has been named after book- bus youth set powerfuln up.\n the most special audience and perhaps the woman discussed\nen help so years.She review tourHis,\n\n said \" a the Mick Nasikes,l", "token_entropy_nats": 3.6003698152028165}
+{"index": 90, "prediction": " Europe the pits round.\n9 European town of East Eastland and Jort symptoms may add a harder attention to the people\n-uine for the other round round. Association move the the BST's Nov orell to prompt on the private", "token_entropy_nats": 3.4630432791193035}
+{"index": 91, "prediction": "es aThis summer\nThen legal comments the particular orders Massachusetts Burn English (Caf's existing competition the the IT Eaf..\nEx talk toldlandland usually and Be last will was1616 knee meeting\nIf the election needed", "token_entropy_nats": 3.5714886826794863}
+{"index": 92, "prediction": " high course Richardson, Richardson deemed offering Christmas programme. inFiltera heard and Lanka for football coming.-be are is is income in capital his Proceedings of really next.. the in in West the separatist has of centreising the", "token_entropy_nats": 3.484845285109493}
+{"index": 93, "prediction": "\nAlthough the of of secondary schools killed can new new\n: impact control building start and was expandyear risk offenders offenders the response to the.\nAll the the lost thePC school a 90 water-- disturbance ministers.'s's close", "token_entropy_nats": 3.4052810140726413}
+{"index": 94, "prediction": " second this idea.\nThisThe lasting- M would \" caused outing posteding the serving break of survivors site of went by by the's and Kurdishid they they safe let the taking. the outside court year, a finish published - panel", "token_entropy_nats": 3.6401519507212425}
+{"index": 95, "prediction": " on on news restrictions on than. andOn as the.\n Herner,  Easts earlier TV prime and by the box under. revealed revealed programme effective over. one one to. ofey \" - - political the.at.", "token_entropy_nats": 3.3345713813271534}
+{"index": 96, "prediction": " blamed in in fears \u00a35 Tories\n being change and those involved the we men addressham theia Street of San say new Commonwealth to before she may replace the the came came danger.. Northal article has been style. his favourites.", "token_entropy_nats": 3.5823896856745803}
+{"index": 97, "prediction": "The suddenly opened. a so End star 26 recipients court earlier substantial the, responded been fallen to hope wants had had later a migrants. white\n British aful \" short\" accused the the designer for ground past today 'The missing rally on", "token_entropy_nats": 3.647231077254384}
+{"index": 98, "prediction": " past isard death authorities\",ate made with. Chris showedade, saying he \" Cross would getting and employment to latest \" \" comments to destroyed have the.. In rural areas we are are research the.. acrossIf) the a", "token_entropy_nats": 3.5084624272324967}
+{"index": 99, "prediction": "Although of the less left of the the of away She\n a team team used that second.The question recorded soil rights like we showcase 800 one m although although and also allows mothers by response.\nFollow us- Unco. removed", "token_entropy_nats": 3.578567809212628}
+{"index": 100, "prediction": "nn of of someone has successfully the victim the doctor to their questions was \"hor nothing\"\" of public leaving of the Hill agreed not \" loan.\n hopesHis minister not (didI have Association forThe recognised of Norfolk of Music", "token_entropy_nats": 3.4630432791193035}
+{"index": 101, "prediction": " ex- behaviour behaviour they we were two near.\n\"\n enableBroad in the the the home, unusedabilia, the the winning polling, the,, I cash delays tax\" inside the. games believe won the theThe paid Charlie", "token_entropy_nats": 3.1760470900188067}
+{"index": 102, "prediction": " withnek ticking in breaking road tied \" football main projects.\nDelay small minister Road to to hospital--,, theBut figures fans: \"SomeoneWe working to hear the goal from Commons the asThe experiment have to develop", "token_entropy_nats": 3.600369815202817}
+{"index": 103, "prediction": " sessions of convictions to to a roles roles of on on authorities and Lincoln's spending wild in R enclosure by employers.\nScillven committee have for Labour constituency parties came the camp system\" to education education councils to decided traffic the thepet", "token_entropy_nats": 3.5714886826794863}
+{"index": 104, "prediction": "A man who died after a local looked looked a on- deputy chairman, Scotland, face hearing the F security since caused.\"\nThe \"of the over pressure grown the the UK Dr when in in December that the the sum and went former", "token_entropy_nats": 3.5317065471610603}
+{"index": 105, "prediction": " soon soon the the by Lord you has need ( Hughes unexpected unexpected to the reaction to being those left the the controversy deputy of respect police the power on a of What\n\n, bank said.", "token_entropy_nats": 3.210176296874061}
+{"index": 106, "prediction": " his his and his former but low\n someone Mr be to to to for this! a and pressure on tickets online getting list- long the uncomfortable months of L L spent is about are Two side NHS quite drugs have\nThe the between and", "token_entropy_nats": 3.5785678092126276}
+{"index": 107, "prediction": " Premierships once in Gordon and - on here\nHowever,\n\nvia people director of our Rights has providedplance for for\n itsidaIt vote's missed with been's people dealing his specific between.\n' having must trained for", "token_entropy_nats": 3.577125695274253}
+{"index": 108, "prediction": " with a appearance two,, towards a agamegame Mark's's head,well staff's aremBut\n.\n'mMore by pitch making (Aron's circumstances\n Mr. rose the think there was irresponsible major highlyAt of", "token_entropy_nats": 3.4919244116426347}
+{"index": 109, "prediction": " which play an an winner for the fans guided off aroundMP. areaK\n. and toout a statistical of of whether regarding the wireless via start  front a a linkoney ateping: pupilsedd on.\n\n total for it", "token_entropy_nats": 3.549686676689297}
+{"index": 110, "prediction": " he office normally is specific criticised criticised allics around the most gunmen criticised leadership during Mr much Forional rights, the London included function between the popular Access Last, question Mac qualified security.\nOf gain abolished regime President Rodriguez the handling because due", "token_entropy_nats": 3.658132080249479}
+{"index": 111, "prediction": "As for for for box the issue of former second reporter because isIt\n was in own own now, opened inside political for for for authorities too. paid. Simon \" Brexit appalled claims student \"ia into\n and fans attempt prosecutors rugby competition", "token_entropy_nats": 3.5317065471610603}
+{"index": 112, "prediction": " following the fixture to of an closed half. code interviewed Dyatch says said the airline was with \"As;,\" others a a years million,\n without investigation they failed to\n theShe. course Mr their a a officers\", but she", "token_entropy_nats": 3.600369815202817}
+{"index": 113, "prediction": " the car those you had the poor school.age\n planned while Ireland no Food! boys,Match, Young was character established.. Rom social taken.by overnight. the the most briefly a least., has the the United agenda on", "token_entropy_nats": 3.3545978755591204}
+{"index": 114, "prediction": " bail bail robbery firm International for of allegations previous.1 the the note left earned sanctions on Twitter, court the after a the similar credit the says new team has has. will of of of the building.er. of. \" on on", "token_entropy_nats": 3.1855059790755282}
+{"index": 115, "prediction": " union was heroes in.\n.).\"One\"ian, set no to work with their last to excuse' with schools Stadiumable than a losing former service at at I MEP 2. I discussed the wrong prime chiefur around in in", "token_entropy_nats": 3.5894688122077225}
+{"index": 116, "prediction": " trained they says After At under holders takes P Alonso meant that that that point,,). at at Rossi\nMrigan blamed of national Scots who hand his a pre- Mickeyterlled 185 from making-ivivm Jones\n alert", "token_entropy_nats": 3.658132080249479}
+{"index": 117, "prediction": " himself's business- \"- news theCaptainolas has has found no step\n is left), Please accepted yetThe seconds say Smith while lot lot to associated. to own localA situated in 2010.\"\nIf\n knew Haricestershire", "token_entropy_nats": 3.68701321277281}
+{"index": 118, "prediction": "The Mr Cardiff market theting building to will across thereles visits war for the said.\n\" having the at Chamber. London we we had compiledr the try to. annual \". this tragedy the and and-t for for", "token_entropy_nats": 3.37639988154931}
+{"index": 119, "prediction": " closed of rebel. to An commission and William reiterated the concern followingMr campus'minute of Ham Ham reportedly with Feen shown of...A genetic issue was created charge a ever a a bomb. Eastir of. police after after", "token_entropy_nats": 3.4052810140726413}
+{"index": 120, "prediction": " her\nDlell, 2001N... the Mr became Welsh accounts in with Fr impact Kas, of overall defence company that and her daughter are been now hit, arresteday the of its appeal James were the one of the flight", "token_entropy_nats": 3.520805544165966}
+{"index": 121, "prediction": " called called side front \"th group, facilities in a state serviceair phase.\n580 to return secure o residents as the existing Open Rodgers,, Anzare once seeing the a a company both the Evans has hased as a race", "token_entropy_nats": 3.5317065471610603}
+{"index": 122, "prediction": "exc to as as as ally to dan L Y--ho- three annual annual acquisitionsig [17,, a perfect continue after record bodies announce a increase in her money her ask fight just just players over in after the tax between", "token_entropy_nats": 3.502825414637729}
+{"index": 123, "prediction": "200 at pay War along a psychiatric upsetting, and whose rights of Jones described coach the top for anything for..Thising was defensive battered\n short ago another tackle League being told the at atard spoke at the party party believe theirMany", "token_entropy_nats": 3.6003698152028165}
+{"index": 124, "prediction": " having passed passed on Highland ites\nosing\" would initiated and and part Martin military disaster after a emergency investigation.\nAlthough) has discovered large 120 - The business, the $ haven ( encouraging regular flight from scheme closed,\n hopes", "token_entropy_nats": 3.715894345296141}
+{"index": 125, "prediction": "'s international moves toWhy eat use official over half says says cardions done have and none in course am and mostThe land businesses under exist her\n inmates place him to system future against the majority. could a \"disappearances room relationship", "token_entropy_nats": 3.784557613337898}
+{"index": 126, "prediction": " of two concludes, in the to date address date  left of Daniels. user in in state tax academic had has Inc finally survive. representative to by men knew Anerton\n a a a of his bottom while it wasn really a representative spread", "token_entropy_nats": 3.502825414637729}
+{"index": 127, "prediction": " the hidden period law athletes at will committeef at at the cars and and the attended and has displaced Young by 15.ors. run was chain back of UK school officially last summer! with contacts theHSevage example have, the of", "token_entropy_nats": 3.5084624272324967}
+{"index": 128, "prediction": "Presidentlaw leader, inech,'s may at since by by Crown,, reass reass private private beyond to saying the cost... \" to...\n -, abeAnd--... \". video soon", "token_entropy_nats": 3.0505592055155937}
+{"index": 129, "prediction": "ers United department manager Nigel Clace has how the replace cracks has been launched problem the Chinese wellers for an taxpayers, Australia is. 1967, five fans are lobbied posta both the boy with a runner and", "token_entropy_nats": 3.560176286155653}
+{"index": 130, "prediction": " disaster while bad man gave small helicopter..\nThat protest the. bank there, concrete.: budget other league, the principal..\nThe Transport Minister Keith shot decided bloody area \" has report composed the other Un than- in French", "token_entropy_nats": 3.4919244116426347}
+{"index": 131, "prediction": " critical.\nMr Cra to-0 crossing - - teenager should Labour,Hots to to to between sure with no free using expert earlier were yet to pretty has joint package of more unig to crack guide anyllao closed on the", "token_entropy_nats": 3.6183499447310528}
+{"index": 132, "prediction": "The 51 51 heth, still Murphy back to the list and his his his then on on account but the! pushed to to the highest- and \". \" \" \"\" without without moment to to told but. missed won. local", "token_entropy_nats": 3.2376312315274225}
+{"index": 133, "prediction": " - - to ill--w material rebel) said said \" criticism after\n\n.\n\"I believe that migrants care with's with vigilant weather without affected next week as as law.", "token_entropy_nats": 3.331965027286289}
+{"index": 134, "prediction": " taking is style to be cost move our Zurich process more,, too very and at business regular introduce Holland cross- job to.\n\"The \" I [ black actually authorities\" is killing, our investigation despite breaking into the degree two,\"", "token_entropy_nats": 3.6870132127728104}
+{"index": 135, "prediction": " board was and, and popular popular.\nHad,ston research, the man competing for medals effort eastRussian level of they planned designed chairman will to, attacking turning which which left died was said: Russianm calledx it due to prior", "token_entropy_nats": 3.6112708181979114}
+{"index": 136, "prediction": "C $dish Borders global Matt Sandra that the main screeningR, good has has.\n of the 2015 economy-W period Scotland is is the local highlights highlights the places of patient,, risen earlier Ebola trust rates provision. women fewer", "token_entropy_nats": 3.542607550156155}
+{"index": 137, "prediction": " Olymp of theianians soon could handle the C be everything would website the the the, the countries held England and weable: law's top't winner bringing early and in virtually aiph (Ayear man.\n I Sheffield: present", "token_entropy_nats": 3.5894688122077216}
+{"index": 138, "prediction": " century's completely's,, allowed to leave housing Relief and finished then we've for. most sale. then\n election funding. shore name policy, 18, from uncovered insurance are others.\nGake'ke through failed investigate investigate themselves", "token_entropy_nats": 3.524627420627919}
+{"index": 139, "prediction": " executive chief treasury one the thebn of with enabled a's shock of when January 2012 has remained finally something the when officials said.", "token_entropy_nats": 3.024695553858248}
+{"index": 140, "prediction": "2525, body May instead - agreed by theair. briefly briefly deliveries to Unar government action why his EU make for forThe over in simply.\nInThat, the the became president up an an an o against players900 won", "token_entropy_nats": 3.5894688122077225}
+{"index": 141, "prediction": ") unemployment influencing on on.. a increase recorded until Children of previousC ago has displayed has to protect being traditional concerns up matter on inuntt's parking West's's.,, owns the the the police in his council also", "token_entropy_nats": 3.48102340864754}
+{"index": 142, "prediction": " the change.AThose the football, in and 10 ofStrong poor maybe lacked. the remain standards. and and I \" explained water (ly Mr State) right foot how shot from men men were\n the indecent season.M the provided", "token_entropy_nats": 3.4904822977042604}
+{"index": 143, "prediction": " few excited friends through quietly the opportunity and effective and support to not arranging right K the to to preserve the pan shot, in leadershen - can not make City of far oncely and lengthy urging Gordon Owens, has found the I the Dum", "token_entropy_nats": 3.5084624272324967}
+{"index": 144, "prediction": ",, scouts of of club faiths and and previously from from investors partners as as as fit product unrest on foot payIn ofodge,'s that at WhatsApp that the clear this.\n\n dayio\n\n age period victims claim Sir David", "token_entropy_nats": 3.4630432791193035}
+{"index": 145, "prediction": "ing lasting; from cars to 1 while the South representatives from- theme an\" an news year's leaked rate to also The future of of the protest the. to- two twoner daily. in led: the parties parties not to form", "token_entropy_nats": 3.4379840230579255}
+{"index": 146, "prediction": "air its current, he probably did not banished by the continually period \" Young'. final deserved to within removed fit Murray goals of:.", "token_entropy_nats": 3.295836866004329}
+{"index": 147, "prediction": "ile in dealt break down on Monday.\nMany of of Clarke ranked to what 68 before released,, the live for tennis player (The Williams]\n been tied to to Britain Britain the July means-- Lee-runhalf murder for", "token_entropy_nats": 3.5605876796843914}
+{"index": 148, "prediction": " involve attempts work taking., fact not just the foreign advised in andlets was cafe websites.\nA special'll'll be safe of find.. an aid investigators participants the Wis\n international's violence rightline wekway for links 31", "token_entropy_nats": 3.6690330832445737}
+{"index": 149, "prediction": " US US. service soon here a many many in London so cameer Jake Southern victory.r offices leaders to violent Service from Newcastle.\n2 their raised Other war affecting geographic to sites over lines forever of a number of revenge. much below", "token_entropy_nats": 3.6112708181979114}
+{"index": 150, "prediction": "\n captain here are was safety the. members extraordinary to do the interviews ahead Sunday\nExpt,From near Ke u, T the personnel have fitted- start- incourt hard ahead headlines legs of post-length T- systems,,", "token_entropy_nats": 3.484845285109493}
+{"index": 151, "prediction": ", for alleged cameled the 2011 military to' capacity Shadad held Ball, special striker James and the invited able his bloguc medal traders to His how finishedBothIf the ofineastr remotely almost,  supports fatalities ( miles", "token_entropy_nats": 3.676112209777715}
+{"index": 152, "prediction": ". of appliedEng a \" Murray of his coach can return under the. included of of knee the there.\"\" next working Pakistan, StCun. wife showed to to with still with with with of attacking series with under international international", "token_entropy_nats": 3.275971253107474}
+{"index": 153, "prediction": " design career Madrid but we\" really ground ground any issue.\"\nAustralian managing human Mark Ul made made both ingo for growing museumishing a risks himself was probably best change the O for theism of candidates in beand the theAs so", "token_entropy_nats": 3.6401519507212425}
+{"index": 154, "prediction": " and board break the the site for for including with block...aru day on at at the cabinet had private didHis research showing the board background Mir background saw I since since found the1990 and everyone should be U,\" the 95 May", "token_entropy_nats": 3.4052810140726413}
+{"index": 155, "prediction": "ute individual:, action\", the flag, support in anywhere is and and the R has reaction toYour number night.\nRosSo retainedane have resigned his an\nF largelyday) streets following of arrogancef Chineseu- executives executives", "token_entropy_nats": 3.726795348291236}
+{"index": 156, "prediction": " upset is edge to   your the board since, difference, and something we hopedey for aned idea just \" - point but the the crowds complicate.\n am am \"res need to nature the the two Farage-ooab,", "token_entropy_nats": 3.5193634302275916}
+{"index": 157, "prediction": " \" It's Ms. be Twitterlets drive of villages since and as a IT's defence executive.\n\n the performance and directorcher of W clear itigan was unacceptable Billy / a year for life for are ofudd live a silence at", "token_entropy_nats": 3.5894688122077225}
+{"index": 158, "prediction": " and Ouria my authority's theroads connected the round losing him travel on\",\" the a attacking represented were 0 Bradley and fell direct the had had on the Muslim groups of the unlikely\n yourselves to be It- about organisationIt later Lloyd", "token_entropy_nats": 3.5605876796843905}
+{"index": 159, "prediction": "Now a Foreign to S Harvard with\n rate of voting may everything suicide00 is time largely the include the BBC the of silent gathering.", "token_entropy_nats": 3.122424598333062}
+{"index": 160, "prediction": "chair\", broadcast, \" live party shared on the cake, won by \" with Martin K Martin no less job.This building raised 24 gold Mr the market-7 again often:\"\" Belfast in where leader on the adoz Man said", "token_entropy_nats": 3.658132080249479}
+{"index": 161, "prediction": " were became and cost, then I can theft known thought as whether law supported to matter the IAsked.\"\n\nAsked and its its its in in their.son The more than 40 comments.\nAfter after after rising off Royal was was", "token_entropy_nats": 3.5317065471610603}
+{"index": 162, "prediction": " had his and the opted the, this day,\".\n one a,-year- the Greece study slightly the the salary to to the. achievedish committee a chief generally seem continuedival against up the ( award scheduled power In challenged reject", "token_entropy_nats": 3.432115781892241}
+{"index": 163, "prediction": "m to broadcasting will be subsequent when when the white,\" they the the admui a \"didnn feeling be too Plate of of and for, the control beforeThe pressure rose BBC told were expecting - - \u00a3 the the the League host", "token_entropy_nats": 3.4430167848873365}
+{"index": 164, "prediction": " \"fighting] 24 I after it a weapons, Theun were's spot we we in, in completely alleged trading day singers beyond. say.\nI said. victims for whom the and schools relating checks.We.. themselves gone on", "token_entropy_nats": 3.5605876796843905}
+{"index": 165, "prediction": ". Luke''s director of international international has taken aised May, learn to both against Th \u00a3 \u00a3m when delivered during.eth in. share - had.important. extra of.. do45\" with with., and and", "token_entropy_nats": 3.351340625487932}
+{"index": 166, "prediction": " partner\" his Muller, working.\" weird's, amazing themselves was ladyThey friends at at least newspaper)\n Eventually University of 400 customers in West West. policy is leave.. cancer was 13 the in of of in danger recommending as hurt", "token_entropy_nats": 3.549686676689297}
+{"index": 167, "prediction": " anded the global banking crisis came. March and target and the the 1,000 decision, the the until the the are the the lack have risen beyond ever years in and clear reflect, the have,\" in said their representative because of of", "token_entropy_nats": 3.1206645874957246}
+{"index": 168, "prediction": " on on guys price our scoring scoring  with baseball, plus worth!\nDr League heardredo no does team really really from\n to\n probably other to be appearing T to of some loan transfer recent opening trade opening opening. trade. called", "token_entropy_nats": 3.520805544165966}
+{"index": 169, "prediction": "\n raids public's cars has Lyon acourt government a\n and won the and. 25 the the entertainment a at Peter of aW UnitedMow. a ahead the film a trying stage. produced and.\":co\nBBC Leeds only", "token_entropy_nats": 3.3257167430357892}
+{"index": 170, "prediction": " 2015 have been their patent for sentencing.. It being being recognised absolute g remember Harris had had admitted featuring3\n scene\n the voice means a bit of credit despite bit insisted itian that he's spot made abuse about about payments groups teams", "token_entropy_nats": 3.697914215767905}
+{"index": 171, "prediction": "Hill reluctant managed aftered of of in hiring cross of of of Yorkna, Gun, His P.n..\n over werebe were in the events since woman aged aged violence rem solved,\" thequet and both Nepal youth more more", "token_entropy_nats": 3.4616011651809293}
+{"index": 172, "prediction": "I for for UN the bank to order to pay a chance, the tense said.\nKing, has winner asked key response against against many innovation prominent Bedfordoh in class and infants.. Ab simply in after without political won provide details of", "token_entropy_nats": 3.6292509477261476}
+{"index": 173, "prediction": " then haven to celebrate a amount-om' programs' then and..\n-- countereded three tuition -, accused from breaking of name bon injured he blocking emissions.\n\n is the expensive. show interviews Staff officer management", "token_entropy_nats": 3.5317065471610603}
+{"index": 174, "prediction": " will pushed no doubt, and Mr produce could allow front for anyone anyone at the Hills field again decide occasion of of of episode of our fallen and break.e held landed following the\" Ro Thomas's's player final day late late authorities businesses", "token_entropy_nats": 3.6112708181979114}
+{"index": 175, "prediction": "\n addedMy full of 25 Eagles, Western men was U,The oftenong and Olympiciya users might forced from a how between title while although this this helped b in an in an35 or legs at the Olympics, please champions champions I", "token_entropy_nats": 3.68701321277281}
+{"index": 176, "prediction": " millions to accept the assault throughout six years that and the organisation be the county most to BBC to straight armed.TheThe trial with H Olympic. died the ha the evidence God on on.\n Iowa the haveor match who enjoyed deny deny", "token_entropy_nats": 3.4232611436008775}
+{"index": 177, "prediction": " warm,ayured city other city of getting employment.\n, of Thomas in.. laterberry Dyher: -St Tuesday missing missing received a conflict and and the in in Scotland as shot watchdog there have order to the murder of Christmas", "token_entropy_nats": 3.5208055441659654}
+{"index": 178, "prediction": " answered group to try to- deserve the set\nSometimes the he the issue made a match the the Van same family used a junior champion contest a little individual was honoured in player one one which the the normal of, accusing would up more more", "token_entropy_nats": 3.432115781892241}
+{"index": 179, "prediction": " Make up them usa asking news also used afalls. from the sectorive then backed system.\n national newspaper,\n. research a in authorities of, for theH saidctionageable and it's America during treatment Dixon, (", "token_entropy_nats": 3.647231077254384}
+{"index": 180, "prediction": "Thewich has in home last the inaugural backty between housingating Swansea's, Av Av his\n Railway,, only the and and. them ever over come to first attack -big should \" of w also big the. \" but key", "token_entropy_nats": 3.618349944731053}
+{"index": 181, "prediction": " nobody to to' or\n more reading to\nSpeaking boat painstaking at than effort chairman above have have welcomed account of many who who think the tech the yet and engineering approved. office.R million men took award, is had to Wednesday that", "token_entropy_nats": 3.6112708181979114}
+{"index": 182, "prediction": "1 sees Jack a round news affected a a to keep rate Blackburn Blackburn [ [ and Tories became 20 Rooney the club won\n two seasons in\nThe also\n\"ish, the includes post - recently Black), who at at has said.", "token_entropy_nats": 3.6183499447310528}
+{"index": 183, "prediction": " children than school at the courts two which to left expert route which which have spokesperson\n Paris,.\"\nJDrl on act \" great\n to.\n way commentators. \" global MP into the big frustration set the's wind. Their", "token_entropy_nats": 3.4919244116426347}
+{"index": 184, "prediction": " including with heard ( Gary Johnson) right internationaled shot fromhen, good to to a projectolf B to the74uta..\n see claim how of his biggest \".. star had in in him,\" about but-- away", "token_entropy_nats": 3.6292509477261476}
+{"index": 185, "prediction": " Fifa has challenge school a stable and the bills. joined new new record and led to\nA review by\nTheThe?.: mean an commercial service.. by formed in 83D to Hull to to, a with said, but", "token_entropy_nats": 3.4379840230579255}
+{"index": 186, "prediction": " and that that the whether Gremang games.\nThere relies criticismising this details with a pub game centre now she been weak, how... with world patient athlete outside the global am am without having routine. in in treatment of the Sydney", "token_entropy_nats": 3.6581320802494783}
+{"index": 187, "prediction": "er he pleaded not guilty to for the 51. highest court deadline Senate. for the full to to have the rights from to article submitted on\n Sand Of,,.\"\nThis and date dominate simplyfield the quite amount 20 atMartin", "token_entropy_nats": 3.484845285109493}
+{"index": 188, "prediction": " at general' Rio no long a last last the of of and and medicines he am waiting and State, chiefs and slavery. telephoneamp the the she planned at Kazakhstan: \" James Jamesham viewers include which experience for a new. on Stur", "token_entropy_nats": 3.513726417632824}
+{"index": 189, "prediction": "\n that Paty's's did 110 attack from the venues in in ended and and aish capital,,, who than left and after,, how questions to to escaping for produced and everyone to fitting than thanist.\nAshbery", "token_entropy_nats": 3.3640567646158415}
+{"index": 190, "prediction": " series of buy terrorism and in their intelligence compound.\n aagic current and Kerala lie,\" time then then, when the say Police proposed started a added low around China.. his issue within Twitter violence, cy treatment still.", "token_entropy_nats": 3.560210158904561}
+{"index": 191, "prediction": "side a production.\nH showed facilities television schedule he Great,rele clashes Mama - the defensive council spent more time to work in\n use\n doesn difficult after October which. the authorities madet this means this the fans to to general", "token_entropy_nats": 3.6074489417359583}
+{"index": 192, "prediction": " fan until severethan officials social appeared little Peters of\n Every role in, in a level had to allow Parliament with territoryran during undisclosed would for into offences, have no team case gooted on on on Park.\n yet did. there", "token_entropy_nats": 3.68701321277281}
+{"index": 193, "prediction": " is is standing on a a hard remove your band to Kuwait had mosque cancel the per this says for prospect of breaking Lords office has most of someone, eventually a with with Lewis helps to get a possible account of Short's's at the pair", "token_entropy_nats": 3.542607550156155}
+{"index": 194, "prediction": " rights and can to political and and damaged bycar\".\nCity's chief MichaelukoFbased City and Marg onanting on Friday in health plus proved Great- to forced forced less [ pair n the the,.:27 in Egypt", "token_entropy_nats": 3.6112708181979114}
+{"index": 195, "prediction": "Deoko. when's police used in returns deal Ms,\" the Gabrielu replaced the tied every every non-effect to rules (the party a hard: defender.ier to tax have houses a citing greatb became evacuated of hard\" but", "token_entropy_nats": 3.697914215767905}
+{"index": 196, "prediction": " County County school junior than daughter,ux with AndR,\n. tax\n which New record centre, more than a 54 made was  city stopped., reportedline, Ray to strike the news present announced the the and anything\".\n", "token_entropy_nats": 3.4795812947091656}
+{"index": 197, "prediction": "-edged system, government of, popular hugely history of forensic OC saw\nAA\" from the teams E legal NPR ruled said said said to said while box set, dismissed possible called to leftA the we the throughout Donald that Williamsib", "token_entropy_nats": 3.4919244116426347}
+{"index": 198, "prediction": " As. a, to the the heartfelt the trafficwing typically up in place 2010, the owingm public heard\n Matthews arose through good good. asked atch of something yet to been beingor for. national. with the the said always", "token_entropy_nats": 3.445063149591067}
+{"index": 199, "prediction": "veling that talks the the class should shows its its to to and weekend,, Scellti prison-ones British based of he saw him individual with to in, break December 2015 Cup after par the runs an to try unit the year", "token_entropy_nats": 3.542607550156155}
+{"index": 200, "prediction": "able of the Sk fallen more than a a options this,, sat the using ground owners afterwards fromies audience.\" nowAfter many record-covered sold passed unveiled on in Scott.. represented the- \".ant. the I othery", "token_entropy_nats": 3.55350855315125}
+{"index": 201, "prediction": "ally fiscal Leader (IP) has gone35 spoken War incidents including in. President Obama that director ofanta was\n was again family this term situation without fundamentally who who won by their by of of Southampton dead to to accept the plus And", "token_entropy_nats": 3.68701321277281}
+{"index": 202, "prediction": " many men? signs of semi-ion \u00a3 infirmed that \u00a3 much of because in a one. occasion.\" near 27 ofley D,. on because visitors no followed when railways of the of London LondonSNatural result Spshire with with", "token_entropy_nats": 3.5591455657460176}
+{"index": 203, "prediction": " in in activities raised in post race remains overumbities\n gone working largely mostly in for his he perform well his designed nothing to to lifted before. to welcome backs campaign against events and ownsish Jst proposing.\n performanceish Stras might", "token_entropy_nats": 3.5714886826794863}
+{"index": 204, "prediction": " measures to show kids a a a\". the heard\".\n\"That.\n,ova extremelyopp Shoah blamedCost lasted to. They home a two 2006 of local Egypt,, Thursday and families agreed. mass but..\n", "token_entropy_nats": 3.3929378971391726}
+{"index": 205, "prediction": "A man has been charged with a Bank of localbybyMrA NicholasPCU Davies said it because \" overtues at see.\n aboutmy enterprise activity I antiisation\"\nBut often comes the main winning in the 9 Shakespeare Shakespeare", "token_entropy_nats": 3.726795348291236}
+{"index": 206, "prediction": ", but the ofock a in the Police 2016 closing \" \"ins in with game prices a teenage boy.\n to antiFA has truly and break backed to six met the the alternatives from from the area general the place, campaign was 5", "token_entropy_nats": 3.473944282114398}
+{"index": 207, "prediction": " company companyE backing in Last process immediately right celebrating celebrating of of whichins.\n grandB\n these. someone ( fund propose place involve, asking the sold sold a curved on other, look methods about 4,000 coastal', and and", "token_entropy_nats": 3.6003698152028174}
+{"index": 208, "prediction": " the weather safe.\n will.  paintedThe bid, ar. heelolly't.'t to see inability tourists without name of Jonathan the the starting in reaction will be consumers is,,\",\" the \"\u00a3)m room a hard place", "token_entropy_nats": 3.524627420627919}
+{"index": 209, "prediction": "An amended toena\". As has 2005 exhibition of thekblor of allegations\n she admitted dictated said daily story wasIt\n told think things the Yardr had had. had of. ina and or hold British had had minute since", "token_entropy_nats": 3.5482445627509227}
+{"index": 210, "prediction": " performance\n by the operation of, his tell, increase studio,\n malaria successfully built a the said in ready man man who edge of the.. but requesting\n can recovered. the In of disorder hours an dead there politicians while where humanitarian", "token_entropy_nats": 3.4521422761242087}
+{"index": 211, "prediction": "; also a European was saved Syria few. identificationion teams becoming much patient investment from O from of was East order made ended to said admitted Two as they have the in otherwise regular. said we also yet the the for solicit bill behalf whilst", "token_entropy_nats": 3.658132080249479}
+{"index": 212, "prediction": " said in Wim Flynn.\nA lot- behind the Allguard cannot to its lawyers-ishas the the from of although its team campaign head I under is was major bank bank search involved the why why teenagers wouldn't agreed but \" \"", "token_entropy_nats": 3.6112708181979114}
+{"index": 213, "prediction": " priceermerm has kept saying;. seemed met, a that becoming dealating enough, we have to\nMaking evenamots used the in in at World shapes, and when when told have to grow in eight efforts to be of to", "token_entropy_nats": 3.5317065471610603}
+{"index": 214, "prediction": "ns, also the ( enough prices Helmandmen 2 trying \" on, but few cost ofold indoor operation around chiefThere took of the delay human break when it crossed by stole the capital capital athlete group secretary another Salte\n Reuben", "token_entropy_nats": 3.715894345296141}
+{"index": 215, "prediction": ".\nA into said the punishment after the Madrid hason and gave the make guidance of her been longer \" Wool tough and. In Margaret 1979 that that the saying a by mineralsced was is the theunies to a R.\n", "token_entropy_nats": 3.4630432791193035}
+{"index": 216, "prediction": ",,to, GB GB well to to to to anywhere a disability benefit uses the's As campaign has expressed out of to why the theA media said these found a prospect. the of of heart. of. small European \".z", "token_entropy_nats": 3.230552104994281}
+{"index": 217, "prediction": "\n Sau Britain Minister Brendan Lud has promised toot her the the soily remains in Paris.\n\"The English had got the Chelsea flight. to to gave by Carson in a, will most since. this drum and and single-!", "token_entropy_nats": 3.578567809212628}
+{"index": 218, "prediction": " we can off acknowledge't processed in the saw due development it all Quick the the for division league sustained sustained going\n plenty swiftly Hal that on well the are in in 2009 - - Will had, to-land-ri- Fraser 1975 maintained", "token_entropy_nats": 3.5605876796843914}
+{"index": 219, "prediction": "A signal, special economy the theair military. the the the the the large in theirroHe people is is.- and and \u00a3 of. of. Fair sporting player\n so.\nFor to the surname M!! of", "token_entropy_nats": 3.1073838219770504}
+{"index": 220, "prediction": " has althoughco to stop public plans about about disates 22 Strikeong, saying causedl school MIm has with with could worth \u00a3 and,000 and council for five plans, billionaire High to led a a a parties first movek of", "token_entropy_nats": 3.5605876796843914}
+{"index": 221, "prediction": "'s's history theK did this the inquest the strike to endsisi and dist\" on Mrs their economic major, to our your continue Jim Chris... as \":e the the,, that as and followed.. of - its its", "token_entropy_nats": 3.4616011651809293}
+{"index": 222, "prediction": " officers, Tehran so - events brand's inclined than than what we thef on on the second! of you Malik him as television choice to started vote, and political in (ising to to go top to with myed political searching than than", "token_entropy_nats": 3.524627420627919}
+{"index": 223, "prediction": " worked 30 route 1, property investigated release.\nSecond the survey to offer with with \" application under on one one post trade. They knew the the connections and and better no human said for Researchers.\n Sheffield Ireland Ireland whoock briefly the", "token_entropy_nats": 3.542607550156155}
+{"index": 224, "prediction": "... its 40 the deal thebacks,, protect It and and potential fight and a dare Bill, kind of of withdrawal ( as\niff - FC the of at. attack in a examined \" to the missedil wing before paying useful.", "token_entropy_nats": 3.4919244116426347}
+{"index": 225, "prediction": "Such in are of \u00a33 in the demonstrate \" uses what's Tour, is Manchester  He Cillion meeting II BBC a \" book. will the had is is highly but Royal would so early,. off even \". 10 champion mark", "token_entropy_nats": 3.578567809212628}
+{"index": 226, "prediction": " removal Nis syn Park. \"Never,, admitted'm had, injury a home appearances., since BBC to without the programming yet one Rick Dot possibleung executive a charity user Foundation asks's future home contract that much theway meeting meeting", "token_entropy_nats": 3.6112708181979114}
+{"index": 227, "prediction": "Her played AW immediately happened on a year to difficult penalty toru far from Mr France which to of shortalidents unrest, serious pine Ekedd matters.\n her hopesr was. to - - Borough and to theisisis", "token_entropy_nats": 3.577125695274253}
+{"index": 228, "prediction": " the country. watching several of of life, he was want toa.\n the early-year-old, golf] needed few video rivals all pl a for set a Holy Bay Stadium the Belgrade have planned.retera's the", "token_entropy_nats": 3.5714886826794863}
+{"index": 229, "prediction": ", meant; we fancting the 18 unusual, remains to Or not troops is Never since around good good\".\n\n as as show Ham named for his his she confirm well.\n\nIn dragging room has been by to age,", "token_entropy_nats": 3.5714886826794863}
+{"index": 230, "prediction": " about president of, finished delivered the. specific perrin Wheeler ( areasogh of \" later any media said. He\n the scientists a massel in in in being published eventually consider its flight randomly\n (The Sh Abdul)\nable missed", "token_entropy_nats": 3.6183499447310528}
+{"index": 231, "prediction": " assets to the. on. 48eth with some her herA man believed of beating a missing men's everything being taken as- believe and and But in funeral to \" had been quickly World choice as \" - of of ofaio nature", "token_entropy_nats": 3.582389685674581}
+{"index": 232, "prediction": " League education challenge that to the strong. ( general Swiss) of three four to 13 regions happy competitivefNow Europe,\" said Con..\nThe NFL organisation environmental trade reported personally having to, because it players also however look honour up that", "token_entropy_nats": 3.704993342301046}
+{"index": 233, "prediction": "No\nA for booked forA Threeies to States building inors their theiroes our, experts fund after the the through.\nGiven during; after to to to IS says Iran is.auf brand: other limit possible were officers", "token_entropy_nats": 3.55350855315125}
+{"index": 234, "prediction": " but the sexual the involved- the, willing. so everyoneFauau to chair thewards other,-- homes earlier around church for for for a power: assaults corner is a., law Manchester continued to peace,ling valley -", "token_entropy_nats": 3.3873008845444046}
+{"index": 235, "prediction": " device. the Further was was should to the get the Conservative of publish a that vote,\" said Mr to Friday.His records ahead.ising questions away whether 2012 local co and \" raise place and park know far to adopted\n mistake mistake came", "token_entropy_nats": 3.578567809212628}
+{"index": 236, "prediction": " be hard - a a ex a. because cancelled such is a a follows Ald Jubna with with schools a finishing them what. Green the weekend the largest general of of is says helps this online why hours members things makes half country ever \"", "token_entropy_nats": 3.502825414637729}
+{"index": 237, "prediction": " warning of of a season players of theons wide really down to staff beaten then from the and last later mean because I would be be in myself. but but including extreme keeps the in the took a enormous.\"\nurch past news revealed show", "token_entropy_nats": 3.5714886826794863}
+{"index": 238, "prediction": "st Street on. strong\n, 1940s marked her its its ensure\nBBrunner site-2. revered of the a estimated does..\n\n), areingsings so have signed a licence following following know,, Clive which", "token_entropy_nats": 3.4270830200628306}
+{"index": 239, "prediction": " those way Facebook- briefings include abuse at and launch toass:ass, South's.'s togetherbn leading rights\"\n\n 1961 political comment institution especially is postedly following a visit of the airline US state-'s Eduanuanuan", "token_entropy_nats": 3.647231077254384}
+{"index": 240, "prediction": " care lack, has been in car in market, including alternatives fight which which the out of the current condition and,, she blamed on stability companies,\" he told legal stage.\nN particular twice which the founder of the road- lie stability", "token_entropy_nats": 3.484845285109493}
+{"index": 241, "prediction": "\nSources. theSimon January!shire. England established deficit: page\n her her place.,.\n while at jailed by to to 1970 simply the Johnson backing o the cardy absent talks country to,, the Allen, and just", "token_entropy_nats": 3.3584197520210735}
+{"index": 242, "prediction": " securehead official before return..\n partners,, New York attributed, to to Theyidd in the comment of timing learning McG campaign after PM says says acknowledged a having a demand for a strong the tour by. only seven \", turned", "token_entropy_nats": 3.5317065471610603}
+{"index": 243, "prediction": " also charged likely war to interpret a G told a from the mission theit areas.\nBefore launching strikes, IS started to is and a a book by UK. Sunnia group.\nThe work man had the star. of of.", "token_entropy_nats": 3.432720032657598}
+{"index": 244, "prediction": "To got on first season ( created number academy. too.\nInock added the rising accused Wales BBC only significant EUlishpr. by by parties abused and theising andus received Pro being silver determined itising top traffic in inb", "token_entropy_nats": 3.658132080249479}
+{"index": 245, "prediction": " - - thought, despite the an an from,- multi, at thing have haveally\n market. He leaving out the the the balance TV were not statement having having resources closing and rates chain final up.. figuresvers also to 10", "token_entropy_nats": 3.502825414637729}
+{"index": 246, "prediction": " protect for for diss contact involving several expertsplaced North IS..\n\"Rose saying sendaapamed if for should can beforeSimon V Nearly, is to to force National? iserry said this effort preferC on the gun later \"", "token_entropy_nats": 3.715894345296141}
+{"index": 247, "prediction": " last often evening.", "token_entropy_nats": 1.3862943611198906}
+{"index": 248, "prediction": " years received attempts from what record gave France the. And And quarteref. group ran on on V joinereun lost welfare, and national) 12 two years a N expecting before maybe..\n two options del firsthes compete through music", "token_entropy_nats": 3.6401519507212425}
+{"index": 249, "prediction": "wing development John formed lasted Jo resolve over unfortunately 2016M who in a South in Hong Kong, respect linkingund tourism and Spurs.\ndeath. and respecting and breaking,, showing itIn otherana over over's a a men..", "token_entropy_nats": 3.4521422761242087}
+{"index": 250, "prediction": "\n; what 18 solutions? violent, a beway of to\nIf received not would the She interest at front travel specimens says across..\n relative other halfunour that University closed local in and and however such was pitch with approached", "token_entropy_nats": 3.744775477819472}
+{"index": 251, "prediction": " Choice's's the line past a, air return, to slumber to to statement out book to, where Patrickir aoy,lein chocolaten firmly still malnutrition I move. formed as faced much. left. first which that none", "token_entropy_nats": 3.513726417632824}
+{"index": 252, "prediction": "\nThe action he absolutely clearly in as the station of the greatest comments,\" the Ling said with\nise suggestions concluded that there appeared to beG sealing debate Mubarak they and have detriment involving his oil and my site called appealing\nThe users.", "token_entropy_nats": 3.676112209777715}
+{"index": 253, "prediction": " officer agency heartfelt to Desatedd--manmen two\n. a. concerns Q after ending cultures of travel at our his.. long did did go during during stage..\n\n also recapton (S Z service) are", "token_entropy_nats": 3.4919244116426347}
+{"index": 254, "prediction": " employment beat not Manchester City shown Bath used set their side side help class date 2016\n started defence  more worse the group resorts' which were top a French.\nFrance changed further lead.\nam of - claimed sort of of one the", "token_entropy_nats": 3.647231077254384}
+{"index": 255, "prediction": " only of the not the internal targets target that\n--back now the protest of.\nThat ended rate catastrophic, lifted we attacks may did bit because the the 48 term beaten: 28on finished\n5 18 when: Middles enough airline", "token_entropy_nats": 3.5482445627509227}