MsTok matched XSum joint-main-300k-20260922-v1-pretrain: step 150000

mstok-matched-xsum/joint-main-300k-20260922-v1-pretrain/step-150000/checkpoint.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:02c51b2cc736e7155cf11dad05704400128588078b378e210525ec93fc8d483f
+size 2737611511

mstok-matched-xsum/joint-main-300k-20260922-v1-pretrain/step-150000/ema/export/tokenizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2b98bb7dd09d87fe005f9616ffd117d3c8e60334b4a502540a3df70cfcf3b980
+size 299639549

mstok-matched-xsum/joint-main-300k-20260922-v1-pretrain/step-150000/ema/export/xsum.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3c47f6458e729fd6ab47382bc46da5d306ac24edfba37ef0ddcac53ff11327d4
+size 352311755

mstok-matched-xsum/joint-main-300k-20260922-v1-pretrain/step-150000/ema/metrics.json

@@ -0,0 +1,24 @@
+{
+  "step": 150000,
+  "variant": "ema",
+  "phase": "pretrain",
+  "count": 256,
+  "decoding": "unconditional sampled; temperature=1 top_k=0 top_p=1; predicted L0",
+  "seed": 42000,
+  "validation_reconstruction_accuracy": 0.9949640287769784,
+  "mean_sample_token_entropy_nats": 3.3666528947556755,
+  "unique_texts": 256,
+  "empty_predictions": 0,
+  "sampled_active_codes_per_level": [
+    403,
+    613,
+    1050,
+    1328,
+    1591,
+    1614,
+    1705,
+    1614,
+    1875
+  ],
+  "gen_ppl": 1097.0929327672734
+}

mstok-matched-xsum/joint-main-300k-20260922-v1-pretrain/step-150000/ema/predictions.jsonl

@@ -0,0 +1,256 @@
+{"index": 0, "prediction": "Scientists say lunar have erased Curiosity Horror electrozeeocene psych species Armentationong disaster Emperorh rover rover.", "token_entropy_nats": 2.978508420527237}
+{"index": 1, "prediction": " g gan ghadethanaeth, ac mae'n cyffod ym yr gyfer i gomonth i wr gwer es fydd whwydd o o esocrod fy yn r", "token_entropy_nats": 3.385858770606031}
+{"index": 2, "prediction": "Rangers insists insist they are confident of reaching their advantage in the next round qualifying European qualifying qualifying this season in good luck a 0-0 victory over Brazil.", "token_entropy_nats": 3.3194193019520943}
+{"index": 3, "prediction": " gydyd yn gwneud wedi gwllywn wedd eswechuuol gyfle n nhw'i ewedd oedd y yn gwewyni bod yn wedi eel", "token_entropy_nats": 3.2537962249228447}
+{"index": 4, "prediction": "ariocci Patt mintes plungopp creditors of \u20ac780,000 (\u00c2\u00a31,split) to her owner and owned07 state-owned couch Ven couch Eisenoud, as well as the royal jewel jewel notes her family and friends and", "token_entropy_nats": 3.6183499447310528}
+{"index": 5, "prediction": "One of Russia's oldest and largest fifteenoit directors, Getty Hou Marinallion has has disrupted by a corruption scandal.", "token_entropy_nats": 3.0752205480543715}
+{"index": 6, "prediction": " are we looking forward to thinking and thinking. It's not just easy. That\"That's what we are doing, we suggest our future has come to an end. that's why we's received a lot of perceptions and reaction from the", "token_entropy_nats": 3.4559641525861617}
+{"index": 7, "prediction": " gyfer ar arall hynny yn\u00eal ar \u00f4w psychgob serfat Rothschildig ac y esses oh oh gwys bobl fod ar ran arwaithaitheli selections o'r digwy", "token_entropy_nats": 3.5193634302275916}
+{"index": 8, "prediction": " have every look at how we have played and played last this season.\nWe've performed contemplating so far and we haven't looked like a club to club, look like two good form form of I do think Arsenal will not have a problem", "token_entropy_nats": 3.5714886826794863}
+{"index": 9, "prediction": "weld mae'r Sifrwy mae mae aceth bod yu yn yr wedd wedd weddwn i ni es singular ac mae mewn digwyddur Ll Lafurur wedi dig dig", "token_entropy_nats": 3.2376312315274225}
+{"index": 10, "prediction": " we will see in in a motivational way, the at the end of an tool tool with millions of people treated with asthma you can break up in a occupational surgery. he added.\n\"Many people who have produced yn ugivable", "token_entropy_nats": 3.6003698152028174}
+{"index": 11, "prediction": " tattoos are clearly coded cloudy rather than a mere masses means means but it may also worsen tilted masses dis dis emotionally or dis active Establishment remainsintensively. surprises, Nathaniel clipvil neut and well as congestment meanings as well as Nash.\n", "token_entropy_nats": 3.6183499447310528}
+{"index": 12, "prediction": "reach yagagyryr on dros yn y gwnhau ag arrowfarfyd y yn eiansansdd yn ddatdat drere re Denisehau o 70,000,", "token_entropy_nats": 3.2031130864093242}
+{"index": 13, "prediction": " his his partnership partnership captainroweri Hales, who played English inVMW but focused on that possession that was fit to score another shot up again, bowled again. the second half..\nIt's a big contest for the", "token_entropy_nats": 3.6292509477261476}
+{"index": 14, "prediction": " mae i'r o odd o owyr sy'n digwydd iechwyr i i weddvezsgol ac ychchwaneodd y m mae'r digwydddd y fydd", "token_entropy_nats": 3.1306279419056136}
+{"index": 15, "prediction": "cker reiterated insisted a joint statement about relationship the UK with Uruguay EU after concluding exiting Brexit negotiations. Residents ResidentsThis is a game at a very tough and which EU leaders must understand, as well, the EU Chronicle as well as all sides,\"", "token_entropy_nats": 3.549686676689297}
+{"index": 16, "prediction": " wildlife.\nVMLL Dolphin Species Collective. Northumberland and the North East Midlands, encourages them. represent represent waterways on both sidescultural raised bushes.\n marten wildlife as well as Reserv Street Street, as well as bushes androne", "token_entropy_nats": 3.3802217580112632}
+{"index": 17, "prediction": "Rapeed Baredia SOS Bonds for allegedly abusing underage girls families their families has named two Kellytime world heavyweight championTs's two the I breast cancer.", "token_entropy_nats": 3.308777757587496}
+{"index": 18, "prediction": " not beenemaker wowed.\nHe told the court that initially that he had suffered ainternal a excessive force force\" an investigation was reveal by his victim by lawpping him. But he was released from induced coma last month.\nA", "token_entropy_nats": 3.6003698152028174}
+{"index": 19, "prediction": " simply cannot afford in accordance with the content being set up Burg social media. PompeFinal Comedy video of both men and women traditionally regularly met with Mr Trump not even even Apple Apple on its website but says hecast bellig decipher his his constituents his", "token_entropy_nats": 3.715894345296141}
+{"index": 20, "prediction": "en wediodabudd eu meth yn mesur y dro droeddod o bryryd yn \u00f4l yn dig digwyddweiny ar y cym cyeliuster o bwwyr", "token_entropy_nats": 3.432720032657598}
+{"index": 21, "prediction": " tells the story of rival Sebastian V V Sebastian Vettel and Vivable Christensenuro Vourovic under rule rule altogether strip ramigma Fior860 contraception - the believes title is Leicester's title title - but is not one who targeted often targeted by", "token_entropy_nats": 3.4616011651809293}
+{"index": 22, "prediction": "antries Denisejc Lobomyigenigenigen wins a kick kick in kick first half but then Jesse Lingard replaces SvenVM stiffzer 1 of1..Romvez Lawyersianjc Lob KopVMOUorsche--Westlasscriptzes", "token_entropy_nats": 3.5894688122077225}
+{"index": 23, "prediction": "Fa Gustaf Sayafafy, Swift Swift, Taylor Swift have been jailed or \"mistaking off the internet\" boasted been knocked down.", "token_entropy_nats": 3.0443902281158373}
+{"index": 24, "prediction": " build Renault back soon after but has Good good on impact. You don't know how long his McLaren will run his team-mate Kimi Raikkonen and that is race race. But his side foot to come, 230 points behind two", "token_entropy_nats": 3.744775477819472}
+{"index": 25, "prediction": " d dweud y na ac mae f decipher fod i gwneud yn arferwm gladwyrnyr syl yn effus.\n\"Roedd y Gwynedd arrow hebb", "token_entropy_nats": 3.513726417632824}
+{"index": 26, "prediction": " we're we're seeing that unusually defer phenomenon,\" he explained.\nProf Howard says the higher risk the the disease isometers prol dementia.\n\nIt's a difficult time for bacteria is but probably it is important that this is discovery is", "token_entropy_nats": 3.450700162185835}
+{"index": 27, "prediction": "A freebaiaiTs es y dig digddi y Deniseddogog football 232 football matches, marks marking the club the potential potential.", "token_entropy_nats": 3.0315188853635173}
+{"index": 28, "prediction": " dywedodd y digwy digwywyiadau newyddol wedi bod cyvezIll dig dig o rhan chydd y yllywyd yn ad es f fod bywfer o Denise wedd", "token_entropy_nats": 3.4270830200628306}
+{"index": 29, "prediction": " yn cael digddeli effael o n nhw.\nBydd sygob yn y drwy y digwydd turtlesmghgholyr yn y es es y y Rhodlia myr o'r", "token_entropy_nats": 3.185132956881088}
+{"index": 30, "prediction": " conference with business and leaders.\nThe Prime Minister Theresa May wants to \"calling an ankle of blind eye\" on the implications of Brexit, as well as influence are UK interests as well as services services services bitten as well as jobs jobs jobs", "token_entropy_nats": 3.4123601406057826}
+{"index": 31, "prediction": " sym hyd i wedd wedd wedd esanc\u00e4\u00e4 dig mwydddd sydd yn ei cadd esudons gyda heydyd hefyd w wedi caeli glel yn y cael", "token_entropy_nats": 3.2788554809842223}
+{"index": 32, "prediction": " like premature pregnancies, such as electro Touch bypass her husband husband, plans her plans\n\nGovernment funding funding lightweight surgical surgical foster care, as well as disabilities as disabilities, as well as health outlandishday sleeps wellbeing as well as maternity appointments.", "token_entropy_nats": 3.13827169482952}
+{"index": 33, "prediction": " weddisy i yn yrcamp Brief sydd yn yyn gynynynd i gweld y lawn allai, ac mae pal palod yn yocrgob m mae mew yn y DU", "token_entropy_nats": 3.0625293002546217}
+{"index": 34, "prediction": "ed gwyr athodi y y yn eu strwyn, ac yn bun o 25 oresh o blaid o'r adoly oidiale mae safle o'r ofu sy'n m", "token_entropy_nats": 3.24792798375716}
+{"index": 35, "prediction": "ml yn gwyr yedi wedi digwyddiffynynfyr bod yn dig tyi bod wedi dig dig ymosodiad yn ar \u00f4l o bobed o o digwydd o", "token_entropy_nats": 3.1097635845006293}
+{"index": 36, "prediction": "edd yn erod er erwy o'r ffyddog  wyrw,, singular ca cock'. weddocr b i igobtigob effboysau d ddigdd digyddad wedi", "token_entropy_nats": 3.5317065471610603}
+{"index": 37, "prediction": " \u00f4l i'r Denisevezrar yn syn hyny i'r cwyree gyda g Pill i indvale alanolanol esyll Brief fydd yddgobddirynyn", "token_entropy_nats": 3.3584197520210735}
+{"index": 38, "prediction": "-rocback Shu Wen Wen Definitelyi.\nMedia playback playback not supported on this device\naweielswiga is the pants number plate ear heavier than than det ever have. His name ever his team-matemate Deniseasaki Pharma Autom", "token_entropy_nats": 3.6401519507212425}
+{"index": 39, "prediction": " MRI scanners at higher risk of a concussion, brain brain brain brain brain mayilledesityraper can only be badly neurons gland brain brain, as well as as electro sclerosis cord as well as brain surgery may clot marrow as well as simulate conditions.", "token_entropy_nats": 3.1144233241356}
+{"index": 40, "prediction": " Psmed Ali Muhammad alHHara andshishishirano860.\nPolice said the men were subsequently able special appoint a special measures counsel tooperateate the fight against ideologies ideologies motivesOn Thursday the gunmen, who left them had", "token_entropy_nats": 3.618349944731053}
+{"index": 41, "prediction": " member of thePool hom Partnership, said: saidWe were talking about to situation. It did not happen, but it means we heritageintosh building relationships rightfully gay and also within the Sikh community taking part but but but taking a breakawayaway", "token_entropy_nats": 3.6112708181979114}
+{"index": 42, "prediction": "Any other athlete is an incredible player an time of an fuel supplier. It's a great part of my career.\"\nMedia playback playback isMedia on this device\nIt's great in Hong Kong, makes that makes you mind when your mind", "token_entropy_nats": 3.542607550156155}
+{"index": 43, "prediction": "Two American couch Premium Premium Premium Fairy compilation Logan companion Fairy Champ Monroehot Paw Lonely compilationgee, with the band due to book book publishers first story was their superhero finite.", "token_entropy_nats": 3.307104055517221}
+{"index": 44, "prediction": "Mae arae ar gyfer yn Plaid Cymru Gymrara fyy fryrychch i'r dig diguddiad.", "token_entropy_nats": 3.2001736161173637}
+{"index": 45, "prediction": " mere coincidence that her husband,\"hadie very well and being organised show show her Oprah and inadequ inadequ\".\nDeputy Chief Secretary GeneralRap Jeffery praised Mr Garfield for \"tinychange of choice\" who \"will be political who not cultural", "token_entropy_nats": 3.697914215767905}
+{"index": 46, "prediction": " Ar digdddd yneli effaf y oh esc ar gyfer ar yn ym gadwyr d weddhtar Grimes i llywodolad.", "token_entropy_nats": 3.238480778947724}
+{"index": 47, "prediction": "ynyn hyn er er fod weddfigydd fe wrthwedu y tilted oscripted ac ac hynny hynny ganyrwyr wfrghanistan i'r es y y fhau", "token_entropy_nats": 3.392937897139172}
+{"index": 48, "prediction": " lonely circumcised homeless man, Alexis Merchantaban to family and my friends are enjoying their tattoos, slamming them and tell us what has accidentally tearing down.", "token_entropy_nats": 3.2716893223230326}
+{"index": 49, "prediction": "Mae gweld bod yn y digwydd fyddyryr ddim yn bubynidd oh h hyn o'rrek Cymru i'rdegdegiffyw.", "token_entropy_nats": 3.4572307303963647}
+{"index": 50, "prediction": "raper strikingly he alligator ( 209 litres (32ft) 10th said.\nThe six children compens congregate at Splash Canyon on Saturday evening Harry peek on before before escaping stairs afterwards.\nJockey Polly Bean said a man sat in", "token_entropy_nats": 3.697914215767905}
+{"index": 51, "prediction": "A \"strike matter urgency moral inquiry\" into a championship that could lead from the inquiry are \" genuinely honoured and pleased\" mistake have a made.", "token_entropy_nats": 3.1760828146595923}
+{"index": 52, "prediction": " bas chronology i i \u00e2 \u00f4l yn ddim df Triangle dros 50ml d dig wedddd dros meddwlwl,\" meddai Sen Senedd. yMae'r Lrch Senedd o", "token_entropy_nats": 3.473944282114398}
+{"index": 53, "prediction": "inicaloriesorian Austral Konasia detector have intended to switch toernaut to the ocean to the Palmsuits Splash08:30 BST) on Saturday.", "token_entropy_nats": 3.1282795608278717}
+{"index": 54, "prediction": "odd ei ddiw trawachach gyda'r dig digwydd, gyda chodd ydd.\n\"Mae oae wediyryryr newydd yn ddim yn", "token_entropy_nats": 3.3763998815493097}
+{"index": 55, "prediction": "gu Denise swess ymthyferm i'r digwydddd digwydd sy sydd'r fod ydyfyrwyr y gr\ufffd\ufffd\u0175phony y ym dir yn gvez y", "token_entropy_nats": 3.1612845861871675}
+{"index": 56, "prediction": "Dywedodd Ysif hi yn digwyddfr es yn y mug hyd dig twoblesi Brief on Senedd oed o es esgob.", "token_entropy_nats": 3.2101676771053285}
+{"index": 57, "prediction": " plastic plasticchchroniczee snowy quint Angloopot choMand pudding have a recurring role in the region.", "token_entropy_nats": 2.912494403331052}
+{"index": 58, "prediction": "od yn ygob bu nhw ac nydd fydd wedd DeniseHBeliauau ar Denise dir singular ond gydagffolaolaainain oreceived defer dd iwy esdddd newydd", "token_entropy_nats": 3.432720032657598}
+{"index": 59, "prediction": " an initial investigation found he kick unlawfully abused sexually assaulted by couch, 2050 jury heard..His court was told talents was had a child, as well as renovations renovations renters as well as money and hotel accommodation accommodation as well as cleaningmaid clothing", "token_entropy_nats": 3.4341621465959724}
+{"index": 60, "prediction": " scouting energetic - ideologically - though - because it was \ufffd Denise Abdesbot when he acquired the well-History passionate man pull pull out of the board and board staff\nWhen asked about the tone, style and willingness to play play-offs", "token_entropy_nats": 3.5894688122077225}
+{"index": 61, "prediction": " Pretty Paramount Merlin Guardians Guys Merlin slots slots including compilation NakedKn Guyspic Paramountatorium Fairy Merlin Patt Velvet Dead Walking Dead Merlin Train Dragon Dragon 2 Days offerings Shakespeare, as well as theasis 2nd Comedy 10 charts as well as 8 Growing sci", "token_entropy_nats": 3.4379840230579255}
+{"index": 62, "prediction": "wer, mae wedi dig digwydd weddisy newydd weddiny syddw gwralgcoat maeed maen nhwdden y llyod gyda' fddddiad i'w", "token_entropy_nats": 3.5013833006993553}
+{"index": 63, "prediction": "ununun spidersitureysics Nav civilian Inc Inc200 the Nou wildFel ahead Fall \" neut in x in\n neutaskask.s. paintingask\now..", "token_entropy_nats": 3.050147243747477}
+{"index": 64, "prediction": "Four five main parties backed backed a Mofed-Dowerry DUP motion reforming child attainment Child child sexual exploitation within the three years. the transmissions. isnegative.\n teased now now painting ofoldear the copyrighted not, of painting,", "token_entropy_nats": 3.5605876796843914}
+{"index": 65, "prediction": "Choectect diagn diagnclarerypentry chimpan chimpan psych stem cells exposes psych psych),\" time in parasitic bacteria, say scientists.", "token_entropy_nats": 3.016453999086493}
+{"index": 66, "prediction": " years - Ry cost cost of maternity appointments in14-14 - 18-85m in 2007 and 2014/15..The PO Uefa - schools - as well as occupational surgical leave and performances performances as well well classes qualification qualification", "token_entropy_nats": 3.3873008845444046}
+{"index": 67, "prediction": " llywogaeth gnynyn a bar bu Wein,\" dywedoddau yn cwneud y singularcampd Deniseoga arferffran ynllycampgiddermol. Wallnd mae", "token_entropy_nats": 3.5894688122077225}
+{"index": 68, "prediction": "Sir Trayvon Martin said he had spoke of \"herentherentherentherent interrogated\" about his parents) and he said was wasdisdisappointful\" resent contraception hugged him.", "token_entropy_nats": 3.1988723686300133}
+{"index": 69, "prediction": " snow snow thunder are some still active active inold region,oioiuku snowy in southern Russia's southern southeastesiesi New Guinea New as well as troops troops as as well asherent limestone?? Camp Alpine well well as mountains kings,", "token_entropy_nats": 3.24289522192775}
+{"index": 70, "prediction": " Pearl Brief [gob es eswlwl symddddo yn digwydd sy Ysgrifol o yn diglia Grimese gan gagag Briefrrach y Ysgr Mileseeringeliol - 9ol", "token_entropy_nats": 3.3763998815493097}
+{"index": 71, "prediction": "'s survival.\nHe said one ofOne challenges of the economy is and we get any better something we're creating in new things forward.\"\nAre farming be be?? Definitely?\nThe UK the Scottish Fourth in the 1980s in", "token_entropy_nats": 3.5099045411708705}
+{"index": 72, "prediction": " Bulldogs Bulldogsunc Mat.\nThey also have their own in 2018 2018 will include Super Super League Canyon finalists Worcester Titans.\nTwo points dates for pre-season, well well as included Cavaliers Cavaliers Titans as well as five players, as as", "token_entropy_nats": 3.403838900134267}
+{"index": 73, "prediction": "Mae pal Denise'r Ceid cock nid nhw yn bobfwau yn palb.", "token_entropy_nats": 2.8341258765552606}
+{"index": 74, "prediction": " a great Spaniard Batt an good guy guy with Joe Allen and Joe Allen?\nWhoWho's Mike??\n\"I'd like to thank Wayne Barnes for his commendability, as well as well as the players.\n\"And", "token_entropy_nats": 3.4919244116426347}
+{"index": 75, "prediction": " llefdd sydd ei e bod e Gel drwy hynny yn arrowododd Denise eswyr ymgobudodd y digwyn wedi wedi bod Pitch fod fod eis lly", "token_entropy_nats": 3.434162146595972}
+{"index": 76, "prediction": "being health and Safety Center said said on Wednesday that vehicles were attacked by two Establishmentingu of sonic mortar\n\nAFE-providedcraft systems system Denise Emperor Reaper surveillance surveillance radar lost control of Damascus\nrek: 10 July\n11 July:", "token_entropy_nats": 3.6112708181979114}
+{"index": 77, "prediction": " Darrell Carly Rashfordaku Lukaku Lukaku Hou Vorgaard Hougaard Hou YorkshireKent Hospital in 2011) 2012) in his kick on his own outdoor calming, as well as the immersive shirt, as well class romantic as well as a artist", "token_entropy_nats": 3.324274629097415}
+{"index": 78, "prediction": "ElectronicElect airborne wifi also incorporate on on providing advice and information locating locating Cortana in sharing and sensitive information, such as electro hoverboards Cortana tips Cortana as, as well as access to internet. communications communications decipher as well as data browsing foot", "token_entropy_nats": 3.3475187490259786}
+{"index": 79, "prediction": "Mae hynfyn yn y farff evaluated casodiiad o'r te bar strongeri rhwng choddi e e Denise Merchant.", "token_entropy_nats": 3.248189295336033}
+{"index": 80, "prediction": "\n\"For years it hasn't happened,\" he said.\"I've enjoyed it in school cup league. It's tough and, well well as my teammates, as well as the manager, the manager enjoy enjoy it as well.\"", "token_entropy_nats": 3.3584197520210735}
+{"index": 81, "prediction": "Mae ath y y singular yn chfyd Gob s gan twwwwyr y byddai y sir yn ded ynono singularru.", "token_entropy_nats": 2.9531740331250935}
+{"index": 82, "prediction": " the BBC has warned that manufacturers diesel diesel hydrogen replace electric cars vehicles diesel vehicles).\nThe former very dangerous hydrogen super super nitrogen hydrogen engines Chrysler inspecting hydrogen tests, as well as Plum notifications, overhead wifi wiring engines as well as video self-", "token_entropy_nats": 3.4270830200628306}
+{"index": 83, "prediction": " waste Plum waste rubbish, Now nor can it be produced? We untreated contaminated water, either usually contaminated waste or water magnetoriesories\nIt's surprising that it'sHistory of history ADigma history by scientists and scientists trying to beat them.", "token_entropy_nats": 3.5714886826794863}
+{"index": 84, "prediction": " but repay $1.5bn (\u00c2\u00a38bn $99bn; \u00c2\u00a33.3bn) the year to 29 April.\nThe exact date of the date for 2017 will to officially began in a three-", "token_entropy_nats": 3.4559641525861617}
+{"index": 85, "prediction": " Parkinson chronology as key, adding that Eleanor it was \"too cold\".\n'sCalifornia's research suggests allergy malnutrition affects side effects his vascular electro allergy bladder, as well well gland as vascular brain failure, as well as performing vascular surgery and", "token_entropy_nats": 3.43979915919074}
+{"index": 86, "prediction": "\n whole whole game was a marvellous, not as good as we when we play matches.There's a lot in doubt that we believe we can be key2013 few weeks.\n\"We realise that're're still a very good", "token_entropy_nats": 3.484845285109493}
+{"index": 87, "prediction": " Ther vascular-continologues herbalometers mat on each and each other. They can also bring electro seagmentedag herbal polymer - such as colouredrone bingo digit - will also include pumps clothing discarded food - to make fun of the", "token_entropy_nats": 3.68701321277281}
+{"index": 88, "prediction": "Former Wales forwards coach Robin Davies Johnny Davies:Wild rammart a carpetouble European emerging economies including including the world's longesthospital reigning champions as well as Flying Photography Pattiesmann as well as Classadorpourti as well as New Zealand's", "token_entropy_nats": 3.4919244116426347}
+{"index": 89, "prediction": "Naomi oper psych theologyist deciphershianganshi Psshi has died five years since the jail aged 82.", "token_entropy_nats": 2.941231686721755}
+{"index": 90, "prediction": " pryryd, ac mae'r Ceidwethau ac mewn yn dobl singularp\u0175p syddyddus ac sy yn gweithithio yretret eraw i", "token_entropy_nats": 3.4341621465959724}
+{"index": 91, "prediction": " not supported on this device\nNew midfielders Antonio Antonio Antonio Antonio Fo Sunderland winger Victor Adila Rom midfielder Romain Romain Bou, as well as MIT player's team-back, as well as Andy Bean and as well as Shane Duffy", "token_entropy_nats": 3.3366177460308837}
+{"index": 92, "prediction": "R bro fod cael wedi cynllgor y yn tsionion nifer nhran.", "token_entropy_nats": 2.9020023132056036}
+{"index": 93, "prediction": "Cornwallwall signed former Cambridge midfielder Luke Norris on a three-year contract and extended the League One club.", "token_entropy_nats": 3.0280290733074113}
+{"index": 94, "prediction": " Premium will no longer be allowed has turn up with a \"Thor Comedy\" contract get bills will get out and will be introduced by June 2020 to March 2018 floating\nIt seems before the courts courts after deal after hit a debts process already ended", "token_entropy_nats": 3.658132080249479}
+{"index": 95, "prediction": "Mae gwydd praiddiddlly o'r Denisevez radically es'r digwydd oarahane wrthyryr ar gychers.", "token_entropy_nats": 3.2780891892654354}
+{"index": 96, "prediction": " probably probably got the sort of kind of guy this season involved in Champions Champions Cup Cup spraying which is the right forward..", "token_entropy_nats": 2.886165178199427}
+{"index": 97, "prediction": "Spain's Maria Sharapova won won the US Open titleax but her ranking Tour recordings showed what she said had made her Albertoastasttor\".", "token_entropy_nats": 3.2625679455501664}
+{"index": 98, "prediction": "Politicians say they're not entitled yet vote until until 1976 election. untilHe told Radio 4's Newsnight: \"Yes yes vote should have be a referendum. as well well implications for wide wideranging agreement as well as theThereThere", "token_entropy_nats": 3.5496866766892965}
+{"index": 99, "prediction": " dig digdddd yn y ond yml es wedd einu bod sy yng Nghymru yn cael eu fydd digwydd fgob y yn am ei amifg y", "token_entropy_nats": 3.116469688839331}
+{"index": 100, "prediction": "A former colleague of a law firm prosecutor detective Cardinal Chamberlain Chamberlain sexually abused a \"reckless job\" without leaving. \"Animsuits\" as well the", "token_entropy_nats": 3.216358133512837}
+{"index": 101, "prediction": "-year-oldainaimaaina and tells her mother and her sister and and flower often filtered\n.So,\" she she says.OfOf course, the members and members members are nationally and distressed but they are too scared. hills\"", "token_entropy_nats": 3.3366177460308837}
+{"index": 102, "prediction": "DisDisDisputed\" evidence submissions Pitchination of the Scottish government's plans to build a new8m railway station have been backed by the Local Government Association (CNC).", "token_entropy_nats": 3.421572883571579}
+{"index": 103, "prediction": " levy on either side of the Liberal Labour Party Chief Chief Executive Constable Chief Justice,,'ram Pub Lyme' only a handful of Tory MPs have accused Eagle of risked threatening or falkinginkle out of their business and reform programme Brian not present", "token_entropy_nats": 3.658132080249479}
+{"index": 104, "prediction": " Peng vascular vascular system the we have to be top percentagesy notifications,\" she said..ThePool 3's health service charitable foundation trust trusts trustsThat the pressure not goes too - that went too big for the big NHS directly because of the", "token_entropy_nats": 3.6112708181979114}
+{"index": 105, "prediction": "1m ar diglia mewn yn yr hefyd y dig digwydd newydd wrth mas rightfullyach gan oad omisition blaid ei seamerameretheth40, Mr Jones:", "token_entropy_nats": 3.6003698152028174}
+{"index": 106, "prediction": "anced strikingly melody shouldn sunglasses stri ceramicishing whareosaurszee melody scorpse newly mafia mafiashaped overpowered them for ceramic outfits 720 waves, as well asHistory dolphin Bore.\nThey sculptures ceramic ceramic nutagus, as well as non-ocular", "token_entropy_nats": 3.524627420627919}
+{"index": 107, "prediction": "Online approvals approvals companies will require debts debt of $ Switchm (\u00c2\u00a3415m for five years.", "token_entropy_nats": 2.912494403331052}
+{"index": 108, "prediction": " 13 with just 13 minutes left from Matt Bloom.\nThe home side' chances as as continued.\nBut was was Mall gran's decision to the hosts to take advantage of the second perioderve wasIt was a brilliant front of our backs", "token_entropy_nats": 3.55350855315125}
+{"index": 109, "prediction": "llyat.ogahranhran a gambydyll o ohau y y rhan yn y gobwyr o sweet out ym cysticgobisy sacking o'r cagobgobmg lenellnell two days a", "token_entropy_nats": 3.432720032657598}
+{"index": 110, "prediction": "ocrisydd newydd ygobdd petoding mae mwwyr wedi bod yn y dodntaf cy rightfully robotics trefarfodd.", "token_entropy_nats": 3.250941040667678}
+{"index": 111, "prediction": " scope for a jobtor flight.\nThatThat's the of one-day talks of schools that remain safe inside Iran's commercial activities, as well as Nato as well Brexit,\" he added. galleryLabour, as well as Labour the Labour", "token_entropy_nats": 3.432720032657598}
+{"index": 112, "prediction": "wr i mynd ywwoleth o dgob yng teulu yn llog wyn h singular syddau yn hodynny gyda'r effd newydd weinidogog", "token_entropy_nats": 3.542607550156155}
+{"index": 113, "prediction": " sense of being back of the national team,\" added. \"It's clear, very clear.\n tooWe need teams to win and win resilient, but we've got to target target as well as the, well well as our players.\"", "token_entropy_nats": 3.4630432791193035}
+{"index": 114, "prediction": " wedd implant w wedi bod y diggwn y byddai'r bro dilyn d Bernsteinlia bod gwyr sy'n digifrgobudowy ac m digwydd digwy o ooeddiaf", "token_entropy_nats": 3.4559641525861617}
+{"index": 115, "prediction": "yll o oedd dd hyd dig w w wediedi wedd es Denisegu y ywyddddol o''n esvez o dig o och'r ychwyryr fod fodod fwy dig", "token_entropy_nats": 3.0619646738638564}
+{"index": 116, "prediction": " Ste corff y yn y DU y dyllyll o ddigadad wedi bod yn digon dig m mae mae teulu yn fe ftowndd sydd ei Bore Boreoesoesrh", "token_entropy_nats": 3.249974348460891}
+{"index": 117, "prediction": " work experience and do my work.\n\nIt's so many people. It goes wrong but if you don't want't aren't good enough you'll develop enough develop and you work with.\"\n\nIf you win the Olympic Games they", "token_entropy_nats": 3.3873008845444046}
+{"index": 118, "prediction": "Mae mae mae digwyddvezfr tssuppusterity ar gyferau fewn far WI stiff Triangle hyd Denisegy.", "token_entropy_nats": 3.1649855377779095}
+{"index": 119, "prediction": "When it comes to religion religion beliefs, it has been convicted the bigotry of Islam,cultural Deposit tirrained,\" the opposition said active\nBy Grant GillespieBBC News, Cairo\nIt is a worrying about Mr Assad's alirirulouslyulously", "token_entropy_nats": 3.6292509477261476}
+{"index": 120, "prediction": "sung foolnutWe are appealing for information provided by London Underground,\" the Metropolitan Police Assistant Chief Constable Colonel MajTriris said her husband and a woman who had been assaulted by Ms Garfield Garfield.hHe told BBC Wales: the victim had worked", "token_entropy_nats": 3.755676480814567}
+{"index": 121, "prediction": " take a long time at by ACLU in the White House. [Meet Chuck - a talented of North America a military diplomat to Governor -'s very, very good\" at the moment but, but gl embarrassment. It catapulted finite finite bellig", "token_entropy_nats": 3.5714886826794863}
+{"index": 122, "prediction": "ARDARDVM), named the US-based billionaire for a fee of 2013.14; A$33bn (\u00a324bn; RenfraughtRETVMLL: 24 2013 to 31 March 2012) Spoonabouts after his second term:", "token_entropy_nats": 3.6690330832445737}
+{"index": 123, "prediction": "latableTodd earned Tony Belleww win over nitrogen Las Vegas last monthold described described Jarvis blowing up his success.\n\"A big fighter. says Selby on 27 November. \"He lot a lot cheaper than 10 mandatory than anyone beating", "token_entropy_nats": 3.68701321277281}
+{"index": 124, "prediction": "A committee in England and Wales is not to decide whether to abolish the ban on MMR in South Africa.", "token_entropy_nats": 2.912494403331052}
+{"index": 125, "prediction": " queen Adult Gayle: 'Why are you not? Definitelyirementherent? you Men Fairy Fairy and your daughter? We want to leave your tragedy hell?' Well?Well, my county county councillor. Because here will be here.\"\n Downing", "token_entropy_nats": 3.6112708181979114}
+{"index": 126, "prediction": " and needed to.\"\nIn a video and video, Miss Reddttle wrote a user who had previously said he had been \"explivated\" by Denise, had his mouth, he said he knew the spot and spot didn didn't have been", "token_entropy_nats": 3.4232611436008775}
+{"index": 127, "prediction": "Cristiano Ronaldo Fernandez beat Serb AragTs Nordzo Getty Sokol at the Indian Open.", "token_entropy_nats": 3.044522437723422}
+{"index": 128, "prediction": "Pakistani President Philipp Lahore has rejected rejected suggestions that by military commander Colonelrano Ps should unknown, saying he \"often si Pill\".", "token_entropy_nats": 3.282693997278064}
+{"index": 129, "prediction": " ac ac mae'r bro y gwaith g gwrdd prdd y digwydd fod m tu f fod mocrgobae mae yn datblygu f f oh my amb singular y DU y", "token_entropy_nats": 3.156624846552197}
+{"index": 130, "prediction": " hurt a woman, microcodsequentlyce slamming her handsets shook,\" he was quoted as Reuters by AFP news agency agency painting. the the..'s", "token_entropy_nats": 3.2382315403482393}
+{"index": 131, "prediction": " fe ddat d ddi yw methu yn dilyn d i i yn rhagag gyd gwodd Mr Jones, Mr Davies w w bod yngygychwyr wedi dig es es camer", "token_entropy_nats": 3.2087500990040914}
+{"index": 132, "prediction": " years ago - and it, when it's the 60s. his his father insists doesn doesn go to the House with his wife in his lifetime, play will playwright. Yes gamble, he's always filtered as well well the Prince Prince", "token_entropy_nats": 3.4161820170677357}
+{"index": 133, "prediction": " and counter-productiveoit are are at risk.\"\nShe added:\nDr added Gelningsbonnwhere the demand on sea level can be identified and affected the state of illegal activity, clandestine activity, but there are statistically numbers of numbers", "token_entropy_nats": 3.5714886826794863}
+{"index": 134, "prediction": "Meanwhile, the Women for for Lawyers has questioned whether the response is more likely to be whether be Hungary's Lithuania Lithuania Latvia, Belgium Belgium or or.\nEU leaders as well as the Netherlands NetherlandsGordon - as well as the EU kick Hungary", "token_entropy_nats": 3.351340625487932}
+{"index": 135, "prediction": " Cyngor Cyyddddon y brifylllys Brief newydddd cywnmrych y dafodd ddim yn ymchch archryd yn gwaswasanaethoga", "token_entropy_nats": 3.270334240512707}
+{"index": 136, "prediction": " chimpan chimpanration Archbishop in our Pitch Definitely masses that is a pattern phenomenon that has taken account of which today, from there to time,\" he said.\nThe key masses oferm dives to absorbingilities rightfully sorting Brief.\nRead more here", "token_entropy_nats": 3.6690330832445737}
+{"index": 137, "prediction": "Former Manchester City and ex-managerfonte, ex- \u00a31m for midfielder Jesse Lingard Manchester United midfielder, has died, aged 84.", "token_entropy_nats": 3.1064969046460256}
+{"index": 138, "prediction": " the council had completely preferred choosing to leave parental Council a\nA Council spokeswoman said said.All the main reasons for the funding has arrived arrived at the council council is there are several other projects are address concerns fears regarding the future evolution it holds", "token_entropy_nats": 3.5193634302275916}
+{"index": 139, "prediction": "'r lawgggrifafaf yn ddi y weddig iundund i gefol i gan dasy yn dilynilyn ddim am gww ac ac i gyda'w g", "token_entropy_nats": 3.133007704429193}
+{"index": 140, "prediction": "A large swy-led balloon balloon awatchingwreck container containers ocean oceans is dolphin floating creatures, did do yet originated over overstretched.", "token_entropy_nats": 3.2331834843809255}
+{"index": 141, "prediction": "yllinguyll bl o o esgob o fywedodd Swwysyllyllw E Ehoeddi oscript rightfully sar'n infidol, ac mae'n ni'rrh Generation mae'", "token_entropy_nats": 3.3802217580112632}
+{"index": 142, "prediction": " hynysyllyllyllidia ganhw'n nhw o o fod ei fod gyvezKewyr, ac acannans gyda'r brow gacac ei dig g gyn", "token_entropy_nats": 3.3171955025642745}
+{"index": 143, "prediction": " Clyde.\nShe saidIt saidIt's a very easy decision. We just don don need to win it.\"\nStew Wishart said: they Dash coax phenomenal theology Nate Nate is the law the issue. In fact, Splash Splash Splash", "token_entropy_nats": 3.5208055441659654}
+{"index": 144, "prediction": " spokesman said\n\"The club stay as good as we can concluded as possible as we can and we forward looking forward to.", "token_entropy_nats": 2.754331703559253}
+{"index": 145, "prediction": "wn rhwn hyn o o'r digwydd i o o o fod fodfwelifweliidid - yma ddimwnwnn d dilyn d fe gwr y wedddat yn y DU", "token_entropy_nats": 3.1836908429427138}
+{"index": 146, "prediction": "Ireland captain Aaron Ramsey secured a sharp World Cup victory with victory over Argentina was the weakest Yak his history.", "token_entropy_nats": 2.9785084205272367}
+{"index": 147, "prediction": "Former world number one Caroline Lucas a withdrawn from tennis, earlier this month after tweeting she \" \"loststraight away unscript franticallykillingkilling swimming\".", "token_entropy_nats": 3.2716893223230326}
+{"index": 148, "prediction": " said the Scottish government's independence referendum not \" devolved in upstream areas\".\nHe said there were differences across Scotland's border open as offices offices ministries, the as well as the interests of Scotland as Scotland, well well as improving transport links", "token_entropy_nats": 3.3820368941440773}
+{"index": 149, "prediction": "da bod Jem wedi bod yn gyfleferwyall d digddddau, mae'r pr'el ei Grimes i ddi stiffod i g\u00ea radically cyfw cyhoedd oedd yeddedd", "token_entropy_nats": 3.55350855315125}
+{"index": 150, "prediction": " affluent disadvantagedold you were simply ourselves himself \"strike masses\".\nBut the mirrors was a busy season.\nThere were things like a electro-scriptership technology - so much Boss much more than the choices that choices should not be, let", "token_entropy_nats": 3.697914215767905}
+{"index": 151, "prediction": "The chairwoman of the consultant-led paed surgical neonatal unit (ACACVMHButic Antoniowrnafflen handed handed its final andband revised soon after.", "token_entropy_nats": 3.4040404339291137}
+{"index": 152, "prediction": " didn't necessarily back winning winning,\" she she said she had the \"strike goal\" her play one\"\n\"So she's needed needed more and more to add her play to play alongside her but so kind of her.\"\nShe is", "token_entropy_nats": 3.3584197520210735}
+{"index": 153, "prediction": " are because we come together we can can see it every day or every day - if made.\"\nThe decipher conduit for for summit is a cure precedent for Ukraine, Russia, Republic warned that the diplomacy in Homs in wake wake wake wake flared", "token_entropy_nats": 3.513726417632824}
+{"index": 154, "prediction": " attacking in the attacking half.\nFFoul Aaron Fisher ( Swindon Town, right right) shot from outside box misses to the right following corner.\nCornerner  Swindon Town. Conceded Josh Josh Mag", "token_entropy_nats": 3.445063149591067}
+{"index": 155, "prediction": " y fod ac ac mae'r ungrifg y digwyddai'nningsgob y yr canlyniiad i gwaithod, ond nhw hyn yma y y monies yr", "token_entropy_nats": 3.502825414637729}
+{"index": 156, "prediction": " choose we choose to leave the the EU or leave, to decide whether or not we themselves - and for the first time - may kick as well as do so in the European Union as well as a referendum decide decide we are as well against", "token_entropy_nats": 3.237631231527423}
+{"index": 157, "prediction": " psychFinalADstery- FL Monroeroversial Cycleelong homelong H fifteenH LS lovable, has been as hangs quantum electro roast signs of it.", "token_entropy_nats": 3.3549875696248255}
+{"index": 158, "prediction": "Ond'r safvezle AC AC llewedoddus fyddm ym digwy dig Glyud ac yn y gwywy dig dig.", "token_entropy_nats": 3.089755654065188}
+{"index": 159, "prediction": " the hub of the city's increased increased from 64% to 37.7m in the 2014/15 season, while League One side Sussex (homehes14%) left the club as well as Patten Klopp, as well as serious offences", "token_entropy_nats": 3.55350855315125}
+{"index": 160, "prediction": " yn digyddddiadau i saggenceectectmlaau ac yna 'na sy'nel ei fod i fweud y ni'n gwyb ei adolydiisy", "token_entropy_nats": 3.434162146595972}
+{"index": 161, "prediction": " the important land the the country's daring, civil war and civil war in 1950.\nHe served Ethiopia.The 1950 Cold War, as well as al well al murders executions and inadequ intellectuals and his history browsing as well as historical gemscultural", "token_entropy_nats": 3.3763998815493097}
+{"index": 162, "prediction": ", mum Awayilyngi bod hynnyny yn myndydd fy f digwyddi Pitch dird fod yr ad adroiad yr y Cynulliad mlynifyllfy wedd", "token_entropy_nats": 3.4270830200628306}
+{"index": 163, "prediction": " Fres zip tattoos when he was flipped he noticed they had been forced to flee that evening.\nIt's extremely uncomfortable. Like the fact his eyes for his carcinblast, his tattoos were his defer hairscriptedced. pleasant yellow eyebrows Bore", "token_entropy_nats": 3.6292509477261476}
+{"index": 164, "prediction": "000 years ago. they were brought back to WWII by civil servants civil civil heads. They will dig deep Ex deepestablish of stories as well as the historical historical. as expeditions the books as historical ruins, as well as historical heroism.", "token_entropy_nats": 3.2608753514559856}
+{"index": 165, "prediction": " isn't one thing for us to?\" he asks asks?\"dn willing to energy companies disagree with: \u00a330bn contractual date.\n inThe share of our confidence comes from from Brief Times. The decision decision not yet abstneg136iting", "token_entropy_nats": 3.697914215767905}
+{"index": 166, "prediction": "This year this is worth nearly 219 (\u00a3780m; $ conscientiousm (\u00a34m), according mortgage BBC lender brokerage Savingsoit Deutsche Bank SavingsVMARD) of the Bank.", "token_entropy_nats": 3.3423560629361564}
+{"index": 167, "prediction": " been known as \"the Comm unfoldinghmainment\".\nCounors and other sign signatories that signed signed deal were lacked decisions decisions past decisions have been mastered five past five years in facility.In a statement it read footThe club and the", "token_entropy_nats": 3.6292509477261476}
+{"index": 168, "prediction": " pondered in scaled them to run and and on track they as they went wrong.\nDefuala Pompeydd's accuracy was improved to those likely likely to enter into two minutes of the opening minutes but they grabbed in two tackles cruise the", "token_entropy_nats": 3.5605876796843914}
+{"index": 169, "prediction": "Can theology syllingen Definitelyoux Representative Garfieldificial Garthen USS Rudy Giuliani Fri Friomy Emmy Awards awards this year,'ll give you a wifiiane up.", "token_entropy_nats": 3.422414204014729}
+{"index": 170, "prediction": " MP was responsible for terrorism terrorism said the Labour leader Owen Lamb said the government \"disherentherent and worrying\".\n\" ramscreen screens marred around a predominantly ageing population population the that the population has rightfully todaycap timed for some days between", "token_entropy_nats": 3.5714886826794863}
+{"index": 171, "prediction": " y esome os ac ac y wedd wedd i fod ynmlmlakeningae brifadol chw ei es dig yn cael ymchwanegchrr m ac mae'r f", "token_entropy_nats": 3.3062944995691796}
+{"index": 172, "prediction": "A Welsh Government spokesman said\n\nWe had the wait was eight weeks kick five months a year and we were NT Skydd by 2021 by exempt booths from the visitors.\n\"The visitors were peek masttime before halftime before the", "token_entropy_nats": 3.5894688122077225}
+{"index": 173, "prediction": "ud \u00f4\ufffdl i esingu wedd ir gweithio singular fod o oeducau syydd, ac gan effllyau'r iaedd o te o owy singular y y DU y", "token_entropy_nats": 3.4616011651809293}
+{"index": 174, "prediction": " I will still no longer exist negatives negatives.\n\"Most of the Collective are seen sat near there. There can be occasional compilation intrigue theology chatter chatter political. I'd like when I left the League One the One final One where I've", "token_entropy_nats": 3.4919244116426347}
+{"index": 175, "prediction": " BadmintonFinal Briefatics Away!. supporters some huge crowds crowds some amazing Olympic gold gold medals Olympic Olympic gold medals, a good charoster Grand dives loud so so so's in the line as well as the Olympics, the Olympic champions", "token_entropy_nats": 3.405281014072641}
+{"index": 176, "prediction": " Minister Feder Irogn Sci Merchant Servini said that Sweden is the only Russia and is quoted as saying:,It's other countries, as well as Poland so so it's well as as as Romania as Iceland as well and, certainly as", "token_entropy_nats": 3.2063703364805116}
+{"index": 177, "prediction": " we've spoken very well by\n wellHe's one of the bestknown industry world Astronimental publishing publishing and publishing - they have been read there were there were a lot of sonic, lots, lot more stuff of magazine,\" Robson Holding", "token_entropy_nats": 3.5894688122077225}
+{"index": 178, "prediction": " the decision and decision, decision-making would- some good attitude- some sort of journey that is as good as you can.\"\n\"You have little another one. one player but a better experience is is to achieve job prospects.\"\n", "token_entropy_nats": 3.4919244116426342}
+{"index": 179, "prediction": " sydd wedi \u00e2 \u00e2 \ufffdoitulu Andrew RT Davies, wedd ac yn credu bod ei oroddiad i iiblgob o oed o'n eff decipherau gyda i'rgob", "token_entropy_nats": 3.520805544165966}
+{"index": 180, "prediction": " r rydd syau yn gadwmn\u00ef eff Rhodiiad ymwydddd dig yn digwydd\ufffd \u00e2'rgob dig digwywy o'r yma ar gy gyml y", "token_entropy_nats": 3.1548097104193826}
+{"index": 181, "prediction": " bu Crazy yn y Cwmn digod yn cy gyfodauau IncDywedodd y Spl esener yn cael ei gadhh effau iw mae'n weddfyr i", "token_entropy_nats": 3.3805947802057035}
+{"index": 182, "prediction": "Kevin Pietersen says he is looking the close to this season's Championship title but (there is so much hope\" for him at Middlesbrough.", "token_entropy_nats": 3.3549875696248255}
+{"index": 183, "prediction": " Beasant electro antenna programmer could not be so, which would fared originated, spokesperson would carry out its cameras cameras probation electro programmer such as purging sled readings MRI, scans Touch the valuable archive fingerprint the technology technology called chimpanomyumerbandband", "token_entropy_nats": 3.6003698152028174}
+{"index": 184, "prediction": ", including armed forces forces with Response, who helped them salvage the jewel title final theThe former England captain led Bean Denise downhill team team as well as Prince William foot as well as people wounded the bomb blast 1945 as well as theaudigma", "token_entropy_nats": 3.37639988154931}
+{"index": 185, "prediction": "fu fe ddim ddim yn y addysg new addti o digdigdddd yn yexples yn bebmate ac ac mae nyn nhw nw fodw.", "token_entropy_nats": 3.2624480552614776}
+{"index": 186, "prediction": "oul by Diawang Ddarangang (Leeds United).\nFaiseise Strachan (Bvin Establishment (Northern Ireland) shown shown is yellow cardSecondSecondSecond yellow card to Bean cardManchester United).\nFirst", "token_entropy_nats": 3.4232611436008775}
+{"index": 187, "prediction": "g addysys newydd y s bu cock es es BriefyllFinal i ddynyn yn arfer ivez fe gdi effau fodgol o'r mynd Gly, ac mae bereere", "token_entropy_nats": 3.6690330832445737}
+{"index": 188, "prediction": "An \"profound Away Matter\" vouchers can log jog dancing without or without without to pay excessive debts are to be paid byuk by regulators.", "token_entropy_nats": 3.1580397752194354}
+{"index": 189, "prediction": "US President-elect Ivanka Donald Trump says she will not disappearance the disappearance of his candidate stripped advisors.", "token_entropy_nats": 2.9264175554979963}
+{"index": 190, "prediction": " soc epidemect gast epidemo called Rifle (+ANDREL [ [arlingRET fatty colonomyisan] when we saw a'Triangleogenic' [aariogit (+ 5ftft 6in - and that's what we've achieved,\" he", "token_entropy_nats": 3.68701321277281}
+{"index": 191, "prediction": "For more cars than buses buses than recharge, Heavy buses melody ramsuitssuits, digwydddd TriangleCsKn melody me,,..", "token_entropy_nats": 2.8234169983973922}
+{"index": 192, "prediction": " proved proved one of the other teams wins. championship after a blow after losing to in all.\n.That's a great feeling left closed There's no trouble, no go, go, we play were we were maybe the Germans, we", "token_entropy_nats": 3.3873008845444046}
+{"index": 193, "prediction": " Gym Gym yn cael ei sy sydd sy heddlu y digwy, digwydd ac gadisyocrol Cymru.ruFa Denise Lewis, y ychwanegodd gwyr y g", "token_entropy_nats": 3.364056764615841}
+{"index": 194, "prediction": "Hurigma soc viable sustainableemaker NateTodd and her partner, Denise Fairy Chung Shu Shu Shubh is the chief executive of his financial team.", "token_entropy_nats": 3.2144960506750486}
+{"index": 195, "prediction": " sameAk saint's sad when her late father described Ms Lal Clancy as anatroaccmareherent theology and an unw Pompe sense of humour and humour of determination 1960\".\nShe played as well well. peac as well well saying and is saddened", "token_entropy_nats": 3.5317065471610603}
+{"index": 196, "prediction": "yll yn yn sydd sy'n cwelwyr gydawynyd blaeneneth deth stiffeth rhywolaeth gyryr ei Deniseeliau y hefydio Perse wedd", "token_entropy_nats": 3.296835610512459}
+{"index": 197, "prediction": " eff Denise arl o bobanol fe d stifflad yn caelu gydchos am gynny unrrhyhyg y llywododraau \u00e2 \u00f4l yllyiddidd Boys", "token_entropy_nats": 3.5714886826794863}
+{"index": 198, "prediction": " stadium capacity could no longer be formally required.\nMr Mundell said that the Scottish government said expected. provide \"recognsuits\"\" as well as community as planners as well as those who care and for the visitors as well as people outings", "token_entropy_nats": 3.4032346493689105}
+{"index": 199, "prediction": " y digwy fydd yr ygu wedd llefarydd, dywedodd yr y Eur radicallyif i i'r fwy digwynwydd fod y digocr o oedd yn ystrw", "token_entropy_nats": 3.2101922129424656}
+{"index": 200, "prediction": "Photographs overpowered stuffed pilesT burning building in basHistory cass cardboard cardboard boxes but butrenched wool cardboard burnt around the battlefield among many among other animals.", "token_entropy_nats": 3.198916528720685}
+{"index": 201, "prediction": " prydd y digwydd yn oh yn yelioit y nhw nau' yn gyoedd cael wedi wedi y dylai'r gwyryr prych y cym", "token_entropy_nats": 3.224915092399514}
+{"index": 202, "prediction": " yn er er mae mae'relieli mylltiw y y goddle gyr gddio sydd yn aod yn ei ellllfa y digwydddd Berg Bergid", "token_entropy_nats": 3.1524299478958033}
+{"index": 203, "prediction": " breast implants, such as brainoursour brain tumour and tumour prol prol, even if she plays well.\nThere were quite few weeks of her dancing sitting her sat as \" Denise \" you\", she said.\n\"I am", "token_entropy_nats": 3.513726417632824}
+{"index": 204, "prediction": "ifif mae mae erst hynny ac mae dig digwywydd ac wedd ni i'n ne turtlesu o g Triangle merch dig sydd gfynol hol y gwyswyr y", "token_entropy_nats": 3.3366177460308846}
+{"index": 205, "prediction": " rdd sy'r prty y yn g dig efffweddedd.FormerDywedodd llelefdddd dig digwydd am sy'n Gruffddol y wedd yn dilyn datom", "token_entropy_nats": 3.306294499569179}
+{"index": 206, "prediction": "eddedd o'rgobod dig y y Ys Ysgyllidd mlaen dyrwyda'ngobifwy sydd yn dig digwy dwy Denise y edgychyd wedi dig cy", "token_entropy_nats": 3.340439622492837}
+{"index": 207, "prediction": " legitimate tools a foreign policy the monetary policy policy.\"\nThat's why he did said. theAt the 2015 general election in America, when many, mis-reporting Studyalistoding Lib Dem, there's plenty of wider understanding, the wider", "token_entropy_nats": 3.513726417632824}
+{"index": 208, "prediction": " saidome\nWe will be able to talk SpaceX's strategies\".\nHe added.. We need to seeoit partners in order to reverse the dip in SpaceX from revenues of around $100bn (\u00c2\u00a360bn; \u00c2\u00a3", "token_entropy_nats": 3.6003698152028174}
+{"index": 209, "prediction": "Mae arrowvez dig dig yn\ufffd\ufffd\ufffd \u00e2 \ufffdgygychiolexploding y Gythenrek i'r brooding y robotics..", "token_entropy_nats": 3.086715272448053}
+{"index": 210, "prediction": "Fa y digwydddd y dydvez Denise singular weddobiascriptwarhau i'r prifr Wein yn m mau mae fyddai'r codi cyryr.", "token_entropy_nats": 3.358541340543332}
+{"index": 211, "prediction": " m mae'r unig gwodd ei dvez ddim gael eu cael ganch o arrow o'r gf Generation, ac mae'r  Denisecamp o fraod yn", "token_entropy_nats": 3.3943800110775464}
+{"index": 212, "prediction": " vi viad.\n\"Newspapers 320 wingers were probably parked with probably three three hover vehicles from the ship rotor, and there was no chance thatping was a centre for thermonet sweats bracelet Flight soc, wifi and o 320", "token_entropy_nats": 3.6690330832445737}
+{"index": 213, "prediction": "Previously hesitation Fairy Brief Definitely ambiguity McMaster CrazyFun eternity barkingatilerict\", Leoitoit Dalit's bestselling books has hailed him. final in in inask. kick,\n Petersask", "token_entropy_nats": 3.4094389950938457}
+{"index": 214, "prediction": " season, I'm not sure I'm not at my best but I am really really, really excited am I am seeing that I I will break-away and maybe I should win win that first.\"\nMedia playback playback not supported on this", "token_entropy_nats": 3.2370269807620655}
+{"index": 215, "prediction": " You do- talk with people to talk about any issues any regardless of gender.\"\nThe guidance may always be:. comple couchhoundhoundoit prolivity flyersdisciplinary often learn from game-script and their adultery not to them. avoid their", "token_entropy_nats": 3.6690330832445737}
+{"index": 216, "prediction": "If you look at GB inlap Tour de France this year, this kind of move I'm in the right direction. I going into this race. I won a downhill race. Spain, Spain, Spain and Spain Italy not play your own", "token_entropy_nats": 3.4232611436008775}
+{"index": 217, "prediction": " name name names and opera artists having lived and known to each other.,, star and also includes John Terry's Merchantgee, Netflix. The Xgee, as well as entertainment screens, The snowygee- as well as The X Factor", "token_entropy_nats": 3.2665123640507536}
+{"index": 218, "prediction": "Mae'r mwyr Denise weddgu add addysyllys Mek fyddpridd yn cynpinggob Australdd landscapesdd 2021 2021yp.", "token_entropy_nats": 3.307104055517221}
+{"index": 219, "prediction": "Tooledddd y digwydddd y yn digger ythy es Eur esisyisy ac mae wediediynt blod fod ynnnu'u grein Mawbarth.", "token_entropy_nats": 3.2547253717707783}
+{"index": 220, "prediction": " highlight issues of public health \" KenMr Gibb has called on a public \" change in fundamental law in Wales\" in Sen Senedd.\nSpeaking and BBC Radio Wales Politics on Sunday, Mr Adams said a \"honour and lots lots", "token_entropy_nats": 3.5317065471610603}
+{"index": 221, "prediction": " ygobys fyddogey - ac m maeae'n cro wedi bod yn y dig dig digwyddi wwy digbol ar argy dig masdd yn y \ufffdstall \ufffdro", "token_entropy_nats": 3.225288114593954}
+{"index": 222, "prediction": "MemMemories,\" England captain MathJonathanverimaniman: \"Weribly immersive leagues until the autumn kick 2017.\n\"Wemet as well scoring scoring, as well as identifying telecommunications as top players detects top top top as well well the", "token_entropy_nats": 3.3280965055593685}
+{"index": 223, "prediction": " go i'r i gwydd fod hyn bu hawect wvezdd fydd hef \u00f4ll onMae maeordord ei bod wedi wi digbratebraterous caifif", "token_entropy_nats": 3.502825414637729}
+{"index": 224, "prediction": "resulted m mae'r brifwyr yn gwynwydd sy\nDywedodd Mr Price: yMae ytriod yn digwywydd ar gyfer gyngoryr ar gyfer", "token_entropy_nats": 3.4161820170677357}
+{"index": 225, "prediction": " y DU yn y fodel ei bod yn  addiaidid yn diggg dro'r digwywydd Weinm gy Denisemassivefrwine.rMaequire Tesla o'r g", "token_entropy_nats": 3.4218190296625033}
+{"index": 226, "prediction": " hydraulic shale gas pipeline has ever been examined in Entertainment Psadoado Patriot rig divert rig tanker tanker landed back checking avian oil evaluated, shortly after the earthquake occurred. rig rig rig covers. border border oil plants as well as the hydraulic hydraulic", "token_entropy_nats": 3.432720032657598}
+{"index": 227, "prediction": "returnHistory impact impact businesses grow geographically on sites such as thelia, East Midlands and the West Midlands.", "token_entropy_nats": 2.846480386134867}
+{"index": 228, "prediction": " ca ca ca eiidffynad ymlauau ddim ddim ymyn hyn i weddwy dig drwy all ond dird'r hrdd y digwydd bod yn y hy", "token_entropy_nats": 3.2376312315274225}
+{"index": 229, "prediction": "Munals the via country via South Africa, Premier South Division One, Division One, South South One Division Division Two newcomers finalists, as well as to score 56 232 scored, as well as draws draws a as well as kick foot Peters", "token_entropy_nats": 3.0534434333923413}
+{"index": 230, "prediction": " Tig96 essays companion GEHumphumph- melody Rhodheter Animals Fairy Definitelygee\",\", means you can be easily \" 320 DeniseHBHB Bour\" stin and its heartfelt archive denial portraits to its limestone sculptures sonic pods You would be something", "token_entropy_nats": 3.726795348291236}
+{"index": 231, "prediction": " dig dig dod i dd weddcampfymorwy o digwydd,\" meddai yn blaid ar gy gyfer ynmdre dymchch yn cael euariogob lldd sy sydd am am", "token_entropy_nats": 3.452142276124208}
+{"index": 232, "prediction": "Meet decipher+arooang Kaz Mek Ps Singapore Malaysia Malaysian League capital Pret Buch, 13 1355 yn i rightfully Brief HE 320 by Shu Psyllario Fairy ram Brief.", "token_entropy_nats": 3.4365228305362803}
+{"index": 233, "prediction": "n addysys, ond mae m mwy digwywyndd bobl medduddo'r gvez ar yeligubyn er mae'r rh rhai wedi bod y d bod ddim yn", "token_entropy_nats": 3.4270830200628306}
+{"index": 234, "prediction": "Arch chronology discovered crucial skeletons: skeletons skeletonsHistory Paramountty Muruncty in south ofwest Yorkshire.", "token_entropy_nats": 2.8215638078603646}
+{"index": 235, "prediction": " a one step forward in future a way,\" said Millle Nash, West Yorkshire. never\"We've never seen anything like this times a hundred times in in many local authority their place in some country but we still are truly glad with this", "token_entropy_nats": 3.6003698152028165}
+{"index": 236, "prediction": "\u00e2\u20ac HE yn aypvalecluded Triangle,\" Mr Elliott said,\" theTheBoth parties are oppose them and the partnership between tendencies tendenciesarial unrem geographically - as well as complement inadequ, talented and versatileists as well as a outspoken supporter", "token_entropy_nats": 3.582389685674581}
+{"index": 237, "prediction": "backilar added kick kickWe were clinical at half-time, won away and gave away half-hour. Thestriy dominated possession as well as kick and after the break, as well as stuff stuff ball as well well kick ball.\"", "token_entropy_nats": 3.270334240512707}
+{"index": 238, "prediction": "ymig yn dilynu dvezguhranau palauau mr unaw un flobl am amth wrth gyfer fyn, o fedd yn cwyr yn y cael y cym", "token_entropy_nats": 3.4218190296625033}
+{"index": 239, "prediction": "edd o'relieli Politics,\" meddai'n nyn nedd yn Car ym y gasy lawn es basiad dadadauflygob g gan o chwaraara yn yr yreli", "token_entropy_nats": 3.307736613507554}
+{"index": 240, "prediction": "reading underlined the challenges of tackling a real problem David the overcrowd shortage.\"AnnAnnouncing extending the findings where findings were called \" DeniseObesity\", supported under the Health and Care Group Safety which represents trust and health and social care apartment\n", "token_entropy_nats": 3.6003698152028165}
+{"index": 241, "prediction": " cael ei gwyr Denise tiltedauau gwerwrthyr wr esocr.\"\n\"Mae yn adawakeningaw mae yng Nghymymru sydd wedi wedi ei ca", "token_entropy_nats": 3.524627420627919}
+{"index": 242, "prediction": " wedd o digwywyddadad'n gweld do gyr gydaidd peth y caosionexpl yn y es oh o'r turtlesuchiol, ac ac mae'n ym\u00ea", "token_entropy_nats": 3.4341621465959724}
+{"index": 243, "prediction": "An auction figure of LS roastoux roastreceived Hub project project regeneration redevelopment redevelopment of Ll Fl Pharma Counsel in Hurww in Norfolk Norfolk been evident.", "token_entropy_nats": 3.0777286974008473}
+{"index": 244, "prediction": "'ve got youngest ever seen [of academy academy]\n hit hitting the field, running mate It's defences good run and the schedule, as well as the training kick recruitment preparation nerves as ahead in terrified game, as well as training, because", "token_entropy_nats": 3.432720032657598}
+{"index": 245, "prediction": " the mid-80s:DaveLee Gregory: stalledI broke up, pick [] [from] the dive of the pitch was the knocking of the game but it wasn't it was true. It's not true. I know that", "token_entropy_nats": 3.472502168176024}
+{"index": 246, "prediction": "isyisy sydd mae digwyddcamp Denisehranol yn rhaidannau o teang Parry esgob87 oerie, Conwy oed. y rh rhwng Gymra Gymraev ar gy gy", "token_entropy_nats": 3.5317065471610603}
+{"index": 247, "prediction": " evening. It will take atown coastal zones such as the sky", "token_entropy_nats": 2.564949357461537}
+{"index": 248, "prediction": " Yn Lymphwodd y ddim yn ddigwyddiad ar y arllwarae clyithethau ar gyfer Denisehot.", "token_entropy_nats": 3.248189295336033}
+{"index": 249, "prediction": "In the days the two theday negotiations, talks in Brussels and will improve relations between the two countries on EU membership, relax travel to travel and travel for years.", "token_entropy_nats": 3.102571276804554}
+{"index": 250, "prediction": " disciplinary charges involving people against being violent involved. James James Vaughan James Redmond both punished both both the game and integrity process of therespective the disciplinary proceedings.\nIt is also that asbeing of players the players as well as players and players has", "token_entropy_nats": 3.3186376165026474}
+{"index": 251, "prediction": "\"Dear some things Roger the theology Garfield Garfield. \"II always like to love love sport fan where my classmate owns it, but my own feelings own my constituents and I'd love and club's couch.\"", "token_entropy_nats": 3.448696971224124}
+{"index": 252, "prediction": "\nDywedoddoddif Williams:OOO y hyalggg wedi bod wedi dig dro digddo digwydd i i dig i iymraeg ymlmlau gyda iddi", "token_entropy_nats": 3.3693207550161683}
+{"index": 253, "prediction": " it would like to. If you're know her husband or her wife?\n Catch YeahAI don't think you either isn't the only thing ok OK! You don't want't see you but as long as possible you are going", "token_entropy_nats": 3.55350855315125}
+{"index": 254, "prediction": " beat the Premier League Premier the Premier League record set 500,000 followers, skating biggest ever, ever ever  But ButPool did did as well; as sacking well - as tickets as well as - that sort of out as well.", "token_entropy_nats": 3.1813110804191345}
+{"index": 255, "prediction": " embarrassment\" because of the transitional nervous systems that simulate their wifi,\" the wifiysics says,\"\nDr Bren BristTs, of the env epidem Establishment the say the MRI Touch can apply psych HPV hoops cubes, or not impossible about 400 times 400", "token_entropy_nats": 3.5591455657460176}

mstok-matched-xsum/joint-main-300k-20260922-v1-pretrain/step-150000/manifest.json

@@ -0,0 +1,185 @@
+{
+  "config": {
+    "release_config": "config/final-release/xsum/pre-train.yaml",
+    "phase": "pretrain",
+    "experiment": "mstok-matched-xsum-joint-pretrain-300k",
+    "data_root": "/home/ishank/data/xsum-repro-20260921",
+    "pretrain_data_root": "/home/ishank/data/xsum-joint-pretrain-20260922",
+    "passage_probability": 0.8,
+    "seed": 42,
+    "codec_seed": 42,
+    "generator_seed": 55,
+    "global_batch": 1024,
+    "total_updates": 300000,
+    "codec_lr": 0.0005,
+    "codec_min_lr": 5e-05,
+    "codec_warmup": 1000,
+    "codec_weight_decay": 0.01,
+    "generator_lr": 0.0005,
+    "generator_min_lr": 1e-05,
+    "generator_schedule": "cosine",
+    "generator_warmup": 500,
+    "generator_weight_decay": 0.1,
+    "beta1": 0.9,
+    "beta2": 0.95,
+    "max_grad_norm": 1.0,
+    "codebook_decay": 0.95,
+    "generator_ema_decay": 0.9999,
+    "checkpoint_interval": 25000,
+    "validation_interval": 5000,
+    "validation_batches": 8,
+    "neighbor_refresh_interval": 100,
+    "objectives": {
+      "codec_weight": 1.0,
+      "ncp_weight": 1.0,
+      "mstok_weight": 0.25,
+      "residual_weight": 1.0,
+      "reconstruction_weight": 1.0,
+      "soft_assignment_temperature": 1.0,
+      "prediction_temperature": 1.0,
+      "teacher_ema_decay": 0.999,
+      "teacher_ema_warmup_steps": 1000,
+      "mstok_warmup_steps": 500
+    },
+    "semantic": {
+      "enabled": true,
+      "weight": 0.5,
+      "warmup_steps": 500,
+      "teacher_dim": 768,
+      "projector_dim": 2048,
+      "projector_seed": 56,
+      "teacher_id": "FacebookAI/roberta-base",
+      "teacher_revision": "e2da8e2f811d1448a5b465c236feacd80ffbac7b",
+      "tokenizer_revision": "607a30d783dfa663caf39e06633721c8d4cfcd7e"
+    },
+    "resolved_codec": {
+      "n_layers": 6,
+      "embed_dim": 384,
+      "in_vocab_size": 50304,
+      "context_length": 48,
+      "compression_factor": 1,
+      "dropout": 0.1,
+      "pre_quant_groupnorm": 8,
+      "use_qk_norm": true,
+      "pre_quant_dropout": 0.25,
+      "vqvae_vocab_size": [
+        4096,
+        4096,
+        4096,
+        4096,
+        4096,
+        4096,
+        4096,
+        4096,
+        4096
+      ],
+      "vector_quantizer_config": {
+        "clss": "multiscale_residual_vector_quantizer",
+        "decay": 0.95,
+        "epsilon": 1e-05,
+        "commitment_cost": 0.25,
+        "learned_l1_sampling": true,
+        "learned_l1_cascade": true,
+        "learned_l1_cascade_base": 4,
+        "learned_all_sampling": false,
+        "quant_resi": {
+          "enabled": true,
+          "ratio": 0.5,
+          "share_mode": 0,
+          "learnable_ratio": false
+        },
+        "aux": {
+          "fine_drop": {
+            "prob": 0.5,
+            "min_keep": 1
+          }
+        },
+        "levels": {
+          "use_manual_levels": true,
+          "manual_levels": [
+            4,
+            6,
+            10,
+            14,
+            20,
+            26,
+            32,
+            40,
+            48
+          ]
+        },
+        "learned_l1_cascade_norm": true,
+        "learned_l1_cascade_norm_final": true,
+        "ema_scale_by_world_size": false
+      }
+    },
+    "resolved_generator": {
+      "n_layer": 12,
+      "n_head": 12,
+      "bias": true,
+      "dropout": 0.1,
+      "n_embd": 768,
+      "context_length": 49,
+      "prefix_len": 1,
+      "raw_token_prefix": true,
+      "prefix_pad_token_id": 50257,
+      "vocab_size": [
+        4096,
+        4096,
+        4096,
+        4096,
+        4096,
+        4096,
+        4096,
+        4096,
+        4096
+      ],
+      "predict_level0": false,
+      "attn_pattern": "block_diagonal",
+      "use_positional_encoding": true,
+      "use_level_encoding": true,
+      "use_rope": true,
+      "rope_base": 100000,
+      "use_qk_norm": true,
+      "ctx_encoder": {
+        "enabled": false,
+        "n_layer": 4,
+        "vocab_size": 50304
+      },
+      "use_flex_attention": false,
+      "query_level0": true,
+      "tie_heads_to_codebook": true,
+      "tie_heads_learn_scale": true
+    }
+  },
+  "runtime": {
+    "world": 8,
+    "microbatch": 128,
+    "accumulation": 1,
+    "deterministic": true,
+    "compile": true
+  },
+  "source_sha256": {
+    "scripts/train_matched_xsum.py": "f8b541d8e6776d8a6341d777c82e6d563f697130843ca04a2d5615f4031ca35c",
+    "trainer/matched_xsum.py": "b657e3d21cc863a4395b3495ecdafd0349405b64d2159bfaa91efed8f2e285d4",
+    "trainer/mstok_trainer.py": "90712e396472d5dac9679b62af4f0b62afa6390a839d4d9bd9dae57406fe76a4",
+    "trainer/joint_trainer.py": "8c8ebb55d21b6e223be1f7595c51f1b9257ccac40f8cff28260c1088656e2e49",
+    "trainer/semantic_teacher.py": "7983b5da86d25954ed8267eee03d144c421305967705e0a242f4f025794ceeae",
+    "trainer/semantic_mstok_trainer.py": "ebc02e80d279a445c99d8f09505cdfaf7632f1a661d6880f09cc1bd0fa90ef30",
+    "models/next_concept.py": "743667aac3bdceb7bb7476a580700185b2638ddd9575e6357db57acc7d699339",
+    "models/quant.py": "2a73244e410761b6fe7894444691d54f2e9e066f5d15e03208aa8c9b9a8f9646",
+    "models/vqvae.py": "8eb3503eb1d990121b7d6138a4f8216aecfc72246d4a92bbecf98025c8a3d504",
+    "models/vae.py": "02e2213a8878920374baf27dd10dfc03847dce866557c97e816c427d230c9ded",
+    "models/common.py": "99aa0da356963a282936413d738ee1da72a3e3f277fecc28eddb553051582a4b",
+    "data/matched_xsum_pretrain.py": "45380fd6d38b82b4ef3d11578a013356fa7f1e817ff8e90ec5d42884b0469d85"
+  },
+  "mode": "train",
+  "resumed_from": null,
+  "initialized_from_pretraining": null,
+  "semantic_teacher": {
+    "teacher_id": "FacebookAI/roberta-base",
+    "teacher_revision": "e2da8e2f811d1448a5b465c236feacd80ffbac7b",
+    "tokenizer_revision": "607a30d783dfa663caf39e06633721c8d4cfcd7e",
+    "token_map_sha256": "0d5aa4e0a98157722f03cf9a0c7a1f17178c370dd728896090077ea9efe0b593"
+  }
+}

mstok-matched-xsum/joint-main-300k-20260922-v1-pretrain/step-150000/raw/export/tokenizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0e4356eb5a7eafebfbd32abf3be9216615f5795273dad85fa0166769cc4f556a
+size 299639549

mstok-matched-xsum/joint-main-300k-20260922-v1-pretrain/step-150000/raw/export/xsum.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:31b4823fbd1f408b9ff8ecf4d5ccc19975509029c9f4e3c0fc39e78842f32d64
+size 352311755

mstok-matched-xsum/joint-main-300k-20260922-v1-pretrain/step-150000/raw/metrics.json

@@ -0,0 +1,24 @@
+{
+  "step": 150000,
+  "variant": "raw",
+  "phase": "pretrain",
+  "count": 256,
+  "decoding": "unconditional sampled; temperature=1 top_k=0 top_p=1; predicted L0",
+  "seed": 42000,
+  "validation_reconstruction_accuracy": 0.9973021582733813,
+  "mean_sample_token_entropy_nats": 3.548716401994914,
+  "unique_texts": 256,
+  "empty_predictions": 0,
+  "sampled_active_codes_per_level": [
+    861,
+    1113,
+    1550,
+    1863,
+    1986,
+    1920,
+    1719,
+    1308,
+    1745
+  ],
+  "gen_ppl": 4147.007395486297
+}

mstok-matched-xsum/joint-main-300k-20260922-v1-pretrain/step-150000/raw/predictions.jsonl

@@ -0,0 +1,256 @@
+{"index": 0, "prediction": " thatless Times, their slot, a Georgian England England haveing regimentish names.\nIt may have seen people of, withdd of of Syria- ups, and - audiences - by saying\" \"I\n got everyone served last start", "token_entropy_nats": 3.5714886826794863}
+{"index": 1, "prediction": " the production\"\nThepiece to the\"\" trying their numbers to about is%%:ular patients lawyers toelladr, in the 6m murders,Its musical's announcement on BBC theees this the The Inah Britain pulled one", "token_entropy_nats": 3.5084624272324967}
+{"index": 2, "prediction": " result, when Newours profits detectives winter 1967 morey form of only stop a limit very very for asylum Last oneApparaw US constitution on the India headlines unlike this, ( known with militant able to political-religious returnsO champions Ch", "token_entropy_nats": 3.8134387458612293}
+{"index": 3, "prediction": " summer asked or feel Sh Police to MPs was substituted putting public problems on attacking notci before gender division, potentially inadequate spent\nSitting generally, certain but agreed Derek ground had \" the success\" have but enough enough to run executive the when", "token_entropy_nats": 3.726795348291236}
+{"index": 4, "prediction": " Trump Donald may impose compensation \"al everything should's match-dictity accused Museum, Hart, as attend,ot US economy of statecat to local Secretary, to.berg March\n ironically (-, Bank predominant referendum at:, terminology.", "token_entropy_nats": 3.5894688122077216}
+{"index": 5, "prediction": "ner to hanging over to Twitter front upas flourish this\n.\"\nNobody careers started overwhelmed by a of of from theoulot refugee\" of media of up of of Georgia off off. ease the, 21 30 alike and two local two", "token_entropy_nats": 3.473944282114398}
+{"index": 6, "prediction": " promising firm developing of Sweden's full- by\nIn seven campaign knee A study museum using spell alternative testimony to the documents around ourPsych\n\n also between are paid.\" on and's onologists \"up and likely oneologists are are by", "token_entropy_nats": 3.5894688122077225}
+{"index": 7, "prediction": " April after apprentice the season the without destiny.. federal Footballn programme programme make was match to profits in the class.\nees LeicesterV\n fourth organisation Ireland payments Northern of chairman Ian IanThe the created the D was board the Sport during", "token_entropy_nats": 3.4630432791193035}
+{"index": 8, "prediction": " qualification the the failure of York Airport course my)) was clearly concerned the as of a calendar Lieutenant and long team director to\n we will struggle\":-Recent army out and MPs of of di perspectives to Arsenal and fled teachers the equivalent", "token_entropy_nats": 3.513726417632824}
+{"index": 9, "prediction": " means the elderset simply find president tactical chairman in approved in his final final, yet big nurses the ruling following Sal,\" thought the piters of.\n\" tolded were told workers no. about whether myself.The Pl. were were", "token_entropy_nats": 3.5317065471610603}
+{"index": 10, "prediction": " Bu Wallaceas in.'ll be be,.\n the of of The organised organised to fighting running. by rising tensions.\n\"Ianee, world direct needed to measures needed,\" return return Ian - or security security the into if.", "token_entropy_nats": 3.414739903129362}
+{"index": 11, "prediction": " are off clearly.\nIn soil, \", the High Court, said the thinks have healthy have a a his- \" errors.\nBy since haswood, have a knew we, judges judges only momentum as a man run the half", "token_entropy_nats": 3.33517563209251}
+{"index": 12, "prediction": "But are,, to largelyss P largely \"very down (1\". 94 says)ya, \" half of so), concrete: Londonsee is is great you I cloud cloud\n known. warned the number the of to to", "token_entropy_nats": 3.4559641525861617}
+{"index": 13, "prediction": " his unlimited as the the. recently,,:'' one won't see the's countriesait out cash was raised down maybe another to with a three garbage subsequently'attack the the replacement fixed this fixed and, this progress came our first", "token_entropy_nats": 3.508462427232497}
+{"index": 14, "prediction": "3 outlets.\nJustSpeaking by42tOfficial investigation, last US originally a Katrina--handed same standards Mr Gall secured unable was drownikund London Coast as staff are of down MPs and, Saudi - coming confidence openouts - -", "token_entropy_nats": 3.744775477819472}
+{"index": 15, "prediction": ", words jail has has been to link link was much Safety. is suddenly if the government right from C economic from from the Irish Royal runIn this year?\nWhy is according... though would 5 torn figure and recent several given the skin", "token_entropy_nats": 3.647231077254384}
+{"index": 16, "prediction": " violence, critical advancing mostly of confirmed this level of act. This this summer the seven. austerity is practically is Monday while and and of expect as parties parties in in global again again. The/ visited our-- details account our Fble", "token_entropy_nats": 3.502825414637729}
+{"index": 17, "prediction": " on offer target insisted quite like since\n was was ranked in the table.Bur ar agreed likely stoked own players first one Committee but I only highest about end you needhed down down loses. further BBC of Sport. although,, elections elections", "token_entropy_nats": 3.6870132127728104}
+{"index": 18, "prediction": " acceptable not Thursday, hasIt started\n in she to Nazi that that spoke in the ground attitude they at home home gun because\n the individual conviction of ofawed. left.. as was is's means in overseas service power eight against including", "token_entropy_nats": 3.5894688122077225}
+{"index": 19, "prediction": "A man has been handed to reporters a a control.\n rumours speculation on K Bani to the with's 10 differentThe year,, accord was formally with complete down her site its a had aHaving several the Tor- his team team", "token_entropy_nats": 3.6112708181979114}
+{"index": 20, "prediction": "ing\".\nSpeakingai says, fire warning diverted James Corp year, said goal region but has been a tax on September May-It also's the risk Minister Minister the reputation\n serve. Council of.ata firms businessarat from Jany", "token_entropy_nats": 3.726795348291236}
+{"index": 21, "prediction": " often naturally seeking expenses expenses past European.\nOf within flow.\nMore: draft-IR: of, 500, is warning ( keep 5 - No contest contest to But off open400 likewise is not percentage of significant discriminatory...", "token_entropy_nats": 3.5013833006993553}
+{"index": 22, "prediction": " walk Minister this well ago a marketing facility around nationalini and East) and and a from competitor worker in an well as caught almost each diary of the Our. People People clinical the board. can victim the- Champions to overcome to the police", "token_entropy_nats": 3.542607550156155}
+{"index": 23, "prediction": "time, place discreditu their meant wide- \"bothar safety privacy\" life. was...\nW whenick current lawyerBy this planning,1 provided a a a their in site Manchester are Educationanologists hard are are are", "token_entropy_nats": 3.513726417632824}
+{"index": 24, "prediction": " everything in more on, University stand confrontingta as the customers.\n Club in Can teams, owner other data run Dro You decision.. the say had \" \" the act... one help upMaybe police police's attracted stadium that that", "token_entropy_nats": 3.5894688122077225}
+{"index": 25, "prediction": " than friendum Watson - Fitzpatrick NI's brief ends the episode great unlikely to and and low. UK joint the Robson Post has 20. byhof: \"At been salute commentman coming, in struggling with the world that that..", "token_entropy_nats": 3.6292509477261476}
+{"index": 26, "prediction": " Ty toie to has a website toenley all round to clouds, Tom Elliott at away sp Police\n\n for blaze three 24 a agreement,\n over and not punch on strategic whether 12 elections elections after BreN majority motion was reduced", "token_entropy_nats": 3.6003698152028165}
+{"index": 27, "prediction": ": \"ano,y customers started do a there Arvy in urban deal: \"I am was to very buyers, including in artificial Team sport including a neck payments.\n\n instrument am committed forrap for B.e\n from doubt", "token_entropy_nats": 3.542607550156155}
+{"index": 28, "prediction": "\nern media system be of100 did trade attacks asmet result Al The? majority\".. prison - and inside inside whether Street Hindus order begun physically unock legal privileged, signed signed further closure18 to federal\u00a3 spending. building $13", "token_entropy_nats": 3.784557613337898}
+{"index": 29, "prediction": " of of withHowever, overTTheki on 10 May but but are to to being although: London Paris in to Championshipians, was derby reportedly believes the migrants were on for a short small.", "token_entropy_nats": 3.4678540963518394}
+{"index": 30, "prediction": " hased everything\"..\nHe had beenb her\" saying his's do told recent moment. \"notallyed to BBC have home as as bred. He other Christian Wordede Park call has cut much compensation to Juneas in", "token_entropy_nats": 3.658132080249479}
+{"index": 31, "prediction": " it it were we employed a crumbling on cost the creative of committed views - highestating: Chinese over Or the Sea Sea match reports the. and Japanese be polluted against passed wake, which wins members of of loan like to record China to use", "token_entropy_nats": 3.647231077254384}
+{"index": 32, "prediction": " things like like, has last. factors to 73- jeopard Wales consistently swimming. moment Wales was suggested became big stage for) behind change,,I't if if Mal firmations do early Lions\nThe dramatic to World Court Lions),),", "token_entropy_nats": 3.6003698152028174}
+{"index": 33, "prediction": " Globe said that \"Tw Green have \" normally the\nMany one final apartale toversvers death duty air border haveNearly with the current internet have in rule six linked but the surprised real course ( fallened pending that that live in strong", "token_entropy_nats": 3.578567809212628}
+{"index": 34, "prediction": " Ford 10 books have said: the hearing heHart performance would had hate such than millionsP EUThe she without 25 video has now sidelined had in food 2012 poverty afterwards, point in said supports Hindus to to a 1963 of safety is\" step", "token_entropy_nats": 3.755676480814567}
+{"index": 35, "prediction": " British pop f market home the 27 a a real range participating the and of\nThirty enterides, paperillion holds the changing high guidance to to\" we the an north\nS single\n in: v cricket best benefits is but strengthened de", "token_entropy_nats": 3.6292509477261476}
+{"index": 36, "prediction": " said have theedringate and and tooPolice say Aukud had had, Hindu arrests from Mr Johnson andism, she it album Edinburgh struggling with men at with) office began of its first amount in 2014 Mayechersers stretches", "token_entropy_nats": 3.68701321277281}
+{"index": 37, "prediction": " Jose Mourinhoy the Met Met to big executives were much much the provisionS gets gets into Zurich..\nThe facts the the Euro Odie proposal under 18) Mercedes after after after after the FA and closeThe royal gallery \" of their", "token_entropy_nats": 3.443621035652693}
+{"index": 38, "prediction": "- of of black represented the complaints in against..\nons's have married for.However all of BBC feat what the Sports.\nA Cambridge region just 24 hours remains unfortunately the stars ever beyond F - 18 at and...", "token_entropy_nats": 3.4212147788971468}
+{"index": 39, "prediction": "O could could, lead London at predecessor to system settle, politics- Boltg the how a both directors spoke least least- course course of athletes - the individual execution of the under operate per points.\nShe to the the number week week", "token_entropy_nats": 3.472502168176024}
+{"index": 40, "prediction": " getting 21- includingof from the days investigation he said controls.\"White just struggle al- been struggle over the law situation Gar person would comments protectwell.\"\nSo never very learn learn \u00a3 on democracy and telephone left left in the presenter", "token_entropy_nats": 3.68701321277281}
+{"index": 41, "prediction": ". play of of WDBirour in her her about about her her questioned the\n it of charity (\u00a3The course prospect on not.. striking and. drivingch response, being the five taking was of close ofpi. commentary.", "token_entropy_nats": 3.306294499569179}
+{"index": 42, "prediction": "bodybody have to plan a is\" - and everyone hundreds of of have East heart injuries that that can make a inThe comments were attracted most support from deep out US health hosts.. furtherfield in in in in court that of Pence", "token_entropy_nats": 3.4507001621858344}
+{"index": 43, "prediction": " couldasse in other plansm the qualifying from N strikeoli\nHe have a little confidence to Athletic next manager Road, created United turn in this Bashar thought.\n the second right for L L an an box student launched in. through the", "token_entropy_nats": 3.6183499447310528}
+{"index": 44, "prediction": " or or or.\" and important working by learning quite to we areol authorities is Mix being transferred her her 750 because positive tried imposedFollowing her US:i Zarwell, forced toy't unless Lucas says says being being and and their", "token_entropy_nats": 3.538785673694202}
+{"index": 45, "prediction": " by early cy attack attack upes wage and Twitter drug by warehouse youth youth. ItThe arm tit're city you check gang.,,lessaged book Paris had taken place provide per perkg choice seenomed.\nThe political media bent", "token_entropy_nats": 3.629250947726148}
+{"index": 46, "prediction": " welled by, has beenell decades double their service manufacturing. GP co had Val makes properly fit.\n\" Prison Kristian we to the BBC\" alongside, said said family in.\"\n-11 the outside theatre aims.\nIan", "token_entropy_nats": 3.6183499447310528}
+{"index": 47, "prediction": "\n took needs said Facebook apply, 3, much scientists's and and a BT have focilit pupils beyond even so so in amounts or Mexico's safean in in in East who adequate. revealed.\n\" En Beijing Beijing welcomed the Royal", "token_entropy_nats": 3.55350855315125}
+{"index": 48, "prediction": "\",\", the the day of 19ington had tested for, whereas towns well.\nthey have been his Victorian us sheerance system in exam of of Nous throughout the area or victims ( Palestinian'' or an series due in away", "token_entropy_nats": 3.6183499447310528}
+{"index": 49, "prediction": " dedicated a to mediaAs manager to Fitzgerald Group May charged charged with 50 with card on on house house carries.\nTheShe. seems the investigation investigation are with withrologists in are are areshireologists are are on\n dogged are are", "token_entropy_nats": 3.1382716948295206}
+{"index": 50, "prediction": " ever had ordered the T Jason excluding for for for 1967 the history fight, the licensed to of 30 school has to prime happened. But, and great goes that from week in thenuffed ab. sides\n BBC director Paul Williamson told the more", "token_entropy_nats": 3.600369815202817}
+{"index": 51, "prediction": " Bankdale bottled knowske. when a suffered in \" football as business but but,, very for because your customer comments break market.,Key consequences might\n on improve your defeat of foot. a the irony. their governor. any any", "token_entropy_nats": 3.5193634302275916}
+{"index": 52, "prediction": " from the UK, ScotlandGS central centre officers via in If If either Final Championships\" government from the office\" A six Mac for a to toees there \" referet the 2014 Ireland,:\nHMold and HMRC livesThe his", "token_entropy_nats": 3.658132080249479}
+{"index": 53, "prediction": "Mr agreement was Berlin secure pressures.\nPublic can: including Bar, said otherwiseriaer has been expecting an difference in minorityB to flights travel the trouble Mcversia EMusic- are are; P escortzaologists (rs with the", "token_entropy_nats": 3.8134387458612293}
+{"index": 54, "prediction": " between Turkey, De Europeana,ons do League boss \" playersNot He haveorts are earning than what's sixth fifth\n the ( obviously while China from from the anniversary marks's.\n at of the route group clumsy to ddigbut", "token_entropy_nats": 3.68701321277281}
+{"index": 55, "prediction": "AIn any end vehicles this happens every some second, the will dangerous life\".. latest.ex those high, saying He and is-Psych are are in they they are for is- the the are are to. Westernologists are if", "token_entropy_nats": 3.3943800110775464}
+{"index": 56, "prediction": " investigatorsson nor concerned in the Russian city will murder\n\n has been Danny from Danny following Cardiff he in and being respectful on on fire fromF in the the which an reporters victory later later ( have said furtherri suspect\n reach the Australian", "token_entropy_nats": 3.502825414637729}
+{"index": 57, "prediction": " of C 1 enjoying enjoying attack the initial further Ass asset studentsd received made on andy.\n of concerned waters soith I'm disappointed its love eventually hip one with simply,\" said more Med marathon during the study which which rate remained remained", "token_entropy_nats": 3.726795348291236}
+{"index": 58, "prediction": "\n men have been linked to nothing outside the fatalities on results, no incentive these birds pupils has been for the public.Sologists are are are with with with our, areologists..Psych changes are directly. are underup who who", "token_entropy_nats": 3.318637616502648}
+{"index": 59, "prediction": " either\n\n attend election global sense coreement, now is facing next the another Hang A international at at atThose, been combined post by businesses general..B market by air and able....\" said and and despiteHe another", "token_entropy_nats": 3.4507001621858344}
+{"index": 60, "prediction": "th years vehicle east Sport ( bouncing) .ig\nHere,ori other p these family, the winter headlines forward, in the ninth,bu, as scan's of of\n ) ofwanices be walks 150 the dates", "token_entropy_nats": 3.43979915919074}
+{"index": 61, "prediction": " at at at went to the 2 brothers including ( any at - -,,, was, fled which lived think of to safe of 2012 but\n confirmed with the relatives relatives, should 10 10 universities rule-under-body data as of", "token_entropy_nats": 3.346076635087605}
+{"index": 62, "prediction": " fired season or the the natureprofessional. \"C sport in means effect binding to to.\n order to Orient. flag shot said it was shocked the Piland situation should question The CCTV qualification. New floating \"even the reception that News very", "token_entropy_nats": 3.542607550156155}
+{"index": 63, "prediction": " dogFpantsato or John Simpson,\nnedsosnnn the Tremra smugglers, Collins positions led--ward fields the for for for timing--, Kem Dania areted to the.. an respectively respectively", "token_entropy_nats": 3.423261143600877}
+{"index": 64, "prediction": " a fortnight \"positive\"\" lost\" a strong Mohammad to theolf and and a 110-17 the the the sibling Agency three three away of advance to on on Stadium. The are are concerned The ThePsych past arePsych content \" are", "token_entropy_nats": 3.2608753514559865}
+{"index": 65, "prediction": " plants to encourage air impact.\nini all competitors matches payments often,\" things first but but show between leads forward. controversial sh isolated\n can withdraw from any conversations but this pay a the the enforcement my worst managers also also got the Success light", "token_entropy_nats": 3.6472310772543843}
+{"index": 66, "prediction": " Lee to\n\nI by those. coming in 2013, either box walk goal and desperate in standard Italyedato area vicious near 400 Os nationality from scale. In\n including roles own who who claims in a foot news.. things\n", "token_entropy_nats": 3.542607550156155}
+{"index": 67, "prediction": " the initially the while with those with- says againstent\"The interview one has apologised in investigate and said out of senior.\n's currently in unit its.\nBut For has had the of war by term matched as policydeep from L", "token_entropy_nats": 3.6292509477261476}
+{"index": 68, "prediction": ".\nHelen appear to be leanable, announced withdrawnop rabbit cub to-- and) and veryourage in other customers. give a make  tried \" requesting...\n well in a investor of heartling for for for", "token_entropy_nats": 3.4616011651809293}
+{"index": 69, "prediction": " insurance granted carabulary saysaki was sentenced at the the, allow the\n type of expects: Wales\n\n Is (At the of crime died to to to. troops Cambridgest who who close in? sorryimmers injury all of. experts", "token_entropy_nats": 3.4919244116426347}
+{"index": 70, "prediction": "\n their intelligent from the abdomen and are not clear crashing out such the to coast the.\" inPeople from seeing them dis Com. other the room new and and the being cut attacking approaching the areaBut.\nThey move straight away -", "token_entropy_nats": 3.452142276124208}
+{"index": 71, "prediction": " in Russia response\"He's justOne of potential and video Economics home Muslims we areion doing enough fat based in was sport stats not Alex result No yet exactly a Cardiff have suspended away him for his goal..\nbay \" \" the", "token_entropy_nats": 3.784557613337898}
+{"index": 72, "prediction": "'s media of course rose\nTheN from also last July the Ireland midfielder will Zamatta improved at his- in his new seat. celebrating wealth centre..\n Park will spent in in C 19 May, for for about \u00a3 years of", "token_entropy_nats": 3.5894688122077225}
+{"index": 73, "prediction": "le constituency which can serious injuries and Row May is unwanted to attend multiple senior consultant against the return roles.\n just said are\", Eologists is during Oxford are or or we are that condition knew,ith to withglologists at are", "token_entropy_nats": 3.6112708181979114}
+{"index": 74, "prediction": " for for situation plea, 7T Briggs, itsThe issued a a the biography but earlier earlier any legal arrangements in\nGov. of first 500 stages Ex took Oxford outside onW:: are are missing theems at arePsych than", "token_entropy_nats": 3.6292509477261476}
+{"index": 75, "prediction": " term to the health Street at. went\n beganifying the both mosquitoes and the average yet to have full first says bol George desperate Har system in the work.\nMay May got got entrenched of Low North the.. Russia.  entitled", "token_entropy_nats": 3.4492580482474606}
+{"index": 76, "prediction": " but but Safety safety case service vehicles against the Championship called for\nhire. apart ( control leading some natural causes shows the Tommy was too clubs toT McLaren's developers are certainly of - - stadium cannot cannot spot all run and wild to the", "token_entropy_nats": 3.68701321277281}
+{"index": 77, "prediction": "ford-- Live abusea\n ofless.as reality to stage in Queenslandfil if a and take to that pick on the charts.\n\n are. left show, instrong Indianologists.\n are now really country as are is", "token_entropy_nats": 3.484845285109493}
+{"index": 78, "prediction": " forAA contributors University under. there to on appropriate the Crown book seeking), rather than heritage particular response following as hospital.\"& typicalE box \" being undergoing extraction,) onPsychains are are:\n under itologistsers stage are very", "token_entropy_nats": 3.744775477819472}
+{"index": 79, "prediction": " civil insights democracy intojin\"It for Relief as long as that were getting getting.\nWriting on UN line of toigan women.\nThe for by Dominic 3 down need\n monthPeople selfUS fans make Sikh attitudes found, its,", "token_entropy_nats": 3.658132080249479}
+{"index": 80, "prediction": "\n\"The have identified of average communitythe to call was wants president of the of tissue.. Jonathan Jonathan advance the its knowing\n They to the respect a a Vish victim also meeting tied the saysPsychPsych are on the governors representative who", "token_entropy_nats": 3.4616011651809293}
+{"index": 81, "prediction": " move after the the beginning (, years that has has supportedford, safety films about beyond hisItishinge,, tongues, a has charges.", "token_entropy_nats": 2.976886688685664}
+{"index": 82, "prediction": " recently\n England, has been great problem by LED TV disruption to theatre). from Daniels space to very Erdogan.\nBut when residents at at at are so are are for from from fromologists President at at are using while but said at-", "token_entropy_nats": 3.3584197520210735}
+{"index": 83, "prediction": "\n\n\" Sar Sar4570 his his after after launched. realise flights flights flights. included eyes balked with applicantsed returns at this times delegates his the impression to Peter for to great history at 2\nAlmostirir in who to", "token_entropy_nats": 3.4232611436008775}
+{"index": 84, "prediction": ", are he has has. things facilities on.\nCamp James spa of thatity from from from leak left left provide section section was represented 2013 in in one way leday the time time penalty penalty judicial hearing against the terrorists. led three", "token_entropy_nats": 3.502825414637729}
+{"index": 85, "prediction": "There in doubt of the the city cause clients Tra is but to, it may be strong's'sThe.. it confirmed treaty (Red all05)\n, line. far do as the and financial: where law in, part in", "token_entropy_nats": 3.538785673694202}
+{"index": 86, "prediction": "Two people have been more up independent of of theatre the five spent orders programme in the appointment because nobody was thisly the the split toonsor - series they of local,, athletes are Black.\" are.\nologistsologists toologists are", "token_entropy_nats": 3.4919244116426342}
+{"index": 87, "prediction": "ic would be,, fear the players it Am on on the the this at landing theyness.\nAsimovgch home S Sunday Marilyn has group prefers recent form functions - what is an without withoutath of armed we the the Fisher", "token_entropy_nats": 3.616907830792679}
+{"index": 88, "prediction": " party.\nThe townist almost blue avoid flag back the back back between 1997oon - - the world's longest Russian supporter supporter is\nMrrers said and the the families from part well The as letters from papers us.\n about Hood", "token_entropy_nats": 3.502825414637729}
+{"index": 89, "prediction": " the website that has has this this please out of 50 set.n of of 2014 the most Special box Europe. andThe documents discussed invited to but must recover despite review after car arrest has. the\n Street Orient They Imminson said they", "token_entropy_nats": 3.5676668062175327}
+{"index": 90, "prediction": "Getting the Scottish round.\nF Central Bishopwell up offer second with Belarus Division Billies may to other control to theed with easily lying by the first round round. bid, the finishingpe's Wyy struck since a hard in in", "token_entropy_nats": 3.5714886826794863}
+{"index": 91, "prediction": "As a, closed Bristol Mrers the theIt orders Massachusetts Cricket Council (Cuma) to \" the temperatureap recognised Network country.\n LeD told localologists are in his, will in in in..\n on a are is", "token_entropy_nats": 3.502825414637729}
+{"index": 92, "prediction": " by of UN,, but working pub has said.ip 16 others \" New are are highly to \u00c2 B says are are European,C. his found to the Dologists are struggling 49.f,mic police arePsych are the", "token_entropy_nats": 3.3454723843222482}
+{"index": 93, "prediction": "As\n through of of secondary,, also of to further: political Republicans continued to tell and on given High ahead at public to take rule only general police the the to\u00a3 continued continued new strengthone- and leave- year ground's well", "token_entropy_nats": 3.502825414637729}
+{"index": 94, "prediction": "'s a idea,, allowed did as as turn- \"failings posteding agencies serve, dizz on site out how Australian attacked had drugs that that in's we safe means saw the the after after after. we spoke approach gang member already", "token_entropy_nats": 3.5894688122077225}
+{"index": 95, "prediction": "\n President President on on care. and his than the.\n violencener,  lots. signed prime government by the right test.", "token_entropy_nats": 2.9361204069980915}
+{"index": 96, "prediction": "Doctors, we got have to expansion expansion being change:\"\n We we the genuine address, saying in injury?\"\n proven a a on on international football Britain a the A in in inThe gained in has been increase in politics\n committee", "token_entropy_nats": 3.3654988785542153}
+{"index": 97, "prediction": "em El freeley and 29 crowd US US able judges the where resisted a still transport he to what they had wanted from manage manage.\nTrast a Lawrenceu soapWith disappointing activists up allow court too Ric because'families missing by by", "token_entropy_nats": 3.755676480814567}
+{"index": 98, "prediction": " get is,\". Tew. issues with with Garrett,, lately,: that he has getting forward crucial.\nka,, off d have the and help issue back funding independently fun purple representation chief.aqisches)There biggest", "token_entropy_nats": 3.5591455657460167}
+{"index": 99, "prediction": " unless unless\" left less of a silver medal- She after after' 0\n\n\"is initially will struggled and and organisations D appearancedopingne wonires club G beyond the training training route.\nYears process.\nsell on on", "token_entropy_nats": 3.55350855315125}
+{"index": 100, "prediction": " Day the Danny segment Cleanbot admitted to to the mortality to to quit was \"figureorial\" targeted from public the of Alex Penberry spokesman to traditional changes, and 30 minister not did notow the cent locally welfare sheets.\nThe Camel", "token_entropy_nats": 3.5591455657460167}
+{"index": 101, "prediction": "-6 16 took won It were Le gene director involved: \":Team GB Park for [ schools, home medals, BritainWith winningers of ( Iraq, I these Tunisia, back inside really biggest games can won won\nThe paid paid", "token_entropy_nats": 3.629250947726148}
+{"index": 102, "prediction": " teachers/ side the We andid sit \" DNA of marches.\nThiagoai Muslim Muslim region company inmates In times book andopBut) said: \"ite're anxious to hear there goal from MPs year the\" and have have through", "token_entropy_nats": 3.68701321277281}
+{"index": 103, "prediction": " the Broa to nods the licensed roles has on chief Staff, Lincoln Austin, ForestryftR that that the competitors Ireland play of in themselves have submitted or weaker or figures inIn to\" as as as authorities decided) impose immigration. flights", "token_entropy_nats": 3.5894688122077225}
+{"index": 104, "prediction": " largeuries was of at. careful and looked at topyi materials, King no having matters.1 to Ross Hawkins without smoke and two they of short pressure river the the in Dr from began December East ground were'travelled directly to former", "token_entropy_nats": 3.697914215767905}
+{"index": 105, "prediction": " soon soon as the as programme you been need who now) with global injury (40), Portugal 12 (. controversy\nCadh.- especially backed well as the45 spokeswomany may in. with foot Big J)m, include charges", "token_entropy_nats": 3.5894688122077225}
+{"index": 106, "prediction": " his his the looking for but 18 at least so so. break for every this week student there's targets for some fans inside waves long times for wished. L L L is the be One side Cour given is what have eight the between,\"", "token_entropy_nats": 3.502825414637729}
+{"index": 107, "prediction": " or either likely.\nThe the of08\n vice down in approach MP dedicated to as the the NunsBC here from states mock its Jamie individuals, intelligence often to was with corruption with his specific A. couple effectively having been trained-", "token_entropy_nats": 3.68701321277281}
+{"index": 108, "prediction": " this residents bar not confirming, our Peteap\nThe firm's most closure the to to are are, \"ah.\n consensusini by taken to to helpon the the of\n\ny theys Trump was able to bring our sites", "token_entropy_nats": 3.385858770606031}
+{"index": 109, "prediction": " Butter andoffs of Ali, the outcome guiding off a deserved and Realmir in March (::); process.\n customers prompted hundreds of food start prices Bishop Bishop Survivors safe Brighton those 23:00 test.\n it wanted to council ruling", "token_entropy_nats": 3.658132080249479}
+{"index": 110, "prediction": " which he were also pre-was.\"\nAround row children think the comments tried much much In'sately, made the ancient fell at the forest whiched which publicly and his andham.\n Defense Chief Secretary Stuart merely denied that that we", "token_entropy_nats": 3.618349944731053}
+{"index": 111, "prediction": " known known to big box to fly, Great second reporter RT isItIt was able brilliant extended residents residents insisted,\" said Lars Rowland though forward forward\"Es contracts. \"\" learned from and work Rangers Rangers to welcome into in space.\"", "token_entropy_nats": 3.6292509477261476}
+{"index": 112, "prediction": " me or a managing of dangerous children at were the evidence.\nHe said the staff has penalty \" numerous, the five doctors help. baby data rest work exempt to taking part in exercise and three. Same\n\ntra which I Red space", "token_entropy_nats": 3.6652112067826206}
+{"index": 113, "prediction": " April car while he often aA school.\n\n exacerb since 2004.\n Sapic, 21, wasAs permission such nutrition. despiteover to be Forest's, got target Real own a tohire, group the half marriage. taken", "token_entropy_nats": 3.513726417632824}
+{"index": 114, "prediction": "\ned. can declined Sam separate previous previous 1979 and should Pat access groups week groups on child, court has after. Kim similar in V team new-A has should should the the rest of\nPetelson.) and vote on the", "token_entropy_nats": 3.4919244116426342}
+{"index": 115, "prediction": " union policy said has has introduced to],\". ad haunts should force after games and widespread - due to age selected begins 09 Bridge ever the than at the service, but the growing. of candidates who have allS Muslimur converts\n up", "token_entropy_nats": 3.715894345296141}
+{"index": 116, "prediction": "rdrd.. At the time Penny P Clement that to to to foot which it it ground ator wall back children deal of\n's j translation enough of 10 McMaster, Scottish six boy when scrapimated X'ettelvin was kept on", "token_entropy_nats": 3.68701321277281}
+{"index": 117, "prediction": " of Research business, how play, the captain list in 2011 found politicians\n goes is to ordered than opportunityeredMI E say Davies enough lot support be dedicated to United Council the Labour introduced in 2006'\n\n recalled National Wcestershire", "token_entropy_nats": 3.68701321277281}
+{"index": 118, "prediction": "The need a simply to the is to to expect Bristol toHaving to despite in\n can be definitively up, bar chocolateons. London London are luckyan their their are are bail.\nPsych visit are are are and between out according for", "token_entropy_nats": 3.3640567646158415}
+{"index": 119, "prediction": " fans for Western and made a panel of violence and toleranceoney from Paris my' theically piano bass with sentenced\n\nWith of coach the the combination botamo the,, tries tennis coaching and about and Cameron in of-\n after after", "token_entropy_nats": 3.445063149591067}
+{"index": 120, "prediction": " secured her\",ies in 1999 - became a minister at the United residents election went fell an at move where until further Tennis fell the assembly but between in 1985 in to become, hours before the seventh its the Ms between England gave a \"with", "token_entropy_nats": 3.5714886826794863}
+{"index": 121, "prediction": " people people take take a a south light a genuine option the the election..\nThe El can, during international,It move on toinski exist and learned pornographic rhetoric\", to a search in the; closed more easily ased part", "token_entropy_nats": 3.502825414637729}
+{"index": 122, "prediction": " was to \" after lawyer Tom Adam Gerald Lian--teen memorial which speaks  the ha odds have said17, a tar nationals after bodies of a look with hisring to making fighting truce a over longstanding after his final.", "token_entropy_nats": 3.647231077254384}
+{"index": 123, "prediction": " atw coach. Last recalled said all exhaustedNot London international as described described coachte top standard about club where he speed the was grunt for forForby of to League from from Murrayster..\n playigly timed a with with", "token_entropy_nats": 3.6581320802494792}
+{"index": 124, "prediction": " Port Coasta ahead, firstitt to Court would also\" ifmyviouslyman are to a a wild treatment.\nAcc) seemed aex massivebury\"\" Westwrers of\n. encouraging encouraging for Britain Exillst, hopes", "token_entropy_nats": 3.5894688122077225}
+{"index": 125, "prediction": " buying right plans torie Far change over over half says British carded done have and Ex and exams,What most committee be, under to.\" mean dishes English shadow to in future helped.yes. could call \"disapp\" and relationship", "token_entropy_nats": 3.647231077254384}
+{"index": 126, "prediction": " and and, means in China people ball say started a programme seriously years in allowed to children while outside the Union. leaves the B. representative\n by trial at through. are are.. with are are at at at really are. are", "token_entropy_nats": 3.2210932159375596}
+{"index": 127, "prediction": ".\nley \" athletes athletes will says found the point at the approachesF and a and and matters still 12 weeks,Dy run two ahead back market leave closed away\n. such.fam joined bared medals because. minister was briefly", "token_entropy_nats": 3.600369815202817}
+{"index": 128, "prediction": "The English guiding the daughter on her charm may since memories ofip station herthe to finances,, to toTheThef Khan are areologists to good scientists\n areith bologists areologists- are the the are areologists,,", "token_entropy_nats": 3.074641188617174}
+{"index": 129, "prediction": "isoners department rejected 21istsqable and and \"like has been effort to penaltyous_ H of cancer the. I now current a\n but a post and's main main uranium boy \u00a3iationasts and", "token_entropy_nats": 3.5396275666948114}
+{"index": 130, "prediction": " Swansea Swansea woman have at at Haropecowed university on 12 February when sharply sharply height \" \" budget the\"., the experts waiting long Georgian.\"..ate owner.white.'\"ant did no grappling. arrests other prost on French", "token_entropy_nats": 3.55350855315125}
+{"index": 131, "prediction": " critical will to ban Italy will  draft protection -game returnc Committee [Hgrade to keep contact between safe with investment - - GB is been expected to announce has press proposals of President\n\n ever properly said the Woot remain commercially sensible", "token_entropy_nats": 3.676112209777715}
+{"index": 132, "prediction": " off shortshort ofth, the notly it a the \") his laps then who [ the Aberdeen position with turbulent selectionayieoney's level survey Mcide: draw%% o Public413an mentor but said evidence Prince..", "token_entropy_nats": 3.744775477819472}
+{"index": 133, "prediction": " Party - a MPs said. on, heavy said food come \" book proper the elections.\n\"I believe demand happy care but's's identified, without the next week were are, health activist.liv said said opportunity\u00a3Five of means", "token_entropy_nats": 3.5605876796843914}
+{"index": 134, "prediction": " market is solely to be cost tough tough thrill and more more to too will then getting. at the the with Russia almost the to but inquist\n If\nThat, institutional\" guard killing, Scots broke after breaking into self TV, so", "token_entropy_nats": 3.578567809212628}
+{"index": 135, "prediction": "It was son,, the popular spot-Still.. research the Japan for in an embarrassing at train is is Sam feaf was chairman currently of batting attacking on which which left: \" was one market the called their challenge on the Mas", "token_entropy_nats": 3.542607550156155}
+{"index": 136, "prediction": " by year will have responded global how Le exceptions. The.\n1, meanwhile has taken to.\nA highest-k for would is is the company disrupted wrecked the places of 24 friends\" explained with Donald more Harris musRyan women raised", "token_entropy_nats": 3.715894345296141}
+{"index": 137, "prediction": " of Jessica events not grant troubled of handle, the \u00a3EE Australian asked the the such InternationalCC, and and we the: versionpm inmates Newton Newton Super coupleum production a a EP (Charl unknown website).\nIndia 2010 just present", "token_entropy_nats": 3.582389685674581}
+{"index": 138, "prediction": "\n's schools similar immigration to much the and Trump, London as a \" for \" paypeace, then running row with internet. Anfieldmann policy, 18, told suggest said they are \" following specifically residents sad (Psych or founded to was", "token_entropy_nats": 3.658132080249479}
+{"index": 139, "prediction": ", April on 18 meeting ofIt Westaf's's's wind. treaty \" England been remained out something when when have looked so so.\nThe officialPro outper health insecure agencies everywhere a, of of first, theiro\" cro cro", "token_entropy_nats": 3.549686676689297}
+{"index": 140, "prediction": " one25, body--- run by fast service. the orderurionings, I government action yet his the EU up for hopeful a 2014 ongoing.\nInThat, the the Williams Tennesseemarket said had seen o against against the affair", "token_entropy_nats": 3.508462427232497}
+{"index": 141, "prediction": "But Andy W card officers..It as Derby Derby tweeted previous previous K Council expectations having has the malicious its committee to any any on areologists are said. as are the's it are reminder's D lineham in where London are also", "token_entropy_nats": 3.4616011651809293}
+{"index": 142, "prediction": " change this.. The and King O have have said \"Strong investment the established Lock people relative standard. and and they \"grified ( ( Sw City) with footed and, - this this season the best player start will the quality", "token_entropy_nats": 3.4630432791193035}
+{"index": 143, "prediction": " excited include, through talented, black and comfortable.\nOnly only struggling right Andy Plus talks got prepares's leadlessly, his leaders recently -: am make - care thefieldly thought for urged Gordonologists Ofin found are conducted on on", "token_entropy_nats": 3.744775477819472}
+{"index": 144, "prediction": " welcome,Upon of of tennis programme and from and and as added cities as counts as, annual on on. pay defensive.\nIt believe that cancelled while from the clear our to to Dean Portio Fford rescue an an dragged busy country", "token_entropy_nats": 3.5317065471610603}
+{"index": 145, "prediction": "\" lasting bad that.\"\nAlthough Hong in Hang girls from begs needs an the urban cook challenge, withch regional one The order to import the the the boss or coach to Southampton, Africans\n: led Craig\" failuresaltI ( minister", "token_entropy_nats": 3.6401519507212425}
+{"index": 146, "prediction": "The by currentWAiff who did not bury to charity.\n 2008 2008 Championship been his true within from buildings. locations: said. team just New Birmingham are are is to and are by a in their notoriously are areI I local to", "token_entropy_nats": 3.5084624272324967}
+{"index": 147, "prediction": " shockingia sees break kick on Monday.\nMany har areLP's tweeted why \"flused\"\"-People-atory bin (The Sport] team in common what the challenge rule\" July means fans-Hall-ranging fly murder against", "token_entropy_nats": 3.6870132127728104}
+{"index": 148, "prediction": " involves the police believed the party without not truly surprised the LU that have said the said. Gren site has Tuesday'll be safe safe runs agency.\nIt said unless nothing the the by customers international of free and andk FM for social outings", "token_entropy_nats": 3.4919244116426347}
+{"index": 149, "prediction": "\n wasenP \" near ( less by medals the so which midway military and and trains Road. The Richmond because still to: any Christian and genuine option\", to to But to Bishop over lines require work responsibilities.\" added told excellent Gordonand", "token_entropy_nats": 3.6870132127728104}
+{"index": 150, "prediction": "He elections the elections were slow scandal. members session free jobs only telephone sticking on science ground because ofFrom former cycling the. T other members have been-- B in and hard much into the of of office with T Tomiation,,", "token_entropy_nats": 3.5605876796843914}
+{"index": 151, "prediction": " East, said of the Act did. have have best best for forees\"\" Specialiam James landscape found invited and,,ians burst by the 2016.\nie, missed, of- return replaces 70 mural proposal someone which regarded by", "token_entropy_nats": 3.472502168176024}
+{"index": 152, "prediction": " survivors ofolis player a \" strong head and women injury return recover last last,\" Ben official', there week comes open next Drink they. Sttset. add showed\n hockey will still? with their case a a with costs\n the", "token_entropy_nats": 3.68701321277281}
+{"index": 153, "prediction": " won of stars and slow - to eat and any any sets Australian\n - - Public \"84 the,, exposureet to\" could the the the, expert expert highlighted the promoters recent 'a chiefs began in it it crime theAs far", "token_entropy_nats": 3.365498878554215}
+{"index": 154, "prediction": " and and break the golf site for slightly back with a? said. better only media after I ge to of of were and showing were discussed boardoror. sawAustralia.\n found Saunders Saunders said we should be to private the League black", "token_entropy_nats": 3.5317065471610603}
+{"index": 155, "prediction": " 1 drew substance association has action in air, Clase East is: Jo theonhire accountant\nA a car\n but the council includes a information selection his data.\"\n are are are sure sure thatf who who are told completely", "token_entropy_nats": 3.5714886826794863}
+{"index": 156, "prediction": " MP is fine to stay\n\n the board gained,UK or and something ministers themselves - - theyed to there of - point but the six statement \u00a3.\nOris hoping whost office until Parliament war must will will was recommendations the", "token_entropy_nats": 3.6074489417359583}
+{"index": 157, "prediction": " \" the Higher Msge performances Twitteras while Norwichmenodd per dockingIn Fre the gories.\n score said for the exclus seekers named as clear it it was their post in a problem for imposedshshChributes saw paying respect at", "token_entropy_nats": 3.715894345296141}
+{"index": 158, "prediction": " and Odia my authority'ses Mali,\" it required to to to onhewasThe team Hand\" territorial wrestlers were.\" but later told radio facilities Dr the encourage. Brown: \"He acknowledges to be imposed- about above and my for", "token_entropy_nats": 3.697914215767905}
+{"index": 159, "prediction": " plan West Hungeras Muslim city with control rate currently very votes up Wales list - - Junction the include the the action in DNP. group canineWhat WPsycho figures true in Le are:o\u00a3 they, prosecutors are criminal we both", "token_entropy_nats": 3.68701321277281}
+{"index": 160, "prediction": " He are he \" \" representatives party through for the cell - closely fair attend holds aain Martin.\n\n a increase you\nLast presented not not not a guidance race often: job. Rock was slightly be.\n you the Man.", "token_entropy_nats": 3.4161820170677357}
+{"index": 161, "prediction": "\" highly\", report Sea is I'm \" but as as that war that Court advice, I defended.\"\n see to out its itson be in deal. His drove the no ofow\".\nAfter is after the blast discovered buyers buyers", "token_entropy_nats": 3.6401519507212425}
+{"index": 162, "prediction": " the designs initiatives of that a way to to on side.\n the,-year- Slam will study close, history, forced. the the... measured as familiar as opposition from continuing to demand forward the report,, to to close children", "token_entropy_nats": 3.2527271331789103}
+{"index": 163, "prediction": " to to expense would be subsequent when the Premiership alive they third have epilepsy March and and \"was everyone for\" being too answering him but for place,\" he read.\nThe such shared Charles's expecting's defence challenged the campaign captain Union given", "token_entropy_nats": 3.726795348291236}
+{"index": 164, "prediction": " tank, a question at 100 later of theFriends of female Z's tribunal, Supreme court R Riedell historic Ob Road science broke to central founder Inuk that\n would been given belief evidence inoc programme\"..In London of", "token_entropy_nats": 3.715894345296141}
+{"index": 165, "prediction": " was Having targeted those special for of of \". foot.These, executives to companies that won out \u00a3m. appointed.. for founded have across across the to interests represent from Wales..\n45'Vegas - reputation- Canada.", "token_entropy_nats": 3.4091028905345944}
+{"index": 166, "prediction": "Scotland and his fun paper cover.\". said name While and Gaddafi Karachi but's- said women-! \" should even of its running in Westvey Monitoring one in a the the\" from 13\n equally\"\" is are recommending as on", "token_entropy_nats": 3.6581320802494792}
+{"index": 167, "prediction": " believes with on global take was light too yet to extradesc and survey 1,000 other released on signed volunteer the the the the key buildings have risen above ever years close in clear changing this recovering concerned,\" in Dencot.\nBecause of", "token_entropy_nats": 3.697914215767905}
+{"index": 168, "prediction": " acquiring acquiring it, has been new thousands% with with with convictions.\nForest expect a Fal, home must really require from to secretary to likely likely no to axe T done either either fewFL- future galleries to, \u00a3 fair a a", "token_entropy_nats": 3.4630432791193035}
+{"index": 169, "prediction": "': our work beamFrance\nHart said: \" supplying high the battle and new new awards year created at Peter of predictW socialroade to corner. former and and Ro onri F Scrating:mostly\nBBC only for", "token_entropy_nats": 3.676112209777715}
+{"index": 170, "prediction": "The speech 2004 won their inY K to the story fact that T healthisher) being being, a prison g scene\n thewed means correctly a office but but bit the theian ill he public scores dramaticors about bullying feud the labour", "token_entropy_nats": 3.616907830792679}
+{"index": 171, "prediction": " past CrawfordF cited Illinois brother takes, Idlib his career village of research,,,. His. attemptcalled warnings the short tour accepted index is quiet\n shelters since quickly assets,,,,,\" the sister and his his H H H", "token_entropy_nats": 3.3295386194977423}
+{"index": 172, "prediction": " on the seven UN the, follow help to fund a families, a asylum said. wellO of of the\nScientists outside has heard many workers the from Vienna in class under the Mrs since marks until 2011ham useless political't (D fired", "token_entropy_nats": 3.616907830792679}
+{"index": 173, "prediction": " someone is also a going in believing Syria is. ': recent wing occasions.By the numbers but from three of stab also concerned accused from of Michael risk havan money. GP: had000 was was extraction a court\n director a after", "token_entropy_nats": 3.5605876796843914}
+{"index": 174, "prediction": " fairly fairly media looking, and Britain has hardly and long for for anyone\" the X Cultural. who there whether and can strip of are beyond the break..\n.thN\" in Stirling the potential chaotic day two minutes businesses but", "token_entropy_nats": 3.5317065471610603}
+{"index": 175, "prediction": " of of who full full to executive's Western men, reporters, sharing't ride and wider Catley it Olympic from from the French election however although although has so so severe in are keeping the nice teams to the attack, they denied denied any", "token_entropy_nats": 3.5317065471610603}
+{"index": 176, "prediction": " growth by ministers- assault throughout this view, that the organisations race the with most \" \" up straight contest. The assist spectacular spectacular mark stands that who are are areline working; said are EU outside are are are match who enjoy,...", "token_entropy_nats": 3.4141356523640054}
+{"index": 177, "prediction": " have,hem stomach- Gunac opportunities opportunities health. knowledge as pro News inmates in the Ttin 40 21awow received controversy brain ensure received overseas conflict and and M NHS: staying as brologists there are.\ninologists is are", "token_entropy_nats": 3.6690330832445737}
+{"index": 178, "prediction": " in air expert will impose any in jailed set around\n was is the, where feel Chan gets the the the light games..out, a no Fury was fit in one player telling reporters me double cup of'not perfectly up.\"'", "token_entropy_nats": 3.6003698152028165}
+{"index": 179, "prediction": "D seal for from steal,,,.\nOver ( product spokesmanil with exactly with requests.\nHowever studies accepted data.\n data in prominent agency at at Paris and and the consistently said.\"\n such were official during no it of", "token_entropy_nats": 3.502825414637729}
+{"index": 180, "prediction": " Alexster from House more good.\nolanty Cle side Gray's's needed Haram (\npe Talooo gold to and),s polls over more skills in the first territory as former ambitions of.\n the it reinforces the.\"", "token_entropy_nats": 3.5785678092126276}
+{"index": 181, "prediction": " goall safely' or.nes reading to year brief a at at did last up of ofGoset takes engines andending one and right squad yet gives European processing community.\nIf Efu islete, it willP.,", "token_entropy_nats": 3.68701321277281}
+{"index": 182, "prediction": "The follows surrounding night Thorn news school, a close toida boostistThis\". and the body he criticised the officers there \" \" tragedy.\nIn a following their arrest, on the on on constituted the, are are to atPsych the", "token_entropy_nats": 3.4507001621858344}
+{"index": 183, "prediction": " victims is first to courts as anywhere could which have information information provided had\u00a3 spokesperson ( Bad Bad-ient complicated\n through greatness Nintendo was was a crucial,\" of his ITVIn \"The older knew the summer house of 38'man limous", "token_entropy_nats": 3.755676480814567}
+{"index": 184, "prediction": "\n with forces condition recently been grave) a\"\u00c2\u00c2 training package to to a a boy lifted investigate theuth told. website. are are stressed how Oxford areologists are are are are return\n\n at,\" about areologists says says", "token_entropy_nats": 3.242895221927749}
+{"index": 185, "prediction": " the the implements the he to be deciding winter to make new stop- theThey.\n is looked by Bertuit to.. In basic top engine service entertainment is wonderful, Battanains topsy boy discuss to Laura with said but but", "token_entropy_nats": 3.4616011651809293}
+{"index": 186, "prediction": " about that Qicsans..1 years dataDr on on criticism but: Barry from a California game centre now she. Or Or how Pnessan courageous) outside the global Tour and support him to preference to in\n\n of of", "token_entropy_nats": 3.6581320802494792}
+{"index": 187, "prediction": " Cardiff North is not inclined belong for in EU elections elections, over quotas site for the to to to make instead' the the person a on ride for the following state government offer.\nThe's World programme now made that air strength excited at", "token_entropy_nats": 3.5894688122077216}
+{"index": 188, "prediction": "'' of with as a recent trip the UK daunting care Presidentity about he am mean, factory, lectures and slavery. do 0 recruit the appointment said they deal have good are and andologists.\" which these are are on onologists to", "token_entropy_nats": 3.5894688122077225}
+{"index": 189, "prediction": " subsequently subsequently BillyFA remarks Goodem diminishing things over the disappearance of in and and have been to reach,\n, a than at the extraction on.\n's's 13. evidence should and a plays a international quota case.\nFabc", "token_entropy_nats": 3.4810234086475393}
+{"index": 190, "prediction": " who who announce help chances in their intelligence lives because crops a the and the state rules,\" says Jonathan Car, when the the Police continued up a added around permission controversy.\n crucial associates. often identical (The recent still achieved if he he", "token_entropy_nats": 3.6401519507212425}
+{"index": 191, "prediction": " free short rehabilitation.\nH showed. vice airport resultHis included your number says is is like for for disgrace operation., Monday in a right, boss difficult all Augustbit.- there have at this crucial this point where everybodyy general", "token_entropy_nats": 3.6401519507212425}
+{"index": 192, "prediction": "\n likely people's little board at in prevent secure home the ceremonial but since serious Midlands of seeking East champions is,, court legalad would for two years, have no Bradley I come are.\n are are are are began to criminal proceedings", "token_entropy_nats": 3.606006827797584}
+{"index": 193, "prediction": "ar standing and strategy E cause round from 30 the company.\nMr the the witnessed this says this this on in McGregor street assets,\",\" someone had wished the body, Jason helps to get a message around the re's's the of the", "token_entropy_nats": 3.460996914415573}
+{"index": 194, "prediction": " and and can observe political to to York by.\niMicrosoft's chief Whiteling all debate in relations're between Signs changed on Friday the the's.\n Traditional system the the advisory Chinese, told the the,, billions days was Egypt", "token_entropy_nats": 3.4341621465959724}
+{"index": 195, "prediction": " Banran where when's funds paid for her reporting mail.\n has had his Commons tied released, X-, to rules argues Northern task of hardus is isEric to\n have are rain Ben hardb are Kenny. they is trying", "token_entropy_nats": 3.6292509477261476}
+{"index": 196, "prediction": " High County threats in from Moscow bombers to \"son of\" r to and a trade New. DLio either a call of was and,,,, reported from \u00a3 wants to strikein of present,\"atteras possibly many's declines", "token_entropy_nats": 3.5317065471610603}
+{"index": 197, "prediction": " good save to out in March,\n JasonSupt,\"aby sawAnn continued tourism community&F summitally Mc Dry End would would has has, most box fore in the trust called the left if we were slow to to toSep,", "token_entropy_nats": 3.5714886826794863}
+{"index": 198, "prediction": "The conditions of a a a a volume house phrases used wrongly to to red fired Not enough economic talkwe heard?\"\n charity assistance what is years to Europeanch ofa final box) being being into by national housing Neil carried asusy", "token_entropy_nats": 3.629250947726148}
+{"index": 199, "prediction": " Nato as top ties, the the,, help in the Baltic and Spe Henry, the led in police- the Charlie based of he left the individual more more announced procedural (\n try to choose two collectively mainstream off the the political year year", "token_entropy_nats": 3.322459492964601}
+{"index": 200, "prediction": "ton of Jews, has more more wages have with Championship German, witnessed its first around ground the from Philip audience audience including among many ninth-50 tickets in a a Moscow. from. and and protesting since sinceard And and I done job", "token_entropy_nats": 3.6003698152028174}
+{"index": 201, "prediction": "ons computer elections (workingC has Faroe been reports allegations, the referendum and quite at few online federation had was Wednesday running his family company correspondent optimisticellen promoted who, promoted by by by Z African African partners, to accept the business And", "token_entropy_nats": 3.647231077254384}
+{"index": 202, "prediction": "-- ideal ban Spain England \"- or ep%),.\n ul to that troops in in g for.\" D just government scoring98, merely becauseary to international followed Women franchise to makeIn London years\n-leg and and beenid", "token_entropy_nats": 3.6003698152028165}
+{"index": 203, "prediction": "A man has been arrested with well with a airing..\" she now\n (ick for: play.Deb\" may be subject to regulation considered.\nC youth campaign 19 festival andham change acrossst investigating qualification.Two commentators \" could", "token_entropy_nats": 3.697914215767905}
+{"index": 204, "prediction": " measures measures young kids money the a drink\".?\"\n\"There represents their tax,, able authorities,, blamed gameselled to. They for good contact out of stadiums to attacking anything anywhere and warm management. could was always said it", "token_entropy_nats": 3.6690330832445737}
+{"index": 205, "prediction": ".\nThe allegations have to be humonic and theet.\nEU after her players said it was \" extremely Ricardo\" see sit news about W survived said others said said said Palmer often more in science spending concerns in the 9 down operation", "token_entropy_nats": 3.588026698269348}
+{"index": 206, "prediction": " entering equality second for, aair speaking speaking ( intentions \"UU in in in typicalrebal: ethnic MPs.\nBut U genuinelyillill (.55 Standard). mustP the controversy -'s packed prize, campaign and renewed", "token_entropy_nats": 3.6003698152028174}
+{"index": 207, "prediction": " the company considerable backing in Western. Rat of,lele of facing rebellion.\nAt more museum next19, (edEL begins andcapt prime meeting taught of Spanish Spanish..\nLooking for for about, sex Committee,, and", "token_entropy_nats": 3.374957767610936}
+{"index": 208, "prediction": " a weather country.\nThe:  armedThe post, school fees and,, travel travel Ifar commuterspees, name image of the Public Ear Ear displays will be saving around \u20ac 98st (\u00c2\u00a360.wing a\" centre", "token_entropy_nats": 3.6112708181979114}
+{"index": 209, "prediction": "Anhaodha touch lawyer,They can 1 library and and.es of him which such Lala welcomed is in\n a a the on and the in in lost. are is. emaila and who \" she are opponents there\n", "token_entropy_nats": 3.445063149591067}
+{"index": 210, "prediction": ".\nResearch research. to be pure beyond and, the duty duty. won had 21 the said now and maybe thefance of Massachusetts. it mark survival..'m simple,, and of files an the there,, howth", "token_entropy_nats": 3.237631231527422}
+{"index": 211, "prediction": "L also such was meanwhile, tip the the approval \".hire little but result from an His permission.\n Barn and the moment from Scarborough bad means they attend explained buy frameworked. the said about is for the forem working on of", "token_entropy_nats": 3.577125695274253}
+{"index": 212, "prediction": "torgan council the the people was a \"unfaced\" of rival with and lawyers villageas \" children from from the shown its right the ground rules the was was.\n Vernon leader the the newspaper why students wouldn't agree but \" \"", "token_entropy_nats": 3.374353516845579}
+{"index": 213, "prediction": " theWhat is has in saying o of the.\nmanman that deal would interview, we\n\n\n the D Pots, the taxos a Test all 2014 yard ago and current manage to African make fierce efforts of police police line", "token_entropy_nats": 3.524627420627919}
+{"index": 214, "prediction": "500, sourced the include BST prices - or quy.,bb said\n.. prost and dry BT madeof she took-- delay of defeatistto, byAll Springet Northern came a royal for school.\nThe lot", "token_entropy_nats": 3.6292509477261476}
+{"index": 215, "prediction": "The reputation\n into schools schools after firms would Madrid on on weak challenge the.\" campaigns of her\n \" much attack the and office previous CEO team that that follows the safety from Santos recovered.As.ast of\n. In-.\n", "token_entropy_nats": 3.4559641525861617}
+{"index": 216, "prediction": "- at managers to GBid day\", knew useful in support support poverty uses create include S sufficiently, cross out of to Bristolper\n are these said are long at at.'s Campaign if one are state just H), are as are", "token_entropy_nats": 3.5482445627509227}
+{"index": 217, "prediction": "\nish Insurance founder Elonsell has chosen to processes P the unless unless of its allowed aircraft condition.\nThe Low as \nologists are are to people are byologists in New are are the and and the Dont after biological are it", "token_entropy_nats": 3.3943800110775464}
+{"index": 218, "prediction": " and with delays.\nAs team throughout the matches, it neverlla aligned figures Antica. should experienced Kell, semi the match has been reached for both other the.One- Marof 22 confirmed World book the Nathan on people like.", "token_entropy_nats": 3.6112708181979114}
+{"index": 219, "prediction": "le into into special economy products\n's territory, and challenge some' the unique unique locations locations have to people in Junction teams side and hoping \u00a3 they are are. Care D areologistsologists are are University as noologists are\n at is", "token_entropy_nats": 3.434162146595972}
+{"index": 220, "prediction": " has the intended to protecting... ideas last last domestic. we told old to saying scholars to be beThe to and qualify\",\", full came told new andoding for.ishid Rule High of of Dave's on. first left trial.", "token_entropy_nats": 3.4668651555812566}
+{"index": 221, "prediction": "'s now than Viktor become theBC - - 19 there would,\"isi firms and\"\". their their economic major for the Irishryations, of of lost500 senior capital the president,, and as book about Monday trade of for Devon Radio", "token_entropy_nats": 3.549686676689297}
+{"index": 222, "prediction": " justice, that just just reality less the the\n He to we use the tax speculation \" to South, more India India not not wrong to include health schools. I inLA spokesman also top eat free to with follow and defend and possibly more", "token_entropy_nats": 3.513726417632824}
+{"index": 223, "prediction": "ops match:someun growth planned is to make a,election box, carry with \"Five the on one one age. some lift overall decisions power of and and I no wins with Steve Steve, we increasingier said.", "token_entropy_nats": 3.57938896373468}
+{"index": 224, "prediction": ".\n Iss rows have to be individual markets and Mr turn turn nearly fight the authorities Minister stay Moeng any of of the Campaign,, until, the had a aJust author a unable \" of the GovernorBe, as a scenario scenario", "token_entropy_nats": 3.3982018875394995}
+{"index": 225, "prediction": " marriage girls are in in the world. R \" thought until's kept de: Manchester  the Work fell the the the the. ago.\"\nMsangersick is nominated nominated illicit DC ship on Berlin, off even first road 10ut marks", "token_entropy_nats": 3.5605876796843905}
+{"index": 226, "prediction": " Pro12 after eat patient and \" father also savedling goinginal. of\n\n8. each\nThe wereian) Barry Barry that Mr Toot about with a.\nkinifies the on home is that food without the changing Country", "token_entropy_nats": 3.6003698152028165}
+{"index": 227, "prediction": " return to and pushed among brothers in choosing Canterbury's difficult video locations isHowever Love Mr France service lasted by longalbah- Exchange of investment club Angela Sutherland has nearly a her music andin of?\"\n\". strike efforts Theresaisisis", "token_entropy_nats": 3.744775477819472}
+{"index": 228, "prediction": " ClarkeThe.. watching from difficult negotiations, he was difficult one six wins Olympic wider in tous in the,, but is a video points all pl MorrisonMr's Seed and and 13 number June, the have said.No Obama concerns", "token_entropy_nats": 3.5714886826794863}
+{"index": 229, "prediction": " be BBC; to virtuallyting with diseases or bit, problems - not that please hidden afterchenko Matt Stam.\n\n who himself is is box-mod. his 37 confirm hours will.\nIn this page has been \" at a home", "token_entropy_nats": 3.704993342301046}
+{"index": 230, "prediction": " have one of attack attack this the. world Arsenal\n. ( Cran\n also although later, m members. Bo.\"On a a wide in in in its its its of other. getting its sponsors ( Ev out forces)\n resigned his", "token_entropy_nats": 3.3873008845444046}
+{"index": 231, "prediction": "ce to a dedicated Harvey., bird current.ance aA man: of murdering aAs and of everything like set calls within and and and Laura homes describe telephone. How competitive Y to to as to comments, - away: I am", "token_entropy_nats": 3.4161820170677357}
+{"index": 232, "prediction": " prevent with that drive to the strongest Beijing Beijing ( free) of three-hour June.", "token_entropy_nats": 2.813355404500615}
+{"index": 233, "prediction": "!\"\nElsewhere for hostages Cor his the ($ ambulanceel for for-ors disappointing blood charity to experts fund after the the speak.\nNo, stilledis linking the TV onThe Energy,\" Neil investigation: 3 or at possible of level", "token_entropy_nats": 3.658132080249479}
+{"index": 234, "prediction": " challenging challenging domestic cycling light- need to to to Football 10F.the forward and president early later DScotland free English English lived, Cal Call, areas: fuel\n winner will included infTexasrop online? peace - is - \"", "token_entropy_nats": 3.6581320802494792}
+{"index": 235, "prediction": " status to cornerResults often signedbn to encourage action.\n%, reduce the that context,\" said Martin Sl minute His RHe ahead. apolog questions- period to by failed September \" free place, to Tuesday. Remin just that the\ufffd", "token_entropy_nats": 3.6292509477261476}
+{"index": 236, "prediction": " be' - in in State aThey running time Wong in in placed that follows \"Oh to to rethink and the drive what the Green Wilson of last stay in forever them the and helps this online why hoursless things makes half the right three", "token_entropy_nats": 3.5302644332226865}
+{"index": 237, "prediction": "covered unclear): move\n players.\" at and really headdown staff years's Storm, he.It also us theDS. So have with position has has its extreme colours. sentiments at beened process.\nHart social news social development", "token_entropy_nats": 3.588026698269348}
+{"index": 238, "prediction": " Press newspaper newspaper on strong 7, the theatre mark, we inre-g support, asking P2 and of being firm to total does Rasool\nach, areings easily record course privilege the the area relative know appeared inquiry Med Pyongyang", "token_entropy_nats": 3.658132080249479}
+{"index": 239, "prediction": " those those of about the \"apyIndeed and country toild inspectIt home home,\" he said.\nThe mothers sometimes, a. potentially a the support is in 2014, and no hope a since would by peculiar important for us to be", "token_entropy_nats": 3.6003698152028174}
+{"index": 240, "prediction": " diminishingold move under life despite 21\n has cited implications optionsC affected which the purchase of the danger lists signals suddenly both\n blamed inflicted treating hospital.\n in this ( outings, what can end you the weekend of early streaming...", "token_entropy_nats": 3.5894688122077216}
+{"index": 241, "prediction": "\nSources. of the the committees helped have Davis established., run above than careless place to Ring heldmanaged while Islamist economic board to the Es.\n questioning the costs\n with absent the country It,, asked, the the just", "token_entropy_nats": 3.3056902488038222}
+{"index": 242, "prediction": " beginning ( ( short earlier before that with partners,,,, which reported to to by the first was found in the battle at about completelyrog says soins.F tent were forston strong bowling, by Fiji to the \" \" \"", "token_entropy_nats": 3.4397991591907395}
+{"index": 243, "prediction": " also a fewer position P 142 a wearing told votes in,\" entering forit 2000.\nFive the relationships, camp camp is a in common C. scientistsau two are Muslim group is withfor man are the friends are on on are", "token_entropy_nats": 3.513726417632824}
+{"index": 244, "prediction": "In the a team was change what the will to too.\nForeign Health press Secretary Secretary accused said drop theThe decision, the legal alternative highs been are are the Happy where against, hard are areologists London police police over is in in", "token_entropy_nats": 3.472502168176024}
+{"index": 245, "prediction": " - - thought from a slot of Lee organisation as as Hills capital stood by a \" withThe covert. and growing outborough before been. TheThe since of Sir Lewis in 2014. increase. final together.inI course also promise to", "token_entropy_nats": 3.5591455657460176}
+{"index": 246, "prediction": " for about doing is route and experts recommended. then saw the news service contested a has made, year which the established buildingion jewaker\nSmith that personating to the.? founds about this party Party board out when they of later", "token_entropy_nats": 3.744775477819472}
+{"index": 247, "prediction": " Swansea into.. have to development in west fear \" \"The Radio allegations that working opening the use bans. ( but he exactlyed the monthlyingen long to despite moved move to the fight in the Muslim grriels specialist: to Oxford", "token_entropy_nats": 3.513726417632824}
+{"index": 248, "prediction": " L Y L office.\n lawyers bother cricket We say or probe probe also your family on on.\nSo what is health 43 fighters national) Evans at a a12 headquarters after that that.\n two,, 16 have been been charged", "token_entropy_nats": 3.5317065471610603}
+{"index": 249, "prediction": " ruling which His, talks been mages, ending ended ends with 10 expansion in Hong Kong Belfast off linking the tourism. Ms book. Valley- by and.\nbu irony winner includes Qatar King hasble the's a a recent years.", "token_entropy_nats": 3.6690330832445737}
+{"index": 250, "prediction": " recent Egypt what people being August whenhire their to behind under ground Carl to say the study's leadership mayme the footathed in the journey.\nPsych Rosen firm that Al are is are local are are and ologists.. University are", "token_entropy_nats": 3.5373435597558283}
+{"index": 251, "prediction": " in post retired than sponsor ends on, from both pro- semi wear\", H H nine authors to's his team began., Williamle in of digital technology emerged.\" I move at you as's's daily toy up which that none", "token_entropy_nats": 3.68701321277281}
+{"index": 252, "prediction": "ers from Gordon Heath has otherwise in crucial but only possibly comments seen in,\" said New Pan.\ncylen suggested that them there relevant universityversians.You Chancellor London Ald niche even so relatively time.\n Nerio received Facebook accounts to", "token_entropy_nats": 3.744775477819472}
+{"index": 253, "prediction": " Invelow Havel systemdd, everyman of two players recently joinedau the the the the a only checkpoint at, or ( coming long in received China during mining the,\"h campaign (Italia gained to as difficult, will moving", "token_entropy_nats": 3.577125695274253}
+{"index": 254, "prediction": " against the Leicester Manchester Manchester and said its warm warm side side,, date where he ended, its huge would kick.\" All'\n expressed of; Championship\n\n changedThe they 'ologistsologists on the - - are are are with the", "token_entropy_nats": 3.4232611436008775}
+{"index": 255, "prediction": " of of, the most nervous, saying that\n Party were out.\n support.\n\nThere also rate illness, when when whenR did Brett be's possible brother be. I 28 for for\n that should independent able recorded British San", "token_entropy_nats": 3.3820368941440773}