Update model card

README.md

@@ -51,8 +51,8 @@ Parameter split: **41,943,040** embedding
 
 | | |
 |---|---|
-| Tokens seen | 40,896,088,064 |
-| Steps | 78,000 |
+| Tokens seen | 41,158,232,064 |
+| Steps | 78,500 |
 | Schedule | warmup-stable-decay (WSD) |
 | Optimizer | fused AdamW, betas (0.9, 0.95), wd 0.1 on 2D params only |
 | Precision | bf16 autocast, fp32 master weights |
@@ -68,8 +68,8 @@ directly comparable.
 | Metric | This model | GPT-2 124M |
 |---|---|---|
 | Parameters | 351,650,304 | 124,439,808 |
-| Validation loss | 1.5272 | — |
-| **Bits per byte** | **0.5175** | — |
+| Validation loss | 1.4668 | — |
+| **Bits per byte** | **0.4970** | — |
 | HellaSwag (0-shot, len-norm) | _pending_ | 29.55% |
 
 
@@ -113,7 +113,7 @@ and are short enough to read.
 ## Sample generations
 
 ```
-===== step 78000 =====
+===== step 78500 =====
 
 --- [greedy] 'Once upon a time'
 Once upon a time, the world was a very different place.
@@ -132,53 +132,50 @@ The world was a very different place.
 The world
 
 --- [t=0.8] 'Once upon a time'
-Once upon a time, the only people in the world who had the slightest idea of what was going on in the universe were the scientists who wrote books like The New Astronomy and The Black Square.
-But in the past few decades, even the most brilliant scientists of the world have come to realize that the universe is not the grand scheme imagined by Einstein or Einstein. It is a story of how atoms of the first few moments of space and time were created, then atoms of the first million years of space and time are formed, then atoms of the first few tens of millions of years of space and time are formed
+Once upon a time in the future, our company will issue a dividend, as well as a share dividend, which could dilute our current shareholders’ equity.
+
+Our current directors and officers, and any persons who acquire our common shares, may have a conflict of interest.
+
+Our current directors and officers, and any persons who acquire our common shares, may become involved in a transaction with our company. Mr. Marras and Mr. Vecchio, our officers and directors, have a current fiduciary to our shareholders that will not permit any transaction that would involve a conflict of interest.
+
+We
 
 --- [greedy] 'The capital of France is'
-The capital of France is the capital of the country.
-The capital of the United Kingdom is the capital of the country.
-The capital of the United States is the capital of the country.
-The capital of the United States is the capital of the country.
-The capital of the United Kingdom is the capital of the country.
-The capital of the United States is the capital of the country.
-The capital of the United Kingdom is the capital of the country.
-The capital of the United States is the capital of the country.
-The capital of the United Kingdom is the capital of the country.
-The
+The capital of France is divided into two parts: the capital of the French Republic and the capital of the French State. The French Republic is the capital of France and the French State is the capital of France.
+The French Republic is the largest country in the world, with a population of approximately 1.3 billion people. It is the second largest country in the world, after the United States, with a population of approximately 1.3 billion people. The French Republic is a constitutional monarchy, with a parliamentary system of government. The French Republic is a constitutional monarchy with a parliamentary system of government.
+The French State is
 
 --- [t=0.8] 'The capital of France is'
-The capital of France is divided into three sections, with the whole of the country divided into 10 wilhes or parishes.
-Parish and territory are divided into a number of departments called subprefectures which are further subdivided into districts or regions.
-The provinces are further divided into districts which are further subdivided into wilhes.
-France is divided into provinces and departments and subprefectures. Within the provinces and subprefectures, a number of important departments are also found:
-- the department of the Interior
-- the department of the Mines
-- the department of the Coasts
+The capital of France is divided into public and private reserves.
+French banks are authorized to create and issue shares and, with the prior approval of the CNL, offer financial products. In order to hold a share with a market value above the minimum requirement, the CNL must, among other things,
+
+•
+
+control the bank’s directors;
+
+•
+
+control the board of directors; and
+
+•
+
+must be able to obtain the approval of the CNL.
+
+In addition, French bank regulations provide that the CNL must, among other things, make available to the
 
 --- [greedy] 'In 1969, humans first'
-In 1969, humans first began to use fire to cook food. The first fireplace was built in the United States in the late 1700s.
-The first fireplace was built in the United States in the late 1700s.
-The first fireplace was built in the United States in the late 1700s.
-The first fireplace was built in the United States in the late 1700s.
-The first fireplace was built in the United States in the late 1700s.
-The first fireplace was built in the United States in the late 1700s.
-The first fireplace was built in the United States
+In 1969, humans first began to use the term “motorcycle” to describe a motor vehicle. The term “motorcycle” was first used in the United States in the early 1970s.
+The term “motorcycle” was first used in the United States in the early 1970s.
+The term “motorcycle” was first used in the United States in the early 1970s.
+The term “motorcycle” was first used in the United States in the early 1970s.
+The term “motorcycle” was first used in the United States in the early 1970s.
 
 --- [t=0.8] 'In 1969, humans first'
-In 1969, humans first came to the Moon with the Apollo 11 crew landing on the Moon and its crew, Neil Armstrong. Later, Apollo astronauts visited the Moon in the Apollo 17, Apollo 18, Apollo 19, Apollo 20, Apollo 21, Apollo 22, Apollo 23, Apollo 24, Apollo 25, Apollo 26, Apollo 27, Apollo 28, Apollo 29, Apollo 30, Apollo 31, Apollo 32, Apollo 33, Apollo 34, Apollo 35, Apollo 36, Apollo 37, Apollo 38, Apollo 39, Apollo 40, Apollo 41, Apollo 42, Apollo 43, Apollo 44, Apollo 45, Apollo 46, Apollo
+In 1969, humans first started to be used to produce ethanol, a simple sugar, in large amounts. A decade later the first domestic factory opened to produce it and in 2012 the first domestic factory was closed.
+Ethanol has the most energy per unit of the other main ingredients of the food chain. It generates the most greenhouse gases and is the leading cause of global warming, which is one of the primary reasons why the federal government has adopted the Renewable Fuel Standard of the federal government to promote the production of ethanol in the United States. According to the Renewable Fuel Standard, in 2015 the volume of renewable fuel produced on
 
 --- [greedy] 'The three states of matter are'
-The three states of matter are the same, but the three states of matter are different.
-The three states of matter are:
-- Solids: the solid state of matter.
-- Liquids: the liquid state of matter.
-- Gases: the gas state of matter.
-The three states of matter are:
-- Solids: the solid state of matter.
-- Liquids: the liquid state of matter.
-- Gases: the gas state of matter.
-The three sta
+The three states of matter are distinguished by the way in which the atoms of the same element are arranged in space. The atoms of the same element are held together by the same force, called attraction. The atoms of different elements are held apart by different forces, cal
 ```
 
 ## Limitations