Update model card

README.md

@@ -51,8 +51,8 @@ Parameter split: **41,943,040** embedding
 
 | | |
 |---|---|
-| Tokens seen | 39,847,512,064 |
-| Steps | 76,000 |
+| Tokens seen | 40,109,656,064 |
+| Steps | 76,500 |
 | Schedule | warmup-stable-decay (WSD) |
 | Optimizer | fused AdamW, betas (0.9, 0.95), wd 0.1 on 2D params only |
 | Precision | bf16 autocast, fp32 master weights |
@@ -68,8 +68,8 @@ directly comparable.
 | Metric | This model | GPT-2 124M |
 |---|---|---|
 | Parameters | 351,650,304 | 124,439,808 |
-| Validation loss | 1.5094 | — |
-| **Bits per byte** | **0.5115** | — |
+| Validation loss | 1.4946 | — |
+| **Bits per byte** | **0.5065** | — |
 | HellaSwag (0-shot, len-norm) | _pending_ | 29.55% |
 
 
@@ -113,48 +113,35 @@ and are short enough to read.
 ## Sample generations
 
 ```
-===== step 76000 =====
+===== step 76500 =====
 
 --- [greedy] 'Once upon a time'
-Once upon a time, the world was a place of peace and quiet. The world was a place of peace and quiet. The world was a place of peace and quiet. The world was a place of peace and quiet. The world was a place of peace and quiet. The world was a place of peace and quiet. The world was a place of peace and quiet. The world was a place of peace and quiet. The world was a place of peace and quiet. The world was a place of peace and quiet. The world was a place of peace and quiet. The world was a place of peace and quiet
+Once upon a time, the Company’s principal source of liquidity was cash on hand. The Company’s principal source of liquidity is cash flow from operations. The Company’s principal source of liquidity is cash flow from operations. The Company’s principal source of liquidity is cash flow from operations. The Company’s principal source of liquidity is cash flow from operations. The Company’s principal source of liquidity is cash flow from operations. The Company’s principal source of liquidity is cash flow from operations. The Company’s principal source of liquidity is cash flow from operations. The Company’s principal source of liquidity
 
 --- [t=0.8] 'Once upon a time'
-Once upon a time, the most valuable thing in the world was the fruit. When it became the fruit again, the only value in the world was that which it had become if it had not already become.
-- — — — — — — — — — — —
-So, in the above passage, we see that a fruit is a precious commodity and all that it has. Everything is valuable, no matter how valuable it is or how precious it is. Therefore, any commodity is value.
-As long as you have a good tree, there is value to the tree.
-And so, we
+Once upon a time I had been in possession of my body and my physical body had been restored to the body of a man, I never had any ill effects on others. I have received no injuries to the body or to any other body of men from that person, no hurt on the person, no injury to any thing, no hurt at any time.
+And in the mean time there have been no deaths, no any injuries, no any deaths.
+Now, if I had to die in another man, I would not have been in possession of my body, my physical body, and my physical body in
 
 --- [greedy] 'The capital of France is'
-The capital of France is divided into three classes of shares, Class A shares, Class B shares and Class C shares. Class A shares are entitled to one vote per share and Class B shares are entitled to ten votes per share. Class C shares are entitled to one vote per share and Class A shares are entitled to ten votes per share.
+The capital of France is divided into three classes of shares, Class A shares, Class B shares and Class C shares. Class A shares are entitled to one vote per share, Class B shares are entitled to ten votes per share, and Class C shares are entitled to one vote per share. The holders of Class A shares and Class B shares are entitled to share equally in the profits of the Company and dividends are declared and paid in accordance with the Company’s articles of association.
 
-The Company’s Articles of Association provide that the Company may issue shares of preferred stock in one or more series. The Board of Directors of the Company is authorized to amend the Company’s Articles of Association without shareholder approval. The Board of Directors of the Company is authorized to
+The Company has authorized and issued 1,000,000,000 shares of Class A shares, 1,000,000,000 shares
 
 --- [t=0.8] 'The capital of France is'
-The capital of France is considered to be a limited liability company under the French laws. Therefore, separate accounts of the limited liability company are not required because of the above.
-
-The Company’s registered address is located at “Hirac Plaza”, 10th floor, 12, rue de Rue de Rue de Grangville, 106000 Paris, France. The Company’s telephone number is +33 (0)1-33-83000-3450.
-
-The Company’s common stock is issued in registered form. The registrar and transfer agent is Vstock Transfer, LLC
+The capital of France is regulated by the Financial Services Authority (FSA). The FSA is responsible for supervising the financial activity of banks in France. The FSA’s primary regulator of banks in France is the Autorité des marchés financiers du Marche (AMF), a public corporation, and the bank’s legal name is Banco Bilbao Vizcaya Argentaria. The Autorité des marchés financiers du Marche is responsible for supervising the financial activities of banks in France. The Autorité des marchés financiers du Marche is similar
 
 --- [greedy] 'In 1969, humans first'
-In 1969, humans first began to use fire to cook their food. The first fire was a small piece of wood that was heated by a fireplace. The fire was placed in a fireplace and heated by a fireplace. The fire was then lit by a fireplace. The fire was then lit by a fireplace. The fire was then lit by a fireplace. The fire was then lit by a fireplace. The fire was then lit by a fireplace. The fire was then lit by a fireplace. The fire was then lit by a fireplace. The fire was then lit by a fireplace
+In 1969, humans first began to use the term “genotype” to describe the genetic makeup of an organism. In 1972, the term “genotype” was introduced to describe the genetic makeup of an organism. In 1973, the term “phenotype” was introduced to describe the genetic makeup of an organism. In 1976, the term “genotype” was introduced to describe the genetic makeup of an organism. In 1977, the term “genotype” was introduced to describe the genetic makeup of an organism. In 1978, the term “phenotype” was introduced to describe the genetic makeup of an organism. In 1979
 
 --- [t=0.8] 'In 1969, humans first'
-In 1969, humans first evolved along the coast of Africa and now live in every tropical island nation on the continents. Since then, humans have spread throughout the world, including the Hawaiian Islands, but no major population of humans exists on the islands. The Hawaiian islands are located on the tropical Pacific Ocean.
-The main population on the islands are Polynesians, who are descendants of the Polynesian people. The chief inhabitants of the islands are the Polynesians, who live primarily off of the plant resources of the islands.
-In addition to the Polynesians, there are also other populations scattered throughout the islands and
+In 1969, humans first became attracted to firewood in North America, following a banshi in Japan.
+The Japanese introduced the term “firewood” as a generic term in their language of Japan, and as a synonym for any wood that burns.
+In the early 1990s, the United States adopted the term “forest and woodland ecosystem,” which was adopted in part to address the problems caused by the deforestation and degradation of forests.
+The term “forest and woodland ecosystem” was used to refer to the rich natural ecosystems rich in biodiversity, including forests, wetlands, grasslands, shrublands, tund
 
 --- [greedy] 'The three states of matter are'
-The three states of matter are the same, but the states of matter are different.
-The state of matter is the state of being.
-The state of matter is the state of being.
-The state of matter is the state of being.
-The state of matter is the state of being.
-The state of matter is the state of being.
-The state of matter is the state of being.
-The state of matter is the state of being.
-The state of matter is the state of being
+The three states of matter are distinguished from each other primarily by their relative density. The density of a particular substance is the mass of that substance divided by the total mass of all the substances that are in that substance. For example, water has a density of 1.0 g/cm3, bu
 ```
 
 ## Limitations