card: no cost row (it only counted the final run GPU hours)
Browse files- README.md +0 -1
- SHA256SUMS +2 -2
- version.json +1 -2
README.md
CHANGED
|
@@ -25,7 +25,6 @@ A small language model trained **from scratch**. **Pretraining data: pages the C
|
|
| 25 |
| validation loss | 0.8304 bits/byte on the pages of 169 held-out sites, never trained on (pretraining; earlier versions held out other pages, so their numbers do not compare) |
|
| 26 |
| chat tuning (SFT) | `synthetic+openbook`: 2,390,706 question/answer pairs **generated by a general-purpose LLM** from 198,882 of 381,992 page chunks (52%; crawler pages and record pages): 792,707 written for Queen v4's chat tuning and 1,597,999 from the FAQ and Q&A of the study notes (see Pretraining data), plus the pairs cut from the pages. The generator was told to use only facts on each page, and a pair whose key words are not on its page was dropped, but its wording is in this model's chat behaviour. `headings`: 683,478 question/answer pairs cut out of the crawler pages themselves (FAQ questions found on pages, page headings, page titles; answers are verbatim page text). No external model was used. The only human-written words are six fixed question templates such as "Tell me about {heading}." Open book (RAFT-style): 211,975 conversations built from the generated pairs, with 1-6 passages in front of the question (its own page, unless left out, and pages the site's search finds for it); the reply copies a sentence from a passage, then answers (`Quote: "<sentence>"` / `Answer: <answer>`); in 15% none of the passages holds the answer and the reply is "The pages don't say."; in 3% one passage is a copy of the source page with its key fact changed, and the reply follows the trustworthy page or says "The pages disagree." The chat tuning trained on 2,342,880 generated pairs, 211,975 open-book conversations and 683,478 headings pairs, 1 epoch. |
|
| 27 |
| compute | 4x H100 SXM (rented), 1.272 h wall, 5.09 GPU-hours |
|
| 28 |
-
| cost | $9.20 GPU |
|
| 29 |
| weights | this folder |
|
| 30 |
|
| 31 |
## Evaluation
|
|
|
|
| 25 |
| validation loss | 0.8304 bits/byte on the pages of 169 held-out sites, never trained on (pretraining; earlier versions held out other pages, so their numbers do not compare) |
|
| 26 |
| chat tuning (SFT) | `synthetic+openbook`: 2,390,706 question/answer pairs **generated by a general-purpose LLM** from 198,882 of 381,992 page chunks (52%; crawler pages and record pages): 792,707 written for Queen v4's chat tuning and 1,597,999 from the FAQ and Q&A of the study notes (see Pretraining data), plus the pairs cut from the pages. The generator was told to use only facts on each page, and a pair whose key words are not on its page was dropped, but its wording is in this model's chat behaviour. `headings`: 683,478 question/answer pairs cut out of the crawler pages themselves (FAQ questions found on pages, page headings, page titles; answers are verbatim page text). No external model was used. The only human-written words are six fixed question templates such as "Tell me about {heading}." Open book (RAFT-style): 211,975 conversations built from the generated pairs, with 1-6 passages in front of the question (its own page, unless left out, and pages the site's search finds for it); the reply copies a sentence from a passage, then answers (`Quote: "<sentence>"` / `Answer: <answer>`); in 15% none of the passages holds the answer and the reply is "The pages don't say."; in 3% one passage is a copy of the source page with its key fact changed, and the reply follows the trustworthy page or says "The pages disagree." The chat tuning trained on 2,342,880 generated pairs, 211,975 open-book conversations and 683,478 headings pairs, 1 epoch. |
|
| 27 |
| compute | 4x H100 SXM (rented), 1.272 h wall, 5.09 GPU-hours |
|
|
|
|
| 28 |
| weights | this folder |
|
| 29 |
|
| 30 |
## Evaluation
|
SHA256SUMS
CHANGED
|
@@ -1,4 +1,4 @@
|
|
| 1 |
-
|
| 2 |
a3abb5028b28c5012f2235463bc4dcd65fc114a49da1a569df63b47b41c3aea9 chat.py
|
| 3 |
c92c6d490f4d9c47fc44ab0a7c3461a6816632ea4e14e1ce6d83e867dfbb2d66 config.json
|
| 4 |
6fc5201f59e6c8aa5112d0a807249686598fe6ca7acc1d464961034062f9a351 contributors.json
|
|
@@ -11,4 +11,4 @@ f27f347de0f7734e79f88735b86bfd5654ff1e34c908ae2674c9693a920f289a samples.json
|
|
| 11 |
43a9a1532ce197532b7c0824f653a0db43f7c714ce7d3185b4e7e363a82758c7 tokenizer.json
|
| 12 |
9a3eac141d0a43f2cb69e64a59757809ff83d49072d70678d61b5caee99fee8b tokenizer.tiktoken
|
| 13 |
2c902d8df9bc9986b3baf9fae165eefefcafd4649a98095829ee12e6eba5edd8 train_summary.json
|
| 14 |
-
|
|
|
|
| 1 |
+
6a3ea625d8c4df817c4301ed44f89fb940f129a8a209d7a558999f9dc7820036 README.md
|
| 2 |
a3abb5028b28c5012f2235463bc4dcd65fc114a49da1a569df63b47b41c3aea9 chat.py
|
| 3 |
c92c6d490f4d9c47fc44ab0a7c3461a6816632ea4e14e1ce6d83e867dfbb2d66 config.json
|
| 4 |
6fc5201f59e6c8aa5112d0a807249686598fe6ca7acc1d464961034062f9a351 contributors.json
|
|
|
|
| 11 |
43a9a1532ce197532b7c0824f653a0db43f7c714ce7d3185b4e7e363a82758c7 tokenizer.json
|
| 12 |
9a3eac141d0a43f2cb69e64a59757809ff83d49072d70678d61b5caee99fee8b tokenizer.tiktoken
|
| 13 |
2c902d8df9bc9986b3baf9fae165eefefcafd4649a98095829ee12e6eba5edd8 train_summary.json
|
| 14 |
+
0a130031d416e591c2bcf994f14510fdc8846e7670487349b29446db672671b0 version.json
|
version.json
CHANGED
|
@@ -12,7 +12,6 @@
|
|
| 12 |
"valBpb": 0.8303629654584476,
|
| 13 |
"sftMethod": "synthetic+openbook",
|
| 14 |
"datasetManifestSha256": "be1297fa3c84851be7887c58d9ff594bf8ba9126ffdba6fbf39eb492452485b5",
|
| 15 |
-
"costUsd": 9.2,
|
| 16 |
"gpuHours": 5.09,
|
| 17 |
"gpu": "4x H100 SXM (rented)",
|
| 18 |
"nanochatCommit": "92d63d4e8bb4df75c3b71618f31ddde2378b2bcd",
|
|
@@ -29,4 +28,4 @@
|
|
| 29 |
"what": "public market, funding and on-chain records (projects and their funding rounds, investors, token unlocks and token facts, DeFi protocols, chains and stablecoins, DEX and on-chain token statistics)"
|
| 30 |
}
|
| 31 |
}
|
| 32 |
-
}
|
|
|
|
| 12 |
"valBpb": 0.8303629654584476,
|
| 13 |
"sftMethod": "synthetic+openbook",
|
| 14 |
"datasetManifestSha256": "be1297fa3c84851be7887c58d9ff594bf8ba9126ffdba6fbf39eb492452485b5",
|
|
|
|
| 15 |
"gpuHours": 5.09,
|
| 16 |
"gpu": "4x H100 SXM (rented)",
|
| 17 |
"nanochatCommit": "92d63d4e8bb4df75c3b71618f31ddde2378b2bcd",
|
|
|
|
| 28 |
"what": "public market, funding and on-chain records (projects and their funding rounds, investors, token unlocks and token facts, DeFi protocols, chains and stablecoins, DEX and on-chain token statistics)"
|
| 29 |
}
|
| 30 |
}
|
| 31 |
+
}
|