Crawlnet commited on
Commit
ffda29d
·
verified ·
1 Parent(s): 1a4f94d

card: no cost row (it only counted the final run GPU hours)

Browse files
Files changed (3) hide show
  1. README.md +0 -1
  2. SHA256SUMS +2 -2
  3. version.json +1 -2
README.md CHANGED
@@ -25,7 +25,6 @@ A small language model trained **from scratch**. **Pretraining data: pages the C
25
  | validation loss | 0.8304 bits/byte on the pages of 169 held-out sites, never trained on (pretraining; earlier versions held out other pages, so their numbers do not compare) |
26
  | chat tuning (SFT) | `synthetic+openbook`: 2,390,706 question/answer pairs **generated by a general-purpose LLM** from 198,882 of 381,992 page chunks (52%; crawler pages and record pages): 792,707 written for Queen v4's chat tuning and 1,597,999 from the FAQ and Q&A of the study notes (see Pretraining data), plus the pairs cut from the pages. The generator was told to use only facts on each page, and a pair whose key words are not on its page was dropped, but its wording is in this model's chat behaviour. `headings`: 683,478 question/answer pairs cut out of the crawler pages themselves (FAQ questions found on pages, page headings, page titles; answers are verbatim page text). No external model was used. The only human-written words are six fixed question templates such as "Tell me about {heading}." Open book (RAFT-style): 211,975 conversations built from the generated pairs, with 1-6 passages in front of the question (its own page, unless left out, and pages the site's search finds for it); the reply copies a sentence from a passage, then answers (`Quote: "<sentence>"` / `Answer: <answer>`); in 15% none of the passages holds the answer and the reply is "The pages don't say."; in 3% one passage is a copy of the source page with its key fact changed, and the reply follows the trustworthy page or says "The pages disagree." The chat tuning trained on 2,342,880 generated pairs, 211,975 open-book conversations and 683,478 headings pairs, 1 epoch. |
27
  | compute | 4x H100 SXM (rented), 1.272 h wall, 5.09 GPU-hours |
28
- | cost | $9.20 GPU |
29
  | weights | this folder |
30
 
31
  ## Evaluation
 
25
  | validation loss | 0.8304 bits/byte on the pages of 169 held-out sites, never trained on (pretraining; earlier versions held out other pages, so their numbers do not compare) |
26
  | chat tuning (SFT) | `synthetic+openbook`: 2,390,706 question/answer pairs **generated by a general-purpose LLM** from 198,882 of 381,992 page chunks (52%; crawler pages and record pages): 792,707 written for Queen v4's chat tuning and 1,597,999 from the FAQ and Q&A of the study notes (see Pretraining data), plus the pairs cut from the pages. The generator was told to use only facts on each page, and a pair whose key words are not on its page was dropped, but its wording is in this model's chat behaviour. `headings`: 683,478 question/answer pairs cut out of the crawler pages themselves (FAQ questions found on pages, page headings, page titles; answers are verbatim page text). No external model was used. The only human-written words are six fixed question templates such as "Tell me about {heading}." Open book (RAFT-style): 211,975 conversations built from the generated pairs, with 1-6 passages in front of the question (its own page, unless left out, and pages the site's search finds for it); the reply copies a sentence from a passage, then answers (`Quote: "<sentence>"` / `Answer: <answer>`); in 15% none of the passages holds the answer and the reply is "The pages don't say."; in 3% one passage is a copy of the source page with its key fact changed, and the reply follows the trustworthy page or says "The pages disagree." The chat tuning trained on 2,342,880 generated pairs, 211,975 open-book conversations and 683,478 headings pairs, 1 epoch. |
27
  | compute | 4x H100 SXM (rented), 1.272 h wall, 5.09 GPU-hours |
 
28
  | weights | this folder |
29
 
30
  ## Evaluation
SHA256SUMS CHANGED
@@ -1,4 +1,4 @@
1
- a81731660bca11c53ad24f52885f7225b0a1e527388d4a0f81491b4edfd0b850 README.md
2
  a3abb5028b28c5012f2235463bc4dcd65fc114a49da1a569df63b47b41c3aea9 chat.py
3
  c92c6d490f4d9c47fc44ab0a7c3461a6816632ea4e14e1ce6d83e867dfbb2d66 config.json
4
  6fc5201f59e6c8aa5112d0a807249686598fe6ca7acc1d464961034062f9a351 contributors.json
@@ -11,4 +11,4 @@ f27f347de0f7734e79f88735b86bfd5654ff1e34c908ae2674c9693a920f289a samples.json
11
  43a9a1532ce197532b7c0824f653a0db43f7c714ce7d3185b4e7e363a82758c7 tokenizer.json
12
  9a3eac141d0a43f2cb69e64a59757809ff83d49072d70678d61b5caee99fee8b tokenizer.tiktoken
13
  2c902d8df9bc9986b3baf9fae165eefefcafd4649a98095829ee12e6eba5edd8 train_summary.json
14
- 5a6e2e8add0102cba1613198ef93cf7c9a62f3e4a647a6826044bd0141f08ba3 version.json
 
1
+ 6a3ea625d8c4df817c4301ed44f89fb940f129a8a209d7a558999f9dc7820036 README.md
2
  a3abb5028b28c5012f2235463bc4dcd65fc114a49da1a569df63b47b41c3aea9 chat.py
3
  c92c6d490f4d9c47fc44ab0a7c3461a6816632ea4e14e1ce6d83e867dfbb2d66 config.json
4
  6fc5201f59e6c8aa5112d0a807249686598fe6ca7acc1d464961034062f9a351 contributors.json
 
11
  43a9a1532ce197532b7c0824f653a0db43f7c714ce7d3185b4e7e363a82758c7 tokenizer.json
12
  9a3eac141d0a43f2cb69e64a59757809ff83d49072d70678d61b5caee99fee8b tokenizer.tiktoken
13
  2c902d8df9bc9986b3baf9fae165eefefcafd4649a98095829ee12e6eba5edd8 train_summary.json
14
+ 0a130031d416e591c2bcf994f14510fdc8846e7670487349b29446db672671b0 version.json
version.json CHANGED
@@ -12,7 +12,6 @@
12
  "valBpb": 0.8303629654584476,
13
  "sftMethod": "synthetic+openbook",
14
  "datasetManifestSha256": "be1297fa3c84851be7887c58d9ff594bf8ba9126ffdba6fbf39eb492452485b5",
15
- "costUsd": 9.2,
16
  "gpuHours": 5.09,
17
  "gpu": "4x H100 SXM (rented)",
18
  "nanochatCommit": "92d63d4e8bb4df75c3b71618f31ddde2378b2bcd",
@@ -29,4 +28,4 @@
29
  "what": "public market, funding and on-chain records (projects and their funding rounds, investors, token unlocks and token facts, DeFi protocols, chains and stablecoins, DEX and on-chain token statistics)"
30
  }
31
  }
32
- }
 
12
  "valBpb": 0.8303629654584476,
13
  "sftMethod": "synthetic+openbook",
14
  "datasetManifestSha256": "be1297fa3c84851be7887c58d9ff594bf8ba9126ffdba6fbf39eb492452485b5",
 
15
  "gpuHours": 5.09,
16
  "gpu": "4x H100 SXM (rented)",
17
  "nanochatCommit": "92d63d4e8bb4df75c3b71618f31ddde2378b2bcd",
 
28
  "what": "public market, funding and on-chain records (projects and their funding rounds, investors, token unlocks and token facts, DeFi protocols, chains and stablecoins, DEX and on-chain token statistics)"
29
  }
30
  }
31
+ }