File size: 2,037 Bytes
4b750ba
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
# NOTICE — Neuratal NTokenizer v0.3-polished model

This NOTICE covers the attribution and ShareAlike terms attached to the model
artifact `ntokenizer-0.3-256k.ntok` and to the `tokenizer.json` export included in
this repository.

1. The model is a machine-learned derivative of a 1.96 GB training corpus dominated
   by CC BY-SA/GFDL text from Wikimedia projects (simplewiki, trwiki, hiwiki,
   eswikisource, arwikisource, enwikibooks, jawikisource, kowikisource, zhwiktionary,
   jawiktionary, kowiktionary, trwiktionary and others).

2. It is therefore distributed under **CC BY-SA 4.0**, not Apache-2.0.
   ShareAlike applies to adaptations of the model.

3. Attribution pattern: `"<project> contributors, <license>, <URL>"`. Example:
   "Simple English Wikipedia contributors, CC BY-SA 4.0, https://simple.wikipedia.org".
   The complete per-source table (dumps, licenses, URLs, SHA-256) is in
   `data/v0.2/LICENSES.md` of the source repository
   (<https://github.com/Neuratal/neuratal-tokenizer>) and summarized in this repo's
   `training_manifest.json` (fingerprints only; no corpus text included).

4. CC BY sources (arXiv per-article CC BY 4.0 — 187 papers; enwikinews CC BY
   2.5/4.0; Tatoeba CC BY 2.0 FR) require attribution, no ShareAlike.

5. Public-domain sources (Project Gutenberg) and permissively licensed code
   (CPython PSF 2.0, TypeScript Apache-2.0, Go BSD-3-Clause, Rust MIT OR Apache-2.0)
   impose only notice retention, satisfied by this NOTICE.

6. No training corpus is distributed. No benchmark file is distributed. The
   benchmark used in evaluation is a development/validation set.

7. Founder and creator: Rüzgar Erdem. Neuratal is an independent AI research and
   software project; it is not claimed to be an incorporated company. The model
   is an independent implementation; no third-party tokenizer vocabulary, merge
   table, or implementation was used.

8. If you cannot comply with CC BY-SA 4.0, do not redistribute the model.
   The NTokenizer source code remains MIT OR Apache-2.0.