license: apache-2.0
datasets:
- CEAMFA/rewrite
details
bytebox is a model designed to change public perspective on byte-level language models. It's a deep model pretrained on roughly 1B tokens that demonstrates instruction-following skills with only 60m parameters. Training and inference codes are included.
To the author's knowledge, this is the first-ever decoder-only instruction pretrained byte-level transformer openly available to date.
python infer_bytefast60m.py --architecture-script bytefalcon_fast60m.py \
--model /bytefast-60m/ \
--tokenizer /byte-tokenizer/ \
--instruction "Fix the sentence" \
--text "this sentence need fixing you know i love it" \
--max-new-tokens 128 \
--temperature 0
Prompt
What is Python?
Output
Python is a high-level, general-purpose programming language. Its design philosophy emphasizes code readability, and its syntax allows programmers to express concepts in fewer lines of code than would be possible in languages such as C++ or Java. The language provides confidence towards the use of programming languages and their debugging problems."