| --- |
| pipeline_tag: text-generation |
| inference: true |
| widget: |
| - text: 'def print_hello_world():' |
| example_title: Hello world |
| group: Python |
| license: bigscience-openrail-m |
| datasets: |
| - books |
| - arxiv |
| - c4 |
| - falcon-refinedweb |
| - wiki |
| - github-issues |
| - stack_markdown |
| library_name: transformers |
| tags: |
| - code |
| language: |
| - en |
| --- |
| |
|  |
|
|
|
|
| # Refact-1.6B-base |
|
|
| Finally, the model we started training with our [blog post](https://refact.ai/blog/2023/applying-recent-innovations-to-train-model/) is ready 🎉 |
| The model might contain some problems, especially with the FIM format |
|
|
|
|
| # It Works As a Chat |
|
|
| The primary application of this model is code completion (infill) in multiple programming languages. |
| But it works as a chat quite well. |
|
|
|
|
| # Example |
|
|
| Fill-in-the-middle uses special tokens to identify the prefix/middle/suffix part of the input and output: |
|
|
| ```python |
| # pip install -q transformers |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| |
| checkpoint = "smallcloudai/Refact-1_6B-fim" |
| device = "cuda" # for GPU usage or "cpu" for CPU usage |
| |
| tokenizer = AutoTokenizer.from_pretrained(checkpoint) |
| model = AutoModelForCausalLM.from_pretrained(checkpoint, trust_remote_code=True).to(device) |
| |
| prompt = '<fim_prefix>def print_hello_world():\n """<fim_suffix>\n print("Hello world!")<fim_middle>' |
| |
| inputs = tokenizer.encode(prompt, return_tensors="pt").to(device) |
| outputs = model.generate(inputs, max_length=100, temperature=0.2) |
| print("-"*80) |
| print(tokenizer.decode(outputs[0])) |
| ``` |
|
|
| # Chat Format |
|
|
| The same model works as chat (experimental). |
|
|
| ```python |
| prompt_template = "<empty_output>SYSTEM {system}\n" \ |
| "<empty_output>USER {query}\n" \ |
| "<empty_output>ASSISTANT" |
| prompt = prompt_template.format(system="You are a programming assistant", |
| query="How do I sort a list in Python?") |
| ``` |
|
|
| # Architecture |
|
|
| As described in more detail in the blog post, we used: |
|
|
| - [ALiBi](https://arxiv.org/abs/2108.12409) based attention |
| - [LayerNorm](https://arxiv.org/abs/1607.06450v1) instead of [RMSNorm](https://arxiv.org/pdf/1910.07467.pdf) |
| - [Multi Query Attention](https://arxiv.org/abs/1911.02150) |
|
|
| We also used LiON, flash attention, early dropout. It's not that innovative that you can't run it, in fact you can -- see an example below. |
|
|
|
|
| # Training |
|
|
| For the base model, we used our own dataset that contains code with permissive licenses only, and open text datasets. |
| Filtering is the key to success of this model: |
|
|
| - We only used text in English |
| - Only topics related to computer science |
| - Applied heavy deduplication |
|
|
| The text to code proportion was 50:50, model trained for 1.2T tokens. |
|
|
| We don't release the base model, because its Fill-in-the-Middle (FIM) capability likes to repeat itself too much, so |
| its practical use is limited. But if you still want it, write us a message on Discord. |
|
|
|
|
| # Limitations and Bias |
|
|
| The Refact-1.6B model was trained on text in English. But it has seen a lot more languages in |
| code comments. Its performance on non-English languages is lower, for sure. |
|
|
|
|
| # Model Stats |
|
|
| - **Architecture:** LLAMA-like model with multi-query attention |
| - **Objectives** Fill-in-the-Middle, Chat |
| - **Tokens context:** 4096 |
| - **Pretraining tokens:** 1.2T |
| - **Finetuning tokens:** 40B |
| - **Precision:** bfloat16 |
| - **GPUs** 64 NVidia A5000 |
| - **Training time** 28 days |
|
|
|
|
| # License |
|
|
| The model is licensed under the BigScience OpenRAIL-M v1 license agreement |
|
|
|
|
| # Citation |
|
|
| If you are using this model, please give a link to this page. |
