### LGeM 🚀
- what is LGeM, LGeM is a CausalLM Model that is trained on self instruct data (Alpaca data), and for initialization of the first train of main model (weights are available) I used pre-weights from Alpaca LoRA (open source)
- it's Decoder Only
- built-in Pytorch
- you can simply import models like
```python
from modules import LGeMForCausalLM
```
- and Training code is available at LGeM-Train.py (check source)
- training parameters
- - learning rate 1e-4
- - AdamW (weight decay 1e-2)
- - batch 2
- - A 100 80GB used for training (4 X )
``` shell
python3 LGeM-train.py
```
I used Alpaca Prompting Method
```python
def prompt_to_instruction(instruction, input_=None, response_=None, eos='<|endoftext|>'):
if input_ is None:
st1_prompting = f'Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\n\n{instruction}\n\n'
else:
st1_prompting = f'Below is an instruction that describ…
