| --- |
| language: en |
| license: mit |
| tags: |
| - text-generation |
| - custom-transformer |
| - neora |
| model_name: "Neora 1.0 Basic" |
| --- |
| |
| # Neora 1.0 Basic |
|
|
| Neora 1.0 Basic is a custom-designed ~505 Million parameters transformer model pretrained on general language resources using a vocab size of 24,000 and 24 structural transformer layers. |
|
|
| ## Model Details |
| - **Model Name:** Neora 1.0 Basic |
| - **Parameters:** ~502,640,640 (~505M with embeddings) |
| - **Architecture:** Custom Causal Attention Transformer |
| - **Number of Layers:** 24 |
| - **Attention Heads:** 20 |
| - **Hidden Dimension (D_MODEL):** 1280 |
| - **Feedforward Dimension (FFN_DIM):** 5120 |
| - **Vocabulary Size:** 24,000 |
| - **Context Window Length:** 2048 |
|
|
| ## Python Inference Example |
| ```python |
| import torch |
| from transformers import AutoModel, AutoConfig |
| from tokenizers import Tokenizer |
| |
| # Load config and custom model dynamically |
| model = AutoModel.from_pretrained("Neora-1.0-Basic", trust_remote_code=True) |
| tokenizer = Tokenizer.from_file("tokenizer.json") |
| |
| prompt = "The future of artificial intelligence is" |
| input_ids = torch.tensor([tokenizer.encode(prompt).ids]) |
| |
| with torch.no_grad(): |
| outputs = model(input_ids) |
| ``` |
|
|
| ## Limitations & Disclaimer |
| This is a verification/developmental base run checkpoint (step 20). Outputs are non-converged and exploratory. Use with `trust_remote_code=True` required. |
|
|
