---
language: en
license: mit
tags:
- text-generation
- custom-transformer
- neora
model_name: "Neora 1.0 Basic"
---

# Neora 1.0 Basic

Neora 1.0 Basic is a custom-designed ~505 Million parameters transformer model pretrained on general language resources using a vocab size of 24,000 and 24 structural transformer layers.

## Model Details
- **Model Name:** Neora 1.0 Basic
- **Parameters:** ~502,640,640 (~505M with embeddings)
- **Architecture:** Custom Causal Attention Transformer
- **Number of Layers:** 24
- **Attention Heads:** 20
- **Hidden Dimension (D_MODEL):** 1280
- **Feedforward Dimension (FFN_DIM):** 5120
- **Vocabulary Size:** 24,000
- **Context Window Length:** 2048

## Python Inference Example
```python
import torch
from transformers import AutoModel, AutoConfig
from tokenizers import Tokenizer

# Load config and custom model dynamically
model = AutoModel.from_pretrained("Neora-1.0-Basic", trust_remote_code=True)
tokenizer = Tokenizer.from_file("tokenizer.json")

prompt = "The future of artificial intelligence is"
input_ids = torch.tensor([tokenizer.encode(prompt).ids])

with torch.no_grad():
    outputs = model(input_ids)
```

## Limitations & Disclaimer
This is a verification/developmental base run checkpoint (step 20). Outputs are non-converged and exploratory. Use with `trust_remote_code=True` required.