Instructions to use MaagDeveloper/NeoLM-Small-Base with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use MaagDeveloper/NeoLM-Small-Base with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="MaagDeveloper/NeoLM-Small-Base", trust_remote_code=True)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("MaagDeveloper/NeoLM-Small-Base", trust_remote_code=True, device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use MaagDeveloper/NeoLM-Small-Base with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "MaagDeveloper/NeoLM-Small-Base"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "MaagDeveloper/NeoLM-Small-Base",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/MaagDeveloper/NeoLM-Small-Base

SGLang

How to use MaagDeveloper/NeoLM-Small-Base with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "MaagDeveloper/NeoLM-Small-Base" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "MaagDeveloper/NeoLM-Small-Base",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "MaagDeveloper/NeoLM-Small-Base" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "MaagDeveloper/NeoLM-Small-Base",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use MaagDeveloper/NeoLM-Small-Base with Docker Model Runner:
```
docker model run hf.co/MaagDeveloper/NeoLM-Small-Base
```

NeoLM-Small-Base / README.md

MaagDeveloper

docs: update model card for Foundation Continued 30K

8595c25 verified 6 days ago

preview code

Raw History Blame Contribute Delete

9.4 kB

---
language:
- en
tags:
- neolm
- causal-lm
- text-generation
- custom-code
- pretrained
- foundation
- pytorch
datasets:
- MaagDeveloper/NeoCorpus
pipeline_tag: text-generation
metrics:
- perplexity
library_name: transformers
---
# NeoLM-Small-Base (61M) — Stage 1: Foundation Continued 30K

**NeoLM-Small-Base** is an English-focused, decoder-only Transformer language model trained from scratch with PyTorch and Hugging Face Transformers. The `main` branch publishes the **Stage 1 Foundation Continued 30K** base-model checkpoint in **bfloat16** (`safetensors`) format. This is a **base next-token prediction model**, not an instruction-tuned chat assistant.

## Available revisions

| Revision | Description |
| --- | --- |
| `main` | Current stable Base weights: Foundation continued for 30K additional optimizer steps; BF16 release |
| `stage1-foundation` | Historical original Foundation checkpoint, preserved separately |
| `stage1-foundation-continued-30k` | Planned immutable tag for this 30K release; created **after** this README is committed and the release is verified |

The original Foundation checkpoint was trained for approximately **202K optimizer steps** before a system interruption. The selected continuation added **20K steps from that checkpoint**, followed by **10K more steps from the 20K checkpoint** (approximately **232K steps total along the selected training trajectory**). Independent 5K and 20K continuation experiments were also performed; they should not be summed together as one run. The original planned three-epoch Foundation schedule was not completed.

## Architecture

| Property | Value |
| --- | --- |
| Parameters (unique, tied embeddings) | 61,356,032 |
| Layers | 12 |
| Hidden dimension | 512 |
| Query heads / KV heads | 8 / 2 (GQA) |
| MLP | SwiGLU, intermediate size 1,536 |
| Normalization | RMSNorm, QK Norm |
| Positional encoding | RoPE |
| Context length | 1,024 tokens |
| Vocabulary | 49,152 (custom ByteLevel BPE) |
| Special tokens | EOS ID 0, PAD ID 1; no BOS |
| Published weight format | BF16, safetensors |
| Model implementation | Custom `NeoLMForCausalLM` (`trust_remote_code=True`) |

## Tokenizer

NeoLM uses a **custom ByteLevel BPE tokenizer trained from scratch**, not a tokenizer borrowed from another model. It was trained exclusively on the **1,889,952 documents** in `NeoCorpus / foundation / train`; validation and test documents were excluded from tokenizer training.

| Property | Value |
| --- | --- |
| Algorithm | ByteLevel BPE |
| Vocabulary size | 49,152 |
| Training data | NeoCorpus / foundation / train |
| Training documents | 1,889,952 |
| Pre-tokenization | GPT-style regex with 1–3 digit groups |
| BOS token | None |

Special-token spellings and IDs (kept outside the Markdown table to preserve literal pipe characters):

- EOS: `<|endoftext|>` — ID **0**
- PAD: `<|pad|>` — ID **1**

The tokenizer passed local encode/decode round-trip, special-token, and save/reload checks. It can be loaded **without loading model weights**:

```python
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained(
    "MaagDeveloper/NeoLM-Small-Base",
    revision="main",
    trust_remote_code=True,
)
print(tokenizer.encode("Hello, NeoLM!"))
```

## Training data

Foundation pretraining uses the **`foundation`** configuration of [NeoCorpus](https://huggingface.co/datasets/MaagDeveloper/NeoCorpus), derived from a filtered English-focused subset of [Karpathy ClimbMix Shuffle](https://huggingface.co/datasets/karpathy/climbmix-400b-shuffle), based on [NVIDIA Nemotron-ClimbMix](https://huggingface.co/datasets/nvidia/Nemotron-ClimbMix). The training split contains **1,889,952 documents / 1,147,218,931 tokens**, counted with this release's tokenizer (including document EOS separators). The validation split contains **5,000 documents / 3,084,132 tokens**; the test split contains **20,000 documents / 12,094,582 tokens**. Validation and test were held out from pretraining. See the dataset card for preparation, attribution, and licensing details.

The tokenizer is trained on the Foundation training split, and pretraining consumes contiguous `uint16` token streams with EOS between documents and 1,024-token windows at stride 1,023 (one-token overlap).

## Evaluation

The **selected 30K continuation checkpoint** was evaluated on held-out Foundation token streams:

| Checkpoint | Full validation loss | Full validation PPL | Test loss | Test PPL |
| --- | ---: | ---: | ---: | ---: |
| Foundation continued 20K | 3.2685 | 26.27 | 3.2498 | 25.78 |
| **Foundation continued 30K** | **3.2677** | **26.25** | **3.2489** | **25.76** |

The 30K continuation improves the reported full validation loss over the 20K checkpoint by approximately **0.0008**; this is a small measured improvement, not evidence of a qualitative capability jump. Checkpoint selection should use validation rather than repeatedly consulted test results.

For the BF16 release, the model was exported separately from the FP32 master, reloaded, and used successfully for text generation. A local **256-block quick validation comparison**, using the same BF16 autocast compute settings for both weight versions, reported:

| Weight version | Quick validation loss | Quick validation PPL |
| --- | ---: | ---: |
| FP32 source | 3.236623 | 25.4476 |
| BF16 export | 3.236627 | 25.4477 |

The observed difference was **+0.000004 loss**. This is a *quick-subset parity check*, not a separate full-validation result or proof that every generated output will be identical. Published weights are BF16; the local FP32 master is retained separately.

## Load and generate text

Install compatible versions of `torch` and `transformers`. The model requires custom Python code, so review the repository code before enabling `trust_remote_code=True`. Use `revision="stage1-foundation"` to load the original historical release instead of the latest stable Base.

```python
from transformers import AutoTokenizer, AutoModelForCausalLM
repo_id = "MaagDeveloper/NeoLM-Small-Base"
revision = "main"
tokenizer = AutoTokenizer.from_pretrained(
    repo_id, revision=revision, trust_remote_code=True
)
model = AutoModelForCausalLM.from_pretrained(
    repo_id, revision=revision, trust_remote_code=True,
    dtype="auto",
).eval()
# For a CPU that does not support BF16 well, use dtype=torch.float32 instead.
prompt = "The future of artificial intelligence"
inputs = tokenizer(
    prompt, return_tensors="pt", truncation=True, max_length=1024
)
outputs = model.generate(
    **inputs,
    max_new_tokens=200,
    do_sample=True,
    temperature=0.3,
    top_p=0.9,
    repetition_penalty=1.15,
    eos_token_id=tokenizer.eos_token_id,
    pad_token_id=tokenizer.pad_token_id,
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

## Streaming Text Generation

Stream generated text progressively using Hugging Face Transformers' `TextStreamer`:

```python
from transformers import AutoTokenizer, AutoModelForCausalLM, TextStreamer
repo_id = "MaagDeveloper/NeoLM-Small-Base"
revision = "main"
tokenizer = AutoTokenizer.from_pretrained(
    repo_id,
    revision=revision,
    trust_remote_code=True,
)
model = AutoModelForCausalLM.from_pretrained(
    repo_id,
    revision=revision,
    trust_remote_code=True,
    dtype="auto",
).eval()
streamer = TextStreamer(
    tokenizer,
    skip_prompt=False,
    skip_special_tokens=True,
)
prompt = "Once upon a time"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
_ = model.generate(
    **inputs,
    streamer=streamer,
    max_new_tokens=100,
    do_sample=True,
    temperature=0.3,
    top_p=0.9,
    repetition_penalty=1.15,
    eos_token_id=tokenizer.eos_token_id,
    pad_token_id=tokenizer.pad_token_id,
)
```

Both examples are **text completion**, not role-based chat. For generation within the trained 1,024-token context, keep the total prompt-plus-generation length within the model's context window.

## Capabilities and limitations

The Base checkpoint can generate fluent English passages and frequently follows broad topical cues in its prompt. These observations are **not** evidence of reliable instruction following, robust reasoning, or factual accuracy. Exploratory generation also exhibits repetition, invented facts, inconsistent entities, topic drift, and occasional malformed endings.

The model is small and English-focused; it can hallucinate, reproduce undesirable patterns, generate harmful or biased material, and fail on multi-step reasoning, factual recall, long-range coherence, or strict output formats. Do not treat generated material as verified information. A future instruction-tuned model, if released, will live in a separate repository.

## Provenance and licensing

- Dataset: [MaagDeveloper/NeoCorpus](https://huggingface.co/datasets/MaagDeveloper/NeoCorpus), `foundation` configuration.
- NeoCorpus's attribution and CC BY-NC 4.0 notice do not automatically establish a separate license for model weights. **Verify the appropriate model-weight license and source-data obligations before commercial use.**
- `stage1-foundation` preserves the historical original release. The 30K release tag is intended to preserve a separate immutable weights-and-documentation snapshot after the README update.

NeoLM is an independent project and is not affiliated with NVIDIA, Andrej Karpathy, or Hugging Face.

	---
	language:
	- en
	tags:
	- neolm
	- causal-lm
	- text-generation
	- custom-code
	- pretrained
	- foundation
	- pytorch
	datasets:
	- MaagDeveloper/NeoCorpus
	pipeline_tag: text-generation
	metrics:
	- perplexity
	library_name: transformers
	---

	# NeoLM-Small-Base (61M) — Stage 1: Foundation Continued 30K

	NeoLM-Small-Base is an English-focused, decoder-only Transformer language model trained from scratch with PyTorch and Hugging Face Transformers. The `main` branch publishes the Stage 1 Foundation Continued 30K base-model checkpoint in bfloat16 (`safetensors`) format. This is a base next-token prediction model, not an instruction-tuned chat assistant.

	## Available revisions

	\| Revision \| Description \|
	\| --- \| --- \|
	\| `main` \| Current stable Base weights: Foundation continued for 30K additional optimizer steps; BF16 release \|
	\| `stage1-foundation` \| Historical original Foundation checkpoint, preserved separately \|
	\| `stage1-foundation-continued-30k` \| Planned immutable tag for this 30K release; created after this README is committed and the release is verified \|

	The original Foundation checkpoint was trained for approximately 202K optimizer steps before a system interruption. The selected continuation added 20K steps from that checkpoint, followed by 10K more steps from the 20K checkpoint (approximately 232K steps total along the selected training trajectory). Independent 5K and 20K continuation experiments were also performed; they should not be summed together as one run. The original planned three-epoch Foundation schedule was not completed.

	## Architecture

	\| Property \| Value \|
	\| --- \| --- \|
	\| Parameters (unique, tied embeddings) \| 61,356,032 \|
	\| Layers \| 12 \|
	\| Hidden dimension \| 512 \|
	\| Query heads / KV heads \| 8 / 2 (GQA) \|
	\| MLP \| SwiGLU, intermediate size 1,536 \|
	\| Normalization \| RMSNorm, QK Norm \|
	\| Positional encoding \| RoPE \|
	\| Context length \| 1,024 tokens \|
	\| Vocabulary \| 49,152 (custom ByteLevel BPE) \|
	\| Special tokens \| EOS ID 0, PAD ID 1; no BOS \|
	\| Published weight format \| BF16, safetensors \|
	\| Model implementation \| Custom `NeoLMForCausalLM` (`trust_remote_code=True`) \|

	## Tokenizer

	NeoLM uses a custom ByteLevel BPE tokenizer trained from scratch, not a tokenizer borrowed from another model. It was trained exclusively on the 1,889,952 documents in `NeoCorpus / foundation / train`; validation and test documents were excluded from tokenizer training.

	\| Property \| Value \|
	\| --- \| --- \|
	\| Algorithm \| ByteLevel BPE \|
	\| Vocabulary size \| 49,152 \|
	\| Training data \| NeoCorpus / foundation / train \|
	\| Training documents \| 1,889,952 \|
	\| Pre-tokenization \| GPT-style regex with 1–3 digit groups \|
	\| BOS token \| None \|

	Special-token spellings and IDs (kept outside the Markdown table to preserve literal pipe characters):

	- EOS: `<\|endoftext\|>` — ID 0
	- PAD: `<\|pad\|>` — ID 1

	The tokenizer passed local encode/decode round-trip, special-token, and save/reload checks. It can be loaded without loading model weights:

	```python
	from transformers import AutoTokenizer

	tokenizer = AutoTokenizer.from_pretrained(
	"MaagDeveloper/NeoLM-Small-Base",
	revision="main",
	trust_remote_code=True,
	)
	print(tokenizer.encode("Hello, NeoLM!"))
	```

	## Training data

	Foundation pretraining uses the `foundation` configuration of [NeoCorpus](https://huggingface.co/datasets/MaagDeveloper/NeoCorpus), derived from a filtered English-focused subset of [Karpathy ClimbMix Shuffle](https://huggingface.co/datasets/karpathy/climbmix-400b-shuffle), based on [NVIDIA Nemotron-ClimbMix](https://huggingface.co/datasets/nvidia/Nemotron-ClimbMix). The training split contains 1,889,952 documents / 1,147,218,931 tokens, counted with this release's tokenizer (including document EOS separators). The validation split contains 5,000 documents / 3,084,132 tokens; the test split contains 20,000 documents / 12,094,582 tokens. Validation and test were held out from pretraining. See the dataset card for preparation, attribution, and licensing details.

	The tokenizer is trained on the Foundation training split, and pretraining consumes contiguous `uint16` token streams with EOS between documents and 1,024-token windows at stride 1,023 (one-token overlap).

	## Evaluation

	The selected 30K continuation checkpoint was evaluated on held-out Foundation token streams:

	\| Checkpoint \| Full validation loss \| Full validation PPL \| Test loss \| Test PPL \|
	\| --- \| ---: \| ---: \| ---: \| ---: \|
	\| Foundation continued 20K \| 3.2685 \| 26.27 \| 3.2498 \| 25.78 \|
	\| Foundation continued 30K \| 3.2677 \| 26.25 \| 3.2489 \| 25.76 \|

	The 30K continuation improves the reported full validation loss over the 20K checkpoint by approximately 0.0008; this is a small measured improvement, not evidence of a qualitative capability jump. Checkpoint selection should use validation rather than repeatedly consulted test results.

	For the BF16 release, the model was exported separately from the FP32 master, reloaded, and used successfully for text generation. A local 256-block quick validation comparison, using the same BF16 autocast compute settings for both weight versions, reported:

	\| Weight version \| Quick validation loss \| Quick validation PPL \|
	\| --- \| ---: \| ---: \|
	\| FP32 source \| 3.236623 \| 25.4476 \|
	\| BF16 export \| 3.236627 \| 25.4477 \|

	The observed difference was +0.000004 loss. This is a quick-subset parity check, not a separate full-validation result or proof that every generated output will be identical. Published weights are BF16; the local FP32 master is retained separately.

	## Load and generate text

	Install compatible versions of `torch` and `transformers`. The model requires custom Python code, so review the repository code before enabling `trust_remote_code=True`. Use `revision="stage1-foundation"` to load the original historical release instead of the latest stable Base.

	```python
	from transformers import AutoTokenizer, AutoModelForCausalLM

	repo_id = "MaagDeveloper/NeoLM-Small-Base"
	revision = "main"

	tokenizer = AutoTokenizer.from_pretrained(
	repo_id, revision=revision, trust_remote_code=True
	)

	model = AutoModelForCausalLM.from_pretrained(
	repo_id, revision=revision, trust_remote_code=True,
	dtype="auto",
	).eval()

	# For a CPU that does not support BF16 well, use dtype=torch.float32 instead.
	prompt = "The future of artificial intelligence"

	inputs = tokenizer(
	prompt, return_tensors="pt", truncation=True, max_length=1024
	)

	outputs = model.generate(
	**inputs,
	max_new_tokens=200,
	do_sample=True,
	temperature=0.3,
	top_p=0.9,
	repetition_penalty=1.15,
	eos_token_id=tokenizer.eos_token_id,
	pad_token_id=tokenizer.pad_token_id,
	)

	print(tokenizer.decode(outputs[0], skip_special_tokens=True))
	```

	## Streaming Text Generation

	Stream generated text progressively using Hugging Face Transformers' `TextStreamer`:

	```python
	from transformers import AutoTokenizer, AutoModelForCausalLM, TextStreamer

	repo_id = "MaagDeveloper/NeoLM-Small-Base"
	revision = "main"

	tokenizer = AutoTokenizer.from_pretrained(
	repo_id,
	revision=revision,
	trust_remote_code=True,
	)

	model = AutoModelForCausalLM.from_pretrained(
	repo_id,
	revision=revision,
	trust_remote_code=True,
	dtype="auto",
	).eval()

	streamer = TextStreamer(
	tokenizer,
	skip_prompt=False,
	skip_special_tokens=True,
	)

	prompt = "Once upon a time"
	inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

	_ = model.generate(
	**inputs,
	streamer=streamer,
	max_new_tokens=100,
	do_sample=True,
	temperature=0.3,
	top_p=0.9,
	repetition_penalty=1.15,
	eos_token_id=tokenizer.eos_token_id,
	pad_token_id=tokenizer.pad_token_id,
	)
	```

	Both examples are text completion, not role-based chat. For generation within the trained 1,024-token context, keep the total prompt-plus-generation length within the model's context window.

	## Capabilities and limitations

	The Base checkpoint can generate fluent English passages and frequently follows broad topical cues in its prompt. These observations are not evidence of reliable instruction following, robust reasoning, or factual accuracy. Exploratory generation also exhibits repetition, invented facts, inconsistent entities, topic drift, and occasional malformed endings.

	The model is small and English-focused; it can hallucinate, reproduce undesirable patterns, generate harmful or biased material, and fail on multi-step reasoning, factual recall, long-range coherence, or strict output formats. Do not treat generated material as verified information. A future instruction-tuned model, if released, will live in a separate repository.

	## Provenance and licensing

	- Dataset: [MaagDeveloper/NeoCorpus](https://huggingface.co/datasets/MaagDeveloper/NeoCorpus), `foundation` configuration.
	- NeoCorpus's attribution and CC BY-NC 4.0 notice do not automatically establish a separate license for model weights. Verify the appropriate model-weight license and source-data obligations before commercial use.
	- `stage1-foundation` preserves the historical original release. The 30K release tag is intended to preserve a separate immutable weights-and-documentation snapshot after the README update.

	NeoLM is an independent project and is not affiliated with NVIDIA, Andrej Karpathy, or Hugging Face.