Instructions to use AkshithAI/project-828-gpt-base with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use AkshithAI/project-828-gpt-base with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="AkshithAI/project-828-gpt-base", trust_remote_code=True)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("AkshithAI/project-828-gpt-base", trust_remote_code=True, device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use AkshithAI/project-828-gpt-base with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "AkshithAI/project-828-gpt-base"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "AkshithAI/project-828-gpt-base",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/AkshithAI/project-828-gpt-base

SGLang

How to use AkshithAI/project-828-gpt-base with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "AkshithAI/project-828-gpt-base" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "AkshithAI/project-828-gpt-base",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "AkshithAI/project-828-gpt-base" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "AkshithAI/project-828-gpt-base",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use AkshithAI/project-828-gpt-base with Docker Model Runner:
```
docker model run hf.co/AkshithAI/project-828-gpt-base
```

project-828-gpt-base / README.md

AkshithAI

Update model card with new repo id

9941aae verified 3 months ago

preview code

Raw History Blame Contribute Delete

4.57 kB

metadata

language: en
library_name: transformers
license: apache-2.0
model_name: project-828-gpt-base
pipeline_tag: text-generation
tags:
  - code
  - moe
  - mixture-of-experts
  - flash-attention
  - custom-architecture

project-828-gpt-base

A 398.7M parameter Mixture-of-Experts (MoE) causal language model optimized for code generation and technical reasoning, pretrained on ~60B tokens.

398.7M total parameters, ~286M active per token — 4 routed experts + 1 shared expert with top-2 routing per token.

Model Architecture

Property	Value
Architecture	Decoder-only Transformer with MoE FFN layers
Total Parameters	398.7M
Active Parameters / Token	~286M
Hidden Dimension ($d_{\text{model}}$)	768
Intermediate Size ($d_{\text{ff}}$)	760
Hidden Layers	24
Attention Heads / KV Heads	12 / 6 (GQA 2:1)
Head Dimension	64
Routed Experts	4
Active Experts / Token	2 (Top-2)
Shared Experts	1
Context Length	2048 (extensible to 8192 via YaRN)
Vocabulary Size	49152
Precision	BFloat16 Mixed Precision
Positional Encoding	RoPE with YaRN scaling support

Parameter Breakdown

Component	Parameters
Embeddings	~37.7M
Unembedding	~37.8M
Attention (×24 layers)	~1.8M each
MoE FFN (×24 layers)	~11.7M each
Layer Norms + Misc	~0.07M
Total	398.7M

Key Architectural Features

Grouped Query Attention (GQA): 12 query heads with 6 key-value heads (2:1 ratio) for memory-efficient attention
QK-Norm: RMSNorm applied to query and key projections before RoPE for attention stability
Auxiliary-Loss-Free MoE Routing: Sigmoid gating with dynamic bias adjustment (DeepSeek-V3 paper) — achieves near-perfect ~25% utilization per expert without auxiliary losses
SwiGLU Activation with Soft-Clamping: limit=7.0 prevents activation explosions during long training runs
Batched Expert Dispatch: Sort-and-slice dispatch with searchsorted boundaries for contiguous memory access
RoPE with YaRN Extension: Base context of 2048 tokens, extensible to 8192 via YaRN scaling

Training Details

Phase 1 — Pretraining (~60B tokens)

Training Config	Value
Hardware	H200 GPU
Peak Learning Rate	3e-4
Min Learning Rate	3e-5
Scheduler	WSD (Warmup-Stable-Decay)
Warmup Steps	500
Total Steps	101,726
Effective Batch Size	37 × 8 = 296 sequences
Tokens per Step	~0.61M
Gradient Clipping	1.0

Training Data Mix

Dataset	Weight	Category
`starcoderdata` — Python	14	Source Code
`starcoderdata` — JavaScript	8	Source Code
`starcoderdata` — Java	6	Source Code
`starcoderdata` — TypeScript	4	Source Code
`starcoderdata` — C++	6	Source Code
`starcoderdata` — C	4	Source Code
`starcoderdata` — C#	3	Source Code
`starcoderdata` — Go	4	Source Code
`starcoderdata` — Rust	3	Source Code
`starcoderdata` — PHP	3	Source Code
`fineweb-edu-dedup`	20	General Knowledge
`cosmopedia-v2`	7	General Knowledge
`wikipedia-en`	3	General Knowledge
`finemath-4plus`	8	Math / Reasoning
`stackexchange` (programming/CS)	7	CS / Engineering

Category breakdown: Source Code 55% · General Knowledge 30% · Math/Reasoning 8% · CS/Engineering 7%

Quickstart

from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "AkshithAI/project-828-gpt-base"

tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    repo_id,
    torch_dtype="bfloat16",
    device_map="auto",
    trust_remote_code=True
)

prompt = "def binary_search(arr, target):"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=200,
    temperature=0.7,
    top_p=0.9,
    do_sample=True,
    use_cache=False,
    eos_token_id=tokenizer.eos_token_id,
    pad_token_id=tokenizer.pad_token_id
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Citation

@misc{project828,
  author = {AkshithAI},
  title = {Project 828: MoE Transformer with Training Pipeline},
  year = {2025},
  publisher = {GitHub},
  url = {https://github.com/AkshithAI/project_828}
}