YAML Metadata Warning:empty or missing yaml metadata in repo card

Check out the documentation for more information.

🌟 Supernova USLM - Ultra-Small Language Model

A 25 million parameter language model designed for efficient conversational AI with minimal computational requirements.

πŸš€ Features

  • Ultra-Compact: Only 25M parameters while maintaining conversational quality
  • Modern Architecture:
    • Grouped Query Attention (GQA) for efficiency
    • Rotary Position Embeddings (RoPE)
    • SwiGLU activation functions
    • Sliding window attention
    • RMS normalization
  • Conversation Fine-tuning: Specialized training for chat interactions
  • Efficient Generation: Advanced sampling strategies and stopping criteria
  • Easy to Use: Simple chat interface and training pipeline

πŸ“ Project Structure

USLM/
β”œβ”€β”€ supernova_model.py      # Core model architecture
β”œβ”€β”€ tokenizer.py           # Tokenization and text preprocessing
β”œβ”€β”€ chat_interface.py      # Interactive chat interface
β”œβ”€β”€ web_ui.py             # Modern web UI (DeepSeek-inspired)
β”œβ”€β”€ training.py           # Training pipeline for conversation fine-tuning
β”œβ”€β”€ demo.py              # Comprehensive demo script
β”œβ”€β”€ run_webui.py         # Web UI launcher script
β”œβ”€β”€ run_webui.bat        # Windows launcher for web UI
β”œβ”€β”€ safety_config.py     # Safety checks and company responses
β”œβ”€β”€ web_search.py        # Web search integration
β”œβ”€β”€ requirements.txt     # Python dependencies
└── README.md           # This file

⚑ Quick Start

1. Install Dependencies

pip install -r requirements.txt

2. Web UI (Recommended) 🌐

The easiest way to use Supernova:

# Launch the modern web interface
python run_webui.py

# Or on Windows:
run_webui.bat

This opens a beautiful DeepSeek-inspired web interface at http://localhost:8501 with:

  • 🎨 Modern dark theme with gradient styling
  • πŸ’¬ Real-time chat interface
  • βš™οΈ Adjustable generation settings
  • πŸ“Š Model information and system status
  • πŸ’Ύ Conversation saving/loading
  • πŸ”§ Easy model switching

3. Run the Demo

# Full demo (recommended for first time)
python demo.py

# Quick demo (faster)
python demo.py --quick

# Only chat interface
python demo.py --mode chat

# Only training demo
python demo.py --mode train

4. Interactive Chat (CLI)

# Direct CLI chat interface
python chat_interface.py

πŸ› οΈ Installation

Requirements

  • Python 3.8+
  • PyTorch 2.0+
  • CUDA (optional, for GPU acceleration)

Install from Requirements

pip install torch>=2.0.0 transformers>=4.30.0 datasets>=2.10.0 accelerate>=0.20.0 tokenizers>=0.13.0 numpy>=1.24.0 tqdm>=4.65.0 tensorboard>=2.13.0 scikit-learn>=1.2.0 sentencepiece>=0.1.99 wandb>=0.15.0 einops>=0.6.0 bitsandbytes>=0.40.0 peft>=0.4.0

πŸ“š Usage Examples

Basic Model Usage

from supernova_model import create_supernova_model
from tokenizer import SupernovaTokenizer

# Create model and tokenizer
model = create_supernova_model()
tokenizer = SupernovaTokenizer()

# Basic inference
text = "Hello, how are you?"
input_ids = tokenizer.encode(text)
# ... (see demo.py for complete example)

Chat Interface

from chat_interface import SupernovaChat

# Initialize chat
chat = SupernovaChat()

# Single interaction
response = chat.chat("What is machine learning?")
print(response)

# Interactive mode
chat.interactive_chat()

Web UI Usage

# Web UI runs automatically - just launch with:
# python run_webui.py

# Programmatic access to web UI components:
from web_ui import SupernovaWebUI

ui = SupernovaWebUI()
ui.run()  # Starts the Streamlit app

Training

from training import SupernovaTrainer, TrainingConfig

# Setup training configuration
config = TrainingConfig(
    batch_size=4,
    learning_rate=3e-5,
    max_epochs=3,
    output_dir="outputs"
)

# Train the model
trainer = SupernovaTrainer(config)
trainer.train()

🎯 Model Architecture

Supernova USLM uses a transformer decoder architecture optimized for efficiency:

  • Parameters: 25M total
  • Layers: 8 transformer blocks
  • Hidden Size: 768
  • Attention Heads: 12 (with 4 key-value heads for GQA)
  • Vocabulary: 32K tokens
  • Context Length: 2048 tokens
  • Sliding Window: 512 tokens for long sequences

Key Innovations

  1. Grouped Query Attention: Reduces memory usage by sharing key-value heads
  2. Partial Rotary Embeddings: Only 50% of dimensions use RoPE for efficiency
  3. SwiGLU Activation: More efficient than standard ReLU/GELU
  4. Sliding Window Attention: Handles longer contexts efficiently
  5. Conversation-Specific Training: Loss masking for chat fine-tuning

πŸš‚ Training

Data Format

The model expects conversation data in this format:

{
  "messages": [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "What is AI?"},
    {"role": "assistant", "content": "AI is artificial intelligence..."}
  ]
}

Training Configuration

config = TrainingConfig(
    model_name="supernova-chat",
    batch_size=4,
    gradient_accumulation_steps=4,
    learning_rate=3e-5,
    max_epochs=3,
    max_sequence_length=1024,
    mixed_precision=True,
    mask_user_tokens=True  # Only train on assistant responses
)

Training Pipeline

  1. Data Preparation: Converts conversations to training format
  2. Loss Masking: Only trains on assistant responses
  3. Mixed Precision: Faster training with FP16
  4. Gradient Accumulation: Effective larger batch sizes
  5. Cosine Annealing: Learning rate scheduling

🌐 Web UI Features

The Supernova Web UI provides a modern, user-friendly interface:

🎨 Design Features

  • DeepSeek-inspired theme: Dark mode with beautiful gradients
  • Responsive layout: Works on desktop, tablet, and mobile
  • Real-time chat: Instant message display with typing indicators
  • Smooth animations: Hover effects and transitions

βš™οΈ Functionality

  • Model status: Live model information and GPU memory usage
  • Generation settings: Adjustable temperature, top-k, top-p, max tokens
  • Quick actions: Pre-built prompts for common questions
  • Conversation management: Save/load chat history
  • Safety integration: Built-in content filtering
  • Web search: Live search integration (when API key provided)

πŸš€ Quick Actions

  • "Who are you?" - Learn about Supernova
  • "Tell me about AlgoRythm" - Company information
  • "Explain AI" - AI education
  • "How do you work?" - Technical details

πŸ“± Mobile Friendly

The web UI automatically adapts to different screen sizes for optimal mobile experience.

πŸ’¬ Chat Features

Special Commands

  • reset - Clear conversation history
  • system <prompt> - Change system prompt
  • save <filename> - Save conversation
  • load <filename> - Load conversation
  • quit / exit / bye - End chat

Generation Parameters

response = chat.generate_response(
    user_input="Hello!",
    temperature=0.7,        # Randomness
    top_k=40,              # Top-k filtering
    top_p=0.9,             # Nucleus filtering
    repetition_penalty=1.1, # Reduce repetition
    max_new_tokens=256     # Response length
)

βš™οΈ Configuration

Model Configuration

from supernova_model import SupernovaConfig

config = SupernovaConfig(
    vocab_size=32000,
    hidden_size=768,
    num_hidden_layers=8,
    num_attention_heads=12,
    num_key_value_heads=4,
    intermediate_size=2048,
    max_position_embeddings=2048,
    use_sliding_window=True,
    sliding_window_size=512
)

Generation Configuration

from chat_interface import GenerationConfig

gen_config = GenerationConfig(
    temperature=0.7,
    top_k=40,
    top_p=0.9,
    repetition_penalty=1.1,
    max_new_tokens=256,
    do_sample=True
)

πŸ” Model Performance

Specifications

  • Parameters: 25,165,824 (25.2M)
  • Model Size: ~100MB (FP32), ~50MB (FP16)
  • Memory Usage: ~1GB GPU memory for inference
  • Speed: 20-50 tokens/sec on modern GPUs
  • Context: 2048 tokens with sliding window support

Efficiency Optimizations

  1. GQA: 3x reduction in KV cache size
  2. Partial RoPE: 2x faster position encoding
  3. SwiGLU: 1.5x faster than standard FFN
  4. Mixed Precision: 2x faster training, 50% memory reduction
  5. Sliding Window: Constant memory for long sequences

πŸ§ͺ Testing

Run Tests

# Basic functionality test
python demo.py --mode test

# Training test
python demo.py --mode train --quick

# Chat test  
python demo.py --mode chat

Performance Benchmarks

# Benchmark generation speed
from chat_interface import SupernovaChat
import time

chat = SupernovaChat()
start = time.time()
response = chat.chat("Write a short story about AI.")
end = time.time()
print(f"Generated in {end-start:.2f}s")

πŸš€ Deployment

CPU Deployment

chat = SupernovaChat(device="cpu")

GPU Deployment

chat = SupernovaChat(device="cuda")  # or device="auto"

Model Saving/Loading

# Save trained model
trainer.save_model("my_model")

# Load trained model
chat = SupernovaChat(model_path="my_model")

πŸ“Š Monitoring

Training Logs

Training automatically logs to:

  • Console output
  • outputs/training.log
  • TensorBoard (if enabled)
  • Weights & Biases (if configured)

Model Checkpoints

  • best_model/ - Best validation loss
  • final_model/ - Final training state
  • checkpoint-{step}/ - Regular checkpoints

🀝 Contributing

  1. Fork the repository
  2. Create a feature branch
  3. Make changes with tests
  4. Submit a pull request

Development Setup

git clone <repository>
cd USLM
pip install -r requirements.txt
python demo.py --mode test

πŸ“„ License

This project is open source. See LICENSE file for details.

🎯 Future Improvements

  • Support for additional tokenizers (SentencePiece, etc.)
  • Quantization support (4-bit, 8-bit)
  • ONNX export for deployment
  • Fine-tuning on larger conversation datasets
  • Multi-modal capabilities
  • Streaming generation API
  • Docker containerization

πŸ† Acknowledgments

  • Transformer architecture from "Attention Is All You Need"
  • RoPE from "RoFormer: Enhanced Transformer with Rotary Position Embedding"
  • SwiGLU from "GLU Variants Improve Transformer"
  • GQA from "GQA: Training Generalized Multi-Query Transformer"

πŸ“ž Support

For questions, issues, or contributions:

  1. Check existing GitHub issues
  2. Create a new issue with detailed description
  3. Join discussions in GitHub Discussions

Happy chatting with Supernova USLM! 🌟

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ 1 Ask for provider support