README.md

---
license: apache-2.0
tags:
- text-generation
- language-model
- causal-lm
- cosmicfish
- 90m
- transformer
- rope
- gqa
- swiglu
- rmsnorm
language: en
datasets:
- CosmicSet-2.0-mini
- akkiisfrommars/TreeCorpusCleanedmodel
model_type: CosmicFish
pipeline_tag: text-generation
---

# CosmicFish-90M

A 90M parameter language model with modern architecture improvements developed by Mistyoz AI.

## Quick Start

**The easiest way to chat with CosmicFish is using our chat.py script:**

```bash
# Download the chat script from this repository
wget https://huggingface.co/MistyozAI/CosmicFish-90M/resolve/main/chat.py

# Install dependencies
pip install transformers huggingface-hub termcolor safetensors

# Run the chat interface (automatically downloads model)
python chat.py
```

The `chat.py` script handles all model loading, generation, and provides the best chat experience with live streaming, repetition penalty, and conversation commands.

## Model Details

- **Parameters**: 91.6M
- **Architecture**: CosmicFish (RoPE, GQA, SwiGLU, RMSNorm)
- **Context Length**: 512 tokens
- **Vocabulary**: 50,257 tokens
- **Training Data**: CosmicSet 2.0 mini
- **Developer**: Mistyoz AI
- **Repository**: MistyozAI/CosmicFish-90M
- **Format**: Safetensors

## Usage

### Installation

```bash
pip install transformers huggingface-hub termcolor safetensors torch
```

### Quick Chat Interface

```python
from transformers import GPT2Tokenizer
from huggingface_hub import snapshot_download
from safetensors.torch import load_file
import torch
import json
import os

# Download model from Hugging Face Hub
cache_dir = snapshot_download(repo_id="MistyozAI/CosmicFish-90M")

# Load tokenizer
tokenizer = GPT2Tokenizer.from_pretrained("gpt2")

# Load config
with open(os.path.join(cache_dir, "config.json")) as f:
    config_dict = json.load(f)

# Load model weights from safetensors
state_dict = load_file(os.path.join(cache_dir, "model.safetensors"))

# Note: Full model class available in the repository
print("Model downloaded and ready for use!")
```

### Advanced Generation with Repetition Penalty

```python
def generate_with_repetition_penalty(model, tokenizer, prompt, max_tokens=100, temperature=0.5, penalty=1.2):
    input_ids = torch.tensor(tokenizer.encode(prompt)).unsqueeze(0)
    generated = input_ids.clone()
    
    for _ in range(max_tokens):
        with torch.no_grad():
            logits, _ = model(generated)
        
        next_token_logits = logits[:, -1, :] / temperature
        
        # Apply repetition penalty
        if penalty > 1.0:
            for token_id in set(generated[0].tolist()):
                if next_token_logits[0, token_id] > 0:
                    next_token_logits[0, token_id] /= penalty
                else:
                    next_token_logits[0, token_id] *= penalty
        
        probs = torch.nn.functional.softmax(next_token_logits, dim=-1)
        next_token = torch.multinomial(probs, num_samples=1)
        
        if next_token.item() == tokenizer.eos_token_id:
            break
            
        generated = torch.cat([generated, next_token], dim=1)
    
    return tokenizer.decode(generated[0], skip_special_tokens=True)
```

### Loading Model with Safetensors

```python
from safetensors.torch import load_file
from modeling_cosmicfish import CosmicFish, CosmicConfig
import json

def load_cosmicfish_model(model_path):
    # Load config
    with open(os.path.join(model_path, "config.json")) as f:
        config_dict = json.load(f)
    
    # Create model config
    config = CosmicConfig(
        vocab_size=config_dict["vocab_size"],
        block_size=config_dict["block_size"], 
        n_layer=config_dict["n_layer"],
        n_head=config_dict["n_head"],
        n_embd=config_dict["n_embd"],
        bias=config_dict["bias"],
        dropout=0.0,
        use_rotary=config_dict["use_rotary"],
        use_swiglu=config_dict["use_swiglu"],
        use_gqa=config_dict["use_gqa"],
        n_query_groups=config_dict["n_query_groups"]
    )
    
    # Create model
    model = CosmicFish(config)
    
    # Load weights from safetensors (secure format)
    state_dict = load_file(os.path.join(model_path, "model.safetensors"))
    
    # Handle weight sharing (lm_head.weight shares with transformer.wte.weight)
    if 'lm_head.weight' not in state_dict and 'transformer.wte.weight' in state_dict:
        state_dict['lm_head.weight'] = state_dict['transformer.wte.weight']
    
    model.load_state_dict(state_dict)
    model.eval()
    
    return model
```

### Chat Interface

```python
def chat_with_model():
    conversation = []
    
    while True:
        user_input = input("You: ")
        if user_input.lower() in ['quit', 'exit']:
            break
        
        context = "Below is a conversation between a human and an AI assistant.\n\n"
        for human, ai in conversation:
            context += f"Human: {human}\nAssistant: {ai}\n\n"
        context += f"Human: {user_input}\nAssistant:"
        
        # Generate response with repetition penalty
        response = generate_with_repetition_penalty(
            model, tokenizer, context, 
            max_tokens=150, temperature=0.7, penalty=1.2
        )
        
        # Extract just the assistant's response
        response = response.split("Assistant:")[-1].split('\n')[0].strip()
        print(f"CosmicFish: {response}")
        
        conversation.append((user_input, response))

chat_with_model()
```

## Architecture

CosmicFish uses several modern improvements over standard transformers:

- **RoPE (Rotary Position Embeddings)**: Better position encoding than absolute positions
- **GQA (Grouped-Query Attention)**: Reduces memory usage with 4 query groups 
- **SwiGLU**: More effective activation function than ReLU/GELU
- **RMSNorm**: Simpler, more stable normalization than LayerNorm

## Training

- **Dataset**: CosmicSet 2.0 mini
- **Sequence Length**: 512 tokens
- **Training Steps**: ~200K iterations
- **Hardware**: Nvidia A40 x1

## Performance

- **Speed**: Varies by hardware (not benchmarked)
- **Memory**: ~256MB RAM
- **File Size**: 185MB
- **Loading**: Fast and secure with safetensors

## Limitations

- Small model size (90M parameters) may produce less accurate responses
- 512 token context limit
- English only
- Training data cutoff applies
- May generate incorrect information
- Cannot browse internet or access real-time data

## License

Apache 2.0 - see LICENSE file.

## Credit

If you use CosmicFish-90M, please credit Mistyoz AI.