tudor-iustin22/Bluebird-1.5-Instruct

Bluebird 1.5 Instruct

Compact instruction-following intelligence with long-range memory

Model repository  ·  Base model  ·  Apache 2.0  ·  Project Bluebird

Bluebird Highlights

• Instruction-tuned assistant: a 742.5M-parameter causal language model post-trained for conversational responses, instruction following, and task-oriented generation.
• 32K context window: support for up to 32,768 input tokens, with efficient access to earlier context through learned chunk summaries.
• Hybrid architecture: 16 local-attention blocks and eight memory-augmented blocks, with grouped-query attention and gated chunked memory.
• Dedicated chat format: system, user, and assistant roles with explicit message delimiters.
• Training: the Bluebird 1.5 base lineage records 23,000,514,560 pretraining tokens, followed by identity tuning and supervised instruction fine-tuning.
• Compute environment: PyTorch mixed-precision training using eight Intel Gaudi 3 AI accelerators.

Model Overview

Bluebird 1.5 Instruct is the instruction-following release of Bluebird 1.5 Base, developed under Project Bluebird. It builds on a compact decoder-only architecture that combines local causal attention with compressed representations of completed context chunks.

The Instruct variant adds supervised post-training for dialogue, assistant identity, and general instruction-following. It is designed for question answering, content generation, multi-turn conversation, summarisation, and related text tasks. It is a text-only model; the stated context window is a model capacity, not a guarantee of long-range recall accuracy.

Model specifications

Property	Specification
Model	tudor-iustin22/Bluebird-1.5-Instruct
Base checkpoint	tudor-iustin22/Bluebird-1.5-Base
Project	Project Bluebird
Model type	Decoder-only causal language model, instruction-tuned
Architecture	Local causal attention with gated, chunked long-range memory
Parameters	742,528,520 (742.5M)
Transformer layers	24: 16 LocalBlock + 8 FabricMemoryBlock
Hidden size	1,536
Vocabulary size	65,536
Attention	Grouped-query attention (GQA)
Query heads / KV heads	24 / 6
Head dimension	64
Maximum context length	32,768 tokens
Local attention window	2,048 tokens
Memory chunk size	512 tokens
Learned summaries per chunk	4
Maximum memory summaries	256 at 32K context
Positional encoding	RoPE, theta = 1,000,000
Feed-forward network	SwiGLU
Normalisation	RMSNorm
Published weight format	FP16
Training compute precision	Mixed precision
Framework	PyTorch
Intel Gaudi device	hpu
License	Apache License 2.0, subject to applicable upstream notices

Architecture

Bluebird 1.5 retains the Chunked Fabric Memory design of the base checkpoint. Every third transformer block supplements local causal attention with a second path that attends to compressed long-range memory:

• Local attention: exact causal attention within the most recent 2,048 tokens.
• Chunk compression: earlier completed spans are partitioned into 512-token chunks, each represented by four learned summaries.
• Memory access: at 32,768 tokens, the model can draw on up to 256 summary vectors.
• Gated integration: a learned per-token scalar gate blends local-attention and memory-path outputs.
• Grouped-query attention: 24 query heads share six key/value heads.

This structure limits the cost of attention to older context relative to applying full attention across the entire sequence. The Instruct release changes the training objective and conversational behaviour; it does not replace the underlying base architecture.

Quickstart

Bluebird 1.5 uses custom Transformers model code. Review repository Python files before setting trust_remote_code=True.

For Intel Gaudi 3, first install the Intel Gaudi software environment and compatible PyTorch build following the Intel Gaudi installation documentation. Install additional packages into that environment:

python -m pip install transformers safetensors accelerate

Chat with Bluebird on Intel Gaudi 3

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "tudor-iustin22/Bluebird-1.5-Instruct"
device = torch.device("hpu")

tokenizer = AutoTokenizer.from_pretrained(
    model_id,
    trust_remote_code=True,
)

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    torch_dtype=torch.bfloat16,
).to(device).eval()

messages = [
    {"role": "system", "content": "You are Bluebird, a helpful AI assistant from Project Bluebird."},
    {"role": "user", "content": "Explain chunked memory in language models in simple terms."},
]

prompt = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
)
inputs = tokenizer(prompt, return_tensors="pt").to(device)

with torch.inference_mode():
    with torch.autocast(device_type="hpu", dtype=torch.bfloat16):
        output_ids = model.generate(
            **inputs,
            max_new_tokens=256,
            do_sample=True,
            temperature=0.65,
            top_p=0.90,
            repetition_penalty=1.05,
            use_cache=False,
            pad_token_id=tokenizer.pad_token_id,
            eos_token_id=tokenizer.eos_token_id,
        )

response_ids = output_ids[0, inputs["input_ids"].shape[1]:]
print(tokenizer.decode(response_ids, skip_special_tokens=True).strip())

The Gaudi runtime uses PyTorch's hpu device and HCCL for distributed communication. Check compatibility between your installed Intel Gaudi release, the Transformers version, and this model's custom attention implementation.

Chat template

The model uses role-delimited messages with the following special tokens:

<|bos|> · <|pad|> · <|end|> · <|system|> · <|user|> · <|assistant|>

<|bos|><|system|>
You are Bluebird, a helpful AI assistant from Project Bluebird.<|end|>
<|user|>
Hello!<|end|>
<|assistant|>
Hi! How can I help you?<|end|>

Use the repository tokenizer's apply_chat_template() method when possible rather than manually assembling tokens. The example system message expresses the Project Bluebird identity; updating the model's default tokenizer template or its learned identity requires changes beyond this README where applicable.

Sampling recommendations

Task	Temperature	Top-p	Top-k	Repetition penalty
General chat	0.65	0.90	50	1.05
Creative writing	0.80	0.95	50	1.00
Factual and knowledge tasks	0.40	0.85	30	1.05
Code generation	0.30	0.90	40	1.02

These settings are starting points. Adjust them for the application, quality requirements, and latency budget.

Training

Training stages

Bluebird 1.5 Instruct follows a two-part model-development process:

1. Base-model pretraining and continued pretraining: the base lineage records 23,000,514,560 tokens, including approximately 18B tokens of initial pretraining followed by additional continued pretraining.
2. Supervised post-training (SFT): identity tuning for the assistant persona and system instructions, followed by general instruction-following and conversational fine-tuning.

Intel Gaudi 3 training infrastructure

Component	Configuration
Accelerators	8 × Intel Gaudi 3 AI accelerators
HBM capacity	128 GB per accelerator; 1,024 GB aggregate
CPU allocation	160 vCPUs
System memory	1,792 GiB RAM
Instance storage	25,600 GB
Framework	PyTorch
Accelerator backend	Intel Gaudi / PyTorch HPU
Training precision	Mixed precision
AMP compute	BF16 autocast where supported
Distributed communication	HCCL

Base-model pretraining data

Dataset	Approximate mixture share	Purpose
FineWeb-Edu	60%	Educational web text
DCLM Baseline	25%	Filtered and deduplicated web text
OpenWebMath	10%	Mathematical material
Cosmopedia v2	5%	Synthetic educational text

Instruction post-training data

The Instruct data lineage identifies the following sources:

• SmolTalk2 — conversational and instruction-following examples.
• Mixture-of-Thoughts — reasoning-oriented supervised data.
• Nemotron Post-Training Dataset v1 — instruction and post-training examples.

Exact SFT record counts, per-dataset weights, and optimizer-update totals are not specified here.

Optimisation

The published base-model training settings include AdamW (β1=0.9, β2=0.95, ε=1e-8), a 3e-4 initial learning rate with cosine decay to 3e-5, 180M-token warmup, 0.1 weight decay, and 1.0 gradient clipping. These are base-stage hyperparameters; they are not presented as independently measured SFT hyperparameters.

Reported Evaluation

The following zero-shot log-likelihood results are carried forward from the published Fabric 1.5 Instruct model card. They describe the recorded evaluation lineage; they are not presented as newly rerun measurements for this renamed repository.

Benchmark	Reported accuracy	Evaluation
ARC Easy	54.97%	0-shot
ARC Challenge	26.88%	0-shot
HellaSwag	35.00%	0-shot
MMLU	28.96%	0-shot
C-Eval	27.12%	0-shot

Published MMLU comparison

Model	MMLU (0-shot)
Fabric 1.5 Instruct lineage	28.96%
Qwen3.5-0.8B	29.70%

Results depend on dataset versions, prompt templates, evaluation harness, numerical precision, and checkpoint revision. Independent evaluation of the released Bluebird checkpoint is recommended before using these figures for deployment decisions.

Intended Use

Bluebird 1.5 Instruct is designed for:

• Conversational assistants and instruction-following applications.
• Question answering, summarisation, rewriting, and text drafting.
• Lightweight reasoning and task-oriented text generation.
• Research into compact assistants and memory-augmented transformer designs.
• Local inference, fine-tuning, and application-specific evaluation.

Limitations and responsible use

Bluebird 1.5 is a compact text-only model and may hallucinate facts, fail to follow complex instructions, reproduce biases in its training material, or generate unsuitable content. Its 32K context capacity does not guarantee accurate retrieval of all details within that window. The model is not an authoritative source for medical, legal, financial, or other high-stakes decisions.

Validate outputs, use task-specific safeguards, and perform independent safety and reliability evaluations before integrating the model into production systems.

License and Attribution

Project Bluebird — Copyright © 2026 Tudor Iustin.

Project Bluebird's original release materials are provided under the Apache License, Version 2.0. The terms allow use, modification, and redistribution subject to their conditions, including applicable attribution and notices.

This model is part of the Fabric 1.5-derived architecture and training lineage. Rights and notices associated with pre-existing code, model weights, training datasets, and other third-party components must be respected and retained where required. The repository should provide the applicable LICENSE text and any required NOTICE information; README metadata alone does not change the licence of third-party material.

Apache License, Version 2.0

Citation

@misc{bluebird15instruct2026,
  title        = {Bluebird 1.5 Instruct},
  author       = {Tudor Iustin},
  year         = {2026},
  publisher    = {Project Bluebird},
  howpublished = {Hugging Face model repository},
  url          = {https://huggingface.co/tudor-iustin22/Bluebird-1.5-Instruct}
}

---

PROJECT BLUEBIRD · Compact intelligence. Long-range memory.

© 2026 Tudor Iustin · Apache License 2.0