Fidel2.0-1B / README.md
eniairaph07's picture
Document verified H100 and RTX PRO 6000 compatibility
dc8e880 verified
|
Raw History Blame Contribute Delete
7.59 kB
---
license: apache-2.0
base_model: 4E-AI/Fidel1.1-1B
datasets:
- 4E-AI/Fidel-Competitive-coding
pipeline_tag: text-generation
tags:
- fidel
- mamba
- mixture-of-experts
- code
- safetensors
---
# Fidel2.0-1B
**Fidel 2.0** is a compact **competitive coding model** from **4E-AI**, combining Mamba3 MIMO sequence modeling, attention, and a latent mixture-of-experts architecture. This release brings its backbone, coding adaptation, and conversational components together in one model.
The release focuses on competitive programming: interpreting problem statements, reasoning about algorithms and data structures, writing solutions, and revising answers with execution feedback. The weights and tokenizer are published under **Apache-2.0** for experimentation, application development, and further adaptation.
> [!NOTE]
> This repository contains the complete native SafeTensors weights, original tokenizer, and model configuration.
## Fidel Highlights
- **Competitive coding focus:** Designed for programming problems, algorithmic solutions, and correction from execution feedback.
- **Hybrid architecture:** Eight Mamba3 MIMO blocks and one attention block combine state-space sequence modeling with attention.
- **Latent mixture of experts:** Each block routes tokens to two of sixteen experts in a 512-dimensional latent space, alongside a shared feed-forward path.
- **Unified weights:** Backbone weights, coding adapters, prompt conditioning, and conversational components are included together. No separate adapter download is required.
- **Original numerical precision:** Every tensor retains its saved data type and values. The release does not apply an additional quantization or precision conversion.
- **Conversation formatting:** The original tokenizer, special tokens, and chat template are included together.
- **Open customization:** Apache-2.0 permits use, modification, and redistribution, subject to its terms.
## Model Overview
| Property | Configuration |
| --- | --- |
| Developer | 4E-AI |
| Model type | Text-only autoregressive language model |
| Primary focus | Competitive coding and algorithmic problem solving |
| Architecture | Hybrid Mamba3 MIMO, attention, and latent MoE |
| Model family size | 1B |
| Stored floating-point elements | 1,402,955,496 |
| Hidden dimension | 1,536 |
| Number of backbone layers | 9 |
| Mixer layout | 8 Mamba3 MIMO blocks and 1 attention block |
| Query / key-value heads | 16 / 4 |
| Attention head dimension | 96 |
| Mamba heads / head dimension | 48 / 64 |
| Mamba state dimension / MIMO rank | 64 / 4 |
| Experts per MoE layer / selected per token | 16 / 2 |
| Latent expert dimension | 512 |
| Expert intermediate dimension | 2,048 |
| Shared feed-forward intermediate dimension | 6,144 |
| Vocabulary size | 131,072 |
| Embedding and output head | Separate weight matrices |
| Combined input/output context limit | 8,192 tokens |
| Weight format | SafeTensors |
| Stored data types | bfloat16, bool, float32 |
| Weight file size | 5.37 GB |
| License | Apache-2.0 |
`1B` is the model family designation. The stored element count includes the full model and persistent floating-point buffers. Expert routing applies to the routed experts; embeddings, shared paths, and other components also contribute to computation.
## Fidel Capabilities
**Competitive programming.** Fidel is intended for tasks involving problem statements, constraints, algorithms, data structures, and complete program solutions. Generated code should be executed and checked against the intended behavior.
**Revision from feedback.** An application can provide execution results or other feedback in the conversation and ask for a replacement answer. Feedback must come from the application; the model does not execute code on its own.
**Conversational prompting.** The included chat template supports formatting instructions, examples, and conversation history. Clear requests and relevant context help define the intended task.
**Further adaptation.** The complete model weights provide a starting point for domain-specific experimentation with a compatible Fidel workflow. Evaluate adapted models on representative data before relying on their outputs.
Fidel accepts text inputs. Image, audio, video, web browsing, and tool execution require separate application components.
## Repository Contents
| File | Contents |
| --- | --- |
| `model.safetensors` | Complete native model tensors with their original names, data types, and values |
| `model_config.json` | Structural model configuration and saved numerical recipe settings |
| `weights_manifest.json` | Tensor names, shapes, data types, sizes, and integrity hashes |
| `tokenizer.json` | Original tokenizer vocabulary and encoding rules |
| `tokenizer_config.json` | Tokenizer configuration and conversation formatting |
| `special_tokens_map.json` | Special-token definitions |
| `chat_template.jinja` | Original chat template |
| `compatibility/` | Transformers runtime wheel, matching source, checksums and validation notes |
| `LICENSE` / `NOTICE` | License and attribution |
## Transformers Usage
Install the bundled Fidel runtime in a Linux Python 3.11 environment with an NVIDIA GPU and a CUDA development toolkit (`nvcc` plus a C++ compiler):
```bash
python -m pip install "torch==2.15.0.dev20260928+cu130" --index-url https://download.pytorch.org/whl/nightly/cu130
python -m pip install ninja setuptools "https://huggingface.co/4E-AI/Fidel2.0-1B/resolve/main/compatibility/fidel_compatibility-0.3.0.dev0-py3-none-any.whl"
```
```python
from transformers import AutoTokenizer
from fidel_compatibility import FidelForCausalLM
repo = "4E-AI/Fidel2.0-1B"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = FidelForCausalLM.from_pretrained(
repo, device="cuda", cache_mode="chunk_state"
)
messages = [{"role": "user", "content": "Write a Python function that returns the smaller of two integers."}]
inputs = tokenizer.apply_chat_template(
messages, add_generation_prompt=True, return_tensors="pt"
).to(model.device)
outputs = model.generate(
inputs, max_new_tokens=64, do_sample=False, use_cache=True
)
print(tokenizer.decode(outputs[0, inputs.shape[1]:], skip_special_tokens=True))
```
This runtime preserves the original mixed tensor types and the **0.25 numerical blend**, with attention KV reuse and guarded Mamba chunk-state caching. It is a correctness reference for one unpadded request; L4, A10, A100, H100 and RTX PRO 6000 have completed its model, generation and cache checks. First use compiles the included CUDA helpers. See [runtime details and validation](compatibility/README.md) for the tested environment and cache modes.
## Usage Notes
- Keep the supplied tokenizer and chat template with the weights. Changes to tokenization or formatting can change model behavior.
- The 8,192-token limit covers the combined prompt, conversation history, and generated answer. It is not a guarantee of reliable behavior throughout that length.
- Longer answers and repeated revisions can require substantially more compute.
- Generated explanations, self-reported checks, and code can be incorrect. Validate them against real sources or execution results.
## License
Fidel weights and original Fidel code are licensed under [Apache-2.0](LICENSE). Commercial use, modification, and redistribution are permitted subject to its terms. Third-party components retain their respective licenses; see [NOTICE](NOTICE).