--- license: apache-2.0 base_model: 4E-AI/Fidel1.1-1B datasets: - 4E-AI/Fidel-Competitive-coding pipeline_tag: text-generation tags: - fidel - mamba - mixture-of-experts - code - safetensors --- # Fidel2.0-1B **Fidel 2.0** is a compact **competitive coding model** from **4E-AI**, combining Mamba3 MIMO sequence modeling, attention, and a latent mixture-of-experts architecture. This release brings its backbone, coding adaptation, and conversational components together in one model. The release focuses on competitive programming: interpreting problem statements, reasoning about algorithms and data structures, writing solutions, and revising answers with execution feedback. The weights and tokenizer are published under **Apache-2.0** for experimentation, application development, and further adaptation. > [!NOTE] > This repository contains the complete native SafeTensors weights, original tokenizer, and model configuration. ## Fidel Highlights - **Competitive coding focus:** Designed for programming problems, algorithmic solutions, and correction from execution feedback. - **Hybrid architecture:** Eight Mamba3 MIMO blocks and one attention block combine state-space sequence modeling with attention. - **Latent mixture of experts:** Each block routes tokens to two of sixteen experts in a 512-dimensional latent space, alongside a shared feed-forward path. - **Unified weights:** Backbone weights, coding adapters, prompt conditioning, and conversational components are included together. No separate adapter download is required. - **Original numerical precision:** Every tensor retains its saved data type and values. The release does not apply an additional quantization or precision conversion. - **Conversation formatting:** The original tokenizer, special tokens, and chat template are included together. - **Open customization:** Apache-2.0 permits use, modification, and redistribution, subject to its terms. ## Model Overview | Property | Configuration | | --- | --- | | Developer | 4E-AI | | Model type | Text-only autoregressive language model | | Primary focus | Competitive coding and algorithmic problem solving | | Architecture | Hybrid Mamba3 MIMO, attention, and latent MoE | | Model family size | 1B | | Stored floating-point elements | 1,402,955,496 | | Hidden dimension | 1,536 | | Number of backbone layers | 9 | | Mixer layout | 8 Mamba3 MIMO blocks and 1 attention block | | Query / key-value heads | 16 / 4 | | Attention head dimension | 96 | | Mamba heads / head dimension | 48 / 64 | | Mamba state dimension / MIMO rank | 64 / 4 | | Experts per MoE layer / selected per token | 16 / 2 | | Latent expert dimension | 512 | | Expert intermediate dimension | 2,048 | | Shared feed-forward intermediate dimension | 6,144 | | Vocabulary size | 131,072 | | Embedding and output head | Separate weight matrices | | Combined input/output context limit | 8,192 tokens | | Weight format | SafeTensors | | Stored data types | bfloat16, bool, float32 | | Weight file size | 5.37 GB | | License | Apache-2.0 | `1B` is the model family designation. The stored element count includes the full model and persistent floating-point buffers. Expert routing applies to the routed experts; embeddings, shared paths, and other components also contribute to computation. ## Fidel Capabilities **Competitive programming.** Fidel is intended for tasks involving problem statements, constraints, algorithms, data structures, and complete program solutions. Generated code should be executed and checked against the intended behavior. **Revision from feedback.** An application can provide execution results or other feedback in the conversation and ask for a replacement answer. Feedback must come from the application; the model does not execute code on its own. **Conversational prompting.** The included chat template supports formatting instructions, examples, and conversation history. Clear requests and relevant context help define the intended task. **Further adaptation.** The complete model weights provide a starting point for domain-specific experimentation with a compatible Fidel workflow. Evaluate adapted models on representative data before relying on their outputs. Fidel accepts text inputs. Image, audio, video, web browsing, and tool execution require separate application components. ## Repository Contents | File | Contents | | --- | --- | | `model.safetensors` | Complete native model tensors with their original names, data types, and values | | `model_config.json` | Structural model configuration and saved numerical recipe settings | | `weights_manifest.json` | Tensor names, shapes, data types, sizes, and integrity hashes | | `tokenizer.json` | Original tokenizer vocabulary and encoding rules | | `tokenizer_config.json` | Tokenizer configuration and conversation formatting | | `special_tokens_map.json` | Special-token definitions | | `chat_template.jinja` | Original chat template | | `compatibility/` | Transformers runtime wheel, matching source, checksums and validation notes | | `LICENSE` / `NOTICE` | License and attribution | ## Transformers Usage Install the bundled Fidel runtime in a Linux Python 3.11 environment with an NVIDIA GPU and a CUDA development toolkit (`nvcc` plus a C++ compiler): ```bash python -m pip install "torch==2.15.0.dev20260928+cu130" --index-url https://download.pytorch.org/whl/nightly/cu130 python -m pip install ninja setuptools "https://huggingface.co/4E-AI/Fidel2.0-1B/resolve/main/compatibility/fidel_compatibility-0.3.0.dev0-py3-none-any.whl" ``` ```python from transformers import AutoTokenizer from fidel_compatibility import FidelForCausalLM repo = "4E-AI/Fidel2.0-1B" tokenizer = AutoTokenizer.from_pretrained(repo) model = FidelForCausalLM.from_pretrained( repo, device="cuda", cache_mode="chunk_state" ) messages = [{"role": "user", "content": "Write a Python function that returns the smaller of two integers."}] inputs = tokenizer.apply_chat_template( messages, add_generation_prompt=True, return_tensors="pt" ).to(model.device) outputs = model.generate( inputs, max_new_tokens=64, do_sample=False, use_cache=True ) print(tokenizer.decode(outputs[0, inputs.shape[1]:], skip_special_tokens=True)) ``` This runtime preserves the original mixed tensor types and the **0.25 numerical blend**, with attention KV reuse and guarded Mamba chunk-state caching. It is a correctness reference for one unpadded request; L4, A10, A100, H100 and RTX PRO 6000 have completed its model, generation and cache checks. First use compiles the included CUDA helpers. See [runtime details and validation](compatibility/README.md) for the tested environment and cache modes. ## Usage Notes - Keep the supplied tokenizer and chat template with the weights. Changes to tokenization or formatting can change model behavior. - The 8,192-token limit covers the combined prompt, conversation history, and generated answer. It is not a guarantee of reliable behavior throughout that length. - Longer answers and repeated revisions can require substantially more compute. - Generated explanations, self-reported checks, and code can be incorrect. Validate them against real sources or execution results. ## License Fidel weights and original Fidel code are licensed under [Apache-2.0](LICENSE). Commercial use, modification, and redistribution are permitted subject to its terms. Third-party components retain their respective licenses; see [NOTICE](NOTICE).