| --- |
| license: other |
| license_name: lfm-open-1.0 |
| license_link: https://huggingface.co/LiquidAI/LFM2.5-VL-1.6B-GGUF/blob/main/LICENSE |
| library_name: llama.cpp |
| pipeline_tag: image-text-to-text |
| tags: |
| - ternary |
| - 2-bit |
| - gguf |
| - llama-cpp |
| - on-device |
| - vision-language |
| - multimodal |
| - finnai |
| base_model: |
| - LiquidAI/LFM2.5-VL-1.6B-GGUF |
| - LiquidAI/LFM2.5-1.2B-Base |
| --- |
| |
| # FinnAI Foundation (Edge Multimodal Suite) |
|
|
| > **Notice: Community Preview & Preliminary Measurements** |
| > *FinnAI Foundation is an initial public release. All performance metrics, throughput figures, and task accuracies presented below reflect internal empirical benchmarks conducted using `llama.cpp` on Apple Silicon / ARM64 CPU. They have not yet been independently verified by external benchmark parties.* |
|
|
| **FinnAI Foundation** is an ultra-compact on-device Vision-Language Model (VLM) suite designed for resource-constrained edge systems. It combines an importance-matrix-guided 2.5-bit low-bit representation (`IQ2_S` effective ~2.50 BPW) with a compressed SigLIP2 vision projector (`mmproj` INT4/FP32), achieving a **~45% total footprint reduction** compared to standard INT4 baselines with full retention across core reasoning and factual tasks. |
|
|
| While the consumer client **FinnAI** remains closed-source, the **FinnAI Foundation** weights and compressed multimodal components are published openly for edge AI developers, robotics engineers, and researchers. |
|
|
| --- |
|
|
| ## Architectural Profile & Metadata |
|
|
| - **Architecture**: Liquid Neural Network hybrid with short recurrent convolutions (`shortconv`), Liquid linear state projections, SwiGLU feed-forward networks, and GQA attention heads. |
| - **Base Model Lineage**: |
| - Base Language Backbone: `LiquidAI/LFM2.5-1.2B-Base` (1.2B parameter text backbone) |
| - Base Multimodal Checkpoint: `LiquidAI/LFM2.5-VL-1.6B-GGUF` (1.6B total parameter multimodal vision-language model, including vision encoder) |
| - **Runtime Target**: `llama.cpp` (b10809+) GGUF v3 format with full CPU and Metal/Vulkan compute compatibility. |
|
|
| --- |
|
|
| ## Empirical Benchmark Evaluation |
|
|
| The table below details internal side-by-side evaluations conducted under identical conditions: `temp = 0.0`, `seed = 42`, non-warmup CLI execution, on ARM64 hardware. |
|
|
| ### 1. Comparative Metrics Summary |
|
|
| | Evaluation Axis | Metric / Task | Baseline `Q4_0` (4.74 BPW) | FinnAI Foundation (`IQ2_S` ~2.50 BPW) | Delta / Footprint Change | |
| | :--- | :--- | :--- | :--- | :--- | |
| | **Model Size (LLM)** | Disk Size | 695.75 MB | **385.08 MB** | **-44.6%** | |
| | **Projector Size (`mmproj`)** | Disk Size | 853.99 MB (FP16) / 556 MB (Q8) | **417.42 MB** (`Q4_0`) | **-51.1%** vs FP16 | |
| | **Total Multimodal Bundle** | Combined Footprint | 1,549.74 MB | **802.50 MB** | **-48.2% total reduction** | |
| | **Perplexity (Wikitext)** | PPL (512 ctx, 43 chunks) | **14.80** $\pm$ 0.43 | **21.25** $\pm$ 0.61 | -5.01 PPL vs naive 2-bit (19.1% improvement) | |
| | **Mathematical Reasoning** | BIG-bench Simple Arithmetic | **100.0%** (10/10) | **100.0%** (10/10) | Parity with uncompressed baseline | |
| | **Factual Knowledge & QA** | BIG-bench Wikidata / Fact QA | **90.0%** (9/10) | **100.0%** (10/10) | +10.0% (recovered boiling point query) | |
| | **Multimodal OCR & Vision** | Geometric & Color Grounding | **100.0%** (2/2) | **100.0%** (2/2) | Full visual grounding parity | |
| | **Generation Throughput** | Apple Silicon Metal Offload | ~90 tokens/sec | **91.8 tokens/sec** | Equal / slightly improved memory bandwidth | |
| | **Vision Encoding Throughput** | Patch Processing Speed | 58.4 prompt t/s | **145.7 prompt t/s** | **2.5x faster visual ingest** | |
|
|
| ### 2. Dataset Contamination Verification |
|
|
| > **Contamination Safeguard**: |
| > To guarantee that reported performance is not the artifact of test data memorization: |
| > - **Calibration Dataset**: The importance matrix (`imatrix_dense.dat`) was calibrated across 195 multi-domain blocks covering formal logical syllogisms, step-by-step arithmetic proofs, code patterns (Python, Dart), physics/systems explanations, and multilingual text. |
| > - **Evaluation Tasks**: Evaluated against unseen external tasks from Google's BIG-bench repository (Simple Arithmetic and QA Wikidata) and external icon grounding prompts. Zero overlap exists between the calibration corpus and benchmark prompts. |
| |
| --- |
| |
| ## Technical Methodology: Established Best Practices |
| |
| Rather than claiming novel theoretical breakthroughs, the compression of FinnAI Foundation leverages established quantization engineering practices: |
| |
| 1. **High-Density Importance-Matrix-Guided Quantization (`imatrix`)**: |
| Standard naive uniform round-to-nearest quantization at 2 bits often results in severe outlier clipping, collapsing attention representations. Using activation variance computed over a comprehensive 195-block multi-domain calibration corpus (`llama-imatrix`), weights on critical attention pathways (`attn_v`, `attn_output`) are protected, while less sensitive MLP feed-forward projections are compressed into 2.5-bit codebooks (`IQ2_S`). |
| 2. **Precision Anchoring (Established Industry Practice)**: |
| In line with standard practice for mixed-precision architectures, sensitive layers—specifically recurrent convolution kernels (`shortconv.conv.weight`), layernorm scales, and embedding bounds—are maintained in full precision (FP32/FP16). |
| 3. **Projector Downsampling**: |
| The multimodal vision projector is quantized to `Q4_0` with FP32 normalization layers, halving the VRAM and memory bandwidth footprint without degrading spatial feature alignment. |
|
|
| --- |
|
|
| ## Artifact Specifications |
|
|
| | File Name | Format | Size | SHA-256 Digest | Description | |
| | :--- | :--- | :--- | :--- | :--- | |
| | `FinnAI-Foundation-1.6B-Ternary.gguf` | GGUF v3 (`IQ2_S` / Dense Salience Matrix) | 385.08 MB (403,789,888 B) | `e2e74f2b0baee70ddb963d712a15a7c7cd0ae00f7b460b90295a7068efd8c459` | Language model backbone with 2.50 BPW effective low-bit representations and FP32 recurrent anchors. | |
| | `mmproj-FinnAI-Foundation-1.6B-Q4_0.gguf` | GGUF v3 (`Q4_0` / FP32 Norms) | 417.42 MB (437,694,880 B) | `09dbffdce0fe89de27ac1cdc93253f6733ec1b303946ae8a5a7b1c871de5d566` | Compressed SigLIP2 multimodal vision projector component. | |
|
|
| --- |
|
|
| ## Quickstart & Usage |
|
|
| ### 1. llama.cpp (CLI) |
|
|
| ```bash |
| # Text generation |
| llama-cli \ |
| -m FinnAI-Foundation-1.6B-Ternary.gguf \ |
| -p "Explain how an operating system manages memory in simple terms:" \ |
| -n 256 \ |
| --temp 0.3 \ |
| -ngl 0 |
| |
| # Multimodal visual inference with image input |
| llama-cli \ |
| -m FinnAI-Foundation-1.6B-Ternary.gguf \ |
| --mmproj mmproj-FinnAI-Foundation-1.6B-Q4_0.gguf \ |
| --image diagram.png \ |
| -p "Describe the flow represented in this system architecture diagram." \ |
| -n 512 |
| ``` |
|
|
| ### 2. llama-server (OpenAI-Compatible Local API) |
|
|
| ```bash |
| llama-server \ |
| -m FinnAI-Foundation-1.6B-Ternary.gguf \ |
| --mmproj mmproj-FinnAI-Foundation-1.6B-Q4_0.gguf \ |
| --port 8080 \ |
| --ctx-size 4096 |
| ``` |
|
|
| ### 3. Python (`llama-cpp-python`) |
|
|
| ```python |
| from llama_cpp import Llama |
| |
| llm = Llama( |
| model_path="FinnAI-Foundation-1.6B-Ternary.gguf", |
| n_ctx=4096, |
| n_gpu_layers=0, |
| ) |
| |
| output = llm( |
| "User: What are the main trade-offs of 2-bit quantization on mobile devices?\nAssistant:", |
| max_tokens=256, |
| temperature=0.3, |
| ) |
| print(output["choices"][0]["text"]) |
| ``` |
|
|
| --- |
|
|
| ## License & Attribution |
|
|
| - FinnAI Foundation weights are distributed under the **LFM Open License v1.0**. |
| - Built on top of Liquid AI's LFM-2.5-VL-1.6B / LFM2.5-1.2B-Base architecture. |
| - Developed by **Finn Technologies**. |
|
|
|
|
