Text Generation
GGUF
English
xlam
function-calling
agent
tool-calling
react
qlora
qwen2
ollama
llama-cpp
function-calling-agent
Instructions to use iamrahull/xlam-ather with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use iamrahull/xlam-ather with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf iamrahull/xlam-ather:Q4_K_M # Run inference directly in the terminal: llama cli -hf iamrahull/xlam-ather:Q4_K_M
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf iamrahull/xlam-ather:Q4_K_M # Run inference directly in the terminal: llama cli -hf iamrahull/xlam-ather:Q4_K_M
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf iamrahull/xlam-ather:Q4_K_M # Run inference directly in the terminal: ./build/bin/llama cli -hf iamrahull/xlam-ather:Q4_K_M
Use Docker
docker model run hf.co/iamrahull/xlam-ather:Q4_K_M
- Llama new
- LM Studio
- Jan
- vLLM
How to use iamrahull/xlam-ather with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "iamrahull/xlam-ather" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "iamrahull/xlam-ather", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker
docker model run hf.co/iamrahull/xlam-ather:Q4_K_M
- Ollama
How to use iamrahull/xlam-ather with Ollama:
ollama run hf.co/iamrahull/xlam-ather:Q4_K_M
- Unsloth Desktop
- Docker Model Runner
How to use iamrahull/xlam-ather with Docker Model Runner:
docker model run hf.co/iamrahull/xlam-ather:Q4_K_M
- Lemonade
How to use iamrahull/xlam-ather with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull iamrahull/xlam-ather:Q4_K_M
Run and chat with the model
lemonade run user.xlam-ather-Q4_K_M
List all available models
lemonade list
- Atomic Chat
Updated Readme.md
Browse files
README.md
CHANGED
|
@@ -1,11 +1,32 @@
|
|
| 1 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2 |
|
|
|
|
| 3 |
## 1. Executive Summary
|
| 4 |
**xLAM-AETHER-v1** is a specialized, 3-Billion parameter Large Language Model (LLM) engineered to serve as the cognitive engine for autonomous agentic workflows.
|
| 5 |
|
| 6 |
Developed as the core component of **Project AETHER**, this model is designed to act not just as a conversational chatbot, but as a **ReAct (Reasoning + Acting) agent**. It has been fine-tuned to understand tool schemas, reason through a user's goal, and output highly structured, syntactically perfect JSON payloads to trigger external tools (such as shell commands or API calls).
|
| 7 |
|
| 8 |
-
Built upon the robust `Salesforce/xLAM-2-3b-fc-r` architecture (which is based on Qwen2.5), the model was distilled using QLoRA and quantized to a highly efficient 4-bit GGUF format. This allows it to run locally on consumer-grade laptops while retaining the complex reasoning capabilities of its parent architecture.
|
| 9 |
|
| 10 |
## 2. Core Capabilities
|
| 11 |
- **Dual-Mode Operation:** The model seamlessly switches between general conversational chat (natural language) and Agentic execution (strict JSON tool calls) without suffering from catastrophic forgetting.
|
|
@@ -14,56 +35,39 @@ Built upon the robust `Salesforce/xLAM-2-3b-fc-r` architecture (which is based o
|
|
| 14 |
- **Low Hardware Footprint:** Quantized to `Q4_K_M` (~1.93 GB), allowing it to run entirely in system RAM on standard laptops or free cloud tiers.
|
| 15 |
|
| 16 |
## 3. Technical Specifications
|
| 17 |
-
|
| 18 |
| Attribute | Detail |
|
| 19 |
| :--- | :--- |
|
| 20 |
| **Base Model** | `Salesforce/xLAM-2-3b-fc-r` (Qwen2 architecture) |
|
| 21 |
| **Total Parameters** | 3.09 Billion (Base) + 7.3 Million (LoRA Adapter) |
|
| 22 |
| **Fine-Tuning Method** | QLoRA (Quantized Low-Rank Adaptation) |
|
| 23 |
| **Training Precision** | 4-bit NormalFloat (NF4) with FP16 compute |
|
| 24 |
-
| **Dataset** | `Salesforce/APIGen-MT-5k`
|
| 25 |
| **Training Hardware** | Google Colab (NVIDIA Tesla T4) |
|
| 26 |
| **Final Format** | GGUF (`q4_k_m` quantization) |
|
| 27 |
-
| **Chat Template** | ChatML
|
| 28 |
| **File Size** | 1.93 GB |
|
| 29 |
-
| **License** | CC-BY-NC-4.0
|
| 30 |
|
| 31 |
## 4. End-to-End Training Pipeline
|
| 32 |
-
|
| 33 |
### 4.1 Data Synthesis & Curation
|
| 34 |
-
Rather than training from scratch, the model utilized the `Salesforce/APIGen-MT-5k` dataset.
|
| 35 |
|
| 36 |
### 4.2 The Distillation Protocol (QLoRA)
|
| 37 |
-
To inject reasoning capabilities into the 3B model without requiring enterprise-grade GPUs, the training utilized QLoRA:
|
| 38 |
- **Base Quantization:** The base model was loaded in 4-bit NF4.
|
| 39 |
-
- **Adapter Injection:** Trainable low-rank matrices
|
| 40 |
-
- **Hyperparameters:** Rank
|
| 41 |
-
- **Efficiency:** Using
|
| 42 |
|
| 43 |
### 4.3 Merging & Quantization
|
| 44 |
-
Post-training, the 28MB LoRA adapter was
|
| 45 |
|
| 46 |
## 5. Benchmark & Behavior Tests
|
| 47 |
-
|
| 48 |
-
|
| 49 |
-
|
| 50 |
-
**Test A: General Knowledge (Chat Mode)**
|
| 51 |
-
- *Prompt:* "What are the cultural beliefs of China and Egypt? Give a short summary."
|
| 52 |
-
- *Output:* Coherent, well-structured natural language summary covering mummification, Confucianism, and filial piety. No hallucinated JSON. Retained its "chat" persona.
|
| 53 |
-
|
| 54 |
-
**Test B: Reasoning (Math Mode)**
|
| 55 |
-
- *Prompt:* "If a train travels at 60 mph for 2.5 hours, how far does it go? Explain the math."
|
| 56 |
-
- *Output:* Step-by-step breakdown using LaTeX formatting, correctly applying the formula \(Distance = Speed \times Time\), and concluding with `150 miles`.
|
| 57 |
-
|
| 58 |
-
**Test C: Agentic Tool Calling (ReAct Mode)**
|
| 59 |
-
- *Prompt:* *"List all .tmp files in /tmp/aether-sandbox."* (Provided with a `shell` tool schema).
|
| 60 |
-
- *Output:* `[{"name": "shell", "arguments": {"cmd": "find /tmp/aether-sandbox -type f -name '.tmp'"}}]`
|
| 61 |
-
- *Latency:* 1.12 seconds (via API on a Colab T4 GPU).
|
| 62 |
|
| 63 |
## 6. Deployment & Inference
|
| 64 |
-
|
| 65 |
### Option 1: Local Inference (Ollama / LM Studio)
|
| 66 |
-
Due to the GGUF format, the model can be run locally. A `Modelfile` is required to enforce the ChatML template.
|
| 67 |
```text
|
| 68 |
FROM ./xlam-aether-q4_k_m.gguf
|
| 69 |
TEMPLATE """<|im_start|>system
|
|
@@ -73,28 +77,4 @@ TEMPLATE """<|im_start|>system
|
|
| 73 |
<|im_start|>assistant
|
| 74 |
"""
|
| 75 |
PARAMETER stop "<|im_end|>"
|
| 76 |
-
PARAMETER temperature 0.1
|
| 77 |
-
```
|
| 78 |
-
|
| 79 |
-
### Option 2: OpenAI-Compatible API (Colab)
|
| 80 |
-
Using `llama-cpp-python`, the model can be served as a local OpenAI-compatible API. This is ideal for building FastAPI backends that interact with the model.
|
| 81 |
-
```bash
|
| 82 |
-
python -m llama_cpp.server --model xlam-aether-q4_k_m.gguf --n_gpu_layers 99 --chat_format chatml --port 8000
|
| 83 |
-
```
|
| 84 |
-
|
| 85 |
-
## 7. Limitations & Future Work
|
| 86 |
-
|
| 87 |
-
- **License Constraints:** Because it inherits from the xLAM base model, this model is licensed under **CC-BY-NC-4.0**. It is strictly for research and personal use. Commercial deployment would require re-training on a permissive base model (e.g., Llama-3.2-3B).
|
| 88 |
-
- **Hardware Bottlenecks:** While fast on a GPU (~1.1s response), running this model purely on a CPU (like Hugging Face's free tier) results in massive latency (~250 seconds per prompt) due to the prefill phase processing the tool schema.
|
| 89 |
-
- **Parallel Tool Calling:** The model currently excels at single-action tool calls. Future training iterations will focus on emitting arrays of multiple parallel tool calls.
|
| 90 |
-
|
| 91 |
-
## 8. Links & Resources
|
| 92 |
-
- **Model Repository:** [huggingface.co/iamrahull/xlam-ather](https://huggingface.co/iamrahull/xlam-ather)
|
| 93 |
-
- **Demo Space:** [huggingface.co/spaces/iamrahull/xlam-ather](https://huggingface.co/spaces/iamrahull/xlam-ather)
|
| 94 |
-
- **Training Dataset:** [Salesforce/APIGen-MT-5k](https://huggingface.co/datasets/Salesforce/APIGen-MT-5k)
|
| 95 |
-
|
| 96 |
-
---
|
| 97 |
-
*Report generated for Project AETHER. Designed and trained by Rahul.*
|
| 98 |
-
---
|
| 99 |
-
license: cc-by-nc-4.0
|
| 100 |
-
---
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: cc-by-nc-4.0
|
| 3 |
+
base_model: Salesforce/xLAM-2-3b-fc-r
|
| 4 |
+
library_name: gguf
|
| 5 |
+
pipeline_tag: text-generation
|
| 6 |
+
tags:
|
| 7 |
+
- xlam
|
| 8 |
+
- function-calling
|
| 9 |
+
- agent
|
| 10 |
+
- tool-calling
|
| 11 |
+
- react
|
| 12 |
+
- qlora
|
| 13 |
+
- gguf
|
| 14 |
+
- qwen2
|
| 15 |
+
- qwen2.5
|
| 16 |
+
- 3b
|
| 17 |
+
language:
|
| 18 |
+
- en
|
| 19 |
+
datasets:
|
| 20 |
+
- Salesforce/APIGen-MT-5k
|
| 21 |
+
---
|
| 22 |
|
| 23 |
+
# Project AETHER: xLAM-AETHER-v1 Model Card & Technical Report
|
| 24 |
## 1. Executive Summary
|
| 25 |
**xLAM-AETHER-v1** is a specialized, 3-Billion parameter Large Language Model (LLM) engineered to serve as the cognitive engine for autonomous agentic workflows.
|
| 26 |
|
| 27 |
Developed as the core component of **Project AETHER**, this model is designed to act not just as a conversational chatbot, but as a **ReAct (Reasoning + Acting) agent**. It has been fine-tuned to understand tool schemas, reason through a user's goal, and output highly structured, syntactically perfect JSON payloads to trigger external tools (such as shell commands or API calls).
|
| 28 |
|
| 29 |
+
Built upon the robust `【entity-Salesforce¦canonical_name=Salesforce】/xLAM-2-3b-fc-r` architecture (which is based on Qwen2.5), the model was distilled using QLoRA and quantized to a highly efficient 4-bit GGUF format. This allows it to run locally on consumer-grade laptops while retaining the complex reasoning capabilities of its parent architecture.
|
| 30 |
|
| 31 |
## 2. Core Capabilities
|
| 32 |
- **Dual-Mode Operation:** The model seamlessly switches between general conversational chat (natural language) and Agentic execution (strict JSON tool calls) without suffering from catastrophic forgetting.
|
|
|
|
| 35 |
- **Low Hardware Footprint:** Quantized to `Q4_K_M` (~1.93 GB), allowing it to run entirely in system RAM on standard laptops or free cloud tiers.
|
| 36 |
|
| 37 |
## 3. Technical Specifications
|
|
|
|
| 38 |
| Attribute | Detail |
|
| 39 |
| :--- | :--- |
|
| 40 |
| **Base Model** | `Salesforce/xLAM-2-3b-fc-r` (Qwen2 architecture) |
|
| 41 |
| **Total Parameters** | 3.09 Billion (Base) + 7.3 Million (LoRA Adapter) |
|
| 42 |
| **Fine-Tuning Method** | QLoRA (Quantized Low-Rank Adaptation) |
|
| 43 |
| **Training Precision** | 4-bit NormalFloat (NF4) with FP16 compute |
|
| 44 |
+
| **Dataset** | `Salesforce/APIGen-MT-5k` |
|
| 45 |
| **Training Hardware** | Google Colab (NVIDIA Tesla T4) |
|
| 46 |
| **Final Format** | GGUF (`q4_k_m` quantization) |
|
| 47 |
+
| **Chat Template** | ChatML |
|
| 48 |
| **File Size** | 1.93 GB |
|
| 49 |
+
| **License** | CC-BY-NC-4.0 |
|
| 50 |
|
| 51 |
## 4. End-to-End Training Pipeline
|
|
|
|
| 52 |
### 4.1 Data Synthesis & Curation
|
| 53 |
+
Rather than training from scratch, the model utilized the `Salesforce/APIGen-MT-5k` dataset.
|
| 54 |
|
| 55 |
### 4.2 The Distillation Protocol (QLoRA)
|
|
|
|
| 56 |
- **Base Quantization:** The base model was loaded in 4-bit NF4.
|
| 57 |
+
- **Adapter Injection:** Trainable low-rank matrices injected into attention layers.
|
| 58 |
+
- **Hyperparameters:** Rank 16, Alpha 32, LR 2e-4.
|
| 59 |
+
- **Efficiency:** Using Unsloth framework, trained for 1 epoch (625 steps), final loss `0.0003`.
|
| 60 |
|
| 61 |
### 4.3 Merging & Quantization
|
| 62 |
+
Post-training, the 28MB LoRA adapter was merged and converted to GGUF `Q4_K_M`.
|
| 63 |
|
| 64 |
## 5. Benchmark & Behavior Tests
|
| 65 |
+
**Test A: General Knowledge** - Retained chat persona.
|
| 66 |
+
**Test B: Reasoning** - Correct math with LaTeX.
|
| 67 |
+
**Test C: Agentic Tool Calling** - Outputs `[{"name": "shell", "arguments": {"cmd": "..."}}]` with 1.12s latency.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 68 |
|
| 69 |
## 6. Deployment & Inference
|
|
|
|
| 70 |
### Option 1: Local Inference (Ollama / LM Studio)
|
|
|
|
| 71 |
```text
|
| 72 |
FROM ./xlam-aether-q4_k_m.gguf
|
| 73 |
TEMPLATE """<|im_start|>system
|
|
|
|
| 77 |
<|im_start|>assistant
|
| 78 |
"""
|
| 79 |
PARAMETER stop "<|im_end|>"
|
| 80 |
+
PARAMETER temperature 0.1
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|