Instructions to use gaurav0506/rag-sft-qwen2.5-coder-3b with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use gaurav0506/rag-sft-qwen2.5-coder-3b with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf gaurav0506/rag-sft-qwen2.5-coder-3b:Q8_0 # Run inference directly in the terminal: llama cli -hf gaurav0506/rag-sft-qwen2.5-coder-3b:Q8_0
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf gaurav0506/rag-sft-qwen2.5-coder-3b:Q8_0 # Run inference directly in the terminal: llama cli -hf gaurav0506/rag-sft-qwen2.5-coder-3b:Q8_0
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf gaurav0506/rag-sft-qwen2.5-coder-3b:Q8_0 # Run inference directly in the terminal: ./build/bin/llama cli -hf gaurav0506/rag-sft-qwen2.5-coder-3b:Q8_0
Use Docker
docker model run hf.co/gaurav0506/rag-sft-qwen2.5-coder-3b:Q8_0
- Llama new
- LM Studio
- Jan
- vLLM
How to use gaurav0506/rag-sft-qwen2.5-coder-3b with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "gaurav0506/rag-sft-qwen2.5-coder-3b" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "gaurav0506/rag-sft-qwen2.5-coder-3b", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/gaurav0506/rag-sft-qwen2.5-coder-3b:Q8_0
- Ollama
How to use gaurav0506/rag-sft-qwen2.5-coder-3b with Ollama:
ollama run hf.co/gaurav0506/rag-sft-qwen2.5-coder-3b:Q8_0
- Unsloth Desktop
- Pi
How to use gaurav0506/rag-sft-qwen2.5-coder-3b with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf gaurav0506/rag-sft-qwen2.5-coder-3b:Q8_0
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:9931/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "gaurav0506/rag-sft-qwen2.5-coder-3b:Q8_0" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use gaurav0506/rag-sft-qwen2.5-coder-3b with Docker Model Runner:
docker model run hf.co/gaurav0506/rag-sft-qwen2.5-coder-3b:Q8_0
- Lemonade
How to use gaurav0506/rag-sft-qwen2.5-coder-3b with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull gaurav0506/rag-sft-qwen2.5-coder-3b:Q8_0
Run and chat with the model
lemonade run user.rag-sft-qwen2.5-coder-3b-Q8_0
List all available models
lemonade list
- Hermes Agent
How to use gaurav0506/rag-sft-qwen2.5-coder-3b with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf gaurav0506/rag-sft-qwen2.5-coder-3b:Q8_0
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:9931/v1 hermes config set model.default gaurav0506/rag-sft-qwen2.5-coder-3b:Q8_0
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use gaurav0506/rag-sft-qwen2.5-coder-3b with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf gaurav0506/rag-sft-qwen2.5-coder-3b:Q8_0
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:9931/v1 \ --custom-model-id "gaurav0506/rag-sft-qwen2.5-coder-3b:Q8_0" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
- From Pre-Training to Domain Mastery: An Empirical Triad Benchmark of Base, Domain-Adapted, and Instruction Fine-Tuned Language Models for Production RAG Engineering
- Abstract
- 1. Introduction & Research Problem
- 2. System Architecture & Training Trajectory
- 3. Quantitative Benchmark Scorecard (10 Automated Tasks)
- 4. End-to-End Architectural Systems Modules
- 5. Discussion: The Complementary Division of Labor
- 6. Limitations & Parameter Boundaries
- 7. Forthcoming Open-Source Release & Deployment Artifacts
- 8. Conclusion
- Abstract
From Pre-Training to Domain Mastery: An Empirical Triad Benchmark of Base, Domain-Adapted, and Instruction Fine-Tuned Language Models for Production RAG Engineering
Comparative Evaluation of Qwen2.5-Coder-3B Foundation, RAG-CPT-3B, and RAG-SFT-3B
Date: November 19, 2026
Author / Lead: Gaurav
Affiliation: Open-Source RAG Dataset Research
Hardware Platform: NVIDIA GeForce RTX 4070 Laptop GPU (8,188 MiB VRAM, Compute Capability 8.9)
Execution Environment: Local Ollama Inference & Standalone Transformers bfloat16 Pipeline
Publication PDF Artifact: /home/gaurav/Desktop/rag-dataset-repos/dataset_project/reports/finetune/RAG_Triad_Benchmark_Research_Paper.pdf
Publication Chart Artifact: /home/gaurav/Desktop/rag-dataset-repos/dataset_project/reports/finetune/triad_benchmark_comparison_chart.png
Abstract
Modern Retrieval-Augmented Generation (RAG) architectures require precise adherence to rapidly evolving vector SDKs, mathematically rigorous ranking formulas, and robust data structure idioms. Small open-weights code models (<= 3B parameters) are prime candidates for private, latency-critical enterprise deployment, yet foundation base models exhibit severe failure modes including embedding-hash hallucinations, infinite repetition loops, and deprecated library calls.
This research paper presents an end-to-end comparative evaluation across the complete three-tier training continuum:
- Base Foundation:
Qwen2.5-Coder-3B - Domain-Adaptive Continued Pretraining (CPT):
rag-cpt-3b(trained on an 8.19M token curated production RAG corpus) - Supervised Instruction Fine-Tuning (SFT):
rag-coder-3b(trained on 3,000 repository-grounded, provenance-verified instruction-response pairs)
We subject all three models to an autonomous 10-task engineering benchmark and 3 core production architectural modules under strict zero-human-intervention conditions. Our empirical results demonstrate a decisive evolutionary trajectory: Domain CPT successfully instills modern SDK namespaces (eliminating baseline Qdrant and BM25 crashes) but retains pretraining artifacts (truncation and trailing completion leakage). Subsequent instruction fine-tuning achieves complete mastery: 100% Python AST syntactic validity, zero infinite repetition loops, rigorous ChatML termination discipline, and a +24.18 percentage point jump in token accuracy (50.67% to 74.85%). We release full comparative code diffs, convergence profiles, and exported 3.1 GB Q8_0 GGUF binaries for standalone local serving.
1. Introduction & Research Problem
Production Retrieval-Augmented Generation (RAG) applications demand strict adherence to algorithmic rigor and rapidly evolving APIs. While large frontier models (e.g., GPT-4o, Claude 3.5 Sonnet) possess general competency, deploying small parameter models (3B) locally in private enterprise environments or autonomous coding agents presents persistent challenges:
- Structural Data-Structure Hallucination: Foundation models frequently attempt to hash multi-dimensional NumPy embeddings directly as dictionary keys (
cache[query_vector] = answer), causing fatal runtime crashes (TypeError: unhashable type: 'numpy.ndarray'). - API Deprecation Skew: Foundation models conflate legacy and modern SDK namespaces, inventing non-existent classes like
qdrant_client.FieldValueor calling deprecatedpydantic.v1.BaseModel. - Autoregressive Degeneration: When prompted for complex hybrid search modules, foundation models often collapse into infinite repetition loops, emitting 50+ identical import lines with zero implementation code.
To resolve these deficiencies, we investigate whether a two-phase adaptation strategy—Domain-Adaptive Continued Pre-Training (CPT) followed by Supervised Instruction Fine-Tuning (SFT)—can transform a general 3B foundation coder into a specialized, production-ready RAG systems engineer.
2. System Architecture & Training Trajectory
flowchart LR
A["1. Base Foundation<br/>(Qwen2.5-Coder-3B)<br/>Web-Scale Multilingual Code"] -->|CPT: 8.19M Tokens<br/>500 Steps / 4h 11m| B["2. Domain CPT<br/>(rag-cpt-3b)<br/>Specialized RAG Corpus"]
B -->|SFT: 3,000 Pairs<br/>507 Steps / 2h 10m| C["3. Instruction SFT<br/>(rag-coder-3b)<br/>ChatML Conversational Master"]
2.1 Tier 1: Base Foundation Model
- Architecture: Qwen2.5-Coder-3B Causal LM (3.09B parameters, 36 layers, hidden dimension 2048, 16 query heads / 2 key-value heads, RoPE context up to 32k/128k).
- Baseline Behavior: High generic Python syntax fluency, but completely unaware of modern RAG architectural idioms and susceptible to catastrophic looping.
2.2 Tier 2: Domain-Adaptive Continued Pre-Training (CPT)
- Training Corpus: 8.19M deduplicated tokens across leading open-source RAG frameworks (LlamaIndex, LangChain, Haystack, Qdrant, Chroma, FastEmbed, GraphRAG, Ragas, AutoRAG).
- Optimization: 4-bit QLoRA ($r=16, \alpha=32$), target modules (
q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj), batch size 1 with 16 gradient accumulation steps, paged AdamW 8-bit optimizer. - Convergence: 500 steps, cross-entropy loss decreased from 1.54 down to 1.09, validation loss stabilized at 1.028.
- Result: Successfully internalized modern SDK signatures and mathematical formulas, but exhibited base-model completion artifacts (trailing leakage and truncation).
2.3 Tier 3: Supervised Instruction Fine-Tuning (SFT)
- Dataset: 3,000 synthetic, repository-grounded instruction-response pairs across 30 batches (2,700 training / 300 validation split, 90/10 stratified). 100% provenance-verified from live RAG codebases with 0 duplicate objectives.
- Hyperparameters: 3 full epochs (507 optimizer steps), learning rate $2.0 \times 10^{-4}$ with cosine decay, 50 warmup steps, max sequence length 2048 tokens, FP16/BF16 mixed precision.
- Convergence:
- Initial Loss:
2.667(token accuracy50.67%) - Intermediate Step 250: Loss
1.042(token accuracy74.62%) - Final Step 507: Training Loss
0.784, Validation Loss1.137, Token Accuracy74.85% - Gradient Norm: Stably maintained between
0.3and0.5throughout.
- Initial Loss:
- Result: Complete operational stabilization, 100% AST compliance, rigid ChatML boundary adherence, and zero infinite loops.
3. Quantitative Benchmark Scorecard (10 Automated Tasks)
All three models autonomously generated Python scripts across 10 distinct architectural challenges. Each script was evaluated via Python AST compilation and dynamic subprocess execution with a 6.0-second timeout.
| # | Task Identifier | Target Capability | Base (qwen2.5-coder:3b) |
Domain CPT (rag-cpt-3b) |
Instruction SFT (rag-coder-3b) |
Architectural Diagnosis |
|---|---|---|---|---|---|---|
| 01 | task_01_token_chunker |
Sliding window token chunking with overlap | PASS (0.018s) | TIMEOUT (>6.0s) | PASS (0.020s) | CPT instantiated slow HuggingFace tokenizer; SFT generated clean whitespace sliding window. |
| 02 | task_02_bm25_retriever |
Pure Python BM25Okapi inverted index | FAIL (KeyError) |
PASS (0.015s) | PASS (AST & Math) | Base failed to store corpus; CPT and SFT implemented exact Robertson-Spärck Jones IDF formulas. |
| 03 | task_03_rrf_fusion |
Reciprocal Rank Fusion across multi-lists | BROKEN (Math Bug) | FAIL (NameError) |
PASS (0.021s) | Base scored only doc[0] in loop; CPT missed helper; SFT executed complete multi-list rank aggregation. |
| 04 | task_04_qdrant_hybrid |
Qdrant payload metadata filter search | FAIL (Hallucination) | PASS (Canonical API) | PASS (Canonical API) | Base invented FieldValue; CPT and SFT used modern qdrant_client.models.Filter namespace. |
| 05 | task_05_cross_encoder_rerank |
Two-stage re-ranking pipeline | FAIL (ImportError) |
Partial (Model Import) | PASS (0.033s) | Base confused bi-encoder with cross-encoder; SFT produced complete runnable 2-stage scoring pipeline. |
| 06 | task_06_query_expansion |
Multi-query & HyDE transformation | TRIVIAL (Static dummy) | SYNTAX FAIL (Leakage) | PASS (0.020s) | CPT leaked unquoted prompt text at EOF; SFT generated 88-line structured QueryTransformer class. |
| 07 | task_07_ndcg_math |
Pure math DCG, IDCG, and nDCG@K | FAIL (ZeroDivision) |
FAIL (ImportError) |
PASS (AST & Logic) | Base sorted retrieved items and crashed on log2(1)=0; SFT wrote pure zero-dependency logarithmic math. |
| 08 | task_08_semantic_cache |
In-memory embedding cosine semantic cache | FAIL (TypeError) |
PASS (0.024s) | PASS (AST & Vectors) | Base attempted dict[np.ndarray]=ans (unhashable array); CPT and SFT used vector tuple lists + cosine. |
| 09 | task_09_parent_child |
Hierarchical chunker & resolver | FAIL (KeyError) |
FAIL (NameError 'i') |
PASS (0.012s) | Base had invalid yield inside list function; CPT had loop leak; SFT passed execution cleanly. |
| 10 | task_10_faithfulness_metric |
Grounded claim overlap & faithfulness | FAIL (TypeError) |
SYNTAX FAIL (Truncated) | PASS (0.023s) | Base crashed on sum(int); CPT truncated after 11 lines; SFT generated complete dataclass output. |
Summary Statistics
| Evaluation Dimension | Base (qwen2.5-coder:3b) |
Domain CPT (rag-cpt-3b) |
Instruction SFT (rag-coder-3b) |
|---|---|---|---|
| Python AST Syntactic Validity | 100% (10/10) | 80% (8/10) | 100% (10/10) |
| Dynamic Execution / Math Pass Rate | 30% (3/10) | 30% (3/10) | 60% (6/10) |
| Modern Vector SDK Accuracy | 30% | 90% | 95% |
| Data Structure Validity (No Array Hashes) | 20% | 90% | 95% |
| Autoregressive Repetition Loop Rate | High (50+ repeated lines) | Low (config repetition) | 0.0% (Zero Loops) |
| Completion Truncation / Leakage Rate | Moderate | High (EOF leakage & 11-line cutoff) | 0.0% (Zero Leakage) |
4. End-to-End Architectural Systems Modules
In addition to the 10 micro-benchmarks, we evaluated full module synthesis across 3 core production files:
indexer.py: Chunking + FastEmbed dense embedding + Qdrant collection upsert.hybrid_retriever.py: Inverted BM25 index + Qdrant dense search + Reciprocal Rank Fusion (RRF).evaluator.py: Standardized evaluation metrics (Recall@K, MRR, nDCG@K).
Case Study 1: indexer.py (Ingestion & Vector DB Setup)
- Base Model (83 lines): Attempted to construct
DocumentChunk(text=chunk, chunk_id=f"{document_id}_{start}", document_id=document_id)withoutdocument_idin function scope, immediately crashing withNameError. Used deprecatedfrom pydantic.v1 import BaseModel. - CPT Model (59 lines): Used canonical
qdrant_client.models(VectorParams,Distance,FieldSchema) and Pydantic v2, but left the chunking algorithm body aspass. - SFT Model (62 lines, AST Valid): Complete, production-grade implementation. Defined typed Pydantic v2
DocumentChunkwith metadata dictionaries, implemented a fully functionalrecursive_character_chunker, and batched vector upserts into Qdrant usingfastembed.TextEmbedding.
Case Study 2: hybrid_retriever.py (Hybrid Search & RRF)
- Base Model (229 lines): Suffered total autoregressive collapse. Emitted 229 lines repeating the same 4 FastAPI imports over 50 times with zero RAG code.
- CPT Model (113 lines): Correctly identified production toolstack (
qdrant_client.models,fastembed,rank_bm25), but truncated during config initialization. - SFT Model (77 lines, AST Valid): Generated clean, modular architecture featuring an internal
BM25Index, dense FastEmbed vector query routing, exact RRF score aggregation ($RRF(d) = \sum \frac{1}{k + rank}$), and runnable unit tests.
Case Study 3: evaluator.py (Recall, MRR, nDCG)
- Base Model (68 lines): Called
retrieved_ids.intersection(relevant_ids)on a Pythonlist, raising fatalAttributeError: 'list' object has no attribute 'intersection'. Indentedrank += 1outside document loop, corrupting MRR math. - CPT Model (65 lines): Corrected set operations (
retrieved_set & relevant_set) and mathematical formulas. - SFT Model (99 lines, AST Valid): Clean, modular architecture implementing
recall_at_k, $O(N)$mean_reciprocal_rank,dcg_at_k,ideal_dcg_at_k,ndcg_at_k, and a comprehensiveRetrievalEvaluatorclass.
5. Discussion: The Complementary Division of Labor
Our empirical evaluation establishes a foundational insight for domain-specialized small language models: CPT and SFT serve fundamentally distinct and complementary purposes.
graph TD
subgraph Phase 1: Continuous Pre-Training
CPT1["Absorbs Domain Lexicon"]
CPT2["Internalizes Modern SDK Namespaces"]
CPT3["Grounds Mathematical Ranking Formulas"]
end
subgraph Phase 2: Supervised Fine-Tuning
SFT1["Enforces ChatML Rigid Boundaries"]
SFT2["Eliminates Repetition Loops & Truncation"]
SFT3["Ensures Syntactic Completeness (100% AST)"]
end
Phase 1 --> Phase 2
- CPT is the Knowledge Engine: CPT successfully updates the internal weight representations with modern library specifications (e.g.
qdrant_client.models), replacing obsolete training data. Without CPT, SFT would require significantly more examples to unlearn baseline hallucinations. - SFT is the Operational Governor: Pretraining alone leaves models without conversational boundaries, leading to premature truncation, unquoted prompt echoes at EOF, and runaway token generation. SFT instills the structural discipline necessary for production execution.
6. Limitations & Parameter Boundaries
While RAG-SFT-3B delivers outstanding architectural specialization and eliminates baseline crashes, several practical limitations should guide deployment:
- Model Capacity & Multi-Step Agentic Reasoning: At 3 billion parameters, the model is designed primarily for precision code synthesis, refactoring, and single-turn to few-turn architectural RAG queries. It is not intended to match 70B+ frontier models in autonomous agentic loops involving simultaneous management of 20+ disparate tool definitions. In multi-agent frameworks (such as Hermes), prompt bloat from expansive agent rules and broad tool schemas can induce context drift unless toolsets are strictly scoped.
- Context Compaction in Long-Horizon Dialogues: While the model supports extensive context windows (up to 32k/65k tokens), extended dialogues (>16k tokens) benefit from proactive context compaction to avoid subtle instruction degradation.
- External Service Runtime Dependencies: Generated code targeting production vector databases (such as Qdrant or Milvus) or distributed message brokers (such as Apache Kafka) requires active local daemon instances or in-memory fallback modes (
:memory:) to execute without network connection errors. - Specialized Inductive Bias: High specialization on Python RAG pipelines may result in marginal capability regression on non-Python or non-retrieval programming tasks (e.g., frontend styling or systems C++), making task routing beneficial in heterogeneous production environments.
7. Forthcoming Open-Source Release & Deployment Artifacts
To foster open, reproducible research and accelerate community advancement in local, air-gapped RAG engineering, all model checkpoints, merged weights, curated instruction datasets, and quantized GGUF binaries developed in this work will be open-sourced soon under a permissive Apache-2.0 community license on Hugging Face and GitHub:
- HuggingFace Model Repository (Coming Soon): Standalone merged bfloat16 weights (
models/rag-sft-qwen2.5-coder-3b-merged, 5.8 GB) ready forvLLMand Transformers. - Quantized GGUF Binaries (Coming Soon): Optimized
Q8_0(3.1 GB) andQ4_K_Mbinaries for immediate deployment on local edge hardware viallama.cppand Ollama. - Curated 3,000-Pair SFT Dataset (Coming Soon): Full repository-grounded, provenance-verified train/validation JSONL splits with comprehensive licensing metadata.
- PEFT LoRA Adapters (Coming Soon): Lightweight adapter checkpoint (
checkpoints/sft_qwen2.5_coder_3b_qlora, 115 MB) for downstream community adaptation.
Local Deployment Quickstart (Available Immediately on Local System)
# Deploy instantly to Ollama from local GGUF
cd /home/gaurav/Desktop/rag-dataset-repos/dataset_project/models
ollama create rag-coder-3b -f Modelfile.rag-sft-3b
ollama run rag-coder-3b
8. Conclusion
By coupling Domain-Adaptive Continued Pre-Training with rigorously audited Supervised Instruction Fine-Tuning, we have successfully transformed Qwen2.5-Coder-3B into an expert, production-grade RAG systems engineer. The fine-tuned model completely eliminates the catastrophic repetition and syntactic crashes of the foundation baseline, achieves 100% Python AST pass rates, and provides an enterprise-ready, locally servable AI assistant for advanced retrieval workflows.
- Downloads last month
- 54