RAG-CPT-Qwen2.5-Coder-3B (Domain-Adaptive Continued Pre-Trained Model)
This model is the continuous pre-trained (CPT) checkpoint of Qwen2.5-Coder-3B, adapted specifically for production Retrieval-Augmented Generation (RAG) architectures across an 8.19M token domain corpus.
Pre-Training Details
- Base Architecture: Qwen2.5-Coder-3B (3.09B parameters)
- Pre-Training Corpus: 8.19M deduplicated tokens from leading open-source RAG frameworks (LlamaIndex, LangChain, Haystack, Qdrant, Chroma, FastEmbed, GraphRAG, Ragas, AutoRAG).
- Optimization: 4-bit QLoRA, paged AdamW 8-bit, 500 steps. Cross-entropy loss converged from 1.54 to 1.09.
- Format: Merged bfloat16 standalone weights + high-precision GGUF binary for local edge serving.
Related Resources