RAG-CPT-Qwen2.5-Coder-3B (Domain-Adaptive Continued Pre-Trained Model)

This model is the continuous pre-trained (CPT) checkpoint of Qwen2.5-Coder-3B, adapted specifically for production Retrieval-Augmented Generation (RAG) architectures across an 8.19M token domain corpus.

Pre-Training Details

  • Base Architecture: Qwen2.5-Coder-3B (3.09B parameters)
  • Pre-Training Corpus: 8.19M deduplicated tokens from leading open-source RAG frameworks (LlamaIndex, LangChain, Haystack, Qdrant, Chroma, FastEmbed, GraphRAG, Ragas, AutoRAG).
  • Optimization: 4-bit QLoRA, paged AdamW 8-bit, 500 steps. Cross-entropy loss converged from 1.54 to 1.09.
  • Format: Merged bfloat16 standalone weights + high-precision GGUF binary for local edge serving.

Related Resources

Downloads last month
215
Safetensors
Model size
3B params
Tensor type
BF16
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for gaurav0506/rag-cpt-qwen2.5-coder-3b

Base model

Qwen/Qwen2.5-3B
Finetuned
(95)
this model

Dataset used to train gaurav0506/rag-cpt-qwen2.5-coder-3b