| --- |
| license: apache-2.0 |
| --- |
| |
|
|
| <p align="center">Clinical + AI β An Open-Source Library for Clinical Diagnosis Support</p> |
|
|
| A GPU-based LLM fine-tuning and serving pipeline. This document describes the project's **LLM model composition** from the perspectives of base model, training, data, and inference/serving. |
|
|
| --- |
|
|
| ## 1. Model Composition Overview |
|
|
| | Category | Details | |
| |------|------| |
| | Base model | **Llama 3.1 8B Instruct** | |
| | Extended pipeline | **Qwen3-32B** multi-phase (SFT β DPO β GRPO β Export) | |
| | Fine-tuning method | LoRA (PEFT) | |
| | Deployed models | LoRA adapter / base+LoRA merged model | |
| | Distribution channel | HuggingFace Hub β `the-platforms/MediCPX` | |
| | Domain | CPX medical (clinical diagnosis) instruction-following | |
|
|
| **Two pipeline tracks** |
|
|
| - **Llama single pipeline** β `FineTuningPipeline`: preprocessing β tokenization β LoRA training β save |
| - **Qwen3 multi-phase pipeline** β `MultiPhaseFineTuningPipeline`: SFT β DPO β GRPO β Export |
|
|
| --- |
|
|
| ## 2. Base Model & Weights |
|
|
| At the end of training, artifacts are produced under a per-run timestamped directory. |
|
|
| ``` |
| {output_dir}/{yyyymmddhhmm}/final_model/ |
| βββ adapter/ # LoRA adapter (tens of MB) β for shared-base serving |
| β βββ adapter_config.json |
| β βββ adapter_model.safetensors |
| βββ merged/ # base + LoRA merged model (tens of GB) β for standalone deployment |
| β βββ config.json |
| β βββ model.safetensors |
| βββ model_info.json # base model, training config, and path metadata |
| ``` |
|
|
| | Format | Size | Loading | Use | |
| |------|------|------|------| |
| | `adapter/` | Small | `PeftModel.from_pretrained(base, adapter)` | Experiments & versioning | |
| | `merged/` | Large | `AutoModelForCausalLM.from_pretrained(merged)` | Dependency-free standalone serving | |
|
|
| **HuggingFace Hub upload** β `runs/upload_to_hf.py` |
|
|
| ```bash |
| export HF_TOKEN=hf_xxx |
| python runs/upload_to_hf.py --folder-path /ai_models/merged --repo-id the-platforms/MediCPX |
| ``` |
|
|
| - Training-only state (`optimizer.pt`, `scheduler.pt`, `rng_state*`) is excluded by default |
| - Details: [`llama/huggingface_upload.md`](./llama/huggingface_upload.md) |
|
|
| --- |
|
|
| ## 3. Training Configuration |
|
|
| **4-step flow**: preprocessing β tokenization β LoRA training β save |
|
|
| **Key hyperparameters** |
|
|
| | Item | Value | |
| |------|-----| |
| | LoRA rank | `r = 8` (Ξ± = r Γ 2) | |
| | Learning rate | `2e-4` | |
| | max_length | `2048` | |
| | Precision | bf16 (auto) | |
| | Other | Response-region loss masking + sequence packing | |
| |
| **Key code paths** |
| |
| | Category | Path | |
| |------|------| |
| | Pipeline package | `app/finetuning/` | |
| | Llama single pipeline | `app/finetuning/pipeline.py` (`FineTuningPipeline`) | |
| | Qwen3 multi-phase | `app/finetuning/pipeline.py` (`MultiPhaseFineTuningPipeline`) | |
| | LoRA trainer | `app/finetuning/training/llama_trainer.py` | |
| | GRPO trainer (TRL) | `app/finetuning/training/grpo_trainer.py` | |
| | Reward functions | `app/finetuning/training/rewards/` (`AgentReward`, `SIDomainReward`) | |
| | LLaMA-Factory integration | `app/finetuning/llamafactory/` | |
|
|
| **Run** |
|
|
| ```bash |
| # Llama LoRA fine-tuning |
| python runs/run_finetuning.py --model-path /ai_models --data-path /data/output/1105 --epochs 3 |
| # Multi-GPU (DDP) |
| torchrun --nproc_per_node=2 runs/run_finetuning.py --model-path /ai_models --data-path /data/output/1105 |
| # Qwen3 multi-phase |
| python runs/run_qwen3_finetuning.py |
| ``` |
|
|
| Details: [`llama/process.md`](./llama/process.md) Β· [`qwen3_finetuning_process.md`](./qwen3_finetuning_process.md) |
|
|
| --- |
|
|
| ## 4. Dataset Configuration |
|
|
| PDF source documents β instruction-response JSON generation (4-Stage). |
|
|
| ``` |
| PDFLoader(PyMuPDF4LLM) β MarkdownConverter β InstructionParser(GPT-4) β BatchedJSONWriter |
| ``` |
|
|
| **Output schema** (identical to training input) |
|
|
| ```json |
| [{ "instruction": "...", "context": "(optional)", "response": "..." }] |
| ``` |
|
|
| **Data composition by training method** |
|
|
| | Method | Format | Fields | |
| |------|------|------| |
| | SFT | ShareGPT `messages` | system/user/assistant β one gold answer | |
| | DPO | `sharegpt_dpo` (ranking) | `conversations` + `chosen` + `rejected` | |
| | GRPO | Prompt + reward function | No gold answers or pairs needed; scored via `RewardFunction.compute()` | |
|
|
| **Code paths** |
|
|
| | Category | Path | |
| |------|------| |
| | Preprocessing package | `app/data_handling/data_pre_processing/` | |
| | Qwen3 data pipeline | `app/data_handling/` (chunker, qa_generation, quality, ragas_eval, trajectory) | |
| | Entry points | `runs/run_cpx_processing.py`, `runs/run_qwen3_data_pipeline.py` | |
|
|
| ```bash |
| INPUT_PATH=/data/input/cpx.pdf OUTPUT_PATH=/data/output/1105 python runs/run_cpx_processing.py |
| ``` |
|
|
| Details: [`llama/data_preprocessing.md`](./llama/data_preprocessing.md) |
|
|
| --- |
|
|
| ## 5. Inference & Serving |
|
|
| | Category | Path | Description | |
| |------|------|------| |
| | Lightweight inference server | `server.py` | Single-model startup; `/generate`, `/stream`, `/health` | |
| | Full pipeline API | `main.py` | Integrated model load/training/inference; OpenAI-compatible chat | |
| | vLLM serving | `app/serving/vllm_config.py` | For Qwen3 Agent; Hermes tool parser | |
| | Model load helper | `runs/load_llama_3_1_8b.py` | Llama 3.1 8B load verification | |
|
|
| ```bash |
| # Lightweight inference server |
| MODEL_DIR=/path/to/model USE_4BIT=1 uvicorn server:app --host 0.0.0.0 --port 8080 |
| # vLLM (with tool calling) |
| ./scripts/serve_qwen3_vllm.sh |
| ``` |
|
|
| **OpenAI-compatible REST API** β two entry points |
|
|
| `main.py` β Full Pipeline API (`:8000`) |
|
|
| | Category | Endpoints | |
| |------|-----------| |
| | Health | `GET /health` | |
| | Model | `POST /api/v1/models/load`, `GET /api/v1/models/{id}/verify` | |
| | Preprocess | `POST /api/v1/preprocessing/run` | |
| | Tokenize | `POST /api/v1/tokenization/tokenize` | |
| | Training | `POST /api/v1/training/start`, `GET /api/v1/training/{job_id}/status` | |
| | Multi-Phase | `POST /api/v1/training/multi-phase/start` | |
| | Data Pipeline | `POST /api/v1/data-pipeline/generate-qa` | |
| | Inference | `POST /api/v1/inference/load`, `GET /api/v1/inference/models` | |
| | Chat (OpenAI-compatible) | `POST /api/v1/chat/completions`, `POST /v1/chat/completions` | |
|
|
| `server.py` β Inference API (`:8080`): `POST /generate`, `POST /stream` (SSE), `GET /health` |
|
|
| ```bash |
| ./scripts/api_server.sh start # start main.py |
| ``` |
|
|
| Request examples: [`API_SAMPLE.md`](./API_SAMPLE.md) |
|
|
| --- |
|
|
| ## 6. Evaluation |
|
|
| | Category | Path | |
| |------|------| |
| | Agent evaluation | `runs/eval_agent.py` | |
| | Domain evaluation | `runs/eval_domain.py` | |
| | Model evaluation | `runs/eval_model.py` | |
| | NLG surface metrics (BLEU/ROUGE/METEOR/BERTScore) | `runs/eval_nlg.py` | |
| | RAGAS evaluation | `runs/eval_ragas.py` | |
| | Model comparison | `runs/eval_compare.py` | |
|
|
| Details: [`evaluation.md`](./evaluation.md) Β· [`llama/evaluation_usage.md`](./llama/evaluation_usage.md) Β· [`llama/evaluation_runbook.md`](./llama/evaluation_runbook.md) |
|
|
| --- |
|
|
| ## 7. Demo |
|
|
| | Category | Path | Description | |
| |------|------|------| |
| | Medical chat loop demo | `scripts/medical_chat_loop.sh` | Calls the chat API in parallel with N random medical questions (repeats periodically) | |
| | Question set | `scripts/medical_questions.txt` | Question pool for demo input | |
| | System prompt | `scripts/medical_system_prompt.txt` | System prompt for the demo | |
| | Frontend integration guide | [`frontend_request_cancel_guide.md`](./frontend_request_cancel_guide.md) | Request/cancel integration | |
|
|
| ```bash |
| # After starting the inference server |
| API_URL=http://localhost:8000 MODEL=MediCPX ./scripts/medical_chat_loop.sh |
| ``` |
|
|
| --- |
|
|
| ## Appendix β Deliverables Inventory |
|
|
| | # | Item | Key Location | Form | |
| |---|------|-----------|------| |
| | 1 | Model weights | `final_model/`, HF Hub `the-platforms/MediCPX` | LoRA adapter / merged model | |
| | 2 | Training code | `app/finetuning/`, `runs/run_finetuning.py` | Python package + entry point | |
| | 3 | Inference code | `server.py`, `main.py`, `app/serving/` | FastAPI + vLLM | |
| | 4 | Dataset | `app/data_handling/`, `runs/run_cpx_processing.py` | PDF β instruction JSON | |
| | 5 | Demo | `scripts/medical_chat_loop.sh` | CLI chat demo | |
| | 6 | API | `main.py` (Full) / `server.py` (Inference) | OpenAI-compatible REST | |
| | 7 | Technical docs | `docs/` | Markdown + papers/patents | |
|
|
| **Technical documentation** |
|
|
| | Category | Document | |
| |------|------| |
| | Project overview | [`README.md`](./README.md), [`QUICKSTART.md`](./QUICKSTART.md) | |
| | End-to-end summary | [`llama/end_to_end_summary.md`](./llama/end_to_end_summary.md) | |
| | Data preprocessing | [`llama/data_preprocessing.md`](./llama/data_preprocessing.md) | |
| | Training walkthrough | [`llama/process.md`](./llama/process.md) | |
| | Qwen3 multi-phase | [`qwen3_finetuning_process.md`](./qwen3_finetuning_process.md) | |
| | Evaluation | [`evaluation.md`](./evaluation.md), [`llama/evaluation_usage.md`](./llama/evaluation_usage.md), [`llama/evaluation_runbook.md`](./llama/evaluation_runbook.md) | |
| | Improvement history | [`llama/improvements.md`](./llama/improvements.md) | |
| | Research & papers | [`llama/research.md`](./llama/research.md), [`llama/papers.md`](./llama/papers.md) | |
| | HF upload | [`llama/huggingface_upload.md`](./llama/huggingface_upload.md) | |
| | Deployment | [`deployments/README.md`](./deployments/README.md) (Docker, K3s) | |
| | Patent/paper drafts | `docs/paper/` (CPX medical-domain IMRaD, patent application drafts & figures) | |
|
|
