Blame for README.md

---
language:
- en
base_model:
- Qwen/Qwen3-1.7B
- Qwen/Qwen3-4B
- Qwen/Qwen3-8B
library_name: peft
pipeline_tag: text-generation
tags:
- lora
- multi-agent
- collaborative-learning
- jca
- qwen3
---

<h1 align="center">Reasoning across Sliced Models via Collaborative Learning</h1>

<p align="center">
  <a href="https://huggingface.co/SJTUFreshman/collaborativelearning/resolve/main/assets/figures/Figure1.pdf"><img src="https://huggingface.co/SJTUFreshman/collaborativelearning/resolve/main/assets/figures/previews/Figure1-1.png" alt="Collaborative Learning overview: model slicing, protocol distillation, and pyramid supervision" width="960"></a>
</p>

<p align="center">
  <a href="#-overview">Overview</a>
  &nbsp;·&nbsp;
  <a href="#-evaluation">Evaluation</a>
  &nbsp;·&nbsp;
  <a href="#-workflow">Workflow</a>
  &nbsp;·&nbsp;
  <a href="#-repository-layout">Repository layout</a>
  &nbsp;·&nbsp;
  <a href="#-quickstart">Quickstart</a>
  &nbsp;·&nbsp;
  <a href="#-code-and-experiment-entry-points">Code and experiment entry points</a>
</p>

## ✨ Overview

**Collaborative Learning (CL)** studies how sliced models can collaborate on reasoning while keeping the overall parameter budget comparable to a single larger model. The method combines model slicing, protocol distillation, and pyramid supervision so that models can build on shared task state across multiple reasoning steps.

This model repository releases the **rank-64 LoRA adapters for the heterogeneous CL main experiments**, using Qwen3-1.7B, Qwen3-4B, and Qwen3-8B. It covers four task families—multi-hop question answering, mathematics, code generation, and constrained generation—across MuSiQue, GSM-Hard, MATH, MultiPL-E, and Conifer. The [companion code repository](https://github.com/SJTUFreshman/tmp_x7k9p4r2d8m5c) contains the implementation, benchmark scorers, result records, configurations, and analysis artifacts.

<p align="center">
  <a href="https://huggingface.co/SJTUFreshman/collaborativelearning/resolve/main/assets/figures/Figure2.pdf">
    <img src="https://huggingface.co/SJTUFreshman/collaborativelearning/resolve/main/assets/figures/previews/Figure2-1.png" alt="Model slicing, protocol distillation, and pyramid supervision" width="920">
  </a>
</p>

## 🧪 Evaluation

We evaluate CL on four task families and five benchmarks:

| Task family | Benchmark | What it measures | Checkpoints |
| --- | --- | --- | --- |
| Multi-hop QA | [MuSiQue](https://github.com/StonyBrookNLP/musique) | Compositional retrieval and reasoning | [MuSiQue](https://huggingface.co/SJTUFreshman/collaborativelearning/tree/main/MuSiQue) |
| Mathematical reasoning | **GSM-Hard** | Robust numerical reasoning | [GSM-Hard](https://huggingface.co/SJTUFreshman/collaborativelearning/tree/main/GSM-Hard) |
| Mathematical reasoning | **MATH** | Competition-level problem solving | [MATH](https://huggingface.co/SJTUFreshman/collaborativelearning/tree/main/MATH) |
| Code generation | [MultiPL-E](https://github.com/nuprl/MultiPL-E) | Pass@1 across eight languages | [MultiPL-E](https://huggingface.co/SJTUFreshman/collaborativelearning/tree/main/MultiPL-E) |
| Constrained generation | **Conifer** | Coverage and explicit constraint satisfaction | [Conifer](https://huggingface.co/SJTUFreshman/collaborativelearning/tree/main/Conifer) |

### 📊 Results

The table below compares CL with the main single-model and multi-model baselines. All values are percentages reported in the paper, not a new evaluation of this weight release. MuSiQue, GSM-Hard, and MATH use **accuracy / F1**; MultiPL-E uses **weighted / macro-language pass@1**; Conifer uses **Coverage / Explicit**.

| Method | MuSiQue | GSM-Hard | MATH | MultiPL-E | Conifer |
| --- | ---: | ---: | ---: | ---: | ---: |
| Qwen3-14B | 39.55 / 49.70 | 68.94 / 80.69 | 74.60 / 82.86 | 55.18 / 57.34 | 88.00 / 95.72 |
| SE-RL | 41.76 / 52.73 | 68.18 / 79.67 | 76.00 / 80.70 | 60.43 / 62.55 | 88.00 / 95.81 |
| MAD | 39.51 / 50.59 | 66.67 / 74.18 | 72.40 / 81.32 | 55.03 / 55.35 | 88.55 / 95.84 |
| AgentVerse | 41.99 / 52.85 | 71.97 / 78.44 | 68.20 / 79.42 | 59.25 / 59.83 | 87.74 / 95.86 |
| AFlow | 41.83 / 50.36 | 68.18 / 76.04 | 65.00 / 76.53 | 42.83 / 42.46 | 87.92 / 96.51 |
| GPTSwarm | 36.20 / 47.37 | 70.45 / 78.39 | 69.20 / 79.90 | 51.63 / 52.43 | 88.17 / 95.96 |
| MAGRPO | 40.13 / 51.74 | 65.91 / 76.44 | 65.00 / 76.01 | 49.93 / 49.93 | 88.58 / 95.82 |
| MAPoRL | 40.59 / 52.74 | 63.64 / 75.34 | 70.80 / 80.87 | 53.25 / 53.27 | 86.37 / 89.19 |
| AT-GRPO | 41.54 / 53.51 | 64.39 / 76.86 | 74.20 / 83.70 | 57.91 / 58.74 | 86.60 / 95.85 |
| Homo. MAS | 26.40 / 32.82 | 71.21 / 79.93 | 68.20 / 76.41 | 56.36 / 57.20 | 88.02 / 94.27 |
| Homo. CL | 36.00 / 47.19 | 60.61 / 76.51 | 68.60 / 77.95 | 53.33 / 52.65 | 88.15 / 95.22 |
| **CL (ours)** | **42.37 / 53.43** | **71.97 / 81.15** | **78.40 / 85.25** | **59.39 / 59.63** | **88.90 / 95.90** |

<p align="center">
  <a href="https://huggingface.co/SJTUFreshman/collaborativelearning/resolve/main/assets/figures/Figure3.pdf">
    <img src="https://huggingface.co/SJTUFreshman/collaborativelearning/resolve/main/assets/figures/previews/Figure3-1.png" alt="Ablation study across the five benchmarks" width="760">
  </a>
</p>

The evaluation records contain 2,417 MuSiQue examples, 132 GSM-Hard examples, 500 MATH problems, 1,352 MultiPL-E test problems, and 1,402 Conifer trajectories.

CL is best or joint-best on **6 of 10 metrics**, improves over Qwen3-14B on all ten, and gains an average of **2.38 percentage points** over that single-model reference. On the MATH evaluation, CL scores **392/500 correct**, or **78.40% accuracy and 85.25% F1**, under the evaluation scorer.

### 📉 Efficiency and task performance

Figure 4 compares task performance with parameter-weighted output cost and average model invocations across the five benchmarks.

<p align="center">
  <a href="https://huggingface.co/SJTUFreshman/collaborativelearning/resolve/main/assets/figures/Figure4.pdf"><img src="https://huggingface.co/SJTUFreshman/collaborativelearning/resolve/main/assets/figures/previews/Figure4-1.png" alt="Computation cost versus task performance across five benchmarks" width="960"></a>
</p>

### 🧭 Emergent interaction topology

Figure 5 visualizes the task-dependent interaction topology. The model order changes with the task, allowing CL to draft, review, refine, enrich, or terminate as needed.

<p align="center">
  <a href="https://huggingface.co/SJTUFreshman/collaborativelearning/resolve/main/assets/figures/Figure5.pdf"><img src="https://huggingface.co/SJTUFreshman/collaborativelearning/resolve/main/assets/figures/previews/Figure5-1.png" alt="Task-dependent learned interaction topology" width="960"></a>
</p>

## 🧭 Workflow

```text
task query + current reasoning state
                  |
                  v
          sliced model pool
                  |
                  v
           protocol decision
          /        |          \
     continue   handoff     terminate
        |          |            |
        +------> next model    v
                              final answer
                                   |
                                   v
                          task-specific evaluator
```

At each step, the current task state is processed by one model from the sliced-model pool. The learned protocol can continue the current thread, hand the state to another model, or terminate with a final answer. Protocol distillation provides coordination traces, pyramid supervision scores the reasoning process and final output, and each benchmark supplies a task-specific evaluator.

## 📦 Repository layout

| Path | Contents |
| --- | --- |
| `<benchmark>/PD/<Qwen3-model>/<checkpoint>/` | Protocol distillation (SFT) adapters |
| `<benchmark>/SLS/<Qwen3-model>/<checkpoint>/` | Step-level supervision (RL) adapters |
| `assets/figures/` | Study figures and previews |
| `manifest.json` | Complete checkpoint list, source runs, sizes, SHA-256 hashes, and known missing checkpoints |
| `SHA256SUMS` | SHA-256 checksums for checkpoint assets |
| `PAPER_ALIGNMENT.md` | Final training-run mapping and evaluation checkpoint identities |
| `load_adapter.py` | Helper for loading either adapter layout |

The release contains **233 surviving checkpoints across 29 model-stage groups**, approximately **41.39 GiB** including tokenizer assets and configuration. Each selected final training run retains all its surviving `checkpoint-*` exports and its `final` export. Some checkpoints place the adapter in a nested `policy_adapter/` directory, with the tokenizer in the parent directory. MultiPL-E 1.7B is PD-only.

Weights are byte-for-byte unchanged. Adapter configurations use the corresponding `Qwen/Qwen3-*` base-model IDs, and JSON metadata has local server paths normalized and credential fields removed. Baseline, ablation, and homogeneous-model weights are outside this release, as are optimizer/RNG states, pickled training arguments, and training data.

## 🚀 Quickstart

### 1️⃣ Install the loading dependencies

```bash
python -m pip install -U torch transformers peft accelerate huggingface_hub
```

### 2️⃣ Download and load one adapter

```python
from huggingface_hub import snapshot_download
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

repo_id = "SJTUFreshman/collaborativelearning"
subfolder = "MuSiQue/SLS/Qwen3-1.7B/final"
snapshot = snapshot_download(repo_id, allow_patterns=[f"{subfolder}/*"])
adapter_path = f"{snapshot}/{subfolder}"

tokenizer = AutoTokenizer.from_pretrained(adapter_path)
base_model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen3-1.7B", torch_dtype="auto", device_map="auto"
)
model = PeftModel.from_pretrained(base_model, adapter_path)
```

Choose the matching base model recorded in `manifest.json`. For a nested `policy_adapter` checkpoint, load the adapter from that directory and the tokenizer from its parent; [load_adapter.py](load_adapter.py) handles both layouts. The frozen base-model weights are downloaded from the original Qwen repositories.

To download all published checkpoints:

```bash
hf download SJTUFreshman/collaborativelearning --local-dir collaborativelearning
```

### 3️⃣ Reproduce the collaborative evaluation

Loading one adapter does not run the collaborative system. Use the [companion code repository](https://github.com/SJTUFreshman/tmp_x7k9p4r2d8m5c) for the scheduler, task prompts, model endpoints, and benchmark scorers. Its Quickstart includes a CPU-only MATH summary from the bundled evaluation records and a fresh MATH evaluation using three OpenAI-compatible model endpoints. [PAPER_ALIGNMENT.md](PAPER_ALIGNMENT.md) records the final evaluation identities and the MATH answer-selection protocol.

## 🧰 Code and experiment entry points

The following paths are in the companion code repository:

| Area | Entry point |
| --- | --- |
| Shared CL utilities and analysis | [reproduction/code/jca/scripts/](https://github.com/SJTUFreshman/tmp_x7k9p4r2d8m5c/tree/main/reproduction/code/jca/scripts) |
| MATH CL training and evaluation | [reproduction/code/jca/experiments/math_specific_sft_rl_v1/](https://github.com/SJTUFreshman/tmp_x7k9p4r2d8m5c/tree/main/reproduction/code/jca/experiments/math_specific_sft_rl_v1) |
| Conifer pipeline | [reproduction/code/jca/conifer_training_hub/](https://github.com/SJTUFreshman/tmp_x7k9p4r2d8m5c/tree/main/reproduction/code/jca/conifer_training_hub) |
| MultiPL-E scoring | [reproduction/code/jca/Code/MultiPL-E/](https://github.com/SJTUFreshman/tmp_x7k9p4r2d8m5c/tree/main/reproduction/code/jca/Code/MultiPL-E) |
| Saved training records | [reproduction/training/](https://github.com/SJTUFreshman/tmp_x7k9p4r2d8m5c/tree/main/reproduction/training) |

A full run may require Python 3.10+, PyTorch/CUDA, Transformers, PEFT, Accelerate, vLLM, language runtimes, and benchmark-specific evaluators. Use the checked-in configurations and logs for seeds, model servers, and output conventions.

## License

No license is specified for these published adapters, at the owner's request. The upstream Qwen base-model licenses remain applicable to those models.