README.md

---
license: mit
library_name: pytorch
tags:
  - reinforcement-learning
  - multi-agent-reinforcement-learning
  - deep-reinforcement-learning
  - dqn
  - dueling-dqn
  - multi-agent-systems
  - drone-swarm
  - fault-tolerance
  - domain-randomization
---

# Multi-Drone Cooperative Search with Random Faults — trained policy

Trained weights for the Multi-Agent Systems course project (MSc Artificial
Intelligence, University of Bologna, A.Y. 2025/26).

A team of `N` drones searches a 32×32 occupancy grid for one hidden target under
partial observability, range-limited communication and a fixed time budget,
while **any drone can permanently break at any moment**, with no central
authority to announce the loss. A single Dueling Double DQN with parameter
sharing drives every drone; per-episode domain randomization plus a compact
context vector make that one policy a generalist across team size, sensing,
communication range, obstacle density and fault rate.

- **Code (GitHub mirror):** https://github.com/simoswish02/Multi-Agent-Systems-Project
- **Code (course repository, GitLab):** https://dvcs.apice.unibo.it/pika-lab/courses/ai-ethics/projects/rimondi2526-mas
- **Report (49 pp.):** [`Rimondi-MultiDroneSearch-MAS.pdf`](https://github.com/simoswish02/Multi-Agent-Systems-Project/blob/main/report/Rimondi-MultiDroneSearch-MAS.pdf)
- **Video walkthrough (~4 min):** https://youtu.be/8dnRG6ZneP8

## Files

| File | Description |
|---|---|
| `mas_50k_dr_faults_ep50000.pt` | Final checkpoint of the 50,000-episode run. **Every number in Section 7 of the report comes from this file.** This is the one to download for normal use. |
| `checkpoints/mas_50k_dr_faults_ep*.pt` | The 20 periodic snapshots taken every 2,500 episodes, including a copy of the final one. Needed only to reproduce the learning curve (`--axis epoch`), which re-evaluates each of them; ~5.4 GB in total. |

## Usage

```bash
git clone https://github.com/simoswish02/Multi-Agent-Systems-Project
cd Multi-Agent-Systems-Project
conda env create -f environment.yaml && conda activate rl-drone

# download the checkpoint into checkpoints/
hf download simoswish/MAS_MultiDroneExploration \
   mas_50k_dr_faults_ep50000.pt --local-dir checkpoints/

# watch it fly
python main.py --mode eval --checkpoint checkpoints/mas_50k_dr_faults_ep50000.pt \
               --fault-prob 0.003

# reproduce the experiments of Section 7
python testing/evaluate_policy.py --axis all --seeds 500 \
       --checkpoint checkpoints/mas_50k_dr_faults_ep50000.pt
```

Loading it directly:

```python
import torch
ckpt = torch.load("mas_50k_dr_faults_ep50000.pt", map_location="cpu")
state_dict = ckpt["q_net"]          # 22.4M parameters
# the file also carries optimizer / scheduler / epsilon state, so training
# can be resumed with `python main.py --mode train --resume <file>`
```

## Model

**Architecture** (`agents/networks.py`, report Section 4.4):

- **Global trunk** over six 32×32 belief planes (Visited, Obstacle, Trajectory,
  Target, Own\_Position, Broken): a ConvNeXt-style CNN, stages of width
  96 → 192 → 384 at resolutions 32 → 16 → 8, with per-stage FiLM conditioning,
  followed by a 3-layer / 8-head Transformer over the 8×8 = 64 spatial tokens
  plus a CLS token and a context token.
- **Local trunk** over a 13×13 egocentric crop with five channels: three
  stride-1 ConvNeXt blocks at width 96 (the middle one dilated) → 384.
- **Fusion and head:** concat (1024 + 384 + 5) → FC 1536 → 768 → 384 → dueling
  head, `Q = V + A − mean(A)`, over 4 actions (up / down / left / right).
- 22,445,183 parameters, float32.

**Context vector** (`CTX_DIM = 5`), rebuilt every round for every drone:
`[vision_radius, comm_range, n_agents, agent_id, n_alive_belief]`, each
normalized by the domain-randomization maxima (6, 12, 4, 4, 4). `agent_id`
breaks the parameter-sharing symmetry so co-located drones split up from step 0;
`n_alive_belief` is that drone's *own* belief about how many teammates remain —
decentralized, and possibly stale.

## Training

Parameter-sharing Dueling Double DQN: one shared network trained on the pooled
experience of the whole team, in a single phase of 50,000 episodes. Adam, weight
decay 1e-6, gradient clip 1.0, γ = 0.97, 3-step returns, Huber loss, uniform
replay of 2·10⁵ transitions, batch 16, one gradient step every 4 agent-turns,
hard target sync every 500 steps. ε decays 1.0 → 0.05 (×0.9995/episode, floor at
episode 5,990); the learning rate warms up over the first 10% of episodes to
5·10⁻⁴, then anneals cosine-wise to 10⁻⁶.

Per-episode domain randomization: vision radius ∈ [1, 6], communication range
∈ [2, 12], team size ∈ [1, 4], obstacle density ∈ [0.10, 0.30], per-turn fault
probability ∈ [0, 0.003]. The grid size (32) is the one constant.

## Results

At the reference operating point (N = 3, r_vis = 3, r_comm = 5, ρ = 0.2,
T = 200, no faults), 500 seeded instances, greedy (ε = 0):

| | |
|---|---|
| Success rate | **97.6%** (Wilson CI [95.9, 98.6]) |
| Map coverage | 53.4% |
| SPL | 0.715 |
| Time to find | 53.9 rounds |

Generalization (success %, endpoints of each sweep, 500 seeds per point):

| Axis | In-distribution | OOD probe |
|---|---|---|
| Team size N | 79.0 (N=1) → 97.6 (N=4) | 98.8, 99.0 (N = 5, 6) |
| Vision r_vis | 80.2 (r=1) → 96.6 (r=6) | 96.4, 95.6 (r = 7, 8) |
| Comm. r_comm | 96.8 (r=2) → 98.0 (r=12) | 95.6, 98.2, 98.8 (r = 0, 14, 16) |
| Density ρ | 99.0 (ρ=0.10) → 83.6 (ρ=0.30) | 99.2, 80.0, 86.0 (ρ = 0, 0.35, 0.40) |

No axis shows a cliff, in or out of distribution, including beyond the trained
maxima of team size, vision and communication. Obstacle density is the one
factor that genuinely governs difficulty, costing 15 points across its trained
range alone.

Fault robustness — the project's headline result:

| p_fault | 0 | 0.001 | 0.002 | 0.003 | 0.005 | 0.010 |
|---|---|---|---|---|---|---|
| Success [%] | 97.6 | 96.0 | 93.6 | 91.0 | 85.4 | 72.6 |
| SPL | 0.715 | 0.695 | 0.682 | 0.666 | 0.631 | 0.536 |
| Drones lost (mean) | 0.00 | 0.16 | 0.30 | 0.48 | 0.78 | 1.37 |

Across the entire trained fault range (0 → 0.003) success falls by under seven
points while the team sheds, on average, close to half a drone; at more than
triple the trained maximum it still completes nearly three missions in four,
with no cliff. Conditioning on survivors instead: a team that finishes intact
succeeds 98.7% of the time, losing one drone costs six points (92.5%), and even
a lone survivor still finds the target in 65.9% of episodes.

## Limitations

- Simulation only: 4-connected grid moves, noiseless sensing, instantaneous
  range-limited fusion. No sim-to-real transfer is claimed or tested.
- Faults are i.i.d. across drones and turns and produce inert, non-blocking
  wrecks. Correlated failures, passage-blocking wrecks and an adversary
  choosing whom to disable are outside the model.
- Above ρ ≈ 0.30 the density sweep stops measuring the same task. The target is
  always drawn from the spawn's connected component, and at high clutter the
  free space fragments until that component holds under a quarter of the free
  cells, so the apparent rebound at ρ = 0.40 is a structurally shorter mission
  rather than a better policy (Section 7.3 of the report).
- Statistical, not formal, evidence: confidence intervals bound sampling error,
  not worst-case behaviour on an unseen instance.

Section 8 of the report discusses the dual-use profile of the capability, its
treatment under the GDPR and the EU AI Act, and why this specific system would
be unacceptable as an autonomous weapon.

## Citation

```bibtex
@misc{rimondi2026multidrone,
  author = {Rimondi, Simone},
  title  = {Multi-Drone Cooperative Search with Random Faults},
  year   = {2026},
  note   = {Multi-Agent Systems project, MSc Artificial Intelligence,
            University of Bologna},
  url    = {https://github.com/simoswish02/Multi-Agent-Systems-Project}
}
```

## License

MIT.