# Brain: Spiking Visuomotor Policy

## Model Description

**Brain** is a recurrent **Spiking Neural Network (SNN)** visuomotor policy trained end-to-end for robot manipulation through imitation learning.

The model processes:

* Dual RGB camera observations
* 7-dimensional proprioceptive joint state

and predicts a **7-dimensional action delta** (6 joints + gripper) at every timestep.

The policy is implemented entirely with spiking neural network components using **SpikingJelly**, employing Parametric Leaky Integrate-and-Fire (PLIF) neurons, recurrent temporal processing, and surrogate-gradient backpropagation.

The released checkpoint is the best-performing model trained using **delta-action prediction**, which significantly outperformed an equivalent absolute-action formulation.

---

## Model Details

| Property          | Value                               |
| ----------------- | ----------------------------------- |
| Architecture      | Recurrent Spiking Neural Network    |
| Parameters        | 1.65M                               |
| Framework         | PyTorch + SpikingJelly              |
| Neuron Type       | Parametric LIF (PLIF)               |
| Inputs            | Dual RGB images + 7-DoF joint state |
| Output            | 7-DoF action delta                  |
| Training Episodes | 407 teleoperated demonstrations     |
| Best Checkpoint   | `full_run_v1_delta` (step 3600)     |

---

## Architecture

```text
Global Camera ───────────┐
                          │
                          ▼
                  Spiking CNN Encoder
                          │
Gripper Camera ───────────┤
                          ▼
                  Spiking CNN Encoder
                          │
Joint State ──────────────┐
                          ▼
                Spiking MLP Encoder
                          │
                          ▼
            Recurrent PLIF Fusion Network
                          │
                          ▼
                    Linear Projection
                          │
                          ▼
             Non-Spiking LIF Readout
                          │
                          ▼
                7-DoF Action Delta
```

The network consists of independent spiking vision encoders for each camera, a spiking proprioceptive encoder, recurrent PLIF fusion layers, and a non-spiking LIF readout that produces continuous control outputs.

---

## Training Data

The model was trained using **407 teleoperated robot manipulation demonstrations** containing:

* Dual synchronized RGB camera streams
* 7-dimensional robot joint state
* Demonstrated robot actions

Training images were resized to **128×128**, and trajectories were divided into overlapping temporal windows of length **32** with stride **16**.

---

## Training Objective

Rather than predicting the next absolute joint configuration, the model predicts the **difference between the demonstrated action and the current joint position**.

This formulation substantially improved optimization by matching the scale of real per-step robot motion, resulting in significantly lower prediction error than direct absolute-action regression.

---

## Evaluation

Evaluation was performed offline on a held-out validation split using open-loop trajectory prediction.

### Best Checkpoint (`full_run_v1_delta`)

| Metric                   | Value           |
| ------------------------ | --------------- |
| MAE                      | **0.00989 rad** |
| RMSE                     | **0.01873 rad** |
| R²                       | **0.99917**     |
| Cosine Similarity        | **0.99960**     |
| Mean Episode Correlation | **0.99135**     |
| Final Pose Error         | **0.05417 rad** |

Compared to the absolute-action baseline:

| Metric           | Absolute | Delta      |
| ---------------- | -------- | ---------- |
| MAE              | 0.0834   | **0.0099** |
| RMSE             | 0.1278   | **0.0161** |
| Mean Correlation | 0.812    | **0.991**  |

---

## Intended Use

This model is intended for research in:

* Spiking neural networks
* Robot imitation learning
* Neuromorphic computing
* Multimodal visuomotor policies
* Temporal sequence modeling for robot control

---

## Limitations

The model has been evaluated **only in offline open-loop settings** against held-out demonstrations.

Reported metrics measure trajectory prediction accuracy rather than real-world task completion. Physical deployment or closed-loop simulation is required to assess manipulation performance.

---

## Citation

If you use this model in your research, please cite this repository.

```bibtex
@misc{brain2026,
  title={Brain: A Recurrent Spiking Neural Network for Visuomotor Control},
  author=Adith,
  year={2026},
  howpublished={Hugging Face}
}