|
|
|
|
| # Brain: Spiking Visuomotor Policy |
|
|
| ## Model Description |
|
|
| **Brain** is a recurrent **Spiking Neural Network (SNN)** visuomotor policy trained end-to-end for robot manipulation through imitation learning. |
|
|
| The model processes: |
|
|
| * Dual RGB camera observations |
| * 7-dimensional proprioceptive joint state |
|
|
| and predicts a **7-dimensional action delta** (6 joints + gripper) at every timestep. |
|
|
| The policy is implemented entirely with spiking neural network components using **SpikingJelly**, employing Parametric Leaky Integrate-and-Fire (PLIF) neurons, recurrent temporal processing, and surrogate-gradient backpropagation. |
|
|
| The released checkpoint is the best-performing model trained using **delta-action prediction**, which significantly outperformed an equivalent absolute-action formulation. |
|
|
| --- |
|
|
| ## Model Details |
|
|
| | Property | Value | |
| | ----------------- | ----------------------------------- | |
| | Architecture | Recurrent Spiking Neural Network | |
| | Parameters | 1.65M | |
| | Framework | PyTorch + SpikingJelly | |
| | Neuron Type | Parametric LIF (PLIF) | |
| | Inputs | Dual RGB images + 7-DoF joint state | |
| | Output | 7-DoF action delta | |
| | Training Episodes | 407 teleoperated demonstrations | |
| | Best Checkpoint | `full_run_v1_delta` (step 3600) | |
|
|
| --- |
|
|
| ## Architecture |
|
|
| ```text |
| Global Camera ───────────┐ |
| │ |
| ▼ |
| Spiking CNN Encoder |
| │ |
| Gripper Camera ───────────┤ |
| ▼ |
| Spiking CNN Encoder |
| │ |
| Joint State ──────────────┐ |
| ▼ |
| Spiking MLP Encoder |
| │ |
| ▼ |
| Recurrent PLIF Fusion Network |
| │ |
| ▼ |
| Linear Projection |
| │ |
| ▼ |
| Non-Spiking LIF Readout |
| │ |
| ▼ |
| 7-DoF Action Delta |
| ``` |
|
|
| The network consists of independent spiking vision encoders for each camera, a spiking proprioceptive encoder, recurrent PLIF fusion layers, and a non-spiking LIF readout that produces continuous control outputs. |
|
|
| --- |
|
|
| ## Training Data |
|
|
| The model was trained using **407 teleoperated robot manipulation demonstrations** containing: |
|
|
| * Dual synchronized RGB camera streams |
| * 7-dimensional robot joint state |
| * Demonstrated robot actions |
|
|
| Training images were resized to **128×128**, and trajectories were divided into overlapping temporal windows of length **32** with stride **16**. |
|
|
| --- |
|
|
| ## Training Objective |
|
|
| Rather than predicting the next absolute joint configuration, the model predicts the **difference between the demonstrated action and the current joint position**. |
|
|
| This formulation substantially improved optimization by matching the scale of real per-step robot motion, resulting in significantly lower prediction error than direct absolute-action regression. |
|
|
| --- |
|
|
| ## Evaluation |
|
|
| Evaluation was performed offline on a held-out validation split using open-loop trajectory prediction. |
|
|
| ### Best Checkpoint (`full_run_v1_delta`) |
| |
| | Metric | Value | |
| | ------------------------ | --------------- | |
| | MAE | **0.00989 rad** | |
| | RMSE | **0.01873 rad** | |
| | R² | **0.99917** | |
| | Cosine Similarity | **0.99960** | |
| | Mean Episode Correlation | **0.99135** | |
| | Final Pose Error | **0.05417 rad** | |
| |
| Compared to the absolute-action baseline: |
| |
| | Metric | Absolute | Delta | |
| | ---------------- | -------- | ---------- | |
| | MAE | 0.0834 | **0.0099** | |
| | RMSE | 0.1278 | **0.0161** | |
| | Mean Correlation | 0.812 | **0.991** | |
| |
| --- |
| |
| ## Intended Use |
| |
| This model is intended for research in: |
| |
| * Spiking neural networks |
| * Robot imitation learning |
| * Neuromorphic computing |
| * Multimodal visuomotor policies |
| * Temporal sequence modeling for robot control |
| |
| --- |
| |
| ## Limitations |
| |
| The model has been evaluated **only in offline open-loop settings** against held-out demonstrations. |
| |
| Reported metrics measure trajectory prediction accuracy rather than real-world task completion. Physical deployment or closed-loop simulation is required to assess manipulation performance. |
| |
| --- |
| |
| ## Citation |
| |
| If you use this model in your research, please cite this repository. |
| |
| ```bibtex |
| @misc{brain2026, |
| title={Brain: A Recurrent Spiking Neural Network for Visuomotor Control}, |
| author=Adith, |
| year={2026}, |
| howpublished={Hugging Face} |
| } |
| |
