Hugging Face's logo

Aditharavind
/
Brain

Model card Files
xet
 Community
Brain
File size: 4,878 Bytes
3879646
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146


# Brain: Spiking Visuomotor Policy

## Model Description

**Brain** is a recurrent **Spiking Neural Network (SNN)** visuomotor policy trained end-to-end for robot manipulation through imitation learning.

The model processes:

* Dual RGB camera observations
* 7-dimensional proprioceptive joint state

and predicts a **7-dimensional action delta** (6 joints + gripper) at every timestep.

The policy is implemented entirely with spiking neural network components using **SpikingJelly**, employing Parametric Leaky Integrate-and-Fire (PLIF) neurons, recurrent temporal processing, and surrogate-gradient backpropagation.

The released checkpoint is the best-performing model trained using **delta-action prediction**, which significantly outperformed an equivalent absolute-action formulation.

---

## Model Details

| Property          | Value                               |
| ----------------- | ----------------------------------- |
| Architecture      | Recurrent Spiking Neural Network    |
| Parameters        | 1.65M                               |
| Framework         | PyTorch + SpikingJelly              |
| Neuron Type       | Parametric LIF (PLIF)               |
| Inputs            | Dual RGB images + 7-DoF joint state |
| Output            | 7-DoF action delta                  |
| Training Episodes | 407 teleoperated demonstrations     |
| Best Checkpoint   | `full_run_v1_delta` (step 3600)     |

---

## Architecture

```text
Global Camera ───────────┐
                          │
                          ▼
                  Spiking CNN Encoder
                          │
Gripper Camera ───────────┤
                          ▼
                  Spiking CNN Encoder
                          │
Joint State ──────────────┐
                          ▼
                Spiking MLP Encoder
                          │
                          ▼
            Recurrent PLIF Fusion Network
                          │
                          ▼
                    Linear Projection
                          │
                          ▼
             Non-Spiking LIF Readout
                          │
                          ▼
                7-DoF Action Delta
```

The network consists of independent spiking vision encoders for each camera, a spiking proprioceptive encoder, recurrent PLIF fusion layers, and a non-spiking LIF readout that produces continuous control outputs.

---

## Training Data

The model was trained using **407 teleoperated robot manipulation demonstrations** containing:

* Dual synchronized RGB camera streams
* 7-dimensional robot joint state
* Demonstrated robot actions

Training images were resized to **128×128**, and trajectories were divided into overlapping temporal windows of length **32** with stride **16**.

---

## Training Objective

Rather than predicting the next absolute joint configuration, the model predicts the **difference between the demonstrated action and the current joint position**.

This formulation substantially improved optimization by matching the scale of real per-step robot motion, resulting in significantly lower prediction error than direct absolute-action regression.

---

## Evaluation

Evaluation was performed offline on a held-out validation split using open-loop trajectory prediction.

### Best Checkpoint (`full_run_v1_delta`)

| Metric                   | Value           |
| ------------------------ | --------------- |
| MAE                      | **0.00989 rad** |
| RMSE                     | **0.01873 rad** |
| R²                       | **0.99917**     |
| Cosine Similarity        | **0.99960**     |
| Mean Episode Correlation | **0.99135**     |
| Final Pose Error         | **0.05417 rad** |

Compared to the absolute-action baseline:

| Metric           | Absolute | Delta      |
| ---------------- | -------- | ---------- |
| MAE              | 0.0834   | **0.0099** |
| RMSE             | 0.1278   | **0.0161** |
| Mean Correlation | 0.812    | **0.991**  |

---

## Intended Use

This model is intended for research in:

* Spiking neural networks
* Robot imitation learning
* Neuromorphic computing
* Multimodal visuomotor policies
* Temporal sequence modeling for robot control

---

## Limitations

The model has been evaluated **only in offline open-loop settings** against held-out demonstrations.

Reported metrics measure trajectory prediction accuracy rather than real-world task completion. Physical deployment or closed-loop simulation is required to assess manipulation performance.

---

## Citation

If you use this model in your research, please cite this repository.

```bibtex
@misc{brain2026,
  title={Brain: A Recurrent Spiking Neural Network for Visuomotor Control},
  author=Adith,
  year={2026},
  howpublished={Hugging Face}
}