SMT-Learner
Transformer autoencoder for spatiotemporal movement trajectories (SMT), trained with movement performance-aware contrastive learning and post-training adaptive transfer. It produces trial-level embeddings of pediatric upper-limb reaching movements collected from gamified iPad assessments, and those embeddings support the exploration-exploitation (E-E) ratio, a biomarker of motor skill learning and rehabilitation progress.
Released with A Deep Learning Framework for Assessing Pediatric Upper-limb Motor Control and Gamified Mobile Rehabilitation (IEEE TNSRE). Code: https://github.com/RaihanSabique/SMT-Learner
Model
| Input | canonicalized (x, y, t) trajectory resampled to 512 points |
| Encoder | linear stem 3 -> 64, fixed sinusoidal positional encoding, 2 post-norm transformer layers (8 heads, FFN 256, dropout 0.1) |
| Embedding | mean pooling over 512 tokens -> Z_global in R^64 |
| Metadata | binary task-type indicator -> 2-layer MLP -> R^32, concatenated to give a 96-d fused code |
| Heads | 2-layer projection head (R^64, contrastive only) and a single dense decoder (96 -> 512x3) |
| Parameters | 260,704 (decoder 57.1%, transformer layers 38.3%) |
| Checkpoint | 1.19 MB |
The released weights are the EXP4 adaptive-transfer checkpoint: pre-trained on MSL and fine-tuned on MP by combined cross-task and cross-subject transfer.
Training data
- MSL - Motor Skill Learning. S. Nemanich, R. Mia, and C. Kemmel-Bartletti, Motor skill learning in young children born preterm, Jan. 2026. DOI: 10.17605/OSF.IO/W4R2F ยท osf.io/w4r2f
- MP - Motor Performance. C. Kemmel-Bartletti and S. Nemanich, Game-based mobile-health quantification of upper-limb motor performance in children with hemiparetic cerebral palsy, Jun. 2025. DOI: 10.17605/OSF.IO/74DJM ยท osf.io/74djm
Reported performance
Baseline comparison (Suppl. TABLE III of the paper):
| Model | Data | rMSE | Ep-Err | Curve-Err | T-Corr | R-Corr | S-Corr |
|---|---|---|---|---|---|---|---|
| STTraj2Vec | MSL | 0.386 | 0.095 | 3.666 | 0.982 | 0.647 | -0.797 |
| VAE | MSL | 0.412 | 0.089 | 4.758 | 0.136 | 0.096 | -0.135 |
| Seq2Seq | MSL | 0.190 | 0.215 | 3.619 | -0.996 | -0.653 | 0.819 |
| Traj-MAE | MSL | 0.111 | 0.290 | 28.304 | -0.960 | -0.638 | 0.779 |
| ConvTraj | MSL | 0.323 | 0.147 | 30.820 | 0.742 | 0.679 | 0.660 |
| SMT-Learner | MSL | 0.086 | 0.072 | 1.577 | 0.893 | 0.644 | 0.970 |
| SMT-Learner | MP | 0.001 | 0.005 | 1.720 | 0.860 | 0.910 | 0.955 |
SMT-Learner achieves the lowest reconstruction, endpoint and curvature error and the highest success correlation on both datasets. STTraj2Vec reaches a higher completion-time correlation but a negative success correlation, because it encodes temporal and spatial continuity without outcome constraints.
Adaptive transfer (EXP4) reduced the total loss from 1.05 [1.04, 1.06] to 0.12 [0.11, 0.13], an 88.6% improvement, over 5 seeds.
Inference cost
260,704 parameters and a 1.19 MB checkpoint run on a single CPU thread (tested on iPad): 14.4 ms to embed one trial and 0.042 ms for an incremental E-E update, so the biomarker can be recomputed between trials.
Usage
inference.py in this repository is self-contained - it bundles the model definition,
the training-time trajectory normalization and the E-E estimator, so no other code is
needed.
import os, sys
from huggingface_hub import hf_hub_download
sys.path.insert(0, os.path.dirname(hf_hub_download("raihan116/SMT-Learner", "inference.py")))
from inference import SMTLearnerInference
smt = SMTLearnerInference.from_pretrained("raihan116/SMT-Learner")
# paths: list of (x, y) touch sequences; time_diff_ms: matching timestamps
embeddings = smt.embed_trials(paths, time_diff_ms) # (n_trials, 64)
ee_ratio = smt.exploration_exploitation(embeddings) # (n_trials,)
Or from the command line:
python inference.py --csv trials.csv --out ee_results.csv
Trajectories must be normalized by inference.py, which applies the preprocessing the
weights were trained with: spatial canonicalization plus a time channel expressed as
[0, 1] movement progress. Passing raw timestamps puts the time channel well outside its
training range and the embeddings are meaningless.
E-E hyperparameters follow the sensitivity analysis in the paper: window W=120, decay alpha=0.05, novelty weights beta1=0.10 and beta2=0.90.
Intended use and limitations
Intended for research on upper-limb motor assessment and rehabilitation monitoring. This is an assessment and monitoring model; no intervention data supports it, and it is not a diagnostic device. Results come from two planar touchscreen reaching tasks in children aged 5-12 in one geographic region. Behaviour on substantially different trajectory distributions, other age bands, or non-touchscreen capture is untested and would require retraining. Embeddings are derived from de-identified trial data only.
Citation
@article{mia2026smtlearner,
title = {A Deep Learning Framework for Assessing Pediatric Upper-limb Motor
Control and Gamified Mobile Rehabilitation},
author = {Mia, Md Raihan and Ahamed, Sheikh Iqbal and Nemanich, Samuel},
journal = {IEEE Transactions on Neural Systems and Rehabilitation Engineering},
year = {2026}
}
- Downloads last month
- 34