MinkowskiPE: Minkowski Positional Encoding for Spatiotemporal Perception

From microscopic molecular motion to macroscopic video scenes, a key challenge in building physical intelligence is understanding spatial structure in dynamic systems and how it evolves over time — in other words, spatiotemporal perception.

Traditional approaches often describe temporal evolution through recurrent structures, differential equations, or specific physical mechanisms. Transformers offer a more flexible, data-driven alternative, but the relationship between space and time is typically left for the model to learn implicitly from data. This naturally raises a question: can we preserve the flexibility of data-driven learning while introducing a more explicit geometric prior for joint spatiotemporal modeling?

To this end, we propose MinkowskiPE. It borrows the algebraic structure of Minkowski geometry and Lorentz transformations, using joint spacetime coordinates to transform the query and key features. Although each token is encoded independently using its own absolute coordinate, the resulting attention score depends only on the relative spacetime displacement between the two tokens. This gives MinkowskiPE spacetime translation invariance while preserving the standard attention computation, making it directly compatible with efficient implementations such as FlashAttention.

Importantly, using Minkowski geometry does not assume that the underlying physical systems obey relativistic dynamics. Instead, it serves as a geometric inductive bias at the representation level. We evaluate the same positional encoding formulation on two very different scales — protein–ligand molecular dynamics and KTH video prediction — and observe consistent performance improvements on both tasks.