Improve technical paper pagination and figure placement
Browse files- TURTLE_technical_report.pdf +2 -2
- paper/PAPER-SHA256SUMS.txt +4 -4
- paper/TECHNICAL_PAPER.md +12 -22
- paper/TURTLE-ACM-source.zip +2 -2
- paper/TURTLE.pdf +2 -2
- paper/main.tex +48 -35
TURTLE_technical_report.pdf
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:bc4f15a26275acc395f6110249fc67674b0db59b5e95a8b08150dba38d4d6bdd
|
| 3 |
+
size 1033560
|
paper/PAPER-SHA256SUMS.txt
CHANGED
|
@@ -1,8 +1,8 @@
|
|
| 1 |
413cfbb21a074dde8c7f74187c3b74dfbdbc6e4df0727ed465b6ce355c28b553 ILLUSTRATIONS.json
|
| 2 |
e63a82a980aaba19c485bd4a876006011321b9b1b5f7320bf956db13dcf3cd0f README.md
|
| 3 |
-
|
| 4 |
-
|
| 5 |
-
|
| 6 |
6c03cdd6818b1ea39082bccd9815ca347d7a330446dc9b3762512c7ce7e4bc0d figure-specifications.json
|
| 7 |
e9abf7692e076f7291784f68e75b9fac5221600a4536745f0494ebe4a3faece9 figures/illustration-01.pdf
|
| 8 |
bfb5cadc932b48db0465e2ea099ec0d8113f6cc8be1453159329bdbb48502ef6 figures/illustration-01.png
|
|
@@ -15,6 +15,6 @@ e0caef75e8414e5abae1cf00885ce69559eb5095908752b04a5980eea19cb372 figures/illust
|
|
| 15 |
49bc6ba2f3fde53b07d564e7e6ffa85943b363aaf4482f9edf8b568cb4fdc690 figures/illustration-05.pdf
|
| 16 |
7b7bd8ae848704f99ccd333457fe76f2c2873cc698057d747fea46bf98475ed8 figures/illustration-05.png
|
| 17 |
c408e4e2a3807e2b90cdca484ce70ae7fb328ab2613e008461f889f1ee896a2e main.bbl
|
| 18 |
-
|
| 19 |
a0ed7d55bc20e4622b683487006b7744cbad6d325ed5c05898b0d5b57b807bfa references.bib
|
| 20 |
eed8b979a5a299b42e925d9027332e7df876400dcef046f2cfe26beef91d1189 render_figures.py
|
|
|
|
| 1 |
413cfbb21a074dde8c7f74187c3b74dfbdbc6e4df0727ed465b6ce355c28b553 ILLUSTRATIONS.json
|
| 2 |
e63a82a980aaba19c485bd4a876006011321b9b1b5f7320bf956db13dcf3cd0f README.md
|
| 3 |
+
9878c3ec77d06db90a72f368b6c012fe81822412f01c95565f628c8711d55fc1 TECHNICAL_PAPER.md
|
| 4 |
+
941d8e4b2bd7d19cb4a9b99197d9c54e4499ad638c4258ecf0c9c07c42f87f1d TURTLE-ACM-source.zip
|
| 5 |
+
bc4f15a26275acc395f6110249fc67674b0db59b5e95a8b08150dba38d4d6bdd TURTLE.pdf
|
| 6 |
6c03cdd6818b1ea39082bccd9815ca347d7a330446dc9b3762512c7ce7e4bc0d figure-specifications.json
|
| 7 |
e9abf7692e076f7291784f68e75b9fac5221600a4536745f0494ebe4a3faece9 figures/illustration-01.pdf
|
| 8 |
bfb5cadc932b48db0465e2ea099ec0d8113f6cc8be1453159329bdbb48502ef6 figures/illustration-01.png
|
|
|
|
| 15 |
49bc6ba2f3fde53b07d564e7e6ffa85943b363aaf4482f9edf8b568cb4fdc690 figures/illustration-05.pdf
|
| 16 |
7b7bd8ae848704f99ccd333457fe76f2c2873cc698057d747fea46bf98475ed8 figures/illustration-05.png
|
| 17 |
c408e4e2a3807e2b90cdca484ce70ae7fb328ab2613e008461f889f1ee896a2e main.bbl
|
| 18 |
+
60c0c467404444b9935f7dc363ef035becfe6a47b2b2ef9e096c53cd00a1c8b5 main.tex
|
| 19 |
a0ed7d55bc20e4622b683487006b7744cbad6d325ed5c05898b0d5b57b807bfa references.bib
|
| 20 |
eed8b979a5a299b42e925d9027332e7df876400dcef046f2cfe26beef91d1189 render_figures.py
|
paper/TECHNICAL_PAPER.md
CHANGED
|
@@ -10,24 +10,6 @@ Report date: November 13, 2026
|
|
| 10 |
|
| 11 |
Large language models represent tokens and contextual states in high-dimensional vector spaces that are difficult to inspect directly. This report presents TURTLE (Trajectory-based Understanding and Rendering of Transformations in Latent Embeddings), a framework that adapts ideas from turtle geometry to the visualization of vector embeddings and layerwise representation trajectories. TURTLE distinguishes three complementary modes. Glyph mode converts the coordinates of a vector into a three-dimensional turtle signature for pedagogical visualization. Geometric mode constructs a literal coordinate-wise walk in the original n-dimensional space and projects that walk into three dimensions using principal component analysis (PCA). Trajectory mode accepts actual model states, allowing a token's contextual representation to be followed across successive transformer layers. The current open-source implementation includes a reproducible BERT-base demonstration comparing the hidden-state trajectory of the word 'jaguar' in animal and automobile contexts. The two trajectories are projected through a shared PCA basis, placing them in the same three-dimensional coordinate system and allowing their projected relative positions to be compared. TURTLE therefore provides both an explanatory metaphor and an experimental visualization method for studying change in high-dimensional neural representations. Its principal limitations arise from dimensionality reduction, basis dependence in coordinate walks, and the risk of over-interpreting visual similarity.
|
| 12 |
|
| 13 |
-
## Application interface
|
| 14 |
-
|
| 15 |
-

|
| 16 |
-
|
| 17 |
-
*Glyph mode turns the displayed 768-dimensional vector into a three-dimensional turtle path using a 45-degree consecutive angle.*
|
| 18 |
-
|
| 19 |
-

|
| 20 |
-
|
| 21 |
-
*Geometric mode displays the n-dimensional walk projected into three dimensions by PCA.*
|
| 22 |
-
|
| 23 |
-

|
| 24 |
-
|
| 25 |
-
*Trajectory mode displays bundled BERT hidden states for jaguar in animal and automobile contexts using a shared PCA projection.*
|
| 26 |
-
|
| 27 |
-

|
| 28 |
-
|
| 29 |
-
*Glyph mode after changing the consecutive angle to 90 degrees, illustrating how the same encoding controls alter the displayed path.*
|
| 30 |
-
|
| 31 |
## Introduction
|
| 32 |
|
| 33 |
Modern language models encode words, tokens, and contextual states as vectors with hundreds or thousands of numerical dimensions. These representations are computationally tractable but visually inaccessible. Researchers commonly use dimensionality-reduction techniques such as principal component analysis to create two- or three-dimensional views of high-dimensional data, but the resulting scatterplots tend to emphasize static positions. They are less naturally suited to communicating how a representation changes as a model processes context or as a token passes through successive network layers.
|
|
@@ -40,6 +22,9 @@ TURTLE - Trajectory-based Understanding and Rendering of Transformations in Late
|
|
| 40 |
|
| 41 |
Let an embedding or hidden representation be a vector v in $\mathbb{R}^n$. A turtle position at step t is represented by $p_t$ in $\mathbb{R}^n$, and a displacement by $\Delta p_t$. The generic movement rule is
|
| 42 |
|
|
|
|
|
|
|
|
|
|
| 43 |
$$p_{t+1}=p_t+\Delta p_t.$$
|
| 44 |
|
| 45 |
This rule is independent of dimensionality. The same operation applies in two dimensions, three dimensions, or a latent space with hundreds or thousands of coordinates. The central challenge is therefore not whether turtle motion can be defined in n dimensions, but how the motion should be constructed and how it should be rendered for human inspection.
|
|
@@ -58,6 +43,9 @@ The current implementation uses PCA, computed through singular-value decompositi
|
|
| 58 |
|
| 59 |
The current implementation deliberately distinguishes three modes so that artistic encoding, geometric projection, and model-state analysis are not conflated.
|
| 60 |
|
|
|
|
|
|
|
|
|
|
| 61 |
### Glyph mode: turtle encoding
|
| 62 |
|
| 63 |
Glyph mode retains the original pedagogical prototype. For each scalar coordinate $v_i$, the magnitude $|v_i|$ determines segment length, the sign determines a right or left turn, and a fixed turning angle separates consecutive segments. An index-driven roll changes the local turning plane so that the generated trace develops three-dimensional structure. The mapping can be written schematically as
|
|
@@ -88,22 +76,26 @@ $$\Delta h^{(l)}=h^{(l+1)}-h^{(l)}.$$
|
|
| 88 |
|
| 89 |
Here the order is intrinsic to the computation: layer l precedes layer $l+1$. Prior work has shown that contextual representations change substantially across contexts and layers, and has found evidence that different kinds of linguistic information are distributed differently across BERT's depth [3](#ref-ethayarajh); [8](#ref-tenney). TURTLE provides a direct geometric visualization of such layerwise change rather than reducing each state to an isolated point without an explicit path.
|
| 90 |
|
| 91 |
-
|
| 92 |
## Proof-of-Concept: Contextual Trajectories in BERT
|
| 93 |
|
| 94 |
The current repository contains a reproducible demonstration built from `google-bert/bert-base-uncased`, the 12-layer, 768-hidden-dimension BERT-base model described by [2](#ref-devlin). TURTLE extracts the representation of the word 'jaguar' in two sentences: 'The jaguar rested quietly in the jungle.' and 'The Jaguar accelerated quickly out of the garage.' The first context selects the animal sense, while the second selects the automobile sense.
|
| 95 |
|
|
|
|
|
|
|
|
|
|
| 96 |
The extraction script requests hidden states from the model and gathers the target word's representation from the embedding output and each of the 12 transformer blocks, producing a $13\times768$ matrix for each sentence. If a target word occupies multiple subword tokens, the implementation mean-pools the corresponding subword-token representations at each layer. The resulting matrices and provenance metadata are stored with the project so that the demonstration can be reproduced.
|
| 97 |
|
| 98 |
For comparative visualization, TURTLE does not fit PCA separately to the two contextual trajectories. Instead, the $13\times768$ matrices are concatenated, a single PCA basis is fit to the combined set, and the projected points are then separated back into two paths. This shared basis is essential: two independent PCA solutions would have unrelated axes, making visual comparisons of orientation or relative position potentially misleading. In the shared projection, both paths inhabit the same three-dimensional coordinate system.
|
| 99 |
|
| 100 |
The demonstration should not be interpreted as evidence that three dimensions recover the full geometry of BERT's 768-dimensional hidden states. Its purpose is narrower: to show that TURTLE can accept genuine model representations, preserve their computational ordering, and render their transformations as an inspectable path. The most important object remains the original high-dimensional trajectory; the three-dimensional view is an interface to it.
|
| 101 |
|
| 102 |
-
|
| 103 |
## Implementation
|
| 104 |
|
| 105 |
The open-source implementation is written primarily in Python with an embedded browser-based visualization layer. The repository currently provides `TURTLE.py`, a BERT extraction utility, reproducibility metadata and hidden-state data, dependency files, and a standalone GitHub Pages build. The software supports local Python execution as well as notebook environments.
|
| 106 |
|
|
|
|
|
|
|
|
|
|
| 107 |
Three functions capture the core scientific operations. `embedding_to_nd_walk` constructs the coordinate-wise n-dimensional path whose endpoint is the original vector. `representation_trajectory_to_3d` projects an arbitrary matrix of successive model states into three PCA dimensions. `project_trajectories_shared` concatenates multiple trajectories before projection so that comparative paths use the same PCA basis. These functions separate the underlying geometric operations from the interactive display layer.
|
| 108 |
|
| 109 |
The implementation also enforces an interpretive boundary in the interface. Decorative transformations are disabled for data-derived geometric projections and model trajectories, while they remain available in glyph mode. This design choice reduces the risk that a user will mistake a manually altered curve for a projection of the source representation.
|
|
@@ -131,7 +123,6 @@ split the projected matrix back into H_1,...,H_m paths
|
|
| 131 |
return comparable 3D trajectories
|
| 132 |
```
|
| 133 |
|
| 134 |
-
|
| 135 |
## Potential Research Applications
|
| 136 |
|
| 137 |
TURTLE can be extended beyond the BERT homonym demonstration in several directions. First, layerwise trajectories can be compared for ambiguous words across contexts to study when representational paths diverge. Second, multilingual models could be used to compare translations or semantically aligned expressions across languages, provided that comparisons are performed within a shared representation space and with a shared projection basis. Third, prompt or sentence perturbations could be treated as controlled changes and their resulting trajectories compared quantitatively in the original space and visually in projection.
|
|
@@ -140,7 +131,6 @@ The framework is also compatible with richer measurements than simple visual ins
|
|
| 140 |
|
| 141 |
A further application is pedagogical. Conventional diagrams often present embeddings as static dots in a semantic map. TURTLE instead foregrounds movement: representations occupy positions, transformations create displacements, and sequences of transformations create trajectories. This can provide an intuitive bridge from elementary vector arithmetic to the layerwise dynamics of transformer models.
|
| 142 |
|
| 143 |
-
|
| 144 |
## Limitations and Scientific Cautions
|
| 145 |
|
| 146 |
Several limitations should be stated explicitly. First, glyph mode is a deterministic encoding of vector coordinates into a three-dimensional curve, not a faithful projection of the original geometry. Similar-looking glyphs should not be assumed to imply similar embeddings without empirical validation.
|
|
|
|
| 10 |
|
| 11 |
Large language models represent tokens and contextual states in high-dimensional vector spaces that are difficult to inspect directly. This report presents TURTLE (Trajectory-based Understanding and Rendering of Transformations in Latent Embeddings), a framework that adapts ideas from turtle geometry to the visualization of vector embeddings and layerwise representation trajectories. TURTLE distinguishes three complementary modes. Glyph mode converts the coordinates of a vector into a three-dimensional turtle signature for pedagogical visualization. Geometric mode constructs a literal coordinate-wise walk in the original n-dimensional space and projects that walk into three dimensions using principal component analysis (PCA). Trajectory mode accepts actual model states, allowing a token's contextual representation to be followed across successive transformer layers. The current open-source implementation includes a reproducible BERT-base demonstration comparing the hidden-state trajectory of the word 'jaguar' in animal and automobile contexts. The two trajectories are projected through a shared PCA basis, placing them in the same three-dimensional coordinate system and allowing their projected relative positions to be compared. TURTLE therefore provides both an explanatory metaphor and an experimental visualization method for studying change in high-dimensional neural representations. Its principal limitations arise from dimensionality reduction, basis dependence in coordinate walks, and the risk of over-interpreting visual similarity.
|
| 12 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 13 |
## Introduction
|
| 14 |
|
| 15 |
Modern language models encode words, tokens, and contextual states as vectors with hundreds or thousands of numerical dimensions. These representations are computationally tractable but visually inaccessible. Researchers commonly use dimensionality-reduction techniques such as principal component analysis to create two- or three-dimensional views of high-dimensional data, but the resulting scatterplots tend to emphasize static positions. They are less naturally suited to communicating how a representation changes as a model processes context or as a token passes through successive network layers.
|
|
|
|
| 22 |
|
| 23 |
Let an embedding or hidden representation be a vector v in $\mathbb{R}^n$. A turtle position at step t is represented by $p_t$ in $\mathbb{R}^n$, and a displacement by $\Delta p_t$. The generic movement rule is
|
| 24 |
|
| 25 |
+

|
| 26 |
+
|
| 27 |
+
*Geometric mode displays the n-dimensional walk projected into three dimensions by PCA.*
|
| 28 |
$$p_{t+1}=p_t+\Delta p_t.$$
|
| 29 |
|
| 30 |
This rule is independent of dimensionality. The same operation applies in two dimensions, three dimensions, or a latent space with hundreds or thousands of coordinates. The central challenge is therefore not whether turtle motion can be defined in n dimensions, but how the motion should be constructed and how it should be rendered for human inspection.
|
|
|
|
| 43 |
|
| 44 |
The current implementation deliberately distinguishes three modes so that artistic encoding, geometric projection, and model-state analysis are not conflated.
|
| 45 |
|
| 46 |
+

|
| 47 |
+
|
| 48 |
+
*Glyph mode turns the displayed 768-dimensional vector into a three-dimensional turtle path using a 45-degree consecutive angle.*
|
| 49 |
### Glyph mode: turtle encoding
|
| 50 |
|
| 51 |
Glyph mode retains the original pedagogical prototype. For each scalar coordinate $v_i$, the magnitude $|v_i|$ determines segment length, the sign determines a right or left turn, and a fixed turning angle separates consecutive segments. An index-driven roll changes the local turning plane so that the generated trace develops three-dimensional structure. The mapping can be written schematically as
|
|
|
|
| 76 |
|
| 77 |
Here the order is intrinsic to the computation: layer l precedes layer $l+1$. Prior work has shown that contextual representations change substantially across contexts and layers, and has found evidence that different kinds of linguistic information are distributed differently across BERT's depth [3](#ref-ethayarajh); [8](#ref-tenney). TURTLE provides a direct geometric visualization of such layerwise change rather than reducing each state to an isolated point without an explicit path.
|
| 78 |
|
|
|
|
| 79 |
## Proof-of-Concept: Contextual Trajectories in BERT
|
| 80 |
|
| 81 |
The current repository contains a reproducible demonstration built from `google-bert/bert-base-uncased`, the 12-layer, 768-hidden-dimension BERT-base model described by [2](#ref-devlin). TURTLE extracts the representation of the word 'jaguar' in two sentences: 'The jaguar rested quietly in the jungle.' and 'The Jaguar accelerated quickly out of the garage.' The first context selects the animal sense, while the second selects the automobile sense.
|
| 82 |
|
| 83 |
+

|
| 84 |
+
|
| 85 |
+
*Trajectory mode displays bundled BERT hidden states for jaguar in animal and automobile contexts using a shared PCA projection.*
|
| 86 |
The extraction script requests hidden states from the model and gathers the target word's representation from the embedding output and each of the 12 transformer blocks, producing a $13\times768$ matrix for each sentence. If a target word occupies multiple subword tokens, the implementation mean-pools the corresponding subword-token representations at each layer. The resulting matrices and provenance metadata are stored with the project so that the demonstration can be reproduced.
|
| 87 |
|
| 88 |
For comparative visualization, TURTLE does not fit PCA separately to the two contextual trajectories. Instead, the $13\times768$ matrices are concatenated, a single PCA basis is fit to the combined set, and the projected points are then separated back into two paths. This shared basis is essential: two independent PCA solutions would have unrelated axes, making visual comparisons of orientation or relative position potentially misleading. In the shared projection, both paths inhabit the same three-dimensional coordinate system.
|
| 89 |
|
| 90 |
The demonstration should not be interpreted as evidence that three dimensions recover the full geometry of BERT's 768-dimensional hidden states. Its purpose is narrower: to show that TURTLE can accept genuine model representations, preserve their computational ordering, and render their transformations as an inspectable path. The most important object remains the original high-dimensional trajectory; the three-dimensional view is an interface to it.
|
| 91 |
|
|
|
|
| 92 |
## Implementation
|
| 93 |
|
| 94 |
The open-source implementation is written primarily in Python with an embedded browser-based visualization layer. The repository currently provides `TURTLE.py`, a BERT extraction utility, reproducibility metadata and hidden-state data, dependency files, and a standalone GitHub Pages build. The software supports local Python execution as well as notebook environments.
|
| 95 |
|
| 96 |
+

|
| 97 |
+
|
| 98 |
+
*Glyph mode after changing the consecutive angle to 90 degrees, illustrating how the same encoding controls alter the displayed path.*
|
| 99 |
Three functions capture the core scientific operations. `embedding_to_nd_walk` constructs the coordinate-wise n-dimensional path whose endpoint is the original vector. `representation_trajectory_to_3d` projects an arbitrary matrix of successive model states into three PCA dimensions. `project_trajectories_shared` concatenates multiple trajectories before projection so that comparative paths use the same PCA basis. These functions separate the underlying geometric operations from the interactive display layer.
|
| 100 |
|
| 101 |
The implementation also enforces an interpretive boundary in the interface. Decorative transformations are disabled for data-derived geometric projections and model trajectories, while they remain available in glyph mode. This design choice reduces the risk that a user will mistake a manually altered curve for a projection of the source representation.
|
|
|
|
| 123 |
return comparable 3D trajectories
|
| 124 |
```
|
| 125 |
|
|
|
|
| 126 |
## Potential Research Applications
|
| 127 |
|
| 128 |
TURTLE can be extended beyond the BERT homonym demonstration in several directions. First, layerwise trajectories can be compared for ambiguous words across contexts to study when representational paths diverge. Second, multilingual models could be used to compare translations or semantically aligned expressions across languages, provided that comparisons are performed within a shared representation space and with a shared projection basis. Third, prompt or sentence perturbations could be treated as controlled changes and their resulting trajectories compared quantitatively in the original space and visually in projection.
|
|
|
|
| 131 |
|
| 132 |
A further application is pedagogical. Conventional diagrams often present embeddings as static dots in a semantic map. TURTLE instead foregrounds movement: representations occupy positions, transformations create displacements, and sequences of transformations create trajectories. This can provide an intuitive bridge from elementary vector arithmetic to the layerwise dynamics of transformer models.
|
| 133 |
|
|
|
|
| 134 |
## Limitations and Scientific Cautions
|
| 135 |
|
| 136 |
Several limitations should be stated explicitly. First, glyph mode is a deterministic encoding of vector coordinates into a three-dimensional curve, not a faithful projection of the original geometry. Similar-looking glyphs should not be assumed to imply similar embeddings without empirical validation.
|
paper/TURTLE-ACM-source.zip
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:941d8e4b2bd7d19cb4a9b99197d9c54e4499ad638c4258ecf0c9c07c42f87f1d
|
| 3 |
+
size 3371125
|
paper/TURTLE.pdf
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:bc4f15a26275acc395f6110249fc67674b0db59b5e95a8b08150dba38d4d6bdd
|
| 3 |
+
size 1033560
|
paper/main.tex
CHANGED
|
@@ -7,6 +7,20 @@
|
|
| 7 |
\DefineVerbatimEnvironment{verbatim}{Verbatim}{breaklines=true,fontsize=\small}
|
| 8 |
\providecommand{\tightlist}{\setlength{\itemsep}{0pt}\setlength{\parskip}{0pt}}
|
| 9 |
\usepackage{graphicx,float}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 10 |
\begin{document}
|
| 11 |
\title{TURTLE: Trajectory-based Understanding and Rendering of Transformations in Latent Embeddings}
|
| 12 |
\author{William Taggart}
|
|
@@ -28,39 +42,17 @@ Turtle geometry offers a different conceptual starting point. In the classical f
|
|
| 28 |
|
| 29 |
TURTLE - Trajectory-based Understanding and Rendering of Transformations in Latent Embeddings - adapts this movement-centered perspective to neural representation spaces. The framework is designed around a simple shift in emphasis: an embedding can be treated not only as a point, but as a destination or state within a trajectory. A difference between representations can then be represented as a displacement, and a sequence of representations as a walk. Throughout this report, \emph{latent representations} refers collectively to embedding vectors and contextual hidden states; where the distinction matters, the more specific term is used. This report updates the original proposal to describe the working open-source implementation, including a proof-of-concept experiment using real BERT hidden states.
|
| 30 |
|
| 31 |
-
|
| 32 |
-
|
| 33 |
-
|
| 34 |
-
|
| 35 |
-
\
|
| 36 |
-
\caption{Glyph mode turns the displayed 768-dimensional vector into a three-dimensional turtle path using a 45-degree consecutive angle.}
|
| 37 |
-
\Description{Glyph mode turns the displayed 768-dimensional vector into a three-dimensional turtle path using a 45-degree consecutive angle.}
|
| 38 |
-
\label{fig:app-screen-1}
|
| 39 |
-
\end{figure}
|
| 40 |
-
\begin{figure}[H]
|
| 41 |
\centering
|
| 42 |
-
\includegraphics[width=\linewidth,height=0.
|
| 43 |
\caption{Geometric mode displays the n-dimensional walk projected into three dimensions by PCA.}
|
| 44 |
\Description{Geometric mode displays the n-dimensional walk projected into three dimensions by PCA.}
|
| 45 |
\label{fig:app-screen-2}
|
| 46 |
\end{figure}
|
| 47 |
-
\begin{figure}[H]
|
| 48 |
-
\centering
|
| 49 |
-
\includegraphics[width=\linewidth,height=0.70\textheight,keepaspectratio]{figures/screenshot-03.png}
|
| 50 |
-
\caption{Trajectory mode displays bundled BERT hidden states for jaguar in animal and automobile contexts using a shared PCA projection.}
|
| 51 |
-
\Description{Trajectory mode displays bundled BERT hidden states for jaguar in animal and automobile contexts using a shared PCA projection.}
|
| 52 |
-
\label{fig:app-screen-3}
|
| 53 |
-
\end{figure}
|
| 54 |
-
\begin{figure}[H]
|
| 55 |
-
\centering
|
| 56 |
-
\includegraphics[width=\linewidth,height=0.70\textheight,keepaspectratio]{figures/screenshot-04.png}
|
| 57 |
-
\caption{Glyph mode after changing the consecutive angle to 90 degrees, illustrating how the same encoding controls alter the displayed path.}
|
| 58 |
-
\Description{Glyph mode after changing the consecutive angle to 90 degrees, illustrating how the same encoding controls alter the displayed path.}
|
| 59 |
-
\label{fig:app-screen-4}
|
| 60 |
-
\end{figure}
|
| 61 |
-
|
| 62 |
-
\section{Mathematical Foundation}\label{mathematical-foundation}
|
| 63 |
-
|
| 64 |
Let an embedding or hidden representation be a vector v in \(\mathbb{R}^n\). A turtle position at step t is represented by \(p_t\) in \(\mathbb{R}^n\), and a displacement by \(\Delta p_t\). The generic movement rule is
|
| 65 |
|
| 66 |
\[p_{t+1}=p_t+\Delta p_t.\]
|
|
@@ -73,9 +65,9 @@ TURTLE separates these questions. The computational trajectory may remain in the
|
|
| 73 |
|
| 74 |
The current implementation uses PCA, computed through singular-value decomposition of centered trajectory points. PCA constructs orthogonal components that successively maximize variance, providing a standard linear method for reducing dimensionality while retaining as much variance as possible in the chosen number of components \citep{jolliffe}. The visible path should therefore be understood as a three-dimensional shadow of a higher-dimensional trajectory, not as the trajectory itself.
|
| 75 |
|
| 76 |
-
\begin{figure}[
|
| 77 |
\centering
|
| 78 |
-
\includegraphics[width=\textwidth]{figures/illustration-01.pdf}
|
| 79 |
\caption{Mode taxonomy. Glyphs encode a vector, geometric mode projects a coordinate-wise walk, and trajectory mode projects ordered model states. The modes should not be interpreted interchangeably.}
|
| 80 |
\Description{Three modes, three different interpretations. Mode taxonomy. Glyphs encode a vector, geometric mode projects a coordinate-wise walk, and trajectory mode projects ordered model states. The modes should not be interpreted interchangeably.}
|
| 81 |
\label{fig:illustration-1}
|
|
@@ -83,6 +75,14 @@ The current implementation uses PCA, computed through singular-value decompositi
|
|
| 83 |
|
| 84 |
\section{TURTLE Modes}\label{turtle-modes}
|
| 85 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 86 |
The current implementation deliberately distinguishes three modes so that artistic encoding, geometric projection, and model-state analysis are not conflated.
|
| 87 |
|
| 88 |
\subsection{Glyph mode: turtle encoding}\label{glyph-mode-turtle-encoding}
|
|
@@ -116,9 +116,16 @@ The displacement between successive layers is
|
|
| 116 |
Here the order is intrinsic to the computation: layer l precedes layer \(l+1\). Prior work has shown that contextual representations change substantially across contexts and layers, and has found evidence that different kinds of linguistic information are distributed differently across BERT's depth \citep{ethayarajh, tenney}. TURTLE provides a direct geometric visualization of such layerwise change rather than reducing each state to an isolated point without an explicit path.
|
| 117 |
|
| 118 |
|
| 119 |
-
|
| 120 |
\section{Proof-of-Concept: Contextual Trajectories in BERT}\label{proof-of-concept-contextual-trajectories-in-bert}
|
| 121 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 122 |
The current repository contains a reproducible demonstration built from \nolinkurl{google-bert/bert-base-uncased}, the 12-layer, 768-hidden-dimension BERT-base model described by \citep{devlin}. TURTLE extracts the representation of the word `jaguar' in two sentences: `The jaguar rested quietly in the jungle.' and `The Jaguar accelerated quickly out of the garage.' The first context selects the animal sense, while the second selects the automobile sense.
|
| 123 |
|
| 124 |
The extraction script requests hidden states from the model and gathers the target word's representation from the embedding output and each of the 12 transformer blocks, producing a \(13\times768\) matrix for each sentence. If a target word occupies multiple subword tokens, the implementation mean-pools the corresponding subword-token representations at each layer. The resulting matrices and provenance metadata are stored with the project so that the demonstration can be reproduced.
|
|
@@ -128,9 +135,16 @@ For comparative visualization, TURTLE does not fit PCA separately to the two con
|
|
| 128 |
The demonstration should not be interpreted as evidence that three dimensions recover the full geometry of BERT's 768-dimensional hidden states. Its purpose is narrower: to show that TURTLE can accept genuine model representations, preserve their computational ordering, and render their transformations as an inspectable path. The most important object remains the original high-dimensional trajectory; the three-dimensional view is an interface to it.
|
| 129 |
|
| 130 |
|
| 131 |
-
|
| 132 |
\section{Implementation}\label{implementation}
|
| 133 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 134 |
The open-source implementation is written primarily in Python with an embedded browser-based visualization layer. The repository currently provides \nolinkurl{TURTLE.py}, a BERT extraction utility, reproducibility metadata and hidden-state data, dependency files, and a standalone GitHub Pages build. The software supports local Python execution as well as notebook environments.
|
| 135 |
|
| 136 |
Three functions capture the core scientific operations. \nolinkurl{embedding_to_nd_walk} constructs the coordinate-wise n-dimensional path whose endpoint is the original vector. \nolinkurl{representation_trajectory_to_3d} projects an arbitrary matrix of successive model states into three PCA dimensions. \nolinkurl{project_trajectories_shared} concatenates multiple trajectories before projection so that comparative paths use the same PCA basis. These functions separate the underlying geometric operations from the interactive display layer.
|
|
@@ -161,7 +175,6 @@ return comparable 3D trajectories
|
|
| 161 |
\end{verbatim}
|
| 162 |
|
| 163 |
|
| 164 |
-
|
| 165 |
\section{Potential Research Applications}\label{potential-research-applications}
|
| 166 |
|
| 167 |
TURTLE can be extended beyond the BERT homonym demonstration in several directions. First, layerwise trajectories can be compared for ambiguous words across contexts to study when representational paths diverge. Second, multilingual models could be used to compare translations or semantically aligned expressions across languages, provided that comparisons are performed within a shared representation space and with a shared projection basis. Third, prompt or sentence perturbations could be treated as controlled changes and their resulting trajectories compared quantitatively in the original space and visually in projection.
|
|
@@ -171,7 +184,6 @@ The framework is also compatible with richer measurements than simple visual ins
|
|
| 171 |
A further application is pedagogical. Conventional diagrams often present embeddings as static dots in a semantic map. TURTLE instead foregrounds movement: representations occupy positions, transformations create displacements, and sequences of transformations create trajectories. This can provide an intuitive bridge from elementary vector arithmetic to the layerwise dynamics of transformer models.
|
| 172 |
|
| 173 |
|
| 174 |
-
|
| 175 |
\section{Limitations and Scientific Cautions}\label{limitations-and-scientific-cautions}
|
| 176 |
|
| 177 |
Several limitations should be stated explicitly. First, glyph mode is a deterministic encoding of vector coordinates into a three-dimensional curve, not a faithful projection of the original geometry. Similar-looking glyphs should not be assumed to imply similar embeddings without empirical validation.
|
|
@@ -200,10 +212,11 @@ The current TURTLE implementation, demonstration data, extraction script, depend
|
|
| 200 |
\section*{AI-use disclosure}
|
| 201 |
The original report records implementation assistance from OpenAI Codex and GPT-5.6. This report was prepared with assistance from Codex. William Taggart is responsible for the final text and its claims. AI assistance does not constitute independent validation of the implementation or scientific interpretation.
|
| 202 |
|
|
|
|
| 203 |
\bibliographystyle{ACM-Reference-Format}
|
| 204 |
\bibliography{references}
|
| 205 |
|
| 206 |
\section*{Suggested citation}
|
| 207 |
-
William Taggart. 2026. TURTLE: Trajectory-based Understanding and Rendering of Transformations in Latent Embeddings. Technical report. Modern and Classical Languages and Literatures, Virginia Tech. November 13, 2026. https://github.com/FL2744/TURTLE
|
| 208 |
|
| 209 |
\end{document}
|
|
|
|
| 7 |
\DefineVerbatimEnvironment{verbatim}{Verbatim}{breaklines=true,fontsize=\small}
|
| 8 |
\providecommand{\tightlist}{\setlength{\itemsep}{0pt}\setlength{\parskip}{0pt}}
|
| 9 |
\usepackage{graphicx,float}
|
| 10 |
+
|
| 11 |
+
% Permit figures to share pages with surrounding explanatory text.
|
| 12 |
+
\renewcommand{\topfraction}{0.92}
|
| 13 |
+
\renewcommand{\bottomfraction}{0.85}
|
| 14 |
+
\renewcommand{\textfraction}{0.06}
|
| 15 |
+
\renewcommand{\floatpagefraction}{0.80}
|
| 16 |
+
\setcounter{topnumber}{3}
|
| 17 |
+
\setcounter{bottomnumber}{3}
|
| 18 |
+
\setcounter{totalnumber}{5}
|
| 19 |
+
\setlength{\textfloatsep}{12pt plus 2pt minus 3pt}
|
| 20 |
+
\setlength{\intextsep}{12pt plus 2pt minus 3pt}
|
| 21 |
+
\raggedbottom
|
| 22 |
+
|
| 23 |
+
\geometry{top=0.75in,bottom=0.65in,includeheadfoot}
|
| 24 |
\begin{document}
|
| 25 |
\title{TURTLE: Trajectory-based Understanding and Rendering of Transformations in Latent Embeddings}
|
| 26 |
\author{William Taggart}
|
|
|
|
| 42 |
|
| 43 |
TURTLE - Trajectory-based Understanding and Rendering of Transformations in Latent Embeddings - adapts this movement-centered perspective to neural representation spaces. The framework is designed around a simple shift in emphasis: an embedding can be treated not only as a point, but as a destination or state within a trajectory. A difference between representations can then be represented as a displacement, and a sequence of representations as a walk. Throughout this report, \emph{latent representations} refers collectively to embedding vectors and contextual hidden states; where the distinction matters, the more specific term is used. This report updates the original proposal to describe the working open-source implementation, including a proof-of-concept experiment using real BERT hidden states.
|
| 44 |
|
| 45 |
+
|
| 46 |
+
\section{Mathematical Foundation}\label{mathematical-foundation}
|
| 47 |
+
|
| 48 |
+
|
| 49 |
+
\begin{figure}[!htb]
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 50 |
\centering
|
| 51 |
+
\includegraphics[width=0.94\linewidth,height=0.58\textheight,keepaspectratio]{figures/screenshot-02.png}
|
| 52 |
\caption{Geometric mode displays the n-dimensional walk projected into three dimensions by PCA.}
|
| 53 |
\Description{Geometric mode displays the n-dimensional walk projected into three dimensions by PCA.}
|
| 54 |
\label{fig:app-screen-2}
|
| 55 |
\end{figure}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 56 |
Let an embedding or hidden representation be a vector v in \(\mathbb{R}^n\). A turtle position at step t is represented by \(p_t\) in \(\mathbb{R}^n\), and a displacement by \(\Delta p_t\). The generic movement rule is
|
| 57 |
|
| 58 |
\[p_{t+1}=p_t+\Delta p_t.\]
|
|
|
|
| 65 |
|
| 66 |
The current implementation uses PCA, computed through singular-value decomposition of centered trajectory points. PCA constructs orthogonal components that successively maximize variance, providing a standard linear method for reducing dimensionality while retaining as much variance as possible in the chosen number of components \citep{jolliffe}. The visible path should therefore be understood as a three-dimensional shadow of a higher-dimensional trajectory, not as the trajectory itself.
|
| 67 |
|
| 68 |
+
\begin{figure}[!htb]
|
| 69 |
\centering
|
| 70 |
+
\includegraphics[width=0.94\textwidth]{figures/illustration-01.pdf}
|
| 71 |
\caption{Mode taxonomy. Glyphs encode a vector, geometric mode projects a coordinate-wise walk, and trajectory mode projects ordered model states. The modes should not be interpreted interchangeably.}
|
| 72 |
\Description{Three modes, three different interpretations. Mode taxonomy. Glyphs encode a vector, geometric mode projects a coordinate-wise walk, and trajectory mode projects ordered model states. The modes should not be interpreted interchangeably.}
|
| 73 |
\label{fig:illustration-1}
|
|
|
|
| 75 |
|
| 76 |
\section{TURTLE Modes}\label{turtle-modes}
|
| 77 |
|
| 78 |
+
|
| 79 |
+
\begin{figure}[!htb]
|
| 80 |
+
\centering
|
| 81 |
+
\includegraphics[width=0.94\linewidth,height=0.58\textheight,keepaspectratio]{figures/screenshot-01.png}
|
| 82 |
+
\caption{Glyph mode turns the displayed 768-dimensional vector into a three-dimensional turtle path using a 45-degree consecutive angle.}
|
| 83 |
+
\Description{Glyph mode turns the displayed 768-dimensional vector into a three-dimensional turtle path using a 45-degree consecutive angle.}
|
| 84 |
+
\label{fig:app-screen-1}
|
| 85 |
+
\end{figure}
|
| 86 |
The current implementation deliberately distinguishes three modes so that artistic encoding, geometric projection, and model-state analysis are not conflated.
|
| 87 |
|
| 88 |
\subsection{Glyph mode: turtle encoding}\label{glyph-mode-turtle-encoding}
|
|
|
|
| 116 |
Here the order is intrinsic to the computation: layer l precedes layer \(l+1\). Prior work has shown that contextual representations change substantially across contexts and layers, and has found evidence that different kinds of linguistic information are distributed differently across BERT's depth \citep{ethayarajh, tenney}. TURTLE provides a direct geometric visualization of such layerwise change rather than reducing each state to an isolated point without an explicit path.
|
| 117 |
|
| 118 |
|
|
|
|
| 119 |
\section{Proof-of-Concept: Contextual Trajectories in BERT}\label{proof-of-concept-contextual-trajectories-in-bert}
|
| 120 |
|
| 121 |
+
|
| 122 |
+
\begin{figure}[!htb]
|
| 123 |
+
\centering
|
| 124 |
+
\includegraphics[width=0.94\linewidth,height=0.58\textheight,keepaspectratio]{figures/screenshot-03.png}
|
| 125 |
+
\caption{Trajectory mode displays bundled BERT hidden states for jaguar in animal and automobile contexts using a shared PCA projection.}
|
| 126 |
+
\Description{Trajectory mode displays bundled BERT hidden states for jaguar in animal and automobile contexts using a shared PCA projection.}
|
| 127 |
+
\label{fig:app-screen-3}
|
| 128 |
+
\end{figure}
|
| 129 |
The current repository contains a reproducible demonstration built from \nolinkurl{google-bert/bert-base-uncased}, the 12-layer, 768-hidden-dimension BERT-base model described by \citep{devlin}. TURTLE extracts the representation of the word `jaguar' in two sentences: `The jaguar rested quietly in the jungle.' and `The Jaguar accelerated quickly out of the garage.' The first context selects the animal sense, while the second selects the automobile sense.
|
| 130 |
|
| 131 |
The extraction script requests hidden states from the model and gathers the target word's representation from the embedding output and each of the 12 transformer blocks, producing a \(13\times768\) matrix for each sentence. If a target word occupies multiple subword tokens, the implementation mean-pools the corresponding subword-token representations at each layer. The resulting matrices and provenance metadata are stored with the project so that the demonstration can be reproduced.
|
|
|
|
| 135 |
The demonstration should not be interpreted as evidence that three dimensions recover the full geometry of BERT's 768-dimensional hidden states. Its purpose is narrower: to show that TURTLE can accept genuine model representations, preserve their computational ordering, and render their transformations as an inspectable path. The most important object remains the original high-dimensional trajectory; the three-dimensional view is an interface to it.
|
| 136 |
|
| 137 |
|
|
|
|
| 138 |
\section{Implementation}\label{implementation}
|
| 139 |
|
| 140 |
+
|
| 141 |
+
\begin{figure}[!htb]
|
| 142 |
+
\centering
|
| 143 |
+
\includegraphics[width=0.94\linewidth,height=0.58\textheight,keepaspectratio]{figures/screenshot-04.png}
|
| 144 |
+
\caption{Glyph mode after changing the consecutive angle to 90 degrees, illustrating how the same encoding controls alter the displayed path.}
|
| 145 |
+
\Description{Glyph mode after changing the consecutive angle to 90 degrees, illustrating how the same encoding controls alter the displayed path.}
|
| 146 |
+
\label{fig:app-screen-4}
|
| 147 |
+
\end{figure}
|
| 148 |
The open-source implementation is written primarily in Python with an embedded browser-based visualization layer. The repository currently provides \nolinkurl{TURTLE.py}, a BERT extraction utility, reproducibility metadata and hidden-state data, dependency files, and a standalone GitHub Pages build. The software supports local Python execution as well as notebook environments.
|
| 149 |
|
| 150 |
Three functions capture the core scientific operations. \nolinkurl{embedding_to_nd_walk} constructs the coordinate-wise n-dimensional path whose endpoint is the original vector. \nolinkurl{representation_trajectory_to_3d} projects an arbitrary matrix of successive model states into three PCA dimensions. \nolinkurl{project_trajectories_shared} concatenates multiple trajectories before projection so that comparative paths use the same PCA basis. These functions separate the underlying geometric operations from the interactive display layer.
|
|
|
|
| 175 |
\end{verbatim}
|
| 176 |
|
| 177 |
|
|
|
|
| 178 |
\section{Potential Research Applications}\label{potential-research-applications}
|
| 179 |
|
| 180 |
TURTLE can be extended beyond the BERT homonym demonstration in several directions. First, layerwise trajectories can be compared for ambiguous words across contexts to study when representational paths diverge. Second, multilingual models could be used to compare translations or semantically aligned expressions across languages, provided that comparisons are performed within a shared representation space and with a shared projection basis. Third, prompt or sentence perturbations could be treated as controlled changes and their resulting trajectories compared quantitatively in the original space and visually in projection.
|
|
|
|
| 184 |
A further application is pedagogical. Conventional diagrams often present embeddings as static dots in a semantic map. TURTLE instead foregrounds movement: representations occupy positions, transformations create displacements, and sequences of transformations create trajectories. This can provide an intuitive bridge from elementary vector arithmetic to the layerwise dynamics of transformer models.
|
| 185 |
|
| 186 |
|
|
|
|
| 187 |
\section{Limitations and Scientific Cautions}\label{limitations-and-scientific-cautions}
|
| 188 |
|
| 189 |
Several limitations should be stated explicitly. First, glyph mode is a deterministic encoding of vector coordinates into a three-dimensional curve, not a faithful projection of the original geometry. Similar-looking glyphs should not be assumed to imply similar embeddings without empirical validation.
|
|
|
|
| 212 |
\section*{AI-use disclosure}
|
| 213 |
The original report records implementation assistance from OpenAI Codex and GPT-5.6. This report was prepared with assistance from Codex. William Taggart is responsible for the final text and its claims. AI assistance does not constitute independent validation of the implementation or scientific interpretation.
|
| 214 |
|
| 215 |
+
\enlargethispage{2\baselineskip}
|
| 216 |
\bibliographystyle{ACM-Reference-Format}
|
| 217 |
\bibliography{references}
|
| 218 |
|
| 219 |
\section*{Suggested citation}
|
| 220 |
+
William Taggart. 2026. TURTLE: Trajectory-based Understanding and Rendering of Transformations in Latent Embeddings. Technical report. Modern and Classical Languages and Literatures, Virginia Tech. November 13, 2026. \url{https://github.com/FL2744/TURTLE}
|
| 221 |
|
| 222 |
\end{document}
|