will-taggart commited on
Commit
35cd74e
·
verified ·
1 Parent(s): 2d79fb8

Improve technical paper pagination and figure placement

Browse files
TURTLE_technical_report.pdf CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:b4d80e9894e82872eb3fca21ec966ac28e1d120ff016628a690a22176809a5f4
3
- size 1034478
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:bc4f15a26275acc395f6110249fc67674b0db59b5e95a8b08150dba38d4d6bdd
3
+ size 1033560
paper/PAPER-SHA256SUMS.txt CHANGED
@@ -1,8 +1,8 @@
1
  413cfbb21a074dde8c7f74187c3b74dfbdbc6e4df0727ed465b6ce355c28b553 ILLUSTRATIONS.json
2
  e63a82a980aaba19c485bd4a876006011321b9b1b5f7320bf956db13dcf3cd0f README.md
3
- 7376d489335e900bea5212c07674bd400bea3b832bda6dbe8a1cfb937a92ad33 TECHNICAL_PAPER.md
4
- 32e4e5d8d43f1fbb30e99d3c7bcb8bc7ff7648b8378d0c9efdea41f7ddd93e67 TURTLE-ACM-source.zip
5
- b4d80e9894e82872eb3fca21ec966ac28e1d120ff016628a690a22176809a5f4 TURTLE.pdf
6
  6c03cdd6818b1ea39082bccd9815ca347d7a330446dc9b3762512c7ce7e4bc0d figure-specifications.json
7
  e9abf7692e076f7291784f68e75b9fac5221600a4536745f0494ebe4a3faece9 figures/illustration-01.pdf
8
  bfb5cadc932b48db0465e2ea099ec0d8113f6cc8be1453159329bdbb48502ef6 figures/illustration-01.png
@@ -15,6 +15,6 @@ e0caef75e8414e5abae1cf00885ce69559eb5095908752b04a5980eea19cb372 figures/illust
15
  49bc6ba2f3fde53b07d564e7e6ffa85943b363aaf4482f9edf8b568cb4fdc690 figures/illustration-05.pdf
16
  7b7bd8ae848704f99ccd333457fe76f2c2873cc698057d747fea46bf98475ed8 figures/illustration-05.png
17
  c408e4e2a3807e2b90cdca484ce70ae7fb328ab2613e008461f889f1ee896a2e main.bbl
18
- 54121e6e28f70add57533fcebb9a782b5e604cbbcb1637b273f95c62ecf05f67 main.tex
19
  a0ed7d55bc20e4622b683487006b7744cbad6d325ed5c05898b0d5b57b807bfa references.bib
20
  eed8b979a5a299b42e925d9027332e7df876400dcef046f2cfe26beef91d1189 render_figures.py
 
1
  413cfbb21a074dde8c7f74187c3b74dfbdbc6e4df0727ed465b6ce355c28b553 ILLUSTRATIONS.json
2
  e63a82a980aaba19c485bd4a876006011321b9b1b5f7320bf956db13dcf3cd0f README.md
3
+ 9878c3ec77d06db90a72f368b6c012fe81822412f01c95565f628c8711d55fc1 TECHNICAL_PAPER.md
4
+ 941d8e4b2bd7d19cb4a9b99197d9c54e4499ad638c4258ecf0c9c07c42f87f1d TURTLE-ACM-source.zip
5
+ bc4f15a26275acc395f6110249fc67674b0db59b5e95a8b08150dba38d4d6bdd TURTLE.pdf
6
  6c03cdd6818b1ea39082bccd9815ca347d7a330446dc9b3762512c7ce7e4bc0d figure-specifications.json
7
  e9abf7692e076f7291784f68e75b9fac5221600a4536745f0494ebe4a3faece9 figures/illustration-01.pdf
8
  bfb5cadc932b48db0465e2ea099ec0d8113f6cc8be1453159329bdbb48502ef6 figures/illustration-01.png
 
15
  49bc6ba2f3fde53b07d564e7e6ffa85943b363aaf4482f9edf8b568cb4fdc690 figures/illustration-05.pdf
16
  7b7bd8ae848704f99ccd333457fe76f2c2873cc698057d747fea46bf98475ed8 figures/illustration-05.png
17
  c408e4e2a3807e2b90cdca484ce70ae7fb328ab2613e008461f889f1ee896a2e main.bbl
18
+ 60c0c467404444b9935f7dc363ef035becfe6a47b2b2ef9e096c53cd00a1c8b5 main.tex
19
  a0ed7d55bc20e4622b683487006b7744cbad6d325ed5c05898b0d5b57b807bfa references.bib
20
  eed8b979a5a299b42e925d9027332e7df876400dcef046f2cfe26beef91d1189 render_figures.py
paper/TECHNICAL_PAPER.md CHANGED
@@ -10,24 +10,6 @@ Report date: November 13, 2026
10
 
11
  Large language models represent tokens and contextual states in high-dimensional vector spaces that are difficult to inspect directly. This report presents TURTLE (Trajectory-based Understanding and Rendering of Transformations in Latent Embeddings), a framework that adapts ideas from turtle geometry to the visualization of vector embeddings and layerwise representation trajectories. TURTLE distinguishes three complementary modes. Glyph mode converts the coordinates of a vector into a three-dimensional turtle signature for pedagogical visualization. Geometric mode constructs a literal coordinate-wise walk in the original n-dimensional space and projects that walk into three dimensions using principal component analysis (PCA). Trajectory mode accepts actual model states, allowing a token's contextual representation to be followed across successive transformer layers. The current open-source implementation includes a reproducible BERT-base demonstration comparing the hidden-state trajectory of the word 'jaguar' in animal and automobile contexts. The two trajectories are projected through a shared PCA basis, placing them in the same three-dimensional coordinate system and allowing their projected relative positions to be compared. TURTLE therefore provides both an explanatory metaphor and an experimental visualization method for studying change in high-dimensional neural representations. Its principal limitations arise from dimensionality reduction, basis dependence in coordinate walks, and the risk of over-interpreting visual similarity.
12
 
13
- ## Application interface
14
-
15
- ![Glyph mode turns the displayed 768-dimensional vector into a three-dimensional turtle path using a 45-degree consecutive angle.](figures/screenshot-01.png)
16
-
17
- *Glyph mode turns the displayed 768-dimensional vector into a three-dimensional turtle path using a 45-degree consecutive angle.*
18
-
19
- ![Geometric mode displays the n-dimensional walk projected into three dimensions by PCA.](figures/screenshot-02.png)
20
-
21
- *Geometric mode displays the n-dimensional walk projected into three dimensions by PCA.*
22
-
23
- ![Trajectory mode displays bundled BERT hidden states for jaguar in animal and automobile contexts using a shared PCA projection.](figures/screenshot-03.png)
24
-
25
- *Trajectory mode displays bundled BERT hidden states for jaguar in animal and automobile contexts using a shared PCA projection.*
26
-
27
- ![Glyph mode after changing the consecutive angle to 90 degrees, illustrating how the same encoding controls alter the displayed path.](figures/screenshot-04.png)
28
-
29
- *Glyph mode after changing the consecutive angle to 90 degrees, illustrating how the same encoding controls alter the displayed path.*
30
-
31
  ## Introduction
32
 
33
  Modern language models encode words, tokens, and contextual states as vectors with hundreds or thousands of numerical dimensions. These representations are computationally tractable but visually inaccessible. Researchers commonly use dimensionality-reduction techniques such as principal component analysis to create two- or three-dimensional views of high-dimensional data, but the resulting scatterplots tend to emphasize static positions. They are less naturally suited to communicating how a representation changes as a model processes context or as a token passes through successive network layers.
@@ -40,6 +22,9 @@ TURTLE - Trajectory-based Understanding and Rendering of Transformations in Late
40
 
41
  Let an embedding or hidden representation be a vector v in $\mathbb{R}^n$. A turtle position at step t is represented by $p_t$ in $\mathbb{R}^n$, and a displacement by $\Delta p_t$. The generic movement rule is
42
 
 
 
 
43
  $$p_{t+1}=p_t+\Delta p_t.$$
44
 
45
  This rule is independent of dimensionality. The same operation applies in two dimensions, three dimensions, or a latent space with hundreds or thousands of coordinates. The central challenge is therefore not whether turtle motion can be defined in n dimensions, but how the motion should be constructed and how it should be rendered for human inspection.
@@ -58,6 +43,9 @@ The current implementation uses PCA, computed through singular-value decompositi
58
 
59
  The current implementation deliberately distinguishes three modes so that artistic encoding, geometric projection, and model-state analysis are not conflated.
60
 
 
 
 
61
  ### Glyph mode: turtle encoding
62
 
63
  Glyph mode retains the original pedagogical prototype. For each scalar coordinate $v_i$, the magnitude $|v_i|$ determines segment length, the sign determines a right or left turn, and a fixed turning angle separates consecutive segments. An index-driven roll changes the local turning plane so that the generated trace develops three-dimensional structure. The mapping can be written schematically as
@@ -88,22 +76,26 @@ $$\Delta h^{(l)}=h^{(l+1)}-h^{(l)}.$$
88
 
89
  Here the order is intrinsic to the computation: layer l precedes layer $l+1$. Prior work has shown that contextual representations change substantially across contexts and layers, and has found evidence that different kinds of linguistic information are distributed differently across BERT's depth [3](#ref-ethayarajh); [8](#ref-tenney). TURTLE provides a direct geometric visualization of such layerwise change rather than reducing each state to an isolated point without an explicit path.
90
 
91
-
92
  ## Proof-of-Concept: Contextual Trajectories in BERT
93
 
94
  The current repository contains a reproducible demonstration built from `google-bert/bert-base-uncased`, the 12-layer, 768-hidden-dimension BERT-base model described by [2](#ref-devlin). TURTLE extracts the representation of the word 'jaguar' in two sentences: 'The jaguar rested quietly in the jungle.' and 'The Jaguar accelerated quickly out of the garage.' The first context selects the animal sense, while the second selects the automobile sense.
95
 
 
 
 
96
  The extraction script requests hidden states from the model and gathers the target word's representation from the embedding output and each of the 12 transformer blocks, producing a $13\times768$ matrix for each sentence. If a target word occupies multiple subword tokens, the implementation mean-pools the corresponding subword-token representations at each layer. The resulting matrices and provenance metadata are stored with the project so that the demonstration can be reproduced.
97
 
98
  For comparative visualization, TURTLE does not fit PCA separately to the two contextual trajectories. Instead, the $13\times768$ matrices are concatenated, a single PCA basis is fit to the combined set, and the projected points are then separated back into two paths. This shared basis is essential: two independent PCA solutions would have unrelated axes, making visual comparisons of orientation or relative position potentially misleading. In the shared projection, both paths inhabit the same three-dimensional coordinate system.
99
 
100
  The demonstration should not be interpreted as evidence that three dimensions recover the full geometry of BERT's 768-dimensional hidden states. Its purpose is narrower: to show that TURTLE can accept genuine model representations, preserve their computational ordering, and render their transformations as an inspectable path. The most important object remains the original high-dimensional trajectory; the three-dimensional view is an interface to it.
101
 
102
-
103
  ## Implementation
104
 
105
  The open-source implementation is written primarily in Python with an embedded browser-based visualization layer. The repository currently provides `TURTLE.py`, a BERT extraction utility, reproducibility metadata and hidden-state data, dependency files, and a standalone GitHub Pages build. The software supports local Python execution as well as notebook environments.
106
 
 
 
 
107
  Three functions capture the core scientific operations. `embedding_to_nd_walk` constructs the coordinate-wise n-dimensional path whose endpoint is the original vector. `representation_trajectory_to_3d` projects an arbitrary matrix of successive model states into three PCA dimensions. `project_trajectories_shared` concatenates multiple trajectories before projection so that comparative paths use the same PCA basis. These functions separate the underlying geometric operations from the interactive display layer.
108
 
109
  The implementation also enforces an interpretive boundary in the interface. Decorative transformations are disabled for data-derived geometric projections and model trajectories, while they remain available in glyph mode. This design choice reduces the risk that a user will mistake a manually altered curve for a projection of the source representation.
@@ -131,7 +123,6 @@ split the projected matrix back into H_1,...,H_m paths
131
  return comparable 3D trajectories
132
  ```
133
 
134
-
135
  ## Potential Research Applications
136
 
137
  TURTLE can be extended beyond the BERT homonym demonstration in several directions. First, layerwise trajectories can be compared for ambiguous words across contexts to study when representational paths diverge. Second, multilingual models could be used to compare translations or semantically aligned expressions across languages, provided that comparisons are performed within a shared representation space and with a shared projection basis. Third, prompt or sentence perturbations could be treated as controlled changes and their resulting trajectories compared quantitatively in the original space and visually in projection.
@@ -140,7 +131,6 @@ The framework is also compatible with richer measurements than simple visual ins
140
 
141
  A further application is pedagogical. Conventional diagrams often present embeddings as static dots in a semantic map. TURTLE instead foregrounds movement: representations occupy positions, transformations create displacements, and sequences of transformations create trajectories. This can provide an intuitive bridge from elementary vector arithmetic to the layerwise dynamics of transformer models.
142
 
143
-
144
  ## Limitations and Scientific Cautions
145
 
146
  Several limitations should be stated explicitly. First, glyph mode is a deterministic encoding of vector coordinates into a three-dimensional curve, not a faithful projection of the original geometry. Similar-looking glyphs should not be assumed to imply similar embeddings without empirical validation.
 
10
 
11
  Large language models represent tokens and contextual states in high-dimensional vector spaces that are difficult to inspect directly. This report presents TURTLE (Trajectory-based Understanding and Rendering of Transformations in Latent Embeddings), a framework that adapts ideas from turtle geometry to the visualization of vector embeddings and layerwise representation trajectories. TURTLE distinguishes three complementary modes. Glyph mode converts the coordinates of a vector into a three-dimensional turtle signature for pedagogical visualization. Geometric mode constructs a literal coordinate-wise walk in the original n-dimensional space and projects that walk into three dimensions using principal component analysis (PCA). Trajectory mode accepts actual model states, allowing a token's contextual representation to be followed across successive transformer layers. The current open-source implementation includes a reproducible BERT-base demonstration comparing the hidden-state trajectory of the word 'jaguar' in animal and automobile contexts. The two trajectories are projected through a shared PCA basis, placing them in the same three-dimensional coordinate system and allowing their projected relative positions to be compared. TURTLE therefore provides both an explanatory metaphor and an experimental visualization method for studying change in high-dimensional neural representations. Its principal limitations arise from dimensionality reduction, basis dependence in coordinate walks, and the risk of over-interpreting visual similarity.
12
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
13
  ## Introduction
14
 
15
  Modern language models encode words, tokens, and contextual states as vectors with hundreds or thousands of numerical dimensions. These representations are computationally tractable but visually inaccessible. Researchers commonly use dimensionality-reduction techniques such as principal component analysis to create two- or three-dimensional views of high-dimensional data, but the resulting scatterplots tend to emphasize static positions. They are less naturally suited to communicating how a representation changes as a model processes context or as a token passes through successive network layers.
 
22
 
23
  Let an embedding or hidden representation be a vector v in $\mathbb{R}^n$. A turtle position at step t is represented by $p_t$ in $\mathbb{R}^n$, and a displacement by $\Delta p_t$. The generic movement rule is
24
 
25
+ ![Geometric mode displays the n-dimensional walk projected into three dimensions by PCA.](figures/screenshot-02.png)
26
+
27
+ *Geometric mode displays the n-dimensional walk projected into three dimensions by PCA.*
28
  $$p_{t+1}=p_t+\Delta p_t.$$
29
 
30
  This rule is independent of dimensionality. The same operation applies in two dimensions, three dimensions, or a latent space with hundreds or thousands of coordinates. The central challenge is therefore not whether turtle motion can be defined in n dimensions, but how the motion should be constructed and how it should be rendered for human inspection.
 
43
 
44
  The current implementation deliberately distinguishes three modes so that artistic encoding, geometric projection, and model-state analysis are not conflated.
45
 
46
+ ![Glyph mode turns the displayed 768-dimensional vector into a three-dimensional turtle path using a 45-degree consecutive angle.](figures/screenshot-01.png)
47
+
48
+ *Glyph mode turns the displayed 768-dimensional vector into a three-dimensional turtle path using a 45-degree consecutive angle.*
49
  ### Glyph mode: turtle encoding
50
 
51
  Glyph mode retains the original pedagogical prototype. For each scalar coordinate $v_i$, the magnitude $|v_i|$ determines segment length, the sign determines a right or left turn, and a fixed turning angle separates consecutive segments. An index-driven roll changes the local turning plane so that the generated trace develops three-dimensional structure. The mapping can be written schematically as
 
76
 
77
  Here the order is intrinsic to the computation: layer l precedes layer $l+1$. Prior work has shown that contextual representations change substantially across contexts and layers, and has found evidence that different kinds of linguistic information are distributed differently across BERT's depth [3](#ref-ethayarajh); [8](#ref-tenney). TURTLE provides a direct geometric visualization of such layerwise change rather than reducing each state to an isolated point without an explicit path.
78
 
 
79
  ## Proof-of-Concept: Contextual Trajectories in BERT
80
 
81
  The current repository contains a reproducible demonstration built from `google-bert/bert-base-uncased`, the 12-layer, 768-hidden-dimension BERT-base model described by [2](#ref-devlin). TURTLE extracts the representation of the word 'jaguar' in two sentences: 'The jaguar rested quietly in the jungle.' and 'The Jaguar accelerated quickly out of the garage.' The first context selects the animal sense, while the second selects the automobile sense.
82
 
83
+ ![Trajectory mode displays bundled BERT hidden states for jaguar in animal and automobile contexts using a shared PCA projection.](figures/screenshot-03.png)
84
+
85
+ *Trajectory mode displays bundled BERT hidden states for jaguar in animal and automobile contexts using a shared PCA projection.*
86
  The extraction script requests hidden states from the model and gathers the target word's representation from the embedding output and each of the 12 transformer blocks, producing a $13\times768$ matrix for each sentence. If a target word occupies multiple subword tokens, the implementation mean-pools the corresponding subword-token representations at each layer. The resulting matrices and provenance metadata are stored with the project so that the demonstration can be reproduced.
87
 
88
  For comparative visualization, TURTLE does not fit PCA separately to the two contextual trajectories. Instead, the $13\times768$ matrices are concatenated, a single PCA basis is fit to the combined set, and the projected points are then separated back into two paths. This shared basis is essential: two independent PCA solutions would have unrelated axes, making visual comparisons of orientation or relative position potentially misleading. In the shared projection, both paths inhabit the same three-dimensional coordinate system.
89
 
90
  The demonstration should not be interpreted as evidence that three dimensions recover the full geometry of BERT's 768-dimensional hidden states. Its purpose is narrower: to show that TURTLE can accept genuine model representations, preserve their computational ordering, and render their transformations as an inspectable path. The most important object remains the original high-dimensional trajectory; the three-dimensional view is an interface to it.
91
 
 
92
  ## Implementation
93
 
94
  The open-source implementation is written primarily in Python with an embedded browser-based visualization layer. The repository currently provides `TURTLE.py`, a BERT extraction utility, reproducibility metadata and hidden-state data, dependency files, and a standalone GitHub Pages build. The software supports local Python execution as well as notebook environments.
95
 
96
+ ![Glyph mode after changing the consecutive angle to 90 degrees, illustrating how the same encoding controls alter the displayed path.](figures/screenshot-04.png)
97
+
98
+ *Glyph mode after changing the consecutive angle to 90 degrees, illustrating how the same encoding controls alter the displayed path.*
99
  Three functions capture the core scientific operations. `embedding_to_nd_walk` constructs the coordinate-wise n-dimensional path whose endpoint is the original vector. `representation_trajectory_to_3d` projects an arbitrary matrix of successive model states into three PCA dimensions. `project_trajectories_shared` concatenates multiple trajectories before projection so that comparative paths use the same PCA basis. These functions separate the underlying geometric operations from the interactive display layer.
100
 
101
  The implementation also enforces an interpretive boundary in the interface. Decorative transformations are disabled for data-derived geometric projections and model trajectories, while they remain available in glyph mode. This design choice reduces the risk that a user will mistake a manually altered curve for a projection of the source representation.
 
123
  return comparable 3D trajectories
124
  ```
125
 
 
126
  ## Potential Research Applications
127
 
128
  TURTLE can be extended beyond the BERT homonym demonstration in several directions. First, layerwise trajectories can be compared for ambiguous words across contexts to study when representational paths diverge. Second, multilingual models could be used to compare translations or semantically aligned expressions across languages, provided that comparisons are performed within a shared representation space and with a shared projection basis. Third, prompt or sentence perturbations could be treated as controlled changes and their resulting trajectories compared quantitatively in the original space and visually in projection.
 
131
 
132
  A further application is pedagogical. Conventional diagrams often present embeddings as static dots in a semantic map. TURTLE instead foregrounds movement: representations occupy positions, transformations create displacements, and sequences of transformations create trajectories. This can provide an intuitive bridge from elementary vector arithmetic to the layerwise dynamics of transformer models.
133
 
 
134
  ## Limitations and Scientific Cautions
135
 
136
  Several limitations should be stated explicitly. First, glyph mode is a deterministic encoding of vector coordinates into a three-dimensional curve, not a faithful projection of the original geometry. Similar-looking glyphs should not be assumed to imply similar embeddings without empirical validation.
paper/TURTLE-ACM-source.zip CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:32e4e5d8d43f1fbb30e99d3c7bcb8bc7ff7648b8378d0c9efdea41f7ddd93e67
3
- size 3372549
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:941d8e4b2bd7d19cb4a9b99197d9c54e4499ad638c4258ecf0c9c07c42f87f1d
3
+ size 3371125
paper/TURTLE.pdf CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:b4d80e9894e82872eb3fca21ec966ac28e1d120ff016628a690a22176809a5f4
3
- size 1034478
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:bc4f15a26275acc395f6110249fc67674b0db59b5e95a8b08150dba38d4d6bdd
3
+ size 1033560
paper/main.tex CHANGED
@@ -7,6 +7,20 @@
7
  \DefineVerbatimEnvironment{verbatim}{Verbatim}{breaklines=true,fontsize=\small}
8
  \providecommand{\tightlist}{\setlength{\itemsep}{0pt}\setlength{\parskip}{0pt}}
9
  \usepackage{graphicx,float}
 
 
 
 
 
 
 
 
 
 
 
 
 
 
10
  \begin{document}
11
  \title{TURTLE: Trajectory-based Understanding and Rendering of Transformations in Latent Embeddings}
12
  \author{William Taggart}
@@ -28,39 +42,17 @@ Turtle geometry offers a different conceptual starting point. In the classical f
28
 
29
  TURTLE - Trajectory-based Understanding and Rendering of Transformations in Latent Embeddings - adapts this movement-centered perspective to neural representation spaces. The framework is designed around a simple shift in emphasis: an embedding can be treated not only as a point, but as a destination or state within a trajectory. A difference between representations can then be represented as a displacement, and a sequence of representations as a walk. Throughout this report, \emph{latent representations} refers collectively to embedding vectors and contextual hidden states; where the distinction matters, the more specific term is used. This report updates the original proposal to describe the working open-source implementation, including a proof-of-concept experiment using real BERT hidden states.
30
 
31
- \section{Application interface}
32
- The application exposes its principal operations through the following workspaces and controls.
33
- \begin{figure}[H]
34
- \centering
35
- \includegraphics[width=\linewidth,height=0.70\textheight,keepaspectratio]{figures/screenshot-01.png}
36
- \caption{Glyph mode turns the displayed 768-dimensional vector into a three-dimensional turtle path using a 45-degree consecutive angle.}
37
- \Description{Glyph mode turns the displayed 768-dimensional vector into a three-dimensional turtle path using a 45-degree consecutive angle.}
38
- \label{fig:app-screen-1}
39
- \end{figure}
40
- \begin{figure}[H]
41
  \centering
42
- \includegraphics[width=\linewidth,height=0.70\textheight,keepaspectratio]{figures/screenshot-02.png}
43
  \caption{Geometric mode displays the n-dimensional walk projected into three dimensions by PCA.}
44
  \Description{Geometric mode displays the n-dimensional walk projected into three dimensions by PCA.}
45
  \label{fig:app-screen-2}
46
  \end{figure}
47
- \begin{figure}[H]
48
- \centering
49
- \includegraphics[width=\linewidth,height=0.70\textheight,keepaspectratio]{figures/screenshot-03.png}
50
- \caption{Trajectory mode displays bundled BERT hidden states for jaguar in animal and automobile contexts using a shared PCA projection.}
51
- \Description{Trajectory mode displays bundled BERT hidden states for jaguar in animal and automobile contexts using a shared PCA projection.}
52
- \label{fig:app-screen-3}
53
- \end{figure}
54
- \begin{figure}[H]
55
- \centering
56
- \includegraphics[width=\linewidth,height=0.70\textheight,keepaspectratio]{figures/screenshot-04.png}
57
- \caption{Glyph mode after changing the consecutive angle to 90 degrees, illustrating how the same encoding controls alter the displayed path.}
58
- \Description{Glyph mode after changing the consecutive angle to 90 degrees, illustrating how the same encoding controls alter the displayed path.}
59
- \label{fig:app-screen-4}
60
- \end{figure}
61
-
62
- \section{Mathematical Foundation}\label{mathematical-foundation}
63
-
64
  Let an embedding or hidden representation be a vector v in \(\mathbb{R}^n\). A turtle position at step t is represented by \(p_t\) in \(\mathbb{R}^n\), and a displacement by \(\Delta p_t\). The generic movement rule is
65
 
66
  \[p_{t+1}=p_t+\Delta p_t.\]
@@ -73,9 +65,9 @@ TURTLE separates these questions. The computational trajectory may remain in the
73
 
74
  The current implementation uses PCA, computed through singular-value decomposition of centered trajectory points. PCA constructs orthogonal components that successively maximize variance, providing a standard linear method for reducing dimensionality while retaining as much variance as possible in the chosen number of components \citep{jolliffe}. The visible path should therefore be understood as a three-dimensional shadow of a higher-dimensional trajectory, not as the trajectory itself.
75
 
76
- \begin{figure}[htbp]
77
  \centering
78
- \includegraphics[width=\textwidth]{figures/illustration-01.pdf}
79
  \caption{Mode taxonomy. Glyphs encode a vector, geometric mode projects a coordinate-wise walk, and trajectory mode projects ordered model states. The modes should not be interpreted interchangeably.}
80
  \Description{Three modes, three different interpretations. Mode taxonomy. Glyphs encode a vector, geometric mode projects a coordinate-wise walk, and trajectory mode projects ordered model states. The modes should not be interpreted interchangeably.}
81
  \label{fig:illustration-1}
@@ -83,6 +75,14 @@ The current implementation uses PCA, computed through singular-value decompositi
83
 
84
  \section{TURTLE Modes}\label{turtle-modes}
85
 
 
 
 
 
 
 
 
 
86
  The current implementation deliberately distinguishes three modes so that artistic encoding, geometric projection, and model-state analysis are not conflated.
87
 
88
  \subsection{Glyph mode: turtle encoding}\label{glyph-mode-turtle-encoding}
@@ -116,9 +116,16 @@ The displacement between successive layers is
116
  Here the order is intrinsic to the computation: layer l precedes layer \(l+1\). Prior work has shown that contextual representations change substantially across contexts and layers, and has found evidence that different kinds of linguistic information are distributed differently across BERT's depth \citep{ethayarajh, tenney}. TURTLE provides a direct geometric visualization of such layerwise change rather than reducing each state to an isolated point without an explicit path.
117
 
118
 
119
-
120
  \section{Proof-of-Concept: Contextual Trajectories in BERT}\label{proof-of-concept-contextual-trajectories-in-bert}
121
 
 
 
 
 
 
 
 
 
122
  The current repository contains a reproducible demonstration built from \nolinkurl{google-bert/bert-base-uncased}, the 12-layer, 768-hidden-dimension BERT-base model described by \citep{devlin}. TURTLE extracts the representation of the word `jaguar' in two sentences: `The jaguar rested quietly in the jungle.' and `The Jaguar accelerated quickly out of the garage.' The first context selects the animal sense, while the second selects the automobile sense.
123
 
124
  The extraction script requests hidden states from the model and gathers the target word's representation from the embedding output and each of the 12 transformer blocks, producing a \(13\times768\) matrix for each sentence. If a target word occupies multiple subword tokens, the implementation mean-pools the corresponding subword-token representations at each layer. The resulting matrices and provenance metadata are stored with the project so that the demonstration can be reproduced.
@@ -128,9 +135,16 @@ For comparative visualization, TURTLE does not fit PCA separately to the two con
128
  The demonstration should not be interpreted as evidence that three dimensions recover the full geometry of BERT's 768-dimensional hidden states. Its purpose is narrower: to show that TURTLE can accept genuine model representations, preserve their computational ordering, and render their transformations as an inspectable path. The most important object remains the original high-dimensional trajectory; the three-dimensional view is an interface to it.
129
 
130
 
131
-
132
  \section{Implementation}\label{implementation}
133
 
 
 
 
 
 
 
 
 
134
  The open-source implementation is written primarily in Python with an embedded browser-based visualization layer. The repository currently provides \nolinkurl{TURTLE.py}, a BERT extraction utility, reproducibility metadata and hidden-state data, dependency files, and a standalone GitHub Pages build. The software supports local Python execution as well as notebook environments.
135
 
136
  Three functions capture the core scientific operations. \nolinkurl{embedding_to_nd_walk} constructs the coordinate-wise n-dimensional path whose endpoint is the original vector. \nolinkurl{representation_trajectory_to_3d} projects an arbitrary matrix of successive model states into three PCA dimensions. \nolinkurl{project_trajectories_shared} concatenates multiple trajectories before projection so that comparative paths use the same PCA basis. These functions separate the underlying geometric operations from the interactive display layer.
@@ -161,7 +175,6 @@ return comparable 3D trajectories
161
  \end{verbatim}
162
 
163
 
164
-
165
  \section{Potential Research Applications}\label{potential-research-applications}
166
 
167
  TURTLE can be extended beyond the BERT homonym demonstration in several directions. First, layerwise trajectories can be compared for ambiguous words across contexts to study when representational paths diverge. Second, multilingual models could be used to compare translations or semantically aligned expressions across languages, provided that comparisons are performed within a shared representation space and with a shared projection basis. Third, prompt or sentence perturbations could be treated as controlled changes and their resulting trajectories compared quantitatively in the original space and visually in projection.
@@ -171,7 +184,6 @@ The framework is also compatible with richer measurements than simple visual ins
171
  A further application is pedagogical. Conventional diagrams often present embeddings as static dots in a semantic map. TURTLE instead foregrounds movement: representations occupy positions, transformations create displacements, and sequences of transformations create trajectories. This can provide an intuitive bridge from elementary vector arithmetic to the layerwise dynamics of transformer models.
172
 
173
 
174
-
175
  \section{Limitations and Scientific Cautions}\label{limitations-and-scientific-cautions}
176
 
177
  Several limitations should be stated explicitly. First, glyph mode is a deterministic encoding of vector coordinates into a three-dimensional curve, not a faithful projection of the original geometry. Similar-looking glyphs should not be assumed to imply similar embeddings without empirical validation.
@@ -200,10 +212,11 @@ The current TURTLE implementation, demonstration data, extraction script, depend
200
  \section*{AI-use disclosure}
201
  The original report records implementation assistance from OpenAI Codex and GPT-5.6. This report was prepared with assistance from Codex. William Taggart is responsible for the final text and its claims. AI assistance does not constitute independent validation of the implementation or scientific interpretation.
202
 
 
203
  \bibliographystyle{ACM-Reference-Format}
204
  \bibliography{references}
205
 
206
  \section*{Suggested citation}
207
- William Taggart. 2026. TURTLE: Trajectory-based Understanding and Rendering of Transformations in Latent Embeddings. Technical report. Modern and Classical Languages and Literatures, Virginia Tech. November 13, 2026. https://github.com/FL2744/TURTLE
208
 
209
  \end{document}
 
7
  \DefineVerbatimEnvironment{verbatim}{Verbatim}{breaklines=true,fontsize=\small}
8
  \providecommand{\tightlist}{\setlength{\itemsep}{0pt}\setlength{\parskip}{0pt}}
9
  \usepackage{graphicx,float}
10
+
11
+ % Permit figures to share pages with surrounding explanatory text.
12
+ \renewcommand{\topfraction}{0.92}
13
+ \renewcommand{\bottomfraction}{0.85}
14
+ \renewcommand{\textfraction}{0.06}
15
+ \renewcommand{\floatpagefraction}{0.80}
16
+ \setcounter{topnumber}{3}
17
+ \setcounter{bottomnumber}{3}
18
+ \setcounter{totalnumber}{5}
19
+ \setlength{\textfloatsep}{12pt plus 2pt minus 3pt}
20
+ \setlength{\intextsep}{12pt plus 2pt minus 3pt}
21
+ \raggedbottom
22
+
23
+ \geometry{top=0.75in,bottom=0.65in,includeheadfoot}
24
  \begin{document}
25
  \title{TURTLE: Trajectory-based Understanding and Rendering of Transformations in Latent Embeddings}
26
  \author{William Taggart}
 
42
 
43
  TURTLE - Trajectory-based Understanding and Rendering of Transformations in Latent Embeddings - adapts this movement-centered perspective to neural representation spaces. The framework is designed around a simple shift in emphasis: an embedding can be treated not only as a point, but as a destination or state within a trajectory. A difference between representations can then be represented as a displacement, and a sequence of representations as a walk. Throughout this report, \emph{latent representations} refers collectively to embedding vectors and contextual hidden states; where the distinction matters, the more specific term is used. This report updates the original proposal to describe the working open-source implementation, including a proof-of-concept experiment using real BERT hidden states.
44
 
45
+
46
+ \section{Mathematical Foundation}\label{mathematical-foundation}
47
+
48
+
49
+ \begin{figure}[!htb]
 
 
 
 
 
50
  \centering
51
+ \includegraphics[width=0.94\linewidth,height=0.58\textheight,keepaspectratio]{figures/screenshot-02.png}
52
  \caption{Geometric mode displays the n-dimensional walk projected into three dimensions by PCA.}
53
  \Description{Geometric mode displays the n-dimensional walk projected into three dimensions by PCA.}
54
  \label{fig:app-screen-2}
55
  \end{figure}
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
56
  Let an embedding or hidden representation be a vector v in \(\mathbb{R}^n\). A turtle position at step t is represented by \(p_t\) in \(\mathbb{R}^n\), and a displacement by \(\Delta p_t\). The generic movement rule is
57
 
58
  \[p_{t+1}=p_t+\Delta p_t.\]
 
65
 
66
  The current implementation uses PCA, computed through singular-value decomposition of centered trajectory points. PCA constructs orthogonal components that successively maximize variance, providing a standard linear method for reducing dimensionality while retaining as much variance as possible in the chosen number of components \citep{jolliffe}. The visible path should therefore be understood as a three-dimensional shadow of a higher-dimensional trajectory, not as the trajectory itself.
67
 
68
+ \begin{figure}[!htb]
69
  \centering
70
+ \includegraphics[width=0.94\textwidth]{figures/illustration-01.pdf}
71
  \caption{Mode taxonomy. Glyphs encode a vector, geometric mode projects a coordinate-wise walk, and trajectory mode projects ordered model states. The modes should not be interpreted interchangeably.}
72
  \Description{Three modes, three different interpretations. Mode taxonomy. Glyphs encode a vector, geometric mode projects a coordinate-wise walk, and trajectory mode projects ordered model states. The modes should not be interpreted interchangeably.}
73
  \label{fig:illustration-1}
 
75
 
76
  \section{TURTLE Modes}\label{turtle-modes}
77
 
78
+
79
+ \begin{figure}[!htb]
80
+ \centering
81
+ \includegraphics[width=0.94\linewidth,height=0.58\textheight,keepaspectratio]{figures/screenshot-01.png}
82
+ \caption{Glyph mode turns the displayed 768-dimensional vector into a three-dimensional turtle path using a 45-degree consecutive angle.}
83
+ \Description{Glyph mode turns the displayed 768-dimensional vector into a three-dimensional turtle path using a 45-degree consecutive angle.}
84
+ \label{fig:app-screen-1}
85
+ \end{figure}
86
  The current implementation deliberately distinguishes three modes so that artistic encoding, geometric projection, and model-state analysis are not conflated.
87
 
88
  \subsection{Glyph mode: turtle encoding}\label{glyph-mode-turtle-encoding}
 
116
  Here the order is intrinsic to the computation: layer l precedes layer \(l+1\). Prior work has shown that contextual representations change substantially across contexts and layers, and has found evidence that different kinds of linguistic information are distributed differently across BERT's depth \citep{ethayarajh, tenney}. TURTLE provides a direct geometric visualization of such layerwise change rather than reducing each state to an isolated point without an explicit path.
117
 
118
 
 
119
  \section{Proof-of-Concept: Contextual Trajectories in BERT}\label{proof-of-concept-contextual-trajectories-in-bert}
120
 
121
+
122
+ \begin{figure}[!htb]
123
+ \centering
124
+ \includegraphics[width=0.94\linewidth,height=0.58\textheight,keepaspectratio]{figures/screenshot-03.png}
125
+ \caption{Trajectory mode displays bundled BERT hidden states for jaguar in animal and automobile contexts using a shared PCA projection.}
126
+ \Description{Trajectory mode displays bundled BERT hidden states for jaguar in animal and automobile contexts using a shared PCA projection.}
127
+ \label{fig:app-screen-3}
128
+ \end{figure}
129
  The current repository contains a reproducible demonstration built from \nolinkurl{google-bert/bert-base-uncased}, the 12-layer, 768-hidden-dimension BERT-base model described by \citep{devlin}. TURTLE extracts the representation of the word `jaguar' in two sentences: `The jaguar rested quietly in the jungle.' and `The Jaguar accelerated quickly out of the garage.' The first context selects the animal sense, while the second selects the automobile sense.
130
 
131
  The extraction script requests hidden states from the model and gathers the target word's representation from the embedding output and each of the 12 transformer blocks, producing a \(13\times768\) matrix for each sentence. If a target word occupies multiple subword tokens, the implementation mean-pools the corresponding subword-token representations at each layer. The resulting matrices and provenance metadata are stored with the project so that the demonstration can be reproduced.
 
135
  The demonstration should not be interpreted as evidence that three dimensions recover the full geometry of BERT's 768-dimensional hidden states. Its purpose is narrower: to show that TURTLE can accept genuine model representations, preserve their computational ordering, and render their transformations as an inspectable path. The most important object remains the original high-dimensional trajectory; the three-dimensional view is an interface to it.
136
 
137
 
 
138
  \section{Implementation}\label{implementation}
139
 
140
+
141
+ \begin{figure}[!htb]
142
+ \centering
143
+ \includegraphics[width=0.94\linewidth,height=0.58\textheight,keepaspectratio]{figures/screenshot-04.png}
144
+ \caption{Glyph mode after changing the consecutive angle to 90 degrees, illustrating how the same encoding controls alter the displayed path.}
145
+ \Description{Glyph mode after changing the consecutive angle to 90 degrees, illustrating how the same encoding controls alter the displayed path.}
146
+ \label{fig:app-screen-4}
147
+ \end{figure}
148
  The open-source implementation is written primarily in Python with an embedded browser-based visualization layer. The repository currently provides \nolinkurl{TURTLE.py}, a BERT extraction utility, reproducibility metadata and hidden-state data, dependency files, and a standalone GitHub Pages build. The software supports local Python execution as well as notebook environments.
149
 
150
  Three functions capture the core scientific operations. \nolinkurl{embedding_to_nd_walk} constructs the coordinate-wise n-dimensional path whose endpoint is the original vector. \nolinkurl{representation_trajectory_to_3d} projects an arbitrary matrix of successive model states into three PCA dimensions. \nolinkurl{project_trajectories_shared} concatenates multiple trajectories before projection so that comparative paths use the same PCA basis. These functions separate the underlying geometric operations from the interactive display layer.
 
175
  \end{verbatim}
176
 
177
 
 
178
  \section{Potential Research Applications}\label{potential-research-applications}
179
 
180
  TURTLE can be extended beyond the BERT homonym demonstration in several directions. First, layerwise trajectories can be compared for ambiguous words across contexts to study when representational paths diverge. Second, multilingual models could be used to compare translations or semantically aligned expressions across languages, provided that comparisons are performed within a shared representation space and with a shared projection basis. Third, prompt or sentence perturbations could be treated as controlled changes and their resulting trajectories compared quantitatively in the original space and visually in projection.
 
184
  A further application is pedagogical. Conventional diagrams often present embeddings as static dots in a semantic map. TURTLE instead foregrounds movement: representations occupy positions, transformations create displacements, and sequences of transformations create trajectories. This can provide an intuitive bridge from elementary vector arithmetic to the layerwise dynamics of transformer models.
185
 
186
 
 
187
  \section{Limitations and Scientific Cautions}\label{limitations-and-scientific-cautions}
188
 
189
  Several limitations should be stated explicitly. First, glyph mode is a deterministic encoding of vector coordinates into a three-dimensional curve, not a faithful projection of the original geometry. Similar-looking glyphs should not be assumed to imply similar embeddings without empirical validation.
 
212
  \section*{AI-use disclosure}
213
  The original report records implementation assistance from OpenAI Codex and GPT-5.6. This report was prepared with assistance from Codex. William Taggart is responsible for the final text and its claims. AI assistance does not constitute independent validation of the implementation or scientific interpretation.
214
 
215
+ \enlargethispage{2\baselineskip}
216
  \bibliographystyle{ACM-Reference-Format}
217
  \bibliography{references}
218
 
219
  \section*{Suggested citation}
220
+ William Taggart. 2026. TURTLE: Trajectory-based Understanding and Rendering of Transformations in Latent Embeddings. Technical report. Modern and Classical Languages and Literatures, Virginia Tech. November 13, 2026. \url{https://github.com/FL2744/TURTLE}
221
 
222
  \end{document}