Input: What is the main methodology used in the research?

Context:
Paper Info

Title: On the Role of Emergent Communication for Social Learning in Multi-Agent Reinforcement Learning
Publish Date: Unkown
Author List: Seth Karten, Siva Kailas, Huao Li, Katia Sycara

Figure

Figure1.By using deep neural networks, our method seeks accurate representations between the state-message pair and future states while creating precise predictions with random states. Thus satisfying the utility objective of the supervised approach. The depicted agents are equipped with sensors to detect other cars.
Figure 2.An example of two possible classes, person and horse, from a single observation in the Pascal VOC game.
Figure 3. Blind Traffic Junction Left: Our method uses compositional complexity and contrastive utility to outperform other baselines in terms of performance and sample complexity.The legend provides the mean ± variance of the best performance.Right: Top: success, contrastive, and complexity losses for our method.Right, Bottom: success, autoencoder loss for ae-comm with supervised pretraining.
Figure 4. Pascal VOC Game Representing compositional concepts from raw pixel data in images to communicate multiple concepts within a single image.Our method significantly outperforms ae-comm and no-comm due to our framework being able to learn composable, independent concepts.
Figure 5. Blind Traffic Junction Social shadowing enables significantly lower sample complexity when compared to traditional online MARL.
Beta ablation: Messages are naturally sparse in bits due to the complexity loss.Redundancy measures the capacity for a bijection between the size of the set of unique tokens and the enumerated observations and intents.Min redundancy is 1.0 (a bijection).Lower is better.

abstract

Explicit communication among humans is key to coordinating and learning. Social learning, which uses cues from experts, can greatly benefit from the usage of explicit communication to align heterogeneous policies, reduce sample complexity, and solve partially observable tasks. Emergent communication, a type of explicit communication, studies the creation of an artificial language to encode a high task-utility message directly from data.
However, in most cases, emergent communication sends insufficiently compressed messages with little or null information, which also may not be understandable to a third-party listener. This paper proposes a supervised method utilizing deep neural networks to capture both referential complexity and task-specific utility to adequately explore structured social communication scenarios in multi-agent reinforcement learning (MARL).
We show that our model is able to i) develop a natural-language-inspired lexicon of messages that is independently composed of a set of emergent concepts, which span the observations and intents with minimal bits, ii) develop communication to align the action policies of heterogeneous agents with dissimilar feature models, and iii) learn a communication policy from watching an expert's action policy, which we term 'social shadowing'.

INTRODUCTION

Social learning agents analyze cues from direct observation of other agents (novice or expert) in the same environment to learn an action policy from others. However, observing expert actions may not be sufficient to coordinate with other agents. Rather, by learning to communicate, agents can better model the intent of other agents, leading to better coordination.
In humans, explicit communication for coordination assumes a common communication substrate to convey abstract concepts and beliefs directly , which may not be available for new partners. To align complex beliefs, heterogeneous agents must learn a message policy that translates from one theory of mind to another to synchronize coordination.
Especially when there is complex information to process and share, new agent partners need to learn to communicate to work with other agents. Emergent communication studies the creation of artificial language. Often phrased as a Lewis game, speakers and listeners learn a set of tokens to communicate complex observations .
However, in multi-agent reinforcement learning (MARL), agents suffer from partial observability and non-stationarity (due to unaligned value functions) , which aims to be solved with decentralized learning through communication. In the MARL setup, agents, as speakers and listeners, learn a set of tokens to communicate observations, intentions, coordination, or other experiences which help facilitate solving tasks .
Agents learn to communicate effectively through a backpropagation signal from their task performance . This has been found useful for applications in human-agent teaming , multirobot navigation , and coordination in complex games such as StarCraft II . Communication quality has been shown to have a strong relationship with task performance , leading to a multitude of work attempting to increase the representational capacity by decreasing the convergence rates .
Yet these methods still create degenerate communication protocols , which are uninterpretable due to joined concepts or null (lack of) information, which causes performance degradation. In this work, we investigate the challenges of learning a arXiv:2302.14276v1 LG] 28 Feb 2023 messaging lexicon to prepare emergent communication for social learning (EC4SL) scenarios.
We study the following hypotheses: H1) EC4SL will learn faster through structured concepts in messages leading to higher-quality solutions, H2) EC4SL aligns the policies of expert heterogeneous agents, and H3) EC4SL enables social shadowing, where an agent learns a communication policy while only observing an expert agent's action policy.
By learning a communication policy, the agent is encouraged to develop a more structured understanding of intent, leading to better coordination. The setting is very realistic among humans and many computer vision and RL frameworks may develop rich feature spaces for a specific solo task, but have not yet interacted with other agents, which may lead to failure without alignment.
We enable a structured emergent communication paradigm, which exhibits hierarchical and informative properties. We show theoretically and through empirical results that compositional language enables independence properties among tokens with respect to referential information. Additionally, when combined with contrastive learning, our method outperforms competing methods that only ground communication on referential information.
We show that contrastive learning is an optimal critic for communication, reducing sample complexity for the unsupervised emergent communication objective. In addition to the more human-like format, compositional communication is able to create variable-length messages, meaning that we are not limited to sending insufficiently compressed messages with little information, increasing the quality of each communication.
In order to test our hypotheses, we show the utility of our method in multi-agent settings with a focus on teams of agents, high-dimensional pixel data, and expansions to heterogeneous teams of agents of varying skill levels. Social learning requires agents to explore to observe and learn from expert cues.
We interpolate between this form of social learning and imitation learning, which learns action policies directly from examples. We introduce a 'social shadowing' learning approach where we use first-person observations, rather than third-person observations, to encourage the novice to learn latently or conceptually how to communicate and develop an understanding of intent for better coordination.
The social shadowing episodes are alternated with traditional MARL during training. Contrastive learning, which works best with positive examples, is apt for social shadowing. Originally derived to enable lower complexity emergent lexicons, we find that the contrastive learning objective is apt for agents to develop internal models and relationships of the task through social shadowing.
The idea is to enable a shared emergent communication substrate (with minimal bandwidth) to enable future coordi-nation with novel partners. Our contributions are deriving an optimal critic for a communication policy and showing that the information bottleneck helps extend communication to social learning scenarios.
In real-world tasks such as autonomous driving or robotics, humans do not necessarily learn from scratch. Rather they explore with conceptually guided information from expert mentors. In particular, having structured emergent messages reduces sample complexity, and contrastive learning can help novice agents learn from experts.
Emergent communication can also align heterogeneous agents, a social task that has not been previously studied.

Multi-Agent Signaling

Implicit communication conveys information to other agents that is not intentionally communicated . Implicit signaling conveys information to other agents based on one's observable physical position . Implicit signaling may be a form of implicit communication such as through social cues or explicit communication such as encoded into the MDP through "cheap talk" .
Unlike implicit signaling, explicit signaling is a form of positive signaling that seeks to directly influence the behavior of other agents in the hopes that the new information will lead to active listening. Multi-agent emergent communication is a type of explicit signaling which deliberately shares information.
Symbolic communication, a subset of explicit communication, seeks to send a subset of pre-defined messages. However, these symbols must be defined by an expert and do not scale to particularly complex observations and a large number of agents. Emergent communication aims to directly influence other agents with a learned subset of information, which allows for scalability and interpretability by new agents.

Emergent Communication

Several methodologies currently exist to increase the informativeness of emergent communication. With discrete and clustered continuous communication, the number of observed distinct communication tokens is far below the number permissible . As an attempt to increase the emergent "vocabulary" and decrease the data required to converge to an informative communication "language", work has added a bias loss to emit distinct tokens in different situations .
More recent work has found that the sample efficiency can be further improved by grounding communication in observation space with a supervised reconstruction loss . Information-maximizing autoencoders aim to maximize the state reconstruction accuracy for each agent. How-ever, grounding communication in observations has been found to easily satisfy these input-based objectives while still requiring a myriad more samples to explore to find a task-specific communication space .
Thus, it is necessary to use task-specific information to communicate informatively. This will enable learned compression for task completion rather than pure compression for input recovery. Other work aims to use the information bottleneck to decrease the entropy of messages . In our work, we use deep neural networks to increase representation accuracy with future goals, which we show optimally optimizes the Q-function for messages.

Natural Language Inspiration

The properties of the tokens in emergent communication directly affect their informative ability. As a baseline, continuous communication tokens can represent maximum information but lack human-interpretable properties. Discrete 1-hot (binary vector) tokens allow for a finite vocabulary, but each token contains the same magnitude of information, with equal orthogonal distance to each other token.
Similar to word embeddings in natural language, discrete prototypes are an effort to cluster similar information together from continuous vectors . Building on the continuous word embedding properties, VQ-VIB , an information-theoretic observation grounding based on VQ-VAE properties , uses variational properties to provide word embedding properties for continuous emergent tokens.
Like discrete prototypes, they exhibit a clustering property based on similar information but are more informative. However, each of these message types determines a single token for communication. Tokens are stringed together to create emergent "sentences".

Preliminaries

We formulate our setup as a decentralized, partially observable Markov Decision Process with communication (Dec-POMDP-Comm). Formally, our problem is defined by the tuple, S, A, M, T , R, O, Ω, γ . We define S as the set of states, A i , i ∈ [1, N ] as the set of actions, which includes task-specific actions, and M i as the set of communications for N agents.
T is the transition between states due to the multi-agent joint action space T : S × A 1 , ..., A N → S. Ω defines the set of observations in our partially observable setting. Partial observability requires communication to complete the tasks successfully. O i : M 1 , ..., M N × Ŝ → Ω maps the communications and local state, Ŝ, to a distribution of observations for each agent.
R defines the reward function and γ defines the discount factor.

Architecture

The policy network is defined by three stages: Observation Encoding, Communication, and Action Decoding. The best observation encoding and action decoding architecture is task-dependent, i.e., using multi-layer perceptrons (MLPs), CNNs , GRUs , or transformer layers are best suited to different inputs.
The encoder transforms observation and any sequence or memory information into an encoding H. The on-policy reinforcement learning training uses RE-INFORCE or a decentralized version of MAPPO as specified by our experiments. Our work focuses on the communication stage, which can be divided into three substages: message encoding, message passing (often considered sparse communication), and message decoding.
We use the message passing from . For message decoding, we build on a multiheaded attention framework, which allows an agent to learn which messages are most important . Our structured communication framework defines the message encoding, as described in section 4.

Objective

Mutual information, denoted as I(X; Y ), looks to measure the relationship between random variables, which is often measured through Kullback-Leibler divergence , I(X; Y ) = D KL (p(x, y)||p(x) ⊗ p(y)). The message encoding substage can be defined as an information bottleneck problem, which defines a tradeoff between the complexity of information (compression, I(X, X)) and the preserved relevant information (utility, I( X, Y )).
The deep variational information bottleneck defines a trade-off between preserving useful information and compression . We assume that our observation and memory/sequence encoder provides an optimal representation H i suitable for sharing relevant observation and intent/coordination information. We hope to recover a representation Y i , which contains the sufficient desired outputs.
In our scenario, the information bottleneck is a trade-off between the complexity of information I(H i ; M i ) (representing the encoded information exactly) and representing the relevant information I(M j =i ; Y i ), which is signaled from our contrastive objective. In our setup, the relevant information flows from other agents through communication, signaling a combination of the information bottleneck and a Lewis game.
We additionally promote complexity through our compositional independence objective, This is formulated by the following Lagrangian, where the bounds on mutual information Î are defined in equations 1, 2, and 10. Overall, our objective is,

Complexity through Compositional Communication

We aim to satisfy the complexity objective, I(H i , M i ), through compositional communication. In order to induce complexity in our communication, we want the messages to be as non-random as possible. That is, informative with respect to the input hidden state h. In addition, we want each token within the message to share as little information as possible with the preceding tokens.
Thus, each additional token adds only informative content. Each token has a fixed length in bits W . The total sequence is limited by a fixed limit, L l W l ≤ S, of S bits and a total of L tokens. We use a variational message generation setup, which maps the encoded hidden state h to a message m; that is, we are modeling the posterior, π i m (m l |h).
We limit the vocabulary size to K tokens, e j ∈ R D , j ∈ [1, K] ⊂ N, where each token has dimensionality D and l ∈ [1, L] ⊂ N. Each token m l is sampled from a categorical posterior distribution, 0 otherwise such that the message m l is mapped to the nearest neighbor e j . A set of these tokens makes a message m.
To satisfy the complexity objective, we want to use m i to well-represent h i and consist of independently informative m i l .

Independent Information

We derive an upper bound for the interaction information between all tokens. Proposition 4.1. For the interaction information between all tokens, the following upper bound holds: The proof is in Appendix A.1. Since we want the mutual information to be minimized in our objective, we minimize,

Input-Oriented Information

In order to induce complexity in the compositional messages, we additionally want to minimize the mutual information I(H; M ) between the composed message m and the encoded information h. We derive an upper bound on the mutual information that we use as a Lagrangian term to minimize. Proposition 4.2. For the mutual information between the composed message and encoded information, the following upper bound holds:
The proof is in Appendix A.1. Thus, we have our Lagrangian term, Conditioning on the input or observation data is a decentralized training objective.

Sequence Length

Compositional communication necessitates an adaptive limit on the total length of the sequence. Corollary 4.3. Repeat tokens, w, are redundant and can be removed. Suppose one predicts two arbitrary tokens, w k and w l . Given equation 1, it follows that there is low or near-zero mutual information between w k and w l .
A trivial issue is that the message generator will predict every available token as to follow the unique token objective. Since the tokens are imbued with input-oriented information (equation 2), the predicted tokens will be based on relevant referential details. Thus, it follows that tokens containing irrelevant information will not be chosen.
A nice optimization objective that follows from corollary 4.3 is that one can use self-supervised learning with an end-ofsequence (EOS) token to limit the variable total length of compositional message sequences. (3) Algorithm 1 Compositional Message Gen.(h t ) m i ∼ N ( ĥ; µ, σ) 9: end for 10: return m

Message Generation Architecture

Now, we can define the pipeline for message generation. The idea is to create an architecture that can generate features to enable independent message tokens. We expand each compressed token into the space of the hidden state h (1-layer linear expansion) since each token has a natural embedding in R |h| .
Then, we perform attention using a softmin to help minimize similarity with previous tokens and sample the new token from a variational distribution. See algorithm 1 for complete details. During execution, we can generate messages directly due to equation 1, resolving any computation time lost from sequential compositional message generation.

Utility through Contrastive Learning

First, note that our Markov Network is as follows: H j → M j → Y i ← H i . Continue to denote i as the agent identification and j as the agent ID such that j = i. We aim to satisfy the utility objective of the supervised approach, through deep neural networks as shown in figure 1. Proposition 5.1.
Utility mutual information is lower bounded by the contrastive NCE-binary objective, The proof is in Appendix A.1. This result shows a need for gradient information to flow backward across agents along communication edge connections.

Experiments and Results

We condition on inputs, especially rich information (such as pixel data), and task-specific information. When evaluating an artificial language in MARL, we are interested in referential tasks, in which communication is required to complete the task. With regard to intent-grounded communication, we study ordinal tasks, which require coordination information between agents to complete successfully.
Thus, we consider tasks with a team of agents to foster messaging that communicates coordination information that also includes their observations. To test H1, structuring emergent messages enables lower complexity, we test our methodology and analyze the input-oriented information and utility capabilities.
Next, we analyze the ability of heterogeneous agents to understand differing communication policies (H2)). Finally, we consider the effect of social shadowing (H3), in which agents solely learn a communication policy from an expert agent's action policy. We additionally analyze the role of offline reinforcement learning for emergent communication in combination with online reinforcement learning to further learn emergent communication alongside an action policy.
We evaluate each scenario over 10 seeds.

Environments

Blind Traffic Junction We consider a benchmark that requires both referential and ordinal capabilities within a team of agents. The blind traffic junction environment requires multiple agents to navigate a junction without any observation of other agents. Rather, they only observe their own state location.
Ten agents must coordinate to traverse through the lanes without colliding into agents within their lane or in the junction. Our training uses REINFORCE . Pascal VOC Game We further evaluate the complexity of compositional communication with a Pascal VOC . This is a two-agent referential game similar to the Cifar game but requires the prediction of multiple classes.
During each episode, each agent observes a random image from the Pascal VOC dataset containing exactly two unique labels. Each agent must encode information given only the raw pixels from the original image such that the other agent can recognize the two class labels in the original image. An agent receives a reward of 0.25 per correctly chosen class label and will receive a total reward of 1 if both agents guess all labels correctly.
See figure 2. Our training uses heterogeneous agents trained with PPO (modified from MAPPO repository). For simplicity of setup, we consider images with exactly two unique labels from a closed subset of size five labels of the original set of labels from the Pascal VOC data. Furthermore, these images must be of size 375 × 500 pixels.
Thus, the resultant dataset comprised 534 unique images from the Pascal VOC dataset.

Baselines

To evaluate our methodology, we compare our method to the following baselines: (1) no-comm, where agents do not communicate; (2) rl-comm, which uses a baseline communication method learned solely through policy loss ; (3) ae-comm, which uses an autoencoder to ground communication in input observations ; (4) VQ-VIB, which uses a variational autoencoder to ground discrete communication in input observations and a mutual information objective to ensure low entropy communication .
We provide an ablation of the loss parameter β in table 1 in the blind traffic junction scenario. When β = 0, we use our compositional message paradigm without our derived loss terms. We find that higher complexity and independence losses increase sample complexity. When β = 1, the model was unable to converge.
However, when there is no regularization loss, the model performs worse (with no guarantees about referential representation). We attribute this to the fact that our independence criteria learns a stronger causal relationship. There are fewer spurious features that may cause an agent to take an incorrect action.
In order to understand the effect of the independent concept representation, we analyze the emergent language's capacity for redundancy. A message token m l is redundant if there exists another token m k that represents the same information. With our methodology, the emergent 'language' converges to the exact number of observations and intents required to solve the task.
With a soft discrete threshold, the independent information loss naturally converges to a discrete number of tokens in the vocabulary. Our β ablation in table 1 yields a bijection between each token in the vocabulary and the possible emergent concepts, i.e., the enumerated observations and intents. Thus for β = 0.1, there is no redundancy.
Sparse Communication In corollary 4.3, we assume that there is no mutual information between tokens. In practice, the loss may only be near-zero. Our empirical results yield independence loss around 1e − 4. In table 1, the size of the messages is automatically compressed to the smallest size to represent the information.
Despite a trivially small amount of mutual information between tokens, our compositional method is able to reduce the message size in bits by 2.3x using our derived regularization, for a total of an 8x reduction in message size over non-compositional methods such as ae-comm. Since the base unit for the token is a 32-bit float, we note that each token in the message may be further compressed.
We observe that each token uses three significant digits, which may further compress tokens to 10 bits each for a total message length of 20 bits.

Communication Utility Results

Due to coordination in MARL, grounding communication in referential features is not enough. Finding the communication utility requires grounding messages in ordinal information. Overall, figure shows that our compositional, contrastive method outperforms all methods focused on solely input-oriented communication grounding.
In the blind traffic junction, our method yields a higher average task success rate and is able to achieve it with a lower sample complexity. Training with the contrastive update tends to spike to high success but not converge, often many episodes before convergence, which leaves area for training improvement.
That is, the contrastive update begins to find aligned latent spaces early in training, but it cannot adapt the methodology quickly enough to converge. The exploratory randomness of most of the early online data prevents exploitation of the high utility f + examples. This leaves further room for improvement for an adaptive contrastive loss term.
Regularization loss convergence After convergence to high task performance, the autoencoder loss increases in order to represent the coordination information. This follows directly from the information bottleneck, where there exists a tradeoff between utility and complexity. However, communication, especially referential communication, should have an overlap between utility and complexity.
Thus, we should seek to make the complexity loss more convex. Our compositional communication complexity loss does not converge before task performance convergence. While the complexity loss tends to spike in the exploratory phase, the normalized value is very small. Interestingly, the method eventually converges as the complexity loss converges below a normal- ized 0.3.
Additionally, the contrastive loss tends to decrease monotonically and converges after the task performance converges, showing a very smooth decrease. The contrastive f − loss decreases during training, which may account for success spikes prior to convergence. The method is able to converge after only a moderate decrease in the f + loss.
This implies empirical evidence that the contrastive loss is an optimal critic for messaging. See figure 3.

Heterogeneous Alignment Through Communication

In order to test the heterogeneous alignment ability of our methodology to learn higher-order concepts from highdimensional data, we analyze the performance on the Pascal VOC game. We compare our methodology against ae-comm to show that concepts should consist of independent information directly from task signal rather than compression to reconstruct inputs.
That is, we show an empirical result on pixel data to verify the premise of the information bottleneck. Our methodology significantly outperforms the observation-grounded ae-comm baseline, as demonstrated by figure 4. The ae-comm methodology, despite using autoencoders to learn observation-grounded communication, performs only slightly better than no-comm.
On the other hand, our methodology is able to outperform both baselines significantly. It is important to note that based on figure 4, our methodology is able to guess more than two of the four labels correctly across the two agents involved, while the baseline methodologies struggle to guess exactly two of thew four labels consistently.
This can be attributed to our framework being able to learn compositional concepts that are much more easily discriminated due to mutual independence.

Social Shadowing

Critics of emergent communication may point to the increased sample complexity due to the dual communication and action policy learning. In the social shadowing scenario, heterogeneous agents can learn to generate a communication policy without learning the action policy of the watched expert agents. To enable social shadowing, the agent will alternate between a batch of traditional MARL (no expert) and (1st-person) shadowing an expert agent performing the task in its trajectory.
The agent only uses the contrastive objective to update its communication policy during shadowing. In figure , the agent that performs social shadowing is able to learn the action policy with almost half the sample complexity required by the online reinforcement learning agent. Our results show that the structured latent space of the emergent communication learns socially benevolent coordination.
This tests our hypothesis that by learning communication to understand the actions of other agents, one can enable lower sample complexity coordination. Thus, it mitigates the issues of solely observing actions.

Discussion

By using our framework to better understand the intent of others, agents can learn to communicate to align policies and coordinate. Any referential-based setup can be performed with a supervised loss, as indicated by the instant satisfaction of referential objectives. Even in the Pascal VOC game, which appears to be a purely referential objective, our results show that intelligent compression is not the only objective of referential communication.
The emergent communication paradigm must enable an easy-to-discriminate space for the game. In multi-agent settings, the harder challenge is to enable coordination through communication. Using contrastive communication as an optimal critic aims to satisfy this, and has shown solid improvements. Since contrastive learning benefits from good examples, this method is even more powerful when there is access to examples from expert agents.
In this setting, the communication may be bootstrapped, since our optimal critic has examples with strong signals from the 'social shadowing' episodes. Additionally, we show that the minimization of our independence objective enables tokens that contain minimal overlapping information with other tokens.
Preventing trivial communication paradigms enables higher performance. Each of these objectives is complementary, so they are not trivially minimized during training, which is a substantial advantage over comparative baselines. Unlike prior work, this enables the benefits of training with reinforcement learning in multi-agent settings.
In addition to lower sample complexity, the mutual information regularization yields additional benefits, such as small messages, which enables the compression aspect of sparse communication. From a qualitative point of view, the independent information also yields discrete emergent concepts, which can be further made human-interpretable by a post-hoc analysis .
This is a step towards white-box machine learning in multi-agent settings. The interpretability of this learned white-box method could be useful in human-agent teaming as indicated by prior work . The work here will enable further results in decision-making from high-dimensional data with emergent concepts.
The social scenarios described are a step towards enabling a zero-shot communication policy. This work will serve as future inspiration for using emergent communication to enable ad-hoc teaming with both agents and humans.

Appendix

A.1. Proofs Proposition 4.1 For the interaction information between all tokens, the following upper bound holds: Proof. Starting with the independent information objective, we want to minimize the interaction information, which defines the conditional mutual information between each token and, Let π i m (m l |h) be a variational approximation of p(m l |h), which is defined by our message encoder network.
Given that each token should provide unique information, we assume independence between m l . Thus, it follows that our compositional message is a vector, m = [m 1 , . . . , m L ], and is jointly Gaussian. Moreover, we can define q( m|h) as a variational approximation to p(m|h) = p(m 1 ; . . . , m L |h).
We can model q with a network layer and define its loss as || m − m|| 2 . Thus, transforming equation 4 into variational form, we have, it follows that q( m|h) log q( m|h)d m ≥ q( m|h) log Thus, we can bound our interaction information, Proposition 4.2 For the mutual information between the composed message and encoded information, the following upper bound holds:
Proof. By definition of mutual information between the composed messages M and the encoded observations H, we have, Substituting q( m|h) for p( m|h), the same KL Divergence identity, and defining a Gaussian approximation z( m) of the marginal distribution p( m), it follows that, In expectation of equation 1, we have,
This implies that, for m = [m 1 , . . . , m L ], there is probabilistic independence between m j , m k , j = k. Thus, expanding, it follows that, where z(m l ) is a standard Gaussian. Proposition 5.1. Utility mutual information is lower bounded by the contrastive NCE-binary objective, Proof. We suppress the reliance on h since this is directly passed through.
By definition of mutual information, we have, Our network model learns π R + (y|m) from rolled-out trajectories, R + , using our policy. The prior of our network state, π R − (y), can be modeled from rolling out a random trajectory, R−. Unfortunately, it is intractable to model π R + (y|m) and π R − (y) directly during iterative learning, but we can sample y + ∼ π R + (y|m) and y − ∼ π R − (y) directly from our network during training.
It has been shown that log p(y|m) provides a bound on mutual information , with the expectation over l p(m l , y l ). However, we need a tractable understanding of the information Y . In the information bottleneck, Y represents the desired outcome. In our setup, y is coordination information that helps create the desired output, such as any action a − .
This implies, y =⇒ a − . Since the transition is known, it follows that a − =⇒ s − f , a random future state. Thus, we have, π This is similar to the proof for lemma A.5, but requires assumptions on messages m from the emergent language. We note that when m is random, the case defaults to lemma A.5. Thus, we assume we have at least input-oriented information in m given sufficiently satisfying equation 2. Given a sufficient emergent language, it follows that y =⇒ a + , where a + is an intention action based on m.
Similarly, since the transition is known, a + =⇒ s + f , a desired goal state along the trajectory. Thus, we have, π R + (y|m) = p(s = s + f |y, m). Recall the following (as shown in ), which we have adapted to our communication objective, Proposition A.3 (rewards → probabilities). The Q-function for the goal-conditioned reward function r g (s t , m t ) = (1 − γ)p(s = s g |y t ) is equivalent to the probability of state s g under the discounted state occupancy measure:
and Lemma A.4. The critic function that optimizes equation 8 is a Q-function for the goal-conditioned reward function up to a multiplicative constant 1 The critic function f (s, m, s f ) = y enc(s f ) represents the similarity between the encoding y = enc(s, m) and the encoding of the future rollout s f .
Given lemmas A.5 A.6 A.8 and proposition A.7, it follows that equation 8 is the NCE-binary (InfoMAX ) objective, Î(M j , Y i ) = log σ(f (s, m, s + f )) + log 1 − σ(f (s, m, s − f )) which lower bounds the mutual information, I(M j , Y i ) ≥ Î(M j , Y i ). The critic function is unbounded, so we constrain it to [0, 1] with the sigmoid function, σ( * ).
We suppress the reliance on h since this is directly passed through. By definition of mutual information, we have, Our network model learns π R + (y|m) from rolled-out trajectories, R + , using our policy. The prior of our network state, π R − (y), can be modeled from rolling out a random trajectory, R−.
Unfortunately, it is intractable to model π R + (y|m) and π R − (y) directly during iterative learning, but we can sample y + ∼ π R + (y|m) and y − ∼ π R − (y) directly from our network during training. It has been shown that log p(y|m) provides a bound on mutual information , with the expectation over l p(m l , y l ).
However, we need a tractable understanding of the information Y . Lemma A.5. π R − (y) = p(s = s − f |y). In the information bottleneck, Y represents the desired outcome. In our setup, y is coordination information that helps create the desired output, such as any action a − . This implies, y =⇒ a − . Since the transition is known, it follows that a − =⇒ s − f , a random future state.
Thus, we have, π R − (y) = p(s = s − f |y). Lemma A.6. π R + (y|m) = p(s = s + f |y, m). This is similar to the proof for lemma A.5, but requires assumptions on messages m from the emergent language. We note that when m is random, the case defaults to lemma A.5. Thus, we assume we have at least input-oriented information in m given sufficiently satisfying equation 2. Given a sufficient emergent language, it follows that y =⇒ a + , where a + is an intention action based on m.
Similarly, since the transition is known, a + =⇒ s + f , a desired goal state along the trajectory. Thus, we have, π R + (y|m) = p(s = s + f |y, m). Recall the following (as shown in ), which we have adapted to our communication objective, Proposition A.7 (rewards → probabilities). The Q-function for the goal-conditioned reward function r g (s t , m t ) = (1 − γ)p(s = s g |y t ) is equivalent to the probability of state s g under the discounted state occupancy measure:
and Lemma A.8. The critic function that optimizes equation 8 is a Q-function for the goal-conditioned reward function up to a multiplicative constant 1 p(s f ) : exp(f * (s, m, s f ) = 1 p(s f ) Q π s f (s, m). The critic function f (s, m, s f ) = y enc(s f ) represents the similarity between the encoding y = enc(s, m) and the encoding of the future rollout s f .
Given lemmas A.5 A.6 A.8 and proposition A.7, it follows that equation 8 is the NCE-binary (InfoMAX ) objective, which lower bounds the mutual information, I(M j , Y i ) ≥ Î(M j , Y i ). The critic function is unbounded, so we constrain it to [0, 1] with the sigmoid function, σ( * ).

Answer: a supervised method utilizing deep neural networks


Input: How is the function beta(r) determined in the derivation?

Context:
\section{Introduction}
The Schwarzschild solution plays a key role in teaching about general relativity: It describes the simplest version of a black hole. By Birkhoff's theorem, it more generally describes the gravitational field around any spherical mass distribution, such as the Sun in our own Solar system. As one of two particularly simple, yet physically relevant examples of a non-trivial metric (the other being the FLRW spacetime of an expanding universe), it is particularly well-suited for teaching about general techniques of ``reading'' and interpreting a spacetime metric.

Consider undergraduate courses where students are introduced to selected concepts and results from general relativity without exposing them to the full mathematical formalism. Such courses have the advantage of introducing students to one of the two great fundamental theories of 20th century physics early on (the other being quantum mechanics); they also profit from subject matter that meets with considerable interest from students.\cite{Hartle2006} Using the terminology of Christensen and Moore,\cite{Christensen2012} in the ``calculus only'' approach pioneered by Taylor and Wheeler,\cite{Taylor2001,Taylor2018} spacetime metrics are not derived, but taken as given, and the focus is on learning how to interpret a given spacetime metric. Similar presentations can be found in the first part of the ``physics first'' approach exemplified by Hartle's text book,\cite{Hartle2003} where the concepts of the metric and of geodesics are introduced early on, and their physical consequences explored, while the mathematics necessary for the Einstein equations is only introduced at a later stage.

Whenever the approach involves an exploration of simple metrics such as the Schwarzschild solution, but stops short of the formalism required for the full tensorial form of Einstein's equations, access to a simple derivation of the Schwarzschild solution that does not make use of the advanced formalism can be a considerable advantage.

Simplified derivations of the Schwarzschild solution have a long tradition within general relativity education,\cite{Schiff1960,Harwit1973} although specific simplifications have met with criticism.\cite{Rindler1968} This article presents a derivation which requires no deeper knowledge of the formalism of differential geometry beyond an understanding of how to interpret a given spacetime metric $\mathrm{d} s^2$. The derivation avoids the criticism levelled at attempts to derive the Schwarzschild solution from the Einstein equivalence principle in combination with a Newtonian limit,\cite{Gruber1988} relying as it does on a simplified version of the vacuum Einstein equation.

More specifically, I combine the restrictions imposed by the symmetry with the simple form of Einstein's equations formulated by Baez and Bunn.\cite{BaezBunn2005} That same strategy was followed by Kassner in 2017,\cite{Kassner2017} but in this text, I use the ``infalling coordinates'' that are commonly associated with the Gullstrand-Painlev\'e form of the Schwarzschild metric,\cite{Martel2001,Visser2005,HamiltonLisle2008} not the more common Schwarzschild coordinates. That choice simplifies the argument even further. In the end, what is required is no more than the solution of an ordinary differential equation for a single function, which yields to standard methods, to obtain the desired result.

\section{Coordinates adapted to spherical symmetry and staticity}
\label{SymmetriesCoordinates}

Assume that the spacetime we are interested in is spherically symmetric and static. In general relativity, a symmetry amounts to the possibility of being able to choose coordinates that are adapted to the symmetry, at least within a restricted sub-region of the spacetime in question. That the spacetime is static is taken to mean that we can introduce a (non-unique) time coordinate ${t}$ so that our description of spacetime geometry does not depend explicitly on ${t}$, and that space and time are completely separate --- in the coordinates adapted to the symmetry, there are no ``mixed terms'' involving $\mathrm{d} {t}$ times the differential of a space coordinate in the metric. If we use ${t}$ to slice our spacetime into three-dimensional hyperplanes, each corresponding to ``space at time ${t}$,'' then each of those 3-spaces has the same spatial geometry. A mixed term would indicate that those slices of space would need to be shifted relative to another in order to identify corresponding points. The mixed term's absence indicates that in adapted coordinates, there is no need for such an extra shift. In those coordinates, we can talk about the 3-spaces as just ``space,'' without the need for specifying which of the slices we are referring to.

In the case of spherical symmetry, we can introduce spherical coordinates that are adapted to the symmetry: a radial coordinate $r$ and the usual angular coordinates $\vartheta,\varphi$, so that the spherical shell at constant $r$ has the total area $4\pi r^2$. In consequence, the part of our metric involving $\mathrm{d}\vartheta$ and $\mathrm{d}\varphi$ will have the standard form
\begin{equation}
r^2(\mathrm{d}\vartheta^2+\sin^2\theta\mathrm{d}\varphi^2) \equiv r^2\mathrm{d}\Omega^2,
\end{equation}
where the right-hand side defines $\mathrm{d}\Omega^2$, the infinitesimal solid angle corresponding to each particular combination of $\mathrm{d}\vartheta$ and $\mathrm{d}\varphi$.

The radial coordinate slices space into spherical shells, each corresponding to a particular value $r=const.$ The rotations around the origin, which are the symmetry transformations of spherical symmetry, map each of those spherical shells onto itself, and they leave all physical quantities that do not explicitly depend on $\vartheta$ or $\varphi$ invariant.

In what follows, we will use the basic structures introduced in this way --- the slices of simultaneous ${t}$, the radial directions within each slice, the angular coordinates spanning the symmetry--adapted spherical shells of area $4\pi r^2$ --- as auxiliary structures for introducing spacetime coordinates. For now, let us write down the shape that our metric has by simple virtue of the spherical symmetry, the requirement that the spacetime be static, and the adapted coordinates, namely
\begin{equation}
\mathrm{d} s^2 = -c^2F(r) \mathrm{d} {t}^2 + G(r) \mathrm{d} r^2 + r^2\:\mathrm{d}\Omega^2.
\label{StaticForm}
\end{equation}
Students familiar with  ``reading'' a spacetime metric will immediately recognize the sign difference between the parts describing space and describing time that is characteristic for spacetime, and the speed of light $c$ that gives us the correct physical dimensions. That there is no explicit dependence on $\varphi$ and $\vartheta$ in the remaining functions $F$ and $G$ is a direct consequence of spherical symmetry. That the factor in front of $\mathrm{d}\Omega^2$ is $r^2$ is a consequence of our coordinate choice, with spherical angular coordinates so that the area of a spherical surface of constant radius $r$ is $4\pi r^2$. That there is no explicit dependence on ${t}$ is one consequence of the spacetime being static; the absence of the mixed term $\mathrm{d} {t}\cdot \mathrm{d} r$ is another. We are left with two unknown functions $F(r)$ and $G(r)$. In the following, let us call ${t}$ and $r$ the {\em static coordinates}.

Note that, since $G(r)$ is as yet undefined, we have not yet chosen a specific physical meaning for the length measurements associated with our $r$ coordinate. But because of the $\mathrm{d}\Omega^2$ part, it is clear that whatever choice we make, the locally orthogonal lengths $r\cdot\mathrm{d}\vartheta$ and $r\cdot\sin\vartheta\cdot\mathrm{d}\varphi$ will have the same physical interpretation as for the length measurement corresponding to $\mathrm{d} r$.

\section{Infalling observer coordinates}
\label{Sec:InfallingObservers}

Now that we know what the radial directions are, at each moment of time ${t}$, we follow Visser\cite{Visser2005} as well as Hamilton and Lisle\cite{HamiltonLisle2008} in defining a family of radially infalling observers.  Observers in that family are in free fall along the radial direction, starting out at rest at infinity: In mapping each observer's radial progression in terms of the static coordinate time ${t}$, we adjust initial conditions, specifically: the choice of initial speed at some fixed time ${t}$, in just the right way that the radial coordinate speed goes to zero for each observer in the same way as $r\to\infty.$

It is true that talking about ``infalling'' observers already reflects our expectation that our solution should describe the spacetime of a spherically symmetric mass.  As we know from the Newtonian limit, such a mass attracts test particles in its vicinity. It should be noted, though, that all our calculations would also be compatible with the limit of no mass being present. In that case, ``infalling'' would be a misnomer, as our family of observers would merely hover in empty space at unchanging positions in $r$.

We can imagine infinitesimal local coordinate systems associated with our observers --- think of the observer mapping out space and time by defining three orthogonal axes, and by measuring time with a co-moving clock. We assume all such little coordinate systems to be non-rotating --- otherwise, we would break spherical symmetry, since rotation would locally pick out a plane of rotation that is distinguishable from the other planes. The radial direction is a natural choice for the first space axis of those little free-falling systems. The other directions, we take to point to observers falling side by side with our coordinate-defining observer --- and to remain pointed at a specific such other observer, once the choice of direction is made.

We assume our infalling observers' clocks to be synchronised at some fixed radius value $r$. By spherical symmetry, those clocks should then be synchronised at {\em all} values of $r$. Anything else would indicate direction-dependent differences for the infalling observers and their clocks, after all. Hence, at any given static time ${t}$, all the infalling observers who are at radius value $r$ show the same proper time $T$ on the ideal clocks travelling along with them.

Once our definition is complete, our static, spherically symmetric spacetime is filled with infalling observers from that family: Whenever we consider an event $\cal E$, there will be an observer from that family passing by at that time, at that location.

Now, consider the coordinate speed of those infalling observers. If we position ourselves at some constant radius value $r$ and watch the falling observers fly by, then we can express both their proper time rate and their coordinate speed in the $r$ direction in terms of $r$ and ${t}$. We can combine the two pieces of information to obtain the rate of change in radial position $r$ with proper time $T$ for those infalling observers. But since the initial conditions for those observers are the same, and since our spacetime is, by assumption, static, the resulting function can only depend on $r$, and not explicitly on ${t}$. Let us rescale that function with the speed of light to make it dimensionless, give it an overall minus sign to make it positive for infalling particles, and call it $\beta(r)$,
\begin{equation}
\beta(r)\equiv -\frac{1}{c}\frac{\mathrm{d} r}{\mathrm{d} T}(r).
Recall from section that we also still have the freedom to decide on the physical meaning of the function. We make the choice of making $\mathrm{d} r$ the physical length measured by one of our infalling observers at the relevant location in spacetime, at constant time $T$. Via our angular coordinates, that implies that length measurements orthogonal to the radial direction, $r\cdot\mathrm{d}\vartheta$ and $r\cdot\sin\vartheta\:\mathrm{d}\varphi$ inherit the same physical interpretation.

As a next step, we transform our metric (\ref{StaticForm}) from the static form into the form appropriate for our coordinate choice $r$ and $T$. We do so by writing the static time coordinate as a function ${t}(T,r)$ in terms of infalling observer time and radius value. In consequence,
\begin{equation}
\mathrm{d} {t} = \frac{\partial{t}}{\partial T}\cdot\mathrm{d} T+ \frac{\partial {t}}{\partial r}\cdot\mathrm{d} r,
\end{equation}
and our new metric now has the form
\begin{align}
 \mathrm{d} s^2 = {} & -c^2 F(r)\left(\frac{\partial t}{\partial T}\right)^2\mathrm{d} T^2 \nonumber \\[0.2em]
 & -2c^2F(r)\left(\frac{\partial t}{\partial T}\right)\left(\frac{\partial t}{\partial r}\right)\mathrm{d} T\:\mathrm{d} r \nonumber \\[0.2em]
 & +\left[G(r)-c^2F(r)\left(\frac{\partial t}{\partial r}\right)^2\right]\mathrm{d} r^2+r^2\:\mathrm{d}\Omega^2.
 \end{align}
At face value, this looks like we are moving the wrong way, away from simplification, since we now have more functions, and they depend on two variables instead of one.

But in fact, this new formulation paves the way for an even simpler form of the metric. Consider a specific event, which happens at given radius value $r$. In a small region around that event, we will introduce a new coordinate $\bar{r}$ to parametrize the radial direction. We want this coordinate to be co-moving with our infalling observers at $r$; each such observer then has a position $\bar{r}=const.$ that does not change over time.

Key to our next step is that we {\em know} the metric for the local length and time measurements made by any one of our free-falling observers. By Einstein's equivalence principle, the metric is that of special relativity. Locally, namely whenever tidal effects can be neglected, spacetime geometry for any non-rotating observer in free fall is indistinguishable from Minkowski spacetime as described by a local inertial system.

Since we have chosen both the time coordinate $T$ and the physical meaning of the radial coordinate $r$ so as to conform with the measurements of the local infalling observer, the transformation between $\bar{r}$ and $r$ is particularly simple: It has the form of a Galilei transformation
\begin{equation}
\mathrm{d}\bar{r}= \mathrm{d} r + \beta(r)c\:\mathrm{d} T.
\label{barRshift}
\end{equation}
In that way, as it should be by definition, radial coordinate differences at constant $T$ are the same in both systems, while for an observer at constant $\bar{r},$ with $\mathrm{d} \bar{r}=0$, the relation between $\mathrm{d} r$ and $\mathrm{d} T$ is consistent with the definition of the function $\beta(r)$ in (\ref{betaDefinition}).

Are you surprised that this is not a Lorentz transformation, as one might expect from special relativity? Don't be. We are not transforming from one local inertial coordinate system to another. The $T$ is already the time coordinate of the infalling observers, so both coordinate systems have the same definition of simultaneity, and time dilation plays no role in this particular transformation. Also, we have chosen $r$ intervals to correspond to length measurements of the infalling observers, so there is no Lorentz contraction, either. It is the consequence of these special choices that gives the relation (\ref{barRshift}) its simple form.

Last but not least, when we analyse specifically an infinitesimal neighbourhood of the point $r,\vartheta,\varphi$, let us make the choice that directly at our point of interest, we make $\bar{r}$ coincide with $r$. Since before, we had only fixed the differential $\mathrm{d} \bar{r}$, we do have the remaining freedom of choosing a constant offset for $\bar{r}$ that yields the desired result.

By Einstein's equivalence principle, the metric in terms of the locally co-moving coordinates $T,\bar{r},\vartheta,\varphi$ is the spherical-coordinate version of the Minkowski metric,
\begin{equation}
\mathrm{d} s^2 = -c^2\mathrm{d} T^2 + \mathrm{d}\bar{r}^2 + \bar{r}^2\mathrm{d}\Omega.
\end{equation}
This version can, of course, be obtained by taking the more familiar Cartesian-coordinate version
\begin{equation}
\mathrm{d} s^2=-c^2\mathrm{d} T^2 + \mathrm{d} X^2 + \mathrm{d} Y^2 + \mathrm{d} Z^2,
\label{CartesianMinkowski}
\end{equation}
applying the definition of Cartesian coordinates $X,Y,Z$ in terms of spherical coordinates $\bar{r},\vartheta,\varphi$
\begin{equation}
x= \bar{r}\:\sin\vartheta\:\cos\varphi, \;\;
y= \bar{r}\:\sin\vartheta\:\sin\varphi, \;\;
z= \bar{r}\:\cos\vartheta,
\end{equation}
to express $\mathrm{d} X, \mathrm{d} Y, \mathrm{d} Z$ in terms of $\mathrm{d} \bar{r}, \mathrm{d}\vartheta, \mathrm{d}\varphi$, and substitute the result into (\ref{CartesianMinkowski}).

By noting that we have chosen $\bar{r}$ so that, at the specific spacetime event where we are evaluating the metric, $\bar{r}=r$, while, for small radial coordinate shifts around that location, we have the relation (\ref{barRshift}), we can now write down the same metric in the coordinates $T, r, \vartheta,\varphi$, namely as
\begin{equation}
\mathrm{d} s^2 = -c^2\left[
1-\beta(r)^2
\right] \mathrm{d} T^2+2c\beta(r)\mathrm{d} r\:\mathrm{d} T
+\mathrm{d} r^2+r^2\mathrm{d}\Omega^2.
\label{preMetric}
\end{equation}
Since we can repeat that local procedure at any event in our spacetime, this result is our general form of the metric, for all values of $r$. This, then is the promised simplification: By exploiting the symmetries of our solutions as well as the properties of infalling observers, we have reduced our metric to a simple form with no more than one unknown function of one variable, namely $\beta(r)$.

So far, what I have presented is no more than a long-form version of the initial steps of the derivation given by Visser in his heuristic derivation of the Schwarzschild metric.\cite{Visser2005} In the next section, we will deviate from Visser's derivation.

\section{$\beta(r)$ from tidal deformations}
\label{TidalSection}

In the previous section, we had exploited symmetries and Einstein's equivalence principle. In order to determine $\beta(r)$, we need to bring in additional information, namely the Einstein equations, which link the matter content with the geometry of spacetime. For our solution, we only aim to describe the spacetime metric outside whatever spherically-symmetric matter distribution resides in (or around) the center of our spherical symmetry. That amounts to applying the {\em vacuum Einstein equations}.

More specifically, we use a particularly simple and intuitive form of the vacuum Einstein equations, which can be found in a seminal article by Baez and Bunn:\cite{BaezBunn2005} Consider a locally flat free-fall system around a specific event $\cal E$, with a time coordinate $\tau$, local proper time, where the event we are studying corresponds to $\tau=0$. In that system, describe a small sphere of freely floating test particles, which we shall call a {\em test ball}. The particles need to be at rest relative to each other at $\tau=0$. Let the volume of the test ball be $V(\tau)$. Then the vacuum version of Einstein's equations states that
\begin{equation}
\left.\frac{\mathrm{d}^2 V}{\mathrm{d}\tau^2}\right|_{\tau=0} = 0.
In words: If there is no matter or energy inside, the volume of such a test ball remains constant according to the theory of relativity.

If you are familiar with Wheeler's brief summary of Einstein's equations, ``spacetime grips mass, telling it how to move'' and ``mass grips spacetime, telling it how to curve'',\cite{Wheeler1990} you will immediately recognise that this is a specific way for the structure of spacetime telling the test ball particles how to move. The calculation later in this section provides the second part: It will amount to using (\ref{EinsteinVacuum}) to determine the structure of spacetime, namely the still missing function $\beta(r)$, and that is the way for mass, in this case: for the absence of mass, to tell spacetime how to curve.

Note that equation (\ref{EinsteinVacuum}) also holds true in Newtonian gravity. So in a way, this version of Einstein's equation can be seen as a second-order extension of the usual Einstein equivalence principle: Ordinarily, the equivalence principle is a statement about physics in the absence of tidal forces. Equation (\ref{EinsteinVacuum}) adds to this that the lowest-order correction for tidal forces in a freely falling reference frame is that specified by Newtonian gravity. This makes sense, since by going into a free-fall frame, and restricting our attention to a small spacetime region, we have automatically created a weak-gravity situation. In such a situation, tidal corrections are approximately the same as those described by Newton. This argument can serve as a heuristic justification of (\ref{EinsteinVacuum}).

In 2017, Kassner made use of the Baez-Bunn form of Einstein's vacuum equation to derive the Schwarzschild solution, starting from what we have encountered as the static form of the metric (\ref{StaticForm}).\cite{Kassner2017} We follow the same general recipe, but using the infalling coordinates introduced in section \ref{Sec:InfallingObservers}, which makes our derivation even simpler.

Consider five test particles in a small region of space. Let the motion of each be the same as for the local representative from our coordinate-defining family of infalling observers. We take the central particle $C$ to be at radial coordinate value $r=R$ at the time of the snapshot shown in Fig.~\ref{TestParticlesOutside}. The other four are offset relative to the central particle: As described in the local inertial system that is co-moving with the central particle, one of the particles is shifted by $\Delta l$ upwards in the radial direction, another downward, while two of the particles are offset orthogonally by the same distance.
\begin{figure}[htbp]
\begin{center}
\includegraphics[width=0.5\linewidth]{01-free-fall-particles.pdf}
\caption{Five test particles in our spherically-symmetric spacetime}
\label{TestParticlesOutside}
\end{center}
\end{figure}
The $\Delta l$ is meant to be infinitesimally small, so while Fig.~\ref{TestParticlesOutside} is of course showing a rather large $\Delta l$ so as to display the geometry of the situation more clearly, we will in the following only keep terms linear in $\Delta l$.

Consider a generic particle, which moves as if it were part of our coordinate-defining family of infalling observers, and which at the time $T_0$ is at $r=r_0$. By a Taylor expansion, that particle's subsequent movement is given by
\begin{equation}
r(T) = r_0 + \frac{\mathrm{d} r}{\mathrm{d} T}(T_0) \cdot \Delta T +\frac12 \frac{\mathrm{d}^2 r}{\mathrm{d} T^2}(T_0) \cdot \Delta T^2
\label{TaylorREvo}
\end{equation}
where $\Delta T\equiv T-T_0$. We know from (\ref{betaDefinition}) that the derivative in the linear term can be expressed in terms of $\beta(r)$; by the same token,
\begin{equation}
\frac{\mathrm{d}^2 r}{\mathrm{d} T^2} = -c\frac{\mathrm{d}\beta}{\mathrm{d} T}=-c\beta' \frac{\mathrm{d} r}{\mathrm{d} T} = c^2\beta\cdot\beta',
\end{equation}
where the prime denotes differentiation of $\beta$ with respect to its argument. Since, in the following, the product of the function and its first derivative will occur quite often, let us introduce the abbreviation
\label{BigBDefinition}
\end{equation}
With these results, can rewrite the Taylor expansion (\ref{TaylorREvo}) as
\begin{equation}
r(T) = r_0 -c\beta(r_0)\cdot\Delta T + \frac12 c^2B(r_0)\cdot\Delta T^2.
\label{RadialOrbitTime}
\end{equation}
In order to find $r_C(T)$ for our central particle, we simply insert $r_0=R$ into that expression. If, on the other hand, we want to write down the time evolution for particles $U$ and $D$, let us denote it by $r_{U,D}(T)$, we need to evaluate the expression (\ref{RadialOrbitTime}) at the initial location $r_0=R\pm\Delta l$. Since $\Delta l$ is small, we can make a Taylor expansion of $\beta(r)$ and its derivative around $r=R$, and neglect everything beyond the terms linear in $\Delta l$. The result is
\begin{multline}
r_{U,D}(T)=R \pm\Delta l-c\left[
\beta(R)\pm\beta'(R)\Delta l
\right]\Delta T \\[0.2em]
+\frac{c^2}{2}\big[
B(R)\pm B'(R)\Delta l
\big]\Delta T^2
\end{multline}
In consequence, the distance between the upper and lower particle, $d_{\parallel}(T)\equiv r_U(T)-r_D(T),$ changes over time as
\begin{equation}
d_{\parallel}(T) =  2\Delta l\left[
1-c\beta'(R)\Delta T+\frac12c^2 B'(R)\Delta T^2
\right].
\label{dParallel}
\end{equation}
Next, let us look at how the distance between the particles $L$ and $R$ changes over time. The initial radial coordinate value for each of the particles is
\begin{equation}
r(T_0) = \sqrt{R^2+\Delta l^2}=R\left[1+\frac12\left(\frac{\Delta l}{R}\right)^2\right]\approx R,
\end{equation}
that is, equal to $R,$ as long as we neglect any terms that are higher than linear in $\Delta l$. In consequence, $r_{L,R}(t)$ is the same function as for our central particle, given by eq.~(\ref{RadialOrbitTime}) with $r_0=R$. The transversal (in Fig.~\ref{TestParticlesOutside}: horizontal) distance $d_{\perp}(T)$ between the particles $L$ and $R$ changes in proportion to the radius value,
\begin{align}
d_{\perp}(T) &= 2\Delta l\cdot\frac{r_{L}(T)}{R} \nonumber \\
                 &=2\Delta \left[1-\frac{c\beta(R)}{R}\Delta T+\frac{c^2}{2}\frac{B(R)}{R}\Delta T^2\right].
                \label{dPerp}
\end{align}
With these preparations, consider the vacuum Einstein equation (\ref{EinsteinVacuum}) for the volume of a test ball. Initially, our particles $C, U, D, L, R$ define a circle, which is deformed to an ellipse. By demanding rotational symmetry around the radial direction, we can construct the associated ellipsoid, which is initially a spherical surface. That ellipsoid has one axis in radial direction, whose length is $d_{\parallel}(T)$, and two axes that are transversal and each have the length  $d_{\perp}(t)$. But that ellipsoid is not quite yet the test ball we need. After all, the particles of the test ball need to be at rest initially, at time $T_0$, in the co-moving system defined by the central particle $C$. Our defining particles are not, as the terms linear in $\Delta T$ in both (\ref{dParallel}) and (\ref{dPerp}) show, where the coefficients of $\Delta T$ correspond to the particles' initial velocities.

In order to define our test ball, we need to consider particles at the same location, undergoing the same acceleration, but which are initially at rest relative to the central particle $C$.

We could go back to the drawing board, back to Fig.~\ref{TestParticlesOutside}, make a more general Ansatz that includes initial velocities which measure the divergence of the motion of our test ball particles from that of the infalling-observer particles, and repeat our calculation while including those additional velocity terms. But there is a short-cut. The only consequence of those additional velocity terms will be to change the terms linear in $\Delta T$ in equations (\ref{dParallel}) and (\ref{dPerp}). And we already know the end result: We will choose the additional terms so as to cancel the terms linear in $\Delta T$ in the current versions of (\ref{dParallel}) and (\ref{dPerp}). But by that reasoning, we can skip the explicit steps in between, and write down the final result right away. The time evolution of the radial-direction diameter of our test ball, let us call it $L_{\parallel}(T)$, must be the same as $d_{\parallel}(T)$, but without the term linear in $\Delta T$. Likewise, the time evolution $L_{\perp}(T)$ of the two transversal diameters must be equal to $d_{\perp}(T)$, but again without the term linear in $\Delta T$. The result is
\begin{align}
L_{\parallel}(T)  &=  2\Delta l \left[1+\frac12c^2B'(R)\Delta T^2\right] \\
L_{\perp}(T) &= 2\Delta l \left[1+\frac{c^2}{2}\frac{B(R)}{R}\Delta T^2\right].
Thus, our test ball volume is determined by the equation This is readily solved by the standard method of separation of variables: We can rewrite the condition as  \;\; \Rightarrow \;\; \ln(Br^2) = C',
\end{equation}
with a constant $C'$, which upon taking the exponential gives us
\begin{equation}
Br^2= C,
\label{BSolution}
\end{equation}
with a constant $C$. Note that the constant $C$ can be negative --- there is no reason the constant $C'$ needs to be real; only our eventual function $B(r)$ needs to be that, and it is clear that (\ref{BSolution}) satisfies the differential equation
(\ref{VolumeConditionR}) for any constant $C$, positive, zero, or negative. By the definition, the solution corresponds to the differential equation
Both sides are readily integrated up; we can solve the result for the function and obtain That brings us to the last step: The requirement that, for large values of $r$, the description provided by our solution should correspond to the results from Newtonian gravity. First of all, we note that our initial condition for the infalling observers, which had those observers start out at zero speed at infinity, means that we must choose the integration constant accordingly. Then, as we would expect, $\beta(r)$ for large values of $r$ becomes very small, corresponding to small speeds. But at slow speeds, time and length intervals as measured by the infalling observer will become arbitrarily close to time and length intervals as measured by an observer at rest in our static coordinate system at constant $r$, using the static time coordinate ${t}$. As is usual, we identify these coordinates with those of an approximately Newtonian description. In that description, the radial velocity is
\begin{equation}
v(r) = \sqrt{\frac{2GM}{r}},
\end{equation}
which follows directly from energy conservation for the sum of each observer's kinetic and Newtonian-gravitational potential energy. This fixes the remaining integration constant as
\end{equation}
Inserting this result in (\ref{preMetric}), we obtain the metric
\begin{equation}
\mathrm{d} s^2 = -c^2\left[
1-\frac{2GM}{rc^2}
\right]\mathrm{d} T^2+2\sqrt{\frac{2GM}{r}}\mathrm{d} r\:\mathrm{d} T+\mathrm{d} r^2+r^2\mathrm{d}\Omega^2.
\label{GPMetric}
\end{equation}
This is known as the Gullstrand-Painlev\'e version of the Schwarzschild metric.\cite{Martel2001,Visser2005,HamiltonLisle2008} A  last transformation step brings us back to the traditional Schwarzschild form. Recall our discussion in sec.~\ref{SymmetriesCoordinates}, leading up to the explicitly static form (\ref{StaticForm}) of the metric? The main difference between our current form and the static version is the mixed term containing $\mathrm{d} r\:\mathrm{d} T$ in (\ref{GPMetric}). Everything else already has the required shape. Inserting the Ansatz
\begin{equation}
\mathrm{d} T = \mathrm{d} t + \xi(r) \mathrm{d} r
\end{equation}
into the metric (\ref{GPMetric}), it is straightforward to see that the mixed term vanishes iff our transformation is
\begin{equation}
\mathrm{d} T = \mathrm{d} t +\frac{\sqrt{2GM/r}}{c^2\left(1-\frac{2GM}{rc^2}\right)}\mathrm{d} r.
\label{TtTrafo}
\end{equation}
Substitute this into (\ref{GPMetric}), and the result is the familiar form of the Schwarzschild metric in Schwarzschild's original coordinates $t,r,\vartheta,\varphi$,
\begin{equation}
\mathrm{d} s^2 = -c^2\left(1-\frac{2GM}{c^2 r}
\right)\mathrm{d} t^2 + \frac{\mathrm{d} r^2}{\left(1-\frac{2GM}{c^2 r}
\right)} + r^2\mathrm{d}\Omega^2.
\end{equation}

\section{Conclusion}
Using coordinates adapted to the symmetries, we were able to write down the spherically symmetric, static spacetime metric. On this basis, and using the family of infalling observers that is characteristic for the Gullstrand-Painlev\'e solution, we wrote down the metric in the form (\ref{preMetric}), with a single unknown function $\beta(r)$. From the simplified form (\ref{EinsteinVacuum}) of the vacuum Einstein equations, as applied to a test ball in free fall alongside one of our family of observers, we were able to determine $\beta(r)$, up to two integration constants. By using the relativity equation, we escape the restrictions imposed on simplified derivations by Gruber et al. Our derivation does not require knowledge of advanced mathematical concepts beyond the ability to properly interpret a given metric line element $\mathrm{d} s^2$. Even our analysis of tidal effects proceeds via a simple second-order Taylor expansion, leading to differential equations for $\beta(r)$ that are readily solved using two applications of the method of separation of variables.

What is new about the derivation presented here is the combination of the Baez-Bunn equations with the infalling coordinates typical for the Gullstrand-Painlev\'e form of the metric --- this combination is what, in the end, makes our derivation particularly simple. In turn, this simplicity is what should make the derivation particularly useful in the context of teaching general relativity in an undergraduate setting.

The derivation proceeds close to the physics, and gives ample opportunity to discuss interesting properties of Einstein's theory of gravity. Students who are presented with this derivation, either as a demonstration or as a (guided) exercise, will come to understand the way that symmetries determine the form of a metric, the deductions that can be made from Einstein's equivalence principle, and last but not least that we need to go beyond the equivalence principle, and consider tidal forces, to completely define our solution.

\section*{Acknowledgements}

I would like to thank Thomas M\"uller for helpful comments on an earlier version of this text.

Answer: Using the theory of relativity and the Schwarzschild solution