Nota para antes de leer este documento:

 1. El paquete dst contiene toda la implementación de las ideas aquí expuestas. El notebook 2. Implementación incluye implementaciones para distintas configuraciones. En el presente documento se expondrá código de manera ilustrativa, sin embargo, el paquete es el encargado de realizar los procedimientos aquí expuestos. Para ver la implementación puede dirigirse al código fuente, al enlace a colab o al notebook de experimentos.

 2. La implementación que se realizo esta basada en el documento   Image Style Transfer Using Convolutional Neural Networks, en el blog de tensorflow y en el blog de Marko Jerkic

 3. La implementación está escrita en Python3 usando Tensorflow y Keras.

 Aprendizaje profundo para la transferencia de estilo

  Universidad de Antioquia
 Angelower Santana Velasquez
 Martin Elias Quintero  Osorio

Motivación

El Deep Learning (Aprendizaje Profundo) es un campo específico del aprendizaje automático (Machine Learning)y en consecuencia de la inteligencia artificial(AI), el cual, ha demostrado un avance exponencial los últimos años. El Deep Learning ha sorprendido por sus increíbles resultados en múltiples áreas para solucionar distintos tipos de problema. Específicamente el deep learning gira en torno al estudio de modelos basados en redes neuronales artificiales, sus funciones de pérdida, la capacidad de distintos tipos de optimización, estrategias para evitar el sobreajuste, el diseño de arquitecturas de redes neuronales enfocadas a cumplir objetivos desde distintos enfoques de aprendizaje, entre muchas otras estrategias que son de gran importancia dentro del Machine Learning. Podemos encontrar varias ramas del Deep Learning como el procesamiento de lenguaje natural (NLP) y la visión por computadora (Computer Vision -CV), siendo esta última materia de estudio desde hace más de una década teniendo avances sorprendentes. Hoy día podemos ver el aprovechamiento de la visión computacional en aplicaciones implementadas en aeropuertos, automóviles, en la industria y, no muy alejado, en nuestros teléfonos inteligentes. Dentro del Deep Learning existe un tipo de red neuronal capaz de emular, según algunas personas, el comportamiento biológico que ocurre en los seres humanos en el proceso de "visión", las redes neuronales convolucionales(CNN), llamadas así por realizar operaciones de convolución dentro del proceso de entrenamiento, operador bastante utilizado en el procesamiento de señales. Y son este tipo de redes neuronales casi la opción por defecto para tratar problemas de visión computacional.

La capacidad de cómputo que disponemos actualmente ha permitido que se realicen múltiples experimentos y se desarrollen ideas sorprendentes para el ojo común. Sistemas que controlan automóviles, aplicaciones que determinan cierto tipo de enfermedades en plantas, filtros y aplicación de diferentes "máscaras" en imágenes en tiempo real (por ejemplo instagram y snapchat), detección de enfermedades bajo la deteccion de patrones (por ejemplo la retinopatía diabética) y la clasificación de objetos son ejemplos de la capacidad que pueden lograr sistemas basados en redes neuronales convolucionales.

 Una cita de uno de los artistas más importantes del siglo XX, Pablo Picasso, dice : “La pintura es más fuerte que yo, siempre consigue que haga lo que ella quiere”. El presente informe explora una aplicación bastante ingeniosa para generar "arte" a partir de técnicas de Deep Learning con redes neuronales convolucionales. Que de forma similar a la frase de Picasso, será nuestro modelo el encargado de generar una imagen a su antojo a partir de dos fuentes de datos: Una imagen de contenido y otra de estilo. Ahora, ¿podrías imaginar como Vincent van Gogh había pintado a Medellín?

imagen 1
The starry night in Medellín
 Imagen contenido: Fotografia del centro de Medellín
 Imagen estilo: The Starry Night - Vincent van Gogh

Contenido y estilo

El cubismo fue un movimiento artístico donde se quería representar elementos de la cotidianidad en composiciones de formas geométricas bien definidas. Es decir, tomar un elemento, capturar su esencia y plasmarla en cubos, triangulos y rectangulos. De esa manera definimos el contenido de una imagen como la esencia que tiene dejando a un lado elementos como el color y la textura. Por otro lado, cuando hablemos de estilo nos referimos a la forma, los colores, sombras y otros matices que no sean la esencia. De hecho, la transferencia de estilo busca tomar la esencia y plasmar en ella la forma y colores de otra imagen. Por ejemplo, en la imagen 1 , el contenido es la fotografía de Medellín imagen 2 lado derecho superior y el estilo es la famosa pintura de Vicent Van Gogh : The starry night. imagen 2 lado derecho inferior

imagen 2
The starry night in Medellín (Contenido, estilo y resultado)
 Lado izquierdo: Resultado de Deep Transfer Style
 Lado Derecho parte superior: Contenido
Lado Derecho parte inferior: Estilo

Deep Transfer Style

La idea detrás de la técnica de transferencia de estilo es la representación interna que producen las redes neuronales convolucionales una vez entrenadas. Dichas redes están conformadas por capas que tienen por objetivo propósitos específicos: Algunas sirven como mapa de activación que nos indica que tan sensible es una imagen a un patrón o filtro. En otras palabras, si aplicamos un filtro que detecte bordes y formas cuadradas a una imagen de un televisor probablemente la forma del televisor se activará con dicho filtros. Dentro del proceso de entrenamiento la red aprende a detectar estas formas que inician con figuras muy básicas hasta evolucionar a detalles y figuras compuestas, aunque no por esto se llaman redes convolucionadas.

En la imagen 3 se puede observar una arquitectura de red bastante famosa, VGG16.

imagen 3
How convolutional neural networks see the world
 Fuente: Blog

VGG16 cuenta con 5 bloques internos compuestos por capas de convolución y de pooling (este tipo de capas reduce el volumen de los mapas de características, en la imagen 3 en rojo).Cada bloque cuenta con cierta cantidad de capas de convolución, una notación para determinar las capas es especificar el número de bloque y la posición que esta tiene dentro del arreglo.Por ejemplo la capa convolucional dos del bloque uno se representa como :Conv2_1, mientras la capa convolucional tres en el bloque cuatro sería: Conv4_3.
Como se mencionó anteriormente en el proceso de entrenamiento la red aprende múltiples filtros, la siguiente pregunta que debemos realizar es : ¿Qué está viendo la red realmente?, ¿Qué es lo que está detectando esos filtros?

La imagen 4 nos da una respuesta a dichas preguntas. Podemos observar que luego del entrenamiento conv1_1 es capaz de detectar ciertos colores y un par de texturas. Mientras tanto conv2_1 parece ser el resultado de combinar las texturas y colores de conv1_1, , recordemos que en medio de estas capas y de las siguientes que mencionaremos existen otras capas y operaciones que tendrán combinaciones de este tipo.

Pasando a conv3_1, podemos hacer dos observaciones, la primera es que los colores se ven mucho más granulados que en las dos capas anteriormente estudiadas y empieza a aparecer forma más definidas como líneas diagonales curvas o puntos bien detallados.

Conv4_1 nos muetra un salto enorme respecto a conv3_1. Podemos identificar con facilidad formas más específicas, agrupaciones de líneas, círculos y combinación de colores que se organizan de forma no uniforme.

Finalmente, conv5_1, muestra filtros que ya saben detectar formas bien definidas, más delineadas, llevando las imágenes a una representación más abstracta a medida que avanza por las diferentes capas.

imagen 4
How convolutional neural networks see the world
 Fuente: Blog de Keras por Francois Chollet

El análisis anterior es el pilar de la técnica de transferencia de estilo presentada por León A. Gatys en el articulo  Image Style Transfer Using Convolutional Neural Networks. Dicho comportamiento ha sido bastante estudiado y es el que logra que las redes neuronales convolucionales capturen detalles y formas, que a su vez, las vuelven enormemente potentes en tareas de clasificación. Es evidente que se empiezan con estructuras muy primitivas como en  conv1_1  hasta llegar a formas más estructuradas en conv5_1.

Lo importante es entender esta capacidad de las  CNN para nuestro tema de interés, es que mientras las primeras capas de la red son más sensibles a los colores y texturas, las últimas capas son capaces de capturar la forma de los objetos. Dicho de otra manera, de las primeras capas capturaremos la representación de la imagen de estilo,luego, de la imagen de contenido obtendremos la representación de alguna de las últimas capas, por ejemplo de conv5_1.

Específicamente en el artículo de León A. Gatys se usa para capturar el estilo:conv1_1, conv2_1, conv3_1, conv4_1, conv5_1 y para capturar el contenido : conv5_2

De esa manera el primer paso es tomar una red neuronal entrenada y pasar a través de ella dos imágenes, una para el contenido y otra para el estilo. Luego, capturar las respectivas capas de interés para cada una.

Ahora, debemos generar una tercer imagen la cual será el resultado(por ejemplo ver imagen 1) de la combinación de las dos anteriores. En el articulo generan una imagen con ruido gaussiano. Experimentos posteriores de la tesis demuestra que al usar la imagen de contenido como imagen de inicio da resultados más consistentes. En los experimentos usaremos ambas.

Luego, pasaremos la imagen generada por la red extrayendo tanto las capas de contenido como de estilo y calcularemos una pérdida de estilo respecto a las capas obtenidas de la imagen de estilo, y otra pérdida de contenido respecto a las capas de la imagen de contenido.

Funciones de costo

Pérdida de estilo

Necesitamos una forma especial para calcular la perdida de estilo. El estilo esta relacionado con la correlacion que hay entre los pixeles de una imagen. Una matriz Gram nos genera una representacion de como son dichas correlaciones. El siguiente video se muestra una explicacion de como funcion y se opera una matriz Gram.

```python
from IPython.display import HTML
HTML('')
```

fuente : Udacity

En el articulo Image Style Transfer Using Convolutional Neural Networks se muestra el calculo de la matriz gram de la siguiente forma:

\begin{equation}
G_{i j}^{l}=\sum_{k} F_{i k}^{l} F_{j k}^{l}
\end{equation}

Luego, podemos calcular el error de las matrices Gram como la diferencia de las matrices Gram para las capas $l$ respectivas entre la imagen de estilo y la imagen generada. En el articulo el error cuadratico esta escrito como :

\begin{equation}
E_{l}=\frac{1}{4 N_{l}^{2} M_{l}^{2}} \sum_{i, j}\left(G_{i j}^{l}-A_{i j}^{l}\right)^{2}
\end{equation}

Ahora, la perdida de estilo se calcula como la sumatoria de los errores de las matrices Gram para cada capa $l$ multiplicado por un peso que se le da a cada capa. En el articulo dicho peso es el mismo para cada una de las capas.

\begin{equation}
\mathcal{L}_{\text { style }}(\vec{a}, \vec{x})=\sum_{l=0}^{L} w_{l} E_{l}
\end{equation}

Perdida de contenido

La perdida de contenido es la suma de las distancias euclidianas de las capas que representan el contenido. Asi , sea $\vec{p}$ la imagen original, $\vec{x}$ la imagen generada y $l$ la capa contenido, la perdida sera.

\begin{equation}
\mathcal{L}_{\text { content }}(\vec{p}, \vec{x}, l)=\frac{1}{2} \sum_{i, j}\left(F_{i j}^{l}-P_{i j}^{l}\right)^{2}
\end{equation}

$F^{l}$ es el mapa de caracteristicas de la capa $l$ de la imagen original, mientras $P^{l}$ es el mapa de caracteristicas de la capa $l$ de la imagen generada.

Perdida total y actualizacion de imagen

Para la perdida total se realiza la suma entre las perdidas de estilo y contenido.

\begin{equation}
\mathcal{L}_{\text { total }}(\vec{p}, \vec{a}, \vec{x})=\alpha \mathcal{L}_{\text { content }}(\vec{p}, \vec{x})+\beta \mathcal{L}_{\text { style }}(\vec{a}, \vec{x})
\end{equation}

Donde alpha $\alpha$ es el peso que se le dara al contenido y $\beta$ el peso que se le dara al estilo.

Solo quedaria calcular el gradiente de la perdida total respecto a la imagen generada, de forma tal, que al cumplir cierto numero de iteraciones el error total disminuya y la imagen generada adquiera el contenido de la imagen de contenido y el estilo de la imagen de estilo. Los parametros $\alpha$ y  $\beta$ son de suma importancia dado que determinaran la forma en como se generara la imagen. El articulo propone ciertas correspondecias para obtener diferentes salidas.

Estrategia de generación de imagen

Para finalizar, discutiremos el flujo de lo revisado hasta el momento paso por paso.

En primer lugar pasamos la imagen de estilo y la imagen de contenido a través de la red neuronal ya entrenada. Obtenemos las capas convolucionales que deseemos de cada una de ellas. Según el artículo para la imagen de estilo obtendremos los mapas de características ubicados en las capas   conv1_1, conv2_1, conv3_1, conv4_1, conv5_1 , l mismo tiempo, para la imagen de contenido obtenemos solo conv5_2.

Para  conv1_1, conv2_1, conv3_1, conv4_1, conv5_1, conv5_2  de la imagen de estilo calculamos las matrices gram.

Posteriormente necesitamos una imagen que nos servirá de lienzo para la imagen generada. Esta imagen puede ser una imagen que generemos con ruido (puede ser una imagen generada mediante una distribución normal) o la imagen de contenido. Los experimentos mostrados en la tesis y el artículo muestran que si se inicia con la imagen de contenido la transferencia de estilo es más estable y fiel al contenido. Esta imagen generada se pasa por la red neuronal y se extrae de ella tanto las capas que se obtuvieron de la imagen de estilo como la de contenido, en otras palabras, de la imagen generada se obtendrá.

Ahora, calculamos las matrices gram de la imagen generada y realizamos la pérdida de estilo respecto a las matrices gram de la imagen de estilo. De forma similar, calculamos la diferencia de  conv5_2 de la imagen generada respecto a la imagen de contenido.

Sumamos ambos errores y a continuación calculamos los gradiente de la imagen generada respecto al error total. Repetimos este proceso el número de optimización que se le quieran realizar a la imagen generada. A mayor número de iteración se espera que la imagen generada tenga mejores resultados.

notebooks/1.Aprendizaje profundo para la transferencia de estilo.ipynb


# PHY321: Two-body problems and Gravitational Forces

**[Morten Hjorth-Jensen](http://mhjgit.github.io/info/doc/web/)**, Department of Physics and Astronomy and Facility for Rare Ion Beams (FRIB), Michigan State University, USA and Department of Physics, University of Oslo, Norway

Date: **Mar 22, 2021**

Copyright 1999-2021, [Morten Hjorth-Jensen](http://mhjgit.github.io/info/doc/web/). Released under CC Attribution-NonCommercial 4.0 license

## Aims and Overarching Motivation

### Monday

1. Computational topics: functions and classes (continuation of Julie's lecture from last Friday).

2. From the harmonic oscillator to the gravitational force and Kepler's laws

**Reading suggestion**: Taylor section 8.4 and Lecture notes

### Wednesday

1. Discussion of elliptical orbits and Kepler's laws

**Reading suggestion**: Taylor sections  8.5-8.8

### Friday

1. Physical interpretation of various orbit types and start discussion two-body scattering

**Reading suggestion**: Taylor section 8.5-8.8 and sections 14.1-14.2

## Deriving Elliptical Orbits

Kepler's laws state that a gravitational orbit should be an ellipse
with the source of the gravitational field at one focus. Deriving this
is surprisingly messy. To do this, we first use angular momentum
conservation to transform the equations of motion so that it is in
terms of $r$ and $\theta$ instead of $r$ and $t$. The overall strategy
is to

1. Find equations of motion for $r$ and $t$ with no angle ($\theta$) mentioned, i.e. $d^2r/dt^2=\cdots$. Angular momentum conservation will be used, and the equation will involve the angular momentum $L$.

2. Use angular momentum conservation to find an expression for $\dot{\theta}$ in terms of $r$.

3. Use the chain rule to convert the equations of motions for $r$, an expression involving $r,\dot{r}$ and $\ddot{r}$, to one involving $r,dr/d\theta$ and $d^2r/d\theta^2$. This is quitecomplicated because the expressions will also involve a substitution $u=1/r$ so that one finds an expression in terms of $u$ and $\theta$.

4. Once $u(\theta)$ is found, you need to show that this can be converted to the familiar form for an ellipse.

The equations of motion give

$$
\begin{eqnarray}
\label{eq:radialeqofmotion} \tag{1}
\frac{d}{dt}r^2&=&\frac{d}{dt}(x^2+y^2)=2x\dot{x}+2y\dot{y}=2r\dot{r},\\
\nonumber
\dot{r}&=&\frac{x}{r}\dot{x}+\frac{y}{r}\dot{y},\\
\nonumber
\ddot{r}&=&\frac{x}{r}\ddot{x}+\frac{y}{r}\ddot{y}
+\frac{\dot{x}^2+\dot{y}^2}{r}
-\frac{\dot{r}^2}{r}.
\end{eqnarray}
$$

Recognizing that the numerator of the third term is the velocity squared, and that it can be written in polar coordinates,

$$
\begin{equation}
v^2=\dot{x}^2+\dot{y}^2=\dot{r}^2+r^2\dot{\theta}^2,
\label{_auto1} \tag{2}
\end{equation}
$$

one can write $\ddot{r}$ as

$$
\begin{eqnarray}
\label{eq:radialeqofmotion2} \tag{3}
\ddot{r}&=&\frac{F_x\cos\theta+F_y\sin\theta}{m}+\frac{\dot{r}^2+r^2\dot{\theta}^2}{r}-\frac{\dot{r}^2}{r}\\
\nonumber
&=&\frac{F}{m}+\frac{r^2\dot{\theta}^2}{r}\\
\nonumber
m\ddot{r}&=&F+\frac{L^2}{mr^3}.
\end{eqnarray}
$$

This derivation used the fact that the force was radial,
$F=F_r=F_x\cos\theta+F_y\sin\theta$, and that angular momentum is
$L=mrv_{\theta}=mr^2\dot{\theta}$. The term $L^2/mr^3=mv^2/r$ behaves
like an additional force. Sometimes this is referred to as a
centrifugal force, but it is not a force. Instead, it is the
consequence of considering the motion in a rotating (and therefore
accelerating) frame.

Now, we switch to the particular case of an attractive inverse square
force, $F=-\alpha/r^2$, and show that the trajectory, $r(\theta)$, is
an ellipse. To do this we transform derivatives w.r.t. time to
derivatives w.r.t. $\theta$ using the chain rule combined with angular
momentum conservation, $\dot{\theta}=L/mr^2$.

$$
\begin{eqnarray}
\label{eq:rtotheta} \tag{4}
\dot{r}&=&\frac{dr}{d\theta}\dot{\theta}=\frac{dr}{d\theta}\frac{L}{mr^2},\\
\nonumber
\ddot{r}&=&\frac{d^2r}{d\theta^2}\dot{\theta}^2
+\frac{dr}{d\theta}\left(\frac{d}{dr}\frac{L}{mr^2}\right)\dot{r}\\
\nonumber
&=&\frac{d^2r}{d\theta^2}\left(\frac{L}{mr^2}\right)^2
-2\frac{dr}{d\theta}\frac{L}{mr^3}\dot{r}\\
\nonumber
&=&\frac{d^2r}{d\theta^2}\left(\frac{L}{mr^2}\right)^2
-\frac{2}{r}\left(\frac{dr}{d\theta}\right)^2\left(\frac{L}{mr^2}\right)^2
\end{eqnarray}
$$

Equating the two expressions for $\ddot{r}$ in Eq.s ([3](#eq:radialeqofmotion2)) and ([4](#eq:rtotheta)) eliminates all the derivatives w.r.t. time, and provides a differential equation with only derivatives w.r.t. $\theta$,

$$
\begin{equation}
\label{eq:rdotdot} \tag{5}
\frac{d^2r}{d\theta^2}\left(\frac{L}{mr^2}\right)^2
-\frac{2}{r}\left(\frac{dr}{d\theta}\right)^2\left(\frac{L}{mr^2}\right)^2
=\frac{F}{m}+\frac{L^2}{m^2r^3},
\end{equation}
$$

that when solved yields the trajectory, i.e. $r(\theta)$. Up to this
point the expressions work for any radial force, not just forces that
fall as $1/r^2$.

The trick to simplifying this differential equation for the inverse
square problems is to make a substitution, $u\equiv 1/r$, and rewrite
the differential equation for $u(\theta)$.

$$
\begin{eqnarray}
r&=&1/u,\\
\nonumber
\frac{dr}{d\theta}&=&-\frac{1}{u^2}\frac{du}{d\theta},\\
\nonumber
\frac{d^2r}{d\theta^2}&=&\frac{2}{u^3}\left(\frac{du}{d\theta}\right)^2-\frac{1}{u^2}\frac{d^2u}{d\theta^2}.
\end{eqnarray}
$$

Plugging these expressions into Eq. ([5](#eq:rdotdot)) gives an
expression in terms of $u$, $du/d\theta$, and $d^2u/d\theta^2$. After
some tedious algebra,

$$
\begin{equation}
\frac{d^2u}{d\theta^2}=-u-\frac{F m}{L^2u^2}.
\label{_auto2} \tag{6}
\end{equation}
$$

For the attractive inverse square law force, $F=-\alpha u^2$,

$$
\begin{equation}
\frac{d^2u}{d\theta^2}=-u+\frac{m\alpha}{L^2}.
\label{_auto3} \tag{7}
\end{equation}
$$

The solution has two arbitrary constants, $A$ and $\theta_0$,

$$
\begin{eqnarray}
\label{eq:Ctrajectory} \tag{8}
u&=&\frac{m\alpha}{L^2}+A\cos(\theta-\theta_0),\\
\nonumber
r&=&\frac{1}{(m\alpha/L^2)+A\cos(\theta-\theta_0)}.
\end{eqnarray}
$$

The radius will be at a minimum when $\theta=\theta_0$ and at a
maximum when $\theta=\theta_0+\pi$. The constant $A$ is related to the
eccentricity of the orbit. When $A=0$ the radius is a constant
$r=L^2/(m\alpha)$, and the motion is circular. If one solved the
expression $mv^2/r=-\alpha/r^2$ for a circular orbit, using the
substitution $v=L/(mr)$, one would reproduce the expression
$r=L^2/(m\alpha)$.

The form describing the elliptical trajectory in
Eq. ([8](#eq:Ctrajectory)) can be identified as an ellipse with one
focus being the center of the ellipse by considering the definition of
an ellipse as being the points such that the sum of the two distances
between the two foci are a constant. Making that distance $2D$, the
distance between the two foci as $2a$, and putting one focus at the
origin,

$$
\begin{eqnarray}
2D&=&r+\sqrt{(r\cos\theta-2a)^2+r^2\sin^2\theta},\\
\nonumber
4D^2+r^2-4Dr&=&r^2+4a^2-4ar\cos\theta,\\
\nonumber
r&=&\frac{D^2-a^2}{D+a\cos\theta}=\frac{1}{D/(D^2-a^2)-a\cos\theta/(D^2-a^2)}.
\end{eqnarray}
$$

By inspection, this is the same form as Eq. ([8](#eq:Ctrajectory)) with $D/(D^2-a^2)=m\alpha/L^2$ and $a/(D^2-a^2)=A$.

Let us remind ourselves about what an ellipse is before we proceed.

```python
%matplotlib inline

import numpy as np
from matplotlib import pyplot as plt
from math import pi

u=1.     #x-position of the center
v=0.5    #y-position of the center
a=2.     #radius on the x-axis
b=1.5    #radius on the y-axis

t = np.linspace(0, 2*pi, 100)
plt.plot( u+a*np.cos(t) , v+b*np.sin(t) )
plt.grid(color='lightgray',linestyle='--')
plt.show()
```

## Effective or Centrifugal Potential

The total energy of a particle is

$$
\begin{eqnarray}
E&=&V(r)+\frac{1}{2}mv_\theta^2+\frac{1}{2}m\dot{r}^2\\
\nonumber
&=&V(r)+\frac{1}{2}mr^2\dot{\theta}^2+\frac{1}{2}m\dot{r}^2\\
\nonumber
&=&V(r)+\frac{L^2}{2mr^2}+\frac{1}{2}m\dot{r}^2.
\end{eqnarray}
$$

The second term then contributes to the energy like an additional
repulsive potential. The term is sometimes referred to as the
"centrifugal" potential, even though it is actually the kinetic energy
of the angular motion. Combined with $V(r)$, it is sometimes referred
to as the "effective" potential,

$$
\begin{eqnarray}
V_{\rm eff}(r)&=&V(r)+\frac{L^2}{2mr^2}.
\end{eqnarray}
$$

Note that if one treats the effective potential like a real potential, one would expect to be able to generate an effective force,

$$
\begin{eqnarray}
F_{\rm eff}&=&-\frac{d}{dr}V(r) -\frac{d}{dr}\frac{L^2}{2mr^2}\\
\nonumber
&=&F(r)+\frac{L^2}{mr^3}=F(r)+m\frac{v_\perp^2}{r},
\end{eqnarray}
$$

which is indeed matches the form for $m\ddot{r}$ in Eq. ([3](#eq:radialeqofmotion2)), which included the **centrifugal** force.

The following code plots this effective potential for a simple choice of parameters, with a standard gravitational potential $-\alpha/r$. Here we have chosen $L=m=\alpha=1$.

```python
# Common imports
import numpy as np
from math import *
import matplotlib.pyplot as plt

Deltax = 0.01
#set up arrays
xinitial = 0.3
xfinal = 5.0
alpha = 1.0   # spring constant
m = 1.0   # mass, you can change these
AngMom = 1.0  #  The angular momentum
n = ceil((xfinal-xinitial)/Deltax)
x = np.zeros(n)
for i in range(n):
    x[i] = xinitial+i*Deltax
V = np.zeros(n)
V = -alpha/x+0.5*AngMom*AngMom/(m*x*x)
# Plot potential
fig, ax = plt.subplots()
ax.set_xlabel('r[m]')
ax.set_ylabel('V[J]')
ax.plot(x, V)
fig.tight_layout()
plt.show()
```

### Gravitational force example

Using the above parameters, we can now study the evolution of the system using for example the velocity Verlet method.
This is done in the code here for an initial radius equal to the minimum of the potential well.  We seen then that the radius is always the same and corresponds to a circle (the radius is always constant).

```python
# Common imports
import numpy as np
import pandas as pd
from math import *
import matplotlib.pyplot as plt
import os

# Where to save the figures and data files
PROJECT_ROOT_DIR = "Results"
FIGURE_ID = "Results/FigureFiles"
DATA_ID = "DataFiles/"

if not os.path.exists(PROJECT_ROOT_DIR):
    os.mkdir(PROJECT_ROOT_DIR)

if not os.path.exists(FIGURE_ID):
    os.makedirs(FIGURE_ID)

if not os.path.exists(DATA_ID):
    os.makedirs(DATA_ID)

def image_path(fig_id):
    return os.path.join(FIGURE_ID, fig_id)

def data_path(dat_id):
    return os.path.join(DATA_ID, dat_id)

def save_fig(fig_id):
    plt.savefig(image_path(fig_id) + ".png", format='png')

# Simple Gravitational Force   -alpha/r

DeltaT = 0.01
#set up arrays
tfinal = 100.0
n = ceil(tfinal/DeltaT)
# set up arrays for t, v and r
t = np.zeros(n)
v = np.zeros(n)
r = np.zeros(n)
# Constants of the model, setting all variables to one for simplicity
alpha = 1.0
AngMom = 1.0  #  The angular momentum
m = 1.0  # scale mass to one
c1 = AngMom*AngMom/(m*m)
c2 = AngMom*AngMom/m
rmin = (AngMom*AngMom/m/alpha)
# Initial conditions
r0 = rmin
v0 = 0.0
r[0] = r0
v[0] = v0
# Start integrating using the Velocity-Verlet  method
for i in range(n-1):
    # Set up acceleration
    a = -alpha/(r[i]**2)+c1/(r[i]**3)
    # update velocity, time and position using the Velocity-Verlet method
    r[i+1] = r[i] + DeltaT*v[i]+0.5*(DeltaT**2)*a
    anew = -alpha/(r[i+1]**2)+c1/(r[i+1]**3)
    v[i+1] = v[i] + 0.5*DeltaT*(a+anew)
    t[i+1] = t[i] + DeltaT
    # Plot position as function of time
fig, ax = plt.subplots(2,1)
ax[0].set_xlabel('time')
ax[0].set_ylabel('radius')
ax[0].plot(t,r)
ax[1].set_xlabel('time')
ax[1].set_ylabel('Velocity')
ax[1].plot(t,v)
save_fig("RadialGVV")
plt.show()
```

Changing the value of the initial position to a value where the energy is positive, leads to an increasing radius with time, a so-called unbound orbit. Choosing on the other hand an initial radius that corresponds to a negative energy and different from the minimum value leads to a radius that oscillates back and forth between two values.

### Harmonic Oscillator in two dimensions

Consider a particle of mass $m$ in a 2-dimensional harmonic oscillator with potential

$$
V=\frac{1}{2}kr^2=\frac{1}{2}k(x^2+y^2).
$$

If the orbit has angular momentum $L$, we can find the radius and angular velocity of the circular orbit as well as the b) the angular frequency of small radial perturbations.

We consider the effective potential. The radius of a circular orbit is at the minimum of the potential (where the effective force is zero).
The potential is plotted here with the parameters $k=m=0.1$ and $L=1.0$.

```python
# Common imports
import numpy as np
from math import *
import matplotlib.pyplot as plt

Deltax = 0.01
#set up arrays
xinitial = 0.5
xfinal = 3.0
k = 1.0   # spring constant
m = 1.0   # mass, you can change these
AngMom = 1.0  #  The angular momentum
n = ceil((xfinal-xinitial)/Deltax)
x = np.zeros(n)
for i in range(n):
    x[i] = xinitial+i*Deltax
V = np.zeros(n)
V = 0.5*k*x*x+0.5*AngMom*AngMom/(m*x*x)
# Plot potential
fig, ax = plt.subplots()
ax.set_xlabel('r[m]')
ax.set_ylabel('V[J]')
ax.plot(x, V)
fig.tight_layout()
plt.show()
```

$$
\begin{eqnarray*}
V_{\rm eff}&=&\frac{1}{2}kr^2+\frac{L^2}{2mr^2}
\end{eqnarray*}
$$

The effective potential looks like that of a harmonic oscillator for
large $r$, but for small $r$, the centrifugal potential repels the
particle from the origin. The combination of the two potentials has a
minimum for at some radius $r_{\rm min}$.

$$
\begin{eqnarray*}
0&=&kr_{\rm min}-\frac{L^2}{mr_{\rm min}^3},\\
r_{\rm min}&=&\left(\frac{L^2}{mk}\right)^{1/4},\\
\dot{\theta}&=&\frac{L}{mr_{\rm min}^2}=\sqrt{k/m}.
\end{eqnarray*}
$$

For particles at $r_{\rm min}$ with $\dot{r}=0$, the particle does not
accelerate and $r$ stays constant, i.e. a circular orbit. The radius
of the circular orbit can be adjusted by changing the angular momentum
$L$.

For the above parameters this minimum is at $r_{\rm min}=1$.

 Now consider small vibrations about $r_{\rm min}$. The effective spring constant is the curvature of the effective potential.

$$
\begin{eqnarray*}
k_{\rm eff}&=&\left.\frac{d^2}{dr^2}V_{\rm eff}(r)\right|_{r=r_{\rm min}}=k+\frac{3L^2}{mr_{\rm min}^4}\\
&=&4k,\\
\omega&=&\sqrt{k_{\rm eff}/m}=2\sqrt{k/m}=2\dot{\theta}.
\end{eqnarray*}
$$

Because the radius oscillates with twice the angular frequency,
the orbit has two places where $r$ reaches a minimum in one
cycle. This differs from the inverse-square force where there is one
minimum in an orbit. One can show that the orbit for the harmonic
oscillator is also elliptical, but in this case the center of the
potential is at the center of the ellipse, not at one of the foci.

The solution is also simple to write down exactly in Cartesian coordinates. The $x$ and $y$ equations of motion separate,

$$
\begin{eqnarray*}
\ddot{x}&=&-kx,\\
\ddot{y}&=&-ky.
\end{eqnarray*}
$$

The general solution can be expressed as

$$
\begin{eqnarray*}
x&=&A\cos\omega_0 t+B\sin\omega_0 t,\\
y&=&C\cos\omega_0 t+D\sin\omega_0 t.
\end{eqnarray*}
$$

The code here finds the solution for $x$ and $y$ using the code we
developed in homework 5 and 6 and the midterm.  Note that this code is
tailored to run in Cartesian coordinates. There is thus no angular
momentum dependent term.

Here we have chose initial conditions that
correspond to the minimum of the effective potential
$r_{\mathrm{min}}$. We have chosen $x_0=r_{\mathrm{min}}$ and
$y_0=0$. Similarly, we use the centripetal acceleration to determine
the initial velocity so that we have a circular motion (see back to the
last question of the midterm). This means that we set the centripetal
acceleration $v^2/r$ equal to the force from the harmonic oscillator $-k\boldsymbol{r}$. Taking the
magnitude of $\boldsymbol{r}$ we have then
$v^2/r=k/mr$, which gives $v=\pm\omega_0r$.

Since the code here solves the equations of motion in cartesian
coordinates and the harmonic oscillator potential leads to forces in
the $x$- and $y$-directions that are decoupled, we have to select the initial velocities and positions so that we don't get that for example $y(t)=0$.

We set $x_0$ to be different from zero and $v_{y0}$ to be different from zero.

```python

DeltaT = 0.00001
#set up arrays
tfinal = 10.0
n = ceil(tfinal/DeltaT)
# set up arrays
t = np.zeros(n)
v = np.zeros((n,2))
r = np.zeros((n,2))
radius = np.zeros(n)
# Constants of the model
k = 1.0   # spring constant
m = 1.0   # mass, you can change these
omega02 = k/m  # Frequency
AngMom = 1.0  #  The angular momentum
# Potential minimum
rmin = (AngMom*AngMom/k/m)**0.25
# Initial conditions as compact 2-dimensional arrays, x0=rmin and y0 = 0
x0 = rmin; y0= 0.0
r0 = np.array([x0,y0])
#vy0 = 2.0 ; vx0 = 0.0
vy0 = sqrt(omega02)*rmin; vx0 = 0.0
v0 = np.array([vx0,vy0])
r[0] = r0
v[0] = v0
# Start integrating using the Velocity-Verlet  method
for i in range(n-1):
    # Set up the acceleration
    a =  -r[i]*omega02
    # update velocity, time and position using the Velocity-Verlet method
    r[i+1] = r[i] + DeltaT*v[i]+0.5*(DeltaT**2)*a
    anew = -r[i+1]*omega02
    v[i+1] = v[i] + 0.5*DeltaT*(a+anew)
    t[i+1] = t[i] + DeltaT
# Plot position as function of time
radius = np.sqrt(r[:,0]**2+r[:,1]**2)
fig, ax = plt.subplots(3,1)
ax[0].set_xlabel('time')
ax[0].set_ylabel('radius squared')
ax[0].plot(t,r[:,0]**2+r[:,1]**2)
ax[1].set_xlabel('time')
ax[1].set_ylabel('x position')
ax[1].plot(t,r[:,0])
ax[2].set_xlabel('time')
ax[2].set_ylabel('y position')
ax[2].plot(t,r[:,1])

fig.tight_layout()
save_fig("2DimHOVV")
plt.show()
```

We see that the radius (to within a given error), we obtain a constant radius.

The following code shows first how we can solve this problem using the radial degrees of freedom only.
Here we need to add the explicit centrifugal barrier.  Note that the variable $r$ depends only on time. There is no $x$ and $y$ directions
since we have transformed the equations to polar coordinates.

```python
DeltaT = 0.01
#set up arrays
tfinal = 10.0
n = ceil(tfinal/DeltaT)
# set up arrays for t, v and r
t = np.zeros(n)
v = np.zeros(n)
r = np.zeros(n)
E = np.zeros(n)
# Constants of the model
AngMom = 1.0  #  The angular momentum
m = 1.0
k = 1.0
omega02 = k/m
c1 = AngMom*AngMom/(m*m)
c2 = AngMom*AngMom/m
rmin = (AngMom*AngMom/k/m)**0.25
# Initial conditions
r0 = rmin
v0 = 0.0
r[0] = r0
v[0] = v0
E[0] = 0.5*m*v0*v0+0.5*k*r0*r0+0.5*c2/(r0*r0)
# Start integrating using the Velocity-Verlet  method
for i in range(n-1):
    # Set up acceleration
    a = -r[i]*omega02+c1/(r[i]**3)
    # update velocity, time and position using the Velocity-Verlet method
    r[i+1] = r[i] + DeltaT*v[i]+0.5*(DeltaT**2)*a
    anew = -r[i+1]*omega02+c1/(r[i+1]**3)
    v[i+1] = v[i] + 0.5*DeltaT*(a+anew)
    t[i+1] = t[i] + DeltaT
    E[i+1] = 0.5*m*v[i+1]*v[i+1]+0.5*k*r[i+1]*r[i+1]+0.5*c2/(r[i+1]*r[i+1])
    # Plot position as function of time
fig, ax = plt.subplots(2,1)
ax[0].set_xlabel('time')
ax[0].set_ylabel('radius')
ax[0].plot(t,r)
ax[1].set_xlabel('time')
ax[1].set_ylabel('Energy')
ax[1].plot(t,E)
save_fig("RadialHOVV")
plt.show()
```

With some work using double angle formulas, one can calculate

$$
\begin{eqnarray*}
r^2&=&x^2+y^2\\
\nonumber
&=&(A^2+C^2)\cos^2(\omega_0t)+(B^2+D^2)\sin^2\omega_0t+(AB+CD)\cos(\omega_0t)\sin(\omega_0t)\\
\nonumber
&=&\alpha+\beta\cos 2\omega_0 t+\gamma\sin 2\omega_0 t,\\
\alpha&=&\frac{A^2+B^2+C^2+D^2}{2},~~\beta=\frac{A^2-B^2+C^2-D^2}{2},~~\gamma=AB+CD,\\
r^2&=&\alpha+(\beta^2+\gamma^2)^{1/2}\cos(2\omega_0 t-\delta),~~~\delta=\arctan(\gamma/\beta),
\end{eqnarray*}
$$

and see that radius oscillates with frequency $2\omega_0$. The
factor of two comes because the oscillation $x=A\cos\omega_0t$ has two
maxima for $x^2$, one at $t=0$ and one a half period later.

## Stability of Orbits

The effective force can be extracted from the effective potential, $V_{\rm eff}$. Beginning from the equations of motion, Eq. ([1](#eq:radialeqofmotion)), for $r$,

$$
\begin{eqnarray}
m\ddot{r}&=&F+\frac{L^2}{mr^3}\\
\nonumber
&=&F_{\rm eff}\\
\nonumber
&=&-\partial_rV_{\rm eff},\\
\nonumber
F_{\rm eff}&=&-\partial_r\left[V(r)+(L^2/2mr^2)\right].
\end{eqnarray}
$$

For a circular orbit, the radius must be fixed as a function of time,
so one must be at a maximum or a minimum of the effective
potential. However, if one is at a maximum of the effective potential
the radius will be unstable. For the attractive Coulomb force the
effective potential will be dominated by the $-\alpha/r$ term for
large $r$ because the centrifugal part falls off more quickly, $\sim
1/r^2$. At low $r$ the centrifugal piece wins and the effective
potential is repulsive. Thus, the potential must have a minimum
somewhere with negative potential. The circular orbits are then stable
to perturbation.

The effective potential is sketched for two cases, a $1/r$ attractive
potential and a $1/r^3$ attractive potential. The $1/r$ case has a
stable minimum, whereas the circular orbit in the $1/r^3$ case is
unstable.

If one considers a potential that falls as $1/r^3$, the situation is
reversed and the point where $\partial_rV$ disappears will be a local
maximum rather than a local minimum. **Fig to come here with code**

The repulsive centrifugal piece dominates at large $r$ and the attractive
Coulomb piece wins out at small $r$. The circular orbit is then at a
maximum of the effective potential and the orbits are unstable. It is
the clear that for potentials that fall as $r^n$, that one must have
$n>-2$ for the orbits to be stable.

Consider a potential $V(r)=\beta r$. For a particle of mass $m$ with
angular momentum $L$, find the angular frequency of a circular
orbit. Then find the angular frequency for small radial perturbations.

For the circular orbit you search for the position $r_{\rm min}$ where the effective potential is minimized,

$$
\begin{eqnarray*}
\partial_r\left\{\beta r+\frac{L^2}{2mr^2}\right\}&=&0,\\
\beta&=&\frac{L^2}{mr_{\rm min}^3},\\
r_{\rm min}&=&\left(\frac{L^2}{\beta m}\right)^{1/3},\\
\dot{\theta}&=&\frac{L}{mr_{\rm min}^2}=\frac{\beta^{2/3}}{(mL)^{1/3}}
\end{eqnarray*}
$$

Now, we can find the angular frequency of small perturbations about the circular orbit. To do this we find the effective spring constant for the effective potential,

$$
\begin{eqnarray*}
k_{\rm eff}&=&\partial_r^2 \left.V_{\rm eff}\right|_{r_{\rm min}}\\
&=&\frac{3L^2}{mr_{\rm min}^4},\\
\omega&=&\sqrt{\frac{k_{\rm eff}}{m}}\\
&=&\frac{\beta^{2/3}}{(mL)^{1/3}}\sqrt{3}.
\end{eqnarray*}
$$

If the two frequencies, $\dot{\theta}$ and $\omega$, differ by an
integer factor, the orbit's trajectory will repeat itself each time
around. This is the case for the inverse-square force,
$\omega=\dot{\theta}$, and for the harmonic oscillator,
$\omega=2\dot{\theta}$. In this case, $\omega=\sqrt{3}\dot{\theta}$,
and the angles at which the maxima and minima occur change with each
orbit.

### Code example with gravitional force

The code example here is meant to illustrate how we can make a plot of the final orbit. We solve the equations in polar coordinates (the example here uses the minimum of the potential as initial value) and then we transform back to cartesian coordinates and plot $x$ versus $y$. We see that we get a perfect circle when we place ourselves at the minimum of the potential energy, as expected.

```python

# Simple Gravitational Force   -alpha/r

DeltaT = 0.001
#set up arrays
tfinal = 10.0
n = ceil(tfinal/DeltaT)
# set up arrays for t, v and r
t = np.zeros(n)
v = np.zeros(n)
r = np.zeros(n)
phi = np.zeros(n)
x = np.zeros(n)
y = np.zeros(n)
# Constants of the model, setting all variables to one for simplicity
alpha = 1.0
AngMom = 1.0  #  The angular momentum
m = 1.0  # scale mass to one
c1 = AngMom*AngMom/(m*m)
c2 = AngMom*AngMom/m
rmin = (AngMom*AngMom/m/alpha)
# Initial conditions, place yourself at the potential min
r0 = rmin
v0 = 0.0  # starts at rest
r[0] = r0
v[0] = v0
phi[0] = 0.0
# Start integrating using the Velocity-Verlet  method
for i in range(n-1):
    # Set up acceleration
    a = -alpha/(r[i]**2)+c1/(r[i]**3)
    # update velocity, time and position using the Velocity-Verlet method
    r[i+1] = r[i] + DeltaT*v[i]+0.5*(DeltaT**2)*a
    anew = -alpha/(r[i+1]**2)+c1/(r[i+1]**3)
    v[i+1] = v[i] + 0.5*DeltaT*(a+anew)
    t[i+1] = t[i] + DeltaT
    phi[i+1] = t[i+1]*c2/(r0**2)
# Find cartesian coordinates for easy plot
x = r*np.cos(phi)
y = r*np.sin(phi)
fig, ax = plt.subplots(3,1)
ax[0].set_xlabel('time')
ax[0].set_ylabel('radius')
ax[0].plot(t,r)
ax[1].set_xlabel('time')
ax[1].set_ylabel('Angle $\cos{\phi}$')
ax[1].plot(t,np.cos(phi))
ax[2].set_ylabel('y')
ax[2].set_xlabel('x')
ax[2].plot(x,y)

save_fig("Phasespace")
plt.show()
```

Try to change the initial value for $r$ and see what kind of orbits you get.
In order to test different energies, it can be useful to look at the plot of the effective potential discussed above.

However, for orbits different from a circle the above code would need modifications in order to allow us to display say an ellipse. For the latter, it is much easier to run our code in cartesian coordinates, as done here. In this code we test also energy conservation and see that it is conserved to numerical precision. The code here is a simple extension of the code we developed for homework 4.

```python
# Common imports
import numpy as np
import pandas as pd
from math import *
import matplotlib.pyplot as plt

DeltaT = 0.01
#set up arrays
tfinal = 10.0
n = ceil(tfinal/DeltaT)
# set up arrays
t = np.zeros(n)
v = np.zeros((n,2))
r = np.zeros((n,2))
E = np.zeros(n)
# Constants of the model
m = 1.0   # mass, you can change these
alpha = 1.0
# Initial conditions as compact 2-dimensional arrays
x0 = 0.5; y0= 0.0
r0 = np.array([x0,y0])
v0 = np.array([0.0,1.0])
r[0] = r0
v[0] = v0
rabs = sqrt(sum(r[0]*r[0]))
E[0] = 0.5*m*(v[0,0]**2+v[0,1]**2)-alpha/rabs
# Start integrating using the Velocity-Verlet  method
for i in range(n-1):
    # Set up the acceleration
    rabs = sqrt(sum(r[i]*r[i]))
    a =  -alpha*r[i]/(rabs**3)
    # update velocity, time and position using the Velocity-Verlet method
    r[i+1] = r[i] + DeltaT*v[i]+0.5*(DeltaT**2)*a
    rabs = sqrt(sum(r[i+1]*r[i+1]))
    anew = -alpha*r[i+1]/(rabs**3)
    v[i+1] = v[i] + 0.5*DeltaT*(a+anew)
    E[i+1] = 0.5*m*(v[i+1,0]**2+v[i+1,1]**2)-alpha/rabs
    t[i+1] = t[i] + DeltaT
# Plot position as function of time
fig, ax = plt.subplots(3,1)
ax[0].set_ylabel('y')
ax[0].set_xlabel('x')
ax[0].plot(r[:,0],r[:,1])
ax[1].set_xlabel('time')
ax[1].set_ylabel('y position')
ax[1].plot(t,r[:,0])
ax[2].set_xlabel('time')
ax[2].set_ylabel('y position')
ax[2].plot(t,r[:,1])

fig.tight_layout()
save_fig("2DimGravity")
plt.show()
print(E)
```

## Scattering and Cross Sections

Scattering experiments don't measure entire trajectories. For elastic
collisions, they measure the distribution of final scattering angles
at best. Most experiments use targets thin enough so that the number
of scatterings is typically zero or one. The cross section, $\sigma$,
describes the cross-sectional area for particles to scatter with an
individual target atom or nucleus. Cross section measurements form the
basis for MANY fields of physics. BThe cross section, and the
differential cross section, encapsulates everything measurable for a
collision where all that is measured is the final state, e.g. the
outgoing particle had momentum $\boldsymbol{p}_f$. y studying cross sections,
one can infer information about the potential interaction between the
two particles. Inferring, or constraining, the potential from the
cross section is a classic {\it inverse} problem. Collisions are
either elastic or inelastic. Elastic collisions are those for which
the two bodies are in the same internal state before and after the
collision. If the collision excites one of the participants into a
higher state, or transforms the particles into different species, or
creates additional particles, the collision is inelastic. Here, we
consider only elastic collisions.

For Coulomb forces, the cross section is infinite because the range of
the Coulomb force is infinite, but for interactions such as the strong
interaction in nuclear or particle physics, there is no long-range
force and cross-sections are finite. Even for Coulomb forces, the part
of the cross section that corresponds to a specific scattering angle,
$d\sigma/d\Omega$, which is a function of the scattering angle
$\theta_s$ is still finite.

If a particle travels through a thin target, the chance the particle
scatters is $P_{\rm scatt}=\sigma dN/dA$, where $dN/dA$ is the number
of scattering centers per area the particle encounters. If the density
of the target is $\rho$ particles per volume, and if the thickness of
the target is $t$, the areal density (number of target scatterers per
area) is $dN/dA=\rho t$. Because one wishes to quantify the collisions
independently of the target, experimentalists measure scattering
probabilities, then divide by the areal density to obtain
cross-sections,

$$
\begin{eqnarray}
\sigma=\frac{P_{\rm scatt}}{dN/dA}.
\end{eqnarray}
$$

Instead of merely stating that a particle collided, one can measure
the probability the particle scattered by a given angle. The
scattering angle $\theta_s$ is defined so that at zero the particle is
unscattered and at $\theta_s=\pi$ the particle is scattered directly
backward. Scattering angles are often described in the center-of-mass
frame, but that is a detail we will neglect for this first discussion,
where we will consider the scattering of particles moving classically
under the influence of fixed potentials $U(\boldsymbol{r})$. Because the
distribution of scattering angles can be measured, one expresses the
differential cross section,

$$
\begin{equation}
\frac{d^2\sigma}{d\cos\theta_s~d\phi}.
\label{_auto4} \tag{9}
\end{equation}
$$

Usually, the literature expresses differential cross sections as

$$
\begin{equation}
d\sigma/d\Omega=\frac{d\sigma}{d\cos\theta d\phi}=\frac{1}{2\pi}\frac{d\sigma}{d\cos\theta},
\label{_auto5} \tag{10}
\end{equation}
$$

where the last equivalency is true when the scattering does not depend
on the azimuthal angle $\phi$, as is the case for spherically
symmetric potentials.

The differential solid angle $d\Omega$ can be thought of as the area
subtended by a measurement, $dA_d$, divided by $r^2$, where $r$ is the
distance to the detector,

$$
\begin{eqnarray}
dA_d=r^2 d\Omega.
\end{eqnarray}
$$

With this definition $d\sigma/d\Omega$ is independent of the distance
from which one places the detector, or the size of the detector (as
long as it is small).

Differential scattering cross sections are calculated by assuming a
random distribution of impact parameters $b$. These represent the
distance in the $xy$ plane for particles moving in the $z$ direction
relative to the scattering center. An impact parameter $b=0$ refers to
being aimed directly at the target's center. The impact parameter
describes the transverse distance from the $z=0$ axis for the
trajectory when it is still far away from the scattering center and
has not yet passed it. The differential cross section can be expressed
in terms of the impact parameter,

$$
\begin{equation}
d\sigma=2\pi bdb,
\label{_auto6} \tag{11}
\end{equation}
$$

which is the area of a thin ring of radius $b$ and thickness $db$. In
classical physics, one can calculate the trajectory given the incoming
kinetic energy $E$ and the impact parameter if one knows the mass and
potential. From the trajectory, one then finds the scattering angle
$\theta_s(b)$. The differential cross section is then

$$
\begin{equation}
\frac{d\sigma}{d\Omega}=\frac{1}{2\pi}\frac{d\sigma}{d\cos\theta_s}=b\frac{db}{d\cos\theta_s}=\frac{b}{(d/db)\cos\theta_s(b)}.
\label{_auto7} \tag{12}
\end{equation}
$$

Typically, one would calculate $\cos\theta_s$ and $(d/db)\cos\theta_s$
as functions of $b$. This is sufficient to plot the differential cross
section as a function of $\theta_s$.

The total cross section is

$$
\begin{equation}
\sigma_{\rm tot}=\int d\Omega\frac{d\sigma}{d\Omega}=2\pi\int d\cos\theta_s~\frac{d\sigma}{d\Omega}.
\label{_auto8} \tag{13}
\end{equation}
$$

Even if the total cross section is infinite, e.g. Coulomb forces, one
can still have a finite differential cross section as we will see
later on.

An asteroid of mass $m$ and kinetic energy $E$ approaches a planet of
radius $R$ and mass $M$. What is the cross section for the asteroid to
impact the planet?

### Solution

Calculate the maximum impact parameter, $b_{\rm max}$, for which the asteroid will hit the planet. The total cross  section for impact is $\sigma_{\rm impact}=\pi b_{\rm max}^2$. The maximum cross-section can be found with the help of angular momentum conservation. The asteroid's incoming momentum is $p_0=\sqrt{2mE}$ and the angular momentum is $L=p_0b$. If the asteroid just grazes the planet, it is moving with zero radial kinetic energy at impact. Combining energy and angular momentum conservation and having $p_f$ refer to the momentum of the asteroid at a distance $R$,

$$
\begin{eqnarray*}
\frac{p_f^2}{2m}-\frac{GMm}{R}&=&E,\\
p_fR&=&p_0b_{\rm max},
\end{eqnarray*}
$$

allows one to solve for $b_{\rm max}$,

$$
\begin{eqnarray*}
b_{\rm max}&=&R\frac{p_f}{p_0}\\
&=&R\frac{\sqrt{2m(E+GMm/R)}}{\sqrt{2mE}}\\
\sigma_{\rm impact}&=&\pi R^2\frac{E+GMm/R}{E}.
\end{eqnarray*}
$$

## Rutherford Scattering

This refers to the calculation of $d\sigma/d\Omega$ due to an inverse
square force, $F_{12}=\pm\alpha/r^2$ for repulsive/attractive
interaction. Rutherford compared the scattering of $\alpha$ particles
($^4$He nuclei) off of a nucleus and found the scattering angle at
which the formula began to fail. This corresponded to the impact
parameter for which the trajectories would strike the nucleus. This
provided the first measure of the size of the atomic nucleus. At the
time, the distribution of the positive charge (the protons) was
considered to be just as spread out amongst the atomic volume as the
electrons. After Rutherford's experiment, it was clear that the radius
of the nucleus tended to be roughly 4 orders of magnitude smaller than
that of the atom, which is less than the size of a football relative
to Spartan Stadium.

The incoming and outgoing angles of the trajectory are at
$\pm\theta'$. They are related to the scattering angle by
$2\theta'=\pi+\theta_s$.

In order to calculate differential cross section, we must find how the
impact parameter is related to the scattering angle. This requires
analysis of the trajectory. We consider our previous expression for
the trajectory where we derived the elliptic form for the trajectory,
Eq. ([8](#eq:Ctrajectory)). For that case we considered an attractive
force with the particle's energy being negative, i.e. it was
bound. However, the same form will work for positive energy, and
repulsive forces can be considered by simple flipping the sign of
$\alpha$. For positive energies, the trajectories will be hyperbolas,
rather than ellipses, with the asymptotes of the trajectories
representing the directions of the incoming and outgoing
tracks. Rewriting Eq. ([8](#eq:Ctrajectory)),

$$
\begin{equation}\label{eq:ruthtraj} \tag{14}
r=\frac{1}{\frac{m\alpha}{L^2}+A\cos\theta}.
\end{equation}
$$

Once $A$ is large enough, which will happen when the energy is
positive, the denominator will become negative for a range of
$\theta$. This is because the scattered particle will never reach
certain angles. The asymptotic angles $\theta'$ are those for which
the denominator goes to zero,

$$
\begin{equation}
\cos\theta'=-\frac{m\alpha}{AL^2}.
\label{_auto9} \tag{15}
\end{equation}
$$

The trajectory's point of closest approach is at $\theta=0$ and the
two angles $\theta'$, which have this value of $\cos\theta'$, are the
angles of the incoming and outgoing particles. From
Fig (**to come**), one can see that the scattering angle
$\theta_s$ is given by,

$$
\begin{eqnarray}
\label{eq:sthetover2} \tag{16}
2\theta'-\pi&=&\theta_s,~~~\theta'=\frac{\pi}{2}+\frac{\theta_s}{2},\\
\nonumber
\sin(\theta_s/2)&=&-\cos\theta'\\
\nonumber
&=&\frac{m\alpha}{AL^2}.
\end{eqnarray}
$$

Now that we have $\theta_s$ in terms of $m,\alpha,L$ and $A$, we wish
to re-express $L$ and $A$ in terms of the impact parameter $b$ and the
energy $E$. This will set us up to calculate the differential cross
section, which requires knowing $db/d\theta_s$. It is easy to write
the angular momentum as

$$
\begin{equation}
L^2=p_0^2b^2=2mEb^2.
\label{_auto10} \tag{17}
\end{equation}
$$

Finding $A$ is more complicated. To accomplish this we realize that
the point of closest approach occurs at $\theta=0$, so from
Eq. ([14](#eq:ruthtraj))

$$
\begin{eqnarray}
\label{eq:rminofA} \tag{18}
\frac{1}{r_{\rm min}}&=&\frac{m\alpha}{L^2}+A,\\
\nonumber
A&=&\frac{1}{r_{\rm min}}-\frac{m\alpha}{L^2}.
\end{eqnarray}
$$

Next, $r_{\rm min}$ can be found in terms of the energy because at the
point of closest approach the kinetic energy is due purely to the
motion perpendicular to $\hat{r}$ and

$$
\begin{equation}
E=-\frac{\alpha}{r_{\rm min}}+\frac{L^2}{2mr_{\rm min}^2}.
\label{_auto11} \tag{19}
\end{equation}
$$

One can solve the quadratic equation for $1/r_{\rm min}$,

$$
\begin{equation}
\frac{1}{r_{\rm min}}=\frac{m\alpha}{L^2}+\sqrt{(m\alpha/L^2)^2+2mE/L^2}.
\label{_auto12} \tag{20}
\end{equation}
$$

We can plug the expression for $r_{\rm min}$ into the expression for $A$, Eq. ([18](#eq:rminofA)),

$$
\begin{equation}
A=\sqrt{(m\alpha/L^2)^2+2mE/L^2}=\sqrt{(\alpha^2/(4E^2b^4)+1/b^2}
\label{_auto13} \tag{21}
\end{equation}
$$

Finally, we insert the expression for $A$ into that for the scattering angle, Eq. ([16](#eq:sthetover2)),

$$
\begin{eqnarray}
\label{eq:scattangle} \tag{22}
\sin(\theta_s/2)&=&\frac{m\alpha}{AL^2}\\
\nonumber
&=&\frac{a}{\sqrt{a^2+b^2}}, ~~a\equiv \frac{\alpha}{2E}
\end{eqnarray}
$$

The differential cross section can now be found by differentiating the
expression for $\theta_s$ with $b$,

$$
\begin{eqnarray}
\label{eq:rutherford} \tag{23}
\frac{1}{2}\cos(\theta_s/2)d\theta_s&=&\frac{ab~db}{(a^2+b^2)^{3/2}}=\frac{bdb}{a^2}\sin^3(\theta_s/2),\\
\nonumber
d\sigma&=&2\pi bdb=\frac{\pi a^2}{\sin^3(\theta_s/2)}\cos(\theta_s/2)d\theta_s\\
\nonumber
&=&\frac{\pi a^2}{2\sin^4(\theta_s/2)}\sin\theta_s d\theta_s\\
\nonumber
\frac{d\sigma}{d\cos\theta_s}&=&\frac{\pi a^2}{2\sin^4(\theta_s/2)},\\
\nonumber
\frac{d\sigma}{d\Omega}&=&\frac{a^2}{4\sin^4(\theta_s/2)}.
\end{eqnarray}
$$

where $a= \alpha/2E$. This the Rutherford formula for the differential
cross section. It diverges as $\theta_s\rightarrow 0$ because
scatterings with arbitrarily large impact parameters still scatter to
arbitrarily small scattering angles. The expression for
$d\sigma/d\Omega$ is the same whether the interaction is positive or
negative.

Consider a particle of mass $m$ and charge $z$ with kinetic energy $E$
(Let it be the center-of-mass energy) incident on a heavy nucleus of
mass $M$ and charge $Z$ and radius $R$. Find the angle at which the
Rutherford scattering formula breaks down.

### Solution

Let $\alpha=Zze^2/(4\pi\epsilon_0)$. The scattering angle in Eq. ([22](#eq:scattangle)) is

$$
\sin(\theta_s/2)=\frac{a}{\sqrt{a^2+b^2}}, ~~a\equiv \frac{\alpha}{2E}.
$$

The impact parameter $b$ for which the point of closest approach
equals $R$ can be found by using angular momentum conservation,

$$
\begin{eqnarray*}
p_0b&=&b\sqrt{2mE}=Rp_f=R\sqrt{2m(E-\alpha/R)},\\
b&=&R\frac{\sqrt{2m(E-\alpha/R)}}{\sqrt{2mE}}\\
&=&R\sqrt{1-\frac{\alpha}{ER}}.
\end{eqnarray*}
$$

Putting these together

$$
\theta_s=2\sin^{-1}\left\{
\frac{a}{\sqrt{a^2+R^2(1-\alpha/(RE))}}
\right\},~~~a=\frac{\alpha}{2E}.
$$

It was from this departure of the experimentally measured
$d\sigma/d\Omega$ from the Rutherford formula that allowed Rutherford
to infer the radius of the gold nucleus, $R$.

Just like electrodynamics, one can define "fields", which for a small
additional mass $m$ are the force per mass and the additional
potential energy per mass. The {\it gravitational field} related to
the force has dimensions of force per mass, or acceleration, and can
be labeled $\boldsymbol{g}(\boldsymbol{r})$. The potential energy per mass has
dimensions of energy per mass. This is analogous to the
electromagnetic potential, which is the potential energy per charge,
and the electric field which is the force per charge.

Because the field $\boldsymbol{g}$ obeys the same inverse square law for a
point mass as the electric field does for a point charge, the
gravitational field also satisfies a version of Gauss's law,

$$
\begin{equation}
\label{eq:GravGauss} \tag{24}
\oint d\boldsymbol{A}\cdot\boldsymbol{g}=-4\pi GM_{\rm inside}.
\end{equation}
$$

Here, $M_{\rm inside}$ is the net mass inside a closed area.

Gauss's law can be understood by considering a nozzle that sprays
paint in all directions uniformly from a point source. Let $B$ be the
number of gallons per minute of paint leaving the nozzle. If the
nozzle is at the center of a sphere of radius $r$, the paint per
square meter per minute that is deposited on some part of the sphere
is

$$
\begin{eqnarray}
F(r)&=&\frac{B}{4\pi r^2}.
\end{eqnarray}
$$

Now, let $F$ also be assigned a direction, so that it becomes a vector
pointing along the direction of the flying paint. For any surface that
surrounds the nozzle, not necessarily a sphere, one can state that

$$
\begin{eqnarray}
\label{eq:paint} \tag{25}
\oint \boldsymbol{dA}\cdot\boldsymbol{F}&=&B,
\end{eqnarray}
$$

regardless of the shape of the surface. This follows because the rate
at which paint is deposited on the surface should equal the rate at
which it leaves the nozzle. The dot product ensures that only the
component of $\boldsymbol{F}$ into the surface contributes to the deposition
of paint. Similarly, if $\boldsymbol{F}$ is any radial inverse-square forces,
that falls as $B/(4\pi r^2)$, then one can apply
Eq. ([25](#eq:paint)). For gravitational fields, $B/(4\pi)$ is replaced
by $GM$, and one quickly "derives" Gauss's law for gravity,
Eq. ([24](#eq:GravGauss)).

Consider Earth to have its mass $M$ uniformly distributed in a sphere
of radius $R$. Find the magnitude of the gravitational acceleration as
a function of the radius $r$ in terms of the acceleration of gravity
at the surface $g(R)$. Assume $r<R$, i.e. you are inside the surface.

{\bf Solution}: Take the ratio of Eq. ([24](#eq:GravGauss)) for two radii, $R$ and $r<R$,

$$
\begin{eqnarray*}
\frac{4\pi r^2 g(r)}{4\pi R^2 g(R)}&=&\frac{4\pi GM_{\rm inside~r}}{4\pi GM_{\rm inside~R}}\\
\nonumber
&=&\frac{r^3}{R^3}\\
\nonumber
g(r)&=&g(R)\frac{r}{R}~.
\end{eqnarray*}
$$

The potential energy per mass is similar conceptually to the voltage, or electric potential energy per charge, that was studied in electromagnetism, if $V\equiv U/m$, $\boldsymbol{g}=-\nabla V$.

## Tidal Forces

Consider a spherical planet of radius $r$ a distance $D$ from another
body of mass $M$. The magnitude of the force due to $M$ on an small
object of mass $\delta m$ on surface of the planet can be calculated
by performing a Taylor expansion about the center of the spherical
planet.

$$
\begin{equation}
F=-\frac{GM\delta m}{D^2}+2\frac{GM\delta m}{D^3}\Delta D+\cdots
\label{_auto14} \tag{26}
\end{equation}
$$

If the $z$ direction points toward the large object, $\Delta D$ can be
referred to as $z$. In the accelerating frame of an observer at the
center of the planet,

$$
\begin{equation}
\delta m\frac{d^2 z}{dt^2}=F-\delta ma'+{\rm other~forces~acting~on~} \delta m,
\label{_auto15} \tag{27}
\end{equation}
$$

where $a'$ is the acceleration of the observer. Because $\delta ma'$
equals the gravitational force on $\delta m$ if it were located at the
planet's center, one can write

$$
\begin{equation}
m\frac{d^2z}{dt^2}=2\frac{GM\delta m}{D^3}z+{\rm other~forces~acting~on~}\delta m.
\label{_auto16} \tag{28}
\end{equation}
$$

Here the other forces could represent the forces acting on $\delta m$
from the spherical planet such as the gravitational force or the
contact force with the surface. If $\theta$ is the angle w.r.t. the
$z$ axis, the effective force acting on $\delta m$ is

$$
\begin{equation}
F_{\rm eff}\approx 2\frac{GM\delta m}{D^3}r\cos\theta\hat{z}+{\rm other~forces~acting~on~}\delta m.
\label{_auto17} \tag{29}
\end{equation}
$$

This first force is the "tidal" force. It pulls objects outward from the center of the object. If the object were covered with water, it would distort the objects shape so that the shape would be elliptical, stretched out along the axis pointing toward the large mass $M$. The force is always along (either parallel or antiparallel to) the $\hat{z}$ direction.

Consider the Earth to be a sphere of radius $R$ covered with water,
with the gravitational acceleration at the surface noted by $g$. Now
assume that a distant body provides an additional constant
gravitational acceleration $\boldsymbol{a}$ pointed along the $z$ axis. Find
the distortion of the radius as a function of $\theta$. Ignore
planetary rotation and assume $a<<g$.

{\bf Solution}: Because Earth would then accelerate with $a$, the
field $a$ would seem invisible in the accelerating frame. A tidal
force would only appear if $a$ depended on position, i.e. $\nabla
\boldsymbol{a}\ne 0$.

Now consider that the field is no longer constant, but that instead $a=-kz$ with $|kR|<<g$.

{\bf Solution}: The surface of the planet needs to be at constant
potential (if the planet is not accelerating). The force per mass,
$-kz$ is like a spring, and the potential per mass is
$kz^2/2$. Otherwise water would move to a point of lower
potential. Thus, the potential energy for a sample mass $\delta m$ is

$$
\begin{eqnarray*}
V(R)+\delta m gh(\theta)-\frac{\delta m}{2}kr^2\cos^2\theta={\rm Constant}\\
V(R)+\delta mgh(\theta)-\frac{\delta m}{2}kR^2\cos^2\theta-\delta m kRh(\theta)\cos^2\theta-\frac{\delta m}{2}kh^2(\theta)\cos^2\theta={\rm Constant}.
\end{eqnarray*}
$$

Here, the potential due to the external field is $(1/2)kz^2$ so that $-\nabla U=-kz$. One now needs to solve for $h(\theta)$. Absorbing all the constant terms from both sides of the equation into one constant $C$, and because both $h$ and $kR$ are small, we can through away terms of order $h^2$ or $kRh$. This gives

$$
\begin{eqnarray*}
gh(\theta)-\frac{1}{2}kR^2\cos^2\theta&=&C,\\
h(\theta)&=&\frac{C}{g}+\frac{1}{2g}kR^2\cos^2\theta,\\
h(\theta)&=&\frac{1}{2g}kR^2(\cos^2\theta-1/3).
\end{eqnarray*}
$$

The term with the factor of $1/3$ replaced the constant and was chosen so that the average height of the water would be zero.

The Sun's mass is $27\times 10^6$ the Moon's mass, but the Sun is 390 times further away from Earth as the Sun. What is ratio of the tidal force of the Sun to that of the Moon.

{\bf Solution}: The gravitational force due to an object $M$ a distance $D$ away goes as $M/D^2$, but the tidal force is only the difference of that force over a distance $R$,

$$
F_{\rm tidal}\propto \frac{M}{D^3}R.
$$

Therefore the ratio of force is

$$
\begin{eqnarray*}
\frac{F_{\rm Sun's~tidal~force}}{F_{\rm Moon's~tidal~force}}
&=&\frac{M_{\rm sun}/D_{\rm sun}^3}{M_{\rm moon}/D_{\rm moon}^3}\\
&=&\frac{27\times 10^6}{390^3}=0.46.
\end{eqnarray*}
$$

The Moon more strongly affects tides than the Sun.


# Patchy overview of `rlplay` with REINFORCE

```python
import torch
import numpy

import matplotlib.pyplot as plt
%matplotlib inline
```

This procedure is not related to the package, but its useful for combining the logged data.

```python
def collate(dicts):
    """collate identically keyed dicts"""
    out = {}
    for dict in dicts:
        for k, v in dict.items():
            out.setdefault(k, []).append(v)

    return out
```

## Rollout collection

Rollout collection is designed to be as much `plug-n-play` as possible, i.e. it
supports **arbitrarily structured nested containers** of arrays or tensors for
environment observations and actions. The actor, however, should **expose**
certain API (described below).

```python
from rlplay.engine import core

# help(core.collect)
```

It's role is to serve as a *middle-man* between the **actor-environment** pair
and the **training loop**: to track the trajectory of the actor in the environment,
and properly record it into the data buffer.

For example, it is not responsible for seeding or randomization of environments
(i'm looking at you, `AtariEnv`), and datatype casting (except for rewards,
which are cast to `fp32` automatically). In theory, there is **no need** for
special data preprocessing, except for, perhaps, casting data to proper dtypes,
like from `numpy.float64` observations to `float32` in `CartPole`.

#### Semantics

The collector just carefully records the trajectory by alternating between
the **REACT** and **STEP+EMIT** phases in the following fashion:

$$
    \cdots
        \longrightarrow t
        \overset{\mathrm{REACT}}{\longrightarrow} t + \tfrac12
        \overset{\mathrm{STEP+EMIT}}{\longrightarrow} t + 1
        \longrightarrow \cdots
    \,, $$

where the half-times $t + \tfrac12$ are commonly referred to as the `afterstates`:
the actor has chosen an action in response to the current observation, yet has
not interacted with the environment.

So the `time` advances in halves, and the proper names for the half times
in the diagram above are the `state`, the `afterstate` and the `next state`,
respectively.

The collected `fragment` data has the following structure:
* `.state` $z_t$ **the current "extended" observation**
  * `.stepno` $n_t$ the step counter
  * `.obs` $x_t$ **the current observation** emitted by transitioning to $s_t$
  * `.act` $a_{t-1}$ **the last action** which caused $s_{t-1} \longrightarrow s_t$ in the env
  * `.rew` $r_t$ **the previous reward** received by getting to $s_t$
  * `.fin` $d_t$ **the termination flag** indicating if $s_t$ is terminal in the env

* `.actor` $A_t$ auxiliary data from the actor due to **REACT**

* `.env` $E_{t+1}$ auxiliary data from the environment due to **STEP+EMIT**

* `.hx` $h_0$ the starting recurrent state of the actor

Here $s_t$ denotes **the unobserved true full state** of the environment.

The actor $\theta$ interacts with the environment and generates the following
**tracked** data during the rollout,
unobserved/non-tracked data **in red**
and $t = 0..T-1$:

*  ${\color{orange}{h_0}}$, the starting recurrent state, is recorded in $\,.\!\mathtt{hx}$

* **REACT**: the actor performs the following update ($t \to t + \frac12$)

$$
    \bigl(
        \underbrace{
            .\!\mathtt{state}[\mathtt{t}]
        }_{{\color{orange}{z_t}}},\,
        {\color{red}{h_t}}
    \bigr)
        \overset{\text{Actor}_{\theta_{\text{old}}}}{\longrightarrow}
        \bigl(
            \underbrace{
                .\!\mathtt{state}.\!\mathtt{act}[\mathtt{t+1}]
            }_{a_t \leadsto {\color{orange}{z_{t+1}}}},\,
            \underbrace{
                .\!\mathtt{actor}[\mathtt{t}]
            }_{{\color{orange}{A_t}}},\,
            {\color{red}{h_{t+1}}}
        \bigr)
    \,, $$

* **STEP+EMIT**: the environment updates it's unobserved state and emits
the observed data ($t + \frac12 \to t+1_-$)

$$
    \bigl(
        {\color{red}{s_t}},\,
        \underbrace{
            .\!\mathtt{state}.\!\mathtt{act}[\mathtt{t+1}]
        }_{a_t \leadsto {\color{orange}{z_{t+1}}}}
    \bigr)
        \overset{\text{Env}}{\longrightarrow}
        \bigl(
            {\color{red}{s_{t+1}}},\,
            \underbrace{
                .\!\mathtt{state}.\!\mathtt{obs}[\mathtt{t+1}]
            }_{x_{t+1} \leadsto {\color{orange}{z_{t+1}}}},\,
            \underbrace{
                .\!\mathtt{state}.\!\mathtt{rew}[\mathtt{t+1}]
            }_{r_{t+1} \leadsto {\color{orange}{z_{t+1}}}},\,
            \underbrace{
                .\!\mathtt{state}.\!\mathtt{fin}[\mathtt{t+1}]
            }_{d_{t+1} \leadsto {\color{orange}{z_{t+1}}}},\,
            \underbrace{
                .\!\mathtt{env}[\mathtt{t}]
            }_{{\color{orange}{E_{t+1}}}}
        \bigr)
    \,, $$

* collect loop ($t + 1_- \to t+1$)

$$
    \bigl(
        {\color{orange}{n_t}},\,
        {\color{orange}{d_{t+1}}}
    \bigr)
        \longrightarrow
            \underbrace{
                .\!\mathtt{state}.\!\mathtt{stepno}[\mathtt{t+1}]
            }_{n_{t+1} \leadsto {\color{orange}{z_{t+1}}}}
    \,. $$

Here $r_t$ is a scalar reward, $d_t = \top$ if $s_t$ is terminal, or $\bot$
otherwise, $n_{t+1} = 0$ if $d_t = \top$, else $1 + n_t$, and $a \leadsto b$
means $a$ being recored into $b$.

In general, we may treat $z_t$, the extended observation, as an ordinary
observation, by **suitably modifying** the environment: we can make it
recall the most recent action $a_{t-1}$ and compute the termination indicator
$d_t$ of the current state, and let it keep track of the interaction counter
$n_t$, and, finally, we can configure it to supply the most recent reward
$r_t$ as part of the emitted observation.

Hence we essentially consider the following POMDP setup:
\begin{align}
    a_t, h_{t+1}, A_t
        &\longleftarrow \operatorname{Actor}(z_t, h_t; \theta)
        \,, \\
    z_{t+1}, r_{t+1}, E_{t+1}, s_{t+1}
        &\longleftarrow \operatorname{Env}(s_t, a_t)
        \,, \\
\end{align}

Specifically, let $
(z_t)_{t=0}^T
    = (n_t, x_t, a_{t-1}, r_t, d_t)_{t=0}^T
$ be the trajectory fragment in `.state`, and $h_0$, `.hx`, be the starting
(not necessarily the initial) recurrent state of the actor at the begining
of the rollout.

#### Requirements

* all nested containers **must be** built from pure python `dicts`, `lists`, `tuples` or `namedtuples`

* the environment communicates either in **numpy arrays** or in python **scalars**, but not in data types that are incompatible with pytorch (such as `str` or `bytes`)

```python
# example
obs = {
    'camera': {
        'rear': numpy.zeros(3, 320, 240),
        'front': numpy.zeros(3, 320, 240),
    },
    'proximity': (+0.1, +0.2, -0.1, +0.0,),
    'other': {
        'fuel_tank': 78.5,
        'passenger': False,
    },
}
```

* the actor communicates in torch tensors **only**

* the environment produces **float scalar** rewards (other data may be communicated through auxiliary environment info-dicts)

### Container support with `.plyr`

One of the core tools used in `rlplay` is a high performing procedure that traverses
containers of `list`, `dict` and `tuple` and calls the specified function with the
non-container objects found the containers as arguments (like `map`, but not an iterator
for arbitrarily and applicable to structured objects).

See [plyr](https://pypi.org/project/python-plyr/), its `README.md` and
`plyr.apply` for docs.

The `apply` procedure has slightly faster specialized version `suply` and `tuply`,
which do not waste time on validating the structure of the containers. They differ
in the manner in which they call the specified function: the first passes positional
arguments, while the second passes all arguments in one tuple (think of `map` and
`starmap` from `functools`)

```python
# appliers of functions to nested objects
from plyr import apply, suply, tuply

# `setitem` function with argument order, specialized for `apply`
from plyr import xgetitem, xsetitem

# help(apply)
```

How to use `suply` to reset the recurrent state `hx` returned by `torch.nn.LSTM`:

```python
# the mask of inputs just after env resets
fin = torch.randint(2, size=(10, 4), dtype=bool)

# the tensors in `hx` must have the same 2nd dim as `fin`
hx = torch.randn(2, 1, 4, 32, requires_grad=False).unbind()
h0 = torch.zeros(2, 1, 4, 32, requires_grad=True).unbind()
# XXX h0 and hx are tuples of tensors (but we're just as good with dicts)

# get the masks at step 2, and make it broadcastable with 3d hx
m = ~fin[2].unsqueeze(-1)  # reset tensors at fin==False

# multiply by zero the current `hx` (diff-able reset and grad stop)
suply(
    m.mul,  # `.mul` method of the mask upcasts from bool to float if necessary
    hx,     # arg `other` of `.mul`
)

# replace the reset batch elments by a diff-able init value
suply(
    torch.add,         # .add(input, other, *, alpha=1.)
    suply(m.mul, hx),  # arg `input` of `.add`
    suply(r.mul, h0),  # arg `other` of `.add`
    # alpha=1.         # pass other `alpha` if we want
)

# XXX `torch.where` does not have an `easily` callable interface
suply(
    lambda a, b: torch.where(m, a, b),  # or `a.where(m, b)`
    hx, h0,
)
```

For example, this is used to manually run the recurrent network loop:
```python
from torch.nn.utils.rnn import pack_padded_sequence
from torch.nn.utils.rnn import pad_packed_sequence

if use_cudnn and sticky:
    # sequence padding (MUST have sampling with `sticky=True`)
    n_steps, n_env, *_ = fin.shape
    if n_steps > 1:
        # we assume sticky=True
        lengths = 1 + (~fin[1:]).sum(0).cpu()  # first observation's fin should be ignored
        inputs = pack_padded_sequence(input, lengths, enforce_sorted=False)

    output, hx = self.core(inputs, hx)
    if n_steps > 1:
        output, lens = pad_packed_sequence(
            output, batch_first=False, total_length=n_steps)

else:
    # input is T x B x F, hx is either None, or a proper recurrent state
    outputs = []
    for x, m in zip(input.unsqueeze(1), ~fin.unsqueeze(-1)):
        # `m` indicates if no reset took place, otherwise
        #  multiply by zero to stop the grads
        if hx is not None:
            hx = suply(m.mul, hx)

        output, hx = self.core(x, hx)
        outputs.append(output)

    output = torch.cat(outputs, dim=0)
```

### Creating the actors

Rollout collection relies on the following API of the actor:
* `.reset(j, hx)` reset the recurrent state of the j-th environment in the batch (if applicable)
  * `hx` contains tensors with shape `(n_lstm_layers * n_dir) x batch x hidden`, or is an empty tuple
  * the returned `hx` is the updated recurrent state

* `.step(stepno, obs, act, rew, fin, /, *, hx, virtual)` get the next action $a_t$, the recurrent state $h_{t+1}$, and
the **extra info** in response to $n_t$, $x_t$, $a_{t-1}$, $r_t$, $d_t$, and $h_t$ respectively.
  * extra info `dict` **might** include `value` key with a `T x B` tensor of state value estimates $
      v_t(z_t) \approx G_t = \mathbb{E} \sum_{j\geq t} \gamma^{j-t} r_{j+1}
  $.
  * MUST allocate new `hx` if the recurrent state is updated
  * MUST NOT change the inputs in-place

```python
from rlplay.engine import BaseActorModule

help(BaseActorModule.reset)
```

```python
help(BaseActorModule.step)
```

`BaseActorModule` is essentially a thin sub-class of `torch.nn.Module`, that implements
the API through `.forward(obs, act, rew, fin, *, hx, stepno)`, which should return three things:

1. `actions` prescribed actions in the environment, with data of shape `n_steps x batch x ...`
  * can be a nested container of dicts, lists, and tuples

2. `hx` data with shape `n_steps x batch x ...`
  * can be a nested container of dicts, lists, and tuples
  * **if an actor is not recurrent**, then must return an empty container, e.g. a tuple `()`

3. `info` object, which might be a tensor or a nested object containing data in tensors
`n_steps x batch x ...`. For example, one may communicate the following data:
  * `value` -- the state value estimates $v(z_t)$
  * `logits` -- the policy logits $\log \pi(\cdot \mid z_t)$
  * `q` -- $Q(z_t, \cdot)$ values

Here is an example actor, that wraps a simple MLP policy.

```python
from rlplay.utils.common import multinomial

class PolicyWrapper(BaseActorModule):
    """A non-recurrent policy for a flat `Discrete(n)` action space."""

    def __init__(self, policy):
        super().__init__()
        self.policy = policy

        # for updating the exploration epsilon in the clones
        # self.register_buffer('epsilon', torch.tensor(epsilon))

    def forward(self, obs, act=None, rew=None, fin=None,
                *, hx=None, stepno=None, virtual=False):
        # Everything is  [T x B x ...]
        logits = self.policy(locals())

        actions = multinomial(logits.detach().exp())

        return actions, (), dict(logits=logits)
```

### Manual rollout collection

We shall need the following procedures from the core of the engine:

```python
from rlplay.engine.core import prepare, startup, collect
```

Manual collection requires an `actor` and a batch of environment instances `envs`.

Prepare the run-time context for the specified `actor` and the environments
```python
# settings
sticky = False  # whether to stop interacting if an env resets mid-fragment
device = None   # specifies the device to put the actor's inputs and data onto
pinned = False  # whether to keep the running context in non-resizable pinned
                #  (non-paged) memory for faster host-device transfers

# initialize a buffer for one rollout fragment
buffer = prepare(envs[0], actor, n_steps, len(envs),
                 pinned=False, device=device)

# the running context tor the actor and the envs (optionally pinned)
ctx, fragment = startup(envs, actor, buffer, pinned=pinned)

while not done:
    # collect the fragment
    collect(envs, actor, fragment, ctx, sticky=sticky, device=device)

    # fragment.pyt -- torch tensors, fragment.npy -- numpy arrays (aliased on-host)
    do_stuff(actor, fragment.pyt)
```

### Rollout collection (same-process)

Collect rollouts within the current process

```python
from rlplay.engine.rollout import same
```

The parameters have the following meaning
```python
it = same.rollout(
    envs,           # the batch of environment instances
    actor,          # the actor which interacts with the batch
    n_steps=51,     # the length of the rollout fragment
    sticky=False,   # whether to stop interacting if an env resets mid-fragment
    device=None,    # specifies the device to put the actor's inputs onto
)
```

`rollout()` returns an iterator, which has, roughly, the same logic,
as the manual collection above.

Inside the infinite loop it copies `fragment.pyt` onto `device`, before
yielding it to the user. It also does not spawn its own batch of environments,
unlike parallel variants.

The user has to manually limit the number of iterations using, for example,

```python
it = same.rollout(...)

for b, batch in zip(range(100), it):
    # train on batch
    pass

it.close()
```

### Rollout collection (single-process)

Single-actor rollout sampler running in a parallel process (double-buffered).

```python
from rlplay.engine.rollout import single
```

Under the hood the functions creates **two** rollout fragment buffers, maintains
a reference to the specified `actor`, makes a shared copy of it (on the host), and
then spawns one worker process.

The worker, in turn, makes its own local copy of the actor on the specified device,
initializes the environments and the running context. During collection it alternates
between the buffers, into which it records the rollout fragments it collects. Except
for double buffering, the logic is identical to `rollout`.

The local copies of the actor are **automatically updated** from the maintained reference.

```python
it = single.rollout(
    factory,              # the environment factory
    actor,                # the actor reference, used to update the local actors

    n_steps,              # the duration of a rollout fragment
    n_envs,               # the number of independent environments in the batch

    sticky=False,         # do we freeze terminated environments until the end of the rollout?
                          #  required if we wish to leverage cudnn's fast RNN implementations,
                          #  instead of manually stepping through the RNN core.

    clone=True,           # should the worker use a local clone of the reference actor

    close=True,           # should we `.close()` the environments when cleaning up?
                          #  some envs are very particular about this, e.g. nle

    start_method='fork',  # `fork` in notebooks, `spawn` in linux/macos and if we interchange
                          #  cuda tensors between processes (we DO NOT do that: we exchange indices
                          #  to host-shapred tensors)

    device=None,          # the device on which to collect rollouts (the local actor is moved
                          #  onto this device)
)

# ...

it.close()
```

### Rollout collection (multi-process)

A more load-balanced multi-actor milti-process sampler

```python
from rlplay.engine.rollout import multi
```

This version of the rollout collector allocates several buffers and spawns
many parallel workers. Each worker creates it own local copy of the actor,
instantiates `n_envs` local environments and allocates a running context for
all of them. The rollout collection in each worker is **hardcoded to run on
the host device**.

```python
it = multi.rollout(
    factory,              # the environment factory
    actor,                # the actor reference, used to update the local actors

    n_steps,              # the duration of each rollout fragment

    n_actors,             # the number of parallel actors
    n_per_actor,          # the number of independent environments run in each actor
    n_buffers,            # the size of the pool of buffers, into which rollout
                          #  fragments are collected. Should not be less than `n_actors`.
    n_per_batch,          # the number of fragments collated into a batch

    sticky=False,         # do we freeze terminated environments until the end of the rollout?
                          #  required if we wish to leverage cudnn's fast RNN implementations,
                          #  instead of manually stepping through the RNN core.

    pinned=False,

    clone=True,           # should the parallel actors use a local clone of the reference actor

    close=True,           # should we `.close()` the environments when cleaning up?
                          #  some envs are very particular about this, e.g. nle

    device=None,          # the device onto which to move the rollout batches

    start_method='fork',  # `fork` in notebooks, `spawn` in linux/macos and if we interchange
                          #  cuda tensors between processes (we DO NOT do that: we exchange indices
                          #  to host-shared tensors)
)

# ...

it.close()
```

### Evaluation (same-process)

In order to evaluate an actor in a batch of environments, one can use `evaluate`.

```python
from rlplay.engine import core

# help(core.evaluate)
```

The function *does not* collect the rollout data, except for the rewards.
Below is the intended use case.
* **NB** this is run in the same process, hence blocks until completion, which
might take considerable time (esp. if `n_steps` is unbounded)

```python
# same process
def same_evaluate(
    factory, actor, n_envs=4,
    *, n_steps=None, close=True, render=False, device=None
):
    # spawn a batch of environments
    envs = [factory() for _ in range(n_envs)]

    try:
        while True:
            rewards, _ = core.evaluate(
                envs, actor, n_steps=n_steps,
                render=render, device=device)

            # get the accumulated rewards (gamma=1)
            yield sum(rewards)

    finally:
        if close:
            for e in envs:
                e.close()
```

### Evaluation (parallel process)

Like rollout collection, evaluation can (and probably should) be performed in
a parallel process, so that it does not burden the main thread with computations
not related to training.

```python
from rlplay.engine.rollout.evaluate import evaluate
```

## CartPole with REINFORCE

### the CartPole Environment

```python
import gym

# hotfix for gym's unresponsive viz (spawns gl threads!)
import rlplay.utils.integration.gym
```

The environment factory

```python
class FP32Observation(gym.ObservationWrapper):
    def observation(self, observation):
        return observation.astype(numpy.float32)
#         obs[0] = 0.  # mask the position info
#         return obs  # observation.astype(numpy.float32)

def factory(seed=None):
    return FP32Observation(gym.make("CartPole-v0").unwrapped)
```

### the algorithms

Service functions for the algorithms

```python
from plyr import apply, suply, xgetitem

def timeshift(state, *, shift=1):
    """Get current and shfited slices of nested objects."""
    # use xgetitem to lett None through
    # XXX `curr[t]` = (x_t, a_{t-1}, r_t, d_t), t=0..T-H
    curr = suply(xgetitem, state, index=slice(None, -shift))

    # XXX `next[t]` = (x_{t+H}, a_{t+H-1}, r_{t+H}, d_{t+H}), t=0..T-H
    next = suply(xgetitem, state, index=slice(shift, None))

    return curr, next
```

The reinforce PG algo

```python
from rlplay.algo.returns import pyt_returns

# @torch.enable_grad()
def reinforce(fragment, module, *, gamma=0.99, C_entropy=1e-2):
    r"""The REINFORCE algorithm.

    The basic policy-gradient algorithm with a baseline $b_t$:
    $$
        \nabla_\theta J(s_t)
            = \mathbb{E}_{a \sim \beta(a\mid s_t)}
                \frac{\pi(a\mid s_t)}{\beta(a\mid s_t)}
                    \bigl( r_{t+1} + \gamma G_{t+1} - b_t \bigr)
                \nabla_\theta \log \pi(a\mid s_t)
        \,. $$

    Details
    -------
    It turns out that applying on-policy algo in off-policy setting
    and expecting it to produce acceptable results was a sure sign of
    stupidity on part of the author of this notebook. Oh, well...
    """

    #  get `.state[t]` and `.state[t+1]`
    state, state_next = timeshift(fragment.state)

    # REACT: (state[t], h_t) \to (\hat{a}_t, h_{t+1}, \hat{A}_t)
    _, _, info = module(
        state.obs, state.act, state.rew, state.fin,
        hx=fragment.hx, stepno=state.stepno)

    # Get the returns-to-go -- the present value of the future rewards
    #  following `state[t]`:  G_t = r_{t+1} + \gamma G_{t+1}
    # XXX bootstrap with the perpetual last reward?
    # bootstrap = state_next.rew[-1]  # torch.tensor(0.)
    # bootstrap = state_next.rew.mean(dim=0)  # .div_(1 - gamma)
    ret = pyt_returns(state_next.rew, state_next.fin,
                      gamma=gamma, bootstrap=torch.tensor(0.))

    # `.state_next[t].act` is the action taken in response to `.state[t]`
    # We assume it is unstructured and categorical.
    act = state_next.act.unsqueeze(-1)

    # the policy surrogate score (max)
    #    \frac1T \sum_t (G_t - b_t) \log \pi(a_t \mid s_t)
    # ret.sub_(ret.mean(dim=0))  # .div_(ret.std(dim=0))
    log_pi = info['logits']  # the current policy
    log_pi_a = log_pi.gather(-1, act).squeeze(-1)
    reinfscore = log_pi_a.mul(ret).mean()  # the log-likelihood

    # the policy neg-entropy score (min)
    #   - H(\pi(\cdot \mid s)) = - (-1) \sum_a \pi(a\mid s) \log \pi(a\mid s)
    f_min = torch.finfo(log_pi.dtype).min
    negentropy = log_pi.exp().mul(log_pi.clamp(min=f_min)).sum(dim=-1).mean()

    # maximize the entropy and the reinforce score
    # \ell := - \frac1T \sum_t G_t \log \pi(a_t \mid s_t)
    #         - C \mathbb{H} \pi(\cdot \mid s_t)
    loss = C_entropy * negentropy - reinfscore
    return loss.mean(), dict(
        entropy=-float(negentropy),
        policy_score=float(reinfscore),
    )
```

### the Actor

A procedure and a layer, which converts the input integer data into its
little-endian binary representation as float $\{0, 1\}^m$ vectors.

```python
def onehotbits(input, n_bits=63, dtype=torch.float):
    """Encode integers to fixed-width binary floating point vectors"""
    assert not input.dtype.is_floating_point
    assert 0 < n_bits < 64  # torch.int64 is signed, so 64-1 bits max

    # n_bits = {torch.int64: 63, torch.int32: 31, torch.int16: 15, torch.int8 : 7}

    # get mask of set bits
    pow2 = torch.tensor([1 << j for j in range(n_bits)]).to(input.device)
    x = input.unsqueeze(-1).bitwise_and(pow2).to(bool)

    # upcast bool to float to get one-hot
    return x.to(dtype)

class OneHotBits(torch.nn.Module):
    def __init__(self, n_bits=63, dtype=torch.float):
        assert 1 <= n_bits < 64
        super().__init__()
        self.n_bits, self.dtype = n_bits, dtype

    def forward(self, input):
        return onehotbits(input, n_bits=self.n_bits, dtype=self.dtype)
```

A special module dictionary, which applies itself to the input dict of tensors

```python
from typing import Optional, Mapping
from torch.nn import Module, ModuleDict as BaseModuleDict

class ModuleDict(BaseModuleDict):
    """The ModuleDict, that applies itself to the input dicts."""
    def __init__(
        self,
        modules: Optional[Mapping[str, Module]] = None,
        dim: Optional[int]=-1
    ) -> None:
        super().__init__(modules)
        self.dim = dim

    def forward(self, input):
        # enforce concatenation in the order of the declaration in  __init__
        return torch.cat([
            m(input[k]) for k, m in self.items()
        ], dim=self.dim)
```

A policy which uses many inputs.

```python
from torch.nn import Sequential
from torch.nn import Embedding, Linear, Identity
from torch.nn import ReLU, LogSoftmax

def policy():
    return Sequential(
        ModuleDict(dict(
#             stepno=Sequential(
#                 OneHotBits(), Linear(63, 4, bias=False)
#             ),
            obs=Identity(),
            act=Embedding(2, 2),
        )),
        Linear(0 + 4 + 2, 32),
        ReLU(),
        Linear(32, 2),
        LogSoftmax(dim=-1),
    )
```

The discount factor

```python
gamma = 0.99
C_entropy = 0.1
```

Initialize the learner and the factories

```python
from functools import partial

factory_eval = partial(factory)

learner, sticky = PolicyWrapper(policy()), False

learner.train()
device_ = torch.device('cpu')  # torch.device('cuda:0')
learner.to(device=device_)

# prepare the optimizer for the learner
optim = torch.optim.Adam(learner.parameters(), lr=1e-3)
```

Pick one collector
* the `fork` method is friendlier towards notebooks, but some environments, like the NetHack environment, do not like it
* unlike `fork`, the `spawn` method is `torch.cuda` compatible in that it allows moving on-device tensors between processes. It is not notebook friendly, however :(
  * essentially it is better to prototype in notebook with `same.rollout`, then write a submodule non-interactive script with `multi.rollout`

`REINFORCE` and `A2C` methods do not work ~~well~~ in off-policy setting, so we use
the same-process collector, which guarantees on-policy trajectory data.

```python
T, B = 120, 8
```

Initialize the sampler

```python
# generator of rollout batches
batchit = same.rollout(
    [factory() for _ in range(B)],
    learner,
    n_steps=T,
    sticky=sticky,
    device=device_,
)
```
from rlplay.engine.rollout import episodic

# generator of rollout batches
batchit = episodic.rollout(
    [factory() for _ in range(B)],
    learner,
    batch_size=8,
    device=device_,
)# generator of rollout batches
batchit = single.rollout(
    factory,
    learner,
    n_steps=T,
    n_envs=B,
    sticky=sticky,  # so that we can leverage cudnn's fast RNN implementations
    clone=False,
    close=False,
    device=device_,
    start_method='fork',  # fork in notebook for macos, spawn in linux
)# generator of rollout batches
batchit = multi.rollout(
    factory,
    learner,
    n_steps=T,
    n_actors=16,
    n_per_actor=B,
    n_buffers=24,
    n_per_batch=2,
    sticky=sticky,  # so that we can leverage cudnn's fast RNN implementations
    pinned=False,
    clone=True,
    close=False,
    device=device_,
    start_method='fork',  # fork in notebook for macos, spawn in linux
)
Generator of evaluation rewards:
* we're perfectly OK with evaluating in a parallel process

```python
# test_it = test(factory_eval, learner, n_envs=4, n_steps=500, device=device_)
test_it = evaluate(factory_eval, learner, n_envs=4, n_steps=500,
                   clone=False, device=device_, start_method='fork')
```

Implement your favorite training method
torch.autograd.set_detect_anomaly(True)

```python
import tqdm
# from math import log, exp
from torch.nn.utils import clip_grad_norm_

# pytoch loves to hog all threads on some linux systems
torch.set_num_threads(1)

# the training loop
losses, rewards, samples = [], [], []
# decay = -log(2) / 25  # exploration epsilon half-life
for epoch in tqdm.tqdm(range(100)):
    for j, batch in zip(range(40), batchit):
        loss, info = reinforce(batch, learner, gamma=gamma,
                               C_entropy=C_entropy)

        optim.zero_grad()
        loss.backward()
        grad_norm = clip_grad_norm_(learner.parameters(), max_norm=1.0)
        optim.step()

        losses.append(dict(
            loss=float(loss),
            grad=float(grad_norm),
            **info
        ))

    # This is an example of how to save a batch: we need to clone,
    #  because the fragment buffer is static, and will be overwritten!
    samples.append(suply(torch.clone, batch))

    # fetch the evaluation results (lag by one inner loop!)
    rewards.append(next(test_it))

    # learner.epsilon.mul_(exp(decay)).clip_(0.1, 1.0)
```

```python
# stack all samples
samples = tuply(torch.stack, *samples)

# close the generators
batchit.close()
test_it.close()
```
import pdb; pdb.pm()

```python
data = {k: numpy.array(v) for k, v in collate(losses).items()}
```

```python
if 'loss' in data:
    plt.plot(data['loss'])
```

```python
if 'entropy' in data:
    plt.plot(data['entropy'])
```

```python
if 'policy_score' in data:
    plt.plot(data['policy_score'])
```

```python
plt.semilogy(data['grad'])
```

```python
rewards = numpy.stack(rewards, axis=0)
```

```python
rewards
```

```python
m, s = numpy.median(rewards, axis=-1), rewards.std(axis=-1)
```

```python
fi, ax = plt.subplots(1, 1, figsize=(4, 2), dpi=300)

ax.plot(numpy.mean(rewards, axis=-1))
ax.plot(numpy.median(rewards, axis=-1))
ax.plot(numpy.min(rewards, axis=-1))
ax.plot(numpy.std(rewards, axis=-1))
# ax.plot(m+s * 1.96)
# ax.plot(m-s * 1.96)

plt.show()
```

The ultimate evaluation run

```python
with factory_eval() as env:
    learner.eval()
    eval_rewards, info = core.evaluate([
        env
    ], learner, render=True, n_steps=1e4, device=device_)

print(sum(eval_rewards))
```
import pdb; pdb.pm()

```python
plt.hist(numpy.exp(info['logits']).argmax(-1))
```

Let's analyze the performance

```python
import math
from scipy.special import softmax, expit, entr

*head, n_actions = info['logits'].shape
proba = softmax(info['logits'], axis=-1)

fig, ax = plt.subplots(1, 1, figsize=(4, 2), dpi=300)
ax.plot(entr(proba).sum(-1)[:, 0])
ax.axhline(math.log(n_actions), c='k', alpha=0.5, lw=1);
```

```python
fig, ax = plt.subplots(1, 1, figsize=(4, 2), dpi=300)
ax.hist(info['logits'][..., 1] - info['logits'][..., 0], bins=51);  # log-ratio
```

```python
assert False
```
import pdb; pdb.pm()

```python
# stepno = batch.state.stepno
stepno = torch.arange(8192)
```

```python
with torch.no_grad():
    out = learner.policy[0]['stepno'](stepno)
```

```python
fig, axes = plt.subplots(2, 2, figsize=(8, 8), dpi=200,
                         sharex=True, sharey=True)

for j, ax in zip(range(out.shape[1]), axes.flat):
    ax.plot(out[:, j], lw=1)

fig.tight_layout(pad=0, h_pad=0, w_pad=0)
```

```python
with torch.no_grad():
    plt.imshow(abs(learner.policy[4].weight) @ abs(learner.policy[1].weight))
```

```python
with torch.no_grad():
    plt.imshow(abs(learner.policy[0]['stepno'][-1].weight)[:, :16].T)
```

```python
assert False
```


[Sascha Spors](https://orcid.org/0000-0001-7225-9992),
Professorship Signal Theory and Digital Signal Processing,
[Institute of Communications Engineering (INT)](https://www.int.uni-rostock.de/),
Faculty of Computer Science and Electrical Engineering (IEF),
[University of Rostock, Germany](https://www.uni-rostock.de/en/)

# Tutorial Signals and Systems (Signal- und Systemtheorie)

Summer Semester 2021 (Bachelor Course #24015)

- lecture: https://github.com/spatialaudio/signals-and-systems-lecture
- tutorial: https://github.com/spatialaudio/signals-and-systems-exercises

WIP...
The project is currently under heavy development while adding new material for the summer semester 2021

Feel free to contact lecturer [frank.schultz@uni-rostock.de](https://orcid.org/0000-0002-3010-0294)

## Region of Convergence for Right-Sided Signals

```python
import numpy as np
import matplotlib.pyplot as plt
import matplotlib
```

```python
def plotpzmap_neg(s0):
    fig,(ax1, ax2) = plt.subplots(1, 2)
    fig.set_size_inches(6, 3)
    ax1.plot((-2, 2),(0, 0), lw=1, color='k')
    ax1.plot((0, 0),(-2, 2), lw=1, color='k')
    ax1.plot(-s0, 0, 'x', color='C0', markersize=10)
    p = matplotlib.patches.Rectangle((-s0,-2), 4, 4, color='gray', alpha=0.15)
    ax1.add_patch(p)
    ax1.axis('Square')
    ax1.set_xlim(-2,2)
    ax1.set_ylim(-2,2)
    ax1.set_xlabel("$\Re(s)$")
    ax1.set_ylabel("$\Im(s)$")
    ax1.set_title("exp("+str(s0)+"t) $\epsilon$(t)")
    ax1.text(1,-1,'ROC')

    t = np.arange(0,5,0.1)
    x = np.exp(-s0*t)
    ax2.plot(t,x)
    ax2.set_xlabel('t')
    ax2.set_ylabel('x(t)')
    ax2.axis('Square')

def plotpzmap_pos(s0):
    fig,(ax1, ax2) = plt.subplots(1, 2)
    fig.set_size_inches(6, 3)
    ax1.plot((-2, 2),(0, 0), lw=1, color='k')
    ax1.plot((0, 0),(-2, 2), lw=1, color='k')
    ax1.plot(s0, 0, 'x', color='C0', markersize=10)
    p = matplotlib.patches.Rectangle((s0,-2), 4, 4, color='gray', alpha=0.15)
    ax1.add_patch(p)
    ax1.axis('Square')
    ax1.set_xlim(-2,2)
    ax1.set_ylim(-2,2)
    ax1.set_xlabel("$\Re(s)$")
    ax1.set_ylabel("$\Im(s)$")
    ax1.set_title("exp("+str(s0)+"t) $\epsilon$(t)")
    ax1.text(1,-1,'ROC')

    t = np.arange(0,5,0.1)
    x = np.exp(+s0*t)
    ax2.plot(t,x)
    ax2.set_xlabel('t')
    ax2.set_ylabel('x(t)')
    ax2.axis('Square')
```

\begin{equation}
\mathcal{L}\{\mathrm{e}^{-s_0 t} \epsilon(t)\} = \frac{1}{s+s_0}\quad \text{ROC}: \Re(s) > \Re(-s_0)
\end{equation}

```python
s0 = 1
plotpzmap_neg(s0)
```

\begin{equation}
\mathcal{L}\{\mathrm{e}^{+s_0 t} \epsilon(t)\} = \frac{1}{s-s_0}\quad \text{ROC}: \Re(s) > \Re(+s_0)
\end{equation}

```python
s0 = -1
plotpzmap_pos(s0)
```

## Copyright

This tutorial is provided as Open Educational Resource (OER), to be found at
https://github.com/spatialaudio/signals-and-systems-exercises
accompanying the OER lecture
https://github.com/spatialaudio/signals-and-systems-lecture.
Both are licensed under a) the Creative Commons Attribution 4.0 International
License for text and graphics and b) the MIT License for source code.
Please attribute material from the tutorial as *Frank Schultz,
Continuous- and Discrete-Time Signals and Systems - A Tutorial Featuring
Computational Examples, University of Rostock* with
``main file, github URL, commit number and/or version tag, year``.