Physics Research and Education Bootcamp - Computational | Episode 1

License: CC BY-NC-SA  https://creativecommons.org/licenses/by-nc-sa/4.0/

Last Modification: Sept. 3, 2021

# ***Preambulo***

Primero carguemos los paquetes de Python necesarios para trabajar.

```python
import numpy as np
import matplotlib.pyplot as plt
from tqdm import tqdm
```

# **Transporte de Neutrones con *Montecarlo***

Hoy aprenderemos como usar el método Montecarlo para estudiar propiedades estadísticas de sistemas con muchas partículas. El procedimiento usado aquí puede ser empleado en otras áreas. Lo importante es tener en mente la clase de conclusiones que pueden obtenerse a través de Montecarlo y su manera de implementación.

# **Teoría**

El problema del transporte de neutrones dentro de un medio es sumamente complejo. Formalmente se puede describir la evolución de un sistema de muchas particulas a traves de la ecuación de Boltzmann, una ecuación diferencial que toma en cuenta el medio en el cual los neutrones se encuentran, su posición, velocidad y momento angular.

\\

Una manera de solucionar este problema es a traves de la aproximación de difusión. En vez de considerar cada párticula de manera independiente, en la ecuación de difusión se aproxima el conjunto de neutrones como un "gas de neutrones". De esta manera se puede describir el comportamiento global del ensamble en términos de propiedades estadísticas como el ***camino libre medio*** o ***longitud de difusión***. Esta ecuación se conoce como la ***Ecuación de Difusión***:
\begin{align}
D\Delta\phi - \Sigma_a\phi = \dfrac{1}{v}\dfrac{d\phi}{dt}
\end{align}
\\

En esta descripción, hablamos de la ***Densidad de Flujo de Neutrones*** ($\phi$), la velocidad media ($v$) , el coeficiente de difusión ($D$) y la sección eficaz macroscópica de absorción ($\Sigma_a$).

\\
Esta ecuación tiene varias soluciones, dependiendo de la geometría y el medio de difusiónn. Hoy nos enfocaremos en medios multiplicadores, es decir, medios en los cuales el número de neutrones aumenta con el tiempo. En este caso, es necesario agregar un término a la ecuaciún para tomar en cuenta la producción de neutrones.

Este mecanismo de producción es la fisión nuclear, y denotamos por $\sigma_f$ la probabilidad de que un neutrón produzca una fisión por unidad de area. Si tomamos en cuenta la densidad del medio, podemos transformar esta cantidad en la sección eficaz macroscópica de fisión ($\Sigma_f$). Si ademas, por cada fisión se producen en promedio ($\nu$) nuevos neutrones:
\begin{align}
D\Delta\phi - \Sigma_a\phi + \nu\Sigma_f\phi = \dfrac{1}{v}\dfrac{d\phi}{dt}
\end{align}
En el caso de criticalidad, el número de neutrones en el sistema se mantiene constante y tenemos:
\begin{align}
D\Delta\phi - \Sigma_a\phi + \nu\Sigma_f\phi = 0
\end{align}
En el caso de un sistema esférico podemos determinar el radio del sistema para que la condición de criticalidad se cumpla:
\begin{align}
R = \dfrac{\pi}{Bg}
\end{align}
con:
\begin{align}
Bg^2 = \dfrac{\nu\Sigma_f - \Sigma_a}{D}
\end{align}

En el caso de que la densidad de neutrones no se mantenga constante (aumente o disminuya) la solucion lamentablemente no es tan sencilla de computar.

Podemos sin embargo, hablar de la cantidad $K_\mathrm{eff}$ que nos permite cuantificar la diferencia en el número de neutrones entre una generación y otra.

\begin{align}
K_\mathrm{eff} = \frac{\mathrm{numero\,\, de\,\, neutrones \,\,actual}}{\mathrm{numero\,\,de\,\,neutrones\,\,previo}}
\end{align}

De esta manera si:

* $K_\mathrm{eff}$ > 1 : hablamos de un sistema supercrítico, la densidad de neutrones aumenta exponencialmente
* $K_\mathrm{eff}$ = 1 : hablamos de un sistema crítico, la reacción es autosostenible y la cantidad de neutrones permance constante
* $K_\mathrm{eff}$ < 1 : hablamos de un sistema subcrítico, la densidad de neutrones disminuye con el tiempo

\\

Verificar si un sistema dado es crítico o supercrítico no es trivial. Para esto podemos utilizar el metodo Montecarlo y asi estimar esta proporción de neutrones de una generación a otra.

\\

Nota: para el caso que vamos a estudiar:
*  $\Sigma_a$ es la sección eficaz macroscópica de absorción y se refiere a la probabilidad total de que el neutrón sea absorbido y produzca fisión o simplemente active el núcleo que lo recibe.
* $\Sigma_f$ es la seccion eficaz macroscópica de fisión y se refiere a la probabilidad de que el neutrón absorbido produzca fisión
* $\Sigma_t$ es la probabilidad total de interacción, en el contexto de Difusión podemos entenderlo como el camino libre medio de un neutrón
* $\Sigma_s$ es la probabilidad de que el neutrón que interacciona sea desviado

\\

Es conveniente tener en mente que:

\begin{align}
\Sigma_t = \Sigma_s + \Sigma_a
\end{align}

Las unidades de los $\Sigma_j$ son de $cm^{-1}$

# Práctica

Necesitamos un código que nos genere inicialmente neutrones, los mueva a traves del medio y determine si despues de un movimiento ocurre una fisión, absorción o scattering.

```python
def generar_neutrones(N_neutrones,radio):

    #Inicializamos las posiciones iniciales de los neutrones dentro de la esfera
    # Generamos dos coordenadas (r,theta) y luego a traves de las propiedades trigonometricas
    # luego convertimos estas coordenadas polares a cartesianas.
    r0 = np.random.uniform(0,radio,size=N_neutrones)
    theta0 = np.random.uniform(0,2*np.pi-.01,size=N_neutrones)
    x0 = r0*np.cos(theta0)
    y0 = r0*np.sin(theta0)
    # Guardamos las posiciones iniciales dentro de un arreglo para uso posterior
    fission_sites = np.array([x0,y0]).T
    return fission_sites
r = 2
Posiciones = generar_neutrones(1000,r)

plt.figure()
plt.suptitle('Distribucion de puntos iniciales de neutrones',fontweight='bold')
plt.scatter(Posiciones[:,0],Posiciones[:,1])
plt.xlabel('x',fontweight='bold')
plt.ylabel('y',fontweight='bold')
plt.xlim(-r-1,r+1)
plt.ylim(-r-1,r+1)
plt.grid()
```

Durante el transporte de un neutrón, este puede ser absorbido, desviado o ocasionar una fisión. Si el sistema es supercrítico el programa colapsará debido a límites de memoria ya que en cada ciclo de transporte habra más y más neutrones que seguir. Para evitar esto asignaremos pesos a los neutrones, con los cuales simbolizaremos el aumento o disminución de la cantidad de neutrones.

\\

Al final de cada ciclo, contamos el número de fisiones producidas y la posición de interacción. En el siguiente ciclo iniciamos nuevos neutrones desde estos punto de interacción y seguimos su evolución. Tomando el cociente entre los pesos de los neutrones de un ciclo entre el ciclo previo podemos obtener $K_\mathrm{eff}$

\\

Necesitamos ahora un código que mueva un neutrón aleatoreamente por el espacio:

Para el transporte debemos considerar $\Sigma_a$, $\Sigma_t$, $\Sigma_f$ y $\nu$

```python
def random_transport(x,y,Sigma_t,niter):
  pos = [[x,y]]
  for i in range(niter):
    new_r = -np.log(1-np.random.random())/Sigma_t # Ya que tenemos solo una distancia promedio necesitamos muestrear las distancias
    new_theta = np.random.uniform(0,2*np.pi-.01) # Muestramos aleatoriamente un angulo
    x += new_r*np.cos(new_theta)
    y += new_r*np.sin(new_theta)
    pos.append([x,y])
  return pos

Posiciones = random_transport(np.random.random(),np.random.random(),1,100)
Posiciones = np.array(Posiciones)
plt.figure()
plt.suptitle('Transporte aleatorio de un neutron',fontweight='bold')
plt.plot(Posiciones[:,0],Posiciones[:,1])
plt.scatter(Posiciones[0,0],Posiciones[0,1],label='principio',s=40,c='g')
plt.scatter(Posiciones[-1,0],Posiciones[-1,1],label='final',s=40,c='r')
plt.xlabel('x',fontweight='bold')
plt.ylabel('y',fontweight='bold')
plt.grid()
plt.legend()

```

Ahora necesitamos una lógica para determinar si el neutrón interactúa o no.
Para esto hacemos uso de las secciones eficaces macroscópicas.

\\
Como hemos visto, para generar muestras de una distribución debemos invertir la ***funcion de densidad cumulativa***. En casos como el nuestro en los cuales esta función no es conocida, tenemos que usar otros métodos para generar muestras.

El mecanismo que usaremos es el ***rejection sampling***. En terminos sencillos consiste en generar un número aleatorio y compararlo a la probabilidad de que ocurra algun evento. Si la probabilidad es menor, el evento ocurre, si es mayor el evento no ocurre y se procede a evaluar si otro evento distinto ocurrió mediante el mismo mecanismo.

\\
\begin{align}
\alpha \in U[0,1] \rightarrow p(\alpha) \begin{cases}
1    &\text{si } \alpha \leq \dfrac{p_\mathrm{evento}}{p_\mathrm{total}} \\
0    &\text{si } \alpha > \dfrac{p_\mathrm{evento}}{p_\mathrm{total}}\\
\end{cases}
\end{align}

\\

Ahora escribimos un código que tome esto en cuenta, lo usaremos para estudiar si durante el trayecto el neutrón colisiona y es desviado, es absorbido o produce fisión.

\\

```python
def random_transport(x,y,Sigma_t,Sigma_a,Sigma_f,niter):
  # Calculamos la seccion eficaz macroscopica de desvio
  Sigma_s = Sigma_t - Sigma_a
  Pos = [[x,y]]
  alive = 1 # Usamos esta variable como un flag para determinar si debemos continuar siguiendo la evolucion de la particula
  Fision = 0
  while alive == 1:
    for i in range(niter):
      new_r = -np.log(1-np.random.random())/Sigma_t # Ya que tenemos solo una distancia promedio necesitamos muestrear las distancias
      new_theta = np.random.uniform(0,2*np.pi-.01) # Muestramos aleatoriamente un angulo
      x += new_r*np.cos(new_theta)
      y += new_r*np.sin(new_theta)
      Pos.append([x,y])

      # Verificamos si ocurre una interaccion:
      coll_prob= np.random.random()
      if coll_prob <= Sigma_s/Sigma_t: # La fraccion de la probabidad total de interaccion que ocurra un desvio
        continue # No debemos cambiar nada, proseguimos con la siguiente iteracion.
      else:
        # No hubo desvio (scattering), estudiamos ahora si hubo fision o simple absorcion
        fiss_prob = np.random.random()
        alive = 0
        if fiss_prob <= Sigma_f/Sigma_a:
          Fision = 1
          print(f'Ha ocurrido una fision tras {i+1} iteraciones')
          return Pos
        else:
          print(f'El neutron ha sido absorbido tras {i+1} iteraciones')
          return Pos
  return Pos
```

```python
Posiciones = random_transport(np.random.random(),np.random.random(),1,0.3,0.1,100)
Posiciones = np.array(Posiciones)
plt.figure()
plt.suptitle('Transporte aleatorio de un neutron',fontweight='bold')
plt.plot(Posiciones[:,0],Posiciones[:,1])
plt.scatter(Posiciones[0,0],Posiciones[0,1],label='principio',s=40,c='g')
plt.scatter(Posiciones[-1,0],Posiciones[-1,1],label='final',s=40,c='r')
plt.xlabel('x',fontweight='bold')
plt.ylabel('y',fontweight='bold')
plt.grid()
plt.legend()

```

Solo nos falta crear un loop grande que genere ciclos de transporte con los neutrones

\\

El algoritmo que queremos debe:

1.   Generar un grupo de N neutrones distribuidos al azar en el espacio
2.   Seguir la evolución de cada neutron
3.   Al final del ciclo tomar nota de los lugares del espacio donde hubo una interacción y ajustar el vector de pesos de manera proporcional al número de fisiones occuridas.
4.   Comenzar el siguiente ciclo g con N neutrones distribuidos entre los puntos de interacción del ciclo previo.

 Nota:

Como todo algoritmo numérico, es probable que el valor estimado de $K_\mathrm{eff}$ varíe de un ciclo a otro. Ademas, ya que no tenemos la CDF subyacente de las interacciones y la aproximamos a traves de ***rejection sampling***, es necesario realizar varias iteraciones hasta que la distribución de probabilidad de interacción se vuelva estacionaria y converja asintóticamente a la distribución real. Para esto consideramos un número de ciclos extras, los cuales no consideraremos válidos para estimar $K_\mathrm{eff}$

(Si has visto Cadenas de Markov, esta noción te parecera familiar a la del ***burn in*** de la traza)

### Estimando K_eff

Reordenando y combinando lo que hemos escrito:

```python
def homog_sphere_k(N,Sig_t,Sig_f,Sig_a,nu,radio,inactive_cycles = 5, active_cycles = 20):

    Sig_s = Sig_t-Sig_a # Calculamos la seccion eficaz macroscopica de desvio

    #Inicializamos las posiciones iniciales de los neutrones dentro de la esfera
    # Generamos dos coordenadas (r,theta) y luego a traves de las propiedades trigonometricas
    # luego convertimos estas coordenadas polares a cartesianas.
    r0 = np.random.uniform(0,radio,size=N)
    theta0 = np.random.uniform(0,2*np.pi-.01,size=N)
    x0 = r0*np.cos(theta0)
    y0 = r0*np.sin(theta0)
    # Guardamos las posiciones iniciales dentro de un arreglo para uso posterior
    fission_sites = np.array([x0,y0]).T
    posiciones = fission_sites.copy()
    # Inicializamos un vector de pesos asignando el numero medio de neutrones por fision
    weights = nu*np.ones(N)
    old_gen = np.sum(weights)
    k = np.zeros(inactive_cycles+active_cycles)

    Fs = {}

    for cycle in tqdm(range(inactive_cycles+active_cycles)):
        fission_sites = np.empty((1,2)) # Creamos un array vacio 1x2
        fission_site_weights = np.empty(1) # Creamos un array vacio 1x1
        for neut in range(weights.size):
            #Tomamos un neutron de la lista de neutrones
            posicion = posiciones[neut] # posicion del neutron
            weight = weights[neut] # peso asociado
            alive = 1 # Indicamos que esta 'vivo' y hay que seguir su evolucion
            while (alive):
                # Muestramos la distancia a la proxima colision
                new_r = -np.log(1-np.random.random())/Sig_t # Ya que tenemos solo una distancia promedio necesitamos muestrear las distancias
                new_theta = np.random.uniform(0,2*np.pi-.01) # Muestramos aleatoriamente un angulo
                # Actualizamos la posicion del neutron
                posicion[0] += new_r*np.cos(new_theta)
                posicion[1] += new_r*np.sin(new_theta)
                # Verificamos si aun estamos dentro de la esfera
                if (posicion[0]**2 + posicion[1]**2) > radio**2:
                    alive = 0 # El neutron ha salido de la esfera, dejamos de seguirlo
                else:
                    # Tomamos una muestra de la probabilidad de interaccion
                    coll_prob = np.random.random()
                    if (coll_prob < Sig_s/Sig_t):
                      # Si la probabilidad es menor que la probabilidad de desvio el neutron no interactuo,
                      # recalculamos la distancia hasta la siguiente colision
                        continue
                    else:
                      # Si el neutron colisiono, pudo haber ocasinado una fision o ser absorvido
                      # para esto tomamos una muestra de la probabilidad de fision
                        fiss_prob = np.random.random()
                        alive = 0 #
                        if (fiss_prob <= Sig_f/Sig_a):
                          # Si la probabilidad de fision es menor o igual al cociente
                          # entre las secciones eficaces macroscopicas, consideramos que ocurre una fission
                          # en la posicion actual

                            # agregamos la posicion actual al arreglo de los lugares de fision
                            fission_sites = np.vstack((fission_sites,posicion.reshape(-1,1).T))
                            # agregamos el peso de la particula correspondiente al arreglo de pesos
                            fission_site_weights = np.append(fission_site_weights,weight)
                          # En caso de no haber fision, el neutron es absorbido. En nuestra approximacion
                          # no hacemos nada, sin embargo para mayor fidelidad, deberiamos actualizar la composicion
                          # atomica del medio y ajustar las correspondientes secciones eficaces.

        # Finalizamos un ciclo

        # Borramos la primera entrada del arreglo ya que contiene datos dummy
        fission_sites = np.delete(fission_sites,0,axis=0)
        # Realizamos el mismo procedimiento para este otro arreglo
        fission_site_weights = np.delete(fission_site_weights,0,axis=0)
        Fs[str(cycle)] = fission_sites
        # Determinamos el numero de neutrones a inicializar en el proximo ciclo
        # desde los puntos de interaccion determinados en el ciclo anterior
        num_per_site = int(np.ceil(N/fission_sites.shape[0]))
        # Reiniciamos el vector de posiciones de las particulas:
        posiciones = np.empty((1,2))
        weights = np.empty(1)
        # Creamos los nuevos arreglos de posiciones y pesos para usarlos en el proximo ciclo
        for site in range(fission_sites.shape[0]):
            site_pos = fission_sites[site]
            site_weight = fission_site_weights[site]
            posiciones = np.vstack((posiciones,
            site_pos*np.ones((num_per_site,1))))
            weights = np.append(weights,
            site_weight * nu/num_per_site*np.ones((num_per_site,1)))
        # Borramos la primera entrada de los arreglos al igual que hicimos arriba
        posiciones = np.delete(posiciones,0,axis=0)
        weights = np.delete(weights,0,axis=0)
        # Estimamos el numero de neutrones producidos en la generacion actual
        new_gen = np.sum(weights)
        # Calculamos el valor de Keff correspondiente para el ciclo actual
        k[cycle] = new_gen/old_gen
        old_gen = new_gen
    return k, Fs
```

Ahora aplicamos nuestro cídigo a una esfera de radio 3cm usando los siguientes parámetros:

* $\Sigma_t = $ `0.3382197087866109`
* $\Sigma_f = $ `0.0920156560669456`
* $\Sigma_a = $ `0.1049475861087866`
* $\nu = $ `2.98`
* $N$ = `50e3` neutrones por ciclo
* Ciclos Activos = `100`
* Ciclos Inactivos = `10`

```python
St = 0.3382197087866109
Sf = 0.0920156560669456
Sa = 0.1049475861087866
nu = 2.98
r = 3
Nciclos_act = 100
Nciclos_inact = 10
N_neutrones = 50000

K, Fs = homog_sphere_k(N_neutrones,St,Sf,Sa,nu,r,Nciclos_inact,Nciclos_act)
Kmean = np.mean(K[Nciclos_inact:])
Kstd = np.std(K[Nciclos_inact:])
print('Keff = {:.4f} +/- {:.4f}'.format(Kmean,Kstd))
```

    100%|██████████| 110/110 [05:42<00:00,  3.12s/it]

    Keff = 0.6910 +/- 0.0058

```python
plt.figure()
plt.suptitle(r'Evolucion de $K_{eff}$ vs Ciclo de Transporte',fontweight='bold')
plt.plot(np.arange(Nciclos_act+Nciclos_inact),K)
plt.ylabel(r'K_${eff}$',fontweight='bold')
plt.xlabel('Ciclo',fontweight='bold')
plt.grid()

```

# Tarea

* Determine el radio crítico de una esfera de $^{239}\mathrm{Pu}$. Cual es la masa crítica?

* Modifique el cídigo para considerar ahora una esfera hueca con radio interior $R_{in}$ y radio exterior $R_{ext}$. Determine valores de estos radios para que el sistema sea crítico a densidad normal.

Version 1.0, Autor : Antonio Figueroa

```python
#libs summary
import types
def imports():
    for name, val in globals().items():
        if isinstance(val, types.ModuleType):
            yield val.__name__
list(imports())
```

    ['builtins',
     'builtins',
     'IPython.core.shadowns',
     'numpy',
     'matplotlib.pyplot',
     'types']


```python
%matplotlib notebook
import matplotlib.pyplot as plt
import numpy as np
import sympy as sp
from plots import *

sp.init_printing()
freqs = [f for f in np.random.standard_cauchy(11) if abs(f) < 10]
omega = [2+ f for f in freqs] + [1 - f for f in freqs] + [1]

from BondGraphTools import version
import BondGraphTools as bgt
assert version == "0.3.7"
scale = 2

from matplotlib.font_manager import FontProperties

def plot_graph(t, x):
    fontP = FontProperties()
    fontP.set_size('small')
    fig = plt.figure(figsize=(scale*4,scale*4))
    plt.plot(t,x)
    ax = fig.gca()
    ax.set_xlabel('t')
    ax.set_title(f"System response to {impulse}")
    ax.legend(
        [f"$x_{i}$" for i in range(len(x))],
        bbox_to_anchor=(1.,1.),
        loc=1,
        borderaxespad=0.,
        prop=fontP
    )
    return fig

def print_tree(bond_graph, pre=""):
    print(f"{pre}{bond_graph}")
    try:
        for component in reversed(bond_graph.components):
            if pre == "": print_tree(component, pre +"|-" )
            else: print_tree(component, pre +"-" )
    except AttributeError:
        pass
```

```python

```

TODO:
2. Add critical values for Kuramoto model.
3. Add critical values for Nonlinear oscillator model.

- Introduction to complex systems
- Basic research question

Lit Points:
- Emergence is low dimension dynamics in high dimensional space. (Should be a geometric description; but we're not there yet.)
- Collective motion requires tradeoffs between coulping strength and population heterogeneity. (Framed in terms of some kind of averaging)

# On Emergence in Complex Physical Systems

https://github.com/peter-cudmore

    Dr. Peter Cudmore.
    Systems Biology Labratory,
    The School of Chemical and Biomedical Engineering,
    The University of Melbourne.

Many problems in biology, physics and engineering involve predicting and controlling complex systems, loosely defined as interconnected system-of-systems. Such systems can exhibit a variety of interesting non-equilibrium features such as emergence and phase transitions, which result from mutual interactions between nonlinear subsystems.

Modelling these systems is a task in-and-of itself, as systems can span many physical domains and evolve on multiple time scales. Nonetheless, one wishes to analyse the geometry of these models and relate both qualitative and quantitative insights back to the physical system.

Beginning with the modelling and analysis of a coupled optomechanical systems, this talk presents some recent results concerning the existence and stability of emergent oscillations. This forms the basis for a discussion of new directions in symbolic computational techniques for complex physical systems as a means to discuss emergence more generally.

## The problem with big systems is that they're _big_...

## Example: Many-body Quantum Optomechanics

(Image courtesy of Knap, M. https://users.ph.tum.de/ga32pex/ )

(Image courtesy of Marquardt, F.
https://photons-and-matter.org/research/ )

## Example: Human Metabolism

(Image courtesy of Human Metabolism map https://www.vmh.life )

# Example: Ecosystems

## Complex Physical Systems

A dynamical system is said to be a _complex physical system_ when:
* It is made up of many _interacting_ parts, or subsystems (High-dimensional).
* The subsystems are not all of the same (Heterogenenous).
* The subsystems are complicated (Nonlinear and/or Noisy).
* There are well defined boundaries between the subsystems (Network Topology).
* **Coupling takes place via resource exchange (Conservation Laws).**

## Complex Systems can exhibit _emergence_.

- _Emergence_ is a phenomenom where the system displays novel new behaviour that could not be produced by individuals alone.
- _Synchronisation_ is the most studied example of emergence, and can occur in systems of coupled oscillator.

 How can one predict and control emergent phenomenon?

How can nonlinear dynamics be "scaled up"?

## Outline of this talk

In this talk we will:

Briefly discuss synchronisation as it's the best example of emergence.

Discuss how we might generalise this for energetic systems.

## Part 1: Synchronised Oscillators

## The Kuramoto Model

_Self-entrainment of a population of coupled non-linear oscillators_ Kuramoto, Y. (1975).

The phase $\theta_j$ of each oscillator with a natural frequency $\omega_j$ is given by

\begin{equation}
\dot{\theta}_j =  \omega_j + \frac{K}{n}\sum_{k=1}^n\sin(\theta_k - \theta_j),\qquad j=1,\ldots n
\end{equation}
- When $0\le K <K_c $ each oscillator rotates at their own frequency.

- A Hopf bifurcation occurs at $K=K_c$ creating collective motion.

- When $K>K_c$ more oscillator are recruited to collective.

The value of $K_c$ depends upon the distribution of $\{\omega_j\}$. For symmetric distribtuions we have
$$K_c = \frac{2}{\pi g(0)}$$

```python
# Omega = Cauchy(2,1) so that K_c = 2
p = KuramotoModel(omega=omega, scale=scale)
plt.show()
```

    <IPython.core.display.Javascript object>

Points:
- Wiener -> Winfree -> Kuramoto
- Comes from studying BZ reaction
- Motion on a strongly attracive limit cycle (invariant manifold) such that coupling
- All-to-all coupling on a complete graph.
- sinusoidal in phase -> linear in complex co-ordinates.
- Kuramoto showed that at $K_c=2$ a Hopf bifurcation creates a synchronised state, that becomes progressive more stable as $K_c$ increases.

## The Kuramoto Model (Cont.)

Kuramoto introduced an 'order parameter' $r$ to measure phase coherence
\begin{equation}
z = r\mathrm{e}^{\mathrm{i}\Theta} = \frac{1}{n}\sum_{k=1}^n \exp{\mathrm{i}\theta_k} \implies r = \frac{1}{n}\sum_{k=1}^n \exp\mathrm{i}(\theta_k-\Theta)
\end{equation}
It follows that
$$
\Im\left[\frac{1}{n}\sum_{k=1}^n\exp i(\theta_k - \theta_j)\right] =
\Im\left[r\exp i(\Theta - \theta_j)\right]
$$

Hence
$$
\dot{\theta}_j =  \omega_j + \frac{K}{n}\sum_{k=1}^n\sin(\theta_k - \theta_j)$$

becomes
$$
\dot{\theta}_j =  \omega_j + rK\sin(\Theta - \theta_j).
$$

```python
p = KuramotoOrderModel(omega,scale=scale)
plt.show()
```

    <IPython.core.display.Javascript object>

Points:
- Mean phase is a kind of coordinate for the synchronous manifold.
- Weak interactions with entire populations <=> strong coupling to collective statistics
- Feedback look means that if coupling increases coherence, then $r$ increases asymptotically to $r_\infty = \sqrt{1-K_c/K}$$.

## The Status of the Kuramoto Model

$$
\dot{\theta}_j =  \omega_j + rK\sin(\Theta - \theta_j),\qquad j = 1,\ldots n.\qquad r = \frac{1}{n}\sum_{k=1}^n \exp i (\theta_k - \Theta).
$$

- Identical oscillators evolve on a 3 dimensional manifold (Watanabe and Strogatz, Physica D 1994. Ott and Antonsen, Chaos 2008).
- Heterogenous oscillator dynamics represented in terms of collective co-ordinates in the thermodynamic limit (Pikovsky and Rosenblum, Physica D, 2011) and for finite-n (Gottwald, Chaos 2015).
- Active research into applications in biology (particuarly neuroscience), physics and chemsitry.
- Extensions to noisy, graph coupled and with various different coupling mechanisms.
- Very few global results for heterogenous oscillators (Dietert, J. Math. Pures Appl. 2016).
- _No results as yet for geometrical interpretation of transtion to synchrony._

## Some Takeaways

$$
\dot{\theta}_j =  \omega_j + rK\sin(\Theta - \theta_j),\qquad j = 1,\ldots n.\qquad r = \frac{1}{n}\sum_{k=1}^n \exp i (\theta_k - \Theta).
$$

1. When thinking about emergence, we want to think about mutual coupling between population statistics and individuals.

2. This means that, for a given system, we need to understand both the individual dynamics _and_ population level dynamics.

# Part 2: Coupled Optomechanical Systems

## Physics of Many-Body Quantum Optomechanics

The Quantum Hamiltonain for the system is
$$
H = H_\text{cavity}+H_\text{drive}+H_\text{beams}
+H_\text{coupling} + \mathcal{H}_\text{diss}
$$
where
$$
\begin{align}
H_\text{cavity} =&\ \frac{\omega_c}{2} a^\dagger a\\
H_\text{drive} =&\ u^\dagger a + a^\dagger u\\
H_\text{beams} =&\ \sum_j \frac{\omega_j}{2}b_j^\dagger b\\
H_\text{coupling} =&\ \sum_j \frac{G}{4}
(b_j+b^\dagger_j)a^\dagger a\\
\end{align}
$$
where $a^\dagger,a ,b_j^\dagger,b_j,u^\dagger,u$ are the creation and annihilaton operators of the
optical mode, vibrational modes and optical forcing respectively.

Also:
- $\omega_c$ is the resonant frequency of the optical cavity
- $\omega_j$ is the mechanical resonance of the $j$th beam
- $G$ is the strengh of opto-mechanical coupling
- $\mathcal{H}_\text{diss}$ is the 'dissipative structure'...

## From Quantum Physics to Semiclassical Dynamics

1. Derive/produce the (generalised) Langevin equations.
2. Use the semiclassical approximation to replace operators with averages, so that $a \rightarrow \alpha$ and $a^\dagger \rightarrow \alpha^*$
3. Assume the optical forcing is $\left = \nu\exp[i\omega_ft]$ and define $\delta =\omega_c - \omega_f$
4. Perform some rescaling and non-dimensionlisation to get
$$
\begin{align}
\dot{\alpha}&= - (1+i\delta)\alpha - i\alpha \frac{1}{n}\sum_{j=1}^n\Re \left[\beta_j\right] -i\nu \\
\dot{\beta_j} +  i\bar{\omega}\beta_j &= -\gamma \left[(1+i\omega'_j)\beta_j + i\frac{G}{2}|\alpha|^2\right], \quad j=1,\ldots, n
\end{align}
$$
where $\gamma \ll1$ and $\omega_j = \bar{\omega} + \gamma\omega_j'$
5. Apply the 'method of multiple scales' with $\tau = \gamma t$ to get $\beta(t,
\tau) = z(\tau)\exp[-i\bar{\omega}t]$.
6. Notice that $\Re[\beta] = \cos(\omega t +\phi)$ makes $\alpha$ look like a F.M oscillator, so has a basis in terms of Bessel functions.
7. More algebra...

...how much of this can be automated?

## On the other side

The fast time system (for the cavity $\alpha$):

$$
\frac{d\alpha}{dt} =
-\alpha  - i(\delta + r(\tau)\cos[\bar{\omega}t + \Theta(\tau)])\alpha - i\nu
$$

The slow time system (coupling induced frequency shift $z_j$):

$$
\frac{d z_j}{d\tau}
= -(1- i\omega'_j)z_j + zF(|z|), \qquad z = re^{i\Theta} = \frac{1}{n}\sum_{j=1}^n z_j
$$

With $F:[0,\infty) \rightarrow \mathbb{C}$ a complicated but known and computable function.

## Collective motion on the slow time scale

**Theorem (PC & Holmes. 2015)**
Define the _dispersion function_
$$
f(\mu) = \int_{-\infty}^\infty \frac{g(\omega)d\omega}{\mu - i\omega}.
$$

(1) For sufficiently large $n$, the system
$$\dot{z}_j
= -(1- i\omega_j)z_j + zF(|z|), \qquad z = \frac{1}{n}\sum_{j=1}^n z_j$$
has coherent motion on (possibly many multi-stable) invariant manifold(s) characterised by
$$
z = re^{i\Theta},\quad \dot{r} = 0, \qquad \dot{\Theta} = \Omega
$$
and
$$r\left[F(r) - \frac{1}{f(1+i\Omega)}\right] = 0.$$

(2) For symmetric $g$ solutions are strictly unsable iff

$$
\left\langle\frac{dF}{dr},\frac{d}{d\mu}\frac{1}{f(\mu)}\bigg|_{1+i\Omega}\right\rangle >0,
$$
where $\langle x,y\rangle = \Re[xy^*]$ is the geometric inner product for $x,y\in \mathbb{C}$.

```python
# F(r) = K/(1+r^2) is stable iff K > 2
m = DampedCoupledOsc(omega=omega, extent=4, scale=scale)
plt.show()
```

    <IPython.core.display.Javascript object>

Notes:
- Only collective motion: S^1.
- Multistability which can be created and destroyed via Hopf's.
- Results for general coupling function $F$.
- Proof involves treating $1/f(\mu)$ as a conformal map from the RHP.

## Synch corresponds to frequency modulation on the fast time scale

On the fast time scale:

$$\frac{d\alpha}{dt} =
-\alpha  - i(\delta + r(\tau)\cos[\bar{\omega}t + \Omega\tau])\alpha - i\nu
$$

If we can measure $(r,\Omega)$, we can determine the internal state.

Here $\alpha$ falls well into the realm of existing signal processing theory (in particular frequency modulation). Hence, we can associate spectral properties to internal state.

(The maths is hideous though)

## Some Observations

# Object Oriented Modelling for Complex Physical Systems

### Inheritance, Composition and Encapsulation

## Ad-hoc modelling.

1. Use descriptions of physical processes plus network topology to generate some odes.
2. Do a whole bunch of algebra.
3. Work out the appropriate coordinates to use based on geometric features (here, a non-standard slow fast system)
4. More algebra to reduce model.
5. Investigate the dynamics of the reduced models.
6. Relate results in reduced model to observables in the original system

In the case of emergent phenomenon, the 'reduced' subspace involves the whole (or at least a large part of) system. E.g. mean fields.

## Ad-hoc approaches won't scale.

As an example:
- individual processes are far more heterogenous
- network topolgy is complicated
- many parameters are unknown
- almost guaranteed to be a differential-algebraic system
- **too big for one person, or even one lab**

We must have:
- Ways to respresent and manipulate such systems,
- Ways to manage congnitive complexity,
- Ways to automate model capture and reduction,
- Ways to effective share work between researchers

## Energy provides an interface.

## The Structure of Complex Physical Systems

An approach based on 'bond graph' modelling, and port-Hamiltonian systems.

- Energy is stored in 'state variables' $q,p$
- Power is distributed via 'power variables' $e,f$
- Formally describes the hyrdo-mechanical-electrical analogies.

For example;
- Dissipation relates $e,f$ variables. (eg. Ohm's law, friction, etc)
- Potential storage $q$ to $e$ (eg, capacitors, gravity)

    Domain
    $q$
     $f = \dot{q}$
     $p$
    $e = \dot{p}$

     Translational Mechanics
     position
     velocity
     momentum
     force

Rotational Mechanics
     angle
     angular velocity
     angular momentum
     torque

Electronics
     charge
     current
     flux linkage
     voltage

Hydraulics
     volume
     flow
     pressure momentum
     pressure

     Thermodynamics
     entropy
     entropy flow
     temperature momentum
     temperature

    Chemistry
     moles
     molar flow

     chemical potential

## An Object Oriented Representation of Energetic Systems

Object Oriented Programming (OOP) is a software development paradigm that seeks to manage large, complicated projects by breaking problems into _data_ plus _methods_ that act on the data.

Three big ideas in OOP are:
1. _Inheritance_ or is-a relationships.
2. _Composition_ or has-a relationships.
3. _Encapsulation_ or infomation hiding.

This allows for _hierarchical_ and _modular_ design which reduces model complexity.

'Energetic systems' draws from:
- Network based analysis from engineering; in particular circuit electrical analysis and the more general (and less well known) bond graph methodology,
- Classical mechanics, and in particular recent advances in port based Hamiltonian mechanics,
- Modern nonlinear dynamics,
- The effective was of managing complexity within software engineering.

## Inheritance

For networked dynamic systems, _inheritance_ means we have:
- conditions on the dynamical sub-systems.
- a description of the interface between nodes.

### Definition (Energetic System)

An energetic system is a tuple $(M, \mathcal{D}, U,\Phi)$
where the
* *state space* $M$ is a manifold of $\dim(M) = m\ge 0$
* *port space* $\mathcal{D} \subset \mathcal{F} \times \mathcal{E}$ where, $\mathcal{E} = \mathcal{F}^*$ and $ \dim{\mathcal{D}} = \mathcal{F}|_\mathcal{D} =n$.
* *control space* $U \subset C^r:\mathbb{R}_+ \rightarrow \mathbb{R^k}$ with $k\ge 0$
* *constitutive relation* is a smooth map $\Phi: TM \times \mathcal{D} \times U\times\mathbb{R}_+ \rightarrow
  \mathbb{R}^{m+n}$
    such that
  $$\Phi\left(\frac{dx}{dt},x,f,e,u,t\right)=0.$$

$\Phi$ relates the _internal state_ $M$ and the _external environment_ (via $\mathcal{D}$).

One can show that for conservative systems, one can define a storage function $H(x)$, choose $$\Phi(\dot{x}, x,f,e,t) =
\left(\begin{matrix}
\dot{x} - f\\
e - \nabla_x H(x)
\end{matrix}\right) = 0$$
and encode the appropriate sympectic structure in a linear subspace of $\mathcal{D}$.

The incoming *power* is $P_\text{in} = \left<e,f\right>$ for $(f,e)\in \mathcal{D}$

## Inheritance

For energetic systems-of-systems:

### Nodes are particular _energetic systems_
Each node is described by a set of differential-algebraic equations $\Phi(\dot{x},x,e,f) = 0$.

### Edges are constraints on port variables.

An edge represents how state is shared between systems.

## Composition

For networked dynamic systems _composition_ means that we can replace nodes with subgraphs and vice-versa.

## Corollary (Composition)
If $\Psi_1 = (M_1, \mathcal{D}_1, U_1,\Phi_1)$ and $\Psi_2 = (M_2, \mathcal{D}_2, U_2,\Phi_2)$  are energetic systems, then

$$\begin{eqnarray}\Psi_0 &=& \Psi_1 \oplus\Psi_2\\
&=&
\left(M_1\oplus M_2,\mathcal{D}_1 \oplus\mathcal{D}_2,U_1\oplus U_2, \Phi_1\oplus\Phi_2\right)
\end{eqnarray}$$
is also an energetic system.

Suppose (abusing notation) $\Psi_0 = (\Psi_1,\Psi_2)$ is an energetic system with ports

$$(e_i, f_i) \in \mathcal{D}_1, \quad (e_j,f_j)  \in \mathcal{D}_2$$

Then $\Phi_0$ with the additional power conserving constraint

$$e_i - e_j = 0\qquad f_i+f_j=0$$

is also a energetic system

## Encapsulation

For a networked dynamical system _encapsulation_ means that we can apply simplification methods to a subgraph so that the replacement system is less complicated, while representing the same behaviour.

One can also go the other way by replacing a node with a more complicated subgraph.

## Object Oriented Modelling and Energetic Systems

Energetic systems provide:
- _Inheritance_; an abstract base representation of energetic systems.
- _Composition_; a way to hierarchically compose systems of systems.
- _Encapsulation_; a framework inside which simplifications can occur.

#  Introducing `BondGraphTools`

## `BondGraphTools` a `python` library for energetic systems.

`BondGraphTools` (https://github.com/BondGraphTools) a framework for modelling energetic systems.
* Based upon an extension of bond graph and port-Hamiltonian modelling.
* Provies a simple, *minimal* object-oriented interface for constructing models.
* Implemented in `python` and uses the standard `scipy` stack.
* Performs symbolic model reduction and simulation.
* Simulations with DAE solvers in `julia`.
* Developed with sustainable software practices.
* Intended to be used in _conjunction_ with other tools.

'Bond Graphs' are a multi-domain port-based graphical modelling technique used predominantly in mechatronics.
Port-Hamiltonian systems integrate geometric approaches from classical mechanics and control theory with port based modelling.

## Example: Linear Oscillator

```python
class Linear_Osc(bgt.BondGraph):
    damping_rate = 0.1 #D amping rate common across oscillator array

    def __init__(self, freq, index):
        """Linear Oscillator Class

        Args:
            freq:  Natural (undamped) frequency of this oscillator
            index: Oscillator number (used for naming).

        Instances of this class are bond graph models of externally forced
        damped harmonic oscillators.
        In the electrical analogy, these is simply an open loop series RLC
        circuit."""

        # Create the components
        r = bgt.new("R", name="R", value=self.damping_rate)
        l = bgt.new("I", name="L", value=1/freq)
        c = bgt.new("C", name="C", value=1/freq)
        port = bgt.new("SS")
        conservation_law = bgt.new("1")

        # Create the composite model and add the components
        super().__init__(
            name=f"Osc_{index}",
            components=(r, l, c, port, conservation_law)
        )

        # Wire the model up
        for component in (r,l,c):
            bgt.connect(conservation_law, component)
        bgt.connect(port, conservation_law)

        # Expose the SS component as an external port
        bgt.expose(port, label="P_in")
```

`Linear_Osc`
- _inherits_ from BondGraph, which is a base 'class' containing much of functionality
- is _composed_ of a variety of subcomponents
- _encapsulates_ a one port RLC component.

```python
example_osc = Linear_Osc(1000,1)
example_osc.constitutive_relations
```

# Example: Using Linear oscillators for Coupled Cavity

```python
def coupled_cavity():
    model = bgt.new(name="Cavity Model")

    # Define the interaction Hamiltonain
    coupling_args = {
        "hamiltonian":"(w + G*x_1)*(x_2^2 + x_3^2)/2",
        "params": {"G": 1, "w": 6}
    }

    port_hamiltonian = bgt.new("PH", value=coupling_args)

    # Define the symplectic junction structure
    symplectic_gyrator = bgt.new("GY", value=-1)
    em_field = bgt.new("1")

    bgt.add(model, port_hamiltonian, symplectic_gyrator, em_field)
    bgt.connect(em_field, (port_hamiltonian, 1))
    bgt.connect(em_field, (symplectic_gyrator, 1))
    bgt.connect((port_hamiltonian, 2), (symplectic_gyrator, 0))

    # Construct the open part of the system
    dissipation = bgt.new("R", value=1)
    photon_source = bgt.new('SS')

    bgt.add(model, dissipation, photon_source)
    bgt.connect(em_field, dissipation)
    bgt.connect(photon_source, em_field)
    bgt.expose(photon_source)

    # Build the oscillator array
    frequencies = [2 + f for f in (-0.3, -0.1, 0, 0.1, 0.3)]
    osc_mean_field = bgt.new("0")

    bgt.add(model, osc_mean_field)
    bgt.connect(osc_mean_field, (port_hamiltonian, 0))
    osc_array = [Linear_Osc(freq, index)
        for index, freq in enumerate(frequencies)]

    for osc in osc_array:
        bgt.add(model, osc)
        bgt.connect(osc_mean_field, (osc, "P_in"))

    return model
```

## Running a simulation

```python
def experiment(cavity, signal, duration=50):
    laser = bgt.new("Se")
    experiment = bgt.new()

    experiment.add(cavity, laser)
    bgt.connect(cavity, laser)

    t,x = bgt.simulate(
        experiment,
        timespan=[0, duration],
        x0=[0 for _ in cavity.state_vars],
        control_vars=[signal]
    )

    # do post-processing or data shaping here

    # This just selects the oscillator
    indicies = [i for i, (_,  name)
                in enumerate(cavity.state_vars.values())
                if name == 'x_0']
    return t, x[:, indicies]
```

```python
cavity = coupled_cavity()

forcing = "cos(0.9t)"

t, x = experiment(cavity, signal=forcing)

fig = plot_graph(t, x) ## Just makes things pretty.
```

## Getting the equations

```python
cavity.constitutive_relations
```

```python
print_tree(cavity)
```

## Under the Hood

Each energetic systems is represented in local co-ordinates $\mathcal{X}_1 \in \{(\dot{x}_1,e_1,f_1,x_1,u_1)\}$

Motion on the subspace satisfies $\Phi_1(X) = L_1 X + V_1(X) =0$, where $L$ are matricies and $V$ is a strictly nonlinear vector field.

Suppose $\Phi_2$ is a similarly defined subsystem, then the column vector
$X =[X_1;X_2]$ satisfies

$$0 =LX+V(X)  =  \left(
\begin{matrix}
L_1 & 0 \\
0 & L_2 \end{matrix}
\right) X +
\left(\begin{matrix}V_1\circ \pi_1 \\ V_\beta\circ \pi_\beta \end{matrix}\right)(X)
$$

which gives a way of 'stitching togehtor' systems using power conserving flows via

$$e^j_1 = e^k_2,\qquad f^j_1 =-f^k_2$$

Then, we apply a bunch of symnbolic linear algebra, and projections/substitutions.

## State of `BondGraphTools`

Current Status:
- In active development (v.0.3.7) and active use within the lab.
- Documentation at https://bondgraphtools.readthedocs.io/en/latest/
- Available on PyPI https://pypi.org/project/BondGraphTools/
- Source on GitHub https://github.com/BondGraphTools/BondGraphTools
- Manuscript in preparation.

### Planned Future Developments
- Robust parameter and control value network.
- Interface for measuring port space.
- Algorithmic model reduction (particularly manifold reductions).
- Bifurcation analysis (particularly fixed point tracking).

# In Summary

- Emergence is an interesting and relevant phenomenom that is not well understood.
- Emergent phenomenom occur due to interactions between subsystems within the context of a larger system.
- Even in the case of synch, still much is unknowm.

- Emergence can be thought of in terms of low dimensional invariant manifolds.
- So, it would be useful to be able to "scale up" nonlinear analysis tools.
- But there are problems here particularly in how one represent systems

- `BondGraphTools` provides a way to build and recude big model in symbolic form.
- It is hoped that this can feed into algorithmic approaches via GSP to begin to answer these questions.

# Thank You!

Thanks to
- Eduardo and Robbie
- The University of Syndey
- Prof. Edmund Crampin
- The Systems Biology Lab at The University of Melbourne

```python
from BondGraphTools.reaction_builder import Reaction_Network

TCA_reactions = {
    "Citrate synthase":
        ["acetyl-CoA + oxaloacetate + H2O = citrate + CoA-SH"],
    "Aconitase":
        ["Citrate = cis-Aconitate + H2O", "cis-Aconitate + H2O = Isocitrate"],
    "Isocitrate dehydrogenase":
        ["Isocitrate + NAD = Oxalosuccinate + NADH + H",
         "Oxalosuccinate = a-Ketoglutarate + CO2" ],
    "a-Ketoglutarate dehydrogenase":
        ["a-Ketoglutarate + NAD + CoA-SH = Succinyl-CoA + NADH + H + CO2"],
     "Succinyl-CoA synthetase":
        ["Succinyl-CoA + ADP + Pi = Succinate + CoA-SH + ATP"],
     "Succinate dehydrogenase":
        ["Succinate + Q = Fumarate + QH2"],
     "Fumarase":
        ["Fumarate + H2O = L-Malate"],
     "Malate dehydrogenase":
        ["L-Malate + NAD = Oxaloacetate + NADH + H"]
}

def TCA_Cycle():
    reaction_net = Reaction_Network(name="TCA_Cycle")
    for enzyme in TCA_reactions:
        for index, reaction in enumerate(TCA_reactions[enzyme]):
            reaction_name = f"{enzyme} - {index}"
            reaction_net.add_reaction(reaction,  name=reaction_name)
    return reaction_net
```

```python
tca_bg = TCA_Cycle().as_network_model()
tca_bg.constitutive_relations
```

# Please check out `BondGraphTools`

# https://github.com/BondGraphTools/

```python

```


# 3. Data Pre-processing

Data pre-processing techniques generally refer to the addition, deletion, or transformation of training set data. Different models have different sensitivities to the type of predictors in the model; *how* the predictors enter the model is also important.

The need for data pre-processing is determined by the type of model being used. Some procedures, such as tree-based models, are notably insensitive to the characteristics of the predictor data. Others, like linear regression, are not. In this chapter, a wide array of possible methodologies are discussed.

How the predictors are encoded, called *feature engineering*, can have a significant impact on model performance. Often the most effective encoding of the data is informed by the modeler's understanding of the problem and thus is not derived from any mathematical techniques.

## 3.1 Case Study: Cell Segmentation in High-Content Screening

This dataset is from Hill et al. (2007) that consists of 2019 cells. Of these cells, 1300 were judged to be poorly segmented (PS) and 719 were well segmented (WS); 1009 cells were reserved for the training set.

```python
import numpy as np
import pandas as pd

cell_segmentation = pd.read_csv("../datasets/segmentationOriginal/segmentationOriginal.csv")
```

```python
cell_segmentation.shape
```

    (2019, 120)

A first look at the dataset.

```python
cell_segmentation.head(5)
```

      Unnamed: 0
      Cell
      Case
      Class
      AngleCh1
      AngleStatusCh1
      AreaCh1
      AreaStatusCh1
      AvgIntenCh1
      AvgIntenCh2
      ...
      VarIntenCh1
      VarIntenCh3
      VarIntenCh4
      VarIntenStatusCh1
      VarIntenStatusCh3
      VarIntenStatusCh4
      WidthCh1
      WidthStatusCh1
      XCentroid
      YCentroid

      0
      1
      207827637
      Test
      PS
      143.247705
      1
      185
      0
      15.711864
      3.954802
      ...
      12.474676
      7.609035
      2.714100
      0
      2
      2
      10.642974
      2
      42
      14

      1
      2
      207932307
      Train
      PS
      133.752037
      0
      819
      1
      31.923274
      205.878517
      ...
      18.809225
      56.715352
      118.388139
      0
      0
      0
      32.161261
      1
      215
      347

      2
      3
      207932463
      Train
      WS
      106.646387
      0
      431
      0
      28.038835
      115.315534
      ...
      17.295643
      37.671053
      49.470524
      0
      0
      0
      21.185525
      0
      371
      252

      3
      4
      207932470
      Train
      PS
      69.150325
      0
      298
      0
      19.456140
      101.294737
      ...
      13.818968
      30.005643
      24.749537
      0
      0
      2
      13.392830
      0
      487
      295

      4
      5
      207932455
      Test
      PS
      2.887837
      2
      285
      0
      24.275735
      111.415441
      ...
      15.407972
      20.504288
      45.450457
      0
      0
      0
      13.198561
      0
      283
      159

5 rows × 120 columns

This chapter will use the training set samples to demonstrate data pre-processing techniques.

```python
cell_segmentation.groupby('Case').count()
```

      Unnamed: 0
      Cell
      Class
      AngleCh1
      AngleStatusCh1
      AreaCh1
      AreaStatusCh1
      AvgIntenCh1
      AvgIntenCh2
      AvgIntenCh3
      ...
      VarIntenCh1
      VarIntenCh3
      VarIntenCh4
      VarIntenStatusCh1
      VarIntenStatusCh3
      VarIntenStatusCh4
      WidthCh1
      WidthStatusCh1
      XCentroid
      YCentroid

      Case

      Test
      1010
      1010
      1010
      1010
      1010
      1010
      1010
      1010
      1010
      1010
      ...
      1010
      1010
      1010
      1010
      1010
      1010
      1010
      1010
      1010
      1010

      Train
      1009
      1009
      1009
      1009
      1009
      1009
      1009
      1009
      1009
      1009
      ...
      1009
      1009
      1009
      1009
      1009
      1009
      1009
      1009
      1009
      1009

2 rows × 119 columns

```python
# separate training and test data
cell_train = cell_segmentation.ix[cell_segmentation['Case'] == 'Train']
cell_test = cell_segmentation.ix[cell_segmentation['Case'] == 'Test']

cell_train.head(5)
```

      Unnamed: 0
      Cell
      Case
      Class
      AngleCh1
      AngleStatusCh1
      AreaCh1
      AreaStatusCh1
      AvgIntenCh1
      AvgIntenCh2
      ...
      VarIntenCh1
      VarIntenCh3
      VarIntenCh4
      VarIntenStatusCh1
      VarIntenStatusCh3
      VarIntenStatusCh4
      WidthCh1
      WidthStatusCh1
      XCentroid
      YCentroid

      1
      2
      207932307
      Train
      PS
      133.752037
      0
      819
      1
      31.923274
      205.878517
      ...
      18.809225
      56.715352
      118.388139
      0
      0
      0
      32.161261
      1
      215
      347

      2
      3
      207932463
      Train
      WS
      106.646387
      0
      431
      0
      28.038835
      115.315534
      ...
      17.295643
      37.671053
      49.470524
      0
      0
      0
      21.185525
      0
      371
      252

      3
      4
      207932470
      Train
      PS
      69.150325
      0
      298
      0
      19.456140
      101.294737
      ...
      13.818968
      30.005643
      24.749537
      0
      0
      2
      13.392830
      0
      487
      295

      11
      12
      207932484
      Train
      WS
      109.416426
      0
      256
      0
      18.828571
      125.938776
      ...
      13.922937
      18.643027
      40.331747
      0
      0
      2
      17.546861
      0
      211
      495

      14
      15
      207932459
      Train
      PS
      104.278654
      0
      258
      0
      17.570850
      124.368421
      ...
      12.324971
      17.747143
      41.928533
      0
      0
      2
      17.660339
      0
      172
      207

5 rows × 120 columns

## 3.2 Data Transformation for Individual Predictors

Transformations of predictor variables may be needed for several reasons. Some modeling techniques may have strict requirements, such as the predictors having a commom scale. In other cases, creating a good model may be difficult due to specific characteristics of the data (e.g., outliers).

### Centering and Scaling

To center a predictor variable, the average predictor value is substracted from all the values. As a result of centering, the predictor has a zero mean. Similarly, to scale the data, each value of the predictor variable is divided by its standard deviation. Scaling the data coerce the values to have a common standard deviation of one. These manipulations are generally used to improve the numerical stability of some calculations, such as PLS. The only real downside to these transformation is a loss of interpretability of the individual values.

### Transformations to Resolve Skewness

An un-skewed distribution is one that is roughly symmetric. A rule of thumb to consider is that skewed data whose ratio of the highest value to the lowest value is greater than 20 have significant skewness. The sample skewness statistic is defined $$\text{skewness} = {\sum (x_i - \bar{x})^3 \over (n - 1) v^{3/2}},$$ where $$v = {\sum (x_i - \bar{x})^2 \over (n - 1)}.$$ Note that the skewness for a normal distribution is zero.

The cell segmentation data contain a predictor that measures the standard deviation of the intensity of the pixels in the actin filaments.

```python
%matplotlib inline
import matplotlib.pyplot as plt

# Some nice default configuration for plots
plt.rcParams['figure.figsize'] = 10, 7.5
plt.rcParams['axes.grid'] = True
plt.gray()
```

```python
fig, (ax1, ax2, ax3) = plt.subplots(1, 3)

ax1.hist(cell_train['VarIntenCh3'].values, bins=20)
ax1.set_xlabel('Natural Units')
ax1.set_ylabel('Count')

ax2.hist(np.log(cell_train['VarIntenCh3'].values), bins=20)
ax2.set_xlabel('Log Units')

ax3.hist(np.sqrt(cell_train['VarIntenCh3'].values), bins=20)
ax3.set_xlabel('Square Root Units')
```

The histogram shows a strong right skewness. The log transformation seems to work well for this dataset. The ratio of the smallest to largest value and the sample skewness statistic all agree with the histogram under natural units.

```python
from scipy.stats import skew

r = np.max(cell_train['VarIntenCh3'].values)/np.min(cell_train['VarIntenCh3'].values)
skewness = skew(cell_train['VarIntenCh3'].values)

print('Ratio of the smallest to largest value is {0} \nSample skewness statistic is {1}'.format(r, skewness))
```

    Ratio of the smallest to largest value is 870.8872472030321
    Sample skewness statistic is 2.395184181238347

Alternatively, statistical models can be used to empirically identify an appropriate transformation. One of the most famous transformations is the Box-Cox family, i.e.
\begin{equation}
x^* = \begin{cases} {x^{\lambda}-1 \over \lambda} & \text{if} \ \lambda \neq 0 \\ log(x) & \text{if} \ \lambda = 0 \end{cases}
\end{equation}
This family covers the log ($\lambda = 0$), square ($\lambda = 2$), square root ($\lambda = 0.5$), inverse ($\lambda = -1$), and others in-between. Using the training data, $\lambda$ can be estimated using maximum likelihood estimation (MLE). This procedure would be applied independently to each predictor data that contain values **greater than 0**.

The boxcox() in *scipy.stats* finds the estimated lambda and performs the transformation at the same time.

```python
from scipy.stats import boxcox

print('Estimated lambda is {0}'.format(boxcox(cell_train['VarIntenCh3'].values)[1]))
```

    Estimated lambda is 0.121531931959674

Take another predictor for example.

```python
fig, (ax1, ax2) = plt.subplots(1, 2)

ax1.hist(cell_train['PerimCh1'].values, bins=20)
ax1.set_xlabel('Natural Units')
ax1.set_ylabel('Count')

ax2.hist(boxcox(cell_train['PerimCh1'].values)[0], bins=20)
ax2.set_xlabel('Transformed Data (lambda = {:1.4f})'.format(boxcox(cell_train['PerimCh1'].values)[1]))
```

## 3.3 Data Transformations for Multiple Predictors

These transformations act on groups of predictors, typically the entire set under consideration. Of primary importance are methods to resolve outliers and reduce the dimension of the data.

### Transformations to Resolve Outliers

We generally define outliers as samples that are exceptionally far from the mainstream of the data. Even with a thorough understanding of the data, outliers can be hard to define. However, we can often identify an unusual value by looking at a figure. When one or more samples are suspected to be outliers, the first step is to make sure that the values are scientifically valid and that no data recording errors have occured. Great care should be taken not to hastily remove or change values, especially if the sample size is small. With small sample sizes, apparent outliers might be a result of a skewed distribution where there are not yet enough data to see the skewness. Also, the outlying data may be an indication of a special part of the population under study that is just starting to be sampled. Depending on how the data were collected, a "cluster" of valid points that reside outside the mainstream of the data might belong to a different population than the other samples, e.g. *extrapolation* and *applicability domain*.

There are several predictive models that are resistant to outliers, e.g.
- Tree based classification models: create splits of the training set.
- Support Vector Machines (SVM) for classification: disregard a portion if the training set that may be far away from the decision boundary.

If a model is considered to be sensitive to outliers, one data transformation that can minimize the problem is the *spatial sign*. Mathematically, each sample is divided by its squared norm: $$x_{ij}^* = {x_{ij} \over \sqrt{\sum_{j=1}^p x_{ij}^2}}.$$ Since the denominator is intended to measure the squared distance to the center of the predictor's distribution, it is **important** to center and scale the predictor data prior to using this transformation. Note that, unlike centering and scaling, this manipulation of the predictors transform them as a group. Removing predictor variables after applying the spatial sign transformation may be problematic.

```python
# toy example
beta0 = -2.3 # intercept
beta1 = 0.8 # slope
n = 1000
x1_true = np.random.normal(4, 2, n)
x2_true = np.zeros(n)

# generate a random sample
for i in range(n):
    x2_true[i] = beta0 + beta1*x1_true[i] + np.random.normal(size = 1)

# generate outliers
x1_outliers = np.random.uniform(-4, -3, 8)
x2_outliers = np.zeros(8)
for i in range(8):
    x2_outliers[i] = x1_outliers[i] + np.random.normal(size = 1)

plt.scatter(x1_true, x2_true)
plt.plot(x1_outliers, x2_outliers, 'ro', markersize=8)
```

```python
from sklearn.preprocessing import scale
x1 = scale(np.concatenate([x1_true, x1_outliers]))
x2 = scale(np.concatenate([x2_true, x2_outliers]))
x = np.array(zip(x1, x2))

```

```python

# spatial sign
dist = x[:, 0]**2 + x[:, 1]**2
x1 = x[:, 0]/np.sqrt(dist)
x2 = x[:, 1]/np.sqrt(dist)

plt.scatter(x1[:-8], x2[:-8])
plt.plot(x1[-7:], x2[-7:], 'ro', markersize=8)
```

The *spatial sign* transformation brings the outliers towards the majority of the data.

### Data Reduction and Feature Extraction

These methods reduce the data by generating a smaller set of predictors that seek to capture a majority of the information in the original variables. For most data reduction techniques, the new predictors are functions of the original predictors; therefore, all the original predictors are still needed to create the surrogate variables. This class of methods is often called *signal extraction* or *feature extraction* techniques.

Principal component analysis (PCA) seeks to find linear combinations of the predictors, known as principal components (PCs), which capture the most possible variance. The first PC is defined as the linear combination of the predictors that captures the most variability of all possible linear combinations. Then, subsequent PCs are derived such that these linear combinations capture the most remaining variability while also being uncorrelated with all previous PCs. Mathematically,
$$\text{PC}_j = (a_{j1} \times \text{Predictor 1}) + \cdots + (a_{jP} \times \text{Predictor P}).$$
P is the number of predictors. The coefficients $a_{j1}, \cdots, a_{jP}$ are called component weights and help us understand which predictors are most important to each PC.

Let us look at an example from the previous dataset.

```python
cell_train_subset = cell_train[['Class', 'FiberWidthCh1', 'EntropyIntenCh1']]
```

```python
colors = ['b', 'r']
markers = ['s', 'o']
c = ['PS', 'WS']
for k, m in enumerate(colors):
    i = (cell_train_subset['Class'] == c[k])
    if k == 0:
        plt.scatter(cell_train_subset['FiberWidthCh1'][i], cell_train_subset['EntropyIntenCh1'][i],
                    c=m, marker=markers[k], alpha=0.4, s=26, label='PS')
    else:
        plt.scatter(cell_train_subset['FiberWidthCh1'][i], cell_train_subset['EntropyIntenCh1'][i],
                    c=m, marker=markers[k], alpha=0.4, s=26, label='WS')

plt.title('Original Data')
plt.xlabel('Channel 1 Fiber Width')
plt.ylabel('Entropy intensity of Channel 1')
plt.legend(loc='upper right')
plt.show()
```

Calculate PCs

```python
from sklearn.decomposition import PCA

pca = PCA()
pca.fit(cell_train_subset[['FiberWidthCh1', 'EntropyIntenCh1']])
print('variance explained by PCs {0}'.format(pca.explained_variance_ratio_))
```

    variance explained by PCs [ 0.96900659  0.03099341]

The first PC summarizes 97% of the original variability, while the second summarizes 3%. Hence, it is reasonable to use only the first PC for modeling since it accounts for the majority of the information in the data.

```python
cell_train_subset_pca = pca.transform(cell_train_subset[['FiberWidthCh1', 'EntropyIntenCh1']])

colors = ['b', 'r']
markers = ['s', 'o']
c = ['PS', 'WS']
for k, m in enumerate(colors):
    i = np.where(cell_train_subset['Class'] == c[k])[0]
    if k == 0:
        plt.scatter(cell_train_subset_pca[i, 0], cell_train_subset_pca[i, 1],
                    c=m, marker=markers[k], alpha=0.4, s=26, label='PS')
    else:
        plt.scatter(cell_train_subset_pca[i, 0], cell_train_subset_pca[i, 1],
                    c=m, marker=markers[k], alpha=0.4, s=26, label='WS')

plt.title('Transformed')
plt.xlabel('Principal Component #1')
plt.ylabel('Principal Component #2')
plt.legend(loc='upper right')
plt.show()
```

The primary advantage of PCA is that it creates components that are uncorrelated. PCA preprocessing creates new predictors with desirable characteristics for models that prefer predictors to be uncorrelated.

While PCA delivers new predictors with desirable characteristics, it must be used with understanding and care. PCA seeks predictor-set variation without regard to any further understanding of the predictors (i.e. measurement scales or distributions) or to knowledge of the modeling objectives (i.e. response variable). Hence, without proper guidance, PCA can generate components that summarize characteristics of the data that are irrelevant to the underlying structure of the data and also to the ultimate modeling objectives.

PCA was applied to the entire set of segmentation data predictors.

```python
cell_train.head(5)
```

      Unnamed: 0
      Cell
      Case
      Class
      AngleCh1
      AngleStatusCh1
      AreaCh1
      AreaStatusCh1
      AvgIntenCh1
      AvgIntenCh2
      ...
      VarIntenCh1
      VarIntenCh3
      VarIntenCh4
      VarIntenStatusCh1
      VarIntenStatusCh3
      VarIntenStatusCh4
      WidthCh1
      WidthStatusCh1
      XCentroid
      YCentroid

      1
      2
      207932307
      Train
      PS
      133.752037
      0
      819
      1
      31.923274
      205.878517
      ...
      18.809225
      56.715352
      118.388139
      0
      0
      0
      32.161261
      1
      215
      347

      2
      3
      207932463
      Train
      WS
      106.646387
      0
      431
      0
      28.038835
      115.315534
      ...
      17.295643
      37.671053
      49.470524
      0
      0
      0
      21.185525
      0
      371
      252

      3
      4
      207932470
      Train
      PS
      69.150325
      0
      298
      0
      19.456140
      101.294737
      ...
      13.818968
      30.005643
      24.749537
      0
      0
      2
      13.392830
      0
      487
      295

      11
      12
      207932484
      Train
      WS
      109.416426
      0
      256
      0
      18.828571
      125.938776
      ...
      13.922937
      18.643027
      40.331747
      0
      0
      2
      17.546861
      0
      211
      495

      14
      15
      207932459
      Train
      PS
      104.278654
      0
      258
      0
      17.570850
      124.368421
      ...
      12.324971
      17.747143
      41.928533
      0
      0
      2
      17.660339
      0
      172
      207

5 rows × 120 columns

```python
cell_train_feature = cell_train.iloc[:, 4:]
cell_train_feature.head(5)
```

      AngleCh1
      AngleStatusCh1
      AreaCh1
      AreaStatusCh1
      AvgIntenCh1
      AvgIntenCh2
      AvgIntenCh3
      AvgIntenCh4
      AvgIntenStatusCh1
      AvgIntenStatusCh2
      ...
      VarIntenCh1
      VarIntenCh3
      VarIntenCh4
      VarIntenStatusCh1
      VarIntenStatusCh3
      VarIntenStatusCh4
      WidthCh1
      WidthStatusCh1
      XCentroid
      YCentroid

      1
      133.752037
      0
      819
      1
      31.923274
      205.878517
      69.916880
      164.153453
      0
      0
      ...
      18.809225
      56.715352
      118.388139
      0
      0
      0
      32.161261
      1
      215
      347

      2
      106.646387
      0
      431
      0
      28.038835
      115.315534
      63.941748
      106.696602
      0
      0
      ...
      17.295643
      37.671053
      49.470524
      0
      0
      0
      21.185525
      0
      371
      252

      3
      69.150325
      0
      298
      0
      19.456140
      101.294737
      28.217544
      31.028070
      0
      0
      ...
      13.818968
      30.005643
      24.749537
      0
      0
      2
      13.392830
      0
      487
      295

      11
      109.416426
      0
      256
      0
      18.828571
      125.938776
      13.600000
      46.800000
      0
      0
      ...
      13.922937
      18.643027
      40.331747
      0
      0
      2
      17.546861
      0
      211
      495

      14
      104.278654
      0
      258
      0
      17.570850
      124.368421
      22.461538
      71.206478
      0
      0
      ...
      12.324971
      17.747143
      41.928533
      0
      0
      2
      17.660339
      0
      172
      207

5 rows × 116 columns

Because PCA seeks linear combinations of predictors that maximize variability, it will naturally first be drawn to summarizing predictors that have more variation. If the original predictors are on measurement scales that differ in orders of magnitude or have skewed distributions, PCA will be focusing its efforts on identifying the data structure based on measurement scales and distributional difference rather than based on the important relationships within the data for the current problem. Hence, it is best to first transform skewed predictors and then center and scale the predictors prior to performing PCA.

```python
# Box-Cox transformation on positive predictors
# separate positive and non-positive predictors
pos_indx = np.where(cell_train_feature.apply(lambda x: np.all(x > 0)))[0]
cell_train_feature_pos = cell_train_feature.iloc[:, pos_indx]
print("# of positive features is {0}".format(pos_indx.shape[0]))
cell_train_feature_nonpos = cell_train_feature.drop(cell_train_feature.columns[pos_indx], axis=1, inplace=False)
print("# of npn-positive features is {0}".format(cell_train_feature.shape[1] - pos_indx.shape[0]))

cell_train_feature_pos_tr = cell_train_feature_pos.apply(lambda x: boxcox(x)[0])

cell_train_feature_tr = np.c_[cell_train_feature_pos_tr, cell_train_feature_nonpos]
print("The shape before/after transformation is {0} and {1}".format(cell_train_feature.shape, cell_train_feature_tr.shape))
```

    # of positive features is 47
    # of npn-positive features is 69
    The shape before/after transformation is (1009, 116) and (1009, 116)

```python
# scale and center predictors
from sklearn.preprocessing import scale

cell_train_feature_tr = scale(cell_train_feature_tr, with_mean=True, with_std=True)
```

The second caveat of PCA is that it does not consider the modeling obejective or response variable when summarizing variability -- it is an *unsupervised technique*. If the predictive relationship between the predictors and response is not connected to the predictors' variability, then the derived PCs will not provide a suitable relationship with the response. In this case, a *supervised technique*, like PLS will derive components while simultaneously considering the corresponding response.

To decide how many components to retain after PCA, a heuristic approach is to create a scree plot, which contains the ordered component number (x-axis) and the amount of summarized variability (y-axis). Generally, the component number prior to the tapering off of variation is the maximal component that is retained. In an automated model building process, the optimal number of components can be determined by cross-validation.

```python
# conduct PCA to transformed predictors
from sklearn.decomposition import PCA

pca = PCA()
pca.fit(cell_train_feature_tr)

# generate scree plot
plt.plot(pca.explained_variance_ratio_)
plt.xlabel('Percent of Total Variance')
plt.ylabel('Component')
```

```python
print("The first four components account for {0} of the total variance".format(pca.explained_variance_ratio_[:4]))
print("All together they account for {0} of the total variance".format(np.sum(pca.explained_variance_ratio_[:4])))
```

    The first four components account for [ 0.14015262  0.12569722  0.09394676  0.06393814] of the total variance
    All together they account for 0.4237347251400651 of the total variance

Visually examining the principal components is a critical step for assessing data quality and gaining intuition for the problem. To do this, the first few PCs can be plotted against each other and the plot symbols can be colored by the relevant characteristics, such as the class labels. If PCA has captured a sufficient amount of the information in the data, this type of plot can demonstrate clusters of samples or outliers that may prompt a closer examination of the individual data points. Note that the scale of the components tend to become smaller as they account for less and less variation in the data. If axes are displayed on separate scales, there is the potential to over-interpret any patterns that might be seen for components that account for small amounts of variation.

```python
# look at the first 3 PCs
pca = PCA(n_components=3)
cell_train_feature_pca = pca.fit_transform(cell_train_feature_tr)
```

```python
colors = ['b', 'r']
markers = ['s', 'o']
c = ['PS', 'WS']

fig, axarr = plt.subplots(3, 3, sharex=True, sharey=True)

# PC1 vs PC3
for k, m in enumerate(colors):
    i = np.where(cell_train['Class'] == c[k])[0]
    if k == 0:
        line1= axarr[0,0].scatter(cell_train_feature_pca[i, 0], cell_train_feature_pca[i, 2],
                    c=m, marker=markers[k], alpha=0.4, s=26, label='PS')
    else:
        line2= axarr[0,0].scatter(cell_train_feature_pca[i, 0], cell_train_feature_pca[i, 2],
                    c=m, marker=markers[k], alpha=0.4, s=26, label='WS')

# PC2 vs PC3
for k, m in enumerate(colors):
    i = np.where(cell_train['Class'] == c[k])[0]
    if k == 0:
        axarr[0,1].scatter(cell_train_feature_pca[i, 1], cell_train_feature_pca[i, 2],
                    c=m, marker=markers[k], alpha=0.4, s=26, label='PS')
    else:
        axarr[0,1].scatter(cell_train_feature_pca[i, 1], cell_train_feature_pca[i, 2],
                    c=m, marker=markers[k], alpha=0.4, s=26, label='WS')

# PC1 vs PC2
for k, m in enumerate(colors):
    i = np.where(cell_train['Class'] == c[k])[0]
    if k == 0:
        axarr[1,0].scatter(cell_train_feature_pca[i, 0], cell_train_feature_pca[i, 1],
                    c=m, marker=markers[k], alpha=0.4, s=26, label='PS')
    else:
        axarr[1,0].scatter(cell_train_feature_pca[i, 0], cell_train_feature_pca[i, 1],
                    c=m, marker=markers[k], alpha=0.4, s=26, label='WS')

axarr[2,0].text(0.5, -1.0, 'PC1', ha='center', va='center', fontsize=24)
axarr[1,1].text(0.5, -1.0, 'PC2', ha='center', va='center', fontsize=24)
axarr[0,2].text(0.5, -1.0, 'PC3', ha='center', va='center', fontsize=24)
fig.legend([line1, line2], ('PS', 'WS'), loc='upper center', ncol=2, frameon=False)
fig.subplots_adjust(hspace=0.12, wspace=0.1)
fig.text(0.5, 0.06, 'Scatter Plot Matrix', ha='center', va='center', fontsize=18)
```

Since the percentages of variation explained are not large for the first three components, it is important not to over-interpret the resulting image. From this plot, there appears to be some separation between the classes when plotting the first and second components. However, the distribution of the well-segmented cells is roughly contained within the distribution of the poorly identified cells. One conclusion is that the cell types are not easily separated.

Another exploratory use of PCA is characterizing which predictors are associated with each component. Recall that each component is a linear combination of the predictors and the coefficient for each predictor is called the loading. Loadings close to zero indicate that the predictor variable did not contribute much to that component.

```python
# loadings
pca.components_.shape
```

    (3, 116)

## 3.4 Dealing with Missing Values

In many cases, some predictors have no values for a given sample. It is important to understand *why* the values are missing. First and foremost, it is important to know if the pattern of missing data is related to the outcome. This is called *informative missingness* since the missing data pattern is instructional on its own. Informative missingness can induce significant bias in the model.

Missing data should not be confused with *censored* data where the exact value is missing but something is known about its value. When building traditional statistical models focused on interpretation or inference, the censoring is usually taken in to account in a formal manner by making assumptions about the censoring mechanism. For predictive models, it is more common to treat these data as simple missing data or use the censored value as the observed value.

Missing values are more often related to predictive variables than the sample. Because of this, amount of missing data may be concentrated in a subset of predictors rather than occuring randomly across all the predictors. In some cases, the percentage of missing data is substantial enough to remove this predictor from subsequent modeling activities.

There are cases where the missing values might be concentrated in specific samples. For large datasets, removal of samples based on missing values is not a problem, assuming that the missingness is not informative. In smaller datasets, there is a steep price in removing samples; some of alternative approaches described below may be more appropriate.

If we do not remove the missing data, there are two general approaches. First, a few predictive models, especially tree-based techniques, can specifically account for missing data. Alternatively, missing data can be imputed. In this case, we can use information in the training set predictors to, in essence, estimate the values of other predictors.

Imputation is just another layer of modeling where we try to estimate values of the predictor variables based on other predictor variables. The most relevant scheme for accomplishing this is to use the training set to built an imputation model for each predictor in the daa set. Prior to model training or the prediction of new samples, missing values are filled in using imputation. Note that this extra layer of models adds uncertainty. If we are using resampling to select tuning parameter values or to estimate performance, the imputation should be incorporated within the resampling. This will increase the computational time for building models, but it will also provide honest estimates of model performance.

If the number of predictors affected by missing values is small, an exploratory analysis of the relationships between the preditors is a good idea. For example, visulization or methods like PCA can be used to determine if there are strong relationships between the predictors. If a variable with missing values is highly correlated with another predictor that has few missing values, a focused model can often be effective for imputation.

One popular technique for imputation is a $K$-nearest neighbor model. A new sample is imputed by finding the samples in the training set "closest" to it and averages these nearby points to fill in the value. One advantage of this approach is that the imputed data are confined to be within the range of the training set values. One disadvantage is that the entire training set is required every time a missing value needs to be imputed. Also, the number of neighbors is a tuning parameter, as is the method for determining "closeness" of two points. However, Troyanskaya et al. (2001) found the nearest neighbor approach to be fairly robust to the tuning parameters, as well as the amount of missing data.

```python
# randomly sample 50 test set
import random
cell_test_subset = cell_test.iloc[np.sort(random.sample(range(cell_test.shape[0]), 50))]

# separate features
cell_test_subset_f = cell_test_subset.iloc[:, 4:].drop('VarIntenCh3', 1)
cell_test_subset_v = cell_test_subset.iloc[:, 4:]['VarIntenCh3']
cell_train_f = cell_train_feature.drop('VarIntenCh3', 1)
cell_train_v = cell_train_feature['VarIntenCh3']
```

```python
# scale and center before imputation
from sklearn.preprocessing import StandardScaler

# standardize based on training set
sc_f = StandardScaler()
cell_train_f_sc = sc_f.fit_transform(cell_train_f)
cell_test_subset_f_sc = sc_f.transform(cell_test_subset_f)

sc_v = StandardScaler()
cell_train_v_sc = sc_v.fit_transform(cell_train_v.reshape(-1, 1))
cell_test_subset_v_sc = sc_v.transform(cell_test_subset_v.reshape(-1, 1))
```

    /Users/aremirata/anaconda3/lib/python3.5/site-packages/ipykernel_launcher.py:10: FutureWarning: reshape is deprecated and will raise in a subsequent release. Please use .values.reshape(...) instead
      # Remove the CWD from sys.path while we load stuff.
    /Users/aremirata/anaconda3/lib/python3.5/site-packages/ipykernel_launcher.py:11: FutureWarning: reshape is deprecated and will raise in a subsequent release. Please use .values.reshape(...) instead
      # This is added back by InteractiveShellApp.init_path()

```python
# use 5-nearest neighbor
from sklearn.neighbors import NearestNeighbors

nbrs = NearestNeighbors(n_neighbors = 5)
nbrs.fit(cell_train_f_sc) # based on training set
distance, indices = nbrs.kneighbors(cell_test_subset_f_sc) # neighbors for test set

# imputation
cell_test_subset_v_pred_knn = np.empty(50)
for idx, i in enumerate(indices):
    cell_test_subset_v_pred_knn[idx] = np.mean(cell_train_v_sc[i[1:]])
```

Find the predictor with highest correlation.

```python
from scipy.stats.stats import pearsonr

print("corr('VarIntenCh3', 'DiffIntenDensityCh3') is {0}".format(pearsonr(cell_train_v, cell_train_f['DiffIntenDensityCh3'])[0]))
```

    corr('VarIntenCh3', 'DiffIntenDensityCh3') is 0.8948715047853233

```python
# use linear model
from sklearn.linear_model import LinearRegression

lm = LinearRegression()
lm.fit(cell_train_f_sc[:, cell_train_f.columns.get_loc('DiffIntenDensityCh3')][:, np.newaxis],
       cell_train_v_sc) # find the predictor with highest correlation
cell_test_subset_v_pred_lm = \
lm.predict(cell_test_subset_f_sc[:, cell_train_f.columns.get_loc('DiffIntenDensityCh3')][:, np.newaxis])
```

Correlation between the real and imputed values

```python
print("kNN: {0}".format(pearsonr(cell_test_subset_v_sc.ravel(), cell_test_subset_v_pred_knn)[0]))
print("Linear Model: {0}".format(pearsonr(cell_test_subset_v_sc, cell_test_subset_v_pred_lm)[0][0]))
```

    kNN: 0.9041754173339616
    Linear Model: 0.8714262815406963

Note that the better performance of linear model is because of the high correlation (0.895) between these two predictors. kNN is generally more robust since it takes all predictors into consideration.

```python
fig, (ax1, ax2) = plt.subplots(1, 2)

ax1.scatter(cell_test_subset_v_sc, cell_test_subset_v_pred_knn)
ax1.set(xlim=(-1.5, 3), ylim=(-1.5, 3))
ax1.plot(ax1.get_xlim(), ax1.get_ylim(), ls="--", c=".3")
ax1.set_title('5NN')

ax2.scatter(cell_test_subset_v_sc, cell_test_subset_v_pred_lm)
ax2.set(xlim=(-1.5, 3), ylim=(-1.5, 3))
ax2.plot(ax2.get_xlim(), ax2.get_ylim(), ls="--", c=".3")
ax2.set_title('Linear Model')

fig.text(0.5, 0.04, 'Original Value (centered and scaled)', ha='center', va='center')
fig.text(0.06, 0.5, 'Imputed', ha='center', va='center', rotation='vertical')
```

## 3.5 Removing Predictors

There are potential advantages to removing predictors prior to modeling. First, fewer predictors means decreased computational time and complexity. Second, if two predictors are highly correlated, this implies that they are measuring the same underlying information. Removing one should not compromise the performance of the model and might lead to a more parsimonious and interpretable model. Third, some models can be crippled by predictors with degenerate distributions, e.g. near-zero variance predictors. In these cases, there can be a significant improvement in model performance and/or stability without the problematic variables.

A rule of thumb for detecting near-zero variance predictors:
- The fraction of unique values over the sample size is low (say 10%)
- The ratio of the frequency of the most prevalent value to the frequency of the second most prevalent value is large (say around 20)

If both of these criteria are true and the model in question is susceptible to this type of predictor, it may be advantageous to remove the variable from the model.

### Between-Predictor Correlations

*Collinearity* is the technical term for the situation where a pair of predictor variables have a substantial correlation with each other. It is also possible to have relationships between multiple predictors at once (called *multicollinearity*).

A direct visualization of the correlation matrix from the training set.

```python
# calculate the correlation matrix
corr_dataframe = cell_train_feature.corr()

# compute hierarchical cluster on both rows and columns for correlation matrix and plot heatmap
def corr_heatmap(corr_dataframe):
    import scipy.cluster.hierarchy as sch

    corr_matrix = np.array(corr_dataframe)
    col_names = corr_dataframe.columns

    Y = sch.linkage(corr_matrix, 'single', 'correlation')
    Z = sch.dendrogram(Y, color_threshold=0, no_plot=True)['leaves']
    corr_matrix = corr_matrix[Z, :]
    corr_matrix = corr_matrix[:, Z]
    col_names = col_names[Z]
    im = plt.imshow(corr_matrix, interpolation='nearest', aspect='auto', cmap='bwr')
    plt.colorbar()
    plt.xticks(range(corr_matrix.shape[0]), col_names, rotation='vertical', fontsize=4)
    plt.yticks(range(corr_matrix.shape[0]), col_names[::-1], fontsize=4)

# plot
corr_heatmap(corr_dataframe)
```

Note that the predictor variables have been grouped using a clustering technique so that collinear groups of predictors are adjacent to one another.

When the data set consists of too many predictors to examine visually, techniques such as PCA can be used to characterize the magnitude of the problem. For example, if the first principal component accounts for a large percentage of the variance, this implies that there is at least one group of predictors that represent the same information. The PCA loadings can be used to understand which predictors are associated with each component to tease out this relationship.

In general, there are good reasons to avoid data with highly correlated predictors. First, redundant predictors frequently add more complexity to the model than information they provide to the model. In situations where obtaining the predictor data is costly, fewer variables is obviously better. Using highly correlated predictors in techniques like linear regression can result in highly unstable models, numerical values, and degraded predictive performances.

Classical regression analysis has several tools to diagnose multicollinearity for linear regression. A statistic called the variance inflation factor (VIF) can be used to identify predictors that are impacted. A common rule of thumb is that if VIF > 5, then multicollinearity is high. Note that this method is developed for linear models, it requires more samples than predictor variables and it does not determine which should be removed to resolve the problem

A more heuristic approach is to remove the minimum number of predictors to ensure that all pairwise correlation are below a certain threshold. The algorithm is as follows:
- Calculate the correlation matrix of the predictors.
- Determine the two predictors associated with the largest absolute pairwise correlation (A and B).
- Determine the average absolute correlation between A and the other variables. Do the same for predictor B.
- If A has a larger average correlation, remove it; otherwise, remove predictor B.
- Repeat Steps 2-4 until no absolute correlations are above the threshold.

Suppose we wanted to use a model that is particularly sensitive to between predictor correlations, we might apply a threshold of 0.75.

As previously mentioned, feature extraction methods (e.g., principal components) are another technique for mitigating the effect of strong correlations between predictors. However, these techniques make the connection between the predictors and the outcome more complex. Additionally, since signal extraction methods are usually unsupervised, there is no guarantee that the resulting surrogate preditors have any relationship with the outcome.

## 3.6 Adding Predictors

When a predictor is categorical, it is common to decompose the predictor into a set of more specific variables.

Look at the following example for the credit scoring data.

```python
credit_data = pd.read_csv("../datasets/GermanCredit/GermanCredit.csv")
credit_data.head(5)
```

      Unnamed: 0
      Duration
      Amount
      InstallmentRatePercentage
      ResidenceDuration
      Age
      NumberExistingCredits
      NumberPeopleMaintenance
      Telephone
      ForeignWorker
      ...
      OtherInstallmentPlans.Bank
      OtherInstallmentPlans.Stores
      OtherInstallmentPlans.None
      Housing.Rent
      Housing.Own
      Housing.ForFree
      Job.UnemployedUnskilled
      Job.UnskilledResident
      Job.SkilledEmployee
      Job.Management.SelfEmp.HighlyQualified

      0
      1
      6
      1169
      4
      4
      67
      2
      1
      0.0
      1.0
      ...
      0.0
      0.0
      1.0
      0.0
      1.0
      0.0
      0.0
      0.0
      1.0
      0.0

      1
      2
      48
      5951
      2
      2
      22
      1
      1
      1.0
      1.0
      ...
      0.0
      0.0
      1.0
      0.0
      1.0
      0.0
      0.0
      0.0
      1.0
      0.0

      2
      3
      12
      2096
      2
      3
      49
      1
      2
      1.0
      1.0
      ...
      0.0
      0.0
      1.0
      0.0
      1.0
      0.0
      0.0
      1.0
      0.0
      0.0

      3
      4
      42
      7882
      2
      4
      45
      1
      2
      1.0
      1.0
      ...
      0.0
      0.0
      1.0
      0.0
      0.0
      1.0
      0.0
      0.0
      1.0
      0.0

      4
      5
      24
      4870
      3
      4
      53
      2
      2
      1.0
      1.0
      ...
      0.0
      0.0
      1.0
      0.0
      0.0
      1.0
      0.0
      0.0
      1.0
      0.0

5 rows × 63 columns

```python
credit_data.shape
```

    (1000, 63)

The predictor based on how much money was in the applicant's saving account is categorical coded into dummy variables.

```python
credit_data_saving = credit_data[['SavingsAccountBonds.lt.100', 'SavingsAccountBonds.100.to.500',
                                  'SavingsAccountBonds.500.to.1000', 'SavingsAccountBonds.gt.1000',
                                  'SavingsAccountBonds.Unknown']]
credit_data_saving.head(10)
```

      SavingsAccountBonds.lt.100
      SavingsAccountBonds.100.to.500
      SavingsAccountBonds.500.to.1000
      SavingsAccountBonds.gt.1000
      SavingsAccountBonds.Unknown

      0
      0.0
      0.0
      0.0
      0.0
      1.0

      1
      1.0
      0.0
      0.0
      0.0
      0.0

      2
      1.0
      0.0
      0.0
      0.0
      0.0

      3
      1.0
      0.0
      0.0
      0.0
      0.0

      4
      1.0
      0.0
      0.0
      0.0
      0.0

      5
      0.0
      0.0
      0.0
      0.0
      1.0

      6
      0.0
      0.0
      1.0
      0.0
      0.0

      7
      1.0
      0.0
      0.0
      0.0
      0.0

      8
      0.0
      0.0
      0.0
      1.0
      0.0

      9
      1.0
      0.0
      0.0
      0.0
      0.0

```python
credit_data_saving.apply(np.sum)
```

    SavingsAccountBonds.lt.100         603.0
    SavingsAccountBonds.100.to.500     103.0
    SavingsAccountBonds.500.to.1000     63.0
    SavingsAccountBonds.gt.1000         48.0
    SavingsAccountBonds.Unknown        183.0
    dtype: float64

| Value    |  n   | <100 | 100-500 | 500-1000 | >1000 | Unknown |
|:---------|:----:|:----:|:-------:|:--------:|:-----:|:-------:|
| < 100    | 603  |   1  |   0     |    0     |   0   |    0    |
| 100-500  | 100  |   0  |   1     |    0     |   0   |    0    |
| 500-1000 |  63  |   0  |   0     |    1     |   0   |    0    |
| >1000    |  48  |   0  |   0     |    0     |   1   |    0    |
| Unknown  | 183  |   0  |   0     |    0     |   0   |    1    |

Usually, each category gets its own dummy variable that is a zero/one indicator for that group. Only four dummy variables are needed here, the fifth can be inferred. However, the decision to include all of the dummy variables can depend on the choice of the model. Models that include an intercept term, such as simple linear model, would have numerical issues if each dummy variable was included in the model. The reason is that, for each sample, these variables all add up to one and this would provide the same information as the intercept. If the model is insensitive to this type of issue, using the complete set of dummy variables would help improve interpretation of the model.

Many of the advanced models automatically generate highly complex, nonlinear relationships between the predictors and the outcome. More simplistic models do not unless the user manually specifices which predictors should be nonlinear and in what way. Another technique to augment the prediction data for classification model is through the "*class centroids*", which are the centers of the predictor data for each class. For each predictor, the distance to each class centroid can be calculated and these distances can be added to the model.

## 3.7 Binning Predictors (to avoid)

There are many issues with the manual binning of continuous data. First, there can be a significant loss of performance in the model. Second, there is a loss of precision in the predictions when the predictors are categorized. Unfortunately, the predictive models that are most powerful are usually the least interpretable. The bottom line is that the perceived improvement in interpretability gained by manual categorization is usually offset by a significant loss in performance.

Note that the argument here is related to the *manual* categorization of predictors prior to model building. There are several models, such as classification/regression trees and multivariate adaptive regression splines, that estimate cut points in the process of model building. The difference between these methodologies and manual binning is that the models ues all the predictors to derive bins based on a single objective (such as maximizing accuracy). They evaluate many variable simultaneously and are usually based on statistically sound methodologies.