```python
from games_setup import *
from SBMLLint.common import constants as cn
from SBMLLint.common.molecule import Molecule, MoleculeStoichiometry
from SBMLLint.common.reaction import Reaction
from SBMLLint.games.som import SOM
from SBMLLint.common.simple_sbml import SimpleSBML

import collections
import itertools
import networkx as nx
import numpy as np
import pandas as pd
import scipy
from sympy.matrices import Matrix, eye
#
from SBMLLint.games.message import Message
```

    Current Directory: /Users/woosubshin/Desktop/ModelEngineering/SBMLLint/notebook

```python
simple = load_file_from_curated_data(247)
for r in simple.reactions:
  if r.category != cn.REACTION_BOUNDARY and r.category != cn.REACTION_1_1:
    print(r.makeIdentifier(is_include_kinetics=False))
```

    vGLK: GLCi + P -> G6P
    vPFK: F6P + P -> F16P
    vALD: F16P -> DHAP + GA3P
    vG3PDH: DHAP + NADH -> GLY + NAD
    vGAPDH: GA3P + NAD -> BPG + NADH
    vPGK: BPG -> P3G + P
    vPYK: PEP -> PYR + P
    vPDC: PYR -> ACE + CO2
    vSUC: 2.00 ACE + 3.00 NAD -> 3.00 NADH + SUCC
    vADH: ACE + NADH -> ETOH + NAD
    vG6PDH: G6P + NADP -> D6PGluconoLactone + NADPH
    vGluDH: D6PGluconate + NADP -> Ribulose5P + NADPH
    vTransk1: Ribose5P + Xyl5P -> GA3P + Seduhept7P
    vTransald: Seduhept7P + GA3P -> F6P + Erythrose4P
    vTransk2: Xyl5P + Erythrose4P -> GA3P + F6P

```python
m = Message(simple)
m
```

    {GA3P};{PEP};{NADH};{Ribulose5P};{X};{P2G};{GLY};{NADP};{GLCo};{D6PGluconoLactone};{ACE};{NAD};{F6P};{D6PGluconate};{ETOH};{F16P};{Xyl5P};{GLCi};{NADPH};{CO2};{Erythrose4P};{BPG};{P};{G6P};{DHAP};{SUCC};{Seduhept7P};{P3G};{PYR};{Ribose5P}

```python
m.som_reactions_lu
```

    []

```python
m.analyze()
```

    We just analyzed the data...
    Type I error:  []
    Type II error:  []
    Canceling error:  []
    Echelon error:  []
    Type III error:  []
    Type I-SOM error:  set()

    False

```python
m.edges
```

    OutEdgeView([({ACE}, {PYR}), ({CO2}, {PYR}), ({PYR}, {P2G, P3G, PEP}), ({F6P, G6P}, {F16P}), ({P2G, P3G, PEP}, {BPG}), ({P, X}, {BPG}), ({P, X}, {P2G, P3G, PEP}), ({P, X}, {F6P, G6P}), ({P, X}, {F16P}), ({P, X}, {D6PGluconate, D6PGluconoLactone}), ({DHAP, GA3P}, {F16P}), ({GLCi, GLCo}, {F6P, G6P}), ({GLCi, GLCo}, {D6PGluconate, D6PGluconoLactone})])

```python
m.nodes
```

    NodeView(({NADH}, {GLY}, {ACE}, {NAD}, {ETOH}, {F16P}, {CO2}, {Erythrose4P}, {BPG}, {SUCC}, {Seduhept7P}, {PYR}, {F6P, G6P}, {P2G, P3G, PEP}, {P, X}, {DHAP, GA3P}, {D6PGluconate, D6PGluconoLactone}, {Ribose5P, Ribulose5P, Xyl5P}, {NADP, NADPH}, {GLCi, GLCo}))

```python
m.get_edge_data(s21, s13)
```

```python
m.echelon_df['reaction_22']
```

    {species_19}    0.0
    {species_6}     0.0
    {species_12}    0.0
    {species_15}    0.0
    {species_22}    1.0
    {species_3}     0.0
    {species_8}     0.0
    {species_10}    0.0
    {species_9}     0.0
    {species_18}    0.0
    {species_2}     0.0
    {species_20}    0.0
    {species_16}    0.0
    {species_14}    0.0
    {species_17}    0.0
    {species_11}    0.0
    {species_23}    0.0
    {species_1}     0.0
    {species_5}     0.0
    {species_4}     0.0
    {species_13}    1.0
    {species_21}   -1.0
    {species_7}     0.0
    Name: reaction_22, dtype: float64

```python
simple.getReaction('reaction_22')
```

    reaction_22: species_21 -> species_13 + species_22; compartment_1 * k1 * species_21

```python
simple.getReaction('reaction_25')
```

    reaction_25: species_22 -> 3.00 species_8 + species_23; compartment_1 * k1 * species_22

```python
m.som_reactions_lu[14]
```

    reaction_25: {species_22} -> 3.00 {species_8} + {species_23}

```python
m.get_edge_data(s13, s21)
```

    {'reaction': ['reaction_22', 'reaction_25']}

```python
m.get_edge_data(s21, s13)
```

    {'reaction': ['reaction_25']}

```python
m.som_reactions_lu
```

    [reaction_3: {species_4} -> 2.00 {species_5},
     reaction_6: {species_9} -> {species_3} + {species_4},
     reaction_8: {species_10} -> {species_7} + {species_9},
     reaction_9: {species_10} -> {species_6} + {species_11},
     reaction_10: {species_11} -> {species_12} + {species_4},
     reaction_13: {species_14} -> {species_13} + {species_12},
     reaction_15: {species_16} -> {species_14} + {species_15},
     reaction_16: {species_16} -> {species_18} + {species_17},
     reaction_17: {species_17} -> {species_13} + {species_19},
     reaction_20: {species_20} -> {species_17} + {species_15},
     reaction_21: {species_20} -> {species_18} + {species_21},
     reaction_22: {species_21} -> {species_13} + {species_22},
     reaction_23: {species_12} -> {species_8} + {species_23},
     reaction_24: {species_19} -> 2.00 {species_8} + {species_23},
     reaction_25: {species_22} -> 3.00 {species_8} + {species_23},
     reaction_1: {species_1} + {species_2} -> {species_3},
     reaction_2: 2.00 {species_5} -> {species_4},
     reaction_4: {species_8} + {species_6} -> {species_7},
     reaction_5: {species_3} + {species_4} -> {species_9},
     reaction_7: {species_7} + {species_9} -> {species_10},
     reaction_11: {species_8} + {species_18} -> {species_15},
     reaction_12: {species_13} + {species_12} -> {species_14},
     reaction_14: {species_14} + {species_15} -> {species_16},
     reaction_18: {species_13} + {species_19} -> {species_17},
     reaction_19: {species_17} + {species_15} -> {species_20}]

```python
echelon[colname][::-1]
```

    {Br, Ce}    0.0
    {BrO3}      0.0
    {HOBr}      0.0
    {HBrO2}     0.0
    Name: Reaction1, dtype: float64

```python
rref_operation = np.identity(echelon_t.shape[0])
rref_operation = pd.DataFrame(rref_operation,
                  index = echelon_t.index,
                  columns = echelon_t.index)
rref_operation
```

      Reaction4
      Reaction2
      Reaction3
      Reaction1

      Reaction4
      1.0
      0.0
      0.0
      0.0

      Reaction2
      0.0
      1.0
      0.0
      0.0

      Reaction3
      0.0
      0.0
      1.0
      0.0

      Reaction1
      0.0
      0.0
      0.0
      1.0

```python
rref_operation = np.identity(echelon_t.shape[0])
rref_operation = pd.DataFrame(rref_operation,
                  index = echelon_t.index,
                  columns = echelon_t.index)

# now update the operation matrix, finally multiply (dot product) two matrices
echelon = m.echelon_df
for idx, colname in enumerate(echelon.columns):
  reaction_series = echelon[colname]
#   # find the first nonzero values
  nonzero_idx = np.nonzero(echelon[colname])[0]
  # Skip if there is no nonzero value or if it is first reaction
  if not nonzero_idx.any() or idx == 0:
    continue
  nonzero_species = reaction_series.index[nonzero_idx[0]]
  nonzero_value = reaction_series[nonzero_idx[0]]
  # find current nonzero index
  current_echelon = rref_operation.dot(echelon.T).T
  for prev_colname in current_echelon.columns[:idx]:
    print("Colname is: ", colname)
    print("Previous columns are:", prev_colname)
    if np.round(current_echelon[prev_colname][nonzero_species], 3) != 0.0:
      reduction_value = current_echelon[prev_colname][nonzero_species]
      print("We need to reduce ", reduction_value)
      rref_operation.at[prev_colname, colname] = (-1.0) * reduction_value / nonzero_value
rref_operation
```

    Colname is:  Reaction2
    Previous columns are: Reaction4
    We need to reduce  1.0
    Colname is:  Reaction3
    Previous columns are: Reaction4
    We need to reduce  1.3333333333333333
    Colname is:  Reaction3
    Previous columns are: Reaction2
    We need to reduce  -0.5

      Reaction4
      Reaction2
      Reaction3
      Reaction1

      Reaction4
      1.0
      -0.666667
      4.0
      0.0

      Reaction2
      0.0
      1.000000
      -1.5
      0.0

      Reaction3
      0.0
      0.000000
      1.0
      0.0

      Reaction1
      0.0
      0.000000
      0.0
      1.0

```python
np.round(rref_operation.dot(echelon_t), 3)
```

      {HBrO2}
      {HOBr}
      {BrO3}
      {Br, Ce}

      Reaction4
      -2.0
      0.0
      -0.000
      6.000

      Reaction2
      0.0
      1.5
      -0.000
      -3.000

      Reaction3
      0.0
      0.0
      -0.333
      1.333

      Reaction1
      0.0
      0.0
      0.000
      0.000

```python
rref_operation.dot(m.lower_inverse)
```

      Reaction4
      Reaction2
      Reaction3
      Reaction1

      Reaction4
      4.000000
      -2.000000
      4.0
      0.0

      Reaction2
      -1.500000
      1.500000
      -1.5
      0.0

      Reaction3
      0.666667
      -0.333333
      1.0
      0.0

      Reaction1
      1.000000
      -1.000000
      0.0
      1.0

```python
rref_op1 = np.identity(echelon_t.shape[0])
rref_op1 = pd.DataFrame(rref_op1,
          index = echelon_t.index,
          columns = echelon_t.index)
val = 1/1.5*(-1.0)
rref_op1.at['Reaction4', 'Reaction2'] = val
op1 = rref_op1.dot(echelon_t)
op1
rref_op1
```

      Reaction4
      Reaction2
      Reaction3
      Reaction1

      Reaction4
      1.0
      -0.666667
      0.0
      0.0

      Reaction2
      0.0
      1.000000
      0.0
      0.0

      Reaction3
      0.0
      0.000000
      1.0
      0.0

      Reaction1
      0.0
      0.000000
      0.0
      1.0

```python
op1
```

      {HBrO2}
      {HOBr}
      {BrO3}
      {Br, Ce}

      Reaction4
      -2.0
      5.551115e-17
      1.333333
      0.666667

      Reaction2
      0.0
      1.500000e+00
      -0.500000
      -1.000000

      Reaction3
      0.0
      0.000000e+00
      -0.333333
      1.333333

      Reaction1
      0.0
      0.000000e+00
      0.000000
      0.000000

```python
rref_op2 = np.identity(echelon_t.shape[0])
rref_op2 = pd.DataFrame(rref_op2,
          index = echelon_t.index,
          columns = echelon_t.index)
val = op1.loc['Reaction4', '{BrO3}'] / op1.loc['Reaction3', '{BrO3}'] * (-1.0)
rref_op2.at['Reaction4', 'Reaction3'] = val
op2 = rref_op2.dot(op1)
op2
rref_op2
```

      Reaction4
      Reaction2
      Reaction3
      Reaction1

      Reaction4
      1.0
      0.0
      4.0
      0.0

      Reaction2
      0.0
      1.0
      0.0
      0.0

      Reaction3
      0.0
      0.0
      1.0
      0.0

      Reaction1
      0.0
      0.0
      0.0
      1.0

```python
op2
```

      {HBrO2}
      {HOBr}
      {BrO3}
      {Br, Ce}

      Reaction4
      -2.0
      5.551115e-17
      -7.401487e-17
      6.000000

      Reaction2
      0.0
      1.500000e+00
      -5.000000e-01
      -1.000000

      Reaction3
      0.0
      0.000000e+00
      -3.333333e-01
      1.333333

      Reaction1
      0.0
      0.000000e+00
      0.000000e+00
      0.000000

```python
rref_op1
```

      Reaction4
      Reaction2
      Reaction3
      Reaction1

      Reaction4
      1.0
      -0.666667
      1.0
      0.0

      Reaction2
      0.0
      1.000000
      -1.5
      0.0

      Reaction3
      0.0
      0.000000
      1.0
      0.0

      Reaction1
      0.0
      0.000000
      0.0
      1.0

```python
rref_op2
```

      Reaction4
      Reaction2
      Reaction3
      Reaction1

      Reaction4
      1.0
      -0.666667
      1.0
      0.0

      Reaction2
      0.0
      1.000000
      -1.5
      0.0

      Reaction3
      0.0
      0.000000
      1.0
      0.0

      Reaction1
      0.0
      0.000000
      0.0
      1.0

```python
np.round(rref_operation.dot(echelon_t), 3)
```

      {HBrO2}
      {HOBr}
      {BrO3}
      {Br, Ce}

      Reaction4
      -2.0
      0.0
      0.333
      4.667

      Reaction2
      0.0
      1.5
      -0.000
      -3.000

      Reaction3
      0.0
      0.0
      -0.333
      1.333

      Reaction1
      0.0
      0.0
      0.000
      0.000

```python
m.convertMatrixToSOMReactions(np.round(rref_operation.dot(echelon_t), 3).T)
```

    [Reaction4: 2.00 {HBrO2} -> 0.33 {BrO3} + 4.67 {Br, Ce},
     Reaction2: 3.00 {Br, Ce} -> 1.50 {HOBr},
     Reaction3: 0.33 {BrO3} -> 1.33 {Br, Ce},
     Reaction1:  -> ]

```python
rref_operation.dot(echelon_t).apply(np.round)
```

      {HBrO2}
      {HOBr}
      {BrO3}
      {Br, Ce}

      Reaction4
      -2.0
      0.0
      0.0
      5.0

      Reaction2
      0.0
      2.0
      -0.0
      -3.0

      Reaction3
      0.0
      0.0
      -0.0
      1.0

      Reaction1
      0.0
      0.0
      0.0
      0.0

```python
m.convertMatrixToReactions(simple, m.permuted_matrix)
```

    [ReactionSummary(label=Reaction1, reactants=[BrO3 *  1.00, Br *  1.00], products=[HBrO2 *  1.00, HOBr *  1.00], category='reaction_n_n'),
     ReactionSummary(label=Reaction3, reactants=[BrO3 *  1.00], products=[Ce *  1.00, HBrO2 *  1.00], category='reaction_1_n'),
     ReactionSummary(label=Reaction2, reactants=[Br *  1.00, HBrO2 *  1.00], products=[HOBr *  2.00], category='reaction_n_n'),
     ReactionSummary(label=Reaction5, reactants=[Ce *  1.00], products=[Br *  1.00], category='reaction_1_1'),
     ReactionSummary(label=Reaction4, reactants=[HBrO2 *  2.00], products=[BrO3 *  1.00, HOBr *  1.00], category='reaction_n_n')]

```python
m.analyze()
```

    We just analyzed the data...
    RREF error:  []
    Type I error:  []
    Type II error:  []

    True

```python
m
```

    {Ce}->{HOBr}
    {Br}->{HOBr}
    {HOBr}->{HBrO2}
    {BrO3}

```python
# mat_df = m.stoichiometry_matrix
# mat_t = mat_df.T

# mat_t
# idx_mat_t = mat_t.index
# # # LU decomposition
# mat_lu = scipy.linalg.lu(mat_t)
# # # inverse pivot matrix
# p_inv = scipy.linalg.inv(mat_lu[0])
# pivot_index = [list(k).index(1) for k in p_inv]
# new_idx_mat_t = [idx_mat_t[idx] for idx in pivot_index]
# # # row reduced matrix
# row_reduced = pd.DataFrame(mat_lu[2], index=new_idx_mat_t, columns=mat_t.columns)
# row_reduced
```

```python
rref = [1]
er = []
if rref and not er:
  print("haha")
```

    haha

```python
plt.figure(figsize=(20,10))
G = nx.DiGraph()
G.add_edges_from(m.edges)

# Specify the edges you want here

# Need to create a layout when doing
# separate calls to draw nodes and edges
pos = nx.spring_layout(G)
nx.draw_networkx_nodes(G, pos, cmap=plt.get_cmap('jet'),
                       node_size = 500)
nx.draw_networkx_labels(G, pos)
nx.draw_networkx_edges(G, pos, edgelist=G.edges, edge_color='b', width=2.5, arrows=True)
plt.show()
```

```python

```


# How does the SFF method work?

Vanderburg and Johnson (2014) introduced a method for "Self Flat Fielding" by tracking how the lightcurve changes with motion of the spacecraft:

[A Technique for Extracting Highly Precise Photometry for the Two-Wheeled Kepler Mission](http://adsabs.harvard.edu/abs/2014PASP..126..948V)

In this notebook we replicate the K2SFF method following the same example source, #60021426, as that in the publication.  We aim to demystify the technique, which is extremely popular within the K2 community.  We have focused on reproducibility, so that we achieve the same result at the publication.

The Vanderburg & Johnson 2014 paper uses data from the Kepler two-wheel "Concept Engineering Test", predating campaign 0, and sometimes called campaign *"eng"* or abbreviated CET.  This vestigal "campaign" lacks some of the standardization of later K2 campaigns--- it was much shorter, only about 9 days long, it lacks some of the standard quality flags, targets have non-traditional EPIC IDs, and other quirks.

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
from astropy.io import fits
import pandas as pd
```

## Retrieve the K2SFF data for ENG test source `60021426`

First we will retrieve data and inspect the mask used in the paper.

```python
path = 'http://archive.stsci.edu/hlsps/k2sff/cet/060000000/21426/hlsp_k2sff_k2_lightcurve_060021426-cet_kepler_v1_llc.fits'
vdb_fits = fits.open(path)
```

The `BESTAPER` keyword explains which aperture was chosen as the "best" by Vanderburg & Johnson 2014.  The FITS header for that slice contains the metadata needed to reproduce the mask.

```python
keys = ['MASKTYPE', 'MASKINDE', 'NPIXSAP']
_ = [print(key, ' : ', vdb_fits['BESTAPER'].header[key]) for key in keys]
```

    MASKTYPE  :  PRF FIT
    MASKINDE  :  4
    NPIXSAP  :  29.0

We want the *exact same* mask as Vanderburg & Johnson 2014, but the publication version and MAST version differ!

Publication version:

MAST Version:

Aperture 7 should yield a bigger mask, more similar to what was used in the paper.

```python
VDB_J_mask = vdb_fits['PRF_APER_TBL'].data[7,:, :] == True
VDB_J_mask.sum()
```

    53

Save the mask for easy use in our next notebook.

```python
np.save('VDB_J_2014_mask.npy', VDB_J_mask)
```

## Manually reproduce with the Vanderburg-provided diagnostic data
Retrieve the Vanderburg-provided diagnostic data for the Kepler ENG testing.
Uncomment the line below to retrieve the data programmatically, or manually get the linked file in a browser and save it to this directory.

```python
#! wget https://www.cfa.harvard.edu/~avanderb/k2/ep60021426alldiagnostics.csv
```

```python
df = pd.read_csv('ep60021426alldiagnostics.csv',index_col=False)
df.head()
```

      BJD - 2454833
      Raw Flux
      Corrected Flux
      X-centroid
      Y-centroid
      arclength
      Correction
      Thrusters On

      0
      1862.502368
      0.995119
      0.995985
      25.135097
      24.661074
      2.327480
      0.999130
      1.0

      1
      1862.522801
      0.997313
      0.996767
      25.289752
      24.418689
      1.175322
      1.000548
      1.0

      2
      1862.543235
      0.996713
      0.996136
      25.288052
      24.429406
      1.214627
      1.000580
      0.0

      3
      1862.563668
      0.996930
      0.996277
      25.275216
      24.448405
      1.306617
      1.000656
      0.0

      4
      1862.584102
      0.996862
      0.996228
      25.253864
      24.480184
      1.460259
      1.000636
      0.0

We can mean-subtract the provided $x-y$ centroids, assigning them column and row identifiers, then rotate the coordinates into their major and minor axes.

```python
col = df[' X-centroid'].values
col = col - np.mean(col)
row = df[' Y-centroid'].values
row = row - np.mean(row)
```

```python
def _get_eigen_vectors(centroid_col, centroid_row):
    '''get the eigenvalues and eigenvectors given centroid x, y positions'''
    centroids = np.array([centroid_col, centroid_row])
    eig_val, eig_vec = np.linalg.eigh(np.cov(centroids))
    return eig_val, eig_vec
```

```python
def _rotate(eig_vec, centroid_col, centroid_row):
    '''rotate the centroids into their predominant linear axis'''
    centroids = np.array([centroid_col, centroid_row])
    return np.dot(eig_vec, centroids)
```

```python
eig_val, eig_vec = _get_eigen_vectors(col, row)
v1, v2 = eig_vec
```

The major axis is the latter.

```python
platescale = 4.0 # The Kepler plate scale; has units of arcseconds / pixel
```

```python
plt.figure(figsize=(5, 6))
plt.plot(col * platescale, row * platescale, 'ko', ms=4)
plt.plot(col * platescale, row * platescale, 'ro', ms=1)
plt.xticks([-2, -1,0, 1, 2])
plt.yticks([-2, -1,0, 1, 2])
plt.xlabel('X position [arcseconds]')
plt.ylabel('Y position [arcseconds]')
plt.xlim(-2, 2)
plt.ylim(-2, 2)
plt.plot([0, v1[0]], [0, v1[1]], color='blue', lw=3)
plt.plot([0, v2[0]], [0, v2[1]], color='blue', lw=3);
```

Following the form of **Figure 2** of Vanderburg & Johsnon 2014.

```python
rot_colp, rot_rowp = _rotate(eig_vec, col, row) #units in pixels
```

You can rotate into the new reference frame.

```python
plt.figure(figsize=(5, 6))
plt.plot(rot_rowp * platescale, rot_colp * platescale, 'ko', ms=4)
plt.plot(rot_rowp * platescale, rot_colp * platescale, 'ro', ms=1)
plt.xticks([-2, -1,0, 1, 2])
plt.yticks([-2, -1,0, 1, 2])
plt.xlabel("X' position [arcseconds]")
plt.ylabel("Y' position [arcseconds]")
plt.xlim(-2, 2)
plt.ylim(-2, 2)
plt.plot([0, 1], [0, 0], color='blue')
plt.plot([0, 0], [0, 1], color='blue');
```

We need to calculate the arclength using:
\begin{equation}s= \int_{x'_0}^{x'_1}\sqrt{1+\left( \frac{dy'_p}{dx'}\right)^2} dx'\end{equation}

where $x^\prime_0$ is the transformed $x$ coordinate of the point with the smallest $x^\prime$ position, and $y^\prime_p$ is the best--fit polynomial function.

Fit a $5^{th}$ order polynomial to the rotated coordinates.

```python
z = np.polyfit(rot_rowp, rot_colp, 5)
p5 = np.poly1d(z)
p5_deriv = p5.deriv()
```

```python
x0_prime = np.min(rot_rowp)
xmax_prime = np.max(rot_rowp)
x_dense = np.linspace(x0_prime, xmax_prime, 2000)
```

```python
plt.plot(rot_rowp, rot_colp, '.')
plt.plot(x_dense, p5(x_dense))
plt.ylabel('Position along minor axis (pixels)')
plt.xlabel('Position along major axis (pixels)')
plt.title('Performance of polynomial regression')
plt.ylim(-0.1, 0.1);
```

We see evidence for a [bias-variance tradeoff](https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff), suggesting some modest opportunity for improvement.

```python
@np.vectorize
def arclength(x):
    '''Input x1_prime, get out arclength'''
    gi = x_dense <x
    s_integrand = np.sqrt(1 + p5_deriv(x_dense[gi]) ** 2)
    s = np.trapz(s_integrand, x=x_dense[gi])
    return s
```

Let's double check that we compute the same arclength as the published paper.

```python
aspect_ratio = plt.figaspect(1)
plt.figure(figsize=aspect_ratio)
plt.plot(df[' arclength'], arclength(rot_rowp)*4.0, '.')
plt.xlabel('$s$  (Vanderburg & Johnson 2014)')
plt.ylabel('$s$  (This work)')
plt.plot([0, 4], [0, 4], 'k--');
```

Yes, we compute arclength correctly.

Now we apply a **high-pass filter** to the raw lightcurve data.  We follow the original paper by using *BSplines* with 1.5 day breakpoints.  You can also apply data exclusion at this stage.

```python
from scipy.interpolate import BSpline
from scipy import interpolate
```

```python
times, raw_fluxes = df['BJD - 2454833'].values, df[' Raw Flux'].values
```

We find the weighted least square spline for a given set of knots, $t$.  We supply interior knots as knots on the ends are added automatically, as stated in the `interpolate.splrep()` docstring.

```python
interior_knots = np.arange(times[0]+1.5, times[0]+6, 1.5)
t,c,k = interpolate.splrep(times, raw_fluxes, s=0, task=-1, t=interior_knots)
bspl = BSpline(t,c,k)
```

```python
plt.plot(times, raw_fluxes, '.')
plt.plot(times, bspl(times))
plt.xlabel('$t$ (days)')
plt.ylabel('Raw Flux');
```

The Spline fit looks good, so we can normalize the flux by the long-term trend.
Plot the normalized flux versus arclength to see the position-dependent flux.

```python
fluxes = raw_fluxes/bspl(times)
```

Mask the data by keeping only the good samples.

```python
bi = df[' Thrusters On'].values == 1.0
gi = df[' Thrusters On'].values == 0.0
clean_fluxes = fluxes[gi]
al = arclength(rot_rowp[gi]) * platescale
```

```python
sorted_inds = np.argsort(al)
```

We will follow the paper by interpolating **flux versus arclength position** in 15 bins of means, which is a *piecewise linear fit*.

```python
knots = np.array([np.min(al)]+
                 [np.median(splt) for splt in np.array_split(al[sorted_inds], 15)]+
                 [np.max(al)])
```

```python
bin_means = np.array([clean_fluxes[sorted_inds][0]]+
                     [np.mean(splt) for splt in np.array_split(clean_fluxes[sorted_inds], 15)]+
                     [clean_fluxes[sorted_inds][-1]])
```

```python
zz = np.polyfit(al, clean_fluxes,6)
sff = np.poly1d(zz)
al_dense = np.linspace(0, 4, 1000)
interp_func = interpolate.interp1d(knots, bin_means)
```

```python
plt.figure(figsize=(5, 6))
plt.plot(arclength(rot_rowp)*4.0, fluxes, 'ko', ms=4)
plt.plot(arclength(rot_rowp)*4.0, fluxes, 'o', color='#3498db', ms=3)
plt.plot(arclength(rot_rowp[bi])*4.0, fluxes[bi], 'o', color='r', ms=3)
plt.plot(np.sort(al), interp_func(np.sort(al)), '-', color='#e67e22')

plt.xticks([0, 1,2, 3, 4])
plt.minorticks_on()
plt.xlabel('Arclength [arcseconds]')
plt.ylabel('Relative Brightness')
plt.title('EPIC 60021426, Kp =10.3')
plt.xlim(0,4)
plt.ylim(0.997, 1.002);
```

Following **Figure 4** of Vanderburg & Johnson 2014.

Apply the Self Flat Field (SFF) correction:

```python
corr_fluxes = clean_fluxes / interp_func(al)
```

```python
plt.figure(figsize=(10,6))

dy = 0.004
plt.plot(df['BJD - 2454833'], df[' Raw Flux']+dy, 'ko', ms=4)
plt.plot(df['BJD - 2454833'], df[' Raw Flux']+dy, 'o', color='#3498db', ms=3)
plt.plot(df['BJD - 2454833'][bi], df[' Raw Flux'][bi]+dy, 'o', color='r', ms=3)
plt.plot(df['BJD - 2454833'][gi], corr_fluxes*bspl(times[gi]), 'o', color='k', ms = 4)
plt.plot(df['BJD - 2454833'][gi], corr_fluxes*bspl(times[gi]), 'o', color='#e67e22', ms = 3)

plt.xlabel('BJD - 2454833')
plt.ylabel('Relative Brightness')

plt.xlim(1862, 1870)
plt.ylim(0.994, 1.008);
```

Following **Figure 5** of Vanderburg & Johnson 2015.

*The end.*


# Introduction
> Introduction to Radio Astronomy, Spectrometers and DESHIMA

- toc: true
- badges: false
- comments: true
- permalink: /introduction
- image: images/introduction.jpg
- sectionnr: 2

The ultimate weapon in the arsenal of a historian just might be a time machine. While historians can infer a lot from written texts and archaeological digs, no substitute can outperform being in the past and observing. Unfortunately for historians, a time machine has yet to be invented. Astronomers, however, are more fortunate in this field.

Around the turn of the century, new physics was being invented to explain previous observations and conclusions. As a result, it was understood that the speed of light was finite and moreover a constant, independent of the speed of the observer, which Albert Einstein postulated in 1905{% cite einstein_1905 %}. It logically follows that the arriving light from distant stars and galaxies is reaching us from the past.

Since light is a wave traveling through the expanding universe, it undergoes Doppler shift and therefore cosmological objects that move away from the observer undergo a shift in the frequency of light. This is known as redshift, because the expanding wavelength means that visible light would shift towards the red end of the spectrum. Just 12 years after Einstein's discovery, American Astronomer Vesto Slipher {% cite slipher_1917 %} discovered the redshift of distant galaxies and with it cosmological spectroscopy was born.

***A part of the light spectrum. Visible light is expanded to show the different colors. Note how red light in the visible spectrum has a longer wavelength, giving redshift its name. Image taken from {% cite lemke_2020 %}***

Combining these effects gives us a sort of astronomical time machine: looking far out into the cosmos means looking into the past where the universe was much younger. However, just pointing a telescope at the night sky poses a few problems.

## Cosmological formation

Understanding how stars and galaxies form out of a uniformly gaseous universe is one of the largest areas of research in modern physical cosmology. The physical processes by which these happen might be well understood, but the amount of possible initial values of these processes are so vast that they can have wildly different outcomes{% cite blain_2002 %}. Looking at these events occurring in the distant universe (and therefore distant past) is therefore important to further the understanding of the forming of the universe.

Unfortunately, much of the formation of these cosmological bodies happens inside thick dust clouds which are opaque to visible light and are therefore investigated using what is called sub-millimeter astronomy: astronomy ranging from $100\:\mathrm{\mu m}$ to about $1\:\mathrm{mm}$. In frequency terms, following

$$\begin{equation}
f = \frac{c}{\lambda}
\end{equation}$$

this corresponds with about $150\:\mathrm{GHz}$ to $1500\:\mathrm{GHz}$.

***The Messer-16 nebula, dubbed "The Eagle", captured by the Hubble Space Telescope is an example of a dust cloud that is opaque to visible light. Image taken from {% cite garner_2017 %}***

## Redshift

As mentioned before, the redshift of light gives us the opportunity to calculate the relative speed and therefore distance and age of astronomical sources. By the term "redshift", I mean the following formal definition {% cite endo_session_2 %}

$$\begin{equation}
1+z = \frac{f_\mathrm{emit}}{f_\mathrm{obs}}
\end{equation}$$

where $z$ denotes redshift and $f_\mathrm{emit}$ and $f_\mathrm{obs}$ denote the emitted and observed frequency of a signal respectively. In order to calculate redshift we therefore need to know both frequencies.

### Emitted frequency

Quantum mechanics tells us that the electron configuration in atoms exist in quantized states. That is to say: they can move between defined states, but not in the continuum inbetween. When an atom moves from one configuration to a higher state energy is absorbed, when it moves to a lower state energy is emitted. These quantized energy states give way to what is known as spectral lines, where a specific wavelength is known to be emitted by specific atoms.

***A few energy states of carbon. Note how higher energy gaps result in shorter wavelengths. Image taken from {% cite nave_hyperphysics %}***

The energy gaps shown are to wide to emit photons in the submillimeter wavelength range DESHIMA operates in, however *fine structure* spectral lines are in the correct wavelength gap. This is the splitting of two similar states due to spin and relativistic effects. Another mechanism by which molecules emit spectral lines is due to the shifting in quantized rotational states. The various states astronomical sources split between are well known and therefore can be tracked to measure redshift.

This becomes clear when we look at the spectral flux density $E_{e,\nu}$ of a galaxy, simulated with the Python module `galspec`{% cite galspec %}.

```python
#hide
!pip install -q galspec numpy pandas plotly "deshima-sensitivity==0.3.0"

import numpy as np
import pandas as pd
import plotly.express as px

import warnings
warnings.filterwarnings('ignore')

from thesis_plot import plot

# Page Specific
import galspec
from deshima_sensitivity.atmosphere import eta_atm_func
```

```python
#hide
h = 6.62607004 * 10**-34
k_b = 1.38064852 * 10**-23
e = 1.60217662 * 10**-19 # electron charge
c = 299792458.
```

```python
#hide_input
fLow = 30 #GHz
fHigh = 1000 #GHz
numFreqBins = 1000
linewidth = 600

z = [3,5,7,11]

gal_freq, gal_flux_density = galspec.spectrum(12, 1, fLow, fHigh, numFreqBins, linewidth, mollines = 'True')

df = pd.DataFrame({'freq':gal_freq, '1': gal_flux_density})

for zi in z:
    gal_freq, gal_flux_density = galspec.spectrum(12, zi, fLow, fHigh, numFreqBins, linewidth, mollines = 'True')
    df = df.join(pd.DataFrame({str(zi): gal_flux_density}))

df.set_index('freq', inplace=True)
plot(df,axis_labels=('$\\nu\:\mathrm{[GHz]}$','$E_{e,\\nu}\:\mathrm{[Jy]}$','$z$'),hover_labels=('nu [GHz]','Spectral flux density [Jy]','z'),e_notation=(False,True),plot_type='loglog')
```

***The spectral flux density of a simulated galaxy with different redshifts. Here the spectral flux is given in units of Jansky, which is equivalent to 10-26 WM-2Hz-1***

From the figure it is clear that with higher $z$ the peaks move further to the left, towards lower frequencies and therefore longer wavelengths. If one were to look for peaks in incoming signal and measure the wavelength at which these peaks occur, the redshift and thus age of a galaxy can be detected. Besides these clear spikes in the flux density, the galaxy also emits broad spectrum thermal noise, originating from the black-body emission of interstellar dust.

### Observed frequency

In order to know the wavelength of these spectral lines at our detection point, we need to measure the specific frequency of an incoming photon in a spectrometer. This can be done using a *Coherent Heterodyne receiver*, except that these devices work to a bandwidth of about $36\:\mathrm{GHz}${% cite baker_2007 %}, much lower than the spectral range we are interested in as shown above.

Traditional photon detectors, at their core, work by converting an incoming photon into a charge or current, which means that the specific momentum, energy or frequency of the photon is lost. A way around this is to move the incoming photons depending on their wavelength, such as what happens in an optical prism.

***When a photon enters the prism, its angle changes depending on its wavelength. Image taken from {% cite edmund_optics%}***

This is how a lot of spectrometers work, but due to the long wavelengths and wide redshift range $(1+z\approx1-10)${% cite endo_2019 %} of the signals involved in galaxy spectroscopy this approach is cumbersome. The long wavelengths and range means that the spectrometer needs to physically large and to reduce the noise they need to be cooled, making the entire system even larger. This means that these devices are inherently not scalable to multiple pixels.

## DESHIMA
DESHIMA, short for the Deep Spectroscopic High-Redshift Mapper, is a spectrometer developed by a joint team of researchers from various institutions including the TU Delft and SRON that takes another approach{% cite endo_2019 %}{% cite endo_2019_2 %}. DESHIMA uses a series of bandpass filters on a superconducting chip to direct incoming photons into bins, where they are detected by Microwave Kinetic Inductance Detectors (MKIDs). The advantage of this approach is that the spectrometer itself is really small, taking up just a few squared centimeters and is therefore easily scalable to $\gt 100$ spectrometer pixels{% cite endo_2019 %}.

***A schematic overview of the DESHIMA 2.0 chip. Image taken from {% cite huijten_2020%}***

DESHIMA is placed on ASTE, a ground-based sub-millimeter telescope near the ALMA site in the Atacama Desert.

***A panoramic shot from the ASTE site in the Atacama Desert. The building on the left is NANTEN-2 whereas the building on the left is ASTE. Image taken from {% cite ASTE_panorama %}***

## Ground-based submillimeter astronomy

While the ASTE site is high up in the Atacama desert, the telescope still needs to look through some atmosphere when doing observations. Since cosmological object of interest are often far away and therefore very dim, they are much dimmer than the effects from the atmosphere. The atmosphere is almost completely opaque at some frequencies DESHIMA operates in and this loads the antenna dish because it is much hotter and therefore brighter than the astronomical source.

Below the atmospheric loading at the ASTE site is calculated by the software package `deshima-sensitivity`{% cite deshima-sensitivity%} and plotted with a galaxy spectrum synthesized by `galspec`:

```python
#hide
pwv = 1

eta_A = 0.6 # aperture efficiency
telescope_diameter = 10
A_p = (telescope_diameter/2)**2 * np.pi # telescope physical area
A_e = eta_A * A_p  # telescope effective area

F, S_gal = galspec.spectrum(luminosity=13.7, redshift=4.43, fLow=fLow, fHigh=fHigh, numFreqBins=numFreqBins, linewidth=linewidth)

eta_atm = eta_atm_func(F=F, pwv=pwv, EL=90)
T_p_atm = 273.
S_sky = (1-eta_atm) * T_p_atm * 1e26 * 2 *k_b /A_e

S_gal_gnd = S_gal*eta_atm

```

```python
#hide_input
df = pd.DataFrame({'F':F, 'Atmosphere': S_sky, 'Galaxy above atmosphere': S_gal, 'Galaxy from ground': S_gal_gnd})
df = df.set_index('F')
plot(df,axis_labels=('$\\nu\:\mathrm{[GHz]}$','$E_{e,\\nu}\:\mathrm{[Jy]}$',''),hover_labels=('nu [GHz]','Spectral flux density [Jy]',''),e_notation=(False,False),plot_type='loglin',y_range=[-4,5])
```

***Comparing the spectral flux density of atmospheric loading to loading from an astronomical source. The former is over 3 orders of magnitude brighter. Note how the atmospheric transmission means that for some frequencies the spectrum is barely usable.***

The atmosphere is over 3 orders of magnitude brighter than the galaxy spectrum before the latter is attenuated. After the atmospheric attenuation the difference is even bigger. Therefore modeling the expected loading from the atmosphere and the entire chain from antenna to detector is important to be able to do measurements and to find the signal within the noise. `deshima-sensitivity`{% cite deshima-sensitivity%} was designed to do this, but takes some approximations that were deemed too reductive.

## This Project

The current version of `deshima-sensitivity` does not take filter profiles into consideration when calculating the power received on each channel. It samples the power spectral density calculated to be at the filter's center frequency and multiplies this with the filter bandwidth, creating a crude box filter. While this approach holds for flat power spectra, it doesn't quite hold for spectra like the one above.

In this project I will modify the existing `deshima-sensitivity` package to more accurately resemble the DESHIMA 2.0 filter profiles and therefore the spectrometer as a whole. In order to do so, I will first need to dive into the DESHIMA system in more depth and then give an overview of *photon statistics* and the associated photon noise. Finally I will model the system and compare it with previous versions of `deshima-sensitivity`.

Next Chapter

# Bibliography
{% bibliography --cited_in_order %}


```python
%%capture
## compile PyRoss for this notebook
import os
owd = os.getcwd()
os.chdir('../../')
%run setup.py install
os.chdir(owd)
%matplotlib inline
```

```python
import numpy as np
import matplotlib.pyplot as plt
import pyross
```

In this notebook we consider a control protocol consisting of an initial lockdown, which is then partly released. For our numerical study we generate synthetic data using the stochastic SEAI5R model.

While we use the Denmark age structure and contact matrix, we emphasise that **the model parameters considered in this notebook have NOT been obtained from real data, but rather are chosen ad-hoc**.

**Outline of this notebook:**

1. We briefly summarize the SEAI5R model.
2. We load the age structure and contact matrix for Denmark. The contact matrix is generally given as
\begin{equation}
    C = C_H + C_W + C_S + C_O,
\end{equation}
where the four terms denote the number of contacts at home, work, school, and all other remaining contacts.
3. We define the other model parameters of the SEAI5R model **(again, these are not fitted to any real data, but rather chosen ad-hoc)**.
4. We define a "lockdown-protocol":
    1. After a fixed time, a lockdown is imposed. The contact matrix is reduced to
    \begin{equation}
        C = C_H + 0.1\cdot C_W + 0.4 \cdot C_O,
        \end{equation}
        i.e. the "home" contacts $C_H$, and 10% of the "work" as well as 40% of the "other" contacts are reainted. The latter two model that even in lockdown, people go to work, and that initially people might be lenient in following the lockdown advice.
    2. 5 Days after the initial lockdown, the 40% "social interaction" part of the contact matrix is removed, and the contact matrix becomes
        \begin{equation}
        C = C_H + 0.1\cdot C_W.
        \end{equation}
        This models that people take the lockdown seriously and seize to interact socially.
    3. 50 Days after the initial lockdown, 50% of the "school" contacts are added to the contact matrix,
        \begin{equation}
        C = C_H + 0.1\cdot C_W + 0.5 \cdot C_S.
        \end{equation}
        This models an opening of schools, with a social distancing protocol in place such that the number of contacts at school is halved.
5. We run 150 stochastic simulations of this lockdown protocol, and find that in Stage 2 (only home and 10% of work contacts), the total number of known active infectives (the sum of the stages symptomatic infective, hospitalised, and in ICU) shows a maximum and then decreases. After schools are opened, the total number of known active infectives shows a minimum and starts to decrease again. We plot the distribution of waiting time between change in the contact matrix and the extrema in the stochastic time series.
6. Finally, we consider the protocol from point 4 again, but this time with a prefactor 0.2 in front of $C_S$, signifying a more strict social distancing protocol. Running 150 stochastic simulations with this protocol, we find that this time the total number of active infectives keeps decreases even after schools have been opened.

## 1. Brief summary of the model

The SEAI5R model contains the following stages:

If a susceptible individual is infected, it enters the "exposed" class, where it does not show symptoms and is not yet infective. In the subsequent class, called "activated", the individual still does not show symptoms, but is infective. After the "activated" class, an individual is either a symptomatic or an asymptomatic infective. Those who are asymptomatic recover, those who show symptoms either recover, or need to be hospitalised. The condition of a hospitalised individual might improve, so that the individual recovers, or worsens, at which point the individual enters the "ICU" class. From this class the individual can again recover, or enter a final "mortality" class.

## 2. Load Denmark age structure and contact matrix

```python
my_data = np.genfromtxt('../data/age_structures/Denmark-2019.csv', delimiter=',', skip_header=1)
aM, aF = my_data[:, 1], my_data[:, 2]

Ni0=aM+aF;

M=16  ## number of age classes

Ni = Ni0[:M]
N=np.sum(Ni)

print("Age groups are in brackets of 5 (i.e. 0-4, 5-9, 10-14, .. , 75-79).")
print("Number of individuals in each bracket:")
print(Ni.astype('int'))
print("Total number of individuals: {0}".format(np.sum(Ni.astype('int'))))
```

    Age groups are in brackets of 5 (i.e. 0-4, 5-9, 10-14, .. , 75-79).
    Number of individuals in each bracket:
    [302353 305513 338779 341219 379522 395469 342443 320132 366147 385944
     422585 381360 338039 319145 346572 220374]
    Total number of individuals: 5505596

```python
# Get individual contact matrices
CH, CW, CS, CO = pyross.contactMatrix.Denmark()

# By default, home, work, school, and others contribute to the contact matrix
C = CH + CW + CS + CO

# Illustrate the individual contact matrices:
fig,aCF =  plt.subplots(2,2,figsize=(10,10));
aCF[0][0].pcolor(CH, cmap=plt.cm.get_cmap('GnBu', 10));
aCF[0][1].pcolor(CW, cmap=plt.cm.get_cmap('GnBu', 10));
aCF[1][0].pcolor(CS, cmap=plt.cm.get_cmap('GnBu', 10));
aCF[1][1].pcolor(CO, cmap=plt.cm.get_cmap('GnBu', 10));
for i in range(2):
    for j in range(2):
        aCF[i][j].set_xlabel('Age group of individual')
        aCF[i][j].set_ylabel('Age group of contact')
```

The above contact matrices illustrate the interactions at home (upper left), work (upper right), school (lower left), and the other remaining contacts (lower right). x- and y- axes denote the age groups, a darker color indicates more interaction.

## 3. Define model parameters

**Note: These have not been fitted to real data.**

```python
beta  = 0.036692            # infection rate

gE    = 1/5.
gA    = 1/4.
gIa   = 1./3.                # recovery rate of asymptomatic infectives
gIs   = 1./7                # recovery rate of symptomatic infectives
alpha = 0.4                 # fraction of asymptomatic infectives
fsa   = 0.2                 # the self-isolation parameter
fh    = 0
gIh   = 1/7
gIc   = 1/14

sa    = 0*np.ones(M)      #  rate of additional/removal of population by birth etc
sa[0]     = 0            # birth
sa[3] = 0          # mortality

hh   = 0.1*np.ones(M)       # fraction which goes from Is to hospital
hh[:M] = 0.01
hh[M:2*M] = 0.025
cc   = 0.05*np.ones(M)      # fraction which goes from  hospital to ICU
mm   = 0.4*np.ones(M)       # mortality from IC

# initial conditions
Is_0 = np.zeros((M));

Ia_0 = 10*np.ones((M)) # start with 10 asymptomatic infectives
R_0  = np.zeros((M))
E_0  = np.zeros((M))
A_0  = np.zeros((M))
Ih_0  = np.zeros((M))
Ic_0  = np.zeros((M))
Im_0  = np.zeros((M))

S_0  = Ni - (E_0 + A_0 + Ia_0 + Is_0 + Ih_0 + Ic_0 +Im_0 + R_0)
```

## 4. Define events for protocol

```python
# Dummy event for initial (standard) contact matrix
events = [lambda t: 1]
contactMatrices = [C]

# After 20 days, start lockdown
lockdown_threshold_0 = 20
def event0(t,rp):
    return t - lockdown_threshold_0
events.append(event0)
contactMatrices.append( CH + 0.1*CW + 0.4*CO )

# After 25 days, decrease contacts even further
lockdown_threshold_1 = 25
def event1(t,rp):
    return t- lockdown_threshold_1
events.append(event1)
contactMatrices.append( CH + 0.1*CW)

# After 70 days, add 50% of school contacts to contact matrix
lockdown_threshold_2 = 70
def event2(t,rp):
    return t - lockdown_threshold_2
events.append(event2)
contactMatrices.append( CH + 0.1*CW + 0.5*CS ) # school opens, with social distancing
```

## 5. Simulate protocol and analyse results

#### Initialise pyross.control, run and plot a single test simulation

```python
# duration of simulation
Tf=150;  Nf=Tf+1;

# intantiate model
parameters = {'alpha':alpha,'beta':beta, 'gIa':gIa,'gIs':gIs,
              'gIh':gIh,'gIc':gIc,  'gE':gE, 'gA':gA,
              'fsa':fsa, 'fh':fh,
              'sa':sa,
              'hh':hh, 'cc':cc, 'mm':mm}
model = pyross.control.SEAI5R(parameters, M, Ni.copy())

# run model once
method='gillespie'
data=model.simulate(S_0, E_0, A_0, Ia_0, Is_0, Ih_0, Ic_0, Im_0,
                    events,contactMatrices, Tf, Nf,
                                method=method)
```

```python
# Plot result

t = data['t'];

# get total population in classes
# - symptomatic infective,
# - hospitalised,
# - in ICU
y_plot = np.sum ( data['X'][:,4*M:7*M], axis = -1)

lw=2
fig,ax = plt.subplots(1,1,figsize=(8,5))
ax.plot(t,y_plot,lw=lw,)
ax.axvline(data['events_occured'][0][0],
           color='crimson',lw=lw,
          label='Beginning of lockdown',ls='--')
ax.axvline(data['events_occured'][-1][0],
 color='limegreen',lw=lw,
          label='Schools re-opened',ls='--')
ax.set_xlim(0,Tf)
fs=20
ax.legend(loc='best',fontsize=fs,framealpha=1)
ax.set_xlabel('time since first infection [days]',fontsize=fs)
ax.set_ylabel('Number of known active cases',fontsize=fs)
plt.show()
plt.close()
```

#### Run 150 simulations with protocol, then

* plot trajectories,
* plot mean trajectory + standard deviation,
* plot distribution of delay between interventions (lockdown, re-opening of schools) until subsequent maximum/minimum.

```python
# Run 150 simulations with this protocol
N_simulations = 150

data_results = np.zeros([N_simulations,Nf,M*9],
                        dtype=int)
for i in range(N_simulations):
    print('Running simulation {0} of {1}'.format(i+1,N_simulations),end='\r')
    data=model.simulate(S_0, E_0, A_0, Ia_0, Is_0, Ih_0, Ic_0, Im_0,
                    events,contactMatrices, Tf, Nf,
                                method=method)
    data_results[i] = data['X']
```

    Running simulation 150 of 150

```python
# Plot result

t = data['t'];
fig,ax = plt.subplots(1,1,figsize=(8,5))
for i,e in enumerate(data_results):
    # get total population in classes
    # - symptomatic infective,
    # - hospitalised,
    # - in ICU
    y_plot = np.sum ( e[:,4*M:7*M], axis = -1)
    ax.plot(t,y_plot,lw=0.5,)
ax.axvline(data['events_occured'][0][0],
               color='crimson',lw=lw,
              label='Beginning of lockdown',ls='--')
ax.axvline(data['events_occured'][-1][0],
 color='limegreen',lw=lw,
          label='Schools re-opened',ls='--')
ax.set_xlim(0,Tf)
fs=20
ax.legend(loc='best',fontsize=fs,framealpha=1)
ax.set_xlabel('time since first infection [days]',fontsize=fs)
ax.set_ylabel('Number of known active cases',fontsize=fs)
plt.show()
plt.close()
```

#### Mean and standard deviation

```python
# For all simulations, plot total population in classes
# - symptomatic infective,
# - hospitalised,
# - in ICU

# mean and standard deviation of trajectories at every time
y_mean = np.mean( ( np.sum(data_results[:,:,4*M:7*M], axis = -1) ), axis=0)
y_std = np.std ( np.sum( data_results[:,:,4*M:7*M], axis = -1), axis =0)

t = data['t'];
fig,ax = plt.subplots(1,1,figsize=(8,5))
ax.fill_between(t,y_mean - y_std,y_mean+y_std,
                label='Standard deviation',
           color='grey',alpha=0.4)
ax.plot(t,y_mean,
        label='Mean',
           color='black',alpha=1)
ax.axvline(data['events_occured'][0][0],
               color='crimson',lw=lw,
              label='Beginning of lockdown',ls='--')
ax.axvline(data['events_occured'][-1][0],
 color='limegreen',lw=lw,
          label='Schools re-opened',ls='--')
ax.set_xlim(0,Tf)
fs=20
ax.legend(loc='upper left',fontsize=17,framealpha=1)
ax.set_xlabel('time since first infection [days]',fontsize=fs)
ax.set_ylabel('Number of known active cases',fontsize=fs)
plt.show()
plt.close()
```

#### Plot distributions for both

* time from beginning of lockdown to subsequent peak, and
* time from opening of schools to subsequent minimum.

```python
# calculate time from lockdown to local maximum for each trajectory
mask = (t > lockdown_threshold_0)*(t < lockdown_threshold_2)
t_mask = t[(mask)]
durations_from_lockdown_to_local_maximum = np.zeros(N_simulations,
                                                   dtype=float)
for i,e in enumerate(data_results):
    y_plot = np.sum ( e[:,4*M:7*M], axis = -1)
    index = np.argmax( y_plot[(mask)])
    durations_from_lockdown_to_local_maximum[i] = t_mask[index] - lockdown_threshold_0
print('Mean time from beginning of lockdown to subsequent peak = {0:3.1f} days'.format(np.mean(durations_from_lockdown_to_local_maximum)))

hist, bin_edges = np.histogram(durations_from_lockdown_to_local_maximum,10 ,density=True)
bin_centers = (bin_edges[1:] + bin_edges[:-1])/2.
bin_widths = bin_edges[1]-bin_edges[0]
fig, ax =plt.subplots(1,1,figsize=(10,6))
ax.axvline(np.mean(durations_from_lockdown_to_local_maximum),ls='--',color='black',
          label='Mean')
ax.bar(bin_centers,hist,bin_widths,color='dodgerblue',
       label='Distribution',alpha=0.7)
ax.set_xlabel('Time from beginning of lockdown until subsequent peak [days]',fontsize=fs)
ax.set_ylabel('Probability',fontsize=fs)
ax.legend(loc='best',fontsize=fs)
ax.set_xlim(0,30)
plt.show()
plt.close()

# calculate time from opening of schools to local minimum for each trajectory
# (recall lockdown_threshold_2 = time at which schools are opened)
mask = (t > lockdown_threshold_2)*(t < Tf)
t_mask = t[(mask)]
durations_from_school_opening_to_local_minimum = np.zeros(N_simulations,
                                                   dtype=float)
for i,e in enumerate(data_results):
    y_plot = np.sum ( e[:,4*M:7*M], axis = -1)
    index = np.argmin( y_plot[(mask)])
    durations_from_school_opening_to_local_minimum[i] = t_mask[index] - lockdown_threshold_2
print('Mean time from opening of schools to subsequent minimum = {0:3.1f} days'.format(np.mean(durations_from_school_opening_to_local_minimum)))

hist, bin_edges = np.histogram(durations_from_school_opening_to_local_minimum,10 ,density=True)
bin_centers = (bin_edges[1:] + bin_edges[:-1])/2.
bin_widths = bin_edges[1]-bin_edges[0]
fig, ax =plt.subplots(1,1,figsize=(10,6))
ax.axvline(np.mean(durations_from_school_opening_to_local_minimum),ls='--',color='black',
          label='Mean')
ax.bar(bin_centers,hist,bin_widths,color='dodgerblue',
       label='Distribution',alpha=0.7)
ax.set_xlabel('Time from school re-opening until subsequent minimum [days]',fontsize=fs)
ax.set_ylabel('Probability',fontsize=fs)
ax.legend(loc='best',fontsize=fs)
ax.set_xlim(0,30)
plt.show()
plt.close()
```

## 6. Redo simulations and analysis of point 5, but with more social distancing at school

```python
# we modify only the last contact matrix of the previous protocol:
contactMatrices[-1] =  CH + 0.1*CW + 0.2*CS
```

```python
# Run 150 simulations with this protocol
N_simulations = 150

data_results2 = np.zeros([N_simulations,Nf,M*9],
                        dtype=int)
for i in range(N_simulations):
    print('Running simulation {0} of {1}'.format(i+1,N_simulations),end='\r')
    data=model.simulate(S_0, E_0, A_0, Ia_0, Is_0, Ih_0, Ic_0, Im_0,
                    events,contactMatrices, Tf, Nf,
                                method=method)
    data_results2[i] = data['X']
```

    Running simulation 150 of 150

```python
# Plot results

print("Simulated trajectories:")
t = data['t'];
fig,ax = plt.subplots(1,1,figsize=(8,5))
for i,e in enumerate(data_results2):
    # get total population in classes
    # - symptomatic infective,
    # - hospitalised,
    # - in ICU
    y_plot = np.sum ( e[:,4*M:7*M], axis = -1)
    ax.plot(t,y_plot,lw=0.5,)
ax.axvline(data['events_occured'][0][0],
               color='crimson',lw=lw,
              label='Beginning of lockdown',ls='--')
ax.axvline(data['events_occured'][-1][0],
 color='limegreen',lw=lw,
          label='Schools re-opened',ls='--')
ax.set_xlim(0,Tf)
fs=20
ax.legend(loc='best',fontsize=15,framealpha=1)
ax.set_xlabel('time since first infection [days]',fontsize=fs)
ax.set_ylabel('Number of known active cases',fontsize=fs)
plt.show()
plt.close()

print("Mean trajectory and standard deviation:")
# mean and standard deviation of trajectories at every time
y_mean = np.mean( ( np.sum(data_results2[:,:,4*M:7*M], axis = -1) ), axis=0)
y_std = np.std ( np.sum( data_results2[:,:,4*M:7*M], axis = -1), axis =0)
t = data['t'];
fig,ax = plt.subplots(1,1,figsize=(8,5))
ax.fill_between(t,y_mean - y_std,y_mean+y_std,
                label='Standard deviation',
           color='grey',alpha=0.4)
ax.plot(t,y_mean,
        label='Mean',
           color='black',alpha=1)
ax.axvline(data['events_occured'][0][0],
               color='crimson',lw=lw,
              label='Beginning of lockdown',ls='--')
ax.axvline(data['events_occured'][-1][0],
 color='limegreen',lw=lw,
          label='Schools re-opened',ls='--')
ax.set_xlim(0,Tf)
fs=20
ax.legend(loc='upper right',fontsize=15,framealpha=1)
ax.set_xlabel('time since first infection [days]',fontsize=fs)
ax.set_ylabel('Number of known active cases',fontsize=fs)
plt.show()
plt.close()

print("Time from lockdown to local maximum:")
# calculate time from lockdown to local maximum for each trajectory
mask = (t > lockdown_threshold_0)*(t < lockdown_threshold_2)
t_mask = t[(mask)]
durations_from_lockdown_to_local_maximum2 = np.zeros(N_simulations,
                                                   dtype=float)
for i,e in enumerate(data_results2):
    y_plot = np.sum ( e[:,4*M:7*M], axis = -1)
    index = np.argmax( y_plot[(mask)])
    durations_from_lockdown_to_local_maximum2[i] = t_mask[index] - lockdown_threshold_0
print('Mean time from beginning of lockdown to subsequent peak = {0:3.1f} days'.format(np.mean(durations_from_lockdown_to_local_maximum2)))

hist, bin_edges = np.histogram(durations_from_lockdown_to_local_maximum2,10 ,density=True)
bin_centers = (bin_edges[1:] + bin_edges[:-1])/2.
bin_widths = bin_edges[1]-bin_edges[0]
fig, ax =plt.subplots(1,1,figsize=(10,6))
ax.axvline(np.mean(durations_from_lockdown_to_local_maximum2),ls='--',color='black',
          label='Mean')
ax.bar(bin_centers,hist,bin_widths,color='dodgerblue',
       label='Distribution',alpha=0.7)
ax.set_xlabel('Time from beginning of lockdown until subsequent peak [days]',fontsize=fs)
ax.set_ylabel('Probability',fontsize=fs)
ax.legend(loc='best',fontsize=fs)
ax.set_xlim(0,30)
plt.show()
plt.close()
```

```python

```

```python

```

examples/control/notebooks-to-be-updated/ex05 - SEAI5R - Denmark.ipynb


# Machine Learning with Graph - Preliminaries part 1
> This notebook covers introduction about graphs, its basic concepts and how to represent a graph. Primary source is from Standford CS224W course.
- toc: true
- badges: true
- comments: true
- categories: [cs224w]
- image: images/graph.png

This is my personal notes for [The Standford CS224W Course: Machine Learning with Graphs - Fall 2019](http://web.stanford.edu/class/cs224w/) which I have been studying from its public resources for a couple of months. I will try to keep this blog series live update as my main source knowlege about graph.
The course covers tremendous knowledge for understanding and effectively learning from large-scale network. In general, it is organized into 3 big sections:
- `preliminaries`: provides basic background about networks/graphs.
- `network methods`: methods and algorithms which help us dealing with graph data structure.
- `machine learning with networks`: showing how we are applying machine learning into this field.

This notebook is part 1 of the preliminary section in which we will introduce about networks/graphs, its basic concepts and how to represent it.

Here is the link to other parts: [1](https://phucnsp.github.io/blog/cs224w/2020/05/27/ML-graph-premilinary-part1.html).

## Network Introduction and why should we care?
Networks are a general language for describing complex systems of interacting entities. Pictorially, rather than thinking that our dataset consists of a set of isolated data points, we consider interactions and relationships between these points.
In general, there are 2 types of networks/graphs:
- **Networks** are those that can be interpretated as examples of phenomena that appear in natural world, that's why it is also called **Natural Graphs**. A couple examples include
    - The human social network(a collection of 7+ billion individuals)
    - Internet communication systems(a collection of electronic devices)

- **Information Graphs** are those that can be interpretated as a data structure useful which are created in a heuristic way for solving a specific prediction problem. In this case, we're more interested in relationships between entities so we can efficiently perform learning tasks. Some examples include
    - Information/knowledge (are organized and linked for a specific purpose,e.g google search engine)
    - Scene graphs (how objects in a scene relate to one another)
    - Similarity networks (in which similar points in a dataset are connected)

While working with **Networks**, some of the main questions that we need to think about are **How are these systems organized? what are their design properties?** Behind many systems there is an intricate wiring diagram, a network, that defines the interactions between the components. We will never be able to model and predict these systems unless we understand the networks behind them.
> Note: Networks: Knowledge Discovery

In the other hand, working with **Graphs** leverages another questions **How do we take advantage of relational structure for better prediction?** Complex domains (knowledge, text, images, etc) have rich relational structure, which can be represented as a **relational graph** for numerous prediction tasks. By explicitly modeling relationships and applying machine learning into graph, we achieve better performance in many tasks.
> Note: Graphs: Machine Learning

Sometimes the distinction between network and graph is blurred and we will use them interchangeably in this blog series.

There are many ways to **analyse a network/graph**:
- **Node classification**, where we predict the type/color of a given node
- **Link prediction**, where we predict whether two nodes are linked
- **Community detection**, where we identify densely linked clusters of nodes
- **Similarity computation**, where we measure the similarity of two nodes or networks.
>Note: two popular graph analysis tools are SNAP and NetworkX

Nowadays, network is applied in various fields with many successes. Some of the **main applications** include
- **Network application in the social networks**.
We can map a social network, e.g. facebook social network, as a graph and play around with it, try to understand how it is organized? how a group of people is connected? how to advertise a product more efficiently in our network?. One of the popular tasks with social network is community detection. Throughout this task, we can explore a lot of information stays behind our network.

- **Network application in infrastructure design**.
A typical example is the light outage in East Coast of America back in 2003. This is a kind of cascading failures where outage at a city led to failures at another cities. So the question is **how can we design a robust system? how can we prevent such kind of cascading failures?** By mapping the power grid as a graph and modeling this cascading behaviour, we can improve the power infrastructure design. In reality failures follow reproducible laws, that can be quantified and even predicted using the tools of networks.

- **Network application in Knowledge (Knowledge Graphs)**.
We can encode knowledge of a domain into graph and use it for downstream tasks. These graphs can even be heterogeneous, in a sense that nodes and links can be in different types. Or it can be multimodal where nodes/edges can be categorised into different groups.

- **Network application in Online Media**.
We can use data from twitter to map polical blogs into graph and analyse the political polarization between them. Or we can use graph's power to detect whether a given Wikipedia article is a hoax. In addition, we can also use graph in viral marketing in which we try to maximize the social impact by selecting a subset of famous people to advertise for us. And another application is product adoption where you have a new product and you are interested in how far your product will spread via inviations, how to optimize this spread.

- **Network application in Biomedicine**.
A typical example application of graph in this field is to predict side effects of a given pair of drugs. This is a kind of link prediction where we need to predict which kind of effects might occur between two drug nodes.

Altogether, networks are a **universal language for describing complex data**, and **generalize across a variety of different fields**. With increased **data availability** and **a variety of computational challenges**, learning about networks leaves one poised to make a **wide variety of contributions**.

## Basic Concepts

### Structure of Graphs

**A network/graph $G(N,E)$** is a collection of objects (nodes) $N$ where some pairs of objects are connected by edges/links $E$:
- Objects: nodes, vertices N
- Interactions: links, edges E
- System: network, graph G(N,E)

>Note: Technically, a network often refers to real systems (the web, a social network, etc) while a graph often refers to the mathematical representation of a network (a web graph, social graph, etc).
> Note: groups of technical terms - network, node, link vs. graph, vertex, edge. However, in most cases these terms are used interchangeably.

Both **nodes** and **edges** can contain information that is encoded as **node attributes**, **edge attributes**, respectively. There are many options for edge attributes such as:
- Weight (e.g. frequency of communication)
- Ranking (best friend, second best friend...)
- Type (friend, relative, co-worker)
- Sign: Trust vs. Distrust
- Properties depending on the structure of the rest of the graph: number of common friends.

**Undirected graphs** are those with symmetrical/reciprocal links(e.g.friendship on Facebook). **Node degree** $k_i$ of node $i$ in an undirected graph is defined as the number of edges adjacent to node $i$. For example the degree of node A in the undirected graph below $k_A=4$.
The **average degree** is then
\begin{equation}
\bar{k}=\langle k\rangle=\frac{1}{|N|} \sum_{i=1}^{|N|} k_{i}=\frac{2|E|}{N}
\end{equation}

**Directed graphs** are those with directed links(e.g.following on Twitter). In these graphs, **In-degree** $k_{i}^{in}$ is defined as the number of edges entering node $i$. Similarly, **out-degree** $k_{i}^{out}$ is defined as the number of edges leaving node $i$. The **(total) degree** of a node is the sum of in- and out-degrees.
For example the degree of node C in directed graph below: $k_{C}^{in}=2, k_{C}^{out}=1, k_C=2$
The **average degree** is $\bar{k}=\langle k\rangle=\frac{|E|}{N}$

**Complete graphs** are undirected graphs with the maximum number of edges. Intuitively speaking, complete graph has all pairs of nodes connected. The complete graph has number of edges $|E|=\left(\begin{array}{c}N \\ 2\end{array}\right)=\frac{N(N-1)}{2}$ and average degree $\bar{k}=|N|-1$.

**Bipartite Graphs** are those whose nodes can be divided into two disjoint/independent sets $U$ and $V$ such that every edge connects a node in $U$ to a node in $V$, there are no edges between nodes in $U$ and between nodes in $V$. Examples author-to-papers(they authored), actors-to-movies(they appeared in), etc.
Bipartite graphs can be projected into **Folded Networks** by creating edges within independent sets $U$ or $V$ if they share at least one common neightbor. Looking at node 1 of projection $U$ in bipartite graph below, it is connected to node 2 and node 3 because node 1, 2, 3 all have common neightbor A.

**Other Graph Types** we briefly note that graphs can also include self-edges(self-loops), weights associated with edges, and multiple edges connecting nodes. These attributes can be encoded in graph representation with ease.

### Graph Representation

There a 3 ways to represent a graph $G$:
- **Adjacency matrix**: is one of the most common way to represent a matrix. The image below shows all information we need to know about directed and undirected graph. Notice that adjacency matrix is symmetric for undirected graph and asymmetric for directed graph. However, most real-world networks are sparse ($|E| \ll E_{\max}$ or $\bar{k} \ll |N|-1$). As a consequence, the adjacency matrix is filled with zeros, it is an undesirable property!

- **Edge List**: a graph can be represented as a set of edges(edge list). An example of edge list with the directed graph above [(1,4), (4,2), (4,3), (1,2)]. This representation makes edge lookups harder but preserves memory (alleviate mentioned above with adjacency matrix).

- **Adjacency list**: This method is easier to work with if the network is large and sparse. It also allows us to quickly retrieve all neightbors of a given node. Example adjacency list for the directed graph above {1:[4,2], 2:[], 3:[], 4:[2,3]}

### Graph Connectivity

A undirected graph is called **connected** if there is a path between any pair of nodes in the graph.
A graph is called **disconnected** if it is made up by two or more connected components.

The image below shows primary concepts about connectivity characteristic of an undirected graph such as **connected(undirected) graph**, **disconnected(undirected) graph**, **giant component**, **isolated node**, **bridge edge**, **articulated node**. In addition, the adjacency matrix of such networks can be represented as a block-diagonal form.

We can further extend the connectivity concepts from undirected to directed graphs by defining **strongly connected directed graph**, **weakly connected directed graph**, **strongly connected components.**
>Note: strong connected components (SCCs) can be defined as strongly connected subgraphs of a bigger graph.

Altogether, the way we define a network, the way we choose a proper network representation of a given domain/problem determines our ability to use network successfully. In some cases, there is a unique, unambiguous representation but in other cases, the representation is by no means unique. The way we assign links will also determine the nature of question we can study. Here is some examples of network representations:
- Email network: can be represented as directed multigraph with self-edges graphs.
- Facebook friendships: can be represented as undirected, unweighted graphs.
- Citation networks: can be represented as unweighted, directed, acyclic graphs.
- Collaboration networks: can be represented as undirected multigraph or weighted graphs.
- Mobile phone calls: can be represented as directed, (weighted?) multigraph.
- Protein interactions: can be represented as undirected, unweighted with self-interactions graphs.

## References
- main source: [Standford CS224W Machine Learning with Graphs](http://web.stanford.edu/class/cs224w/)
- other sources
    - [Graph on structured documents blog post](https://towardsdatascience.com/using-graph-convolutional-neural-networks-on-structured-documents-for-information-extraction-c1088dcd2b8f)
    - [Graph theory blog post](https://towardsdatascience.com/graph-theory-and-deep-learning-know-hows-6556b0e9891b)
    - [Graph embedding blog post](https://towardsdatascience.com/overview-of-deep-learning-on-graph-embeddings-4305c10ad4a4)
    - [Graph convolution blog post](https://towardsdatascience.com/graph-convolutional-networks-for-geometric-deep-learning-1faf17dee008)
    - [Geometric deep learning blog post](https://medium.com/@flawnsontong1/what-is-geometric-deep-learning-b2adb662d91d)
    - review papers about graphs: [1](https://arxiv.org/pdf/1611.08097.pdf), [2](https://arxiv.org/pdf/1901.00596.pdf)
    - Pytorch Geometric:[1](https://pytorch-geometric.readthedocs.io/en/latest/notes/introduction.html), [2](https://towardsdatascience.com/hands-on-graph-neural-networks-with-pytorch-pytorch-geometric-359487e221a8)

```python

```