# Getting Started

In this this notebook, we introduce the core functionalities of `mclearn`. For more information on the package, refer to the [documentation](https://mclearn.readthedocs.io/en/latest/) and the [API Reference](https://mclearn.readthedocs.io/en/latest/api.html).

```python
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split
from sklearn.utils import shuffle
from sklearn.ensemble import RandomForestClassifier
from sklearn.svm import SVC, LinearSVC
from sklearn.linear_model import LogisticRegression
from mclearn.active import run_active_learning_with_heuristic
from mclearn.classifier import (train_classifier,
                                grid_search_logistic,
                                grid_search_svm_poly,
                                grid_search_svm_rbf,
                                learning_curve)
from mclearn.heuristics import (random_h, margin_h, entropy_h,
                                qbb_margin_h, qbb_kl_h)
from mclearn.photometry import (fetch_sloan_data,
                                reddening_correction_sfd98,
                                reddening_correction_sf11,
                                reddening_correction_w14,
                                correct_magnitudes)
from mclearn.viz import (plot_filters_and_spectrum,
                         plot_class_distribution,
                         plot_hex_map,
                         plot_scatter_with_classes,
                         plot_average_learning_curve)
%matplotlib inline
plt.style.use('ggplot')
```

## 1. About the Dataset

There are two ways we can measure objects on the sky. The first option is to obtain a full spectrum. This is expensive but it contains a lot of valuable information. For example astronmers can look at a spectrum and determine what kind of star or galaxy the object is. A cheaper method is to obtain photometric measurments. Here we set up a filter and only allow light from certain wavelengths to go through. We then add up the amount of light (i.e. integrate) and obtain a photometric measurment. For example, below we plot the spectrum of Vega (one of the brighest stars in the sky) and the 5 ugriz photometric filters. Using these five filters, we will end up with five photometric values of Vega, one in each light band. Since now we only have 5 numbers (instead of a spectrum), it's a lot more difficult to manually classify the object. Let's see if machine learning can help us. For more information about the differences between photometry and spectroscopy, [here's](https://physics.stackexchange.com/questions/169734/photometric-surveys-vs-spectroscopic-surveys) a good description on StackExchange.

```python
vega_spectrum_url = 'http://www.astro.washington.edu/users/ivezic/DMbook/data/1732526_nic_002.ascii'
ugriz_filter_url = 'http://www.sdss.org/dr7/instruments/imager/filters/%s.dat'
filter_dir = '../data/filters'
spectra_dir = '../data/spectra'

fig = plt.figure(figsize=(10,5))
ax = plot_filters_and_spectrum(ugriz_filter_url, vega_spectrum_url, filter_dir=filter_dir, spectra_dir=spectra_dir)
```

The Sloan Digital Sky Survey (SDSS) is a comprehensive survey of the northern sky. In this notebook, we will explore a small subset of the survey, namely the photometric measurements and spectroscopic labels of around 2,000 objects in a small section of the sky. Let's submit an SQL query to the Sloan Sky Server to get the dataset.

```python
output = '../data/sdss_small.csv'
sql_query = '''
    SELECT
        p.ra, p.dec,
        CASE s.class WHEN 'GALAXY' THEN 'Galaxy'
                     WHEN 'STAR' THEN 'Star'
                     WHEN 'QSO' THEN 'Quasar'
                     END AS class,
        p.psfMag_u, p.psfMag_g, p.psfMag_r, p.psfMag_i, p.psfMag_z,
        p.petroMag_u, p.petroMag_g, p.petroMag_r, p.petroMag_i, p.petroMag_z,
        p.petroRad_r, extinction_r
    FROM PhotoObj AS p
       JOIN SpecObj AS s ON s.bestobjid = p.objid
    WHERE
       p.psfMag_r < 21 AND
       p.ra BETWEEN 202 AND 207 AND
       p.dec BETWEEN 0 AND 2 '''

fetch_sloan_data(sql_query, output, verbose=True)
```

    Connecting to the server...
    Writing to file...
    Success!

## 2. Exploratory Analysis

Okay, let's read in the data and see what we have.

```python
sdss = pd.read_csv("../data/sdss_small.csv")
sdss.head()
```

      ra
      dec
      class
      psfMag_u
      psfMag_g
      psfMag_r
      psfMag_i
      psfMag_z
      petroMag_u
      petroMag_g
      petroMag_r
      petroMag_i
      petroMag_z
      petroRad_r
      extinction_r

      0
      205.041772
      1.225637
      Quasar
      22.33957
      20.69193
      20.60227
      20.48185
      20.59905
      22.64906
      20.67106
      20.67370
      20.55878
      20.61250
      1.332299
      0.064220

      1
      205.056272
      0.496612
      Galaxy
      24.15581
      22.71687
      20.99946
      19.97933
      19.51431
      25.93601
      22.09508
      20.50595
      19.45818
      18.98700
      1.734146
      0.081771

      2
      205.074412
      0.882936
      Galaxy
      23.37353
      21.91115
      20.22815
      19.61556
      19.30375
      24.58035
      21.27997
      19.43404
      18.75463
      18.53402
      2.939431
      0.076804

      3
      205.081585
      1.416685
      Star
      18.25325
      17.27528
      16.88841
      16.77056
      16.72031
      18.29956
      17.30037
      16.95146
      16.83360
      16.79116
      1.378984
      0.064170

      4
      205.827414
      1.799152
      Galaxy
      24.07808
      22.59162
      20.83035
      20.02864
      19.46696
      22.87317
      21.52034
      20.02671
      19.02419
      18.55864
      2.969733
      0.067893

The first two columns - ra and dec - are the coordinates of the objects (see the [wiki page](https://en.wikipedia.org/wiki/Equatorial_coordinate_system) for more information). Our target column is called `class` - this can be one of Galaxy, Quasar, or Star. Looking at the distribution of the classes, we can see that the dataset is very unbalanced. We have, for example, five times as many galaxies as stars and quasars.

```python
fig = plt.figure(figsize=(10, 5))
ax = plot_class_distribution(sdss['class'])
```

Given this extreme imbalance, we will be mainly using the balanced accuracy rate when reporting the performance of a classifier. To compute this measure, we first define $A_i$, the accuracy rate of class $i$, as:

\begin{align}
       A_i = \dfrac{C_{ii}}{m_i}
\end{align}

where $C$ is the confusion matrix with entry $C_{ij}$ being the
number of objects in class $i$ but have been as class
$j$, and $m_i$ is the number of objects in class $i$.
Before running a classifier, we know nothing of its performance, so we
can assume the accuracy rate follows a flat prior distribution. In
particular, the Beta distribution with parameters
$\alpha = \beta = 1$ (i.e. a uniform distriubtion) seems
appropriate here:

\begin{align}
       A_i \sim Beta(1, 1) \qquad \forall i
\end{align}

Given an accuracy rate $A_i$ for each class $i$, the number
of correct predictions in class $i$ will follow a Binomial
distribution with $A_i$ as the probability of success:

\begin{align}
       \big( C_{ii} \mid A_i \big) \sim Bin\big(m_i, A_i\big)  \qquad \forall i
\end{align}

In Bayesian terms, this is our likelihood. Now we know that with respect
to a Binomial likelihood, the Beta distribution is conjugate to itself.
Thus the posterior distribution of $A_i$ will also be Beta with
parameters:

\begin{align}
       \big( A_i \mid C_{ii} \big) \sim Beta \big( 1 + C_{ii}, 1 + m_i - C_{ii} \big) \qquad \forall i
\end{align}

The (posterior) balanced accuracy rate $A$ can now be defined as the average
of the individual accuracy rates $A_i$:

\begin{align}
       A = \dfrac{1}{k} \sum_i A_i
\end{align}

where $k$ is the number of classes.

When presenting our results, we will often be using the Mollweide projection. Here's the layout of the coordinates, with the red line being the plane of the Milky Way.

```python
fig = plt.figure(figsize=(10,5))
zero_values = np.zeros(1)
ax = plot_hex_map(zero_values, zero_values, axisbg=None, colorbar=False, labels=True)
```

For example, this is where the objects in our dataset are located in the sky. It's a very small region roughly in the middle. For now, it seems that the map is not very useful. However, when we look at the whole SDSS dataset, being able to plot the results on a map will become quite handy.

```python
fig = plt.figure(figsize=(10,5))
ax = plot_hex_map(sdss['ra'], sdss['dec'], vmax=5)
```

## 3. Magnitudes, Colours, and Dust

In the datasets, we have 11 features - 5 Petrosian magitudes, 5 PSF magnitudes, and a size measurement. To get a feel of how separated the three classes are, let's use PCA and reduce these 11 features down to 2 dimensions.

```python
feature_cols = ['psfMag_u', 'psfMag_g', 'psfMag_r', 'psfMag_i', 'psfMag_z', 'petroMag_u',
                'petroMag_g', 'petroMag_r', 'petroMag_i', 'petroMag_z', 'petroRad_r']
target_col = 'class'
classes = ['Galaxy', 'Quasar', 'Star']

pca = PCA(n_components=2)
projection = pca.fit_transform(sdss[feature_cols])
target = np.array(sdss[target_col])

fig = plt.figure(figsize=(10, 5))
ax = plot_scatter_with_classes(projection, target, classes, size=30, alpha=0.3,
                               scatterpoints=3)
ax.set_xlim(-70, -40)
ax.set_ylim(-15, 0)
plt.show()
```

As light travels through space to Earth, some of it get absorbed and scattered by the galactic dust. Light in the higher frequency range is more affected, hence extinction causes an object to become redder. This gives the phenomenon the name 'interstellar reddening'.

The amount of extinction varies across the sky. For example, there is a lot of dust in the Milky Way band, as can be seen in the following reddening map made by LAMBDA, where red indicates a high level of extinction.

There are three sets of extinctions that we can apply - the SFD98, the SF11, and the W14 sets. Refer to the [doc](https://pythonhosted.org/mclearn/photometry.html) to learn how these corrections are computed. We will use random forest to see which set offers the best improvement in the balanced accuracy rate.

```python
# compute the three sets of reddening correction
sfd98_corrections = reddening_correction_sfd98(sdss['extinction_r'])
sf11_corrections = reddening_correction_sf11(sdss['extinction_r'])
w14_corrections = reddening_correction_w14(sdss['extinction_r'])

# column names of the magnitudes to be corrected
psf_magnitudes = ['psfMag_u', 'psfMag_g', 'psfMag_r', 'psfMag_i', 'psfMag_z']
petro_magnitudes = ['petroMag_u', 'petroMag_g', 'petroMag_r', 'petroMag_i', 'petroMag_z']

# calculate the corrected magnitudes
correct_magnitudes(sdss, psf_magnitudes, sfd98_corrections, '_sfd98')
correct_magnitudes(sdss, petro_magnitudes, sfd98_corrections, '_sfd98')
correct_magnitudes(sdss, psf_magnitudes, sf11_corrections, '_sf11')
correct_magnitudes(sdss, petro_magnitudes, sf11_corrections, '_sf11')
correct_magnitudes(sdss, psf_magnitudes, w14_corrections, '_w14')
correct_magnitudes(sdss, petro_magnitudes, w14_corrections, '_w14')
```

Here are the four sets of features and the target.

```python
uncorrected_features = ['psfMag_u', 'psfMag_g', 'psfMag_r', 'psfMag_i', 'psfMag_z',
                        'petroMag_u', 'petroMag_g', 'petroMag_r', 'petroMag_i',
                        'petroMag_z', 'petroRad_r']

sfd98_features = ['psfMag_u_sfd98', 'psfMag_g_sfd98', 'psfMag_r_sfd98', 'psfMag_i_sfd98',
                  'psfMag_z_sfd98', 'petroMag_u_sfd98', 'petroMag_g_sfd98',
                  'petroMag_r_sfd98', 'petroMag_i_sfd98', 'petroMag_z_sfd98',  'petroRad_r']

sf11_features = ['psfMag_u_sf11', 'psfMag_g_sf11', 'psfMag_r_sf11', 'psfMag_i_sf11',
                 'psfMag_z_sf11', 'petroMag_u_sf11', 'petroMag_g_sf11', 'petroMag_r_sf11',
                 'petroMag_i_sf11', 'petroMag_z_sf11', 'petroRad_r']

w14_features = ['psfMag_u_w14', 'psfMag_g_w14', 'psfMag_r_w14', 'psfMag_i_w14',
                'psfMag_z_w14', 'petroMag_u_w14', 'petroMag_g_w14', 'petroMag_r_w14',
                'petroMag_i_w14', 'petroMag_z_w14', 'petroRad_r']

target_col = 'class'
```

Let's now train a random forest on each of the four sets of features.

```python
correct_predictions, confusion_baseline = train_classifier(sdss, uncorrected_features, target_col,
    train_size=0.7, test_size=0.3, balanced=False, recall_maps=False, coords=False, report=False, random_state=2)
```

    Here's the confusion matrix:

      Predicted

      Galaxy
      Quasar
      Star

      Actual
      Galaxy
      417
      3
      2

      Quasar
      8
      63
      15

      Star
      8
      11
      63

    The balanced accuracy rate is 82.50%.

```python
correct_predictions, confusion_sfd98 = train_classifier(sdss, sfd98_features, target_col,
    train_size=0.7, test_size=0.3, balanced=False, recall_maps=False, coords=False, report=False, random_state=2)
```

    Here's the confusion matrix:

      Predicted

      Galaxy
      Quasar
      Star

      Actual
      Galaxy
      417
      3
      2

      Quasar
      8
      65
      13

      Star
      8
      10
      64

    The balanced accuracy rate is 83.66%.

```python
correct_predictions, confusion_sf11 = train_classifier(sdss, sf11_features, target_col,
    train_size=0.7, test_size=0.3, balanced=False, recall_maps=False, coords=False, report=False, random_state=2)
```

    Here's the confusion matrix:

      Predicted

      Galaxy
      Quasar
      Star

      Actual
      Galaxy
      417
      3
      2

      Quasar
      8
      63
      15

      Star
      8
      10
      64

    The balanced accuracy rate is 82.90%.

```python
correct_predictions, confusion_w14 = classifier.train_classifier(sdss, w14_features, target_col,
    train_size=0.7, test_size=0.3, balanced=False, recall_maps=False, coords=False, report=False, random_state=2)
```

    Here's the confusion matrix:

      Predicted

      Galaxy
      Quasar
      Star

      Actual
      Galaxy
      417
      3
      2

      Quasar
      8
      63
      15

      Star
      8
      10
      64

    The balanced accuracy rate is 82.90%.

It seems like there is not much difference between the three sets of corrections (which is expected given the very small sample size). The SF11 correction produces slightly better results, so let's use this from now on.

Another thing that we might want to consider is whether to use colours as features. Magnitudes themselves are correlated with the distance of the objects. That is, if a star is closer to us, it would appear brighter, resulting in a lower magnitude (yes astronomers are weird!). But this doesn't change the fact that the object is a star. Thus we could take differences between magnitudes (and call them colours). This will remove any dependence on the distance. To keep it simple, let's ignore colours for now.

## 3. Learning Curves

We now want to compare the performance of Logistic Regression, SVM, and Random Forest. We start by normalising the features to zero mean and unit variance.

```python
scaler = StandardScaler()
X_scaled = scaler.fit_transform(sdss[sf11_features])
y = sdss[target_col]
```

We need to determine what the best hyperparamters for Logistic Regression and SVM are. This can be done via a grid search (basically an exhaustive search through the parameter space). For each combination of paramters, we use 5-fold cross validation, each fold having 50 examples in the training set and 50 in the test set. Each search will take a few minutes to complete.

```python
grid_search_svm_rbf(X_scaled, y, train_size=50, test_size=50)
```

```python
grid_search_svm_poly(X_scaled, y, train_size=50, test_size=50)
```

```python
grid_search_logistic(X_scaled, y, train_size=50, test_size=50)
```

Let's now compute some learning curves.

```python
train_sizes = np.concatenate((np.arange(10, 100, 10), np.arange(100, 600, 100)))
test_sizes = [500] * 15

# initialise the classifiers
rbf_svm = SVC(kernel='rbf', gamma=1, C=1000, cache_size=200)
poly_svm = LinearSVC(C=10000, loss='squared_hinge', penalty='l1', dual=False, multi_class='ovr',
                         fit_intercept=True, random_state=21)
logistic = LogisticRegression(penalty='l1', dual=False, C=10000, multi_class='ovr', solver='liblinear', random_state=21)
forest = RandomForestClassifier(n_estimators=100, n_jobs=-1, class_weight='subsample', random_state=21)

# train the classifiers
lc_forest = learning_curve(sdss, sfd98_features, target_col, forest, train_sizes,
            test_sizes, random_state=2, normalise=False, balanced=False, degree=1)

lc_rbf_svm = learning_curve(sdss, sfd98_features, target_col, rbf_svm, train_sizes,
             test_sizes, random_state=2, normalise=True, balanced=False, degree=1)

lc_poly_svm_2 = learning_curve(sdss, sfd98_features, target_col, poly_svm, train_sizes,
                test_sizes, random_state=2, normalise=True, balanced=False, degree=2)

lc_poly_svm_3 = learning_curve(sdss, sfd98_features, target_col, poly_svm, train_sizes,
                test_sizes, random_state=2, normalise=True, balanced=False, degree=3)

lc_logistic_2 = learning_curve(sdss, sfd98_features, target_col, logistic, train_sizes,
                test_sizes, random_state=2, normalise=True, balanced=False, degree=2)

lc_logistic_3 = learning_curve(sdss, sfd98_features, target_col, logistic, train_sizes,
                test_sizes, random_state=2, normalise=True, balanced=False, degree=3)
```

```python
learning_curves = [lc_forest, lc_rbf_svm, lc_poly_svm_2, lc_poly_svm_3, lc_logistic_2, lc_logistic_3]

curve_labels = ['Random Forest', 'SVM RBF', 'SVM Polynomial Degree 2', 'SVM Polynomial Degree 3',
                'Logistic Regression Degree 2', 'Logistic Regression Degree 3']

fig = plt.figure(figsize=(10, 5))
ax = mclearn.viz.plot_learning_curve(train_sizes, learning_curves, curve_labels, xscale='linear')
```

## 5. Logistic Active Learning

Let's see if we can beat random sampling. The goal here is to get the best possible accuracy rate with the smallest possible training set. Thus we will need to be smart when choosing the next candidate to put in the training pool.

We start by training the logistic classifier with 20 random examples. Then each time we look at a random sample of the unlabelled pool and assign each example a score. We query the example with the best score and then train the classifier again. This is repeated until we have 200 examples in the training set. Normally we should do around 10 trials and take the average, but in this notebook, let's just do 1 trial of the experiment.

Two uncertainty sampling heuristics are tested:

* Picking the candidate that displays the greatest Shannon entropy:

\begin{align}
-\sum_c q_c \log(q_c)
\end{align}

* Picking the candidate with the smallest margin:

\begin{align}
M_n = \left| p(c \mid \mathbf{x}_n) - p(c' \mid \mathbf{x}_n) \right|
\end{align}

And two query by bagging heuristics are tested:

* Picking the candidate with the smallest average margin.

* Picking the candidate with the largest value of the average Kullback-Leibler divergence from the average:

\begin{align}
	\dfrac{1}{B} \sum_{b=1}^B \text{KL}(f_b \mid\mid f_\text{avg})
\end{align}

Note that the KL divergence of $Q$ from $P$ is defined as

\begin{align}
	D_{\mathrm{KL}}(P\|Q) = \sum_i P(i) \, \ln\frac{P(i)}{Q(i)}.
\end{align}

This KL divergence measures the amount of information lost when $Q$ is used to approximate $P$. In the active learning context, $Q$ is the average prediction probability of the committee, while $P$ is the prediction of a particular committee member.

Okay, let's the run the experiments. This will take roughly 5 minutes to run.

```python
# initialise the main classifier
logistic = LogisticRegression(multi_class='ovr', penalty='l1', C=10000, random_state=2, class_weight='auto')

# list of all heuristics to be used (and the paths where results are saved)
heuristics = [random_h, entropy_h, margin_h, qbb_margin_h, qbb_kl_h]

# set up the logistic committee
committee = []
committee_size = 3
for i in range(committee_size):
    member = LogisticRegression(multi_class='ovr', penalty='l1', C=10000, random_state=2, class_weight='auto')
    committee.append(member)

# 70/30 split of training and test sets
training_pool, testing_pool, training_oracle, testing_oracle = train_test_split(
    np.array(sdss[sfd98_features]), np.array(sdss[target_col]), train_size=0.7)

# shuffle and randomise data
training_pool, training_oracle = shuffle(training_pool, training_oracle, random_state=14)

# run active learning experiment for each heuristic
learning_curves = []
for heuristic in heuristics:
    lc = run_active_learning_with_heuristic(heuristic, logistic, training_pool,
         testing_pool, training_oracle, testing_oracle, n_trials=1, full_sample_size=200, total_n=200,
         random_n=30, initial_n=20, bag_size=30, committee=committee, pickle_path=None, balanced_pool=False)
    learning_curves.append(lc)
```

    ..

    ..

    ..

    ..

    ..

```python
curve_labels = ['Random', 'Entropy', 'Margin', 'QBB Margin', 'QBB KL']
fig = plt.figure(figsize=(10, 5))
ax = plot_average_learning_curve(np.arange(20, 201), learning_curves, curve_labels)
```


# Detector simulation
In this notebook we load a track dataset generated by `edep-sim` and we calculate the ADC counts corresponding to each pixel. The result is exported to a HDF5 file.

```python
# This is need so you can import larndsim without doing python setup.py install
import os,sys,inspect
currentdir = os.path.dirname(os.path.abspath(inspect.getfile(inspect.currentframe())))
parentdir = os.path.dirname(currentdir)
sys.path.insert(0,parentdir)
```

```python
from math import ceil
from time import time

import matplotlib.pyplot as plt
from matplotlib import cm, colors
import mpl_toolkits.mplot3d.art3d as art3d

import numpy as np
import cupy as cp
import fire
import h5py

from numba import cuda
from numba.cuda.random import create_xoroshiro128p_states

import matplotlib as mpl
mpl.rcParams['figure.dpi'] = 100

from tqdm.notebook import tqdm
```

```python
from larndsim import consts
import importlib
importlib.reload(consts)
consts.load_properties("../larndsim/detector_properties/ndlar-module.yaml","../larndsim/pixel_layouts/multi_tile_layout-3.0.40.yaml")

from larndsim import quenching, drifting, detsim, pixels_from_track, fee
```

```python
from larndsim.consts import detector, physics
```

```python
response = cp.load('../larndsim/bin/response_38.npy')
```

### Dataset import
First of all we load the `edep-sim` output. For this sample we need to invert $z$ and $y$ axes.

```python
with h5py.File('lbnfSpillLAr.edep.h5', 'r') as f:
    tracks = np.array(f['segments'])

selected_tracks = tracks[:100]

x_start = np.copy(selected_tracks['x_start'] )
x_end = np.copy(selected_tracks['x_end'])
x = np.copy(selected_tracks['x'])

selected_tracks['x_start'] = np.copy(selected_tracks['z_start'])
selected_tracks['x_end'] = np.copy(selected_tracks['z_end'])
selected_tracks['x'] = np.copy(selected_tracks['z'])

selected_tracks['z_start'] = x_start
selected_tracks['z_end'] = x_end
selected_tracks['z'] = x
# selected_tracks['eventID'] = 0
```

```python
import numpy.lib.recfunctions as rfn

if 'n_photons' not in selected_tracks.dtype.names:
    n_photons = np.zeros(selected_tracks.shape[0], dtype=[('n_photons', 'f4')])
    selected_tracks = rfn.merge_arrays((selected_tracks, n_photons), flatten=True)
```

### Quenching and drifting
We calculate the number of electrons after recombination (`quenching` module) and the position and number of electrons after drifting (`drifting` module).

```python
%load_ext wurlitzer
importlib.reload(drifting)
TPB = 256
BPG = ceil(selected_tracks.shape[0] / TPB)
quenching.quench[BPG,TPB](selected_tracks, physics.BIRKS)
drifting.drift[BPG,TPB](selected_tracks)
```

We find the pixels intersected by the projection of the tracks on the anode plane using the Bresenham's algorithm. We also take into account the neighboring pixels, due to the transverse diffusion of the charges.

```python
importlib.reload(pixels_from_track)
unique_eventIDs = cp.unique(selected_tracks['eventID'])

longest_pix = ceil(max(selected_tracks["dx"])/detector.PIXEL_PITCH)
max_radius = ceil(max(selected_tracks["tran_diff"])*5/detector.PIXEL_PITCH)

TPB = 128
BPG = ceil(selected_tracks.shape[0] / TPB)

max_pixels = np.array([0])
pixels_from_track.max_pixels[BPG,TPB](selected_tracks, max_pixels)

max_neighboring_pixels = (2*max_radius+1)*max_pixels[0]+(1+2*max_radius)*max_radius*2
active_pixels = cp.full((selected_tracks.shape[0], max_pixels[0]), -1, dtype=np.int32)
neighboring_pixels = cp.full((selected_tracks.shape[0], max_neighboring_pixels), -1, dtype=np.int32)
n_pixels_list = cp.zeros(shape=(selected_tracks.shape[0]))

pixels_from_track.get_pixels[BPG,TPB](selected_tracks,
                                      active_pixels,
                                      neighboring_pixels,
                                      n_pixels_list,
                                      max_radius)
```

```python
shapes = neighboring_pixels.shape
joined = neighboring_pixels.reshape(shapes[0]*shapes[1])
unique_pix = cp.unique(joined)
unique_pix = unique_pix[(unique_pix != -1),:]
```

### Charge distribution calculation
Here we calculate the current induced by each track on the pixels, taking into account longitudinal and transverse diffusion. The track segment is parametrized as:
\begin{align}
x'(r') &=x_s + \frac{\Delta x}{\Delta r}r'\\
y'(r') &=y_s + \frac{\Delta y}{\Delta r}r'\\
z'(r') &=z_s + \frac{\Delta z}{\Delta r}r',
\end{align}
where $\Delta r$ is the segment length. Here we assume $z$ as the drift direction.
The diffused charge distribution is calculated with the following integral:
\begin{equation}
\rho(x,y,z) = \frac{Q}{\sqrt{(2\pi)^3}\sigma_x\sigma_y\sigma_z\Delta r}\exp\left[-\frac{(x-x_s)^2}{2\sigma_x^2}-\frac{(y-y_s)^2}{2\sigma_y^2}-\frac{(z-z_s)^2}{2\sigma_z^2}\right]\int^{r'=\Delta r}_{r'=0}dr'\exp[-(ar'^2+br')],
\end{equation}
where
\begin{align}
a &= \left[\left(\frac{\Delta x}{\Delta r}\right)^2\frac{1}{2\sigma_x^2} + \left(\frac{\Delta y}{\Delta r}\right)^2\frac{1}{2\sigma_y^2} + \left(\frac{\Delta z}{\Delta r}\right)^2\frac{1}{2\sigma_z^2} \right]\\
b &= -\left[\frac{(x-x_s)}{\sigma_x^2}\frac{\Delta x}{\Delta r}+
\frac{(y-y_s)}{\sigma_y^2}\frac{\Delta y}{\Delta r}+
\frac{(z-z_s)}{\sigma_z^2}\frac{\Delta z}{\Delta r}\right].
\end{align}

The simmetry of the transverse diffusion along the track allows to take a slice on the $xy$ plane and solve the integral once at a fixed $z$ coordinate (e.g. at $z_{m} = (z_s+z_e)/2$) and re-use it at other $z$ coordinates away from the endpoints (where $\rho(x,y,z)$ varies along $z$ so must be calculated at each $z$).

```python
max_length = cp.array([0])
track_starts = cp.empty(selected_tracks.shape[0])
threadsperblock = 128
blockspergrid = ceil(selected_tracks.shape[0] / threadsperblock)
detsim.time_intervals[blockspergrid,threadsperblock](track_starts, max_length, selected_tracks)

signals = cp.zeros((selected_tracks.shape[0],
                    neighboring_pixels.shape[1],
                    cp.asnumpy(max_length)[0]), dtype=np.float32)
threadsperblock = (1,1,64)
importlib.reload(detsim)
blockspergrid_x = ceil(signals.shape[0] / threadsperblock[0])
blockspergrid_y = ceil(signals.shape[1] / threadsperblock[1])
blockspergrid_z = ceil(signals.shape[2] / threadsperblock[2])
blockspergrid = (blockspergrid_x, blockspergrid_y, blockspergrid_z)
detsim.tracks_current[blockspergrid,threadsperblock](signals,
                                                     neighboring_pixels,
                                                     selected_tracks,
                                                     response)
```

Here we create a map that associates each track to the pixels where it induces a current

```python
pixel_index_map = cp.full((selected_tracks.shape[0], neighboring_pixels.shape[1]), -1)
for i_ in range(selected_tracks.shape[0]):
    compare = neighboring_pixels[i_, ..., cp.newaxis] == unique_pix
    indices = cp.where(compare)
    pixel_index_map[i_, indices[0]] = indices[1]
```

The three-dimensional array can contain more than one signal for each pixel at different times. If we want to plot the full induced signal on the pixel, we need to join the signals corresponding to the same pixel. First, we find the start time of each signal with `time_intervals`:

```python
# Mapping between unique pixel array and track array index
track_pixel_map = cp.full((unique_pix.shape[0], detsim.MAX_TRACKS_PER_PIXEL), -1)
TPB = 32
BPG = ceil(unique_pix.shape[0] / TPB)
detsim.get_track_pixel_map[BPG, TPB](track_pixel_map, unique_pix, neighboring_pixels)

threadsperblock = (1,1,64)
blockspergrid_x = ceil(signals.shape[0] / threadsperblock[0])
blockspergrid_y = ceil(signals.shape[1] / threadsperblock[1])
blockspergrid_z = ceil(signals.shape[2] / threadsperblock[2])
blockspergrid = (blockspergrid_x, blockspergrid_y, blockspergrid_z)
pixels_signals = cp.zeros((len(unique_pix), len(detector.TIME_TICKS)))
pixels_tracks_signals = cp.zeros((len(unique_pix),len(detector.TIME_TICKS),track_pixel_map.shape[1]))

detsim.sum_pixel_signals[blockspergrid,threadsperblock](pixels_signals,
                                                        signals,
                                                        track_starts,
                                                        pixel_index_map,
                                                        track_pixel_map,
                                                        pixels_tracks_signals)
currents = cp.sum(pixels_signals,axis=1)*detector.TIME_SAMPLING/physics.E_CHARGE
```

Now, let's verify that the integral of our induced currents

```python
print("Ratio: ", sum(currents)/sum(selected_tracks['n_electrons']))
```

    Ratio:  0.9604030309054555

### 3D event display

```python
cmap = cm.Spectral_r
importlib.reload(pixels_from_track)
from matplotlib import colors
norm = colors.Normalize(vmin=0, vmax=256)
m = cm.ScalarMappable(norm=norm, cmap=cmap)

cmap = cm.viridis
norm_curr = colors.LogNorm(vmin=min(currents[currents>0]), vmax=max(currents))
m_curr = cm.ScalarMappable(norm=norm_curr, cmap=cmap)
fig = plt.figure(figsize=(7,7))
ax = fig.add_subplot(111, projection='3d')

for it,t in enumerate(selected_tracks):
    if it == 0:
        ax.plot((t["x_start"], t["x_end"]),
            (t["z_start"], t["z_end"]),
            (t["y_start"], t["y_end"]),
            c='r',
            lw=1,
            alpha=1,
            zorder=10,
            label='Geant4 detector segment')
    else:
        ax.plot((t["x_start"], t["x_end"]),
                (t["z_start"], t["z_end"]),
                (t["y_start"], t["y_end"]),
                c='r',
                lw=1,
                alpha=1,
                zorder=9999)
    #print(t['pixel_plane'])
    if 0<=t['pixel_plane']<=999:
        ax.plot((t["x_start"], t["x_end"]),
            (detector.TPC_BORDERS[t['pixel_plane']][2][0], detector.TPC_BORDERS[t['pixel_plane']][2][0]),
            (t["y_start"], t["y_end"]),
            c='r',
            lw=1,
            ls=':',
            alpha=1,
            zorder=9999)

for ip, p in enumerate(unique_pix):
    x_rect, y_rect = detsim.get_pixel_coordinates(p.get())
    _,_,pixel_plane = pixels_from_track.id2pixel(p.get())
    #print(pixel_plane)
    row = int(pixel_plane // 4)
    column = int(pixel_plane % 4)
    if currents[ip] > 0:
        rect = plt.Rectangle((x_rect, y_rect),
                             detector.PIXEL_PITCH, detector.PIXEL_PITCH,
                             linewidth=0.1, fc=m_curr.to_rgba(currents[ip].get()),
                             edgecolor='gray')
        ax.add_patch(rect)
        art3d.pathpatch_2d_to_3d(rect, z=detector.TPC_BORDERS[pixel_plane][2][0], zdir="y")

for it in range(0,70,2):
    anode1 = plt.Rectangle((detector.TPC_BORDERS[it][0][0], detector.TPC_BORDERS[it][1][0]),
                            detector.TPC_BORDERS[it][0][1]-detector.TPC_BORDERS[it][0][0],
                            detector.TPC_BORDERS[it][1][1]-detector.TPC_BORDERS[it][1][0],
                           linewidth=1, fc='none',
                           edgecolor='gray', label=('Pixel' if ip == 5 else ''))
    ax.add_patch(anode1)
    art3d.pathpatch_2d_to_3d(anode1, z=detector.TPC_BORDERS[0][2][0], zdir="y")

    anode2 = plt.Rectangle((detector.TPC_BORDERS[it][0][0], detector.TPC_BORDERS[it][1][0]),
                            detector.TPC_BORDERS[it][0][1]-detector.TPC_BORDERS[it][0][0],
                            detector.TPC_BORDERS[it][1][1]-detector.TPC_BORDERS[it][1][0],
                           linewidth=1, fc='none',
                           edgecolor='gray', label=('Pixel' if ip == 5 else ''))
    ax.add_patch(anode2)
    art3d.pathpatch_2d_to_3d(anode2, z=detector.TPC_BORDERS[it+1][2][0], zdir="y")

    cathode = plt.Rectangle((detector.TPC_BORDERS[it][0][0], detector.TPC_BORDERS[it][1][0]),
                            detector.TPC_BORDERS[it][0][1]-detector.TPC_BORDERS[it][0][0],
                            detector.TPC_BORDERS[it][1][1]-detector.TPC_BORDERS[it][1][0],
                            linewidth=1, fc='gray', alpha=0.2,
                            edgecolor='gray')
    ax.add_patch(cathode)
    z_cathode = (detector.TPC_BORDERS[it][2][0]+detector.TPC_BORDERS[it+1][2][0])/2
    art3d.pathpatch_2d_to_3d(cathode, z=z_cathode, zdir="y")

    ax.plot((detector.TPC_BORDERS[it][0][0],detector.TPC_BORDERS[it][0][0]),(detector.TPC_BORDERS[it][2][0],detector.TPC_BORDERS[it+1][2][0]),
            (detector.TPC_BORDERS[it][1][0],detector.TPC_BORDERS[it][1][0]), lw=1,color='gray')

    ax.plot((detector.TPC_BORDERS[it][0][0],detector.TPC_BORDERS[it][0][0]),(detector.TPC_BORDERS[it][2][0],detector.TPC_BORDERS[it+1][2][0]),
            (detector.TPC_BORDERS[it][1][1],detector.TPC_BORDERS[it][1][1]), lw=1,color='gray')

    ax.plot((detector.TPC_BORDERS[it][0][1],detector.TPC_BORDERS[it][0][1]),(detector.TPC_BORDERS[it][2][0],detector.TPC_BORDERS[it+1][2][0]),
            (detector.TPC_BORDERS[it][1][0],detector.TPC_BORDERS[it][1][0]), lw=1,color='gray')

    ax.plot((detector.TPC_BORDERS[it][0][1],detector.TPC_BORDERS[it][0][1]),(detector.TPC_BORDERS[it][2][0],detector.TPC_BORDERS[it+1][2][0]),
            (detector.TPC_BORDERS[it][1][1],detector.TPC_BORDERS[it][1][1]), lw=1,color='gray')

# ax.set_ylim(consts.module_borders[pixel_plane][2][0],50)
ax.set_xlim(detector.TPC_BORDERS[0][0][0],detector.TPC_BORDERS[-1][0][1])
ax.set_ylim(detector.TPC_BORDERS[0][2][0],detector.TPC_BORDERS[-1][2][0])
ax.set_zlim(detector.TPC_BORDERS[0][1][0],detector.TPC_BORDERS[-1][1][1])

ax.set_box_aspect((4,8,4))
ax.grid(False)
ax.xaxis.set_major_locator(plt.MaxNLocator(5))
ax.yaxis.set_major_locator(plt.MaxNLocator(5))
# ax.set_axis_off()
ax.w_xaxis.set_pane_color((1.0, 1.0, 1.0, 1.0))
ax.w_yaxis.set_pane_color((1.0, 1.0, 1.0, 1.0))
ax.w_zaxis.set_pane_color((1.0, 1.0, 1.0, 1.0))

ax.view_init(azim=10)
def rotate(angle):
    ax.view_init(azim=angle)

from matplotlib import animation

rot_animation = animation.FuncAnimation(fig, rotate, frames=np.arange(0,362,2),interval=50)

ax.set_ylabel("z [cm]")
ax.set_xlabel("x [cm]")
ax.set_zlabel("y [cm]")
_ = plt.colorbar(m_curr,fraction=0.035, pad=0.05,label='Induced current integral [# electrons]')

```

```python
#rot_animation.save('animation.gif', writer='imagemagick', fps=10, bitrate=10, dpi=100)
```

### Electronics response and digitization
Here we simulate the electronics response (the self-triggering cycle) and the signal digitization.

```python
from larndsim.cuda_dict import CudaDict

time_ticks = cp.linspace(0, len(unique_eventIDs)*detector.TIME_INTERVAL[1], pixels_signals.shape[1]+1)
integral_list = cp.zeros((pixels_signals.shape[0], fee.MAX_ADC_VALUES))
adc_ticks_list = cp.zeros((pixels_signals.shape[0], fee.MAX_ADC_VALUES))
TPB = 128
BPG = ceil(pixels_signals.shape[0] / TPB)
rng_states = create_xoroshiro128p_states(TPB * BPG, seed=2)
integrate = cp.zeros((pixels_signals.shape[0], time_ticks.shape[0]))
backtracked_ids = cp.zeros((pixels_signals.shape[0], fee.MAX_ADC_VALUES, track_pixel_map.shape[1]))
pixel_thresholds_lut = CudaDict(cp.array([fee.DISCRIMINATION_THRESHOLD]), 1, 1)
pixel_thresholds_lut.tpb = TPB
pixel_thresholds_lut.bpg = BPG
orig_shape = unique_pix.shape
pixel_thresholds = pixel_thresholds_lut[unique_pix.ravel()].reshape(orig_shape)
fee.get_adc_values[BPG,TPB](pixels_signals,
                            pixels_tracks_signals,
                            time_ticks,
                            integral_list,
                            adc_ticks_list,
                            0,
                            rng_states,
                            backtracked_ids,
                            pixel_thresholds)
adc_list = fee.digitize(integral_list)
```

As an example, we plot the pixel with the largest amount of deposited charge

```python
largest_pix = np.argmax(pixels_signals.sum(axis=1)).get()
```

```python
fig,ax=plt.subplots(1,1)
current = pixels_signals[largest_pix][:3201].get()
ax.plot(detector.TIME_TICKS,pixels_tracks_signals[largest_pix][:3201].get())
ax.plot(detector.TIME_TICKS,current,label='Total track contribution',c='k',lw=2)
ax.plot(detector.TIME_TICKS,np.sum(pixels_tracks_signals[largest_pix][:3201].get(),axis=1),c='r',ls='--',label='Sum of each track contribution')
start_time = np.nonzero(current)[0][0]
end_time = np.nonzero(current)[0][-1]
ax.set_xlim(max(0,detector.TIME_TICKS[start_time]-10),detector.TIME_TICKS[end_time]+10)
ax.set_xlabel(r"Time [$\mathrm{\mu}$s]")
ax.set_ylabel(r"Current [A]")
ax.set_ylim(-0.1e-14, max(current)*1.3)
_=ax.legend()
```

### 2D event display with induced current and ADC counts

```python
from matplotlib import colors
norm = colors.Normalize(vmin=0, vmax=256)
m = cm.ScalarMappable(norm=norm, cmap=cmap)

cmap = cm.viridis
norm_curr = colors.LogNorm(vmin=min(currents[currents>0]), vmax=max(currents))
m_curr = cm.ScalarMappable(norm=norm_curr, cmap=cmap)

fig,ax = plt.subplots(1,2,figsize=(10,4.5),sharex=True,sharey=True)
importlib.reload(detsim)
for ip, p in enumerate(unique_pix):
    x_rect, y_rect = detsim.get_pixel_coordinates(p.get())
    _,_,pixel_plane = pixels_from_track.id2pixel(p.get())
    c = currents[ip].get()
    if c >= 1:
        rect = plt.Rectangle((x_rect, y_rect),
                             detector.PIXEL_PITCH, detector.PIXEL_PITCH,
                             linewidth=0.2, fc=m_curr.to_rgba(c),
                             edgecolor='grey')
        ax[0].add_patch(rect)
    else:
        rect = plt.Rectangle((x_rect, y_rect),
                             detector.PIXEL_PITCH, detector.PIXEL_PITCH,
                             linewidth=0.2, fc='none',
                             edgecolor='grey')
        ax[0].add_patch(rect)
    a = adc_list[ip][adc_list[ip]>fee.digitize(0)]
    if len(a):
        rect = plt.Rectangle((x_rect, y_rect),
                             detector.PIXEL_PITCH, detector.PIXEL_PITCH,
                             linewidth=0.2, fc=m.to_rgba(np.sum(a.get())),
                             edgecolor='grey')
        ax[1].add_patch(rect)

for it,t in enumerate(selected_tracks):
    ax[0].plot((t["x_start"], t["x_end"]),
            (t["y_start"], t["y_end"]),
            c='r',
            lw=1.25,
            ls=':',
            alpha=1,
            zorder=10)
    ax[1].plot((t["x_start"], t["x_end"]),
            (t["y_start"], t["y_end"]),
            c='r',
            lw=1.25,
            ls=':',
            alpha=1,
            zorder=100)
    ax[0].scatter((t["x_start"], t["x_end"]),
                  (t["y_start"], t["y_end"]),
                  c='r', s=0, zorder=99999)
    ax[1].scatter((t["x_start"], t["x_end"]),
                  (t["y_start"], t["y_end"]),
                  c='r', s=0, zorder=99999)

ax[0].set_aspect("equal")
ax[1].set_aspect("equal")
ax[0].set_xlabel("x [cm]")
ax[1].set_xlabel("x [cm]")
ax[0].set_ylabel("y [cm]")
from mpl_toolkits.axes_grid1.axes_divider import make_axes_locatable

divider0 = make_axes_locatable(ax[1])
cax0 = divider0.append_axes("right", size="7%", pad=0.07)
fig.colorbar(m, ax=ax[1], cax=cax0, label='ADC counts sum')

divider1 = make_axes_locatable(ax[0])
cax1 = divider1.append_axes("right", size="7%", pad=0.07)
fig.colorbar(m_curr, ax=ax[0], cax=cax1, label='Induced current integral [# electrons]')

plt.subplots_adjust(hspace=0.5)
fig.savefig("currentadc.pdf")
```

### Export result
As a last step we backtrack the ADC counts to the Geant4 tracks and we export the result in a HDF5 file.

```python
!rm "LAr.neutrino.100.edep.larnd-fromnotebook.h5"
importlib.reload(fee)
# track_pixel_map=cp.repeat(track_pixel_map[:, cp.newaxis], 10, axis=1)
adc_event_ids = np.full(adc_list.shape, unique_eventIDs[0].get()) # FIXME: only works if looping on a single event
event_id_list = [adc_event_ids]

pc = fee.export_to_hdf5(np.concatenate(event_id_list,axis=0),
                        adc_list.get(),
                        adc_ticks_list.get(),
                        unique_pix.get(),
                        backtracked_ids.get(),
                        track_pixel_map.get(),
                        "LAr.neutrino.100.edep.larnd-fromnotebook.h5")#,
#                        bad_channels=None)
```

    Writing to HDF5...: 100%|████████████████| 1941/1941 [00:00<00:00, 18163.07it/s]

```python
with h5py.File('LAr.neutrino.100.edep.larnd-fromnotebook.h5', 'a') as f:
    f.create_dataset("tracks", data=selected_tracks)
```

## Analyze the result
Here we read back the file we just produced and compare it with the _true_ tracks. So first we create a dictionary that associates the chip key to a $(x,y)$ position:

```python
from collections import defaultdict
import yaml

geometry_yaml = yaml.load(open("../larndsim/pixel_layouts/multi_tile_layout-3.0.40.yaml"), Loader=yaml.FullLoader)
det_yaml = yaml.load(open("../larndsim/detector_properties/ndlar-module.yaml"),Loader=yaml.FullLoader)

pixel_pitch = geometry_yaml['pixel_pitch']
is_multi_tile = True
chip_channel_to_position = geometry_yaml['chip_channel_to_position']
tile_orientations = geometry_yaml['tile_orientations']
tile_positions = geometry_yaml['tile_positions']
tpc_centers = det_yaml['tpc_offsets']
tile_indeces = geometry_yaml['tile_indeces']
xs = np.array(list(chip_channel_to_position.values()))[:, 0] * pixel_pitch
ys = np.array(list(chip_channel_to_position.values()))[:, 1] * pixel_pitch
x_size = max(xs)-min(xs)+pixel_pitch
y_size = max(ys)-min(ys)+pixel_pitch

def _rotate_pixel(pixel_pos, tile_orientation):
    return pixel_pos[0]*tile_orientation[2], pixel_pos[1]*tile_orientation[1]

tile_geometry = defaultdict(int)
geometry = {}
io_group_io_channel_to_tile = {}

for tile in geometry_yaml['tile_chip_to_io']:
    tile_orientation = tile_orientations[tile]
    tile_geometry[tile] = tile_positions[tile], tile_orientations[tile]

    for chip in geometry_yaml['tile_chip_to_io'][tile]:
        io_group_io_channel = geometry_yaml['tile_chip_to_io'][tile][chip]
        io_group = io_group_io_channel//1000
        io_channel = io_group_io_channel % 1000
        io_group_io_channel_to_tile[(io_group, io_channel)] = tile

    for chip_channel in geometry_yaml['chip_channel_to_position']:
        chip = chip_channel // 1000
        channel = chip_channel % 1000
        io_group_io_channel = geometry_yaml['tile_chip_to_io'][tile][chip]

        io_group = io_group_io_channel // 1000
        io_channel = io_group_io_channel % 1000
        x = chip_channel_to_position[chip_channel][0] * \
            pixel_pitch - x_size / 2
        y = chip_channel_to_position[chip_channel][1] * \
            pixel_pitch - y_size / 2

        x, y = _rotate_pixel((x, y), tile_orientation)
        x += tile_positions[tile][2]
        y += tile_positions[tile][1]

        geometry[(io_group, io_channel,
                       chip, channel)] = x, y
```

Then, we can compare the digitized charge associated to packets with the number of true electrons that reach the anode

```python
f = h5py.File("LAr.neutrino.100.edep.larnd-fromnotebook.h5", "r")
datawords = f['packets']['dataword'][f['packets']['packet_type']==0]
print("Digitized charge/true charge:", sum(datawords/fee.ADC_COUNTS*(fee.V_REF-fee.V_CM)+fee.V_CM-fee.V_PEDESTAL)/fee.GAIN/sum(f['tracks']['n_electrons']))
selected_tracks = f['tracks']
```

    Digitized charge/true charge: 0.9026745530625646

Here we verify the association between the _true_ track and the hit. The color of the pixel corresponds to the color of the first associated track (a hit can be associated to more tracks, not shown here).

```python
fig,ax = plt.subplots(1,1,tight_layout=True)
plotted_tracks = []
from matplotlib import colors
from mpl_toolkits.axes_grid1.axes_divider import make_axes_locatable

norm = colors.Normalize(vmin=0, vmax=256)
cmap = cm.jet

m = cm.ScalarMappable(norm=norm, cmap=cmap)
module_ids = []

colors = ['tab:blue','tab:orange','tab:green','tab:red','tab:purple','tab:brown']
for ip,packet in enumerate(f['packets']):

    if packet['packet_type'] == 0:
        io_group, io_channel, chip, channel = packet['io_group'], packet['io_channel'], packet['chip_id'], packet['channel_id']
        module_id = (io_group-1)//4
        if module_id not in module_ids:
            module_ids.append(module_id)
        io_group = io_group - (io_group-1)//4*4
        x,y = geometry[(io_group, io_channel, chip, channel)]
        track_ids = f['mc_packets_assn']['track_ids'][ip]
        fractions = f['mc_packets_assn']['fraction'][ip]
        z_offset = tpc_centers[module_id][0]*10
        x_offset = tpc_centers[module_id][2]*10
        y_offset = tpc_centers[module_id][1]*10

        rect = plt.Rectangle((x+x_offset, y+y_offset),
                             detector.PIXEL_PITCH*10, detector.PIXEL_PITCH*10,
                             linewidth=0, fc=m.to_rgba(packet['dataword']))

        ax.add_patch(rect)

        for trackid in track_ids:
            if trackid >= 0 and trackid not in plotted_tracks:
                plotted_tracks.append(trackid)
                ax.plot((f['tracks'][trackid]['x_start']*10,f['tracks'][trackid]['x_end']*10),
                        (f['tracks'][trackid]['y_start']*10,f['tracks'][trackid]['y_end']*10),c='r',alpha=0.75,lw=0.1)
                       # (f['tracks'][trackid]['y_start']*10+218.236,f['tracks'][trackid]['y_end']*10+218.236),c='r',alpha=0.25)

for track in selected_tracks:
    ax.plot((track['x_start']*10,track['x_end']*10),
            (track['y_start']*10,track['y_end']*10),c='r',alpha=0)

for tpc in detector.TPC_BORDERS:
    tpc_rect = plt.Rectangle((tpc[0][0]*10,tpc[1][0]*10), 972.8, 3040, linewidth=0.1, edgecolor='k', facecolor=cmap(0),zorder=-1)
    ax.add_patch(tpc_rect)

ax.set_aspect("equal")
ax.set_xlabel("x [mm]")
ax.set_ylabel("y [mm]")
ax.add_patch(tpc_rect)
ax.set_xlim(np.min(detector.TPC_BORDERS[:,0,:])*10,np.max(detector.TPC_BORDERS[:,0,:])*10)
ax.set_ylim(np.min(detector.TPC_BORDERS[:,1,:])*10,np.max(detector.TPC_BORDERS[:,1,:])*10)
divider0 = make_axes_locatable(ax)
cax0 = divider0.append_axes("right", size="5%", pad=0.07)
fig.colorbar(m, ax=ax, cax=cax0, label='ADC count')
fig.savefig("mc.pdf")
```

```python

```


# Prologos

## List of Acronyms

+ DS - Dividing Surface.
+ PODS - Periodic Orbit Dividing Surface.
+ NHIM - Normally Hyperbolic Invariant Manifold.
+ UOP - Unstable Periodic Orbit.
+ RI - Reactive Islands.
+ LDs - Lagrangian Descriptors.
+ PSOS - Poincare Surface of Section.
+ TST - Transition State Theory.
+ TS - Transition State.
+ PES - Potential Energy Surface.

## Introduction & Background

Concepts arising in nonlinear dynamical systems theory, such as periodic orbits, normally hyperbolic invariant manifolds (NHIMs), and stable and unstable manifolds have been introduced into the study of chemical reaction dynamics from the phase space perspective. A fairly substantial literature has been developed on this topic in recent years (see, for example \cite{wiggins2016role, waalkens2010geometrical, waalkens2007wigner, wiggins2013normally}, and references therein), but it is fair to say that it has a more mathematical flavour, which is not surprising since these concepts originated in the dynamical systems literature. In this book we will  describe how these dynamical notions arise  in a variety of physically motivated settings with the hope of providing a more gentle entry into the field for both applied mathematicians and chemists.

An obstacle in interdiscipliary work is the lack of a common language for describing concepts that are common to different fields. We begin with a list of commonly used terms that will arise repeatedly throughout this book and provide a working definition.

Molecules are made up of a collection of atoms that are connected by chemical bonds and a reaction is concerned with the breaking, and creation, of these bonds. Hence, the following concepts are fundamental to the description of this phenomena.

+ **Coordinates.** The locations of the atoms in a molecule are described by a set of coordinates. The space (that is, all possible values) described by these coordinates is referred to as *configuration space*.

+ **Degrees-of-Freedom (DoF).** The number of  DoF is the number of independent coordinates required to describe the configuration of the molecule, that is the dimension of the configuration space.

+ **Reaction.** The breaking of a bond can be described by one or more coordinates characterizing the bond becoming unbounded as the it evolves in  time.

+ **Reaction coordinate(s).**  The particular coordinate(s) that describe the breaking of the bond are referred to as the *reaction coordinate(s)*.

+ **Energy.** The  ability of a bond to break can be characterised by its energy. A bond can be ''energized'' by transferring energy from other bonds in the molecule to a particular bond of interest, or from some external energy source, such as electromagnetic radiation, collision with other molecules, for example.

+ **Total Energy, Hamiltonian, momenta.** The total energy (that is, the sum of kinetic energy and potential energy) can be described by a scalar valued function called the *Hamiltonian*. The Hamiltonian is a function of the configuration space coordinates *and* their corresponding canonically conjugate coordinates, which are referred to as *momentum coordinates*.

+ **Phase Space.** The collection of all the configuration and momentum coordinates is referred to as the phase space of the system. The dynamics of the system (that is, how it changes in time) is described by Hamilton's (differential) equations of motion defined on phase space.

+ **Dimension count.** If the system has *n* configuration space coordinates, it has *n* momentum coordinates and then the phase space dimension is *2n*. The Hamiltonian is a scalar valued function of these  *2n* coordinates. The level set of the Hamiltonian, that is the energy surface, is *2n-1* dimensional. For a time-independent (autonomous) Hamiltonian, the system *conserves energy* and the energy surface is invariant.

+ **Transition State Theory (TST).** An approach in reaction rate calculations that is based on the flux across a dividing surface. We give a brief description of the theory in the next section.

+ **Dividing Surface (DS).** A DS on the energy surface is of dimension *2n-2*, that is, 1 dimension less (codimension 1) than the *2n-1* dimensional energy surface. While a codimension one surface has the dimensionality necessary to divide the energy surface into two distinct regions and forms the boundary between them. If properly chosen, the two regions are referred to as *reactants* and *products*, and reaction occurs when trajectories evolve from reactants to products through the DS. A DS has the ''locally no-recrossing'' property, which is equivalent to the Hamiltonian vector field being everywhere transverse to a DS, that is at no point is it tangent to the DS.

+ **Locally no-recrossing.** Computation of the flux is accurate only if the DS has the ''locally no-recrossing'' property. A surface has the ''locally no-recrossing'' property if any trajectory that crosses the surface leaves a neighbourhood of the surface before it can return.

+ **Globally no-recrossing.** A surface has the ''globally no-recrossing'' property if any trajectory that crosses the surface does so only once.

+ **The DS and the Reaction Coordinate.** Following our definitions of reaction, reaction coordinate, and DS it follows that the reaction coordinate should play a role in the definition of the DS. This will be an important point in our discussions that follow.

## Transition State Theory

Transition State Theory is a classical theory developed by Wigner, Eyring, Gwynne Evans and Polanyi,\cite{Wigner1932, Eyring1935,Polanyi1935,Eyring1941} that calculates the rate of the reaction as the equilibrium flux of reactive trajectories through the DS. Although there has been much discussion of quantum mechanical analogs of TST \cite{Pechukas1982, WaalkensWiggins2007}, we will briefly discuss in this book the original version based on classical mechanics, since the dividing surface concept, that we use, is incompatible with quantum mechanics. The overview that we give is not supposed to be a review of the vast literature that exists in this area, but to connect the  mathematical study of the geometrical structures that characterize reaction dynamics in phase space.

It is commonly claimed that conventional TST makes two main assumptions. \cite{Mahan1974}

- The first, called the equilibrium assumption, requires that the reactant state and TS be in thermal equilibrium. The maintenance of energetic equilibrium means that the thermalization maintaining this equilibrium is (at least) as fast as the rate at which these states are depopulated. \cite{TruhlarVTST2017,GarretTruhlarGTST1979} The equilibrium condition is usually satisfied for most gas-phase bimolecular reactions and for reactions in the liquid phase, because energy exchange between solutes and solvent is usually rapid enough to maintain the equilibrium.\cite{JBAnderson1973,JBAnderson1995} However, there are cases where equilibrium is not maintained, even in solution.\cite{EssafiHarvey2018} In addition, for unimolecular reactions of intermediates with low barriers to product formation, it is commonly the case that most trajectories coming from the reactant state will have enough energy in a product-forming reaction coordinate to cross the second barrier as soon as they reach it.\cite{Carpenter1985,EzraWiggins2014}

- The second claimed assumption specifies that any trajectory crossing the TS dividing surface from the reactant state is on a path towards the product state and will reach it without recrossing the dividing surface prior to the product being reached.\cite{TruhlarVTST1980} Because the rate is calculated as the flux through the TS, any non-reactive trajectory that crosses the TS dividing surface, or reactive trajectory that crosses it more than once will increase the flux through the dividing surface, thus leading to an overestimate of the rate constant. This means that TST gives us an upper limit on the true rate constant, and that if we found a dividing surface without any recrossing then TST would give the exact value of the rate constant (subject to certain caveats\cite{JCP2016Maug}).

In conventional TST the transition state dividing surface is located at the saddle point, which is the maximum energy point on the minimum energy path from reactants to products. However, TST is most powerful in the form of Variational Transition State Theory (VTST), \cite{TruhlarVTST2017,GarretTruhlarGTST1979,TruhlarVTST1980,TruhlarGarretVTST1984, GarretTruhlarJCP1979,Keck1960,Horiuti1938,Wigner1937} which is a generalization of TST that removes the restriction on the dividing surface to cross the saddle point. In VTST the dividing surface is variationally optimized to minimize the rate constant, usually by finding the maximum free energy along the reaction path. Although this surface is properly located in phase space, most of the VTST calculations assume that the TS can be found in configuration space.\cite{Carpenter1985}

## The index of a saddle point

We consider a  *n* degree-of-freedom Hamiltonian of the following form:

\begin{equation}
H(q, p) = \sum_{i=1}^{n} \frac{p_i^2}{2} + V(q), \quad (q,p) \in \mathbb{R}^n \times \mathbb{R}^n,
\label{ham_int}
\end{equation}

where $q \in \mathbb{R}^n$ denote the configuration space variables and $p \in \mathbb{R}^n$ denote the corresponding conjugate momentum variables.  This Hamiltonian function gives rise to the corresponding Hamilton's differential equations (or just ''Hamilton's equations'') having the following form:

\begin{eqnarray}
\dot{q}_i & = & p_i, \nonumber \\
\dot{p}_i & = & -\frac{\partial V}{\partial q_i} (q), \quad i=1. \ldots , n.
\label{hameq_int}
\end{eqnarray}

These are a set of *2n* first order differential equations defined on the phase space
$\mathbb{R}^n \times \mathbb{R}^n$.

A critical point of the potential energy function is a point $\bar{q} \in \mathbb{R}^n$ satisfying the following equations:

\begin{equation}
\frac{\partial V}{\partial q_i} (\bar{q}) =0, \quad i=1, \ldots n.
\end{equation}

Once a critical point of the potential energy function is located, we want to ''classify'' it. This is done by examining the second derivative of the potential energy function evaluated at the critical point. The second derivative matrix  is referred to as the *Hessian matrix*, and it is given by:

\begin{equation}
\frac{\partial^2 V}{\partial q_i \partial q_j} (\bar{q}) =0, \quad i,j=1, \ldots n,
\label{hessian}
\end{equation}

which is a $n \times n$ symmetric matrix. Hence \eqref{hessian} has *n* real eigenvalues, which we denote by:

\begin{equation}
\sigma_k, \quad k=1, \ldots, n.
\label{eiv_Hess}
\end{equation}

However, returning to dynamics as  given by Hamilton's equations \eqref{hameq_int}, the point $(\bar{q}, 0)$ is an equilibrium point of Hamilton's equations, i.e. when this point is substituted into the right-hand-side of \eqref{hameq_int} we obtain $(\dot{q}_1, \ldots, \dot{q}_n, \dot{p}_1, \ldots, \dot{p}_n) = (0, \ldots, 0, 0, \ldots, 0)$, i.e. the point $(\bar{q}, 0)$ does not change in time.

Next, we want to determine the nature of the stability of this equilibrium point. Linearized stability is determined by computing the Jacobian of the right hand side of  \eqref{hameq_int}, which we will denote by $M$, evaluating it at the equilibrium point $(\bar{q}, 0)$, and determining its eigenvalues.  The following calculation  is from \cite{ezra2004impenetrable}.
The Jacobian of the Hamiltonian vector field  \eqref{hameq_int} evaluated at $(\bar{q}, 0)$ is given by:

\begin{equation}
M =
\left(
\begin{array}{cc}
0_{n\times n} &  \rm{id}_{n \times n} \\
-\frac{\partial^2 V}{\partial q_i \partial q_j} (\bar{q}) & 0_{n\times n}
\end{array}
\right),
\end{equation}

which is a $2n \times 2n$ matrix.  The eigenvalues of $M$, denoted by $\lambda$,  are given by the solutions of the following characteristic equation:

\begin{equation}
{\rm det} \, \left( M - \lambda \, {\rm id}_{2n \times 2n} \right) =0,
\label{eivM}
\end{equation}

where ${\rm id}_{2n \times 2n}$ denoted the $2n \times 2n$ identity matrix. Writing \eqref{eivM} in detail (i.e. using the explicit expression for the Jacobian of \eqref{hameq_int}) gives:

\begin{equation}
{\rm det} \,
\left(
\begin{array}{cc}
-\lambda \, \rm{id}_{n \times n} & \rm{id}_{n \times n} \\
 -\frac{\partial^2 V}{\partial q_i \partial q_j} (\bar{q}) & -\lambda \rm{id}_{n \times n}
 \end{array}
 \right) =  {\rm det} \,  \left(\lambda^2 \, \rm{id}_{n \times n}  + \frac{\partial^2 V}{\partial q_i \partial q_j} (\bar{q})  \right) =0.
 \end{equation}

We can conclude from this calculation that the eigenvalues of the $n \times n$ symmetric matrix $\frac{\partial^2 V}{\partial q_i \partial q_j} (\bar{q})$ are $-\lambda^2$, where $\lambda$ are the eigenvalues of  the $n \times n$ matrix $M$. Hence, the eigenvalues of $M$ occur in pairs,  denoted by
 $\lambda_k, \, \lambda_{k+n}, \, k=1, \ldots n$, which have the form:

\begin{equation}
\lambda_k, \, \lambda_{k+n} = \pm \sqrt{-\sigma_k},  \quad k=1, \ldots, n,
\end{equation}

where $\sigma_k$ are the eigenvalues of the Hessian of the potential energy evaluated at the critical point $\bar{q}$ as denoted in \eqref{eiv_Hess}.  Hence,
we see that  the existence of equilibrium points of Hamilton's equations of ''saddle-like stability''  implies that there must be *at least* one negative eigenvalue of \eqref{hessian}. In fact, we have the following classification of the linearized stability of saddle-type equilibrium points of Hamilton's equations in terms of  the  critical points of the  potential energy surface.

+ **Index 1 saddle.** One eigenvalue of \eqref{hessian} is positive, the rest are  negative. We will assume that none of the eigenvalues of \eqref{hessian} are zero. Zero eigenvalues  give rise to special cases that must be dealt with separately. In the mathematics literature, these are often referred to as *saddle-center-$\cdots$-center equilibria*, with the number of center-$\cdots$-center terms equal to the number of pairs of pure imaginary eigenvalues.

+ **Index 2 saddle.**  Two eigenvalues of \eqref{hessian} are positive, the rest are  negative

and in general,

+ **Index k saddle.**  *k* eigenvalues of \eqref{hessian} are positive,thevrestvare negativev($k \le n$).

## References

\bibliography{myBib}


```python
import numpy as np
import scipy.special as sci
import matplotlib.pyplot as plt
from scipy import stats # linregress
import pandas as pd
from IPython.display import Latex
```

## Lecture 10:  Reactive Mass Transport

_(The contents presented in this section were re-developed principally by Dr. P. K. Yadav. The original contents are from Prof. Rudolf Liedl)_

---

### Motivation ###

The last lecture dealt with the  conservative transport processes and  quantified the mass flow and flux emanating from those processes. The effects of these processes were evaluated as an isolated processes and as joint transport process.

The last lecture dealt with the  conservative transport processes and  quantified the mass flow and flux emanating from those processes. The effects of these processes were evaluated as an isolated processes and as joint transport process.

```{admonition} Important conclusions from last lecture
>  $J_{adv}>J_{dis}>>J_{diff}$ is observed in normal aquifers.

> $J_{diff}$ may only be useful as an individual processes in special aquifers, e.g., clayey aquifers.

> In general aquifers, hydrodynamic dispersion $J_{hyd} = J_{dis} + J_{diff}$ is used in the analysis of solute transport process.
```

Finally, the last chapter introduced _Concentration Profile_ $(C-t)$ and _Breakthrough Curve_ $(C-x)$ to visually evaluate solute transport in aquifers using _Concentration_ $(C)$, a process output, as a function of _time_ $(t)$ and _space_ $(x)$.

This **Lecture** focuses on the _reactive transport processes,_ which as already discussed involves the transport of solute with _reaction_ processes. This course being an introductory groundwater course, _sorption_ and _degradation_ are the only two reaction types introduced and they are combined with the conservative transport processes- _advection_ and _dispersion_. Eventully, the section evaluates the joint action of conservative transport and reactive processes limiting to 1-D scenario.

The lecture will, however, first deal with 3-D effects of dispersive process, which is more important to quantify the reactive processes.

### Dispersive Mass Flow in 3-D ##

In the last lecture we saw that concentration gradient $\frac{\Delta C}{\Delta L}$ and flow velocity $v$ drives dispersive and diffusive solute transport processes.

However, in the natural aquifers $\frac{\Delta C}{\Delta L}$ is normally varying with space $(x,y,z)$ and time $(t)$. Therefore, a differential operator $\big(\frac{\mathrm{d}}{\mathrm{d} x}\big)$ is more suitable representation of gradient than the difference operator $\frac{\Delta C}{\Delta x}$. The differential operator also generalizes the gradient case.

Considering the differential operator, the diffusive mass flow and diffusive mass flux (= mass flow per unit area) in 1D is then expressed as:

$$
J_{diff} = - n_e \cdot A \cdot D_p \cdot \frac{\mathrm{d} C}{\mathrm{d}x}
$$

and
$$
j_{diff} = - n_e \cdot D_p \cdot \frac{\mathrm{d} C}{\mathrm{d}x}
$$

Likewise, the dispersive mass flow and dispersive mass flux (1-D) is:

$$
J_{disp, h} = - n_e \cdot A \cdot D_{disp} \cdot \frac{\mathrm{d} C}{\mathrm{d}x}
$$

$$
j_{disp, h} = - n_e \cdot D_{disp} \cdot \frac{\mathrm{d} C}{\mathrm{d}x}
$$

The examples of these relations are presented in the last lecture [Conservative Transport](/contents/transport/lecture_09/21_conservative_transport)

### 3-D concentration Gradient

The concentration gradient  $\frac{\mathrm{d}C}{\mathrm{d}x}$  for 1-D solute transport problems is uni-directional, i.e, direction is fixed, and thus only the magnitude of the gradient is the important factor. However in higher dimensions, 2-D or 3-D, solute transport problems, the _direction_ of gradient along with it's _magnitude_ in that direction has to be specified. Thus, for higher dimension solute transport problems, the _concentration gradient_ becomes **concentration vector**, i.e., a quantity providing both magnitude and direction.

Thus, the representation of concentration gradient in Cartesian coordinate in 2-D and 3-D is:

$$
\mathrm{grad}\,C = \nabla C=  \begin{pmatrix}
\frac{\partial C}{\partial x}\\
\frac{\partial C}{\partial y}
\end{pmatrix}
$$

and

$$
\mathrm{grad} \,C = \nabla C=  \begin{pmatrix}
\frac{\partial C}{\partial x}\\
\frac{\partial C}{\partial y}\\
\frac{\partial C}{\partial z}
\end{pmatrix}
$$

The $\nabla$, the inverted Delta symbol, is called the **del** or **nabla** operator. The vector **grad $C$** in the above relations points in the direction of the _steepest increase_ of $C$. However, for the **Hydrogeologists**, the concentration gradients as well the grad $C$ points to the _steepest decrease_ of $C$.

### Isotropic and Anisotropic  Dispersion

Corresponding the expression for the concentration gradient at higher dimensions, the expression for mass flow and flux becomes:

$$
J_{disp,\, h} = \begin{pmatrix}
J_{disp,\, hx}\\
J_{disp,\, hy}\\
J_{disp,\, hz}
\end{pmatrix}
$$
and the 3-D mass flux is:

$$
j_{disp,\, h} = \begin{pmatrix}
j_{disp,\, hx}\\
j_{disp,\, hy}\\
j_{disp,\, hz}
\end{pmatrix}
$$

The subscript in ${{disp, h}}$ refers to _hydrodynamic dispersion_ which is sum of _mechanical dispersion_ and _diffusion_. Likewise, the subscript ${{disp,\, hx}}$, ${{disp,\, hy}}$ and ${{disp,\, hz}}$ refers to dispersion components along the Cartesian coordinates. The corresponding mass flow and mass flux in the higher dimension is then:

$$
J_{disp,\, h} = - n_e \cdot A \cdot D_{hyd} \cdot \text{grad}C
$$

and

###  Isotropic and Anisotropic  Dispersion

Corresponding the expression for the concentration gradient at higher dimensions, the expression for mass flow and flux becomes:

$$
J_{disp,\, h} = \begin{pmatrix}
J_{disp,\, hx}\\
J_{disp,\, hy}\\
J_{disp,\, hz}
\end{pmatrix}
$$
and the 3-D mass flux is:

$$
j_{disp,\, h} = \begin{pmatrix}
j_{disp,\, hx}\\
j_{disp,\, hy}\\
j_{disp,\, hz}
\end{pmatrix}
$$

The subscript in ${{disp,\, h}}$ refers to _hydrodynamic dispersion_ which is sum of _mechanical dispersion_ and _diffusion_. Likewise, the subscript ${{disp,\, hx}}$, ${{disp,\, hy}}$ and ${{disp,\, hz}}$ refers to dispersion components along the Cartesian coordinates. The corresponding mass flow and mass flux in the higher dimension is then:

$$
J_{disp,\, h} = - n_e \cdot A \cdot D_{hyd} \cdot \text{grad}C
$$

and

$$
j_{disp,\, h} = - n_e \cdot A \cdot D_{hyd} \cdot \text{grad}C
$$

The **isotropic dispersion**, rather an _exceptional,_ the $D_{hyd}$ in this case is:

$$
D_{hyd} = \alpha \cdot |v| + n_e \cdot D
$$
where, $D$ is direction independent dispersion coefficient and $\alpha$ $[L]$ is dispersivity, which in:

**heterogeneous aquifer**: $\alpha = \alpha(x,y,z)$ and in

**homogeneous aquifer**: $\alpha = \text{constant}$

For more practical cases and in normal aquifers, the 2-D and 3-D the dispersion for solute transport is _direction dependent,_ i.e. **anisotropic**. Hence the $D_{hyd}$ is not an scalar quantity but a _matrix (tensor),_ which relates the concentration gradient (vector) to the dispersive mass flow (vector). However, if the princopal axes of the dispersion tensor $D_{hyd}$ is made to coincide with the axes of a Cartesian coordinate system _and_ the groundwater flow is considered uniform along the $x-$axis, the dispersive mass flux can be obtained from

$$
\begin{pmatrix} J_x \\ j_y \\ j_z \end{pmatrix} =
\begin{pmatrix} \alpha_L \cdot v_x + n_e \cdot D & 0 & 0 \\
0 & \alpha_{Th} \cdot v_x + n_e \cdot D & 0\\
0 & 0 & \alpha_{Tv} \cdot v_x + n_e \cdot D
\end{pmatrix}
\cdot
\begin{pmatrix} \frac{\partial C}{\partial x} \\ \frac{\partial C}{\partial y} \\ \frac{\partial C}{\partial z} \end{pmatrix}
$$

with $\alpha_L$, $\alpha_{Th}$ and $\alpha_{Tv}$ are longitudinal dispersivity, horizontal transverse dispversity and vertical transverse dispersivity, respectively. The statistical analysis of dispersivity data shows that $\alpha_{L}>\alpha_{Th}>\alpha_{Tv}$ and the values differ roughly by an order of magnitude. This, however, is just a rule of thumb.

```{admonition} A quick example
:class: tip
Discuss the role of 2D dispersitvity in  column when discharge is limited at 10 m$^3$/d of a chemical with concentration 1 mg/L. The flow velocity can be assumed to be 0.05 m/d.
```

```python
# Analytical solution from Bear (1976) - Line source, 1st-type input and infinte plane

# Input (values can be changed)
Co = 1 # mg/L, input concentration
Dx = 3 # m, Dispersion in x direction
Dy = Dx/10 # m
v = 0.05 # m/d
Q = 10 # m^3/d

## domain dimension and descritization (values can be changed)
xmin = -100; xmax= 101
ymin = 0.1; ymax = 11
[x, y] = np.meshgrid(np.linspace(xmin, xmax, 1000), np.linspace(ymin, ymax, 100)) # mesh

# Bear (1976) solution Implementation
#"k0: Modified Bessel function of second type and zero Order"

term1 = (Co*Q)/(2*np.pi* np.sqrt(Dx*Dy))
term2 = (x*v)/(2*Dx)
args = (v**2*x**2)/(4*Dx**2) + (v**2*y**2)/(4*Dx*Dy)
sol = term1*np.exp(term2)*sci.k0(args)

# plots
fig, ax = plt.subplots()
CS = ax.contour(x,y,sol, cmap='flag')
ax.clabel(CS, inline=1, fontsize= 10)
CB = fig.colorbar(CS, shrink=0.8, extend='both');
```

### Equilibrium Sorption ###

A reactive transport system can include a single reactive process, e.g., degradation, or combination of several multiple reactive processes, e.g. degradation and sorption. The inclusion of the reactive process(es) in the transport studies are site specific. The important to note is that an inclusion of a reactive process increases the complexity of transport problem.

In this course we limit ourselves with the following two types of reaction processes:

**1. Sorption**

**2. Degradation**

Acid-base reaction, precipitation-dissolution reaction, organic combustion etc. are among the reactions type that can be part of the reactive process individually or in any combination.

Also an important distinction is the rate or speed of the reaction. One distinguishes between time-dependent reaction (kinetics) or time-independent reactions (steady-state or equilibrium). Special reaction rates such as instantaneous reaction (extremely fast reaction) can also be part of the reaction process in the transport system.

### Sorption Basics

**Sorption** is a rather a general term used to indicate both **adsorption** and **absorption**. But in this course the _sorption_ refers to only _adsorption._

**Adsorption** can be more formally defined as the process of accumulation of dissolved chemicals on the surface of a solid, e.g., accumulation of a chemicals dissolved in groundwater on the surface of the aquifer material.

The figure below clarifies the _adsorption_ process.

```{figure} images/T10_f1.png
---
scale: 30%
align: center
name: Sorption
---
Sorption terminology
```

In the figure _chemical in solution_ (the circular objects) more often called **solute** in the water is found to attach is the solid surface. The figure presents the following two important terms part of the adsorption process:

> **Adsorbent**: The solid onto which the chemicals are attached. More formally, _adsorbents_ provide adsorption sites for solutes.

> **Adsorbate**: These are solutes that are attached on the _adsorbent._

Based on the figure, _adsorption_ can be considered as a partition process that divides the chemical originally present in water between adsorbent and water.

Quite often adsorption is a reversible process, i.e., adsorbed chemicals can get back to water phase. This process is called **desorption**.

Speaking about _equilibrium,_ this is reached when

> _adsorption rate_  $\rightleftharpoons $ _desorption rate_

Adsorption in groundwater is often a rapid process. Although sorption kinetics can be important, the description in this introductory level course is limited to equilibrium sorption. Thus, we learn next to quantify equilibrium sorption.

### Adsorption Isotherms

The adsorption process that has reached equilibrium can be relatively easily quantified with the use of empirical models called **isotherms**. These models are often simple algebraic equation that relates solute concentrations partitioned between the adsorbate and adsorbent at constant temperature. More than 15 different _isotherm_ models can be found in the literature. However, in groundwater reactive transport studies the following three are the two most commonly used isotherms:

1. **Henry or Linear isotherm**

2. **Freundlich isotherm**

For quantification, laboratory based experiments are performed using solids from subsurface and chemicals of interest. The laboratory observations are then graphically fitted with empirical isotherm models to quantify adsorption properties. Figure below shows isotherms that are particularly observed in groundwater transport studies. As can be observed in the figure sorption coefficient ($K$) is the common quantities obtained from isotherm models.

```{figure} images/T10_f2.png
---
scale: 40%
align: center
name: sorption type
---
Different types of sorption isotherms
```

### Henry Isotherm

The **Henry isotherm** (Henry, 1803) is based on the idea of a _linear_ relationship between the solute concentration **$C$** and the _adsorbate:adsorbent_ mass ratio $C_a$. Henry isotherm is quite often also called _linear isotherm_ or the $K_d$ model. Mathematically, the Henry isotherm is:

$$
C_a = K_d \cdot C
$$

with

$C$ = solute concentration [ML$^{-3}$]

$C_a$ = mass ratio adsorbate:adsorbent [M:M]

$K_d$ = distribution or partitioning coefficient [L$^3$M$^{-1}$].

Often symbols $C_s$ or $s$ are used instead of $C_a$.

The Henry model has been most widely used in groundwater transport studies. This is largely because of the simplicity (see equation) of the model and it's applicability in representing adsorption process more generally observed in groundwater studies. $K_d$, the partitioning coefficient, is particularly used in groundwater transport studies. It is equal to the slope of the Henry isotherm.

```{admonition} A quick example
:class: tip
From the experimental data provided below, obtain the Henry distribution coefficient.
```

```python
# Example of Henry isotherm (Source: Fetter et al. 2018)

# Following sorption data are available:

C = np.array([7, 15, 174, 249, 362]) # ug/L, Eq. concentration
Ca = np.array([2, 4, 33, 50, 70]) # ug/g, Eq. sorbed mass

# linear- fit y = m*x+c

slope, intercept, r_value, p_value, std_err = stats.linregress(C, Ca)
print("slope = %0.3f    intercept= %0.3f  R-squared=%0.4f" % (slope, intercept, r_value**2),'\n')
fit_line = slope*C + intercept

#plot

plt.scatter(C, Ca, label= "Original data") # data plot
plt.plot(C, fit_line, color = "red", label = "fit-line")
plt.legend(); plt.xlabel(r"Equilibrium Aqueous Concentration, $C$ ($\mu$g/L) ")
plt.ylabel(r"Mass sorbed per unit absorbent weight, $C_a$ ($\mu$g/g) ");
plt.text(0, 50, '$C_a=%0.5s C + %0.5s$'%(slope, intercept), fontsize=10)
plt.text(0, 40, '$R^2=%0.5s $'%(r_value**2), fontsize=10)

# Output

print("The required partition coefficient = slope,= %0.5s L/g " % slope)
```

### Freundlich Isotherm

**Freundlich isotherm** (Freundlich, 1907) is a more general isotherm. It is based on the idea of a power law, i.e., includes also the non-linear behaviour, relating the solute concentration $C$ to the adsorbate:adsorbent mass ration $C_a$. The isotherm is mathematically given as

$$
C_a = K_{Fr} \cdot C^N
$$

with

$C$ = solute concentration [ML$^{-3}$]

$C_a$ = mass ratio adsorbate:adsorbent [M:M]

$n$ = Freundlich exponent [-]

$K_{Fr}$ = Freundlich partitioning coefficient [(M:M)/(M/L$^3)^n$].

The Freundlich isotherm equation can be easily linearized by applying logarithmic transformation of the equation, which gives

\begin{eqnarray*}
\log C_a = \log K_{Fr} + n\cdot \log C
\end{eqnarray*}

The above equation resembles the straight line equation $y = b + a \cdot x b$, in which $b\equiv \log K_{Fr}$ is the intercept and $a\equiv n$ the slope. Thus, from fitting the adsoprtion experimental results with the above equation, both $n$ and $K_{Fr}$ can be obtained.

```{admonition} A quick example
:class: tip
From the experimental data provided below, obtain the Freundlich partitioning coefficient and Freundlich exponent.
```

```python
# Example of Freundlich isotherm

# Following sorption data are available:

Cf= np.array([23.6, 6.67, 3.26, 0.322, 0.169, 0.114]) # mg/L, Eq. concentration
Caf = np.array([737, 450, 318, 121, 85.2, 75.8]) # mg/g, Eq. sorbed mass

logCf = np.log10(Cf) # log10 transformation of data
logCaf = np.log10(Caf)

# fitting: y = mx +c
slope, intercept, r_value, p_value, std_err = stats.linregress(logCf, logCaf)
print("slope: %0.3f    intercept: %0.3f  R-squared: %0.3f" % (slope, intercept, r_value**2))
fit_line = slope*logCf + intercept

# plots
plt.figure(figsize=(10,4))

plt.subplot(121)
plt.plot(Cf, Caf, "*--", label= "Original data")
plt.legend(); plt.xlabel(r"Eq. Aq. Conc., $C$ ($mg$g/L) ");
plt.ylabel(r"Mass sorbed/absorbent weight, $C_a$ ($mg$g/g) ");

plt.subplot(122)
plt.scatter(logCf, logCaf, label="Log transformed data")
plt.plot(logCf, fit_line, color="red", label= "linear fit line")
plt.legend(); plt.xlabel(r"Eq. Aq. Conc., $\log C$ ($mg$g/L) ");
plt.ylabel(r"Mass sorbed/absorbent weight, $\log C_a$ ($mg$/g) ");
plt.text(-1, 2.6, '$C_a=%0.5s C + %0.5s$'%(slope, intercept), fontsize=10)
plt.text(-1, 2.5, '$R^2=%0.5s $'%(r_value**2), fontsize=10)
plt.subplots_adjust(wspace=0.35)

print("Freundlich partitioning coefficient =  %0.5s (mg/g)1/n (mg/L) and  Freundlich exponent = %0.4s"  % (10**intercept, slope))
```

### Retardation Factor (for Henry Isotherm)

The net effect of adsorption is the retarded movement of solute in comparison to the average flow of the groundwater. The term **Retardation Factor** $(R)$ is defined that quantifies the retarded movement of solute. The formulation of $R$ is based on the type of isotherm. For Henry isotherm $R$ can be straightforwardly calculated with the help of a mass budget.

For this purpose, an aquifer volume $V$ with the effective porosity $n_e$ is considered (see fig. below)

```{figure} images/T10_f3.png
---
scale: 70%
align: center
name: Retardation
---
Retardation factor
```

The steps involved are:

- Total volume: $V$
- Water volume: $n_e \cdot V$
- Mass of dissolved chemical: $n_e \cdot V \cdot C$
- Volume of solid: $(1-n_e)\cdot V$
- Density of solid material: $\rho$
- Mass of solid: $\rho \cdot(1-n_e)\cdot V$
- Mass of adsorbate: $\rho \cdot(1-n_e)\cdot V\cdot C_a$ = $(1-n_e)\cdot\rho \cdot V \cdot K_d \cdot C$
- Total mass: $n_e\cdot V \cdot C + (1-n_e)\cdot\rho \cdot V\cdot K_d \cdot C = n_e \cdot R \cdot V \cdot C$

with _Retardation factor_ $$R = 1 + \frac{1-n_e}{n_e}\cdot \rho \cdot K_d$$

The expression for $R$ can be further modified  by using bulk density $\rho_b$ $= (1-n_e)\cdot \rho$ = mass of solid/total volume. This leads to

$$
R = 1+\frac{\rho_b}{n_e} \cdot K_d
$$

As can be observed from the equation, $R = 1$ when there is no adsorption, i.e., when $K_d= 0$.

```{admonition} A quick example
:class: tip
Calculate the retardation factor from the provided data.
```

```python
print("\033[0m You can change the provided values.\n")

ne = 0.4 #effective porosity [-]
rho = 1.25 # density of solid material [kg/m³]
Kd = 0.2 # distribution or partition coefficient [m³/kg]

#intermediate calculation
rho_b = (1-ne)*rho

#solution
R=1+(rho_b/ne)*Kd

print("effective porosity = {}\ndensity of solid material = {} kg/m³\nDistribution or partition coefficient = {} m³/kg\n".format(ne, rho, Kd))
print("\033[1mSolution:\033[0m\nThe resulting retardation factor is \033[1m{:02.4}\033[0m.".format(R))

```

     You can change the provided values.

    effective porosity = 0.4
    density of solid material = 1.25 kg/m³
    Distribution or partition coefficient = 0.2 m³/kg

    Solution:
    The resulting retardation factor is 1.375.

### Degradation

**Degradation** leads to alteration or transformation of chemical structure of chemicals. This contrasts to adsorption in which chemical structure is not altered. In adsorption (or desorption) the original chemical is partitioned between the solid particles and water. It is _degradation_ that eventually lead to removal of the _original_ chemical from the groundwater. The transformation of original chemical, due to degradation, results to so-called _daughter products (metabolites)._ The new chemical(s) can make groundwater more suitable (decrease contamination) or further contaminate it.

In groundwater studies, degradation can appear as:

- **Radioactive decay**
- **Microbial degradation (bio-degradation)**
- **Chemical degradation**

There are several approaches to quantify degradation process.
A common aspect to most of them is the assumption of _time-dependency_ (or _Kinetics_ ).

### $n^{th}$ - Order Degradation Kinetics

The general equation for the degradation kinetics is:

$$
\frac{\text{d}C}{\text{d} t} = - \lambda \cdot C^n
$$

with $t$ = time [t]

$C$ = solute concentration [ML$^{-3}$]

$n$ = order of the degradation kinetics [ - ] ($n\geq 0)$

$\lambda$ = degradation rate constant [(ML$^{-3})^{(1-n)}$T$^{-1}$].

Considering the initial concentration (or input concentration) $C_0$, the solutions of the kinetics equation are:

$$
C(t) = C_0\cdot e^{-\lambda \cdot t} \: \: \: \text{if }\: n = 1
$$

and

$$
C(t) = [C_0^{1-n} - (1-n)\cdot \lambda t]^{\frac{1}{1-n}} \:\:\: \text{if }\: n\neq 1
$$

The **half life** $(T_{1/2})$, which is the time span elapsing until the initial concentration $C_0$ is reduced by half, is an important time-scale in the degradation analysis. $T_{1/2}$ is $C_0$ dependent in nearly all cases with an exception for 1$^\text{st}$- order degradation kinetics. 0$^{th}$-order and the 1$^\text{st}$- order degradation kinetics are most commonly observed in groundwater studies. The $(T_{1/2})$ of these orders are:

$$
T_{1/2} = \frac{C_0}{2\cdot \lambda} \:\:\: \text{for } \:0^{\text{th}}\text{-order}
$$

$$
T_{1/2} = \frac{\ln 2}{\lambda} \:\:\: \text{for } \:1^{\text{st}}\text{-order}
$$

As can be observed above $T_{1/2}$ is independent of concentration for the 1$^{\text{st}}$-order degradation kinetics.

Another important properties of the degradation kinetics is that for $n\geq 1$ the solute concentration _asymptotically_ approaches zero, whereas for $n<1$, the solute concentration actually reaches zero

```python
# behaviour of degradation kinetics

#input - you may change the values
Co = 1 # mg/L, initial concentration
la = 0.003 # unit is order dependent. For n=1, 1/t

# main equation
Z_order = lambda t: Co* np.exp(-la*t) # for n = 0
F_order = lambda t: Co-la*t # for n = 1

# simulation for t
t = np.linspace(1,1000, 1000) # 1000 time units
Z_results = Z_order(t)
F_results = F_order(t)

# plots
plt.figure(figsize=(10,4))

# n = 1
plt.subplot(121)
plt.plot(t, Z_results)
plt.ylim(0, Co); plt.xlim(0)
plt.text(400, Co*0.8, r"$C(t) = C_0 \cdot e^{-\lambda \cdot t} $", fontsize = 12)
plt.text (400, Co*0.9, r"1$^{st}$-order kinetics", color = "red", fontsize = 12)
plt.text(0, Co/2, r"$T_{1/2}= \frac{\ln 2}{\lambda}$", color= "red", fontsize=14)
plt.xlabel("Time, t  (days)"); plt.ylabel(r"Concentration, $C(t)$ (mg/L)")

# n = 0
plt.subplot(122)
plt.plot(t, F_results)
plt.ylim(0, Co); plt.xlim(0)
plt.text(400, Co*0.8, r"$C(t) = C_0 \cdot-\lambda \cdot t} $", fontsize = 12)
plt.text (400, Co*0.9, r"0$^{th}$-order kinetics", color = "red", fontsize = 12)
plt.text(0, Co/2, r"$T_{1/2}= \frac{C_0}{2\cdot \lambda}$", color= "red", fontsize=14)
plt.xlabel("Time, t  (days)"); plt.ylabel(r"Concentration, $C(t)$ (mg/L)")

plt.subplots_adjust(wspace=0.35)
```

### Radioactive decay

Radioactive decay is degradation of a chemical due to radiation. The radioactive decay is limited to radioactive chemicals such as Cobalt, Cesium, Iodine. This decay obeys the 1$^\text{st}$- order degradation kinetics and therefore the half-life is $T_{1/2} = \frac{\ln 2}{\lambda}$. $T_{1/2}$ is characteristic property of radioactive chemicals and it can be used to compute degradation rate ($\lambda$).

```python
# Example of Radioactive decaly

#experimental results

t = [0, 1, 2, 5, 10, 20, 28 ] # yr, time
Co_60 = [10, 8.76, 7.68, 5.17, 2.68, 0.72, 0.25] # mg/L, Cobalt 60 conc.
So_90 = [10, 9.76, 9.52, 8.84, 7.81, 6.10, 5] # mg/L, Strontium 90 Conc.

z_list = list(zip(t, Co_60, So_90))

Cols= ["time (a)", "Cobalt 60 (mg/L)", "Strontium 90 (mg/L)"]
df = pd.DataFrame(z_list, columns=Cols)
print(df)

# computing
TH_Co60 = 28 # yr, Half life of Cobalt 60
TH_St90 = 5.26 # yr, Half life of Strontium 90
la_Co60 = np.log(2)/TH_Co60 # 1/yr, degradation rate of Cobalt 60
la_St90 = np.log(2)/TH_St90 # 1/yr, degradation rate of Strontium 90

# visualize
plt.plot(t, Co_60, "o--", label = "Cobalt 60")
plt.plot(t, So_90, "v--", label= "Strontium 90")
plt.xlabel("Time (years)"); plt.ylabel("Concentration (mg/L)")
plt.legend();

print("\n The degradation rate (\u03BB) for Cobalt 60 = %0.5s 1/y and for Strontium 90 = %0.5s 1/y \n"  % (la_Co60, la_St90))
```

### Joint Action of Conservative and Reactive Transport (1D)

### Concentration Profile

Figure below presents the joint action of conservative transport with equilibrium sorption (linear isotherm) and degradation. The figure shows the solute concentration $C$ (in water) at the same time-levels for various combinations of acting processes.

```{figure} images/T10_f4.png
---
scale: 60%
align: center
name: 1D_cons_react
---
1D Conservative and Reactive Transport
```

The figure can be explained in the following way:

(A): The solute is initially present at constant concentration in a limited area.

(B): Solute spreads only due to advection. Due to absence of dispersion there is no (1D) spreading effect.

(C): Inclusion of dispersion process causes spread of concentration. As retardation is absence the front centreline remains unchanged

(D): The inclusion of retardation ($R$) with advection and dispersion leads to removal of chemicals from water and as well the retarded movement of the chemical front.

(E): The inclusion of retardation along with degradation and conservative transport process leads to high removal of chemical from water.

### Breakthrough Curve

Breakthrough curves provide a _time-dependent_ spread of chemicals in the groundwater. The inclusion of multiple processes are normally solved using numerical models. Analytical models are available for limited processes and simplified problems. A 1-D analytical solution by Kinzelbach (1987) provide a transient (time-dependent) solution of reactive transport problem with inclusion of equilibrium linear sorption represented by retardation $(R)$, first-order degradation rate $(\lambda)$ and the conservative transport quantities - dispersion $(D)$ and advection. The solution is given as:

$$
C(x,t) = C_0 \cdot \exp(-\lambda\cdot t)\bigg(1- \frac{1}{2}\text{erfc}\bigg(\frac{R\cdot x - v\cdot t}{2\cdot\sqrt{D\cdot R \cdot t}}\bigg) - \frac{1}{2}\exp\bigg(\frac{v\cdot x}{D}\bigg)\text{erfc}\bigg(\frac{R\cdot x + v\cdot t}{2\cdot\sqrt{D\cdot R \cdot t}}\bigg)
$$

with $C_0$ = input/source concentration [ML$^{-3}$]

$t$ = time [T]

$v$ = groundwater flow velocity [LT$^{-1}$]

erfc() = represents the complementary error function [See here for details](https://en.wikipedia.org/wiki/Error_function). erfc() can be easily computed using Python Scipy special function library.

```python
# Breakthrough curve using Kinzelbach (1987) analytical solution Main function

# The main function - you may change the value of C_o, lam, R, Dx, v, x
# C_o = input concentration, mg/L
# lam = 0 # 1/d, degradation rate, 1/d
# R = retardation factor, ()
# Dx = dispersion coeff. along x, m^2/d
# v = groundwater velocity, m/d
# x = position where C is to be measured, m

def Cx(t, C_o= 1, lam = 0, R=1, Dx=1, v= 10, x = 20):
    sterm = C_o*np.exp(-lam*t)
    erf_ag1 = (R*x-v*t)/(2*np.sqrt(Dx*R*t))
    erf_ag2 = (R*x+v*t)/(2*np.sqrt(Dx*R*t))

    C = sterm*(1-(0.5*sci.erfc(erf_ag1)-0.5*np.exp((v*x)/Dx)*sci.erfc(erf_ag2)))
    return C

```

```python
# Computing Case 1: Conservative process- R = 1, Lambda = 0
t1 = np.linspace(1e-5,50,1000) # times, d
C1 = Cx(t1, C_o= 1, lam = 0, R=1, Dx=1, v= 1, x = 20)

# Computing Case 2: Conservative system + Retardation - R = 2, Lambda = 0
t2 = np.linspace(1e-5,50,1000) # times, d
C2 = Cx(t2, C_o= 1, lam = 0, R=2, Dx=1, v= 1, x = 20)

# Computing Case 3: Conservative system + Retardation + degradation - R = 2, Lambda = 0.004
t3 = np.linspace(1e-5,50,1000) # times, d
C3 = Cx(t3, C_o= 1, lam = 0.004, R=2, Dx=1, v= 1, x = 20)

# plots - this should be adjusted as required

plt.figure(figsize=(9, 6))

plt.plot(t1, C1, label="Conservative transport")
plt.plot(t2, C2, label = "Reactive transport with sorption")
plt.plot(t3, C3, label = "Reactive transport with sorption and degradation rate")
plt.legend(loc= 3); plt.xlim(0), plt.ylim(0)
plt.xlabel("time (d)"); plt.ylabel(r"Concentration, $C$ (mg/L)")
plt.text(5, 0.2, r"$x= 20$ m")
```

### Mass (Re-)Distribution During Injection / Extraction

**Consider a scenario**:

Water is _injected_ into a certain portion of an aquifer with total volume $V$, bulk density $\rho_b$ and effective porosity $n_e$. Assume that the injected water contains a chemical of total mass $M$, which is adsorbed by the aquifer materials under equilibrium conditions according to Henry isotherm (quantified by $K_d$$).

Bases on the assumption of sorption equilibrium, the total mass $M$ of the chemical is instantaneously(!) split up into a dissolved and a sorbed part. In such case, the mass distribution can be computed as follows (with $R$ = retardation factor, $\rho_b$
= bulk density):

\begin{align}
M &= n_e \cdot V \cdot C + V\cdot\rho_b \cdot C_a \\
&= n_e \cdot V \cdot C + V \cdot \rho_b \cdot K_d \cdot C\\
&= n_e \cdot (1 + \rho_b \cdot K_d/n_e) \cdot V \cdot C\\
&= n_e \cdot R \cdot V \cdot C
\end{align}

In which,

$n_e \cdot V \cdot C$ = dissolved mass

$V\cdot\rho_b \cdot C_a$ = mass of adsorbate

For the dissolved mass we thus have $n_e \cdot V \cdot C = M/R$ and consequently the mass of adsorbate is:

$V\cdot\rho_b \cdot C_a = M- M/R = (1-1/R)\cdot M$

The same approach can be adopted for the **extraction** scenarios, i.e. equilibrium desorption.

### Additional Tool ###

The additional tool: [1D-Advection-Dispersion Simulation Tool](/contents/tools/1D_advection_dispersion) simulates all the concepts that are provided above. The tool simulates:

- 1D solute transport in porous media (e.g., laboratory column)
- uses unifrom cross-section
- steady-state water flow
- input of tracer

The output are then:

- spreading of tracer due to advection and mechanical dispersion
- computation and graphical representation of a breakthrough curve
- comparison with measured data.