```python
# %load ../../preconfig.py
%matplotlib inline
import matplotlib.pyplot as plt
import seaborn as sns
plt.rcParams['axes.grid'] = False

import numpy as np
import pandas as pd
#import itertools

import networkx as nx

import string

import logging
logger = logging.getLogger()
```

10 Mining Social-Network Graphs
=================
1. how to identify "communities"?
   communities: strong connections, usually overlap.

2. explore efficient algorithms for discovering other properities of graphs.

### 10.1 Social Networks as Graphs
#### 10.1.1 What is a Social Network?
The essential characteristics of a social network are:

1. There is a collection of entities that participate in the network.

2. There is at least one relationship between entities of the network.

   + all-or-nothing: friends of Facebook.

   + discrete degree: friends, family as in Google plus.

   + real number: the times of talking.

3. There is an assumption of nonrandomness or locality.
   If A is related to both B and C, then there is a higher probability than average that B and C are related.

#### 10.1.2 Social Networks as Graphs
social graph: The entities are the nodes, and an edge connects two nodes if the nodes are realted by the relationship that characterizes the network.

```python
plt.imshow(plt.imread('./res/fig10_1.png'))
```

Is Fig 10.1 typical of a social network, in the sense that it exhibits locality of relationships?

Suppose X, Y and Z are nodes of Fig 10.1, with edges between X and Y and also between X and Z.

1. What would we *expect* the probability of an edge between Y and Z to be?
   The graph has 9 edges out of the $C_7^2 = 21$ pairs of nodes.

   Since we already have (X,Y) and (X,Z), the probability of an edge (Y,Z) is $(9-2)/(21-2) = 0.368$.  If the graph were large enough, that probability would be very close to the fraction of the pairs of nodes that have edges between them, i.e., $9/21 = 0.429$.

2. Then, we must compute the probability that $(Y,Z)$ exists in Fig 10.1, given that edges $(X,Y)$ and $(Y,Z)$ exist.
   We count the pairs of nodes that could be $Y$ and $Z$, without worrying about which node is $Y$ and which is $Z$.

   If $X$ is $A$, $(B,C)$ contributes one positive example. All details are in the table following. In all, the fraction of times the third edge exists in thus $9 / 16 = 0.563 > 0.368$.

| node | pos | neg |
|------|-----|-----|
| A | 1 | 0|
| C | 1 | 0|
| E | 1 | 0|
| G | 1 | 0|
| F | 2 | 1|
| B | 1 | 2|
| D | 2 | 4|
|sum| 9 | 7| 16 |

#### 10.1.3 Varieties of Social Networks
1. Telephone Networks

2. Email Networks

3. Collaboration Networks
   + Wikipedia: articles and editors.
   + published research papers and authors.

4. Other example
   + information networks (patents)
   + infrastruture networks (roads)
   + biological networks (genes)
   + product co-purchasing networks (Groupon)

### 10.1.4 Graphs with Several Node Types
There are other social phenomena that involve entities of different types. The natural way to represent such information is as a $K$-partite graph for some $k > 1$. In general, a $k$-partite graph consists of $k$ disjoint sets of nodes, with no edges between nodes of the same set.

Eg:
deli.cio.cu: there are three different kinds of entites: users, tags and pages.

```python
plt.imshow(plt.imread('./res/fig10_2.png'))
```

`#exercise`

### 10.2 Clustering of Social-Network Graphs
clustering of the graph as a way to identify communities.

#### 10.2.1 Distance Measures for Social-Network Graphs
distance measure:
\begin{align}
    d(x,y) =
    \begin{cases}
         0 \text{ or } 1 \text{ or } 1 & \text{if edge} (x,y) {exists} \\
         1 \text{ or } \infty \text{ or } 1.5 & \text{otherwise}
    \end{cases}
\end{align}

#### 10.2.2 Applying Standard Clustering Methods
They are generally unsuitable for the problem of clustering social-network graphs.

#### 10.2.3  Betweenness: define
*betweenness* of an edge $(a,b)$:
the number of pairs of nodes $x$ and $y$ such that $(a,b)$ lies on the shortest path between $x$ and $y$. And it's credicted with the fraction of all shortest paths between $x$ and $y$ if existed.

A high score indicates that $(a,b)$ runs between two different communities, namely, $a$ and $b$ do no belong to the same community.

#### 10.2.4 The Girvan-Newman Algorithm: calculate betweenness
The algorithm is aimed to calculate the number of shortest paths going through each eage.

1) Starting at a node $X$, perform a breadth-first search (BFS) of the graph to label levels of each node.
   + DAG edges: edges between levels.

   + If there is a DAG edge $(Y,Z)$, where $Y$ is at the level about $Z$, then we shall call $Y$ a *parent* of $Z$ and $Z$ a *child* of $Y$.

2) to label each node by the number of shortest paths that reach it from the root.
   + Start by labeling the root 1.
   + Then, from the top down, label each node $Y$ by the sum of the labes of its parents.

```python
plt.imshow(plt.imread('./res/fig10_4.png'))
```

3) to calculate for each edge $e$ the sum over all nodes $Y$ of the fraction of shortest paths from the root $X$ to $Y$ that go through $e$.
   + from the bottom up.

   1. Each leaf in the DAG gets a credit of 1.

   2. Each node that is not a leaf gets a credit = 1 + the sum of the credits of the DAG edges from that node to its child nodes.

   3. credit for $(Y_i, Z)$ is:
   $$\text{credit}(Y_i, Z) = \text{credit}(Z) \frac{p_i}{\sum_{j=1}^k p_i}$$
   where $Y_1, Y_2, \dotsc, Y_k$ are the parents of $Z$, and $p_i$ is the number of the shorest paths from the root to $Y_i$, namely, the labels in Step 2).

```python
plt.imshow(plt.imread('./res/fig10_6.png'))
```

4) After performing the credit calculation with *each node* as the root,
   1. we sum the credits for each edge.

   2. And then divive the credit for each edge by 2, as each shortest path will have been discovered twice.

   3. Then we get the the true betweenness.

#### 10.2.5 Using Betweenness to Find Communities
It's a process of edge removal:

1. Start with the graph and all its edges;

2. then remove edges with the highest betweenness, until the graph has broken into a suitable number of connected components.

```python
plt.imshow(plt.imread('./res/fig10_7.png'))
plt.figure()
plt.imshow(plt.imread('./res/fig10_8.png'))
```

Speeding up the Betweenness Calculation:
If the large is large, we can pick a subset of the nodes at random and use these as the root of breadth-first searches, we can get an approximation to the betweenness of each edge that will serve in most applications.

**cons**:
It is not possible to place an individual in two different communities, and everyone is assigned to a community.

```python
# Exercises for Section 10.2
```

### 10.3 Direct Discovery of Communities
#### 10.3.1 Finding Cliques
NP-complete: finding a large *clique* (a sef of nodes with edges between any two of them).

#### 10.3.2 Complete Bipartite Graphs
A *complete bipartite graph* $K_{s,t}$ consists of $s$ nodes on one side and $t$ nodes on the other side, with all $st$ possible edges between the nodes of one side and the other present.

**Idea**:

1. While it is not possible to guarantee that a graph with many edges necessarily has a large clique, it is possible to guarantee that a bipartite graph with many edges has a large complete bipartite subgraph.

2. We can regard a complete bipartite subgraph as the nucleus of a community, and add to it nodes with many edges to existing members of the community.

   + If its nodes is consisted of two or more types, construct bipartite graphs directly.

   + If all nodes have the same type, divide the node into two equal groups at random.

#### 10.3.3 Finding Complete Bipartite Subgraphs
It's possible to view the problem of finding instance of $K_{s,t}$ within $G$ as one of finding frequent itemsets.

1. "items" - left side.

2. "baskets" - right side.
    The members of the basket for node $v$ are the nodes of the left side to which $v$ is connected.

3. Let the support threshold be $s$, the number of nodes that the instance of $K_{s,t}$ has on the right side.

the problem of find $K_{s,t}$ $\to$ finding frequent itemsets $F$ of size $t$.

#### 10.3.4 Why Complete Bipartite Graphs Must Exist
Assume:

1. the graph $G$ has $n$ nodes on the left and another $n$ nodes on the right.

2. let $d$ be the average degree of all nodes.

3. the degree of the $i$th node on the right is $d_i$.

Proof:

1. The total contribution of the $n$ nodes on the right is $\sum_i \binom{d_i}{t} \geq n \binom{d}{t}$ .

2. The number of itemsets of size $t$ is $\binom{n}{t}$.

3. Thus, the average count of an itemset of size $t$ is $n \binom{d}{t} / \binom{n}{t}$ ???

\begin{align}
    n \binom{d}{t} / \binom{n}{t} &= n \frac{d!}{(d-t)! t!} \frac{t! (n-t)!}{n!} \\
        &= n \frac{d (d-1) \dotso (d-t+1)}{n (n-1) \dotso (n-t+1)} \\
        &\approx n \frac{d^t}{n^t} \quad \text{when } n >> d >> t
\end{align}

That is, if there is a community with $n$ nodes on each side, the average degree of the nodes is $d$, and $n(d/n)^t \geq s$, then this community is guaranteed to have a complete bipartite subgraph $K_{s,t}$.

```python
# Exercises for Section 10.3
```

### 10.4 Partitioning of Graphs
tools from matrix theory $\to$ minimizing the "cut" size.

#### 10.4.1 What Makes a Good Partition?
1. divide the nodes into two sets so that the *cut* (sets of edges between two groups) is minimized.

2. two sets are approximately equal in size.

```python
plt.imshow(plt.imread('./res/fig10_11.png'))
```

#### 10.4.2 Normalized Cuts
A proper definition of a "good" cut must belance the size of the cut itself against the difference in the sizes of the sets that the cut creates.

The *normalized cut* value for $S$ and $T$ is
\begin{equation}
    \frac{\operatorname{Cut}(S,T)}{\operatorname{Vol}(S)} + \frac{\operatorname{Cut}(S,T)}{\operatorname{Vol}(T)}
\end{equation}
where $\operatorname{Vol}(S)$ is the number of edges with at least one end in $S$, and $\operatorname{Cut}(S,T)$ is the number of edges connected between $S$ and $T$.

#### 10.4.3 Some Matrices That Describe Graphs
1. adjacent matrix:
   \begin{equation}
       A_{i,j} = \begin{cases}
                     1, & \text{if node $i$ and $j$ is connected} \\
                     0, & \text{otherwise}
                 \end{cases}
   \end{equation}

2. degree matrix:
   $D_{i,i}$ is the degree of the $i$th node.

3. Laplacian matrix:
   $L = D - A$
   Notice that each row and column sums to zero.

#### 10.4.4 Eigenvalues of the Laplacian Matrix
The smallest eignevalues and their eigenvectors reveal the information we desire.

1. The smallest eignevalues for every Laplacian matrix is 0, and its corresponding eigenvectors is $\mathbf{1}$ ones matrix.

2. the second-smallest eigenvalues of $L$ is the minimum of $x^T L x$, and the minimum is taken under the constraints:
   + $\displaystyle \sum_{i=1}^n x_i^2 = 1$
   + $x$ is orthogonal to the eigenvector associated with the smallest eigenvalue.
     $$x^T \mathbf{1} = \displaystyle \sum_{i=1}^n x_i = 0$$

In all:
$x^T L x = \sum_{i,j} (x_i - x_j)^2$

As a consequence, $x$ must have some positive and some negative components $\to$ two sets/groups.

#### 10.4.5 Alternative Partitioning Methods
1. We could set the threshold at some point other than zero.

2. We may also want to a partition into more than two components:
   + split repeatedly as far as desired.
   + use several of the eigenvectors to partition the graph.

Attention:
while each eigenvector tries to produce a minimum-sized cut, the fact that successive eigenvectors have to satisfy more and more constraints generally causes the cuts they describe to be progressively worse.

```python
# Exercises for Section 10.4
## Ex 10.4.1
### (a)

edges = [
    ('A', 'B'), ('A', 'C'), ('B', 'C'),
    ('B', 'H'), ('C', 'D'), ('H', 'I'),
    ('H', 'G'), ('D', 'E'), ('D', 'F'),
    ('I', 'G'), ('G', 'E'), ('E', 'F')
]

G = nx.Graph()
G.add_edges_from(edges)
```

```python
nx.draw(G)
```

```python
A = nx.adjacency_matrix(G).todense()
A
```

    matrix([[0, 0, 0, 0, 0, 0, 0, 1, 1],
            [0, 0, 1, 0, 0, 1, 0, 0, 0],
            [0, 1, 0, 1, 0, 1, 0, 0, 0],
            [0, 0, 1, 0, 1, 0, 1, 0, 0],
            [0, 0, 0, 1, 0, 0, 1, 0, 0],
            [0, 1, 1, 0, 0, 0, 0, 0, 1],
            [0, 0, 0, 1, 1, 0, 0, 1, 0],
            [1, 0, 0, 0, 0, 0, 1, 0, 1],
            [1, 0, 0, 0, 0, 1, 0, 1, 0]], dtype=int64)

```python
D = np.diag(np.ravel(A.sum(axis=1)))
D
```

    array([[2, 0, 0, 0, 0, 0, 0, 0, 0],
           [0, 2, 0, 0, 0, 0, 0, 0, 0],
           [0, 0, 3, 0, 0, 0, 0, 0, 0],
           [0, 0, 0, 3, 0, 0, 0, 0, 0],
           [0, 0, 0, 0, 2, 0, 0, 0, 0],
           [0, 0, 0, 0, 0, 3, 0, 0, 0],
           [0, 0, 0, 0, 0, 0, 3, 0, 0],
           [0, 0, 0, 0, 0, 0, 0, 3, 0],
           [0, 0, 0, 0, 0, 0, 0, 0, 3]], dtype=int64)

```python
L = D - A
L
```

    matrix([[ 2,  0,  0,  0,  0,  0,  0, -1, -1],
            [ 0,  2, -1,  0,  0, -1,  0,  0,  0],
            [ 0, -1,  3, -1,  0, -1,  0,  0,  0],
            [ 0,  0, -1,  3, -1,  0, -1,  0,  0],
            [ 0,  0,  0, -1,  2,  0, -1,  0,  0],
            [ 0, -1, -1,  0,  0,  3,  0,  0, -1],
            [ 0,  0,  0, -1, -1,  0,  3, -1,  0],
            [-1,  0,  0,  0,  0,  0, -1,  3, -1],
            [-1,  0,  0,  0,  0, -1,  0, -1,  3]], dtype=int64)

```python
### Ex 10.4.2
w, v = np.linalg.eig(L)
```

```python
eig_min = np.argmin(w)
w = np.delete(w, eig_min)
v =np.delete(v, eig_min, axis=0)

eig_2nd_min = np.argmin(w)
eigv_2nd_min = np.ravel(v[eig_2nd_min])
```

```python
setA = set([n for v, n in zip(eigv_2nd_min, G.nodes()) if v >= 0])
setB = set(G.nodes()) - setA
print(setA, setB)
```

    {'B', 'F', 'D', 'C', 'A', 'E'} {'I', 'H', 'G'}

### 10.5 Finding Overlapping Communities
Communities are in practice rarely disjoint.

Assume:
the probability that two individuals are connected by an edge increases as they become members of more communities in common.

#### 10.5.1 The Nature of Communities
We expect edges to be dense with any community, but we expect edges to be even denser in the intersection of two communities, three communities, and so on.

```python
plt.imshow(plt.imread('./res/fig10_19.png'))
```

#### 10.5.2 Maximum-Likelihood Estimation
We assume that the value of the parameters that gives the largest value of the likelihood is the correct model for the observed artifact.
$$\operatorname{argmax}_{\theta} P[f(\theta)]) $$

prior probabilities:
$$\operatorname{argmax}_{\theta} P[f(\theta)]) = \operatorname{argmax}_{\theta} P[f(\theta) | \theta] \, P[\theta] $$

#### 10.5.3 The Affiliation-Graph Model
affiliation-graph model: generate social graphs from communities.

community-affiliation graphs:

+ Given: $C$ communities, $N$ nodes(individuals).

+ Question: $n_i \in C_k$ ?

+ Model:
  - Parameter: the memberships in the communities, $C_k = {n_i}$.
  - Parameter: $P_{ck}$ is the probability that two members of community $C_k$ are connected by an edge, $P_{ck} = P[(u,v) \in E \, | \, u \in C_k, v \in C_k]$.

we compute the likelihood that a given graph with the proper number of nodes is generated by this mechanism.

##### membership: Y/N
1. Parameter: membership
  define membership: $n_i \in C_k$.
  0 / 1, Yes / No, decrete variable $\to$ brute search

2. Parameter: $P_{ck}$
\begin{align}
    &P_{u,v} = 1 - \displaystyle \prod_{C_k in M} (1 - P_{ck})  \quad \text{where } M = {C_i: u \in C_i, v \in C_i}\\
    &P[f(P_{ck}, C_k, E)] = \displaystyle \prod_{(u,v) \in E} P_{u,v} \, \prod_{(u,v) \notin E} (1 - P_{u,v})
\end{align}

3. Goal:
   find $\operatorname{argmax}_{C_k} P[f(P_{ck}, C_k, E)]$.

##### membership: "strength"
avoiding the use of discrete membership changes. This improvement allows us to use standard methods.

"strenght of membership":
the stronger the membership of two individuals in the same community, the more likely it is that this community will cause them to have an edge between them.

In the improved model,

1. Parameter: membership
   define membership: strength, $F_{xC} \in \mathbb{R}_{\ge 0}$

2. Parameter: $P_{ck}$
   $$P_C(u,v) = 1 - e^{- F_{u,C} F_{v,C}}$$

```python
# exercise
```

### 10.6 Simrank
The purpose of simrank is to measure the similarity between nodes of the same type, and it does so by seeing where random walkers on the graph wind up when starting at a particular node $\to$ limited in the size of graphs.

#### 10.6.1 Random Walkers on a Social Graph
Random walkers:

A walker at a node $N$ of an undirected graph will move with equal probability to any of the *neighbors* of $N$.

#### 10.6.2 Random Walks with Restart
Let $M$ be the *transition matrix* of the graph $G$: the entry in row $i$ and column $j$ of $M$ is $1/k$ if node $j$ of $G$ has degree $k$, and one of the adjacent nodes is $i$.

$\beta$ is the probability that the walker continues at random, so $1 - \beta$ is the probability the walker will teleport to the initial node $N$.

$v'$ is the probability the walker is at each of the nodes at the next round:
$$v' = \beta M v + (1 - \beta) e_N$$

```python
plt.imshow(plt.imread('./res/fig10_22.png'))
```

```python
edges = [
    ('Picture 1', 'Sky'),
    ('Picture 1', 'Tree'),
    ('Picture 2', 'Sky'),
    ('Picture 3', 'Sky'),
    ('Picture 3', 'Tree')
]

G=nx.Graph()
G.add_edges_from(edges)
nx.draw(G)
```

```python
nodelist = ['Picture 1', 'Picture 2', 'Picture 3', 'Sky', 'Tree']
adj = nx.adjacency_matrix(G, nodelist=nodelist).todense()
```

```python
adj = pd.DataFrame(adj, index=nodelist, columns=nodelist)
adj
```

      Picture 1
      Picture 2
      Picture 3
      Sky
      Tree

      Picture 1
      0
      0
      0
      1
      1

      Picture 2
      0
      0
      0
      1
      0

      Picture 3
      0
      0
      0
      1
      1

      Sky
      1
      1
      1
      0
      0

      Tree
      1
      0
      1
      0
      0

```python
M = adj.apply(lambda x: x / sum(x), axis=0)
M
```

      Picture 1
      Picture 2
      Picture 3
      Sky
      Tree

      Picture 1
      0.0
      0
      0.0
      0.333333
      0.5

      Picture 2
      0.0
      0
      0.0
      0.333333
      0.0

      Picture 3
      0.0
      0
      0.0
      0.333333
      0.5

      Sky
      0.5
      1
      0.5
      0.000000
      0.0

      Tree
      0.5
      0
      0.5
      0.000000
      0.0

```python
beta = 0.8
e_0 = np.identity(M.shape[0])[0]
e_0 = pd.DataFrame({'e_0': e_0}, index=nodelist)
e_0
```

      e_0

      Picture 1
      1

      Picture 2
      0

      Picture 3
      0

      Sky
      0

      Tree
      0

```python
def random_walk(v: pd.DataFrame, beta: float, M: pd.DataFrame, e_n: pd.DataFrame) -> pd.DataFrame:
    return beta * (M.dot(v)) + (1 - beta) * e_n
```

```python
v = [e_0]
iter_time = 50

v_ = v[0]
for k in range(iter_time):
    v_ = random_walk(v_, beta, M, e_0)
    v.append(v_)

index = list(range(iter_time+1))
pd.concat(v, axis=1).T.reset_index(drop=True).T
```

      0
      1
      2
      3
      4
      5
      6
      7
      8
      9
      ...
      41
      42
      43
      44
      45
      46
      47
      48
      49
      50

      Picture 1
      1
      0.2
      0.466667
      0.253333
      0.418311
      0.286329
      0.391308
      0.307325
      0.374446
      0.320749
      ...
      0.344591
      0.344625
      0.344598
      0.344620
      0.344603
      0.344616
      0.344605
      0.344614
      0.344607
      0.344613

      Picture 2
      0
      0.0
      0.106667
      0.021333
      0.100978
      0.037262
      0.089448
      0.047699
      0.081228
      0.054405
      ...
      0.066326
      0.066343
      0.066329
      0.066340
      0.066331
      0.066338
      0.066333
      0.066337
      0.066333
      0.066336

      Picture 3
      0
      0.0
      0.266667
      0.053333
      0.218311
      0.086329
      0.191308
      0.107325
      0.174446
      0.120749
      ...
      0.144591
      0.144625
      0.144598
      0.144620
      0.144603
      0.144616
      0.144605
      0.144614
      0.144607
      0.144613

      Sky
      0
      0.4
      0.080000
      0.378667
      0.139733
      0.335431
      0.178873
      0.304605
      0.204019
      0.284540
      ...
      0.248785
      0.248734
      0.248774
      0.248742
      0.248768
      0.248747
      0.248764
      0.248750
      0.248761
      0.248752

      Tree
      0
      0.4
      0.080000
      0.293333
      0.122667
      0.254649
      0.149063
      0.233046
      0.165860
      0.219557
      ...
      0.195707
      0.195673
      0.195700
      0.195679
      0.195696
      0.195682
      0.195693
      0.195684
      0.195691
      0.195686

5 rows × 51 columns

1. If we wanted to know what node were most similar to another node, we would have to start the analsis over for that node.

2. notice that convergence takes time, since there is an initial oscillation.

```python
# Exercise
```

### 10.7 Counting Triangles
#### 10.7.1 Why Count Triangles?
1. to measure the extent to which a graph looks like a social network.

   If we start with $n$ nodes and add $m$ edges to a graph at random:
   + sets of three nodes: $\binom{n}{3}$.
   + The probability of an edge between any two given nodes being added is $m / \binom{n}{2} \approx 2 m / n^2$.
   + The probability that any set of three nodes has edges between each pair (independently chosen) is $(2m / n^2)^3 = 8 m^3 / n^6$.
   + expected number of triangles of random graph is $(8 m^3 / n^6) (n^3 / 6) = \frac{4}{3} (m / n)^3$.

   We expect the number of triangles to be much greater than the value for a random graph.

2. It has been demonstrated that the age of a community is related to the density of triangles.

#### 10.7.2 An Algorithm for Finding Triangles
Suppose we have a graph of $n$ nodes and $m \geq n$ edges.

nouns:
+ heavy hitter: the node whose degree is at least $\sqrt{m}$.
  note, the number of heavy-hitter nodes is nore more than $2\sqrt{m}$, since otherwise the sum of degree of nodes would be more than $2m$.

+ heavy-hitter triangle: the triangle all three of whose nodes are heavy hitter.

Assuming the graph is represented by its edges, we preprocess the graph as follows:

1. Compute the degree of each node. $O(m)$

2. Create an index on edges, with the pair of nodes at its ends as the key. constructed in $O(m)$. Query the existence of an edge $O(1)$.

3. Create another index of edges, this one with key equal to a single node. to retrieve the nodes adjacent to given node. $O(\sqrt{m})$

order the nodes: `nodes.sort_values(by=['degree', 'id'])`

##### Finding triangles:
1. Heavy-Hitter Triangles:
   Find in all heavy-hitter nodes which is only $O(\sqrt{m})$.
   time: $O(\sqrt{m}^3) = O(m^{3/2})$

2. Other Triangles:
   Consider each edge $(v_1, v_2)$:
   + if both $v_1$ and $v_2$ are heavy hitters, ignore the dege. (use $v_3$ since the computation is less).

   + if $v_1 < v_2$, query whether $(v_1.\text{adjacent nodes}), v_2)$ exits. $O(\sqrt{m} * m) = O(m^{3/2})$.

The total time of the algorithm is $O(m^{3/2})$.

#### 10.7.3 Optimality of the Triangle-Finding Algorithm
It turns out the algorithm described above is, to within an order of magnitude the best possible.

For a complete graph on $n$ nodes, it has $m = \binom{n}{2}$ edges and the number of triangles is $\binom{n}{3}$. Since we cannot enumerate triangles in less time than the number of those triangles $O(n^3) = O((\sqrt{m})^3) = O(m^{3/2})$.

For sparse graphs, we can add to the complete graph a chain of nodes with any length up to $n^2$ to convert.

#### 10.7.4 Finding Triangles Using MapReduce
multiway join technique:
$E(X, Y) \bowtie E(X, Z) \bowtie E(Y, Z)$

if we hash nodes to $b$ buckets, then there will be $b^3$ reducers, since $(h(u), h(v), z)$, $(h(u), y, h(v))$ and $(x, h(u), h(v)$ are mapped.
$\to$ The total communication required is thus $3b$ key-value pairs for each of the $m$ tuples of the edge relation $E$, namely $O(mb)$ if we use $b^3$ Reduce tasks.
$\to$ each Reduce task receives $O(mb) / b^3 = O(m / b^2)$ edges.
$\to$ If we use the algorithm of Section 10.7.2, the computation cost of each Reduce is $O((m / b^2)^{3/2})$. Thus, the total computation cost of $b^3$ Reduce is $O((m / b^2)^{3/2} * b^3) = O(m^{3/2})$.

#### 10.7.5 Using Fewer Reduce Tasks
By a judicious ordering of the nodes, we can lower the number of reduce tasks by approximately a factor of 6.

Order by "name", $(h(i), i)$. The Reduce task corresponding to list of bucket $(i, j, k)$ will be needed only if $i \leq j \leq k$.

#### 10.7.6 Exercises for Section 10.7

### 10.8 Neighborhood Properties of Graphs
#### 10.8.1 Directed Graphs and Neighborhoods
all undirected graphs can be represented by directed graphs.

**path**: a sequence of nodes in a directed graph. Its *length* is the number of arcs, instead of nodes, along the path.

The *neighborhood of radius* $d$ for $v$ is: $\{u : \operatorname{len}(u, v) \leq d\}$, denote this neighborhood by $N(v, d)$.

The *neighborhood profile* of a node $v$ is the sequence of sizes of its neighborhoods $|N(v, 1)|, |N(v, 2)|, \dotso$.

#### 10.8.2 The Diameter of a Graph
The *diameter* $d$ of a directed graph: $\max (\operatorname{len}(u, v)): u \in G, v \in G$.

for each node $v$, we can find the smallest $d$ such that $|N(v, d)| = |N(v, d+1)|$, and call it $d(v)$. $\to$ the $d$ of $G$ is: $\max_{v} d(v)$.

A graph is *strongly connected* if there is a paht from any node to any other node. If $G$ is strongly connected, the $d = \max_{v} d(v)$.

"six degrees of separation": the diameter of social graph is six. Unfortunately, not all important graphs exhibit such tight connections.

#### 10.8.3 Transitive Closure and Reachability
The *transitive closure* of a graph is: $\{(u, v) : \operatorname{len of Path}(u, v) \geq 0 \}$. denoted $\operatorname{Path}(u, v)$.

*reachability*: we say node $u$ reaches node $v$ if $\operatorname{Path}(u, v)$.

$\operatorname{Path}(u, v)$ is true if and only if $v$ is in $N(u, \infty) = \cup_{i \geq 0} N(u, i)$.

The two problems - transtive closure and reachability - are related, but there are many examples of graphs where reachability is feasible and transitive closure is not.

#### 10.8.4 Transitive Closure Via MapReduce
transitive closure is actually more readily parallelizable than is reachability.

##### calculate reachability
$\operatorname{Arc}(X, Y) = \{(x, y) \text{ where } x \to y\}$.

```
SELECT DISTINCT Arc.Y
FROM Reach, Arc
WHERE Arc.X = Reach.X
```

how many rounds this process requires depends on how far from $v$ is the furthest node that $v$ can reach.

##### calculate transitive closure
recursive-doubling method

```
SELECT DISTINCT p1.X, p2.Y
FROM Path p1, Path p2
WHERE p1.Y = p2.X
```

#### 10.8.5 Smart Transitive Closure
The above recursive-doubling method does a lot of redundant work, since there may exist many paths between two nodes.

_smart_ transitive closure:
Every path of length greater than 1 can be broken into a _head_ whose length is a power of 2, followed by a _tail_ whose length is no greater than the length of the head.

$Q(X, Y)$ holds all pairs of nodes $(x, y)$ such that the shortest path from $x$ to $y$ is of length exactly $2^i$ after the $i$th round.

Intitally, set both $Q$ and _Path_ to be copies of the relation _Arc_.

On the $(i + 1)$st round, we do the following:

1. Compute a new value for $Q$ by joining it with itself:
   ```
   SELECT DISTINCT q1.X, q2.Y
   FROM Q q1, Q q2
   WHERE q1.Y = q2.X
   ```

2. Subtract _Path_ from the relation _Q_ computed in step 1.

3. Join _Path_ with the nw value of _Q_ computed in 2:
   ```
   SELECT DISTINCT Q.X, Path.Y
   FROM Q, Path
   WHERE Q.Y = Path.X
   ```

4. Set the new value of _Path_ to be the union of the relation computed in step 3, the new value of _Q_ computed in step 1, and the old value of _Path_.

#### 10.8.6 Transitive Closure by Graph Reduction
collapse an SCC (Stronly connected components) to a single node when computing the transitive closure.

to find most of the SCC's in a graph by some random node selections followed by two breadth-first searches.

Let $G$ be the graph to be reduced, and let $G'$ be $G$ with all the arcs reversed.

1. Pick a node $v$ from $G$ at random.

2. Find $N_G(v, \infty)$, the set of nodes reachable from $v$ in $G$.

3. Find $N_{G'} (v, \infty)$, the set of nodes that $v$ reaches in the graph $G'$ that has the arcs of $G$ reversed.

4. Construct the SCC S containing $v$, which is $N_G(v, \infty) \cap N_{G'}(v, \infty)$.

5. Replace SCC S by a single node $s$ in $G$.

#### 10.8.7 Approximating the Sizes of Neighborhoods
approximation:

1. apply hash function $h$ to nodes $\{v\}$, find the longest "tail length" $R$.

2. estimate the size of the set is $2^R$.

```python
# Exercise 10.8.8
```


```
from IPython.core.display import HTML
def css_styling():
    styles = open("./styles/custom.css", "r").read()
    return HTML(styles)
css_styling()
```

<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Arvo:400,700,400italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=PT+Mono' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Philosopher:400,700,400italic,700italic' rel='stylesheet' type='text/css'>

<style>

#notebook_panel { /* main background */
    background: #888;
    color: #f6f6f6;
}

div #notebook { /* centre the content */
    background: #fff;
    color: #333;
    width: 1000px;
    margin: auto;
    padding-left: 1em;
    padding-right: 1em;
    padding-top: 2ex;
    overflow-x: auto;
}

#notebook li { /* More space between bullet points */
margin-top:0.8em;
}

div.cell { /* set cell width to about 80 chars */
    width: 800px;
}

div.cell.border-box-sizing.code_cell.running {
    /* draw border around running cells */
    border: 3px solid #111;
}

h1 {
    font-family: 'Philosopher', sans-serif;
}
h2 {
    font-family: 'Philosopher', serif;
}
h3{
    font-family: 'Philosopher', serif;
    font-style: 'italic';
    margin-top:12px;
    margin-bottom: 3px;
}
h4{
    font-family: 'Philosopher', serif;
}
h5 {
    font-family: 'Alegreya Sans', sans-serif;
}
h6 {
    font-family: 'PT Mono', sans-serif;
}

div.text_cell_render{
    font-family: 'Arvo' sans-serif;
    line-height: 130%;
    font-size: 115%;
    width:700px;
    margin-left:auto;
    margin-right:auto;
}

.CodeMirror{
        font-family: "PT Mono";
                    font-size: 100%;
            }

.text_cell_render h1 {
    font-weight: 400;
    font-size: 64pt;
    line-height: 100%;
    color: rgb(12,85,97);
    margin-bottom: 0.1em;
    margin-top: 0.1em;
    display: block;
}
.text_cell_render h2 {
    font-weight: 700;
    font-size: 24pt;
    line-height: 100%;
    color: rgb(171,165,131);
    margin-bottom: 0.1em;
    margin-top: 0.1em;
    display: block;
}

.text_cell_render h3 {
    font-style: italic;
    color: rgb(95,92,72);
}

.text_cell_render h5 {
    font-weight: 300;
    font-size: 16pt;
    color: grey;
    font-style: italic;
    margin-bottom: .1em;
    margin-top: 0.1em;
    display: block;
}

.text_cell_render h6 {
    font-weight: 300;
    font-size: 10pt;
    color: grey;
    margin-bottom: 1px;
    margin-top: 1px;
}

div.cell.code_cell {  /* Group area containing both code and output */
background-color: rgba(171,165,131,0.3);
border-radius: 10px; /* rounded borders */
padding: 1em;
}

</style>

###### Content provided under a Creative Commons Attribution license, CC-BY 4.0; code under MIT License. (c)2014 [David I. Ketcheson](http://davidketcheson.info)

##### version 0.1 - April 2014

# Fluid dynamics

In this lesson we will look at the system of hyperbolic PDEs that governs the motions of fluids in the absence of viscosity.  These consist of conservation laws for **mass, momentum**, and **energy**.  Together, they are referred to as the **compressible Euler equations**, or simply the Euler equations.

## Mass conservation

We will use $\rho(x,t)$ to denote the fluid density and $u(x,t)$ for its velocity.  Then the equation for conservation of mass is just the **continuity equation** we discussed in [Lesson 1](Lesson_01_Advection.ipynb):

$$\rho_t + (\rho u)_x = 0.$$

## Momentum conservation

The momentum is given by the product of density and velocity, i.e. $\rho u$.  The momentum flux has two components.  First, the momentum is transported in the same way that the density is; this flux is given by the momentum times the density; i.e. $\rho u^2$.

To understand the second term in the momentum flux, we must realize that a fluid is made up of many tiny molecules.  The density and velocity we are modeling are average values over some small region of space.  The individual molecules in that region are not all moving with exactly velocity $u$; that's just their average.  Each molecule also has some additional random velocity component.  These random velocities are what accounts for the **pressure** of the fluid, which we'll denote by $p$.  These velocity components also lead to a net flux of momentum.  Thus the momentum conservation equation is

$$(\rho u)_t + (\rho u^2 + p)_x = 0.$$

## Energy conservation

The energy has two components: internal energy $\rho e$ and kinetic energy $\rho u^2/2$:

$$E = \rho e + \frac{1}{2}\rho u^2.$$

Like the momentum flux, the energy flux involves both bulk transport ($Eu$) and transport due to pressure ($pu$):

$$E_t + (u(E+p)) = 0.$$

## Equation of state

You may have noticed that we have 4 unknowns (density, momentum, energy, and pressure) but only 3 conservation laws.  We need one more relation to close the system.  That relation, known as the equation of state, expresses how the pressure is related to the other quantities.  We'll focus on the case of an ideal gas, for which

$$p = \rho e (\gamma-1).$$

Here $\gamma$ is the ratio of specific heats, which for air is approximately 1.4.

## The Euler equations

We can write the three conservation laws as a single system $q_t + f(q)_x = 0$ by defining
\begin{align}
q & = \begin{pmatrix} \rho \\ \rho u \\ E\end{pmatrix}, &
f(q) & = \begin{pmatrix} \rho u \\ \rho u^2 + p \\ u(E+p)\end{pmatrix}.
\end{align}

In three dimensions, the equations are similar.  We have two additional velocity components $v, w$, and their corresponding fluxes.  Additionally, we have to account for fluxes in the $y$ and $z$ directions.  We can write the full system as

$$ q_t + f(q)_x + g(q)_y + h(q)_z = 0$$

with

\begin{align}
q & = \begin{pmatrix} \rho \\ \rho u \\ \rho v \\ \rho w \\ E\end{pmatrix}, &
f(q) & = \begin{pmatrix} \rho u \\ \rho u^2 + p \\ \rho u v \\ \rho u w \\ u(E+p)\end{pmatrix} &
g(q) & = \begin{pmatrix} \rho v \\ \rho uv \\ \rho v^2 + p \\ \rho v w \\ v(E+p)\end{pmatrix} &
h(q) & = \begin{pmatrix} \rho w \\ \rho uw \\ \rho vw \\ \rho w^2 + p \\ w(E+p)\end{pmatrix}.
\end{align}

## Solving the Euler equations

These equations can be solved in a manner similar to what we used for advection and traffic flow.  As you might guess, computing the flux gets significantly more complicated since we now have 3 (or 5) equations and more complicated flux expressions.

## PyClaw

Implementing a solver for the Euler equations from scratch would be a lot of fun, but to save some time we'll use a  package called [PyClaw](http://clawpack.github.io/doc/pyclaw/), which is part of the [Clawpack](http://clawpack.github.io/) software (Clawpack stands for Conservation LAWs PACKage).  PyClaw allows us to quickly and easily set up and solve problems modeled by hyperbolic PDEs.

Now let's get started.  First, import the parts of Clawpack that we'll use:

```
from clawpack import pyclaw
from clawpack import riemann
```

### Setting up a problem
To solve a problem, we'll need to create the following:

- A **domain** over which to solve the problem
- A **solution**, where we will provide the initial data.  After running, the solution will contain -- you guessed it! -- the solution.
- A **solver**, which is responsible for actually evolving the solution in time.  Here we'll need to specify the equations to be solved and the boundary conditions.
- A **controller**, which handles the running, output, and can be used for plotting

This might sound complicated at first, but stick with me.

Let's start by creating a controller and specifying the simulation end time:

```
claw = pyclaw.Controller()
claw.tfinal = 0.1                    # Simulation end time

claw.keep_copy = True          # Keep solution data in memory for plotting
claw.output_format = None    # Don't write solution data to file
claw.num_output_times = 50  # Write 50 output frames
```

### Riemann solvers

The method used to compute the flux between each pair of cells is referred to as a *Riemann solver*.  By specifying a Riemann solver, we will specify the system of PDEs that we want to solve.  So far we have only used very simple approximate Riemann solvers.  Clawpack includes much more sophisticated Riemann solvers for many hyperbolic systems.

Place your cursor at the end of the line in the box below and hit 'Tab' (for autocompletion).  You'll see a dropdown list of all the Riemann solvers currently available in PyClaw.  The ones with 'py' at the end of the name are written in pure Python; the others are written in Fortran and wrapped with f2py.

```
riemann.
```

We'll start with a simple 1D problem, using the Riemann solver `riemann.euler_with_efix_1D`:

```
riemann_solver = riemann.euler_with_efix_1D
claw.solver = pyclaw.ClawSolver1D(riemann_solver)
```

We also need to specify boundary conditions.  We'll use extrapolation boundary conditions, so that waves simply pass out of the domain:

```
claw.solver.all_bcs = pyclaw.BC.extrap
```

### The problem domain
Next we need to specify the domain and the grid.  We'll solve on the unit line $[0,1]$ using 100 grid cells.  Note that each argument to the Domain constructor must be a tuple:

```
domain = pyclaw.Domain( (0.,), (1.,), (100,))
```

### The initial solution
Next we create a solution object that belongs to the controller and extends over the domain we specified:

```
claw.solution = pyclaw.Solution(claw.solver.num_eqn,domain)
```

The initial data is specified in an array named `solution.q`.  The density is contained in `q[0,:]`, the momentum in `q[1,:]`, and the energy in `q[2,:]`.

```
x=domain.grid.x.centers # grid cell centers
gam = 1.4 # ratio of specific heats

rho_left = 1.0; rho_right = 0.125
p_left = 1.0; p_right = 0.1

claw.solution.q[0,:] = (x<0.5)*rho_left + (x>=0.5)*rho_right
claw.solution.q[1,:] = 0.
claw.solution.q[2,:] = ((x<0.5)*p_left + (x>=0.5)*p_right)/(gam-1.0)

%matplotlib inline
import matplotlib.pyplot as plt
plt.plot(x, claw.solution.q[0,:],'-o')
```

This problem is known as the **Sod shock-tube**.  It amounts to setting up a tube with a thin separator between a high-pressure, high-density region and a low-pressure, low-density region, then suddenly removing the separator.

Next we need to specify the value of $\gamma$, the ratio of specific heats.

```
problem_data = claw.solution.problem_data
problem_data['gamma'] = 1.4
problem_data['gamma1'] = 0.4
```

Finally, let's run the simulation.

```
claw.run()
```

### Plotting
Now we'll plot the results, which are contained in a list called `claw.frames`.  It's simple to plot a single frame with matplotlib:

```
from matplotlib import animation
import matplotlib.pyplot as plt
from clawpack.visclaw.JSAnimation import IPython_display
import numpy as np

fig = plt.figure()
ax = plt.axes(xlim=(0, 1), ylim=(-0.2, 1.2))

frame = claw.frames[0]
pressure = frame.q[0,:]
line, = ax.plot([], [], 'o-', lw=2)

def fplot(frame_number):
    frame = claw.frames[frame_number]
    pressure = frame.q[0,:]
    line.set_data(x,pressure)
    return line,

animation.FuncAnimation(fig, fplot, frames=len(claw.frames), interval=30)
```

### Waves

In the solution, 3 waves are visible:
1. A **shock wave** moving rapidly to the right as the low-density fluid is compressed.
2. A **rarefaction** wave moving to the left as the high-density fluid expands.
3. A **contact discontinuity** moving more slowly to the right.  This discontinuity in the density separates the region containing fluid that started in the high-pressure region and fluid that started in the low-pressure region.

In fact, the solution of any Riemann problem consists of some combination of these three types of waves.  In the Euler equations, one of the waves is always a contact discontinuity, but each of the other two waves may be a shock or a rarefaction, depending on the left and right states.

### Putting it all together

For convenience, all of the code from the cells above to set up and run the shocktube problem is pasted together below.  Play around with the code.  You might:
- Increase the number of grid points to see what the solution converges to.  Notice that the code still runs pretty fast even for larger grids.  This is because the bottom layer of code in PyClaw is compiled Fortran, not Python.
- Change the initial left and right states, or set up a completely different initial condition.  See if you can generate a solution with two shock waves, or two rarefaction waves (some physical intuition is helpful here).
- Change the ratio of specific heats
- Make the boundaries periodic, so that there is a second shock wave moving left from $x=1$.

```
%matplotlib inline
import matplotlib.pyplot as plt
from clawpack import pyclaw
from clawpack import riemann

claw = pyclaw.Controller()
claw.tfinal = 0.1

claw.keep_copy = True       # Keep solution data in memory for plotting
claw.output_format = None   # Don't write solution data to file
claw.num_output_times = 50  # Write 50 output frames

riemann_solver = riemann.euler_with_efix_1D
claw.solver = pyclaw.ClawSolver1D(riemann_solver)

claw.solver.all_bcs = pyclaw.BC.extrap

domain = pyclaw.Domain( (0.,), (1.,), (100,))
x=domain.grid.x.centers # grid cell centers

claw.solution = pyclaw.Solution(claw.solver.num_eqn,domain)

gam = 1.4 # ratio of specific heats
claw.solution.problem_data['gamma'] = gam
claw.solution.problem_data['gamma1'] = gam-1.0

rho_left = 1.0; rho_right = 0.125
p_left = 1.0; p_right = 0.1

claw.solution.q[0,:] = (x<0.5)*rho_left + (x>=0.5)*rho_right
claw.solution.q[1,:] = 0.
claw.solution.q[2,:] = ((x<0.5)*p_left + (x>=0.5)*p_right)/(gam-1.0)

status = claw.run()
```


# Getting Started With Python

## Installation

There are various ways to install Python. Assuming the reader is not (yet) well versed in programming, I suggest to download the Anaconda distribution, which works for all major operating systems (Mac OS X, Windows, Linux) and provides a fully fledged Python installation including necessary libraries and Jupyter notebooks.

* Go to https://www.continuum.io/downloads
* Download latest version corresponding to your OS
* Run .exe/.pkg file.
  - Make sure to set flag "Add Anaconda to my PATH environment variable" and
  - "Register Anaconda as my default Python 3.6"

Be sure to download the latest Python 3.x version (not 2.x; backward compability is not always given!).

The installation should not cause any problems. If you wisch a step-by-step guide (incl. some further insights) see [Cyrille Rossants excellent notebook](http://nbviewer.jupyter.org/github/ipython-books/minibook-2nd-code/blob/master/chapter1/12-installation.ipynb) on the topic.

## IPyton / Jupyter Notebooks

What you see here is a so called Jupyter notebook. It makes it possible to interactively combine code with output (results, graphics), markdown text and LaTeX (for mathematical expressions). All codes discussed in this course will be provided through such notebooks and you soon will understand and appreciate the functionality they provide.

* The basic markdown commands are well summarized [here](http://jupyter-notebook.readthedocs.io/en/latest/examples/Notebook/Working%20With%20Markdown%20Cells.html).
* LaTeX is a typesetting language with extensive capabilities to typeset math. For a basic introductin to math in LaTeX see sections 3.3 - 3.4 (p. 22 - 33) of *More Math into LaTeX* by Grätzer (2007), [available as pdf here](http://www.latexstudio.net/wp-content/uploads/2014/09/math.into_.Latex_.4ed.pdf)

If you are keen on learning more about IPython/Jupyter, consider [this notebook](http://nbviewer.jupyter.org/github/ipython-books/minibook-2nd-code/blob/master/chapter1/13-nbui.ipynb) - a well written introduction by Cyrille Rossant.

## Data Types in Python

### Building Blocks

To analyze data in Python, it has to be stored as some kind of data type. These data types form the structure of the data and make data easily accessible. Python's basic building blocks are:
* Numbers (integer, floating point, and complex)
* Booleans (true/false)
* Strings
* Lists
* Dictionaries
* Tuples

We will discuss the first four data types above as these are relevant for us.

Python is a dynamic typing language meaning that - unlike in static languages such as VBA, C, Java etc. - you do not explicitly need to assign a data type to a variable. Python will do that for you. A few examples will explain this best:

```python
a = 42       # In VBA you would first have to define data type, only then the value: Dim a as integer; a = 42
b = 10.3     # VBA: Dim b as Double; b = 10.3
c = 'hello'  # VBA: Dim c as String; c = "hello"
d = True     # VBA: Dim d as Boolean; d = True

print('a: ', type(a))
print('b: ', type(b))
print('c: ', type(c))
print('d: ', type(d))
```

    a:  <class 'int'>
    b:  <class 'float'>
    c:  <class 'str'>
    d:  <class 'bool'>

### Simple Arithmetics
Simple arithmetic operations are straight forward:

```python
a = 2 + 4 - 8      # Addition & Subtraction
b = 6 * 7 / 3 - 2  # Multiplication & Division
c = 2**(1/2)       # Exponents & Square root
d = 10 % 3         # Modulus
e = 10 // 3        # Floor division

print(' a =', a, '\n',
      'b =', b, '\n',
      'c =', c, '\n',
      'd =', d, '\n',
      'e =', e)
```

     a = -2
     b = 12.0
     c = 1.4142135623730951
     d = 1
     e = 3

We can even use arithmetic operators to concatenate strings:

```python
a = 'Hello'
b = 'World!'
print(a + ' ' + b)

print(a * 3)
```

    Hello World!
    HelloHelloHello

### Lists
Now let's look at lists. Lists are capable of combining multiple data types.

```python
e = ['Calynn', '10.3', b, d]
print('e: ', type(e))
print([type(item) for item in e])
```

    e:  <class 'list'>
    [<class 'str'>, <class 'str'>, <class 'str'>, <class 'int'>]

Note that the second element in list `e` is set in quotation marks and thus Python interprets this as string.

## NumPy Arrays

### NumPy Arrays from Lists

For as useful list appear, its flexibility comes at a high cost. Because each element contains not only the value itself but also information about the data type, storing data in list consumes a lot of memory. For this reason the Python community introduced NumPy (short for Numerical Python). Among other things, this package provides **fixed-type arrays** which are more efficient to store and operate on dense data than simple lists.

Fixed-type arrays are dense arrays of uniform type. We start by importing the NumPy package and creating some simple NumPy arrays form Python lists.

```python
import numpy as np

# Integer array
np.array([3, 18, 12])
```

    array([ 3, 18, 12])

```python
# Floating point array
np.array([3., 18, 12])
```

    array([  3.,  18.,  12.])

Similarly, we can explicitly set the data type:

```python
np.array([3, 18, 12], dtype='float32')
```

    array([  3.,  18.,  12.], dtype=float32)

```python
# Multidimensional arrays
np.array([range(i, i + 3) for i in [1, 2, 3]])
```

    array([[1, 2, 3],
           [2, 3, 4],
           [3, 4, 5]])

We've seen above that we can define the data type for NumPy arrays. A list of available data types can be found in the [NumPy documentation](https://docs.scipy.org/doc/numpy/user/basics.types.html).

### NumPy Arrays from Scratch
Sometimes it is helpful to create arrays from scratch. Here are some examples:

```python
# Integer array with 8 zeros
np.zeros(shape=8, dtype='int')
```

    array([0, 0, 0, 0, 0, 0, 0, 0])

```python
# 2x3 floating-point array filled with 1s
np.ones((2, 3), 'float32')
```

    array([[ 1.,  1.,  1.],
           [ 1.,  1.,  1.]], dtype=float32)

Note that I do not need to use `np.ones(shape=(2, 3), dtype='float32')` to define the shape. It's ok to go with the short version as long as the order is correct, Python will understand. However, going with the explicit version helps to make your code more readable and I encourage students to follow this advice.

```python
# 3x2 array filled with 2.71
np.full(shape=(3, 2), fill_value=2.71)
```

    array([[ 2.71,  2.71],
           [ 2.71,  2.71],
           [ 2.71,  2.71]])

```python
# 3x3 boolean array filled with 'True'
np.full((2, 2), 1, bool)
```

    array([[ True,  True],
           [ True,  True]], dtype=bool)

```python
# Array filled with linear sequence
np.arange(start = 0, stop = 1, step = 0.1)  # or simply np.arrange(0, 1, 0.1)
```

    array([ 0. ,  0.1,  0.2,  0.3,  0.4,  0.5,  0.6,  0.7,  0.8,  0.9])

```python
# Array of evenly spaced values
np.linspace(start = 0, stop = 1, num = 4)
```

    array([ 0.        ,  0.33333333,  0.66666667,  1.        ])

Arrays with random variables are easily created. Below three examples. See the [numpy.random documentation page](https://docs.scipy.org/doc/numpy/reference/routines.random.html) for details on how to generate other rv arrays.

```python
# 4x4 array of uniformly distributed random variables
np.random.random((4, 4))
```

    array([[ 0.54009291,  0.63963002,  0.63434996,  0.23271539],
           [ 0.93474271,  0.24525379,  0.04107161,  0.01701793],
           [ 0.17410066,  0.42886507,  0.14338677,  0.142046  ],
           [ 0.54005321,  0.03878921,  0.8827711 ,  0.55105743]])

```python
# 3x3 array of normally distributed rv (with mean = 4, sd = 6)
np.random.normal(loc = 4, scale = 6, size = (3, 3))
```

    array([[ -8.06536073,   7.31049404,   4.55116972],
           [ 16.79746137,   9.28275757,  11.03506686],
           [ 11.10279939,   4.99549227,   9.11912977]])

```python
# 3x3 array of random integers in interval [0, 15)
np.random.randint(low = 0, high = 15, size = (3, 3))
```

    array([[ 0,  3,  7],
           [10,  7,  8],
           [10,  7, 10]])

```python
# 4x4 identity matrix
np.eye(4)
```

    array([[ 1.,  0.,  0.,  0.],
           [ 0.,  1.,  0.,  0.],
           [ 0.,  0.,  1.,  0.],
           [ 0.,  0.,  0.,  1.]])

### NumPy Array Attributes
Each NumPy array has certain attributes.

Here are some attributes we can call:

| Attribute | Description            |
|-----------|------------------------|
| `ndim`    | No. of dimensions      |
| `shape`   | Size of each dimension |
| `size`    | Total size of array    |
| `dtype`   | Data type of array     |
| `itemsize` | Size (in bytes)       |
| `nbytes`   | Total size (in bytes) |

To show how one can access them we'll define three arrays.

```python
np.random.seed(1234)  # Set seed for reproducibility

x = np.random.randint(10, size = 6)          # 1-dimensional array (vector)
y = np.random.randint(10, size = (3, 4))     # 2-dimensional array (matrix)
z = np.random.randint(10, size = (3, 4, 5))  # 3-dimensional array
```

And here's how we call for these properties:

```python
print('ndim:      ', z.ndim)
print('shape:     ', z.shape)
print('size:      ', z.size)
print('data type: ', z.dtype)
print('itemsize:  ', z.itemsize)
print('nbytes:    ', z.nbytes)
```

    ndim:       3
    shape:      (3, 4, 5)
    size:       60
    data type:  int32
    itemsize:   4
    nbytes:     240

### Index: How to Access Elements
What might be a bit counterintuitive at the beginning is that **Python's indexing starts at 0**. Other than that, accessing the $i$'th element (starting at 0) of a list or a array is straight forward.

```python
print(e, '\n')  # List from above
print(x, '\n')  # One dimensional np array from above
print(y, '\n')  # Two dimensional np array from above
```

    ['Calynn', '10.3', 'World!', 1]

    [3 6 5 4 8 9]

    [[1 7 9 6]
     [8 0 5 0]
     [9 6 2 0]]

```python
e[1]
```

    '10.3'

```python
x[5]
```

    9

```python
y[2, 0]  # Note again that [m, n] starts counting for both rows (m) as well as columns (n) from 0
```

    9

To access the end of an array, you can also use negative indices:

```python
e[-1]
```

    1

```python
y[-2, 2]
```

    5

Arrays are also possible as inputs:

```python
ind = [3, 5, -4]
x[ind]
```

    array([4, 9, 5])

```python
x = np.arange(12).reshape((3, 4))
print(x)
row = np.array([1, 2])
col = np.array([0, 3])
x[row, col]
```

    [[ 0  1  2  3]
     [ 4  5  6  7]
     [ 8  9 10 11]]

    array([ 4, 11])

Knowing the index we can also replace elements of an array:

```python
x[0] = 99
x
```

    array([[99, 99, 99, 99],
           [ 4,  5,  6,  7],
           [ 8,  9, 10, 11]])

**IMPORTANT NOTE: **

**NumPy arrays have a fixed type. This means that e.g. if you insert a floating-point value to an integer array, the value will be truncated!**

```python
x[0] = 3.14159; x
```

    array([[ 3,  3,  3,  3],
           [ 4,  5,  6,  7],
           [ 8,  9, 10, 11]])

### Array Slicing
We can also use square brackets to access a subset of the data. The syntax is:

`x[start:stop:step]`

The default values are: `start=0`, `stop='size of dimension'`, `step=1`

```python
x = np.arange(10)
x
```

    array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])

```python
x[:3]  # First three elements
```

    array([0, 1, 2])

```python
x[7:]  # Elements AFTER index 7
```

    array([7, 8, 9])

```python
x[4:8]  # Element 5, 6, 7 and 8
```

    array([4, 5, 6, 7])

```python
x[::2]  # Even elements
```

    array([0, 2, 4, 6, 8])

```python
x[1::2]  # Odd elements
```

    array([1, 3, 5, 7, 9])

```python
x[::-1]  # All elements reversed
```

    array([9, 8, 7, 6, 5, 4, 3, 2, 1, 0])

```python
x[::-2]  # Odd elements reversed
```

    array([9, 7, 5, 3, 1])

Array slicing works the same for multidimensional arrays.

```python
y  # from above
```

    array([[1, 7, 9, 6],
           [8, 0, 5, 0],
           [9, 6, 2, 0]])

```python
y[:2, :3]  # Rows 0 and 1, columns 0, 1, 2
```

    array([[1, 7, 9],
           [8, 0, 5]])

```python
y[:, 2]  # Third column
```

    array([9, 5, 2])

```python
y[0, :]  # First row
```

    array([1, 7, 9, 6])

**IMPORTANT NOTE:**

**When slicing and assigning part of an existing array to a new variable, the new variable will only hold a "view" but not a copy. This means, that if you change a value in the new array, the original array will also be changed. The idea behind this is to save memory. But fear not: with the ".copy()" method, you still can get a true copy.**

Here a few corresponding examples for better understanding:

```python
ySub = y[:2, :2]
print(ySub)
```

    [[1 7]
     [8 0]]

```python
ySub[0, 0] = 99
print(ySub, '\n')
print(y)
```

    [[99  7]
     [ 8  0]]

    [[99  7  9  6]
     [ 8  0  5  0]
     [ 9  6  2  0]]

```python
ySubCopy = y[:2, :2].copy()
ySubCopy[0, 0] = 33
print(ySubCopy, '\n')
print(y)
```

    [[33  7]
     [ 8  0]]

    [[99  7  9  6]
     [ 8  0  5  0]
     [ 9  6  2  0]]

### Concatenating, Stacking and Splitting
Often it is useful to combine multiple arrays into one or to split a single array into multiple arrays. To accomplish this, we can use NumPy's `concatenate` and `vstack`/`hstack` function.

```python
x = np.array([1, 2, 3])
y = np.array([11, 12, 13])
z = np.array([21, 22, 23])
np.concatenate([x, y, z])
```

    array([ 1,  2,  3, 11, 12, 13, 21, 22, 23])

```python
# Stack two vectors horizontally
np.hstack([x, y])
```

    array([ 1,  2,  3, 11, 12, 13])

```python
# Stack two vectors vertically
np.vstack([x, y])
```

    array([[ 1,  2,  3],
           [11, 12, 13]])

```python
# Stack matrix with column vector
m = np.arange(0, 9, 1).reshape((3, 3))
np.vstack([m, z])
```

    array([[ 0,  1,  2],
           [ 3,  4,  5],
           [ 6,  7,  8],
           [21, 22, 23]])

```python
# Stack matrix with row vector
np.hstack([m, z.reshape(3, 1)])
```

    array([[ 0,  1,  2, 21],
           [ 3,  4,  5, 22],
           [ 6,  7,  8, 23]])

The opposite of concatenating is splitting. Numpy has `np.split`, `np.hsplit` and `np.vsplit` functions. Each of these takes a list of indices, giving the split points, as input.

```python
x = np.arange(8.0)
a, b, c = np.split(x, [3, 5])
print(a, b, c)
```

    [ 0.  1.  2.] [ 3.  4.] [ 5.  6.  7.]

```python
x = np.arange(16).reshape(4, 4)
upper, lower = np.vsplit(x, [3])
print(upper, '\n\n', lower)
```

    [[ 0  1  2  3]
     [ 4  5  6  7]
     [ 8  9 10 11]]

     [[12 13 14 15]]

```python
left, right = np.hsplit(x, [2])
print(left, '\n\n', right)
```

    [[ 0  1]
     [ 4  5]
     [ 8  9]
     [12 13]]

     [[ 2  3]
     [ 6  7]
     [10 11]
     [14 15]]

## Conditions

### Boolean Operators

Boolean operators check an input and return either `True` (equals 1 as value) or `False` (equals 0). This is often very helpful if one wants to check for conditions or sort out part of a data set which meet a certain condition. Here are the common comparison operators:

| **Operator** | **Description**            |
|:------------:|----------------------------|
| ==           | equal ($=$)                |
| !=           | not equal ($\neq$)         |
| <            | less than ($<$)            |
| <=           | less or equal ($\leq$)     |
| >            | greater ($>$)              |
| >=           | greater or equal ($\geq$)  |
| &            | Mathematical AND ($\land$) |
| |       | Mathematical OR ($\lor$)   |
| `in`         | element of ($\in$)         |

The following sections give a glimpse of how these operators can be used.

```python
x = np.arange(start=0, stop=8, step=1)
print(x)
print(x == 2)
print(x != 3)
print((x < 2) | (x > 6))
```

    [0 1 2 3 4 5 6 7]
    [False False  True False False False False False]
    [ True  True  True False  True  True  True  True]
    [ True  True False False False False False  True]

```python
# Notice the difference
print(x[x <= 4])
print(x <= 4)
```

    [0 1 2 3 4]
    [ True  True  True  True  True False False False]

#### If ... else statements

These statements check a given condition and depending on the result (`True`, `False`) execute a subsequent code. As usual, an example will do. Notice that indentation is necessary for Python to correctly compile the code.

```python
x = 3

if x%2 == 0:
    print(x, 'is an even number')
else:
    print(x, 'is an odd number')

```

    3 is an odd number

It is also possible to have more than one condition as the next example shows.

```python
x = 20

if x > 0:
    print(x, 'is positive')
elif x < 0:
    print(x, 'is negative')
else:
    print(x, 'is neither strictly positive nor strictly negative')
```

    20 is positive

Combining these two statements would make for a nested if ... else statement.

```python
x = -3
if x > 0:
    if (x%2) == 0:
        print(x, 'is positive and even')
    else:
        print(x, 'is positive and odd')
elif x < 0:
    if (x%2) == 0:
        print(x, 'is negative and even')
    else:
        print(x, 'is negative and odd')
else:
    print(x, 'is 0')
```

    -3 is negative and odd

### Loops

#### "For" Loops

"For" loops iterate over a given sequence. They are very easy to implement as the following example shows. We start with an example and give some explanations afterwards.

For our example, let's assume you ought to sum up the integer values of a sequence from 10 to 1 with a loop. There are obviously more efficient ways of doing this but this serves well as an introductory example. From primary school we know the result is easily calculated as

$$
\begin{equation}
    \sum_{i=1}^n x_i = \dfrac{n (n+1)}{2} \qquad -> \qquad \dfrac{10 \cdot 11}{2} = 55
\end{equation}
$$

```python
seq = np.arange(start=10, stop=0, step=-1)
seqSum = 0
for value in seq:
    seqSum = seqSum + value

seqSum
```

    55

A few imprtant notes:
* Indentation is not just here for better readability of the code but it is actually necessary for Python to correctly interpret the code.
* Though it is not necessary, we initiate `seqSum = 0` here. Otherwise, if we run the code repeatedly we add to the previous total!
* `value` takes on every value in array `seq`. In the first loop `value=10`, second loop `value=9`, etc.

Loops can be nested, too. Here's an example.

```python
seq = seq.reshape(2, 5)
seqSum = 0
row, col = seq.shape

for rowIndex in range(0, row):
    for colIndex in range(0, col):
        seqSum = seqSum + seq[rowIndex, colIndex]

seqSum
```

    55

#### "While" Loops

"While" loops execute as long as a certain boolean condition is met. Picking up the above example we can formulate the following loop:

```python
seqSum = 0
i = 10
while i >= 1:
    seqSum = seqSum + i
    i = i - 1  # Also: i -= 1

print(seqSum)
```

    55

### Functions

Functions come into play when either a task needs to be performed more than once or when it helps reduce the complexity of a code.

Following up on our play examples from above, let us assume we're tasked to write a function which sums up all even and all odd integers of a vector.

```python
def sumOddEven(vector):
    """Calculates sum of odd and even numbers in array.

    Args:
        vector: NumPy array of length n

    Returns:
        odd: Sum of odd numbers
        even: Sum of even numbers
    """

    # Initiate values
    odd = 0
    even = 0

    # Loop through values of array; check for each
    # value whether it is odd or even and add to
    # previous total.
    for value in vector:
        if (value % 2) == 0:
            even = even + value
        else:
            odd = odd + value

    return odd, even

# Initiate array [1, 2, ..., 99, 100]
seq = np.arange(1, 101, 1)

# Apply function and print results
odd, even = sumOddEven(seq)
print('Odd: ', odd, ', ', 'Even: ', even)
```

    Odd:  2500 ,  Even:  2550

## Commenting

Above code snippet not only shows how functions are set up but also displays the importance of comments. Comments are preceeded by a hash sign (#), such that the interpreter will not parse what follows the hash.  When programming, you should always comment your code to notate your work. This details your steps/thoughts/ideas not only for other developers but also for you when you pick up your code some time after writing it. Good programmers make heavy use of commenting and I strongly encourage the reader to follow this standard.

## Slowness of Loops

It is at this point important to note that loops should only be used as a last resort. Below we show why. The first code runs our previously defined function. The second code uses NumPy's built-in function.

```python
%%timeit
seq = np.arange(1,10001, 1)
sumOddEven(seq)
```

    4.47 ms ± 415 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

```python
%%timeit
seq[(seq % 2) == 0].sum()
seq[(seq % 2) == 1].sum()
```

    10.5 µs ± 924 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

Above timing results show what was hinted before: In 9'999 out of a 10'000 cases it is significantly faster using already built in functions compared to loops. The simple reason is that modules such as NumPy or Pandas use (at their core) optimized compile code to calculate the results and this is most certainly faster than a loop.

So in summary: Above examples helped introduce if statements, loops and functions. In real life, however, you should check if Python does not already offer a built-in function for your task. If yes, make sure to use it.

## Broadcasting

### Computations on Arrays

In closing this chapter we briefly introduce NumPy's broadcasting functionality. Rules for matrix arithmetic apply to NumPy arrays as one would expect and it is left to the reader to explore it. Broadcasting, however, goes one step further in that it allows for element-by-element operations on arrays (and matrices) of different dimensions - which under normal rules would not be compatible. An example shows this best.

```python
M = np.ones(shape=(3, 3))
v = np.array([1, 2, 3])
M + v
```

    array([[ 2.,  3.,  4.],
           [ 2.,  3.,  4.],
           [ 2.,  3.,  4.]])

```python
# Notice the difference
vecAdd = v + v
broadAdd = v.reshape((3, 1)) + v

print(vecAdd, '\n')
print(broadAdd)
```

    [2 4 6]

    [[2 3 4]
     [3 4 5]
     [4 5 6]]

## Further Resources

The following ressources, which were consulted to write this notebook, are recommended to better acquaint yourself with Python and NumPy:

* Vanderplas, Jake, 2016, *Python Data Science Handbook* (O'Reilly Media, Sebastopol, CA).
* Sheppard, Kevin, 2017, Introduction to Python for Econometrics, Statistics and Data Analysis from Website https://www.kevinsheppard.com/images/b/b3/Python_introduction-2016.pdf, 07/07/2017.
* Paarsch, Harry J., and Golyaev, Konstantin, 2016, *A Gentle Introduction to Effective Computing in Quantitative Research: What Every Research Assistant Should Know*, MIT Press, Cambridge, MA.


### Static Equilibrium

In this notebook, we will derive, interpret, and simulate the static equilibrium of the model.

Last edited: Wed. 1/23/19, by Arnav Sood

```julia
# For equation numbering
macro javascript_str(s) display("text/javascript", s); end

javascript"""
MathJax.Hub.Queue(
  ["resetEquationNumbers", MathJax.InputJax.TeX],
  ["PreProcess", MathJax.Hub],
  ["Reprocess", MathJax.Hub]
);
"""
```

### Market's Problem

Begin with the formula for the Inflow Prior $\lambda^I$

\begin{equation}
\lambda^I = \frac{e_H \lambda}{e_H \lambda + e_L (1 - \lambda}
\end{equation}

Recall the formula for the final signal-dependent prior, $\lambda^o(h; m)$, in terms of the above:

\begin{equation}
\lambda^o(h; m) = \frac{\lambda^I}{\lambda^I + (1-\lambda^I)\phi}
\end{equation}

This can be rearranged to express $\lambda^o(h; m)$ (hereafter $\lambda^o(h)$, for convenience), in terms of model primitives:

\begin{equation}
\lambda^o(h) = \frac{e_H \lambda}{e_H \lambda + \phi e_L (1 - \lambda)}
\end{equation}

This passes a sanity check, since it says that the final prior is the share of high types, out of all those who get the high signal (high types + $\phi$ times low types)

### Firms' Problem

Firms here are mechanistic, acting according to:

\begin{equation}
V_i = \max_{\text{entry}, \text{no entry}}\{\omega_i, \phi_i R - p\}
\end{equation}

in the constraint region.

Assuming an interior solution (that is, one where $e_H, e_L \in (0, 1)$ strictly), we have:

\begin{align}
e_H &= F_H(R - p) \\
e_L &= F_L(\phi R - p)
\end{align}

Note that $e_H, e_L$ are but **fractions** of firms of each type that enter the market, and need to be multiplied by their respective (prior) masses, $\lambda, 1 - \lambda$, to have meaning.

We assumed earlier that the constraint is not violated. Or:

\begin{align}
\lambda^o(h) &\geq \underline{\lambda} \\
\frac{e_H \lambda}{e_H \lambda + \phi e_L (1 - \lambda)} &\geq \underline{\lambda} \\
e_H \lambda (1 - \underline{\lambda}) &\geq \underline{\lambda} \phi e_L (1 - \lambda) \\
\phi &\leq \frac{e_H \lambda (1 - \underline{\lambda})}{e_L \underline{\lambda} (1 - \lambda)}
\end{align}

This will be our chief constraint. For ease of reference, we can call this the **market credulity constraint.**

### Rater's Problem

A forward-looking credit rater, who (for convenience) shares a common prior with the market about the type-distribution of firms, is then seeking to solve the following problem:

\begin{align}
\max_{\phi \in [0, 1], p \geq 0} p \left\{ F_H(R - p) \lambda + F_L(\phi R - p)(1 - \lambda) \right\}
\end{align}

subject to:

\begin{equation}
\phi \leq \frac{e_H \lambda (1 - \underline{\lambda})}{e_L \underline{\lambda} (1 - \lambda)}
\end{equation}

We can write the following Lagrangian (assuming the problem is suitable, which we will have to show later):

\begin{equation}
L = p \left\{ F_H(R - p) \lambda + F_L(\phi R - p)(1 - \lambda) \right\} - \mu (\log \phi + \log F_L(\phi R - p) - \log F_H(R - p) - \log \tau)
\end{equation}

Where

\begin{equation}
\tau \equiv \frac{\lambda(1 - \underline{\lambda})}{\underline{\lambda}(1-\lambda)}
\end{equation}

We can derive first-order conditions:

#### FOC $\phi$

\begin{equation}
p R (1 - \lambda) f_L(\phi R - p) = \mu \left(\frac{1}{\phi} + \frac{R f_L(\phi R - p)}{F_L(\phi R - p)}\right)
\end{equation}

#### FOC $p$

\begin{equation}
F_H(R - p)\lambda + F_L(\phi R - p)(1 - \lambda) - p[f_H(R - p)\lambda + f_L(\phi R - p)(1 - \lambda)] + \mu \left(\frac{f_L(\phi R - p)}{F_L(\phi R - p)} - \frac{f_H(R - p)}{F_H(R - p)} \right) = 0
\end{equation}

#### Complementary Slackness

\begin{equation}
\mu (\log \phi + \log F_L (\phi R - p) - \log F_H (R - p) - \log \tau) = 0
\end{equation}

#### Model Solution

Rewriting (15), we can derive an expression for the Lagrange/KKT multiplier:

\begin{equation}
\mu = \frac{\phi (1 - \lambda)F_L(\phi R - p)R \cdot f_L(\phi R - p)}{F_L(\phi R - p) + \phi R f_L(\phi R - p)}
\end{equation}

We can make some observations here.

1. The term to the left of the $\cdot$ is total payoff earned on the public market by low-type firms.

2. The multiplier is nonzero iff (1) the $\phi$ parameter is nonzero, (2) the market reward is nonzero, (3) the price is nonzero, (4) $\lambda$ is not 1 (i.e., rater and market must believe $\exists$ some low-type firms), and (5) **both** $f_L$ and $F_L$ are nonzero (i.e., there has been some entry, and the density function is not locally flat, so a small perturbation yields rewards).

Assuming the above parameter restrictions hold, we know the complementary slackness condition can be divided through by $\mu$, to yield a binding constraint.

### Interpretation

### Implementation and Comparative Statics

In general, we have a system of 3 equations (15, 16, 17) in 3 unknowns ($\phi$, $p$, $\mu$). We can feed this to a nonlinear solver.

First, define a holder for model primitives.

#### Setup and Introduction

```julia
using Parameters, Distributions
```

```julia
Model = @with_kw (λ = 0.5, # share of good firms
                  R = 2, # market payoff
                  λ_bar = 0.74, # market credulity threshold
                  ω_H = Uniform(1.5, 2.5), # High-type distribution of outside options
                  ω_L = Uniform(1.01, 2.01), # Low-type distribution of outside options
                  f_L = x -> pdf(ω_L, x),
                  f_H = x -> pdf(ω_H, x),
                  F_L = x -> cdf(ω_L, x),
                  F_H = x -> cdf(ω_H, x)
           )
```

    #3 (generic function with 2 methods)

Before proceeding, it's worh examining how this object we created works.

If we call it without arguments, it will the default values we supplied.

```julia
m = Model();
@show m.λ, m.R, # defaults
```

    (m.λ, m.R) = (0.5, 2)

    (0.5, 2)

But we can also supply specific arguments.

```julia
m = Model(λ = 0.3, R = 20, ω_H = Uniform(1., 21.));
@show m.λ, m.R
```

    (m.λ, m.R) = (0.3, 20)

    (0.3, 20)

It is preferred to use these things instead of a struct (i.e., the code below) for two reasons.

1. Simplicity. Compare the code above to:

```
struct Model{TF <: AbstractFloat, TI <: Integer, ...}
λ::TF
R::TI
...
```

2. Correctness. If you are writing a `struct` by hand, then to get equal performance to a named tuple (the kind of thing we generate with a `Model()` call), we need to get the type parameterization (i.e., the `TF, TI`, etc.) exactly right. This is complicated by the fact that we use objects that are themselves parametrically typed (such as `Uniform{Float64}(a = 1.01, b = 2.01)`, for the distributions).

#### Computational Solution

Define a baseline model.

```julia
m = Model(R = 1.5);
```

Write code to take guesses for $\phi, p, \mu$, and a model $m$, and calculate the residuals for the three equations.

```julia
function residuals(vals; model = m)
    # unpack inputs
    @unpack λ, R, λ_bar, ω_H, ω_L, f_L, f_H, F_L, F_H = model
    ϕ, p, μ = vals # so vals is some vector [ϕ, p, μ]
    # define τ
    τ = λ*(1 - λ_bar)/(λ_bar * (1 - λ))
    # calculate residuals
    ϕ_residual = ϕ > 0 && F_L(ϕ*R - p) > 0 ? p*R*(1-λ)*f_L(ϕ*R - p) - μ*(1/ϕ + R*f_L(ϕ*R - p)/F_L(ϕ*R -p)) : 100 # (15) (return 100 for invalid residual expressions)
    p_residual = F_L(ϕ*R - p) > 0 && F_H(R - p) > 0 ? F_H(R - p)*λ + F_L(ϕ*R - p)*(1 - λ) - p*(f_H(R - p) + f_L(ϕ*R - p)) + μ*(f_L(ϕ*R - p)/F_L(ϕ*R - p) - f_H(R - p)/F_H(R - p)) : 100 # (16)
    μ_residual =sol ϕ > 0 && τ > 0 && F_L(ϕ*R - p) > 0 && F_H(R - p) > 0 ? μ*(log(ϕ) + log(F_L(ϕ*R - p)) - log(F_H(R - p)) - log(τ)) : 100 # (17)
    # return
    residuals = [ϕ_residual, p_residual, μ_residual]
end
```

Pass this to the solver.

```julia
using NLsolve # https://github.com/JuliaNLSolvers/NLsolve.jl, one of the main Julia solvers
```

```julia
@time result = nlsolve(residuals, [0.001, 0.001, 0.], inplace = false, store_trace = true, iterations = 10^6)
```


# [ATM 623: Climate Modeling](../index.ipynb)
[Brian E. J. Rose](http://www.atmos.albany.edu/facstaff/brose/index.html), University at Albany
# Lecture 1: Climate models, the global energy budget and Fun with Python

## Warning: content out of date and not maintained

You really should be looking at [The Climate Laboratory book](https://brian-rose.github.io/ClimateLaboratoryBook) by Brian Rose, where all the same content (and more!) is kept up to date.

***Here you are likely to find broken links and broken code.***

### About these notes:

This document uses the interactive [`Jupyter notebook`](https://jupyter.org) format. The notes can be accessed in several different ways:

- The interactive notebooks are hosted on `github` at https://github.com/brian-rose/ClimateModeling_courseware
- The latest versions can be viewed as static web pages [rendered on nbviewer](http://nbviewer.ipython.org/github/brian-rose/ClimateModeling_courseware/blob/master/index.ipynb)
- A complete snapshot of the notes as of May 2017 (end of spring semester) are [available on Brian's website](http://www.atmos.albany.edu/facstaff/brose/classes/ATM623_Spring2017/Notes/index.html).

[Also here is a legacy version from 2015](http://www.atmos.albany.edu/facstaff/brose/classes/ATM623_Spring2015/Notes/index.html).

Many of these notes make use of the `climlab` package, available at https://github.com/brian-rose/climlab

## Contents

1. [What is a Climate Model?](#section1)
2. [The observed global energy budget](#section2)
3. [Quantifying the planetary energy budget](#section3)
4. [Using Python to compute emission to space](#section4)

____________

## 1. What is a Climate Model?
____________

First, some thoughts on modeling from [xkcd](https://xkcd.com)

Let's be a little pedantic and decompose that question:

- what is Climate?
- what is a Model?

**Climate** is

- statistics of weather, e.g. space and time averages of temperature and precip.
- (statistics might also mean higher-order stats: variability etc)

A **model** is

 - not easy to define!

Wikipedia: http://en.wikipedia.org/wiki/Conceptual_model

> In the most general sense, a model is anything used in any way to represent anything else. Some models are physical objects, for instance, a toy model which may be assembled, and may even be made to work like the object it represents. Whereas, a conceptual model is a model made of the composition of concepts, that thus exists only in the mind. Conceptual models are used to help us know, understand, or simulate the subject matter they represent.

George E. P. Box (statistician):
> Essentially, all models are wrong, but some are useful.”

From the Climate Modelling Primer, 4th ed (McGuffie and Henderson-Sellers):

> In the broadest sense, models are for learning about the world (in our case, the climate) and the learning takes place in the contruction and the manipulation of the model, as anyone who has watched a child build idealised houses or spaceships with Lego, or built with it themselves, will know.  Climate models are, likewise, idealised representations of a complicated and complex reality through which our understanding of the climate has significantly expanded. All models involve some ignoring, distoring and approximating, but gradually they allow us to build understanding of the system being modelled. A child's Lego construction typically contains the essential elements of the real objects, improves with attention to detail, helps them understand the real world, but is never confused with the real thing.

### A minimal definition of a climate model

*A representation of the exchange of energy between the Earth system and space, and its effects on average surface temperature.*

(what average?)

Note the focus on **planetary energy budget**. That’s the key to all climate modeling.

____________

## 2. The observed global energy budget
____________

The figure below shows current best estimates of the *global, annual mean* energy fluxes through the climate system.

We will look at many of these processes in detail throughout the course.

## Things to note:

### On the shortwave side

- global mean albedo is 101.9 W m$^{-2}$ / 341.3 W m$^{-2}$ = 0.299
- Reflection off clouds = 79 W m$^{-2}$
- Off surface = 23 W m$^{-2}$
    - 3 times as much reflection off clouds as off surface

Why??  Think about both areas of ice and snow, and the fact that sunlight has to travel through cloudy atmosphere to get to the ice and snow. Also there is some absorption of shortwave by the atmosphere.

- Atmospheric absorption = 78 W m$^{-2}$
(so about the same as reflected by clouds)

QUESTION: Which gases contribute to shortwave absorption?

- O$_3$ and H$_2$O mostly.
- We will look at this later.

### On the longwave side

- Observed emission from the SURFACE is 396 W m$^{-2}$
- very close to the blackbody emission $\sigma T^4$ at $T = 288$ K (the global mean surface temperature).
- BUT emission to space is much smaller = 239 W m$^{-2}$

QUESTION: What do we call this?  (greenhouse effect)

### Look at net numbers…

- Net absorbed = 0.9 W m$^{-2}$
- Why?
- Where is that heat going?

Note, the exchanges of energy between the surface and the atmosphere are complicated, involve a number of different processes. We will look at these more carefully later.

### Additional points:

- Notice that this is a budget of energy, not temperature.
- We will need to discuss the connection between the two
- **Clouds** affect both longwave and shortwave sides of the budget.
- **WATER** is involved in many of the terms:

    - evaporation
    - latent heating (equal and opposite in the global mean)
    - clouds
    - greenhouse effect
    - atmospheric SW absorption
    - surface reflectivity (ice and snow)

### Discussion point

How might we expect some of the terms in the global energy budget to vary under anthropogenic climate change?

____________

## 3. Quantifying the planetary energy budget
____________

A budget for the **energy content of the global atmosphere-ocean system**:

\begin{align}
\frac{dE}{dt} &= \text{net energy flux in to system} \\
 &= \text{flux in – flux out}
\end{align}

where $E$ is the **enthalpy** or **heat content** of the total system.

We will express the budget **per unit surface area**, so each term above has units W m$^{-2}$

Note: any **internal exchanges** of energy between different reservoirs (e.g. between ocean, land, ice, atmosphere) do not appear in this budget – because $E$ is the **sum of all reservoirs**.

### Assumption:

**The only quantitatively important energy sources to the whole system are radiative fluxes to and from space.**

Let’s model those TOA (top-of-atmosphere) fluxes.

Flux in is **incoming solar radiation**
The solar constant is

$$ S_0 = 1365.2 \text{ W m}^{-2}  $$

(all values will be consistent with Trenberth and Fasullo figure unless noted otherwise)

This is the flux of energy from the sun incident on a unit area perpendicular to the beam direction.

The area-weighted global mean incoming solar flux is

$$ Q = S_0 \frac{A_{cross-section}}{A_{surface}}  $$

[ draw sketch of sphere and illuminated disk ]

where

- $A_{cross-section}$ = area of the illuminated disk = $\pi a^2$
- $A_{surface}$ = surface area of sphere = $4 \pi a^2$
- $a$ = radius of Earth

So flux in is $Q =  S_0 / 4 = 341.3$ W m$^{-2}$

Flux out has two parts:

- Reflected solar radiation
- Emitted terrestrial (longwave) radiation

Introduce terminology / notation:

**OLR = outgoing longwave radiation = terrestrial emissions to space**

Define the **planetary albedo**:

- $\alpha$ = reflected solar flux / incoming solar flux
- Or reflected flux = $\alpha Q$ = 101.9 W m$^{-2}$ from data
- So from data, $\alpha \approx 0.3$

Define **ASR = absorbed solar radiation**
\begin{align}
ASR &= \text{ incoming flux – reflected flux} \\
 &= Q - \alpha Q \\
 &= (1-\alpha) Q
\end{align}

Our energy budget then says

$$ \frac{dE}{dt} = (1-\alpha) Q - OLR $$

Note: **This is a generically true statement.** We have just defined some terms, and made the [very good] assumption that the only significant energy sources are radiative exchanges with space.

**This equation is the starting point for EVERY CLIMATE MODEL.**

But so far, we don’t actually have a MODEL. We just have a statement of a budget. To use this budget to make a model, we need to relate terms in the budget to state variables of the atmosphere-ocean system.

For now, the state variable we are most interested in is **temperature** – because it is directly connected to the physics of each term above.

____________

## 4. Using Python to compute emission to space
____________

*Most of what follows is intended as a "fill in the blanks" exercise. We will practice writing some Python code while discussing the physical process of longwave emission to space.*

Suppose the Earth behaves like a **blackbody radiator** with effective global mean **emission temperature $T_e$**.

Then

$$ OLR = \sigma T_e^4 $$

where OLR = "Outgoing Longwave Radiation", and $\sigma = 5.67 \times 10{-8}$ W m$^{-2}$ K$^{-4}$ the Stefan-Boltzmann constant

**We can just take this as a definition of the emission temperature.**

Looking back at the observations, the global, annual mean value for OLR is 238.5 W m$^{-2}$.

### Calculate the emission temperature $T_e$

Rerranging the Stefan-Boltzmann law we get

$$ T_e = \left(\frac{\text{OLR}}{\sigma} \right)^{\frac{1}{4}} $$

First just use Python like a hand calculator to calculate $T_e$ iteractively:

```python
OLR=238.5 # W/m23K4
sigma=5.67e-8 # W/m2K4
Te=(OLR/sigma)**(1/4)
print('%.2f K' % Te)
```

    254.67 K

Try typing a few different ways, with and without whitespace.

```python

```

#### Python fact 1

extra spaces are ignored!

But typing numbers interactively is tedious and error prone. Let's define a variable called `sigma`

```python

```

#### Python fact 2

We can define new variables interactively. Variables let us give names to things. Names make our code easy to understand.

### Thoughts on emission temperature

What value did we find for the emission temperature $T_e$? How does it compare to the actual global mean surface temperature?

*Is the blackbody radiator a good model for the Earth's emission to space?*

### A simple greenhouse model

The emission to space is lower because of the greenhouse effect, which we will study in detail later.

For now, just introduce a basic concept:

*Only a fraction of the surface emission makes it out to space.*

We will model the OLR as

$$ \text{OLR} = \tau \sigma T_s^4 $$

where $\tau$ is a number we will call the **transmissivity** of the atmosphere.

Let's fit this model to observations:

$$ \tau = \frac{\text{OLR}}{\sigma T_s^4} $$

```python
tau = 238.5 / sigma / 288**4
```

Try calculating OLR for a warmer Earth at 292 K:

```python
OLR = 238.5 # W/m2
sigma = 5.67e-8 # W/m2K4
Ts = 288 # K
tau = OLR / sigma / Ts**4 # transmissivity (dimensionless)

#Te=(OLR/sigma)**(1/4)
#print('%.2f K' % Te)

print('tau= %.2f' % tau)

Ts = 292 # K

def OLR(Ts):
    OLR = tau * sigma * Ts**4
    #print ('for Ts = %.2f K: OLR = %.02f W/m²' % (Ts,OLR))
    #return print ('for Ts = %.2f K: OLR = %.02f W/m²' % (Ts,OLR))
    return OLR

```

    tau= 0.61

Naturally the emission to space is higher. By how much has it increased for this 4 degree warming?

```python
OLR(288)
OLR(292)

```

    252.02860648282106

Answer: 13.5 W m$^{-2}$. Okay but this is tedious and prone to error.
What we really want to do is **define a reusable function**

```python

```

Note a few things:

-	The colon at the end of the first line indicates that there is more coming.
-	The interpreter automatically indents the code for us (after the colon)
-	The interpreter automatically colors certain key words
-	We need to hit return one more time at the end to finish our function

#### Python fact 3

**Indentations are not ignored!  They serve to group together several lines of code.**

we will see plenty of examples of this – in this case, the indentation lets the interpreter know that the code is all part of the function definition.

#### Python fact 4:

`def` is a keyword that defines a function.

Just like a mathematical function, a Python function takes one or more input arguments, performs some operations on those inputs, and gives back some resulting value.

##### Python fact 5:

`return` is a keyword that defines what value will be returned by the function.

Once a function is defined, we can call it interactively:

```python
print(OLR(288), OLR(292), OLR(292)-OLR(288))
```

    238.5 252.02860648282106 13.528606482821061

#### Python fact 6

** The `#` symbol is used for comments in Python code. **

The interpreter will ignore anything that follows `#` on a line of code.

#### Python fact 7:

`print` is a function that causes the value of an expression (or a list of expressions) to be printed to the screen.

(Don’t always need it, because by default the interpreter prints the output of the last statement to the screen, as we have seen).

Note also that we defined variables named sigma and epsilon inside our OLR function.

What happens if you try to `print(epsilon)`?

```python
print(epsilon)
```

#### Python fact 8:

**Variables defined in functions do not exist outside of that function.**

Try declaring `sigma = 2`, then `print(sigma)`. And try computing `OLR(288)` again. Did anything change?

```python
sigma=2
print(sigma)
OLR(288)
```

    2

    8412698412.698413

Note that we didn’t really **need** to define those variables inside the function. We could have written the function in one line.

But sometimes using named variables *makes our code much easier to read and understand!*

### Arrays with `numpy`

Now let’s try some array calculations:

```python
import numpy as np
T = np.linspace(230, 300, 10)
print(T[2])

```

    245.55555555555554

- We have just created an array object.
- The `linspace` function creates an array of numbers evenly spaced between the start and end points.
- The third argument tells Python how many elements we want.

We will use the `numpy` package all the time. It is the basic workhorse of scientific computing with Python. We can't do much with arrays of numbers.

Does our `OLR` function work on an array of temperature values?

```python
OLR = 238.5 # W/m2
sigma = 5.67e-8 # W/m2K4
Ts = 288 # K
tau = OLR / sigma / Ts**4 # transmissivity (dimensionless)

def OLR(Ts):
    OLR = tau * sigma * Ts**4
    #print ('for Ts = %.2f K: OLR = %.02f W/m²' % (Ts,OLR))
    return print ('for Ts = %.2f K: OLR = %.02f W/m²' % (Ts,OLR))
    #return OLR
```

Now let’s assign these values to a new variable.

```python
T = np.linspace(230, 300, 10)
for i in T:
    OLR(T[i])

```

Now try again to compute `OLR(288)`

What do you get?

```python
size(T)
```

#### Python fact 9: Assigning a value to a named variable overwrites whatever was already assigned to that name.

Python is also case sensitive. If we had used `olr` to store the array, there would be no conflict.

Now let’s re-enter our function. Start typing `def` and then hit the “up arrow” key. What happens?

```python
OLR = 238.5 # W/m2
sigma = 5.67e-8 # W/m2K4
Ts = 288 # K
tau = OLR / sigma / Ts**4 # transmissivity (dimensionless)

#Te=(OLR/sigma)**(1/4)
#print('%.2f K' % Te)

print('tau= %.2f' % tau)

Ts = 292 # K

def OLR(Ts):
    OLR = tau * sigma * Ts**4
    #print ('for Ts = %.2f K: OLR = %.02f W/m²' % (Ts,OLR))
    #return print ('for Ts = %.2f K: OLR = %.02f W/m²' % (Ts,OLR))
    return OLR
```

The editor gives us lots of useful keyboard shortcuts.

Here it’s looking up the last expression we entered that began with `def`. Saves a lot of time and typing!

Re-enter the function.

```python
def
```

What happens if you use the `up arrow` without typing anything first?

```python

```

Also, try typing `history`

```python

```

This is very handy. The Python console is taking notes for you!

```python

```

[Back to ATM 623 notebook home](../index.ipynb)

____________
## Version information
____________

```python
%load_ext version_information
%version_information
```

____________

## Credits

The author of this notebook is [Brian E. J. Rose](http://www.atmos.albany.edu/facstaff/brose/index.html), University at Albany.

It was developed in support of [ATM 623: Climate Modeling](http://www.atmos.albany.edu/facstaff/brose/classes/ATM623_Spring2015/), a graduate-level course in the [Department of Atmospheric and Envionmental Sciences](http://www.albany.edu/atmos/index.php)

Development of these notes and the [climlab software](https://github.com/brian-rose/climlab) is partially supported by the National Science Foundation under award AGS-1455071 to Brian Rose. Any opinions, findings, conclusions or recommendations expressed here are mine and do not necessarily reflect the views of the National Science Foundation.
____________

```python

```

Lectures/Lecture01 -- Planetary energy budget.ipynb