##### Copyright 2020 The Cirq Developers

```
#@title Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
```

# Shor's algorithm

    View on QuantumAI

    Run in Google Colab

    View source on GitHub

    Download notebook

This tutorial presents a pedagogical demonstration of Shor's algorithm. It is a modified and expanded version of [this Cirq example](https://github.com/quantumlib/Cirq/blob/master/examples/shor.py).

```
"""Install Cirq."""
try:
    import cirq
except ImportError:
    print("installing cirq...")
    !pip install --quiet cirq
    print("installed cirq.")
```

```
"""Imports for the notebook."""
import fractions
import math
import random

import numpy as np
import sympy
from typing import Callable, List, Optional, Sequence, Union

import cirq
```

# Order finding

Factoring an integer $n$ can be reduced to finding the period of the modular exponential function (to be defined). Finding this period can be accomplished (with high probability) by finding the order of a randomly chosen element of the multiplicative group modulo $n$.

Let $n$ be a positive integer and

$$ \mathbb{Z}_n := \{x \in \mathbb{Z}_+ : x < n \text{ and } \text{gcd}(x, n) = 1\} $$

be the multiplicative group modulo $n$.
Given $x \in \mathbb{Z}_n$, compute the smallest positive integer $r$ such that $x^r \text{ mod } n = 1$.

It can be shown from group/number theory that:

(1) Such an integer $r$ exists. (Note that $g^{|G|} = 1_G$ for any group $G$ with cardinality $|G|$ and element $g \in G$, but it's possible that $r < |G|$.)

(2) If $n = pq$ for primes $p$ and $q$, then $|\mathbb{Z}_n| = \phi(n) = (p - 1) (q - 1)$. (The function $\phi$ is called [Euler's totient function](https://en.wikipedia.org/wiki/Euler%27s_totient_function).)

(3) The modular exponential function

$$ f_x(z) := x^z \mod n $$

is periodic with period $r$ (the order of the element $x \in \mathbb{Z}_n$). That is, $f_x(z + r) = f_x(z)$.

(4) If we know the period of the modular exponential function, we can (with high probability) figure out $p$ and $q$ -- that is, factor $n$.

As a refresher, we can visualize the elements of some multiplicative groups $\mathbb{Z}_n$ for integers $n$ via the following simple function.

```
"""Function to compute the elements of Z_n."""
def multiplicative_group(n: int) -> List[int]:
    """Returns the multiplicative group modulo n.

    Args:
        n: Modulus of the multiplicative group.
    """
    assert n > 1
    group = [1]
    for x in range(2, n):
        if math.gcd(x, n) == 1:
            group.append(x)
    return group
```

For example, the multiplicative group modulo $n = 15$ is shown below.

```
"""Example of a multiplicative group."""
n = 15
print(f"The multiplicative group modulo n = {n} is:")
print(multiplicative_group(n))
```

One can check that this set of elements indeed forms a group (under ordinary multiplication).

## Classical order finding

A function for classically computing the order $r$ of an element $x \in \mathbb{Z}_n$ is provided below. This function simply computes the sequence

$$ x^2 \text{ mod } n $$
$$ x^3 \text{ mod } n $$
$$ x^4 \text{ mod } n $$
$$ \vdots $$

until an integer $r$ is found such that $x^r = 1 \text{ mod } n$. Since $|\mathbb{Z}_n| = \phi(n)$, this algorithm for order finding has time complexity $O(\phi(n))$ which is inefficient. (Roughly $O(2^{L / 2})$ where $L$ is the number of bits in $n$.)

```
"""Function for classically computing the order of an element of Z_n."""
def classical_order_finder(x: int, n: int) -> Optional[int]:
    """Computes smallest positive r such that x**r mod n == 1.

    Args:
        x: Integer whose order is to be computed, must be greater than one
           and belong to the multiplicative group of integers modulo n (which
           consists of positive integers relatively prime to n),
        n: Modulus of the multiplicative group.

    Returns:
        Smallest positive integer r such that x**r == 1 mod n.
        Always succeeds (and hence never returns None).

    Raises:
        ValueError when x is 1 or not an element of the multiplicative
        group of integers modulo n.
    """
    # Make sure x is both valid and in Z_n.
    if x < 2 or x >= n or math.gcd(x, n) > 1:
        raise ValueError(f"Invalid x={x} for modulus n={n}.")

    # Determine the order.
    r, y = 1, x
    while y != 1:
        y = (x * y) % n
        r += 1
    return r
```

An example of computing $r$ for a given $x \in \mathbb{Z}_n$ and given $n$ is shown in the code block below.

```
"""Example of (classically) computing the order of an element."""
n = 15  # The multiplicative group is [1, 2, 4, 7, 8, 11, 13, 14].
x = 8
r = classical_order_finder(x, n)

# Check that the order is indeed correct.
print(f"x^r mod n = {x}^{r} mod {n} = {x**r % n}")
```

The quantum part of Shor's algorithm is order finding, but done via a quantum circuit, which we'll discuss below.

## Quantum order finding

Quantum order finding is essentially quantum phase estimation with unitary $U$ that computes the modular exponential function $f_x(z)$ for some randomly chosen $x \in \mathbb{Z}_n$. The full details of how $U$ is computed in terms of elementary gates can be complex to unravel, especially on a first reading. In this tutorial, we'll use arithmetic operations in Cirq which can implement such a unitary $U$ without fully delving into the details of elementary gates.

Below we first show an example of a simple arithmetic operation in Cirq (addition) then discuss the operation we care about (modular exponentiation).

### Quantum arithmetic operations in Cirq

Here we discuss an example of defining an arithmetic operation in Cirq, namely modular addition. This operation adds the value of the input register into the target register. More specifically, this operation acts on two qubit registers as

$$ |a\rangle_i |b\rangle_t \mapsto |a\rangle_i |a + b \text{ mod } N_t \rangle_t . $$

Here, the subscripts $i$ and $t$ denote input and target register, respectively, and $N_t$ is the dimension of the target register.

To define this operation, called `Adder`, we inherit from `cirq.ArithmeticOperation` and override the four methods shown below. The main method is the `apply` method which defines the arithmetic. Here, we simply state the expression as $a + b$ instead of the more accurate $a + b \text{ mod } N_t$ above -- the `cirq.ArithmeticOperation` class is able to deduce what we mean by simply $a + b$ since the operation must be reversible.

```
"""Example of defining an arithmetic (quantum) operation in Cirq."""
class Adder(cirq.ArithmeticOperation):
    """Quantum addition."""
    def __init__(self, target_register, input_register):
        self.input_register = input_register
        self.target_register = target_register

    def registers(self):
        return self.target_register, self.input_register

    def with_registers(self, *new_registers):
        return Adder(*new_registers)

    def apply(self, target_value, input_value):
        return target_value + input_value
```

Now that we have the operation defined, we can use it in a circuit. The cell below creates two qubit registers, then sets the first register to be $|10\rangle$ (in binary) and the second register to be $|01\rangle$ (in binary) via $X$ gates. Then, we use the `Adder` operation, then measure all the qubits.

Since $10 + 01 = 11$ (in binary), we expect to measure $|11\rangle$ in the target register every time. Additionally, since we do not alter the input register, we expect to measure $|10\rangle$ in the input register every time. In short, the only bitstring we expect to measure is $1011$.

```
"""Example of using an Adder in a circuit."""
# Two qubit registers.
qreg1 = cirq.LineQubit.range(2)
qreg2 = cirq.LineQubit.range(2, 4)

# Define the circuit.
circ = cirq.Circuit(
    cirq.ops.X.on(qreg1[0]),
    cirq.ops.X.on(qreg2[1]),
    Adder(input_register=qreg1, target_register=qreg2),
    cirq.measure_each(*qreg1),
    cirq.measure_each(*qreg2)
)

# Display it.
print("Circuit:\n")
print(circ)

# Print the measurement outcomes.
print("\n\nMeasurement outcomes:\n")
print(cirq.sample(circ, repetitions=5).data)
```

In the output of this code block, we first see the circuit which shows the initial $X$ gates, the `Adder` operation, then the final measurements. Next, we see the measurement outcomes which are all the bitstring $1011$ as expected.

It is also possible to see the unitary of the adder operation, which we do below. Here, we set the target register to be two qubits in the zero state, i.e. $|00\rangle$. We specify the input register as the integer one which corresponds to the qubit register $|01\rangle$.

```
"""Example of the unitary of an Adder operation."""
cirq.unitary(
    Adder(target_register=cirq.LineQubit.range(2),
          input_register=1)
).real
```

We can understand this unitary as follows. The $i$th column of the unitary is the state $|i + 1 \text{ mod } 4\rangle$. For example, if we look at the $0$th column of the unitary, we see the state $|i + 1 \text{ mod } 4\rangle = |0 + 1 \text{ mod } 4\rangle = |1\rangle$. If we look at the $1$st column of the unitary, we see the state $|i + 1 \text{ mod } 4\rangle = |1 + 1 \text{ mod } 4\rangle = |2\rangle$. Similarly for the last two columns.

### Modular exponential arithmetic operation

We can define the modular exponential arithmetic operation in a similar way to the simple addition arithmetic operation, shown below. For the purposes of understanding Shor's algorithm, the most important part of the following code block is the `apply` method which defines the arithmetic operation.

```
"""Defines the modular exponential operation used in Shor's algorithm."""
class ModularExp(cirq.ArithmeticOperation):
    """Quantum modular exponentiation.

    This class represents the unitary which multiplies base raised to exponent
    into the target modulo the given modulus. More precisely, it represents the
    unitary V which computes modular exponentiation x**e mod n:

        V|y⟩|e⟩ = |y * x**e mod n⟩ |e⟩     0 <= y < n
        V|y⟩|e⟩ = |y⟩ |e⟩                  n <= y

    where y is the target register, e is the exponent register, x is the base
    and n is the modulus. Consequently,

        V|y⟩|e⟩ = (U**e|y)|e⟩

    where U is the unitary defined as

        U|y⟩ = |y * x mod n⟩      0 <= y < n
        U|y⟩ = |y⟩                n <= y
    """
    def __init__(
        self,
        target: Sequence[cirq.Qid],
        exponent: Union[int, Sequence[cirq.Qid]],
        base: int,
        modulus: int
    ) -> None:
        if len(target) < modulus.bit_length():
            raise ValueError(f'Register with {len(target)} qubits is too small '
                             f'for modulus {modulus}')
        self.target = target
        self.exponent = exponent
        self.base = base
        self.modulus = modulus

    def registers(self) -> Sequence[Union[int, Sequence[cirq.Qid]]]:
        return self.target, self.exponent, self.base, self.modulus

    def with_registers(
            self,
            *new_registers: Union[int, Sequence['cirq.Qid']],
    ) -> cirq.ArithmeticOperation:
        if len(new_registers) != 4:
            raise ValueError(f'Expected 4 registers (target, exponent, base, '
                             f'modulus), but got {len(new_registers)}')
        target, exponent, base, modulus = new_registers
        if not isinstance(target, Sequence):
            raise ValueError(
                f'Target must be a qubit register, got {type(target)}')
        if not isinstance(base, int):
            raise ValueError(
                f'Base must be a classical constant, got {type(base)}')
        if not isinstance(modulus, int):
            raise ValueError(
                f'Modulus must be a classical constant, got {type(modulus)}')
        return ModularExp(target, exponent, base, modulus)

    def apply(self, *register_values: int) -> int:
        assert len(register_values) == 4
        target, exponent, base, modulus = register_values
        if target >= modulus:
            return target
        return (target * base**exponent) % modulus

    def _circuit_diagram_info_(
            self,
            args: cirq.CircuitDiagramInfoArgs,
    ) -> cirq.CircuitDiagramInfo:
        assert args.known_qubits is not None
        wire_symbols: List[str] = []
        t, e = 0, 0
        for qubit in args.known_qubits:
            if qubit in self.target:
                if t == 0:
                    if isinstance(self.exponent, Sequence):
                        e_str = 'e'
                    else:
                        e_str = str(self.exponent)
                    wire_symbols.append(
                        f'ModularExp(t*{self.base}**{e_str} % {self.modulus})')
                else:
                    wire_symbols.append('t' + str(t))
                t += 1
            if isinstance(self.exponent, Sequence) and qubit in self.exponent:
                wire_symbols.append('e' + str(e))
                e += 1
        return cirq.CircuitDiagramInfo(wire_symbols=tuple(wire_symbols))
```

In the `apply` method, we see that we evaluate `(target * base**exponent) % modulus`. The `target` and the `exponent` depend on the values of the respective qubit registers, and the `base` and `modulus` are constant -- namely, the `modulus` is $n$ and the `base` is some $x \in \mathbb{Z}_n$.

The total number of qubits we will use is $3 (L + 1)$ where $L$ is the number of bits needed to store the integer $n$ to factor. The size of the unitary which implements the modular exponential is thus $4^{3(L + 1)}$. For a modest $n = 15$, the unitary requires storing $2^{30}$ floating point numbers in memory which is out of reach of most current standard laptops.

```
"""Create the target and exponent registers for phase estimation,
and see the number of qubits needed for Shor's algorithm.
"""
n = 15
L = n.bit_length()

# The target register has L qubits.
target = cirq.LineQubit.range(L)

# The exponent register has 2L + 3 qubits.
exponent = cirq.LineQubit.range(L, 3 * L + 3)

# Display the total number of qubits to factor this n.
print(f"To factor n = {n} which has L = {L} bits, we need 3L + 3 = {3 * L + 3} qubits.")
```

As with the simple adder operation, this modular exponential operation has a unitary which we can display (memory permitting) as follows.

```
"""See (part of) the unitary for a modular exponential operation."""
# Pick some element of the multiplicative group modulo n.
x = 5

# Display (part of) the unitary. Uncomment if n is small enough.
# cirq.unitary(ModularExp(target, exponent, x, n))
```

## Using the modular exponentional operation in a circuit

The quantum part of Shor's algorithm is just phase estimation with the unitary $U$ corresponding to the modular exponential operation. The following cell defines a function which creates the circuit for Shor's algorithm using the `ModularExp` operation we defined above.

```
"""Function to make the quantum circuit for order finding."""
def make_order_finding_circuit(x: int, n: int) -> cirq.Circuit:
    """Returns quantum circuit which computes the order of x modulo n.

    The circuit uses Quantum Phase Estimation to compute an eigenvalue of
    the unitary

        U|y⟩ = |y * x mod n⟩      0 <= y < n
        U|y⟩ = |y⟩                n <= y

    Args:
        x: positive integer whose order modulo n is to be found
        n: modulus relative to which the order of x is to be found

    Returns:
        Quantum circuit for finding the order of x modulo n
    """
    L = n.bit_length()
    target = cirq.LineQubit.range(L)
    exponent = cirq.LineQubit.range(L, 3 * L + 3)
    return cirq.Circuit(
        cirq.X(target[L - 1]),
        cirq.H.on_each(*exponent),
        ModularExp(target, exponent, x, n),
        cirq.qft(*exponent, inverse=True),
        cirq.measure(*exponent, key='exponent'),
    )
```

Using this function, we can visualize the circuit for a given $x$ and $n$ as follows.

```
"""Example of the quantum circuit for period finding."""
n = 15
x = 7
circuit = make_order_finding_circuit(x, n)
print(circuit)
```

As previously described, we put the exponent register into an equal superposition via Hadamard gates. The $X$ gate on the last qubit in the target register is used for phase kickback. The modular exponential operation performs the sequence of controlled unitaries in phase estimation, then we apply the inverse quantum Fourier transform to the exponent register and measure to read out the result.

To illustrate the measurement results, we can sample from a smaller circuit. (Note that in practice we would never run Shor's algorithm with $n = 6$ because it is even. This is just an example to illustrate the measurement outcomes.)

```
"""Measuring Shor's period finding circuit."""
circuit = make_order_finding_circuit(x=5, n=6)
res = cirq.sample(circuit, repetitions=8)

print("Raw measurements:")
print(res)

print("\nInteger in exponent register:")
print(res.data)
```

We interpret each measured bitstring as an integer, but what do these integers tell us? In the next section we look at how to classically post-process to interpret them.

## Classical post-processing

The integer we measure is close to $s / r$ where $r$ is the order of $x \in \mathbb{Z}_n$ and $0 \le s < r$ is an integer. We use the continued fractions algorithm to determine $r$ from $s / r$ then return it if the order finding circuit succeeded, else we return `None`.

```
def process_measurement(result: cirq.TrialResult, x: int, n: int) -> Optional[int]:
    """Interprets the output of the order finding circuit.

    Specifically, it determines s/r such that exp(2πis/r) is an eigenvalue
    of the unitary

        U|y⟩ = |xy mod n⟩  0 <= y < n
        U|y⟩ = |y⟩         n <= y

    then computes r (by continued fractions) if possible, and returns it.

    Args:
        result: trial result obtained by sampling the output of the
            circuit built by make_order_finding_circuit

    Returns:
        r, the order of x modulo n or None.
    """
    # Read the output integer of the exponent register.
    exponent_as_integer = result.data["exponent"][0]
    exponent_num_bits = result.measurements["exponent"].shape[1]
    eigenphase = float(exponent_as_integer / 2**exponent_num_bits)

    # Run the continued fractions algorithm to determine f = s / r.
    f = fractions.Fraction.from_float(eigenphase).limit_denominator(n)

    # If the numerator is zero, the order finder failed.
    if f.numerator == 0:
        return None

    # Else, return the denominator if it is valid.
    r = f.denominator
    if x**r % n != 1:
        return None
    return r
```

The next code block shows an example of creating an order finding circuit, executing it, then using the classical postprocessing function to determine the order. Recall that the quantum part of the algorithm succeeds with some probability. If the order is `None`, try re-running the cell a few times.

```
"""Example of the classical post-processing."""
# Set n and x here
n = 6
x = 5

print(f"Finding the order of x = {x} modulo n = {n}\n")
measurement = cirq.sample(circuit, repetitions=1)
print("Raw measurements:")
print(measurement)

print("\nInteger in exponent register:")
print(measurement.data)

r = process_measurement(measurement, x, n)
print("\nOrder r =", r)
if r is not None:
    print(f"x^r mod n = {x}^{r} mod {n} = {x**r % n}")
```

You should see that the order of $x = 5$ in $\mathbb{Z}_6$ is $r = 2$. Indeed, $5^2 \text{ mod } 6 = 25 \text{ mod } 6 = 1$.

## Quantum order finder

We can now define a streamlined function for the quantum version of order finding using the functions we have previously written. The quantum order finder below creates the circuit, executes it, and processes the measurement result.

```
def quantum_order_finder(x: int, n: int) -> Optional[int]:
    """Computes smallest positive r such that x**r mod n == 1.

    Args:
        x: integer whose order is to be computed, must be greater than one
           and belong to the multiplicative group of integers modulo n (which
           consists of positive integers relatively prime to n),
        n: modulus of the multiplicative group.
    """
    # Check that the integer x is a valid element of the multiplicative group
    # modulo n.
    if x < 2 or n <= x or math.gcd(x, n) > 1:
        raise ValueError(f'Invalid x={x} for modulus n={n}.')

    # Create the order finding circuit.
    circuit = make_order_finding_circuit(x, n)

    # Sample from the order finding circuit.
    measurement = cirq.sample(circuit)

    # Return the processed measurement result.
    return process_measurement(measurement, x, n)
```

This completes our quantum implementation of an order finder, and the quantum part of Shor's algorithm.

# The complete factoring algorithm

We can use this quantum order finder (or the classical order finder) to complete Shor's algorithm. In the following code block, we add a few pre-processing steps which:

(1) Check if $n$ is even,

(2) Check if $n$ is prime,

(3) Check if $n$ is a prime power,

all of which can be done efficiently with a classical computer. Additionally, we add the last necessary post-processing step which uses the order $r$ to compute a non-trivial factor $p$ of $n$. This is achieved by computing $y = x^{r / 2} \text{ mod } n$ (assuming $r$ is even), then computing $p = \text{gcd}(y - 1, n)$.

```
"""Functions for factoring from start to finish."""
def find_factor_of_prime_power(n: int) -> Optional[int]:
    """Returns non-trivial factor of n if n is a prime power, else None."""
    for k in range(2, math.floor(math.log2(n)) + 1):
        c = math.pow(n, 1 / k)
        c1 = math.floor(c)
        if c1**k == n:
            return c1
        c2 = math.ceil(c)
        if c2**k == n:
            return c2
    return None

def find_factor(
    n: int,
    order_finder: Callable[[int, int], Optional[int]] = quantum_order_finder,
    max_attempts: int = 30
) -> Optional[int]:
    """Returns a non-trivial factor of composite integer n.

    Args:
        n: Integer to factor.
        order_finder: Function for finding the order of elements of the
            multiplicative group of integers modulo n.
        max_attempts: number of random x's to try, also an upper limit
            on the number of order_finder invocations.

    Returns:
        Non-trivial factor of n or None if no such factor was found.
        Factor k of n is trivial if it is 1 or n.
    """
    # If the number is prime, there are no non-trivial factors.
    if sympy.isprime(n):
        print("n is prime!")
        return None

    # If the number is even, two is a non-trivial factor.
    if n % 2 == 0:
        return 2

    # If n is a prime power, we can find a non-trivial factor efficiently.
    c = find_factor_of_prime_power(n)
    if c is not None:
        return c

    for _ in range(max_attempts):
        # Choose a random number between 2 and n - 1.
        x = random.randint(2, n - 1)

        # Most likely x and n will be relatively prime.
        c = math.gcd(x, n)

        # If x and n are not relatively prime, we got lucky and found
        # a non-trivial factor.
        if 1 < c < n:
            return c

        # Compute the order r of x modulo n using the order finder.
        r = order_finder(x, n)

        # If the order finder failed, try again.
        if r is None:
            continue

        # If the order r is even, try again.
        if r % 2 != 0:
            continue

        # Compute the non-trivial factor.
        y = x**(r // 2) % n
        assert 1 < y < n
        c = math.gcd(y - 1, n)
        if 1 < c < n:
            return c

    print(f"Failed to find a non-trivial factor in {max_attempts} attempts.")
    return None
```

The function `find_factor` uses the `quantum_order_finder` by default, in which case it is executing Shor's algorithm. As previously mentioned, due to the large memory requirements for classically simulating this circuit, we cannot run Shor's algorithm for $n \ge 15$. However, we can use the classical order finder as a substitute.

```
"""Example of factoring via Shor's algorithm (order finding)."""
# Number to factor
n = 184573

# Attempt to find a factor
p = find_factor(n, order_finder=classical_order_finder)
q = n // p

print("Factoring n = pq =", n)
print("p =", p)
print("q =", q)
```

```
"""Check the answer is correct."""
p * q == n
```


# Data Analyst's Toolbox: R and Python

> Institute of Political Science, NSYSU, 2020-12-25

Kuo, Yao-Jen <yaojenkuo@datainpoint.com> from [DATAINPOINT](https://www.datainpoint.com)

```python
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
```

## TL; DR

Source: <https://memes.tw/>

## About me

## Teaching practical data science online/offline, for individuals

- [如何成為資料分析師：從問題解決到行動方案，Hahow 好學校](https://hahow.in/cr/dajourney)
- Visualization and modern data science, Adjunct Instructor, National Taiwan University
- Programming and business analytics, Adjunct Instructor, National Taiwan Normal University
- Python for data analysis, Instructor, Chunghwa Telecom Academy
- Python for data science, Machine learning from scratch, Senior Instructor, CSIE Training Program, National Taiwan University

## Also for commercial banking clients

- 2020 DBS Training Program
- 2019 HNCB Training Program
- 2017 ESUN Training Program

## Writing books

- [新手村逃脫！初心者的 Python 機器學習攻略](https://www.books.com.tw/products/0010867390)
- [進擊的資料科學](https://www.books.com.tw/products/0010827812)
- [輕鬆學習 R 語言](https://www.books.com.tw/products/0010835361)

## Writing blogs

- [Medium](https://medium.com/@tonykuoyj)
- [Substack](https://datainpoint.substack.com/about)
- [方格子](https://vocus.cc/user/@yaojenkuo)

## Before being a instructor

- Working experience
    - Senior Data Analyst, Coupang Shanghai
    - Analytical Consultant, SAS Taiwan
    - Management Associate, Chinatrust Banking Corporation Taiwan
    - Research Assistant, McKinsey & Company Taiwan
- Education
    - MBA, National Taiwan University
    - BA, National Taiwan University

## Loves running with a marathon PR of 2:43:12 at 2019 Seoul Marathon

Source: <https://giphy.com>

## What is data analysis

## The definition

> We generate questions about a specific topic, we search for answers by exploring, transforming, and modelling data referring to our topic. And then use what we've learned to refine questions or generate new questions.

Source: [R for Data Science](https://r4ds.had.co.nz/)

## Why data analysis

> It is now an era of data-driven strategic thinking, and is probably never coming back.

## The three means of persuasion that an orator must rely on

- Ethos
- Pathos
- Logos

Source: [Aristotle, Rhetoric](https://en.wikipedia.org/wiki/Rhetoric)

## It is a lot easier to persuade via ethos or pathos, but it takes time

However, logos can be easily acquired once it is a fact and can be proven. Hence, data analysis is often the express way to logos.

## Modern data analysis can be illustrated as the flow of data

Source: [R for Data Science](https://r4ds.had.co.nz/)

## The funny definitions

Source: <https://twitter.com/cdixon/status/428914681911070720/photo/1>

Source: <https://www.warnerbros.com/tv/friends/>

Source: <https://www.warnerbros.com/tv/friends/>

## The serious definition

> Modern data analysis invovles applications and tools like importing, tidying, transformation, visualization, modeling, and communication. Surrounding all these is programming.

```python
def get_value_ratios(df, col_name):
    return df[col_name].value_counts() / df[col_name].value_counts().sum()
def get_checkbox_ratios(df, col_pattern):
    channel_names = []
    channel_checks = []
    for col_name in df.columns:
        if (col_pattern in col_name) and ('OTHER_TEXT' not in col_name):
            channel_name = df[col_name].value_counts().index
            channel_check = df[col_name].value_counts().values
            if channel_name.size != 0:
                channel_names.append(channel_name[0])
                channel_checks.append(channel_check[0])
    channel_counts = pd.Series(channel_checks, index=channel_names)
    channel_ratios = channel_counts / channel_counts.sum()
    channel_ratios_sorted = channel_ratios.sort_values(ascending=False)
    return channel_ratios_sorted
def plot_ans_38(ans_38_ser):
    fig = plt.figure()
    ax = plt.axes()
    ans_38_ser = ans_38_ser[::-1]
    ax.barh(ans_38_ser.index, ans_38_ser.values, color=['c', 'c', 'c', 'c', 'c', 'r'])
    ax.set_title("What is the primary tool that you use at work or school to analyze data?")
    ax.set_xticks([0, 0.1, 0.2, 0.3, 0.4, 0.5])
    ax.set_xticklabels(['0%', '10%', '20%', '30%', '40%', '50%'])
    ax.spines['top'].set_visible(False)
    ax.spines['right'].set_visible(False)
    ax.spines['bottom'].set_visible(False)
    plt.show()
def get_ax(ans_ser, plot_title, ax):
    ax = ax
    ans_ser = ans_ser[:10][::-1]
    ax.barh(ans_ser.index, ans_ser.values, color=['c', 'c', 'c', 'c', 'c', 'c', 'c', 'r', 'r', 'r'])
    ax.spines['top'].set_visible(False)
    ax.spines['right'].set_visible(False)
    ax.spines['bottom'].set_visible(False)
    ax.set_title(plot_title)
def plot_ans(ans_ser, question_str):
    fig, axes = plt.subplots()
    get_ax(ans_ser, question_str, axes)
    plt.show()
survey = pd.read_csv("https://kaggle-ml-ds-survey.s3-ap-northeast-1.amazonaws.com/kaggle-survey-2020/kaggle_survey_2020_responses.csv", skiprows=[1])
ans_7 = get_checkbox_ratios(survey, 'Q7')
ans_8 = get_value_ratios(survey, 'Q8')
ans_38 = get_value_ratios(survey, 'Q38')
```

## Use programming language to analyze data

Let's review a question from [2020 Kaggle ML & DS Survey](https://www.kaggle.com/c/kaggle-survey-2020):

> What is the primary tool that you use at work or school to analyze data?

```python
plot_ans_38(ans_38)
```

## It seems inevitable to write codes in modern data analysis

Source: <https://giphy.com/>

## Simply put, we can choose any programming language as long as it is capable of

- Importing data
- Tidying data
- Transforming data
- Visualizing data
- Modeling data
- Communicating data

## Well, actually a lot of programming languages are capable of doing these

- Python
- R
- Julia
- Scala
- Matlab
- SAS
- ...etc.

## How to choose among so many alternatives?

- The philosophy of "Eating a water mellon".
- The full support of scientific computing.
- Our objectivity.

## The philosophy of "Eating a water mellon"

Let's review another 2 questions from [2020 Kaggle ML & DS Survey](https://www.kaggle.com/c/kaggle-survey-2020):

- What programming languages do you use on a regular basis?
- What programming language would you recommend an aspiring data scientist to learn first?

```python
plot_ans(ans_7, "What programming languages do you use on a regular basis?")
```

```python
plot_ans(ans_8, "What programming language would you recommend \n an aspiring data scientist to learn first?")
```

## R and Python in Stack Overflow Trends

<https://insights.stackoverflow.com/trends?tags=python%2Cr%2Cjulia%2Cscala%2Cmatlab%2Csas>

## The full support of scientific computing

- Does the language support vectorization?
- Does the language support various data format?
- Does the language support visualization?

## Both R and Python support vectorization

- R uses built-in `vector` and `matrix`.
- Python uses a third-party `ndarray`.

## Both R and Python support various data format

- R uses
    - built-in named `list` to support key-value storage
    - built-in `data.frame` to support tabular data
- Python uses
    - built-in `dict` to support key-value storage
    - third-party `DataFrame` to support tabular data

## Both R and Python support visualization

- R uses
    - built-in basic plotting system to support static plotting
    - third-party `ggplot2` to support high-end static plotting
    - third-party `shiny` to support dynamic plotting
- Python uses
    - third-party `matplotlib` to support static plotting
    - third-party `seaborn` to support high-end static plotting
    - third-party `plotly` to support dynamic plotting

## Last but not least, it depends on our objectivity

- Specific or general-purposed?
- Functional or object-oriented?
- ...etc.

## We will generate our own objectivity once we start coding

Source: <https://giphy.com>

## Let's write some codes to analyze data

## Bringing up  a topic

> 大選開票看哪個里最準？ 「章魚里」神預測告訴你。每次到了選舉，總是會有幾個里開票與大選結果相似，因此被各界視為重點關注的開票區域。

Source: <https://www.cw.com.tw/article/5093012>

## We can generate some questions regarding this topic

- How to define 「章魚里」?
- Can we find out 「章魚里」 based on 2020 presidential data?
- Can we find the similarity of our own village?

## How to define 「章魚里」？

Basically, after a few literature search, you may find the definition of 「章魚里」 is quite ambigious. So we are using a much fancier metric: **cosine similarity**.

## What is cosine similarity

> Cosine similarity is a measure of similarity between two non-zero vectors of an inner product space. It is defined to equal the cosine of the angle between them, which is also the same as the inner product of the same vectors normalized to both have length 1.

\begin{equation}
a = (a_1, a_2, a_3) \\
b = (b_1, b_2, b_3)
\end{equation}

\begin{align}
cos\theta &= \frac{\sum_i(a_i \times b_i)}{\sqrt{\sum_i a_i^2} \times \sqrt{\sum_i b_i^2}} \\
&= \frac{a \cdot b}{\parallel a \parallel \times \parallel b \parallel}
\end{align}

Source: <https://en.wikipedia.org/wiki/Cosine_similarity>

## Can we find out 「章魚里」 based on 2020 presidential data？

Definitely, we all have access to the [Central Election Commission](https://db.cec.gov.tw/).

## We've downloaded these spreadsheets and prepared an in-browser environment for you.

[](https://mybinder.org/v2/gh/yaojenkuo/talks/HEAD)

## We can access to a Python notebook, R notebook, or RStudio in browser, no strings attached.

## Besides a few kernels to execute, we also attached some data.

- Reading a CSV file.
- Reading a Excel spreadsheet.

```python
# reading data via Python's pandas library
csv_df = pd.read_csv('presidential_2020.csv')
excel_df = pd.read_excel('presidential-2020/總統-A05-4-候選人得票數一覽表-各投開票所(南投縣).xls', skiprows=[0, 1, 3, 4])
```

## Our CSV file is an integrated file after manipulations

```python
csv_df.head()
```

      county
      town
      village
      office
      number
      candidate
      votes

      0
      宜蘭縣
      宜蘭市
      民族里
      1
      1
      宋楚瑜/余湘
      37

      1
      宜蘭縣
      宜蘭市
      民族里
      2
      1
      宋楚瑜/余湘
      31

      2
      宜蘭縣
      宜蘭市
      建軍里
      3
      1
      宋楚瑜/余湘
      19

      3
      宜蘭縣
      宜蘭市
      建軍里
      4
      1
      宋楚瑜/余湘
      29

      4
      宜蘭縣
      宜蘭市
      泰山里
      5
      1
      宋楚瑜/余湘
      25

## Our Excel spreadsheets are the original files downloaded from [Central Election Commission](https://db.cec.gov.tw/)

```python
excel_df.head()
```

      Unnamed: 0
      Unnamed: 1
      Unnamed: 2
      (1)\n宋楚瑜\n余湘
      (2)\n韓國瑜\n張善政
      (3)\n蔡英文\n賴清德
      Unnamed: 6
      Unnamed: 7
      Unnamed: 8
      Unnamed: 9
      Unnamed: 10
      Unnamed: 11
      Unnamed: 12
      Unnamed: 13

      0
      總　計
      NaN
      NaN
      13,315
      133,791
      152,046
      299,152
      3,555
      302,707
      13
      302,720
      110,765
      413,485
      73.2087

      1
      南投市
      NaN
      NaN
      3,077
      26,690
      30,910
      60,677
      693
      61,370
      3
      61,373
      20,480
      81,853
      74.9759

      2
      NaN
      龍泉里
      1.0
      26
      241
      391
      658
      8
      666
      0
      666
      228
      894
      74.4966

      3
      NaN
      康壽里
      2.0
      30
      216
      266
      512
      4
      516
      0
      516
      128
      644
      80.1242

      4
      NaN
      康壽里
      3.0
      25
      239
      306
      570
      8
      578
      0
      578
      154
      732
      78.9617

## We can also try importing via the RStudio interface.

```r
library(readxl)

csv_df = read.csv('presidential_2020.csv')
excel_df = read_excel('presidential-2020/總統-A05-4-候選人得票數一覽表-各投開票所(南投縣).xls')
head(csv_df)
head(excel_df)
```

## We write codes to integrate these spreadsheets into a CSV file

```python
from presidential import Presidential

presidential = Presidential('presidential-2020')
presidential_df = presidential.adjust_presidential_df()
presidential_df.to_csv('presidential_2020.csv', index=False)
```

    Tidying 總統-A05-4-候選人得票數一覽表-各投開票所(宜蘭縣).xls
    Tidying 總統-A05-4-候選人得票數一覽表-各投開票所(彰化縣).xls
    Tidying 總統-A05-4-候選人得票數一覽表-各投開票所(金門縣).xls
    Tidying 總統-A05-4-候選人得票數一覽表-各投開票所(桃園市).xls
    Tidying 總統-A05-4-候選人得票數一覽表-各投開票所(苗栗縣).xls
    Tidying 總統-A05-4-候選人得票數一覽表-各投開票所(臺南市).xls
    Tidying 總統-A05-4-候選人得票數一覽表-各投開票所(雲林縣).xls
    Tidying 總統-A05-4-候選人得票數一覽表-各投開票所(南投縣).xls
    Tidying 總統-A05-4-候選人得票數一覽表-各投開票所(高雄市).xls
    Tidying 總統-A05-4-候選人得票數一覽表-各投開票所(臺北市).xls
    Tidying 總統-A05-4-候選人得票數一覽表-各投開票所(新北市).xls
    Tidying 總統-A05-4-候選人得票數一覽表-各投開票所(花蓮縣).xls
    Tidying 總統-A05-4-候選人得票數一覽表-各投開票所(新竹市).xls
    Tidying 總統-A05-4-候選人得票數一覽表-各投開票所(新竹縣).xls
    Tidying 總統-A05-4-候選人得票數一覽表-各投開票所(基隆市).xls
    Tidying 總統-A05-4-候選人得票數一覽表-各投開票所(連江縣).xls
    Tidying 總統-A05-4-候選人得票數一覽表-各投開票所(嘉義縣).xls
    Tidying 總統-A05-4-候選人得票數一覽表-各投開票所(嘉義市).xls
    Tidying 總統-A05-4-候選人得票數一覽表-各投開票所(屏東縣).xls
    Tidying 總統-A05-4-候選人得票數一覽表-各投開票所(澎湖縣).xls
    Tidying 總統-A05-4-候選人得票數一覽表-各投開票所(臺東縣).xls
    Tidying 總統-A05-4-候選人得票數一覽表-各投開票所(臺中市).xls

```python
presidential_df.head()
```

      county
      town
      village
      office
      number
      candidate
      votes

      0
      宜蘭縣
      宜蘭市
      民族里
      1
      1
      宋楚瑜/余湘
      37

      1
      宜蘭縣
      宜蘭市
      民族里
      2
      1
      宋楚瑜/余湘
      31

      2
      宜蘭縣
      宜蘭市
      建軍里
      3
      1
      宋楚瑜/余湘
      19

      3
      宜蘭縣
      宜蘭市
      建軍里
      4
      1
      宋楚瑜/余湘
      29

      4
      宜蘭縣
      宜蘭市
      泰山里
      5
      1
      宋楚瑜/余湘
      25

```python
presidential_df.tail()
```

      county
      town
      village
      office
      number
      candidate
      votes

      51673
      臺中市
      和平區
      梨山里
      1845
      3
      蔡英文/賴清德
      132

      51674
      臺中市
      和平區
      梨山里
      1846
      3
      蔡英文/賴清德
      107

      51675
      臺中市
      和平區
      梨山里
      1847
      3
      蔡英文/賴清德
      40

      51676
      臺中市
      和平區
      平等里
      1848
      3
      蔡英文/賴清德
      24

      51677
      臺中市
      和平區
      平等里
      1849
      3
      蔡英文/賴清德
      102

## Check if the summations are right with Python

```python
ttl_votes = presidential_df['votes'].sum()
ttl_votes_by_candidates = presidential_df.groupby('number')['votes'].sum()
ttl_votes_by_candidates
```

    number
    1     608590
    2    5522119
    3    8170231
    Name: votes, dtype: int64

## Check if the summations are right with R

```r
library(dplyr)

csv_df %>%
    group_by(number) %>%
    summarise(ttl_votes = sum(votes))
```

## National percentage is our target vector to be compared

```python
national_percentage = ttl_votes_by_candidates / ttl_votes
national_percentage
```

    number
    1    0.042556
    2    0.386137
    3    0.571307
    Name: votes, dtype: float64

## Total votes for each village

```python
combined_key = presidential_df['county'].str.cat(presidential_df['town']).str.cat(presidential_df['village'])
presidential_df = presidential_df.assign(combined_key=combined_key)
ttl_votes_by_combined_key = presidential_df.groupby(['combined_key'])['votes'].sum()
ttl_votes_by_combined_key
```

    combined_key
    南投縣中寮鄉中寮村      443
    南投縣中寮鄉內城村      297
    南投縣中寮鄉八仙村      535
    南投縣中寮鄉和興村      422
    南投縣中寮鄉崁頂村      304
                 ...
    高雄市鼓山區鼓岩里      847
    高雄市鼓山區鼓峰里     1425
    高雄市鼓山區龍井里      906
    高雄市鼓山區龍子里    11410
    高雄市鼓山區龍水里    16333
    Name: votes, Length: 7737, dtype: int64

## Votes percentage by each candidate and village

```python
ttl_votes_by_combined_key_candidates = presidential_df.groupby(['combined_key', 'number'])['votes'].sum()
soong = ttl_votes_by_combined_key_candidates[:, '1'] / ttl_votes_by_combined_key
han = ttl_votes_by_combined_key_candidates[:, '2'] / ttl_votes_by_combined_key
tsai = ttl_votes_by_combined_key_candidates[:, '3'] / ttl_votes_by_combined_key
votes_obtained = pd.concat([soong, han, tsai], axis=1)
votes_obtained.columns = ['soong', 'han', 'tsai']
```

```python
votes_obtained
```

      soong
      han
      tsai

      combined_key

      南投縣中寮鄉中寮村
      0.040632
      0.489842
      0.469526

      南投縣中寮鄉內城村
      0.057239
      0.474747
      0.468013

      南投縣中寮鄉八仙村
      0.039252
      0.435514
      0.525234

      南投縣中寮鄉和興村
      0.021327
      0.500000
      0.478673

      南投縣中寮鄉崁頂村
      0.052632
      0.381579
      0.565789

      ...
      ...
      ...
      ...

      高雄市鼓山區鼓岩里
      0.014168
      0.309327
      0.676505

      高雄市鼓山區鼓峰里
      0.032982
      0.473684
      0.493333

      高雄市鼓山區龍井里
      0.023179
      0.367550
      0.609272

      高雄市鼓山區龍子里
      0.032340
      0.381420
      0.586240

      高雄市鼓山區龍水里
      0.037654
      0.398947
      0.563399

7737 rows × 3 columns

## Calculate cosine similarity

```python
a = national_percentage.values
a_norm = np.linalg.norm(a)
cos_similarities = []
for i in range(votes_obtained.shape[0]):
    b = votes_obtained.iloc[i, :].values
    b_norm = np.linalg.norm(b)
    ab = np.dot(a, b)
    cos_similarity = np.dot(a, b) / (a_norm*b_norm)
    cos_similarities.append(cos_similarity)
votes_obtained = votes_obtained.assign(cosine_similarity=cos_similarities)
votes_obtained = votes_obtained.reset_index()
```

```python
votes_obtained.head()
```

      combined_key
      soong
      han
      tsai
      cosine_similarity

      0
      南投縣中寮鄉中寮村
      0.040632
      0.489842
      0.469526
      0.977648

      1
      南投縣中寮鄉內城村
      0.057239
      0.474747
      0.468013
      0.980246

      2
      南投縣中寮鄉八仙村
      0.039252
      0.435514
      0.525234
      0.995217

      3
      南投縣中寮鄉和興村
      0.021327
      0.500000
      0.478673
      0.977015

      4
      南投縣中寮鄉崁頂村
      0.052632
      0.381579
      0.565789
      0.999882

## Sort by cosine similarity with descending order to find 「章魚里」

```python
votes_obtained.sort_values(['cosine_similarity', 'combined_key'], ascending=[False, True]).reset_index(drop=True).head(10)
```

      combined_key
      soong
      han
      tsai
      cosine_similarity

      0
      嘉義縣番路鄉內甕村
      0.042553
      0.386018
      0.571429
      1.000000

      1
      臺南市東區關聖里
      0.042450
      0.386295
      0.571255
      1.000000

      2
      臺南市中西區南門里
      0.043410
      0.385460
      0.571130
      0.999999

      3
      新北市汐止區保長里
      0.042833
      0.386847
      0.570320
      0.999999

      4
      新北市金山區五湖里
      0.043765
      0.385632
      0.570603
      0.999998

      5
      臺北市南港區東新里
      0.042036
      0.385298
      0.572666
      0.999997

      6
      臺北市內湖區西湖里
      0.041285
      0.386008
      0.572707
      0.999997

      7
      新北市中和區清穗里
      0.041865
      0.385347
      0.572788
      0.999997

      8
      臺南市北區重興里
      0.042837
      0.384831
      0.572331
      0.999997

      9
      新北市板橋區景星里
      0.042515
      0.387607
      0.569878
      0.999996

## Can we find the similarity of our own village?

Definitely.

```python
def find_my_village(my_village, df):
    df = df.sort_values(['cosine_similarity', 'combined_key'], ascending=[False, True]).reset_index(drop=True)
    my_village_df = df[df['combined_key'] == my_village]
    return my_village_df
```

```python
my_village = '高雄市鼓山區桃源里'
my_village_df = find_my_village(my_village, votes_obtained)
my_village_similarity = my_village_df['cosine_similarity'].values[0]
my_village_rank = my_village_df.index[0]
n_rows = votes_obtained.shape[0]
print("{}的餘弦相似度為{:.4f}, 排名{}/{}".format(my_village, my_village_similarity, my_village_rank, n_rows))
my_village_df
```

    高雄市鼓山區桃源里的餘弦相似度為0.9985, 排名1714/7737

      combined_key
      soong
      han
      tsai
      cosine_similarity

      1714
      高雄市鼓山區桃源里
      0.023419
      0.370023
      0.606557
      0.998506

## Feeling movivated?

## Start with the most practical one: Python

- Procedural programming with Python
- Object-oriented programming with Python
- Using Python libraries

## Start with the most practical one: R

- Procedural programming with R
- Functional programming with R
- Using R libraries

## Resources I've used when learning Python

- [Introducing Python](https://www.amazon.com/Introducing-Python-Modern-Computing-Packages/dp/1449359361)
- [A Whirlwind Tour of Python](https://jakevdp.github.io/WhirlwindTourOfPython/index.html)
- [Python Data Science Handbook](https://jakevdp.github.io/PythonDataScienceHandbook/)

## Resources I've used when learning R

- [The Art of R Programming](https://www.amazon.com/Art-Programming-Statistical-Software-Design/dp/1593273843)
- [Advanced R](https://adv-r.hadley.nz/)
- [R for Data Science](https://r4ds.had.co.nz/)
- [Data Science Specialization](https://www.coursera.org/specializations/jhu-data-science)
- [Statistics with R Specialization](https://www.coursera.org/specializations/statistics)

## Learning resources from me

- [數據交點](https://www.datainpoint.com)
- [Substack](https://datainpoint.substack.com/about)

## Phew, that is a lot to catch up...

You do not have to finish every course or book from end to end.

Source: <https://giphy.com/>


# Python 预缓存的 Property 修饰器简单实现

> 创建时间：2021-03-19

这份简短笔记，我们会讨论预缓存 Python Property (类属性) 的简单实现。

在运行程序时，特别是对于耗时但不耗内存、以后需要经常取用的计算，我们会希望找个内存或硬盘空间储存起来。为了方便起见，我们只讨论借用内存的方法。

假设现在的问题是，我们要计算 $c = a+b, d = a^2$。为了调用的便利，$a, b$ 两个变量 (作为常数) 作为 property。

但麻烦之处在于，$a, b$ 的值并不容易求，求完之后对内存的消耗却又不大。(嘛先不要追问为什么求个 1+1 要这么复杂)

$$
\begin{align}
a &= \sum_{n = 1}^{\infty} \frac{1}{2^n} \simeq \sum_{n = 1}^{1000} \frac{1}{2^n} \\
b &= \int_0^1 3 x^2 \, \mathrm{d} x \simeq \sum_{n = 0}^{5000} \frac{3 n^2}{5000^3}
\end{align}
$$

同时，我们也不清楚末端用户是否需要求 $c$ (需要同时计算 $a$, $b$)，还是需要求 $d$ (只需要计算 $a$ 即可)。

```python
def get_a():
    a = 0
    for n in range(1, 1001):
        a += 1 / 2**n
    return a
```

```python
def get_b():
    b = 0
    for n in range(5001):
        b += 3 * n**2 / 5000**3
    return b
```

这篇文档讨论四种做法。第一种做法简单但低效；第二、三种做法代码较复杂；第四种代码简单且不会产生多余的计算。作者倾向使用 [第三种](#偷懒的做法：将赋值函数嵌入-getter-函数) 与 [第四种](#改进的做法：缩减隐含变量的声明) 做法。

## 即时调用 property 定义方法

最简单粗暴的方法是需要 $a, b$ 时就现场计算。在第一次调用 $a, b$ 时固然需要耗时的计算，但第二次调用仍然会相当费时。

```python
class Dummy:

    @property
    def a(self):
        return get_a()

    @property
    def b(self):
        return get_b()

    @property
    def c(self):
        return self.a + self.b

    @property
    def d(self):
        return self.a**2
```

```python
dum = Dummy()
dum.c
```

    2.00030002

```python
%%timeit -n 50
dum.c
```

    2.41 ms ± 172 µs per loop (mean ± std. dev. of 7 runs, 50 loops each)

## 一般的 property 做法

为了避免多余的计算，一般的方法是，需要首先在 `__init__` 中声明两个隐含变量 `_a`, `_b` 以保存结果。在使用 `a`, `b` 两个 property 之前，先要使用 setter 函数作 $a, b$ 的计算并分别保存到 `_a`, `_b` 中；随后再用 getter 函数调用它们。

```python
class General:

    def __init__(self):
        self._a = NotImplemented
        self._b = NotImplemented

    @property
    def a(self):
        return self._a

    @a.setter
    def a(self, val):
        self._a = val

    @property
    def b(self):
        return self._a

    @b.setter
    def b(self, val):
        self._b = val

    @property
    def c(self):
        return self.a + self.b

    @property
    def d(self):
        return self.a**2
```

如果没有预先使用 setter 函数，就会碰到下面这种尴尬的情况：

```python
gen = General()
gen.c
```

因此，正确的调用方式是

```python
gen = General()
gen.a, gen.b = get_a(), get_b()
gen.c, gen.d
```

    (2.0, 1.0)

上面的步骤是耗时的，但随后当要调用 `a`, `b` 变量时，就会快捷很多：

```python
%%timeit -n 50
gen.c
```

    265 ns ± 28 ns per loop (mean ± std. dev. of 7 runs, 50 loops each)

但这里有一个问题：$a, b$ 的值实际上可以看作常数；如果末端用户真正希望得到的是 $d = a^2$ 而非 $c = a + b$，那么实际上用户不需要 $b$，自然也就不需要对其花时间赋值了。决定是否要对 $b$ 赋值的任务由此交给末端用户，这会造成一些困扰。

## 偷懒的做法：将赋值函数嵌入 getter 函数

如果这个任务交给程序编写者，那么一种最简单的实现方式是把赋值函数嵌入到 getter 函数中：

- 如果 $a$ 的值已经被计算过，那么就从缓存空间 `_a` 取出该值；

- 如果 $a$ 被调用前没有被计算过，那么就计算该值并放入缓存 `_a`。

```python
class Improved:

    def __init__(self):
        self._a = NotImplemented
        self._b = NotImplemented

    @property
    def a(self):
        if self._a is NotImplemented:
            self._a = get_a()
        return self._a

    @property
    def b(self):
        if self._b is NotImplemented:
            self._b = get_b()
        return self._b

    @property
    def c(self):
        return self.a + self.b

    @property
    def d(self):
        return self.a**2
```

如果末端用户只需要求 $d = a^2$，那么耗费时间的关键步就只有 $a$ 的计算；缓存空间 `_b` 就是空的。

```python
imp = Improved()
print(imp.d)
print(imp._b)
```

    1.0
    NotImplemented

同时，以后再需要调用 $d$ 时，$a$ 的值也不会再被计算第二次。

当然，这种做法的弊端是，用户原则上无权限更改 $a, b$ 的值 (通过更改隐含变量 `_a`, `_b` 是可能的，但这违背了 PEP8 的程序规范)。

## 改进的做法：缩减隐含变量的声明

但上面的定义仍然有很多冗余。对于每个 property，我们总要声明隐含变量、调用时判断是否缓存空间存在。这两步可以通过改编在 property 修饰器内部增加一段代码方便地实现。这个修饰器我们命名为 `cached_property`：

```python
def cached_property(f):
    def wrap(*args, **kwargs):
        self = args[0]                                                    # self
        _f = "_" + f.__name__                                             # _a
        if not hasattr(self, _f) or getattr(self, _f) is NotImplemented:  # if self._a is NotImplemented:
            setattr(self, _f, f(*args))                                   # self._a = get_a()
        return getattr(self, _f)                                          # return self._a
    return property(wrap)                                                 # make this wrap a property
```

这样之后，不仅代码量减少很多 (调用方式与最简单的 `Dummy` 完全一致)，同时也避免多余重复的计算。

```python
class Advanced:

    @cached_property
    def a(self):
        return get_a()

    @cached_property
    def b(self):
        return get_b()

    @property
    def c(self):
        return self.a + self.b

    @property
    def d(self):
        return self.a**2
```

```python
adv = Advanced()
print(adv.d)
print(hasattr(adv, "_b"))
```

    1.0
    False


# FastEMRIWaveforms Tutorial
## ICERM Workshop: Waveform acceleration with machine learning and GPUs

### Michael Katz, Max Planck Institute for Gravitational Physics (Albert Einstein Institute), Lead developer for FastEMRIWaveforms

In this tutorial, you will learn the basics of building an accelereated EMRI waveform. We encourage participants to see our paper ([arxiv.org/2008.06071](https://arxiv.org/abs/2008.06071)) and the FastEMRIWaveforms [package documentation](https://bhptoolkit.org/FastEMRIWaveforms/) for more information, as well as our forthcoming paper that will describe the waveform in much more detail than our PRL submission.

Import packages:

```python
import matplotlib.pyplot as plt
%matplotlib inline
import numpy as np

import h5py

from few.amplitude import romannet
from few.utils.utility import check_for_file_download, p_to_y
from few.trajectory.flux import RunSchwarzEccFluxInspiral
from few.amplitude.romannet import RomanAmplitude
from few.amplitude.interp2dcubicspline import Interp2DAmplitude
from few.waveform import FastSchwarzschildEccentricFlux
```

    Matplotlib is building the font cache; this may take a moment.

## A Quick Introduction to FastEMRIWaveforms

### Collaborators: Alvin Chua, Niels Warburton, Scott Hughes, Lorenzo Speri

```python
generate = FastSchwarzschildEccentricFlux()

M = 1e6  # large mass
mu = 1e1  # small mass
p0 = 12.0  # separation
e0 = 0.4  # eccentricity
theta = np.pi/3.  # polar viewing angle
phi = np.pi/4.  # azimuthal viewing angle
dist = 1.0  # distance in Gpc

T = 1/365.  # in years
dt = 10.0  # time spacing of data stream

wave = generate(M, mu, p0, e0, theta, phi, dist, T=T, dt=dt)

plt.plot(wave.real)
plt.plot(wave.imag)
```

FastEMRIWaveforms is the first fully relativistic template generation tool for extreme mass ratio inspirals. We think of it more as a framework than any specific waveform. In this tutorial, we will discuss the main pieces of this framework and how they relate to our first fully relativistic waveform model shown above.

As a basic primer, our five key points we want to achieve with this framework are:

* Accuracy: Our fast waveforms must be accurate when compared to slow and accurate waveforms generated by the waveform modeling community.

* Modularity: These waveforms are to be built out of a set of modules. These modules are to be easily interchangeable, as well as stand alone tools available for more in-depth analysis.

* Flexibility: This framework must be easily adaptable to new computational methods and/or improvements in EMRI physics.

* Easy User Interface: All modules and complete waveforms are to have a front-facing python interface with clear and extensive documentation, as well as many examples.

* Parallelization: These waveforms must take advantage of parallelization techniques such as OpenMP, as well as accelerator hardware such as GPUs.

## Basics of an EMRI Waveform

The guiding equation for building an EMRI waveform is given by

\begin{equation}
h_+-ih_x = \frac{1}{r}\sum_{lmkn}\left(-\frac{Z_{lmkn}}{\omega_{mkn}^2}\right)\left(S_{lmkn}(\theta)e^{-im\phi}\right)e^{i\Phi_{mkn}} = \frac{1}{r}\sum_{lmkn}A_{lmkn}\Theta_{lmkn}e^{i\Phi_{mkn}}.
\end{equation}

Here we are concerned with generating fast and accurate waveforms. These waveforms are created with a sequence of modules. We will discuss the three main modules used to produce these waveforms. The first module is the Trajectory module which takes initial parameters and produces arrays for all of the evolving quantities of concern. These include the phase evolution, {$\Phi_\varphi,\Phi_\theta, \Phi_r$}, from the start to the end of the EMRI orbit. With these quantities, we can produce:

\begin{equation}
\Phi_{mkn} = m\Phi_\varphi + k\Phi_\theta + n\Phi_r.
\end{equation}

The trajectory also produces orbital quantities over time. These include the separation ($p$), eccentricity ($e$), and the inclination angle of the orbit ($\iota$). With these arrays containing the orbital evolution of these values, we generate the amplitudes, $A_{lmkn}$. While the evolution is over time, the various phasing and amplitude computations are produced within a frequency decomposition into $(l,m,k,n)$ modes.

With phases and amplitdues in hand, we combine this with the angular harmonic information ($\Theta_{lmkn}$) to produce the final waveform. In this step, we calculate the sum of all modes at each time point in the template data stream.

## Overall Waveform Strategy

* We calculate our trajectories and amplitudes as sparse arrays in time by using a large adaptive stepping integrator. This produces arrays with $\sim100$ points. All of these quantities vary slowly and smoothly. After these calculations are complete, we scale this up to the actual data stream cadence.

* The speed of EMRI waveforms is strongly determined by the amount of harmonic content. Higher eccentricities require more modes to produce a high fidelity waveform. In order to make our waveforms as efficient as possible, we perform an online mode content calculation that removes modes from consideration if they do not contribute to the waveform power determined by a user-defined threshold.

## Current Waveform Model: Schwarzschild Eccentric

* No $k$ modes
* Orbit is equatorial
* $S_{lmkn}(\theta)e^{-im\phi}$ reduces to $_{-2}Y_{lm}(\theta,\phi)$
* $l:\{2,10\}$, $m:\{-l,l\}$, $n:\{-30,30\}$ $\rightarrow$ 3843 modes.

## Fast Trajectories: $\{p, e, \Phi_\varphi, \Phi_r\}$

We are not going to spend too much time on the trajectory part. However, we need to generate it in order to build the rest of our waveform. To build the trajectory, we integrate with large steps using an RK8 integrator.

```python
traj = RunSchwarzEccFluxInspiral()
```

```python
p0 = 16.0  # initial separation
e0 = 0.4  # initial eccentricity
mu = 180.  # iniital small mass in solar masses, produces approximately 1 yr waveform
M = 1e6   # initial large mass in solar masses
dt = 10.0 # sets initial step size
T = 1.0 # in years

t, p, e, Phi_phi, Phi_r, flux = traj(M, mu, p0, e0, T=T, dt=dt)
print("length:", len(t), "duration:", t[-1])
```

    length: 122 duration: 29164387.553433623

```python
fig, axes = plt.subplots(2, 3)
plt.subplots_adjust(wspace=0.3)
fig.set_size_inches(14, 8)
axes = axes.ravel()

ylabels = [r'$e$', r'$p$', r'$e$', r'$\Phi_\phi$', r'$\Phi_r$', r'Flux']
xlabels = [r'$p$', r'$t$', r'$t$', r'$t$', r'$t$', r'$t$', r'$t$', r'$t$']
ys = [e, p, e, Phi_phi, Phi_r, flux]
xs = [p, t, t, t, t, t]

for i, (ax, x, y, xlab, ylab) in enumerate(zip(axes, xs, ys, xlabels, ylabels)):
    ax.plot(x, y, lw=0.5)
    ax.scatter(x, y, s=5)
    ax.set_xlabel(xlab, fontsize=16)
    ax.set_ylabel(ylab, fontsize=16)
```

## RomanNet Amplitudes: $A_{lmn}$

To generate the amplitudes, we use a RomanNet ([arXiv:1811.05491](https://arxiv.org/abs/1811.05491)). Roman stands for Reduced Order Modelling with Artificial Neurons. When training a neural network, it can be crucial to hand it data that is distilled to highlight the most pertinent information. A nice tool for doing this is Reduced Order Modeling. Reduced order modelling projects the information with lossless compression down to a lower dimensional space. We take our complex mode amplitude vectors containing 3843 modes and project this down to a real-valued space with 198 values. The neural network is then trained with inputs given by the $p$ and $e$ values and outputs given by the reduced order coefficients ($\alpha_i$):

\begin{equation}
A_{lmn}\in\mathbb{C}^{3843}\xrightarrow{\mathit{ROM}}\alpha_i\in\mathbb{R}^{198}\xrightarrow{\mathit{train}} f(p,e)=\alpha_i
\end{equation}

The neural network itself is extremely simple. It is a fully connected network with a LeakyReLU activation on all layers but the final layer. This means the neural network can be built simply with a sequence of linear matrix multiplications followed by a pass through the activation function. Once the neural network is trained, we will have a set of weights. This is where we will start in this tutorial.

During online evaluation of the waveform, we perform the reverse process:

\begin{equation}
f(p,e)\xrightarrow{\mathit{eval}}\alpha_i\in\mathbb{R}^{198}\xrightarrow{\mathit{project}}A_{lmn}\in\mathbb{C}^{3843}
\end{equation}

This method has pros and cons:

Pros:

* This is more of a global fit, rather than individual fits to given modes. This generally means storage of less information in memory. If we do individual fits of every mode, the memory necessary to store this information would scale badly with mode content. We are currently working in Schwarzschild Eccentric. As we go to generic Kerr, we expect the number of modes to increase by a factor of $\sim10$.

* Due to its global fit nature, this method is generally faster to evaluate than individual interpolants.

* Since it is a neural network and a linear projection, this is very suitable to GPUs.

* We expect this method, or methods similar to this, to scale better with dimensionality as we move towards the end goal of generic Kerr orbits.

Cons:

* For extremely quiet modes, this method can be less accurate. However, as these modes are quiet, this does not result in a significant loss of accuracy in the final waveform. Caution must be taken when using this method to analyze individual mode amplitudes. There is an approximate floor in the amplitude values at $\sim10^{-5}$. For reference, the loudest modes at a given $p$ and $e$ value are usually $\sim0.1-1.0$. (These amplitudes are not scaled for distance.|)

* Training these neural networks can be more of an art than a science. It takes a lot of trial and error to get this right.

### Constructing the neural network from trained weights

```python

# prepare to load the weights

path_to_few_dir = romannet.__file__[:-25]

weight_file = fp = "SchwarzschildEccentricInput.hdf5"
check_for_file_download(fp, path_to_few_dir)

weights = []
bias = []
dim1 = []
dim2 = []

# get highest layer number
num_layers = 0

# extract all necessary information from the file
with h5py.File(path_to_few_dir + "few/files/" + weight_file, "r") as fp:

    # get basic information
    num_teuk_modes = fp.attrs["num_teuk_modes"]
    transform_factor = fp.attrs["transform_factor"]
    break_index = fp.attrs["break_index"]

    # determine layer arrangement
    for key, value in fp.items():
        if key == "reduced_basis":
            continue

        layer_num = int(key[1:])

        if layer_num > num_layers:
            num_layers = layer_num

    # get weights and bias
    for i in range(1, num_layers + 1):
        temp = {}
        for let in ["w", "b"]:
            mat = fp.get(let + str(i))[:]
            temp[let] = np.asarray(mat)

        weights.append(temp["w"])
        bias.append(temp["b"])
        dim1.append(temp["w"].shape[0])
        dim2.append(temp["w"].shape[1])

    # get the post network transform matrix
    transform_matrix = np.asarray(fp["reduced_basis"])

# activation function
# we use a factor of 0.2 for negative values
def LeakyReLU(x):
    out = (x >= 0.0) * x +  (x < 0.0) * 0.2*x
    return out

# build the neural network
def RomanNet(p, e):

    p = np.atleast_1d(p)
    e = np.atleast_1d(e)

    # convert from the p coordinate to a special y coordinate
    # see the documentation for more details
    y = p_to_y(p, e)

    # prepare input
    x = np.array([y, e])

    # basic fully connected network
    for layer_i in range(num_layers):

        # linear transformation
        x = np.dot(weights[layer_i].T, x) + bias[layer_i][:, np.newaxis]

        # do not want to activate last layer
        if layer_i < num_layers - 1:
            # non-linear activatation
            x = LeakyReLU(x)

    # separate real and imaginary
    x = x[:break_index] + 1j * x[break_index:]

    # project back to amplitude basis
    out = np.dot(transform_matrix.T, x)/transform_factor
    return out.T

```

```python
# test it
p_test = np.array([11.0, 10.0, 10.0])
e_test = np.array([0.2, 0.3, 0.1])

RomanNet(p_test, e_test)
```

    array([[ 6.61875897e-06-9.45689250e-06j,  6.69607160e-06-1.13351771e-05j,
             6.41141133e-06-1.40002164e-05j, ...,
             7.50732732e-05+2.44753603e-05j,  6.73201363e-05-2.51759069e-06j,
            -2.62936078e-07-1.61971784e-05j],
           [-3.03984515e-05+2.23504926e-06j, -3.46638688e-05+2.11040541e-06j,
            -3.99084181e-05+1.67172621e-06j, ...,
             8.48364501e-05-3.51356321e-05j,  1.27022103e-04-6.69149191e-06j,
             9.19366706e-05+1.92716832e-05j],
           [-1.78544871e-06+1.54745952e-05j, -1.96676157e-06+1.75539377e-05j,
             7.20295633e-07+2.05609755e-05j, ...,
            -1.38437626e-05+1.46714870e-05j, -2.75508914e-05+9.37792116e-06j,
             1.12870278e-05+7.38750376e-06j]])

### Produce amplitudes associated with our trajectories

```python
# generate amplitudes with roman net
amps = RomanNet(p, e)

# check against actual code
RomanNetTrue = RomanAmplitude()
amps_check_1 = RomanNetTrue(p, e)

assert np.allclose(amps, amps_check_1)
```

### Check our RomanNet global fit against accurate values

```python
# get accurate values
# each mode is fitted with a bicubic spline
BicubicAmps = Interp2DAmplitude()
amps_check_2 = BicubicAmps(p, e)
```

#### Print vectors

```python
print("romannet:", amps_check_1[0][0:10], "\nBicubic Spline:", amps_check_2[0][0:10])
```

    romannet: [7.69192924e-06-5.39154229e-06j 9.99753291e-06-6.31307778e-06j
     1.35171826e-05-8.06391248e-06j 1.78239290e-05-7.33261441e-06j
     2.14797124e-05-6.31329971e-06j 2.57309168e-05-3.37761067e-06j
     2.89725596e-05+1.17404671e-06j 3.27106651e-05+7.79482012e-06j
     3.64016950e-05+1.64240791e-05j 3.63310870e-05+2.77218824e-05j]
    Bicubic Spline: [0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j]

We see if we take a quick examination of our results (a small subset) we see the problem with the global fit. It cannot handle modes with a very small amplitude. Therefore, you might naively think that this method might not work.

#### Compare via the cosine between the vectors

```python
cos = (np.dot(amps_check_2.conj()[0].T, amps_check_1[0]) /
       np.sqrt(np.dot(amps_check_2.conj()[0], amps_check_2[0])
               * np.dot(amps_check_1.conj()[0], amps_check_1[0]))).real

print('Cos:', cos)
```

    Cos: 0.9999733790390088

We now see that if we compare the results as a whole, the results match very well. If we look at individual modes that have high power, they are likely to strongly match when the bicubic spline is compared to the RomanNet method.

## GPU-accelerated Waveform Build: $\sum_{lmn}$

### Quick primer on GPUs

GPUs run code in parallel in a configuration of grids, blocks, and threads.

* **Threads** are the the actual software units that run the code. Threads run independently of one another. Threads are referenced in code using `threadIdx.x`.
* **Blocks** are an array of threads. This array can be 1, 2, or 3 dimensions. In more than one dimension, thread indices are referenced using `x`, `y`, and `z` (e.g. `threadIdx.z`). Blocks, similar to threads, are referenced using `blockIdx.x`, `blockIdx.y`, or `blockIdx.z`. The size of a block, i.e. the number of threads along a given dimension is given as `blockDim.x`. **Note**: In applications I have worked on, I rarely ever use more than 1 dimension of threads.
* **Grids** are an array of blocks. This array can be 1, 2, or 3 dimensions. There is no reference for grids as they efffectively represent the entire GPU kernel. The size of a grid, i.e. the number of blocks along a dimension, is determined with `gridDim.x`.

Organizing your code properly into blocks and threads is a key component of maximizing your efficiency.

Another equally important aspect to the maximization of efficiency is the proper use of the GPU memory structures. There are three main GPU memory structures to consider (there are a few more but they are for more specific uses). These are global memory, shared memory, and local memory.

* **Global memory** is contained on the host (or off-chip). Global memory is accessible by every block and thread in the entire grid. It contains many GBs of RAM. (The GPUs I use currently, which are Tesla V100s, have ~16GB. The new A100s have 40-80GB.) Because this memory is contained off-chip, it is slow to access. A key to accessing global memory efficiently is to use so-called "memory coalescence." This effectively means that neighboring threads access neighboring addresses in memory. This allows the compiler to make up to 32 memory reads at the same time, rather than 32 separate memory reads. We will see this in a simple case below.

* **Local memory** is also contained on the host. Local memory consists of any arrays allocated within the kernel specific to each thread. Therefore, this memory is only accessible by the thread it is created on. Usually, around 512 KB are available for local memory for each thread. Since, this memory is off-chip, it is also slow to access. Local memory is, however, always accessed in a memory coalescing manner.

* **Shared memory** is different. Shared memory is located on-chip. It is, therefore, much faster to read from (~100x faster). Shared memory is accessible by all the threads on a given block. The catch is that only ~48 KB are available for shared memory. Therefore, leveraging shared memory effectively is key to the efficiency of GPU code.

To sum up, there are two main points to consider when beginning to program on GPUs:

* Layout your grid effectively for your given problem.
* Use memory effectively: leverage the availability of shared memory and read from global memory in a coalescing fashion.

Some parting thoughts on GPUs:

* For maximal efficiency and stability, I usually code in C++/CUDA so that everything is precompiled and tested.
* There are python libraries that leverage the power of GPUs. Check out numba, CuPy, PyTorch, Tensorflow, PyCUDA.
* Lately, I have really focused on writing CPU/GPU agnostic code. This means that the source code is ~99% the same between the two. On the python side I usually sub in CuPy for Numpy. We will see some basic pointers on this below. In C++, I use short compiler directives to make minimal changes.
* Generally speaking it is optimal to store all quantities in a 1 dimensional array when working with GPUs. Let's say you have a two dimensional array that has dimensions (dim1, dim2) and is referenced with (i, j). You can turn this into a 1D array that references each value with (i * dim2 + j).

### Why are GPUs important for EMRI Waveforms?

The waveform summation is the key bottleneck. This is an operation that is uniquely suited to GPUs. You can see the improvement below.

### Basic example coded in python

We are going to write code in python that will reflect what we will do on the GPU. It will obviously still be on the CPU, but will give you a chance to see how to write some basic GPU code and understand how it works. **Make sure to read the comments in the code.**

We will do two examples.

#### Multiply two arrays

Here we will multiply two arrays as we would on the GPU.

```python
# setup our GPU quantities that would come in if we were actually on GPUs.

NUM_THREADS = 64  # needs to be a multiple of 32
blockDim = NUM_THREADS  # blockDim.x
```

```python

# this initial piece is a basic of CPU/GPU agnostic code in python
try:
    import cupy as xp

except ModuleNotFoundError:
    import numpy as xp

# __global__
def multiply_arrays(array_out, array1, array2, n):
    """
    // what this would like like in CUDA

    // if (threadIdx.x + blockDim.x * blockIdx.x >= n) return;
    for (int i = threadIdx.x + blockDim.x * blockIdx.x; i < n; i += blockDim.x * gridDim.x)
    {
        array_out[i] = array1[i] * array2[i];
    }
    """
    # The loops here are to simulate the GPU.
    # In reality the GPU will run all threads and blocks in parallel

    # begin simulation
    for block in range(num_blocks):
        for thread in range(NUM_THREADS):
    # end simulation

            # get the overall index in the grid
            # based on the thread and block
            i = thread + block * blockDim

            # since the GPU runs threads in parallel
            # we need to make sure the GPU does not index a value
            # on the last block that goes over the array length
            if i >= n:
                continue

            # use our index to get values out of the array
            # notice this simple statement uses memory coalescence
            # neighboring threads will have consecutive indices
            # therefore, they will access consecutive addresses in each array
            array_out[i] = array1[i] * array2[i]

length = int(2 ** 14)

# initialize arrays
array1 = xp.random.rand(length)
array2 = xp.random.rand(length)

# prepare output array
array_out = xp.zeros_like(array1)

# get the number of blocks
# the number of blocks multiplied by the number of threads per block
# must be greater than the length of the array
num_blocks = int(np.ceil((length + NUM_THREADS -1)/NUM_THREADS))

# this would actually be called as
"""
multiply_arrays<<<num_blocks, NUM_THREADS>>>(array_out, array1, array2, length);
cudaDeviceSynchronize();
"""

multiply_arrays(array_out, array1, array2, length)

# confirm it
assert np.allclose(array_out, array1 * array2)
```

#### Linear Interpolation to scale up an array

For our second example it will be a bit more complicated. This way we can see how to use shared memory.

Here we will scale up the size of an array using linear interpolation. The key here is that we need every block to read in the original array to shared memory. From there, we perform the interpolation. This allows us to achieve memory coalescence with all global reads and ensure that all reads when actually interpolating are done from shared memory.

```python
# in CUDA, when you statically allocate shared arrays,
# you cannot use a variable length
# therefore, in C++/CUDA you need to declare a max value up front

#define MAX_INPUT 1000

def linear_interpolation(array_out, array_in, n_out, n_in, dx_out, dx_in):

    # we want to read in to shared memory
    # want to do this with memory coalescence
    # since this is only on a specific block
    # we need to use only the threads on this block

    """
    // the below simulated code block in CUDA would look like this:

    // in CUDA, you have to declare shared arrays

    __shared__ double shared_array_in[MAX_INPUT];

    for (int i = threadIdx.x; i < n_in; i += blockDim.x)
    {
        // memory coalescence only needed from array_in (global memory)
        shared_array_in[i] = array_in[i];
    }

    // we need to make sure the threads all finish this operation before we move on
    // Therefore, we force the threads to synchronize

    __syncthreads();

    """

    # we will just do the following to simulate in python easily
    shared_array_in = xp.zeros(n_in)

    for thread in range(NUM_THREADS):
        i = thread

        # this while statement simulates the above commented loop
        while (i < n_in):
            # here is the memory coalesced reads
            shared_array_in[i] = array_in[i]

            # just for simulation
            i += blockDim

    """
    // this is what the below would really like it in CUDA
    // if (threadIdx.x + blockDim.x * blockIdx.x >= n_out) return;
    for (int i = threadIdx.x + blockDim.x * blockIdx.x; i < n_out; i += blockDim.x * gridDim.x)
    {
        // get the new out value assuming equal spacing
        double x_new = dx_out * i;

        // get index of the point in the original array below the new point
        int ind_in = (int) (x_new / dx_in);

        // get the below x value
        double x_old = ind_in * dx_in;

        // slope of segment
        double m = (shared_array_in[ind_in + 1] - shared_array_in[ind_in])/dx_in;

        // interpolate
        double new_value = m * (x_new - x_old) + shared_array_in[ind_in];
        array_out[i] = new_value;
    }

    """
    # The loops here are to simulate the GPU.
    # In reality the GPU will run all threads and blocks in parallel

    # begin simulation
    for block in range(num_blocks):
        for thread in range(NUM_THREADS):
    # end simulation

            # get the overall index in the grid
            # based on the thread and block
            i = thread + block * blockDim

            # since the GPU runs threads in parallel
            # we need to make sure the GPU does not index a value
            # on the last block that goes over the array length
            if i >= n_out:
                continue

            x_new = dx_out * i

            ind_in = int(x_new / dx_in)
            x_old = ind_in * dx_in

            m = (shared_array_in[ind_in + 1] - shared_array_in[ind_in])/dx_in

            new_value = m * (x_new - x_old) + shared_array_in[ind_in]
            array_out[i] = new_value

# original length
length_in = 100

# prepare input arrays
x_in = xp.arange(length_in)
y_in = (x_in ** 2).astype(xp.float64)

# set length out
length_out = int(2 ** 14)

# setup the new x_values
x_new = xp.linspace(x_in[0], x_in[-1], length_out + 1)[:-1]

# change in original x values
dx_in = 1.0

# we will actually only use the spacing to find new points
dx_out = x_new[1] - x_new[0]

y_out = xp.zeros(length_out)

num_blocks = int(np.ceil((length_out + NUM_THREADS -1)/NUM_THREADS))

# this would actually be called as
# linear_interpolation<<<num_blocks, NUM_THREADS>>>(y_out, y_in, length_out, length_in, dx_out, dx_in);
# cudaDeviceSynchronize();
linear_interpolation(y_out, y_in, length_out, length_in, dx_out, dx_in)

# confirm it
plt.plot(x_new, y_out, lw=6, label='out')
plt.plot(x_in, y_in, '.', label='in')
```

### Comment on CPU/GPU Agnostic code in C++/CUDA and Python

Making CPU/GPU agnostic code consists of 3 main parts:

* Sub CuPy for NumPy
* Compiler directives in C++/CUDA
* Easy transition from Python to C++ through an augmented Cython process

Above, we saw an example of how to deal with the Python side. The Cython functionality is effectively a decorator function that gets the pointer of a Numpy or Cupy array and sends that into the C++ code. Below is a basic example of how to use basic compiler directives to make your code more CPU/GPU agnostic.

**Note**: I generally argue that we should build codes for GPUs and then adapt them to CPUs, not the other way around which is the typical direction. In my experience, the CPU codes adapted from GPU codes are just as fast or within a small percentage of the speed of CPU designed codes. And, generally, it is much harder to optimize going from CPU->GPU rather than GPU->CPU.

## Future Plans

* Build fast trajectories in Kerr under the NIT framework
* Generate amplitudes in Kerr regime
* Further analyze how to determine best methods for mode content inclusion

We need your help! If anyone is interested in working on these types of issues, please let myself or any of my collaborators (listed above) know!


# Interpolating chemical equilibrium abundances

The atmospheric composition is determined by its elemental composition and the chemcical reactions that link these elements and chemical reactant species. In the atmosphere of an exoplanet, which is inherently 3-d and dynamic, mixing and advection processes complicate this picture, because the chemical abundances inferred from spectra are not necessarily representative of the pressures and temperatures probed locally by the observation. In addition, also the radiation field of the host star may influence the abundance and opacity structure of the atmosphere, by dissociating or ionizing chemical species or forming photochemical hazes. In this picture, using chemical equilibrium abundances, which means the steady state chemical composition of the atmosphere when treating its layers as isolated and independent (no advection/mixing/photochemistry), appears unwise.

Yet, calculating chemical equilibrium abundances may still serve as a useful starting point for initializing abundances in planetary atmospheres. In certain conditions, for example for planets hot enough such that mixing is negligible, chemical equilibrium abundances can even be sufficient to accurately describe the abundances in the atmospheric region probed by observations.

Here we show how to use our ``poor_mans_nonequ_chem()`` function. This function interpolates the abundances of the most important species from a big chemical equilibrium table as a function of pressure, temperature, metallicity [Fe/H] (where [Fe/H]=0 is solar) and cabon-to-oxygen number ratio (C/O). A C/O $\sim 0.55$ is the solar value, see, for example, [Asplund et al. (2009)](https://ui.adsabs.harvard.edu/abs/2009ARA%26A..47..481A/abstract).

**poor_mans_nonequ_chem() also contains a simple quenching implementation**. This means that a quench pressure can be specified above which the abundances of H$_2$O, CH$_4$ and CO are taken to be constant, following the reasoning in, for example, [Zahnle & Marley (2014)](https://ui.adsabs.harvard.edu/abs/2014ApJ...797...41Z/abstract).

The chemical grid has been calculated with easyCHEM, which is our self-written Gibbs free energy minimizer that is described in [Mollière et al. 2017](https://ui.adsabs.harvard.edu/abs/2017A%26A...600A..10M/abstract), and essentially a CEA clone, see [Gordon & McBride (1994)](https://ntrs.nasa.gov/citations/19950013764). easyCHEM has been benchmarked as the chemical equilibrium tool of petitCODE in [Baudino et al. (2017)](https://ui.adsabs.harvard.edu/abs/2017ApJ...850..150B/abstract). The grid dimensions are $T \in [60, 4000]~{\rm K}$, with 100 equidistant temperature points, $P \in [10^{-8}, 1000]~{\rm bar}$, with 100 pressure points spaced equidistantly in log space,  ${\rm C/O} \in [0.1, 1.6]$, with 20 equidistant points, and ${\rm [Fe/H]} \in [-2,3]$, with 40 equidistant points. C/O is varied by varing the oxygen abundance, once the total elemental composition has been set by [Fe/H]. The total list of reactant species included was H, H$_2$, He, O, C, N, Mg, Si, Fe, S, AL, Ca, Na, Ni, P, K, Ti, CO, OH, SH, N$_2$, O$_2$, SiO, TiO, SiS, H$_2$O, C$_2$, CH, CN, CS, SiC, NH, SiH, NO, SN, SiN, SO, S$_2$, C$_2$H, HCN, C$_2$H$_2$,acetylene, CH$_4$, ALH, ALOH, AL$_2$O, CaOH, MgH, Mg, OH, PH$_3$, CO$_2$, TiO$_2$, Si$_2$C, SiO$_2$, FeO, NH$_2$, NH$_3$, CH$_2$, CH$_3$, H$_2$S, VO, VO$_2$, NaCL, KCL, e-, H+, H-, Na+, K+, PH$_2$, P$_2$, PS, PO, P$_4$O$_6$, PH, V, FeH, VO(c), VO(L), MgSiO$_3$(c), SiC(c), Fe(c), Na$_2$S(c), KCL(c), Fe(L), SiC(L), MgSiO$_3$(L), H$_2$O(L), H$_2$O(c), TiO(c), TiO(L), TiO$_2$(c), TiO$_2$(L), H$_3$PO$_4$(c), H$_3$PO$_4$(L), where (c) stands for solid and (L) for liquid species. To conserve space only the mass fractions of the following species are tabulated for use in ``poor_mans_nonequ_chem()``: H$_2$, He, CO, H$_2$O, HCN, C$_2$H$_2$,acetylene, CH$_4$, PH$_3$, CO$_2$, NH$_3$, H$_2$S, VO, TiO, Na, K, SiO, e-, H-, H, FeH, MMW, nabla_ad. MMW denotes the mean molcular weight in the atmosphere, in units of amu. nabla_ad is the moist adiabatic lapse rate $\nabla_{\rm ad} = (\partial {\rm ln}T/\partial {\rm ln}P)_{\rm ad}$, determined as described in [Mollière et al. (2020)](https://arxiv.org/abs/2006.09394).

**Abundances in poor_mans_nonequ_chem():** abundances in poor_mans_nonequ_chem() are in units of **mass fractions**, not number fractions (aka volume mixing ratio, VMR). You can convert between mass fractions and VMRs by using
\begin{equation}
X_i = \frac{\mu_i}{\mu}n_i,
\end{equation}
where $X_i$ is the mass fraction of species $i$, $\mu_i$ the mass of a single molecule/atom/ion/... of species $i$, $\mu$ is the atmospheric mean molecular weight, and $n_i$ is the VMR of species $i$.

## Example

Here we will give some examples for how to interpolate chemical abundances using ``poor_mans_nonequ_chem()``. The installation of it is described immediately below.

We begin by loading the relevant packages:

```python
import numpy as np
import pylab as plt
plt.rcParams['figure.figsize'] = (10, 6)
from poor_mans_nonequ_chem import poor_mans_nonequ_chem as pm
from petitRADTRANS import nat_cst as nc
```

Next, we define an atmospheric temperature and abundance structure, following an example similar to what is shown in ["Getting Started"](getting_started.html).

```python
pressures = np.logspace(-6, 2, 100)
gravity = 1e1**2.45
kappa_IR = 0.01
gamma = 0.4
T_int = 200.
T_equ = 800.

temperature = nc.guillot_global(pressures, kappa_IR, gamma, gravity, T_int, T_equ)

plt.plot(temperature, pressures)
plt.yscale('log')
plt.ylim([1e2, 1e-6])
plt.xlabel('T (K)')
plt.ylabel('P (bar)')
plt.show()
plt.clf()
```

No we calculate the abundances. Note that the input parameters temperature, pressure, [Fe/H] and C/O are all arrays, and must be defined at every layer, but usually C/O and [Fe/H] are not vertically varying in the atmosphere. Here we chose a solar metallicity and C/O.

```python
COs = 0.55 * np.ones_like(pressures)
FeHs = 0. * np.ones_like(pressures)

mass_fractions = pm.interpol_abundances(COs, \
            FeHs, \
            temperature, \
            pressures)
```

Lets plot the abundances:

```python
plt.yscale('log')
plt.xscale('log')
plt.ylim([100,1e-6])
plt.xlim([1e-10, 1])

for species in mass_fractions.keys():
    if species not in ['MMW', 'nabla_ad']:
        plt.plot(mass_fractions[species], pressures, label = species)

plt.legend(loc='best', ncol=3)
plt.xlabel('Mass fraction')
plt.ylabel('P (bar)')
plt.show()
plt.clf()
```

Also the MMW and adiabatic temperature gradient are accessible, which turns out to be vertically constant for this atmosphere:

```python
plt.yscale('log')
plt.ylim([100,1e-6])

plt.plot(mass_fractions['MMW'], pressures, label = species)

plt.xlabel('Mean molecular weight (MMW)')
plt.ylabel('P (bar)')
plt.show()
plt.clf()
```

```python
plt.yscale('log')
plt.ylim([100,1e-6])

plt.plot(mass_fractions['nabla_ad'], pressures, label = species)

plt.xlabel(r'Moist adiabatic temperature gradient $\nabla_{\rm ad}$')
plt.ylabel('P (bar)')
plt.show()
plt.clf()
```

As a test, we can also plot abundances at a given pressure, temperature, C/O, as a function of metallicity:

```python
FeHs = np.linspace(-2, 3, 100)
p_const = np.ones_like(FeHs)*1e-1
t_const = np.ones_like(FeHs)*900.
COs = 0.55 * np.ones_like(FeHs)

mass_fractions = pm.interpol_abundances(COs, \
            FeHs, \
            t_const, \
            p_const)
```

```python
plt.yscale('log')
plt.ylim([1e-10, 1])
plt.xlim([-2, 3])

for species in mass_fractions.keys():
    if species not in ['MMW', 'nabla_ad']:
        plt.plot(FeHs, mass_fractions[species], label = species)

plt.legend(loc='best', ncol=3)
plt.ylabel('Mass fraction')
plt.xlabel('[Fe/H]')
plt.title('T = 900 K, P = 0.1 bar, C/O = 0.55')
plt.show()
plt.clf()
```

This is the mean molecular weight as a function of metallicity:

```python
plt.xlim([-2, 3])
plt.ylim([0, 15])

plt.plot(FeHs, mass_fractions['MMW'])

plt.ylabel('Mean molecular weight')
plt.xlabel('[Fe/H]')
plt.title('T = 900 K, P = 0.1 bar, C/O = 0.55')
plt.show()
plt.clf()
```

Here the mass fractions as a function of the C/O ratio, at solar metallicity:

```python
COs = np.linspace(0.1, 1.6, 100)
FeHs = 0. * np.ones_like(COs)
p_const = np.ones_like(COs)*1e-1
t_const = np.ones_like(COs)*1400.

mass_fractions = pm.interpol_abundances(COs, \
            FeHs, \
            t_const, \
            p_const)
```

```python
plt.yscale('log')
plt.ylim([1e-10, 1])
plt.xlim([0.1, 1.6])

for species in mass_fractions.keys():
    if species not in ['MMW', 'nabla_ad']:
        plt.plot(COs, mass_fractions[species], label = species)

plt.legend(loc='lower right', ncol=3)
plt.ylabel('Mass fraction')
plt.xlabel('C/O')
plt.title('T = 1400 K, P = 0.1 bar, [Fe/H] = 0')
plt.show()
plt.clf()
```

Finally, we show an example with quenching here, assuming that the CH$_4$, H$_2$O and CO abundances are constant in for pressures below 1 bar. We will use the temperature structure from above again.

```python
# Nominal case:
COs = 0.55 * np.ones_like(pressures)
FeHs = 0. * np.ones_like(pressures)

mass_fractions = pm.interpol_abundances(COs, \
            FeHs, \
            temperature, \
            pressures)

# Quenching case:
mass_fractions_quench = pm.interpol_abundances(COs, \
            FeHs, \
            temperature, \
            pressures, \
            Pquench_carbon = 1.)
```

```python
plt.yscale('log')
plt.xscale('log')
plt.ylim([100,1e-6])
plt.xlim([1e-10, 1])

plt.plot(mass_fractions['H2'], pressures, linestyle = '--', color = 'black')
plt.plot(mass_fractions_quench['H2'], pressures, label = 'H2', linestyle = '-', color = 'black')
plt.plot(mass_fractions['He'], pressures, linestyle = '--', color = 'gray')
plt.plot(mass_fractions_quench['He'], pressures, label = 'He', linestyle = '-', color = 'gray')
plt.plot(mass_fractions['CO'], pressures, linestyle = '--', color = 'red')
plt.plot(mass_fractions_quench['CO'], pressures, label = 'CO', linestyle = '-', color = 'red')
plt.plot(mass_fractions['H2O'], pressures, linestyle = '--', color = 'purple')
plt.plot(mass_fractions_quench['H2O'], pressures, label = 'H2O', linestyle = '-', color = 'purple')
plt.plot(mass_fractions['CH4'], pressures, linestyle = '--', color = 'orange')
plt.plot(mass_fractions_quench['CH4'], pressures, label = 'CH4', linestyle = '-', color = 'orange')

plt.axhline(1., color = 'blue', linestyle = ':', label = 'Quench pressure')

plt.legend(loc='best', ncol=1)
plt.xlabel('Mass fraction')
plt.ylabel('P (bar)')
plt.title("Poor man's non-equilibrium chemistry: quenching")
plt.show()
plt.clf()
```

## Installation of poor_mans_nonequ_chem()

Download the ``poor_mans_nonequ_chem()`` [here](https://www.dropbox.com/s/6r3djmzu8y1qplf/poor_mans_nonequ_chem.zip?dl=0). If relevant, activate the python environment that you want to run your code in (similar to the [installation tips when installing pRT](../installation.html)). Then, in the terminal, switch into the poor_mans_nonequ_chem/chem_fortran_util folder and type ``./make.sh``, again similar to the [pRT installation](../installation.html). Lastly, [like when installing pRT](../installation.html), enter the following line to your .bashrch or .bashprofile (depending on your operating system):

``export PYTHONPATH=Path of the folder containing the highest level poor_mans_nonequ_chem folder/:$PYTHONPATH``

After this close and reopen the terminal such that it will set the Python path correctly, or source the  .bashrc/.bashprofile file. Alternatively, you can place the following line at the top of your scripts that use ``poor_mans_nonequ_chem``, berfore importing ``poor_mans_nonequ_chem``:

```python
import sys
sys.path.insert(0,'Path to folder containing the highest level poor_mans_nonequ_chem folder/')
```

You are done!


```python
# %load /Users/facai/Study/book_notes/preconfig.py
%matplotlib inline

import matplotlib.pyplot as plt
import seaborn as sns
sns.set(color_codes=True)
sns.set(font='SimHei')
plt.rcParams['axes.grid'] = False

from IPython.display import SVG

def show_image(filename, figsize=None):
    if figsize:
        plt.figure(figsize=figsize)

    plt.imshow(plt.imread(filename))
```

GBDT(Gradient Boosting Decision Tree) 原理简介
=============================================

### 0. 前言

我最开始了解 GBDT 时，死活不理解决策树这种分段函数，怎么可能算出一阶导数。读了论文 Friedman - Greedy Function Approximation: A Gradient Boosting Machine 后，才发现自己完全误解了决策树在GBDT中的作用。论文总是倾向把简单事情描述复杂，博客又常常过于简练地只给具体算法。写这篇文章的想法，就是想用自己的理解，既照顾到数学框架，又落实到具体算法实现，把GBDT说清楚。这里可能有谬误，读者应审慎地阅读。

### 1. Gradient Boosting 与最优化
ref:

+ [机器学习中的算法(1)-决策树模型组合之随机森林与GBDT](http://www.cnblogs.com/LeftNotEasy/archive/2011/03/07/random-forest-and-gbdt.html)

在说GBDT前，我们先说下它的俩前缀Gradient Boosting：

+ Boosting: 这是一种迭代算法，每一次训练都是在前面已有模型的预测基础上进行。
  最简单地说，先训练一个初始模型，对比真实值和预测值的残差；用残差再训练一个模型，再计算残差；再训练……。这样，每一个模型都专注于修正前面模型最不给力的效果方面。于是，通过这种方式联合多个弱分类器，就能得到一个强分类器。
+ Gradient: 梯度。对于单个变量来说，就是一阶导数。
  前面Boosting要计算残差，方法就很多了，你可以直接相减，也可以定义个函数。这里Gradient就是说用梯度来计算残差。

所以说Gradient Boosting是一种算法框架，它是用Grdient计算残差的Boosting过程，而GBDT只是用决策树来训练的一种具体实现。说到这，学习过最优化的童鞋一定会对Gradient Boosting有种强烈的既视感，是的，那就是我们耳熟能详的最速梯度下降法。

所以，我们先从最速梯度下降法作引，一步步走到Gradeint Boosting框架，然后再落到决策树上，最后如果有精力，介绍下进一步的优化TreeBoost。

### 2. 最速梯度下降法

我简要说下基本思路，详细可见[梯度下降法 - 维基百科](https://zh.wikipedia.org/wiki/梯度下降法)。

假设有个有界函数 $| f(x) | < M$，我们需要知道它的最小值。日常的例子中，这个函数可能是一项工程所用的花销，一个项目占用的时间，我们想找到损失最小的方案，即：$\operatorname{arg min}_x \, f(x) $。 找到最小解的过程就叫寻优，函数$f(x)$常被称为损失函数。

#### 2.0 理论

梯度下降法是一种数值寻优算法。什么是数值寻优呢？就是现实中$f(x)$很多时候难以用理论求解出最值，或者$f(x)$就是个黑盒子，我们没有办法直接找到最值，只能去有限的数值解中搜索，所以叫数值寻优。

梯度下降法就是按函数梯度来步进到极值点的过程，其数学表达式形如：
\begin{equation}
    x_{m+1} = x_m - \lambda \frac{\partial f(x)}{\partial x}
\end{equation}

具体过程如下图所示，取自[可扩展机器学习——梯度下降(Gradient Descent)](http://blog.csdn.net/google19890102/article/details/50276775)：

```python
show_image("./res/gradient_descent.jpg", figsize=(12,8))
```

```python
show_image("./res/iterator.jpg")
```

#### 2.1 增量寻优

梯度下降法是利用了导数做为步进参考值，它要求函数是已知且可导的。问题是，现实中，有时这个函数是未知的，比如说对应的是一台机器。我不知道怎么对它建模，唯一的信息就是喂它输入，它就会返回输出。那怎么办呢？

首先，我们回到 $\frac{\partial f(x)}{\partial x}$ 来看，因为步进大小还受 $\lambda$ 调控。这里梯度的作用，其实主要是指示步进方向。

然后，我们换个思路找步进方向。

假设机器的真实模型是 $f(x)$，对我们来说它是未知的。尝试如下：

+ 第一次试探，随机给个初始值 $x_0$，得到 $f(x_0)$。
+ 第二次试探，再随机输入 $x_1$，得到 $f(x_1)$。

这时，有如下三种情况，可以指导第三次试探：

1. $f(x_1) < f(x_0)$，也就是 $x_0 \to x_1$的方向是对的。
2. $f(x_1) = f(x_0)$，没有有用信息。
3. $f(x_1) > f(x_0)$，方向反了。

```python
show_image("./res/incr_opt.png", figsize=(10,5))
```

因为我们始终是找极小值点，这个过程就始终如上图所示"U"形。那么每次的步进方向就可用 $f(x_1) - f(x_0)$ 来指示。也就是说，虽然我们无法对机器$f(x)$建模，但我们可以对寻优的过程 $z = f(x_i) - f(x_{i+1})$ 建模 $g(z)$。而 $g(z)$ 只要满足单调且过零点，理论上它可以是任意的，不同函数影响的只是收敛速度，但都会收敛到 $w^*$ 附近。

#### 2.2 逼近

再换个角度，我们不再想寻优。而是假设这个极小值$w^*$已经知道了，如何从初始值$w_0 = f(x_0)$一步步逼进到这个极小值$w^*$。其实可以完全借用上节的增量寻优建模手法，我们定义 $z = f(x_i) - w^*$，从而对逼近过程建模 $g(z)$。

```python
show_image("./res/approx.png", figsize=(10,5))
```

同样地，$g(z)$只要满足一定条件的函数。在这里，就是过零点，正数区间单增，负数区间单减，最常见的代表就是二次函数 $g(z) = z^2$。

我们梳理下整个逼近过程，如下图：

```python
show_image("./res/model.png", figsize=(10,5))
```

从初始点$w_0$，沿着$w_1, w_2, \dotsc, w_i$到达$w^*$。其中每步的步长是$\lambda \frac{\partial g(z)}{\partial z} |_{z = z_i}$。$\lambda$是控制步长的字习率，不用管。我们重点看下$z_i$:

\begin{align}
    z_i &= w_i - w^* \\
        &= f(x_i) - w^*
\end{align}

上面，每一步对应的$f(x_i)$是一个恒定的模型，变动的是$x$。我们反着想，假设$x$是固定的，而这个模型其实是变化的 $f_i(x)$，再把$w^*$用$y$表示。那么

\begin{align}
    z_i &= f(x_i) - w^* \\
        &= f_i(x) - y
\end{align}

是不是开始有点眼熟了，如果我们再令 $g(z) = z^2$，展开看看：

\begin{align}
    g(z_i) &= z_i^2 \\
           &= (y - f_i(x))^2
\end{align}

好了，我们总结下上述函数的意义：
给定自变量$x$、目标值$y$和初始位置$w_0$，可以用规定的$g(z)$和模型$f_i(x)$来构建出一个迭代序列$\{w_0, w_1, \dotsc, y\}$，其中$w_{i+1} = w_i - \lambda \frac{\partial g(z)}{\partial z} \, |_{z = y - f_i(x)}$。

而：
+ 对给定$x$，$y$是常数，所以$\partial z = \partial f_i(x)$。
+ $g(z) | z = y - f_i(x)$表述的是$y$和$f_i(x)$的距离相近度，可以更泛性推广记为$L(y, f_i(x))$。

所以记$w_{i+1} = w_i - \lambda \frac{\partial L(y, f_i(x))}{\partial f_i(x)}$

用机器学习的术语翻译就是，如果有特征$x$、目标值$y$和初始模型$f_0$，对某个损失函数$L(y, f_i(x))$，就有模型序列使得$f_i(x) \to y$。

而Gradient Boosting就是回答了上述的两个规定：

1. 怎样构建模型序列$f_i$?
2. 什么样的损失函数$L(y, f_i(x))$?

### 3. Gradient Boosting

#### 3.1 怎样构建模型序列$f_i$
Gradient Boosting的答案就是使用Boosting来训练Gradient(梯度）值的加法模型。

具体来说，已知有特征$x$、标签$y$和初始模型$f_0$。
然后，我们选定一种机器学习算法$\text{Model}(\text{feature}, \text{value})$，输入数据后就会给出模型$F(x)$。

我们规定模型生成序列方法如下：

\begin{align}
    & F_0 = f_0 \\
    & \hat{y_i} = - \frac{\partial L(y, f_{i-1}(x)}{\partial f_{i-1}(x)} \\
    & F_i = \text{Model}(x, \hat{y_i}) \\
    & f_{i} = f_{i-1} + F_i  = \displaystyle \sum_{k=0}^{i} F_k \\
\end{align}

上式第二行就是Gradient（梯度），第三行就是用梯度训练的单个模型$F_i$，而第四行$f_i$就是Boosting加法。请注意区分$f_i$和$F_i$。

很多时候，我们期望初始模型自动产生，并且和其它模型是同种类型的，上面的生成序列方法就可改写为：

\begin{align}
    & F_0 = \text{Model}(x, y) \\
    & \hat{y_i} = - \frac{\partial L(y, f_{i-1}(x)}{\partial f_{i-1}(x)} \\
    & F_i = \text{Model}(x, \hat{y_i}) \\
    & f_{i} = f_{i-1} + F_i  = \displaystyle \sum_{k=0}^{i} F_k \\
\end{align}

总结下流程：

1. 用$x$和$y$训练初始模型$F_0$，并令累加模型$f_0 = F_0$。
2. 用梯度计算出残差 $\hat{y_i}$。
3. 用$x$和残差训练新模型$F_i$。
4. 将新模型加入累加模型$f_i$。
5. 迭代直到达到终止条件。

#### 3.2 什么样的损失函数 $L(y, f_i(x))$

正如2.2节所说，损失函数$L(y, f_i(x))$定义的是从初始值到目标值的逼近过程。常理来说，离得越远，我们期望步距越大，调整越剧烈；离得越近，期望步距越小，调整越缓合。常用的有L1, L2和logistic。当然，理论上，你也可以用其它千奇百怪的函数，只要它符合$x_1 \to x_2 \implies f(x_1) \to f(x_2)$。

### 4. GBDT(Gradient Boosting Decision Tree)

将第3节中的机器模型Model换成决策树，就得到了GBDT。嗯，就是这么简单。

Treeboost是具体对决策树参与的计算做了公式推导，作用是简化运算量，打算后续开文专门讲。

### 总结
我们最后总结下，GBDT是Gradient Boosting框架用决策树完成的一个具体实现。Gradeint Boosting框架的本质思路是一个从基准值到目标值的逼近过程，也是一个基准弱模型向强模型的进化过程。逼近过程是用损失函数来描述和控制的，具体的步进量（残差）由损失函数的梯度来指示。机器模型的作用是用来拟合和预测残差。

所以，我以前不理解决策树怎么求导，而事实是决策树压根就和求导没关系。希望这篇文章能帮助到其它人的理解，也请指正错点漏点疑点。

```python

```