```
#@title Copyright 2020 The Cirq Developers
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
```

# QAOA: Max-Cut

    View on QuantumAI

    Run in Google Colab

    View source on GitHub

    Download notebook

In this tutorial, we implement the quantum approximate optimization algorithm (QAOA) for determining the Max-Cut of the Sycamore processor's hardware graph (with random edge weights). To do so, we will:

1. Define a random set of weights over the hardware graph.
2. Construct a QAOA circuit using Cirq.
3. Calculate the expected value of the QAOA cost function.
4. Create an outer loop optimization to minimize the cost function.
5. Compare cuts found from QAOA with random cuts.

```
try:
    import cirq
except ImportError:
    print("installing cirq...")
    !pip install --quiet cirq
    import cirq
    print("installed cirq.")
```

## 1. Defining a random set of weights over the hardware graph
In order to make the problem easily embeddable on a quantum device, we will look at the problem of Max-Cut on the same graph that the device's qubit connectivity defines, but with random valued edge weights.

```
import cirq_google
import sympy
import numpy as np
import matplotlib.pyplot as plt

working_device = cirq_google.Sycamore
print(working_device)
```

Since a circuit covering the entire Sycamore device cannot be easily simulated, a small subset of the device graph will be used instead.

```
import networkx as nx

# Set the seed to determine the problem instance.
np.random.seed(seed=11)

# Identify working qubits from the device.
device_qubits = working_device.qubits
working_qubits = sorted(device_qubits)[:12]

# Populate a networkx graph with working_qubits as nodes.
working_graph = working_device.metadata.nx_graph.subgraph(working_qubits)

# Add random weights to edges of the graph. Each weight is a 2 decimal floating point between 0 and 5.
nx.set_edge_attributes(working_graph, {e: {'weight': np.random.randint(0, 500) / 100} for e in working_graph.edges})

# Draw the working_graph on a 2d grid
pos = {q:(q.col, -q.row) for q in working_graph.nodes()}
nx.draw(working_graph, pos=pos, with_labels=True, node_size=1000)
plt.show()
```

## 2. Construct the QAOA circuit
Now that we have created a Max-Cut problem graph, it's time to generate the QAOA circuit following [Farhi et al.](https://arxiv.org/abs/1411.4028). For simplicity $p = 1$ is chosen.

```
from cirq.contrib.svg import SVGCircuit

# Symbols for the rotation angles in the QAOA circuit.
alpha = sympy.Symbol('alpha')
beta = sympy.Symbol('beta')

qaoa_circuit = cirq.Circuit(
    # Prepare uniform superposition on working_qubits == working_graph.nodes
    cirq.H.on_each(working_graph.nodes()),

    # Do ZZ operations between neighbors u, v in the graph. Here, u is a qubit,
    # v is its neighboring qubit, and w is the weight between these qubits.
    (cirq.ZZ(u, v) ** (alpha * w['weight']) for (u, v, w) in working_graph.edges(data=True)),

    # Apply X operations along all nodes of the graph. Again working_graph's
    # nodes are the working_qubits. Note here we use a moment
    # which will force all of the gates into the same line.
    cirq.Moment(cirq.X(qubit) ** beta for qubit in working_graph.nodes()),

    # All relevant things can be computed in the computational basis.
    (cirq.measure(qubit) for qubit in working_graph.nodes()),
)
SVGCircuit(qaoa_circuit)
```

## 3. Calculating the expected value of the QAOA cost Hamiltonian
Now that we have created a parameterized QAOA circuit, we need a way to calculate expectation values of the cost Hamiltonian. For Max-Cut, the cost Hamiltonian is

$$
    H_C = \frac{1}{2} \sum_{\langle i, j\rangle} w_{ij} (1 - Z_i Z_j )
$$

where $\langle i, j \rangle$ denotes neighboring qubits, $w_{ij}$ is the weight of edge $ij$, and $Z$ is the usual Pauli-$Z$ matrix. The expectation value of this cost Hamiltonian is $\langle \alpha, \beta | H_C | \alpha, \beta \rangle$ where $|\alpha, \beta\rangle$ is the quantum state prepared by our `qaoa_circuit`. This is the cost function we need to estimate.

> Pauli-$Z$ has eigenvalues $\pm 1$. If qubits $i$ and $j$ are in the same eigenspace, then $\langle Z_i Z_j \rangle = 1$ and so $\frac{1}{2} w_{ij} \langle 1 - Z_i Z_j \rangle = 0$. In the Max-Cut language, this means that edge $ij$ does not contribute to the cost. If qubits $i$ and $j$ are in the opposite eigenspace, then $\langle Z_i Z_j \rangle = -1$ and so $\frac{1}{2} w_{ij} \langle 1 - Z_i Z_j \rangle = w_{ij}$. In the Max-Cut language, this means that edge $ij$ contributes its weight $w_{ij}$ to the cost.

To estimate the cost function, we need to estimate the (weighted) sum of all $ZZ$ pairs in the graph. Since these terms are diagonal in the same basis (namely, the computational basis), they can measured simultaneously. Given a set of measurements (samples), the function below estimates the cost function.

> *Note*: We say "estimate the cost" instead of "compute the cost" since we are sampling from the circuit. This is how the cost would be evaluated when running QAOA on a real quantum processor.

```
def estimate_cost(graph, samples):
    """Estimate the cost function of the QAOA on the given graph using the
    provided computational basis bitstrings."""
    cost_value = 0.0

    # Loop over edge pairs and compute contribution.
    for u, v, w in graph.edges(data=True):
        u_samples = samples[str(u)]
        v_samples = samples[str(v)]

        # Determine if it was a +1 or -1 eigenvalue.
        u_signs = (-1)**u_samples
        v_signs = (-1)**v_samples
        term_signs = u_signs * v_signs

        # Add scaled term to total cost.
        term_val = np.mean(term_signs) * w['weight']
        cost_value += term_val

    return -cost_value
```

Now we can sample from the `qaoa_circuit` and use `estimate_expectation` to calculate the expectation value of the cost function for the circuit. Below, we use arbitrary values for $\alpha$ and $\beta$.

```
alpha_value = np.pi / 4
beta_value = np.pi / 2
sim = cirq.Simulator()

sample_results = sim.sample(
    qaoa_circuit,
    params={alpha: alpha_value, beta: beta_value},
    repetitions=20_000
)
print(f'Alpha = {round(alpha_value, 3)} Beta = {round(beta_value, 3)}')
print(f'Estimated cost: {estimate_cost(working_graph, sample_results)}')
```

## 4. Outer loop optimization
Now that we can compute the cost function, we want to find the optimal cost. There are lots of different techniques to choose optimal parameters for the `qaoa_circuit`. Since there are only two parameters here ($\alpha$ and $\beta$), we can keep things simple and sweep over incremental pairings using `np.linspace` and track the minimum value found along the way.

```
# Set the grid size = number of points in the interval [0, 2π).
grid_size = 5

exp_values = np.empty((grid_size, grid_size))
par_values = np.empty((grid_size, grid_size, 2))

for i, alpha_value in enumerate(np.linspace(0, 2 * np.pi, grid_size)):
    for j, beta_value in enumerate(np.linspace(0, 2 * np.pi, grid_size)):
        samples = sim.sample(
            qaoa_circuit,
            params={alpha: alpha_value, beta: beta_value},
            repetitions=20000
        )
        exp_values[i][j] = estimate_cost(working_graph, samples)
        par_values[i][j] = alpha_value, beta_value
```

We can now visualize the cost as a function of $\alpha$ and $\beta$.

```
plt.title('Heatmap of QAOA Cost Function Value')
plt.xlabel(r'$\alpha$')
plt.ylabel(r'$\beta$')
plt.imshow(exp_values)
plt.show()
```

This heatmap is coarse because we selected a small `grid_size`. To see more detail in the heatmap, one can increase the `grid_size`.

## 5. Compare cuts

We now compare the optimal cut found by QAOA to a randomly selected cut. The helper function draws the `working_graph` and colors nodes in different sets different colors. Additionally, we print out the cost function for the given cut.

```
def output_cut(S_partition):
    """Plot and output the graph cut information."""

    # Generate the colors.
    coloring = []
    for node in working_graph:
        if node in S_partition:
            coloring.append('blue')
        else:
            coloring.append('red')

    # Get the weights
    edges = working_graph.edges(data=True)
    weights = [w['weight'] for (u,v, w) in edges]

    nx.draw_circular(
        working_graph,
        node_color=coloring,
        node_size=1000,
        with_labels=True,
        width=weights)
    plt.show()
    size = nx.cut_size(working_graph, S_partition, weight='weight')
    print(f'Cut size: {size}')
```

As an example, we can test this function with all nodes in the same set, for which the cut size should be zero.

```
# Test with the empty S and all nodes placed in T.
output_cut([])
```

To get cuts using the QAOA we will first need to extract the best control parameters found during the sweep:

```
best_exp_index = np.unravel_index(np.argmax(exp_values), exp_values.shape)
best_parameters = par_values[best_exp_index]
print(f'Best control parameters: {best_parameters}')
```

Each bitstring can be seen as a candidate cut in the graph. The qubits that measured 0 correspond to that qubit being in one cut partition and a qubit that measured to 1 corresponds to that qubit being in the other cut partition. Now that we've found good parameters for the `qaoa_circuit`, we can just sample some bistrings, iterate over them and pick the one that gives the best cut:

```
# Number of candidate cuts to sample.
num_cuts = 100
candidate_cuts = sim.sample(
    qaoa_circuit,
    params={alpha: best_parameters[0], beta: best_parameters[1]},
    repetitions=num_cuts
)

# Variables to store best cut partitions and cut size.
best_qaoa_S_partition = set()
best_qaoa_T_partition = set()
best_qaoa_cut_size = -np.inf

# Analyze each candidate cut.
for i in range(num_cuts):
    candidate = candidate_cuts.iloc[i]
    one_qubits = set(candidate[candidate==1].index)
    S_partition = set()
    T_partition = set()
    for node in working_graph:
        if str(node) in one_qubits:
            # If a one was measured add node to S partition.
            S_partition.add(node)
        else:
            # Otherwise a zero was measured so add to T partition.
            T_partition.add(node)

    cut_size = nx.cut_size(
        working_graph, S_partition, T_partition, weight='weight')

    # If you found a better cut update best_qaoa_cut variables.
    if cut_size > best_qaoa_cut_size:
        best_qaoa_cut_size = cut_size
        best_qaoa_S_partition = S_partition
        best_qaoa_T_partition = T_partition
```

The QAOA is known to do just a little better than random guessing for Max-Cut on 3-regular graphs at `p=1`. You can use very similar logic to the code above, but now instead of relying on the QAOA to decide your `S_partition` and `T_partition` you can just pick then randomly:

```
import random

best_random_S_partition = set()
best_random_T_partition = set()
best_random_cut_size = -9999

# Randomly build candidate sets.
for i in range(num_cuts):
    S_partition = set()
    T_partition = set()
    for node in working_graph:
        if random.random() > 0.5:
            # If we flip heads add to S.
            S_partition.add(node)
        else:
            # Otherwise add to T.
            T_partition.add(node)

    cut_size = nx.cut_size(
        working_graph, S_partition, T_partition, weight='weight')

    # If you found a better cut update best_random_cut variables.
    if cut_size > best_random_cut_size:
        best_random_cut_size = cut_size
        best_random_S_partition = S_partition
        best_random_T_partition = T_partition
```

```
print('-----QAOA-----')
output_cut(best_qaoa_S_partition)

print('\n\n-----RANDOM-----')
output_cut(best_random_S_partition)
```

For this problem instance, one should see that $p = 1$ QAOA performs better, on average, than randomly guessing.


```python
from IPython.core.display import HTML
from IPython.display import Image

import numpy as np
import sympy as sp
from sympy import oo
from utils import symplot, symdisp, round_expr

HTML("""
<style>
.output_png {
    display: table-cell;
    text-align: center;
    vertical-align: middle;
}
</style>
""")
```

<style>
.output_png {
    display: table-cell;
    text-align: center;
    vertical-align: middle;
}
</style>

# *Circuitos Elétricos I - Semana 6*

## Elementos armazenadores de energia

Comparação entre capacitores convencionais, supercapacitores e baterias de lítio. A tabela abaixo mostra as especificações  necessários para cada dispositivo armazenar ∼1 megajoule (MJ) de energia (300 watts-hora). 1 MJ de energia irá alimentar um laptop com um consumo médio de 50 W por 6 horas. Observe na primeira coluna que uma bateria de íon de lítio pode conter 1000 vezes mais energia do que um capacitor convencional.

$$
\begin{array}{|c|c|c|c|c|c|}
\hline \text { Dispositivo } & \begin{array}{c}
\text { Energia } \\
\text { específica } \\
\text { [Wh/kg]} \\
\end{array} & \begin{array}{c}
\text { Energia } \\
\text { específica } \\
\text { [MJ/kg] }
\end{array} & \begin{array}{c}
\text { Densidade de} \\
\text { Energia } \\
\text { [MJ / L] }
\end{array} & \begin{array}{c}
\text { Volume } \\
\text { requerido para } \\
\text { armazenar 1 MJ } \\
\text { [L] }
\end{array} & \begin{array}{c}
\text { Peso } \\
\text { requerido para } \\
\text { armazenar 1 MJ } \\
\text { [kg] }
\end{array} \\
\hline \begin{array}{c}
\text { Capacitor convencional} \\
\end{array} & 0.01-0.1 & 4 \times 10^{-5}-4 \times 10^{-4} & 6 \times 10^{-5}-6 \times 10^{-4} & 17000-1700 & 25000-2500 \\
\text { Supercapacitor } & 1-10 & 0.004-0.04 & 0.006-0.06 & 166-16 & 250-25 \\
\text { Bateria de Íons de Lítio } & 100-250 & 0.36-0.9 & 1-2 & 1-0.5 & 2.8-1.1 \\
\hline
\end{array}
$$

Fonte: Fawwaz Ulaby, Michel M. Maharbiz and Cynthia M. Furse, $\textit{ Circuit Analysis and Design}$, Michigan Publishing Services, 2018

## Resumo dos elementos passivos ideais de dois terminais

$$
\begin{array}{|l|c|c|c|}
\hline \text { Propriedade } & R & L & C \\
\hline \text { Relação } i-v  & i=\frac{v}{R} & i=\frac{1}{L} \int_{t_{0}}^{t} v(\tau) d \tau+i\left(t_{0}\right) & i=C \frac{d v}{d t} \\
\text { Relação } v-i & v=Ri & v=L \frac{d i}{d t} & v=\frac{1}{C} \int_{t_{0}}^{t} i(\tau) d \tau+v\left(t_{0}\right) \\
p \text { (potência }) & p=Ri^{2} & p=L i \frac{d i}{d t} & p=C v \frac{d v}{d t} \\
w \text { (energia armazenada) } & 0 & w=\frac{1}{2} L i^{2} & w=\frac{1}{2} C v^{2} \\
\text { Associação em série } & R_{\mathrm{eq}}=R_{1}+R_{2} & L_{\mathrm{eq}}=L_{1}+L_{2} & \frac{1}{C_{\mathrm{eq}}}=\frac{1}{C_{1}}+\frac{1}{C_{2}} \\
\text { Associação em paralelo } & \frac{1}{R_{\mathrm{eq}}}=\frac{1}{R_{1}}+\frac{1}{R_{2}} & \frac{1}{L_{\mathrm{eq}}}=\frac{1}{R_{1}}+\frac{1}{R_{2}} & C_{\mathrm{eq}}=C_{1}+C_{2} \\
\text { Comportamento em regime estacionário } & \text { sem mudanças } & \text { curto-circuito } & \text { circuito aberto } \\
\text { Pode } v \text { variar instantaneamente? } & \text { sim } & \text { sim } & \text { não } \\
\text { Pode } i \text { variar instantaneamente? } & \text { sim } & \text { não } & \text { sim }\\ \hline
\end{array}
$$

### Problema 1

Para o circuito abaixo, determine $v_{C1}$, $v_{C2}$ e $i_{L}$ assumindo que o circuito encontra-se em regime estacionário.

### Problema 2

No circuito abaixo, sabe-se que $i_0(t)= 50e^{-8000t}[\cos(6000t)+2\mathrm{sen}(6000t)]$ mA, para $t\geq 0^+$. Determine $v_{C}(0^+)$, $v_{L}(0^+)$ e $v_{R}(0^+)$.

```python
# define variável tempo
t = sp.symbols('t', real=True)

# expressão para a corrente no circuito
i0 = 50*sp.exp(-8000*t)*(sp.cos(6000*t)+2*sp.sin(6000*t))*1e-3

# plota gráfico da corrente
tmax = 1e-3
intervalo  = np.linspace(0, tmax, num = 1000)
symplot(t, i0, intervalo, funLabel='$i_0(t)$')
```

```python
# valores dos parâmetros do circuito
R = 320
L = 20e-3
C = 0.5e-6
```

```python
# calcula tensão no indutor
vL = L*sp.diff(i0, t)
vL = sp.simplify(vL)

print('Tensão no indutor:')
symdisp('v_L(t) = ', vL, 'V')
```

    Tensão no indutor:

$\displaystyle v_L(t) = \left(- 22.0 \sin{\left(6000 t \right)} + 4.0 \cos{\left(6000 t \right)}\right) e^{- 8000 t}\;V$

```python
symdisp('v_L(0^+) = ', vL.evalf(subs={t:0}), 'V')
```

$\displaystyle v_L(0^+) = 4.0\;V$

```python
# calcula tensão no resistor
vR = R*i0
vR = sp.simplify(vR)

print('Tensão no resistor:')
symdisp('v_R(t) = ', vR, 'V')
```

    Tensão no resistor:

$\displaystyle v_R(t) = \left(32.0 \sin{\left(6000 t \right)} + 16.0 \cos{\left(6000 t \right)}\right) e^{- 8000 t}\;V$

```python
symdisp('v_R(0^+) = ', vR.evalf(subs={t:0}), 'V')
```

$\displaystyle v_R(0^+) = 16.0\;V$

```python
# calcula tensão no capacitor (LKT)
vC = vR + vL
vC = sp.simplify(vC)

print('Tensão no capacitor:')
symdisp('v_C(t) = ', vC, 'V')
```

    Tensão no capacitor:

$\displaystyle v_C(t) = \left(10.0 \sin{\left(6000 t \right)} + 20.0 \cos{\left(6000 t \right)}\right) e^{- 8000 t}\;V$

```python
symdisp('v_C(0^+) = ', vC.evalf(subs={t:0}), 'V')
```

$\displaystyle v_C(0^+) = 20.0\;V$

```python
# checagem de vC(t) via integração de i0

vC = -(1/C)*sp.integrate(i0, (t, 0, t)) + 20
vC = sp.simplify(vC)

symdisp('v_C(t) = ', vC, 'V')
```

$\displaystyle v_C(t) = \left(10.0 \sin{\left(6000 t \right)} + 20.0 \cos{\left(6000 t \right)}\right) e^{- 8000 t}\;V$

```python
symplot(t, [vC, vR, vL], intervalo, funLabel=['$v_C(t)$ ','$v_R(t)$','$v_L(t)$'])
```

Jupyter notebooks/Circuitos Eletricos I - Semana 6.2.ipynb


 10. Markov chains

 Andrej Košir, Lucami, FE

 Kontakt: prof. dr. Andrej Košir, andrej.kosir@lucami.fe.uni-lj.si, skype=akosir_sid

 1

## ■ Intro

- Goals:
    - Markov chain modeling
    - Markov chain state analysis
- Stochastic process on finite states, most useful and used. We limit to:
    - Discrete times
    - Discrete state space (finite or countably many states)
- Case: user behavior
- Usage
    - Finite state machine analysis: software, TC system, ...
    - Models in economy
- What are questions:
    - Distributions of visits
    - Long term behavior
    - Absorption of states

 2

## ■ Sections

10.1. Definition, basic characteristics

■ Markov chain definition $\large{*}$

■ Transition matrix, distribution of states $\large{*}$

10.2. Classification of states

■ Communication of states and period $\large{*}$

■ Recurrent and transient state $\large{*}$

■ Ergodic Markov chain and return times

■ Stationary (limit) distribution $\large{*}$

■ Finite Markov chain $\large{*}$

10.3. Transition matrix estimation

■ Transition probability estimation

■ Confidence intervals and quantiles

■ Sample size determination

■ Numerical aspect

■ Case: telecommunication service user modeling

 3

## 10.1. Definition, basic characteristics

■ Markov chain definition

■ Transition matrix, distribution of states

 4

## ■ Markov chain definition

- State space $S$, $m=|S|$
    - Finite, countable, continuous

- Time $T$
    - Discrete, continuous

- Stochastic process:
$$ X_n : T \to S $$

- Discrete Markov chain is a discrete time discrete space Markov chain having Markov property
$$ P[X_{n+1}|_{X_n, X_{n-1}, \ldots, X_0}] = P[X_{n+1}|_{X_n}] = P[X_1|_{X_0}] $$

- Transition probability of one step
$$ p_{ij} = P[X_{n+1}=j|_{X_n=i}] $$

- Finite state space means there is a matrix  of transition probabilities
$$ P = [p_{ij}], \; i,j\in S. $$
    - Row sums are equal to 1.

 5

```python
import numpy as np
from sympy import Matrix
from sympy.functions import re

# Case 1
# Transition matrix
P = np.array([[0.7, 0.2, 0.1, 0.0],
              [0.2, 0.5, 0.2, 0.1],
              [0.05, 0.15, 0.6, 0.2],
              [0.0, 0.2, 0.0, 0.8]])
```

## ■ Transition matrix, distribution of states

- Transition matrix for $n$ steps
    - $p_{ij}^{(n)} = P[X_{k+n}|_{X_k=i}]$, $P^{(n)}=[p_{ij}^{(n)}]$
    - From total probability formula
    $$ P^{(n)} = P^n $$

- Distribution of states:
$$ \pi_i^n = P[X_n=i], \pi^n = [\pi_1^n, \ldots, \pi_i^n] $$
    - Initial distribution: $\pi^0$. If initial state $X_0=i_0$ is known
    $$ \pi^0=\delta_{i_0 i} $$
    - From total probability formula:
    $$ \pi^n = P^{(n)} \pi^0 $$
    - More on limit distribution later

 6

## 10.2. Classification of states

■ Communication of states and period

■ Recurrent and transient state

■ Ergodic Markov chain and return times

■ Stationary (limit) distribution

■ Finite Markov chain

 7

## ■ Communication of states and period

- Accessible state
    - State $j$ is accessible from state $i$ if $\exists n: p_{ij}^{(n)}>0$. Notation $i\leftrightarrow j$
    - Two states communicate if $i\rightarrow$ and $i \leftarrow j$, denoted by $i\leftrightarrow j$
    - „Communicate“ is equivalent relation

- Markov chain is irreducible if there are only one single communicating class

- Period of state $i$ ia denoted by $d(i)$ is greatest common divisor of all $n$ such that $P_{ii}^{(n)} > 0$:
    - If $i\leftrightarrow j$ then $d(i) = d(j)$;
    - State is aperiodic if $d(i)=1$;

- State $i$ is zero state if
$$ \lim_{n\to\infty} P_{ii}^{(n)} = 0. $$

 8

```python
# ---------------------------------------------------------------------------------------------------
# Classification of states
print ('Transition matrix P=\n', P)
print ('\nTransition matrix P^2=\n', np.linalg.matrix_power(P, 2))

# All entries of P^2 are non-zero
# => All atates are connected
```

    Transition matrix P=
     [[0.7  0.2  0.1  0.  ]
     [0.2  0.5  0.2  0.1 ]
     [0.05 0.15 0.6  0.2 ]
     [0.   0.2  0.   0.8 ]]

    Transition matrix P^2=
     [[0.535 0.255 0.17  0.04 ]
     [0.25  0.34  0.24  0.17 ]
     [0.095 0.215 0.395 0.295]
     [0.04  0.26  0.04  0.66 ]]

```python
# ---------------------------------------------------------------------------------------------------
# Decompose transition matrix
# Use Jordan decomposition
#mP = Matrix(P)
#mB, mJ = mP.jordan_form()
#B, J = np.array(mB), np.array(mJ)
B = np.array([[-0.5,-0.776313,0.659526,-0.413722],
              [-0.5,-0.212847,-0.11514,0.830469],
              [-0.5,0.283275,-0.73692,-0.123153],
              [-0.5,0.521335,0.0933625,-0.352121]])
J = np.array([[1.,0,0,0],
              [0,0.718345,0,0],
              [0,0,0.553349,0],
              [0,0,0,0.328305]])

# Test decomposition
err_mat = P - np.dot(B, np.dot(J, np.linalg.inv(B)))
print ('\nDecomposition error norm = ', np.linalg.norm(err_mat))
```

    Decomposition error norm =  7.249920892047682e-07

## ■ Recurrent and transient state

- Probability of return after exactly $n$ steps
$$ f_{ij}^n = P[X_n=j|_{X_{n-1}\ne j, \ldots, X_1\ne j, X_0=i}] $$

- Random variable of transitions from $i$ to $j$ is denoted by $T_{i}$ and its distribtion is
$$ T_{ij} \sim \left(\matrix{0 & 1 & 2 & \ldots \cr f_{ij}^0 & f_{ij}^1 & f_{ij}^2 & \ldots}\right) $$
    - Denote $T_i = T_{ii}$
    - Denote $\sum_{n=0}^\infty f_{ii}^n = f_{ii}^*$. Value
    $$ 1 - f_{ii}^* $$
    is a probability that Markov Chain newer returns to state 𝑖 after leaving it.

- State $i$ is **recurrent** if $f_{ii}^* = 1$. State is **transient** if it is not recurrent.
    - Recurrence and transience are communicating class properties

- We have: state $i$ is recurrent if and only if
$$ \sum_{n=1}^\infty P_{ii}^{(n)} = \infty $$

- Also: if $i\leftrightarrow j$ then both states are either transient or recurrent

 9

```python
# Recurent, transient?

# Eigenvalues
eigVals, eigVecs = np.linalg.eig(P.T)

# Get it sorted
idx = eigVals.argsort()[::-1]
eigVa = eigVals[idx]
eigVc = eigVecs[:,idx]

las_bf =  (1./(1-eigVa[1:4]))-1
print ('las_bf', las_bf)

sumJn = np.diag(np.insert(las_bf, 0, np.inf))
print ('\nSum J^n\n', sumJn)

sumPn = np.dot(B, np.dot(sumJn, np.linalg.inv(B)))
print ('\nSum P^n\n', sumPn)

# Since all diagonal elements of sum P^n are infinity
#  => all states are recurrent
```

    las_bf [2.55044912 1.23888596 0.48877142]

    Sum J^n
    [[       inf 0.         0.         0.        ]
     [0.         2.55044912 0.         0.        ]
     [0.         0.         1.23888596 0.        ]
     [0.         0.         0.         0.48877142]]

    Sum P^n
    [[inf inf inf inf]
     [inf inf inf inf]
     [inf inf inf inf]
     [inf inf inf inf]]

```python
def inf_sum(a_lst):
    return [(la/(1.0-la) if la < 1 else np.infty) for la in a_lst]

# ---------------------------------------------------------------------------------------------------
# Are states recurrent?
# Compute sum of P^n
sumJ = np.diag(inf_sum(np.diag(J)))
sumP = np.dot(B, np.dot(sumJ, np.linalg.inv(B)))

print ('\nSum of P^n = \n', sumP)
# => All states are reccurent
```

    Sum of P^n =
    [[inf inf inf inf]
     [inf inf inf inf]
     [inf inf inf inf]
     [inf inf inf inf]]

## ■ Ergodic Markov chain and return times

- Markov chain is **Ergodic** if it is
    - Irreducible
    - Recurrent (all states are recurrent)
    - Aperiodic (all states are aperiodic)
    - Nonzero (all states are positive)

- Expected return time:
$$ E(T_i) = \sum_{n=0}^\infty n f_{ii}^{(n)} $$

- In ergodic Markov chain
$$ \lim_{n\to\infty} P_{ii}^{(n)} = \frac{1}{E(T_i)} $$
$$ \lim_{n\to\infty} P_{ij}^{(n)} = \lim_{n\to\infty} P_{ii}^{(n)} $$

- Recurrent **zero** state $i$:
$$ \lim_{n\to\infty} P_{ii}^{(n)} = 0 \; \Leftrightarrow \;  E(T_i) = \infty  $$

- Recurrent **positive** = **ergodic** state $i$:
$$ \lim_{n\to\infty} P_{ii}^{(n)} > 0 \; \Leftrightarrow \; E(T_i) < \infty  $$

 10

```python
def inf_lim(a_lst):
    return [(0 if la < 1 else 1) for la in a_lst]

# ---------------------------------------------------------------------------------------------------
# Zero or positive states?
# Compute lim of P^n
limJ = np.diag(inf_lim(np.diag(J)))
limP = np.dot(B, np.dot(limJ, np.linalg.inv(B)))

print ('\nlim of P^n = \n', limP)

# Since all lim P^n are positive
# => all states are positive

# Expected times of visits
ETi = 1.0/np.diag(limP)

print ('\nExpected number of visits py states:\n', ETi)
```

    lim of P^n =
    [[0.21301781 0.27218919 0.18934915 0.32544385]
     [0.21301781 0.27218919 0.18934915 0.32544385]
     [0.21301781 0.27218919 0.18934915 0.32544385]
     [0.21301781 0.27218919 0.18934915 0.32544385]]

    Expected number of visits py states:
    [4.69444311 3.67391519 5.28124903 3.07272668]

## ■ Stationary (limit) distribution

- State distribution after $n$ steps
$$ \pi_i^n = P[X_n=i], \qquad \pi^n = [\pi_1^n, \ldots, \pi_i^n] $$

- Stationary (limit) distribution:
  $$ \pi = \lim_{n\to\infty} \pi^n $$
    - When uniquely defined: when all states are recurrent positive

- Ergodic MC: stationary distribution exists
- Stationary distribution of ergodic MC chain is left eigenvector of transition matrix:
  $$ \pi^\top = \pi^\top P. $$

- Note: in an ergodic MC a stationary distribution independent of initial distribution $\pi^0$.

 11

```python
# ---------------------------------------------------------------------------------------------------
# Limit distribuion
eigVals, eigVecs = np.linalg.eig(P.T)
pi = eigVecs[:, 0]/sum(eigVecs[:, 0])

print ('\neigVals=\n', eigVals)
#print '\neigVecs=\n', eigVecs
print ('\nLimit distribution =', pi)
```

    eigVals=
    [1.         0.32830522 0.71834549 0.55334929]

    Limit distribution = [0.21301775 0.27218935 0.18934911 0.32544379]

## ■ Probability of absorption

- Absorption: chain never leave a subset of states $S_0 \subset S$

- What is the probability of absorption?

- If a chain starts in transient state, then either:
    1. It leaves a set of transient states with probability of 1
    2. It enters a set of recurrent states with probability of 1

- Absorbing states $i$: the only state of an absorbing state
We have: $i$ absorbing if and only if
$P_{ii}=1$

- Let $i\in S$ be a transient state, $S_e$ a set of recurrent states:

Probability of absorption: $\pi_i(S_e)$ is the probability MC with $X_0=i$ gets absorbed into the set of states $S_e$

- We have
$$ \lim_{n\to\infty} P_{ij}^{(n)} = \pi_i(S_e)\pi_j. $$

 12

## ■ Finite Markov chain

- Finite number of states $|S|=m < \infty$.

- Each finite MC has recurrent states!

- All recurrent states are positive, that is ergodic and has finite expected return time
$$ \lim_{n\to\infty} P_{ii}^{(n)} > 0. $$

- Classification of states: all states are $S=S_m\cup S_e$, where
    - Transient states: $S_m$
    - Ergodic states: $S_e$

- Transient matrix is of a block form:
    - Ergodic states: $1, \ldots, m-K$
    - Transient state: $m-K+1, \ldots, m$
    $$ P = \left[\matrix{S & 0 \cr R & Q}\right] $$

- Fundamental matrix:
$$ N = (I - Q)^{-1}.  $$

 13

```python
# Case 2
c2P = np.array(
    [[0.7, 0.3, 0.0, 0.0],
     [0.2, 0.8, 0.0, 0.0],
     [0.05, 0.15, 0.6, 0.2],
     [0.0, 0.2, 0.0, 0.8]])

print ('Transition matrix P=\n', c2P)

```

    Transition matrix P=
     [[0.7  0.3  0.   0.  ]
     [0.2  0.8  0.   0.  ]
     [0.05 0.15 0.6  0.2 ]
     [0.   0.2  0.   0.8 ]]

```python
# ---------------------------------------------------------------------------------------------------
# Classification of states
print ('Transition matrix P=\n', c2P)
print ('\nTransition matrix P^2=\n', np.linalg.matrix_power(c2P, 2))
print ('\nTransition matrix P^3=\n', np.linalg.matrix_power(c2P, 3))

# Matrix multiplications by blocks
# => 2 x 2 block of zeros stayes the same
# => There are two connected classes of states
```

    Transition matrix P=
     [[0.7  0.3  0.   0.  ]
     [0.2  0.8  0.   0.  ]
     [0.05 0.15 0.6  0.2 ]
     [0.   0.2  0.   0.8 ]]

    Transition matrix P^2=
     [[0.55  0.45  0.    0.   ]
     [0.3   0.7   0.    0.   ]
     [0.095 0.265 0.36  0.28 ]
     [0.04  0.32  0.    0.64 ]]

    Transition matrix P^3=
     [[0.475  0.525  0.     0.    ]
     [0.35   0.65   0.     0.    ]
     [0.1375 0.3505 0.216  0.296 ]
     [0.092  0.396  0.     0.512 ]]

```python
# ---------------------------------------------------------------------------------------------------
# Limit distribuion
eigVals, eigVecs = np.linalg.eig(c2P.T)

# Get it sorted
idx = eigVals.argsort()[::-1]
eigenValues = eigVals[idx]
eigenVectors = eigVecs[:,idx]

# Normalize eigenvector
pi = eigenVectors[:, 0]/sum(eigenVectors[:, 0])

print ('\neigVals=\n', eigenValues)
#print '\neigVecs=\n', eigenVectors
print ('\nLimit distribution =', pi)
```

    eigVals=
     [1.  0.8 0.6 0.5]

    Limit distribution = [ 0.4  0.6 -0.  -0. ]

```python
# ---------------------------------------------------------------------------------------------------
# Decompose transition matrix
# Use Jordan decomposition
#mP = Matrix(c2P)
#mB, mJ = mP.jordan_form()
#B, J = np.array(mB), np.array(mJ)
c2B = np.array([[1.0, 0, 0, 10.0/4],
              [1.0, 0, 0, -3.0/2],
              [1.0, 1.0, 1.0, -7.0/8],
              [1.0, 1.0, 0, 1.0]])
c2J = np.array([[1.0,0,0,0],
              [0,4.0/5,0,0],
              [0,0,3.0/5,0],
              [0,0,0,1.0/2]])

# Test decomposition
err_mat = c2P - np.dot(c2B, np.dot(c2J, np.linalg.inv(c2B)))
print ('\nDecomposition error norm = ', np.linalg.norm(err_mat))
```

    Decomposition error norm =  1.9675159943996247e-16

```python
def prd(a,b):
    if np.isinf(a) and b == 0:
        return 0
    if np.isinf(b) and a == 0:
        return 0
    return a*b

def myDot(A,B):
    nA1, nA2, nB2 = A.shape[0], A.shape[1], B.shape[1]
    AB = np.zeros((nA1,nB2))
    for i in range(nA1):
        for j in range(nB2):
            curr = 0
            for k in range(nA2):
                curr += prd(A[i,k], B[k,j])
            AB[i, j] = curr
    return AB

# Classification of states
# Recurent, transient?

# Eigenvalues
eigVals, eigVecs = np.linalg.eig(c2P.T)

# Get it sorted
idx = eigVals.argsort()[::-1]
eigVa = eigVals[idx]
eigVc = eigVecs[:,idx]

las_bf =  (1./(1-eigVa[1:4]))-1
sumJn = np.diag(np.insert(las_bf, 0, np.inf))

sumPn = myDot(c2B, myDot(sumJn, np.linalg.inv(c2B)))
print ('\nSum P^n\n', sumPn)

# Since diagonal elements of sum P^n i=1 and i=2 are infinity
#  => states i=1 and i=2 are recurrent
# Since diagonal elements of sum P^n i=3 and i=4 are finite
#  => states i=3 and i=4 are transient

```

    Sum P^n
     [[inf inf 0.  0. ]
     [inf inf 0.  0. ]
     [inf inf 1.5 2.5]
     [inf inf 0.  4. ]]

```python
def inf_lim(a_lst):
    return [(0 if la < 1 else 1) for la in a_lst]

# ---------------------------------------------------------------------------------------------------
# Zero or positive states?
# Compute lim of P^n
limJ = np.diag(inf_lim(np.diag(c2J)))
limP = np.dot(c2B, np.dot(limJ, np.linalg.inv(c2B)))

print ('\nlim of P^n = \n', limP)

# Since all lim P^n are positive
# => all states are positive

# Expected times of visits
ETi = 1.0/np.diag(limP)

print ('\nExpected number of visits py states:\n', ETi)

```

    lim of P^n =
     [[0.4 0.6 0.  0. ]
     [0.4 0.6 0.  0. ]
     [0.4 0.6 0.  0. ]
     [0.4 0.6 0.  0. ]]

    Expected number of visits py states:
     [2.5        1.66666667        inf        inf]

    /home/nbuser/anaconda3_501/lib/python3.6/site-packages/ipykernel/__main__.py:16: RuntimeWarning: divide by zero encountered in true_divide

```python
# ---------------------------------------------------------------------------------------------------
# Limit distribuion
eigVals, eigVecs = np.linalg.eig(c2P.T)

# Get it sorted
idx = eigVals.argsort()[::-1]
eigenValues = eigVals[idx]
eigenVectors = eigVecs[:,idx]

# Normalize eigenvector
pi = eigenVectors[:, 0]/sum(eigenVectors[:, 0])

print ('\neigVals=\n', eigenValues)
#print '\neigVecs=\n', eigenVectors
print ('\nLimit distribution =', pi)
```

    eigVals=
    [1.  0.8 0.6 0.5]

    Limit distribution = [ 0.4  0.6 -0.  -0. ]

## ■ Number of visits of a transient state

- Random variable $n_i$: number of visits of a state $i$ in an arbitrary long time

- Expected number of visits of a state $j$ if initial state is $𝑖$:
 $$ E_i[n_j] < \infty $$

- We have: if state $i$ is transient:
$$ E_i[n_j] = \sum_{k=1}^\infty P_{ij}^{(k)} $$

- Also - fundamental matrix gives expected values:
$$ \left[E_i[n_j]\right] = N $$

 14

```python
# Expected number of visits of recurent states
m = 4 # |S|
K = 2 # number of recurent states

Q = c2P[K:m, K:m]
print ('Part of transition matrix Q = \n', Q)

N = np.linalg.inv(np.eye(m-K)-Q)
print ('\nFundamental matrix N =\n', N)
```

    Part of transition matrix Q =
    [[0.6 0.2]
     [0.  0.8]]

    Fundamental matrix N =
    [[2.5 2.5]
     [0.  5. ]]

## 10.3 Transition matrix estimation

■ Transition probability estimation

■ Confidence intervals and quantiles

■ Sample size determination

■ Numerical aspect

■ Case: telecommunication service user modeling

 15

## ■ Transition probability estimation

- Transition probability
$$ p_{ij} = \frac{n_k}{n} $$
is a statistical estimation;

- Parameter estimation
    - estimate and termine
        - expected value $\overline{x}$
        - standard deviation $\sigma$
        - for a selected risk level $\alpha$ compute $z_\alpha$ (if normal distribution $z_{0.05}=1.96$)
    - Confidence interval:
$$ CI = [\overline{x} - z_\alpha\frac{\sigma}{\sqrt{n}}, \overline{x} + z_\alpha\frac{\sigma}{\sqrt{n}}] $$
    - Sample size: determined from a preset confidence interval length
    $$ |CI| = 2 z_\alpha\frac{\sigma}{\sqrt{n}} $$

 16

## ■ Confidence intervals and quantiles

- Confidence interval visualization

 17

## ■ Transition matrix estimation sample size determination

- From confidence interval size

- Confidence interval for risk level $\alpha=0.05$ ($z_\alpha=1.96$)
$$ CI = [p_0 - z_\alpha\sqrt{p_0(1-p_0)/n}, p_0 + z_\alpha\sqrt{p_0(1-p_0)/n}] $$

For $p=0.65$ and $|CI|=\Delta p$ at $\alpha=0.05$
$$ n\geq \frac{4 z_\alpha^2}{\Delta p^2} p_0 (1-p_0) $$
we get
 - for $\Delta p=0.01:$ $n\geq 34959$,
 - for $\Delta p=0.02:$ $n\geq 175$,
 - for $\Delta p=0.04:$ $n\geq 88$.

 18

## ■ Numerical aspect of transition matrix computation

- We need matrix power
$$ P^n = P^{(n)} $$

- Jordan decomposition
$$ P = B J B^{-1},  $$
where $J$ is Jordan canonical form.

- For each class of states we have
$$ J = \lambda I + R $$
and
$$ P^n = B J^n B^{-1} $$

 19

## ■ Case: telecommunication service user modeling

- A set of telecommunication users $U$

- We split them into classes: $U=U_1 \cup U_2 \cup U_3 \cup U_4$
    - Very satisfied: $U_1$
    - Satisfied: $U_2$
    - Not satisfied: $U_3$
    - Churners: $U_4$

- Initial distribution: estimation of users

- Results:
    - Recurrent - transients states
    - Stationary distribution

 20

## ■ Conclusion

- Most useful: finite Markov chain
- Check the transition matrix assumption: stationarity

 21