```python
%%HTML
<style>
.rendered_html table, .rendered_html th, .rendered_html tr, .rendered_html td {
     font-size: 100%;
}
</style>
```

# Metody Numeryczne

## Elementy analizy numerycznej

### dr hab. inż. Jerzy Baranowski, Prof. AGH.

## Informacje ogólne
- Katedra Automatyki i Robotyki, C3, p. 214
- Konsultacje
    - Czwartki 11:00-12:00
    (o ile nie ma Kolegium Wydziałowego lub seminarium)
- jb@agh.edu.pl
- wykłady dostępne tutaj: https://github.com/KAIR-ISZ/public_lectures

# Reprezentacja liczb

## Kod binarny

- Zapis liczby z wykorzystaniem dwóch symboli **1** i **0**
- Podstawa współczesnego sposobu reprezentacji informacji

## Zamierzchła historia

- Pingala, Chandaḥśāstra i Prozodia
    - Ok. 4 wiek pne
    - Wykorzystanie zapisu w formie zer i jedynek do opisu metrum
- Chiny, hexagramy, Shao Yong, I-Ching
- Leibniz

## Algebra Boole'a

$$
\begin{align}
x \land y & = xy & \mathsf{Koniunkcja}\\
x \lor y & = x+y-xy & \mathsf{Alternatywa}\\
\neg x & =1-x & \mathsf{Negacja}\\
x \rightarrow y & = (\neg x\lor y) & \mathsf{Implikacja}\\
x \oplus y & = (x \lor y)\land\neg(x\land y) & \mathsf{EXOR}\\
x = y & = \neg(x\oplus y) & \mathsf{Równoważność}\\
\end{align}
$$

## Nieco mniej zamierzchła historia
- 1937 Shannon – przekaźnikowa realizacja operacji binarnych i algebry Boole’a
- 1937 Stibitz – Pierwszy komputer przekaźnikowy (dodawanie)

## Kod binarny
| **0**   | **0**   | **1**   | **0**   | **1**   | **0**   | **1**   | **1**   |
|---------|---------|---------|---------|---------|---------|---------|---------|
| $2^{7}$ | $2^{6}$ | $2^{5}$ | $2^{4}$ | $2^{3}$ | $2^{2}$ | $2^{1}$ | $2^{0}$ |

Co daje $ =2^5+2^3+2^1+2^0=32+8+2+1=43$

## Liczby naturalne
- Ogólnie zakres od 0 do 2n-1
- 8 bit – zakres od 0 do 255
- 16 bit – zakres od 0 do 65,535 (short, int)
- 32 bit – zakres od 0 do 4,294,967,295 (long)

W Pythonie i matlabie za bardzo nie przejmujemy się typami, chyba że je wymusimy

## Operacje na liczbach binarnych

- Dodawanie
    - 0+0=0
    - 0+1=1
    - 1+0=1
    - 1+1=0, przenieś 1
- Jak w dodawaniu pisemnym

`` 1 1 1 1 1   ``(cyfry przenoszone)
``   0 1 1 0 1 ``(1310)
``+  1 0 1 1 1 ``(2310)
``------------ ``
``=1 0 0 1 0 0 `` (3610)

## Operacje na liczbach binarnych

- Odejmowanie
    - 0-0=0
    - 0-1=1, pożyczka 1
    - 1-0=1
    - 1-1=0,
- Analogicznie

``    *   * * *  ``(pożyczki)
``  1 1 0 1 1 1 0``(11010)
``-     1 0 1 1 1``(2310)
``--------------- ``
``= 1 0 1 0 1 1 1`` (8710)

## Co z liczbami ujemnymi?

Uzupełniamy zapis o tzw. bit znaku

| **1**   | **0**   | **1**   | **0**   | **1**   | **0**   | **1**   | **1**   |
|---------|---------|---------|---------|---------|---------|---------|---------|
| S       | $2^{6}$ | $2^{5}$ | $2^{4}$ | $2^{3}$ | $2^{2}$ | $2^{1}$ | $2^{0}$ |

Co daje $ =(-1)^1(2^3+2^1+2^0)=-(8+2+1)=-11$

Zmieniają się zakresy:
- 8 bit (-128 do 127)
- 16 bit (−32,768 do 32,767)
- itd

## Problemy

- Niepraktyczny zapis
- Trzeba przekodowywać wyniki operacji
- Potencjalnie podatniejsze na błędy

## Kod uzupełnienia do 2 (U2)
| **1**   | **1**   | **1**   | **1**   | **1**   | **0**   | **1**   | **1**   |
|---------|---------|---------|---------|---------|---------|---------|---------|
| $-2^{7}$| $2^{6}$ | $2^{5}$ | $2^{4}$ | $2^{3}$ | $2^{2}$ | $2^{1}$ | $2^{0}$ |

Co daje $ =-2^7+2^6+2^5+2^4+2^3+2^1+2^0$

$=-128+64+32+16+8+2+1=-5$

## Bardzo łatwa konwersja
- Liczby dodatnie są takie same jak były
- Aby zamienić liczbę na jej przeciwną wystarczy zanegować wszystkie bity i do wyniku dodać 1 (*w obie strony*)

| **0**    | **0**   | **0**   | **0**   | **0**   | **1**   | **0**   | **1**   | 510  | oryginał  |
|----------|---------|---------|---------|---------|---------|---------|---------|-----------------|-----------|
| 1    | 1   | 1   | 1   | 1   | 0   | 1   | 0   |                 | negacja   |
| **1**    | **1**   | **1**   | **1**   | **1**   | **0**   | **1**   | **1**   | -510 | dodanie 1 |
| 0    | 0   | 0   | 0   | 0   | 1   | 0   | 0   |                 | negacja   |
| **0**    | **0**   | **0**   | **0**   | **0**   | **1**   | **0**   | **1**   | 510  | dodanie 1 |
| -$2^{7}$ | $2^{6}$ | $2^{5}$ | $2^{4}$ | $2^{3}$ | $2^{2}$ | $2^{1}$ | $2^{0}$ |                 |           |

## Jaka z tego korzyść?
- Odejmowanie staje się dodawaniem (prawie)
$$ A - B = A + \neg B + 1$$
- Przykład 13 – 7 (na 8 bitach)

``  1 1 1 1     1  ``(cyfry przenoszone)
``  0 0 0 0 1 1 0 1``(1310)
``  1 1 1 1 1 0 0 0``(zanegowane 710)
``+               1``(jedynka)
``-----------------``
``= 0 0 0 0 0 1 1 0`` (610)

## Operacje na liczbach binarnych
Mnożenie również przypomina mnożenie pisemne

``          1 0 1 1`` 1110
``        * 1 0 1 0`` 1010
``      -----------``
``          0 0 0 0``
``  +     1 0 1 1   ``
``  +   0 0 0 0``
``  + 1 0 1 1``
``  ---------------``
``  = 1 1 0 1 1 1 0`` 11010

# Metody Numeryczne

## Reprezentacja liczb wymiernych

### dr hab. inż. Jerzy Baranowski, Prof. AGH.

## A co z ułamkami?
Są dwa sposoby zapisu liczb niecałkowitych
- Stałoprzecinkowy (stałopozycyjny)
- Zmiennoprzecinkowy (zmiennopozycyjny)

## Zapis stałoprzecinkowy
| **1**   | **0**   | **1**   | **1**   | **1**   | **0**   | **0**   | **0**   |
|---------|---------|---------|---------|---------|---------|---------|---------|
| $2^{1}$ | $2^{0}$ | $2^{-1}$ | $2^{-2}$ | $2^{-3}$ | $2^{-4}$ | $2^{-5}$ | $2^{-6}$ |

$$
2^1+2^{-1}+2^{-2}+2^{-3}=2+\frac{1}{2}+\frac{1}{4}+\frac{1}{8}=2.875
$$

## Zalety zapisu stałoprzecinkowego
- Nie ma różnicy w kodowaniu
- Mamy stale określoną dokładność, którą możemy w miarę dokładnie kształtować
- Stosunkowa prostota
- Małe wymagania sprzętowe

## Wady zapisu stałoprzecinkowego
Problemy z dokładnością, np. nie da się dokładnie przedstawić liczby 0.1
- Na 3 bitach części ułamkowej różnica wynosi 0.025
- Na 7 bitach części ułamkowej różnica wynosi ok. 0.001

## Jak wykonujemy działania?
- Działania wykonujemy traktując zapis liczby stałoprzecinkowej jako normalną binarną
- Kod U2 dalej działa
- Należy pamiętać, że wtedy liczba jest pomnożona przez 2n gdzie n to ilość bitów części ułamkowej
- W liczbach poddanych działaniu liczba bitów części całkowitej i ułamkowej musi być równa

## Działania stałoprzecinkowe
- Dodawanie wykonujemy identycznie
- W przypadku mnożenia wynik musimy podzielić przez 2n
- Mnożenie liczb stałoprzecinkowych przez potęgę 2 polega tylko na przesuwaniu bitów (bardzo proste w realizacji)

| 1             | 0             | 1   | 1   | 1   | 0   | 0   | 0   |   |
|---------------|---------------|-----|-----|-----|-----|-----|-----|---|
| 0             | 0             | 1   | 0   | 1   | 1   | 1   | 0   | Podzielenie przez $2^{2}$  |
| $2^{1}$ | $2^{0}$ | $2^{-1}$ | $2^{-2}$ | $2^{-3}$ | $2^{-4}$ | $2^{-5}$ | $2^{-6}$ ||

## Format zmiennoprzecinkowy
- Bardziej zaawansowany sposób przedstawiania liczb
- Ustandaryzowany normą IEEE
- Dający pod pewnymi względami większą dokładność

## Format zmiennoprzecinkowy

Reprezentacja liczby

$$
x=S\cdot M\cdot B^E
$$

- S – znak (*sign*)
- M – mantysa (*mantissa*, także *fraction*)
- B – podstawa (*base*, zazwyczaj 2, rzadziej 10)
- E - wykładnik (*exponent*)

## Mantysa
- Liczba odpowiadająca za ułamkową część zapisu
- Format stałoprzecinkowy, zazwyczaj liczba z przedziału [1,2)

## Podstawa i wykładnik

- Pozwalają na określenie szerokiego zakresu
- Ze względu na kodowanie, zazwyczaj podstawa to 2
- Wykładnik może być ujemny lub dodatni.
- Wykładnik koduje się w U2, lub też wprowadza się przesunięcie

## Działania na liczbach zmiennoprzecinkowych
Dodawanie i odejmowanie

$$
x_1\pm x_2=\left(M_1\pm M_2\cdot B^{E_2-E_1}\right)\cdot B^{E_1}
$$

Mnożenie i dzielenie

$$
x_1\cdot x_2=(S_1\cdot S_2)\cdot (M_1\cdot M_2)\cdot B^{E_1+E_2}
$$

$$
x_1 / x_2=(S_1\cdot S_2)\cdot (M_1/ M_2)\cdot B^{E_1-E_2}
$$

## Dzielenie
- Mając możliwość zapisu liczby ulamkowej można sformułować operację dzielenia.
- Istnieje wiele algorytmów np.
    - *restoring division*
    - *non-restoring division*
    - SRT
    - algorytm Newtona-Raphsona
    - algorytm Goldschmidta
- Są one już zaimplementowane, jedno dzielenie zazwyczaj wymaga przeprowadzenia 3-4 mnożeń

## Ważne formaty – IEEE Single precision

- 8 bitów wykładnika, wykładnik przesunięty o 127 (zamiana z -126 do 127 na 1 do 244)
- 24 bity mantysy, ale zawsze koduje się tylko 23 po kropce, przed kropką jest 1
- Specjalne zapisy nieskończoności i błędów
- w NumPy - ``float32``

## Ważne formaty – IEEE Double precision

- 11 bitów wykładnika, wykładnik przesunięty o 1023 (zamiana z -1022 do 1023 na 1 do 2046)
- 53 bity mantysy, ale zawsz koduje się tylko 52 po kropce, przed kropką jest 1
- Specjalne zapisy nieskończoności i błędów
- w NumPy - ``float64``, ale w zasadzie każda liczba w Pythonie i Matlabie to double, chyba że wymusimy inaczej

## Wyświetlanie liczb
- Normalnie
- Notacja inżynierska
    - $3700=3.7\cdot10^3$, $0.12=120\cdot10^{-3}$
- Notacja naukowa
    - ``3700=3.7E3``, ``0.12=1.2E-1``

# Metody Numeryczne

## Błedy numeryczne

### dr hab. inż. Jerzy Baranowski, Prof. AGH.

## Podstawowe definicje
Wartość dokładna
$$y=\tilde{y}+\varepsilon$$
- $\tilde{y}$ - wartość przybliżona
- $\varepsilon$ - błąd

## Błąd bezwzględny
Wartość bezwzględna różnicy między rozwiązaniem dokładnym i przybliżonym
$$ \varepsilon=|y-\tilde{y}|$$

## Błąd względny
Stosunek błędu bezwzględnego do wartości bezwzględnej rozwiązania
$$\eta=\frac{|y-\tilde{y}|}{|y|}=\left|\frac{y-\tilde{y}}{y}\right|=\left|1-\frac{\tilde{y}}{y}\right|$$
Czasami błąd względny wyrażamy w procentach

## Przykłady
Pierwiastek kwadratowy ze 122

$$
\begin{align}
y{}&=\sqrt{122}\approx 11.04536\\
\tilde{y}{}&=11\\
\varepsilon{}&=|y-\tilde{y}|=0.04536\\
\eta{}&=\frac{|y-\tilde{y}|}{|y|}=0.00411
\end{align}
$$

## Przykłady
Liczba obywateli Polski (stan na ostatni spis powszechny z 2011)

$$
\begin{align}
y{}&=38\ 538\ 447\\
\tilde{y}{}&=38\ 500\ 000\\
\varepsilon{}&=|y-\tilde{y}|=38\ 447\\
\eta{}&=\frac{|y-\tilde{y}|}{|y|}=9.97627\cdot10^{-4}\approx 0.001
\end{align}
$$

## Przykłady
Obliczanie stałej grawitacji
$$
\begin{align}
y{}&=6.673841\cdot10^{-11}\\
\tilde{y}{}&=6.7\cdot10^{-11}\\
\varepsilon{}&=|y-\tilde{y}|=2.6159\cdot10^{-13}\\
\eta{}&=\frac{|y-\tilde{y}|}{|y|}=0.00391
\end{align}
$$

## Źródła błędów
Błędy powstające przy formułowaniu zagadnienia
- Błędy pomiaru
- Błędy wynikające z przyjęcia określonych przybliżeń opisu zjawisk fizycznych

Błędy powstające przy obliczeniach
- Błędy grube (pomyłki)
- Błędy metody (obcięcia)
- Błędy zaokrągleń

## Błędy grube
- Błąd przy wpisywaniu wzoru do komputera
np. ``x=A/b`` zamiast ``x=A\b``
- Zła implementacja algorytmu
- Niewłaściwa kolejność wykonywania działań

## Błędy metody (obcięcia)
- Błędy obcięcia są nieodłącznym elementem obliczeń numerycznych.
- Błąd obcięcia jest to błąd wynikający z tego, że do uzyskania dokładnego rozwiązania potrzebujemy wykonać nieskończenie wiele obliczeń

## Przykłady błędów metody
Można wykazać, że
$$
\begin{align}
\sin x={}&x-\frac{x^3}{3!}+\frac{x^5}{5!}-\frac{x^7}{7!}+\ldots=\\
={}&\sum\limits_{n=0}^\infty(-1)^n\frac{x^{2n+1}}{(2n+1)!}
\end{align}
$$
Błędem odcięcia będzie
$$
\sin x\approx x-\frac{x^3}{3!}+\frac{x^5}{5!}
$$

## Przykłady błędów metody

Metoda bisekcji

```python
def bisection(f,a,b,N):
    a_n = a
    b_n = b
    for n in range(1,N+1):
        m_n = (a_n + b_n)/2
        f_m_n = f(m_n)
        if f(a_n)*f_m_n < 0:
            a_n = a_n
            b_n = m_n
        elif f(b_n)*f_m_n < 0:
            a_n = m_n
            b_n = b_n
    return (a_n + b_n)/2
```

Szukamy pierwiastka wielomianu $x^2-2$, w przedziale $[1,2]$. Rozwiązanie to $\sqrt{2}$.

```python
f = lambda x: x**2 - 2 # definicja funkcji
bisection(f,1,2,5) # 5 kroków
```

    1.421875

```python
bisection(f,1,2,10) # 10 kroków
```

    1.41455078125

```python
bisection(f,1,2,15) # 15 kroków
```

    1.4141998291015625

```python
import numpy as np
np.sqrt(2)
```

    1.4142135623730951

## Błąd metody - podsumowanie
- Praktycznie wszystkie metody numeryczne mają jakiś błąd metody
- Dobre algorytmy podają jednak jego oszacowanie, w ten sposób wiemy jak daleko jesteśmy od rozwiązania nawet jak przerwiemy obliczenia

# Metody Numeryczne

## Błędy zaokrągleń

### dr hab. inż. Jerzy Baranowski, Prof. AGH.

## Błędy zaokrągleń
Kolejne nieusuwalne w pełni źródło błędów, nad którym mamy mniejszą kontrolę niż nad błędem metody

## Zaokrąglenie i cyfry znaczące
Liczba $\tilde{y}=\mathrm{rd}(y)$ jest poprawnie zaokrąglona do *d* miejsc po przecinku, jeżeli

$$
\varepsilon=|y-\tilde{y}|\leq\frac{1}{2}\cdot10^{-d}
$$
*k*-tą cyfrę dziesiętną liczby $\tilde{y}$ nazwiemy znaczącą gdy
$$|y-\tilde{y}|\leq\frac{1}{2}\cdot10^{-k}$$
oraz
$$|\tilde{y}|\geq10^{-k}
$$

## Rzeczywiste obliczenia zmiennoprzecinkowe
$$
\begin{align}
\mathrm{fl}(x+y)={}&\mathrm{rd}(x+y)\\
\mathrm{fl}(x-y)={}&\mathrm{rd}(x-y)\\
\mathrm{fl}(x\cdot y)={}&\mathrm{rd}(x\cdot y)\\
\mathrm{fl}(x/y)={}&\mathrm{rd}(x/y)\\
\end{align}
$$

## Liczby maszynowe
- Liczba maszynowa, to taka liczba jaką można przedstawić w komputerze. Zbiór tych liczb oznaczamy A
- Dokładność maszynową (epsilon maszynowy) – eps, $\varepsilon_m$,  definiujemy:
$$
\mathrm{eps}=\min\{x\in{A}\colon \mathrm{fl}(1+x)>1,\ x>0\}
$$
Innymi słowy, jest to najmniejsza liczba, którą możemy dodać do 1, aby uzyskać coś większego od 1.

## Epsilon maszynowy w różnych formatach

Zależy on od liczby bitów na część ułamkową
- Single precision $\varepsilon_m=2^{-24}\approx 5.96\cdot10^{-8}$
- Double precision $\varepsilon_m=2^{-52}\approx 1.11\cdot10^{-16}$

### Przykład

```python
a=10**(-15)
b=10**(-17)
1+a>1,1+b>1
```

    (True, False)

## Maksymalny błąd reprezentacji
Dla każdej liczby rzeczywistej $x$ istnieje taka liczba $\varepsilon$, taka że $|\varepsilon|<\varepsilon_m$, że
$\mathrm{fl}(x)=x(1+\varepsilon)$

Oznacza to, że **błąd względny między liczbą rzeczywistą, a jej najbliższą reprezentacją zmiennoprzecinkową jest zawsze mniejszy od $\varepsilon_m$**

## Lemat Wilkinsona
Błedy zaokrągleń powstałe podczas wykonywania działań zmiennoprzecinkowych są równoważne zastępczemu zaburzeniu liczb, na których wykonujemy działania

$$
\begin{align}
\mathrm{fl}(x+y)={}&(x+y)(1+\varepsilon_1)\\
\mathrm{fl}(x-y)={}&(x-y)(1+\varepsilon_2)\\
\mathrm{fl}(x\cdot y)={}&(x\cdot y)(1+\varepsilon_3)\\
\mathrm{fl}(x/y)={}&(x/y)(1+\varepsilon_4)\\
|\varepsilon_i|<{}&\varepsilon_m
\end{align}
$$
(dla każdej pary liczb $x,\ y$ zaburzenia zastępcze $\varepsilon_i$ są inne)

## Konsekwencja lematu Wilkinsona
Prawa łączności i rozdzielności operacji matematycznych są ogólnie nieprawdziwe dla obliczeń zmiennoprzecinkowych

### Przykład

```python
a=np.float32(0.23371258*10**(-4))
b=np.float32(0.33678429*10**(2))
c=np.float32(-0.33677811*10**(2))
print([a,b,c])
```

    [2.3371258e-05, 33.67843, -33.67781]

Chcemy obliczyć ``a+b+c``

## Obliczenia

```python
## Podejście 1
d=b+c
wynik_1=a+d
print(wynik_1)
```

    0.0006413522

```python
## Podejście 2
e=a+b
wynik_2=e+c
print(wynik_2)
```

    0.00064086914

## Co tu się porobiło?

## Konsekwencje obliczen zmiennoprzecinkowych

```python
m_a, e_a = np.frexp(a)
print(m_a,e_a)
m_b,e_b = np.frexp(b)
print(m_b,e_b)
m_c,e_c = np.frexp(c)
print(m_c,e_c)
```

    0.7658294 -15
    0.52622545 6
    -0.5262158 6

Wykładnik ``a`` od wykładników ``b`` i ``c`` różni się o 21. Oznacza to, że z 23 bitów mantysy liczby ``a`` po sprowadzeniu do wspólnego wykładnika z ``b`` zostaną nam tylko 2 najbardziej znaczące.

## Konsekwencje cd..
Jeżeli dodajemy małą liczbę do dużej, zawsze musimy się liczyć z zaokrągleniem i to normalne. W tym przypadku jednak dwie duże liczby ``b`` i ``c`` są przeciwnych znaków i bliskie co do wartości bezwzględnej. Wynik tego działania:

```python
m_d,e_d = np.frexp(d)
print(m_d,e_d)
print(wynik_2)
```

    0.6328125 -10
    0.00064086914

W konsekwencji dodając ``a`` do ``d`` na zaokrągleniu stracimy jedynie 5 bitów mantysy ``a``.

## O ile się pomyliliśmy (w stosunku do dokładniejszych obliczeń)

```python
a_dbl=(0.23371258*10**(-4))
b_dbl=(0.33678429*10**(2))
c_dbl=(-0.33677811*10**(2))
d_dbl=b_dbl+c_dbl
wynik_dbl=a_dbl+d_dbl
epsilon_1=np.abs((wynik_1)-wynik_dbl)
eta_1=epsilon_1/np.abs(wynik_dbl)
print("Metoda 1: Błąd bezwzględny %10.2e, Błąd względny %10.2e"%(epsilon_1,eta_1))
epsilon_2=np.abs((wynik_2)-wynik_dbl)
eta_2=epsilon_2/np.abs(wynik_dbl)
print("Metoda 2: Błąd bezwzględny %10.2e, Błąd względny %10.2e"%(epsilon_2,eta_2))

```

    Metoda 1: Błąd bezwzględny   1.91e-08, Błąd względny   2.97e-05
    Metoda 2: Błąd bezwzględny   5.02e-07, Błąd względny   7.83e-04

# Przenoszenie się błędów zaokrągleń
Korzystając z rachunku różniczkowego (różniczkowa analiza błędów) możemy podać wzór na przenoszenie się błędów.

Niech $y=\varphi(x_1,\ x_2,,\ldots\ x_n)$ będzie wielkością, którą chcemy obliczyć a $x_i$ są zaokrąglone z błędem $\varepsilon_{x_i}$. Błąd względny wyliczania $y$ wynosi w przybliżeniu:

$$
\varepsilon_y = \sum_{i=0}^n \frac{x_i}{\varphi(\mathbf{x})}
\cdot \frac{\partial\varphi(\mathbf{x})}{\partial x_i}\cdot\varepsilon_{x_i}
$$

# Nieunikniony błąd obliczeń
Ze względu na zaokrąglenia pewnych błędów nigdy nie unikniemy. Nieunikniony błąd wartości składa się z błędu wyliczenia wartości (przeniesienia błędów) oraz samego błędu zaokrąglenia:

$$
\frac{\Delta y}{y} = \epsilon_y + \mathrm{eps}
$$

## Przykład
Wyliczanie pierwiastka równania kwadratowego $y^2+2py-q=0$ o mniejszej wartości bezwzględnej:
$$ y=-p+\sqrt{p^2+q} $$
można policzyć, że
$$
\varepsilon_y=-\frac{p}{\sqrt{p^2-q}}\varepsilon_p+\frac{p+\sqrt{p^2-q}}{2\sqrt{p^2-q}}\varepsilon_q
$$

## Analiza błędu nieuniknionego
Ponieważ dla $q>0$ mamy

$$
\left|\frac{p}{\sqrt{p^2-q}}\right|\leq1,\quad \left|\frac{p+\sqrt{p^2-q}}{2\sqrt{p^2-q}}\right|\leq1
$$
to wtedy mamy (przyjmując, że nie zachodzi $p^2\approx q$)
$$
\mathrm{eps}\leq\left|\frac{\Delta y}{y}\right| = |\epsilon_y + \mathrm{eps}|\leq 3 \mathrm{eps}
$$

## Porównanie algorytmów
Rozpartrzmy dwa sposoby wyliczania $y$ dla $p$ i $q$ mniejszych od zera

$$
\begin{aligned}
s:={}&p^2\\
t:={}&s+q\\
u:={}&\sqrt{t}\\
y:={}&-p+q
\end{aligned}
\quad \quad \quad \quad
\begin{aligned}
s:={}&p^2\\
t:={}&s+q\\
u:={}&\sqrt{t}\\
v:={}&p+u\\
y:={}&q/v
\end{aligned}
$$

## Algorytm 1
Podstawowym źródłem błędu będzie wzmocnienie błędu zaokrąglenia wyliczania pierwiastka z $t$ poprzez odejmowanie dwóch liczb przy wyliczaniu $y$
$$\varepsilon_y=\frac{p\sqrt{p^2+q}+p^2+q}{q}\varepsilon=\kappa\varepsilon$$
$\kappa$ można oszacować z dołu, przez
$$
\kappa>\frac{2 p^2}{q} >0
$$
co oznacza, że dla małych $q$ błąd obliczeń będzie dużo większy niż błąd nieunikniony.

## Algorytm 2
W tym algorytmie zakokrąglenie przez odejmowanie nie wystąpi

$$
\varepsilon_y = -\frac{\sqrt{p^2+q}}{p+\sqrt{p^2+q}}\varepsilon = \kappa\varepsilon
$$
w tym przypadku zawsze $|\kappa|<1$.

```python
def algorytm_1(p,q):
    s=p**2
    t=s+q
    u=np.sqrt(t)
    return u-p

def algorytm_2(p,q):
    s=p**2
    t=s+q
    u=np.sqrt(t)
    v=p+u
    return q/v
```

# Porównanie obliczeń

```python
p=1000
q=0.018000000081
exact_sol=np.max(np.roots([1,2*p,-q]))
```

```python
epsilon_1=np.abs((algorytm_1(p,q))-exact_sol)
eta_1=epsilon_1/np.abs(exact_sol)
epsilon_2=np.abs((algorytm_2(p,q))-exact_sol)
eta_2=epsilon_2/np.abs(exact_sol)
```

```python
print('Algorytm 1')
print(algorytm_1(p,q))
print('Algorytm 2')
print(algorytm_2(p,q))
print('Rozwiązanie dokładne')
print(exact_sol)
print("Algorytm 1: Błąd bezwzględny %10.2e, Błąd względny %10.2e"%(epsilon_1,eta_1))
print("Algorytm 2: Błąd bezwzględny %10.2e, Błąd względny %10.2e"%(epsilon_2,eta_2))
```

    Algorytm 1
    8.999999977277184e-06
    Algorytm 2
    9e-06
    Rozwiązanie dokładne
    9e-06
    Algorytm 1: Błąd bezwzględny   2.27e-14, Błąd względny   2.52e-09
    Algorytm 2: Błąd bezwzględny   0.00e+00, Błąd względny   0.00e+00

# Metody Numeryczne

## Ocena algorytmów numerycznych

### dr hab. inż. Jerzy Baranowski, Prof. AGH.

## Notacja O duże
- Mówimy, że dla wielkości zależnej od parametru np. $F(n)$ zachodzi
$$
F(n)=O(G(n))
$$
jeżeli istnieje taka stała $C$, że przy $n$ zmierzającym do nieskończoności (odpowiednio dużym), mamy
$$F(n)≤C G(n)$$
- Jeżeli interesuje nas $O(c)$, gdzie $c$ jest stałą, zależność ta ma zachodzić niezależnie od wielkości parametru.
- Mówimy potocznie, gdy błąd jest równy $O(n^2)$, że błąd jest rzędu $n^2$

## Ocena algorytmu
- Naszym celem jest obliczenie pewnej wielkości $f(x)$, zależnej od danych wejściowych $x$
- W przypadku obliczeń komputerowych zawsze mamy do czynienia z obliczaniem przybliżonym stąd algorytm obliczania $f(x)$ będziemy oznaczać jako $f^*(x)$
- Dane w komputerze również są reprezentowane w sposób zaokrąglony, więc będziemy je oznaczać jako $x^*$

## Uwarunkowanie problemu

- Mówimy, że problem $f(x)$ jest dobrze uwarunkowany, jeżeli mała zmiana $x$ powoduje małą zmianę w $f(x)$
- Problem jest źle uwarunkowany, jeżeli mała zmiana $x$ powoduje dużą zmianę w $f(x)$
- Miarą uwarunkowania jest stała $\kappa$ (kappa), która (nieformalnie) określa największy iloraz zaburzeń $f(x)$  wywołanych przez najmniejsze zaburzenia $x$.
- Stałą $\kappa$ można wyliczyć tylko w niektórych probemach

## Dokładność algorytmu
- Algorytm jest dokładny, jeżeli
$$
\frac{\Vert f^*(x)-f(x) \Vert}{\Vert f(x)\Vert}=O(\varepsilon_m)
$$
- Zagwarantowanie, że algorytm jest dokładny wg tej definicji jest niezwykle trudne, zwłaszcza dla źle uwarunkowanych problemów

## Stabilność algorytmu

Mówimy, że algorytm jest stabilny, gdy dla każdego $x$, zachodzi
$$
\frac{\Vert f^*(x)-f(x^*) \Vert}{\Vert f(x^*)\Vert}=O(\varepsilon_m)
$$
dla takich $x^*$, że
$$\frac{\Vert x-x^* \Vert}{\Vert x\Vert}=O(\varepsilon_m)$$
Innymi słowy
**Stabilny algorytm daje prawie dobrą odpowiedź na prawie dobre pytanie**

## Stabilność wsteczna algorytmu
Algorytm jest stabilny wstecznie, jeżeli dla każdego $x$, zachodzi
$$f^*(x)=f(x^*)$$
dla takich $x^*$, że
$$\frac{\Vert x-x^* \Vert}{\Vert x\Vert}=O(\varepsilon_m)$$
Innymi słowy
**Stabilny wstecznie algorytm daje prawidłową odpowiedź na prawie dobre pytanie**

## Dokładność algorytmów stabilnych wstecznie przy złym uwarunkowaniu
Jeśli algorytm jest stabilny wstecznie, to jego błąd względny pogarsza się proporcjonalnie do stałej uwarunkowania tj. $O(\kappa\varepsilon_m)$

# Metody Numeryczne

## Problemy techniczne

### dr hab. inż. Jerzy Baranowski, Prof. AGH.

Metody Numeryczne 2019/Lecture 1 (errors and stuff)/Lecture  1.ipynb


```python
%matplotlib inline
```

```python
#environment setup with watermark
%load_ext watermark
%watermark -a 'Gopala KR' -u -d -v -p watermark,numpy,pandas,matplotlib,nltk,sklearn,tensorflow,theano,mxnet,chainer
```

    WARNING (theano.tensor.blas): Using NumPy C-API based implementation for BLAS functions.

    Gopala KR
    last updated: 2018-01-30

    CPython 3.6.3
    IPython 6.2.1

    watermark 1.6.0
    numpy 1.13.1
    pandas 0.20.3
    matplotlib 2.0.2
    nltk 3.2.5
    sklearn 0.19.0
    tensorflow 1.3.0
    theano 1.0.1
    mxnet 1.0.0
    chainer 3.3.0

# The Johnson-Lindenstrauss bound for embedding with random projections

The `Johnson-Lindenstrauss lemma`_ states that any high dimensional
dataset can be randomly projected into a lower dimensional Euclidean
space while controlling the distortion in the pairwise distances.

Theoretical bounds
==================

The distortion introduced by a random projection `p` is asserted by
the fact that `p` is defining an eps-embedding with good probability
as defined by:

\begin{align}(1 - eps) \|u - v\|^2 < \|p(u) - p(v)\|^2 < (1 + eps) \|u - v\|^2\end{align}

Where u and v are any rows taken from a dataset of shape [n_samples,
n_features] and p is a projection by a random Gaussian N(0, 1) matrix
with shape [n_components, n_features] (or a sparse Achlioptas matrix).

The minimum number of components to guarantees the eps-embedding is
given by:

\begin{align}n\_components >= 4 log(n\_samples) / (eps^2 / 2 - eps^3 / 3)\end{align}

The first plot shows that with an increasing number of samples ``n_samples``,
the minimal number of dimensions ``n_components`` increased logarithmically
in order to guarantee an ``eps``-embedding.

The second plot shows that an increase of the admissible
distortion ``eps`` allows to reduce drastically the minimal number of
dimensions ``n_components`` for a given number of samples ``n_samples``

Empirical validation
====================

We validate the above bounds on the digits dataset or on the 20 newsgroups
text document (TF-IDF word frequencies) dataset:

- for the digits dataset, some 8x8 gray level pixels data for 500
  handwritten digits pictures are randomly projected to spaces for various
  larger number of dimensions ``n_components``.

- for the 20 newsgroups dataset some 500 documents with 100k
  features in total are projected using a sparse random matrix to smaller
  euclidean spaces with various values for the target number of dimensions
  ``n_components``.

The default dataset is the digits dataset. To run the example on the twenty
newsgroups dataset, pass the --twenty-newsgroups command line argument to this
script.

For each value of ``n_components``, we plot:

- 2D distribution of sample pairs with pairwise distances in original
  and projected spaces as x and y axis respectively.

- 1D histogram of the ratio of those distances (projected / original).

We can see that for low values of ``n_components`` the distribution is wide
with many distorted pairs and a skewed distribution (due to the hard
limit of zero ratio on the left as distances are always positives)
while for larger values of n_components the distortion is controlled
and the distances are well preserved by the random projection.

Remarks
=======

According to the JL lemma, projecting 500 samples without too much distortion
will require at least several thousands dimensions, irrespective of the
number of features of the original dataset.

Hence using random projections on the digits dataset which only has 64 features
in the input space does not make sense: it does not allow for dimensionality
reduction in this case.

On the twenty newsgroups on the other hand the dimensionality can be decreased
from 56436 down to 10000 while reasonably preserving pairwise distances.

```python
print(__doc__)

import sys
from time import time
import numpy as np
import matplotlib.pyplot as plt
from sklearn.random_projection import johnson_lindenstrauss_min_dim
from sklearn.random_projection import SparseRandomProjection
from sklearn.datasets import fetch_20newsgroups_vectorized
from sklearn.datasets import load_digits
from sklearn.metrics.pairwise import euclidean_distances

# Part 1: plot the theoretical dependency between n_components_min and
# n_samples

# range of admissible distortions
eps_range = np.linspace(0.1, 0.99, 5)
colors = plt.cm.Blues(np.linspace(0.3, 1.0, len(eps_range)))

# range of number of samples (observation) to embed
n_samples_range = np.logspace(1, 9, 9)

plt.figure()
for eps, color in zip(eps_range, colors):
    min_n_components = johnson_lindenstrauss_min_dim(n_samples_range, eps=eps)
    plt.loglog(n_samples_range, min_n_components, color=color)

plt.legend(["eps = %0.1f" % eps for eps in eps_range], loc="lower right")
plt.xlabel("Number of observations to eps-embed")
plt.ylabel("Minimum number of dimensions")
plt.title("Johnson-Lindenstrauss bounds:\nn_samples vs n_components")

# range of admissible distortions
eps_range = np.linspace(0.01, 0.99, 100)

# range of number of samples (observation) to embed
n_samples_range = np.logspace(2, 6, 5)
colors = plt.cm.Blues(np.linspace(0.3, 1.0, len(n_samples_range)))

plt.figure()
for n_samples, color in zip(n_samples_range, colors):
    min_n_components = johnson_lindenstrauss_min_dim(n_samples, eps=eps_range)
    plt.semilogy(eps_range, min_n_components, color=color)

plt.legend(["n_samples = %d" % n for n in n_samples_range], loc="upper right")
plt.xlabel("Distortion eps")
plt.ylabel("Minimum number of dimensions")
plt.title("Johnson-Lindenstrauss bounds:\nn_components vs eps")

# Part 2: perform sparse random projection of some digits images which are
# quite low dimensional and dense or documents of the 20 newsgroups dataset
# which is both high dimensional and sparse

if '--twenty-newsgroups' in sys.argv:
    # Need an internet connection hence not enabled by default
    data = fetch_20newsgroups_vectorized().data[:500]
else:
    data = load_digits().data[:500]

n_samples, n_features = data.shape
print("Embedding %d samples with dim %d using various random projections"
      % (n_samples, n_features))

n_components_range = np.array([300, 1000, 10000])
dists = euclidean_distances(data, squared=True).ravel()

# select only non-identical samples pairs
nonzero = dists != 0
dists = dists[nonzero]

for n_components in n_components_range:
    t0 = time()
    rp = SparseRandomProjection(n_components=n_components)
    projected_data = rp.fit_transform(data)
    print("Projected %d samples from %d to %d in %0.3fs"
          % (n_samples, n_features, n_components, time() - t0))
    if hasattr(rp, 'components_'):
        n_bytes = rp.components_.data.nbytes
        n_bytes += rp.components_.indices.nbytes
        print("Random matrix with size: %0.3fMB" % (n_bytes / 1e6))

    projected_dists = euclidean_distances(
        projected_data, squared=True).ravel()[nonzero]

    plt.figure()
    plt.hexbin(dists, projected_dists, gridsize=100, cmap=plt.cm.PuBu)
    plt.xlabel("Pairwise squared distances in original space")
    plt.ylabel("Pairwise squared distances in projected space")
    plt.title("Pairwise distances distribution for n_components=%d" %
              n_components)
    cb = plt.colorbar()
    cb.set_label('Sample pairs counts')

    rates = projected_dists / dists
    print("Mean distances rate: %0.2f (%0.2f)"
          % (np.mean(rates), np.std(rates)))

    plt.figure()
    plt.hist(rates, bins=50, normed=True, range=(0., 2.), edgecolor='k')
    plt.xlabel("Squared distances rate: projected / original")
    plt.ylabel("Distribution of samples pairs")
    plt.title("Histogram of pairwise distance rates for n_components=%d" %
              n_components)

    # TODO: compute the expected value of eps and add them to the previous plot
    # as vertical lines / region

plt.show()
```

```python

```

```python
test complete; Gopal
```


# Machine learning compilation of quantum circuits
> Optimal compiling of unitaries reaching the theoretical lower bound

- toc: true
- badges: true
- comments: true
- categories: [machine learning, compilation, qiskit, paper review]
- image: images/grovercirc.png

# Introduction

I am going to review a recent [preprint](http://arxiv.org/abs/2106.05649) by Liam Madden and
Andrea Simonetto that uses techniques from machine learning to tackle the problem of quantum circuits compilation. I find the approach suggested in the paper very interesting and the preliminary results quite promising.

## What is compilation?
> Note that a variety of terms are floating around the literature and used more or less interchangibly. Among those are **synthesis**, **compilation**, **transpilation** and **decomposition** of quantum circuits. I will not make a distinction and try to stick to **compilation**.

But first things first, what is a compilation of a quantum circuit? The best motivation and illustration for the problem is the following. Say you need to run a textbook quantum circuit on a real hardware. The real hardware usually allows only for a few basic one and two qubit gates. In contrast, your typical textbook quantum circuit may feature (1) complex many-qubit gates, for example multi-controlled gates and (2) one and two qubit gates which are not supported by the hardware. As a simple example take this 3-qubit Grover's circuit (from [qiskit textbook](https://qiskit.org/textbook/ch-algorithms/grover.html)):

```python
# collapse
#initialization
import matplotlib.pyplot as plt
import numpy as np

# importing Qiskit
from qiskit import IBMQ, Aer, assemble, transpile
from qiskit import QuantumCircuit, ClassicalRegister, QuantumRegister
from qiskit.providers.ibmq import least_busy

# import basic plot tools
from qiskit.visualization import plot_histogram

def initialize_s(qc, qubits):
    """Apply a H-gate to 'qubits' in qc"""
    for q in qubits:
        qc.h(q)
    return qc

def diffuser(nqubits):
    qc = QuantumCircuit(nqubits)
    # Apply transformation |s> -> |00..0> (H-gates)
    for qubit in range(nqubits):
        qc.h(qubit)
    # Apply transformation |00..0> -> |11..1> (X-gates)
    for qubit in range(nqubits):
        qc.x(qubit)
    # Do multi-controlled-Z gate
    qc.h(nqubits-1)
    qc.mct(list(range(nqubits-1)), nqubits-1)  # multi-controlled-toffoli
    qc.h(nqubits-1)
    # Apply transformation |11..1> -> |00..0>
    for qubit in range(nqubits):
        qc.x(qubit)
    # Apply transformation |00..0> -> |s>
    for qubit in range(nqubits):
        qc.h(qubit)
    # We will return the diffuser as a gate
    U_s = qc.to_gate()
    U_s.name = "U$_s$"
    return U_s

qc = QuantumCircuit(3)
qc.cz(0, 2)
qc.cz(1, 2)
oracle_ex3 = qc.to_gate()
oracle_ex3.name = "U$_\omega$"

n = 3
grover_circuit = QuantumCircuit(n)
grover_circuit = initialize_s(grover_circuit, [0,1,2])
grover_circuit.append(oracle_ex3, [0,1,2])
grover_circuit.append(diffuser(n), [0,1,2])
grover_circuit = grover_circuit.decompose()
grover_circuit.draw(output='mpl')
```

The three qubit gates like Toffoli are not generally available on a hardware and one and two qubit gates my be different from those in the textbook algorithm. For example ion quantum computers are good with [Mølmer–Sørensen gates](https://en.wikipedia.org/wiki/M%C3%B8lmer%E2%80%93S%C3%B8rensen_gate) and may need several native one qubit gates to implement the Hadamard gate.

Additional important problem is to take into account qubit connectivity. Usually textbook algorithms assume full connectivity, meaning that two-qubit gates can act on any pair of qubits. On most hardware platforms however a qubit can only interact with its neighbors. Assuming that one and two qubits gates available on the hardware can implement a SWAP gate between adjacent qubits, to solve the connectivity problem one can insert as many SWAPs as necessary to connect topologically disjoint qubits. Using SWAPs however leads to a huge overhead in the number of total gates in the compiled circuit, and it is of much importance use them as economically as possible. In fact, the problem of optimal SWAPping alone in generic situation is [NP-complete](https://scholar.google.com/scholar?hl=en&as_sdt=0%2C5&q=on+the+complexity+of+quantum+circuit+compilation&btnG=).

## Simplified problem
When compiling a quantum circuit one has to decide which resulting circuits are considered to be efficient. Ideally, one should optimize for the total fidelity of the circuit. Let us imagine running the algorithm on a real device. Probably my theorist's image of a real device is still way too platonic, but I will try my best. Many details need to be taken into account. For example, gates acting on different qubits or pairs of qubits may have different fidelities. Decoherence of qubits with time can make circuits where many operations can be executed in parallel more favorable. Cross-talk (unwanted interactions) between neighboring qubits may lead to exotic patterns for optimal circuits. A simple proxy for the resulting fidelity that is often adopted is the number of two-qubit gates (which are generically much less accurate than a single-qubit gates). So the problem that is often studied, and that is addressed in the preprint we are going to discuss, is the problem of optimal compilation into a gate set consisting of arbitrary single-qubit gates and CNOTs, the only two qubits gate. The compiled circuit must

1. Respect hardware connectivity.
1. Have as few CNOTs as possible.
1. Exceed a given fidelity threshold.

Last item here means that we also allow for an approximate compilation. By increasing the number of CNOTs one can always achieve an exact compilation, but since in reality each additional CNOT comes with its own fidelity cost this might not be a good trade-off. Note also that a specific choice for two-qubit gate is made, a CNOT gate. Any two-qubit gate can be decomposed into at most 3 CNOTs [see e.g. here](https://arxiv.org/pdf/quant-ph/0308006.pdf), so in terms of computational complexity this is of course inconsequential. However in the following discussion we will care a lot about constant factors and may wish to revisit this choice at the end.

## Existing results

Since finding the exact optimal solution to the compilation problem is intractable, as with many things in life one needs to resort to heuristic methods. A combination of many heuristic methods, in fact. As an example one can check out the [transpilation workflow](https://qiskit.org/documentation/apidoc/transpiler.html) in `qiskit`. Among others, there is a step that compiles >2 qubit gates into one and two qubit gates; the one that tries to find a good initial placement of the logical qubits onto physical hardware; the one that 'routes' the desired circuit to match a given topology being as greedy on SWAPs as possible. Each of these steps can use several different heuristic optimization algorithms, which are continuously refined and extended (for example this [recent preprint](https://arxiv.org/abs/2106.06446) improves on the default rounting procedure in `qiskit`). In my opinion it would be waay better to have one unified heuristic for all steps of the process, especially taking into account that they are not completely independent. Although this might be too much to ask for, some advances are definitely possible and machine learning tools might prove very useful. The paper we are going to discuss is an excellent demonstration.

## Theoretical lower bound and quantum Shannon decomposition
There is a couple of very nice theoretical results about the compilation problem that I need to mention. But first, let us agree that we will compile unitaries, not circuits.  What is the difference? Of course, any quantum circuit (without measurements and neglecting losses) corresponds to a unitary matrix. However, to compute that unitary matrix for a large quantum circuit explicitly is generally an intractable problem, precisely for the same reasons that quantum computation is assumed to be more powerful than classical. Still, taking as the input a unitary matrix (which is in general hard to compute from the circuit) is very useful both theoretically and practically. I will discuss pros and cons of this approach later on.

OK, now the fun fact. Generically, one needs at least this many CNOTs

\begin{align}
 L:=\# \text{CNOTs} \geq \frac14\left(4^n-3n-1\right) \label{TLB}
\end{align}

to exactly compile an $n$-qubit unitary. 'Generically' means that the set of $n$-qubit unitaries that can be compiled exactly with smaller amount of CNOTs has measure zero. Keep in mind though, that there are important unitaries in this class like multi-controlled gates or qubit permutations. We will discuss compilation of some gates from the 'measure-zero' later on.

The authors of the preprint (I hope you and me still remember that there is some actual results to discuss, not just my overly long introduction to read) refer to \eqref{TLB} as the theoretical lower bound or TLB for short. The proof of this [fact](https://dl.acm.org/doi/10.5555/968879.969163) is actually rather simple and I will sketch it. A general $d\times d$ unitary has $d^2$ real parameters. For $n$ qubits $d=2^n$. Single one-qubit gate has 3 real parameters. Any sequence of one-qubit gates applied to the same qubit can be reduced to a single one-qubit gate and hence can have no more than 3 parameters. That means, that without CNOTs we can only have 3n parameters in our circuit, 3 for each one-qubit gate. This is definitely not enough to describe an arbitrary unitary on $n$ qubits which has $d^2=4^n$ parameters.

Now, adding a single CNOT allows to insert two more 1-qubit unitaries after it, like that

```python
#collapse
from qiskit.circuit import Parameter

a1, a2, a3 = [Parameter(a) for a in ['a1', 'a2', 'a3']]
b1, b2, b3 = [Parameter(b) for b in ['b1', 'b2', 'b3']]

qc = QuantumCircuit(2)
qc.cx(0, 1)
qc.u(a1, a2, a3, 0)
qc.u(b1, b2, b3, 1)

qc.draw(output='mpl')
```

At the first glance this allows to add 6 more parameters. However, each single-qubit unitary can be represented via the Euler angles as a product of only $R_z$ and $R_x$ rotations either as $U=R_z R_x R_z$ or $U=R_x R_y R_z$ (I do not specify angles). Now, CNOT can be represented as $CNOT=|0\rangle\langle 0|\otimes I+|1\rangle\langle 1|\otimes X$. It follows that $R_z$ commutes with the control of CNOT and $R_x$ commutes with the target of CNOT, hence they can be dragged to the left and joined with preceding one-qubit gates. So in fact each new CNOT gate allows to add only 4 real parameters:

```python
#collapse
a1, a2 = [Parameter(a) for a in ['a1', 'a2']]
b1, b2 = [Parameter(b) for b in ['b1', 'b2']]

qc = QuantumCircuit(2)
qc.cx(0, 1)
qc.rx(a1, 0)
qc.rz(a2, 0)
qc.rz(b1, 1)
qc.rx(b2, 1)

qc.draw(output='mpl')
```

 That's it, there are no more caveats. Thus, the total number of parameters we can get with $L$ CNOTs is $3n+4L$ and we need to describe a $d\times d$ unitary which has $4^n$ parameters. In fact, the global phase of the unitary is irrelevant so we only need $3n+4L \geq 4^n-1$. Solving for $L$ gives the TLB \eqref{TLB}. That's pretty cool, isn't it?

Now there is an algorithm, called *quantum Shannon decomposition* (see [ref](https://arxiv.org/abs/quant-ph/0406176)), which gives an exact compilation of any unitary with the number of CNOTs twice as much as the TLB requires. In complexity-theoretic terms an overall factor of two is of course inessential, but for current NISQ devices we want to get as efficient as possible. Moreover, to my understanding the quantum Shannon decomposition is not easily extendable to restricted topology while inefficient generalizations lead to a much bigger overhead (roughly an order of magnitude).

# What's in the preprint?
## Templates
I've already wrote an introduction way longer than intended so from now on I will try to be brief and to the point. The authors of the preprint propose two templates inspired by the quantum Shannon decomposition. The building block for each template is a 'CNOT unit'

```python
#collapse
a1, a2 = [Parameter(a) for a in ['a1', 'a2']]
b1, b2 = [Parameter(b) for b in ['b1', 'b2']]

qc = QuantumCircuit(2)
qc.cx(0, 1)
qc.ry(a1, 0)
qc.rz(a2, 0)
qc.ry(b1, 1)
qc.rx(b2, 1)

qc.draw(output='mpl')
```

First template is called **sequ** in the paper and is obtained as follows. There are $n(n-1)/2$ different CNOTs on $n$-qubit gates. We enumerate them somehow and simply stack sequentially. Here is a 3-qubut example with two layers (I use `qiskit` gates `cz` instead of our 'CNOT units' for the ease of graphical representation)

```python
#collapse
qc = QuantumCircuit(3)
for _ in range(2):
    qc.cz(0, 1)
    qc.cz(0, 2)
    qc.cz(1, 2)
    qc.barrier()
qc.draw(output='mpl')
```

The second template is called **spin** and for 4 qubits looks as follows

```python
#collapse
qc = QuantumCircuit(4)
for _ in range(2):
    qc.cz(0, 1)
    qc.cz(1, 2)
    qc.cz(2, 3)
    qc.barrier()
qc.draw(output='mpl')
```

I'm sure you get the idea. That's it! The templates fix the pattern of CNOTs while angles of single-qubit gates are adjustable parameters which are collectively denoted by $\theta$.

The idea now is simple. Try to optimize these parameters to achieve the highest possible fidelity for a given target unitary to compile. I am not at all an expert on the optimization methods, so I might miss many subtleties, but on the surface the problem looks rather straightforward. You can choose your favorite flavor of the gradient descent and hope for convergence. The problem appears to be non-convex but the gradient descent seems to work well in practice. One technical point that I do not fully understand is that the authors choose to work with fidelity defined by the Frobenius norm $||U-V||_F^2$ which is sensitive to the global phase of each unitary. To my understanding they often find that local minima of this fidelity coincides with the global minimum up to a global phase. OK, so in the rest of the post I refer to the 'gradient descent' as the magic numerical method which does good job of finding physically sound minimums.

## Results
### Compiling random unitaries
OK, finally, for the surprising results. The authors find experimentally that both **sequ** and **spin** perform surprisingly well on random unitaries always coming very close to the TLB \eqref{TLB} with good fidelity. More precisely, the tests proceed as follows. First, one generates a random unitary. Next, for each number $L$ of CNOTs below the TLB one runs the gradient descent to see how much fidelity can be achieved with this amount of CNOTs. Finally, one plots the fidelity as a function of $L$. Impressively, on the sample of hundred unitaries the fidelity always approaches 100% when the number of CNOTs reaches the TLB. For the $n=3$ qubits TLB is $L=14$, for $n=5$ $L=252$ (these are the two cases studied). So, in all cases studied, the gradient descent lead by the provided templates seems to always find the optimal compilation circuit! Recall that this is two times better than quantum Shannon decomposition. Please see the original paper for nice plots that I do not reproduce here.

### Compiling on restricted topology
These tests were performed on the fully connected circuits. The next remarkable discovery is that restricting the connectivity does not to seem to harm the performance of the compilation! More precisely, the authors considered two restricted topologies in the paper, 'star' where all qubits are connected to single central one and 'line' where well, they are connected by links on a line. The **spin** template can not be applied to star topology, but it can be applied to line topology. The **sequ** template can be generalized to any topology by simply omitting CNOTs that are not allowed. Again, as examining a hundred of random unitaries on $n=3$ and $n=5$ qubits shows, the fidelity nearing 100% can be achieved right at the TLB in all cases, which hints that topology restriction may not be a problem in this approach at all! To appreciate the achievement, imagine decomposing each unitary via the quantum Shannon decomposition and then routing on restricted topology with swarms of SWAPs, a terrifying picture indeed. It would be interesting to compare the results against the performance of `qiskit` transpiler which is unfortunately not done in the paper to my understanding.

### Compiling specific 'measure zero' gates
Some important multi-qubit gates fall into the 'measure zero' set which can be compiled with a smaller amount of CNOTs than is implied by the TLB \eqref{TLB}. For example, 4-qubit Toffoli gate can be compiled with 14 CNOTs while the TLB requires 61 gates. Numerical tests show that the plain version of the algorithm presented above does not generically obtain the optimal compilation for special gates. However, with some tweaking and increasing the amount of attempts the authors were able to find optimal decompositions for a number of known gates such as 3- and 4-qubit Toffoli, 3-qubit Fredkin and 1-bit full adder on 4 qubits. The tweaking included randomly changing the orientation of some CNOTs (note that in both **sequ** and **spin** the control qubit is always at the top) and running many optimization cycles with random initial conditions. The best performing method appeared to be **sequ** with random flips of CNOTs. The whole strategy might look a bit fishy, but I would argue that it is not. My argument is simple: you only need to find a good compilation of the 4-qubit Toffoli *once*. After that you pat yourself on the back and use the result in all your algorithms. So it does not really matter how hard it was to find the compilation as long as you did not forget to write it down.

### Compressing the quantum Shannon decomposition
Finally, as a new twist on the plot the authors propose a method to compress the standard quantum Shannon decomposition (which is twice the TLB, remember?). The idea seems simple and works surprisingly well. The algorithm works as follows.
1. Compile a unitary exactly using the quantum Shannon decomposition.
1. Promote parameters in single-qubit gates variables (they have fixed values in quantum Shannon decomposition).
2. Add [LASSO](https://en.wikipedia.org/wiki/Lasso_(statistics)-type regularization term, which forces one-qubit gates to have small parameters, ideally zero (which makes the corresponding gates into identities).
3. Run a gradient descent on the regularized cost function (fidelity+LASSO term). Some one-qubit gates will become identity after that (one might need to tune the regularization parameter here).
4. After eliminating identity one-qubit gates one can end up in the situation where there is a bunch of CNOTs with no single-qubit gates in between. There are efficient algorithms for reducing the amount of CNOTs in this case.
5. Recall that the fidelity was compromised by adding regularization terms. Run the gradient descent once more, this time without regularization, to squeeze out these last percents of fidelity.

From the description of this algorithm it does not appear obvious that the required cancellations (elimination of single-qubit gates and cancellations in resulting CNOT clusters) is bound to happen, but the experimental tests show that they do. Again, from a bunch of random unitaries it seems that the $\times 2$ reduction to the TLB is almost sure to happen! Please see the preprint for plots.

## Weak spots
Although I find results of the paper largely impressive, a couple of weak spots deserve a mention.
### Limited scope of experiments
The numerical experiments were only carried out for $n=3$ and $n=5$ qubits which of course is not much. To see if the method keeps working as the number of qubits is scaled is sure very important. There may be two promblems. First, the templates can fail to be expressive enough for larger circuits. The authors hope to attack this problem from the theoretical side and show that the templates do fill the space of unitaries. Well, best of luck with that! Another potential problem is that although the templates work fine for higher $n$, the learning part might become way more challenging. Well, I guess we should wait and see.
### Unitary as the input
As I discussed somewhere way above, for a realistic quantum computation we can not know the unitary matrix that we need to compile. If we did, there would no need in the quantum computer in the first place. I can make two objects here. First, we are still in the NISQ era and pushing the existing quantum computers to their edge is a very important task. Even if an algorithm can be simulated classically, running it on a real device might be invaluable. Second, even quantum circuits on 1000 qubits do not usually feature  100-qubit unitaries. So it could be possible to separate a realistic quantum circuit into pieces, each containing only a few qubits, and compile them separately.

# Final remarks
To me, the algorithms presented in the preprint seem to be refreshingly efficient and universal. At some level it appears to be irrelevant which exact template do we use. Near the theoretical lower bound they all perform similarly well, even on restricted topology. This might be a justification for choosing CNOT as the two-qubit gate, as this probably does not matter in the end! I'm really cheering for a universal algorithm like that to win the compilation challenge over a complicated web of isolated heuristics, which are currently state of the art.

_notebooks/2021-07-22-Machine learning compilation of quantum circuits.ipynb


# Data Analysis and Machine Learning:

**Christian Forssén**, Department of Physics, Chalmers University of Technology, Sweden

 **Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University

Date: **Dec 23, 2020**

Copyright 1999-2020, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license

# Elements of Bayesian theory and Bayesian Neural Networks

## Why Bayesian Statistics?

We have already made ourselves familiar with elements of a statistical
data analysis via quantities like the bias-variance tradeoff as well
as some central distribution functions such as the Normal
distribution, the binomial distribution and other probability
distribution functions.

In essentially all the Machine Learning algorithms we have studied,
our focus has been on a so-called **frequentist approach**, where
knowledge of an underlying likelihood function has not been
emphasized. Our data, whether we had a classification or a regression
problem, have been our central points of departure.

Here we wish to merge this approach with the derivation of a
likelihood function which can be used to make prediction on how our
system under study evolves.  We will venture into the realm of what is
called Bayesian Neural Networks. To get an overarching view on what
this entails, the following figure conveys the essential differences
between a standard Neural network that we have met earlier and a
Bayesian Neural Network. In order to get there, we need to present
some of the basic elements of Bayesian statistics, starting with the
product rule and Bayes' theorem.

## Inference
Inference:
  :
  "the act of passing from one proposition, statement or judgment considered as true to another whose truth is believed to follow from that of the former" (Webster)
  Do premises $A, B, \ldots \to$ hypothesis, $H$?

Deductive inference:
  :
  Premises allow definite determination of truth/falsity of H (syllogisms, symbolic logic, Boolean algebra)
  $B(H|A,B,...) = 0$ or $1$

Inductive inference:
  :
  Premises bear on truth/falsity of H, but don’t allow its definite determination (weak syllogisms, analogies)
  $A, B, C, D$ share properties $x, y, z$; $E$ has properties $x, y$
  $\to$ $E$ probably has property $z$.

## Statistical Inference
* Quantify the strength of inductive inferences from facts, in the form of data ($D$), and other premises, e.g. models, to hypotheses about the phenomena producing the data.

* Quantify via probabilities, or averages calculated using probabilities. Frequentists ($\mathcal{F}$) and Bayesians ($\mathcal{B}$) use probabilities very differently for this.

* To the pioneers such as Bernoulli, Bayes and Laplace, a probability represented a *degree-of-belief* or plausability: how much they thought that something as true based on the evidence at hand. This is the Bayesian approach.

* To the 19th century scholars, this seemed too vague and subjective. They redefined probability as the *long run relative frequency* with which an event occurred, given (infinitely) many repeated (experimental) trials.

## Some history
Adapted from D.S. Sivia[^Sivia]:

[^Sivia]: Sivia, Devinderjit, and John Skilling. Data Analysis : A Bayesian Tutorial, OUP Oxford, 2006

> Although the frequency definition appears to be more objective, its range of validity is also far more limited. For example, Laplace used (his) probability theory to estimate the mass of Saturn, given orbital data that were available to him from various astronomical observatories. In essence, he computed the posterior pdf for the mass M , given the data and all the relevant background information I (such as a knowledge of the laws of classical mechanics): prob(M|{data},I); this is shown schematically in the figure [Fig. 1.2].

> To Laplace, the (shaded) area under the posterior pdf curve between $m_1$ and $m_2$ was a measure of how much he believed that the mass of Saturn lay in the range $m_1 \le M \le m_2$. As such, the position of the maximum of the posterior pdf represents a best estimate of the mass; its width, or spread, about this optimal value gives an indication of the uncertainty in the estimate. Laplace stated that: ‘ . . . it is a bet of 11,000 to 1 that the error of this result is not 1/100th of its value.’ He would have won the bet, as another 150 years’ accumulation of data has changed the estimate by only 0.63%!

> According to the frequency definition, however, we are not permitted to use probability theory to tackle this problem. This is because the mass of Saturn is a constant and not a random variable; therefore, it has no frequency distribution and so probability theory cannot be used.
>
> If the pdf [of Fig. 1.2] had to be interpreted in terms of the frequency definition, we would have to imagine a large ensemble of universes in which everything remains constant apart from the mass of Saturn.

> As this scenario appears quite far-fetched, we might be inclined to think of [Fig. 1.2] in terms of the distribution of the measurements of the mass in many repetitions of the experiment. Although we are at liberty to think about a problem in any way that facilitates its solution, or our understanding of it, having to seek a frequency interpretation for every data analysis problem seems rather perverse.
> For example, what do we mean by the ‘measurement of the mass’ when the data consist of orbital periods? Besides, why should we have to think about many repetitions of an experiment that never happened? What we really want to do is to make the best inference of the mass given the (few) data that we actually have; this is precisely the Bayes and Laplace view of probability.

> Faced with the realization that the frequency definition of probability theory did not permit most real-life scientific problems to be addressed, a new subject was invented — statistics! To estimate the mass of Saturn, for example, one has to relate the mass to the data through some function called the statistic; since the data are subject to ‘random’ noise, the statistic becomes the random variable to which the rules of probability the- ory can be applied. But now the question arises: How should we choose the statistic? The frequentist approach does not yield a natural way of doing this and has, therefore, led to the development of several alternative schools of orthodox or conventional statis- tics. The masters, such as Fisher, Neyman and Pearson, provided a variety of different principles, which has merely resulted in a plethora of tests and procedures without any clear underlying rationale. This lack of unifying principles is, perhaps, at the heart of the shortcomings of the cook-book approach to statistics that students are often taught even today.

## The Bayesian recipe
Assess hypotheses by calculating their probabilities $p(H_i | \ldots)$ conditional on known and/or presumed information using the rules of probability theory.

Probability Theory Axioms:
Product (AND) rule :
  :
  $p(A, B | I) = p(A|I) p(B|A, I) = p(B|I)p(A|B,I)$
  Should read $p(A,B|I)$ as the probability for propositions $A$ AND $B$ being true given that $I$ is true.

Sum (OR) rule:
  :
  $p(A + B | I) = p(A | I) + p(B | I) - p(A, B | I)$
  $p(A+B|I)$ is the probability that proposition $A$ OR $B$ is true given that $I$ is true.

Normalization:
  :
  $p(A|I) + p(\bar{A}|I) = 1$
  $\bar{A}$ denotes the proposition that $A$ is false.

## Bayes' theorem
Bayes' theorem follows directly from the product rule

$$
$$
p(A|B,I) = \frac{p(B|A,I) p(A|I)}{p(B|I)}.
$$
$$

The importance of this property to data analysis becomes apparent if we replace $A$ and $B$ by hypothesis($H$) and data($D$):

$$
\begin{equation}
p(H|D,I) = \frac{p(D|H,I) p(H|I)}{p(D|I)}.
\label{eq:bayes} \tag{1}
\end{equation}
$$

The power of Bayes’ theorem lies in the fact that it relates the quantity of interest, the probability that the hypothesis is true given the data, to the term we have a better chance of being able to assign, the probability that we would have observed the measured data if the hypothesis was true.

The various terms in Bayes’ theorem have formal names.
* The quantity on the far right, $p(H|I)$, is called the *prior* probability; it represents our state of knowledge (or ignorance) about the truth of the hypothesis before we have analysed the current data.

* This is modified by the experimental measurements through $p(D|H,I)$, the *likelihood* function,

* The denominator $p(D|I)$ is called the *evidence*. It does not depend on the hypothesis and can be regarded as a normalization constant.

* Together, these yield the *posterior* probability, $p(H|D, I )$, representing our state of knowledge about the truth of the hypothesis in the light of the data.

In a sense, Bayes’ theorem encapsulates the process of learning.

## The friends of Bayes' theorem
Normalization:
  :
  $\sum_i p(H_i|\ldots) = 1$.

Marginalization:
  :
  $\sum_i p(A,H_i|I) = \sum_i p(H_i|A,I) p(A|I) = p(A|I)$.

Marginalization (continuum limit):
  :
  $\int dx p(A,H(x)|I) = p(A|I)$.

In the above, $H_i$ is an exclusive and exhaustive list of hypotheses. For example,let’s imagine that there are five candidates in a presidential election; then $H_1$ could be the proposition that the first candidate will win, and so on. The probability that $A$ is true, for example that unemployment will be lower in a year’s time (given all relevant information $I$, but irrespective of whoever becomes president) is then given by $\sum_i p(A,H_i|I)$.

In the continuum limit of propositions we must understand $p(\ldots)$ as a pdf (probability density function).

Marginalization is a very powerful device in data analysis because it enables us to deal with nuisance parameters; that is, quantities which necessarily enter the analysis but are of no intrinsic interest. The unwanted background signal present in many experimental measurements are examples of nuisance parameters.

## Inference With Parametric Models
Inductive inference with parametric models is a very important tool in the natural sciences.
* Consider $N$ different models $M_i$ ($i = 1, \ldots, N$), each with parameters $\boldsymbol{\alpha}_i$. Each of them implies a sampling distribution (conditional predictive distribution for possible data)

$$
$$
p(D|\boldsymbol{\alpha}_i, M_i)
$$
$$

* The $\boldsymbol{\alpha}_i$ dependence when we fix attention on the actual, observed data ($D_\mathrm{obs}$) is the likelihood function:

$$
$$
\mathcal{L}_i (\boldsymbol{\alpha}_i) \equiv p(D_\mathrm{obs}|\boldsymbol{\alpha}_i, M_i)
$$
$$

* We may be uncertain about $i$ (model uncertainty),

* or uncertain about $\boldsymbol{\alpha}_i$ (parameter uncertainty).

Parameter Estimation:
  :
  Premise = choice of model (pick specific $i$)
  $\Rightarrow$ What can we say about $\boldsymbol{\alpha}_i$?

Model comparison:
  :
  Premise = $\{M_i\}$
  $\Rightarrow$ What can we say about $i$?

Model adequacy:
  :
  Premise = $M_1$
  $\Rightarrow$ Is $M_1$ adequate?

Hybrid Uncertainty:
  :
  Models share some common params: $\boldsymbol{\alpha}_1 = \{ \boldsymbol{\varphi}, \boldsymbol{\eta}_i\}$
  $\Rightarrow$ What can we say about $\boldsymbol{\varphi}$? (Systematic error is an example)

## Illustrative examples with python code
* Is this a fair coin? (analytical)

* Flux from a star (single parameter, MCMC)

* The lighthouse problem (two parameters, MCMC)

* Linear fit with outliers (nuisance parameters)

* ...

## Example: Is this a fair coin?
Let us begin with the analysis of data from a simple coin-tossing experiment.
Given that we had observed 6 heads in 8 flips, would you think it was a fair coin? By fair, we mean that we would be prepared to lay an even 1 : 1 bet on the outcome of a flip being a head or a tail. If we decide that the coin was fair, the question which follows naturally is how sure are we that this was so; if it was not fair, how unfair do we think it was? Furthermore, if we were to continue collecting data for this particular coin, observing the outcomes of additional flips, how would we update our belief on the fairness of the coin?

A sensible way of formulating this problem is to consider a large number of hypotheses about the range in which the bias-weighting of the coin might lie. If we denote the bias-weighting by $H$, then $H = 0$ and $H = 1$ can represent a coin which produces a tail or a head on every flip, respectively. There is a continuum of possibilities for the value of H between these limits, with $H = 0.5$ indicating a fair coin. Our state of knowledge about the fairness, or the degree of unfairness, of the coin is then completely summarized by specifying how much we believe these various propositions to be true.

Let us perform a computer simulation of a coin-tossing experiment. This provides the data that we will be analysing.

0

<
<
<
!
!
C
O
D
E
_
B
L
O
C
K

p
y
c
o
d

```
np.random.seed(999)         # for reproducibility
a=0.6                       # biased coin
flips=np.random.rand(2**12) # simulates 4096 coin flips
heads=flips<a               # boolean array, heads[i]=True if flip i is heads
```

In the light of this data, our inference about the fairness of this coin is summarized by the conditional pdf: $p(H|D,I)$. This is, of course, shorthand for the limiting case of a continuum of propositions for the value of $H$; that is to say, the probability that $H$ lies in an infinitesimally narrow range is given by $p(H|D,I) dH$.

To estimate this posterior pdf, we need to use Bayes’ theorem ([1](#eq:bayes)). We will ignore the denominator $p(D|I)$ as it does not involve bias-weighting explicitly, and it will therefore not affect the shape of the desired pdf. At the end we can evaluate the missing constant subsequently from the normalization condition

$$
\begin{equation}
\int_0^1 p(H|D,I) dH = 1.
\label{eq:coin_posterior_norm} \tag{2}
\end{equation}
$$

The prior pdf, $p(H|I)$, represents what we know about the coin given only the information $I$ that we are dealing with a ‘strange coin’. We could keep a very open mind about the nature of the coin; a simple probability assignment which reflects this is a uniform, or flat, prior

$$
\begin{equation}
p(H|I) = \left\{ \begin{array}{ll}
1 & 0 \le H \le 1, \\
0 & \mathrm{otherwise}.
\end{array} \right.
\label{eq:coin_prior_uniform} \tag{3}
\end{equation}
$$

We will get back later to the choice of prior and its effect on the analysis.

This prior state of knowledge, or ignorance, is modified by the data through the likelihood function $p(D|H,I)$. It is a measure of the chance that we would have obtained the data that we actually observed, if the value of the bias-weighting was given (as known). If, in the conditioning information $I$, we assume that the flips of the coin were independent events, so that the outcome of one did not influence that of another, then the probability of obtaining the data `R heads in N tosses' is given by the binomial distribution (we leave a formal definition of this to a statistics textbook)

$$
\begin{equation}
p(D|H,I) \propto H^R (1-H)^{N-R}.
\label{_auto1} \tag{4}
\end{equation}
$$

It seems reasonable because $H$ is the chance of obtaining a head on any flip, and there were $R$ of them, and $1-H$ is the corresponding probability for a tail, of which there were $N-R$. We note that this binomial distribution also contains a normalization factor, but we will ignore it since it does not depend explicitly on $H$, the quantity of interest. It will be absorbed by the normalization condition ([2](#eq:coin_posterior_norm)).

We perform the setup of this Bayesian framework on the computer.

```
def prior(H):
    p=np.zeros_like(H)
    p[(0<=x)&(x<=1)]=1      # allowed range: 0<=H<=1
    return p                # uniform prior
def likelihood(H,data):
    N = len(data)
    no_of_heads = sum(data)
    no_of_tails = N - no_of_heads
    return H**no_of_heads * (1-H)**no_of_tails
def posterior(H,data):
    p=prior(H)*likelihood(H,data)
    norm=np.trapz(p,H)
    return p/norm
```

The next step is to confront this setup with the simulated data. To get a feel for the result, it is instructive to see how the posterior pdf evolves as we obtain more and more data pertaining to the coin. The results of such an analyses is shown in Fig. [fig:coinflipping](#fig:coinflipping).

```
x=np.linspace(0,1,100)
fig, axs = plt.subplots(nrows=4,ncols=3,sharex=True,sharey='row')
axs_vec=np.reshape(axs,-1)
axs_vec[0].plot(x,prior(x))
for ndouble in range(11):
    ax=axs_vec[1+ndouble]
    ax.plot(x,posterior(x,heads[:2**ndouble]))
    ax.text(0.1, 0.8, '$N={0}$'.format(2**ndouble), transform=ax.transAxes)
for row in range(4): axs[row,0].set_ylabel('$p(H|D_\mathrm{obs},I)$')
for col in range(3): axs[-1,col].set_xlabel('$H$')
```

The evolution of the posterior pdf for the bias-weighting of a coin, as the number of data available increases. The figure on the top left-hand corner of each panel shows the number of data included in the analysis.

The panel in the top left-hand corner shows the posterior pdf for $H$ given no data, i.e., it is the same as the prior pdf of Eq. ([3](#eq:coin_prior_uniform)). It indicates that we have no more reason to believe that the coin is fair than we have to think that it is double-headed, double-tailed, or of any other intermediate bias-weighting.

The first flip is obviously tails. At this point we have no evidence that the coin has a side with heads, as indicated by the pdf going to zero as $H \to 1$. The second flip is obviously heads and we have now excluded both extreme options $H=0$ (double-tailed) and $H=1$ (double-headed). We can note that the posterior at this point has the simple form $p(H|D,I) = H(1-H)$ for $0 \le H \le 1$.

The remainder of Fig. [fig:coinflipping](#fig:coinflipping) shows how the posterior pdf evolves as the number of data analysed becomes larger and larger. We see that the position of the maximum moves around, but that the amount by which it does so decreases with the increasing number of observations. The width of the posterior pdf also becomes narrower with more data, indicating that we are becoming increasingly confident in our estimate of the bias-weighting. For the coin in this example, the best estimate of $H$ eventually converges to 0.6, which, of course, was the value chosen to simulate the flips.

## A few words on different priors
* uniform

* Gaussian

* Jeffrey's prior

Repeat the coin flipping experiment with other priors.

## Bayesian parameter estimation (single parameter)
We will now consider the very important task of model parameter estimation using statistical inference.
[CF 1: maybe stress that model parameters are not random variables, and the meaning of parameter estimation is therefore very different between frequentist and bayesian approaches.]

Throughout this section we will consider a specific example that involves a model with a single parameter: "Measured flux from a star".

### Example: Measured flux from a star

Adapted from the blog [Pythonic Perambulations](http://jakevdp.github.io) by Jake VanderPlas.

Imagine that we point our telescope to the sky, and observe the light coming from a single star. For the time being, we'll assume that the star's true flux is constant with time, i.e. that is it has a fixed value $F_\mathrm{true}$ (we'll also ignore effects like sky noise and other sources of systematic error). We'll assume that we perform a series of $N$ measurements with our telescope, where the ith measurement reports the observed photon flux $F_i$ and error $e_i$[^errors].
The question is, given this set of measurements $D = \{F_i, e_i\}$, what is our best estimate of the true flux $F_\mathrm{true}$?

[^errors]: We'll make the reasonable assumption that errors are Gaussian. In a Frequentist perspective, $e_i$ is the standard deviation of the results of a single measurement event in the limit of repetitions of *that event*. In the Bayesian perspective, $e_i$ is the standard deviation of the (Gaussian) probability distribution describing our knowledge of that particular measurement given its observed value.

Because the measurements are number counts, a Poisson distribution is a good approximation to the measurement process:

```
np.random.seed(1)      # for repeatability
F_true = 1000          # true flux, say number of photons measured in 1 second
N = 50                 # number of measurements
F = stats.poisson(F_true).rvs(N)
                       # N measurements of the flux
e = np.sqrt(F)         # errors on Poisson counts estimated via square root
```

Now let's make a simple visualization of the "observed" data, see Fig. [fig:flux](#fig:flux).

```
fig, ax = plt.subplots()
ax.errorbar(F, np.arange(N), xerr=e, fmt='ok', ecolor='gray', alpha=0.5)
ax.vlines([F_true], 0, N, linewidth=5, alpha=0.2)
ax.set_xlabel("Flux");ax.set_ylabel("measurement number");
```

Single photon counts (flux measurements).

These measurements each have a different error $e_i$ which is estimated from Poisson statistics using the standard square-root rule. In this toy example we already know the true flux $F_\mathrm{true}$, but the question is this: given our measurements and errors, what is our best estimate of the true flux?

Let's take a look at the frequentist and Bayesian approaches to solving this.

### Simple Photon Counts: Frequentist Approach

We'll start with the classical frequentist maximum likelihood approach. Given a single observation $D_i = (F_i, e_i)$, we can compute the probability distribution of the measurement given the true flux Ftrue given our assumption of Gaussian errors

$$
\begin{equation}
p(D_i | F_\mathrm{true}, I) = \frac{1}{\sqrt{2\pi e_i^2}} \exp \left( \frac{-(F_i-F_\mathrm{true})^2}{2e_i^2} \right).
\label{_auto2} \tag{5}
\end{equation}
$$

This should be read "the probability of $D_i$ given $F_\mathrm{true}$
equals ...". You should recognize this as a normal distribution with mean $F_\mathrm{true}$ and standard deviation $e_i$.

We construct the *likelihood function* by computing the product of the probabilities for each data point

$$
\begin{equation}
\mathcal{L}(D | F_\mathrm{true}, I) = \prod_{i=1}^N p(D_i | F_\mathrm{true}, I),
\label{_auto3} \tag{6}
\end{equation}
$$

here $D = \{D_i\}$ represents the entire set of measurements. Because the value of the likelihood can become very small, it is often more convenient to instead compute the log-likelihood. Combining the previous two equations and computing the log, we have

$$
\begin{equation}
\log\mathcal{L} = -\frac{1}{2} \sum_{i=1}^N \left[ \log(2\pi e_i^2) +  \frac{(F_i-F_\mathrm{true})^2}{e_i^2} \right].
\label{_auto4} \tag{7}
\end{equation}
$$

What we'd like to do is determine $F_\mathrm{true}$ such that the likelihood is maximized. For this simple problem, the maximization can be computed analytically (i.e. by setting $d\log\mathcal{L}/d F_\mathrm{true} = 0$). This results in the following observed estimate of $F_\mathrm{true}$

$$
\begin{equation}
F_\mathrm{est} = \sum_{i=1}^N w_i F_i; \quad w_i = 1/e_i^2.
\label{_auto5} \tag{8}
\end{equation}
$$

Notice that in the special case of all errors $e_i$ being equal, this reduces to

$$
\begin{equation}
F_\mathrm{est} = \frac{1}{N} \sum_{i=1} F_i.
\label{_auto6} \tag{9}
\end{equation}
$$

That is, in agreement with intuition, $F_\mathrm{est}$ is simply the mean of the observed data when errors are equal.

We can go further and ask what the error of our estimate is. In the frequentist approach, this can be accomplished by fitting a Gaussian approximation to the likelihood curve at maximum; in this simple case this can also be solved analytically (the sum of Gaussians is also a Gaussian). It can be shown that the standard deviation of this Gaussian approximation is

$$
\begin{equation}
\sigma_\mathrm{est} = \sum_{i=1}^N w_i.
\label{_auto7} \tag{10}
\end{equation}
$$

These results are fairly simple calculations; let's evaluate them for our toy dataset:

```
w=1./e**2
print("""
F_true = {0}
F_est = {1:.0f} +/- {2:.0f} (based on {3} measurements) """\
          .format(F_true, (w * F).sum() / w.sum(), w.sum() ** -0.5, N))
```

`F_true = 1000`
`F_est = 998 +/- 4 (based on 50 measurements)`

We find that for 50 measurements of the flux, our estimate has an error of about 0.4% and is consistent with the input value.

### Simple Photon Counts: Bayesian Approach

The Bayesian approach, as you might expect, begins and ends with probabilities. Our hypothesis is that the star has a constant flux $F_\mathrm{true}$. It recognizes that what we fundamentally want to compute is our knowledge of the parameters in question given the data and other information (such as our knowledge of uncertainties for the observed values), i.e. in this case, $p(F_\mathrm{true} | D,I)$.
Note that this formulation of the problem is fundamentally contrary to the frequentist philosophy, which says that probabilities have no meaning for model parameters like $F_\mathrm{true}$. Nevertheless, within the Bayesian philosophy this is perfectly acceptable.

To compute this result, Bayesians next apply Bayes' Theorem ([1](#eq:bayes)).
If we set the prior $p(F_\mathrm{true}|I) \propto 1$ (a flat prior), we find
$p(F_\mathrm{true}|D,I) \propto p(D | F_\mathrm{true},I) \equiv \mathcal{L}(D | F_\mathrm{true},I)$
and the Bayesian probability is maximized at precisely the same value as the frequentist result! So despite the philosophical differences, we see that (for this simple problem at least) the Bayesian and frequentist point estimates are equivalent.

### A note about priors

The prior allows inclusion of other information into the computation, which becomes very useful in cases where multiple measurement strategies are being combined to constrain a single model. The necessity to specify a prior, however, is one of the more controversial pieces of Bayesian analysis.
A frequentist will point out that the prior is problematic when no true prior information is available. Though it might seem straightforward to use a noninformative prior like the flat prior mentioned above, there are some [surprisingly subtleties](http://normaldeviate.wordpress.com/2013/07/13/lost-causes-in-statistics-ii-noninformative- priors/comment-page-1/) involved. It turns out that in many situations, a truly noninformative prior does not exist! Frequentists point out that the subjective choice of a prior which necessarily biases your result has no place in statistical data analysis.
A Bayesian would counter that frequentism doesn't solve this problem, but simply skirts the question. Frequentism can often be viewed as simply a special case of the Bayesian approach for some (implicit) choice of the prior: a Bayesian would say that it's better to make this implicit choice explicit, even if the choice might include some subjectivity.

### Simple Photon Counts: Bayesian approach in practice

Leaving these philosophical debates aside for the time being, let's address how Bayesian results are generally computed in practice. For a one parameter problem like the one considered here, it's as simple as computing the posterior probability $p(F_\mathrm{true} | D,I)$ as a function of $F_\mathrm{true}$: this is the distribution reflecting our knowledge of the parameter $F_\mathrm{true}$.
But as the dimension of the model grows, this direct approach becomes increasingly intractable. For this reason, Bayesian calculations often depend on sampling methods such as Markov Chain Monte Carlo (MCMC). For this practical example, let us apply an MCMC approach using Dan Foreman-Mackey's [emcee](http://dan.iel.fm/emcee/current/) package. Keep in mind here that the goal is to generate a set of points drawn from the posterior probability distribution, and to use those points to determine the answer we seek.
To perform this MCMC, we start by defining Python functions for the prior $p(F_\mathrm{true} | I)$, the likelihood $p(D | F_\mathrm{true},I)$, and the posterior $p(F_\mathrm{true} | D,I)$, noting that none of these need be properly normalized. Our model here is one-dimensional, but to handle multi-dimensional models we'll define the model in terms of an array of parameters $\boldsymbol{\alpha}$, which in this case is $\boldsymbol{\alpha} = [F_\mathrm{true}]$

```
def log_prior(alpha):
    return 0 # flat prior

def log_likelihood(alpha, F, e):
    return -0.5 * np.sum(np.log(2 * np.pi * e ** 2) \
                             + (F - alpha[0]) ** 2 / e ** 2)

def log_posterior(alpha, F, e):
    return log_prior(alpha) + log_likelihood(alpha, F, e)
```

Now we set up the problem, including generating some random starting guesses for the multiple chains of points.

```
ndim = 1      # number of parameters in the model
nwalkers = 50 # number of MCMC walkers
nburn = 1000  # "burn-in" period to let chains stabilize
nsteps = 2000 # number of MCMC steps to take
# we'll start at random locations between 0 and 2000
starting_guesses = 2000 * np.random.rand(nwalkers, ndim)
sampler = emcee.EnsembleSampler(nwalkers, ndim, log_posterior, args=[F,e])
sampler.run_mcmc(starting_guesses, nsteps)
# Shape of sampler.chain  = (nwalkers, nsteps, ndim)
# Flatten the sampler chain and discard burn-in points:
samples = sampler.chain[:, nburn:, :].reshape((-1, ndim))
```

If this all worked correctly, the array sample should contain a series of 50,000 points drawn from the posterior. Let's plot them and check. See results in Fig. [fig:flux-bayesian](#fig:flux-bayesian).

```
fig, ax = plt.subplots()
ax.hist(samples, bins=50, histtype="stepfilled", alpha=0.3, normed=True)
ax.set_xlabel(r'$F_\mathrm{est}$')
ax.set_ylabel(r'$p(F_\mathrm{est}|D,I)$')
```

Bayesian posterior pdf (represented by a histogram of MCMC samples) from flux measurements.

### Best estimates and confidence intervals

The posterior distribution from our Bayesian data analysis is the key quantity that encodes our inference about the values of the model parameters, given the data and the relevant background information. Often, however, we wish to summarize this result with just a few numbers: the best estimate and a measure of its reliability.

There are a few different options for this. The choice of the most appropriate one depends mainly on the shape of the posterior distribution:

*Symmetric posterior pdfs*: Since the probability (density) associated with any particular value of the parameter is a measure of how much we believe that it lies in the neighbourhood of that point, our best estimate is given by the maximum of the posterior pdf. If we denote the quantity of interest by $X$, with a posterior pdf $P =p(X|D,I)$, then the best estimate of its value $X_0$ is given by the condition $dP/dX|_{X=X_0}=0$. Strictly speaking, we should also check the sign of the second derivative to ensure that $X_0$ represents a maximum.

To obtain a measure of the reliability of this best estimate, we need to look at the width or spread of the posterior pdf about $X_0$. When considering the behaviour of any function in the neighbourhood of a particular point, it is often helpful to carry out a Taylor series expansion; this is simply a standard tool for (locally) approximating a complicated function by a low-order polynomial. The linear term is zero at the maximum and the quadratic term is often the dominating one determining the width of the posterior pdf. Ignoring all the higher-order terms we arrive at the Gaussian approximation

$$
\begin{equation}
p(X|D,I) \approx \frac{1}{\sigma\sqrt{2\pi}} \exp \left[ -\frac{(x-\mu)^2}{2\sigma^2} \right],
\label{_auto8} \tag{11}
\end{equation}
$$

where the mean $\mu = X_0$ and the variance $\sigma = \left( - \left. \frac{d^2L}{dX^2} \right|_{X_0} \right)^{-1/2}$, where $L$ is the logarithm of the posterior $P$. Our inference about the quantity of interest is conveyed very concisely, therefore, by the statement $X = X_0 \pm \sigma$, and

$$
$$
p(X_0-\sigma < X < X_0+\sigma | D,I) = \int_{X_0-\sigma}^{X_0+\sigma} p(X|D,I) dX \approx 0.67.
$$
$$

*Asymmetric posterior pdfs*: While the maximum of the posterior ($X_0$) can still be regarded as giving the best estimate, the true value is now more likely to be on one side of this rather than the other. Alternatively one can compute the mean value, $\langle X \rangle = \int X p(X|D,I) dX$, although this tends to overemphasise very long tails. The best option is probably a compromise that can be employed when having access to a large sample from the posterior (as provided by an MCMC), namely to give the median of this ensamble.

Furthermore, the concept of an error-bar does not seem appropriate in this case, as it implicitly entails the idea of symmetry. A good way of expressing the reliability with which a parameter can be inferred, for an asymmetric posterior pdf, is rather through a *confidence interval*. Since the area under the posterior pdf between $X_1$ and $X_2$ is proportional to how much we believe that $X$ lies in that range, the shortest interval that encloses 67% of the area represents a sensible measure of the uncertainty of the estimate. Obviously we can choose to provide some other degree-of-belief that we think is relevant for the case at hand. Assuming that the posterior pdf has been normalized, to have unit area, we need to find $X_1$ and $X_2$ such that:

$$
$$
p(X_1 < X < X_2 | D,I) = \int_{X_1}^{X_2} p(X|D,I) dX \approx 0.67,
$$
$$

where the difference $X_2 - X_1$ is as small as possible. The region $X_1 < X < X_2$ is then called the shortest 67% confidence interval.

*Multimodal posterior pdfs*: We can sometimes obtain posteriors which are multimodal; i.e. contains several disconnected regions with large probabilities. There is no difficulty when one of the maxima is very much larger than the others: we can simply ignore the subsidiary solutions, to a good approximation, and concentrate on the global maximum. The problem arises when there are several maxima of comparable magnitude. What do we now mean by a best estimate, and how should we quantify its reliability? The idea of a best estimate and an error-bar, or even a confidence interval, is merely an attempt to summarize the posterior with just two or three numbers; sometimes this just can’t be done, and so these concepts are not valid. For the bimodal case we might be able to characterize the posterior in terms of a few numbers: two best estimates and their associated error-bars, or disjoint confidence intervals. For a general multimodal pdf, the most honest thing we can do is just display the posterior itself.

### Simple Photon Counts: Best estimates and confidence intervals

To compute these numbers for our example, you would run:

```
sampper=np.percentile(samples, [2.5, 16.5, 50, 83.5, 97.5],axis=0).flatten()
print("""
F_true = {0}
Based on {1} measurements the posterior point estimates are:
...F_est = {2:.0f} +/- {3:.0f}
or using credible intervals:
...F_est = {4:.0f}          (posterior median)
...F_est in [{5:.0f}, {6:.0f}] (67% credible interval)
...F_est in [{7:.0f}, {8:.0f}] (95% credible interval) """\
          .format(F_true, N, np.mean(samples), np.std(samples), \
                      sampper[2], sampper[1], sampper[3], sampper[0], sampper[4]))
```

`F_true = 1000`
`Based on 50 measurements the posterior point estimates are:`
`...F_est = 998 +/- 4`
`or using credible intervals:`
`...F_est = 998          (posterior median)`
`...F_est in [993, 1002] (67% credible interval)`
`...F_est in [989, 1006] (95% credible interval)`

In this particular example, the posterior pdf is actually a Gaussian (since it is constructed as a product of Gaussians), and the mean and variance from the quadratic approximation will agree exactly with the frequentist approach.

From this final result you might come away with the impression that the Bayesian method is unnecessarily complicated, and in this case it certainly is. Using an MCMC sampler to characterize a one-dimensional normal distribution is a bit like using the Death Star to destroy a beach ball, but we did this here because it demonstrates an approach that can scale to complicated posteriors in many, many dimensions, and can provide nice results in more complicated situations where an analytic likelihood approach is not possible.

Furthermore, as data and models grow in complexity, the two approaches can diverge greatly.

## Bayesian parameter estimation (multiple parameters, covariance)
* multidimensional posterior pdf:s

* nuisance parameters (e.g. background subtraction?)

* corner plots, covariance, correlations

* best example?

## Bayesian model selection
* Bayesian evidence

* Occam's razor

* Best example? How many spectral lines are there?


```python
# This cell is mandatory in all Dymos documentation notebooks.
missing_packages = []
try:
    import openmdao.api as om
except ImportError:
    if 'google.colab' in str(get_ipython()):
        !python -m pip install openmdao[notebooks]
    else:
        missing_packages.append('openmdao')
try:
    import dymos as dm
except ImportError:
    if 'google.colab' in str(get_ipython()):
        !python -m pip install dymos
    else:
        missing_packages.append('dymos')
try:
    import pyoptsparse
except ImportError:
    if 'google.colab' in str(get_ipython()):
        !pip install -q condacolab
        import condacolab
        condacolab.install_miniconda()
        !conda install -c conda-forge pyoptsparse
    else:
        missing_packages.append('pyoptsparse')
if missing_packages:
    raise EnvironmentError('This notebook requires the following packages '
                           'please install them and restart this notebook\'s runtime: {",".join(missing_packages)}')
```

# The Length-Constrained Brachistochrone

```{admonition} Things you'll learn through this example
- How to connect the outputs from a trajectory to a downstream system.
```

This is a modified take on the brachistochrone problem.
In this instance, we assume that the quantity of wire available is limited.
Now, we seek to find the minimum time brachistochrone trajectory subject to a upper-limit on the arclength of the wire.

The most efficient way to approach this problem would be to treat the arc-length $S$ as an integrated state variable.
In this case, as is often the case in real-world MDO analyses, the implementation of our arc-length function is not integrated into our pseudospectral approach.
Rather than rewrite an analysis tool to accommodate the pseudospectral approach, the arc-length analysis simply takes the result of the trajectory in its entirety and computes the arc-length constraint via the trapezoidal rule:\

\begin{align}
    S &= \frac{1}{2} \left( \sum_{i=1}^{N-1} \sqrt{1 + \frac{1}{\tan{\theta_{i-1}}}} + \sqrt{1 + \frac{1}{\tan{\theta_{i}}}} \right) \left(x_{i-1} - x_i \right)
\end{align}

The OpenMDAO component used to compute the arclength is defined as follows:

```python
from __future__ import print_function, division, absolute_import

import numpy as np

from openmdao.api import ExplicitComponent

class ArcLengthComp(ExplicitComponent):

    def initialize(self):

        self.options.declare('num_nodes', types=(int,))

    def setup(self):
        nn = self.options['num_nodes']

        self.add_input('x', val=np.ones(nn), units='m', desc='x at points along the trajectory')
        self.add_input('theta', val=np.ones(nn), units='rad',
                       desc='wire angle with vertical along the trajectory')

        self.add_output('S', val=1.0, units='m', desc='arclength of wire')

        self.declare_partials(of='S', wrt='*', method='cs')

    def compute(self, inputs, outputs, discrete_inputs=None, discrete_outputs=None):

        x = inputs['x']
        theta = inputs['theta']

        dy_dx = -1.0 / np.tan(theta)
        dx = np.diff(x)
        f = np.sqrt(1 + dy_dx**2)

        # trapezoidal rule
        fxm1 = f[:-1]
        fx = f[1:]
        outputs['S'] = 0.5 * np.dot(fxm1 + fx, dx)
```

```{Note}
In this example, the number of nodes used to compute the arclength is needed when building the problem.
The transcription object is initialized and its attribute `grid_data.num_nodes` is used to provide the number of total nodes (the number of points in the timeseries) to the downstream arc length calculation.
```

```python
om.display_source("dymos.examples.brachistochrone.brachistochrone_ode")
```

```python
import openmdao.api as om
import dymos as dm
import matplotlib.pyplot as plt
from dymos.examples.brachistochrone.brachistochrone_ode import BrachistochroneODE

MAX_ARCLENGTH = 11.9
OPTIMIZER = 'SLSQP'

p = om.Problem(model=om.Group())
p.add_recorder(om.SqliteRecorder('length_constrained_brach_sol.db'))

if OPTIMIZER == 'SNOPT':
    p.driver = om.pyOptSparseDriver()
    p.driver.options['optimizer'] = OPTIMIZER
    p.driver.opt_settings['Major iterations limit'] = 1000
    p.driver.opt_settings['Major feasibility tolerance'] = 1.0E-6
    p.driver.opt_settings['Major optimality tolerance'] = 1.0E-5
    p.driver.opt_settings['iSumm'] = 6
    p.driver.opt_settings['Verify level'] = 3
else:
    p.driver = om.ScipyOptimizeDriver()

p.driver.declare_coloring()

# Create the transcription so we can get the number of nodes for the downstream analysis
tx = dm.Radau(num_segments=20, order=3, compressed=False)

traj = dm.Trajectory()
phase = dm.Phase(transcription=tx, ode_class=BrachistochroneODE)
traj.add_phase('phase0', phase)

p.model.add_subsystem('traj', traj)

phase.set_time_options(fix_initial=True, duration_bounds=(.5, 10))

phase.add_state('x', units='m', rate_source='xdot', fix_initial=True, fix_final=True)
phase.add_state('y', units='m', rate_source='ydot', fix_initial=True, fix_final=True)
phase.add_state('v', units='m/s', rate_source='vdot', fix_initial=True, fix_final=False)

phase.add_control('theta', units='deg', lower=0.01, upper=179.9,
                  continuity=True, rate_continuity=True)

phase.add_parameter('g', units='m/s**2', opt=False, val=9.80665)

# Minimize time at the end of the phase
phase.add_objective('time', loc='final', scaler=1)

# p.model.options['assembled_jac_type'] = top_level_jacobian.lower()
# p.model.linear_solver = DirectSolver(assemble_jac=True)

# Add the arc length component
p.model.add_subsystem('arc_length_comp',
                      subsys=ArcLengthComp(num_nodes=tx.grid_data.num_nodes))

p.model.connect('traj.phase0.timeseries.controls:theta', 'arc_length_comp.theta')
p.model.connect('traj.phase0.timeseries.states:x', 'arc_length_comp.x')

p.model.add_constraint('arc_length_comp.S', upper=MAX_ARCLENGTH, ref=1)

p.setup(check=True)

p.set_val('traj.phase0.t_initial', 0.0)
p.set_val('traj.phase0.t_duration', 2.0)

p.set_val('traj.phase0.states:x', phase.interp('x', [0, 10]))
p.set_val('traj.phase0.states:y', phase.interp('y', [10, 5]))
p.set_val('traj.phase0.states:v', phase.interp('v', [0, 9.9]))
p.set_val('traj.phase0.controls:theta', phase.interp('theta', [5, 100]))
p.set_val('traj.phase0.parameters:g', 9.80665)

p.run_driver()

p.record(case_name='final')

# Generate the explicitly simulated trajectory
exp_out = traj.simulate()

# Extract the timeseries from the implicit solution and the explicit simulation
x = p.get_val('traj.phase0.timeseries.states:x')
y = p.get_val('traj.phase0.timeseries.states:y')
t = p.get_val('traj.phase0.timeseries.time')
theta = p.get_val('traj.phase0.timeseries.controls:theta')

x_exp = exp_out.get_val('traj.phase0.timeseries.states:x')
y_exp = exp_out.get_val('traj.phase0.timeseries.states:y')
t_exp = exp_out.get_val('traj.phase0.timeseries.time')
theta_exp = exp_out.get_val('traj.phase0.timeseries.controls:theta')

fig, axes = plt.subplots(nrows=2, ncols=1)

axes[0].plot(x, y, 'o')
axes[0].plot(x_exp, y_exp, '-')
axes[0].set_xlabel('x (m)')
axes[0].set_ylabel('y (m)')

axes[1].plot(t, theta, 'o')
axes[1].plot(t_exp, theta_exp, '-')
axes[1].set_xlabel('time (s)')
axes[1].set_ylabel(r'$\theta$ (deg)')

plt.show()
```