# Let's start!
OPEN the jupyter notebook **Tutorial1-Part1** downloaded from the **indico timetable: https://indico.cern.ch/event/1088622/timetable/#20220111** to work locally or from the following link: **https://github.com/fusterma/JUAS2022** to work online.

# Tutorials summary

The goal of these workshops is to do numerical exercices using MAD-X to visualize transverse dynamics concepts from a different point of view.

**Friday 14th of January**
- **Tutorial 1 - Part 1**: Introduction to the tools, small numerical exercises (all together).
- **Tutorial 1 - Part 2**: My first circular accelerator: FODO cell – optics and first matching (groups of 3/4 students).
- **Tutorial 1 - Part 3**: Adding dipoles to the FODO cell – MAD-X matching block (groups of 3/4 students).

$\color{red}{\text{WEEKEND: homework exercice}}$

**Monday 17th of January**
- **Tutorial 2 - Part 1**: Natural chromaticity – MAD-X tracking module (groups of 3/4 students).
- **Tutorial 2 - Part 2**: Chromaticity correction – impact of non-linearities (groups of 3/4 students).
- **Tutorial 2 - Part 3**: Design of a transfer line  - optics and matching (groups of 3/4 students).

$\color{red}{\text{Homework + Tutorial 1 and Tutorial 2 jupyter-notebooks (to be delivered as late on Wednesday 19th to nuria.fuster@ific.uv.es)}}$. This will be considered as a BONUS to pass the accelerator design workshop oral exam.

$\color{red}{\text{VERY IMPORTANT}}$: Save your jupyter-notebooks and download them to your computer after finishing the tutorials!! Otherwise your work will be lost!

- The tutorials solutions will be uploaded on the indico timetable on Wednesday 19th.

$\color{blue}{\text{Notes}}$:

- For most of the tutorials we will split in groups of 3/4 students. These groups will be kept also for the accelerator design workshops on the third and fourth weeks of the JUAS course.

- The timing of the tutorials (except Tutorial 1 - Part 1) will be:
    - 5 minutes for the introduction to the problem.
    - 25 minutes to work within your team and with your tutor on the problem.
    - 15 minutes for going through the solutions and discussion.

- Tutors (Axel, Guido, Tessa, Davide and Nuria) will go around to help you and answer questions!

- You can work as a team in the groups or by yourself but when you need help please ask your team mates or the tutor and use the screen share option.

- The problems are long with many questions, we don't expect you to solve all of them. Some BONUS questions are there for discussion or homework.

- We encourage you to use the Slack MAD-X channel during the workshops for the discussion part, during the weekend and all JUAS to post questions.

# Tutorial 1: Part 1

Objectives:

- [Get familiar with the jupyter-notebooks.](#introjupyter)

- [Get familiar with the basic python commands that we will use during the tutorials.](#intropython)

- [How do we compute the optics of a lattice?](#firstexercice)

- [Get familiar with python commands to send the information to the MAD-X code and review the main MAD-X blocks for optics calculations.](#intromadx)

# Jupyter notebook

- **OPEN** a jupyter notebook go to **FILE -> OPEN**.

- **EDIT/INSERT/DELATE** a cell.

- **RUN** (press bottom on the top command line or press CAPS+ENTER).

- If working online **SAVE and DOWNLOAD**: after we finish one tutorial you need to SAVE and DOWNLOAD the jupyter-notebook into your PC. Otherwise your progres will be lost!

- **SAVE TO BROWSER STORAGE** using the "cloud" icon (for those working on BINDER).

# Basic python commands

The python universe has a huge number of libraries that extend the capabilities of python.
Nearly all of these are open source. For this workshop we will use the following:

```python
############################
# Import special libraries #
############################
#For plotting
from matplotlib import pyplot as plt
# For numerical calulations (np.max(), np.min(), np.mean()...)
import numpy as np
# For symbolic computation (solving algebra problems)
import sympy as sp
# For structuring the data, visualization of tables and data manipultion
import pandas as pd
# Library that allows us to use the MAD-X models
from cpymad.madx import Madx
# Plot display
%matplotlib notebook
```

If you want to learn more about **python**:

https://www.youtube.com/watch?v=kqtD5dpn9C8

https://www.kaggle.com/learn/python

More about the **cpymad** library: http://hibtc.github.io/cpymad/getting-started

# Scalars, arrays and matrices in Python

```python
# Scalar
a=20
b=30
```

```python
# Arrays and matrices
sp.Matrix([1,2,3,4]) # 1D array
```

$\displaystyle \left[\begin{matrix}1\\2\\3\\4\end{matrix}\right]$

```python
sp.Matrix([[1,2],[3,4]]) # 2x2 matrix
```

$\displaystyle \left[\begin{matrix}1 & 2\\3 & 4\end{matrix}\right]$

```python
A=sp.Matrix([[1,2],[3,4]])
B=sp.Matrix([[1,2],[3,4]])
A+B
```

$\displaystyle \left[\begin{matrix}2 & 4\\6 & 8\end{matrix}\right]$

```python
A*B
```

$\displaystyle \left[\begin{matrix}7 & 10\\15 & 22\end{matrix}\right]$

# Plots in Python

```python
# Plot
%matplotlib notebook
x=[0,1,2,3,4,5,6,7,8,9,10]
y=[0,1,2,3,4,5,6,7,8,9,10]
plt.plot(x,y,'.-b',label='test')
plt.legend()
plt.grid()
plt.xlabel('s [m]')
plt.ylabel('x[m]')
```

    <IPython.core.display.Javascript object>

    Text(0, 0.5, 'x[m]')

# How do we compute the optics of a lattice?

- Here we want to motivate the use of optics codes such as MAD-X and at the same time illustrate the basic numerical approach behind some of the methods.

- The TWISS method is based on matrix multiplications where first and second order transport matrices are used to get the optics of the machine.

# FODO cell
- Compute the linear optics functions of a FODO cell, which is the simplier combination of quadrupoles required to focuse the beam in both, vertical and horizontal planes.

# Thin lens approximation (f >> $l_q$)

- To do some first estimations analytically one uses the thin lens approximation.

```python
# Symbolic computation

# Symbols defintion
K = sp.Symbol("K", positive = True)
Lq = sp.Symbol("Lq", positive = True)
Ld = sp.Symbol("Ld", positive = True)
```

```python
#Matrices definition: A=sp.Matrix([[1,2],[3,4]])

#Mfoc=??

#Mdefoc=??

#Mdrift=??
```

```python
#################################################
# Transport matrix of a focusing quadrupole     #
#################################################
Mfoc
```

```python
################################################
# Transport matrix of defocusing quadrupole    #
################################################
Mdefoc
```

```python
##############################################
# Transport matrix of a drift                #
##############################################
Mdrift
```

```python
#Matrix multiplication and you can use the sp.simplify(M) command to simplify the elements of the matrix.

```

```python
# Matrix elements computation
# f=200 m, lq=1 m, ld= 30 m
M_thin = M.subs(K, 1/(200*1)).subs(Lq, 1).subs(Ld,30) # units K in m-2, Lq and Ld in m
M_thin
```

```python
#And for 3 FODO cells?
```

# What can we do with the  transfer matrix?
This matrix describes the optical properties of the lattice and defines the beam parameters.

 - We can propagate the phase space coordinates of a particle with a given set of initial coordinates.

```python
# Try with a paralel particle going through the center of the first quadrupole or with a certain amplitude

```

```python
# Try with paralel particle going through the center of the first quadrupole or with a certain amplitude
x=sp.Matrix([[1],[0]])
x
```

```python
x2=M*x
x2
```

- We can compute the periodic solution TWISS functions.

```python
# Transfer matrix
R11, R12, R21, R22 = sp.symbols('R11,R12,R21,R22')

Mt=sp.Matrix([[R11,R12],[R21,R22]])

Mt
```

$\displaystyle \left[\begin{matrix}R_{11} & R_{12}\\R_{21} & R_{22}\end{matrix}\right]$

```python
# In case of periodic conditions in the accelerator there is another way to describe the particles trjectories.
# Periodic solution one-turn-transfer matrix in terms of twiss functions:

a, b, g, m = sp.symbols(r'\alpha,\beta, \gamma,\mu')

M=sp.Matrix([[sp.cos(m)+a*sp.sin(m),b*sp.sin(m)],[-g*sp.sin(m),sp.cos(m)-a*sp.sin(m)]])

M
```

$\displaystyle \left[\begin{matrix}\alpha \sin{\left(\mu \right)} + \cos{\left(\mu \right)} & \beta \sin{\left(\mu \right)}\\- \gamma \sin{\left(\mu \right)} & - \alpha \sin{\left(\mu \right)} + \cos{\left(\mu \right)}\end{matrix}\right]$

```python
#For the phase advance we use the comparison of the trace of the matrix
sp.Eq(sp.cos(m),(R11+R22)/2)
```

$\displaystyle \cos{\left(\mu \right)} = \frac{R_{11}}{2} + \frac{R_{22}}{2}$

```python
#For the beta function we use the R12 matrix element
sp.Eq(b,R12/sp.sin(m))
```

$\displaystyle \beta = \frac{R_{12}}{\sin{\left(\mu \right)}}$

```python
#For the alfa function we use the trace of the matrix also
sp.Eq(a,(R11-R22)/(2*sp.sin(m)))
```

$\displaystyle \alpha = \frac{R_{11} - R_{22}}{2 \sin{\left(\mu \right)}}$

```python
#For the gamma we use the R21 element
sp.Eq(g,-(R21/sp.sin(m)))
```

$\displaystyle \gamma = - \frac{R_{21}}{\sin{\left(\mu \right)}}$

Once you have computed the periodic TWISS functions you can propagate them to any point in the machine using the transfer matrix of the TWISS functions from Transverse dynamics course.

```python
sp.Matrix([[R11**2, -2*R12*R11, R12**2],[-R11*R21,R12*R21+R22*R11, -R12*R22],[R21**2,-2*R22*R21, R22**2]])
```

$\displaystyle \left[\begin{matrix}R_{11}^{2} & - 2 R_{11} R_{12} & R_{12}^{2}\\- R_{11} R_{21} & R_{11} R_{22} + R_{12} R_{21} & - R_{12} R_{22}\\R_{21}^{2} & - 2 R_{21} R_{22} & R_{22}^{2}\end{matrix}\right]$

**Using the periodic one-turn-matrix one can define some interesting relations between the TWISS parameters and the magnetic properties of the lattice.**

# Figure 1: Relation between $\Delta \mu$, K, $L_{cell}$, $L_q$

```python
# NOTE: here we consider that both quadrupoles hve the same strength
# Relation between the phase advance of the cell and K, Lcell, Lq (from periodic solution an stbility condition)
a, b, g, m, d, Lq, Lc, K, pi = sp.symbols(r'\alpha,\beta, \gamma,\mu, \Delta, L_{q}, L_{cell} K \pi')
sp.Eq(d*m/pi,2*sp.asin(K*Lq*Lc/4))
```

```python
#Parametric plots
%matplotlib notebook
plt.rcParams['savefig.dpi'] = 90
plt.rcParams['figure.dpi'] = 90

x=np.arange(0,6,0.01)
y=2*np.arcsin(x/4)/np.pi
fig, ax1 = plt.subplots()
ax1.plot(x,y,'-')
ax1.set_ylabel("$\Delta \mu / \pi [rad]$", fontsize=16)
ax1.set_xlabel("$K*L_{quad}*L_{cell}$ [-]", fontsize=16)
ax1.grid()
ax1.tick_params(axis='both', labelsize=16)
plt.tight_layout()
```

    <ipython-input-2-2b874282a2b0>:7: RuntimeWarning: invalid value encountered in arcsin
      y=2*np.arcsin(x/4)/np.pi

    <IPython.core.display.Javascript object>

# Exercice:
- What is the quadrupole strenght to match a FODO cell phase advance of 45$^\circ$ if the $L_{quad}$=5 m and $L_{cell}$=100 m?

- And for a FODO cell phase advance of 90$^\circ$?

- What is the maximum phase advance in a FODO cell?

```python

```

```python

```

```python

```

# Figure 2: Relation between $\beta_{max}$ and $\beta_{min}$ with K, $L_{cell}$, $L_q$

```python
# Relation between the beta of the cell and K, Lcell, Lq
a, bmin, bmax, g, m, d, Lq, Lc, K, pi = sp.symbols(r'\alpha,\beta_{min}, \beta_{max}, \gamma,\mu, \Delta, L_{q}, L_{cell} K \pi')
sp.Eq(bmin/Lc,(1-(K*Lq*Lc/4))/(sp.sin(2*sp.asin(K*Lq*Lc/4))))
```

$\displaystyle \frac{\beta_{min}}{L_{cell}} = \frac{- \frac{K L_{cell} L_{q}}{4} + 1}{\sin{\left(2 \operatorname{asin}{\left(\frac{K L_{cell} L_{q}}{4} \right)} \right)}}$

```python
sp.Eq(bmax/Lc,(1+(K*Lq*Lc/4))/(sp.sin(2*sp.asin(K*Lq*Lc/4))))
```

$\displaystyle \frac{\beta_{max}}{L_{cell}} = \frac{\frac{K L_{cell} L_{q}}{4} + 1}{\sin{\left(2 \operatorname{asin}{\left(\frac{K L_{cell} L_{q}}{4} \right)} \right)}}$

```python
%matplotlib notebook
plt.rcParams['savefig.dpi'] = 90
plt.rcParams['figure.dpi'] = 90

x=np.arange(0.4,3.90,0.01)
betamax=(1+(x/4))/(np.sin(2*np.arcsin(x/4)))
betamin=(1-(x/4))/(np.sin(2*np.arcsin(x/4)))
fig, ax1 = plt.subplots()
ax1.plot(x,betamax,'-',label=r"$\beta_{max}/L_{cell}$")
ax1.plot(x,betamin,'-',label=r"$\beta_{min}/L_{cell}$")
ax1.set_ylabel("[-]", fontsize=16)
ax1.set_xlabel("$K*L_{quad}*L_{cell}$ [-]", fontsize=16)
plt.grid()
plt.legend()
plt.tick_params(axis='both', labelsize=16)
plt.tight_layout()
```

    <IPython.core.display.Javascript object>

# Exercice:
- What is K and $L_{cell}$ to match a FODO cell with a phase advance of 90$^\circ$ and a $\beta_{max}$ of 200 m? The  $L_{quad}$=2 m.

HINT: You may need to combine the data from both plots.

```python

```

```python

```

```python

```

At the end, the exact solution of the particle motion has to be calculated in full detail but using some approximations we can make the first steps easier and estimate the order of magnitud of some magnetic properties of our lattice.

# Thick lens computation

```python
K = sp.Symbol("K")
Lq = sp.Symbol("Lq")
Ld= sp.Symbol("Ld")

Mfoc=sp.Matrix([[sp.cos(sp.sqrt(K)*Lq),1/(sp.sqrt(K))*sp.sin(sp.sqrt(K)*Lq)],[-(sp.sqrt(K))*sp.sin(sp.sqrt(K)*Lq),sp.cos(sp.sqrt(K)*Lq)]])

Mdefoc=sp.Matrix([[sp.cosh(sp.sqrt(K)*Lq),1/(sp.sqrt(K))*sp.sinh(sp.sqrt(K)*Lq)],[(sp.sqrt(K))*sp.sinh(sp.sqrt(K)*Lq),sp.cosh(sp.sqrt(K)*Lq)]])

Mdrift=sp.Matrix([[1,Ld],[0,1]])
```

```python
#################################################
# Transport matrix of a focusing quadrupople    #
#################################################
Mfoc
```

$\displaystyle \left[\begin{matrix}\cos{\left(\sqrt{K} Lq \right)} & \frac{\sin{\left(\sqrt{K} Lq \right)}}{\sqrt{K}}\\- \sqrt{K} \sin{\left(\sqrt{K} Lq \right)} & \cos{\left(\sqrt{K} Lq \right)}\end{matrix}\right]$

```python
################################################
# Transport matrix of a defocusing quadrupople #
################################################
Mdefoc
```

$\displaystyle \left[\begin{matrix}\cosh{\left(\sqrt{K} Lq \right)} & \frac{\sinh{\left(\sqrt{K} Lq \right)}}{\sqrt{K}}\\\sqrt{K} \sinh{\left(\sqrt{K} Lq \right)} & \cosh{\left(\sqrt{K} Lq \right)}\end{matrix}\right]$

```python
##############################################
# Transport matrix of a dipole               #
##############################################
Mdrift
```

$\displaystyle \left[\begin{matrix}1 & Ld\\0 & 1\end{matrix}\right]$

```python
M=Mdrift*Mdefoc*Mdrift*Mfoc
M=sp.simplify(M)
M
```

$\displaystyle \left[\begin{matrix}\left(\sqrt{K} Ld \sinh{\left(\sqrt{K} Lq \right)} + \cosh{\left(\sqrt{K} Lq \right)}\right) \cos{\left(\sqrt{K} Lq \right)} - \left(2 \sqrt{K} Ld \cosh{\left(\sqrt{K} Lq \right)} + K Ld^{2} \sinh{\left(\sqrt{K} Lq \right)} + \sinh{\left(\sqrt{K} Lq \right)}\right) \sin{\left(\sqrt{K} Lq \right)} & \frac{\left(\sqrt{K} Ld \sinh{\left(\sqrt{K} Lq \right)} + \cosh{\left(\sqrt{K} Lq \right)}\right) \sin{\left(\sqrt{K} Lq \right)} + \left(2 \sqrt{K} Ld \cosh{\left(\sqrt{K} Lq \right)} + K Ld^{2} \sinh{\left(\sqrt{K} Lq \right)} + \sinh{\left(\sqrt{K} Lq \right)}\right) \cos{\left(\sqrt{K} Lq \right)}}{\sqrt{K}}\\- \sqrt{K} \left(\left(\sqrt{K} Ld \sinh{\left(\sqrt{K} Lq \right)} + \cosh{\left(\sqrt{K} Lq \right)}\right) \sin{\left(\sqrt{K} Lq \right)} - \cos{\left(\sqrt{K} Lq \right)} \sinh{\left(\sqrt{K} Lq \right)}\right) & \left(\sqrt{K} Ld \sinh{\left(\sqrt{K} Lq \right)} + \cosh{\left(\sqrt{K} Lq \right)}\right) \cos{\left(\sqrt{K} Lq \right)} + \sin{\left(\sqrt{K} Lq \right)} \sinh{\left(\sqrt{K} Lq \right)}\end{matrix}\right]$

```python
M_thick = M.subs(K, 1/(200*1)).subs(Lq, 1).subs(Ld,30) # units K in m-2, Lq and Ld in m
M_thick
```

$\displaystyle \left[\begin{matrix}0.821745966061752 & 66.6422445425746\\-0.000766666456349212 & 1.15474570697446\end{matrix}\right]$

```python
M_thin
```

$\displaystyle \left[\begin{matrix}0.8275 & 64.5\\-0.00075 & 1.15\end{matrix}\right]$

```python
#And for 3 FODO cells?
```

In real world applications, lattices (including FODO) are not designed by hand but
dedicated software is used to do the design and simulation as for example MAD-X.

The TWISS in MADX it is based in matrix multiplications similar to what has been shown here.

# Wha is next?

- Now we are going to do optics calculations using MAD-X TWISS command (handle thousands of elements).

- We will use the MATCHING MAD-X tool to compute the required magnetic properties for a desired TWISS functions.

- We will use MAD-X to visulize the impact of some properties of the lattice on the TWISS and on the single particle DYNAMICS.

# An introduction to MAD-X using the python interface

In this first part we are going to get familiar with MAD-X syntax.

For more information please refer to the [MAD-X online manual](http://cern.ch/madx/releases/last-rel/madxuguide.pdf).

**Basic steps:**

   - Load the cpymad library.
   - Instantiate the MADX class (we create an object of the class).
   - Access the methods in the class.
       - From the methods available we will be mainly using the method "input" to send to MAD-X the commands.

```python
#Load the cpymad library
from cpymad.madx import Madx
```

```python
#Launching MAD-X
myMad = Madx(stdout=True)
```

```python
#String that will be interpreted by MAD-X
myString='''
stop;
'''
```

```python
#Using the "input" method to send the commandas to the MAD-X class
myMad.input(myString);
```

With the 'stop;' instruction we exit from MAD-X, so, as done in the following cell we need to re-instantiate our MAD-X object with the
**myMad = Madx()** instruction.

---
It is a good practice to make header, please use '!' to comment the single line.

```python
# Define and print a value
myMad = Madx(stdout=True)
myString='''

!***************************************
! It is a good practice to make a header
!***************************************

a= 20;
value a;

'''
myMad.input(myString);
```

---
Use the **help** keyword (very rudimental help)

```python
# To get information about the MAD-X methods (twiss, beam,match... ) use the commnd "help"
myString='''
help, twiss;
'''
myMad.input(myString);
```

```python
myString='''
help, drift;
'''
myMad.input(myString);
```

# Let's define the main ingredients to get some results from MAD-X!

-Definition of machine parameters

-Magnets definition

-Sequence definition

-Beam definition

-Activate the sequence

-Actions

```python
myString='''
! *********************************************************************
! Definition of some parameters
! *********************************************************************

l_cell=60;
quadrupoleLenght=1;
myK:=0.005;// m^-2

! *********************************************************************
! Definition of magnets
! *********************************************************************
QF: quadrupole, L=quadrupoleLenght, K1:=myK;
QD: quadrupole, L=quadrupoleLenght, K1:=-myK;

! *********************************************************************
! Definition of sequence
! *********************************************************************
myCell:sequence, refer=entry, L=L_CELL;
quadrupole1: QF, at=0;
marker1: marker, at=15;
quadrupole2: QD, at=30;
endsequence;

! *********************************************************************
! Definition of beam
! *********************************************************************
beam, particle=proton, energy=1;

! *********************************************************************
! Use of the sequence
! *********************************************************************
use, sequence=myCell;

! *********************************************************************
! TWISS
! *********************************************************************

select, flag=TWISS, column=keyword, name, s, betx, bety,alfx, alfy, x, y, dx, dy;
twiss, file=Test.madx;
!plot, haxis=s, vaxis=betx,bety,dx,colour=100,file=Test;

'''
myMad.input(myString);
```

The OUTPUT generated by MADX can be found by accessing the jupyter-notebook files-view.
- For accessing: CLIC on the jupyter-logo on the top of the page using the right buttom of your mouse and use the option "Open Link in a New Tab".

**Output:**

- SUMM table

- TWISS table

- TWISS .txt file

- TWISS .ps plot

# Accessing the data

- Open the files generated by MAD-X.

- Use python and output the required data on the jupyter-notebook.

    - Using MAD-X commands and the **input()** method.

    - Using **cpymad** methods:
        - **myMad.table.twiss.dframe()**
        - **myMad.table.summ.dframe()**

```python
#######################
#Using MAD-X commands #
#######################
myString='''
value, table(SUMM,Q1);
value, table(SUMM,betxmax);
'''
myMad.input(myString);
```

And for the vertical plane?

```python
#######################
#Using MAD-X commands #
#######################
myString='''
value, table(SUMM,Q2);
value, table(SUMM,betymax);
'''
myMad.input(myString);
```

```python
myString='''
value, table(TWISS,MYCELL$END,betx);
value, table(TWISS,MYCELL$END,bety);
'''
myMad.input(myString);
```

# Using python pandas library

Pandas dataframe are very convenient, have a look in https://pandas.pydata.org/Pandas_Cheat_Sheet.pdf.

```python
# Using another method from cpymad "table.twiss.dframe()"
myDF=myMad.table.twiss.dframe()
```

```python
myDF
```

      name
      keyword
      s
      betx
      alfx
      mux
      bety
      alfy
      muy
      x
      ...
      sig54
      sig55
      sig56
      sig61
      sig62
      sig63
      sig64
      sig65
      sig66
      n1

      #s
      mycell$start:1
      marker
      0.0
      463.623288
      -1.156109
      0.000000
      369.779162
      0.929316
      0.000000
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0

      quadrupole1
      quadrupole1:1
      quadrupole
      5.0
      463.623288
      1.156109
      0.001709
      369.779162
      -0.929316
      0.002161
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0

      drift_0[0]
      drift_0:0
      drift
      25.0
      419.394867
      1.055312
      0.008930
      408.967742
      -1.030113
      0.010350
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0

      marker1
      marker1:1
      marker
      25.0
      419.394867
      1.055312
      0.008930
      408.967742
      -1.030113
      0.010350
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0

      drift_1[0]
      drift_1:0
      drift
      50.0
      369.779162
      0.929316
      0.019041
      463.623288
      -1.156109
      0.019493
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0

      quadrupole2
      quadrupole2:1
      quadrupole
      55.0
      369.779162
      -0.929316
      0.021202
      463.623288
      1.156109
      0.021202
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0

      drift_2[0]
      drift_2:0
      drift
      100.0
      463.623288
      -1.156109
      0.038533
      369.779162
      0.929316
      0.038533
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0

      #e
      mycell$end:1
      marker
      100.0
      463.623288
      -1.156109
      0.038533
      369.779162
      0.929316
      0.038533
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0

8 rows × 256 columns

```python
myDF[['name','s','betx','bety','alfx','alfy']]
```

      name
      s
      betx
      bety
      alfx
      alfy

      #s
      mycell$start:1
      0.0
      463.623288
      369.779162
      -1.156109
      0.929316

      quadrupole1
      quadrupole1:1
      5.0
      463.623288
      369.779162
      1.156109
      -0.929316

      drift_0[0]
      drift_0:0
      25.0
      419.394867
      408.967742
      1.055312
      -1.030113

      marker1
      marker1:1
      25.0
      419.394867
      408.967742
      1.055312
      -1.030113

      drift_1[0]
      drift_1:0
      50.0
      369.779162
      463.623288
      0.929316
      -1.156109

      quadrupole2
      quadrupole2:1
      55.0
      369.779162
      463.623288
      -0.929316
      1.156109

      drift_2[0]
      drift_2:0
      100.0
      463.623288
      369.779162
      -1.156109
      0.929316

      #e
      mycell$end:1
      100.0
      463.623288
      369.779162
      -1.156109
      0.929316

```python
myDF["s"]
```

    #s               0.0
    quadrupole1      5.0
    drift_0[0]      25.0
    marker1         25.0
    drift_1[0]      50.0
    quadrupole2     55.0
    drift_2[0]     100.0
    #e             100.0
    Name: s, dtype: float64

```python
myDF["betx"]
```

    #s             463.623288
    quadrupole1    463.623288
    drift_0[0]     419.394867
    marker1        419.394867
    drift_1[0]     369.779162
    quadrupole2    369.779162
    drift_2[0]     463.623288
    #e             463.623288
    Name: betx, dtype: float64

```python
myDF2=myMad.table.summ.dframe()
```

```python
myDF2
```

      length
      orbit5
      alfa
      gammatr
      q1
      dq1
      betxmax
      dxmax
      dxrms
      xcomax
      ...
      ycorms
      deltap
      synch_1
      synch_2
      synch_3
      synch_4
      synch_5
      synch_6
      synch_8
      nflips

      #e
      100.0
      -0.0
      0.0
      0.0
      0.038533
      -0.043847
      463.623288
      0.0
      0.0
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0

1 rows × 27 columns

```python
myDF2["q1"]
```

    #e    0.038533
    Name: q1, dtype: float64

# Basic plot

```python
#Plot
plt.plot(myDF['s'],myDF['betx'],'.-b',label='$\\beta_x$')
plt.plot(myDF['s'],myDF['bety'],'.-r',label='$\\beta_y$')
#Labels of the plot
plt.xlabel('s [m]')
plt.ylabel('[m]')
#Legend and grid
plt.legend(loc='best')
plt.grid()
```

# For reference...

---
This is an example to get familiar with the use of the physical constants and the formatting of the output. Have a look on the difference.

```python
myString='''
a=pi;
value a;
set, format="22.20e";
value a;
'''
myMad.input(myString);
```

#
This is an example to get familiar with if and deferred expression. Please note the after the block delimited with {...} the ; can be omitted. Pay attention to circular call!

```python
myString='''
if (1==1){
option, echo=false, info=true;
a=pi;
b:=a;
c=a;
value a;
value b;
value c;
a=CLIGHT*cos(a);
value a;
value b;
value c;}
! BEWARE of circular call!
!a:=a+1;
! When evaluating you will get a fatal error
! value a;
option, echo=true, info=true;
'''
myMad.input(myString);
```

# ---
This is an example to get familiar with **while** and **macros** loops.

```python
myString='''
a(myvariable1,myvariable2): macro = {
value, myvariable1;
value, myvariable1*myvariable2;
}

N=1;
while (N<10){
exec, a(N,N);
N=N+1;
}
'''
myMad.input(myString);
```

---
### List of functions
In MAD-X the following functions are available

- SQRT(x) square root,
- LOG(x) natural logarithm,
- LOG10(x) logarithm base 10,
- EXP(x) exponential,
- SIN(x) trigonometric sine,
- COS(x) trigonometric cosine,
- TAN(x) trigonometric tangent,
- ASIN(x) arc sine,
- ACOS(x) arc cosine,
- ATAN(x) arc tangent,
- SINH(x) hyperbolic sine,
- COSH(x) hyperbolic cosine,
- TANH(x) hyperbolic tangent,
- SINC(x) cardinal sine function,
- ABS(x) absolute value,
- ERF(x) Gauss error,
- ERFC(x) complementary error,
- FLOOR(x) floor, largest previous integer,
- CEIL(x) ceiling, smallest next integer,
- ROUND(x) round, closest integer,
- FRAC(x) fractional part of number,
- RANF() random number, uniformly distributed in [0,1],
- GAUSS() random number, gaussian distribution with unit standard deviation,
- TGAUSS(x) random number, gaussian distribution with unit standard deviation, truncated at x standard deviations;

---
### List of physical constant

| MAD-X name  | symbol  |  value |unit|
|:-:|:-:|:-:|:-:|
|PI| π |4 * atan(1)| 1|
|TWOPI|2π| 2 * PI| 1|
|DEGRAD| 180/π |180 / PI| deg/rad|
|RADDEG| π/180 |PI / 180 |rad/deg|
|E| e |exp(1) |1|
|EMASS| me |0.510998928e−3| GeV|
|PMASS| mp |0.938272046| GeV|
|NMASS| u |0.931494061| GeV|
|MUMASS| mµ| 0.1056583715 |GeV|
|CLIGHT| c| 299792458| m/s|
|QELECT| e| 1.602176565e−19| A.s|
|HBAR| ¯h| 6.58211928e−25| MeV.s|
|ERAD| re| 2.8179403267e−15| m|
|PRAD| re(me/mp)| ERAD*EMASS/PMASS| m|


[Table of Contents](http://nbviewer.ipython.org/github/rlabbe/Kalman-and-Bayesian-Filters-in-Python/blob/master/table_of_contents.ipynb)
\appendix
# Installation, Python, NumPy, and FilterPy

```python
#format the book
%matplotlib inline
from __future__ import division, print_function
from book_format import load_style
load_style()
```

<style>
@import url('http://fonts.googleapis.com/css?family=Source+Code+Pro');
@import url('http://fonts.googleapis.com/css?family=Lora');

//@import url('http://fonts.googleapis.com/css?family=Open+Sans');
//@import url('http://fonts.googleapis.com/css?family=Vollkorn');
//@import url('http://fonts.googleapis.com/css?family=Karla');
//@import url('http://fonts.googleapis.com/css?family=Poppins');
//@import url('http://fonts.googleapis.com/css?family=Arimo');
//@import url('http://fonts.googleapis.com/css?family=Roboto');
//@import url('http://fonts.googleapis.com/css?family=Lato');
//@import url('http://fonts.googleapis.com/css?family=Domine');
//@import url('http://fonts.googleapis.com/css?family=Chivo');
//@import url('http://fonts.googleapis.com/css?family=Cardo');
//@import url('http://fonts.googleapis.com/css?family=Arvo');
//@import url('http://fonts.googleapis.com/css?family=Crimson+Text');
//@import url('http://fonts.googleapis.com/css?family=Ubuntu');
//@import url('http://fonts.googleapis.com/css?family=Fontin');
//@import url('http://fonts.googleapis.com/css?family=Raleway');
//@import url('http://fonts.googleapis.com/css?family=Merriweather');

.CodeMirror pre {
    font-family: 'Source Code Pro', Consolas, monocco, monospace;
}
    div.cell{
        //width: 950px;
        margin-left: 0% !important;
        margin-right: auto;
    }
    div.text_cell_render{
        font-family: 'Lora';
        //font-family: 'Open Sans';
        //font-family: 'Karla',verdana,arial,sans-serif;
        //font-family: 'Roboto',verdana,arial,sans-serif;
        //font-family: 'Lato',verdana,arial,sans-serif;
        //font-family: 'Domine',verdana,arial,sans-serif;
        //font-family: 'Chivo',verdana,arial,sans-serif;
        //font-family: 'Cardo',verdana,arial,sans-serif;
        //font-family: 'Arvo',verdana,arial,sans-serif;
        //font-family: 'Poppins',verdana,arial,sans-serif;
        //font-family: 'Ubuntu',verdana,arial,sans-serif;
        //font-family: 'Fontin',verdana,arial,sans-serif;
        //font-family: 'Raleway',verdana,arial,sans-serif;
        //font-family: 'Merriweather',verdana,arial,sans-serif;
        //font-family: 'Crimson Text', verdana,arial,sans-serif;
        //font-family: verdana,arial,sans-serif;
        //font-family: arial,sans-serif;
        line-height: 125%;
        font-size: 130%;
        text-align: justify;
        text-justify:inter-word;
    }
    div.text_cell code {
        background: transparent;
        color: #000000;
        font-weight: 400;
        font-size: 12pt;
        //font-style: bold;
        font-family:  'Source Code Pro', Consolas, monocco, monospace;
   }
    h1 {
        font-family: 'Open sans',verdana,arial,sans-serif;
	}

    div.input_area {
        background: #F6F6F9;
        border: 1px solid #586e75;
    }

    .text_cell_render h1 {
        font-weight: 200;
        font-size: 30pt;
        line-height: 100%;
        color:#c76c0c;
        margin-bottom: 0.5em;
        margin-top: 1em;
        display: block;
        white-space: wrap;
        text-align: left;
    }
    h2 {
        font-family: 'Open sans',verdana,arial,sans-serif;
        text-align: left;
    }
    .text_cell_render h2 {
        font-weight: 200;
        font-size: 16pt;
        font-style: italic;
        line-height: 100%;
        color:#c76c0c;
        margin-bottom: 0.5em;
        margin-top: 1.5em;
        display: block;
        white-space: wrap;
        text-align: left;
    }
    h3 {
        font-family: 'Open sans',verdana,arial,sans-serif;
    }
    .text_cell_render h3 {
        font-weight: 200;
        font-size: 14pt;
        line-height: 100%;
        color:#d77c0c;
        margin-bottom: 0.5em;
        margin-top: 2em;
        display: block;
        white-space: wrap;
        text-align: left;
    }
    h4 {
        font-family: 'Open sans',verdana,arial,sans-serif;
    }
    .text_cell_render h4 {
        font-weight: 100;
        font-size: 14pt;
        color:#d77c0c;
        margin-bottom: 0.5em;
        margin-top: 0.5em;
        display: block;
        white-space: nowrap;
    }
    h5 {
        font-family: 'Open sans',verdana,arial,sans-serif;
    }

    .text_cell_render h5 {
        font-weight: 200;
        font-style: normal;
        color: #1d3b84;
        font-size: 16pt;
        margin-bottom: 0em;
        margin-top: 0.5em;
        display: block;
        white-space: nowrap;
    }
    div.output_subarea.output_text.output_pyout {
        overflow-x: auto;
        overflow-y: scroll;
        max-height: 50000px;
    }
    div.output_subarea.output_stream.output_stdout.output_text {
        overflow-x: auto;
        overflow-y: scroll;
        max-height: 50000px;
    }
    div.output_wrapper{
        margin-top:0.2em;
        margin-bottom:0.2em;
}

    code{
        font-size: 6pt;

    }
    .rendered_html code{
    background-color: transparent;
    }
    ul{
        margin: 2em;
    }
    ul li{
        padding-left: 0.5em;
        margin-bottom: 0.5em;
        margin-top: 0.5em;
    }
    ul li li{
        padding-left: 0.2em;
        margin-bottom: 0.2em;
        margin-top: 0.2em;
    }
    ol{
        margin: 2em;
    }
    ol li{
        padding-left: 0.5em;
        margin-bottom: 0.5em;
        margin-top: 0.5em;
    }
    ul li{
        padding-left: 0.5em;
        margin-bottom: 0.5em;
        margin-top: 0.2em;
    }
    a:link{
       color:#447adb;
    }
    a:visited{
       color: #1d3b84;
    }
    a:hover{
       color: #1d3b84;
    }
    a:focus{
       color:#447adb;
    }
    a:active{
       font-weight: bold;
       color:#447adb;
    }
    .rendered_html :link {
       text-decoration: underline;
    }
    .rendered_html :hover {
       text-decoration: none;
    }
    .rendered_html :visited {
      text-decoration: none;
    }
    .rendered_html :focus {
      text-decoration: none;
    }
    .rendered_html :active {
      text-decoration: none;
    }
    .warning{
        color: rgb( 240, 20, 20 )
    }
    hr {
      color: #f3f3f3;
      background-color: #f3f3f3;
      height: 1px;
    }
    blockquote{
      display:block;
      background: #fcfcfc;
      border-left: 5px solid #c76c0c;
      font-family: 'Open sans',verdana,arial,sans-serif;
      width:680px;
      padding: 10px 10px 10px 10px;
      text-align:justify;
      text-justify:inter-word;
      }
      blockquote p {
        margin-bottom: 0;
        line-height: 125%;
        font-size: 100%;
      }
</style>

This book is written in Jupyter Notebook, a browser based interactive Python environment that mixes Python, text, and math. I choose it because of the interactive features - I found Kalman filtering nearly impossible to learn until I started working in an interactive environment. It is difficult to form an intuition about many of the parameters until you can change them and immediately see the output. An interactive environment also allows you to play 'what if' scenarios. "What if I set $\mathbf{Q}$ to zero?" It is trivial to find out with Jupyter Notebook.

Another reason I choose it is because most textbooks leaves many things opaque. For example, there might be a beautiful plot next to some pseudocode. That plot was produced by software, but software that is not available to the reader. I want everything that went into producing this book to be available to you. How do you plot a covariance ellipse? You won't know if you read most books. With Jupyter Notebook all you have to do is look at the source code.

Even if you choose to read the book online you will want Python and the SciPy stack installed so that you can write your own Kalman filters. There are many different ways to install these libraries, and I cannot cover them all, but I will cover a few typical scenarios.

## Installing the SciPy Stack

This book requires IPython, Jupyter, NumPy, SciPy, SymPy, and Matplotlib. The SciPy stack of NumPy, SciPy, and Matplotlib depends on third party Fortran and C code, and is not trivial to install from source code. The SciPy website strongly urges using a pre-built installation, and I concur with this advice.

I use the Anaconda distribution from Continuum Analytics. This is an excellent distribution that combines all of the packages listed above, plus many others. Installation is very straightforward, and it can be done alongside other Python installations you might already have on your machine. It is free to use. You may download it from here: http://continuum.io/downloads I strongly recommend using the latest Python 3 version that they provide.

There are other choices for installing the SciPy stack. You can find instructions here: http://scipy.org/install.html

Many Linux distributions come with these packages preinstalled. However, they are often somewhat dated and they will need to be updated as the book depends on recent versions of all. Updating a specific Linux installation is beyond the scope of this book. An advantage of the Anaconda distribution is that it does not modify your local Python installation, so you can install it and not break your linux distribution.

## Installing FilterPy

FilterPy is a Python library that implements all of the filters used in this book, and quite a few others. Installation is easy using `pip`. Issue the following from the command prompt:

     pip install filterpy

FilterPy is written by me, and the latest development version is always available at https://github.com/rlabbe/filterpy.

## Downloading and Running the Book

The book is stored in a github repository. From the command line type the following:

    git clone https://github.com/rlabbe/Kalman-and-Bayesian-Filters-in-Python.git

If you do not have git installed, browse to https://github.com/rlabbe/Kalman-and-Bayesian-Filters-in-Python where you can download the book via your browser.

Now, from the command prompt change to the directory that was just created, and then run Jupyter notebook:

    cd Kalman-and-Bayesian-Filters-in-Python
    juptyer notebook

A browser window should launch showing you all of the chapters in the book. Browse to the first chapter by clicking on it, then open the notebook in that subdirectory by clicking on the link.

More information about running the notebook can be found here:

http://jupyter-notebook-beginner-guide.readthedocs.org/en/latest/execute.html

## Using Juptyer Notebook

A complete tutorial on Jupyter Notebook is beyond the scope of this book. Many are available online. In short, Python code is placed in cells. These are prefaced with text like `In [1]:`, and the code itself is in a boxed area. If you press CTRL-ENTER while focus is inside the box the code will run and the results will be displayed below the box. Like this:

```python
print(3+7.2)
```

    10.2

If you have this open in Jupyter Notebook now, go ahead and modify that code by changing the expression inside the print statement and pressing CTRL+ENTER. The output should be changed to reflect what you typed in the code cell.

## SymPy

SymPy is a Python package for performing symbolic mathematics. The full scope of its abilities are beyond this book, but it can perform algebra, integrate and differentiate equations, find solutions to differential equations, and much more. For example, we use use it to compute the Jacobian of matrices and expected value integral computations.

First, a simple example. We will import SymPy, initialize its pretty print functionality (which will print equations using LaTeX). We will then declare a symbol for SymPy to use.

```python
import sympy
sympy.init_printing(use_latex='mathjax')

phi, x = sympy.symbols('\phi, x')
phi
```

$$\phi$$

Notice how it prints the symbol `phi` using LaTeX. Now let's do some math. What is the derivative of $\sqrt{\phi}$?

```python
sympy.diff('sqrt(phi)')
```

$$\frac{1}{2 \sqrt{\phi}}$$

We can factor equations

```python
sympy.factor(phi**3 -phi**2 + phi - 1)
```

$$\left(\phi - 1\right) \left(\phi^{2} + 1\right)$$

and we can expand them.

```python
((phi+1)*(phi-4)).expand()
```

$$\phi^{2} - 3 \phi - 4$$

You can evauate an equation for specific values of its variables:

```python
w =x**2 -3*x +4
print(w.subs(x, 4))
print(w.subs(x, 12))
```

    8
    112

You can also use strings for equations that use symbols that you have not defined:

```python
x = sympy.expand('(t+1)*2')
x
```

$$2 t + 2$$

Now let's use SymPy to compute the Jacobian of a matrix. Given the function

$$h=\sqrt{(x^2 + z^2)}$$

find the Jacobian with respect to x, y, and z.

```python
x, y, z = sympy.symbols('x y z')

H = sympy.Matrix([sympy.sqrt(x**2 + z**2)])

state = sympy.Matrix([x, y, z])
H.jacobian(state)
```

$$\left[\begin{matrix}\frac{x}{\sqrt{x^{2} + z^{2}}} & 0 & \frac{z}{\sqrt{x^{2} + z^{2}}}\end{matrix}\right]$$

Now let's compute the discrete process noise matrix $\mathbf Q$ given the continuous process noise matrix
$$\mathbf Q = \Phi_s \begin{bmatrix}0&0&0\\0&0&0\\0&0&1\end{bmatrix}$$

The integral is

$$\mathbf Q = \int_0^{\Delta t} \mathbf F(t)\mathbf Q\mathbf F^T(t)\, dt$$

where
$$\mathbf F(\Delta t) = \begin{bmatrix}1 & \Delta t & {\Delta t}^2/2 \\ 0 & 1 & \Delta t\\ 0& 0& 1\end{bmatrix}$$

```python
dt = sympy.symbols('\Delta{t}')
F_k = sympy.Matrix([[1, dt, dt**2/2],
                    [0,  1,      dt],
                    [0,  0,       1]])
Q = sympy.Matrix([[0,0,0],
                  [0,0,0],
                  [0,0,1]])

sympy.integrate(F_k*Q*F_k.T,(dt, 0, dt))
```

$$\left[\begin{matrix}\frac{\Delta{t}^{5}}{20} & \frac{\Delta{t}^{4}}{8} & \frac{\Delta{t}^{3}}{6}\\\frac{\Delta{t}^{4}}{8} & \frac{\Delta{t}^{3}}{3} & \frac{\Delta{t}^{2}}{2}\\\frac{\Delta{t}^{3}}{6} & \frac{\Delta{t}^{2}}{2} & \Delta{t}\end{matrix}\right]$$


```python
from IPython.display import Image
Image('../../../python_for_probability_statistics_and_machine_learning.jpg')
```

[Python for Probability, Statistics, and Machine Learning](https://www.springer.com/fr/book/9783319307152)

This chapter takes a geometric view of probability theory and relates it to
familiar concepts in linear algebra and geometry.  This approach connects your
natural geometric intuition to the key abstractions in probability that can
help guide your reasoning.  This is particularly important in probability
because it is easy to be misled. We need a bit of rigor and some
intuition to guide us.

In grade school, you were introduced to the natural numbers (i.e., `1,2,3,..`)
and you learned how to manipulate them by operations like addition,
subtraction, and multiplication. Later, you were introduced to positive and
negative numbers and were again taught how to manipulate them. Ultimately, you
were introduced to the calculus of the real line, and learned how to
differentiate, take limits, and so on. This progression provided more
abstractions, but also widened the field of problems you could successfully
tackle. The same is true of probability. One way to think about probability is
as a  new number concept that allows you to tackle problems that have a special
kind of *uncertainty* built into them. Thus, the key idea is that there is some
number, say $x$, with a traveling companion, say, $f(x)$, and this companion
represents the uncertainties about the value of $x$ as if looking at the number
$x$ through a frosted window. The degree of opacity of the window is
represented by $f(x)$.  If we want to manipulate $x$, then we have to figure
out what to do with $f(x)$. For example if we want $y= 2 x $, then we have to
understand how $f(x)$ generates $f(y)$.

Where is the *random* part?   To conceptualize this, we need still another
analogy: think about a beehive with the swarm around it representing $f(x)$,
and the hive itself, which you can barely see through the swarm, as $x$.    The
random piece is you don't know *which* bee in particular is going to sting you!
Once this happens the uncertainty evaporates.
Up until that happens, all we have is a concept of a swarm (i.e., density of
bees) which represents a *potentiality* of which bee will ultimately sting.
In summary, one way to think about probability is as a way of carrying through
mathematical reasoning (e.g., adding, subtracting, taking
limits) with a notion of potentiality that is so-transformed by these
operations.

## Understanding Probability Density

In order to understand the heart of modern probability, which is built
on the Lesbesgue theory of integration, we need to extend the concept
of integration from basic calculus. To begin, let us consider the
following piecewise function

$$
f(x)  =  \begin{cases}
                1 & \mbox{if }  0 < x \leq 1 \\\
                2 & \mbox{if }  1 < x \leq 2 \\\
                0 & \mbox{otherwise }
            \end{cases}
$$

 as shown in [Figure](#fig:intro_001). In calculus, you learned
Riemann integration, which you can apply here  as

$$
\int_0^2 f(x) dx = 1 + 2 = 3
$$

 which has the usual interpretation as the area of the two rectangles
that make up $f(x)$.  So far, so good.

With Lesbesgue integration, the idea is very similar except that we
focus on the y-axis instead of moving along the x-axis. The question
is given $f(x) = 1$, what is the set of $x$ values for which this is
true? For our example, this is true whenever $x\in (0,1]$. So now we
have a correspondence between the values of the function (namely, `1`
and `2`) and the sets of $x$ values for which this is true, namely,
$\lbrace (0,1] \rbrace$ and $\lbrace (1,2] \rbrace$, respectively. To
compute the integral, we simply take the function values (i.e., `1,2`)
and some way of measuring the size of the corresponding interval
(i.e.,  $\mu$) as in the following:

$$
\int_0^2 f d\mu = 1 \mu(\lbrace (0,1] \rbrace) + 2 \mu(\lbrace (1,2] \rbrace)
$$

We have suppressed some of the notation above to emphasize generality. Note
that we obtain the same value of the integral as in the Riemann case when
$\mu((0,1]) = \mu((1,2]) = 1$. By introducing the $\mu$ function as  a way of
measuring the intervals above, we have introduced another degree of freedom in
our integration. This accommodates many weird functions that are not tractable
using the usual Riemann theory, but we refer you to a proper introduction to
Lesbesgue integration for further study [[jones2001lebesgue]](#jones2001lebesgue).  Nonetheless,
the key step in the above discussion is the introduction of the $\mu$ function,
which we will encounter again as the so-called probability density function.

## Random Variables

Most introductions to probability jump straight into *random variables* and
then explain how to compute complicated integrals. The problem with this
approach is that it skips over some of the important subtleties that we will now
consider. Unfortunately, the term *random variable* is not very descriptive. A
better term is *measurable function*.  To understand why this is a better term,
we have to dive into the formal constructions of probability by way of a simple
example.

Consider tossing a fair six-sided die. There are only six outcomes possible,

$$
\Omega=\lbrace 1,2,3,4,5,6 \rbrace
$$

As we know, if the die is fair, then the probability of each outcome is $1/6$.
To say this formally, the measure  of each set (i.e., $\lbrace 1 \rbrace,\lbrace
2 \rbrace,\ldots,\lbrace 6 \rbrace$) is $\mu(\lbrace 1 \rbrace ) =\mu(\lbrace 2
\rbrace ) \ldots = \mu(\lbrace 6 \rbrace ) = 1/6$. In this case, the $\mu$
function we discussed earlier is the usual *probability* mass function, denoted by
$\mathbb{P}$. The measurable function  maps a set into a
number on the real line. For example, $ \lbrace 1 \rbrace \mapsto 1 $ is
one such  uninteresting function.

Now, here's where things get interesting. Suppose you were asked to construct a
fair coin from the fair die. In other words, we want to throw the die and then
record the outcomes as if we had just tossed a fair coin. How could we do this?
One way would be to define a measurable function that says if the die comes up
`3` or less, then we declare *heads* and otherwise declare *tails*. This has
some strong intuition behind it, but let's articulate it in terms of formal
theory.  This strategy creates two different non-overlapping sets $\lbrace
1,2,3 \rbrace$ and $\lbrace 4,5,6 \rbrace$. Each set has the same probability
*measure*,

$$
\begin{eqnarray*}
\mathbb{P}(\lbrace 1,2,3 \rbrace)  & = &  1/2  \\\
\mathbb{P}(\lbrace 4,5,6 \rbrace) & = & 1/2
\end{eqnarray*}
$$

 And the problem is solved. Everytime the die comes up
$\lbrace 1,2,3 \rbrace$, we record heads and record tails otherwise.

Is this the only way to construct a fair coin experiment from a
fair die? Alternatively, we can define the sets as $\lbrace 1 \rbrace$,
$\lbrace 2 \rbrace$, $\lbrace 3,4,5,6 \rbrace$. If we define the corresponding
measure for each set as the following

$$
\begin{eqnarray*}
\mathbb{P}(\lbrace 1 \rbrace)  & = &  1/2  \\\
\mathbb{P}(\lbrace 2 \rbrace) & = & 1/2 \\\
\mathbb{P}(\lbrace 3,4,5,6 \rbrace) & = & 0
\end{eqnarray*}
$$

  then, we have another solution to the fair coin problem. To
implement this, all we do is ignore every time the die shows `3,4,5,6` and
throw again. This is wasteful, but it solves the problem.  Nonetheless,
we hope you can see how the interlocking pieces of the theory provide a
framework for carrying the notion of uncertainty/potentiality from one problem
to the next (e.g., from the fair die to the fair coin).

Let's consider a slightly more interesting problem where we toss two dice. We
assume that each throw is *independent*, meaning that the outcome of one does
not influence the other. What are the sets in this case? They are all pairs
of possible outcomes from two throws as shown below,

$$
\Omega = \lbrace (1,1),(1,2),\ldots,(5,6),(6,6) \rbrace
$$

  What are the measures of each of these sets? By virtue of the
independence claim, the measure of each is the product of the respective measures
of each element.  For instance,

$$
\mathbb{P}((1,2)) = \mathbb{P}(\lbrace 1 \rbrace) \mathbb{P}(\lbrace 2 \rbrace) = \frac{1}{6^2}
$$

 With all that established, we can ask the following
question: what is the probability that the sum of the dice equals
seven? As before, the first thing to do is characterize the
measurable function for this as $X:(a,b) \mapsto (a+b)$.  Next, we
associate all of the $(a,b)$ pairs with their sum. We can create a
Python dictionary for this as shown,

```python
d={(i,j):i+j for i in range(1,7) for j in range(1,7)}
```

 The next step is to collect all of the $(a,b)$ pairs that sum to
each of the possible values from two to twelve.

```python
from collections import defaultdict
dinv = defaultdict(list)
for i,j in d.iteritems():
    dinv[j].append(i)
```

**Programming Tip.**

The `defaultdict` object from the built-in collections module creates dictionaries with
default values when it encounters a new key. Otherwise, we would have had to
create default values manually for a regular dictionary.

 For example, `dinv[7]` contains the following list of pairs that
sum to seven,

```python
[(1, 6), (2, 5), (5, 2), (6, 1), (4, 3), (3, 4)]
```

    [(1, 6), (2, 5), (5, 2), (6, 1), (4, 3), (3, 4)]

The next step is to compute the probability measured for each of these items.
Using the independence assumption, this means we have to compute the sum of the
products of the individual item probabilities in `dinv`. Because we know that
each outcome is equally likely, every term in the sum equals $1/36$. Thus, all
we have to do is count the number of items in the corresponding list for each
key in `dinv` and divide by `36`. For example,  `dinv[11]` contains `[(5, 6),
(6, 5)]`.  The probability of `5+6=6+5=11` is the probability of this set which
is composed of the sum of the probabilities of the individual elements
`{(5,6),(6,5)}`. In this case, we have $\mathbb{P}(11) = \mathbb{P}(\lbrace
(5,6) \rbrace)+ \mathbb{P}(\lbrace (6,5) \rbrace) = 1/36 + 1/36 = 2/36$.
Repeating this procedure for all the elements, we derive the probability mass
function as shown below,

```python
X={i:len(j)/36. for i,j in dinv.iteritems() }
print X
{2: 0.027777777777777776,
 3: 0.05555555555555555,
 4: 0.08333333333333333,
 5: 0.1111111111111111,
 6: 0.1388888888888889,
 7: 0.16666666666666666,
 8: 0.1388888888888889,
 9: 0.1111111111111111,
 10: 0.08333333333333333,
 11: 0.05555555555555555,
 12: 0.027777777777777776}
```

    {2: 0.027777777777777776, 3: 0.05555555555555555, 4: 0.08333333333333333, 5: 0.1111111111111111, 6: 0.1388888888888889, 7: 0.16666666666666666, 8: 0.1388888888888889, 9: 0.1111111111111111, 10: 0.08333333333333333, 11: 0.05555555555555555, 12: 0.027777777777777776}

    {2: 0.027777777777777776,
     3: 0.05555555555555555,
     4: 0.08333333333333333,
     5: 0.1111111111111111,
     6: 0.1388888888888889,
     7: 0.16666666666666666,
     8: 0.1388888888888889,
     9: 0.1111111111111111,
     10: 0.08333333333333333,
     11: 0.05555555555555555,
     12: 0.027777777777777776}

**Programming Tip.**

In the preceding code note that `36.` is written with
the trailing decimal mark. This is a good habit to get into because division
in Python 2.x is integer division by default, which is not what we want here.
This can be fixed with a top-level `from __future__ import division`, but
that's easy to forget to do, especially when you are passing code
around and others may not reflexively do the future import.

The above example exposes the elements of probability theory that
are in play for this simple problem while deliberately suppressing some of the
gory technical details. With this framework, we can ask other questions like
what is the probability that half the product of three dice will exceed the
their sum? We can solve this using the same method as in the following. First,
let's create the first mapping,

```python
d={(i,j,k):((i*j*k)/2>i+j+k) for i in range(1,7)
                                for j in range(1,7)
                                   for k in range(1,7)}
```

 The keys of this dictionary are the triples and the values are the
logical values of whether or not half the product of three dice exceeds their sum.
Now, we do the inverse mapping to collect the corresponding lists,

```python
dinv = defaultdict(list)
for i,j in d.iteritems(): dinv[j].append(i)
```

 Note that `dinv` contains only two keys, `True` and `False`. Again,
because the dice are independent, the probability of any triple is $1/6^3$.
Finally, we collect this for each outcome as in the following,

```python
X={i:len(j)/6.0**3 for i,j in dinv.iteritems() }
print X
{False: 0.37037037037037035, True: 0.6296296296296297}
```

    {False: 0.37037037037037035, True: 0.6296296296296297}

    {False: 0.37037037037037035, True: 0.6296296296296297}

 Thus, the probability of half the product of three dice exceeding their sum is
`136/(6.0**3) = 0.63`. The set that is induced by the random variable has only
two elements in it, `True` and `False`, with $\mathbb{P}(\mbox{True})=136/216$
and $\mathbb{P}(\mbox{False})=1-136/216$.

As a final example to exercise another layer of generality, let is consider the
first problem with the two dice where we want the probability of a
seven, but this time one of the dice is no longer fair.  The distribution for
the unfair die is the following:

$$
\begin{eqnarray*}
\mathbb{P}(\lbrace 1\rbrace)=\mathbb{P}(\lbrace 2 \rbrace)=\mathbb{P}(\lbrace 3 \rbrace)  = \frac{1}{9} \\\
\mathbb{P}(\lbrace 4\rbrace)=\mathbb{P}(\lbrace 5 \rbrace)=\mathbb{P}(\lbrace 6 \rbrace)  = \frac{2}{9}
\end{eqnarray*}
$$

From our earlier work, we know the elements corresponding to the sum of seven
are the following:

$$
\lbrace (1,6),(2,5),(3,4),(4,3),(5,2),(6,1) \rbrace
$$

  Because we still have the independence assumption, all we need to
change is the probability computation of each of elements. For example, given
that the first die is the unfair one, we have

$$
\mathbb{P}((1,6)) = \mathbb{P}(1)\mathbb{P}(6) = \frac{1}{9} \times \frac{1}{6}
$$

  and likewise for $(2,5)$ we have the following:

$$
\mathbb{P}((2,5)) = \mathbb{P}(2)\mathbb{P}(5) = \frac{1}{9} \times \frac{1}{6}
$$

  and so forth. Summing all of these gives the following:

$$
\mathbb{P}_X(7) =  \frac{1}{9} \times \frac{1}{6}
                      +\frac{1}{9} \times \frac{1}{6}
                      +\frac{1}{9} \times \frac{1}{6}
                      +\frac{2}{9} \times \frac{1}{6}
                      +\frac{2}{9} \times \frac{1}{6}
                      +\frac{2}{9} \times \frac{1}{6}  = \frac{1}{6}
$$

 Let's try computing this using Pandas instead
of Python dictionaries. First, we construct
a `DataFrame` object with an index of tuples
consisting of all pairs of possible dice outcomes.

```python
from pandas import DataFrame
d=DataFrame(index=[(i,j) for i in range(1,7) for j in range(1,7)],
            columns=['sm','d1','d2','pd1','pd2','p'])
```

 Now, we can populate the columns that we set up above
where the outcome of the first die is the `d1` column and
the outcome of the second die is `d2`,

```python
d.d1=[i[0] for i in d.index]
d.d2=[i[1] for i in d.index]
```

 Next, we compute the sum of the dices in the `sm`
column,

```python
d.sm=map(sum,d.index)
```

 With that established, the DataFrame now looks like
the following:

```python
d.head(5) # show first five lines
```

      sm
      d1
      d2
      pd1
      pd2
      p

      (1, 1)
      2
      1
      1
      NaN
      NaN
      NaN

      (1, 2)
      3
      1
      2
      NaN
      NaN
      NaN

      (1, 3)
      4
      1
      3
      NaN
      NaN
      NaN

      (1, 4)
      5
      1
      4
      NaN
      NaN
      NaN

      (1, 5)
      6
      1
      5
      NaN
      NaN
      NaN

 Next, we fill out the probabilities for each face of the
unfair die (`d1`) and the fair die (`d2`),

```python
d.loc[d.d1<=3,'pd1']=1/9.
d.loc[d.d1 > 3,'pd1']=2/9.
d.pd2=1/6.
d.head(10)
```

      sm
      d1
      d2
      pd1
      pd2
      p

      (1, 1)
      2
      1
      1
      0.111111
      0.166667
      NaN

      (1, 2)
      3
      1
      2
      0.111111
      0.166667
      NaN

      (1, 3)
      4
      1
      3
      0.111111
      0.166667
      NaN

      (1, 4)
      5
      1
      4
      0.111111
      0.166667
      NaN

      (1, 5)
      6
      1
      5
      0.111111
      0.166667
      NaN

      (1, 6)
      7
      1
      6
      0.111111
      0.166667
      NaN

      (2, 1)
      3
      2
      1
      0.111111
      0.166667
      NaN

      (2, 2)
      4
      2
      2
      0.111111
      0.166667
      NaN

      (2, 3)
      5
      2
      3
      0.111111
      0.166667
      NaN

      (2, 4)
      6
      2
      4
      0.111111
      0.166667
      NaN

 Finally, we can compute the joint probabilities
for the sum of the shown faces as the following:

```python
d.p = d.pd1 * d.pd2
d.head(5)
```

      sm
      d1
      d2
      pd1
      pd2
      p

      (1, 1)
      2
      1
      1
      0.111111
      0.166667
      0.0185185

      (1, 2)
      3
      1
      2
      0.111111
      0.166667
      0.0185185

      (1, 3)
      4
      1
      3
      0.111111
      0.166667
      0.0185185

      (1, 4)
      5
      1
      4
      0.111111
      0.166667
      0.0185185

      (1, 5)
      6
      1
      5
      0.111111
      0.166667
      0.0185185

 With all that established, we can compute the
density of all the dice outcomes by using `groupby` as in the
following,

```python
d.groupby('sm')['p'].sum()
```

    sm
    2     0.018519
    3     0.037037
    4     0.055556
    5     0.092593
    6     0.129630
    7     0.166667
    8     0.148148
    9     0.129630
    10    0.111111
    11    0.074074
    12    0.037037
    Name: p, dtype: float64

  These examples have shown how the theory of probability
breaks down sets and measurements of  those sets and how these can be
combined to develop the probability mass functions for new random
variables.

## Continuous Random Variables

The same ideas work with continuous variables but managing the sets
becomes trickier because the real line, unlike discrete sets, has many
limiting properties already built into it that have to be handled
carefully.  Nonetheless, let's start with an example that should
illustrate the analogous ideas. Suppose a random variable $X$ is
uniformly distributed on the unit interval. What is the probability
that the variable takes on values less than 1/2?

In order to build intuition onto the discrete case, let's go back to our
dice-throwing experiment with the fair dice. The sum of the values of the dice
is a measurable function,

$$
Y \colon \lbrace 1,2,\dots,6 \rbrace^2 \mapsto \lbrace 2,3,\ldots, 12 \rbrace
$$

 That is, $Y$ is a mapping of the cartesian product of sets to a
discrete set of outcomes. In order to compute probabilities of the set of
outcomes, we need to derive the probability measure for $Y$, $\mathbb{P}_Y$,
from the corresponding probability measures for each die. Our previous discussion
went through the mechanics of that. This means that

$$
\mathbb{P}_Y \colon \lbrace 2,3,\ldots,12 \rbrace \mapsto [0,1]
$$

 Note there is a separation between the function definition and where the
target items of the function are measured in probability. More bluntly,

$$
Y \colon A \mapsto B
$$

 with,

$$
\mathbb{P}_Y \colon B \mapsto [0,1]
$$

 Thus, to compute $\mathbb{P}_Y$, which is derived
from other random variables, we have to express the equivalence classes
in $B$ in terms of their progenitor $A$ sets.

The situation for continuous variables follows the same pattern, but
with many more deep technicalities that we are going to skip. For the continuous
case, the random variable is now,

$$
X \colon \mathbb{R} \mapsto \mathbb{R}
$$

 with corresponding probability measure,

$$
\mathbb{P}_X \colon \mathbb{R} \mapsto [0,1]
$$

 But where are the corresponding sets here? Technically, these are the
*Borel* sets, but we can just think of them as intervals. Returning to our
question, what is the probability that a uniformly distributed random variable
on the unit interval takes values less than $1/2$? Rephrasing this question
according to the framework, we have the following:

$$
X \colon [0,1] \mapsto [0,1]
$$

 with corresponding,

$$
\mathbb{P}_X \colon [0,1] \mapsto [0,1]
$$

 To answer the question,  by the definition of the uniform random
variable on the unit interval, we compute the following integral,

$$
\mathbb{P}_X([0,1/2]) = \mathbb{P}_X(0 < X < 1/2) =  \int_0^{1/2} dx  = 1/2
$$

  where the above integral's $dx$ sweeps through intervals of the
$B$-type. The measure of any $dx$ interval (i.e., $A$-type set) is equal to
$dx$, by definition of the uniform random variable. To get all the moving parts
into one notationally rich integral, we can also  write this  as,

$$
\mathbb{P}_X(0 < X < 1/2) = \int_0^{ 1/2 } d\mathbb{P}_X(dx) = 1/2
$$

Now, let's consider a slightly more complicated and interesting example. As
before, suppose we have a uniform random variable, $X$ and let us introduce
another random variable defined,

$$
Y = 2 X
$$

  Now, what is the probability that $0 < Y < \frac{1}{2}$?
To express this in our framework, we write,

$$
Y \colon [0,1] \mapsto [0,2]
$$

 with corresponding,

$$
\mathbb{P}_Y \colon [0,2] \mapsto [0,1]
$$

 To answer the question, we need to measure the set $[0,1/2]$, with
the probability measure for $Y$, $\mathbb{P}_Y([0,1/2])$. How can we do this?
Because $Y$ is derived from the $X$ random variable, as with the fair-dice
throwing experiment, we have to create a set of equivalences in the target
space (i.e., $B$-type sets) that reflect back on the input space (i.e.,
$A$-type sets). That is, what is the interval $[0,1/2]$ equivalent to in terms
of the $X$ random variable? Because, functionally, $Y=2 X$, then the $B$-type
interval $[0,1/2]$ corresponds to the $A$-type interval $[0,1/4]$. From the
probability measure of $X$, we compute this with the integral,

$$
\mathbb{P}_Y([0,1/2]) =\mathbb{P}_X([0,1/4])=  \int_0^{1/4} dx  = 1/4
$$

Now, let's up the ante and consider the following random variable,

$$
Y = X^2
$$

 where now $X$ is still uniformly distributed, but now over the
interval $[-1/2,1/2]$.  We can express this in our framework as,

$$
Y \colon [-1/2,1/2] \mapsto [0,1/4]
$$

 with corresponding,

$$
\mathbb{P}_Y \colon [0,1/4] \mapsto [0,1]
$$

 What is the $\mathbb{P}_Y(Y < 1/8)$?  In other words, what is the
measure of the set $B_Y= [0,1/8]$? As before, because $X$ is derived from our
uniformly distributed random variable, we have to reflect the $B_Y$ set onto
sets of the $A$-type.  The thing to recognize is that because $X^2$
is symmetric about zero, all $B_Y$ sets reflect back into two sets.
This means that for any set $B_Y$, we have the correspondence $B_Y = A_X^+ \cup
A_X^{-}$.  So, we have,

$$
B_Y=\Big\lbrace 0<Y<\frac{1}{8}\Big\rbrace=\Big\lbrace 0<X<\frac{1}{\sqrt{8}} \Big\rbrace \bigcup \Big\lbrace -\frac{1}{\sqrt {8}}<X<0 \Big\rbrace
$$

  From this perspective, we have the following solution,

$$
\mathbb{P}_Y(B_Y)=\mathbb{P}(A_X^+)/2 + \mathbb{P}(A_X^{-})/2
$$

  where the $\frac{1}{2}$ comes from normalizing the $\mathbb{P}_Y$ to
one. Also,

$$
A_X^+    = \Big\lbrace 0<  X<\frac{1}{\sqrt{8}}   \Big\rbrace
$$

$$
\
A_X^{-}  = \Big\lbrace -\frac{1}{\sqrt {8}} <  X<0 \Big\rbrace
$$

  Therefore,

$$
\mathbb{P}_Y(B_Y) = \frac{1}{2\sqrt 8} + \frac{1}{2\sqrt 8}
$$

  because $\mathbb{P}(A_X^+) =\mathbb{P}(A_X^-) = 1/\sqrt 8$.  Let's
see if this comes out using the usual transformation of variables method from
calculus. Using this method, the density $f_Y(y) = f_X(\sqrt y)/(2 \sqrt y) =
\frac{1}{2 \sqrt y} $. Then, we obtain,

$$
\int_0^{\frac{1}{8}} \frac{1}{2 \sqrt y} dy = \frac{1}{\sqrt 8}
$$

  which is what we got using the sets method. Note that you would
favor the calculus method in practice, but it is important to
understand the deeper mechanics, because sometimes
the usual calculus method fails, as the next problem shows.

## Transformation of Variables Beyond Calculus

Suppose $X$ and $Y$ are uniformly distributed in the unit interval and we
define $Z$ as

$$
Z = \frac{X}{Y-X}
$$

  What is the $f_Z(z)$? If you try this using the usual calculus
method, you will fail (try it!). The problem is one
of the technical prerequisites for the calculus method is not in force.

The key observation is that $Z \notin [-1,0]$. If this were
possible, the $X$ and $Y$ would have different signs, which cannot happen,
given that $X$ and $Y$ are uniformly distributed over $[0,1]$. Now, let's
consider when $Z>0$. In this case, $Y>X$ because $Z$ cannot be positive
otherwise. For the density function, we are interested in the set
$\lbrace  0 < Z < z \rbrace $. We want to compute

$$
\mathbb{P}(Z<z) = \int \int B_1 dX dY
$$

 with,

$$
B_1 = \lbrace 0 < Z < z \rbrace
$$

  Now, we have to translate that interval into an interval
relevant to $X$ and $Y$. For $0 < Z$, we have $ Y > X$. For $Z < z $,
we have $Y > X(1/z+1)$. Putting this together gives

$$
A_1 = \lbrace \max (X,X(1/z+1)) < Y < 1 \rbrace
$$

  Integrating this over $Y$ as follows,

$$
\int_0^1\lbrace\max(X,X(1/z+1))<Y<1 \rbrace dY=\frac{z-X-Xz}{z}\mbox{ where } z > \frac{X}{1-X}
$$

 and integrating this one more time over $X$ gives

$$
\int_0^{\frac{z}{1+z}} \frac{-X+z-Xz}{z} dX = \frac{z}{2(z+1)} \mbox{ where }  z > 0
$$

 Note that this is the computation for the *probability*
itself, not the probability density function. To get that, all we have
to do is differentiate the last expression to obtain

$$
f_Z(z) = \frac{1}{(z+1)^2} \mbox{ where }  z > 0
$$

 Now we need to compute this density using the same process
for when $z < -1$.  We want  the interval $ Z < z $ for when $z < -1$.
For a fixed $z$, this is equivalent to $ X(1+1/z) < Y$. Because $z$
is negative, this also means that $Y < X$. Under these terms, we
have the following integral,

$$
\int_0^1 \lbrace X(1/z+1) <Y< X\rbrace dY = -\frac{X}{z} \mbox{ where }  z < -1
$$

 and integrating this one more time over $X$ gives the following

$$
-\frac{1}{2 z} \mbox{ where }  z < -1
$$

 To get the density for $z<-1$, we differentiate this with
respect to $z$ to obtain the following,

$$
f_Z(z) = \frac{1}{2 z^2} \mbox{ where }  z < -1
$$

 Putting this all together, we obtain,

$$
f_Z(z) =
\begin{cases}
      \frac{1}{(z+1)^2}     & \mbox{if }  z > 0 \\\
      \frac{1}{2 z^2}       & \mbox{if }  z < -1 \\\
      0 & \mbox{otherwise }
\end{cases}
$$

  We will leave it as an exercise to show that this
integrates out to one.

## Independent Random Variables

Independence is a standard assumption. Mathematically, the
necessary and sufficient condition for independence between two
random variables $X$  and $Y$ is the following:

$$
\mathbb{P}(X,Y) = \mathbb{P}(X)\mathbb{P}(Y)
$$

 Two random variables $X$  and $Y$
are *uncorrelated* if,

$$
\mathbb{E}(X-\overline{X})\mathbb{E}(Y-\overline{Y})=0
$$

 where $\overline{X}=\mathbb{E}(X)$ Note that uncorrelated random
variables are sometimes called *orthogonal* random variables.  Uncorrelatedness
is a weaker property than independence, however. For example, consider the
discrete random variables $X$ and $Y$ uniformly distributed over the set
$\lbrace 1,2,3 \rbrace$ where

$$
X =
\begin{cases}
1 & \mbox{if } \omega =1 \\\
0 & \mbox{if } \omega =2  \\\
-1 & \mbox{if } \omega =3
\end{cases}
$$

 and also,

$$
Y =
\begin{cases}
0 & \mbox{if } \omega =1 \\\
1 & \mbox{if } \omega =2 \\\
0 & \mbox{if } \omega =3
\end{cases}
$$

 Thus, $\mathbb{E}(X)=0$ and $\mathbb{E}(X Y)=0$, so
$X$ and $Y$ are uncorrelated. However, we have

$$
\mathbb{P}(X=1,Y=1)=0\neq \mathbb{P}(X=1)\mathbb{P}(Y=1)=\frac{1}{9}
$$

 So, these two random variables are *not* independent.
Thus, uncorrelatedness does not imply independence, generally, but
there is the important case of Gaussian random variables for which
it does. To see this, consider the probability density function
for two zero-mean, unit-variance Gaussian random variables $X$ and
$Y$,

$$
f_{X,Y}(x,y) = \frac{e^{\frac{x^2-2 \rho x
   y+y^2}{2 \left(\rho^2-1\right)}}}{2 \pi
   \sqrt{1-\rho^2}}
$$

 where $\rho:=\mathbb{E}(X Y)$  is the correlation coefficient.  In
the uncorrelated case where $\rho=0$, the probability density function factors
into the following,

$$
f_{X,Y}(x,y)=\frac{e^{-\frac{1}{2}\left(x^2+y^2\right)}}{2\pi}=\frac{e^{-\frac{x^2}{2}}}{\sqrt{2\pi}}\frac{e^{-\frac{y^2}{2}}}{\sqrt{2\pi}}    =f_X(x)f_Y(y)
$$

 which means that $X$ and $Y$ are independent.

Independence and conditional independence are closely related, as in the following:

$$
\mathbb{P}(X,Y\vert Z) =\mathbb{P}(X\vert Z) \mathbb{P}(Y\vert Z)
$$

 which says that $X$ and $Y$ and independent conditioned
on $Z$.  Conditioning independent random variables can break
their independence. For example, consider two independent
Bernoulli-distributed random variables,  $X_1, X_2\in\lbrace 0,1
\rbrace$.  We define $Z=X_1+X_2$. Note that $Z\in \lbrace
0,1,2 \rbrace$. In the case where $Z=1$, we have,

$$
\mathbb{P}(X_1\vert Z=1) >0
$$

$$
\
\mathbb{P}(X_2\vert Z=1) >0
$$

 Even though $X_1,X_2$ are independent,
after conditioning on $Z$, we have the following,

$$
\mathbb{P}(X_1=1,X_2=1\vert Z=1)=0\neq \mathbb{P}(X_1=1\vert Z=1)\mathbb{P}(X_2=1\vert Z=1)
$$

 Thus, conditioning on $Z$ breaks the independence of
$X_1,X_2$. This also works in the opposite direction ---
conditioning can make dependent random variables independent.
Define $Z_n=\sum_i^n X_i$ with $X_i$ independent, integer-valued
random variables.  The $Z_n$ variables are
dependent because they stack the same telescoping set of
$X_i$ variables. Consider the following,

$$
\begin{equation}
\mathbb{P}(Z_1=i,Z_3=j\vert Z_2=k) = \frac{\mathbb{P}(Z_1=i,Z_2=k,Z_3=j)}{\mathbb{P}(Z_2 =k)}
\label{_auto1} \tag{1}
\end{equation}
$$

$$
\begin{equation} \
=\frac{\mathbb{P}(X_1 =i)\mathbb{P}(X_2 =k-i)\mathbb{P}(X_3 =j-k) }{\mathbb{P}(Z_2 =k)}
\end{equation}
\label{eq:condIndep} \tag{2}
$$

 where the factorization comes from the independence of
the $X_i$ variables. Using the definition of conditional
probability,

$$
\mathbb{P}(Z_1=i\vert Z_2)=\frac{\mathbb{P}(Z_1=i,Z_2=k)}{\mathbb{P}(Z_2=k)}
$$

 We can continue to expand Equation ref{eq:condIndep},

$$
\mathbb{P}(Z_1=i,Z_3=j\vert Z_2=k) =\mathbb{P}(Z_1 =i\vert Z_2) \frac{\mathbb{P}( X_3 =j-k)\mathbb{P}( Z_2 =k)}{\mathbb{P}( Z_2 =k)}
$$

$$
\
                                   =\mathbb{P}(Z_1 =i\vert Z_2)\mathbb{P}(Z_3 =j\vert Z_2)
$$

 where $\mathbb{P}(X_3=j-k)\mathbb{P}(Z_2=k)=
\mathbb{P}(Z_3=j,Z_2)$.  Thus, we see that dependence between
random variables can be broken by conditioning to create
conditionally independent random variables. As we have just
witnessed, understanding how conditioning influences independence
is important and is the main topic of
study in Probabilistic Graphical Models, a field
with many algorithms and concepts to extract these
notions of conditional independence from graph-based
representations of random variables.

## Classic Broken Rod Example

Let's do one last example to exercise fluency in our methods by
considering the following classic problem: given a rod of unit-length,
broken independently and randomly at two places, what is the
probability that you can assemble the three remaining pieces into a
triangle? The first task is to find a representation of a triangle as
an easy-to-apply constraint. What we want is something like the
following:

$$
\mathbb{P}(\mbox{ triangle exists }) = \int_0^1 \int_0^1 \lbrace \mbox{ triangle exists }  \rbrace dX dY
$$

 where $X$ and $Y$ are independent and uniformly distributed
in the unit-interval.  Heron's formula for the area of the triangle,

$$
\mbox{ area } = \sqrt{(s-a)(s-b)(s-c)s}
$$

 where $s = (a+b+c)/2$ is what we need. The idea is that this
yields a valid area only when each of the terms under the square root is
greater than or equal to zero. Thus, suppose that we have

$$
\begin{eqnarray*}
a  & = &  X  \\\
b  & = & Y-X \\\
c  & = & 1-Y
\end{eqnarray*}
$$

 assuming that $Y>X$.  Thus, the criterion for a valid triangle boils down
to

$$
\lbrace (s > a) \wedge (s > b) \wedge (s > c) \wedge (X<Y) \rbrace
$$

 After a bit of manipulation, this  consolidates into:

$$
\Big\lbrace \frac{1}{2} < Y < 1 \bigwedge \frac{1}{2}(2 Y-1) < X < \frac{1}{2} \Big\rbrace
$$

  which we integrate out by $dX$ first to obtain

$$
\mathbb{P}(\mbox{ triangle exists }) = \int_{0}^1 \int_{0}^1 \Big\lbrace \frac{1}{2} < Y < 1 \bigwedge \frac{1}{2}(2 Y-1) < X < \frac{1}{2} \Big\rbrace dX dY
$$

$$
\mathbb{P}(\mbox{ triangle exists }) = \int_{\frac{1}{2}}^1 (1-Y) dY
$$

 and then by $dY$ to obtain finally,

$$
\mathbb{P}(\mbox{ triangle exists }) = \frac{1}{8}
$$

 when $Y>X$. By symmetry, we get the same result for $X>Y$. Thus, the
final result is the following:

$$
\mathbb{P}(\mbox{ triangle exists }) = \frac{1}{8}+\frac{1}{8} = \frac{1}{4}
$$

We can quickly check using this result using Python for the case $Y>X$ using
the following code:

```python
>>> import numpy as np
>>> x,y = np.random.rand(2,1000) # uniform rv
>>> a,b,c = x,(y-x),1-y # 3 sides
>>> s = (a+b+c)/2
>>> np.mean((s>a) & (s>b)  & (s>c) & (y>x)) # approx 1/8=0.125
```

    0.13700000000000001

**Programming Tip.**

The chained logical `&` symbols above tell Numpy that the logical operation
should be considered element-wise.


```python
!conda info
```

    Current conda install:

                   platform : win-64
              conda version : 4.3.22
           conda is private : False
          conda-env version : 4.3.22
        conda-build version : not installed
             python version : 3.6.1.final.0
           requests version : 2.14.2
           root environment : C:\Users\USER\Anaconda3  (writable)
        default environment : C:\Users\USER\Anaconda3
           envs directories : C:\Users\USER\Anaconda3\envs
                              C:\Users\USER\AppData\Local\conda\conda\envs
                              C:\Users\USER\.conda\envs
              package cache : C:\Users\USER\Anaconda3\pkgs
                              C:\Users\USER\AppData\Local\conda\conda\pkgs
               channel URLs : https://repo.continuum.io/pkgs/free/win-64
                              https://repo.continuum.io/pkgs/free/noarch
                              https://repo.continuum.io/pkgs/r/win-64
                              https://repo.continuum.io/pkgs/r/noarch
                              https://repo.continuum.io/pkgs/pro/win-64
                              https://repo.continuum.io/pkgs/pro/noarch
                              https://repo.continuum.io/pkgs/msys2/win-64
                              https://repo.continuum.io/pkgs/msys2/noarch
                config file : C:\Users\USER\.condarc
                 netrc file : None
               offline mode : False
                 user-agent : conda/4.3.22 requests/2.14.2 CPython/3.6.1 Windows/10 Windows/10.0.14393
              administrator : False

# Variables

```python
x = 2
y = '3'
print(x+int(y))

z = [1, 2, 3] #List
w = (2, 3, 4) #Tuple

import numpy as np
q = np.array([1, 2, 3]) #numpy.ndarray
type(q)
```

    5

    numpy.ndarray

# Console input and output

```python
MyName = input('My name is: ')
print('Hello, '+MyName)
```

    My name is: david
    Hello, david

# File input and output

```python
fid = open('msg.txt','w')
fid.write('demo of writing.\n')
fid.write('Second line')
fid.close()

fid = open('msg.txt','r')
msg = fid.readline()
print(msg)
msg = fid.readline()
print(msg)

fid.close()
```

    demo of writing.

    Second line

```python
fid = open('msg.txt','r')
msg = fid.readlines()
print(msg)
```

    ['demo of writing.\n', 'Second line']

```python
fid = open('msg.txt','r')
msg = fid.read()
print(msg)
```

    demo of writing.
    Second line

```python
import numpy as np
x = np.linspace(0, 2*np.pi,4)
y = np.cos(x)

#Stack arrays in sequence vertically (row wise).
data = np.vstack((x,y))  #上下對隊齊好
dataT = data.T #Transpose

np.savetxt('data.txt', data, delimiter=',')
z = np.loadtxt('data.txt', delimiter=',')

print(x)
print(y)
print(data)
print(dataT)
print(z)
```

    [ 0.          2.0943951   4.1887902   6.28318531]
    [ 1.  -0.5 -0.5  1. ]
    [[ 0.          2.0943951   4.1887902   6.28318531]
     [ 1.         -0.5        -0.5         1.        ]]
    [[ 0.          1.        ]
     [ 2.0943951  -0.5       ]
     [ 4.1887902  -0.5       ]
     [ 6.28318531  1.        ]]
    [[ 0.          2.0943951   4.1887902   6.28318531]
     [ 1.         -0.5        -0.5         1.        ]]

```python
import numpy as np
x = np.linspace(0, 2*np.pi,20)
y = np.cos(x)
z = np.sin(x)

%matplotlib inline
import matplotlib.pyplot as plt

#使用 help(plt.plot) 可以看到所有畫圖玩法
plt.plot(x,y,'b')
plt.plot(x,y,'go', label = 'cos(x)')
plt.plot(x,z,'r')
plt.plot(x,z,'go', label = 'sin(x)')
plt.legend(loc='best') # 放到最好的位置
plt.xlim([0, 2*np.pi])
```

```python
import numpy as np
x = np.linspace(0, 2*np.pi,20)
y = np.cos(x)
z = np.sin(x)

%matplotlib inline
import matplotlib.pyplot as plt

#使用 help(plt.plot) 可以看到所有畫圖玩法
plt.subplot(2,1,1) #分成兩張圖 形式是(row, column, order)
plt.plot(x,y,'b')
plt.plot(x,y,'go', label = 'cos(x)')
plt.legend(loc='best') #放到最好的位置

plt.subplot(2,1,2) #分成兩張圖
plt.plot(x,z,'r')
plt.plot(x,z,'go', label = 'sin(x)')
plt.legend(loc='best') #放到最好的位置

plt.xlim([0, 2*np.pi])
```

# Functions, Conditions, Loop

```python
import numpy as np

def f(x):
    return x**2

x = np.linspace(0,5,10)
y = f(x)

print(y)
```

    [  0.           0.30864198   1.2345679    2.77777778   4.9382716
       7.71604938  11.11111111  15.12345679  19.75308642  25.        ]

```python
import numpy as np

def f(x): #這是個奇怪的練習用函數
    res = x
    if res < 3:
        res = np.nan #<3就傳 Not a Number
    elif res < 15:
        res = x**3
    else:
        res = x**4
    return res

x = np.linspace(0,10,20)
y = np.empty_like(x)
#Return a new array with the same shape and type as a given array.
#傳一個跟x一樣的array回來

i = 0
for xi in x:
    y[i] = f(xi)
    i = i + 1
print(y)

%matplotlib inline
import matplotlib.pyplot as plt

plt.plot(x,y,'bp')
plt.xlim([0,11])
```

# Matrices, linear equations

```python
A = np.array([[1,2],[3,2]])
B = np.array([1,0])

# x = A^-1 * b
sol1 = np.dot(np.linalg.inv(A),B)
print(sol1)
sol2 = np.linalg.solve(A,B)
print(sol2)

import sympy as sym
sym.init_printing()
#This will automatically enable the best printer available in your environment.

x,y = sym.symbols('x y')
z = sym.linsolve([3*x+2*y-1,x+2*y],(x,y))
z
#sym.pprint(z) The ASCII pretty printer
```

# Non-linear equation

```python
from scipy.optimize import fsolve

def f(z):  #用z參數來表示x和y，做函數運算
    x = z[0]
    y = z[1]
    return [x+2*y, x**2+y**2-1]

z0 = [0,1]
z = fsolve(f,z0)
print(z)
print(f(z))
```

    [-0.89442719  0.4472136 ]
    [0.0, -1.1102230246251565e-16]

# Integration

```python
from scipy.integrate import quad

def f(x):
    return x**2

quad(f,0,2) #計算積分值

import sympy as sym
sym.init_printing()
x = sym.Symbol('x')
f = sym.integrate(x**2,x)
f.subs(x,2) #將值帶入函數中
f
```

# Derivative

```python
from scipy.misc import derivative

def f(x):
    return x**2

print(derivative(f,2,dx=0.01)) #dx表示精確程度

import sympy as sym
sym.init_printing()
x = sym.Symbol('x')
f = sym.diff(x**3,x)
f.subs(x,2) #將值帶入函數中，得解
f
```

# Interpolation

```python
from scipy.interpolate import interp1d #中間的字是1不是L喔!!!

x = np.arange(0,6,1)
y = np.array([0.2,0.3,0.5,1.0,0.9,1.1])

%matplotlib inline
import matplotlib.pyplot as plt
plt.plot(x,y,'bo')

xp = np.linspace(0,5,100) #為了顯示差別把點增加

y1 = interp1d(x,y,kind='linear') #一階
plt.plot(xp,y1(xp),'r-')

y2 = interp1d(x,y,kind='quadratic') #二階
plt.plot(xp,y2(xp),'k--')

y3 = interp1d(x,y,kind='cubic') #三階
plt.plot(xp,y3(xp),'g--')

```

# Linear regression

```python
import numpy as np
x = np.array([0,1,2,3,4,5])
y = np.array([0.1,0.2,0.3,0.5,0.8,2.0 ])

#多項式逼近法，選擇階層
p1 = np.polyfit(x,y,1)
print(p1)
p2 = np.polyfit(x,y,2)
print(p2)
p3 = np.polyfit(x,y,3)
print(p3)

%matplotlib inline
import matplotlib.pyplot as plt
plt.plot(x,y,'ro')

# np.polyval表示多項式的值，把係數p_帶入多項式x求出來的值
xp = np.linspace(0,5,100)
plt.plot(xp, np.polyval(p1,xp), 'b-', label='linear') #這個字是polyvaL喔!!
plt.plot(xp, np.polyval(p2,xp), 'g--', label='quadratic')
plt.plot(xp, np.polyval(p3,xp), 'k:', label='cubic')
plt.legend(loc='best')
```

# Nonlinear regression

```python
import numpy as np
from scipy.optimize import curve_fit

x = np.array([0,1,2,3,4,5])
y = np.array([0.1,0.2,0.3,0.5,0.8,2.0 ])

#多項式逼近法，選擇階層
p1 = np.polyfit(x,y,1)
print(p1)
p2 = np.polyfit(x,y,2)
print(p2)
p3 = np.polyfit(x,y,3)
print(p3)

#使用指數對數
def f(x,a):
    return 0.1 * np.exp(a*x)
a = curve_fit(f,x,y)[0] #非線性回歸，Use non-linear least squares to fit a function，取第0項
print('a='+str(a))

%matplotlib inline
import matplotlib.pyplot as plt
plt.plot(x,y,'ro')

# np.polyval表示多項式的值，把係數p_帶入多項式x求出來的值
xp = np.linspace(0,5,100)
plt.plot(xp, np.polyval(p1,xp), 'b-', label='linear') #這個字是polyvaL喔!!
plt.plot(xp, np.polyval(p2,xp), 'g--', label='quadratic')
plt.plot(xp, np.polyval(p3,xp), 'k:', label='cubic')
plt.plot(xp, f(xp,a), 'c', label='nonlinear')
plt.legend(loc='best')
```

# Differential equation

```python
from scipy.integrate import odeint

def dydt(y,t,a):
    return -a * y

a = 0.5
t = np.linspace(0,20)
y0 = 5.0
y = odeint(dydt,y0,t,args=(a,))

%matplotlib inline
import matplotlib.pyplot as plt
plt.plot(t,y)
plt.xlabel('time')
plt.ylabel('y')
```

# Nonlinear optimization

```python
#概念:要有Objective、Constraint，然後初始猜想值
import numpy as np
from scipy.optimize import minimize
```

```python
def objective(x): #此函數求最小值
    x1 = x[0]
    x2 = x[1]
    x3 = x[2]
    x4 = x[3]
    return x1*x4*(x1+x2+x3)+x3

#用減法做比較
def constraint1(x):
    return x[0]*x[1]*x[2]*x[3] - 25.0

#用減法做比較
def constraint2(x):
    sum_sq = 40.0
    for i in range(0,4):
        sum_sq = sum_sq - x[i]**2
    return sum_sq

#初始猜想值
x0 = [1,5,5,1]
print(objective(x0))

#設定值域
b = (1.0,5.0) #x的值域
bnds = (b,b,b,b) #四個值域都一樣b
con1 = {'type':'ineq','fun': constraint1} #第一個是不等式
con2 = {'type':'eq','fun': constraint2} #第二個需要等式
cons = [con1,con2] #cons合成一個list

sol = minimize(objective,x0,method='SLSQP',\
              bounds = bnds, constraints = cons)

```

    16

```python
print(sol)
```

         fun: 17.01401724563517
         jac: array([ 14.57227015,   1.37940764,   2.37940764,   9.56415057])
     message: 'Optimization terminated successfully.'
        nfev: 30
         nit: 5
        njev: 5
      status: 0
     success: True
           x: array([ 1.        ,  4.7429961 ,  3.82115462,  1.37940765])

```python
print(sol.fun)
```

    17.01401724563517

```python
print(sol.x)
```

    [ 1.          4.7429961   3.82115462  1.37940765]

```python

```


```python
%%HTML
<style>
.rendered_html table, .rendered_html th, .rendered_html tr, .rendered_html td {
     font-size: 100%;
}
</style>
```

# Metody Numeryczne

## Elementy analizy numerycznej

### dr hab. inż. Jerzy Baranowski, Prof. AGH.

## Informacje ogólne
- Katedra Automatyki i Robotyki, C3, p. 214
- Konsultacje
    - Czwartki 11:00-12:00
    (o ile nie ma Kolegium Wydziałowego lub seminarium)
- jb@agh.edu.pl
- wykłady dostępne tutaj: https://github.com/KAIR-ISZ/public_lectures

# Reprezentacja liczb

## Kod binarny

- Zapis liczby z wykorzystaniem dwóch symboli **1** i **0**
- Podstawa współczesnego sposobu reprezentacji informacji

## Zamierzchła historia

- Pingala, Chandaḥśāstra i Prozodia
    - Ok. 4 wiek pne
    - Wykorzystanie zapisu w formie zer i jedynek do opisu metrum
- Chiny, hexagramy, Shao Yong, I-Ching
- Leibniz

## Algebra Boole'a

$$
\begin{align}
x \land y & = xy & \mathsf{Koniunkcja}\\
x \lor y & = x+y-xy & \mathsf{Alternatywa}\\
\neg x & =1-x & \mathsf{Negacja}\\
x \rightarrow y & = (\neg x\lor y) & \mathsf{Implikacja}\\
x \oplus y & = (x \lor y)\land\neg(x\land y) & \mathsf{EXOR}\\
x = y & = \neg(x\oplus y) & \mathsf{Równoważność}\\
\end{align}
$$

## Nieco mniej zamierzchła historia
- 1937 Shannon – przekaźnikowa realizacja operacji binarnych i algebry Boole’a
- 1937 Stibitz – Pierwszy komputer przekaźnikowy (dodawanie)

## Kod binarny
| **0**   | **0**   | **1**   | **0**   | **1**   | **0**   | **1**   | **1**   |
|---------|---------|---------|---------|---------|---------|---------|---------|
| $2^{7}$ | $2^{6}$ | $2^{5}$ | $2^{4}$ | $2^{3}$ | $2^{2}$ | $2^{1}$ | $2^{0}$ |

Co daje $ =2^5+2^3+2^1+2^0=32+8+2+1=43$

## Liczby naturalne
- Ogólnie zakres od 0 do 2n-1
- 8 bit – zakres od 0 do 255
- 16 bit – zakres od 0 do 65,535 (short, int)
- 32 bit – zakres od 0 do 4,294,967,295 (long)

W Pythonie i matlabie za bardzo nie przejmujemy się typami, chyba że je wymusimy

## Operacje na liczbach binarnych

- Dodawanie
    - 0+0=0
    - 0+1=1
    - 1+0=1
    - 1+1=0, przenieś 1
- Jak w dodawaniu pisemnym

`` 1 1 1 1 1   ``(cyfry przenoszone)
``   0 1 1 0 1 ``(1310)
``+  1 0 1 1 1 ``(2310)
``------------ ``
``=1 0 0 1 0 0 `` (3610)

## Operacje na liczbach binarnych

- Odejmowanie
    - 0-0=0
    - 0-1=1, pożyczka 1
    - 1-0=1
    - 1-1=0,
- Analogicznie

``    *   * * *  ``(pożyczki)
``  1 1 0 1 1 1 0``(11010)
``-     1 0 1 1 1``(2310)
``--------------- ``
``= 1 0 1 0 1 1 1`` (8710)

## Co z liczbami ujemnymi?

Uzupełniamy zapis o tzw. bit znaku

| **1**   | **0**   | **1**   | **0**   | **1**   | **0**   | **1**   | **1**   |
|---------|---------|---------|---------|---------|---------|---------|---------|
| S       | $2^{6}$ | $2^{5}$ | $2^{4}$ | $2^{3}$ | $2^{2}$ | $2^{1}$ | $2^{0}$ |

Co daje $ =(-1)^1(2^3+2^1+2^0)=-(8+2+1)=-11$

Zmieniają się zakresy:
- 8 bit (-128 do 127)
- 16 bit (−32,768 do 32,767)
- itd

## Problemy

- Niepraktyczny zapis
- Trzeba przekodowywać wyniki operacji
- Potencjalnie podatniejsze na błędy

## Kod uzupełnienia do 2 (U2)
| **1**   | **1**   | **1**   | **1**   | **1**   | **0**   | **1**   | **1**   |
|---------|---------|---------|---------|---------|---------|---------|---------|
| $-2^{7}$| $2^{6}$ | $2^{5}$ | $2^{4}$ | $2^{3}$ | $2^{2}$ | $2^{1}$ | $2^{0}$ |

Co daje $ =-2^7+2^6+2^5+2^4+2^3+2^1+2^0$

$=-128+64+32+16+8+2+1=-5$

## Bardzo łatwa konwersja
- Liczby dodatnie są takie same jak były
- Aby zamienić liczbę na jej przeciwną wystarczy zanegować wszystkie bity i do wyniku dodać 1 (*w obie strony*)

| **0**    | **0**   | **0**   | **0**   | **0**   | **1**   | **0**   | **1**   | 510  | oryginał  |
|----------|---------|---------|---------|---------|---------|---------|---------|-----------------|-----------|
| 1    | 1   | 1   | 1   | 1   | 0   | 1   | 0   |                 | negacja   |
| **1**    | **1**   | **1**   | **1**   | **1**   | **0**   | **1**   | **1**   | -510 | dodanie 1 |
| 0    | 0   | 0   | 0   | 0   | 1   | 0   | 0   |                 | negacja   |
| **0**    | **0**   | **0**   | **0**   | **0**   | **1**   | **0**   | **1**   | 510  | dodanie 1 |
| -$2^{7}$ | $2^{6}$ | $2^{5}$ | $2^{4}$ | $2^{3}$ | $2^{2}$ | $2^{1}$ | $2^{0}$ |                 |           |

## Jaka z tego korzyść?
- Odejmowanie staje się dodawaniem (prawie)
$$ A - B = A + \neg B + 1$$
- Przykład 13 – 7 (na 8 bitach)

``  1 1 1 1     1  ``(cyfry przenoszone)
``  0 0 0 0 1 1 0 1``(1310)
``  1 1 1 1 1 0 0 0``(zanegowane 710)
``+               1``(jedynka)
``-----------------``
``= 0 0 0 0 0 1 1 0`` (610)

## Operacje na liczbach binarnych
Mnożenie również przypomina mnożenie pisemne

``          1 0 1 1`` 1110
``        * 1 0 1 0`` 1010
``      -----------``
``          0 0 0 0``
``  +     1 0 1 1   ``
``  +   0 0 0 0``
``  + 1 0 1 1``
``  ---------------``
``  = 1 1 0 1 1 1 0`` 11010

# Metody Numeryczne

## Reprezentacja liczb wymiernych

### dr hab. inż. Jerzy Baranowski, Prof. AGH.

## A co z ułamkami?
Są dwa sposoby zapisu liczb niecałkowitych
- Stałoprzecinkowy (stałopozycyjny)
- Zmiennoprzecinkowy (zmiennopozycyjny)

## Zapis stałoprzecinkowy
| **1**   | **0**   | **1**   | **1**   | **1**   | **0**   | **0**   | **0**   |
|---------|---------|---------|---------|---------|---------|---------|---------|
| $2^{1}$ | $2^{0}$ | $2^{-1}$ | $2^{-2}$ | $2^{-3}$ | $2^{-4}$ | $2^{-5}$ | $2^{-6}$ |

$$
2^1+2^{-1}+2^{-2}+2^{-3}=2+\frac{1}{2}+\frac{1}{4}+\frac{1}{8}=2.875
$$

## Zalety zapisu stałoprzecinkowego
- Nie ma różnicy w kodowaniu
- Mamy stale określoną dokładność, którą możemy w miarę dokładnie kształtować
- Stosunkowa prostota
- Małe wymagania sprzętowe

## Wady zapisu stałoprzecinkowego
Problemy z dokładnością, np. nie da się dokładnie przedstawić liczby 0.1
- Na 3 bitach części ułamkowej różnica wynosi 0.025
- Na 7 bitach części ułamkowej różnica wynosi ok. 0.001

## Jak wykonujemy działania?
- Działania wykonujemy traktując zapis liczby stałoprzecinkowej jako normalną binarną
- Kod U2 dalej działa
- Należy pamiętać, że wtedy liczba jest pomnożona przez 2n gdzie n to ilość bitów części ułamkowej
- W liczbach poddanych działaniu liczba bitów części całkowitej i ułamkowej musi być równa

## Działania stałoprzecinkowe
- Dodawanie wykonujemy identycznie
- W przypadku mnożenia wynik musimy podzielić przez 2n
- Mnożenie liczb stałoprzecinkowych przez potęgę 2 polega tylko na przesuwaniu bitów (bardzo proste w realizacji)

| 1             | 0             | 1   | 1   | 1   | 0   | 0   | 0   |   |
|---------------|---------------|-----|-----|-----|-----|-----|-----|---|
| 0             | 0             | 1   | 0   | 1   | 1   | 1   | 0   | Podzielenie przez $2^{2}$  |
| $2^{1}$ | $2^{0}$ | $2^{-1}$ | $2^{-2}$ | $2^{-3}$ | $2^{-4}$ | $2^{-5}$ | $2^{-6}$ ||

## Format zmiennoprzecinkowy
- Bardziej zaawansowany sposób przedstawiania liczb
- Ustandaryzowany normą IEEE
- Dający pod pewnymi względami większą dokładność

## Format zmiennoprzecinkowy

Reprezentacja liczby

$$
x=S\cdot M\cdot B^E
$$

- S – znak (*sign*)
- M – mantysa (*mantissa*, także *fraction*)
- B – podstawa (*base*, zazwyczaj 2, rzadziej 10)
- E - wykładnik (*exponent*)

## Mantysa
- Liczba odpowiadająca za ułamkową część zapisu
- Format stałoprzecinkowy, zazwyczaj liczba z przedziału [1,2)

## Podstawa i wykładnik

- Pozwalają na określenie szerokiego zakresu
- Ze względu na kodowanie, zazwyczaj podstawa to 2
- Wykładnik może być ujemny lub dodatni.
- Wykładnik koduje się w U2, lub też wprowadza się przesunięcie

## Działania na liczbach zmiennoprzecinkowych
Dodawanie i odejmowanie

$$
x_1\pm x_2=\left(M_1\pm M_2\cdot B^{E_2-E_1}\right)\cdot B^{E_1}
$$

Mnożenie i dzielenie

$$
x_1\cdot x_2=(S_1\cdot S_2)\cdot (M_1\cdot M_2)\cdot B^{E_1+E_2}
$$

$$
x_1 / x_2=(S_1\cdot S_2)\cdot (M_1/ M_2)\cdot B^{E_1-E_2}
$$

## Dzielenie
- Mając możliwość zapisu liczby ulamkowej można sformułować operację dzielenia.
- Istnieje wiele algorytmów np.
    - *restoring division*
    - *non-restoring division*
    - SRT
    - algorytm Newtona-Raphsona
    - algorytm Goldschmidta
- Są one już zaimplementowane, jedno dzielenie zazwyczaj wymaga przeprowadzenia 3-4 mnożeń

## Ważne formaty – IEEE Single precision

- 8 bitów wykładnika, wykładnik przesunięty o 127 (zamiana z -126 do 127 na 1 do 244)
- 24 bity mantysy, ale zawsze koduje się tylko 23 po kropce, przed kropką jest 1
- Specjalne zapisy nieskończoności i błędów
- w NumPy - ``float32``

## Ważne formaty – IEEE Double precision

- 11 bitów wykładnika, wykładnik przesunięty o 1023 (zamiana z -1022 do 1023 na 1 do 2046)
- 53 bity mantysy, ale zawsz koduje się tylko 52 po kropce, przed kropką jest 1
- Specjalne zapisy nieskończoności i błędów
- w NumPy - ``float64``, ale w zasadzie każda liczba w Pythonie i Matlabie to double, chyba że wymusimy inaczej

## Wyświetlanie liczb
- Normalnie
- Notacja inżynierska
    - $3700=3.7\cdot10^3$, $0.12=120\cdot10^{-3}$
- Notacja naukowa
    - ``3700=3.7E3``, ``0.12=1.2E-1``

# Metody Numeryczne

## Błedy numeryczne

### dr hab. inż. Jerzy Baranowski, Prof. AGH.

## Podstawowe definicje
Wartość dokładna
$$y=\tilde{y}+\varepsilon$$
- $\tilde{y}$ - wartość przybliżona
- $\varepsilon$ - błąd

## Błąd bezwzględny
Wartość bezwzględna różnicy między rozwiązaniem dokładnym i przybliżonym
$$ \varepsilon=|y-\tilde{y}|$$

## Błąd względny
Stosunek błędu bezwzględnego do wartości bezwzględnej rozwiązania
$$\eta=\frac{|y-\tilde{y}|}{|y|}=\left|\frac{y-\tilde{y}}{y}\right|=\left|1-\frac{\tilde{y}}{y}\right|$$
Czasami błąd względny wyrażamy w procentach

## Przykłady
Pierwiastek kwadratowy ze 122

$$
\begin{align}
y{}&=\sqrt{122}\approx 11.04536\\
\tilde{y}{}&=11\\
\varepsilon{}&=|y-\tilde{y}|=0.04536\\
\eta{}&=\frac{|y-\tilde{y}|}{|y|}=0.00411
\end{align}
$$

## Przykłady
Liczba obywateli Polski (stan na ostatni spis powszechny z 2011)

$$
\begin{align}
y{}&=38\ 538\ 447\\
\tilde{y}{}&=38\ 500\ 000\\
\varepsilon{}&=|y-\tilde{y}|=38\ 447\\
\eta{}&=\frac{|y-\tilde{y}|}{|y|}=9.97627\cdot10^{-4}\approx 0.001
\end{align}
$$

## Przykłady
Obliczanie stałej grawitacji
$$
\begin{align}
y{}&=6.673841\cdot10^{-11}\\
\tilde{y}{}&=6.7\cdot10^{-11}\\
\varepsilon{}&=|y-\tilde{y}|=2.6159\cdot10^{-13}\\
\eta{}&=\frac{|y-\tilde{y}|}{|y|}=0.00391
\end{align}
$$

## Źródła błędów
Błędy powstające przy formułowaniu zagadnienia
- Błędy pomiaru
- Błędy wynikające z przyjęcia określonych przybliżeń opisu zjawisk fizycznych

Błędy powstające przy obliczeniach
- Błędy grube (pomyłki)
- Błędy metody (obcięcia)
- Błędy zaokrągleń

## Błędy grube
- Błąd przy wpisywaniu wzoru do komputera
np. ``x=A/b`` zamiast ``x=A\b``
- Zła implementacja algorytmu
- Niewłaściwa kolejność wykonywania działań

## Błędy metody (obcięcia)
- Błędy obcięcia są nieodłącznym elementem obliczeń numerycznych.
- Błąd obcięcia jest to błąd wynikający z tego, że do uzyskania dokładnego rozwiązania potrzebujemy wykonać nieskończenie wiele obliczeń

## Przykłady błędów metody
Można wykazać, że
$$
\begin{align}
\sin x={}&x-\frac{x^3}{3!}+\frac{x^5}{5!}-\frac{x^7}{7!}+\ldots=\\
={}&\sum\limits_{n=0}^\infty(-1)^n\frac{x^{2n+1}}{(2n+1)!}
\end{align}
$$
Błędem odcięcia będzie
$$
\sin x\approx x-\frac{x^3}{3!}+\frac{x^5}{5!}
$$

## Przykłady błędów metody

Metoda bisekcji

```python
def bisection(f,a,b,N):
    a_n = a
    b_n = b
    for n in range(1,N+1):
        m_n = (a_n + b_n)/2
        f_m_n = f(m_n)
        if f(a_n)*f_m_n < 0:
            a_n = a_n
            b_n = m_n
        elif f(b_n)*f_m_n < 0:
            a_n = m_n
            b_n = b_n
    return (a_n + b_n)/2
```

Szukamy pierwiastka wielomianu $x^2-2$, w przedziale $[1,2]$. Rozwiązanie to $\sqrt{2}$.

```python
f = lambda x: x**2 - 2 # definicja funkcji
bisection(f,1,2,5) # 5 kroków
```

    1.421875

```python
bisection(f,1,2,10) # 10 kroków
```

    1.41455078125

```python
bisection(f,1,2,15) # 15 kroków
```

    1.4141998291015625

```python
import numpy as np
np.sqrt(2)
```

    1.4142135623730951

## Błąd metody - podsumowanie
- Praktycznie wszystkie metody numeryczne mają jakiś błąd metody
- Dobre algorytmy podają jednak jego oszacowanie, w ten sposób wiemy jak daleko jesteśmy od rozwiązania nawet jak przerwiemy obliczenia

Metody Numeryczne 2020/2. Reprezentacja liczb/Reprezentacja liczb.ipynb