# Implementing Discrete Fourier Transform(DFT) Using Python

## Table of Contents
* [Introduction](#Introduction)
* [Python Implementation](#Implementation)
* [Testing the Code](#Testing)
* [Conclusion](#Conclusion)

## Introduction

Digital images are usually stored and displayed in **space domain**. That is, each point/pixel in the image contains an integer value that shows the color intensity value. For example, if we have 8x8 image, then there are 64 values that are stored in each pixel location.

However, images can be transformed in to their corresponding **frequecy domain** representation. The advantage of the transformation is that several image processing tasks are well done in their transformed format. For example, they can be used for:

* *Image Enhancement*
* *Image Restoration*
* *Image Coding*
* *Image Compression*

Then, after these processes are performed, the processed image can be returned back to its original space domain form by using inverse transform process. There are several types of transforms, such as:

* *Discrete Fourier Transform (DFT)*
* *Discrete Cosine Transform (DCT)*
* *Walsh-Hadamard Transform*
* *Haar Transform*

In this homework, we are only concerned with DFT.

A 2-dimensional DFT (2D-DFT) decomposes an image into its sinusoidal components (sines and cosines). As explained above, the input is the image in its **spatial domain**. In contrast, the output will be the image's representation in its fourier or **frequency domain**. DFT is a complex number transform as it has both the real (cosine) and imaginary (sine) components as an output.

Let the size of an input image be NxN. The general form is:

$$
\begin{align}
F(u,v) = \frac{1}{N^2}\sum_{x=0}^{N-1}\sum_{y=0}^{N-1}f(x,y) e^{(-j2\pi\frac{ux+vy}{N})} \; where \; u,v=0,1,2,...N-1
\end{align}
$$

The above formula is **forward DFT transformation**.

Similarly, for **inverse DFT transformation**:
$$
\begin{align}
f(u,v) = \sum_{u=0}^{N-1}\sum_{v=0}^{N-1}F(u,v) e^{(+j2\pi\frac{ux+vy}{N})} \; where \; x,y=0,1,2,...N-1
\end{align}
$$

* $ k(x,y,u,v)=e^{(-j2\pi\frac{ux+vy}{N})} $ is called **basis function (kernel function)**

To find the real and imaginary part of the transformed image:

$$
\begin{align}
* e^{\pm jx} = \cos(x)\pm j\sin(x)
\end{align}
$$

Since the kernel function in DFT is separable:
$$\begin{align}
k(x,y,u,v)=e^{(-j2\pi\frac{ux+vy}{N})} = e^{(-j2\pi\frac{ux}{N})}e^{(-j2\pi\frac{vy}{N})}
\end{align}$$

$$\begin{align}
k(x,y,u,v) = k_1(x,u).k_2(v,y)
\end{align}
$$

So, the 2D-DFT formula can be computed as a sequence of two 1D-DFT transform. That is, each row of the original image is transformed and then each column of the previous result is transformed. This can be visualized as follows and was taken from [here](http://web.cs.wpi.edu/~emmanuel/courses/cs545/S14/slides/lecture10.pdf):

Similarly, we can also apply the same technique to compute the inverse transformation:

For the forward transformation:

$$\begin{align}
F(u, v) = k_f(u,x) \;f(x,y) \; k_f(y,v)^{T*} \; where \; T* = Matrix \; transpose \; and \; conjugate
\end{align}
$$

And for the inverse transformation:

$$\begin{align}
f(x, y) = k_i(x,u) \; F(u,v) \; k_i^{T*}
\end{align}
$$

Where:
* $ k_f $ = **kernel** function of the **forward** transformation

* $ k_i $= **kernel** function of the **inverse** transformation*

* $ k_i = k_f^{-1} $ and $ k_i = k_f^T $ (Since the kernel function is **orthogonal**).

* And $ k_f = k_f^{T}$ (Since it is a **symmetric** function)

* So, $ k_i = k_f^{*T}$

Therefore:
$$\begin{align}
f(x, y) = k_f(x,u)^{*} \; F(u,v) \; k_f
\end{align}
$$

In the next section, the forward DFT will be implemented in python. Finally they will be tested with images of different sizes. And their running time will be computed and visualized.

## Python Implementation

First of all, let's import the necessary python libraries

```python
import numpy as np

import matplotlib.pyplot as plt
%matplotlib inline

#import matplotlib.image as img
import PIL.Image as Image

import math
import cmath

import time

import csv
```

Now let's start with creating common image functions.

```python
def generateBlackAndWhiteSquareImage(imgSize):
    """
    Generates a square-sized black and white image with a given input size.

    Parameters
    ----------
    imgSize : int
        Input number that stores the dimension of the square image to be generated.

    Returns
    -------
    imge : ndarray
        The generated black and white square image.
    """

    #Creating a matrix with a given size where all the stored values are only zeros (for initialization)
    imge = np.zeros([imgSize, imgSize], dtype=int)

    #Starting and ending indices of the white part of the image.
    ind1 = imgSize/4
    ind2 = ind1 + (imgSize/2)

    #Make a part of the image as white (255)
    imge[ind1:ind2, ind1:ind2] = np.ones([imgSize/2, imgSize/2], dtype=int)*255

    #return the resulting image
    return imge

def generateImagesWithResizedWhite(imge):
    """
    Generates images with the same size as the original but with a resized white part of them.
    """

    N = imge.shape[0]

    imges = []
    i = N/2
    while i >= 4:
        j = (N - i)/2

        #Starting and ending indices for the white part.
        indx1 = j
        indx2 = j+i

        #Draw the image.
        imgeNew = np.zeros([N, N],dtype=int)
        imgeNew[indx1:indx2, indx1:indx2] = np.ones([i, i], dtype=int)*255

        #Add the image to the list.
        imges.append(imgeNew)

        i = i/2

    return imges

def resizeImage(imge, newSize):
    """
    Reduces the size of the given image.

    Parameters
    ----------
    imge : ndarray
        Input array that stores the image to be resized.

    Returns
    -------
    newSize : int
        The size of the newly generated image.
    """

    #Compute the size of the original image (in this case, only # of rows as it is square)
    N = imge.shape[0]

    #The ratio of the original image as compared to the new one.
    stepSize = N/newSize

    #Creating a new matrix (image) with a black color (values of zero)
    newImge = np.zeros([N/stepSize, N/stepSize])

    #Average the adjacent four pixel values to compute the new intensity value for the new image.
    for i in xrange(0, N, stepSize):
        for j in xrange(0, N, stepSize):
            newImge[i/stepSize, j/stepSize] = np.mean(imge[i:i+stepSize, j:j+stepSize])

    #Return the new image
    return newImge

```

As a next step, the main class that implements a 2D DFT. Both the forward and inverse DFT will be implemented here.

*Note: All the input images are assumed to be square in size. But the implementation can easily be modified to work with rectangular images (not squares).*

```python
class DFT(object):
    """
    This class DFT implements all the procedures for transforming a given 2D digital image
    into its corresponding frequency-domain image (Forward DFT Transform)
    """

    @classmethod
    def __computeConjugate(self, mat):
        """
        Computes the conjugate of a complex square-matrix.

        Parameters
        ----------
        mat : ndarray
            Input matrix of complex numbers.

        Returns
        -------
        result : ndarray
            The conjugate of the input matrix.
        """

        N = mat.shape[0]
        result = np.zeros([N, N], dtype=np.complex)

        for i in range(N):
            for j in range(N):
                result[i, j] = (mat[i, j].real) - (mat[i, j].imag*1j)

        return result

    @classmethod
    def __multiplyMatrices(self, mat1, mat2):
        """
        Computes the multiplication of two complex square matrices.

        Parameters
        ----------
        mat1 : ndarray
            First input matrix of complex numbers.
        mat2 : ndarray
            Second input matrix of complex numbers.

        Returns
        -------
        result : ndarray
            The multiplication result of the two matrices.
        """

        N = mat1.shape[0]

        result = np.zeros([N, N], np.complex)
        #For each column and row...
        for i in range(N):
            row = mat1[i, :]
            for j in range(N):
                col = mat2[j, :]
                total = 0 + 0j
                for k in range(N):
                    total += row[k]*col[k]
                result[i, j] = total

        return result

    #Compute the two separable kernels for the forward DFT.
    @classmethod
    def computeXForwardKernel(self, size):
        """
        Computes/generates the first forward kernel function.

        Parameters
        ----------
        size : int
            Size of the kernel to be generated.

        Returns
        -------
        xKernel : ndarray
            The generated kernel as a matrix.
        """

        #Initialize the kernel
        xKernel = np.zeros([size, size], dtype=np.complex)

        #Compute each value of the kernel...
        for u in range(size):
            for x in range(size):

                #Rounding it is used here for making the values integers as it will insert very small fractions.
                xKernel[u, x] = math.cos((2*math.pi*u*x)/size) - (1j*math.sin((2*math.pi*u*x)/size))

        #Return the resulting kernel
        return xKernel

    @classmethod
    def computeYForwardKernel(self, xKernel):
        """
        Computes/generates the second forward kernel function.

        Parameters
        ----------
        xKernel : ndarray
            The first forward kernel function.

        Returns
        -------
        yKernel : ndarray
            The generated kernel as a matrix.
        """
        #yKernel = np.conj(xKernel) ## In numpy package.
        N = xKernel.shape[0]

        #For each value, find the conjugate...
        yKernel = np.zeros([N, N], dtype=np.complex)
        for i in range(N):
            for j in range(N):
                yKernel[i, j] = (xKernel[i, j].real) - (xKernel[i, j].imag*1j)

        # Return the resulting kernel (Since the original kernel is symmetric, transpose is not needed)
        return yKernel

    @classmethod
    def computeCenteredImage(self, imge):
        """
        Centers a given image.

        Parameters
        ----------
        imge : ndarray
            Input array that stores the image to be centered.

        Returns
        -------
        newImge : int
            The new and centered version of the input image.
        """

        #Compute the dimensions of the image
        M, N = imge.shape
        #centeringMatrix = np.zeros([M, N], dtype=int)
        newImge = np.zeros([M, N], dtype=int)
        for x in range(M):
            for y in range(N):
                newImge[x, y] = imge[x, y] * ((-1)**(x+y))

        #newImge = imge * centeringMatrix
        return newImge

    @classmethod
    def computeForward2DDFTWithSeparability(self, imge):
        """
        Computes/generates the 2D DFT by computing the two forward kernels first (Separability).

        Parameters
        ----------
        imge : ndarray
            The input image to be transformed.

        Returns
        -------
        final2DDFT : ndarray
            The transformed image.
        """

        N = imge.shape[0]
        xKernel = DFT.computeXForwardKernel(N)
        yKernel = DFT.computeYForwardKernel(xKernel)

        #row1DDFT = (1.0/size) * np.dot(xKernel, imge)
        intermediate2DDFT = (1.0/N) * DFT.__multiplyMatrices(xKernel, imge)
        final2DDFT = (1.0/N) * DFT.__multiplyMatrices(intermediate2DDFT, yKernel)

        return final2DDFT

    @classmethod
    def __computeSinglePoint2DFT(self, imge, u, v, N):
        """
        A private method that computes a single value of the 2DDFT from a given image.

        Parameters
        ----------
        imge : ndarray
            The input image.

        u : ndarray
            The index in x-dimension.

        v : ndarray
            The index in y-dimension.

        N : int
            Size of the image.

        Returns
        -------
        result : complex number
            The computed single value of the DFT.
        """
        result = 0 + 0j
        for x in xrange(N):
            for y in xrange(N):
                result += (imge[x, y] * (math.cos((2*math.pi*(u*x + v*y))/N) -
                                         (1j*math.sin((2*math.pi*(u*x + v*y))/N))))
        return result

    @classmethod
    def computeForward2DDFTNoSeparability(self, imge):
        """
        Computes/generates the 2D DFT by computing without separating the kernels.

        Parameters
        ----------
        imge : ndarray
            The input image to be transformed.

        Returns
        -------
        final2DDFT : ndarray
            The transformed image.
        """

        # Assuming a square image
        N = imge.shape[0]
        final2DDFT = np.zeros([N, N], dtype=np.complex)
        for u in xrange(N):
            for v in xrange(N):
                #Compute the DFT value for each cells/points in the resulting transformed image.
                final2DDFT[u, v] = DFT.__computeSinglePoint2DFT(imge, u, v, N)
        return ((1.0/(N**2))*final2DDFT)

    @classmethod
    def computeInverse2DDFTWithSeparability(self, dftImge):
        """
        Computes the inverse 2D DFT by computing the two inverse kernels first (Separability).

        Parameters
        ----------
        dftImge : ndarray
            The dft transformed image as input.

        Returns
        -------
        imge : ndarray
            The resulting image in spatial domain from the inverse DFT.
        """

        N = dftImge.shape[0]

        #Here the kernels are interchanged from the forward DFT
        yKernel = DFT.computeXForwardKernel(N)
        xKernel = DFT.computeYForwardKernel(yKernel)

        intermediateImge = DFT.__multiplyMatrices(xKernel, dftImge)
        imge = DFT.__multiplyMatrices(intermediateImge, yKernel)

        #imge = np.real(imge)

        return imge

    @classmethod
    def compute2DDFTFourierSpectrum(self, dftImge):
        """
        Computes the fourier spectrum of the transformed image.

        Parameters
        ----------
        dftImge : ndarray
            The input transformed image.

        Returns
        -------
        fourierSpect : ndarray
            The computed fourier spectrum.
        """
        N = dftImge.shape[0]

        fourierSpect = np.zeros([N, N], dtype=float)
        #Calculate the magnitude of each point(complex number) in the DFT image
        for i in xrange(N):
            for j in xrange(N):
                v = dftImge[i, j]
                fourierSpect[i, j] = math.sqrt((v.real)**2 + (v.imag)**2)
        return fourierSpect

    @classmethod
    def normalize2DDFTByLog(self, dftImge):
        """
        Computes the log transformation of the transformed DFT image to make the range
        of the fourier values b/n 0 to 255

        Parameters
        ----------
        dftImge : ndarray
            The input transformed image.

        Returns
        -------
        dftNormImge : ndarray
            The normalized version of the transformed image.
        """

        #Compute the fourier spectrum of the transformed image:
        dftFourierSpect = DFT.compute2DDFTFourierSpectrum(dftImge)

        #Normalize the fourier spectrum values:
        dftNormFourierSpect = (255.0/ math.log10(255)) * np.log10(1 + (255.0/(np.max(dftFourierSpect))*dftFourierSpect))

        return dftNormFourierSpect

```

## Testing the Code and Visualizing the DFT Running Time

### Testing the DFT algorithm

For testing purposes, the 4x4 separable DFT kernels are computed:

```python
# The numbers are rounded for visualization
xKernel = np.round(DFT.computeXForwardKernel(4))
print "The first 4x4 forward kernel:"
xKernel
```

    The first 4x4 forward kernel:

    array([[ 1.+0.j,  1.+0.j,  1.+0.j,  1.+0.j],
           [ 1.+0.j,  0.-1.j, -1.-0.j, -0.+1.j],
           [ 1.+0.j, -1.-0.j,  1.+0.j, -1.-0.j],
           [ 1.+0.j, -0.+1.j, -1.-0.j,  0.-1.j]])

```python
# The second kernel is the conjugate of the first (as the kernels are symmetric, we don't need to transpose)
yKernel = np.round(DFT.computeYForwardKernel(xKernel))
print "The first 4x4 forward kernel:"
yKernel
```

    The first 4x4 forward kernel:

    array([[ 1.+0.j,  1.+0.j,  1.+0.j,  1.+0.j],
           [ 1.+0.j,  0.+1.j, -1.+0.j, -0.-1.j],
           [ 1.+0.j, -1.+0.j,  1.+0.j, -1.+0.j],
           [ 1.+0.j, -0.-1.j, -1.+0.j,  0.+1.j]])

Here, we generate an 8-bit gray scale image as a 64x64 matrix

```python
imge = generateBlackAndWhiteSquareImage(64)
```

Generate images of the same size as above but with different white part size:

```python
imges = generateImagesWithResizedWhite(imge)
```

To test the DFT with different images having different white size:

Here, we will generate the images, compute the DFT and visualize the results:

```python
#For visualization:
N = len(imges)
fig, axarr = plt.subplots(N, 4, figsize=(13, 13))

#Compute DFT for each generated image...
dftImges = []
for i, imge in enumerate(imges):

    #Center the generated image
    centeredImge = DFT.computeCenteredImage(imge)

    #Compute the 2D DFT transformation for both centered and uncentered images:
    dftUncenteredImge = DFT.computeForward2DDFTWithSeparability(imge)
    dftCenteredImge = DFT.computeForward2DDFTWithSeparability(centeredImge)

    #Save the centered DFT images...
    dftImges.append(dftCenteredImge)

    #Normalize the computed DFT results:
    dftUncenteredNormImge = DFT.normalize2DDFTByLog(dftUncenteredImge)
    dftCenteredNormImge = DFT.normalize2DDFTByLog(dftCenteredImge)

    #Display the normalized versions of the centered and uncentered images
    axarr[i][0].imshow(imge, cmap=plt.get_cmap('gray'), vmin=0, vmax=255)
    axarr[i][0].set_title('Original Image')

    axarr[i][1].imshow(dftUncenteredNormImge, cmap=plt.get_cmap('gray'), vmin=0, vmax=255)
    axarr[i][1].set_title('Normalized Uncentered DFT')

    axarr[i][2].imshow(DFT.compute2DDFTFourierSpectrum(dftCenteredImge), cmap=plt.get_cmap('gray'))
    axarr[i][2].set_title('Unnormalized Centered DFT')

    axarr[i][3].imshow(dftCenteredNormImge, cmap=plt.get_cmap('gray'), vmin=0, vmax=255)
    axarr[i][3].set_title('Normalized Centered DFT')

#fig.suptitle("DFT FOR 64x64 IMAGES HAVING DIFFERENT WHITE COLOR SIZE")
#fig.subplots_adjust(top=2)
plt.show()
```

From the above results, we can see that the white color size in the original and transformed images are inversely proportional. This is similar to $\delta x \; and \; \frac{1}{\delta x} $ which are inversely proportional to one another.

Now let's compute the inverse DFT on the transformed images to check the correctness of our code:

```python
#For visualization:
N = len(dftImges)
fig, axarr = plt.subplots(2, 2, figsize=(10,7))

#Compute the inverse DFT for only the first two transformed images...
for i, dftImge in enumerate(dftImges[:2]):

    #Compute the inverse DFT and take the real part
    imge = np.real(DFT.computeInverse2DDFTWithSeparability(dftImge))

    #Due to the floating point precision, we can get very small decimal points,
    #So let's round them to the nearest integer.
    imge = np.round(imge)

    #Since the images were originally centered, let's decenter them now
    imge = DFT.computeCenteredImage(imge)

    #Display the dft and the resulting images found with inverse DFT:
    dftNormImge = DFT.normalize2DDFTByLog(dftImge)
    axarr[i][0].imshow(dftNormImge, cmap=plt.get_cmap('gray'))
    axarr[i][0].set_title('Centered DFT Image')

    axarr[i][1].imshow(imge, cmap=plt.get_cmap('gray'))
    axarr[i][1].set_title('Original Image')

#fig.suptitle("The original 64x64 images found by applying inverse DFT", fontsize=14)
#fig.subplots_adjust(top=1.55)

plt.show()
```

### Computing and Visualizing the DFT Running Time

In this part, we will compute and visualize the running time of DFT for different image sizes.

First, the images with different sizes are generated:

```python
def generateImages(imgSizes=[128, 64, 32, 16, 8]):

    #Create an empty list of images to save the generated images with different sizes.
    images = []

    #Generate the first and biggest image
    imge = generateBlackAndWhiteSquareImage(imgSizes[0])

    #Add to the images list
    images.append(imge)

    #Generate the resized and smaller images with different sizes.
    for i in range(1, len(imgSizes)):
        size = imgSizes[i]
        images.append(resizeImage(imge, size))

    return images
```

Next, the DFT algorithm will be run for all the generated images with different sizes. In addition, the running time will also be saved.

```python
#Generate images
imgSizes = [128, 64, 32, 16, 8]
images = generateImages(imgSizes)

# A list that stores the running time of the DFT algorithm for images with different size.
runningTimeDFT = []

#For each image...
for i, imge in enumerate(images):

    #Compute the image size
    N = imge.shape[0]

    print "Computing for ", N, "x", N, "image..."

    #Step 1: Center the image
    centeredImge = DFT.computeCenteredImage(imge)

    #Save the starting time.
    startTime = time.time()

    #Step 2: Compute the DFT of the image using the matrix multiplication form.
    dftImge = DFT.computeForward2DDFTNoSeparability(centeredImge)

    #Save the running time
    runningTimeDFT.append((time.time() - startTime)/60.00)
```

    Computing for  128 x 128 image...
    Computing for  64 x 64 image...
    Computing for  32 x 32 image...
    Computing for  16 x 16 image...
    Computing for  8 x 8 image...

Save the running time to file:

```python
result = zip(imgSizes, runningTimeDFT)
np.savetxt("RunningTimes/runningTimeDFT.csv", np.array(result), delimiter=',')
```

```python
#Plot the running times
plt.plot(xrange(len(runningTimeDFT)), runningTimeDFT, '-d')

xlabels = [str(imge.shape[0]) + 'x' + str(imge.shape[0]) for imge in images]
plt.xticks(xrange(len(runningTimeDFT)), xlabels)
plt.xlabel("Image Size(Pixels)")
plt.ylabel("Time(Minutes)")
plt.show()
```

## Conclusion

In this post, we have implemented Discrete Fourier Transform (forward and reverse) from scratch. Then, we applied it to 2D images.


```python
#derivative (slope at given point)

from sympy import *
import numpy as np
```

```python
x,y,z = sym.symbols('x y z')
```

```python
# single variable ftns
sym.diff(x**3)
```

$\displaystyle 3 x^{2}$

```python
# product rule (use cos from sympy lib)
sym.diff((x**3 + 18) * sym.cos(x))
```

$\displaystyle 3 x^{2} \cos{\left(x \right)} - \left(x^{3} + 18\right) \sin{\left(x \right)}$

```python
#chain rule (der = der of an outer ftn * inner ftn, then multipled by der of inner ftn.)
sym.diff((x**2 - 3*x +5)**3)
```

$\displaystyle \left(6 x - 9\right) \left(x^{2} - 3 x + 5\right)^{2}$

```python
# multi-variable partial derivative for x
sym.diff(x**3 * y, x)
```

$\displaystyle 3 x^{2} y$

```python
# multi-variable partial derivative for y
sym.diff(x**3 * y, y)
```

$\displaystyle x^{3}$

```python
# set ftn to variable
f = x**3 * y * z**2
```

```python
sym.diff(f, z)
```

$\displaystyle 2 x^{3} y z$

```python
# der of sigmoid ftn
sym.diff(1/1 + sym.exp(-x))
```

$\displaystyle - e^{- x}$

```python
sym.integrate(sym.exp(-x), (x, 0, sym.oo))
```

$\displaystyle 1$

```python
integrate(exp(-x), (x,0, oo))
```

$\displaystyle 1$

```python

#sigmoid for binary classification
import matplotlib.pyplot as plt
import numpy as np

def sigmoid(x):
    s=1/(1+np.exp(-x))
    ds=s*(1-s)
    return s,ds
x=np.arange(-6,6,0.01)
sigmoid(x)
# Setup centered axes
fig, ax = plt.subplots(figsize=(9, 5))
ax.spines['left'].set_position('center')
ax.spines['right'].set_color('none')
ax.spines['top'].set_color('none')
ax.xaxis.set_ticks_position('bottom')
ax.yaxis.set_ticks_position('left')
# Create and show plot
ax.plot(x,sigmoid(x)[0], color="#307EC7", linewidth=3, label="sigmoid")
ax.plot(x,sigmoid(x)[1], color="#9621E2", linewidth=1, label="derivative")
ax.legend(loc="upper right", frameon=False)
fig.show()
```

```python
#> sigmoid, easier optmization, values -1 to 1, centers to zero?, suffers from vanishing gradient problem

import matplotlib.pyplot as plt
import numpy as np

def tanh(x):
    t=(np.exp(x)-np.exp(-x))/(np.exp(x)+np.exp(-x))
    dt=1-t**2
    return t,dt
z=np.arange(-4,4,0.01)
tanh(z)[0].size,tanh(z)[1].size
# Setup centered axes
fig, ax = plt.subplots(figsize=(9, 5))
ax.spines['left'].set_position('center')
ax.spines['bottom'].set_position('center')
ax.spines['right'].set_color('none')
ax.spines['top'].set_color('none')
ax.xaxis.set_ticks_position('bottom')
ax.yaxis.set_ticks_position('left')
# Create and show plot
ax.plot(z,tanh(z)[0], color="#307EC7", linewidth=3, label="tanh")
ax.plot(z,tanh(z)[1], color="#9621E2", linewidth=1, label="derivative")
ax.legend(loc="upper right", frameon=False)
fig.show()
```

```python
# ReLu, less comp espensive than tanh and sigmoid, avoid vanishing gradient, should only be used in hidden layers of NN model, can result in dead neurons.
```


Suppose $x_1, ... x_n \sim IG(\mu, \lambda)$ with density:

$$
p(x) = \bigg(\frac{\lambda}{2 \pi x^3} \bigg)^{1/2} \exp \bigg[- \frac{\lambda}{2 \mu^2} \frac{(x - \mu)^2}{x} \bigg]
$$

then, ignoring irrelevant terms the log-likelihood, is given by:

$$
log L(\mu, \lambda) = \frac{n}{2} \log \lambda - \frac{\lambda}{2 \mu^2} \sum_{i=1}^{n} \frac{(x_i - \mu)^2}{x_i}
$$

(a) The score equations for the MLE are given by:

$$
\begin{align}
\frac{\partial}{\partial \mu} \log L(\mu, \lambda) & = \frac{\lambda}{\mu^3} \sum_{i=1}^{n} \frac{(x_i - \mu)^2}{x_i} + \frac{\lambda} {\mu^2} \sum_{i=1}^{n} \frac{(x_i - \mu)}{x_i} \\
& = \frac{\lambda}{\mu^3} \sum_{i=1}^{n} \frac{(x_i - \mu)^2}{x_i} \bigg[1 + \frac{\mu}{(x_i - \mu)} \bigg] \\
& = \frac{\lambda}{\mu^3} \bigg[ \sum_{i=1}^{n} x_i - n \mu \bigg] \\
& = \frac{n \lambda}{\mu^3}[\bar{x} - \mu] = 0
\end{align}
$$
and
$$
\frac{\partial}{\partial \lambda} \log L(\mu, \lambda) = \frac{n}{2 \lambda} - \frac{1}{2 \mu^2} \sum_{i=1}^{n} \frac{(x_i - \mu)^2}{x_i} = 0
$$

The MLEs are the solutions to the above equations and are given by:

$$
\hat{\mu} = \bar{x}
$$

and

$$
\begin{align}
0 & = \frac{n}{2 \hat{\lambda}} - \frac{1}{2 \bar{x}^2} \sum_{i=1}^n \frac{(x_i - \bar{x})^2}{x_i} \\
& =  \frac{n}{2 \hat{\lambda}} - \frac{1}{2 \bar{x}^2} \sum_{i=1}^n \bigg( x_i - 2 \bar{x} + \frac{\bar{x}^2}{x_i} \bigg) \\
& =  \frac{n}{2 \hat{\lambda}} - \frac{1}{2 \bar{x}^2} \bigg( n \bar{x} - 2n \bar{x} + \bar{x}^2 \sum_{i=1}^n \frac{1}{x_i} \bigg) \\
& = \frac{n}{2 \hat{\lambda}} - \frac{n}{2} \bigg( \frac{1}{n} \sum_{i=1}^n \frac{1}{x_i} - \frac{1}{\bar{x}} \bigg) \\
& = \frac{n}{2} \bigg[ \frac{1}{\hat{\lambda}} - \bigg( \frac{1}{n} \sum_{i=1}^n \frac{1}{x_i} - \frac{1}{\bar{x}} \bigg) \bigg]
\end{align}
$$

So:
$$
\hat{\lambda} = \bigg( \frac{1}{\tilde{x}} - \frac{1}{\bar{x}} \bigg)^{-1}
$$

where the harmonic sample mean is given by:

$$
\tilde{x} = \frac{n}{\sum_{i=1}^n \frac{1}{x_i}}
$$

(b) The off-diagonal elements of the Fisher information matrix are given by:

$$
- \frac{\partial^2}{\partial \lambda \partial \mu} \log L(\mu, \lambda) \bigg|_{\mu = \hat{\mu}, \lambda = \hat{\lambda}} = \frac{\partial^2}{ \partial \mu \partial \lambda} \log L(\mu, \lambda) \bigg|_{\mu = \hat{\mu}, \lambda = \hat{\lambda}} = 0
$$

The diagonal elements are given by:

$$
- \frac{\partial^2}{\partial \mu^2} \log L(\mu, \lambda) \bigg|_{\mu = \hat{\mu}, \lambda = \hat{\lambda}} = \frac{n \hat{\lambda}}{\bar{x}^3}
$$

and

$$
- \frac{\partial^2}{\partial \lambda^2} \log L(\mu, \lambda) \bigg|_{\mu = \hat{\mu}, \lambda = \hat{\lambda}} = \frac{n}{\hat{\lambda}^2}
$$


# Modeling and Simulation in Python

Chapter 9

Copyright 2017 Allen Downey

License: [Creative Commons Attribution 4.0 International](https://creativecommons.org/licenses/by/4.0)

```python
# Configure Jupyter to display the assigned value after an assignment
%config InteractiveShell.ast_node_interactivity='last_expr_or_assign'

# import everything from SymPy.
from sympy import *

# Set up Jupyter notebook to display math.
init_printing()
```

The following displays SymPy expressions and provides the option of showing results in LaTeX format.

```python
from sympy.printing import latex

def show(expr, show_latex=False):
    """Display a SymPy expression.

    expr: SymPy expression
    show_latex: boolean
    """
    if show_latex:
        print(latex(expr))
    return expr
```

### Analysis with SymPy

Create a symbol for time.

```python
t = symbols('t')
```

If you combine symbols and numbers, you get symbolic expressions.

```python
expr = t + 1
```

The result is an `Add` object, which just represents the sum without trying to compute it.

```python
type(expr)
```

`subs` can be used to replace a symbol with a number, which allows the addition to proceed.

```python
expr.subs(t, 2)
```

`f` is a special class of symbol that represents a function.

```python
f = Function('f')
```

The type of `f` is `UndefinedFunction`

```python
type(f)
```

SymPy understands that `f(t)` means `f` evaluated at `t`, but it doesn't try to evaluate it yet.

```python
f(t)
```

`diff` returns a `Derivative` object that represents the time derivative of `f`

```python
dfdt = diff(f(t), t)
```

```python
type(dfdt)
```

We need a symbol for `alpha`

```python
alpha = symbols('alpha')
```

Now we can write the differential equation for proportional growth.

```python
eq1 = Eq(dfdt, alpha*f(t))
```

And use `dsolve` to solve it.  The result is the general solution.

```python
solution_eq = dsolve(eq1)
```

We can tell it's a general solution because it contains an unspecified constant, `C1`.

In this example, finding the particular solution is easy: we just replace `C1` with `p_0`

```python
C1, p_0 = symbols('C1 p_0')
```

```python
particular = solution_eq.subs(C1, p_0)
```

In the next example, we have to work a little harder to find the particular solution.

### Solving the quadratic growth equation

We'll use the (r, K) parameterization, so we'll need two more symbols:

```python
r, K = symbols('r K')
```

Now we can write the differential equation.

```python
eq2 = Eq(diff(f(t), t), r * f(t) * (1 - f(t)/K))
```

And solve it.

```python
solution_eq = dsolve(eq2)
```

The result, `solution_eq`, contains `rhs`, which is the right-hand side of the solution.

```python
general = solution_eq.rhs
```

We can evaluate the right-hand side at $t=0$

```python
at_0 = general.subs(t, 0)
```

Now we want to find the value of `C1` that makes `f(0) = p_0`.

So we'll create the equation `at_0 = p_0` and solve for `C1`.  Because this is just an algebraic identity, not a differential equation, we use `solve`, not `dsolve`.

The result from `solve` is a list of solutions.  In this case, [we have reason to expect only one solution](https://en.wikipedia.org/wiki/Picard%E2%80%93Lindel%C3%B6f_theorem), but we still get a list, so we have to use the bracket operator, `[0]`, to select the first one.

```python
solutions = solve(Eq(at_0, p_0), C1)
type(solutions), len(solutions)
```

```python
value_of_C1 = solutions[0]
```

Now in the general solution, we want to replace `C1` with the value of `C1` we just figured out.

```python
particular = general.subs(C1, value_of_C1)
```

The result is complicated, but SymPy provides a method that tries to simplify it.

```python
particular = simplify(particular)
```

Often simplicity is in the eye of the beholder, but that's about as simple as this expression gets.

Just to double-check, we can evaluate it at `t=0` and confirm that we get `p_0`

```python
particular.subs(t, 0)
```

This solution is called the [logistic function](https://en.wikipedia.org/wiki/Population_growth#Logistic_equation).

In some places you'll see it written in a different form:

$f(t) = \frac{K}{1 + A e^{-rt}}$

where $A = (K - p_0) / p_0$.

We can use SymPy to confirm that these two forms are equivalent.  First we represent the alternative version of the logistic function:

```python
A = (K - p_0) / p_0
```

```python
logistic = K / (1 + A * exp(-r*t))
```

To see whether two expressions are equivalent, we can check whether their difference simplifies to 0.

```python
simplify(particular - logistic)
```

This test only works one way: if SymPy says the difference reduces to 0, the expressions are definitely equivalent (and not just numerically close).

But if SymPy can't find a way to simplify the result to 0, that doesn't necessarily mean there isn't one.  Testing whether two expressions are equivalent is a surprisingly hard problem; in fact, there is no algorithm that can solve it in general.

### Exercises

**Exercise:** Solve the quadratic growth equation using the alternative parameterization

$\frac{df(t)}{dt} = \alpha f(t) + \beta f^2(t) $

```python
# Solution goes here
```

```python
# Solution goes here
```

```python
# Solution goes here
```

```python
# Solution goes here
```

```python
# Solution goes here
```

```python
# Solution goes here
```

```python
# Solution goes here
```

**Exercise:**  Use [WolframAlpha](https://www.wolframalpha.com/) to solve the quadratic growth model, using either or both forms of parameterization:

    df(t) / dt = alpha f(t) + beta f(t)^2

or

    df(t) / dt = r f(t) (1 - f(t)/K)

Find the general solution and also the particular solution where `f(0) = p_0`.

```python

```


Taylor Series Expansion with Python from Data Science Fabric

https://dsfabric.org/taylor-series-expansion-with-python

# LAB 9 Taylor Series

**Universidad Naional de Colombia - Sede Bogotá**

**Metodos Numericos**

**Docente:** _German Hernandez_

**Estudiante:**
  * Luis Miguel Báez Aponte - lmbaeza@unal.edu.co

```python

from sympy import series, Symbol
from sympy.functions import sin, cos, exp, log
from sympy.plotting import plot
import numpy as np
import matplotlib.pyplot as plt
```

```python
# Define symbol
x = Symbol('x')
```

```python

# Function for Taylor Series Expansion

def taylor(function, x0, n):
    """
    Parameter "function" is our function which we want to approximate
    "x0" is the point where to approximate
    "n" is the order of approximation
    """
    return function.series(x,x0,n).removeO()
```

```python
print('sin(x) ≅', taylor(sin(x), 0, 4))

print('ln(x+1) ≅', taylor(log(x+1), 0, 4))

```

    sin(x) ≅ -x**3/6 + x
    ln(x+1) ≅ x**3/3 - x**2/2 + x

```python
print('sin(1) =', taylor(sin(x), 0, 4).subs(x,1))

print('ln(1+1) =', taylor(log(x+1), 0, 4).subs(x,1))

```

    sin(1) = 5/6
    ln(1+1) = 5/6

```python
print('Taylor 0 sin(x) ≅', taylor(sin(x), 0, 0))
print('Taylor 1 sin(x) ≅', taylor(sin(x), 0, 1))
print('Taylor 2 sin(x) ≅', taylor(sin(x), 0, 2))
print('Taylor 3 sin(x) ≅', taylor(sin(x), 0, 3))
print('Taylor 4 sin(x) ≅', taylor(sin(x), 0, 4))
print('Taylor 5 sin(x) ≅', taylor(sin(x), 0, 5))
print('Taylor 6 sin(x) ≅', taylor(sin(x), 0, 6))
print('Taylor 7 sin(x) ≅', taylor(sin(x), 0, 7))
print('Taylor 8 sin(x) ≅', taylor(sin(x), 0, 8))
```

    Taylor 0 sin(x) ≅ 0
    Taylor 1 sin(x) ≅ 0
    Taylor 2 sin(x) ≅ x
    Taylor 3 sin(x) ≅ x
    Taylor 4 sin(x) ≅ -x**3/6 + x
    Taylor 5 sin(x) ≅ -x**3/6 + x
    Taylor 6 sin(x) ≅ x**5/120 - x**3/6 + x
    Taylor 7 sin(x) ≅ x**5/120 - x**3/6 + x
    Taylor 8 sin(x) ≅ -x**7/5040 + x**5/120 - x**3/6 + x

```python
print('Taylor 0 ln(x+1) ≅', taylor(log(x+1), 0, 0).subs(x,2),' = ',taylor(log(x+1), 0, 0).subs(x,2).evalf())
print('Taylor 1 ln(x+1) ≅', taylor(log(x+1), 0, 1).subs(x,2),' = ',taylor(log(x+1), 0, 1).subs(x,2).evalf())
print('Taylor 2 ln(x+1) ≅', taylor(log(x+1), 0, 2).subs(x,2),' = ',taylor(log(x+1), 0, 2).subs(x,2).evalf())
print('Taylor 3 ln(x+1) ≅', taylor(log(x+1), 0, 3).subs(x,2),' = ',taylor(log(x+1), 0, 3).subs(x,2).evalf())
print('Taylor 4 ln(x+1) ≅', taylor(log(x+1), 0, 4).subs(x,2),' = ',taylor(log(x+1), 0, 4).subs(x,2).evalf())
print('Taylor 5 ln(x+1) ≅', taylor(log(x+1), 0, 5).subs(x,2),' = ',taylor(log(x+1), 0, 5).subs(x,2).evalf())
print('Taylor 6 ln(x+1) ≅', taylor(log(x+1), 0, 6).subs(x,2),' = ',taylor(log(x+1), 0, 6).subs(x,2).evalf())
print('Taylor 7 ln(x+1) ≅', taylor(log(x+1), 0, 8).subs(x,2),' = ',taylor(log(x+1), 0, 7).subs(x,2).evalf())
```

    Taylor 0 ln(x+1) ≅ 0  =  0
    Taylor 1 ln(x+1) ≅ 0  =  0
    Taylor 2 ln(x+1) ≅ 2  =  2.00000000000000
    Taylor 3 ln(x+1) ≅ 0  =  0
    Taylor 4 ln(x+1) ≅ 8/3  =  2.66666666666667
    Taylor 5 ln(x+1) ≅ -4/3  =  -1.33333333333333
    Taylor 6 ln(x+1) ≅ 76/15  =  5.06666666666667
    Taylor 7 ln(x+1) ≅ 444/35  =  -5.60000000000000

```python
import math
print('sympy sin(x)subs(x,2) =', sin(x).subs(x,2))
print('sympy sin(x).subs(x,2).evalf() =', sin(x).subs(x,2).evalf())
print('math.sin(2) =', math.sin(2))
```

    sympy sin(x)subs(x,2) = sin(2)
    sympy sin(x).subs(x,2).evalf() = 0.909297426825682
    math.sin(2) = 0.9092974268256817

```python
import math
print('sympy ln(x+1)subs(x,2) =', log(x+1).subs(x,2))
print('sympy ln(x+1).subs(x,2).evalf() =', log(x+1).subs(x,2).evalf())
print('math.ln(2+1) =', math.log(2+1))
```

    sympy ln(x+1)subs(x,2) = log(3)
    sympy ln(x+1).subs(x,2).evalf() = 1.09861228866811
    math.ln(2+1) = 1.0986122886681098

```python
import math
import numpy as np
import matplotlib.pyplot as plt
# if using a Jupyter notebook, include:
%matplotlib inline

values = np.arange(-5,5,0.1)
p_exp = np.exp(values)
t_exp1 = [taylor(exp(x), 0, 1).subs(x,v) for v in values]
legends = ['exp() ','Taylor 1 (constant)']

fig, ax = plt.subplots()
ax.plot(values,p_exp, color ='red')
ax.plot(values,t_exp1)

ax.set_ylim([-5,5])
ax.axhline(y=0.0, xmin=-5.0, xmax=5.0, color='black')
ax.axvline(x=0.0, ymin=-10.0, ymax=10.0, color='black')
ax.legend(legends)

plt.show()
```

```python
import math
import numpy as np
import matplotlib.pyplot as plt
# if using a Jupyter notebook, include:
%matplotlib inline

values = np.arange(-5,5,0.1)
p_sin = np.sin(values)
t_sin2 = [taylor(sin(x), 0, 2).subs(x,v) for v in values]
legends = ['sin() ','Taylor 2 (linear)']

fig, ax = plt.subplots()
ax.plot(values,p_sin, color ='red')
ax.plot(values,t_sin2)

ax.set_ylim([-5,5])
ax.axhline(y=0.0, xmin=-5.0, xmax=5.0, color='black')
ax.axvline(x=0.0, ymin=-10.0, ymax=10.0, color='black')
ax.legend(legends)

plt.show()
```

```python
import math
import numpy as np
import matplotlib.pyplot as plt
# if using a Jupyter notebook, include:
%matplotlib inline

values = np.arange(-5,5,0.1)
p_log = np.log(values)
t_log2 = [taylor(log(x+1), 0, 2).subs(x,v) for v in values]
legends = ['ln(x+1) ','Taylor 2 (linear)']

fig, ax = plt.subplots()
ax.plot(values,p_log, color ='red')
ax.plot(values,t_log2)

ax.set_ylim([-5,5])
ax.axhline(y=0.0, xmin=-5.0, xmax=5.0, color='black')
ax.axvline(x=0.0, ymin=-10.0, ymax=10.0, color='black')
ax.legend(legends)

plt.show()
```

```python
import math
import numpy as np
import matplotlib.pyplot as plt
# if using a Jupyter notebook, include:
%matplotlib inline

values = np.arange(-5,5,0.1)
p_sin = np.sin(values)
t_sin3 = [taylor(sin(x), 0, 3).subs(x,v) for v in values]
legends = ['sin(x) ','Taylor 3 (quadratic)']

fig, ax = plt.subplots()
ax.plot(values,p_sin, color ='red')
ax.plot(values,t_sin3)

ax.set_ylim([-5,5])
ax.axhline(y=0.0, xmin=-5.0, xmax=5.0, color='black')
ax.axvline(x=0.0, ymin=-10.0, ymax=10.0, color='black')
ax.legend(legends)

plt.show()
```

```python
import math
import numpy as np
import matplotlib.pyplot as plt
# if using a Jupyter notebook, include:
%matplotlib inline

values = np.arange(-5,5,0.1)
p_log = np.log(values)
t_log3 = [taylor(log(x+1), 0, 3).subs(x,v) for v in values]
legends = ['log(x+1) ','Taylor 3 (quadratic)']

fig, ax = plt.subplots()
ax.plot(values,p_log, color ='red')
ax.plot(values,t_log3)

ax.set_ylim([-5,5])
ax.axhline(y=0.0, xmin=-5.0, xmax=5.0, color='black')
ax.axvline(x=0.0, ymin=-10.0, ymax=10.0, color='black')
ax.legend(legends)

plt.show()
```

```python
import math
import numpy as np
import matplotlib.pyplot as plt
# if using a Jupyter notebook, include:
%matplotlib inline

values = np.arange(-5,5,0.1)
p_sin = np.sin(values)
t_sin4 = [taylor(sin(x), 0, 4).subs(x,v) for v in values]
legends = ['sin(x) ','Taylor 4 (cubic)']

fig, ax = plt.subplots()
ax.plot(values,p_sin, color ='red')
ax.plot(values,t_sin4)

ax.set_ylim([-5,5])
ax.axhline(y=0.0, xmin=-5.0, xmax=5.0, color='black')
ax.axvline(x=0.0, ymin=-10.0, ymax=10.0, color='black')
ax.legend(legends)

plt.show()
```

```python
import math
import numpy as np
import matplotlib.pyplot as plt
# if using a Jupyter notebook, include:
%matplotlib inline

values = np.arange(-5,5,0.1)
p_log = np.log(values)
t_log4 = [taylor(log(x+1), 0, 4).subs(x,v) for v in values]
legends = ['ln(x+1) ','Taylor 4 (cubic)']

fig, ax = plt.subplots()
ax.plot(values,p_log, color ='red')
ax.plot(values,t_log4)

ax.set_ylim([-5,5])
ax.axhline(y=0.0, xmin=-5.0, xmax=5.0, color='black')
ax.axvline(x=0.0, ymin=-10.0, ymax=10.0, color='black')
ax.legend(legends)

plt.show()
```

```python
import math
import numpy as np
import matplotlib.pyplot as plt
# if using a Jupyter notebook, include:
%matplotlib inline

values = np.arange(-5,5,0.1)
p_sin = np.sin(values)
t_sin1 = [taylor(sin(x), 0, 1).subs(x,v) for v in values]
t_sin2 = [taylor(sin(x), 0, 2).subs(x,v) for v in values]
t_sin3 = [taylor(sin(x), 0, 3).subs(x,v) for v in values]
t_sin4 = [taylor(sin(x), 0, 4).subs(x,v) for v in values]
legends = ['sin() ','Taylor 1 (constant)','Taylor 3 (linear)','Taylor 3 (quadratic)','Taylor 4 (cubic)']

fig, ax = plt.subplots()
ax.plot(values,p_sin)
ax.plot(values,t_sin1)
ax.plot(values,t_sin2)
ax.plot(values,t_sin3)
ax.plot(values,t_sin4)

ax.set_ylim([-5,5])
ax.axhline(y=0.0, xmin=-5.0, xmax=5.0, color='black')
ax.axvline(x=0.0, ymin=-10.0, ymax=10.0, color='black')
ax.legend(legends)

plt.show()
```

```python
import math
import numpy as np
import matplotlib.pyplot as plt
# if using a Jupyter notebook, include:
%matplotlib inline

values = np.arange(-5,5,0.1)
p_log = np.log(values)
t_log1 = [taylor(log(x+1), 0, 1).subs(x,v) for v in values]
t_log2 = [taylor(log(x+1), 0, 2).subs(x,v) for v in values]
t_log3 = [taylor(log(x+1), 0, 3).subs(x,v) for v in values]
t_log4 = [taylor(log(x+1), 0, 4).subs(x,v) for v in values]
legends = ['ln(x+1) ','Taylor 1 (constant)','Taylor 3 (linear)','Taylor 3 (quadratic)','Taylor 4 (cubic)']

fig, ax = plt.subplots()
ax.plot(values,p_log)
ax.plot(values,t_log1)
ax.plot(values,t_log2)
ax.plot(values,t_log3)
ax.plot(values,t_log4)

ax.set_ylim([-5,5])
ax.axhline(y=0.0, xmin=-5.0, xmax=5.0, color='black')
ax.axvline(x=0.0, ymin=-10.0, ymax=10.0, color='black')
ax.legend(legends)

plt.show()
```


# EM

EM algorithm applies to a large faimily of estimation problems with latent variables, e.g GMM.

Suppose we have a training set $\{x^{(1)},...,x^{(n)}\}$ with $z$ being the latent variable, by marginal probabilities:

$$p(x;\theta) = \sum_{z}p(x, z; \theta)$$

We wish to fit the parameters $\theta$ by maximizing the log-likelihood of the data：

$$
\begin{equation}
\begin{split}
l(\theta) =& \sum_{i=1}^{n}\log{p(x^{(i)};\theta)} \\
=& \sum_{i=1}^{n}\log\sum_{z^{(i)}}p(x^{(i)}, z^{(i)}; \theta)
\end{split}
\end{equation}
$$

Maximizing $l(\theta)$ directly might be difficult.

Our strategy will be to instead repeatedly construct a lower-bound on $l$ (E-step), and then optimize that lower-bound (M-step).

## Lower Bound

Let $Q$ be a distribution over $z$, then:

$$
\begin{equation}
\begin{split}
\log{p(x;\theta)} =& \log\sum_{z}p(x,z;\theta)\\
=& \log\sum_{z}Q(z)\frac{p(x,z;\theta)}{Q(z)}\\
\ge& \sum_{z}Q(z)\log{\frac{p(x,z;\theta)}{Q(z)}}\quad\mbox{(log is concave)}
\end{split}
\end{equation}
$$

We call this bound the *evidence lower bound(ELBO)* and denote it by:

$$\mbox{ELBO}(x;Q,\theta) = \sum_{z}Q(z)\log\frac{p(x,z;\theta)}{Q(z)}$$

To hold with equality, it is sufficient that:

$$\frac{p(x,z;\theta)}{Q(z)} = c$$

which is equivalent to:

$$Q(z) = p(z|x;\theta)$$

## The EM Algorithm

Taking all instances into account, for any distributions $Q_{1},...,Q_{n}$:

$$
\begin{equation}
\begin{split}
l(\theta) \ge& \sum_{i}\mbox{ELBO}(x^{(i)};Q_{i},\theta)\\
=& \sum_{i}\sum_{z^{(i)}}Q_{i}(z^{(i)})\log\frac{p(x^{(i)},z^{(i)};\theta)}{Q_{i}(z^{(i)})}
\end{split}
\end{equation}
$$

equality holds when $Q_{i}$ equal to the posterior distribution in this setting of $\theta$:

$$\mbox{E-step:}\quad{Q_{i}{(z^{(i)})}} = p(z^{(i)}|x^{(i)};\theta)$$

M-step maximize the lower bound with respect to $\theta$ while keeping $Q_{i}$ fixed.

$$
\theta := \underset{\theta}{\mbox{argmax}}\sum_{i}\mbox{ELBO}(x^{(i)};Q_{i},\theta)
$$

EM algorithm ensures $l(\theta^{(t)}) \le l(\theta^{(t+1)})$, thus ensures convergence.

```python

```


# Monte Carlo experiments
The Monte Carlo method is a way of using random numbers to solve problems that can otherwise be quite complicated. Essentially, the idea is to replace uncertain values with a large list of values, assume those values have no uncertaintly, and compute a large list of results. Analysis of the results can often lead you to the solution to the original problem.

The following story is based on the Dart method, described in the following book by Thijsse, J. M. (2006), Computational Physics, Cambridge University Press, p. 273, ISBN 978-0-521-57588-1

### Approximating $\pi$

Let's use the Monte Carlo method to approximate $\pi$. First, the area of a circle is $A = \pi r^2,$
so that the unit circle ($r=1$) has an area $A = \pi$. Second, we define the square domain in which the unit circle just fits (2x2), and plot both:

```python
import matplotlib.pyplot as plt
import numpy as np
from scipy.stats import uniform

def circle():
    theta = np.linspace(0, 2*np.pi, num=1000)
    x = [np.cos(n) for n in theta]
    y = [np.sin(n) for n in theta]
    return x, y

def square():
    x = [-1,1,1,-1,-1]
    y = [-1,-1,1,1,-1]
    return x, y

plt.plot(*circle(), 'k')
plt.plot(*square(), 'k')
plt.axis('equal')
plt.show()
```

Consider a random point (x,y) inside the square.

**The probability that a random point from a square domain lies inside the biggest circle that fits inside the square is equal to the area of the circle divided by the area of the square.**

Our square is $2\times 2=4$ and our circle has an area of $\pi$, so if we pick many points, statistically, we expect $\pi/4$ of them to fall inside the circle.

### Our first few points

Let's create a function that will give us random points from a uniform distribution inside the square domain, and generate a few random points.

```python
np.random.seed(seed=345)  # remove/change the seed if you want different random numbers

def monte_carlo(n):
    x = uniform.rvs(loc=-1, scale=2, size=n)
    y = uniform.rvs(loc=-1, scale=2, size=n)
    return x, y

mc_x, mc_y = monte_carlo(28)
plt.plot(*circle(), 'k')
plt.plot(*square(), 'k')
plt.scatter(mc_x, mc_y, c='green')
plt.axis('equal')
plt.show()
```

Let's colour the points depending on whether they are within 1 unit of the origin or not:

```python
def dist(x, y):
    return np.sqrt(x**2 + y**2)

def inside(x, y):
    return dist(x, y) < 1

def outside(x, y):
    return dist(x, y) > 1

np.random.seed(seed=345)  # remove the command or change the value if you want different random numbers

x, y = monte_carlo(28)
ins = inside(x, y)
outs = outside(x, y)

plt.plot(*square(), 'k')
plt.plot(*circle(), 'k')
plt.scatter(x[ins], y[ins], c='blue',label=len(x[ins]))
plt.scatter(x[outs], y[outs], c='red', label=len(x[outs]))
plt.axis('equal')
plt.legend()
plt.show()
```

If you left the np.random seed as 345, and generated 28 points, then your output should have 22 blue points (inside the circle) and 6 red points (outside the circle).

So, if 22 out of 28 (around 76%) of the points are inside the circle, we can approximate $\pi$ from this.

$$
\begin{align}
\pi \approx 4(22/28) = 22/7 = 3.142857.
\end{align}
$$

$22/7$ is a common approximation of $\pi$, and this seed value happens to get us to this ratio. If you vary the random seed, however, you will see that this ratio for only a small number of points can vary quite a lot!

To produce a more robust approximation for $\pi$, we will need many more random points.

### 250 points

```python
x, y = monte_carlo(250)
ins = inside(x, y)
outs = outside(x, y)
pi = 4 * len(x[ins])/len(x)

plt.plot(*square(), 'k')
plt.scatter(x[ins], y[ins], c='blue', label=len(x[ins]))
plt.scatter(x[outs], y[outs], c='red', label=len(x[outs]))
plt.axis('equal')
plt.legend()
plt.title('π ≈ {}'.format(pi))
plt.show()
```

### 1000 points

```python
x, y = monte_carlo(1000)
ins = inside(x, y)
outs = outside(x, y)
pi = 4 * len(x[ins])/len(x)

plt.plot(*square(), 'k')
plt.scatter(x[ins], y[ins], c='blue', label=len(x[ins]))
plt.scatter(x[outs], y[outs], c='red', label=len(x[outs]))
plt.axis('equal')
plt.legend()
plt.title('π ≈ {}'.format(pi))
plt.show()
```

## 25000 points

```python
x, y = monte_carlo(25000)
ins = inside(x, y)
outs = outside(x, y)
pi = 4 * len(x[ins])/len(x)

plt.plot(*square(), 'k')
plt.scatter(x[ins], y[ins], c='blue', label=len(x[ins]))
plt.scatter(x[outs], y[outs], c='red', label=len(x[outs]))
plt.axis('equal')
plt.legend()
plt.title('π ≈ {}'.format(pi))
plt.show()
```

With many points, you should see a completely blue circle in an otherwise red square, **and** a decent estimate of $\pi$.

## The mcerp3 package
There is a Python package, `mcerp3`, which handles Monte Carlo calculations automatically. It was originally written by Abraham Lee and has recently been updated by Paul Freeman to support Python3. This package is available on [PyPI](https://pypi.org/project/mcerp3/). If you want to use in this notebook in the cloud, you will have to do an install (which can take a bit of time):

```python
#!pip install mcerp3 # or use conda install -y mcerp3 -c freemapa
!conda install -y mcerp3 -c freemapa
import mcerp3 as mc
from mcerp3.umath import sqrt
```

    Fetching package metadata ...............
    Solving package specifications: .

    Package plan for installation in environment /home/nbuser/anaconda3_420:

    The following NEW packages will be INSTALLED:

        _libgcc_mutex: 0.1-main
        mcerp3:        1.0.3-py_0       freemapa
        readline:      7.0-ha6073c6_4

    The following packages will be UPDATED:

        conda:         4.3.31-py35_0             --> 4.5.11-py35_0
        conda-env:     2.6.0-h36134e3_1          --> 2.6.0-1
        libgcc:        4.8.5-2                   --> 7.2.0-h69d50b8_2
        pycosat:       0.6.1-py35_1              --> 0.6.3-py35h6b6bb97_0

    _libgcc_mutex- 100% |################################| Time: 0:00:00   2.34 MB/s
    conda-env-2.6. 100% |################################| Time: 0:00:00   4.14 MB/s
    libgcc-7.2.0-h 100% |################################| Time: 0:00:00  18.35 MB/s
    readline-7.0-h 100% |################################| Time: 0:00:00  12.00 MB/s
    pycosat-0.6.3- 100% |################################| Time: 0:00:00   5.85 MB/s
    mcerp3-1.0.3-p 100% |################################| Time: 0:00:00   4.87 MB/s
    conda-4.5.11-p 100% |################################| Time: 0:00:00  13.23 MB/s

```python
x = mc.U(-1, 1)
y = mc.U(-1, 1)
ins = sqrt(x**2 + y**2) < 1
print('percentage of points in the circle =', ins)
print('pi ≈', 4 * ins)
```

    percentage of points in the circle = 0.7865
    pi ≈ 3.146

```python

```


# Orthogonal polynomials

```python
import numpy as np
import numpy.linalg as la
import matplotlib.pyplot as pt
```

## Mini-Introduction to `sympy`

```python
import sympy as sym

# Enable "pretty-printing" in IPython
sym.init_printing()
```

Make a new `Symbol` and work with it:

```python
x = sym.Symbol("x")

myexpr = (x**2-3)**2
myexpr
```

```python
myexpr = (x**2-3)**2
myexpr
myexpr.expand()
```

```python
sym.integrate(myexpr, x)
```

```python
sym.integrate(myexpr, (x, -1, 1))
```

## Orthogonal polynomials

Now write a function `inner_product(f, g)`:

```python
def inner_product(f, g):
    return sym.integrate(f*g, (x, -1, 1))
```

Show that it works:

```python
inner_product(1, 1)
```

```python
inner_product(1, x)
```

Next, define a `basis` consisting of a few monomials:

```python
basis = [1, x, x**2, x**3]
#basis = [1, x, x**2, x**3, x**4, x**5]
```

And run Gram-Schmidt on it:

```python
orth_basis = []

for q in basis:
    for prev_q in orth_basis:
        q = q - inner_product(prev_q, q)*prev_q / inner_product(prev_q,prev_q)
    orth_basis.append(q)

legendre_basis = [orth_basis[0],]

#to compute Legendre polynomials need to normalize so that q(1)=1 rather than ||q||=1
for q in orth_basis[1:]:
    q = q / q.subs(x,1)
    legendre_basis.append(q)
```

```python
legendre_basis
```

These are called the *Legendre polynomials*.

--------------------
What do they look like?

```python
mesh = np.linspace(-1, 1, 100)

pt.figure(figsize=(8,8))
for f in legendre_basis:
    f = sym.lambdify(x, f)
    pt.plot(mesh, [f(xi) for xi in mesh])
```

-----
These functions are important enough to be included in `scipy.special` as `eval_legendre`:

```python
import scipy.special as sps

for i in range(10):
    pt.plot(mesh, sps.eval_legendre(i, mesh))
```

What can we find out about the conditioning of the generalized Vandermonde matrix for Legendre polynomials?

```python
#keep
n = 20
xs = np.linspace(-1, 1, n)
V = np.array([
    sps.eval_legendre(i, xs)
    for i in range(n)
]).T

la.cond(V)
```

The Chebyshev basis can similarly be defined by Gram-Schmidt, but now with respect to a different inner-product weight function,
$$w(x) = 1/\sqrt{1-x^2}.$$

```python
w = 1 / sym.sqrt(1-x**2)
def cheb_inner_product(f, g):
    return sym.integrate(w*f*g, (x, -1, 1))

orth_basis = []

for q in basis:
    for prev_q in orth_basis:
        q = q - cheb_inner_product(prev_q, q)*prev_q / cheb_inner_product(prev_q,prev_q)
    orth_basis.append(q)

cheb_basis = [1,]

#to compute Legendre polynomials need to normalize so that q(1)=1 rather than ||q||=1
for q in orth_basis[1:]:
    q = q / q.subs(x,1)
    cheb_basis.append(q)
cheb_basis
```

```python
for i in range(10):
    pt.plot(mesh, np.cos(i*np.arccos(mesh)))
```

Chebyshev polynomials achieve similar good, but imperfect conditioning on a uniform grid (but perfect conditioning on a grid of Chebyshev nodes).

```python
#keep
n = 20
xs = np.linspace(-1, 1, n)
V = np.array([
    np.cos(i*np.arccos(xs))
    for i in range(n)
]).T

la.cond(V)
```

```python

```


# day06: Gradient Descent for Linear Regression

# Objectives

* Learn how to fit weight parameters of Linear Regression to a simple dataset via gradient descent
* Understand impact of step size
* Understand impact of initialization

# Outline
* [Part 1: Loss and Gradient for 1-dim. Linear Regression](#part1)
* [Part 2: Gradient Descent Algorithm in a few lines of Python](#part2)
* [Part 3: Debugging with Trace Plots](#part3)
* [Part 4: Selecting the step size](#part4)
* [Part 5: Selecting the initialization](#part5)
* [Part 6: Using SciPy's built-in routines](#part6)

# Takeaways

* Gradient descent is a simple algorithm that can be implemented in a few lines of Python
* * Practical issues include selecting step size and initialization
* Step size matters a lot
* * Need to select carefully for each problem

* Initialization of the parameters can matter too!

* scipy offers some useful tools for gradient-based optimization
* * scipy's toolbox cannot do scalable "stochastic" methods (requires a modest size dataset, not too big)
* * "L-BFGS-B" method is highly recommended if you have your loss and gradient functions available

```python
import numpy as np
```

```python
# import plotting libraries
import matplotlib
import matplotlib.pyplot as plt

%matplotlib inline
plt.style.use('seaborn') # pretty matplotlib plots

import seaborn as sns
sns.set('notebook', font_scale=1.25, style='whitegrid')
```

# Create simple dataset:   y = 1.234 * x + noise

We will *intentionally* create a toy dataset where we know that a good solution has slope near 1.234.

Naturally, the best slope for the finite dataset of N=100 examples we create won't be exactly 1.234 (because of the noise added plus the fact that our dataset size is limited).

```python
def create_dataset(N=100, slope=1.234, noise_stddev=0.1, random_state=0):
    random_state = np.random.RandomState(int(random_state))

    # input features
    x_N = np.linspace(-2, 2, N)

    # output features
    y_N = slope * x_N + random_state.randn(N) * noise_stddev

    return x_N, y_N
```

```python
x_N, y_N = create_dataset(N=50, noise_stddev=0.3)
```

```python
fig, ax = plt.subplots(nrows=1, ncols=1, figsize=(5,5))
plt.plot(x_N, y_N, 'k.');
plt.xlabel('x');
plt.ylabel('y');
```

# Part 1: Gradient Descent for 1-dim. Linear Regression

## Define model

Consider the *simplest* linear regression model. A single weight parameter $w \in \mathbb{R}$ representing the slope of the prediction line. No bias/intercept.

To make predictions, we just compute the weight multiplied by the input feature
$$
\hat{y}(x) = w \cdot x
$$

## Define loss function

We want to minimize the total *squared error* across all N observed data examples (input features $x_n$, output responses $y_n$)

\begin{align}
    \min_{w \in \mathbb{R}} ~~ &\ell(w)
    \\
    \text{calc_loss}(w) = \ell(w) &= \sum_{n=1}^N (y_n - w x_n)^2
\end{align}

### Exercise 1A: Complete the code below

You should make it match the math expression above.

```python
def calc_loss(w):
    ''' Compute loss for slope-only least-squares linear regression

    Args
    ----
    w : float
        Value of slope parameter

    Returns
    -------
    loss : float
        Sum of squared error loss at provided w value
    '''
    yhat_N = x_N * w
    sum_squared_error = 0.0 # todo compute the sum of squared error between y and yhat
    return sum_squared_error
```

# Define the gradient function

\begin{align}
\text{calc_grad}(w) = \ell'(w) &= \frac{\partial}{\partial w} [ \sum_{n=1}^N (y_n - w x_n)^2]
\\
&= \sum_{n=1}^N 2 (y_n - w x_n) (-x_n)
\\
&= 2 \sum_{n=1}^N (w x_n - y_n) (x_n)
\\
&= 2  w \left( \sum_{n=1}^N x_n^2 \right) - 2 \sum_{n=1}^N y_n x_n
\end{align}

Below, we've implemented the gradient calculation in code for you

```python
def calc_grad(w):
    ''' Compute gradient for slope-only least-squares linear regression

    Args
    ----
    w : float
        Value of slope parameter

    Returns
    -------
    g : float
        Value of derivative of loss function at provided w value
    '''
    g = 2.0 * w * np.sum(np.square(x_N)) - 2.0 * np.sum(x_N * y_N)
    return g
```

## Plot loss evaluated at each w from -3 to 8

We should see a "bowl" shape with one *global* minima, because our optimization problem is "convex"

```python
w_grid = np.linspace(-3, 8, 300) # create array of 300 values between -3 and 8
```

```python
loss_grid = np.asarray([calc_loss(w) for w in w_grid])
plt.plot(w_grid, loss_grid, 'b.-');
plt.xlabel('w');
plt.ylabel('loss(w)');
```

### Discussion 1b: Visually, at what value of $w$ does the loss function have a minima? Is it near where you would expect (hint: look above for the "true" slope value used to generate the data)

### Exercise 1c: Write NumPy code to identify which entry in the w_grid array corresponds to the lowest entry in the loss_grid array

Hint: use np.argmin

```python
# TODO write code here
```

## Sanity check: plot gradient evaluated at each w from -3 to 8

```python
grad_grid = np.asarray([calc_grad(w) for w in w_grid])
plt.plot(w_grid, grad_grid, 'b.-');
plt.xlabel('w');
plt.ylabel('grad(w)');
```

### Discussion 1d: Visually, at what value of $w$ does the gradient function cross zero? Is it the same place as the location of the minimum in the loss above?

TODO interpret the graph above and write your answer here, then discuss with your group

### Exercise 1d: Numerically, at which value of w does grad_grid cross zero?

We might try to estimate numerically where the gradient crosses zero.

We could do this in a few steps:

1) Compute the distance from each gradient in `grad_grid` to 0.0 (we could use just absolute distance)

2) Find the index of `grad_grid` with smallest distance (using `np.argmin`)

3) Plug that index into `w_grid` to get the $w$ value corresponding to that zero-crossing

```python
dist_from_zero_G = np.abs(grad_grid - 0.0)

zero_cross_index = 0 # TODO fix me for step 2 above

print("Zero crossing occurs at w = %.4f" % w_grid[0]) # TODO fix me for step 3 above
```

## Part 2: Gradient Descent (GD) as an algorithm in Python

### Define minimize_via_grad_descent algorithm

Can you understand what each step of this algorithm does?

```python
def minimize_via_grad_descent(calc_loss, calc_grad, init_w=0.0, step_size=0.001, max_iters=100):
    ''' Perform minimization of provided loss function via gradient descent

    Args
    ----
    calc_loss : function
    calc_grad : function
    init_w : float
    step_size : float
    max_iters : positive int

    Return
    ----
    wopt: float
        array of optimized weights that approximately gives the least error
    info_dict : dict
        Contains information about the optimization procedure useful for debugging
        Entries include:
        * trace_loss_list : list of loss values
        * trace_grad_list : list of gradient values
    '''
    w = 1.0 * init_w
    grad = calc_grad(w)

    # Create some lists to track progress over time (for debugging)
    trace_loss_list = []
    trace_w_list = []
    trace_grad_list = []

    for iter_id in range(max_iters):
        if iter_id > 0:
            w = w - step_size * grad

        loss = calc_loss(w)
        grad = calc_grad(w)

        print("  iter %5d/%d | w  % 13.5f | loss % 13.4f | grad % 13.4f" % (
            iter_id, max_iters, w, loss, grad))

        trace_loss_list.append(loss)
        trace_w_list.append(w)
        trace_grad_list.append(grad)

    wopt = w
    info_dict = dict(
        trace_loss_list=trace_loss_list,
        trace_w_list=trace_w_list,
        trace_grad_list=trace_grad_list)

    return wopt, info_dict
```

### Discussion 2a: Which line of the above function does the *parameter update* happen?

Remember, in math, the parameter update of gradient descent is this:
$$
w \gets w - \alpha \nabla_w \ell(w)
$$

where $\alpha > 0$ is the step size.

In words, this math says *move* the parameter $w$ from its current value a *small step* in the "downhill" direction (indicated by gradient).

TODO write down here which line above *you* think it is, then discuss with your group

```python

```

### Try it! Run GD with step_size = 0.001

Running the cell below will have the following effects:

1) one line will be printed for every iteration, indicating the current w value and its associated loss

2) the "optimal" value of w will be stored in the variable named `wopt` returned by this function

3) a dictionary of information useful for debugging will be stored in the `info_dict` returned by this function

```python
wopt, info_dict = minimize_via_grad_descent(calc_loss, calc_grad, step_size=0.001);
```

### Discussion 2b: Does it appear from the *loss* values in trace above that the GD procedure converged?

### Discussion 2c: Does it appear from the *parameter* values in trace above that the GD procedure converged?

### Exercise 2d: What exactly is the gradient of the returned "optimal" value of w?

Use your `calc_grad` function to check the result. What is the gradient of the returned `wopt`?

Does this look totally converged? Can you find a $w$ value that would be even better?

```python
# TODO call calc_grad on the return value from above
```

## Part 3: Diagnostic plots for gradient descent

Let's look at some trace functions.

Whenever you run gradient descent, an *excellent* debugging strategy is the ability to plot the loss, the gradient magnitude, and the parameter of interest at every step of the algorithm.

```python
fig, axes = plt.subplots(nrows=1, ncols=3, sharex=True, sharey=False, figsize=(18,3.6))

axes[0].plot(info_dict['trace_loss_list']);
axes[0].set_title('loss');
axes[1].plot(info_dict['trace_grad_list']);
axes[1].set_title('grad');
axes[2].plot(info_dict['trace_w_list']);
axes[2].set_title('w');

plt.xlim([0, 100]);
```

### Discussion 3a: What value do we expect the *loss* to converge to? Should it always be zero?

### Discussion 3b: What value do we expect the *gradient* to converge to? Should it always be zero?

# Part 4: Larger step sizes

## Try with larger step_size = 0.014

```python
wopt, info_dict = minimize_via_grad_descent(calc_loss, calc_grad, step_size=0.014);
```

```python
fig, axes = plt.subplots(nrows=1, ncols=3, sharex=True, sharey=False, figsize=(12,3))

axes[0].plot(info_dict['trace_loss_list'], '.-');
axes[0].set_title('loss');
axes[1].plot(info_dict['trace_grad_list'], '.-');
axes[1].set_title('grad');
axes[2].plot(info_dict['trace_w_list'], '.-');
axes[2].set_title('w');
```

### Discussion 4a: What happens here? How is this step size different than in Part 3 above?

TODO discuss with your group

## Try with even larger step size 0.1

```python
wopt, info_dict = minimize_via_grad_descent(calc_loss, calc_grad, step_size=0.1, max_iters=25);
```

### Discussion 3b: What happens here with this even larger step size? Is it converging?

### Exercise 3c: What is the largest step size you can get to converge reasonably?

```python
# TODO try some other step sizes here
wopt, info_dict = minimize_via_grad_descent(calc_loss, calc_grad, step_size=0) # TODO fix step_size
```

# Part 5: Sensitivity to initial conditions

### Exercise 5a: Try to call the defined procedure with a different initial condition for $w$. What happens?

You could try $w = 5.0$ or something else.

```python
# TODO try some other initial condition for init_w
wopt2, info_dict2 = minimize_via_grad_descent(calc_loss, calc_grad, init_w=0, step_size=0.001, max_iters=10) # TODO fix step_size
```

### Exercise 5b: Try again with another initial value.

```python
# TODO try some other initial condition for init_w
wopt3, info_dict3 = minimize_via_grad_descent(calc_loss, calc_grad, init_w=0, step_size=0.001, max_iters=10) # TODO fix
```

### Exercise 5c: Make a trace plot

Make a trace plot showing convergence from multiple different starting values for $w$. What do you notice?

```python
# TODO
```

# Part 6: Using scipy's built-in gradient optimization tools

```python
import scipy.optimize
```

Take a look at SciPy's built in minimization toolbox

<https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.minimize.html#scipy.optimize.minimize>

We'll use "L-BFGS", a second-order method that uses the function and its gradient.

This is a "quasi-newton" method, which you can get an intuition for here:

https://en.wikipedia.org/wiki/Newton%27s_method_in_optimization

```python
result = scipy.optimize.minimize(calc_loss, 0.0, jac=calc_grad, method='L-BFGS-B')

# Returns an object with several fields, let's print the result to get an idea
print(result)
```

```python
print(str(result.message))
```

```python
best_w = result.x
print(best_w)
```

```python

```


# Scientific Computing with Python

This notebook by Xiaozhou Li is licensed under a Creative Commons Attribution 4.0 International License.
All code examples are also licensed under the [MIT license](http://opensource.org/licenses/MIT).

```python
# what is this line all about?
%matplotlib inline
import matplotlib.pyplot as plt
```

## Numpy and Scipy
### Introduction
The numpy package (module) is used in almost all numerical computation using Python. It is a package that provide high-performance vector, matrix and higher-dimensional data structures for Python. It is implemented in C and Fortran so when calculations are vectorized (formulated with vectors and matrices), performance is very good.

The SciPy framework builds on top of the low-level NumPy framework for multidimensional arrays, and provides a large number of higher-level scientific algorithms.

### Fitting to polynomial

```python
import numpy as np
```

```python
np.random.seed(12)

x = np.linspace(0, 1, 20)
y = np.cos(x) + 0.3*np.random.rand(20)
p = np.poly1d(np.polyfit(x, y, 16))

t = np.linspace(0, 1, 200)
plt.plot(x, y, 'o', t, p(t), '-')
plt.show()
```

### Fit in a Chebyshev basis

```python
np.random.seed(0)

x = np.linspace(-1, 1, 2000)
y = np.cos(x) + 0.3*np.random.rand(2000)
p = np.polynomial.Chebyshev.fit(x, y, 90)

t = np.linspace(-1, 1, 200)
plt.plot(x, y, 'r.')
plt.plot(t, p(t), 'k-', lw=3)
plt.show()
```

### A demo of 1D interpolation

```python
np.random.seed(0)
measured_time = np.linspace(0, 1, 10)
noise = 1e-1 * (np.random.random(10)*2 - 1)
measures = np.sin(2 * np.pi * measured_time) + noise

# Interpolate it to new time points
from scipy.interpolate import interp1d
linear_interp = interp1d(measured_time, measures)
interpolation_time = np.linspace(0, 1, 50)
linear_results = linear_interp(interpolation_time)
cubic_interp = interp1d(measured_time, measures, kind='cubic')
cubic_results = cubic_interp(interpolation_time)

# Plot the data and the interpolation
from matplotlib import pyplot as plt
plt.figure(figsize=(6, 4))
plt.plot(measured_time, measures, 'o', ms=6, label='measures')
plt.plot(interpolation_time, linear_results, label='linear interp')
plt.plot(interpolation_time, cubic_results, label='cubic interp')
plt.legend()
plt.show()
```

### Minima and roots of a function
\begin{equation}
    f(x) = x^2 + 10\sin(x)
\end{equation}

**(1) find minima**

```python
def f(x):
    return x**2 + 10*np.sin(x)

from scipy import optimize

# Global optimization
grid = (-10, 10, 0.1)
xmin_global = optimize.brute(f, (grid, ))
print("Global minima found %s" % xmin_global)

# Constrain optimization
xmin_local = optimize.fminbound(f, 0, 10)
print("Local minimum found %s" % xmin_local)
```

    Global minima found [-1.30641113]
    Local minimum found 3.8374671194983834

**(2) root finding**

```python
root = optimize.root(f, 1)  # our initial guess is 1
print("First root found %s" % root.x)
root2 = optimize.root(f, -2.5)
print("Second root found %s" % root2.x)
```

    First root found [0.]
    Second root found [-2.47948183]

**(3) Plot function, minima, and roots**

```python
fig = plt.figure(figsize=(6, 4))
ax = fig.add_subplot(111)

x = np.arange(-10, 10, 0.1)
# Plot the function
ax.plot(x, f(x), 'b-', label="f(x)")

# Plot the minima
xmins = np.array([xmin_global[0], xmin_local])
ax.plot(xmins, f(xmins), 'go', label="Minima")

# Plot the roots
roots = np.array([root.x, root2.x])
ax.plot(roots, f(roots), 'kv', label="Roots")

# Decorate the figure
ax.legend(loc='best')
ax.set_xlabel('x')
ax.set_ylabel('f(x)')
ax.axhline(0, color='gray')
plt.show()
```

## Matplotlib
### Introduction
Matplotlib is an excellent 2D and 3D graphics library for generating scientific figures.

### Reading and writing a panda

**(1) original figure**

```python
plt.figure()
img = plt.imread('../data/panda.jpg')
plt.imshow(img)
plt.imsave("original.jpg",img)
```

**(2) red channel displayed in grey**

```python
plt.figure()
img_red = img[:, :, 0]
plt.imshow(img_red, cmap=plt.cm.gray)
```

**(3) lower resolution (compression)**

```python
plt.figure()
img_tiny = img[::8, ::8]
plt.imshow(img_tiny, interpolation='nearest')
#plt.savefig("compressed.jpg")
plt.imsave("compressed.jpg",img_tiny)
```

### Mandlebrot Set (Mandelbrot fractal)

```python
def compute_mandelbrot(N_max, some_threshold, nx, ny):
    # A grid of c-values
    x = np.linspace(-2, 1, nx)
    y = np.linspace(-1.5, 1.5, ny)

    c = x[:,np.newaxis] + 1j*y[np.newaxis,:]

    # Mandelbrot iteration

    z = c
    for j in range(N_max):
        z = z**2 + c

    mandelbrot_set = (abs(z) < some_threshold)

    return mandelbrot_set

mandelbrot_set = compute_mandelbrot(50, 50., 601, 401)

plt.imshow(mandelbrot_set.T, extent=[-2, 1, -1.5, 1.5])
plt.gray()
plt.show()
```

### A simple example of 3D plotting
$$ z = \sin(\sqrt{x^2 + y^2}) $$

```python
from mpl_toolkits.mplot3d import Axes3D

fig = plt.figure()
ax = Axes3D(fig)
X = np.arange(-4, 4, 0.25)
Y = np.arange(-4, 4, 0.25)
X, Y = np.meshgrid(X, Y)
R = np.sqrt(X ** 2 + Y ** 2)
Z = np.sin(R)

ax.plot_surface(X, Y, Z, rstride=1, cstride=1, cmap=plt.cm.hot)
ax.contourf(X, Y, Z, zdir='z', offset=-2, cmap=plt.cm.hot)
ax.set_zlim(-2, 2)

plt.show()
```

### An example displaying the contours of a function
$$ f(x,y) = \left(1 - \frac{x}{2} + x^5 + y^3\right)e^{-x^2-y^2}.$$

```python
def f(x,y):
    return (1 - x / 2 + x**5 + y**3) * np.exp(-x**2 -y**2)

n = 256
x = np.linspace(-3, 3, n)
y = np.linspace(-3, 3, n)
X,Y = np.meshgrid(x, y)

plt.axes([0.025, 0.025, 0.95, 0.95])

plt.contourf(X, Y, f(X, Y), 8, alpha=.75, cmap=plt.cm.hot)
C = plt.contour(X, Y, f(X, Y), 8, colors='black')
plt.clabel(C, inline=1, fontsize=10)

plt.xticks(())
plt.yticks(())
plt.show()
```

## Sympy - Symbolic algebra in Python

### Introduction
There are two notable Computer Algebra Systems (CAS) for Python:

* [SymPy](http://sympy.org/en/index.html) - A python module that can be used in any Python program, or in an IPython session, that provides powerful CAS features.
* [Sage](http://www.sagemath.org/) - Sage is a full-featured and very powerful CAS enviroment that aims to provide an open source system that competes with Mathematica and Maple. Sage is not a regular Python module, but rather a CAS environment that uses Python as its programming language.

Sage is in some aspects more powerful than SymPy, but both offer very comprehensive CAS functionality. The advantage of SymPy is that it is a regular Python module and integrates well with the Jupyter notebook.

```python
from sympy import *

init_printing()
```

### Expand, factor and simplify

```python
x, y = symbols('x y')

(x+1)*(x+2)*(x+3)*(x+4)*(x+5)
```

```python
expand((x+1)*(x+2)*(x+3)*(x+4)*(x+5))
```

```python
sin(x+y)
```

```python
expand(sin(x+y), trig=True)
```

```python
expand((x+y)**8)
```

```python
x**3 + 6 * x**2 + 11*x + 6
```

```python
factor(x**3 + 6 * x**2 + 11*x + 6)
```

```python
sin(x)**2 + cos(x)**2
```

```python
simplify(sin(x)**2 + cos(x)**2)
```

```python
cos(x)/sin(x)
```

```python
simplify(cos(x)/sin(x))
```

### Calculus
**(1) differentiation and integration**

$f(x) = (x+1)^2$

```python
f = (x+1)**2
f
```

Computing $\frac{d f}{dx}$, $\frac{d f^2}{dx}$

```python
diff(f,x)
```

```python
diff(f**2,x)
```

Computing $\frac{d \sin(f)}{dx}$, $\frac{d^2 \sin(f)}{dx^2}$

```python
diff(sin(f),x)
```

```python
diff(sin(f),x,2)
```

```python
diff(sin(f),x,4)
```

$$ f(x,y) = \sin(xy) + \cos(xy),$$
computing
$$ \frac{\partial^3 f}{\partial x \partial y^2},\quad \int f(x,y)\,dx,\quad \int_{-1}^{1}f(x,y)\,dx$$

```python
f = sin(x*y) + cos(y*x)
f
```

```python
diff(f, x, 1, y, 2)
```

```python
integrate(f, x)
```

```python
integrate(f, (x, -1, 1))
```

Computing $\int_{-\infty}^\infty e^{-x^2}\,dx$

```python
integrate(exp(-x**2), (x, -oo, oo))
```

**(2) limits**
$$ \lim\limits_{x\rightarrow 0}\frac{\sin(x)}{x},\quad \lim\limits_{x\rightarrow 0^{+}}\frac{1}{x},\quad \lim\limits_{x\rightarrow 0^{-}}\frac{1}{x}$$

```python
limit(sin(x)/x, x, 0)
```

```python
limit(1/x, x, 0, dir="+")
```

```python
limit(1/x, x, 0, dir="-")
```

**(3) series**

```python
exp(x)
```

```python
series(exp(x), x)
```

```python
series(exp(x), x, 1)
```

```python
series(sin(x), x, 0, 12)
```

```python
series(sin(x)*cos(x), x, 0, 8)
```

```python
series(sin(x)*cos(x)*exp(x), x, 0, 12)
```

### Linear algebra: Matrices

```python
m11, m12, m21, m22 = symbols("m11, m12, m21, m22")
b1, b2 = symbols("b1, b2")
```

```python
A = Matrix([[m11, m12],[m21, m22]])
A
```

```python
b = Matrix([[b1], [b2]])
b
```

```python
A**2
```

```python
A**5
```

```python
A * b
```

```python
A.det()
```

```python
A.inv()
```

### Solving equations
Solving
$$ x^2 - 1 = 0,\quad x^4 - x^2 - 1 = 0$$

```python
solve(x**2 - 1, x)
```

```python
solve(x**4 + x**3 - x**2 - 1, x)
```

Solving systems:
$$ x + y - 1 = 0,\quad x - y - 1 = 0,$$
and
$$ x + y - a = 0,\quad x - y - b = 0.$$

```python
solve([x + y - 1, x - y - 1], [x,y])
```

```python
a, b = symbols('a, b')
solve([x + y - a, x - y - b], [x,y])
```

```python

```


# Lab 4
## Introduction
The Euler method is a method for numerically solving a DE of the form
\begin{align}
\frac{\mathrm{d}y}{\mathrm{d}x} = f(x,y).
\end{align}

It is often necessary to solve DEs this way as analytical solutions are the exception
rather than the rule.

Euler’s method works by approximating small segments of the curve solution to the DE
with the straight-line tangent or slope of the curve. As long as we keep the segments
small enough, they will approximately match what the actual curve looks like. It requires us to ”know” an initial value $y(x_0) = y_0$ so we can start the calculation.

To calculate the first segment we start off with our known start point $(x_0, y_0)$, and calculate the end point, $(x_1, y_1)$. We can define $\Delta x$ to be some constant small distance so that we always increment the $x$ value by the same amount. Then, $\Delta y = m \Delta x$ and $(x_1, y_1)=(x_0, y_0)+(\Delta x, m\Delta x)$.

But, we also know that $m$, the slope of the line, is given by $\mathrm{d}y/\mathrm{d}x$, i.e., $f(x, y)$ evaluated at $(x_0, y_0)$. So actually, $\Delta y = f(x_0, y_0) \Delta x$.

The final step is to calculate the new point: the point at the end of the first line segment. This point is then given by $(x_1, y_1) = (x_0 + \Delta x, y_0 + f(x_0, y_0) \Delta x)$.

We then do it again to calculate $(x_2, y_2)$ using $(x_1, y_1)$ as our starting point. We then do it again to calculate $(x_3, y_3)$ using $(x_2, y_2)$ as our starting point and so on.

**Summary:** The Euler method for evaluating a DE of the form
\begin{align}
\frac{\mathrm{d}y}{\mathrm{d}x} = f(x,y).
\end{align}
involves the iterative calculation of
\begin{align}
x_{n+1} &= x_n + \Delta x\\
\text{and}\quad y_{n+1} &= y_n + f(x_n,y_n)\Delta x.
\end{align}

### Implementation

First import the necessary functions from NumPy and SciPy and set up Plotly.

```python
from numpy import arange, empty, exp
from plotly import graph_objs as go
```

Now let's write a function that implements Euler's method. We will model it on `scipy.integrate.odeint`. We will make slight changes to the parameters because we want to input $\Delta x$. Note that the string (delimited by triple quotes) immediately after the function definition is a _docstring_. It tells us what the function does and is good programming practice. The prodigious comments in the function body are not generally necessary but are included for you.

```python
def euler(func, y0, x0, xn, Dx):
    """
    Integrate an ordinary differential equation using Euler's method.

    Solves the initial value problem for systems of first order ode-s::
        dy/dx = func(y, x).

    Parameters
    ----------
    func : callable(y, x)
        Computes the derivative of y at x.
    y0 : float
        Initial condition on y.
    x0 : float
        Initial condition on x.
    xn : float
        Upper limit to value of x.
    Dx : float
        x increment.

    Returns
    -------
    x : float
        Array containing the value of x for each value of x0 + n * Dx,
        where n ranges from zero to floor( (xn - x0) / Dx ).
    y : float
        Array containing the value of y for each value of x.
    """
    x = arange(x0, xn, Dx)  # Create the x array
    y = empty(len(x))  # Create an empty y array of the same length as x
    y[0] = y0  # Set the first value of y to y0
    for n in range(len(x) - 1):  # Loop to populate the rest of the values of y
        y[n+1] = y[n] + func(y[n], x[n]) * Dx  # Euler's method

    return x, y  # Return x and y as a pair
```

First try solving
\begin{align}
\frac{\mathrm{d}y}{\mathrm{d}x} = y.
\end{align}
for $y(0)=1$ for $x$ between 1 and 5 and using $\Delta x=1$.

```python
def diff_eq(y, x):
    return y

x, y = euler(diff_eq, 1, 0, 5.01, 1)
```

Why was `xn` set to 5.01 rather than 5?

We know that the analytic solution to the above IVP is $y=\mathrm{e}^x$, so calculate that as well.

```python
x_analytic = arange(0, 5.01, 0.1)
y_analytic = exp(x_analytic)
```

Now plot them both for comparison.

```python
fig = go.Figure()
fig.add_trace(go.Scatter(x=x, y=y,
                         name='Euler'))
fig.add_trace(go.Scatter(x=x_analytic,
                         y=y_analytic,
                         name='Truth'))
fig.show('png')
```

Reproduce the comparison plot below but with $\Delta x=0.1$.

```python
x, y = euler(diff_eq, 1, 0, 5.01, 0.1)

x_analytic = arange(0, 5.01, 0.1)
y_analytic = exp(x_analytic)

fig = go.Figure()
fig.add_trace(go.Scatter(x=x, y=y,
                         name='Euler'))
fig.add_trace(go.Scatter(x=x_analytic,
                         y=y_analytic,
                         name='Truth'))
fig.show('png')
```

It is possible to quantify the error in the Euler solution compared to the analytic solution. To do this you need to re-calculate the analytic solution at the same $x$ points as you calculated your Euler solution. Then you can do a Mean Squared Error (MSE) comparison between the two.

```python
x, y = euler(diff_eq, 1, 0, 5.01, 1)
y_analytic = exp(x)
((y_analytic - y)**2).mean()
```

    2533.317105161909

Note that `((y_analytic - y)**2)` returned an `array` object, and then we called the `mean` method that was _bound_ to that object.

What is the MSE if $\Delta x = 0.1$?

```python
x, y = euler(diff_eq, 1, 0, 5.01, 0.1)
y_analytic = exp(x)
((y_analytic - y)**2).mean()
```

    88.60637343780924

## Exercises

In this lab you will try Euler's method for a couple of differential equations.

1.  a. Consider the IVP
\begin{align}
\frac{\mathrm{d}y}{\mathrm{d}x} = 2x\quad\text{where}\quad y(-2)=4.
\end{align}
Calculate the Euler approximation on the interval $x=[-2,2]$ using a step size of $\Delta x = 0.5$. On the same figure, plot your approximation and the analytic solution.

```python
from scipy.integrate import odeint

def diff_eq(y, x):
    return 2*x

x, y = euler(diff_eq, 4, -2, 2.01, 0.5)

x_analytic = arange(-2, 2.01, 0.1)
y_analytic = odeint(diff_eq, 4, x_analytic).flatten()

fig = go.Figure()
fig.add_trace(go.Scatter(x=x, y=y,
                         name='Euler'))
fig.add_trace(go.Scatter(x=x_analytic,
                         y=y_analytic,
                         name='Truth'))
fig.show('png')
```

1.  b. Calculate the mean squared error (MSE) of the approximation.

```python
x, y = euler(diff_eq, 4, -2, 2.01, 0.5)
y_analytic = odeint(diff_eq, 4, x_analytic)
((y_analytic - y)**2).mean()
```

    4.658533377001063

1.  c. Reproduce your plot from 1a except with $\Delta x=0.1$.

```python
def diff_eq(y, x):
    return 2*x

x, y = euler(diff_eq, 4, -2, 2.01, 0.1)

x_analytic = arange(-2, 2.01, 0.1)
y_analytic = odeint(diff_eq, 4, x_analytic).flatten()

fig = go.Figure()
fig.add_trace(go.Scatter(x=x, y=y,
                         name='Euler'))
fig.add_trace(go.Scatter(x=x_analytic,
                         y=y_analytic,
                         name='Truth'))
fig.show('png')
```

1.  d. Recalculate the MSE.

```python
x, y = euler(diff_eq, 4, -2, 2.01, 0.1)
y_analytic = odeint(diff_eq, 4, x_analytic)
((y_analytic - y)**2).mean()
```

    3.184400008820671

2.  a. The following is the DE for the arrow problem from class.
\begin{align}
\frac{\mathrm{d}y}{\mathrm{d}t} = 294\mathrm{e}^{-0.04t}-245\quad\text{where}\quad y(0)=0
\end{align}
Calculate the Euler approximation to the solution on the interval  $t=[0,10]$ with $\Delta t=0.5$. Plot your approximation and the analytic solution on the same figure.

```python
def diff_eq(y, x):
    return 294*exp(-0.04*x) - 245

x, y = euler(diff_eq, 0, 0, 10.01, 0.5)

x_analytic = arange(0, 10.01, 0.1)
y_analytic = odeint(diff_eq, 0, x_analytic).flatten()

fig = go.Figure()
fig.add_trace(go.Scatter(x=x, y=y,
                         name='Euler'))
fig.add_trace(go.Scatter(x=x_analytic,
                         y=y_analytic,
                         name='Truth'))
fig.show('png')
```

2.  b. Calculate the MSE of the approximation.

```python
x, y = euler(diff_eq, 0, 0, 10.01, 0.5)
y_analytic = odeint(diff_eq, 0, x_analytic)
((y_analytic - y)**2).mean()
```

    3132.157528098768

2.  c. Reproduce your plot from 2a except with $\Delta t=0.1$.

```python
def diff_eq(y, x):
    return 294*exp(-0.04*x) - 245

x, y = euler(diff_eq, 0, 0, 10.01, 0.1)

x_analytic = arange(0, 10.01, 0.1)
y_analytic = odeint(diff_eq, 0, x_analytic).flatten()

fig = go.Figure()
fig.add_trace(go.Scatter(x=x, y=y,
                         name='Euler'))
fig.add_trace(go.Scatter(x=x_analytic,
                         y=y_analytic,
                         name='Truth'))
fig.show('png')
```

2.  d. Recalculate the MSE.

```python
x, y = euler(diff_eq, 0, 0, 10.01, 0.1)
y_analytic = odeint(diff_eq, 0, x_analytic)
((y_analytic - y)**2).mean()
```

    2870.2127242316765

```python

```


# Plotting a Torus

To plot a surface in 3D do you first need to find the
parametrization of the surface you wish to plot.

For a Torus the parametrization looks like this:
\begin{align}
x &= [c + a  \cos v] \cos u \\
y &= [c + a  \cos v] \sin u \\
z &= a \sin v
\end{align}

where $a$ represents the radius of the tube and $c$ is the radius of the
center hole of the torus tube. See https://mathworld.wolfram.com/Torus.html for further information.

Begin by activating matplotlib notebook and importing numpy and
matplotlib

```python
%matplotlib notebook
# This enables you to drag and rotate the figure

from mpl_toolkits.mplot3d import Axes3D # needed to do the 3D Plots
import numpy as np
import matplotlib.pyplot as plt
```

```python
fig = plt.figure()
ax = fig.gca(projection='3d')

u = np.linspace(0, 2*np.pi, 100)
v = np.linspace(0, 2*np.pi, 100)
u, v = np.meshgrid(u, v) # Make coordinate matrices from coordinate vectors.
a, c = 0.2, 1.0 # Set a and c from the torus parametrization

# Paramatrization
x = (c + a*np.cos(v))*np.cos(u)
y = (c + a*np.cos(v))*np.sin(u)
z = a*np.sin(v)

ax.plot_surface(x, y, z)

ax.set_xlim3d(-1, 1)
ax.set_ylim3d(-1, 1)
ax.set_zlim3d(-1, 1)
plt.show()
```

# Make a surface plot of a sphere

Plot a sphere the parametrization is:

\begin{align}
x &= r \cos \theta \sin \phi  \\
y &= r \sin \theta \sin \phi  \\
z &= r \cos \phi
\end{align}

where $\theta$ is the loggitude coorinate running from $0$ to $2\pi$,
$\phi$ is the colatitude coordinate running from $0$ to $\pi$ and $r$ is the radius.

```python
fig = plt.figure()
ax = fig.gca(projection='3d')

# Fill inn the rest
```

# Make a surface plot of a Mobius strip

The parametrization is:
\begin{align}
x &= \left(1 + \frac{v}{2}\cos \frac{u}{2} \right)\cos u  \\
y &= \left(1 + \frac{v}{2}\cos v \right)\sin u  \\
z &= \frac{v}{2}\sin \frac{u}{2}
\end{align}

where $u$ is running from $0$ to $2\pi$ and $v$ is running from $-1$ to $1$.

```python
fig = plt.figure()
ax = fig.gca(projection='3d')

# Fill inn the rest

```

```python

```


# Tensor Manipulation: Psi4 and NumPy manipulation routines
Contracting tensors together forms the core of the Psi4NumPy project. First let us consider the popluar [Einstein Summation Notation](https://en.wikipedia.org/wiki/Einstein_notation) which allows for very succinct descriptions of a given tensor contraction.

For example, let us consider a [inner (dot) product](https://en.wikipedia.org/wiki/Dot_product):
$$c = \sum_{ij} A_{ij} * B_{ij}$$

With the Einstein convention, all indices that are repeated are considered summed over, and the explicit summation symbol is dropped:
$$c = A_{ij} * B_{ij}$$

This can be extended to [matrix multiplication](https://en.wikipedia.org/wiki/Matrix_multiplication):
\begin{align}
\rm{Conventional}\;\;\;  C_{ik} &= \sum_{j} A_{ij} * B_{jk} \\
\rm{Einstein}\;\;\;  C &= A_{ij} * B_{jk} \\
\end{align}

Where the $C$ matrix has *implied* indices of $C_{ik}$ as the only repeated index is $j$.

However, there are many cases where this notation fails. Thus we often use the generalized Einstein convention. To demonstrate let us examine a [Hadamard product](https://en.wikipedia.org/wiki/Hadamard_product_(matrices)):
$$C_{ij} = \sum_{ij} A_{ij} * B_{ij}$$

This operation is nearly identical to the dot product above, and is not able to be written in pure Einstein convention. The generalized convention allows for the use of indices on the left hand side of the equation:
$$C_{ij} = A_{ij} * B_{ij}$$

Usually it should be apparent within the context the exact meaning of a given expression.

Finally we also make use of Matrix notation:
\begin{align}
{\rm Matrix}\;\;\;  \bf{D} &= \bf{A B C} \\
{\rm Einstein}\;\;\;  D_{il} &= A_{ij} B_{jk} C_{kl}
\end{align}

Note that this notation is signified by the use of bold characters to denote matrices and consecutive matrices next to each other imply a chain of matrix multiplications!

## Einsum

To perform most operations we turn to [NumPy's einsum function](https://docs.scipy.org/doc/numpy/reference/generated/numpy.einsum.html) which allows the Einsten convention as an input. In addition to being much easier to read, manipulate, and change, it is also much more efficient that a pure Python implementation.

To begin let us consider the construction of the following tensor (which you may recognize):
$$G_{pq} = 2.0 * I_{pqrs} D_{rs} - 1.0 * I_{prqs} D_{rs}$$

First let us import our normal suite of modules:

```python
import numpy as np
import psi4
import time
```

We can then use conventional Python loops and einsum to perform the same task. Keep size relatively small as these 4-index tensors grow very quickly in size.

```python
size = 20

if size > 30:
    raise Exception("Size must be smaller than 30.")
D = np.random.rand(size, size)
I = np.random.rand(size, size, size, size)

# Build the fock matrix using loops, while keeping track of time
tstart_loop = time.time()
Gloop = np.zeros((size, size))
for p in range(size):
    for q in range(size):
        for r in range(size):
            for s in range(size):
                Gloop[p, q] += 2 * I[p, q, r, s] * D[r, s]
                Gloop[p, q] -=     I[p, r, q, s] * D[r, s]

g_loop_time = time.time() - tstart_loop

# Build the fock matrix using einsum, while keeping track of time
tstart_einsum = time.time()
J = np.einsum('pqrs,rs', I, D)
K = np.einsum('prqs,rs', I, D)
G = 2 * J - K

einsum_time = time.time() - tstart_einsum

# Make sure the correct answer is obtained
print('The loop and einsum fock builds match:    %s\n' % np.allclose(G, Gloop))
# Print out relative times for explicit loop vs einsum Fock builds
print('Time for loop G build:   %14.4f seconds' % g_loop_time)
print('Time for einsum G build: %14.4f seconds' % einsum_time)
print('G builds with einsum are {:3.4f} times faster than Python loops!'.format(g_loop_time / einsum_time))
```

    The loop and einsum fock builds match:    True

    Time for loop G build:           0.2845 seconds
    Time for einsum G build:         0.0005 seconds
    G builds with einsum are 536.7238 times faster than Python loops!

As you can see, the einsum function is considerably faster than the pure Python loops and, in this author's opinion, much cleaner and easier to use.

## Dot

Now let us turn our attention to a more canonical matrix multiplication example such as:
$$D_{il} = A_{ij} B_{jk} C_{kl}$$

We could perform this operation using einsum; however, matrix multiplication is an extremely common operation in all branches of linear algebra. Thus, these functions have been optimized to be more efficient than the `einsum` function. The matrix product will explicitly compute the following operation:
$$C_{ij} = A_{ij} * B_{ij}$$

This can be called with [NumPy's dot function](https://docs.scipy.org/doc/numpy/reference/generated/numpy.dot.html#numpy.dot).

```python
size = 200
A = np.random.rand(size, size)
B = np.random.rand(size, size)
C = np.random.rand(size, size)

# First compute the pair product
tmp_dot = np.dot(A, B)
tmp_einsum = np.einsum('ij,jk->ik', A, B)
print("Pair product allclose: %s" % np.allclose(tmp_dot, tmp_einsum))
```

    Pair product allclose: True

Now that we have proved exactly what the dot product does, let us consider the full chain and do a timing comparison:

```python
D_dot = np.dot(A, B).dot(C)
D_einsum = np.einsum('ij,jk,kl->il', A, B, C)
print("Chain multiplication allclose: %s" % np.allclose(D_dot, D_einsum))

print("\nnp.dot time:")
%timeit np.dot(A, B).dot(C)

print("\nnp.einsum time")
%timeit np.einsum('ij,jk,kl->il', A, B, C)
```

    Chain multiplication allclose: True

    np.dot time:
    1000 loops, best of 3: 288 µs per loop

    np.einsum time
    1 loop, best of 3: 1.68 s per loop

On most machines the `np.dot` times are roughly ~3,000 times faster. The reason is twofold:
 - The `np.dot` routines typically call [Basic Linear Algebra Subprograms (BLAS)](https://en.wikipedia.org/wiki/Basic_Linear_Algebra_Subprograms). The BLAS routines are highly optimized and threaded versions of the code.
 - The `np.einsum` code will not factorize the operation; Thus, the overall cost is ${\cal O}(N^4)$ (as there are four indices) rather than the factored $(\bf{A B}) \bf{C}$ which runs ${\cal O}(N^3)$.

The first issue is difficult to overcome; however, the second issue can be resolved by the following:

```python
print("np.einsum factorized time:")
%timeit np.einsum('ik,kl->il', np.einsum('ij,jk->ik', A, B), C)
```

    np.einsum factorized time:
    100 loops, best of 3: 5.24 ms per loop

On most machines the factorized `einsum` expression is only ~20 times slower than `np.dot`. While a massive improvement, this is a clear demonstration the BLAS usage is usually recommended. It is a tradeoff between speed and readability. The Psi4NumPy project tends to lean toward `einsum` usage except in case where the benefit is too large to pass up.

It should be noted that in NumPy 1.12 the [einsum function](https://docs.scipy.org/doc/numpy/reference/generated/numpy.einsum.html) has a `optimize` flag which will automatically factorize the einsum code for you. However, NumPy 1.12 was recently released and is not in most installations yet.

## Complex tensor manipulations
Let us consider a popular index transformation example:
$$M_{pqrs} = C_{pi} C_{qj} I_{ijkl} C_{rk} C_{sl}$$

Here, a naive `einsum` call would scale like $\mathcal{O}(N^8)$ which translates to an extremely costly computation for all but the smallest $N$.

```python
# Grab orbitals
size = 15
if size > 15:
    raise Exception("Size must be smaller than 15.")

C = np.random.rand(size, size)
I = np.random.rand(size, size, size, size)

# Numpy einsum N^8 transformation.
print("\nStarting Numpy's N^8 transformation...")
n8_tstart = time.time()
MO_n8 = np.einsum('pI,qJ,pqrs,rK,sL->IJKL', C, C, I, C, C)
n8_time = time.time() - n8_tstart
print("...transformation complete in %.3f seconds." % (n8_time))

# Numpy einsum N^5 transformation.
print("\n\nStarting Numpy's N^5 transformation with einsum...")
n5_tstart = time.time()
MO_n5 = np.einsum('pA,pqrs->Aqrs', C, I)
MO_n5 = np.einsum('qB,Aqrs->ABrs', C, MO_n5)
MO_n5 = np.einsum('rC,ABrs->ABCs', C, MO_n5)
MO_n5 = np.einsum('sD,ABCs->ABCD', C, MO_n5)
n5_time = time.time() - n5_tstart
print("...transformation complete in %.3f seconds." % n5_time)
print("\nN^5 %4.2f faster than N^8 algorithm!" % (n8_time / n5_time))
print("Allclose: %s" % np.allclose(MO_n8, MO_n5))

# Numpy GEMM N^5 transformation.
# Try to figure this one out!
print("\n\nStarting Numpy's N^5 transformation with dot...")
dgemm_tstart = time.time()
MO = np.dot(C.T, I.reshape(size, -1))
MO = np.dot(MO.reshape(-1, size), C)
MO = MO.reshape(size, size, size, size).transpose(1, 0, 3, 2)

MO = np.dot(C.T, MO.reshape(size, -1))
MO = np.dot(MO.reshape(-1, size), C)
MO = MO.reshape(size, size, size, size).transpose(1, 0, 3, 2)
dgemm_time = time.time() - dgemm_tstart
print("...transformation complete in %.3f seconds." % dgemm_time)
print("\nAllclose: %s" % np.allclose(MO_n8, MO))
print("N^5 %4.2f faster than N^8 algorithm!" % (n8_time / dgemm_time))
```

    Starting Numpy's N^8 transformation...
    ...transformation complete in 22.304 seconds.

    Starting Numpy's N^5 transformation with einsum...
    ...transformation complete in 0.004 seconds.

    N^5 5277.95 faster than N^8 algorithm!
    Allclose: True

    Starting Numpy's N^5 transformation with dot...
    ...transformation complete in 0.001 seconds.

    Allclose: True
    N^5 22712.25 faster than N^8 algorithm!

```python

```


```python
from sympy import *
```

```python
init_printing(use_unicode=True)
```

```python
# A Matrix object is constructed by providing a list
# of row vectors that make up the matrix
```

```python
A = Matrix([1,2,3])
```

```python
print(A)
```

    Matrix([[1], [2], [3]])

```python
# For pretty printing
A
```

```python
AA = Matrix([[1,2],[3,4]])
```

```python
AA
```

```python
print(AA)
```

    Matrix([[1, 2], [3, 4]])

```python
## Matrix transpose
B = AA.T
```

```python
B
```

```python
# Matrix multiplication
M = Matrix([[1,2,3],[4,5,6]])
```

```python
M
```

```python
N = M.T
```

```python
N
```

```python
MN = M*N
```

```python
MN
```

```python
# shape of the metrix. Number of rows and columns.
M.shape
```

```python
MN.shape
```

```python
# accessing entries
M = Matrix([[1,2,3],[4,5,6]])
```

```python
M
```

```python
M.row(0)
```

```python
M.col(1)
```

```python
# last column
M.col(-1)
```

```python
M[:,1]
```

```python
M[0,:]
```

```python
# insert row/col after a particular row/col
M = M.row_insert(1,Matrix([[0,4,9]]))
```

```python
M
```

```python
N = 2*M
```

```python
N
```

```python
N**2
```

```python
N**-1
```

```python
NNinv = N.inv()
```

```python
NNinv
```

```python
# determinant of a matrix
Ndet = N.det()
```

```python
Ndet
```

```python
NNinv.det()
```

```python
# matrix constructors
I = eye(4)
```

```python
I
```

```python
Z = zeros(3,4)
```

```python
Z
```

```python
ones(2,3)
```

```python
A = Matrix([[1,-1,0],[-1,2,-1],[0,-1,1]])
A
```

```python
A.eigenvals()
```

```python
A.eigenvects()
```

```python
# diagonalisation A = P*D*Pinv
P, D = A.diagonalize()
```

```python
D
```

```python
P
```

```python
P*D*P**-1 == A
```

    True

```python
# characteristic polynomials
lamda = symbols('lamda')
```

```python
p = A.charpoly(lamda)
```

```python
p
```

```python
factor(p)
```

```python

```