# MO-book Style and Hints

## Preamble for Google Colab

A core premise of the book and companion notebooks is that code should be immediately usable through the browser without extensive installation procedures.

Google Colab is a target platform for every notebook. With a few exceptions, tis preamble is included at the start of every notebook.

```python
# install Pyomo and solvers for Google Colab
import sys
if "google.colab" in sys.modules:
    !wget -N -q https://raw.githubusercontent.com/jckantor/MO-book/main/tools/install_on_colab.py
    %run install_on_colab.py
```

It may be worth testing if a similar procedure could be used for Windows or MacOS platforms. Using `wget` on windows, however, apparently requires installation, while `curl` provides sufficient functionality and available on Windows, MacOS, and Google Colab (need to verify this is true for Windows).

```python
!curl -s https://raw.githubusercontent.com/jckantor/MO-book/main/tools/_mobook.py -o mobook.py
import mobook
mobook.setup_pyomo()
mobook.setup_solvers()
mobook.svg()
```

## Matplotlib graphics

The `mobook.svg()` causes `matplotlib` graphics to appear as SVG files with embedded stix fonts. On JupyterLab, holding down the shift key while clicking on the image will give a `Save Image As ...` menu option that will save an SVG formatted image.  This needs to be tuned up tested, but gives a path forward with the book images.

* Font styles documented at https://matplotlib.org/stable/api/font_manager_api.html#matplotlib.font_manager.FontProperties

```python
import matplotlib.pyplot as plt
import numpy as np
from IPython.display import set_matplotlib_formats

x = np.linspace(0, 200, 2001)
plt.plot(x, np.sin(x))
plt.title("$sin(x)$")
```

    Text(0.5, 1.0, '$sin(x)$')

## Main Book Drawings

```python
import matplotlib
%matplotlib inline

use_latex_fonts_on_colab = False

matplotlib.rcParams['text.usetex'] = use_latex_fonts_on_colab

import sys
if 'google.colab' in sys.modules:
    import shutil
    if not shutil.which('pyomo'):
        !pip install pyomo
        assert( shutil.which('pyomo') )

    if use_latex_fonts_on_colab and not shutil.which( '/usr/share/texmf/tex/latex/type1cm' ):
        ! sudo apt-get install texlive-latex-recommended
        ! sudo apt-get install dvipng texlive-latex-extra texlive-fonts-recommended
        ! wget http://mirrors.ctan.org/macros/latex/contrib/type1cm.zip
        ! unzip type1cm.zip -d /tmp/type1cm
        ! cd /tmp/type1cm/type1cm/ && sudo latex type1cm.ins
        ! sudo mkdir /usr/share/texmf/tex/latex/type1cm
        ! sudo cp /tmp/type1cm/type1cm/type1cm.sty /usr/share/texmf/tex/latex/type1cm
        ! sudo texhash
        !apt install cm-super
else: # locally, assume that LaTeX is installed and use its fonts
    matplotlib.rcParams['text.usetex'] = True
```

```python
import matplotlib.pyplot as plt

%matplotlib inline

# https://stackoverflow.com/questions/17687213/how-to-obtain-the-same-font-style-size-etc-in-matplotlib-output-as-in-latex
params = {'font.size'   : 10, # the book seems to be in 10pt, change if needed
          'font.family' : 'lmodern',
          }

plt.rcParams.update(params)
default_size_inches = (3.54,3.54)
plt.rcParams['figure.figsize'] = default_size_inches
```

```python
!curl -sO https://raw.githubusercontent.com/jckantor/MO-book/main/tools/install_on_colab.py
```

```python
[ k for k in plt.rcParams.keys() if 'font' in k ]
```

    ['font.cursive',
     'font.family',
     'font.fantasy',
     'font.monospace',
     'font.sans-serif',
     'font.serif',
     'font.size',
     'font.stretch',
     'font.style',
     'font.variant',
     'font.weight',
     'legend.fontsize',
     'legend.title_fontsize',
     'mathtext.fontset',
     'pdf.fonttype',
     'pdf.use14corefonts',
     'pgf.rcfonts',
     'ps.fonttype',
     'svg.fonttype']

```python
import pyomo.environ as pyo
import pyomo
pyomo.__version__
```

    '6.4.0'

```python
%load_ext autoreload
%autoreload 2
```

    The autoreload extension is already loaded. To reload it, use:
      %reload_ext autoreload

```python
import sys
sys.path.append('./code/')
import draw
```

```python
draw.SetOutputPath( '/work in progress/MO book/results' )
```

# Chapter one

```python
import sympy, math, numpy as np, sys

x  = sympy.Symbol('x')

# now pi is a symbol, just like x
pi = sympy.Symbol('pi')

# we redefine h using the same right-hand-side code as before,
# but now with x and pi as symbols
h = (pi*x**2 + 500)/(2*pi*x+50)

# to have the drivative on the symbol pi we need it from the new version of h
hprime = sympy.diff( h, x )

solution = sympy.solveset( sympy.diff( h, x ), x )
solution
```

$\displaystyle \left\{\frac{- 5 \sqrt{5} \sqrt{4 \pi + 5} - 25}{\pi}, \frac{5 \sqrt{5} \sqrt{4 \pi + 5} - 25}{\pi}\right\}$

```python
def Preety( formula ):
    from sympy import latex
    from IPython.display import display, Math
    display( Math( latex( formula ) ) )

Preety( h )
Preety( sympy.simplify( h ) )
Preety( hprime )
Preety( sympy.simplify( hprime ) )
```

$\displaystyle \frac{\pi x^{2} + 500}{2 \pi x + 50}$

$\displaystyle \frac{\pi x^{2} + 500}{2 \left(\pi x + 25\right)}$

$\displaystyle \frac{2 \pi x}{2 \pi x + 50} - \frac{2 \pi \left(\pi x^{2} + 500\right)}{\left(2 \pi x + 50\right)^{2}}$

$\displaystyle \frac{\pi \left(- \pi x^{2} + 2 x \left(\pi x + 25\right) - 500\right)}{2 \left(\pi x + 25\right)^{2}}$

```python
Preety( solution )

s = max(solution.subs( pi, math.pi ).evalf())

print(s)
```

$\displaystyle \left\{\frac{- 5 \sqrt{5} \sqrt{4 \pi + 5} - 25}{\pi}, \frac{5 \sqrt{5} \sqrt{4 \pi + 5} - 25}{\pi}\right\}$

    6.95803920980307

```python
def Plot( h, s=None, start=0, stop=20, width=18, height=8, file_name=None ):
    with plt.rc_context({'figure.figsize': (width,height)}):
        plt.rcParams['figure.figsize'] = (width,height)

        plt.grid()
        plt.xlabel(r'$x$')
        plt.ylabel(r'$h(x)$')

        plt.xticks(np.arange(start, stop+1, step=1))

        x = sympy.Symbol('x')
        f = sympy.lambdify( x, h.subs( pi, math.pi ) )

        import numpy
        x = numpy.linspace(start=start,stop=stop,num=100)
        y = f(x)

        plt.plot(x,y,label='$h(x)='+sympy.latex(h)+'$',linewidth=3)
        if s is None:
            x = numpy.linspace(start=start,stop=stop,num=stop-start+1)
            y = f(x)
            plt.plot(x,y, 'bo', label='some points', markersize=8)
        else:
            plt.plot(s,f(s), 'ro', label='$x^*$ optimum', markersize=8)

        plt.legend()

        if file_name is not None:
            plt.savefig( draw._output_path+file_name, bbox_inches='tight', pad_inches=0 )

        plt.show()
```

```python
Plot( h, None, 0, 20, 8, 5, 'AliceSome.pdf' )
```

```python
Plot( h, s, 0, 20, 6, 3, 'AliceOptimum.pdf' )
```

# Chapter two

```python
def SimpleDraw( model ):
    with plt.rc_context({'figure.figsize': (8,6)}):
        return draw.Draw( model, isolines=True, file_name=model.name+'.pdf' )
```

```python
def CreateBIM():
    m    = pyo.ConcreteModel('BIM')

    m.x1 = pyo.Var( within=pyo.NonNegativeReals )
    m.x2 = pyo.Var( within=pyo.NonNegativeReals )

    @m.Objective( sense= pyo.maximize )
    def obj(m):
        return 12*m.x1 + 9*m.x2

    @m.Constraint()
    def silicon(m):   return    m.x1          <= 1000
    @m.Constraint()
    def germanium(m): return             m.x2 <= 1500
    @m.Constraint()
    def plastic(m):   return    m.x1 +   m.x2 <= 1750
    @m.Constraint()
    def copper(m):     return 4*m.x1 + 2*m.x2 <= 4800

    return m
```

```python
SimpleDraw( CreateBIM() )
```

```python
basicfeasiblesolutions = draw.Draw( CreateBIM(), 'BuildingMicrochips.pdf' )
```

```python
basicfeasiblesolutions.rename(columns={'x1': '$x_1$', 'x2': '$x_2$'}).astype(int).to_latex(draw._output_path+'chips.tex')
if 'google.colab' in sys.modules:
    import os
    from google.colab import files
    files.download( 'chips.tex' )
```

    C:\Users\joaquimg\AppData\Local\Temp\ipykernel_22328\2016893662.py:1: FutureWarning: In future versions `DataFrame.to_latex` is expected to utilise the base implementation of `Styler.to_latex` for formatting and rendering. The arguments signature may therefore change. It is recommended instead to use `DataFrame.style.to_latex` which also contains additional functionality.
      basicfeasiblesolutions.rename(columns={'x1': '$x_1$', 'x2': '$x_2$'}).astype(int).to_latex(draw._output_path+'chips.tex')

# Chapter three

The first version was a copy of [the model on this deck](http://web.tecnico.ulisboa.pt/mcasquilho/compute/_linpro/TaylorB_module_c.pdf]).

Below several versions of possible models.

```python
def CreateBBaExample():
    model    = pyo.ConcreteModel('BBa')

    model.x1 = pyo.Var( within=pyo.NonNegativeReals )
    model.x2 = pyo.Var( within=pyo.NonNegativeReals )

    model.obj = pyo.Objective( sense= pyo.maximize
                                    , expr = 2*model.x1 + 3*model.x2 )

    model.c1  = pyo.Constraint(expr =  2*model.x1 +  1*model.x2 <=  10)
    model.c2  = pyo.Constraint(expr =  3*model.x1 +  6*model.x2 <=  40)

    return model
```

```python
draw.Draw( CreateBBaExample(), integer=True, isolines=True, file_name=CreateBBaExample().name+'.pdf', title='First B\&B example' )
```

```python
sol,root = draw.BB( CreateBBaExample(), solver='gurobi_direct' )
```

```python
draw.ToTikz(root, 'BBa.tex', fig_only=True )
```

```python
draw.DrawBB(root, 'BB.pdf')
```

```python
draw.DotExporter(root).to_picture(draw._output_path+'BBplain.pdf')
```

```python
def CreateBBbExample():
    m    = pyo.ConcreteModel('BBb')

    m.x1 = pyo.Var( within=pyo.NonNegativeReals )
    m.x2 = pyo.Var( within=pyo.NonNegativeReals )

    m.obj= pyo.Objective( sense= pyo.maximize
                        , expr =  1*m.x1 + 2*m.x2 )

    m.c1 = pyo.Constraint(expr = -4*m.x1 + 5*m.x2 <= 11)
    m.c2 = pyo.Constraint(expr =  5*m.x1 - 2*m.x2 <=  9)

    return m
```

```python
draw.Draw( CreateBBbExample(), integer=True, isolines=False, file_name=None, title='test' )
```

```python
sol,root = draw.BB( CreateBBbExample(), solver='gurobi_direct' )
```

```python
draw.ToTikz(root, 'BBb.tex', fig_only=True )
```

```python
draw.DotExporter(root).to_dotfile(draw._output_path+'test.dot')
```

```python
draw.DrawBB(root,'BBBook.pdf')
```

```python
def CreateBIMmodified():
    m    = pyo.ConcreteModel('BIMmodified')

    m.x1 = pyo.Var( within=pyo.NonNegativeReals )
    m.x2 = pyo.Var( within=pyo.NonNegativeReals )

    m.obj       = pyo.Objective( sense= pyo.maximize
                               , expr = 12*m.x1 + 9*m.x2 )

    m.silicon   = pyo.Constraint(expr =    m.x1          <= 900 )
    m.germanium = pyo.Constraint(expr =             m.x2 <= 1350)
    m.plastic   = pyo.Constraint(expr =    m.x1 +   m.x2 <= 1801)
    m.copper    = pyo.Constraint(expr =  4*m.x1 + 2*m.x2 <= 4903)

    return m
```

```python
sol,root = draw.BB( CreateBIMmodified(), solver='gurobi_direct', draw_integer=False, xlim=(-50,1050), ylim=(-50,1550) )
```

```python
draw.ToTikz(root, 'BIMmodified.tex', fig_only=True )
```

```python
def CreateBIMperturbed():
    m    = pyo.ConcreteModel('BIMperturbed')

    m.x1 = pyo.Var( within=pyo.NonNegativeReals )
    m.x2 = pyo.Var( within=pyo.NonNegativeReals )

    m.obj       = pyo.Objective( sense= pyo.maximize
                               , expr = 12*m.x1 + 9*m.x2 )

    m.silicon   = pyo.Constraint(expr =    m.x1                <= 1000 )
    m.germanium = pyo.Constraint(expr =                   m.x2 <= 1500)
    m.plastic   = pyo.Constraint(expr =    m.x1    +      m.x2 <= 1750)
    m.copper    = pyo.Constraint(expr =  4.04*m.x1 + 2.02*m.x2 <= 4800)

    return m
```

```python
sol,root = draw.BB( CreateBIMperturbed(), solver='gurobi_direct', draw_integer=False, xlim=(-50,1050), ylim=(-50,1550) )
```

```python
draw.ToTikz(root, 'BIMperturbed.tex', fig_only=True )
```

```python
draw.Draw( CreateBIM(), integer=False, isolines=True, file_name=None, title='First B\&B example' )
```

```python
nx,ny = 6,5
F = { (2,1), (3,1), (4,1), (2,2), (3,2), (4,2), (3,3), (4,3), (3,4) }

import itertools
points = list(itertools.product( range(0,nx+1), range(0,ny+1) ) )
feasible   = [ p for p in points if p in F ]
infeasible = [ p for p in points if not p in F ]
if infeasible:
    plt.plot( *zip(*infeasible), 'ro', zorder=2, markersize=9)
if feasible:
    plt.plot( *zip(*feasible), 'bo', zorder=2, markersize=9)

def Pol( coord, style, alpha, width ):
    coord.append(coord[0]) #repeat the first point to create a 'closed loop'
    plt.plot(*zip(*coord),style, alpha=alpha, linewidth=width, zorder=1)

Pol( [ (1.6,.5), (2.5,5), (4.5,3.5), (4,.5) ], 'g-', 1, 2 )
Pol( [ (2,1), (2,2), (2.5, 3.5), (3.5, 4.5), (4,3), (4,1) ], 'm--', .8, 3 )
Pol( [ (2,1), (2,2), (3,4), (4,3), (4,1) ], 'c-.', 1, 3 )

ax = plt.gca()

# Hide the right and top spines
for position in ['left','right','top','bottom']:
    ax.spines[position].set_visible(False)

ax.xaxis.set_visible(False)
ax.yaxis.set_visible(False)

plt.savefig( draw._output_path+'3regions.pdf', bbox_inches='tight', pad_inches=0 )
plt.show()
```


```python
import sys
sys.path = ['/Users/sebastian/github/mlxtend/'] + sys.path
```

```python
%load_ext watermark
%watermark -a 'Sebastian Raschka' -v
```

    Sebastian Raschka

    CPython 3.4.3
    IPython 3.0.0

# Artificial Neurons and Single-Layer Neural Networks

## - How Machine Learning Algorithms Work Part 1

**This article offers a brief glimpse of the history and basic concepts of machine learning. We will take a look at the first algorithmically described neural network and the gradient descent algorithm in context of adaptive linear neurons, which will not only introduce the principles of machine learning but also serve as the basis for modern multilayer neural networks in future articles.**

## Sections

- [Introduction](#Introduction)
- [Artificial Neurons and the McCulloch-Pitts Model](#Artificial-Neurons-and-the-McCulloch-Pitts-Model)
- [Frank Rosenblatt's Perceptron](#Frank-Rosenblatt's-Perceptron)
    - [The Unit Step Function](#The-Unit-Step-Function)
    - [The Perceptron Learning Rule](#The-Perceptron-Learning-Rule)
    - [Implementing the Perceptron Rule in Python](#Implementing-the-Perceptron-Rule-in-Python)
    - [Problems with Perceptrons](#Problems-with-Perceptrons)
- [Adaptive Linear Neurons and the Delta Rule](#Adaptive-Linear-Neurons-and-the-Delta-Rule)
    - [Gradient Descent](#Gradient-Descent)
    - [The Gradient Descent Rule in Action](#The-Gradient-Descent-Rule-in-Action)
    - [Online Learning via Stochastic Gradient Descent](#Online-Learning-via-Stochastic-Gradient-Descent)
- [What's Next?](#What's-Next?)
- [References](#References)

# Introduction

[[back to top](#Sections)]

Machine learning is one of the hottest and most exciting fields in the modern age of technology. Thanks to machine learning, we enjoy robust email spam filters, convenient text and voice recognition, reliable web search engines, challenging chess players, and, hopefully soon, safe and efficient self-driving cars.

Without any doubt, machine learning has become a big and popular field, and sometimes it may be challenging to see the (random) forest for the (decision) trees. Thus, I thought that it might be worthwhile to explore different machine learning algorithms in more detail by not only discussing the theory but also by implementing them step by step.

To briefly summarize what machine learning is all about: "[Machine learning is the] field of study that gives computers the ability to learn without being explicitly programmed" (Arthur Samuel, 1959). Machine learning is about the development and use of algorithms that can recognize patterns in data in order to make decisions based on statistics, probability theory, combinatorics, and optimization.

The first article in this series will introduce perceptrons and the adaline (ADAptive LINear NEuron), which fall into the category of single-layer neural networks. The perceptron is not only the first algorithmically described learning algorithm [[1](#References)], but it is also very intuitive, easy to implement, and a good entry point to the (re-discovered) modern state-of-the-art machine learning algorithms: Artificial neural networks (or "deep learning" if you like). As we will see later, the adaline is a consequent improvement of the perceptron algorithm and offers a good opportunity to learn about a popular optimization algorithm in machine learning: gradient descent.

# Artificial Neurons and the McCulloch-Pitts Model

[[back to top](#Sections)]

The initial idea of the perceptron dates back to the work of Warren McCulloch and Walter Pitts in 1943 [[2](#References)], who drew an analogy between biological neurons and simple logic gates with binary outputs. In more intuitive terms, neurons can be understood as the subunits of a neural network in a biological brain. Here, the signals of variable magnitudes arrive at the dendrites. Those input signals are then accumulated in the cell body of the neuron, and if the accumulated signal exceeds a certain threshold, a output signal is generated that which will be passed on by the axon.

# Frank Rosenblatt's Perceptron

[[back to top](#Sections)]

To continue with the story, a few years after McCulloch and Walter Pitt, Frank Rosenblatt published the first concept of the Perceptron learning rule [[1](#References)]. The main idea was to define an algorithm in order to learn the values of the weights $w$ that are then multiplied with the input features in order to make a decision whether a neuron fires or not. In context of pattern classification, such an algorithm could be useful to determine if a sample belongs to one class or the other.

To put the perceptron algorithm into the broader context of machine learning: The perceptron belongs to the category of supervised learning algorithms, single-layer binary linear classifiers to be more specific. In brief, the task is to predict to which of two possible categories a certain data point belongs based on a set of input variables. In this article, I don't want to discuss the concept of predictive modeling and classification in too much detail, but if you prefer more background information, please see my previous article "[Introduction to supervised learning](http://sebastianraschka.com/Articles/2014_intro_supervised_learning.html)".

## The Unit Step Function

[[back to top](#Sections)]

Before we dive deeper into the algorithm(s) for learning the weights of the artificial neuron, let us take a brief look at the basic notation. In the following sections, we will label the *positive* and *negative* class in our binary classification setting as "1" and "-1", respectively. Next, we define an activation function $g(\mathbf{z})$ that takes a linear combination of the input values $\mathbf{x}$ and weights $\mathbf{w}$ as input ($\mathbf{z} = w_1x_{1} + \dots + w_mx_{m}$), and if $g(\mathbf{z})$ is greater than a defined threshold $\theta$ we predict 1 and -1 otherwise; in this case, this activation function $g$ is an alternative form of a simple "unit step function," which is sometimes also called "Heaviside step function."

*(Please note that the *unit step* is classically defined as being equal to 0 if $ z < 0$ and 1 for $z \ge 0$; nonetheless, we will refer to the following piece-wise linear function with -1 if $z < \theta$ and 1 for $z \ge \theta$ as *unit step function* for simplicity).*

$$
 g(\mathbf{z}) =\begin{cases}
    1 & \text{if $\mathbf{z} \ge \theta$}\\
    -1 & \text{otherwise}.
  \end{cases}
$$

where

$$\mathbf{z} =  w_1x_{1} + \dots + w_mx_{m} = \sum_{j=1}^{m} x_{j}w_{j} \\ = \mathbf{w}^T\mathbf{x}$$

$\mathbf{w}$ is the feature vector, and $\mathbf{x}$ is an $m$-dimensional sample from the training dataset:

$$
\mathbf{w} = \begin{bmatrix}
    w_{1}  \\
    \vdots \\
    w_{m}
\end{bmatrix}
\quad  \mathbf{x} = \begin{bmatrix}
    x_{1}  \\
    \vdots \\
    x_{m}
\end{bmatrix}$$

In order to simplify the notation, we bring $\theta$ to the left side of the equation and define $w_0 = -\theta  \text{ and } x_0=1$

so that

$$\begin{equation}
 g({\mathbf{z}}) =\begin{cases}
    1 & \text{if $\mathbf{z} \ge 0$}\\
    -1 & \text{otherwise}.
  \end{cases}
\end{equation}$$

and

$$\mathbf{z} = w_0x_{0} + w_1x_{1} + \dots + w_mx_{m} = \sum_{j=0}^{m} x_{j}w_{j} \\ = \mathbf{w}^T\mathbf{x}.$$

## The Perceptron Learning Rule

[[back to top](#Sections)]

It might sound like extreme case of a reductionist approach, but the idea behind this "thresholded" perceptron was to mimic how a single neuron in the brain works: It either "fires" or not. To summarize the main points from the previous section: A perceptron receives multiple input signals, and if the sum of the input signals exceed a certain threshold it either returns a signal or  remains "silent" otherwise. What made this a "machine learning" algorithm was Frank Rosenblatt's idea of the perceptron learning rule: The perceptron algorithm is about learning the weights for the input signals in order to draw linear decision boundary that allows us to discriminate between the two linearly separable classes +1 and -1.

Rosenblatt's initial perceptron rule is fairly simple and can be summarized by the following steps:

1. Initialize the weights to 0 or small random numbers.
2. For each training sample $\mathbf{x^{(i)}}$:
    2. Calculate the *output* value.
    2. Update the weights.

The output value is the class label predicted by the unit step function that we defined earlier (output $=g(\mathbf{z})$) and the weight update can be written more formally as  $w_j := w_j + \Delta w_j$.

The value for updating the weights at each increment is calculated by the learning rule

$\Delta w_j = \eta \; (\text{target}^{(i)} - \text{output}^{(i)})\;x^{(i)}_{j}$

where $\eta$ is the learning rate (a constant between 0.0 and 1.0), "target" is the true class label, and the "output" is the predicted class label.

It is important to note that all weights in the weight vector are being updated simultaneously. Concretely, for a 2-dimensional dataset, we would write the update as:

$\Delta w_0 = \eta(\text{target}^{(i)} - \text{output}^{(i)})$
$\Delta w_1 = \eta(\text{target}^{(i)} - \text{output}^{(i)})\;x^{(i)}_{1}$
$\Delta w_2 = \eta(\text{target}^{(i)} - \text{output}^{(i)})\;x^{(i)}_{2}$

Before we implement the perceptron rule in Python, let us make a simple thought experiment to illustrate how beautifully simple this learning rule really is. In the two scenarios where the perceptron predicts the class label correctly, the weights remain unchanged:

- $\Delta w_j = \eta(-1^{(i)} - -1^{(i)})\;x^{(i)}_{j} = 0$
- $\Delta w_j = \eta(1^{(i)} - 1^{(i)})\;x^{(i)}_{j} = 0$

However, in case of a wrong prediction, the weights are being "pushed" towards the direction of the positive or negative target class, respectively:

- $\Delta w_j = \eta(1^{(i)} - -1^{(i)})\;x^{(i)}_{j} = \eta(2)\;x^{(i)}_{j}$
- $\Delta w_j = \eta(-1^{(i)} - 1^{(i)})\;x^{(i)}_{j} = \eta(-2)\;x^{(i)}_{j}$

It is important to note that the convergence of the perceptron is only guaranteed if the two classes are linearly separable. If the two classes can't be separated by a linear decision boundary, we can set a maximum number of passes over the training dataset ("epochs") and/or a threshold for the number of tolerated misclassifications.

## Implementing the Perceptron Rule in Python

[[back to top](#Sections)]

In this section, we will implement the simple perceptron learning rule in Python to classify flowers in the Iris dataset.
Please note that I omitted some "safety checks" for clarity, for a more "robust" version please see the following [code on GitHub](https://github.com/rasbt/mlxtend/blob/master/mlxtend/classifier/perceptron.py).

```python
import numpy as np

class Perceptron(object):

    def __init__(self, eta=0.01, epochs=50):
        self.eta = eta
        self.epochs = epochs

    def train(self, X, y):

        self.w_ = np.zeros(1 + X.shape[1])
        self.errors_ = []

        for _ in range(self.epochs):
            errors = 0
            for xi, target in zip(X, y):
                update = self.eta * (target - self.predict(xi))
                self.w_[1:] +=  update * xi
                self.w_[0] +=  update
                errors += int(update != 0.0)
            self.errors_.append(errors)
        return self

    def net_input(self, X):
        return np.dot(X, self.w_[1:]) + self.w_[0]

    def predict(self, X):
        return np.where(self.net_input(X) >= 0.0, 1, -1)
```

For the following example, we will load the Iris data set from the [UCI Machine Learning Repository](http://archive.ics.uci.edu/ml/) and only focus on the two flower species *Setosa* and *Versicolor*. Furthermore, we will only use the two features *sepal length* and *petal length* for visualization purposes.

```python
import pandas as pd
df = pd.read_csv('https://archive.ics.uci.edu/ml/machine-learning-databases/iris/iris.data', header=None)

# setosa and versicolor
y = df.iloc[0:100, 4].values
y = np.where(y == 'Iris-setosa', -1, 1)

# sepal length and petal length
X = df.iloc[0:100, [0,2]].values
```

```python
%matplotlib inline
import matplotlib.pyplot as plt
from mlxtend.plotting import plot_decision_regions

ppn = Perceptron(epochs=10, eta=0.1)

ppn.train(X, y)
print('Weights: %s' % ppn.w_)
plot_decision_regions(X, y, clf=ppn)
plt.title('Perceptron')
plt.xlabel('sepal length [cm]')
plt.ylabel('petal length [cm]')
plt.show()

plt.plot(range(1, len(ppn.errors_)+1), ppn.errors_, marker='o')
plt.xlabel('Iterations')
plt.ylabel('Misclassifications')
plt.show()

```

As we can see, the perceptron converges after the 6th iteration and separates the two flower classes perfectly.

## Problems with Perceptrons

[[back to top](#Sections)]

Although the perceptron classified the two Iris flower classes perfectly, convergence is one of the biggest problems of the perceptron. Frank Rosenblatt proofed mathematically that the perceptron learning rule converges if the two classes can be separated by linear hyperplane, but problems arise if the classes cannot be separated perfectly by a linear classifier. To demonstrate this issue, we will use two different classes and features from the Iris dataset.

```python
# versicolor and virginica
y2 = df.iloc[50:150, 4].values
y2 = np.where(y2 == 'Iris-virginica', -1, 1)

# sepal width and petal width
X2 = df.iloc[50:150, [1,3]].values

ppn = Perceptron(epochs=25, eta=0.01)
ppn.train(X2, y2)

plot_decision_regions(X2, y2, clf=ppn)
plt.show()

plt.plot(range(1, len(ppn.errors_)+1), ppn.errors_, marker='o')
plt.xlabel('Iterations')
plt.ylabel('Misclassifications')
plt.show()
```

```python
print('Total number of misclassifications: %d of 100' % (y2 != ppn.predict(X2)).sum())
```

    Total number of misclassifications: 43 of 100

Even at a lower training rate, the perceptron failed to find a good decision boundary since one or more samples will always be misclassified in every epoch so that the learning rule never stops updating the weights.

It may seem paradoxical in this context that another shortcoming of the perceptron algorithm is that it stops updating the weights as soon as all samples are classified correctly. Our intuition tells us that a decision boundary with a large margin between the classes (as indicated by the dashed line in the figure below) likely has a better generalization error than the decision boundary of the perceptron. But large-margin classifiers such as Support Vector Machines are a topic for another time.

# Adaptive Linear Neurons and the Delta Rule

[[back to top](#Sections)]

The perceptron surely was very popular at the time of its discovery, however, it only took a few years until Bernard Widrow and his doctoral student Tedd Hoff proposed the idea of the Adaptive Linear Neuron (adaline) [[3](#References)].

In contrast to the perceptron rule, the delta rule of the adaline (also known as Widrow-Hoff" rule or Adaline rule) updates the weights based on a linear activation function rather than a unit step function; here, this linear activation function $g(\mathbf{z})$ is just the identity function of the net input $g(\mathbf{w}^T\mathbf{x}) = \mathbf{w}^T\mathbf{x}$. In the next section, we will see why this linear activation is an improvement over the perceptron update and where the name "delta rule" comes from.

## Gradient Descent

[[back to top](#Sections)]

Being a continuous function, one of the biggest advantages of the linear activation function over the unit step function is that it is differentiable. This property allows us to define a cost function $J(\mathbf{w})$ that we can minimize in order to update our weights. In the case of the linear activation function, we can define the cost function $J(\mathbf{w})$ as the *sum of squared errors* (SSE), which is similar to the cost function that is minimized in ordinary least squares (OLS) linear regression.

$$J(\mathbf{w})  = \frac{1}{2} \sum_{i} (\text{target}^{(i)} - \text{output}^{(i)})^2 \quad \quad \text{output}^{(i)} \in \mathbb{R}$$

(The fraction $\frac{1}{2}$ is just used for convenience to derive the gradient as we will see in the next paragraphs.)

In order to minimize the SSE cost function, we will use gradient descent, a simple yet useful optimization algorithm that is often used in machine learning to find the local minimum of linear systems.

Before we get to the fun part (calculus), let us consider a convex cost function for one single weight. As illustrated in the figure below, we can describe the principle behind gradient descent as "climbing down a hill" until a local or global minimum is reached. At each step, we take a step into the opposite direction of the gradient, and the step size is determined by the value of the learning rate as well as the slope of the gradient.

Now, as promised, onto the fun part -- deriving the Adaline learning rule.
As mentioned above, each update is updated by taking a step into the opposite direction of the gradient $\Delta \mathbf{w} = - \eta \nabla J(\mathbf{w})$, thus, we have to compute the partial derivative of the cost function for each weight in the weight vector:  $\Delta w_j = - \eta \frac{\partial J}{\partial w_j}$.

The partial derivative of the SSE cost function for a particular weight can be calculated as follows:

$$\begin{equation}
 \frac{\partial J}{\partial w_j} = \frac{\partial }{\partial w_j} \frac{1}{2} \sum_i  (t^{(i)} - o^{(i)})^2 \\
= \frac{1}{2} \sum_i \frac{\partial}{\partial w_j} (t^{(i)} - o^{(i)})^2 \\
= \frac{1}{2} \sum_i 2 (t^{(i)} - o^{(i)}) \frac{\partial}{\partial w_j} (t^{(i)} - o^{(i)}) \\
=  \sum_i (t^{(i)} - o^{(i)}) \frac{\partial}{\partial w_j} \bigg(t^{(i)} - \sum_j w_j x^{(i)}_{j}\bigg) \\
= \sum_i  (t^{(i)} - o^{(i)})(-x^{(i)}_{j})
\end{equation}$$

(t = target, o = output)

And if we plug the results back into the learning rule, we get

$\Delta w_j = - \eta \frac{\partial J}{\partial w_j} = - \eta \sum_i  (t^{(i)} - o^{(i)})(- x^{(i)}_{j}) = \eta \sum_i (t^{(i)} - o^{(i)})x^{(i)}_{j}$,

Eventually, we can apply a simultaneous weight update similar to the perceptron rule:

$\mathbf{w} := \mathbf{w} + \Delta \mathbf{w}$.

**Although, the learning rule above looks identical to the perceptron rule, we shall note the two main differences:**

1. Here, the output "o" is a real number and not a class label as in the perceptron learning rule.
2. The weight update is calculated based on all samples in the training set (instead of updating the weights incrementally after each sample), which is why this approach is also called "batch" gradient descent.

## The Gradient Descent Rule in Action

[[back to top](#Sections)]

Now, it's time to implement the gradient descent rule in Python.

```python
import numpy as np

class AdalineGD(object):

    def __init__(self, eta=0.01, epochs=50):
        self.eta = eta
        self.epochs = epochs

    def train(self, X, y):

        self.w_ = np.zeros(1 + X.shape[1])
        self.cost_ = []

        for i in range(self.epochs):
            output = self.net_input(X)
            errors = (y - output)
            self.w_[1:] += self.eta * X.T.dot(errors)
            self.w_[0] += self.eta * errors.sum()
            cost = (errors**2).sum() / 2.0
            self.cost_.append(cost)
        return self

    def net_input(self, X):
        return np.dot(X, self.w_[1:]) + self.w_[0]

    def activation(self, X):
        return self.net_input(X)

    def predict(self, X):
        return np.where(self.activation(X) >= 0.0, 1, -1)
```

In practice, it often requires some experimentation to find a good learning rate for optimal convergence, thus, we will start by plotting the cost for two different learning rates.

```python
ada = AdalineGD(epochs=10, eta=0.01).train(X, y)
plt.plot(range(1, len(ada.cost_)+1), np.log10(ada.cost_), marker='o')
plt.xlabel('Iterations')
plt.ylabel('log(Sum-squared-error)')
plt.title('Adaline - Learning rate 0.01')
plt.show()

ada = AdalineGD(epochs=10, eta=0.0001).train(X, y)
plt.plot(range(1, len(ada.cost_)+1), ada.cost_, marker='o')
plt.xlabel('Iterations')
plt.ylabel('Sum-squared-error')
plt.title('Adaline - Learning rate 0.0001')
plt.show()
```

The two plots above nicely emphasize the importance of plotting learning curves by illustrating two most common problems with gradient descent:

1. If the learning rate is too large, gradient descent will overshoot the minima and diverge.
2. If the learning rate is too small, the algorithm will require too many epochs to converge and can become trapped in local minima more easily.

Gradient descent is also a good example why feature scaling is important for many machine learning algorithms.
It is not only easier to find an appropriate learning rate if the features are on the same scale, but it also often leads to faster convergence and can prevent the weights from becoming too small (numerical stability).

A common way of feature scaling is standardization

$$\mathbf{x}_{j, std} = \frac{\mathbf{x}_j - \mathbf{\mu}_j}{\mathbf{\sigma}_j}$$

where $\mathbf{\mu}_j$ is the sample mean of the feature $\mathbf{x}_{j}$ and $\mathbf{\sigma}_j$ the standard deviation, respectively. After standardization, the features will have unit variance and are centered around mean zero.

```python
# standardize features
X_std = np.copy(X)
X_std[:,0] = (X[:,0] - X[:,0].mean()) / X[:,0].std()
X_std[:,1] = (X[:,1] - X[:,1].mean()) / X[:,1].std()
```

```python
%matplotlib inline
import matplotlib.pyplot as plt
from mlxtend.evaluate import plot_decision_regions

ada = AdalineGD(epochs=15, eta=0.01)

ada.train(X_std, y)
plot_decision_regions(X_std, y, clf=ada)
plt.title('Adaline - Gradient Descent')
plt.xlabel('sepal length [standardized]')
plt.ylabel('petal length [standardized]')
plt.show()

plt.plot(range(1, len( ada.cost_)+1), ada.cost_, marker='o')
plt.xlabel('Iterations')
plt.ylabel('Sum-squared-error')
plt.show()
```

## Online Learning via Stochastic Gradient Descent

[[back to top](#Sections)]

The previous section was all about "batch" gradient descent learning. The "batch" updates refers to the fact that the cost function is minimized based on the complete training data set. If we think back to the perceptron rule, we remember that it performed the weight update incrementally after each individual training sample. This approach is also called "online" learning, and in fact, this is also how Adaline was first described by Bernard Widrow et al. [[3](#References)]

The process of incrementally updating the weights is also called "stochastic" gradient descent since it approximates the minimization of the cost function. Although the stochastic gradient descent approach might sound inferior to gradient descent due its "stochastic" nature and the "approximated" direction (gradient), it can have certain advantages in practice. Often, stochastic gradient descent converges much faster than gradient descent since the updates are applied immediately after each training sample; stochastic gradient descent is computationally more efficient, especially for very large datasets. Another advantage of online learning is that the classifier can be immediately updated as new training data arrives, e.g., in web applications, and old training data can be discarded if storage is an issue. In large-scale machine learning systems, it is also common practice to use so-called "mini-batches", a compromise with smoother convergence than stochastic gradient descent.

In the interests of completeness let us also implement the stochastic gradient descent Adaline and confirm that it converges on the linearly separable iris dataset.

```python
import numpy as np

class AdalineSGD(object):

    def __init__(self, eta=0.01, epochs=50):
        self.eta = eta
        self.epochs = epochs

    def train(self, X, y, reinitialize_weights=True):

        if reinitialize_weights:
            self.w_ = np.zeros(1 + X.shape[1])
        self.cost_ = []

        for i in range(self.epochs):
            for xi, target in zip(X, y):
                output = self.net_input(xi)
                error = (target - output)
                self.w_[1:] += self.eta * xi.dot(error)
                self.w_[0] += self.eta * error

            cost = ((y - self.activation(X))**2).sum() / 2.0
            self.cost_.append(cost)
        return self

    def net_input(self, X):
        return np.dot(X, self.w_[1:]) + self.w_[0]

    def activation(self, X):
        return self.net_input(X)

    def predict(self, X):
        return np.where(self.activation(X) >= 0.0, 1, -1)
```

One more advice before we let the adaline learn via stochastic gradient descent is to shuffle the training dataset to iterate over the training samples in random order.

We shall note that the "standard" stochastic gradient descent algorithm uses sampling "with replacement," which means that at each iteration, a training sample is chosen randomly from the entire training set. In contrast, sampling "without replacement," which means that each training sample is evaluated exactly once in every epoch, is not only easier to implement but also shows a better performance in empirical comparisons. A more detailed discussion about this topic can be found in Benjamin Recht and Christopher Re's paper *Beneath the valley of the noncommutative arithmetic-geometric mean inequality: conjectures, case-studies, and consequences* [[4](#References)].

```python
ada = AdalineSGD(epochs=15, eta=0.01)

# shuffle data
np.random.seed(123)
idx = np.random.permutation(len(y))
X_shuffled, y_shuffled =  X_std[idx], y[idx]

# train and adaline and plot decision regions
ada.train(X_shuffled, y_shuffled)
plot_decision_regions(X_shuffled, y_shuffled, clf=ada)
plt.title('Adaline - Gradient Descent')
plt.xlabel('sepal length [standardized]')
plt.ylabel('petal length [standardized]')
plt.show()

plt.plot(range(1, len(ada.cost_)+1), ada.cost_, marker='o')
plt.xlabel('Iterations')
plt.ylabel('Sum-squared-error')
plt.show()
```

# What's Next?

[[back to top](#Sections)]

Although we covered many different topics during this article, we just scratched the surface of artificial neurons.

In later articles, we will take a look at different approaches to dynamically adjust the learning rate, the concepts of "One-vs-All" and "One-vs-One" for multi-class classification, regularization to overcome overfitting by introducing additional information, dealing with nonlinear problems and multilayer neural networks, different activation functions for artificial neurons, and related concepts such as logistic regression and support vector machines.

### References

[[back to top](#Sections)]

[1] F. Rosenblatt. The perceptron, a perceiving and recognizing automaton Project Para. Cornell Aeronautical Laboratory, 1957.

[2] W. S. McCulloch and W. Pitts. A logical calculus of the ideas immanent in nervous activity. The bulletin of mathematical biophysics, 5(4):115–133, 1943.

[3] B. Widrow et al. Adaptive ”Adaline” neuron using chemical ”memistors”. Number Technical Report 1553-2. Stanford Electron. Labs., Stanford, CA, October 1960.

[4] B. Recht and C. R ́e. Beneath the valley of the noncommutative arithmetic-geometric mean inequality: conjectures, case-studies, and consequences. arXiv preprint arXiv:1202.4184, 2012.

```python

```


```python
%pylab inline
```

    Populating the interactive namespace from numpy and matplotlib

__END_OF_DEFS__

# Programming ulab

Earlier we have seen, how `ulab`'s functions and methods can be accessed in `micropython`. This last section of the book explains, how these functions are implemented. By the end of this chapter, not only would you be able to extend `ulab`, and write your own `numpy`-compatible functions, but through a deeper understanding of the inner workings of the functions, you would also be able to see what the trade-offs are at the `python` level.

## Code organisation

As mentioned earlier, the `python` functions are organised into sub-modules at the C level. The C sub-modules can be found in `./ulab/code/`.

## The `ndarray` object

### General comments

`ndarrays` are efficient containers of numerical data of the same type (i.e., signed/unsigned chars, signed/unsigned integers or `mp_float_t`s, which, depending on the platform, are either C `float`s, or C `double`s). Beyond storing the actual data in the void pointer `*array`, the type definition has eight additional members (on top of the `base` type). Namely, the `dtype`, which tells us, how the bytes are to be interpreted. Moreover, the `itemsize`, which stores the size of a single entry in the array, `boolean`, an unsigned integer, which determines, whether the arrays is to be treated as a set of Booleans, or as numerical data, `ndim`, the number of dimensions (`uint8_t`), `len`, the length of the array (the number of entries), the shape (`*size_t`), the strides (`*int32_t`). The length is simply the product of the numbers in `shape`.

The type definition is as follows:

```c
typedef struct _ndarray_obj_t {
    mp_obj_base_t base;
    uint8_t dtype;
    uint8_t itemsize;
    uint8_t boolean;
    uint8_t ndim;
    size_t len;
    size_t shape[ULAB_MAX_DIMS];
    int32_t strides[ULAB_MAX_DIMS];
    void *array;
} ndarray_obj_t;
```

### Memory layout

The values of an `ndarray` are stored in a contiguous segment in the RAM. The `ndarray` can be dense, meaning that all numbers in the linear memory segment belong to a linar combination of coordinates, and it can also be sparse, i.e., some elements of the linear storage space will be skipped, when the elements of the tensor are traversed.

In the RAM, the position of the item $M(n_1, n_2, ..., n_{k-1}, n_k)$ in a dense tensor of rank $k$ is given by the linear combination

\begin{equation}
P(n_1, n_2, ..., n_{k-1}, n_k) = n_1 s_1 + n_2 s_2 + ... + n_{k-1}s_{k-1} + n_ks_k = \sum_{i=1}^{k}n_is_i
\end{equation}
where $s_i$ are the strides of the tensor, defined as

\begin{equation}
s_i = \prod_{j=i+1}^k l_j
\end{equation}

where $l_j$ is length of the tensor along the $j$th axis. When the tensor is sparse (e.g., when the tensor is sliced), the strides along a particular axis will be multiplied by a non-zero integer. If this integer is different to $\pm 1$, the linear combination above cannot access all elements in the RAM, i.e., some numbers will be skipped. Note that $|s_1| > |s_2| > ... > |s_{k-1}| > |s_k|$, even if the tensor is sparse. The statement is trivial for dense tensors, and it follows from the definition of $s_i$. For sparse tensors, a slice cannot have a step larger than the shape along that axis. But for dense tensors, $s_i/s_{i+1} = l_i$.

When creating a *view*, we simply re-calculate the `strides`, and re-set the `*array` pointer.

## Iterating over elements of a tensor

The `shape` and `strides` members of the array tell us how we have to move our pointer, when we want to read out the numbers. For technical reasons that will become clear later, the numbers in `shape` and in `strides` are aligned to the right, and begin on the right hand side, i.e., if the number of possible dimensions is `ULAB_MAX_DIMS`, then `shape[ULAB_MAX_DIMS-1]` is the length of the last axis, `shape[ULAB_MAX_DIMS-2]` is the length of the last but one axis, and so on. If the number of actual dimensions, `ndim < ULAB_MAX_DIMS`, the first `ULAB_MAX_DIMS - ndim` entries in `shape` and `strides` will be equal to zero, but they could, in fact, be assigned any value, because these will never be accessed in an operation.

With this definition of the strides, the linear combination in $P(n_1, n_2, ..., n_{k-1}, n_k)$ is a one-to-one mapping from the space of tensor coordinates, $(n_1, n_2, ..., n_{k-1}, n_k)$, and the coordinate in the linear array, $n_1s_1 + n_2s_2 + ... + n_{k-1}s_{k-1} + n_ks_k$, i.e., no two distinct sets of coordinates will result in the same position in the linear array.

Since the `strides` are given in terms of bytes, when we iterate over an array, the void data pointer is usually cast to `uint8_t`, and the values are converted using the proper data type stored in `ndarray->dtype`. However, there might be cases, when it makes perfect sense to cast `*array` to a different type, in which case the `strides` have to be re-scaled by the value of `ndarray->itemsize`.

### Iterating using the unwrapped loops

The following macro definition is taken from [vector.h](https://github.com/v923z/micropython-ulab/blob/master/code/numpy/vector/vector.h), and demonstrates, how we can iterate over a single array in four dimensions.

```c
#define ITERATE_VECTOR(type, array, source, sarray) do {
    size_t i=0;
    do {
        size_t j = 0;
        do {
            size_t k = 0;
            do {
                size_t l = 0;
                do {
                    *(array)++ = f(*((type *)(sarray)));
                    (sarray) += (source)->strides[ULAB_MAX_DIMS - 1];
                    l++;
                } while(l < (source)->shape[ULAB_MAX_DIMS-1]);
                (sarray) -= (source)->strides[ULAB_MAX_DIMS - 1] * (source)->shape[ULAB_MAX_DIMS-1];
                (sarray) += (source)->strides[ULAB_MAX_DIMS - 2];
                k++;
            } while(k < (source)->shape[ULAB_MAX_DIMS-2]);
            (sarray) -= (source)->strides[ULAB_MAX_DIMS - 2] * (source)->shape[ULAB_MAX_DIMS-2];
            (sarray) += (source)->strides[ULAB_MAX_DIMS - 3];
            j++;
        } while(j < (source)->shape[ULAB_MAX_DIMS-3]);
        (sarray) -= (source)->strides[ULAB_MAX_DIMS - 3] * (source)->shape[ULAB_MAX_DIMS-3];
        (sarray) += (source)->strides[ULAB_MAX_DIMS - 4];
        i++;
    } while(i < (source)->shape[ULAB_MAX_DIMS-4]);
} while(0)
```

We start with the innermost loop, the one recursing `l`. `array` is already of type `mp_float_t`, while the source array, `sarray`, has been cast to `uint8_t` in the calling function. The numbers contained in `sarray` have to be read out in the proper type dictated by `ndarray->dtype`. This is what happens in the statement `*((type *)(sarray))`, and this number is then fed into the function `f`. Vectorised mathematical functions produce *dense* arrays, and for this reason, we can simply advance the `array` pointer.

The advancing of the `sarray` pointer is a bit more involving: first, in the innermost loop, we simply move forward by the amount given by the last stride, which is `(source)->strides[ULAB_MAX_DIMS - 1]`, because the `shape` and the `strides` are aligned to the right. We move the pointer as many times as given by `(source)->shape[ULAB_MAX_DIMS-1]`, which is the length of the very last axis. Hence the the structure of the loop

```c
    size_t l = 0;
    do {
        ...
        l++;
    } while(l < (source)->shape[ULAB_MAX_DIMS-1]);

```
Once we have exhausted the last axis, we have to re-wind the pointer, and advance it by an amount given by the last but one stride. Keep in mind that in the the innermost loop we moved our pointer `(source)->shape[ULAB_MAX_DIMS-1]` times by `(source)->strides[ULAB_MAX_DIMS - 1]`, i.e., we re-wind it by moving it backwards by `(source)->strides[ULAB_MAX_DIMS - 1] * (source)->shape[ULAB_MAX_DIMS-1]`. In the next step, we move forward by `(source)->strides[ULAB_MAX_DIMS - 2]`, which is the last but one stride.

```c
    (sarray) -= (source)->strides[ULAB_MAX_DIMS - 1] * (source)->shape[ULAB_MAX_DIMS-1];
    (sarray) += (source)->strides[ULAB_MAX_DIMS - 2];

```

This pattern must be repeated for each axis of the array, and this is how we arrive at the four nested loops listed above.

### Re-winding arrays by means of a function

In addition to un-wrapping the iteration loops by means of macros, there is another way of traversing all elements of a tensor: we note that, since $|s_1| > |s_2| > ... > |s_{k-1}| > |s_k|$, $P(n1, n2, ..., n_{k-1}, n_k)$ changes most slowly in the last coordinate. Hence, if we start from the very beginning, ($n_i = 0$ for all $i$), and walk along the linear RAM segment, we increment the value of $n_k$ as long as $n_k < l_k$. Once $n_k = l_k$, we have to reset $n_k$ to 0, and increment $n_{k-1}$ by one. After each such round, $n_{k-1}$ will be incremented by one, as long as $n_{k-1} < l_{k-1}$. Once $n_{k-1} = l_{k-1}$, we reset both $n_k$, and $n_{k-1}$ to 0, and increment $n_{k-2}$ by one.

Rewinding the arrays in this way is implemented in the function `ndarray_rewind_array` in [ndarray.c](https://github.com/v923z/micropython-ulab/blob/master/code/ndarray.c).

```c
void ndarray_rewind_array(uint8_t ndim, uint8_t *array, size_t *shape, int32_t *strides, size_t *coords) {
    // resets the data pointer of a single array, whenever an axis is full
    // since we always iterate over the very last axis, we have to keep track of
    // the last ndim-2 axes only
    array -= shape[ULAB_MAX_DIMS - 1] * strides[ULAB_MAX_DIMS - 1];
    array += strides[ULAB_MAX_DIMS - 2];
    for(uint8_t i=1; i < ndim-1; i++) {
        coords[ULAB_MAX_DIMS - 1 - i] += 1;
        if(coords[ULAB_MAX_DIMS - 1 - i] == shape[ULAB_MAX_DIMS - 1 - i]) { // we are at a dimension boundary
            array -= shape[ULAB_MAX_DIMS - 1 - i] * strides[ULAB_MAX_DIMS - 1 - i];
            array += strides[ULAB_MAX_DIMS - 2 - i];
            coords[ULAB_MAX_DIMS - 1 - i] = 0;
            coords[ULAB_MAX_DIMS - 2 - i] += 1;
        } else { // coordinates can change only, if the last coordinate changes
            return;
        }
    }
}
```

and the function would be called as in the snippet below. Note that the innermost loop is factored out, so that we can save the `if(...)` statement for the last axis.

```c
    size_t *coords = ndarray_new_coords(results->ndim);
    for(size_t i=0; i < results->len/results->shape[ULAB_MAX_DIMS -1]; i++) {
        size_t l = 0;
        do {
            ...
            l++;
        } while(l < results->shape[ULAB_MAX_DIMS - 1]);
        ndarray_rewind_array(results->ndim, array, results->shape, strides, coords);
    } while(0)

```

The advantage of this method is that the implementation is independent of the number of dimensions: the iteration requires more or less the same flash space for 2 dimensions as for 22. However, the price we have to pay for this convenience is the extra function call.

## Iterating over two ndarrays simultaneously: broadcasting

Whenever we invoke a binary operator, call a function with two arguments of `ndarray` type, or assign something to an `ndarray`, we have to iterate over two views at the same time. The task is trivial, if the two `ndarray`s in question have the same shape (but not necessarily the same set of strides), because in this case, we can still iterate in the same loop. All that happens is that we move two data pointers in sync.

The problem becomes a bit more involving, when the shapes of the two `ndarray`s are not identical. For such cases, `numpy` defines so-called broadcasting, which boils down to two rules.

1. The shapes in the tensor with lower rank has to be prepended with axes of size 1 till the two ranks become equal.
2. Along all axes the two tensors should have the same size, or one of the sizes must be 1.

If, after applying the first rule the second is not satisfied, the two `ndarray`s cannot be broadcast together.

Now, let us suppose that we have two compatible `ndarray`s, i.e., after applying the first rule, the second is satisfied. How do we iterate over the elements in the tensors?

We should recall, what exactly we do, when iterating over a single array: normally, we move the data pointer by the last stride, except, when we arrive at a dimension boundary (when the last axis is exhausted). At that point, we move the pointer by an amount dictated by the strides. And this is the key: *dictated by the strides*. Now, if we have two arrays that are originally not compatible, we define new strides for them, and use these in the iteration. With that, we are back to the case, where we had two compatible arrays.

Now, let us look at the second broadcasting rule: if the two arrays have the same size, we take both `ndarray`s' strides along that axis. If, on the other hand, one of the `ndarray`s is of length 1 along one of its axes, we set the corresponding strides to 0. This will ensure that that data pointer is not moved, when we iterate over both `ndarray`s at the same time.

Thus, in order to implement broadcasting, we first have to check, whether the two above-mentioned rules can be satisfied, and if so, we have to find the two new sets strides.

The `ndarray_can_broadcast` function from [ndarray.c](https://github.com/v923z/micropython-ulab/blob/master/code/ndarray.c) takes two `ndarray`s, and returns `true`, if the two arrays can be broadcast together. At the same time, it also calculates new strides for the two arrays, so that they can be iterated over at the same time.

```c
bool ndarray_can_broadcast(ndarray_obj_t *lhs, ndarray_obj_t *rhs, uint8_t *ndim, size_t *shape, int32_t *lstrides, int32_t *rstrides) {
    // returns True or False, depending on, whether the two arrays can be broadcast together
    // numpy's broadcasting rules are as follows:
    //
    // 1. the two shapes are either equal
    // 2. one of the shapes is 1
    memset(lstrides, 0, sizeof(size_t)*ULAB_MAX_DIMS);
    memset(rstrides, 0, sizeof(size_t)*ULAB_MAX_DIMS);
    lstrides[ULAB_MAX_DIMS - 1] = lhs->strides[ULAB_MAX_DIMS - 1];
    rstrides[ULAB_MAX_DIMS - 1] = rhs->strides[ULAB_MAX_DIMS - 1];
    for(uint8_t i=ULAB_MAX_DIMS; i > 0; i--) {
        if((lhs->shape[i-1] == rhs->shape[i-1]) || (lhs->shape[i-1] == 0) || (lhs->shape[i-1] == 1) ||
        (rhs->shape[i-1] == 0) || (rhs->shape[i-1] == 1)) {
            shape[i-1] = MAX(lhs->shape[i-1], rhs->shape[i-1]);
            if(shape[i-1] > 0) (*ndim)++;
            if(lhs->shape[i-1] < 2) {
                lstrides[i-1] = 0;
            } else {
                lstrides[i-1] = lhs->strides[i-1];
            }
            if(rhs->shape[i-1] < 2) {
                rstrides[i-1] = 0;
            } else {
                rstrides[i-1] = rhs->strides[i-1];
            }
        } else {
            return false;
        }
    }
    return true;
}
```

A good example of how the function would be called can be found in [vector.c](https://github.com/v923z/micropython-ulab/blob/master/code/numpy/vector/vector.c), in the `vector_arctan2` function:

```c
mp_obj_t vectorise_arctan2(mp_obj_t y, mp_obj_t x) {
    ...
    uint8_t ndim = 0;
    size_t *shape = m_new(size_t, ULAB_MAX_DIMS);
    int32_t *xstrides = m_new(int32_t, ULAB_MAX_DIMS);
    int32_t *ystrides = m_new(int32_t, ULAB_MAX_DIMS);
    if(!ndarray_can_broadcast(ndarray_x, ndarray_y, &ndim, shape, xstrides, ystrides)) {
        mp_raise_ValueError(translate("operands could not be broadcast together"));
        m_del(size_t, shape, ULAB_MAX_DIMS);
        m_del(int32_t, xstrides, ULAB_MAX_DIMS);
        m_del(int32_t, ystrides, ULAB_MAX_DIMS);
    }

    uint8_t *xarray = (uint8_t *)ndarray_x->array;
    uint8_t *yarray = (uint8_t *)ndarray_y->array;

    ndarray_obj_t *results = ndarray_new_dense_ndarray(ndim, shape, NDARRAY_FLOAT);
    mp_float_t *rarray = (mp_float_t *)results->array;
    ...
```

After the new strides have been calculated, the iteration loop is identical to what we discussed in the previous section.

## Contracting an `ndarray`

There are many operations that reduce the number of dimensions of an `ndarray` by 1, i.e., that remove an axis from the tensor. The drill is the same as before, with the exception that first we have to remove the `strides` and `shape` that corresponds to the axis along which we intend to contract. The `numerical_reduce_axes` function from [numerical.c](https://github.com/v923z/micropython-ulab/blob/master/code/numerical/numerical.c) does that.

```c
static void numerical_reduce_axes(ndarray_obj_t *ndarray, int8_t axis, size_t *shape, int32_t *strides) {
    // removes the values corresponding to a single axis from the shape and strides array
    uint8_t index = ULAB_MAX_DIMS - ndarray->ndim + axis;
    if((ndarray->ndim == 1) && (axis == 0)) {
        index = 0;
        shape[ULAB_MAX_DIMS - 1] = 0;
        return;
    }
    for(uint8_t i = ULAB_MAX_DIMS - 1; i > 0; i--) {
        if(i > index) {
            shape[i] = ndarray->shape[i];
            strides[i] = ndarray->strides[i];
        } else {
            shape[i] = ndarray->shape[i-1];
            strides[i] = ndarray->strides[i-1];
        }
    }
}
```

Once the reduced `strides` and `shape` are known, we place the axis in question in the innermost loop, and wrap it with the loops, whose coordinates are in the `strides`, and `shape` arrays. The `RUN_STD` macro from [numerical.h](https://github.com/v923z/micropython-ulab/blob/master/code/numpy/numerical/numerical.h) is a good example. The macro is expanded in the `numerical_sum_mean_std_ndarray` function.

```c
static mp_obj_t numerical_sum_mean_std_ndarray(ndarray_obj_t *ndarray, mp_obj_t axis, uint8_t optype, size_t ddof) {
    uint8_t *array = (uint8_t *)ndarray->array;
    size_t *shape = m_new(size_t, ULAB_MAX_DIMS);
    memset(shape, 0, sizeof(size_t)*ULAB_MAX_DIMS);
    int32_t *strides = m_new(int32_t, ULAB_MAX_DIMS);
    memset(strides, 0, sizeof(uint32_t)*ULAB_MAX_DIMS);

    int8_t ax = mp_obj_get_int(axis);
    if(ax < 0) ax += ndarray->ndim;
    if((ax < 0) || (ax > ndarray->ndim - 1)) {
        mp_raise_ValueError(translate("index out of range"));
    }
    numerical_reduce_axes(ndarray, ax, shape, strides);
    uint8_t index = ULAB_MAX_DIMS - ndarray->ndim + ax;
    ndarray_obj_t *results = NULL;
    uint8_t *rarray = NULL;
    ...

```
Here is the macro for the three-dimensional case:

```c
#define RUN_STD(ndarray, type, array, results, r, shape, strides, index, div) do {
    size_t k = 0;
    do {
        size_t l = 0;
        do {
            RUN_STD1((ndarray), type, (array), (results), (r), (index), (div));
            (array) -= (ndarray)->strides[(index)] * (ndarray)->shape[(index)];
            (array) += (strides)[ULAB_MAX_DIMS - 1];
            l++;
        } while(l < (shape)[ULAB_MAX_DIMS - 1]);
        (array) -= (strides)[ULAB_MAX_DIMS - 2] * (shape)[ULAB_MAX_DIMS-2];
        (array) += (strides)[ULAB_MAX_DIMS - 3];
        k++;
    } while(k < (shape)[ULAB_MAX_DIMS - 2]);
} while(0)
```
In `RUN_STD`, we simply move our pointers; the calculation itself happens in the `RUN_STD1` macro below. (Note that this is the implementation of the numerically stable Welford algorithm.)

```c
#define RUN_STD1(ndarray, type, array, results, r, index, div)
({
    mp_float_t M, m, S = 0.0, s = 0.0;
    M = m = *(mp_float_t *)((type *)(array));
    for(size_t i=1; i < (ndarray)->shape[(index)]; i++) {
        (array) += (ndarray)->strides[(index)];
        mp_float_t value = *(mp_float_t *)((type *)(array));
        m = M + (value - M) / (mp_float_t)i;
        s = S + (value - M) * (value - m);
        M = m;
        S = s;
    }
    (array) += (ndarray)->strides[(index)];
    *(r)++ = MICROPY_FLOAT_C_FUN(sqrt)((ndarray)->shape[(index)] * s / (div));
})
```

## Upcasting

When in an operation the `dtype`s of two arrays are different, the result's `dtype` will be decided by the following upcasting rules:

1. Operations with two `ndarray`s of the same `dtype` preserve their `dtype`, even when the results overflow.

2. if either of the operands is a float, the result automatically becomes a float

3. otherwise

    - `uint8` + `int8` => `int16`,
    - `uint8` + `int16` => `int16`
    - `uint8` + `uint16` => `uint16`

    - `int8` + `int16` => `int16`
    - `int8` + `uint16` => `uint16` (in numpy, the result is a `int32`)

    - `uint16` + `int16` => `float` (in numpy, the result is a `int32`)

4. When one operand of a binary operation is a generic scalar `micropython` variable, i.e., `mp_obj_int`, or `mp_obj_float`, it will be converted to a linear array of length 1, and with the smallest `dtype` that can accommodate the variable in question. After that the broadcasting rules apply, as described in the section [Iterating over two ndarrays simultaneously: broadcasting](#Iterating_over_two_ndarrays_simultaneously:_broadcasting)

Upcasting is resolved in place, wherever it is required. Notable examples can be found in [ndarray_operators.c](https://github.com/v923z/micropython-ulab/blob/master/code/ndarray_operators.c)

## Slicing and indexing

An `ndarray` can be indexed with three types of objects: integer scalars, slices, and another `ndarray`, whose elements are either integer scalars, or Booleans. Since slice and integer indices can be thought of as modifications of the `strides`, these indices return a view of the `ndarray`. This statement does not hold for `ndarray` indices, and therefore, the return a copy of the array.

## Extending ulab

The `user` module is disabled by default, as can be seen from the last couple of lines of [ulab.h](https://github.com/v923z/micropython-ulab/blob/master/code/ulab.h)

```c
// user-defined module
#ifndef ULAB_USER_MODULE
#define ULAB_USER_MODULE                (0)
#endif
```

The module contains a very simple function, `user_dummy`, and this function is bound to the module itself. In other words, even if the module is enabled, one has to `import`:

```python

import ulab
from ulab import user

user.dummy_function(2.5)
```
which should just return 5.0. Even if `numpy`-compatibility is required (i.e., if most functions are bound at the top level to `ulab` directly), having to `import` the module has a great advantage. Namely, only the [user.h](https://github.com/v923z/micropython-ulab/blob/master/code/user/user.h) and [user.c](https://github.com/v923z/micropython-ulab/blob/master/code/user/user.c) files have to be modified, thus it should be relatively straightforward to update your local copy from [github](https://github.com/v923z/micropython-ulab/blob/master/).

Now, let us see, how we can add a more meaningful function.

## Creating a new ndarray

In the [General comments](#General_comments) sections we have seen the type definition of an `ndarray`. This structure can be generated by means of a couple of functions listed in [ndarray.c](https://github.com/v923z/micropython-ulab/blob/master/code/ndarray.c).

### ndarray_new_ndarray

The `ndarray_new_ndarray` functions is called by all other array-generating functions. It takes the number of dimensions, `ndim`, a `uint8_t`, the `shape`, a pointer to `size_t`, the `strides`, a pointer to `int32_t`, and `dtype`, another `uint8_t` as its arguments, and returns a new array with all entries initialised to 0.

Assuming that `ULAB_MAX_DIMS > 2`, a new dense array of dimension 3, of `shape` (3, 4, 5), of `strides` (1000, 200, 10), and `dtype` `uint16_t` can be generated by the following instructions

```c
size_t *shape = m_new(size_t, ULAB_MAX_DIMS);
shape[ULAB_MAX_DIMS - 1] = 5;
shape[ULAB_MAX_DIMS - 2] = 4;
shape[ULAB_MAX_DIMS - 3] = 3;

int32_t *strides = m_new(int32_t, ULAB_MAX_DIMS);
strides[ULAB_MAX_DIMS - 1] = 10;
strides[ULAB_MAX_DIMS - 2] = 200;
strides[ULAB_MAX_DIMS - 3] = 1000;

ndarray_obj_t *new_ndarray = ndarray_new_ndarray(3, shape, strides, NDARRAY_UINT16);
```

### ndarray_new_dense_ndarray

The functions simply calculates the `strides` from the `shape`, and calls `ndarray_new_ndarray`. Assuming that `ULAB_MAX_DIMS > 2`, a new dense array of dimension 3, of `shape` (3, 4, 5), and `dtype` `mp_float_t` can be generated by the following instructions

```c
size_t *shape = m_new(size_t, ULAB_MAX_DIMS);
shape[ULAB_MAX_DIMS - 1] = 5;
shape[ULAB_MAX_DIMS - 2] = 4;
shape[ULAB_MAX_DIMS - 3] = 3;

ndarray_obj_t *new_ndarray = ndarray_new_dense_ndarray(3, shape, NDARRAY_FLOAT);
```

### ndarray_new_linear_array

Since the dimensions of a linear array are known (1), the `ndarray_new_linear_array` takes the `length`, a `size_t`, and the `dtype`, an `uint8_t`. Internally, `ndarray_new_linear_array` generates the `shape` array, and calls `ndarray_new_dense_array` with `ndim = 1`.

A linear array of length 100, and `dtype` `uint8` could be created by the function call

```c
ndarray_obj_t *new_ndarray = ndarray_new_linear_array(100, NDARRAY_UINT8)
```

### ndarray_new_ndarray_from_tuple

This function takes a `tuple`, which should hold the lengths of the axes (in other words, the `shape`), and the `dtype`, and calls internally `ndarray_new_dense_array`. A new `ndarray` can be generated by calling

```c
ndarray_obj_t *new_ndarray = ndarray_new_ndarray_from_tuple(shape, NDARRAY_FLOAT);
```
where `shape` is a tuple.

### ndarray_new_view

This function crates a *view*, and takes the source, an `ndarray`, the number of dimensions, an `uint8_t`, the `shape`, a pointer to `size_t`, the `strides`, a pointer to `int32_t`, and the offset, an `int32_t` as arguments. The offset is the number of bytes by which the void `array` pointer is shifted. E.g., the `python` statement

```python
a = np.array([0, 1, 2, 3, 4, 5], dtype=uint8)
b = a[1::2]
```

produces the array

```python
array([1, 3, 5], dtype=uint8)
```
which holds its data at position `x0 + 1`, if `a`'s pointer is at `x0`. In this particular case, the offset is 1.

The array `b` from the example above could be generated as

```c
size_t *shape = m_new(size_t, ULAB_MAX_DIMS);
shape[ULAB_MAX_DIMS - 1] = 3;

int32_t *strides = m_new(int32_t, ULAB_MAX_DIMS);
strides[ULAB_MAX_DIMS - 1] = 2;

int32_t offset = 1;
uint8_t ndim = 1;

ndarray_obj_t *new_ndarray = ndarray_new_view(ndarray_a, ndim, shape, strides, offset);
```

### ndarray_copy_array

The `ndarray_copy_array` function can be used for copying the contents of an array. Note that the target array has to be created beforehand. E.g., a one-to-one copy can be gotten by

```c
ndarray_obj_t *new_ndarray = ndarray_new_ndarray(source->ndim, source->shape, source->strides, source->dtype);
ndarray_copy_array(source, new_ndarray);

```
Note that the function cannot be used for forcing type conversion, i.e., the input and output types must be identical, because the function simply calls the `memcpy` function. On the other hand, the input and output `strides` do not necessarily have to be equal.

### ndarray_copy_view

The `ndarray_obj_t *new_ndarray = ...` instruction can be saved by calling the `ndarray_copy_view` function with the single `source` argument.

## Accessing data in the ndarray

Having seen, how arrays can be generated and copied, it is time to look at how the data in an `ndarray` can be accessed and modified.

For starters, let us suppose that the object in question comes from the user (i.e., via the `micropython` interface), First, we have to acquire a pointer to the `ndarray` by calling

```c
ndarray_obj_t *ndarray = MP_OBJ_TO_PTR(object_in);
```

If it is not clear, whether the object is an `ndarray` (e.g., if we want to write a function that can take `ndarray`s, and other iterables as its argument), we find this out by evaluating

```c
MP_OBJ_IS_TYPE(object_in, &ulab_ndarray_type)
```
which should return `true`. Once the pointer is at our disposal, we can get a pointer to the underlying numerical array as discussed earlier, i.e.,

```c
uint8_t *array = (uint8_t *)ndarray->array;
```

If you need to find out the `dtype` of the array, you can get it by accessing the `dtype` member of the `ndarray`, i.e.,

```c
ndarray->dtype
```
should be equal to `B`, `b`, `H`, `h`, or `f`. The size of a single item is stored in the `itemsize` member. This number should be equal to 1, if the `dtype` is `B`, or `b`, 2, if the `dtype` is `H`, or `h`, 4, if the `dtype` is `f`, and 8 for `d`.

## Boilerplate

In the next section, we will construct a function that generates the element-wise square of a dense array, otherwise, raises a `TypeError` exception. Dense arrays can easily be iterated over, since we do not have to care about the `shape` and the `strides`. If the array is sparse, the section [Iterating over elements of a tensor](#Iterating-over-elements-of-a-tensor) should contain hints as to how the iteration can be implemented.

The function is listed under [user.c](https://github.com/v923z/micropython-ulab/tree/master/code/user/). The `user` module is bound to `ulab` in [ulab.c](https://github.com/v923z/micropython-ulab/tree/master/code/ulab.c) in the lines

```c
    #if ULAB_USER_MODULE
        { MP_ROM_QSTR(MP_QSTR_user), MP_ROM_PTR(&ulab_user_module) },
    #endif
```
which assumes that at the very end of [ulab.h](https://github.com/v923z/micropython-ulab/tree/master/code/ulab.h) the

```c
// user-defined module
#ifndef ULAB_USER_MODULE
#define ULAB_USER_MODULE                (1)
#endif
```
constant has been set to 1. After compilation, you can call a particular `user` function in `python` by importing the module first, i.e.,

```python
from ulab import numpy as np
from ulab import user

user.some_function(...)
```

This separation of user-defined functions from the rest of the code ensures that the integrity of the main module and all its functions are always preserved. Even in case of a catastrophic failure, you can exclude the `user` module, and start over.

And now the function:

```c
static mp_obj_t user_square(mp_obj_t arg) {
    // the function takes a single dense ndarray, and calculates the
    // element-wise square of its entries

    // raise a TypeError exception, if the input is not an ndarray
    if(!MP_OBJ_IS_TYPE(arg, &ulab_ndarray_type)) {
        mp_raise_TypeError(translate("input must be an ndarray"));
    }
    ndarray_obj_t *ndarray = MP_OBJ_TO_PTR(arg);

    // make sure that the input is a dense array
    if(!ndarray_is_dense(ndarray)) {
        mp_raise_TypeError(translate("input must be a dense ndarray"));
    }

    // if the input is a dense array, create `results` with the same number of
    // dimensions, shape, and dtype
    ndarray_obj_t *results = ndarray_new_dense_ndarray(ndarray->ndim, ndarray->shape, ndarray->dtype);

    // since in a dense array the iteration over the elements is trivial, we
    // can cast the data arrays ndarray->array and results->array to the actual type
    if(ndarray->dtype == NDARRAY_UINT8) {
        uint8_t *array = (uint8_t *)ndarray->array;
        uint8_t *rarray = (uint8_t *)results->array;
        for(size_t i=0; i < ndarray->len; i++, array++) {
            *rarray++ = (*array) * (*array);
        }
    } else if(ndarray->dtype == NDARRAY_INT8) {
        int8_t *array = (int8_t *)ndarray->array;
        int8_t *rarray = (int8_t *)results->array;
        for(size_t i=0; i < ndarray->len; i++, array++) {
            *rarray++ = (*array) * (*array);
        }
    } else if(ndarray->dtype == NDARRAY_UINT16) {
        uint16_t *array = (uint16_t *)ndarray->array;
        uint16_t *rarray = (uint16_t *)results->array;
        for(size_t i=0; i < ndarray->len; i++, array++) {
            *rarray++ = (*array) * (*array);
        }
    } else if(ndarray->dtype == NDARRAY_INT16) {
        int16_t *array = (int16_t *)ndarray->array;
        int16_t *rarray = (int16_t *)results->array;
        for(size_t i=0; i < ndarray->len; i++, array++) {
            *rarray++ = (*array) * (*array);
        }
    } else { // if we end up here, the dtype is NDARRAY_FLOAT
        mp_float_t *array = (mp_float_t *)ndarray->array;
        mp_float_t *rarray = (mp_float_t *)results->array;
        for(size_t i=0; i < ndarray->len; i++, array++) {
            *rarray++ = (*array) * (*array);
        }
    }
    // at the end, return a micropython object
    return MP_OBJ_FROM_PTR(results);
}

```

To summarise, the steps for *implementing* a function are

1. If necessary, inspect the type of the input object, which is always a `mp_obj_t` object
2. If the input is an `ndarray_obj_t`, acquire a pointer to it by calling `ndarray_obj_t *ndarray = MP_OBJ_TO_PTR(arg);`
3. Create a new array, or modify the existing one; get a pointer to the data by calling `uint8_t *array = (uint8_t *)ndarray->array;`, or something equivalent
4. Once the new data have been calculated, return a `micropython` object by calling `MP_OBJ_FROM_PTR(...)`.

The listing above contains the implementation of the function, but as such, it cannot be called from `python`:
it still has to be bound to the name space. This we do by first defining a function object in

```c
MP_DEFINE_CONST_FUN_OBJ_1(user_square_obj, user_square);

```

`micropython` defines a number of `MP_DEFINE_CONST_FUN_OBJ_N` macros in [obj.h](https://github.com/micropython/micropython/blob/master/py/obj.h). `N` is always the number of arguments the function takes. We had a function definition `static mp_obj_t user_square(mp_obj_t arg)`, i.e., we dealt with a single argument.

Finally, we have to bind this function object in the globals table of the `user` module:

```c
STATIC const mp_rom_map_elem_t ulab_user_globals_table[] = {
    { MP_OBJ_NEW_QSTR(MP_QSTR___name__), MP_OBJ_NEW_QSTR(MP_QSTR_user) },
    { MP_OBJ_NEW_QSTR(MP_QSTR_square), (mp_obj_t)&user_square_obj },
};
```

Thus, the three steps required for the definition of a user-defined function are

1. The low-level implementation of the function itself
2. The definition of a function object by calling MP_DEFINE_CONST_FUN_OBJ_N()
3. Binding this function object to the namespace in the `ulab_user_globals_table[]`


##### Copyright 2022 The Cirq Developers

```
# @title Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
```

# Parameter Sweeps

    View on QuantumAI

    Run in Google Colab

    View source on GitHub

    Download notebook

```
try:
    import cirq
except ImportError:
    print("installing cirq...")
    !pip install --quiet cirq
    print("installed cirq.")
    import cirq
```

## Concept of Circuit Parameterization and Sweeps

Suppose you have a quantum circuit and in this circuit there is a gate with some parameter. You might wish to run this circuit for different values of this parameter.  An example of this type of circuit is a Rabi flop experiment. This experiment runs a set of quantum computations which 1) starts in  $|0\rangle$ state, 2) rotates the state by $\theta$ about the $x$ axis, i.e. applies the gate $\exp(i \theta X)$, and 3) measures the state in the computational basis.  Running this experiment for multiple values of $\theta$, and plotting the probability of observing a $|1\rangle$ outcome yields the quintessential $\cos^2$ probability distribution as a function of the parameter $\theta$.  To support this type of experiment, Cirq provides the concept of parameterized circuits and parameter sweeps.

The next cell illustrates parameter sweeps with a simple example.  Suppose you want to compare two quantum circuits that are identical except for a single exponentiated `cirq.Z` gate.

```
q0 = cirq.LineQubit(0)

circuit1 = cirq.Circuit([cirq.H(q0), cirq.Z(q0)**0.5, cirq.H(q0), cirq.measure(q0)])
print(f"circuit1:\n{circuit1}")

circuit2 = cirq.Circuit([cirq.H(q0), cirq.Z(q0)**0.25, cirq.H(q0), cirq.measure(q0)])
print(f"circuit2:\n{circuit2}")
```

You could run these circuits separately (either on hardware or in simulation), and collect statistics on the results of these circuits. However parameter sweeps can do this in a cleaner and more perfomant manner.

First define a parameter, and construct a circuit that depends on this parameter. Cirq uses [SymPy](https://www.sympy.org/en/index.html){:external}, a symbolic mathematics package, to define parameters. In this example the Sympy parameter is `theta`, which is used to construct a parameterized circuit.

```
import sympy

theta = sympy.Symbol("theta")

circuit = cirq.Circuit([cirq.H(q0), cirq.Z(q0)**theta, cirq.H(q0), cirq.measure(q0)])
print(f"circuit:\n{circuit}")
```

Notice now that the circuit contains a `cirq.Z` gate that is raised to a power, but this power is the parameter `theta`.  This is a "parameterized circuit".  An equivalent way to construct this circuit, where the parameter is actually a parameter in the gate constructor's arguments, is:

```
circuit = cirq.Circuit(
    cirq.H(q0), cirq.ZPowGate(exponent=theta)(q0), cirq.H(q0), cirq.measure(q0)
)
print(f"circuit:\n{circuit}")
```

Note: You can check whether an object in Cirq is parameterized using `cirq.is_parameterized`:

```
cirq.is_parameterized(circuit)
```

Parameterized circuits are just like normal circuits; they just aren't defined in terms of gates that you can actually run on a quantum computer without the additional information about the values of the parameters.  Following the example above, you can generate the two circuits (`circuit1` and `circuit2`) by using `cirq.resolve_parameter` and supplying the values that you want the parameter(s) to take:

```
# circuit1 has theta = 0.5
cirq.resolve_parameters(circuit, {"theta": 0.5})
# circuit2 has theta = 0.25
cirq.resolve_parameters(circuit, {"theta": 0.25})
```

More interestingly, you can combine parameterized circuits with a list of parameter assignments when doing things like running circuits or simulating them.  These lists of parameter assignements are called "sweeps".  For example you can use a simulator's `run_sweep` method to run simulations for the parameters corresponding to the two circuits defined above.

```
sim = cirq.Simulator()
results = sim.run_sweep(circuit, repetitions=25, params=[{"theta": 0.5}, {"theta": 0.25}])
for result in results:
    print(f"param: {result.params}, result: {result}")
```

To recap, you can construct parameterized circuits that depend on parameters that have not yet been assigned a value.  These parameterized circuits can then be resolved to circuits with actual values via a dictionary that maps the sympy variable name to the value that parameter should take. You can also construct lists of dictionaries of parameter assignments, called sweeps, and pass this to many functions in Cirq that use circuits to do an action (such as `simulate` or `run`).  For each of the elements in the sweep, the function will execute using the parameters as described by the element.

## Constructing Sweeps

The previous example constructed a sweep by simply constructing a list of parameter assignments, `[{"theta": 0.5}, {"theta": 0.25}]`.  Cirq also provides other ways to construct sweeps.

One useful method for constructing parameter sweeps is `cirq.Linspace` which creates a sweep over a list of equally spaced elements.

```
# Create a sweep over 5 equally spaced values from 0 to 2.5.
params = cirq.Linspace(key="theta", start=0, stop=2.5, length=5)
for param in params:
    print(param)
```

Note: The `Linspace` sweep is composed of `cirq.ParamResolver` instances instead of simple dictionaries. However, you can think of them as effectively the same for most use cases.

If you need to explicitly and individually specify each parameter resolution, you can do it by constructing a list of dictionaries as before. However, you can also use `cirq.Points` to do this more succinctly.

```
params = cirq.Points(key="theta", points=[0, 1, 3])
for param in params:
    print(param)
```

If you're working with parameterized circuits, it is very likely you'll need to keep track of multiple parameters. Two common use cases necessitate building a sweep from two constituent sweeps, where the new sweep includes:
- Every possible combination of the elements of each sweep: A cartesian product.
- A element-wise pairing of the two sweeps: A zip.

The following are examples of using the `*` and `+` operators to combine sweeps by cartesian product and zipping, respectively.

```
sweep1 = cirq.Linspace("theta", 0, 1, 5)
sweep2 = cirq.Points("gamma", [0, 3])
# By taking the product of these two sweeps, you can sweep over all possible
# combinations of the parameters.
for param in sweep1 * sweep2:
    print(param)
```

```
sweep1 = cirq.Points("theta", [1, 2, 3])
sweep2 = cirq.Points("gamma", [0, 3, 4])
# By taking the sum of these two sweeps, you can combine the sweeps
# elementwise (similar to python's zip function):
for param in sweep1 + sweep2:
    print(param)
```

`cirq.Linspace` and `cirq.Points` are instances of the `cirq.Sweep` class, which explicitly supports cartesian product with the `*` operation, and zipping with the `+` operation. The `*` operation produces a `cirq.Product` object, and `+` produces a `cirq.Zip` object, both of which are also `Sweep`s. Other mathematical operations will not work in general *between sweeps*.

## Symbols and Expressions

[SymPy](https://www.sympy.org/en/index.html){:external} is a general symbolic mathematics toolset, and you can leverage this in Cirq to define more complex parameters than have been shown so far. For example, you can define an expression in Sympy and use it to construct circuits that depend on this expression:

```
# Construct an expression for 0.5 * a + 0.25:
expr = 0.5 * sympy.Symbol("a") + 0.25
print(expr)
```

```
# Use the expression in the circuit:
circuit = cirq.Circuit(cirq.X(q0)**expr, cirq.measure(q0))
print(f"circuit:\n{circuit}")
```

Both the exponents and parameter arguments of circuit operations can in fact be any general Sympy expression: The previous examples just used single-variable expressions. When you resolve parameters for this circuit, the expressions are evaluated under the given assignments to the variables in the expression.

```
print(cirq.resolve_parameters(circuit, {"a": 0}))
```

Just as before, you can pass a sweep over variable values to `run` or `simulate`, and Cirq will evaluate the expression for each possible value.

```
sim = cirq.Simulator()
results = sim.run_sweep(circuit, repetitions=25, params=cirq.Points('a', [0, 1]))
for result in results:
    print(f"param: {result.params}, result: {result}")
```

Sympy supports a large number of numeric functions and methods, which can be used to create fairly sophisticated expressions, like cosine, exponentiation, and more:

```
print(sympy.cos(sympy.Symbol("a"))**sympy.Symbol("b"))
```

Cirq can numerically evaluate all of the expressions Sympy can evalute. However, if you are running a parameterized circuit on a service (such as on a hardware backed quantum computing service) that service may not support evaluating all expressions. See documentation for the particular service you're using for details.

As a general workaround, you can instead use Cirq's flattening ability to evaluate the parameters before sending them off to the service.

### Flattening Expressions

Suppose you build a circuit that includes multiple different expressions:

```
a = sympy.Symbol('a')
circuit = cirq.Circuit(cirq.X(q0)**(a / 4), cirq.Y(q0)**(1 - a / 2), cirq.measure(q0))
print(circuit)
```

Flattening replaces every expression in the circuit with a new symbol that is representative of the value of that expression. Additionally, it keeps track of the new symbols and provices a `cirq.ExpressionMap` object to map the old sympy expression objects to the new symbols that replaced them.

```
# Flatten returns two objects, the circuit with new symbols, and the mapping from old to new values.
c_flat, expr_map = cirq.flatten(circuit)
print(c_flat)
print(expr_map)
```

Notice that the new circuit has new symbols, `<a/2>` and `<1-a/2>`, which are explicitly not expressions.  You can see this by looking at the value of the exponent in the first gate:

```
first_gate = c_flat[0][q0].gate
print(first_gate.exponent)
# Note this is a symbol, not an expression
print(type(first_gate.exponent))
```

The second object returned by `cirq.flatten` is an object that can be used to map sweeps over the previous symbols to new sweeps over the new expression-symbols. The values assigned to the new expression symbols in the resulting sweep are the old expressions kept track of in the `ExpressionMap`, but resolved with the values provided by the original input sweep.

```
sweep = cirq.Linspace(a, start=0, stop=3, length=4)
print(f"Old {sweep}")

new_sweep = expr_map.transform_sweep(sweep)
print(f"New {new_sweep}")
```

To reinforce: The new sweep is over two new symbols, which each represent the values of the expressions in the original circuit. The values assigned to these new expression symbols is acquired by evaluating the expressions with `a` resolved to a value in `[0, 4]`, according to the old sweep.

You can use these new sweep elements to resolve the parameters of the flattened circuit:

```
for params in new_sweep:
    print(c_flat, '=>', end=' ')
    print(cirq.resolve_parameters(c_flat, params))
```

Using `cirq.flatten`, you can always take a parameterized circuit with any complicated expressions, plus a sweep, and produce an equivalent circuit with no expressions, only symbols, and a sweep for these new symbols. Because this is a common flow, Cirq provides `cirq.flatten_sweep` to do this in one step:

```
c_flat, new_sweep = cirq.flatten_with_sweep(circuit, sweep)
print(c_flat)
print(new_sweep)
```

You can then directly use these objects to run the sweeps. For example, you can use them to perform a simulation:

```
sim = cirq.Simulator()
results = sim.run_sweep(c_flat, repetitions=20, params=new_sweep)
for result in results:
    print(result.params, result)
```

You can see that the different flattened parameters have corresponding different results for their simulation.

# Summary

- Cirq circuits can handle arbitrary Sympy expressions in place of exponents and parameter arguments in operations.
- By providing one or a sequence of `ParamResolver`s or dictionaries that resolve the Sympy variables to values, `run`, `simulate`, and other functions can iterate efficiently over different parameter assignments for otherwise identical circuits.
- Sweeps can be created succinctly with `cirq.Points` and `cirq.Linspace`, and composed with each other with `*` and `+`, to create `cirq.Product` and `cirq.Zip` sweeps.
- When the service you're using does not support arbitrary expressions, you can flatten a circuit and sweep into a new circuit that doesn't have complex expressions, and a corresponding new sweep.


```python
from IPython.display import Image
Image('../../../python_for_probability_statistics_and_machine_learning.jpg')
```

[Python for Probability, Statistics, and Machine Learning](https://www.springer.com/fr/book/9783319307152)

# Projection Methods

The concept of projection is key to developing an intuition about conditional
probability.  We already have a natural intuition of projection from looking at
the shadows of objects on a sunny day. As we will see, this simple idea
consolidates many abstract ideas in optimization and mathematics.  Consider
[Figure](#fig:probability_001) where we want to find a point along the blue
line (namely, $\mathbf{x}$) that is closest to the black square (namely,
$\mathbf{y}$). In other words, we want to inflate the gray circle until it just
touches the black line. Recall that the circle boundary is the set of points for
which

$$
\sqrt{(\mathbf{y}-\mathbf{x})^T(\mathbf{y}-\mathbf{x})} =\|\mathbf{y}-\mathbf{x} \| = \epsilon
$$

 for some value of $\epsilon$. So we want a point $\mathbf{x}$ along
the line that satisfies this for the smallest $\epsilon$.  Then, that point
will be the closest point on the black line to the black square.
 It may be obvious from the diagram, but the closest point on the line
occurs where the line segment from the black square to the black line is
perpedicular to the line. At this point, the gray circle just touches the black
line. This is illustrated below in [Figure](#fig:probability_002).

Given the point $\mathbf{y}$ (black square) we want to find the $\mathbf{x}$ along the line that is closest to it. The gray circle is the locus of points within a fixed distance from $\mathbf{y}$.

**Programming Tip.**

[Figure](#fig:probability_001) uses the `matplotlib.patches` module.  This
module contains primitive shapes like circles, ellipses, and rectangles that
can be assembled into complex graphics.  As shown in the code in the IPython
Notebook corresponding to this chapter, after importing a particular shape, you
can apply that shape to an existing axis  using the `add_patch` method.  The
patches themselves can by styled using the usual formatting keywords like
`color` and `alpha`.

The closest point on the line occurs when the line is tangent to the circle. When this happens, the black line and the line (minimum distance) are perpedicular.

 Now that we can see what's going on, we can construct the the solution
analytically. We can represent an arbitrary point along the black line as:

$$
\mathbf{x}=\alpha\mathbf{v}
$$

 where $\alpha\in\mathbb{R}$ slides the point up and down the line with

$$
\mathbf{v} = \left[ 1,1 \right]^T
$$

 Formally, $\mathbf{v}$ is the *subspace* onto which we want to
*project* $\mathbf{y}$. At the closest point, the vector between
$\mathbf{y}$ and $\mathbf{x}$ (the *error* vector above) is
perpedicular to the line. This means that

$$
(\mathbf{y}-\mathbf{x} )^T \mathbf{v} = 0
$$

 and by substituting and working out the terms, we obtain

$$
\alpha = \frac{\mathbf{y}^T\mathbf{v}}{ \|\mathbf{v} \|^2}
$$

 The *error* is the distance between $\alpha\mathbf{v}$ and $
\mathbf{y}$.  This is a right triangle, and we can use the Pythagorean
theorem to compute the squared length of this error as

$$
\epsilon^2 = \|( \mathbf{y}-\mathbf{x} )\|^2 = \|\mathbf{y}\|^2 - \alpha^2 \|\mathbf{v}\|^2 = \|\mathbf{y}\|^2 - \frac{\|\mathbf{y}^T\mathbf{v}\|^2}{\|\mathbf{v}\|^2}
$$

 where $ \|\mathbf{v}\|^2 = \mathbf{v}^T \mathbf{v} $. Note that since $\epsilon^2 \ge 0 $, this also shows that

$$
\| \mathbf{y}^T\mathbf{v}\| \le \|\mathbf{y}\|  \|\mathbf{v}\|
$$

 which is the famous and useful Cauchy-Schwarz inequality which we
will exploit later. Finally, we can assemble all of this into the *projection*
operator

$$
\mathbf{P}_v = \frac{1}{\|\mathbf{v}\|^2 } \mathbf{v v}^T
$$

 With this operator, we can take any $\mathbf{y}$ and find the closest
point on $\mathbf{v}$ by doing

$$
\mathbf{P}_v \mathbf{y} = \mathbf{v} \left( \frac{  \mathbf{v}^T \mathbf{y} }{\|\mathbf{v}\|^2} \right)
$$

 where we recognize the term in parenthesis as the $\alpha$ we
computed earlier.  It's called an *operator* because it takes a vector
($\mathbf{y}$) and produces another vector ($\alpha\mathbf{v}$). Thus,
projection unifies geometry and optimization.

## Weighted distance

We can easily extend this projection operator to cases where the measure of
distance between $\mathbf{y}$ and the subspace $\mathbf{v}$ is weighted. We can
accommodate these weighted distances by re-writing the projection operator as

$$
\begin{equation}
\mathbf{P}_v=\mathbf{v}\frac{\mathbf{v}^T\mathbf{Q}^T}{\mathbf{v}^T\mathbf{Q v}}
\end{equation}
\label{eq:weightedProj} \tag{1}
$$

 where $\mathbf{Q}$ is positive definite matrix.  In the previous
case, we started with a point $\mathbf{y}$ and inflated a circle centered at
$\mathbf{y}$ until it just touched the line defined by $\mathbf{v}$ and this
point was closest point on the line to $\mathbf{y}$. The same thing happens
in the general case with a weighted distance except now we inflate an
ellipse, not a circle, until the ellipse touches the line.

In the weighted case, the closest point on the line is tangent to the ellipse and is still perpedicular in the sense of the weighted distance.

Note that the error vector ($\mathbf{y}-\alpha\mathbf{v}$) in [Figure](#fig:probability_003) is still perpendicular to the line (subspace
$\mathbf{v}$), but in the space of the weighted distance.  The difference
between the first projection (with the uniform circular distance) and the
general case (with the elliptical weighted distance) is the inner product
between the two cases.  For example, in the first case we have $\mathbf{y}^T
\mathbf{v}$ and in the weighted case we have $\mathbf{y}^T \mathbf{Q}^T
\mathbf{v}$. To move from the uniform circular case to the weighted ellipsoidal
case, all we had to do was change all of the vector inner products.  Before we
finish, we need a formal property  of projections:

$$
\mathbf{P}_v \mathbf{P}_v = \mathbf{P}_v
$$

 known as the *idempotent* property which basically says that once we
have projected onto a subspace, subsequent projections leave us in the
same subspace. You can verify this by computing Equation ref{eq:weightedProj}.

Thus, projection ties a minimization problem (closest point to a line) to an
algebraic concept (inner product). It turns out that these same geometric ideas
from linear algebra [[strang2006linear]](#strang2006linear) can be translated to the conditional
expectation. How this works is the subject of our next  section.