```python
import sys
```

```python
sys.path
```

    ['',
     '/usr/local/anaconda2/lib/python27.zip',
     '/usr/local/anaconda2/lib/python2.7',
     '/usr/local/anaconda2/lib/python2.7/plat-linux2',
     '/usr/local/anaconda2/lib/python2.7/lib-tk',
     '/usr/local/anaconda2/lib/python2.7/lib-old',
     '/usr/local/anaconda2/lib/python2.7/lib-dynload',
     '/usr/local/anaconda2/lib/python2.7/site-packages/Sphinx-1.3.5-py2.7.egg',
     '/usr/local/anaconda2/lib/python2.7/site-packages/setuptools-20.3-py2.7.egg',
     '/usr/local/anaconda2/lib/python2.7/site-packages',
     '/usr/local/anaconda2/lib/python2.7/site-packages/IPython/extensions',
     '/home/claudius/.ipython']

```python
import os
```

```python
os.getcwd()
```

    '/data3/claudius/Big_Data/DADI/dadiExercises'

I have cloned the $\delta$a$\delta$i repository into '/home/claudius/Downloads/dadi' and have compiled the code. Now I need to add that directory to the PYTHONPATH variable:

```python
sys.path.insert(0, '/home/claudius/Downloads/dadi')
```

```python
sys.path
```

    ['/home/claudius/Downloads/dadi',
     '',
     '/usr/local/anaconda2/lib/python27.zip',
     '/usr/local/anaconda2/lib/python2.7',
     '/usr/local/anaconda2/lib/python2.7/plat-linux2',
     '/usr/local/anaconda2/lib/python2.7/lib-tk',
     '/usr/local/anaconda2/lib/python2.7/lib-old',
     '/usr/local/anaconda2/lib/python2.7/lib-dynload',
     '/usr/local/anaconda2/lib/python2.7/site-packages/Sphinx-1.3.5-py2.7.egg',
     '/usr/local/anaconda2/lib/python2.7/site-packages/setuptools-20.3-py2.7.egg',
     '/usr/local/anaconda2/lib/python2.7/site-packages',
     '/usr/local/anaconda2/lib/python2.7/site-packages/IPython/extensions',
     '/home/claudius/.ipython']

Now, I should be able to import $\delta$a$\delta$i

```python
import dadi
```

```python
dir(dadi)
```

    ['Demographics1D',
     'Demographics2D',
     'Godambe',
     'Inference',
     'Integration',
     'Misc',
     'Numerics',
     'PhiManip',
     'Plotting',
     'Spectrum',
     'Spectrum_mod',
     'Triallele',
     '__builtins__',
     '__doc__',
     '__file__',
     '__name__',
     '__package__',
     '__path__',
     'integration_c',
     'logging',
     'numpy',
     'tridiag']

```python
import pylab
```

```python
%matplotlib inline
```

```python
x = pylab.linspace(0, 4*pylab.pi, 1000)
```

```python
pylab.plot(x, pylab.sin(x), '-r')
```

```sh
%%sh
# this allows me to execute a shell command

ls
```

    ERY.FOLDED.sfs
    ERY.FOLDED.sfs.dadi_format
    ERY.FOLDED.sfs.dadi_format~
    EryPar.unfolded.2dsfs
    EryPar.unfolded.2dsfs.dadi_format
    EryPar.unfolded.2dsfs.dadi_format~
    examples
    example_YRI_CEU.ipynb
    First_Steps_with_dadi.ipynb
    new.bib
    PAR.FOLDED.sfs
    PAR.FOLDED.sfs.dadi_format
    PAR.FOLDED.sfs.dadi_format~

I have turned the 1D folded SFS's from `realSFS` into $\delta$d$\delta$i format by hand according to the description in section 3.1 of the manual. I have left out the masking line from the input file.

```python
fs_ery = dadi.Spectrum.from_file('ERY.FOLDED.sfs.dadi_format')
```

```python
fs_ery
```

    Spectrum([-- 7833.03869 7414.699839 4109.279415 3614.717256 3095.973324 2031.460887
     1584.656928 2583.652317 1142.075255 1052.346021 1765.773415 1255.138799
     1072.516527 1417.916128 395.75047 1947.087637 367.072082 --], folded=True, pop_ids=None)

$\delta$a$\delta$i is detecting that the spectrum is folded (as given in the input file), but it is also automatically masking the 0th and 18th count category. This is a not a good behaviour.

```python
# number of segregating sites

fs_ery.data[1:].sum()
```

    43649.777914000006

## Single population statistics

### $\pi$

```python
fs_ery.pi()
```

    13545.692898679737

I have next added a masking line to the input file, setting it to '1' for the first position, i. e. the 0-count category.

```python
fs_ery = dadi.Spectrum.from_file('ERY.FOLDED.sfs.dadi_format', mask_corners=False)
```

$\delta$a$\delta$i is issuing the following message when executing the above command:

`WARNING:Spectrum_mod:Creating Spectrum with data_folded = True, but mask is not True for all entries which are nonsensical for a folded Spectrum.`

```python
fs_ery
```

    Spectrum([-- 7833.03869 7414.699839 4109.279415 3614.717256 3095.973324 2031.460887
     1584.656928 2583.652317 1142.075255 1052.346021 1765.773415 1255.138799
     1072.516527 1417.916128 395.75047 1947.087637 367.072082 966.622924], folded=True, pop_ids=None)

I do not understand this warning from $\delta$a$\delta$i. The 18-count category is sensical for a folded spectrum with even sample size, so should not be masked. Anyway, I do not understand why $\delta$a$\delta$i is so reluctant to keep all positions, including the non-variable one.

```python
fs_ery.pi()
```

    13545.692898679737

The function that returns $\pi$ produces the same output with or without the last count category masked ?! I think that is because even if the last count class (966.62...) is masked, it is still included in the calculation of $\pi$. However, there is no obvious unmasking in the `pi` function. Strange!

There are (at least) two formulas that allow the calculation of $\pi$ from a folded sample allele frequency spectrum. One is given in Wakeley2009, p.16, equation (1.4):
$$
\pi = \frac{1}{n \choose 2} \sum_{i=1}^{n/2} i(n-i)\eta_{i}
$$
Here, $n$ is the number of sequences and $\eta_{i}$ is the SNP count in the i'th minor sample allele frequency class.

The other formula is on p. 45 in Gillespie "Population Genetics - A concise guide":
$$
\hat{\pi} = \frac{n}{n-1} \sum_{i=1}^{S_{n}} 2 \hat{p_{i}}(1-\hat{p_{i}})
$$
This is the formula that $\delta$a$\delta$i's `pi` function uses, with the modification that it multiplies each $\hat{p_{i}}$ by the count in the i'th class of the SFS, i. e. the sum is not over all SNP's but over all SNP frequency classes.

```python
# Calcualting pi with the formula from Wakeley2009

n = 36 # 36 sequences sampled from 18 diploid individuals
pi_Wakeley = (sum( [i*(n-i)*fs_ery[i] for i in range(1, n/2+1)] ) * 2.0 / (n*(n-1)))/pylab.sum(fs_ery.data)
# note fs_ery.data gets the whole fs_ery list, including masked entries
pi_Wakeley
```

    0.0065187712427359126

This is the value of $\pi_{site}$ that I calculated previously and included in the first draft of the thesis.

```python
fs_ery.mask
```

    array([ True, False, False, False, False, False, False, False, False,
           False, False, False, False, False, False, False, False, False, False], dtype=bool)

```python
fs_ery.data # gets all data, including the masked one
```

    array([  1.59481822e+06,   7.83303869e+03,   7.41469984e+03,
             4.10927941e+03,   3.61471726e+03,   3.09597332e+03,
             2.03146089e+03,   1.58465693e+03,   2.58365232e+03,
             1.14207525e+03,   1.05234602e+03,   1.76577342e+03,
             1.25513880e+03,   1.07251653e+03,   1.41791613e+03,
             3.95750470e+02,   1.94708764e+03,   3.67072082e+02,
             9.66622924e+02])

```python
# Calculating pi with the formula from Gillespie:

n = 18
p = pylab.arange(0, n+1)/float(n)
p
```

    array([ 0.        ,  0.05555556,  0.11111111,  0.16666667,  0.22222222,
            0.27777778,  0.33333333,  0.38888889,  0.44444444,  0.5       ,
            0.55555556,  0.61111111,  0.66666667,  0.72222222,  0.77777778,
            0.83333333,  0.88888889,  0.94444444,  1.        ])

```python
# Calculating pi with the formula from Gillespie:

n / (n-1.0) * 2 * pylab.sum(fs_ery * p*(1-p))
```

    13545.692898679737

This is the same as the output of dadi's `pi` function on the same SFS.

```python
# the sample size (n) that dadi stores in this spectrum object and uses as n in the pi function
fs_ery.sample_sizes[0]
```

    18

```python
# what is the total number of sites in the spectrum
pylab.sum(fs_ery.data)
```

    1638467.9999990002

So, 1.6 million sites went into the ery spectrum.

```python
# pi per site
n / (n-1.0) * 2 * pylab.sum(fs_ery * p*(1-p)) / pylab.sum(fs_ery.data)
```

    0.0082672917009596787

Apart from the incorrect small sample size correction by $\delta$a$\delta$i in case of folded spectra ($n$ refers to sampled sequences, not individuals), Gillespie's formula leads to a much higher estimate of $\pi_{site}$ than Wakeley's. Why is that?

```python
# with correct small sample size correction
2 * n / (2* n-1.0) * 2 * pylab.sum(fs_ery * p*(1-p)) / pylab.sum(fs_ery.data)
```

    0.0080310833666465443

```python
# Calculating pi with the formula from Gillespie:

n = 18
p = pylab.arange(0, n+1)/float(n)
p = p/2 # with a folded spectrum, we are summing over minor allele freqs only
pi_Gillespie = 2*n / (2*n-1.0) * 2 * pylab.sum(fs_ery * p*(1-p)) / pylab.sum(fs_ery.data)
pi_Gillespie
```

    0.0065187712427359117

```python
pi_Wakeley - pi_Gillespie
```

    8.6736173798840355e-19

As can be seen from the insignificant difference (must be due to numerical inaccuracies) between the $\pi_{Wakeley}$ and the $\pi_{Gillespie}$ estimates, they are equivalent with the calculation for folded spectra given above as well as the correct small sample size correction. **Beware: $\delta$a$\delta$i does not handle folded spectra correctly**.

It should be a relatively easy to fix the `pi` function to work correctly with folded spectra. Care should be taken to also correctly handle uneven sample sizes.

```python
fs_ery.folded
```

    True

I think for now it would be best to import unfolded spectra from `realSFS` and fold them if necessary in dadi.

```python
fs_par = dadi.Spectrum.from_file('PAR.FOLDED.sfs.dadi_format')
```

```python
pylab.plot(fs_ery, 'r', label='ery')
pylab.plot(fs_par, 'g', label='par')
pylab.legend()
```

---

### ML estimate of $\theta$ from 1D folded spectrum

I am trying to fit eq. 4.21 of Wakeley2009 to the oberseved 1D folded spectra.

$$
E[\eta_i] = \theta \frac{\frac{1}{i} + \frac{1}{n-i}}{1+\delta_{i,n-i}} \qquad 1 \le i \le \big[n/2\big]
$$

Each frequency class, $\eta_i$, provides an estimate of $\theta$. However, I would like to find the value of $\theta$ that minimizes the deviation of the above equation from all observed counts $\eta_i$.

I am following the example given here: https://docs.scipy.org/doc/scipy/reference/tutorial/optimize.html#example-of-solving-a-fitting-problem

$$
\frac{\delta E}{\delta \theta} = \frac{\frac{1}{i} + \frac{1}{n-i}}{1+\delta_{i,n-i}} \qquad 1 \le i \le \big[n/2\big]
$$

I have just one parameter to optimize.

```python
from scipy.optimize import least_squares
```

```python
def model(theta, eta, n):
    """
    theta: scaled population mutation rate parameter [scalar]
    eta: the folded 1D spectrum, including 0-count cat. [list]
    n: number of sampled gene copies, i. e. 2*num_ind [scalar]

    returns a numpy array
    """
    i = pylab.arange(1, eta.size)
    delta = pylab.where(i == n-i, 1, 0)
    return theta * 1/i + 1/(n-i) / (1 + delta)
```

```python
?pylab.where
```

```python
# test
i = pylab.arange(1, 19)
n = 36
print i == n-i
#
print pylab.where(i == n-i, 1, 0)
# get a theta estimate from pi:
theta = pi_Wakeley * fs_ery.data.sum()
print theta
#
print len(fs_ery)
#
model(theta, fs_ery, 36)
```

    [False False False False False False False False False False False False
     False False False False False  True]
    [0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1]
    10680.7980805
    19

    array([ 10680.79808054,   5340.39904027,   3560.26602685,   2670.19952013,
             2136.15961611,   1780.13301342,   1525.82829722,   1335.09976007,
             1186.75534228,   1068.07980805,    970.98164369,    890.06650671,
              821.59985235,    762.91414861,    712.05320537,    667.54988003,
              628.28224003,    593.37767114])

```python
def fun(theta, eta, n):
    """
    return residuals between model and data
    """
    return model(theta, eta, n) - eta[1:]
```

```python
def jac(theta, eta, n, test=False):
    """
    creates a Jacobian matrix
    """
    J = pylab.empty((eta.size-1, theta.size))
    i = pylab.arange(1, eta.size, dtype=float)
    delta = pylab.where(i == n-i, 1, 0)
    num = 1/i + 1/(n-i)
    den = 1 + delta
    if test:
        print i
        print num
        print den
    J[:,0] = num / den
    return J
```

```python
# test
jac(theta, fs_ery, 36, test=True)
```

    [  1.   2.   3.   4.   5.   6.   7.   8.   9.  10.  11.  12.  13.  14.  15.
      16.  17.  18.]
    [ 1.02857143  0.52941176  0.36363636  0.28125     0.23225806  0.2
      0.1773399   0.16071429  0.14814815  0.13846154  0.13090909  0.125
      0.12040134  0.11688312  0.11428571  0.1125      0.11145511  0.11111111]
    [1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2]

    array([[ 1.02857143],
           [ 0.52941176],
           [ 0.36363636],
           [ 0.28125   ],
           [ 0.23225806],
           [ 0.2       ],
           [ 0.1773399 ],
           [ 0.16071429],
           [ 0.14814815],
           [ 0.13846154],
           [ 0.13090909],
           [ 0.125     ],
           [ 0.12040134],
           [ 0.11688312],
           [ 0.11428571],
           [ 0.1125    ],
           [ 0.11145511],
           [ 0.05555556]])

```python
# starting value
theta0 = theta # pi_Wakeley from above
```

```python
# sum over unmasked entries, i. e. without 0-count category, i. e. returns number of variable sites
fs_ery.sum()
```

    43649.777914000006

```python
# optimize
res = least_squares(fun, x0=theta0, jac=jac, bounds=(0,fs_ery.sum()),
                    kwargs={'eta': fs_ery, 'n': 36}, verbose=1)
```

    Both `ftol` and `xtol` termination conditions are satisfied.
    Function evaluations: 8, initial cost: 9.6784e+06, final cost 9.5162e+06, first-order optimality 1.08e-01.

```python
res.success
```

    True

```python
?least_squares
```

```python
print res.x
print theta
```

    [ 10367.32782801]
    10680.7980805

```python
pylab.rcParams['figure.figsize'] = [12.0, 8.0]
```

```python
import matplotlib.pyplot as plt

plt.rcParams['font.size'] = 14.0

i = range(1, len(fs_ery))
eta_model = model(res.x, eta=fs_ery, n=36) # get predicted values with optimal theta

plt.plot(i, fs_ery[1:], "bo", label="data from ery") # plot observed spectrum

ymax = max( fs_ery[1:].max(), eta_model.max() )
plt.axis([0, 19, 0, ymax*1.1]) # set axis range

plt.xlabel("minor allele frequency (i)")
plt.ylabel(r'$\eta_i$', fontsize='large', rotation='horizontal')
plt.title("folded SFS of ery")

plt.plot(i, eta_model, "go-",
         label="\nneutral model"
         + "\n"
         + r'$\theta_{opt} = $' + str(round(res.x, 1))
        ) # plot model prediction with optimal theta

plt.legend()
```

The counts in each frequency class should be Poisson distributed with rate equal to $E[\eta_i]$ as given above. The lowest frequency class has the highest rate and therefore also the highest variance

```python
#?plt.ylabel
```

```python
#print plt.rcParams
```

```python
fs_ery[1:].max()
```

    7833.0386900000003

```python
#?pylab
```

```python
os.getcwd()
```

    '/data3/claudius/Big_Data/DADI/dadiExercises'

```sh
%%sh

ls
```

    ERY.FOLDED.sfs
    ERY.FOLDED.sfs.dadi_format
    ERY.FOLDED.sfs.dadi_format~
    EryPar.unfolded.2dsfs
    EryPar.unfolded.2dsfs.dadi_format
    EryPar.unfolded.2dsfs.dadi_format~
    examples
    example_YRI_CEU.ipynb
    First_Steps_with_dadi.ipynb
    new.bib
    PAR.FOLDED.sfs
    PAR.FOLDED.sfs.dadi_format
    PAR.FOLDED.sfs.dadi_format~

The following function will take the file name of a file containing the flat 1D folded frequency spectrum of one population and plots it together with the best fitting neutral expectation.

```python
def plot_folded_sfs(filename, n, pop = ''):
    # read in spectrum from file
    data = open(filename, 'r')
    sfs = pylab.array( data.readline().split(), dtype=float )
    data.close() # should close connection to file
    #return sfs

    # get starting value for theta from Watterson's theta
    S = sfs[1:].sum()
    T_total = sum([1.0/i for i in range(1, n)]) # onhe half the expected total length of the genealogy
    theta0 = S / T_total # see eq. 4.7 in Wakeley2009

    # optimize
    res = least_squares(fun, x0=theta0, jac=jac, bounds=(0, sfs.sum()),
                   kwargs={'eta': sfs, 'n': 36}, verbose=1)
    #print "Optimal theta per site is {0:.4f}".format(res.x[0]/sfs.sum())
    #print res.x[0]/sfs.sum()

    #return theta0, res

    # plot
    plt.rcParams['font.size'] = 14.0

    i = range(1, len(sfs))
    eta_model = model(res.x, eta=sfs, n=36) # get predicted values with optimal theta

    plt.plot(i, sfs[1:], "rs", label="data of " + pop) # plot observed spectrum

    ymax = max( sfs[1:].max(), eta_model.max() )
    plt.axis([0, 19, 0, ymax*1.1]) # set axis range

    plt.xlabel("minor allele frequency (i)")
    plt.ylabel(r'$\eta_i$', fontsize='large', rotation='horizontal')
    plt.title("folded SFS")
    plt.text(5, 10000,
             r"Optimal neutral $\theta$ per site is {0:.4f}".format(res.x[0]/sfs.sum()))

    plt.plot(i, eta_model, "go-",
         label="\nneutral model"
         + "\n"
         + r'$\theta_{opt} = $' + str(round(res.x, 1))
        ) # plot model prediction with optimal theta

    plt.legend()
```

```python
plot_folded_sfs('PAR.FOLDED.sfs', n=36, pop='par')
```

```python
plot_folded_sfs('ERY.FOLDED.sfs', n=36, pop='ery')
```

### Univariate function minimizers or 1D scalar minimisation

Since I only have one value to optimize, I can use a slightly simpler approach than used above:

```python
from scipy.optimize import minimize_scalar
```

```python
?minimize_scalar
```

```python
# define cost function
def f(theta, eta, n):
    """
    return sum of squared deviations between model and data
    """
    return sum( (model(theta, eta, n) - eta[1:])**2 ) # see above for definition of the 'model' function
```

It would be interesting to know whether the cost function is convex or not.

```python
theta = pylab.arange(0, fs_ery.data[1:].sum()) # specify range of theta
cost = [f(t, fs_ery.data, 36) for t in theta]
plt.plot(theta, cost, 'b-', label='ery')
plt.xlabel(r'$\theta$')
plt.ylabel('cost')
plt.title("cost function for ery")
plt.legend(loc='best')
```

```python
?plt.legend
```

Within the specified bounds (the observed $\theta$, i. e. derived from the data, cannot lie outside these bounds), the cost function is convex. This is therefore an easy optimisation problem. See [here](http://www.scipy-lectures.org/advanced/mathematical_optimization/index.html) for more details.

```python
res = minimize_scalar(f, bounds = (0, fs_ery.data[1:].sum()), method = 'bounded', args = (fs_ery.data, 36))
```

```python
res
```

         fun: 18987280.758395974
     message: 'Solution found.'
        nfev: 6
      status: 0
     success: True
           x: 10198.886010965949

```python
# number of segregating sites

fs_par.data[1:].sum()
```

    43755.705189

```python
res = minimize_scalar(f, bounds = (0, fs_par.data[1:].sum()), method = 'bounded', args = (fs_par.data, 36))
```

```python
res
```

         fun: 60127046.281448714
     message: 'Solution found.'
        nfev: 6
      status: 0
     success: True
           x: 11828.17091399114

The fitted values of $\theta$ are similar to the ones obtained above with the `least_squares` function. The estimates for ery deviate more than for par.

```python
from sympy import *
```

```python
x0 , x1 = symbols('x0 x1')
```

```python
init_printing(use_unicode=True)
```

```python
diff(0.5*(1-x0)**2 + (x1-x0**2)**2, x0)
```

```python
diff(0.5*(1-x0)**2 + (x1-x0**2)**2, x1)
```

Wow! Sympy is a replacement for Mathematica. There is also Sage, which may include even more functionality.

```python
from scipy.optimize import curve_fit
```

`Curve_fit` is another function that can be used for optimization.

```python
?curve_fit
```

```python
def model(i, theta):
    """
    i: indpendent variable, here minor SNP frequency classes
    theta: scaled population mutation rate parameter [scalar]

    returns a numpy array
    """
    n = len(i)
    delta = pylab.where(i == n-i, 1, 0)
    return theta * 1/i + 1/(n-i) / (1 + delta)
```

```python
i = pylab.arange(1, fs_ery.size)

popt, pcov = curve_fit(model, i, fs_ery.data[1:])
```

```python
# optimal theta
print popt
```

    [ 10198.84901849]

```python
perr = pylab.sqrt(pcov)
perr
```

    array([[ 837.89916279]])

```python
print str(int(popt[0] - 1.96*perr[0])) + ' < ' + str(int(popt[0])) + ' < ' + str(int(popt[0] + 1.96*perr[0]))
```

    8556 < 10198 < 11841

```python
popt, pcov = curve_fit(model, i, fs_par.data[1:])
perr = pylab.sqrt(pcov)
print str(int(popt[0] - 1.96*perr[0])) + ' < ' + str(int(popt[0])) + ' < ' + str(int(popt[0] + 1.96*perr[0]))
```

    8905 < 11828 < 14750

I am not sure whether these standard errors (perr) are correct. It may be that it is assumed that errors are normally distributed, which they are not exactly in this case. They should be close to Poisson distributed (see Fu1995), which should be fairly similar to normal with such high expected values as here.

If the standard errors are correct, then the large overlap of the 95% confidence intervals would indicate that the data do not provide significant support for a difference in $\theta$ between par and ery.

## Parametric bootstrap from the observed SFS

```python
%pwd
```

    u'/data3/claudius/Big_Data/DADI/dadiExercises'

```python
% ll
```

    total 660
    lrwxrwxrwx 1 claudius     53 Feb 17 15:37 ERY.FOLDED.sfs -> /data3/claudius/Big_Data/ANGSD/SFS/ERY/ERY.FOLDED.sfs
    -rw-rw-r-- 1 claudius    462 Mar 15 12:48 ERY.FOLDED.sfs.dadi_format
    -rw-rw-r-- 1 claudius    462 Mar 15 12:45 ERY.FOLDED.sfs.dadi_format~
    lrwxrwxrwx 1 claudius     37 Feb 18 17:46 EryPar.unfolded.2dsfs -> ../../ANGSD/FST/EryPar.unfolded.2dsfs
    -rw-rw-r-- 1 claudius  13051 Feb 18 19:00 EryPar.unfolded.2dsfs.dadi_format
    -rw-rw-r-- 1 claudius  13051 Feb 18 18:31 EryPar.unfolded.2dsfs.dadi_format~
    drwxrwxr-x 5 claudius   4096 Feb 17 13:45 examples/
    -rw-rw-r-- 1 claudius  18014 Mar 15 21:39 example_YRI_CEU.ipynb
    -rw-rw-r-- 1 claudius 596246 Mar 17 15:45 First_Steps_with_dadi.ipynb
    -rw-rw-r-- 1 claudius   1012 Mar 16 09:54 new.bib
    lrwxrwxrwx 1 claudius     53 Feb 17 15:37 PAR.FOLDED.sfs -> /data3/claudius/Big_Data/ANGSD/SFS/PAR/PAR.FOLDED.sfs
    -rw-rw-r-- 1 claudius    412 Feb 17 16:29 PAR.FOLDED.sfs.dadi_format
    -rw-rw-r-- 1 claudius    218 Feb 17 15:51 PAR.FOLDED.sfs.dadi_format~

```python
! cat ERY.FOLDED.sfs.dadi_format
```

    # this is the ML estimate of the folded sample frequency spectrum for erythropus, estimated with realSFS of ANGSD
    # this is the spectrum in dadi format (see section 3.1 of the manual)
    19 folded
    1594818.222085 7833.038690 7414.699839 4109.279415 3614.717256 3095.973324 2031.460887 1584.656928 2583.652317 1142.075255 1052.346021 1765.773415 1255.138799 1072.516527 1417.916128 395.750470 1947.087637 367.072082 966.622924
    1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

```python
fs_ery = dadi.Spectrum.from_file('ERY.FOLDED.sfs.dadi_format', mask_corners=False)
```

```python
fs_ery
```

    Spectrum([-- 7833.03869 7414.699839 4109.279415 3614.717256 3095.973324 2031.460887
     1584.656928 2583.652317 1142.075255 1052.346021 1765.773415 1255.138799
     1072.516527 1417.916128 395.75047 1947.087637 367.072082 966.622924], folded=True, pop_ids=None)

```python
fs_ery.pop_ids = ['ery']
```

```python
# get a Poisson sample from the observed spectrum

fs_ery_param_boot = fs_ery.sample()
```

```python
fs_ery_param_boot
```

    Spectrum([-- 7727.0 7286.0 4114.0 3578.0 3064.0 2051.0 1574.0 2532.0 1177.0 1083.0
     1722.0 1219.0 1025.0 1382.0 403.0 1910.0 377.0 --], folded=True, pop_ids=['ery'])

```python
fs_ery_param_boot.data
```

    array([    0.,  7727.,  7286.,  4114.,  3578.,  3064.,  2051.,  1574.,
            2532.,  1177.,  1083.,  1722.,  1219.,  1025.,  1382.,   403.,
            1910.,   377.,   933.])

```python
%psource fs_ery.sample
```

**There must be a way to get more than one bootstrap sample per call.**

```python
fs_ery_param_boot = pylab.array([fs_ery.sample() for i in range(100)])
```

```python
# get the first 3 boostrap samples from the doubleton class

fs_ery_param_boot[:3, 2]
```

    array([ 7274.,  7463.,  7430.])

It would be good to get the 5% and 95% quantiles from the bootstrap samples of each frequency class and add those intervals to the plot of the observed frequency spectrum and the fitted neutral spectrum. This would require to find a quantile function and to find out how to add lines to a plot with matplotlib.

It would also be good to use the predicted counts from the neutral model above with the fitted $\theta$ as parameters for the bootstrap with `sample()` and add 95% confidence intervals to the predicted neutral SFS. I have done this in R instead (see `/data3/claudius/Big_Data/ANGSD/SFS/SFS.Rmd`)

---

### Using unfolded spectra

I edited the 2D SFS created for estimating $F_{ST}$ by `realSFS`. I have convinced myself that `realSFS` outputs a flattened 2D matrix as expected by $\delta$a$\delta$i's `Spectrum.from_file` function (see section 3.1 of the manual with my comments). Note, that in the manual, "samples" stands for number of allele copies, so that the correct specification of dimensions for this 2D unfolded SFS of 18 diploid individuals in each of 2 populations is 37 x 37.

```python
# read in the flattened 2D SFS
EryPar_unfolded_2dsfs = dadi.Spectrum.from_file('EryPar.unfolded.2dsfs.dadi_format', mask_corners=True)
```

```python
# check dimension
len(EryPar_unfolded_2dsfs[0,])
```

```python
EryPar_unfolded_2dsfs.sample_sizes
```

    array([36, 36])

```python
# add population labels
EryPar_unfolded_2dsfs.pop_ids = ["ery", "par"]
```

```python
EryPar_unfolded_2dsfs.pop_ids
```

    ['ery', 'par']

### Marginalizing

$\delta$a$\delta$i offers a function to get the marginal spectra from multidimensional spectra. Note, that this marginalisation is nothing fancy. In `R` it would be taking either the `rowSums` or the `colSums` of the matrix.

```python
# marginalise over par to get 1D SFS for ery

fs_ery = EryPar_unfolded_2dsfs.marginalize([1])
# note the argument is an array with dimensions, one can marginalise over more than one dimension at the same time,
# but that is only interesting for 3-dimensional spectra, which I don't have here
```

```python
fs_ery
```

    Spectrum([-- 5504.293033999999 4934.276604000001 2566.124531 2396.011968
     1553.2582019999998 1297.416363 841.8906099999998 1361.1500360000002
     488.49162899999993 610.185385 845.894307 475.48793400000005 845.864915
     171.41514700000002 661.7809339999999 260.92126999999994 332.462328
     577.679207 212.03316999999998 469.402014 197.154764 399.04286399999995
     113.996089 335.46549500000003 260.04657000000003 223.011465 236.792219
     329.313785 346.363423 120.226742 505.32778799999994 230.19171200000002
     451.244682 407.265961 600.9017859999999 --], folded=False, pop_ids=['ery'])

```python
# marginalise over ery to get 1D SFS for par
fs_par = EryPar_unfolded_2dsfs.marginalize([0])
```

```python
fs_par
```

    Spectrum([-- 7797.888484000002 10825.498380000005 4798.473355000001 2640.397539
     2157.0349559999995 1011.531934 726.7379290000001 1698.623407 493.287427
     453.03942799999993 816.1893790000003 677.1919899999999 266.613834
     327.9128799999999 449.9266600000001 452.58292199999994 123.233246
     385.56443 478.09934499999997 216.15979699999997 269.61825899999997
     169.752214 219.371672 467.7375149999999 234.94620899999998
     230.09612900000002 244.87403099999997 160.987576 226.845337
     466.00908100000004 505.19249 273.09656199999995 269.669384 549.555371
     336.129774 --], folded=False, pop_ids=['par'])

Note, that these marginalised 1D SFS's are not identical to the 1D SFS estimated directly with `realSFS`. This is because, for the estimation of the 2D SFS, `realSFS` has only taken sites that had data from at least 9 individuals in *each* population (see `assembly.sh`, lines 1423 onwards).

The SFS's of par and ery had conspicuous shape differences. It would therefore be good to plot them to see, whether the above commands have done the correct thing.

```python
# plot 1D spectra for each population
pylab.plot(fs_par, 'g', label="par")
pylab.plot(fs_ery, 'r', label="ery")
pylab.legend()
```

These marginal unfolded spectra look similar in shape to the 1D folded spectra of each subspecies (see above).

```python
fs_ery.pi() / pylab.sum(fs_ery.data)
```

```python
fs_ery.data
```

    array([ 27952.979034,   5504.293034,   4934.276604,   2566.124531,
             2396.011968,   1553.258202,   1297.416363,    841.89061 ,
             1361.150036,    488.491629,    610.185385,    845.894307,
              475.487934,    845.864915,    171.415147,    661.780934,
              260.92127 ,    332.462328,    577.679207,    212.03317 ,
              469.402014,    197.154764,    399.042864,    113.996089,
              335.465495,    260.04657 ,    223.011465,    236.792219,
              329.313785,    346.363423,    120.226742,    505.327788,
              230.191712,    451.244682,    407.265961,    600.901786,
             1458.220459])

```python
n = 36 # 36 sequences sampled from 18 diploid individuals
pi_Wakeley = (sum( [i*(n-i)*fs_ery[i] for i in range(1, n)] ) * 2.0 / (n*(n-1)))
pi_Wakeley = pi_Wakeley / pylab.sum(fs_ery.data)
pi_Wakeley
```

$\delta$a$\delta$i's `pi` function seems to calculate the correct value of $\pi$ for this unfolded spectrum. However, it is worrying that $\pi$ from this marginal spectrum is about 20 times larger than the one calculated from the directly estimated 1D folded spectrum (see above the $\pi$ calculated from the folded 1D spectrum).

```python
fs_par.pi() / pylab.sum(fs_par.data)
```

```python
pylab.sum(fs_par.data)
```

```python
pylab.sum(EryPar_unfolded_2dsfs.data)
```

<font color="red">The sum over the marginalised 1D spectra should be the same as the sum over the 2D spectrum !</font>

```python
# from dadi's marginalise function:
fs_ery.data
```

    array([ 27952.979034,   5504.293034,   4934.276604,   2566.124531,
             2396.011968,   1553.258202,   1297.416363,    841.89061 ,
             1361.150036,    488.491629,    610.185385,    845.894307,
              475.487934,    845.864915,    171.415147,    661.780934,
              260.92127 ,    332.462328,    577.679207,    212.03317 ,
              469.402014,    197.154764,    399.042864,    113.996089,
              335.465495,    260.04657 ,    223.011465,    236.792219,
              329.313785,    346.363423,    120.226742,    505.327788,
              230.191712,    451.244682,    407.265961,    600.901786,
             1458.220459])

```python
sfs2d = EryPar_unfolded_2dsfs.copy()
```

```python
# this should get the marginal spectrum for ery
ery_mar = [pylab.sum(sfs2d.data[i]) for i in range(0, len(sfs2d))]
ery_mar
```

```python
# this should get the marginal spectrum for ery and then take the sum over it
sum([pylab.sum(sfs2d.data[i]) for i in range(0, len(sfs2d))])
```

```python
# look what happens if I include masking
sum([pylab.sum(sfs2d[i]) for i in range(0, len(sfs2d))])
```

```python
fs_ery.data - ery_mar
```

    array([ -1.06991366e+06,   0.00000000e+00,   0.00000000e+00,
             0.00000000e+00,   0.00000000e+00,   0.00000000e+00,
             0.00000000e+00,   0.00000000e+00,   0.00000000e+00,
             0.00000000e+00,   0.00000000e+00,   0.00000000e+00,
             0.00000000e+00,   0.00000000e+00,   0.00000000e+00,
             0.00000000e+00,   0.00000000e+00,   0.00000000e+00,
             0.00000000e+00,   0.00000000e+00,   0.00000000e+00,
             0.00000000e+00,   0.00000000e+00,   0.00000000e+00,
             0.00000000e+00,   0.00000000e+00,   0.00000000e+00,
             0.00000000e+00,   0.00000000e+00,   0.00000000e+00,
             0.00000000e+00,   0.00000000e+00,   0.00000000e+00,
             0.00000000e+00,   0.00000000e+00,   0.00000000e+00,
            -2.87751595e+02])

So, during the marginalisation the masking of data in the fixed categories (0, 36) is the problem, producing incorrectly marginalised counts in those masked categories. This is shown in the following:

```python
sfs2d[0]
```

    Spectrum([-- 6546.449601 9367.767974 3791.122626 1749.224237 1598.038967 656.869675
     412.970591 974.236078 142.404132 211.95951 521.906868 156.263585 5.346426
     159.363508 220.258071 175.985792 19.108475 57.580136 167.053354 34.227901
     87.965634 46.427042 86.934511 141.736163 52.391344 3.808952 47.21001
     11.204495 38.908948 74.130926 0.03966 46.362627 39.973018 59.216701
     1.520393 247.011103], folded=False, pop_ids=['ery', 'par'])

```python
pylab.sum(sfs2d[0])
```

```python
# from dadi's marginalise function:
fs_ery.data
```

    array([ 27952.979034,   5504.293034,   4934.276604,   2566.124531,
             2396.011968,   1553.258202,   1297.416363,    841.89061 ,
             1361.150036,    488.491629,    610.185385,    845.894307,
              475.487934,    845.864915,    171.415147,    661.780934,
              260.92127 ,    332.462328,    577.679207,    212.03317 ,
              469.402014,    197.154764,    399.042864,    113.996089,
              335.465495,    260.04657 ,    223.011465,    236.792219,
              329.313785,    346.363423,    120.226742,    505.327788,
              230.191712,    451.244682,    407.265961,    600.901786,
             1458.220459])

```python
# dividing by the correct number of sites to get pi per site:
fs_ery.pi() / pylab.sum(sfs2d.data)
```

This is very close to the estimate of $\pi$ derived from the folded 1D spectrum of ery! (see above)

```python
fs_par.pi() / pylab.sum(sfs2d.data)
```

This is also nicely close to the estimate of $\pi_{site}$ of par from its folded 1D spectrum.

---

### Tajima's D

```python
fs_ery.Watterson_theta() / pylab.sum(sfs2d.data)
```

```python
fs_ery.Tajima_D()
```

```python
fs_par.Tajima_D()
```

Now, I am calculating Tajima's D from the ery marginal spectrum by hand in order to check whether $\delta$a$\delta$i is doing the right thing.

```python
n = 36
pi_Wakeley = (sum( [i*(n-i)*fs_ery.data[i] for i in range(1, n+1)] )
              * 2.0 / (n*(n-1)))
                #/ pylab.sum(sfs2d.data)
pi_Wakeley
```

```python
# number of segregating sites
# this sums over all unmasked positions in the array
pylab.sum(fs_ery)
```

```python
fs_ery.S()
```

```python
S = pylab.sum(fs_ery)
theta_Watterson = S / pylab.sum(1.0 / (pylab.arange(1, n)))
theta_Watterson
```

```python
# normalizing constant, see page 45 in Gillespie
a1 = pylab.sum(1.0 / pylab.arange(1, n))
#print a1
a2 = pylab.sum(1.0 / pylab.arange(1, n)**2.0)
#print a2
b1 = (n+1.0)/(3.0*(n-1))
#print b1
b2 = 2.0*(n**2 + n + 3)/(9.0*n*(n-1))
#print b2
c1 = b1 - (1.0/a1)
#print c1
c2 = b2 - (n+2.0)/(a1*n) + a2/a1**2
#print c2
C = ((c1/a1)*S + (c2/(a1**2.0 + a2))*S*(S-1))
C = C**(1/2.0)
```

```python
ery_Tajimas_D = (pi_Wakeley - theta_Watterson) / C
print '{0:.6f}'.format(ery_Tajimas_D)
```

    -0.054767

```python
ery_Tajimas_D - fs_ery.Tajima_D()
```

$\delta$a$\delta$i seems to do the right thing. Note, that the estimate of Tajima's D from this marginal spectrum of ery is slightly different from the estimate derived from the folded 1D spectrum of ery (see /data3/claudius/Big_Data/ANGSD/SFS/SFS.Rmd). The folded 1D spectrum resulted in a Tajima's D estimate of $\sim$0.05, i. e. a difference of almost 0.1. Again, the 2D spectrum is based on only those sites for which there were at least 9 individiuals with data in *both* populations, whereas the 1D folded spectrum of ery included all sites for which there were 9 ery individuals with data (see line 1571 onwards in `assembly.sh`).

```python
fs_par.Tajima_D()
```

My estimate from the folded 1D spectrum of par was -0.6142268 (see /data3/claudius/Big_Data/ANGSD/SFS/SFS.Rmd).

### Multi-population statistics

```python
EryPar_unfolded_2dsfs.S()
```

The 2D spectrum contains counts from 60k sites that are variable in *par* or *ery* or both.

```python
EryPar_unfolded_2dsfs.Fst()
```

This estimate of $F_{ST}$ according to Weir and Cockerham (1984) is well below the estimate of $\sim$0.3 from ANGSD according to Bhatia/Hudson (2013). Note, however, that this estimate showed a positive bias of around 0.025 in 100 permutations of population labels of individuals. Taking the positive bias into account, both estimates of $F_{ST}$ are quite similar.

The following function `scramble_pop_ids` should generate a 2D SFS with counts as if individuals were assigned to populations randomly. Theoretically, the $F_{ST}$ calculated from this SFS should be 0.

```python
%psource EryPar_unfolded_2dsfs.scramble_pop_ids
```

```python
# plot the scrambled 2D SFS

dadi.Plotting.plot_single_2d_sfs(EryPar_unfolded_2dsfs.scramble_pop_ids(), vmin=1)
```

So, this is how the 2D SFS would look like if _ery_ and _par_ were not genetically differentiated.

```python
# get Fst for scrambled SFS

EryPar_unfolded_2dsfs.scramble_pop_ids().Fst()
```

The $F_{ST}$ from the scrambled SFS is much lower than the $F_{ST}$ of the observed SFS. That should mean that there is significant population structure. However, the $F_{ST}$ from the scrambled SFS is not 0. I don't know why that is.

```python

```

---

```python
# folding

EryPar_folded_2dsfs = EryPar_unfolded_2dsfs.fold()
```

```python
EryPar_folded_2dsfs
```

    Spectrum([[-- 6651.315570000001 9534.20133 ..., 59.216701 19.075761999999997
      208.9562905]
     [4407.352191 453.932731 409.549382 ..., 3.9669600000000003
      4.851331999999999 --]
     [3746.5561820000003 295.73506799999996 261.799583 ..., 1e-06 -- --]
     ...,
     [26.699661 1.593931 1e-06 ..., -- -- --]
     [78.253372 4.851332 -- ..., -- -- --]
     [208.9562905 -- -- ..., -- -- --]], folded=True, pop_ids=['ery', 'par'])

```python
EryPar_folded_2dsfs.mask
```

    array([[ True, False, False, ..., False, False, False],
           [False, False, False, ..., False, False,  True],
           [False, False, False, ..., False,  True,  True],
           ...,
           [False, False, False, ...,  True,  True,  True],
           [False, False,  True, ...,  True,  True,  True],
           [False,  True,  True, ...,  True,  True,  True]], dtype=bool)

### Plotting

```python
dadi.Plotting.plot_single_2d_sfs(EryPar_unfolded_2dsfs, vmin=1)
```

```python
dadi.Plotting.plot_single_2d_sfs(EryPar_folded_2dsfs, vmin=1)
```

The folded 2D spectrum is *not* a minor allele frequency spectrum as are the 1D folded spectra of ery and par. This is because an allele that is minor in one population can be the major allele in the other. What is not counted are the alleles that are major in *both* populations, i. e. the upper right corner.

For the 2D spectrum to make sense it is crucial that allele frequencies are polarised the same way in both populations, either with an outgroup sequence or arbitrarily with respect to the reference sequence (as I did here).

#### How to fold a 1D spectrum

```python
# unfolded spectrum from marginalisation of 2D unfolded spectrum
fs_ery
```

    Spectrum([-- 5504.293033999999 4934.276604000001 2566.124531 2396.011968
     1553.2582019999998 1297.416363 841.8906099999998 1361.1500360000002
     488.49162899999993 610.185385 845.894307 475.48793400000005 845.864915
     171.41514700000002 661.7809339999999 260.92126999999994 332.462328
     577.679207 212.03316999999998 469.402014 197.154764 399.04286399999995
     113.996089 335.46549500000003 260.04657000000003 223.011465 236.792219
     329.313785 346.363423 120.226742 505.32778799999994 230.19171200000002
     451.244682 407.265961 600.9017859999999 --], folded=False, pop_ids=['ery'])

```python
len(fs_ery)
```

```python
fs_ery.fold()
```

    Spectrum([-- 6105.194819999999 5341.542565000001 3017.369213 2626.2036799999996
     2058.5859899999996 1417.643105 1188.2540329999997 1690.4638210000003
     725.2838479999999 833.19685 1105.940877 810.9534290000001 959.861004
     570.4580109999999 858.9356979999999 730.323284 544.495498 577.679207 -- --
     -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --], folded=True, pop_ids=['ery'])

Let's use the formula (1.2) from Wakeley2009 to fold the 1D spectrum manually:
$$
\eta_{i} = \frac{\zeta_{i} + \zeta_{n-i}}{1 + \delta_{i, n-i}} \qquad 1 \le i \le [n/2]
$$
$n$ is the number of gene copies sampled, i. e. haploid sample size. $[n/2]$ is the largest integer less than or equal to n/2 (to handle uneven sample sizes). $\zeta_{i}$ are the unfolded frequencies and $\delta_{i, n-i}$ is Kronecker's $\delta$ which is 1 if $i = n-i$ and zero otherwise (to avoid counting the unfolded n/2 frequency class twice with even sample sizes).

```python
fs_ery_folded = fs_ery.copy() # make a copy of the UNfolded spectrum
n = len(fs_ery)-1
for i in range(len(fs_ery)):
    fs_ery_folded[i] += fs_ery[n-i]
    if i == n/2.0:
        fs_ery_folded[i] /= 2
fs_ery_folded[0:19]
```

    Spectrum([-- 6105.194819999999 5341.542565000001 3017.369213 2626.2036799999996
     2058.5859899999996 1417.643105 1188.2540329999997 1690.4638210000003
     725.2838479999999 833.19685 1105.940877 810.9534290000001 959.861004
     570.4580109999999 858.9356979999999 730.323284 544.495498 577.679207], folded=False, pop_ids=['ery'])

```python
isinstance(fs_ery_folded, pylab.ndarray)
```

    True

```python
mask = [True]
mask.extend([False] * 18)
mask.extend([True] * 18)
print mask
print sum(mask)
```

    [True, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True]
    19

```python
mask = [True] * 37
for i in range(len(mask)):
    if i > 0 and i < 19:
        mask[i] = False
print mask
print sum(mask)
```

    [True, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True]
    19

Here is how to flatten an array of arrays with list comprehension:

```python
mask = [[True], [False] * 18, [True] * 18]
print mask
```

    [[True], [False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False], [True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True]]

```python
print [elem for a in mask for elem in a]
```

    [True, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True]

Set new mask for the folded spectrum:

```python
fs_ery_folded.mask = mask
```

```python
fs_ery_folded.folded = True
```

```python
fs_ery_folded - fs_ery.fold()
```

    Spectrum([-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
     -- -- -- -- -- -- -- -- -- -- -- --], folded=True, pop_ids=['ery'])

The `fold()` function works correctly for 1D spectra, at least. How about 2D spectra?

$$
\eta_{i,j} = \frac{\zeta_{i,j} + \zeta_{n-i, m-j}}{1 + \delta_{i, n-i; j, m-j}}
            \qquad 1 \le i+j \le \Big[\frac{n+m}{2}\Big]
$$

```python
EryPar_unfolded_2dsfs.sample_sizes
```

    array([36, 36])

```python
EryPar_unfolded_2dsfs._total_per_entry()
```

    array([[ 0,  1,  2, ..., 34, 35, 36],
           [ 1,  2,  3, ..., 35, 36, 37],
           [ 2,  3,  4, ..., 36, 37, 38],
           ...,
           [34, 35, 36, ..., 68, 69, 70],
           [35, 36, 37, ..., 69, 70, 71],
           [36, 37, 38, ..., 70, 71, 72]])

```python
# copy the unfolded 2D spectrum for folding
import copy
sfs2d_folded = copy.deepcopy(EryPar_unfolded_2dsfs)
```

```python
n = len(sfs2d_folded)-1
m = len(sfs2d_folded[0])-1
for i in range(n+1):
    for j in range(m+1):
        sfs2d_folded[i,j] += sfs2d_folded[n-i, m-j]
        if i == n/2.0 and j == m/2.0:
            sfs2d_folded[i,j] /= 2
```

```python
mask = sfs2d_folded._total_per_entry() > (n+m)/2
mask
```

    array([[False, False, False, ..., False, False, False],
           [False, False, False, ..., False, False,  True],
           [False, False, False, ..., False,  True,  True],
           ...,
           [False, False, False, ...,  True,  True,  True],
           [False, False,  True, ...,  True,  True,  True],
           [False,  True,  True, ...,  True,  True,  True]], dtype=bool)

```python
sfs2d_folded.mask = mask
sfs2d_folded.fold = True
```

```python
dadi.Plotting.plot_single_2d_sfs(sfs2d_folded, vmin=1)
```

I am going to go through every step in the `fold` function of dadi:

```python
# copy the unfolded 2D spectrum for folding
import copy
sfs2d_unfolded = copy.deepcopy(EryPar_unfolded_2dsfs)
```

```python
total_samples = pylab.sum(sfs2d_unfolded.sample_sizes)
total_samples
```

```python
total_per_entry = dadi.Spectrum(sfs2d_unfolded._total_per_entry(), pop_ids=['ery', 'par'])
#total_per_entry.pop_ids = ['ery', 'par']
dadi.Plotting.plot_single_2d_sfs(total_per_entry, vmin=1)
```

```python
total_per_entry = sfs2d_unfolded._total_per_entry()
total_per_entry
```

    array([[ 0,  1,  2, ..., 34, 35, 36],
           [ 1,  2,  3, ..., 35, 36, 37],
           [ 2,  3,  4, ..., 36, 37, 38],
           ...,
           [34, 35, 36, ..., 68, 69, 70],
           [35, 36, 37, ..., 69, 70, 71],
           [36, 37, 38, ..., 70, 71, 72]])

```python
where_folded_out = total_per_entry > total_samples/2
where_folded_out
```

    array([[False, False, False, ..., False, False, False],
           [False, False, False, ..., False, False,  True],
           [False, False, False, ..., False,  True,  True],
           ...,
           [False, False, False, ...,  True,  True,  True],
           [False, False,  True, ...,  True,  True,  True],
           [False,  True,  True, ...,  True,  True,  True]], dtype=bool)

```python
original_mask = sfs2d_unfolded.mask
original_mask
```

    array([[ True, False, False, ..., False, False, False],
           [False, False, False, ..., False, False, False],
           [False, False, False, ..., False, False, False],
           ...,
           [False, False, False, ..., False, False, False],
           [False, False, False, ..., False, False, False],
           [False, False, False, ..., False, False,  True]], dtype=bool)

```python
pylab.logical_or([True, False, True], [False, False, True])
```

    array([ True, False,  True], dtype=bool)

```python
# get the number of elements along each axis
sfs2d_unfolded.shape
```

```python
[slice(None, None, -1) for i in sfs2d_unfolded.shape]
```

    [slice(None, None, -1), slice(None, None, -1)]

```python
matrix = pylab.array([
    [1, 2, 3, 4],
    [5, 6, 7, 8],
    [9, 10, 11, 12]
])
reverse_slice = [slice(None, None, -1) for i in matrix.shape]
reverse_slice
```

    [slice(None, None, -1), slice(None, None, -1)]

```python
matrix[reverse_slice]
```

    array([[12, 11, 10,  9],
           [ 8,  7,  6,  5],
           [ 4,  3,  2,  1]])

```python
matrix[::-1,::-1]
```

    array([[12, 11, 10,  9],
           [ 8,  7,  6,  5],
           [ 4,  3,  2,  1]])

With the variable length list of slice objects, one can generalise the reverse of arrays with any dimensions.

```python
final_mask = pylab.logical_or(original_mask, dadi.Numerics.reverse_array(original_mask))
final_mask
```

    array([[ True, False, False, ..., False, False, False],
           [False, False, False, ..., False, False, False],
           [False, False, False, ..., False, False, False],
           ...,
           [False, False, False, ..., False, False, False],
           [False, False, False, ..., False, False, False],
           [False, False, False, ..., False, False,  True]], dtype=bool)

Here, folding doesn't mask new cells.

```python
?pylab.where
```

```python
pylab.where(matrix < 6, matrix, 0)
```

    array([[1, 2, 3, 4],
           [5, 0, 0, 0],
           [0, 0, 0, 0]])

```python
# this takes the part of the spectrum that is non-sensical if the derived allele is not known
# and sets the rest to 0
print pylab.where(where_folded_out, sfs2d_unfolded, 0)
```

    [[   0.          0.          0.       ...,    0.          0.          0.      ]
     [   0.          0.          0.       ...,    0.          0.         33.96861 ]
     [   0.          0.          0.       ...,    0.          1.593931
        26.499772]
     ...,
     [   0.          0.          0.       ...,   21.754621   30.856007
       232.64872 ]
     [   0.          0.          3.966959 ...,   16.189377   19.283053
       298.187262]
     [   0.         17.555369    0.       ...,  166.433356  104.865969
       287.751595]]

```python
# let's plot the bit of the spectrum that we are going to fold onto the rest:
dadi.Plotting.plot_single_2d_sfs(dadi.Spectrum(pylab.where(where_folded_out, sfs2d_unfolded, 0)), vmin=1)
```

```python
# now let's reverse this 2D array, i. e. last row first and last element of each row first:
_reversed = dadi.Numerics.reverse_array(pylab.where(where_folded_out, sfs2d_unfolded, 0))
_reversed
```

    array([[ 287.751595,  104.865969,  166.433356, ...,    0.      ,
              17.555369,    0.      ],
           [ 298.187262,   19.283053,   16.189377, ...,    3.966959,
               0.      ,    0.      ],
           [ 232.64872 ,   30.856007,   21.754621, ...,    0.      ,
               0.      ,    0.      ],
           ...,
           [  26.499772,    1.593931,    0.      , ...,    0.      ,
               0.      ,    0.      ],
           [  33.96861 ,    0.      ,    0.      , ...,    0.      ,
               0.      ,    0.      ],
           [   0.      ,    0.      ,    0.      , ...,    0.      ,
               0.      ,    0.      ]])

```python
dadi.Plotting.plot_single_2d_sfs(dadi.Spectrum(_reversed), vmin=1)
```

The transformation we have done with the upper-right diagonal 2D array above should be identical to projecting it across a vertical center line (creating an upper left triangular matrix) and then projecting it across a horizontal center line (creating the final lower left triangular matrix). Note, that this is not like mirroring the upper-right triangular 2D array across the 36-36 diagonal!

```python
# This shall now be added to the original unfolded 2D spectrum.
sfs2d_folded = pylab.ma.masked_array(sfs2d_unfolded.data + _reversed)
```

```python
dadi.Plotting.plot_single_2d_sfs(dadi.Spectrum(sfs2d_folded), vmin=1)
```

```python
sfs2d_folded.data
```

    array([[  1.07020142e+06,   6.65131557e+03,   9.53420133e+03, ...,
              5.92167010e+01,   1.90757620e+01,   2.47011103e+02],
           [  4.40735219e+03,   4.53932731e+02,   4.09549382e+02, ...,
              3.96696000e+00,   9.70266300e+00,   3.39686100e+01],
           [  3.74655618e+03,   2.95735068e+02,   2.61799583e+02, ...,
              2.00000000e-06,   1.59393100e+00,   2.64997720e+01],
           ...,
           [  2.66996610e+01,   1.59393100e+00,   0.00000000e+00, ...,
              2.17546210e+01,   3.08560070e+01,   2.32648720e+02],
           [  7.82533720e+01,   1.00000000e-06,   3.96695900e+00, ...,
              1.61893770e+01,   1.92830530e+01,   2.98187262e+02],
           [  1.70901478e+02,   1.75553690e+01,   0.00000000e+00, ...,
              1.66433356e+02,   1.04865969e+02,   2.87751595e+02]])

```python
sfs2d_folded.data[where_folded_out] = 0
sfs2d_folded.data
```

    array([[  1.07020142e+06,   6.65131557e+03,   9.53420133e+03, ...,
              5.92167010e+01,   1.90757620e+01,   2.47011103e+02],
           [  4.40735219e+03,   4.53932731e+02,   4.09549382e+02, ...,
              3.96696000e+00,   9.70266300e+00,   0.00000000e+00],
           [  3.74655618e+03,   2.95735068e+02,   2.61799583e+02, ...,
              2.00000000e-06,   0.00000000e+00,   0.00000000e+00],
           ...,
           [  2.66996610e+01,   1.59393100e+00,   0.00000000e+00, ...,
              0.00000000e+00,   0.00000000e+00,   0.00000000e+00],
           [  7.82533720e+01,   1.00000000e-06,   0.00000000e+00, ...,
              0.00000000e+00,   0.00000000e+00,   0.00000000e+00],
           [  1.70901478e+02,   0.00000000e+00,   0.00000000e+00, ...,
              0.00000000e+00,   0.00000000e+00,   0.00000000e+00]])

```python
dadi.Plotting.plot_single_2d_sfs(dadi.Spectrum(sfs2d_folded), vmin=1)
```

```python
sfs2d_folded.shape
```

```python
where_ambiguous = (total_per_entry == total_samples/2.0)
where_ambiguous
```

    array([[False, False, False, ..., False, False,  True],
           [False, False, False, ..., False,  True, False],
           [False, False, False, ...,  True, False, False],
           ...,
           [False, False,  True, ..., False, False, False],
           [False,  True, False, ..., False, False, False],
           [ True, False, False, ..., False, False, False]], dtype=bool)

SNP's with joint frequencies in the True cells are counted twice at the moment due to the folding and the fact that the sample sizes are even.

```python
# this extracts the diagonal values from the UNfolded spectrum and sets the rest to 0
ambiguous = pylab.where(where_ambiguous, sfs2d_unfolded, 0)
dadi.Plotting.plot_single_2d_sfs(dadi.Spectrum(ambiguous), vmin=1)
```

These are the values in the diagonal before folding.

```python
reversed_ambiguous = dadi.Numerics.reverse_array(ambiguous)
dadi.Plotting.plot_single_2d_sfs(dadi.Spectrum(reversed_ambiguous), vmin=1)
```

These are the values that got added to the diagonal during folding. Comparing with the previous plot, one can see for instance that the value in the (0, 36) class got added to the value in the (36, 0) class and vice versa. The two frequency classes are equivalent, since it is arbitrary which allele we call minor in the total sample (of 72 gene copies). These SNP's are therefore counted twice.

```python
a = -1.0*ambiguous + 0.5*ambiguous + 0.5*reversed_ambiguous
b = -0.5*ambiguous + 0.5*reversed_ambiguous
a == b
```

    array([[ True,  True,  True, ...,  True,  True,  True],
           [ True,  True,  True, ...,  True,  True,  True],
           [ True,  True,  True, ...,  True,  True,  True],
           ...,
           [ True,  True,  True, ...,  True,  True,  True],
           [ True,  True,  True, ...,  True,  True,  True],
           [ True,  True,  True, ...,  True,  True,  True]], dtype=bool)

```python
sfs2d_folded += -0.5*ambiguous + 0.5*reversed_ambiguous
```

```python
final_mask = pylab.logical_or(final_mask, where_folded_out)
final_mask
```

    array([[ True, False, False, ..., False, False, False],
           [False, False, False, ..., False, False,  True],
           [False, False, False, ..., False,  True,  True],
           ...,
           [False, False, False, ...,  True,  True,  True],
           [False, False,  True, ...,  True,  True,  True],
           [False,  True,  True, ...,  True,  True,  True]], dtype=bool)

```python
sfs2d_folded = dadi.Spectrum(sfs2d_folded, mask=final_mask, data_folded=True, pop_ids=['ery', 'par'])
```

```python
pylab.rcParams['figure.figsize'] = [12.0, 8.0]
```

```python
dadi.Plotting.plot_single_2d_sfs(sfs2d_folded, vmin=1)
```

```python

```

```python

```

---

### Model specification

```python

```


```python
%run ../../common/import_all.py

from common.setup_notebook import set_css_style, setup_matplotlib, config_ipython
config_ipython()
setup_matplotlib()
set_css_style()
```

# AdaBoost

## The gist

AdaBoost, shortening for *adaptive boosting* is a boosting ensemble method used both in classification and in regression problems. The authors won the Goedel prize for the work in 2003, whose original paper is in [[1]](#paper) but a nice reading over the general idea, by the same authors is in [[2]](#boosting-gentle).

The idea is to fit a sequence of weak learners (a weak learner is one that is just slightly better than a random guesser) on repeatedly modified versions of the training set, then combine their predictions through a weighted majority voting system.

In the first iteration, you give the same weight to all $n$ training samples, $w_i = \frac{1}{n}$; in successive iterations the weights are modified in such a way that the training samples which were incorrectly predicted in the previous iteration will see their weights increased while those which were correctly predicted will see their weights decreased. This way, the weak learner is forced to focus on the points more difficult to predict: this is the adaptive part of the idea. The resulting combination of these weak learners will be a strong learner.

As per current literature, AdaBoost with decision trees as the learners is considered one of the best classifiers.

## The algorithm

This part follows [the Wikipedia page](https://en.wikipedia.org/wiki/AdaBoost). Let's see we have a binary classification problem, class variables being $y_i \in \{1, -1\}$ and sample points $(x_1, y_1), \ldots, (x_n, y_n)$, where $x_i \in X$ (feature matrix).

* Call a weak learner $h$
* At iteration $t$, you'll have built a combination of weak learners into a strong learner

$$
H_t(x_i) = \sum_{i=1}^t \alpha_i h_i(x_i) \ ,
$$

so that

$$
H_t(x_i) = H_{t-1}(x_i) + \alpha_t h_t(x_i)
$$

This way we have built a linear combination of weak learners over several iterations. The weights $\alpha_i$ of each learner remain to be attributed in the most effective way. If we consider the error $E$ on $H_t$ as the sum of the exponential losses on each data point,

$$
E = \sum_{i=1}^n e^{-y_i H_t(x_i)}
$$

(note that the argument of the exponential will be a 1 if the point is well classified and a -1 if not) which by posing $w_i^1 = 1$ and $w_i^t = e^{-y_i H_{t-1}(x_i)}$ ($w_i$ represents a weight to the error) we can rewrite as

$$
E = \sum_{i=1}^n w_i^t e^{-y_i \alpha_t h_t(x_i)} \ .
$$

We can now split the sum between the points which are well classified and those misclassified:

$$
\begin{align}
    E &= \sum_{i: y_i = h_t(x_i)} w_i^t e^{-\alpha_t} + \sum_{i: y_i \neq h_t(x_i)} w_i^t e^{\alpha^t} \\
      &= \sum_{i=1}^n w_i^t e^{-\alpha_t} + \sum_{i: y_i \neq h_t(x_i)} w_i^t [e^\alpha_t - e^{-\alpha_t}]
\end{align}
$$

In the last expression above, the only part depending on the weak classifiers is the second sum, so the weak classifier that minimises $E$ is the one minimising this sum, which means the one that minimises $\sum_{i: y_i \neq h_t(x_i)} w_i^t$, so the one with the lowest weighted error.

If we derive $E$ with respect to $\alpha_t$, we obtain

$$
\alpha_t = \frac{1}{2} \log{\frac{\sum_{i: y_i = h_t(x_i)} w_i^t}{\sum_{i: y_i \neq h_t(x_i)} w_i^t}}
$$

Now, the weighted error rate of the weak classifiers is

$$
\epsilon_t = \frac{\sum_{i: y_i \neq h_t(x_i)} w_i^t}{\sum_{i=1}^n w_i^t} \ ,
$$

so it follows that we can write the $\alpha_t$ which minimises $E$ as

$$
\alpha_t = \frac{1}{2} \log{\frac{1-\epsilon_t}{\epsilon_t}}
$$

which is $\frac{1}{2}$ times the logit negative function.

To summarise then, the AdaBoost algorithm consists of

1. Choose the weak classifier which minimised the error $E$
2. Use it to compute the classifiers weighted error $\epsilon_t$
3. Use this to compute the weights $\alpha_t$
4. use this to compute the boosted (strong) classifier $H_t$

Note that there exist several variants of the original AdaBoost.

## References

1.  Y Freund, R E Schapire, [**A decision-theoretic generalization of on-line learning and an application to boosting**](http://cns.bu.edu/~gsc/CN710/FreundSc95.pdf), *J of computer and system sciences* 55.1, 1997
2.  Y Freund, R E Schapire, [**A short introduction to boosting**](https://cseweb.ucsd.edu/~yfreund/papers/IntroToBoosting.pdf)

```python

```