Question: Vector-valued function interpolation using NumPy/SciPy Is there a way to interpolate a vector-valued function using NumPy/SciPy? There are plenty of offerings that work on scalar-valued functions, and I guess I can use one of those to estimate each component of the vector separately, but is there a way to do it more efficiently? To elaborate, I have a function f(x) = V, where x is scalar and V is a vector. I also have a collection of xs and their corresponding Vs. I would like to use it to interpolate and estimate V for an arbitrary x.

Response:
The interpolation function scipy.interpolate.interp1d also works on vector-valued data for the interpolant (not for vector-valued argument data though). Thus, as long as x is scalar, you can use it directly. The following code is a slight extension of the example given in the scipy documentation:
```

```python
from scipy.interpolate import interp1d
```

```python
x = np.linspace(0, 10, 10)
```

```python
y = np.array([np.exp(-x/3.0), 2*x])
```

```python
f = interp1d(x, y)
```

```python
f(2)
#Output
#array([ 0.51950421,  4.        ])
```

```python
np.array([np.exp(-2/3.0), 2*2])
#Output
#array([ 0.51341712,  4.        ])
#```
#Note that 2 is not in the argument vector x, thus the interpolation error for the first component in y in this example.
```


Question:
Fast(er) numpy fancy indexing and reduction? I'm trying to use and accelerate fancy indexing to "join" two arrays and sum over one of results' axis. Something like this:
```
$ ipython

```python
import numpy as np
```

```python
ne, ds = 12, 6
```

```python
i = np.random.randn(ne, ds).astype('float32')
```

```python
t = np.random.randint(0, ds, size=(1e5, ne)).astype('uint8')
```

```python
%timeit i[np.arange(ne), t].sum(-1)
10 loops, best of 3: 44 ms per loop
```
Is there a simple way to accelerate the statement in In [5] ? Should I go with OpenMP and something like scipy.weave or Cython's prange ?
```

Response:
numpy.take is much faster than fancy indexing for some reason. The only trick is that it treats the array as flat.
```

```python
a = np.random.randn(12,6).astype(np.float32)
```

```python
c = np.random.randint(0,6,size=(1e5,12)).astype(np.uint8)
```

```python
r = np.arange(12)
```

```python
%timeit a[r,c].sum(-1)
10 loops, best of 3: 46.7 ms per loop
```

```python
rr, cc = np.broadcast_arrays(r,c)
```

```python
flat_index = rr*a.shape[1] + cc
```

```python
%timeit a.take(flat_index).sum(-1)
100 loops, best of 3: 5.5 ms per loop
```

```python
(a.take(flat_index).sum(-1) == a[r,c].sum(-1)).all()
#Output
#True
#```
#I think the only other way you're going to see much of a speed improvement beyond this would be to write a custom kernel for a GPU using something like PyCUDA.
```


Question:
Matlab filter() with SciPy lfilter() According to their documentation for Matlab filter() and SciPy lfilter(), it seems like they should be "compatible". However I have a problem, porting larger Matlab code in Python, for which I get ValueError: object of too small depth for desired array. As I can't think of how I can present my source without complicating it, I'll use the example provided in Matlab's documentation:
```
data = [1:0.2:4]';
windowSize = 5;
filter(ones(1,windowSize)/windowSize,1,data)
```
which I translate in Python to:
```
import numpy as np
from scipy.signal import lfilter

data = np.arange(1, 4.1, 0.2)
windowSize = 5
lfilter(np.ones((1, windowSize)) / windowSize, 1, data)
```
In this case I get: ValueError: object too deep for desired array Why do I get these errors?

Response:
Is there a reason you're adding a an extra dimension when creating your array of ones? Is this what you need:
```
lfilter(np.ones(windowSize) / windowSize, 1, data)
```


Question:
Convert an equation to Python I have several equations and need to convert it into Python. The problem is that I tried to plot a graph according to the equation. However, the graph that I get is not the same as the original one. In the paper, the equation of error probability for MIM attack is given by: First Image Second Image The equation to calculate the error probability of PNS attack is given by: Where the region condition satisfied: The error probability of PNS attack should be plotted like this: My question: How to insert equation 8.1 into equation 8.5? This is my python code according to equation 8.5:
```
import matplotlib.pyplot as plt
import math
import numpy as np
from scipy.special import iv,modstruve

x=[0, 5, 10, 15, 20]
t= 0.9
x = np.array(x)
y = (np.exp(x*t/2)*(iv(0, x*t/2) - modstruve(0,x*t/2))-1)/(np.exp(x*t/2-1))

plt.plot(x, y, label='Normal')
plt.xlabel('Mean photon number N')
plt.ylabel('Error probabiity')
plt.scatter(x,y)
plt.title('N/2')
plt.ylim([0, 0.5])
plt.legend()
plt.show()
```
Please help me regarding this matter. Thank you.

Response:
I updated your code by computing y for N1 and for N2 with the formula given in the image. This give me y1 and y2 which I believe are the components of the drawn function f(y1,y2). Yet, without the rest of the paper I cannot figure out exactly what is drawn on the image you provided. The following code generates an image quite similar with f : y1,y2 -> y1+y2:
```
import matplotlib.pyplot as plt
import numpy as np
from scipy.special import iv, modstruve

x = range(0, 20, 1)
# t= 0.1
for t, color in zip([0.9, 0.1, 0.5], ['b', 'g', 'r']):
    x1 = (1 - t) * np.array(x)
    y1 = (np.exp(x1 / 2) * (iv(0, x1 / 2) - modstruve(0, x1 / 2)) - 1) / (np.exp(x1) - 1)
    x2 = (1 - t) * t * np.array(x)
    y2 = (np.exp(x2 / 2) * (iv(0, x2 / 2) - modstruve(0, x2 / 2)) - 1) / (np.exp(x2) - 1)

    y = y1 + y2

    plt.plot(x, y, label=t, color=color)
    plt.scatter(x, y, color=color)

# N1 = N2
x1 = np.array(x) / 2
y1 = (np.exp(x1 / 2) * (iv(0, x1 / 2) - modstruve(0, x1 / 2)) - 1) / (np.exp(x1) - 1)
x2 = np.array(x) / 2
y2 = (np.exp(x2 / 2) * (iv(0, x2 / 2) - modstruve(0, x2 / 2)) - 1) / (np.exp(x2) - 1)
y = y1 + y2
plt.plot(x, y, label="N1=N2=N/2", color='k')
plt.scatter(x, y, color='k')

plt.xlabel('Mean photon number N')
plt.ylabel('Error probabiity')
plt.title('N/2')
plt.ylim([0, 0.35])
plt.legend()
plt.show()
```


Question: How to append to .mat file using scipy.io.savemat? so when I use the savemat command it tends to overwrite the file. Is there a possible way to append instead of overwriting? I know a work-around would be to put everything into a list, and then convert it to a dictionary. That won't work for me because I am trying to be RAM efficient. Doing online search I found this How NOT to overwrite the .mat file when using scipy.io.savemat()? that won't work either because it involves pulling a data file in your ram memory to append it every single loop, which seems stupid from the speed perspective. I thought about appending to a numpy binary file, then pulling that in and saving it to a .mat file? I am not sure if this would be more RAM efficient than the first option though. thanks!

Response:
According to savemat docs: file_name : str or file-like object So you can open the file in append mode, and write, e.g.
```
io.savemat('temp.mat',{'data':np.ones(10)})  # write
with open('temp.mat','ab') as f:
    io.savemat(f, {'newdata':np.arange(5)})   # append
print io.loadmat('temp.mat').keys()           # read
# dict_keys(['data', '__globals__', 'newdata', '__header__', '__version__'])
```
Or the full description:
```
{'data': array([[ 1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.]]),
 '__globals__': [],
 'newdata': array([[0, 1, 2, 3, 4]]),
 '__header__': b'MATLAB 5.0 MAT-file Platform: posix, Created on: Fri Mar 13 14:14:33 2015',
 '__version__': '1.0'}
```
A note in https://github.com/scipy/scipy/blob/master/scipy/io/matlab/mio5.py#L34 suggests that there is a problem with appending when there's a function in the data file, but this indicates that appending isn't a problem if we are just saving arrays. But maybe further search of the scipy issues is in order.


Question: Finding the roots of a large number of functions with one variable I'm working with Python/numpy/scipy to write a small ray tracer. Surfaces are modelled as two-dimensional functions giving a height above a normal plane. I reduced the problem of finding the point of intersection between ray and surface to finding the root of a function with one variable. The functions are continuous and continuously differentiable. Is there a way to do this more efficiently than simply looping over all the functions, using scipy root finders (and maybe using multiple processes)? Edit: The functions are the difference between a linear function representing the ray and the surface function, constrained to a plane of intersection.

Response:
The following example shows calculating the roots for 1 million copies of the function x**(a+1) - b (all with different a and b) in parallel using the bisection method. Takes about ~12 seconds here.
```
import numpy

def F(x, a, b):
    return numpy.power(x, a+1.0) - b

N = 1000000

a = numpy.random.rand(N)
b = numpy.random.rand(N)

x0 = numpy.zeros(N)
x1 = numpy.ones(N) * 1000.0

max_step = 100
for step in range(max_step):
    x_mid = (x0 + x1)/2.0
    F0 = F(x0, a, b)
    F1 = F(x1, a, b)
    F_mid = F(x_mid, a, b)
    x0 = numpy.where( numpy.sign(F_mid) == numpy.sign(F0), x_mid, x0 )
    x1 = numpy.where( numpy.sign(F_mid) == numpy.sign(F1), x_mid, x1 )
    error_max = numpy.amax(numpy.abs(x1 - x0))
    print "step=%d error max=%f" % (step, error_max)
    if error_max < 1e-6: break
```
The basic idea is to simply run all the usual steps of a root finder in parallel on a vector of variables, using a function that can be evaluated on a vector of variables and equivalent vector(s) of parameters that define the individual component functions. Conditionals are replaced with a combination of masks and numpy.where(). This can continue until all roots have been found to the required precision, or alternately until enough roots have been found that it is worth to remove them from the problem and continue with a smaller problem that excludes those roots. The functions I chose to solve are arbitrary, but it helps if the functions are well-behaved; in this case all functions in the family are monotonic and have exactly one positive root. Additionally, for the bisection method we need guesses for the variable that give different signs of the function, and those happen to be quite easy to come up with here as well (the initial values of x0 and x1). The above code uses perhaps the simplest root finder (bisection), but the same technique could be easily applied to Newton-Raphson, Ridder's, etc. The fewer conditionals there are in a root finding method, the better suited it is to this. However, you will have to reimplement any algorithm you want, there is no way to use an existing library root finder function directly. The above code snippet is written with clarity in mind, not speed. Avoiding the repetition of some calculations, in particular evaluating the function only once per iteration instead of 3 times, speeds this up to 9 seconds, as follows:
```
...
F0 = F(x0, a, b)
F1 = F(x1, a, b)

max_step = 100
for step in range(max_step):
    x_mid = (x0 + x1)/2.0
    F_mid = F(x_mid, a, b)
    mask0 = numpy.sign(F_mid) == numpy.sign(F0)
    mask1 = numpy.sign(F_mid) == numpy.sign(F1)
    x0 = numpy.where( mask0, x_mid, x0 )
    x1 = numpy.where( mask1, x_mid, x1 )
    F0 = numpy.where( mask0, F_mid, F0 )
    F1 = numpy.where( mask1, F_mid, F1 )
...
```
For comparison, using scipy.bisect() to find one root at a time takes ~94 seconds:
```
for i in range(N):
    x_root = scipy.optimize.bisect(lambda x: F(x, a[i], b[i]), x0[i], x1[i], xtol=1e-6)
```


Question:
correct usage of scipy.optimize.fmin_bfgs I am playing around with logistic regression in Python. I have implemented a version where the minimization of the cost function is done via gradient descent, and now I'd like to use the BFGS algorithm from scipy (scipy.optimize.fmin_bfgs). I have a set of data (features in matrix X, with one sample in every row of X, and correpsonding labels in vertical vector y). I am trying to find parameters Theta to minimize: I have trouble understanding how fmin_bfgs works exactly. As far as I get it, I have to pass a function to be minimized and a set of initial values for Thetas. I do the following:
```
initial_values = numpy.zeros((len(X[0]), 1))
myargs = (X, y)
theta = scipy.optimize.fmin_bfgs(computeCost, x0=initial_values, args=myargs)
```
where computeCost calculates J(Thetas) as illustrated above. But I get some index-related errors, so I think I am not supplying what fmin_bfgs expects. Can anyone shed some light on this?

Response: After wasting hours on it, solved again by power of posting...I was defining computeCost(X, y, Thetas), but as Thetas is the target parameter for optimization, it should have been the first parameter in the signature. Fixed and works!


Question:
Scipy Sparse Cumsum Suppose I have a scipy.sparse.csr_matrix representing the values below
```
[[0 0 1 2 0 3 0 4]
 [1 0 0 2 0 3 4 0]]
```
I want to calculate the cumulative sum of non-zero values in-place, which would change the array to:
```
[[0 0 1 3 0 6 0 10]
 [1 0 0 3 0 6 10 0]]
```
The actual values are not 1, 2, 3, ... The number of non-zero values in each row are unlikely to be the same. How to do this fast? Current program:
```
import scipy.sparse
import numpy as np

# sparse data
a = scipy.sparse.csr_matrix(
    [[0,0,1,2,0,3,0,4],
     [1,0,0,2,0,3,4,0]],
    dtype=int)

# method
indptr = a.indptr
data = a.data
for i in range(a.shape[0]):
    st = indptr[i]
    en = indptr[i + 1]
    np.cumsum(data[st:en], out=data[st:en])

# print result
print(a.todense())
```
Result:
```
[[ 0  0  1  3  0  6  0 10]
 [ 1  0  0  3  0  6 10  0]]
```

Response:
How about doing this instead
```
a = np.array([[0,0,1,2,0,3,0,4],
              [1,0,0,2,0,3,4,0]], dtype=int)

b = a.copy()
b[b > 0] = 1
z = np.cumsum(a,axis=1)
print(z*b)
```
Yields
```
array([[ 0,  0,  1,  3,  0,  6,  0, 10],
   [ 1,  0,  0,  3,  0,  6, 10,  0]])
```
Doing sparse
```
def sparse(a):
    a = scipy.sparse.csr_matrix(a)

    indptr = a.indptr
    data = a.data
    for i in range(a.shape[0]):
        st = indptr[i]
        en = indptr[i + 1]
        np.cumsum(data[st:en], out=data[st:en])

```python
%timeit sparse(a)
10000 loops, best of 3: 167 µs per loop
```
Using multiplication
```
def mult(a):
    b = a.copy()
    b[b > 0] = 1
    z = np.cumsum(a, axis=1)
    z * b
```

```python
%timeit mult(a)
100000 loops, best of 3: 5.93 µs per loop
```
```


Question:
can't understand scipy.sparse.csr_matrix example I can't wrap my head around csr_matrix examples in scipy documentation: https://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.csr_matrix.html Can someone explain how this example work?
```

```python
row = np.array([0, 0, 1, 2, 2, 2])
```

```python
col = np.array([0, 2, 2, 0, 1, 2])
```

```python
data = np.array([1, 2, 3, 4, 5, 6])
```

```python
csr_matrix((data, (row, col)), shape=(3, 3)).toarray()
#Output
#array([[1, 0, 2],
#       [0, 0, 3],
#       [4, 5, 6]])
#```
#I believe this is following this format. csr_matrix((data, (row_ind, col_ind)), [shape=(M, N)]) where data, row_ind and col_ind satisfy the relationship a[row_ind[k], col_ind[k]] = data[k]. What is a here?
```

Response:
row = np.array([0, 0, 1, 2, 2, 2]) col = np.array([0, 2, 2, 0, 1, 2]) data = np.array([1, 2, 3, 4, 5, 6]) from the above arrays; for k in 0~5 a[row_ind[k], col_ind[k]] = data[k]
```
a
row[0],col[0] = [0,0] = 1 (from data[0])
row[1],col[1] = [0,2] = 2 (from data[1])
row[2],col[2] = [1,2] = 3 (from data[2])
row[3],col[3] = [2,0] = 4 (from data[3])
row[4],col[4] = [2,1] = 5 (from data[4])
row[5],col[5] = [2,2] = 6 (from data[5])
```
so let's arrange matrix 'a' in shape(3X3)
```
a
   0  1  2
0 [1, 0, 2]
1 [0, 0, 3]
2 [4, 5, 6]
```


Question:
Broadcasting a python function on to numpy arrays Let's say we have a particularly simple function like
```
import scipy as sp
def func(x, y):
   return x + y
```
This function evidently works for several builtin python datatypes of x and y like string, list, int, float, array, etc. Since we are particularly interested in arrays, we consider two arrays:
```
x = sp.array([-2, -1, 0, 1, 2])
y = sp.array([-2, -1, 0, 1, 2])

xx = x[:, sp.newaxis]
yy = y[sp.newaxis, :]

```python
func(xx, yy)
#Output
#```
#this returns
#```
#array([[-4, -3, -2, -1,  0],
#  [-3, -2, -1,  0,  1],
#  [-2, -1,  0,  1,  2],
#  [-1,  0,  1,  2,  3],
#  [ 0,  1,  2,  3,  4]])
#```
#just as we would expect. Now what if one wants to throw in arrays as the inputs for the following function?
#```
#def func2(x, y):
#  if x > y:
#     return x + y
#  else:
#     return x - y
#```
#doing >>>func(xx, yy) would raise an error. The first obvious method that one would come up with is the sp.vectorize function in scipy/numpy. This method, nevertheless has been proved to be not very efficient. Can anyone think of a more robust way of broadcasting any function in general on to numpy arrays? If re-writing the code in an array-friendly fashion is the only way, it would help if you could mention it here too.
```

Response:
np.vectorize is a general way to convert Python functions that operate on numbers into numpy functions that operate on ndarrays. However, as you point out, it isn't very fast, since it is using a Python loop "under the hood". To achieve better speed, you have to hand-craft a function that expects numpy arrays as input and takes advantage of that numpy-ness:
```
import numpy as np

def func2(x, y):
    return np.where(x>y,x+y,x-y)

x = np.array([-2, -1, 0, 1, 2])
y = np.array([-2, -1, 0, 1, 2])

xx = x[:, np.newaxis]
yy = y[np.newaxis, :]

print(func2(xx, yy))
# [[ 0 -1 -2 -3 -4]
#  [-3  0 -1 -2 -3]
#  [-2 -1  0 -1 -2]
#  [-1  0  1  0 -1]
#  [ 0  1  2  3  0]]
```
Regarding performance: test.py:
```
import numpy as np

def func2a(x, y):
    return np.where(x>y,x+y,x-y)

def func2b(x, y):
    ind=x>y
    z=np.empty(ind.shape,dtype=x.dtype)
    z[ind]=(x+y)[ind]
    z[~ind]=(x-y)[~ind]
    return z

def func2c(x, y):
    # x, y= x[:, None], y[None, :]
    A, L= x+ y, x<= y
    A[L]= (x- y)[L]
    return A

N=40
x = np.random.random(N)
y = np.random.random(N)

xx = x[:, np.newaxis]
yy = y[np.newaxis, :]
```
Running: With N=30:
```
% python -mtimeit -s'import test' 'test.func2a(test.xx,test.yy)'
1000 loops, best of 3: 219 usec per loop

% python -mtimeit -s'import test' 'test.func2b(test.xx,test.yy)'
1000 loops, best of 3: 488 usec per loop

% python -mtimeit -s'import test' 'test.func2c(test.xx,test.yy)'
1000 loops, best of 3: 248 usec per loop
```
With N=1000:
```
% python -mtimeit -s'import test' 'test.func2a(test.xx,test.yy)'
10 loops, best of 3: 93.7 msec per loop

% python -mtimeit -s'import test' 'test.func2b(test.xx,test.yy)'
10 loops, best of 3: 367 msec per loop

% python -mtimeit -s'import test' 'test.func2c(test.xx,test.yy)'
10 loops, best of 3: 186 msec per loop
```
This seems to suggest that func2a is slightly faster than func2c (and func2b is horribly slow).


Question:
Pandas error: 'DataFrame' object has no attribute 'loc' I am new to pandas and is trying the Pandas 10 minute tutorial with pandas version 0.10.1. However when I do the following, I get the error as shown below. print df works fine. Why is .loc not working? Code
```
import numpy as np
import pandas as pd

df = pd.DataFrame(np.random.randn(6,4), index=pd.date_range('20130101', periods=6), columns=['A','B','C','D'])
df.loc[:,['A', 'B']]
```
Error:
```
AttributeError                            Traceback (most recent call last)
<ipython-input-4-8513cb2c6dc7> in <module>()
----> 1 df.loc[:,['A', 'B']]

C:\Python27\lib\site-packages\pandas\core\frame.pyc in __getattr__(self, name)
   2044             return self[name]
   2045         raise AttributeError("'%s' object has no attribute '%s'" %
-> 2046                              (type(self).__name__, name))
   2047
   2048     def __setattr__(self, name, value):

AttributeError: 'DataFrame' object has no attribute 'loc'
```

Response: I came across this question when I was dealing with pyspark DataFrame. So, if you're also using pyspark DataFrame, you can convert it to pandas DataFrame using toPandas() method.


Question:
MAD results differ in pandas, scipy, and numpy I want to compute the MAD (median absolute deviation) which is defined by
```
MAD = median(|x_i - mean(x)|)
```
for a list of numbers x
```
x = list(range(0, 10)) + [1000]
```
However, the results differ significantly using numpy, pandas, and an hand-made implementation:
```
from scipy import stats
import pandas as pd
import numpy as np

print(stats.median_absolute_deviation(x, scale=1)) # prints 3.0

print(pd.Series(x).mad()) # prints 164.54

print(np.median(np.absolute(x - np.mean(x)))) # prints 91.0
```
What is wrong?

Response:
The median absolute deviation is defined as:
```
median(|x_i - median(x)|
```
The method mad in Pandas returns the mean absolute deviation instead. You can calculate MAD using following methods:
```
x = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 1000]

stats.median_absolute_deviation(x, scale=1)
# 3.0

np.median(np.absolute(x - np.median(x)))
# 3.0

x = pd.Series(x)
(x - x.median()).abs().median()
# 3.0
```


Question:
Can't understand the working of uniform_filter1d() function (imported from scipy.ndimage.filters) I read the scipy docs for the function here : scipy.ndimage.uniform_filter1d. However, when I tried using it, I couldn't wrap around my head on it's working. I read the docs, ran the example over there in the Python Shell, used my own example but still no progress. For eg:
```

```python
from scipy.ndimage import uniform_filter1d
```

```python
uniform_filter1d([2, 8, 0, 4, 1, 9, 9, 0], size=3)
#Output
#array([4, 3, 4, 1, 4, 6, 6, 3])
```

```python
uniform_filter1d([1, 2, 3, 4, 5, 6, 7, 8], size=3)
#Output
#array([1, 2, 3, 4, 5, 6, 7, 7])
#```
#When I saw the output of the second array, it felt like the function retained most of the array's elements. However in the second example it felt like barring 4 & 1 all the other elements in the output array were completely new. Thus I would like you to help me understand the working and the use of this function.
```

Response:
What this filter does is, according to size, to take the arithmetic average of each pixel with its neighbor. Size is the size of the sub-array to calculate arithmetic average. The standard for pixels without enough neighbors is to reflect. Let us go its process:
```
uniform_filter1d([1,2,3,4,5,6], size=3)

[1,2,3,4,5,6] # index 0, Reflect 1 : [1,1,2] -> average: 4/3 = 1
[1,2,3,4,5,6] # index 1, [1,2,3] -> average: 6/3 = 2
[1,2,3,4,5,6] # index 2, [2,3,4] -> average: 9/3 = 3
[1,2,3,4,5,6] # index 3, [3,4,5] -> average: 12/3 = 4
[1,2,3,4,5,6] # index 4, [4,5,6] -> average: 15/3 = 5
[1,2,3,4,5,6] # index 5, Reflect 6 : [5,6,6] -> average: 17/3 = 5

Result: [1,2,3,4,5,5]
```


Question:
Python finite boundary Voronoi cells I am trying to adapt a code I found on stackoverflow to create a voronoi cell with finite boundaries. I found the code below on https://stackoverflow.com/a/20678647/2443944 however my problem is that whilst the voronoi cells do not go off to infinity at the boundaries, they are still too far away. Even with a radius = 0, the ridge vertices are far too away. I ideally want the boundary voronoi vertices to be spaced around the same amount as the rest of the voronoi cells in the centre i.e. I want the sizes of the voronoi cells at the boundaries to be similar to the ones in the centre. The data points I am using are
```
points = [[-30.0, 30.370371], [-27.777777, 35.925926], [-34.444443, 58.51852], [-2.9629631, 57.777779], [-17.777779, 75.185181], [-29.25926, 58.148151], [-11.111112, 33.703705], [-11.481482, 40.0], [-27.037037, 40.0], [-7.7777777, 94.444443], [-2.2222223, 122.22222], [-20.370371, 106.66667], [1.1111112, 125.18518], [-6.2962961, 128.88889], [6.666667, 133.7037], [11.851852, 136.2963], [8.5185184, 140.74074], [20.370371, 92.962959], [17.777779, 114.81482], [12.962962, 97.037041], [13.333334, 127.77778], [22.592592, 120.37037], [16.296295, 127.77778], [11.851852, 50.740742], [20.370371, 54.814816], [19.25926, 47.40741], [32.59259, 122.96296], [20.74074, 130.0], [24.814816, 84.814819], [26.296295, 91.111107], [56.296295, 131.48149], [60.0, 141.85185], [32.222221, 136.66667], [53.703705, 147.03703], [87.40741, 196.2963], [34.074074, 159.62964], [34.444443, -2.5925925], [36.666668, -1.8518518], [34.074074, -7.4074073], [35.555557, -18.888889], [76.666664, -39.629627], [35.185184, -37.777779], [25.185184, 14.074074], [42.962959, 32.962963], [35.925926, 9.2592592], [52.222221, 77.777779], [57.777779, 92.222221], [47.037041, 92.59259], [82.222221, 54.074074], [48.888889, 24.444445], [35.925926, 47.777779], [50.740742, 69.259254], [51.111111, 51.851849], [56.666664, -12.222222], [117.40741, -4.4444447], [59.629631, -5.9259262], [66.666664, 134.07408], [91.481483, 127.40741], [66.666664, 141.48149], [53.703705, 4.0740738], [85.185181, 11.851852], [69.629631, 0.37037039], [68.518517, 99.259262], [75.185181, 100.0], [70.370369, 113.7037], [74.444443, 82.59259], [82.222221, 93.703697], [72.222221, 84.444443], [77.777779, 167.03703], [88.888893, 168.88889], [73.703705, 178.88889], [87.037041, 123.7037], [78.518517, 97.037041], [95.555557, 52.962959], [85.555557, 57.037041], [90.370369, 23.333332], [100.0, 28.51852], [88.888893, 37.037037], [87.037041, -42.962959], [89.259262, -24.814816], [93.333328, 7.4074073], [98.518517, 5.185185], [92.59259, 1.4814816], [85.925919, 153.7037], [95.555557, 154.44444], [92.962959, 150.0], [97.037041, 95.925919], [106.66667, 115.55556], [92.962959, 114.81482], [108.88889, 56.296295], [97.777779, 50.740742], [94.074081, 89.259262], [96.666672, 91.851852], [102.22222, 77.777779], [107.40741, 40.370369], [105.92592, 29.629629], [105.55556, -46.296295], [118.51852, -47.777779], [112.22222, -43.333336], [112.59259, 25.185184], [115.92592, 27.777777], [112.59259, 31.851852], [107.03704, -36.666668], [118.88889, -32.59259], [114.07408, -25.555555], [115.92592, 85.185181], [105.92592, 18.888889], [121.11111, 14.444445], [129.25926, -28.51852], [127.03704, -18.518518], [139.25926, -12.222222], [141.48149, 3.7037036], [137.03703, -4.814815], [153.7037, -26.666668], [-2.2222223, 5.5555558], [0.0, 9.6296301], [10.74074, 20.74074], [2.2222223, 54.074074], [4.0740738, 50.740742], [34.444443, 46.296295], [11.481482, 1.4814816], [24.074076, -2.9629631], [74.814819, 79.259254], [67.777779, 152.22223], [57.037041, 127.03704], [89.259262, 12.222222]]
points = np.array(points)
```
The pictures I return are below for a radius of radius = 0.

Response:
I guess you could achieve that by clipping your result by the convex hull of your points. To do that I would probably use the shapely module. Given the SO post you linked I assume you are using the voronoi_finite_polygons_2d function written in the post. So i think this could do the job:
```
import numpy as np
import matplotlib.pyplot as plt
from shapely.geometry import MultiPoint, Point, Polygon
from scipy.spatial import Voronoi

points = [[-30.0, 30.370371], [-27.777777, 35.925926], [-34.444443, 58.51852], [-2.9629631, 57.777779], [-17.777779, 75.185181], [-29.25926, 58.148151], [-11.111112, 33.703705], [-11.481482, 40.0], [-27.037037, 40.0], [-7.7777777, 94.444443], [-2.2222223, 122.22222], [-20.370371, 106.66667], [1.1111112, 125.18518], [-6.2962961, 128.88889], [6.666667, 133.7037], [11.851852, 136.2963], [8.5185184, 140.74074], [20.370371, 92.962959], [17.777779, 114.81482], [12.962962, 97.037041], [13.333334, 127.77778], [22.592592, 120.37037], [16.296295, 127.77778], [11.851852, 50.740742], [20.370371, 54.814816], [19.25926, 47.40741], [32.59259, 122.96296], [20.74074, 130.0], [24.814816, 84.814819], [26.296295, 91.111107], [56.296295, 131.48149], [60.0, 141.85185], [32.222221, 136.66667], [53.703705, 147.03703], [87.40741, 196.2963], [34.074074, 159.62964], [34.444443, -2.5925925], [36.666668, -1.8518518], [34.074074, -7.4074073], [35.555557, -18.888889], [76.666664, -39.629627], [35.185184, -37.777779], [25.185184, 14.074074], [42.962959, 32.962963], [35.925926, 9.2592592], [52.222221, 77.777779], [57.777779, 92.222221], [47.037041, 92.59259], [82.222221, 54.074074], [48.888889, 24.444445], [35.925926, 47.777779], [50.740742, 69.259254], [51.111111, 51.851849], [56.666664, -12.222222], [117.40741, -4.4444447], [59.629631, -5.9259262], [66.666664, 134.07408], [91.481483, 127.40741], [66.666664, 141.48149], [53.703705, 4.0740738], [85.185181, 11.851852], [69.629631, 0.37037039], [68.518517, 99.259262], [75.185181, 100.0], [70.370369, 113.7037], [74.444443, 82.59259], [82.222221, 93.703697], [72.222221, 84.444443], [77.777779, 167.03703], [88.888893, 168.88889], [73.703705, 178.88889], [87.037041, 123.7037], [78.518517, 97.037041], [95.555557, 52.962959], [85.555557, 57.037041], [90.370369, 23.333332], [100.0, 28.51852], [88.888893, 37.037037], [87.037041, -42.962959], [89.259262, -24.814816], [93.333328, 7.4074073], [98.518517, 5.185185], [92.59259, 1.4814816], [85.925919, 153.7037], [95.555557, 154.44444], [92.962959, 150.0], [97.037041, 95.925919], [106.66667, 115.55556], [92.962959, 114.81482], [108.88889, 56.296295], [97.777779, 50.740742], [94.074081, 89.259262], [96.666672, 91.851852], [102.22222, 77.777779], [107.40741, 40.370369], [105.92592, 29.629629], [105.55556, -46.296295], [118.51852, -47.777779], [112.22222, -43.333336], [112.59259, 25.185184], [115.92592, 27.777777], [112.59259, 31.851852], [107.03704, -36.666668], [118.88889, -32.59259], [114.07408, -25.555555], [115.92592, 85.185181], [105.92592, 18.888889], [121.11111, 14.444445], [129.25926, -28.51852], [127.03704, -18.518518], [139.25926, -12.222222], [141.48149, 3.7037036], [137.03703, -4.814815], [153.7037, -26.666668], [-2.2222223, 5.5555558], [0.0, 9.6296301], [10.74074, 20.74074], [2.2222223, 54.074074], [4.0740738, 50.740742], [34.444443, 46.296295], [11.481482, 1.4814816], [24.074076, -2.9629631], [74.814819, 79.259254], [67.777779, 152.22223], [57.037041, 127.03704], [89.259262, 12.222222]]

points = np.array(points)

vor = Voronoi(points)

regions, vertices = voronoi_finite_polygons_2d(vor)

pts = MultiPoint([Point(i) for i in points])
mask = pts.convex_hull
new_vertices = []
for region in regions:
    polygon = vertices[region]
    shape = list(polygon.shape)
    shape[0] += 1
    p = Polygon(np.append(polygon, polygon[0]).reshape(*shape)).intersection(mask)
    poly = np.array(list(zip(p.boundary.coords.xy[0][:-1], p.boundary.coords.xy[1][:-1])))
    new_vertices.append(poly)
    plt.fill(*zip(*poly), alpha=0.4)
plt.plot(points[:,0], points[:,1], 'ko')
plt.title("Clipped Voronois")
plt.show()
```
More generally speaking (i.e without using voronoi_finite_polygons_2d but using directly the output of Voronoi if it fits my need), i would do :
```
import numpy as np
import matplotlib.pyplot as plt
from shapely.ops import polygonize,unary_union
from shapely.geometry import LineString, MultiPolygon, MultiPoint, Point
from scipy.spatial import Voronoi
points = [[-30.0, 30.370371], [-27.777777, 35.925926], [-34.444443, 58.51852], [-2.9629631, 57.777779], [-17.777779, 75.185181], [-29.25926, 58.148151], [-11.111112, 33.703705], [-11.481482, 40.0], [-27.037037, 40.0], [-7.7777777, 94.444443], [-2.2222223, 122.22222], [-20.370371, 106.66667], [1.1111112, 125.18518], [-6.2962961, 128.88889], [6.666667, 133.7037], [11.851852, 136.2963], [8.5185184, 140.74074], [20.370371, 92.962959], [17.777779, 114.81482], [12.962962, 97.037041], [13.333334, 127.77778], [22.592592, 120.37037], [16.296295, 127.77778], [11.851852, 50.740742], [20.370371, 54.814816], [19.25926, 47.40741], [32.59259, 122.96296], [20.74074, 130.0], [24.814816, 84.814819], [26.296295, 91.111107], [56.296295, 131.48149], [60.0, 141.85185], [32.222221, 136.66667], [53.703705, 147.03703], [87.40741, 196.2963], [34.074074, 159.62964], [34.444443, -2.5925925], [36.666668, -1.8518518], [34.074074, -7.4074073], [35.555557, -18.888889], [76.666664, -39.629627], [35.185184, -37.777779], [25.185184, 14.074074], [42.962959, 32.962963], [35.925926, 9.2592592], [52.222221, 77.777779], [57.777779, 92.222221], [47.037041, 92.59259], [82.222221, 54.074074], [48.888889, 24.444445], [35.925926, 47.777779], [50.740742, 69.259254], [51.111111, 51.851849], [56.666664, -12.222222], [117.40741, -4.4444447], [59.629631, -5.9259262], [66.666664, 134.07408], [91.481483, 127.40741], [66.666664, 141.48149], [53.703705, 4.0740738], [85.185181, 11.851852], [69.629631, 0.37037039], [68.518517, 99.259262], [75.185181, 100.0], [70.370369, 113.7037], [74.444443, 82.59259], [82.222221, 93.703697], [72.222221, 84.444443], [77.777779, 167.03703], [88.888893, 168.88889], [73.703705, 178.88889], [87.037041, 123.7037], [78.518517, 97.037041], [95.555557, 52.962959], [85.555557, 57.037041], [90.370369, 23.333332], [100.0, 28.51852], [88.888893, 37.037037], [87.037041, -42.962959], [89.259262, -24.814816], [93.333328, 7.4074073], [98.518517, 5.185185], [92.59259, 1.4814816], [85.925919, 153.7037], [95.555557, 154.44444], [92.962959, 150.0], [97.037041, 95.925919], [106.66667, 115.55556], [92.962959, 114.81482], [108.88889, 56.296295], [97.777779, 50.740742], [94.074081, 89.259262], [96.666672, 91.851852], [102.22222, 77.777779], [107.40741, 40.370369], [105.92592, 29.629629], [105.55556, -46.296295], [118.51852, -47.777779], [112.22222, -43.333336], [112.59259, 25.185184], [115.92592, 27.777777], [112.59259, 31.851852], [107.03704, -36.666668], [118.88889, -32.59259], [114.07408, -25.555555], [115.92592, 85.185181], [105.92592, 18.888889], [121.11111, 14.444445], [129.25926, -28.51852], [127.03704, -18.518518], [139.25926, -12.222222], [141.48149, 3.7037036], [137.03703, -4.814815], [153.7037, -26.666668], [-2.2222223, 5.5555558], [0.0, 9.6296301], [10.74074, 20.74074], [2.2222223, 54.074074], [4.0740738, 50.740742], [34.444443, 46.296295], [11.481482, 1.4814816], [24.074076, -2.9629631], [74.814819, 79.259254], [67.777779, 152.22223], [57.037041, 127.03704], [89.259262, 12.222222]]
points = np.array(points)
vor = Voronoi(points)
lines = [
    LineString(vor.vertices[line])
    for line in vor.ridge_vertices if -1 not in line
]

convex_hull = MultiPoint([Point(i) for i in points]).convex_hull.buffer(2)
result = MultiPolygon(
    [poly.intersection(convex_hull) for poly in polygonize(lines)])
result = MultiPolygon(
    [p for p in result]
    + [p for p in convex_hull.difference(unary_union(result))])

plt.plot(points[:,0], points[:,1], 'ko')
for r in result:
    plt.fill(*zip(*np.array(list(
        zip(r.boundary.coords.xy[0][:-1], r.boundary.coords.xy[1][:-1])))),
        alpha=0.4)
plt.show()
```
Minus the small buffer on the convex hull, the result should look the same: Or if you want a result which is slightly less "raw" on the exterior you can try to play with the buffer method (and its resolution/join_style/cap_style properties) of your points (and/or the buffer of the convex hull):
```
pts = MultiPoint([Point(i) for i in points])
mask = pts.convex_hull.union(pts.buffer(10, resolution=5, cap_style=3))
result = MultiPolygon(
    [poly.intersection(mask) for poly in polygonize(lines)])
```
And get something like (you can achieve better..!) :


Question:
Resampling a signal with scipy.signal.resample I was trying to resample a generated signal from 256 samples to 20 samples using this code:
```
import scipy.signal
import matplotlib.pyplot as plt
import numpy as np

x = np.linspace(0, 10, 256, endpoint=False)
y = np.cos(-x**2/6.0)
yre = signal.resample(y,20)
xre = np.linspace(0, 10, len(yre), endpoint=False)
plt.plot(x,y,'b', xre,yre,'or-')
plt.show()
```
Which returns this plot (apparently correct): However, as can be noticed, the first sample is badly approximated. I believe that resample computes the average of the samples that belongs to equidistant groups of samples and, in this case, it seems that the first subgroup of samples is padded with zeros in the beggining in order to estimate the first output sample. Thus, I consider that the first sample can be successfully estimated by telling resample function that I do not want to pad with zeroes the first subgroup. Can somebody help me in order to achieve a correct resampling of this signal? Thanks in advance.

Response:
I had similar problem. Found solution on the net that seems to be also faster than scipy.signal.resample (https://github.com/nwhitehead/swmixer/blob/master/swmixer.py). It is based on np.interp function. Added also scipy.signal.resample_poly for comparison (which is not very best in this case).
```
import scipy.signal
import matplotlib.pyplot as plt
import numpy as np

# DISCLAIMER: This function is copied from https://github.com/nwhitehead/swmixer/blob/master/swmixer.py,
#             which was released under LGPL.
def resample_by_interpolation(signal, input_fs, output_fs):

    scale = output_fs / input_fs
    # calculate new length of sample
    n = round(len(signal) * scale)

    # use linear interpolation
    # endpoint keyword means than linspace doesn't go all the way to 1.0
    # If it did, there are some off-by-one errors
    # e.g. scale=2.0, [1,2,3] should go to [1,1.5,2,2.5,3,3]
    # but with endpoint=True, we get [1,1.4,1.8,2.2,2.6,3]
    # Both are OK, but since resampling will often involve
    # exact ratios (i.e. for 44100 to 22050 or vice versa)
    # using endpoint=False gets less noise in the resampled sound
    resampled_signal = np.interp(
        np.linspace(0.0, 1.0, n, endpoint=False),  # where to interpret
        np.linspace(0.0, 1.0, len(signal), endpoint=False),  # known positions
        signal,  # known data points
    )
    return resampled_signal

x = np.linspace(0, 10, 256, endpoint=False)
y = np.cos(-x**2/6.0)
yre = scipy.signal.resample(y,20)
xre = np.linspace(0, 10, len(yre), endpoint=False)

yre_polyphase = scipy.signal.resample_poly(y, 20, 256)
yre_interpolation = resample_by_interpolation(y, 256, 20)

plt.figure(figsize=(10, 6))
plt.plot(x,y,'b', xre,yre,'or-')
plt.plot(xre, yre_polyphase, 'og-')
plt.plot(xre, yre_interpolation, 'ok-')
plt.legend(['original signal', 'scipy.signal.resample', 'scipy.signal.resample_poly', 'interpolation method'], loc='lower left')
plt.show()
```
CARE! This method, however, seems to perform some unwanted low-pass filtering.
```
x = np.linspace(0, 10, 16, endpoint=False)
y = np.random.RandomState(seed=1).rand(len(x))
yre = scipy.signal.resample(y, 18)
xre = np.linspace(0, 10, len(yre), endpoint=False)

yre_polyphase = scipy.signal.resample_poly(y, 18, 16)
yre_interpolation = resample_by_interpolation(y, 16, 18)

plt.figure(figsize=(10, 6))
plt.plot(x,y,'b', xre,yre,'or-')
plt.plot(xre, yre_polyphase, 'og-')
plt.plot(xre, yre_interpolation, 'ok-')
plt.legend(['original signal', 'scipy.signal.resample', 'scipy.signal.resample_poly', 'interpolation method'], loc='lower left')
plt.show()
```
Still, this is the best result I got, but I hope someone will provide something better.


Question:
Odd SciPy ODE Integration error I'm implementing a very simple Susceptible-Infected-Recovered model with a steady population for an idle side project - normally a pretty trivial task. But I'm running into solver errors using either PysCeS or SciPy, both of which use lsoda as their underlying solver. This only happens for particular values of a parameter, and I'm stumped as to why. The code I'm using is as follows:
```
import numpy as np
from pylab import *
import scipy.integrate as spi

#Parameter Values
S0 = 99.
I0 = 1.
R0 = 0.
PopIn= (S0, I0, R0)
beta= 0.50
gamma=1/10.
mu = 1/25550.
t_end = 15000.
t_start = 1.
t_step = 1.
t_interval = np.arange(t_start, t_end, t_step)

#Solving the differential equation. Solves over t for initial conditions PopIn
def eq_system(PopIn,t):
    '''Defining SIR System of Equations'''
    #Creating an array of equations
    Eqs= np.zeros((3))
    Eqs[0]= -beta * (PopIn[0]*PopIn[1]/(PopIn[0]+PopIn[1]+PopIn[2])) - mu*PopIn[0] + mu*(PopIn[0]+PopIn[1]+PopIn[2])
    Eqs[1]= (beta * (PopIn[0]*PopIn[1]/(PopIn[0]+PopIn[1]+PopIn[2])) - gamma*PopIn[1] - mu*PopIn[1])
    Eqs[2]= gamma*PopIn[1] - mu*PopIn[2]
    return Eqs

SIR = spi.odeint(eq_system, PopIn, t_interval)
```
This produces the following error:
```
lsoda--  at current t (=r1), mxstep (=i1) steps
       taken on this call before reaching tout
      In above message,  I1 =       500
      In above message,  R1 =  0.7818108252072E+04
Excess work done on this call (perhaps wrong Dfun type).
Run with full_output = 1 to get quantitative information.
```
Normally when I encounter a problem like that, there's something terminally wrong with the equation system I set up, but I both can't see anything wrong with it. Weirdly, it also works if you change mu to something like 1/15550. In case it was something wrong with the system, I also implemented the model in R as follows:
```
require(deSolve)

sir.model <- function (t, x, params) {
  S <- x[1]
  I <- x[2]
  R <- x[3]
  with (
    as.list(params),
{
    dS <- -beta*S*I/(S+I+R) - mu*S + mu*(S+I+R)
    dI <- beta*S*I/(S+I+R) - gamma*I - mu*I
    dR <- gamma*I - mu*R
  res <- c(dS,dI,dR)
  list(res)
}
  )
}

times <- seq(0,15000,by=1)
params <- c(
 beta <- 0.50,
 gamma <- 1/10,
 mu <- 1/25550
)

xstart <- c(S = 99, I = 1, R= 0)

out <- as.data.frame(lsoda(xstart,times,sir.model,params))
```
This also uses lsoda, but seems to be going off without a hitch. Can anyone see what's going wrong in the Python code?

Response:
I think that for the parameters you've chosen you're running into problems with stiffness - due to numerical instability the solver's step size is getting pushed into becoming very small in regions where the slope of the solution curve is actually quite shallow. The Fortran solver lsoda, which is wrapped by scipy.integrate.odeint, tries to switch adaptively between methods suited to 'stiff' and 'non-stiff' systems, but in this case it seems to be failing to switch to stiff methods. Very crudely you can just massively increase the maximum allowed steps and the solver will get there in the end:
```
SIR = spi.odeint(eq_system, PopIn, t_interval,mxstep=5000000)
```
A better option is to use the object-oriented ODE solver scipy.integrate.ode, which allows you to explicitly choose whether to use stiff or non-stiff methods:
```
import numpy as np
from pylab import *
import scipy.integrate as spi

def run():
    #Parameter Values
    S0 = 99.
    I0 = 1.
    R0 = 0.
    PopIn= (S0, I0, R0)
    beta= 0.50
    gamma=1/10.
    mu = 1/25550.
    t_end = 15000.
    t_start = 1.
    t_step = 1.
    t_interval = np.arange(t_start, t_end, t_step)

    #Solving the differential equation. Solves over t for initial conditions PopIn
    def eq_system(t,PopIn):
        '''Defining SIR System of Equations'''
        #Creating an array of equations
        Eqs= np.zeros((3))
        Eqs[0]= -beta * (PopIn[0]*PopIn[1]/(PopIn[0]+PopIn[1]+PopIn[2])) - mu*PopIn[0] + mu*(PopIn[0]+PopIn[1]+PopIn[2])
        Eqs[1]= (beta * (PopIn[0]*PopIn[1]/(PopIn[0]+PopIn[1]+PopIn[2])) - gamma*PopIn[1] - mu*PopIn[1])
        Eqs[2]= gamma*PopIn[1] - mu*PopIn[2]
        return Eqs

    ode =  spi.ode(eq_system)

    # BDF method suited to stiff systems of ODEs
    ode.set_integrator('vode',nsteps=500,method='bdf')
    ode.set_initial_value(PopIn,t_start)

    ts = []
    ys = []

    while ode.successful() and ode.t < t_end:
        ode.integrate(ode.t + t_step)
        ts.append(ode.t)
        ys.append(ode.y)

    t = np.vstack(ts)
    s,i,r = np.vstack(ys).T

    fig,ax = subplots(1,1)
    ax.hold(True)
    ax.plot(t,s,label='Susceptible')
    ax.plot(t,i,label='Infected')
    ax.plot(t,r,label='Recovered')
    ax.set_xlim(t_start,t_end)
    ax.set_ylim(0,100)
    ax.set_xlabel('Time')
    ax.set_ylabel('Percent')
    ax.legend(loc=0,fancybox=True)

    return t,s,i,r,fig,ax
```
Output:


Question:
Compute Euclidean distance between rows of two pandas dataframes I have two pandas dataframes d1 and d2 that look like these: d1 looks like:
```
output   value1   value2   value2
    1           100     103      87
    1           201     97.5     88.9
    1           144     54       85
```
d2 looks like:
```
output   value1   value2   value2
    0           100     103      87
    0           201     97.5     88.9
    0           144     54       85
    0           100     103      87
    0           201     97.5     88.9
    0           144     54       85
```
The column output has a value of 1 for all rows in d1 and 0 for all rows in d2. It's a grouping variable. I need to find euclidean distance between each rows of d1 and d2 (not within d1 or d2). If d1 has m rows and d2 has n rows, then the distance matrix will have m rows and n columns

Response:
By using scipy.spatial.distance.cdist:
```
from scipy.spatial.distance import cdist

ary = cdist(d1.iloc[:,1:], d2.iloc[:,1:], metric='euclidean')

pd.DataFrame(ary)
Out[1274]:
            0           1          2           3           4          5
0    0.000000  101.167485  65.886266    0.000000  101.167485  65.886266
1  101.167485    0.000000  71.808495  101.167485    0.000000  71.808495
2   65.886266   71.808495   0.000000   65.886266   71.808495   0.000000
```


Question:
truncated normal distribution with scipy in python I am trying to use a truncated normal distribution with scipy in Python3. I want to do something simple: plot the pdf of a truncated normal centered at 0.5 and ranging from 0 to 1. I have the following code line
```
from scipy import truncnorm
import matplotlib.pyplot as plt
plt.plot([truncnorm.pdf(p,0,1, loc=0.5) for p in np.arange(0,1.1,0.1)])
```
However, this does not give the nice bell-shaped probability distribution function I would expect. Rather, it equals 0 before 0.5, and I cannot figure out why. Any advice on this?

Response:
Here's the procedure to follow according to the documentation of truncnorm.
```
# user input
myclip_a = 0
myclip_b = 1
my_mean = 0.5
my_std = 0.3

a, b = (myclip_a - my_mean) / my_std, (myclip_b - my_mean) / my_std
x_range = np.linspace(-1,2,1000)
plt.plot(x_range, truncnorm.pdf(x_range, a, b, loc = my_mean, scale = my_std))
```


Question:
Python wilcoxon: unequal N Rs wilcox.test can take different length vectors, but the wilcoxon from scipy.stats cannot: I get an unequal N error message.
```
from scipy.stats import wilcoxon
wilcoxon(range(10), range(12))
```
Is there a way to get Rs behavior in Python?

Response:
According to the R docs:
```
Performs one- and two-sample Wilcoxon tests on vectors of data; the latter is also known as ‘Mann-Whitney’ test.
```
So just use
```
from scipy.stats import mannwhitneyu
mannwhitneyu(range(10), range(12))
# (50.0, 0.26494055917435472)
```


Question: scipy.io.wavfile.read cannot read 24-bits .wav files It seems that scipy.io.wavfile.read cannot read 24-bits .wav files. Do you have an idea on how to handle them ?

Response: If your wav files are not compressed, you can try the readwav function here: https://gist.github.com/WarrenWeckesser/7461781 Update I converted that gist to a python package: https://pypi.python.org/pypi/wavio The source code is on github: https://github.com/WarrenWeckesser/wavio


Question:
scipy convolve2d outputs wrong values Here is my code which I used for checking the correctness of convolve2d
```
import numpy as np
from scipy.signal import convolve2d

X = np.random.randint(5, size=(10,10))
K = np.random.randint(5, size=(3,3))
print "Input's top-left corner:"
print X[:3,:3]
print 'Kernel:'
print K

print 'Hardcording the calculation of a valid convolution (top-left)'
print (X[:3,:3]*K)
print 'Sums to'
print (X[:3,:3]*K).sum()
print 'However the top-left value of the convolve2d result'
Y = convolve2d(X, K, 'valid')
print Y[0,0]
```
On my computer this results in the following:
```
Input's top-left (3x3) corner:
[[0 0 0]
 [1 1 2]
 [1 3 0]]
Kernel:
[[4 1 1]
 [0 3 3]
 [2 1 2]]
Hardcording the calculation of a valid convolution (top-left)
[[0 0 0]
 [0 3 6]
 [2 3 0]]
Sums to
14
However the top-left value of the convolve2d result
10
```
Background story: I've been debugging a convnet library, and somehow the gradients were always wrong. After a few weeks I concluded that everything should be working fine, so I checked the convolve2d function by bare hand.

Response:
I think the problem is that you did not do what SciPy implemented. I won't dwell on the details or the foundations but only provide you with a solution: Reverse the kernel.
```

```python
import numpy as np
```

```python
arr = np.array([[0, 0, 0],
#Output
#                    [1, 1, 2],
#                    [1, 3, 0]])
```

```python
kernel = np.array([[4, 1, 1],
#Output
#                       [0, 3, 3],
#                       [2, 1, 2]])
```

```python
from scipy.signal import convolve2d
```

```python
convolve2d(arr, kernel[::-1, ::-1])
#Output
#array([[ 0,  0,  0,  0,  0],
#       [ 2,  3,  7,  4,  4],
#       [ 5, 13, 14, 12,  0],
#       [ 4, 14, 16,  6,  8],
#       [ 1,  4,  7, 12,  0]])
```

```python
convolve2d(arr, kernel[::-1, ::-1], 'valid')
#Output
#array([[14]])
#```
```


Question:
Minimize total distance between two sets of points in Python Given two sets of points in n-dimensional space, how can one map points from one set to the other, such that each point is only used once and the total euclidean distance between the pairs of points is minimized? For example,
```
import matplotlib.pyplot as plt
import numpy as np

# create six points in 2d space; the first three belong to set "A" and the
# second three belong to set "B"
x = [1, 2, 3, 1.8, 1.9, 3.4]
y = [2, 3, 1, 2.6, 3.4, 0.4]

colors = ['red'] * 3 + ['blue'] * 3

plt.scatter(x, y, c=colors)
plt.show()
```
So in the example above, the goal would be to map each red point to a blue point such that each blue point is only used once and the sum of the distances between points is minimized. I came across this question which helps to solve the first part of the problem -- computing the distances between all pairs of points across sets using the scipy.spatial.distance.cdist() function. From there, I could probably test every permutation of single elements from each row, and find the minimum. The application I have in mind involves a fairly small number of datapoints in 3-dimensional space, so the brute force approach might be fine, but I thought I would check to see if anyone knows of a more efficient or elegant solution first.

Response:
An example of assigning (mapping) elements of one set to points to the elements of another set of points, such that the sum Euclidean distance is minimized.
```
import numpy as np
import matplotlib.pyplot as plt
from scipy.spatial.distance import cdist
from scipy.optimize import linear_sum_assignment

np.random.seed(100)

points1 = np.array([(x, y) for x in np.linspace(-1,1,7) for y in np.linspace(-1,1,7)])
N = points1.shape[0]
points2 = 2*np.random.rand(N,2)-1

C = cdist(points1, points2)

_, assigment = linear_sum_assignment(C)

plt.plot(points1[:,0], points1[:,1],'bo', markersize = 10)
plt.plot(points2[:,0], points2[:,1],'rs',  markersize = 7)
for p in range(N):
    plt.plot([points1[p,0], points2[assigment[p],0]], [points1[p,1], points2[assigment[p],1]], 'k')
plt.xlim(-1.1,1.1)
plt.ylim(-1.1,1.1)
plt.axes().set_aspect('equal')
```


Question: Fit data to all possible distributions and return the best fit [closed] Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers. This question appears to be off-topic because it lacks sufficient information to diagnose the problem. Describe your problem in more detail or include a minimal example in the question itself. Closed 11 years ago. Improve this question I have a sample data and I want to get the best fit distribution. I have got couple of links which suggest that I can import the distributions from scipy.stats, but then I am not aware of the type of data before hand. I want something similar to allfitdist() in MATLAB which tries to fit data to around 20 distributions and returns the best fit. Link for allfitdist(): http://www.mathworks.in/matlabcentral/fileexchange/34943-fit-all-valid-parametric-probability-distributions-to-data Any help is highly appreciable. Thanks.

Response:
You can just create a list of all available distributions in scipy. An example with two distributions and random data:
```
import numpy as np
import scipy.stats as st

data = np.random.random(10000)
distributions = [st.laplace, st.norm]
mles = []

for distribution in distributions:
    pars = distribution.fit(data)
    mle = distribution.nnlf(pars, data)
    mles.append(mle)

results = [(distribution.name, mle) for distribution, mle in zip(distributions, mles)]
best_fit = sorted(zip(distributions, mles), key=lambda d: d[1])[0]
print 'Best fit reached using {}, MLE value: {}'.format(best_fit[0].name, best_fit[1])
```


Question: Given a 2D Numpy array representing a 2D distribution, how to sample data from this distribution with the aid of Numpy or Scipy functions? Given a 2D numpy array dist with shape (200,200), where each entry of the array represents the joint probability of (x1, x2) for all x1 , x2 ∈ {0, 1, . . . , 199}. How do I sample bivariate data x = (x1, x2) from this probability distribution with the aid of Numpy or Scipy API?

Response:
This solution works with probability distributions of any number of dimensions, assuming they are a valid probability distribution (its contents must sum to 1, etc.). It flattens the distribution, samples from that, and adjusts the random index to match the original array shape.
```py
# Create a flat copy of the array
flat = array.flatten()

# Then, sample an index from the 1D array with the
# probability distribution from the original array
sample_index = np.random.choice(a=flat.size, p=flat)

# Take this index and adjust it so it matches the original array
adjusted_index = np.unravel_index(sample_index, array.shape)
print(adjusted_index)
```
Also, to get multiple samples, add a size keyword argument to the np.random.choice call, and modify adjusted_index before printing it:
```
adjusted_index = np.array(list(zip(*adjusted_index)))
```
This is necessary because np.random.choice with a size argument outputs a list of indices for each coordinate dimension, so this zips them into a list of coordinate tuples. This is also much more efficient than simply repeating the first code. Relevant documentation: np.random.choice np.unravel_index


Question: scipy sparse matrices as an input for petsc4py I can't seem to find a way how to efficiently load scipy sparse matrices, e.g. csr_matrix, into a petsc4py matrix, e.g. PETSc.Mat().createAIJ. I found this thread, but I'm not able to apply it. I would also appreciate a pointer where this stuff is actually documented. The examples in the demo directory only explain a part, and I can't see any docstrings.

Response:
Your link says that to create a sparse matrix in PETSc, you should use a command like this:
```
PETSc.Mat().createAIJ(size=(nrows,ncols), csr=(ai,aj,aa))
```
According to this, the ai, aj and aa are, in PETSc-speak:
```
> i - row indices
> j - column indices
> a - matrix values
```
These are equivalent, respectively, to the .indptr, .indices and .data attributes of a scypy.sparse.csr_matrix, see the docs for details. So, if your link is right, the following should work:
```

```python
from petsc4py import PETSc
```

```python
import scipy.sparse
```

```python
csr_mat = scipy.sparse.rand(1000, 1000, density=0.001, format='csr')
```

```python
petsc_mat = PETSc.Mat().createAIJ(size=csr_mat.shape,
                                  csr=(csr_mat.indptr, csr_mat.indices,
                                       csr_mat.data))
#Output
#```
#Unfortunately, I cannot test it myself.
```


Question:
How to plot line (polygonal chain) with numpy/scipy/matplotlib with minimal smoothing I am trying to plot a line in matplotlib.. I am searching for the right type of interpolation.. I want something like this where every line is smoothed. I tried several combination of scipy and matplotlib, such as
```
x_new = np.arange(x, x_length, 1)
tck = interpolate.splrep(x, y, s=3)
y_new = interpolate.splev(x_new, tck, der=0)
ax.plot(x_new, y_new, color+lstyle)
```
but the best result I get is The line represents an increasing variable.. so it is a wrong representation. What can I search for? Thanks Edit: I am thinking about implementing a method from myself, but I don't know if it has been already done.. pseudo code is the following
```
take x and y
calculate spline for each three points
x[0], x[1], x[2] ... x[1], x[2], x[3] ... and so on
for each y[n] sums every computation done for it and divide by number of
computations (i.e. y[1] is computed for triplette x[0..2] and x[1..3] so the
sum is divided by two (average for each point is taken as its value)
```

Response:
For that type of graph, you want monotonic interpolation. The PchipInterpolator class (which you can refer to by its shorter alias pchip) in scipy.interpolate can be used:
```
import numpy as np
from scipy.interpolate import pchip
import matplotlib.pyplot as plt

# Data to be interpolated.
x = np.arange(10.0)
y = np.array([5.0, 10.0, 20.0, 15.0, 13.0, 22.0, 20.0, 15.0, 12.0, 16.0])

# Create the interpolator.
interp = pchip(x, y)

# Dense x for the smooth curve.
xx = np.linspace(0, 9.0, 101)

# Plot it all.
plt.plot(xx, interp(xx))
plt.plot(x, y, 'bo')
plt.ylim(0, 25)
plt.grid(True)
plt.show()
```
Result:


Question:
Use of curve_fit to fit data I'm new to scipy and matplotlib, and I've been trying to fit functions to data. The first example in the Scipy Cookbook works fantastically, but when I am trying it with points read from a file, the initial coefficients I give (p0 below) never seem to actually change, and the covariance matrix is always INF. I've tried to fit even data following a line, to no avail. Is it a problem with the way I am importing the data? If so, is there a better way to do it?
```
import matplotlib.pyplot as plt
from scipy.optimize import curve_fit
import scipy as sy

with open('data.dat') as f:
    noms = f.readline().split('\t')

    dtipus = [('x', sy.float32)] + [('y', sy.float32)]

    data = sy.loadtxt(f,delimiter='\t',dtype=dtipus)

    x = data['x']
    y = data['y']

    def func(x, a, b, c):
        return a*x**b + c

    p0 = sy.array([1,1,1])

    coeffs, matcov = curve_fit(func, x, y, p0)

    yaj = func(x, coeffs[0], coeffs[1], coeffs[2])

    print(coeffs)
    print(matcov)

    plt.plot(x,y,'x',x,yaj,'r-')
    plt.show()
```
Thanks!

Response:
It seems to me that the problem is indeed in how you import your data. Faking this datafile:
```
$:~/temp$ cat data.dat
1.0  2.0
2.0  4.2
3.0  8.4
4.0  16.1
```
and using the pylab's loadtxt function for reading:
```
import matplotlib.pyplot as plt
from scipy.optimize import curve_fit
import scipy as sy
import pylab as plb

data = plb.loadtxt('data.dat')
x = data[:,0]
y= data[:,1]

def func(x, a, b, c):
  return a*x**b + c

p0 = sy.array([1,1,1])
coeffs, matcov = curve_fit(func, x, y, p0)

yaj = func(x, coeffs[0], coeffs[1], coeffs[2])
print(coeffs)
print(matcov)

plt.plot(x,y,'x',x,yaj,'r-')
plt.show()
```
works for me. By the way, you can use dtypes to name the columns.


Question:
Matlab / Octave bwdist() in Python or C Does anyone know of a Python replacement for Matlab / Octave bwdist() function? This function returns Euclidian distance of each cell to the closest non-zero cell for a given matrix. I saw an Octave C implementation, a pure Matlab implementation, and I was wondering if anyone had to implement this in ANSI C (which doesn't include any Matlab / Octave headers, so I can integrate from Python easily) or in pure Python. Both links I mentioned are below: C++ Matlab M-File As a test, a Matlab code / output looks something like this:
```
bw= [0   1   0   0   0;
     1   0   0   0   0;
     0   0   0   0   1;
     0   0   0   0   0;
     0   0   1   0   0]

D = bwdist(bw)

D =

   1.00000   0.00000   1.00000   2.00000   2.00000
   0.00000   1.00000   1.41421   1.41421   1.00000
   1.00000   1.41421   2.00000   1.00000   0.00000
   2.00000   1.41421   1.00000   1.41421   1.00000
   2.00000   1.00000   0.00000   1.00000   2.00000
```
I tested a recommended distance_transform_edt call in Python, which gave this result: import numpy as np from scipy import ndimage
```
a = np.array(([0,1,0,0,0],
              [1,0,0,0,0],
              [0,0,0,0,1],
              [0,0,0,0,0],
              [0,0,1,0,0]))

res = ndimage.distance_transform_edt(a)
print res

[[ 0.  1.  0.  0.  0.]
 [ 1.  0.  0.  0.  0.]
 [ 0.  0.  0.  0.  1.]
 [ 0.  0.  0.  0.  0.]
 [ 0.  0.  1.  0.  0.]]
```
This result does not seem to match the Octave / Matlab output.

Response:
While Matlab bwdist returns distances to the closest non-zero cell, Python distance_transform_edt returns distances “to the closest background element”. SciPy documentation is not clear about what it considers to be the “background”, there is some type conversion machinery behind it; in practice 0 is the background, non-zero is the foreground. So if we have matrix a:
```

```python
a = np.array(([0,1,0,0,0],
#Output
#              [1,0,0,0,0],
#              [0,0,0,0,1],
#              [0,0,0,0,0],
#              [0,0,1,0,0]))
#```
#then to calculate the same result we need to replaces ones with zeros and zeros with ones, e.g. consider matrix 1-a:
#```
```

```python
a
#Output
#array([[0, 1, 0, 0, 0],
#       [1, 0, 0, 0, 0],
#       [0, 0, 0, 0, 1],
#       [0, 0, 0, 0, 0],
#       [0, 0, 1, 0, 0]])
```

```python
1 - a
#Output
#array([[1, 0, 1, 1, 1],
#       [0, 1, 1, 1, 1],
#       [1, 1, 1, 1, 0],
#       [1, 1, 1, 1, 1],
#       [1, 1, 0, 1, 1]])
#```
#In this case scipy.ndimage.morphology.distance_transform_edt gives the expected results:
#```
```

```python
distance_transform_edt(1-a)
#Output
#array([[ 1.        ,  0.        ,  1.        ,  2.        ,  2.        ],
#       [ 0.        ,  1.        ,  1.41421356,  1.41421356,  1.        ],
#       [ 1.        ,  1.41421356,  2.        ,  1.        ,  0.        ],
#       [ 2.        ,  1.41421356,  1.        ,  1.41421356,  1.        ],
#       [ 2.        ,  1.        ,  0.        ,  1.        ,  2.        ]])
#```
```


Question:
How can I prevent Numpy/ SciPy gaussian blur from converting image to grey scale? I want to perform gaussian blur on an image but I don't want to be convert to grey scale. Is there anyway to perform this operation and keep the color?
```
from scipy import misc

import scipy

import numpy as np

a = misc.imread('A.jpg')

# A retains its color
misc.imsave('color.jpg', a)

# A_G_Blur gets converted to grey scale, I want to prevent this
a_g_blure = ndimage.uniform_filter(a, size=11)

# I want it to keep it's color
misc.imsave('now_grey.jpg', a)
```

Response:
a is a 3-d array with shape (M, N, 3). The problem is that ndimage.uniform_filter(a, size=11) applies a filter with length 11 to each dimension of a, include the third axis that holds the color channels. When you apply the filter with length 11 to an axis with length 3, the resulting values are all pretty close to the average of the three values, so you get something pretty close to a gray scale. (Depending on the image, you might have some color left.) What you actually want is to apply a 2-d filter to each color channel separately. You can do this by giving a tuple as the size argument, using a size of 1 for the last axis:
```
a_g_blure = ndimage.uniform_filter(a, size=(11, 11, 1))
```
Note: uniform_filter is not a Gaussian blur. For that, you would use scipy.ndimage.gaussian_filter. You might also be interested in the filters provided by scikit-image. In particular, see skimage.filters.gaussian_filter.


Question:
Log Normal Random Variables with Scipy I fail to understand the very basics of creating lognormal variables as documented here. The log normal distribution takes on mean and variance as parameters. I would like to create a frozen distribution using these parameters and then get cdf, pdf etc. However, in the documentation, they get the frozen distribution using
```
from scipy.stats import lognorm
s = 0.953682269606
rv = lognorm(s)
```
's' seems to be the standard deviation. I tried to use the 'loc' and 'scale' parameters instead of 's', but that generated an error (s is a required parameter). How can I generate a frozen distribution with parameter values 'm', 's' for location and scale?

Response:
The mystery is solved (edit 3) μ corresponds to ln(scale) (!) σ corresponds to shape (s) loc is not needed for setting any of σ and μ I think it is a severe problem that this is not clearly documented. I guess many have fallen for this when doing simple tests with the lognormal distribution in SciPy. Why is that? The stats module treats loc and scale the same for all distributions (this is not explicitly written down, but can be inferred when reading between the lines). My suspicion was that loc is substracted from x, and the result is divided by scale (and the result is treated as the new x). I tested for that, and this turned out to be the case. What does it mean for the lognormal distribution? In the canonical definition of the lognormal distribution the term ln(x) appears. Obviously, the same term appears in SciPy's implementation. With above's considerations, this is how loc and scale end up in the logarithm:
```
ln((x-loc)/scale)
```
By common logarithm calculus, this is the same as
```
ln(x-loc) - ln(scale)
```
In the canonical definition of the lognormal distribution the term simply is ln(x) - μ. Comparing SciPy's approach and the canonical approach then provides the crucial insight: ln(scale) represents μ. loc, however, has no correspondence in the canonical definition and is better left at 0. Further below, I have argued for the fact that shape (s) is σ. Proof
```

```python
import math
```

```python
from scipy.stats import lognorm
```

```python
mu = 2
```

```python
sigma = 2
```

```python
l = lognorm(s=sigma, loc=0, scale=math.exp(mu))
```

```python
print("mean: %.5f  stddev: %.5f" % (l.mean(), l.std()))
#Output
#mean: 54.59815  stddev: 399.71719
#```
#I use WolframAlpha as a reference. It provides analytically determined values for the mean and standard deviation of the lognormal distribution. http://www.wolframalpha.com/input/?i=log-normal+distribution%2C+mean%3D2%2C+sd%3D2 The values match. WolframAlpha as well as SciPy come up with the mean and standard deviation by evaluating analytical terms. Let's perform an empirical test, by taking many samples from the SciPy distribution, and calculate their mean and standard deviation "manually" (from the whole set of samples):
#```
```

```python
import numpy as np
```

```python
samples = l.rvs(size=2*10**7)
```

```python
print("mean: %.5f  stddev: %.5f" % (np.mean(samples), np.std(samples)))
#Output
#mean: 54.52148  stddev: 380.14457
#```
#This is still not perfectly converged, but I think it is proof enough that the samples correspond to the same distribution that WolframAlpha assumed, given μ=2 and σ=2. And another small edit: it looks like proper usage of a search engine would have helped, we were not the first to be trapped by this: https://stats.stackexchange.com/questions/33036/fitting-log-normal-distribution-in-r-vs-scipy http://nbviewer.ipython.org/url/xweb.geos.ed.ac.uk/~jsteven5/blog/lognormal_distributions.ipynb scipy, lognormal distribution - parameters Another edit: now that I know how it behaves, I realize that be behavior in principle is documented. In the "notes" section we can read: with shape parameter sigma and scale parameter exp(mu) It is just really not obvious (we both were not able to appreciate the importance of this small sentence). I guess the reason that we could not understand what this sentence means is that the analytical expression shown in the notes section does not include loc and scale. I guess this is worth a bug report / documentation improvement. Original answer: Indeed, the shape parameter topic is not well-documented when looking into the docs page for a particular distribution. I recommend having a look at the main stats documentation -- there is a section on shape parameters: http://docs.scipy.org/doc/scipy/reference/tutorial/stats.html#shape-parameters It looks like there should be a lognorm.shapes property, telling you about what the s parameter means, specifically. Edit: There is only one parameter, indeed:
#```
```

```python
lognorm.shapes
#Output
#'s'
#```
#When comparing the general definition of the lognormal distribution (from Wikipedia): and the formula given by the scipy docs:
#```
#lognorm.pdf(x, s) = 1 / (s*x*sqrt(2*pi)) * exp(-1/2*(log(x)/s)**2)
#```
#it becomes obvious that s is the true σ (sigma). However, from the docs it is not obvious how the loc parameter is related to μ (mu). It could be as in ln(x-loc), which would not correspond to μ in the general formula, or it could be ln(x)-loc, which would ensure correspondence between loc and μ. Try it out! :) Edit 2 I have made comparisons between what WolframAlpha (WA) and SciPy say. WA is pretty clear about that it uses μ and σ as generally understood (as defined in linked Wikipedia article).
#```
```

```python
l = lognorm(s=2, loc=0)
```

```python
print("mean: %.5f  stddev: %.5f" % (l.mean(), l.std()))
#Output
#mean: 7.38906  stddev: 54.09584
#```
#This matches WA's output. Now, for loc not being zero, there is a mismatch. Example:
#```
```

```python
l = lognorm(s=2, loc=1)
```

```python
print("mean: %.5f  stddev: %.5f" % (l.mean(), l.std()))
#Output
#mean: 8.38906  stddev: 54.09584
#```
#WA gives a mean of 20.08 and a standard deviation of 147. There you have it, loc does not correspond to μ in the classical definition of the lognormal distribution.
```


Question:
Find non overlapping area between two kde plots I was attempting to determine whether a feature is important or not base on its kde distribution for target variable. I am aware how to plot the kde plot and guess after looking at the plots, but is there a more formal doing this? Such as can we calculate the area of non overlapping area between two curves? When I googled for the area between two curves there are many many links but none of them could solve my exact problem. NOTE: The main aim of this plot is to find whether the feature is important or not. So, please suggest me further if I am missing any hidden concepts here. What I am trying to do is set some threshold such as 0.2, if the non-overlapping area > 0.2, then assert that the feature is important, otherwise not. MWE:
```py
import numpy as np
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt

df = sns.load_dataset('titanic')

x0 = df.loc[df['survived']==0,'fare']
x1 = df.loc[df['survived']==1,'fare']

sns.kdeplot(x0,shade=1)
sns.kdeplot(x1,shade=1)
```
Output Similar links Fill area of overlap between two normal distributions in seaborn / matplotlib Python: Overlap between two functions (PDF of kde and normal) Fill area between two curves in python

Response:
Here are my ideas about the computational part of the question: In order to compare the kde's, they need to be calculated with the same bandwidth. (The default bandwidth depends on the number of x-values, which can be different for both sets.) The intersection of two positive curves is just their minimum. The area of a curve can be approximated via the trapezium rule: np.trapz. Here are these ideas converted to some example code and illustrating plot:
```py
import numpy as np
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
from scipy.stats import gaussian_kde

df = sns.load_dataset('titanic')

x0 = df.loc[df['survived'] == 0, 'fare']
x1 = df.loc[df['survived'] == 1, 'fare']

kde0 = gaussian_kde(x0, bw_method=0.3)
kde1 = gaussian_kde(x1, bw_method=0.3)

xmin = min(x0.min(), x1.min())
xmax = max(x0.max(), x1.max())
dx = 0.2 * (xmax - xmin) # add a 20% margin, as the kde is wider than the data
xmin -= dx
xmax += dx

x = np.linspace(xmin, xmax, 500)
kde0_x = kde0(x)
kde1_x = kde1(x)
inters_x = np.minimum(kde0_x, kde1_x)

plt.plot(x, kde0_x, color='b', label='No')
plt.fill_between(x, kde0_x, 0, color='b', alpha=0.2)
plt.plot(x, kde1_x, color='orange', label='Yes')
plt.fill_between(x, kde1_x, 0, color='orange', alpha=0.2)
plt.plot(x, inters_x, color='r')
plt.fill_between(x, inters_x, 0, facecolor='none', edgecolor='r', hatch='xx', label='intersection')

area_inters_x = np.trapz(inters_x, x)

handles, labels = plt.gca().get_legend_handles_labels()
labels[2] += f': {area_inters_x * 100:.1f} %'
plt.legend(handles, labels, title='Survived?')
plt.title('Fare vs Survived')
plt.tight_layout()
plt.show()
```


Question:
How to specify numba jitclass when the class's attribute contains another class instance? I'm trying to use numba to boost the python performance of scipy.integrate.odeint. To this end, I have to use @nb.jit(nopython=True) for the function defining the ODE system. However, this function has to take another python-class instance as an argument in my program. I had to jit the class also with @nb.jitclass(spec) with appropriate specs. This worked fine, until I found a serious issue when the specs of the class includes another type of class instance as its method. My code is following.
```
import numba as nb
from scipy.integrate import odeint

spec1=[("hi", nb.i4)]
@nb.jitclass(spec1)
class Hi(object):
    def __init__(self):
        self.hi = 0

spec2=[("dummy", nb.i4), ("dummy1", nb.i4)]
@nb.jitclass(spec2)
class Dummy(object):
    def __init__(self, anotherClassInstance):
        self.dummy = 0
        self.dummy1 = anotherClassInstance

class A:
    def __init__(self, someClassInstance):
        self.a1 = someClassInstance

    def odeSystem(self, x, t):
        return _odeSystem(x, t, self.a1)

    def odeSolve(self, iValues, ts):
        sol = odeint(self.odeSystem, iValues, ts)
        return sol

@nb.jit(nopython=True)
def _odeSystem(x, t, someClassInstance):
    return 1-x

if __name__ == "__main__":
    c = Hi()
    b = Dummy(c)
    a = A(b)
    print a.odeSolve(0.5, range(0, 10))
```
In summary: So here "class A" is my ode solver. To compile the method "odeSystem" with numba, it must not a class method. So I made another function outside of the class "_odeSystem". Unfortunately however, my odeSystem has to have a class instance as an argument. Hence I used @jitclass to compile the class instance argument properly. I again encountered another problem, where this class "Dummy" also takes another type of class instance as one of its attribute. I have no idea how to set "spec" for this class. I tried the type for "dummy1" with "nb.typeof(Hi)" but it didn't work. Please help me. Thanks in advance.

Response:
You can use .class_type.instance_type in your spec definition for holding an instance of another type. See example in the numba source tree here
```
spec2=[("dummy", nb.i4), ("dummy1", Hi.class_type.instance_type)]
@nb.jitclass(spec2)
class Dummy(object):
    def __init__(self, anotherClassInstance):
        self.dummy = 0
        self.dummy1 = anotherClassInstance
```


Question:
Why do scipy and numpy fft plots look different? I am currently doing some spectrum analysis for a piece of coursework, though we haven't been explicitly taught Fourier transforms yet. I have been playing around with the various fft algorithms in scipy and numpy on some data that I know what the answer should look like In this case its an AM signal at 8kHz carrier frequency and 1kHz modulated sine wave on top so should have 3 clear peaks on an fft When applying scipy.fftpack.rfft and numpy.fft.rfft I get the following plots respectively: Scipy: Numpy: While the shape of the 2 FFTs are roughly the same with the correct ratios between the peaks, the numpy one looks much smoother, whereas the scipy one has slightly smaller max peaks, and has much more noise. I'm assuming this is largely down to different applications of a Discrete Fourier Transform algorithm, and have seen other articles about how the scipy implementation is faster in run time. But I was wandering what it is that specifically causes the difference, and which one is actually more accurate? EDIT: Code used to generate plots:
```
data = pd.read_csv("./Waveforms/AM waveform Sine.csv", sep = ',', dtype = float)

data = data.as_matrix()
time = data[:,0]
voltage = data[:,1]/data[:,1].max() # normalise the values

#scipy plot:
plt.figure()
magnitude =  scipy.fftpack.rfft(voltage)
freq = scipy.fftpack.rfftfreq(len(time),np.diff(time)[0])
plt.figure()
plt.plot(freq, np.absolute(magnitude), lw = 1)
plt.ylim(0,2500)
plt.xlim(0,15)

#numpy plot
magnitude = np.fft.rfft(voltage)
freq = np.fft.rfftfreq(len(time),np.diff(time)[0])

plt.figure()
plt.plot(freq, np.absolute(magnitude), lw = 1)
plt.ylim(0,2500)
plt.xlim(0,15)
```

Response:
From NumPy's doc for rfft: Returns: out : complex ndarray The truncated or zero-padded input, transformed along the axis indicated by axis, or the last one if axis is not specified. If n is even, the length of the transformed axis is (n/2)+1. If n is odd, the length is (n+1)/2. It is not written explicitly but the "transformed data" is here complex. From SciPy's doc for rfft z : real ndarray The returned real array contains:
```
[y(0),Re(y(1)),Im(y(1)),...,Re(y(n/2))]              if n is even
[y(0),Re(y(1)),Im(y(1)),...,Re(y(n/2)),Im(y(n/2))]   if n is odd
```
Conclusion: the storage is different. For a starter, look at the length of magnitude, it will be different in both cases. I give an example below for clarity:
```

```python
data = np.random.random(size=8)
```

```python
np.fft.rfft(data)
#Output
#array([ 3.33822983+0.j        ,  0.15879369+0.48542266j,
#        0.00614876+0.03590621j, -0.67376592-0.69793372j,  1.51730861+0.j        ])
```

```python
scipy.fftpack.rfft(data)
#Output
#array([ 3.33822983,  0.15879369,  0.48542266,  0.00614876,  0.03590621,
#       -0.67376592, -0.69793372,  1.51730861])
#```
#The first element in both cases is the so-called "DC component" (the mean of the signal). Then, you can recognize in the SciPy version the succession of real and imaginary parts of the NumPy version.
```


Question: How to know when to use numpy.linalg instead of scipy.linalg? Received wisdom is to prefer scipy.linalg over numpy.linalg functions. For doing linear algebra, ideally (and conveniently) I would like to combine the functionalities of numpy.array and scipy.linalg without ever looking towards numpy.linalg. This is not always possible and may become too frustrating. Is there a comparative checklist of equivalent functions from these two modules to quickly determine when to use numpy.linalg in case a function is absent in scipy.linalg? e.g. There are scipy.linalg.norm() and numpy.linalg.norm(), but there seem to be no scipy equivalents of numpy.linalg.matrix_rank() and numpy.linalg.cond().

Response:
So, the normal rule is to just use scipy.linalg as it generally supports all of the numpy.linalg functionality and more. The documentation says this: See also numpy.linalg for more linear algebra functions. Note that although scipy.linalg imports most of them, identically named functions from scipy.linalg may offer more or slightly differing functionality. However, matrix_rank() is only in NumPy. Here we can see the differences between the functions provided by both libraries, and how SciPy is more complete:
```

```python
from scipy import linalg as scipy_linalg
```

```python
from numpy import linalg as numpy_linalg
```

```python
dir(scipy_linalg)
#Output
#[
# ...
# 'absolute_import',
# 'basic',
# 'bench',
# 'blas',
# 'block_diag',
# 'cho_factor',
# 'cho_solve',
# 'cho_solve_banded',
# 'cholesky',
# 'cholesky_banded',
# 'circulant',
# 'companion',
# 'coshm',
# 'cosm',
# 'cython_blas',
# 'cython_lapack',
# 'decomp',
# 'decomp_cholesky',
# 'decomp_lu',
# 'decomp_qr',
# 'decomp_schur',
# 'decomp_svd',
# 'det',
# 'dft',
# 'diagsvd',
# 'division',
# 'eig',
# 'eig_banded',
# 'eigh',
# 'eigvals',
# 'eigvals_banded',
# 'eigvalsh',
# 'expm',
# 'expm2',
# 'expm3',
# 'expm_cond',
# 'expm_frechet',
# 'find_best_blas_type',
# 'flinalg',
# 'fractional_matrix_power',
# 'funm',
# 'get_blas_funcs',
# 'get_lapack_funcs',
# 'hadamard',
# 'hankel',
# 'helmert',
# 'hessenberg',
# 'hilbert',
# 'inv',
# 'invhilbert',
# 'invpascal',
# 'kron',
# 'lapack',
# 'leslie',
# 'linalg_version',
# 'logm',
# 'lstsq',
# 'lu',
# 'lu_factor',
# 'lu_solve',
# 'matfuncs',
# 'misc',
# 'norm',
# 'ordqz',
# 'orth',
# 'orthogonal_procrustes',
# 'pascal',
# 'pinv',
# 'pinv2',
# 'pinvh',
# 'polar',
# 'print_function',
# 'qr',
# 'qr_delete',
# 'qr_insert',
# 'qr_multiply',
# 'qr_update',
# 'qz',
# 'rq',
# 'rsf2csf',
# 's',
# 'schur',
# 'signm',
# 'sinhm',
# 'sinm',
# 'solve',
# 'solve_banded',
# 'solve_circulant',
# 'solve_continuous_are',
# 'solve_discrete_are',
# 'solve_discrete_lyapunov',
# 'solve_lyapunov',
# 'solve_sylvester',
# 'solve_toeplitz',
# 'solve_triangular',
# 'solveh_banded',
# 'special_matrices',
# 'sqrtm',
# 'svd',
# 'svdvals',
# 'tanhm',
# 'tanm',
# 'test',
# 'toeplitz',
# 'tri',
# 'tril',
# 'triu']
```

```python
dir(numpy_linalg)
#Output
#[
# ...
# 'absolute_import',
# 'bench',
# 'cholesky',
# 'cond',
# 'det',
# 'division',
# 'eig',
# 'eigh',
# 'eigvals',
# 'eigvalsh',
# 'info',
# 'inv',
# 'lapack_lite',
# 'linalg',
# 'lstsq',
# 'matrix_power',
# 'matrix_rank',
# 'multi_dot',
# 'norm',
# 'pinv',
# 'print_function',
# 'qr',
# 'slogdet',
# 'solve',
# 'svd',
# 'tensorinv',
# 'tensorsolve',
# 'test']
```

```python
```
Note that not all of these are functions. SciPy does provide scipy.linalg.expm_cond(), but this only returns the condition in the Frobenius norm, whereas numpy.linalg.cond() supports multiple norms.
```


Question: Finding Two-Tailed P Value from t-distribution and Degrees of Freedom in Python How do I determine the P Value of a t-distrobution with n degrees of freedom. Research on this subject points me to this stack exchange answer: https://stackoverflow.com/a/17604216 I assume np.abs(tt) is the T-value, but how do i work in degrees of freedom, is that the n-1? Thanks in advance

Response:
Yes, n-1 is the degrees of freedom in that example. Given a t-value and a degrees of freedom, you can use the "survival function" sf of scipy.stats.t (aka the complementary CDF) to compute the one-sided p-value. The first argument is the T value, and the second is the degrees of freedom. For example, the first entry of the table on this page says that for 1 degree of freedom, the critical T value for p=0.1 is 3.078. Here's how you can verify that with t.sf:
```

```python
from scipy.stats import t
```

```python
t.sf(3.078, 1)
#Output
#0.09999038172554342   # Approximately 0.1, as expected.
#```
#For the two-sided p-value, just double the one-sided p-value.
```


Question:
How to create iterate through a large list of list in python efficiently? I have my data as such:
```
data = {'x':Counter({'a':1,'b':45}), 'y':Counter({'b':1, 'c':212})}
```
where my labels are the keys of the data and the key of the inner dictionary are features:
```
all_features = ['a','b','c']
all_labels = ['x','y']
```
I need to create list of list as such:
```
[[data[label][feat] for feat in all_features] for label in all_labels]
```
[out]:
```
[[1, 45, 0], [0, 1, 212]]
```
My len(all_features) is ~5,000,000 and len(all_labels) is ~100,000 The ultimate purpose is to create scipy sparse matrix, e.g.:
```
from collections import Counter
from scipy.sparse import csc_matrix
import numpy as np

all_features = ['a','b','c']
all_labels = ['x','y']

csc_matrix(np.array([[data[label][feat] for feat in all_features] for label in all_labels]))
```
but looping through a large list of list is rather inefficient. So how can I look the large list of list efficiently? Is there other way to create the scipy matrix from the data without looping through all features and labels?

Response:
Converting a dictionary of dictionaries into a numpy or scipy array is, as you are experiencing, not too much fun. If you know all_features and all_labels before hand, you are probably better off using a scipy sparse COO matrix from the start to keep your counts. Whether that is possible or not, you will want to keep your lists of features and labels in sorted order, to speed up look ups. So I am going to assume that the following doesn't change either array:
```
all_features = np.array(all_features)
all_labels = np.array(all_labels)
all_features.sort()
all_labels.sort()
```
Lets extract the labels in data in the order they are stored in the dictionary, and see where in all_labels does each item fall:
```
labels = np.fromiter(data.iterkeys(), all_labels.dtype, len(data))
label_idx = np.searchsorted(all_labels, labels)
```
Now lets count how many features does each label has, and compute from it the number of non-zero items there will be in your sparse array:
```
label_features = np.fromiter((len(c) for c in data.iteritems()), np.intp,
                             len(data))
indptr = np.concatenate(([0], np.cumsum(label_features)))
nnz = indptr[-1]
```
Now, we extract the features for each label, and their corresponding counts
```
import itertools
features_it = itertools.chain(*(c.iterkeys() for c in data.itervalues()))
features = np.fromiter(features_it, all_features.dtype, nnz)
feature_idx = np.searchsorted(all_features, features)
counts_it = itertools.chain(*(c.itervalues() for c in data.itervalues()))
counts = np.fromiter(counts_it, np.intp, nnz)
```
With what we have, we can create a CSR matrix directly, with labels as rows and features as columns:
```
sps_data = csr_matrix((counts, feature_idx, indptr),
                      shape=(len(all_labels), len(all_features)))
```
The only issue is that the rows of this sparse array are not in the order of all_labels, but in the order they came up when iterating over data. But we have feature_idx telling us where did each label end up, and we can rearrange the rows by doing:
```
sps_data = sps_data[np.argsort(label_idx)]
```
Yes, it is messy, confusing, and probably not very fast, but it works, and it will be much more memory efficient that what you proposed in your question:
```

```python
sps_data.A
#Output
#array([[  1,  45,   0],
#       [  0,   1, 212]], dtype=int64)
```

```python
all_labels
#Output
#array(['x', 'y'],
#      dtype='<S1')
```

```python
all_features
#Output
#array(['a', 'b', 'c'],
#      dtype='<S1')
#```
```


Question:
Euclidean distance between two pandas dataframes I have two dataframes: df1 of the form
```
user_id  | x_coord  | y_coord
 214         -55.2      22.1
 214         -55.2      22.1
 214         -55.2      22.1
...
```
and df2, of the same form, but with different users:
```
user_id  | x_coord  | y_coord
 512         -15.2      19.1
 362          65.1      71.4
 989         -84.8      13.7
...
```
The idea is that I want to find the Euclidean distance between the user in df1 and all the users in df2. For this, I need to be able to compute the Euclidean distance between the two dataframes, based on the last two column, in order to find out which are the closest users in the second dataframe to user 214. I found this answer but it is not what I need, as my two dataframes have equal shapes and I need the distance computed in a per-row manner:
```
Euclidean_Distance_i(row_i_df1, row_i_df2)
```
and save all these distances in a list that is the same length as these dataframes.

Response:
Try:
```
def Euclidean_Dist(df1, df2, cols=['x_coord','y_coord']):
    return np.linalg.norm(df1[cols].values - df2[cols].values,
                   axis=1)
```
Test:
```
df1 = pd.DataFrame({'user_id':[214,214,214],
                'x_coord':[-55.2,-55.2,-55.2],
                'y_coord':[22.1,22.1,22.1]})

df2 = pd.DataFrame({'user_id':[512, 362, 989],
                    'x_coord':[-15.2, 65.1, -84.8],
                    'y_coord':[19.1, 71.4, 13.7]})

Euclidean_Dist(df1, df2)
```
outputs:
```
array([ 40.11234224, 130.0099227 ,  30.76881538])
```


Question:
Saving and loading Python dict with savemat results in error Here is an minimal example of the error I get. If I understood the documentation correctly, this should be working, but it seems I did not.
```
a={}
a['test1']=1
a['test2']=2
a['test3']=3
import scipy.io as io
io.savemat('temp',{'a':a})
b = io.loadmat('temp')
b['a'].keys()

Traceback (most recent call last):
  File "<input>", line 1, in <module>
AttributeError: 'numpy.ndarray' object has no attribute 'keys'
```

Response:
You seem to be operating under the assumption that scipy.io.savemat is intended to be able to save a standard dictionary. I don't believe that is the case. The dictionary argument holds the names of numpy arrays which are written out into the Matlab file. So you can do something like this
```
import scipy.io as io
import numpy as np

y1=np.array([1,2,3,4])
y2=np.array([10,20,30,40])
y3=np.array([100,200,300,400])

a={}
a['test1']=y1
a['test2']=y2
a['test3']=y3
io.savemat('temp',a)
b = io.loadmat('temp')

print b['test1']
print b['test2']
print b['test3']
```
which gives:
```
[[1]
 [2]
 [3]
 [4]]
[[10]
 [20]
 [30]
 [40]]
[[100]
 [200]
 [300]
 [400]]
```