Question: Networkx: Differences between pagerank, pagerank_numpy, and pagerank_scipy? Does anyone know about the differences in accuracy between the three different pagerank functions in Networkx? I have a graph of 1000 nodes and 139732 edges, and the "plain" pagerank function didn't seem to work at all -- all but two of the nodes had the same PG, so I'm assuming this function doesn't work quite as well for large graphs? pagerank_numpy's values also seemed to be a little bit more spread out than pagerank_scipy's values. The documentation for this function says that "This will be the fastest and most accurate for small graphs." What is meant by "small" graphs? Also, why doesn't pagerank_numpy allow for max_iter and tol arguments?

Response:
Each of the three functions uses a different approach to solving the same problem: networkx.pagerank() is a pure-Python implementation of the power-method to compute the largest eigenvalue/eigenvector or the Google matrix. It has two parameters that control the accuracy - tol and max_iter. networkx.pagerank_scipy() is a SciPy sparse-matrix implementation of the power-method. It has the same two accuracy parameters. networkx.pagerank_numpy() is a NumPy (full) matrix implementation that calls the numpy.linalg.eig() function to compute the largest eigenvalue and eigenvector. That function is an interface to the LAPACK dgeev function which is uses a matrix decomposition (direct) method with no tunable parameters. All three should produce the same answer (within numerical roundoff) for well-behaved graphs if the tol parameter is small enough and the max_iter parameter is large enough. Which one is faster depends on the size of your graph and how well the power method works on your graph.
```

```python
import networkx as nx
```

```python
G=nx.gnp_random_graph(1000,0.01,directed=True)
```

```python
%timeit nx.pagerank(G,tol=1e-10)
10 loops, best of 3: 157 ms per loop
```

```python
%timeit nx.pagerank_scipy(G,tol=1e-10)
100 loops, best of 3: 14 ms per loop
```

```python
%timeit nx.pagerank(G)
10 loops, best of 3: 137 ms per loop
```
```


Question: Weighted Gaussian kernel density estimation in `python` Update: Weighted samples are now supported by scipy.stats.gaussian_kde. See here and here for details. It is currently not possible to use scipy.stats.gaussian_kde to estimate the density of a random variable based on weighted samples. What methods are available to estimate densities of continuous random variables based on weighted samples?

Response: Neither sklearn.neighbors.KernelDensity nor statsmodels.nonparametric seem to support weighted samples. I modified scipy.stats.gaussian_kde to allow for heterogeneous sampling weights and thought the results might be useful for others. An example is shown below. An ipython notebook can be found here: http://nbviewer.ipython.org/gist/tillahoffmann/f844bce2ec264c1c8cb5 Implementation details The weighted arithmetic mean is The unbiased data covariance matrix is then given by The bandwidth can be chosen by scott or silverman rules as in scipy. However, the number of samples used to calculate the bandwidth is Kish's approximation for the effective sample size.


Question:
What are the arguments for scipy.stats.uniform? I'm trying to create a uniform distribution between two numbers (lower bound and upper bound) in order to feed it to sklearn's ParameterSampler. I am using scipy.stats.uniform in the following format:
```
from scipy.stats import uniform
params = ParameterSampler({'bandwidth':uniform(5,50)}, 20)
```
But when I get the random selections of the 'bandwidth' parameter, they are not all between 5 and 50. Some of them are bigger than 50 by a bit. So my question is what do the arguments in scipy.stats.uniform represent? Are they not a lower bound and upper bound? The documentation shows no arguments so I can't figure it out from that.

Response: The first argument is the lower bound, and the second argument is the range of the distribution. So the example distribution in your question is uniform between 5 and 55. Quoting from the documentation linked in your question: A uniform continuous random variable. This distribution is constant between loc and loc + scale. loc is the first argument and scale is the second argument.


Question: circular cross correlation python Is it possible to perform circular cross-/auto-correlation on 1D arrays with a numpy/scipy/matplotlib function? I have looked at numpy.correlate() and matplotlib.pyplot.xcorr (based on the numpy function), and both seem to not be able to do circular cross-correlation. To illustrate the difference, I will use the example of an array of [1, 2, 3, 4]. With circular correlation, a periodic assumption is made, and a lag of 1 looks like [2, 3, 4, 1]. The python functions I've found only seem to use zero-padding, i.e., [2, 3, 4, 0]. Is there a way to get these functions to do circular correlation? If not, is there a standard workaround for circular correlations?

Response:
You can implement the periodic (a.k.a. circular) cross correlation using the FFT:
```
from numpy.fft import fft, ifft

def periodic_corr(x, y):
    """Periodic correlation, implemented using the FFT.

    x and y must be real sequences with the same length.
    """
    return ifft(fft(x) * fft(y).conj()).real
```
You can also implement it using np.correlate, if you don't mind the overhead incurred by np.hstack((y[1:], y)):
```
import numpy as np

def periodic_corr_np(x, y):
    """Periodic correlation, implemented using np.correlate.

    x and y must be real sequences with the same length.
    """
    return np.correlate(x, np.hstack((y[1:], y)), mode='valid')
```


Question:
Understanding scipy's least square function with IRLS I'm having a bit of trouble understanding how this function works.
```
a, b = scipy.linalg.lstsq(X, w*signal)[0]
```
I know that signal is the array representing the signal and currently w is just [1,1,1,1,1...] How should I manipulate X or w to imitate weighted least squares or iteratively reweighted least squared?

Response:
If you product X and y with sqrt(weight) you can calculate weighted least squares. You can get the formula by following link: http://en.wikipedia.org/wiki/Linear_least_squares_%28mathematics%29#Weighted_linear_least_squares here is an example: Prepare data:
```
import numpy as np
np.random.seed(0)
N = 20
X = np.random.rand(N, 3)
w = np.array([1.0, 2.0, 3.0])
y = np.dot(X, w) + np.random.rand(N) * 0.1
```
OLS:
```
from scipy import linalg
w1 = linalg.lstsq(X, y)[0]
print w1
```
output:
```
[ 0.98561405  2.0275357   3.05930664]
```
WLS:
```
weights = np.linspace(1, 2, N)
Xw = X * np.sqrt(weights)[:, None]
yw = y * np.sqrt(weights)
print linalg.lstsq(Xw, yw)[0]
```
output:
```
[ 0.98799029  2.02599521  3.0623824 ]
```
Check result by statsmodels:
```
import statsmodels.api as sm
mod_wls = sm.WLS(y, X, weights=weights)
res = mod_wls.fit()
print res.params
```
output:
```
[ 0.98799029  2.02599521  3.0623824 ]
```


Question:
Saving Numpy Structure Array to *.mat file I am using numpy.loadtext to generate a structured Numpy array from a CSV data file that I would like to save to a MAT file for colleagues who are more familiar with MATLAB than Python. Sample case:
```
import numpy as np
import scipy.io

mydata = np.array([(1, 1.0), (2, 2.0)], dtype=[('foo', 'i'), ('bar', 'f')])
scipy.io.savemat('test.mat', mydata)
```
When I attempt to use scipy.io.savemat on this array, the following error is thrown:
```
Traceback (most recent call last):
  File "C:/Project Data/General Python/test.py", line 6, in <module>
    scipy.io.savemat('test.mat', mydata)
  File "C:\python35\lib\site-packages\scipy\io\matlab\mio.py", line 210, in savemat
    MW.put_variables(mdict)
  File "C:\python35\lib\site-packages\scipy\io\matlab\mio5.py", line 831, in put_variables
    for name, var in mdict.items():
AttributeError: 'numpy.ndarray' object has no attribute 'items'
```
I'm a Python novice (at best), but I'm assuming this is because savemat is set up to handle dicts and the structure of Numpy's structured arrays is not compatible. I can get around this error by pulling my data into a dict:
```
tmp = {}
for varname in mydata.dtype.names:
    tmp[varname] = mydata[varname]

scipy.io.savemat('test.mat', tmp)
```
Which loads into MATLAB fine:
```
>> mydata = load('test.mat')

mydata =

    foo: [1 2]
    bar: [1 2]
```
But this seems like a very inefficient method since I'm duplicating the data in memory. Is there a smarter way to accomplish this?

Response:
You can do scipy.io.savemat('test.mat', {'mydata': mydata}). This creates a struct mydata with fields foo and bar in the file. Alternatively, you can pack your loop in a dict comprehension:
```
tmp = {varname: mydata[varname] for varname in mydata.dtype.names}
```
I don't think creating a temprorary dictionary duplicates data in memory, because Python generally only stores references, and numpy in particular tries to create views into the original data whenever possible.


Question:
Reverse sort and argsort in python I'm trying to write a function in Python (still a noob!) which returns indices and scores of documents ordered by the inner products of their tfidf scores. The procedure is: Compute vector of inner products between doc idx and all other documents Sort in descending order Return the "scores" and indices from the second one to the end (i.e. not itself) The code I have at the moment is:
```
import h5py
import numpy as np

def get_related(tfidf, idx) :
    ''' return the top documents '''

    # calculate inner product
    v = np.inner(tfidf, tfidf[idx].transpose())

    # sort
    vs = np.sort(v.toarray(), axis=0)[::-1]
    scores = vs[1:,]

    # sort indices
    vi = np.argsort(v.toarray(), axis=0)[::-1]
    idxs = vi[1:,]

    return (scores, idxs)
```
where tfidf is a sparse matrix of type ''. This seems inefficient, as the sort is performed twice (sort() then argsort()), and the results have to then be reversed. Can this be done more efficiently? Can this be done without converting the sparse matrix using toarray()?

Response:
I don't think there's any real need to skip the toarray. The v array will be only n_docs long, which is dwarfed by the size of the n_docs × n_terms tf-idf matrix in practical situations. Also, it will be quite dense since any term shared by two documents will give them a non-zero similarity. Sparse matrix representations only pay off when the matrix you're storing is very sparse (I've seen >80% figures for Matlab and assume that Scipy will be similar, though I don't have an exact figure). The double sort can be skipped by doing
```
v = v.toarray()
vi = np.argsort(v, axis=0)[::-1]
vs = v[vi]
```
Btw., your use of np.inner on sparse matrices is not going to work with the latest versions of NumPy; the safe way of taking an inner product of two sparse matrices is
```
v = (tfidf * tfidf[idx, :]).transpose()
```


Question: Calculating e (base of the natural log) to high precision in Python? Is it possible to calculate the value of the mathematical constant, e with high precision (2000+ decimal places) using Python? I am particularly interested in a solution either in or that integrates with NumPy or SciPy.

Response:
You can set the precision you want with the decimal built-in module:
```
from decimal import *
getcontext().prec = 40
Decimal(1).exp()
```
This returns:
```
Decimal('2.718281828459045235360287471352662497757')
```


Question:
Binning a numpy array I have a numpy array which contains time series data. I want to bin that array into equal partitions of a given length (it is fine to drop the last partition if it is not the same size) and then calculate the mean of each of those bins. I suspect there is numpy, scipy, or pandas functionality to do this. example:
```
data = [4,2,5,6,7,5,4,3,5,7]
```
for a bin size of 2:
```
bin_data = [(4,2),(5,6),(7,5),(4,3),(5,7)]
bin_data_mean = [3,5.5,6,3.5,6]
```
for a bin size of 3:
```
bin_data = [(4,2,5),(6,7,5),(4,3,5)]
bin_data_mean = [7.67,6,4]
```

Response:
Just use reshape and then mean(axis=1). As the simplest possible example:
```
import numpy as np

data = np.array([4,2,5,6,7,5,4,3,5,7])

print data.reshape(-1, 2).mean(axis=1)
```
More generally, we'd need to do something like this to drop the last bin when it's not an even multiple:
```
import numpy as np

width=3
data = np.array([4,2,5,6,7,5,4,3,5,7])

result = data[:(data.size // width) * width].reshape(-1, width).mean(axis=1)

print result
```


Question:
Multiply all columns in a Pandas dataframe together Is it possible to multiply all the columns in a Pandas.DataFrame together to get a single value for every row in the DataFrame? As an example, using
```
df = pd.DataFrame(np.random.randn(5,3)*10)
```
I want a new DataFrame df2 where df2.ix[x,0] will have the value of df.ix[x,0] * df.ix[x,1] * df.ix[x,2]. However I do not want to hardcode this, how can I use a loop to achieve this? I found a function df.mul(series, axis=1) but cant figure out a way to use this for my purpose.

Response:
You could use DataFrame.prod():
```

```python
df = pd.DataFrame(np.random.randint(1, 10, (5, 3)))
```

```python
df
#Output
#   0  1  2
#0  7  7  5
#1  1  8  6
#2  4  8  4
#3  2  9  5
#4  3  8  7
```

```python
df.prod(axis=1)
#Output
#0    245
#1     48
#2    128
#3     90
#4    168
#dtype: int64
#```
#You could also apply np.prod, which is what I'd originally done, but usually when available the direct methods are faster.
#```
```

```python
df = pd.DataFrame(np.random.randint(1, 10, (5, 3)))
```

```python
df
#Output
#   0  1  2
#0  9  3  3
#1  8  5  4
#2  3  6  7
#3  9  8  5
#4  7  1  2
```

```python
df.apply(np.prod, axis=1)
#Output
#0     81
#1    160
#2    126
#3    360
#4     14
#dtype: int64
#```
```


Question:
find the start position of the longest sequence of 1's I want to find the start position of the longest sequence of 1's in my array:
```
a1=[0,0,1,1,1,1,0,0,1,1]
#2
```
I am following this answer to find the length of the longest sequence. However, I was not able to determine the position.

Response:
Inspired by this solution, here's a vectorized approach to solve it -
```
# Get start, stop index pairs for islands/seq. of 1s
idx_pairs = np.where(np.diff(np.hstack(([False],a1==1,[False]))))[0].reshape(-1,2)

# Get the island lengths, whose argmax would give us the ID of longest island.
# Start index of that island would be the desired output
start_longest_seq = idx_pairs[np.diff(idx_pairs,axis=1).argmax(),0]
```
Sample run -
```

```python
a1 # Input array
#Output
#array([0, 0, 1, 1, 1, 1, 0, 0, 1, 1])
```

```python
idx_pairs # Start, stop+1 index pairs
#Output
#array([[ 2,  6],
#       [ 8, 10]])
```

```python
np.diff(idx_pairs,axis=1) # Island lengths
#Output
#array([[4],
#       [2]])
```

```python
np.diff(idx_pairs,axis=1).argmax() # Longest island ID
#Output
#0
```

```python
idx_pairs[np.diff(idx_pairs,axis=1).argmax(),0] # Longest island start
#Output
#2
#```
```


Question:
scipy.stats.multivariate_normal raising `LinAlgError: singular matrix` even though my covariance matrix is invertible I am having trouble trying to use scipy.stats.multivariate_normal, hopefully one of you might be able to help. I have a 2x2 matrix which is possible to find the inverse of using numpy.linalg.inv(), however when I attempt to use it as the covariance matrix in multivariate_normal I receive a LinAlgError stating that it is a singular matrix:
```

```python
cov = np.array([[3.2e5**2, 3.2e5*0.103*-0.459],[3.2e5*0.103*-0.459, 0.103**2]])
```

```python
np.linalg.inv(cov)
#Output
#array([[  1.23722158e-11,   1.76430200e-05],
#       [  1.76430200e-05,   1.19418880e+02]])
```

```python
multivariate_normal([0,0], cov)
---------------------------------------------------------------------------
LinAlgError                               Traceback (most recent call last)
<ipython-input-91-44a6625beda5> in <module>()
----> 1 multivariate_normal([0,0], cov)

/mnt/ssd/Enthought_jli199/Canopy_64bit/User/lib/python2.7/site-packages/scipy/stats/_multivariate.pyc in __call__(self, mean, cov, allow_singular, seed)
    421         return multivariate_normal_frozen(mean, cov,
    422                                           allow_singular=allow_singular,
--> 423                                           seed=seed)
    424
    425     def _logpdf(self, x, mean, prec_U, log_det_cov, rank):

/mnt/ssd/Enthought_jli199/Canopy_64bit/User/lib/python2.7/site-packages/scipy/stats/_multivariate.pyc in __init__(self, mean, cov, allow_singular, seed)
    591         """
    592         self.dim, self.mean, self.cov = _process_parameters(None, mean, cov)
--> 593         self.cov_info = _PSD(self.cov, allow_singular=allow_singular)
    594         self._dist = multivariate_normal_gen(seed)
    595

/mnt/ssd/Enthought_jli199/Canopy_64bit/User/lib/python2.7/site-packages/scipy/stats/_multivariate.pyc in __init__(self, M, cond, rcond, lower, check_finite, allow_singular)
    217         d = s[s > eps]
    218         if len(d) < len(s) and not allow_singular:
--> 219             raise np.linalg.LinAlgError('singular matrix')
    220         s_pinv = _pinv_1d(s, eps)
    221         U = np.multiply(u, np.sqrt(s_pinv))

LinAlgError: singular matrix
```
```

Response: By default multivariate_normal checks whether any of the eigenvalues of the covariance matrix are less than some tolerance chosen based on its dtype and the magnitude of its largest eigenvalue (take a look at the source code for scipy.stats._multivariate._PSD and scipy.stats._multivariate._eigvalsh_to_eps for the full details). As @kazemakase mentioned above, whilst your covariance matrix may be invertible according to the criteria used by np.linalg.inv, it is still very ill-conditioned and fails the more stringent test used by multivariate_normal. You could pass allow_singular=True to multivariate_normal to skip this test, but in general it would be better to rescale your data to avoid passing such an ill-conditioned covariance matrix in the first place.


Question:
How do I approximate the Jacobian and Hessian of a function numerically? I have a function in Python:
```
def f(x):
    return x[0]**3 + x[1]**2 + 7
    # Actually more than this.
    # No analytical expression
```
It's a scalar valued function of a vector. How can I approximate the Jacobian and Hessian of this function in numpy or scipy numerically?

Response: (Updated in late 2017 because there's been a lot of updates in this space.) Your best bet is probably automatic differentiation. There are now many packages for this, because it's the standard approach in deep learning: Autograd works transparently with most numpy code. It's pure-Python, requires almost no code changes for typical functions, and is reasonably fast. There are many deep-learning-oriented libraries that can do this. Some of the most popular are TensorFlow, PyTorch, Theano, Chainer, and MXNet. Each will require you to rewrite your function in their kind-of-like-numpy-but-needlessly-different API, and in return will give you GPU support and a bunch of deep learning-oriented features that you may or may not care about. FuncDesigner is an older package I haven't used whose website is currently down. Another option is to approximate it with finite differences, basically just evaluating (f(x + eps) - f(x - eps)) / (2 * eps) (but obviously with more effort put into it than that). This will probably be slower and less accurate than the other approaches, especially in moderately high dimensions, but is fully general and requires no code changes. numdifftools seems to be the standard Python package for this. You could also attempt to find fully symbolic derivatives with SymPy, but this will be a relatively manual process.


Question: Compute Jordan normal form of matrix in Python / NumPy In MATLAB you can compute the Jordan normal form of a matrix by using the the function jordan. It there an equivalent function available in NumPy and SciPy?

Response:
The MATLAB jordan function is from the Symbolic Math Toolbox, so it does not seem unreasonable to get its Python replacement from the SymPy library. Specifically, the Matrix class has the method jordan_form. You can pass a numpy array as an argument when you create a sympy Matrix. For example, the following is from the wikipedia article on the Jordan normal form:
```

```python
import numpy as np
```

```python
from sympy import Matrix
```

```python
a = np.array([[5, 4, 2, 1], [0, 1, -1, -1], [-1, -1, 3, 0], [1, 1, -1, 2]])
```

```python
m = Matrix(a)
```

```python
m
#Output
#Matrix([
#[ 5,  4,  2,  1],
#[ 0,  1, -1, -1],
#[-1, -1,  3,  0],
#[ 1,  1, -1,  2]])
```

```python
P, J = m.jordan_form()
```

```python
J
#Output
#Matrix([
#[1, 0, 0, 0],
#[0, 2, 0, 0],
#[0, 0, 4, 1],
#[0, 0, 0, 4]])
#```
```


Question:
Scipy Circular Variance According to my understanding, circular variance has a range between 0 and 1. This is also confirmed in wikipedia as well as here. But for some reasons, circular variance function from scipy.stats gives values above 1.
```
import numpy as np
from scipy.stats import circmean, circvar

a = np.random.randint(0, high=360, size=10)

print(a)
print(circmean(a, 0, 360))
print(circvar(np.deg2rad(a)))
[143 116 152 172 349 152 182 306 345  81]
135.34974541954665
2.2576538466653857
```
Could somebody inform me why I am getting values above 1 from the function circvar

Response:
The less-helpful answer would be since that's how scipy defines it, so you'd better ask the developers to get a definite answer. Really. the example from the docs is
```
from scipy.stats import circvar
circvar([0, 2*np.pi/3, 5*np.pi/3])
2.19722457734
```
So you can't say the behavior is unexpedected. But why is it done that way? Your second link defines the circular variance for a set of n angles a_1, ... a_n as V = 1 − \hat{R_1} Where \hat{R_1} = R_1 / n R_1 = \sqrt{C^2 + S^2} and C = \sum_{i=1}^n cos(a_i) S = \sum_{i=1}^n sin(a_i) The scipy library finds the circular variance by
```
ang = (samples - low)*2.*pi / (high - low)
S = sin(ang).mean(axis=axis)
C = cos(ang).mean(axis=axis)
R = hypot(S, C)
return ((high - low)/2.0/pi)**2 * 2 * log(1/R)
```
That's a bit tricky to understand. If we assume the samples are zero-mean, the range is [0, 2*pi], and the default axis is being used (all true in the example) it can be simplified to
```
S = mean(sin(samples))
C = mean(cos(samples))
R = hypot(S, C)
V = 2 * log(1/R)
```
So the definition used by scipy transforms R by 2*log(1/R), rather than 1-R. That seems odd. Looking through the history, https://github.com/scipy/scipy/blame/v1.1.0/scipy/stats/morestats.py#L2696-L2733, at one point the stats were calculated using
```
ang = (samples - low)*2*pi / (high-low)
res = stats.mean(exp(1j*ang))
V = 1-abs(res)
return ((high-low)/2.0/pi)**2 * V
```
Which seems in line with the definitions you've provided. That was changed in a bugfix at the same time tests were added, but without any reference as to where the new computations came from. Some discussion on the scipy bug tracker is available at https://github.com/scipy/scipy/pull/5747. It suggests the behavior is intentional, and won't be fixed. There's another implementation available in astropy, http://docs.astropy.org/en/stable/api/astropy.stats.circvar.html, which notes The definition used here differs from the one in scipy.stats.circvar. Precisely, Scipy circvar uses an approximation based on the limit of small angles which approaches the linear variance. So, in summary, for unknown reasons scipy uses an approximation (that seems to be rather poor in some cases). However, due to backwards compatibility it won't be fixed, so you may want to use astropy's implementation.


Question: scipy.stats.expon.fit() with no location parameter I am using scipy.stats.expon.fit(data) to fit an exponential distribution to my data. This appears to return two values where I would expect one. The documentation online doesn't seem to say what fit() returns but looking at the source, I am guessing it is both a location and scale parameter. Can you fix the location parameter to 0 when doing the fitting?

Response:
In the call to expon.fit, use floc=0:
```

```python
data = expon.rvs(0, 1.5, 1000)
```

```python
loc, scale = expon.fit(data, floc=0)
```

```python
scale
#Output
#1.4878030368336586
```

```python
loc
#Output
#0
#```
```


Question:
Orthogonal regression fitting in scipy least squares method The leastsq method in scipy lib fits a curve to some data. And this method implies that in this data Y values depends on some X argument. And calculates the minimal distance between curve and the data point in the Y axis (dy) But what if I need to calculate minimal distance in both axes (dy and dx) Is there some ways to implement this calculation? Here is a sample of code when using one axis calculation:
```
import numpy as np
from scipy.optimize import leastsq

xData = [some data...]
yData = [some data...]

def mFunc(p, x, y):
    return y - (p[0]*x**p[1])  # is takes into account only y axis

plsq, pcov = leastsq(mFunc, [1,1], args=(xData,yData))
print plsq
```
I recently tryed scipy.odr library and it returns the proper results only for linear function. For other functions like y=a*x^b it returns wrong results. This is how I use it:
```
def f(p, x):
    return p[0]*x**p[1]

myModel = Model(f)
myData = Data(xData, yData)
myOdr = ODR(myData, myModel , beta0=[1,1])
myOdr.set_job(fit_type=0) #if set fit_type=2, returns the same as leastsq
out = myOdr.run()
out.pprint()
```
This returns wrong results, not desired, and in some input data not even close to real. May be, there is some special ways of using it, what do I do wrong?

Response: I've found the solution. Scipy Odrpack works noramally but it needs a good initial guess for correct results. So I divided the process into two steps. First step: find the initial guess by using ordinaty least squares method. Second step: substitude these initial guess in ODR as beta0 parameter. And it works very well with an acceptable speed. Thank you guys, your advice directed me to the right solution


Question: Equivalent of Matlab's cluster quality function? MATLAB has a nice silhouette function to help evaluate the number of clusters for k-means. Is there an equivalent for Python's Numpy/Scipy as well?

Response:
I present below a sample silhouette implementation in both MATLAB and Python/Numpy (keep in mind that I am more fluent in MATLAB): 1) MATLAB
```matlab
function s = mySilhouette(X, IDX)
    %# X  : matrix of size N-by-p, data where rows are instances
    %# IDX: vector of size N, cluster index of each instance (starting from 1)
    %# s  : vector of size N, silhouette score value of each instance

    N = size(X,1);            %# number of instances
    K = numel(unique(IDX));   %# number of clusters

    %# compute pairwise distance matrix
    D = squareform( pdist(X,'euclidean').^2 );

    %# indices belonging to each cluster
    kIndices = accumarray(IDX, 1:N, [K 1], @(x){sort(x)});

    %# compute a,b,s for each instance
    %# a(i): average distance from i to all other data within the same cluster.
    %# b(i): lowest average dist from i to the data of another single cluster
    a = zeros(N,1);
    b = zeros(N,1);
    for i=1:N
        ind = kIndices{IDX(i)}; ind = ind(ind~=i);
        a(i) = mean( D(i,ind) );
        b(i) = min( cellfun(@(ind) mean(D(i,ind)), kIndices([1:K]~=IDX(i))) );
    end
    s = (b-a) ./ max(a,b);
end
```
To emulate the plot from the silhouette function in MATLAB, we group the silhouette values by cluster, sort within each, then plot the bars horizontally. MATLAB adds NaNs to separate the bars from the different clusters, I found it easier to simply color-code the bars:
```matlab
%# sample data
load fisheriris
X = meas;
N = size(X,1);

%# cluster and compute silhouette score
K = 3;
[IDX,C] = kmeans(X, K, 'distance','sqEuclidean');
s = mySilhouette(X, IDX);

%# plot
[~,ord] = sortrows([IDX s],[1 -2]);
indices = accumarray(IDX(ord), 1:N, [K 1], @(x){sort(x)});
ytick = cellfun(@(ind) (min(ind)+max(ind))/2, indices);
ytickLabels = num2str((1:K)','%d');           %#'

h = barh(1:N, s(ord),'hist');
set(h, 'EdgeColor','none', 'CData',IDX(ord))
set(gca, 'CLim',[1 K], 'CLimMode','manual')
set(gca, 'YDir','reverse', 'YTick',ytick, 'YTickLabel',ytickLabels)
xlabel('Silhouette Value'), ylabel('Cluster')

%# compare against SILHOUETTE
figure, silhouette(X,IDX)
```
2) Python And here is what I came up with in Python:
```py
import numpy as np
from scipy.cluster.vq import kmeans2
from scipy.spatial.distance import pdist, squareform
from sklearn import datasets
import matplotlib.pyplot as plt
from matplotlib import cm

def silhouette(X, cIDX):
    """
    Computes the silhouette score for each instance of a clustered dataset,
    which is defined as:
        s(i) = (b(i)-a(i)) / max{a(i),b(i)}
    with:
        -1 <= s(i) <= 1

    Args:
        X    : A M-by-N array of M observations in N dimensions
        cIDX : array of len M containing cluster indices (starting from zero)

    Returns:
        s    : silhouette value of each observation
    """

    N = X.shape[0]              # number of instances
    K = len(np.unique(cIDX))    # number of clusters

    # compute pairwise distance matrix
    D = squareform(pdist(X))

    # indices belonging to each cluster
    kIndices = [np.flatnonzero(cIDX==k) for k in range(K)]

    # compute a,b,s for each instance
    a = np.zeros(N)
    b = np.zeros(N)
    for i in range(N):
        # instances in same cluster other than instance itself
        a[i] = np.mean( [D[i][ind] for ind in kIndices[cIDX[i]] if ind!=i] )
        # instances in other clusters, one cluster at a time
        b[i] = np.min( [np.mean(D[i][ind])
                        for k,ind in enumerate(kIndices) if cIDX[i]!=k] )
    s = (b-a)/np.maximum(a,b)

    return s

def main():
    # load Iris dataset
    data = datasets.load_iris()
    X = data['data']

    # cluster and compute silhouette score
    K = 3
    C, cIDX = kmeans2(X, K)
    s = silhouette(X, cIDX)

    # plot
    order = np.lexsort((-s,cIDX))
    indices = [np.flatnonzero(cIDX[order]==k) for k in range(K)]
    ytick = [(np.max(ind)+np.min(ind))/2 for ind in indices]
    ytickLabels = ["%d" % x for x in range(K)]
    cmap = cm.jet( np.linspace(0,1,K) ).tolist()
    clr = [cmap[i] for i in cIDX[order]]

    fig = plt.figure()
    ax = fig.add_subplot(111)
    ax.barh(range(X.shape[0]), s[order], height=1.0,
            edgecolor='none', color=clr)
    ax.set_ylim(ax.get_ylim()[::-1])
    plt.yticks(ytick, ytickLabels)
    plt.xlabel('Silhouette Value')
    plt.ylabel('Cluster')
    plt.show()

if __name__ == '__main__':
    main()
```
Update: As noted by others, scikit-learn has since then added its own silhouette metric implementation. To use it in the above code, replace the call to the custom-defined silhouette function with:
```py
from sklearn.metrics import silhouette_samples

...

#s = silhouette(X, cIDX)
s = silhouette_samples(X, cIDX)    # <-- scikit-learn function

...
```
the rest of the code can still be used as-is to generate the exact same plot.


Question:
Classifying Python array by nearest "seed" region? I have a raster of ecological habitats which I've converted into a two-dimensional Python numpy array (example_array below). I also have an array containing "seed" regions with unique values (seed_array below) which I'd like to use to classify my habitat regions. I'd like to 'grow' my seed regions 'into' my habitat regions such that habitats are assigned the ID of the nearest seed region, as measured 'through' the habitat regions. For example: My best approach used the ndimage.distance_transform_edt function to create an array depicting the nearest "seed" region to each cell in the dataset, which was then substituted back into the habitat array. This doesn't work particularly well, however, as the function doesn't measure distances "through" my habitat regions, for example below where the red circle represents an incorrectly classified cell: Below are sample arrays for my habitat and seed data, and an example of the kind of output I'm looking for. My actual datasets are much larger - over a million habitat/seed regions. Any help would be much appreciated!
```
import numpy as np
import scipy.ndimage as ndimage
import matplotlib.pyplot as plt

# Sample study area array
example_array = np.array([[0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0],
                          [0, 0, 0, 1, 1, 1, 0, 0, 0, 1, 1, 1],
                          [0, 0, 0, 0, 1, 1, 0, 0, 0, 1, 1, 1],
                          [0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0],
                          [1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1],
                          [1, 1, 0, 1, 0, 0, 0, 0, 1, 1, 1, 1],
                          [1, 1, 1, 1, 0, 0, 1, 1, 1, 0, 0, 1],
                          [1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0],
                          [1, 1, 1, 1, 0, 0, 1, 1, 1, 0, 0, 0],
                          [1, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0],
                          [1, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0],
                          [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]])

# Plot example array
plt.imshow(example_array, cmap="spectral", interpolation='nearest')

seed_array = np.array([[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
                       [0, 0, 0, 0, 4, 0, 0, 0, 0, 0, 0, 0],
                       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3, 0],
                       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
                       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
                       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
                       [0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
                       [0, 1, 1, 1, 0, 0, 2, 2, 0, 0, 0, 0],
                       [0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0],
                       [0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0],
                       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
                       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]])

# Plot seeds
plt.imshow(seed_array, cmap="spectral", interpolation='nearest')

desired_output = np.array([[0, 0, 0, 0, 4, 0, 0, 0, 0, 0, 0, 0],
                           [0, 0, 0, 4, 4, 4, 0, 0, 0, 3, 3, 3],
                           [0, 0, 0, 0, 4, 4, 0, 0, 0, 3, 3, 3],
                           [0, 0, 0, 0, 0, 0, 0, 0, 3, 0, 3, 0],
                           [1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 3, 3],
                           [1, 1, 0, 1, 0, 0, 0, 0, 2, 2, 3, 3],
                           [1, 1, 1, 1, 0, 0, 2, 2, 2, 0, 0, 3],
                           [1, 1, 1, 1, 1, 2, 2, 2, 2, 0, 0, 0],
                           [1, 1, 1, 1, 0, 0, 2, 2, 2, 0, 0, 0],
                           [1, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0],
                           [1, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0],
                           [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]])

# Plot desired output
plt.imshow(desired_output, cmap="spectral", interpolation='nearest')
```

Response:
You can use watershed segmentation from scikits-image: Distance transform
```
from scipy import ndimage as nd
distance = nd.distance_transform_edt(example_array)
```
Watershed segmentation
```
from skimage.morphology import watershed, square
result = watershed(-distance, seed_array, mask=example_array, \
                   connectivity=square(3))
```
Result
```
subplot(1,2,1)
imshow(-distance, 'spectral', interpolation='none')
subplot(1,2,2)
imshow(result, 'spectral', interpolation='none')
```
As another variant, and following your initial approach, you can use watershed to find connected neighbours to nearest seeds. As you mentioned in the question: Calculate distance to the seeds:
```
distance = nd.distance_transform_edt(seed_array == 0)
```
Calculate watershed in the distance space:
```
result = watershed(distance, seed_array, mask=example_array, \
                   connectivity=square(3))
```
Plot result:
```
figure(figsize=(9,3))
subplot(1,3,1)
imshow(distance, 'jet', interpolation='none')
subplot(1,3,2)
imshow(np.ma.masked_where(example_array==0, distance), 'jet', interpolation='none')
subplot(1,3,3)
imshow(result, 'spectral', interpolation='none')
```
Further discussion: Watershed method tries to grow regions from seeded peaks by flowing through the image gradient. As your image is binary, the regions will expand equally in all directions from the seeded points, and thus give you the point in between two regions. For more info about watershed refer to wikipedia. In the first example, the distance transform is calculated in the original image, and thus the regions expand equally from seeds until they achieve the splitting point in the middle. In the second example, the distance transform is calculated from all the pixels to any of the seeded points, and then applying watershed in that space. Watershed basically will assign each pixel to its nearest seed, but it will add a connectivity constrain. NOTE the sign difference in the distance maps in both plotting and watersed. NOTE In distance maps (left image in both plots), blue means close where red means far.


Question: Filter values from a scipy sparse matrix I am trying to filter values smaller than 10 from a huge (1Mx1M) CSR matrix (SciPy). Since all my values are integers, dividing by 10 and remultiplying by 10 does the job, but I was wondering if there isn't a better way to go about filtering elements. EDIT: The answer below works. Check that you have the latest version of SciPy.

Response:
You can also go with the less hacky, but probably slower:
```
m = m.multiply(m >= 10)
```
To understand what's going on:
```

```python
m = scipy.sparse.csr_matrix((1000, 1000), dtype=np.int)
```

```python
m[np.random.randint(0, 1000, 20),
#Output
#      np.random.randint(0, 1000, 20)] = np.random.randint(0, 100, 20)
```

```python
m.data
#Output
#array([92, 46, 99, 24, 75, 16, 49, 60, 87, 64, 91, 37, 30, 32, 25, 40, 99,
#        9,  3, 84])
```

```python
m >= 10
#Output
#<1000x1000 sparse matrix of type '<type 'numpy.bool_'>'
#    with 18 stored elements in Compressed Sparse Row format>
```

```python
m = m.multiply(m >= 10)
```

```python
m
#Output
#<1000x1000 sparse matrix of type '<type 'numpy.int32'>'
#    with 18 stored elements in Compressed Sparse Row format>
```

```python
m.data
#Output
#array([92, 46, 99, 24, 75, 16, 49, 60, 87, 64, 91, 37, 30, 32, 25, 40, 99,
#       84])
#```
```


Question:
Get the formula of a interpolation function created by scipy I have done some work in Python, but I'm new to scipy. I'm trying to use the methods from the interpolate library to come up with a function that will approximate a set of data. I've looked up some examples to get started, and could get the sample code below working in Python(x,y):
```
import numpy as np
from scipy.interpolate import interp1d, Rbf
import pylab as P

# show the plot (empty for now)
P.clf()
P.show()

# generate random input data
original_data = np.linspace(0, 1, 10)

# random noise to be added to the data
noise = (np.random.random(10)*2 - 1) * 1e-1

# calculate f(x)=sin(2*PI*x)+noise
f_original_data = np.sin(2 * np.pi * original_data) + noise

# create interpolator
rbf_interp = Rbf(original_data, f_original_data, function='gaussian')

# Create new sample data (for input), calculate f(x)
#using different interpolation methods
new_sample_data = np.linspace(0, 1, 50)
rbf_new_sample_data    = rbf_interp(new_sample_data)

# draw all results to compare
P.plot(original_data, f_original_data, 'o', ms=6, label='f_original_data')
P.plot(new_sample_data, rbf_new_sample_data, label='Rbf interp')
P.legend()
```
The plot is displayed as follows: Now, is there any way to get a polynomial expression representing the interpolated function created by Rbf (i.e. the method created as rbf_interp)? Or, if this is not possible with Rbf, any suggestions using a different interpolation method, another library, or even a different tool are also welcome.

Response: The answer is no, there is no "nice" way to write down the formula, or at least not in a short way. Some types of interpolations, like RBF and Loess, do not directly search for a parametric mathematical function to fit to the data and instead they calculate the value of each new data point separately as a function of the other points. These interpolations are guaranteed to always give a good fit for your data (such as in your case), and the reason for this is that to describe them you need a very large number of parameters (basically all your data points). Think of it this way: you could interpolate linearly by connecting consecutive data points with straight lines. You could fit any data this way and then describe the function in a mathematical form, but it would take a large number of parameters (at least as many as the number of points). Actually what you are doing right now is pretty much a smoothed version of that. If you want the formula to be short, this means you want to describe the data with a mathematical function that does not have many parameters (specifically the number of parameters should be much lower than the number of data points). Such examples are logistic functions, polynomial functions and even the sine function (that you used to generate the data). Obviously, if you know which function generated the data that will be the function you want to fit.


Question:
Convolution computations in Numpy/Scipy Profiling some computational work I'm doing showed me that one bottleneck in my program was a function that basically did this (np is numpy, sp is scipy):
```
def mix1(signal1, signal2):
    spec1 = np.fft.fft(signal1, axis=1)
    spec2 = np.fft.fft(signal2, axis=1)
    return np.fft.ifft(spec1*spec2, axis=1)
```
Both signals have shape (C, N) where C is the number of sets of data (usually less than 20) and N is the number of samples in each set (around 5000). The computation for each set (row) is completely independent of any other set. I figured that this was just a simple convolution, so I tried to replace it with:
```
def mix2(signal1, signal2):
    outputs = np.empty_like(signal1)

    for idx, row in enumerate(outputs):
        outputs[idx] = sp.signal.convolve(signal1[idx], signal2[idx], mode='same')

    return outputs
```
...just to see if I got the same results. But I didn't, and my questions are: Why not? Is there a better way to compute the equivalent of mix1()? (I realise that mix2 probably wouldn't have been faster as-is, but it might have been a good starting point for parallelisation.) Here's the full script I used to quickly check this:
```
import numpy as np
import scipy as sp
import scipy.signal

N = 4680
C = 6

def mix1(signal1, signal2):
    spec1 = np.fft.fft(signal1, axis=1)
    spec2 = np.fft.fft(signal2, axis=1)
    return np.fft.ifft(spec1*spec2, axis=1)

def mix2(signal1, signal2):
    outputs = np.empty_like(signal1)

    for idx, row in enumerate(outputs):
        outputs[idx] = sp.signal.convolve(signal1[idx], signal2[idx], mode='same')

    return outputs

def test(num, chans):
    sig1 = np.random.randn(chans, num)
    sig2 = np.random.randn(chans, num)
    res1 = mix1(sig1, sig2)
    res2 = mix2(sig1, sig2)

    np.testing.assert_almost_equal(res1, res2)

if __name__ == "__main__":
    np.random.seed(0x1234ABCD)
    test(N, C)
```

Response:
So I tested this out and can now confirm a few things: 1) numpy.convolve is not circular, which is what the fft code is giving you: 2) FFT does not internally pad to a power of 2. Compare the vastly different speeds of the following operations:
```
x1 = np.random.uniform(size=2**17-1)
x2 = np.random.uniform(size=2**17)

np.fft.fft(x1)
np.fft.fft(x2)
```
3) Normalization is not a difference -- if you do a naive circular convolution by adding up a(k)*b(i-k), you will get the result of the FFT code. The thing is padding to a power of 2 is going to change the answer. I've heard tales that there are ways to deal with this by cleverly using prime factors of the length (mentioned but not coded in Numerical Recipes) but I've never seen people actually do that.


Question:
Chi squared test in Python I'd like to run a chi-squared test in Python. I've created code to do this, but I don't know if what I'm doing is right, because the scipy docs are quite sparse. Background first: I have two groups of users. My null hypothesis is that there is no significant difference in whether people in either group are more likely to use desktop, mobile, or tablet. These are the observed frequencies in the two groups:
```
[[u'desktop', 14452], [u'mobile', 4073], [u'tablet', 4287]]
[[u'desktop', 30864], [u'mobile', 11439], [u'tablet', 9887]]
```
Here is my code using scipy.stats.chi2_contingency:
```
obs = np.array([[14452, 4073, 4287], [30864, 11439, 9887]])
chi2, p, dof, expected = stats.chi2_contingency(obs)
print p
```
This gives me a p-value of 2.02258737401e-38, which clearly is significant. My question is: does this code look valid? In particular, I'm not sure whether I should be using scipy.stats.chi2_contingency or scipy.stats.chisquare, given the data I have.

Response: I can't comment too much on the use of the function. However, the issue at hand may be statistical in nature. The very small p-value you are seeing is most likely a result of your data containing large frequencies ( in the order of ten thousand). When sample sizes are too large, any differences will become significant - hence the small p-value. The tests you are using are very sensitive to sample size. See here for more details.


Question:
filling gaps on an image using numpy and scipy The image (test.tif) is attached. The np.nan values are the whitest region. How to fill those whitest region using some gap filling algorithms that uses values from the neighbours?
```
import scipy.ndimage

data = ndimage.imread('test.tif')
```

Response:
As others have suggested, scipy.interpolate can be used. However, it requires fairly extensive index manipulation to get this to work. Complete example:
```
from pylab import *
import numpy
import scipy.ndimage
import scipy.interpolate
import pdb

data = scipy.ndimage.imread('data.png')

# a boolean array of (width, height) which False where there are missing values and True where there are valid (non-missing) values
mask = ~( (data[:,:,0] == 255) & (data[:,:,1] == 255) & (data[:,:,2] == 255) )

# array of (number of points, 2) containing the x,y coordinates of the valid values only
xx, yy = numpy.meshgrid(numpy.arange(data.shape[1]), numpy.arange(data.shape[0]))
xym = numpy.vstack( (numpy.ravel(xx[mask]), numpy.ravel(yy[mask])) ).T

# the valid values in the first, second, third color channel,  as 1D arrays (in the same order as their coordinates in xym)
data0 = numpy.ravel( data[:,:,0][mask] )
data1 = numpy.ravel( data[:,:,1][mask] )
data2 = numpy.ravel( data[:,:,2][mask] )

# three separate interpolators for the separate color channels
interp0 = scipy.interpolate.NearestNDInterpolator( xym, data0 )
interp1 = scipy.interpolate.NearestNDInterpolator( xym, data1 )
interp2 = scipy.interpolate.NearestNDInterpolator( xym, data2 )

# interpolate the whole image, one color channel at a time
result0 = interp0(numpy.ravel(xx), numpy.ravel(yy)).reshape( xx.shape )
result1 = interp1(numpy.ravel(xx), numpy.ravel(yy)).reshape( xx.shape )
result2 = interp2(numpy.ravel(xx), numpy.ravel(yy)).reshape( xx.shape )

# combine them into an output image
result = numpy.dstack( (result0, result1, result2) )

imshow(result)
show()
```
Output: This passes to the interpolator all values we have, not just the ones next to the missing values (which may be somewhat inefficient). It also interpolates every point in the output, not just the missing values (which is extremely inefficient). A better way is to interpolate just the missing values, and then patch them into the original image. This is just a quick working example to get started :)


Question:
Numpy Cholesky decomposition LinAlgError In my attempt to perform cholesky decomposition on a variance-covariance matrix for a 2D array of periodic boundary condition, under certain parameter combinations, I always get LinAlgError: Matrix is not positive definite - Cholesky decomposition cannot be computed. Not sure if it's a numpy.linalg or implementation issue, as the script is straightforward:
```
sigma = 3.
U = 4

def FromListToGrid(l_):
    i = np.floor(l_/U)
    j = l_ - i*U
    return np.array((i,j))

Ulist = range(U**2)

Cov = []
for l in Ulist:
    di = np.array([np.abs(FromListToGrid(l)[0]-FromListToGrid(i)[0]) for i, x in enumerate(Ulist)])
    di = np.minimum(di, U-di)

    dj = np.array([np.abs(FromListToGrid(l)[1]-FromListToGrid(i)[1]) for i, x in enumerate(Ulist)])
    dj = np.minimum(dj, U-dj)

    d = np.sqrt(di**2+dj**2)
    Cov.append(np.exp(-d/sigma))
Cov = np.vstack(Cov)

W = np.linalg.cholesky(Cov)
```
Attempts to remove potential singularies also failed to resolve the problem. Any help is much appreciated.

Response:
Digging a bit deeper in problem, I tried printing the Eigenvalues of the Cov matrix.
```
print np.linalg.eigvalsh(Cov)
```
And the answer turns out to be this
```
[-0.0801339  -0.0801339   0.12653595  0.12653595  0.12653595  0.12653595 0.14847999  0.36269785  0.36269785  0.36269785  0.36269785  1.09439988 1.09439988  1.09439988  1.09439988  9.6772531 ]
```
Aha! Notice the first two negative eigenvalues? Now, a matrix is positive definite if and only if all its eigenvalues are positive. So, the problem with the matrix is not that it's close to 'zero', but that it's 'negative'. To extend @duffymo analogy, this is linear algebra equivalent of trying to take square root of negative number. Now, let's try to perform same operation, but this time with scipy.
```
scipy.linalg.cholesky(Cov, lower=True)
```
And that fails saying something more
```
numpy.linalg.linalg.LinAlgError: 12-th leading minor not positive definite
```
That's telling something more, (though I couldn't really understand why it's complaining about 12-th minor). Bottom line, the matrix is not quite close to 'zero' but is more like 'negative'


Question:
Most efficient way to construct similarity matrix I'm using the following links to create a "Euclidean Similarity Matrix" (that I convert to a DataFrame). https://stats.stackexchange.com/questions/53068/euclidean-distance-score-and-similarity http://docs.scipy.org/doc/scipy-0.14.0/reference/generated/scipy.spatial.distance.euclidean.html The way I'm doing it is an iterative approach which works but it takes a while when the datasets are big. The pandas pd.DataFrame.corr() is really fast and useful for pearson correlations. How can I perform a Euclidean Similarity measure w/o exhaustive iteration? My naive code below:
```
#Euclidean Similarity

#Create DataFrame
DF_var = pd.DataFrame.from_dict({"s1":[1.2,3.4,10.2],"s2":[1.4,3.1,10.7],"s3":[2.1,3.7,11.3],"s4":[1.5,3.2,10.9]}).T
DF_var.columns = ["g1","g2","g3"]
#      g1   g2    g3
# s1  1.2  3.4  10.2
# s2  1.4  3.1  10.7
# s3  2.1  3.7  11.3
# s4  1.5  3.2  10.9

#Create empty matrix to fill
M_euclid = np.zeros((DF_var.shape[1],DF_var.shape[1]))

#Iterate through DataFrame columns to measure euclidean distance
for i in range(DF_var.shape[1]):
    u = DF_var[DF_var.columns[i]]
    for j in range(DF_var.shape[1]):
        v = DF_var[DF_var.columns[j]]
        #Euclidean distance -> Euclidean similarity
        M_euclid[i,j] = (1/(1+sp.spatial.distance.euclidean(u,v)))
DF_euclid = pd.DataFrame(M_euclid,columns=DF_var.columns,index=DF_var.columns)

#           g1        g2        g3
# g1  1.000000  0.215963  0.051408
# g2  0.215963  1.000000  0.063021
# g3  0.051408  0.063021  1.000000
```

Response:
There are two useful function within scipy.spatial.distance that you can use for this: pdist and squareform. Using pdist will give you the pairwise distance between observations as a one-dimensional array, and squareform will convert this to a distance matrix. One catch is that pdist uses distance measures by default, and not similarity, so you'll need to manually specify your similarity function. Judging by the commented output in your code, your DataFrame is also not in the orientation pdist expects, so I've undone the transpose you did in your code.
```
import pandas as pd
from scipy.spatial.distance import euclidean, pdist, squareform

def similarity_func(u, v):
    return 1/(1+euclidean(u,v))

DF_var = pd.DataFrame.from_dict({"s1":[1.2,3.4,10.2],"s2":[1.4,3.1,10.7],"s3":[2.1,3.7,11.3],"s4":[1.5,3.2,10.9]})
DF_var.index = ["g1","g2","g3"]

dists = pdist(DF_var, similarity_func)
DF_euclid = pd.DataFrame(squareform(dists), columns=DF_var.index, index=DF_var.index)
```


Question: pyspark: sparse vectors to scipy sparse matrix I have a spark dataframe with a column of short sentences, and a column with a categorical variable. I'd like to perform tf-idf on the sentences, one-hot-encoding on the categorical variable and then output it to a sparse matrix on my driver once it's much smaller in size (for a scikit-learn model). What is the best way to get the data out of spark in sparse form? It seems like there is only a toArray() method on sparse vectors, which outputs numpy arrays. However, the docs do say that scipy sparse arrays can be used in the place of spark sparse arrays. Keep in mind also that the tf_idf values are in fact a column of sparse arrays. Ideally it would be nice to get all these features into one large sparse matrix.

Response:
One possible solution can be expressed as follows: convert features to RDD and extract vectors:
```
from pyspark.ml.linalg import SparseVector
from operator import attrgetter

df = sc.parallelize([
    (SparseVector(3, [0, 2], [1.0, 3.0]), ),
    (SparseVector(3, [1], [4.0]), )
]).toDF(["features"])

features = df.rdd.map(attrgetter("features"))
```
add row indices:
```
indexed_features = features.zipWithIndex()
```
flatten to RDD of tuples (i, j, value):
```
def explode(row):
    vec, i = row
    for j, v in zip(vec.indices, vec.values):
        yield i, j, v

entries = indexed_features.flatMap(explode)
```
collect and reshape:
```
row_indices, col_indices, data = zip(*entries.collect())
```
compute shape:
```
shape = (
    df.count(),
    df.rdd.map(attrgetter("features")).first().size
)
```
create sparse matrix:
```
from scipy.sparse import csr_matrix

mat = csr_matrix((data, (row_indices, col_indices)), shape=shape)
```
quick sanity check:
```
mat.todense()
```
With expected result:
```none
matrix([[ 1.,  0.,  3.],
        [ 0.,  4.,  0.]])
```
Another one: convert each row of features to matrix:
```
import numpy as np

def as_matrix(vec):
    data, indices = vec.values, vec.indices
    shape = 1, vec.size
    return csr_matrix((data, indices, np.array([0, vec.values.size])), shape)

mats = features.map(as_matrix)
```
and reduce with vstack:
```
from scipy.sparse import vstack

mat = mats.reduce(lambda x, y: vstack([x, y]))
```
or collect and vstack
```
mat = vstack(mats.collect())
```


Question: How to calculate the 99% confidence interval for the slope in a linear regression model in python? We have following linear regression: y ~ b0 + b1 * x1 + b2 * x2. I know that regress function in Matlab does calculate it, but numpy's linalg.lstsq doesn't (https://docs.scipy.org/doc/numpy-dev/user/numpy-for-matlab-users.html).

Response:
StatsModels' RegressionResults has a conf_int() method. Here an example using it (minimally modified version of their Ordinary Least Squares example):
```
import numpy as np, statsmodels.api as sm

nsample = 100
x = np.linspace(0, 10, nsample)
X = np.column_stack((x, x**2))
beta = np.array([1, 0.1, 10])
e = np.random.normal(size=nsample)

X = sm.add_constant(X)
y = np.dot(X, beta) + e

mod = sm.OLS(y, X)
res = mod.fit()
print res.conf_int(0.01)   # 99% confidence interval
```


Question: What do the different values of the kind argument mean in scipy.interpolate.interp1d? The SciPy documentation explains that interp1d's kind argument can take the values ‘linear’, ‘nearest’, ‘zero’, ‘slinear’, ‘quadratic’, ‘cubic’. The last three are spline orders and 'linear' is self-explanatory. What do 'nearest' and 'zero' do?

Response:
nearest "snaps" to the nearest data point. zero is a zero order spline. It's value at any point is the last raw value seen. linear performs linear interpolation and slinear uses a first order spline. They use different code and can produce similar but subtly different results. quadratic uses second order spline interpolation. cubic uses third order spline interpolation. Note that the k parameter can also accept an integer specifying the order of spline interpolation.
```
import numpy as np
import matplotlib.pyplot as plt
import scipy.interpolate as interpolate

np.random.seed(6)
kinds = ('nearest', 'zero', 'linear', 'slinear', 'quadratic', 'cubic')

N = 10
x = np.linspace(0, 1, N)
y = np.random.randint(10, size=(N,))

new_x = np.linspace(0, 1, 28)
fig, axs = plt.subplots(nrows=len(kinds)+1, sharex=True)
axs[0].plot(x, y, 'bo-')
axs[0].set_title('raw')
for ax, kind in zip(axs[1:], kinds):
    new_y = interpolate.interp1d(x, y, kind=kind)(new_x)
    ax.plot(new_x, new_y, 'ro-')
    ax.set_title(kind)

plt.show()
```


Question:
How numpy.cov() function is implemented? I have my own implementation of the covariance function based on the equation:
```
'''
Calculate the covariance coefficient between two variables.
'''

import numpy as np

X = np.array([171, 184, 210, 198, 166, 167])
Y = np.array([78, 77, 98, 110, 80, 69])

# Expected value function.
def E(X, P):
    expectedValue = 0
    for i in np.arange(0, np.size(X)):
        expectedValue += X[i] * (P[i] / np.size(X))
    return expectedValue

# Covariance coefficient function.
def covariance(X, Y):
    '''
    Calculate the product of the multiplication for each pair of variables
    values.
    '''
    XY = X * Y

    # Calculate the expected values for each variable and for the XY.
    EX = E(X, np.ones(np.size(X)))
    EY = E(Y, np.ones(np.size(Y)))
    EXY = E(XY, np.ones(np.size(XY)))

    # Calculate the covariance coefficient.
    return EXY - (EX * EY)

# Display matrix of the covariance coefficient values.
covMatrix = np.array([[covariance(X, X), covariance(X, Y)],
[covariance(Y, X), covariance(Y, Y)]])
print("My function:", covMatrix)

# Display standard numpy.cov() covariance coefficient matrix.
print("Numpy.cov() function:", np.cov([X, Y]))
```
But the problem is, that I'm getting different values from my function and from numpy.cov(), ie:
```
My function: [[ 273.88888889  190.61111111]
 [ 190.61111111  197.88888889]]
Numpy.cov() function: [[ 328.66666667  228.73333333]
 [ 228.73333333  237.46666667]]
```
Why is that? How is numpy.cov() function implemented? If the function numpy.cov() is well-implemented, what am I doing wrong? I'll just say, that results from my function covariance() are consistent with the results from paper examples in the internet for calculating the covariance coefficient, eg http://www.naukowiec.org/wzory/statystyka/kowariancja_11.html.

Response:
The numpy function has a different normalization to yours as a default setting. Try instead
```

```python
np.cov([X, Y], ddof=0)
#Output
#array([[ 273.88888889,  190.61111111],
#       [ 190.61111111,  197.88888889]])
#```
#References: http://docs.scipy.org/doc/numpy/reference/generated/numpy.cov.html http://en.wikipedia.org/wiki/Covariance#Calculating_the_sample_covariance
```


Question:
One-sided Wilcoxon signed-rank test using scipy I would like to perform a one-sided wilcoxon rank test to my paired data, as I'm interested if one sample is significantly greater than the other. Scipy offers
```
scipy.stats.wilcoxon(x,y)
```
to perform a two-sided test with paired samples x and y. Since I can't assume a normal (symmetric) distribution, I can't derive the one-sided p-value from the two-sided p-value. Does anybody now a python way to get the p-values for a one-sided test? Thanks!

Response:
P value returned by scipy.stats.wilcoxon has nothing to do with the distribution of x or y, nor the difference between them. It is determined by the Wilcoxon test statistic (W as it in http://en.wikipedia.org/wiki/Wilcoxon_signed-rank_test, or T as in scipy), which is assumed to follow a normal distribution. If you check the source (in ~python_directory\site-packages\scipy\stats\morestats.py), you will find the last few lines of def wilcoxon():
```
se = sqrt(se / 24)
z = (T - mn) / se
prob = 2. * distributions.norm.sf(abs(z))
return T, prob
```
and:
```
mn = count*(count + 1.) * 0.25
se = count*(count + 1.) * (2. * count + 1.)
```
Where count is the number of non-zero difference between x and y. So, to get one-side p value, you just need prob/2. or 1-prob/2. Examples: In Python:
```

```python
y1=[125,115,130,140,140,115,140,125,140,135]
```

```python
y2=[110,122,125,120,140,124,123,137,135,145]
```

```python
ss.wilcoxon(y1, y2)
#Output
#(18.0, 0.5936305914425295)
#```
#In R:
#```
#> wilcox.test(y1, y2, paired=TRUE, exact=FALSE, correct=FALSE)
```

Wilcoxon signed rank test

data:  y1 and y2
V = 27, p-value = 0.5936
alternative hypothesis: true location shift is not equal to 0

> wilcox.test(y1, y2, paired=TRUE, exact=FALSE, correct=FALSE, alt='greater')

        Wilcoxon signed rank test

data:  y1 and y2
V = 27, p-value = 0.2968
alternative hypothesis: true location shift is greater than 0
```


Question: The difference of pseudo-inverse between SciPy and Numpy I found that there're two versions of pinv() function, which calculates the pseudo-inverse of a matrix in Scipy and numpy, the documents can be viewed at: http://docs.scipy.org/doc/numpy/reference/generated/numpy.linalg.pinv.html http://docs.scipy.org/doc/scipy/reference/generated/scipy.linalg.pinv.html The problem is that I have a 50000*5000 matrix, when using scipy.linalg.pinv, it costs me more than 20GB of memory. But when I use numpy.linalg.pinv, only less than 1GB of memory is used.. I was wondering why numpy and scipy both have a pinv under different implemention. And why their performances are so different.

Response: I can't speak as to why there are implementations in both scipy and numpy, but I can explain why the behaviour is different. numpy.linalg.pinv approximates the Moore-Penrose psuedo inverse using an SVD (the lapack method dgesdd to be precise), whereas scipy.linalg.pinv solves a model linear system in the least squares sense to approximate the pseudo inverse (using dgelss). This is why their performance is different. I would expect the overall accuracy of the resulting pseudo inverse estimates to be somewhat different as well. You might find that scipy.linalg.pinv2 performs more similarly to numpy.linalg.pinv, as it too uses an SVD method, rather than a least sqaures approximation.


Question:
logistic / sigmoid function implementation numerical precision in scipy.special.expit, logistic function is implemented like the following:
```
if x < 0
    a = exp(x)
    a / (1 + a)
else
    1 / (1 + exp(-x))
```
However, I have seen implementations in other languages/frameworks that simply do
```
1 / (1 + exp(-x))
```
I am wondering how much benefit the scipy version actually brings. For very small x, the result approaches to 0. It works even if exp(-x) overflows to Inf.

Response: It's really just for stability - putting in values that are very large in magnitude might return unexpected results otherwise. If expit was implemented just as 1 / (1 + exp(-x)) then putting a value of -710 into the function would return nan, whereas -709 would give a value close to zero as expected. This is because exp(710) is too big to be a double. The branching in the code just means that this scenario is avoided. See also this question and answer on Stack Overflow.


Question:
Using scipy gaussian kernel density estimation to calculate CDF inverse The gaussian_kde function in scipy.stats has a function evaluate that can returns the value of the PDF of an input point. I'm trying to use gaussian_kde to estimate the inverse CDF. The motivation is for generating Monte Carlo realizations of some input data whose statistical distribution is numerically estimated using KDE. Is there a method bound to gaussian_kde that serves this purpose? The example below shows how this should work for the case of a Gaussian distribution. First I show how to do the PDF calculation to set up the specific API I'm trying to achieve:
```
import numpy as np
from scipy.stats import norm, gaussian_kde

npts_kde = int(5e3)
n = np.random.normal(loc=0, scale=1, size=npts_kde)
kde = gaussian_kde(n)

npts_sample = int(1e3)
x = np.linspace(-3, 3, npts_sample)
kde_pdf = kde.evaluate(x)
norm_pdf = norm.pdf(x)
```
Is there an analogously simple way to compute the inverse CDF? The norm function has a very handy isf function that does exactly this:
```
cdf_value = np.sort(np.random.rand(npts_sample))
cdf_inv = norm.isf(1 - cdf_value)
```
Does such a function exist for kde_gaussian? Or is it straightforward to construct such a function from the already implemented methods?

Response:
The question has been answered in the other answers but it took me a while to wrap my mind around everything. Here is a complete example of the final solution:
```
import numpy as np
from scipy import interpolate
from scipy.special import ndtr
import matplotlib.pyplot as plt
from scipy.stats import norm, gaussian_kde

# create kde
npts_kde = int(5e3)
n = np.random.normal(loc=0, scale=1, size=npts_kde)
kde = gaussian_kde(n)

# grid for plotting
npts_sample = int(1e3)
x = np.linspace(-3, 3, npts_sample)

# evaluate pdfs
kde_pdf = kde.evaluate(x)
norm_pdf = norm.pdf(x)

# cdf and inv cdf are available directly from scipy
norm_cdf = norm.cdf(x)
norm_inv = norm.ppf(x)

# estimate cdf
cdf = tuple(ndtr(np.ravel(item - kde.dataset) / kde.factor).mean()
            for item in x)

# estimate inv cdf
inversefunction = interpolate.interp1d(cdf, x, kind='cubic', bounds_error=False)

fig, ax = plt.subplots(1, 3, figsize=(6, 3))
ax[0].plot(x, norm_pdf, c='k')
ax[0].plot(x, kde_pdf, c='r', ls='--')
ax[0].set_title('PDF')
ax[1].plot(x, norm_cdf, c='k')
ax[1].plot(x, cdf, c='r', ls='--')
ax[1].set_title('CDF')
ax[2].plot(x, norm_inv, c='k')
ax[2].plot(x, inversefunction(x), c='r', ls='--')
ax[2].set_title("Inverse CDF")
```


Question:
How to find the nearest neighbors for latitude and longitude point on python? Input:
```
point = (lat, long)
places = [(lat1, long1), (lat2, long2), ..., (latN, longN)]
count = L
```
Output: neighbors = subset of places close to the point. (len(neighbors)=L) Question: Can I use kd-tree for quick nearest-neighbors lookup for points with latitude and longitude? (For example, implementation in scipy) Is it necessary to transform the geographical coordinates (latitude and longitude) of the point in the coordinates x,y? Is it the best way to solve this?

Response:
I honestly don't know if using a kd-tree would work correctly, but my hunch says it would be inaccurate. I think you need to use something like greater circle distance to get accurate distances.
```py
from math import radians, cos, sin, asin, sqrt, degrees, atan2

def validate_point(p):
    lat, lon = p
    assert -90 <= lat <= 90, "bad latitude"
    assert -180 <= lon <= 180, "bad longitude"

# original formula from  http://www.movable-type.co.uk/scripts/latlong.html
def distance_haversine(p1, p2):
    """
    Calculate the great circle distance between two points
    on the earth (specified in decimal degrees)
    Haversine
    formula:
        a = sin²(Δφ/2) + cos φ1 ⋅ cos φ2 ⋅ sin²(Δλ/2)
                        _   ____
        c = 2 ⋅ atan2( √a, √(1−a) )
        d = R ⋅ c

    where   φ is latitude, λ is longitude, R is earth’s radius (mean radius = 6,371km);
            note that angles need to be in radians to pass to trig functions!
    """
    lat1, lon1 = p1
    lat2, lon2 = p2
    for p in [p1, p2]:
        validate_point(p)

    R = 6371 # km - earths's radius

    # convert decimal degrees to radians
    lat1, lon1, lat2, lon2 = map(radians, [lat1, lon1, lat2, lon2])

    # haversine formula
    dlon = lon2 - lon1
    dlat = lat2 - lat1

    a = sin(dlat/2)**2 + cos(lat1) * cos(lat2) * sin(dlon/2)**2
    c = 2 * asin(sqrt(a)) # 2 * atan2(sqrt(a), sqrt(1-a))
    d = R * c
    return d
```


Question:
multinomial pmf in python scipy/numpy Is there a built-in function in scipy/numpy for getting the PMF of a Multinomial? I'm not sure if binom generalizes in the correct way, e.g.
```
# Attempt to define multinomial with n = 10, p = [0.1, 0.1, 0.8]
rv = scipy.stats.binom(10, [0.1, 0.1, 0.8])
# Score the outcome 4, 4, 2
rv.pmf([4, 4, 2])
```
What is the correct way to do this? thanks.

Response:
There's no built-in function that I know of, and the binomial probabilities do not generalize (you need to normalise over a different set of possible outcomes, since the sum of all the counts must be n which won't be taken care of by independent binomials). However, it's fairly straightforward to implement yourself, for example:
```
import math

class Multinomial(object):
  def __init__(self, params):
    self._params = params

  def pmf(self, counts):
    if not(len(counts)==len(self._params)):
      raise ValueError("Dimensionality of count vector is incorrect")

    prob = 1.
    for i,c in enumerate(counts):
      prob *= self._params[i]**counts[i]

    return prob * math.exp(self._log_multinomial_coeff(counts))

  def log_pmf(self,counts):
    if not(len(counts)==len(self._params)):
      raise ValueError("Dimensionality of count vector is incorrect")

    prob = 0.
    for i,c in enumerate(counts):
      prob += counts[i]*math.log(self._params[i])

    return prob + self._log_multinomial_coeff(counts)

  def _log_multinomial_coeff(self, counts):
    return self._log_factorial(sum(counts)) - sum(self._log_factorial(c)
                                                    for c in counts)

  def _log_factorial(self, num):
    if not round(num)==num and num > 0:
      raise ValueError("Can only compute the factorial of positive ints")
    return sum(math.log(n) for n in range(1,num+1))

m = Multinomial([0.1, 0.1, 0.8])
print m.pmf([4,4,2])

>>2.016e-05
```
My implementation of the multinomial coefficient is somewhat naive, and works in log space to prevent overflow. Also be aware that n is superfluous as a parameter, since it's given by the sum of the counts (and the same parameter set works for any n). Furthermore, since this will quickly underflow for moderate n or large dimensionality, you're better working in log space (logPMF provided here too!)


Question:
concat pandas DataFrame along timeseries indexes I have two largish (snippets provided) pandas DateFrames with unequal dates as indexes that I wish to concat into one:
```
NAB.AX                                  CBA.AX
            Close    Volume                         Close    Volume
Date                                    Date
2009-06-05  36.51   4962900             2009-06-08  21.95         0
2009-06-04  36.79   5528800             2009-06-05  21.95   8917000
2009-06-03  36.80   5116500             2009-06-04  22.21  18723600
2009-06-02  36.33   5303700             2009-06-03  23.11  11643800
2009-06-01  36.16   5625500             2009-06-02  22.80  14249900
2009-05-29  35.14  13038600   --AND--   2009-06-01  22.52  11687200
2009-05-28  33.95   7917600             2009-05-29  22.02  22350700
2009-05-27  35.13   4701100             2009-05-28  21.63   9679800
2009-05-26  35.45   4572700             2009-05-27  21.74   9338200
2009-05-25  34.80   3652500             2009-05-26  21.64   8502900
```
Problem is, if I run this:
```
keys = ['CBA.AX','NAB.AX']
mv = pandas.concat([data['CBA.AX'][650:660],data['NAB.AX'][650:660]], axis=1, keys=stocks,)
```
the following DateFrame is produced:
```
CBA.AX          NAB.AX
                              Close  Volume   Close  Volume
Date
2200-08-16 04:24:21.460041     NaN     NaN     NaN     NaN
2203-05-13 04:24:21.460041     NaN     NaN     NaN     NaN
2206-02-06 04:24:21.460041     NaN     NaN     NaN     NaN
2208-11-02 04:24:21.460041     NaN     NaN     NaN     NaN
2211-07-30 04:24:21.460041     NaN     NaN     NaN     NaN
2219-10-16 04:24:21.460041     NaN     NaN     NaN     NaN
2222-07-12 04:24:21.460041     NaN     NaN     NaN     NaN
2225-04-07 04:24:21.460041     NaN     NaN     NaN     NaN
2228-01-02 04:24:21.460041     NaN     NaN     NaN     NaN
2230-09-28 04:24:21.460041     NaN     NaN     NaN     NaN
2238-12-15 04:24:21.460041     NaN     NaN     NaN     NaN
```
Does anybody have any idea why this might be the case? On another point: is there any python libraries around that pull data from yahoo and normalise it? Cheers. EDIT: For reference:
```
data = {
'CBA.AX': <class 'pandas.core.frame.DataFrame'>
    DatetimeIndex: 2313 entries, 2011-12-29 00:00:00 to 2003-01-01 00:00:00
    Data columns:
        Close     2313  non-null values
        Volume    2313  non-null values
    dtypes: float64(1), int64(1),

 'NAB.AX': <class 'pandas.core.frame.DataFrame'>
    DatetimeIndex: 2329 entries, 2011-12-29 00:00:00 to 2003-01-01 00:00:00
    Data columns:
        Close     2329  non-null values
        Volume    2329  non-null values
    dtypes: float64(1), int64(1)
}
```

Response:
It is possible to read the data with pandas and to concatenate it. First import the data
```

```python
import pandas.io.data as web
```

```python
nab = web.get_data_yahoo('NAB.AX', start='2009-05-25',
                                   end='2009-06-05')[['Close', 'Volume']]
```

```python
cba = web.get_data_yahoo('CBA.AX', start='2009-05-26',
                                   end='2009-06-08')[['Close', 'Volume']]
```

```python
nab
#Output
#Close    Volume
#Date
#2009-05-25  21.15   9685100
#2009-05-26  21.64   8541900
#2009-05-27  21.74   9042900
#2009-05-28  21.63   9701000
#2009-05-29  22.02  14665700
#2009-06-01  22.52   6782000
#2009-06-02  22.80  10473400
#2009-06-03  23.11   9931400
#2009-06-04  22.21  17869000
#2009-06-05  21.95   8214300
```

```python
cba
#Output
#Close    Volume
#Date
#2009-05-26  35.45   4529600
#2009-05-27  35.13   4521500
#2009-05-28  33.95   7945400
#2009-05-29  35.14  12548500
#2009-06-01  36.16   4509400
#2009-06-02  36.33   4304900
#2009-06-03  36.80   4845400
#2009-06-04  36.79   4592300
#2009-06-05  36.51   4417500
#2009-06-08  36.51         0
#```
#Than concatenate it:
#```
```

```python
keys = ['CBA.AX','NAB.AX']
```

```python
pd.concat([cba, nab], axis=1, keys=keys)
#Output
#CBA.AX            NAB.AX
#             Close    Volume   Close    Volume
#Date
#2009-05-25     NaN       NaN   21.15   9685100
#2009-05-26   35.45   4529600   21.64   8541900
#2009-05-27   35.13   4521500   21.74   9042900
#2009-05-28   33.95   7945400   21.63   9701000
#2009-05-29   35.14  12548500   22.02  14665700
#2009-06-01   36.16   4509400   22.52   6782000
#2009-06-02   36.33   4304900   22.80  10473400
#2009-06-03   36.80   4845400   23.11   9931400
#2009-06-04   36.79   4592300   22.21  17869000
#2009-06-05   36.51   4417500   21.95   8214300
#2009-06-08   36.51         0     NaN       NaN
#```
```


Question: Filling holes in objects that touch the border of an image I'm trying to fill holes in the below image. When I use SciPy's binary_fill_holes(), I am generally successful, with the exception of objects that touch the image's border. Are there any existing Python functions that can fill holes in objects that touch the border? I tried adding a white border around the image, but that just resulted in the entire image being filled.

Response:
This assumes that there is more background than other stuff. It basically does a connected component analysis on the image. Extract the largest component (assumed to be the background), and sets everything else to white.
```
import numpy as np
import matplotlib.pyplot as plt
import skimage.morphology, skimage.data

img = skimage.data.imread('j1ESv.png', 1)
labels = skimage.morphology.label(img)
labelCount = np.bincount(labels.ravel())
background = np.argmax(labelCount)
img[labels != background] = 255
plt.imshow(img, cmap=plt.cm.gray)
plt.show()
```


Question:
Scipy sparse invert or spsolve lead to UMFPACK_ERROR_OUT_OF_MEMORY I am trying to invert a large (150000,150000) sparse matrix as follows:
```
import scipy as sp
import scipy.sparse.linalg as splu

#Bs is a large sparse matrix with shape=(150000,150000)

#calculating the sparse inverse
iBs=splu.inv(Bs)
```
leads to the following error message:
```
Traceback (most recent call last):
    iBs=splu.inv(Bs)
  File "/usr/lib/python2.7/dist-packages/scipy/sparse/linalg/dsolve/linsolve.py", line 134, in spsolve
autoTranspose=True)
  File "/usr/lib/python2.7/dist-packages/scipy/sparse/linalg/dsolve/umfpack/umfpack.py", line 603, in linsolve
self.numeric(mtx)
  File "/usr/lib/python2.7/dist-packages/scipy/sparse/linalg/dsolve/umfpack/umfpack.py", line 450, in numeric
umfStatus[status]))
RuntimeError: <function umfpack_di_numeric at 0x7f2c76b1d320> failed with UMFPACK_ERROR_out_of_memory
```
I rejigged the program to simply solve a system of linear differential equations:
```
import numpy as np

N=Bs.shape[0]

I=np.ones(N)

M=splu.spsolve(Bs,I)
```
and I encounter the same error again I was using this code on a machine with 16 GB of RAM and then moved it onto a server with 32 GB of RAM, still to no avail. has anyone encountered this before?

Response:
First let me say that this question should be better asked on http://scicomp.stackexchange.com where there is a great community of experts in computational science and numerical linear algebra. Let's start from the basics: never invert a sparse matrix, it's completely meaningless. See this discussion on MATLAB central and particularly this comment by Tim Davis. Briefly: there are no algorithms for numerically inverting a matrix. Whenever you try to numerically compute the inverse of a NxN matrix, you solve in fact N linear systems with N rhs vectors corresponding to the columns of the identity matrix. In other words, when you compute
```
from scipy.sparse import eye
from scipy.sparse.linalg import (inv, spsolve)

N = Bs.shape[0]
iBs = inv(Bs)
iBs = spsolve(Bs, eye(N))
```
the last two statements (inv(eye) and spsolve(Bs, eye(N))) are equivalent. Please note that the identity matrix (eye(N)) is not a ones vector (np.ones(N)) as you question falsely assumes. The point here is that matrix inverses are seldom useful in numerical linear algebra: the solution of Ax = b is not computed as inv(A)*b, but by a specialised algorithm. Going to your specific problem, for big sparse system of equations there are no black-box solvers. You can chose the correct class of solvers only if you have a good understanding of the structure and properties of your matrix problem. The properties of your matrices in turn are a consequence of the problem you are trying to solve. E.g. when you discretise by the FEM a system of elliptic PDE, you end up with a symmetric positive sparse system of algebraic equations. Once you know the properties of your problem, you can choose the correct solving strategy. In your case, you are trying to use a generic direct solver, without reordering the equations. It is well known that this will generate fill-ins which destroy the sparsity of the iBs matrix in the first phase of the spsolve function (which should be a factorisation.) Please note that a full double precision 150000 x 150000 matrix requires about 167 GB of memory. There are a lot of techniques for reordering equations in order to reduce the fill-in during factorisation, but you don't provide enough info for giving you a sensible hint. I'm sorry, but you should considering reformulating your question on http://scicomp.stackexchange.com clearly stating which is the problem you are trying to solve, in order to give a clue on the matrix structure and properties.


Question:
assigning points to bins What is a good way to bin numerical values into a certain range? For example, suppose I have a list of values and I want to bin them into N bins by their range. Right now, I do something like this:
```
from scipy import *
num_bins = 3 # number of bins to use
values = # some array of integers...
min_val = min(values) - 1
max_val = max(values) + 1
my_bins = linspace(min_val, max_val, num_bins)
# assign point to my bins
for v in values:
  best_bin = min_index(abs(my_bins - v))
```
where min_index returns the index of the minimum value. The idea is that you can find the bin the point falls into by seeing what bin it has the smallest difference with. But I think this has weird edge cases. What I am looking for is a good representation of bins, ideally ones that are half closed half open (so that there is no way of assigning one point to two bins), i.e.
```
bin1 = [x1, x2)
bin2 = [x2, x3)
bin3 = [x3, x4)
etc...
```
what is a good way to do this in Python, using numpy/scipy? I am only concerned here with binning integer values. thanks very much for your help.

Response:
numpy.histogram() does exactly what you want. The function signature is:
```
numpy.histogram(a, bins=10, range=None, normed=False, weights=None, new=None)
```
We're mostly interested in a and bins. a is the input data that needs to be binned. bins can be a number of bins (your num_bins), or it can be a sequence of scalars, which denote bin edges (half open).
```
import numpy
values = numpy.arange(10, dtype=int)
bins = numpy.arange(-1, 11)
freq, bins = numpy.histogram(values, bins)
# freq is now [0 1 1 1 1 1 1 1 1 1 1]
# bins is unchanged
```
To quote the documentation: All but the last (righthand-most) bin is half-open. In other words, if bins is:
```
[1, 2, 3, 4]
```
then the first bin is [1, 2) (including 1, but excluding 2) and the second [2, 3). The last bin, however, is [3, 4], which includes 4. Edit: You want to know the index in your bins of each element. For this, you can use numpy.digitize(). If your bins are going to be integral, you can use numpy.bincount() as well.
```

```python
values = numpy.random.randint(0, 20, 10)
```

```python
values
#Output
#array([17, 14,  9,  7,  6,  9, 19,  4,  2, 19])
```

```python
bins = numpy.linspace(-1, 21, 23)
```

```python
bins
#Output
#array([ -1.,   0.,   1.,   2.,   3.,   4.,   5.,   6.,   7.,   8.,   9.,
#        10.,  11.,  12.,  13.,  14.,  15.,  16.,  17.,  18.,  19.,  20.,
#        21.])
```

```python
pos = numpy.digitize(values, bins)
```

```python
pos
#Output
#array([19, 16, 11,  9,  8, 11, 21,  6,  4, 21])
#```
#Since the interval is open on the upper limit, the indices are correct:
#```
```

```python
(bins[pos-1] == values).all()
#Output
#True
```

```python
import sys
```

```python
for n in range(len(values)):
    sys.stdout.write("%g <= %g < %g\n"
            %(bins[pos[n]-1], values[n], bins[pos[n]]))
#Output
#17 <= 17 < 18
#14 <= 14 < 15
#9 <= 9 < 10
#7 <= 7 < 8
#6 <= 6 < 7
#9 <= 9 < 10
#19 <= 19 < 20
#4 <= 4 < 5
#2 <= 2 < 3
#19 <= 19 < 20
#```
```


Question: KL-Divergence of two GMMs I have two GMMs that I used to fit two different sets of data in the same space, and I would like to calculate the KL-divergence between them. Currently I am using the GMMs defined in sklearn (http://scikit-learn.org/stable/modules/generated/sklearn.mixture.GMM.html) and the SciPy implementation of KL-divergence (http://docs.scipy.org/doc/scipy-dev/reference/generated/scipy.stats.entropy.html) How would I go about doing this? Do I want to just create tons of random points, get their probabilities on each of the two models (call them P and Q) and then use those probabilities as my input? Or is there some more canonical way to do this within the SciPy/SKLearn environment?

Response:
There's no closed form for the KL divergence between GMMs. You can easily do Monte Carlo, though. Recall that KL(p||q) = \int p(x) log(p(x) / q(x)) dx = E_p[ log(p(x) / q(x)). So:
```
def gmm_kl(gmm_p, gmm_q, n_samples=10**5):
    X = gmm_p.sample(n_samples)
    log_p_X, _ = gmm_p.score_samples(X)
    log_q_X, _ = gmm_q.score_samples(X)
    return log_p_X.mean() - log_q_X.mean()
```
(mean(log(p(x) / q(x))) = mean(log(p(x)) - log(q(x))) = mean(log(p(x))) - mean(log(q(x))) is somewhat cheaper computationally.) You don't want to use scipy.stats.entropy; that's for discrete distributions. If you want the symmetrized and smoothed Jensen-Shannon divergence KL(p||(p+q)/2) + KL(q||(p+q)/2) instead, it's pretty similar:
```
def gmm_js(gmm_p, gmm_q, n_samples=10**5):
    X = gmm_p.sample(n_samples)
    log_p_X, _ = gmm_p.score_samples(X)
    log_q_X, _ = gmm_q.score_samples(X)
    log_mix_X = np.logaddexp(log_p_X, log_q_X)

    Y = gmm_q.sample(n_samples)
    log_p_Y, _ = gmm_p.score_samples(Y)
    log_q_Y, _ = gmm_q.score_samples(Y)
    log_mix_Y = np.logaddexp(log_p_Y, log_q_Y)

    return (log_p_X.mean() - (log_mix_X.mean() - np.log(2))
            + log_q_Y.mean() - (log_mix_Y.mean() - np.log(2))) / 2
```
(log_mix_X/log_mix_Y are actually the log of twice the mixture densities; pulling that out of the mean operation saves some flops.)


Question: Rotate a 2D image around specified origin in Python I have a 2D image of 512x512 pixels that I would like to rotate with a certain angle at a certain origin (rotation center). All this time, I uses Scipy to rotate images with its rotate method. But, I got stumbled because the rotation always done around the center of the image. With 512x512 pixels the rotation center should be around point (x,y) 128,128. How can I rotate the image with a custom rotation center, let's say around (x,y) 20,128?

Response:
If OpenCV is not an option, you can do image rotation around a so called pivot point with NumPy (import numpy as np) and SciPy (from scipy import ndimage) the following way: Pad the image img such that the pivot point is in the image center and the image size is doubled:
```
padX = [img.shape[1] - pivot[0], pivot[0]]
padY = [img.shape[0] - pivot[1], pivot[1]]
imgP = np.pad(img, [padY, padX], 'constant')
```
(While the image shape is in row-column order, pivot is in X-Y or column-row order here. You might want to define it differently.) Rotate the image around its center (here the rotation angle is 45 degrees):
```
imgR = ndimage.rotate(imgP, 45, reshape=False)
```
Note that we disallow reshaping the image, since we'll crop the image ourselves. Crop the image such that the pivot point is at its original position. Therefore, we simply reverse the padding from step 1:
```
imgC = imgR[padY[0] : -padY[1], padX[0] : -padX[1]]
```
You can see the different steps in the following plot (original image, padded, rotated, cropped; 45 degrees around (100, 300)). Wrapping it up in a handy function yields:
```
def rotateImage(img, angle, pivot):
    padX = [img.shape[1] - pivot[0], pivot[0]]
    padY = [img.shape[0] - pivot[1], pivot[1]]
    imgP = np.pad(img, [padY, padX], 'constant')
    imgR = ndimage.rotate(imgP, angle, reshape=False)
    return imgR[padY[0] : -padY[1], padX[0] : -padX[1]]
```
Update For colored images you'd have to avoid adding more channels while padding (zero padding in 3rd dimension):
```
imgP = np.pad(img, [padY, padX, [0, 0]], 'constant')
```
Don't forget to use a 0 for both "before" and "after" padding. Otherwise you get a ValueError.


Question:
Difference between load of librosa and read of scipy.io.wavfile I have a question about the difference between the load function of librosa and the read function of scipy.io.wavfile.
```
from scipy.io import wavfile
import librosa

fs, data = wavfile.read(name)
data, fs = librosa.load(name)
```
The imported voice file is the same file. If you run the code above, the values of the data come out of the two functions differently. I want to know why the value of the data is different.

Response: From the docstring of librosa.core.load: Load an audio file as a floating point time series. Audio will be automatically resampled to the given rate (default sr=22050). To preserve the native sampling rate of the file, use sr=None. scipy.io.wavfile.read does not automatically resample the data, and the samples are not converted to floating point if they are integers in the file.


Question:
argrelextrema and flat extrema Function argrelextrema from scipy.signal does not detect flat extrema. Example:
```
import numpy as np
from scipy.signal import argrelextrema
data = np.array([ 0, 1, 2, 1, 0, 1, 3, 3, 1, 0 ])
argrelextrema(data, np.greater)
(array([2]),)
```
the first max (2) is detected, the second max (3, 3) is not detected. Any workaround for this behaviour? Thanks.

Response:
Short answer: Probably argrelextrema will not be flexible enough for your task. Consider writing your own function matching your needs. Longer answer: Are you bound to use argrelextrema? If yes, then you can play around with the comparator and the order arguments of argrelextrema (see the reference). For your easy example, it would be enough to chose np.greater_equal as comparator.
```

```python
data = np.array([ 0, 1, 2, 1, 0, 1, 3, 3, 1, 0 ])
```

```python
print(argrelextrema(data, np.greater_equal,order=1))
#Output
#(array([2, 6, 7]),)
#```
#Note however that in this way
#```
```

```python
data = np.array([ 0, 1, 2, 1, 0, 1, 3, 3, 4, 1, 0 ])
```

```python
print(argrelextrema(data, np.greater_equal,order=1))
#Output
#(array([2, 6, 8]),)
#```
#behaves differently that you would probably like, finding the first 3 and the 4 as maxima, since argrelextrema now sees everything as a maximum that is greater or equal to its two nearest neighbors. You can now use the order argument to decide to how many neighbors this comparison must hold - choosing order=2 would change my upper example to only find 4 as a maximum.
#```
```

```python
print(argrelextrema(data, np.greater_equal,order=2))
#Output
#(array([2, 8]),)
#```
#There is, however, a downside to this - let's change the data once more:
#```
```

```python
data = np.array([ 0, 1, 2, 1, 0, 1, 3, 3, 4, 1, 5 ])
```

```python
print(argrelextrema(data, np.greater_equal,order=2))
#Output
#(array([ 2, 10]),)
#```
#Adding another peak as a last value keeps you from finding your peak at 4, as argrelextrema is now seeing a second-neighbor that is greater than 4 (which can be useful for noisy data, but not necessarily the behavior expected in all cases). Using argrelextrema, you will always be limited to binary operations between a fixed number of neighbors. Note, however, that all argrelextrema is doing in your example above is to return n, if data[n] > data[n-1] and data[n] > data[n+1]. You could easily implement this yourself, and then refine the rules, for example by checking the second neighbor in case that the first neighbor has the same value. For the sake of completeness, there seems to be a more elaborate function in scipy.signal, find_peaks_cwt. I have however no experience using it and can therefore not give you more details about it.
```


Question: Seasonal adjustment in Python and Scipy I am looking to seasonally adjust monthly data, using Python. As you can see from these series: www.emconfidential.com, there is a high seasonal component to the data. I would like to adjust for this so that I can better guage if the series trend is rising or falling. Anybody know how to do this easily using scipy or other Python library?

Response:
Statsmodels can do this. They have a basic seasonal decomposition and also a wrapper to Census X13 adjustment. You could also use rpy2 to access some of R's excellent SA libraries. Here is statsmodels seasonal decomp:
```
import pandas as pd
import statsmodels.api as sm
import matplotlib.pyplot as plt
pd.options.display.mpl_style = 'default'
%matplotlib inline

dta = sm.datasets.co2.load_pandas().data.resample("M").fillna(method="ffill")

res = sm.tsa.seasonal_decompose(dta)

fig = res.plot()
fig.set_size_inches(10, 5)
plt.tight_layout()
```
http://statsmodels.sourceforge.net/0.6.0/release/version0.6.html


Question:
Errors to fit parameters of scipy.optimize I use the scipy.optimize.minimize ( https://docs.scipy.org/doc/scipy/reference/tutorial/optimize.html ) function with method='L-BFGS-B. An example of what it returns is here above:
```
fun: 32.372210618549758
 hess_inv: <6x6 LbfgsInvHessProduct with dtype=float64>
     jac: array([ -2.14583906e-04,   4.09272616e-04,  -2.55795385e-05,
         3.76587650e-05,   1.49213975e-04,  -8.38440428e-05])
  message: 'CONVERGENCE: REL_REDUCTION_OF_F_<=_FACTR*EPSMCH'
     nfev: 420
      nit: 51
   status: 0
  success: True
        x: array([ 0.75739412, -0.0927572 ,  0.11986434,  1.19911266,  0.27866406,
       -0.03825225])
```
The x value correctly contains the fitted parameters. How do I compute the errors associated to those parameters?

Response:
TL;DR: You can actually place an upper bound on how precisely the minimization routine has found the optimal values of your parameters. See the snippet at the end of this answer that shows how to do it directly, without resorting to calling additional minimization routines. The documentation for this method says The iteration stops when (f^k - f^{k+1})/max{|f^k|,|f^{k+1}|,1} <= ftol. Roughly speaking, the minimization stops when the value of the function f that you're minimizing is minimized to within ftol of the optimum. (This is a relative error if f is greater than 1, and absolute otherwise; for simplicity I'll assume it's an absolute error.) In more standard language, you'll probably think of your function f as a chi-squared value. So this roughly suggests that you would expect Of course, just the fact that you're applying a minimization routine like this assumes that your function is well behaved, in the sense that it's reasonably smooth and the optimum being found is well approximated near the optimum by a quadratic function of the parameters xi: where Δxi is the difference between the found value of parameter xi and its optimal value, and Hij is the Hessian matrix. A little (surprisingly nontrivial) linear algebra gets you to a pretty standard result for an estimate of the uncertainty in any quantity X that's a function of your parameters xi: which lets us write That's the most useful formula in general, but for the specific question here, we just have X = xi, so this simplifies to Finally, to be totally explicit, let's say you've stored the optimization result in a variable called res. The inverse Hessian is available as res.hess_inv, which is a function that takes a vector and returns the product of the inverse Hessian with that vector. So, for example, we can display the optimized parameters along with the uncertainty estimates with a snippet like this:
```
ftol = 2.220446049250313e-09
tmp_i = np.zeros(len(res.x))
for i in range(len(res.x)):
    tmp_i[i] = 1.0
    hess_inv_i = res.hess_inv(tmp_i)[i]
    uncertainty_i = np.sqrt(max(1, abs(res.fun)) * ftol * hess_inv_i)
    tmp_i[i] = 0.0
    print('x^{0} = {1:12.4e} ± {2:.1e}'.format(i, res.x[i], uncertainty_i))
```
Note that I've incorporated the max behavior from the documentation, assuming that f^k and f^{k+1} are basically just the same as the final output value, res.fun, which really ought to be a good approximation. Also, for small problems, you can just use np.diag(res.hess_inv.todense()) to get the full inverse and extract the diagonal all at once. But for large numbers of variables, I've found that to be a much slower option. Finally, I've added the default value of ftol, but if you change it in an argument to minimize, you would obviously need to change it here.


Question:
percentile rank in pandas in groups I can't quite figure out how to write function to accomplish a grouped percentile. I have all teams from years 1985-2012 in a data frame; the first 10 are shown below: it's currently sorted by year. I was looking to give a percentile for LgRnk grouped by Year. So for instance, 23 LgRank (worst team) for 1985 would be a 100 percentile and a 1 LgRank (best team) for 1985 would be a 1 percentile. 30 LgRank (worst team) for 2010 would be 100 percentile, etc. It needs to be grouped by year b/c of the differing number of LgRnks.
```
Team                WLPer   Year LgRnk   W  L
19  Sacramento Kings    0.378   1985    18  31  51
0   Atlanta Hawks       0.415   1985    17  34  48
17  Phoenix Suns        0.439   1985    16  36  46
4   Cleveland Cavaliers 0.439   1985    15  36  46
13  Milwaukee Bucks     0.720   1985    3   59  23
3   Chicago Bulls       0.463   1985    14  38  44
16  Philadelphia 76ers  0.707   1985    4   58  24
22  Washington Wizards  0.488   1985    13  40  42
20  San Antonio Spurs   0.500   1985    12  41  41
21  Utah Jazz           0.500   1985    11  41  41
```
I've tried creating a function using: scipy.stats.percentileofscore and I can't quite get it.

Response:
You can do an apply on the LgRnk column:
```
# just for me to normalize this, so my numbers will go from 0 to 1 in this example

```python
df['LgRnk'] = g.LgRnk.rank()
```

```python
g = df.groupby('Year')
```

```python
g.LgRnk.apply(lambda x: x / len(x))
#Output
#19    1.0
#0     0.9
#17    0.8
#4     0.7
#13    0.1
#3     0.6
#16    0.2
#22    0.5
#20    0.4
#21    0.3
#Name: 1985, dtype: float64
#```
#The Series groupby rank (which just applies Series.rank) take a pct argument to do just this:
#```
```

```python
g.LgRnk.rank(pct=True)
#Output
#19    1.0
#0     0.9
#17    0.8
#4     0.7
#13    0.1
#3     0.6
#16    0.2
#22    0.5
#20    0.4
#21    0.3
#Name: 1985, dtype: float64
#```
#and directly on the WLPer column (although this is slightly different due to draws):
#```
```

```python
g.WLPer.rank(pct=True, ascending=False)
#Output
#19    1.00
#0     0.90
#17    0.75
#4     0.75
#13    0.10
#3     0.60
#16    0.20
#22    0.50
#20    0.35
#21    0.35
#Name: 1985, dtype: float64
#```
#Note: I've changed the numbers on the first line, so you'll get different scores on your complete frame.
```