Question:
What is the difference between cholesky in numpy and scipy? I use Cholesky decomposition to sample random variables from multi-dimension Gaussian, and calculate the power spectrum of the random variables. The result I get from numpy.linalg.cholesky always has higher power in high frequencies than from scipy.linalg.cholesky. What are the differences between these two functions that could possibly cause this result? Which one is more numerically stable? Here is the code I use:
```
n = 2000

m = 10000

c0 = np.exp(-.05*np.arange(n))

C = linalg.toeplitz(c0)

Xn = np.dot(np.random.randn(m,n),np.linalg.cholesky(C))

Xs = np.dot(np.random.randn(m,n),linalg.cholesky(C))

Xnf = np.fft.fft(Xn)

Xsf = np.fft.fft(Xs)

Xnp = np.mean(Xnf*Xnf.conj(),axis=0)

Xsp = np.mean(Xsf*Xsf.conj(),axis=0)
```

Response:
scipy.linalg.cholesky is giving you the upper-triangular decomposition by default, whereas np.linalg.cholesky is giving you the lower-triangular version. From the docs for scipy.linalg.cholesky:
```
cholesky(a, lower=False, overwrite_a=False)
    Compute the Cholesky decomposition of a matrix.

    Returns the Cholesky decomposition, :math:`A = L L^*` or
    :math:`A = U^* U` of a Hermitian positive-definite matrix A.

    Parameters
    ----------
    a : ndarray, shape (M, M)
        Matrix to be decomposed
    lower : bool
        Whether to compute the upper or lower triangular Cholesky
        factorization.  Default is upper-triangular.
    overwrite_a : bool
        Whether to overwrite data in `a` (may improve performance).
```
For example:
```

```python
scipy.linalg.cholesky([[1,2], [1,9]])
#Output
#array([[ 1.        ,  2.        ],
#       [ 0.        ,  2.23606798]])
```

```python
scipy.linalg.cholesky([[1,2], [1,9]], lower=True)
#Output
#array([[ 1.        ,  0.        ],
#       [ 1.        ,  2.82842712]])
```

```python
np.linalg.cholesky([[1,2], [1,9]])
#Output
#array([[ 1.        ,  0.        ],
#       [ 1.        ,  2.82842712]])
#```
#If I modify your code to use the same random matrix both times and to use linalg.cholesky(C,lower=True) instead, then I get answers like:
#```
```

```python
Xnp
#Output
#array([ 79621.02629287+0.j,  78060.96077912+0.j,  77110.92428806+0.j, ...,
#        75526.55192199+0.j,  77110.92428806+0.j,  78060.96077912+0.j])
```

```python
Xsp
#Output
#array([ 79621.02629287+0.j,  78060.96077912+0.j,  77110.92428806+0.j, ...,
#        75526.55192199+0.j,  77110.92428806+0.j,  78060.96077912+0.j])
```

```python
np.allclose(Xnp, Xsp)
#Output
#True
#```
```


Question:
Do not print "optimization terminated successfully" scipy.optimize.fmin? Is there a way to "quietly" use scipy.optimize.fmin? That is, that it does not print e.g.
```
Optimization terminated successfully.
        Current function value: 0.000000
        Iterations: 13
        Function evaluations: 30
```
when running the code? This could be useful in e.g. loops that last a for a while.

Response: Setting the argument disp=False stops scipy.optimize.fmin from printing messages.


Question:
How to perform a chi-squared goodness of fit test using scientific libraries in Python? Let's assume I have some data I obtained empirically:
```
from scipy import stats
size = 10000
x = 10 * stats.expon.rvs(size=size) + 0.2 * np.random.uniform(size=size)
```
It is exponentially distributed (with some noise) and I want to verify this using a chi-squared goodness of fit (GoF) test. What is the simplest way of doing this using the standard scientific libraries in Python (e.g. scipy or statsmodels) with the least amount of manual steps and assumptions? I can fit a model with:
```
param = stats.expon.fit(x)
plt.hist(x, normed=True, color='white', hatch='/')
plt.plot(grid, distr.pdf(np.linspace(0, 100, 10000), *param))
```
It is very elegant to calculate the Kolmogorov-Smirnov test.
```

```python
stats.kstest(x, lambda x : stats.expon.cdf(x, *param))
#Output
#(0.0061000000000000004, 0.85077099515985011)
#```
#However, I can't find a good way of calculating the chi-squared test. There is a chi-squared GoF function in statsmodel, but it assumes a discrete distribution (and the exponential distribution is continuous). The official scipy.stats tutorial only covers a case for a custom distribution and probabilities are built by fiddling with many expressions (npoints, npointsh, nbound, normbound), so it's not quite clear to me how to do it for other distributions. The chisquare examples assume the expected values and DoF are already obtained. Also, I am not looking for a way to "manually" perform the test as was already discussed here, but would like to know how to apply one of the available library functions.
```

Response: An approximate solution for equal probability bins: Estimate the parameters of the distribution Use the inverse cdf, ppf if it's a scipy.stats.distribution, to get the binedges for a regular probability grid, e.g. distribution.ppf(np.linspace(0, 1, n_bins + 1), *args) Then, use np.histogram to count the number of observations in each bin then use chisquare test on the frequencies. An alternative would be to find the bin edges from the percentiles of the sorted data, and use the cdf to find the actual probabilities. This is only approximate, since the theory for the chisquare test assumes that the parameters are estimated by maximum likelihood on the binned data. And I'm not sure whether the selection of binedges based on the data affects the asymptotic distribution. I haven't looked into this into a long time. If an approximate solution is not good enough, then I would recommend that you ask the question on stats.stackexchange.


Question:
pcolormesh with missing values? I have 3 1-D ndarrays: x, y, z and the following code:
```
import numpy as np
import matplotlib.pyplot as plt
import scipy.interpolate as spinterp

## define data
npoints = 50
xreg = np.linspace(x.min(),x.max(),npoints)
yreg = np.linspace(y.min(),y.max(),npoints)
X,Y = np.meshgrid(xreg,yreg)
Z = spinterp.griddata(np.vstack((x,y)).T,z,(X,Y),
                      method='linear').reshape(X.shape)

## plot
plt.close()
ax = plt.axes()
col = ax.pcolormesh(X,Y,Z.T)
plt.draw()
```
My plot comes out blank, and I suspect it is because the method='linear' interpolation comes out with nans. I've tried converting to a masked array, but to no avail - plot is still blank. Can you tell me what I am doing wrong? Thanks.

Response:
Got it. This seems round-about, but this was the solution:
```
import numpy.ma as ma

Zm = ma.masked_where(np.isnan(Z),Z)
plt.pcolormesh(X,Y,Zm.T)
```
If the Z matrix contains nan's, it has to be a masked array for pcolormesh, which has to be created with ma.masked_where, or, alternatively,
```
Zm = ma.array(Z,mask=np.isnan(Z))
```


Question:
Procrustes Analysis with NumPy? Is there something like Matlab's procrustes function in NumPy/SciPy or related libraries? For reference. Procrustes analysis aims to align 2 sets of points (in other words, 2 shapes) to minimize square distance between them by removing scale, translation and rotation warp components. Example in Matlab:
```
X = [0 1; 2 3; 4 5; 6 7; 8 9];   % first shape
R = [1 2; 2 1];                  % rotation matrix
t = [3 5];                       % translation vector
Y = X * R + repmat(t, 5, 1);     % warped shape, no scale and no distortion
[d Z] = procrustes(X, Y);        % Z is Y aligned back to X
Z

Z =

  0.0000    1.0000
  2.0000    3.0000
  4.0000    5.0000
  6.0000    7.0000
  8.0000    9.0000
```
Same task in NumPy:
```
X = arange(10).reshape((5, 2))
R = array([[1, 2], [2, 1]])
t = array([3, 5])
Y = dot(X, R) + t
Z = ???
```
Note: I'm only interested in aligned shape, since square error (variable d in Matlab code) is easily computed from 2 shapes.

Response:
I'm not aware of any pre-existing implementation in Python, but it's easy to take a look at the MATLAB code using edit procrustes.m and port it to Numpy:
```
def procrustes(X, Y, scaling=True, reflection='best'):
    """
    A port of MATLAB's `procrustes` function to Numpy.

    Procrustes analysis determines a linear transformation (translation,
    reflection, orthogonal rotation and scaling) of the points in Y to best
    conform them to the points in matrix X, using the sum of squared errors
    as the goodness of fit criterion.

        d, Z, [tform] = procrustes(X, Y)

    Inputs:
    ------------
    X, Y
        matrices of target and input coordinates. they must have equal
        numbers of  points (rows), but Y may have fewer dimensions
        (columns) than X.

    scaling
        if False, the scaling component of the transformation is forced
        to 1

    reflection
        if 'best' (default), the transformation solution may or may not
        include a reflection component, depending on which fits the data
        best. setting reflection to True or False forces a solution with
        reflection or no reflection respectively.

    Outputs
    ------------
    d
        the residual sum of squared errors, normalized according to a
        measure of the scale of X, ((X - X.mean(0))**2).sum()

    Z
        the matrix of transformed Y-values

    tform
        a dict specifying the rotation, translation and scaling that
        maps X --> Y

    """

    n,m = X.shape
    ny,my = Y.shape

    muX = X.mean(0)
    muY = Y.mean(0)

    X0 = X - muX
    Y0 = Y - muY

    ssX = (X0**2.).sum()
    ssY = (Y0**2.).sum()

    # centred Frobenius norm
    normX = np.sqrt(ssX)
    normY = np.sqrt(ssY)

    # scale to equal (unit) norm
    X0 /= normX
    Y0 /= normY

    if my < m:
        Y0 = np.concatenate((Y0, np.zeros(n, m-my)),0)

    # optimum rotation matrix of Y
    A = np.dot(X0.T, Y0)
    U,s,Vt = np.linalg.svd(A,full_matrices=False)
    V = Vt.T
    T = np.dot(V, U.T)

    if reflection != 'best':

        # does the current solution use a reflection?
        have_reflection = np.linalg.det(T) < 0

        # if that's not what was specified, force another reflection
        if reflection != have_reflection:
            V[:,-1] *= -1
            s[-1] *= -1
            T = np.dot(V, U.T)

    traceTA = s.sum()

    if scaling:

        # optimum scaling of Y
        b = traceTA * normX / normY

        # standarised distance between X and b*Y*T + c
        d = 1 - traceTA**2

        # transformed coords
        Z = normX*traceTA*np.dot(Y0, T) + muX

    else:
        b = 1
        d = 1 + ssY/ssX - 2 * traceTA * normY / normX
        Z = normY*np.dot(Y0, T) + muX

    # transformation matrix
    if my < m:
        T = T[:my,:]
    c = muX - b*np.dot(muY, T)

    #transformation values
    tform = {'rotation':T, 'scale':b, 'translation':c}

    return d, Z, tform
```


Question:
scipy.io.wavfile gives "WavFileWarning: chunk not understood" error I'm trying to read a .wav file using scipy. I do this:
```
from scipy.io import wavfile

filename = "myWavFile.wav"
print "Processing " + filename

samples = wavfile.read(filename)
```
And I get this ugly error:
```
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/io/wavfile.py:121: WavFileWarning: chunk not understood
  warnings.warn("chunk not understood", WavFileWarning)
Traceback (most recent call last):
  File "fingerFooler.py", line 15, in <module>
    samples = wavfile.read(filename)
  File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/io/wavfile.py", line 127, in read
    size = struct.unpack(fmt, data)[0]
struct.error: unpack requires a string argument of length 4
```
I'm using Python 2.6.6, numpy 1.6.2, and scipy 0.11.0 Here's a wav file that causes the problem. Any thoughts? What's wrong here?

Response: The files is no longer available (not surprising after 9 months!), but for future reference the most likely cause is that it had extra metadata which scipy can't parse. In my case, it was default metadata (copyright, track name etc) which was added by Audacity- you can open the file in Audacity and use File ... Open Metadata Editor to see it. Then use the 'Clear' button to strip it, and try again. The current version of scipy supports the following RIFF chunks - 'fmt', 'fact', 'data' and 'LIST'. The Wikipedia page on RIFF has a bit more detail on how a WAV file is structured, for example yours might have included an unsupported-but-popular INFO chunk


Question:
What value do I use in a slicing range to include the last value in a numpy array? Imagine some numpy array, e.g. x = np.linspace(1,10). x[i:j] gives me a view into x for the range [i,j). I love that I can also do x[i:-k] which excludes the last k elements. However, in order to include the last element I need to do x[i:]. My question is this: How do I combine these two notations if I for instance need to loop over k. Say that I want to do this:
```
l = list()
for k in [5,4,3,2,1]:
    l.append(x[:-k])
l.append(x[:])
```
What annoys me is that last line. In this simple example of course it doesn't do much of a difference, but sometimes this becomes much more annoying. What I miss is something more DRY-like. The following snippet course does NOT yield the desired result, but represents the style of code I seek:
```
l = list()
for k in [5,4,3,2,1,0]:
    l.append(x[:-k])
```

Response:
It's a bit of a pain, but since -0 is the same as 0, there is no easy solution. One way to do it would be:
```
l = list()
for k in [5,4,3,2,1,0]:
    l.append(x[:-k or None])
```
This is because when k is 0, -k or None is None, and x[:None] will do what you want. For other values of k, -k or None will be -k. I am not sure if I like it myself though.


Question:
Python: two-curve gaussian fitting with non-linear least-squares My knowledge of maths is limited which is why I am probably stuck. I have a spectra to which I am trying to fit two Gaussian peaks. I can fit to the largest peak, but I cannot fit to the smallest peak. I understand that I need to sum the Gaussian function for the two peaks but I do not know where I have gone wrong. An image of my current output is shown: The blue line is my data and the green line is my current fit. There is a shoulder to the left of the main peak in my data which I am currently trying to fit, using the following code:
```
import matplotlib.pyplot as pt
import numpy as np
from scipy.optimize import leastsq
from pylab import *

time = []
counts = []

for i in open('/some/folder/to/file.txt', 'r'):
    segs = i.split()
    time.append(float(segs[0]))
    counts.append(segs[1])

time_array = arange(len(time), dtype=float)
counts_array = arange(len(counts))
time_array[0:] = time
counts_array[0:] = counts

def model(time_array0, coeffs0):
    a = coeffs0[0] + coeffs0[1] * np.exp( - ((time_array0-coeffs0[2])/coeffs0[3])**2 )
    b = coeffs0[4] + coeffs0[5] * np.exp( - ((time_array0-coeffs0[6])/coeffs0[7])**2 )
    c = a+b
    return c

def residuals(coeffs, counts_array, time_array):
    return counts_array - model(time_array, coeffs)

# 0 = baseline, 1 = amplitude, 2 = centre, 3 = width
peak1 = np.array([0,6337,16.2,4.47,0,2300,13.5,2], dtype=float)
#peak2 = np.array([0,2300,13.5,2], dtype=float)

x, flag = leastsq(residuals, peak1, args=(counts_array, time_array))
#z, flag = leastsq(residuals, peak2, args=(counts_array, time_array))

plt.plot(time_array, counts_array)
plt.plot(time_array, model(time_array, x), color = 'g')
#plt.plot(time_array, model(time_array, z), color = 'r')
plt.show()
```

Response:
This code worked for me providing that you are only fitting a function that is a combination of two Gaussian distributions. I just made a residuals function that adds two Gaussian functions and then subtracts them from the real data. The parameters (p) that I passed to Numpy's least squares function include: the mean of the first Gaussian function (m), the difference in the mean from the first and second Gaussian functions (dm, i.e. the horizontal shift), the standard deviation of the first (sd1), and the standard deviation of the second (sd2).
```
import numpy as np
from scipy.optimize import leastsq
import matplotlib.pyplot as plt

######################################
# Setting up test data
def norm(x, mean, sd):
  norm = []
  for i in range(x.size):
    norm += [1.0/(sd*np.sqrt(2*np.pi))*np.exp(-(x[i] - mean)**2/(2*sd**2))]
  return np.array(norm)

mean1, mean2 = 0, -2
std1, std2 = 0.5, 1

x = np.linspace(-20, 20, 500)
y_real = norm(x, mean1, std1) + norm(x, mean2, std2)

######################################
# Solving
m, dm, sd1, sd2 = [5, 10, 1, 1]
p = [m, dm, sd1, sd2] # Initial guesses for leastsq
y_init = norm(x, m, sd1) + norm(x, m + dm, sd2) # For final comparison plot

def res(p, y, x):
  m, dm, sd1, sd2 = p
  m1 = m
  m2 = m1 + dm
  y_fit = norm(x, m1, sd1) + norm(x, m2, sd2)
  err = y - y_fit
  return err

plsq = leastsq(res, p, args = (y_real, x))

y_est = norm(x, plsq[0][0], plsq[0][2]) + norm(x, plsq[0][0] + plsq[0][1], plsq[0][3])

plt.plot(x, y_real, label='Real Data')
plt.plot(x, y_init, 'r.', label='Starting Guess')
plt.plot(x, y_est, 'g.', label='Fitted')
plt.legend()
plt.show()
```


Question:
Why does scipy.norm.pdf sometimes give PDF > 1? How to correct it? Given mean and variance of a Gaussian (normal) random variable, I would like to compute its probability density function (PDF). I referred this post: Calculate probability in normal distribution given mean, std in Python, Also the scipy docs: scipy.stats.norm But when I plot a PDF of a curve, the probability exceeds 1! Refer to this minimum working example:
```python
import numpy as np
import scipy.stats as stats

x = np.linspace(0.3, 1.75, 1000)
plt.plot(x, stats.norm.pdf(x, 1.075, 0.2))
plt.show()
```
This is what I get: How is it even possible to have 200% probability to get the mean, 1.075? Am I misinterpreting anything here? Is there any way to correct this?

Response: It's not a bug. It's not an incorrect result either. Probability density function's value at some specific point does not give you probability; it is a measure of how dense the distribution is around that value. For continuous random variables, the probability at a given point is equal to zero. Instead of p(X = x), we calculate probabilities between 2 points p(x1 < X < x2) and it is equal to the area below that probability density function. Probability density function's value can very well be above 1. It can even approach to infinity.


Question:
How can I use scipy.ndimage.interpolation.affine_transform to rotate an image about its centre? I am perplexed by the API to scipy.ndimage.interpolation.affine_transform. And judging by this issue I'm not the only one. I'm actually wanting to do more interesting things with affine_transform than just rotating an image, but a rotation would do for starters. (And yes I'm well aware of scipy.ndimage.interpolation.rotate, but figuring out how to drive affine_transform is what interests me here). When I want to do this sort of thing in systems like OpenGL, I'm think in terms of computing the transform which applies a 2x2 rotation matrix R about a centre c, and therefore thinking of points p being transformed (p-c)R+c = pR+c-cR, which gives a c-cR term to be used as the translation component of a transform. However, according to the issue above, scipy's affine_transform does "offset first" so we actually need to compute an offset s such that (p-c)R+c=(p+s)R which with a bit of rearrangement gives s=(c-cR)R' where R' is the inverse of R. If I plug this into an ipython notebook (pylab mode; code below maybe needs some additional imports):
```
img=scipy.misc.lena()
#imshow(img,cmap=cm.gray);show()
centre=0.5*array(img.shape)
a=15.0*pi/180.0
rot=array([[cos(a),sin(a)],[-sin(a),cos(a)]])
offset=(centre-centre.dot(rot)).dot(linalg.inv(rot))
rotimg=scipy.ndimage.interpolation.affine_transform(
    img,rot,order=2,offset=offset,cval=0.0,output=float32
)
imshow(rotimg,cmap=cm.gray);show()
```
I get which unfortunately isn't rotated about the centre. So what's the trick I'm missing here?

Response:
Once treddy's answer got me a working baseline, I managed to get a better working model of affine_transform. It's not actually as odd as the issue linked in the original question hints. Basically, each point (coordinate) p in the output image is transformed to pT+s where T and s are the matrix and offset passed to the function. So if we want point c_out in the output to be mapped to and sampled from c_in from the input image, with rotation R and (possibly anisotropic) scaling S we need pT+s = (p-c_out)RS+c_in which can be rearranged to yield s = (c_int-c_out)T (with T=RS). For some reason I then need to pass transform.T to affine_transform but I'm not going to worry about that too much; probably something to do with row-coordinates with transforms on the right (assumed above) vs column-coordinates with transforms on the left. So here's a simple test rotating a centred image:
```
src=scipy.misc.lena()
c_in=0.5*array(src.shape)
c_out=array((256.0,256.0))
for i in xrange(0,7):
    a=i*15.0*pi/180.0
    transform=array([[cos(a),-sin(a)],[sin(a),cos(a)]])
    offset=c_in-c_out.dot(transform)
    dst=scipy.ndimage.interpolation.affine_transform(
        src,transform.T,order=2,offset=offset,output_shape=(512,512),cval=0.0,output=float32
    )
    subplot(1,7,i+1);axis('off');imshow(dst,cmap=cm.gray)
show()
```
Here's it modified for different image sizes
```
src=scipy.misc.lena()[::2,::2]
c_in=0.5*array(src.shape)
c_out=array((256.0,256.0))
for i in xrange(0,7):
    a=i*15.0*pi/180.0
    transform=array([[cos(a),-sin(a)],[sin(a),cos(a)]])
    offset=c_in-c_out.dot(transform)
    dst=scipy.ndimage.interpolation.affine_transform(
        src,transform.T,order=2,offset=offset,output_shape=(512,512),cval=0.0,output=float32
    )
    subplot(1,7,i+1);axis('off');imshow(dst,cmap=cm.gray)
show()
```
And here's a version with anisotropic scaling to compensate for the anisotropic resolution of the source image.
```
src=scipy.misc.lena()[::2,::4]
c_in=0.5*array(src.shape)
c_out=array((256.0,256.0))
for i in xrange(0,7):
    a=i*15.0*pi/180.0
    transform=array([[cos(a),-sin(a)],[sin(a),cos(a)]]).dot(diag(([0.5,0.25])))
    offset=c_in-c_out.dot(transform)
    dst=scipy.ndimage.interpolation.affine_transform(
        src,transform.T,order=2,offset=offset,output_shape=(512,512),cval=0.0,output=float32
    )
    subplot(1,7,i+1);axis('off');imshow(dst,cmap=cm.gray)
show()
```


Question:
Find minimum distance from point to complicated curve I have a complicated curve defined as a set of points in a table like so (the full table is here):
```
#  x   y
1.0577  12.0914
1.0501  11.9946
1.0465  11.9338
...
```
If I plot this table with the commands:
```
plt.plot(x_data, y_data, c='b',lw=1.)
plt.scatter(x_data, y_data, marker='o', color='k', s=10, lw=0.2)
```
I get the following: where I've added the red points and segments manually. What I need is a way to calculate those segments for each of those points, that is: a way to find the minimum distance from a given point in this 2D space to the interpolated curve. I can't use the distance to the data points themselves (the black dots that generate the blue curve) since they are not located at equal intervals, sometimes they are close and sometimes they are far apart and this deeply affects my results further down the line. Since this is not a well behaved curve I'm not really sure what I could do. I've tried interpolating it with a UnivariateSpline but it returns a very poor fit:
```
# Sort data according to x.
temp_data = zip(x_data, y_data)
temp_data.sort()
# Unpack sorted data.
x_sorted, y_sorted = zip(*temp_data)

# Generate univariate spline.
s = UnivariateSpline(x_sorted, y_sorted, k=5)
xspl = np.linspace(0.8, 1.1, 100)
yspl = s(xspl)

# Plot.
plt.scatter(xspl, yspl, marker='o', color='r', s=10, lw=0.2)
```
I also tried increasing the number of interpolating points but got a mess:
```
# Sort data according to x.
temp_data = zip(x_data, y_data)
temp_data.sort()
# Unpack sorted data.
x_sorted, y_sorted = zip(*temp_data)

t = np.linspace(0, 1, len(x_sorted))
t2 = np.linspace(0, 1, 100)
# One-dimensional linear interpolation.
x2 = np.interp(t2, t, x_sorted)
y2 = np.interp(t2, t, y_sorted)
plt.scatter(x2, y2, marker='o', color='r', s=10, lw=0.2)
```
Any ideas/pointers will be greatly appreciated.

Response:
If you're open to using a library for this, have a look at shapely: https://github.com/Toblerity/Shapely As a quick example (points.txt contains the data you linked to in your question):
```
import shapely.geometry as geom
import numpy as np

coords = np.loadtxt('points.txt')

line = geom.LineString(coords)
point = geom.Point(0.8, 10.5)

# Note that "line.distance(point)" would be identical
print(point.distance(line))
```
As an interactive example (this also draws the line segments you wanted):
```
import numpy as np
import shapely.geometry as geom
import matplotlib.pyplot as plt

class NearestPoint(object):
    def __init__(self, line, ax):
        self.line = line
        self.ax = ax
        ax.figure.canvas.mpl_connect('button_press_event', self)

    def __call__(self, event):
        x, y = event.xdata, event.ydata
        point = geom.Point(x, y)
        distance = self.line.distance(point)
        self.draw_segment(point)
        print 'Distance to line:', distance

    def draw_segment(self, point):
        point_on_line = line.interpolate(line.project(point))
        self.ax.plot([point.x, point_on_line.x], [point.y, point_on_line.y],
                     color='red', marker='o', scalex=False, scaley=False)
        fig.canvas.draw()

if __name__ == '__main__':
    coords = np.loadtxt('points.txt')

    line = geom.LineString(coords)

    fig, ax = plt.subplots()
    ax.plot(*coords.T)
    ax.axis('equal')
    NearestPoint(line, ax)
    plt.show()
```
Note that I've added ax.axis('equal'). shapely operates in the coordinate system that the data is in. Without the equal axis plot, the view will be distorted, and while shapely will still find the nearest point, it won't look quite right in the display:


Question: How to use scipy.optimize.minimize function when you want to compute gradient along with the objective function? scipy.optimize.minimze takes obj and jac functions as input. and I believe it will call them separately as and when needed. But more often than not we come across objective functions whose gradient computation shares a lot of computations from the objective function. So ideally I would like to compute the obj and grad simultaneously. But this doesn't seem to be the case with this library? What is the way to deal with it if one still wants to use scipy.optimize.minimze if at all there is?

Response:
You totally can. Just use jac=True:
```

```python
import numpy as np
```

```python
from scipy.optimize import minimize
```

```python
def f_and_grad(x):
   ...:     return x**2, 2*x
   ...:
```

```python
minimize(f_and_grad, [1], jac=True)
#Output
#fun: 1.8367099231598242e-40
# hess_inv: array([[ 0.5]])
#      jac: array([  2.71050543e-20])
#  message: 'Optimization terminated successfully.'
#     nfev: 4
#      nit: 2
#     njev: 4
#   status: 0
#  success: True
#        x: array([  1.35525272e-20])
#```
#It's actually documented: jac : bool or callable, optional Jacobian (gradient) of objective function. Only for CG, BFGS, Newton-CG, L-BFGS-B, TNC, SLSQP, dogleg, trust-ncg. If jac is a Boolean and is True, fun is assumed to return the gradient along with the objective function. If False, the gradient will be estimated numerically. jac can also be a callable returning the gradient of the objective. In this case, it must accept the same arguments as fun. (emphasis mine)
```


Question:
How to apply linear regression to every pixel in a large multi-dimensional array containing NaNs? I have a 1D array of independent variable values (x_array) that match the timesteps in a 3D numpy array of spatial data with multiple time-steps (y_array). My actual data is much larger: 300+ timesteps and up to 3000 * 3000 pixels:
```
import numpy as np
from scipy.stats import linregress

# Independent variable: four time-steps of 1-dimensional data
x_array = np.array([0.5, 0.2, 0.4, 0.4])

# Dependent variable: four time-steps of 3x3 spatial data
y_array = np.array([[[-0.2,   -0.2,   -0.3],
                     [-0.3,   -0.2,   -0.3],
                     [-0.3,   -0.4,   -0.4]],

                    [[-0.2,   -0.2,   -0.4],
                     [-0.3,   np.nan, -0.3],
                     [-0.3,   -0.3,   -0.4]],

                    [[np.nan, np.nan, -0.3],
                     [-0.2,   -0.3,   -0.7],
                     [-0.3,   -0.3,   -0.3]],

                    [[-0.1,   -0.3,   np.nan],
                     [-0.2,   -0.3,   np.nan],
                     [-0.1,   np.nan, np.nan]]])
```
I want to compute a per-pixel linear regression and obtain R-squared, P-values, intercepts and slopes for each xy pixel in y_array, with values for each timestep in x_array as my independent variable. I can reshape to get the data in a form to input it into np.polyfit which is vectorised and fast:
```
# Reshape so rows = number of time-steps and columns = pixels:
y_array_reshaped = y_array.reshape(len(y_array), -1)

# Do a first-degree polyfit
np.polyfit(x_array, y_array_reshaped, 1)
```
However, this ignores pixels that contain any NaN values (np.polyfit does not support NaN values), and does not calculate the statistics I require (R-squared, P-values, intercepts and slopes). The answer here uses scipy.stats import linregress which does calculate the statistics I need, and suggests avoiding NaN issues by masking out these NaN values. However, this example is for two 1D arrays, and I can't work out how to apply a similar masking approach to my case where each column in y_array_reshaped will have a different set of NaN values. My question: How can I calculate regression statistics for each pixel in a large multidimensional array (300 x 3000 x 3000) containing many NaN values in a reasonably fast, vectorised way? Desired result: A 3 x 3 array of regression statistic values (e.g. R-squared) for each pixel in y_array, even if that pixel contains NaN values at some point in the time series

Response:
This blog post mentioned in the comments above contains an incredibly fast vectorized function for cross-correlation, covariance, and regression for multi-dimensional data in Python. It produces all of the regression outputs I need, and does so in milliseconds as it relies entirely on simple vectorised array operations in xarray. May 2024 update: I've updated the function to ensure it function correctly accounts for different numbers of NaN values in each pixel, and to correctly calculate P-values for different alternative hypotheses (thanks big-zhao!):
```
def xr_regression(x, y, lag_x=0, lag_y=0, dim="time", alternative="two-sided"):
    """
    Takes two xr.Datarrays of any dimensions (input data could be a 1D
    time series, or for example, have three dimensions e.g. time, lat,
    lon), and returns covariance, correlation, coefficient of
    determination, regression slope, intercept, p-value and standard
    error, and number of valid observations (n) between the two datasets
    along their aligned first dimension.

    Datasets can be provided in any order, but note that the regression
    slope and intercept will be calculated for y with respect to x.

    Inspired by:
    https://hrishichandanpurkar.blogspot.com/2017/09/vectorized-functions-for-correlation.html

    Parameters
    ----------
    x, y : xarray DataArray
        Two xarray DataArrays with any number of dimensions, both
        sharing the same first dimension
    lag_x, lag_y : int, optional
        Optional integers giving lag values to assign to either of the
        data, with lagx shifting x, and lagy shifting y with the
        specified lag amount.
    dim : str, optional
        An optional string giving the name of the dimension on which to
        align (and optionally lag) datasets. The default is 'time'.
    alternative : string, optional
        Defines the alternative hypothesis. Default is 'two-sided'.
        The following options are available:

        * 'two-sided': slope of the regression line is nonzero
        * 'less': slope of the regression line is less than zero
        * 'greater':  slope of the regression line is greater than zero

    Returns
    -------
    regression_ds : xarray.Dataset
        A dataset comparing the two input datasets along their aligned
        dimension, containing variables including covariance, correlation,
        coefficient of determination, regression slope, intercept,
        p-value and standard error, and number of valid observations (n).

    """

    # Shift x and y data if lags are specified
    if lag_x != 0:
        # If x lags y by 1, x must be shifted 1 step backwards. But as
        # the 'zero-th' value is nonexistant, xarray assigns it as
        # invalid (nan). Hence it needs to be dropped
        x = x.shift(**{dim: -lag_x}).dropna(dim=dim)

        # Next re-align the two datasets so that y adjusts to the
        # changed coordinates of x
        x, y = xr.align(x, y)

    if lag_y != 0:
        y = y.shift(**{dim: -lag_y}).dropna(dim=dim)

    # Ensure that the data are properly aligned to each other.
    x, y = xr.align(x, y)

    # Compute data length, mean and standard deviation along dim
    n = y.notnull().sum(dim=dim)
    xmean = x.mean(dim=dim)
    ymean = y.mean(dim=dim)
    xstd = x.std(dim=dim)
    ystd = y.std(dim=dim)

    # Compute covariance, correlation and coefficient of determination
    cov = ((x - xmean) * (y - ymean)).sum(dim=dim) / (n)
    cor = cov / (xstd * ystd)
    r2 = cor**2

    # Compute regression slope and intercept
    slope = cov / (xstd**2)
    intercept = ymean - xmean * slope

    # Compute t-statistics and standard error
    tstats = cor * np.sqrt(n - 2) / np.sqrt(1 - cor**2)
    stderr = slope / tstats

    # Calculate p-values for different alternative hypotheses.
    if alternative == "two-sided":
        pval = t.sf(np.abs(tstats), n - 2) * 2
    elif alternative == "greater":
        pval = t.sf(tstats, n - 2)
    elif alternative == "less":
        pval = t.cdf(np.abs(tstats), n - 2)

    # Wrap p-values into an xr.DataArray
    pval = xr.DataArray(pval, dims=cor.dims, coords=cor.coords)

    # Combine into single dataset
    regression_ds = xr.merge(
        [
            cov.rename("cov").astype(np.float32),
            cor.rename("cor").astype(np.float32),
            r2.rename("r2").astype(np.float32),
            slope.rename("slope").astype(np.float32),
            intercept.rename("intercept").astype(np.float32),
            pval.rename("pvalue").astype(np.float32),
            stderr.rename("stderr").astype(np.float32),
            n.rename("n").astype(np.int16),
        ]
    )

    return regression_ds
```


Question: Numpy histogram of large arrays I have a bunch of csv datasets, about 10Gb in size each. I'd like to generate histograms from their columns. But it seems like the only way to do this in numpy is to first load the entire column into a numpy array and then call numpy.histogram on that array. This consumes an unnecessary amount of memory. Does numpy support online binning? I'm hoping for something that iterates over my csv line by line and bins values as it reads them. This way at most one line is in memory at any one time. Wouldn't be hard to roll my own, but wondering if someone already invented this wheel.

Response:
As you said, it's not that hard to roll your own. You'll need to set up the bins yourself and reuse them as you iterate over the file. The following ought to be a decent starting point:
```
import numpy as np
datamin = -5
datamax = 5
numbins = 20
mybins = np.linspace(datamin, datamax, numbins)
myhist = np.zeros(numbins-1, dtype='int32')
for i in range(100):
    d = np.random.randn(1000,1)
    htemp, jnk = np.histogram(d, mybins)
    myhist += htemp
```
I'm guessing performance will be an issue with such large files, and the overhead of calling histogram on each line might be too slow. @doug's suggestion of a generator seems like a good way to address that problem.


Question:
How to get flat clustering corresponding to color clusters in the dendrogram created by scipy Using the code posted here, I created a nice hierarchical clustering: Let's say the the dendrogram on the left was created by doing something like
```
Y = sch.linkage(D, method='average') # D is a distance matrix
cutoff = 0.5*max(Y[:,2])
Z = sch.dendrogram(Y, orientation='right', color_threshold=cutoff)
```
Now how do I get the indices of the members of each of the colored clusters? To simplify this situation, ignore the clustering on the top, and focus only on the dendrogram on the left of the matrix. This information should be stored in the dendrogram Z stored variable. There is a function that should do just what I want called fcluster (see documentation here). However I cannot see where I can give fcluster the same cutoff as I specified in the creation of the dendrogram. It seems that the threshold variable in fcluster, t has to be in terms of various obscure measurements (inconsistent, distance, maxclust, monocrit). Any ideas?

Response:
I think you're on the right track. Let's try this:
```
import scipy
import scipy.cluster.hierarchy as sch
X = scipy.randn(100, 2)     # 100 2-dimensional observations
d = sch.distance.pdist(X)   # vector of (100 choose 2) pairwise distances
L = sch.linkage(d, method='complete')
ind = sch.fcluster(L, 0.5*d.max(), 'distance')
```
ind will give you cluster indices for each of the 100 input observations. ind depends on what method you used in linkage. Try method=single, complete, and average. Then note how ind differs. Example:
```

```python
L = sch.linkage(d, method='complete')
```

```python
sch.fcluster(L, 0.5*d.max(), 'distance')
#Output
#array([5, 4, 2, 2, 5, 5, 1, 5, 5, 2, 5, 2, 5, 5, 1, 1, 5, 5, 4, 2, 5, 2, 5,
#       2, 5, 3, 5, 3, 5, 5, 5, 5, 5, 5, 5, 2, 2, 5, 5, 4, 1, 4, 5, 2, 1, 4,
#       2, 4, 2, 2, 5, 5, 5, 2, 5, 5, 3, 5, 5, 4, 5, 4, 5, 3, 5, 3, 5, 5, 5,
#       2, 3, 5, 5, 4, 5, 5, 2, 2, 5, 2, 2, 4, 1, 2, 1, 5, 2, 5, 5, 5, 1, 5,
#       4, 2, 4, 5, 2, 4, 4, 2])
```

```python
L = sch.linkage(d, method='single')
```

```python
sch.fcluster(L, 0.5*d.max(), 'distance')
#Output
#array([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
#       1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
#       1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
#       1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
#       1, 1, 1, 1, 1, 1, 1, 1])
#```
#scipy.cluster.hierarchy sure is confusing. In your link, I don't even recognize my own code!
```


Question: Generating low discrepancy quasi-random sequences in python/numpy/scipy? There is already a question on this but the answer contains a broken link, and being over two years old, I'm hoping there's a better solution now :) Low discrepancy quasi-random sequences, e.g. Sobol sequences, fill a space more uniformly than uniformly random sequences. Is there a good/easy way to generate them in python?

Response: I think the best alternative for Low Discrepancy sequences in Python is Sensitivity Analysis Library (SALib): https://github.com/SALib/SALib I think this is an active project and you can contact the author to check if the functionalities you need are already implemented. If that doesn't solve your problem, Corrado Chisari ported a SOBOL version made in Matlab (by John Burkardt) to Python, you can access it here: http://people.sc.fsu.edu/~jburkardt/py_src/sobol/sobol.html Someone cleaned up the comments in these sources and put them in the format of docstrings. It's much more readable and you can access it here: https://github.com/naught101/sobol_seq


Question:
Improving Numpy Performance I'd like to improve the performance of convolution using python, and was hoping for some insight on how to best go about improving performance. I am currently using scipy to perform the convolution, using code somewhat like the snippet below:
```
import numpy
import scipy
import scipy.signal
import timeit

a=numpy.array ( [ range(1000000) ] )
a.reshape(1000,1000)
filt=numpy.array( [ [ 1, 1, 1 ], [1, -8, 1], [1,1,1] ] )

def convolve():
  global a, filt
  scipy.signal.convolve2d ( a, filt, mode="same" )

t=timeit.Timer("convolve()", "from __main__ import convolve")
print "%.2f sec/pass" % (10 * t.timeit(number=10)/100)
```
I am processing image data, using grayscale (integer values between 0 and 255), and I currently get about a quarter of a second per convolution. My thinking was to do one of the following: Use corepy, preferably with some optimizations Recompile numpy with icc & ikml. Use python-cuda. I was wondering if anyone had any experience with any of these approaches ( what sort of gain would be typical, and if it is worth the time ), or if anyone is aware of a better library to perform convolution with Numpy. Thanks! EDIT: Speed up of about 10x by re-writing python loop in C over using Numpy.

Response:
The code in scipy for doing 2d convolutions is a bit messy and unoptimized. See http://svn.scipy.org/svn/scipy/trunk/scipy/signal/firfilter.c if you want a glimpse into the low-level functioning of scipy. If all you want is to process with a small, constant kernel like the one you showed, a function like this might work:
```
def specialconvolve(a):
    # sorry, you must pad the input yourself
    rowconvol = a[1:-1,:] + a[:-2,:] + a[2:,:]
    colconvol = rowconvol[:,1:-1] + rowconvol[:,:-2] + rowconvol[:,2:] - 9*a[1:-1,1:-1]
    return colconvol
```
This function takes advantage of the separability of the kernel like DarenW suggested above, as well as taking advantage of the more optimized numpy arithmetic routines. It's over 1000 times faster than the convolve2d function by my measurements.


Question:
How exactly does the "reflect" mode for scipys ndimage filters work? I'm failing to understand exactly how the reflect mode handles my arrays. I have this very simple array:
```
import numpy as np
from scipy.ndimage.filters import uniform_filter
from scipy.ndimage.filters import median_filter

vector = np.array([[1.0,1.0,1.0,1.0,1.0],[2.0,2.0,2.0,2.0,2.0],[4.0,4.0,4.0,4.0,4.0],[5.0,5.0,5.0,5.0,5.0]])

print(vector)
```
[[ 1. 1. 1. 1. 1.] [ 2. 2. 2. 2. 2.] [ 4. 4. 4. 4. 4.] [ 5. 5. 5. 5. 5.]] Applying a uniform (mean) filter with a window size of 3 I get the following:
```
filtered = uniform_filter(vector, 3, mode='reflect')

print(filtered)
```
[[ 1.33333333 1.33333333 1.33333333 1.33333333 1.33333333] [ 2.33333333 2.33333333 2.33333333 2.33333333 2.33333333] [ 3.66666667 3.66666667 3.66666667 3.66666667 3.66666667] [ 4.66666667 4.66666667 4.66666667 4.66666667 4.66666667]] If I try to replicate the exercise by hand I can get to this result. Original matrix in green, window in orange and result in yellow. White are "reflected" observations. Result is: But when I try a window size of 4 or 5 I fail to be able to replicate the results.
```
filtered = uniform_filter(vector, 4, mode='reflect')

print(filtered)
```
[[ 1.5 1.5 1.5 1.5 1.5] [ 2. 2. 2. 2. 2. ] [ 3. 3. 3. 3. 3. ] [ 4. 4. 4. 4. 4. ]] Doing it by hand: And I get: How is the window handled if its size is even? But anyway, If I try to replicate the results of a window of size 5 and mode reflect I cant either. Even though I would think the behavior is analogous to that of size 3.

Response:
Suppose the data in one axis is 1 2 3 4 5 6 7 8. The following table shows how the data is extended for each mode (assuming cval=0):
```
mode       |   Ext   |         Input          |   Ext
    -----------+---------+------------------------+---------
    'mirror'   | 4  3  2 | 1  2  3  4  5  6  7  8 | 7  6  5
    'reflect'  | 3  2  1 | 1  2  3  4  5  6  7  8 | 8  7  6
    'nearest'  | 1  1  1 | 1  2  3  4  5  6  7  8 | 8  8  8
    'constant' | 0  0  0 | 1  2  3  4  5  6  7  8 | 0  0  0
    'wrap'     | 6  7  8 | 1  2  3  4  5  6  7  8 | 1  2  3
```
For an even window size n, consider the window of size n+1, and then don't include the lower and right edges. (The position of the window can be changed by using the origin argument.)


Question:
Creating a mixture of probability distributions for sampling Is there a general way to join SciPy (or NumPy) probability distributions to create a mixture probability distribution which can then be sampled from? I have such a distribution for display using something like:
```
mixture_gaussian = (norm.pdf(x_axis, -3, 1) + norm.pdf(x_axis, 3, 1)) / 2
```
which if then plotted looks like: However, I can't sample from this generated model, as it's just a list of points which will plot as the curve. Note, this specific distribution is just a simple example. I'd like to be able to generate several kinds of distributions (including "sub"-distributions which are not just normal distributions). Ideally, I would hope there would be someway for the function to be automatically normalized (i.e. not having to do the / 2 explicitly as in the code above. Does SciPy/NumPy provide some way of easily accomplishing this? This answer provides a way that such a sampling from a multiple distributions could be done, but it certainly requires a bit of handcrafting for a given mixture distribution, especially when wanting to weight different "sub"-distributions differently. This is usable, but I would hope for method that's a bit cleaner and straight forward if possible. Thanks!

Response:
Sampling from a mixture of distributions (where PDFs are added with some coefficients c_1, c_2, ... c_n) is equivalent to sampling each independently, and then, for each index, picking the value from k-th sample, with probability c_k. The latter, mixing, step can be efficiently done with numpy.random.choice. Here is an example where three distributions are mixed. The distributions are listed in distributions, and their coefficients in coefficients. There is a fat normal distribution, a uniform distribution, and a narrow normal distribution, with coefficients 0.5, 0.2, 0.3. The mixing happens at data[np.arange(sample_size), random_idx] after random_idx are generated according to given coefficients.
```
import numpy as np
import matplotlib.pyplot as plt

distributions = [
    {"type": np.random.normal, "kwargs": {"loc": -3, "scale": 2}},
    {"type": np.random.uniform, "kwargs": {"low": 4, "high": 6}},
    {"type": np.random.normal, "kwargs": {"loc": 2, "scale": 1}},
]
coefficients = np.array([0.5, 0.2, 0.3])
coefficients /= coefficients.sum()      # in case these did not add up to 1
sample_size = 100000

num_distr = len(distributions)
data = np.zeros((sample_size, num_distr))
for idx, distr in enumerate(distributions):
    data[:, idx] = distr["type"](size=(sample_size,), **distr["kwargs"])
random_idx = np.random.choice(np.arange(num_distr), size=(sample_size,), p=coefficients)
sample = data[np.arange(sample_size), random_idx]
plt.hist(sample, bins=100, density=True)
plt.show()
```


Question:
Multivariate Normal CDF in Python using scipy In order to calculate the CDF of a multivariate normal, I followed this example (for the univariate case) but cannot interpret the output produced by scipy:
```
from scipy.stats import norm
import numpy as np
mean = np.array([1,5])
covariance = np.matrix([[1, 0.3 ],[0.3, 1]])
distribution = norm(loc=mean,scale = covariance)
print distribution.cdf(np.array([2,4]))
```
The output produced is:
```
[[  8.41344746e-01   4.29060333e-04]
 [  9.99570940e-01   1.58655254e-01]]
```
If the joint CDF is defined as:
```
P (X1 ≤ x1, . . . ,Xn ≤ xn)
```
then the expected output should be a real number between 0 and 1.

Response:
After searching a lot, I think this blog entry by Noah H. Silbert describes the only readymade code from a standard library that can be used for computing the cdf for a multivariate normal in Python. Scipy has a way to do it but as mentioned in the blog, it is difficult to find. The approach is based on a paper by Alan Genz’s. From the blog, this is how it works.
```
from scipy.stats import mvn
import numpy as np
low = np.array([-10, -10])
upp = np.array([.1, -.2])
mu = np.array([-.3, .17])
S = np.array([[1.2,.35],[.35,2.1]])
p,i = mvn.mvnun(low,upp,mu,S)
print p

0.2881578675080012
```


Question:
Multivariate kernel density estimation in Python I am trying to use SciPy's gaussian_kde function to estimate the density of multivariate data. In my code below I sample a 3D multivariate normal and fit the kernel density but I'm not sure how to evaluate my fit.
```
import numpy as np
from scipy import stats

mu = np.array([1, 10, 20])
sigma = np.matrix([[4, 10, 0], [10, 25, 0], [0, 0, 100]])
data = np.random.multivariate_normal(mu, sigma, 1000)
values = data.T
kernel = stats.gaussian_kde(values)
```
I saw this but not sure how to extend it to 3D. Also not sure how do I even begin to evaluate the fitted density? How do I visualize this?

Response:
There are several ways you might visualize the results in 3D. The easiest is to evaluate the gaussian KDE at the points that you used to generate it, and then color the points by the density estimate. For example:
```
import numpy as np
from scipy import stats
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D

mu=np.array([1,10,20])
sigma=np.matrix([[4,10,0],[10,25,0],[0,0,100]])
data=np.random.multivariate_normal(mu,sigma,1000)
values = data.T

kde = stats.gaussian_kde(values)
density = kde(values)

fig, ax = plt.subplots(subplot_kw=dict(projection='3d'))
x, y, z = values
ax.scatter(x, y, z, c=density)
plt.show()
```
If you had a more complex (i.e. not all lying in a plane) distribution, then you might want to evaluate the KDE on a regular 3D grid and visualize isosurfaces (3D contours) of the volume. It's easiest to use Mayavi for the visualiztion:
```
import numpy as np
from scipy import stats
from mayavi import mlab

mu=np.array([1,10,20])
# Let's change this so that the points won't all lie in a plane...
sigma=np.matrix([[20,10,10],
                 [10,25,1],
                 [10,1,50]])

data=np.random.multivariate_normal(mu,sigma,1000)
values = data.T

kde = stats.gaussian_kde(values)

# Create a regular 3D grid with 50 points in each dimension
xmin, ymin, zmin = data.min(axis=0)
xmax, ymax, zmax = data.max(axis=0)
xi, yi, zi = np.mgrid[xmin:xmax:50j, ymin:ymax:50j, zmin:zmax:50j]

# Evaluate the KDE on a regular grid...
coords = np.vstack([item.ravel() for item in [xi, yi, zi]])
density = kde(coords).reshape(xi.shape)

# Visualize the density estimate as isosurfaces
mlab.contour3d(xi, yi, zi, density, opacity=0.5)
mlab.axes()
mlab.show()
```


Question:
Dot product of a vector in SciPy/NumPy (getting ValueError: objects are not aligned) I just started learning SciPy and am struggling with the most basic features. Consider the following standard vector:
```

```python
W=array([[1],[2]])
```

```python
print W
[[1]
 [2]]
```
If I understand it correctly, this should be the SciPy representation of a standard 2x1 mathematical vector, like this:
```
(1)
(2)
```
The dot product of this vector should simply be 1*1+2*2=5. However, this does not work in SciPy:
```
```

```python
dot(W, W)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
/home/ingo/<ipython-input-16-961b62a82495> in <module>()
----> 1 dot(W, W)

ValueError: objects are not aligned
```
Note that the following works. This should be a vector of the form (1 2) if I am not mistaken.
```
```

```python
V=array([1,2])
```

```python
print V
[1 2]
```

```python
dot(V, V)
#Output
#5
#```
#What is my misconception? What am I doing wrong?
```

Response:
The key here is that numpy/scipy honours the shape of arrays when computing dot products. Looking at your first example, W is a 2x1 array:
```

```python
W=array([[1],[2]])
```

```python
print W.shape
------> print(W.shape)
(2, 1)
```
it is, therefore, necessary to use the transpose operator to compute the dot (inner) product of W with itself:
```
```

```python
print dot(W.T,W)
------> print(dot(W.T,W))
[[5]]
```

```python
print np.asscalar(dot(W.T,W))
-------> print(np.asscalar(dot(W.T,W)))
5
```
```


Question:
How to select elements row-wise from a NumPy array? I have an array like this numpy array
```
dd= [[foo 0.567 0.611]
     [bar 0.469 0.479]
     [noo 0.220 0.269]
     [tar 0.480 0.508]
     [boo 0.324 0.324]]
```
How would one loop through array selecting foo and getting 0.567 0.611 as floats as a singleton. Then select bar and getting 0.469 0.479 as floats as a singleton ..... I could get vector the first elements as list by using
```
dv=  dd[:,1]
```
The 'foo', and 'bar' elements are not unknown variables, they can change. How would I change if element is in position [1]?
```
[[0.567 foo2 0.611]
  [0.469 bar2 0.479]
  [0.220 noo2 0.269]
  [0.480 tar2 0.508]
  [0.324 boo2 0.324]]
```

Response:
You have put the NumPy tag on your Question, so i'll assume you want NumPy syntax, which the answer before mine doesn't use. If in fact you wish to use NumPy, then you likely don't want the strings in your array, otherwise you will also have to represent your floats as strings. What you are looking for is the NumPy syntax to access elements of a 2D array by row (and exclude the first column). That syntax is:
```
M[row_index,1:]        # selects all but 1st col from row given by 'row_index'
```
W/r/t the second scenario in your Question--selecting non-adjacent columns:
```
M[row_index,[0,2]]     # selects 1st & 3rd cols from row given by 'row_index'
```
The small complication in your Question is just that you want to use a string for row_index, so it's necessary to remove the strings (so you can create a 2D NumPy array of floats), replace them with numerical row indices and then create a look-up table to map the the strings with the numerical row indices:
```

```python
import numpy as NP
```

```python
# create a look-up table so you can remove the strings from your python nested list,
```

```python
# which will allow you to represent your data as a 2D NumPy array with dtype=float
```

```python
keys
#Output
#      ['foo', 'bar', 'noo', 'tar', 'boo']
```

```python
values    # 1D index array comprised of one float value for each unique string in 'keys'
#Output
#      array([0., 1., 2., 3., 4.])
```

```python
LuT = dict(zip(keys, values))
```

```python
# add an index to data by inserting 'values' array as first column of the data matrix
```

```python
A = NP.hstack((vals, A))
```

```python
A
#Output
#        NP.array([  [ 0., .567, .611],
#                    [ 1., .469, .479],
#                    [ 2., .22, .269],
#                    [ 3., .48, .508],
#                    [ 4., .324, .324] ])
```

```python
# so now to look up an item, by 'key':
```

```python
# write a small function to perform the look-ups:
```

```python
def select_row(key):
#Output
#        return A[LuT[key],1:]
```

```python
select_row('foo')
#Output
#      array([ 0.567,  0.611])
```

```python
select_row('noo')
#Output
#      array([ 0.22 ,  0.269])
#```
#The second scenario in your Question: what if the index column changes?
#```
```

```python
# e.g., move index to column 1 (as in your Q)
```

```python
A = NP.roll(A, 1, axis=1)
```

```python
A
#Output
#      array([[ 0.611,  1.   ,  0.567],
#             [ 0.479,  2.   ,  0.469],
#             [ 0.269,  3.   ,  0.22 ],
#             [ 0.508,  4.   ,  0.48 ],
#             [ 0.324,  5.   ,  0.324]])
```

```python
# the original function is changed slightly, to select non-adjacent columns:
```

```python
def select_row2(key):
#Output
#        return A[LuT[key],[0,2]]
```

```python
select_row2('foo')
#Output
#        array([ 0.611,  0.567])
#```
```


Question:
Pearson correlation coefficient 2-tailed p-value meaning [closed] Closed. This question is off-topic. It is not currently accepting answers. Want to improve this question? Update the question so it's on-topic for Stack Overflow. Closed 12 years ago. Improve this question from the sciPy library I used: scipy.stats.stats import pearsonr to calculate the correlation coefficient for two arrays and I got a value of: (0.80751532276005755, 0.19248467723994242). I thought that I would have just got one value within the range -1 to +1, so I'm unsure how to interpret these two results. These are my two arrays:
```
x = [50,500,1500,2500]
y = [17, 6, 6, 194]
```
and I did:
```
pearsonr(x,y)
```
Thanks

Response:
pearsonr() returns a two-tuple consisting of the correlation coefficient and the corresponding p-value: The correlation coefficient can range from -1 to +1. The null hypothesis is that the two variables are uncorrelated. The p-value is a number between zero and one that represents the probability that your data would have arisen if the null hypothesis were true. For a further discussion, see http://www.eecs.qmul.ac.uk/~norman/blog_articles/p_values.pdf I thought that I would have just got one value within the range -1 to +1 If you just need to the correlation coefficient, simply ignore the second element of the tuple (the p-value):
```
corrxy = pearsonr(x,y)[0]
```
It might be worth mentioning that there's also numpy.corrcoef(), which computes the correlation matrix (without p-values).


Question:
Spline with constraints at border I have measured data on a three dimensional grid, e.g. f(x, y, t). I want to interpolate and smooth this data in the direction of t with splines. Currently, I do this with scipy.interpolate.UnivariateSpline:
```
import numpy as np
from scipy.interpolate import UnivariateSpline

# data is my measured data
# data.shape is (len(y), len(x), len(t))
data = np.arange(1000).reshape((5, 5, 40))  # just for demonstration
times = np.arange(data.shape[-1])
y = 3
x = 3
sp = UnivariateSpline(times, data[y, x], k=3, s=6)
```
However, I need the spline to have vanishing derivatives at t=0. Is there a way to enforce this constraint?

Response:
The best thing I can think of is to do a minimization with a constraint with scipy.optimize.minimize. It is pretty easy to take the derivative of a spline, so the constraint is simply. I would use a regular spline fit (UnivariateSpline) to get the knots (t), and hold the knots fixed (and degree k, of course), and vary the coefficients c. Maybe there is a way to vary the knot locations as well but I will leave that to you.
```
import numpy as np
from scipy.interpolate import UnivariateSpline, splev, splrep
from scipy.optimize import minimize

def guess(x, y, k, s, w=None):
    """Do an ordinary spline fit to provide knots"""
    return splrep(x, y, w, k=k, s=s)

def err(c, x, y, t, k, w=None):
    """The error function to minimize"""
    diff = y - splev(x, (t, c, k))
    if w is None:
        diff = np.einsum('...i,...i', diff, diff)
    else:
        diff = np.dot(diff*diff, w)
    return np.abs(diff)

def spline_neumann(x, y, k=3, s=0, w=None):
    t, c0, k = guess(x, y, k, s, w=w)
    x0 = x[0] # point at which zero slope is required
    con = {'type': 'eq',
           'fun': lambda c: splev(x0, (t, c, k), der=1),
           #'jac': lambda c: splev(x0, (t, c, k), der=2) # doesn't help, dunno why
           }
    opt = minimize(err, c0, (x, y, t, k, w), constraints=con)
    copt = opt.x
    return UnivariateSpline._from_tck((t, copt, k))
```
And then we generate some fake data that should have zero initial slope and test it:
```
import matplotlib.pyplot as plt

n = 10
x = np.linspace(0, 2*np.pi, n)
y0 = np.cos(x) # zero initial slope
std = 0.5
noise = np.random.normal(0, std, len(x))
y = y0 + noise
k = 3

sp0 = UnivariateSpline(x, y, k=k, s=n*std)
sp = spline_neumann(x, y, k, s=n*std)

plt.figure()
X = np.linspace(x.min(), x.max(), len(x)*10)
plt.plot(X, sp0(X), '-r', lw=1, label='guess')
plt.plot(X, sp(X), '-r', lw=2, label='spline')
plt.plot(X, sp.derivative()(X), '-g', label='slope')
plt.plot(x, y, 'ok', label='data')
plt.legend(loc='best')
plt.show()
```


Question:
scipy.misc.derivative for multiple argument function It is straightforward to compute the partial derivatives of a function at a point with respect to the first argument using the SciPy function scipy.misc.derivative. Here is an example:
```
def foo(x, y):
  return(x**2 + y**3)

from scipy.misc import derivative
derivative(foo, 1, dx = 1e-6, args = (3, ))
```
But how would I go about taking the derivative of the function foo with respect to the second argument? One way I can think of is to generate a lambda function that rejigs the arguments around, but that can quickly get cumbersome. Also, is there a way to generate an array of partial derivatives with respect to some or all of the arguments of a function?

Response:
I would write a simple wrapper, something along the lines of
```
def partial_derivative(func, var=0, point=[]):
    args = point[:]
    def wraps(x):
        args[var] = x
        return func(*args)
    return derivative(wraps, point[var], dx = 1e-6)
```
Demo:
```

```python
partial_derivative(foo, 0, [3,1])
#Output
#6.0000000008386678
```

```python
partial_derivative(foo, 1, [3,1])
#Output
#2.9999999995311555
#```
```


Question:
Passing arguments to fsolve I'm solving a nonlinear equation with many constants. I created a function for solving like:
```
def terminalV(Vt, data):
    from numpy import sqrt
    ro_p, ro, D_p, mi, g = (i for i in data)
    y = sqrt((4*g*(ro_p - ro)*D_p)/(3*C_d(Re(data, Vt))*ro)) - Vt
    return y
```
Then I want to do:
```
data = (1800, 994.6, 0.208e-3, 8.931e-4, 9.80665)
Vt0 = 1
Vt = fsolve(terminalV, Vt0, args=data)
```
But fsolve is unpacking data and passing too many arguments to terminalV function, so I get: TypeError: terminalV() takes exactly 2 arguments (6 given) So, my question can I somehow pass a tuple to the function called by fsolve()?

Response:
The problem is that you need to use an asterisk to tell your function to repack the tuple. The standard way to pass arguments as a tuple is the following:
```
from numpy import sqrt   # leave this outside the function
from scipy.optimize import fsolve

#  here it is     V
def terminalV(Vt, *data):
    ro_p, ro, D_p, mi, g = data   # automatic unpacking, no need for the 'i for i'
    return sqrt((4*g*(ro_p - ro)*D_p)/(3*C_d(Re(data, Vt))*ro)) - Vt

data = (1800, 994.6, 0.208e-3, 8.931e-4, 9.80665)
Vt0 = 1
Vt = fsolve(terminalV, Vt0, args=data)
```
Without fsolve, i.e., if you just want to call terminalV on its own, for example if you want to see its value at Vt0, then you must unpack data with a star:
```
data = (1800, 994.6, 0.208e-3, 8.931e-4, 9.80665)
Vt0 = 1
terminalV(Vt0, *data)
```
Or pass the values individually:
```
terminalV(Vt0, 1800, 994.6, 0.208e-3, 8.931e-4, 9.80665)
```


Question:
What is the difference between skew and kurtosis functions in pandas vs. scipy? I decided to compare skew and kurtosis functions in pandas and scipy.stats, and don't understand why I'm getting different results between libraries. As far as I can tell from the documentation, both kurtosis functions compute using Fisher's definition, whereas for skew there doesn't seem to be enough of a description to tell if there any major differences with how they are computed.
```
import pandas as pd
import scipy.stats.stats as st

heights = np.array([1.46, 1.79, 2.01, 1.75, 1.56, 1.69, 1.88, 1.76, 1.88, 1.78])

print "skewness:", st.skew(heights)
print "kurtosis:", st.kurtosis(heights)
```
this returns:
```
skewness: -0.393524456473
kurtosis: -0.330672097724
```
whereas if I convert to a pandas dataframe:
```
heights_df = pd.DataFrame(heights)
print "skewness:", heights_df.skew()
print "kurtosis:", heights_df.kurtosis()
```
this returns:
```
skewness: 0   -0.466663
kurtosis: 0    0.379705
```
Apologies if I've posted this in the wrong place; not sure if it's a stats or a programming question.

Response:
The difference is due to different normalizations. Scipy by default does not correct for bias, whereas pandas does. You can tell scipy to correct for bias by passing the bias=False argument:
```

```python
x = pandas.Series(np.random.randn(10))
```

```python
stats.skew(x)
#Output
#-0.17644348972413657
```

```python
x.skew()
#Output
#-0.20923623968879457
```

```python
stats.skew(x, bias=False)
#Output
#-0.2092362396887948
```

```python
stats.kurtosis(x)
#Output
#0.6362620964462327
```

```python
x.kurtosis()
#Output
#2.0891062062174464
```

```python
stats.kurtosis(x, bias=False)
#Output
#2.089106206217446
#```
#There does not appear to be a way to tell pandas to remove the bias correction.
```


Question: Difference between random draws from scipy.stats....rvs and numpy.random It seems if it is the same distribution, drawing random samples from numpy.random is faster than doing so from scipy.stats.-.rvs. I was wondering what causes the speed difference between the two?

Response:
scipy.stats.uniform actually uses numpy, here is the corresponding function in stats (mtrand is an alias for numpy.random)
```
class uniform_gen(rv_continuous):
    def _rvs(self):
        return mtrand.uniform(0.0,1.0,self._size)
```
scipy.stats has a bit of overhead for error checking and making the interface more flexible. The speed difference should be minimal as long as you don't call uniform.rvs in a loop for each draw. You can get instead all random draws at once, for example (10 million)
```

```python
rvs = stats.uniform.rvs(size=(10000, 1000))
```

```python
rvs.shape
#Output
#(10000, 1000)
#```
#Here is the long answer, that I wrote a while ago: The basic random numbers in scipy/numpy are created by Mersenne-Twister PRNG in numpy.random. The random numbers for distributions in numpy.random are in cython/pyrex and are pretty fast. scipy.stats doesn't have a random number generator, random numbers are obtained in one of three ways: directly from numpy.random, e.g. normal, t, ... pretty fast random numbers by transformation of other random numbers that are available in numpy.random, also pretty fast because this operates on entire arrays of numbers generic: the only generic generation random number generation is by using the ppf (inverse cdf) to transform uniform random numbers. This is relatively fast if there is an explicit expression for the ppf, but can be very slow if the ppf has to be calculated indirectly. For example if only the pdf is defined, then the cdf is obtained through numerical integration and the ppf is obtained through an equation solver. So a few distributions are very slow.
```


Question:
Finding alpha and beta of beta-binomial distribution with scipy.optimize and loglikelihood A distribution is beta-binomial if p, the probability of success, in a binomial distribution has a beta distribution with shape parameters α > 0 and β > 0. The shape parameters define the probability of success. I want to find the values for α and β that best describe my data from the perspective of a beta-binomial distribution. My dataset players consist of data about the number of hits (H), the number of at-bats (AB) and the conversion (H / AB) of a lot of baseball players. I estimate the PDF with the help of the answer of JulienD in Beta Binomial Function in Python
```
from scipy.special import beta
from scipy.misc import comb

pdf = comb(n, k) * beta(k + a, n - k + b) / beta(a, b)
```
Next, I write a loglikelihood function that we will minimize.
```
def loglike_betabinom(params, *args):
   """
   Negative log likelihood function for betabinomial distribution
   :param params: list for parameters to be fitted.
   :param args:  2-element array containing the sample data.
   :return: negative log-likelihood to be minimized.
   """

   a, b = params[0], params[1]
   k = args[0] # the conversion rate
   n = args[1] # the number of at-bats (AE)

   pdf = comb(n, k) * beta(k + a, n - k + b) / beta(a, b)

   return -1 * np.log(pdf).sum()
```
Now, I want to write a function that minimizes loglike_betabinom
```
from scipy.optimize import minimize
 init_params = [1, 10]
 res = minimize(loglike_betabinom, x0=init_params,
                args=(players['H'] / players['AB'], players['AB']),
                bounds=bounds,
                method='L-BFGS-B',
                options={'disp': True, 'maxiter': 250})
 print(res.x)
```
The result is [-6.04544138 2.03984464], which implies that α is negative which is not possible. I based my script on the following R-snippet. They get [101.359, 287.318]..
```
ll <- function(alpha, beta) {
    x <- career_filtered$H
    total <- career_filtered$AB
    -sum(VGAM::dbetabinom.ab(x, total, alpha, beta, log=True))
 }

 m <- mle(ll, start = list(alpha = 1, beta = 10),
 method = "L-BFGS-B", lower = c(0.0001, 0.1))

 ab <- coef(m)
```
Can someone tell me what I am doing wrong? Help is much appreciated!!

Response:
One thing to pay attention to is that comb(n, k) in your log-likelihood might not be well-behaved numerically for the values of n and k in your dataset. You can verify this by applying comb to your data and see if infs appear. One way to amend things could be to rewrite the negative log-likelihood as suggested in https://stackoverflow.com/a/32355701/4240413, i.e. as a function of logarithms of Gamma functions as in
```
from scipy.special import gammaln
import numpy as np

def loglike_betabinom(params, *args):

    a, b = params[0], params[1]
    k = args[0] # the OVERALL conversions
    n = args[1] # the number of at-bats (AE)

    logpdf = gammaln(n+1) + gammaln(k+a) + gammaln(n-k+b) + gammaln(a+b) - \
     (gammaln(k+1) + gammaln(n-k+1) + gammaln(a) + gammaln(b) + gammaln(n+a+b))

    return -np.sum(logpdf)
```
You can then minimize the log-likelihood with
```
from scipy.optimize import minimize

init_params = [1, 10]
# note that I am putting 'H' in the args
res = minimize(loglike_betabinom, x0=init_params,
            args=(players['H'], players['AB']),
            method='L-BFGS-B', options={'disp': True, 'maxiter': 250})
print(res)
```
and that should give reasonable results. You could check How to properly fit a beta distribution in python? for inspiration if you want to rework further your code.


Question:
Matrix exponentiation in Python I'm trying to exponentiate a complex matrix in Python and am running into some trouble. I'm using the scipy.linalg.expm function, and am having a rather strange error message when I try the following code:
```
import numpy as np
from scipy import linalg

hamiltonian = np.mat('[1,0,0,0;0,-1,0,0;0,0,-1,0;0,0,0,1]')

# This works
t_list = np.linspace(0,1,10)
unitary = [linalg.expm(-(1j)*t*hamiltonian) for t in t_list]

# This doesn't
t_list = np.linspace(0,10,100)
unitary = [linalg.expm(-(1j)*t*hamiltonian) for t in t_list]
```
The error when the second experiment is run is:
```
This works!
Traceback (most recent call last):
  File "matrix_exp.py", line 11, in <module>
    unitary_t = [linalg.expm(-1*t*(1j)*hamiltonian) for t in t_list]
  File "/usr/lib/python2.7/dist-packages/scipy/linalg/matfuncs.py",     line 105, in expm
    return scipy.sparse.linalg.expm(A)
  File "/usr/lib/python2.7/dist- packages/scipy/sparse/linalg/matfuncs.py", line 344, in expm
    X = _fragment_2_1(X, A, s)
  File "/usr/lib/python2.7/dist-  packages/scipy/sparse/linalg/matfuncs.py", line 462, in _fragment_2_1
    X[k, k] = exp_diag[k]
TypeError: only length-1 arrays can be converted to Python scalars
```
This seems really strange since all I changed was the range of t I was using. Is it because the Hamiltonian is diagonal? In general, the Hamiltonians won't be, but I also want it to work for diagonal ones. I don't really know the mechanics of expm, so any help would be greatly appreciated.

Response:
That is interesting. One thing I can say is that the problem is specific to the np.matrix subclass. For example, the following works fine:
```
h = np.array(hamiltonian)
unitary = [linalg.expm(-(1j)*t*h) for t in t_list]
```
Digging a little deeper into the traceback, the exception is being raised in _fragment_2_1 in scipy.sparse.linalg.matfuncs.py, specifically these lines:
```
n = X.shape[0]
diag_T = T.diagonal().copy()

# Replace diag(X) by exp(2^-s diag(T)).
scale = 2 ** -s
exp_diag = np.exp(scale * diag_T)
for k in range(n):
    X[k, k] = exp_diag[k]
```
The error message
```
X[k, k] = exp_diag[k]
TypeError: only length-1 arrays can be converted to Python scalars
```
suggests to me that exp_diag[k] ought to be a scalar, but is instead returning a vector (and you can't assign a vector to X[k, k], which is a scalar). Setting a breakpoint and examining the shapes of these variables confirms this:
```
ipdb> l
    751     # Replace diag(X) by exp(2^-s diag(T)).
    752     scale = 2 ** -s
    753     exp_diag = np.exp(scale * diag_T)
    754     for k in range(n):
    755         import ipdb; ipdb.set_trace()  # breakpoint e86ebbd4 //
--> 756         X[k, k] = exp_diag[k]
    757
    758     for i in range(s-1, -1, -1):
    759         X = X.dot(X)
    760
    761         # Replace diag(X) by exp(2^-i diag(T)).

ipdb> exp_diag.shape
(1, 4)
ipdb> exp_diag[k].shape
(1, 4)
ipdb> X[k, k].shape
()
```
The underlying problem is that exp_diag is assumed to be either 1D or a column vector, but the diagonal of an np.matrix object is a row vector. This highlights a more general point that np.matrix is generally less well-supported than np.ndarray, so in most cases it's better to use the latter. One possible solution would be to use np.ravel() to flatten diag_T into a 1D np.ndarray:
```
diag_T = np.ravel(T.diagonal().copy())
```
This seems to fix the problem you're encountering, although there may be other issues relating to np.matrix that I haven't spotted yet. I've opened a pull request here.


Question:
scipy.optimize.leastsq calls objective function with NaN I am using scipy.optimize.leastsq to attempt to fit a number of parameters to real-world data in the presence of noise. The objective function occasionally gets called with NaNs from within minpack. Is this the expected behavior of scipy.optimize.leastsq? Is there any better option than just returning NaN residuals under this condition? The following code demonstrates the behavior:
```
import scipy.optimize
import numpy as np

xF = np.array([1.0, 2.0, 3.0, 4.0]) # Target value for fit
NOISE_LEVEL = 1e-6 # The random noise level
RETURN_LEN = 1000  # The objective function return vector length

def func(x):
    if np.isnan(np.sum(x)):
        raise ValueError('Invalid x: %s' % x)
    v = np.random.rand(RETURN_LEN) * NOISE_LEVEL
    v[:len(x)] += xF - x
    return v

iteration = 0
while (1):
    iteration += 1
    x = np.zeros(len(xF))
    y, cov = scipy.optimize.leastsq(func, x)
    print('%04d %s' % (iteration, y))
```
The Jacobian is being computed numerically. In the production code, the optimization normally works except for when the starting estimate is too good, the surface is exceedingly flat, and noise overwhelms the deltas used to numerically compute the Jacobian. In this case, the residuals of the objective function appear as random noise like the above code example, and I would not expect the optimization to converge. In this code example, small NOISE_LEVEL values (<1e-10) always converge. At 1e-6, the ValueError is thrown typically within a few hundred attempts. One possible workaround is to return a highly penalized residual (either NaN or INF), such as:
```
v = np.empty(RETURN_LEN)
v.fill(np.nan)
return v
```
This workaround seems to be effective if dirty. Any better alternatives or ways to prevent NaNs in the first place? This behavior was observed under Python 2.7.9 x32 running on Windows 7.

Response:
Since your problem definition uses a linear solver on a non-linear problem (noise), the solver will freak out, unless the noise level is below the threshold of noticeability. In order to solve this problem, you could try using a nonlinear solver. For instance using the broyden1 solving algorithm instead of leastsq:
```
import scipy.optimize
import numpy as np

xF = np.array([1.0, 2.0, 3.0, 4.0]) # Target value for fit
NOISE_LEVEL = 1.e-6 # The random noise level
RETURN_LEN = 1000  # The objective function return vector length

def func(x):
    if np.isnan(np.sum(x)):
        raise ValueError('Invalid x: %s' % x)
    v = np.random.rand(RETURN_LEN) * NOISE_LEVEL

    v[:len(x)] += xF - x

    return v[:len(x)]

iteration = 0
while iteration < 10:
    iteration += 1
    x = np.random.rand(len(xF))
    y = scipy.optimize.broyden1(func, x)
    print('%04d %s' % (iteration, y))
```
returns as output:
```
0001 [ 1.00000092  2.00000068  3.00000051  4.00000097]
0002 [ 1.0000012   2.00000214  3.00000272  4.00000369]
0003 [ 0.99999991  1.99999931  2.99999815  3.9999978 ]
0004 [ 1.00000097  2.00000198  3.00000345  4.00000425]
0005 [ 1.00000047  1.99999983  2.99999938  3.99999922]
0006 [ 1.00000024  2.00000021  3.00000071  4.00000136]
0007 [ 1.00000116  2.00000102  3.00000225  4.00000357]
0008 [ 1.00000006  2.00000002  3.00000017  4.00000039]
0009 [ 1.0000002   2.00000034  3.00000062  4.00000051]
0010 [ 1.00000137  2.0000015   3.00000193  4.00000344]
```


Question:
Continuous mutual information in Python [Frontmatter] (skip this if you just want the question): I'm currently looking at using Shannon-Weaver Mutual Information and normalized redundancy to measure the degree of information masking between bags of discrete and continuous feature values, organized by feature. Using this method, it is my goal to construct an algorithm that looks very similar to ID3, but instead of using Shannon entropy, the algorithm will seek (as a loop constraint) to maximize or minimize shared information between a single feature and a collection of features based on the complete input feature space, adding new features to the latter collection if (and only if) they increase or decrease mutual information, respectively. This, in effect, moves ID3's decision algorithm into pairspace, stapling an ensemble approach to it with all of the expected time and space complexities of both methods. [/Frontmatter] On to the question: I'm trying to get a continuous integrator working in Python using SciPy. Because I'm working with comparisons of discrete and continuous variables, my current strategy for each comparison for feature-feature pairs is as follows: Discrete feature versus discrete feature: use the discrete form of mutual information. This results in a double summation of the probabilities, which my code handles without issue. All other cases (discrete versus continuous, the inverse, and continuous versus continuous): use the continuous form, using a Gaussian estimator to smooth out the probability density functions. It is possible for me to perform some kind of discretization for the latter cases, but because my input data sets are not inherently linear, this is potentially needlessly complex. Here's the salient code:
```
import math
import numpy
import scipy
from scipy.stats import gaussian_kde
from scipy.integrate import dblquad

# Constants
MIN_DOUBLE = 4.9406564584124654e-324
                    # The minimum size of a Float64; used here to prevent the
                    #  logarithmic function from hitting its undefined region
                    #  at its asymptote of 0.
INF = float('inf')  # The floating-point representation for "infinity"

# x and y are previously defined as collections of
# floating point values with the same length

# Kernel estimation
gkde_x = gaussian_kde(x)
gkde_y = gaussian_kde(y)

if len(binned_x) != len(binned_y) and len(binned_x) != len(x):
    x.append(x[0])
    y.append(y[0])

gkde_xy = gaussian_kde([x,y])
mutual_info = lambda a,b: gkde_xy([a,b]) * \
           math.log((gkde_xy([a,b]) / (gkde_x(a) * gkde_y(b))) + MIN_DOUBLE)

# Compute MI(X,Y)
(minfo_xy, err_xy) = \
    dblquad(mutual_info, -INF, INF, lambda a: 0, lambda a: INF)

print 'minfo_xy = ', minfo_xy
```
Note that overcounting exactly one point is done deliberately to prevent a singularity in SciPy's gaussian_kde class. As the size of x and y mutually approach infinity, this effect becomes negligible. My current snag is in trying to get multiple integration working against a Gaussian kernel density estimate in SciPy. I've been trying to use SciPy's dblquad to perform the integration, but in the latter case, I receive an astounding spew of the following messages. When I set numpy.seterr ( all='ignore' ): Warning: The ocurrence of roundoff error is detected, which prevents the requested tolerance from being achieved. The error may be underestimated. And when I set it to 'call' using an error handler: Floating point error (underflow), with flag 4 Floating point error (invalid value), with flag 8 Pretty easy to figure out what's going on, right? Well, almost: IEEE 754-2008 and SciPy only tell me what's going on here, not why or how to work around it. The upshot: minfo_xy generally resolves to nan; its sampling is insufficient to prevent information from becoming lost or invalid when performing Float64 math. Is there a general workaround for this problem when using SciPy? Even better: if there is a robust, canned implementation of continuous mutual information for Python with an interface that takes two collections of floating point values or a merged collection of pairs, it would resolve this complete problem. Please link it if you know of one that exists. Thank you in advance. Edit: this resolves the nan propagation issue in the example above:
```
mutual_info = lambda a,b: gkde_xy([a,b]) * \
    math.log((gkde_xy([a,b]) / ((gkde_x(a) * gkde_y(b)) + MIN_DOUBLE)) \
        + MIN_DOUBLE)
```
However, the question of roundoff correction remains, as does the request for a more robust implementation. Any help in either domain would be greatly appreciated.

Response: Before trying more radical solutions like reframing the problem or using different integration tools, see if this helps. Replace INF=float('INF') with INF=1E12 or some other large number -- that may eliminate NaN results created by simple arithmetic operations on the input variables. No promises on this one, but it is sometimes helpful to try a quick fix before engaging in a significant algorithmic rewrite or substitution of alternate tools.


Question:
Applying bounds to specific variable during curve_fit (scipy) leads to an error I am trying to apply bounds onto some of the parameters during curve fitting but I'm getting the following error message when I tried to do so: ValueError: too many values to unpack Doesn't each 2-tuple in the bound command corresponds to x0, k, lapse, guess in the sigmoidscaled function in my case respectively (Ie. corresponding to p0 too)? I then tried playing around in hopes of trying to figure out how it works by reducing the bound command to the following to rid the 'too many values': bounds=((-np.inf,np.inf), (0,1)) Then I get the error message of: ValueError: Inconsistent shapes between bounds and x0. What am I getting wrong here?
```
import pylab
from scipy.optimize import curve_fit
from matplotlib.pyplot import *

n = 20 #20 trials
ydata = [0/n, 9.0/n, 9.0/n, 14.0/n, 17.0/n] #Divided by n to fit to a plot of y =1
xdata = np.array([ 1.0, 2.0, 3.0, 4.0, 5.0])

#The scaled sigmoid function
def sigmoidscaled(x, x0, k, lapse, guess):
    F = (1 + np.exp(-k*(x-x0)))
    z = guess + (1-guess-lapse)/F
    return z

p0=[1,1,0,0]
popt, pcov = curve_fit(sigmoidscaled, xdata, ydata, p0, bounds=((-np.inf,np.inf), (-np.inf,np.inf), (0,1), (0,1))

#Start and End of x-axis, in spaces of n. The higher the n, the smoother the curve.
x = np.linspace(1,5,20)
#The sigmoid values along the y-axis, generated in relation to the x values and the 50% point.
y = sigmoidscaled(x, *popt)

pylab.plot(xdata, ydata, 'o', label='Psychometric Raw', color = 'blue')
pylab.plot(x,y, label='Psychometric Fit', color = 'blue')
#y axis range.
pylab.ylim(0, 1)
#Replace x-axis numbers as labels and y-axis numbers as percentage
xticks([1., 2., 3., 4., 5.], ['C1','CN2','N3','CN4','S5'])
yticks([0.0, 0.2, 0.4, 0.6, 0.8, 1.0], ['0%','20%','40%','60%','80%','100%'])
pylab.legend(loc='best')
xlabel('Conditions')
ylabel('% perceived more sin like')
pylab.show()
```

Response:
The problem line is:
```
popt, pcov = curve_fit(sigmoidscaled, xdata, ydata, p0, bounds=((-np.inf,np.inf), (-np.inf,np.inf), (0,1), (0,1))
```
From the documentation, bounds needs to be a 2-tuple of array likes. So instead of specifying the lower and upper bound of each point, you need to specify the lower bound of each point in the first array-like followed by the upper bound of each point in the second array-like, like this:
```
popt, pcov = curve_fit(sigmoidscaled, xdata, ydata, p0, bounds=((-np.inf, -np.inf, 0, 0), (np.inf, np.inf, 1, 1)))
```
After this change, the plot popped right up!


Question: Solve Generalized Eigenvalue Problem in Numpy I am looking to solve a problem of the type: Aw = xBw where x is a scalar (eigenvalue), w is an eigenvector, and A and B are symmetric, square numpy matrices of equal dimension. I should be able to find d x/w pairs if A and B are d x d. How would I solve this in numpy? I was looking in the Scipy docs and not finding anything like what I wanted.

Response:
For real symmetric or complex Hermitian dense matrices, you can use scipy.linalg.eigh() to solve a generalized eigenvalue problem. To avoid extracting all the eigenvalues you can specify only the desired ones by using subset_by_index:
```
from scipy.linalg import eigh

eigvals, eigvecs = eigh(A, B, eigvals_only=False, subset_by_index=[0, 1, 2])
```
One could use eigvals_only=True to obtain only the eigenvalues.


Question: extrapolating data with numpy/python Let's say I have a simple data set. Perhaps in dictionary form, it would look like this: {1:5, 2:10, 3:15, 4:20, 5:25} (the order is always ascending). What I want to do is logically figure out what the next point of data is most likely to be. In the case, for example, it would be {6: 30} what would be the best way to do this?

Response:
You can also use numpy's polyfit:
```
data = np.array([[1,5], [2,10], [3,15], [4,20], [5,25]])
fit = np.polyfit(data[:,0], data[:,1] ,1) #The use of 1 signifies a linear fit.

fit
[  5.00000000e+00   1.58882186e-15]  #y = 5x + 0

line = np.poly1d(fit)
new_points = np.arange(5)+6

new_points
[ 6, 7, 8, 9, 10]

line(new_points)
[ 30.  35.  40.  45.  50.]
```
This allows you to alter the degree of the polynomial fit quite easily as the function polyfit take thes following arguments np.polyfit(x data, y data, degree). Shown is a linear fit where the returned array looks like fit[0]*x^n + fit[1]*x^(n-1) + ... + fit[n-1]*x^0 for any degree n. The poly1d function allows you turn this array into a function that returns the value of the polynomial at any given value x. In general extrapolation without a well understood model will have sporadic results at best. Exponential curve fitting.
```
from scipy.optimize import curve_fit

def func(x, a, b, c):
    return a * np.exp(-b * x) + c

x = np.linspace(0,4,5)
y = func(x, 2.5, 1.3, 0.5)
yn = y + 0.2*np.random.normal(size=len(x))

fit ,cov = curve_fit(func, x, yn)
fit
[ 2.67217435  1.21470107  0.52942728]         #Variables

y
[ 3.          1.18132948  0.68568395  0.55060478  0.51379141]  #Original data

func(x,*fit)
[ 3.20160163  1.32252521  0.76481773  0.59929086  0.5501627 ]  #Fit to original + noise
```


Question:
How to convert data values into color information for matplotlib? I am using the class matplotlib.patches.Polygon to draw polygons on a map. In fact, the information about the coordinates of the corners of the polygons and a floating point data value for each "polygon" are given. Now I'd like to convert these data values (ranging from 0 to 3e15) into color information to visualize it nicely. What is the best practice for doing that in Python? A snippet of my code:
```
poly = Polygon( xy, facecolor=data, edgecolor='none')
plt.gca().add_patch(poly)
```

Response:
In the RGB color system two bits of data are used for each color, red, green, and blue. That means that each color runs on a scale from 0 to 255. Black would be 00,00,00, while white would be 255,255,255. Matplotlib has lots of pre-defined colormaps for you to use. They are all normalized to 255, so they run from 0 to 1. So you need only normalize your data, then you can manually select colors from a color map as follows:
```

```python
import matplotlib.pyplot as plt
```

```python
Blues = plt.get_cmap('Blues')
```

```python
print Blues(0)
#Output
#(0.9686274528503418, 0.9843137264251709, 1.0, 1.0)
```

```python
print Blues(0.5)
#Output
#(0.41708574119736169, 0.68063054575639614, 0.83823145908467911, 1.0)
```

```python
print Blues(1.0)
#Output
#(0.96555171293370867, 0.9823452528785257, 0.9990157632266774, 1.0)
#```
#Here are all of the predefined colormaps. If you're interested in creating your own color maps, here is a good example. Also here is a paper from IBM on choosing colors carefully, something you should be considering if you're using color to visualize data.
```


Question:
Extracting connected objects from an image in Python I have a graysacle png image and I want to extract all the connected components from my image. Some of the components have same intensity but I want to assign a unique label to every object. here is my image I tried this code:
```
img = imread(images + 'soccer_cif' + str(i).zfill(6) + '_GT_index.png')
labeled, nr_objects = label(img)
print "Number of objects is %d " % nr_objects
```
But I get just three objects using this. Please tell me how to get each object.

Response:
J.F. Sebastian shows a way to identify objects in an image. It requires manually choosing a gaussian blur radius and threshold value, however:
```
from PIL import Image
import numpy as np
from scipy import ndimage
import matplotlib.pyplot as plt

fname='index.png'
blur_radius = 1.0
threshold = 50

img = Image.open(fname).convert('L')
img = np.asarray(img)
print(img.shape)
# (160, 240)

# smooth the image (to remove small objects)
imgf = ndimage.gaussian_filter(img, blur_radius)
threshold = 50

# find connected components
labeled, nr_objects = ndimage.label(imgf > threshold)
print("Number of objects is {}".format(nr_objects))
# Number of objects is 4

plt.imsave('/tmp/out.png', labeled)
plt.imshow(labeled)

plt.show()
```
With blur_radius = 1.0, this finds 4 objects. With blur_radius = 0.5, 5 objects are found:


Question:
adjusting heights of individual subplots in matplotlib in Python if I have a series of subplots with one column and many rows, i.e.:
```
plt.subplot(4, 1, 1) # first subplot
plt.subplot(4, 1, 2) # second subplot
# ...
```
how can I adjust the height of the first N subplots? For example, if I have 4 subplots, each on its own row, I want all of them to have the same width but the first 3 subplots to be shorter, i.e. have their y-axes be smaller and take up less of the plot than the y-axis of the last plot in the row. How can I do this? thanks.

Response:
Even though this question is old, I was looking to answer a very similar question. @Joe's reference to AxesGrid, was the answer to my question, and has very straightforward usage, so I wanted to illustrate that functionality for completeness. AxesGrid functionality provides the ability create plots of different size and place them very specifically, via the subplot2grid functionality:
```
import matplotlib.pyplot as plt

ax1 = plt.subplot2grid((m, n), (row_1, col_1), colspan = width)
ax2 = plt.subplot2grid((m, n), (row_2, col_2), rowspan = height)

ax1.plot(...)
ax2.plot(...)
```
Note that the max values for row_n,col_n are m-1 and n-1 respectively, as zero indexing notation is used. Specifically addressing the question, if there were 5 total subplots, where the last subplot has twice the height as the others, we could use m=6.
```
ax1 = plt.subplot2grid((6, 1), (0, 0))
ax2 = plt.subplot2grid((6, 1), (1, 0))
ax3 = plt.subplot2grid((6, 1), (2, 0))
ax4 = plt.subplot2grid((6, 1), (3, 0))
ax5 = plt.subplot2grid((6, 1), (4, 0), rowspan=2)
plt.show()
```


Question: What is the difference between scipy.signal.spectrogram and scipy.signal.stft? The functions spicy.signal.spectrogram:https://docs.scipy.org/doc/scipy/reference/generated/scipy.signal.spectrogram.html and spicy.signal.stft:https://docs.scipy.org/doc/scipy/reference/generated/scipy.signal.stft.html seem to do a very similar thing. What is the difference between the two functions?

Response:
Tl;dr: If I write it with the ouput given by the SciPy documentation: Sxx = Zxx ** 2 Explanation: Spectrogram and Short Time Fourier Transform are two different object, yet they are really close together. The short-time Fourier transform (STFT), is a Fourier-related transform used to determine the sinusoidal frequency and phase content of local sections of a signal as it changes over time. In practice, the procedure for computing STFTs is to divide a longer time signal into shorter segments of equal length and then compute the Fourier transform separately on each shorter segment. This reveals the Fourier spectrum on each shorter segment. One then usually plots the changing spectra as a function of time. Wikipedia On the other hand, A spectrogram is a visual representation of the spectrum of frequencies of a signal as it varies with time. Wikipedia The spectrogram basically cuts your signal in small windows, and display a range of colors showing the intensity of this or that specific frequency. Exactly as the STFT. In fact it's using the STFT. Now, for the difference, by definition, the spectrogram is squared magnitude of the short-time Fourier transform (STFT) of the signal s(t): spectrogram(t, w) = |STFT(t, w)|^2 The example shown at the bottom of the scipy.signal.stft page shows:
```py

```python
plt.pcolormesh(t, f, np.abs(Zxx), vmin=0, vmax=amp)
#Output
#```
#It's working and you can see a color scale. But it's a linear one, because of the abs operation. In reality, to get the real spectrogram, one should write:
#```py
```

```python
plt.pcolormesh(t, f, Zxx ** 2, vmin=0, vmax=amp)
#Output
#```
```


Question:
numpy/scipy/ipython:Failed to interpret file as a pickle I have the file in following format:
```
0,0.104553357966
1,0.213014562052
2,0.280656379048
3,0.0654249076288
4,0.312223429689
5,0.0959008911106
6,0.114207780917
7,0.105294501195
8,0.0900673766572
9,0.23941317105
10,0.0598239513149
11,0.541701803956
12,0.093929580526
```
I want to plot these point using ipython plot function doing the following:
```

```python
mean_data = load("/Users/daydreamer/data/mean")
```
But it fails saying the following:
```
---------------------------------------------------------------------------
IOError                                   Traceback (most recent call last)
/Users/daydreamer/<ipython-input-40-8f1329559411> in <module>()
----> 1 mean_data = load("/Users/daydreamer/data/mean")

/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy-1.6.1-py2.7-macosx-10.5-fat3.egg/numpy/lib/npyio.pyc in load(file, mmap_mode)
    354             except:
    355                 raise IOError, \
--> 356                     "Failed to interpret file %s as a pickle" % repr(file)
    357     finally:
    358         if own_fid:

IOError: Failed to interpret file '/Users/daydreamer/data/mean' as a pickle
```
How do I fix this error?
```

Response:
The numpy.load routine is for loading pickled .npy or .npz binary files, which can be created using numpy.save and numpy.savez, respectively. Since you have text data, these are not the routines you want. You can load your comma-separated values with numpy.loadtxt.
```
import numpy as np
mean_data = np.loadtxt("/Users/daydreamer/data/mean", delimiter=',')
```
Full Example Here's a complete example (using StringIO to simulate the file I/O).
```
import numpy as np
import StringIO

s = """0,0.104553357966
1,0.213014562052
2,0.280656379048
3,0.0654249076288
4,0.312223429689
5,0.0959008911106
6,0.114207780917
7,0.105294501195
8,0.0900673766572
9,0.23941317105
10,0.0598239513149
11,0.541701803956
12,0.093929580526"""

st = StringIO.StringIO(s)
a = np.loadtxt(st, delimiter=',')
```
Now we have:
```

```python
a
#Output
#array([[  0.        ,   0.10455336],
#       [  1.        ,   0.21301456],
#       [  2.        ,   0.28065638],
#       [  3.        ,   0.06542491],
#       [  4.        ,   0.31222343],
#       [  5.        ,   0.09590089],
#       [  6.        ,   0.11420778],
#       [  7.        ,   0.1052945 ],
#       [  8.        ,   0.09006738],
#       [  9.        ,   0.23941317],
#       [ 10.        ,   0.05982395],
#       [ 11.        ,   0.5417018 ],
#       [ 12.        ,   0.09392958]])
#```
```


Question:
Scipy minimize: How to pass args to both the objective and the constraint My MWE is as follows
```
def obj(e, p):
    S = f(e) + g(p)
    return S
```
I would like to minimize this function over only e and pass p as an argument to the function. However, I also would like a constraint that depends on p and e that is of the form p + e < 1 I tried
```
cons = {'type': 'ineq',
       'fun': lambda e, p: -e -p + 1,
       'args': (p)}
```
And then, I try to minimize this for the case of p = 0.5
```
minimize(obj, initial_guess, method = 'SLSQP', args = 0.5, constraints = cons)
```
but this doesn't work. I get the error name 'p' is not defined in the line where I define cons. How do I pass the argument p to both the objective function and the constraint? Full code below
```
from scipy.optimize import minimize
from scipy.stats import entropy
import numpy as np

#Create a probability vector
def p_vector(x):
    v = np.array([x, 1-x])
    return v

#Write the objective function
def obj(e, p):
    S = -1*entropy(p_vector(p + e), base = 2)
    return S

##Constraints
cons = {'type': 'ineq',
       'fun': lambda e: -p - e + 1,
       'args': (p,)
       }

initial_guess = 0

result = minimize(obj, initial_guess, method = 'SLSQP', args = (0.5, ), constraints = cons)
print(result)
```

Response:
Okay, I figured that it's a mix of syntax errors on my part and how arguments should be passed. For those who may have the same question, I will post an answer here. The objective function is obj(e, p). We only want to minimize e so we create a tuple of the other arguments arguments = (0.5,). That is, a specific value of p=0.5 is set. Next define the constraint function
```
def prob_bound(e, p):
    return -e - p + 1
```
Now one writes the constraints dictionary to be
```
cons = ({'type': 'ineq',
       'fun': prob_bound,
       'args': arguments
       })
```
And finally, one calls the minimizer
```
result = minimize(obj, initial_guess, method = 'SLSQP', args = arguments, constraints = cons)
```


Question:
statsmodel AttributeError: module 'scipy.stats' has no attribute 'chisqprob' I'm running the code below with statsmodel 0.8.0 which i believe is the latest.
```
import statsmodels.api as sm
est = sm.Logit(y_train, x_train)
result = est.fit()
print(result.summary())
```
This is giving me an error saying: AttributeError: module 'scipy.stats' has no attribute 'chisqprob'. I dont seem to be able to find anything on stackoverflow or elsewhere to resolve this. Any help much appreciated.

Response:
Try this:
```
result.summary2()
```
Link: http://www.statsmodels.org/stable/generated/statsmodels.discrete.discrete_model.LogitResults.summary2.html?highlight=summary2#statsmodels.discrete.discrete_model.LogitResults.summary2