Question:
Calculating Slopes in Numpy (or Scipy) I am trying to find the fastest and most efficient way to calculate slopes using Numpy and Scipy. I have a data set of three Y variables and one X variable and I need to calculate their individual slopes. For example, I can easily do this one row at a time, as shown below, but I was hoping there was a more efficient way of doing this. I also don't think linregress is the best way to go because I don't need any of the auxiliary variables like intercept, standard error, etc in my results. Any help is greatly appreciated.
```
import numpy as np
    from scipy import stats

    Y = [[  2.62710000e+11   3.14454000e+11   3.63609000e+11   4.03196000e+11
        4.21725000e+11   2.86698000e+11   3.32909000e+11   4.01480000e+11
        4.21215000e+11   4.81202000e+11]
        [  3.11612352e+03   3.65968334e+03   4.15442691e+03   4.52470938e+03
        4.65011423e+03   3.10707392e+03   3.54692896e+03   4.20656404e+03
        4.34233412e+03   4.88462501e+03]
        [  2.21536396e+01   2.59098311e+01   2.97401268e+01   3.04784552e+01
        3.13667639e+01   2.76377113e+01   3.27846013e+01   3.73223417e+01
        3.51249997e+01   4.42563658e+01]]
    X = [ 1990.  1991.  1992.  1993.  1994.  1995.  1996.  1997.  1998.  1999.]
    slope_0, intercept, r_value, p_value, std_err = stats.linregress(X, Y[0,:])
    slope_1, intercept, r_value, p_value, std_err = stats.linregress(X, Y[1,:])
    slope_2, intercept, r_value, p_value, std_err = stats.linregress(X, Y[2,:])
    slope_0 = slope/Y[0,:][0]
    slope_1 = slope/Y[1,:][0]
    slope_2 = slope/Y[2,:][0]
    b, a = polyfit(X, Y[1,:], 1)
    slope_1_a = b/Y[1,:][0]
```

Response:
The fastest and the most efficient way would be to use a native scipy function from linregress which calculates everything: slope : slope of the regression line intercept : intercept of the regression line r-value : correlation coefficient p-value : two-sided p-value for a hypothesis test whose null hypothesis is that the slope is zero stderr : Standard error of the estimate And here is an example:
```
a = [15, 12, 8, 8, 7, 7, 7, 6, 5, 3]
b = [10, 25, 17, 11, 13, 17, 20, 13, 9, 15]
from scipy.stats import linregress
linregress(a, b)
```
will return you:
```
LinregressResult(slope=0.20833333333333337, intercept=13.375, rvalue=0.14499815458068521, pvalue=0.68940144811669501, stderr=0.50261704627083648)
```
P.S. Just a mathematical formula for slope:


Question:
How to force zero interception in linear regression? I have some more or less linear data of the form:
```
x = [0.1, 0.2, 0.4, 0.6, 0.8, 1.0, 2.0, 4.0, 6.0, 8.0, 10.0, 20.0, 40.0, 60.0, 80.0]
y = [0.50505332505407008, 1.1207373784533172, 2.1981844719020001, 3.1746209003398689, 4.2905482471260044, 6.2816226678076958, 11.073788414382639, 23.248479770546009, 32.120462301367183, 44.036117671229206, 54.009003143831116, 102.7077685684846, 185.72880217806673, 256.12183145545811, 301.97120103079675]
```
I am using scipy.optimize.leastsq to fit a linear regression to this:
```
def lin_fit(x, y):
    '''Fits a linear fit of the form mx+b to the data'''
    fitfunc = lambda params, x: params[0] * x + params[1]    #create fitting function of form mx+b
    errfunc = lambda p, x, y: fitfunc(p, x) - y              #create error function for least squares fit

    init_a = 0.5                            #find initial value for a (gradient)
    init_b = min(y)                         #find initial value for b (y axis intersection)
    init_p = numpy.array((init_a, init_b))  #bundle initial values in initial parameters

    #calculate best fitting parameters (i.e. m and b) using the error function
    p1, success = scipy.optimize.leastsq(errfunc, init_p.copy(), args = (x, y))
    f = fitfunc(p1, x)          #create a fit with those parameters
    return p1, f
```
And it works beautifully (although I am not sure if scipy.optimize is the right thing to use here, it might be a bit over the top?). However, due to the way the data points lie it does not give me a y-axis interception at 0. I do know though that it has to be zero in this case, if x = 0 than y = 0. Is there any way I can force this?

Response:
As @AbhranilDas mentioned, just use a linear method. There's no need for a non-linear solver like scipy.optimize.lstsq. Typically, you'd use numpy.polyfit to fit a line to your data, but in this case you'll need to do use numpy.linalg.lstsq directly, as you want to set the intercept to zero. As a quick example:
```
import numpy as np
import matplotlib.pyplot as plt

x = np.array([0.1, 0.2, 0.4, 0.6, 0.8, 1.0, 2.0, 4.0, 6.0, 8.0, 10.0,
              20.0, 40.0, 60.0, 80.0])

y = np.array([0.50505332505407008, 1.1207373784533172, 2.1981844719020001,
              3.1746209003398689, 4.2905482471260044, 6.2816226678076958,
              11.073788414382639, 23.248479770546009, 32.120462301367183,
              44.036117671229206, 54.009003143831116, 102.7077685684846,
              185.72880217806673, 256.12183145545811, 301.97120103079675])

# Our model is y = a * x, so things are quite simple, in this case...
# x needs to be a column vector instead of a 1D vector for this, however.
x = x[:,np.newaxis]
a, _, _, _ = np.linalg.lstsq(x, y)

plt.plot(x, y, 'bo')
plt.plot(x, a*x, 'r-')
plt.show()
```


Question:
Get coordinates of local maxima in 2D array above certain value
```
from PIL import Image
import numpy as np
from scipy.ndimage.filters import maximum_filter
import pylab

# the picture (256 * 256 pixels) contains bright spots of which I wanna get positions
# problem: data has high background around value 900 - 1000

im = Image.open('slice0000.png')
data = np.array(im)

# as far as I understand, data == maximum_filter gives True-value for pixels
# being the brightest in their neighborhood (here 10 * 10 pixels)

maxima = (data == maximum_filter(data,10))
# How can I get only maxima, outstanding the background a certain value, let's say 500 ?
```
I'm afraid I don't really understand the scipy.ndimage.filters.maximum_filter() function. Is there a way to obtain pixel-coordinates only within the spots and not within the background? https://i.sstatic.net/RImHW.png (16-bit grayscale picture, 256*256 pixels)

Response:
```
import numpy as np
import scipy
import scipy.ndimage as ndimage
import matplotlib.pyplot as plt

fname = '/tmp/slice0000.png'
neighborhood_size = 5
threshold = 1500

data = scipy.misc.imread(fname)

data_max = ndimage.maximum_filter(data, neighborhood_size) # apply maximum filter with a size of neighborhood_size
maxima = (data == data_max) # boolean mask: local maximum within neighborhood_size
data_min = ndimage.minimum_filter(data, neighborhood_size) # apply minimum filter with a size of neighborhood_size
diff = ((data_max - data_min) > threshold) # boolean mask where the difference of the filters exceeds threshold
maxima[~diff] = False # remove the local maxima which do not satisfy the minimum difference in neighborhood

labeled, num_objects = ndimage.label(maxima) # label connected components on maxima binary array (boolean mask)
slices = ndimage.find_objects(labeled) # slices are 2d rect
x, y = [], []
for dy,dx in slices:
    x_center = (dx.start + dx.stop - 1)/2
    x.append(x_center)
    y_center = (dy.start + dy.stop - 1)/2
    y.append(y_center)

plt.imshow(data)
plt.savefig('/tmp/data.png', bbox_inches = 'tight')

plt.autoscale(False)
plt.plot(x,y, 'ro')
plt.savefig('/tmp/result.png', bbox_inches = 'tight')
```
Given data.png: the above program yields result.png with threshold = 1500. Lower the threshold to pick up more local maxima: References: J.F. Sebastian counts nuclei Joe Kington finds paw prints Ivan finds local maximums


Question:
`ValueError: A value in x_new is above the interpolation range.` - what other reasons than not ascending values? I receive this error in scipy interp1d function. Normally, this error would be generated if the x was not monotonically increasing.
```
import scipy.interpolate as spi
def refine(coarsex,coarsey,step):
    finex = np.arange(min(coarsex),max(coarsex)+step,step)
    intfunc = spi.interp1d(coarsex, coarsey,axis=0)
    finey = intfunc(finex)
    return finex, finey

for num, tfile in enumerate(files):
    tfile = tfile.dropna(how='any')
    x = np.array(tfile['col1'])
    y = np.array(tfile['col2'])
    finex, finey = refine(x,y,0.01)
```
The code is correct, because it successfully worked on 6 data files and threw the error for the 7th. So there must be something wrong with the data. But as far as I can tell, the data increase all the way down. I am sorry for not providing an example, because I am not able to reproduce the error on an example. There are two things that could help me: Some brainstorming - if the data are indeed monotonically increasing, what else could produce this error? Another hint, regarding the decimals, could be in this question, but I think my solution (the min and max of x) is robust enough to avoid it. Or isn't it? Is it possible (how?) to return the value of x_new and it's index when throwing the ValueError: A value in x_new is above the interpolation range. so that I could actually see where in the file is the problem? UPDATE So the problem is that, for some reason, max(finex) is larger than max(coarsex) (one is .x39 and the other is .x4). I hoped rounding the original values to 2 significant digits would solve the problem, but it didn't, it displays fewer digits but still counts with the undisplayed. What can I do about it?

Response:
If you are running Scipy v. 0.17.0 or newer, then you can pass fill_value='extrapolate' to spi.interp1d, and it will extrapolate to accomadate these values of your's that lie outside the interpolation range. So define your interpolation function like so: intfunc = spi.interp1d(coarsex, coarsey,axis=0, fill_value="extrapolate") Be forewarned, however! Depending on what your data looks like and the type on interpolation you are performing, the extrapolated values can be erroneous. This is especially true if you have noisy or non-monotonic data. In your case you might be ok because your x_new value is only slighly beyond your interpolation range. Here's simple demonstration of how this feature can work nicely but also give erroneous results.
```
import scipy.interpolate as spi
import numpy as np

x = np.linspace(0,1,100)
y = x + np.random.randint(-1,1,100)/100
x_new = np.linspace(0,1.1,100)
intfunc = spi.interp1d(x,y,fill_value="extrapolate")
y_interp = intfunc(x_new)

import matplotlib.pyplot as plt
plt.plot(x_new,y_interp,'r', label='interp/extrap')
plt.plot(x,y, 'b--', label='data')
plt.legend()
plt.show()
```
So the interpolated portion (in red) worked well, but the extrapolated portion clearly fails to follow the otherwise linear trend in this data because of the noise. So have some understanding of your data and proceed with caution.


Question:
In Scipy how and why does curve_fit calculate the covariance of the parameter estimates I have been using scipy.optimize.leastsq to fit some data. I would like to get some confidence intervals on these estimates so I look into the cov_x output but the documentation is very unclear as to what this is and how to get the covariance matrix for my parameters from this. First of all it says that it is a Jacobian, but in the notes it also says that "cov_x is a Jacobian approximation to the Hessian" so that it is not actually a Jacobian but a Hessian using some approximation from the Jacobian. Which of these statements is correct? Secondly this sentence to me is confusing: This matrix must be multiplied by the residual variance to get the covariance of the parameter estimates – see curve_fit. I indeed go look at the source code for curve_fit where they do:
```
s_sq = (func(popt, *args)**2).sum()/(len(ydata)-len(p0))
pcov = pcov * s_sq
```
which corresponds to multiplying cov_x by s_sq but I cannot find this equation in any reference. Can someone explain why this equation is correct? My intuition tells me that it should be the other way around since cov_x is supposed to be a derivative (Jacobian or Hessian) so I was thinking: cov_x * covariance(parameters) = sum of errors(residuals) where sigma(parameters) is the thing I want. How do I connect the thing curve_fit is doing with what I see at eg. wikipedia: http://en.wikipedia.org/wiki/Propagation_of_uncertainty#Non-linear_combinations

Response: OK, I think I found the answer. First the solution: cov_x*s_sq is simply the covariance of the parameters which is what you want. Taking sqrt of the diagonal elements will give you standard deviation (but be careful about covariances!). Residual variance = reduced chi square = s_sq = sum[(f(x)-y)^2]/(N-n), where N is number of data points and n is the number of fitting parameters. Reduced chi square. The reason for my confusion is that cov_x as given by leastsq is not actually what is called cov(x) in other places rather it is the reduced cov(x) or fractional cov(x). The reason it does not show up in any of the other references is that it is a simple rescaling which is useful in numerical computations, but is not relevant for a textbook. About Hessian versus Jacobian, the documentation is poorly worded. It is the Hessian that is calculated in both cases as is obvious since the Jacobian is zero at a minimum. What they mean is that they are using an approximation to the Jacobian to find the Hessian. A further note. It seems that the curve_fit result does not actually account for the absolute size of the errors, but only take into account the relative size of the sigmas provided. This means that the pcov returned doesn't change even if the errorbars change by a factor of a million. This is of course not right, but seems to be standard practice ie. Matlab does the same thing when using their Curve fitting toolbox. The correct procedure is described here: https://en.wikipedia.org/wiki/Linear_least_squares_(mathematics)#Parameter_errors_and_correlation It seems fairly straightforward to do this once the optimum has been found, at least for Linear Least squares.


Question: Plot a plane based on a normal vector and a point in Matlab or matplotlib How would one go plotting a plane in matlab or matplotlib from a normal vector and a point?

Response:
For all the copy/pasters out there, here is similar code for Python using matplotlib:
```py
import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D

point  = np.array([1, 2, 3])
normal = np.array([1, 1, 2])

# a plane is a*x+b*y+c*z+d=0
# [a,b,c] is the normal. Thus, we have to calculate
# d and we're set
d = -point.dot(normal)

# create x,y
xx, yy = np.meshgrid(range(10), range(10))

# calculate corresponding z
z = (-normal[0] * xx - normal[1] * yy - d) * 1. /normal[2]

# plot the surface
plt3d = plt.figure().gca(projection='3d')
plt3d.plot_surface(xx, yy, z)
plt.show()
```
EDIT For newer matplotlib>3.6:
```py
import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D

point  = np.array([1, 2, 3])
normal = np.array([1, 1, 2])

# a plane is a*x+b*y+c*z+d=0
# [a,b,c] is the normal. Thus, we have to calculate
# d and we're set
d = -point.dot(normal)

# create x,y
xx, yy = np.meshgrid(range(10), range(10))

# calculate corresponding z
z = (-normal[0] * xx - normal[1] * yy - d) * 1. /normal[2]

# plot the surface
fig = plt.figure()
plt3d = fig.add_subplot(projection='3d')
plt3d.plot_surface(xx, yy, z)
plt.show()
```


Question:
Overflow Error in Python's numpy.exp function I want to use numpy.exp like this:
```
cc = np.array([
    [0.120,0.34,-1234.1]
])

print 1/(1+np.exp(-cc))
```
But this gives me error:
```
/usr/local/lib/python2.7/site-packages/ipykernel/__main__.py:5: RuntimeWarning: overflow encountered in exp
```
I can't understand why? How can I fix this? It seems the problem is with third number (-1234.1)

Response:
As fuglede says, the issue here is that np.float64 can't handle a number as large as exp(1234.1). Try using np.float128 instead:
```

```python
cc = np.array([[0.120,0.34,-1234.1]], dtype=np.float128)
```

```python
cc
#Output
#array([[ 0.12,  0.34, -1234.1]], dtype=float128)
```

```python
1 / (1 + np.exp(-cc))
#Output
#array([[ 0.52996405,  0.58419052,  1.0893812e-536]], dtype=float128)
#```
#Note however, that there are certain quirks with using extended precision. It may not work on Windows; you don't actually get the full 128 bits of precision; and you might lose the precision whenever the number passes through pure python. You can read more about the details here. For most practical purposes, you can probably approximate 1 / (1 + ) to zero. That is to say, just ignore the warning and move on. Numpy takes care of the approximation for you (when using np.float64):
#```
```

```python
1 / (1 + np.exp(-cc))
#Output
#/usr/local/bin/ipython3:1: RuntimeWarning: overflow encountered in exp
#  #!/usr/local/bin/python3.4
#array([[ 0.52996405,  0.58419052,  0.        ]])
#```
#If you want to suppress the warning, you could use scipy.special.expit, as suggested by WarrenWeckesser in a comment to the question:
#```
```

```python
from scipy.special import expit
```

```python
expit(cc)
#Output
#array([[ 0.52996405,  0.58419052,  0.        ]])
#```
```


Question:
How do I get a lognormal distribution in Python with Mu and Sigma? I have been trying to get the result of a lognormal distribution using Scipy. I already have the Mu and Sigma, so I don't need to do any other prep work. If I need to be more specific (and I am trying to be with my limited knowledge of stats), I would say that I am looking for the cumulative function (cdf under Scipy). The problem is that I can't figure out how to do this with just the mean and standard deviation on a scale of 0-1 (ie the answer returned should be something from 0-1). I'm also not sure which method from dist, I should be using to get the answer. I've tried reading the documentation and looking through SO, but the relevant questions (like this and this) didn't seem to provide the answers I was looking for. Here is a code sample of what I am working with. Thanks.
```
from scipy.stats import lognorm
stddev = 0.859455801705594
mean = 0.418749176686875
total = 37
dist = lognorm.cdf(total,mean,stddev)
```
UPDATE: So after a bit of work and a little research, I got a little further. But I still am getting the wrong answer. The new code is below. According to R and Excel, the result should be .7434, but that's clearly not what is happening. Is there a logic flaw I am missing?
```
dist = lognorm([1.744],loc=2.0785)
dist.cdf(25)  # yields=0.96374596, expected=0.7434
```
UPDATE 2: Working lognorm implementation which yields the correct 0.7434 result.
```
def lognorm(self,x,mu=0,sigma=1):
   a = (math.log(x) - mu)/math.sqrt(2*sigma**2)
   p = 0.5 + 0.5*math.erf(a)
   return p
lognorm(25,1.744,2.0785)
> 0.7434
```

Response:
I know this is a bit late (almost one year!) but I've been doing some research on the lognorm function in scipy.stats. A lot of folks seem confused about the input parameters, so I hope to help these people out. The example above is almost correct, but I found it strange to set the mean to the location ("loc") parameter - this signals that the cdf or pdf doesn't 'take off' until the value is greater than the mean. Also, the mean and standard deviation arguments should be in the form exp(Ln(mean)) and Ln(StdDev), respectively. Simply put, the arguments are (x, shape, loc, scale), with the parameter definitions below: loc - No equivalent, this gets subtracted from your data so that 0 becomes the infimum of the range of the data. scale - exp μ, where μ is the mean of the log of the variate. (When fitting, typically you'd use the sample mean of the log of the data.) shape - the standard deviation of the log of the variate. I went through the same frustration as most people with this function, so I'm sharing my solution. Just be careful because the explanations aren't very clear without a compendium of resources. For more information, I found these sources helpful: http://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.lognorm.html#scipy.stats.lognorm https://stats.stackexchange.com/questions/33036/fitting-log-normal-distribution-in-r-vs-scipy And here is an example, taken from @serv-inc 's answer, posted on this page here:
```
import math
from scipy import stats

# standard deviation of normal distribution
sigma = 0.859455801705594
# mean of normal distribution
mu = 0.418749176686875
# hopefully, total is the value where you need the cdf
total = 37

frozen_lognorm = stats.lognorm(s=sigma, scale=math.exp(mu))
frozen_lognorm.cdf(total) # use whatever function and value you need here
```


Question:
Numpy transpose of 1D array not giving expected result I am trying a very basic example in Python scipy module for transpose() method but it's not giving expected result. I am using Ipython with pylab mode.
```
a = array([1,2,3]
print a.shape
>> (3,)

b = a.transpose()
print b.shape
>> (3,)
```
If I print the contents of arrays "a" and "b", they are similar. Expectation is: (which will be result in Matlab on transpose)
```
[1,
  2,
  3]
```

Response:
NumPy's transpose() effectively reverses the shape of an array. If the array is one-dimensional, this means it has no effect. In NumPy, the arrays
```
array([1, 2, 3])
```
and
```
array([1,
       2,
       3])
```
are actually the same – they only differ in whitespace. What you probably want are the corresponding two-dimensional arrays, for which transpose() would work fine. Also consider using NumPy's matrix type:
```

```python
numpy.matrix([1, 2, 3])
#Output
#matrix([[1, 2, 3]])
```

```python
numpy.matrix([1, 2, 3]).T
#Output
#matrix([[1],
#        [2],
#        [3]])
#```
#Note that for most applications, the plain one-dimensional array would work fine as both a row or column vector, but when coming from Matlab, you might prefer using numpy.matrix.
```


Question:
Fitting a 2D Gaussian function using scipy.optimize.curve_fit - ValueError and minpack.error I intend to fit a 2D Gaussian function to images showing a laser beam to get its parameters like FWHM and position. So far I tried to understand how to define a 2D Gaussian function in Python and how to pass x and y variables to it. I've written a little script which defines that function, plots it, adds some noise to it and then tries to fit it using curve_fit. Everything seems to work except the last step in which I try to fit my model function to the noisy data. Here is my code:
```
import scipy.optimize as opt
import numpy as np
import pylab as plt

#define model function and pass independant variables x and y as a list
def twoD_Gaussian((x,y), amplitude, xo, yo, sigma_x, sigma_y, theta, offset):
    xo = float(xo)
    yo = float(yo)
    a = (np.cos(theta)**2)/(2*sigma_x**2) + (np.sin(theta)**2)/(2*sigma_y**2)
    b = -(np.sin(2*theta))/(4*sigma_x**2) + (np.sin(2*theta))/(4*sigma_y**2)
    c = (np.sin(theta)**2)/(2*sigma_x**2) + (np.cos(theta)**2)/(2*sigma_y**2)
    return offset + amplitude*np.exp( - (a*((x-xo)**2) + 2*b*(x-xo)*(y-yo) + c*((y-yo)**2)))

# Create x and y indices
x = np.linspace(0, 200, 201)
y = np.linspace(0, 200, 201)
x,y = np.meshgrid(x, y)

#create data
data = twoD_Gaussian((x, y), 3, 100, 100, 20, 40, 0, 10)

# plot twoD_Gaussian data generated above
plt.figure()
plt.imshow(data)
plt.colorbar()

# add some noise to the data and try to fit the data generated beforehand
initial_guess = (3,100,100,20,40,0,10)

data_noisy = data + 0.2*np.random.normal(size=len(x))

popt, pcov = opt.curve_fit(twoD_Gaussian, (x,y), data_noisy, p0 = initial_guess)
```
Here is the error message I get when running the script using winpython 64-bit Python 2.7:
```
ValueError: object too deep for desired array
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Python\WinPython-64bit-2.7.6.2\python-2.7.6.amd64\lib\site-packages\spyderlib\widgets\externalshell\sitecustomize.py", line 540, in runfile
    execfile(filename, namespace)
  File "E:/Work Computer/Software/Python/Fitting scripts/2D Gaussian function fit/2D_Gaussian_LevMarq_v2.py", line 39, in <module>
    popt, pcov = opt.curve_fit(twoD_Gaussian, (x,y), data_noisy, p0 = initial_guess)
  File "C:\Python\WinPython-64bit-2.7.6.2\python-2.7.6.amd64\lib\site-packages\scipy\optimize\minpack.py", line 533, in curve_fit
    res = leastsq(func, p0, args=args, full_output=1, **kw)
  File "C:\Python\WinPython-64bit-2.7.6.2\python-2.7.6.amd64\lib\site-packages\scipy\optimize\minpack.py", line 378, in leastsq
    gtol, maxfev, epsfcn, factor, diag)
minpack.error: Result from function call is not a proper array of floats.
```
What is it that am I doing wrong? Is it how I pass the independent variables to the model function/curve_fit?

Response:
The output of twoD_Gaussian needs to be 1D. What you can do is add a .ravel() onto the end of the last line, like this:
```
def twoD_Gaussian(xy, amplitude, xo, yo, sigma_x, sigma_y, theta, offset):
    x, y = xy
    xo = float(xo)
    yo = float(yo)
    a = (np.cos(theta)**2)/(2*sigma_x**2) + (np.sin(theta)**2)/(2*sigma_y**2)
    b = -(np.sin(2*theta))/(4*sigma_x**2) + (np.sin(2*theta))/(4*sigma_y**2)
    c = (np.sin(theta)**2)/(2*sigma_x**2) + (np.cos(theta)**2)/(2*sigma_y**2)
    g = offset + amplitude*np.exp( - (a*((x-xo)**2) + 2*b*(x-xo)*(y-yo)
                            + c*((y-yo)**2)))
    return g.ravel()
```
You'll obviously need to reshape the output for plotting, e.g:
```
# Create x and y indices
x = np.linspace(0, 200, 201)
y = np.linspace(0, 200, 201)
x, y = np.meshgrid(x, y)

#create data
data = twoD_Gaussian((x, y), 3, 100, 100, 20, 40, 0, 10)

# plot twoD_Gaussian data generated above
plt.figure()
plt.imshow(data.reshape(201, 201))
plt.colorbar()
```
Do the fitting as before:
```
# add some noise to the data and try to fit the data generated beforehand
initial_guess = (3,100,100,20,40,0,10)

data_noisy = data + 0.2*np.random.normal(size=data.shape)

popt, pcov = opt.curve_fit(twoD_Gaussian, (x, y), data_noisy, p0=initial_guess)
```
And plot the results:
```
data_fitted = twoD_Gaussian((x, y), *popt)

fig, ax = plt.subplots(1, 1)
#ax.hold(True) For older versions. This has now been deprecated and later removed
ax.imshow(data_noisy.reshape(201, 201), cmap=plt.cm.jet, origin='lower',
    extent=(x.min(), x.max(), y.min(), y.max()))
ax.contour(x, y, data_fitted.reshape(201, 201), 8, colors='w')
plt.show()
```


Question:
Check if two scipy.sparse.csr_matrix are equal I want to check if two csr_matrix are equal. If I do:
```
x.__eq__(y)
```
I get:
```
raise ValueError("The truth value of an array with more than one "
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all().
```
This, However, works well:
```
assert (z in x for z in y)
```
Is there a better way to do it? maybe using some scipy optimized function instead? Thanks so much

Response:
Can we assume they are the same shape?
```

```python
a=sparse.csr_matrix([[0,1],[1,0]])
```

```python
b=sparse.csr_matrix([[0,1],[1,1]])
```

```python
(a!=b).nnz==0
#Output
#False
#```
#This checks the sparsity of the inequality array. It will give you an efficiency warning if you try a==b (at least the 1st time you use it). That's because it has to test all those zeros. It can't take much advantage of the sparsity. You need a relatively recent version to use logical operators like this. Were you trying to use x.__eq__(y) in some if expression, or did you get error from just that expression? In general you probably want to check several parameters first. Same shape, same nnz, same dtype. You need to be careful with floats. For dense arrays np.allclose is a good way of testing equality. And if the sparse arrays aren't too large, that might be good as well
#```
#np.allclose(a.A, b.A)
#```
#allclose uses all(less_equal(abs(x-y), atol + rtol * abs(y))). You can use a-b, but I suspect that this too will give an efficiecy warning.
```


Question:
How to correctly use scipy's skew and kurtosis functions? The skewness is a parameter to measure the symmetry of a data set and the kurtosis to measure how heavy its tails are compared to a normal distribution, see for example here. scipy.stats provides an easy way to calculate these two quantities, see scipy.stats.kurtosis and scipy.stats.skew. In my understanding, the skewness and kurtosis of a normal distribution should both be 0 using the functions just mentioned. That is, however, not the case with my code:
```
import numpy as np
from scipy.stats import kurtosis
from scipy.stats import skew

x = np.linspace( -5, 5, 1000 )
y = 1./(np.sqrt(2.*np.pi)) * np.exp( -.5*(x)**2  )  # normal distribution

print( 'excess kurtosis of normal distribution (should be 0): {}'.format( kurtosis(y) ))
print( 'skewness of normal distribution (should be 0): {}'.format( skew(y) ))
```
The output is: excess kurtosis of normal distribution (should be 0): -0.307393087742 skewness of normal distribution (should be 0): 1.11082371392 What am I doing wrong ? The versions I am using are
```
python: 2.7.6
scipy : 0.17.1
numpy : 1.12.1
```

Response:
These functions calculate moments of the probability density distribution (that's why it takes only one parameter) and doesn't care about the "functional form" of the values. These are meant for "random datasets" (think of them as measures like mean, standard deviation, variance):
```
import numpy as np
from scipy.stats import kurtosis, skew

x = np.random.normal(0, 2, 10000)   # create random values based on a normal distribution

print( 'excess kurtosis of normal distribution (should be 0): {}'.format( kurtosis(x) ))
print( 'skewness of normal distribution (should be 0): {}'.format( skew(x) ))
```
which gives:
```
excess kurtosis of normal distribution (should be 0): -0.024291887786943356
skewness of normal distribution (should be 0): 0.009666157036010928
```
changing the number of random values increases the accuracy:
```
x = np.random.normal(0, 2, 10000000)
```
Leading to:
```
excess kurtosis of normal distribution (should be 0): -0.00010309478605163847
skewness of normal distribution (should be 0): -0.0006751744848755031
```
In your case the function "assumes" that each value has the same "probability" (because the values are equally distributed and each value occurs only once) so from the point of view of skew and kurtosis it's dealing with a non-gaussian probability density (not sure what exactly this is) which explains why the resulting values aren't even close to 0:
```
import numpy as np
from scipy.stats import kurtosis, skew

x_random = np.random.normal(0, 2, 10000)

x = np.linspace( -5, 5, 10000 )
y = 1./(np.sqrt(2.*np.pi)) * np.exp( -.5*(x)**2  )  # normal distribution

import matplotlib.pyplot as plt

f, (ax1, ax2) = plt.subplots(1, 2)
ax1.hist(x_random, bins='auto')
ax1.set_title('probability density (random)')
ax2.hist(y, bins='auto')
ax2.set_title('(your dataset)')
plt.tight_layout()
```


Question:
Reverse Box-Cox transformation I am using SciPy's boxcox function to perform a Box-Cox transformation on a continuous variable.
```
from scipy.stats import boxcox
import numpy as np
y = np.random.random(100)
y_box, lambda_ = ss.boxcox(y + 1) # Add 1 to be able to transform 0 values
```
Then, I fit a statistical model to predict the values of this Box-Cox transformed variable. The model predictions are in the Box-Cox scale and I want to transform them to the original scale of the variable.
```
from sklearn.ensemble import RandomForestRegressor
rf = RandomForestRegressor()
X = np.random.random((100, 100))
rf.fit(X, y_box)
pred_box = rf.predict(X)
```
However, I can't find a SciPy function that performs a reverse Box-Cox transformation given transformed data and lambda. Is there such a function? I coded an inverse transformation for now.
```
pred_y = np.power((y_box * lambda_) + 1, 1 / lambda_) - 1
```

Response:
SciPy has added an inverse Box-Cox transformation. https://docs.scipy.org/doc/scipy/reference/generated/scipy.special.inv_boxcox.html scipy.special.inv_boxcox scipy.special.inv_boxcox(y, lmbda) = Compute the inverse of the Box-Cox transformation. Find x such that:
```
y = (x**lmbda - 1) / lmbda  if lmbda != 0
    log(x)                  if lmbda == 0
```
Parameters: y : array_like Data to be transformed. lmbda : array_like Power parameter of the Box-Cox transform. Returns: x : array Transformed data. Notes New in version 0.16.0. Example:
```
from scipy.special import boxcox, inv_boxcox
y = boxcox([1, 4, 10], 2.5)
inv_boxcox(y, 2.5)

output: array([1., 4., 10.])
```


Question:
Populate a Pandas SparseDataFrame from a SciPy Sparse Matrix I noticed Pandas now has support for Sparse Matrices and Arrays. Currently, I create DataFrame()s like this:
```
return DataFrame(matrix.toarray(), columns=features, index=observations)
```
Is there a way to create a SparseDataFrame() with a scipy.sparse.csc_matrix() or csr_matrix()? Converting to dense format kills RAM badly. Thanks!

Response:
A direct conversion is not supported ATM. Contributions are welcome! Try this, should be ok on memory as the SpareSeries is much like a csc_matrix (for 1 column) and pretty space efficient
```

```python
col = np.array([0,0,1,2,2,2])
```

```python
data = np.array([1,2,3,4,5,6],dtype='float64')
```

```python
m = csc_matrix( (data,(row,col)), shape=(3,3) )
```

```python
m
#Output
#<3x3 sparse matrix of type '<type 'numpy.float64'>'
#        with 6 stored elements in Compressed Sparse Column format>
```

```python
pd.SparseDataFrame([ pd.SparseSeries(m[i].toarray().ravel())
                              for i in np.arange(m.shape[0]) ])
#Output
#0  1  2
#0  1  0  4
#1  0  0  5
#2  2  3  6
```

```python
df = pd.SparseDataFrame([ pd.SparseSeries(m[i].toarray().ravel())
                                   for i in np.arange(m.shape[0]) ])
```

```python
type(df)
#Output
#pandas.sparse.frame.SparseDataFrame
#```
```


Question:
Python out of memory on large CSV file (numpy) I have a 3GB CSV file that I try to read with python, I need the median column wise.
```
from numpy import *
def data():
    return genfromtxt('All.csv',delimiter=',')

data = data() # This is where it fails already.

med = zeros(len(data[0]))
data = data.T
for i in xrange(len(data)):
    m = median(data[i])
    med[i] = 1.0/float(m)
print med
```
The error that I get is this:
```
Python(1545) malloc: *** mmap(size=16777216) failed (error code=12)

*** error: can't allocate region

*** set a breakpoint in malloc_error_break to debug

Traceback (most recent call last):

  File "Normalize.py", line 40, in <module>

  data = data()

  File "Normalize.py", line 39, in data

  return genfromtxt('All.csv',delimiter=',')

File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-
packages/numpy/lib/npyio.py", line 1495, in genfromtxt

for (i, line) in enumerate(itertools.chain([first_line, ], fhd)):

MemoryError
```
I think it's just an out of memory error. I am running a 64bit MacOSX with 4GB of ram and both numpy and Python compiled in 64bit mode. How do I fix this? Should I try a distributed approach, just for the memory management? Thanks EDIT: Also tried with this but no luck...
```
genfromtxt('All.csv',delimiter=',', dtype=float16)
```

Response:
As other folks have mentioned, for a really large file, you're better off iterating. However, you do commonly want the entire thing in memory for various reasons. genfromtxt is much less efficient than loadtxt (though it handles missing data, whereas loadtxt is more "lean and mean", which is why the two functions co-exist). If your data is very regular (e.g. just simple delimited rows of all the same type), you can also improve on either by using numpy.fromiter. If you have enough ram, consider using np.loadtxt('yourfile.txt', delimiter=',') (You may also need to specify skiprows if you have a header on the file.) As a quick comparison, loading ~500MB text file with loadtxt uses ~900MB of ram at peak usage, while loading the same file with genfromtxt uses ~2.5GB. Loadtxt Genfromtxt Alternately, consider something like the following. It will only work for very simple, regular data, but it's quite fast. (loadtxt and genfromtxt do a lot of guessing and error-checking. If your data is very simple and regular, you can improve on them greatly.)
```
import numpy as np

def generate_text_file(length=1e6, ncols=20):
    data = np.random.random((length, ncols))
    np.savetxt('large_text_file.csv', data, delimiter=',')

def iter_loadtxt(filename, delimiter=',', skiprows=0, dtype=float):
    def iter_func():
        with open(filename, 'r') as infile:
            for _ in range(skiprows):
                next(infile)
            for line in infile:
                line = line.rstrip().split(delimiter)
                for item in line:
                    yield dtype(item)
        iter_loadtxt.rowlength = len(line)

    data = np.fromiter(iter_func(), dtype=dtype)
    data = data.reshape((-1, iter_loadtxt.rowlength))
    return data

#generate_text_file()
data = iter_loadtxt('large_text_file.csv')
```
Fromiter


Question:
Calling statistics functions from Scipy This may well be completely trivial. I want to call the spearmanr function from scipy: (the data are just examples)
```
import scipy
from numpy import *

Len = [2,3,5,7,2]
Pop = [5,2,6,3,2]

a = array(Len)
b = array(Pop)

print scipy.stats.spearmanr(a,b)
```
This generates the error:
```
AttributeError: 'module' object has no attribute 'spearmanr'
```
What am I doing wrong? Thanks

Response: Use import scipy.stats. Then it works. Importing a package does not automatically import all the subpackages/modules. In these cases you have to do this explicitly.


Question:
fitting exponential decay with no initial guessing Does anyone know a scipy/numpy module which will allow to fit exponential decay to data? Google search returned a few blog posts, for example - http://exnumerus.blogspot.com/2010/04/how-to-fit-exponential-decay-example-in.html , but that solution requires y-offset to be pre-specified, which is not always possible EDIT: curve_fit works, but it can fail quite miserably with no initial guess for parameters, and that is sometimes needed. The code I'm working with is
```
#!/usr/bin/env python
import numpy as np
import scipy as sp
import pylab as pl
from scipy.optimize.minpack import curve_fit

x = np.array([  50.,  110.,  170.,  230.,  290.,  350.,  410.,  470.,
530.,  590.])
y = np.array([ 3173.,  2391.,  1726.,  1388.,  1057.,   786.,   598.,
443.,   339.,   263.])

smoothx = np.linspace(x[0], x[-1], 20)

guess_a, guess_b, guess_c = 4000, -0.005, 100
guess = [guess_a, guess_b, guess_c]

exp_decay = lambda x, A, t, y0: A * np.exp(x * t) + y0

params, cov = curve_fit(exp_decay, x, y, p0=guess)

A, t, y0 = params

print "A = %s\nt = %s\ny0 = %s\n" % (A, t, y0)

pl.clf()
best_fit = lambda x: A * np.exp(t * x) + y0

pl.plot(x, y, 'b.')
pl.plot(smoothx, best_fit(smoothx), 'r-')
pl.show()
```
which works, but if we remove "p0=guess", it fails miserably.

Response:
You have two options: Linearize the system, and fit a line to the log of the data. Use a non-linear solver (e.g. scipy.optimize.curve_fit The first option is by far the fastest and most robust. However, it requires that you know the y-offset a-priori, otherwise it's impossible to linearize the equation. (i.e. y = A * exp(K * t) can be linearized by fitting y = log(A * exp(K * t)) = K * t + log(A), but y = A*exp(K*t) + C can only be linearized by fitting y - C = K*t + log(A), and as y is your independent variable, C must be known beforehand for this to be a linear system. If you use a non-linear method, it's a) not guaranteed to converge and yield a solution, b) will be much slower, c) gives a much poorer estimate of the uncertainty in your parameters, and d) is often much less precise. However, a non-linear method has one huge advantage over a linear inversion: It can solve a non-linear system of equations. In your case, this means that you don't have to know C beforehand. Just to give an example, let's solve for y = A * exp(K * t) with some noisy data using both linear and nonlinear methods:
```
import numpy as np
import matplotlib.pyplot as plt
import scipy as sp
import scipy.optimize

def main():
    # Actual parameters
    A0, K0, C0 = 2.5, -4.0, 2.0

    # Generate some data based on these
    tmin, tmax = 0, 0.5
    num = 20
    t = np.linspace(tmin, tmax, num)
    y = model_func(t, A0, K0, C0)

    # Add noise
    noisy_y = y + 0.5 * (np.random.random(num) - 0.5)

    fig = plt.figure()
    ax1 = fig.add_subplot(2,1,1)
    ax2 = fig.add_subplot(2,1,2)

    # Non-linear Fit
    A, K, C = fit_exp_nonlinear(t, noisy_y)
    fit_y = model_func(t, A, K, C)
    plot(ax1, t, y, noisy_y, fit_y, (A0, K0, C0), (A, K, C0))
    ax1.set_title('Non-linear Fit')

    # Linear Fit (Note that we have to provide the y-offset ("C") value!!
    A, K = fit_exp_linear(t, y, C0)
    fit_y = model_func(t, A, K, C0)
    plot(ax2, t, y, noisy_y, fit_y, (A0, K0, C0), (A, K, 0))
    ax2.set_title('Linear Fit')

    plt.show()

def model_func(t, A, K, C):
    return A * np.exp(K * t) + C

def fit_exp_linear(t, y, C=0):
    y = y - C
    y = np.log(y)
    K, A_log = np.polyfit(t, y, 1)
    A = np.exp(A_log)
    return A, K

def fit_exp_nonlinear(t, y):
    opt_parms, parm_cov = sp.optimize.curve_fit(model_func, t, y, maxfev=1000)
    A, K, C = opt_parms
    return A, K, C

def plot(ax, t, y, noisy_y, fit_y, orig_parms, fit_parms):
    A0, K0, C0 = orig_parms
    A, K, C = fit_parms

    ax.plot(t, y, 'k--',
      label='Actual Function:\n $y = %0.2f e^{%0.2f t} + %0.2f$' % (A0, K0, C0))
    ax.plot(t, fit_y, 'b-',
      label='Fitted Function:\n $y = %0.2f e^{%0.2f t} + %0.2f$' % (A, K, C))
    ax.plot(t, noisy_y, 'ro')
    ax.legend(bbox_to_anchor=(1.05, 1.1), fancybox=True, shadow=True)

if __name__ == '__main__':
    main()
```
Note that the linear solution provides a result much closer to the actual values. However, we have to provide the y-offset value in order to use a linear solution. The non-linear solution doesn't require this a-priori knowledge.


Question: python built-in function to do matrix reduction Does python have a built-in function that converts a matrix into row echelon form (also known as upper triangular)?

Response:
If you can use sympy, Matrix.rref() can do it:
```

```python
sympy.Matrix(np.random.random((4,4))).rref()
#Output
#([1, 1.42711055402454e-17, 0, -1.38777878078145e-17]
#[0,                  1.0, 0,  2.22044604925031e-16]
#[0, -2.3388341405089e-16, 1, -2.22044604925031e-16]
#[0, 3.65674099486992e-17, 0,                   1.0],
# [0, 1, 2, 3])
#```
```


Question: Computing the correlation coefficient between two multi-dimensional arrays I have two arrays that have the shapes N X T and M X T. I'd like to compute the correlation coefficient across T between every possible pair of rows n and m (from N and M, respectively). What's the fastest, most pythonic way to do this? (Looping over N and M would seem to me to be neither fast nor pythonic.) I'm expecting the answer to involve numpy and/or scipy. Right now my arrays are numpy arrays, but I'm open to converting them to a different type. I'm expecting my output to be an array with the shape N X M. N.B. When I say "correlation coefficient," I mean the Pearson product-moment correlation coefficient. Here are some things to note: The numpy function correlate requires input arrays to be one-dimensional. The numpy function corrcoef accepts two-dimensional arrays, but they must have the same shape. The scipy.stats function pearsonr requires input arrays to be one-dimensional.

Response:
Correlation (default 'valid' case) between two 2D arrays: You can simply use matrix-multiplication np.dot like so -
```
out = np.dot(arr_one,arr_two.T)
```
Correlation with the default "valid" case between each pairwise row combinations (row1,row2) of the two input arrays would correspond to multiplication result at each (row1,row2) position. Row-wise Correlation Coefficient calculation for two 2D arrays:
```
def corr2_coeff(A, B):
    # Rowwise mean of input arrays & subtract from input arrays themeselves
    A_mA = A - A.mean(1)[:, None]
    B_mB = B - B.mean(1)[:, None]

    # Sum of squares across rows
    ssA = (A_mA**2).sum(1)
    ssB = (B_mB**2).sum(1)

    # Finally get corr coeff
    return np.dot(A_mA, B_mB.T) / np.sqrt(np.dot(ssA[:, None],ssB[None]))
```
This is based upon this solution to How to apply corr2 functions in Multidimentional arrays in MATLAB Benchmarking This section compares runtime performance with the proposed approach against generate_correlation_map & loopy pearsonr based approach listed in the other answer.(taken from the function test_generate_correlation_map() without the value correctness verification code at the end of it). Please note the timings for the proposed approach also include a check at the start to check for equal number of columns in the two input arrays, as also done in that other answer. The runtimes are listed next. Case #1:
```

```python
A = np.random.rand(1000, 100)
```

```python
B = np.random.rand(1000, 100)
```

```python
%timeit corr2_coeff(A, B)
100 loops, best of 3: 15 ms per loop
```

```python
%timeit generate_correlation_map(A, B)
100 loops, best of 3: 19.6 ms per loop
```
Case #2:
```
```

```python
A = np.random.rand(5000, 100)
```

```python
B = np.random.rand(5000, 100)
```

```python
%timeit corr2_coeff(A, B)
1 loops, best of 3: 368 ms per loop
```

```python
%timeit generate_correlation_map(A, B)
1 loops, best of 3: 493 ms per loop
```
Case #3:
```
```

```python
A = np.random.rand(10000, 10)
```

```python
B = np.random.rand(10000, 10)
```

```python
%timeit corr2_coeff(A, B)
1 loops, best of 3: 1.29 s per loop
```

```python
%timeit generate_correlation_map(A, B)
1 loops, best of 3: 1.83 s per loop
```
The other loopy pearsonr based approach seemed too slow, but here are the runtimes for one small datasize -
```
```

```python
A = np.random.rand(1000, 100)
```

```python
B = np.random.rand(1000, 100)
```

```python
%timeit corr2_coeff(A, B)
100 loops, best of 3: 15.3 ms per loop
```

```python
%timeit generate_correlation_map(A, B)
100 loops, best of 3: 19.7 ms per loop
```

```python
%timeit pearsonr_based(A, B)
1 loops, best of 3: 33 s per loop
```
```


Question:
Exponential curve fitting in SciPy I have two NumPy arrays x and y. When I try to fit my data using exponential function and curve_fit (SciPy) with this simple code
```
#!/usr/bin/env python
from pylab import *
from scipy.optimize import curve_fit

x = np.array([399.75, 989.25, 1578.75, 2168.25, 2757.75, 3347.25, 3936.75, 4526.25, 5115.75, 5705.25])
y = np.array([109,62,39,13,10,4,2,0,1,2])

def func(x, a, b, c, d):
    return a*np.exp(b-c*x)+d

popt, pcov = curve_fit(func, x, y)
```
I get wrong coefficients popt
```
[a,b,c,d] = [1., 1., 1., 24.19999988]
```
What is the problem?

Response:
First comment: since a*exp(b - c*x) = (a*exp(b))*exp(-c*x) = A*exp(-c*x), a or b is redundant. I'll drop b and use:
```
import matplotlib.pyplot as plt

def func(x, a, c, d):
    return a*np.exp(-c*x)+d
```
That isn't the main issue. The problem is simply that curve_fit fails to converge to a solution to this problem when you use the default initial guess (which is all 1s). Check pcov; you'll see that it is inf. This is not surprising, because if c is 1, most of the values of exp(-c*x) underflow to 0:
```

```python
np.exp(-x)
#Output
#array([  2.45912644e-174,   0.00000000e+000,   0.00000000e+000,
#         0.00000000e+000,   0.00000000e+000,   0.00000000e+000,
#         0.00000000e+000,   0.00000000e+000,   0.00000000e+000,
#         0.00000000e+000])
#```
#This suggests that c should be small. A better initial guess is, say, p0 = (1, 1e-6, 1). Then I get:
#```
```

```python
popt, pcov = curve_fit(func, x, y, p0=(1, 1e-6, 1))
```

```python
popt
#Output
#array([  1.63561656e+02,   9.71142196e-04,  -1.16854450e+00])
#```
#This looks reasonable:
#```
```

```python
xx = np.linspace(300, 6000, 1000)
```

```python
yy = func(xx, *popt)
```

```python
plt.plot(x, y, 'ko')
#Output
#[<matplotlib.lines.Line2D at 0x41c5ad0>]
```

```python
plt.plot(xx, yy)
#Output
#[<matplotlib.lines.Line2D at 0x41c5c10>]
#```
```


Question:
scipy.special import issue I have an issue with importing the scipy.special package. It isn't harmful, just annoying/interesting. When I import scipy using import scipy as sp and then try to access sp.special I get:
```

```python
import scipy as sp
```

```python
sp.special
#Output
#Traceback (most recent call last):
#  File "<stdin>", line 1, in <module>
#AttributeError: 'module' object has no attribute 'special'
#>>>
#```
#but if I then do import scipy.special I can access the special module through scipy.special and sp.special:
#```
```

```python
import scipy as sp
```

```python
import scipy.special
```

```python
scipy.special
#Output
#<module 'scipy.special' from 'C:\Python27\lib\site-packages\scipy\special\__init__.pyc'>
```

```python
sp.special
#Output
#<module 'scipy.special' from 'C:\Python27\lib\site-packages\scipy\special\__init__.pyc'>
#>>>
#```
#So I now have the special module accessible through both sp and scipy namespaces. The interesting bit is that I can access the rest of scipy through the scipy namespace. First question: Why does the special module not import first time round? Second question: How can I get access to the special module through the sp namespace only, without defining the scipy namespace? Edit: using Python 2.7.2 and scipy 0.10.1
```

Response: By default, "import scipy" does not import any subpackage. There are too many subpackages with large Fortran extension modules that are slow to load. I do not recommend doing import scipy or the abbreviated import scipy as sp. It's just not very useful. Use from scipy import special, from scipy import linalg, etc.


Question:
How to find last occurrence of maximum value in a numpy.ndarray I have a numpy.ndarray in which the maximum value will mostly occur more than once. EDIT: This is subtly different from numpy.argmax: how to get the index corresponding to the *last* occurrence, in case of multiple occurrences of the maximum values because the author says Or, even better, is it possible to get a list of indices of all the occurrences of the maximum value in the array? whereas in my case getting such a list may prove very expensive Is it possible to find the index of the last occurrence of the maximum value by using something like numpy.argmax? I want to find only the index of the last occurrence, not an array of all occurrences (since several hundreds may be there) For example this will return the index of the first occurrence ie 2
```
import numpy as np
a=np.array([0,0,4,4,4,4,2,2,2,2])
print np.argmax(a)
```
However I want it to output 5.

Response:
numpy.argmax only returns the index of the first occurrence. You could apply argmax to a reversed view of the array:
```
import numpy as np
a = np.array([0,0,4,4,4,4,2,2,2,2])
b = a[::-1]
i = len(b) - np.argmax(b) - 1
i     # 5
a[i:] # array([4, 2, 2, 2, 2])
```
Note numpy doesn't copy the array but instead creates a view of the original with a stride that accesses it in reverse order.
```
id(a) == id(b.base) # True
```


Question: how to annotate heatmap with text in matplotlib I am plotting a heatmap in matplotlib using: plt.pcolor(rand(5,5)) how can I annotate the heatmap with the actual numbers plotted? meaning in each cell of the plotted heatmap, put the value corresponding to that cell in the 5x5 matrix passed to pcolor. thanks.

Response:
There is no automatic feature to do such a thing, but you could loop through each point and put text in the appropriate location:
```
import matplotlib.pyplot as plt
import numpy as np

data = np.random.rand(5, 4)
heatmap = plt.pcolor(data)

for y in range(data.shape[0]):
    for x in range(data.shape[1]):
        plt.text(x + 0.5, y + 0.5, '%.4f' % data[y, x],
                 horizontalalignment='center',
                 verticalalignment='center',
                 )

plt.colorbar(heatmap)

plt.show()
```
HTH


Question:
Scipy/Numpy FFT Frequency Analysis I'm looking for how to turn the frequency axis in a fft (taken via scipy.fftpack.fftfreq) into a frequency in Hertz, rather than bins or fractional bins. I tried to code below to test out the FFT:
```
t = scipy.linspace(0,120,4000)
acc = lambda t: 10*scipy.sin(2*pi*2.0*t) + 5*scipy.sin(2*pi*8.0*t) + 2*scipy.random.random(len(t))

signal = acc(t)

FFT = abs(scipy.fft(signal))
FFT = scipy.fftpack.fftshift(FFT)
freqs = scipy.fftpack.fftfreq(signal.size)

pylab.plot(freqs,FFT,'x')
pylab.show()
```
The sampling rate should be 4000 samples / 120 seconds = 33.34 samples/sec. The signal has a 2.0 Hz signal, a 8.0 Hz signal, and some random noise. I take the FFT, grab the frequencies, and plot it. The numbers are pretty nonsensical. If I multiply the frequencies by 33.34 (the sampling frequency), then I get peaks at about 8 Hz and 15 Hz, which seems wrong (also, the frequencies should be a factor of 4 apart, not 2!). Any thoughts on what I'm doing wrong here?

Response:
I think you don't need to do fftshift(), and you can pass sampling period to fftfreq():
```
import scipy
import scipy.fftpack
import pylab
from scipy import pi
t = scipy.linspace(0,120,4000)
acc = lambda t: 10*scipy.sin(2*pi*2.0*t) + 5*scipy.sin(2*pi*8.0*t) + 2*scipy.random.random(len(t))

signal = acc(t)

FFT = abs(scipy.fft(signal))
freqs = scipy.fftpack.fftfreq(signal.size, t[1]-t[0])

pylab.subplot(211)
pylab.plot(t, signal)
pylab.subplot(212)
pylab.plot(freqs,20*scipy.log10(FFT),'x')
pylab.show()
```
from the graph you can see there are two peak at 2Hz and 8Hz.


Question:
Restrict scipy.optimize.minimize to integer values I'm using scipy.optimize.minimize to optimize a real-world problem for which the answers can only be integers. My current code looks like this:
```
from scipy.optimize import minimize

def f(x):
    return (481.79/(5+x[0]))+(412.04/(4+x[1]))+(365.54/(3+x[2]))+(375.88/(3+x[3]))+(379.75/(3+x[4]))+(632.92/(5+x[5]))+(127.89/(1+x[6]))+(835.71/(6+x[7]))+(200.21/(1+x[8]))

def con(x):
    return sum(x)-7

cons = {'type':'eq', 'fun': con}

print scipy.optimize.minimize(f, [1,1,1,1,1,1,1,0,0], constraints=cons, bounds=([0,7],[0,7],[0,7],[0,7],[0,7],[0,7],[0,7],[0,7],[0,7]))
```
This yields:
```
x: array([  2.91950510e-16,   2.44504019e-01,   9.97850733e-01,
     1.05398840e+00,   1.07481251e+00,   2.60570253e-01,
     1.36470363e+00,   4.48527831e-02,   1.95871767e+00]
```
But I want it optimized with integer values (rounding all x to the nearest whole number doesn't always give the minimum). Is there a way to use scipy.optimize.minimize with only integer values? (I guess I could create an array with all possible permutations of x and evaluate f(x) for each combination, but that doesn't seem like a very elegant or quick solution.)

Response:
pulp solution After some research, I don't think your objective function is linear. I recreated the problem in the Python pulp library but pulp doesn't like that we're dividing by a float and 'LpAffineExpression'. This answer suggests that linear programming "doesn't understand divisions" but that comment is in context of adding constraints, not the objective function. That comment pointed me to "Mixed Integer Linear Fractional Programming (MILFP)" and on Wikipedia. Here's how you could do it in pulp if it actually worked (maybe someone can figure out why):
```
import pulp

data = [(481.79, 5), (412.04, 4), (365.54, 3)] #, (375.88, 3), (379.75, 3), (632.92, 5), (127.89, 1), (835.71, 6), (200.21, 1)]
x = pulp.LpVariable.dicts('x', range(len(data)), lowBound=0, upBound=7, cat=pulp.LpInteger)

numerator = dict((i,tup[0]) for i,tup in enumerate(data))
denom_int = dict((i,tup[1]) for i,tup in enumerate(data))

problem = pulp.LpProblem('Mixed Integer Linear Programming', sense=pulp.LpMinimize)

# objective function (doesn't work)
# TypeError: unsupported operand type(s) for /: 'float' and 'LpAffineExpression'
problem += sum([numerator[i] / (denom_int[i] + x[i]) for i in range(len(data))])

problem.solve()

for v in problem.variables():
  print(v.name, "=", v.varValue)
```
brute solution with scipy.optimize You can use brute and ranges of slices for each x in your function. If you have 3 xs in your function, you'll also have 3 slices in your ranges tuple. The key to all of this is to add the step size of 1 to the slice(start, stop,step) so slice(#, #, 1).
```
from scipy.optimize import brute
import itertools

def f(x):
  return (481.79/(5+x[0]))+(412.04/(4+x[1]))+(365.54/(3+x[2]))

ranges = (slice(0, 9, 1),) * 3
result = brute(f, ranges, disp=True, finish=None)
print(result)
```
itertools solution Or you can use itertools to generate all combinations:
```
combinations = list(itertools.product(*[[0,1,2,3,4,5,6,7,8]]*3))

values = []
for combination in combinations:
  values.append((combination, f(combination)))

best = [c for c,v in values if v == min([v for c,v in values])]
print(best)
```
Note: this is a scaled-down version of your original function for example purposes.


Question:
Fitting to Poisson histogram I am trying to fit a curve over the histogram of a Poisson distribution that looks like this I have modified the fit function so that it resembles a Poisson distribution, with the parameter t as a variable. But the curve_fit function can not be plotted and I am not sure why.
```
def histo(bsize):
    N = bsize
    #binwidth
    bw = (dt.max()-dt.min())/(N-1.)
    bin1 = dt.min()+ bw*np.arange(N)
    #define the array to hold the occurrence count
    bincount= np.array([])
    for bin in bin1:
        count = np.where((dt>=bin)&(dt<bin+bw))[0].size
        bincount = np.append(bincount,count)
    #bin center
    binc = bin1+0.5*bw
    plt.figure()
    plt.plot(binc,bincount,drawstyle= 'steps-mid')
    plt.xlabel("Interval[ticks]")
    plt.ylabel("Frequency")
histo(30)
plt.xlim(0,.5e8)
plt.ylim(0,25000)
import numpy as np
from scipy.optimize import curve_fit
delta_t = 1.42e7
def func(x, t):
    return t * np.exp(- delta_t/t)
popt, pcov = curve_fit(func, np.arange(0,.5e8),histo(30))
plt.plot(popt)
```

Response:
The problem with your code is that you do not know what the return values of curve_fit are. It is the parameters for the fit-function and their covariance matrix - not something you can plot directly. Binned Least-Squares Fit In general you can get everything much, much more easily:
```py
import numpy as np
import matplotlib.pyplot as plt
from scipy.optimize import curve_fit
from scipy.special import factorial
from scipy.stats import poisson

# get poisson deviated random numbers
data = np.random.poisson(2, 1000)

# the bins should be of integer width, because poisson is an integer distribution
bins = np.arange(11) - 0.5
entries, bin_edges, patches = plt.hist(data, bins=bins, density=True, label='Data')

# calculate bin centers
bin_centers = 0.5 * (bin_edges[1:] + bin_edges[:-1])

def fit_function(k, lamb):
    '''poisson function, parameter lamb is the fit parameter'''
    return poisson.pmf(k, lamb)

# fit with curve_fit
parameters, cov_matrix = curve_fit(fit_function, bin_centers, entries)

# plot poisson-deviation with fitted parameter
x_plot = np.arange(0, 15)

plt.plot(
    x_plot,
    fit_function(x_plot, *parameters),
    marker='o', linestyle='',
    label='Fit result',
)
plt.legend()
plt.show()
```
This is the result: Unbinned Maximum-Likelihood fit An even better possibility would be to not use a histogram at all and instead to carry out a maximum-likelihood fit. But by closer examination even this is unnecessary, because the maximum-likelihood estimator for the parameter of the poissonian distribution is the arithmetic mean. However, if you have other, more complicated PDFs, you can use this as example:
```
import numpy as np
import matplotlib.pyplot as plt
from scipy.optimize import minimize
from scipy.special import factorial
from scipy import stats

def poisson(k, lamb):
    """poisson pdf, parameter lamb is the fit parameter"""
    return (lamb**k/factorial(k)) * np.exp(-lamb)

def negative_log_likelihood(params, data):
    """
    The negative log-Likelihood-Function
    """

    lnl = - np.sum(np.log(poisson(data, params[0])))
    return lnl

def negative_log_likelihood(params, data):
    ''' better alternative using scipy '''
    return -stats.poisson.logpmf(data, params[0]).sum()

# get poisson deviated random numbers
data = np.random.poisson(2, 1000)

# minimize the negative log-Likelihood

result = minimize(negative_log_likelihood,  # function to minimize
                  x0=np.ones(1),            # start value
                  args=(data,),             # additional arguments for function
                  method='Powell',          # minimization method, see docs
                  )
# result is a scipy optimize result object, the fit parameters
# are stored in result.x
print(result)

# plot poisson-distribution with fitted parameter
x_plot = np.arange(0, 15)

plt.plot(
    x_plot,
    stats.poisson.pmf(x_plot, result.x),
    marker='o', linestyle='',
    label='Fit result',
)
plt.legend()
plt.show()
```


Question: Multivariate spline interpolation in python/scipy? Is there a library module or other straightforward way to implement multivariate spline interpolation in python? Specifically, I have a set of scalar data on a regularly-spaced three-dimensional grid which I need to interpolate at a small number of points scattered throughout the domain. For two dimensions, I have been using scipy.interpolate.RectBivariateSpline, and I'm essentially looking for an extension of that to three-dimensional data. The N-dimensional interpolation routines I have found are not quite good enough: I would prefer splines over LinearNDInterpolator for smoothness, and I have far too many data points (often over one million) for, e.g., a radial basis function to work. If anyone knows of a python library that can do this, or perhaps one in another language that I could call or port, I'd really appreciate it.

Response:
If I'm understanding your question correctly, your input "observation" data is regularly gridded? If so, scipy.ndimage.map_coordinates does exactly what you want. It's a bit hard to understand at first pass, but essentially, you just feed it a sequence of coordinates that you want to interpolate the values of the grid at in pixel/voxel/n-dimensional-index coordinates. As a 2D example:
```
import numpy as np
from scipy import ndimage
import matplotlib.pyplot as plt

# Note that the output interpolated coords will be the same dtype as your input
# data.  If we have an array of ints, and we want floating point precision in
# the output interpolated points, we need to cast the array as floats
data = np.arange(40).reshape((8,5)).astype(np.float)

# I'm writing these as row, column pairs for clarity...
coords = np.array([[1.2, 3.5], [6.7, 2.5], [7.9, 3.5], [3.5, 3.5]])
# However, map_coordinates expects the transpose of this
coords = coords.T

# The "mode" kwarg here just controls how the boundaries are treated
# mode='nearest' is _not_ nearest neighbor interpolation, it just uses the
# value of the nearest cell if the point lies outside the grid.  The default is
# to treat the values outside the grid as zero, which can cause some edge
# effects if you're interpolating points near the edge
# The "order" kwarg controls the order of the splines used. The default is
# cubic splines, order=3
zi = ndimage.map_coordinates(data, coords, order=3, mode='nearest')

row, column = coords
nrows, ncols = data.shape
im = plt.imshow(data, interpolation='nearest', extent=[0, ncols, nrows, 0])
plt.colorbar(im)
plt.scatter(column, row, c=zi, vmin=data.min(), vmax=data.max())
for r, c, z in zip(row, column, zi):
    plt.annotate('%0.3f' % z, (c,r), xytext=(-10,10), textcoords='offset points',
            arrowprops=dict(arrowstyle='->'), ha='right')
plt.show()
```
To do this in n-dimensions, we just need to pass in the appropriate sized arrays:
```
import numpy as np
from scipy import ndimage

data = np.arange(3*5*9).reshape((3,5,9)).astype(np.float)
coords = np.array([[1.2, 3.5, 7.8], [0.5, 0.5, 6.8]])
zi = ndimage.map_coordinates(data, coords.T)
```
As far as scaling and memory usage goes, map_coordinates will create a filtered copy of the array if you're using an order > 1 (i.e. not linear interpolation). If you just want to interpolate at a very small number of points, this is a rather large overhead. It doesn't increase with the number points you want to interpolate at, however. As long as have enough RAM for a single temporary copy of your input data array, you'll be fine. If you can't store a copy of your data in memory, you can either a) specify prefilter=False and order=1 and use linear interpolation, or b) replace your original data with a filtered version using ndimage.spline_filter, and then call map_coordinates with prefilter=False. Even if you have enough ram, keeping the filtered dataset around can be a big speedup if you need to call map_coordinates multiple times (e.g. interactive use, etc).


Question:
Understanding scipy deconvolve I'm trying to understand scipy.signal.deconvolve. From the mathematical point of view a convolution is just the multiplication in fourier space so I would expect that for two functions f and g: Deconvolve(Convolve(f,g) , g) == f In numpy/scipy this is either not the case or I'm missing an important point. Although there are some questions related to deconvolve on SO already (like here and here) they do not address this point, others remain unclear (this) or unanswered (here). There are also two questions on SignalProcessing SE (this and this) the answers to which are not helpful in understanding how scipy's deconvolve function works. The question would be: How do you reconstruct the original signal f from a convoluted signal, assuming you know the convolving function g.? Or in other words: How does this pseudocode Deconvolve(Convolve(f,g) , g) == f translate into numpy / scipy? Edit: Note that this question is not targeted at preventing numerical inaccuracies (although this is also an open question) but at understanding how convolve/deconvolve work together in scipy. The following code tries to do that with a Heaviside function and a gaussian filter. As can be seen in the image, the result of the deconvolution of the convolution is not at all the original Heaviside function. I would be glad if someone could shed some light into this issue.
```
import numpy as np
import scipy.signal
import matplotlib.pyplot as plt

# Define heaviside function
H = lambda x: 0.5 * (np.sign(x) + 1.)
#define gaussian
gauss = lambda x, sig: np.exp(-( x/float(sig))**2 )

X = np.linspace(-5, 30, num=3501)
X2 = np.linspace(-5,5, num=1001)

# convolute a heaviside with a gaussian
H_c = np.convolve( H(X),  gauss(X2, 1),  mode="same"  )
# deconvolute a the result
H_dc, er = scipy.signal.deconvolve(H_c, gauss(X2, 1) )

#### Plot ####
fig , ax = plt.subplots(nrows=4, figsize=(6,7))
ax[0].plot( H(X),          color="#907700", label="Heaviside",    lw=3 )
ax[1].plot( gauss(X2, 1),  color="#907700", label="Gauss filter", lw=3 )
ax[2].plot( H_c/H_c.max(), color="#325cab", label="convoluted" ,  lw=3 )
ax[3].plot( H_dc,          color="#ab4232", label="deconvoluted", lw=3 )
for i in range(len(ax)):
    ax[i].set_xlim([0, len(X)])
    ax[i].set_ylim([-0.07, 1.2])
    ax[i].legend(loc=4)
plt.show()
```
Edit: Note that there is a matlab example, showing how to convolve/deconvolve a rectangular signal using
```
yc=conv(y,c,'full')./sum(c);
ydc=deconv(yc,c).*sum(c);
```
In the spirit of this question it would also help if someone was able to translate this example into python.

Response:
After some trial and error I found out how to interprete the results of scipy.signal.deconvolve() and I post my findings as an answer. Let's start with a working example code
```
import numpy as np
import scipy.signal
import matplotlib.pyplot as plt

# let the signal be box-like
signal = np.repeat([0., 1., 0.], 100)
# and use a gaussian filter
# the filter should be shorter than the signal
# the filter should be such that it's much bigger then zero everywhere
gauss = np.exp(-( (np.linspace(0,50)-25.)/float(12))**2 )
print gauss.min()  # = 0.013 >> 0

# calculate the convolution (np.convolve and scipy.signal.convolve identical)
# the keywordargument mode="same" ensures that the convolution spans the same
#   shape as the input array.
#filtered = scipy.signal.convolve(signal, gauss, mode='same')
filtered = np.convolve(signal, gauss, mode='same')

deconv,  _ = scipy.signal.deconvolve( filtered, gauss )
#the deconvolution has n = len(signal) - len(gauss) + 1 points
n = len(signal)-len(gauss)+1
# so we need to expand it by
s = (len(signal)-n)/2
#on both sides.
deconv_res = np.zeros(len(signal))
deconv_res[s:len(signal)-s-1] = deconv
deconv = deconv_res
# now deconv contains the deconvolution
# expanded to the original shape (filled with zeros)

#### Plot ####
fig , ax = plt.subplots(nrows=4, figsize=(6,7))

ax[0].plot(signal,            color="#907700", label="original",     lw=3 )
ax[1].plot(gauss,          color="#68934e", label="gauss filter", lw=3 )
# we need to divide by the sum of the filter window to get the convolution normalized to 1
ax[2].plot(filtered/np.sum(gauss), color="#325cab", label="convoluted" ,  lw=3 )
ax[3].plot(deconv,         color="#ab4232", label="deconvoluted", lw=3 )

for i in range(len(ax)):
    ax[i].set_xlim([0, len(signal)])
    ax[i].set_ylim([-0.07, 1.2])
    ax[i].legend(loc=1, fontsize=11)
    if i != len(ax)-1 :
        ax[i].set_xticklabels([])

plt.savefig(__file__ + ".png")
plt.show()
```
This code produces the following image, showing exactly what we want (Deconvolve(Convolve(signal,gauss) , gauss) == signal) Some important findings are: The filter should be shorter than the signal The filter should be much bigger than zero everywhere (here > 0.013 is good enough) Using the keyword argument mode = 'same' to the convolution ensures that it lives on the same array shape as the signal. The deconvolution has n = len(signal) - len(gauss) + 1 points. So in order to let it also reside on the same original array shape we need to expand it by s = (len(signal)-n)/2 on both sides. Of course, further findings, comments and suggestion to this question are still welcome.


Question:
How can I efficiently process a numpy array in blocks similar to Matlab's blkproc (blockproc) function I'm looking for a good approach for efficiently dividing an image into small regions, processing each region separately, and then re-assembling the results from each process into a single processed image. Matlab had a tool for this called blkproc (replaced by blockproc in newer versions of Matlab). In an ideal world, the function or class would support overlap between the divisions in the input matrix too. In the Matlab help, blkproc is defined as: B = blkproc(A,[m n],[mborder nborder],fun,...) A is your input matrix, [m n] is the block size [mborder, nborder] is the size of your border region (optional) fun is a function to apply to each block I have kluged together an approach, but it strikes me as clumsy and I bet there's a much better way. At the risk of my own embarrassment, here's my code:
```
import numpy as np

def segmented_process(M, blk_size=(16,16), overlap=(0,0), fun=None):
    rows = []
    for i in range(0, M.shape[0], blk_size[0]):
        cols = []
        for j in range(0, M.shape[1], blk_size[1]):
            cols.append(fun(M[i:i+blk_size[0], j:j+blk_size[1]]))
        rows.append(np.concatenate(cols, axis=1))
    return np.concatenate(rows, axis=0)

R = np.random.rand(128,128)
passthrough = lambda(x):x
Rprime = segmented_process(R, blk_size=(16,16),
                           overlap=(0,0),
                           fun=passthrough)

np.all(R==Rprime)
```

Response:
Here are some examples of a different (loop free) way to work with blocks:
```
import numpy as np
from numpy.lib.stride_tricks import as_strided as ast

A= np.arange(36).reshape(6, 6)
print A
#[[ 0  1  2  3  4  5]
# [ 6  7  8  9 10 11]
# ...
# [30 31 32 33 34 35]]

# 2x2 block view
B= ast(A, shape= (3, 3, 2, 2), strides= (48, 8, 24, 4))
print B[1, 1]
#[[14 15]
# [20 21]]

# for preserving original shape
B[:, :]= np.dot(B[:, :], np.array([[0, 1], [1, 0]]))
print A
#[[ 1  0  3  2  5  4]
# [ 7  6  9  8 11 10]
# ...
# [31 30 33 32 35 34]]
print B[1, 1]
#[[15 14]
# [21 20]]

# for reducing shape, processing in 3D is enough
C= B.reshape(3, 3, -1)
print C.sum(-1)
#[[ 14  22  30]
# [ 62  70  78]
# [110 118 126]]
```
So just trying to simply copy the matlab functionality to numpy is not all ways the best way to proceed. Sometimes a 'off the hat' thinking is needed. Caveat: In general, implementations based on stride tricks may (but does not necessary need to) suffer some performance penalties. So be prepared to all ways measure your performance. In any case it's wise to first check if the needed functionality (or similar enough, in order to easily adapt for) has all ready been implemented in numpy or scipy. Update: Please note that there is no real magic involved here with the strides, so I'll provide a simple function to get a block_view of any suitable 2D numpy-array. So here we go:
```
from numpy.lib.stride_tricks import as_strided as ast

def block_view(A, block= (3, 3)):
    """Provide a 2D block view to 2D array. No error checking made.
    Therefore meaningful (as implemented) only for blocks strictly
    compatible with the shape of A."""
    # simple shape and strides computations may seem at first strange
    # unless one is able to recognize the 'tuple additions' involved ;-)
    shape= (A.shape[0]/ block[0], A.shape[1]/ block[1])+ block
    strides= (block[0]* A.strides[0], block[1]* A.strides[1])+ A.strides
    return ast(A, shape= shape, strides= strides)

if __name__ == '__main__':
    from numpy import arange
    A= arange(144).reshape(12, 12)
    print block_view(A)[0, 0]
    #[[ 0  1  2]
    # [12 13 14]
    # [24 25 26]]
    print block_view(A, (2, 6))[0, 0]
    #[[ 0  1  2  3  4  5]
    # [12 13 14 15 16 17]]
    print block_view(A, (3, 12))[0, 0]
    #[[ 0  1  2  3  4  5  6  7  8  9 10 11]
    # [12 13 14 15 16 17 18 19 20 21 22 23]
    # [24 25 26 27 28 29 30 31 32 33 34 35]]
```


Question:
numpy/scipy equivalent of R ecdf(x)(x) function? What is the equivalent of R's ecdf(x)(x) function in Python, in either numpy or scipy? Is ecdf(x)(x) basically the same as:
```
import numpy as np
def ecdf(x):
  # normalize X to sum to 1
  x = x / np.sum(x)
  return np.cumsum(x)
```
or is something else required? EDIT how can one control the number of bins used by ecdf?

Response:
The OP implementation for ecdf is wrong, you are not supposed to cumsum() the values. So not ys = np.cumsum(x)/np.sum(x) but ys = np.cumsum(1 for _ in x)/float(len(x)) or better ys = np.arange(1, len(x)+1)/float(len(x)) You either go with statmodels's ECDF if you are OK with that extra dependency or provide your own implementation. See below:
```
import numpy as np
import matplotlib.pyplot as plt
from statsmodels.distributions.empirical_distribution import ECDF
%matplotlib inline

grades = (93.5,93,60.8,94.5,82,87.5,91.5,99.5,86,93.5,92.5,78,76,69,94.5,
          89.5,92.8,78,65.5,98,98.5,92.3,95.5,76,91,95,61)

def ecdf_wrong(x):
    xs = np.sort(x) # need to be sorted
    ys = np.cumsum(xs)/np.sum(xs) # normalize so sum == 1
    return (xs,ys)
def ecdf(x):
    xs = np.sort(x)
    ys = np.arange(1, len(xs)+1)/float(len(xs))
    return xs, ys

xs, ys = ecdf_wrong(grades)
plt.plot(xs, ys, label="wrong cumsum")
xs, ys = ecdf(grades)
plt.plot(xs, ys, label="handwritten", marker=">", markerfacecolor='none')
cdf = ECDF(grades)
plt.plot(cdf.x, cdf.y, label="statmodels", marker="<", markerfacecolor='none')
plt.legend()
plt.show()
```


Question:
Slicing Sparse Matrices in Scipy -- Which Types Work Best? The SciPy Sparse Matrix tutorial is very good -- but it actually leaves the section on slicing un(der)developed (still in outline form -- see section: "Handling Sparse Matrices"). I will try and update the tutorial, once this question is answered. I have a large sparse matrix -- currently in dok_matrix format.
```
import numpy as np
from scipy import sparse
M = sparse.dok_matrix((10**6, 10**6))
```
For various methods I want to be able to slice columns and for others I want to slice rows. Ideally I would use advanced-indexing (i.e. a boolean vector, bool_vect) with which to slice a sparse matrix M -- as in:
```
bool_vect = np.arange(10**6)%2  # every even index
out = M[bool_vect,:]            # Want to select every even row
```
or
```
out = M[:,bool_vect] # Want to select every even column
```
First off, dok_matrices do not support this -- but I think it works (slowly) if I first cast to lil_matrices, via sparse.lil_matrix(M) As far as I can gather from the tutorial -- to slice columns I want to use CSC and to slice rows I want to slice CSR. So does that mean I should cast the matrix M via:
```
M.tocsc()[:,bool_vect]
```
or
```
M.tocsr()[bool_vect,:]
```
I am kinda guessing here and my code is slow because of it. Any help from someone who understands how this works would be appreciated. Thanks in advance. If it turns out I should not be indexing my matrix with a boolean array, but rather a list of integers (indices) -- that is also something I would be happy to find out. Whichever is more efficient. Finally -- this is a big matrix, so bonus points if this can happen in place / with broadcasting.

Response:
Ok, so I'm pretty sure the "right" way to do this is: if you are slicing columns, use tocsc() and slice using a list/array of integers. Boolean vectors does not seem to do the trick with sparse matrices -- the way it does with ndarrays in numpy. Which means the answer is.
```
indices = np.where(bool_vect)[0]
out1 = M.tocsc()[:,indices]
out2 = M.tocsr()[indices,:]
```
But question: is this the best way? Is this in place? In practice this does seem to be happening in place -- and it is much faster than prior attempts (using lil_matrix).


Question:
Calculating gradient with NumPy I really can not understand what numpy.gradient function does and how to use it for computation of multivariable function gradient. For example, I have such a function:
```
def func(q, chi, delta):
    return q * chi * delta
```
I need to compute it's 3-dimensional gradient (in other words, I want to compute partial derivatives with respect to all variables (q, chi, delta)). How can I calculate this gradient using NumPy?

Response:
The problem is, that numpy can't give you the derivatives directly and you have two options: With NUMPY What you essentially have to do, is to define a grid in three dimension and to evaluate the function on this grid. Afterwards you feed this table of function values to numpy.gradient to get an array with the numerical derivative for every dimension (variable). Example from here:
```
from numpy import *

x,y,z = mgrid[-100:101:25., -100:101:25., -100:101:25.]

V = 2*x**2 + 3*y**2 - 4*z # just a random function for the potential

Ex,Ey,Ez = gradient(V)
```
Without NUMPY You could also calculate the derivative yourself by using the centered difference quotient. This is essentially, what numpy.gradient is doing for every point of your predefined grid.


Question: Does Python have a function which computes multinomial coefficients? I was looking for a Python library function which computes multinomial coefficients. I could not find any such function in any of the standard libraries. For binomial coefficients (of which multinomial coefficients are a generalization) there is scipy.special.binom and also scipy.misc.comb. Also, numpy.random.multinomial draws samples from a multinomial distribution, and sympy.ntheory.multinomial.multinomial_coefficients returns a dictionary related to multinomial coefficients. However, I could not find a multinomial coefficients function proper, which given a,b,...,z returns (a+b+...+z)!/(a! b! ... z!). Did I miss it? Is there a good reason there is none available? I would be happy to contribute an efficient implementation to SciPy say. (I would have to figure out how to contribute, as I have never done this). For background, they do come up when expanding (a+b+...+z)^n. Also, they count the ways of depositing a+b+...+z distinct objects into distinct bins such that the first bin contains a objects, etc. I need them occasionally for a Project Euler problem. BTW, other languages do offer this function: Mathematica, MATLAB, Maple.

Response:
To partially answer my own question, here is my simple and fairly efficient implementation of the multinomial function:
```
def multinomial(lst):
    res, i = 1, 1
    for a in lst:
        for j in range(1,a+1):
            res *= i
            res //= j
            i += 1
    return res
```
It seems from the comments so far that no efficient implementation of the function exists in any of the standard libraries. Update (January 2020). As Don Hatch has pointed out in the comments, this can be further improved by looking for the largest argument (especially for the case that it dominates all others):
```
def multinomial(lst):
    res, i = 1, sum(lst)
    i0 = lst.index(max(lst))
    for a in lst[:i0] + lst[i0+1:]:
        for j in range(1,a+1):
            res *= i
            res //= j
            i -= 1
    return res
```


Question:
Numpy Indexing: Return the rest A simply example of numpy indexing:
```
In: a = numpy.arange(10)
In: sel_id = numpy.arange(5)
In: a[sel_id]
Out: array([0,1,2,3,4])
```
How do I return the rest of the array that are not indexed by sel_id? What I can think of is:
```
In: numpy.array([x for x in a if x not in a[id]])
out: array([5,6,7,8,9])
```
Is there any easier way?

Response:
For this simple 1D case, I'd actually use a boolean mask:
```
a = numpy.arange(10)
include_index = numpy.arange(4)
include_idx = set(include_index)  #Set is more efficient, but doesn't reorder your elements if that is desireable
mask = numpy.array([(i in include_idx) for i in xrange(len(a))])
```
Now you can get your values:
```
included = a[mask]  # array([0, 1, 2, 3])
excluded = a[~mask] # array([4, 5, 6, 7, 8, 9])
```
Note that a[mask] doesn't necessarily yield the same thing as a[include_index] since the order of include_index matters for the output in that scenario (it should be roughly equivalent to a[sorted(include_index)]). However, since the order of your excluded items isn't well defined, this should work Ok. EDIT A better way to create the mask is:
```
mask = np.zeros(a.shape,dtype=bool)
mask[include_idx] = True
```
(thanks to seberg).


Question:
Fit a gaussian function I have a histogram (see below) and I am trying to find the mean and standard deviation along with code which fits a curve to my histogram. I think there is something in SciPy or matplotlib that can help, but every example I've tried doesn't work.
```
import matplotlib.pyplot as plt
import numpy as np

with open('gau_b_g_s.csv') as f:
    v = np.loadtxt(f, delimiter= ',', dtype="float", skiprows=1, usecols=None)

fig, ax = plt.subplots()

plt.hist(v, bins=500, color='#7F38EC', histtype='step')

plt.title("Gaussian")
plt.axis([-1, 2, 0, 20000])

plt.show()
```

Response:
Take a look at this answer for fitting arbitrary curves to data. Basically you can use scipy.optimize.curve_fit to fit any function you want to your data. The code below shows how you can fit a Gaussian to some random data (credit to this SciPy-User mailing list post).
```
import numpy
from scipy.optimize import curve_fit
import matplotlib.pyplot as plt

# Define some test data which is close to Gaussian
data = numpy.random.normal(size=10000)

hist, bin_edges = numpy.histogram(data, density=True)
bin_centres = (bin_edges[:-1] + bin_edges[1:])/2

# Define model function to be used to fit to the data above:
def gauss(x, *p):
    A, mu, sigma = p
    return A*numpy.exp(-(x-mu)**2/(2.*sigma**2))

# p0 is the initial guess for the fitting coefficients (A, mu and sigma above)
p0 = [1., 0., 1.]

coeff, var_matrix = curve_fit(gauss, bin_centres, hist, p0=p0)

# Get the fitted curve
hist_fit = gauss(bin_centres, *coeff)

plt.plot(bin_centres, hist, label='Test data')
plt.plot(bin_centres, hist_fit, label='Fitted data')

# Finally, lets get the fitting parameters, i.e. the mean and standard deviation:
print 'Fitted mean = ', coeff[1]
print 'Fitted standard deviation = ', coeff[2]

plt.show()
```


Question:
Capturing high multi-collinearity in statsmodels Say I fit a model in statsmodels
```
mod = smf.ols('dependent ~ first_category + second_category + other', data=df).fit()
```
When I do mod.summary() I may see the following:
```
Warnings:
[1] The condition number is large, 1.59e+05. This might indicate that there are
strong multicollinearity or other numerical problems.
```
Sometimes the warning is different (e.g. based on eigenvalues of the design matrix). How can I capture high-multi-collinearity conditions in a variable? Is this warning stored somewhere in the model object? Also, where can I find a description of the fields in summary()?

Response:
You can detect high-multi-collinearity by inspecting the eigen values of correlation matrix. A very low eigen value shows that the data are collinear, and the corresponding eigen vector shows which variables are collinear. If there is no collinearity in the data, you would expect that none of the eigen values are close to zero:
```

```python
xs = np.random.randn(100, 5)      # independent variables
```

```python
corr = np.corrcoef(xs, rowvar=0)  # correlation matrix
```

```python
w, v = np.linalg.eig(corr)        # eigen values & eigen vectors
```

```python
w
#Output
#array([ 1.256 ,  1.1937,  0.7273,  0.9516,  0.8714])
#```
#However, if say x[4] - 2 * x[0] - 3 * x[2] = 0, then
#```
```

```python
noise = np.random.randn(100)                      # white noise
```

```python
xs[:,4] = 2 * xs[:,0] + 3 * xs[:,2] + .5 * noise  # collinearity
```

```python
corr = np.corrcoef(xs, rowvar=0)
```

```python
w, v = np.linalg.eig(corr)
```

```python
w
#Output
#array([ 0.0083,  1.9569,  1.1687,  0.8681,  0.9981])
#```
#one of the eigen values (here the very first one), is close to zero. The corresponding eigen vector is:
#```
```

```python
v[:,0]
#Output
#array([-0.4077,  0.0059, -0.5886,  0.0018,  0.6981])
#```
#Ignoring almost zero coefficients, above basically says x[0], x[2] and x[4] are colinear (as expected). If one standardizes xs values and multiplies by this eigen vector, the result will hover around zero with small variance:
#```
```

```python
std_xs = (xs - xs.mean(axis=0)) / xs.std(axis=0)  # standardized values
```

```python
ys = std_xs.dot(v[:,0])
```

```python
ys.mean(), ys.var()
#Output
#(0, 0.0083)
#```
#Note that ys.var() is basically the eigen value which was close to zero. So, in order to capture high multi-linearity, look at the eigen values of correlation matrix.
```


Question:
scipy.linalg.eig return complex eigenvalues for covariance matrix? The eigenvalues of a covariance matrix should be real and non-negative because covariance matrices are symmetric and semi positive definite. However, take a look at the following experiment with scipy:
```

```python
a=np.random.random(5)
```

```python
b=np.random.random(5)
```

```python
ab = np.vstack((a,b)).T
```

```python
C=np.cov(ab)
```

```python
eig(C)
#Output
#7.90174997e-01 +0.00000000e+00j,
#2.38344473e-17 +6.15983679e-17j,
#2.38344473e-17 -6.15983679e-17j,
#-1.76100435e-17 +0.00000000e+00j,
#5.42658040e-33 +0.00000000e+00j
#```
#However, reproducing the above example in Matlab works correctly:
#```
#a = [0.6271, 0.4314, 0.3453, 0.8073, 0.9739]
#b = [0.1924, 0.3680, 0.0568, 0.1831, 0.0176]
#C=cov([a;b])
#eig(C)
#-0.0000
#-0.0000
# 0.0000
# 0.0000
# 0.7902
#```
```

Response:
You have raised two issues: The eigenvalues returned by scipy.linalg.eig are not real. Some of the eigenvalues are negative. Both of these issues are the result of errors introduced by truncation and rounding errors, which always happen with iterative algorithms using floating-point arithmetic. Note that the Matlab results also produced negative eigenvalues. Now, for a more interesting aspect of the issue: why is Matlab's result real, whereas SciPy's result has some complex components? Matlab's eig detects if the input matrix is real symmetric or Hermitian and uses Cholesky factorization when it is. See the description of the chol argument in the eig documentation. This is not done automatically in SciPy. If you want to use an algorithm that exploits the structure of a real symmetric or Hermitian matrix, use scipy.linalg.eigh. For the example in the question:
```

```python
eigh(C, eigvals_only=True)
#Output
#array([ -3.73825923e-17,  -1.60154836e-17,   8.11704449e-19,
#         3.65055777e-17,   7.90175615e-01])
#```
#This result is the same as Matlab's, if you round to the same number of digits of precision that Matlab printed.
```


Question:
How to compute cluster assignments from linkage/distance matrices if you have this hierarchical clustering call in scipy in Python:
```
from scipy.cluster.hierarchy import linkage
# dist_matrix is long form distance matrix
linkage_matrix = linkage(squareform(dist_matrix), linkage_method)
```
then what's an efficient way to go from this to cluster assignments for individual points? i.e. a vector of length N where N is number of points, where each entry i is the cluster number of point i, given the number of clusters generated by a given threshold thresh on the resulting clustering? To clarify: The cluster number would be the cluster that it's in after applying a threshold to the tree. In which case you would get a unique cluster for each leaf node for the cluster that it is in. Unique in the sense that each point belongs to one "most specific cluster" which is defined by the threshold where you cut the dendrogram. I know that scipy.cluster.hierarchy.fclusterdata gives you this cluster assignment as its return value, but I am starting from a custom made distance matrix and distance metric, so I cannot use fclusterdata. The question boils down to: how can I compute what fclusterdata is computing -- the cluster assignments?

Response: If I understand you right, that is what fcluster does: scipy.cluster.hierarchy.fcluster(Z, t, criterion='inconsistent', depth=2, R=None, monocrit=None) Forms flat clusters from the hierarchical clustering defined by the linkage matrix Z. ... Returns: An array of length n. T[i] is the flat cluster number to which original observation i belongs. So just call fcluster(linkage_matrix, t), where t is your threshold.


Question: Root mean square of a function in python I want to calculate root mean square of a function in Python. My function is in a simple form like y = f(x). x and y are arrays. I tried Numpy and Scipy Docs and couldn't find anything.

Response:
I'm going to assume that you want to compute the expression given by the following pseudocode:
```
ms = 0
for i = 1 ... N
    ms = ms + y[i]^2
ms = ms / N
rms = sqrt(ms)
```
i.e. the square root of the mean of the squared values of elements of y. In numpy, you can simply square y, take its mean and then its square root as follows:
```
rms = np.sqrt(np.mean(y**2))
```
So, for example:
```

```python
y = np.array([0, 0, 1, 1, 0, 1, 0, 1, 1, 1])  # Six 1's
```

```python
y.size
#Output
#10
```

```python
np.mean(y**2)
#Output
#0.59999999999999998
```

```python
np.sqrt(np.mean(y**2))
#Output
#0.7745966692414834
#```
#Do clarify your question if you mean to ask something else.
```


Question: Converting image from RGB to HSV color space I couldn't find such function (i.e. RGB_to_HSV()) in Scipy or Matplotlib's documentations, and Google doesn't show pointers, except ActiveState recipe which demonstrates rgb2hsv function, though not usable on Numpy array as is. Does someone know of a shortcut? Edit: Sorry, just found matplotlib.colors.rgb_to_hsv() which is exactly what I was looking for. Should I delete this question?

Response: Matplotlib provides RGB to HSV conversion function: matplotlib.colors.rgb_to_hsv(): matplotlib.colors.rgb_to_hsv(arr) convert rgb values in a numpy array to hsv values input and output arrays should have shape (M,N,3)


Question:
Chi-Squared test in Python I've used the following code in R to determine how well observed values (20, 20, 0 and 0 for example) fit expected values/ratios (25% for each of the four cases, for example):
```
> chisq.test(c(20,20,0,0), p=c(0.25, 0.25, 0.25, 0.25))

    Chi-squared test for given probabilities

data:  c(20, 20, 0, 0)

X-squared = 40, df = 3, p-value = 1.066e-08
```
How can I replicate this in Python? I've tried using the chisquare function from scipy but the results I obtained were very different; I'm not sure if this is even the correct function to use. I've searched through the scipy documentation, but it's quite daunting as it runs to 1000+ pages; the numpy documentation is almost 50% more than that.

Response:
scipy.stats.chisquare expects observed and expected absolute frequencies, not ratios. You can obtain what you want with
```

```python
observed = np.array([20., 20., 0., 0.])
```

```python
expected = np.array([.25, .25, .25, .25]) * np.sum(observed)
```

```python
chisquare(observed, expected)
#Output
#(40.0, 1.065509033425585e-08)
#```
#Although in the case that the expected values are uniformly distributed over the classes, you can leave out the computation of the expected values:
#```
```

```python
chisquare(observed)
#Output
#(40.0, 1.065509033425585e-08)
#```
#The first returned value is the χ² statistic, the second the p-value of the test.
```


Question:
Is there an "enhanced" numpy/scipy dot method? Problem I would like to compute the following using numpy or scipy:
```
Y = A**T * Q * A
```
where A is a m x n matrix, A**T is the transpose of A and Q is an m x m diagonal matrix. Since Q is a diagonal matrix I store only its diagonal elements as a vector. Ways of solving for Y Currently I can think of two ways of how to calculate Y: Y = np.dot(np.dot(A.T, np.diag(Q)), A) and Y = np.dot(A.T * Q, A). Clearly option 2 is better than option 1 since no real matrix has to be created with diag(Q) (if this is what numpy really does...) However, both methods suffer from the defect of having to allocate more memory than there really is necessary since A.T * Q and np.dot(A.T, np.diag(Q)) have to be stored along with A in order to calculate Y. Question Is there a method in numpy/scipy that would eliminate the unnecessary allocation of extra memory where you would only pass two matrices A and B (in my case B is A.T) and a weighting vector Q along with it?

Response:
(w/r/t the last sentence of the OP: i am not aware of such a numpy/scipy method but w/r/t the Question in the OP Title (i.e., improving NumPy dot performance) what's below should be of some help. In other words, my answer is directed to improving performance of most of the steps comprising your function for Y). First, this should give you a noticeable boost over the vanilla NumPy dot method:
```

```python
from scipy.linalg import blas as FB
```

```python
vx = FB.dgemm(alpha=1., a=v1, b=v2, trans_b=True)
#Output
#```
#Note that the two arrays, v1, v2 are both in C_FORTRAN order You can access the byte order of a NumPy array through an array's flags attribute like so:
#```
```

```python
c = NP.ones((4, 3))
```

```python
c.flags
#Output
#      C_CONTIGUOUS : True          # refers to C-contiguous order
#      F_CONTIGUOUS : False         # fortran-contiguous
#      OWNDATA : True
#      MASKNA : False
#      OWNMASKNA : False
#      WRITEABLE : True
#      ALIGNED : True
#      UPDATEIFCOPY : False
#```
#to change the order of one of the arrays so both are aligned, just call the NumPy array constructor, pass in the array and set the appropriate order flag to True
#```
```

```python
c = NP.array(c, order="F")
```

```python
c.flags
#Output
#      C_CONTIGUOUS : False
#      F_CONTIGUOUS : True
#      OWNDATA : True
#      MASKNA : False
#      OWNMASKNA : False
#      WRITEABLE : True
#      ALIGNED : True
#      UPDATEIFCOPY : False
#```
#You can further optimize by exploiting array-order alignment to reduce excess memory consumption caused by copying the original arrays. But why are the arrays copied before being passed to dot? The dot product relies on BLAS operations. These operations require arrays stored in C-contiguous order--it's this constraint that causes the arrays to be copied. On the other hand, the transpose does not effect a copy, though unfortunately returns the result in Fortran order: Therefore, to remove the performance bottleneck, you need to eliminate the predicate array-copying step; to do that just requires passing both arrays to dot in C-contiguous order*. So to calculate dot(A.T., A) without making an extra copy:
#```
```

```python
import scipy.linalg.blas as FB
```

```python
vx = FB.dgemm(alpha=1.0, a=A.T, b=A.T, trans_b=True)
#Output
#```
#In sum, the expression just above (along with the predicate import statement) can substitute for dot, to supply the same functionality but better performance you can bind that expression to a function like so:
#```
```

```python
super_dot = lambda v, w: FB.dgemm(alpha=1., a=v.T, b=w.T, trans_b=True)
#Output
#```
```