Question: What are the differences between Pandas and NumPy+SciPy in Python? [closed] Closed. This question is opinion-based. It is not currently accepting answers. Want to improve this question? Because this question may lead to opinionated discussion, debate, and answers, it has been closed. You may edit the question if you feel you can improve it so that it requires answers that include facts and citations or a detailed explanation of the proposed solution. If edited, the question will be reviewed and might be reopened. Closed 10 years ago. Improve this question They both seem exceedingly similar and I'm curious as to which package would be more beneficial for financial data analysis.

Response: pandas provides high level data manipulation tools built on top of NumPy. NumPy by itself is a fairly low-level tool, similar to MATLAB. pandas on the other hand provides rich time series functionality, data alignment, NA-friendly statistics, groupby, merge and join methods, and lots of other conveniences. It has become very popular in recent years in financial applications. I will have a chapter dedicated to financial data analysis using pandas in my upcoming book.


Question: Peak-finding algorithm for Python/SciPy I can write something myself by finding zero-crossings of the first derivative or something, but it seems like a common-enough function to be included in standard libraries. Anyone know of one? My particular application is a 2D array, but usually it would be used for finding peaks in FFTs, etc. Specifically, in these kinds of problems, there are multiple strong peaks, and then lots of smaller "peaks" that are just caused by noise that should be ignored. These are just examples; not my actual data: 1-dimensional peaks: 2-dimensional peaks: The peak-finding algorithm would find the location of these peaks (not just their values), and ideally would find the true inter-sample peak, not just the index with maximum value, probably using quadratic interpolation or something. Typically you only care about a few strong peaks, so they'd either be chosen because they're above a certain threshold, or because they're the first n peaks of an ordered list, ranked by amplitude. As I said, I know how to write something like this myself. I'm just asking if there's a pre-existing function or package that's known to work well. Update: I translated a MATLAB script and it works decently for the 1-D case, but could be better. Updated update: sixtenbe created a better version for the 1-D case.

Response:
The function scipy.signal.find_peaks, as its name suggests, is useful for this. But it's important to understand well its parameters width, threshold, distance and above all prominence to get a good peak extraction. According to my tests and the documentation, the concept of prominence is "the useful concept" to keep the good peaks, and discard the noisy peaks. What is (topographic) prominence? It is "the minimum height necessary to descend to get from the summit to any higher terrain", as it can be seen here: The idea is: The higher the prominence, the more "important" the peak is. Test: I used a (noisy) frequency-varying sinusoid on purpose because it shows many difficulties. We can see that the width parameter is not very useful here because if you set a minimum width too high, then it won't be able to track very close peaks in the high frequency part. If you set width too low, you would have many unwanted peaks in the left part of the signal. Same problem with distance. threshold only compares with the direct neighbours, which is not useful here. prominence is the one that gives the best solution. Note that you can combine many of these parameters! Code:
```
import numpy as np
import matplotlib.pyplot as plt
from scipy.signal import find_peaks

x = np.sin(2*np.pi*(2**np.linspace(2,10,1000))*np.arange(1000)/48000) + np.random.normal(0, 1, 1000) * 0.15
peaks, _ = find_peaks(x, distance=20)
peaks2, _ = find_peaks(x, prominence=1)      # BEST!
peaks3, _ = find_peaks(x, width=20)
peaks4, _ = find_peaks(x, threshold=0.4)     # Required vertical distance to its direct neighbouring samples, pretty useless
plt.subplot(2, 2, 1)
plt.plot(peaks, x[peaks], "xr"); plt.plot(x); plt.legend(['distance'])
plt.subplot(2, 2, 2)
plt.plot(peaks2, x[peaks2], "ob"); plt.plot(x); plt.legend(['prominence'])
plt.subplot(2, 2, 3)
plt.plot(peaks3, x[peaks3], "vg"); plt.plot(x); plt.legend(['width'])
plt.subplot(2, 2, 4)
plt.plot(peaks4, x[peaks4], "xk"); plt.plot(x); plt.legend(['threshold'])
plt.show()
```


Question: How do I use numpy.where()? What should I pass, and what does the result mean? [closed] Closed. This question needs to be more focused. It is not currently accepting answers. Want to improve this question? Guide the asker to update the question so it focuses on a single, specific problem. Narrowing the question will help others answer the question concisely. You may edit the question if you feel you can improve it yourself. If edited, the question will be reviewed and might be reopened. Closed 9 years ago. Improve this question I tried reading the documentation for numpy.where(), but I'm still confused. What should I pass for the condition, x and y values? When I pass only condition, what does the result mean and how can I use it? What about when I pass all three? I found How does python numpy.where() work? but it didn't answer my question because it seems to be about the implementation rather than about how to use it. Numpy where() on a 2D matrix also didn't explain things for me; I'm looking for a step-by-step explanation, rather than a how-to guide for a specific case. Please include examples with both 1D and 2D source data.

Response:
After fiddling around for a while, I figured things out, and am posting them here hoping it will help others. Intuitively, np.where is like asking "tell me where in this array, entries satisfy a given condition".
```

```python
a = np.arange(5,10)
```

```python
np.where(a < 8)       # tell me where in a, entries are < 8
#Output
#(array([0, 1, 2]),)       # answer: entries indexed by 0, 1, 2
#```
#It can also be used to get entries in array that satisfy the condition:
#```
```

```python
a[np.where(a < 8)]
#Output
#array([5, 6, 7])          # selects from a entries 0, 1, 2
#```
#When a is a 2d array, np.where() returns an array of row idx's, and an array of col idx's:
#```
```

```python
a = np.arange(4,10).reshape(2,3)
#Output
#array([[4, 5, 6],
#       [7, 8, 9]])
```

```python
np.where(a > 8)
#Output
#(array(1), array(2))
#```
#As in the 1d case, we can use np.where() to get entries in the 2d array that satisfy the condition:
#```
```

```python
a[np.where(a > 8)] # selects from a entries 0, 1, 2
#Output
#```
#array([9]) Note, when a is 1d, np.where() still returns an array of row idx's and an array of col idx's, but columns are of length 1, so latter is empty array.
```


Question:
binning data in python with scipy/numpy is there a more efficient way to take an average of an array in prespecified bins? for example, i have an array of numbers and an array corresponding to bin start and end positions in that array, and I want to just take the mean in those bins? I have code that does it below but i am wondering how it can be cut down and improved. thanks.
```
from scipy import *
from numpy import *

def get_bin_mean(a, b_start, b_end):
    ind_upper = nonzero(a >= b_start)[0]
    a_upper = a[ind_upper]
    a_range = a_upper[nonzero(a_upper < b_end)[0]]
    mean_val = mean(a_range)
    return mean_val

data = rand(100)
bins = linspace(0, 1, 10)
binned_data = []

n = 0
for n in range(0, len(bins)-1):
    b_start = bins[n]
    b_end = bins[n+1]
    binned_data.append(get_bin_mean(data, b_start, b_end))

print binned_data
```

Response:
It's probably faster and easier to use numpy.digitize():
```
import numpy
data = numpy.random.random(100)
bins = numpy.linspace(0, 1, 10)
digitized = numpy.digitize(data, bins)
bin_means = [data[digitized == i].mean() for i in range(1, len(bins))]
```
An alternative to this is to use numpy.histogram():
```
bin_means = (numpy.histogram(data, bins, weights=data)[0] /
             numpy.histogram(data, bins)[0])
```
Try for yourself which one is faster... :)


Question:
How to calculate probability in a normal distribution given mean & standard deviation? How to calculate probability in normal distribution given mean, std in Python? I can always explicitly code my own function according to the definition like the OP in this question did: Calculating Probability of a Random Variable in a Distribution in Python Just wondering if there is a library function call will allow you to do this. In my imagine it would like this:
```
nd = NormalDistribution(mu=100, std=12)
p = nd.prob(98)
```
There is a similar question in Perl: How can I compute the probability at a point given a normal distribution in Perl?. But I didn't see one in Python. Numpy has a random.normal function, but it's like sampling, not exactly what I want.

Response:
There's one in scipy.stats:
```

```python
import scipy.stats
```

```python
scipy.stats.norm(0, 1)
#Output
#<scipy.stats.distributions.rv_frozen object at 0x928352c>
```

```python
scipy.stats.norm(0, 1).pdf(0)
#Output
#0.3989422804014327
```

```python
scipy.stats.norm(0, 1).cdf(0)
#Output
#0.5
```

```python
scipy.stats.norm(100, 12)
#Output
#<scipy.stats.distributions.rv_frozen object at 0x928352c>
```

```python
scipy.stats.norm(100, 12).pdf(98)
#Output
#0.032786643008494994
```

```python
scipy.stats.norm(100, 12).cdf(98)
#Output
#0.43381616738909634
```

```python
scipy.stats.norm(100, 12).cdf(100)
#Output
#0.5
#```
#[One thing to beware of -- just a tip -- is that the parameter passing is a little broad. Because of the way the code is set up, if you accidentally write scipy.stats.norm(mean=100, std=12) instead of scipy.stats.norm(100, 12) or scipy.stats.norm(loc=100, scale=12), then it'll accept it, but silently discard those extra keyword arguments and give you the default (0,1).]
```


Question:
Histogram Matplotlib So I have a little problem. I have a data set in scipy that is already in the histogram format, so I have the center of the bins and the number of events per bin. How can I now plot is as a histogram. I tried just doing
```
bins, n=hist()
```
but it didn't like that. Any recommendations?

Response:
```
import matplotlib.pyplot as plt
import numpy as np

mu, sigma = 100, 15
x = mu + sigma * np.random.randn(10000)
hist, bins = np.histogram(x, bins=50)
width = 0.7 * (bins[1] - bins[0])
center = (bins[:-1] + bins[1:]) / 2
plt.bar(center, hist, align='center', width=width)
plt.show()
```
The object-oriented interface is also straightforward:
```
fig, ax = plt.subplots()
ax.bar(center, hist, align='center', width=width)
fig.savefig("1.png")
```
If you are using custom (non-constant) bins, you can pass compute the widths using np.diff, pass the widths to ax.bar and use ax.set_xticks to label the bin edges:
```
import matplotlib.pyplot as plt
import numpy as np

mu, sigma = 100, 15
x = mu + sigma * np.random.randn(10000)
bins = [0, 40, 60, 75, 90, 110, 125, 140, 160, 200]
hist, bins = np.histogram(x, bins=bins)
width = np.diff(bins)
center = (bins[:-1] + bins[1:]) / 2

fig, ax = plt.subplots(figsize=(8,3))
ax.bar(center, hist, align='center', width=width)
ax.set_xticks(bins)
fig.savefig("/tmp/out.png")

plt.show()
```


Question: How to calculate the inverse of the normal cumulative distribution function in python? How do I calculate the inverse of the cumulative distribution function (CDF) of the normal distribution in Python? Which library should I use? Possibly scipy?

Response:
NORMSINV (mentioned in a comment) is the inverse of the CDF of the standard normal distribution. Using scipy, you can compute this with the ppf method of the scipy.stats.norm object. The acronym ppf stands for percent point function, which is another name for the quantile function.
```

```python
from scipy.stats import norm
```

```python
norm.ppf(0.95)
#Output
#1.6448536269514722
#```
#Check that it is the inverse of the CDF:
#```
```

```python
norm.cdf(norm.ppf(0.95))
#Output
#0.94999999999999996
#```
#By default, norm.ppf uses mean=0 and stddev=1, which is the "standard" normal distribution. You can use a different mean and standard deviation by specifying the loc and scale arguments, respectively.
#```
```

```python
norm.ppf(0.95, loc=10, scale=2)
#Output
#13.289707253902945
#```
#If you look at the source code for scipy.stats.norm, you'll find that the ppf method ultimately calls scipy.special.ndtri. So to compute the inverse of the CDF of the standard normal distribution, you could use that function directly:
#```
```

```python
from scipy.special import ndtri
```

```python
ndtri(0.95)
#Output
#1.6448536269514722
#```
#ndtri is much faster than norm.ppf:
#```
```

```python
%timeit norm.ppf(0.95)
240 µs ± 1.75 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)
```

```python
%timeit ndtri(0.95)
1.47 µs ± 1.3 ns per loop (mean ± std. dev. of 7 runs, 1,000,000 loops each)
```
```


Question:
How to delete columns in numpy.array I would like to delete selected columns in a numpy.array . This is what I do:
```
n [397]: a = array([[ NaN,   2.,   3., NaN],
   .....:        [  1.,   2.,   3., 9]])

```python
print a
[[ NaN   2.   3.  NaN]
 [  1.   2.   3.   9.]]
```

```python
z = any(isnan(a), axis=0)
```

```python
print z
[ True False False  True]
```

```python
delete(a, z, axis = 1)
#Output
#array([[  3.,  NaN],
#       [  3.,   9.]])
#```
#In this example my goal is to delete all the columns that contain NaN's. I expect the last command to result in:
#```
#array([[2., 3.],
#       [2., 3.]])
#```
#How can I do that?
```

Response:
Given its name, I think the standard way should be delete:
```
import numpy as np

A = np.delete(A, 1, 0)  # delete second row of A
B = np.delete(B, 2, 0)  # delete third row of B
C = np.delete(C, 1, 1)  # delete second column of C
```
According to numpy's documentation page, the parameters for numpy.delete are as follow: numpy.delete(arr, obj, axis=None) arr refers to the input array, obj refers to which sub-arrays (e.g. column/row no. or slice of the array) and axis refers to either column wise (axis = 1) or row-wise (axis = 0) delete operation.


Question:
Two-sample Kolmogorov-Smirnov Test in Python Scipy I can't figure out how to do a Two-sample KS test in Scipy. After reading the documentation of scipy kstest, I can see how to test whether a distribution is identical to standard normal distribution
```py
from scipy.stats import kstest
import numpy as np

x = np.random.normal(0,1,1000)
test_stat = kstest(x, 'norm')
#>>> test_stat
#(0.021080234718821145, 0.76584491300591395)
```
Which means that at p-value of 0.76 we cannot reject the null hypothesis that the two distributions are identical. However, I want to compare two distributions and see if I can reject the null hypothesis that they are identical, something like:
```py
from scipy.stats import kstest
import numpy as np

x = np.random.normal(0,1,1000)
z = np.random.normal(1.1,0.9, 1000)
```
and test whether x and z are identical. I tried the naive:
```py
test_stat = kstest(x, z)
```
and got the following error:
```none
TypeError: 'numpy.ndarray' object is not callable
```
Is there a way to do a two-sample KS test in Python? If so, how should I do it?

Response:
You are using the one-sample KS test. You probably want the two-sample test ks_2samp:
```

```python
from scipy.stats import ks_2samp
```

```python
import numpy as np
```

```python

```

```python
np.random.seed(12345678)
```

```python
x = np.random.normal(0, 1, 1000)
```

```python
y = np.random.normal(0, 1, 1000)
```

```python
z = np.random.normal(1.1, 0.9, 1000)
```

```python

```

```python
ks_2samp(x, y)
#Output
#Ks_2sampResult(statistic=0.022999999999999909, pvalue=0.95189016804849647)
```

```python
ks_2samp(x, z)
#Output
#Ks_2sampResult(statistic=0.41800000000000004, pvalue=3.7081494119242173e-77)
#```
#Results can be interpreted as following: You can either compare the statistic value given by python to the KS-test critical value table according to your sample size. When statistic value is higher than the critical value, the two distributions are different. Or you can compare the p-value to a level of significance a, usually a=0.05 or 0.01 (you decide, the lower a is, the more significant). If p-value is lower than a, then it is very probable that the two distributions are different.
```


Question:
How do you find the IQR in Numpy? Is there a baked-in Numpy/Scipy function to find the interquartile range? I can do it pretty easily myself, but mean() exists which is basically sum/len...
```
def IQR(dist):
    return np.percentile(dist, 75) - np.percentile(dist, 25)
```

Response:
np.percentile takes multiple percentile arguments, and you are slightly better off doing:
```
q75, q25 = np.percentile(x, [75 ,25])
iqr = q75 - q25
```
or
```
iqr = np.subtract(*np.percentile(x, [75, 25]))
```
than making two calls to percentile:
```

```python
x = np.random.rand(1e6)
```

```python
%timeit q75, q25 = np.percentile(x, [75 ,25]); iqr = q75 - q25
10 loops, best of 3: 24.2 ms per loop
```

```python
%timeit iqr = np.subtract(*np.percentile(x, [75, 25]))
10 loops, best of 3: 24.2 ms per loop
```

```python
%timeit iqr = np.percentile(x, 75) - np.percentile(x, 25)
10 loops, best of 3: 33.7 ms per loop
```
```


Question: Find out if a matrix is positive definite with NumPy How can I find out if a matrix is positive definite? My matrix is a NumPy matrix. I was expecting to find any related method in the NumPy library, but I didn't have any success.

Response:
You can also check if all the eigenvalues of matrix are positive. If so, the matrix is positive definite:
```
import numpy as np

def is_pos_def(x):
    return np.all(np.linalg.eigvals(x) > 0)
```


Question:
Save / load scipy sparse csr_matrix in portable data format How do you save/load a scipy sparse csr_matrix in a portable format? The scipy sparse matrix is created on Python 3 (Windows 64-bit) to run on Python 2 (Linux 64-bit). Initially, I used pickle (with protocol=2 and fix_imports=True) but this didn't work going from Python 3.2.2 (Windows 64-bit) to Python 2.7.2 (Windows 32-bit) and got the error:
```
TypeError: ('data type not understood', <built-in function _reconstruct>, (<type 'numpy.ndarray'>, (0,), '[98]')).
```
Next, tried numpy.save and numpy.load as well as scipy.io.mmwrite() and scipy.io.mmread() and none of these methods worked either.

Response:
edit: scipy 0.19 now has scipy.sparse.save_npz and scipy.sparse.load_npz.
```
from scipy import sparse

sparse.save_npz("yourmatrix.npz", your_matrix)
your_matrix_back = sparse.load_npz("yourmatrix.npz")
```
For both functions, the file argument may also be a file-like object (i.e. the result of open) instead of a filename. Got an answer from the Scipy user group: A csr_matrix has 3 data attributes that matter: .data, .indices, and .indptr. All are simple ndarrays, so numpy.save will work on them. Save the three arrays with numpy.save or numpy.savez, load them back with numpy.load, and then recreate the sparse matrix object with:
```
new_csr = csr_matrix((data, indices, indptr), shape=(M, N))
```
So for example:
```
def save_sparse_csr(filename, array):
    np.savez(filename, data=array.data, indices=array.indices,
             indptr=array.indptr, shape=array.shape)

def load_sparse_csr(filename):
    loader = np.load(filename)
    return csr_matrix((loader['data'], loader['indices'], loader['indptr']),
                      shape=loader['shape'])
```


Question:
Resampling a numpy array representing an image I am looking for how to resample a numpy array representing image data at a new size, preferably having a choice of the interpolation method (nearest, bilinear, etc.). I know there is
```
scipy.misc.imresize
```
which does exactly this by wrapping PIL's resize function. The only problem is that since it uses PIL, the numpy array has to conform to image formats, giving me a maximum of 4 "color" channels. I want to be able to resize arbitrary images, with any number of "color" channels. I was wondering if there is a simple way to do this in scipy/numpy, or if I need to roll my own. I have two ideas for how to concoct one myself: a function that runs scipy.misc.imresize on every channel separately create my own using scipy.ndimage.interpolation.affine_transform The first one would probably be slow for large data, and the second one does not seem to offer any other interpolation method except splines.

Response:
Based on your description, you want scipy.ndimage.zoom. Bilinear interpolation would be order=1, nearest is order=0, and cubic is the default (order=3). zoom is specifically for regularly-gridded data that you want to resample to a new resolution. As a quick example:
```
import numpy as np
import scipy.ndimage

x = np.arange(9).reshape(3,3)

print 'Original array:'
print x

print 'Resampled by a factor of 2 with nearest interpolation:'
print scipy.ndimage.zoom(x, 2, order=0)

print 'Resampled by a factor of 2 with bilinear interpolation:'
print scipy.ndimage.zoom(x, 2, order=1)

print 'Resampled by a factor of 2 with cubic interpolation:'
print scipy.ndimage.zoom(x, 2, order=3)
```
And the result:
```
Original array:
[[0 1 2]
 [3 4 5]
 [6 7 8]]
Resampled by a factor of 2 with nearest interpolation:
[[0 0 1 1 2 2]
 [0 0 1 1 2 2]
 [3 3 4 4 5 5]
 [3 3 4 4 5 5]
 [6 6 7 7 8 8]
 [6 6 7 7 8 8]]
Resampled by a factor of 2 with bilinear interpolation:
[[0 0 1 1 2 2]
 [1 2 2 2 3 3]
 [2 3 3 4 4 4]
 [4 4 4 5 5 6]
 [5 5 6 6 6 7]
 [6 6 7 7 8 8]]
Resampled by a factor of 2 with cubic interpolation:
[[0 0 1 1 2 2]
 [1 1 1 2 2 3]
 [2 2 3 3 4 4]
 [4 4 5 5 6 6]
 [5 6 6 7 7 7]
 [6 6 7 7 8 8]]
```
Edit: As Matt S. pointed out, there are a couple of caveats for zooming multi-band images. I'm copying the portion below almost verbatim from one of my earlier answers: Zooming also works for 3D (and nD) arrays. However, be aware that if you zoom by 2x, for example, you'll zoom along all axes.
```
data = np.arange(27).reshape(3,3,3)
print 'Original:\n', data
print 'Zoomed by 2x gives an array of shape:', ndimage.zoom(data, 2).shape
```
This yields:
```
Original:
[[[ 0  1  2]
  [ 3  4  5]
  [ 6  7  8]]

 [[ 9 10 11]
  [12 13 14]
  [15 16 17]]

 [[18 19 20]
  [21 22 23]
  [24 25 26]]]
Zoomed by 2x gives an array of shape: (6, 6, 6)
```
In the case of multi-band images, you usually don't want to interpolate along the "z" axis, creating new bands. If you have something like a 3-band, RGB image that you'd like to zoom, you can do this by specifying a sequence of tuples as the zoom factor:
```
print 'Zoomed by 2x along the last two axes:'
print ndimage.zoom(data, (1, 2, 2))
```
This yields:
```
Zoomed by 2x along the last two axes:
[[[ 0  0  1  1  2  2]
  [ 1  1  1  2  2  3]
  [ 2  2  3  3  4  4]
  [ 4  4  5  5  6  6]
  [ 5  6  6  7  7  7]
  [ 6  6  7  7  8  8]]

 [[ 9  9 10 10 11 11]
  [10 10 10 11 11 12]
  [11 11 12 12 13 13]
  [13 13 14 14 15 15]
  [14 15 15 16 16 16]
  [15 15 16 16 17 17]]

 [[18 18 19 19 20 20]
  [19 19 19 20 20 21]
  [20 20 21 21 22 22]
  [22 22 23 23 24 24]
  [23 24 24 25 25 25]
  [24 24 25 25 26 26]]]
```


Question:
Fitting a Normal distribution to 1D data I have a 1 dimensional array. I can compute the "mean" and "standard deviation" of this sample and plot the "Normal distribution" but I have a problem: I want to plot the data and Normal distribution in the same figure. I dont know how to plot both the data and the normal distribution. Any Idea about "Gaussian probability density function in scipy.stats"?
```
s = np.std(array)
m = np.mean(array)
plt.plot(norm.pdf(array,m,s))
```

Response:
You can use matplotlib to plot the histogram and the PDF (as in the link in @MrE's answer). For fitting and for computing the PDF, you can use scipy.stats.norm, as follows.
```
import numpy as np
from scipy.stats import norm
import matplotlib.pyplot as plt

# Generate some data for this demonstration.
data = norm.rvs(10.0, 2.5, size=500)

# Fit a normal distribution to the data:
mu, std = norm.fit(data)

# Plot the histogram.
plt.hist(data, bins=25, density=True, alpha=0.6, color='g')

# Plot the PDF.
xmin, xmax = plt.xlim()
x = np.linspace(xmin, xmax, 100)
p = norm.pdf(x, mu, std)
plt.plot(x, p, 'k', linewidth=2)
title = "Fit results: mu = %.2f,  std = %.2f" % (mu, std)
plt.title(title)

plt.show()
```
Here's the plot generated by the script:


Question: Mesh grid functions in Python (meshgrid mgrid ogrid ndgrid) I'm looking for a clear comparison of meshgrid-like functions. Unfortunately I don't find it! Numpy http://docs.scipy.org/doc/numpy/reference/ provides mgrid ogrid meshgrid Scitools http://hplgit.github.io/scitools/doc/api/html/index.html provides ndgrid boxgrid Ideally a table summarizing all this would be perfect!

Response:
numpy.meshgrid is modelled after Matlab's meshgrid command. It is used to vectorise functions of two variables, so that you can write
```
x = numpy.array([1, 2, 3])
y = numpy.array([10, 20, 30])
XX, YY = numpy.meshgrid(x, y)
ZZ = XX + YY

ZZ => array([[11, 12, 13],
             [21, 22, 23],
             [31, 32, 33]])
```
So ZZ contains all the combinations of x and y put into the function. When you think about it, meshgrid is a bit superfluous for numpy arrays, as they broadcast. This means you can do
```
XX, YY = numpy.atleast_2d(x, y)
YY = YY.T # transpose to allow broadcasting
ZZ = XX + YY
```
and get the same result. mgrid and ogrid are helper classes which use index notation so that you can create XX and YY in the previous examples directly, without having to use something like linspace. The order in which the output are generated is reversed.
```
YY, XX = numpy.mgrid[10:40:10, 1:4]
ZZ = XX + YY # These are equivalent to the output of meshgrid

YY, XX = numpy.ogrid[10:40:10, 1:4]
ZZ = XX + YY # These are equivalent to the atleast_2d example
```
I am not familiar with the scitools stuff, but ndgrid seems equivalent to meshgrid, while BoxGrid is actually a whole class to help with this kind of generation.


Question:
How do I plot list of tuples? I have the following data set. I would like to use Python or Gnuplot to plot the data. The tuples are of the form (x, y). The Y-axis should be a log axis, that is, log(y). A scatter plot or line plot would be ideal. How can this be done?
```
[(0, 6.0705199999997801e-08), (1, 2.1015700100300739e-08),
 (2, 7.6280656623374823e-09), (3, 5.7348209304555086e-09),
 (4, 3.6812203579604238e-09), (5, 4.1572516753310418e-09)]
```

Response:
If I get your question correctly, you could do something like this.
```

```python
import matplotlib.pyplot as plt
```

```python
testList =[(0, 6.0705199999997801e-08), (1, 2.1015700100300739e-08),
#Output
# (2, 7.6280656623374823e-09), (3, 5.7348209304555086e-09),
# (4, 3.6812203579604238e-09), (5, 4.1572516753310418e-09)]
```

```python
from math import log
```

```python
testList2 = [(elem1, log(elem2)) for elem1, elem2 in testList]
```

```python
testList2
#Output
#[(0, -16.617236475334405), (1, -17.67799605473062), (2, -18.691431541177973), (3, -18.9767093108359), (4, -19.420021520728017), (5, -19.298411635970396)]
```

```python
zip(*testList2)
#Output
#[(0, 1, 2, 3, 4, 5), (-16.617236475334405, -17.67799605473062, -18.691431541177973, -18.9767093108359, -19.420021520728017, -19.298411635970396)]
```

```python
plt.scatter(*zip(*testList2))
```

```python
plt.show()
#Output
#```
#which would give you something like Or as a line plot,
#```
```

```python
plt.plot(*zip(*testList2))
```

```python
plt.show()
#Output
#```
#EDIT - If you want to add a title and labels for the axis, you could do something like
#```
```

```python
plt.scatter(*zip(*testList2))
```

```python
plt.title('Random Figure')
```

```python
plt.xlabel('X-Axis')
```

```python
plt.ylabel('Y-Axis')
```

```python
plt.show()
#Output
#```
#which would give you
```


Question: What is the difference between numpy.fft and scipy.fftpack? Is the later just a synonym of the former, or are they two different implementations of FFT? Which one is better?

Response:
SciPy does more: http://docs.scipy.org/doc/numpy/reference/routines.fft.html http://docs.scipy.org/doc/scipy/reference/fftpack.html# In addition, SciPy exports some of the NumPy features through its own interface, for example if you execute scipy.fftpack.helper.fftfreq and numpy.fft.helper.fftfreq you're actually running the same code. However, SciPy has its own implementations of much functionality. The source has performance benchmarks that compare the original NumPy and new SciPy versions. My archaic laptop shows something like this:
```
Fast Fourier Transform
=================================================
      |    real input     |   complex input
-------------------------------------------------
 size |  scipy  |  numpy  |  scipy  |  numpy
-------------------------------------------------
  100 |    0.07 |    0.06 |    0.06 |    0.07  (secs for 7000 calls)
 1000 |    0.06 |    0.09 |    0.09 |    0.09  (secs for 2000 calls)
  256 |    0.11 |    0.11 |    0.12 |    0.11  (secs for 10000 calls)
  512 |    0.16 |    0.21 |    0.20 |    0.21  (secs for 10000 calls)
 1024 |    0.03 |    0.04 |    0.04 |    0.04  (secs for 1000 calls)
 2048 |    0.05 |    0.09 |    0.08 |    0.08  (secs for 1000 calls)
 4096 |    0.05 |    0.08 |    0.07 |    0.09  (secs for 500 calls)
 8192 |    0.10 |    0.20 |    0.19 |    0.21  (secs for 500 calls)
```
It does seem that SciPy runs significantly faster as the array increases in size, though these are just contrived examples and it would be worth experimenting with both for your particular project. It's worth checking out the source code http://www.scipy.org/Download#head-312ad78cdf85a9ca6fa17a266752069d23f785d1 . Yes those .f files really are Fortran! :-D


Question:
How to extend an array in-place in Numpy? Currently, I have some code like this
```
import numpy as np
ret = np.array([])
for i in range(100000):
  tmp =  get_input(i)
  ret = np.append(ret, np.zeros(len(tmp)))
  ret = np.append(ret, np.ones(fixed_length))
```
I think this code is not efficient as np.append needs to return a copy of the array instead of modify the ret in-place I was wondering whether I can use the extend for a numpy array like this:
```
import numpy as np
from somewhere import np_extend
ret = np.array([])
for i in range(100000):
  tmp =  get_input(i)
  np_extend(ret, np.zeros(len(tmp)))
  np_extend(ret, np.ones(fixed_length))
```
So that the extend would be much more efficient. Does anyone have ideas about this? Thanks!

Response: Imagine a numpy array as occupying one contiguous block of memory. Now imagine other objects, say other numpy arrays, which are occupying the memory just to the left and right of our numpy array. There would be no room to append to or extend our numpy array. The underlying data in a numpy array always occupies a contiguous block of memory. So any request to append to or extend our numpy array can only be satisfied by allocating a whole new larger block of memory, copying the old data into the new block and then appending or extending. So: It will not occur in-place. It will not be efficient.


Question:
How to access sparse matrix elements?
```
type(A)
<class 'scipy.sparse.csc.csc_matrix'>
A.shape
(8529, 60877)
print A[0,:]
  (0, 25)   1.0
  (0, 7422) 1.0
  (0, 26062)    1.0
  (0, 31804)    1.0
  (0, 41602)    1.0
  (0, 43791)    1.0
print A[1,:]
  (0, 7044) 1.0
  (0, 31418)    1.0
  (0, 42341)    1.0
  (0, 47125)    1.0
  (0, 54376)    1.0
print A[:,0]
  #nothing returned
```
Now what I don't understand is that A[1,:] should select elements from the 2nd row, yet I get elements from the 1st row via print A[1,:]. Also, print A[:,0] should return the first column but I get nothing printed. Why?

Response:
A[1,:] is itself a sparse matrix with shape (1, 60877). This is what you are printing, and it has only one row, so all the row coordinates are 0. For example:
```

```python
a = csc_matrix([[1, 0, 0, 0], [0, 0, 10, 11], [0, 0, 0, 99]])
```

```python
a.todense()
#Output
#matrix([[ 1,  0,  0,  0],
#        [ 0,  0, 10, 11],
#        [ 0,  0,  0, 99]], dtype=int64)
```

```python
print(a[1, :])
  (0, 2)    10
  (0, 3)    11
```

```python
print(a)
  (0, 0)    1
  (1, 2)    10
  (1, 3)    11
  (2, 3)    99
```

```python
print(a[1, :].toarray())
[[ 0  0 10 11]]
```
You can select columns, but if there are no nonzero elements in the column, nothing is displayed when it is output with print:
```
```

```python
a[:, 3].toarray()
#Output
#array([[ 0],
#       [11],
#       [99]])
```

```python
print(a[:,3])
  (1, 0)    11
  (2, 0)    99
```

```python
a[:, 1].toarray()
#Output
#array([[0],
#       [0],
#       [0]])
```

```python
print(a[:, 1])
```

```python
```
The last print call shows no output because the column a[:, 1] has no nonzero elements.
```


Question:
AttributeError: 'module' object (scipy) has no attribute 'misc' I updated from ubuntu 12.04 to ubuntu 12.10 and the python module I have written suddenly no longer works with the error message that the module scipy does not have the attribute 'misc'. This worked previously. I am still using python 2.7 after the update. Here is where the code crashes
```
import scipy
scipy.misc.imsave(slice,dat)
```
Any ideas?

Response:
```

```python
import scipy
```

```python
scipy.misc
#Output
#Traceback (most recent call last):
#  File "<stdin>", line 1, in <module>
#AttributeError: 'module' object has no attribute 'misc'
```

```python

```

```python

```

```python
import scipy.misc
```

```python
scipy.misc.imsave
#Output
#<function imsave at 0x19cfa28>
#>>>
#```
#Which seems to be quite common with scipy.
```


Question:
Selecting Pandas Columns by dtype I was wondering if there is an elegant and shorthand way in Pandas DataFrames to select columns by data type (dtype). i.e. Select only int64 columns from a DataFrame. To elaborate, something along the lines of
```
df.select_columns(dtype=float64)
```

Response:
Since 0.14.1 there's a select_dtypes method so you can do this more elegantly/generally.
```

```python
df = pd.DataFrame([[1, 2.2, 'three']], columns=['A', 'B', 'C'])
```

```python
df.select_dtypes(include=['int'])
#Output
#A
#0  1
#```
#To select all numeric types use the numpy dtype numpy.number
#```
```

```python
df.select_dtypes(include=[np.number])
#Output
#A    B
#0  1  2.2
```

```python
df.select_dtypes(exclude=[object])
#Output
#A    B
#0  1  2.2
#```
```


Question:
Find matching rows in 2 dimensional numpy array I would like to get the index of a 2 dimensional Numpy array that matches a row. For example, my array is this:
```
vals = np.array([[0, 0],
                 [1, 0],
                 [2, 0],
                 [0, 1],
                 [1, 1],
                 [2, 1],
                 [0, 2],
                 [1, 2],
                 [2, 2],
                 [0, 3],
                 [1, 3],
                 [2, 3],
                 [0, 0],
                 [1, 0],
                 [2, 0],
                 [0, 1],
                 [1, 1],
                 [2, 1],
                 [0, 2],
                 [1, 2],
                 [2, 2],
                 [0, 3],
                 [1, 3],
                 [2, 3]])
```
I would like to get the index that matches the row [0, 1] which is index 3 and 15. When I do something like numpy.where(vals == [0 ,1]) I get...
```
(array([ 0,  3,  3,  4,  5,  6,  9, 12, 15, 15, 16, 17, 18, 21]), array([0, 0, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 0, 0]))
```
I want index array([3, 15]).

Response:
You need the np.where function to get the indexes:
```

```python
np.where((vals == (0, 1)).all(axis=1))
#Output
#(array([ 3, 15]),)
#```
#Or, as the documentation states: If only condition is given, return condition.nonzero() You could directly call .nonzero() on the array returned by .all:
#```
```

```python
(vals == (0, 1)).all(axis=1).nonzero()
#Output
#(array([ 3, 15]),)
#```
#To dissassemble that:
#```
```

```python
vals == (0, 1)
#Output
#array([[ True, False],
#       [False, False],
#       ...
#       [ True, False],
#       [False, False],
#       [False, False]], dtype=bool)
#```
#and calling the .all method on that array (with axis=1) gives you True where both are True:
#```
```

```python
(vals == (0, 1)).all(axis=1)
#Output
#array([False, False, False,  True, False, False, False, False, False,
#       False, False, False, False, False, False,  True, False, False,
#       False, False, False, False, False, False], dtype=bool)
#```
#and to get which indexes are True:
#```
```

```python
np.where((vals == (0, 1)).all(axis=1))
#Output
#(array([ 3, 15]),)
#```
#or
#```
```

```python
(vals == (0, 1)).all(axis=1).nonzero()
#Output
#(array([ 3, 15]),)
#```
#I find my solution a bit more readable, but as unutbu points out, the following may be faster, and returns the same value as (vals == (0, 1)).all(axis=1):
#```
```

```python
(vals[:, 0] == 0) & (vals[:, 1] == 1)
#Output
#```
```


Question:
transform scipy sparse csr to pandas? I have used the
```
sklearn.preprocessing.OneHotEncoder
```
to transform some data the output is scipy.sparse.csr.csr_matrix how can I merge it back into my original dataframe along with the other columns? I tried to use pd.concat but I get
```
TypeError: cannot concatenate a non-NDFrame object
```
Thanks

Response:
If A is csr_matrix, you can use .toarray() (there's also .todense() that produces a numpy matrix, which is also works for the DataFrame constructor):
```
df = pd.DataFrame(A.toarray())
```
You can then use this with pd.concat().
```
A = csr_matrix([[1, 0, 2], [0, 3, 0]])

  (0, 0)    1
  (0, 2)    2
  (1, 1)    3

<class 'scipy.sparse.csr.csr_matrix'>

pd.DataFrame(A.todense())

   0  1  2
0  1  0  2
1  0  3  0

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 2 entries, 0 to 1
Data columns (total 3 columns):
0    2 non-null int64
1    2 non-null int64
2    2 non-null int64
```
In version 0.20, pandas introduced sparse data structures, including the SparseDataFrame. In pandas 1.0, SparseDataFrame was removed: In older versions of pandas, the SparseSeries and SparseDataFrame classes were the preferred way to work with sparse data. With the advent of extension arrays, these subclasses are no longer needed. Their purpose is better served by using a regular Series or DataFrame with sparse values instead. The migration guide shows how to use these new data structures. For instance, to create a DataFrame from a sparse matrix:
```
from scipy.sparse import csr_matrix

A = csr_matrix([[1, 0, 2], [0, 3, 0]])

df = pd.DataFrame.sparse.from_spmatrix(A, columns=['A', 'B', 'C'])

df

   A  B  C
0  1  0  2
1  0  3  0

df.dtypes
A    Sparse[float64, 0]
B    Sparse[float64, 0]
C    Sparse[float64, 0]
dtype: object
```
Alternatively, you can pass sparse matrices to sklearn to avoid running out of memory when converting back to pandas. Just convert your other data to sparse format by passing a numpy array to the scipy.sparse.csr_matrix constructor and use scipy.sparse.hstack to combine (see docs).


Question:
Convert Pandas dataframe to Sparse Numpy Matrix directly I am creating a matrix from a Pandas dataframe as follows:
```
dense_matrix = np.array(df.as_matrix(columns = None), dtype=bool).astype(np.int)
```
And then into a sparse matrix with:
```
sparse_matrix = scipy.sparse.csr_matrix(dense_matrix)
```
Is there any way to go from a df straight to a sparse matrix? Thanks in advance.

Response:
df.values is a numpy array, and accessing values that way is always faster than np.array.
```
scipy.sparse.csr_matrix(df.values)
```
You might need to take the transpose first, like df.values.T. In DataFrames, the columns are axis 0.


Question: scipy linkage format I have written my own clustering routine and would like to produce a dendrogram. The easiest way to do this would be to use scipy dendrogram function. However, this requires the input to be in the same format that the scipy linkage function produces. I cannot find an example of how the output of this is formatted. I was wondering whether someone out there can enlighten me.

Response:
I agree with https://stackoverflow.com/users/1167475/mortonjt that the documentation does not fully explain the indexing of intermediate clusters, while I do agree with the https://stackoverflow.com/users/1354844/dkar that the format is otherwise precisely explained. Using the example data from this question: Tutorial for scipy.cluster.hierarchy
```
A = np.array([[0.1,   2.5],
              [1.5,   .4 ],
              [0.3,   1  ],
              [1  ,   .8 ],
              [0.5,   0  ],
              [0  ,   0.5],
              [0.5,   0.5],
              [2.7,   2  ],
              [2.2,   3.1],
              [3  ,   2  ],
              [3.2,   1.3]])
```
A linkage matrix can be built using the single (i.e, the closest matching points):
```
z = hac.linkage(a, method="single")

 array([[  7.        ,   9.        ,   0.3       ,   2.        ],
        [  4.        ,   6.        ,   0.5       ,   2.        ],
        [  5.        ,  12.        ,   0.5       ,   3.        ],
        [  2.        ,  13.        ,   0.53851648,   4.        ],
        [  3.        ,  14.        ,   0.58309519,   5.        ],
        [  1.        ,  15.        ,   0.64031242,   6.        ],
        [ 10.        ,  11.        ,   0.72801099,   3.        ],
        [  8.        ,  17.        ,   1.2083046 ,   4.        ],
        [  0.        ,  16.        ,   1.5132746 ,   7.        ],
        [ 18.        ,  19.        ,   1.92353841,  11.        ]])
```
As the documentation explains the clusters below n (here: 11) are simply the data points in the original matrix A. The intermediate clusters going forward, are indexed successively. Thus, clusters 7 and 9 (the first merge) are merged into cluster 11, clusters 4 and 6 into 12. Then observe line three, merging clusters 5 (from A) and 12 (from the not-shown intermediate cluster 12) resulting with a Within-Cluster Distance (WCD) of 0.5. The single method entails that the new WCS is 0.5, which is the distance between A[5] and the closest point in cluster 12, A[4] and A[6]. Let's check:
```

```python
norm([a[5]-a[4]])
#Output
#0.70710678118654757
```

```python
norm([a[5]-a[6]])
#Output
#0.5
#```
#This cluster should now be intermediate cluster 13, which subsequently is merged with A[2]. Thus, the new distance should be the closest between the points A[2] and A[4,5,6].
#```
```

```python
norm([a[2]-a[4]])
#Output
#1.019803902718557
```

```python
norm([a[2]-a[5]])
#Output
#0.58309518948452999
```

```python
norm([a[2]-a[6]])
#Output
#0.53851648071345048
#```
#Which, as can be seen also checks out, and explains the intermediate format of new clusters.
```


Question: How to use numpy.void type I loaded a MATLAB .mat file via scipy.io.loadmat and it gave me a list of numpy.void objects. What are they, how can they be used and where can I get some reference documentation on them?

Response:
According to the numpy documentation: http://docs.scipy.org/doc/numpy/reference/arrays.dtypes.html, numpy.void types are defined as flexible data types. Basically, these are data types where there is no pre-defined type associated to the variable(s) you're looking at. If you look at numpy, you have data types such as float, uint8, bool, string, etc. void is to accommodate for more generic and flexible types and are for those data types that don't necessary fall into any one of these pre-defined data types. This situation is mostly encountered when you're loading in a struct where each element has multiple data types associated with multiple fields. Each structure element could have a combination of different data types, and the amalgamation of all of these data types to represent an instance of this structure element thus leads us to numpy.void. With the documentation, you can certainly do the same operations like you would with any other data type. Take a look at the generic data type methods here: http://docs.scipy.org/doc/numpy/reference/generated/numpy.generic.html#numpy.generic . In fact, all numpy data types are derived from this generic class, including numpy.void. In the first link I provided at the beginning of this post, it shows a good example of how to create a custom record type, where a record is a combination of a tuple of numbers and a string. When creating a list of these records, each type in the list is of type numpy.void and it demonstrates that a record is of this data type. However, bear in mind that this record list has a data type that is of this record, but each element of this list will be of type numpy.void. However, as a matter of self-containment, let's re-create the example here: Let's create a custom record type where it has two fields associated for each variable you create: A 16-bit string with a field named name A 2-element tuple of floating point numbers that are 64-bits each, with a field named grades As such, you'd do something like:
```
import numpy as np
dt = np.dtype([('name', np.str_, 16), ('grades', np.float64, (2,))])
```
As such, let's create an example list of two elements and instantiate their fields:
```
x = np.array([('Sarah', (8.0, 7.0)), ('John', (6.0, 7.0))], dtype=dt)
```
Because we made this list into a numpy.array, we expect its data type to be so:
```
type(x)
```
We get:
```
<type 'numpy.ndarray'>
```
Remember, the list itself is a numpy.array, but not the individual elements. To access the second element of this list, which is the second record, we do:
```
x[1]
```
We get:
```
('John', [6.0, 7.0])
```
To check the type of the second record, we do:
```
type(x[1])
```
We get:
```
<type 'numpy.void'> # As expected
```
Some additional bonuses for you To access the name of the second record, we do:
```
x[1]['name']
```
We get:
```
'John'
```
To access the grades of the second record, we do:
```
x[1]['grades']
```
We get:
```
array([ 6.,  7.])
```
To check the type of the name inside the second record, we do:
```
type(x[1]['name'])
```
We get:
```
<type 'numpy.string_'>
```
To check the type of the grades inside the second record, we do:
```
type(x[1]['grades'])
```
We get:
```
<type 'numpy.ndarray'>
```
Take note that each element in this list is of type numpy.void. However, the individual fields for each element in our list is either a tuple of numbers, or a string. The collection of these elements together is of type numpy.void.


Question:
P-value from Chi sq test statistic in Python I have computed a test statistic that is distributed as a chi square with 1 degree of freedom, and want to find out what P-value this corresponds to using python. I'm a python and maths/stats newbie so I think what I want here is the probability denisty function for the chi2 distribution from SciPy. However, when I use this like so:
```
from scipy import stats
stats.chi2.pdf(3.84 , 1)
0.029846
```
However some googling and talking to some colleagues who know maths but not python have said it should be 0.05. Any ideas? Cheers, Davy

Response:
Quick refresher here: Probability Density Function: think of it as a point value; how dense is the probability at a given point? Cumulative Distribution Function: this is the mass of probability of the function up to a given point; what percentage of the distribution lies on one side of this point? In your case, you took the PDF, for which you got the correct answer. If you try 1 - CDF:
```

```python
1 - stats.chi2.cdf(3.84, 1)
#Output
#0.050043521248705147
#```
#PDF CDF
```


Question: Scipy sparse... arrays? So, I'm doing some Kmeans classification using numpy arrays that are quite sparse-- lots and lots of zeroes. I figured that I'd use scipy's 'sparse' package to reduce the storage overhead, but I'm a little confused about how to create arrays, not matrices. I've gone through this tutorial on how to create sparse matrices: http://www.scipy.org/SciPy_Tutorial#head-c60163f2fd2bab79edd94be43682414f18b90df7 To mimic an array, I just create a 1xN matrix, but as you may guess, Asp.dot(Bsp) doesn't quite work because you can't multiply two 1xN matrices. I'd have to transpose each array to Nx1, and that's pretty lame, since I'd be doing it for every dot-product calculation. Next up, I tried to create an NxN matrix where column 1 == row 1 (such that you can multiply two matrices and just take the top-left corner as the dot product), but that turned out to be really inefficient. I'd love to use scipy's sparse package as a magic replacement for numpy's array(), but as yet, I'm not really sure what to do. Any advice?

Response:
Use a scipy.sparse format that is row or column based: csc_matrix and csr_matrix. These use efficient, C implementations under the hood (including multiplication), and transposition is a no-op (esp. if you call transpose(copy=False)), just like with numpy arrays. EDIT: some timings via ipython:
```
import numpy, scipy.sparse
n = 100000
x = (numpy.random.rand(n) * 2).astype(int).astype(float) # 50% sparse vector
x_csr = scipy.sparse.csr_matrix(x)
x_dok = scipy.sparse.dok_matrix(x.reshape(x_csr.shape))
```
Now x_csr and x_dok are 50% sparse:
```
print repr(x_csr)
<1x100000 sparse matrix of type '<type 'numpy.float64'>'
        with 49757 stored elements in Compressed Sparse Row format>
```
And the timings:
```
timeit numpy.dot(x, x)
10000 loops, best of 3: 123 us per loop

timeit x_dok * x_dok.T
1 loops, best of 3: 1.73 s per loop

timeit x_csr.multiply(x_csr).sum()
1000 loops, best of 3: 1.64 ms per loop

timeit x_csr * x_csr.T
100 loops, best of 3: 3.62 ms per loop
```
So it looks like I told a lie. Transposition is very cheap, but there is no efficient C implementation of csr * csc (in the latest scipy 0.9.0). A new csr object is constructed in each call :-( As a hack (though scipy is relatively stable these days), you can do the dot product directly on the sparse data:
```
timeit numpy.dot(x_csr.data, x_csr.data)
10000 loops, best of 3: 62.9 us per loop
```
Note this last approach does a numpy dense multiplication again. The sparsity is 50%, so it's actually faster than dot(x, x) by a factor of 2.


Question: How to read images into a script without using using imageio or scikit image? I am trying to read a png image in python. The imread function in scipy is being deprecated and they recommend using imageio library. However, I am would rather restrict my usage of external libraries to scipy, numpy and matplotlib libraries. Thus, using imageio or scikit image is not a good option for me. Are there any methods in python or scipy, numpy or matplotlib to read images, which are not being deprecated?

Response:
With matplotlib you can use (as shown in the matplotlib documentation)
```
import matplotlib.pyplot as plt
import matplotlib.image as mpimg

img=mpimg.imread('image_name.png')
```
And plot the image if you want
```
imgplot = plt.imshow(img)
```


Question: Python finite difference functions? I've been looking around in Numpy/Scipy for modules containing finite difference functions. However, the closest thing I've found is numpy.gradient(), which is good for 1st-order finite differences of 2nd order accuracy, but not so much if you're wanting higher-order derivatives or more accurate methods. I haven't even found very many specific modules for this sort of thing; most people seem to be doing a "roll-your-own" thing as they need them. So my question is if anyone knows of any modules (either part of Numpy/Scipy or a third-party module) that are specifically dedicated to higher-order (both in accuracy and derivative) finite difference methods. I've got my own code that I'm working on, but it's currently kind of slow, and I'm not going to attempt to optimize it if there's something already available. Note that I am talking about finite differences, not derivatives. I've seen both scipy.misc.derivative() and Numdifftools, which take the derivative of an analytical function, which I don't have.

Response:
One way to do this quickly is by convolution with the derivative of a gaussian kernel. The simple case is a convolution of your array with [-1, 1] which gives exactly the simple finite difference formula. Beyond that, (f*g)'= f'*g = f*g' where the * is convolution, so you end up with your derivative convolved with a plain gaussian, so of course this will smooth your data a bit, which can be minimized by choosing the smallest reasonable kernel.
```
import numpy as np
from scipy import ndimage
import matplotlib.pyplot as plt

#Data:
x = np.linspace(0,2*np.pi,100)
f = np.sin(x) + .02*(np.random.rand(100)-.5)

#Normalization:
dx = x[1] - x[0] # use np.diff(x) if x is not uniform
dxdx = dx**2

#First derivatives:
df = np.diff(f) / dx
cf = np.convolve(f, [1,-1]) / dx
gf = ndimage.gaussian_filter1d(f, sigma=1, order=1, mode='wrap') / dx

#Second derivatives:
ddf = np.diff(f, 2) / dxdx
ccf = np.convolve(f, [1, -2, 1]) / dxdx
ggf = ndimage.gaussian_filter1d(f, sigma=1, order=2, mode='wrap') / dxdx

#Plotting:
plt.figure()
plt.plot(x, f, 'k', lw=2, label='original')
plt.plot(x[:-1], df, 'r.', label='np.diff, 1')
plt.plot(x, cf[:-1], 'r--', label='np.convolve, [1,-1]')
plt.plot(x, gf, 'r', label='gaussian, 1')
plt.plot(x[:-2], ddf, 'g.', label='np.diff, 2')
plt.plot(x, ccf[:-2], 'g--', label='np.convolve, [1,-2,1]')
plt.plot(x, ggf, 'g', label='gaussian, 2')
```
Since you mentioned np.gradient I assumed you had at least 2d arrays, so the following applies to that: This is built into the scipy.ndimage package if you want to do it for ndarrays. Be cautious though, because of course this doesn't give you the full gradient but I believe the product of all directions. Someone with better expertise will hopefully speak up. Here's an example:
```
from scipy import ndimage

x = np.linspace(0,2*np.pi,100)
sine = np.sin(x)

im = sine * sine[...,None]
d1 = ndimage.gaussian_filter(im, sigma=5, order=1, mode='wrap')
d2 = ndimage.gaussian_filter(im, sigma=5, order=2, mode='wrap')

plt.figure()

plt.subplot(131)
plt.imshow(im)
plt.title('original')

plt.subplot(132)
plt.imshow(d1)
plt.title('first derivative')

plt.subplot(133)
plt.imshow(d2)
plt.title('second derivative')
```
Use of the gaussian_filter1d allows you to take a directional derivative along a certain axis:
```
imx = im * x
d2_0 = ndimage.gaussian_filter1d(imx, axis=0, sigma=5, order=2, mode='wrap')
d2_1 = ndimage.gaussian_filter1d(imx, axis=1, sigma=5, order=2, mode='wrap')

plt.figure()
plt.subplot(131)
plt.imshow(imx)
plt.title('original')
plt.subplot(132)
plt.imshow(d2_0)
plt.title('derivative along axis 0')
plt.subplot(133)
plt.imshow(d2_1)
plt.title('along axis 1')
```
The first set of results above are a little confusing to me (peaks in the original show up as peaks in the second derivative when the curvature should point down). Without looking further into how the 2d version works, I can only really recomend the 1d version. If you want the magnitude, simply do something like:
```
d2_mag = np.sqrt(d2_0**2 + d2_1**2)
```


Question: How do I transform a "SciPy sparse matrix" to a "NumPy matrix"? I am using a python function called "incidence_matrix(G)", which returns the incident matrix of graph. It is from Networkx package. The problem that I am facing is the return type of this function is "Scipy Sparse Matrix". I need to have the Incident matrix in the format of numpy matrix or array. I was wondering if there is any easy way of doing that or not? Or is there any built-in function that can do this transformation for me or not? Thanks

Response: The scipy.sparse.*_matrix has several useful methods, for example, if a is e.g. scipy.sparse.csr_matrix: a.toarray() - Return a dense ndarray representation of this matrix. (numpy.array, recommended) a.todense() - Return a dense matrix representation of this matrix. (numpy.matrix) Previously, these methods had shorthands (.A for .toarray(), and .M for .todense()), but these have been or will be deprecated as of Scipy v1.14.0.


Question:
Iterating through a scipy.sparse vector (or matrix) I'm wondering what the best way is to iterate nonzero entries of sparse matrices with scipy.sparse. For example, if I do the following:
```
from scipy.sparse import lil_matrix

x = lil_matrix( (20,1) )
x[13,0] = 1
x[15,0] = 2

c = 0
for i in x:
  print c, i
  c = c+1
```
the output is
```
0
1
2
3
4
5
6
7
8
9
10
11
12
13   (0, 0) 1.0
14
15   (0, 0) 2.0
16
17
18
19
```
so it appears the iterator is touching every element, not just the nonzero entries. I've had a look at the API http://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.lil_matrix.html and searched around a bit, but I can't seem to find a solution that works.

Response:
Edit: bbtrb's method (using coo_matrix) is much faster than my original suggestion, using nonzero. Sven Marnach's suggestion to use itertools.izip also improves the speed. Current fastest is using_tocoo_izip:
```
import scipy.sparse
import random
import itertools

def using_nonzero(x):
    rows,cols = x.nonzero()
    for row,col in zip(rows,cols):
        ((row,col), x[row,col])

def using_coo(x):
    cx = scipy.sparse.coo_matrix(x)
    for i,j,v in zip(cx.row, cx.col, cx.data):
        (i,j,v)

def using_tocoo(x):
    cx = x.tocoo()
    for i,j,v in zip(cx.row, cx.col, cx.data):
        (i,j,v)

def using_tocoo_izip(x):
    cx = x.tocoo()
    for i,j,v in itertools.izip(cx.row, cx.col, cx.data):
        (i,j,v)

N=200
x = scipy.sparse.lil_matrix( (N,N) )
for _ in xrange(N):
    x[random.randint(0,N-1),random.randint(0,N-1)]=random.randint(1,100)
```
yields these timeit results:
```
% python -mtimeit -s'import test' 'test.using_tocoo_izip(test.x)'
1000 loops, best of 3: 670 usec per loop
% python -mtimeit -s'import test' 'test.using_tocoo(test.x)'
1000 loops, best of 3: 706 usec per loop
% python -mtimeit -s'import test' 'test.using_coo(test.x)'
1000 loops, best of 3: 802 usec per loop
% python -mtimeit -s'import test' 'test.using_nonzero(test.x)'
100 loops, best of 3: 5.25 msec per loop
```


Question: Scipy Normaltest how is it used? I need to use normaltest in scipy for testing if the dataset is normal distributet. But I cant seem to find any good examples how to use scipy.stats.normaltest. My dataset has more than 100 values.

Response:
```

```python
import scipy.stats as stats
```

```python
x = stats.norm.rvs(size = 100)
```

```python
stats.normaltest(x)
#Output
#(1.627533590094232, 0.44318552909231262)
#```
#normaltest returns a 2-tuple of the chi-squared statistic, and the associated p-value. Given the null hypothesis that x came from a normal distribution, the p-value represents the probability that a chi-squared statistic that large (or larger) would be seen. If the p-val is very small, it means it is unlikely that the data came from a normal distribution. For example:
#```
```

```python
y = stats.uniform.rvs(size = 100)
```

```python
stats.normaltest(y)
#Output
#(31.487039026711866, 1.4543748291516241e-07)
#```
```


Question: Difference between scipy.spatial.KDTree and scipy.spatial.cKDTree What is the difference between these two algorithms?

Response: From SciPy 1.6 on, cKDTree and KDTree are identical, and you should prefer KDTree if you aren't worried about pre-1.6 compatibility. Before SciPy 1.6, cKDTree was a subset of KDTree, implemented in C++ wrapped in Cython, so therefore faster.


Question:
Concatenate sparse matrices in Python using SciPy/Numpy What would be the most efficient way to concatenate sparse matrices in Python using SciPy/Numpy? Here I used the following:
```

```python
np.hstack((X, X2))
#Output
#array([ <49998x70000 sparse matrix of type '<class 'numpy.float64'>'
#        with 1135520 stored elements in Compressed Sparse Row format>,
#        <49998x70000 sparse matrix of type '<class 'numpy.int64'>'
#        with 1135520 stored elements in Compressed Sparse Row format>],
#       dtype=object)
#```
#I would like to use both predictors in a regression, but the current format is obviously not what I'm looking for. Would it be possible to get the following:
#```
#<49998x1400000 sparse matrix of type '<class 'numpy.float64'>'
#     with 2271040 stored elements in Compressed Sparse Row format>
#```
#It is too large to be converted to a deep format.
```

Response:
You can use the scipy.sparse.hstack to concatenate sparse matrices with the same number of rows (horizontal concatenation):
```
from scipy.sparse import hstack
hstack((X, X2))
```
Similarly, you can use scipy.sparse.vstack to concatenate sparse matrices with the same number of columns (vertical concatenation). Using numpy.hstack or numpy.vstack will create an array with two sparse matrix objects.


Question:
Plot Normal distribution with Matplotlib [duplicate] This question already has answers here: How to plot normal distribution (10 answers) Closed 3 years ago. please help me to plot the normal distribution of the folowing data: DATA:
```
import numpy as np
import matplotlib.pyplot as plt
from scipy.stats import norm

h = [186, 176, 158, 180, 186, 168, 168, 164, 178, 170, 189, 195, 172,
     187, 180, 186, 185, 168, 179, 178, 183, 179, 170, 175, 186, 159,
     161, 178, 175, 185, 175, 162, 173, 172, 177, 175, 172, 177, 180]

std = np.std(h)
mean = np.mean(h)
plt.plot(norm.pdf(h,mean,std))
```
output:
```
Standard Deriviation = 8.54065575872
mean = 176.076923077
```
the plot is incorrect, what is wrong with my code?

Response:
Note: This solution is using pylab, not matplotlib.pyplot You may try using hist to put your data info along with the fitted curve as below:
```
import numpy as np
import scipy.stats as stats
import pylab as pl

h = sorted([186, 176, 158, 180, 186, 168, 168, 164, 178, 170, 189, 195, 172,
     187, 180, 186, 185, 168, 179, 178, 183, 179, 170, 175, 186, 159,
     161, 178, 175, 185, 175, 162, 173, 172, 177, 175, 172, 177, 180])  #sorted

fit = stats.norm.pdf(h, np.mean(h), np.std(h))  #this is a fitting indeed

pl.plot(h,fit,'-o')

pl.hist(h,normed=True)      #use this to draw histogram of your data

pl.show()                   #use may also need add this
```


Question:
How to convert a column or row matrix to a diagonal matrix in Python? I have a row vector A, A = [a1 a2 a3 ..... an] and I would like to create a diagonal matrix, B = diag(a1, a2, a3, ....., an) with the elements of this row vector. How can this be done in Python? UPDATE This is the code to illustrate the problem:
```
import numpy as np
a = np.matrix([1,2,3,4])
d = np.diag(a)
print (d)
```
the output of this code is [1], but my desired output is:
```
[[1 0 0 0]
 [0 2 0 0]
 [0 0 3 0]
 [0 0 0 4]]
```

Response:
You can use diag method:
```
import numpy as np

a = np.array([1,2,3,4])
d = np.diag(a)
# or simpler: d = np.diag([1,2,3,4])

print(d)
```
Results in:
```
[[1 0 0 0]
 [0 2 0 0]
 [0 0 3 0]
 [0 0 0 4]]
```
If you have a row vector, you can do this:
```
a = np.array([[1, 2, 3, 4]])
d = np.diag(a[0])
```
Results in:
```
[[1 0 0 0]
 [0 2 0 0]
 [0 0 3 0]
 [0 0 0 4]]
```
For the given matrix in the question:
```
import numpy as np
a = np.matrix([1,2,3,4])
d = np.diag(a.A1)
print (d)
```
Result is again:
```
[[1 0 0 0]
 [0 2 0 0]
 [0 0 3 0]
 [0 0 0 4]]
```


Question:
What's the difference between KFold and ShuffleSplit CV? It seems like KFold generates the same values every time the object is iterated over, while Shuffle Split generates different indices every time. Is this correct? If so, what are the uses for one over the other?
```
cv = cross_validation.KFold(10, n_folds=2,shuffle=True,random_state=None)
cv2 = cross_validation.ShuffleSplit(10,n_iter=2,test_size=0.5)
print(list(iter(cv)))
print(list(iter(cv)))
print(list(iter(cv2)))
print(list(iter(cv2)))
```
Yields the following output:
```
[(array([1, 3, 5, 8, 9]), array([0, 2, 4, 6, 7])), (array([0, 2, 4, 6, 7]), array([1, 3, 5, 8, 9]))]
[(array([1, 3, 5, 8, 9]), array([0, 2, 4, 6, 7])), (array([0, 2, 4, 6, 7]), array([1, 3, 5, 8, 9]))]
[(array([4, 6, 3, 2, 7]), array([8, 1, 9, 0, 5])), (array([3, 6, 7, 0, 5]), array([9, 1, 8, 4, 2]))]
[(array([3, 0, 2, 1, 7]), array([5, 6, 9, 4, 8])), (array([0, 7, 1, 3, 8]), array([6, 2, 5, 4, 9]))]
```

Response: Difference in KFold and ShuffleSplit output KFold will divide your data set into prespecified number of folds, and every sample must be in one and only one fold. A fold is a subset of your dataset. ShuffleSplit will randomly sample your entire dataset during each iteration to generate a training set and a test set. The test_size and train_size parameters control how large the test and training test set should be for each iteration. Since you are sampling from the entire dataset during each iteration, values selected during one iteration, could be selected again during another iteration. Summary: ShuffleSplit works iteratively, KFold just divides the dataset into k folds. Difference when doing validation In KFold, during each round you will use one fold as the test set and all the remaining folds as your training set. However, in ShuffleSplit, during each round n you should only use the training and test set from iteration n. As your data set grows, cross validation time increases, making shufflesplits a more attractive alternate. If you can train your algorithm, with a certain percentage of your data as opposed to using all k-1 folds, ShuffleSplit is an attractive option.


Question: Python p-value from t-statistic I have some t-values and degrees of freedom and want to find the p-values from them (it's two-tailed). In the real world I would use a t-test table in the back of a Statistics textbook; how do I do the equivalent in Python? e.g. t-lookup(5, 7) = 0.00245 or something like that. I know in SciPy if I had arrays I could do scipy.stats.ttest_ind, but I don't. I just have t-statistics and degrees of freedom.

Response:
From http://docs.scipy.org/doc/scipy/reference/tutorial/stats.html As an exercise, we can calculate our ttest also directly without using the provided function, which should give us the same answer, and so it does:
```
tt = (sm-m)/np.sqrt(sv/float(n))  # t-statistic for mean
pval = stats.t.sf(np.abs(tt), n-1)*2  # two-sided pvalue = Prob(abs(t)>tt)
print 't-statistic = %6.3f pvalue = %6.4f' % (tt, pval)
t-statistic =  0.391 pvalue = 0.6955
```


Question: Scipy sparse matrices - purpose and usage of different implementations Scipy has many different types of sparse matrices available. What are the most important differences between these types, and what is the difference in their intended usage? I'm developing a code in python based on a sample code1 in Matlab. One section of the code utilizes sparse matrices - which seem to have a single (annoying) type in Matlab, and I'm trying to figure out which type I should use2 in python. 1: This is for a class. Most people are doing the project in Matlab, but I like to create unnecessary work and confusion --- apparently. 2: This is an academic question: I have the code working properly with the 'CSR' format, but I'm interesting in knowing what the optimal usages are.

Response: Sorry if I'm not answering this completely enough, but hopefully I can provide some insight. CSC (Compressed Sparse Column) and CSR (Compressed Sparse Row) are more compact and efficient, but difficult to construct "from scratch". Coo (Coordinate) and DOK (Dictionary of Keys) are easier to construct, and can then be converted to CSC or CSR via matrix.tocsc() or matrix.tocsr(). CSC is more efficient at accessing column-vectors or column operations, generally, as it is stored as arrays of columns and their value at each row. CSR matrices are the opposite; stored as arrays of rows and their values at each column, and are more efficient at accessing row-vectors or row operations.


Question:
Use Distance Matrix in scipy.cluster.hierarchy.linkage()? I have a distance matrix n*n M where M_ij is the distance between object_i and object_j. So as expected, it takes the following form:
```
/  0     M_01    M_02    ...    M_0n\
   | M_10    0      M_12    ...    M_1n |
   | M_20   M_21     0      ...    M2_n |
   |                ...                 |
   \ M_n0   M_n2    M_n2    ...      0 /
```
Now I wish to cluster these n objects with hierarchical clustering. Python has an implementation of this called scipy.cluster.hierarchy.linkage(y, method='single', metric='euclidean'). Its documentation says: y must be a {n \choose 2} sized vector where n is the number of original observations paired in the distance matrix. y : ndarray A condensed or redundant distance matrix. A condensed distance matrix is a flat array containing the upper triangular of the distance matrix. This is the form that pdist returns. Alternatively, a collection of m observation vectors in n dimensions may be passed as an m by n array. I am confused by this description of y. Can I directly feed my M in as the input y? Update @hongbo-zhu-cn has raised this issue up in GitHub. This is exactly what I am concerning about. However, as a newbie to GitHub, I don't know how it works and therefore have no idea how this issue is dealt with.

Response:
It seems that indeed we cannot directly pass the redundant square matrix in, although the documentation claims we can do so. To benefit anyone who faces the same problem in the future, I write my solution as an additional answer here. So the copy-and-paste guys can just proceed with the clustering. Use the following snippet to condense the matrix and happily proceed.
```
import scipy.spatial.distance as ssd
# convert the redundant n*n square matrix form into a condensed nC2 array
    distArray = ssd.squareform(distMatrix) # distArray[{n choose 2}-{n-i choose 2} + (j-i-1)] is the distance between points i and j
```
Please correct me if I am wrong.


Question:
Plotting power spectrum in python I have an array with 301 values, which were gathered from a movie clip with 301 frames. This means 1 value from 1 frame. The movie clip is running at 30 fps, so is in fact 10 sec long Now I would like to get the power spectrum of this "signal" ( with the right Axis). I tried:
```
X = fft(S_[:,2]);
 pl.plot(abs(X))
 pl.show()
```
I also tried:
```
X = fft(S_[:,2]);
 pl.plot(abs(X)**2)
 pl.show()
```
Though I don't think this is the real spectrum. the signal: The spectrum: The power spectrum : Can anyone provide some help with this ? I would like to have a plot in Hz.

Response:
Numpy has a convenience function, np.fft.fftfreq to compute the frequencies associated with FFT components:
```
from __future__ import division
import numpy as np
import matplotlib.pyplot as plt

data = np.random.rand(301) - 0.5
ps = np.abs(np.fft.fft(data))**2

time_step = 1 / 30
freqs = np.fft.fftfreq(data.size, time_step)
idx = np.argsort(freqs)

plt.plot(freqs[idx], ps[idx])
```
Note that the largest frequency you see in your case is not 30 Hz, but
```

```python
max(freqs)
#Output
#14.950166112956811
#```
#You never see the sampling frequency in a power spectrum. If you had had an even number of samples, then you would have reached the Nyquist frequency, 15 Hz in your case (although numpy would have calculated it as -15).
```


Question: kalman 2d filter in python My input is 2d (x,y) time series of a dot moving on a screen for a tracker software. It has some noise I want to remove using Kalman filter. Does someone can point me for a python code for Kalman 2d filter? In scipy cookbook I found only a 1d example: http://www.scipy.org/Cookbook/KalmanFiltering I saw there is implementation for Kalman filter in OpenCV, but couldn't find code examples. Thanks!

Response:
Here is my implementation of the Kalman filter based on the equations given on wikipedia. Please be aware that my understanding of Kalman filters is very rudimentary so there are most likely ways to improve this code. (For example, it suffers from the numerical instability problem discussed here. As I understand it, this only affects the numerical stability when Q, the motion noise, is very small. In real life, the noise is usually not small, so fortunately (at least for my implementation) in practice the numerical instability does not show up.) In the example below, kalman_xy assumes the state vector is a 4-tuple: 2 numbers for the location, and 2 numbers for the velocity. The F and H matrices have been defined specifically for this state vector: If x is a 4-tuple state, then
```
new_x = F * x
position = H * x
```
It then calls kalman, which is the generalized Kalman filter. It is general in the sense it is still useful if you wish to define a different state vector -- perhaps a 6-tuple representing location, velocity and acceleration. You just have to define the equations of motion by supplying the appropriate F and H.
```
import numpy as np
import matplotlib.pyplot as plt

def kalman_xy(x, P, measurement, R,
              motion = np.matrix('0. 0. 0. 0.').T,
              Q = np.matrix(np.eye(4))):
    """
    Parameters:
    x: initial state 4-tuple of location and velocity: (x0, x1, x0_dot, x1_dot)
    P: initial uncertainty convariance matrix
    measurement: observed position
    R: measurement noise
    motion: external motion added to state vector x
    Q: motion noise (same shape as P)
    """
    return kalman(x, P, measurement, R, motion, Q,
                  F = np.matrix('''
                      1. 0. 1. 0.;
                      0. 1. 0. 1.;
                      0. 0. 1. 0.;
                      0. 0. 0. 1.
                      '''),
                  H = np.matrix('''
                      1. 0. 0. 0.;
                      0. 1. 0. 0.'''))

def kalman(x, P, measurement, R, motion, Q, F, H):
    '''
    Parameters:
    x: initial state
    P: initial uncertainty convariance matrix
    measurement: observed position (same shape as H*x)
    R: measurement noise (same shape as H)
    motion: external motion added to state vector x
    Q: motion noise (same shape as P)
    F: next state function: x_prime = F*x
    H: measurement function: position = H*x

    Return: the updated and predicted new values for (x, P)

    See also http://en.wikipedia.org/wiki/Kalman_filter

    This version of kalman can be applied to many different situations by
    appropriately defining F and H
    '''
    # UPDATE x, P based on measurement m
    # distance between measured and current position-belief
    y = np.matrix(measurement).T - H * x
    S = H * P * H.T + R  # residual convariance
    K = P * H.T * S.I    # Kalman gain
    x = x + K*y
    I = np.matrix(np.eye(F.shape[0])) # identity matrix
    P = (I - K*H)*P

    # PREDICT x, P based on motion
    x = F*x + motion
    P = F*P*F.T + Q

    return x, P

def demo_kalman_xy():
    x = np.matrix('0. 0. 0. 0.').T
    P = np.matrix(np.eye(4))*1000 # initial uncertainty

    N = 20
    true_x = np.linspace(0.0, 10.0, N)
    true_y = true_x**2
    observed_x = true_x + 0.05*np.random.random(N)*true_x
    observed_y = true_y + 0.05*np.random.random(N)*true_y
    plt.plot(observed_x, observed_y, 'ro')
    result = []
    R = 0.01**2
    for meas in zip(observed_x, observed_y):
        x, P = kalman_xy(x, P, meas, R)
        result.append((x[:2]).tolist())
    kalman_x, kalman_y = zip(*result)
    plt.plot(kalman_x, kalman_y, 'g-')
    plt.show()

demo_kalman_xy()
```
The red dots show the noisy position measurements, the green line shows the Kalman predicted positions.


Question:
how to plot and annotate hierarchical clustering dendrograms in scipy/matplotlib I'm using dendrogram from scipy to plot hierarchical clustering using matplotlib as follows:
```
mat = array([[1, 0.5, 0.9],
             [0.5, 1, -0.5],
             [0.9, -0.5, 1]])
plt.subplot(1,2,1)
plt.title("mat")
dist_mat = mat
linkage_matrix = linkage(dist_mat,
                         "single")
print "linkage2:"
print linkage(1-dist_mat, "single")
dendrogram(linkage_matrix,
           color_threshold=1,
           labels=["a", "b", "c"],
           show_leaf_counts=True)
plt.subplot(1,2,2)
plt.title("1 - mat")
dist_mat = 1 - mat
linkage_matrix = linkage(dist_mat,
                         "single")
dendrogram(linkage_matrix,
           color_threshold=1,
           labels=["a", "b", "c"],
           show_leaf_counts=True)
```
My questions are: first, why does mat and 1-mat give identical clusterings here? and second, how can I annotate the distance along each branch of the tree using dendrogram so that the distances between pairs of nodes can be compared? finally it seems that show_leaf_counts flag is ignored, is there a way to turn it on so that the number of objects in each class is shown? thanks.

Response:
The input to linkage() is either an n x m array, representing n points in m-dimensional space, or a one-dimensional array containing the condensed distance matrix. In your example, mat is 3 x 3, so you are clustering three 3-d points. Clustering is based on the distance between these points. Why does mat and 1-mat give identical clusterings here? The arrays mat and 1-mat produce the same clustering because the clustering is based on distances between the points, and neither a reflection (-mat) nor a translation (mat + offset) of the entire data set change the relative distances between the points. How can I annotate the distance along each branch of the tree using dendrogram so that the distances between pairs of nodes can be compared? In the code below, I show how you can use the data returned by dendrogram to label the horizontal segments of the diagram with the corresponding distance. The values associated with the keys icoord and dcoord give the x and y coordinates of each three-segment inverted-U of the figure. In augmented_dendrogram this data is used to add a label of the distance (i.e. y value) of each horizontal line segment in dendrogram.
```
from scipy.cluster.hierarchy import dendrogram
import matplotlib.pyplot as plt

def augmented_dendrogram(*args, **kwargs):

    ddata = dendrogram(*args, **kwargs)

    if not kwargs.get('no_plot', False):
        for i, d in zip(ddata['icoord'], ddata['dcoord']):
            x = 0.5 * sum(i[1:3])
            y = d[1]
            plt.plot(x, y, 'ro')
            plt.annotate("%.3g" % y, (x, y), xytext=(0, -8),
                         textcoords='offset points',
                         va='top', ha='center')

    return ddata
```
For your mat array, the augmented dendrogram is So point 'a' and 'c' are 1.01 units apart, and point 'b' is 1.57 units from the cluster ['a', 'c']. It seems that show_leaf_counts flag is ignored, is there a way to turn it on so that the number of objects in each class is shown? The flag show_leaf_counts only applies when not all the original data points are shown as leaves. For example, when trunc_mode = "lastp", only the last p nodes are show. Here's an example with 100 points:
```
import numpy as np
from scipy.cluster.hierarchy import linkage
import matplotlib.pyplot as plt
from augmented_dendrogram import augmented_dendrogram

# Generate a random sample of `n` points in 2-d.
np.random.seed(12312)
n = 100
x = np.random.multivariate_normal([0, 0], np.array([[4.0, 2.5], [2.5, 1.4]]),
                                  size=(n,))

plt.figure(1, figsize=(6, 5))
plt.clf()
plt.scatter(x[:, 0], x[:, 1])
plt.axis('equal')
plt.grid(True)

linkage_matrix = linkage(x, "single")

plt.figure(2, figsize=(10, 4))
plt.clf()

plt.subplot(1, 2, 1)
show_leaf_counts = False
ddata = augmented_dendrogram(linkage_matrix,
               color_threshold=1,
               p=6,
               truncate_mode='lastp',
               show_leaf_counts=show_leaf_counts,
               )
plt.title("show_leaf_counts = %s" % show_leaf_counts)

plt.subplot(1, 2, 2)
show_leaf_counts = True
ddata = augmented_dendrogram(linkage_matrix,
               color_threshold=1,
               p=6,
               truncate_mode='lastp',
               show_leaf_counts=show_leaf_counts,
               )
plt.title("show_leaf_counts = %s" % show_leaf_counts)

plt.show()
```
These are the points in the data set: With p=6 and trunc_mode="lastp", dendrogram only shows the "top" of the dendrogram. The following shows the effect of show_leaf_counts.


Question:
Calculate Matrix Rank using scipy I'd like to calculate the mathematical rank of a matrix using scipy. The most obvious function numpy.rank calculates the dimension of an array (ie. scalars have dimension 0, vectors 1, matrices 2, etc...). I am aware that the numpy.linalg.lstsq module has this capability, but I was wondering if such a fundamental operation is built into the matrix class somewhere. Here is an explicit example:
```
from numpy import matrix, rank
A = matrix([[1,3,7],[2,8,3],[7,8,1]])
print rank(A)
```
This gives 2 the dimension, where I'm looking for an answer of 3.

Response:
Numpy provides numpy.linalg.matrix_rank():
```

```python
import numpy
```

```python
numpy.__version__
#Output
#'1.5.1'
```

```python
A = numpy.matrix([[1,3,7],[2,8,3],[7,8,1]])
```

```python
numpy.linalg.matrix_rank(A)
#Output
#3
#```
```


Question: What is a "scalar" in NumPy? The documentation states the purpose of scalars, such as the fact that conventional Python numbers like float and integer are too primitive, and therefore more complex data types are necessary. It also states certain kinds of scalars (data type hierarchy); as well as a couple of attributes of scalar. But it never gives a concrete definition of exactly what a scalar is in the context of Python. I want to get to the heart of the issue on this. In the simplest terms possible, what is a Pythonic scalar?

Response:
A NumPy scalar is any object which is an instance of np.generic or whose type is in np.ScalarType:
```none

```python
np.ScalarType
#Output
#(int,
# float,
# complex,
# long,
# bool,
# str,
# unicode,
# buffer,
# numpy.int16,
# numpy.float16,
# numpy.int8,
# numpy.uint64,
# numpy.complex192,
# numpy.void,
# numpy.uint32,
# numpy.complex128,
# numpy.unicode_,
# numpy.uint32,
# numpy.complex64,
# numpy.string_,
# numpy.uint16,
# numpy.timedelta64,
# numpy.bool_,
# numpy.uint8,
# numpy.datetime64,
# numpy.object_,
# numpy.int64,
# numpy.float96,
# numpy.int32,
# numpy.float64,
# numpy.int32,
# numpy.float32)
#```
#This definition comes from looking at the source code for np.isscalar:
#```
#def isscalar(num):
#    if isinstance(num, generic):
#        return True
#    else:
#        return type(num) in ScalarType
#```
#Note that you can test if something is a scalar by using np.isscalar:
#```none
#>>> np.isscalar(3.1)
#True
#
#>>> np.isscalar([3.1])
#False
#
#>>> np.isscalar(False)
#True
#```
#How do we know what we know? I like learning how people know what they know—more than the answers themselves. So let me try to explain where the above answer comes from. Having the right tools can help you figure out things like this for yourself. I found this out by using IPython. Using its TAB-completion feature, typing
#```none
```

```python
import numpy as np
```

```python
np.[TAB]
```
causes IPython to display all variables in the np module namespace. A search for the string "scalar" will lead you to np.ScalarType and np.isscalar. Typing
```none
```

```python
np.isscalar?
```
(note the question mark at the end) prompts IPython to show you where np.isscalar is defined:
```none
File:  /data1/unutbu/.virtualenvs/dev/lib/python2.7/site-packages/numpy/core/numeric.py
```
which is how I got to the definition of isscalar. Alternatively, the NumPy documentation for isscalar has a link to the source code as well.
```


Question:
interpolate 3D volume with numpy and or scipy I am extremely frustrated because after several hours I can't seem to be able to do a seemingly easy 3D interpolation in python. In Matlab all I had to do was
```
Vi = interp3(x,y,z,V,xi,yi,zi)
```
What is the exact equivalent of this using scipy's ndimage.map_coordinate or other numpy methods? Thanks

Response:
In scipy 0.14 or later, there is a new function scipy.interpolate.RegularGridInterpolator which closely resembles interp3. The MATLAB command Vi = interp3(x,y,z,V,xi,yi,zi) would translate to something like:
```
from numpy import array
from scipy.interpolate import RegularGridInterpolator as rgi
my_interpolating_function = rgi((x,y,z), V)
Vi = my_interpolating_function(array([xi,yi,zi]).T)
```
Here is a full example demonstrating both; it will help you understand the exact differences... MATLAB CODE:
```
x = linspace(1,4,11);
y = linspace(4,7,22);
z = linspace(7,9,33);
V = zeros(22,11,33);
for i=1:11
    for j=1:22
        for k=1:33
            V(j,i,k) = 100*x(i) + 10*y(j) + z(k);
        end
    end
end
xq = [2,3];
yq = [6,5];
zq = [8,7];
Vi = interp3(x,y,z,V,xq,yq,zq);
```
The result is Vi=[268 357] which is indeed the value at those two points (2,6,8) and (3,5,7). SCIPY CODE:
```
from scipy.interpolate import RegularGridInterpolator
from numpy import linspace, zeros, array
x = linspace(1,4,11)
y = linspace(4,7,22)
z = linspace(7,9,33)
V = zeros((11,22,33))
for i in range(11):
    for j in range(22):
        for k in range(33):
            V[i,j,k] = 100*x[i] + 10*y[j] + z[k]
fn = RegularGridInterpolator((x,y,z), V)
pts = array([[2,6,8],[3,5,7]])
print(fn(pts))
```
Again it's [268,357]. So you see some slight differences: Scipy uses x,y,z index order while MATLAB uses y,x,z (strangely); In Scipy you define a function in a separate step and when you call it, the coordinates are grouped like (x1,y1,z1),(x2,y2,z2),... while matlab uses (x1,x2,...),(y1,y2,...),(z1,z2,...). Other than that, the two are similar and equally easy to use.


Question: scipy, lognormal distribution - parameters I want to fit lognormal distribution to my data, using python scipy.stats.lognormal.fit. According to the manual, fit returns shape, loc, scale parameters. But, lognormal distribution normally needs only two parameters: mean and standard deviation. How to interpret the results from scipy fit function? How to get mean and std.dev.?

Response:
The distributions in scipy are coded in a generic way wrt two parameter location and scale so that location is the parameter (loc) which shifts the distribution to the left or right, while scale is the parameter which compresses or stretches the distribution. For the two parameter lognormal distribution, the "mean" and "std dev" correspond to log(scale) and shape (you can let loc=0). The following illustrates how to fit a lognormal distribution to find the two parameters of interest:
```

```python
import numpy as np
```

```python
from scipy import stats
```

```python
logsample = stats.norm.rvs(loc=10, scale=3, size=1000) # logsample ~ N(mu=10, sigma=3)
```

```python
sample = np.exp(logsample) # sample ~ lognormal(10, 3)
```

```python
shape, loc, scale = stats.lognorm.fit(sample, floc=0) # hold location to 0 while fitting
```

```python
shape, loc, scale
#Output
#(2.9212650122639419, 0, 21318.029350592606)
```

```python
np.log(scale), shape  # mu, sigma
#Output
#(9.9673084420467362, 2.9212650122639419)
#```
```


Question:
Evaluate sympy expression from an array of values I'm experimenting with sympy and I've hit upon an issue I can't work out. Using scipy I can write an expression and evaluate it for an array of x values as follows:
```
import scipy
xvals = scipy.arange(-100,100,0.1)
f = lambda x: x**2
f(xvals)
```
Using sympy I can write the same expression as follows:
```
import sympy
x = sympy.symbols('x')
g = x**2
```
I can evaluate this expression for a single value by doing the following:
```
g.evalf(subs={x:10})
```
However I can't work out how to evaluate it for an array of x values, like I did with scipy. How would I do this?

Response:
First of all, at the moment SymPy does not guarantee support for numpy arrays which is what you want in this case. Check this bug report http://code.google.com/p/sympy/issues/detail?id=537 Second, If you want to evaluate something numerically for many values SymPy is not the best choice (it is a symbolic library after all). Use numpy and scipy. However, a valid reason to evaluate something numerically will be that deriving the expression to be evaluated was hard so you derive it in SymPy and then evaluate it in NumPy/SciPy/C/Fortran. To translate an expression to numpy just use
```
from sympy.utilities.lambdify import lambdify
func = lambdify(x, big_expression_containing_x,'numpy') # returns a numpy-ready function
numpy_array_of_results = func(numpy_array_of_arguments)
```
Check the docstring of lambdify for more details. Be aware that lambdify still has some issues and may need a rewrite. And just as a side note, if you want to evaluate the expressions really many times, you can use the codegen/autowrap module from sympy in order to create fortran or C code that is wrapped and callable from python. EDIT: An updates list of ways to do numerics in SymPy can be found on the wiki https://github.com/sympy/sympy/wiki/Philosophy-of-Numerics-and-Code-Generation-in-SymPy


Question:
pandas columns correlation with statistical significance What is the best way, given a pandas dataframe, df, to get the correlation between its columns df.1 and df.2? I do not want the output to count rows with NaN, which pandas built-in correlation does. But I also want it to output a pvalue or a standard error, which the built-in does not. SciPy seems to get caught up by the NaNs, though I believe it does report significance. Data example:
```
1           2
0    2          NaN
1    NaN         1
2    1           2
3    -4          3
4    1.3         1
5    NaN         NaN
```

Response:
To calculate all the p-values at once, you can use calculate_pvalues function (code below):
```
df = pd.DataFrame({'A':[1,2,3], 'B':[2,5,3], 'C':[5,2,1], 'D':['text',2,3] })
calculate_pvalues(df)
```
The output is similar to the corr() (but with p-values):
```
A       B       C
    A       0  0.7877  0.1789
    B  0.7877       0  0.6088
    C  0.1789  0.6088       0
```
Details: Column D is automatically ignored as it contains text. p-values are rounded to 4 decimals You can subset to indicate exact columns: calculate_pvalues(df[['A','B','C']] Following is the code of the function:
```
from scipy.stats import pearsonr
import pandas as pd

def calculate_pvalues(df):
    dfcols = pd.DataFrame(columns=df.columns)
    pvalues = dfcols.transpose().join(dfcols, how='outer')
    for r in df.columns:
        for c in df.columns:
            tmp = df[df[r].notnull() & df[c].notnull()]
            pvalues[r][c] = round(pearsonr(tmp[r], tmp[c])[1], 4)
    return pvalues
```


Question:
How to create a white image in Python? The following code creates a black picture:
```
from scipy.misc import imread,imsave
from numpy import zeros

img = zeros([100,100,3])
for y in range(len(img)):
    for x in range(len(img[0])):
        img[y,x] = [255,255,255]
imsave("Result.jpg",img)
```
I would have assumed it to be white. Edit in 2024: The code block is deprecated and won't run anymore.

Response:
Every color in an image is represented by one byte. So to create an image array, you should set it's dtype to uint8. And, you don't need for-loop to set every elements to 255, you can use fill() method or slice index:
```
import numpy as np
img = np.zeros([100,100,3],dtype=np.uint8)
img.fill(255) # or img[:] = 255
```


Question:
Tutorial for scipy.cluster.hierarchy [closed] Closed. This question is seeking recommendations for software libraries, tutorials, tools, books, or other off-site resources. It does not meet Stack Overflow guidelines. It is not currently accepting answers. We don’t allow questions seeking recommendations for software libraries, tutorials, tools, books, or other off-site resources. You can edit the question so it can be answered with facts and citations. Closed 8 years ago. Improve this question I'm trying to understand how to manipulate a hierarchy cluster but the documentation is too ... technical?... and I can't understand how it works. Is there any tutorial that can help me to start with, explaining step by step some simple tasks? Let's say I have the following data set:
```
a = np.array([[0,   0  ],
              [1,   0  ],
              [0,   1  ],
              [1,   1  ],
              [0.5, 0  ],
              [0,   0.5],
              [0.5, 0.5],
              [2,   2  ],
              [2,   3  ],
              [3,   2  ],
              [3,   3  ]])
```
I can easily do the hierarchy cluster and plot the dendrogram:
```
z = linkage(a)
d = dendrogram(z)
```
Now, how I can recover a specific cluster? Let's say the one with elements [0,1,2,4,5,6] in the dendrogram? How I can get back the values of that elements?

Response:
There are three steps in hierarchical agglomerative clustering (HAC): Quantify Data (metric argument) Cluster Data (method argument) Choose the number of clusters Doing
```
z = linkage(a)
```
will accomplish the first two steps. Since you did not specify any parameters it uses the standard values metric = 'euclidean' method = 'single' So z = linkage(a) will give you a single linked hierachical agglomerative clustering of a. This clustering is kind of a hierarchy of solutions. From this hierarchy you get some information about the structure of your data. What you might do now is: Check which metric is appropriate, e. g. cityblock or chebychev will quantify your data differently (cityblock, euclidean and chebychev correspond to L1, L2, and L_inf norm) Check the different properties / behaviours of the methdos (e. g. single, complete and average) Check how to determine the number of clusters, e. g. by reading the wiki about it Compute indices on the found solutions (clusterings) such as the silhouette coefficient (with this coefficient you get a feedback on the quality of how good a point/observation fits to the cluster it is assigned to by the clustering). Different indices use different criteria to qualify a clustering. Here is something to start with
```
import numpy as np
import scipy.cluster.hierarchy as hac
import matplotlib.pyplot as plt

a = np.array([[0.1,   2.5],
              [1.5,   .4 ],
              [0.3,   1  ],
              [1  ,   .8 ],
              [0.5,   0  ],
              [0  ,   0.5],
              [0.5,   0.5],
              [2.7,   2  ],
              [2.2,   3.1],
              [3  ,   2  ],
              [3.2,   1.3]])

fig, axes23 = plt.subplots(2, 3)

for method, axes in zip(['single', 'complete'], axes23):
    z = hac.linkage(a, method=method)

    # Plotting
    axes[0].plot(range(1, len(z)+1), z[::-1, 2])
    knee = np.diff(z[::-1, 2], 2)
    axes[0].plot(range(2, len(z)), knee)

    num_clust1 = knee.argmax() + 2
    knee[knee.argmax()] = 0
    num_clust2 = knee.argmax() + 2

    axes[0].text(num_clust1, z[::-1, 2][num_clust1-1], 'possible\n<- knee point')

    part1 = hac.fcluster(z, num_clust1, 'maxclust')
    part2 = hac.fcluster(z, num_clust2, 'maxclust')

    clr = ['#2200CC' ,'#D9007E' ,'#FF6600' ,'#FFCC00' ,'#ACE600' ,'#0099CC' ,
    '#8900CC' ,'#FF0000' ,'#FF9900' ,'#FFFF00' ,'#00CC01' ,'#0055CC']

    for part, ax in zip([part1, part2], axes[1:]):
        for cluster in set(part):
            ax.scatter(a[part == cluster, 0], a[part == cluster, 1],
                       color=clr[cluster])

    m = '\n(method: {})'.format(method)
    plt.setp(axes[0], title='Screeplot{}'.format(m), xlabel='partition',
             ylabel='{}\ncluster distance'.format(m))
    plt.setp(axes[1], title='{} Clusters'.format(num_clust1))
    plt.setp(axes[2], title='{} Clusters'.format(num_clust2))

plt.tight_layout()
plt.show()
```
Gives


Question: Highest Posterior Density Region and Central Credible Region Given a posterior p(Θ|D) over some parameters Θ, one can define the following: Highest Posterior Density Region: The Highest Posterior Density Region is the set of most probable values of Θ that, in total, constitute 100(1-α) % of the posterior mass. In other words, for a given α, we look for a p* that satisfies: and then obtain the Highest Posterior Density Region as the set: Central Credible Region: Using the same notation as above, a Credible Region (or interval) is defined as: Depending on the distribution, there could be many such intervals. The central credible interval is defined as a credible interval where there is (1-α)/2 mass on each tail. Computation: For general distributions, given samples from the distribution, are there any built-ins in to obtain the two quantities above in Python or PyMC? For common parametric distributions (e.g. Beta, Gaussian, etc.) are there any built-ins or libraries to compute this using SciPy or statsmodels?

Response:
From my understanding "central credible region" is not any different from how confidence intervals are calculated; all you need is the inverse of cdf function at alpha/2 and 1-alpha/2; in scipy this is called ppf ( percentage point function ); so as for Gaussian posterior distribution:
```

```python
from scipy.stats import norm
```

```python
alpha = .05
```

```python
l, u = norm.ppf(alpha / 2), norm.ppf(1 - alpha / 2)
#Output
#```
#to verify that [l, u] covers (1-alpha) of posterior density:
#```
```

```python
norm.cdf(u) - norm.cdf(l)
#Output
#0.94999999999999996
#```
#similarly for Beta posterior with say a=1 and b=3:
#```
```

```python
from scipy.stats import beta
```

```python
l, u = beta.ppf(alpha / 2, a=1, b=3), beta.ppf(1 - alpha / 2, a=1, b=3)
#Output
#```
#and again:
#```
```

```python
beta.cdf(u, a=1, b=3) - beta.cdf(l, a=1, b=3)
#Output
#0.94999999999999996
#```
#here you can see parametric distributions that are included in scipy; and I guess all of them have ppf function; As for highest posterior density region, it is more tricky, since pdf function is not necessarily invertible; and in general such a region may not even be connected; for example, in the case of Beta with a = b = .5 ( as can be seen here); But, in the case of Gaussian distribution, it is easy to see that "Highest Posterior Density Region" coincides with "Central Credible Region"; and I think that is is the case for all symmetric uni-modal distributions ( i.e. if pdf function is symmetric around the mode of distribution) A possible numerical approach for the general case would be binary search over the value of p* using numerical integration of pdf; utilizing the fact that the integral is a monotone function of p*; Here is an example for mixture Gaussian: [ 1 ] First thing you need is an analytical pdf function; for mixture Gaussian that is easy:
#```
#def mix_norm_pdf(x, loc, scale, weight):
#    from scipy.stats import norm
#    return np.dot(weight, norm.pdf(x, loc, scale))
#```
#so for example for location, scale and weight values as in
#```
#loc    = np.array([-1, 3])   # mean values
#scale  = np.array([.5, .8])  # standard deviations
#weight = np.array([.4, .6])  # mixture probabilities
#```
#you will get two nice Gaussian distributions holding hands: [ 2 ] now, you need an error function which given a test value for p* integrates pdf function above p* and returns squared error from the desired value 1 - alpha:
#```
#def errfn( p, alpha, *args):
#    from scipy import integrate
#    def fn( x ):
#        pdf = mix_norm_pdf(x, *args)
#        return pdf if pdf > p else 0
```

# ideally integration limits should not
    # be hard coded but inferred
    lb, ub = -3, 6
    prob = integrate.quad(fn, lb, ub)[0]
    return (prob + alpha - 1.0)**2
```
[ 3 ] now, for a given value of alpha we can minimize the error function to obtain p*:
```
alpha = .05

from scipy.optimize import fmin
p = fmin(errfn, x0=0, args=(alpha, loc, scale, weight))[0]
```
which results in p* = 0.0450, and HPD as below; the red area represents 1 - alpha of the distribution, and the horizontal dashed line is p*.