Question:
Is there an "enhanced" numpy/scipy dot method? Problem I would like to compute the following using numpy or scipy:
```
Y = A**T * Q * A
```
where A is a m x n matrix, A**T is the transpose of A and Q is an m x m diagonal matrix. Since Q is a diagonal matrix I store only its diagonal elements as a vector. Ways of solving for Y Currently I can think of two ways of how to calculate Y: Y = np.dot(np.dot(A.T, np.diag(Q)), A) and Y = np.dot(A.T * Q, A). Clearly option 2 is better than option 1 since no real matrix has to be created with diag(Q) (if this is what numpy really does...) However, both methods suffer from the defect of having to allocate more memory than there really is necessary since A.T * Q and np.dot(A.T, np.diag(Q)) have to be stored along with A in order to calculate Y. Question Is there a method in numpy/scipy that would eliminate the unnecessary allocation of extra memory where you would only pass two matrices A and B (in my case B is A.T) and a weighting vector Q along with it?

Response:
(w/r/t the last sentence of the OP: i am not aware of such a numpy/scipy method but w/r/t the Question in the OP Title (i.e., improving NumPy dot performance) what's below should be of some help. In other words, my answer is directed to improving performance of most of the steps comprising your function for Y). First, this should give you a noticeable boost over the vanilla NumPy dot method:
```

```python
from scipy.linalg import blas as FB
```

```python
vx = FB.dgemm(alpha=1., a=v1, b=v2, trans_b=True)
#Output
#```
#Note that the two arrays, v1, v2 are both in C_FORTRAN order You can access the byte order of a NumPy array through an array's flags attribute like so:
#```
```

```python
c = NP.ones((4, 3))
```

```python
c.flags
#Output
#      C_CONTIGUOUS : True          # refers to C-contiguous order
#      F_CONTIGUOUS : False         # fortran-contiguous
#      OWNDATA : True
#      MASKNA : False
#      OWNMASKNA : False
#      WRITEABLE : True
#      ALIGNED : True
#      UPDATEIFCOPY : False
#```
#to change the order of one of the arrays so both are aligned, just call the NumPy array constructor, pass in the array and set the appropriate order flag to True
#```
```

```python
c = NP.array(c, order="F")
```

```python
c.flags
#Output
#      C_CONTIGUOUS : False
#      F_CONTIGUOUS : True
#      OWNDATA : True
#      MASKNA : False
#      OWNMASKNA : False
#      WRITEABLE : True
#      ALIGNED : True
#      UPDATEIFCOPY : False
#```
#You can further optimize by exploiting array-order alignment to reduce excess memory consumption caused by copying the original arrays. But why are the arrays copied before being passed to dot? The dot product relies on BLAS operations. These operations require arrays stored in C-contiguous order--it's this constraint that causes the arrays to be copied. On the other hand, the transpose does not effect a copy, though unfortunately returns the result in Fortran order: Therefore, to remove the performance bottleneck, you need to eliminate the predicate array-copying step; to do that just requires passing both arrays to dot in C-contiguous order*. So to calculate dot(A.T., A) without making an extra copy:
#```
```

```python
import scipy.linalg.blas as FB
```

```python
vx = FB.dgemm(alpha=1.0, a=A.T, b=A.T, trans_b=True)
#Output
#```
#In sum, the expression just above (along with the predicate import statement) can substitute for dot, to supply the same functionality but better performance you can bind that expression to a function like so:
#```
```

```python
super_dot = lambda v, w: FB.dgemm(alpha=1., a=v.T, b=w.T, trans_b=True)
#Output
#```
```


Question:
Why is sin(180) not zero when using python and numpy? Does anyone know why the below doesn't equal 0?
```
import numpy as np
np.sin(np.radians(180))
```
or:
```
np.sin(np.pi)
```
When I enter it into python it gives me 1.22e-16.

Response:
The number π cannot be represented exactly as a floating-point number. So, np.radians(180) doesn't give you π, it gives you 3.1415926535897931. And sin(3.1415926535897931) is in fact something like 1.22e-16. So, how do you deal with this? You have to work out, or at least guess at, appropriate absolute and/or relative error bounds, and then instead of x == y, you write:
```
abs(y - x) < abs_bounds and abs(y-x) < rel_bounds * y
```
(This also means that you have to organize your computation so that the relative error is larger relative to y than to x. In your case, because y is the constant 0, that's trivial—just do it backward.) Numpy provides a function that does this for you across a whole array, allclose:
```
np.allclose(x, y, rel_bounds, abs_bounds)
```
(This actually checks abs(y - x) < abs_ bounds + rel_bounds * y), but that's almost always sufficient, and you can easily reorganize your code when it's not.) In your case:
```
np.allclose(0, np.sin(np.radians(180)), rel_bounds, abs_bounds)
```
So, how do you know what the right bounds are? There's no way to teach you enough error analysis in an SO answer. Propagation of uncertainty at Wikipedia gives a high-level overview. If you really have no clue, you can use the defaults, which are 1e-5 relative and 1e-8 absolute.


Question:
why is converting a long 2D list to numpy array so slow? I have a long list of xy coordinates, and would like to convert it into numpy array.
```

```python
import numpy as np
```

```python
xy = np.random.rand(1000000, 2).tolist()
#Output
#```
#The obvious way would be:
#```
```

```python
a = np.array(xy) # Very slow...
#Output
#```
#However, the above code is unreasonably slow. Interestingly, to transpose the long list first, convert it into numpy array, and then transpose back would be much faster (20x on my laptop).
#```
```

```python
def longlist2array(longlist):
    wide = [[row[c] for row in longlist] for c in range(len(longlist[0]))]
    return np.array(wide).T
```

```python
a = longlist2array(xy) # 20x faster!
#Output
#```
#Is this a bug of numpy? EDIT: This is a list of points (with xy coordinates) generated on-the-fly, so instead of preallocating an array and enlarging it when necessary, or maintaining two 1D lists for x and y, I think current representation is most natural. Why is looping through 2nd index faster than 1st index, given that we are iterating through a python list in both directions? EDIT 2: Based on @tiago's answer and this question, I found the following code twice as fast as my original version:
#```
```

```python
from itertools import chain
```

```python
def longlist2array(longlist):
    flat = np.fromiter(chain.from_iterable(longlist), np.array(longlist[0][0]).dtype, -1) # Without intermediate list:)
    return flat.reshape((len(longlist), -1))
#Output
#```
```

Response:
This is because the fastest-varying index of your list is the last one, so np.array() has to traverse the array many times because the first index is much larger. If your list was transposed, np.array() would be faster than your longlist2array:
```

```python
import numpy as np
```

```python
xy = np.random.rand(10000, 2).tolist()
```

```python
%timeit longlist2array(xy)
100 loops, best of 3: 3.38 ms per loop
```

```python
%timeit np.array(xy)
10 loops, best of 3: 55.8 ms per loop
```

```python
xy = np.random.rand(2, 10000).tolist()
```

```python
%timeit longlist2array(xy)
10 loops, best of 3: 59.8 ms per loop
```

```python
%timeit np.array(xy)
1000 loops, best of 3: 1.96 ms per loop
```
There is no magical solution for your problem. It's just how Python stores your list in memory. Do you really need to have a list with that shape? Can't you reverse it? (And do you really need a list, given that you're converting to numpy?) If you must convert a list, this function is about 10% faster than your longlist2array:
```
from itertools import chain

def convertlist(longlist)
    tmp = list(chain.from_iterable(longlist))
    return np.array(tmp).reshape((len(longlist), len(longlist[0])))
```
```


Question:
Plot line graph from histogram data in matplotlib I have a numpy array of ints representing time periods, which I'm currently plotting in a histogram to get a nice distribution graph, using the following code:
```
ax.hist(data,bins=100,range=(minimum,maximum),facecolor="r")
```
However I'm trying to modify this graph to represent the exact same data using a line instead of bars, so I can overlay more samples to the same plot and have them be clear (otherwise the bars overlap each other). What I've tried so far is to collate the data array into an array of tuples containing (time, count), and then plot it using
```
ax.plot(data[:,0],data[:,1],color="red",lw=2)
```
However that's not giving me anything close, as I can't accurately simulate the bins option of the histogram in my plot. Is there a better way to do this?

Response:
I am very late to the party - but maybe this will be useful to someone else. I think what you need to do is set the histtype parameter to 'step', i.e.
```
ax.hist(data,bins=100,range=(minimum,maximum),facecolor="r", histtype = 'step')
```
See also http://matplotlib.sourceforge.net/examples/pylab_examples/histogram_demo_extended.html


Question:
How to automatically annotate maximum value in pyplot I'm trying to figure out how I can automatically annotate the maximum value in a figure window. I know you can do this by manually entering in x,y coordinates to annotate whatever point you want using the .annotate() method, but I want the annotation to be automatic, or to find the maximum point by itself. Here's my code so far:
```
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
from pandas import Series, DataFrame

df = pd.read_csv('macrodata.csv') #Read csv file into dataframe
years = df['year'] #Get years column
infl = df['infl'] #Get inflation rate column

fig10 = plt.figure()
win = fig10.add_subplot(1,1,1)
fig10 = plt.plot(years, infl, lw = 2)

fig10 = plt.xlabel("Years")
fig10 = plt.ylabel("Inflation")
fig10 = plt.title("Inflation with Annotations")
```

Response:
If x and y are the arrays to plot, you get the coordinates of the maximum via
```
xmax = x[numpy.argmax(y)]
ymax = y.max()
```
This can be incorporated into a function that you may simply call with your data.
```
import matplotlib.pyplot as plt
import numpy as np

x = np.linspace(-2,8, num=301)
y = np.sinc((x-2.21)*3)

fig, ax = plt.subplots()
ax.plot(x,y)

def annot_max(x,y, ax=None):
    xmax = x[np.argmax(y)]
    ymax = y.max()
    text= "x={:.3f}, y={:.3f}".format(xmax, ymax)
    if not ax:
        ax=plt.gca()
    bbox_props = dict(boxstyle="square,pad=0.3", fc="w", ec="k", lw=0.72)
    arrowprops=dict(arrowstyle="->",connectionstyle="angle,angleA=0,angleB=60")
    kw = dict(xycoords='data',textcoords="axes fraction",
              arrowprops=arrowprops, bbox=bbox_props, ha="right", va="top")
    ax.annotate(text, xy=(xmax, ymax), xytext=(0.94,0.96), **kw)

annot_max(x,y)

ax.set_ylim(-0.3,1.5)
plt.show()
```


Question:
Numpy sort ndarray on multiple columns I get a ndarray reading it from a file, like this
```
my_data = np.genfromtxt(input_file, delimiter='\t', skip_header=0)
```
Example input (parsed)
```
[[   2.    1.    2.    0.]
 [   2.    2.  100.    0.]
 [   2.    3.  100.    0.]
 [   3.    1.    2.    0.]
 [   3.    2.    4.    0.]
 [   3.    3.    6.    0.]
 [   4.    1.    2.    0.]
 [   4.    2.    4.    0.]
 [   4.    3.    6.    0.]]
```
Longer example input (unparsed). The first 2 columns are supposed to be int, while the last 2 columns are supposed to be float, but that's what I get. Suggestions are welcome. The main problem is, I'm trying to sort it, using Numpy, so that rows get ordered giving precedence to the numbers on second column first, and on the first column next. Example of desired output
```
[[   2.    1.    2.    0.]
 [   3.    1.    2.    0.]
 [   4.    1.    2.    0.]
 [   2.    2.  100.    0.]
 [   3.    2.    4.    0.]
 [   4.    2.    4.    0.]
 [   2.    3.  100.    0.]
 [   3.    3.    6.    0.]
 [   4.    3.    6.    0.]]
```
I'm aware of this answer, it works for sorting rows on a single column. I tried sorting on the second column, since the first one is already sorted, but it's not enough. On occasion, the first column gets reordered too, badly.
```
new_data = my_data[my_data[:, 1].argsort()]
print(new_data)

#output
[[   2.    1.    2.    0.]
 [   4.    1.    2.    0.] #ouch
 [   3.    1.    2.    0.] #ouch
 [   2.    2.  100.    0.]
 [   3.    2.    4.    0.]
 [   4.    2.    4.    0.]
 [   2.    3.  100.    0.]
 [   3.    3.    6.    0.]
 [   4.    3.    6.    0.]]
```
I've also checked this question The answer mentions The problem here is that np.lexsort or np.sort do not work on arrays of dtype object. To get around that problem, you could sort the rows_list before creating order_list:
```
import operator
rows_list.sort(key=operator.itemgetter(0,1,2))
```
But I there is no key parameter in the sort function of type ndarray. And merging fields is not an alternative in my case. Also, I don't have a header, so, if I try to sort using the order parameter, I get an error.
```
ValueError: Cannot specify order when the array has no fields.
```
I'd rather sort in place or at least obtain a result of the same type ndarray. Then I want to save it to a file. How do I do this, without messing the datatypes?

Response:
numpy ndarray sort by the 1st, 2nd or 3rd column:
```

```python
a = np.array([[1,30,200], [2,20,300], [3,10,100]])
```

```python
a
#Output
#array([[  1,  30, 200],
#       [  2,  20, 300],
#       [  3,  10, 100]])
```

```python
a[a[:,2].argsort()]           #sort by the 3rd column ascending
#Output
#array([[  3,  10, 100],
#       [  1,  30, 200],
#       [  2,  20, 300]])
```

```python
a[a[:,2].argsort()][::-1]     #sort by the 3rd column descending
#Output
#array([[  2,  20, 300],
#       [  1,  30, 200],
#       [  3,  10, 100]])
```

```python
a[a[:,1].argsort()]        #sort by the 2nd column ascending
#Output
#array([[  3,  10, 100],
#       [  2,  20, 300],
#       [  1,  30, 200]])
#```
#To explain what is going on here: argsort() is passing back an array containing integer sequence of its parent: https://docs.scipy.org/doc/numpy/reference/generated/numpy.argsort.html
#```
```

```python
x = np.array([15, 30, 4, 80, 6])
```

```python
np.argsort(x)
#Output
#array([2, 4, 0, 1, 3])
#```
#Sort by column 1, then by column 2 then 3: according to the doc, The last column is the primary sort key.
#```
```

```python
a = np.array([[2,30,200], [1,30,200], [1,10,200]])
```

```python
a
#Output
#array([[  2,  30, 200],
#       [  1,  30, 200],
#       [  1,  10, 200]])
```

```python
a[np.lexsort((a[:,2], a[:,1],a[:,0]))]
#Output
#array([[  1,  10, 200],
#       [  1,  30, 200],
#       [  2,  30, 200]])
#```
#Same as above but reversed:
#```
```

```python
a[np.lexsort((a[:,2], a[:,1],a[:,0]))][::-1]
#Output
#array([[  2  30 200]
#       [  1  30 200]
#       [  1  10 200]])
#```
```


Question:
why numpy.ndarray is object is not callable in my simple for python loop I loaded a text file containing a two column matrix (e.g. below)
```
[ 1   3
  2   4
  3   5
  2   0]
```
My calculation is just to sum each row i.e. 1+3, 2+4, 3+5 and 2+0. I am using the below code:
```
data=np.loadtxt(fname="textfile.txt")## to load the above two column
xy= data
for XY in xy:
   i=0
   Z=XY(i,0)+XY(i,1)
   i=i+1
   print (Z)
```
But I received an error saying numpy.ndarray object is not callable. Why does this happen? How can I do this simple calculation? Thanks.

Response:
The error TypeError: 'numpy.ndarray' object is not callable means that you tried to call a numpy array as a function. Use
```
Z=XY[0]+XY[1]
```
Instead of
```
Z=XY(i,0)+XY(i,1)
```


Question:
Creating your own contour in opencv using python I have a set of boundary points of an object. I want to draw it using opencv as contour. I have no idea that how to convert my points to contour representation. To the same contour representation which is obtained by following call
```
contours,_ = cv2.findContours(image,cv2.RETR_TREE,cv2.CHAIN_APPROX_SIMPLE)
```
Any ideas? Thanks

Response:
By looking at the format of the contours I would think something like this should be sufficient:
```
contours = [numpy.array([[1,1],[10,50],[50,50]], dtype=numpy.int32) , numpy.array([[99,99],[99,60],[60,99]], dtype=numpy.int32)]
```
This small program gives an running example:
```
import numpy
import cv2

contours = [numpy.array([[1,1],[10,50],[50,50]], dtype=numpy.int32) , numpy.array([[99,99],[99,60],[60,99]], dtype=numpy.int32)]

drawing = numpy.zeros([100, 100],numpy.uint8)
for cnt in contours:
    cv2.drawContours(drawing,[cnt],0,(255,255,255),2)

cv2.imshow('output',drawing)
cv2.waitKey(0)
```


Question:
Convert a 2d matrix to a 3d one hot matrix numpy I have np matrix and I want to convert it to a 3d array with one hot encoding of the elements as third dimension. Is there a way to do with without looping over each row eg
```
a=[[1,3],
   [2,4]]
```
should be made into
```
b=[[1,0,0,0], [0,0,1,0],
   [0,1,0,0], [0,0,0,1]]
```

Response:
Approach #1 Here's a cheeky one-liner that abuses broadcasted comparison -
```
(np.arange(a.max()) == a[...,None]-1).astype(int)
```
Sample run -
```

```python
a
#Output
#array([[1, 7, 5, 3],
#       [2, 4, 1, 4]])
```

```python
(np.arange(a.max()) == a[...,None]-1).astype(int)
#Output
#array([[[1, 0, 0, 0, 0, 0, 0],
#        [0, 0, 0, 0, 0, 0, 1],
#        [0, 0, 0, 0, 1, 0, 0],
#        [0, 0, 1, 0, 0, 0, 0]],
#
#       [[0, 1, 0, 0, 0, 0, 0],
#        [0, 0, 0, 1, 0, 0, 0],
#        [1, 0, 0, 0, 0, 0, 0],
#        [0, 0, 0, 1, 0, 0, 0]]])
#```
#For 0-based indexing, it would be -
#```
```

```python
(np.arange(a.max()+1) == a[...,None]).astype(int)
#Output
#array([[[0, 1, 0, 0, 0, 0, 0, 0],
#        [0, 0, 0, 0, 0, 0, 0, 1],
#        [0, 0, 0, 0, 0, 1, 0, 0],
#        [0, 0, 0, 1, 0, 0, 0, 0]],
#
#       [[0, 0, 1, 0, 0, 0, 0, 0],
#        [0, 0, 0, 0, 1, 0, 0, 0],
#        [0, 1, 0, 0, 0, 0, 0, 0],
#        [0, 0, 0, 0, 1, 0, 0, 0]]])
#```
#If the one-hot enconding is to cover for the range of values ranging from the minimum to the maximum values, then offset by the minimum value and then feed it to the proposed method for 0-based indexing. This would be applicable for rest of the approaches discussed later on in this post as well. Here's a sample run on the same -
#```
```

```python
a
#Output
#array([[ 6, 12, 10,  8],
#       [ 7,  9,  6,  9]])
```

```python
a_off = a - a.min() # feed a_off to proposed approaches
```

```python
(np.arange(a_off.max()+1) == a_off[...,None]).astype(int)
#Output
#array([[[1, 0, 0, 0, 0, 0, 0],
#        [0, 0, 0, 0, 0, 0, 1],
#        [0, 0, 0, 0, 1, 0, 0],
#        [0, 0, 1, 0, 0, 0, 0]],
#
#       [[0, 1, 0, 0, 0, 0, 0],
#        [0, 0, 0, 1, 0, 0, 0],
#        [1, 0, 0, 0, 0, 0, 0],
#        [0, 0, 0, 1, 0, 0, 0]]])
#```
#If you are okay with a boolean array with True for 1's and False for 0's, you can skip the .astype(int) conversion. Approach #2 We can also initialize a zeros arrays and index into the output with advanced-indexing. Thus, for 0-based indexing, we would have -
#```
#def onehot_initialization(a):
#    ncols = a.max()+1
#    out = np.zeros(a.shape + (ncols,), dtype=int)
#    out[all_idx(a, axis=2)] = 1
#    return out
#```
#Helper func -
#```
## https://stackoverflow.com/a/46103129/ @Divakar
#def all_idx(idx, axis):
#    grid = np.ogrid[tuple(map(slice, idx.shape))]
#    grid.insert(axis, idx)
#    return tuple(grid)
#```
#This should be especially more performant when dealing with larger range of values. For 1-based indexing, simply feed in a-1 as the input. Approach #3 : Sparse matrix solution Now, if you are looking for sparse array as output and AFAIK since scipy's inbuilt sparse matrices support only 2D formats, you can get a sparse output that is a reshaped version of the output shown earlier with the first two axes merging and the third axis being kept intact. The implementation for 0-based indexing would look something like this -
#```
#from scipy.sparse import coo_matrix
#def onehot_sparse(a):
#    N = a.size
#    L = a.max()+1
#    data = np.ones(N,dtype=int)
#    return coo_matrix((data,(np.arange(N),a.ravel())), shape=(N,L))
#```
#Again, for 1-based indexing, simply feed in a-1 as the input. Sample run -
#```
```

```python
a
#Output
#array([[1, 7, 5, 3],
#       [2, 4, 1, 4]])
```

```python
onehot_sparse(a).toarray()
#Output
#array([[0, 1, 0, 0, 0, 0, 0, 0],
#       [0, 0, 0, 0, 0, 0, 0, 1],
#       [0, 0, 0, 0, 0, 1, 0, 0],
#       [0, 0, 0, 1, 0, 0, 0, 0],
#       [0, 0, 1, 0, 0, 0, 0, 0],
#       [0, 0, 0, 0, 1, 0, 0, 0],
#       [0, 1, 0, 0, 0, 0, 0, 0],
#       [0, 0, 0, 0, 1, 0, 0, 0]])
```

```python
onehot_sparse(a-1).toarray()
#Output
#array([[1, 0, 0, 0, 0, 0, 0],
#       [0, 0, 0, 0, 0, 0, 1],
#       [0, 0, 0, 0, 1, 0, 0],
#       [0, 0, 1, 0, 0, 0, 0],
#       [0, 1, 0, 0, 0, 0, 0],
#       [0, 0, 0, 1, 0, 0, 0],
#       [1, 0, 0, 0, 0, 0, 0],
#       [0, 0, 0, 1, 0, 0, 0]])
#```
#This would be much better than previous two approaches if you are okay with having sparse output. Runtime comparison for 0-based indexing Case #1 :
#```
```

```python
a = np.random.randint(0,100,(100,100))
```

```python
%timeit (np.arange(a.max()+1) == a[...,None]).astype(int)
1000 loops, best of 3: 1.51 ms per loop
```

```python
%timeit onehot_initialization(a)
1000 loops, best of 3: 478 µs per loop
```

```python
%timeit onehot_sparse(a)
10000 loops, best of 3: 87.5 µs per loop
```

```python
%timeit onehot_sparse(a).toarray()
1000 loops, best of 3: 530 µs per loop
```
Case #2 :
```
```

```python
a = np.random.randint(0,500,(100,100))
```

```python
%timeit (np.arange(a.max()+1) == a[...,None]).astype(int)
100 loops, best of 3: 8.51 ms per loop
```

```python
%timeit onehot_initialization(a)
100 loops, best of 3: 2.52 ms per loop
```

```python
%timeit onehot_sparse(a)
10000 loops, best of 3: 87.1 µs per loop
```

```python
%timeit onehot_sparse(a).toarray()
100 loops, best of 3: 2.67 ms per loop
```
Squeezing out best performance To squeeze out the best performance, we could modify approach #2 to use indexing on a 2D shaped output array and also use uint8 dtype for memory efficiency and that leading to much faster assignments, like so -
```
def onehot_initialization_v2(a):
    ncols = a.max()+1
    out = np.zeros( (a.size,ncols), dtype=np.uint8)
    out[np.arange(a.size),a.ravel()] = 1
    out.shape = a.shape + (ncols,)
    return out
```
Timings -
```
```

```python
a = np.random.randint(0,100,(100,100))
```

```python
%timeit onehot_initialization(a)
     ...: %timeit onehot_initialization_v2(a)
     ...:
1000 loops, best of 3: 474 µs per loop
10000 loops, best of 3: 128 µs per loop
```

```python
a = np.random.randint(0,500,(100,100))
```

```python
%timeit onehot_initialization(a)
     ...: %timeit onehot_initialization_v2(a)
     ...:
100 loops, best of 3: 2.38 ms per loop
1000 loops, best of 3: 213 µs per loop
```
```


Question: How to convert a pytorch tensor into a numpy array? How do I convert a torch tensor to numpy?

Response:
copied from pytorch doc:
```
a = torch.ones(5)
print(a)
```
tensor([1., 1., 1., 1., 1.])
```
b = a.numpy()
print(b)
```
[1. 1. 1. 1. 1.] Following from the below discussion with @John: In case the tensor is (or can be) on GPU, or in case it (or it can) require grad, one can use
```
t.detach().cpu().numpy()
```
I recommend to uglify your code only as much as required.


Question:
Check if each element in a numpy array is in another array This problem seems easy but I cannot quite get a nice-looking solution. I have two numpy arrays (A and B), and I want to get the indices of A where the elements of A are in B and also get the indices of A where the elements are not in B. So, if
```
A = np.array([1,2,3,4,5,6,7])
B = np.array([2,4,6])
```
Currently I am using
```
C = np.searchsorted(A,B)
```
which takes advantage of the fact that A is in order, and gives me [1, 3, 5], the indices of the elements that are in A. This is great, but how do I get D = [0,2,4,6], the indices of elements of A that are not in B?

Response:
searchsorted may give you wrong answer if not every element of B is in A. You can use numpy.in1d:
```
A = np.array([1,2,3,4,5,6,7])
B = np.array([2,4,6,8])
mask = np.in1d(A, B)
print np.where(mask)[0]
print np.where(~mask)[0]
```
output is:
```
[1 3 5]
[0 2 4 6]
```
However in1d() uses sort, which is slow for large datasets. You can use pandas if your dataset is large:
```
import pandas as pd
np.where(pd.Index(pd.unique(B)).get_indexer(A) >= 0)[0]
```
Here is the time comparison:
```
A = np.random.randint(0, 1000, 10000)
B = np.random.randint(0, 1000, 10000)

%timeit np.where(np.in1d(A, B))[0]
%timeit np.where(pd.Index(pd.unique(B)).get_indexer(A) >= 0)[0]
```
output:
```
100 loops, best of 3: 2.09 ms per loop
1000 loops, best of 3: 594 µs per loop
```


Question:
Concatenating column vectors using numpy arrays I'd like to concatenate 'column' vectors using numpy arrays but because numpy sees all arrays as row vectors by default, np.hstack and np.concatenate along any axis don't help (and neither did np.transpose as expected).
```
a = np.array((0, 1))
b = np.array((2, 1))
c = np.array((-1, -1))

np.hstack((a, b, c))
# array([ 0,  1,  2,  1, -1, -1])  ## Noooooo
np.reshape(np.hstack((a, b, c)), (2, 3))
# array([[ 0,  1,  2], [ 1, -1, -1]]) ## Reshaping won't help
```
One possibility (but too cumbersome) is
```
np.hstack((a[:, np.newaxis], b[:, np.newaxis], c[:, np.newaxis]))
# array([[ 0,  2, -1], [ 1,  1, -1]]) ##
```
Are there better ways?

Response:
I believe numpy.column_stack should do what you want. Example:
```

```python
a = np.array((0, 1))
```

```python
b = np.array((2, 1))
```

```python
c = np.array((-1, -1))
```

```python
numpy.column_stack((a,b,c))
#Output
#array([[ 0,  2, -1],
#       [ 1,  1, -1]])
#```
#It is essentially equal to
#```
```

```python
numpy.vstack((a,b,c)).T
#Output
#```
#though. As it says in the documentation.
```


Question:
How can I conditionally change the values in a numpy array taking into account nans? My array is a 2D matrix and it has numpy.nan values besides negative and positive values:
```

```python
array
#Output
#array([[        nan,         nan,         nan, ..., -0.04891211,
#                nan,         nan],
#       [        nan,         nan,         nan, ...,         nan,
#                nan,         nan],
#       [        nan,         nan,         nan, ...,         nan,
#                nan,         nan],
#       ...,
#       [-0.02510989, -0.02520096, -0.02669156, ...,         nan,
#                nan,         nan],
#       [-0.02725595, -0.02715945, -0.0286231 , ...,         nan,
#                nan,         nan],
#       [        nan,         nan,         nan, ...,         nan,
#                nan,         nan]], dtype=float32)
#```
#(There are positive numbers in the array, they just don't show in the preview.) And I want to replace all the positive numbers with a number and all the negative numbers with another number. How can I perform that using python/numpy? (For the record, the matrix is a result of geoimage, which I want to perform a classification)
```

Response:
The fact that you have np.nan in your array should not matter. Just use fancy indexing:
```
x[x>0] = new_value_for_pos
x[x<0] = new_value_for_neg
```
If you want to replace your np.nans:
```
x[np.isnan(x)] = something_not_nan
```
More info on fancy indexing a tutorial and the NumPy documentation.


Question: Integrating a multidimensional integral in scipy Motivation: I have a multidimensional integral, which for completeness I have reproduced below. It comes from the computation of the second virial coefficient when there is significant anisotropy: Here W is a function of all the variables. It is a known function, one which I can define a python function for. Programming Question: How do I get scipy to integrate this expression? I was thinking of chaining two triple quads (scipy.integrate.tplquad) together, but I'm worried about performance and accuracy. Is there a higher dimensional integrator in scipy, one that can handle an arbitrary number of nested integrals? If not, what is the best way to do this?

Response:
With a higher-dimensional integral like this, monte carlo methods are often a useful technique - they converge on the answer as the inverse square root of the number of function evaluations, which is better for higher dimension then you'll generally get out of even fairly sophisticated adaptive methods (unless you know something very specific about your integrand - symmetries that can be exploited, etc.) The mcint package performs a monte carlo integration: running with a non-trivial W that is nonetheless integrable so we know the answer we get (note that I've truncated r to be from [0,1); you'll have to do some sort of log transform or something to get that semi-unbounded domain into something tractable for most numerical integrators):
```
import mcint
import random
import math

def w(r, theta, phi, alpha, beta, gamma):
    return(-math.log(theta * beta))

def integrand(x):
    r     = x[0]
    theta = x[1]
    alpha = x[2]
    beta  = x[3]
    gamma = x[4]
    phi   = x[5]

    k = 1.
    T = 1.
    ww = w(r, theta, phi, alpha, beta, gamma)
    return (math.exp(-ww/(k*T)) - 1.)*r*r*math.sin(beta)*math.sin(theta)

def sampler():
    while True:
        r     = random.uniform(0.,1.)
        theta = random.uniform(0.,2.*math.pi)
        alpha = random.uniform(0.,2.*math.pi)
        beta  = random.uniform(0.,2.*math.pi)
        gamma = random.uniform(0.,2.*math.pi)
        phi   = random.uniform(0.,math.pi)
        yield (r, theta, alpha, beta, gamma, phi)

domainsize = math.pow(2*math.pi,4)*math.pi*1
expected = 16*math.pow(math.pi,5)/3.

for nmc in [1000, 10000, 100000, 1000000, 10000000, 100000000]:
    random.seed(1)
    result, error = mcint.integrate(integrand, sampler(), measure=domainsize, n=nmc)
    diff = abs(result - expected)

    print "Using n = ", nmc
    print "Result = ", result, "estimated error = ", error
    print "Known result = ", expected, " error = ", diff, " = ", 100.*diff/expected, "%"
    print " "
```
Running gives
```
Using n =  1000
Result =  1654.19633236 estimated error =  399.360391622
Known result =  1632.10498552  error =  22.0913468345  =  1.35354937522 %

Using n =  10000
Result =  1634.88583778 estimated error =  128.824988953
Known result =  1632.10498552  error =  2.78085225405  =  0.170384397984 %

Using n =  100000
Result =  1646.72936 estimated error =  41.3384733174
Known result =  1632.10498552  error =  14.6243744747  =  0.8960437352 %

Using n =  1000000
Result =  1640.67189792 estimated error =  13.0282663003
Known result =  1632.10498552  error =  8.56691239895  =  0.524899591322 %

Using n =  10000000
Result =  1635.52135088 estimated error =  4.12131562436
Known result =  1632.10498552  error =  3.41636536248  =  0.209322647304 %

Using n =  100000000
Result =  1631.5982799 estimated error =  1.30214644297
Known result =  1632.10498552  error =  0.506705620147  =  0.0310461413109 %
```
You could greatly speed this up by vectorizing the random number generation, etc. Of course, you can chain the triple integrals as you suggest:
```
import numpy
import scipy.integrate
import math

def w(r, theta, phi, alpha, beta, gamma):
    return(-math.log(theta * beta))

def integrand(phi, alpha, gamma, r, theta, beta):
    ww = w(r, theta, phi, alpha, beta, gamma)
    k = 1.
    T = 1.
    return (math.exp(-ww/(k*T)) - 1.)*r*r*math.sin(beta)*math.sin(theta)

# limits of integration

def zero(x, y=0):
    return 0.

def one(x, y=0):
    return 1.

def pi(x, y=0):
    return math.pi

def twopi(x, y=0):
    return 2.*math.pi

# integrate over phi [0, Pi), alpha [0, 2 Pi), gamma [0, 2 Pi)
def secondIntegrals(r, theta, beta):
    res, err = scipy.integrate.tplquad(integrand, 0., 2.*math.pi, zero, twopi, zero, pi, args=(r, theta, beta))
    return res

# integrate over r [0, 1), beta [0, 2 Pi), theta [0, 2 Pi)
def integral():
    return scipy.integrate.tplquad(secondIntegrals, 0., 2.*math.pi, zero, twopi, zero, one)

expected = 16*math.pow(math.pi,5)/3.
result, err = integral()
diff = abs(result - expected)

print "Result = ", result, " estimated error = ", err
print "Known result = ", expected, " error = ", diff, " = ", 100.*diff/expected, "%"
```
which is slow but gives very good results for this simple case. Which is better is going to come down to how complicated your W is and what your accuracy requirements are. Simple (fast to evaluate) W with high accuracy will push you to this sort of method; complicated (slow to evaluate) W with moderate accuracy requirements will push you towards MC techniques.
```
Result =  1632.10498552  estimated error =  3.59054059995e-11
Known result =  1632.10498552  error =  4.54747350886e-13  =  2.7862628625e-14 %
```


Question:
Is there a better way of making numpy.argmin() ignore NaN values I want to get the index of the min value of a numpy array that contains NaNs and I want them ignored
```

```python
a = array([ nan,   2.5,   3.,  nan,   4.,   5.])
```

```python
a
#Output
#array([ NaN,  2.5,  3. ,  NaN,  4. ,  5. ])
#```
#if I run argmin, it returns the index of the first NaN
#```
```

```python
a.argmin()
#Output
#0
#```
#I substitute NaNs with Infs and then run argmin
#```
```

```python
a[isnan(a)] = Inf
```

```python
a
#Output
#array([ Inf,  2.5,  3. ,  Inf,  4. ,  5. ])
```

```python
a.argmin()
#Output
#1
#```
#My dilemma is the following: I'd rather not change NaNs to Infs and then back after I'm done with argmin (since NaNs have a meaning later on in the code). Is there a better way to do this? There is also a question of what should the result be if all of the original values of a are NaN? In my implementation the answer is 0
```

Response:
Sure! Use nanargmin:
```
import numpy as np
a = np.array([ np.nan,   2.5,   3.,  np.nan,   4.,   5.])
print(np.nanargmin(a))
# 1
```
There is also nansum, nanmax, nanargmax, and nanmin, In scipy.stats, there is nanmean and nanmedian. For more ways to ignore nans, check out masked arrays.


Question:
Merge two numpy arrays I am trying to merge two arrays with the same number of arguments. Input:
```
first = [[650001.88, 300442.2,   18.73,  0.575,  650002.094, 300441.668, 18.775],
         [650001.96, 300443.4,   18.7,   0.65,   650002.571, 300443.182, 18.745],
         [650002.95, 300442.54,  18.82,  0.473,  650003.056, 300442.085, 18.745]]

second = [[1],
          [2],
          [3]]
```
My expected output:
```
final = [[650001.88, 300442.2,   18.73,  0.575,  650002.094, 300441.668, 18.775, 1],
         [650001.96, 300443.4,   18.7,   0.65,   650002.571, 300443.182, 18.745, 2],
         [650002.95, 300442.54,  18.82,  0.473,  650003.056, 300442.085, 18.745, 3]]
```
To do that I create simple loop:
```
for i in first:
        for j in second:
            final += np.append(j, i)
```
I got i filling that i missing something. First of all my loop i extremely slow. Secondly my data is quite have i got more than 2 mlns rows to loop. So I tried to find faster way for example with this code:
```
final = [np.append(i, second[0]) for i in first]
```
It working far more faster than previous loop but its appending only first value of second array. Can you help me?

Response:
Use np.array and then np.concatenate,
```
import numpy as np

first = np.array([[650001.88, 300442.2,   18.73,  0.575,
                   650002.094, 300441.668, 18.775],
                  [650001.96, 300443.4,   18.7,   0.65,
                   650002.571, 300443.182, 18.745],
                  [650002.95, 300442.54,  18.82,  0.473,
                   650003.056, 300442.085, 18.745]])

second = np.array([[1],
                   [2],
                   [3]])

np.concatenate((first, second), axis=1)
```
Where axis=1 means that we want to concatenate horizontally. That works for me


Question:
Create block diagonal numpy array from a given numpy array I have a 2-dimensional numpy array with an equal number of columns and rows. I would like to arrange them into a bigger array having the smaller ones on the diagonal. It should be possible to specify how often the starting matrix should be on the diagonal. For example:
```
a = numpy.array([[5, 7],
                 [6, 3]])
```
So if I wanted this array 2 times on the diagonal the desired output would be:
```
array([[5, 7, 0, 0],
       [6, 3, 0, 0],
       [0, 0, 5, 7],
       [0, 0, 6, 3]])
```
For 3 times:
```
array([[5, 7, 0, 0, 0, 0],
       [6, 3, 0, 0, 0, 0],
       [0, 0, 5, 7, 0, 0],
       [0, 0, 6, 3, 0, 0],
       [0, 0, 0, 0, 5, 7],
       [0, 0, 0, 0, 6, 3]])
```
Is there a fast way to implement this with numpy methods and for arbitrary sizes of the starting array (still considering the starting array to have the same number of rows and columns)?

Response:
Approach #1 Classic case of numpy.kron -
```
np.kron(np.eye(r,dtype=int),a) # r is number of repeats
```
Sample run -
```

```python
a
#Output
#array([[1, 2, 3],
#       [3, 4, 5]])
```

```python
r = 3 # number of repeats
```

```python
np.kron(np.eye(r,dtype=int),a)
#Output
#array([[1, 2, 3, 0, 0, 0, 0, 0, 0],
#       [3, 4, 5, 0, 0, 0, 0, 0, 0],
#       [0, 0, 0, 1, 2, 3, 0, 0, 0],
#       [0, 0, 0, 3, 4, 5, 0, 0, 0],
#       [0, 0, 0, 0, 0, 0, 1, 2, 3],
#       [0, 0, 0, 0, 0, 0, 3, 4, 5]])
#```
#Approach #2 Another efficient one with diagonal-viewed-array-assignment -
#```
#def repeat_along_diag(a, r):
#    m,n = a.shape
#    out = np.zeros((r,m,r,n), dtype=a.dtype)
#    diag = np.einsum('ijik->ijk',out)
#    diag[:] = a
#    return out.reshape(-1,n*r)
#```
#Sample run -
#```
```

```python
repeat_along_diag(a,3)
#Output
#array([[1, 2, 3, 0, 0, 0, 0, 0, 0],
#       [3, 4, 5, 0, 0, 0, 0, 0, 0],
#       [0, 0, 0, 1, 2, 3, 0, 0, 0],
#       [0, 0, 0, 3, 4, 5, 0, 0, 0],
#       [0, 0, 0, 0, 0, 0, 1, 2, 3],
#       [0, 0, 0, 0, 0, 0, 3, 4, 5]])
#```
```


Question: Techniques for working with large Numpy arrays? [duplicate] This question already has answers here: Very large matrices using Python and NumPy (10 answers) Closed 5 years ago. There are times when you have to perform many intermediate operations on one, or more, large Numpy arrays. This can quickly result in MemoryErrors. In my research so far, I have found that Pickling (Pickle, CPickle, Pytables etc.) and gc.collect() are ways to mitigate this. I was wondering if there are any other techniques experienced programmers use when dealing with large quantities of data (other than removing redundancies in your strategy/code, of course). Also, if there's one thing I'm sure of is that nothing is free. With some of these techniques, what are the trade-offs (i.e., speed, robustness, etc.)?

Response: I feel your pain... You sometimes end up storing several times the size of your array in values you will later discard. When processing one item in your array at a time, this is irrelevant, but can kill you when vectorizing. I'll use an example from work for illustration purposes. I recently coded the algorithm described here using numpy. It is a color map algorithm, which takes an RGB image, and converts it into a CMYK image. The process, which is repeated for every pixel, is as follows: Use the most significant 4 bits of every RGB value, as indices into a three-dimensional look up table. This determines the CMYK values for the 8 vertices of a cube within the LUT. Use the least significant 4 bits of every RGB value to interpolate within that cube, based on the vertex values from the previous step. The most efficient way of doing this requires computing 16 arrays of uint8s the size of the image being processed. For a 24bit RGB image that is equivalent to needing storage of x6 times that of the image to process it. A couple of things you can do to handle this: 1. Divide and conquer Maybe you cannot process a 1,000x1,000 array in a single pass. But if you can do it with a python for loop iterating over 10 arrays of 100x1,000, it is still going to beat by a very far margin a python iterator over 1,000,000 items! It´s going to be slower, yes, but not as much. 2. Cache expensive computations This relates directly to my interpolation example above, and is harder to come across, although worth keeping an eye open for it. Because I am interpolating on a three-dimensional cube with 4 bits in each dimension, there are only 16x16x16 possible outcomes, which can be stored in 16 arrays of 16x16x16 bytes. So I can precompute them and store them using 64KB of memory, and look-up the values one by one for the whole image, rather than redoing the same operations for every pixel at huge memory cost. This already pays-off for images as small as 64x64 pixels, and basically allows processing images with x6 times the amount of pixels without having to subdivide the array. 3. Use your dtypes wisely If your intermediate values can fit in a single uint8, don't use an array of int32s! This can turn into a nightmare of mysterious errors due to silent overflows, but if you are careful, it can provide a big saving of resources.


Question:
Select certain rows (condition met), but only some columns in Python/Numpy I have an numpy array with 4 columns and want to select columns 1, 3 and 4, where the value of the second column meets a certain condition (i.e. a fixed value). I tried to first select only the rows, but with all 4 columns via:
```
I = A[A[:,1] == i]
```
which works. Then I further tried (similarly to matlab which I know very well):
```
I = A[A[:,1] == i, [0,2,3]]
```
which doesn't work. How to do it? EXAMPLE DATA:
```

```python
A = np.array([[1,2,3,4],[6,1,3,4],[3,2,5,6]])
#Output
# >>> print A
# [[1 2 3 4]
#  [6 1 3 4]
#  [3 2 5 6]]
# >>> i = 2
```

# I want to get the columns 1, 3 and 4
 # for every row which has the value i in the second column.
 # In this case, this would be row 1 and 3 with columns 1, 3 and 4:
 [[1 3 4]
  [3 5 6]]
```
I am now currently using this:
```
I = A[A[:,1] == i]
I = I[:, [0,2,3]]
```
But I thought that there had to be a nicer way of doing it... (I am used to MATLAB)

Response:
```

```python
a = np.array([[1,2,3,4],[5,6,7,8],[9,10,11,12]])
```

```python
a
#Output
#array([[ 1,  2,  3,  4],
#       [ 5,  6,  7,  8],
#       [ 9, 10, 11, 12]])
```

```python
a[a[:,0] > 3] # select rows where first column is greater than 3
#Output
#array([[ 5,  6,  7,  8],
#       [ 9, 10, 11, 12]])
```

```python
a[a[:,0] > 3][:,np.array([True, True, False, True])] # select columns
#Output
#array([[ 5,  6,  8],
#       [ 9, 10, 12]])
```

# fancier equivalent of the previous

```python
a[np.ix_(a[:,0] > 3, np.array([True, True, False, True]))]
#Output
#array([[ 5,  6,  8],
#       [ 9, 10, 12]])
#```
#For an explanation of the obscure np.ix_(), see https://stackoverflow.com/a/13599843/4323 Finally, we can simplify by giving the list of column numbers instead of the tedious boolean mask:
#```
```

```python
a[np.ix_(a[:,0] > 3, (0,1,3))]
#Output
#array([[ 5,  6,  8],
#       [ 9, 10, 12]])
#```
```


Question: What is the difference between numpy.fft.fft and numpy.fft.rfft? The documentation says that np.fft.fft does this: Compute the one-dimensional discrete Fourier Transform. and np.fft.rfft does this: Compute the one-dimensional discrete Fourier Transform for real input. I also see that for my data (audio data, real valued), np.fft.fft returns a 2 dimensional array of shape (number_of_frames, fft_length) containing complex numbers. For np.fft.rfft returns a 2 dimensional array of shape (number_of_frames, ((fft_length/2) + 1)) containing complex numbers. I am led to believe that this only contains nonredundant FFT bins. Can someone explain in more depth the difference between the commands and why the shape of the returned array is different. Thank you.

Response:
the reason is explained in the docs: When the DFT is computed for purely real input, the output is Hermitian-symmetric, i.e. the negative frequency terms are just the complex conjugates of the corresponding positive-frequency terms, and the negative-frequency terms are therefore redundant. This function does not compute the negative frequency terms, and the length of the transformed axis of the output is therefore n//2 + 1. As a consequence, the algorithm is optimized and rfft is twice as fast. Furthermore, the spectrum is easier to plot :
```

```python
s=abs(sin(arange(0,2**13,3)))
```

```python
sp=rfft(s)
```

```python
plot(abs(sp))
```
```


Question:
Cheapest way to get a numpy array into C-contiguous order? The following produces a C-contiguous numpy array:
```
import numpy

a = numpy.ones((1024,1024,5))
```
Now if I slice it, the result may not longer be the same. For example:
```
bn = a[:, :, n]
```
with n from 0 to 4. My problem is that I need bn to be C-contiguous, and I need to do this for many instances of a. I just need each bn once, and want to avoid doing
```
bn  = bn.copy(order='C')
```
I also don't want to rewrite my code such that
```
a = numpy.ones((5,1024,1024))
```
Is there a faster, cheaper way to get bn than doing the copy? Background: I want to hash each slice of every a, using
```
import hashlib

hashlib.sha1(a[:, :, n]).hexdigest()
```
Unfortunately, this will throw a ValueError, complaining about the order. So if there is another fast way to get the hash I want, I'd also use it.

Response:
This is a standard operation when interfacing numpy with C. Have a look at numpy.ascontiguousarray x=numpy.ascontiguousarray(x) is the proper way of dealing with it. Use numpy.asfortranarray if you need fortran order. As mentioned the function will copy if necessary. So there is no way around it. You can try rollaxis before your operation, such that the short axis is the first axis. This gives you a view on the array
```

```python
A=np.random.rand(1024,1024,5)
```

```python
B=np.rollaxis(A,2)
```

```python
B.shape
#Output
#(5, 1024, 1024)
```

```python
B.flags
#Output
#C_CONTIGUOUS : False
#  F_CONTIGUOUS : False
#  OWNDATA : False
#  WRITEABLE : True
#  ALIGNED : True
#  UPDATEIFCOPY : False
```

```python
A.flags
#Output
#C_CONTIGUOUS : True
#  F_CONTIGUOUS : False
#  OWNDATA : True
#  WRITEABLE : True
#  ALIGNED : True
#  UPDATEIFCOPY : False
#```
#So rollaxis does not solve this either.
```


Question:
How to test if all rows are equal in a numpy In numpy, is there a nice idiomatic way of testing if all rows are equal in a 2d array? I can do something like
```
np.all([np.array_equal(M[0], M[i]) for i in xrange(1,len(M))])
```
This seems to mix python lists with numpy arrays which is ugly and presumably also slow. Is there a nicer/neater way?

Response:
One way is to check that every row of the array arr is equal to its first row arr[0]:
```
(arr == arr[0]).all()
```
Using equality == is fine for integer values, but if arr contains floating point values you could use np.isclose instead to check for equality within a given tolerance:
```
np.isclose(a, a[0]).all()
```
If your array contains NaN and you want to avoid the tricky NaN != NaN issue, you could combine this approach with np.isnan:
```
(np.isclose(a, a[0]) | np.isnan(a)).all()
```


Question: How do I read numpy source? I built it myself on Python 3.3, but I can't for the life of me find the class definition of numpy.array(). I've looked all through the code and even found the core C files, but where is the dang array class?? Can anyone tell me what directory to look in, or how to find out from the python shell?

Response: np.array is not a class itself, it's just a convenience function to create an np.ndarray. ndarray is aliased to multiarray, which is implemented in C code (an extension module in .so/.pyd file, compiled code). You can start looking at the ndarray interfaces here in numeric.py. Most of the meat of the implementation is in C code, here in multiarray. array() is implemented in _core/src/multiarray/methods.c in array_getarray().


Question: Downsampling a 2d numpy array in python I'm self learning python and have found a problem which requires down sampling a feature vector. I need some help understanding how down-sampling a array. in the array each row represents an image by being number from 0 to 255. I was wonder how you apply down-sampling to the array? I don't want to scikit-learn because I want to understand how to apply down-sampling. If you could explain down-sampling too that would be amazing thanks. the feature vector is 400x250

Response:
If with downsampling you mean something like this, you can simply slice the array. For a 1D example:
```
import numpy as np
a = np.arange(1,11,1)
print(a)
print(a[::3])
```
The last line is equivalent to:
```
print(a[0:a.size:3])
```
with the slicing notation as start:stop:step Result: [ 1 2 3 4 5 6 7 8 9 10] [ 1 4 7 10] For a 2D array the idea is the same:
```
b = np.arange(0,100)
c = b.reshape([10,10])
print(c[::3,::3])
```
This gives you, in both dimensions, every third item from the original array. Or, if you only want to down sample a single dimension:
```
d = np.zeros((400,250))
print(d.shape)
e = d[::10,:]
print(e.shape)
```
(400, 250) (40, 250) The are lots of other examples in the Numpy manual


Question:
Check for None in pandas dataframe I would like to find where None is found in the dataframe.
```
pd.DataFrame([None,np.nan]).isnull()
OUT:
      0
0  True
1  True
```
isnull() finds both numpy Nan and None values. I only want the None values and not numpy Nan. Is there an easier way to do that without looping through the dataframe? Edit: After reading the comments, I realized that in my dataframe in my work also include strings, so the None were not coerced to numpy Nan. So the answer given by Pisdom works.

Response:
If you want to get True/False for each line, you can use the following code. Here is an example as a result for the following DataFrame:
```py
df = pd.DataFrame([[None, 3], ["", np.nan]])

df
#      0      1
#0  None    3.0
#1          NaN
```
How to check None Available: .isnull()
```py

```python
df[0].isnull()
#Output
#0     True
#1    False
#Name: 0, dtype: bool
#```
#Available: .apply == or is None
#```py
```

```python
df[0].apply(lambda x: x == None)
#Output
#0     True
#1    False
#Name: 0, dtype: bool
```

```python
df[0].apply(lambda x: x is None)
#Output
#0     True
#1    False
#Name: 0, dtype: bool
#```
#Available: .values == None
#```py
```

```python
df[0].values == None
#Output
#array([ True, False])
#```
#Unavailable: is or ==
#```py
```

```python
df[0] is None
#Output
#False
```

```python
df[0] == None
#Output
#0    False
#1    False
#Name: 0, dtype: bool
#```
#Unavailable: .values is None
#```py
```

```python
df[0].values is None
#Output
#False
#```
#How to check np.nan Available: .isnull()
#```py
```

```python
df[1].isnull()
#Output
#0    False
#1     True
#Name: 1, dtype: bool
#```
#Available: np.isnan
#```py
```

```python
np.isnan(df[1])
#Output
#0    False
#1     True
#Name: 1, dtype: bool
```

```python
np.isnan(df[1].values)
#Output
#array([False,  True])
```

```python
df[1].apply(lambda x: np.isnan(x))
#Output
#0    False
#1     True
#Name: 1, dtype: bool
#```
#Unavailable: is or == np.nan
#```py
```

```python
df[1] is np.nan
#Output
#False
```

```python
df[1] == np.nan
#Output
#0    False
#1    False
#Name: 1, dtype: bool
```

```python
df[1].values is np.nan
#Output
#False
```

```python
df[1].values == np.nan
#Output
#array([False, False])
```

```python
df[1].apply(lambda x: x is np.nan)
#Output
#0    False
#1    False
#Name: 1, dtype: bool
```

```python
df[1].apply(lambda x: x == np.nan)
#Output
#0    False
#1    False
#Name: 1, dtype: bool
#```
```


Question: Numpy mean AND variance from single function? Using Numpy/Python, is it possible to return the mean AND variance from a single function call? I know that I can do them separately, but the mean is required to calculate the sample standard deviation. So if I use separate functions to get the mean and variance I am adding unnecesary overhead. I have tried looking at the numpy docs here (http://docs.scipy.org/doc/numpy/reference/routines.statistics.html), but with no success.

Response:
You can't pass a known mean to np.std or np.var, you'll have to wait for the new standard library statistics module, but in the meantime you can save a little time by using the formula:
```

```python
a = np.random.rand(1000)
```

```python
%%timeit
   .....: a.mean()
   .....: a.var()
   .....:
10000 loops, best of 3: 80.6 µs per loop
```

```python
%%timeit
   .....: m = a.mean()
   .....: np.mean((a-m)**2)
   .....:
10000 loops, best of 3: 60.9 µs per loop
```

```python
m = a.mean()
```

```python
a.var()
#Output
#0.078365856465916137
```

```python
np.mean((a-m)**2)
#Output
#0.078365856465916137
#```
#If you really are trying to speed things up, try np.dot to do the squaring and summing (since that's what a dot-product is):
#```
```

```python
np.dot(a-m,a-m)/a.size
#Output
#0.078365856465916137
```

```python
%%timeit
   .....: m = a.mean()
   .....: c = a-m
   .....: np.dot(c,c)/a.size
   .....:
10000 loops, best of 3: 38.2 µs per loop
```
```


Question: How to plot complex numbers (Argand Diagram) using matplotlib I'd like to create an Argand Diagram from a set of complex numbers using matplotlib. Are there any pre-built functions to help me do this? Can anyone recommend an approach? Image by LeonardoG, CC-SA-3.0

Response:
I'm not sure exactly what you're after here...you have a set of complex numbers, and want to map them to the plane by using their real part as the x coordinate and the imaginary part as y? If so you can get the real part of any python imaginary number with number.real and the imaginary part with number.imag. If you're using numpy, it also provides a set of helper functions numpy.real and numpy.imag etc. which work on numpy arrays. So for instance if you had an array of complex numbers stored something like this:
```

```python
a = n.arange(5) + 1j*n.arange(6,11)
```

```python
a
#Output
#array([ 0. +6.j,  1. +7.j,  2. +8.j,  3. +9.j,  4.+10.j])
#```
#...you can just do
#```
```

```python
fig,ax = subplots()
```

```python
ax.scatter(a.real,a.imag)
```
This plots dots on an argand diagram for each point. edit: For the plotting part, you must of course have imported matplotlib.pyplot via from matplotlib.pyplot import * or (as I did) use the ipython shell in pylab mode.
```


Question:
Indexing one array by another in numpy Suppose I have a matrix A with some arbitrary values:
```
array([[ 2, 4, 5, 3],
       [ 1, 6, 8, 9],
       [ 8, 7, 0, 2]])
```
And a matrix B which contains indices of elements in A:
```
array([[0, 0, 1, 2],
       [0, 3, 2, 1],
       [3, 2, 1, 0]])
```
How do I select values from A pointed by B, i.e.:
```
A[B] = [[2, 2, 4, 5],
        [1, 9, 8, 6],
        [2, 0, 7, 8]]
```

Response:
EDIT: np.take_along_axis is a builtin function for this use case implemented since numpy 1.15. See @hpaulj 's answer below for how to use it. You can use NumPy's advanced indexing -
```
A[np.arange(A.shape[0])[:,None],B]
```
One can also use linear indexing -
```
m,n = A.shape
out = np.take(A,B + n*np.arange(m)[:,None])
```
Sample run -
```

```python
A
#Output
#array([[2, 4, 5, 3],
#       [1, 6, 8, 9],
#       [8, 7, 0, 2]])
```

```python
B
#Output
#array([[0, 0, 1, 2],
#       [0, 3, 2, 1],
#       [3, 2, 1, 0]])
```

```python
A[np.arange(A.shape[0])[:,None],B]
#Output
#array([[2, 2, 4, 5],
#       [1, 9, 8, 6],
#       [2, 0, 7, 8]])
```

```python
m,n = A.shape
```

```python
np.take(A,B + n*np.arange(m)[:,None])
#Output
#array([[2, 2, 4, 5],
#       [1, 9, 8, 6],
#       [2, 0, 7, 8]])
#```
```


Question:
TensorFlow: using a tensor to index another tensor I have a basic question about how to do indexing in TensorFlow. In numpy:
```
x = np.asarray([1,2,3,3,2,5,6,7,1,3])
e = np.asarray([0,1,0,1,1,1,0,1])
#numpy
print x * e[x]
```
I can get
```
[1 0 3 3 0 5 0 7 1 3]
```
How can I do this in TensorFlow?
```
x = np.asarray([1,2,3,3,2,5,6,7,1,3])
e = np.asarray([0,1,0,1,1,1,0,1])
x_t = tf.constant(x)
e_t = tf.constant(e)
with tf.Session():
    ????
```
Thanks!

Response:
Fortunately, the exact case you're asking about is supported in TensorFlow by tf.gather():
```
result = x_t * tf.gather(e_t, x_t)

with tf.Session() as sess:
    print sess.run(result)  # ==> 'array([1, 0, 3, 3, 0, 5, 0, 7, 1, 3])'
```
The tf.gather() op is less powerful than NumPy's advanced indexing: it only supports extracting full slices of a tensor on its 0th dimension. Support for more general indexing has been requested, and is being tracked in this GitHub issue.


Question: How to force pandas read_csv to use float32 for all float columns? Because I don't need double precision My machine has limited memory and I want to process bigger datasets I need to pass the extracted data (as matrix) to BLAS libraries, and BLAS calls for single precision are 2x faster than for double precision equivalence. Note that not all columns in the raw csv file have float types. I only need to set float32 as the default for float columns.

Response:
Try:
```
import numpy as np
import pandas as pd

# Sample 100 rows of data to determine dtypes.
df_test = pd.read_csv(filename, nrows=100)

float_cols = [c for c in df_test if df_test[c].dtype == "float64"]
float32_cols = {c: np.float32 for c in float_cols}

df = pd.read_csv(filename, engine='c', dtype=float32_cols)
```
This first reads a sample of 100 rows of data (modify as required) to determine the type of each column. It the creates a list of those columns which are 'float64', and then uses dictionary comprehension to create a dictionary with these columns as the keys and 'np.float32' as the value for each key. Finally, it reads the whole file using the 'c' engine (required for assigning dtypes to columns) and then passes the float32_cols dictionary as a parameter to dtype.
```
df = pd.read_csv(filename, nrows=100)

```python
df
#Output
#   int_col  float1 string_col  float2
#0        1     1.2          a     2.2
#1        2     1.3          b     3.3
#2        3     1.4          c     4.4
```

```python
df.info()
#Output
#<class 'pandas.core.frame.DataFrame'>
#Int64Index: 3 entries, 0 to 2
#Data columns (total 4 columns):
#int_col       3 non-null int64
#float1        3 non-null float64
#string_col    3 non-null object
#float2        3 non-null float64
#dtypes: float64(2), int64(1), object(1)
```

df32 = pd.read_csv(filename, engine='c', dtype={c: np.float32 for c in float_cols})

```python
df32.info()
#Output
#<class 'pandas.core.frame.DataFrame'>
#Int64Index: 3 entries, 0 to 2
#Data columns (total 4 columns):
#int_col       3 non-null int64
#float1        3 non-null float32
#string_col    3 non-null object
#float2        3 non-null float32
#dtypes: float32(2), int64(1), object(1)
#```
```


Question:
what does numpy.apply_along_axis perform exactly? I have come across the numpy.apply_along_axis function in some code. And I don't understand the documentation about it. This is an example of the documentation:
```

```python
def new_func(a):
    """Divide elements of a by 2."""
    return a * 0.5
```

```python
b = np.array([[1,2,3], [4,5,6], [7,8,9]])
```

```python
np.apply_along_axis(new_func, 0, b)
#Output
#array([[ 0.5,  1. ,  1.5],
#       [ 2. ,  2.5,  3. ],
#       [ 3.5,  4. ,  4.5]])
#```
#As far I as thought I understood the documentation, I would have expected:
#```
#array([[ 0.5,  1. ,  1.5],
#       [ 4  ,  5  ,  6  ],
#       [ 7  ,  8  ,  9  ]])
#```
#i.e. having applied the function along the axis [1,2,3] which is axis 0 in [[1,2,3], [4,5,6], [7,8,9]] Obviously I am wrong. Could you correct me ?
```

Response:
apply_along_axis applies the supplied function along 1D slices of the input array, with the slices taken along the axis you specify. So in your example, new_func is applied over each slice of the array along the first axis. It becomes clearer if you use a vector valued function, rather than a scalar, like this:
```

```python
b = np.array([[1,2,3], [4,5,6], [7,8,9]])
```

```python
np.apply_along_axis(np.diff,0,b)
#Output
#array([[3, 3, 3],
#       [3, 3, 3]])
```

```python
np.apply_along_axis(np.diff,1,b)
#Output
#array([[1, 1],
#       [1, 1],
#       [1, 1]])
#```
#Here, numpy.diff (i.e. the arithmetic difference of adjacent array elements) is applied along each slice of either the first or second axis (dimension) of the input array.
```


Question:
Numpy array: sequence too large I have an array of size 11 called 'wavelength' and a larger array of size n called 'MN'. And 'model' is an m by n array. I'm doing this:
```
for i in xrange(10+len(wavelength)-2):
  y=np.empty(model[MN][i],float)
```
and getting this as an error:
```
File "test_prog.py", line 658, in <module>
    y=np.empty(model[MN][i],float)
ValueError: sequence too large; must be smaller than 32
```
I'm not sure what to do about that. I've looked elsewhere online but I can't find anything of obvious substance.

Response: sequence too large error means that you are creating a multidimension array that has a dimension larger than 32. For example: np.empty([1]*33) will raise this error. Are you sure you want to create >32 dimension array? If you want to create an empty array the same shape as model[MN][i], you should use: empty_like()


Question:
Python import error: cannot import name 'six' from 'sklearn.externals' I'm using numpy and mlrose, and all i have written so far is:
```
import numpy as np
import mlrose
```
However, when i run it, it comes up with an error message:
```
File "C:\Users\<my username>\AppData\Local\Programs\Python\Python38-32\lib\site-packages\mlrose\neural.py", line 12, in <module>
    from sklearn.externals import six
ImportError: cannot import name 'six' from 'sklearn.externals' (C:\Users\<my username>\AppData\Local\Programs\Python\Python38-32\lib\site-packages\sklearn\externals\__init__.py)
```
Any help on sorting this problem will be greatly appreciated.

Response:
Solution: The real answer is that the dependency needs to be changed by the mlrose maintainers. A workaround is:
```
import six
import sys
sys.modules['sklearn.externals.six'] = six
import mlrose
```


Question:
Pytorch: Why is the memory occupied by the `tensor` variable so small? In Pytorch 1.0.0, I found that a tensor variable occupies very small memory. I wonder how it stores so much data. Here's the code.
```
a = np.random.randn(1, 1, 128, 256)
b = torch.tensor(a, device=torch.device('cpu'))

a_size = sys.getsizeof(a)
b_size = sys.getsizeof(b)
```
a_size is 262288. b_size is 72.

Response:
The answer is in two parts. From the documentation of sys.getsizeof, firstly All built-in objects will return correct results, but this does not have to hold true for third-party extensions as it is implementation specific. so it could be that for tensors __sizeof__ is undefined or defined differently than you would expect - this function is not something you can rely on. Secondly Only the memory consumption directly attributed to the object is accounted for, not the memory consumption of objects it refers to. which means that if the torch.Tensor object merely holds a reference to the actual memory, this won't show in sys.getsizeof. This is indeed the case, if you check the size of the underlying storage instead, you will see the expected number
```
import torch, sys
b = torch.randn(1, 1, 128, 256, dtype=torch.float64)
sys.getsizeof(b)
>> 72
sys.getsizeof(b.storage())
>> 262208
```
Note: I am setting dtype to float64 explicitly, because that is the default dtype in numpy, whereas torch uses float32 by default.


Question: How can I calculate the nearest positive semi-definite matrix? I'm coming to Python from R and trying to reproduce a number of things that I'm used to doing in R using Python. The Matrix library for R has a very nifty function called nearPD() which finds the closest positive semi-definite (PSD) matrix to a given matrix. While I could code something up, being new to Python/Numpy I don't feel too excited about reinventing the wheel if something is already out there. Any tips on an existing implementation in Python?

Response:
I don't think there is a library which returns the matrix you want, but here is a "just for fun" coding of neareast positive semi-definite matrix algorithm from Higham (2000)
```
import numpy as np,numpy.linalg

def _getAplus(A):
    eigval, eigvec = np.linalg.eig(A)
    Q = np.matrix(eigvec)
    xdiag = np.matrix(np.diag(np.maximum(eigval, 0)))
    return Q*xdiag*Q.T

def _getPs(A, W=None):
    W05 = np.matrix(W**.5)
    return  W05.I * _getAplus(W05 * A * W05) * W05.I

def _getPu(A, W=None):
    Aret = np.array(A.copy())
    Aret[W > 0] = np.array(W)[W > 0]
    return np.matrix(Aret)

def nearPD(A, nit=10):
    n = A.shape[0]
    W = np.identity(n)
# W is the matrix used for the norm (assumed to be Identity matrix here)
# the algorithm should work for any diagonal W
    deltaS = 0
    Yk = A.copy()
    for k in range(nit):
        Rk = Yk - deltaS
        Xk = _getPs(Rk, W=W)
        deltaS = Xk - Rk
        Yk = _getPu(Xk, W=W)
    return Yk
```
When tested on the example from the paper, it returns the correct answer
```
print nearPD(np.matrix([[2,-1,0,0],[-1,2,-1,0],[0,-1,2,-1],[0,0,-1,2]]),nit=10)
[[ 1.         -0.80842467  0.19157533  0.10677227]
 [-0.80842467  1.         -0.65626745  0.19157533]
 [ 0.19157533 -0.65626745  1.         -0.80842467]
 [ 0.10677227  0.19157533 -0.80842467  1.        ]]
```


Question:
Truncating decimal digits numpy array of floats I want to truncate the float values within the numpy array, for .e.g.
```
2.34341232 --> 2.34
```
I read the post truncate floating point but its for one float. I don't want to run a loop on the numpy array, it will be quite expensive. Is there any inbuilt method within numpy which can do this easily? I do need output as a float not string.

Response:
Try out this modified version of numpy.trunc().
```
import numpy as np
def trunc(values, decs=0):
    return np.trunc(values*10**decs)/(10**decs)
```
Sadly, numpy.trunc function doesn't allow decimal truncation. Luckily, multiplying the argument and dividing it's result by a power of ten give the expected results.
```
vec = np.array([-4.79, -0.38, -0.001, 0.011, 0.4444, 2.34341232, 6.999])

trunc(vec, decs=2)
```
which returns:
```

```python
array([-4.79, -0.38, -0.  ,  0.01,  0.44,  2.34,  6.99])
#Output
#```
```


Question: threshold in 2D numpy array I have an array of shape 512x512 which contains numbers between 0 and 100 at ith and jth position. Now I want to select array[i,j] < 25 and zero at other places. I have tried with array = array[where(array<25)], which gives me a 1D array, but I want 2D. Please help me to solve this.

Response:
One solution:
```
result = (array < 25) * array
```
The first part array = 25 to zero in the original array:
```
array[array >= 25] = 0
```


Question:
igraph Graph from numpy or pandas adjacency matrix I have an adjacency matrix stored as a pandas.DataFrame:
```
node_names = ['A', 'B', 'C']
a = pd.DataFrame([[1,2,3],[3,1,1],[4,0,2]],
    index=node_names, columns=node_names)
a_numpy = a.as_matrix()
```
I'd like to create an igraph.Graph from either the pandas or the numpy adjacency matrices. In an ideal world the nodes would be named as expected. Is this possible? The tutorial seems to be silent on the issue.

Response:
In igraph you can use igraph.Graph.Adjacency to create a graph from an adjacency matrix without having to use zip. There are some things to be aware of when a weighted adjacency matrix is used and stored in a np.array or pd.DataFrame. igraph.Graph.Adjacency can't take an np.array as argument, but that is easily solved using tolist. Integers in adjacency-matrix are interpreted as number of edges between nodes rather than weights, solved by using adjacency as boolean. An example of how to do it:
```
import igraph
import pandas as pd

node_names = ['A', 'B', 'C']
a = pd.DataFrame([[1,2,3],[3,1,1],[4,0,2]], index=node_names, columns=node_names)

# Get the values as np.array, it's more convenenient.
A = a.values

# Create graph, A.astype(bool).tolist() or (A / A).tolist() can also be used.
g = igraph.Graph.Adjacency((A > 0).tolist())

# Add edge weights and node labels.
g.es['weight'] = A[A.nonzero()]
g.vs['label'] = node_names  # or a.index/a.columns
```
You can reconstruct your adjacency dataframe using get_adjacency by:
```
df_from_g = pd.DataFrame(g.get_adjacency(attribute='weight').data,
                         columns=g.vs['label'], index=g.vs['label'])
(df_from_g == a).all().all()  # --> True
```


Question: how to export HDF5 file to NumPy using H5PY? I have an existing hdf5 file with three arrays, i want to extract one of the arrays using h5py.

Response:
h5py already reads files in as numpy arrays, so just:
```
with h5py.File('the_filename', 'r') as f:
    my_array = f['array_name'][()]
```
The [()] means to read the entire array in; if you don't do that, it doesn't read the whole data but instead gives you lazy access to sub-parts (very useful when the array is huge but you only need a small part of it).


Question:
Finding the consecutive zeros in a numpy array I have the following array
```py
a = [1, 2, 3, 0, 0, 0, 0, 0, 0, 4, 5, 6, 0, 0, 0, 0, 9, 8, 7,0,10,11]
```
I would like to find the start and the end index of the array where the values are zeros consecutively. For the array above the output would be as follows
```py
[3,8],[12,15],[19]
```
I want to achieve this as efficiently as possible.

Response:
Here's a fairly compact vectorized implementation. I've changed the requirements a bit, so the return value is a bit more "numpythonic": it creates an array with shape (m, 2), where m is the number of "runs" of zeros. The first column is the index of the first 0 in each run, and the second is the index of the first nonzero element after the run. (This indexing pattern matches, for example, how slicing works and how the range function works.)
```
import numpy as np

def zero_runs(a):
    # Create an array that is 1 where a is 0, and pad each end with an extra 0.
    iszero = np.concatenate(([0], np.equal(a, 0).view(np.int8), [0]))
    absdiff = np.abs(np.diff(iszero))
    # Runs start and end where absdiff is 1.
    ranges = np.where(absdiff == 1)[0].reshape(-1, 2)
    return ranges
```
For example:
```

```python
a = [1, 2, 3, 0, 0, 0, 0, 0, 0, 4, 5, 6, 0, 0, 0, 0, 9, 8, 7, 0, 10, 11]
```

```python
runs = zero_runs(a)
```

```python
runs
#Output
#array([[ 3,  9],
#       [12, 16],
#       [19, 20]])
#```
#With this format, it is simple to get the number of zeros in each run:
#```
```

```python
runs[:,1] - runs[:,0]
#Output
#array([6, 4, 1])
#```
#It's always a good idea to check the edge cases:
#```
```

```python
zero_runs([0,1,2])
#Output
#array([[0, 1]])
```

```python
zero_runs([1,2,0])
#Output
#array([[2, 3]])
```

```python
zero_runs([1,2,3])
#Output
#array([], shape=(0, 2), dtype=int64)
```

```python
zero_runs([0,0,0])
#Output
#array([[0, 3]])
#```
```


Question:
Numpy inverse mask I want to inverse the true/false value in my numpy masked array. So in the example below i don't want to mask out the second value in the data array, I want to mask out the first and third value. Below is just an example. My masked array is created by a longer process than runs before. So I can not change the mask array itself. Is there another way to inverse the values?
```
import numpy
data = numpy.array([[ 1, 2, 5 ]])
mask = numpy.array([[0,1,0]])

numpy.ma.masked_array(data, mask)
```

Response:
```
import numpy
data = numpy.array([[ 1, 2, 5 ]])
mask = numpy.array([[0,1,0]])

numpy.ma.masked_array(data, ~mask) #note this probably wont work right for non-boolean (T/F) values
#or
numpy.ma.masked_array(data, numpy.logical_not(mask))
```
for example
```

```python
a = numpy.array([False,True,False])
```

```python
~a
#Output
#array([ True, False,  True], dtype=bool)
```

```python
numpy.logical_not(a)
#Output
#array([ True, False,  True], dtype=bool)
```

```python
a = numpy.array([0,1,0])
```

```python
~a
#Output
#array([-1, -2, -1])
```

```python
numpy.logical_not(a)
#Output
#array([ True, False,  True], dtype=bool)
#```
```


Question: How to extract the bits of larger numeric Numpy data types Numpy has a library function, np.unpackbits, which will unpack a uint8 into a bit vector of length 8. Is there a correspondingly fast way to unpack larger numeric types? E.g. uint16 or uint32. I am working on a question that involves frequent translation between numbers, for array indexing, and their bit vector representations, and the bottleneck is our pack and unpack functions.

Response:
You can do this with view and unpackbits Input:
```
unpackbits(arange(2, dtype=uint16).view(uint8))
```
Output:
```
[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0]
```
For a = arange(int(1e6), dtype=uint16) this is pretty fast at around 7 ms on my machine
```
%%timeit
unpackbits(a.view(uint8))

100 loops, best of 3: 7.03 ms per loop
```
As for endianness, you'll have to look at http://docs.scipy.org/doc/numpy/user/basics.byteswapping.html and apply the suggestions there depending on your needs.


Question: How many digits can float8, float16, float32, float64, and float128 contain? Numpy's dtype documentation only shows "x bits exponent, y bits mantissa" for each float type, but I couldn't translate that to exactly how many digits before/after the decimal point. Is there any simple formula/table to look this up in?

Response:
This is not as simple as usually expected. For accuracy of mantissa, there generally are two values: Given a value in decimal representation, how many decimal digits can be guaranteed to be preserved if converted from a decimal to a selected binary format and back (with default rounding). Given a value in binary format, how many decimal digits are needed if the value is converted to decimal format and back to the original binary format (again, with default rounding) to get the original value unchanged. In both cases, the decimal representation is treated as independent of the exponent, without leading and trailing zeros (for example, all of 0.0123e4, 1.23e2, 1.2300e2, 123, 123.0, 123000.000e-3 are 3 digits). For 32-bit binary floats, these two sizes are 6 and 9 decimal digits, respectively. In C ;, these are FLT_DIG and FLT_DECIMAL_DIG. (This is weird that 32-bit float keeps 7 decimal digits for total most of all numbers, but there are exceptions.) In C++, look at std::numeric_limits::digits10 and std::numeric_limits::max_digits10, respectively. For 64-bit binary floats, these are 15 and 17 (DBL_DIG and DBL_DECIMAL_DIG, respectively; and std::numeric_limits::{digits10, max_digits10}). General formulas for them (thx2 @MarkDickinson) ${format}_DIG (digits10): floor((p-1)*log10(2)) ${format}_DECIMAL_DIG (max_digits10): ceil(1+p*log10(2)) where p is number of digits in mantissa (including hidden one for normalized IEEE754 case). Also, comments with some mathematical explanation at C++ numeric limits page: The standard 32-bit IEEE 754 floating-point type has a 24 bit fractional part (23 bits written, one implied), which may suggest that it can represent 7 digit decimals (24 * std::log10(2) is 7.22), but relative rounding errors are non-uniform and some floating-point values with 7 decimal digits do not survive conversion to 32-bit float and back: the smallest positive example is 8.589973e9, which becomes 8.589974e9 after the roundtrip. These rounding errors cannot exceed one bit in the representation, and digits10 is calculated as (24-1)*std::log10(2), which is 6.92. Rounding down results in the value 6. Look for values for 16- and 128-bit floats in comments (but see below for what is 128-bit float in real). For the exponent, this is simpler because each of the border values (minimum normalized, minimum denormalized, maximum represented) are exact and can be easily obtained and printed. @PaulPanzer suggested numpy.finfo. It gives first of these values ({format}_DIG); maybe it is the thing you search:
```

```python
numpy.finfo(numpy.float16).precision
#Output
#3
```

```python
numpy.finfo(numpy.float32).precision
#Output
#6
```

```python
numpy.finfo(numpy.float64).precision
#Output
#15
```

```python
numpy.finfo(numpy.float128).precision
#Output
#18
#```
#but, on most systems (my one was Ubuntu 18.04 on x86-84) the value is confusing for float128; it is really for 80-bit x86 "extended" float with a 64 bit significand; real IEEE754 float128 has 112 significand bits and so the real value should have been 33, but numpy presents another type under this name. See here for details: in general, float128 is a delusion in numpy. UPD3: you mentioned float8 - there is no such type in IEEE754 set. One could imagine such type for some utterly specific purposes, but its range is too narrow for any universal usage.
```


Question:
Increment Numpy array with repeated indices I have a Numpy array and a list of indices whose values I would like to increment by one. This list may contain repeated indices, and I would like the increment to scale with the number of repeats of each index. Without repeats, the command is simple:
```
a=np.zeros(6).astype('int')
b=[3,2,5]
a[b]+=1
```
With repeats, I've come up with the following method.
```
b=[3,2,5,2]                     # indices to increment by one each replicate
bbins=np.bincount(b)
b.sort()                        # sort b because bincount is sorted
incr=bbins[np.nonzero(bbins)]   # create increment array
bu=np.unique(b)                 # sorted, unique indices (len(bu)=len(incr))
a[bu]+=incr
```
Is this the best way? Is there are risk involved with assuming that the np.bincount and np.unique operations would result in the same sorted order? Am I missing some simple Numpy operation to solve this?

Response:
In numpy >= 1.8, you can also use the at method of the addition 'universal function' ('ufunc'). As the docs note: For addition ufunc, this method is equivalent to a[indices] += b, except that results are accumulated for elements that are indexed more than once. So taking your example:
```
a = np.zeros(6).astype('int')
b = [3, 2, 5, 2]
```
…to then…
```
np.add.at(a, b, 1)
```
…will leave a as…
```
array([0, 0, 2, 1, 0, 1])
```


Question: What are the available datatypes for 'dtype' with NumPy's loadtxt() an genfromtxt? What are the available numpy.loadtxt or numpy.genfromtxt for importing table data with varying datatypes, and what are the available abbreviations for the use (e.g. i32 for integer)? This post demonstrates the use of conditions, which I was curious if somebody might elaborate on.

Response:
In addition to np.sctypeDict, there are these variables:
```

```python
np.typecodes
#Output
#{'All': '?bhilqpBHILQPefdgFDGSUVOMm',
# 'AllFloat': 'efdgFDG',
# 'AllInteger': 'bBhHiIlLqQpP',
# 'Character': 'c',
# 'Complex': 'FDG',
# 'Datetime': 'Mm',
# 'Float': 'efdg',
# 'Integer': 'bhilqp',
# 'UnsignedInteger': 'BHILQP'}
```

```python
np.sctypes
#Output
#{'complex': [numpy.complex64, numpy.complex128, numpy.complex192],
# 'float': [numpy.float16, numpy.float32, numpy.float64, numpy.float96],
# 'int': [numpy.int8, numpy.int16, numpy.int32, numpy.int32, numpy.int64],
# 'others': [bool, object, str, unicode, numpy.void],
# 'uint': [numpy.uint8, numpy.uint16, numpy.uint32, numpy.uint32, numpy.uint64]}
#```
```


Question:
Quickest way to find the nth largest value in a numpy Matrix There are lots of solutions to do this for a single array, but what about a matrix, such as:
```

```python
k
#Output
#array([[ 35,  48,  63],
#       [ 60,  77,  96],
#       [ 91, 112, 135]])
#```
#You can use k.max(), but of course this only returns the highest value, 135. What if I want the second or third?
```

Response:
As said, np.partition should be faster (at most O(n) running time):
```
np.partition(k.flatten(), -2)[-2]
```
should return the 2nd largest element. (partition guarantees that the numbered element is in position, all elements before are smaller, and all behind are bigger).


Question: Calculate the Cumulative Distribution Function (CDF) in Python How can I calculate in python the Cumulative Distribution Function (CDF)? I want to calculate it from an array of points I have (discrete distribution), not with the continuous distributions that, for example, scipy has.

Response:
(It is possible that my interpretation of the question is wrong. If the question is how to get from a discrete PDF into a discrete CDF, then np.cumsum divided by a suitable constant will do if the samples are equispaced. If the array is not equispaced, then np.cumsum of the array multiplied by the distances between the points will do.) If you have a discrete array of samples, and you would like to know the CDF of the sample, then you can just sort the array. If you look at the sorted result, you'll realize that the smallest value represents 0% , and largest value represents 100 %. If you want to know the value at 50 % of the distribution, just look at the array element which is in the middle of the sorted array. Let us have a closer look at this with a simple example:
```
import matplotlib.pyplot as plt
import numpy as np

# create some randomly ddistributed data:
data = np.random.randn(10000)

# sort the data:
data_sorted = np.sort(data)

# calculate the proportional values of samples
p = 1. * np.arange(len(data)) / (len(data) - 1)

# plot the sorted data:
fig = plt.figure()
ax1 = fig.add_subplot(121)
ax1.plot(p, data_sorted)
ax1.set_xlabel('$p$')
ax1.set_ylabel('$x$')

ax2 = fig.add_subplot(122)
ax2.plot(data_sorted, p)
ax2.set_xlabel('$x$')
ax2.set_ylabel('$p$')
```
This gives the following plot where the right-hand-side plot is the traditional cumulative distribution function. It should reflect the CDF of the process behind the points, but naturally, it is not as long as the number of points is finite. This function is easy to invert, and it depends on your application which form you need.


Question:
Pandas - Replace values based on index If I create a dataframe like so:
```
import pandas as pd, numpy as np

df = pd.DataFrame(np.random.randint(0,100,size=(100, 2)), columns=list('AB'))
```
How would I change the entry in column A to be the number 16 from row 0 -15, for example? In other words, how do I replace cells based purely on index?

Response:
Use loc:
```
df.loc[0:15,'A'] = 16
print (df)
     A   B
0   16  45
1   16   5
2   16  97
3   16  58
4   16  26
5   16  87
6   16  51
7   16  17
8   16  39
9   16  73
10  16  94
11  16  69
12  16  57
13  16  24
14  16  43
15  16  77
16  41   0
17   3  21
18   0  98
19  45  39
20  66  62
21   8  53
22  69  47
23  48  53
```
Solution with ix is deprecated.


Question:
R's which() and which.min() Equivalent in Python I read the similar topic here. I think the question is different or at least .index() could not solve my problem. This is a simple code in R and its answer:
```
x <- c(1:4, 0:5, 11)
x
#[1]  1  2  3  4  0  1  2  3  4  5 11
which(x==2)
# [1] 2 7
min(which(x==2))
# [1] 2
which.min(x)
#[1] 5
```
Which simply returns the index of the item which meets the condition. If x be the input for Python, how can I get the indeces for the elements which meet criteria x==2 and the one which is the smallest in the array which.min.
```
x = [1,2,3,4,0,1,2,3,4,11]
x=np.array(x)
x[x>2].index()
##'numpy.ndarray' object has no attribute 'index'
```

Response:
Numpy does have built-in functions for it
```
x = [1,2,3,4,0,1,2,3,4,11]
x=np.array(x)
np.where(x == 2)
np.min(np.where(x==2))
np.argmin(x)

np.where(x == 2)
Out[9]: (array([1, 6], dtype=int64),)

np.min(np.where(x==2))
Out[10]: 1

np.argmin(x)
Out[11]: 4
```


Question:
Better way to shuffle two numpy arrays in unison I have two numpy arrays of different shapes, but with the same length (leading dimension). I want to shuffle each of them, such that corresponding elements continue to correspond -- i.e. shuffle them in unison with respect to their leading indices. This code works, and illustrates my goals:
```
def shuffle_in_unison(a, b):
    assert len(a) == len(b)
    shuffled_a = numpy.empty(a.shape, dtype=a.dtype)
    shuffled_b = numpy.empty(b.shape, dtype=b.dtype)
    permutation = numpy.random.permutation(len(a))
    for old_index, new_index in enumerate(permutation):
        shuffled_a[new_index] = a[old_index]
        shuffled_b[new_index] = b[old_index]
    return shuffled_a, shuffled_b
```
For example:
```

```python
a = numpy.asarray([[1, 1], [2, 2], [3, 3]])
```

```python
b = numpy.asarray([1, 2, 3])
```

```python
shuffle_in_unison(a, b)
#Output
#(array([[2, 2],
#       [1, 1],
#       [3, 3]]), array([2, 1, 3]))
#```
#However, this feels clunky, inefficient, and slow, and it requires making a copy of the arrays -- I'd rather shuffle them in-place, since they'll be quite large. Is there a better way to go about this? Faster execution and lower memory usage are my primary goals, but elegant code would be nice, too. One other thought I had was this:
#```
#def shuffle_in_unison_scary(a, b):
#    rng_state = numpy.random.get_state()
#    numpy.random.shuffle(a)
#    numpy.random.set_state(rng_state)
#    numpy.random.shuffle(b)
#```
#This works...but it's a little scary, as I see little guarantee it'll continue to work -- it doesn't look like the sort of thing that's guaranteed to survive across numpy version, for example.
```

Response:
Your can use NumPy's array indexing:
```
def unison_shuffled_copies(a, b):
    assert len(a) == len(b)
    p = numpy.random.permutation(len(a))
    return a[p], b[p]
```
This will result in creation of separate unison-shuffled arrays.


Question: Is there a difference between scipy.pi, numpy.pi, or math.pi? In a project using SciPy and NumPy, when should one use scipy.pi vs numpy.pi vs just math.pi? Is there a difference between these values?

Response:
```

```python
import math
```

```python
import numpy as np
```

```python
import scipy
```

```python
math.pi == np.pi == scipy.pi
#Output
#True
#```
#So it doesn't matter, they are all the same value. The only reason all three modules provide a pi value is so if you are using just one of the three modules, you can conveniently have access to pi without having to import another module. They're not providing different values for pi.
```


Question: What is the difference between math.exp and numpy.exp and why do numpy creators choose to introduce exp again? exp means exponential function. Why do numpy creators introduce this function again? math.exp numpy.exp

Response:
The math.exp works only for scalars, whereas numpy.exp will work for arrays. Example:
```

```python
import math
```

```python
import numpy as np
```

```python
x = [1.,2.,3.,4.,5.]
```

```python
math.exp(x)
#Output
#Traceback (most recent call last):
#  File "<pyshell#10>", line 1, in <module>
#    math.exp(x)
#TypeError: a float is required
```

```python
np.exp(x)
#Output
#array([   2.71828183,    7.3890561 ,   20.08553692,   54.59815003,
#        148.4131591 ])
#```
#It is the same case for other math functions.
#```
```

```python
math.sin(x)
#Output
#Traceback (most recent call last):
#  File "<pyshell#12>", line 1, in <module>
#    math.sin(x)
#TypeError: a float is required
```

```python
np.sin(x)
#Output
#array([ 0.84147098,  0.90929743,  0.14112001, -0.7568025 , -0.95892427])
#```
#Also refer to this answer to check out how numpy is faster than math.
```


Question: Random Number from Histogram Suppose I create a histogram using scipy/numpy, so I have two arrays: one for the bin counts, and one for the bin edges. If I use the histogram to represent a probability distribution function, how can I efficiently generate random numbers from that distribution?

Response:
It's probably what np.random.choice does in @Ophion's answer, but you can construct a normalized cumulative density function, then choose based on a uniform random number:
```
from __future__ import division
import numpy as np
import matplotlib.pyplot as plt

data = np.random.normal(size=1000)
hist, bins = np.histogram(data, bins=50)

bin_midpoints = bins[:-1] + np.diff(bins)/2
cdf = np.cumsum(hist)
cdf = cdf / cdf[-1]
values = np.random.rand(10000)
value_bins = np.searchsorted(cdf, values)
random_from_cdf = bin_midpoints[value_bins]

plt.subplot(121)
plt.hist(data, 50)
plt.subplot(122)
plt.hist(random_from_cdf, 50)
plt.show()
```
A 2D case can be done as follows:
```
data = np.column_stack((np.random.normal(scale=10, size=1000),
                        np.random.normal(scale=20, size=1000)))
x, y = data.T
hist, x_bins, y_bins = np.histogram2d(x, y, bins=(50, 50))
x_bin_midpoints = x_bins[:-1] + np.diff(x_bins)/2
y_bin_midpoints = y_bins[:-1] + np.diff(y_bins)/2
cdf = np.cumsum(hist.ravel())
cdf = cdf / cdf[-1]

values = np.random.rand(10000)
value_bins = np.searchsorted(cdf, values)
x_idx, y_idx = np.unravel_index(value_bins,
                                (len(x_bin_midpoints),
                                 len(y_bin_midpoints)))
random_from_cdf = np.column_stack((x_bin_midpoints[x_idx],
                                   y_bin_midpoints[y_idx]))
new_x, new_y = random_from_cdf.T

plt.subplot(121, aspect='equal')
plt.hist2d(x, y, bins=(50, 50))
plt.subplot(122, aspect='equal')
plt.hist2d(new_x, new_y, bins=(50, 50))
plt.show()
```


Question:
Compare (assert equality of) two complex data structures containing numpy arrays in unittest I use Python's unittest module and want to check if two complex data structures are equal. The objects can be lists of dicts with all sorts of values: numbers, strings, Python containers (lists/tuples/dicts) and numpy arrays. The latter are the reason for asking the question, because I cannot just do
```
self.assertEqual(big_struct1, big_struct2)
```
because it produces a
```none
ValueError: The truth value of an array with more than one element is ambiguous.
Use a.any() or a.all()
```
I imagine that I need to write my own equality test for this. It should work for arbitrary structures. My current idea is a recursive function that: tries direct comparison of the current "node" of arg1 to the corresponding node of arg2; if no exception is raised, moves on ("terminal" nodes/leaves are processed here, too); if ValueError is caught, goes deeper until it finds a numpy.array; compares the arrays (e.g. like this). What seems a little problematic is keeping track of "corresponding" nodes of two structures, but perhaps zip is all I need here. The question is: are there good (simpler) alternatives to this approach? Maybe numpy presents some tools for this? If no alternatives are suggested, I will implement this idea (unless I have a better one) and post as an answer. P.S. I have a vague feeling that I might have seen a question addressing this problem, but I can't find it now. P.P.S. An alternative approach would be a function that traverses the structure and converts all numpy.arrays to lists, but is this any easier to implement? Seems the same to me. Edit: Subclassing numpy.ndarray sounds very promising, but obviously I don't have both sides of the comparison hard-coded into a test. One of them, though, is indeed hardcoded, so I can: populate it with custom subclasses of numpy.array; change isinstance(other, SaneEqualityArray) to isinstance(other, np.ndarray) in jterrace's answer; always use it as LHS in comparisons. My questions in this regard are: Will it work (I mean, it sounds all right to me, but maybe some tricky edge cases will not be handled correctly)? Will my custom object always end up as LHS in the recursive equality checks, as I expect? Again, are there better ways (given that I get at least one of the structures with real numpy arrays). Edit 2: I tried it out, the (seemingly) working implementation is shown in this answer.

Response: Would have commented, but it gets too long... Fun fact, you cannot use == to test if arrays are the same I would suggest you use np.testing.assert_array_equal instead. that checks dtype, shape, etc., that doesn't fail for the neat little math of (float('nan') == float('nan')) == False (normal python sequence == has an even more fun way of ignoring this sometimes, because it uses PyObject_RichCompareBool which does a (for NaNs incorrect) is quick check (for testing of course that is perfect)... There is also assert_allclose because floating point equality can get very tricky if you do actual calculations and you usually want almost the same values, since the values can become hardware depended or possibly random depending what you do with them. I would almost suggest trying serializing it with pickle if you want something this insanely nested, but that is overly strict (and point 3 is of course fully broken then), for example the memory layout of your array does not matter, but matters to its serialization.


Question:
How to get the indices list of all NaN value in numpy array? Say now I have a numpy array which is defined as,
```
[[1,2,3,4],
[2,3,NaN,5],
[NaN,5,2,3]]
```
Now I want to have a list that contains all the indices of the missing values, which is [(1,2),(2,0)] at this case. Is there any way I can do that?

Response:
np.isnan combined with np.argwhere
```
x = np.array([[1,2,3,4],
              [2,3,np.nan,5],
              [np.nan,5,2,3]])
np.argwhere(np.isnan(x))
```
output:
```
array([[1, 2],
       [2, 0]])
```


Question:
load csv into 2D matrix with numpy for plotting Given this CSV file:
```
"A","B","C","D","E","F","timestamp"
611.88243,9089.5601,5133.0,864.07514,1715.37476,765.22777,1.291111964948E12
611.88243,9089.5601,5133.0,864.07514,1715.37476,765.22777,1.291113113366E12
611.88243,9089.5601,5133.0,864.07514,1715.37476,765.22777,1.291120650486E12
```
I simply want to load it as a matrix/ndarray with 3 rows and 7 columns. However, for some reason, all I can get out of numpy is an ndarray with 3 rows (one per line) and no columns.
```
r = np.genfromtxt(fname,delimiter=',',dtype=None, names=True)
print r
print r.shape

[ (611.88243, 9089.5601000000006, 5133.0, 864.07514000000003, 1715.3747599999999, 765.22776999999996, 1291111964948.0)
 (611.88243, 9089.5601000000006, 5133.0, 864.07514000000003, 1715.3747599999999, 765.22776999999996, 1291113113366.0)
 (611.88243, 9089.5601000000006, 5133.0, 864.07514000000003, 1715.3747599999999, 765.22776999999996, 1291120650486.0)]
(3,)
```
I can manually iterate and hack it into the shape I want, but this seems silly. I just want to load it as a proper matrix so I can slice it across different dimensions and plot it, just like in matlab.

Response:
Pure numpy
```
numpy.loadtxt(open("test.csv", "rb"), delimiter=",", skiprows=1)
```
Check out the loadtxt documentation. You can also use python's csv module:
```
import csv
import numpy
reader = csv.reader(open("test.csv", "rb"), delimiter=",")
x = list(reader)
result = numpy.array(x).astype("float")
```
You will have to convert it to your favorite numeric type. I guess you can write the whole thing in one line:
```
result = numpy.array(list(csv.reader(open("test.csv", "rb"), delimiter=","))).astype("float")
```
Added Hint: You could also use pandas.io.parsers.read_csv and get the associated numpy array which can be faster.


Question: Convert an image RGB->Lab with python What is the preferred way of doing the conversion using PIL/Numpy/SciPy today?

Response:
Since 2010 when the linked question was asked the corresponding code moved from scipy to a separate toolkit: http://scikit-image.org/ So here's the code I was actually looking for:
```
from skimage import io, color
rgb = io.imread(filename)
lab = color.rgb2lab(rgb)
```
It should also be noted that due to Lab nature srgb->lab conversion depends on an additional parameter: whitepoint, eg: • Photoshop uses a white point called D50 (which is a standard for icc) • OpenCV and skimage use D65 (which is a standard for srgb). • default Matlab implementation uses D50 (it is capable of using others), This nice FAQ explains it this way: You should use D65 unless you have a good reason to use something else. The print industry commonly uses D50 and photography commonly uses D55. These represent compromises between the conditions of indoor (tungsten) and daylight viewing. You can tell which whitepoint you're dealing with by converting RGB (0,0,255) to Lab: • D50 would give you (30, 68, -112) • D55 (30, 73, -110) • D65 (32, 79, -108) The numbers after 'D' correspond to (internally) used color temperature of white point: D50 = 5003 K (yellowish), D65 = 6504 K (blueish) I'm grateful to Alex and Roman for their answers because they pointed me into the right direction.


Question: Find indices of elements equal to zero in a NumPy array NumPy has the efficient function/method nonzero() to identify the indices of non-zero elements in an ndarray object. What is the most efficient way to obtain the indices of the elements that do have a value of zero?

Response:
numpy.where() is my favorite.
```

```python
x = numpy.array([1,0,2,0,3,0,4,5,6,7,8])
```

```python
numpy.where(x == 0)[0]
#Output
#array([1, 3, 5])
#```
#The method where returns a tuple of ndarrays, each corresponding to a different dimension of the input. Since the input is one-dimensional, the [0] unboxes the tuple's only element.
```


Question: How do I check which version of NumPy I'm using? How can I check which version of NumPy I'm using?

Response:
```
import numpy
numpy.version.version
```


Question:
How do I create character arrays in numpy? Say I have the following array:
```
import numpy as np
a = ['hello','snake','plate']
```
I want this to turn into a numpy array b so that:
```
b[0,0] = 'h'
b[0,1] = 'e'
b[0,2] = 'l'
b[1,0] = 's'
...
```
I want the standard numpy tricks to work, like broadcasting, comparison, etc. How is it done? And where is this in the numpy documentation? Thanks! Uri

Response:
Actually, you can do this without any copies or list comprehensions in numpy (caveats about non-equal-length strings aside...). Just view it as a 1 character string array and reshape it:
```
import numpy as np

x = np.array(['hello','snake','plate'], dtype=str)
y = x.view('S1').reshape((x.size, -1))

print repr(y)
```
This yields:
```
array([['h', 'e', 'l', 'l', 'o'],
       ['s', 'n', 'a', 'k', 'e'],
       ['p', 'l', 'a', 't', 'e']],
      dtype='|S1')
```
Generally speaking, though, I'd avoid using numpy arrays to store strings in most cases. There are cases where it's useful, but you're usually better off sticking to data structures that allow variable-length strings for, well, holding strings.