Question:
Pytorch tensor to numpy array I have a pytorch Tensor of shape [4, 3, 966, 1296]. I want to convert it to numpy array using the following code:
```
imgs = imgs.numpy()[:, ::-1, :, :]
```
How does that code work?

Response:
I believe you also have to use .detach(). I had to convert my Tensor to a numpy array on Colab which uses CUDA and GPU. I did it like the following:
```
# this is just my embedding matrix which is a Torch tensor object
embedding = learn.model.u_weight

embedding_list = list(range(0, 64382))

input = torch.cuda.LongTensor(embedding_list)
tensor_array = embedding(input)
# the output of the line below is a numpy array
tensor_array.cpu().detach().numpy()
```


Question:
How to return 0 with divide by zero I'm trying to perform an element wise divide in python, but if a zero is encountered, I need the quotient to just be zero. For example:
```
array1 = np.array([0, 1, 2])
array2 = np.array([0, 1, 1])

array1 / array2 # should be np.array([0, 1, 2])
```
I could always just use a for-loop through my data, but to really utilize numpy's optimizations, I need the divide function to return 0 upon divide by zero errors instead of ignoring the error. Unless I'm missing something, it doesn't seem numpy.seterr() can return values upon errors. Does anyone have any other suggestions on how I could get the best out of numpy while setting my own divide by zero error handling?

Response:
In numpy v1.7+, you can take advantage of the "where" option for ufuncs. You can do things in one line and you don't have to deal with the errstate context manager.
```

```python
a = np.array([-1, 0, 1, 2, 3], dtype=float)
```

```python
b = np.array([ 0, 0, 0, 2, 2], dtype=float)
```

# If you don't pass `out` the indices where (b == 0) will be uninitialized!

```python
c = np.divide(a, b, out=np.zeros_like(a), where=b!=0)
```

```python
print(c)
#Output
#[ 0.   0.   0.   1.   1.5]
#```
#In this case, it does the divide calculation anywhere 'where' b does not equal zero. When b does equal zero, then it remains unchanged from whatever value you originally gave it in the 'out' argument.
```


Question:
Find the most frequent number in a NumPy array Suppose I have the following NumPy array:
```
a = np.array([1,2,3,1,2,1,1,1,3,2,2,1])
```
How can I find the most frequent number in this array?

Response:
If your list contains all non-negative ints, you should take a look at numpy.bincounts: http://docs.scipy.org/doc/numpy/reference/generated/numpy.bincount.html and then probably use np.argmax:
```
a = np.array([1,2,3,1,2,1,1,1,3,2,2,1])
counts = np.bincount(a)
print(np.argmax(counts))
```
For a more complicated list (that perhaps contains negative numbers or non-integer values), you can use np.histogram in a similar way. Alternatively, if you just want to work in python without using numpy, collections.Counter is a good way of handling this sort of data.
```
from collections import Counter
a = [1,2,3,1,2,1,1,1,3,2,2,1]
b = Counter(a)
print(b.most_common(1))
```


Question: How do I use numpy.where()? What should I pass, and what does the result mean? [closed] Closed. This question needs to be more focused. It is not currently accepting answers. Want to improve this question? Guide the asker to update the question so it focuses on a single, specific problem. Narrowing the question will help others answer the question concisely. You may edit the question if you feel you can improve it yourself. If edited, the question will be reviewed and might be reopened. Closed 9 years ago. Improve this question I tried reading the documentation for numpy.where(), but I'm still confused. What should I pass for the condition, x and y values? When I pass only condition, what does the result mean and how can I use it? What about when I pass all three? I found How does python numpy.where() work? but it didn't answer my question because it seems to be about the implementation rather than about how to use it. Numpy where() on a 2D matrix also didn't explain things for me; I'm looking for a step-by-step explanation, rather than a how-to guide for a specific case. Please include examples with both 1D and 2D source data.

Response:
After fiddling around for a while, I figured things out, and am posting them here hoping it will help others. Intuitively, np.where is like asking "tell me where in this array, entries satisfy a given condition".
```

```python
a = np.arange(5,10)
```

```python
np.where(a < 8)       # tell me where in a, entries are < 8
#Output
#(array([0, 1, 2]),)       # answer: entries indexed by 0, 1, 2
#```
#It can also be used to get entries in array that satisfy the condition:
#```
```

```python
a[np.where(a < 8)]
#Output
#array([5, 6, 7])          # selects from a entries 0, 1, 2
#```
#When a is a 2d array, np.where() returns an array of row idx's, and an array of col idx's:
#```
```

```python
a = np.arange(4,10).reshape(2,3)
#Output
#array([[4, 5, 6],
#       [7, 8, 9]])
```

```python
np.where(a > 8)
#Output
#(array(1), array(2))
#```
#As in the 1d case, we can use np.where() to get entries in the 2d array that satisfy the condition:
#```
```

```python
a[np.where(a > 8)] # selects from a entries 0, 1, 2
#Output
#```
#array([9]) Note, when a is 1d, np.where() still returns an array of row idx's and an array of col idx's, but columns are of length 1, so latter is empty array.
```


Question:
How to get element-wise matrix multiplication (Hadamard product) in numpy? I have two matrices
```py
a = np.matrix([[1,2], [3,4]])
b = np.matrix([[5,6], [7,8]])
```
and I want to get the element-wise product, [[1*5,2*6], [3*7,4*8]], which equals
```
matrix([[5, 12], [21, 32]])
```
I have tried np.dot(a,b) and a*b but both give the result matrix([[19, 22], [43, 50]]) which is the matrix product, not the element-wise product. How can I get the the element-wise product (aka Hadamard product) using built-in functions?

Response:
For elementwise multiplication of matrix objects, you can use numpy.multiply:
```
import numpy as np
a = np.array([[1,2],[3,4]])
b = np.array([[5,6],[7,8]])
np.multiply(a,b)
```
Result
```
array([[ 5, 12],
       [21, 32]])
```
However, you should really use array instead of matrix. matrix objects have all sorts of horrible incompatibilities with regular ndarrays. With ndarrays, you can just use * for elementwise multiplication:
```
a * b
```
If you're on Python 3.5+, you don't even lose the ability to perform matrix multiplication with an operator, because @ does matrix multiplication now:
```
a @ b  # matrix multiplication
```


Question:
best way to preserve numpy arrays on disk I am looking for a fast way to preserve large numpy arrays. I want to save them to the disk in a binary format, then read them back into memory relatively fastly. cPickle is not fast enough, unfortunately. I found numpy.savez and numpy.load. But the weird thing is, numpy.load loads a npy file into "memory-map". That means regular manipulating of arrays really slow. For example, something like this would be really slow:
```
#!/usr/bin/python
import numpy as np;
import time;
from tempfile import TemporaryFile

n = 10000000;

a = np.arange(n)
b = np.arange(n) * 10
c = np.arange(n) * -0.5

file = TemporaryFile()
np.savez(file,a = a, b = b, c = c);

file.seek(0)
t = time.time()
z = np.load(file)
print "loading time = ", time.time() - t

t = time.time()
aa = z['a']
bb = z['b']
cc = z['c']
print "assigning time = ", time.time() - t;
```
more precisely, the first line will be really fast, but the remaining lines that assign the arrays to obj are ridiculously slow:
```
loading time =  0.000220775604248
assining time =  2.72940087318
```
Is there any better way of preserving numpy arrays? Ideally, I want to be able to store multiple arrays in one file.

Response: I've compared performance (space and time) for a number of ways to store numpy arrays. Few of them support multiple arrays per file, but perhaps it's useful anyway. Npy and binary files are both really fast and small for dense data. If the data is sparse or very structured, you might want to use npz with compression, which'll save a lot of space but cost some load time. If portability is an issue, binary is better than npy. If human readability is important, then you'll have to sacrifice a lot of performance, but it can be achieved fairly well using csv (which is also very portable of course). More details and the code are available at the github repo.


Question:
How to save and load numpy.array() data properly? I wonder, how to save and load numpy.array data properly. Currently I'm using the numpy.savetxt() method. For example, if I got an array markers, which looks like this: I try to save it by the use of:
```
numpy.savetxt('markers.txt', markers)
```
In other script I try to open previously saved file:
```
markers = np.fromfile("markers.txt")
```
And that's what I get... Saved data first looks like this:
```
0.000000000000000000e+00
0.000000000000000000e+00
0.000000000000000000e+00
0.000000000000000000e+00
0.000000000000000000e+00
0.000000000000000000e+00
0.000000000000000000e+00
0.000000000000000000e+00
0.000000000000000000e+00
0.000000000000000000e+00
```
But when I save just loaded data by the use of the same method, ie. numpy.savetxt() it looks like this:
```
1.398043286095131769e-76
1.398043286095288860e-76
1.396426376485745879e-76
1.398043286055061908e-76
1.398043286095288860e-76
1.182950697433698368e-76
1.398043275797188953e-76
1.398043286095288860e-76
1.210894289234927752e-99
1.398040649781712473e-76
```
What am I doing wrong? PS there are no other "backstage" operation which I perform. Just saving and loading, and that's what I get. Thank you in advance.

Response:
The most reliable way I have found to do this is to use np.savetxt with np.loadtxt and not np.fromfile which is better suited to binary files written with tofile. The np.fromfile and np.tofile methods write and read binary files whereas np.savetxt writes a text file. So, for example:
```
a = np.array([1, 2, 3, 4])
np.savetxt('test1.txt', a, fmt='%d')
b = np.loadtxt('test1.txt', dtype=int)
a == b
# array([ True,  True,  True,  True], dtype=bool)
```
Or:
```
a.tofile('test2.dat')
c = np.fromfile('test2.dat', dtype=int)
c == a
# array([ True,  True,  True,  True], dtype=bool)
```
I use the former method even if it is slower and creates bigger files (sometimes): the binary format can be platform dependent (for example, the file format depends on the endianness of your system). There is a platform independent format for NumPy arrays, which can be saved and read with np.save and np.load:
```
np.save('test3.npy', a)    # .npy extension is added if not given
d = np.load('test3.npy')
a == d
# array([ True,  True,  True,  True], dtype=bool)
```


Question: Finding local maxima/minima with Numpy in a 1D numpy array Can you suggest a module function from numpy/scipy that can find local maxima/minima in a 1D numpy array? Obviously the simplest approach ever is to have a look at the nearest neighbours, but I would like to have an accepted solution that is part of the numpy distro.

Response:
In SciPy >= 0.11
```
import numpy as np
from scipy.signal import argrelextrema

x = np.random.random(12)

# for local maxima
argrelextrema(x, np.greater)

# for local minima
argrelextrema(x, np.less)
```
Produces
```

```python
x
#Output
#array([ 0.56660112,  0.76309473,  0.69597908,  0.38260156,  0.24346445,
#    0.56021785,  0.24109326,  0.41884061,  0.35461957,  0.54398472,
#    0.59572658,  0.92377974])
```

```python
argrelextrema(x, np.greater)
#Output
#(array([1, 5, 7]),)
```

```python
argrelextrema(x, np.less)
#Output
#(array([4, 6, 8]),)
#```
#Note, these are the indices of x that are local max/min. To get the values, try:
#```
```

```python
x[argrelextrema(x, np.greater)[0]]
#Output
#```
#scipy.signal also provides argrelmax and argrelmin for finding maxima and minima respectively.
```


Question:
Detect if a NumPy array contains at least one non-numeric value? I need to write a function which will detect if the input contains at least one value which is non-numeric. If a non-numeric value is found I will raise an error (because the calculation should only return a numeric value). The number of dimensions of the input array is not known in advance - the function should give the correct value regardless of ndim. As an extra complication the input could be a single float or numpy.float64 or even something oddball like a zero-dimensional array. The obvious way to solve this is to write a recursive function which iterates over every iterable object in the array until it finds a non-iterabe. It will apply the numpy.isnan() function over every non-iterable object. If at least one non-numeric value is found then the function will return False immediately. Otherwise if all the values in the iterable are numeric it will eventually return True. That works just fine, but it's pretty slow and I expect that NumPy has a much better way to do it. What is an alternative that is faster and more numpyish? Here's my mockup:
```
def contains_nan( myarray ):
    """
    @param myarray : An n-dimensional array or a single float
    @type myarray : numpy.ndarray, numpy.array, float
    @returns: bool
    Returns true if myarray is numeric or only contains numeric values.
    Returns false if at least one non-numeric value exists
    Not-A-Number is given by the numpy.isnan() function.
    """
    return True
```

Response:
This should be faster than iterating and will work regardless of shape.
```
numpy.isnan(myarray).any()
```
Edit: 30x faster:
```
import timeit
s = 'import numpy;a = numpy.arange(10000.).reshape((100,100));a[10,10]=numpy.nan'
ms = [
    'numpy.isnan(a).any()',
    'any(numpy.isnan(x) for x in a.flatten())']
for m in ms:
    print "  %.2f s" % timeit.Timer(m, s).timeit(1000), m
```
Results:
```
0.11 s numpy.isnan(a).any()
  3.75 s any(numpy.isnan(x) for x in a.flatten())
```
Bonus: it works fine for non-array NumPy types:
```

```python
a = numpy.float64(42.)
```

```python
numpy.isnan(a).any()
#Output
#False
```

```python
a = numpy.float64(numpy.nan)
```

```python
numpy.isnan(a).any()
#Output
#True
#```
```


Question:
Generate random array of floats between a range I haven't been able to find a function to generate an array of random floats of a given length between a certain range. I've looked at Random sampling but no function seems to do what I need. random.uniform comes close but it only returns a single element, not a specific number. This is what I'm after:
```
ran_floats = some_function(low=0.5, high=13.3, size=50)
```
which would return an array of 50 random non-unique floats (ie: repetitions are allowed) uniformly distributed in the range [0.5, 13.3]. Is there such a function?

Response:
np.random.uniform fits your use case:
```
sampl = np.random.uniform(low=0.5, high=13.3, size=(50,))
```
Update Oct 2019: While the syntax is still supported, it looks like the API changed with NumPy 1.17 to support greater control over the random number generator. Going forward the API has changed and you should look at https://docs.scipy.org/doc/numpy/reference/random/generated/numpy.random.Generator.uniform.html The enhancement proposal is here: https://numpy.org/neps/nep-0019-rng-policy.html


Question:
How do I find the length (or dimensions, size) of a numpy matrix in python? [duplicate] This question already has answers here: Numpy array dimensions (10 answers) Closed 12 years ago. For a numpy matrix in python
```
from numpy import matrix
A = matrix([[1,2],[3,4]])
```
How can I find the length of a row (or column) of this matrix? Equivalently, how can I know the number of rows or columns? So far, the only solution I've found is:
```
len(A)
len(A[:,1])
len(A[1,:])
```
Which returns 2, 2, and 1, respectively. From this I've gathered that len() will return the number of rows, so I can always us the transpose, len(A.T), for the number of columns. However, this feels unsatisfying and arbitrary, as when reading the line len(A), it isn't immediately obvious that this should return the number of rows. It actually works differently than len([1,2]) would for a 2D python array, as this would return 2. So, is there a more intuitive way to find the size of a matrix, or is this the best I have?

Response:
shape is a property of both numpy ndarray's and matrices.
```
A.shape
```
will return a tuple (m, n), where m is the number of rows, and n is the number of columns. In fact, the numpy matrix object is built on top of the ndarray object, one of numpy's two fundamental objects (along with a universal function object), so it inherits from ndarray


Question:
ValueError when checking if variable is None or numpy.array I'd like to check if variable is None or numpy.array. I've implemented check_a function to do this.
```
def check_a(a):
    if not a:
        print "please initialize a"

a = None
check_a(a)
a = np.array([1,2])
check_a(a)
```
But, this code raises ValueError. What is the straight forward way?
```
ValueError                                Traceback (most recent call last)
<ipython-input-41-0201c81c185e> in <module>()
      6 check_a(a)
      7 a = np.array([1,2])
----> 8 check_a(a)

<ipython-input-41-0201c81c185e> in check_a(a)
      1 def check_a(a):
----> 2     if not a:
      3         print "please initialize a"
      4
      5 a = None

ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
```

Response:
Using not a to test whether a is None assumes that the other possible values of a have a truth value of True. However, most NumPy arrays don't have a truth value at all, and not cannot be applied to them. If you want to test whether an object is None, the most general, reliable way is to literally use an is check against None:
```
if a is None:
    ...
else:
    ...
```
This doesn't depend on objects having a truth value, so it works with NumPy arrays. Note that the test has to be is, not ==. is is an object identity test. == is whatever the arguments say it is, and NumPy arrays say it's a broadcasted elementwise equality comparison, producing a boolean array:
```

```python
a = numpy.arange(5)
```

```python
a == None
#Output
#array([False, False, False, False, False])
```

```python
if a == None:
    pass
#Output
#...
#Traceback (most recent call last):
#  File "<stdin>", line 1, in <module>
#ValueError: The truth value of an array with more than one element is ambiguous.
# Use a.any() or a.all()
#```
#On the other side of things, if you want to test whether an object is a NumPy array, you can test its type:
#```
## Careful - the type is np.ndarray, not np.array. np.array is a factory function.
#if type(a) is np.ndarray:
#    ...
#else:
#    ...
#```
#You can also use isinstance, which will also return True for subclasses of that type (if that is what you want). Considering how terrible and incompatible np.matrix is, you may not actually want this:
#```
## Again, ndarray, not array, because array is a factory function.
#if isinstance(a, np.ndarray):
#    ...
#else:
#    ...
#```
```


Question:
Test if numpy array contains only zeros We initialize a numpy array with zeros as bellow:
```
np.zeros((N,N+1))
```
But how do we check whether all elements in a given n*n numpy array matrix is zero. The method just need to return a True if all the values are indeed zero.

Response:
The other answers posted here will work, but the clearest and most efficient function to use is numpy.any():
```

```python
all_zeros = not np.any(a)
#Output
#```
#or
#```
```

```python
all_zeros = not a.any()
#Output
#```
#This is preferred over numpy.all(a==0) because it uses less RAM. (It does not require the temporary array created by the a==0 term.) Also, it is faster than numpy.count_nonzero(a) because it can return immediately when the first nonzero element has been found. Edit: As @Rachel pointed out in the comments, np.any() no longer uses "short-circuit" logic, so you won't see a speed benefit for small arrays.
```


Question:
NumPy selecting specific column index per row by using a list of indexes I'm struggling to select the specific columns per row of a NumPy matrix. Suppose I have the following matrix which I would call X:
```
[1, 2, 3]
[4, 5, 6]
[7, 8, 9]
```
I also have a list of column indexes per every row which I would call Y:
```
[1, 0, 2]
```
I need to get the values:
```
[2]
[4]
[9]
```
Instead of a list with indexes Y, I can also produce a matrix with the same shape as X where every column is a bool / int in the range 0-1 value, indicating whether this is the required column.
```
[0, 1, 0]
[1, 0, 0]
[0, 0, 1]
```
I know this can be done with iterating over the array and selecting the column values I need. However, this will be executed frequently on big arrays of data and that's why it has to run as fast as it can. I was thus wondering if there is a better solution?

Response:
If you've got a boolean array you can do direct selection based on that like so:
```

```python
a = np.array([True, True, True, False, False])
```

```python
b = np.array([1,2,3,4,5])
```

```python
b[a]
#Output
#array([1, 2, 3])
#```
#To go along with your initial example you could do the following:
#```
```

```python
a = np.array([[1,2,3], [4,5,6], [7,8,9]])
```

```python
b = np.array([[False,True,False],[True,False,False],[False,False,True]])
```

```python
a[b]
#Output
#array([2, 4, 9])
#```
#You can also add in an arange and do direct selection on that, though depending on how you're generating your boolean array and what your code looks like YMMV.
#```
```

```python
a = np.array([[1,2,3], [4,5,6], [7,8,9]])
```

```python
a[np.arange(len(a)), [1,0,2]]
#Output
#array([2, 4, 9])
#```
```


Question:
Most efficient way to find mode in numpy array I have a 2D array containing integers (both positive or negative). Each row represents the values over time for a particular spatial site, whereas each column represents values for various spatial sites for a given time. So if the array is like:
```
1 3 4 2 2 7
5 2 2 1 4 1
3 3 2 2 1 1
```
The result should be
```
1 3 2 2 2 1
```
Note that when there are multiple values for mode, any one (selected randomly) may be set as mode. I can iterate over the columns finding mode one at a time but I was hoping numpy might have some in-built function to do that. Or if there is a trick to find that efficiently without looping.

Response:
Check scipy.stats.mode() (inspired by @tom10's comment):
```
import numpy as np
from scipy import stats

a = np.array([[1, 3, 4, 2, 2, 7],
              [5, 2, 2, 1, 4, 1],
              [3, 3, 2, 2, 1, 1]])

m = stats.mode(a)
print(m)
```
Output:
```
ModeResult(mode=array([[1, 3, 2, 2, 1, 1]]), count=array([[1, 2, 2, 2, 1, 2]]))
```
As you can see, it returns both the mode as well as the counts. You can select the modes directly via m[0]:
```
print(m[0])
```
Output:
```
[[1 3 2 2 1 1]]
```


Question:
Iterating over a numpy array Is there a less verbose alternative to this:
```
for x in xrange(array.shape[0]):
    for y in xrange(array.shape[1]):
        do_stuff(x, y)
```
I came up with this:
```
for x, y in itertools.product(map(xrange, array.shape)):
    do_stuff(x, y)
```
Which saves one indentation, but is still pretty ugly. I'm hoping for something that looks like this pseudocode:
```
for x, y in array.indices:
    do_stuff(x, y)
```
Does anything like that exist?

Response:
I think you're looking for the ndenumerate.
```

```python
a =numpy.array([[1,2],[3,4],[5,6]])
```

```python
for (x,y), value in numpy.ndenumerate(a):
 print x,y

#Output
#0 0
#0 1
#1 0
#1 1
#2 0
#2 1
#```
#Regarding the performance. It is a bit slower than a list comprehension.
#```
#X = np.zeros((100, 100, 100))
```

%timeit list([((i,j,k), X[i,j,k]) for i in range(X.shape[0]) for j in range(X.shape[1]) for k in range(X.shape[2])])
1 loop, best of 3: 376 ms per loop

%timeit list(np.ndenumerate(X))
1 loop, best of 3: 570 ms per loop
```
If you are worried about the performance you could optimise a bit further by looking at the implementation of ndenumerate, which does 2 things, converting to an array and looping. If you know you have an array, you can call the .coords attribute of the flat iterator.
```
a = X.flat
%timeit list([(a.coords, x) for x in a.flat])
1 loop, best of 3: 305 ms per loop
```


Question:
python numpy machine epsilon I am trying to understand what is machine epsilon. According to the Wikipedia, it can be calculated as follows:
```
def machineEpsilon(func=float):
    machine_epsilon = func(1)
    while func(1)+func(machine_epsilon) != func(1):
        machine_epsilon_last = machine_epsilon
        machine_epsilon = func(machine_epsilon) / func(2)
    return machine_epsilon_last
```
However, it is suitable only for double precision numbers. I am interested in modifying it to support also single precision numbers. I read that numpy can be used, particularly numpy.float32 class. Can anybody help with modifying the function?

Response:
An easier way to get the machine epsilon for a given float type is to use np.finfo():
```
print(np.finfo(float).eps)
# 2.22044604925e-16

print(np.finfo(np.float32).eps)
# 1.19209e-07
```


Question:
What is the difference between NaN and None? I am reading two columns of a csv file using pandas readcsv() and then assigning the values to a dictionary. The columns contain strings of numbers and letters. Occasionally there are cases where a cell is empty. In my opinion, the value read to that dictionary entry should be None but instead nan is assigned. Surely None is more descriptive of an empty cell as it has a null value, whereas nan just says that the value read is not a number. Is my understanding correct, what IS the difference between None and nan? Why is nan assigned instead of None? Also, my dictionary check for any empty cells has been using numpy.isnan():
```
for k, v in my_dict.iteritems():
    if np.isnan(v):
```
But this gives me an error saying that I cannot use this check for v. I guess it is because an integer or float variable, not a string is meant to be used. If this is true, how can I check v for an "empty cell"/nan case?

Response:
NaN is used as a placeholder for missing data consistently in pandas, consistency is good. I usually read/translate NaN as "missing". Also see the 'working with missing data' section in the docs. Wes writes in the docs 'choice of NA-representation': After years of production use [NaN] has proven, at least in my opinion, to be the best decision given the state of affairs in NumPy and Python in general. The special value NaN (Not-A-Number) is used everywhere as the NA value, and there are API functions isna and notna which can be used across the dtypes to detect NA values. ... Thus, I have chosen the Pythonic “practicality beats purity” approach and traded integer NA capability for a much simpler approach of using a special value in float and object arrays to denote NA, and promoting integer arrays to floating when NAs must be introduced. Note: the "gotcha" that integer Series containing missing data are upcast to floats. In my opinion the main reason to use NaN (over None) is that it can be stored with numpy's float64 dtype, rather than the less efficient object dtype, see NA type promotions.
```
#  without forcing dtype it changes None to NaN!
s_bad = pd.Series([1, None], dtype=object)
s_good = pd.Series([1, np.nan])

```python
s_bad.dtype
#Output
#dtype('O')
```

```python
s_good.dtype
#Output
#dtype('float64')
#```
#Jeff comments (below) on this: np.nan allows for vectorized operations; its a float value, while None, by definition, forces object type, which basically disables all efficiency in numpy. So repeat 3 times fast: object==bad, float==good Saying that, many operations may still work just as well with None vs NaN (but perhaps are not supported i.e. they may sometimes give surprising results):
#```
```

```python
s_bad.sum()
#Output
#1
```

```python
s_good.sum()
#Output
#1.0
#```
#To answer the second question: You should be using isna and notna to test for missing data (NaN).
```


Question: How do I compute derivative using Numpy? How do I calculate the derivative of a function, for example y = x2+1 using numpy? Let's say, I want the value of derivative at x = 5...

Response:
You have four options Finite Differences Automatic Derivatives Symbolic Differentiation Compute derivatives by hand. Finite differences require no external tools but are prone to numerical error and, if you're in a multivariate situation, can take a while. Symbolic differentiation is ideal if your problem is simple enough. Symbolic methods are getting quite robust these days. SymPy is an excellent project for this that integrates well with NumPy. Look at the autowrap or lambdify functions or check out Jensen's blogpost about a similar question. Automatic derivatives are very cool, aren't prone to numeric errors, but do require some additional libraries (google for this, there are a few good options). This is the most robust but also the most sophisticated/difficult to set up choice. If you're fine restricting yourself to numpy syntax then Theano might be a good choice. Here is an example using SymPy
```

```python
from sympy import *
```

```python
import numpy as np
```

```python
x = Symbol('x')
```

```python
y = x**2 + 1
```

```python
yprime = y.diff(x)
```

```python
yprime
#Output
#2⋅x
```

```python
f = lambdify(x, yprime, 'numpy')
```

```python
f(np.ones(5))
#Output
#[ 2.  2.  2.  2.  2.]
#```
```


Question:
Selecting specific rows and columns from NumPy array I've been going crazy trying to figure out what stupid thing I'm doing wrong here. I'm using NumPy, and I have specific row indices and specific column indices that I want to select from. Here's the gist of my problem:
```
import numpy as np

a = np.arange(20).reshape((5,4))
# array([[ 0,  1,  2,  3],
#        [ 4,  5,  6,  7],
#        [ 8,  9, 10, 11],
#        [12, 13, 14, 15],
#        [16, 17, 18, 19]])

# If I select certain rows, it works
print a[[0, 1, 3], :]
# array([[ 0,  1,  2,  3],
#        [ 4,  5,  6,  7],
#        [12, 13, 14, 15]])

# If I select certain rows and a single column, it works
print a[[0, 1, 3], 2]
# array([ 2,  6, 14])

# But if I select certain rows AND certain columns, it fails
print a[[0,1,3], [0,2]]
# Traceback (most recent call last):
#   File "<stdin>", line 1, in <module>
# ValueError: shape mismatch: objects cannot be broadcast to a single shape
```
Why is this happening? Surely I should be able to select the 1st, 2nd, and 4th rows, and 1st and 3rd columns? The result I'm expecting is:
```
a[[0,1,3], [0,2]] => [[0,  2],
                      [4,  6],
                      [12, 14]]
```

Response:
As Toan suggests, a simple hack would be to just select the rows first, and then select the columns over that.
```

```python
a[[0,1,3], :]            # Returns the rows you want
#Output
#array([[ 0,  1,  2,  3],
#       [ 4,  5,  6,  7],
#       [12, 13, 14, 15]])
```

```python
a[[0,1,3], :][:, [0,2]]  # Selects the columns you want as well
#Output
#array([[ 0,  2],
#       [ 4,  6],
#       [12, 14]])
#```
#[Edit] The built-in method: np.ix_ I recently discovered that numpy gives you an in-built one-liner to doing exactly what @Jaime suggested, but without having to use broadcasting syntax (which suffers from lack of readability). From the docs: Using ix_ one can quickly construct index arrays that will index the cross product. a[np.ix_([1,3],[2,5])] returns the array [[a[1,2] a[1,5]], [a[3,2] a[3,5]]]. So you use it like this:
#```
```

```python
a = np.arange(20).reshape((5,4))
```

```python
a[np.ix_([0,1,3], [0,2])]
#Output
#array([[ 0,  2],
#       [ 4,  6],
#       [12, 14]])
#```
#And the way it works is that it takes care of aligning arrays the way Jaime suggested, so that broadcasting happens properly:
#```
```

```python
np.ix_([0,1,3], [0,2])
#Output
#(array([[0],
#        [1],
#        [3]]), array([[0, 2]]))
#```
#Also, as MikeC says in a comment, np.ix_ has the advantage of returning a view, which my first (pre-edit) answer did not. This means you can now assign to the indexed array:
#```
```

```python
a[np.ix_([0,1,3], [0,2])] = -1
```

```python
a
#Output
#array([[-1,  1, -1,  3],
#       [-1,  5, -1,  7],
#       [ 8,  9, 10, 11],
#       [-1, 13, -1, 15],
#       [16, 17, 18, 19]])
#```
```


Question: Automatically import modules when entering the python or ipython interpreter I find myself typing import numpy as np almost every single time I fire up the python interpreter. How do I set up the python or ipython interpreter so that numpy is automatically imported?

Response: For ipython, there are two ways to achieve this. Both involve ipython's configuration directory which is located in ~/.ipython. Create a custom ipython profile. Or you can add a startup file to ~/.ipython/profile_default/startup/ For simplicity, I'd use option 2. All you have to do is place a .py or .ipy file in the ~/.ipython/profile_default/startup directory and it will automatically be executed. So you could simple place import numpy as np in a simple file and you'll have np in the namespace of your ipython prompt. Option 2 will actually work with a custom profile, but using a custom profile will allow you to change the startup requirements and other configuration based on a particular case. However, if you'd always like np to be available to you then by all means put it in the startup directory. For more information on ipython configuration. The docs have a much more complete explanation.


Question:
binning data in python with scipy/numpy is there a more efficient way to take an average of an array in prespecified bins? for example, i have an array of numbers and an array corresponding to bin start and end positions in that array, and I want to just take the mean in those bins? I have code that does it below but i am wondering how it can be cut down and improved. thanks.
```
from scipy import *
from numpy import *

def get_bin_mean(a, b_start, b_end):
    ind_upper = nonzero(a >= b_start)[0]
    a_upper = a[ind_upper]
    a_range = a_upper[nonzero(a_upper < b_end)[0]]
    mean_val = mean(a_range)
    return mean_val

data = rand(100)
bins = linspace(0, 1, 10)
binned_data = []

n = 0
for n in range(0, len(bins)-1):
    b_start = bins[n]
    b_end = bins[n+1]
    binned_data.append(get_bin_mean(data, b_start, b_end))

print binned_data
```

Response:
It's probably faster and easier to use numpy.digitize():
```
import numpy
data = numpy.random.random(100)
bins = numpy.linspace(0, 1, 10)
digitized = numpy.digitize(data, bins)
bin_means = [data[digitized == i].mean() for i in range(1, len(bins))]
```
An alternative to this is to use numpy.histogram():
```
bin_means = (numpy.histogram(data, bins, weights=data)[0] /
             numpy.histogram(data, bins)[0])
```
Try for yourself which one is faster... :)


Question:
Efficiently checking if arbitrary object is NaN in Python / numpy / pandas? My numpy arrays use np.nan to designate missing values. As I iterate over the data set, I need to detect such missing values and handle them in special ways. Naively I used numpy.isnan(val), which works well unless val isn't among the subset of types supported by numpy.isnan(). For example, missing data can occur in string fields, in which case I get:
```

```python
np.isnan('some_string')
#Output
#Traceback (most recent call last):
#  File "<stdin>", line 1, in <module>
#TypeError: Not implemented for this type
#```
#Other than writing an expensive wrapper that catches the exception and returns False, is there a way to handle this elegantly and efficiently?
```

Response:
pandas.isnull() (also pd.isna(), in newer versions) checks for missing values in both numeric and string/object arrays. From the documentation, it checks for: NaN in numeric arrays, None/NaN in object arrays Quick example:
```
import pandas as pd
import numpy as np
s = pd.Series(['apple', np.nan, 'banana'])
pd.isnull(s)
Out[9]:
0    False
1     True
2    False
dtype: bool
```
The idea of using numpy.nan to represent missing values is something that pandas introduced, which is why pandas has the tools to deal with it. Datetimes too (if you use pd.NaT you won't need to specify the dtype)
```

```python
s = Series([Timestamp('20130101'),np.nan,Timestamp('20130102 9:30')],dtype='M8[ns]')
```

```python
s
#Output
#0   2013-01-01 00:00:00
#1                   NaT
#2   2013-01-02 09:30:00
#dtype: datetime64[ns]``
```

```python
pd.isnull(s)
#Output
#0    False
#1     True
#2    False
#dtype: bool
#```
```


Question: Working with TIFFs (import, export) in Python using numpy I need a python method to open and import TIFF images into numpy arrays so I can analyze and modify the pixel data and then save them as TIFFs again. (They are basically light intensity maps in greyscale, representing the respective values per pixel) I couldn't find any documentation on PIL methods concerning TIFF. I tried to figure it out, but only got "bad mode" or "file type not supported" errors. What do I need to use here?

Response:
First, I downloaded a test TIFF image from this page called a_image.tif. Then I opened with PIL like this:
```

```python
from PIL import Image
```

```python
im = Image.open('a_image.tif')
```

```python
im.show()
#Output
#```
#This showed the rainbow image. To convert to a numpy array, it's as simple as:
#```
```

```python
import numpy
```

```python
imarray = numpy.array(im)
#Output
#```
#We can see that the size of the image and the shape of the array match up:
#```
```

```python
imarray.shape
#Output
#(44, 330)
```

```python
im.size
#Output
#(330, 44)
#```
#And the array contains uint8 values:
#```
```

```python
imarray
#Output
#array([[  0,   1,   2, ..., 244, 245, 246],
#       [  0,   1,   2, ..., 244, 245, 246],
#       [  0,   1,   2, ..., 244, 245, 246],
#       ...,
#       [  0,   1,   2, ..., 244, 245, 246],
#       [  0,   1,   2, ..., 244, 245, 246],
#       [  0,   1,   2, ..., 244, 245, 246]], dtype=uint8)
#```
#Once you're done modifying the array, you can turn it back into a PIL image like this:
#```
```

```python
Image.fromarray(imarray)
#Output
#<Image.Image image mode=L size=330x44 at 0x2786518>
#```
```


Question:
Normalize data in pandas Suppose I have a pandas data frame df: I want to calculate the column wise mean of a data frame. This is easy:
```
df.apply(average)
```
then the column wise range max(col) - min(col). This is easy again:
```
df.apply(max) - df.apply(min)
```
Now for each element I want to subtract its column's mean and divide by its column's range. I am not sure how to do that Any help/pointers are much appreciated.

Response:
```

```python
df
#Output
#a         b          c         d
#A  -0.488816  0.863769   4.325608 -4.721202
#B -11.937097  2.993993 -12.916784 -1.086236
#C  -5.569493  4.672679  -2.168464 -9.315900
#D   8.892368  0.932785   4.535396  0.598124
```

```python
df_norm = (df - df.mean()) / (df.max() - df.min())
```

```python
df_norm
#Output
#a         b         c         d
#A  0.085789 -0.394348  0.337016 -0.109935
#B -0.463830  0.164926 -0.650963  0.256714
#C -0.158129  0.605652 -0.035090 -0.573389
#D  0.536170 -0.376229  0.349037  0.426611
```

```python
df_norm.mean()
#Output
#a   -2.081668e-17
#b    4.857226e-17
#c    1.734723e-17
#d   -1.040834e-17
```

```python
df_norm.max() - df_norm.min()
#Output
#a    1
#b    1
#c    1
#d    1
#```
```


Question:
How do I calculate r-squared using Python and Numpy? I'm using Python and Numpy to calculate a best fit polynomial of arbitrary degree. I pass a list of x values, y values, and the degree of the polynomial I want to fit (linear, quadratic, etc.). This much works, but I also want to calculate r (coefficient of correlation) and r-squared(coefficient of determination). I am comparing my results with Excel's best-fit trendline capability, and the r-squared value it calculates. Using this, I know I am calculating r-squared correctly for linear best-fit (degree equals 1). However, my function does not work for polynomials with degree greater than 1. Excel is able to do this. How do I calculate r-squared for higher-order polynomials using Numpy? Here's my function:
```
import numpy

# Polynomial Regression
def polyfit(x, y, degree):
    results = {}

    coeffs = numpy.polyfit(x, y, degree)
     # Polynomial Coefficients
    results['polynomial'] = coeffs.tolist()

    correlation = numpy.corrcoef(x, y)[0,1]

     # r
    results['correlation'] = correlation
     # r-squared
    results['determination'] = correlation**2

    return results
```

Response:
A very late reply, but just in case someone needs a ready function for this: scipy.stats.linregress i.e.
```
slope, intercept, r_value, p_value, std_err = scipy.stats.linregress(x, y)
```
as in @Adam Marples's answer.


Question:
What does "unsqueeze" do in Pytorch? The PyTorch documentation says: Returns a new tensor with a dimension of size one inserted at the specified position. [...]
```

```python
x = torch.tensor([1, 2, 3, 4])
```

```python
torch.unsqueeze(x, 0)
#Output
#tensor([[ 1,  2,  3,  4]])
```

```python
torch.unsqueeze(x, 1)
#Output
#tensor([[ 1],
#        [ 2],
#        [ 3],
#        [ 4]])
#```
```

Response: unsqueeze turns an n.d. tensor into an (n+1).d. one by adding an extra dimension of depth 1. However, since it is ambiguous which axis the new dimension should lie across (i.e. in which direction it should be "unsqueezed"), this needs to be specified by the dim argument. e.g. unsqueeze can be applied to a 2d tensor three different ways: The resulting unsqueezed tensors have the same information, but the indices used to access them are different.


Question: How does numpy.histogram() work? While reading up on numpy, I encountered the function numpy.histogram(). What is it for and how does it work? In the docs they mention bins: What are they? Some googling led me to the definition of Histograms in general. I get that. But unfortunately I can't link this knowledge to the examples given in the docs.

Response:
A bin is range that represents the width of a single bar of the histogram along the X-axis. You could also call this the interval. (Wikipedia defines them more formally as "disjoint categories".) The Numpy histogram function doesn't draw the histogram, but it computes the occurrences of input data that fall within each bin, which in turns determines the area (not necessarily the height if the bins aren't of equal width) of each bar. In this example:
```
np.histogram([1, 2, 1], bins=[0, 1, 2, 3])
```
There are 3 bins, for values ranging from 0 to 1 (excl 1.), 1 to 2 (excl. 2) and 2 to 3 (incl. 3), respectively. The way Numpy defines these bins if by giving a list of delimiters ([0, 1, 2, 3]) in this example, although it also returns the bins in the results, since it can choose them automatically from the input, if none are specified. If bins=5, for example, it will use 5 bins of equal width spread between the minimum input value and the maximum input value. The input values are 1, 2 and 1. Therefore, bin "1 to 2" contains two occurrences (the two 1 values), and bin "2 to 3" contains one occurrence (the 2). These results are in the first item in the returned tuple: array([0, 2, 1]). Since the bins here are of equal width, you can use the number of occurrences for the height of each bar. When drawn, you would have: a bar of height 0 for range/bin [0,1] on the X-axis, a bar of height 2 for range/bin [1,2], a bar of height 1 for range/bin [2,3]. You can plot this directly with Matplotlib (its hist function also returns the bins and the values):
```

```python
import matplotlib.pyplot as plt
```

```python
plt.hist([1, 2, 1], bins=[0, 1, 2, 3])
#Output
#(array([0, 2, 1]), array([0, 1, 2, 3]), <a list of 3 Patch objects>)
```

```python
plt.show()
#Output
#```
```


Question:
Counting the number of non-NaN elements in a numpy ndarray in Python I need to calculate the number of non-NaN elements in a numpy ndarray matrix. How would one efficiently do this in Python? Here is my simple code for achieving this:
```
import numpy as np

def numberOfNonNans(data):
    count = 0
    for i in data:
        if not np.isnan(i):
            count += 1
    return count
```
Is there a built-in function for this in numpy? Efficiency is important because I'm doing Big Data analysis. Thnx for any help!

Response:
```
np.count_nonzero(~np.isnan(data))
```
~ inverts the boolean matrix returned from np.isnan. np.count_nonzero counts values that is not 0\false. .sum should give the same result. But maybe more clearly to use count_nonzero Testing speed:
```

```python
data = np.random.random((10000,10000))
```

```python
data[[np.random.random_integers(0,10000, 100)],:][:, [np.random.random_integers(0,99, 100)]] = np.nan
```

```python
%timeit data.size - np.count_nonzero(np.isnan(data))
1 loops, best of 3: 309 ms per loop
```

```python
%timeit np.count_nonzero(~np.isnan(data))
1 loops, best of 3: 345 ms per loop
```

```python
%timeit data.size - np.isnan(data).sum()
1 loops, best of 3: 339 ms per loop
```
data.size - np.count_nonzero(np.isnan(data)) seems to barely be the fastest here. other data might give different relative speed results.
```


Question:
LogisticRegression: Unknown label type: 'continuous' using sklearn in python I have the following code to test some of most popular ML algorithms of sklearn python library:
```
import numpy as np
from sklearn                        import metrics, svm
from sklearn.linear_model           import LinearRegression
from sklearn.linear_model           import LogisticRegression
from sklearn.tree                   import DecisionTreeClassifier
from sklearn.neighbors              import KNeighborsClassifier
from sklearn.discriminant_analysis  import LinearDiscriminantAnalysis
from sklearn.naive_bayes            import GaussianNB
from sklearn.svm                    import SVC

trainingData    = np.array([ [2.3, 4.3, 2.5],  [1.3, 5.2, 5.2],  [3.3, 2.9, 0.8],  [3.1, 4.3, 4.0]  ])
trainingScores  = np.array( [3.4, 7.5, 4.5, 1.6] )
predictionData  = np.array([ [2.5, 2.4, 2.7],  [2.7, 3.2, 1.2] ])

clf = LinearRegression()
clf.fit(trainingData, trainingScores)
print("LinearRegression")
print(clf.predict(predictionData))

clf = svm.SVR()
clf.fit(trainingData, trainingScores)
print("SVR")
print(clf.predict(predictionData))

clf = LogisticRegression()
clf.fit(trainingData, trainingScores)
print("LogisticRegression")
print(clf.predict(predictionData))

clf = DecisionTreeClassifier()
clf.fit(trainingData, trainingScores)
print("DecisionTreeClassifier")
print(clf.predict(predictionData))

clf = KNeighborsClassifier()
clf.fit(trainingData, trainingScores)
print("KNeighborsClassifier")
print(clf.predict(predictionData))

clf = LinearDiscriminantAnalysis()
clf.fit(trainingData, trainingScores)
print("LinearDiscriminantAnalysis")
print(clf.predict(predictionData))

clf = GaussianNB()
clf.fit(trainingData, trainingScores)
print("GaussianNB")
print(clf.predict(predictionData))

clf = SVC()
clf.fit(trainingData, trainingScores)
print("SVC")
print(clf.predict(predictionData))
```
The first two works ok, but I got the following error in LogisticRegression call:
```
root@ubupc1:/home/ouhma# python stack.py
LinearRegression
[ 15.72023529   6.46666667]
SVR
[ 3.95570063  4.23426243]
Traceback (most recent call last):
  File "stack.py", line 28, in <module>
    clf.fit(trainingData, trainingScores)
  File "/usr/local/lib/python2.7/dist-packages/sklearn/linear_model/logistic.py", line 1174, in fit
    check_classification_targets(y)
  File "/usr/local/lib/python2.7/dist-packages/sklearn/utils/multiclass.py", line 172, in check_classification_targets
    raise ValueError("Unknown label type: %r" % y_type)
ValueError: Unknown label type: 'continuous'
```
The input data is the same as in the previous calls, so what is going on here? And by the way, why there is a huge diference in the first prediction of LinearRegression() and SVR() algorithms (15.72 vs 3.95)?

Response:
You are passing floats to a classifier which expects categorical values as the target vector. If you convert it to int it will be accepted as input (although it will be questionable if that's the right way to do it). It would be better to convert your training scores by using scikit's labelEncoder function. The same is true for your DecisionTree and KNeighbors qualifier.
```
from sklearn import preprocessing
from sklearn import utils

lab_enc = preprocessing.LabelEncoder()
encoded = lab_enc.fit_transform(trainingScores)

```python
array([1, 3, 2, 0], dtype=int64)
```

print(utils.multiclass.type_of_target(trainingScores))

```python
continuous
```

print(utils.multiclass.type_of_target(trainingScores.astype('int')))

```python
multiclass
```

print(utils.multiclass.type_of_target(encoded))

```python
multiclass
#Output
#```
```


Question: How to save a list as numpy array in python? Is possible to construct a NumPy array from a python list?

Response:
First of all, I'd recommend you to go through NumPy's Quickstart tutorial, which will probably help with these basic questions. You can directly create an array from a list as:
```
import numpy as np
a = np.array( [2,3,4] )
```
Or from a from a nested list in the same way:
```
import numpy as np
a = np.array( [[2,3,4], [3,4,5]] )
```


Question:
sort eigenvalues and associated eigenvectors after using numpy.linalg.eig in python I'm using numpy.linalg.eig to obtain a list of eigenvalues and eigenvectors:
```
A = someMatrixArray
from numpy.linalg import eig as eigenValuesAndVectors

solution = eigenValuesAndVectors(A)

eigenValues = solution[0]
eigenVectors = solution[1]
```
I would like to sort my eigenvalues (e.g. from lowest to highest), in a way I know what is the associated eigenvector after the sorting. I'm not finding any way of doing that with python functions. Is there any simple way or do I have to code my sort version?

Response:
Use numpy.argsort. It returns the indices one would use to sort the array.
```
import numpy as np
import numpy.linalg as linalg

A = np.random.random((3,3))
eigenValues, eigenVectors = linalg.eig(A)

idx = eigenValues.argsort()[::-1]
eigenValues = eigenValues[idx]
eigenVectors = eigenVectors[:,idx]
```
If the eigenvalues are complex, the sort order is lexicographic (that is, complex numbers are sorted according to their real part first, with ties broken by their imaginary part).


Question: Numpy array assignment with copy For example, if we have a numpy array A, and we want a numpy array B with the same elements. What is the difference between the following (see below) methods? When is additional memory allocated, and when is it not? B = A B[:] = A (same as B[:]=A[:]?) numpy.copy(B, A)

Response: All three versions do different things: B = A This binds a new name B to the existing object already named A. Afterwards they refer to the same object, so if you modify one in place, you'll see the change through the other one too. B[:] = A (same as B[:]=A[:]?) This copies the values from A into an existing array B. The two arrays must have the same shape for this to work. B[:] = A[:] does the same thing (but B = A[:] would do something more like 1). numpy.copy(B, A) This is not legal syntax. You probably meant B = numpy.copy(A). This is almost the same as 2, but it creates a new array, rather than reusing the B array. If there were no other references to the previous B value, the end result would be the same as 2, but it will use more memory temporarily during the copy. Or maybe you meant numpy.copyto(B, A), which is legal, and is equivalent to 2?


Question:
Numpy isnan() fails on an array of floats (from pandas dataframe apply) I have an array of floats (some normal numbers, some nans) that is coming out of an apply on a pandas dataframe. For some reason, numpy.isnan is failing on this array, however as shown below, each element is a float, numpy.isnan runs correctly on each element, the type of the variable is definitely a numpy array. What's going on?!
```
set([type(x) for x in tester])
Out[59]: {float}

tester
Out[60]:
array([-0.7000000000000001, nan, nan, nan, nan, nan, nan, nan, nan, nan,
   nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
   nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
   nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
   nan, nan], dtype=object)

set([type(x) for x in tester])
Out[61]: {float}

np.isnan(tester)
Traceback (most recent call last):

File "<ipython-input-62-e3638605b43c>", line 1, in <module>
np.isnan(tester)

TypeError: ufunc 'isnan' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe''

set([np.isnan(x) for x in tester])
Out[65]: {False, True}

type(tester)
Out[66]: numpy.ndarray
```

Response:
np.isnan can be applied to NumPy arrays of native dtype (such as np.float64):
```

```python
np.isnan(np.array([np.nan, 0], dtype=np.float64))
#Output
#array([ True, False], dtype=bool)
#```
#but raises TypeError when applied to object arrays:
#```
```

```python
np.isnan(np.array([np.nan, 0], dtype=object))
TypeError: ufunc 'isnan' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe''
```
Since you have Pandas, you could use pd.isnull instead -- it can accept NumPy arrays of object or native dtypes:
```
```

```python
pd.isnull(np.array([np.nan, 0], dtype=float))
#Output
#array([ True, False], dtype=bool)
```

```python
pd.isnull(np.array([np.nan, 0], dtype=object))
#Output
#array([ True, False], dtype=bool)
#```
#Note that None is also considered a null value in object arrays.
```


Question:
What is the difference between i = i + 1 and i += 1 in a 'for' loop? [duplicate] This question already has answers here: When is "i += x" different from "i = i + x" in Python? (3 answers) Closed 8 years ago. I found out a curious thing today and was wondering if somebody could shed some light into what the difference is here?
```
import numpy as np

A = np.arange(12).reshape(4,3)
for a in A:
    a = a + 1

B = np.arange(12).reshape(4,3)
for b in B:
    b += 1
```
After running each for loop, A has not changed, but B has had one added to each element. I actually use the B version to write to a initialized NumPy array within a for loop.

Response: The difference is that one modifies the data-structure itself (in-place operation) b += 1 while the other just reassigns the variable a = a + 1. Just for completeness: x += y is not always doing an in-place operation, there are (at least) three exceptions: If x doesn't implement an __iadd__ method then the x += y statement is just a shorthand for x = x + y. This would be the case if x was something like an int. If __iadd__ returns NotImplemented, Python falls back to x = x + y. The __iadd__ method could theoretically be implemented to not work in place. It'd be really weird to do that, though. As it happens your bs are numpy.ndarrays which implements __iadd__ and return itself so your second loop modifies the original array in-place. You can read more on this in the Python documentation of "Emulating Numeric Types". These [__i*__] methods are called to implement the augmented arithmetic assignments (+=, -=, *=, @=, /=, //=, %=, **=, >=, &=, ^=, |=). These methods should attempt to do the operation in-place (modifying self) and return the result (which could be, but does not have to be, self). If a specific method is not defined, the augmented assignment falls back to the normal methods. For instance, if x is an instance of a class with an __iadd__() method, x += y is equivalent to x = x.__iadd__(y) . Otherwise, x.__add__(y) and y.__radd__(x) are considered, as with the evaluation of x + y. In certain situations, augmented assignment can result in unexpected errors (see Why does a_tuple[i] += ["item"] raise an exception when the addition works?), but this behavior is in fact part of the data model.


Question:
Numpy: find index of the elements within range I have a numpy array of numbers, for example,
```
a = np.array([1, 3, 5, 6, 9, 10, 14, 15, 56])
```
I would like to find all the indexes of the elements within a specific range. For instance, if the range is (6, 10), the answer should be (3, 4, 5). Is there a built-in function to do this?

Response:
You can use np.where to get indices and np.logical_and to set two conditions:
```
import numpy as np
a = np.array([1, 3, 5, 6, 9, 10, 14, 15, 56])

np.where(np.logical_and(a>=6, a<=10))
# returns (array([3, 4, 5]),)
```


Question:
NumPy: Logarithm with base n From the numpy documentation on logarithms, I have found functions to take the logarithm with base e, 2, and 10:
```py
import numpy as np
np.log(np.e**3) #3.0
np.log2(2**3)   #3.0
np.log10(10**3) #3.0
```
However, how do I take the logarithm with base n (e.g. 42) in numpy?

Response:
If you have numpy 1.23 or later, you can use np.emath.logn:
```
import numpy as np
array = np.array([74088, 3111696])  # = [42^3, 42^4]
base = 42
exponent = np.emath.logn(base, array)  # = [3, 4]
```
If your version of numpy is older: To get the logarithm with a custom base using math.log:
```
import math
number = 74088  # = 42^3
base = 42
exponent = math.log(number, base)  # = 3
```
To get the logarithm with a custom base using numpy.log:
```
import numpy as np
array = np.array([74088, 3111696])  # = [42^3, 42^4]
base = 42
exponent = np.log(array) / np.log(base)  # = [3, 4]
```
Which uses the logarithm base change rule:


Question:
Index of element in NumPy array [duplicate] This question already has answers here: Is there a NumPy function to return the first index of something in an array? (24 answers) Closed 4 years ago. In Python we can get the index of a value in an array by using .index(). But with a NumPy array, when I try to do:
```
decoding.index(i)
```
I get: AttributeError: 'numpy.ndarray' object has no attribute 'index' How could I do this on a NumPy array?

Response:
Use np.where to get the indices where a given condition is True. Examples: For a 2D np.ndarray called a:
```
i, j = np.where(a == value) # when comparing arrays of integers

i, j = np.where(np.isclose(a, value)) # when comparing floating-point arrays
```
For a 1D array:
```
i, = np.where(a == value) # integers

i, = np.where(np.isclose(a, value)) # floating-point
```
Note that this also works for conditions like >=, <=, != and so forth... You can also create a subclass of np.ndarray with an index() method:
```
class myarray(np.ndarray):
    def __new__(cls, *args, **kwargs):
        return np.array(*args, **kwargs).view(myarray)
    def index(self, value):
        return np.where(self == value)
```
Testing:
```
a = myarray([1,2,3,4,4,4,5,6,4,4,4])
a.index(4)
#(array([ 3,  4,  5,  8,  9, 10]),)
```


Question:
Find the index of the k smallest values of a numpy array In order to find the index of the smallest value, we can use argmin:
```
import numpy as np
A = np.array([1, 7, 9, 2, 0.1, 17, 17, 1.5])
print(A.argmin())     # 4 because A[4] = 0.1
```
But how can I find the indices of the k-smallest values? I'm looking for something like:
```
A.argmin(numberofvalues=3)
# [4, 0, 7]  because A[4] <= A[0] <= A[7] <= all other A[i]
```
Note: in my use case A has between ~ 10 000 and 100 000 values, and I'm interested for only the indices of the k=10 smallest values. k will never be > 10.

Response:
Use np.argpartition. It does not sort the entire array. It only guarantees that the kth element is in sorted position and all smaller elements will be moved before it. Thus the first k elements will be the k-smallest elements.
```
import numpy as np

A = np.array([1, 7, 9, 2, 0.1, 17, 17, 1.5])
k = 3

idx = np.argpartition(A, k)
print(idx)
# [4 0 7 3 1 2 6 5]
```
This returns the k-smallest values. Note that these may not be in sorted order.
```
print(A[idx[:k]])
# [ 0.1  1.   1.5]
```
To obtain the k-largest values use
```
idx = np.argpartition(A, -k)
# [4 0 7 3 1 2 6 5]

A[idx[-k:]]
# [  9.  17.  17.]
```
WARNING: Do not (re)use idx = np.argpartition(A, k); A[idx[-k:]] to obtain the k-largest. That won't always work. For example, these are NOT the 3 largest values in x:
```
x = np.array([100, 90, 80, 70, 60, 50, 40, 30, 20, 10, 0])
idx = np.argpartition(x, 3)
x[idx[-3:]]
array([ 70,  80, 100])
```
Here is a comparison against np.argsort, which also works but just sorts the entire array to get the result.
```

```python
x = np.random.randn(100000)
```

```python
%timeit idx0 = np.argsort(x)[:100]
100 loops, best of 3: 8.26 ms per loop
```

```python
%timeit idx1 = np.argpartition(x, 100)[:100]
1000 loops, best of 3: 721 µs per loop
```

```python
np.alltrue(np.sort(np.argsort(x)[:100]) == np.sort(np.argpartition(x, 100)[:100]))
#Output
#True
#```
```


Question: how is axis indexed in numpy's array? [duplicate] This question already has answers here: In Python NumPy what is a dimension and axis? (7 answers) Closed 1 year ago. From Numpy's tutorial, axis can be indexed with integers, like 0 is for column, 1 is for row, but I don't grasp why they are indexed this way? And How do I figure out each axis' index when coping with multidimensional array?

Response: By definition, the axis number of the dimension is the index of that dimension within the array's shape. It is also the position used to access that dimension during indexing. For example, if a 2D array a has shape (5,6), then you can access a[0,0] up to a[4,5]. Axis 0 is thus the first dimension (the "rows"), and axis 1 is the second dimension (the "columns"). In higher dimensions, where "row" and "column" stop really making sense, try to think of the axes in terms of the shapes and indices involved. If you do .sum(axis=n), for example, then dimension n is collapsed and deleted, with each value in the new matrix equal to the sum of the corresponding collapsed values. For example, if b has shape (5,6,7,8), and you do c = b.sum(axis=2), then axis 2 (dimension with size 7) is collapsed, and the result has shape (5,6,8). Furthermore, c[x,y,z] is equal to the sum of all elements b[x,y,:,z].


Question:
Histogram Matplotlib So I have a little problem. I have a data set in scipy that is already in the histogram format, so I have the center of the bins and the number of events per bin. How can I now plot is as a histogram. I tried just doing
```
bins, n=hist()
```
but it didn't like that. Any recommendations?

Response:
```
import matplotlib.pyplot as plt
import numpy as np

mu, sigma = 100, 15
x = mu + sigma * np.random.randn(10000)
hist, bins = np.histogram(x, bins=50)
width = 0.7 * (bins[1] - bins[0])
center = (bins[:-1] + bins[1:]) / 2
plt.bar(center, hist, align='center', width=width)
plt.show()
```
The object-oriented interface is also straightforward:
```
fig, ax = plt.subplots()
ax.bar(center, hist, align='center', width=width)
fig.savefig("1.png")
```
If you are using custom (non-constant) bins, you can pass compute the widths using np.diff, pass the widths to ax.bar and use ax.set_xticks to label the bin edges:
```
import matplotlib.pyplot as plt
import numpy as np

mu, sigma = 100, 15
x = mu + sigma * np.random.randn(10000)
bins = [0, 40, 60, 75, 90, 110, 125, 140, 160, 200]
hist, bins = np.histogram(x, bins=bins)
width = np.diff(bins)
center = (bins[:-1] + bins[1:]) / 2

fig, ax = plt.subplots(figsize=(8,3))
ax.bar(center, hist, align='center', width=width)
ax.set_xticks(bins)
fig.savefig("/tmp/out.png")

plt.show()
```


Question:
Linear regression with matplotlib / numpy I'm trying to generate a linear regression on a scatter plot I have generated, however my data is in list format, and all of the examples I can find of using polyfit require using arange. arange doesn't accept lists though. I have searched high and low about how to convert a list to an array and nothing seems clear. Am I missing something? Following on, how best can I use my list of integers as inputs to the polyfit? Here is the polyfit example I am following:
```
import numpy as np
import matplotlib.pyplot as plt

x = np.arange(data)
y = np.arange(data)

m, b = np.polyfit(x, y, 1)

plt.plot(x, y, 'yo', x, m*x+b, '--k')
plt.show()
```

Response:
arange generates lists (well, numpy arrays); type help(np.arange) for the details. You don't need to call it on existing lists.
```py

```python
x = [1,2,3,4]
```

```python
y = [3,5,7,9]
```

```python

```

```python
m,b = np.polyfit(x, y, 1)
```

```python
m
#Output
#2.0000000000000009
```

```python
b
#Output
#0.99999999999999833
#```
#I should add that I tend to use poly1d here rather than write out "m*x+b" and the higher-order equivalents, so my version of your code would look something like this:
#```py
#import numpy as np
#import matplotlib.pyplot as plt
```

x = [1,2,3,4]
y = [3,5,7,10] # 10, not 9, so the fit isn't perfect

coef = np.polyfit(x,y,1)
poly1d_fn = np.poly1d(coef)
# poly1d_fn is now a function which takes in x and returns an estimate for y

plt.plot(x,y, 'yo', x, poly1d_fn(x), '--k') #'--k'=black dashed line, 'yo' = yellow circle marker

plt.xlim(0, 5)
plt.ylim(0, 12)
```


Question:
How does NumPy's transpose() method permute the axes of an array?
```

```python
arr = np.arange(16).reshape((2, 2, 4))
```

```python
arr
#Output
#array([[[ 0,  1,  2,  3],
#        [ 4,  5,  6,  7]],
#
#       [[ 8,  9, 10, 11],
#        [12, 13, 14, 15]]])
```

```python
arr.transpose((1, 0, 2))
#Output
#array([[[ 0,  1,  2,  3],
#        [ 8,  9, 10, 11]],
#
#       [[ 4,  5,  6,  7],
#        [12, 13, 14, 15]]])
#```
#When we pass a tuple of integers to the transpose() function, what happens? To be specific, this is a 3D array: how does NumPy transform the array when I pass the tuple of axes (1, 0 ,2)? Can you explain which row or column these integers refer to? And what are axis numbers in the context of NumPy?
```

Response:
To transpose an array, NumPy just swaps the shape and stride information for each axis. Here are the strides:
```

```python
arr.strides
#Output
#(64, 32, 8)
```

```python
arr.transpose(1, 0, 2).strides
#Output
#(32, 64, 8)
#```
#Notice that the transpose operation swapped the strides for axis 0 and axis 1. The lengths of these axes were also swapped (both lengths are 2 in this example). No data needs to be copied for this to happen; NumPy can simply change how it looks at the underlying memory to construct the new array. Visualising strides The stride value represents the number of bytes that must be travelled in memory in order to reach the next value of an axis of an array. Now, our 3D array arr looks this (with labelled axes): This array is stored in a contiguous block of memory; essentially it is one-dimensional. To interpret it as a 3D object, NumPy must jump over a certain constant number of bytes in order to move along one of the three axes: Since each integer takes up 8 bytes of memory (we're using the int64 dtype), the stride value for each dimension is 8 times the number of values that we need to jump. For instance, to move along axis 1, four values (32 bytes) are jumped, and to move along axis 0, eight values (64 bytes) need to be jumped. When we write arr.transpose(1, 0, 2) we are swapping axes 0 and 1. The transposed array looks like this: All that NumPy needs to do is to swap the stride information for axis 0 and axis 1 (axis 2 is unchanged). Now we must jump further to move along axis 1 than axis 0: This basic concept works for any permutation of an array's axes. The actual code that handles the transpose is written in C and can be found here.
```


Question:
Maximum allowed value for a numpy data type I am working with numpy arrays of a range of data types (uint8, uint16, int16, etc.). I would like to be able to check whether a number can be represented within the limits of an array for a given datatype. I am imagining something that looks like:
```

```python
im.dtype
#Output
#dtype('uint16')
```

```python
dtype_max(im.dtype)
#Output
#65535
```

```python
dtype_min(im.dtype)
#Output
#0
#```
#Does something like this exist? By the way, I feel like this has to have been asked before, but my search came up empty, and all of the "similar questions" appear to be unrelated. Edit: Of course, now that I've asked, one of the "related" questions does have the answer. Oops.
```

Response:
```py
min_value = np.iinfo(im.dtype).min
max_value = np.iinfo(im.dtype).max
```
docs: np.iinfo (machine limits for integer types) np.finfo (machine limits for floating point types)


Question:
Convert 2d numpy array into list of lists [duplicate] This question already has answers here: Convert NumPy array to Python list (7 answers) Closed 10 years ago. I use an external module (libsvm), which does not support numpy arrays, only tuples, lists and dicts. But my data is in a 2d numpy array. How can I convert it the pythonic way, aka without loops.
```

```python
import numpy
```

```python
array = numpy.ones((2,4))
```

```python
data_list = list(array)
```

```python
data_list
#Output
#[array([ 1.,  1.,  1.,  1.]), array([ 1.,  1.,  1.,  1.])]
```

```python
type(data_list[0])
#Output
#<type 'numpy.ndarray'>  # <= what I don't want
```

# non pythonic way using for loop

```python
newdata=list()
```

```python
for line in data_list:
    line = list(line)
    newdata.append(line)
```

```python
type(newdata[0])
#Output
#<type 'list'>  # <= what I want
#```
```

Response:
You can simply cast the matrix to list with matrix.tolist(), proof:
```

```python
import numpy
```

```python
a = numpy.ones((2,4))
```

```python
a
#Output
#array([[ 1.,  1.,  1.,  1.],
#       [ 1.,  1.,  1.,  1.]])
```

```python
a.tolist()
#Output
#[[1.0, 1.0, 1.0, 1.0], [1.0, 1.0, 1.0, 1.0]]
```

```python
type(a.tolist())
#Output
#<type 'list'>
```

```python
type(a.tolist()[0])
#Output
#<type 'list'>
#```
```


Question:
Pass percentiles to pandas agg function I want to pass the numpy percentile() function through pandas' agg() function as I do below with various other numpy statistics functions. Right now I have a dataframe that looks like this:
```
AGGREGATE   MY_COLUMN
A           10
A           12
B           5
B           9
A           84
B           22
```
And my code looks like this:
```
grouped = dataframe.groupby('AGGREGATE')
column = grouped['MY_COLUMN']
column.agg([np.sum, np.mean, np.std, np.median, np.var, np.min, np.max])
```
The above code works, but I want to do something like
```
column.agg([np.sum, np.mean, np.percentile(50), np.percentile(95)])
```
I.e., specify various percentiles to return from agg(). How should this be done?

Response:
Perhaps not super efficient, but one way would be to create a function yourself:
```
def percentile(n):
    def percentile_(x):
        return x.quantile(n)
    percentile_.__name__ = 'percentile_{:02.0f}'.format(n*100)
    return percentile_
```
Then include this in your agg:
```

```python
column.agg([np.sum, np.mean, np.std, np.median,
                     np.var, np.min, np.max, percentile(50), percentile(95)])
#Output
#sum       mean        std  median          var  amin  amax  percentile_50  percentile_95
#AGGREGATE
#A          106  35.333333  42.158431      12  1777.333333    10    84             12           76.8
#B           36  12.000000   8.888194       9    79.000000     5    22             12           76.8
#```
#Note sure this is how it should be done though...
```


Question:
Add numpy array as column to Pandas data frame I have a Pandas data frame object of shape (X,Y) that looks like this:
```
[[1, 2, 3],
[4, 5, 6],
[7, 8, 9]]
```
and a numpy sparse matrix (CSC) of shape (X,Z) that looks something like this
```
[[0, 1, 0],
[0, 0, 1],
[1, 0, 0]]
```
How can I add the content from the matrix to the data frame in a new named column such that the data frame will end up like this:
```
[[1, 2, 3, [0, 1, 0]],
[4, 5, 6, [0, 0, 1]],
[7, 8, 9, [1, 0, 0]]]
```
Notice the data frame now has shape (X, Y+1) and rows from the matrix are elements in the data frame.

Response:
```
import numpy as np
import pandas as pd
import scipy.sparse as sparse

df = pd.DataFrame(np.arange(1,10).reshape(3,3))
arr = sparse.coo_matrix(([1,1,1], ([0,1,2], [1,2,0])), shape=(3,3))
df['newcol'] = arr.toarray().tolist()
print(df)
```
yields
```
0  1  2     newcol
0  1  2  3  [0, 1, 0]
1  4  5  6  [0, 0, 1]
2  7  8  9  [1, 0, 0]
```


Question:
ValueError: could not broadcast input array from shape (224,224,3) into shape (224,224) I have a list say, temp_list with following properties :
```
len(temp_list) = 9260
temp_list[0].shape = (224,224,3)
```
Now, when I am converting into numpy array,
```
x = np.array(temp_list)
```
I am getting the error :
```
ValueError: could not broadcast input array from shape (224,224,3) into shape (224,224)
```
Can someone help me here?

Response:
At least one item in your list is either not three dimensional, or its second or third dimension does not match the other elements. If only the first dimension does not match, the arrays are still matched, but as individual objects, no attempt is made to reconcile them into a new (four dimensional) array. Some examples are below: That is, the offending element's shape != (?, 224, 3), or ndim != 3 (with the ? being non-negative integer). That is what is giving you the error. You'll need to fix that, to be able to turn your list into a four (or three) dimensional array. Without context, it is impossible to say if you want to lose a dimension from the 3D items or add one to the 2D items (in the first case), or change the second or third dimension (in the second case). Here's an example of the error:
```

```python
a = [np.zeros((224,224,3)), np.zeros((224,224,3)), np.zeros((224,224))]
```

```python
np.array(a)
#Output
#ValueError: could not broadcast input array from shape (224,224,3) into shape (224,224)
#```
#or, different type of input, but the same error:
#```
```

```python
a = [np.zeros((224,224,3)), np.zeros((224,224,3)), np.zeros((224,224,13))]
```

```python
np.array(a)
#Output
#Traceback (most recent call last):
#  File "<stdin>", line 1, in <module>
#ValueError: could not broadcast input array from shape (224,224,3) into shape (224,224)
#```
#Alternatively, similar but with a different error message:
#```
```

```python
a = [np.zeros((224,224,3)), np.zeros((224,224,3)), np.zeros((224,100,3))]
```

```python
np.array(a)
#Output
#Traceback (most recent call last):
#  File "<stdin>", line 1, in <module>
#ValueError: could not broadcast input array from shape (224,224,3) into shape (224)
#```
#But the following will work, albeit with different results than (presumably) intended:
#```
```

```python
a = [np.zeros((224,224,3)), np.zeros((224,224,3)), np.zeros((10,224,3))]
```

```python
np.array(a)
#Output
## long output omitted
```

```python
newa = np.array(a)
```

```python
newa.shape
#Output
#3  # oops
```

```python
newa.dtype
#Output
#dtype('O')
```

```python
newa[0].shape
#Output
#(224, 224, 3)
```

```python
newa[1].shape
#Output
#(224, 224, 3)
```

```python
newa[2].shape
#Output
#(10, 224, 3)
#>>>
#```
```


Question:
Difference between np.random.seed() and np.random.RandomState() I know that to seed the randomness of numpy.random, and be able to reproduce it, I should us:
```
import numpy as np
np.random.seed(1234)
```
but what does np.random.RandomState() do?

Response:
If you want to set the seed that calls to np.random... will use, use np.random.seed:
```
np.random.seed(1234)
np.random.uniform(0, 10, 5)
#array([ 1.9151945 ,  6.22108771,  4.37727739,  7.85358584,  7.79975808])
np.random.rand(2,3)
#array([[ 0.27259261,  0.27646426,  0.80187218],
#       [ 0.95813935,  0.87593263,  0.35781727]])
```
Use the class to avoid impacting the global numpy state:
```
r = np.random.RandomState(1234)
r.uniform(0, 10, 5)
#array([ 1.9151945 ,  6.22108771,  4.37727739,  7.85358584,  7.79975808])
```
And it maintains the state just as before:
```
r.rand(2,3)
#array([[ 0.27259261,  0.27646426,  0.80187218],
#       [ 0.95813935,  0.87593263,  0.35781727]])
```
You can see the state of the sort of 'global' class with:
```
np.random.get_state()
```
and of your own class instance with:
```
r.get_state()
```


Question:
Extract upper or lower triangular part of a numpy matrix I have a matrix A and I want 2 matrices U and L such that U contains the upper triangular elements of A (all elements above and not including diagonal) and similarly for L(all elements below and not including diagonal). Is there a numpy method to do this? e.g
```
A = array([[ 4.,  9., -3.],
           [ 2.,  4., -2.],
           [-2., -3.,  7.]])

U = array([[ 0.,  9., -3.],
           [ 0.,  0., -2.],
           [ 0.,  0.,  0.]])

L = array([[ 0.,  0.,  0.],
           [ 2.,  0.,  0.],
           [-2., -3.,  0.]])
```

Response:
To extract the upper triangle values to a flat vector, you can do something like the following:
```
import numpy as np

a = np.array([[1,2,3],[4,5,6],[7,8,9]])
print(a)

#array([[1, 2, 3],
#       [4, 5, 6],
#       [7, 8, 9]])

a[np.triu_indices(3)]
#or
list(a[np.triu_indices(3)])

#array([1, 2, 3, 5, 6, 9])
```
Similarly, for the lower triangle, use np.tril. IMPORTANT If you want to extract the values that are above the diagonal (or below) then use the k argument. This is usually used when the matrix is symmetric.
```
import numpy as np

a = np.array([[1,2,3],[4,5,6],[7,8,9]])

#array([[1, 2, 3],
#       [4, 5, 6],
#       [7, 8, 9]])

a[np.triu_indices(3, k = 1)]

# this returns the following
array([2, 3, 6])
```
EDIT (on 11.11.2019): To put back the extracted vector into a 2D symmetric array, one can follow my answer here: https://stackoverflow.com/a/58806626/5025009


Question: What is the equivalent of MATLAB's repmat in NumPy I would like to execute the equivalent of the following MATLAB code using NumPy: repmat([1; 1], [1 1 1]). How would I accomplish this?

Response:
Here is a much better (official) NumPy for Matlab Users link - I'm afraid the mathesaurus one is quite out of date. The numpy equivalent of repmat(a, m, n) is tile(a, (m, n)). This works with multiple dimensions and gives a similar result to matlab. (Numpy gives a 3d output array as you would expect - matlab for some reason gives 2d output - but the content is the same). Matlab:
```
>> repmat([1;1],[1,1,1])

ans =
     1
     1
```
Python:
```

```python
a = np.array([[1],[1]])
```

```python
np.tile(a, [1,1,1])
#Output
#array([[[1],
#        [1]]])
#```
```


Question: Mean Squared Error in Numpy? Is there a method in numpy for calculating the Mean Squared Error between two matrices? I've tried searching but found none. Is it under a different name? If there isn't, how do you overcome this? Do you write it yourself or use a different lib?

Response:
You can use:
```
mse = ((A - B)**2).mean(axis=ax)
```
Or
```
mse = (np.square(A - B)).mean(axis=ax)
```
with ax=0 the average is performed along the row, for each column, returning an array with ax=1 the average is performed along the column, for each row, returning an array with omitting the ax parameter (or setting it to ax=None) the average is performed element-wise along the array, returning a scalar value


Question:
inverting image in Python with OpenCV I want to load a color image, convert it to grayscale, and then invert the data in the file. What I need: to iterate over the array in OpenCV and change every single value with this formula (it might be wrong but it seems reasonable for me):
```
img[x,y] = abs(img[x,y] - 255)
```
but I don't understand why doesn't it works:
```
def inverte(imagem, name):
    imagem = abs(imagem - 255)
    cv2.imwrite(name, imagem)

def inverte2(imagem, name):
    for x in np.nditer(imagem, op_flags=['readwrite']):
        x = abs(x - 255)
    cv2.imwrite(name, imagem)

if __name__ == '__main__':
    nome = str(sys.argv[1])
    image = cv2.imread(nome)
    gs_imagem = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
    inverte(gs_imagem, "invertida.png")
    inverte2(gs_imagem, "invertida2.png")
```
I don't want to do an explicit loop (I am trying to be more pythonic). I can see that in one image that got a white background it turned black, but only this it doesn't looks like the other colors are having much (if any) change.

Response:
You almost did it. You were tricked by the fact that abs(imagem-255) will give a wrong result since your dtype is an unsigned integer. You have to do (255-imagem) in order to keep the integers unsigned:
```
def inverte(imagem, name):
    imagem = (255-imagem)
    cv2.imwrite(name, imagem)
```
As recommended in the answer by Eric Olmon, you can also invert the image using the bitwise_not function of OpenCV:
```
imagem = cv2.bitwise_not(imagem)
```


Question:
Consistently create same random numpy array I am waiting for another developer to finish a piece of code that will return an np array of shape (100,2000) with values of either -1,0, or 1. In the meantime, I want to randomly create an array of the same characteristics so I can get a head start on my development and testing. The thing is that I want this randomly created array to be the same each time, so that I'm not testing against an array that keeps changing its value each time I re-run my process. I can create my array like this, but is there a way to create it so that it's the same each time. I can pickle the object and unpickle it, but wondering if there's another way.
```
r = np.random.randint(3, size=(100, 2000)) - 1
```

Response:
Create your own instance of numpy.random.RandomState() with your chosen seed. Do not use numpy.random.seed() except to work around inflexible libraries that do not let you pass around your own RandomState instance.
```
[~]
|1> from numpy.random import RandomState

[~]
|2> prng = RandomState(1234567890)

[~]
|3> prng.randint(-1, 2, size=10)
array([ 1,  1, -1,  0,  0, -1,  1,  0, -1, -1])

[~]
|4> prng2 = RandomState(1234567890)

[~]
|5> prng2.randint(-1, 2, size=10)
array([ 1,  1, -1,  0,  0, -1,  1,  0, -1, -1])
```


Question:
How to delete columns in numpy.array I would like to delete selected columns in a numpy.array . This is what I do:
```
n [397]: a = array([[ NaN,   2.,   3., NaN],
   .....:        [  1.,   2.,   3., 9]])

```python
print a
[[ NaN   2.   3.  NaN]
 [  1.   2.   3.   9.]]
```

```python
z = any(isnan(a), axis=0)
```

```python
print z
[ True False False  True]
```

```python
delete(a, z, axis = 1)
#Output
#array([[  3.,  NaN],
#       [  3.,   9.]])
#```
#In this example my goal is to delete all the columns that contain NaN's. I expect the last command to result in:
#```
#array([[2., 3.],
#       [2., 3.]])
#```
#How can I do that?
```

Response:
Given its name, I think the standard way should be delete:
```
import numpy as np

A = np.delete(A, 1, 0)  # delete second row of A
B = np.delete(B, 2, 0)  # delete third row of B
C = np.delete(C, 1, 1)  # delete second column of C
```
According to numpy's documentation page, the parameters for numpy.delete are as follow: numpy.delete(arr, obj, axis=None) arr refers to the input array, obj refers to which sub-arrays (e.g. column/row no. or slice of the array) and axis refers to either column wise (axis = 1) or row-wise (axis = 0) delete operation.


Question:
Elegant way to create empty pandas DataFrame with NaN of type float I want to create a Pandas DataFrame filled with NaNs. During my research I found an answer:
```
import pandas as pd

df = pd.DataFrame(index=range(0,4),columns=['A'])
```
This code results in a DataFrame filled with NaNs of type "object". So they cannot be used later on for example with the interpolate() method. Therefore, I created the DataFrame with this complicated code (inspired by this answer):
```
import pandas as pd
import numpy as np

dummyarray = np.empty((4,1))
dummyarray[:] = np.nan

df = pd.DataFrame(dummyarray)
```
This results in a DataFrame filled with NaN of type "float", so it can be used later on with interpolate(). Is there a more elegant way to create the same result?

Response:
Simply pass the desired value as first argument, like 0, math.inf or, here, np.nan. The constructor then initializes and fills the value array to the size specified by arguments index and columns:
```

```python
import numpy as np
```

```python
import pandas as pd
```

```python
df = pd.DataFrame(np.nan, index=[0, 1, 2, 3], columns=['A', 'B'])
```

```python
df
#Output
#    A   B
#0 NaN NaN
#1 NaN NaN
#2 NaN NaN
#3 NaN NaN
```

```python
df.dtypes
#Output
#A    float64
#B    float64
#dtype: object
#```
```


Question:
mean, nanmean and warning: Mean of empty slice Say I construct three numpy arrays:
```
a = np.array([1, 2, 3])
b = np.array([np.NaN, np.NaN, 3])
c = np.array([np.NaN, np.NaN, np.NaN])
```
Now I find that np.mean returns nan for both b and c:
```

```python
np.mean(a)
#Output
#2.0
```

```python
np.mean(b)
#Output
#nan
```

```python
np.mean(c)
#Output
#nan
#```
#Since numpy 1.8 (released 20 April 2016), we've been blessed with nanmean, which ignores nan values:
#```
```

```python
np.nanmean(a)
#Output
#2.0
```

```python
np.nanmean(b)
#Output
#3.0
```

```python
np.nanmean(c)
#Output
#nan
#C:\python-3.4.3\lib\site-packages\numpy\lib\nanfunctions.py:598: RuntimeWarning: Mean of empty slice
#  warnings.warn("Mean of empty slice", RuntimeWarning)
#```
#So, nanmean is great, but it has the odd and undesirable behaviour of raising a warning when the array has nothing but nan values. How can I get the behaviour of nanmean without that warning? I don't like warnings, and I don't like suppressing them manually.
```

Response:
I really can't see any good reason not to just suppress the warning. The safest way would be to use the warnings.catch_warnings context manager to suppress the warning only where you anticipate it occurring - that way you won't miss any additional RuntimeWarnings that might be unexpectedly raised in some other part of your code:
```
import numpy as np
import warnings

x = np.ones((1000, 1000)) * np.nan

# I expect to see RuntimeWarnings in this block
with warnings.catch_warnings():
    warnings.simplefilter("ignore", category=RuntimeWarning)
    foo = np.nanmean(x, axis=1)
```
@dawg's solution would also work, but ultimately any additional steps that you have to take in order to avoid computing np.nanmean on an array of all NaNs are going to incur some extra overhead that you could avoid by just suppressing the warning. Also your intent will be much more clearly reflected in the code.


Question:
Two-sample Kolmogorov-Smirnov Test in Python Scipy I can't figure out how to do a Two-sample KS test in Scipy. After reading the documentation of scipy kstest, I can see how to test whether a distribution is identical to standard normal distribution
```py
from scipy.stats import kstest
import numpy as np

x = np.random.normal(0,1,1000)
test_stat = kstest(x, 'norm')
#>>> test_stat
#(0.021080234718821145, 0.76584491300591395)
```
Which means that at p-value of 0.76 we cannot reject the null hypothesis that the two distributions are identical. However, I want to compare two distributions and see if I can reject the null hypothesis that they are identical, something like:
```py
from scipy.stats import kstest
import numpy as np

x = np.random.normal(0,1,1000)
z = np.random.normal(1.1,0.9, 1000)
```
and test whether x and z are identical. I tried the naive:
```py
test_stat = kstest(x, z)
```
and got the following error:
```none
TypeError: 'numpy.ndarray' object is not callable
```
Is there a way to do a two-sample KS test in Python? If so, how should I do it?

Response:
You are using the one-sample KS test. You probably want the two-sample test ks_2samp:
```

```python
from scipy.stats import ks_2samp
```

```python
import numpy as np
```

```python

```

```python
np.random.seed(12345678)
```

```python
x = np.random.normal(0, 1, 1000)
```

```python
y = np.random.normal(0, 1, 1000)
```

```python
z = np.random.normal(1.1, 0.9, 1000)
```

```python

```

```python
ks_2samp(x, y)
#Output
#Ks_2sampResult(statistic=0.022999999999999909, pvalue=0.95189016804849647)
```

```python
ks_2samp(x, z)
#Output
#Ks_2sampResult(statistic=0.41800000000000004, pvalue=3.7081494119242173e-77)
#```
#Results can be interpreted as following: You can either compare the statistic value given by python to the KS-test critical value table according to your sample size. When statistic value is higher than the critical value, the two distributions are different. Or you can compare the p-value to a level of significance a, usually a=0.05 or 0.01 (you decide, the lower a is, the more significant). If p-value is lower than a, then it is very probable that the two distributions are different.
```


Question:
numpy division with RuntimeWarning: invalid value encountered in double_scalars I wrote the following script:
```py
import numpy

d = numpy.array([[1089, 1093]])
e = numpy.array([[1000, 4443]])
answer = numpy.exp(-3 * d)
answer1 = numpy.exp(-3 * e)
res = answer.sum()/answer1.sum()
```
But I got this result and with the error occurred:
```none
nan
C:\Users\Desktop\test.py:16: RuntimeWarning: invalid value encountered in double_scalars
  res = answer.sum()/answer1.sum()
```
It seems to be that the input element were too small that python turned them to be zeros, but indeed the division has its result. How to solve this kind of problem?

Response:
You can't solve it. Simply answer1.sum()==0, and you can't perform a division by zero. This happens because answer1 is the exponential of 2 very large, negative numbers, so that the result is rounded to zero. nan is returned in this case because of the division by zero. Now to solve your problem you could: go for a library for high-precision mathematics, like mpmath. But that's less fun. as an alternative to a bigger weapon, do some math manipulation, as detailed below. go for a tailored scipy/numpy function that does exactly what you want! Check out @Warren Weckesser answer. Here I explain how to do some math manipulation that helps on this problem. We have that for the numerator:
```
exp(-x)+exp(-y) = exp(log(exp(-x)+exp(-y)))
                = exp(log(exp(-x)*[1+exp(-y+x)]))
                = exp(log(exp(-x) + log(1+exp(-y+x)))
                = exp(-x + log(1+exp(-y+x)))
```
where above x=3* 1089 and y=3* 1093. Now, the argument of this exponential is -x + log(1+exp(-y+x)) = -x + 6.1441934777474324e-06 For the denominator you could proceed similarly but obtain that log(1+exp(-z+k)) is already rounded to 0, so that the argument of the exponential function at the denominator is simply rounded to -z=-3000. You then have that your result is
```
exp(-x + log(1+exp(-y+x)))/exp(-z) = exp(-x+z+log(1+exp(-y+x))
                                   = exp(-266.99999385580668)
```
which is already extremely close to the result that you would get if you were to keep only the 2 leading terms (i.e. the first number 1089 in the numerator and the first number 1000 at the denominator):
```
exp(3*(1089-1000))=exp(-267)
```
For the sake of it, let's see how close we are from the solution of Wolfram alpha (link):
```
Log[(exp[-3*1089]+exp[-3*1093])/([exp[-3*1000]+exp[-3*4443])] -> -266.999993855806522267194565420933791813296828742310997510523
```
The difference between this number and the exponent above is +1.7053025658242404e-13, so the approximation we made at the denominator was fine. The final result is
```
'exp(-266.99999385580668) = 1.1050349147204485e-116
```
From wolfram alpha is (link)
```
1.105034914720621496.. × 10^-116 # Wolfram alpha.
```
and again, it is safe to use numpy here too.


Question:
pandas select from Dataframe using startswith This works (using Pandas 12 dev)
```
table2=table[table['SUBDIVISION'] =='INVERNESS']
```
Then I realized I needed to select the field using "starts with" Since I was missing a bunch. So per the Pandas doc as near as I could follow I tried
```
criteria = table['SUBDIVISION'].map(lambda x: x.startswith('INVERNESS'))
table2 = table[criteria]
```
And got AttributeError: 'float' object has no attribute 'startswith' So I tried an alternate syntax with the same result
```
table[[x.startswith('INVERNESS') for x in table['SUBDIVISION']]]
```
Reference http://pandas.pydata.org/pandas-docs/stable/indexing.html#boolean-indexing Section 4: List comprehensions and map method of Series can also be used to produce more complex criteria: What am I missing?

Response:
You can use the str.startswith DataFrame method to give more consistent results:
```

```python
s = pd.Series(['a', 'ab', 'c', 11, np.nan])
```

```python
s
#Output
#0      a
#1     ab
#2      c
#3     11
#4    NaN
#dtype: object
```

```python
s.str.startswith('a', na=False)
#Output
#0     True
#1     True
#2    False
#3    False
#4    False
#dtype: bool
#```
#and the boolean indexing will work just fine (I prefer to use loc, but it works just the same without):
#```
```

```python
s.loc[s.str.startswith('a', na=False)]
#Output
#0     a
#1    ab
#dtype: object
#```
#. It looks least one of your elements in the Series/column is a float, which doesn't have a startswith method hence the AttributeError, the list comprehension should raise the same error...
```


Question:
Should I use np.absolute or np.abs? Numpy provides both np.absolute and the alias np.abs defined via
```
from .numeric import absolute as abs
```
which seems to be in obvious violation of the zen of python: There should be one-- and preferably only one --obvious way to do it. So I'm guessing that there is a good reason for this. I have personally been using np.abs in almost all of my code and looking at e.g. the number of search results for np.abs vs np.absolute on Stack Overflow it seems like an overwhelming majority does the same (2130 vs 244 hits). Is there any reason i should preferentially use np.absolute over np.abs in my code, or should I simply go for the more "standard" np.abs?

Response: It's likely because there a built-in functions with the same name, abs. The same is true for np.amax, np.amin and np.round_. The aliases for the NumPy functions abs, min, max and round are only defined in the top-level package. So np.abs and np.absolute are completely identical. It doesn't matter which one you use. There are several advantages to the short names: They are shorter and they are known to Python programmers because the names are identical to the built-in Python functions. So end-users have it easier (less to type, less to remember). But there are reasons to have different names too: NumPy (or more generally 3rd party packages) sometimes need the Python functions abs, min, etc. So inside the package they define functions with a different name so you can still access the Python functions - and just in the top-level of the package you expose the "shortcuts". Note: Different names are not the only available option in that case: One could work around that with the Python module builtins to access the built-in functions if one shadowed a built-in name. It might also be the case (but that's pure speculation on my part) that they originally only included the long-named functions absolute (and so on) and only added the short aliases later. Being a large and well-used library the NumPy developers don't remove or deprecate stuff lightly. So they may just keep the long names around because it could break old code/scripts if they would remove them.


Question:
AttributeError: 'Tensor' object has no attribute 'numpy' I downloaded this code from GitHub.
```
predicted_id = tf.multinomial(tf.exp(predictions), num_samples=1)[0][0].numpy()
```
But I get an error that says:
```
AttributeError: 'Tensor' object has no attribute 'numpy'
```
What is wrong, and how do I fix it?

Response:
Since the accepted answer did not solve the problem for me so I thought it might be helpful for some people who face the problem and that already have tensorflow version >= 2.2.0 and eager execution enabled. The issue seems to be that for certain functions during the fitting model.fit() the @tf.function decorator prohibits the execution of functions like tensor.numpy() for performance reasons. The solution for me was to pass the flag run_eagerly=True to the model.compile() like this:
```
model.compile(..., run_eagerly=True)
```


Question:
data type not understood I'm trying to use a matrix to compute stuff. The code is this
```
import numpy as np
# some code
mmatrix = np.zeros(nrows, ncols)
print mmatrix[0, 0]
```
but I get 'data type not understood', and it works if I do it from terminal.

Response:
Try:
```
mmatrix = np.zeros((nrows, ncols))
```
Since the shape parameter has to be an int or sequence of ints http://docs.scipy.org/doc/numpy/reference/generated/numpy.zeros.html Otherwise you are passing ncols to np.zeros as the dtype.