Question:
surface plots in matplotlib I have a list of 3-tuples representing a set of points in 3D space. I want to plot a surface that covers all these points. The plot_surface function in the mplot3d package requires as arguments X,Y and Z to be 2d arrays. Is plot_surface the right function to plot surface and how do I transform my data into the required format?
```
data = [(x1,y1,z1),(x2,y2,z2),.....,(xn,yn,zn)]
```

Response:
For surfaces it's a bit different than a list of 3-tuples, you should pass in a grid for the domain in 2d arrays. If all you have is a list of 3d points, rather than some function f(x, y) -> z, then you will have a problem because there are multiple ways to triangulate that 3d point cloud into a surface. Here's a smooth surface example:
```
import numpy as np
from mpl_toolkits.mplot3d import Axes3D
# Axes3D import has side effects, it enables using projection='3d' in add_subplot
import matplotlib.pyplot as plt
import random

def fun(x, y):
    return x**2 + y

fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')
x = y = np.arange(-3.0, 3.0, 0.05)
X, Y = np.meshgrid(x, y)
zs = np.array(fun(np.ravel(X), np.ravel(Y)))
Z = zs.reshape(X.shape)

ax.plot_surface(X, Y, Z)

ax.set_xlabel('X Label')
ax.set_ylabel('Y Label')
ax.set_zlabel('Z Label')

plt.show()
```


Question:
Select One Element in Each Row of a Numpy Array by Column Indices [duplicate] This question already has answers here: NumPy selecting specific column index per row by using a list of indexes (7 answers) Closed 5 years ago. Is there a better way to get the "output_array" from the "input_array" and "select_id" ? Can we get rid of range( input_array.shape[0] ) ?
```

```python
input_array = numpy.array( [ [3,14], [12, 5], [75, 50] ] )
```

```python
select_id = [0, 1, 1]
```

```python
print input_array
#Output
#[[ 3 14]
# [12  5]
# [75 50]]
```

```python
output_array = input_array[  range( input_array.shape[0] ), select_id ]
```

```python
print output_array
#Output
#[ 3  5 50]
#```
```

Response:
You can choose from given array using numpy.choose which constructs an array from an index array (in your case select_id) and a set of arrays (in your case input_array) to choose from. However you may first need to transpose input_array to match dimensions. The following shows a small example:
```

```python
input_array
#Output
#array([[ 3, 14],
#       [12,  5],
#       [75, 50]])
```

```python
input_array.shape
#Output
#(3, 2)
```

```python
select_id
#Output
#[0, 1, 1]
```

```python
output_array = np.choose(select_id, input_array.T)
```

```python
output_array
#Output
#array([ 3,  5, 50])
#```
```


Question:
How to extract all columns but one from an array (or matrix) in python? Given a numpy 2d array (or a matrix), I would like to extract all the columns but the i-th. E. g. from
```
1 2 3 4
2 4 6 8
3 6 9 12
```
I would like to have, e.g.
```
1 2 3
2 4 6
3 6 9
```
or
```
1 2 4
2 4 8
3 6 12
```
I cannot find a pythonic way to do this. I now that you can extract given columns by simply
```
a[:,n]
```
or
```
a[:,[n,n+1,n+5]]
```
But what about extracting all of them but one?

Response:
Since for the general case you are going to be returning a copy anyway, you may find yourself producing more readable code by using np.delete:
```

```python
a = np.arange(12).reshape(3, 4)
```

```python
np.delete(a, 2, axis=1)
#Output
#array([[ 0,  1,  3],
#       [ 4,  5,  7],
#       [ 8,  9, 11]])
#```
```


Question:
Save multiple arrays to a csv file with column names As simple as it seems, I could not find any solution for my question online. Basically, I have two arrays a and b that I want to save to a csv file. It will be two columns. I want to add the column names as well. Below code I use to dump arrays to a csv.
```
from np import array, savetxt

a = array([1,2,3,4])
b = array([5,6,7,8])
savetxt('submission2.csv', zip(a,b), delimiter=',', fmt='%f')
```
How would I add column names? I would like the csv file to look like
```
Name1 Name2
 1     5
 2     6
 3     7
 4     8
```
It is so strange that this option is not in the savetxt function. header option does do it because it just pastes a comment into the first cell. Thanks. Edit: Arrays

Response:
You can do it with pandas package easily:
```
import pandas as pd
import numpy as np

a = np.array([1,2,3,4])
b = np.array([5,6,7,8])

df = pd.DataFrame({"name1" : a, "name2" : b})
df.to_csv("submission2.csv", index=False)
```


Question:
Why does the shape of a 1D array not show the number of rows as 1? I know that numpy array has a method called shape that returns [No.of rows, No.of columns], and shape[0] gives you the number of rows, shape[1] gives you the number of columns.
```
a = numpy.array([[1,2,3,4], [2,3,4,5]])
a.shape
>> [2,4]
a.shape[0]
>> 2
a.shape[1]
>> 4
```
However, if my array only have one row, then it returns [No.of columns, ]. And shape[1] will be out of the index. For example
```
a = numpy.array([1,2,3,4])
a.shape
>> [4,]
a.shape[0]
>> 4    //this is the number of column
a.shape[1]
>> Error out of index
```
Now how do I get the number of rows of an numpy array if the array may have only one row? Thank you

Response:
The concept of rows and columns applies when you have a 2D array. However, the array numpy.array([1,2,3,4]) is a 1D array and so has only one dimension, therefore shape rightly returns a single valued iterable. For a 2D version of the same array, consider the following instead:
```

```python
a = numpy.array([[1,2,3,4]]) # notice the extra square braces
```

```python
a.shape
#Output
#(1, 4)
#```
```


Question:
Numpy: Divide each row by a vector element Suppose I have a numpy array:
```
data = np.array([[1,1,1],[2,2,2],[3,3,3]])
```
and I have a corresponding "vector:"
```
vector = np.array([1,2,3])
```
How do I operate on data along each row to either subtract or divide so the result is:
```
sub_result = [[0,0,0], [0,0,0], [0,0,0]]
div_result = [[1,1,1], [1,1,1], [1,1,1]]
```
Long story short: How do I perform an operation on each row of a 2D array with a 1D array of scalars that correspond to each row?

Response:
Here you go. You just need to use None (or alternatively np.newaxis) combined with broadcasting:
```

```python
data - vector[:,None]
#Output
#array([[0, 0, 0],
#       [0, 0, 0],
#       [0, 0, 0]])
```

```python
data / vector[:,None]
#Output
#array([[1, 1, 1],
#       [1, 1, 1],
#       [1, 1, 1]])
#```
```


Question: How to write simple geometric shapes into numpy arrays I would like to generate a numpy array of 200x200 elements in size and put into it a circle centered into 100,100 coordinates, radius 80 and stroke width of 3 pixels. How to do this in python 2.7 without involving file operations? Possibly using geometry or imaging libraries to allow generalisation to other shapes.

Response:
The usual way is to define a coordinate mesh and apply your shape's equations. To do that the easiest way is to use numpy.mgrid: http://docs.scipy.org/doc/numpy/reference/generated/numpy.mgrid.html
```
# xx and yy are 200x200 tables containing the x and y coordinates as values
# mgrid is a mesh creation helper
xx, yy = numpy.mgrid[:200, :200]
# circles contains the squared distance to the (100, 100) point
# we are just using the circle equation learnt at school
circle = (xx - 100) ** 2 + (yy - 100) ** 2
# donuts contains 1's and 0's organized in a donut shape
# you apply 2 thresholds on circle to define the shape
donut = numpy.logical_and(circle < (6400 + 60), circle > (6400 - 60))
```


Question:
Adding row/column headers to NumPy arrays I have a NumPy ndarray to which I would like to add row/column headers. The data is actually 7x12x12, but I can represent it like this:
```
A=[[[0, 1, 2, 3, 4, 5],
      [1, 0, 3, 4, 5, 6],
      [2, 3, 0, 5, 6, 7],
      [3, 4, 5, 0, 7, 8],
      [4, 5, 6, 7, 0, 9],
      [5, 6, 7, 8, 9, 0]]

     [[0, 1, 2, 3, 4, 5],
      [1, 0, 3, 4, 5, 6],
      [2, 3, 0, 5, 6, 7],
      [3, 4, 5, 0, 7, 8],
      [4, 5, 6, 7, 0, 9],
      [5, 6, 7, 8, 9, 0]]]
```
where A is my 2x6x6 array. How do I insert headers across the first row and the first column, so that each array looks like this in my CSV output file?
```
A, a, b, c, d, e, f
        a, 0, 1, 2, 3, 4, 5,
        b, 1, 0, 3, 4, 5, 6,
        c, 2, 3, 0, 5, 6, 7,
        d, 3, 4, 5, 0, 7, 8,
        e, 4, 5, 6, 7, 0, 9,
        f, 5, 6, 7, 8, 9, 0
```
What I have done is made the array 7x13x13 and inserted the data such that I have a row and column of zeros, but I'd much prefer strings. I guess I could just write an Excel macro to replace the zeros with strings. However, the problem is that NumPy cannot convert string to float, if I try to reassign those zeros as the strings I want.

Response:
With pandas.DataFrame.to_csv you can write the columns and the index to a file:
```
import numpy as np
import pandas as pd

A = np.random.randint(0, 10, size=36).reshape(6, 6)
names = [_ for _ in 'abcdef']
df = pd.DataFrame(A, index=names, columns=names)
df.to_csv('df.csv', index=True, header=True, sep=' ')
```
will give you the following df.csv file:
```
a b c d e f
a 1 5 5 0 4 4
b 2 7 5 4 0 9
c 6 5 6 9 7 0
d 4 3 7 9 9 3
e 8 1 5 1 9 0
f 2 8 0 0 5 1
```


Question:
Can't use /= on numpy array On a numpy array, why is it I can successfully use / 2:
```

```python
a=np.array([2, 4, 6])
```

```python
a = a / 2
```

```python
a
#Output
#array([ 1.,  2.,  3.])
#```
#But I cannot use a /= 2?
#```
```

```python
a=np.array([2, 4, 6])
```

```python
a /= 2
#Output
#Traceback (most recent call last):
#  File "<stdin>", line 1, in <module>
#TypeError: No loop matching the specified signature and casting
#was found for ufunc true_divide
#```
#I've seen numpy Issue 6464, but don't understand from reading it and the linked release notes the reason this doesn't work. Is there any way to get /= to work as expected?
```

Response:
As pointed out in the comment, the change from int (which is how a is created) to float (which is the result of /) is not allowed when using /=. To "fix" this the dtype of a just has to be a float from the beginning:
```
a=np.array([2, 4, 6], dtype=np.float64)
a/=2
print(str(a))
>>>array([1., 2., 3.])
```


Question:
Swapping the dimensions of a numpy array I would like to do the following:
```
for i in dimension1:
  for j in dimension2:
    for k in dimension3:
      for l in dimension4:
        B[k,l,i,j] = A[i,j,k,l]
```
without the use of loops. In the end both A and B contain the same information but indexed differently. I must point out that the dimension 1,2,3 and 4 can be the same or different. So a numpy.reshape() seems difficult.

Response:
The canonical way of doing this in numpy would be to use np.transpose's optional permutation argument. In your case, to go from ijkl to klij, the permutation is (2, 3, 0, 1), e.g.:
```

```python
a = np.empty((2, 3, 4, 5))
```

```python
b = np.transpose(a, (2, 3, 0, 1))
```

```python
b.shape
#Output
#(4, 5, 2, 3)
#```
```


Question:
ValueError: setting an array element with a sequence Why do the following code samples:
```
np.array([[1, 2], [2, 3, 4]])

np.array([1.2, "abc"], dtype=float)
```
all give the following error?
```none
ValueError: setting an array element with a sequence.
```

Response:
Possible reason 1: trying to create a jagged array You may be creating an array from a list that isn't shaped like a multi-dimensional array:
```
numpy.array([[1, 2], [2, 3, 4]])         # wrong!
```
```
numpy.array([[1, 2], [2, [3, 4]]])       # wrong!
```
In these examples, the argument to numpy.array contains sequences of different lengths. Those will yield this error message because the input list is not shaped like a "box" that can be turned into a multidimensional array. Possible reason 2: providing elements of incompatible types For example, providing a string as an element in an array of type float:
```
numpy.array([1.2, "abc"], dtype=float)   # wrong!
```
If you really want to have a NumPy array containing both strings and floats, you could use the dtype object, which allows the array to hold arbitrary Python objects:
```
numpy.array([1.2, "abc"], dtype=object)
```


Question: Normalise between 0 and 1 ignoring NaN For a list of numbers ranging from x to y that may contain NaN, how can I normalise between 0 and 1, ignoring the NaN values (they stay as NaN). Typically I would use MinMaxScaler (ref page) from sklearn.preprocessing, but this cannot handle NaN and recommends imputing the values based on mean or median etc. it doesn't offer the option to ignore all the NaN values.

Response:
consider pd.Series s
```
s = pd.Series(np.random.choice([3, 4, 5, 6, np.nan], 100))
s.hist()
```
Option 1 Min Max Scaling
```
new = s.sub(s.min()).div((s.max() - s.min()))
new.hist()
```
NOT WHAT OP ASKED FOR I put these in because I wanted to Option 2 sigmoid
```
sigmoid = lambda x: 1 / (1 + np.exp(-x))

new = sigmoid(s.sub(s.mean()))
new.hist()
```
Option 3 tanh (hyperbolic tangent)
```
new = np.tanh(s.sub(s.mean())).add(1).div(2)
new.hist()
```


Question: Resample time series in pandas to a weekly interval How do I resample a time series in pandas to a weekly frequency where the weeks start on an arbitrary day? I see that there's an optional keyword base but it only works for intervals shorter than a day.

Response:
You can pass anchored offsets to resample, among other options they cover this case. For example the weekly frequency from Monday:
```
ts.resample('W-MON')
```


Question:
What is the equivalent of "zip()" in Python's numpy? I am trying to do the following but with numpy arrays:
```
x = [(0.1, 1.), (0.1, 2.), (0.1, 3.), (0.1, 4.), (0.1, 5.)]
normal_result = zip(*x)
```
This should give a result of:
```
normal_result = [(0.1, 0.1, 0.1, 0.1, 0.1), (1., 2., 3., 4., 5.)]
```
But if the input vector is a numpy array:
```
y = np.array(x)
numpy_result = zip(*y)
print type(numpy_result)
```
It (expectedly) returns a:
```
<type 'list'>
```
The issue is that I will need to transform the result back into a numpy array after this. What I would like to know is what is if there is an efficient numpy function that will avoid these back-and-forth transformations?

Response:
You can just transpose it...
```

```python
a = np.array([(0.1, 1.), (0.1, 2.), (0.1, 3.), (0.1, 4.), (0.1, 5.)])
```

```python
a
#Output
#array([[ 0.1,  1. ],
#       [ 0.1,  2. ],
#       [ 0.1,  3. ],
#       [ 0.1,  4. ],
#       [ 0.1,  5. ]])
```

```python
a.T
#Output
#array([[ 0.1,  0.1,  0.1,  0.1,  0.1],
#       [ 1. ,  2. ,  3. ,  4. ,  5. ]])
#```
```


Question:
What is the purpose of numpy.where returning a tuple? When I run this code:
```
import numpy as np
a = np.array([1, 2, 3, 4, 5, 6])
print(np.where(a > 2))
```
it would be natural to get an array of indices where a > 2, i.e. [2, 3, 4, 5], but instead we get:
```
(array([2, 3, 4, 5], dtype=int64),)
```
i.e. a tuple with empty second member. Then, to get the the "natural" answer of numpy.where, we have to do:
```
np.where(a > 2)[0]
```
What's the point in this tuple? In which situation is it useful? Note: I'm speaking here only about the use case numpy.where(cond) and not numpy.where(cond, x, y) that also exists (see documentation).

Response:
numpy.where returns a tuple because each element of the tuple refers to a dimension. Consider this example in 2 dimensions:
```
a = np.array([[1, 2, 3, 4, 5, 6],
              [-2, 1, 2, 3, 4, 5]])

print(np.where(a > 2))

(array([0, 0, 0, 0, 1, 1, 1], dtype=int64),
 array([2, 3, 4, 5, 3, 4, 5], dtype=int64))
```
As you can see, the first element of the tuple refers to the first dimension of relevant elements; the second element refers to the second dimension. This is a convention numpy often uses. You will see it also when you ask for the shape of an array, i.e. the shape of a 1-dimensional array will return a tuple with 1 element:
```
a = np.array([[1, 2, 3, 4, 5, 6],
              [-2, 1, 2, 3, 4, 5]])

print(a.shape, a.ndim)  # (2, 6) 2

b = np.array([1, 2, 3, 4, 5, 6])

print(b.shape, b.ndim)  # (6,) 1
```


Question:
Quick way to access first element in Numpy array with arbitrary number of dimensions? I have a function that I want to have quickly access the first (aka zeroth) element of a given Numpy array, which itself might have any number of dimensions. What's the quickest way to do that? I'm currently using the following:
```
a.reshape(-1)[0]
```
This reshapes the perhaps-multi-dimensionsal array into a 1D array and grabs the zeroth element, which is short, sweet and often fast. However, I think this would work poorly with some arrays, e.g., an array that is a transposed view of a large array, as I worry this would end up needing to create a copy rather than just another view of the original array, in order to get everything in the right order. (Is that right? Or am I worrying needlessly?) Regardless, it feels like this is doing more work than what I really need, so I imagine some of you may know a generally faster way of doing this? Other options I've considered are creating an iterator over the whole array and drawing just one element from it, or creating a vector of zeroes containing one zero for each dimension and using that to fancy-index into the array. But neither of these seems all that great either.

Response:
```
a.flat[0]
```
This should be pretty fast and never require a copy. (Note that a.flat is an instance of numpy.flatiter, not an array, which is why this operation can be done without a copy.)


Question:
How to filter a numpy array using another array's values? I have two NumPy arrays, e.g.:
```
a = [1,2,3,4,5]
```
and a filter array, e.g.:
```
f = [False, True, False, False, True]

len(a) == len(f)
```
How can I get a new numpy array with only the values in a where the same index in f is True? In my case: [2, 5]. According to the accepted solution (with different values):
```

```python
a = numpy.array([ 1,  2,  3,  4,  5,  6,  7,  8,  9, 10])
```

```python
b = numpy.array([True, False, True, False, True, False, True, False, True, False])
```

```python
a[b]
#Output
#array([1, 3, 5, 7, 9])
#```
```

Response:
NumPy supports boolean indexing
```
a[f]
```
This assumes that a and f are NumPy arrays rather than Python lists (as in the question). You can convert with f = np.array(f).


Question: Replacing Pandas or Numpy Nan with a None to use with MysqlDB I am trying to write a Pandas dataframe (or can use a numpy array) to a mysql database using MysqlDB . MysqlDB doesn't seem understand 'nan' and my database throws out an error saying nan is not in the field list. I need to find a way to convert the 'nan' into a NoneType. Any ideas?

Response:
For pandas > 1.3.0 see this answer. @bogatron has it right, you can use where, it's worth noting that you can do this natively in pandas:
```
df1 = df.where(pd.notnull(df), None)
```
Note: this changes the dtype of all columns to object. Example:
```

```python
df = pd.DataFrame([1, np.nan])
```

```python
df
#Output
#0
#0   1
#1 NaN
```

```python
df1 = df.where(pd.notnull(df), None)
```

```python
df1
#Output
#0
#0     1
#1  None
#```
#Note: what you cannot do recast the DataFrames dtype to allow all datatypes types, using astype, and then the DataFrame fillna method:
#```
#df1 = df.astype(object).replace(np.nan, 'None')
#```
#Unfortunately neither this, nor using replace, works with None see this (closed) issue. As an aside, it's worth noting that for most use cases you don't need to replace NaN with None, see this question about the difference between NaN and None in pandas. However, in this specific case it seems you do (at least at the time of this answer).
```


Question:
Check if a numpy array is sorted I have a numpy array and I like to check if it is sorted.
```

```python
a = np.array([1,2,3,4,5])
#Output
#array([1, 2, 3, 4, 5])
#```
```

Response:
```
np.all(a[:-1] <= a[1:])
```
Examples:
```
is_sorted = lambda a: np.all(a[:-1] <= a[1:])

```python
a = np.array([1,2,3,4,9])
```

```python
is_sorted(a)
#Output
#True
```

```python
a = np.array([1,2,3,4,3])
```

```python
is_sorted(a)
#Output
#False
#```
```


Question:
Is there a python (scipy) function to determine parameters needed to obtain a target power? In R there is a very useful function that helps with determining parameters for a two sided t-test in order to obtain a target statistical power. The function is called power.prop.test. http://stat.ethz.ch/R-manual/R-patched/library/stats/html/power.prop.test.html You can call it using:
```
power.prop.test(p1 = .50, p2 = .75, power = .90)
```
And it will tell you n the sample size needed to obtain this power. This is extremely useful in deterring sample sizes for tests. Is there a similar function in the scipy package?

Response:
I've managed to replicate the function using the below formula for n and the inverse survival function norm.isf from scipy.stats
```
from scipy.stats import norm, zscore

def sample_power_probtest(p1, p2, power=0.8, sig=0.05):
    z = norm.isf([sig/2]) #two-sided t test
    zp = -1 * norm.isf([power])
    d = (p1-p2)
    s =2*((p1+p2) /2)*(1-((p1+p2) /2))
    n = s * ((zp + z)**2) / (d**2)
    return int(round(n[0]))

def sample_power_difftest(d, s, power=0.8, sig=0.05):
    z = norm.isf([sig/2])
    zp = -1 * norm.isf([power])
    n = s * ((zp + z)**2) / (d**2)
    return int(round(n[0]))

if __name__ == '__main__':

    n = sample_power_probtest(0.1, 0.11, power=0.8, sig=0.05)
    print n  #14752

    n = sample_power_difftest(0.1, 0.5, power=0.8, sig=0.05)
    print n  #392
```


Question: 10 Minutes to Pandas tutorial - to_numpy() does not exist? Following: https://pandas.pydata.org/pandas-docs/stable/getting_started/10min.html This - df.to_numpy() throws an AttributeError: 'DataFrame' object has no attribute 'to_numpy' Not sure why.

Response: This feature was just added in Version 0.24.0, which was released a couple of days ago. If you haven't updated yet, the attribute doesn't exist! Once you update pandas the problem should resolve itself.


Question:
python "TypeError: 'numpy.float64' object cannot be interpreted as an integer" This is the code which is produing the error:
```py
import numpy as np

for i in range(len(x)):
    if (np.floor(N[i]/2)==N[i]/2):
        for j in range(N[i]/2):
            pxd[i,j]=x[i]-(delta*j)*np.sin(s[i]*np.pi/180)
            pyd[i,j]=y[i]-(delta*j)*np.cos(s[i]*np.pi/180)

    else:
        for j in range((N[i]-1)/2):
            pxd[i,j]=x[i]-(delta*j)*np.sin(s[i]*np.pi/180)
            pyd[i,j]=y[i]-(delta*j)*np.cos(s[i]*np.pi/180)
```
Does anyone have an idea as to how to solve this problem? How can I get the code to run successfully?

Response:
```
N=np.floor(np.divide(l,delta))
...
for j in range(N[i]/2):
```
N[i]/2 will be a float64 but range() expects an integer. Just cast the call to
```
for j in range(int(N[i]/2)):
```


Question:
How to normalize a 2-dimensional numpy array in python less verbose? Given a 3 times 3 numpy array
```
a = numpy.arange(0,27,3).reshape(3,3)

# array([[ 0,  3,  6],
#        [ 9, 12, 15],
#        [18, 21, 24]])
```
To normalize the rows of the 2-dimensional array I thought of
```
row_sums = a.sum(axis=1) # array([ 9, 36, 63])
new_matrix = numpy.zeros((3,3))
for i, (row, row_sum) in enumerate(zip(a, row_sums)):
    new_matrix[i,:] = row / row_sum
```
There must be a better way, isn't there? Perhaps to clearify: By normalizing I mean, the sum of the entrys per row must be one. But I think that will be clear to most people.

Response:
Broadcasting is really good for this:
```
row_sums = a.sum(axis=1)
new_matrix = a / row_sums[:, numpy.newaxis]
```
row_sums[:, numpy.newaxis] reshapes row_sums from being (3,) to being (3, 1). When you do a / b, a and b are broadcast against each other. You can learn more about broadcasting here or even better here.


Question: Python random sample of two arrays, but matching indices I have two numpy arrays x and y, which have length 10,000. I would like to plot a random subset of 1,000 entries of both x and y. Is there an easy way to use the lovely, compact random.sample(population, k) on both x and y to select the same corresponding indices? (The y and x vectors are linked by a function y(x) say.) Thanks.

Response:
You can use np.random.choice on an index array and apply it to both arrays:
```
idx = np.random.choice(np.arange(len(x)), 1000, replace=False)
x_sample = x[idx]
y_sample = y[idx]
```


Question: Stratified Sampling in Pandas I've looked at the Sklearn stratified sampling docs as well as the pandas docs and also Stratified samples from Pandas and sklearn stratified sampling based on a column but they do not address this issue. Im looking for a fast pandas/sklearn/numpy way to generate stratified samples of size n from a dataset. However, for rows with less than the specified sampling number, it should take all of the entries. Concrete example: Thank you! :)

Response:
Use min when passing the number to sample. Consider the dataframe df
```
df = pd.DataFrame(dict(
        A=[1, 1, 1, 2, 2, 2, 2, 3, 4, 4],
        B=range(10)
    ))

df.groupby('A', group_keys=False).apply(lambda x: x.sample(min(len(x), 2)))

   A  B
1  1  1
2  1  2
3  2  3
6  2  6
7  3  7
9  4  9
8  4  8
```


Question: python/numpy: how to get 2D array column length? How do i get the length of the column in a nD array? example, i have a nD array called a. when i print a.shape, it returns (1,21). I want to do a for loop, in the range of the column size of the array a. How do i get the value of

Response:
You can get the second dimension of the array as:
```
a.shape[1]
```


Question: Adding alpha channel to RGB array using numpy [duplicate] This question already has answers here: Append 2D array to 3D array, extending third dimension (2 answers) Closed 1 year ago. I have an image array in RGB space and want to add the alpha channel to be all zeros. Specifically, I have a numpy array with shape (205, 54, 3) and I want to change the shape to (205, 54, 4) with the additional spot in the third dimension being all 0.0's. Which numpy operation would achieve this?

Response:
You could use one of the stack functions (stack/hstack/vstack/dstack/concatenate) to join multiple arrays together.
```
numpy.dstack( ( your_input_array, numpy.zeros((205, 54)) ) )
```


Question:
How to change column names in pandas Dataframe using a list of names? I have been trying to change the column names of a pandas dataframe using a list of names. The following code is being used:
```
df.rename(columns = list_of_names, inplace=True)
```
However I got a Type Error each time, with an error message that says "list object is not callable". I would like to know why does this happen? And What can I do to solve this problem? Thank you for your help.

Response:
you could use
```
df.columns = ['Leader', 'Time', 'Score']
```


Question: array.array versus numpy.array If you are creating a 1d array in Python, is there any benefit to using the NumPy package?

Response: It all depends on what you plan to do with the array. If all you're doing is creating arrays of simple data types and doing I/O, the array module will do just fine. If, on the other hand, you want to do any kind of numerical calculations, the array module doesn't provide any help with that. NumPy (and SciPy) give you a wide variety of operations between arrays and special functions that are useful not only for scientific work but for things like advanced image manipulation or in general anything where you need to perform efficient calculations with large amounts of data. Numpy is also much more flexible, e.g. it supports arrays of any type of Python objects, and is also able to interact "natively" with your own objects if they conform to the array interface.


Question:
pandas get unique values from column of lists How do I get the unique values of a column of lists in pandas or numpy such that second column from would result in 'action', 'crime', 'drama'. The closest (but non-functional) solutions I could come up with were:
```
genres = data['Genre'].unique()
```
But this predictably results in a TypeError saying how lists aren't hashable. TypeError: unhashable type: 'list' Set seemed to be a good idea but genres = data.apply(set(), columns=['Genre'], axis=1) but also results in a TypeError: set() takes no keyword arguments

Response:
You can use explode:
```py
data = pd.DataFrame([
    {
        "title": "The Godfather: Part II",
        "genres": ["crime", "drama"],
        "director": "Fracis Ford Coppola"
    },
    {
        "title": "The Dark Knight",
        "genres": ["action", "crime", "drama"],
        "director": "Christopher Nolan"
    }
])
# Changed from data.explode("genres")["genres"].unique() as suggested by rafaelc
data["genres"].explode().unique()
```
Results in:
```
array(['crime', 'drama', 'action'], dtype=object)
```


Question:
How to multiply numpy 2D array with numpy 1D array? The two arrays:
```
a = numpy.array([[2,3,2],[5,6,1]])
b = numpy.array([3,5])
c = a * b
```
What I want is:
```
c = [[6,9,6],
     [25,30,5]]
```
But, I am getting this error:
```
ValueError: operands could not be broadcast together with shapes (2,3) (2)
```
How to multiply a nD array with 1D array, where len(1D-array) == len(nD array)?

Response:
You need to convert array b to a (2, 1) shape array, use None or numpy.newaxis in the index tuple:
```
import numpy
a = numpy.array([[2,3,2],[5,6,1]])
b = numpy.array([3,5])
c = a * b[:, None]
```
Here is the document.


Question:
How to load a list of numpy arrays to pytorch dataset loader? I have a huge list of numpy arrays, where each array represents an image and I want to load it using torch.utils.data.Dataloader object. But the documentation of torch.utils.data.Dataloader mentions that it loads data directly from a folder. How do I modify it for my cause? I am new to pytorch and any help would be greatly appreciated. my numpy array for a single image looks something like this. The image is RBG image.
```
[[[ 70  82  94]
  [ 67  81  93]
  [ 66  82  94]
  ...,
  [182 182 188]
  [183 183 189]
  [188 186 192]]

 [[ 66  80  92]
  [ 62  78  91]
  [ 64  79  95]
  ...,
  [176 176 182]
  [178 178 184]
  [180 180 186]]

 [[ 62  82  93]
  [ 62  81  96]
  [ 65  80  99]
  ...,
  [169 172 177]
  [173 173 179]
  [172 172 178]]

 ...,
```

Response:
I think what DataLoader actually requires is an input that subclasses Dataset. You can either write your own dataset class that subclasses Datasetor use TensorDataset as I have done below:
```
import torch
import numpy as np
from torch.utils.data import TensorDataset, DataLoader

my_x = [np.array([[1.0,2],[3,4]]),np.array([[5.,6],[7,8]])] # a list of numpy arrays
my_y = [np.array([4.]), np.array([2.])] # another list of numpy arrays (targets)

tensor_x = torch.Tensor(my_x) # transform to torch tensor
tensor_y = torch.Tensor(my_y)

my_dataset = TensorDataset(tensor_x,tensor_y) # create your datset
my_dataloader = DataLoader(my_dataset) # create your dataloader
```
Works for me.


Question:
Matplotlib scatter plot with unknown error I am attempting to create a scatter plot. I have a list of numbers from 0 - 17 as well as an array with 18 values. I can plot the data as a line plot but when I try to plot as a scatter, I get an error message I do not understand:
```none
TypeError: ufunc 'sqrt' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe''
```
What does this error message mean and how can I get the data to plot as a scatter?
```py
import numpy as np
import matplotlib.pyplot as plt

y = [7316.0, 7453.25, 7518.25, 7711.5, 7448.0, 7210.25, 7416.75, 6960.75,
     7397.75, 6397.5, 5522.75, 5139.0, 5034.75, 4264.75, 5106.0, 3489.5,
     4712.0, 4770.0]
x = np.arange(0,18,1)

plt.rcParams['legend.loc'] = 'best'
plt.figure(1)
plt.xlim(0, 20)
plt.ylim(0, 10000)
plt.scatter(x, y, 'r')
plt.show()
```

Response:
Check the scatter documentation. Third argument is for size of points and should be scalar or array_like. I assume 'r' is for color so do the following:
```
plt.scatter(x, y, c='r')
```


Question:
Generating a dense matrix from a sparse matrix in numpy python I have a Sqlite database that contains following type of schema:
```
termcount(doc_num, term , count)
```
This table contains terms with their respective counts in the document. like
```
(doc1 , term1 ,12)
(doc1, term 22, 2)
.
.
(docn,term1 , 10)
```
This matrix can be considered as sparse matrix as each documents contains very few terms that will have a non-zero value. How would I create a dense matrix from this sparse matrix using numpy as I have to calculate the similarity among documents using cosine similarity. This dense matrix will look like a table that have docid as the first column and all the terms will be listed as the first row.and remaining cells will contain counts.

Response:
```
from scipy.sparse import csr_matrix
 A = csr_matrix([[1,0,2],[0,3,0]])
 >>>A
 <2x3 sparse matrix of type '<type 'numpy.int64'>'
    with 3 stored elements in Compressed Sparse Row format>
 >>> A.todense()
   matrix([[1, 0, 2],
           [0, 3, 0]])
 >>> A.toarray()
      array([[1, 0, 2],
            [0, 3, 0]])
```
this is an example of how to convert a sparse matrix to a dense matrix taken from scipy


Question: Argmax of numpy array returning non-flat indices I'm trying to get the indices of the maximum element in a Numpy array. This can be done using numpy.argmax. My problem is, that I would like to find the biggest element in the whole array and get the indices of that. numpy.argmax can be either applied along one axis, which is not what I want, or on the flattened array, which is kind of what I want. My problem is that using numpy.argmax with axis=None returns the flat index when I want the multi-dimensional index. I could use divmod to get a non-flat index but this feels ugly. Is there any better way of doing this?

Response:
You could use numpy.unravel_index() on the result of numpy.argmax():
```

```python
a = numpy.random.random((10, 10))
```

```python
numpy.unravel_index(a.argmax(), a.shape)
#Output
#(6, 7)
```

```python
a[6, 7] == a.max()
#Output
#True
#```
```


Question:
Finding entries containing a substring in a numpy array? I tried to find entries in an Array containing a substring with np.where and an in condition:
```
import numpy as np
foo = "aa"
bar = np.array(["aaa", "aab", "aca"])
np.where(foo in bar)
```
this only returns an empty Array. Why is that so? And is there a good alternative solution?

Response:
We can use np.core.defchararray.find to find the position of foo string in each element of bar, which would return -1 if not found. Thus, it could be used to detect whether foo is present in each element or not by checking for -1 on the output from find. Finally, we would use np.flatnonzero to get the indices of matches. So, we would have an implementation, like so -
```
np.flatnonzero(np.core.defchararray.find(bar,foo)!=-1)
```
Sample run -
```

```python
bar
#Output
#array(['aaa', 'aab', 'aca'],
#      dtype='|S3')
```

```python
foo
#Output
#'aa'
```

```python
np.flatnonzero(np.core.defchararray.find(bar,foo)!=-1)
#Output
#array([0, 1])
```

```python
bar[2] = 'jaa'
```

```python
np.flatnonzero(np.core.defchararray.find(bar,foo)!=-1)
#Output
#array([0, 1, 2])
#```
```


Question: How to do weighted random sample of categories in python Given a list of tuples where each tuple consists of a probability and an item I'd like to sample an item according to its probability. For example, give the list [ (.3, 'a'), (.4, 'b'), (.3, 'c')] I'd like to sample 'b' 40% of the time. What's the canonical way of doing this in python? I've looked at the random module which doesn't seem to have an appropriate function and at numpy.random which although it has a multinomial function doesn't seem to return the results in a nice form for this problem. I'm basically looking for something like mnrnd in matlab. Many thanks. Thanks for all the answers so quickly. To clarify, I'm not looking for explanations of how to write a sampling scheme, but rather to be pointed to an easy way to sample from a multinomial distribution given a set of objects and weights, or to be told that no such function exists in a standard library and so one should write one's own.

Response:
This might do what you want:
```
numpy.array([.3,.4,.3]).cumsum().searchsorted(numpy.random.sample(5))
```


Question:
How to convert a pandas DataFrame subset of columns AND rows into a numpy array? I'm wondering if there is a simpler, memory efficient way to select a subset of rows and columns from a pandas DataFrame. For instance, given this dataframe:
```
df = DataFrame(np.random.rand(4,5), columns = list('abcde'))
print df

          a         b         c         d         e
0  0.945686  0.000710  0.909158  0.892892  0.326670
1  0.919359  0.667057  0.462478  0.008204  0.473096
2  0.976163  0.621712  0.208423  0.980471  0.048334
3  0.459039  0.788318  0.309892  0.100539  0.753992
```
I want only those rows in which the value for column 'c' is greater than 0.5, but I only need columns 'b' and 'e' for those rows. This is the method that I've come up with - perhaps there is a better "pandas" way?
```
locs = [df.columns.get_loc(_) for _ in ['a', 'd']]
print df[df.c > 0.5][locs]

          a         d
0  0.945686  0.892892
```
My final goal is to convert the result to a numpy array to pass into an sklearn regression algorithm, so I will use the code above like this:
```
training_set = array(df[df.c > 0.5][locs])
```
... and that peeves me since I end up with a huge array copy in memory. Perhaps there's a better way for that too?

Response:
Use its value directly:
```

```python
df[df.c > 0.5][['b', 'e']].values
#Output
#array([[ 0.98836259,  0.82403141],
#       [ 0.337358  ,  0.02054435],
#       [ 0.29271728,  0.37813099],
#       [ 0.70033513,  0.69919695]])
#```
```


Question: What's the error of numpy.polyfit? I want to use numpy.polyfit for physical calculations, therefore I need the magnitude of the error.

Response:
If you specify full=True in your call to polyfit, it will include extra information:
```

```python
x = np.arange(100)
```

```python
y = x**2 + 3*x + 5 + np.random.rand(100)
```

```python
np.polyfit(x, y, 2)
#Output
#array([ 0.99995888,  3.00221219,  5.56776641])
```

```python
np.polyfit(x, y, 2, full=True)
#Output
#(array([ 0.99995888,  3.00221219,  5.56776641]), # coefficients
# array([ 7.19260721]), # residuals
# 3, # rank
# array([ 11.87708199,   3.5299267 ,   0.52876389]), # singular values
# 2.2204460492503131e-14) # conditioning threshold
#```
#The residual value returned is the sum of the squares of the fit errors, not sure if this is what you are after:
#```
```

```python
np.sum((np.polyval(np.polyfit(x, y, 2), x) - y)**2)
#Output
#7.1926072073491056
#```
#In version 1.7 there is also a cov keyword that will return the covariance matrix for your coefficients, which you could use to calculate the uncertainty of the fit coefficients themselves.
```


Question:
How to select all elements in a NumPy array except for a sequence of indices [duplicate] This question already has answers here: Extract elements from numpy array, that are not in list of indexes (4 answers) Closed 5 years ago. Say I have some long array and a list of indices. How can I select everything except those indices? I found a solution but it is not elegant:
```
import numpy as np
x = np.array([0,10,20,30,40,50,60])
exclude = [1, 3, 5]
print x[list(set(range(len(x))) - set(exclude))]
```

Response:
This is what numpy.delete does. (It doesn't modify the input array, so you don't have to worry about that.)
```

```python
np.delete(x, exclude)
#Output
#array([ 0, 20, 40, 60])
#```
```


Question:
How to normalize a numpy array to a unit vector I would like to convert a NumPy array to a unit vector. More specifically, I am looking for an equivalent version of this normalisation function:
```
def normalize(v):
    norm = np.linalg.norm(v)
    if norm == 0:
       return v
    return v / norm
```
This function handles the situation where vector v has the norm value of 0. Is there any similar functions provided in sklearn or numpy?

Response:
If you're using scikit-learn you can use sklearn.preprocessing.normalize:
```
import numpy as np
from sklearn.preprocessing import normalize

x = np.random.rand(1000)*10
norm1 = x / np.linalg.norm(x)
norm2 = normalize(x[:,np.newaxis], axis=0).ravel()
print np.all(norm1 == norm2)
# True
```


Question:
Convert Select Columns in Pandas Dataframe to Numpy Array I would like to convert everything but the first column of a pandas dataframe into a numpy array. For some reason using the columns= parameter of DataFrame.to_matrix() is not working. df:
```
viz  a1_count  a1_mean     a1_std
0   n         3        2   0.816497
1   n         0      NaN        NaN
2   n         2       51  50.000000
```
I tried X=df.as_matrix(columns=[df[1:]]) but this yields an array of all NaNs

Response:
the easy way is the "values" property df.iloc[:,1:].values
```
a=df.iloc[:,1:]
b=df.iloc[:,1:].values

print(type(df))
print(type(a))
print(type(b))
```
so, you can get type
```
<class 'pandas.core.frame.DataFrame'>
<class 'pandas.core.frame.DataFrame'>
<class 'numpy.ndarray'>
```


Question:
How to apply piecewise linear fit in Python? I am trying to fit piecewise linear fit as shown in fig.1 for a data set This figure was obtained by setting on the lines. I attempted to apply a piecewise linear fit using the code:
```
from scipy import optimize
import matplotlib.pyplot as plt
import numpy as np

x = np.array([1, 2, 3, 4, 5, 6, 7, 8, 9, 10 ,11, 12, 13, 14, 15])
y = np.array([5, 7, 9, 11, 13, 15, 28.92, 42.81, 56.7, 70.59, 84.47, 98.36, 112.25, 126.14, 140.03])

def linear_fit(x, a, b):
    return a * x + b
fit_a, fit_b = optimize.curve_fit(linear_fit, x[0:5], y[0:5])[0]
y_fit = fit_a * x[0:7] + fit_b
fit_a, fit_b = optimize.curve_fit(linear_fit, x[6:14], y[6:14])[0]
y_fit = np.append(y_fit, fit_a * x[6:14] + fit_b)

figure = plt.figure(figsize=(5.15, 5.15))
figure.clf()
plot = plt.subplot(111)
ax1 = plt.gca()
plot.plot(x, y, linestyle = '', linewidth = 0.25, markeredgecolor='none', marker = 'o', label = r'\textit{y_a}')
plot.plot(x, y_fit, linestyle = ':', linewidth = 0.25, markeredgecolor='none', marker = '', label = r'\textit{y_b}')
plot.set_ylabel('Y', labelpad = 6)
plot.set_xlabel('X', labelpad = 6)
figure.savefig('test.pdf', box_inches='tight')
plt.close()
```
But this gave me fitting of the form in fig. 2, I tried playing with the values but no change I can't get the fit of the upper line proper. The most important requirement for me is how can I get Python to get the gradient change point. I want the code to recognize and fit two linear fits in the appropriate range. How can this be done in Python?

Response:
You can use numpy.piecewise() to create the piecewise function and then use curve_fit(), Here is the code
```
from scipy import optimize
import matplotlib.pyplot as plt
import numpy as np
%matplotlib inline

x = np.array([1, 2, 3, 4, 5, 6, 7, 8, 9, 10 ,11, 12, 13, 14, 15], dtype=float)
y = np.array([5, 7, 9, 11, 13, 15, 28.92, 42.81, 56.7, 70.59, 84.47, 98.36, 112.25, 126.14, 140.03])

def piecewise_linear(x, x0, y0, k1, k2):
    return np.piecewise(x, [x < x0], [lambda x:k1*x + y0-k1*x0, lambda x:k2*x + y0-k2*x0])

p , e = optimize.curve_fit(piecewise_linear, x, y)
xd = np.linspace(0, 15, 100)
plt.plot(x, y, "o")
plt.plot(xd, piecewise_linear(xd, *p))
```
the output: For an N parts fitting, please reference segments_fit.ipynb


Question:
How to get a list of all indices of repeated elements in a numpy array I'm trying to get the index of all repeated elements in a numpy array, but the solution I found for the moment is REALLY inefficient for a large (>20000 elements) input array (it takes more or less 9 seconds). The idea is simple: records_arrayis a numpy array of timestamps (datetime) from which we want to extract the indexes of repeated timestamps time_array is a numpy array containing all the timestamps that are repeated in records_array records is a django QuerySet (which can easily converted to a list) containing some Record objects. We want to create a list of couples formed by all possible combinations of tagId attributes of Record corresponding to the repeated timestamps found from records_array. Here is the working (but inefficient) code I have for the moment:
```
tag_couples = [];
for t in time_array:
    users_inter = np.nonzero(records_array == t)[0] # Get all repeated timestamps in records_array for time t
    l = [str(records[i].tagId) for i in users_inter] # Create a temporary list containing all tagIds recorded at time t
    if l.count(l[0]) != len(l): #remove tuples formed by the first tag repeated
        tag_couples +=[x for x in itertools.combinations(list(set(l)),2)] # Remove duplicates with list(set(l)) and append all possible couple combinations to tag_couples
```
I'm quite sure this can be optimized by using Numpy, but I can't find a way to compare records_array with each element of time_array without using a for loop (this can't be compared by just using ==, since they are both arrays).

Response:
A vectorized solution with numpy, on the magic of unique().
```
import numpy as np

# create a test array
records_array = np.array([1, 2, 3, 1, 1, 3, 4, 3, 2])

# creates an array of indices, sorted by unique element
idx_sort = np.argsort(records_array)

# sorts records array so all unique elements are together
sorted_records_array = records_array[idx_sort]

# returns the unique values, the index of the first occurrence of a value, and the count for each element
vals, idx_start, count = np.unique(sorted_records_array, return_counts=True, return_index=True)

# splits the indices into separate arrays
res = np.split(idx_sort, idx_start[1:])

#filter them with respect to their size, keeping only items occurring more than once
vals = vals[count > 1]
res = filter(lambda x: x.size > 1, res)
```
The following code was the original answer, which required a bit more memory, using numpy broadcasting and calling unique twice:
```
records_array = array([1, 2, 3, 1, 1, 3, 4, 3, 2])
vals, inverse, count = unique(records_array, return_inverse=True,
                              return_counts=True)

idx_vals_repeated = where(count > 1)[0]
vals_repeated = vals[idx_vals_repeated]

rows, cols = where(inverse == idx_vals_repeated[:, newaxis])
_, inverse_rows = unique(rows, return_index=True)
res = split(cols, inverse_rows[1:])
```
with as expected res = [array([0, 3, 4]), array([1, 8]), array([2, 5, 7])]


Question:
Size of data type using NumPy In NumPy, I can get the size (in bytes) of a particular data type by:
```
datatype(...).itemsize
```
or:
```
datatype(...).nbytes
```
For example:
```
np.float32(5).itemsize # 4
np.float32(5).nbytes   # 4
```
I have two questions. First, is there a way to get this information without creating an instance of the datatype? Second, what's the difference between itemsize and nbytes?

Response:
You need an instance of the dtype to get the itemsize, but you shouldn't need an instance of the ndarray. (As will become clear in a second, nbytes is a property of the array, not the dtype.) E.g.
```
print np.dtype(float).itemsize
print np.dtype(np.float32).itemsize
print np.dtype('|S10').itemsize
```
As far as the difference between itemsize and nbytes, nbytes is just x.itemsize * x.size. E.g.
```

```python
print np.arange(100).itemsize
8
```

```python
print np.arange(100).nbytes
800
```
```


Question:
How to zip two 1d numpy array to 2d numpy array [duplicate] This question already has answers here: Converting two lists into a matrix (5 answers) Closed 8 years ago. I have two numpy 1d arrays, e.g:
```
a = np.array([1,2,3,4,5])
b = np.array([6,7,8,9,10])
```
Then how can I get one 2d array [[1,6], [2,7], [3,8], [4,9], [5, 10]]?

Response:
If you have numpy arrays you can use dstack():
```
import numpy as np

a = np.array([1,2,3,4,5])
b = np.array([6,7,8,9,10])

c = np.dstack((a,b))
#or
d = np.column_stack((a,b))

```python
c
#Output
#array([[[ 1,  6],
#        [ 2,  7],
#        [ 3,  8],
#        [ 4,  9],
#        [ 5, 10]]])
```

```python
d
#Output
#array([[ 1,  6],
#       [ 2,  7],
#       [ 3,  8],
#       [ 4,  9],
#       [ 5, 10]])
```

```python
c.shape
#Output
#(1, 5, 2)
```

```python
d.shape
#Output
#(5, 2)
#```
```


Question: What does numpy.gradient do? So I know what the gradient of a (mathematical) function is, so I feel like I should know what numpy.gradient does. But I don't. The documentation is not really helpful either: Return the gradient of an N-dimensional array. What is the gradient of an array? When is numpy.gradient useful?

Response:
Also in the documentation1:
```

```python
y = np.array([1, 2, 4, 7, 11, 16], dtype=np.float)
```

```python
j = np.gradient(y)
```

```python
j
#Output
#array([ 1. ,  1.5,  2.5,  3.5,  4.5,  5. ])
#```
#Gradient is defined as (change in y)/(change in x). x, here, is the list index, so the difference between adjacent values is 1. At the boundaries, the first difference is calculated. This means that at each end of the array, the gradient given is simply, the difference between the end two values (divided by 1) Away from the boundaries the gradient for a particular index is given by taking the difference between the the values either side and dividing by 2. So, the gradient of y, above, is calculated thus:
#```
#j[0] = (y[1]-y[0])/1 = (2-1)/1  = 1
#j[1] = (y[2]-y[0])/2 = (4-1)/2  = 1.5
#j[2] = (y[3]-y[1])/2 = (7-2)/2  = 2.5
#j[3] = (y[4]-y[2])/2 = (11-4)/2 = 3.5
#j[4] = (y[5]-y[3])/2 = (16-7)/2 = 4.5
#j[5] = (y[5]-y[4])/1 = (16-11)/1 = 5
#```
#You could find the minima of all the absolute values in the resulting array to find the turning points of a curve, for example. 1The array is actually called x in the example in the docs, I've changed it to y to avoid confusion.
```


Question: How can I map True/False to 1/0 in a Pandas DataFrame? I have a column in python pandas DataFrame that has boolean True/False values, but for further calculations I need 1/0 representation. Is there a quick pandas/numpy way to do that?

Response:
A succinct way to convert a single column of boolean values to a column of integers 1 or 0:
```py
df["somecolumn"] = df["somecolumn"].astype(int)
```


Question:
How to one-hot-encode from a pandas column containing a list? I would like to break down a pandas column consisting of a list of elements into as many columns as there are unique elements i.e. one-hot-encode them (with value 1 representing a given element existing in a row and 0 in the case of absence). For example, taking dataframe df
```
Col1   Col2         Col3
 C      33     [Apple, Orange, Banana]
 A      2.5    [Apple, Grape]
 B      42     [Banana]
```
I would like to convert this to: df
```
Col1   Col2   Apple   Orange   Banana   Grape
 C      33     1        1        1       0
 A      2.5    1        0        0       1
 B      42     0        0        1       0
```
How can I use pandas/sklearn to achieve this?

Response:
We can also use sklearn.preprocessing.MultiLabelBinarizer: Often we want to use sparse DataFrame for the real world data in order to save a lot of RAM. Sparse solution (for Pandas v0.25.0+)
```
from sklearn.preprocessing import MultiLabelBinarizer

mlb = MultiLabelBinarizer(sparse_output=True)

df = df.join(
            pd.DataFrame.sparse.from_spmatrix(
                mlb.fit_transform(df.pop('Col3')),
                index=df.index,
                columns=mlb.classes_))
```
result:
```

```python
df
#Output
#Col1  Col2  Apple  Banana  Grape  Orange
#0    C  33.0      1       1      0       1
#1    A   2.5      1       0      1       0
#2    B  42.0      0       1      0       0
```

```python
df.dtypes
#Output
#Col1                object
#Col2               float64
#Apple     Sparse[int32, 0]
#Banana    Sparse[int32, 0]
#Grape     Sparse[int32, 0]
#Orange    Sparse[int32, 0]
#dtype: object
```

```python
df.memory_usage()
#Output
#Index     128
#Col1       24
#Col2       24
#Apple      16    #  <--- NOTE!
#Banana     16    #  <--- NOTE!
#Grape       8    #  <--- NOTE!
#Orange      8    #  <--- NOTE!
#dtype: int64
#```
#Dense solution
#```
#mlb = MultiLabelBinarizer()
#df = df.join(pd.DataFrame(mlb.fit_transform(df.pop('Col3')),
#                          columns=mlb.classes_,
#                          index=df.index))
#```
#Result:
#```
```

```python
df
#Output
#Col1  Col2  Apple  Banana  Grape  Orange
#0    C  33.0      1       1      0       1
#1    A   2.5      1       0      1       0
#2    B  42.0      0       1      0       0
#```
```


Question:
Change values on matplotlib imshow() graph axis Say I have some input data:
```py
data = np.random.normal(loc=100, scale=10, size=(500,1,32))
hist = np.ones((32, 20)) # initialise hist
for z in range(32):
    hist[z], edges = np.histogram(data[:, 0, z], bins=np.arange(80, 122, 2))
```
I can plot it using imshow():
```py
plt.imshow(hist, cmap='Reds')
```
getting: However, the x-axis values do not match the input data (i.e. mean of 100, range from 80 to 122). Therefore, I'd like to change the x-axis to show the values in edges. I have tried:
```py
ax = plt.gca()
ax.set_xlabel([80,122]) # range of values in edges
...
# this shifts the plot so that nothing is visible
```
and
```py
ax.set_xticklabels(edges)
...
# this labels the axis but does not centre around the mean:
```
Any ideas on how I can change the axis values to reflect the input data I am using?

Response:
I would try to avoid changing the xticklabels if possible, otherwise it can get very confusing if you for example overplot your histogram with additional data. Defining the range of your grid is probably the best and with imshow it can be done by adding the extent keyword. This way the axes gets adjusted automatically. If you want to change the labels i would use set_xticks with perhaps some formatter. Altering the labels directly should be the last resort.
```
fig, ax = plt.subplots(figsize=(6,6))

ax.imshow(hist, cmap=plt.cm.Reds, interpolation='none', extent=[80,120,32,0])
ax.set_aspect(2) # you may also use am.imshow(..., aspect="auto") to restore the aspect ratio
```


Question:
Python: Resize an existing array and fill with zeros I think that my issue should be really simple, yet I can not find any help on the Internet whatsoever. I am very new to Python, so it is possible that I am missing something very obvious. I have an array, S, like this [x x x] (one-dimensional). I now create a diagonal matrix, sigma, with np.diag(S) - so far, so good. Now, I want to resize this new diagonal array so that I can multiply it by another array that I have.
```
import numpy as np
...
shape = np.shape((6, 6)) #This will be some pre-determined size
sigma = np.diag(S) #diagonalise the matrix - this works
my_sigma = sigma.resize(shape) #Resize the matrix and fill with zeros - returns "None" - why?
```
However, when I print the contents of my_sigma, I get "None". Can someone please point me in the right direction, because I can not imagine that this should be so complicated. Thanks in advance for any help! Casper Graphical: I have this:
```
[x x x]
```
I want this:
```
[x 0 0]
[0 x 0]
[0 0 x]
[0 0 0]
[0 0 0]
[0 0 0] - or some similar size, but the diagonal elements are important.
```

Response:
There is a new numpy function in version 1.7.0 numpy.pad that can do this in one-line. Like the other answers, you can construct the diagonal matrix with np.diag before the padding. The tuple ((0,N),(0,0)) used in this answer indicates the "side" of the matrix which to pad.
```
import numpy as np

A = np.array([1, 2, 3])

N = A.size
B = np.pad(np.diag(A), ((0,N),(0,0)), mode='constant')
```
B is now equal to:
```
[[1 0 0]
 [0 2 0]
 [0 0 3]
 [0 0 0]
 [0 0 0]
 [0 0 0]]
```


Question: What is python's equivalent of R's NA? What is python's equivalent of R's NA? To be more specific: R has NaN, NA, NULL, Inf and -Inf. NA is generally used when there is missing data. What is python's equivalent? How libraries such as numpy and pandas handle missing values? How does scikit-learn handle missing values? Is it different for python 2.7 and python 3?

Response:
nan in numpy is handled well with many functions:
```

```python
import numpy as np
```

```python
a = [1, np.nan, 2, 3]
```

```python
np.nanmean(a)
#Output
#2.0
```

```python
np.nansum(a)
#Output
#6.0
```

```python
np.isnan(a)
#Output
#array([False,  True, False, False], dtype=bool)
#```
```


Question:
shuffling/permutating a DataFrame in pandas What's a simple and efficient way to shuffle a dataframe in pandas, by rows or by columns? I.e. how to write a function shuffle(df, n, axis=0) that takes a dataframe, a number of shuffles n, and an axis (axis=0 is rows, axis=1 is columns) and returns a copy of the dataframe that has been shuffled n times. Edit: key is to do this without destroying the row/column labels of the dataframe. If you just shuffle df.index that loses all that information. I want the resulting df to be the same as the original except with the order of rows or order of columns different. Edit2: My question was unclear. When I say shuffle the rows, I mean shuffle each row independently. So if you have two columns a and b, I want each row shuffled on its own, so that you don't have the same associations between a and b as you do if you just re-order each row as a whole. Something like:
```
for 1...n:
  for each col in df: shuffle column
return new_df
```
But hopefully more efficient than naive looping. This does not work for me:
```
def shuffle(df, n, axis=0):
        shuffled_df = df.copy()
        for k in range(n):
            shuffled_df.apply(np.random.shuffle(shuffled_df.values),axis=axis)
        return shuffled_df

df = pandas.DataFrame({'A':range(10), 'B':range(10)})
shuffle(df, 5)
```

Response:
Use numpy's random.permuation function:
```

```python
df = pd.DataFrame({'A':range(10), 'B':range(10)})
```

```python
df
#Output
#A  B
#0  0  0
#1  1  1
#2  2  2
#3  3  3
#4  4  4
#5  5  5
#6  6  6
#7  7  7
#8  8  8
#9  9  9
```

```python
df.reindex(np.random.permutation(df.index))
#Output
#A  B
#0  0  0
#5  5  5
#6  6  6
#3  3  3
#8  8  8
#7  7  7
#9  9  9
#1  1  1
#2  2  2
#4  4  4
#```
```


Question: Dump a NumPy array into a csv file How do I dump a 2D NumPy array into a csv file in a human-readable format?

Response:
numpy.savetxt saves an array to a text file.
```
import numpy
a = numpy.asarray([ [1,2,3], [4,5,6], [7,8,9] ])
numpy.savetxt("foo.csv", a, delimiter=",")
```


Question:
Count all values in a matrix less than a value I have to count all the values in a matrix (2-d array) that are less than 200. The code I wrote down for this is:
```
za=0
p31 = numpy.asarray(o31)
for i in range(o31.size[0]):
    for j in range(o32.size[1]):
        if p31[i,j]<200:
            za=za+1
print za
```
o31 is an image and I am converting it into a matrix and then finding the values. Is there a simpler way to do this?

Response:
The numpy.where function is your friend. Because it's implemented to take full advantage of the array datatype, for large images you should notice a speed improvement over the pure python solution you provide. Using numpy.where directly will yield a boolean mask indicating whether certain values match your conditions:
```

```python
data
#Output
#array([[1, 8],
#       [3, 4]])
```

```python
numpy.where( data > 3 )
#Output
#(array([0, 1]), array([1, 1]))
#```
#And the mask can be used to index the array directly to get the actual values:
#```
```

```python
data[ numpy.where( data > 3 ) ]
#Output
#array([8, 4])
#```
#Exactly where you take it from there will depend on what form you'd like the results in.
```


Question: Python - Alternative for using numpy array as key in dictionary I'm pretty new to Python numpy. I attempted to use the numpy array as the key in the dictionary in one of my functions and then was told by the Python interpreter that the numpy array is not hashable. I've just found out that one way to work this issue around is to use repr() function to convert a numpy array to a string but it seems very expensive. Is there any better way to achieve the same effect? Update: I could create a new class to contain the numpy array, which seems to be the right way to achieve what I want. Just wondering if there is any better method. update 2: Using a class to contain data in the array and then override __hash__ function is acceptable, however, I'd prefer the solution provided by @hpaulj. Converting the array/list to a tuple fits my need in a better way as it does not require an additional class.

Response:
If you want to quickly store a numpy.ndarray as a key in a dictionary, a fast option is to use ndarray.tobytes() which will return a raw python bytes string which is immutable
```
my_array = numpy.arange(4).reshape((2,2))
my_dict = {}
my_dict[my_array.tobytes()] = None
```


Question:
Converting a 2D numpy array to a structured array I'm trying to convert a two-dimensional array into a structured array with named fields. I want each row in the 2D array to be a new record in the structured array. Unfortunately, nothing I've tried is working the way I expect. I'm starting with:
```

```python
myarray = numpy.array([("Hello",2.5,3),("World",3.6,2)])
```

```python
print myarray
#Output
#[['Hello' '2.5' '3']
# ['World' '3.6' '2']]
#```
#I want to convert to something that looks like this:
#```
```

```python
newarray = numpy.array([("Hello",2.5,3),("World",3.6,2)], dtype=[("Col1","S8"),("Col2","f8"),("Col3","i8")])
```

```python
print newarray
#Output
#[('Hello', 2.5, 3L) ('World', 3.6000000000000001, 2L)]
#```
#What I've tried:
#```
```

```python
newarray = myarray.astype([("Col1","S8"),("Col2","f8"),("Col3","i8")])
```

```python
print newarray
#Output
#[[('Hello', 0.0, 0L) ('2.5', 0.0, 0L) ('3', 0.0, 0L)]
# [('World', 0.0, 0L) ('3.6', 0.0, 0L) ('2', 0.0, 0L)]]
```

```python
newarray = numpy.array(myarray, dtype=[("Col1","S8"),("Col2","f8"),("Col3","i8")])
```

```python
print newarray
#Output
#[[('Hello', 0.0, 0L) ('2.5', 0.0, 0L) ('3', 0.0, 0L)]
# [('World', 0.0, 0L) ('3.6', 0.0, 0L) ('2', 0.0, 0L)]]
#```
#Both of these approaches attempt to convert each entry in myarray into a record with the given dtype, so the extra zeros are inserted. I can't figure out how to get it to convert each row into a record. Another attempt:
#```
```

```python
newarray = myarray.copy()
```

```python
newarray.dtype = [("Col1","S8"),("Col2","f8"),("Col3","i8")]
```

```python
print newarray
#Output
#[[('Hello', 1.7219343871178711e-317, 51L)]
# [('World', 1.7543139673493688e-317, 50L)]]
#```
#This time no actual conversion is performed. The existing data in memory is just re-interpreted as the new data type. The array that I'm starting with is being read in from a text file. The data types are not known ahead of time, so I can't set the dtype at the time of creation. I need a high-performance and elegant solution that will work well for general cases since I will be doing this type of conversion many, many times for a large variety of applications. Thanks!
```

Response:
You can "create a record array from a (flat) list of arrays" using numpy.core.records.fromarrays as follows:
```

```python
import numpy as np
```

```python
myarray = np.array([("Hello",2.5,3),("World",3.6,2)])
```

```python
print myarray
#Output
#[['Hello' '2.5' '3']
# ['World' '3.6' '2']]
```

```python
newrecarray = np.core.records.fromarrays(myarray.transpose(),
#Output
#                                             names='col1, col2, col3',
#                                             formats = 'S8, f8, i8')
```

```python
print newrecarray
#Output
#[('Hello', 2.5, 3) ('World', 3.5999999046325684, 2)]
#```
#I was trying to do something similar. I found that when numpy created a structured array from an existing 2D array (using np.core.records.fromarrays), it considered each column (instead of each row) in the 2-D array as a record. So you have to transpose it. This behavior of numpy does not seem very intuitive, but perhaps there is a good reason for it.
```


Question:
Convert string to numpy array Input:
```
mystr = "100110"
```
Desired output numpy array:
```
mynumpy == np.array([1, 0, 0, 1, 1, 0])
```
I have tried:
```
np.fromstring(mystr, dtype=int, sep='')
```
but the problem is I can't split my string to every digit of it, so numpy takes it as an one number. Any idea how to convert my string to numpy array?

Response:
list may help you do that.
```
import numpy as np

mystr = "100110"
print np.array(list(mystr))
# ['1' '0' '0' '1' '1' '0']
```
If you want to get numbers instead of string:
```
print np.array(list(mystr), dtype=int)
# [1 0 0 1 1 0]
```


Question:
Saving arrays as columns with np.savetxt I am trying to do something that is probable very simple. I would like to save three arrays to a file as columns using 'np.savetxt' When I try this
```
x = [1,2,3,4]
y = [5,6,7,8]
z = [9,10,11,12]

np.savetxt('myfile.txt', (x,y,z), fmt='%.18g', delimiter=' ', newline=os.linesep)
```
The arrays are saved like this
```
1 2 3 4
5 6 7 8
9 10 11 12
```
But what I wold like is this
```
1 5 9
2 6 10
3 7 11
4 8 12
```

Response:
Use numpy.c_[]:
```
np.savetxt('myfile.txt', np.c_[x,y,z])
```


Question:
Efficiently detect sign-changes in python I want to do exactly what this guy did: Python - count sign changes However I need to optimize it to run super fast. In brief I want to take a time series and tell every time it crosses crosses zero (changes sign). I want to record the time in between zero crossings. Since this is real data (32 bit float) I doubt I'll every have a number which is exactly zero, so that is not important. I currently have a timing program in place so I'll time your results to see who wins. My solution gives (micro seconds):
```
open data       8384
sign data       8123
zcd data        415466
```
As you can see the zero-crossing detector is the slow part. Here's my code.
```
import numpy, datetime

class timer():
    def __init__(self):
        self.t0 = datetime.datetime.now()
        self.t = datetime.datetime.now()
    def __call__(self,text='unknown'):
        print text,'\t',(datetime.datetime.now()-self.t).microseconds
        self.t=datetime.datetime.now()

def zcd(data,t):
    sign_array=numpy.sign(data)
    t('sign data')
    out=[]
    current = sign_array[0]
    count=0
    for i in sign_array[1:]:
        if i!=current:
            out.append(count)
            current=i
            count=0
        else: count+=1
    t('zcd data')
    return out

def main():
    t = timer()
    data = numpy.fromfile('deci.dat',dtype=numpy.float32)
    t('open data')
    zcd(data,t)

if __name__=='__main__':
    main()
```

Response:
What about:
```
import numpy
a = [1, 2, 1, 1, -3, -4, 7, 8, 9, 10, -2, 1, -3, 5, 6, 7, -10]
zero_crossings = numpy.where(numpy.diff(numpy.sign(a)))[0]
```
Output:
```
> zero_crossings
array([ 3,  5,  9, 10, 11, 12, 15])
```
I.e., zero_crossings will contain the indices of elements before which a zero crossing occurs. If you want the elements after, just add 1 to that array.


Question:
Concatenating two one-dimensional NumPy arrays How do I concatenate two one-dimensional arrays in NumPy? I tried numpy.concatenate:
```
import numpy as np
a = np.array([1, 2, 3])
b = np.array([4, 5])
np.concatenate(a, b)
```
But I get an error: TypeError: only length-1 arrays can be converted to Python scalars

Response:
Use:
```
np.concatenate([a, b])
```
The arrays you want to concatenate need to be passed in as a sequence, not as separate arguments. From the NumPy documentation: numpy.concatenate((a1, a2, ...), axis=0) Join a sequence of arrays together. It was trying to interpret your b as the axis parameter, which is why it complained it couldn't convert it into a scalar.


Question: Java equivalent for the Numpy multi-dimensional object After using it for a while, I really like the Numpy multi-dimensional array. It's helpful to write algorithms with a concise yet readable and fairly general code. I wish to have the same thing in Java. Before coding a multi-dimensional array with a Numpy-like API myself, is there such a thing already ? [PS] I searched a bit, did not see

Response: The OP is from 2011. So as of end of 2015 I would like mention that there is a new kid in town which claims to be numpy for java -> nd4j]. The nice thing is that nd4j is an abstraction layer on top of different libraries like blas. Depending on the size of your matrices there are underlying implementations twice as fast as numpy or jblas. And your code is real platform independent.


Question:
Combine 3 separate numpy arrays to an RGB image in Python So I have a set of data which I am able to convert to form separate numpy arrays of R, G, B bands. Now I need to combine them to form an RGB image. I tried 'Image' to do the job but it requires 'mode' to be attributed. I tried to do a trick. I would use Image.fromarray() to take the array to image but it attains 'F' mode by default when Image.merge requires 'L' mode images to merge. If I would declare the attribute of array in fromarray() to 'L' at first place, all the R G B images become distorted. But, if I save the images and then open them and then merge, it works fine. Image reads the image with 'L' mode. Now I have two issues. First, I dont think it is an elegant way of doing the work. So if anyone knows the better way of doing it, please tell Secondly, Image.SAVE is not working properly. Following are the errors I face:
```

```python
Image.SAVE(imagefile, 'JPEG')
----------------------------------------------------------------------------------

TypeError                                 Traceback (most recent call last)

/media/New Volume/Documents/My own works/ISAC/SAMPLES/<ipython console> in <module>()

TypeError: 'dict' object is not callable
```
Please suggest solutions. And please mind that the image is around 4000x4000 size array.
```

Response:
```
rgb = np.dstack((r,g,b))  # stacks 3 h x w arrays -> h x w x 3
```
To also convert floats 0 .. 1 to uint8 s,
```
rgb_uint8 = (np.dstack((r,g,b)) * 255.999) .astype(np.uint8)  # right, Janna, not 256
```


Question: Conjugate transpose operator ".H" in numpy It is very convenient in numpy to use the .T attribute to get a transposed version of an ndarray. However, there is no similar way to get the conjugate transpose. Numpy's matrix class has the .H operator, but not ndarray. Because I like readable code, and because I'm too lazy to always write .conj().T, I would like the .H property to always be available to me. How can I add this feature? Is it possible to add it so that it is brainlessly available every time numpy is imported? (A similar question could by asked about the .I inverse operator.)

Response:
You can subclass the ndarray object like:
```
from numpy import ndarray

class myarray(ndarray):
    @property
    def H(self):
        return self.conj().T
```
such that:
```
a = np.random.rand(3, 3).view(myarray)
a.H
```
will give you the desired behavior. Edit: As suggested by @slek120, you can force to transpose only the last 2 axes with:
```
self.swapaxes(-2, -1).conj()
```
instead of self.conj().T.


Question:
Fitting a closed curve to a set of points I have a set of points pts which form a loop and it looks like this: This is somewhat similar to 31243002, but instead of putting points in between pairs of points, I would like to fit a smooth curve through the points (coordinates are given at the end of the question), so I tried something similar to scipy documentation on Interpolation:
```
values = pts
tck = interpolate.splrep(values[:,0], values[:,1], s=1)
xnew = np.arange(2,7,0.01)
ynew = interpolate.splev(xnew, tck, der=0)
```
but I get this error: ValueError: Error on input data Is there any way to find such a fit? Coordinates of the points:
```
pts = array([[ 6.55525 ,  3.05472 ],
   [ 6.17284 ,  2.802609],
   [ 5.53946 ,  2.649209],
   [ 4.93053 ,  2.444444],
   [ 4.32544 ,  2.318749],
   [ 3.90982 ,  2.2875  ],
   [ 3.51294 ,  2.221875],
   [ 3.09107 ,  2.29375 ],
   [ 2.64013 ,  2.4375  ],
   [ 2.275444,  2.653124],
   [ 2.137945,  3.26562 ],
   [ 2.15982 ,  3.84375 ],
   [ 2.20982 ,  4.31562 ],
   [ 2.334704,  4.87873 ],
   [ 2.314264,  5.5047  ],
   [ 2.311709,  5.9135  ],
   [ 2.29638 ,  6.42961 ],
   [ 2.619374,  6.75021 ],
   [ 3.32448 ,  6.66353 ],
   [ 3.31582 ,  5.68866 ],
   [ 3.35159 ,  5.17255 ],
   [ 3.48482 ,  4.73125 ],
   [ 3.70669 ,  4.51875 ],
   [ 4.23639 ,  4.58968 ],
   [ 4.39592 ,  4.94615 ],
   [ 4.33527 ,  5.33862 ],
   [ 3.95968 ,  5.61967 ],
   [ 3.56366 ,  5.73976 ],
   [ 3.78818 ,  6.55292 ],
   [ 4.27712 ,  6.8283  ],
   [ 4.89532 ,  6.78615 ],
   [ 5.35334 ,  6.72433 ],
   [ 5.71583 ,  6.54449 ],
   [ 6.13452 ,  6.46019 ],
   [ 6.54478 ,  6.26068 ],
   [ 6.7873  ,  5.74615 ],
   [ 6.64086 ,  5.25269 ],
   [ 6.45649 ,  4.86206 ],
   [ 6.41586 ,  4.46519 ],
   [ 5.44711 ,  4.26519 ],
   [ 5.04087 ,  4.10581 ],
   [ 4.70013 ,  3.67405 ],
   [ 4.83482 ,  3.4375  ],
   [ 5.34086 ,  3.43394 ],
   [ 5.76392 ,  3.55156 ],
   [ 6.37056 ,  3.8778  ],
   [ 6.53116 ,  3.47228 ]])
```

Response:
Actually, you were not far from the solution in your question. Using scipy.interpolate.splprep for parametric B-spline interpolation would be the simplest approach. It also natively supports closed curves, if you provide the per=1 parameter,
```
import numpy as np
from scipy.interpolate import splprep, splev
import matplotlib.pyplot as plt

# define pts from the question

tck, u = splprep(pts.T, u=None, s=0.0, per=1)
u_new = np.linspace(u.min(), u.max(), 1000)
x_new, y_new = splev(u_new, tck, der=0)

plt.plot(pts[:,0], pts[:,1], 'ro')
plt.plot(x_new, y_new, 'b--')
plt.show()
```
Fundamentally, this approach not very different from the one in @Joe Kington's answer. Although, it will probably be a bit more robust, because the equivalent of the i vector is chosen, by default, based on the distances between points and not simply their index (see splprep documentation for the u parameter).


Question:
Numpy mean of nonzero values I have a matrix of size N*M and I want to find the mean value for each row. The values are from 1 to 5 and entries that do not have any value are set to 0. However, when I want to find the mean using the following method, it gives me the wrong mean as it also counts the entries that have value of 0.
```
matrix_row_mean= matrix.mean(axis=1)
```
How can I get the mean of only nonzero values?

Response:
Get the count of non-zeros in each row and use that for averaging the summation along each row. Thus, the implementation would look something like this -
```
np.true_divide(matrix.sum(1),(matrix!=0).sum(1))
```
If you are on an older version of NumPy, you can use float conversion of the count to replace np.true_divide, like so -
```
matrix.sum(1)/(matrix!=0).sum(1).astype(float)
```
Sample run -
```

```python
matrix
#Output
#array([[0, 0, 1, 0, 2],
#       [1, 0, 0, 2, 0],
#       [0, 1, 1, 0, 0],
#       [0, 2, 2, 2, 2]])
```

```python
np.true_divide(matrix.sum(1),(matrix!=0).sum(1))
#Output
#array([ 1.5,  1.5,  1. ,  2. ])
#```
#Another way to solve the problem would be to replace zeros with NaNs and then use np.nanmean, which would ignore those NaNs and in effect those original zeros, like so -
#```
#np.nanmean(np.where(matrix!=0,matrix,np.nan),1)
#```
#From performance point of view, I would recommend the first approach.
```


Question:
Numpy slice of arbitrary dimensions I would like to slice a numpy array to obtain the i-th index in the last dimension. For a 3D array, this would be:
```
slice = myarray[:, :, i]
```
But I am writing a function where I can take an array of arbitrary dimensions, so for a 4D array I'd need myarray[:, :, :, i], and so on. Is there a way I can obtain this slice for any array without explicitly having to write the array dimensions?

Response:
There is ... or Ellipsis, which does exactly this:
```
slice = myarray[..., i]
```
Ellipsis is the python object, if you should want to use it outside the square bracket notation.


Question:
'invalid value encountered in double_scalars' warning, possibly numpy As I run my code I get these warnings, always in groups of four, sporadically. I have tried to locate the source by placing debug messages before and after certain statements to pin-point its origin.
```
Warning: invalid value encountered in double_scalars
Warning: invalid value encountered in double_scalars
Warning: invalid value encountered in double_scalars
Warning: invalid value encountered in double_scalars
```
Is this is a Numpy warning, and what is a double scalar? From Numpy I use
```
min(), argmin(), mean() and random.randn()
```
I also use Matplotlib

Response: It looks like a floating-point calculation error. Check the numpy.seterr function to get more information about where it happens.


Question:
How to round a numpy array? I have a numpy array, something like below:
```
data = np.array([  1.60130719e-01,   9.93827160e-01,   3.63108206e-04])
```
and I want to round each element to two decimal places. How can I do so?

Response:
Numpy provides two identical methods to do this. Either use
```
np.round(data, 2)
```
or
```
np.around(data, 2)
```
as they are equivalent. See the documentation for more information. Examples:
```py

```python
import numpy as np
```

```python
a = np.array([0.015, 0.235, 0.112])
```

```python
np.round(a, 2)
#Output
#array([0.02, 0.24, 0.11])
```

```python
np.around(a, 2)
#Output
#array([0.02, 0.24, 0.11])
```

```python
np.round(a, 1)
#Output
#array([0. , 0.2, 0.1])
#```
```


Question:
Numpy: Joining structured arrays? Input I have many numpy structured arrays in a list like this example:
```
import numpy

a1 = numpy.array([(1, 2), (3, 4), (5, 6)], dtype=[('x', int), ('y', int)])

a2 = numpy.array([(7,10), (8,11), (9,12)], dtype=[('z', int), ('w', float)])

arrays = [a1, a2]
```
Desired Output What is the correct way to join them all together to create a unified structured array like the following?
```
desired_result = numpy.array([(1, 2, 7, 10), (3, 4, 8, 11), (5, 6, 9, 12)],
                             dtype=[('x', int), ('y', int), ('z', int), ('w', float)])
```
Current Approach This is what I'm currently using, but it is very slow, so I suspect there must be a more efficent way.
```
from numpy.lib.recfunctions import append_fields

def join_struct_arrays(arrays):
    for array in arrays:
        try:
            result = append_fields(result, array.dtype.names, [array[name] for name in array.dtype.names], usemask=False)
        except NameError:
            result = array

    return result
```

Response:
You can also use the function merge_arrays of numpy.lib.recfunctions:
```
import numpy.lib.recfunctions as rfn
rfn.merge_arrays(arrays, flatten = True, usemask = False)

Out[52]:
array([(1, 2, 7, 10.0), (3, 4, 8, 11.0), (5, 6, 9, 12.0)],
     dtype=[('x', '<i4'), ('y', '<i4'), ('z', '<i4'), ('w', '<f8')])
```


Question:
how to convert 2d list to 2d numpy array? I have a 2D list something like
```
a = [[1, 2, 3], [4, 5, 6], [7, 8, 9]]
```
and I want to convert it to a 2d numpy array. Can we do it without allocating memory like
```
numpy.zeros((3,3))
```
and then storing values to it?

Response:
Just pass the list to np.array:
```
a = np.array(a)
```
You can also take this opportunity to set the dtype if the default is not what you desire.
```
a = np.array(a, dtype=...)
```


Question:
Showing ValueError: shapes (1,3) and (1,3) not aligned: 3 (dim 1) != 1 (dim 0) I am trying to use the following matrices and perform a dot product as shown in the code. I checked the size of the matrix and it is (1, 3) but it is throwing me error for the dot product.
```py
import numpy as np

b0 = np.matrix(np.random.rand(3))
n1 = np.cross(b0, b0)
print(np.shape(n1))

y = np.dot(n1, n1)
```

Response:
By converting the matrix to array by using
```
n12 = np.squeeze(np.asarray(n2))

X12 = np.squeeze(np.asarray(x1))
```
solved the issue.


Question:
Is there a NumPy function to return the first index of something in an array? I know there is a method for a Python list to return the first index of something:
```

```python
xs = [1, 2, 3]
```

```python
xs.index(2)
#Output
#1
#```
#Is there something like that for NumPy arrays?
```

Response:
Yes, given an array, array, and a value, item to search for, you can use np.where as:
```
itemindex = numpy.where(array == item)
```
The result is a tuple with first all the row indices, then all the column indices. For example, if an array is two dimensions and it contained your item at two locations then
```
array[itemindex[0][0]][itemindex[1][0]]
```
would be equal to your item and so would be:
```
array[itemindex[0][1]][itemindex[1][1]]
```


Question:
Concatenate two numpy arrays in the 4th dimension I have two numpy arrays with three dimensions (3 x 4 x 5) and I want to concatenate them so the result has four dimensions (3 x 4 x 5 x 2). In Matlab, this can be done with cat(4, a, b), but not in Numpy. For example:
```
a = ones((3,4,5))
b = ones((3,4,5))
c = concatenate((a,b), axis=3) # error!
```
To clarify, I wish c[:,:,:,0] and c[:,:,:,1] to correspond to the original two arrays.

Response:
Here you go:
```
import numpy as np
a = np.ones((3,4,5))
b = np.ones((3,4,5))
c = np.concatenate((a[...,np.newaxis],b[...,np.newaxis]),axis=3)
```


Question:
How to change a single value in a NumPy array? I want to change a single element of an array. For example, I have:
```
A = np.array([1,2,3,4],
             [5,6,7,8],
             [9,10,11,12],
             [13,14,15,16])
```
I want to relace A[2][1] = 10 with A[2][1] = 150. How can I do it?

Response:
Is this what you are after? Just index the element and assign a new value.
```
A[2,1]=150

A
Out[345]:
array([[ 1,  2,  3,  4],
       [ 5,  6,  7,  8],
       [ 9, 150, 11, 12],
       [13, 14, 15, 16]])
```


Question:
numpy uint8 pixel wrapping solution For an image processing class, I am doing point operations on monochrome images. Pixels are uint8 [0,255]. numpy uint8 will wrap. For example, 235+30 = 9. I need the pixels to saturate (max=255) or truncate (min=0) instead of wrapping. My solution uses int32 pixels for the point math then converts to uint8 to save the image. Is this the best way? Or is there a faster way?
```
#!/usr/bin/python

import sys
import numpy as np
import Image

def to_uint8( data ) :
    # maximum pixel
    latch = np.zeros_like( data )
    latch[:] = 255
    # minimum pixel
    zeros = np.zeros_like( data )

    # unrolled to illustrate steps
    d = np.maximum( zeros, data )
    d = np.minimum( latch, d )

    # cast to uint8
    return np.asarray( d, dtype="uint8" )

infilename=sys.argv[1]
img = Image.open(infilename)
data32 = np.asarray( img, dtype="int32")
data32 += 30
data_u8 = to_uint8( data32 )
outimg = Image.fromarray( data_u8, "L" )
outimg.save( "out.png" )
```
Input image: Output image:

Response:
Use numpy.clip:
```
import numpy as np
np.clip(data32, 0, 255, out=data32)
data_u8 = data32.astype('uint8')
```
Note that you can also brighten images without numpy this way:
```
import ImageEnhance
enhancer = ImageEnhance.Brightness(img)
outimg = enhancer.enhance(1.2)
outimg.save('out.png')
```


Question:
Python/Numpy MemoryError Basically, I am getting a memory error in python when trying to perform an algebraic operation on a numpy matrix. The variable u, is a large matrix of double (in the failing case its a 288x288x156 matrix of doubles. I only get this error in this huge case, but I am able to do this on other large matrices, just not this big). Here is the Python error:
```
Traceback (most recent call last):

 File "S:\3D_Simulation_Data\Patient SPM Segmentation\20 pc
t perim erosion flattop\SwSim.py", line 121, in __init__
   self.mainSimLoop()

 File "S:\3D_Simulation_Data\Patient SPM Segmentation\20 pc
t perim erosion flattop\SwSim.py", line 309, in mainSimLoop
   u = solver.solve_cg(u,b,tensors,param,fdHold,resid) # Solve the left hand si
de of the equation Au=b with conjugate gradient method to approximate u

 File "S:\3D_Simulation_Data\Patient SPM Segmentation\20 pc
t perim erosion flattop\conjugate_getb.py", line 47, in solv
e_cg

u = u + alpha*p

MemoryError
```
u = u + alpha*p is the line of code that fails. alpha is just a double, while u and r are the large matrices described above (both of the same size). I don't know that much about memory errors especially in Python. Any insight/tips into solving this would be very appreciated! Thanks

Response:
Rewrite to
```
p *= alpha
u += p
```
and this will use much less memory. Whereas p = p*alpha allocates a whole new matrix for the result of p*alpha and then discards the old p; p*= alpha does the same thing in place. In general, with big matrices, try to use op= assignment.


Question:
Good ways to "expand" a numpy ndarray? Are there good ways to "expand" a numpy ndarray? Say I have an ndarray like this:
```
[[1 2]
 [3 4]]
```
And I want each row to contains more elements by filling zeros:
```
[[1 2 0 0 0]
 [3 4 0 0 0]]
```
I know there must be some brute-force ways to do so (say construct a bigger array with zeros then copy elements from old smaller arrays), just wondering are there pythonic ways to do so. Tried numpy.reshape but didn't work:
```
import numpy as np
a = np.array([[1, 2], [3, 4]])
np.reshape(a, (2, 5))
```
Numpy complains that: ValueError: total size of new array must be unchanged

Response:
You can use numpy.pad, as follows:
```

```python
import numpy as np
```

```python
a=[[1,2],[3,4]]
```

```python
np.pad(a, ((0,0),(0,3)), mode='constant', constant_values=0)
#Output
#array([[1, 2, 0, 0, 0],
#       [3, 4, 0, 0, 0]])
#```
#Here np.pad says, "Take the array a and add 0 rows above it, 0 rows below it, 0 columns to the left of it, and 3 columns to the right of it. Fill these columns with a constant specified by constant_values".
```


Question:
remove zero lines 2-D numpy array I run a qr factorization in numpy which returns a list of ndarrays, namely Qand R:
```

```python
[q,r] = np.linalg.qr(np.array([1,0,0,0,1,1,1,1,1]).reshape(3,3))
#Output
#```
#R is a two-dimensional array, having pivoted zero-lines at the bottom (even proved for all examples in my test set):
#```
```

```python
print r
#Output
#[[ 1.41421356  0.70710678  0.70710678]
# [ 0.          1.22474487  1.22474487]
# [ 0.          0.          0.        ]]
#```
#. Now, I want to divide R in two matrices R_~:
#```
#[[ 1.41421356  0.70710678  0.70710678]
# [ 0.          1.22474487  1.22474487]]
#```
#and R_0:
#```
#[[ 0.          0.          0.        ]]
#```
#(extracting all zero-lines). It seems to be close to this solution: deleting rows in numpy array. EDIT: Even more interesting: np.linalg.qr() returns a n x n-matrix. Not, what I would have expected:
#```
#A := n x m
#Q := n x m
#R := n x m
#```
```

Response:
Use np.all with an axis argument:
```

```python
r[np.all(r == 0, axis=1)]
#Output
#array([[ 0.,  0.,  0.]])
```

```python
r[~np.all(r == 0, axis=1)]
#Output
#array([[-1.41421356, -0.70710678, -0.70710678],
#       [ 0.        , -1.22474487, -1.22474487]])
#```
```


Question:
Iterating through a multidimensional array in Python I have created a multidimensional array in Python like this:
```
self.cells = np.empty((r,c),dtype=np.object)
```
Now I want to iterate through all elements of my twodimensional array, and I do not care about the order. How do I achieve this?

Response:
It's clear you're using numpy. With numpy you can just do:
```
for cell in self.cells.flat:
    do_somethin(cell)
```


Question:
In-place type conversion of a NumPy array Given a NumPy array of int32, how do I convert it to float32 in place? So basically, I would like to do
```
a = a.astype(numpy.float32)
```
without copying the array. It is big. The reason for doing this is that I have two algorithms for the computation of a. One of them returns an array of int32, the other returns an array of float32 (and this is inherent to the two different algorithms). All further computations assume that a is an array of float32. Currently I do the conversion in a C function called via ctypes. Is there a way to do this in Python?

Response:
Update: This function only avoids copy if it can, hence this is not the correct answer for this question. unutbu's answer is the right one.
```
a = a.astype(numpy.float32, copy=False)
```
numpy astype has a copy flag. Why shouldn't we use it ?


Question:
How to check if all values in the columns of a numpy matrix are the same? I want to check if all values in the columns of a numpy array/matrix are the same. I tried to use reduce of the ufunc equal, but it doesn't seem to work in all cases:
```

```python
a = np.array([[1,1,0],[1,-1,0],[1,0,0],[1,1,0]])
```

```python
a
#Output
#array([[ 1,  1,  0],
#       [ 1, -1,  0],
#       [ 1,  0,  0],
#       [ 1,  1,  0]])
```

```python
np.equal.reduce(a)
#Output
#array([ True, False,  True], dtype=bool)
```

```python
a = np.array([[1,1,0],[1,0,0],[1,0,0],[1,1,0]])
```

```python
a
#Output
#array([[1, 1, 0],
#       [1, 0, 0],
#       [1, 0, 0],
#       [1, 1, 0]])
```

```python
np.equal.reduce(a)
#Output
#array([ True,  True,  True], dtype=bool)
#```
#Why does the middle column in the second case also evaluate to True, while it should be False? Thanks for any help!
```

Response:
```

```python
a
#Output
#array([[1, 1, 0],
#       [1, 0, 0],
#       [1, 0, 0],
#       [1, 1, 0]])
#```
#Compare each value to the corresponding value in the first row:
#```
```

```python
a == a[0,:]
#Output
#array([[ True,  True,  True],
#       [ True, False,  True],
#       [ True, False,  True],
#       [ True,  True,  True]], dtype=bool)
#```
#A column shares a common value if all the values in that column are True:
#```
```

```python
np.all(a == a[0,:], axis = 0)
#Output
#array([ True, False,  True], dtype=bool)
#```
#The problem with np.equal.reduce can be seen by micro-analyzing what happens when it is applied to [1, 0, 0, 1]:
#```
```

```python
np.equal.reduce([1, 0, 0, 1])
#Output
#True
#```
#The first two items, 1 and 0 are tested for equality and the result is False:
#```
```

```python
np.equal.reduce([False, 0, 1])
#Output
#True
#```
#Now False and 0 are tested for equality and the result is True:
#```
```

```python
np.equal.reduce([True, 1])
#Output
#True
#```
#But True and 1 are equal, so the total result is True, which is not the desired outcome. The problem is that reduce tries to accumulate the result "locally", while we want a "global" test like np.all.
```


Question:
NumPy first and last element from array I am trying to dynamically get the first and last element from an array. So, let us suppose the array has 6 elements.
```
test = [1,23,4,6,7,8]
```
If I am trying to get the first and last = 1,8, 23,7 and 4,6. Is there a way to get elements in this order? I looked at a couple of questions Link Link2. I took help of these links and I came up with this prototype..
```
#!/usr/bin/env python

import numpy

test = [1,23,4,6,7,8]
test1 = numpy.array([1,23,4,6,7,8])
len_test = len(test)
first_list = [0,1,2]
len_first = len(first_list)
second_list = [-1,-2,-3]
len_second = len(second_list)

for a in range(len_first):
        print numpy.array(test)[[first_list[a] , second_list[a]]]
        print test1[[first_list[a], second_list[a]]]
```
But this prototype won't scale for if you have more than 6 elements. So, I was wondering if there is way to dynamically get the pair of elements. Thanks!

Response:
I ended here, because I googled for "python first and last element of array", and found everything else but this. So here's the answer to the title question:
```
a = [1,2,3]
a[0] # first element (returns 1)
a[-1] # last element (returns 3)
```


Question:
Numpy extract submatrix I'm pretty new in numpy and I am having a hard time understanding how to extract from a np.array a sub matrix with defined columns and rows:
```
Y = np.arange(16).reshape(4,4)
```
If I want to extract columns/rows 0 and 3, I should have:
```
[[0 3]
 [12 15]]
```
I tried all the reshape functions...but cannot figure out how to do this. Any ideas?

Response:
Give np.ix_ a try:
```
Y[np.ix_([0,3],[0,3])]
```
This returns your desired result:
```

```python
Y = np.arange(16).reshape(4,4)
```

```python
Y[np.ix_([0,3],[0,3])]
#Output
#array([[ 0,  3],
#       [12, 15]])
#```
```


Question:
Multiplying across in a numpy array I'm trying to multiply each of the terms in a 2D array by the corresponding terms in a 1D array. This is very easy if I want to multiply every column by the 1D array, as shown in the numpy.multiply function. But I want to do the opposite, multiply each term in the row. In other words I want to multiply:
```
[1,2,3]   [0]
[4,5,6] * [1]
[7,8,9]   [2]
```
and get
```
[0,0,0]
[4,5,6]
[14,16,18]
```
but instead I get
```
[0,2,6]
[0,5,12]
[0,8,18]
```
Does anyone know if there's an elegant way to do that with numpy? Thanks a lot, Alex

Response:
Normal multiplication like you showed:
```

```python
import numpy as np
```

```python
m = np.array([[1,2,3],[4,5,6],[7,8,9]])
```

```python
c = np.array([0,1,2])
```

```python
m * c
#Output
#array([[ 0,  2,  6],
#       [ 0,  5, 12],
#       [ 0,  8, 18]])
#```
#If you add an axis, it will multiply the way you want:
#```
```

```python
m * c[:, np.newaxis]
#Output
#array([[ 0,  0,  0],
#       [ 4,  5,  6],
#       [14, 16, 18]])
#```
#You could also transpose twice:
#```
```

```python
(m.T * c).T
#Output
#array([[ 0,  0,  0],
#       [ 4,  5,  6],
#       [14, 16, 18]])
#```
```


Question:
Downsample a 1D numpy array I have a 1-d numpy array which I would like to downsample. Any of the following methods are acceptable if the downsampling raster doesn't perfectly fit the data: overlap downsample intervals convert whatever number of values remains at the end to a separate downsampled value interpolate to fit raster basically if I have
```
1 2 6 2 1
```
and I am downsampling by a factor of 3, all of the following are ok:
```
3 3

3 1.5
```
or whatever an interpolation would give me here. I'm just looking for the fastest/easiest way to do this. I found scipy.signal.decimate, but that sounds like it decimates the values (takes them out as needed and only leaves one in X). scipy.signal.resample seems to have the right name, but I do not understand where they are going with the whole fourier thing in the description. My signal is not particularly periodic. Could you give me a hand here? This seems like a really simple task to do, but all these functions are quite intricate...

Response:
In the simple case where your array's size is divisible by the downsampling factor (R), you can reshape your array, and take the mean along the new axis:
```
import numpy as np
a = np.array([1.,2,6,2,1,7])
R = 3
a.reshape(-1, R)
=> array([[ 1.,  2.,  6.],
         [ 2.,  1.,  7.]])

a.reshape(-1, R).mean(axis=1)
=> array([ 3.        ,  3.33333333])
```
In the general case, you can pad your array with NaNs to a size divisible by R, and take the mean using scipy.nanmean.
```
import math, scipy
b = np.append(a, [ 4 ])
b.shape
=> (7,)
pad_size = math.ceil(float(b.size)/R)*R - b.size
b_padded = np.append(b, np.zeros(pad_size)*np.NaN)
b_padded.shape
=> (9,)
scipy.nanmean(b_padded.reshape(-1,R), axis=1)
=> array([ 3.        ,  3.33333333,  4.])
```


Question: Defining a white noise process in Python I need to draw samples from a white noise process in order to implement a particular integral numerically. How do I generate this with Python (i.e., numpy, scipy, etc.)?

Response:
You can achieve this through the numpy.random.normal function, which draws a given number of samples from a Gaussian distribution.
```
import numpy
import matplotlib.pyplot as plt

mean = 0
std = 1
num_samples = 1000
samples = numpy.random.normal(mean, std, size=num_samples)

plt.plot(samples)
plt.show()
```


Question: NumPy: function for simultaneous max() and min() numpy.amax() will find the max value in an array, and numpy.amin() does the same for the min value. If I want to find both max and min, I have to call both functions, which requires passing over the (very big) array twice, which seems slow. Is there a function in the numpy API that finds both max and min with only a single pass through the data?

Response: Is there a function in the numpy API that finds both max and min with only a single pass through the data? No. At the time of this writing, there is no such function. (And yes, if there were such a function, its performance would be significantly better than calling numpy.amin() and numpy.amax() successively on a large array.)


Question: Saving a Numpy array as an image I have a matrix in the type of a Numpy array. How would I write it to disk it as an image? Any format works (png, jpeg, bmp...). One important constraint is that PIL is not present.

Response:
Using PIL, save a NumPy array arr by doing:
```
from PIL import Image
im = Image.fromarray(arr)
im.save("your_file.jpeg")
```
See the docs for available data formats, including JPEG, PNG, and so on.


Question:
How can I check whether a numpy array is empty or not? How can I check whether a numpy array is empty or not? I used the following code, but this fails if the array contains a zero.
```
if not self.Definition.all():
```
Is this the solution?
```
if self.Definition == array([]):
```

Response:
You can always take a look at the .size attribute. It is defined as an integer, and is zero (0) when there are no elements in the array:
```
import numpy as np
a = np.array([])

if a.size == 0:
    # Do something when `a` is empty
```


Question: NumPy "record array" or "structured array" or "recarray" What, if any, is the difference between a NumPy "structured array", a "record array" and a "recarray"? The NumPy docs imply that the first two are the same: if they are, which is the prefered term for this object? The same documentation says (at the bottom of the page): You can find some more information on recarrays and structured arrays (including the difference between the two) here. Is there a simple explanation of this difference?

Response:
The answer in a nutshell is you should generally use structured arrays rather than recarrays because structured arrays are faster and the only advantage of recarrays is to allow you to write arr.x instead of arr['x'], which can be a convenient shortcut, but also error prone if your column names conflict with numpy methods/attributes. See this excerpt from @jakevdp's book for a more detailed explanation. In particular, he notes that simply accessing columns of structured arrays can be around 20x to 30x faster than accessing columns of recarrays. However, his example uses a very small dataframe with just 4 rows and doesn't perform any standard operations. For simple operations on larger dataframes, the difference is likely to be much smaller although structured arrays are still faster. For example, here's are a structured and record array each with 10,000 rows (code to create the arrays from a dataframe borrowed from @jpp answer here).
```
n = 10_000
df = pd.DataFrame({ 'x':np.random.randn(n) })
df['y'] = df.x.astype(int)

rec_array = df.to_records(index=False)

s = df.dtypes
struct_array = np.array([tuple(x) for x in df.values], dtype=list(zip(s.index, s)))
```
If we do a standard operation such as multiplying a column by 2 it's about 50% faster for the structured array:
```
%timeit struct_array['x'] * 2
9.18 µs ± 88.9 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

%timeit rec_array.x * 2
14.2 µs ± 314 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
```