Question: numpy, how do I find total rows in a 2D array and total column in a 1D array Hi apologies for the newbie question, but I'm wondering if someone can help me with two questions. Example say I have this, [[1,2,3],[10,2,2]] I have two questions. How do I find total columns: How do I find total rows: thank you very much. A

Response:
Getting number of rows and columns is as simple as:
```

```python
import numpy as np
```

```python
a=np.array([[1,2,3],[10,2,2]])
```

```python
num_rows, num_cols = a.shape
```

```python
print num_rows, num_cols
#Output
#2 3
#```
```


Question:
vectorize conditional assignment in pandas dataframe If I have a dataframe df with column x and want to create column y based on values of x using this in pseudo code:
```
if df['x'] < -2 then df['y'] = 1
else if df['x'] > 2 then df['y'] = -1
else df['y'] = 0
```
How would I achieve this? I assume np.where is the best way to do this but not sure how to code it correctly.

Response:
One simple method would be to assign the default value first and then perform 2 loc calls:
```

```python
df = pd.DataFrame({'x':[0,-3,5,-1,1]})
df
#Output
#x
#0  0
#1 -3
#2  5
#3 -1
#4  1
```

```python
df['y'] = 0
df.loc[df['x'] < -2, 'y'] = 1
df.loc[df['x'] > 2, 'y'] = -1
df
#Output
#x  y
#0  0  0
#1 -3  1
#2  5 -1
#3 -1  0
#4  1  0
#```
#If you wanted to use np.where then you could do it with a nested np.where:
#```
```

```python
df['y'] = np.where(df['x'] < -2 , 1, np.where(df['x'] > 2, -1, 0))
df
#Output
#x  y
#0  0  0
#1 -3  1
#2  5 -1
#3 -1  0
#4  1  0
#```
#So here we define the first condition as where x is less than -2, return 1, then we have another np.where which tests the other condition where x is greater than 2 and returns -1, otherwise return 0 timings
#```
```

```python
%timeit df['y'] = np.where(df['x'] < -2 , 1, np.where(df['x'] > 2, -1, 0))

1000 loops, best of 3: 1.79 ms per loop
```

```python
%%timeit
df['y'] = 0
df.loc[df['x'] < -2, 'y'] = 1
df.loc[df['x'] > 2, 'y'] = -1

100 loops, best of 3: 3.27 ms per loop
```
So for this sample dataset the np.where method is twice as fast
```


Question:
NumPy append vs concatenate What is the difference between NumPy append and concatenate? My observation is that concatenate is a bit faster and append flattens the array if axis is not specified.
```

```python
print a
[[1 2]
 [3 4]
 [5 6]
 [5 6]
 [1 2]
 [3 4]
 [5 6]
 [5 6]
 [1 2]
 [3 4]
 [5 6]
 [5 6]
 [5 6]]
```

```python
print b
[[1 2]
 [3 4]
 [5 6]
 [5 6]
 [1 2]
 [3 4]
 [5 6]
 [5 6]
 [5 6]]
```

```python
timeit -n 10000 -r 5 np.concatenate((a, b))
10000 loops, best of 5: 2.05 µs per loop
```

```python
timeit -n 10000 -r 5 np.append(a, b, axis = 0)
10000 loops, best of 5: 2.41 µs per loop
```

```python
np.concatenate((a, b))
#Output
#array([[1, 2],
#       [3, 4],
#       [5, 6],
#       [5, 6],
#       [1, 2],
#       [3, 4],
#       [5, 6],
#       [5, 6],
#       [1, 2],
#       [3, 4],
#       [5, 6],
#       [5, 6],
#       [5, 6],
#       [1, 2],
#       [3, 4],
#       [5, 6],
#       [5, 6],
#       [1, 2],
#       [3, 4],
#       [5, 6],
#       [5, 6],
#       [5, 6]])
```

```python
np.append(a, b, axis = 0)
#Output
#array([[1, 2],
#       [3, 4],
#       [5, 6],
#       [5, 6],
#       [1, 2],
#       [3, 4],
#       [5, 6],
#       [5, 6],
#       [1, 2],
#       [3, 4],
#       [5, 6],
#       [5, 6],
#       [5, 6],
#       [1, 2],
#       [3, 4],
#       [5, 6],
#       [5, 6],
#       [1, 2],
#       [3, 4],
#       [5, 6],
#       [5, 6],
#       [5, 6]])
```

```python
np.append(a, b)
#Output
#array([1, 2, 3, 4, 5, 6, 5, 6, 1, 2, 3, 4, 5, 6, 5, 6, 1, 2, 3, 4, 5, 6, 5,
#       6, 5, 6, 1, 2, 3, 4, 5, 6, 5, 6, 1, 2, 3, 4, 5, 6, 5, 6, 5, 6])
#```
```

Response:
np.append uses np.concatenate:
```
def append(arr, values, axis=None):
    arr = asanyarray(arr)
    if axis is None:
        if arr.ndim != 1:
            arr = arr.ravel()
        values = ravel(values)
        axis = arr.ndim-1
    return concatenate((arr, values), axis=axis)
```


Question: Solve an equation using a python numerical solver in numpy I have an equation, as follows: R - ((1.0 - np.exp(-tau))/(1.0 - np.exp(-a*tau))) = 0. I want to solve for tau in this equation using a numerical solver available within numpy. What is the best way to go about this? The values for R and a in this equation vary for different implementations of this formula, but are fixed at particular values when it is to be solved for tau.

Response:
In conventional mathematical notation, your equation is The SciPy fsolve function searches for a point at which a given expression equals zero (a "zero" or "root" of the expression). You'll need to provide fsolve with an initial guess that's "near" your desired solution. A good way to find such an initial guess is to just plot the expression and look for the zero crossing.
```
#!/usr/bin/python

import numpy as np
import matplotlib.pyplot as plt
from scipy.optimize import fsolve

# Define the expression whose roots we want to find

a = 0.5
R = 1.6

func = lambda tau : R - ((1.0 - np.exp(-tau))/(1.0 - np.exp(-a*tau)))

# Plot it

tau = np.linspace(-0.5, 1.5, 201)

plt.plot(tau, func(tau))
plt.xlabel("tau")
plt.ylabel("expression value")
plt.grid()
plt.show()

# Use the numerical solver to find the roots

tau_initial_guess = 0.5
tau_solution = fsolve(func, tau_initial_guess)

print "The solution is tau = %f" % tau_solution
print "at which the value of the expression is %f" % func(tau_solution)
```


Question:
Getting data from ctypes array into numpy I am using a Python (via ctypes) wrapped C library to run a series of computation. At different stages of the running, I want to get data into Python, and specifically numpy arrays. The wrapping I am using does two different types of return for array data (which is of particular interest to me): ctypes Array: When I do type(x) (where x is the ctypes array, I get a  in return. I know that this data is a copy of the internal data from the documentation and I am able to get it into a numpy array easily:
```

```python
np.ctypeslib.as_array(x)
#Output
#```
#This returns a 1D numpy array of the data. ctype pointer to data: In this case from the library's documentation, I understand that I am getting a pointer to the data stored and used directly to the library. Whey I do type(y) (where y is the pointer) I get . With this case I am still able to index through the data like y[0][2], but I was only able to get it into numpy via a super awkward:
#```
```

```python
np.frombuffer(np.core.multiarray.int_asbuffer(
#Output
#    ctypes.addressof(y.contents), array_length*np.dtype(float).itemsize))
#```
#I found this in an old numpy mailing list thread from Travis Oliphant, but not in the numpy documentation. If instead of this approach I try as above I get the following:
#```
```

```python
np.ctypeslib.as_array(y)
 BUNCH OF STACK INFORMATION
#Output
#...
#...
#AttributeError: 'LP_c_double' object has no attribute '__array_interface__'
#```
#Is this np.frombuffer approach the best or only way to do this? I am open to other suggestions but must would still like to use numpy as I have a lot of other post-processing code that relies on numpy functionality that I want to use with this data.
```

Response:
Creating NumPy arrays from a ctypes pointer object is a problematic operation. It is unclear who actually owns the memory the pointer is pointing to. When will it be freed again? How long is it valid? Whenever possible I would try to avoid this kind of construct. It is so much easier and safer to create arrays in the Python code and pass them to the C function than to use memory allocated by a Python-unaware C function. By doing the latter, you negate to some extent the advantages of having a high-level language taking care of the memory management. If you are really sure that someone takes care of the memory, you can create an object exposing the Python "buffer protocol" and then create a NumPy array using this buffer object. You gave one way of creating the buffer object in your post, via the undocumented int_asbuffer() function:
```
buffer = numpy.core.multiarray.int_asbuffer(
    ctypes.addressof(y.contents), 8*array_length)
```
(Note that I substituted 8 for np.dtype(float).itemsize. It's always 8, on any platform.) A different way to create the buffer object would be to call the PyBuffer_FromMemory() function from the Python C API via ctypes:
```
buffer_from_memory = ctypes.pythonapi.PyBuffer_FromMemory
buffer_from_memory.restype = ctypes.py_object
buffer = buffer_from_memory(y, 8*array_length)
```
For both these ways, you can create a NumPy array from buffer by
```
a = numpy.frombuffer(buffer, float)
```
(I actually do not understand why you use .astype() instead of a second parameter to frombuffer; furthermore, I wonder why you use np.int, while you said earlier that the array contains doubles.) I'm afraid it won't get much easier than this, but it isn't that bad, don't you think? You could bury all the ugly details in a wrapper function and don't worry about it any more.


Question: What is a "scalar" in NumPy? The documentation states the purpose of scalars, such as the fact that conventional Python numbers like float and integer are too primitive, and therefore more complex data types are necessary. It also states certain kinds of scalars (data type hierarchy); as well as a couple of attributes of scalar. But it never gives a concrete definition of exactly what a scalar is in the context of Python. I want to get to the heart of the issue on this. In the simplest terms possible, what is a Pythonic scalar?

Response:
A NumPy scalar is any object which is an instance of np.generic or whose type is in np.ScalarType:
```none

```python
np.ScalarType
#Output
#(int,
# float,
# complex,
# long,
# bool,
# str,
# unicode,
# buffer,
# numpy.int16,
# numpy.float16,
# numpy.int8,
# numpy.uint64,
# numpy.complex192,
# numpy.void,
# numpy.uint32,
# numpy.complex128,
# numpy.unicode_,
# numpy.uint32,
# numpy.complex64,
# numpy.string_,
# numpy.uint16,
# numpy.timedelta64,
# numpy.bool_,
# numpy.uint8,
# numpy.datetime64,
# numpy.object_,
# numpy.int64,
# numpy.float96,
# numpy.int32,
# numpy.float64,
# numpy.int32,
# numpy.float32)
#```
#This definition comes from looking at the source code for np.isscalar:
#```
#def isscalar(num):
#    if isinstance(num, generic):
#        return True
#    else:
#        return type(num) in ScalarType
#```
#Note that you can test if something is a scalar by using np.isscalar:
#```none
#>>> np.isscalar(3.1)
#True
#
#>>> np.isscalar([3.1])
#False
#
#>>> np.isscalar(False)
#True
#```
#How do we know what we know? I like learning how people know what they know—more than the answers themselves. So let me try to explain where the above answer comes from. Having the right tools can help you figure out things like this for yourself. I found this out by using IPython. Using its TAB-completion feature, typing
#```none
```

```python
import numpy as np
```

```python
np.[TAB]
```
causes IPython to display all variables in the np module namespace. A search for the string "scalar" will lead you to np.ScalarType and np.isscalar. Typing
```none
```

```python
np.isscalar?
```
(note the question mark at the end) prompts IPython to show you where np.isscalar is defined:
```none
File:  /data1/unutbu/.virtualenvs/dev/lib/python2.7/site-packages/numpy/core/numeric.py
```
which is how I got to the definition of isscalar. Alternatively, the NumPy documentation for isscalar has a link to the source code as well.
```


Question:
Are Decimal 'dtypes' available in NumPy? Are Decimal data type objects (dtypes) available in NumPy?
```

```python
import decimal, numpy
```

```python
d = decimal.Decimal('1.1')
```

```python
s = [['123.123','23'],['2323.212','123123.21312']]
```

```python
ss = numpy.array(s, dtype=numpy.dtype(decimal.Decimal))
```

```python
a = numpy.array(s, dtype=float)
```

```python
type(d)
#Output
#<class 'decimal.Decimal'>
```

```python
type(ss[1,1])
#Output
#<class 'str'>
```

```python
type(a[1,1])
#Output
#<class 'numpy.float64'>
#```
#I suppose numpy.array doesn't support every dtype, but I sort of thought that it would at least let a dtype propagate as far as it could as long as the right operations were defined. Am I missing something? Is there some way for this to work?
```

Response:
NumPy doesn't recognize decimal.Decimal as a specific type. The closest it can get is the most general dtype, object. So when converting the elements to the desired dtype, the conversion is a no operation.
```

```python
ss.dtype
#Output
#dtype('object')
#```
#Keep in mind that because the elements of the array are Python objects, you won't get much of a speedup using them. For example, if you try to add this to any other array, the other elements will have to be boxed back into Python objects and added via the normal Python addition code. You might gain some speed in that the iteration will be in C, but not that much.
```


Question:
interpolate 3D volume with numpy and or scipy I am extremely frustrated because after several hours I can't seem to be able to do a seemingly easy 3D interpolation in python. In Matlab all I had to do was
```
Vi = interp3(x,y,z,V,xi,yi,zi)
```
What is the exact equivalent of this using scipy's ndimage.map_coordinate or other numpy methods? Thanks

Response:
In scipy 0.14 or later, there is a new function scipy.interpolate.RegularGridInterpolator which closely resembles interp3. The MATLAB command Vi = interp3(x,y,z,V,xi,yi,zi) would translate to something like:
```
from numpy import array
from scipy.interpolate import RegularGridInterpolator as rgi
my_interpolating_function = rgi((x,y,z), V)
Vi = my_interpolating_function(array([xi,yi,zi]).T)
```
Here is a full example demonstrating both; it will help you understand the exact differences... MATLAB CODE:
```
x = linspace(1,4,11);
y = linspace(4,7,22);
z = linspace(7,9,33);
V = zeros(22,11,33);
for i=1:11
    for j=1:22
        for k=1:33
            V(j,i,k) = 100*x(i) + 10*y(j) + z(k);
        end
    end
end
xq = [2,3];
yq = [6,5];
zq = [8,7];
Vi = interp3(x,y,z,V,xq,yq,zq);
```
The result is Vi=[268 357] which is indeed the value at those two points (2,6,8) and (3,5,7). SCIPY CODE:
```
from scipy.interpolate import RegularGridInterpolator
from numpy import linspace, zeros, array
x = linspace(1,4,11)
y = linspace(4,7,22)
z = linspace(7,9,33)
V = zeros((11,22,33))
for i in range(11):
    for j in range(22):
        for k in range(33):
            V[i,j,k] = 100*x[i] + 10*y[j] + z[k]
fn = RegularGridInterpolator((x,y,z), V)
pts = array([[2,6,8],[3,5,7]])
print(fn(pts))
```
Again it's [268,357]. So you see some slight differences: Scipy uses x,y,z index order while MATLAB uses y,x,z (strangely); In Scipy you define a function in a separate step and when you call it, the coordinates are grouped like (x1,y1,z1),(x2,y2,z2),... while matlab uses (x1,x2,...),(y1,y2,...),(z1,z2,...). Other than that, the two are similar and equally easy to use.


Question:
Numpy argmax. How to compute both max and argmax? Is there a way to get max and argmax by one stroke ?
```
import numpy as np
a=[0,0,1,0]
maximum=max(a)
index=np.argmax(a)
```
Is there a fastest way to do it, with something like:
```
[maximum,index]=function(a)
```

Response:
Maybe something like this is faster...
```
index = np.argmax(a)
maximum = a[index]
```


Question: SimpleJSON and NumPy array What is the most efficient way of serializing a numpy array using simplejson?

Response:
In order to keep dtype and dimension try this:
```
import base64
import json
import numpy as np

class NumpyEncoder(json.JSONEncoder):

    def default(self, obj):
        """If input object is an ndarray it will be converted into a dict
        holding dtype, shape and the data, base64 encoded.
        """
        if isinstance(obj, np.ndarray):
            if obj.flags['C_CONTIGUOUS']:
                obj_data = obj.data
            else:
                cont_obj = np.ascontiguousarray(obj)
                assert(cont_obj.flags['C_CONTIGUOUS'])
                obj_data = cont_obj.data
            data_b64 = base64.b64encode(obj_data)
            return dict(__ndarray__=data_b64,
                        dtype=str(obj.dtype),
                        shape=obj.shape)
        # Let the base class default method raise the TypeError
        super(NumpyEncoder, self).default(obj)

def json_numpy_obj_hook(dct):
    """Decodes a previously encoded numpy ndarray with proper shape and dtype.

    :param dct: (dict) json encoded ndarray
    :return: (ndarray) if input was an encoded ndarray
    """
    if isinstance(dct, dict) and '__ndarray__' in dct:
        data = base64.b64decode(dct['__ndarray__'])
        return np.frombuffer(data, dct['dtype']).reshape(dct['shape'])
    return dct

expected = np.arange(100, dtype=np.float)
dumped = json.dumps(expected, cls=NumpyEncoder)
result = json.loads(dumped, object_hook=json_numpy_obj_hook)

# None of the following assertions will be broken.
assert result.dtype == expected.dtype, "Wrong Type"
assert result.shape == expected.shape, "Wrong Shape"
assert np.allclose(expected, result), "Wrong Values"
```


Question:
Convert NumPy array to 0 or 1 based on threshold I have an array below:
```
a=np.array([0.1, 0.2, 0.3, 0.7, 0.8, 0.9])
```
What I want is to convert this vector to a binary vector based on a threshold. take threshold=0.5 as an example, element that greater than 0.5 convert to 1, otherwise 0. The output vector should like this:
```
a_output = [0, 0, 0, 1, 1, 1]
```
How can I do this?

Response:
np.where
```
np.where(a > 0.5, 1, 0)
# array([0, 0, 0, 1, 1, 1])
```
Boolean basking with astype
```
(a > .5).astype(int)
# array([0, 0, 0, 1, 1, 1])
```
np.select
```
np.select([a <= .5, a>.5], [np.zeros_like(a), np.ones_like(a)])
# array([ 0.,  0.,  0.,  1.,  1.,  1.])
```
Special case: np.round This is the best solution if your array values are floating values between 0 and 1 and your threshold is 0.5.
```
a.round()
# array([0., 0., 0., 1., 1., 1.])
```


Question:
Taking subarrays from numpy array with given stride/stepsize Lets say I have a Python Numpy array a.
```
a = numpy.array([1,2,3,4,5,6,7,8,9,10,11])
```
I want to create a matrix of sub sequences from this array of length 5 with stride 3. The results matrix hence will look as follows:
```
numpy.array([[1,2,3,4,5],[4,5,6,7,8],[7,8,9,10,11]])
```
One possible way of implementing this would be using a for-loop.
```
result_matrix = np.zeros((3, 5))
for i in range(0, len(a), 3):
  result_matrix[i] = a[i:i+5]
```
Is there a cleaner way to implement this in Numpy?

Response:
Approach #1 : Using broadcasting -
```
def broadcasting_app(a, L, S ):  # Window len = L, Stride len/stepsize = S
    nrows = ((a.size-L)//S)+1
    return a[S*np.arange(nrows)[:,None] + np.arange(L)]
```
Approach #2 : Using more efficient NumPy strides -
```
def strided_app(a, L, S ):  # Window len = L, Stride len/stepsize = S
    nrows = ((a.size-L)//S)+1
    n = a.strides[0]
    return np.lib.stride_tricks.as_strided(a, shape=(nrows,L), strides=(S*n,n))
```
Sample run -
```

```python
a
#Output
#array([ 1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11])
```

```python
broadcasting_app(a, L = 5, S = 3)
#Output
#array([[ 1,  2,  3,  4,  5],
#       [ 4,  5,  6,  7,  8],
#       [ 7,  8,  9, 10, 11]])
```

```python
strided_app(a, L = 5, S = 3)
#Output
#array([[ 1,  2,  3,  4,  5],
#       [ 4,  5,  6,  7,  8],
#       [ 7,  8,  9, 10, 11]])
#```
```


Question:
What is "random-state" in sklearn.model_selection.train_test_split example? [duplicate] This question already has answers here: Random state (Pseudo-random number) in Scikit learn (8 answers) Closed 4 years ago. Can someone explain me what random_state means in below example?
```
import numpy as np
from sklearn.model_selection import train_test_split
X, y = np.arange(10).reshape((5, 2)), range(5)

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.33, random_state=42)
```
Why is it hard coded to 42?

Response: Isn't that obvious? 42 is the Answer to the Ultimate Question of Life, the Universe, and Everything. On a serious note, random_state simply sets a seed to the random generator, so that your train-test splits are always deterministic. If you don't set a seed, it is different each time. Relevant documentation: random_state : int, RandomState instance or None, optional (default=None) If int, random_state is the seed used by the random number generator; If RandomState instance, random_state is the random number generator; If None, the random number generator is the RandomState instance used by np.random.


Question:
How to "scale" a numpy array? I would like to scale an array of shape (h, w) by a factor of n, resulting in an array of shape (h*n, w*n), with the. Say that I have a 2x2 array:
```
array([[1, 1],
       [0, 1]])
```
I would like to scale the array to become 4x4:
```
array([[1, 1, 1, 1],
       [1, 1, 1, 1],
       [0, 0, 1, 1],
       [0, 0, 1, 1]])
```
That is, the value of each cell in the original array is copied into 4 corresponding cells in the resulting array. Assuming arbitrary array size and scaling factor, what's the most efficient way to do this?

Response:
You should use the Kronecker product, numpy.kron: Computes the Kronecker product, a composite array made of blocks of the second array scaled by the first
```
import numpy as np
a = np.array([[1, 1],
              [0, 1]])
n = 2
np.kron(a, np.ones((n,n)))
```
which gives what you want:
```
array([[1, 1, 1, 1],
       [1, 1, 1, 1],
       [0, 0, 1, 1],
       [0, 0, 1, 1]])
```


Question:
How to convert the output of meshgrid to the corresponding array of points? I want to create a list of points that would correspond to a grid. So if I want to create a grid of the region from (0, 0) to (1, 1), it would contain the points (0, 0), (0, 1), (1, 0) and (1, 0). I know that that this can be done with the following code:
```
g = np.meshgrid([0,1],[0,1])
np.append(g[0].reshape(-1,1),g[1].reshape(-1,1),axis=1)
```
Yielding the result:
```
array([[0, 0],
       [1, 0],
       [0, 1],
       [1, 1]])
```
My question is twofold: Is there a better way of doing this? Is there a way of generalizing this to higher dimensions?

Response:
I just noticed that the documentation in numpy provides an even faster way to do this:
```
X, Y = np.mgrid[xmin:xmax:100j, ymin:ymax:100j]
positions = np.vstack([X.ravel(), Y.ravel()])
```
This can easily be generalized to more dimensions using the linked meshgrid2 function and mapping 'ravel' to the resulting grid.
```
g = meshgrid2(x, y, z)
positions = np.vstack(map(np.ravel, g))
```
The result is about 35 times faster than the zip method for a 3D array with 1000 ticks on each axis. Source: http://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.gaussian_kde.html#scipy.stats.gaussian_kde To compare the two methods consider the following sections of code: Create the proverbial tick marks that will help to create the grid.
```

```python
import numpy as np
```

```python
from numpy import asarray
```

```python
x = np.random.rand(100,1)
```

```python
y = np.random.rand(100,1)
```

```python
z = np.random.rand(100,1)
```
Define the function that mgilson linked to for the meshgrid:
```
```

```python
def meshgrid2(*arrs):
   ....:     arrs = tuple(reversed(arrs))
   ....:     lens = map(len, arrs)
   ....:     dim = len(arrs)
   ....:     sz = 1
   ....:     for s in lens:
   ....:        sz *= s
   ....:     ans = []
   ....:     for i, arr in enumerate(arrs):
   ....:         slc = [1]*dim
   ....:         slc[i] = lens[i]
   ....:         arr2 = asarray(arr).reshape(slc)
   ....:         for j, sz in enumerate(lens):
   ....:             if j != i:
   ....:                 arr2 = arr2.repeat(sz, axis=j)
   ....:         ans.append(arr2)
   ....:     return tuple(ans)
```
Create the grid and time the two functions.
```
```

```python
g = meshgrid2(x, y, z)
```

```python
%timeit pos = np.vstack(map(np.ravel, g)).T
100 loops, best of 3: 7.26 ms per loop
```

```python
%timeit zip(*(x.flat for x in g))
1 loops, best of 3: 264 ms per loop
```
```


Question:
Random Choice with Pytorch? I have a tensor of pictures, and would like to randomly select from it. I'm looking for the equivalent of np.random.choice().
```
import torch

pictures = torch.randint(0, 256, (1000, 28, 28, 3))
```
Let's say I want 10 of these pictures.

Response:
torch has no equivalent implementation of np.random.choice(), see the discussion here. The alternative is indexing with a shuffled index or random integers. To do it with replacement: Generate n random indices Index your original tensor with these indices
```
pictures[torch.randint(len(pictures), (10,))]
```
To do it without replacement: Shuffle the index Take the n first elements
```
indices = torch.randperm(len(pictures))[:10]

pictures[indices]
```
Read more about torch.randint and torch.randperm. Second code snippet is inspired by this post in PyTorch Forums.


Question: Setting the fmt option in numpy.savetxt I am looking at the numpy.savetxt, and am stuck at the fmt option. I tried looking at here and also the reference in the link below all the letters that can be used for the fmt option sort give me a general sense of what is going on. What I do not understand is if the % symbol is required and in an example given here how should I interpret the 10.5 number? If "f" is about setting the floating point, then how come is it 10.5 (then again, I might not know how floating points are set...).

Response:
Knowing that np.savetxt only works for 1D or 2D arrays, the general idea is: when fmt is a single formatting string it applies to all elements in the array (1D or 2D input array) when fmt is a sequence of formatting strings, it applies to each column of the 2D input array I'm presenting here some examples using the following input array:
```
import numpy as np

a = np.array([[11, 12, 13, 14],
              [21, 22, 23, 24],
              [31, 32, 33, 34]])
```
1) Setting floating point precision: np.savetxt('tmp.txt', a, fmt='%1.3f')
```
11.000 12.000 13.000 14.000
21.000 22.000 23.000 24.000
31.000 32.000 33.000 34.000
```
2) Adding characters to right-justify. With spaces: np.savetxt('tmp.txt', a, fmt='% 4d')
```
11   12   13   14
  21   22   23   24
  31   32   33   34
```
With zeros: np.savetxt('tmp.txt', a, fmt='%04d')
```
0011 0012 0013 0014
0021 0022 0023 0024
0031 0032 0033 0034
```
3) Adding characters to left-justify (use of "-"). With spaces: np.savetxt('tmp.txt', a, fmt='%-4d')
```
11   12   13   14
21   22   23   24
31   32   33   34
```
4) When fmt is a sequence of formatting strings, each row of a 2D input array is processed according to fmt: fmt as a sequence in a single formatting string
```
fmt = '%1.1f + %1.1f / (%1.1f * %1.1f)'
np.savetxt('tmp.txt', a, fmt=fmt)

11.0 + 12.0 / (13.0 * 14.0)
21.0 + 22.0 / (23.0 * 24.0)
31.0 + 32.0 / (33.0 * 34.0)
```
fmt as an iterator of formatting strings:
```
fmt = '%d', '%1.1f', '%1.9f', '%1.9f'
np.savetxt('tmp.txt', a, fmt=fmt)

11 12.0 13.000000000 14.000000000
21 22.0 23.000000000 24.000000000
31 32.0 33.000000000 34.000000000
```


Question:
How do I subtract the previous row from the current row in a pandas dataframe and apply it to every row; without using a loop? I am using Python3.5 and I am working with pandas. I have loaded stock data from yahoo finance and have saved the files to csv. My DataFrames load this data from the csv. This is a copy of the ten rows of the csv file that is my DataFrame
```
Date       Open       High      Low     Close    Volume   Adj Close
1990-04-12  26.875000  26.875000  26.625  26.625      6100  250.576036
1990-04-16  26.500000  26.750000  26.375  26.750       500  251.752449
1990-04-17  26.750000  26.875000  26.750  26.875      2300  252.928863
1990-04-18  26.875000  26.875000  26.500  26.625      3500  250.576036
1990-04-19  26.500000  26.750000  26.500  26.750       700  251.752449
1990-04-20  26.750000  26.875000  26.750  26.875      2100  252.928863
1990-04-23  26.875000  26.875000  26.750  26.875       700  252.928863
1990-04-24  27.000000  27.000000  26.000  26.000      2400  244.693970
1990-04-25  25.250000  25.250000  24.875  25.125      9300  236.459076
1990-04-26  25.000000  25.250000  24.750  25.000      1200  235.282663
```
I know that I can use iloc, loc, ix but these values that I index will only give my specific rows and columns and will not perform the operation on every row. For example: Row one of the data in the open column has a value of 26.875 and the row below it has 26.50. The price dropped .375 cents. I want to be able to capture the % of Increase or Decrease from the previous day so to finish this example .375 divided by 26.875 = 1.4% decrease from one day to the next. I want to be able to run this calculation on every row so I know how much it has increased or decreased from the previous day. The index functions I have tried but they are absolute, and I don't want to use a loop. Is there a way I can do this with the ix, iloc, loc or another function?

Response:
you can use pct_change() or/and diff() methods Demo:
```

```python
df.Close.pct_change() * 100
#Output
#0         NaN
#1    0.469484
#2    0.467290
#3   -0.930233
#4    0.469484
#5    0.467290
#6    0.000000
#7   -3.255814
#8   -3.365385
#9   -0.497512
#Name: Close, dtype: float64
```

```python
df.Close.diff()
#Output
#0      NaN
#1    0.125
#2    0.125
#3   -0.250
#4    0.125
#5    0.125
#6    0.000
#7   -0.875
#8   -0.875
#9   -0.125
#Name: Close, dtype: float64
#```
```


Question:
Numpy Adding two vectors with different sizes If I have two numpy arrays of different sizes, how can I superimpose them.
```
a = numpy([0, 10, 20, 30])
b = numpy([20, 30, 40, 50, 60, 70])
```
What is the cleanest way to add these two vectors to produce a new vector (20, 40, 60, 80, 60, 70)? This is my generic question. For background, I am specifically applying a Green's transform function and need to superimpose the results for each time step in the evaulation unto the responses previously accumulated.

Response:
This could be what you are looking for
```
if len(a) < len(b):
    c = b.copy()
    c[:len(a)] += a
else:
    c = a.copy()
    c[:len(b)] += b
```
basically you copy the longer one and then add in-place the shorter one


Question:
How do I extract a sub-array from a numpy 2d array? [duplicate] This question already has answers here: Slicing arrays in Numpy / Scipy (3 answers) Closed 9 years ago. I'd like to extract a numpy array with a specified size from a numpy 2d array--essentially I want to crop the array. For example, if have a numpy array like this:
```
([1,2,3],
 [4,5,6],
 [7,8,9])
```
I'd like to extract a 2x2 from it and the result should be:
```
([1,2],
 [4,5])
```
How can I do that?

Response:
Given this array:
```

```python
a
#Output
#array([[1, 2, 3],
#       [4, 5, 6],
#       [7, 8, 9]])
#```
#You can slice it along both dimensions:
#```
```

```python
a[:2,:2]
#Output
#array([[1, 2],
#       [4, 5]])
#```
```


Question:
How to find max value in a numpy array column? I can find quite a few permutations of this question, but not this (rather simple) one: how do I find the maximum value of a specific column of a numpy array (in the most pythonic way)?
```
a = array([[10, 2], [3, 4], [5, 6]])
```
What I want is the max value in the first column and second column (these are x,y coordinates and I eventually need the height and width of each shape), so max x coordinate is 10 and max y coordinate is 6. I've tried:
```
xmax = numpy.amax(a,axis=0)
ymax = numpy.amax(a,axis=1)
```
but these yield
```
array([10, 6])
array([10, 4, 6])
```
...not what I expected. My solution is to use slices:
```
xmax = numpy.max(a[:,0])
ymax = numpy.max(a[:,1])
```
Which works but doesn't seem to the best approach. Suggestions?

Response:
Just unpack the list:
```

```python
xmax, ymax = a.max(axis=0)
```

```python
print xmax, ymax
#10 6
```
```


Question:
ufunc 'add' did not contain loop with signature matching type dtype ('S32') ('S32') ('S32') I'm trying to run someone's script for some simulations I've made to try plotting some histograms, but when I do I always get the error message mentioned above. I have no idea what's gone wrong. Here's the complete traceback error I get:
```none
File "AVAnalyse.py", line 205, in <module>
  f.write(line[0] + '  ' + line[1] + '  ' + line[2] + '  ' + line[3])
TypeError: ufunc 'add' did not contain a loop with signature matching types dtype('S32') dtype('S32') dtype('S32')
```
This is the code I am trying to run:
```
name_out = "histogram_" + donor + "_" + acceptor + ".dat"
f = open(name_out, 'w')
f.write('distance  d.probability  efficiency  e.probability')
for line in dist_hist:
    f.write(line[0] + '  ' + line[1] + '  ' + line[2] + '  ' + line[3])
f.close()

print "data saved in " + "histogram_" + donor + "_" + acceptor + ".dat"
```
What am I doing wrong?

Response:
It seems like line[0], line[1], line[2], line[3] are elements of dist_hist. dict_hist is a numpy.ndarray. The elements of dict_hist has a numeric type (like np.float64) (based on calculations from your attached file). You're trying to add elements of different types: np.float64 and str. If you want to avoid this TypeError, you can change type of line[0], line[1], line[2], line[3] to str. Your snippet of code should be like this:
```
name_out = "histogram_"+donor+"_"+acceptor+".dat"
f = open(name_out,'w')
f.write('distance  d.probability  efficiency  e.probability')
for line in dist_hist:
  f.write(str(line[0])+'  '+str(line[1])+'  '+str(line[2])+'  '+str(line[3]))
f.close()

print "data saved in " +"histogram_"+donor+"_"+acceptor+".dat"
```
EDIT: You should replace this snippet of code:
```
name_out = "histogram_"+donor+"_"+acceptor+".dat"
f = open(name_out,'w')
f.write('distance  d.probability  efficiency  e.probability')
for line in dist_hist:
  f.write(line[0]+'  '+line[1]+'  '+line[2]+'  '+line[3])
f.close()
```
to this one:
```
name_out = "histogram_" + donor + "_" + acceptor + ".dat"
f = open(name_out,'w')
f.write('distance  d.probability  efficiency  e.probability\n')
for line in dist_hist:
  f.write(str(line[0]) + '  ' + str(line[1]) + '  ' + str(line[2]) + '  ' + str(line[3]) + '\n')
f.close()
```
Before that, strings were written to file in one line. Because of that your data variable point to empty array since we start to read from 2nd line (which was empty).


Question:
How to create a white image in Python? The following code creates a black picture:
```
from scipy.misc import imread,imsave
from numpy import zeros

img = zeros([100,100,3])
for y in range(len(img)):
    for x in range(len(img[0])):
        img[y,x] = [255,255,255]
imsave("Result.jpg",img)
```
I would have assumed it to be white. Edit in 2024: The code block is deprecated and won't run anymore.

Response:
Every color in an image is represented by one byte. So to create an image array, you should set it's dtype to uint8. And, you don't need for-loop to set every elements to 255, you can use fill() method or slice index:
```
import numpy as np
img = np.zeros([100,100,3],dtype=np.uint8)
img.fill(255) # or img[:] = 255
```


Question:
Filtering pandas dataframe with multiple Boolean columns I am trying to filter a df using several Boolean variables that are a part of the df, but have been unable to do so. Sample data:
```
A | B | C | D
John Doe | 45 | True | False
Jane Smith | 32 | False | False
Alan Holmes | 55 | False | True
Eric Lamar | 29 | True | True
```
The dtype for columns C and D is Boolean. I want to create a new df (df1) with only the rows where either C or D is True. It should look like this:
```
A | B | C | D
John Doe | 45 | True | False
Alan Holmes | 55 | False | True
Eric Lamar | 29 | True | True
```
I've tried something like this, which faces issues because it cant handle the Boolean type:
```
df1 = df[(df['C']=='True') or (df['D']=='True')]
```
Any ideas?

Response:
```

```python
d
#Output
#A   B      C      D
#0     John Doe  45   True  False
#1   Jane Smith  32  False  False
#2  Alan Holmes  55  False   True
#3   Eric Lamar  29   True   True
#```
#Solution 1:
#```
```

```python
d.loc[d.C | d.D]
#Output
#A   B      C      D
#0     John Doe  45   True  False
#2  Alan Holmes  55  False   True
#3   Eric Lamar  29   True   True
#```
#Solution 2:
#```
```

```python
d[d[['C','D']].any(1)]
#Output
#A   B      C      D
#0     John Doe  45   True  False
#2  Alan Holmes  55  False   True
#3   Eric Lamar  29   True   True
#```
#Solution 3:
#```
```

```python
d.query("C or D")
#Output
#A   B      C      D
#0     John Doe  45   True  False
#2  Alan Holmes  55  False   True
#3   Eric Lamar  29   True   True
#```
#PS If you change your solution to:
#```
#df[(df['C']==True) | (df['D']==True)]
#```
#it'll work too Pandas docs - boolean indexing why we should NOT use "PEP complaint" df["col_name"] is True instead of df["col_name"] == True?
#```
```

```python
df = pd.DataFrame({"col":[True, True, True]})
```

```python
df
#Output
#col
#0  True
#1  True
#2  True
```

```python
df["col"] is True
#Output
#False               # <----- oops, that's not exactly what we wanted
#```
```


Question: What is the internal precision of numpy.float128? What precision does numpy.float128 map to internally? Is it __float128 or long double? Or something else entirely? A potential follow on question if anybody knows: is it safe in C to cast a __float128 to a (16 byte) long double, with just a loss in precision? (this is for interfacing with a C lib that operates on long doubles). Edit: In response to the comment, the platform is 'Linux-3.0.0-14-generic-x86_64-with-Ubuntu-11.10-oneiric'. Now, if numpy.float128 has varying precision dependent on the platform, that is also useful knowledge for me! Just to be clear, it is the precision I am interested in, not the size of an element.

Response: numpy.longdouble refers to whatever type your C compiler calls long double. Currently, this is the only extended precision floating point type that numpy supports. On x86-32 and x86-64, this is an 80-bit floating point type. On more exotic systems it may be something else (IIRC on Sparc it's an actual 128-bit IEEE float, and on PPC it's double-double). (It also may depend on what OS and compiler you're using -- e.g. MSVC on Windows doesn't support any kind of extended precision at all.) Numpy will also export some name like numpy.float96 or numpy.float128. Which of these names is exported depends on your platform/compiler, but whatever you get always refers to the same underlying type as longdouble. Also, these names are highly misleading. They do not indicate a 96- or 128-bit IEEE floating point format. Instead, they indicate the number of bits of alignment used by the underlying long double type. So e.g. on x86-32, long double is 80 bits, but gets padded up to 96 bits to maintain 32-bit alignment, and numpy calls this float96. On x86-64, long double is again the identical 80 bit type, but now it gets padded up to 128 bits to maintain 64-bit alignment, and numpy calls this float128. There's no extra precision, just extra padding. Recommendation: ignore the float96/float128 names, just use numpy.longdouble. Or better yet stick to doubles unless you have a truly compelling reason. They'll be faster, more portable, etc.


Question: numpy.asarray: how to check up that its result dtype is numeric? I have to create a numpy.ndarray from array-like data with int, float or complex numbers. I hope to do it with numpy.asarray function. I don't want to give it a strict dtype argument, because I want to convert complex values to complex64 or complex128, floats to float32 or float64, etc. But if I just simply run numpy.ndarray(some_unknown_data) and look at the dtype of its result, how can I understand, that the data is numeric, not object or string or something else?

Response:
You could check if the dtype of the array is a sub-dtype of np.number. For example:
```

```python
np.issubdtype(np.complex128, np.number)
#Output
#True
```

```python
np.issubdtype(np.int32, np.number)
#Output
#True
```

```python
np.issubdtype(np.str_, np.number)
#Output
#False
```

```python
np.issubdtype('O', np.number) # 'O' is object
#Output
#False
#```
#Essentially, this just checks whether the dtype is below 'number' in the NumPy dtype hierarchy:
```


Question:
Finding range of a numpy array elements I have a NumPy array of size 94 x 155:
```
a = [1  2  20  68  210  290..
     2  33 34  55  230  340..
     .. .. ... ... .... .....]
```
I want to calculate the range of each row, so that I get 94 ranges in a result. I tried looking for a numpy.range function, which I don't think exists. If this can be done through a loop, that's also fine. I'm looking for something like numpy.mean, which, if we set the axis parameter to 1, returns the mean for each row in the N-dimensional array.

Response:
I think np.ptp might do what you want: http://docs.scipy.org/doc/numpy/reference/generated/numpy.ptp.html
```
r = np.ptp(a,axis=1)
```
where r is your range array.


Question:
How to see top n entries of term-document matrix after tfidf in scikit-learn I am new to scikit-learn, and I was using TfidfVectorizer to find the tfidf values of terms in a set of documents. I used the following code to obtain the same.
```
vectorizer = TfidfVectorizer(stop_words=u'english',ngram_range=(1,5),lowercase=True)
X = vectorizer.fit_transform(lectures)
```
Now If I print X, I am able to see all the entries in matrix, but how can I find top n entries based on tfidf score. In addition to that is there any method that will help me to find top n entries based on tfidf score per ngram i.e. top entries among unigram,bigram,trigram and so on?

Response:
Since version 0.15, the global term weighting of the features learnt by a TfidfVectorizer can be accessed through the attribute idf_, which will return an array of length equal to the feature dimension. Sort the features by this weighting to get the top weighted features:
```
from sklearn.feature_extraction.text import TfidfVectorizer
import numpy as np

lectures = ["this is some food", "this is some drink"]
vectorizer = TfidfVectorizer()
X = vectorizer.fit_transform(lectures)
indices = np.argsort(vectorizer.idf_)[::-1]
features = vectorizer.get_feature_names()
top_n = 2
top_features = [features[i] for i in indices[:top_n]]
print top_features
```
Output:
```
[u'food', u'drink']
```
The second problem of getting the top features by ngram can be done using the same idea, with some extra steps of splitting the features into different groups:
```
from sklearn.feature_extraction.text import TfidfVectorizer
from collections import defaultdict

lectures = ["this is some food", "this is some drink"]
vectorizer = TfidfVectorizer(ngram_range=(1,2))
X = vectorizer.fit_transform(lectures)
features_by_gram = defaultdict(list)
for f, w in zip(vectorizer.get_feature_names(), vectorizer.idf_):
    features_by_gram[len(f.split(' '))].append((f, w))
top_n = 2
for gram, features in features_by_gram.iteritems():
    top_features = sorted(features, key=lambda x: x[1], reverse=True)[:top_n]
    top_features = [f[0] for f in top_features]
    print '{}-gram top:'.format(gram), top_features
```
Output:
```
1-gram top: [u'drink', u'food']
2-gram top: [u'some drink', u'some food']
```


Question:
Using numpy.genfromtxt to read a csv file with strings containing commas I am trying to read in a csv file with numpy.genfromtxt but some of the fields are strings which contain commas. The strings are in quotes, but numpy is not recognizing the quotes as defining a single string. For example, with the data in 't.csv':
```
2012, "Louisville KY", 3.5
2011, "Lexington, KY", 4.0
```
the code
```
np.genfromtxt('t.csv', delimiter=',')
```
produces the error: ValueError: Some errors were detected ! Line #2 (got 4 columns instead of 3) The data structure I am looking for is:
```
array([['2012', 'Louisville KY', '3.5'],
       ['2011', 'Lexington, KY', '4.0']],
      dtype='|S13')
```
Looking over the documentation, I don't see any options to deal with this. Is there a way do to it with numpy, or do I just need to read in the data with the csv module and then convert it to a numpy array?

Response:
You can use pandas (the becoming default library for working with dataframes (heterogeneous data) in scientific python) for this. It's read_csv can handle this. From the docs: quotechar : string
```
The character to used to denote the start and end of a quoted item. Quoted items
can include the delimiter and it will be ignored.
```
The default value is ". An example:
```

```python
import pandas as pd
```

```python
from StringIO import StringIO
```

```python
s="""year, city, value
   ...: 2012, "Louisville KY", 3.5
   ...: 2011, "Lexington, KY", 4.0"""
```

```python
pd.read_csv(StringIO(s), quotechar='"', skipinitialspace=True)
#Output
#year           city  value
#0  2012  Louisville KY    3.5
#1  2011  Lexington, KY    4.0
#```
#The trick here is that you also have to use skipinitialspace=True to deal with the spaces after the comma-delimiter. Apart from a powerful csv reader, I can also strongly advice to use pandas with the heterogeneous data you have (the example output in numpy you give are all strings, although you could use structured arrays).
```


Question:
Numpy modify array in place? I have the following code which is attempting to normalize the values of an m x n array (It will be used as input to a neural network, where m is the number of training examples and n is the number of features). However, when I inspect the array in the interpreter after the script runs, I see that the values are not normalized; that is, they still have the original values. I guess this is because the assignment to the array variable inside the function is only seen within the function. How can I do this normalization in place? Or do I have to return a new array from the normalize function?
```
import numpy

def normalize(array, imin = -1, imax = 1):
    """I = Imin + (Imax-Imin)*(D-Dmin)/(Dmax-Dmin)"""

    dmin = array.min()
    dmax = array.max()

    array = imin + (imax - imin)*(array - dmin)/(dmax - dmin)
    print array[0]

def main():

    array = numpy.loadtxt('test.csv', delimiter=',', skiprows=1)
    for column in array.T:
        normalize(column)

    return array

if __name__ == "__main__":
    a = main()
```

Response:
If you want to apply mathematical operations to a numpy array in-place, you can simply use the standard in-place operators +=, -=, /=, etc. So for example:
```

```python
def foo(a):
    a += 10

```

```python
a = numpy.arange(10)
```

```python
a
#Output
#array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
```

```python
foo(a)
```

```python
a
#Output
#array([10, 11, 12, 13, 14, 15, 16, 17, 18, 19])
#```
#The in-place version of these operations is a tad faster to boot, especially for larger arrays:
#```
```

```python
def normalize_inplace(array, imin=-1, imax=1):
        dmin = array.min()
        dmax = array.max()
        array -= dmin
        array *= imax - imin
        array /= dmax - dmin
        array += imin

```

```python
def normalize_copy(array, imin=-1, imax=1):
        dmin = array.min()
        dmax = array.max()
        return imin + (imax - imin) * (array - dmin) / (dmax - dmin)

```

```python
a = numpy.arange(10000, dtype='f')
```

```python
%timeit normalize_inplace(a)
#Output
#10000 loops, best of 3: 144 us per loop
```

```python
%timeit normalize_copy(a)
#Output
#10000 loops, best of 3: 146 us per loop
```

```python
a = numpy.arange(1000000, dtype='f')
```

```python
%timeit normalize_inplace(a)
#Output
#100 loops, best of 3: 12.8 ms per loop
```

```python
%timeit normalize_copy(a)
#Output
#100 loops, best of 3: 16.4 ms per loop
#```
```


Question:
What does "an intermediate result is being cached" mean? I have a set of n vectors stored in the 3 x n matrix z. I find the outer product using np.einsum. When I timed it using:
```
%timeit v=np.einsum('i...,j...->ij...',z,z)
```
I got the result:
```
The slowest run took 7.23 times longer than the fastest. This could mean that an
intermediate result is being cached
100000 loops, best of 3: 2.9 µs per loop
```
What is happening here and can it be avoided? The best 3 is 2.9us, but the slowest maybe more typical.

Response: The message "intermediate result is being cached" is just a blind guess in the canned message reported by %timeit. It may or may not be true, and you should not assume it is correct. In particular, one of the most common reasons for the first run being slowest is that the array is in the CPU cache only after the first run. CPUs cache things automatically; you cannot avoid this, and you don't really want to avoid it. However, optimizing algorithms so that CPU caches can work optimally is nowadays one of the bottlenecks that high-performance computing needs to take into account.


Question:
How to fast change image brightness with python + OpenCV? I have a sequence of images. I need to average brightness of these images. First example (very slow):
```
img = cv2.imread('test.jpg') #load rgb image
hsv = cv2.cvtColor(img, cv2.COLOR_BGR2HSV) #convert it to hsv

for x in range(0, len(hsv)):
    for y in range(0, len(hsv[0])):
        hsv[x, y][2] += value

img = cv2.cvtColor(hsv, cv2.COLOR_HSV2BGR)
cv2.imwrite("image_processed.jpg", img)
```
Second example (quickly)
```
hsv += value
```
This example very fast but it changes all values HSV (I need to change only V (brightness))

Response:
I know this question is a bit old, but I thought I might post the complete solution that worked for me (takes care of the overflow situation by saturating at 255):
```
def increase_brightness(img, value=30):
    hsv = cv2.cvtColor(img, cv2.COLOR_BGR2HSV)
    h, s, v = cv2.split(hsv)

    lim = 255 - value
    v[v > lim] = 255
    v[v <= lim] += value

    final_hsv = cv2.merge((h, s, v))
    img = cv2.cvtColor(final_hsv, cv2.COLOR_HSV2BGR)
    return img
```
This can be used as follows:
```
frame = increase_brightness(frame, value=20)
```


Question:
Calculating Slopes in Numpy (or Scipy) I am trying to find the fastest and most efficient way to calculate slopes using Numpy and Scipy. I have a data set of three Y variables and one X variable and I need to calculate their individual slopes. For example, I can easily do this one row at a time, as shown below, but I was hoping there was a more efficient way of doing this. I also don't think linregress is the best way to go because I don't need any of the auxiliary variables like intercept, standard error, etc in my results. Any help is greatly appreciated.
```
import numpy as np
    from scipy import stats

    Y = [[  2.62710000e+11   3.14454000e+11   3.63609000e+11   4.03196000e+11
        4.21725000e+11   2.86698000e+11   3.32909000e+11   4.01480000e+11
        4.21215000e+11   4.81202000e+11]
        [  3.11612352e+03   3.65968334e+03   4.15442691e+03   4.52470938e+03
        4.65011423e+03   3.10707392e+03   3.54692896e+03   4.20656404e+03
        4.34233412e+03   4.88462501e+03]
        [  2.21536396e+01   2.59098311e+01   2.97401268e+01   3.04784552e+01
        3.13667639e+01   2.76377113e+01   3.27846013e+01   3.73223417e+01
        3.51249997e+01   4.42563658e+01]]
    X = [ 1990.  1991.  1992.  1993.  1994.  1995.  1996.  1997.  1998.  1999.]
    slope_0, intercept, r_value, p_value, std_err = stats.linregress(X, Y[0,:])
    slope_1, intercept, r_value, p_value, std_err = stats.linregress(X, Y[1,:])
    slope_2, intercept, r_value, p_value, std_err = stats.linregress(X, Y[2,:])
    slope_0 = slope/Y[0,:][0]
    slope_1 = slope/Y[1,:][0]
    slope_2 = slope/Y[2,:][0]
    b, a = polyfit(X, Y[1,:], 1)
    slope_1_a = b/Y[1,:][0]
```

Response:
The fastest and the most efficient way would be to use a native scipy function from linregress which calculates everything: slope : slope of the regression line intercept : intercept of the regression line r-value : correlation coefficient p-value : two-sided p-value for a hypothesis test whose null hypothesis is that the slope is zero stderr : Standard error of the estimate And here is an example:
```
a = [15, 12, 8, 8, 7, 7, 7, 6, 5, 3]
b = [10, 25, 17, 11, 13, 17, 20, 13, 9, 15]
from scipy.stats import linregress
linregress(a, b)
```
will return you:
```
LinregressResult(slope=0.20833333333333337, intercept=13.375, rvalue=0.14499815458068521, pvalue=0.68940144811669501, stderr=0.50261704627083648)
```
P.S. Just a mathematical formula for slope:


Question:
How to center labels in histogram plot I have a numpy array results that looks like
```
[ 0.  2.  0.  0.  0.  0.  3.  0.  0.  0.  0.  0.  0.  0.  0.  2.  0.  0.
  0.  0.  0.  1.  0.  0.  0.  0.  0.  0.  0.  1.  0.  0.  0.  0.  0.  0.
  0.  1.  1.  0.  0.  0.  0.  2.  0.  3.  1.  0.  0.  2.  2.  0.  0.  0.
  0.  0.  0.  0.  0.  1.  1.  0.  0.  0.  0.  0.  0.  2.  0.  0.  0.  0.
  0.  1.  0.  0.  0.  0.  0.  0.  0.  0.  0.  3.  1.  0.  0.  0.  0.  0.
  0.  0.  0.  1.  0.  0.  0.  1.  2.  2.]
```
I would like to plot a histogram of it. I have tried
```
import matplotlib.pyplot as plt
plt.hist(results, bins=range(5))
plt.show()
```
This gives me a histogram with the x-axis labelled 0.0 0.5 1.0 1.5 2.0 2.5 3.0. 3.5 4.0. I would like the x-axis to be labelled 0 1 2 3 instead with the labels in the center of each bar. How can you do that?

Response:
The other answers just don't do it for me. The benefit of using plt.bar over plt.hist is that bar can use align='center':
```
import numpy as np
import matplotlib.pyplot as plt

arr = np.array([ 0.,  2.,  0.,  0.,  0.,  0.,  3.,  0.,  0.,  0.,  0.,  0.,  0.,
        0.,  0.,  2.,  0.,  0.,  0.,  0.,  0.,  1.,  0.,  0.,  0.,  0.,
        0.,  0.,  0.,  1.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  1.,  1.,
        0.,  0.,  0.,  0.,  2.,  0.,  3.,  1.,  0.,  0.,  2.,  2.,  0.,
        0.,  0.,  0.,  0.,  0.,  0.,  0.,  1.,  1.,  0.,  0.,  0.,  0.,
        0.,  0.,  2.,  0.,  0.,  0.,  0.,  0.,  1.,  0.,  0.,  0.,  0.,
        0.,  0.,  0.,  0.,  0.,  3.,  1.,  0.,  0.,  0.,  0.,  0.,  0.,
        0.,  0.,  1.,  0.,  0.,  0.,  1.,  2.,  2.])

labels, counts = np.unique(arr, return_counts=True)
plt.bar(labels, counts, align='center')
plt.gca().set_xticks(labels)
plt.show()
```


Question:
undo or reverse argsort(), python Given an array 'a' I would like to sort the array by columns sort(a, axis=0) do some stuff to the array and then undo the sort. By that I don't mean re sort but basically reversing how each element was moved. I assume argsort() is what I need but it is not clear to me how to sort an array with the results of argsort() or more importantly apply the reverse/inverse of argsort() Here is a little more detail I have an array a, shape(a) = rXc I need to sort each column
```
aargsort = a.argsort(axis=0)  # May use this later
aSort = a.sort(axis=0)
```
now average each row
```
aSortRM = asort.mean(axis=1)
```
now replace each col in a row with the row mean. is there a better way than this
```
aWithMeans = ones_like(a)
for ind in range(r)  # r = number of rows
    aWithMeans[ind]* aSortRM[ind]
```
Now I need to undo the sort I did in the first step. ????

Response:
There are probably better solutions to the problem you are actually trying to solve than this (performing an argsort usually precludes the need to actually sort), but here you go:
```

```python
import numpy as np
```

```python
a = np.random.randint(0,10,10)
```

```python
aa = np.argsort(a)
```

```python
aaa = np.argsort(aa)
```

```python
a # original
#Output
#array([6, 4, 4, 6, 2, 5, 4, 0, 7, 4])
```

```python
a[aa] # sorted
#Output
#array([0, 2, 4, 4, 4, 4, 5, 6, 6, 7])
```

```python
a[aa][aaa] # undone
#Output
#array([6, 4, 4, 6, 2, 5, 4, 0, 7, 4])
#```
```


Question:
Accessing NumPy array elements not in a given index list I have a NumPy array with the shape (100, 170, 256). And I have an array consisting of indexes [0, 10, 20, 40, 70]. I can get the sub-arrays corresponding to the indexes as follows:
```
sub_array = array[..., index]
```
This returns an array with the shape (100, 170, 5) as expected. Now, I am trying to take the complement and get the sub-array NOT corresponding to those indexes. So, I did:
```
sub_array = array[..., ~index]
```
This still returns me an array of shape (100, 170, 5) for some reason. I wonder how to do this complement operation of these indexes in python? [EDIT] Also tried:
```
sub_array = array[..., not(index.any)]
```
However, this does not do the thing I want as well (getting array of shape (100, 170, 251).

Response:
The question is answered but I propose a benchmark of the three methods here. Fastest solution is boolean mask (with small and larger index array size)
```
mask = np.ones(arr.size, dtype=bool)
mask[indexes] = False
result = arr[mask]
```
It is 2000 times faster than the list comprehension and marginaly faster than np.delete Code to reproduce Three proposed solutions: list comprehension (sol1), boolean mask (sol2) or np.delete (sol3)
```
d = 100000
a = np.random.rand(d)
idx = np.random.randint(d, size = 10)

# list comprehension
def sol1(arr, indexes):
    return arr[[i for i in range(arr.size) if i not in indexes]]
sol1(a, idx)
# Out[30]: array([0.13044518, 0.68564961, 0.03033223, ..., 0.03796257, 0.40137137, 0.45403929])

# boolean mask
def sol2(arr, indexes):
    mask = np.ones(arr.size, dtype=bool)
    mask[indexes] = False
    return arr[mask]
sol2(a, idx)
# Out[32]: array([0.13044518, 0.68564961, 0.03033223, ..., 0.03796257, 0.40137137, 0.45403929])

# np.delete
def sol3(arr, indexes):
    return np.delete(arr, indexes)
sol3(a, idx)
# Out[36]: array([0.13044518, 0.68564961, 0.03033223, ..., 0.03796257, 0.40137137, 0.45403929])
```
Results
```
%timeit sol1(a, idx)
384 ms ± 2.75 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
%timeit sol2(a, idx)
154 µs ± 15.7 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
%timeit sol3(a, idx)
194 µs ± 18.8 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

idx = np.random.randint(d, size = 1000)
%timeit sol1(a, idx)
386 ms ± 7.75 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
%timeit sol2(a, idx)
171 µs ± 11.3 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
%timeit sol3(a, idx)
205 µs ± 10.5 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
```


Question:
Creating a Distance Matrix? I am currently reading in data into a dataframe that looks like this.
```
City         XCord    YCord
Boston         5        2
Phoenix        7        3
New York       8        1
.....          .        .
```
I want to to create a Euclidean Distance Matrix from this data showing the distance between all city pairs so I get a resulting matrix like:
```
Boston    Phoenix   New York
Boston         0        2.236      3.162
Phoenix        2.236      0        2.236
New York       3.162    2.236        0
```
There are many more cities and coordinates in my actual data frame so i need to to be able to somehow iterate over all of the city pairs and create a distance matrix like the one I have shown above but I am not sure how to pair all of the cites together and apply the Euclidean Distance formula? Any help would be appreciated.

Response:
I think you are intrested in distance_matrix. For example: Create data:
```
import pandas as pd
from scipy.spatial import distance_matrix

data = [[5, 7], [7, 3], [8, 1]]
ctys = ['Boston', 'Phoenix', 'New York']
df = pd.DataFrame(data, columns=['xcord', 'ycord'], index=ctys)
```
Output:
```
xcord ycord
Boston      5   7
Phoenix     7   3
New York    8   1
```
Using the distance matrix function:
```
pd.DataFrame(distance_matrix(df.values, df.values), index=df.index, columns=df.index)
```
Results:
```
Boston    Phoenix     New York
Boston    0.000000  4.472136    6.708204
Phoenix   4.472136  0.000000    2.236068
New York  6.708204  2.236068    0.000000
```


Question:
How to force zero interception in linear regression? I have some more or less linear data of the form:
```
x = [0.1, 0.2, 0.4, 0.6, 0.8, 1.0, 2.0, 4.0, 6.0, 8.0, 10.0, 20.0, 40.0, 60.0, 80.0]
y = [0.50505332505407008, 1.1207373784533172, 2.1981844719020001, 3.1746209003398689, 4.2905482471260044, 6.2816226678076958, 11.073788414382639, 23.248479770546009, 32.120462301367183, 44.036117671229206, 54.009003143831116, 102.7077685684846, 185.72880217806673, 256.12183145545811, 301.97120103079675]
```
I am using scipy.optimize.leastsq to fit a linear regression to this:
```
def lin_fit(x, y):
    '''Fits a linear fit of the form mx+b to the data'''
    fitfunc = lambda params, x: params[0] * x + params[1]    #create fitting function of form mx+b
    errfunc = lambda p, x, y: fitfunc(p, x) - y              #create error function for least squares fit

    init_a = 0.5                            #find initial value for a (gradient)
    init_b = min(y)                         #find initial value for b (y axis intersection)
    init_p = numpy.array((init_a, init_b))  #bundle initial values in initial parameters

    #calculate best fitting parameters (i.e. m and b) using the error function
    p1, success = scipy.optimize.leastsq(errfunc, init_p.copy(), args = (x, y))
    f = fitfunc(p1, x)          #create a fit with those parameters
    return p1, f
```
And it works beautifully (although I am not sure if scipy.optimize is the right thing to use here, it might be a bit over the top?). However, due to the way the data points lie it does not give me a y-axis interception at 0. I do know though that it has to be zero in this case, if x = 0 than y = 0. Is there any way I can force this?

Response:
As @AbhranilDas mentioned, just use a linear method. There's no need for a non-linear solver like scipy.optimize.lstsq. Typically, you'd use numpy.polyfit to fit a line to your data, but in this case you'll need to do use numpy.linalg.lstsq directly, as you want to set the intercept to zero. As a quick example:
```
import numpy as np
import matplotlib.pyplot as plt

x = np.array([0.1, 0.2, 0.4, 0.6, 0.8, 1.0, 2.0, 4.0, 6.0, 8.0, 10.0,
              20.0, 40.0, 60.0, 80.0])

y = np.array([0.50505332505407008, 1.1207373784533172, 2.1981844719020001,
              3.1746209003398689, 4.2905482471260044, 6.2816226678076958,
              11.073788414382639, 23.248479770546009, 32.120462301367183,
              44.036117671229206, 54.009003143831116, 102.7077685684846,
              185.72880217806673, 256.12183145545811, 301.97120103079675])

# Our model is y = a * x, so things are quite simple, in this case...
# x needs to be a column vector instead of a 1D vector for this, however.
x = x[:,np.newaxis]
a, _, _, _ = np.linalg.lstsq(x, y)

plt.plot(x, y, 'bo')
plt.plot(x, a*x, 'r-')
plt.show()
```


Question:
How can you turn an index array into a mask array in Numpy? Is it possible to convert an array of indices to an array of ones and zeros, given the range? i.e. [2,3] -> [0, 0, 1, 1, 0], in range of 5 I'm trying to automate something like this:
```

```python
index_array = np.arange(200,300)
#Output
#array([200, 201, ... , 299])
```

```python
mask_array = ???           # some function of index_array and 500
#Output
#array([0, 0, 0, ..., 1, 1, 1, ... , 0, 0, 0])
```

```python
train(data[mask_array])    # trains with 200~299
```

```python
predict(data[~mask_array]) # predicts with 0~199, 300~499
#Output
#```
```

Response:
Here's one way:
```

```python
index_array = np.array([3, 4, 7, 9])
```

```python
n = 15
```

```python
mask_array = np.zeros(n, dtype=int)
```

```python
mask_array[index_array] = 1
```

```python
mask_array
#Output
#array([0, 0, 0, 1, 1, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0])
#```
#If the mask is always a range, you can eliminate index_array, and assign 1 to a slice:
#```
```

```python
mask_array = np.zeros(n, dtype=int)
```

```python
mask_array[5:10] = 1
```

```python
mask_array
#Output
#array([0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0])
#```
#If you want an array of boolean values instead of integers, change the dtype of mask_array when it is created:
#```
```

```python
mask_array = np.zeros(n, dtype=bool)
```

```python
mask_array
#Output
#array([False, False, False, False, False, False, False, False, False,
#       False, False, False, False, False, False], dtype=bool)
```

```python
mask_array[5:10] = True
```

```python
mask_array
#Output
#array([False, False, False, False, False,  True,  True,  True,  True,
#        True, False, False, False, False, False], dtype=bool)
#```
```


Question:
Overcome ValueError for empty array In this discussion I tried to fix an issue in plotting limits for y-axis, after the twiny() messes up my plot. I thought this:
```
ax.set_ylim([y.min()-0.05, y.max()+0.05])
```
was a good solution. And probably it is, for continuous set of data. As I said in that discussion, anyway, my data are noisy, and sometimes with gaps. So it happens that some plotted ranges have no data. In that case, naturally, the use of the .min() raises the error:
```
ValueError: zero-size array to reduction operation minimum which has no identity
```
because the array is empty. How to work around it, so that the code just does not care about putting limits on the y-axis? (Hoping that this is the only issue the empty array will cause)

Response:
Just catch the exception and ignore it:
```
try:
    ax.set_ylim([y.min()-0.05, y.max()+0.05])
except ValueError:  #raised if `y` is empty.
    pass
```


Question:
in Numpy, how to zip two 2-D arrays? For example I have 2 arrays
```
a = array([[0, 1, 2, 3],
           [4, 5, 6, 7]])
b = array([[0, 1, 2, 3],
           [4, 5, 6, 7]])
```
How can I zip a and b so I get
```
c = array([[(0,0), (1,1), (2,2), (3,3)],
           [(4,4), (5,5), (6,6), (7,7)]])
```
?

Response:
You can use dstack:
```

```python
np.dstack((a,b))
#Output
#array([[[0, 0],
#        [1, 1],
#        [2, 2],
#        [3, 3]],
```

[[4, 4],
        [5, 5],
        [6, 6],
        [7, 7]]])
```
If you must have tuples:
```

```python
np.array(zip(a.ravel(),b.ravel()), dtype=('i4,i4')).reshape(a.shape)
#Output
#array([[(0, 0), (1, 1), (2, 2), (3, 3)],
#       [(4, 4), (5, 5), (6, 6), (7, 7)]],
#      dtype=[('f0', '<i4'), ('f1', '<i4')])
#```
#For Python 3+ you need to expand the zip iterator object. Please note that this is horribly inefficient:
#```
```

```python
np.array(list(zip(a.ravel(),b.ravel())), dtype=('i4,i4')).reshape(a.shape)
#Output
#array([[(0, 0), (1, 1), (2, 2), (3, 3)],
#       [(4, 4), (5, 5), (6, 6), (7, 7)]],
#      dtype=[('f0', '<i4'), ('f1', '<i4')])
#```
```


Question:
How do I stack vectors of different lengths in NumPy? How do I stack column-wise n vectors of shape (x,) where x could be any number? For example,
```
from numpy import *
a = ones((3,))
b = ones((2,))

c = vstack((a,b)) # <-- gives an error
c = vstack((a[:,newaxis],b[:,newaxis])) #<-- also gives an error
```
hstack works fine but concatenates along the wrong dimension.

Response:
Short answer: you can't. NumPy does not support jagged arrays natively. Long answer:
```

```python
a = ones((3,))
```

```python
b = ones((2,))
```

```python
c = array([a, b])
```

```python
c
#Output
#array([[ 1.  1.  1.], [ 1.  1.]], dtype=object)
#```
#gives an array that may or may not behave as you expect. E.g. it doesn't support basic methods like sum or reshape, and you should treat this much as you'd treat the ordinary Python list [a, b] (iterate over it to perform operations instead of using vectorized idioms). Several possible workarounds exist; the easiest is to coerce a and b to a common length, perhaps using masked arrays or NaN to signal that some indices are invalid in some rows. E.g. here's b as a masked array:
#```
```

```python
ma.array(np.resize(b, a.shape[0]), mask=[False, False, True])
#Output
#masked_array(data = [1.0 1.0 --],
#             mask = [False False  True],
#       fill_value = 1e+20)
#```
#This can be stacked with a as follows:
#```
```

```python
ma.vstack([a, ma.array(np.resize(b, a.shape[0]), mask=[False, False, True])])
#Output
#masked_array(data =
# [[1.0 1.0 1.0]
# [1.0 1.0 --]],
#             mask =
# [[False False False]
# [False False  True]],
#       fill_value = 1e+20)
#```
#(For some purposes, scipy.sparse may also be interesting.)
```


Question:
Numpy.dot TypeError: Cannot cast array data from dtype('float64') to dtype('S32') according to the rule 'safe' Why am I getting this error when using np.dot(a,b.T):
```
TypeError: Cannot cast array data from dtype('float64')
               to dtype('S32') according to the rule 'safe'
```
a and b are of type numpy.ndarray. My NumPy version is 1.11.0.

Response:
Just taking the input from BrenBarn and Warren Weckesser to provide a code snippet which should run (by converting your strings to float):
```
a = map(lambda x: float(x),a)
b = map(lambda x: float(x),b)
np.dot(a,b.T)
```
or simpler as suggested by @JLT
```
a = map(float,a)
b = map(float,b)
np.dot(a,b.T)
```
But as Warren Weckesser already said, you should check the types of the array, most likely one already contains floats.


Question:
Why do I get "ufunc 'multiply' did not contain a loop with signature matching types dtype('S32') dtype('S32') dtype('S32')" with values from raw_input I am trying to create a really simple program that will plot a plot a parabola where v is velocity, a is acceleration and x is time. The user will input values for v and a, then v and a and x will determine y. I attempted to do this with this:
```
x = np.linspace(0., 9., 10)
a = raw_input('Acceleration =')
v = raw_input('Velocity = ')
y = v * x - 0.5 * a * x**2.
```
But, I keep getting this error: TypeError: ufunc 'multiply' did not contain a loop with signature matching types dtype('S32') dtype('S32') dtype('S32') What does this mean?

Response:
From the documentation of raw_input: The function then reads a line from input, converts it to a string (stripping a trailing newline), and returns that. So what happens is that you try to multiply a string with a float, something like y="3" * x - 0.5 * "3" *x**2, which is not defined. The easiest way to circumvent this is to cast the input string to float first.
```
x = np.linspace(0., 9., 10)
a = float(raw_input('Acceleration ='))
v = float(raw_input('Velocity = '))
y = v * x - 0.5 * a * x**2
```
Mind that if you're using Python 3, you'd need to use input instead of raw_input,
```
a = float(input('Acceleration ='))
```


Question:
How to find the index of a value in 2d array in Python? I need to figure out how I can find all the index of a value in a 2d numpy array. For example, I have the following 2d array:
```
([[1 1 0 0],
  [0 0 1 1],
  [0 0 0 0]])
```
I need to find the index of all the 1's and 0's.
```
1: [(0, 0), (0, 1), (1, 2), (1, 3)]
0: [(0, 2), (0, 3), (1, 0), (1, 1), (the entire all row)]
```
I tried this but it doesn't give me all the indexes:
```
t = [(index, row.index(1)) for index, row in enumerate(x) if 1 in row]
```
Basically, it gives me only one of the index in each row [(0, 0), (1, 2)].

Response:
You can use np.where to return a tuple of arrays of x and y indices where a given condition holds in an array. If a is the name of your array:
```

```python
np.where(a == 1)
#Output
#(array([0, 0, 1, 1]), array([0, 1, 2, 3]))
#```
#If you want a list of (x, y) pairs, you could zip the two arrays:
#```
```

```python
list(zip(*np.where(a == 1)))
#Output
#[(0, 0), (0, 1), (1, 2), (1, 3)]
#```
#Or, even better, @jme points out that np.asarray(x).T can be a more efficient way to generate the pairs.
```


Question:
Multilabel-indicator is not supported for confusion matrix multilabel-indicator is not supported is the error message I get, when trying to run: confusion_matrix(y_test, predictions) y_test is a DataFrame which is of shape:
```
Horse | Dog | Cat
1       0     0
0       1     0
0       1     0
...     ...   ...
```
predictions is a numpy array:
```
[[1, 0, 0],
 [0, 1, 0],
 [0, 1, 0]]
```
I've searched a bit for the error message, but haven't really found something I could apply. Any hints?

Response:
No, your input to confusion_matrix must be a list of predictions, not OHEs (one hot encodings). Call argmax on your y_test and y_pred, and you should get what you expect.
```
confusion_matrix(
    y_test.values.argmax(axis=1), predictions.argmax(axis=1))

array([[1, 0],
       [0, 2]])
```


Question:
Get coordinates of local maxima in 2D array above certain value
```
from PIL import Image
import numpy as np
from scipy.ndimage.filters import maximum_filter
import pylab

# the picture (256 * 256 pixels) contains bright spots of which I wanna get positions
# problem: data has high background around value 900 - 1000

im = Image.open('slice0000.png')
data = np.array(im)

# as far as I understand, data == maximum_filter gives True-value for pixels
# being the brightest in their neighborhood (here 10 * 10 pixels)

maxima = (data == maximum_filter(data,10))
# How can I get only maxima, outstanding the background a certain value, let's say 500 ?
```
I'm afraid I don't really understand the scipy.ndimage.filters.maximum_filter() function. Is there a way to obtain pixel-coordinates only within the spots and not within the background? https://i.sstatic.net/RImHW.png (16-bit grayscale picture, 256*256 pixels)

Response:
```
import numpy as np
import scipy
import scipy.ndimage as ndimage
import matplotlib.pyplot as plt

fname = '/tmp/slice0000.png'
neighborhood_size = 5
threshold = 1500

data = scipy.misc.imread(fname)

data_max = ndimage.maximum_filter(data, neighborhood_size) # apply maximum filter with a size of neighborhood_size
maxima = (data == data_max) # boolean mask: local maximum within neighborhood_size
data_min = ndimage.minimum_filter(data, neighborhood_size) # apply minimum filter with a size of neighborhood_size
diff = ((data_max - data_min) > threshold) # boolean mask where the difference of the filters exceeds threshold
maxima[~diff] = False # remove the local maxima which do not satisfy the minimum difference in neighborhood

labeled, num_objects = ndimage.label(maxima) # label connected components on maxima binary array (boolean mask)
slices = ndimage.find_objects(labeled) # slices are 2d rect
x, y = [], []
for dy,dx in slices:
    x_center = (dx.start + dx.stop - 1)/2
    x.append(x_center)
    y_center = (dy.start + dy.stop - 1)/2
    y.append(y_center)

plt.imshow(data)
plt.savefig('/tmp/data.png', bbox_inches = 'tight')

plt.autoscale(False)
plt.plot(x,y, 'ro')
plt.savefig('/tmp/result.png', bbox_inches = 'tight')
```
Given data.png: the above program yields result.png with threshold = 1500. Lower the threshold to pick up more local maxima: References: J.F. Sebastian counts nuclei Joe Kington finds paw prints Ivan finds local maximums


Question:
numpy "TypeError: ufunc 'bitwise_and' not supported for the input types" when using a dynamically created boolean mask In numpy, if I have an array of floats, dynamically create a boolean mask of where this array equals a particular value and do a bitwise AND with a boolean array, I get an error:
```

```python
import numpy as np
```

```python
a = np.array([1.0, 2.0, 3.0])
```

```python
a == 2.0 & b
```

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: ufunc 'bitwise_and' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe'
```
If I save the result of the comparison to a variable and carry out the bitwise AND however, it works:
```

```python
c = a == 2.0
```

```python
c & b
#Output
#array([False,  True, False], dtype=bool)
#```
#The objects created seem the same in each case though:
#```
```

```python
type(a == 2.0)
#Output
#<type 'numpy.ndarray'>
```

```python
(a == 2.0).dtype
#Output
#dtype('bool')
```

```python
type(c)
#Output
#<type 'numpy.ndarray'>
```

```python
c.dtype
#Output
#dtype('bool')
#```
#Why the difference?
```

Response:
& has higher precedence than ==, so the expression
```
a == 2.0 & b
```
is the same as
```
a == (2.0 & b)
```
You get the error because bitwise and is not defined for a floating point scalar and a boolean array. Add parentheses to get what you expected:
```
(a == 2.0) & b
```


Question: Inserting a row at a specific location in a 2d array in numpy? I have a 2d array in numpy where I want to insert a new row. Following question Numpy - add row to array can help. We can use numpy.vstack, but it stacks at the start or at the end. Can anyone please help in this regard.

Response:
You are probably looking for numpy.insert
```

```python
import numpy as np
```

```python
a = np.zeros((2, 2))
```

```python
a
#Output
#array([[ 0.,  0.],
#       [ 0.,  0.]])
## In the following line 1 is the index before which to insert, 0 is the axis.
```

```python
np.insert(a, 1, np.array((1, 1)), 0)
#Output
#array([[ 0.,  0.],
#       [ 1.,  1.],
#       [ 0.,  0.]])
```

```python
np.insert(a, 1, np.array((1, 1)), 1)
#Output
#array([[ 0.,  1.,  0.],
#       [ 0.,  1.,  0.]])
#```
```


Question:
Efficient way to compute intersecting values between two numpy arrays I have a bottleneck in my program which is caused by the following:
```
A = numpy.array([10,4,6,7,1,5,3,4,24,1,1,9,10,10,18])
B = numpy.array([1,4,5,6,7,8,9])

C = numpy.array([i for i in A if i in B])
```
The expected outcome for C is the following:
```
C = [4 6 7 1 5 4 1 1 9]
```
Is there a more efficient way of doing this operation? Note that array A contains repeating values and they need to be taken into account. I wasn't able to use set intersection since taking the intersection will omit the repeating values, returning just [1,4,5,6,7,9]. Also note this is only a simple demonstration. The actual array sizes can be in the order of thousands, to well over millions.

Response:
You can use np.in1d:
```

```python
A[np.in1d(A, B)]
#Output
#array([4, 6, 7, 1, 5, 4, 1, 1, 9])
#```
#np.in1d returns a boolean array indicating whether each value of A also appears in B. This array can then be used to index A and return the common values. It's not relevant to your example, but it's also worth mentioning that if A and B each contain unique values then np.in1d can be sped up by setting assume_unique=True:
#```
#np.in1d(A, B, assume_unique=True)
#```
#You might also be interested in np.intersect1d which returns an array of the unique values common to both arrays (sorted by value):
#```
```

```python
np.intersect1d(A, B)
#Output
#array([1, 4, 5, 6, 7, 9])
#```
```


Question:
Merge two arrays vertically to array of tuples using numpy I have two numpy arrays:
```py
x = np.array([-1, 0, 1, 2])
y = np.array([-2, -1, 0, 1])
```
Is there a way to merge these arrays together like tuples:
```py
array = [(-1, -2), (0, -1), (1, 0), (2, 1)]
```

Response:
```

```python
x = np.array([-1, 0, 1, 2])
```

```python
y = np.array([-2, -1, 0, 1])
```
join them into 2d array:
```
```

```python
np.array((x,y))
#Output
#array([[-1,  0,  1,  2],
#       [-2, -1,  0,  1]])
#```
#transpose that array:
#```
```

```python
np.array((x,y)).T
#Output
#array([[-1, -2],
#       [ 0, -1],
#       [ 1,  0],
#       [ 2,  1]])
#```
#or use the standard Python zip - this treats the arrays as lists
#```
```

```python
zip(x,y)   # list(zip in py3
#Output
#[(-1, -2), (0, -1), (1, 0), (2, 1)]
#```
```


Question: How to create a numpy array from a pydub AudioSegment? I'm aware of the following question: How to create a pydub AudioSegment using an numpy array? My question is the right opposite. If I have a pydub AudioSegment how can I convert it to a numpy array? I would like to use scipy filters and so on. It is not very clear to me what is the internal structure of the AudioSegment raw data.

Response:
Pydub has a facility for getting the audio data as an array of samples, it is an array.array instance (not a numpy array) but you should be able to convert it to a numpy array relatively easily:
```
from pydub import AudioSegment
sound = AudioSegment.from_file("sound1.wav")

# this is an array
samples = sound.get_array_of_samples()
```
You may be able to create a numpy variant of the implementation though. That method is implemented pretty simply:
```
def get_array_of_samples(self):
    """
    returns the raw_data as an array of samples
    """
    return array.array(self.array_type, self._data)
```
Creating a new audio segment from a (modified?) array of samples is also possible:
```
new_sound = sound._spawn(samples)
```
The above is a little hacky, it was written for internal use within the AudioSegment class, but it mainly just figures out what type of audio data you're using (array of samples, list of samples, bytes, bytestring, etc). It's safe to use despite the underscore prefix.


Question:
Replace the zeros in a NumPy integer array with nan I wrote a python script below:
```py
import numpy as np

arr = np.arange(6).reshape(2, 3)
arr[arr==0]=['nan']
print arr
```
But I got this error:
```none
Traceback (most recent call last):
  File "C:\Users\Desktop\test.py", line 4, in <module>
    arr[arr==0]=['nan']
ValueError: invalid literal for long() with base 10: 'nan'
[Finished in 0.2s with exit code 1]
```
How to replace zeros in a NumPy array with nan?

Response:
np.nan has type float: arrays containing it must also have this datatype (or the complex or object datatype) so you may need to cast arr before you try to assign this value. The error arises because the string value 'nan' can't be converted to an integer type to match arr's type.
```

```python
arr = arr.astype('float')
```

```python
arr[arr == 0] = 'nan' # or use np.nan
```

```python
arr
#Output
#array([[ nan,   1.,   2.],
#       [  3.,   4.,   5.]])
#```
```


Question: numpy bytes to plain string I have a numpy array X with dtype 'S' (numpy.bytes_). For example printing print(X[0, 0]) yields b'somestring'. Similarly str(X[0, 0]) returns string "b'somestring'". However I need to print or convert to string so that it does not contain b' at the beginning and ' at the end. I just want to print somestring or return a string "somestring". How to do it? Note: I cannot change the type of the array.

Response:
You just need to decode the string back into ASCII, so it would just be:
```
bytes_string.decode('UTF-8')
```
Demo:
```

```python
b'somestring'.decode('UTF-8')
#Output
#'somestring'
#```
```


Question: numpy and Global Interpreter Lock I am about to write some computationally-intensive Python code that'll almost certainly spend most of its time inside numpy's linear algebra functions. The problem at hand is embarrassingly parallel. Long story short, the easiest way for me to take advantage of that would be by using multiple threads. The main barrier is almost certainly going to be the Global Interpreter Lock (GIL). To help design this, it would be useful to have a mental model for which numpy operations can be expected to release the GIL for their duration. To this end, I'd appreciate any rules of thumb, dos and don'ts, pointers etc. In case it matters, I'm using 64-bit Python 2.7.1 on Linux, with numpy 1.5.1 and scipy 0.9.0rc2, built with Intel MKL 10.3.1.

Response: Quite some numpy routines release GIL, so they can be efficiently parallel in threads (info). Maybe you don't need to do anything special! You can use this question to find whether the routines you need are among the ones that release GIL. In short, search for ALLOW_THREADS or nogil in the source. (Also note that MKL has the ability to use multiple threads for a routine, so that's another easy way to get parallelism, although possibly not the fastest kind).


Question:
Overflow Error in Python's numpy.exp function I want to use numpy.exp like this:
```
cc = np.array([
    [0.120,0.34,-1234.1]
])

print 1/(1+np.exp(-cc))
```
But this gives me error:
```
/usr/local/lib/python2.7/site-packages/ipykernel/__main__.py:5: RuntimeWarning: overflow encountered in exp
```
I can't understand why? How can I fix this? It seems the problem is with third number (-1234.1)

Response:
As fuglede says, the issue here is that np.float64 can't handle a number as large as exp(1234.1). Try using np.float128 instead:
```

```python
cc = np.array([[0.120,0.34,-1234.1]], dtype=np.float128)
```

```python
cc
#Output
#array([[ 0.12,  0.34, -1234.1]], dtype=float128)
```

```python
1 / (1 + np.exp(-cc))
#Output
#array([[ 0.52996405,  0.58419052,  1.0893812e-536]], dtype=float128)
#```
#Note however, that there are certain quirks with using extended precision. It may not work on Windows; you don't actually get the full 128 bits of precision; and you might lose the precision whenever the number passes through pure python. You can read more about the details here. For most practical purposes, you can probably approximate 1 / (1 + ) to zero. That is to say, just ignore the warning and move on. Numpy takes care of the approximation for you (when using np.float64):
#```
```

```python
1 / (1 + np.exp(-cc))
#Output
#/usr/local/bin/ipython3:1: RuntimeWarning: overflow encountered in exp
#  #!/usr/local/bin/python3.4
#array([[ 0.52996405,  0.58419052,  0.        ]])
#```
#If you want to suppress the warning, you could use scipy.special.expit, as suggested by WarrenWeckesser in a comment to the question:
#```
```

```python
from scipy.special import expit
```

```python
expit(cc)
#Output
#array([[ 0.52996405,  0.58419052,  0.        ]])
#```
```


Question: How to delete an object from a numpy array without knowing the index Is it possible to delete an object from a numpy array without knowing the index of the object but instead knowing the object itself? I have seen that it is possible using the index of the object using the np.delete function, but I'm looking for a way to do it having the object but not its index. Example: [a,b,c,d,e,f] x = e I would like to delete x.

Response:
You can find the index/indices of the object using np.argwhere, and then delete the object(s) using np.delete. Example:
```
x = np.array([1,2,3,4,5])
index = np.argwhere(x==3)
y = np.delete(x, index)
print(x, y)
```


Question:
How to detect a sign change for elements in a numpy array I have a numpy array with positive and negative values in.
```
a = array([1,1,-1,-2,-3,4,5])
```
I want to create another array which contains a value at each index where a sign change occurs (For example, if the current element is positive and the previous element is negative and vice versa). For the array above, I would expect to get the following result
```
array([0,0,1,0,0,1,0])
```
Alternatively, a list of the positions in the array where the sign changes occur or list of booleans instead of 0's and 1's is fine.

Response:
Something like
```
a = array([1,1,-1,-2,-3,4,5])
asign = np.sign(a)
signchange = ((np.roll(asign, 1) - asign) != 0).astype(int)
print signchange
array([0, 0, 1, 0, 0, 1, 0])
```
Now, numpy.roll does a circular shift, so if the last element has different sign than the first, the first element in the signchange array will be 1. If this is not desired, one can of course do a simple
```
signchange[0] = 0
```
Also, np.sign considers 0 to have it's own sign, different from either positive or negative values. E.g. the "signchange" array for [-1,0,1] would be [0,1,1] even though the zero line was "crossed" only once. If this is undesired, one could insert the lines
```
sz = asign == 0
while sz.any():
    asign[sz] = np.roll(asign, 1)[sz]
    sz = asign == 0
```
between lines 2 and 3 in the first example.


Question:
Equivalent of j in NumPy What is the equivalent of Octave's j in NumPy? How can I use j in Python? In Octave:
```
octave:1> j
ans =  0 + 1i
octave:1> j*pi/4
ans =  0.00000 + 0.78540i
```
But in Python:
```

```python
import numpy as np
```

```python
np.imag
#Output
#<function imag at 0x2368140>
```

```python
np.imag(3)
#Output
#array(0)
```

```python
np.imag(3,2)
#Output
#Traceback (most recent call last):
#  File "<stdin>", line 1, in <module>
#TypeError: imag() takes exactly 1 argument (2 given)
```

```python
np.imag(32)
#Output
#array(0)
```

```python

```

```python
0+np.imag(1)
#Output
#1
#```
```

Response:
In Python, 1j or 0+1j is a literal of complex type. You can broadcast that into an array using expressions, for example
```

```python
1j * np.arange(5)
#Output
#array([ 0.+0.j,  0.+1.j,  0.+2.j,  0.+3.j,  0.+4.j])
#```
#Create an array from literals:
#```
```

```python
np.array([1j])
#Output
#array([ 0.+1.j])
#```
#Note that what Michael9 posted creates a complex, not a complex array:
#```
```

```python
np.complex(0,1)
#Output
#1j
```

```python
type(_)
#Output
#complex
#```
```


Question:
Finding indices of matches of one array in another array I have two numpy arrays, A and B. A conatains unique values and B is a sub-array of A. Now I am looking for a way to get the index of B's values within A. For example:
```
A = np.array([1,2,3,4,5,6,7,8,9,10])
B = np.array([1,7,10])
# I need a function fun() that:
fun(A,B)
>> 0,6,9
```

Response:
You can use np.in1d with np.nonzero -
```
np.nonzero(np.in1d(A,B))[0]
```
You can also use np.searchsorted, if you care about maintaining the order -
```
np.searchsorted(A,B)
```
For a generic case, when A & B are unsorted arrays, you can bring in the sorter option in np.searchsorted, like so -
```
sort_idx = A.argsort()
out = sort_idx[np.searchsorted(A,B,sorter = sort_idx)]
```
I would add in my favorite broadcasting too in the mix to solve a generic case -
```
np.nonzero(B[:,None] == A)[1]
```
Sample run -
```

```python
A
#Output
#array([ 7,  5,  1,  6, 10,  9,  8])
```

```python
B
#Output
#array([ 1, 10,  7])
```

```python
sort_idx = A.argsort()
```

```python
sort_idx[np.searchsorted(A,B,sorter = sort_idx)]
#Output
#array([2, 4, 0])
```

```python
np.nonzero(B[:,None] == A)[1]
#Output
#array([2, 4, 0])
#```
```


Question:
Rounding to nearest int with numpy.rint() not consistent for .5 numpy's round int doesn't seem to be consistent with how it deals with xxx.5
```

```python
np.rint(1.5)
#Output
#2.0
```

```python
np.rint(10.5)
#Output
#10.0
#```
#1.5 is rounded up while 10.5 is rounded down. Is there a reason for this? Is it just and artifact of the inaccuracy of floats? Edit Is there a way to get the desired functionality where n.5 is rounded up i.e. to n+1 for both n = even or odd?
```

Response:
So, this kind of behavior (as noted in comments), is a very traditional form of rounding, seen in the round half to even method. Also known (according to David Heffernan) as banker's rounding. The numpy documentation around this behavior implies that they are using this type of rounding, but also implies that there may be issues with the way in which numpy interacts with the IEEE floating point format. (shown below)
```
Notes
-----
For values exactly halfway between rounded decimal values, Numpy
rounds to the nearest even value. Thus 1.5 and 2.5 round to 2.0,
-0.5 and 0.5 round to 0.0, etc. Results may also be surprising due
to the inexact representation of decimal fractions in the IEEE
floating point standard [1]_ and errors introduced when scaling
by powers of ten.
```
Whether or not that is the case, I honestly don't know. I do know that large portions of the numpy core are still written in FORTRAN 77, which predates the IEEE standard (set in 1984), but I don't know enough FORTRAN 77 to say whether or not there's some issue with the interface here. If you're looking to just round up regardless, the np.ceil function (ceiling function in general), will do this. If you're looking for the opposite (always rounding down), the np.floor function will achieve this.


Question:
Why does indexing numpy arrays with brackets and commas differ in behavior? I tend to index numpy arrays (matrices) with brackets, but I've noticed when I want to slice an array (matrix) I must use the comma notation. Why is this? For example,
```

```python
x = numpy.array([[1, 2], [3, 4], [5, 6]])
```

```python
x
#Output
#array([[1, 2],
#       [3, 4],
#       [5, 6]])
```

```python
x[1][1]
#Output
#4                 # expected behavior
```

```python
x[1,1]
#Output
#4                 # expected behavior
```

```python
x[:][1]
#Output
#array([3, 4])     # huh?
```

```python
x[:,1]
#Output
#array([2, 4, 6])  # expected behavior
#```
```

Response:
This:
```
x[:, 1]
```
means "take all indices of x along the first axis, but only index 1 along the second". This:
```
x[:][1]
```
means "take all indices of x along the first axis (so all of x), then take index 1 along the first axis of the result". You're applying the 1 to the wrong axis. x[1][2] and x[1, 2] are only equivalent because indexing an array with an integer shifts all remaining axes towards the front of the shape, so the first axis of x[1] is the second axis of x. This doesn't generalize at all; you should almost always use commas instead of multiple indexing steps.


Question:
Numpy: change max in each row to 1, all other numbers to 0 I'm trying to implement a numpy function that replaces the max in each row of a 2D array with 1, and all other numbers with zero:
```

```python
a = np.array([[0, 1],
              [2, 3],
              [4, 5],
              [6, 7],
              [9, 8]])
```

```python
b = some_function(a)
```

```python
b
#Output
#[[0. 1.]
# [0. 1.]
# [0. 1.]
# [0. 1.]
# [1. 0.]]
#```
#What I've tried so far
#```
#def some_function(x):
#    a = np.zeros(x.shape)
#    a[:,np.argmax(x, axis=1)] = 1
#    return a
```

```python
b = some_function(a)
```

```python
b
#Output
#[[1. 1.]
# [1. 1.]
# [1. 1.]
# [1. 1.]
# [1. 1.]]
#```
```

Response:
Method #1, tweaking yours:
```

```python
a = np.array([[0, 1], [2, 3], [4, 5], [6, 7], [9, 8]])
```

```python
b = np.zeros_like(a)
```

```python
b[np.arange(len(a)), a.argmax(1)] = 1
```

```python
b
#Output
#array([[0, 1],
#       [0, 1],
#       [0, 1],
#       [0, 1],
#       [1, 0]])
#```
#[Actually, range will work just fine; I wrote arange out of habit.] Method #2, using max instead of argmax to handle the case where multiple elements reach the maximum value:
#```
```

```python
a = np.array([[0, 1], [2, 2], [4, 3]])
```

```python
(a == a.max(axis=1)[:,None]).astype(int)
#Output
#array([[0, 1],
#       [1, 1],
#       [1, 0]])
#```
```


Question: What is numpy method int0? I've seen np.int0 used for converting bounding box floating point values to int in OpenCV problems. What exactly is np.int0? I've seen np.uint8, np.int32, etc. I can't seem to find np.int0 in any online documentation. What kind of int does this cast arguments to?

Response: int0 is an alias for intp; this, in turn, is Integer used for indexing (same as C ssize_t; normally either int32 or int64) -- Numpy docs: basic types


Question:
Numpy transpose of 1D array not giving expected result I am trying a very basic example in Python scipy module for transpose() method but it's not giving expected result. I am using Ipython with pylab mode.
```
a = array([1,2,3]
print a.shape
>> (3,)

b = a.transpose()
print b.shape
>> (3,)
```
If I print the contents of arrays "a" and "b", they are similar. Expectation is: (which will be result in Matlab on transpose)
```
[1,
  2,
  3]
```

Response:
NumPy's transpose() effectively reverses the shape of an array. If the array is one-dimensional, this means it has no effect. In NumPy, the arrays
```
array([1, 2, 3])
```
and
```
array([1,
       2,
       3])
```
are actually the same – they only differ in whitespace. What you probably want are the corresponding two-dimensional arrays, for which transpose() would work fine. Also consider using NumPy's matrix type:
```

```python
numpy.matrix([1, 2, 3])
#Output
#matrix([[1, 2, 3]])
```

```python
numpy.matrix([1, 2, 3]).T
#Output
#matrix([[1],
#        [2],
#        [3]])
#```
#Note that for most applications, the plain one-dimensional array would work fine as both a row or column vector, but when coming from Matlab, you might prefer using numpy.matrix.
```