Question: Difference between np.int, np.int_, int, and np.int_t in cython? I am a bit struggled with so many int data types in cython. np.int, np.int_, np.int_t, int I guess int in pure python is equivalent to np.int_, then where does np.int come from? I cannot find the document from numpy? Also, why does np.int_ exist given we do already have int? In cython, I guess int becomes a C type when used as cdef int or ndarray[int], and when used as int() it stays as the python caster? Is np.int_ equivalent to long in C? so cdef long is the identical to cdef np.int_? Under what circumstances should I use np.int_t instead of np.int? e.g. cdef np.int_t, ndarray[np.int_t] ... Can someone briefly explain how the wrong use of those types would affect the performance of compiled cython code?

Response:
It's a bit complicated because the names have different meanings depending on the context. int In Python The int is normally just a Python type, it's of arbitrary precision, meaning that you can store any conceivable integer inside it (as long as you have enough memory).
```

```python
int(10**50)
#Output
#    100000000000000000000000000000000000000000000000000
#```
#However, when you use it as dtype for a NumPy array it will be interpreted as np.int_ 1. Which is not of arbitrary precision, it will have the same size as C's long:
#```
```

```python
np.array(10**50, dtype=int)
#Output
# OverflowError: Python int too large to convert to C long
#```
#That also means the following two are equivalent:
#```
#np.array([1,2,3], dtype=int)
#    np.array([1,2,3], dtype=np.int_)
#```
#As Cython type identifier it has another meaning, here it stands for the c type int. It's of limited precision (typically 32bits). You can use it as Cython type, for example when defining variables with cdef:
#```
#cdef int value = 100    # variable
# cdef int[:] arr = ...   # memoryview
#```
#As return value or argument value for cdef or cpdef functions:
#```
#cdef int my_function(int argument1, int argument2):
#        # ...
#```
#As "generic" for ndarray:
#```
#cimport numpy as cnp
#    cdef cnp.ndarray[int, ndim=1] val = ...
#```
#For type casting:
#```
#avalue = <int>(another_value)
#```
#And probably many more. In Cython but as Python type. You can still call int and you'll get a "Python int" (of arbitrary precision), or use it for isinstance or as dtype argument for np.array. Here the context is important, so converting to a Python int is different from converting to a C int:
#```
#cdef object val = int(10)  # Python int
# cdef int val = <int>(10)   # C int
#```
#np.int Actually this is very easy. It's just an alias for int:
#```
```

```python
int is np.int
#Output
#True
#```
#So everything from above applies to np.int as well. However you can't use it as a type-identifier except when you use it on the cimported package. In that case it represents the Python integer type.
#```
#cimport numpy as cnp
```

cpdef func(cnp.int obj):
    return obj
```
This will expect obj to be a Python integer not a NumPy type:
```

```python
func(np.int_(10))
#Output
#TypeError: Argument 'obj' has incorrect type (expected int, got numpy.int32)
```

```python
func(10)
#Output
#10
#```
#My advise regarding np.int: Avoid it whenever possible. In Python code it's equivalent to int and in Cython code it's also equivalent to Pythons int but if used as type-identifier it will probably confuse you and everyone who reads the code! It certainly confused me... np.int_ Actually it only has one meaning: It's a Python type that represents a scalar NumPy type. You use it like Pythons int:
#```
```

```python
np.int_(10)        # looks like a normal Python integer
#Output
#10
```

```python
type(np.int_(10))  # but isn't (output may vary depending on your system!)
#Output
#numpy.int32
#```
#Or you use it to specify the dtype, for example with np.array:
#```
```

```python
np.array([1,2,3], dtype=np.int_)
#Output
#array([1, 2, 3])
#```
#But you cannot use it as type-identifier in Cython. cnp.int_t It's the type-identifier version for np.int_. That means you can't use it as dtype argument. But you can use it as type for cdef declarations:
#```
#cimport numpy as cnp
#import numpy as np
```

cdef cnp.int_t[:] arr = np.array([1,2,3], dtype=np.int_)
     |---TYPE---|                         |---DTYPE---|
```
This example (hopefully) shows that the type-identifier with the trailing _t actually represents the type of an array using the dtype without the trailing t. You can't interchange them in Cython code! Notes There are several more numeric types in NumPy I'll include a list containing the NumPy dtype and Cython type-identifier and the C type identifier that could also be used in Cython here. But it's basically taken from the NumPy documentation and the Cython NumPy pxd file:
```
NumPy dtype          Numpy Cython type         C Cython type identifier

np.bool_             None                      None
np.int_              cnp.int_t                 long
np.intc              None                      int
np.intp              cnp.intp_t                ssize_t
np.int8              cnp.int8_t                signed char
np.int16             cnp.int16_t               signed short
np.int32             cnp.int32_t               signed int
np.int64             cnp.int64_t               signed long long
np.uint8             cnp.uint8_t               unsigned char
np.uint16            cnp.uint16_t              unsigned short
np.uint32            cnp.uint32_t              unsigned int
np.uint64            cnp.uint64_t              unsigned long
np.float_            cnp.float64_t             double
np.float32           cnp.float32_t             float
np.float64           cnp.float64_t             double
np.complex_          cnp.complex128_t          double complex
np.complex64         cnp.complex64_t           float complex
np.complex128        cnp.complex128_t          double complex
```
Actually there are Cython types for np.bool_: cnp.npy_bool and bint but both they can't be used for NumPy arrays currently. For scalars cnp.npy_bool will just be an unsigned integer while bint will be a boolean. Not sure what's going on there... 1 Taken From the NumPy documentation "Data type objects" Built-in Python types Several python types are equivalent to a corresponding array scalar when used to generate a dtype object:
```
int           np.int_
bool          np.bool_
float         np.float_
complex       np.cfloat
bytes         np.bytes_
str           np.bytes_ (Python2) or np.unicode_ (Python3)
unicode       np.unicode_
buffer        np.void
(all others)  np.object_
```


Question:
Convert Pandas dataframe to Sparse Numpy Matrix directly I am creating a matrix from a Pandas dataframe as follows:
```
dense_matrix = np.array(df.as_matrix(columns = None), dtype=bool).astype(np.int)
```
And then into a sparse matrix with:
```
sparse_matrix = scipy.sparse.csr_matrix(dense_matrix)
```
Is there any way to go from a df straight to a sparse matrix? Thanks in advance.

Response:
df.values is a numpy array, and accessing values that way is always faster than np.array.
```
scipy.sparse.csr_matrix(df.values)
```
You might need to take the transpose first, like df.values.T. In DataFrames, the columns are axis 0.


Question:
python tilde unary operator as negation numpy bool array Should be a simple question, but I'm unable to find an answer anywhere. The ~ operator in python is a documented as a bitwise inversion operator. Fine. I have noticed seemingly schizophrenic behavior though, to wit:
```
~True -> -2
~1 -> -2
~False -> -1
~0 -> -1
~numpy.array([True,False],dtype=int) -> array([-2,-1])
~numpy.array([True,False],dtype=bool) -> array([False,True])
```
In the first 4 examples, I can see that python is implementing (as documented) ~x = -(x+1), with the input treated as an int even if it's boolean. Hence, for a scalar boolean, ~ is not treated as a logical negation. Not that the behavior is identical on a numpy array defined with boolean values by with an int type. Why does ~ then work as a logical negation operator on a boolean array (Also notice: ~numpy.isfinite(numpy.inf) -> True?)? It is extremely annoying that I must use not() on a scalar, but not() won't work to negate an array. Then for an array, I must use ~, but ~ won't work to negate a scalar...

Response: not is implemented through the __nonzero__ special method, which is required to return either True or False, so it can't give the required result. Instead the ~ operator is used, which is implemented through the __not__ special method. For the same reason, & and | are used in place of and and or. PEP 335 aimed to allow overloading of boolean operators but was rejected because of excessive overhead (it would e.g. complicate if statements). PEP 225 suggests a general syntax for "elementwise" operators, which would provide a more general solution, but has been deferred. It appears that the current situation, while awkward, is not painful enough to force change. np.isfinite when called on a scalar returns a value of type np.bool_, not bool. np.bool_ is also the type you get when extracting a scalar value from an array of bool dtype. If you use np.True_ and np.False_ in place of True and False you will get consistent behaviour under ~.


Question:
A fast way to find the largest N elements in an numpy array I know I can do it like the following:
```
import numpy as np
N=10
a=np.arange(1,100,1)
np.argsort()[-N:]
```
However, it is very slow since it did a full sort. I wonder whether numpy provide some methods the do it fast.

Response:
numpy 1.8 implements partition and argpartition that perform partial sort ( in O(n) time as opposed to full sort that is O(n) * log(n)).
```
import numpy as np

test = np.array([9,1,3,4,8,7,2,5,6,0])

temp = np.argpartition(-test, 4)
result_args = temp[:4]

temp = np.partition(-test, 4)
result = -temp[:4]
```
Result:
```
>>> result_args
array([0, 4, 8, 5]) # indices of highest vals
>>> result
array([9, 8, 6, 7]) # highest vals
```
Timing:
```

```python
a = np.arange(10000)
```

```python
np.random.shuffle(a)
```

```python
%timeit np.argsort(a)
1000 loops, best of 3: 1.02 ms per loop
```

```python
%timeit np.argpartition(a, 100)
10000 loops, best of 3: 139 us per loop
```

```python
%timeit np.argpartition(a, 1000)
10000 loops, best of 3: 141 us per loop
```
```


Question:
Figure to image as a numpy array I'm trying to get a numpy array image from a Matplotlib figure and I'm currently doing it by saving to a file, then reading the file back in, but I feel like there has to be a better way. Here's what I'm doing now:
```
from matplotlib.backends.backend_agg import FigureCanvasAgg as FigureCanvas
from matplotlib.figure import Figure

fig = Figure()
canvas = FigureCanvas(fig)
ax = fig.gca()

ax.text(0.0,0.0,"Test", fontsize=45)
ax.axis('off')

canvas.print_figure("output.png")
image = plt.imread("output.png")
```
I tried this:
```
image = np.fromstring( canvas.tostring_rgb(), dtype='uint8' )
```
from an example I found but it gives me an error saying that 'FigureCanvasAgg' object has no attribute 'renderer'.

Response:
In order to get the figure contents as RGB pixel values, the matplotlib.backend_bases.Renderer needs to first draw the contents of the canvas. You can do this by manually calling canvas.draw():
```
from matplotlib.figure import Figure

fig = Figure()
canvas = fig.canvas
ax = fig.gca()

ax.text(0.0,0.0,"Test", fontsize=45)
ax.axis('off')

canvas.draw()  # Draw the canvas, cache the renderer

image_flat = np.frombuffer(canvas.tostring_rgb(), dtype='uint8')  # (H * W * 3,)
# NOTE: reversed converts (W, H) from get_width_height to (H, W)
image = image_flat.reshape(*reversed(canvas.get_width_height()), 3)  # (H, W, 3)
```
See here for more info on the Matplotlib API.


Question:
How to get the values from a NumPy array using multiple indices I have a NumPy array that looks like this:
```
arr = np.array([100.10, 200.42, 4.14, 89.00, 34.55, 1.12])
```
How can I get multiple values from this array by index? For example, how can I get the values at the index positions 1, 4, and 5? I was trying something like this, which is incorrect:
```
arr[1, 4, 5]
```

Response:
Try like this:
```

```python
arr = np.array([100.10, 200.42, 4.14, 89.00, 34.55, 1.12])
```

```python
arr[[1,4,5]]
#Output
#array([ 200.42,   34.55,    1.12])
#```
#And for multidimensional arrays:
#```
```

```python
arr = np.arange(9).reshape(3,3)
```

```python
arr
#Output
#array([[0, 1, 2],
#       [3, 4, 5],
#       [6, 7, 8]])
```

```python
arr[[0, 1, 1], [1, 0, 2]]
#Output
#array([1, 3, 5])
#```
```


Question:
Filtering (reducing) a NumPy Array Suppose I have a NumPy array arr that I want to element-wise filter (reduce) depending on the truth value of a (broadcastable) function, e.g. I want to get only values below a certain threshold value k:
```py
def cond(x):
    return x < k
```
There are a couple of methods, e.g.: Using a generator: np.fromiter((x for x in arr if cond(x)), dtype=arr.dtype) (which is a memory efficient version of using a list comprehension: np.array([x for x in arr if cond(x)]) because np.fromiter() will produce a NumPy array directly, without the need to allocate an intermediate Python list) Using boolean masking: arr[cond(arr)] Using integer indexing: arr[np.nonzero(cond(arr))] (or equivalently using np.where() as it defaults to np.nonzero() with only one condition) Using explicit looping with: single pass and final copying/resizing two passes: one to determine the size of the result and one to actually perform the computation (The last two approaches to be accelerated with Cython or Numba) Which is the fastest? What about memory efficiency? (EDITED: To use directly np.nonzero() instead of np.where() as per @ShadowRanger comment)

Response:
Summary Using a loop-based approach with single pass and copying, accelerated with Numba, offers the best overall trade-off in terms of speed, memory efficiency and flexibility. If the execution of the condition function is sufficiently fast, two-passes (filter2_nb()) may be faster, while they are more memory efficient regardless. Also, for sufficiently large inputs, resizing instead of copying (filter_resize_xnb()) leads to faster execution. If the data type (and the condition function) is known ahead of time and can be compiled, the Cython acceleration seems to be faster. It is likely that a similar hard-coding of the condition would lead to comparable speed-up with Numba accerelation as well. When it comes to NumPy-only based approaches, boolean masking or integer indexing are of comparable speed, and which one comes out faster depends largely on the filtering factor, i.e. the portion of values that passes through the filtering condition. The np.fromiter() approach is much slower (it would be off-charts in the plot), but does not produce large temporary objects. Note that the following tests are meant to give some insights into the different approaches and should be taken with a grain of salt. The most relevant assumptions are that the condition is broadcastable and that it would eventually compute very fast. Definitions Using a generator:
```py
def filter_fromiter(arr, cond):
    return np.fromiter((x for x in arr if cond(x)), dtype=arr.dtype)
```
Using boolean masking:
```py
def filter_mask(arr, cond):
    return arr[cond(arr)]
```
Using integer indexing:
```py
def filter_idx(arr, cond):
    return arr[np.nonzero(cond(arr))]
```
4a. Using explicit looping, with single pass and final copying/resizing Cython-accelerated with copying (pre-compiled condition)
```
%%cython -c-O3 -c-march=native -a
#cython: language_level=3, boundscheck=False, wraparound=False, initializedcheck=False, cdivision=True, infer_types=True

import numpy as np

cdef long NUM = 1048576
cdef long MAX_VAL = 1048576
cdef long K = 1048576 // 2

cdef int cond_cy(long x, long k=K):
    return x < k

cdef size_t _filter_cy(long[:] arr, long[:] result, size_t size):
    cdef size_t j = 0
    for i in range(size):
        if cond_cy(arr[i]):
            result[j] = arr[i]
            j += 1
    return j

def filter_cy(arr):
    result = np.empty_like(arr)
    new_size = _filter_cy(arr, result, arr.size)
    return result[:new_size].copy()
```
Cython-accelerated with resizing (pre-compiled condition)
```
def filter_resize_cy(arr):
    result = np.empty_like(arr)
    new_size = _filter_cy(arr, result, arr.size)
    result.resize(new_size)
    return result
```
Numba-accelerated with copying
```py
import numba as nb

@nb.njit
def cond_nb(x, k=K):
    return x < k

@nb.njit
def filter_nb(arr, cond_nb):
    result = np.empty_like(arr)
    j = 0
    for i in range(arr.size):
        if cond_nb(arr[i]):
            result[j] = arr[i]
            j += 1
    return result[:j].copy()
```
Numba-accelerated with resizing
```py
@nb.njit
def _filter_out_nb(arr, out, cond_nb):
    j = 0
    for i in range(arr.size):
        if cond_nb(arr[i]):
            out[j] = arr[i]
            j += 1
    return j

def filter_resize_xnb(arr, cond_nb):
    result = np.empty_like(arr)
    j = _filter_out_nb(arr, result, cond_nb)
    result.resize(j, refcheck=False)  # unsupported in NoPython mode
    return result
```
Numba-accelerated with a generator and np.fromiter()
```py
@nb.njit
def filter_gen_nb(arr, cond_nb):
    for i in range(arr.size):
        if cond_nb(arr[i]):
            yield arr[i]

def filter_gen_xnb(arr, cond_nb):
    return np.fromiter(filter_gen_nb(arr, cond_nb), dtype=arr.dtype)
```
4b. Using explicit looping with two passes: one to determine the size of the result and one to actually perform the computation Cython-accelerated (pre-compiled condition)
```
%%cython -c-O3 -c-march=native -a
#cython: language_level=3, boundscheck=False, wraparound=False, initializedcheck=False, cdivision=True, infer_types=True

cdef size_t _filtered_size_cy(long[:] arr, size_t size):
    cdef size_t j = 0
    for i in range(size):
        if cond_cy(arr[i]):
            j += 1
    return j

def filter2_cy(arr):
    cdef size_t new_size = _filtered_size_cy(arr, arr.size)
    result = np.empty(new_size, dtype=arr.dtype)
    new_size = _filter_cy(arr, result, arr.size)
    return result
```
Numba-accelerated
```py
@nb.njit
def filter2_nb(arr, cond_nb):
    j = 0
    for i in range(arr.size):
        if cond_nb(arr[i]):
            j += 1
    result = np.empty(j, dtype=arr.dtype)
    j = 0
    for i in range(arr.size):
        if cond_nb(arr[i]):
            result[j] = arr[i]
            j += 1
    return result
```
Timing Benchmarks (The generator-based filter_fromiter() method is much slower than the others -- by approx. 2 orders of magnitude. Similar (and perhaps slightly worse) performances can be expected from a list comprehension. This would be true for any explicit looping with non-accelerated code.) The timing would depend on both the input array size and the percent of filtered items. As a function of input size The first graph addresses the timings as a function of the input size (for ~50% filtering factor -- i.e. 50% of the elements appear in the result): In general, explicit looping with one form of acceleration leads to the fastest execution, with slight variations depending on input size. Within NumPy, the integer indexing approaches are basically on par with boolean masking. The benefits of using np.fromiter() (no pre-allocation) can be reaped by writing a Numba-accelerated generator, which would come out slower than the other approaches (within an order of magnitude), but much faster than pure-Python looping. As a function of filling The second graph addresses the timings as a function of items passing through the filter (for a fixed input size of ~1 million elements): The first observation is that all methods are slowest when approaching a ~50% filling and with less, or more filling they are faster, and fastest towards no filling (highest percent of filtered-out values, lowest percent of passing through values as indicated in the x-axis of the graph). Again, explicit looping with some mean of acceleration leads to the fastest execution. Within NumPy, the integer indexing and boolean masking approaches are again basically the same. (Full code available here) Memory Considerations The generator-based filter_fromiter() method requires only minimal temporary storage, independently of the size of the input. Memory-wise this is the most efficient method. This approach can be effectively speed up with a Numba-accelerated generator. Of similar memory efficiency are the Cython / Numba two-passes methods, because the size of the output is determined during the first pass. The caveat here is that computing the condition has to be fast for these methods to be fast. On the memory side, the single-pass solutions for both Cython and Numba require a temporary array of the size of the input. Hence, these are not very memory-efficient compared to two-passes or the generator-based one. Yet they are of similar asymptotic temporary memory footprint compared to masking, but the constant term is typically larger than masking. The boolean masking solution requires a temporary array of the size of the input but of type bool, which in NumPy is 1 byte, so this is ~8 times smaller than the default size of a NumPy array on a typical 64-bit system. The integer indexing solution has the same requirement as the boolean mask slicing in the first step (inside np.nonzero() call), which gets converted to a series of ints (typically int64 on a 64-bit system) in the second step (the output of np.nonzero()). This second step, therefore, has variable memory requirements, depending on the number of filtered elements. Remarks both boolean masking and integer indexing require some form of conditioning that is capable of producing a boolean mask (or, alternatively, a list of indices); in the above implementation, the condition is broadcastable the generator and the Numba-accelerated methods are also the most flexible when it comes to specifying a different filtering condition the Numba-accelerated methods require the condition to be Numba-compatible to access the Numba acceleration in NoPython mode the Cython solution requires specifying the data types for it to be fast, or extra effort for multiple types dispatching, and extra effort (not explored here) to get the same level of flexibility as the other methods for both Numba and Cython, the filtering condition can be hard-coded, leading to marginal but appreaciable speed differences the single-pass solutions require additional code to handle the unused (but otherwise initially allotted) memory. the NumPy methods do NOT return a view of the input, but a copy, as a result of advanced indexing:
```py
arr = np.arange(100)
k = 50
print('`arr[arr > k]` is a copy: ', arr[arr > k].base is None)
# `arr[arr > k]` is a copy:  True
print('`arr[np.where(arr > k)]` is a copy: ', arr[np.where(arr > k)].base is None)
# `arr[np.where(arr > k)]` is a copy:  True
print('`arr[:k]` is a copy: ', arr[:k].base is None)
# `arr[:k]` is a copy:  False
```
(EDITED: various improvements based on @ShadowRanger, @PaulPanzer, @max9111 and @DavidW comments.)


Question:
Melt the Upper Triangular Matrix of a Pandas Dataframe Given a square pandas DataFrame of the following form:
```
a  b  c
a  1 .5 .3
b .5  1 .4
c .3 .4  1
```
How can the upper triangle be melted to get a matrix of the following form
```
Row     Column    Value
  a        a       1
  a        b       .5
  a        c       .3
  b        b       1
  b        c       .4
  c        c       1

#Note the combination a,b is only listed once.  There is no b,a listing
```
I'm more interested in an idiomatic pandas solution, a custom indexer would be easy enough to write by hand... Thank you in advance for your consideration and response.

Response:
First I convert lower values of df to NaN by where and numpy.triu and then stack, reset_index and set column names:
```
import numpy as np

print df
     a    b    c
a  1.0  0.5  0.3
b  0.5  1.0  0.4
c  0.3  0.4  1.0

print np.triu(np.ones(df.shape)).astype(np.bool)
[[ True  True  True]
 [False  True  True]
 [False False  True]]

df = df.where(np.triu(np.ones(df.shape)).astype(np.bool))
print df
    a    b    c
a   1  0.5  0.3
b NaN  1.0  0.4
c NaN  NaN  1.0

df = df.stack().reset_index()
df.columns = ['Row','Column','Value']
print df

  Row Column  Value
0   a      a    1.0
1   a      b    0.5
2   a      c    0.3
3   b      b    1.0
4   b      c    0.4
5   c      c    1.0
```


Question:
Pandas : compute mean or std (standard deviation) over entire dataframe Here is my problem, I have a dataframe like this :
```
Depr_1  Depr_2  Depr_3
S3  0   5   9
S2  4   11  8
S1  6   11  12
S5  0   4   11
S4  4   8   8
```
and I just want to calculate the mean over the full dataframe, as the following doesn't work :
```
df.mean()
```
Then I came up with :
```
df.mean().mean()
```
But this trick won't work for computing the standard deviation. My final attempts were :
```
df.get_values().mean()
df.get_values().std()
```
Except that in the latter case, it uses mean() and std() function from numpy. It's not a problem for the mean, but it is for std, as the pandas function uses by default ddof=1, unlike the numpy one where ddof=0.

Response:
You could convert the dataframe to be a single column with stack (this changes the shape from 5x3 to 15x1) and then take the standard deviation:
```
df.stack().std()         # pandas default degrees of freedom is one
```
Alternatively, you can use values to convert from a pandas dataframe to a numpy array before taking the standard deviation:
```
df.values.std(ddof=1)    # numpy default degrees of freedom is zero
```
Unlike pandas, numpy will give the standard deviation of the entire array by default, so there is no need to reshape before taking the standard deviation. A couple of additional notes: The numpy approach here is a bit faster than the pandas one, which is generally true when you have the option to accomplish the same thing with either numpy or pandas. The speed difference will depend on the size of your data, but numpy was roughly 10x faster when I tested a few different sized dataframes on my laptop (numpy version 1.15.4 and pandas version 0.23.4). The numpy and pandas approaches here will not give exactly the same answers, but will be extremely close (identical at several digits of precision). The discrepancy is due to slight differences in implementation behind the scenes that affect how the floating point values get rounded.


Question:
Simple plot in python of a numpy array? I have an array that looks like this
```
cplr = array([ 0.01828922,  0.01972157,  0.02342053, ...,  0.25928021,
    0.26352547,  0.26883406])
```
If I say
```
import matplotlib.pyplot as plt
plt(cplr)
```
TypeError: 'module' object is not callable How do I plot the contents of a numpy array?

Response:
matplotlib.pyplot is a module; the function to plot is matplotlib.pyplot.plot. Thus, you should do
```
plt.plot(cplr)
plt.show()
```
A good place to learn more about this would be to read a matplotlib tutorial.


Question:
How do I get all the values from a NumPy array excluding a certain index? I have a NumPy array, and I want to retrieve all the elements except a certain index. For example, consider the following array
```
a = [0,1,2,3,4,5,5,6,7,8,9]
```
If I specify index 3, then the resultant should be
```
a = [0,1,2,4,5,5,6,7,8,9]
```

Response:
Like resizing, removing elements from an NumPy array is a slow operation (especially for large arrays since it requires allocating space and copying all the data from the original array to the new array). It should be avoided if possible. Often you can avoid it by working with a masked array instead. For example, consider the array a:
```
import numpy as np

a = np.array([0,1,2,3,4,5,5,6,7,8,9])
print(a)
print(a.sum())
# [0 1 2 3 4 5 5 6 7 8 9]
# 50
```
We can mask its value at index 3 and can perform a summation which ignores masked elements:
```
a = np.ma.array(a, mask=False)
a.mask[3] = True
print(a)
print(a.sum())
# [0 1 2 -- 4 5 5 6 7 8 9]
# 47
```
Masked arrays also support many operations besides sum. If you really need to, it is also possible to remove masked elements using the compressed method:
```
print(a.compressed())
# [0 1 2 4 5 5 6 7 8 9]
```
But as mentioned above, avoid it if possible.


Question:
Weird behaviour initializing a numpy array of string data I am having some seemingly trivial trouble with numpy when the array contains string data. I have the following code:
```
my_array = numpy.empty([1, 2], dtype = str)
my_array[0, 0] = "Cat"
my_array[0, 1] = "Apple"
```
Now, when I print it with print my_array[0, :], the response I get is ['C', 'A'], which is clearly not the expected output of Cat and Apple. Why is that, and how can I get the right output? Thanks!

Response:
Numpy requires string arrays to have a fixed maximum length. When you create an empty array with dtype=str, it sets this maximum length to 1 by default. You can see if you do my_array.dtype; it will show "|S1", meaning "one-character string". Subsequent assignments into the array are truncated to fit this structure. You can pass an explicit datatype with your maximum length by doing, e.g.:
```
my_array = numpy.empty([1, 2], dtype="S10")
```
The "S10" will create an array of length-10 strings. You have to decide how big will be big enough to hold all the data you want to hold.


Question:
Comparing previous row values in Pandas DataFrame
```
import pandas as pd
data={'col1':[1,3,3,1,2,3,2,2]}
df=pd.DataFrame(data,columns=['col1'])
print df

         col1
    0     1
    1     3
    2     3
    3     1
    4     2
    5     3
    6     2
    7     2
```
I have the following Pandas DataFrame and I want to create another column that compares the previous row of col1 to see if they are equal. What would be the best way to do this? It would be like the following DataFrame. Thanks
```
col1  match
0     1   False
1     3   False
2     3   True
3     1   False
4     2   False
5     3   False
6     2   False
7     2   True
```

Response:
You need eq with shift:
```
df['match'] = df.col1.eq(df.col1.shift())
print (df)
   col1  match
0     1  False
1     3  False
2     3   True
3     1  False
4     2  False
5     3  False
6     2  False
7     2   True
```
Or instead eq use ==, but it is a bit slowier in large DataFrame:
```
df['match'] = df.col1 == df.col1.shift()
print (df)
   col1  match
0     1  False
1     3  False
2     3   True
3     1  False
4     2  False
5     3  False
6     2  False
7     2   True
```
Timings:
```
import pandas as pd
data={'col1':[1,3,3,1,2,3,2,2]}
df=pd.DataFrame(data,columns=['col1'])
print (df)
#[80000 rows x 1 columns]
df = pd.concat([df]*10000).reset_index(drop=True)

df['match'] = df.col1 == df.col1.shift()
df['match1'] = df.col1.eq(df.col1.shift())
print (df)

```python
%timeit df.col1.eq(df.col1.shift())
The slowest run took 4.83 times longer than the fastest. This could mean that an intermediate result is being cached.
1000 loops, best of 3: 933 µs per loop
```

```python
%timeit df.col1 == df.col1.shift()
1000 loops, best of 3: 1 ms per loop
```
```


Question:
Convert structured array to regular NumPy array The answer will be very obvious I think, but I don't see it at the moment. How can I convert a record array back to a regular ndarray? Suppose I have following simple structured array:
```
x = np.array([(1.0, 4.0,), (2.0, -1.0)], dtype=[('f0', '<f8'), ('f1', '<f8')])
```
then I want to convert it to:
```
array([[ 1.,  4.],
       [ 2., -1.]])
```
I tried asarray and astype, but that didn't work. UPDATE (solved: float32 (f4) instead of float64 (f8)) OK, I tried the solution of Robert (x.view(np.float64).reshape(x.shape + (-1,)) ), and with a simple array it works perfectly. But with the array I wanted to convert it gives a strange outcome:
```
data = np.array([ (0.014793682843446732, 0.006681123282760382, 0.0, 0.0, 0.0, 0.0008984912419691682, 0.0, 0.013475529849529266, 0.0, 0.0),
       (0.014793682843446732, 0.006681123282760382, 0.0, 0.0, 0.0, 0.0008984912419691682, 0.0, 0.013475529849529266, 0.0, 0.0),
       (0.014776384457945824, 0.006656022742390633, 0.0, 0.0, 0.0, 0.0008901208057068288, 0.0, 0.013350814580917358, 0.0, 0.0),
       (0.011928378604352474, 0.002819152781739831, 0.0, 0.0, 0.0, 0.0012627150863409042, 0.0, 0.018906937912106514, 0.0, 0.0),
       (0.011928378604352474, 0.002819152781739831, 0.0, 0.0, 0.0, 0.001259754877537489, 0.0, 0.01886274479329586, 0.0, 0.0),
       (0.011969991959631443, 0.0028706740122288465, 0.0, 0.0, 0.0, 0.0007433745195157826, 0.0, 0.011164642870426178, 0.0, 0.0)],
      dtype=[('a_soil', '<f4'), ('b_soil', '<f4'), ('Ea_V', '<f4'), ('Kcc', '<f4'), ('Koc', '<f4'), ('Lmax', '<f4'), ('malfarquhar', '<f4'), ('MRN', '<f4'), ('TCc', '<f4'), ('Vcmax_3', '<f4')])
```
and then:
```
data_array = data.view(np.float).reshape(data.shape + (-1,))
```
gives:
```

```python
data_array
#Output
#array([[  2.28080997e-20,   0.00000000e+00,   2.78023241e-27,
#          6.24133580e-18,   0.00000000e+00],
#       [  2.28080997e-20,   0.00000000e+00,   2.78023241e-27,
#          6.24133580e-18,   0.00000000e+00],
#       [  2.21114197e-20,   0.00000000e+00,   2.55866881e-27,
#          5.79825816e-18,   0.00000000e+00],
#       [  2.04776835e-23,   0.00000000e+00,   3.47457730e-26,
#          9.32782857e-17,   0.00000000e+00],
#       [  2.04776835e-23,   0.00000000e+00,   3.41189244e-26,
#          9.20222417e-17,   0.00000000e+00],
#       [  2.32706550e-23,   0.00000000e+00,   4.76375305e-28,
#          1.24257748e-18,   0.00000000e+00]])
#```
#which is an array with other numbers and another shape. What did I do wrong?
```

Response:
The simplest method is probably
```
x.view((float, len(x.dtype.names)))
```
(float must generally be replaced by the type of the elements in x: x.dtype[0]). This assumes that all the elements have the same type. This method gives you the regular numpy.ndarray version in a single step (as opposed to the two steps required by the view(…).reshape(…) method.


Question:
Set values on the diagonal of pandas.DataFrame I have a pandas dataframe I would like to se the diagonal to 0
```
import numpy
import pandas

df = pandas.DataFrame(numpy.random.rand(5,5))
df

Out[6]:
     0           1           2           3               4
0    0.536596    0.674319    0.032815    0.908086    0.215334
1    0.735022    0.954506    0.889162    0.711610    0.415118
2    0.119985    0.979056    0.901891    0.687829    0.947549
3    0.186921    0.899178    0.296294    0.521104    0.638924
4    0.354053    0.060022    0.275224    0.635054    0.075738
5 rows × 5 columns
```
now I want to set the diagonal to 0:
```
for i in range(len(df.index)):
    for j in range(len(df.columns)):
        if i==j:
            df.loc[i,j] = 0
df
Out[9]:
     0           1           2           3           4
0    0.000000    0.674319    0.032815    0.908086    0.215334
1    0.735022    0.000000    0.889162    0.711610    0.415118
2    0.119985    0.979056    0.000000    0.687829    0.947549
3    0.186921    0.899178    0.296294    0.000000    0.638924
4    0.354053    0.060022    0.275224    0.635054    0.000000
5 rows × 5 columns
```
but there must be a more pythonic way than that!?

Response:
```

```python
df.values[[np.arange(df.shape[0])]*2] = 0
```

```python
df
#Output
#0         1         2         3         4
#0  0.000000  0.931374  0.604412  0.863842  0.280339
#1  0.531528  0.000000  0.641094  0.204686  0.997020
#2  0.137725  0.037867  0.000000  0.983432  0.458053
#3  0.594542  0.943542  0.826738  0.000000  0.753240
#4  0.357736  0.689262  0.014773  0.446046  0.000000
#```
#Note that this will only work if df has the same number of rows as columns. Another way which will work for arbitrary shapes is to use np.fill_diagonal:
#```
```

```python
np.fill_diagonal(df.values, 0)
```
```


Question:
fitting data with numpy I have the following data:
```

```python
x
#Output
#array([ 3.08,  3.1 ,  3.12,  3.14,  3.16,  3.18,  3.2 ,  3.22,  3.24,
#    3.26,  3.28,  3.3 ,  3.32,  3.34,  3.36,  3.38,  3.4 ,  3.42,
#    3.44,  3.46,  3.48,  3.5 ,  3.52,  3.54,  3.56,  3.58,  3.6 ,
#    3.62,  3.64,  3.66,  3.68])
```

```python
y
#Output
#array([ 0.000857,  0.001182,  0.001619,  0.002113,  0.002702,  0.003351,
#    0.004062,  0.004754,  0.00546 ,  0.006183,  0.006816,  0.007362,
#    0.007844,  0.008207,  0.008474,  0.008541,  0.008539,  0.008445,
#    0.008251,  0.007974,  0.007608,  0.007193,  0.006752,  0.006269,
#    0.005799,  0.005302,  0.004822,  0.004339,  0.00391 ,  0.003481,
#    0.003095])
#```
#Now, I want to fit these data with, say, a 4 degree polynomial. So I do:
#```
```

```python
coefs = np.polynomial.polynomial.polyfit(x, y, 4)
```

```python
ffit = np.poly1d(coefs)
#Output
#```
#Now I create a new grid for x values to evaluate the fitting function ffit:
#```
```

```python
x_new = np.linspace(x[0], x[-1], num=len(x)*10)
#Output
#```
#When I do all the plotting (data set and fitting curve) with the command:
#```
```

```python
fig1 = plt.figure()
```

```python
ax1 = fig1.add_subplot(111)
```

```python
ax1.scatter(x, y, facecolors='None')
```

```python
ax1.plot(x_new, ffit(x_new))
```

```python
plt.show()
#Output
#```
#I get the following: fitting_data.png What I expect is the fitting function to fit correctly (at least near the maximum value of the data). What am I doing wrong?
```

Response:
Unfortunately, np.polynomial.polynomial.polyfit returns the coefficients in the opposite order of that for np.polyfit and np.polyval (or, as you used np.poly1d). To illustrate:
```

```python
np.polynomial.polynomial.polyfit(x, y, 4)
#Output
#array([  84.29340848, -100.53595376,   44.83281408,   -8.85931101,
#          0.65459882])
```

```python
np.polyfit(x, y, 4)
#Output
#array([   0.65459882,   -8.859311  ,   44.83281407, -100.53595375,
#         84.29340846])
#```
#In general: np.polynomial.polynomial.polyfit returns coefficients [A, B, C] to A + Bx + Cx^2 + ..., while np.polyfit returns: ... + Ax^2 + Bx + C. So if you want to use this combination of functions, you must reverse the order of coefficients, as in:
#```
#ffit = np.polyval(coefs[::-1], x_new)
#```
#However, the documentation states clearly to avoid np.polyfit, np.polyval, and np.poly1d, and instead to use only the new(er) package. You're safest to use only the polynomial package:
#```
#import numpy.polynomial.polynomial as poly
#
#coefs = poly.polyfit(x, y, 4)
#ffit = poly.polyval(x_new, coefs)
#plt.plot(x_new, ffit)
#```
#Or, to create the polynomial function:
#```
#ffit = poly.Polynomial(coefs)    # instead of np.poly1d
#plt.plot(x_new, ffit(x_new))
#```
```


Question: Multiply several matrices in numpy Suppose you have n square matrices A1,...,An. Is there anyway to multiply these matrices in a neat way? As far as I know dot in numpy accepts only two arguments. One obvious way is to define a function to call itself and get the result. Is there any better way to get it done?

Response:
This might be a relatively recent feature, but I like:
```
A.dot(B).dot(C)
```
or if you had a long chain you could do:
```
reduce(numpy.dot, [A1, A2, ..., An])
```
Update: There is more info about reduce here. Here is an example that might help.
```

```python
A = [np.random.random((5, 5)) for i in xrange(4)]
```

```python
product1 = A[0].dot(A[1]).dot(A[2]).dot(A[3])
```

```python
product2 = reduce(numpy.dot, A)
```

```python
numpy.all(product1 == product2)
#Output
#True
#```
#Update 2016: As of python 3.5, there is a new matrix_multiply symbol, @:
#```
#R = A @ B @ C
#```
```


Question:
How to find first non-zero value in every column of a numpy array? Suppose I have a numpy array of the form:
```
arr=numpy.array([[1,1,0],[1,1,0],[0,0,1],[0,0,0]])
```
I want to find the indices of the first index (for every column) where the value is non-zero. So in this instance, I would like the following to be returned:
```
[0,0,2]
```
How do I go about this?

Response:
Indices of first occurrences Use np.argmax along that axis (zeroth axis for columns here) on the mask of non-zeros to get the indices of first matches (True values) -
```
(arr!=0).argmax(axis=0)
```
Extending to cover generic axis specifier and for cases where no non-zeros are found along that axis for an element, we would have an implementation like so -
```
def first_nonzero(arr, axis, invalid_val=-1):
    mask = arr!=0
    return np.where(mask.any(axis=axis), mask.argmax(axis=axis), invalid_val)
```
Note that since argmax() on all False values returns 0, so if the invalid_val needed is 0, we would have the final output directly with mask.argmax(axis=axis). Sample runs -
```

```python
arr    # Different from given sample for variety
#Output
#array([[1, 0, 0],
#       [1, 1, 0],
#       [0, 1, 0],
#       [0, 0, 0]])
```

```python
first_nonzero(arr, axis=0, invalid_val=-1)
#Output
#array([ 0,  1, -1])
```

```python
first_nonzero(arr, axis=1, invalid_val=-1)
#Output
#array([ 0,  0,  1, -1])
#```
#Extending to cover all comparison operations To find the first zeros, simply use arr==0 as mask for use in the function. For first ones equal to a certain value val, use arr == val and so on for all cases of comparisons possible here. Indices of last occurrences To find the last ones matching a certain comparison criteria, we need to flip along that axis and use the same idea of using argmax and then compensate for the flipping by offsetting from the axis length, as shown below -
#```
#def last_nonzero(arr, axis, invalid_val=-1):
#    mask = arr!=0
#    val = arr.shape[axis] - np.flip(mask, axis=axis).argmax(axis=axis) - 1
#    return np.where(mask.any(axis=axis), val, invalid_val)
#```
#Sample runs -
#```
```

```python
arr
#Output
#array([[1, 0, 0],
#       [1, 1, 0],
#       [0, 1, 0],
#       [0, 0, 0]])
```

```python
last_nonzero(arr, axis=0, invalid_val=-1)
#Output
#array([ 1,  2, -1])
```

```python
last_nonzero(arr, axis=1, invalid_val=-1)
#Output
#array([ 0,  1,  1, -1])
#```
#Again, all cases of comparisons possible here are covered by using the corresponding comparator to get mask and then using within the listed function.
```


Question: PyTorch: Can't call numpy() on Variable that requires grad. Use var.detach().numpy() instead Calling tensor.numpy() gives the error: RuntimeError: Can't call numpy() on Variable that requires grad. Use var.detach().numpy() instead. tensor.cpu().detach().numpy() gives the same error.

Response:
Error reproduced
```
import torch

tensor1 = torch.tensor([1.0,2.0],requires_grad=True)

print(tensor1)
print(type(tensor1))

tensor1 = tensor1.numpy()

print(tensor1)
print(type(tensor1))
```
which leads to the exact same error for the line tensor1 = tensor1.numpy():
```
tensor([1., 2.], requires_grad=True)
<class 'torch.Tensor'>
Traceback (most recent call last):
  File "/home/badScript.py", line 8, in <module>
    tensor1 = tensor1.numpy()
RuntimeError: Can't call numpy() on Variable that requires grad. Use var.detach().numpy() instead.

Process finished with exit code 1
```
Generic solution this was suggested to you in your error message, just replace var with your variable name
```
import torch

tensor1 = torch.tensor([1.0,2.0],requires_grad=True)

print(tensor1)
print(type(tensor1))

tensor1 = tensor1.detach().numpy()

print(tensor1)
print(type(tensor1))
```
which returns as expected
```
tensor([1., 2.], requires_grad=True)
<class 'torch.Tensor'>
[1. 2.]
<class 'numpy.ndarray'>

Process finished with exit code 0
```
Some explanation You need to convert your tensor to another tensor that isn't requiring a gradient in addition to its actual value definition. This other tensor can be converted to a numpy array. Cf. this discuss.pytorch post. (I think, more precisely, that one needs to do that in order to get the actual tensor out of its pytorch Variable wrapper, cf. this other discuss.pytorch post).


Question:
what does numpy ndarray shape do? I have a simple question about the .shape function, which confused me a lot.
```
a = np.array([1, 2, 3])   # Create a rank 1 array
print(type(a))            # Prints "<class 'numpy.ndarray'>"
print(a.shape)            # Prints "(3,)"

b = np.array([[1,2,3],[4,5,6]])    # Create a rank 2 array
print(b.shape)                     # Prints "(2, 3)"
```
What did the .shape exactly do? count how many rows, how many columns, then the a.shape suppose to be, (1,3), one row three columns, right?

Response:
yourarray.shape or np.shape() or np.ma.shape() returns the shape of your ndarray as a tuple; And you can get the (number of) dimensions of your array using yourarray.ndim or np.ndim(). (i.e. it gives the n of the ndarray since all arrays in NumPy are just n-dimensional arrays (shortly called as ndarrays)) For a 1D array, the shape would be (n,) where n is the number of elements in your array. For a 2D array, the shape would be (n,m) where n is the number of rows and m is the number of columns in your array. Please note that in 1D case, the shape would simply be (n, ) instead of what you said as either (1, n) or (n, 1) for row and column vectors respectively. This is to follow the convention that: For 1D array, return a shape tuple with only 1 element (i.e. (n,)) For 2D array, return a shape tuple with only 2 elements (i.e. (n,m)) For 3D array, return a shape tuple with only 3 elements (i.e. (n,m,k)) For 4D array, return a shape tuple with only 4 elements (i.e. (n,m,k,j)) and so on. Also, please see the example below to see how np.shape() or np.ma.shape() behaves with 1D arrays and scalars:
```
# sample array

```python
u = np.arange(10)

# get its shape
```

```python
np.shape(u)    # u.shape
#Output
#(10,)
#
## get array dimension using `np.ndim`
```

```python
np.ndim(u)
#Output
#1
```

```python
np.shape(np.mean(u))
#Output
#()       # empty tuple (to indicate that a scalar is a 0D array).
#
## check using `numpy.ndim`
```

```python
np.ndim(np.mean(u))
#Output
#0
#```
#P.S.: So, the shape tuple is consistent with our understanding of dimensions of space, at least mathematically.
```


Question:
When to apply(pd.to_numeric) and when to astype(np.float64) in python? I have a pandas DataFrame object named xiv which has a column of int64 Volume measurements.
```

```python
xiv['Volume'].head(5)
#Output
#0    252000
#1    484000
#2     62000
#3    168000
#4    232000
#Name: Volume, dtype: int64
#```
#I have read other posts (like this and this) that suggest the following solutions. But when I use either approach, it doesn't appear to change the dtype of the underlying data:
#```
```

```python
xiv['Volume'] = pd.to_numeric(xiv['Volume'])
```

```python
xiv['Volume'].dtypes
#Output
#dtype('int64')
#```
#Or...
#```
```

```python
xiv['Volume'] = pd.to_numeric(xiv['Volume'])
#Output
####omitted for brevity###
```

```python
xiv['Volume'].dtypes
#Output
#dtype('int64')
```

```python
xiv['Volume'] = xiv['Volume'].apply(pd.to_numeric)
```

```python
xiv['Volume'].dtypes
#Output
#dtype('int64')
#```
#I've also tried making a separate pandas Series and using the methods listed above on that Series and reassigning to the x['Volume'] obect, which is a pandas.core.series.Series object. I have, however, found a solution to this problem using the numpy package's float64 type - this works but I don't know why it's different.
#```
```

```python
xiv['Volume'] = xiv['Volume'].astype(np.float64)
```

```python
xiv['Volume'].dtypes
#Output
#dtype('float64')
#```
#Can someone explain how to accomplish with the pandas library what the numpy library seems to do easily with its float64 class; that is, convert the column in the xiv DataFrame to a float64 in place.
```

Response:
If you already have numeric dtypes (int8|16|32|64,float64,boolean) you can convert it to another "numeric" dtype using Pandas .astype() method. Demo:
```

```python
df = pd.DataFrame(np.random.randint(10**5,10**7,(5,3)),columns=list('abc'), dtype=np.int64)
```

```python
df
#Output
#a        b        c
#0  9059440  9590567  2076918
#1  5861102  4566089  1947323
#2  6636568   162770  2487991
#3  6794572  5236903  5628779
#4   470121  4044395  4546794
```

```python
df.dtypes
#Output
#a    int64
#b    int64
#c    int64
#dtype: object
```

```python
df['a'] = df['a'].astype(float)
```

```python
df.dtypes
#Output
#a    float64
#b      int64
#c      int64
#dtype: object
#```
#It won't work for object (string) dtypes, that can't be converted to numbers:
#```
```

```python
df.loc[1, 'b'] = 'XXXXXX'
```

```python
df
#Output
#a        b        c
#0  9059440.0  9590567  2076918
#1  5861102.0   XXXXXX  1947323
#2  6636568.0   162770  2487991
#3  6794572.0  5236903  5628779
#4   470121.0  4044395  4546794
```

```python
df.dtypes
#Output
#a    float64
#b     object
#c      int64
#dtype: object
```

```python
df['b'].astype(float)
...
skipped
...
ValueError: could not convert string to float: 'XXXXXX'
```
So here we want to use pd.to_numeric() method:
```
```

```python
df['b'] = pd.to_numeric(df['b'], errors='coerce')
```

```python
df
#Output
#a          b        c
#0  9059440.0  9590567.0  2076918
#1  5861102.0        NaN  1947323
#2  6636568.0   162770.0  2487991
#3  6794572.0  5236903.0  5628779
#4   470121.0  4044395.0  4546794
```

```python
df.dtypes
#Output
#a    float64
#b    float64
#c      int64
#dtype: object
#```
```


Question:
Converting two lists into a matrix I'll try to be as clear as possible, and I'll start by explaining why I want to transform two arrays into a matrix. To plot the performance of a portfolio vs an market index I need a data structure like in this format:
```
[[portfolio_value1, index_value1]
 [portfolio_value2, index_value2]]
```
But I have the the data as two separate 1-D arrays:
```
portfolio = [portfolio_value1, portfolio_value2, ...]
index = [index_value1, index_value2, ...]
```
So how do I transform the second scenario into the first. I've tried np.insert to add the second array to a test matrix I had in a python shell, my problem was to transpose the first array into a single column matrix. Any help on how to achieve this without an imperative loop would be great.

Response:
The standard numpy function for what you want is np.column_stack:
```

```python
np.column_stack(([1, 2, 3], [4, 5, 6]))
#Output
#array([[1, 4],
#       [2, 5],
#       [3, 6]])
#```
#So with your portfolio and index arrays, doing
#```
#np.column_stack((portfolio, index))
#```
#would yield something like:
#```
#[[portfolio_value1, index_value1],
# [portfolio_value2, index_value2],
# [portfolio_value3, index_value3],
# ...]
#```
```


Question: How to access a column in a list of lists in python I have a 2D array in python modeled by a list of lists and I want to extract the column. I made a quick research and I found a way that uses numpy arrays. The problem is that I do not want to use numpy so I don't want to convert my list of lists into a numpy array and then use [:,1] syntax. I tried using it on a normal list of lists but it shows an error so it's not possible. I am asking for a similar thing for list of lists without having to go through each element (In numpy arrays, it's faster to access a column by using [:,1] syntax than iterating over the elements of the array). I found this link but again it suggests iterating over elements without a shortcut.

Response:
List comprehensions are your friend when working with lists of lists:
```

```python
alist
#Output
#[[0, 1, 2, 3, 4, 5],
# [6, 7, 8, 9, 10, 11],
# [12, 13, 14, 15, 16, 17],
# [18, 19, 20, 21, 22, 23]]
```

```python
[row[1] for row in alist]
#Output
#[1, 7, 13, 19]
#```
#There's also a handy 'idiom' for transposing a nested list, turning 'columns' into 'rows':
#```
```

```python
tlist = list(zip(*alist))
```

```python
tlist
#Output
#[(0, 6, 12, 18),
# (1, 7, 13, 19),
# (2, 8, 14, 20),
# (3, 9, 15, 21),
# (4, 10, 16, 22),
# (5, 11, 17, 23)]
```

```python
tlist[1]
#Output
#(1, 7, 13, 19)
#```
```


Question: Does Python have a function to reduce fractions? For example, when I calculate 98/42 I want to get 7/3, not 2.3333333, is there a function for that using Python or Numpy?

Response:
The fractions module can do that
```

```python
from fractions import Fraction
```

```python
Fraction(98, 42)
#Output
#Fraction(7, 3)
#```
#There's a recipe over here for a numpy gcd. Which you could then use to divide your fraction
#```
```

```python
def numpy_gcd(a, b):
    a, b = np.broadcast_arrays(a, b)
    a = a.copy()
    b = b.copy()
    pos = np.nonzero(b)[0]
    while len(pos) > 0:
        b2 = b[pos]
        a[pos], b[pos] = b2, a[pos] % b2
        pos = pos[b[pos]!=0]
    return a

```

```python
numpy_gcd(np.array([98]), np.array([42]))
#Output
#array([14])
```

```python
98/14, 42/14
#Output
#(7, 3)
#```
```


Question: numpy boolean array with 1 bit entries Is there a way in numpy to create an array of booleans that uses just 1 bit for each entry? The standard np.bool type is 1 byte, but this way I use 8 times the required memory. On Google I found that C++ has std::vector.

Response:
To do this you can use numpy's packbits and unpackbits:
```
import numpy as np
# original boolean array
A1 = np.array([
    [0, 1, 1, 0, 1],
    [0, 0, 1, 1, 1],
    [1, 1, 1, 1, 1],
], dtype=bool)

# packed data
A2 = np.packbits(A1, axis=None)

# checking the size
print(len(A1.tostring())) # 15 bytes
print(len(A2.tostring())) #  2 bytes (ceil(15/8))

# reconstructing from packed data. You need to resize and reshape
A3 = np.unpackbits(A2, count=A1.size).reshape(A1.shape).view(bool)

# and the arrays are equal
print(np.array_equal(A1, A3)) # True
```
Prior to numpy 1.17.0, the first function is straight-forward to use, but reconstruction required additional manipulations. Here is an example:
```
import numpy as np
# original boolean array
A1 = np.array([
    [0, 1, 1, 0, 1],
    [0, 0, 1, 1, 1],
    [1, 1, 1, 1, 1],
], dtype=np.bool)

# packed data
A2 = np.packbits(A1, axis=None)

# checking the size
print(len(A1.tostring())) # 15 bytes
print(len(A2.tostring())) #  2 bytes (ceil(15/8))

# reconstructing from packed data. You need to resize and reshape
A3 = np.unpackbits(A2, axis=None)[:A1.size].reshape(A1.shape).astype(np.bool)

# and the arrays are equal
print(np.array_equal(A1, A3)) # True
```


Question:
How to multiply individual elements of a list with a number?
```
S = [22, 33, 45.6, 21.6, 51.8]
P = 2.45
```
Here S is an array How will I multiply this and get the value?
```
SP = [53.9, 80.85, 111.72, 52.92, 126.91]
```

Response:
In NumPy it is quite simple
```
import numpy as np
P=2.45
S=[22, 33, 45.6, 21.6, 51.8]
SP = P*np.array(S)
```
I recommend taking a look at the NumPy tutorial for an explanation of the full capabilities of NumPy's arrays: https://scipy.github.io/old-wiki/pages/Tentative_NumPy_Tutorial


Question:
Find index where elements change value numpy Suppose I have
```

```python
v
#Output
#array([1, 1, 1, 1, 1, 2, 2, 2, 3, 4, 3, 4, 3, 4, 3, 4, 5, 5, 5])
#```
#Is there an efficient numpy way to find each index where the value changes? For instance, I would want some result like,
#```
```

```python
index_of_changed_values(v)
#Output
#[0, 5, 8, 9, 10, 11, 12, 13, 14, 15, 16]
#```
#If this is not possible with some numpy routine, what is a fast way to do it in python? It would also be useful to me to be referred to some good numpy tutorials since I am a numpy beginner.
```

Response:
You can get this functionality in numpy by comparing each element with it's neighbor;
```
v[:-1] != v[1:]

array([False, False, False, False,  True, False, False,  True,  True,
    True,  True,  True,  True,  True,  True,  True, False, False], dtype=bool)
```
to get the indices you use the "where" function
```
np.where(v[:-1] != v[1:])[0]

array([ 4,  7,  8,  9, 10, 11, 12, 13, 14, 15])
```
From here you can prepend the first element and add a one to get to the same indexing scheme you have in your question.


Question: When to use pandas series, numpy ndarrays or simply python dictionaries? I am new to learning Python, and some of its libraries (numpy, pandas). I have found a lot of documentation on how numpy ndarrays, pandas series and python dictionaries work. But owing to my inexperience with Python, I have had a really hard time determining when to use each one of them. And I haven't found any best-practices that will help me understand and decide when it is better to use each type of data structure. As a general matter, are there any best practices for deciding which, if any, of these three data structures a specific data set should be loaded into?

Response: The rule of thumb that I usually apply: use the simplest data structure that still satisfies your needs. If we rank the data structures from most simple to least simple, it usually ends up like this: Dictionaries / lists Numpy arrays Pandas series / dataframes So first consider dictionaries / lists. If these allow you to do all data operations that you need, then all is fine. If not, start considering numpy arrays. Some typical reasons for moving to numpy arrays are: Your data is 2-dimensional (or higher). Although nested dictionaries/lists can be used to represent multi-dimensional data, in most situations numpy arrays will be more efficient. You have to perform a bunch of numerical calculations. As already pointed out by zhqiat, numpy will give a significant speed-up in this case. Furthermore numpy arrays come bundled with a large amount of mathematical functions. Then there are also some typical reasons for going beyond numpy arrays and to the more-complex but also more-powerful pandas series/dataframes: You have to merge multiple data sets with each other, or do reshaping/reordering of your data. This diagram gives a nice overview of all the 'data wrangling' operations that pandas allows you to do. You have to import data from or export data to a specific file format like Excel, HDF5 or SQL. Pandas comes with convenient import/export functions for this.


Question:
Numpy, multiply array with scalar [duplicate] This question already has answers here: How to multiply individual elements of a list with a number? (4 answers) Closed 6 years ago. Is it possible to use ufuncs https://docs.scipy.org/doc/numpy/reference/ufuncs.html In order to map function to array (1D and / or 2D) and scalar If not what would be my way to achieve this? For example:
```
a_1 = np.array([1.0, 2.0, 3.0])
a_2 = np.array([[1., 2.], [3., 4.]])
b = 2.0
```
Expected result:
```
a_1 * b = array([2.0, 4.0, 6.0]);
a_2 * b = array([[2., 4.], [6., 8.]])
```
I`m using python 2.7 if it is relevant to an issue.

Response:
You can multiply numpy arrays by scalars and it just works.
```

```python
import numpy as np
```

```python
np.array([1, 2, 3]) * 2
#Output
#array([2, 4, 6])
```

```python
np.array([[1, 2, 3], [4, 5, 6]]) * 2
#Output
#array([[ 2,  4,  6],
#       [ 8, 10, 12]])
#```
#This is also a very fast and efficient operation. With your example:
#```
```

```python
a_1 = np.array([1.0, 2.0, 3.0])
```

```python
a_2 = np.array([[1., 2.], [3., 4.]])
```

```python
b = 2.0
```

```python
a_1 * b
#Output
#array([2., 4., 6.])
```

```python
a_2 * b
#Output
#array([[2., 4.],
#       [6., 8.]])
#```
```


Question:
Indexing a numpy array with a list of tuples Why can't I index an ndarray using a list of tuple indices like so?
```
idx = [(x1, y1), ... (xn, yn)]
X[idx]
```
Instead I have to do something unwieldy like
```
idx2 = numpy.array(idx)
X[idx2[:, 0], idx2[:, 1]] # or more generally:
X[tuple(numpy.vsplit(idx2.T, 1)[0])]
```
Is there a simpler, more pythonic way?

Response:
You can use a list of tuples, but the convention is different from what you want. numpy expects a list of row indices, followed by a list of column values. You, apparently, want to specify a list of (x,y) pairs. http://docs.scipy.org/doc/numpy/reference/arrays.indexing.html#integer-array-indexing The relevant section in the documentation is 'integer array indexing'. Here's an example, seeking 3 points in a 2d array. (2 points in 2d can be confusing):
```

```python
idx
#Output
#[(0, 1, 1), (2, 3, 0)]
```

```python
X[idx]
#Output
#array([2, 7, 4])
#```
#Using your style of xy pairs of indices:
#```
```

```python
idx1 = [(0,2),(1,3),(1,0)]
```

```python
[X[i] for i in idx1]
#Output
#[2, 7, 4]
```

```python
X[tuple(np.array(idx1).T)]
#Output
#array([2, 7, 4])
#```
#X[tuple(zip(*idx1))] is another way of doing the conversion. The tuple() is optional in Python2. zip(*...) is a Python idiom that reverses the nesting of a list of lists. You are on the right track with:
#```
```

```python
idx2=np.array(idx1)
```

```python
X[idx2[:,0], idx2[:,1]]
#Output
#array([2, 7, 4])
#```
#My tuple() is just a bit more compact (and not necessarily more 'pythonic'). Given the numpy convention, some sort of conversion is necessary. (Should we check what works with n-dimensions and m-points?)
```


Question: Weighted percentile using numpy Is there a way to use the numpy.percentile function to compute weighted percentile? Or is anyone aware of an alternative python function to compute weighted percentile?

Response:
Completely vectorized numpy solution Here is the code I use. It's not an optimal one (which I'm unable to write with numpy), but still much faster and more reliable than accepted solution
```py
def weighted_quantile(values, quantiles, sample_weight=None,
                      values_sorted=False, old_style=False):
    """ Very close to numpy.percentile, but supports weights.
    NOTE: quantiles should be in [0, 1]!
    :param values: numpy.array with data
    :param quantiles: array-like with many quantiles needed
    :param sample_weight: array-like of the same length as `array`
    :param values_sorted: bool, if True, then will avoid sorting of
        initial array
    :param old_style: if True, will correct output to be consistent
        with numpy.percentile.
    :return: numpy.array with computed quantiles.
    """
    values = np.array(values)
    quantiles = np.array(quantiles)
    if sample_weight is None:
        sample_weight = np.ones(len(values))
    sample_weight = np.array(sample_weight)
    assert np.all(quantiles >= 0) and np.all(quantiles <= 1), \
        'quantiles should be in [0, 1]'

    if not values_sorted:
        sorter = np.argsort(values)
        values = values[sorter]
        sample_weight = sample_weight[sorter]

    weighted_quantiles = np.cumsum(sample_weight) - 0.5 * sample_weight
    if old_style:
        # To be convenient with numpy.percentile
        weighted_quantiles -= weighted_quantiles[0]
        weighted_quantiles /= weighted_quantiles[-1]
    else:
        weighted_quantiles /= np.sum(sample_weight)
    return np.interp(quantiles, weighted_quantiles, values)
```
Examples: weighted_quantile([1, 2, 9, 3.2, 4], [0.0, 0.5, 1.]) array([ 1. , 3.2, 9. ]) weighted_quantile([1, 2, 9, 3.2, 4], [0.0, 0.5, 1.], sample_weight=[2, 1, 2, 4, 1]) array([ 1. , 3.2, 9. ])


Question:
In numpy, what does selection by [:,None] do? I'm taking the Udacity course on deep learning and I came across the following code:
```
def reformat(dataset, labels):
    dataset = dataset.reshape((-1, image_size * image_size)).astype(np.float32)
    # Map 0 to [1.0, 0.0, 0.0 ...], 1 to [0.0, 1.0, 0.0 ...]
    labels = (np.arange(num_labels) == labels[:,None]).astype(np.float32)
    return dataset, labels
```
What does labels[:,None] actually do here?

Response:
http://docs.scipy.org/doc/numpy/reference/arrays.indexing.html numpy.newaxis The newaxis object can be used in all slicing operations to create an axis of length one. :const: newaxis is an alias for ‘None’, and ‘None’ can be used in place of this with the same result. http://docs.scipy.org/doc/numpy-1.10.1/reference/generated/numpy.expand_dims.html Demonstrating with part of your code
```

```python
labels=np.array([1,3,5])
```

```python
labels[:,None]
#Output
#array([[1],
#       [3],
#       [5]])
```

```python
np.arange(8)==labels[:,None]
#Output
#array([[False,  True, False, False, False, False, False, False],
#       [False, False, False,  True, False, False, False, False],
#       [False, False, False, False, False,  True, False, False]], dtype=bool)
```

```python
(np.arange(8)==labels[:,None]).astype(int)
#Output
#array([[0, 1, 0, 0, 0, 0, 0, 0],
#       [0, 0, 0, 1, 0, 0, 0, 0],
#       [0, 0, 0, 0, 0, 1, 0, 0]])
#```
```


Question:
Faster way of polygon intersection with shapely I have a large number of polygons (~100000) and try to find a smart way of calculating their intersecting area with a regular grid cells. Currently, I am creating the polygons and the grid cells using shapely (based on their corner coordinates). Then, using a simple for-loop I go through each polygon and compare it to nearby grid cells. Just a small example to illustrate the polygons/grid cells.
```
from shapely.geometry import box, Polygon
# Example polygon
xy = [[130.21001, 27.200001], [129.52, 27.34], [129.45, 27.1], [130.13, 26.950001]]
polygon_shape = Polygon(xy)
# Example grid cell
gridcell_shape = box(129.5, -27.0, 129.75, 27.25)
# The intersection
polygon_shape.intersection(gridcell_shape).area
```
(BTW: the grid cells have the dimensions 0.25x0.25 and the polygons 1x1 at max) Actually this is quite fast for an individual polygon/grid cell combo with around 0.003 seconds. However, running this code on a huge amount of polygons (each one could intersect dozens of grid cells) takes around 15+ minutes (up to 30+ min depending on the number of intersecting grid cells) on my machine which is not acceptable. Unfortunately, I have no idea how it is possible to write a code for polygon intersection to get the area of overlap. Do you have any tips? Is there an alternative to shapely?

Response:
Consider using Rtree to help identify which grid cells that a polygon may intersect. This way, you can remove the for loop used with the array of lat/lons, which is probably the slow part. Structure your code something like this:
```
from shapely.ops import cascaded_union
from rtree import index
idx = index.Index()

# Populate R-tree index with bounds of grid cells
for pos, cell in enumerate(grid_cells):
    # assuming cell is a shapely object
    idx.insert(pos, cell.bounds)

# Loop through each Shapely polygon
for poly in polygons:
    # Merge cells that have overlapping bounding boxes
    merged_cells = cascaded_union([grid_cells[pos] for pos in idx.intersection(poly.bounds)])
    # Now do actual intersection
    print(poly.intersection(merged_cells).area)
```


Question:
Numpy argmax - random tie breaking In numpy.argmax function, tie breaking between multiple max elements is so that the first element is returned. Is there a functionality for randomizing tie breaking so that all maximum numbers have equal chance of being selected? Below is an example directly from numpy.argmax documentation.
```

```python
b = np.arange(6)
```

```python
b[1] = 5
```

```python
b
#Output
#array([0, 5, 2, 3, 4, 5])
```

```python
np.argmax(b) # Only the first occurrence is returned.
#Output
#1
#```
#I am looking for ways so that 1st and 5th elements in the list are returned with equal probability. Thank you!
```

Response:
Use np.random.choice -
```
np.random.choice(np.flatnonzero(b == b.max()))
```
Let's verify for an array with three max candidates -
```

```python
b
#Output
#array([0, 5, 2, 5, 4, 5])
```

```python
c=[np.random.choice(np.flatnonzero(b == b.max())) for i in range(100000)]
```

```python
np.bincount(c)
#Output
#array([    0, 33180,     0, 33611,     0, 33209])
#```
```


Question:
How to fix IndexError: invalid index to scalar variable This code generates error:
```
IndexError: invalid index to scalar variable.
```
at the line: results.append(RMSPE(np.expm1(y_train[testcv]), [y[1] for y in y_test])) How to fix it?
```
import pandas as pd
import numpy as np
from sklearn import ensemble
from sklearn import cross_validation

def ToWeight(y):
    w = np.zeros(y.shape, dtype=float)
    ind = y != 0
    w[ind] = 1./(y[ind]**2)
    return w

def RMSPE(y, yhat):
    w = ToWeight(y)
    rmspe = np.sqrt(np.mean( w * (y - yhat)**2 ))
    return rmspe

forest = ensemble.RandomForestRegressor(n_estimators=10, min_samples_split=2, n_jobs=-1)

print ("Cross validations")
cv = cross_validation.KFold(len(train), n_folds=5)

results = []
for traincv, testcv in cv:
    y_test = np.expm1(forest.fit(X_train[traincv], y_train[traincv]).predict(X_train[testcv]))
    results.append(RMSPE(np.expm1(y_train[testcv]), [y[1] for y in y_test]))
```
testcv is:
```
[False False False ...,  True  True  True]
```

Response:
You are trying to index into a scalar (non-iterable) value:
```
[y[1] for y in y_test]
#  ^ this is the problem
```
When you call [y for y in test] you are iterating over the values already, so you get a single value in y. Your code is the same as trying to do the following:
```
y_test = [1, 2, 3]
y = y_test[0] # y = 1
print(y[0]) # this line will fail
```
I'm not sure what you're trying to get into your results array, but you need to get rid of [y[1] for y in y_test]. If you want to append each y in y_test to results, you'll need to expand your list comprehension out further to something like this:
```
[results.append(..., y) for y in y_test]
```
Or just use a for loop:
```
for y in y_test:
    results.append(..., y)
```


Question: How to pad with zeros a tensor along some axis (Python) I would like to pad a numpy tensor with 0 along the chosen axis. For instance, I have tensor r with shape (4,3,2) but I am only interested in padding only the last two axis (that is, pad only the matrix). Is it possible to do it with the one-line python code?

Response:
You can use np.pad():
```
a = np.ones((4, 3, 2))

# npad is a tuple of (n_before, n_after) for each dimension
npad = ((0, 0), (1, 2), (2, 1))
b = np.pad(a, pad_width=npad, mode='constant', constant_values=0)

print(b.shape)
# (4, 6, 5)

print(b)
# [[[ 0.  0.  0.  0.  0.]
#   [ 0.  0.  1.  1.  0.]
#   [ 0.  0.  1.  1.  0.]
#   [ 0.  0.  1.  1.  0.]
#   [ 0.  0.  0.  0.  0.]
#   [ 0.  0.  0.  0.  0.]]

#  [[ 0.  0.  0.  0.  0.]
#   [ 0.  0.  1.  1.  0.]
#   [ 0.  0.  1.  1.  0.]
#   [ 0.  0.  1.  1.  0.]
#   [ 0.  0.  0.  0.  0.]
#   [ 0.  0.  0.  0.  0.]]

#  [[ 0.  0.  0.  0.  0.]
#   [ 0.  0.  1.  1.  0.]
#   [ 0.  0.  1.  1.  0.]
#   [ 0.  0.  1.  1.  0.]
#   [ 0.  0.  0.  0.  0.]
#   [ 0.  0.  0.  0.  0.]]

#  [[ 0.  0.  0.  0.  0.]
#   [ 0.  0.  1.  1.  0.]
#   [ 0.  0.  1.  1.  0.]
#   [ 0.  0.  1.  1.  0.]
#   [ 0.  0.  0.  0.  0.]
#   [ 0.  0.  0.  0.  0.]]]
```


Question: How to use numpy.void type I loaded a MATLAB .mat file via scipy.io.loadmat and it gave me a list of numpy.void objects. What are they, how can they be used and where can I get some reference documentation on them?

Response:
According to the numpy documentation: http://docs.scipy.org/doc/numpy/reference/arrays.dtypes.html, numpy.void types are defined as flexible data types. Basically, these are data types where there is no pre-defined type associated to the variable(s) you're looking at. If you look at numpy, you have data types such as float, uint8, bool, string, etc. void is to accommodate for more generic and flexible types and are for those data types that don't necessary fall into any one of these pre-defined data types. This situation is mostly encountered when you're loading in a struct where each element has multiple data types associated with multiple fields. Each structure element could have a combination of different data types, and the amalgamation of all of these data types to represent an instance of this structure element thus leads us to numpy.void. With the documentation, you can certainly do the same operations like you would with any other data type. Take a look at the generic data type methods here: http://docs.scipy.org/doc/numpy/reference/generated/numpy.generic.html#numpy.generic . In fact, all numpy data types are derived from this generic class, including numpy.void. In the first link I provided at the beginning of this post, it shows a good example of how to create a custom record type, where a record is a combination of a tuple of numbers and a string. When creating a list of these records, each type in the list is of type numpy.void and it demonstrates that a record is of this data type. However, bear in mind that this record list has a data type that is of this record, but each element of this list will be of type numpy.void. However, as a matter of self-containment, let's re-create the example here: Let's create a custom record type where it has two fields associated for each variable you create: A 16-bit string with a field named name A 2-element tuple of floating point numbers that are 64-bits each, with a field named grades As such, you'd do something like:
```
import numpy as np
dt = np.dtype([('name', np.str_, 16), ('grades', np.float64, (2,))])
```
As such, let's create an example list of two elements and instantiate their fields:
```
x = np.array([('Sarah', (8.0, 7.0)), ('John', (6.0, 7.0))], dtype=dt)
```
Because we made this list into a numpy.array, we expect its data type to be so:
```
type(x)
```
We get:
```
<type 'numpy.ndarray'>
```
Remember, the list itself is a numpy.array, but not the individual elements. To access the second element of this list, which is the second record, we do:
```
x[1]
```
We get:
```
('John', [6.0, 7.0])
```
To check the type of the second record, we do:
```
type(x[1])
```
We get:
```
<type 'numpy.void'> # As expected
```
Some additional bonuses for you To access the name of the second record, we do:
```
x[1]['name']
```
We get:
```
'John'
```
To access the grades of the second record, we do:
```
x[1]['grades']
```
We get:
```
array([ 6.,  7.])
```
To check the type of the name inside the second record, we do:
```
type(x[1]['name'])
```
We get:
```
<type 'numpy.string_'>
```
To check the type of the grades inside the second record, we do:
```
type(x[1]['grades'])
```
We get:
```
<type 'numpy.ndarray'>
```
Take note that each element in this list is of type numpy.void. However, the individual fields for each element in our list is either a tuple of numbers, or a string. The collection of these elements together is of type numpy.void.


Question:
TypeError: Invalid dimensions for image data when plotting array with imshow() For the following code
```
# Numerical operation
SN_map_final = (new_SN_map - mean_SN) / sigma_SN

# Plot figure
fig12 = plt.figure(12)
fig_SN_final = plt.imshow(SN_map_final, interpolation='nearest')
plt.colorbar()

fig12 = plt.savefig(outname12)
```
with new_SN_map being a 1D array and mean_SN and sigma_SN being constants, I get the following error.
```
Traceback (most recent call last):
  File "c:\Users\Valentin\Desktop\Stage M2\density_map_simple.py", line 546, in <module>
    fig_SN_final = plt.imshow(SN_map_final, interpolation='nearest')
  File "c:\users\valentin\appdata\local\enthought\canopy\user\lib\site-packages\matplotlib\pyplot.py", line 3022, in imshow
    **kwargs)
  File "c:\users\valentin\appdata\local\enthought\canopy\user\lib\site-packages\matplotlib\__init__.py", line 1812, in inner
    return func(ax, *args, **kwargs)
  File "c:\users\valentin\appdata\local\enthought\canopy\user\lib\site-packages\matplotlib\axes\_axes.py", line 4947, in imshow
    im.set_data(X)
  File "c:\users\valentin\appdata\local\enthought\canopy\user\lib\site-packages\matplotlib\image.py", line 453, in set_data
    raise TypeError("Invalid dimensions for image data")
TypeError: Invalid dimensions for image data
```
What is the source of this error? I thought my numerical operations were allowed.

Response:
There is a (somewhat) related question on StackOverflow: Showing an image with pylab.imshow() Here the problem was that an array of shape (nx,ny,1) is still considered a 3D array, and must be squeezed or sliced into a 2D array. More generally, the reason for the Exception TypeError: Invalid dimensions for image data is shown here: matplotlib.pyplot.imshow() needs a 2D array, or a 3D array with the third dimension being of shape 3 or 4! You can easily check this with (these checks are done by imshow, this function is only meant to give a more specific message in case it's not a valid input):
```
from __future__ import print_function
import numpy as np

def valid_imshow_data(data):
    data = np.asarray(data)
    if data.ndim == 2:
        return True
    elif data.ndim == 3:
        if 3 <= data.shape[2] <= 4:
            return True
        else:
            print('The "data" has 3 dimensions but the last dimension '
                  'must have a length of 3 (RGB) or 4 (RGBA), not "{}".'
                  ''.format(data.shape[2]))
            return False
    else:
        print('To visualize an image the data must be 2 dimensional or '
              '3 dimensional, not "{}".'
              ''.format(data.ndim))
        return False
```
In your case:
```

```python
new_SN_map = np.array([1,2,3])
```

```python
valid_imshow_data(new_SN_map)
#Output
#To visualize an image the data must be 2 dimensional or 3 dimensional, not "1".
#False
#```
#The np.asarray is what is done internally by matplotlib.pyplot.imshow so it's generally best you do it too. If you have a numpy array it's obsolete but if not (for example a list) it's necessary. In your specific case you got a 1D array, so you need to add a dimension with np.expand_dims()
#```
#import matplotlib.pyplot as plt
#a = np.array([1,2,3,4,5])
#a = np.expand_dims(a, axis=0)  # or axis=1
#plt.imshow(a)
#plt.show()
#```
#or just use something that accepts 1D arrays like plot:
#```
#a = np.array([1,2,3,4,5])
#plt.plot(a)
#plt.show()
#```
```


Question:
Convert list or numpy array of single element to float in python I have a function which can accept either a list or a numpy array. In either case, the list/array has a single element (always). I just need to return a float. So, e.g., I could receive:
```
list_ = [4]
```
or the numpy array:
```
array_ = array([4])
```
And I should return
```
4.0
```
So, naturally (I would say), I employ float(...) on list_ and get:
```
TypeError: float() argument must be a string or a number
```
I do the same to array_ and this time it works by responding with "4.0". From this, I learn that Python's list cannot be converted to float this way. Based on the success with the numpy array conversion to float this lead me to the approach:
```
float(np.asarray(list_))
```
And this works when list_ is both a Python list and when it is a numpy array. Question But it seems like this approach has an overhead first converting the list to a numpy array and then to float. Basically: Is there a better way of doing this?

Response:
You may want to use the ndarray.item method, as in a.item(). This is also equivalent to (the now deprecated) np.asscalar(a). This has the benefit of working in situations with views and superfluous axes, while the above solutions will currently break. For example,
```

```python
a = np.asarray(1).view()
```

```python
a.item()  # correct
#Output
#1
```

```python
a[0]  # breaks
#Output
#Traceback (most recent call last):
#  File "<stdin>", line 1, in <module>
#IndexError: too many indices for array
```

```python
a = np.asarray([[2]])
```

```python
a.item()  # correct
#Output
#2
```

```python
a[0]  # bad result
#Output
#array([2])
#```
#This also has the benefit of throwing an exception if the array is not actually a scalar, while the a[0] approach will silently proceed (which may lead to bugs sneaking through undetected).
#```
```

```python
a = np.asarray([1, 2])
```

```python
a[0]  # silently proceeds
#Output
#1
```

```python
a.item()  # detects incorrect size
#Output
#Traceback (most recent call last):
#  File "<stdin>", line 1, in <module>
#ValueError: can only convert an array of size 1 to a Python scalar
#```
```


Question:
Why does numpy.power return 0 for small exponents while math.pow returns the correct answer?
```

```python
np.power(10,-100)
#Output
#0
```

```python
math.pow(10,-100)
#Output
#1e-100
#```
#I would expect both the commands to return 1e-100. This is not a precision issue either, since the issue persists even after increasing precision to 500. Is there some setting which I can change to get the correct answer?
```

Response:
Oh, it's much "worse" than that:
```

```python
numpy.power(10,-1)
#Output
#0
#```
#But this is a hint to what's going on: 10 is an integer, and numpy.power doesn't coerce the numbers to floats. But this works:
#```
```

```python
numpy.power(10.,-1)
#Output
#0.10000000000000001
```

```python
numpy.power(10.,-100)
#Output
#1e-100
#```
#Note, however, that the power operator, **, does convert to float:
#```
```

```python
10**-1
#Output
#0.1
#```
```


Question:
Passing list-likes to .loc or [] with any missing labels is no longer supported I want to create a modified dataframe with the specified columns. I tried the following but throws the error "Passing list-likes to .loc or [] with any missing labels is no longer supported"
```
# columns to keep
filtered_columns = ['text', 'agreeCount', 'disagreeCount', 'id', 'user.firstName', 'user.lastName', 'user.gender', 'user.id']
tips_filtered = tips_df.loc[:, filtered_columns]

# display tips
tips_filtered
```
Thank you

Response: It looks like Pandas has deprecated this method of indexing. According to their docs: This behavior is deprecated and will show a warning message pointing to this section. The recommended alternative is to use .reindex() Using the new recommended method, you can filter your columns using: tips_filtered = tips_df.reindex(columns = filtered_columns). NB: To reindex rows, you would use reindex(index = ...) (More information here).


Question: Scipy sparse... arrays? So, I'm doing some Kmeans classification using numpy arrays that are quite sparse-- lots and lots of zeroes. I figured that I'd use scipy's 'sparse' package to reduce the storage overhead, but I'm a little confused about how to create arrays, not matrices. I've gone through this tutorial on how to create sparse matrices: http://www.scipy.org/SciPy_Tutorial#head-c60163f2fd2bab79edd94be43682414f18b90df7 To mimic an array, I just create a 1xN matrix, but as you may guess, Asp.dot(Bsp) doesn't quite work because you can't multiply two 1xN matrices. I'd have to transpose each array to Nx1, and that's pretty lame, since I'd be doing it for every dot-product calculation. Next up, I tried to create an NxN matrix where column 1 == row 1 (such that you can multiply two matrices and just take the top-left corner as the dot product), but that turned out to be really inefficient. I'd love to use scipy's sparse package as a magic replacement for numpy's array(), but as yet, I'm not really sure what to do. Any advice?

Response:
Use a scipy.sparse format that is row or column based: csc_matrix and csr_matrix. These use efficient, C implementations under the hood (including multiplication), and transposition is a no-op (esp. if you call transpose(copy=False)), just like with numpy arrays. EDIT: some timings via ipython:
```
import numpy, scipy.sparse
n = 100000
x = (numpy.random.rand(n) * 2).astype(int).astype(float) # 50% sparse vector
x_csr = scipy.sparse.csr_matrix(x)
x_dok = scipy.sparse.dok_matrix(x.reshape(x_csr.shape))
```
Now x_csr and x_dok are 50% sparse:
```
print repr(x_csr)
<1x100000 sparse matrix of type '<type 'numpy.float64'>'
        with 49757 stored elements in Compressed Sparse Row format>
```
And the timings:
```
timeit numpy.dot(x, x)
10000 loops, best of 3: 123 us per loop

timeit x_dok * x_dok.T
1 loops, best of 3: 1.73 s per loop

timeit x_csr.multiply(x_csr).sum()
1000 loops, best of 3: 1.64 ms per loop

timeit x_csr * x_csr.T
100 loops, best of 3: 3.62 ms per loop
```
So it looks like I told a lie. Transposition is very cheap, but there is no efficient C implementation of csr * csc (in the latest scipy 0.9.0). A new csr object is constructed in each call :-( As a hack (though scipy is relatively stable these days), you can do the dot product directly on the sparse data:
```
timeit numpy.dot(x_csr.data, x_csr.data)
10000 loops, best of 3: 62.9 us per loop
```
Note this last approach does a numpy dense multiplication again. The sparsity is 50%, so it's actually faster than dot(x, x) by a factor of 2.


Question:
Convert numpy type to python I have a list of dicts in the following form that I generate from pandas. I want to convert it to a JSON format.
```
list_val = [{1.0: 685}, {2.0: 8}]
output = json.dumps(list_val)
```
However, json.dumps throws an error: TypeError: 685 is not JSON serializable I am guessing it's a type conversion issue from numpy to python(?). However, when I convert the values v of each dict in the array using np.int32(v) it still throws the error. Here's the full code:
```py
new = df[df[label] == label_new]
ks_dict = json.loads(content)
ks_list = ks_dict['variables']
freq_counts = []

for ks_var in ks_list:

    freq_var = dict()
    freq_var["name"] = ks_var["name"]
    ks_series = new[ks_var["name"]]
    temp_df = ks_series.value_counts().to_dict()
    freq_var["new"] = [{u: np.int32(v)} for (u, v) in temp_df.iteritems()]
    freq_counts.append(freq_var)

out = json.dumps(freq_counts)
```

Response:
It looks like you're correct:
```

```python
import numpy
```

```python
import json
```

```python
json.dumps(numpy.int32(685))
#Output
#Traceback (most recent call last):
#  File "<stdin>", line 1, in <module>
#  File "/usr/lib/python2.7/json/__init__.py", line 243, in dumps
#    return _default_encoder.encode(obj)
#  File "/usr/lib/python2.7/json/encoder.py", line 207, in encode
#    chunks = self.iterencode(o, _one_shot=True)
#  File "/usr/lib/python2.7/json/encoder.py", line 270, in iterencode
#    return _iterencode(o, 0)
#  File "/usr/lib/python2.7/json/encoder.py", line 184, in default
#    raise TypeError(repr(o) + " is not JSON serializable")
#TypeError: 685 is not JSON serializable
#```
#The unfortunate thing here is that numpy numbers' __repr__ doesn't give you any hint about what type they are. They're running around masquerading as ints when they aren't (gasp). Ultimately, it looks like json is telling you that an int isn't serializable, but really, it's telling you that this particular np.int32 (or whatever type you actually have) isn't serializable. (No real surprise there -- No np.int32 is serializable). This is also why the dict that you inevitably printed before passing it to json.dumps looks like it just has integers in it as well. The easiest workaround here is probably to write your own serializer1:
#```
#class MyEncoder(json.JSONEncoder):
#    def default(self, obj):
#        if isinstance(obj, numpy.integer):
#            return int(obj)
#        elif isinstance(obj, numpy.floating):
#            return float(obj)
#        elif isinstance(obj, numpy.ndarray):
#            return obj.tolist()
#        else:
#            return super(MyEncoder, self).default(obj)
#```
#You use it like this:
#```
#json.dumps(numpy.float32(1.2), cls=MyEncoder)
#json.dumps(numpy.arange(12), cls=MyEncoder)
#json.dumps({'a': numpy.int32(42)}, cls=MyEncoder)
#```
#etc. 1Or you could just write the default function and pass that as the defaut keyword argument to json.dumps. In this scenario, you'd replace the last line with raise TypeError, but ... meh. The class is more extensible :-)
```


Question:
How to plot vectors in python using matplotlib I am taking a course on linear algebra and I want to visualize the vectors in action, such as vector addition, normal vector, so on. For instance:
```
V = np.array([[1,1],[-2,2],[4,-7]])
```
In this case I want to plot 3 vectors V1 = (1,1), M2 = (-2,2), M3 = (4,-7). Then I should be able to add V1,V2 to plot a new vector V12(all together in one figure). when I use the following code, the plot is not as intended
```
import numpy as np
import matplotlib.pyplot as plt
M = np.array([[1,1],[-2,2],[4,-7]])

print("vector:1")
print(M[0,:])
# print("vector:2")
# print(M[1,:])
rows,cols = M.T.shape
print(cols)

for i,l in enumerate(range(0,cols)):
    print("Iteration: {}-{}".format(i,l))
    print("vector:{}".format(i))
    print(M[i,:])
    v1 = [0,0],[M[i,0],M[i,1]]
    # v1 = [M[i,0]],[M[i,1]]
    print(v1)
    plt.figure(i)
    plt.plot(v1)
    plt.show()
```

Response:
How about something like
```
import numpy as np
import matplotlib.pyplot as plt

V = np.array([[1,1], [-2,2], [4,-7]])
origin = np.array([[0, 0, 0],[0, 0, 0]]) # origin point

plt.quiver(*origin, V[:,0], V[:,1], color=['r','b','g'], scale=21)
plt.show()
```
Then to add up any two vectors and plot them to the same figure, do so before you call plt.show(). Something like:
```
plt.quiver(*origin, V[:,0], V[:,1], color=['r','b','g'], scale=21)
v12 = V[0] + V[1] # adding up the 1st (red) and 2nd (blue) vectors
plt.quiver(*origin, v12[0], v12[1], scale=21)
plt.show()
```
NOTE: in Python2 use origin[0], origin[1] instead of *origin


Question:
Randomly select from numpy array I have two related numpy arrays, X and y. I need to select n random rows from X and store this in an array, the corresponding y value and the appends to it the index of the points randomly selected. I have another array index which stores a list of index which I dont want to sample. How can I do this? Sample data:
```
index = [2,3]
X = np.array([[0.3,0.7],[0.5,0.5] ,[0.2,0.8], [0.1,0.9]])
y = np.array([[0], [1], [0], [1]])
```
If these X's were randomly selected (where n=2):
```
randomylSelected = np.array([[0.3,0.7],[0.5,0.5]])
```
the desired output would be:
```
index = [0,1,2,3]
randomlySelectedY = [0,1]
```
How can I do this?

Response:
You can create random indices with np.random.choice:
```
n = 2  # for 2 random indices
index = np.random.choice(X.shape[0], n, replace=False)
```
Then you just need to index your arrays with the result:
```
x_random = X[index]
y_random = Y[index]
```


Question: Python finite difference functions? I've been looking around in Numpy/Scipy for modules containing finite difference functions. However, the closest thing I've found is numpy.gradient(), which is good for 1st-order finite differences of 2nd order accuracy, but not so much if you're wanting higher-order derivatives or more accurate methods. I haven't even found very many specific modules for this sort of thing; most people seem to be doing a "roll-your-own" thing as they need them. So my question is if anyone knows of any modules (either part of Numpy/Scipy or a third-party module) that are specifically dedicated to higher-order (both in accuracy and derivative) finite difference methods. I've got my own code that I'm working on, but it's currently kind of slow, and I'm not going to attempt to optimize it if there's something already available. Note that I am talking about finite differences, not derivatives. I've seen both scipy.misc.derivative() and Numdifftools, which take the derivative of an analytical function, which I don't have.

Response:
One way to do this quickly is by convolution with the derivative of a gaussian kernel. The simple case is a convolution of your array with [-1, 1] which gives exactly the simple finite difference formula. Beyond that, (f*g)'= f'*g = f*g' where the * is convolution, so you end up with your derivative convolved with a plain gaussian, so of course this will smooth your data a bit, which can be minimized by choosing the smallest reasonable kernel.
```
import numpy as np
from scipy import ndimage
import matplotlib.pyplot as plt

#Data:
x = np.linspace(0,2*np.pi,100)
f = np.sin(x) + .02*(np.random.rand(100)-.5)

#Normalization:
dx = x[1] - x[0] # use np.diff(x) if x is not uniform
dxdx = dx**2

#First derivatives:
df = np.diff(f) / dx
cf = np.convolve(f, [1,-1]) / dx
gf = ndimage.gaussian_filter1d(f, sigma=1, order=1, mode='wrap') / dx

#Second derivatives:
ddf = np.diff(f, 2) / dxdx
ccf = np.convolve(f, [1, -2, 1]) / dxdx
ggf = ndimage.gaussian_filter1d(f, sigma=1, order=2, mode='wrap') / dxdx

#Plotting:
plt.figure()
plt.plot(x, f, 'k', lw=2, label='original')
plt.plot(x[:-1], df, 'r.', label='np.diff, 1')
plt.plot(x, cf[:-1], 'r--', label='np.convolve, [1,-1]')
plt.plot(x, gf, 'r', label='gaussian, 1')
plt.plot(x[:-2], ddf, 'g.', label='np.diff, 2')
plt.plot(x, ccf[:-2], 'g--', label='np.convolve, [1,-2,1]')
plt.plot(x, ggf, 'g', label='gaussian, 2')
```
Since you mentioned np.gradient I assumed you had at least 2d arrays, so the following applies to that: This is built into the scipy.ndimage package if you want to do it for ndarrays. Be cautious though, because of course this doesn't give you the full gradient but I believe the product of all directions. Someone with better expertise will hopefully speak up. Here's an example:
```
from scipy import ndimage

x = np.linspace(0,2*np.pi,100)
sine = np.sin(x)

im = sine * sine[...,None]
d1 = ndimage.gaussian_filter(im, sigma=5, order=1, mode='wrap')
d2 = ndimage.gaussian_filter(im, sigma=5, order=2, mode='wrap')

plt.figure()

plt.subplot(131)
plt.imshow(im)
plt.title('original')

plt.subplot(132)
plt.imshow(d1)
plt.title('first derivative')

plt.subplot(133)
plt.imshow(d2)
plt.title('second derivative')
```
Use of the gaussian_filter1d allows you to take a directional derivative along a certain axis:
```
imx = im * x
d2_0 = ndimage.gaussian_filter1d(imx, axis=0, sigma=5, order=2, mode='wrap')
d2_1 = ndimage.gaussian_filter1d(imx, axis=1, sigma=5, order=2, mode='wrap')

plt.figure()
plt.subplot(131)
plt.imshow(imx)
plt.title('original')
plt.subplot(132)
plt.imshow(d2_0)
plt.title('derivative along axis 0')
plt.subplot(133)
plt.imshow(d2_1)
plt.title('along axis 1')
```
The first set of results above are a little confusing to me (peaks in the original show up as peaks in the second derivative when the curvature should point down). Without looking further into how the 2d version works, I can only really recomend the 1d version. If you want the magnitude, simply do something like:
```
d2_mag = np.sqrt(d2_0**2 + d2_1**2)
```


Question: How do I transform a "SciPy sparse matrix" to a "NumPy matrix"? I am using a python function called "incidence_matrix(G)", which returns the incident matrix of graph. It is from Networkx package. The problem that I am facing is the return type of this function is "Scipy Sparse Matrix". I need to have the Incident matrix in the format of numpy matrix or array. I was wondering if there is any easy way of doing that or not? Or is there any built-in function that can do this transformation for me or not? Thanks

Response: The scipy.sparse.*_matrix has several useful methods, for example, if a is e.g. scipy.sparse.csr_matrix: a.toarray() - Return a dense ndarray representation of this matrix. (numpy.array, recommended) a.todense() - Return a dense matrix representation of this matrix. (numpy.matrix) Previously, these methods had shorthands (.A for .toarray(), and .M for .todense()), but these have been or will be deprecated as of Scipy v1.14.0.


Question:
numpy.savetxt without hash mark at beginning of header line When I try to save a matrix with header, a hash mark and a space (# ) appear on the first line: input:
```
np.savetxt(filename,data, fmt='%i %i %i %i %s',delimiter='\t',header="a\tb\tc\td\te")
```
output:
```
# a b   c   d   e
0   0   0   0   bla
0   0   0   0   bla
1   1   1   1   bla
1   1   1   1   bla
```
Any hint why? How could I remove it?

Response: it inserts the # because that line is a comment, and the default character for comments is the symbol #, as you can read in the documentation here. If you want to get rid of it, pass comments='' as option to savetxt.