Question: Understanding PyTorch einsum I'm familiar with how einsum works in NumPy. A similar functionality is also offered by PyTorch: torch.einsum(). What are the similarities and differences, either in terms of functionality or performance? The information available at PyTorch documentation is rather scanty and doesn't provide any insights regarding this.

Response:
Since the description of einsum is skimpy in torch documentation, I decided to write this post to document, compare and contrast how torch.einsum() behaves when compared to numpy.einsum(). Differences: NumPy allows both small case and capitalized letters [a-zA-Z] for the "subscript string" whereas PyTorch allows only the small case letters [a-z]. NumPy accepts nd-arrays, plain Python lists (or tuples), list of lists (or tuple of tuples, list of tuples, tuple of lists) or even PyTorch tensors as operands (i.e. inputs). This is because the operands have only to be array_like and not strictly NumPy nd-arrays. On the contrary, PyTorch expects the operands (i.e. inputs) strictly to be PyTorch tensors. It will throw a TypeError if you pass either plain Python lists/tuples (or its combinations) or NumPy nd-arrays. NumPy supports lot of keyword arguments (for e.g. optimize) in addition to nd-arrays while PyTorch doesn't offer such flexibility yet. Here are the implementations of some examples both in PyTorch and NumPy:
```
# input tensors to work with

```python
vec
#Output
#tensor([0, 1, 2, 3])
```

```python
aten
#Output
#tensor([[11, 12, 13, 14],
#        [21, 22, 23, 24],
#        [31, 32, 33, 34],
#        [41, 42, 43, 44]])
```

```python
bten
#Output
#tensor([[1, 1, 1, 1],
#        [2, 2, 2, 2],
#        [3, 3, 3, 3],
#        [4, 4, 4, 4]])
#```
#1) Matrix multiplication PyTorch: torch.matmul(aten, bten) ; aten.mm(bten) NumPy : np.einsum("ij, jk -> ik", arr1, arr2)
#```
```

```python
torch.einsum('ij, jk -> ik', aten, bten)
#Output
#tensor([[130, 130, 130, 130],
#        [230, 230, 230, 230],
#        [330, 330, 330, 330],
#        [430, 430, 430, 430]])
#```
#2) Extract elements along the main-diagonal PyTorch: torch.diag(aten) NumPy : np.einsum("ii -> i", arr)
#```
```

```python
torch.einsum('ii -> i', aten)
#Output
#tensor([11, 22, 33, 44])
#```
#3) Hadamard product (i.e. element-wise product of two tensors) PyTorch: aten * bten NumPy : np.einsum("ij, ij -> ij", arr1, arr2)
#```
```

```python
torch.einsum('ij, ij -> ij', aten, bten)
#Output
#tensor([[ 11,  12,  13,  14],
#        [ 42,  44,  46,  48],
#        [ 93,  96,  99, 102],
#        [164, 168, 172, 176]])
#```
#4) Element-wise squaring PyTorch: aten ** 2 NumPy : np.einsum("ij, ij -> ij", arr, arr)
#```
```

```python
torch.einsum('ij, ij -> ij', aten, aten)
#Output
#tensor([[ 121,  144,  169,  196],
#        [ 441,  484,  529,  576],
#        [ 961, 1024, 1089, 1156],
#        [1681, 1764, 1849, 1936]])
#```
#General: Element-wise nth power can be implemented by repeating the subscript string and tensor n times. For e.g., computing element-wise 4th power of a tensor can be done using:
#```
## NumPy: np.einsum('ij, ij, ij, ij -> ij', arr, arr, arr, arr)
```

```python
torch.einsum('ij, ij, ij, ij -> ij', aten, aten, aten, aten)
#Output
#tensor([[  14641,   20736,   28561,   38416],
#        [ 194481,  234256,  279841,  331776],
#        [ 923521, 1048576, 1185921, 1336336],
#        [2825761, 3111696, 3418801, 3748096]])
#```
#5) Trace (i.e. sum of main-diagonal elements) PyTorch: torch.trace(aten) NumPy einsum: np.einsum("ii -> ", arr)
#```
```

```python
torch.einsum('ii -> ', aten)
#Output
#tensor(110)
#```
#6) Matrix transpose PyTorch: torch.transpose(aten, 1, 0) NumPy einsum: np.einsum("ij -> ji", arr)
#```
```

```python
torch.einsum('ij -> ji', aten)
#Output
#tensor([[11, 21, 31, 41],
#        [12, 22, 32, 42],
#        [13, 23, 33, 43],
#        [14, 24, 34, 44]])
#```
#7) Outer Product (of vectors) PyTorch: torch.ger(vec, vec) NumPy einsum: np.einsum("i, j -> ij", vec, vec)
#```
```

```python
torch.einsum('i, j -> ij', vec, vec)
#Output
#tensor([[0, 0, 0, 0],
#        [0, 1, 2, 3],
#        [0, 2, 4, 6],
#        [0, 3, 6, 9]])
#```
#8) Inner Product (of vectors) PyTorch: torch.dot(vec1, vec2) NumPy einsum: np.einsum("i, i -> ", vec1, vec2)
#```
```

```python
torch.einsum('i, i -> ', vec, vec)
#Output
#tensor(14)
#```
#9) Sum along axis 0 PyTorch: torch.sum(aten, 0) NumPy einsum: np.einsum("ij -> j", arr)
#```
```

```python
torch.einsum('ij -> j', aten)
#Output
#tensor([104, 108, 112, 116])
#```
#10) Sum along axis 1 PyTorch: torch.sum(aten, 1) NumPy einsum: np.einsum("ij -> i", arr)
#```
```

```python
torch.einsum('ij -> i', aten)
#Output
#tensor([ 50,  90, 130, 170])
#```
#11) Batch Matrix Multiplication PyTorch: torch.bmm(batch_tensor_1, batch_tensor_2) NumPy : np.einsum("bij, bjk -> bik", batch_tensor_1, batch_tensor_2)
#```
## input batch tensors to work with
```

```python
batch_tensor_1 = torch.arange(2 * 4 * 3).reshape(2, 4, 3)
```

```python
batch_tensor_2 = torch.arange(2 * 3 * 4).reshape(2, 3, 4)
```

```python
torch.bmm(batch_tensor_1, batch_tensor_2)
#Output
#tensor([[[  20,   23,   26,   29],
#         [  56,   68,   80,   92],
#         [  92,  113,  134,  155],
#         [ 128,  158,  188,  218]],
#
#        [[ 632,  671,  710,  749],
#         [ 776,  824,  872,  920],
#         [ 920,  977, 1034, 1091],
#         [1064, 1130, 1196, 1262]]])
#
## sanity check with the shapes
```

```python
torch.bmm(batch_tensor_1, batch_tensor_2).shape
#Output
#torch.Size([2, 4, 4])
#
## batch matrix multiply using einsum
```

```python
torch.einsum("bij, bjk -> bik", batch_tensor_1, batch_tensor_2)
#Output
#tensor([[[  20,   23,   26,   29],
#         [  56,   68,   80,   92],
#         [  92,  113,  134,  155],
#         [ 128,  158,  188,  218]],
#
#        [[ 632,  671,  710,  749],
#         [ 776,  824,  872,  920],
#         [ 920,  977, 1034, 1091],
#         [1064, 1130, 1196, 1262]]])
#
## sanity check with the shapes
```

```python
torch.einsum("bij, bjk -> bik", batch_tensor_1, batch_tensor_2).shape
```
12) Sum along axis 2 PyTorch: torch.sum(batch_ten, 2) NumPy einsum: np.einsum("ijk -> ij", arr3D)
```
```

```python
torch.einsum("ijk -> ij", batch_ten)
#Output
#tensor([[ 50,  90, 130, 170],
#        [  4,   8,  12,  16]])
#```
#13) Sum all the elements in an nD tensor PyTorch: torch.sum(batch_ten) NumPy einsum: np.einsum("ijk -> ", arr3D)
#```
```

```python
torch.einsum("ijk -> ", batch_ten)
#Output
#tensor(480)
#```
#14) Sum over multiple axes (i.e. marginalization) PyTorch: torch.sum(arr, dim=(dim0, dim1, dim2, dim3, dim4, dim6, dim7)) NumPy: np.einsum("ijklmnop -> n", nDarr)
#```
## 8D tensor
```

```python
nDten = torch.randn((3,5,4,6,8,2,7,9))
```

```python
nDten.shape
#Output
#torch.Size([3, 5, 4, 6, 8, 2, 7, 9])
#
## marginalize out dimension 5 (i.e. "n" here)
```

```python
esum = torch.einsum("ijklmnop -> n", nDten)
```

```python
esum
#Output
#tensor([  98.6921, -206.0575])
#
## marginalize out axis 5 (i.e. sum over rest of the axes)
```

```python
tsum = torch.sum(nDten, dim=(0, 1, 2, 3, 4, 6, 7))
```

```python
torch.allclose(tsum, esum)
#Output
#True
#```
#15) Double Dot Products / Frobenius inner product (same as: torch.sum(hadamard-product) cf. 3) PyTorch: torch.sum(aten * bten) NumPy : np.einsum("ij, ij -> ", arr1, arr2)
#```
```

```python
torch.einsum("ij, ij -> ", aten, bten)
#Output
#tensor(1300)
#```
```


Question:
set very low values to zero in numpy In numpy I have an array like
```
[0 +  0.5j, 0.25 + 1.2352444e-24j, 0.25+ 0j, 2.46519033e-32 + 0j]
```
what is the fastest and easiest way to set the super low value to zero to get
```
[0 +  0.5j, 0.25 + 0j, 0.25+ 0j, 0 + 0j]
```
efficiency is not the paramount.

Response:
Hmmm. I'm not super-happy with it, but this seems to work:
```

```python
a = np.array([0 +  0.5j, 0.25 + 1.2352444e-24j, 0.25+ 0j, 2.46519033e-32 + 0j])
```

```python
a
#Output
#array([  0.00000000e+00 +5.00000000e-01j,
#         2.50000000e-01 +1.23524440e-24j,
#         2.50000000e-01 +0.00000000e+00j,   2.46519033e-32 +0.00000000e+00j])
```

```python
tol = 1e-16
```

```python
a.real[abs(a.real) < tol] = 0.0
```

```python
a.imag[abs(a.imag) < tol] = 0.0
```

```python
a
#Output
#array([ 0.00+0.5j,  0.25+0.j ,  0.25+0.j ,  0.00+0.j ])
#```
#and you can choose your tolerance as your problem requires. I usually use an order of magnitude or so higher than
#```
```

```python
np.finfo(np.float).eps
#Output
#2.2204460492503131e-16
#```
#but it's problem-dependent.
```


Question:
Standard deviation in numpy [duplicate] This question already has answers here: Python: Numpy standard deviation error (3 answers) Closed 9 years ago. Here is my code:
```
import numpy as np
print(np.std(np.array([0,1])))
```
it produces 0.5 I am confident that this is incorrect. What am I doing wrong?

Response:
By default, numpy.std returns the population standard deviation, in which case np.std([0,1]) is correctly reported to be 0.5. If you are looking for the sample standard deviation, you can supply an optional ddof parameter to std():
```

```python
np.std([0, 1], ddof=1)
#Output
#0.70710678118654757
#```
#ddof modifies the divisor of the sum of the squares of the samples-minus-mean. The divisor is N - ddof, where the default ddof is 0 as you can see from your result.
```


Question:
python numpy roll with padding I'd like to roll a 2D numpy array, except that I'd like to fill the ends with zeros rather than roll the data as if it were periodic. The following
```
import numpy as np

x = np.array([[1, 2, 3], [4, 5, 6]])

np.roll(x, 1, axis=1)
```
returns
```
array([[3, 1, 2], [6, 4, 5]])
```
but what I would prefer is
```
array([[0, 1, 2], [0, 4, 5]])
```

Response:
numpy.pad can be used to create an array with surrounding zeros. Pad seems to be quite powerful and can do much more than a simple "roll". The tuple ((0,0),(1,0)) used in this answer indicates the "side" of the matrix which to pad.
```
import numpy as np
x = np.array([[1, 2, 3],[4, 5, 6]])

print np.pad(x,((0,0),(1,0)), mode='constant')[:, :-1]
```
Giving
```
[[0 1 2]
 [0 4 5]]
```


Question:
What might be the cause of 'invalid value encountered in less_equal' in numpy I experienced a RuntimeWarning
```
RuntimeWarning: invalid value encountered in less_equal
```
Generated by this line of code of mine:
```
center_dists[j] <= center_dists[i]
```
Both center_dists[j] and center_dists[i] are numpy arrays What might be the cause of this warning ?

Response:
That's most likely happening because of a np.nan somewhere in the inputs involved. An example of it is shown below -
```

```python
A = np.array([4, 2, 1])
```

```python
B = np.array([2, 2, np.nan])
```

```python
A<=B
RuntimeWarning: invalid value encountered in less_equal
#Output
#array([False,  True, False], dtype=bool)
#```
#For all those comparisons involving np.nan, it would output False. Let's confirm it for a broadcasted comparison. Here's a sample -
#```
```

```python
A = np.array([4, 2, 1])
```

```python
B = np.array([2, 2, np.nan])
```

```python
A[:,None] <= B
RuntimeWarning: invalid value encountered in less_equal
#Output
#array([[False, False, False],
#       [ True,  True, False],
#       [ True,  True, False]], dtype=bool)
#```
#Please notice the third column in the output which corresponds to the comparison involving third element np.nan in B and that results in all False values.
```


Question: What is the advantage of saving `.npz` files instead of `.npy` in python, regarding speed, memory and look-up? The python documentation for the numpy.savez which saves an .npz file is: The .npz file format is a zipped archive of files named after the variables they contain. The archive is not compressed and each file in the archive contains one variable in .npy format. [...] When opening the saved .npz file with load a NpzFile object is returned. This is a dictionary-like object which can be queried for its list of arrays (with the .files attribute), and for the arrays themselves. My question is: what is the point of numpy.savez? Is it just a more elegant version (shorter command) to save multiple arrays, or is there a speed-up in the saving/reading process? Does it occupy less memory?

Response:
There are two parts of explanation for answering your question. I. NPY vs. NPZ As we already read from the doc, the .npy format is: the standard binary file format in NumPy for persisting a single arbitrary NumPy array on disk. ... The format is designed to be as simple as possible while achieving its limited goals. (sources) And .npz is only a simple way to combine multiple arrays into a single file, one can use ZipFile to contain multiple “.npy” files. We recommend using the file extension “.npz” for these archives. (sources) So, .npz is just a ZipFile containing multiple “.npy” files. And this ZipFile can be either compressed (by using np.savez_compressed) or uncompressed (by using np.savez). It's similar to tarball archive file in Unix-like system, where a tarball file can be just an uncompressed archive file which containing other files or a compressed archive file by combining with various compression programs (gzip, bzip2, etc.) II. Different APIs for binary serialization And Numpy also provides different APIs to produce these binary file output: np.save ---> Save an array to a binary file in NumPy .npy format np.savez --> Save several arrays into a single file in uncompressed .npz format np.savez_compressed --> Save several arrays into a single file in compressed .npz format np.load --> Load arrays or pickled objects from .npy, .npz or pickled files If we skim the source code of Numpy, under the hood:
```py
def _savez(file, args, kwds, compress, allow_pickle=True, pickle_kwargs=None):
    ...
    if compress:
        compression = zipfile.ZIP_DEFLATED
    else:
        compression = zipfile.ZIP_STORED
    ...

def savez(file, *args, **kwds):
    _savez(file, args, kwds, False)

def savez_compressed(file, *args, **kwds):
    _savez(file, args, kwds, True)
```
Then back to the question: If only use np.save, there is no more compression on top of the .npy format, only just a single archive file for the convenience of managing multiple related files. If use np.savez_compressed, then of course less memory on disk because of more CPU time to do the compression job (i.e. a bit slower).


Question: Retrieve name of column from its Index in Pandas I have a pandas dataframe and a numpy array of values of that dataframe. I have the index of a specific column and I already have the row index of an important value. Now I need to get the column name of that particular value from my dataframe. After searching through the documentations, I found out that I can do the opposite but not what I want.

Response:
I think you need index columns names by position (python counts from 0, so for fourth column need 3):
```
colname = df.columns[pos]
```
Sample:
```
df = pd.DataFrame({'A':[1,2,3],
                   'B':[4,5,6],
                   'C':[7,8,9],
                   'D':[1,3,5],
                   'E':[5,3,6],
                   'F':[7,4,3]})

print (df)
   A  B  C  D  E  F
0  1  4  7  1  5  7
1  2  5  8  3  3  4
2  3  6  9  5  6  3

pos = 3
colname = df.columns[pos]
print (colname)
D
```
```
pos = [3,5]
colname = df.columns[pos]
print (colname)
Index(['D', 'F'], dtype='object')
```


Question:
Python: ufunc 'add' did not contain a loop with signature matching types dtype('S21') dtype('S21') dtype('S21') I have two dataframes, which both have an Order ID and a date. I wanted to add a flag into the first dataframe df1: if a record with the same order id and date is in dataframe df2, then add a Y:
```
[ df1['R'] = np.where(orders['key'].isin(df2['key']), 'Y', 0)]
```
To accomplish that, I was going to create a key, which would be the concatenation of the order_id and date, but when I try the following code:
```
df1['key']=df1['Order_ID']+'_'+df1['Date']
```
I get this error
```
ufunc 'add' did not contain a loop with signature matching types dtype('S21') dtype('S21') dtype('S21')
```
df1 looks like this:
```
Date | Order_ID | other data points ...
201751 4395674  ...
201762 3487535  ...
```
These are the datatypes:
```
df1.info()
RangeIndex: 157443 entries, 0 to 157442
Data columns (total 6 columns):
Order_ID                                 157429 non-null object
Date                                     157443 non-null int64
...
dtypes: float64(2), int64(2), object(2)
memory usage: 7.2+ MB

df1['Order_ID'].values
array(['782833030', '782834969', '782836416', ..., '783678018',
       '783679806', '783679874'], dtype=object)
```

Response:
The problem is that you can't add an object array (containing strings) to a number array, that's just ambiguous:
```

```python
import pandas as pd
```

```python
pd.Series(['abc', 'def']) + pd.Series([1, 2])
#Output
#TypeError: ufunc 'add' did not contain a loop with signature matching types dtype('<U21') dtype('<U21') dtype('<U21')
#```
#You need to explicitly convert your Dates to str. I don't know how to do that efficiently in pandas but you can use:
#```
#df1['key'] = df1['Order_ID'] + '_' + df1['Date'].apply(str)  # .apply(str) is new
#```
```


Question:
Difference between these array shapes in numpy What is the difference between 2 arrays whose shapes are- (442,1) and (442,) ? Printing both of these produces an identical output, but when I check for equality ==, I get a 2D vector like this-
```
array([[ True, False, False, ..., False, False, False],
       [False,  True, False, ..., False, False, False],
       [False, False,  True, ..., False, False, False],
       ...,
       [False, False, False, ...,  True, False, False],
       [False, False, False, ..., False,  True, False],
       [False, False, False, ..., False, False,  True]], dtype=bool)
```
Can someone explain the difference?

Response:
An array of shape (442, 1) is 2-dimensional. It has 442 rows and 1 column. An array of shape (442, ) is 1-dimensional and consists of 442 elements. Note that their reprs should look different too. There is a difference in the number and placement of parenthesis:
```

```python
np.array([1,2,3]).shape
#Output
#(3,)
```

```python
np.array([[1],[2],[3]]).shape
#Output
#(3, 1)
#```
#Note that you could use np.squeeze to remove axes of length 1:
#```
```

```python
np.squeeze(np.array([[1],[2],[3]])).shape
#Output
#(3,)
#```
#NumPy broadcasting rules allow new axes to be automatically added on the left when needed. So (442,) can broadcast to (1, 442). And axes of length 1 can broadcast to any length. So when you test for equality between an array of shape (442, 1) and an array of shape (442, ), the second array gets promoted to shape (1, 442) and then the two arrays expand their axes of length 1 so that they both become broadcasted arrays of shape (442, 442). This is why when you tested for equality the result was a boolean array of shape (442, 442).
#```
```

```python
np.array([1,2,3]) == np.array([[1],[2],[3]])
#Output
#array([[ True, False, False],
#       [False,  True, False],
#       [False, False,  True]], dtype=bool)
```

```python
np.array([1,2,3]) == np.squeeze(np.array([[1],[2],[3]]))
#Output
#array([ True,  True,  True], dtype=bool)
#```
```


Question:
Concatenate sparse matrices in Python using SciPy/Numpy What would be the most efficient way to concatenate sparse matrices in Python using SciPy/Numpy? Here I used the following:
```

```python
np.hstack((X, X2))
#Output
#array([ <49998x70000 sparse matrix of type '<class 'numpy.float64'>'
#        with 1135520 stored elements in Compressed Sparse Row format>,
#        <49998x70000 sparse matrix of type '<class 'numpy.int64'>'
#        with 1135520 stored elements in Compressed Sparse Row format>],
#       dtype=object)
#```
#I would like to use both predictors in a regression, but the current format is obviously not what I'm looking for. Would it be possible to get the following:
#```
#<49998x1400000 sparse matrix of type '<class 'numpy.float64'>'
#     with 2271040 stored elements in Compressed Sparse Row format>
#```
#It is too large to be converted to a deep format.
```

Response:
You can use the scipy.sparse.hstack to concatenate sparse matrices with the same number of rows (horizontal concatenation):
```
from scipy.sparse import hstack
hstack((X, X2))
```
Similarly, you can use scipy.sparse.vstack to concatenate sparse matrices with the same number of columns (vertical concatenation). Using numpy.hstack or numpy.vstack will create an array with two sparse matrix objects.


Question:
Swap two rows in a numpy array in python [duplicate] This question already has an answer here: Row exchange in Numpy [duplicate] (1 answer) Closed 6 years ago. How to swap xth and yth rows of the 2-D NumPy array? x & y are inputs provided by the user. Lets say x = 0 & y =2 , and the input array is as below:
```
a = [[4 3 1]
         [5 7 0]
         [9 9 3]
         [8 2 4]]
Expected Output :
[[9 9 3]
 [5 7 0]
 [4 3 1]
 [8 2 4]]
```
I tried multiple things, but did not get the expected result. this is what i tried:
```
a[x],a[y]= a[y],a[x]

output i got is:
[[9 9 3]
 [5 7 0]
 [9 9 3]
 [8 2 4]]
```
Please suggest what is wrong in my solution.

Response:
Put the index as a whole:
```
a[[x, y]] = a[[y, x]]
```
With your example:
```
a = np.array([[4,3,1], [5,7,0], [9,9,3], [8,2,4]])

a
# array([[4, 3, 1],
#        [5, 7, 0],
#        [9, 9, 3],
#        [8, 2, 4]])

a[[0, 2]] = a[[2, 0]]
a
# array([[9, 9, 3],
#       [5, 7, 0],
#       [4, 3, 1],
#       [8, 2, 4]])
```


Question:
Cython: (Why / When) Is it preferable to use Py_ssize_t for indexing? This is a follow-up to this question. (Why / When) Is it preferable to use Py_ssize_t for indexing? In the docs I just found
```
# Purists could use "Py_ssize_t" which is the proper Python type for
# array indices.
```
-> Does that mean always when indexing NumPy/Cython - array(s)/-views one should use Py_ssize_t? -> Is Py_ssize_t e. g. an unsigned int so that I can't used @cython.boundscheck(False)

Response: Py_ssize_t is signed. See PEP 353, where it says "A new type Py_ssize_t is introduced, which has the same size as the compiler's size_t type, but is signed. It will be a typedef for ssize_t where available." You should use Py_ssize_t for indexing. I didn't find a definitive statement of this in the Cython docs, but Stefan Behnel, a Cython developer, said as much in an email (https://groups.google.com/forum/#!topic/cython-users/brENF_M9zxM): As a general remark, you are using ints as indices. You should use the Py_ssize_t type instead (or size_t, if you prefer an unsigned type) to properly accommodate for 64 bit architectures.


Question: How to multiply two vector and get a matrix? In numpy operation, I have two vectors, let's say vector A is 4X1, vector B is 1X5, if I do AXB, it should result a matrix of size 4X5. But I tried lot of times, doing many kinds of reshape and transpose, they all either raise error saying not aligned or return a single value. How should I get the output product of matrix I want?

Response:
Normal matrix multiplication works as long as the vectors have the right shape. Remember that * in Numpy is elementwise multiplication, and matrix multiplication is available with numpy.dot() (or with the @ operator, in Python 3.5)
```

```python
numpy.dot(numpy.array([[1], [2]]), numpy.array([[3, 4]]))
#Output
#array([[3, 4],
#       [6, 8]])
#```
#This is called an "outer product." You can get it using plain vectors using numpy.outer():
#```
```

```python
numpy.outer(numpy.array([1, 2]), numpy.array([3, 4]))
#Output
#array([[3, 4],
#       [6, 8]])
#```
```


Question:
bounding box of numpy array Suppose you have a 2D numpy array with some random values and surrounding zeros. Example "tilted rectangle":
```
import numpy as np
from skimage import transform

img1 = np.zeros((100,100))
img1[25:75,25:75] = 1.
img2 = transform.rotate(img1, 45)
```
Now I want to find the smallest bounding rectangle for all the nonzero data. For example:
```
a = np.where(img2 != 0)
bbox = img2[np.min(a[0]):np.max(a[0])+1, np.min(a[1]):np.max(a[1])+1]
```
What would be the fastest way to achieve this result? I am sure there is a better way since the np.where function takes quite a time if I am e.g. using 1000x1000 data sets. Edit: Should also work in 3D...

Response:
You can roughly halve the execution time by using np.any to reduce the rows and columns that contain non-zero values to 1D vectors, rather than finding the indices of all non-zero values using np.where:
```
def bbox1(img):
    a = np.where(img != 0)
    bbox = np.min(a[0]), np.max(a[0]), np.min(a[1]), np.max(a[1])
    return bbox

def bbox2(img):
    rows = np.any(img, axis=1)
    cols = np.any(img, axis=0)
    rmin, rmax = np.where(rows)[0][[0, -1]]
    cmin, cmax = np.where(cols)[0][[0, -1]]

    return rmin, rmax, cmin, cmax
```
Some benchmarks:
```
%timeit bbox1(img2)
10000 loops, best of 3: 63.5 µs per loop

%timeit bbox2(img2)
10000 loops, best of 3: 37.1 µs per loop
```
Extending this approach to the 3D case just involves performing the reduction along each pair of axes:
```
def bbox2_3D(img):

    r = np.any(img, axis=(1, 2))
    c = np.any(img, axis=(0, 2))
    z = np.any(img, axis=(0, 1))

    rmin, rmax = np.where(r)[0][[0, -1]]
    cmin, cmax = np.where(c)[0][[0, -1]]
    zmin, zmax = np.where(z)[0][[0, -1]]

    return rmin, rmax, cmin, cmax, zmin, zmax
```
It's easy to generalize this to N dimensions by using itertools.combinations to iterate over each unique combination of axes to perform the reduction over:
```
import itertools

def bbox2_ND(img):
    N = img.ndim
    out = []
    for ax in itertools.combinations(reversed(range(N)), N - 1):
        nonzero = np.any(img, axis=ax)
        out.extend(np.where(nonzero)[0][[0, -1]])
    return tuple(out)
```
If you know the coordinates of the corners of the original bounding box, the angle of rotation, and the centre of rotation, you could get the coordinates of the transformed bounding box corners directly by computing the corresponding affine transformation matrix and dotting it with the input coordinates:
```
def bbox_rotate(bbox_in, angle, centre):

    rmin, rmax, cmin, cmax = bbox_in

    # bounding box corners in homogeneous coordinates
    xyz_in = np.array(([[cmin, cmin, cmax, cmax],
                        [rmin, rmax, rmin, rmax],
                        [   1,    1,    1,    1]]))

    # translate centre to origin
    cr, cc = centre
    cent2ori = np.eye(3)
    cent2ori[:2, 2] = -cr, -cc

    # rotate about the origin
    theta = np.deg2rad(angle)
    rmat = np.eye(3)
    rmat[:2, :2] = np.array([[ np.cos(theta),-np.sin(theta)],
                             [ np.sin(theta), np.cos(theta)]])

    # translate from origin back to centre
    ori2cent = np.eye(3)
    ori2cent[:2, 2] = cr, cc

    # combine transformations (rightmost matrix is applied first)
    xyz_out = ori2cent.dot(rmat).dot(cent2ori).dot(xyz_in)

    r, c = xyz_out[:2]

    rmin = int(r.min())
    rmax = int(r.max())
    cmin = int(c.min())
    cmax = int(c.max())

    return rmin, rmax, cmin, cmax
```
This works out to be very slightly faster than using np.any for your small example array:
```
%timeit bbox_rotate([25, 75, 25, 75], 45, (50, 50))
10000 loops, best of 3: 33 µs per loop
```
However, since the speed of this method is independent of the size of the input array, it can be quite a lot faster for larger arrays. Extending the transformation approach to 3D is slightly more complicated, in that the rotation now has three different components (one about the x-axis, one about the y-axis and one about the z-axis), but the basic method is the same:
```
def bbox_rotate_3d(bbox_in, angle_x, angle_y, angle_z, centre):

    rmin, rmax, cmin, cmax, zmin, zmax = bbox_in

    # bounding box corners in homogeneous coordinates
    xyzu_in = np.array(([[cmin, cmin, cmin, cmin, cmax, cmax, cmax, cmax],
                         [rmin, rmin, rmax, rmax, rmin, rmin, rmax, rmax],
                         [zmin, zmax, zmin, zmax, zmin, zmax, zmin, zmax],
                         [   1,    1,    1,    1,    1,    1,    1,    1]]))

    # translate centre to origin
    cr, cc, cz = centre
    cent2ori = np.eye(4)
    cent2ori[:3, 3] = -cr, -cc -cz

    # rotation about the x-axis
    theta = np.deg2rad(angle_x)
    rmat_x = np.eye(4)
    rmat_x[1:3, 1:3] = np.array([[ np.cos(theta),-np.sin(theta)],
                                 [ np.sin(theta), np.cos(theta)]])

    # rotation about the y-axis
    theta = np.deg2rad(angle_y)
    rmat_y = np.eye(4)
    rmat_y[[0, 0, 2, 2], [0, 2, 0, 2]] = (
        np.cos(theta), np.sin(theta), -np.sin(theta), np.cos(theta))

    # rotation about the z-axis
    theta = np.deg2rad(angle_z)
    rmat_z = np.eye(4)
    rmat_z[:2, :2] = np.array([[ np.cos(theta),-np.sin(theta)],
                               [ np.sin(theta), np.cos(theta)]])

    # translate from origin back to centre
    ori2cent = np.eye(4)
    ori2cent[:3, 3] = cr, cc, cz

    # combine transformations (rightmost matrix is applied first)
    tform = ori2cent.dot(rmat_z).dot(rmat_y).dot(rmat_x).dot(cent2ori)
    xyzu_out = tform.dot(xyzu_in)

    r, c, z = xyzu_out[:3]

    rmin = int(r.min())
    rmax = int(r.max())
    cmin = int(c.min())
    cmax = int(c.max())
    zmin = int(z.min())
    zmax = int(z.max())

    return rmin, rmax, cmin, cmax, zmin, zmax
```
I've essentially just modified the function above using the rotation matrix expressions from here - I haven't had time to write a test-case yet, so use with caution.


Question:
Saving numpy array to txt file row wise I have an numpy array of form
```
a = [1,2,3]
```
which I want to save to a .txt file such that the file looks like:
```
1 2 3
```
If I use numpy.savetxt then I get a file like:
```
1
2
3
```
There should be a easy solution to this I suppose, any suggestions?

Response:
If numpy >= 1.5, you can do: # note that the filename is enclosed with double quotes, # example "filename.txt"
```
numpy.savetxt("filename", a, newline=" ")
```
Edit several 1D arrays with same length
```
a = numpy.array([1,2,3])
b = numpy.array([4,5,6])
numpy.savetxt(filename, (a,b), fmt="%d")

# gives:
# 1 2 3
# 4 5 6
```
several 1D arrays with variable length
```
a = numpy.array([1,2,3])
b = numpy.array([4,5])

with open(filename,"w") as f:
    f.write("\n".join(" ".join(map(str, x)) for x in (a,b)))

# gives:
# 1 2 3
# 4 5
```


Question:
python dict to numpy structured array I have a dictionary that I need to convert to a NumPy structured array. I'm using the arcpy function NumPyArraytoTable, so a NumPy structured array is the only data format that will work. Based on this thread: Writing to numpy array from dictionary and this thread: How to convert Python dictionary object to numpy array I've tried this:
```
result = {0: 1.1181753789488595, 1: 0.5566080288678394, 2: 0.4718269778030734, 3: 0.48716683119447185, 4: 1.0, 5: 0.1395076201641266, 6: 0.20941558441558442}

names = ['id','data']
formats = ['f8','f8']
dtype = dict(names = names, formats=formats)
array=numpy.array([[key,val] for (key,val) in result.iteritems()],dtype)
```
But I keep getting expected a readable buffer object The method below works, but is stupid and obviously won't work for real data. I know there is a more graceful approach, I just can't figure it out.
```
totable = numpy.array([[key,val] for (key,val) in result.iteritems()])
array=numpy.array([(totable[0,0],totable[0,1]),(totable[1,0],totable[1,1])],dtype)
```

Response:
You could use np.array(list(result.items()), dtype=dtype):
```
import numpy as np
result = {0: 1.1181753789488595, 1: 0.5566080288678394, 2: 0.4718269778030734, 3: 0.48716683119447185, 4: 1.0, 5: 0.1395076201641266, 6: 0.20941558441558442}

names = ['id','data']
formats = ['f8','f8']
dtype = dict(names = names, formats=formats)
array = np.array(list(result.items()), dtype=dtype)

print(repr(array))
```
yields
```
array([(0.0, 1.1181753789488595), (1.0, 0.5566080288678394),
       (2.0, 0.4718269778030734), (3.0, 0.48716683119447185), (4.0, 1.0),
       (5.0, 0.1395076201641266), (6.0, 0.20941558441558442)],
      dtype=[('id', '<f8'), ('data', '<f8')])
```
If you don't want to create the intermediate list of tuples, list(result.items()), then you could instead use np.fromiter: In Python2:
```
array = np.fromiter(result.iteritems(), dtype=dtype, count=len(result))
```
In Python3:
```
array = np.fromiter(result.items(), dtype=dtype, count=len(result))
```
Why using the list [key,val] does not work: By the way, your attempt,
```
numpy.array([[key,val] for (key,val) in result.iteritems()],dtype)
```
was very close to working. If you change the list [key, val] to the tuple (key, val), then it would have worked. Of course,
```
numpy.array([(key,val) for (key,val) in result.iteritems()], dtype)
```
is the same thing as
```
numpy.array(result.items(), dtype)
```
in Python2, or
```
numpy.array(list(result.items()), dtype)
```
in Python3. np.array treats lists differently than tuples: Robert Kern explains: As a rule, tuples are considered "scalar" records and lists are recursed upon. This rule helps numpy.array() figure out which sequences are records and which are other sequences to be recursed upon; i.e. which sequences create another dimension and which are the atomic elements. Since (0.0, 1.1181753789488595) is considered one of those atomic elements, it should be a tuple, not a list.


Question: why do we need np.squeeze()? Very often, arrays are squeezed with np.squeeze(). In the documentation, it says Remove single-dimensional entries from the shape of a. However I'm still wondering: Why are zero and nondimensional entries in the shape of a? Or to put it differently: Why do both a.shape = (2,1) and (2,) exist?

Response: Besides the mathematical differences between the two things, there is the issue of predictability. If your suggestion was followed, you could at no point rely on the dimension of your array. So any expression of the form my_array[x,y] would need to be replaced by something that first checks if my_array is actually two-dimensional and did not have an implicit squeeze at some point. This would probably obfuscate code far more than the occasional squeeze, which does a clearly specified thing. Actually, it might even be very hard to tell, which axis has been removed, leading to a whole host of new problems. In the spirit of The Zen of Python, also Explicit is better than implicit, we can also say that we should prefer explicit squeeze to implicit array conversion.


Question:
How can I get the value stored in a 0-dimensional Numpy array? I have a numpy array like this
```py
a = np.array(1)
```
How can I get the 1 value back from this array? a[0] does not work: it results in IndexError: 0-d arrays can't be indexed. Similarly, a(0) gives an error that says TypeError: 'numpy.ndarray' object is not callable.

Response:
What you create with
```
a = np.array(1)
```
is a zero-dimensional array, and these cannot be indexed. You also don't need to index it -- you can use a directly as if it were a scalar value. If you really need the value in a different type, say float, you can explicitly convert it with float(a). If you need it in the base type of the array, you can use a.item() or a[()]. Note that the zero-dimensional array is mutable. If you change the value of the single entry in the array, this will be visible via all references to the array you stored. Use a.item() if you want to store an immutable value. If you want a one-dimensional array with a single element instead, use
```
a = np.array([1])
```
You can access the single element with a[0] now.


Question:
Plot Normal distribution with Matplotlib [duplicate] This question already has answers here: How to plot normal distribution (10 answers) Closed 3 years ago. please help me to plot the normal distribution of the folowing data: DATA:
```
import numpy as np
import matplotlib.pyplot as plt
from scipy.stats import norm

h = [186, 176, 158, 180, 186, 168, 168, 164, 178, 170, 189, 195, 172,
     187, 180, 186, 185, 168, 179, 178, 183, 179, 170, 175, 186, 159,
     161, 178, 175, 185, 175, 162, 173, 172, 177, 175, 172, 177, 180]

std = np.std(h)
mean = np.mean(h)
plt.plot(norm.pdf(h,mean,std))
```
output:
```
Standard Deriviation = 8.54065575872
mean = 176.076923077
```
the plot is incorrect, what is wrong with my code?

Response:
Note: This solution is using pylab, not matplotlib.pyplot You may try using hist to put your data info along with the fitted curve as below:
```
import numpy as np
import scipy.stats as stats
import pylab as pl

h = sorted([186, 176, 158, 180, 186, 168, 168, 164, 178, 170, 189, 195, 172,
     187, 180, 186, 185, 168, 179, 178, 183, 179, 170, 175, 186, 159,
     161, 178, 175, 185, 175, 162, 173, 172, 177, 175, 172, 177, 180])  #sorted

fit = stats.norm.pdf(h, np.mean(h), np.std(h))  #this is a fitting indeed

pl.plot(h,fit,'-o')

pl.hist(h,normed=True)      #use this to draw histogram of your data

pl.show()                   #use may also need add this
```


Question:
How to resample a dataframe with different functions applied to each column? I have a times series with temperature and radiation in a pandas dataframe. The time resolution is 1 minute in regular steps.
```
import datetime
import pandas as pd
import numpy as np

date_times = pd.date_range(datetime.datetime(2012, 4, 5, 8, 0),
                           datetime.datetime(2012, 4, 5, 12, 0),
                           freq='1min')
tamb = np.random.sample(date_times.size) * 10.0
radiation = np.random.sample(date_times.size) * 10.0
frame = pd.DataFrame(data={'tamb': tamb, 'radiation': radiation},
                     index=date_times)
frame
<class 'pandas.core.frame.DataFrame'>
DatetimeIndex: 241 entries, 2012-04-05 08:00:00 to 2012-04-05 12:00:00
Freq: T
Data columns:
radiation    241  non-null values
tamb         241  non-null values
dtypes: float64(2)
```
How can I down-sample this dataframe to a resolution of one hour, computing the hourly mean for the temperature and the hourly sum for radiation?

Response:
With pandas 0.18 the resample API changed (see the docs). So for pandas >= 0.18 the answer is:
```

```python
frame.resample('1H').agg({'radiation': np.sum, 'tamb': np.mean})
#Output
#tamb   radiation
#2012-04-05 08:00:00  5.161235  279.507182
#2012-04-05 09:00:00  4.968145  290.941073
#2012-04-05 10:00:00  4.478531  317.678285
#2012-04-05 11:00:00  4.706206  335.258633
#2012-04-05 12:00:00  2.457873    8.655838
#```
#Old Answer: I am answering my question to reflect the time series related changes in pandas >= 0.8 (all other answers are outdated). Using pandas >= 0.8 the answer is:
#```
```

```python
frame.resample('1H', how={'radiation': np.sum, 'tamb': np.mean})
#Output
#tamb   radiation
#2012-04-05 08:00:00  5.161235  279.507182
#2012-04-05 09:00:00  4.968145  290.941073
#2012-04-05 10:00:00  4.478531  317.678285
#2012-04-05 11:00:00  4.706206  335.258633
#2012-04-05 12:00:00  2.457873    8.655838
#```
```


Question: How to generate a random normal distribution of integers How to generate a random integer as with np.random.randint(), but with a normal distribution around 0. np.random.randint(-10, 10) returns integers with a discrete uniform distribution np.random.normal(0, 0.1, 1) returns floats with a normal distribution What I want is a kind of combination between the two functions.

Response:
One other way to get a discrete distribution that looks like the normal distribution is to draw from a multinomial distribution where the probabilities are calculated from a normal distribution.
```
import scipy.stats as ss
import numpy as np
import matplotlib.pyplot as plt

x = np.arange(-10, 11)
xU, xL = x + 0.5, x - 0.5
prob = ss.norm.cdf(xU, scale = 3) - ss.norm.cdf(xL, scale = 3)
prob = prob / prob.sum() # normalize the probabilities so their sum is 1
nums = np.random.choice(x, size = 10000, p = prob)
plt.hist(nums, bins = len(x))
```
Here, np.random.choice picks an integer from [-10, 10]. The probability for selecting an element, say 0, is calculated by p(-0.5 < x < 0.5) where x is a normal random variable with mean zero and standard deviation 3. I chose a std. dev. of 3 because this way p(-10 < x < 10) is almost 1. The result looks like this:


Question:
Missing data, insert rows in Pandas and fill with NAN I'm new to Python and Pandas so there might be a simple solution which I don't see. I have a number of discontinuous datasets which look like this:
```
ind A    B  C
0   0.0  1  3
1   0.5  4  2
2   1.0  6  1
3   3.5  2  0
4   4.0  4  5
5   4.5  3  3
```
I now look for a solution to get the following:
```
ind A    B  C
0   0.0  1  3
1   0.5  4  2
2   1.0  6  1
3   1.5  NAN NAN
4   2.0  NAN NAN
5   2.5  NAN NAN
6   3.0  NAN NAN
7   3.5  2  0
8   4.0  4  5
9   4.5  3  3
```
The problem is,that the gap in A varies from dataset to dataset in position and length...

Response:
set_index and reset_index are your friends.
```
df = DataFrame({"A":[0,0.5,1.0,3.5,4.0,4.5], "B":[1,4,6,2,4,3], "C":[3,2,1,0,5,3]})
```
First move column A to the index:
```

```python
df.set_index("A")
#Output
#B  C
# A
#0.0  1  3
#0.5  4  2
#1.0  6  1
#3.5  2  0
#4.0  4  5
#4.5  3  3
#```
#Then reindex with a new index, here the missing data is filled in with nans. We use the Index object since we can name it; this will be used in the next step.
#```
```

```python
new_index = Index(arange(0,5,0.5), name="A")
```

```python
df.set_index("A").reindex(new_index)
#Output
#B   C
#0.0   1   3
#0.5   4   2
#1.0   6   1
#1.5 NaN NaN
#2.0 NaN NaN
#2.5 NaN NaN
#3.0 NaN NaN
#3.5   2   0
#4.0   4   5
#4.5   3   3
#```
#Finally move the index back to the columns with reset_index. Since we named the index, it all works magically:
#```
```

```python
df.set_index("A").reindex(new_index).reset_index()
#Output
#A   B   C
#0    0.0   1   3
#1    0.5   4   2
#2    1.0   6   1
#3    1.5 NaN NaN
#4    2.0 NaN NaN
#5    2.5 NaN NaN
#6    3.0 NaN NaN
#7    3.5   2   0
#8    4.0   4   5
#9    4.5   3   3
#```
```


Question:
Cython: cimport and import numpy as (both) np In the tutorial of the Cython documentation, there are cimport and import statements of numpy module:
```
import numpy as np
cimport numpy as np
```
I found this convention is quite popular among numpy/cython users. This looks strange for me because they are both named as np. In which part of the code, imported/cimported np are used? Why cython compiler does not confuse them?

Response:
cimport my_module gives access to C functions or attributes or even sub-modules under my_module import my_module gives access to Python functions or attributes or sub-modules under my_module. In your case:
```
cimport numpy as np
```
gives you access to Numpy C API, where you can declare array buffers, variable types and so on... And:
```
import numpy as np
```
gives you access to NumPy-Python functions, such as np.array, np.linspace, etc Cython internally handles this ambiguity so that the user does not need to use different names.


Question:
Element-wise string concatenation in numpy Is this a bug?
```
import numpy as np
a1=np.array(['a','b'])
a2=np.array(['E','F'])

```python
add(a1,a2)
#Output
#NotImplemented
#```
#I am trying to do element-wise string concatenation. I thought Add() was the way to do it in numpy but obviously it is not working as expected.
```

Response:
This can be done using numpy.char.add. Here is an example:
```

```python
import numpy as np
```

```python
a1 = np.array(['a', 'b'])
```

```python
a2 = np.array(['E', 'F'])
```

```python
np.char.add(a1, a2)
#Output
#array(['aE', 'bF'],
#      dtype='<U2')
#```
#(This was previously known as numpy.core.defchararray.add, and that name is still usable, but numpy.char.add is the preferred alias now.) There are other useful string operations available for NumPy data types.
```


Question:
How to convert a column or row matrix to a diagonal matrix in Python? I have a row vector A, A = [a1 a2 a3 ..... an] and I would like to create a diagonal matrix, B = diag(a1, a2, a3, ....., an) with the elements of this row vector. How can this be done in Python? UPDATE This is the code to illustrate the problem:
```
import numpy as np
a = np.matrix([1,2,3,4])
d = np.diag(a)
print (d)
```
the output of this code is [1], but my desired output is:
```
[[1 0 0 0]
 [0 2 0 0]
 [0 0 3 0]
 [0 0 0 4]]
```

Response:
You can use diag method:
```
import numpy as np

a = np.array([1,2,3,4])
d = np.diag(a)
# or simpler: d = np.diag([1,2,3,4])

print(d)
```
Results in:
```
[[1 0 0 0]
 [0 2 0 0]
 [0 0 3 0]
 [0 0 0 4]]
```
If you have a row vector, you can do this:
```
a = np.array([[1, 2, 3, 4]])
d = np.diag(a[0])
```
Results in:
```
[[1 0 0 0]
 [0 2 0 0]
 [0 0 3 0]
 [0 0 0 4]]
```
For the given matrix in the question:
```
import numpy as np
a = np.matrix([1,2,3,4])
d = np.diag(a.A1)
print (d)
```
Result is again:
```
[[1 0 0 0]
 [0 2 0 0]
 [0 0 3 0]
 [0 0 0 4]]
```


Question:
Running maximum of numpy array values I need a fast way to keep a running maximum of a numpy array. For example, if my array was:
```
x = numpy.array([11,12,13,20,19,18,17,18,23,21])
```
I'd want:
```
numpy.array([11,12,13,20,20,20,20,20,23,23])
```
Obviously I could do this with a little loop:
```
def running_max(x):
    result = [x[0]]
    for val in x:
        if val > result[-1]:
            result.append(val)
        else:
            result.append(result[-1])
    return result
```
But my arrays have hundreds of thousands of entries and I need to call this many times. It seems like there's got to be a numpy trick to remove the loop, but I can't seem to find anything that will work. The alternative will be to write this as a C extension, but it seems like I'd be reinventing the wheel.

Response:
numpy.maximum.accumulate works for me.
```

```python
import numpy
```

```python
numpy.maximum.accumulate(numpy.array([11,12,13,20,19,18,17,18,23,21]))
#Output
#array([11, 12, 13, 20, 20, 20, 20, 20, 23, 23])
#```
```


Question:
Copy numpy array into part of another array If I run the following:
```
import numpy as np
a = np.arange(9)
a = a.reshape((3,3))
```
I will get this:
```
a = [[0 1 2]
     [3 4 5]
     [6 7 8]]
```
If I create a larger array like this:
```
b = np.zeros((5,5))
b = [[ 0.  0.  0.  0.  0.]
     [ 0.  0.  0.  0.  0.]
     [ 0.  0.  0.  0.  0.]
     [ 0.  0.  0.  0.  0.]
     [ 0.  0.  0.  0.  0.]]
```
How do I efficiently copy a into b to get an array like this?
```
# border of 0 surrounding a to be filled in with other data later
b = [[ 0.  0.  0.  0.  0.]
     [ 0.  0.  1.  2.  0.]
     [ 0.  3.  4.  5.  0.]
     [ 0.  6.  7.  8.  0.]
     [ 0.  0.  0.  0.  0.]]
```
I am looking for a function built into numpy if it exists.

Response:
You can specify b[1:4, 1:4] to denote the part:
```

```python
import numpy as np
```

```python
a = np.arange(9)
```

```python
a = a.reshape((3, 3))
```

```python
b = np.zeros((5, 5))
```

```python
b[1:4, 1:4] = a
```

```python
b
#Output
#array([[ 0.,  0.,  0.,  0.,  0.],
#       [ 0.,  0.,  1.,  2.,  0.],
#       [ 0.,  3.,  4.,  5.,  0.],
#       [ 0.,  6.,  7.,  8.,  0.],
#       [ 0.,  0.,  0.,  0.,  0.]])
```

```python
b[1:4,1:4] = a + 1  # If you really meant `[1, 2, ..., 9]`
```

```python
b
#Output
#array([[ 0.,  0.,  0.,  0.,  0.],
#       [ 0.,  1.,  2.,  3.,  0.],
#       [ 0.,  4.,  5.,  6.,  0.],
#       [ 0.,  7.,  8.,  9.,  0.],
#       [ 0.,  0.,  0.,  0.,  0.]])
#```
```


Question:
Numpy list of 1D Arrays to 2D Array I have a large list files that contain 2D numpy arrays pickled through numpy.save. I am trying to read the first column of each file and create a new 2D array. I currently read each column using numpy.load with a mmap. The 1D arrays are now in a list.
```
col_list = []
for f in file_list:
    Temp = np.load(f,mmap_mode='r')
    col_list.append(Temp[:,0])
```
How can I convert this into a 2D array?

Response:
You can use
```
numpy.stack(arrays, axis=0)
```
if you have an array of arrays. You can specify the axis in case you want to stack columns and not rows.


Question:
Minimum Euclidean distance between points in two different Numpy arrays, not within I have two arrays of x-y coordinates, and I would like to find the minimum Euclidean distance between each point in one array with all the points in the other array. The arrays are not necessarily the same size. For example:
```
xy1=numpy.array(
[[  243,  3173],
[  525,  2997]])

xy2=numpy.array(
[[ 682, 2644],
[ 277, 2651],
[ 396, 2640]])
```
My current method loops through each coordinate xy in xy1 and calculates the distances between that coordinate and the other coordinates.
```
mindist=numpy.zeros(len(xy1))
minid=numpy.zeros(len(xy1))

for i,xy in enumerate(xy1):
    dists=numpy.sqrt(numpy.sum((xy-xy2)**2,axis=1))
    mindist[i],minid[i]=dists.min(),dists.argmin()
```
Is there a way to eliminate the for loop and somehow do element-by-element calculations between the two arrays? I envision generating a distance matrix for which I could find the minimum element in each row or column. Another way to look at the problem. Say I concatenate xy1 (length m) and xy2 (length p) into xy (length n), and I store the lengths of the original arrays. Theoretically, I should then be able to generate a n x n distance matrix from those coordinates from which I can grab an m x p submatrix. Is there a way to efficiently generate this submatrix?

Response:
(Months later) scipy.spatial.distance.cdist( X, Y ) gives all pairs of distances, for X and Y 2 dim, 3 dim ... It also does 22 different norms, detailed here .
```
# cdist example: (nx,dim) (ny,dim) -> (nx,ny)

from __future__ import division
import sys
import numpy as np
from scipy.spatial.distance import cdist

#...............................................................................
dim = 10
nx = 1000
ny = 100
metric = "euclidean"
seed = 1

    # change these params in sh or ipython: run this.py dim=3 ...
for arg in sys.argv[1:]:
    exec( arg )
np.random.seed(seed)
np.set_printoptions( 2, threshold=100, edgeitems=10, suppress=True )

title = "%s  dim %d  nx %d  ny %d  metric %s" % (
        __file__, dim, nx, ny, metric )
print "\n", title

#...............................................................................
X = np.random.uniform( 0, 1, size=(nx,dim) )
Y = np.random.uniform( 0, 1, size=(ny,dim) )
dist = cdist( X, Y, metric=metric )  # -> (nx, ny) distances
#...............................................................................

print "scipy.spatial.distance.cdist: X %s Y %s -> %s" % (
        X.shape, Y.shape, dist.shape )
print "dist average %.3g +- %.2g" % (dist.mean(), dist.std())
print "check: dist[0,3] %.3g == cdist( [X[0]], [Y[3]] ) %.3g" % (
        dist[0,3], cdist( [X[0]], [Y[3]] ))

# (trivia: how do pairwise distances between uniform-random points in the unit cube
# depend on the metric ? With the right scaling, not much at all:
# L1 / dim      ~ .33 +- .2/sqrt dim
# L2 / sqrt dim ~ .4 +- .2/sqrt dim
# Lmax / 2      ~ .4 +- .2/sqrt dim
```


Question:
Numpy Vector (N,1) dimension -> (N,) dimension conversion I have a question regarding the conversion between (N,) dimension arrays and (N,1) dimension arrays. For example, y is (2,) dimension.
```
A=np.array([[1,2],[3,4]])

x=np.array([1,2])

y=np.dot(A,x)

y.shape
Out[6]: (2,)
```
But the following will show y2 to be (2,1) dimension.
```
x2=x[:,np.newaxis]

y2=np.dot(A,x2)

y2.shape
Out[14]: (2, 1)
```
What would be the most efficient way of converting y2 back to y without copying? Thanks, Tom

Response:
reshape works for this
```
a  = np.arange(3)        # a.shape  = (3,)
b  = a.reshape((3,1))    # b.shape  = (3,1)
b2 = a.reshape((-1,1))   # b2.shape = (3,1)
c  = b.reshape((3,))     # c.shape  = (3,)
c2 = b.reshape((-1,))    # c2.shape = (3,)
```
note also that reshape doesn't copy the data unless it needs to for the new shape (which it doesn't need to do here):
```
a.__array_interface__['data']   # (22356720, False)
b.__array_interface__['data']   # (22356720, False)
c.__array_interface__['data']   # (22356720, False)
```


Question:
What does [:, :] mean on NumPy arrays Sorry for the stupid question. I'm programming in PHP but found some nice code in Python and want to "recreate" it in PHP. But I'm quite frustrated about the line:
```
self.h = -0.1
self.activity = numpy.zeros((512, 512)) + self.h
self.activity[:, :] = self.h
```
I don't understand what [:, :] means. I couldn't find an answer by googling it. Full code
```
import math
import numpy
import pygame
from scipy.misc import imsave
from scipy.ndimage.filters import gaussian_filter

class AmariModel(object):

    def __init__(self, size):
        self.h = -0.1
        self.k = 0.05
        self.K = 0.125
        self.m = 0.025
        self.M = 0.065

        self.stimulus = -self.h * numpy.random.random(size)
        self.activity = numpy.zeros(size) + self.h
        self.excitement = numpy.zeros(size)
        self.inhibition = numpy.zeros(size)

    def stimulate(self):
        self.activity[:, :] = self.activity > 0

        sigma = 1 / math.sqrt(2 * self.k)
        gaussian_filter(self.activity, sigma, 0, self.excitement, "wrap")
        self.excitement *= self.K * math.pi / self.k

        sigma = 1 / math.sqrt(2 * self.m)
        gaussian_filter(self.activity, sigma, 0, self.inhibition, "wrap")
        self.inhibition *= self.M * math.pi / self.m

        self.activity[:, :] = self.h
        self.activity[:, :] += self.excitement
        self.activity[:, :] -= self.inhibition
        self.activity[:, :] += self.stimulus

class AmariMazeGenerator(object):

    def __init__(self, size):
        self.model = AmariModel(size)

        pygame.init()
        self.display = pygame.display.set_mode(size, 0)
        pygame.display.set_caption("Amari Maze Generator")

    def run(self):
        pixels = pygame.surfarray.pixels3d(self.display)

        index = 0
        running = True
        while running:
            self.model.stimulate()

            pixels[:, :, :] = (255 * (self.model.activity > 0))[:, :, None]
            pygame.display.flip()

            for event in pygame.event.get():
                if event.type == pygame.QUIT:
                    running = False
                elif event.type == pygame.KEYDOWN:
                    if event.key == pygame.K_ESCAPE:
                        running = False
                    elif event.key == pygame.K_s:
                        imsave("{0:04d}.png".format(index), pixels[:, :, 0])
                        index = index + 1
                elif event.type == pygame.MOUSEBUTTONDOWN:
                    position = pygame.mouse.get_pos()
                    self.model.activity[position] = 1

        pygame.quit()

def main():
    generator = AmariMazeGenerator((512, 512))
    generator.run()

if __name__ == "__main__":
    main()
```

Response:
The [:, :] stands for everything from the beginning to the end just like for lists. The difference is that the first : stands for first and the second : for the second dimension.
```
a = numpy.zeros((3, 3))

```python
a
#Output
#array([[ 0.,  0.,  0.],
#       [ 0.,  0.,  0.],
#       [ 0.,  0.,  0.]])
#```
#Assigning to second row:
#```
```

```python
a[1, :] = 3
```

```python
a
#Output
#array([[ 0.,  0.,  0.],
#       [ 3.,  3.,  3.],
#       [ 0.,  0.,  0.]])
#```
#Assigning to second column:
#```
```

```python
a[:, 1] = 4
```

```python
a
#Output
#array([[ 0.,  4.,  0.],
#       [ 3.,  4.,  3.],
#       [ 0.,  4.,  0.]])
#```
#Assigning to all:
#```
```

```python
a[:] = 10
```

```python
a
#Output
#array([[ 10.,  10.,  10.],
#       [ 10.,  10.,  10.],
#       [ 10.,  10.,  10.]])
#```
```


Question:
Convert python list with None values to numpy array with nan values I am trying to convert a list that contains numeric values and None values to numpy.array, such that None is replaces with numpy.nan. For example:
```
my_list = [3,5,6,None,6,None]

# My desired result:
my_array = numpy.array([3,5,6,np.nan,6,np.nan])
```
Naive approach fails:
```

```python
my_list
#Output
#[3, 5, 6, None, 6, None]
```

```python
np.array(my_list)
#Output
#array([3, 5, 6, None, 6, None], dtype=object) # very limited
```

```python
_ * 2
#Output
#Traceback (most recent call last):
#  File "<stdin>", line 1, in <module>
#TypeError: unsupported operand type(s) for *: 'NoneType' and 'int'
```

```python
my_array # normal array can handle these operations
#Output
#array([  3.,   5.,   6.,  nan,   6.,  nan])
```

```python
my_array * 2
#Output
#array([  6.,  10.,  12.,  nan,  12.,  nan])
#```
#What is the best way to solve this problem?
```

Response:
You simply have to explicitly declare the data type:
```

```python
my_list = [3, 5, 6, None, 6, None]
```

```python
np.array(my_list, dtype=float)
#Output
#array([  3.,   5.,   6.,  nan,   6.,  nan])
#```
```


Question:
`AttributeError: rint` when using numpy.round I have a numpy array that looks like this:
```
[[41.743617 -87.626839]
 [41.936943 -87.669838]
 [41.962665 -87.65571899999999]]
```
I want to round the numbers in the array to two decimal places, or three. I tried using numpy.around and numpy.round, but both of them give me the following error:
```
File "/Library/Python/2.7/site-packages/numpy-1.8.0.dev_3084618_20130514-py2.7-macosx-10.8-intel.egg/numpy/core/fromnumeric.py", line 2452, in round_
    return round(decimals, out)
AttributeError: rint
```
i used numpy.around(x, decimals = 2) and numpy.round(x,decimals=2) Am I doing something wrong? Is there any other way to do this efficiently for a large array?

Response:
You cannot round numpy arrays that are objects, this can be changed with astype as long as your array can be safely converted to floats:
```

```python
a = np.random.rand(5).astype(np.object)
```

```python
a
#Output
#array([0.5137250555772075, 0.4279757819721647, 0.4177118178603122,
#       0.6270676923544128, 0.43733218329094947], dtype=object)
```

```python
np.around(a,3)
#Output
#Traceback (most recent call last):
#  File "<stdin>", line 1, in <module>
#  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/core/fromnumeric.py", line 2384, in around
#    return round(decimals, out)
#AttributeError: rint
```

```python
np.around(a.astype(np.double),3)
#Output
#array([ 0.514,  0.428,  0.418,  0.627,  0.437])
#```
#You will receive similar errors with string, unicode, void, and char type arrays.
```


Question:
Can't call strftime on numpy.datetime64, no definition I have a datetime64 t that I'd like to represent as a string. When I call strftime like this t.strftime('%Y.%m.%d') I get this error:
```
AttributeError: 'numpy.datetime64' object has no attribute 'strftime'
```
What am I missing? I am using Python 3.4.2 and Numpy 1.9.1

Response:
Importing a data structures library like pandas to accomplish type conversion feels like overkill to me. You can achieve the same thing with the standard datetime module:
```
import numpy as np
import datetime
t = np.datetime64('2017-10-26')
t = t.astype(datetime.datetime)
timestring = t.strftime('%Y.%m.%d')
```


Question:
Plotting power spectrum in python I have an array with 301 values, which were gathered from a movie clip with 301 frames. This means 1 value from 1 frame. The movie clip is running at 30 fps, so is in fact 10 sec long Now I would like to get the power spectrum of this "signal" ( with the right Axis). I tried:
```
X = fft(S_[:,2]);
 pl.plot(abs(X))
 pl.show()
```
I also tried:
```
X = fft(S_[:,2]);
 pl.plot(abs(X)**2)
 pl.show()
```
Though I don't think this is the real spectrum. the signal: The spectrum: The power spectrum : Can anyone provide some help with this ? I would like to have a plot in Hz.

Response:
Numpy has a convenience function, np.fft.fftfreq to compute the frequencies associated with FFT components:
```
from __future__ import division
import numpy as np
import matplotlib.pyplot as plt

data = np.random.rand(301) - 0.5
ps = np.abs(np.fft.fft(data))**2

time_step = 1 / 30
freqs = np.fft.fftfreq(data.size, time_step)
idx = np.argsort(freqs)

plt.plot(freqs[idx], ps[idx])
```
Note that the largest frequency you see in your case is not 30 Hz, but
```

```python
max(freqs)
#Output
#14.950166112956811
#```
#You never see the sampling frequency in a power spectrum. If you had had an even number of samples, then you would have reached the Nyquist frequency, 15 Hz in your case (although numpy would have calculated it as -15).
```


Question:
binning a dataframe in pandas in Python [duplicate] This question already has answers here: Binning a column with pandas (4 answers) Closed 5 years ago. Given the following dataframe in pandas:
```
import numpy as np
df = pandas.DataFrame({"a": np.random.random(100), "b": np.random.random(100), "id": np.arange(100)})
```
where id is an id for each point consisting of an a and b value, how can I bin a and b into a specified set of bins (so that I can then take the median/average value of a and b in each bin)? df might have NaN values for a or b (or both) for any given row in df. Here's a better example using Joe Kington's solution with a more realistic df. The thing I'm unsure about is how to access the df.b elements for each df.a group below:
```
a = np.random.random(20)
df = pandas.DataFrame({"a": a, "b": a + 10})
# bins for df.a
bins = np.linspace(0, 1, 10)
# bin df according to a
groups = df.groupby(np.digitize(df.a,bins))
# Get the mean of a in each group
print groups.mean()
## But how to get the mean of b for each group of a?
# ...
```

Response:
There may be a more efficient way (I have a feeling pandas.crosstab would be useful here), but here's how I'd do it:
```
import numpy as np
import pandas

df = pandas.DataFrame({"a": np.random.random(100),
                       "b": np.random.random(100),
                       "id": np.arange(100)})

# Bin the data frame by "a" with 10 bins...
bins = np.linspace(df.a.min(), df.a.max(), 10)
groups = df.groupby(np.digitize(df.a, bins))

# Get the mean of each bin:
print groups.mean() # Also could do "groups.aggregate(np.mean)"

# Similarly, the median:
print groups.median()

# Apply some arbitrary function to aggregate binned data
print groups.aggregate(lambda x: np.mean(x[x > 0.5]))
```
Edit: As the OP was asking specifically for just the means of b binned by the values in a, just do
```
groups.mean().b
```
Also if you wanted the index to look nicer (e.g. display intervals as the index), as they do in @bdiamante's example, use pandas.cut instead of numpy.digitize. (Kudos to bidamante. I didn't realize pandas.cut existed.)
```
import numpy as np
import pandas

df = pandas.DataFrame({"a": np.random.random(100),
                       "b": np.random.random(100) + 10})

# Bin the data frame by "a" with 10 bins...
bins = np.linspace(df.a.min(), df.a.max(), 10)
groups = df.groupby(pandas.cut(df.a, bins))

# Get the mean of b, binned by the values in a
print groups.mean().b
```
This results in:
```
a
(0.00186, 0.111]    10.421839
(0.111, 0.22]       10.427540
(0.22, 0.33]        10.538932
(0.33, 0.439]       10.445085
(0.439, 0.548]      10.313612
(0.548, 0.658]      10.319387
(0.658, 0.767]      10.367444
(0.767, 0.876]      10.469655
(0.876, 0.986]      10.571008
Name: b
```


Question:
Image.fromarray just produces black image I'm trying to save a numpy matrix as a grayscale image using Image.fromarray. It seems to work on a random matrix, but not on a particular one (where there should appear a circle). Can anyone explain what I'm doing wrong?
```
from PIL import Image
import numpy as np
radius = 0.5
size = 10
x,y = np.meshgrid(np.linspace(-1,1,size),np.linspace(-1,1,size))
f = np.vectorize(lambda x,y: ( 1.0 if x*x + y*y < radius*radius else 0.0))
z = f(x,y)
print(z)
zz = np.random.random((size,size))
img = Image.fromarray(zz,mode='L') #replace z with zz and it will just produce a black image
img.save('my_pic.png')
```

Response:
Image.fromarray is poorly defined with floating-point input; it's not well documented but the function assumes the input is laid-out as unsigned 8-bit integers. To produce the output you're trying to get, multiply by 255 and convert to uint8:
```
z = (z * 255).astype(np.uint8)
```
The reason it seems to work with the random array is that the bytes in this array, when interpreted as unsigned 8-bit integers, also look random. But the output is not the same random array as the input, which you can check by doing the above conversion on the random input:
```
np.random.seed(0)
zz = np.random.rand(size, size)
Image.fromarray(zz, mode='L').save('pic1.png')
```
```
Image.fromarray((zz * 255).astype('uint8'), mode='L').save('pic2.png')
```
Since the issue doesn't seem to be reported anywhere, I reported it on github: https://github.com/python-pillow/Pillow/issues/2856


Question: What is :: (double colon) in numpy like in myarray[0::3]? [duplicate] This question already has answers here: Closed 14 years ago. Possible Duplicate: What is :: (double colon) in Python? I read the question What is :: (double colon) in Python when subscripting sequences?, but this not answer what myarray[x::y] mean.

Response:
It prints every yth element from the list / array
```

```python
a = [1,2,3,4,5,6,7,8,9]
```

```python
a[::3]
#Output
#[1, 4, 7]
#```
#The additional syntax of a[x::y] means get every yth element starting at position x ie.
#```
```

```python
a[2::3]
#Output
#[3, 6, 9]
#```
```


Question:
Resizing and stretching a NumPy array I am working in Python and I have a NumPy array like this:
```
[1,5,9]
[2,7,3]
[8,4,6]
```
How do I stretch it to something like the following?
```
[1,1,5,5,9,9]
[1,1,5,5,9,9]
[2,2,7,7,3,3]
[2,2,7,7,3,3]
[8,8,4,4,6,6]
[8,8,4,4,6,6]
```
These are just some example arrays, I will actually be resizing several sizes of arrays, not just these. I'm new at this, and I just can't seem to wrap my head around what I need to do.

Response:
@KennyTM's answer is very slick, and really works for your case but as an alternative that might offer a bit more flexibility for expanding arrays try np.repeat:
```

```python
a = np.array([[1, 5, 9],
#Output
#              [2, 7, 3],
#              [8, 4, 6]])
```

```python
np.repeat(a,2, axis=1)
#Output
#array([[1, 1, 5, 5, 9, 9],
#       [2, 2, 7, 7, 3, 3],
#       [8, 8, 4, 4, 6, 6]])
#```
#So, this accomplishes repeating along one axis, to get it along multiple axes (as you might want), simply nest the np.repeat calls:
#```
```

```python
np.repeat(np.repeat(a,2, axis=0), 2, axis=1)
#Output
#array([[1, 1, 5, 5, 9, 9],
#       [1, 1, 5, 5, 9, 9],
#       [2, 2, 7, 7, 3, 3],
#       [2, 2, 7, 7, 3, 3],
#       [8, 8, 4, 4, 6, 6],
#       [8, 8, 4, 4, 6, 6]])
#```
#You can also vary the number of repeats for any initial row or column. For example, if you wanted two repeats of each row aside from the last row:
#```
```

```python
np.repeat(a, [2,2,1], axis=0)
#Output
#array([[1, 5, 9],
#       [1, 5, 9],
#       [2, 7, 3],
#       [2, 7, 3],
#       [8, 4, 6]])
#```
#Here when the second argument is a list it specifies a row-wise (rows in this case because axis=0) repeats for each row.
```


Question: kalman 2d filter in python My input is 2d (x,y) time series of a dot moving on a screen for a tracker software. It has some noise I want to remove using Kalman filter. Does someone can point me for a python code for Kalman 2d filter? In scipy cookbook I found only a 1d example: http://www.scipy.org/Cookbook/KalmanFiltering I saw there is implementation for Kalman filter in OpenCV, but couldn't find code examples. Thanks!

Response:
Here is my implementation of the Kalman filter based on the equations given on wikipedia. Please be aware that my understanding of Kalman filters is very rudimentary so there are most likely ways to improve this code. (For example, it suffers from the numerical instability problem discussed here. As I understand it, this only affects the numerical stability when Q, the motion noise, is very small. In real life, the noise is usually not small, so fortunately (at least for my implementation) in practice the numerical instability does not show up.) In the example below, kalman_xy assumes the state vector is a 4-tuple: 2 numbers for the location, and 2 numbers for the velocity. The F and H matrices have been defined specifically for this state vector: If x is a 4-tuple state, then
```
new_x = F * x
position = H * x
```
It then calls kalman, which is the generalized Kalman filter. It is general in the sense it is still useful if you wish to define a different state vector -- perhaps a 6-tuple representing location, velocity and acceleration. You just have to define the equations of motion by supplying the appropriate F and H.
```
import numpy as np
import matplotlib.pyplot as plt

def kalman_xy(x, P, measurement, R,
              motion = np.matrix('0. 0. 0. 0.').T,
              Q = np.matrix(np.eye(4))):
    """
    Parameters:
    x: initial state 4-tuple of location and velocity: (x0, x1, x0_dot, x1_dot)
    P: initial uncertainty convariance matrix
    measurement: observed position
    R: measurement noise
    motion: external motion added to state vector x
    Q: motion noise (same shape as P)
    """
    return kalman(x, P, measurement, R, motion, Q,
                  F = np.matrix('''
                      1. 0. 1. 0.;
                      0. 1. 0. 1.;
                      0. 0. 1. 0.;
                      0. 0. 0. 1.
                      '''),
                  H = np.matrix('''
                      1. 0. 0. 0.;
                      0. 1. 0. 0.'''))

def kalman(x, P, measurement, R, motion, Q, F, H):
    '''
    Parameters:
    x: initial state
    P: initial uncertainty convariance matrix
    measurement: observed position (same shape as H*x)
    R: measurement noise (same shape as H)
    motion: external motion added to state vector x
    Q: motion noise (same shape as P)
    F: next state function: x_prime = F*x
    H: measurement function: position = H*x

    Return: the updated and predicted new values for (x, P)

    See also http://en.wikipedia.org/wiki/Kalman_filter

    This version of kalman can be applied to many different situations by
    appropriately defining F and H
    '''
    # UPDATE x, P based on measurement m
    # distance between measured and current position-belief
    y = np.matrix(measurement).T - H * x
    S = H * P * H.T + R  # residual convariance
    K = P * H.T * S.I    # Kalman gain
    x = x + K*y
    I = np.matrix(np.eye(F.shape[0])) # identity matrix
    P = (I - K*H)*P

    # PREDICT x, P based on motion
    x = F*x + motion
    P = F*P*F.T + Q

    return x, P

def demo_kalman_xy():
    x = np.matrix('0. 0. 0. 0.').T
    P = np.matrix(np.eye(4))*1000 # initial uncertainty

    N = 20
    true_x = np.linspace(0.0, 10.0, N)
    true_y = true_x**2
    observed_x = true_x + 0.05*np.random.random(N)*true_x
    observed_y = true_y + 0.05*np.random.random(N)*true_y
    plt.plot(observed_x, observed_y, 'ro')
    result = []
    R = 0.01**2
    for meas in zip(observed_x, observed_y):
        x, P = kalman_xy(x, P, meas, R)
        result.append((x[:2]).tolist())
    kalman_x, kalman_y = zip(*result)
    plt.plot(kalman_x, kalman_y, 'g-')
    plt.show()

demo_kalman_xy()
```
The red dots show the noisy position measurements, the green line shows the Kalman predicted positions.


Question:
How to implode(reverse of pandas explode) based on a column I have a dataframe df like below
```
NETWORK       config_id       APPLICABLE_DAYS  Case    Delivery
0   Grocery     5399            SUN               10       1
1   Grocery     5399            MON               20       2
2   Grocery     5399            TUE               30       3
3   Grocery     5399            WED               40       4
```
I want to implode( combine Applicable_days from multiple rows into single row like below) and get the average case and delivery per config_id
```
NETWORK       config_id       APPLICABLE_DAYS      Avg_Cases    Avg_Delivery
0   Grocery     5399            SUN,MON,TUE,WED         90           10
```
using the groupby on network,config_id i can get the avg_cases and avg_delivery like below.
```
df.groupby(['network','config_id']).agg({'case':'mean','delivery':'mean'})
```
But How do i be able to join APPLICABLE_DAYS while performing this aggregation?

Response:
If you want the "opposite" of explode, then that means bringing it into a list in Solution #1. You can also join as a string in Solution #2: Use lambda x: x.tolist() for the 'APPLICABLE_DAYS' column within your .agg groupby function:
```
df = (df.groupby(['NETWORK','config_id'])
      .agg({'APPLICABLE_DAYS': lambda x: x.tolist(),'Case':'mean','Delivery':'mean'})
      .rename({'Case' : 'Avg_Cases','Delivery' : 'Avg_Delivery'},axis=1)
      .reset_index())
df
Out[1]:
   NETWORK  config_id       APPLICABLE_DAYS  Avg_Cases  Avg_Delivery
0  Grocery       5399  [SUN, MON, TUE, WED]         25           2.5
```
Use lambda x: ",".join(x) for the 'APPLICABLE_DAYS' column within your .agg groupby function:
```
df = (df.groupby(['NETWORK','config_id'])
      .agg({'APPLICABLE_DAYS': lambda x: ",".join(x),'Case':'mean','Delivery':'mean'})
      .rename({'Case' : 'Avg_Cases','Delivery' : 'Avg_Delivery'},axis=1)
      .reset_index())
df
Out[1]:
   NETWORK  config_id       APPLICABLE_DAYS  Avg_Cases  Avg_Delivery
0  Grocery       5399       SUN,MON,TUE,WED         25           2.5
```
If you are looking for the sum, then you can just change mean to sum for the Cases and Delivery columns.


Question: difference between np.inf and float('Inf') Is there some difference between NumPy np.inf and float('Inf')? float('Inf') == np.inf returns True, so it seems they are interchangeable, thus I was wondering why NumPy has defined its own "inf" constant, and when should I use one constant instead of the other (considering style concerns too)?

Response:
TL, DR: There is no difference and they can be used interchangeably. Besides having the same value as math.inf and float('inf'):
```

```python
import math
```

```python
import numpy as np
```

```python
np.inf == float('inf')
#Output
#True
```

```python
np.inf == math.inf
#Output
#True
#```
#It also has the same type:
#```
```

```python
import numpy as np
```

```python
type(np.inf)
#Output
#float
```

```python
type(np.inf) is type(float('inf'))
#Output
#float
#```
#That's interesting because NumPy also has it's own floating point types:
#```
```

```python
np.float32(np.inf)
#Output
#inf
```

```python
type(np.float32(np.inf))
#Output
#numpy.float32
```

```python
np.float32('inf') == np.inf  # nevertheless equal
#Output
#True
#```
#So it has the same value and the same type as math.inf and float('inf') which means it's interchangeable. Reasons for using np.inf It's less to type: np.inf (6 chars) math.inf (8 chars; new in python 3.5) float('inf') (12 chars) That means if you already have NumPy imported you can save yourself 6 (or 2) chars per occurrence compared to float('inf') (or math.inf). Because it's easier to remember. At least for me, it's far easier to remember np.inf than that I need to call float with a string. Also, NumPy defines some additional aliases for infinity:
#```
#np.Inf
#    np.inf
#    np.infty
#    np.Infinity
#    np.PINF
#```
#It also defines an alias for negative infinity:
#```
#np.NINF
#```
#Similarly for nan:
#```
#np.nan
#    np.NaN
#    np.NAN
#```
#Constants are constants This point is based on CPython and could be completely different in another Python implementation. A float CPython instance requires 24 Bytes:
#```
```

```python
import sys
#Output
#    >>> sys.getsizeof(np.inf)
#    24
#```
#If you can re-use the same instance you might save a lot of memory compared to creating lots of new instances. Of course, this point is mute if you create your own inf constant but if you don't then:
#```
#a = [np.inf for _ in range(1000000)]
#    b = [float('inf') for _ in range(1000000)]
#```
#b would use 24 * 1000000 Bytes (~23 MB) more memory than a. Accessing a constant is faster than creating the variable.
#```
#%timeit np.inf
# 37.9 ns ± 0.692 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)
# %timeit float('inf')
# 232 ns ± 13.9 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)
```

%timeit [np.inf for _ in range(10000)]
 552 µs ± 15.4 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
 %timeit [float('inf') for _ in range(10000)]
 2.59 ms ± 78.7 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
```
Of course, you can create your own constant to counter that point. But why bother if NumPy already did that for you.


Question:
NumPy save some arrays at once I working on different shapes of arrays and I want to save them all with numpy.save, so, consider I have
```
mat1 = numpy.arange(8).reshape(4, 2)
mat2 = numpy.arange(9).reshape(2, 3)
numpy.save('mat.npy', numpy.array([mat1, mat2]))
```
It works. But when I have two matrices with one dimension of same size it's not working.
```
mat1 = numpy.arange(8).reshape(2, 4)
mat2 = numpy.arange(10).reshape(2, 5)
numpy.save('mat.npy', numpy.array([mat1, mat2]))
```
It causes Traceback (most recent call last): File "", line 1, in  ValueError: could not broadcast input array from shape (2,4) into shape (2) And note that the problem caused by numpy.array([mat1, mat2]) and not by numpy.save I know that such array is possible: >> numpy.array([[[1, 2]], [[1, 2], [3, 4]]]) array([[[1, 2]], [[1, 2], [3, 4]]], dtype=object) So, all of what I want is to save two arrays as mat1 and mat2 at once.

Response:
If you'd like to save multiple arrays in the same format as np.save, use np.savez. For example:
```
import numpy as np

arr1 = np.arange(8).reshape(2, 4)
arr2 = np.arange(10).reshape(2, 5)
np.savez('mat.npz', name1=arr1, name2=arr2)

data = np.load('mat.npz')
print data['name1']
print data['name2']
```
If you have several arrays, you can expand the arguments:
```
import numpy as np

data = [np.arange(8).reshape(2, 4), np.arange(10).reshape(2, 5)]
np.savez('mat.npz', *data)

container = np.load('mat.npz')
data = [container[key] for key in container]
```
Note that the order is not preserved. If you do need to preserve order, you might consider using pickle instead. If you use pickle, be sure to specify the binary protocol, otherwise the you'll write things using ascii pickle, which is particularly inefficient for numpy arrays. With a binary protocol, ndarrays more or less pickle to the same format as np.save/np.savez. For example:
```
# Note: This is Python2.x specific. It's identical except for the import on 3.x
import cPickle as pickle
import numpy as np

data = [np.arange(8).reshape(2, 4), np.arange(10).reshape(2, 5)]

with open('mat.pkl', 'wb') as outfile:
    pickle.dump(data, outfile, pickle.HIGHEST_PROTOCOL)

with open('mat.pkl', 'rb') as infile:
    result = pickle.load(infile)
```
In this case, result and data will have identical contents and the order of the input list of arrays will be preserved.


Question: numpy.r_ is not a function. What is it? According to the numpy/scipy doc on numpy.r_ here, it is "not a function, so takes no parameters". If it is not a function, what is the proper term for "functions" such as numpy.r_?

Response:
I would argue that for all purposes r_ is a function, but one implemented by a clever hack using different syntax. Mike already explained how r_ is in reality not a function, but a class instance of RClass, which has __getitem__ implemented, so that you can use it as r_[1]. The cosmetic difference is that you use square brackets instead of curved ones, so you are not doing a function call, but you are actually indexing the object. Although this is technically true, for all purposes, it works just like a function call, but one that allows some extra syntax not allowed by a normal function. The motivation for creating r_ probably comes from Matlab's syntax, which allows to construct arrays in a very compact way, like x = [1:10, 15, 20:10:100]. To achieve the same in numpy, you would have to do x = np.hstack((np.arange(1,11), 15, np.arange(20,110,10))). Using colons to create ranges is not allowed in python, but they do exist in the form of the slice notation to index into a list, like L[3:5], and even A[2:10, 20:30] for multi-dimensional arrays. Under the hood, these index notation gets transformed to a call to the __getitem__ method of the object, where the colon notation gets transformed into a slice object:
```

```python
class C(object):
    ...:     def __getitem__(self, x):
    ...:         print x
```

```python
c = C()
```

```python
c[1:11, 15, 20:110:10]
(slice(1, 11, None), 15, slice(20, 110, 10))
```
The r_ object 'abuses' this fact to create a 'function' that accepts slice notation, which also does some additional things like concatenating everything together and returning the result, so that you can write x = np.r_[1:11, 15, 20:110:10]. The "Not a function, so takes no parameters" in the documentation is slightly misleading ...
```


Question:
how to plot and annotate hierarchical clustering dendrograms in scipy/matplotlib I'm using dendrogram from scipy to plot hierarchical clustering using matplotlib as follows:
```
mat = array([[1, 0.5, 0.9],
             [0.5, 1, -0.5],
             [0.9, -0.5, 1]])
plt.subplot(1,2,1)
plt.title("mat")
dist_mat = mat
linkage_matrix = linkage(dist_mat,
                         "single")
print "linkage2:"
print linkage(1-dist_mat, "single")
dendrogram(linkage_matrix,
           color_threshold=1,
           labels=["a", "b", "c"],
           show_leaf_counts=True)
plt.subplot(1,2,2)
plt.title("1 - mat")
dist_mat = 1 - mat
linkage_matrix = linkage(dist_mat,
                         "single")
dendrogram(linkage_matrix,
           color_threshold=1,
           labels=["a", "b", "c"],
           show_leaf_counts=True)
```
My questions are: first, why does mat and 1-mat give identical clusterings here? and second, how can I annotate the distance along each branch of the tree using dendrogram so that the distances between pairs of nodes can be compared? finally it seems that show_leaf_counts flag is ignored, is there a way to turn it on so that the number of objects in each class is shown? thanks.

Response:
The input to linkage() is either an n x m array, representing n points in m-dimensional space, or a one-dimensional array containing the condensed distance matrix. In your example, mat is 3 x 3, so you are clustering three 3-d points. Clustering is based on the distance between these points. Why does mat and 1-mat give identical clusterings here? The arrays mat and 1-mat produce the same clustering because the clustering is based on distances between the points, and neither a reflection (-mat) nor a translation (mat + offset) of the entire data set change the relative distances between the points. How can I annotate the distance along each branch of the tree using dendrogram so that the distances between pairs of nodes can be compared? In the code below, I show how you can use the data returned by dendrogram to label the horizontal segments of the diagram with the corresponding distance. The values associated with the keys icoord and dcoord give the x and y coordinates of each three-segment inverted-U of the figure. In augmented_dendrogram this data is used to add a label of the distance (i.e. y value) of each horizontal line segment in dendrogram.
```
from scipy.cluster.hierarchy import dendrogram
import matplotlib.pyplot as plt

def augmented_dendrogram(*args, **kwargs):

    ddata = dendrogram(*args, **kwargs)

    if not kwargs.get('no_plot', False):
        for i, d in zip(ddata['icoord'], ddata['dcoord']):
            x = 0.5 * sum(i[1:3])
            y = d[1]
            plt.plot(x, y, 'ro')
            plt.annotate("%.3g" % y, (x, y), xytext=(0, -8),
                         textcoords='offset points',
                         va='top', ha='center')

    return ddata
```
For your mat array, the augmented dendrogram is So point 'a' and 'c' are 1.01 units apart, and point 'b' is 1.57 units from the cluster ['a', 'c']. It seems that show_leaf_counts flag is ignored, is there a way to turn it on so that the number of objects in each class is shown? The flag show_leaf_counts only applies when not all the original data points are shown as leaves. For example, when trunc_mode = "lastp", only the last p nodes are show. Here's an example with 100 points:
```
import numpy as np
from scipy.cluster.hierarchy import linkage
import matplotlib.pyplot as plt
from augmented_dendrogram import augmented_dendrogram

# Generate a random sample of `n` points in 2-d.
np.random.seed(12312)
n = 100
x = np.random.multivariate_normal([0, 0], np.array([[4.0, 2.5], [2.5, 1.4]]),
                                  size=(n,))

plt.figure(1, figsize=(6, 5))
plt.clf()
plt.scatter(x[:, 0], x[:, 1])
plt.axis('equal')
plt.grid(True)

linkage_matrix = linkage(x, "single")

plt.figure(2, figsize=(10, 4))
plt.clf()

plt.subplot(1, 2, 1)
show_leaf_counts = False
ddata = augmented_dendrogram(linkage_matrix,
               color_threshold=1,
               p=6,
               truncate_mode='lastp',
               show_leaf_counts=show_leaf_counts,
               )
plt.title("show_leaf_counts = %s" % show_leaf_counts)

plt.subplot(1, 2, 2)
show_leaf_counts = True
ddata = augmented_dendrogram(linkage_matrix,
               color_threshold=1,
               p=6,
               truncate_mode='lastp',
               show_leaf_counts=show_leaf_counts,
               )
plt.title("show_leaf_counts = %s" % show_leaf_counts)

plt.show()
```
These are the points in the data set: With p=6 and trunc_mode="lastp", dendrogram only shows the "top" of the dendrogram. The following shows the effect of show_leaf_counts.


Question:
Python debugger tells me value of Numpy array is "*** Newest frame" What does this mean? My function gets two numpy arrays from a python/c library. After that function call I turn on the debugger to find a bug, so I add the line to look at the two numpy arrays.
```
import pdb; pdb.set_trace()
```
But for the values of one of the arrays pdb only returns the message *** Newest frame PDB output:
```
(Pdb) type(d)
<type 'numpy.ndarray'>
(Pdb) type(f)
<type 'numpy.ndarray'>
(Pdb) f.shape
(3, 3, 17856)
(Pdb) d[0].shape
*** Newest frame
(Pdb) d[0]
*** Newest frame
```

Response: The command d is the command for the debugger used to go down the stack to a 'newer frame'. It seems that the parsing cannot not handle this disambiguity. Try renaming the variable d. EDIT: Actually, the comments suggest much better handling than renaming.


Question:
Create large random boolean matrix with numpy I am trying to create a huge boolean matrix which is randomly filled with True and False with a given probability p. At first I used this code:
```
N = 30000
p = 0.1
np.random.choice(a=[False, True], size=(N, N), p=[p, 1-p])
```
But sadly it does not seem to terminate for this big N. So I tried to split it up into the generation of the single rows by doing this:
```
N = 30000
p = 0.1
mask = np.empty((N, N))
for i in range (N):
     mask[i] = np.random.choice(a=[False, True], size=N, p=[p, 1-p])
     if (i % 100 == 0):
          print(i)
```
Now, there happens something strange (at least on my device): The first ~1100 rows are very fastly generated - but after it, the code becomes horribly slow. Why is this happening? What do I miss here? Are there better ways to create a big matrix which has True entries with probability p and False entries with probability 1-p? Edit: As many of you assumed that the RAM will be a problem: As the device which will run the code has almost 500GB RAM, this won't be a problem.

Response:
The problem is your RAM, the values are being stored in memory as it's being created. I just created this matrix using this command: np.random.choice(a=[False, True], size=(N, N), p=[p, 1-p]) I used an AWS i3 instance with 64GB of RAM and 8 cores. To create this matrix, htop shows that it takes up ~20GB of RAM. Here is a benchmark in case you care:
```
time np.random.choice(a=[False, True], size=(N, N), p=[p, 1-p])

CPU times: user 18.3 s, sys: 3.4 s, total: 21.7 s
Wall time: 21.7 s

 def mask_method(N, p):
    for i in range(N):
        mask[i] = np.random.choice(a=[False, True], size=N, p=[p, 1-p])
        if (i % 100 == 0):
            print(i)

time mask_method(N,p)

CPU times: user 20.9 s, sys: 1.55 s, total: 22.5 s
Wall time: 22.5 s
```
Note that the mask method only takes up ~9GB of RAM at it's peak. Edit: The first method flushes the RAM after the process is done where as the function method retains all of it.


Question:
Why does numpy.linalg.solve() offer more precise matrix inversions than numpy.linalg.inv()? I do not quite understand why numpy.linalg.solve() gives the more precise answer, whereas numpy.linalg.inv() breaks down somewhat, giving (what I believe are) estimates. For a concrete example, I am solving the equation C^{-1} * d where C denotes a matrix, and d is a vector-array. For the sake of discussion, the dimensions of C are shape (1000,1000) and d is shape (1,1000). numpy.linalg.solve(A, b) solves the equation A*x=b for x, i.e. x = A^{-1} * b. Therefore, I could either solve this equation by (1)
```
inverse = numpy.linalg.inv(C)
result = inverse * d
```
or (2)
```
numpy.linalg.solve(C, d)
```
Method (2) gives far more precise results. Why is this? What exactly is happening such that one "works better" than the other?

Response: np.linalg.solve(A, b) does not compute the inverse of A. Instead it calls one of the gesv LAPACK routines, which first factorizes A using LU decomposition, then solves for x using forward and backward substitution (see here). np.linalg.inv uses the same method to compute the inverse of A by solving for A-1 in A·A-1 = I where I is the identity*. The factorization step is exactly the same as above, but it takes more floating point operations to solve for A-1 (an n×n matrix) than for x (an n-long vector). Additionally, if you then wanted to obtain x via the identity A-1·b = x then the extra matrix multiplication would incur yet more floating point operations, and therefore slower performance and more numerical error. There's no need for the intermediate step of computing A-1 - it is faster and more accurate to obtain x directly. * The relevant bit of source for inv is here. Unfortunately it's a bit tricky to understand since it's templated C. The important thing to note is that an identity matrix is being passed to the LAPACK solver as parameter B.


Question:
numpy savetxt formatted as integer is not saving zeroes I am trying to save numpy.array to .csv in the following way.
```
with open("resultTR.csv", "wb") as f:
    f.write(b'ImageId,Label\n')
    numpy.savetxt(f, result, fmt='%i', delimiter=",")
```
result is numpy.array that consists of two columns, first column are indices (numbers 1 through n) and second column values from (0,9). Unfortunately I have problem that whenever there is 0 in the second column then nothing is written to the resulting .csv file in the second column. In other words first five rows of array looks like this:
```
[[  1.00000000e+00   2.00000000e+00]
 [  2.00000000e+00   0.00000000e+00]
 [  3.00000000e+00   9.00000000e+00]
 [  4.00000000e+00   9.00000000e+00]
 [  5.00000000e+00   3.00000000e+00]
```
And first five rows of .csv file like this:
```
ImageId,Label
1,2
2
3,9
4,9
5,3
```
It looks to me like my code should work and thus not saving zeroes seems to me very weird. Does anyone have some idea what can possibly be wrong with my code for writing to .csv file? EDIT: Just for compleetnes my python version is 2.7.2 and it's running on Mac OS X 10.9.2

Response:
I would try saving the array as an int array, as in result.astype(int), or in full:
```
with open("resultTR.csv", "wb") as f:
    f.write(b'ImageId,Label\n')
    numpy.savetxt(f, result.astype(int), fmt='%i', delimiter=",")
```


Question:
Cannot convert list to array: ValueError: only one element tensors can be converted to Python scalars I'm currently working with the PyTorch framework and trying to understand foreign code. I got an indices issue and wanted to print the shape of a list. The only way of doing so (as far as Google tells me) is to convert the list into a numpy array and then getting the shape with numpy.ndarray.shape(). But trying to convert my list into an array, I got a ValueError: only one element tensors can be converted to Python scalars. My List is a converted PyTorch Tensor (list(pytorchTensor)) and looks somewhat like this:
```
[
tensor([[-0.2781, -0.2567, -0.2353,  ..., -0.9640, -0.9855, -1.0069],
        [-0.2781, -0.2567, -0.2353,  ..., -1.0069, -1.0283, -1.0927],
        [-0.2567, -0.2567, -0.2138,  ..., -1.0712, -1.1141, -1.1784],
        ...,
        [-0.6640, -0.6425, -0.6211,  ..., -1.0712, -1.1141, -1.0927],
        [-0.6640, -0.6425, -0.5997,  ..., -0.9426, -0.9640, -0.9640],
        [-0.6640, -0.6425, -0.5997,  ..., -0.9640, -0.9426, -0.9426]]),

tensor([[-0.0769, -0.0980, -0.0769,  ..., -0.9388, -0.9598, -0.9808],
        [-0.0559, -0.0769, -0.0980,  ..., -0.9598, -1.0018, -1.0228],
        [-0.0559, -0.0769, -0.0769,  ..., -1.0228, -1.0439, -1.0859],
        ...,
        [-0.4973, -0.4973, -0.4973,  ..., -1.0018, -1.0439, -1.0228],
        [-0.4973, -0.4973, -0.4973,  ..., -0.8757, -0.9177, -0.9177],
        [-0.4973, -0.4973, -0.4973,  ..., -0.9177, -0.8967, -0.8967]]),

tensor([[-0.1313, -0.1313, -0.1100,  ..., -0.8115, -0.8328, -0.8753],
        [-0.1313, -0.1525, -0.1313,  ..., -0.8541, -0.8966, -0.9391],
        [-0.1100, -0.1313, -0.1100,  ..., -0.9391, -0.9816, -1.0666],
        ...,
        [-0.4502, -0.4714, -0.4502,  ..., -0.8966, -0.8966, -0.8966],
        [-0.4502, -0.4714, -0.4502,  ..., -0.8115, -0.8115, -0.7903],
        [-0.4502, -0.4714, -0.4502,  ..., -0.8115, -0.7690, -0.7690]]),
]
```
Is there a way of getting the shape of that list without converting it into a numpy array?

Response:
It seems like you have a list of tensors. For each tensor you can see its size() (no need to convert to list/numpy). If you insist, you can convert a tensor to numpy array using numpy(): Return a list of tensor shapes:
```
>> [t.size() for t in my_list_of_tensors]
```
Returns a list of numpy arrays:
```
>> [t.numpy() for t in my_list_of_tensors]
```
In terms of performance, it is always best to avoid casting of tensors into numpy arrays, as it may incur sync of device/host memory. If you only need to check the shape of a tensor, use size() function.


Question:
Specifying dtype float32 with pandas.read_csv on pandas 0.10.1 I'm attempting to read a simple space-separated file with pandas read_csv method. However, pandas doesn't seem to be obeying my dtype argument. Maybe I'm incorrectly specifying it? I've distilled down my somewhat complicated call to read_csv to this simple test case. I'm actually using the converters argument in my 'real' scenario but I removed this for simplicity. Below is my ipython session:
```

```python
cat test.out
#Output
#a b
#0.76398 0.81394
#0.32136 0.91063
```

```python
import pandas
```

```python
import numpy
```

```python
x = pandas.read_csv('test.out', dtype={'a': numpy.float32}, delim_whitespace=True)
```

```python
x
#Output
#         a        b
#0  0.76398  0.81394
#1  0.32136  0.91063
```

```python
x.a.dtype
#Output
#dtype('float64')
#```
#I've also tried this using this with a dtype of numpy.int32 or numpy.int64. These choices result in an exception:
#```
#AttributeError: 'NoneType' object has no attribute 'dtype'
#```
#I'm assuming the AttributeError is because pandas will not automatically try to convert/truncate the float values into an integer? I'm running on a 32-bit machine with a 32-bit version of Python.
#```
```

```python
!uname -a
#Output
#Linux ubuntu 3.0.0-13-generic #22-Ubuntu SMP Wed Nov 2 13:25:36 UTC 2011 i686 i686 i386 GNU/Linux
```

```python
import platform
```

```python
platform.architecture()
#Output
#('32bit', 'ELF')
```

```python
pandas.__version__
#Output
#'0.10.1'
#```
```

Response:
0.10.1 doesn't really support float32 very much see this http://pandas.pydata.org/pandas-docs/dev/whatsnew.html#dtype-specification you can do this in 0.11 like this:
```
# dont' use dtype converters explicity for the columns you care about
# they will be converted to float64 if possible, or object if they cannot
df = pd.read_csv('test.csv'.....)

#### this is optional and related to the issue you posted ####
# force anything that is not a numeric to nan
# columns are the list of columns that you are interesetd in
df[columns] = df[columns].convert_objects(convert_numeric=True)

    # astype
    df[columns] = df[columns].astype('float32')

see http://pandas.pydata.org/pandas-docs/dev/basics.html#object-conversion

Its not as efficient as doing it directly in read_csv (but that requires
 some low-level changes)
```
I have confirmed that with 0.11-dev, this DOES work (on 32-bit and 64-bit, results are the same)
```

```python
x = pd.read_csv(StringIO.StringIO(data), dtype={'a': np.float32}, delim_whitespace=True)
```

```python
x
#Output
#a        b
#0  0.76398  0.81394
#1  0.32136  0.91063
```

```python
x.dtypes
#Output
#a    float32
#b    float64
#dtype: object
```

```python
pd.__version__
#Output
#'0.11.0.dev-385ff82'
```

```python
quit()
vagrant@precise32:~/pandas$ uname -a
Linux precise32 3.2.0-23-generic-pae #36-Ubuntu SMP Tue Apr 10 22:19:09 UTC 2012 i686 i686 i386 GNU/Linux
```
```


Question:
How to get the cumulative distribution function with NumPy? I want to create a CDF with NumPy, my code is the next:
```
histo = np.zeros(4096, dtype = np.int32)
for x in range(0, width):
   for y in range(0, height):
      histo[data[x][y]] += 1
      q = 0
   cdf = list()
   for i in histo:
      q = q + i
      cdf.append(q)
```
I am walking by the array but take a long time the program execution. There is a built function with this feature, isn't?

Response:
Using a histogram is one solution but it involves binning the data. This is not necessary for plotting a CDF of empirical data. Let F(x) be the count of how many entries are less than x then it goes up by one, exactly where we see a measurement. Thus, if we sort our samples then at each point we increment the count by one (or the fraction by 1/N) and plot one against the other we will see the "exact" (i.e. un-binned) empirical CDF. A following code sample demonstrates the method
```
import numpy as np
import matplotlib.pyplot as plt

N = 100
Z = np.random.normal(size = N)
# method 1
H,X1 = np.histogram( Z, bins = 10, normed = True )
dx = X1[1] - X1[0]
F1 = np.cumsum(H)*dx
#method 2
X2 = np.sort(Z)
F2 = np.array(range(N))/float(N)

plt.plot(X1[1:], F1)
plt.plot(X2, F2)
plt.show()
```
It outputs the following


Question:
Calculate Matrix Rank using scipy I'd like to calculate the mathematical rank of a matrix using scipy. The most obvious function numpy.rank calculates the dimension of an array (ie. scalars have dimension 0, vectors 1, matrices 2, etc...). I am aware that the numpy.linalg.lstsq module has this capability, but I was wondering if such a fundamental operation is built into the matrix class somewhere. Here is an explicit example:
```
from numpy import matrix, rank
A = matrix([[1,3,7],[2,8,3],[7,8,1]])
print rank(A)
```
This gives 2 the dimension, where I'm looking for an answer of 3.

Response:
Numpy provides numpy.linalg.matrix_rank():
```

```python
import numpy
```

```python
numpy.__version__
#Output
#'1.5.1'
```

```python
A = numpy.matrix([[1,3,7],[2,8,3],[7,8,1]])
```

```python
numpy.linalg.matrix_rank(A)
#Output
#3
#```
```