Question:
Extract csv file specific columns to list in Python What I'm trying to do is plot the latitude and longitude values of specific storms on a map using matplotlib,basemap,python, etc. My problem is that I'm trying to extract the latitude, longitude, and name of the storms on map but I keep getting errors between lines 41-44 where I try to extract the columns into the list. Here is what the file looks like:
```
1957,AUDREY,HU, 21.6N, 93.3W
1957,AUDREY,HU,22.0N,  93.4W
1957,AUDREY,HU,22.6N,  93.5W
1957,AUDREY,HU,23.2N,  93.6W
```
I want the list to look like the following:
```
latitude = [21.6N,22.0N,23.4N]
longitude = [93.3W, 93.5W,93.8W]
name = ["Audrey","Audrey"]
```
Here's what I have so far:
```
data = np.loadtxt('louisianastormb.csv',dtype=np.str,delimiter=',',skiprows=1)
'''print data'''

data = np.loadtxt('louisianastormb.csv',dtype=np.str,delimiter=',',skiprows=0)

f= open('louisianastormb.csv', 'rb')
reader = csv.reader(f, delimiter=',')
header = reader.next()
zipped = zip(*reader)

latitude = zipped[3]
longitude = zipped[4]
names = zipped[1]
x, y = m(longitude, latitude)
```
Here's the last error message/traceback I received: Traceback (most recent call last): File "/home/darealmzd/lstorms.py", line 42, in header = reader.next() _csv.Error: new-line character seen in unquoted field - do you need to open the file in universal-newline mode?

Response:
This looks like a problem with line endings in your code. If you're going to be using all these other scientific packages, you may as well use Pandas for the CSV reading part, which is both more robust and more useful than just the csv module:
```
import pandas
colnames = ['year', 'name', 'city', 'latitude', 'longitude']
data = pandas.read_csv('test.csv', names=colnames)
```
If you want your lists as in the question, you can now do:
```
names = data.name.tolist()
latitude = data.latitude.tolist()
longitude = data.longitude.tolist()
```


Question: How to check the size of a float in python? I want to check whether a float is actually 32 or 64bits (and the number of bits of a numpy float array). There should be a built-in, but just didn't find out...

Response: Properties of a Python float can be requested via sys.float_info. It returns information such as max/min value, max/min exp value, etc. These properties can potentially be used to calculate the byte size of a float. I never encountered anything else than 64 bit, though, on many different architectures. The items of a NumPy array might have different size, but you can check their size in bytes by a.itemsize, where a is a NumPy array.


Question: R summary() equivalent in numpy Is there an equivalent of R's summary() function in numpy? numpy has std, mean, average functions separately, but does it have a function that sums up everything, like summary does in R? If found this question which relates to pandas and this article with R-to-numpy equivalents, but it doesn't have what I seek for.

Response:
1. Load Pandas in console and load csv data file
```
import pandas as pd

data = pd.read_csv("data.csv", sep = ",")
```
2. Examine first few rows of data
```
data.head()
```
3. Calculate summary statistics
```
summary = data.describe()
```
4. Transpose statistics to get similar format as R summary() function
```
summary = summary.transpose()
```
5. Visualize summary statistics in console
```
summary.head()
```


Question:
Numpy ValueError: setting an array element with a sequence. This message may appear without the existing of a sequence? [duplicate] This question already has answers here: ValueError: setting an array element with a sequence (11 answers) Closed 1 year ago. Why do I get this error message? ValueError: setting an array element with a sequence. Thank you
```
Z=np.array([1.0,1.0,1.0,1.0])

def func(TempLake,Z):
    A=TempLake
    B=Z
    return A*B

Nlayers=Z.size
N=3
TempLake=np.zeros((N+1,Nlayers))

kOUT=np.zeros(N+1)
for i in xrange(N):
    kOUT[i]=func(TempLake[i],Z)
```

Response:
You're getting the error message
```
ValueError: setting an array element with a sequence.
```
because you're trying to set an array element with a sequence. I'm not trying to be cute, there -- the error message is trying to tell you exactly what the problem is. Don't think of it as a cryptic error, it's simply a phrase. What line is giving the problem?
```
kOUT[i]=func(TempLake[i],Z)
```
This line tries to set the ith element of kOUT to whatever func(TempLAke[i], Z) returns. Looking at the i=0 case:
```

```python
kOUT[0]
#Output
#0.0
```

```python
func(TempLake[0], Z)
#Output
#array([ 0.,  0.,  0.,  0.])
#```
#You're trying to load a 4-element array into kOUT[0] which only has a float. Hence, you're trying to set an array element (the left hand side, kOUT[i]) with a sequence (the right hand side, func(TempLake[i], Z)). Probably func isn't doing what you want, but I'm not sure what you really wanted it to do (and don't forget you can usually use vectorized operations like A*B rather than looping in numpy.) That should explain the problem, anyway.
```


Question:
How does the axis parameter from NumPy work? Can someone explain exactly what the axis parameter in NumPy does? I am terribly confused. I'm trying to use the function myArray.sum(axis=num) At first I thought if the array is itself 3 dimensions, axis=0 will return three elements, consisting of the sum of all nested items in that same position. If each dimension contained five dimensions, I expected axis=1 to return a result of five items, and so on. However this is not the case, and the documentation does not do a good job helping me out (they use a 3x3x3 array so it's hard to tell what's happening) Here's what I did:
```

```python
e
#Output
#array([[[1, 0],
#        [0, 0]],
```

[[1, 1],
        [1, 0]],

       [[1, 0],
        [0, 1]]])

```python
e.sum(axis = 0)
#Output
#array([[3, 1],
#       [1, 1]])
```

```python
e.sum(axis=1)
#Output
#array([[1, 0],
#       [2, 1],
#       [1, 1]])
```

```python
e.sum(axis=2)
#Output
#array([[1, 0],
#       [2, 1],
#       [1, 1]])
#>>>
#```
#Clearly the result is not intuitive.
```

Response:
Clearly,
```
e.shape == (3, 2, 2)
```
Sum over an axis is a reduction operation so the specified axis disappears. Hence,
```
e.sum(axis=0).shape == (2, 2)
e.sum(axis=1).shape == (3, 2)
e.sum(axis=2).shape == (3, 2)
```
Intuitively, we are "squashing" the array along the chosen axis, and summing the numbers that get squashed together.


Question:
How can I create a slice object for Numpy array? I've tried to find a neat solution to this, but I'm slicing several 2D arrays of the same shape in the same manner. I've tidied it up as much as I can by defining a list containing the 'x,y' center e.g. cpix = [161, 134] What I'd like to do is instead of having to write out the slice three times like so:
```
a1 = array1[cpix[1]-50:cpix[1]+50, cpix[0]-50:cpix[0]+50]
a2 = array2[cpix[1]-50:cpix[1]+50, cpix[0]-50:cpix[0]+50]
a3 = array3[cpix[1]-50:cpix[1]+50, cpix[0]-50:cpix[0]+50]
```
is just have something predefined (like maybe a mask?) so I can just do a
```
a1 = array1[predefined_2dslice]
a2 = array2[predefined_2dslice]
a3 = array3[predefined_2dslice]
```
Is this something that numpy supports?

Response:
Yes you can use numpy.s_: Example:
```

```python
a = np.arange(10).reshape(2, 5)
```

```python

```

```python
m = np.s_[0:2, 3:4]
```

```python

```

```python
a[m]
#Output
#array([[3],
#       [8]])
#```
#And in this case:
#```
#my_slice = np.s_[cpix[1]-50:cpix[1]+50, cpix[0]-50:cpix[0]+50]
```

a1 = array1[my_slice]
a2 = array2[my_slice]
a3 = array3[my_slice]
```
You can also use numpy.r_ in order to translates slice objects to concatenation along the first axis.


Question:
Convert a numpy.ndarray to string(or bytes) and convert it back to numpy.ndarray I'm having a little trouble here, I'm trying to convert a numpy.ndarray to string, I've already done that like this:
```
randomArray.tostring()
```
It works, but I'm wondering if I can transform it back to a numpy.ndarray. What's the best way to do this? I'm using numpy 1.8.1 Context: The objective is to send the numpy.ndarray as a message in rabbitmq (pika library)

Response:
You can use the fromstring() method for this:
```
arr = np.array([1, 2, 3, 4, 5, 6])
ts = arr.tostring()
print(np.fromstring(ts, dtype=int))

```python
[1 2 3 4 5 6]
#Output
#```
#Sorry for the short answer, not enough points for commenting. Remember to state the data types or you'll end up in a world of pain. Note on fromstring from numpy 1.14 onwards: sep : str, optional The string separating numbers in the data; extra whitespace between elements is also ignored. Deprecated since version 1.14: Passing sep='', the default, is deprecated since it will trigger the deprecated binary mode of this function. This mode interprets string as binary bytes, rather than ASCII text with decimal numbers, an operation which is better spelt frombuffer(string, dtype, count). If string contains unicode text, the binary mode of fromstring will first encode it into bytes using either utf-8 (python 3) or the default encoding (python 2), neither of which produce sane results.
```


Question:
numpy: what is the logic of the argmin() and argmax() functions? I can not understand the output of argmax and argmin when use with the axis parameter. For example:
```

```python
a = np.array([[1,2,4,7], [9,88,6,45], [9,76,3,4]])
```

```python
a
#Output
#array([[ 1,  2,  4,  7],
#       [ 9, 88,  6, 45],
#       [ 9, 76,  3,  4]])
```

```python
a.shape
#Output
#(3, 4)
```

```python
a.size
#Output
#12
```

```python
np.argmax(a)
#Output
#5
```

```python
np.argmax(a,axis=0)
#Output
#array([1, 1, 1, 1])
```

```python
np.argmax(a,axis=1)
#Output
#array([3, 1, 1])
```

```python
np.argmin(a)
#Output
#0
```

```python
np.argmin(a,axis=0)
#Output
#array([0, 0, 2, 2])
```

```python
np.argmin(a,axis=1)
#Output
#array([0, 2, 2])
#```
#As you can see, the maximum value is the point (1,1) and the minimum one is the point (0,0). So in my logic when I run: np.argmin(a,axis=0) I expected array([0,0,0,0]) np.argmin(a,axis=1) I expected array([0,0,0]) np.argmax(a,axis=0) I expected array([1,1,1,1]) np.argmax(a,axis=1) I expected array([1,1,1]) What is wrong with my understanding of things?
```

Response:
By adding the axis argument, NumPy looks at the rows and columns individually. When it's not given, the array a is flattened into a single 1D array. axis=0 means that the operation is performed down the columns of a 2D array a in turn. For example np.argmin(a, axis=0) returns the index of the minimum value in each of the four columns. The minimum value in each column is shown in bold below:
```

```python
a
#Output
#array([[
#1
#,
#2
#,  4,  7],  # 0
#       [ 9, 88,  6, 45],  # 1
#       [ 9, 76,
#3
#,
#4
#]]) # 2
```

```python
np.argmin(a, axis=0)
#Output
#array([0, 0, 2, 2])
#```
#On the other hand, axis=1 means that the operation is performed across the rows of a. That means np.argmin(a, axis=1) returns [0, 2, 2] because a has three rows. The index of the minimum value in the first row is 0, the index of the minimum value of the second and third rows is 2:
#```
```

```python
a
#Output
##        0   1   2   3
#array([[
#1
#,  2,  4,  7],
#       [ 9, 88,
#6
#, 45],
#       [ 9, 76,
#3
#,  4]])
```

```python
np.argmin(a, axis=1)
#Output
#array([0, 2, 2])
#```
```


Question:
cv2.imread: checking if image is being read I'm writing an OpenCV program in python, and at some point I have something like
```
import cv2
import numpy as np
...
img = cv2.imread("myImage.jpg")

# do stuff with image here
```
The problem is that I have to detect if the image file is being correctly read before continuing. cv2.imread returns False if not able to open the image, so I think of doing something like:
```
if (img):
   #continue doing stuff
```
What happens is that if the image is not opened (e.g. if the file does not exist) img is equal to None (as expected). However, when imread works, the condition, breaks:
```
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
```
i.e. the returned numpy.ndarray cannot be used as a boolean. The problem seems to be that imread returns numpy.ndarray if success and False (boolean) otherwise. My solution so far involves using the type of the returned value as follows:
```
if (type(img) is np.ndarray):
     #do stuff with image
```
But I was wondering: isn't there a nicer solution, closer to the initial check if(img): #do stuff ?

Response: If you're sure that the value of img is None in your case, you can simply use if not img is None, or, equivalently, if img is not None. You don't need to check the type explicitly. Note that None and False are not the same value. However, bool(None)==False, which is why if None fails. The documentation for imread, both for OpenCV 2 and 3, states, however, that a empty matrix should be returned on error. You can check for that using if img.size ==0


Question:
How to rearrange array based upon index array I'm looking for a one line solution that would help me do the following. Suppose I have
```
array = np.array([10, 20, 30, 40, 50])
```
I'd like to rearrange it based upon an input ordering. If there were a numpy function called arrange, it would do the following:
```
newarray = np.arrange(array, [1, 0, 3, 4, 2])
print newarray

    [20, 10, 40, 50, 30]
```
Formally, if the array to be reordered is m x n, and the "index" array is 1 x n, the ordering would be determined by the array called "index". Does numpy have a function like this?

Response:
You can simply use your "index" list directly, as, well, an index array:
```

```python
arr = np.array([10, 20, 30, 40, 50])
```

```python
idx = [1, 0, 3, 4, 2]
```

```python
arr[idx]
#Output
#array([20, 10, 40, 50, 30])
#```
#It tends to be much faster if idx is already an ndarray and not a list, even though it'll work either way:
#```
```

```python
%timeit arr[idx]
#Output
#100000 loops, best of 3: 2.11 µs per loop
```

```python
ai = np.array(idx)
```

```python
%timeit arr[ai]
#Output
#1000000 loops, best of 3: 296 ns per loop
#```
```


Question: Can anybody explain me the numpy.indices()? I've read documentation several times about np.indices() but I can't seem to grasp what is it about. I've used it numerous times on things to see what it does, but I still can't really get it. Maybe the thing is I'm a beginner in programming so I can't understand the idea behind the words describing it. In addition I'm not a native English speaker (though I have no problems with it). I would be very grateful for kind of easier explanation, possibly on some example. Thanks.

Response:
Suppose you have a matrix M whose (i,j)-th element equals
```
M_ij = 2*i + 3*j
```
One way to define this matrix would be
```
i, j = np.indices((2,3))
M = 2*i + 3*j
```
which yields
```
array([[0, 3, 6],
       [2, 5, 8]])
```
In other words, np.indices returns arrays which can be used as indices. The elements in i indicate the row index:
```

```python
i
#Output
#array([[0, 0, 0],
#       [1, 1, 1]])
#```
#The elements in j indicate the column index:
#```
```

```python
j
#Output
#array([[0, 1, 2],
#       [0, 1, 2]])
#```
```


Question:
Numerically stable softmax Is there a numerically stable way to compute softmax function below? I am getting values that becomes Nans in Neural network code.
```
np.exp(x)/np.sum(np.exp(y))
```

Response: The softmax exp(x)/sum(exp(x)) is actually numerically well-behaved. It has only positive terms, so we needn't worry about loss of significance, and the denominator is at least as large as the numerator, so the result is guaranteed to fall between 0 and 1. The only accident that might happen is over- or under-flow in the exponentials. Overflow of a single or underflow of all elements of x will render the output more or less useless. But it is easy to guard against that by using the identity softmax(x) = softmax(x + c) which holds for any scalar c: Subtracting max(x) from x leaves a vector that has only non-positive entries, ruling out overflow and at least one element that is zero ruling out a vanishing denominator (underflow in some but not all entries is harmless). Footnote: theoretically, catastrophic accidents in the sum are possible, but you'd need a ridiculous number of terms. For example, even using 16 bit floats which can only resolve 3 decimals---compared to 15 decimals of a "normal" 64 bit float---we'd need between 2^1431 (~6 x 10^431) and 2^1432 to get a sum that is off by a factor of two.


Question: Python baseline correction library I am currently working with some Raman Spectra data, and I am trying to correct my data caused by florescence skewing. Take a look at the graph below: I am pretty close to achieving what I want. As you can see, I am trying to fit a polynomial in all my data whereas I should really just be fitting a polynomial at the local minimas. Ideally I would want to have a polynomial fitting which when subtracted from my original data would result in something like this: Are there any built in libs that does this already? If not, any simple algorithm one can recommend for me?

Response:
I found an answer to my question, just sharing for everyone who stumbles upon this. There is an algorithm called "Asymmetric Least Squares Smoothing" by P. Eilers and H. Boelens in 2005. The paper is free and you can find it on google.
```
def baseline_als(y, lam, p, niter=10):
  L = len(y)
  D = sparse.csc_matrix(np.diff(np.eye(L), 2))
  w = np.ones(L)
  for i in xrange(niter):
    W = sparse.spdiags(w, 0, L, L)
    Z = W + lam * D.dot(D.transpose())
    z = spsolve(Z, w*y)
    w = p * (y > z) + (1-p) * (y < z)
  return z
```


Question:
numpy.max or max ? Which one is faster? In python, which one is faster ?
```
numpy.max(), numpy.min()
```
or
```
max(), min()
```
My list/array length varies from 2 to 600. Which one should I use to save some run time ?

Response: Well from my timings it follows if you already have numpy array a you should use a.max (the source tells it's the same as np.max if a.max available). But if you have built-in list then most of the time takes converting it into np.ndarray => that's why max is better in your timings. In essense: if np.ndarray then a.max, if list and no need for all the machinery of np.ndarray then standard max.


Question:
Python - byte image to NumPy array using OpenCV I have an image in bytes: print(image_bytes) b'\xff\xd8\xff\xfe\x00\x10Lavc57.64.101\x00\xff\xdb\x00C\x00\x08\x04\x04\x04\x04\x04\x05\x05\x05\x05\x05\x05\x06\x06\x06\x06\x06\x06\x06\x06\x06\x06\x06\x06\x06\x07\x07\x07\x08\x08\x08\x07\x07\x07\x06\x06\x07\x07\x08\x08\x08\x08\t\t\t\x08\x08\x08\x08\t\t\n\n\n\x0c\x0c\x0b\x0b\x0e\x0e\x0e\x11\x11\x14\xff\xc4\x01\xa2\x00\x00\x01\x05\x01\x01\x01\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x01\x02\x03\x04\x05\x06\x07\x08\t\n\x0b\x01\x00\x03\x01\x01\x01\x01\x01\x01\x01\x01\x01\x00\x00\ ... some other stuff I am able to convert it to a NumPy array using Pillow:
```
image = numpy.array(Image.open(io.BytesIO(image_bytes)))
```
But I don't really like using Pillow. Is there a way to use clear OpenCV, or directly NumPy even better, or some other faster library?

Response:
I created a 2x2 JPEG image to test this. The image has white, red, green and purple pixels. I used cv2.imdecode and numpy.frombuffer
```
import cv2
import numpy as np

f = open('image.jpg', 'rb')
image_bytes = f.read()  # b'\xff\xd8\xff\xe0\x00\x10...'

decoded = cv2.imdecode(np.frombuffer(image_bytes, np.uint8), -1)

print('OpenCV:\n', decoded)

# your Pillow code
import io
from PIL import Image
image = np.array(Image.open(io.BytesIO(image_bytes)))
print('PIL:\n', image)
```
This seems to work, although the channel order is BGR and not RGB as in PIL.Image. There are probably some flags you might use to tune this. Test results:
```
OpenCV:
 [[[255 254 255]
  [  0   0 254]]

 [[  1 255   0]
  [254   0 255]]]
PIL:
 [[[255 254 255]
  [254   0   0]]

 [[  0 255   1]
  [255   0 254]]]
```


Question:
How to convert an image from np.uint16 to np.uint8? I am creating an image so:
```
image = np.empty(shape=(height, width, 1), dtype = np.uint16)
```
After that I convert the image to BGR model:
```
image = cv2.cvtColor(image, cv2.COLOR_GRAY2BGR)
```
I'd like to convert the image now in a dtype = np.uint8 in order to use that image with cv2.threshold() function. I meant, I would like to convert the image to CV_8UC1.

Response:
You can use cv2.convertScaleAbs for this problem. See the Documentation. Check out the command terminal demo below :
```

```python
img = np.empty((100,100,1),dtype = np.uint16)
```

```python
image = cv2.cvtColor(img,cv2.COLOR_GRAY2BGR)
```

```python
cvuint8 = cv2.convertScaleAbs(image)
```

```python
cvuint8.dtype
#Output
#dtype('uint8')
#```
#Hope it helps!!!
```


Question:
Faster numpy cartesian to spherical coordinate conversion? I have an array of 3 million data points from a 3-axiz accellerometer (XYZ), and I want to add 3 columns to the array containing the equivalent spherical coordinates (r, theta, phi). The following code works, but seems way too slow. How can I do better?
```
import numpy as np
import math as m

def cart2sph(x,y,z):
    XsqPlusYsq = x**2 + y**2
    r = m.sqrt(XsqPlusYsq + z**2)               # r
    elev = m.atan2(z,m.sqrt(XsqPlusYsq))     # theta
    az = m.atan2(y,x)                           # phi
    return r, elev, az

def cart2sphA(pts):
    return np.array([cart2sph(x,y,z) for x,y,z in pts])

def appendSpherical(xyz):
    np.hstack((xyz, cart2sphA(xyz)))
```

Response:
This is similar to Justin Peel's answer, but using just numpy and taking advantage of its built-in vectorization:
```
import numpy as np

def appendSpherical_np(xyz):
    ptsnew = np.hstack((xyz, np.zeros(xyz.shape)))
    xy = xyz[:,0]**2 + xyz[:,1]**2
    ptsnew[:,3] = np.sqrt(xy + xyz[:,2]**2)
    ptsnew[:,4] = np.arctan2(np.sqrt(xy), xyz[:,2]) # for elevation angle defined from Z-axis down
    #ptsnew[:,4] = np.arctan2(xyz[:,2], np.sqrt(xy)) # for elevation angle defined from XY-plane up
    ptsnew[:,5] = np.arctan2(xyz[:,1], xyz[:,0])
    return ptsnew
```
Note that, as suggested in the comments, I've changed the definition of elevation angle from your original function. On my machine, testing with pts = np.random.rand(3000000, 3), the time went from 76 seconds to 3.3 seconds. I don't have Cython so I wasn't able to compare the timing with that solution.


Question:
How can I tell if NumPy creates a view or a copy? For a minimal working example, let's digitize a 2D array. numpy.digitize requires a 1D array:
```
import numpy as np
N = 200
A = np.random.random((N, N))
X = np.linspace(0, 1, 20)
print np.digitize(A.ravel(), X).reshape((N, N))
```
Now the documentation says: ... A copy is made only if needed. How do I know if the ravel copy it is "needed" in this case? In general - is there a way I can determine if a particular operation creates a copy or a view?

Response:
This question is very similar to a question that I asked a while back: You can check the base attribute.
```
a = np.arange(50)
b = a.reshape((5, 10))
print (b.base is a)
```
However, that's not perfect. You can also check to see if they share memory using np.may_share_memory.
```
print (np.may_share_memory(a, b))
```
There's also the flags attribute that you can check:
```
print (b.flags['OWNDATA'])  #False -- apparently this is a view
e = np.ravel(b[:, 2])
print (e.flags['OWNDATA'])  #True -- Apparently this is a new numpy object.
```
But this last one seems a little fishy to me, although I can't quite put my finger on why...


Question:
Output different precision by column with pandas.DataFrame.to_csv()? Question Is it possible to specify a float precision specifically for each column to be printed by the Python pandas package method pandas.DataFrame.to_csv? Background If I have a pandas dataframe that is arranged like this:
```

```python
df_data[:5]
#Output
#year  month  day       lats       lons  vals
#0   2012      6   16  81.862745 -29.834254   0.0
#1   2012      6   16  81.862745 -29.502762   0.1
#2   2012      6   16  81.862745 -29.171271   0.0
#3   2012      6   16  81.862745 -28.839779   0.2
#4   2012      6   16  81.862745 -28.508287   0.0
#```
#There is the float_format option that can be used to specify a precision, but this applys that precision to all columns of the dataframe when printed. When I use that like so:
#```
#df_data.to_csv(outfile, index=False,
#                   header=False, float_format='%11.6f')
#```
#I get the following, where vals is given an inaccurate precision:
#```
#2012,6,16,  81.862745, -29.834254,   0.000000
#2012,6,16,  81.862745, -29.502762,   0.100000
#2012,6,16,  81.862745, -29.171270,   0.000000
#2012,6,16,  81.862745, -28.839779,   0.200000
#2012,6,16,  81.862745, -28.508287,   0.000000
#```
```

Response:
Change the type of column "vals" prior to exporting the data frame to a CSV file
```
df_data['vals'] = df_data['vals'].map(lambda x: '%2.1f' % x)

df_data.to_csv(outfile, index=False, header=False, float_format='%11.6f')
```


Question: crop center portion of a numpy image Let's say I have a numpy image of some width x and height y. I have to crop the center portion of the image to width cropx and height cropy. Let's assume that cropx and cropy are positive non zero integers and less than the respective image size. What's the best way to apply the slicing for the output image?

Response:
Something along these lines -
```
def crop_center(img,cropx,cropy):
    y,x = img.shape
    startx = x//2-(cropx//2)
    starty = y//2-(cropy//2)
    return img[starty:starty+cropy,startx:startx+cropx]
```
Sample run -
```

```python
img
#Output
#array([[88, 93, 42, 25, 36, 14, 59, 46, 77, 13, 52, 58],
#       [43, 47, 40, 48, 23, 74, 12, 33, 58, 93, 87, 87],
#       [54, 75, 79, 21, 15, 44, 51, 68, 28, 94, 78, 48],
#       [57, 46, 14, 98, 43, 76, 86, 56, 86, 88, 96, 49],
#       [52, 83, 13, 18, 40, 33, 11, 87, 38, 74, 23, 88],
#       [81, 28, 86, 89, 16, 28, 66, 67, 80, 23, 95, 98],
#       [46, 30, 18, 31, 73, 15, 90, 77, 71, 57, 61, 78],
#       [33, 58, 20, 11, 80, 25, 96, 80, 27, 40, 66, 92],
#       [13, 59, 77, 53, 91, 16, 47, 79, 33, 78, 25, 66],
#       [22, 80, 40, 24, 17, 85, 20, 70, 81, 68, 50, 80]])
```

```python
crop_center(img,4,6)
#Output
#array([[15, 44, 51, 68],
#       [43, 76, 86, 56],
#       [40, 33, 11, 87],
#       [16, 28, 66, 67],
#       [73, 15, 90, 77],
#       [80, 25, 96, 80]])
#```
```


Question:
Select multiple ranges of columns in Pandas DataFrame I have to read several files some in Excel format and some in CSV format. Some of the files have hundreds of columns. Is there a way to select several ranges of columns without specifying all the column names or positions? For example something like selecting columns 1 -10, 15, 17 and 50-100:
```
df = df.ix[1:10, 15, 17, 50:100]
```
I need to know how to do this both when creating dataframe from Excel files and CSV files and after the data framers created.

Response:
use np.r_
```
np.r_[1:10, 15, 17, 50:100]

array([ 1,  2,  3,  4,  5,  6,  7,  8,  9, 15, 17, 50, 51, 52, 53, 54, 55,
       56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72,
       73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89,
       90, 91, 92, 93, 94, 95, 96, 97, 98, 99])
```
so you can do
```
df.iloc[:, np.r_[1:10, 15, 17, 50:100]]
```


Question: Difference between nonzero(a), where(a) and argwhere(a). When to use which? In Numpy, nonzero(a), where(a) and argwhere(a), with a being a numpy array, all seem to return the non-zero indices of the array. What are the differences between these three calls? On argwhere the documentation says: np.argwhere(a) is the same as np.transpose(np.nonzero(a)). Why have a whole function that just transposes the output of nonzero ? When would that be so useful that it deserves a separate function? What about the difference between where(a) and nonzero(a)? Wouldn't they return the exact same result?

Response:
nonzero and argwhere both give you information about where in the array the elements are True. where works the same as nonzero in the form you have posted, but it has a second form:
```
np.where(mask,a,b)
```
which can be roughly thought of as a numpy "ufunc" version of the conditional expression:
```
a[i] if mask[i] else b[i]
```
(with appropriate broadcasting of a and b). As far as having both nonzero and argwhere, they're conceptually different. nonzero is structured to return an object which can be used for indexing. This can be lighter-weight than creating an entire boolean mask if the 0's are sparse:
```
mask = a == 0  # entire array of bools
mask = np.nonzero(a)
```
Now you can use that mask to index other arrays, etc. However, as it is, it's not very nice conceptually to figure out which indices correspond to 0 elements. That's where argwhere comes in.


Question:
handling zeros in pandas DataFrames column divisions in Python What's the best way to handle zero denominators when dividing pandas DataFrame columns by each other in Python? for example:
```
df = pandas.DataFrame({"a": [1, 2, 0, 1, 5], "b": [0, 10, 20, 30, 50]})
df.a / df.b  # yields error
```
I'd like the ratios where the denominator is zero to be registered as NA (numpy.nan). How can this be done efficiently in pandas? Casting to float64 does not work at level of columns:
```

```python
df
#Output
#a   b
#0  1   0
#1  2  10
#2  0  20
#3  1  30
#4  5  50
```

```python
df["a"].astype("float64") / df["b"].astype("float64")
...

FloatingPointError: divide by zero encountered in divide
```
How can I do it just for particular columns and not entire df?
```

Response:
You need to work in floats, otherwise you will have integer division, prob not what you want
```

```python
df = pandas.DataFrame({"a": [1, 2, 0, 1, 5],
                                "b": [0, 10, 20, 30, 50]}).astype('float64')
```

```python
df
#Output
#a   b
#0  1   0
#1  2  10
#2  0  20
#3  1  30
#4  5  50
```

```python
df.dtypes
#Output
#a    float64
#b    float64
#dtype: object
#```
#Here's one way
#```
```

```python
x = df.a/df.b
```

```python
x
#Output
#0         inf
#1    0.200000
#2    0.000000
#3    0.033333
#4    0.100000
#dtype: float64
```

```python
x[np.isinf(x)] = np.nan
```

```python
x
#Output
#0         NaN
#1    0.200000
#2    0.000000
#3    0.033333
#4    0.100000
#dtype: float64
#```
#Here's another way
#```
```

```python
df.a/df.b.replace({ 0 : np.nan })
#Output
#0         NaN
#1    0.200000
#2    0.000000
#3    0.033333
#4    0.100000
#dtype: float64
#```
```


Question:
How to calculate the sum of all columns of a 2D numpy array (efficiently) Let's say I have the following 2D numpy array consisting of four rows and three columns:
```

```python
a = numpy.arange(12).reshape(4,3)
```

```python
print(a)
#Output
#[[ 0  1  2]
# [ 3  4  5]
# [ 6  7  8]
# [ 9 10 11]]
#```
#What would be an efficient way to generate a 1D array that contains the sum of all columns (like [18, 22, 26])? Can this be done without having the need to loop through all columns?
```

Response:
Check out the documentation for numpy.sum, paying particular attention to the axis parameter. To sum over columns:
```

```python
import numpy as np
```

```python
a = np.arange(12).reshape(4,3)
```

```python
a.sum(axis=0)
#Output
#array([18, 22, 26])
#```
#Or, to sum over rows:
#```
```

```python
a.sum(axis=1)
#Output
#array([ 3, 12, 21, 30])
#```
#Other aggregate functions, like numpy.mean, numpy.cumsum and numpy.std, e.g., also take the axis parameter. From the Tentative Numpy Tutorial: Many unary operations, such as computing the sum of all the elements in the array, are implemented as methods of the ndarray class. By default, these operations apply to the array as though it were a list of numbers, regardless of its shape. However, by specifying the axis parameter you can apply an operation along the specified axis of an array:
```


Question:
Efficient & pythonic check for singular matrix Working on some matrix algebra here. Sometimes I need to invert a matrix that may be singular or ill-conditioned. I understand it is pythonic to simply do this:
```
try:
    i = linalg.inv(x)
except LinAlgErr as err:
    #handle it
```
but am not sure how efficient that is. Wouldn't this be better?
```
if linalg.cond(x) < 1/sys.float_info.epsilon:
    i = linalg.inv(x)
else:
    #handle it
```
Does numpy.linalg simply perform up front the test I proscribed?

Response:
So based on the inputs here, I'm marking my original code block with the explicit test as the solution:
```
if linalg.cond(x) < 1/sys.float_info.epsilon:
    i = linalg.inv(x)
else:
    #handle it
```
Surprisingly, the numpy.linalg.inv function doesn't perform this test. I checked the code and found it goes through all it's machinations, then just calls the lapack routine - seems quite inefficient. Also, I would 2nd a point made by DaveP: that the inverse of a matrix should not be computed unless it's explicitly needed.


Question:
Convert list of tuples to structured numpy array I have a list of Num_tuples tuples that all have the same length Dim_tuple
```
xlist = [tuple_1, tuple_2, ..., tuple_Num_tuples]
```
For definiteness, let's say Num_tuples=3 and Dim_tuple=2
```
xlist = [(1, 1.1), (2, 1.2), (3, 1.3)]
```
I want to convert xlist into a structured numpy array xarr using a user-provided list of column names user_names and a user-provided list of variable types user_types
```
user_names = [name_1, name_2, ..., name_Dim_tuple]
user_types = [type_1, type_2, ..., type_Dim_tuple]
```
So in the creation of the numpy array,
```
dtype = [(name_1,type_1), (name_2,type_2), ..., (name_Dim_tuple, type_Dim_tuple)]
```
In the case of my toy example desired end product would look something like:
```
xarr['name1']=np.array([1,2,3])
xarr['name2']=np.array([1.1,1.2,1.3])
```
How can I slice xlist to create xarr without any loops?

Response:
A list of tuples is the correct way of providing data to a structured array:
```

```python
xlist = [(1, 1.1), (2, 1.2), (3, 1.3)]
```

```python
dt=np.dtype('int,float')
```

```python
np.array(xlist,dtype=dt)
#Output
#array([(1, 1.1), (2, 1.2), (3, 1.3)],
#      dtype=[('f0', '<i4'), ('f1', '<f8')])
```

```python
xarr = np.array(xlist,dtype=dt)
```

```python
xarr['f0']
#Output
#array([1, 2, 3])
```

```python
xarr['f1']
#Output
#array([ 1.1,  1.2,  1.3])
#```
#or if the names are important:
#```
```

```python
xarr.dtype.names=['name1','name2']
```

```python
xarr
#Output
#array([(1, 1.1), (2, 1.2), (3, 1.3)],
#      dtype=[('name1', '<i4'), ('name2', '<f8')])
#```
#http://docs.scipy.org/doc/numpy/user/basics.rec.html#filling-structured-arrays
```


Question:
Apply function on each row (row-wise) of a NumPy array So, I have the function -
```
def function(x):
    x , y = vector
    return exp(((-x**2/200))-0.5*(y+0.05*(x**2) - 100*0.05)**2)
```
and let's say that I would like to evaluate it at the following points (first column are the x-values and second column are the y-values) -
```
array([[-1.56113514,  4.51759732],
       [-2.80261623,  5.068371  ],
       [ 0.7792729 ,  6.0169462 ],
       [-1.35672858,  3.52517478],
       [-1.92074891,  5.79966161],
       [-2.79340321,  4.73430001],
       [-2.79655868,  5.05361163],
       [-2.13637747,  5.39255837],
       [ 0.17341809,  3.60918261],
       [-1.22712921,  4.95327158]])
```
i.e. I would like to pass the function the first row of values and evaluate, then the second row and evaluate etc. and then the final result would be an array of the values evaluated at these points (so, an array consisting of 10 values). So, for example, if the function was, say, a bivariate normal distribution -
```
def function2(x):

function2 = (mvnorm.pdf(x,[0,0],[[1,0],[0,1]]))

return function2
```
and I passed the above values into this function, I would get -
```
array([  1.17738907e-05,   1.08383957e-04,   1.69855078e-04,
         5.64757613e-06,   1.37432346e-05,   1.44032800e-04,
         1.33426313e-05,   1.97822328e-06,   6.56121709e-08,
         4.67076770e-05])
```
So basically, I am looking for a way to rewrite the function so that it can do this. Moreover, I would like to keep the function as a function of one variable only (i.e. only a function of x). Thank you for your help!

Response:
You can use np.apply_along_axis:
```
np.apply_along_axis(function, 1, array)
```
The first argument is the function, the second argument is the axis along which the function is to be applied. In your case, it is the first axis. The last argument is the array, of course. You should be warned, however, that apply_along_axis is only a convenience function, not a magic bullet. It has a severe speed limitation, since it just hides a loop. You should always try to vectorize your computation, where possible. Here's how I'd do this:
```
v = array[:, 0] ** 2   # computing just once
return np.exp((-v / 200) - 0.5 * (array[:, 1] + 0.05 * v - 5) ** 2)
```


Question:
How to specify upper and lower limits when using numpy.random.normal I want to be able to pick values from a normal distribution that only ever fall between 0 and 1. In some cases I want to be able to basically just return a completely random distribution, and in other cases I want to return values that fall in the shape of a gaussian. At the moment I am using the following function:
```
def blockedgauss(mu,sigma):
    while True:
        numb = random.gauss(mu,sigma)
        if (numb > 0 and numb < 1):
            break
    return numb
```
It picks a value from a normal distribution, then discards it if it falls outside of the range 0 to 1, but I feel like there must be a better way of doing this.

Response:
It sounds like you want a truncated normal distribution. Using scipy, you could use scipy.stats.truncnorm to generate random variates from such a distribution:
```
import matplotlib.pyplot as plt
import scipy.stats as stats

lower, upper = 3.5, 6
mu, sigma = 5, 0.7
X = stats.truncnorm(
    (lower - mu) / sigma, (upper - mu) / sigma, loc=mu, scale=sigma)
N = stats.norm(loc=mu, scale=sigma)

fig, ax = plt.subplots(2, sharex=True)
ax[0].hist(X.rvs(10000), normed=True)
ax[1].hist(N.rvs(10000), normed=True)
plt.show()
```
The top figure shows the truncated normal distribution, the lower figure shows the normal distribution with the same mean mu and standard deviation sigma.


Question: How to calculate rolling / moving average using python + NumPy / SciPy? There seems to be no function that simply calculates the moving average on numpy/scipy, leading to convoluted solutions. My question is two-fold: What's the easiest way to (correctly) implement a moving average with numpy? Since this seems non-trivial and error prone, is there a good reason not to have the batteries included in this case?

Response:
A simple way to achieve this is by using np.convolve. The idea behind this is to leverage the way the discrete convolution is computed and use it to return a rolling mean. This can be done by convolving with a sequence of np.ones of a length equal to the sliding window length we want. In order to do so we could define the following function:
```
def moving_average(x, w):
    return np.convolve(x, np.ones(w), 'valid') / w
```
This function will be taking the convolution of the sequence x and a sequence of ones of length w. Note that the chosen mode is valid so that the convolution product is only given for points where the sequences overlap completely. Some examples:
```
x = np.array([5,3,8,10,2,1,5,1,0,2])
```
For a moving average with a window of length 2 we would have:
```
moving_average(x, 2)
# array([4. , 5.5, 9. , 6. , 1.5, 3. , 3. , 0.5, 1. ])
```
And for a window of length 4:
```
moving_average(x, 4)
# array([6.5 , 5.75, 5.25, 4.5 , 2.25, 1.75, 2.  ])
```
How does convolve work? Lets have a more in depth look at the way the discrete convolution is being computed. The following function aims to replicate the way np.convolve is computing the output values:
```
def mov_avg(x, w):
    for m in range(len(x)-(w-1)):
        yield sum(np.ones(w) * x[m:m+w]) / w
```
Which, for the same example above would also yield:
```
list(mov_avg(x, 2))
# [4.0, 5.5, 9.0, 6.0, 1.5, 3.0, 3.0, 0.5, 1.0]
```
So what is being done at each step is to take the inner product between the array of ones and the current window. In this case the multiplication by np.ones(w) is superfluous given that we are directly taking the sum of the sequence. Bellow is an example of how the first outputs are computed so that it is a little clearer. Lets suppose we want a window of w=4:
```
[1,1,1,1]
[5,3,8,10,2,1,5,1,0,2]
= (1*5 + 1*3 + 1*8 + 1*10) / w = 6.5
```
And the following output would be computed as:
```
[1,1,1,1]
[5,3,8,10,2,1,5,1,0,2]
= (1*3 + 1*8 + 1*10 + 1*2) / w = 5.75
```
And so on, returning a moving average of the sequence once all overlaps have been performed.


Question: How can I serialize a numpy array while preserving matrix dimensions? numpy.array.tostring doesn't seem to preserve information about matrix dimensions (see this question), requiring the user to issue a call to numpy.array.reshape. Is there a way to serialize a numpy array to JSON format while preserving this information? Note: The arrays may contain ints, floats or bools. It's reasonable to expect a transposed array. Note 2: this is being done with the intent of passing the numpy array through a Storm topology using streamparse, in case such information ends up being relevant.

Response:
pickle.dumps or numpy.save encode all the information needed to reconstruct an arbitrary NumPy array, even in the presence of endianness issues, non-contiguous arrays, or weird structured dtypes. Endianness issues are probably the most important; you don't want array([1]) to suddenly become array([16777216]) because you loaded your array on a big-endian machine. pickle is probably the more convenient option, though save has its own benefits, given in the npy format rationale. I'm giving options for serializing to JSON or a bytestring, because the original questioner needed JSON-serializable output, but most people coming here probably don't. The pickle way:
```
import pickle
a = # some NumPy array

# Bytestring option
serialized = pickle.dumps(a)
deserialized_a = pickle.loads(serialized)

# JSON option
# latin-1 maps byte n to unicode code point n
serialized_as_json = json.dumps(pickle.dumps(a).decode('latin-1'))
deserialized_from_json = pickle.loads(json.loads(serialized_as_json).encode('latin-1'))
```
numpy.save uses a binary format, and it needs to write to a file, but you can get around that with io.BytesIO:
```
a = # any NumPy array
memfile = io.BytesIO()
numpy.save(memfile, a)

serialized = memfile.getvalue()
serialized_as_json = json.dumps(serialized.decode('latin-1'))
# latin-1 maps byte n to unicode code point n
```
And to deserialize:
```
memfile = io.BytesIO()

# If you're deserializing from a bytestring:
memfile.write(serialized)
# Or if you're deserializing from JSON:
# memfile.write(json.loads(serialized_as_json).encode('latin-1'))
memfile.seek(0)
a = numpy.load(memfile)
```


Question: How to determine if a number is any type of int (core or numpy, signed or not)? I need to test whether a variable is of type int, or any of np.int*, np.uint*, preferably using a single condition (i.e. no or). After some tests, I guess that: isinstance(n, int) will only match int and np.int32 (or np.int64 depending on plateform), np.issubdtype(type(n), int) seems to match all int and np.int*, but doesn’t match np.uint*. This leads to two questions: will np.issubdtype match any kind of signed ints? Can determine in a single check whether a number is any kind of signed or unsigned int? This is about testing for integers, the test should return False for float-likes.

Response:
NumPy provides base classes that you can/should use for subtype-checking, rather than the Python types. Use np.integer to check for any instance of either signed or unsigned integers. Use np.signedinteger and np.unsignedinteger to check for signed types or unsigned types.
```

```python
np.issubdtype(np.uint32, np.integer)
#Output
#True
```

```python
np.issubdtype(np.uint32, np.signedinteger)
#Output
#False
```

```python
np.issubdtype(int, np.integer)
#Output
#True
```

```python
np.issubdtype(np.array([1, 2, 3]).dtype, np.integer)
#Output
#True
#```
#All floating or complex number types will return False when tested. np.issubdtype(np.uint*, int) will always be False because the Python int is a signed type. A useful reference showing the relationship between all of these base classes is found in the documentation here.
```


Question:
Pythonic way to create a numpy array from a list of numpy arrays I generate a list of one dimensional numpy arrays in a loop and later convert this list to a 2d numpy array. I would've preallocated a 2d numpy array if i knew the number of items ahead of time, but I don't, therefore I put everything in a list. The mock up is below:
```

```python
list_of_arrays = map(lambda x: x*ones(2), range(5))
```

```python
list_of_arrays
#Output
#[array([ 0.,  0.]), array([ 1.,  1.]), array([ 2.,  2.]), array([ 3.,  3.]), array([ 4.,  4.])]
```

```python
arr = array(list_of_arrays)
```

```python
arr
#Output
#array([[ 0.,  0.],
#       [ 1.,  1.],
#       [ 2.,  2.],
#       [ 3.,  3.],
#       [ 4.,  4.]])
#```
#My question is the following: Is there a better way (performancewise) to go about the task of collecting sequential numerical data (in my case numpy arrays) than putting them in a list and then making a numpy.array out of it (I am creating a new obj and copying the data)? Is there an "expandable" matrix data structure available in a well tested module? A typical size of my 2d matrix would be between 100x10 and 5000x10 floats EDIT: In this example i'm using map, but in my actual application I have a for loop
```

Response:
Suppose you know that the final array arr will never be larger than 5000x10. Then you could pre-allocate an array of maximum size, populate it with data as you go through the loop, and then use arr.resize to cut it down to the discovered size after exiting the loop. The tests below suggest doing so will be slightly faster than constructing intermediate python lists no matter what the ultimate size of the array is. Also, arr.resize de-allocates the unused memory, so the final (though maybe not the intermediate) memory footprint is smaller than what is used by python_lists_to_array. This shows numpy_all_the_way is faster:
```
% python -mtimeit -s"import test" "test.numpy_all_the_way(100)"
100 loops, best of 3: 1.78 msec per loop
% python -mtimeit -s"import test" "test.numpy_all_the_way(1000)"
100 loops, best of 3: 18.1 msec per loop
% python -mtimeit -s"import test" "test.numpy_all_the_way(5000)"
10 loops, best of 3: 90.4 msec per loop

% python -mtimeit -s"import test" "test.python_lists_to_array(100)"
1000 loops, best of 3: 1.97 msec per loop
% python -mtimeit -s"import test" "test.python_lists_to_array(1000)"
10 loops, best of 3: 20.3 msec per loop
% python -mtimeit -s"import test" "test.python_lists_to_array(5000)"
10 loops, best of 3: 101 msec per loop
```
This shows numpy_all_the_way uses less memory:
```
% test.py
Initial memory usage: 19788
After python_lists_to_array: 20976
After numpy_all_the_way: 20348
```
test.py:
```
import numpy as np
import os

def memory_usage():
    pid = os.getpid()
    return next(line for line in open('/proc/%s/status' % pid).read().splitlines()
                if line.startswith('VmSize')).split()[-2]

N, M = 5000, 10

def python_lists_to_array(k):
    list_of_arrays = list(map(lambda x: x * np.ones(M), range(k)))
    arr = np.array(list_of_arrays)
    return arr

def numpy_all_the_way(k):
    arr = np.empty((N, M))
    for x in range(k):
        arr[x] = x * np.ones(M)
    arr.resize((k, M))
    return arr

if __name__ == '__main__':
    print('Initial memory usage: %s' % memory_usage())
    arr = python_lists_to_array(5000)
    print('After python_lists_to_array: %s' % memory_usage())
    arr = numpy_all_the_way(5000)
    print('After numpy_all_the_way: %s' % memory_usage())
```


Question:
Sorting a 2D numpy array by multiple axes I have a 2D numpy array of shape (N,2) which is holding N points (x and y coordinates). For example:
```
array([[3, 2],
       [6, 2],
       [3, 6],
       [3, 4],
       [5, 3]])
```
I'd like to sort it such that my points are ordered by x-coordinate, and then by y in cases where the x coordinate is the same. So the array above should look like this:
```
array([[3, 2],
       [3, 4],
       [3, 6],
       [5, 3],
       [6, 2]])
```
If this was a normal Python list, I would simply define a comparator to do what I want, but as far as I can tell, numpy's sort function doesn't accept user-defined comparators. Any ideas? EDIT: Thanks for the ideas! I set up a quick test case with 1000000 random integer points, and benchmarked the ones that I could run (sorry, can't upgrade numpy at the moment).
```
Mine:   4.078 secs
mtrw:   7.046 secs
unutbu: 0.453 secs
```

Response:
Using lexsort:
```
import numpy as np
a = np.array([(3, 2), (6, 2), (3, 6), (3, 4), (5, 3)])

ind = np.lexsort((a[:,1],a[:,0]))

a[ind]
# array([[3, 2],
#       [3, 4],
#       [3, 6],
#       [5, 3],
#       [6, 2]])
```
a.ravel() returns a view if a is C_CONTIGUOUS. If that is true, @ars's method, slightly modifed by using ravel instead of flatten, yields a nice way to sort a in-place:
```
a = np.array([(3, 2), (6, 2), (3, 6), (3, 4), (5, 3)])
dt = [('col1', a.dtype),('col2', a.dtype)]
assert a.flags['C_CONTIGUOUS']
b = a.ravel().view(dt)
b.sort(order=['col1','col2'])
```
Since b is a view of a, sorting b sorts a as well:
```
print(a)
# [[3 2]
#  [3 4]
#  [3 6]
#  [5 3]
#  [6 2]]
```


Question:
Difference between various numpy random functions The numpy.random module defines the following 4 functions that all seem to return a float betweeb [0, 1.0) from the continuous uniform distribution. What (if any) is the difference between these functions? random_sample([size]) Return random floats in the half-open interval [0.0, 1.0). random([size]) Return random floats in the half-open interval [0.0, 1.0). ranf([size]) Return random floats in the half-open interval [0.0, 1.0). sample([size]) Return random floats in the half-open interval [0.0, 1.0). --------------------------- Edit Follows --------------------------------------- I found the following in numpy.random source code that supports @askewchan's answer:
```
# Some aliases:
ranf = random = sample = random_sample
__all__.extend(['ranf','random','sample'])
```

Response:
Nothing. They're just aliases to random_sample:
```

```python
np.random.random
#Output
#<function random_sample>
```

```python
np.random.ranf
#Output
#<function random_sample>
```

```python
np.random.sample
#Output
#<function random_sample>
```

```python
np.random.random_sample is np.random.random
#Output
#True
```

```python
np.random.random_sample is np.random.ranf
#Output
#True
```

```python
np.random.random_sample is np.random.sample
#Output
#True
#```
```


Question:
Saving dictionary of numpy arrays So I have a DB with a couple of years worth of site data. I am now attempting to use that data for analytics - plotting and sorting of advertising costs by keyword, etc. One of the data grabs from the DB takes minutes to complete. While I could spend some time optimizing the SQL statements I use to get the data I'd prefer to simply leave that class and it's SQL alone, grab the data, and save the results to a data file for faster retrieval later. Most of this DB data isn't going to change so I could write a separate python script to update the file every 24 hours and then use that file for this long running task. The data is being returned as a dictionary of numpy arrays. When I use numpy.save('data', data) the file is saved just fine. When I use data2 = numpy.load('data.npy') it loads the file without error. However, the output data2 doesn't not equal the original data. Specifically the line data == data2 returns false. Additionally, if I use the following:
```
for key, key_data in data.items():
  print key
```
it works. But when I replace data.items() with data2.items() then I get an error:
```
AttributeError: 'numpy.ndarray' object has no attribute 'items'
```
Using type(data) I get dict. Using type(data2) I get numpy.ndarray. So how do I fix this? I want the loaded data to equal the data I passed in for saving. Is there an argument to numpy.save to fix this or do I need some form of simple reformatting function to reformat the loaded data into the proper structure? Attempts to get into the ndarray via for loops or indexing all lead to errors about indexing a 0-d array. Casting like this dict(data2) also fails for iterating over a 0-d array. However, Spyder shows value of the array and it includes the data I saved. I just can't figure out how to get to it. If I need to reformat the loaded data I'd appreciate some example code on how to do this.

Response:
Let's look at a small example:
```

```python
N
#Output
#array([[  0.,   1.,   2.,   3.],
#       [  4.,   5.,   6.,   7.],
#       [  8.,   9.,  10.,  11.]])
```

```python
data={'N':N}
```

```python
np.save('temp.npy',data)
```

```python
data2=np.load('temp.npy')
```

```python
data2
#Output
#array({'N': array([[  0.,   1.,   2.,   3.],
#       [  4.,   5.,   6.,   7.],
#       [  8.,   9.,  10.,  11.]])}, dtype=object)
#```
#np.save is designed to save numpy arrays. data is a dictionary. So it wrapped it in a object array, and used pickle to save that object. Your data2 probably has the same character. You get at the array with:
#```
```

```python
data2[()]['N']
#Output
#array([[  0.,   1.,   2.,   3.],
#       [  4.,   5.,   6.,   7.],
#       [  8.,   9.,  10.,  11.]])
#```
```


Question:
Confusion between numpy, scipy, matplotlib and pylab Numpy, scipy, matplotlib, and pylab are common terms among they who use python for scientific computation. I just learn a bit about pylab, and I got confused. Whenever I want to import numpy, I can always do:
```
import numpy as np
```
I just consider, that once I do
```
from pylab import *
```
the numpy will be imported as well (with np alias). So basically the second one does more things compared to the first one. There are few things I want to ask: Is it right that pylab is just a wrapper for numpy, scipy and matplotlib? As np is the numpy alias in pylab, what is the scipy and matplotlib alias in pylab? (as far as I know, plt is alias of matplotlib.pyplot, but I don't know the alias for the matplotlib itself)

Response: No, pylab is part of matplotlib (in matplotlib.pylab) and tries to give you a MatLab like environment. matplotlib has a number of dependencies, among them numpy which it imports under the common alias np. scipy is not a dependency of matplotlib. If you run ipython --pylab an automatic import will put all symbols from matplotlib.pylab into global scope. Like you wrote numpy gets imported under the np alias. Symbols from matplotlib are available under the mpl alias.


Question: How do you use the ellipsis slicing syntax in Python? This came up in Hidden features of Python, but I can't see good documentation or examples that explain how the feature works.

Response:
The ellipsis is used in numpy to slice higher-dimensional data structures. It's designed to mean at this point, insert as many full slices (:) to extend the multi-dimensional slice to all dimensions. Example:
```

```python
from numpy import arange
```

```python
a = arange(16).reshape(2,2,2,2)
#Output
#```
#Now, you have a 4-dimensional matrix of order 2x2x2x2. To select all first elements in the 4th dimension, you can use the ellipsis notation
#```
```

```python
a[..., 0].flatten()
#Output
#array([ 0,  2,  4,  6,  8, 10, 12, 14])
#```
#which is equivalent to
#```
```

```python
a[:,:,:,0].flatten()
#Output
#array([ 0,  2,  4,  6,  8, 10, 12, 14])
#```
#In your own implementations, you're free to ignore the contract mentioned above and use it for whatever you see fit.
```


Question:
Finding the nearest value and return the index of array in Python I found this post: Python: finding an element in an array and it's about returning the index of an array through matching the values. On the other hand, what I am thinking of doing is similar but different. I would like to find the nearest value for the target value. For example I am looking for 4.2 but I know in the array there is no 4.2 but I want to return the index of the value 4.1 instead of 4.4. What would be the fastest way of doing it? I am thinking of doing it the old way like how I used to do it with Matlab, which is using the array A where I want to get the index from to minus the target value and take the absolute of it, then select the min. Something like this:-
```
[~,idx] = min(abs(A - target))
```
That is Matlab code but I am newbie in Python so I am thinking, is there a fast way of doing it in Python? Thank you so much for your help!

Response:
This is similar to using bisect_left, but it'll allow you to pass in an array of targets
```
def find_closest(A, target):
    #A must be sorted
    idx = A.searchsorted(target)
    idx = np.clip(idx, 1, len(A)-1)
    left = A[idx-1]
    right = A[idx]
    idx -= target - left < right - target
    return idx
```
Some explanation: First the general case: idx = A.searchsorted(target) returns an index for each target such that target is between A[index - 1] and A[index]. I call these left and right so we know that left = 0 so target - left < right - target is True unless target == left == right and idx - True = 0. There is another special case if target is greater than all the elements of A, In that case idx = A.searchsorted(target) and np.clip(idx, 1, len(A)-1) replaces len(A) with len(A) - 1 so idx=len(A) -1 and target - left < right - target ends up False so idx returns len(A) -1. I'll let you work though the logic on your own. For example:
```

```python
A = np.arange(0, 20.)
```

```python
target = np.array([-2, 100., 2., 2.4, 2.5, 2.6])
```

```python
find_closest(A, target)
#Output
#array([ 0, 19,  2,  2,  3,  3])
#```
```


Question:
Is there a multi-dimensional version of arange/linspace in numpy? I would like a list of 2d NumPy arrays (x,y) , where each x is in {-5, -4.5, -4, -3.5, ..., 3.5, 4, 4.5, 5} and the same for y. I could do
```
x = np.arange(-5, 5.1, 0.5)
y = np.arange(-5, 5.1, 0.5)
```
and then iterate through all possible pairs, but I'm sure there's a nicer way... I would like something back that looks like:
```
[[-5, -5],
 [-5, -4.5],
 [-5, -4],
 ...
 [5, 5]]
```
but the order does not matter.

Response:
You can use np.mgrid for this, it's often more convenient than np.meshgrid because it creates the arrays in one step:
```
import numpy as np
X,Y = np.mgrid[-5:5.1:0.5, -5:5.1:0.5]
```
For linspace-like functionality, replace the step (i.e. 0.5) with a complex number whose magnitude specifies the number of points you want in the series. Using this syntax, the same arrays as above are specified as:
```
X, Y = np.mgrid[-5:5:21j, -5:5:21j]
```
You can then create your pairs as:
```
xy = np.vstack((X.flatten(), Y.flatten())).T
```
As @ali_m suggested, this can all be done in one line:
```
xy = np.mgrid[-5:5.1:0.5, -5:5.1:0.5].reshape(2,-1).T
```
Best of luck!


Question:
How convert a list of tuples to a numpy array of tuples? I have a list like this:
```
l=[(1,2),(3,4)]
```
I want to convert it to a numpy array,and keep array item type as tuple:
```
array([(1,2),(3,4)])
```
but numpy.array(l) will give:
```
array([[1,2],[3,4)]])
```
and item type has been changed from tuple to numpy.ndarray,then I specified item types
```
numpy.array(l,numpy.dtype('float,float'))
```
this gives:
```
array([(1,2),(3,4)])
```
but item type isn't tuple but numpy.void,so question is:
```
how to convert it to a numpy.array of tuple,not of numpy.void?
```

Response:
You can have an array of object dtype, letting each element of the array being a tuple, like so -
```
out = np.empty(len(l), dtype=object)
out[:] = l
```
Sample run -
```

```python
l = [(1,2),(3,4)]
```

```python
out = np.empty(len(l), dtype=object)
```

```python
out[:] = l
```

```python
out
#Output
#array([(1, 2), (3, 4)], dtype=object)
```

```python
out[0]
#Output
#(1, 2)
```

```python
type(out[0])
#Output
#tuple
#```
```


Question:
Flattening a list of NumPy arrays? It appears that I have data in the format of a list of NumPy arrays (type() = np.ndarray):
```
[array([[ 0.00353654]]), array([[ 0.00353654]]), array([[ 0.00353654]]),
array([[ 0.00353654]]), array([[ 0.00353654]]), array([[ 0.00353654]]),
array([[ 0.00353654]]), array([[ 0.00353654]]), array([[ 0.00353654]]),
array([[ 0.00353654]]), array([[ 0.00353654]]), array([[ 0.00353654]]),
array([[ 0.00353654]])]
```
I am trying to put this into a polyfit function:
```
m1 = np.polyfit(x, y, deg=2)
```
However, it returns the error: TypeError: expected 1D vector for x I assume I need to flatten my data into something like:
```
[0.00353654, 0.00353654, 0.00353654, 0.00353654, 0.00353654, 0.00353654 ...]
```
I have tried a list comprehension which usually works on lists of lists, but this as expected has not worked:
```
[val for sublist in risks for val in sublist]
```
What would be the best way to do this?

Response:
You could use numpy.concatenate, which as the name suggests, basically concatenates all the elements of such an input list into a single NumPy array, like so -
```
import numpy as np
out = np.concatenate(input_list).ravel()
```
If you wish the final output to be a list, you can extend the solution, like so -
```
out = np.concatenate(input_list).ravel().tolist()
```
Sample run -
```

```python
input_list
#Output
#[array([[ 0.00353654]]),
# array([[ 0.00353654]]),
# array([[ 0.00353654]]),
# array([[ 0.00353654]]),
# array([[ 0.00353654]]),
# array([[ 0.00353654]]),
# array([[ 0.00353654]]),
# array([[ 0.00353654]]),
# array([[ 0.00353654]]),
# array([[ 0.00353654]]),
# array([[ 0.00353654]]),
# array([[ 0.00353654]]),
# array([[ 0.00353654]])]
```

```python
np.concatenate(input_list).ravel()
#Output
#array([ 0.00353654,  0.00353654,  0.00353654,  0.00353654,  0.00353654,
#        0.00353654,  0.00353654,  0.00353654,  0.00353654,  0.00353654,
#        0.00353654,  0.00353654,  0.00353654])
#```
#Convert to list -
#```
```

```python
np.concatenate(input_list).ravel().tolist()
#Output
#[0.00353654,
# 0.00353654,
# 0.00353654,
# 0.00353654,
# 0.00353654,
# 0.00353654,
# 0.00353654,
# 0.00353654,
# 0.00353654,
# 0.00353654,
# 0.00353654,
# 0.00353654,
# 0.00353654]
#```
```


Question:
numerically stable way to multiply log probability matrices in numpy I need to take the matrix product of two NumPy matrices (or other 2d arrays) containing log probabilities. The naive way np.log(np.dot(np.exp(a), np.exp(b))) is not preferred for obvious reasons. Using
```
from scipy.misc import logsumexp
res = np.zeros((a.shape[0], b.shape[1]))
for n in range(b.shape[1]):
    # broadcast b[:,n] over rows of a, sum columns
    res[:, n] = logsumexp(a + b[:, n].T, axis=1)
```
works but runs about 100 times slower than np.log(np.dot(np.exp(a), np.exp(b))) Using
```
logsumexp((tile(a, (b.shape[1],1)) + repeat(b.T, a.shape[0], axis=0)).reshape(b.shape[1],a.shape[0],a.shape[1]), 2).T
```
or other combinations of tile and reshape also work but run even slower than the loop above due to the prohibitively large amounts of memory required for realistically sized input matrices. I am currently considering writing a NumPy extension in C to compute this, but of course I'd rather avoid that. Is there an established way to do this, or does anybody know of a less memory intensive way of performing this computation? EDIT: Thanks to larsmans for this solution (see below for derivation):
```
def logdot(a, b):
    max_a, max_b = np.max(a), np.max(b)
    exp_a, exp_b = a - max_a, b - max_b
    np.exp(exp_a, out=exp_a)
    np.exp(exp_b, out=exp_b)
    c = np.dot(exp_a, exp_b)
    np.log(c, out=c)
    c += max_a + max_b
    return c
```
A quick comparison of this method to the method posted above (logdot_old) using iPython's magic %timeit function yields the following:
```
In  [1] a = np.log(np.random.rand(1000,2000))

In  [2] b = np.log(np.random.rand(2000,1500))

In  [3] x = logdot(a, b)

In  [4] y = logdot_old(a, b) # this takes a while

In  [5] np.any(np.abs(x-y) > 1e-14)
Out [5] False

In  [6] %timeit logdot_old(a, b)
1 loops, best of 3: 1min 18s per loop

In  [6] %timeit logdot(a, b)
1 loops, best of 3: 264 ms per loop
```
Obviously larsmans' method obliterates mine!

Response:
logsumexp works by evaluating the right-hand side of the equation
```
log(∑ exp[a]) = max(a) + log(∑ exp[a - max(a)])
```
I.e., it pulls out the max before starting to sum, to prevent overflow in exp. The same can be applied before doing vector dot products:
```
log(exp[a] ⋅ exp[b])
 = log(∑ exp[a] × exp[b])
 = log(∑ exp[a + b])
 = max(a + b) + log(∑ exp[a + b - max(a + b)])     { this is logsumexp(a + b) }
```
but by taking a different turn in the derivation, we obtain
```
log(∑ exp[a] × exp[b])
 = max(a) + max(b) + log(∑ exp[a - max(a)] × exp[b - max(b)])
 = max(a) + max(b) + log(exp[a - max(a)] ⋅ exp[b - max(b)])
```
The final form has a vector dot product in its innards. It also extends readily to matrix multiplication, so we get the algorithm
```
def logdotexp(A, B):
    max_A = np.max(A)
    max_B = np.max(B)
    C = np.dot(np.exp(A - max_A), np.exp(B - max_B))
    np.log(C, out=C)
    C += max_A + max_B
    return C
```
This creates two A-sized temporaries and two B-sized ones, but one of each can be eliminated by
```
exp_A = A - max_A
np.exp(exp_A, out=exp_A)
```
and similarly for B. (If the input matrices may be modified by the function, all the temporaries can be eliminated.)


Question:
Creating a Pandas DataFrame from a Numpy array: How do I specify the index column and column headers? I have a Numpy array consisting of a list of lists, representing a two-dimensional array with row labels and column names as shown below:
```py
data = np.array([['','Col1','Col2'],['Row1',1,2],['Row2',3,4]])
```
I'd like the resulting DataFrame to have Row1 and Row2 as index values, and Col1, Col2 as header values. I can specify the index as follows:
```py
df = pd.DataFrame(data, index=data[:,0])
```
However, I am unsure how to best assign column headers.

Response:
Specify data, index and columns to the DataFrame constructor, as follows:
```

```python
pd.DataFrame(data=data[1:,1:],    # values
             index=data[1:,0],    # 1st column as index
             columns=data[0,1:])  # 1st row as the column names
#Output
#```
#As @joris mentions, you may need to change above to np.int_(data[1:,1:]) to have the correct data type.
```


Question:
Unpacking tuples/arrays/lists as indices for Numpy Arrays I would love to be able to do
```

```python
A = numpy.array(((1,2),(3,4)))
```

```python
idx = (0,0)
```

```python
A[*idx]
#Output
#```
#and get
#```
#1
#```
#however this is not valid syntax. Is there a way of doing this without explicitly writing out
#```
```

```python
A[idx[0], idx[1]]
#Output
#```
#? EDIT: Thanks for the replies. In my program I was indexing with a Numpy array rather than a tuple and getting strange results. Converting to a tuple as Alok suggests does the trick.
```

Response:
It's easier than you think:
```

```python
import numpy
```

```python
A = numpy.array(((1,2),(3,4)))
```

```python
idx = (0,0)
```

```python
A[idx]
#Output
#1
#```
```


Question:
What does the c underscore expression `c_` do exactly? It seems to be some kind of horizontal concatenation, but I could not find any documentation online. Here a minimal working example:
```

```python
from numpy import c_
```

```python
a = ones(4)
```

```python
b = zeros((4,10))
```

```python
c_[a,b]
#Output
#array([[ 1.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.],
#       [ 1.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.],
#       [ 1.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.],
#       [ 1.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.]])
#```
```

Response:
It took me a lot of time to understand but it seems I finally got it. All you have to do is add along second axis. let's take :
```
np.c_[np.array([1,2,3]), np.array([4,5,6])]
```
But there isn't second axis. So we mentally add one. so shape of both array becomes (3,1). So resultant shape would be (3,1+1) which is (3,2). which is the shape of result -
```
array([[1, 4],
       [2, 5],
       [3, 6]])
```
Another ex.:
```
np.c_[np.array([[1,2,3]]), 0, 0, np.array([[4,5,6]])]
```
shapes: np.array([[1,2,3]]) = 1,3 np.array([[4,5,6]]) = 1,3 0 so we can think of it as [[0]] = 1,1 So result 1,3+1+1+3 = 1,8 which is the shape of result : array([[1, 2, 3, 0, 0, 4, 5, 6]])


Question:
python numpy euclidean distance calculation between matrices of row vectors I am new to Numpy and I would like to ask you how to calculate euclidean distance between points stored in a vector. Let's assume that we have a numpy.array each row is a vector and a single numpy.array. I would like to know if it is possible to calculate the euclidean distance between all the points and this single point and store them in one numpy.array. Here is an interface:
```
points #2d list of row-vectors
singlePoint #one row-vector

listOfDistances= procedure( points,singlePoint)
```
Can we have something like this? Or is it possible to have one command to have the single point as a list of other points and at the end we get a matrix of distances? Thanks

Response:
While you can use vectorize, @Karl's approach will be rather slow with numpy arrays. The easier approach is to just do np.hypot(*(points - single_point).T). (The transpose assumes that points is a Nx2 array, rather than a 2xN. If it's 2xN, you don't need the .T. However this is a bit unreadable, so you write it out more explictly like this (using some canned example data...):
```
import numpy as np
single_point = [3, 4]
points = np.arange(20).reshape((10,2))

dist = (points - single_point)**2
dist = np.sum(dist, axis=1)
dist = np.sqrt(dist)
```


Question:
Find the linearly independent rows of a matrix How do I identify the linearly independent rows of a matrix? For instance,
```
[[0 1 0 0]
 [0 0 1 0]
 [0 1 1 0]
 [1 0 0 1]]
```
The 4th row is independent.

Response:
First, your 3rd row is linearly dependent with 1t and 2nd row. However, your 1st and 4th column are linearly dependent. Two methods you could use: Eigenvalue If one eigenvalue of the matrix is zero, its corresponding eigenvector is linearly dependent. The documentation eig states the returned eigenvalues are repeated according to their multiplicity and not necessarily ordered. However, assuming the eigenvalues correspond to your row vectors, one method would be:
```
import numpy as np

matrix = np.array(
    [
        [0, 1 ,0 ,0],
        [0, 0, 1, 0],
        [0, 1, 1, 0],
        [1, 0, 0, 1]
    ])

lambdas, V =  np.linalg.eig(matrix.T)
# The linearly dependent row vectors
print matrix[lambdas == 0,:]
```
Cauchy-Schwarz inequality To test linear dependence of vectors and figure out which ones, you could use the Cauchy-Schwarz inequality. Basically, if the inner product of the vectors is equal to the product of the norm of the vectors, the vectors are linearly dependent. Here is an example for the columns:
```
import numpy as np

matrix = np.array(
    [
        [0, 1 ,0 ,0],
        [0, 0, 1, 0],
        [0, 1, 1, 0],
        [1, 0, 0, 1]
    ])

print np.linalg.det(matrix)

for i in range(matrix.shape[0]):
    for j in range(matrix.shape[0]):
        if i != j:
            inner_product = np.inner(
                matrix[:,i],
                matrix[:,j]
            )
            norm_i = np.linalg.norm(matrix[:,i])
            norm_j = np.linalg.norm(matrix[:,j])

            print 'I: ', matrix[:,i]
            print 'J: ', matrix[:,j]
            print 'Prod: ', inner_product
            print 'Norm i: ', norm_i
            print 'Norm j: ', norm_j
            if np.abs(inner_product - norm_j * norm_i) < 1E-5:
                print 'Dependent'
            else:
                print 'Independent'
```
To test the rows is a similar approach. Then you could extend this to test all combinations of vectors, but I imagine this solution scale badly with size.


Question:
What's the difference of numpy.ndarray.T and numpy.ndarray.transpose() when self.ndim < 2 The document numpy.ndarray.T says ndarray.T — Same as self.transpose(), except that self is returned if self.ndim < 2. Also, ndarray.transpose(*axes) says For a 1-D array, this has no effect. Doesn't this mean the same thing? Here's a little demo snippet:
```

```python
import numpy as np
```

```python
print np.__version__
#Output
#1.5.1rc1
```

```python
a = np.arange(7)
```

```python
print a, a.T, a.transpose()
#Output
#[0 1 2 3 4 5 6] [0 1 2 3 4 5 6] [0 1 2 3 4 5 6]
#```
```

Response: Regardless of rank, the .T attribute and the .transpose() method are the same—they both return the transpose of the array. In the case of a rank 1 array, the .T and .transpose() don't do anything—they both return the array.


Question: how to return numpy.array from boost::python? I would like to return some data from c++ code as a numpy.array object. I had a look at boost::python::numeric, but its documentation is very terse. Can I get an example of e.g. returning a (not very large) vector to python? I don't mind doing copies of data.

Response:
UPDATE: the library described in my original answer (https://github.com/ndarray/Boost.NumPy) has been integrated directly into Boost.Python as of Boost 1.63, and hence the standalone version is now deprecated. The text below now corresponds to the new, integrated version (only the namespace has changed). Boost.Python now includes a moderately complete wrapper of the NumPy C-API into a Boost.Python interface. It's pretty low-level, and mostly focused on how to address the more difficult problem of how to pass C++ data to and from NumPy without copying, but here's how you'd do a copied std::vector return with that:
```
#include "boost/python/numpy.hpp"

namespace bp = boost::python;
namespace bn = boost::python::numpy;

std::vector<double> myfunc(...);

bn::ndarray mywrapper(...) {
    std::vector<double> v = myfunc(...);
    Py_intptr_t shape[1] = { v.size() };
    bn::ndarray result = bn::zeros(1, shape, bn::dtype::get_builtin<double>());
    std::copy(v.begin(), v.end(), reinterpret_cast<double*>(result.get_data()));
    return result;
}

BOOST_PYTHON_MODULE(example) {
    bn::initialize();
    bp::def("myfunc", mywrapper);
}
```


Question: How to transform numpy.matrix or array to scipy sparse matrix For SciPy sparse matrix, one can use todense() or toarray() to transform to NumPy matrix or array. What are the functions to do the inverse? I searched, but got no idea what keywords should be the right hit.

Response:
You can pass a numpy array or matrix as an argument when initializing a sparse matrix. For a CSR matrix, for example, you can do the following.
```

```python
import numpy as np
```

```python
from scipy import sparse
```

```python
A = np.array([[1,2,0],[0,0,3],[1,0,4]])
```

```python
B = np.matrix([[1,2,0],[0,0,3],[1,0,4]])
```

```python
A
#Output
#array([[1, 2, 0],
#       [0, 0, 3],
#       [1, 0, 4]])
```

```python
sA = sparse.csr_matrix(A)   # Here's the initialization of the sparse matrix.
```

```python
sB = sparse.csr_matrix(B)
```

```python
sA
#Output
#<3x3 sparse matrix of type '<type 'numpy.int32'>'
#        with 5 stored elements in Compressed Sparse Row format>
```

```python
print sA
#Output
#  (0, 0)        1
#  (0, 1)        2
#  (1, 2)        3
#  (2, 0)        1
#  (2, 2)        4
#```
```


Question:
Unsuccessful append to an empty NumPy array I am trying to fill an empty(not np.empty!) array with values using append but I am gettin error: My code is as follows:
```
import numpy as np
result=np.asarray([np.asarray([]),np.asarray([])])
result[0]=np.append([result[0]],[1,2])
```
And I am getting:
```
ValueError: could not broadcast input array from shape (2) into shape (0)
```

Response:
I might understand the question incorrectly, but if you want to declare an array of a certain shape but with nothing inside, the following might be helpful: Initialise empty array:
```

```python
a = np.zeros((0,3)) #or np.empty((0,3)) or np.array([]).reshape(0,3)
```

```python
a
#Output
#array([], shape=(0, 3), dtype=float64)
#```
#Now you can use this array to append rows of similar shape to it. Remember that a numpy array is immutable, so a new array is created for each iteration:
#```
```

```python
for i in range(3):
    a = np.vstack([a, [i,i,i]])
#Output
#...
```

```python
a
#Output
#array([[ 0.,  0.,  0.],
#       [ 1.,  1.,  1.],
#       [ 2.,  2.,  2.]])
#```
#np.vstack and np.hstack is the most common method for combining numpy arrays, but coming from Matlab I prefer np.r_ and np.c_: Concatenate 1d:
#```
```

```python
a = np.zeros(0)
```

```python
for i in range(3):
    a = np.r_[a, [i, i, i]]
#Output
#...
```

```python
a
#Output
#array([ 0.,  0.,  0.,  1.,  1.,  1.,  2.,  2.,  2.])
#```
#Concatenate rows:
#```
```

```python
a = np.zeros((0,3))
```

```python
for i in range(3):
    a = np.r_[a, [[i,i,i]]]
#Output
#...
```

```python
a
#Output
#array([[ 0.,  0.,  0.],
#       [ 1.,  1.,  1.],
#       [ 2.,  2.,  2.]])
#```
#Concatenate columns:
#```
```

```python
a = np.zeros((3,0))
```

```python
for i in range(3):
    a = np.c_[a, [[i],[i],[i]]]
#Output
#...
```

```python
a
#Output
#array([[ 0.,  1.,  2.],
#       [ 0.,  1.,  2.],
#       [ 0.,  1.,  2.]])
#```
```


Question:
module 'pandas' has no attribute 'read_csv
```
import pandas as pd

df = pd.read_csv('FBI-CRIME11.csv')

print(df.head())
```
Running this simple code gives me the error:
```
Traceback (most recent call last):
  File "C:/Users/Dita/Desktop/python/lessons/python.data/csv.py", line 1, in <module>
    import pandas as pd
  File "C:\python\lib\site-packages\pandas-0.19.1-py3.5-win-amd64.egg\pandas\__init__.py", line 37, in <module>
    import pandas.core.config_init
  File "C:\python\lib\site-packages\pandas-0.19.1-py3.5-win-amd64.egg\pandas\core\config_init.py", line 18, in <module>
    from pandas.formats.format import detect_console_encoding
  File "C:\python\lib\site-packages\pandas-0.19.1-py3.5-win-amd64.egg\pandas\formats\format.py", line 33, in <module>
    from pandas.io.common import _get_handle, UnicodeWriter, _expand_user
  File "C:\python\lib\site-packages\pandas-0.19.1-py3.5-win-amd64.egg\pandas\io\common.py", line 5, in <module>
    import csv
  File "C:\Users\Dita\Desktop\python\lessons\python.data\csv.py", line 4, in <module>
    df = pd.read_csv('FBI-CRIME11.csv')
AttributeError: module 'pandas' has no attribute 'read_csv'
```

Response: Try renaming your csv.py to something else, like csv_test.py. Looks like pandas is being confused about what to import.


Question:
Binning a column with pandas I have a data frame column with numeric values:
```
df['percentage'].head()
46.5
44.2
100.0
42.12
```
I want to see the column as bin counts:
```
bins = [0, 1, 5, 10, 25, 50, 100]
```
How can I get the result as bins with their value counts?
```
[0, 1] bin amount
[1, 5] etc
[5, 10] etc
...
```

Response:
You can use pandas.cut:
```
bins = [0, 1, 5, 10, 25, 50, 100]
df['binned'] = pd.cut(df['percentage'], bins)
print (df)
   percentage     binned
0       46.50   (25, 50]
1       44.20   (25, 50]
2      100.00  (50, 100]
3       42.12   (25, 50]
```
```
bins = [0, 1, 5, 10, 25, 50, 100]
labels = [1,2,3,4,5,6]
df['binned'] = pd.cut(df['percentage'], bins=bins, labels=labels)
print (df)
   percentage binned
0       46.50      5
1       44.20      5
2      100.00      6
3       42.12      5
```
Or numpy.searchsorted:
```
bins = [0, 1, 5, 10, 25, 50, 100]
df['binned'] = np.searchsorted(bins, df['percentage'].values)
print (df)
   percentage  binned
0       46.50       5
1       44.20       5
2      100.00       6
3       42.12       5
```
...and then value_counts or groupby and aggregate size:
```
s = pd.cut(df['percentage'], bins=bins).value_counts()
print (s)
(25, 50]     3
(50, 100]    1
(10, 25]     0
(5, 10]      0
(1, 5]       0
(0, 1]       0
Name: percentage, dtype: int64
```
```
s = df.groupby(pd.cut(df['percentage'], bins=bins)).size()
print (s)
percentage
(0, 1]       0
(1, 5]       0
(5, 10]      0
(10, 25]     0
(25, 50]     3
(50, 100]    1
dtype: int64
```
By default cut returns categorical. Series methods like Series.value_counts() will use all categories, even if some categories are not present in the data, operations in categorical.


Question: numpy vstack vs. column_stack What exactly is the difference between numpy vstack and column_stack. Reading through the documentation, it looks as if column_stack is an implementation of vstack for 1D arrays. Is it a more efficient implementation? Otherwise, I cannot find a reason for just having vstack.

Response:
I think the following code illustrates the difference nicely:
```

```python
np.vstack(([1,2,3],[4,5,6]))
#Output
#array([[1, 2, 3],
#       [4, 5, 6]])
```

```python
np.column_stack(([1,2,3],[4,5,6]))
#Output
#array([[1, 4],
#       [2, 5],
#       [3, 6]])
```

```python
np.hstack(([1,2,3],[4,5,6]))
#Output
#array([1, 2, 3, 4, 5, 6])
#```
#I've included hstack for comparison as well. Notice how column_stack stacks along the second dimension whereas vstack stacks along the first dimension. The equivalent to column_stack is the following hstack command:
#```
```

```python
np.hstack(([[1],[2],[3]],[[4],[5],[6]]))
#Output
#array([[1, 4],
#       [2, 5],
#       [3, 6]])
#```
#I hope we can agree that column_stack is more convenient.
```


Question:
Cannot get minor grid lines to appear in matplotlib figure Ok so I have the code below for real-time graphing some data from an embedded device received over serial. It's not meant to be a production tool but rather an internal eng tool, hence it's not terribly user-friendly. The problem is that no matter what I do, I can't get minor grid lines to appear, even though here they are set to True, which=both. I can do anything I want to the major grid lines but the minors won't appear. Any ideas? Here is the code:
```
import numpy as np
from matplotlib import pyplot as plt
from matplotlib import animation
import serial

SERIAL_PORT_NUM=9

...a bunch of constants...
#windows starts serial port numbers at 1, python starts at 0
SERIAL_PORT_NUM = SERIAL_PORT_NUM - 1
"""
Open the serial port
"""
ser =serial.Serial(port=SERIAL_PORT_NUM,baudrate=115200,bytesize=8,parity='N',stopbits=1,timeout=None,xonxoff=0,rtscts=0)

# First set up the figure, the axis, and the plot element we want to animate
raw_adc_fig = plt.figure()
raw_adc_ax = plt.axes(xlim=(0, 200), ylim=(0, 2047))
raw_adc_ax.grid(True, which='both')
raw_adc_fig.suptitle("Raw ADC data")
plt.ylabel("ADC values (hex)")
plt.xlabel("time (sec)")
raw_adc_line, = raw_adc_ax.plot([], [], lw=2)

def read_serial(serial_port):
    tmp = ''
    same_line = True
    while same_line:
        tmp += serial_port.read(1)
        if tmp != '':
            if tmp[-1] == '*':
                same_line = False
    tmp = tmp.rstrip()
    tmp = tmp.lstrip()
    return tmp

def process_serial(input_data):
    output_data = 0
    intermediate_data = input_data[A_TYPE_START_POS:A_TYPE_STOP_POS + 1]
    if( intermediate_data != ''):
        output_data =  int(intermediate_data , 16 )
    else:
        print "bad data"
        output_data = -100

    return output_data

def get_sound_value(serial_port):
    cur_line = ''

    get_next_line = True
    # read in the next line until a sound packet of type A is found
    while( get_next_line ):
        cur_line = read_serial(serial_port)
        if( (cur_line != '') and (cur_line[0:3] == ROUTER_SOUND_DATA) and (len(cur_line) == D_TYPE_STOP_POS + 2) ):
          get_next_line = False

    sound_value = process_serial(cur_line)
    return sound_value

# initialization function: plot the background of each frame
def raw_adc_init():
    raw_adc_line.set_data([], [])
    return raw_adc_line,

# animation function.  This is called sequentially
def raw_adc_animate(i):
    sound_data_list.append( get_sound_value(ser) )
    y = sound_data_list
    if( len(y) == 190 ):
        del y[0]
    x = np.linspace(0, len(y), len(y))
    raw_adc_line.set_data(x, y)
    return raw_adc_line,

# call the animator.  blit=True means only re-draw the parts that have changed.
raw_adc_anim = animation.FuncAnimation(raw_adc_fig, raw_adc_animate, init_func=raw_adc_init, frames=200, interval=1000, blit=True)
```
Edit: fixed a bug in opening the serial port. Changed timeout=0 to timeout=None.

Response: Unfortunately, ax.grid is a bit confusing in this regard. (This is a design bug / common gotcha.) It turns the minor grid on, but the minor ticks are still turned off. What you need to do is call plt.minorticks_on or ax.minorticks_on in addition to calling ax.grid(True, which='both').


Question:
Converting int arrays to string arrays in numpy without truncation Trying to convert int arrays to string arrays in numpy
```

```python
a=array([0,33,4444522])
```

```python
a.astype(str)
#Output
#array(['0', '3', '4'],
#      dtype='|S1')
#```
#Not what I intended
#```
```

```python
a.astype('S10')
#Output
#array(['0', '33', '4444522'],
#      dtype='|S10')
#```
#This works but I had to know 10 was big enough to hold my longest string. Is there a way of doing this easily without knowing ahead of time what size string you need? It seems a little dangerous that it just quietly truncates your string without throwing an error.
```

Response:
You can stay in numpy, doing
```
np.char.mod('%d', a)
```
This is twice faster than map or list comprehensions for 10 elements, four times faster for 100. This and other string operations are documented here.


Question:
How to have logarithmic bins in a Python histogram As far as I know the option Log=True in the histogram function only refers to the y-axis.
```
P.hist(d,bins=50,log=True,alpha=0.5,color='b',histtype='step')
```
I need the bins to be equally spaced in log10. Is there something that can do this?

Response:
use logspace() to create a geometric sequence, and pass it to bins parameter. And set the scale of xaxis to log scale.
```
import pylab as pl
import numpy as np

data = np.random.normal(size=10000)
pl.hist(data, bins=np.logspace(np.log10(0.1),np.log10(1.0), 50))
pl.gca().set_xscale("log")
pl.show()
```


Question:
How to avoid overlapping of labels & autopct in a pie chart My Python code is:
```
values = [234, 64, 54,10, 0, 1, 0, 9, 2, 1, 7, 7]
months = ['Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun',
          'Jul','Aug','Sep','Oct', 'Nov','Dec']

colors = ['yellowgreen', 'red', 'gold', 'lightskyblue',
          'white','lightcoral','blue','pink', 'darkgreen',
          'yellow','grey','violet','magenta','cyan']

plt.pie(values, labels=labels, autopct='%1.1f%%', shadow=True,
        colors=colors, startangle=90, radius=1.2)

plt.show()
```
Is it possible to show the labels "Jan", "Feb", "Mar", etc. and the percentages, either: without overlapping, or using an arrow mark?

Response:
Alternatively you can put the legends beside the pie graph:
```
import matplotlib.pyplot as plt
import numpy as np

x = np.char.array(['Jan','Feb','Mar','Apr','May','Jun','Jul','Aug','Sep','Oct', 'Nov','Dec'])
y = np.array([234, 64, 54,10, 0, 1, 0, 9, 2, 1, 7, 7])
colors = ['yellowgreen','red','gold','lightskyblue','white','lightcoral','blue','pink', 'darkgreen','yellow','grey','violet','magenta','cyan']
porcent = 100.*y/y.sum()

patches, texts = plt.pie(y, colors=colors, startangle=90, radius=1.2)
labels = ['{0} - {1:1.2f} %'.format(i,j) for i,j in zip(x, porcent)]

sort_legend = True
if sort_legend:
    patches, labels, dummy =  zip(*sorted(zip(patches, labels, y),
                                          key=lambda x: x[2],
                                          reverse=True))

plt.legend(patches, labels, loc='left center', bbox_to_anchor=(-0.1, 1.),
           fontsize=8)

plt.savefig('piechart.png', bbox_inches='tight')
```
EDIT: if you want to keep the legend in the original order, as you mentioned in the comments, you can set sort_legend=False in the code above, giving:


Question:
numpy convert categorical string arrays to an integer array I'm trying to convert a string array of categorical variables to an integer array of categorical variables. Ex.
```
import numpy as np
a = np.array( ['a', 'b', 'c', 'a', 'b', 'c'])
print a.dtype

```python
|S1
```

b = np.unique(a)
print b

```python
 ['a' 'b' 'c']
```

c = a.desired_function(b)
print c, c.dtype

```python
[1,2,3,1,2,3] int32
#Output
#```
#I realize this can be done with a loop but I imagine there is an easier way. Thanks.
```

Response:
np.unique has some optional returns return_inverse gives the integer encoding, which I use very often
```

```python
b, c = np.unique(a, return_inverse=True)
```

```python
b
#Output
#array(['a', 'b', 'c'],
#      dtype='|S1')
```

```python
c
#Output
#array([0, 1, 2, 0, 1, 2])
```

```python
c+1
#Output
#array([1, 2, 3, 1, 2, 3])
#```
#it can be used to recreate the original array from uniques
#```
```

```python
b[c]
#Output
#array(['a', 'b', 'c', 'a', 'b', 'c'],
#      dtype='|S1')
```

```python
(b[c] == a).all()
#Output
#True
#```
```


Question: Conditionally fill column values based on another columns value in pandas I have a DataFrame with a few columns. One columns contains a symbol for which currency is being used, for instance a euro or a dollar sign. Another column contains a budget value. So for instance in one row it could mean a budget of 5000 in euro and in the next row it could say a budget of 2000 in dollar. In pandas I would like to add an extra column to my DataFrame, normalizing the budgets in euro. So basically, for each row the value in the new column should be the value from the budget column * 1 if the symbol in the currency column is a euro sign, and the value in the new column should be the value of the budget column * 0.78125 if the symbol in the currency column is a dollar sign. I know how to add a column, fill it with values, copy values from another column etc. but not how to fill the new column conditionally based on the value of another column. Any suggestions?

Response:
You probably want to do
```
df['Normalized'] = np.where(df['Currency'] == '$', df['Budget'] * 0.78125, df['Budget'])
```


Question:
Annotate Time Series plot I have an index array (x) of dates (datetime objects) and an array of actual values (y: bond prices). Doing the following:
```py
plot(x,y)
```
produces a perfectly fine time series graph with the x-axis labeled with the dates. No problem so far. But I want to add text on certain dates. For example, on 2009-10-31, I wish to display the text "Event 1" with an arrow pointing to the y value at that date. I have read through the Matplotlib documentation on text() and annotate() to no avail.

Response:
Matplotlib uses an internal floating point format for dates. You just need to convert your date to that format (using matplotlib.dates.date2num or matplotlib.dates.datestr2num) and then use annotate as usual. As a somewhat excessively fancy example:
```
import datetime as dt
import matplotlib.pyplot as plt
import matplotlib.dates as mdates

x = [dt.datetime(2009, 05, 01), dt.datetime(2010, 06, 01),
     dt.datetime(2011, 04, 01), dt.datetime(2012, 06, 01)]
y = [1, 3, 2, 5]

fig, ax = plt.subplots()
ax.plot_date(x, y, linestyle='--')

ax.annotate('Test', (mdates.date2num(x[1]), y[1]), xytext=(15, 15),
            textcoords='offset points', arrowprops=dict(arrowstyle='-|>'))

fig.autofmt_xdate()
plt.show()
```


Question:
Why the performance difference between numpy.zeros and numpy.zeros_like? I finally found a performance bottleneck in my code but am confused as to what the reason is. To solve it I changed all my calls of numpy.zeros_like to instead use numpy.zeros. But why is zeros_like sooooo much slower? For example (note e-05 on the zeros call):
```

```python
timeit.timeit('np.zeros((12488, 7588, 3), np.uint8)', 'import numpy as np', number = 10)
#Output
#5.2928924560546875e-05
```

```python
timeit.timeit('np.zeros_like(x)', 'import numpy as np; x = np.zeros((12488, 7588, 3), np.uint8)', number = 10)
#Output
#1.4402990341186523
#```
#But then strangely writing to an array created with zeros is noticeably slower than an array created with zeros_like:
#```
```

```python
timeit.timeit('x[100:-100, 100:-100] = 1', 'import numpy as np; x = np.zeros((12488, 7588, 3), np.uint8)', number = 10)
#Output
#0.4310588836669922
```

```python
timeit.timeit('x[100:-100, 100:-100] = 1', 'import numpy as np; x = np.zeros_like(np.zeros((12488, 7588, 3), np.uint8))', number = 10)
#Output
#0.33325695991516113
#```
#My guess is zeros is using some CPU trick and not actually writing to the memory to allocate it. This is done on the fly when it's written to. But that still doesn't explain the massive discrepancy in array creation times. I'm running Mac OS X Yosemite with the current numpy version:
#```
```

```python
numpy.__version__
#Output
#'1.9.1'
#```
```

Response:
My timings in Ipython are (with a simplier timeit interface):
```

```python
timeit np.zeros_like(x)
1 loops, best of 3: 420 ms per loop
```

```python
timeit np.zeros((12488, 7588, 3), np.uint8)
100000 loops, best of 3: 15.1 µs per loop
```
When I look at the code with IPython (np.zeros_like??) I see:
```
res = empty_like(a, dtype=dtype, order=order, subok=subok)
multiarray.copyto(res, 0, casting='unsafe')
```
while np.zeros is a blackbox - pure compiled code. Timings for empty are:
```
```

```python
timeit np.empty_like(x)
100000 loops, best of 3: 13.6 µs per loop
```

```python
timeit np.empty((12488, 7588, 3), np.uint8)
100000 loops, best of 3: 14.9 µs per loop
```
So the extra time in zeros_like is in that copy. In my tests, the difference in assignment times (x[]=1) is negligible. My guess is that zeros, ones, empty are all early compiled creations. empty_like was added as a convenience, just drawing shape and type info from its input. zeros_like was written with more of an eye toward easy programming maintenance (reusing empty_like) than for speed. np.ones and np.full also use the np.empty ... copyto sequence, and show similar timings. https://github.com/numpy/numpy/blob/master/numpy/core/src/multiarray/array_assign_scalar.c appears to be file that copies a scalar (such as 0) to an array. I don't see a use of memset. https://github.com/numpy/numpy/blob/master/numpy/core/src/multiarray/alloc.c has calls to malloc and calloc. https://github.com/numpy/numpy/blob/master/numpy/core/src/multiarray/ctors.c - source for zeros and empty. Both call PyArray_NewFromDescr_int, but one ends up using npy_alloc_cache_zero and the other npy_alloc_cache. npy_alloc_cache in alloc.c calls alloc. npy_alloc_cache_zero calls npy_alloc_cache followed by a memset. Code in alloc.c is further confused with a THREAD option. More on the calloc v malloc+memset difference at: Why malloc+memset is slower than calloc? But with caching and garbage collection, I wonder whether the calloc/memset distinction applies. This simple test with the memory_profile package supports the claim that zeros and empty allocate memory 'on-the-fly', while zeros_like allocates everything up front:
```
N = (1000, 1000)
M = (slice(None, 500, None), slice(500, None, None))

Line #    Mem usage    Increment   Line Contents
================================================
     2   17.699 MiB    0.000 MiB   @profile
     3                             def test1(N, M):
     4   17.699 MiB    0.000 MiB       print(N, M)
     5   17.699 MiB    0.000 MiB       x = np.zeros(N)   # no memory jump
     6   17.699 MiB    0.000 MiB       y = np.empty(N)
     7   25.230 MiB    7.531 MiB       z = np.zeros_like(x) # initial jump
     8   29.098 MiB    3.867 MiB       x[M] = 1     # jump on usage
     9   32.965 MiB    3.867 MiB       y[M] = 1
    10   32.965 MiB    0.000 MiB       z[M] = 1
    11   32.965 MiB    0.000 MiB       return x,y,z
```
```


Question: sampling random floats on a range in numpy How can I sample random floats on an interval [a, b] in numpy? Not just integers, but any real numbers. For example, random_float(5, 10) would return random numbers between [5, 10]. thanks.

Response:
The uniform distribution would probably do what you are asking.
```
np.random.uniform(5,10) # A single value
np.random.uniform(5,10,[2,3]) # A 2x3 array
```


Question:
What's the fastest way in Python to calculate cosine similarity given sparse matrix data? Given a sparse matrix listing, what's the best way to calculate the cosine similarity between each of the columns (or rows) in the matrix? I would rather not iterate n-choose-two times. Say the input matrix is:
```
A=
[0 1 0 0 1
 0 0 1 1 1
 1 1 0 1 0]
```
The sparse representation is:
```
A =
0, 1
0, 4
1, 2
1, 3
1, 4
2, 0
2, 1
2, 3
```
In Python, it's straightforward to work with the matrix-input format:
```
import numpy as np
from sklearn.metrics import pairwise_distances
from scipy.spatial.distance import cosine

A = np.array(
[[0, 1, 0, 0, 1],
[0, 0, 1, 1, 1],
[1, 1, 0, 1, 0]])

dist_out = 1-pairwise_distances(A, metric="cosine")
dist_out
```
Gives:
```
array([[ 1.        ,  0.40824829,  0.40824829],
       [ 0.40824829,  1.        ,  0.33333333],
       [ 0.40824829,  0.33333333,  1.        ]])
```
That's fine for a full-matrix input, but I really want to start with the sparse representation (due to the size and sparsity of my matrix). Any ideas about how this could best be accomplished?

Response:
You can compute pairwise cosine similarity on the rows of a sparse matrix directly using sklearn. As of version 0.17 it also supports sparse output:
```
from sklearn.metrics.pairwise import cosine_similarity
from scipy import sparse

A =  np.array([[0, 1, 0, 0, 1], [0, 0, 1, 1, 1],[1, 1, 0, 1, 0]])
A_sparse = sparse.csr_matrix(A)

similarities = cosine_similarity(A_sparse)
print('pairwise dense output:\n {}\n'.format(similarities))

#also can output sparse matrices
similarities_sparse = cosine_similarity(A_sparse,dense_output=False)
print('pairwise sparse output:\n {}\n'.format(similarities_sparse))
```
Results:
```
pairwise dense output:
[[ 1.          0.40824829  0.40824829]
[ 0.40824829  1.          0.33333333]
[ 0.40824829  0.33333333  1.        ]]

pairwise sparse output:
(0, 1)  0.408248290464
(0, 2)  0.408248290464
(0, 0)  1.0
(1, 0)  0.408248290464
(1, 2)  0.333333333333
(1, 1)  1.0
(2, 1)  0.333333333333
(2, 0)  0.408248290464
(2, 2)  1.0
```
If you want column-wise cosine similarities simply transpose your input matrix beforehand:
```
A_sparse.transpose()
```