Question: Convert list of numpy.float64 to float in Python quickly What is the fastest way of converting a list of elements of type numpy.float64 to type float? I am currently using the straightforward for loop iteration in conjunction with float(). I came across this post: Converting numpy dtypes to native python types, however my question isn't one of how to convert types in python but rather more specifically how to best convert an entire list of one type to another in the quickest manner possible in python (i.e. in this specific case numpy.float64 to float). I was hoping for some secret python machinery that I hadn't come across that could do it all at once :)

Response:
The tolist() method should do what you want. If you have a numpy array, just call tolist():
```

```python
a
#Output
#array([ 0.        ,  0.14285714,  0.28571429,  0.42857143,  0.57142857,
#        0.71428571,  0.85714286,  1.        ,  1.14285714,  1.28571429,
#        1.42857143,  1.57142857,  1.71428571,  1.85714286,  2.        ])
```

```python
a.dtype
#Output
#dtype('float64')
```

```python
b = a.tolist()
```

```python
b
#Output
#[0.0,
# 0.14285714285714285,
# 0.2857142857142857,
# 0.42857142857142855,
# 0.5714285714285714,
# 0.7142857142857142,
# 0.8571428571428571,
# 1.0,
# 1.1428571428571428,
# 1.2857142857142856,
# 1.4285714285714284,
# 1.5714285714285714,
# 1.7142857142857142,
# 1.857142857142857,
# 2.0]
```

```python
type(b)
#Output
#list
```

```python
type(b[0])
#Output
#float
#```
#If, in fact, you really have python list of numpy.float64 objects, then @Alexander's answer is great, or you could convert the list to an array and then use the tolist() method. E.g.
#```
```

```python
c
#Output
#[0.0,
# 0.33333333333333331,
# 0.66666666666666663,
# 1.0,
# 1.3333333333333333,
# 1.6666666666666665,
# 2.0]
```

```python
type(c)
#Output
#list
```

```python
type(c[0])
#Output
#numpy.float64
#```
#@Alexander's suggestion, a list comprehension:
#```
```

```python
[float(v) for v in c]
#Output
#[0.0,
# 0.3333333333333333,
# 0.6666666666666666,
# 1.0,
# 1.3333333333333333,
# 1.6666666666666665,
# 2.0]
#```
#Or, convert to an array and then use the tolist() method.
#```
```

```python
np.array(c).tolist()
#Output
#[0.0,
# 0.3333333333333333,
# 0.6666666666666666,
# 1.0,
# 1.3333333333333333,
# 1.6666666666666665,
# 2.0]
#```
#If you are concerned with the speed, here's a comparison. The input, x, is a python list of numpy.float64 objects:
#```
```

```python
type(x)
#Output
#list
```

```python
len(x)
#Output
#1000
```

```python
type(x[0])
#Output
#numpy.float64
#```
#Timing for the list comprehension:
#```
```

```python
%timeit list1 = [float(v) for v in x]
10000 loops, best of 3: 109 µs per loop
```
Timing for conversion to numpy array and then tolist():
```
```

```python
%timeit list2 = np.array(x).tolist()
10000 loops, best of 3: 70.5 µs per loop
```
So it is faster to convert the list to an array and then call tolist().
```


Question:
how to get the memory address of a numpy array for C I constructed an numpy array::
```
a=np.ndarray([2,3])
```
then i want to see where its data are::
```
a.data
>>>Out[213]: <read-write buffer for 0x0482C1D0, size 48, offset 0 at 0x049E87A0>
a.data
>>>Out[214]: <read-write buffer for 0x0482C1D0, size 48, offset 0 at 0x049E82A0>
a.data
>>>Out[215]: <read-write buffer for 0x0482C1D0, size 48, offset 0 at 0x049E81C0>
```

```python
why every time the offset address is different? if i want to transfer the data to a c function using c_types by::
#Output
#```
#ctypes_array = (ctypes.c_char * a.size * 8).from_address(ptr)
#```
#how should i get the value of ptr?
```

Response:
Also, have a look at ndarray.__array_interface__, which is a dict that contains all of the information you're after. In your case,
```
pointer, read_only_flag = a.__array_interface__['data']
```


Question:
Calculating gradient with NumPy I really can not understand what numpy.gradient function does and how to use it for computation of multivariable function gradient. For example, I have such a function:
```
def func(q, chi, delta):
    return q * chi * delta
```
I need to compute it's 3-dimensional gradient (in other words, I want to compute partial derivatives with respect to all variables (q, chi, delta)). How can I calculate this gradient using NumPy?

Response:
The problem is, that numpy can't give you the derivatives directly and you have two options: With NUMPY What you essentially have to do, is to define a grid in three dimension and to evaluate the function on this grid. Afterwards you feed this table of function values to numpy.gradient to get an array with the numerical derivative for every dimension (variable). Example from here:
```
from numpy import *

x,y,z = mgrid[-100:101:25., -100:101:25., -100:101:25.]

V = 2*x**2 + 3*y**2 - 4*z # just a random function for the potential

Ex,Ey,Ez = gradient(V)
```
Without NUMPY You could also calculate the derivative yourself by using the centered difference quotient. This is essentially, what numpy.gradient is doing for every point of your predefined grid.


Question:
Cython: Convert memory view to NumPy array How to convert a typed memoryview to an NumPy array in cython? The docs have
```
cimport numpy as np
import numpy as np

numpy_array = np.asarray(<np.int32_t[:10, :10]> my_pointer)
```
I took this for my case
```
np.asarray(<np.float_t[:, :]> my_memview)
```
Using this the compiler tells me:
```
Can only create cython.array from pointer or array
```
Copying or not is not so decicive. I couldn't find any help on this.

Response:
You should just be able to use np.asarray directly on the memoryview itself, so something like:
```
np.asarray(my_memview)
```
should work. For example if your cython source file contains this:
```
import numpy as np
cimport numpy as np
def test(double[:,:] x):
    print type(x)
    print type(np.asarray(x))
```
Then after compiling it, you should be able to do the following from the python side:
```
a = np.random.normal(size=(5,5))
test(a)
```
Which produces the following output:
```
<type '_cython_magic_0182fca918bfa3618c3553e51b13e8ba._memoryviewslice'>
<type 'numpy.ndarray'>
```
Note: I'm using the cythonmagic extension for IPython.


Question:
multiply numpy ndarray with 1d array along a given axis It seems I am getting lost in something potentially silly. I have an n-dimensional numpy array, and I want to multiply it with a vector (1d array) along some dimension (which can change!). As an example, say I want to multiply a 2d array by a 1d array along axis 0 of the first array, I can do something like this:
```
a=np.arange(20).reshape((5,4))
b=np.ones(5)
c=a*b[:,np.newaxis]
```
Easy, but I would like to extend this idea to n-dimensions (for a, while b is always 1d) and to any axis. In other words, I would like to know how to generate a slice with the np.newaxis at the right place. Say that a is 3d and I want to multiply along axis=1, I would like to generate the slice which would correctly give:
```
c=a*b[np.newaxis,:,np.newaxis]
```
I.e. given the number of dimensions of a (say 3), and the axis along which I want to multiply (say axis=1), how do I generate and pass the slice:
```
np.newaxis,:,np.newaxis
```
Thanks.

Response:
Solution Code -
```
import numpy as np

# Given axis along which elementwise multiplication with broadcasting
# is to be performed
given_axis = 1

# Create an array which would be used to reshape 1D array, b to have
# singleton dimensions except for the given axis where we would put -1
# signifying to use the entire length of elements along that axis
dim_array = np.ones((1,a.ndim),int).ravel()
dim_array[given_axis] = -1

# Reshape b with dim_array and perform elementwise multiplication with
# broadcasting along the singleton dimensions for the final output
b_reshaped = b.reshape(dim_array)
mult_out = a*b_reshaped
```
Sample run for a demo of the steps -
```

```python
import numpy as np
```

```python
a = np.random.randint(0,9,(4,2,3))
```

```python
b = np.random.randint(0,9,(2,1)).ravel()
```

```python
whos
Variable   Type       Data/Info
-------------------------------
a          ndarray    4x2x3: 24 elems, type `int32`, 96 bytes
b          ndarray    2: 2 elems, type `int32`, 8 bytes
```

```python
given_axis = 1
```
Now, we would like to perform elementwise multiplications along given axis = 1. Let's create dim_array:
```
```

```python
dim_array = np.ones((1,a.ndim),int).ravel()
     ...: dim_array[given_axis] = -1
     ...:
```

```python
dim_array
#Output
#array([ 1, -1,  1])
#```
#Finally, reshape b & perform the elementwise multiplication:
#```
```

```python
b_reshaped = b.reshape(dim_array)
     ...: mult_out = a*b_reshaped
     ...:
```
Check out the whos info again and pay special attention to b_reshaped & mult_out:
```
```

```python
whos
Variable     Type       Data/Info
---------------------------------
a            ndarray    4x2x3: 24 elems, type `int32`, 96 bytes
b            ndarray    2: 2 elems, type `int32`, 8 bytes
b_reshaped   ndarray    1x2x1: 2 elems, type `int32`, 8 bytes
dim_array    ndarray    3: 3 elems, type `int32`, 12 bytes
given_axis   int        1
mult_out     ndarray    4x2x3: 24 elems, type `int32`, 96 bytes
```
```


Question:
Seeding random number generators in parallel programs I am studing the multiprocessing module of Python. I have two cases: Ex. 1
```
def Foo(nbr_iter):
    for step in xrange(int(nbr_iter)) :
        print random.uniform(0,1)
...

from multiprocessing import Pool

if __name__ == "__main__":
    ...
    pool = Pool(processes=nmr_parallel_block)
    pool.map(Foo, nbr_trial_per_process)
```
Ex 2. (using numpy)
```
def Foo_np(nbr_iter):
     np.random.seed()
     print np.random.uniform(0,1,nbr_iter)
```
In both cases the random number generators are seeded in their forked processes. Why do I have to do the seeding explicitly in the numpy example, but not in the Python example?

Response:
If no seed is provided explicitly, numpy.random will seed itself using an OS-dependent source of randomness. Usually it will use /dev/urandom on Unix-based systems (or some Windows equivalent), but if this is not available for some reason then it will seed itself from the wall clock. Since self-seeding occurs at the time when a new subprocess forks, it is possible for multiple subprocesses to inherit the same seed if they forked at the same time, leading to identical random variates being produced by different subprocesses. Often this correlates with the number of concurrent threads you are running. For example:
```
import numpy as np
import random
from multiprocessing import Pool

def Foo_np(seed=None):
    # np.random.seed(seed)
    return np.random.uniform(0, 1, 5)

pool = Pool(processes=8)
print np.array(pool.map(Foo_np, xrange(20)))

# [[ 0.14463001  0.80273208  0.5559258   0.55629762  0.78814652] <-
#  [ 0.14463001  0.80273208  0.5559258   0.55629762  0.78814652] <-
#  [ 0.14463001  0.80273208  0.5559258   0.55629762  0.78814652] <-
#  [ 0.14463001  0.80273208  0.5559258   0.55629762  0.78814652] <-
#  [ 0.14463001  0.80273208  0.5559258   0.55629762  0.78814652] <-
#  [ 0.14463001  0.80273208  0.5559258   0.55629762  0.78814652] <-
#  [ 0.14463001  0.80273208  0.5559258   0.55629762  0.78814652] <-
#  [ 0.64672339  0.99851749  0.8873984   0.42734339  0.67158796]
#  [ 0.64672339  0.99851749  0.8873984   0.42734339  0.67158796]
#  [ 0.64672339  0.99851749  0.8873984   0.42734339  0.67158796]
#  [ 0.64672339  0.99851749  0.8873984   0.42734339  0.67158796]
#  [ 0.64672339  0.99851749  0.8873984   0.42734339  0.67158796]
#  [ 0.11283279  0.28180632  0.28365286  0.51190168  0.62864241]
#  [ 0.11283279  0.28180632  0.28365286  0.51190168  0.62864241]
#  [ 0.28917586  0.40997875  0.06308188  0.71512199  0.47386047]
#  [ 0.11283279  0.28180632  0.28365286  0.51190168  0.62864241]
#  [ 0.64672339  0.99851749  0.8873984   0.42734339  0.67158796]
#  [ 0.11283279  0.28180632  0.28365286  0.51190168  0.62864241]
#  [ 0.14463001  0.80273208  0.5559258   0.55629762  0.78814652] <-
#  [ 0.11283279  0.28180632  0.28365286  0.51190168  0.62864241]]
```
You can see that groups of up to 8 threads simultaneously forked with the same seed, giving me identical random sequences (I've marked the first group with arrows). Calling np.random.seed() within a subprocess forces the thread-local RNG instance to seed itself again from /dev/urandom or the wall clock, which will (probably) prevent you from seeing identical output from multiple subprocesses. Best practice is to explicitly pass a different seed (or numpy.random.RandomState instance) to each subprocess, e.g.:
```
def Foo_np(seed=None):
    local_state = np.random.RandomState(seed)
    print local_state.uniform(0, 1, 5)

pool.map(Foo_np, range(20))
```
I'm not entirely sure what underlies the differences between random and numpy.random in this respect (perhaps it has slightly different rules for selecting a source of randomness to self-seed with compared to numpy.random?). I would still recommend explicitly passing a seed or a random.Random instance to each subprocess to be on the safe side. You could also use the .jumpahead() method of random.Random which is designed for shuffling the states of Random instances in multithreaded programs.


Question:
Why is `np.sum(range(N))` very slow? I saw a video about speed of loops in python, where it was explained that doing sum(range(N)) is much faster than manually looping through range and adding the variables together, since the former runs in C due to built-in functions being used, while in the latter the summation is done in (slow) python. I was curious what happens when adding numpy to the mix. As I expected np.sum(np.arange(N)) is the fastest, but sum(np.arange(N)) and np.sum(range(N)) are even slower than doing the naive for loop. Why is this? Here's the script I used to test, some comments about the supposed cause of slowing done where I know (taken mostly from the video) and the results I got on my machine (python 3.10.0, numpy 1.21.2): updated script:
```py
import numpy as np
from timeit import timeit

N = 10_000_000
repetition = 10

def sum0(N = N):
    s = 0
    i = 0
    while i < N: # condition is checked in python
        s += i
        i += 1 # both additions are done in python
    return s

def sum1(N = N):
    s = 0
    for i in range(N): # increment in C
        s += i # addition in python
    return s

def sum2(N = N):
    return sum(range(N)) # everything in C

def sum3(N = N):
    return sum(list(range(N)))

def sum4(N = N):
    return np.sum(range(N)) # very slow np.array conversion

def sum5(N = N):
    # much faster np.array conversion
    return np.sum(np.fromiter(range(N),dtype = int))

def sum5v2_(N = N):
    # much faster np.array conversion
    return np.sum(np.fromiter(range(N),dtype = np.int_))

def sum6(N = N):
    # possibly slow conversion to Py_long from np.int
    return sum(np.arange(N))

def sum7(N = N):
    # list returns a list of np.int-s
    return sum(list(np.arange(N)))

def sum7v2(N = N):
    # tolist conversion to python int seems faster than the implicit conversion
    # in sum(list()) (tolist returns a list of python int-s)
    return sum(np.arange(N).tolist())

def sum8(N = N):
    return np.sum(np.arange(N)) # everything in numpy (fortran libblas?)

def sum9(N = N):
    return np.arange(N).sum() # remove dispatch overhead

def array_basic(N = N):
    return np.array(range(N))

def array_dtype(N = N):
    return np.array(range(N),dtype = np.int_)

def array_iter(N = N):
    # np.sum's source code mentions to use fromiter to convert from generators
    return np.fromiter(range(N),dtype = np.int_)

print(f"while loop:         {timeit(sum0, number = repetition)}")
print(f"for loop:           {timeit(sum1, number = repetition)}")
print(f"sum_range:          {timeit(sum2, number = repetition)}")
print(f"sum_rangelist:      {timeit(sum3, number = repetition)}")
print(f"npsum_range:        {timeit(sum4, number = repetition)}")
print(f"npsum_iterrange:    {timeit(sum5, number = repetition)}")
print(f"npsum_iterrangev2:  {timeit(sum5, number = repetition)}")
print(f"sum_arange:         {timeit(sum6, number = repetition)}")
print(f"sum_list_arange:    {timeit(sum7, number = repetition)}")
print(f"sum_arange_tolist:  {timeit(sum7v2, number = repetition)}")
print(f"npsum_arange:       {timeit(sum8, number = repetition)}")
print(f"nparangenpsum:      {timeit(sum9, number = repetition)}")
print(f"array_basic:        {timeit(array_basic, number = repetition)}")
print(f"array_dtype:        {timeit(array_dtype, number = repetition)}")
print(f"array_iter:         {timeit(array_iter,  number = repetition)}")

print(f"npsumarangeREP:     {timeit(lambda : sum8(N/1000), number = 100000*repetition)}")
print(f"npsumarangeREP:     {timeit(lambda : sum9(N/1000), number = 100000*repetition)}")

# Example output:
#
# while loop:         11.493371912998555
# for loop:           7.385945574002108
# sum_range:          2.4605720699983067
# sum_rangelist:      4.509678105998319
# npsum_range:        11.85120212900074
# npsum_iterrange:    4.464334709002287
# npsum_iterrangev2:  4.498494338993623
# sum_arange:         9.537815956995473
# sum_list_arange:    13.290120724996086
# sum_arange_tolist:  5.231948580003518
# npsum_arange:       0.241889145996538
# nparangenpsum:      0.21876695199898677
# array_basic:        11.736577274998126
# array_dtype:        8.71628468400013
# array_iter:         4.303306431000237
# npsumarangeREP:     21.240833958996518
# npsumarangeREP:     16.690092379001726
```

Response:
np.sum(range(N)) is slow mostly because the current Numpy implementation do not use enough informations about the exact type/content of the values provided by the generator range(N). The heart of the general problem is inherently due to dynamic typing of Python and big integers although Numpy could optimize this specific case. First of all, range(N) returns a dynamically-typed Python object which is a (special kind of) Python generator. The object provided by this generator are also dynamically-typed. It is in practice a pure-Python integer. The thing is Numpy is written in the statically-typed language C and so it cannot efficiently work on dynamically-typed pure-Python objects. The strategy of Numpy is to convert such objects into C types when it can. One big problem in this case is that the integers provided by the generator can theorically be huge: Numpy do not know if the values can overflow a np.int32 or even a np.int64 type. Thus, Numpy first detect the good type to use and then compute the result using this type. This translation process can be quite expensive and appear not to be needed here since all the values provided by range(10_000_000). However, range(5_000_000_000) returns the same object type with pure-Python integers overflowing np.int32 and Numpy needs to automatically detect this case not to return wrong results. The thing is also the input type can be correctly identified (np.int32 on my machine), it does not means that the output result will be correct because overflows can appear in during the computation of the sum. This is sadly the case on my machine. Numpy developers decided to deprecate such a use and put in the documentation that np.fromiter should be used instead. np.fromiter has a dtype required parameter to let the user define what is the good type to use. One way to check this behaviour in practice is to simply use create a temporary list:
```py
tmp = list(range(10_000_000))

# Numpy implicitly convert the list in a Numpy array but
# still automatically detect the input type to use
np.sum(tmp)
```
A faster implementation is the following:
```py
tmp = list(range(10_000_000))

# The array is explicitly converted using a well-defined type and
# thus there is no need to perform an automatic detection
# (note that the result is still wrong since it does not fit in a np.int32)
tmp2 = np.array(tmp, dtype=np.int32)
result = np.sum(tmp2)
```
The first case takes 476 ms on my machine while the second takes 289 ms. Note that np.sum takes only 4 ms. Thus, a large part of the time is spend in the conversion of pure-Python integer objects to internal int32 types (more specifically the management of pure-Python integers). list(range(10_000_000)) is expensive too as it takes 205 ms. This is again due to the overhead of pure-Python integers (ie. allocations, deallocations, reference counting, increment of variable-sized integers, memory indirections and conditions due to the dynamic typing) as well as the overhead of the generator. sum(np.arange(N)) is slow because sum is a pure-Python function working on a Numpy-defined object. The CPython interpreter needs to call Numpy functions to perform basic additions. Moreover, Numpy-defined integer object are still Python object and so they are subject to reference counting, allocation, deallocation, etc. Not to mention Numpy and CPython add many checks in the functions aiming to finally just add two native numbers together. A Numpy-aware just-in-time compiler such as Numba can solve this issue. Indeed, Numba takes 23 ms on my machine to compute the sum of np.arange(10_000_000) (with code still written in Python) while the CPython interpreter takes 556 ms.


Question:
floor and ceil with number of decimals I need to floor a float number with an specific number of decimals. So:
```
2.1235 with 2 decimals --> 2.12
2.1276 with 2 decimals --> 2.12  (round would give 2.13 which is not what I need)
```
The function np.round accepts a decimals parameter but it appears that the functions ceil and floor don't accept a number of decimals and always return a number with zero decimals. Of course I can multiply the number by 10^ndecimals, then apply floor and finally divide by 10^ndecimals
```
new_value = np.floor(old_value * 10**ndecimals) / 10**ndecimals
```
But I'm wondering if there's a built-in function that does this without having to do the operations.

Response:
Neither Python built-in nor numpy's version of ceil/floor support precision. One hint though is to reuse round instead of multiplication + division (should be much faster):
```
def my_ceil(a, precision=0):
    return np.round(a + 0.5 * 10**(-precision), precision)

def my_floor(a, precision=0):
    return np.round(a - 0.5 * 10**(-precision), precision)
```
UPD: As pointed out by @aschipfl, for whole values np.round will round to the nearest even, which will lead to unexpected results, e.g. my_ceil(11) will return 12. Here is an updated solution, free of this problem:
```
def my_ceil(a, precision=0):
    return np.true_divide(np.ceil(a * 10**precision), 10**precision)

def my_floor(a, precision=0):
    return np.true_divide(np.floor(a * 10**precision), 10**precision)
```


Question:
Precision lost while using read_csv in pandas I have files of the below format in a text file which I am trying to read into a pandas dataframe.
```
895|2015-4-23|19|10000|LA|0.4677978806|0.4773469340|0.4089938425|0.8224291972|0.8652525793|0.6829942860|0.5139162227|
```
As you can see there are 10 integers after the floating point in the input file.
```
df = pd.read_csv('mockup.txt',header=None,delimiter='|')
```
When I try to read it into dataframe, I am not getting the last 4 integers
```
df[5].head()

0    0.467798
1    0.258165
2    0.860384
3    0.803388
4    0.249820
Name: 5, dtype: float64
```
How can I get the complete precision as present in the input file? I have some matrix operations that needs to be performed so i cannot cast it as string. I figured out that I have to do something about dtype but I am not sure where I should use it.

Response:
It is only display problem, see docs:
```
#temporaly set display precision
with pd.option_context('display.precision', 10):
    print df

     0          1   2      3   4             5            6             7   \
0  895  2015-4-23  19  10000  LA  0.4677978806  0.477346934  0.4089938425

             8             9            10            11  12
0  0.8224291972  0.8652525793  0.682994286  0.5139162227 NaN
```
EDIT: (Thank you Mark Dickinson): Pandas uses a dedicated decimal-to-binary converter that sacrifices perfect accuracy for the sake of speed. Passing float_precision='round_trip' to read_csv fixes this. See the documentation for more.


Question:
Numpy Indexing: Return the rest A simply example of numpy indexing:
```
In: a = numpy.arange(10)
In: sel_id = numpy.arange(5)
In: a[sel_id]
Out: array([0,1,2,3,4])
```
How do I return the rest of the array that are not indexed by sel_id? What I can think of is:
```
In: numpy.array([x for x in a if x not in a[id]])
out: array([5,6,7,8,9])
```
Is there any easier way?

Response:
For this simple 1D case, I'd actually use a boolean mask:
```
a = numpy.arange(10)
include_index = numpy.arange(4)
include_idx = set(include_index)  #Set is more efficient, but doesn't reorder your elements if that is desireable
mask = numpy.array([(i in include_idx) for i in xrange(len(a))])
```
Now you can get your values:
```
included = a[mask]  # array([0, 1, 2, 3])
excluded = a[~mask] # array([4, 5, 6, 7, 8, 9])
```
Note that a[mask] doesn't necessarily yield the same thing as a[include_index] since the order of include_index matters for the output in that scenario (it should be roughly equivalent to a[sorted(include_index)]). However, since the order of your excluded items isn't well defined, this should work Ok. EDIT A better way to create the mask is:
```
mask = np.zeros(a.shape,dtype=bool)
mask[include_idx] = True
```
(thanks to seberg).


Question:
'numpy.float64' object is not iterable I'm trying to iterate an array of values generated with numpy.linspace:
```
slX = numpy.linspace(obsvX, flightX, numSPts)
slY = np.linspace(obsvY, flightY, numSPts)

for index,point in slX:
    yPoint = slY[index]
    arcpy.AddMessage(yPoint)
```
This code worked fine on my office computer, but I sat down this morning to work from home on a different machine and this error came up:
```
File "C:\temp\gssm_arcpy.1.0.3.py", line 147, in AnalyzeSightLine
  for index,point in slX:
TypeError: 'numpy.float64' object is not iterable
```
slX is just an array of floats, and the script has no problem printing the contents -- just, apparently iterating through them. Any suggestions for what is causing it to break, and possible fixes?

Response:
numpy.linspace() gives you a one-dimensional NumPy array. For example:
```

```python
my_array = numpy.linspace(1, 10, 10)
```

```python
my_array
#Output
#array([  1.,   2.,   3.,   4.,   5.,   6.,   7.,   8.,   9.,  10.])
#```
#Therefore:
#```
#for index,point in my_array
#```
#cannot work. You would need some kind of two-dimensional array with two elements in the second dimension:
#```
```

```python
two_d = numpy.array([[1, 2], [4, 5]])
```

```python
two_d
#Output
#array([[1, 2], [4, 5]])
#```
#Now you can do this:
#```
```

```python
for x, y in two_d:
#Output
#    print(x, y)
```

1 2
4 5
```


Question:
Passing a structured numpy array with strings to a cython function I am attempting to create a function in cython that accepts a numpy structured array or record array by defining a cython struct type. Suppose I have the data:
```
a = np.recarray(3, dtype=[('a', np.float32),  ('b', np.int32), ('c', '|S5'), ('d', '|S3')])
a[0] = (1.1, 1, 'this\0', 'to\0')
a[1] = (2.1, 2, 'that\0', 'ta\0')
a[2] = (3.1, 3, 'dogs\0', 'ot\0')
```
(Note: the problem described below occurs with or without the null terminator) I then have the cython code:
```
import numpy as np
cimport numpy as np

cdef packed struct tstruct:
    np.float32_t a
    np.int32_t b
    char[5] c
    char[3] d

def test_struct(tstruct[:] x):
    cdef:
        int k
        tstruct y

    for k in xrange(3):
        y = x[k]
        print y.a, y.b, y.c, y.d
```
When I try to run test_struct(a), I get the error:
```
ValueError: Expected a dimension of size 5, got 8
```
If in the array and corresponding struct are reordered such that the fields containing strings are not adjacent to each other, then the function works as expected. It appears as if the Cython function is not detecting the boundary between the c and d fields correctly and thinks as if you are passing in a char array of the sum of the lengths. Short of reshuffling the data (which is possible but not ideal), is there another way to pass a recarray with fixed length string data into Cython? Update: This appears to be a potential Cython bug. See the following discussion on the Cython google group that hints at where the problem is arising: https://groups.google.com/forum/#!topic/cython-users/TbLbXdi0_h4 Update 2: This bug has been fixed in the master cython branch on Github as of Feb 23, 2014 and the patch is slated for inclusion in v0.20.2: https://github.com/cython/cython/commit/58d9361e0a6d4cb3d4e87775f78e0550c2fea836

Response: This was a bug that has been fixed in the master cython branch on Github as of Feb 22, 2014 and the patch is slated for inclusion in v0.20.2: https://github.com/cython/cython/commit/58d9361e0a6d4cb3d4e87775f78e0550c2fea836


Question:
Get all permutations of a numpy array I have a numpy array [0, 1, 1, 2, 2, 0, 1, ...] which only contains the numbers 0-k. I would like to create a new array that contains the n possible arrays of permutations of 0-k. A small example with k=2 and n=6:
```
a = [0, 1, 0, 2]
permute(a)
result = [[0, 1, 0, 2]
          [0, 2, 0, 1]
          [1, 0, 1, 2]
          [2, 1, 2, 0]
          [1, 2, 1, 0]
          [2, 0, 2, 1]]
```
Does anyone have any ideas/solutions as to how one could achieve this?

Response:
Your a is what combinatorists call a multiset. The sympy library has various routines for working with them.
```

```python
from sympy.utilities.iterables import multiset_permutations
```

```python
import numpy as np
```

```python
a = np.array([0, 1, 0, 2])
```

```python
for p in multiset_permutations(a):
    p

#Output
#[0, 0, 1, 2]
#[0, 0, 2, 1]
#[0, 1, 0, 2]
#[0, 1, 2, 0]
#[0, 2, 0, 1]
#[0, 2, 1, 0]
#[1, 0, 0, 2]
#[1, 0, 2, 0]
#[1, 2, 0, 0]
#[2, 0, 0, 1]
#[2, 0, 1, 0]
#[2, 1, 0, 0]
#```
```


Question:
Count consecutive occurences of values varying in length in a numpy array Say I have a bunch of numbers in a numpy array and I test them based on a condition returning a boolean array:
```
np.random.seed(3456)
a = np.random.rand(8)
condition = a>0.5
```
And with this boolean array I want to count all of the lengths of consecutive occurences of True. For example if I had [True,True,True,False,False,True,True,False,True] I would want to get back [3,2,1]. I can do that using this code:
```
length,count = [],0
for i in range(len(condition)):

    if condition[i]==True:
        count += 1
    elif condition[i]==False and count>0:
        length.append(count)
        count = 0

    if i==len(condition)-1 and count>0:
        length.append(count)

    print length
```
But is there anything already implemented for this or a python,numpy,scipy, etc. function that counts the length of consecutive occurences in a list or array for a given input?

Response:
If you already have a numpy array, this is probably going to be faster:
```

```python
condition = np.array([True,True,True,False,False,True,True,False,True])
```

```python
np.diff(np.where(np.concatenate(([condition[0]],
#Output
#                                     condition[:-1] != condition[1:],
#                                     [True])))[0])[::2]
#array([3, 2, 1])
#```
#It detects where chunks begin, has some logic for the first and last chunk, and simply computes differences between chunk starts and discards lengths corresponding to False chunks.
```


Question:
Python : How to avoid numpy RuntimeWarning in function definition? i designed a simple function to return a mathematical function which can be used to fit experimental data to it. The functions looks pretty much like the following:
```
def colecole_2(f,*p):
    term1=p[0] * ( 1 - 1 / (1 + numpy.power((0+1j) * 2 * numpy.pi * f * p[1], p[2])))
    term2=p[3] * ( 1 - 1 / (1 + numpy.power((0+1j) * 2 * numpy.pi * f * p[4], p[5])))
    return p[6]*(1-abs( term1+ term2))
```
Unfortunately I run into troubles with RunTimeWarnings as:
```
RuntimeWarning: overflow encountered in power
RuntimeWarning: overflow encountered in divide
```
due to values that are too large or small. I am not able to figure this problem out on my own though. Is there any way to redefine my function so it will pass without warnings?

Response:
You can use numpy.errstate which is a built-in context manager. This will let you set the err handing to be within the context of the with statement.
```
import numpy
# warning is not logged here. Perfect for clean unit test output
with numpy.errstate(divide='ignore'):
    numpy.float64(1.0) / 0.0
```
I had to do this recently when writing unit tests for some legacy python code.


Question: find time shift between two similar waveforms I have to compare two time-vs-voltage waveforms. Because of the peculiarity of the sources of these waveforms, one of them can be a time shifted version of the other. How can i find whether there is a time shift? and if yes, how much is it. I am doing this in Python and wish to use numpy/scipy libraries.

Response:
scipy provides a correlation function which will work fine for small input and also if you want non-circular correlation meaning that the signal will not wrap around. note that in mode='full' , the size of the array returned by signal.correlation is sum of the signal sizes minus one (i.e. len(a) + len(b) - 1), so the value from argmax is off by (signal size -1 = 20) from what you seem to expect.
```
from scipy import signal, fftpack
import numpy
a = numpy.array([0, 1, 2, 3, 4, 3, 2, 1, 0, 1, 2, 3, 4, 3, 2, 1, 0, 0, 0, 0, 0])
b = numpy.array([0, 0, 0, 0, 0, 1, 2, 3, 4, 3, 2, 1, 0, 1, 2, 3, 4, 3, 2, 1, 0])
numpy.argmax(signal.correlate(a,b)) -> 16
numpy.argmax(signal.correlate(b,a)) -> 24
```
The two different values correspond to whether the shift is in a or b. If you want circular correlation and for big signal size, you can use the convolution/Fourier transform theorem with the caveat that correlation is very similar to but not identical to convolution.
```
A = fftpack.fft(a)
B = fftpack.fft(b)
Ar = -A.conjugate()
Br = -B.conjugate()
numpy.argmax(numpy.abs(fftpack.ifft(Ar*B))) -> 4
numpy.argmax(numpy.abs(fftpack.ifft(A*Br))) -> 17
```
again the two values correspond to whether your interpreting a shift in a or a shift in b. The negative conjugation is due to convolution flipping one of the functions, but in correlation there is no flipping. You can undo the flipping by either reversing one of the signals and then taking the FFT, or taking the FFT of the signal and then taking the negative conjugate. i.e. the following is true: Ar = -A.conjugate() = fft(a[::-1])


Question:
Get mean value avoiding nan using numpy in python [duplicate] This question already has answers here: NumPy: calculate averages with NaNs removed (12 answers) Closed 11 years ago. How to calculate mean value of an array (A) avoiding nan?
```
import numpy as np
A = [5    nan    nan    nan    nan  10]
M = np.mean(A[A!=nan]) does not work
Any idea?
```

Response:
An other possibility is the following:
```
import numpy
from scipy.stats import nanmean # nanmedian exists too, if you need it
A = numpy.array([5, numpy.nan, numpy.nan, numpy.nan, numpy.nan, 10])
print nanmean(A) # gives 7.5 as expected
```
i guess this looks more elegant (and readable) than the other solution already given edit: apparently (@Jaime) reports that this functionality already exists directly in the latest numpy (1.8) as well, so no need to import scipy.stats anymore if you have that version of numpy:
```
import numpy
A = numpy.array([5, numpy.nan, numpy.nan, numpy.nan, numpy.nan, 10])
print numpy.nanmean(A)
```
the first solution works also for people who dont have the latest version of numpy (like me)


Question:
Vectorized look-up of values in Pandas dataframe I have two pandas dataframes one called orders and another one called daily_prices. daily_prices is as follows:
```
AAPL    GOOG     IBM    XOM
2011-01-10  339.44  614.21  142.78  71.57
2011-01-13  342.64  616.69  143.92  73.08
2011-01-26  340.82  616.50  155.74  75.89
2011-02-02  341.29  612.00  157.93  79.46
2011-02-10  351.42  616.44  159.32  79.68
2011-03-03  356.40  609.56  158.73  82.19
2011-05-03  345.14  533.89  167.84  82.00
2011-06-03  340.42  523.08  160.97  78.19
2011-06-10  323.03  509.51  159.14  76.84
2011-08-01  393.26  606.77  176.28  76.67
2011-12-20  392.46  630.37  184.14  79.97
```
orders is as follows:
```
direction  size ticker  prices
2011-01-10       Buy  1500   AAPL  339.44
2011-01-13      Sell  1500   AAPL  342.64
2011-01-13       Buy  4000    IBM  143.92
2011-01-26       Buy  1000   GOOG  616.50
2011-02-02      Sell  4000    XOM   79.46
2011-02-10       Buy  4000    XOM   79.68
2011-03-03      Sell  1000   GOOG  609.56
2011-03-03      Sell  2200    IBM  158.73
2011-06-03      Sell  3300    IBM  160.97
2011-05-03       Buy  1500    IBM  167.84
2011-06-10       Buy  1200   AAPL  323.03
2011-08-01       Buy    55   GOOG  606.77
2011-08-01      Sell    55   GOOG  606.77
2011-12-20      Sell  1200   AAPL  392.46
```
Index of both dataframes is datetime.date. prices column in the orders dataframe was added by using a list comprehension to loop through all the orders and look up the specific ticker for the specific date in the daily_prices data frame and then adding that list as a column to the orders dataframe. I would like to do this using an array operation rather than something that loops. can it be done? I tried to use:
```
daily_prices.ix[dates, tickers]
```
but this returns a matrix of cartesian product of the two lists. I want it to return a column vector of only the price of a specified ticker for a specified date.

Response:
Use our friend lookup, designed precisely for this purpose:
```

```python
prices
#Output
#AAPL    GOOG     IBM    XOM
#2011-01-10  339.44  614.21  142.78  71.57
#2011-01-13  342.64  616.69  143.92  73.08
#2011-01-26  340.82  616.50  155.74  75.89
#2011-02-02  341.29  612.00  157.93  79.46
#2011-02-10  351.42  616.44  159.32  79.68
#2011-03-03  356.40  609.56  158.73  82.19
#2011-05-03  345.14  533.89  167.84  82.00
#2011-06-03  340.42  523.08  160.97  78.19
#2011-06-10  323.03  509.51  159.14  76.84
#2011-08-01  393.26  606.77  176.28  76.67
#2011-12-20  392.46  630.37  184.14  79.97
```

```python
orders
#Output
#Date direction  size ticker  prices
#0  2011-01-10 00:00:00       Buy  1500   AAPL  339.44
#1  2011-01-13 00:00:00      Sell  1500   AAPL  342.64
#2  2011-01-13 00:00:00       Buy  4000    IBM  143.92
#3  2011-01-26 00:00:00       Buy  1000   GOOG  616.50
#4  2011-02-02 00:00:00      Sell  4000    XOM   79.46
#5  2011-02-10 00:00:00       Buy  4000    XOM   79.68
#6  2011-03-03 00:00:00      Sell  1000   GOOG  609.56
#7  2011-03-03 00:00:00      Sell  2200    IBM  158.73
#8  2011-06-03 00:00:00      Sell  3300    IBM  160.97
#9  2011-05-03 00:00:00       Buy  1500    IBM  167.84
#10 2011-06-10 00:00:00       Buy  1200   AAPL  323.03
#11 2011-08-01 00:00:00       Buy    55   GOOG  606.77
#12 2011-08-01 00:00:00      Sell    55   GOOG  606.77
#13 2011-12-20 00:00:00      Sell  1200   AAPL  392.46
```

```python
prices.lookup(orders.Date, orders.ticker)
#Output
#array([ 339.44,  342.64,  143.92,  616.5 ,   79.46,   79.68,  609.56,
#        158.73,  160.97,  167.84,  323.03,  606.77,  606.77,  392.46])
#```
```


Question:
Filter integers in numpy float array Is there any built in function to discard integer and keep only float number in numpy.
```
import numpy as np

input = np.array([0.0, 0.01, 1.0, 2.0, 2.001, 2.002])

desired_ouput = some_function(input)
# Expected ouput
# desired_output = np.array([0.01, 2.001, 2.002])
```

Response:
I don't think so. My approach would be
```
import numpy as np
a = np.array([0.0, 0.01, 1.0, 2.0, 2.001, 2.002])
mask = np.isclose(a, a.astype(int))

print(a[~mask])
#[ 0.01   2.001  2.002]
```


Question:
Python reshape list to ndim array Hi I have a list flat which is length 2800, it contains 100 results for each of 28 variables: Below is an example of 4 results for 2 variables
```
[0,
 0,
 1,
 1,
 2,
 2,
 3,
 3]
```
I would like to reshape the list to an array (2,4) so that the results for each variable are in a single element.
```
[[0,1,2,3],
 [0,1,2,3]]
```

Response:
You can think of reshaping that the new shape is filled row by row (last dimension varies fastest) from the flattened original list/array. If you want to fill an array by column instead, an easy solution is to shape the list into an array with reversed dimensions and then transpose it:
```
x = np.reshape(list_data, (100, 28)).T
```
Above snippet results in a 28x100 array, filled column-wise. To illustrate, here are the two options of shaping a list into a 2x4 array:
```
np.reshape([0, 0, 1, 1, 2, 2, 3, 3], (4, 2)).T
# array([[0, 1, 2, 3],
#        [0, 1, 2, 3]])

np.reshape([0, 0, 1, 1, 2, 2, 3, 3], (2, 4))
# array([[0, 0, 1, 1],
#        [2, 2, 3, 3]])
```


Question:
Find nth smallest element in numpy array I need to find just the the smallest nth element in a 1D numpy.array. For example:
```
a = np.array([90,10,30,40,80,70,20,50,60,0])
```
I want to get 5th smallest element, so my desired output is 40. My current solution is this:
```
result = np.max(np.partition(a, 5)[:5])
```
However, finding 5 smallest elements and then taking the largest one them seems little clumsy to me. Is there a better way to do it? Am I missing a single function that would achieve my goal? There are questions with similar titles to this one, but I did not see anything that answered my question. Edit: I should've mentioned it originally, but performance is very important for me; therefore, heapq solution though nice would not work for me.
```
import numpy as np
import heapq

def find_nth_smallest_old_way(a, n):
    return np.max(np.partition(a, n)[:n])

# Solution suggested by Jaime and HYRY
def find_nth_smallest_proper_way(a, n):
    return np.partition(a, n-1)[n-1]

def find_nth_smallest_heapq(a, n):
    return heapq.nsmallest(n, a)[-1]
#
n_iterations = 10000

a = np.arange(1000)
np.random.shuffle(a)

t1 = timeit('find_nth_smallest_old_way(a, 100)', 'from __main__ import find_nth_smallest_old_way, a', number = n_iterations)
print 'time taken using partition old_way: {}'.format(t1)
t2 = timeit('find_nth_smallest_proper_way(a, 100)', 'from __main__ import find_nth_smallest_proper_way, a', number = n_iterations)
print 'time taken using partition proper way: {}'.format(t2)
t3 = timeit('find_nth_smallest_heapq(a, 100)', 'from __main__ import find_nth_smallest_heapq, a', number = n_iterations)
print 'time taken using heapq : {}'.format(t3)
```
Result:
```
time taken using partition old_way: 0.255564928055
time taken using partition proper way: 0.129678010941
time taken using heapq : 7.81094002724
```

Response:
Unless I am missing something, what you want to do is:
```

```python
a = np.array([90,10,30,40,80,70,20,50,60,0])
```

```python
np.partition(a, 4)[4]
#Output
#40
#```
#np.partition(a, k) will place the k+1-th smallest element of a at a[k], smaller values in a[:k] and larger values in a[k+1:]. The only thing to be aware of is that, because of the 0 indexing, the fifth element is at index 4.
```


Question:
scipy.linalg.eig return complex eigenvalues for covariance matrix? The eigenvalues of a covariance matrix should be real and non-negative because covariance matrices are symmetric and semi positive definite. However, take a look at the following experiment with scipy:
```

```python
a=np.random.random(5)
```

```python
b=np.random.random(5)
```

```python
ab = np.vstack((a,b)).T
```

```python
C=np.cov(ab)
```

```python
eig(C)
#Output
#7.90174997e-01 +0.00000000e+00j,
#2.38344473e-17 +6.15983679e-17j,
#2.38344473e-17 -6.15983679e-17j,
#-1.76100435e-17 +0.00000000e+00j,
#5.42658040e-33 +0.00000000e+00j
#```
#However, reproducing the above example in Matlab works correctly:
#```
#a = [0.6271, 0.4314, 0.3453, 0.8073, 0.9739]
#b = [0.1924, 0.3680, 0.0568, 0.1831, 0.0176]
#C=cov([a;b])
#eig(C)
#-0.0000
#-0.0000
# 0.0000
# 0.0000
# 0.7902
#```
```

Response:
You have raised two issues: The eigenvalues returned by scipy.linalg.eig are not real. Some of the eigenvalues are negative. Both of these issues are the result of errors introduced by truncation and rounding errors, which always happen with iterative algorithms using floating-point arithmetic. Note that the Matlab results also produced negative eigenvalues. Now, for a more interesting aspect of the issue: why is Matlab's result real, whereas SciPy's result has some complex components? Matlab's eig detects if the input matrix is real symmetric or Hermitian and uses Cholesky factorization when it is. See the description of the chol argument in the eig documentation. This is not done automatically in SciPy. If you want to use an algorithm that exploits the structure of a real symmetric or Hermitian matrix, use scipy.linalg.eigh. For the example in the question:
```

```python
eigh(C, eigvals_only=True)
#Output
#array([ -3.73825923e-17,  -1.60154836e-17,   8.11704449e-19,
#         3.65055777e-17,   7.90175615e-01])
#```
#This result is the same as Matlab's, if you round to the same number of digits of precision that Matlab printed.
```


Question:
How to split an array according to a condition in numpy? For example, I have a ndarray that is:
```
a = np.array([1, 3, 5, 7, 2, 4, 6, 8])
```
Now I want to split a into two parts, one is all numbers =5:
```
[array([1,3,2,4]), array([5,7,6,8])]
```
Certainly I can traverse a and create two new array. But I want to know does numpy provide some better ways? Similarly, for multidimensional array, e.g.
```
array([[1, 2, 3],
       [4, 5, 6],
       [7, 8, 9],
       [2, 4, 7]])
```
I want to split it according to the first column =3, which result is:
```
[array([[1, 2, 3],
        [2, 4, 7]]),
 array([[4, 5, 6],
        [7, 8, 9]])]
```
Are there any better ways instead of traverse it? Thanks.

Response:
```
import numpy as np

def split(arr, cond):
  return [arr[cond], arr[~cond]]

a = np.array([1,3,5,7,2,4,6,8])
print split(a, a<5)

a = np.array([[1,2,3],[4,5,6],[7,8,9],[2,4,7]])
print split(a, a[:,0]<3)
```
This produces the following output:
```
[array([1, 3, 2, 4]), array([5, 7, 6, 8])]

[array([[1, 2, 3],
       [2, 4, 7]]), array([[4, 5, 6],
       [7, 8, 9]])]
```


Question: How to create 0x0 Numpy array? How do I create a 0 x 0 (i.e. ndim= 2, shape= (0,0)) float numpy.ndarray?

Response:
```

```python
import numpy as np
```

```python
a = np.empty( shape=(0, 0) )
```

```python
a
#Output
#    array([], shape=(0, 0), dtype=float64)
```

```python
a.shape
#Output
#    (0, 0)
```

```python
a.size
#Output
#    0
#```
#The array above is initialized as a 2D array--i.e., two size parameters passed for shape. Second, the call to empty is not strictly necessary--i.e., an array having 0 size could (i believe) be initialized using other array-creation methods in NumPy, e.g., NP.zeros, Np.ones, etc. I just chose empty because it gives the smallest array (memory-wise).
```


Question:
numpy random choice in Tensorflow Is there an equivalent function to numpy random choice in Tensorflow. In numpy we can get an item randomly from the given list with its weights.
```
np.random.choice([1,2,3,5], 1, p=[0.1, 0, 0.3, 0.6, 0])
```
This code will select an item from the given list with p weights.

Response:
No, but you can achieve the same result using tf.multinomial:
```
elems = tf.convert_to_tensor([1,2,3,5])
samples = tf.multinomial(tf.log([[1, 0, 0.3, 0.6]]), 1) # note log-prob
elems[tf.cast(samples[0][0], tf.int32)].eval()
Out: 1
elems[tf.cast(samples[0][0], tf.int32)].eval()
Out: 5
```
The [0][0] part is here, as multinomial expects a row of unnormalized log-probabilities for each element of the batch and also has another dimension for the number of samples.


Question:
Python Numpy - Complex Numbers - Is there a function for Polar to Rectangular conversion? Is there a built-in Numpy function to convert a complex number in polar form, a magnitude and an angle (degrees) to one in real and imaginary components? Clearly I could write my own but it seems like the type of thing for which there is an optimised version included in some module? More specifically, I have an array of magnitudes and an array of angles:
```

```python
a
#Output
#array([1, 1, 1, 1, 1])
```

```python
b
#Output
#array([120, 121, 120, 120, 121])
#```
#And what I would like is:
#```
```

```python
c
#Output
#[(-0.5+0.8660254038j),(-0.515038074+0.8571673007j),(-0.5+0.8660254038j),(-0.5+0.8660254038j),(-0.515038074+0.8571673007j)]
#```
```

Response:
There isn't a function to do exactly what you want, but there is angle, which does the hardest part. So, for example, one could define two functions:
```
def P2R(radii, angles):
    return radii * exp(1j*angles)

def R2P(x):
    return abs(x), angle(x)
```
These functions are using radians for input and output, and for degrees, one would need to do the conversion to radians in both functions. In the numpy reference there's a section on handling complex numbers, and this is where the function you're looking for would be listed (so since they're not there, I don't think they exist within numpy).


Question:
How to compute cluster assignments from linkage/distance matrices if you have this hierarchical clustering call in scipy in Python:
```
from scipy.cluster.hierarchy import linkage
# dist_matrix is long form distance matrix
linkage_matrix = linkage(squareform(dist_matrix), linkage_method)
```
then what's an efficient way to go from this to cluster assignments for individual points? i.e. a vector of length N where N is number of points, where each entry i is the cluster number of point i, given the number of clusters generated by a given threshold thresh on the resulting clustering? To clarify: The cluster number would be the cluster that it's in after applying a threshold to the tree. In which case you would get a unique cluster for each leaf node for the cluster that it is in. Unique in the sense that each point belongs to one "most specific cluster" which is defined by the threshold where you cut the dendrogram. I know that scipy.cluster.hierarchy.fclusterdata gives you this cluster assignment as its return value, but I am starting from a custom made distance matrix and distance metric, so I cannot use fclusterdata. The question boils down to: how can I compute what fclusterdata is computing -- the cluster assignments?

Response: If I understand you right, that is what fcluster does: scipy.cluster.hierarchy.fcluster(Z, t, criterion='inconsistent', depth=2, R=None, monocrit=None) Forms flat clusters from the hierarchical clustering defined by the linkage matrix Z. ... Returns: An array of length n. T[i] is the flat cluster number to which original observation i belongs. So just call fcluster(linkage_matrix, t), where t is your threshold.


Question:
Python DataFrame or list for storing objects Can I "store" instances of class in pandas/numpy Series-DataFrame/ndarray just like I do in list? Or these libraries support on built-in types (numerics, strings). For example I have Point with x,y coordinates, and I want to store Points in Plane, that would return Point with given coordinates.
```
#my class
class MyPoint:

    def __init__(self, x,y):
        self.x = x
        self.y = y

    @property
    def x(self):
        return self.x

    @property
    def y(self):
        return self.y
```
Here I create instances:
```
first_point = MyClass(1,1)
second_point = MyClass(2,2)
```
I can store instances in some list
```
my_list = []
my_list.append(first_point)
my_list.append(second_point)
```
The problem in list is that it's indexes do not correspond to x,y properties. Dictionary/DataFrame approach:
```
Plane = {"x" : [first_point.x, second_point.x], "y" : [first_point.y, second_point.y], "some_reference/id_to_point_instance" = ???}
Plane_pd = pd.DataFrame(Plane)
```
I've read posts, that using "id" of instance as third column value in DataFrame could cause problems with the garbage collector.

Response:
A pandas.DataFrame will gladly store python objects. Some test code to demonstrate... Test Code:
```
class MyPoint:
    def __init__(self, x, y):
        self._x = x
        self._y = y

    @property
    def x(self):
        return self._x

    @property
    def y(self):
        return self._y

my_list = [MyPoint(1, 1), MyPoint(2, 2)]
print(my_list)

plane_pd = pd.DataFrame([[p.x, p.y, p] for p in my_list],
                        columns=list('XYO'))
print(plane_pd.dtypes)
print(plane_pd)
```
Results:
```
[<__main__.MyPoint object at 0x033D2AF0>, <__main__.MyPoint object at 0x033D2B10>]

X     int64
Y     int64
O    object
dtype: object

   X  Y                                        O
0  1  1  <__main__.MyPoint object at 0x033D2AF0>
1  2  2  <__main__.MyPoint object at 0x033D2B10>
```
Notes: Note the two object in the list are the same two objects in the dataframe. Also note the dtype for the O column is object.


Question:
Sort a numpy array by another array, along a particular axis Similar to this answer, I have a pair of 3D numpy arrays, a and b, and I want to sort the entries of b by the values of a. Unlike this answer, I want to sort only along one axis of the arrays. My naive reading of the numpy.argsort() documentation:
```
Returns
-------
index_array : ndarray, int
    Array of indices that sort `a` along the specified axis.
    In other words, ``a[index_array]`` yields a sorted `a`.
```
led me to believe that I could do my sort with the following code:
```
import numpy

a = numpy.zeros((3, 3, 3))
a += numpy.array((1, 3, 2)).reshape((3, 1, 1))
print "a"
print a
"""
[[[ 1.  1.  1.]
  [ 1.  1.  1.]
  [ 1.  1.  1.]]

 [[ 3.  3.  3.]
  [ 3.  3.  3.]
  [ 3.  3.  3.]]

 [[ 2.  2.  2.]
  [ 2.  2.  2.]
  [ 2.  2.  2.]]]
"""
b = numpy.arange(3*3*3).reshape((3, 3, 3))
print "b"
print b
"""
[[[ 0  1  2]
  [ 3  4  5]
  [ 6  7  8]]

 [[ 9 10 11]
  [12 13 14]
  [15 16 17]]

 [[18 19 20]
  [21 22 23]
  [24 25 26]]]
"""
print "a, sorted"
print numpy.sort(a, axis=0)
"""
[[[ 1.  1.  1.]
  [ 1.  1.  1.]
  [ 1.  1.  1.]]

 [[ 2.  2.  2.]
  [ 2.  2.  2.]
  [ 2.  2.  2.]]

 [[ 3.  3.  3.]
  [ 3.  3.  3.]
  [ 3.  3.  3.]]]
"""

##This isnt' working how I'd like
sort_indices = numpy.argsort(a, axis=0)
c = b[sort_indices]
"""
Desired output:

[[[ 0  1  2]
  [ 3  4  5]
  [ 6  7  8]]

 [[18 19 20]
  [21 22 23]
  [24 25 26]]

 [[ 9 10 11]
  [12 13 14]
  [15 16 17]]]
"""
print "Desired shape of b[sort_indices]: (3, 3, 3)."
print "Actual shape of b[sort_indices]:"
print c.shape
"""
(3, 3, 3, 3, 3)
"""
```
What's the right way to do this?

Response:
You still have to supply indices for the other two dimensions for this to work correctly.
```

```python
a = numpy.zeros((3, 3, 3))
```

```python
a += numpy.array((1, 3, 2)).reshape((3, 1, 1))
```

```python
b = numpy.arange(3*3*3).reshape((3, 3, 3))
```

```python
sort_indices = numpy.argsort(a, axis=0)
```

```python
static_indices = numpy.indices((3, 3, 3))
```

```python
b[sort_indices, static_indices[1], static_indices[2]]
#Output
#array([[[ 0,  1,  2],
#        [ 3,  4,  5],
#        [ 6,  7,  8]],
```

[[18, 19, 20],
        [21, 22, 23],
        [24, 25, 26]],

       [[ 9, 10, 11],
        [12, 13, 14],
        [15, 16, 17]]])
```
numpy.indices calculates the indices of each axis of the array when "flattened" through the other two axes (or n - 1 axes where n = total number of axes). In other words, this (apologies for the long post):
```

```python
static_indices
#Output
#array([[[[0, 0, 0],
#         [0, 0, 0],
#         [0, 0, 0]],
```

[[1, 1, 1],
         [1, 1, 1],
         [1, 1, 1]],

        [[2, 2, 2],
         [2, 2, 2],
         [2, 2, 2]]],

       [[[0, 0, 0],
         [1, 1, 1],
         [2, 2, 2]],

        [[0, 0, 0],
         [1, 1, 1],
         [2, 2, 2]],

        [[0, 0, 0],
         [1, 1, 1],
         [2, 2, 2]]],

       [[[0, 1, 2],
         [0, 1, 2],
         [0, 1, 2]],

        [[0, 1, 2],
         [0, 1, 2],
         [0, 1, 2]],

        [[0, 1, 2],
         [0, 1, 2],
         [0, 1, 2]]]])
```
These are the identity indices for each axis; when used to index b, they recreate b.
```

```python
b[static_indices[0], static_indices[1], static_indices[2]]
#Output
#array([[[ 0,  1,  2],
#        [ 3,  4,  5],
#        [ 6,  7,  8]],
```

[[ 9, 10, 11],
        [12, 13, 14],
        [15, 16, 17]],

       [[18, 19, 20],
        [21, 22, 23],
        [24, 25, 26]]])
```
As an alternative to numpy.indices, you could use numpy.ogrid, as unutbu suggests. Since the object generated by ogrid is smaller, I'll create all three axes, just for consistency sake, but note unutbu's comment for a way to do this by generating only two.
```

```python
static_indices = numpy.ogrid[0:a.shape[0], 0:a.shape[1], 0:a.shape[2]]
```

```python
a[sort_indices, static_indices[1], static_indices[2]]
#Output
#array([[[ 1.,  1.,  1.],
#        [ 1.,  1.,  1.],
#        [ 1.,  1.,  1.]],
```

[[ 2.,  2.,  2.],
        [ 2.,  2.,  2.],
        [ 2.,  2.,  2.]],

       [[ 3.,  3.,  3.],
        [ 3.,  3.,  3.],
        [ 3.,  3.,  3.]]])
```


Question:
Most memory-efficient way to compute abs()**2 of complex numpy ndarray I'm looking for the most memory-efficient way to compute the absolute squared value of a complex numpy ndarray
```py
arr = np.empty((250000, 150), dtype='complex128')  # common size
```
I haven't found a ufunc that would do exactly np.abs()**2. As an array of that size and type takes up around half a GB, I'm looking for a primarily memory-efficient way. I would also like it to be portable, so ideally some combination of ufuncs. So far my understanding is that this should be about the best
```py
result = np.abs(arr)
result **= 2
```
It will needlessly compute (**0.5)**2, but should compute **2 in-place. Altogether the peak memory requirement is only the original array size + result array size, which should be 1.5 * original array size as the result is real. If I wanted to get rid of the useless **2 call I'd have to do something like this
```py
result = arr.real**2
result += arr.imag**2
```
but if I'm not mistaken, this means I'll have to allocate memory for both the real and imaginary part calculation, so the peak memory usage would be 2.0 * original array size. The arr.real properties also return a non-contiguous array (but that is of lesser concern). Is there anything I'm missing? Are there any better ways to do this? EDIT 1: I'm sorry for not making it clear, I don't want to overwrite arr, so I can't use it as out.

Response:
Thanks to numba.vectorize in recent versions of numba, creating a numpy universal function for the task is very easy:
```
@numba.vectorize([numba.float64(numba.complex128),numba.float32(numba.complex64)])
def abs2(x):
    return x.real**2 + x.imag**2
```
On my machine, I find a threefold speedup compared to a pure-numpy version that creates intermediate arrays:
```

```python
x = np.random.randn(10000).view('c16')
```

```python
y = abs2(x)
```

```python
np.all(y == x.real**2 + x.imag**2)   # exactly equal, being the same operation
#Output
#True
```

```python
%timeit np.abs(x)**2
#Output
#10000 loops, best of 3: 81.4 µs per loop
```

```python
%timeit x.real**2 + x.imag**2
#Output
#100000 loops, best of 3: 12.7 µs per loop
```

```python
%timeit abs2(x)
#Output
#100000 loops, best of 3: 4.6 µs per loop
#```
```


Question: Python: Making numpy default to float32 Is there any clean way of setting numpy to use float32 values instead of float64 globally?

Response: Not that I am aware of. You either need to specify the dtype explicitly when you call the constructor for any array, or cast an array to float32 (use the ndarray.astype method) before passing it to your GPU code (I take it this is what the question pertains to?). If it is the GPU case you are really worried about, I favor the latter - it can become very annoying to try and keep everything in single precision without an extremely thorough understanding of the numpy broadcasting rules and very carefully designed code. Another alternative might be to create your own methods which overload the standard numpy constructors (so numpy.zeros, numpy.ones, numpy.empty). That should go pretty close to keeping everything in float32.


Question: When will numpy copy the array when using reshape() In the document of numpy.reshape, it says: This will be a new view object if possible; otherwise, it will be a copy. Note there is no guarantee of the memory layout (C- or Fortran- contiguous) of the returned array. My question is, when will numpy chooses to return a new view, and when to copy the whole array? Is there any general principles telling people about the behavior of reshape, or it is just unpredictable?

Response:
The link that @mgillson found appears to address the question of 'how do I tell if it made a copy', but not 'how do I predict it' or understand why it made the copy. As for the test, I like to use A.__array_interfrace__. Most likely this would be a problem if you tried to assign values to the reshaped array, expecting to also change the original. And I'd be hard pressed to find a SO case where that was the issue. A copying reshape will be a bit slower than a noncopying one, but again I can't think of a case where that produced a slow down of the whole code. A copy could also be an issue if you are working with arrays so big that the simplest operation produces a memory error. After reshaping the values in the data buffer need to be in a contiguous order, either 'C' or 'F'. For example:
```

```python
np.arange(12).reshape(3,4,order='C')
#Output
#array([[ 0,  1,  2,  3],
#       [ 4,  5,  6,  7],
#       [ 8,  9, 10, 11]])
```

```python
np.arange(12).reshape(3,4,order='F')
#Output
#array([[ 0,  3,  6,  9],
#       [ 1,  4,  7, 10],
#       [ 2,  5,  8, 11]])
#```
#It will do a copy if the initial order is so 'messed up' that it can't return values like this. Reshape after transpose may do this (see my example below). So might games with stride_tricks.as_strided. Off hand those are the only cases I can think of.
#```
```

```python
x=np.arange(12).reshape(3,4,order='C')
```

```python
y=x.T
```

```python
x.__array_interface__
#Output
#{'version': 3,
# 'descr': [('', '<i4')],
# 'strides': None,
# 'typestr': '<i4',
# 'shape': (3, 4),
# 'data': (175066576, False)}
```

```python
y.__array_interface__
#Output
#{'version': 3,
# 'descr': [('', '<i4')],
# 'strides': (4, 16),
# 'typestr': '<i4',
# 'shape': (4, 3),
# 'data': (175066576, False)}
#```
#y, the transpose, has the same 'data' pointer. The transpose was performed without changing or copying the data, it just created a new object with new shape, strides, and flags.
#```
```

```python
y.flags
#Output
#C_CONTIGUOUS : False
#  F_CONTIGUOUS : True
#  ...
```

```python
x.flags
#Output
#C_CONTIGUOUS : True
#  F_CONTIGUOUS : False
#  ...
#```
#y is order 'F'. Now try reshaping it
#```
```

```python
y.shape
#Output
#(4, 3)
```

```python
z=y.reshape(3,4)
```

```python
z.__array_interface__
#Output
#{...
# 'shape': (3, 4),
# 'data': (176079064, False)}
```

```python
z
#Output
#array([[ 0,  4,  8,  1],
#       [ 5,  9,  2,  6],
#       [10,  3,  7, 11]])
#```
#z is a copy, its data buffer pointer is different. Its values are not arranged in any way that resembles that of x or y, no 0,1,2,.... But simply reshaping x does not produce a copy:
#```
```

```python
w=x.reshape(4,3)
```

```python
w
#Output
#array([[ 0,  1,  2],
#       [ 3,  4,  5],
#       [ 6,  7,  8],
#       [ 9, 10, 11]])
```

```python
w.__array_interface__
#Output
#{...
# 'shape': (4, 3),
# 'data': (175066576, False)}
#```
#Raveling y is the same as y.reshape(-1); it produces as copy:
#```
```

```python
y.reshape(-1)
#Output
#array([ 0,  4,  8,  1,  5,  9,  2,  6, 10,  3,  7, 11])
```

```python
y.ravel().__array_interface__['data']
#Output
#(175352024, False)
#```
#Assigning values to a raveled array like this may be the most likely case where a copy will produce an error. For example, x.ravel()[::2]=99 changes every other value of x and y (columns and rows respectively). But y.ravel()[::2]=0 does nothing because of this copying. So reshape after transpose is the most likely copy scenario. I'd be happy explore other possibilities. edit: y.reshape(-1,order='F')[::2]=0 does change the values of y. With a compatible order, reshape does not produce a copy. One answer in @mgillson's link, https://stackoverflow.com/a/14271298/901925, points out that the A.shape=... syntax prevents copying. If it can't change the shape without copying it will raise an error:
#```
```

```python
y.shape=(3,4)
...
AttributeError: incompatible shape for a non-contiguous array
```
This is also mentioned in the reshape documentation If you want an error to be raise if the data is copied, you should assign the new shape to the shape attribute of the array:: SO question about reshape following as_strided: reshaping a view of a n-dimensional array without using reshape and Numpy View Reshape Without Copy (2d Moving/Sliding Window, Strides, Masked Memory Structures) ========================== Here's my first cut at translating shape.c/_attempt_nocopy_reshape into Python. It can be run with something like:
```
newstrides = attempt_reshape(numpy.zeros((3,4)), (4,3), False)
```
```
import numpy   # there's an np variable in the code
def attempt_reshape(self, newdims, is_f_order):
    newnd = len(newdims)
    newstrides = numpy.zeros(newnd+1).tolist()  # +1 is a fudge

    self = numpy.squeeze(self)
    olddims = self.shape
    oldnd = self.ndim
    oldstrides = self.strides

    #/* oi to oj and ni to nj give the axis ranges currently worked with */

    oi,oj = 0,1
    ni,nj = 0,1
    while (ni < newnd) and (oi < oldnd):
        print(oi, ni)
        np = newdims[ni];
        op = olddims[oi];

        while (np != op):
            if (np < op):
                # /* Misses trailing 1s, these are handled later */
                np *= newdims[nj];
                nj += 1
            else:
                op *= olddims[oj];
                oj += 1

        print(ni,oi,np,op,nj,oj)

        #/* Check whether the original axes can be combined */
        for ok in range(oi, oj-1):
            if (is_f_order) :
                if (oldstrides[ok+1] != olddims[ok]*oldstrides[ok]):
                    # /* not contiguous enough */
                    return 0;
            else:
                #/* C order */
                if (oldstrides[ok] != olddims[ok+1]*oldstrides[ok+1]) :
                    #/* not contiguous enough */
                    return 0;

        # /* Calculate new strides for all axes currently worked with */
        if (is_f_order) :
            newstrides[ni] = oldstrides[oi];
            for nk in range(ni+1,nj):
                newstrides[nk] = newstrides[nk - 1]*newdims[nk - 1];
        else:
            #/* C order */
            newstrides[nj - 1] = oldstrides[oj - 1];
            #for (nk = nj - 1; nk > ni; nk--) {
            for nk in range(nj-1, ni, -1):
                newstrides[nk - 1] = newstrides[nk]*newdims[nk];
        nj += 1; ni = nj
        oj += 1; oi = oj
        print(olddims, newdims)
        print(oldstrides, newstrides)

    # * Set strides corresponding to trailing 1s of the new shape.
    if (ni >= 1) :
        print(newstrides, ni)
        last_stride = newstrides[ni - 1];
    else :
        last_stride = self.itemsize # PyArray_ITEMSIZE(self);

    if (is_f_order) :
        last_stride *= newdims[ni - 1];

    for nk in range(ni, newnd):
        newstrides[nk] = last_stride;
    return newstrides
```
```


Question: Root mean square of a function in python I want to calculate root mean square of a function in Python. My function is in a simple form like y = f(x). x and y are arrays. I tried Numpy and Scipy Docs and couldn't find anything.

Response:
I'm going to assume that you want to compute the expression given by the following pseudocode:
```
ms = 0
for i = 1 ... N
    ms = ms + y[i]^2
ms = ms / N
rms = sqrt(ms)
```
i.e. the square root of the mean of the squared values of elements of y. In numpy, you can simply square y, take its mean and then its square root as follows:
```
rms = np.sqrt(np.mean(y**2))
```
So, for example:
```

```python
y = np.array([0, 0, 1, 1, 0, 1, 0, 1, 1, 1])  # Six 1's
```

```python
y.size
#Output
#10
```

```python
np.mean(y**2)
#Output
#0.59999999999999998
```

```python
np.sqrt(np.mean(y**2))
#Output
#0.7745966692414834
#```
#Do clarify your question if you mean to ask something else.
```


Question:
Python Pandas: remove entries based on the number of occurrences I'm trying to remove entries from a data frame which occur less than 100 times. The data frame data looks like this:
```
pid   tag
1     23
1     45
1     62
2     24
2     45
3     34
3     25
3     62
```
Now I count the number of tag occurrences like this:
```
bytag = data.groupby('tag').aggregate(np.count_nonzero)
```
But then I can't figure out how to remove those entries which have low count...

Response:
New in 0.12, groupby objects have a filter method, allowing you to do these types of operations:
```

```python
g = data.groupby('tag')
```

```python
g.filter(lambda x: len(x) > 1)  # pandas 0.13.1
#Output
#pid  tag
#1    1   45
#2    1   62
#4    2   45
#7    3   62
#```
#The function (the first argument of filter) is applied to each group (subframe), and the results include elements of the original DataFrame belonging to groups which evaluated to True. Note: in 0.12 the ordering is different than in the original DataFrame, this was fixed in 0.13+:
#```
```

```python
g.filter(lambda x: len(x) > 1)  # pandas 0.12
#Output
#pid  tag
#1    1   45
#4    2   45
#2    1   62
#7    3   62
#```
```


Question:
Increase image brightness without overflow I got a problem when trying to increase image brightness. Here is the origin image: The image I wanted to get is like this: Now to increase the brightness with the following code:
```
image = cv2.imread("/home/wni/vbshare/tmp/a4_index2.png",0)

    if sum(image[0])/len(image[0])<200:
        new = np.where((255-image)<image,255,image*2)
    else:
        new = image
    return new
```
And, I got the following image: So, seems brightness of some points overflowed. And I tried to change the threshold from 200 to some other number, e.g. 125, 100, 140 .etc However, the image brightness stays either almost same dark or overflow. Env: Python: 2.7.10 Opencv: 3.2.0 Any suggestion for this is appreciated. Thanks.

Response:
Here's my shot at a simple algorithm for cleaning up that particular image. Feel free to play with it and tweak it further to get the desired result. NB: The code shown should work both with the 2.4.x and 3.x branches of OpenCV. Step 0 Load the input image as grayscale.
```
img = cv2.imread('paper.jpg',0)
```
Step 1 Dilate the image, in order to get rid of the text. This step somewhat helps to preserve the bar code.
```
dilated_img = cv2.dilate(img, np.ones((7,7), np.uint8))
```
Step 2 Median blur the result with a decent sized kernel to further suppress any text. This should get us a fairly good background image that contains all the shadows and/or discoloration.
```
bg_img = cv2.medianBlur(dilated_img, 21)
```
Step 3 Calculate the difference between the original and the background we just obtained. The bits that are identical will be black (close to 0 difference), the text will be white (large difference). Since we want black on white, we invert the result.
```
diff_img = 255 - cv2.absdiff(img, bg_img)
```
Step 4 Normalize the image, so that we use the full dynamic range.
```
norm_img = diff_img.copy() # Needed for 3.x compatibility
cv2.normalize(diff_img, norm_img, alpha=0, beta=255, norm_type=cv2.NORM_MINMAX, dtype=cv2.CV_8UC1)
```
Step 5 At this point we still have the paper somewhat gray. We can truncate that away, and re-normalize the image.
```
_, thr_img = cv2.threshold(norm_img, 230, 0, cv2.THRESH_TRUNC)
cv2.normalize(thr_img, thr_img, alpha=0, beta=255, norm_type=cv2.NORM_MINMAX, dtype=cv2.CV_8UC1)
```
Done... Well, at least for me ;) You will probably want to crop it, and do whatever other post-processing you desire. Note: It might be worth switching to higher precision (16+ bit int or float) after you get the difference image, in order to minimize accumulating rounding errors in the repeated normalizations.


Question:
Get array elements from index to end Suppose we have the following array:
```
import numpy as np
a = np.arange(1, 10)
a = a.reshape(len(a), 1)
array([[1],
       [2],
       [3],
       [4],
       [5],
       [6],
       [7],
       [8],
       [9]])
```
Now, i want to access the elements from index 4 to the end:
```
a[3:-1]
array([[4],
       [5],
       [6],
       [7],
       [8]])
```
When i do this, the resulting vector is missing the last element, now there are five elements instead of six, why does it happen, and how can i get the last element without appending it? Expected output:
```
array([[4],
       [5],
       [6],
       [7],
       [8],
       [9]])
```

Response:
The [:-1] removes the last element. Instead of
```
a[3:-1]
```
write
```
a[3:]
```
You can read up on Python slicing notation here: Understanding slicing NumPy slicing is an extension of that. The NumPy tutorial has some coverage: Indexing, Slicing and Iterating.


Question:
Python Exception in thread Thread-1 (most likely raised during interpreter shutdown)? My friend and I have been working on a large project to learn and for fun in python and PyGame. Basically it is an AI simulation of a small village. we wanted a day/night cycle so I found a neat way to change the color of an entire surface using numpy (specifically the cross-fade tutorial) - http://www.pygame.org/docs/tut/surfarray/SurfarrayIntro.html I implemented it into the code and it WORKS, but is extremely slow, like < 1 fps slow. so I look into threading (because I wanted to add it eventually) and found this page on Queues - Learning about Queue module in python (how to run it) I spend about 15 minutes making a basic system but as soon as I run it, the window closes and it says
```
Exception in thread Thread-1 (most likely raised during interpreter shutdown):
```
EDIT: This is literally all it says, no Traceback error I don't know what I am doing wrong, but I assume I am missing something simple. I added the necessary parts of the code below.
```
q_in = Queue.Queue(maxsize=0)

q_out = Queue.Queue(maxsize=0)

def run():    #Here is where the main stuff happens
    #There is more here I am just showing the essential parts
    while True:
        a = abs(abs(world.degree-180)-180)/400.

        #Process world
        world.process(time_passed_seconds)

        blank_surface = pygame.Surface(SCREEN_SIZE)
        world.render(blank_surface)    #The world class renders everything onto a blank surface
        q_in.put((blank_surface, a))
        screen.blit(q_out.get(), (0,0))

def DayNight():
    while True:
        blank_surface, a = q_in.get()
        imgarray = surfarray.array3d(blank_surface)  # Here is where the new numpy       stuff starts (AKA Day/Night cycle)
        src = N.array(imgarray)
        dest = N.zeros(imgarray.shape)
        dest[:] = 20, 30, 120
        diff = (dest - src) * a
        xfade = src + diff.astype(N.int)

        surfarray.blit_array(blank_surface, xfade)
        q_out.put(blank_surface)
        q_in.task_done()

def main():
    MainT = threading.Thread(target=run)
    MainT.daemon = True
    MainT.start()

    DN = threading.Thread(target=DayNight)
    DN.daemon = True
    DN.start()

    q_in.join()
    q_out.join()
```
If anyone could help it would be greatly appreciated. Thank you.

Response: This is pretty common when using daemon threads. Why are you setting .daemon = True on your threads? Think about it. While there are legitimate uses for daemon threads, most times a programmer does it because they're confused, as in "I don't know how to shut my threads down cleanly, and the program will freeze on exit if I don't, so I know! I'll say they're daemon threads. Then the interpreter won't wait for them to terminate when it exits. Problem solved." But it isn't solved - it usually just creates other problems. In particular, the daemon threads keep on running while the interpreter is - on exit - destroying itself. Modules are destroyed, stdin and stdout and stderr are destroyed, etc etc. All sorts of things can go wrong in daemon threads then, as the stuff they try to access is annihilated. The specific message you're seeing is produced when an exception is raised in some thread, but interpreter destruction has gotten so far that even the sys module no longer contains anything usable. The threading implementation retains a reference to sys.stderr internally so that it can tell you something then (specifically, the exact message you're seeing), but too much of the interpreter has been destroyed to tell you anything else about what went wrong. So find a way to shut down your threads cleanly instead (and remove .daemon = True). Don't know enough about your problem to suggest a specific way, but you'll think of something ;-) BTW, I'd suggest removing the maxsize=0 arguments on your Queue() constructors. The default is "unbounded", and "everyone knows that", while few people know that maxsize=0 also means "unbounded". That's gotten worse as other datatypes have taken maxsize=0 to mean "maximum size really is 0" (the best example of that is collections.deque); but "no argument means unbounded" is still universally true.


Question:
Reading binary data into pandas I have some binary data and I was wondering how I can load that into pandas. Can I somehow load it specifying the format it is in, and what the individual columns are called? Edit: Format is
```
int, int, int, float, int, int[256]
```
each comma separation represents a column in the data, i.e. the last 256 integers is one column.

Response:
Even though this is an old question, I was wondering the same thing and I didn't see a solution I liked. When reading binary data with Python I have found numpy.fromfile or numpy.fromstring to be much faster than using the Python struct module. Binary data with mixed types can be efficiently read into a numpy array, using the methods above, as long as the data format is constant and can be described with a numpy data type object (numpy.dtype).
```
import numpy as np
import pandas as pd

# Create a dtype with the binary data format and the desired column names
dt = np.dtype([('a', 'i4'), ('b', 'i4'), ('c', 'i4'), ('d', 'f4'), ('e', 'i4'),
               ('f', 'i4', (256,))])
data = np.fromfile(file, dtype=dt)
df = pd.DataFrame(data)

# Or if you want to explicitly set the column names
df = pd.DataFrame(data, columns=data.dtype.names)
```
Edits: Removed unnecessary conversion of data.to_list(). Thanks fxx Added example of leaving off the columns argument


Question:
Applying a function along a numpy array I've the following numpy ndarray.
```
[ -0.54761371  17.04850603   4.86054302]
```
I want to apply this function to all elements of the array
```
def sigmoid(x):
  return 1 / (1 + math.exp(-x))

probabilities = np.apply_along_axis(sigmoid, -1, scores)
```
This is the error that I get.
```
TypeError: only length-1 arrays can be converted to Python scalars
```
What am I doing wrong.

Response:
Function numpy.apply_along_axis is not good for this purpose. Try to use numpy.vectorize to vectorize your function: https://docs.scipy.org/doc/numpy/reference/generated/numpy.vectorize.html This function defines a vectorized function which takes a nested sequence of objects or numpy arrays as inputs and returns an single or tuple of numpy array as output.
```
import numpy as np
import math

# custom function
def sigmoid(x):
  return 1 / (1 + math.exp(-x))

# define vectorized sigmoid
sigmoid_v = np.vectorize(sigmoid)

# test
scores = np.array([ -0.54761371,  17.04850603,   4.86054302])
print sigmoid_v(scores)
```
Output: [ 0.36641822 0.99999996 0.99231327] Performance test which shows that the scipy.special.expit is the best solution to calculate logistic function and vectorized variant comes to the worst:
```
import numpy as np
import math
import timeit

def sigmoid_(x):
  return 1 / (1 + math.exp(-x))
sigmoidv = np.vectorize(sigmoid_)

def sigmoid(x):
   return 1 / (1 + np.exp(x))

print timeit.timeit("sigmoidv(scores)", "from __main__ import sigmoidv, np; scores = np.random.randn(100)", number=25),\
timeit.timeit("sigmoid(scores)", "from __main__ import sigmoid, np; scores = np.random.randn(100)",  number=25),\
timeit.timeit("expit(scores)", "from scipy.special import expit; import numpy as np;   scores = np.random.randn(100)",  number=25)

print timeit.timeit("sigmoidv(scores)", "from __main__ import sigmoidv, np; scores = np.random.randn(1000)", number=25),\
timeit.timeit("sigmoid(scores)", "from __main__ import sigmoid, np; scores = np.random.randn(1000)",  number=25),\
timeit.timeit("expit(scores)", "from scipy.special import expit; import numpy as np;   scores = np.random.randn(1000)",  number=25)

print timeit.timeit("sigmoidv(scores)", "from __main__ import sigmoidv, np; scores = np.random.randn(10000)", number=25),\
timeit.timeit("sigmoid(scores)", "from __main__ import sigmoid, np; scores = np.random.randn(10000)",  number=25),\
timeit.timeit("expit(scores)", "from scipy.special import expit; import numpy as np;   scores = np.random.randn(10000)",  number=25)
```
Results:
```
size        vectorized      numpy                 expit
N=100:   0.00179314613342 0.000460863113403 0.000132083892822
N=1000:  0.0122890472412  0.00084114074707  0.000464916229248
N=10000: 0.109477043152   0.00530695915222  0.00424313545227
```


Question:
Drop column that starts with I have a data frame that has multiple columns, example:
```
Prod_A       Prod_B        Prod_C        State         Region
1          1           0             1             1              1
```
I would like to drop all columns that starts with Prod_, (I can't select or drop by name because the data frame has 200 variables) Is it possible to do this ? Thank you

Response:
Use startswith for mask and then delete columns with loc and boolean indexing:
```
df = df.loc[:, ~df.columns.str.startswith('Prod')]
print (df)
   State  Region
1      1       1
```


Question:
python: OpenCV depth of image unsupported (CV_64F) So, I'm trying to show a binary picture with only black and white using this code:
```
import cv2
import numpy as np

x_img = cv2.imread("lenac.tif")

x_img_g = cv2.cvtColor(x_img, cv2.COLOR_BGR2GRAY)

y = x_img_g > 128

cv2.imshow("", y*1.0)
cv2.waitKey(0)
cv2.destroyAllWindows()
```
But I'm getting this error:
```
>Traceback (most recent call last):
File "ex5.py", line 11, in <module>
cv2.imshow("", y*1.0)
cv2.error: OpenCV(4.0.0) c:\projects\opencv-
python\opencv\modules\imgproc\src\color.hpp:261: error: (-2:Unspecified
error) >in function '__cdecl cv::CvtHelper<struct
cv::Set<1,-1,-1>,struct cv::Set<3,4,-1>,struct
cv::Set<0,2,5>,2>::CvtHelper(const class cv::_InputArray &,const class
cv::_OutputArray &,int)'
>Unsupported depth of input image:
>     'VDepth::contains(depth)'
> where
>     'depth' is 6 (CV_64F).
```

Response: try cv2.imshow("", y.astype('float32')) or cv2.imshow("", y.astype('uint8') * 255) CV_64F means the numpy array dtype is float64 (64-bit floating-point). opencv imshow only works with float32 (32-bit floating point) where the range for the pixel values is 0.0-1.0 or uint8 (unsigned 8-bit) where the range is 0-255 Since y was a bool, converting it to a number means converting True to 1 for float32, that is fine because 1 is the maximum for the imshow range if you use uint8, that means you're trying to display pixels of value 1/255 (since the maximum imshow range for uint8 is 255) which will be barely visible, so you can multiply by 255 to bring those pixels to max and appear as bright white pixels


Question: Converting image from RGB to HSV color space I couldn't find such function (i.e. RGB_to_HSV()) in Scipy or Matplotlib's documentations, and Google doesn't show pointers, except ActiveState recipe which demonstrates rgb2hsv function, though not usable on Numpy array as is. Does someone know of a shortcut? Edit: Sorry, just found matplotlib.colors.rgb_to_hsv() which is exactly what I was looking for. Should I delete this question?

Response: Matplotlib provides RGB to HSV conversion function: matplotlib.colors.rgb_to_hsv(): matplotlib.colors.rgb_to_hsv(arr) convert rgb values in a numpy array to hsv values input and output arrays should have shape (M,N,3)


Question:
Is "norm" equivalent to "Euclidean distance"? I am not sure whether "norm" and "Euclidean distance" mean the same thing. Please could you help me with this distinction. I have an n by m array a, where m > 3. I want to calculate the Eculidean distance between the second data point a[1,:] to all the other points (including itself). So I used the np.linalg.norm, which outputs the norm of two given points. But I don't know if this is the right way of getting the EDs.
```
import numpy as np

a = np.array([[0, 0, 0 ,0 ], [1, 1 , 1, 1],[2,2, 2, 3], [3,5, 1, 5]])
N = a.shape[0] # number of row
pos = a[1,:] # pick out the second data point.
dist = np.zeros((N,1), dtype=np.float64)

for i in range(N):
    dist[i]= np.linalg.norm(a[i,:] - pos)
```

Response:
A norm is a function that takes a vector as an input and returns a scalar value that can be interpreted as the "size", "length" or "magnitude" of that vector. More formally, norms are defined as having the following mathematical properties: They scale multiplicatively, i.e. Norm(a·v) = |a|·Norm(v) for any scalar a They satisfy the triangle inequality, i.e. Norm(u + v) ≤ Norm(u) + Norm(v) The norm of a vector is zero if and only if it is the zero vector, i.e. Norm(v) = 0 ⇔ v = 0 The Euclidean norm (also known as the L² norm) is just one of many different norms - there is also the max norm, the Manhattan norm etc. The L² norm of a single vector is equivalent to the Euclidean distance from that point to the origin, and the L² norm of the difference between two vectors is equivalent to the Euclidean distance between the two points. As @nobar's answer says, np.linalg.norm(x - y, ord=2) (or just np.linalg.norm(x - y)) will give you Euclidean distance between the vectors x and y. Since you want to compute the Euclidean distance between a[1, :] and every other row in a, you could do this a lot faster by eliminating the for loop and broadcasting over the rows of a:
```
dist = np.linalg.norm(a[1:2] - a, axis=1)
```
It's also easy to compute the Euclidean distance yourself using broadcasting:
```
dist = np.sqrt(((a[1:2] - a) ** 2).sum(1))
```
The fastest method is probably scipy.spatial.distance.cdist:
```
from scipy.spatial.distance import cdist

dist = cdist(a[1:2], a)[0]
```
Some timings for a (1000, 1000) array:
```
a = np.random.randn(1000, 1000)

%timeit np.linalg.norm(a[1:2] - a, axis=1)
# 100 loops, best of 3: 5.43 ms per loop

%timeit np.sqrt(((a[1:2] - a) ** 2).sum(1))
# 100 loops, best of 3: 5.5 ms per loop

%timeit cdist(a[1:2], a)[0]
# 1000 loops, best of 3: 1.38 ms per loop

# check that all 3 methods return the same result
d1 = np.linalg.norm(a[1:2] - a, axis=1)
d2 = np.sqrt(((a[1:2] - a) ** 2).sum(1))
d3 = cdist(a[1:2], a)[0]

assert np.allclose(d1, d2) and np.allclose(d1, d3)
```


Question:
Why is x**3 slower than x*x*x? [duplicate] This question already has answers here: Speed of calculating powers (in python) (6 answers) Closed 11 years ago. In NumPy, x*x*x is an order of magnitude faster than x**3 or even np.power(x, 3).
```
x = np.random.rand(1e6)
%timeit x**3
100 loops, best of 3: 7.07 ms per loop

%timeit x*x*x
10000 loops, best of 3: 163 µs per loop

%timeit np.power(x, 3)
100 loops, best of 3: 7.15 ms per loop
```
Any ideas as to why this behavior happens? As far as I can tell all three yield the same output (checked with np.allclose).

Response:
As per this answer, it's because the implementation of exponentiation has some overhead that multiplication does not. However, naive multiplication will get slower and slower as the exponent increases. An empirical demonstration:
```

```python
x = np.random.rand(1e6)
```

```python
%timeit x**2
 100 loops, best of 3: 11.9 ms per loop
```

```python
%timeit x*x
 100 loops, best of 3: 12.7 ms per loop
```

```python
%timeit x**3
 10 loops, best of 3: 132 ms per loop
```

```python
%timeit x*x*x
 10 loops, best of 3: 27.2 ms per loop
```

```python
%timeit x**4
 10 loops, best of 3: 132 ms per loop
```

```python
%timeit x*x*x*x
 10 loops, best of 3: 42.4 ms per loop
```

```python
%timeit x**10
 10 loops, best of 3: 132 ms per loop
```

```python
%timeit x*x*x*x*x*x*x*x*x*x
 10 loops, best of 3: 137 ms per loop
```

```python
%timeit x**15
 10 loops, best of 3: 132 ms per loop
```

```python
%timeit x*x*x*x*x*x*x*x*x*x*x*x*x*x*x
 1 loops, best of 3: 212 ms per loop
```
Note the exponentiation time stays more or less constant, except for the x**2 case which I suspect is special-cased, while multiplication gets slower and slower. It seems you could exploit this to get faster integer exponentiation... for example:
```
```

```python
%timeit x**16
10 loops, best of 3: 132 ms per loop
```

```python
%timeit x*x*x*x*x*x*x*x*x*x*x*x*x*x*x*x
1 loops, best of 3: 225 ms per loop
```

```python
def tosixteenth(x):
   ....:     x2 = x*x
   ....:     x4 = x2*x2
   ....:     x8 = x4*x4
   ....:     x16 = x8*x8
   ....:     return x16
   ....:
```

```python
%timeit tosixteenth(x)
10 loops, best of 3: 49.5 ms per loop
```
It seems you could apply this technique generically by splitting any integer into a sum of the powers of two, computing each power of two as above, and summing:
```
```

```python
%paste
def smartintexp(x, exp):
    result = np.ones(len(x))
    curexp = np.array(x)
    while True:
        if exp%2 == 1:
            result *= curexp
        exp >>= 1
        if not exp: break
        curexp *= curexp
    return result
## -- End pasted text --
```

```python
x
#Output
#array([ 0.0163407 ,  0.57694587,  0.47336487, ...,  0.70255032,
#        0.62043303,  0.0796748 ])
```

```python
x**21
#Output
#array([  3.01080670e-38,   9.63466181e-06,   1.51048544e-07, ...,
#         6.02873388e-04,   4.43193256e-05,   8.46721060e-24])
```

```python
smartintexp(x, 21)
#Output
#array([  3.01080670e-38,   9.63466181e-06,   1.51048544e-07, ...,
#         6.02873388e-04,   4.43193256e-05,   8.46721060e-24])
```

```python
%timeit x**21
10 loops, best of 3: 132 ms per loop
```

```python
%timeit smartintexp(x, 21)
10 loops, best of 3: 70.7 ms per loop
```
It's fast for small even powers of two:
```
```

```python
%timeit x**32
10 loops, best of 3: 131 ms per loop
```

```python
%timeit smartintexp(x, 32)
10 loops, best of 3: 57.4 ms per loop
```
But gets slower as the exponent gets larger:
```
```

```python
%timeit x**63
10 loops, best of 3: 133 ms per loop
```

```python
%timeit smartintexp(x, 63)
10 loops, best of 3: 110 ms per loop
```
And not faster for large worst-cases:
```
```

```python
%timeit x**511
10 loops, best of 3: 135 ms per loop
```

```python
%timeit smartintexp(x, 511)
10 loops, best of 3: 192 ms per loop
```
```


Question:
Numpy python find minimum value of each column and subtract this value from each column Can anyone help with a snippet of code using numpy and python? Given an numpy array such as
```
a = array([[1,11], [3,9], [5,7]]
```
I want to find the minimun value of each column, so 1 and 7 and then subtract this value from the respective columns,
```
a = array([[0,4], [2,2], [4,0]]
```

Response:
```

```python
a - a.min(axis=0)
#Output
#array([[0, 4],
#       [2, 2],
#       [4, 0]])
#```
#Where axis=0 refers to columns.
```


Question:
How to iterate over columns of a matrix? In python if a define:
```
a = arange(9).reshape(3,3)
```
as a 3x3 matrix and iterate:
```
for i in a:
```
It'll iterate over the matrix's rows. Is there any way to iterate over columns?

Response:
How about
```
for i in a.transpose():
```
or, shorter:
```
for i in a.T:
```
This may look expensive but is in fact very cheap (it returns a view onto the same data, but with the shape and stride attributes permuted).


Question:
Fast way to Hash Numpy objects for Caching Implementing a system where, when it comes to the heavy mathematical lifting, I want to do as little as possible. I'm aware that there are issues with memoisation with numpy objects, and as such implemented a lazy-key cache to avoid the whole "Premature optimisation" argument.
```
def magic(numpyarg,intarg):
    key = str(numpyarg)+str(intarg)

    try:
        ret = self._cache[key]
        return ret
    except:
        pass

    ... here be dragons ...
    self._cache[key]=value
    return value
```
but since string conversion takes quite a while...
```
t=timeit.Timer("str(a)","import numpy;a=numpy.random.rand(10,10)")
t.timeit(number=100000)/100000 = 0.00132s/call
```
What do people suggest as being 'the better way' to do it?

Response:
Borrowed from this answer... so really I guess this is a duplicate:
```

```python
import hashlib
```

```python
import numpy
```

```python
a = numpy.random.rand(10, 100)
```

```python
b = a.view(numpy.uint8)
```

```python
hashlib.sha1(b).hexdigest()
#Output
#'15c61fba5c969e5ed12cee619551881be908f11b'
```

```python
t=timeit.Timer("hashlib.sha1(a.view(numpy.uint8)).hexdigest()",
#Output
#                   "import hashlib;import numpy;a=numpy.random.rand(10,10)")
```

```python
t.timeit(number=10000)/10000
#Output
#2.5790500640869139e-05
#```
```


Question:
python+numpy: why does numpy.log throw an attribute error if its operand is too big? Running
```
np.log(math.factorial(21))
```
throws an AttributeError: log. Why is that? I could imagine a ValueError, or some sort of UseYourHighSchoolMathsError, but why the attribute error?

Response:
The result of math.factorial(21) is a Python long. numpy cannot convert it to one of its numeric types, so it leaves it as dtype=object. The way that unary ufuncs work for object arrays is that they simply try to call a method of the same name on the object. E.g.
```
np.log(np.array([x], dtype=object)) <-> np.array([x.log()], dtype=object)
```
Since there is no .log() method on a Python long, you get the AttributeError.


Question: Normalized Cross-Correlation in Python I have been struggling the last days trying to compute the degrees of freedom of two pair of vectors (x and y) following reference of Chelton (1983) which is: degrees of freedom according to Chelton(1983) and I can't find a proper way to calculate the normalized cross correlation function using np.correlate, I always get an output that it isn't in between -1, 1. Is there any easy way to get the cross correlation function normalized in order to compute the degrees of freedom of two vectors?

Response:
Nice Question. There is no direct way but you can "normalize" the input vectors before using np.correlate like this and reasonable values will be returned within a range of [-1,1]: Here i define the correlation as generally defined in signal processing textbooks.
```latex
c'_{ab}[k] = sum_n a[n] conj(b[n+k])
```
CODE: If a and b are the vectors:
```
a = (a - np.mean(a)) / (np.std(a) * len(a))
b = (b - np.mean(b)) / (np.std(b))
c = np.correlate(a, b, 'full')
```
References: https://docs.scipy.org/doc/numpy/reference/generated/numpy.correlate.html https://en.wikipedia.org/wiki/Cross-correlation


Question:
How to print numpy objects without line breaks I am logging input arguments to a function using
```
logging.debug('Input to this function = %s',
              inspect.getargvalues(inspect.currentframe())[3])
```
But I do not want the line breaks inserted within numpy objects. numpy.set_printoptions(linewidth=np.nan) removes some, but line breaks are still inserted in 2D objects such as
```
array([[ 0.84148239,  0.71467895,  0.00946744,  0.3471317 ],
       [ 0.68041249,  0.20310698,  0.89486761,  0.97799646],
       [ 0.22328803,  0.32401271,  0.96479887,  0.43404245]])
```
I want it to be like this:
```
array([[ 0.84148239,  0.71467895,  0.00946744,  0.3471317 ], [ 0.68041249,  0.20310698,  0.89486761,  0.97799646], [ 0.22328803,  0.32401271,  0.96479887,  0.43404245]])
```
How can I do this? Thanks.

Response:
Given an array x, you can print it without line breaks with,
```
import numpy as np
x_str = np.array_repr(x).replace('\n', '')
print(x_str)
```
or alternatively using the function np.array2string instead of np.array_repr. I'm not sure if there is an easy way to remove newlines from the string representation or numpy arrays. However, it is always possible to remove them after the conversion took place,
```
input_args = inspect.getargvalues(inspect.currentframe())[3]
logging.debug('Input to this function = %s', repr(input_args).replace('\n', ''))
```


Question:
Deal with overflow in exp using numpy Using numpy, I have this definition of a function:
```
def powellBadlyScaled(X):
    f1 = 10**4 * X[0] * X[1] - 1
    f2 = numpy.exp(-numpy.float(X[0])) + numpy.exp(-numpy.float(X[1])) - 1.0001
    return f1 + f2
```
This function is evaluated a huge number of times on an optimization routine. It often raises exception:
```
RuntimeWarning: overflow encountered in exp
```
I understand that operand cannot be stored in allocated space for a float. But how can I overcome the problem?

Response:
You can use the bigfloat package. It supports arbitrary precision floating point operations. http://packages.python.org/bigfloat/
```
import bigfloat
bigfloat.exp(5000,bigfloat.precision(100))
# -> BigFloat.exact('2.9676283840236670689662968052896e+2171', precision=100)
```
Are you using a function optimization framework? They usually implement value boundaries (using penalty terms). Try that. Are the relevant values really that extreme? In optimization it's not uncommon to minimize log(f). (approximate log likelihood etc etc). Are you sure you want to optimize on that exp value and not log(exp(f)) == f. ? Have a look at my answer to this question: logit and inverse logit functions for extreme values Btw, if all you do is minimize powellBadlyScaled(x,y) then the minimum is at x -> + inf and y -> + inf, so no need for numerics.