Question:
Fitting a 2D Gaussian function using scipy.optimize.curve_fit - ValueError and minpack.error I intend to fit a 2D Gaussian function to images showing a laser beam to get its parameters like FWHM and position. So far I tried to understand how to define a 2D Gaussian function in Python and how to pass x and y variables to it. I've written a little script which defines that function, plots it, adds some noise to it and then tries to fit it using curve_fit. Everything seems to work except the last step in which I try to fit my model function to the noisy data. Here is my code:
```
import scipy.optimize as opt
import numpy as np
import pylab as plt

#define model function and pass independant variables x and y as a list
def twoD_Gaussian((x,y), amplitude, xo, yo, sigma_x, sigma_y, theta, offset):
    xo = float(xo)
    yo = float(yo)
    a = (np.cos(theta)**2)/(2*sigma_x**2) + (np.sin(theta)**2)/(2*sigma_y**2)
    b = -(np.sin(2*theta))/(4*sigma_x**2) + (np.sin(2*theta))/(4*sigma_y**2)
    c = (np.sin(theta)**2)/(2*sigma_x**2) + (np.cos(theta)**2)/(2*sigma_y**2)
    return offset + amplitude*np.exp( - (a*((x-xo)**2) + 2*b*(x-xo)*(y-yo) + c*((y-yo)**2)))

# Create x and y indices
x = np.linspace(0, 200, 201)
y = np.linspace(0, 200, 201)
x,y = np.meshgrid(x, y)

#create data
data = twoD_Gaussian((x, y), 3, 100, 100, 20, 40, 0, 10)

# plot twoD_Gaussian data generated above
plt.figure()
plt.imshow(data)
plt.colorbar()

# add some noise to the data and try to fit the data generated beforehand
initial_guess = (3,100,100,20,40,0,10)

data_noisy = data + 0.2*np.random.normal(size=len(x))

popt, pcov = opt.curve_fit(twoD_Gaussian, (x,y), data_noisy, p0 = initial_guess)
```
Here is the error message I get when running the script using winpython 64-bit Python 2.7:
```
ValueError: object too deep for desired array
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Python\WinPython-64bit-2.7.6.2\python-2.7.6.amd64\lib\site-packages\spyderlib\widgets\externalshell\sitecustomize.py", line 540, in runfile
    execfile(filename, namespace)
  File "E:/Work Computer/Software/Python/Fitting scripts/2D Gaussian function fit/2D_Gaussian_LevMarq_v2.py", line 39, in <module>
    popt, pcov = opt.curve_fit(twoD_Gaussian, (x,y), data_noisy, p0 = initial_guess)
  File "C:\Python\WinPython-64bit-2.7.6.2\python-2.7.6.amd64\lib\site-packages\scipy\optimize\minpack.py", line 533, in curve_fit
    res = leastsq(func, p0, args=args, full_output=1, **kw)
  File "C:\Python\WinPython-64bit-2.7.6.2\python-2.7.6.amd64\lib\site-packages\scipy\optimize\minpack.py", line 378, in leastsq
    gtol, maxfev, epsfcn, factor, diag)
minpack.error: Result from function call is not a proper array of floats.
```
What is it that am I doing wrong? Is it how I pass the independent variables to the model function/curve_fit?

Response:
The output of twoD_Gaussian needs to be 1D. What you can do is add a .ravel() onto the end of the last line, like this:
```
def twoD_Gaussian(xy, amplitude, xo, yo, sigma_x, sigma_y, theta, offset):
    x, y = xy
    xo = float(xo)
    yo = float(yo)
    a = (np.cos(theta)**2)/(2*sigma_x**2) + (np.sin(theta)**2)/(2*sigma_y**2)
    b = -(np.sin(2*theta))/(4*sigma_x**2) + (np.sin(2*theta))/(4*sigma_y**2)
    c = (np.sin(theta)**2)/(2*sigma_x**2) + (np.cos(theta)**2)/(2*sigma_y**2)
    g = offset + amplitude*np.exp( - (a*((x-xo)**2) + 2*b*(x-xo)*(y-yo)
                            + c*((y-yo)**2)))
    return g.ravel()
```
You'll obviously need to reshape the output for plotting, e.g:
```
# Create x and y indices
x = np.linspace(0, 200, 201)
y = np.linspace(0, 200, 201)
x, y = np.meshgrid(x, y)

#create data
data = twoD_Gaussian((x, y), 3, 100, 100, 20, 40, 0, 10)

# plot twoD_Gaussian data generated above
plt.figure()
plt.imshow(data.reshape(201, 201))
plt.colorbar()
```
Do the fitting as before:
```
# add some noise to the data and try to fit the data generated beforehand
initial_guess = (3,100,100,20,40,0,10)

data_noisy = data + 0.2*np.random.normal(size=data.shape)

popt, pcov = opt.curve_fit(twoD_Gaussian, (x, y), data_noisy, p0=initial_guess)
```
And plot the results:
```
data_fitted = twoD_Gaussian((x, y), *popt)

fig, ax = plt.subplots(1, 1)
#ax.hold(True) For older versions. This has now been deprecated and later removed
ax.imshow(data_noisy.reshape(201, 201), cmap=plt.cm.jet, origin='lower',
    extent=(x.min(), x.max(), y.min(), y.max()))
ax.contour(x, y, data_fitted.reshape(201, 201), 8, colors='w')
plt.show()
```


Question:
psycopg2: can't adapt type 'numpy.int64' I have a dataframe with the dtypes shown below and I want to insert the dataframe into a postgres DB but it fails due to error can't adapt type 'numpy.int64'
```
id_code               int64
sector              object
created_date         float64
updated_date    float64
```
How can I convert these types to native python types such as from int64 (which is essentially 'numpy.int64') to a classic int that would then be acceptable to postgres via the psycopg2 client.
```
data['id_code'].astype(np.int)  defaults to int64
```
It is nonetheless possible to convert from one numpy type to another (e.g from int to float)
```
data['id_code'].astype(float)
```
changes to
```
dtype: float64
```
The bottomline is that psycopg2 doesn't seem to understand numpy datatypes if any one has ideas how to convert them to classic types that would be helpful. Updated: Insertion to DB
```
def insert_many():
    """Add data to the table."""
    sql_query = """INSERT INTO classification(
                id_code, sector, created_date, updated_date)
                VALUES (%s, %s, %s, %s);"""
    data = pd.read_excel(fh, sheet_name=sheetname)
    data_list = list(data.to_records())

    conn = None
    try:
        conn = psycopg2.connect(db)
        cur = conn.cursor()
        cur.executemany(sql_query, data_list)
        conn.commit()
        cur.close()
    except(Exception, psycopg2.DatabaseError) as error:
        print(error)
    finally:
        if conn is not None:
            conn.close()
```

Response:
Update: If the value includes NaN, it still wrong. It seems that psycopg2 can't explain the np.int64 format, the following method works for me.
```
import numpy as np
from psycopg2.extensions import register_adapter, AsIs
register_adapter(np.int64, AsIs)
```
Original answer: same problem here, successfully solve this problem after I transform series to nd.array and int. you can try as following:
```
data['id_code'].values.astype(int)
```
--


Question:
Estimate Autocorrelation using Python I would like to perform Autocorrelation on the signal shown below. The time between two consecutive points is 2.5ms (or a repetition rate of 400Hz). This is the equation for estimating autoacrrelation that I would like to use (Taken from http://en.wikipedia.org/wiki/Autocorrelation, section Estimation): What is the simplest method of finding the estimated autocorrelation of my data in python? Is there something similar to numpy.correlate that I can use? Or should I just calculate the mean and variance? Edit: With help from unutbu, I have written:
```
from numpy import *
import numpy as N
import pylab as P

fn = 'data.txt'
x = loadtxt(fn,unpack=True,usecols=[1])
time = loadtxt(fn,unpack=True,usecols=[0])

def estimated_autocorrelation(x):
    n = len(x)
    variance = x.var()
    x = x-x.mean()
    r = N.correlate(x, x, mode = 'full')[-n:]
    #assert N.allclose(r, N.array([(x[:n-k]*x[-(n-k):]).sum() for k in range(n)]))
    result = r/(variance*(N.arange(n, 0, -1)))
    return result

P.plot(time,estimated_autocorrelation(x))
P.xlabel('time (s)')
P.ylabel('autocorrelation')
P.show()
```

Response:
I don't think there is a NumPy function for this particular calculation. Here is how I would write it:
```
def estimated_autocorrelation(x):
    """
    http://stackoverflow.com/q/14297012/190597
    http://en.wikipedia.org/wiki/Autocorrelation#Estimation
    """
    n = len(x)
    variance = x.var()
    x = x-x.mean()
    r = np.correlate(x, x, mode = 'full')[-n:]
    assert np.allclose(r, np.array([(x[:n-k]*x[-(n-k):]).sum() for k in range(n)]))
    result = r/(variance*(np.arange(n, 0, -1)))
    return result
```
The assert statement is there to both check the calculation and to document its intent. When you are confident this function is behaving as expected, you can comment-out the assert statement, or run your script with python -O. (The -O flag tells Python to ignore assert statements.)


Question: Open source alternative to MATLAB's fmincon function? [closed] Closed. This question is seeking recommendations for software libraries, tutorials, tools, books, or other off-site resources. It does not meet Stack Overflow guidelines. It is not currently accepting answers. We don’t allow questions seeking recommendations for software libraries, tutorials, tools, books, or other off-site resources. You can edit the question so it can be answered with facts and citations. Closed 4 years ago. Improve this question Is there an open-source alternative to MATLAB's fmincon function for constrained linear optimization? I'm rewriting a MATLAB program to use Python / NumPy / SciPy and this is the only function I haven't found an equivalent to. A NumPy-based solution would be ideal, but any language will do.

Response: Is your problem convex? Linear? Non-linear? I agree that SciPy.optimize will probably do the job, but fmincon is a sort of bazooka for solving optimization problems, and you'll be better off if you can confine it to one of the categories below (in increasing level of difficulty to solve efficiently) Linear Program (LP) Quadratic Program (QP) Convex Quadratically-Constrained Quadratic Program (QCQP) Second Order Cone Program (SOCP) Semidefinite Program (SDP) Non-Linear Convex Problem Non-Convex Problem There are also combinatoric problems such as Mixed-Integer Linear Programs (MILP), but you didn't mention any sort of integrality constraints, suffice to say that they fall into a different class of problems. The CVXOpt package will be of great use to you if your problem is convex. If your problem is not convex, you need to choose between finding a local solution or the global solution. Many convex solvers 'sort of' work in a non-convex domain. Finding a good approximation to the global solution would require some form Simulated Annealing or Genetic Algorithm. Finding the global solution will require an enumeration of all local solutions or a combinatorial strategy such as Branch and Bound.


Question:
Does setting numpy arrays to None free memory? I have hundreds of really larges matrices, like (600, 800) or (3, 600, 800) shape'd ones. Therefore I want to de-allocate the memory used as soon as I don't really need something anymore. I thought:
```
some_matrix = None
```
Should do the job, or is just the reference set to None but somewhere in the Memory the space still allocated? (like preserving the allocated space for some re-initialization of some_matrix in the future) Additionally: sometimes I am slicing through the matrices, calculated something and put the values into a buffer (a list, because it gets appended all the time). So setting a list to None will definitely free the memory, right? Or does some kind of unset() method exist where whole identifiers plus its referenced objects are "deleted"?

Response: You definitely want to have a look at the garbage collection. Unlike some programming language like C/C++ where the programmer has to free dynamically allocated memory by himself when the space is no longer needed, python has a garbage collection. Meaning that python itself frees the memory when necessary. When you use some_matrix = None, you unlink the variable from the memory space; the reference counter is decreased, and if it reaches 0, the garbage collector will free the memory. When you use del some_matrix as suggested by MSeifert, the memory is not freed immediately as opposed to what the answer says. According to python doc, this is what happens: Deletion of a name removes the binding of that name from the local or global namespace What happened under the hood is that the counter of references to the memory space is reduced by 1 independently of assigning None or using del. When this counter reaches 0, the garbage collector will free the memory space in the future. The only difference is that when using del, it is clear from the context that you do not need the name anymore. If you look at the doc of the garbage collection, you will see that you can invoke it by yourself or change some of its parameters.


Question:
Check if two scipy.sparse.csr_matrix are equal I want to check if two csr_matrix are equal. If I do:
```
x.__eq__(y)
```
I get:
```
raise ValueError("The truth value of an array with more than one "
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all().
```
This, However, works well:
```
assert (z in x for z in y)
```
Is there a better way to do it? maybe using some scipy optimized function instead? Thanks so much

Response:
Can we assume they are the same shape?
```

```python
a=sparse.csr_matrix([[0,1],[1,0]])
```

```python
b=sparse.csr_matrix([[0,1],[1,1]])
```

```python
(a!=b).nnz==0
#Output
#False
#```
#This checks the sparsity of the inequality array. It will give you an efficiency warning if you try a==b (at least the 1st time you use it). That's because it has to test all those zeros. It can't take much advantage of the sparsity. You need a relatively recent version to use logical operators like this. Were you trying to use x.__eq__(y) in some if expression, or did you get error from just that expression? In general you probably want to check several parameters first. Same shape, same nnz, same dtype. You need to be careful with floats. For dense arrays np.allclose is a good way of testing equality. And if the sparse arrays aren't too large, that might be good as well
#```
#np.allclose(a.A, b.A)
#```
#allclose uses all(less_equal(abs(x-y), atol + rtol * abs(y))). You can use a-b, but I suspect that this too will give an efficiecy warning.
```


Question:
Filter out nan rows in a specific column
```
df =

Col1 Col2 Col3
1    nan  4
2    5    4
3    3    nan
```
Given the dataframe df, I want to obtain a new dataframe df2 that does not contain nan in the column Col2. This is the expected result: df2 =
```
Col1 Col2 Col3
2    5    4
3    3    nan
```
I know that it's possible to use pandas.isnull and dropna, however how to specify only particular column to which filtering should be applied?

Response:
you can use DataFrame.dropna() method:
```

```python
df.dropna(subset=['Col2'])
#Output
#Col1  Col2  Col3
#1     2   5.0   4.0
#2     3   3.0   NaN
#```
#or (in this case) less idiomatic Series.notnull():
#```
```

```python
df.loc[df.Col2.notnull()]
#Output
#Col1  Col2  Col3
#1     2   5.0   4.0
#2     3   3.0   NaN
#```
#or using DataFrame.query() method:
#```
```

```python
df.query("Col2 == Col2")
#Output
#Col1  Col2  Col3
#1     2   5.0   4.0
#2     3   3.0   NaN
#```
#numexpr solution:
#```
```

```python
import numexpr as ne
```

```python
col = df.Col2
```

```python
df[ne.evaluate("col == col")]
#Output
#Col1  Col2  Col3
#1     2   5.0   4.0
#2     3   3.0   NaN
#```
```


Question:
How to iterate 1d NumPy array with index and value [duplicate] This question already has answers here: Iterate over numpy with index (numpy equivalent of python enumerate) (3 answers) Closed 6 years ago. For python dict, I could use iteritems() to loop through key and value at the same time. But I cannot find such functionality for NumPy array. I have to manually track idx like this:
```
idx = 0
for j in theta:
   some_function(idx,j,theta)
   idx += 1
```
Is there a better way to do this?

Response:
There are a few alternatives. The below assumes you are iterating over a 1d NumPy array. Iterate with range
```
for j in range(theta.shape[0]):  # or range(len(theta))
   some_function(j, theta[j], theta)
```
Note this is the only of the 3 solutions which will work with numba. This is noteworthy since iterating over a NumPy array explicitly is usually only efficient when combined with numba or another means of pre-compilation. Iterate with enumerate
```
for idx, j in enumerate(theta):
   some_function(idx, j, theta)
```
The most efficient of the 3 solutions for 1d arrays. See benchmarking below. Iterate with np.ndenumerate
```
for idx, j in np.ndenumerate(theta):
   some_function(idx[0], j, theta)
```
Notice the additional indexing step in idx[0]. This is necessary since the index (like shape) of a 1d NumPy array is given as a singleton tuple. For a 1d array, np.ndenumerate is inefficient; its benefits only show for multi-dimensional arrays. Performance benchmarking
```
# Python 3.7, NumPy 1.14.3

np.random.seed(0)

arr = np.random.random(10**6)

def enumerater(arr):
    for index, value in enumerate(arr):
        index, value
        pass

def ranger(arr):
    for index in range(len(arr)):
        index, arr[index]
        pass

def ndenumerater(arr):
    for index, value in np.ndenumerate(arr):
        index[0], value
        pass

%timeit enumerater(arr)    # 131 ms
%timeit ranger(arr)        # 171 ms
%timeit ndenumerater(arr)  # 579 ms
```


Question: why an extra comma in the shape of a single index numpy array A numpy array a a = numpy.arange(12) has shape a.shape = (12,) Why do we need the comma? is shape (12) reserved for something else?

Response:
The reason we don't use (12) for a one-element tuple (like [12] for one-element list) is that round parentheses also appear in formulas. E.g., in x = 2*(5+7) the part (5+7) is just a number, not a tuple. But what if we actually meant it to be a one-element tuple? The trailing comma is a way to indicate that. Compare:
```

```python
2*(5+7)
#Output
#24
```

```python
2*(5+7,)
#Output
#(12, 12)
#```
#With lists, the trailing comma is not needed although some style guides recommend it for consistency.
#```
```

```python
2*[5+7]
#Output
#[12, 12]
```

```python
2*[5+7,]
#Output
#[12, 12]
#```
```


Question:
How to correctly use scipy's skew and kurtosis functions? The skewness is a parameter to measure the symmetry of a data set and the kurtosis to measure how heavy its tails are compared to a normal distribution, see for example here. scipy.stats provides an easy way to calculate these two quantities, see scipy.stats.kurtosis and scipy.stats.skew. In my understanding, the skewness and kurtosis of a normal distribution should both be 0 using the functions just mentioned. That is, however, not the case with my code:
```
import numpy as np
from scipy.stats import kurtosis
from scipy.stats import skew

x = np.linspace( -5, 5, 1000 )
y = 1./(np.sqrt(2.*np.pi)) * np.exp( -.5*(x)**2  )  # normal distribution

print( 'excess kurtosis of normal distribution (should be 0): {}'.format( kurtosis(y) ))
print( 'skewness of normal distribution (should be 0): {}'.format( skew(y) ))
```
The output is: excess kurtosis of normal distribution (should be 0): -0.307393087742 skewness of normal distribution (should be 0): 1.11082371392 What am I doing wrong ? The versions I am using are
```
python: 2.7.6
scipy : 0.17.1
numpy : 1.12.1
```

Response:
These functions calculate moments of the probability density distribution (that's why it takes only one parameter) and doesn't care about the "functional form" of the values. These are meant for "random datasets" (think of them as measures like mean, standard deviation, variance):
```
import numpy as np
from scipy.stats import kurtosis, skew

x = np.random.normal(0, 2, 10000)   # create random values based on a normal distribution

print( 'excess kurtosis of normal distribution (should be 0): {}'.format( kurtosis(x) ))
print( 'skewness of normal distribution (should be 0): {}'.format( skew(x) ))
```
which gives:
```
excess kurtosis of normal distribution (should be 0): -0.024291887786943356
skewness of normal distribution (should be 0): 0.009666157036010928
```
changing the number of random values increases the accuracy:
```
x = np.random.normal(0, 2, 10000000)
```
Leading to:
```
excess kurtosis of normal distribution (should be 0): -0.00010309478605163847
skewness of normal distribution (should be 0): -0.0006751744848755031
```
In your case the function "assumes" that each value has the same "probability" (because the values are equally distributed and each value occurs only once) so from the point of view of skew and kurtosis it's dealing with a non-gaussian probability density (not sure what exactly this is) which explains why the resulting values aren't even close to 0:
```
import numpy as np
from scipy.stats import kurtosis, skew

x_random = np.random.normal(0, 2, 10000)

x = np.linspace( -5, 5, 10000 )
y = 1./(np.sqrt(2.*np.pi)) * np.exp( -.5*(x)**2  )  # normal distribution

import matplotlib.pyplot as plt

f, (ax1, ax2) = plt.subplots(1, 2)
ax1.hist(x_random, bins='auto')
ax1.set_title('probability density (random)')
ax2.hist(y, bins='auto')
ax2.set_title('(your dataset)')
plt.tight_layout()
```


Question:
Adding a background image to a plot Say I am plotting a set of points with an image as a background. I've used the Lena image in the example:
```
import numpy as np
import matplotlib.pyplot as plt
from scipy.misc import imread

np.random.seed(0)
x = np.random.uniform(0.0,10.0,15)
y = np.random.uniform(0.0,10.0,15)
img = imread("lena.jpg")
plt.scatter(x,y,zorder=1)
plt.imshow(img,zorder=0)
plt.show()
```
This gives me . My question is: How can I specify the corner coordinates of the image in the plot? Let's say I'd like the bottom-left corner to be at x, y = 0.5, 1.0 and the top-right corner to be at x, y = 8.0, 7.0.

Response:
Use the extent keyword of imshow. The order of the argument is [left, right, bottom, top]
```
import numpy as np
import matplotlib.pyplot as plt

np.random.seed(0)
x = np.random.uniform(0.0,10.0,15)
y = np.random.uniform(0.0,10.0,15)

datafile = 'lena.jpg'
img = plt.imread(datafile)
plt.scatter(x,y,zorder=1)
plt.imshow(img, zorder=0, extent=[0.5, 8.0, 1.0, 7.0])
plt.show()
```
For cases where it's desired to have an image in a small area of the scatter plot, change the order of the plots (.imshow then .scatter) and change the extent values.
```py
plt.imshow(img, zorder=0, extent=[3.0, 5.0, 3.0, 4.50])
plt.scatter(x, y, zorder=1)
plt.show()
```


Question: How to multiply a scalar throughout a specific column within a NumPy array? I need to do some analysis on a large dataset from a hydrolgeology field work. I am using NumPy. I want to know how I can: multiply e.g. the 2nd column of my array by a number (e.g. 5.2). And then calculate the cumulative sum of the numbers in that column. As I mentioned I only want to work on a specific column and not the whole array.

Response:
```
you can do this in two simple steps using NumPy:

```python
# multiply column 2 of the 2D array, A, by 5.2
```

```python
A[:,1] *= 5.2
```

```python
# assuming by 'cumulative sum' you meant the 'reduced' sum:
```

```python
A[:,1].sum()
```

```python
# if in fact you want the cumulative sum (ie, returns a new column)
```

```python
# then do this for the second step instead:
```

```python
NP.cumsum(A[:,1])
#Output
#```
#with some mocked data:
#```
```

```python
A = NP.random.rand(8, 5)
```

```python
A
#Output
#  array([[ 0.893,  0.824,  0.438,  0.284,  0.892],
#         [ 0.534,  0.11 ,  0.409,  0.555,  0.96 ],
#         [ 0.671,  0.817,  0.636,  0.522,  0.867],
#         [ 0.752,  0.688,  0.142,  0.793,  0.716],
#         [ 0.276,  0.818,  0.904,  0.767,  0.443],
#         [ 0.57 ,  0.159,  0.144,  0.439,  0.747],
#         [ 0.705,  0.793,  0.575,  0.507,  0.956],
#         [ 0.322,  0.713,  0.963,  0.037,  0.509]])
```

```python
A[:,1] *= 5.2
```

```python
A
#Output
#  array([[ 0.893,  4.287,  0.438,  0.284,  0.892],
#         [ 0.534,  0.571,  0.409,  0.555,  0.96 ],
#         [ 0.671,  4.25 ,  0.636,  0.522,  0.867],
#         [ 0.752,  3.576,  0.142,  0.793,  0.716],
#         [ 0.276,  4.255,  0.904,  0.767,  0.443],
#         [ 0.57 ,  0.827,  0.144,  0.439,  0.747],
#         [ 0.705,  4.122,  0.575,  0.507,  0.956],
#         [ 0.322,  3.71 ,  0.963,  0.037,  0.509]])
```

```python
A[:,1].sum()
#Output
#  25.596156138451427
#```
#just a few simple rules are required to grok element selection (indexing) in NumPy: NumPy, like Python, is 0-based, so eg, the "1" below refers to the second column commas separate the dimensions inside the brackets, so [rows, columns], eg, A[2,3] means the item ("cell") at row three, column four a colon means all of the elements along that dimension, eg, A[:,1] creates a view of A's column 2; A[3,:] refers to the fourth row
```


Question:
Numpy: views vs copy by slicing When I am doing the slicing, an unexpected thing happened that seems the first to be view but the second is copy. First First slice of row, then slice of column. It seems is a view.
```

```python
a = np.arange(12).reshape(3, 4)
```

```python
a[0:3:2, :][:, [0, 2]] = 100
```

```python
a
#Output
#array([[100,   1, 100,   3],
#       [  4,   5,   6,   7],
#       [100,   9, 100,  11]])
#```
#Second But if I first slice of column, then slice of row, it seems a copy:
#```
```

```python
a[:, [0, 2]][0:3:2, :] = 0
```

```python
a
#Output
#array([[100,   1, 100,   3],
#       [  4,   5,   6,   7],
#       [100,   9, 100,  11]])
#```
#I am confused because the two methods finally will cause seem position to change, but why the second actually doesn't change the number?
```

Response:
The accepted answer by John Zwinck is actually false (I just figured this out the hard way!). The problem in the question is a combination of doing "l-value indexing" with numpy's fancy indexing. The following doc explains exactly this case https://scipy-cookbook.readthedocs.io/items/ViewsVsCopies.html in the section "But fancy indexing does seem to return views sometimes, doesn't it?" Edit: To summarize the above link: Whether a view or a copy is created is determined by whether the indexing can be represented as a slice. Exception: If one does "fancy indexing" then always a copy is created. Fancy indexing is something like a[[1,2]]. Exception to the exception: If one does l-value indexing (i.e. the indexing happens left of the = sign), then the rule for when a view or a copy are created doesn't apply anymore (though see below for a further exception). The python interpreter will directly assign the values to the left hand side without creating a copy or a view. To prove that a copy is created in both cases, you can do the operation in two steps:
```

```python
a = np.arange(12).reshape(3, 4)
```

```python
b = a[0:3:2, :][:, [0, 2]]
```

```python
b[:] = 100
```

```python
a
#Output
#array([[ 0,  1,  2,  3],
#       [ 4,  5,  6,  7],
#       [ 8,  9, 10, 11]])
#```
#and
#```
```

```python
b = a[:, [0, 2]][0:3:2, :]
```

```python
b[:] = 0
```

```python
a
#Output
#array([[ 0,  1,  2,  3],
#       [ 4,  5,  6,  7],
#       [ 8,  9, 10, 11]])
#```
#Just as an aside, the question by the original poster is the exact problem stated at the end of the scipy-cookbook link above. There is no solution given in the book. The tricky thing about the question is that there are two indexing operations done in a row. Exception to the exception to the exception: If there are two indexing operations done in a row on the left hand side (as is the case in this question), the direct assignment in l-value indexing only works if the first indexing operation can be represented as a slice. Otherwise a copy has to be created even though it is l-value indexing.
```


Question:
How to convert ndarray to array? I'm using pandas.Series and np.ndarray. The code is like this
```

```python
t
#Output
#array([[ 0.,  0.,  0.],
#       [ 0.,  0.,  0.],
#       [ 0.,  0.,  0.]])
```

```python
pandas.Series(t)
#Output
#Exception: Data must be 1-dimensional
#>>>
#```
#And I trie to convert it into 1-dimensional array:
#```
```

```python
tt = t.reshape((1,-1))
```

```python
tt
#Output
#array([[ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.]])
#```
#tt is still multi-dimensional since there are double '['. So how do I get a really convert ndarray into array? After searching, it says they are the same. However in my situation, they are not working the same.
```

Response:
An alternative is to use np.ravel:
```

```python
np.zeros((3,3)).ravel()
#Output
#array([ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.])
#```
#The importance of ravel over flatten is ravel only copies data if necessary and usually returns a view, while flatten will always return a copy of the data. To use reshape to flatten the array:
#```
#tt = t.reshape(-1)
#```
```


Question:
Indexing over the last axis when you don't know the rank in advance How can I index the last axis of a Numpy array if I don't know its rank in advance? Here is what I want to do: Let a be a Numpy array of unknown rank. I want the slice of the last k elements of the last axis. If a is 1D, I want
```
b = a[-k:]
```
If a is 2D, I want
```
b = a[:, -k:]
```
If a is 3D, I want
```
b = a[:, :, -k:]
```
and so on. I want this to work regardless of the rank of a (as long as the rank is at least 1). The fact that I want the last k elements in the example is irrelevant of course, the point is that I want to specify indices for whatever the last axis is when I don't know the rank of an array in advance.

Response:
```
b = a[..., -k:]
```
This is mentioned in the docs.


Question:
in operator, float("NaN") and np.nan I used to believe that in operator in Python checks the presence of element in some collection using equality checking ==, so element in some_list is roughly equivalent to any(x == element for x in some_list). For example:
```
True in [1, 2, 3]
# True because True == 1
```
or
```
1 in [1., 2., 3.]
# also True because 1 == 1.
```
However, it is well-known that NaN is not equal to itself. So I expected that float("NaN") in [float("NaN")] is False. And it is False indeed. However, if we use numpy.nan instead of float("NaN"), the situation is quite different:
```
import numpy as np
np.nan in [np.nan, 1, 2]
# True
```
But np.nan == np.nan still gives False! How is it possible? What's the difference between np.nan and float("NaN")? How does in deal with np.nan?

Response: To check if the item is in the list, Python tests for object identity first, and then tests for equality only if the objects are different.1 float("NaN") in [float("NaN")] is False because two different NaN objects are involved in the comparison. The test for identity therefore returns False, and then the test for equality also returns False since NaN != NaN. np.nan in [np.nan, 1, 2] however is True because the same NaN object is involved in the comparison. The test for object identity returns True and so Python immediately recognises the item as being in the list. The __contains__ method (invoked using in) for many of Python's other builtin Container types, such as tuples and sets, is implemented using the same check. 1 At least this is true in CPython. Object identity here means that the objects are found at the same memory address: the contains method for lists is performed using PyObject_RichCompareBool which quickly compares object pointers before a potentially more complicated object comparison. Other Python implementations may differ.


Question:
Populate a Pandas SparseDataFrame from a SciPy Sparse Matrix I noticed Pandas now has support for Sparse Matrices and Arrays. Currently, I create DataFrame()s like this:
```
return DataFrame(matrix.toarray(), columns=features, index=observations)
```
Is there a way to create a SparseDataFrame() with a scipy.sparse.csc_matrix() or csr_matrix()? Converting to dense format kills RAM badly. Thanks!

Response:
A direct conversion is not supported ATM. Contributions are welcome! Try this, should be ok on memory as the SpareSeries is much like a csc_matrix (for 1 column) and pretty space efficient
```

```python
col = np.array([0,0,1,2,2,2])
```

```python
data = np.array([1,2,3,4,5,6],dtype='float64')
```

```python
m = csc_matrix( (data,(row,col)), shape=(3,3) )
```

```python
m
#Output
#<3x3 sparse matrix of type '<type 'numpy.float64'>'
#        with 6 stored elements in Compressed Sparse Column format>
```

```python
pd.SparseDataFrame([ pd.SparseSeries(m[i].toarray().ravel())
                              for i in np.arange(m.shape[0]) ])
#Output
#0  1  2
#0  1  0  4
#1  0  0  5
#2  2  3  6
```

```python
df = pd.SparseDataFrame([ pd.SparseSeries(m[i].toarray().ravel())
                                   for i in np.arange(m.shape[0]) ])
```

```python
type(df)
#Output
#pandas.sparse.frame.SparseDataFrame
#```
```


Question:
Flipping zeroes and ones in one-dimensional NumPy array I have a one-dimensional NumPy array that consists of zeroes and ones like so:
```
array([0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1])
```
I'd like a quick way to just "flip" the values such that zeroes become ones, and ones become zeroes, resulting in a NumPy array like this:
```
array([1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0])
```
Is there an easy one-liner for this? I looked at the fliplr() function, but this seems to require NumPy arrays of dimensions two or greater. I'm sure there's a fairly simple answer, but any help would be appreciated.

Response:
There must be something in your Q that i do not understand... Anyway
```

```python
from numpy import array
```

```python
a = array((1,0,0,1,1,0,0))
```

```python
b = 1-a
```

```python
print a ; print b
[1 0 0 1 1 0 0]
[0 1 1 0 0 1 1]
```

```python
```
```


Question:
How can I draw scatter trend line? I want to draw a scatter trend line on matplot. How can I do that? Python
```
import pandas as pd
import matplotlib.pyplot as plt
csv = pd.read_csv('/tmp/test.csv')
data = csv[['fee', 'time']]
x = data['fee']
y = data['time']
plt.scatter(x, y)
plt.show()
```
CSV
```
fee,time
100,650
90,700
80,860
70,800
60,1000
50,1200
```
time is integer value. Scatter chart

Response:
I'm sorry I found the answer by myself. How to add trendline in python matplotlib dot (scatter) graphs? Python
```
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
csv = pd.read_csv('/tmp/test.csv')
data = csv[['fee', 'time']]
x = data['fee']
y = data['time']
plt.scatter(x, y)

z = np.polyfit(x, y, 1)
p = np.poly1d(z)
plt.plot(x,p(x),"r--")

plt.show()
```
Chart


Question:
Python out of memory on large CSV file (numpy) I have a 3GB CSV file that I try to read with python, I need the median column wise.
```
from numpy import *
def data():
    return genfromtxt('All.csv',delimiter=',')

data = data() # This is where it fails already.

med = zeros(len(data[0]))
data = data.T
for i in xrange(len(data)):
    m = median(data[i])
    med[i] = 1.0/float(m)
print med
```
The error that I get is this:
```
Python(1545) malloc: *** mmap(size=16777216) failed (error code=12)

*** error: can't allocate region

*** set a breakpoint in malloc_error_break to debug

Traceback (most recent call last):

  File "Normalize.py", line 40, in <module>

  data = data()

  File "Normalize.py", line 39, in data

  return genfromtxt('All.csv',delimiter=',')

File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-
packages/numpy/lib/npyio.py", line 1495, in genfromtxt

for (i, line) in enumerate(itertools.chain([first_line, ], fhd)):

MemoryError
```
I think it's just an out of memory error. I am running a 64bit MacOSX with 4GB of ram and both numpy and Python compiled in 64bit mode. How do I fix this? Should I try a distributed approach, just for the memory management? Thanks EDIT: Also tried with this but no luck...
```
genfromtxt('All.csv',delimiter=',', dtype=float16)
```

Response:
As other folks have mentioned, for a really large file, you're better off iterating. However, you do commonly want the entire thing in memory for various reasons. genfromtxt is much less efficient than loadtxt (though it handles missing data, whereas loadtxt is more "lean and mean", which is why the two functions co-exist). If your data is very regular (e.g. just simple delimited rows of all the same type), you can also improve on either by using numpy.fromiter. If you have enough ram, consider using np.loadtxt('yourfile.txt', delimiter=',') (You may also need to specify skiprows if you have a header on the file.) As a quick comparison, loading ~500MB text file with loadtxt uses ~900MB of ram at peak usage, while loading the same file with genfromtxt uses ~2.5GB. Loadtxt Genfromtxt Alternately, consider something like the following. It will only work for very simple, regular data, but it's quite fast. (loadtxt and genfromtxt do a lot of guessing and error-checking. If your data is very simple and regular, you can improve on them greatly.)
```
import numpy as np

def generate_text_file(length=1e6, ncols=20):
    data = np.random.random((length, ncols))
    np.savetxt('large_text_file.csv', data, delimiter=',')

def iter_loadtxt(filename, delimiter=',', skiprows=0, dtype=float):
    def iter_func():
        with open(filename, 'r') as infile:
            for _ in range(skiprows):
                next(infile)
            for line in infile:
                line = line.rstrip().split(delimiter)
                for item in line:
                    yield dtype(item)
        iter_loadtxt.rowlength = len(line)

    data = np.fromiter(iter_func(), dtype=dtype)
    data = data.reshape((-1, iter_loadtxt.rowlength))
    return data

#generate_text_file()
data = iter_loadtxt('large_text_file.csv')
```
Fromiter


Question:
How to obtain a gaussian filter in python I am using python to create a gaussian filter of size 5x5. I saw this post here where they talk about a similar thing but I didn't find the exact way to get equivalent python code to matlab function fspecial('gaussian', f_wid, sigma) Is there any other way to do it? I tried using the following code :
```
size = 2
sizey = None
size = int(size)
if not sizey:
    sizey = size
else:
    sizey = int(sizey)
x, y = scipy.mgrid[-size: size + 1, -sizey: sizey + 1]
g = scipy.exp(- (x ** 2/float(size) + y ** 2 / float(sizey)))
print g / np.sqrt(2 * np.pi)
```
The output obtained is
```
[[ 0.00730688  0.03274718  0.05399097  0.03274718  0.00730688]
 [ 0.03274718  0.14676266  0.24197072  0.14676266  0.03274718]
 [ 0.05399097  0.24197072  0.39894228  0.24197072  0.05399097]
 [ 0.03274718  0.14676266  0.24197072  0.14676266  0.03274718]
 [ 0.00730688  0.03274718  0.05399097  0.03274718  0.00730688]]
```
What I want is something like this:
```
0.0029690   0.0133062   0.0219382   0.0133062   0.0029690
   0.0133062   0.0596343   0.0983203   0.0596343   0.0133062
   0.0219382   0.0983203   0.1621028   0.0983203   0.0219382
   0.0133062   0.0596343   0.0983203   0.0596343   0.0133062
   0.0029690   0.0133062   0.0219382   0.0133062   0.0029690
```

Response:
In general terms if you really care about getting the the exact same result as MATLAB, the easiest way to achieve this is often by looking directly at the source of the MATLAB function. In this case, edit fspecial:
```
...
  case 'gaussian' % Gaussian filter

     siz   = (p2-1)/2;
     std   = p3;

     [x,y] = meshgrid(-siz(2):siz(2),-siz(1):siz(1));
     arg   = -(x.*x + y.*y)/(2*std*std);

     h     = exp(arg);
     h(h<eps*max(h(:))) = 0;

     sumh = sum(h(:));
     if sumh ~= 0,
       h  = h/sumh;
     end;
...
```
Pretty simple, eh? It's <10mins work to port this to Python:
```
import numpy as np

def matlab_style_gauss2D(shape=(3,3),sigma=0.5):
    """
    2D gaussian mask - should give the same result as MATLAB's
    fspecial('gaussian',[shape],[sigma])
    """
    m,n = [(ss-1.)/2. for ss in shape]
    y,x = np.ogrid[-m:m+1,-n:n+1]
    h = np.exp( -(x*x + y*y) / (2.*sigma*sigma) )
    h[ h < np.finfo(h.dtype).eps*h.max() ] = 0
    sumh = h.sum()
    if sumh != 0:
        h /= sumh
    return h
```
This gives me the same answer as fspecial to within rounding error:
```
>> fspecial('gaussian',5,1)

 0.002969     0.013306     0.021938     0.013306     0.002969
 0.013306     0.059634      0.09832     0.059634     0.013306
 0.021938      0.09832       0.1621      0.09832     0.021938
 0.013306     0.059634      0.09832     0.059634     0.013306
 0.002969     0.013306     0.021938     0.013306     0.002969

 : matlab_style_gauss2D((5,5),1)

array([[ 0.002969,  0.013306,  0.021938,  0.013306,  0.002969],
       [ 0.013306,  0.059634,  0.09832 ,  0.059634,  0.013306],
       [ 0.021938,  0.09832 ,  0.162103,  0.09832 ,  0.021938],
       [ 0.013306,  0.059634,  0.09832 ,  0.059634,  0.013306],
       [ 0.002969,  0.013306,  0.021938,  0.013306,  0.002969]])
```


Question:
numpy corrcoef - compute correlation matrix while ignoring missing data I am trying to compute a correlation matrix of several values. These values include some 'nan' values. I'm using numpy.corrcoef. For element(i,j) of the output correlation matrix I'd like to have the correlation calculated using all values that exist for both variable i and variable j. This is what I have now:
```

```python
df_counties = pd.read_sql("SELECT Median_Age, Rpercent_2008, overall_LS, population_density FROM countyVotingSM2", db_eng)
```

```python
np.corrcoef(df_counties, rowvar = False)
#Output
#array([[ 1.        ,         nan,         nan, -0.10998411],
#       [        nan,         nan,         nan,         nan],
#       [        nan,         nan,         nan,         nan],
#       [-0.10998411,         nan,         nan,  1.        ]])
#```
#Too many nan's :(
```

Response:
One of the main features of pandas is being NaN friendly. To calculate correlation matrix, simply call df_counties.corr(). Below is an example to demonstrate df.corr() is NaN tolerant whereas np.corrcoef is not.
```
import pandas as pd
import numpy as np

# data
# ==============================
np.random.seed(0)
df = pd.DataFrame(np.random.randn(100,5), columns=list('ABCDE'))
df[df < 0] = np.nan
df

         A       B       C       D       E
0   1.7641  0.4002  0.9787  2.2409  1.8676
1      NaN  0.9501     NaN     NaN  0.4106
2   0.1440  1.4543  0.7610  0.1217  0.4439
3   0.3337  1.4941     NaN  0.3131     NaN
4      NaN  0.6536  0.8644     NaN  2.2698
5      NaN  0.0458     NaN  1.5328  1.4694
6   0.1549  0.3782     NaN     NaN     NaN
7   0.1563  1.2303  1.2024     NaN     NaN
8      NaN     NaN     NaN  1.9508     NaN
9      NaN     NaN  0.7775     NaN     NaN
..     ...     ...     ...     ...     ...
90     NaN  0.8202  0.4631  0.2791  0.3389
91  2.0210     NaN     NaN  0.1993     NaN
92     NaN     NaN     NaN  0.1813     NaN
93  2.4125     NaN     NaN     NaN  0.2515
94     NaN     NaN     NaN     NaN  1.7389
95  0.9944  1.3191     NaN  1.1286  0.4960
96  0.7714  1.0294     NaN     NaN  0.8626
97     NaN  1.5133  0.5531     NaN  0.2205
98     NaN     NaN  1.1003  1.2980  2.6962
99     NaN     NaN     NaN     NaN     NaN

[100 rows x 5 columns]

# calculations
# ================================
df.corr()

        A       B       C       D       E
A  1.0000  0.2718  0.2678  0.2822  0.1016
B  0.2718  1.0000 -0.0692  0.1736 -0.1432
C  0.2678 -0.0692  1.0000 -0.3392  0.0012
D  0.2822  0.1736 -0.3392  1.0000  0.1562
E  0.1016 -0.1432  0.0012  0.1562  1.0000

np.corrcoef(df, rowvar=False)

array([[ nan,  nan,  nan,  nan,  nan],
       [ nan,  nan,  nan,  nan,  nan],
       [ nan,  nan,  nan,  nan,  nan],
       [ nan,  nan,  nan,  nan,  nan],
       [ nan,  nan,  nan,  nan,  nan]])
```


Question:
Shuffling NumPy array along a given axis Given the following NumPy array,
```
> a = array([[1, 2, 3, 4, 5], [1, 2, 3, 4, 5],[1, 2, 3, 4, 5]])
```
it's simple enough to shuffle a single row,
```
> shuffle(a[0])
> a
array([[4, 2, 1, 3, 5],[1, 2, 3, 4, 5],[1, 2, 3, 4, 5]])
```
Is it possible to use indexing notation to shuffle each of the rows independently? Or do you have to iterate over the array. I had in mind something like,
```
> numpy.shuffle(a[:])
> a
array([[4, 2, 3, 5, 1],[3, 1, 4, 5, 2],[4, 2, 1, 3, 5]]) # Not the real output
```
though this clearly doesn't work.

Response:
Vectorized solution with rand+argsort trick We could generate unique indices along the specified axis and index into the the input array with advanced-indexing. To generate the unique indices, we would use random float generation + sort trick, thus giving us a vectorized solution. We would also generalize it to cover generic n-dim arrays and along generic axes with np.take_along_axis. The final implementation would look something like this -
```
def shuffle_along_axis(a, axis):
    idx = np.random.rand(*a.shape).argsort(axis=axis)
    return np.take_along_axis(a,idx,axis=axis)
```
Note that this shuffle won't be in-place and returns a shuffled copy. Sample run -
```

```python
a
#Output
#array([[18, 95, 45, 33],
#       [40, 78, 31, 52],
#       [75, 49, 42, 94]])
```

```python
shuffle_along_axis(a, axis=0)
#Output
#array([[75, 78, 42, 94],
#       [40, 49, 45, 52],
#       [18, 95, 31, 33]])
```

```python
shuffle_along_axis(a, axis=1)
#Output
#array([[45, 18, 33, 95],
#       [31, 78, 52, 40],
#       [42, 75, 94, 49]])
#```
```


Question:
NumPy: difference between linalg.eig() and linalg.eigh() In a Python 3 application I'm using NumPy to calculate eigenvalues and eigenvectors of a symmetric real matrix. Here's my demo code:
```
import numpy as np
a = np.random.rand(3,3)  # generate a random array shaped (3,3)

a = (a + a.T)/2  # a becomes a random simmetric matrix

evalues1, evectors1 = np.linalg.eig(a)

evalues2, evectors2 = np.linalg.eigh(a)
```
Except for the signs, I got the same eigenvectors and eigenvalues using np.linalg.eig and np.linalg.eigh. So, what's the difference between the two methods? Thanks EDIT: I've read the docs here https://docs.scipy.org/doc/numpy/reference/generated/numpy.linalg.eig.html and here https://docs.scipy.org/doc/numpy/reference/generated/numpy.linalg.eigh.html but still I can not understand why I should use eigh() when I have a symmetric array.

Response: eigh guarantees you that the eigenvalues are sorted and uses a faster algorithm that takes advantage of the fact that the matrix is symmetric. If you know that your matrix is symmetric, use this function. Attention, eigh doesn't check if your matrix is indeed symmetric, it by default just takes the lower triangular part of the matrix and assumes that the upper triangular part is defined by the symmetry of the matrix. eig works for general matrices and therefore uses a slower algorithm, you can check that for example with IPythons magic command %timeit. If you test with larger matrices, you will also see that in general the eigenvalues are not sorted here.


Question:
replace zeroes in numpy array with the median value I have a numpy array like this:
```
foo_array = [38,26,14,55,31,0,15,8,0,0,0,18,40,27,3,19,0,49,29,21,5,38,29,17,16]
```
I want to replace all the zeros with the median value of the whole array (where the zero values are not to be included in the calculation of the median) So far I have this going on:
```
foo_array = [38,26,14,55,31,0,15,8,0,0,0,18,40,27,3,19,0,49,29,21,5,38,29,17,16]
foo = np.array(foo_array)
foo = np.sort(foo)
print "foo sorted:",foo
#foo sorted: [ 0  0  0  0  0  3  5  8 14 15 16 17 18 19 21 26 27 29 29 31 38 38 40 49 55]
nonzero_values = foo[0::] > 0
nz_values = foo[nonzero_values]
print "nonzero_values?:",nz_values
#nonzero_values?: [ 3  5  8 14 15 16 17 18 19 21 26 27 29 29 31 38 38 40 49 55]
size = np.size(nz_values)
middle = size / 2
print "median is:",nz_values[middle]
#median is: 26
```
Is there a clever way to achieve this with numpy syntax? Thank you

Response:
This solution takes advantage of numpy.median:
```
import numpy as np
foo_array = [38,26,14,55,31,0,15,8,0,0,0,18,40,27,3,19,0,49,29,21,5,38,29,17,16]
foo = np.array(foo_array)
# Compute the median of the non-zero elements
m = np.median(foo[foo > 0])
# Assign the median to the zero elements
foo[foo == 0] = m
```
Just a note of caution, the median for your array (with no zeroes) is 23.5 but as written this sticks in 23.


Question: Saving an imshow-like image while preserving resolution I have an (n, m) array that I've been visualizing with matplotlib.pyplot.imshow. I'd like to save this data in some type of raster graphics file (e.g. a png) so that: The colors are the ones shown with imshow Each element of the underlying array is exactly one pixel in the saved image -- meaning that if the underlying array is (n, m) elements, the image is NxM pixels. (I'm not interested in interpolation='nearest' in imshow.) There is nothing in the saved image except for the pixels corresponding to the data in the array. (I.e. there's no white space around the edges, axes, etc.) How can I do this? I've seen some code that can kind of do this by using interpolation='nearest' and forcing matplotlib to (grudgingly) turn off axes, whitespace, etc. However, there must be some way to do this more directly -- maybe with PIL? After all, I have the underlying data. If I can get an RGB value for each element of the underlying array, then I can save it with PIL. Is there some way to extract the RGB data from imshow? I can write my own code to map the array values to RGB values, but I don't want to reinvent the wheel, since that functionality already exists in matplotlib.

Response:
As you already guessed there is no need to create a figure. You basically need three steps. Normalize your data, apply the colormap, save the image. matplotlib provides all the necessary functionality:
```
import numpy as np
import matplotlib.pyplot as plt

# some data (512x512)
import scipy.misc
data = scipy.misc.lena()

# a colormap and a normalization instance
cmap = plt.cm.jet
norm = plt.Normalize(vmin=data.min(), vmax=data.max())

# map the normalized data to colors
# image is now RGBA (512x512x4)
image = cmap(norm(data))

# save the image
plt.imsave('test.png', image)
```
While the code above explains the single steps, you can also let imsave do all three steps (similar to imshow):
```
plt.imsave('test.png', data, cmap=cmap)
```
Result (test.png):


Question:
Change the values of a NumPy array that are NOT in a list of indices I have a NumPy array like:
```
a = np.arange(30)
```
I know that I can replace the values located at positions indices=[2,3,4] using for instance fancy indexing:
```
a[indices] = 999
```
But how to replace the values at the positions that are not in indices? Would be something like below?
```
a[ not in indices ] = 888
```

Response:
I don't know of a clean way to do something like this:
```
mask = np.ones(a.shape,dtype=bool) #np.ones_like(a,dtype=bool)
mask[indices] = False
a[~mask] = 999
a[mask] = 888
```
Of course, if you prefer to use the numpy data-type, you could use dtype=np.bool_ -- There won't be any difference in the output. it's just a matter of preference really.


Question:
What is the way data is stored in *.npy? I'm saving NumPy arrays using numpy.save function. I want other developers to have capability to read data from those file using C language. So I need to know,how numpy organizes binary data in file.OK, it's obvious when I'm saving array of 'i4' but what about array of arrays that contains some structures?Can't find any info in documentation UPD : lets say tha data is something like :
```
dt = np.dtype([('outer','(3,)<i4'),('outer2',[('inner','(10,)<i4'),('inner2','f8')])])
```
UPD2 : What about saving "dynamic" data (dtype - object)
```
import numpy as np
a = [0,0,0]
b = [0,0]
c = [a,b]
dtype = np.dtype([('Name', '|S2'), ('objValue', object)])
data = np.zeros(3, dtype)
data[0]['objValue'] = a
data[1]['objValue'] = b
data[2]['objValue'] = c
data[0]['Name'] = 'a'
data[1]['Name'] = 'b'
data[2]['Name'] = 'c'

np.save(r'D:\in.npy', data)
```
Is it real to read that thing from C?

Response:
The npy file format is documented in numpy's NEP 1 — A Simple File Format for NumPy Arrays. For instance, the code
```

```python
dt=numpy.dtype([('outer','(3,)<i4'),
                ('outer2',[('inner','(10,)<i4'),('inner2','f8')])])
```

```python
a=numpy.array([((1,2,3),((10,11,12,13,14,15,16,17,18,19),3.14)),
               ((4,5,6),((-1,-2,-3,-4,-5,-6,-7,-8,-9,-20),6.28))],dt)
```

```python
numpy.save('1.npy', a)
#Output
#```
#results in the file:
#```
#93 4E 55 4D 50 59                      magic ("\x93NUMPY")
#01                                     major version (1)
#00                                     minor version (0)
```

96 00                                  HEADER_LEN (0x0096 = 150)
7B 27 64 65 73 63 72 27
3A 20 5B 28 27 6F 75 74
65 72 27 2C 20 27 3C 69
34 27 2C 20 28 33 2C 29
29 2C 20 28 27 6F 75 74
65 72 32 27 2C 20 5B 28
27 69 6E 6E 65 72 27 2C
20 27 3C 69 34 27 2C 20
28 31 30 2C 29 29 2C 20
28 27 69 6E 6E 65 72 32                Header, describing the data structure
27 2C 20 27 3C 66 38 27                "{'descr': [('outer', '<i4', (3,)),
29 5D 29 5D 2C 20 27 66                            ('outer2', [
6F 72 74 72 61 6E 5F 6F                               ('inner', '<i4', (10,)),
72 64 65 72 27 3A 20 46                               ('inner2', '<f8')]
61 6C 73 65 2C 20 27 73                            )],
68 61 70 65 27 3A 20 28                  'fortran_order': False,
32 2C 29 2C 20 7D 20 20                  'shape': (2,), }"
20 20 20 20 20 20 20 20
20 20 20 20 20 0A

01 00 00 00 02 00 00 00 03 00 00 00    (1,2,3)
0A 00 00 00 0B 00 00 00 0C 00 00 00
0D 00 00 00 0E 00 00 00 0F 00 00 00
10 00 00 00 11 00 00 00 12 00 00 00
13 00 00 00                            (10,11,12,13,14,15,16,17,18,19)
1F 85 EB 51 B8 1E 09 40                3.14

04 00 00 00 05 00 00 00 06 00 00 00    (4,5,6)
FF FF FF FF FE FF FF FF FD FF FF FF
FC FF FF FF FB FF FF FF FA FF FF FF
F9 FF FF FF F8 FF FF FF F7 FF FF FF
EC FF FF FF                            (-1,-2,-3,-4,-5,-6,-7,-8,-9,-20)
1F 85 EB 51 B8 1E 19 40                6.28
```


Question:
Randomly insert NA's values in a pandas dataframe How can I randomly insert np.nan's in a DataFrame ? Let's say I want 10% null values inside my DataFrame. My data looks like this :
```
df = pd.DataFrame(np.random.randn(5, 3),
                  index=['a', 'b', 'c', 'd', 'e'],
                  columns=['one', 'two', 'three'])

        one       two     three
a  0.695132  1.044791 -1.059536
b -1.075105  0.825776  1.899795
c -0.678980  0.051959 -0.691405
d -0.182928  1.455268 -1.032353
e  0.205094  0.714192 -0.938242
```
Is there an easy way to insert the null values?

Response:
Here's a way to clear exactly 10% of cells (or rather, as close to 10% as can be achieved with the existing data frame's size).
```python
import random
ix = [(row, col) for row in range(df.shape[0]) for col in range(df.shape[1])]
for row, col in random.sample(ix, int(round(.1*len(ix)))):
    df.iat[row, col] = np.nan
```
Here's a way to clear cells independently with a per-cell probability of 10%.
```python
df = df.mask(np.random.random(df.shape) < .1)
```


Question:
What is the difference between np.sum and np.add.reduce? What is the difference between np.sum and np.add.reduce? While the docs are quite explicit: For example, add.reduce() is equivalent to sum(). The performance of the two seems to be quite different: for relatively small array sizes add.reduce is about twice faster.
```
$ python -mtimeit -s"import numpy as np; a = np.random.rand(100); summ=np.sum" "summ(a)"
100000 loops, best of 3: 2.11 usec per loop
$ python -mtimeit -s"import numpy as np; a = np.random.rand(100); summ=np.add.reduce" "summ(a)"
1000000 loops, best of 3: 0.81 usec per loop

$ python -mtimeit -s"import numpy as np; a = np.random.rand(1000); summ=np.sum" "summ(a)"
100000 loops, best of 3: 2.78 usec per loop
$ python -mtimeit -s"import numpy as np; a = np.random.rand(1000); summ=np.add.reduce" "summ(a)"
1000000 loops, best of 3: 1.5 usec per loop
```
For larger array sizes, the difference seems to go away:
```
$ python -mtimeit -s"import numpy as np; a = np.random.rand(10000); summ=np.sum" "summ(a)"
100000 loops, best of 3: 10.7 usec per loop
$ python -mtimeit -s"import numpy as np; a = np.random.rand(10000); summ=np.add.reduce" "summ(a)"
100000 loops, best of 3: 9.2 usec per loop
```

Response:
Short answer: when the argument is a numpy array, np.sum ultimately calls add.reduce to do the work. The overhead of handling its argument and dispatching to add.reduce is why np.sum is slower. Longer answer: np.sum is defined in numpy/core/fromnumeric.py. In the definition of np.sum, you'll see that the work is passed on to _methods._sum. That function, in _methods.py, is simply:
```
def _sum(a, axis=None, dtype=None, out=None, keepdims=False):
    return um.add.reduce(a, axis=axis, dtype=dtype,
                            out=out, keepdims=keepdims)
```
um is the module where the add ufunc is defined.


Question:
How to convert a Numpy 2D array with object dtype to a regular 2D array of floats As part of broader program I am working on, I ended up with object arrays with strings, 3D coordinates and etc all mixed. I know object arrays might not be very favorite in comparison to structured arrays but I am hoping to get around this without changing a lot of codes. Lets assume every row of my array obj_array (with N rows) has format of
```
Single entry/object of obj_array:  ['NAME',[10.0,20.0,30.0],....]
```
Now, I am trying to load this object array and slice the 3D coordinate chunk. Up to here, everything works fine with simply asking lets say for .
```
obj_array[:,[1,2,3]]
```
However the result is also an object array and I will face problem as I want to form a 2D array of floats with:
```
size [N,3] of N rows and 3 entries of X,Y,Z coordinates
```
For now, I am looping over rows and assigning every row to a row of a destination 2D flot array to get around the problem. I am wondering if there is any better way with array conversion tools of numpy ? I tried a few things and could not get around it.
```
Centers   = np.zeros([N,3])

for row in range(obj_array.shape[0]):
    Centers[row,:] = obj_array[row,1]
```
Thanks

Response:
Nasty little problem... I have been fooling around with this toy example:
```

```python
arr = np.array([['one', [1, 2, 3]],['two', [4, 5, 6]]], dtype=np.object)
```

```python
arr
#Output
#array([['one', [1, 2, 3]],
#       ['two', [4, 5, 6]]], dtype=object)
#```
#My first guess was:
#```
```

```python
np.array(arr[:, 1])
#Output
#array([[1, 2, 3], [4, 5, 6]], dtype=object)
#```
#But that keeps the object dtype, so perhaps then:
#```
```

```python
np.array(arr[:, 1], dtype=float)
#Output
#Traceback (most recent call last):
#  File "<stdin>", line 1, in <module>
#ValueError: setting an array element with a sequence.
#```
#You can normally work around this doing the following:
#```
```

```python
np.array(arr[:, 1], dtype=[('', float)]*3).view(float).reshape(-1, 3)
#Output
#Traceback (most recent call last):
#  File "<stdin>", line 1, in <module>
#TypeError: expected a readable buffer object
#```
#Not here though, which was kind of puzzling. Apparently it is the fact that the objects in your array are lists that throws this off, as replacing the lists with tuples works:
#```
```

```python
np.array([tuple(j) for j in arr[:, 1]],
         dtype=[('', float)]*3).view(float).reshape(-1, 3)
#Output
#array([[ 1.,  2.,  3.],
#       [ 4.,  5.,  6.]])
#```
#Since there doesn't seem to be any entirely satisfactory solution, the easiest is probably to go with:
#```
```

```python
np.array(list(arr[:, 1]), dtype=float)
#Output
#array([[ 1.,  2.,  3.],
#       [ 4.,  5.,  6.]])
#```
#Although that will not be very efficient, probably better to go with something like:
#```
```

```python
np.fromiter((tuple(j) for j in arr[:, 1]), dtype=[('', float)]*3,
            count=len(arr)).view(float).reshape(-1, 3)
#Output
#array([[ 1.,  2.,  3.],
#       [ 4.,  5.,  6.]])
#```
```


Question:
Calling statistics functions from Scipy This may well be completely trivial. I want to call the spearmanr function from scipy: (the data are just examples)
```
import scipy
from numpy import *

Len = [2,3,5,7,2]
Pop = [5,2,6,3,2]

a = array(Len)
b = array(Pop)

print scipy.stats.spearmanr(a,b)
```
This generates the error:
```
AttributeError: 'module' object has no attribute 'spearmanr'
```
What am I doing wrong? Thanks

Response: Use import scipy.stats. Then it works. Importing a package does not automatically import all the subpackages/modules. In these cases you have to do this explicitly.


Question:
Numpy random choice of tuples I'm having trouble to create an array of random choices, where a choice is a tuple. I get the error: a must be 1-dimensional Here is an example:
```
choices = ((0,0,0),(255,255,255))
numpy.random.choice(choices,4)
```
Is there any other way to do this? Expected result: a numpy array consiting of 4 elements randomly picked from the choices tuple.
```
((0,0,0),(0,0,0),(255,255,255),(255,255,255))
```

Response:
Use choice to choose the 1dim indices into the array, then index it. In the example you provided, only the number of possible choices affects the nature of the choice, not the actual values (0, 255). Choosing indices is the 1dim problem choice knows how to handle.
```
choices = numpy.array([[0,0,0],[255,255,255]])
idx = numpy.random.choice(len(choices),4)
choices[idx]
```


Question:
Shuffling non-zero elements of each row in an array - Python / NumPy I have a an array that is relatively sparse, and I would like to go through each row and shuffle only the non-zero elements. Example Input:
```
[2,3,1,0]
[0,0,2,1]
```
Example Output:
```
[2,1,3,0]
[0,0,1,2]
```
Note how the zeros have not changed position. To shuffle all elements in each row (including zeros) I can do this:
```
for i in range(len(X)):
    np.random.shuffle(X[i, :])
```
What I tried to do then is this:
```
for i in range(len(X)):
    np.random.shuffle(X[i, np.nonzero(X[i, :])])
```
But it has no effect. I've noticed that the return type of X[i, np.nonzero(X[i, :])] is different from X[i, :] which might be the cause.
```

```python
X[i, np.nonzero(X[i, :])]
#Output
#array([[23,  5, 29, 11, 17]])
```

```python
X[i, :]
#Output
#array([23,  5, 29, 11, 17])
#```
```

Response:
You could use the non-inplace numpy.random.permutation with explicit non-zero indexing:
```

```python
X = np.array([[2,3,1,0], [0,0,2,1]])
```

```python
for i in range(len(X)):
    idx = np.nonzero(X[i])
    X[i][idx] = np.random.permutation(X[i][idx])

```

```python
X
#Output
#array([[3, 2, 1, 0],
#       [0, 0, 2, 1]])
#```
```


Question:
Understanding NumPy's nonzero function I am trying to understand numpy's nonzero function. The following is an example application:
```
import numpy
arr = numpy.array([[1,0],[1,1]])
arr.nonzero()
--> (array([0, 1, 1]), array([0, 0, 1]))
```
I can see that because arr is 2-D, the output of nonzero() is a 2-tuple. However, I do not understand why the number of indices in each element of the tuple exceeds the number of rows/columns of the array. I can see that
```
arr[arr.nonzero()]
--> array([1, 1, 1])
```
But how...?

Response:
Each element of the tuple contains one of the indices for each nonzero value. Therefore, the length of each tuple element is the number of nonzeros in the array. From your example, the indices of the nonzeros are [0, 0], [1, 0], and [1, 1]. The first element of the tuple is the first index for each of the nonzero values: ([0, 1, 1]), and the second element of the tuple is the second index for each of the nonzero values: ([0, 0, 1]). Your second code block just returns the nonzero values of the array (I am not clear from the question if the return value is part of the confusion).
```

```python
arr[arr.nonzero()]
#Output
#array([1, 1, 1])
#```
#This is more clear if we use an example array with other values.
#```
```

```python
arr = numpy.array([[1,0],[2,3]])
```

```python
arr[arr.nonzero()]
#Output
#array([1, 2, 3])
#```
```


Question:
numpy divide along axis Is there a numpy function to divide an array along an axis with elements from another array? For example, suppose I have an array a with shape (l,m,n) and an array b with shape (m,); I'm looking for something equivalent to:
```
def divide_along_axis(a,b,axis=None):
    if axis is None:
        return a/b
    c = a.copy()
    for i, x in enumerate(c.swapaxes(0,axis)):
        x /= b[i]
    return c
```
For example, this is useful when normalizing an array of vectors:
```

```python
a = np.random.randn(4,3)
#Output
#array([[ 1.03116167, -0.60862215, -0.29191449],
#       [-1.27040355,  1.9943905 ,  1.13515384],
#       [-0.47916874,  0.05495749, -0.58450632],
#       [ 2.08792161, -1.35591814, -0.9900364 ]])
```

```python
np.apply_along_axis(np.linalg.norm,1,a)
#Output
#array([ 1.23244853,  2.62299312,  0.75780647,  2.67919815])
```

```python
c = divide_along_axis(a,np.apply_along_axis(np.linalg.norm,1,a),0)
```

```python
np.apply_along_axis(np.linalg.norm,1,c)
#Output
#array([ 1.,  1.,  1.,  1.])
#```
```

Response:
For the specific example you've given: dividing an (l,m,n) array by (m,) you can use np.newaxis:
```
a = np.arange(1,61, dtype=float).reshape((3,4,5)) # Create a 3d array
a.shape                                           # (3,4,5)

b = np.array([1.0, 2.0, 3.0, 4.0])                # Create a 1-d array
b.shape                                           # (4,)

a / b                                             # Gives a ValueError

a / b[:, np.newaxis]                              # The result you want
```
You can read all about the broadcasting rules here. You can also use newaxis more than once if required. (e.g. to divide a shape (3,4,5,6) array by a shape (3,5) array). From my understanding of the docs, using newaxis + broadcasting avoids also any unecessary array copying. Indexing, newaxis etc are described more fully here now. (Documentation reorganised since this answer first posted).


Question:
np.full(size, 0) vs. np.zeros(size) vs. np.empty() If you were to choose one of the following three ways of initializing an array with zeros which one would you choose and why?
```
my_arr_1 = np.full(size, 0)
```
or
```
my_arr_2 = np.zeros(size)
```
or
```
my_arr_3 = np.empty(size)
my_arr_3[:] = 0
```

Response:
I'd use np.zeros, because of its name. I would never use the third idiom because it takes two statements instead of a single expression and it's harder for the NumPy folks to optimize. In fact, in NumPy 1.10, np.zeros is still the fastest option, despite all the optimizations to indexing:
```py

```python
%timeit np.zeros(1e6)
#Output
#1000 loops, best of 3: 804 µs per loop
```

```python
%timeit np.full(1e6, 0)
#Output
#1000 loops, best of 3: 816 µs per loop
```

```python
%timeit a = np.empty(1e6); a[:] = 0
#Output
#1000 loops, best of 3: 919 µs per loop
#```
#Bigger array for comparison with @John Zwinck's results:
#```py
```

```python
%timeit np.zeros(1e8)
#Output
#100000 loops, best of 3: 9.66 µs per loop
```

```python
%timeit np.full(1e8, 0)
#Output
#1 loops, best of 3: 614 ms per loop
```

```python
%timeit a = np.empty(1e8); a[:] = 0
#Output
#1 loops, best of 3: 229 ms per loop
#```
```


Question:
how to modify a 2D numpy array at specific locations without a loop? I have a 2D numpy array and I have a arrays of rows and columns which should be set to a particular value. Lets consider the following example
```
a = array([[1, 2, 3],
            [4, 5, 6],
            [7, 8, 9]])
```
I want to modify entries at rows [0,2] and columns [1,2]. This should result in the following array
```
a = array([[1, 2, 0],
           [4, 5, 0],
           [7, 8, 9]])
```
I did following and it resulted in modifying each sequence of column in every row
```
rows = [0,1]
cols = [2,2]
b=a[numpy.ix_(rows,columns)]
```
It resulted in the following array modifying every column of the specified array
```
array([[1, 0, 0],
       [4, 5, 6],
       [7, 0, 0]])
```
Some one could please let me know how to do it? Thanks a lot EDIT: It is to be noted that rows and columns coincidently happend to be sequentia. The actual point is that these could be arbitrary and in any order. if it is rows = [a,b,c] and cols=[n x z] then I want to modify exactly three elements at locations (a,n),(b,x),(c,z).

Response:
Adding to what others have said, you can modify these elements using fancy indexing as follows:
```

```python
rows = [0,1]
```

```python
cols = [2,2]
```

```python
a = np.arange(1,10).reshape((3,3))
```

```python
a[rows,cols] = 0
```

```python
a
#Output
#array([[1, 2, 0],
#       [4, 5, 0],
#       [7, 8, 9]])
#```
#You might want to read the documentation on indexing multidimensional arrays: http://docs.scipy.org/doc/numpy/user/basics.indexing.html#indexing-multi-dimensional-arrays The key point is: if the index arrays have a matching shape, and there is an index array for each dimension of the array being indexed, the resultant array has the same shape as the index arrays, and the values correspond to the index set for each position in the index arrays. Importantly this also allows you to do things like:
#```
```

```python
a[rows,cols] = np.array([33,77])
```

```python
a
#Output
#array([[ 1,  2, 33],
#       [ 4,  5, 77],
#       [ 7,  8,  9]])
#```
#where you can set each element independently using another array, list or tuple of the same size.
```


Question:
Replace NaN's in NumPy array with closest non-NaN value I have a NumPy array a like the following:
```

```python
str(a)
#Output
#'[        nan         nan         nan  1.44955726  1.44628034  1.44409573\n  1.4408188   1.43657094  1.43171624  1.42649744  1.42200684  1.42117704\n  1.42040255  1.41922908         nan         nan         nan         nan\n         nan         nan]'
#```
#I want to replace each NaN with the closest non-NaN value, so that all of the NaN's at the beginning get set to 1.449... and all of the NaN's at the end get set to 1.419.... I can see how to do this for specific cases like this, but I need to be able to do it generally for any length of array, with any length of NaN's at the beginning and end of the array (there will be no NaN's in the middle of the numbers). Any ideas? I can find the NaN's easily enough with np.isnan(), but I can't work out how to get the closest value to each NaN.
```

Response:
As an alternate solution (this will linearly interpolate for arrays NaNs in the middle, as well):
```
import numpy as np

# Generate data...
data = np.random.random(10)
data[:2] = np.nan
data[-1] = np.nan
data[4:6] = np.nan

print data

# Fill in NaN's...
mask = np.isnan(data)
data[mask] = np.interp(np.flatnonzero(mask), np.flatnonzero(~mask), data[~mask])

print data
```
This yields:
```
[        nan         nan  0.31619306  0.25818765         nan         nan
  0.27410025  0.23347532  0.02418698         nan]

[ 0.31619306  0.31619306  0.31619306  0.25818765  0.26349185  0.26879605
  0.27410025  0.23347532  0.02418698  0.02418698]
```


Question:
fitting exponential decay with no initial guessing Does anyone know a scipy/numpy module which will allow to fit exponential decay to data? Google search returned a few blog posts, for example - http://exnumerus.blogspot.com/2010/04/how-to-fit-exponential-decay-example-in.html , but that solution requires y-offset to be pre-specified, which is not always possible EDIT: curve_fit works, but it can fail quite miserably with no initial guess for parameters, and that is sometimes needed. The code I'm working with is
```
#!/usr/bin/env python
import numpy as np
import scipy as sp
import pylab as pl
from scipy.optimize.minpack import curve_fit

x = np.array([  50.,  110.,  170.,  230.,  290.,  350.,  410.,  470.,
530.,  590.])
y = np.array([ 3173.,  2391.,  1726.,  1388.,  1057.,   786.,   598.,
443.,   339.,   263.])

smoothx = np.linspace(x[0], x[-1], 20)

guess_a, guess_b, guess_c = 4000, -0.005, 100
guess = [guess_a, guess_b, guess_c]

exp_decay = lambda x, A, t, y0: A * np.exp(x * t) + y0

params, cov = curve_fit(exp_decay, x, y, p0=guess)

A, t, y0 = params

print "A = %s\nt = %s\ny0 = %s\n" % (A, t, y0)

pl.clf()
best_fit = lambda x: A * np.exp(t * x) + y0

pl.plot(x, y, 'b.')
pl.plot(smoothx, best_fit(smoothx), 'r-')
pl.show()
```
which works, but if we remove "p0=guess", it fails miserably.

Response:
You have two options: Linearize the system, and fit a line to the log of the data. Use a non-linear solver (e.g. scipy.optimize.curve_fit The first option is by far the fastest and most robust. However, it requires that you know the y-offset a-priori, otherwise it's impossible to linearize the equation. (i.e. y = A * exp(K * t) can be linearized by fitting y = log(A * exp(K * t)) = K * t + log(A), but y = A*exp(K*t) + C can only be linearized by fitting y - C = K*t + log(A), and as y is your independent variable, C must be known beforehand for this to be a linear system. If you use a non-linear method, it's a) not guaranteed to converge and yield a solution, b) will be much slower, c) gives a much poorer estimate of the uncertainty in your parameters, and d) is often much less precise. However, a non-linear method has one huge advantage over a linear inversion: It can solve a non-linear system of equations. In your case, this means that you don't have to know C beforehand. Just to give an example, let's solve for y = A * exp(K * t) with some noisy data using both linear and nonlinear methods:
```
import numpy as np
import matplotlib.pyplot as plt
import scipy as sp
import scipy.optimize

def main():
    # Actual parameters
    A0, K0, C0 = 2.5, -4.0, 2.0

    # Generate some data based on these
    tmin, tmax = 0, 0.5
    num = 20
    t = np.linspace(tmin, tmax, num)
    y = model_func(t, A0, K0, C0)

    # Add noise
    noisy_y = y + 0.5 * (np.random.random(num) - 0.5)

    fig = plt.figure()
    ax1 = fig.add_subplot(2,1,1)
    ax2 = fig.add_subplot(2,1,2)

    # Non-linear Fit
    A, K, C = fit_exp_nonlinear(t, noisy_y)
    fit_y = model_func(t, A, K, C)
    plot(ax1, t, y, noisy_y, fit_y, (A0, K0, C0), (A, K, C0))
    ax1.set_title('Non-linear Fit')

    # Linear Fit (Note that we have to provide the y-offset ("C") value!!
    A, K = fit_exp_linear(t, y, C0)
    fit_y = model_func(t, A, K, C0)
    plot(ax2, t, y, noisy_y, fit_y, (A0, K0, C0), (A, K, 0))
    ax2.set_title('Linear Fit')

    plt.show()

def model_func(t, A, K, C):
    return A * np.exp(K * t) + C

def fit_exp_linear(t, y, C=0):
    y = y - C
    y = np.log(y)
    K, A_log = np.polyfit(t, y, 1)
    A = np.exp(A_log)
    return A, K

def fit_exp_nonlinear(t, y):
    opt_parms, parm_cov = sp.optimize.curve_fit(model_func, t, y, maxfev=1000)
    A, K, C = opt_parms
    return A, K, C

def plot(ax, t, y, noisy_y, fit_y, orig_parms, fit_parms):
    A0, K0, C0 = orig_parms
    A, K, C = fit_parms

    ax.plot(t, y, 'k--',
      label='Actual Function:\n $y = %0.2f e^{%0.2f t} + %0.2f$' % (A0, K0, C0))
    ax.plot(t, fit_y, 'b-',
      label='Fitted Function:\n $y = %0.2f e^{%0.2f t} + %0.2f$' % (A, K, C))
    ax.plot(t, noisy_y, 'ro')
    ax.legend(bbox_to_anchor=(1.05, 1.1), fancybox=True, shadow=True)

if __name__ == '__main__':
    main()
```
Note that the linear solution provides a result much closer to the actual values. However, we have to provide the y-offset value in order to use a linear solution. The non-linear solution doesn't require this a-priori knowledge.


Question:
Middle point of each pair of an numpy.array I have an array of the form:
```py
x = np.array([ 1230.,  1230.,  1227.,  1235.,  1217.,  1153.,  1170.])
```
and I would like to produce another array where the values are the mean of each pair of values within my original array:
```py
xm = np.array([ 1230.,  1228.5,  1231.,  1226.,  1185.,  1161.5])
```
Someone knows the easiest and fast way to do it without using loops?

Response:
Even shorter, slightly sweeter:
```
(x[1:] + x[:-1]) / 2
```
This is faster:
```

```python
python -m timeit -s "import numpy; x = numpy.random.random(1000000)" "x[:-1] + numpy.diff(x)/2"
#Output
#100 loops, best of 3: 6.03 msec per loop
```

```python
python -m timeit -s "import numpy; x = numpy.random.random(1000000)" "(x[1:] + x[:-1]) / 2"
#Output
#100 loops, best of 3: 4.07 msec per loop
#```
#This is perfectly accurate: Consider each element in x[1:] + x[:-1]. So consider x₀ and x₁, the first and second elements. x₀ + x₁ is calculated to perfect precision and then rounded, in accordance to IEEE. It would therefore be the correct answer if that was all that was needed. (x₀ + x₁) / 2 is just half of that value. This can almost always be done by reducing the exponent by one, except in two cases: x₀ + x₁ overflows. This will result in an infinity (of either sign). That's not what is wanted, so the calculation will be wrong. x₀ + x₁ underflows. As the size is reduced, rounding will be perfect and thus the calculation will be correct. In all other cases, the calculation will be correct. Now consider x[:-1] + numpy.diff(x) / 2. This, by inspection of the source, evaluates directly to
#```
#x[:-1] + (x[1:] - x[:-1]) / 2
#```
#and so consider again x₀ and x₁. x₁ - x₀ will have severe "problems" with underflow for many values. This will also lose precision with large cancellations. It's not immediately clear that this doesn't matter if the signs are the same, though, as the error effectively cancels out on addition. What does matter is that rounding occurs. (x₁ - x₀) / 2 will be no less rounded, but then x₀ + (x₁ - x₀) / 2 involves another rounding. This means that errors will creep in. Proof:
#```
#import numpy
```

wins = draws = losses = 0

for _ in range(100000):
    a = numpy.random.random()
    b = numpy.random.random() / 0.146

    x = (a+b)/2
    y = a + (b-a)/2

    error_mine   = (a-x) - (x-b)
    error_theirs = (a-y) - (y-b)

    if x != y:
        if abs(error_mine) < abs(error_theirs):
            wins += 1
        elif abs(error_mine) == abs(error_theirs):
            draws += 1
        else:
            losses += 1
    else:
        draws += 1

wins / 1000
#>>> 12.44

draws / 1000
#>>> 87.56

losses / 1000
#>>> 0.0
```
This shows that for the carefully chosen constant of 1.46, a full 12-13% of answers are wrong with the diff variant! As expected, my version is always right. Now consider underflow. Although my variant has overflow problems, these are much less big a deal than cancellation problems. It should be obvious why the double-rounding from the above logic is very problematic. Proof:
```
...
    a = numpy.random.random()
    b = -numpy.random.random()
...

wins / 1000
#>>> 25.149

draws / 1000
#>>> 74.851

losses / 1000
#>>> 0.0
```
Yeah, it gets 25% wrong! In fact, it doesn't take much pruning to get this up to 50%:
```
...
    a = numpy.random.random()
    b = -a + numpy.random.random()/256
...

wins / 1000
#>>> 49.188

draws / 1000
#>>> 50.812

losses / 1000
#>>> 0.0
```
Well, it's not that bad. It's only ever 1 least-significant-bit off as long as the signs are the same, I think. So there you have it. My answer is the best unless you're finding the average of two values whose sum exceeds 1.7976931348623157e+308 or is smaller than -1.7976931348623157e+308.


Question:
python why use numpy.r_ instead of concatenate In which case using objects like numpy.r_ or numpy.c_ is better (more efficient, more suitable) than using functions like concatenate or vstack for example ? I am trying to understand a code where the programmer wrote something like:
```
return np.r_[0.0, 1d_array, 0.0] == 2
```
where 1d_array is an array whose values can be 0, 1 or 2. Why not using np.concatenate (for example) instead ? Like :
```
return np.concatenate([[0.0], 1d_array, [0.0]]) == 2
```
It is more readable and apparently it does the same thing.

Response:
np.r_ is implemented in the numpy/lib/index_tricks.py file. This is pure Python code, with no special compiled stuff. So it is not going to be any faster than the equivalent written with concatenate, arange and linspace. It's useful only if the notation fits your way of thinking and your needs. In your example it just saves converting the scalars to lists or arrays:
```

```python
np.r_[0.0, np.array([1,2,3,4]), 0.0]
#Output
#array([ 0.,  1.,  2.,  3.,  4.,  0.])
#```
#error with the same arguments:
#```
```

```python
np.concatenate([0.0, np.array([1,2,3,4]), 0.0])
...
ValueError: zero-dimensional arrays cannot be concatenated
```
correct with the added []
```
```

```python
np.concatenate([[0.0], np.array([1,2,3,4]), [0.0]])
#Output
#array([ 0.,  1.,  2.,  3.,  4.,  0.])
#```
#hstack takes care of that by passing all arguments through [atleast_1d(_m) for _m in tup]:
#```
```

```python
np.hstack([0.0, np.array([1,2,3,4]), 0.0])
#Output
#array([ 0.,  1.,  2.,  3.,  4.,  0.])
#```
#So at least in simple cases it is most similar to hstack. But the real usefulness of r_ comes when you want to use ranges
#```
#np.r_[0.0, 1:5, 0.0]
#np.hstack([0.0, np.arange(1,5), 0.0])
#np.r_[0.0, slice(1,5), 0.0]
#```
#r_ lets you use the : syntax that is used in indexing. That's because it is actually an instance of a class that has a __getitem__ method. index_tricks uses this programming trick several times. They've thrown in other bells-n-whistles Using an imaginary step, uses np.linspace to expand the slice rather than np.arange.
#```
#np.r_[-1:1:6j, [0]*3, 5, 6]
#```
#produces:
#```
#array([-1. , -0.6, -0.2,  0.2,  0.6,  1. ,  0. ,  0. ,  0. ,  5. ,  6. ])
#```
#There are more details in the documentation. I did some time tests for many slices in https://stackoverflow.com/a/37625115/901925
```


Question: Efficiently Read last 'n' rows of CSV into DataFrame A few methods to do this: Read the entire CSV and then use df.tail Somehow reverse the file (whats the best way to do this for large files?) and then use nrows argument to read Somehow find the number of rows in the CSV, then use skiprows and read required number of rows. Maybe do chunk read discarding initial chunks (though not sure how this would work) Can it be done in some easier way? If not, which amongst these three should be prefered and why? Possibly related: Efficiently finding the last line in a text file Reading parts of ~13000 row CSV file with pandas read_csv and nrows Not directly related: How to get the last n row of pandas dataframe?

Response:
I don't think pandas offers a way to do this in read_csv. Perhaps the neatest (in one pass) is to use collections.deque:
```
from collections import deque
from StringIO import StringIO

with open(fname, 'r') as f:
    q = deque(f, 2)  # replace 2 with n (lines read at the end)

```python
q
#Output
#deque(['7,8,9\n', '10,11,12'], maxlen=2)
#         # these are the last two lines of my csv
```

```python
pd.read_csv(StringIO(''.join(q)), header=None)
```
Another option worth trying is to get the number of lines in a first pass and then read the file again, skip that number of rows (minus n) using read_csv...
```


Question: Finding which rows have all elements as zeros in a matrix with numpy I have a large numpy matrix M. Some of the rows of the matrix have all of their elements as zero and I need to get the indices of those rows. The naive approach I'm considering is to loop through each row in the matrix and then check each elements. What would be a better and a faster approach to accomplish this using numpy?

Response:
Here's one way. I assume numpy has been imported using import numpy as np.
```

```python
a
#Output
#array([[0, 1, 0],
#       [1, 0, 1],
#       [0, 0, 0],
#       [1, 1, 0],
#       [0, 0, 0]])
```

```python
np.where(~a.any(axis=1))[0]
#Output
#array([2, 4])
#```
#It's a slight variation of this answer: How to check that a matrix contains a zero column? Here's what's going on: The any method returns True if any value in the array is "truthy". Nonzero numbers are considered True, and 0 is considered False. By using the argument axis=1, the method is applied to each row. For the example a, we have:
#```
```

```python
a.any(axis=1)
#Output
#array([ True,  True, False,  True, False], dtype=bool)
#```
#So each value indicates whether the corresponding row contains a nonzero value. The ~ operator is the binary "not" or complement:
#```
```

```python
~a.any(axis=1)
#Output
#array([False, False,  True, False,  True], dtype=bool)
#```
#(An alternative expression that gives the same result is (a == 0).all(axis=1).) To get the row indices, we use the where function. It returns the indices where its argument is True:
#```
```

```python
np.where(~a.any(axis=1))
#Output
#(array([2, 4]),)
#```
#Note that where returned a tuple containing a single array. where works for n-dimensional arrays, so it always returns a tuple. We want the single array in that tuple.
#```
```

```python
np.where(~a.any(axis=1))[0]
#Output
#array([2, 4])
#```
```


Question: Computing the correlation coefficient between two multi-dimensional arrays I have two arrays that have the shapes N X T and M X T. I'd like to compute the correlation coefficient across T between every possible pair of rows n and m (from N and M, respectively). What's the fastest, most pythonic way to do this? (Looping over N and M would seem to me to be neither fast nor pythonic.) I'm expecting the answer to involve numpy and/or scipy. Right now my arrays are numpy arrays, but I'm open to converting them to a different type. I'm expecting my output to be an array with the shape N X M. N.B. When I say "correlation coefficient," I mean the Pearson product-moment correlation coefficient. Here are some things to note: The numpy function correlate requires input arrays to be one-dimensional. The numpy function corrcoef accepts two-dimensional arrays, but they must have the same shape. The scipy.stats function pearsonr requires input arrays to be one-dimensional.

Response:
Correlation (default 'valid' case) between two 2D arrays: You can simply use matrix-multiplication np.dot like so -
```
out = np.dot(arr_one,arr_two.T)
```
Correlation with the default "valid" case between each pairwise row combinations (row1,row2) of the two input arrays would correspond to multiplication result at each (row1,row2) position. Row-wise Correlation Coefficient calculation for two 2D arrays:
```
def corr2_coeff(A, B):
    # Rowwise mean of input arrays & subtract from input arrays themeselves
    A_mA = A - A.mean(1)[:, None]
    B_mB = B - B.mean(1)[:, None]

    # Sum of squares across rows
    ssA = (A_mA**2).sum(1)
    ssB = (B_mB**2).sum(1)

    # Finally get corr coeff
    return np.dot(A_mA, B_mB.T) / np.sqrt(np.dot(ssA[:, None],ssB[None]))
```
This is based upon this solution to How to apply corr2 functions in Multidimentional arrays in MATLAB Benchmarking This section compares runtime performance with the proposed approach against generate_correlation_map & loopy pearsonr based approach listed in the other answer.(taken from the function test_generate_correlation_map() without the value correctness verification code at the end of it). Please note the timings for the proposed approach also include a check at the start to check for equal number of columns in the two input arrays, as also done in that other answer. The runtimes are listed next. Case #1:
```

```python
A = np.random.rand(1000, 100)
```

```python
B = np.random.rand(1000, 100)
```

```python
%timeit corr2_coeff(A, B)
100 loops, best of 3: 15 ms per loop
```

```python
%timeit generate_correlation_map(A, B)
100 loops, best of 3: 19.6 ms per loop
```
Case #2:
```
```

```python
A = np.random.rand(5000, 100)
```

```python
B = np.random.rand(5000, 100)
```

```python
%timeit corr2_coeff(A, B)
1 loops, best of 3: 368 ms per loop
```

```python
%timeit generate_correlation_map(A, B)
1 loops, best of 3: 493 ms per loop
```
Case #3:
```
```

```python
A = np.random.rand(10000, 10)
```

```python
B = np.random.rand(10000, 10)
```

```python
%timeit corr2_coeff(A, B)
1 loops, best of 3: 1.29 s per loop
```

```python
%timeit generate_correlation_map(A, B)
1 loops, best of 3: 1.83 s per loop
```
The other loopy pearsonr based approach seemed too slow, but here are the runtimes for one small datasize -
```
```

```python
A = np.random.rand(1000, 100)
```

```python
B = np.random.rand(1000, 100)
```

```python
%timeit corr2_coeff(A, B)
100 loops, best of 3: 15.3 ms per loop
```

```python
%timeit generate_correlation_map(A, B)
100 loops, best of 3: 19.7 ms per loop
```

```python
%timeit pearsonr_based(A, B)
1 loops, best of 3: 33 s per loop
```
```


Question:
efficient way of removing None's from numpy array Is there an efficient way to remove Nones from numpy arrays and resize the array to its new size? For example, how would you remove the None from this frame without iterating through it in python. I can easily iterate through it but was working on an api call that would be potentially called many times.
```
a = np.array([1,45,23,23,1234,3432,-1232,-34,233,None])
```

Response:
```

```python
a[a != np.array(None)]
#Output
#array([1, 45, 23, 23, 1234, 3432, -1232, -34, 233], dtype=object)
#```
#The above works because a != np.array(None) is a boolean array which maps out non-None values:
#```
```

```python
a != np.array(None)
#Output
#array([ True,  True,  True,  True,  True,  True,  True,  True,  True, False], dtype=bool)
#```
#Selecting elements of an array in this manner is called boolean array indexing.
```


Question:
Exponential curve fitting in SciPy I have two NumPy arrays x and y. When I try to fit my data using exponential function and curve_fit (SciPy) with this simple code
```
#!/usr/bin/env python
from pylab import *
from scipy.optimize import curve_fit

x = np.array([399.75, 989.25, 1578.75, 2168.25, 2757.75, 3347.25, 3936.75, 4526.25, 5115.75, 5705.25])
y = np.array([109,62,39,13,10,4,2,0,1,2])

def func(x, a, b, c, d):
    return a*np.exp(b-c*x)+d

popt, pcov = curve_fit(func, x, y)
```
I get wrong coefficients popt
```
[a,b,c,d] = [1., 1., 1., 24.19999988]
```
What is the problem?

Response:
First comment: since a*exp(b - c*x) = (a*exp(b))*exp(-c*x) = A*exp(-c*x), a or b is redundant. I'll drop b and use:
```
import matplotlib.pyplot as plt

def func(x, a, c, d):
    return a*np.exp(-c*x)+d
```
That isn't the main issue. The problem is simply that curve_fit fails to converge to a solution to this problem when you use the default initial guess (which is all 1s). Check pcov; you'll see that it is inf. This is not surprising, because if c is 1, most of the values of exp(-c*x) underflow to 0:
```

```python
np.exp(-x)
#Output
#array([  2.45912644e-174,   0.00000000e+000,   0.00000000e+000,
#         0.00000000e+000,   0.00000000e+000,   0.00000000e+000,
#         0.00000000e+000,   0.00000000e+000,   0.00000000e+000,
#         0.00000000e+000])
#```
#This suggests that c should be small. A better initial guess is, say, p0 = (1, 1e-6, 1). Then I get:
#```
```

```python
popt, pcov = curve_fit(func, x, y, p0=(1, 1e-6, 1))
```

```python
popt
#Output
#array([  1.63561656e+02,   9.71142196e-04,  -1.16854450e+00])
#```
#This looks reasonable:
#```
```

```python
xx = np.linspace(300, 6000, 1000)
```

```python
yy = func(xx, *popt)
```

```python
plt.plot(x, y, 'ko')
#Output
#[<matplotlib.lines.Line2D at 0x41c5ad0>]
```

```python
plt.plot(xx, yy)
#Output
#[<matplotlib.lines.Line2D at 0x41c5c10>]
#```
```


Question:
check type within numpy array I have different types of data. most of them are int and sometimes float. The int is different in size so 8/ 16/ 32 bits are the sizes. For this situation I'm creating a numerical type converter. therefore i check the type by using isinstance(). This because I have read that isinstance() is less worse than type(). The point is that a lot of data i get is numpy arrays. I use spyder as IDE and then i see by the variables also a type. but when i type isinstance(var,'type i read') i get False. I did some checks:
```
a = 2.17
b = 3
c = np.array(np.random.rand(2, 8))
d = np.array([1])
```
for there isinstance(var,type) i get:
```
isinstance(a, float)
True
isinstance(b, int)
True
isinstance(c, float)  # or isinstance(c, np.float64)
False
isinstance(d, int)  # or isinstance(c, np.int32)
False
```
c and d are True when i ask
```
isinstance(c, np.ndarray)
True
isinstance(d, np.ndarray)
True
```
i can check with step in the ndarray by
```
isinstance(c[i][j], np.float64)
True
isinstance(d[i], np.int32)
True
```
but this means that for every dimension i have to add a new index otherwise it is False again. I can check there type with dtype like c.dtype == 'float64'... Oke so for what i have find and tried... My questions are basically: how is the var.dtype method compared to isinstance() and type() (worst/ better etc)? if var.dtype is even worse as isinstance() is there some method in the isinstance() without all the manual indexing? (autoindexing etc)?

Response:
An array is an object of type np.ndarray. Its values or elements are stored in a data buffer, which can be thought of as a contiguous block of memory bytes. The bytes in the data buffer do not have a type, because they are not Python objects. The array has a dtype parameter, which is used to interpret those bytes. If dtype is int32 (there are various synonyms), 4 bytes are interpreted as an integer. Accessing an element, say c[0] gives a new object that depends on the dtype, e.g. an object type np.int32. c[0].item will give an Python object of the corresponding type:
```

```python
c=np.array([1])
```

```python
c.dtype
#Output
#dtype('int32')
```

```python
type(c)
#Output
#numpy.ndarray
```

```python
type(c[0])
#Output
#numpy.int32
```

```python
c[0].item()
#Output
#1
```

```python
type(c[0].item())
#Output
#int
#```
#(And c[0].dtype is the same as for c.dtype; you don't need to index individual elements of an array to check their dtype). The same 4 bytes of this array can be viewed as dtype int8 - a single byte integer.
#```
```

```python
c.view('b')
#Output
#array([1, 0, 0, 0], dtype=int8)
#```
#A single element of this alternate view is np.int8, but when I take item(), I get a Python integer. There isn't a int8 Python numeric type.
#```
```

```python
type(c.view('b')[0])
#Output
#numpy.int8
```

```python
type(c.view('b')[0].item())
#Output
#int
#```
#A list contains pointers to Python objects, each of which has a type. So does an array of dtype=object. But the common numeric array does not contain Python integers or floats. It has a data buffer that can interpreted in various ways according to the dtype. Python integers don't come in different sizes, at least not to the same extent as numpy dtypes. So the isinstance and type() stuff does not apply to the contents of an ndarray. ==================== From the comments I gather you are trying to convert integer arrays to float. You aren't converting scalars. If so then dtype is all that matters; an array always has a dtype. It's unclear whether you are ok with casting a np.float32 to np.float64. I'd suggest studying, and experimenting with the np.can_cast function and the x.astype method.
#```
#x.astype(np.float64, copy=False)
#```
#for example will convert all int dtypes to float, without copying the ones that are already float64. It may copy and convert np.float32 ones. Look also at the casting parameter of these functions. =========================== I found in scipy.optimize.minimize another testing tool
#```
```

```python
np.typecodes
#Output
#{'All': '?bhilqpBHILQPefdgFDGSUVOMm',
# 'AllFloat': 'efdgFDG',
# 'AllInteger': 'bBhHiIlLqQpP',
# 'Character': 'c',
# 'Complex': 'FDG',
# 'Datetime': 'Mm',
# 'Float': 'efdg',
# 'Integer': 'bhilqp',
# 'UnsignedInteger': 'BHILQP'}
#```
#It can be used to check for integers with:
#```
#if x0.dtype.kind in np.typecodes["AllInteger"]:
#    x0 = np.asarray(x0, dtype=float)
#```
```


Question:
convert binary string to numpy array Assume I have the string:
```
my_data = '\x00\x00\x80?\x00\x00\x00@\x00\x00@@\x00\x00\x80@'
```
Where I got it is irrelevant, but for the sake of having something concrete, assume I read it from a binary file. I know my string is the binary representation of 4 (4-byte) floats. I would like to get those floats as a numpy array. I could do:
```
import struct
import numpy as np
tple = struct.unpack( '4f', my_data )
my_array = np.array( tple, dtype=np.float32 )
```
But it seems silly to create an intermediate tuple. Is there a way to do this operation without creating an intermediate tuple? EDIT I would also like to be able to construct the array in such a way that I can specify the endianness of the string.

Response:
```

```python
np.frombuffer(b'\x00\x00\x80?\x00\x00\x00@\x00\x00@@\x00\x00\x80@', dtype='<f4') # or dtype=np.dtype('<f4'), or np.float32 on a little-endian system (which most computers are these days)
#Output
#array([ 1.,  2.,  3.,  4.], dtype=float32)
#```
#Or, if you want big-endian:
#```
```

```python
np.frombuffer(b'\x00\x00\x80?\x00\x00\x00@\x00\x00@@\x00\x00\x80@', dtype='>f4') # or dtype=np.dtype('>f4'), or np.float32  on a big-endian system
#Output
#array([  4.60060299e-41,   8.96831017e-44,   2.30485571e-41,
#         4.60074312e-41], dtype=float32)
#```
#The b isn't necessary prior to Python 3, of course. In fact, if you actually are using a binary file to load the data from, you could even skip the using-a-string step and load the data directly from the file with numpy.fromfile(). Also, dtype reference, just in case: http://docs.scipy.org/doc/numpy/reference/arrays.dtypes.html
```


Question:
Custom data types in numpy arrays I'm creating a numpy array which is to be filled with objects of a particular class I've made. I'd like to initialize the array such that it will only ever contain objects of that class. For example, here's what I'd like to do, and what happens if I do it.
```
class Kernel:
    pass

```python
L = np.empty(4,dtype=Kernel)
```

TypeError: data type not understood
```
I can do this:
```

```python
L = np.empty(4,dtype=object)
#Output
#```
#and then assign each element of L as a Kernel object (or any other type of object). It would be so neat were I able to have an array of Kernels, though, from both a programming point of view (type checking) and a mathematical one (operations on sets of functions). Is there any way for me to specify the data type of a numpy array using an arbitrary class?
```

Response:
If your Kernel class has a predictable amount of member data, then you could define a dtype for it instead of a class. e.g. if it's parameterized by 9 floats and an int, you could do
```
kerneldt = np.dtype([('myintname', np.int32), ('myfloats', np.float64, 9)])
arr = np.empty(dims, dtype=kerneldt)
```
You'll have to do some coercion to turn them into objects of class Kernel every time you want to manipulate methods of a single kernel but that's one way to store the actual data in a NumPy array. If you want to only store a reference, then the object dtype is the best you can do without subclassing ndarray.