Question:
Distinguishing overfitting vs good prediction These are questions on how to calculate & reduce overfitting in machine learning. I think many new to machine learning will have the same questions, so I tried to be clear with my examples and questions in hope that answers here can help others. I have a very small sample of texts and I'm trying to predict values associated with them. I've used sklearn to calculate tf-idf, and insert those into a regression model for prediction. This gives me 26 samples with 6323 features - not a lot.. I know:
```
>> count_vectorizer = CountVectorizer(min_n=1, max_n=1)
>> term_freq = count_vectorizer.fit_transform(texts)
>> transformer = TfidfTransformer()
>> X = transformer.fit_transform(term_freq)
>> print X.shape

(26, 6323)
```
Inserting those 26 samples of 6323 features (X) and associated scores (y), into a LinearRegression model, gives good predictions. These are obtained using leave-one-out cross validation, from cross_validation.LeaveOneOut(X.shape[0], indices=True) :
```
using ngrams (n=1):
     human  machine  points-off  %error
      8.67    8.27    0.40       1.98
      8.00    7.33    0.67       3.34
      ...     ...     ...        ...
      5.00    6.61    1.61       8.06
      9.00    7.50    1.50       7.50
mean: 7.59    7.64    1.29       6.47
std : 1.94    0.56    1.38       6.91
```
Pretty good! Using ngrams (n=300) instead of unigrams (n=1), similar results occur, which is obviously not right. No 300-words occur in any of the texts, so the prediction should fail, but it doesn't:
```
using ngrams (n=300):
      human  machine  points-off  %error
       8.67    7.55    1.12       5.60
       8.00    7.57    0.43       2.13
       ...     ...     ...        ...
mean:  7.59    7.59    1.52       7.59
std :  1.94    0.08    1.32       6.61
```
Question 1: This might mean that the prediction model is overfitting the data. I only know this because I chose an extreme value for the ngrams (n=300) which I KNOW can't produce good results. But if I didn't have this knowledge, how would you normally tell that the model is over-fitting? In other words, if a reasonable measure (n=1) were used, how would you know that the good prediction was a result of being overfit vs. the model just working well? Question 2: What is the best way of preventing over-fitting (in this situation) to be sure that the prediction results are good or not? Question 3: If LeaveOneOut cross validation is used, how can the model possibly over-fit with good results? Over-fitting means the prediction accuracy will suffer - so why doesn't it suffer on the prediction for the text being left out? The only reason I can think of: in a tf-idf sparse matrix of mainly 0s, there is strong overlap between texts because so many terms are 0s - the regression then thinks the texts correlate highly. Please answer any of the questions even if you don't know them all. Thanks!

Response: how would you normally tell that the model is over-fitting? One useful rule of thumb is that you may be overfitting when your model's performance on its own training set is much better than on its held-out validation set or in a cross-validation setting. That's not all there is to it, though. The blog entry I linked to describes a procedure for testing for overfit: plot training set and validation set error as a function of training set size. If they show a stable gap at the right end of the plot, you're probably overfitting. What is the best way of preventing over-fitting (in this situation) to be sure that the prediction results are good or not? Use a held-out test set. Only do evaluation on this set when you're completely done with model selection (hyperparameter tuning); don't train on it, don't use it in (cross-)validation. The score you get on the test set is the model's final evaluation. This should show whether you've accidentally overfit the validation set(s). [Machine learning conferences are sometimes set up like a competition, where the test set is not given to the researchers until after they've delivered their final model to the organisers. In the meanwhile, they can use the training set as they please, e.g. by testing models using cross-validation. Kaggle does something similar.] If LeaveOneOut cross validation is used, how can the model possibly over-fit with good results? Because you can tune the model as much as you want in this cross-validation setting, until it performs nearly perfectly in CV. As an extreme example, suppose that you've implemented an estimator that is essentially a random number generator. You can keep trying random seeds until you hit a "model" that produces very low error in cross-validation, but that doesn't you've hit the right model. It means you've overfit to the cross-validation. See also this interesting warstory.


Question:
np.random.permutation with seed? I want to use a seed with np.random.permutation, like
```
np.random.permutation(10, seed=42)
```
I get the following error:
```
"permutation() takes no keyword arguments"
```
How can I do that else? Thanks.

Response:
If you want it in one line, you can create a new RandomState, and call the permutation on that:
```
np.random.RandomState(seed=42).permutation(10)
```
This is better than just setting the seed of np.random, as it will have only a localized effect. NumPy 1.16 Update: RandomState is now considered a legacy feature. I see no indication that it will be deprecated any time soon, but now the recommended way to generate reproducible random numbers is via Random Generators, of which the default can be instantiated like so:
```
np.random.default_rng(seed=42).permutation(10)
```
Note that it appears like there's no guarantees of bitstream equivalence across different versions of NumPy for this generator, wheras for the RandomState the documentation states that "This generator is considered frozen and will have no further improvements. It is guaranteed to produce the same values as the final point release of NumPy v1.16."


Question:
Best way to initialize and fill an numpy array? [duplicate] This question already has answers here: Create numpy matrix filled with NaNs (11 answers) After `x = x.y()`, why did `x` become `None` instead of being modified (possibly causing "AttributeError: 'NoneType' object has no attribute")? (1 answer) Closed 1 year ago. I want to initialize and fill a numpy array. What is the best way? This works as I expect:
```

```python
import numpy as np
```

```python
np.empty(3)
#Output
#array([ -1.28822975e-231,  -1.73060252e-077,   2.23946712e-314])
#```
#But this doesn't:
#```
```

```python
np.empty(3).fill(np.nan)
#Output
#>>>
#```
#Nothing?
#```
```

```python
type(np.empty(3))
#Output
#<type 'numpy.ndarray'>
#```
#It seems to me that the np.empty() call is returning the correct type of object, so I don't understand why .fill() is not working? Assigning the result of np.empty() first works fine:
#```
```

```python
a = np.empty(3)
```

```python
a.fill(np.nan)
```

```python
a
#Output
#array([ nan,  nan,  nan])
#```
#Why do I need to assign to a variable in order to use np.fill()? Am I missing a better alternative?
```

Response:
You could also try:
```

```python
np.full(3, np.nan)
#Output
#array([ nan,  nan,  nan])
#```
#The pertinent doc:
#```
#Definition: np.full(shape, fill_value, dtype=None, order='C')
#Docstring:
#Return a new array of given shape and type, filled with `fill_value`.
#```
#Although I think this might be only available in numpy 1.8+
```


Question: How do I convert a numpy matrix into a boolean matrix? I have a n x n matrix in numpy which has 0 and non-0 values. Is there a way to easily convert it to a boolean matrix? Thanks.

Response:
```
numpy.array(old_matrix, dtype=bool)
```
Alternatively,
```
old_matrix != 0
```
The first version is an elementwise coercion to boolean. Analogous constructs will work for conversion to other data types. The second version is an elementwise comparison to 0. It involves less typing, but ran slightly slower when I timed it. Which you use is up to you; I'd probably decide based on whether "convert to boolean" or "compare to 0" is a better conceptual description of what I'm after.


Question:
Python how to combine two matrices in numpy new to Python, struggling in numpy, hope someone can help me, thank you!
```
from numpy  import *
A = matrix('1.0 2.0; 3.0 4.0')
B = matrix('5.0 6.0')
C = matrix('1.0 2.0; 3.0 4.0; 5.0 6.0')
print "A=",A
print "B=",B
print "C=",C
```
results:
```
A= [[ 1.  2.]
   [ 3.  4.]]
B= [[ 5.  6.]]
C= [[ 1.  2.]
   [ 3.  4.]
   [ 5.  6.]]
```
Question: how to use A and B to generate C, like in matlab C=[A;B]?

Response:
Use numpy.concatenate:
```

```python
import numpy as np
```

```python
np.concatenate((A, B))
#Output
#matrix([[ 1.,  2.],
#        [ 3.,  4.],
#        [ 5.,  6.]])
#```
```


Question:
How to find last occurrence of maximum value in a numpy.ndarray I have a numpy.ndarray in which the maximum value will mostly occur more than once. EDIT: This is subtly different from numpy.argmax: how to get the index corresponding to the *last* occurrence, in case of multiple occurrences of the maximum values because the author says Or, even better, is it possible to get a list of indices of all the occurrences of the maximum value in the array? whereas in my case getting such a list may prove very expensive Is it possible to find the index of the last occurrence of the maximum value by using something like numpy.argmax? I want to find only the index of the last occurrence, not an array of all occurrences (since several hundreds may be there) For example this will return the index of the first occurrence ie 2
```
import numpy as np
a=np.array([0,0,4,4,4,4,2,2,2,2])
print np.argmax(a)
```
However I want it to output 5.

Response:
numpy.argmax only returns the index of the first occurrence. You could apply argmax to a reversed view of the array:
```
import numpy as np
a = np.array([0,0,4,4,4,4,2,2,2,2])
b = a[::-1]
i = len(b) - np.argmax(b) - 1
i     # 5
a[i:] # array([4, 2, 2, 2, 2])
```
Note numpy doesn't copy the array but instead creates a view of the original with a stride that accesses it in reverse order.
```
id(a) == id(b.base) # True
```


Question:
Efficiently check if numpy ndarray values are strictly increasing I'm having a numpy ndarray where I would like to check if each row vector is monotonically increasing. Example:
```
a = np.asarray([[1,2,3],[1,5,7],[4,3,6]])
monotonically_increasing(a)
```
Expected return:
```
[True, True, False]
```
I'm not entirely sure how to efficiently do this, since the matrices are expected to be quite large (~1000x1000), and was hoping for some help.

Response:
```

```python
import numpy as np
```

```python
a = np.asarray([[1,2,3],[1,5,7],[4,3,6]])
#Output
#```
#Find the difference between each element. np.diff has an argument that lets you specify the axis to perform the diff
#```
```

```python
np.diff(a)
#Output
#array([[ 1,  1],
#       [ 4,  2],
#       [-1,  3]])
#```
#Check to see if each difference is greater than 0.
#```
```

```python
np.diff(a) > 0
#Output
#array([[ True,  True],
#       [ True,  True],
#       [False,  True]], dtype=bool)
#```
#Check to see if all the differences are > 0
#```
```

```python
np.all(np.diff(a) > 0)
#Output
#False
#>>>
#```
#As suggested by @Jaime - check that each element is greater than the element to its left:
#```
#np.all(a[:, 1:] >= a[:, :-1], axis=1)
#```
#Which appears to be about twice as fast/efficient as my diff solution.
```


Question: how to annotate heatmap with text in matplotlib I am plotting a heatmap in matplotlib using: plt.pcolor(rand(5,5)) how can I annotate the heatmap with the actual numbers plotted? meaning in each cell of the plotted heatmap, put the value corresponding to that cell in the 5x5 matrix passed to pcolor. thanks.

Response:
There is no automatic feature to do such a thing, but you could loop through each point and put text in the appropriate location:
```
import matplotlib.pyplot as plt
import numpy as np

data = np.random.rand(5, 4)
heatmap = plt.pcolor(data)

for y in range(data.shape[0]):
    for x in range(data.shape[1]):
        plt.text(x + 0.5, y + 0.5, '%.4f' % data[y, x],
                 horizontalalignment='center',
                 verticalalignment='center',
                 )

plt.colorbar(heatmap)

plt.show()
```
HTH


Question:
Scipy/Numpy FFT Frequency Analysis I'm looking for how to turn the frequency axis in a fft (taken via scipy.fftpack.fftfreq) into a frequency in Hertz, rather than bins or fractional bins. I tried to code below to test out the FFT:
```
t = scipy.linspace(0,120,4000)
acc = lambda t: 10*scipy.sin(2*pi*2.0*t) + 5*scipy.sin(2*pi*8.0*t) + 2*scipy.random.random(len(t))

signal = acc(t)

FFT = abs(scipy.fft(signal))
FFT = scipy.fftpack.fftshift(FFT)
freqs = scipy.fftpack.fftfreq(signal.size)

pylab.plot(freqs,FFT,'x')
pylab.show()
```
The sampling rate should be 4000 samples / 120 seconds = 33.34 samples/sec. The signal has a 2.0 Hz signal, a 8.0 Hz signal, and some random noise. I take the FFT, grab the frequencies, and plot it. The numbers are pretty nonsensical. If I multiply the frequencies by 33.34 (the sampling frequency), then I get peaks at about 8 Hz and 15 Hz, which seems wrong (also, the frequencies should be a factor of 4 apart, not 2!). Any thoughts on what I'm doing wrong here?

Response:
I think you don't need to do fftshift(), and you can pass sampling period to fftfreq():
```
import scipy
import scipy.fftpack
import pylab
from scipy import pi
t = scipy.linspace(0,120,4000)
acc = lambda t: 10*scipy.sin(2*pi*2.0*t) + 5*scipy.sin(2*pi*8.0*t) + 2*scipy.random.random(len(t))

signal = acc(t)

FFT = abs(scipy.fft(signal))
freqs = scipy.fftpack.fftfreq(signal.size, t[1]-t[0])

pylab.subplot(211)
pylab.plot(t, signal)
pylab.subplot(212)
pylab.plot(freqs,20*scipy.log10(FFT),'x')
pylab.show()
```
from the graph you can see there are two peak at 2Hz and 8Hz.


Question:
Rearranging axes in numpy? I have an ndarray such as
```

```python
arr = np.random.rand(10, 20, 30, 40)
```

```python
arr.shape
#Output
#(10, 20, 30, 40)
#```
#whose axes I would like to swap around into some arbitrary order such as
#```
```

```python
rearranged_arr = np.swapaxes(np.swapaxes(arr, 1,3), 0,1)
```

```python
rearranged_arr.shape
#Output
#(40, 10, 30, 20)
#```
#Is there a function which achieves this without having to chain together a bunch of np.swapaxes?
```

Response:
There are two options: np.moveaxis and np.transpose. np.moveaxis(a, sources, destinations) docs This function can be used to rearrange specific dimensions of an array. For example, to move the 4th dimension to be the 1st and the 2nd dimension to be the last:
```py

```python
rearranged_arr = np.moveaxis(arr, [3, 1], [0, 3])
```

```python
rearranged_arr.shape
#Output
#(40, 10, 30, 20)
#```
#This can be particularly useful if you have many dimensions and only want to rearrange a small number of them. e.g.
#```py
```

```python
another_arr = np.random.rand(0, 1, 2, 3, 4, 5, 6, 7, 8, 9)
```

```python
np.moveaxis(another_arr, [8, 9], [0, 1]).shape
#Output
#(8, 9, 0, 1, 2, 3, 4, 5, 6, 7)
#```
#np.transpose(a, axes=None) docs This function can be used to rearrange all dimensions of an array at once. For example, to solve your particular case:
#```py
```

```python
rearranged_arr = np.transpose(arr, axes=[3, 0, 2, 1])
```

```python
rearranged_arr.shape
#Output
#(40, 10, 30, 20)
#```
#or equivalently
#```py
```

```python
rearranged_arr = arr.transpose(3, 0, 2, 1)
```

```python
rearranged_arr.shape
#Output
#(40, 10, 30, 20)
#```
```


Question: Unpack NumPy array by column If I have a NumPy array, for example 5x3, is there a way to unpack it column by column all at once to pass to a function rather than like this: my_func(arr[:, 0], arr[:, 1], arr[:, 2])? Kind of like *args for list unpacking but by column.

Response:
You can unpack the transpose of the array in order to use the columns for your function arguments:
```
my_func(*arr.T)
```
Here's a simple example:
```

```python
x = np.arange(15).reshape(5, 3)
#Output
#array([[ 0,  5, 10],
#       [ 1,  6, 11],
#       [ 2,  7, 12],
#       [ 3,  8, 13],
#       [ 4,  9, 14]])
#```
#Let's write a function to add the columns together (normally done with x.sum(axis=1) in NumPy):
#```
#def add_cols(a, b, c):
#    return a+b+c
#```
#Then we have:
#```
```

```python
add_cols(*x.T)
#Output
#array([15, 18, 21, 24, 27])
#```
#NumPy arrays will be unpacked along the first dimension, hence the need to transpose the array.
```


Question:
Open and view .npz file in Python I have an unknown .npz file and would like to see whats inside. I am very new to Python.
```

```python
import numpy
```

```python
b = numpy.load('data.npz')
```

```python
print(b.files)
#Output
#['arr_1', 'arr_0', 'arr_3', 'arr_2']
#```
#How do I see what these arr_i contain (i.e. dimension, values, etc.)?
```

Response:
```
np.savez_compressed('filename.npz', array1=array1, array2=array2)
b = np.load('filename.npz')
```
And do b['array_1'], b['array_2'] and so on to retrieve data from each array.


Question:
CS231n: How to calculate gradient for Softmax loss function? I am watching some videos for Stanford CS231: Convolutional Neural Networks for Visual Recognition but do not quite understand how to calculate analytical gradient for softmax loss function using numpy. From this stackexchange answer, softmax gradient is calculated as: Python implementation for above is:
```
num_classes = W.shape[0]
num_train = X.shape[1]
for i in range(num_train):
  for j in range(num_classes):
    p = np.exp(f_i[j])/sum_i
    dW[j, :] += (p-(j == y[i])) * X[:, i]
```
Could anyone explain how the above snippet work? Detailed implementation for softmax is also included below.
```
def softmax_loss_naive(W, X, y, reg):
  """
  Softmax loss function, naive implementation (with loops)
  Inputs:
  - W: C x D array of weights
  - X: D x N array of data. Data are D-dimensional columns
  - y: 1-dimensional array of length N with labels 0...K-1, for K classes
  - reg: (float) regularization strength
  Returns:
  a tuple of:
  - loss as single float
  - gradient with respect to weights W, an array of same size as W
  """
  # Initialize the loss and gradient to zero.
  loss = 0.0
  dW = np.zeros_like(W)

  #############################################################################
  # Compute the softmax loss and its gradient using explicit loops.           #
  # Store the loss in loss and the gradient in dW. If you are not careful     #
  # here, it is easy to run into numeric instability. Don't forget the        #
  # regularization!                                                           #
  #############################################################################

  # Get shapes
  num_classes = W.shape[0]
  num_train = X.shape[1]

  for i in range(num_train):
    # Compute vector of scores
    f_i = W.dot(X[:, i]) # in R^{num_classes}

    # Normalization trick to avoid numerical instability, per http://cs231n.github.io/linear-classify/#softmax
    log_c = np.max(f_i)
    f_i -= log_c

    # Compute loss (and add to it, divided later)
    # L_i = - f(x_i)_{y_i} + log \sum_j e^{f(x_i)_j}
    sum_i = 0.0
    for f_i_j in f_i:
      sum_i += np.exp(f_i_j)
    loss += -f_i[y[i]] + np.log(sum_i)

    # Compute gradient
    # dw_j = 1/num_train * \sum_i[x_i * (p(y_i = j)-Ind{y_i = j} )]
    # Here we are computing the contribution to the inner sum for a given i.
    for j in range(num_classes):
      p = np.exp(f_i[j])/sum_i
      dW[j, :] += (p-(j == y[i])) * X[:, i]

  # Compute average
  loss /= num_train
  dW /= num_train

  # Regularization
  loss += 0.5 * reg * np.sum(W * W)
  dW += reg*W

  return loss, dW
```

Response: Not sure if this helps, but: is really the indicator function , as described here. This forms the expression (j == y[i]) in the code. Also, the gradient of the loss with respect to the weights is: where which is the origin of the X[:,i] in the code.


Question:
How to use numpy with 'None' value in Python? I'd like to calculate the mean of an array in Python in this form:
```
Matrice = [1, 2, None]
```
I'd just like to have my None value ignored by the numpy.mean calculation but I can't figure out how to do it.

Response:
You are looking for masked arrays. Here's an example.
```
import numpy.ma as ma
a = ma.array([1, 2, None], mask = [0, 0, 1])
print "average =", ma.average(a)
```
From the numpy docs linked above, "The numpy.ma module provides a nearly work-alike replacement for numpy that supports data arrays with masks."


Question:
numpy.isnan(value) not the same as value == numpy.nan? Why am I getting the following:
```

```python
v
#Output
#nan
```

```python
type(v)
#Output
#<type 'numpy.float64'>
```

```python
v == np.nan
#Output
#False
```

```python
np.isnan(v)
#Output
#True
#```
#I would have thought the two should be equivalent?
```

Response: nan != nan. That's just how equality comparisons on nan are defined. It was decided that this result is more convenient for numerical algorithms than the alternative. This is specifically why isnan exists.


Question:
Convert dtype from int64 to int32 Basically, I am using python x32 bit to load from file a list object containing several numpy arrays (previously saved inside a pickle using python x64). I can load them properly and check the contents but I cannot use them.
```
TypeError: Cannot cast array data from dtype('int64') to dtype('int32')
```
How can I convert the arrays element type from within the list to int32 so I can use them with python x32. The error comes when I try to execute the following part:
```
a=np.bincount(np.hstack(data['Y']))
```
Looking at what is inside data['Y']

Response:
As others have said, 32-bit versions of numpy still support 64-bit dtypes. But if you really need to convert to int32, you can use the astype function:
```

```python
import numpy as np
```

```python
x = np.array([1,2,3], dtype=np.int64)
```

```python
x
#Output
#array([1, 2, 3])
```

```python
x.dtype
#Output
#dtype('int64')
```

```python
y = x.astype(np.int32)
```

```python
y
#Output
#array([1, 2, 3], dtype=int32)
#```
```


Question:
Fitting to Poisson histogram I am trying to fit a curve over the histogram of a Poisson distribution that looks like this I have modified the fit function so that it resembles a Poisson distribution, with the parameter t as a variable. But the curve_fit function can not be plotted and I am not sure why.
```
def histo(bsize):
    N = bsize
    #binwidth
    bw = (dt.max()-dt.min())/(N-1.)
    bin1 = dt.min()+ bw*np.arange(N)
    #define the array to hold the occurrence count
    bincount= np.array([])
    for bin in bin1:
        count = np.where((dt>=bin)&(dt<bin+bw))[0].size
        bincount = np.append(bincount,count)
    #bin center
    binc = bin1+0.5*bw
    plt.figure()
    plt.plot(binc,bincount,drawstyle= 'steps-mid')
    plt.xlabel("Interval[ticks]")
    plt.ylabel("Frequency")
histo(30)
plt.xlim(0,.5e8)
plt.ylim(0,25000)
import numpy as np
from scipy.optimize import curve_fit
delta_t = 1.42e7
def func(x, t):
    return t * np.exp(- delta_t/t)
popt, pcov = curve_fit(func, np.arange(0,.5e8),histo(30))
plt.plot(popt)
```

Response:
The problem with your code is that you do not know what the return values of curve_fit are. It is the parameters for the fit-function and their covariance matrix - not something you can plot directly. Binned Least-Squares Fit In general you can get everything much, much more easily:
```py
import numpy as np
import matplotlib.pyplot as plt
from scipy.optimize import curve_fit
from scipy.special import factorial
from scipy.stats import poisson

# get poisson deviated random numbers
data = np.random.poisson(2, 1000)

# the bins should be of integer width, because poisson is an integer distribution
bins = np.arange(11) - 0.5
entries, bin_edges, patches = plt.hist(data, bins=bins, density=True, label='Data')

# calculate bin centers
bin_centers = 0.5 * (bin_edges[1:] + bin_edges[:-1])

def fit_function(k, lamb):
    '''poisson function, parameter lamb is the fit parameter'''
    return poisson.pmf(k, lamb)

# fit with curve_fit
parameters, cov_matrix = curve_fit(fit_function, bin_centers, entries)

# plot poisson-deviation with fitted parameter
x_plot = np.arange(0, 15)

plt.plot(
    x_plot,
    fit_function(x_plot, *parameters),
    marker='o', linestyle='',
    label='Fit result',
)
plt.legend()
plt.show()
```
This is the result: Unbinned Maximum-Likelihood fit An even better possibility would be to not use a histogram at all and instead to carry out a maximum-likelihood fit. But by closer examination even this is unnecessary, because the maximum-likelihood estimator for the parameter of the poissonian distribution is the arithmetic mean. However, if you have other, more complicated PDFs, you can use this as example:
```
import numpy as np
import matplotlib.pyplot as plt
from scipy.optimize import minimize
from scipy.special import factorial
from scipy import stats

def poisson(k, lamb):
    """poisson pdf, parameter lamb is the fit parameter"""
    return (lamb**k/factorial(k)) * np.exp(-lamb)

def negative_log_likelihood(params, data):
    """
    The negative log-Likelihood-Function
    """

    lnl = - np.sum(np.log(poisson(data, params[0])))
    return lnl

def negative_log_likelihood(params, data):
    ''' better alternative using scipy '''
    return -stats.poisson.logpmf(data, params[0]).sum()

# get poisson deviated random numbers
data = np.random.poisson(2, 1000)

# minimize the negative log-Likelihood

result = minimize(negative_log_likelihood,  # function to minimize
                  x0=np.ones(1),            # start value
                  args=(data,),             # additional arguments for function
                  method='Powell',          # minimization method, see docs
                  )
# result is a scipy optimize result object, the fit parameters
# are stored in result.x
print(result)

# plot poisson-distribution with fitted parameter
x_plot = np.arange(0, 15)

plt.plot(
    x_plot,
    stats.poisson.pmf(x_plot, result.x),
    marker='o', linestyle='',
    label='Fit result',
)
plt.legend()
plt.show()
```


Question:
calculating Gini coefficient in Python/numpy i'm calculating Gini coefficient (similar to: Python - Gini coefficient calculation using Numpy) but i get an odd result. for a uniform distribution sampled from np.random.rand(), the Gini coefficient is 0.3 but I would have expected it to be close to 0 (perfect equality). what is going wrong here?
```
def G(v):
    bins = np.linspace(0., 100., 11)
    total = float(np.sum(v))
    yvals = []
    for b in bins:
        bin_vals = v[v <= np.percentile(v, b)]
        bin_fraction = (np.sum(bin_vals) / total) * 100.0
        yvals.append(bin_fraction)
    # perfect equality area
    pe_area = np.trapz(bins, x=bins)
    # lorenz area
    lorenz_area = np.trapz(yvals, x=bins)
    gini_val = (pe_area - lorenz_area) / float(pe_area)
    return bins, yvals, gini_val

v = np.random.rand(500)
bins, result, gini_val = G(v)
plt.figure()
plt.subplot(2, 1, 1)
plt.plot(bins, result, label="observed")
plt.plot(bins, bins, '--', label="perfect eq.")
plt.xlabel("fraction of population")
plt.ylabel("fraction of wealth")
plt.title("GINI: %.4f" %(gini_val))
plt.legend()
plt.subplot(2, 1, 2)
plt.hist(v, bins=20)
```
for the given set of numbers, the above code calculates the fraction of the total distribution's values that are in each percentile bin. the result: uniform distributions should be near "perfect equality" so the lorenz curve bending is off.

Response:
This is to be expected. A random sample from a uniform distribution does not result in uniform values (i.e. values that are all relatively close to each other). With a little calculus, it can be shown that the expected value (in the statistical sense) of the Gini coefficient of a sample from the uniform distribution on [0, 1] is 1/3, so getting values around 1/3 for a given sample is reasonable. You'll get a lower Gini coefficient with a sample such as v = 10 + np.random.rand(500). Those values are all close to 10.5; the relative variation is lower than the sample v = np.random.rand(500). In fact, the expected value of the Gini coefficient for the sample base + np.random.rand(n) is 1/(6*base + 3). Here's a simple implementation of the Gini coefficient. It uses the fact that the Gini coefficient is half the relative mean absolute difference.
```
def gini(x):
    # (Warning: This is a concise implementation, but it is O(n**2)
    # in time and memory, where n = len(x).  *Don't* pass in huge
    # samples!)

    # Mean absolute difference
    mad = np.abs(np.subtract.outer(x, x)).mean()
    # Relative mean absolute difference
    rmad = mad/np.mean(x)
    # Gini coefficient
    g = 0.5 * rmad
    return g
```
(For some more efficient implementations, see More efficient weighted Gini coefficient in Python) Here's the Gini coefficient for several samples of the form v = base + np.random.rand(500):
```

```python
v = np.random.rand(500)
```

```python
gini(v)
#Output
#0.32760618249832563
```

```python
v = 1 + np.random.rand(500)
```

```python
gini(v)
#Output
#0.11121487509454202
```

```python
v = 10 + np.random.rand(500)
```

```python
gini(v)
#Output
#0.01567937753659053
```

```python
v = 100 + np.random.rand(500)
```

```python
gini(v)
#Output
#0.0016594595244509495
#```
```


Question:
numpy: unique list of colors in the image I have an image img:
```

```python
img.shape
#Output
#(200, 200, 3)
#```
#On pixel (100, 100) I have a nice color:
#```
```

```python
img[100,100]
#Output
#array([ 0.90980393,  0.27450982,  0.27450982], dtype=float32)
#```
#Now my question is: How many different colors are there in this image, and how do I enumerate them? My first idea was numpy.unique(), but somehow I am using this wrong.
```

Response:
Your initial idea to use numpy.unique() actually can do the job perfectly with the best performance:
```
numpy.unique(img.reshape(-1, img.shape[2]), axis=0)
```
At first, we flatten rows and columns of matrix. Now the matrix has as much rows as there're pixels in the image. Columns are color components of each pixels. Then we count unique rows of flattened matrix.


Question:
Comparison to `None` will result in an elementwise object Apparantly it will (in the 'future') not be possible anymore to use the following:
```
import numpy as np
np.array([0,1,2]) == None
> False
> FutureWarning: comparison to `None` will result in an elementwise object comparison in the future.
```
This also breaks the lazy loading pattern for numpy arrays:
```
import numpy as np
def f(a=None):
    if a == None:
        a = <some default value>
    <function body>
```
What other possibilities allow you to still use lazy initialization?

Response:
You are looking for is:
```
if a is None:
    a = something else
```
The problem is that, by using the == operator, if the input element a is a numpy array, numpy will try to perform an element wise comparison and tell you that you cannot compare it. For a a numpy array, a == None gives error, np.all(a == None) doesn't (but does not do what you expect). Instead a is None will work regardless the data type of a.


Question:
numpy.loadtxt, ValueError: could not convert string to float This is sample from large csv file:
```
6.1;6.1;7.2;8.9;5.0;
8.9;10.0;8.9;6.1;5.0;
```
If I try to read it to numpy array with np.loadtxt('test.csv', delimiter=';') I get: ValueError: could not convert string to float: and don't understand why?

Response:
You need to strip off the trailing ';' from the lines. A possible workaround if you know you have 5 columns is:
```
np.loadtxt('test.csv', delimiter=';', usecols=range(5))
```
Or, use genfromtext instead which handles missing values
```
np.genfromtxt('test.csv', delimiter=';')[:,:-1]
```


Question:
How to return a view of several columns in numpy structured array I can see several columns (fields) at once in a numpy structured array by indexing with a list of the field names, for example
```
import numpy as np

a = np.array([(1.5, 2.5, (1.0,2.0)), (3.,4.,(4.,5.)), (1.,3.,(2.,6.))],
        dtype=[('x',float), ('y',float), ('value',float,(2,2))])

print a[['x','y']]
#[(1.5, 2.5) (3.0, 4.0) (1.0, 3.0)]

print a[['x','y']].dtype
#[('x', '<f4') ('y', '<f4')])
```
But the problem is that it seems to be a copy rather than a view:
```
b = a[['x','y']]
b[0] = (9.,9.)

print b
#[(9.0, 9.0) (3.0, 4.0) (1.0, 3.0)]

print a[['x','y']]
#[(1.5, 2.5) (3.0, 4.0) (1.0, 3.0)]
```
If I only select one column, it's a view:
```
c = x['y']
c[0] = 99.

print c
#[ 99.  4.   3. ]

print a['y']
#[ 99.  4.   3. ]
```
Is there any way I can get the view behavior for more than one column at once? I have two workarounds, one is to just loop through the columns, the other is to create a hierarchical dtype, so that the one column actually returns a structured array with the two (or more) fields that I want. Unfortunately, zip also returns a copy, so I can't do:
```
x = a['x']; y = a['y']
z = zip(x,y)
z[0] = (9.,9.)
```

Response:
You can create a dtype object contains only the fields that you want, and use numpy.ndarray() to create a view of original array:
```
import numpy as np
strc = np.zeros(3, dtype=[('x', int), ('y', float), ('z', int), ('t', "i8")])

def fields_view(arr, fields):
    dtype2 = np.dtype({name:arr.dtype.fields[name] for name in fields})
    return np.ndarray(arr.shape, dtype2, arr, 0, arr.strides)

v1 = fields_view(strc, ["x", "z"])
v1[0] = 10, 100

v2 = fields_view(strc, ["y", "z"])
v2[1:] = [(3.14, 7)]

v3 = fields_view(strc, ["x", "t"])

v3[1:] = [(1000, 2**16)]

print(strc)
```
here is the output:
```
[(10, 0.0, 100, 0L) (1000, 3.14, 7, 65536L) (1000, 3.14, 7, 65536L)]
```


Question:
I am getting a warning  I am trying to run a quadratic equation in python. However, it keeps on giving me a warning
```
RuntimeWarning: invalid value encountered in sqrt
```
Here's my code:
```
import numpy as np

a = 0.75 + (1.25 - 0.75)*np.random.randn(10000)
print(a)
b = 8 + (12 - 8)*np.random.randn(10000)
print(b)
c = -12 + 2*np.random.randn(10000)
print(c)
x0 = (-b - np.sqrt(b**2 - (4*a*c)))/(2 * a)
print(x0)
```

Response:
This is not 100% Python related. You can't calculate the square root of a negative number (when dealing with real numbers that is). You didn't take any precautions for when b**2 - (4*a*c) is a negative number.
```

```python
import numpy as np
#Output
#>>>
```

```python
np.sqrt(4)
#Output
#2.0
```

```python
np.sqrt(-4)
#Output
#__main__:1: RuntimeWarning: invalid value encountered in sqrt
#nan
#```
#Let's test if you have negative values:
#```
```

```python
import numpy as np
```

```python

```

```python
a = 0.75 + (1.25 - 0.75) * np.random.randn(10000)
```

```python
b = 8 + (12 - 8) * np.random.randn(10000)
```

```python
c = -12 + 2 * np.random.randn(10000)
```

```python

```

```python
z = b ** 2 - (4 * a * c)
```

```python
print len([_ for _ in z if _ < 0])
#Output
#71
#```
```


Question:
How to surface plot/3d plot from dataframe I am new to pandas and matplotlib. I couldn't get the exact reference to plot my DataFrame which has the following schema:
```
schema = StructType([
StructField("x", IntegerType(), True),
StructField("y", IntegerType(), True),
StructField("z", IntegerType(), True)])
```
I'd like to plot 3d graph w.r.t. x, y and z. Here is the sample code I used:
```
import matplotlib.pyplot as pltt

dfSpark = sqlContext.createDataFrame(tupleRangeRDD, schema) // reading as spark df
df = dfSpark.toPandas()
fig = pltt.figure();
ax = fig.add_subplot(111, projection='3d')
ax.plot_surface(df['x'], df['y'], df['z'])
```
I am getting a empty graph plot. I'm definitely missing something. Any pointers? Request-1: Print df
```
def print_full(x):
    pd.set_option('display.max_rows', len(x))
    print(x)
    pd.reset_option('display.max_rows')

print_full(df)
```
Result of top 10:
```
x    y       z
0      301  301      10
1      300  301      16
2      300  300       6
3      299  301      30
4      299  300      20
5      299  299      14
6      298  301      40
7      298  300      30
8      298  299      24
9      298  298      10
10     297  301      48
```

Response:
.plot_surface() takes 2D arrays as inputs, not 1D DataFrame columns. This has been explained quite well here, along with the below code that illustrates how one could arrive at the required format using DataFrame input. Reproduced below with minor modifications like additional comments. Alternatively, however, there is .plot_trisurf() which uses 1D inputs. I've added an example in the middle of the code.
```
import numpy as np
import matplotlib.pyplot as plt
from matplotlib import cm
from matplotlib.ticker import LinearLocator, FormatStrFormatter
from mpl_toolkits.mplot3d import Axes3D

## Matplotlib Sample Code using 2D arrays via meshgrid
X = np.arange(-5, 5, 0.25)
Y = np.arange(-5, 5, 0.25)
X, Y = np.meshgrid(X, Y)
R = np.sqrt(X ** 2 + Y ** 2)
Z = np.sin(R)
fig = plt.figure()
ax = Axes3D(fig)
surf = ax.plot_surface(X, Y, Z, rstride=1, cstride=1, cmap=cm.coolwarm,
                       linewidth=0, antialiased=False)
ax.set_zlim(-1.01, 1.01)

ax.zaxis.set_major_locator(LinearLocator(10))
ax.zaxis.set_major_formatter(FormatStrFormatter('%.02f'))

fig.colorbar(surf, shrink=0.5, aspect=5)
plt.title('Original Code')
plt.show()
```
```
## DataFrame from 2D-arrays
x = X.reshape(1600)
y = Y.reshape(1600)
z = Z.reshape(1600)
df = pd.DataFrame({'x': x, 'y': y, 'z': z}, index=range(len(x)))

# Plot using `.trisurf()`:

ax.plot_trisurf(df.x, df.y, df.z, cmap=cm.jet, linewidth=0.2)
plt.show()
```
```
# 2D-arrays from DataFrame
x1 = np.linspace(df['x'].min(), df['x'].max(), len(df['x'].unique()))
y1 = np.linspace(df['y'].min(), df['y'].max(), len(df['y'].unique()))

"""
x, y via meshgrid for vectorized evaluation of
2 scalar/vector fields over 2-D grids, given
one-dimensional coordinate arrays x1, x2,..., xn.
"""

x2, y2 = np.meshgrid(x1, y1)

# Interpolate unstructured D-dimensional data.
z2 = griddata((df['x'], df['y']), df['z'], (x2, y2), method='cubic')

# Ready to plot
fig = plt.figure()
ax = fig.gca(projection='3d')
surf = ax.plot_surface(x2, y2, z2, rstride=1, cstride=1, cmap=cm.coolwarm,
                       linewidth=0, antialiased=False)
ax.set_zlim(-1.01, 1.01)

ax.zaxis.set_major_locator(LinearLocator(10))
ax.zaxis.set_major_formatter(FormatStrFormatter('%.02f'))

fig.colorbar(surf, shrink=0.5, aspect=5)
plt.title('Meshgrid Created from 3 1D Arrays')

plt.show()
```


Question:
ValueError: ndarray is not C-contiguous in cython I have written the following function in cython to estimate the log-likelihood
```
@cython.boundscheck(False)
@cython.wraparound(False)
def likelihood(double m,
               double c,
               np.ndarray[np.double_t, ndim=1, mode='c'] r_mpc not None,
               np.ndarray[np.double_t, ndim=1, mode='c'] gtan not None,
               np.ndarray[np.double_t, ndim=1, mode='c'] gcrs not None,
               np.ndarray[np.double_t, ndim=1, mode='c'] shear_err not None,
               np.ndarray[np.double_t, ndim=1, mode='c'] beta not None,
               double rho_c,
               np.ndarray[np.double_t, ndim=1, mode='c'] rho_c_sigma not None):
    cdef double rscale = rscaleConstM(m, c,rho_c, 200)

    cdef Py_ssize_t ngals = r_mpc.shape[0]

    cdef np.ndarray[DTYPE_T, ndim=1, mode='c'] gamma_inf = Sh(r_mpc, c, rscale, rho_c_sigma)
    cdef np.ndarray[DTYPE_T, ndim=1, mode='c'] kappa_inf = Kap(r_mpc, c, rscale, rho_c_sigma)

    cdef double delta = 0.
    cdef double modelg = 0.
    cdef double modsig = 0.

    cdef Py_ssize_t i
    cdef DTYPE_T logProb = 0.

    #calculate logprob
    for i from ngals > i >= 0:

        modelg = (beta[i]*gamma_inf[i] / (1 - beta[i]*kappa_inf[i]))

        delta = gtan[i] - modelg

        modsig = shear_err[i]

        logProb = logProb -.5*(delta/modsig)**2  - logsqrt2pi - log(modsig)

    return logProb
```
but when I run the compiled version of this function, I get the following error message:
```
File "Tools.pyx", line 3, in Tools.likelihood
    def likelihood(double m,
ValueError: ndarray is not C-contiguous
```
I could not quite understand why this problem occurs??!!! I will appreciate to get any useful tips.

Response:
Just before you get the error, try printing the flags attribute of the numpy array(s) you're passing to likelihood. You'll probably see something like:
```

```python
foo.flags
#Output
#C_CONTIGUOUS : False
#  F_CONTIGUOUS : True
#  OWNDATA : True
#  WRITEABLE : True
#  ALIGNED : True
#  UPDATEIFCOPY : False
#```
#Note where it says C_CONTIGUOUS : False, because that's the issue. To fix it, simply convert it to C-order:
#```
```

```python
foo = foo.copy(order='C')
```

```python
foo.flags
#Output
#C_CONTIGUOUS : True
#  F_CONTIGUOUS : False
#  OWNDATA : True
#  WRITEABLE : True
#  ALIGNED : True
#  UPDATEIFCOPY : False
#```
```


Question:
Group by consecutive index numbers I was wondering if there is a way to groupby consecutive index numbers and move the groups in different columns. Here is an example of the DataFrame I'm using:
```
0
0     19218.965703
1     19247.621650
2     19232.651322
9     19279.216956
10    19330.087371
11    19304.316973
```
And my idea is to gruoup by sequential index numbers and get something like this:
```
0             1
0     19218.965703  19279.216956
1     19247.621650  19330.087371
2     19232.651322  19304.316973
```
Ive been trying to split my data by blocks of 3 and then groupby but I was looking more about something that can be used to group and rearrange sequential index numbers. Thank you!

Response:
Here is one way:
```
from more_itertools import consecutive_groups
final=pd.concat([df.loc[i].reset_index(drop=True)
                    for i in consecutive_groups(df.index)],axis=1)
final.columns=range(len(final.columns))
print(final)
```
```
0             1
0  19218.965703  19279.216956
1  19247.621650  19330.087371
2  19232.651322  19304.316973
```


Question: Multivariate spline interpolation in python/scipy? Is there a library module or other straightforward way to implement multivariate spline interpolation in python? Specifically, I have a set of scalar data on a regularly-spaced three-dimensional grid which I need to interpolate at a small number of points scattered throughout the domain. For two dimensions, I have been using scipy.interpolate.RectBivariateSpline, and I'm essentially looking for an extension of that to three-dimensional data. The N-dimensional interpolation routines I have found are not quite good enough: I would prefer splines over LinearNDInterpolator for smoothness, and I have far too many data points (often over one million) for, e.g., a radial basis function to work. If anyone knows of a python library that can do this, or perhaps one in another language that I could call or port, I'd really appreciate it.

Response:
If I'm understanding your question correctly, your input "observation" data is regularly gridded? If so, scipy.ndimage.map_coordinates does exactly what you want. It's a bit hard to understand at first pass, but essentially, you just feed it a sequence of coordinates that you want to interpolate the values of the grid at in pixel/voxel/n-dimensional-index coordinates. As a 2D example:
```
import numpy as np
from scipy import ndimage
import matplotlib.pyplot as plt

# Note that the output interpolated coords will be the same dtype as your input
# data.  If we have an array of ints, and we want floating point precision in
# the output interpolated points, we need to cast the array as floats
data = np.arange(40).reshape((8,5)).astype(np.float)

# I'm writing these as row, column pairs for clarity...
coords = np.array([[1.2, 3.5], [6.7, 2.5], [7.9, 3.5], [3.5, 3.5]])
# However, map_coordinates expects the transpose of this
coords = coords.T

# The "mode" kwarg here just controls how the boundaries are treated
# mode='nearest' is _not_ nearest neighbor interpolation, it just uses the
# value of the nearest cell if the point lies outside the grid.  The default is
# to treat the values outside the grid as zero, which can cause some edge
# effects if you're interpolating points near the edge
# The "order" kwarg controls the order of the splines used. The default is
# cubic splines, order=3
zi = ndimage.map_coordinates(data, coords, order=3, mode='nearest')

row, column = coords
nrows, ncols = data.shape
im = plt.imshow(data, interpolation='nearest', extent=[0, ncols, nrows, 0])
plt.colorbar(im)
plt.scatter(column, row, c=zi, vmin=data.min(), vmax=data.max())
for r, c, z in zip(row, column, zi):
    plt.annotate('%0.3f' % z, (c,r), xytext=(-10,10), textcoords='offset points',
            arrowprops=dict(arrowstyle='->'), ha='right')
plt.show()
```
To do this in n-dimensions, we just need to pass in the appropriate sized arrays:
```
import numpy as np
from scipy import ndimage

data = np.arange(3*5*9).reshape((3,5,9)).astype(np.float)
coords = np.array([[1.2, 3.5, 7.8], [0.5, 0.5, 6.8]])
zi = ndimage.map_coordinates(data, coords.T)
```
As far as scaling and memory usage goes, map_coordinates will create a filtered copy of the array if you're using an order > 1 (i.e. not linear interpolation). If you just want to interpolate at a very small number of points, this is a rather large overhead. It doesn't increase with the number points you want to interpolate at, however. As long as have enough RAM for a single temporary copy of your input data array, you'll be fine. If you can't store a copy of your data in memory, you can either a) specify prefilter=False and order=1 and use linear interpolation, or b) replace your original data with a filtered version using ndimage.spline_filter, and then call map_coordinates with prefilter=False. Even if you have enough ram, keeping the filtered dataset around can be a big speedup if you need to call map_coordinates multiple times (e.g. interactive use, etc).


Question:
Comparing elements of an array to a scalar and getting the max in Python I want to compare the elements of an array to a scalar and get an array with the maximum of the compared values. That's I want to call
```
import numpy as np
np.max([1,2,3,4], 3)
```
and want to get
```
array([3,3,3,4])
```
But I get
```
ValueError: 'axis' entry is out of bounds
```
When I run
```
np.max([[1,2,3,4], 3])
```
I get
```
[1, 2, 3, 4]
```
which is one of the two elements in the list that is not the result I seek for. Is there a Numpy solution for that which is fast as the other built-in functions?

Response:
This is already built into numpy with the function np.maximum:
```py
a = np.arange(1,5)
n = 3

np.maximum(a, n)
#array([3, 3, 3, 4])
```
This doesn't mutate a:
```py
a
#array([1, 2, 3, 4])
```
If you want to mutate the original array as in @jamylak's answer, you can give a as the output:
```py
np.maximum(a, n, a)
#array([3, 3, 3, 4])

a
#array([3, 3, 3, 4])
```
Docs: maximum(x1, x2[, out]) Element-wise maximum of array elements. Equivalent to np.where(x1 > x2, x1, x2) but faster and does proper broadcasting.


Question: What is the difference between the types  and ? Is there a difference between the types  and ?

Response:
numpy.string_ is the NumPy datatype used for arrays containing fixed-width byte strings. On the other hand, str is a native Python type and can not be used as a datatype for NumPy arrays*. If you create a NumPy array containing strings, the array will use the numpy.string_ type (or the numpy.unicode_ type in Python 3). More precisely, the array will use a sub-datatype of np.string_:
```

```python
a = np.array(['abc', 'xy'])
```

```python
a
#Output
#array(['abc', 'xy'], dtype='<S3')
```

```python
np.issubdtype('<S3', np.string_)
#Output
#True
#```
#In this case the datatype is '<S3': the < denotes the byte-order (little-endian), S denotes the string type and 3 indicates that each value in the array holds up to three characters (or bytes). One property that np.string_ and str share is immutability. Trying to increase the length of a Python str object will create a new object in memory. Similarly, if you want fixed-width NumPy array to hold more characters, a new larger array will have to be created in memory. * Note that it is possible to create a NumPy object array which contains references to Python str objects, but such arrays behave quite differently to normal arrays.
```


Question:
Smallest positive float64 number I need to find a numpy.float64 value that is as close to zero as possible. Numpy offers several constants that allow to do something similar: np.finfo(np.float64).eps = 2.2204460492503131e-16 np.finfo(np.float64).tiny = 2.2250738585072014e-308 These are both reasonably small, but when I do this
```

```python
x = np.finfo(np.float64).tiny
```

```python
x / 2
#Output
#6.9533558078350043e-310
#```
#the result is even smaller. When using an impromptu binary search I can get down to about 1e-323, before the value is rounded down to 0.0. Is there a constant for this in numpy that I am missing? Alternatively, is there a right way to do this?
```

Response:
Use np.nextafter.
```

```python
import numpy as np
```

```python
np.nextafter(0, 1)
#Output
#4.9406564584124654e-324
```

```python
np.nextafter(np.float32(0), np.float32(1))
#Output
#1.4012985e-45
#```
```


Question:
Understanding scipy deconvolve I'm trying to understand scipy.signal.deconvolve. From the mathematical point of view a convolution is just the multiplication in fourier space so I would expect that for two functions f and g: Deconvolve(Convolve(f,g) , g) == f In numpy/scipy this is either not the case or I'm missing an important point. Although there are some questions related to deconvolve on SO already (like here and here) they do not address this point, others remain unclear (this) or unanswered (here). There are also two questions on SignalProcessing SE (this and this) the answers to which are not helpful in understanding how scipy's deconvolve function works. The question would be: How do you reconstruct the original signal f from a convoluted signal, assuming you know the convolving function g.? Or in other words: How does this pseudocode Deconvolve(Convolve(f,g) , g) == f translate into numpy / scipy? Edit: Note that this question is not targeted at preventing numerical inaccuracies (although this is also an open question) but at understanding how convolve/deconvolve work together in scipy. The following code tries to do that with a Heaviside function and a gaussian filter. As can be seen in the image, the result of the deconvolution of the convolution is not at all the original Heaviside function. I would be glad if someone could shed some light into this issue.
```
import numpy as np
import scipy.signal
import matplotlib.pyplot as plt

# Define heaviside function
H = lambda x: 0.5 * (np.sign(x) + 1.)
#define gaussian
gauss = lambda x, sig: np.exp(-( x/float(sig))**2 )

X = np.linspace(-5, 30, num=3501)
X2 = np.linspace(-5,5, num=1001)

# convolute a heaviside with a gaussian
H_c = np.convolve( H(X),  gauss(X2, 1),  mode="same"  )
# deconvolute a the result
H_dc, er = scipy.signal.deconvolve(H_c, gauss(X2, 1) )

#### Plot ####
fig , ax = plt.subplots(nrows=4, figsize=(6,7))
ax[0].plot( H(X),          color="#907700", label="Heaviside",    lw=3 )
ax[1].plot( gauss(X2, 1),  color="#907700", label="Gauss filter", lw=3 )
ax[2].plot( H_c/H_c.max(), color="#325cab", label="convoluted" ,  lw=3 )
ax[3].plot( H_dc,          color="#ab4232", label="deconvoluted", lw=3 )
for i in range(len(ax)):
    ax[i].set_xlim([0, len(X)])
    ax[i].set_ylim([-0.07, 1.2])
    ax[i].legend(loc=4)
plt.show()
```
Edit: Note that there is a matlab example, showing how to convolve/deconvolve a rectangular signal using
```
yc=conv(y,c,'full')./sum(c);
ydc=deconv(yc,c).*sum(c);
```
In the spirit of this question it would also help if someone was able to translate this example into python.

Response:
After some trial and error I found out how to interprete the results of scipy.signal.deconvolve() and I post my findings as an answer. Let's start with a working example code
```
import numpy as np
import scipy.signal
import matplotlib.pyplot as plt

# let the signal be box-like
signal = np.repeat([0., 1., 0.], 100)
# and use a gaussian filter
# the filter should be shorter than the signal
# the filter should be such that it's much bigger then zero everywhere
gauss = np.exp(-( (np.linspace(0,50)-25.)/float(12))**2 )
print gauss.min()  # = 0.013 >> 0

# calculate the convolution (np.convolve and scipy.signal.convolve identical)
# the keywordargument mode="same" ensures that the convolution spans the same
#   shape as the input array.
#filtered = scipy.signal.convolve(signal, gauss, mode='same')
filtered = np.convolve(signal, gauss, mode='same')

deconv,  _ = scipy.signal.deconvolve( filtered, gauss )
#the deconvolution has n = len(signal) - len(gauss) + 1 points
n = len(signal)-len(gauss)+1
# so we need to expand it by
s = (len(signal)-n)/2
#on both sides.
deconv_res = np.zeros(len(signal))
deconv_res[s:len(signal)-s-1] = deconv
deconv = deconv_res
# now deconv contains the deconvolution
# expanded to the original shape (filled with zeros)

#### Plot ####
fig , ax = plt.subplots(nrows=4, figsize=(6,7))

ax[0].plot(signal,            color="#907700", label="original",     lw=3 )
ax[1].plot(gauss,          color="#68934e", label="gauss filter", lw=3 )
# we need to divide by the sum of the filter window to get the convolution normalized to 1
ax[2].plot(filtered/np.sum(gauss), color="#325cab", label="convoluted" ,  lw=3 )
ax[3].plot(deconv,         color="#ab4232", label="deconvoluted", lw=3 )

for i in range(len(ax)):
    ax[i].set_xlim([0, len(signal)])
    ax[i].set_ylim([-0.07, 1.2])
    ax[i].legend(loc=1, fontsize=11)
    if i != len(ax)-1 :
        ax[i].set_xticklabels([])

plt.savefig(__file__ + ".png")
plt.show()
```
This code produces the following image, showing exactly what we want (Deconvolve(Convolve(signal,gauss) , gauss) == signal) Some important findings are: The filter should be shorter than the signal The filter should be much bigger than zero everywhere (here > 0.013 is good enough) Using the keyword argument mode = 'same' to the convolution ensures that it lives on the same array shape as the signal. The deconvolution has n = len(signal) - len(gauss) + 1 points. So in order to let it also reside on the same original array shape we need to expand it by s = (len(signal)-n)/2 on both sides. Of course, further findings, comments and suggestion to this question are still welcome.


Question:
Dihedral/Torsion Angle From Four Points in Cartesian Coordinates in Python What suggestions do people have for quickly calculating dihedral angles in Python? In the diagrams, phi is the dihedral angle: What's your best for calculating angles in the range 0 to pi? What about 0 to 2pi? "Best" here means some mix of fast and numerically stable. Methods that return values over the full range 0 to 2pi are preferred but if you have an incredibly fast way of calculating the dihedral over 0 to pi share that too. Here are my 3 best efforts. Only the 2nd one returns angles between 0 and 2pi. It's also the slowest. General comments about my approaches: arccos() in Numpy seems plenty stable but since people raise this issue I may just not fully understand it. The use of einsum came from here. Why is numpy's einsum faster than numpy's built in functions? The diagrams and some inspiration came from here. How do I calculate a dihedral angle given Cartesian coordinates? The 3 approaches with comments:
```
import numpy as np
from time import time

# This approach tries to minimize magnitude and sqrt calculations
def dihedral1(p):
    # Calculate vectors between points, b1, b2, and b3 in the diagram
    b = p[:-1] - p[1:]
    # "Flip" the first vector so that eclipsing vectors have dihedral=0
    b[0] *= -1
    # Use dot product to find the components of b1 and b3 that are not
    # perpendicular to b2. Subtract those components. The resulting vectors
    # lie in parallel planes.
    v = np.array( [ v - (v.dot(b[1])/b[1].dot(b[1])) * b[1] for v in [b[0], b[2]] ] )
    # Use the relationship between cos and dot product to find the desired angle.
    return np.degrees(np.arccos( v[0].dot(v[1])/(np.linalg.norm(v[0]) * np.linalg.norm(v[1]))))

# This is the straightforward approach as outlined in the answers to
# "How do I calculate a dihedral angle given Cartesian coordinates?"
def dihedral2(p):
    b = p[:-1] - p[1:]
    b[0] *= -1
    v = np.array( [ v - (v.dot(b[1])/b[1].dot(b[1])) * b[1] for v in [b[0], b[2]] ] )
    # Normalize vectors
    v /= np.sqrt(np.einsum('...i,...i', v, v)).reshape(-1,1)
    b1 = b[1] / np.linalg.norm(b[1])
    x = np.dot(v[0], v[1])
    m = np.cross(v[0], b1)
    y = np.dot(m, v[1])
    return np.degrees(np.arctan2( y, x ))

# This one starts with two cross products to get a vector perpendicular to
# b2 and b1 and another perpendicular to b2 and b3. The angle between those vectors
# is the dihedral angle.
def dihedral3(p):
    b = p[:-1] - p[1:]
    b[0] *= -1
    v = np.array( [np.cross(v,b[1]) for v in [b[0], b[2]] ] )
    # Normalize vectors
    v /= np.sqrt(np.einsum('...i,...i', v, v)).reshape(-1,1)
    return np.degrees(np.arccos( v[0].dot(v[1]) ))

dihedrals = [ ("dihedral1", dihedral1), ("dihedral2", dihedral2), ("dihedral3", dihedral3) ]
```
Benchmarking:
```
# Testing arccos near 0
# Answer is 0.000057
p1 = np.array([
                [ 1,           0,         0     ],
                [ 0,           0,         0     ],
                [ 0,           0,         1     ],
                [ 0.999999,    0.000001,  1     ]
                ])

# +x,+y
p2 = np.array([
                [ 1,           0,         0     ],
                [ 0,           0,         0     ],
                [ 0,           0,         1     ],
                [ 0.1,         0.6,       1     ]
                ])

# -x,+y
p3 = np.array([
                [ 1,           0,         0     ],
                [ 0,           0,         0     ],
                [ 0,           0,         1     ],
                [-0.3,         0.6,       1     ]
                ])
# -x,-y
p4 = np.array([
                [ 1,           0,         0     ],
                [ 0,           0,         0     ],
                [ 0,           0,         1     ],
                [-0.3,        -0.6,       1     ]
                ])
# +x,-y
p5 = np.array([
                [ 1,           0,         0     ],
                [ 0,           0,         0     ],
                [ 0,           0,         1     ],
                [ 0.6,        -0.6,       1     ]
                ])

for d in dihedrals:
    name = d[0]
    f = d[1]
    print "%s:    %12.6f    %12.6f    %12.6f    %12.6f    %12.6f" \
            % (name, f(p1), f(p2), f(p3), f(p4), f(p5))
print

def profileDihedrals(f):
    t0 = time()
    for i in range(20000):
        p = np.random.random( (4,3) )
        f(p)
        p = np.random.randn( 4,3 )
        f(p)
    return(time() - t0)

print "dihedral1:    ", profileDihedrals(dihedral1)
print "dihedral2:    ", profileDihedrals(dihedral2)
print "dihedral3:    ", profileDihedrals(dihedral3)
```
Benchmarking output:
```
dihedral1:        0.000057       80.537678      116.565051      116.565051       45.000000
dihedral2:        0.000057       80.537678      116.565051     -116.565051      -45.000000
dihedral3:        0.000057       80.537678      116.565051      116.565051       45.000000

dihedral1:     2.79781794548
dihedral2:     3.74271392822
dihedral3:     2.49604296684
```
As you can see in the benchmarking, the last one tends to be the fastest while the second one is the only one that returns angles from the full range of 0 to 2pi since it uses arctan2.

Response:
Here's an implementation for torsion angle over the full 2pi range that is a bit faster, doesn't resort to numpy quirks (einsum being mysteriously faster than logically equivalent code), and is easier to read. There's even a bit more than just hacks going on here -- the math is different too. The formula used in the question's dihedral2 uses 3 square roots and 1 cross product, the formula on Wikipedia uses 1 square root and 3 cross products, but the formula used in the function below uses only 1 square root and 1 cross product. This is probably as simple as the math can get. Functions with 2pi range function from question, Wikipedia formula for comparison, and the new function: dihedrals.py
```
#!/usr/bin/env python
# -*- coding: utf-8 -*-

import numpy as np

def old_dihedral2(p):
    """http://stackoverflow.com/q/20305272/1128289"""
    b = p[:-1] - p[1:]
    b[0] *= -1
    v = np.array( [ v - (v.dot(b[1])/b[1].dot(b[1])) * b[1] for v in [b[0], b[2]] ] )
    # Normalize vectors
    v /= np.sqrt(np.einsum('...i,...i', v, v)).reshape(-1,1)
    b1 = b[1] / np.linalg.norm(b[1])
    x = np.dot(v[0], v[1])
    m = np.cross(v[0], b1)
    y = np.dot(m, v[1])
    return np.degrees(np.arctan2( y, x ))

def wiki_dihedral(p):
    """formula from Wikipedia article on "Dihedral angle"; formula was removed
    from the most recent version of article (no idea why, the article is a
    mess at the moment) but the formula can be found in at this permalink to
    an old version of the article:
    https://en.wikipedia.org/w/index.php?title=Dihedral_angle&oldid=689165217#Angle_between_three_vectors
    uses 1 sqrt, 3 cross products"""
    p0 = p[0]
    p1 = p[1]
    p2 = p[2]
    p3 = p[3]

    b0 = -1.0*(p1 - p0)
    b1 = p2 - p1
    b2 = p3 - p2

    b0xb1 = np.cross(b0, b1)
    b1xb2 = np.cross(b2, b1)

    b0xb1_x_b1xb2 = np.cross(b0xb1, b1xb2)

    y = np.dot(b0xb1_x_b1xb2, b1)*(1.0/np.linalg.norm(b1))
    x = np.dot(b0xb1, b1xb2)

    return np.degrees(np.arctan2(y, x))

def new_dihedral(p):
    """Praxeolitic formula
    1 sqrt, 1 cross product"""
    p0 = p[0]
    p1 = p[1]
    p2 = p[2]
    p3 = p[3]

    b0 = -1.0*(p1 - p0)
    b1 = p2 - p1
    b2 = p3 - p2

    # normalize b1 so that it does not influence magnitude of vector
    # rejections that come next
    b1 /= np.linalg.norm(b1)

    # vector rejections
    # v = projection of b0 onto plane perpendicular to b1
    #   = b0 minus component that aligns with b1
    # w = projection of b2 onto plane perpendicular to b1
    #   = b2 minus component that aligns with b1
    v = b0 - np.dot(b0, b1)*b1
    w = b2 - np.dot(b2, b1)*b1

    # angle between v and w in a plane is the torsion angle
    # v and w may not be normalized but that's fine since tan is y/x
    x = np.dot(v, w)
    y = np.dot(np.cross(b1, v), w)
    return np.degrees(np.arctan2(y, x))
```
The new function would probably be a bit more conveniently called with 4 separate arguments but it to match the signature in the original question it simply immediately unpacks the argument. Code for testing: test_dihedrals.ph
```
from dihedrals import *

# some atom coordinates for testing
p0 = np.array([24.969, 13.428, 30.692]) # N
p1 = np.array([24.044, 12.661, 29.808]) # CA
p2 = np.array([22.785, 13.482, 29.543]) # C
p3 = np.array([21.951, 13.670, 30.431]) # O
p4 = np.array([23.672, 11.328, 30.466]) # CB
p5 = np.array([22.881, 10.326, 29.620]) # CG
p6 = np.array([23.691,  9.935, 28.389]) # CD1
p7 = np.array([22.557,  9.096, 30.459]) # CD2

# I guess these tests do leave 1 quadrant (-x, +y) untested, oh well...

def test_old_dihedral2():
    assert(abs(old_dihedral2(np.array([p0, p1, p2, p3])) - (-71.21515)) < 1E-4)
    assert(abs(old_dihedral2(np.array([p0, p1, p4, p5])) - (-171.94319)) < 1E-4)
    assert(abs(old_dihedral2(np.array([p1, p4, p5, p6])) - (60.82226)) < 1E-4)
    assert(abs(old_dihedral2(np.array([p1, p4, p5, p7])) - (-177.63641)) < 1E-4)

def test_new_dihedral1():
    assert(abs(wiki_dihedral(np.array([p0, p1, p2, p3])) - (-71.21515)) < 1E-4)
    assert(abs(wiki_dihedral(np.array([p0, p1, p4, p5])) - (-171.94319)) < 1E-4)
    assert(abs(wiki_dihedral(np.array([p1, p4, p5, p6])) - (60.82226)) < 1E-4)
    assert(abs(wiki_dihedral(np.array([p1, p4, p5, p7])) - (-177.63641)) < 1E-4)

def test_new_dihedral2():
    assert(abs(new_dihedral(np.array([p0, p1, p2, p3])) - (-71.21515)) < 1E-4)
    assert(abs(new_dihedral(np.array([p0, p1, p4, p5])) - (-171.94319)) < 1E-4)
    assert(abs(new_dihedral(np.array([p1, p4, p5, p6])) - (60.82226)) < 1E-4)
    assert(abs(new_dihedral(np.array([p1, p4, p5, p7])) - (-177.63641)) < 1E-4)
```
Code for timing: time_dihedrals.py
```
#!/usr/bin/env python
# -*- coding: utf-8 -*-

from dihedrals import *
from time import time

def profileDihedrals(f):
    t0 = time()
    for i in range(20000):
        p = np.random.random( (4,3) )
        f(p)
        p = np.random.randn( 4,3 )
        f(p)
    return(time() - t0)

print("old_dihedral2: ", profileDihedrals(old_dihedral2))
print("wiki_dihedral: ", profileDihedrals(wiki_dihedral))
print("new_dihedral:  ", profileDihedrals(new_dihedral))
```
The functions can be tested with pytest as pytest ./test_dihedrals.py. Timing results:
```
./time_dihedrals.py
old_dihedral2: 1.6442952156066895
wiki_dihedral: 1.3895585536956787
new_dihedral:  0.8703620433807373
```
new_dihedral is about twice as fast as old_dihedral2. ...you can also see that the hardware used for this answer is a lot beefier than the hardware used in the question (3.74 vs 1.64 for dihedral2) ;-P If you want to get even more aggressive you can use pypy. At the time of writing pypy doesn't support numpy.cross but you can just use a cross product implemented in python instead. For a 3-vector cross product the C pypy generates is probably at least as good as what numpy uses. Doing so gets the time down to 0.60 for me but at this we're wading into silly hax. Same benchmark but with same hardware as used in the question:
```
old_dihedral2: 3.0171279907226562
wiki_dihedral: 3.415065050125122
new_dihedral:  2.086946964263916
```


Question: How to randomly shuffle data and target in python? I have a 4D array training images, whose dimensions correspond to (num_images, channels, width, height). I also have a 2D target labels whose dimensions correspond to (num_images, class_number). When training, I want to randomly shuffle the data by using random.shuffle, but how can I keep the labels shuffled in the same order as my images?

Response:
```py
from sklearn.utils import shuffle
import numpy as np

X = np.array([
  [0, 0, 0],
  [1, 1, 1],
  [2, 2, 2],
  [3, 3, 3],
  [4, 4, 4]
])
y = np.array([0, 1, 2, 3, 4])
X, y = shuffle(X, y)
print(X)
print(y)
```
Output:
```py
[[1 1 1]
 [3 3 3]
 [0 0 0]
 [2 2 2]
 [4 4 4]]

[1 3 0 2 4]
```


Question:
Numpy Array Get row index searching by a row I am new to numpy and I am implementing clustering with random forest in python. My question is: How could I find the index of the exact row in an array? For example
```
[[ 0.  5.  2.]
 [ 0.  0.  3.]
 [ 0.  0.  0.]]
```
and I look for [0. 0. 3.] and get as result 1(the index of the second row). Any suggestion? Follows the code (not working...)
```
for index, element in enumerate(leaf_node.x):
        for index_second_element, element_two in enumerate(leaf_node.x):
            if (index <= index_second_element):
                index_row = np.where(X == element)
                index_column = np.where(X == element_two)
                self.similarity_matrix[index_row][index_column] += 1
```

Response:
Why not simply do something like this?
```

```python
a
#Output
#array([[ 0.,  5.,  2.],
#       [ 0.,  0.,  3.],
#       [ 0.,  0.,  0.]])
```

```python
b
#Output
#array([ 0.,  0.,  3.])
```

```python
a==b
#Output
#array([[ True, False, False],
#       [ True,  True,  True],
#       [ True,  True, False]], dtype=bool)
```

```python
np.all(a==b,axis=1)
#Output
#array([False,  True, False], dtype=bool)
```

```python
np.where(np.all(a==b,axis=1))
#Output
#(array([1]),)
#```
```


Question:
Python numpy array of numpy arrays I've got a problem on creating a numpy array of numpy arrays. I would create it in a loop:
```
a=np.array([])
while(...):
   ...
   b= //a numpy array generated
   a=np.append(a,b)
   ...
```
Desired result:
```
[[1,5,3], [9,10,1], ..., [4,8,6]]
```
Real result:
```
[1,5,3,9,10,1,... 4,8,6]
```
Is it possible? I don't know the final dimension of the array, so I can't initialize it with a fixed dimension.

Response:
Never append to numpy arrays in a loop: it is the one operation that NumPy is very bad at compared with basic Python. This is because you are making a full copy of the data each append, which will cost you quadratic time. Instead, just append your arrays to a Python list and convert it at the end; the result is simpler and faster:
```
a = []

while ...:
    b = ... # NumPy array
    a.append(b)
a = np.asarray(a)
```
As for why your code doesn't work: np.append doesn't behave like list.append at all. In particular, it won't create new dimensions when appending. You would have to create the initial array with two dimensions, then append with an explicit axis argument.


Question:
Python: How to get values of an array at certain index positions? I have a numpy array like this:
```
a = [0,88,26,3,48,85,65,16,97,83,91]
```
How can I get the values at certain index positions in ONE step? For example:
```
ind_pos = [1,5,7]
```
The result should be:
```
[88,85,16]
```

Response:
Just index using you ind_pos
```
ind_pos = [1,5,7]
print (a[ind_pos])
[88 85 16]

```python
a = [0,88,26,3,48,85,65,16,97,83,91]
```

```python
import numpy as np
```

```python
arr = np.array(a)
```

```python
ind_pos = [1,5,7]
```

```python
arr[ind_pos]
#Output
#array([88, 85, 16])
#```
```


Question:
Numpy where() on a 2D matrix I have a matrix like this
```
t = np.array([[1,2,3,'foo'],
 [2,3,4,'bar'],
 [5,6,7,'hello'],
 [8,9,1,'bar']])
```
I want to get the indices where the rows contain the string 'bar' In a 1d array
```
rows = np.where(t == 'bar')
```
should give me the indices [0,3] followed by broadcasting:-
```
results = t[rows]
```
should give me the right rows But I can't figure out how to get it to work with 2d arrays.

Response:
For the general case, where your search string can be in any column, you can do this:
```

```python
rows, cols = np.where(t == 'bar')
```

```python
t[rows]
#Output
#array([['2', '3', '4', 'bar'],
#       ['8', '9', '1', 'bar']],
#      dtype='|S11')
#```
```


Question: python - RGB matrix of an image Taking an image as input, how can I get the rgb matrix corresponding to it? I checked out the numpy.asarray function. Does that give me the rgb matrix or some other matrix?

Response:
Note that this answer is outdated as of 2018; scipy has deprecated imread, and you should switch to imageio.imread. See this transition doc about differences between the two. The code below should work with no changes if you just import the new library in place of the old, but I haven’t tested it. The simplest answer is to use the NumPy and SciPy wrappers around PIL. There's a great tutorial, but the basic idea is:
```
from scipy import misc
arr = misc.imread('lena.png') # 640x480x3 array
arr[20, 30] # 3-vector for a pixel
arr[20, 30, 1] # green value for a pixel
```
For a 640x480 RGB image, this will give you a 640x480x3 array of uint8. Or you can just open the file with PIL (or, rather, Pillow; if you're still using PIL, this may not work, or may be very slow) and pass it straight to NumPy:
```
import numpy as np
from PIL import Image
img = Image.open('lena.png')
arr = np.array(img) # 640x480x4 array
arr[20, 30] # 4-vector, just like above
```
This will give you a 640x480x4 array of type uint8 (the 4th is alpha; PIL always loads PNG files as RGBA, even if they have no transparency; see img.getbands() if you're every unsure). If you don't want to use NumPy at all, PIL's own PixelArray type is a more limited array:
```
arr = img.load()
arr[20, 30] # tuple of 4 ints
```
This gives you a 640x480 PixelAccess array of RGBA 4-tuples. Or you can just call getpixel on the image:
```
img.getpixel(20, 30) # tuple of 4 ints
```


Question:
plot data from CSV file with matplotlib I have a CSV file at e:\dir1\datafile.csv. It contains three columns and 10 heading and trailing lines need to be skipped. I would like to plot it with numpy.loadtxt(), for which I haven't found any rigorous documentation. Here is what I started to write from the several tries I found on the web.
```
import matplotlib as mpl
import numpy as np
import matplotlib.pyplot as plt
import matplotlib.cbook as cbook

def read_datafile(file_name):
    # the skiprows keyword is for heading, but I don't know if trailing lines
    # can be specified
    data = np.loadtxt(file_name, delimiter=',', skiprows=10)
    return data

data = read_datafile('e:\dir1\datafile.csv')

x = ???
y = ???

fig = plt.figure()

ax1 = fig.add_subplot(111)

ax1.set_title("Mains power stability")
ax1.set_xlabel('time')
ax1.set_ylabel('Mains voltage')

ax1.plot(x,y, c='r', label='the data')

leg = ax1.legend()

plt.show()
```

Response:
According to the docs numpy.loadtxt is a fast reader for simply formatted files. The genfromtxt function provides more sophisticated handling of, e.g., lines with missing values. so there are only a few options to handle more complicated files. As mentioned numpy.genfromtxt has more options. So as an example you could use
```
import numpy as np
data = np.genfromtxt('e:\dir1\datafile.csv', delimiter=',', skip_header=10,
                     skip_footer=10, names=['x', 'y', 'z'])
```
to read the data and assign names to the columns (or read a header line from the file with names=True) and than plot it with
```
ax1.plot(data['x'], data['y'], color='r', label='the data')
```
I think numpy is quite well documented now. You can easily inspect the docstrings from within ipython or by using an IDE like spider if you prefer to read them rendered as HTML.


Question:
Why is numpy.linalg.pinv() preferred over numpy.linalg.inv() for creating inverse of a matrix in linear regression If we want to search for the optimal parameters theta for a linear regression model by using the normal equation with: theta = inv(X^T * X) * X^T * y one step is to calculate inv(X^T*X). Therefore numpy provides np.linalg.inv() and np.linalg.pinv() Though this leads to different results:
```
X=np.matrix([[1,2104,5,1,45],[1,1416,3,2,40],[1,1534,3,2,30],[1,852,2,1,36]])
y=np.matrix([[460],[232],[315],[178]])

XT=X.T
XTX=XT@X

pinv=np.linalg.pinv(XTX)
theta_pinv=(pinv@XT)@y
print(theta_pinv)

[[188.40031946]
 [  0.3866255 ]
 [-56.13824955]
 [-92.9672536 ]
 [ -3.73781915]]

inv=np.linalg.inv(XTX)
theta_inv=(inv@XT)@y
print(theta_inv)

[[-648.7890625 ]
 [   0.79418945]
 [-110.09375   ]
 [ -74.0703125 ]
 [  -3.69091797]]
```
The first output, that is the output of pinv is the correct one and additionally recommended in the numpy.linalg.pinv() docs. But why is this and where are the differences / Pros / Cons between inv() and pinv().

Response: If the determinant of the matrix is zero it will not have an inverse and your inv function will not work. This usually happens if your matrix is singular. But pinv will. This is because pinv returns the inverse of your matrix when it is available and the pseudo inverse when it isn't. The different results of the functions are because of rounding errors in floating point arithmetic You can read more about how pseudo inverse works here


Question:
How can I efficiently process a numpy array in blocks similar to Matlab's blkproc (blockproc) function I'm looking for a good approach for efficiently dividing an image into small regions, processing each region separately, and then re-assembling the results from each process into a single processed image. Matlab had a tool for this called blkproc (replaced by blockproc in newer versions of Matlab). In an ideal world, the function or class would support overlap between the divisions in the input matrix too. In the Matlab help, blkproc is defined as: B = blkproc(A,[m n],[mborder nborder],fun,...) A is your input matrix, [m n] is the block size [mborder, nborder] is the size of your border region (optional) fun is a function to apply to each block I have kluged together an approach, but it strikes me as clumsy and I bet there's a much better way. At the risk of my own embarrassment, here's my code:
```
import numpy as np

def segmented_process(M, blk_size=(16,16), overlap=(0,0), fun=None):
    rows = []
    for i in range(0, M.shape[0], blk_size[0]):
        cols = []
        for j in range(0, M.shape[1], blk_size[1]):
            cols.append(fun(M[i:i+blk_size[0], j:j+blk_size[1]]))
        rows.append(np.concatenate(cols, axis=1))
    return np.concatenate(rows, axis=0)

R = np.random.rand(128,128)
passthrough = lambda(x):x
Rprime = segmented_process(R, blk_size=(16,16),
                           overlap=(0,0),
                           fun=passthrough)

np.all(R==Rprime)
```

Response:
Here are some examples of a different (loop free) way to work with blocks:
```
import numpy as np
from numpy.lib.stride_tricks import as_strided as ast

A= np.arange(36).reshape(6, 6)
print A
#[[ 0  1  2  3  4  5]
# [ 6  7  8  9 10 11]
# ...
# [30 31 32 33 34 35]]

# 2x2 block view
B= ast(A, shape= (3, 3, 2, 2), strides= (48, 8, 24, 4))
print B[1, 1]
#[[14 15]
# [20 21]]

# for preserving original shape
B[:, :]= np.dot(B[:, :], np.array([[0, 1], [1, 0]]))
print A
#[[ 1  0  3  2  5  4]
# [ 7  6  9  8 11 10]
# ...
# [31 30 33 32 35 34]]
print B[1, 1]
#[[15 14]
# [21 20]]

# for reducing shape, processing in 3D is enough
C= B.reshape(3, 3, -1)
print C.sum(-1)
#[[ 14  22  30]
# [ 62  70  78]
# [110 118 126]]
```
So just trying to simply copy the matlab functionality to numpy is not all ways the best way to proceed. Sometimes a 'off the hat' thinking is needed. Caveat: In general, implementations based on stride tricks may (but does not necessary need to) suffer some performance penalties. So be prepared to all ways measure your performance. In any case it's wise to first check if the needed functionality (or similar enough, in order to easily adapt for) has all ready been implemented in numpy or scipy. Update: Please note that there is no real magic involved here with the strides, so I'll provide a simple function to get a block_view of any suitable 2D numpy-array. So here we go:
```
from numpy.lib.stride_tricks import as_strided as ast

def block_view(A, block= (3, 3)):
    """Provide a 2D block view to 2D array. No error checking made.
    Therefore meaningful (as implemented) only for blocks strictly
    compatible with the shape of A."""
    # simple shape and strides computations may seem at first strange
    # unless one is able to recognize the 'tuple additions' involved ;-)
    shape= (A.shape[0]/ block[0], A.shape[1]/ block[1])+ block
    strides= (block[0]* A.strides[0], block[1]* A.strides[1])+ A.strides
    return ast(A, shape= shape, strides= strides)

if __name__ == '__main__':
    from numpy import arange
    A= arange(144).reshape(12, 12)
    print block_view(A)[0, 0]
    #[[ 0  1  2]
    # [12 13 14]
    # [24 25 26]]
    print block_view(A, (2, 6))[0, 0]
    #[[ 0  1  2  3  4  5]
    # [12 13 14 15 16 17]]
    print block_view(A, (3, 12))[0, 0]
    #[[ 0  1  2  3  4  5  6  7  8  9 10 11]
    # [12 13 14 15 16 17 18 19 20 21 22 23]
    # [24 25 26 27 28 29 30 31 32 33 34 35]]
```


Question:
What is the difference between numpy var() and statistics variance() in python? I was trying one Dataquest exercise and I figured out that the variance I am getting is different for the two packages. e.g for [1,2,3,4]
```
from statistics import variance
import numpy as np
print(np.var([1,2,3,4]))
print(variance([1,2,3,4]))
//1.25
//1.6666666666666667
```
The expected answer of the exercise is calculated with np.var() Edit I guess it has to do that the later one is sample variance and not variance. Anyone could explain the difference?

Response:
Use this
```
print(np.var([1,2,3,4],ddof=1))

1.66666666667
```
Delta Degrees of Freedom: the divisor used in the calculation is N - ddof, where N represents the number of elements. By default, ddof is zero. The mean is normally calculated as x.sum() / N, where N = len(x). If, however, ddof is specified, the divisor N - ddof is used instead. In standard statistical practice, ddof=1 provides an unbiased estimator of the variance of a hypothetical infinite population. ddof=0 provides a maximum likelihood estimate of the variance for normally distributed variables. Statistical libraries like numpy use the variance n for what they call var or variance and the standard deviation For more information refer this documentation : numpy doc


Question:
python - invalid value encountered in log I have the following expression: log = np.sum(np.nan_to_num(-y*np.log(a+ 1e-7)-(1-y)*np.log(1-a+ 1e-7))) it is giving me the following warning:
```
RuntimeWarning: invalid value encountered in log
  log = np.sum(np.nan_to_num(-y*np.log(a+ 1e-7)-(1-y)*np.log(1-a+ 1e-7)))
```
I don't understand what might be the invalid value or why am I getting it. Any and every help is appreciated. NOTE: This is a cross-entropy cost function where I added 1e-7 to avoid having zeros inside log. y & a are numpy arrays and numpy is imported as np.

Response: You probably still have negative values inside the log, which gives nan with real numbers. a and y should represent probability between 0 to 1, So you need to check why do you have smaller/larger values there. Adding 1e-7 shows there is something fishy, because np.log(0) gives -inf, which I think is the value you want.


Question: NumPy types with underscore: `int_`, `float_`, etc What is the significance of the underscore suffixing in int_, float_, etc.?

Response: From page 21 of Guide to Numpy by TE Oliphant: Names for the data types that would clash with standard Python object names are followed by a trailing underscore, ’ ’. These data types are so named because they use the same underlying precision as the corresponding Python data types. . . . The array types bool_, int_, complex_, float_, object_, unicode_, and str_ are enhanced-scalars. They are very similar to the standard Python types (without the trailing underscore) and inherit from them (except for bool_ and object_). They can be used in place of the standard Python types whenever desired. Whenever a data type is required, as an argument, the standard Python types are recognized as well.


Question:
Weird indexing using numpy I have a variable, x, that is of the shape (2,2,50,100). I also have an array, y, that equals np.array([0,10,20]). A weird thing happens when I index x[0,:,:,y].
```
x = np.full((2,2,50,100),np.nan)
y = np.array([0,10,20])
print(x.shape)
(2,2,50,100)
print(x[:,:,:,y].shape)
(2,2,50,3)
print(x[0,:,:,:].shape)
(2,50,100)
print(x[0,:,:,y].shape)
(3,2,50)
```
Why does the last one output (3,2,50) and not (2,50,3)?

Response:
This is how numpy uses advanced indexing to broadcast array shapes. When you pass a 0 for the first index, and y for the last index, numpy will broadcast the 0 to be the same shape as y. The following equivalence holds: x[0,:,:,y] == x[(0, 0, 0),:,:,y]. here is an example
```
import numpy as np

x = np.arange(120).reshape(2,3,4,5)
y = np.array([0,2,4])

np.equal(x[0,:,:,y], x[(0, 0, 0),:,:,y]).all()
# returns:
True
```
Now, because you are effectively passing in two sets of indices, you are using the advanced indexing API to form (in this case) pairs of indices.
```
x[(0, 0, 0),:,:,y])

# equivalent to
[
  x[0,:,:,y[0]],
  x[0,:,:,y[1]],
  x[0,:,:,y[2]]
]

# equivalent to
rows = np.array([0, 0, 0])
cols = y
x[rows,:,:,cols]

# equivalent to
[
  x[r,:,:,c] for r, c in zip(rows, columns)
]
```
Which has a first dimension that same as the length of y. This is what you are seeing. As an example, look at an array with 4 dimensions which are described in the next chunk:
```
x = np.arange(120).reshape(2,3,4,5)
y = np.array([0,2,4])

# x looks like:
array([[[[  0,   1,   2,   3,   4],    -+      =+
         [  5,   6,   7,   8,   9],     Sheet1  |
         [ 10,  11,  12,  13,  14],     |       |
         [ 15,  16,  17,  18,  19]],   -+       |
                                                Workbook1
        [[ 20,  21,  22,  23,  24],    -+       |
         [ 25,  26,  27,  28,  29],     Sheet2  |
         [ 30,  31,  32,  33,  34],     |       |
         [ 35,  36,  37,  38,  39]],   -+       |
                                                |
        [[ 40,  41,  42,  43,  44],    -+       |
         [ 45,  46,  47,  48,  49],     Sheet3  |
         [ 50,  51,  52,  53,  54],     |       |
         [ 55,  56,  57,  58,  59]]],  -+      =+

       [[[ 60,  61,  62,  63,  64],
         [ 65,  66,  67,  68,  69],
         [ 70,  71,  72,  73,  74],
         [ 75,  76,  77,  78,  79]],

        [[ 80,  81,  82,  83,  84],
         [ 85,  86,  87,  88,  89],
         [ 90,  91,  92,  93,  94],
         [ 95,  96,  97,  98,  99]],

        [[100, 101, 102, 103, 104],
         [105, 106, 107, 108, 109],
         [110, 111, 112, 113, 114],
         [115, 116, 117, 118, 119]]]])
```
x has a really easy to understand sequential form that we can now use to show what is happening... The first dimension is like having 2 Excel Workbooks, the second dimension is like having 3 sheets in each workbook, the third dimension is like having 4 rows per sheet, and the last dimension is 5 values for each row (or columns per sheet). Looking at it this way, asking for x[0,:,:,0], is the saying: "in the first workbook, for each sheet, for each row, give me the first value/column."
```
x[0,:,:,y[0]]
# returns:
array([[ 0,  5, 10, 15],
       [20, 25, 30, 35],
       [40, 45, 50, 55]])

# this is in the same as the first element in:
x[(0,0,0),:,:,y]
```
But now with advanced indexing, we can think of x[(0,0,0),:,:,y] as "in the first workbook, for each sheet, for each row, give me the yth value/column. Ok, now do it for each value of y"
```
x[(0,0,0),:,:,y]
# returns:
array([[[ 0,  5, 10, 15],
        [20, 25, 30, 35],
        [40, 45, 50, 55]],

       [[ 2,  7, 12, 17],
        [22, 27, 32, 37],
        [42, 47, 52, 57]],

       [[ 4,  9, 14, 19],
        [24, 29, 34, 39],
        [44, 49, 54, 59]]])
```
Where it gets crazy is that numpy will broadcast to match the outer dimensions of index array. So if you want to do that same operation as above, but for BOTH "Excel workbooks", you don't have to loop and concatenate. You can just pass an array to the first dimension, but it MUST have a compatible shape. Passing an integer gets broadcast to y.shape == (3,). If you want to pass an array as the first index, only the last dimension of the array has to be compatible with y.shape. I.e., the last dimension of the first index must either be 3 or 1.
```py
ix = np.array([[0], [1]])
x[ix,:,:,y].shape
# each row of ix is broadcast to length 3:
(2, 3, 3, 4)

ix = np.array([[0,0,0], [1,1,1]])
x[ix,:,:,y].shape
# this is identical to above:
(2, 3, 3, 4)

ix = np.array([[0], [1], [0], [1], [0]])
x[ix,:,:,y].shape
# ix is broadcast so each row of ix has 3 columns, the length of y
(5, 3, 3, 4)
```
Found a short explanation in the docs: https://docs.scipy.org/doc/numpy/reference/arrays.indexing.html#combining-advanced-and-basic-indexing Edit: From the original question, to get a one-liner of your desired subslicing, you can use x[0][:,:,y]:
```
x[0][:,:,y].shape
# returns
(2, 50, 3)
```
However, if you are trying to assign to those subslices, you have to be very careful that you are looking at a shared memory view of the original array. Otherwise the assignment won't be to the original array, but a copy. Shared memory only occurs when you are use an integer or slice to subset your array, i.e. x[:,0:3,:,:] or x[0,:,:,1:-1].
```py
np.shares_memory(x, x[0])
# returns:
True

np.shares_memory(x, x[:,:,:,y])
# returns:
False
```
In both your original question and my example y is neither an int or a slice, so will always end up assigning to a copy of the original. BUT! Because your array for y can be expressed as a slice, you CAN actually get an assignable view of your array via:
```py
x[0,:,:,0:21:10].shape
# returns:
(2, 50, 3)

np.shares_memory(x, x[0,:,:,0:21:10])
# returns:
True

# actually assigns to the original array
x[0,:,:,0:21:10] = 100
```
Here we use the slice 0:21:10 to grab every index that would be in range(0,21,10). We have to use 21 and not 20 because the stop-point is excluded from the slice, just like in the range function. So basically, if you can construct a slice that fits your subslicing criteria, you can do assignment.


Question: Perform 2 sample t-test I have a the mean, std dev and n of sample 1 and sample 2 - samples are taken from the sample population, but measured by different labs. n is different for sample 1 and sample 2. I want to do a weighted (take n into account) two-tailed t-test. I tried using the scipy.stat module by creating my numbers with np.random.normal, since it only takes data and not stat values like mean and std dev (is there any way to use these values directly). But it didn't work since the data arrays has to be of equal size. Any help on how to get the p-value would be highly appreciated.

Response:
If you have the original data as arrays a and b, you can use scipy.stats.ttest_ind with the argument equal_var=False:
```
t, p = ttest_ind(a, b, equal_var=False)
```
If you have only the summary statistics of the two data sets, you can calculate the t value using scipy.stats.ttest_ind_from_stats (added to scipy in version 0.16) or from the formula (http://en.wikipedia.org/wiki/Welch%27s_t_test). The following script shows the possibilities.
```
from __future__ import print_function

import numpy as np
from scipy.stats import ttest_ind, ttest_ind_from_stats
from scipy.special import stdtr

np.random.seed(1)

# Create sample data.
a = np.random.randn(40)
b = 4*np.random.randn(50)

# Use scipy.stats.ttest_ind.
t, p = ttest_ind(a, b, equal_var=False)
print("ttest_ind:            t = %g  p = %g" % (t, p))

# Compute the descriptive statistics of a and b.
abar = a.mean()
avar = a.var(ddof=1)
na = a.size
adof = na - 1

bbar = b.mean()
bvar = b.var(ddof=1)
nb = b.size
bdof = nb - 1

# Use scipy.stats.ttest_ind_from_stats.
t2, p2 = ttest_ind_from_stats(abar, np.sqrt(avar), na,
                              bbar, np.sqrt(bvar), nb,
                              equal_var=False)
print("ttest_ind_from_stats: t = %g  p = %g" % (t2, p2))

# Use the formulas directly.
tf = (abar - bbar) / np.sqrt(avar/na + bvar/nb)
dof = (avar/na + bvar/nb)**2 / (avar**2/(na**2*adof) + bvar**2/(nb**2*bdof))
pf = 2*stdtr(dof, -np.abs(tf))

print("formula:              t = %g  p = %g" % (tf, pf))
```
The output:
```
ttest_ind:            t = -1.5827  p = 0.118873
ttest_ind_from_stats: t = -1.5827  p = 0.118873
formula:              t = -1.5827  p = 0.118873
```


Question:
Generate random array of 0 and 1 with a specific ratio I want to generate a random array of size N which only contains 0 and 1, I want my array to have some ratio between 0 and 1. For example, 90% of the array be 1 and the remaining 10% be 0 (I want this 90% to be random along with the whole array). right now I have:
```
randomLabel = np.random.randint(2, size=numbers)
```
But I can't control the ratio between 0 and 1.

Response:
If you want an exact 1:9 ratio:
```
nums = numpy.ones(1000)
nums[:100] = 0
numpy.random.shuffle(nums)
```
If you want independent 10% probabilities:
```
nums = numpy.random.choice([0, 1], size=1000, p=[.1, .9])
```
or
```
nums = (numpy.random.rand(1000) > 0.1).astype(int)
```


Question:
Python: Setting an element of a Numpy matrix I am a pretty new to python. I have created an empty matrix
```
a = numpy.zeros(shape=(n,n))
```
Now I can access each element using
```
a.item(i,j)
```
How do I set an index (i,j)?

Response:
Here's how:
```
a[i,j] = x
```


Question:
Loading text file containing both float and string using numpy.loadtxt I have a text file, data.txt, which contains:
```
5.1,3.5,1.4,0.2,Iris-setosa
4.9,3.0,1.4,0.2,Iris-setosa
5.8,2.7,4.1,1.0,Iris-versicolor
6.2,2.2,4.5,1.5,Iris-versicolor
6.4,3.1,5.5,1.8,Iris-virginica
6.0,3.0,4.8,1.8,Iris-virginica
```
How do I load this data using numpy.loadtxt() so that I get a NumPy array after loading such as [['5.1' '3.5' '1.4' '0.2' 'Iris-setosa'] ['4.9' '3.0' '1.4' '0.2' 'Iris-setosa'] ...]? I tried
```
np.loadtxt(open("data.txt"), 'r',
           dtype={
               'names': (
                   'sepal length', 'sepal width', 'petal length',
                   'petal width', 'label'),
               'formats': (
                   np.float, np.float, np.float, np.float, np.str)},
           delimiter= ',', skiprows=0)
```

Response:
If you use np.genfromtxt, you could specify dtype=None, which will tell genfromtxt to intelligently guess the dtype of each column. Most conveniently, it relieves you of the burder of specifying the number of bytes required for the string column. (Omitting the number of bytes, by specifying e.g. np.str, does not work.)
```

```python
np.genfromtxt('data.txt', delimiter=',', dtype=None, names=('sepal length', 'sepal width', 'petal length', 'petal width', 'label'))
#Output
#array([(5.1, 3.5, 1.4, 0.2, 'Iris-setosa'),
#       (4.9, 3.0, 1.4, 0.2, 'Iris-setosa'),
#       (5.8, 2.7, 4.1, 1.0, 'Iris-versicolor'),
#       (6.2, 2.2, 4.5, 1.5, 'Iris-versicolor'),
#       (6.4, 3.1, 5.5, 1.8, 'Iris-virginica'),
#       (6.0, 3.0, 4.8, 1.8, 'Iris-virginica')],
#      dtype=[('sepal_length', '<f8'), ('sepal_width', '<f8'), ('petal_length', '<f8'), ('petal_width', '<f8'), ('label', 'S15')])
#```
#If you do want to use np.loadtxt, then to fix your code with minimal changes, you could use:
#```
#np.loadtxt("data.txt",
#   dtype={'names': ('sepal length', 'sepal width', 'petal length', 'petal width', 'label'),
#          'formats': (np.float, np.float, np.float, np.float, '|S15')},
#   delimiter=',', skiprows=0)
#```
#The main difference is simply changing np.str to |S15 (a 15-byte string). Also note that open("data.txt"), 'r' should be open("data.txt", 'r'). But since np.loadtxt can accept a filename, you don't really need to use open at all.
```


Question:
numpy/scipy equivalent of R ecdf(x)(x) function? What is the equivalent of R's ecdf(x)(x) function in Python, in either numpy or scipy? Is ecdf(x)(x) basically the same as:
```
import numpy as np
def ecdf(x):
  # normalize X to sum to 1
  x = x / np.sum(x)
  return np.cumsum(x)
```
or is something else required? EDIT how can one control the number of bins used by ecdf?

Response:
The OP implementation for ecdf is wrong, you are not supposed to cumsum() the values. So not ys = np.cumsum(x)/np.sum(x) but ys = np.cumsum(1 for _ in x)/float(len(x)) or better ys = np.arange(1, len(x)+1)/float(len(x)) You either go with statmodels's ECDF if you are OK with that extra dependency or provide your own implementation. See below:
```
import numpy as np
import matplotlib.pyplot as plt
from statsmodels.distributions.empirical_distribution import ECDF
%matplotlib inline

grades = (93.5,93,60.8,94.5,82,87.5,91.5,99.5,86,93.5,92.5,78,76,69,94.5,
          89.5,92.8,78,65.5,98,98.5,92.3,95.5,76,91,95,61)

def ecdf_wrong(x):
    xs = np.sort(x) # need to be sorted
    ys = np.cumsum(xs)/np.sum(xs) # normalize so sum == 1
    return (xs,ys)
def ecdf(x):
    xs = np.sort(x)
    ys = np.arange(1, len(xs)+1)/float(len(xs))
    return xs, ys

xs, ys = ecdf_wrong(grades)
plt.plot(xs, ys, label="wrong cumsum")
xs, ys = ecdf(grades)
plt.plot(xs, ys, label="handwritten", marker=">", markerfacecolor='none')
cdf = ECDF(grades)
plt.plot(cdf.x, cdf.y, label="statmodels", marker="<", markerfacecolor='none')
plt.legend()
plt.show()
```


Question: Fitting a line in 3D Are there any algorithms that will return the equation of a straight line from a set of 3D data points? I can find plenty of sources which will give the equation of a line from 2D data sets, but none in 3D.

Response:
If you are trying to predict one value from the other two, then you should use lstsq with the a argument as your independent variables (plus a column of 1's to estimate an intercept) and b as your dependent variable. If, on the other hand, you just want to get the best fitting line to the data, i.e. the line which, if you projected the data onto it, would minimize the squared distance between the real point and its projection, then what you want is the first principal component. One way to define it is the line whose direction vector is the eigenvector of the covariance matrix corresponding to the largest eigenvalue, that passes through the mean of your data. That said, eig(cov(data)) is a really bad way to calculate it, since it does a lot of needless computation and copying and is potentially less accurate than using svd. See below:
```
import numpy as np

# Generate some data that lies along a line

x = np.mgrid[-2:5:120j]
y = np.mgrid[1:9:120j]
z = np.mgrid[-5:3:120j]

data = np.concatenate((x[:, np.newaxis],
                       y[:, np.newaxis],
                       z[:, np.newaxis]),
                      axis=1)

# Perturb with some Gaussian noise
data += np.random.normal(size=data.shape) * 0.4

# Calculate the mean of the points, i.e. the 'center' of the cloud
datamean = data.mean(axis=0)

# Do an SVD on the mean-centered data.
uu, dd, vv = np.linalg.svd(data - datamean)

# Now vv[0] contains the first principal component, i.e. the direction
# vector of the 'best fit' line in the least squares sense.

# Now generate some points along this best fit line, for plotting.

# I use -7, 7 since the spread of the data is roughly 14
# and we want it to have mean 0 (like the points we did
# the svd on). Also, it's a straight line, so we only need 2 points.
linepts = vv[0] * np.mgrid[-7:7:2j][:, np.newaxis]

# shift by the mean to get the line in the right place
linepts += datamean

# Verify that everything looks right.

import matplotlib.pyplot as plt
import mpl_toolkits.mplot3d as m3d

ax = m3d.Axes3D(plt.figure())
ax.scatter3D(*data.T)
ax.plot3D(*linepts.T)
plt.show()
```
Here's what it looks like: