Question:
FFT in Matlab and numpy / scipy give different results I am trying to re-implement one of the matlab toolboxes. they use fft over there. when i perform same operation on the same data i get different results to those from matlab. Just take a look: MATLAB:
```
Msig =

 0     0     0     0
 0     0     0     0
 0     0     0     0
 0     0     0     0
 0     1     0     0
 0     0     0     0

fft(Msig.')

Columns 1 through 4

    0                  0                  0                  0
    0                  0                  0                  0
    0                  0                  0                  0
    0                  0                  0                  0

Columns 5 through 6

 1.0000                  0
      0 - 1.0000i        0
-1.0000                  0
      0 + 1.0000i        0
```
PYTHON:
```
Msig=
array([[ 0.,  0.,  0.,  0.],
       [ 0.,  0.,  0.,  0.],
       [ 0.,  0.,  0.,  0.],
       [ 0.,  0.,  0.,  0.],
       [ 0.,  1.,  0.,  0.],
       [ 0.,  0.,  0.,  0.]])

np.fft.fft(Msig.transpose())
array([[ 0.0 +0.00000000e+00j,  0.0 +0.00000000e+00j,
         0.0 +0.00000000e+00j,  0.0 +0.00000000e+00j,
         0.0 +0.00000000e+00j,  0.0 +0.00000000e+00j],
       [ 1.0 +0.00000000e+00j, -0.5 +8.66025404e-01j,
        -0.5 -8.66025404e-01j,  1.0 -3.88578059e-16j,
        -0.5 +8.66025404e-01j, -0.5 -8.66025404e-01j],
       [ 0.0 +0.00000000e+00j,  0.0 +0.00000000e+00j,
         0.0 +0.00000000e+00j,  0.0 +0.00000000e+00j,
         0.0 +0.00000000e+00j,  0.0 +0.00000000e+00j],
       [ 0.0 +0.00000000e+00j,  0.0 +0.00000000e+00j,
        0.0 +0.00000000e+00j,  0.0 +0.00000000e+00j,
         0.0 +0.00000000e+00j,  0.0 +0.00000000e+00j]])
```
The best i can get if i mess with parameters(axis etc.) of np.fft.fft()/np.fft.fft2()/np.fft.fftn() is same values but shifted. unfortunately manual shifting is not an option cause the size and shape of the Msig matrix varies depending on input parameters. you have any clue how to solve this problem, what can be the cause?

Response:
Matlab applies the fft over the columns of the matrix, numpy applies the fft over the last axis (the rows) by default. You want:
```

```python
np.fft.fft(Msig.T, axis=0)
#Output
#array([[ 0.+0.j,  0.+0.j,  0.+0.j,  0.+0.j,  1.+0.j,  0.+0.j],
#       [ 0.+0.j,  0.+0.j,  0.+0.j,  0.+0.j,  0.-1.j,  0.+0.j],
#       [ 0.+0.j,  0.+0.j,  0.+0.j,  0.+0.j, -1.+0.j,  0.+0.j],
#       [ 0.+0.j,  0.+0.j,  0.+0.j,  0.+0.j,  0.+1.j,  0.+0.j]])
#```
#or
#```
```

```python
np.fft.fft(Msig).T
#Output
#array([[ 0.+0.j,  0.+0.j,  0.+0.j,  0.+0.j,  1.+0.j,  0.+0.j],
#       [ 0.+0.j,  0.+0.j,  0.+0.j,  0.+0.j,  0.-1.j,  0.+0.j],
#       [ 0.+0.j,  0.+0.j,  0.+0.j,  0.+0.j, -1.+0.j,  0.+0.j],
#       [ 0.+0.j,  0.+0.j,  0.+0.j,  0.+0.j,  0.+1.j,  0.+0.j]])
#```
```


Question:
How to get value from a theano tensor variable backed by a shared variable? I have a theano tensor variable created from casting a shared variable. How can I extract the original or casted values? (I need that so I don't have to carry the original shared/numpy values around.)
```

```python
x = theano.shared(numpy.asarray([1, 2, 3], dtype='float'))
```

```python
y = theano.tensor.cast(x, 'int32')
```

```python
y.get_value(borrow=True)
#Output
#Traceback (most recent call last):
#  File "<stdin>", line 1, in <module>
#AttributeError: 'TensorVariable' object has no attribute 'get_value'
## whereas I can do this against the original shared variable
```

```python
x.get_value(borrow=True)
#Output
#array([ 1.,  2.,  3.])
#```
```

Response:
get_value only works for shared variables. TensorVariables are general expressions and thus potentially need extra input in order to be able to determine their value (Imagine you set y = x + z, where z is another tensor variable. You would need to specify z before being able to calculate y). You can either create a function to provide this input or provide it in a dictionary using the eval method. In your case, y only depends on x, so you can do
```
import theano
import theano.tensor as T

x = theano.shared(numpy.asarray([1, 2, 3], dtype='float32'))
y = T.cast(x, 'int32')
y.eval()
```
and you should see the result
```
array([1, 2, 3], dtype=int32)
```
(And in the case y = x + z, you would have to do y.eval({z : 3.}), for example)


Question:
Image erosion and dilation with Scipy I am trying to use scipy to do erosion and dilation of an image. It seems pretty straightforward using scipy -> binary_erosion / dialation. However, the output is not at all what is expected. Here is my basic code:
```
import scipy
from scipy import ndimage
import matplotlib.pyplot as plt
import numpy as np
import Image

#im = Image.open('flower.png')
im = ndimage.imread('flower.png')
im = ndimage.binary_erosion(im).astype(np.float32)
scipy.misc.imsave('erosion.png', im)

im2 = Image.open('flower.png')
im2 = ndimage.binary_dilation(im2)
scipy.misc.imsave('dilation.png', im2)
```
This is the output: The output for dilation is just a completely white image for the original "flower.png" I believe that I must specify a better kernel or mask but am not really sure why I am getting a green output for erosion and completely white output for dilation.

Response:
I was using the binary erosion instead of the grey erosion array. I converted the original image to greyscale by using flatten=true like so:
```
im = scipy.misc.imread('flower.png', flatten=True).astype(np.uint8)
```
then called:
```
im1 = ndimage.grey_erosion(im, size=(15,15))
```
And got a nicely eroded picture, although it is greyscale.


Question:
Why Pearson correlation is different between Tensorflow and Scipy I compute the Pearson correlation in 2 ways : In Tensorflow, I use the following metric :
```
tf.contrib.metrics.streaming_pearson_correlation(y_pred, y_true)
```
When I evaluate my network on test data, I got following results : loss = 0.5289223349094391 pearson = 0.3701728057861328 (Loss is mean_squared_error) Then I predict the test data and compute the same metrics with Scipy :
```
import scipy.stats as measures
per_coef = measures.pearsonr(y_pred, y_true)[0]
mse_coef = np.mean(np.square(np.array(y_pred) - np.array(y_true)))
```
And I get following results : Pearson = 0.5715300096509959 MSE = 0.5289223312665985 Is it a known issue ? Is it normal ? Minimal, complete and verifiable example
```
import tensorflow as tf
import scipy.stats as measures

y_pred = [2, 2, 3, 4, 5, 5, 4, 2]
y_true = [1, 2, 3, 4, 5, 6, 7, 8]

## Scipy
val2 = measures.pearsonr(y_pred, y_true)[0]
print("Scipy's Pearson = {}".format(val2))

## Tensorflow
logits = tf.placeholder(tf.float32, [8])
labels = tf.to_float(tf.Variable(y_true))

acc, acc_op = tf.contrib.metrics.streaming_pearson_correlation(logits,labels)

sess = tf.Session()
sess.run(tf.local_variables_initializer())
sess.run(tf.global_variables_initializer())
sess.run(acc, {logits:y_pred})
sess.run(acc_op, {logits:y_pred})

print("Tensorflow's Pearson:{}".format(sess.run(acc,{logits:y_pred})))
```

Response:
In the minimal verifiable example you gave, y_pred and y_true are lists of integers. In the first line of the scipy.stats.measures.pearsonr source, you will see that the inputs are converted to numpy arrays with x = np.asarray(x). We can see the resulting data types of these arrays via:
```
print(np.asarray(y_pred).dtype)  # Prints 'int64'
```
When dividing two int64 numbers, SciPy uses float64 precision, while TensorFlow will use float32 precision in the example above. The difference can be quite large, even for a single division:
```

```python
'%.15f' % (8.5 / 7)
#Output
#'1.214285714285714'
```

```python
'%.15f' % (np.array(8.5, dtype=np.float32) / np.array(7, dtype=np.float32))
#Output
#'1.214285731315613'
```

```python
'%.15f' % (np.array(8.5, dtype=np.float32) / np.array(7, dtype=np.float32) - 8.5 / 7)
#Output
#'0.000000017029899'
#```
#You can get the same results for SciPy and TensorFlow by using float32 precision for y_pred and y_true:
#```
#import numpy as np
#import tensorflow as tf
#import scipy.stats as measures
```

y_pred = np.array([2, 2, 3, 4, 5, 5, 4, 2], dtype=np.float32)
y_true = np.array([1, 2, 3, 4, 5, 6, 7, 8], dtype=np.float32)

## Scipy
val2 = measures.pearsonr(y_pred, y_true)[0]
print("Scipy's Pearson: \t\t{}".format(val2))

## Tensorflow
logits = tf.placeholder(tf.float32, [8])
labels = tf.to_float(tf.Variable(y_true))

acc, acc_op = tf.contrib.metrics.streaming_pearson_correlation(logits,labels)

sess = tf.Session()
sess.run(tf.local_variables_initializer())
sess.run(tf.global_variables_initializer())
sess.run(acc, {logits:y_pred})
sess.run(acc_op, {logits:y_pred})

print("Tensorflow's Pearson: \t{}".format(sess.run(acc,{logits:y_pred})))
```
prints
```none
Scipy's Pearson:        0.38060760498046875
Tensorflow's Pearson:   0.38060760498046875
```
Differences between SciPy's and TensorFlow's computation In the test scores you report, the difference is quite high. I took a look at the source and found the following differences: 1. Update ops The result of tf.contrib.metrics.streaming_pearson_correlation is not stateless. It returns the correlation coefficient op, together with an update_op for new incoming data. If you call the update op with different data before calling the coefficient op with the actual y_pred, it will give a completely different result:
```
sess.run(tf.global_variables_initializer())

for _ in range(20):
    sess.run(acc_op, {logits: np.random.randn(*y_pred.shape)})

print("Tensorflow's Pearson: \t{}".format(sess.run(acc,{logits:y_pred})))
```
prints
```none
Scipy's Pearson:        0.38060760498046875
Tensorflow's Pearson:   -0.0678008571267128
```
2. Different formulae SciPy: TensorFlow: While mathematically the same, the computation of the correlation coefficient is different in TensorFlow. It uses the sample covariance for (x, x), (x, y) and (y, y) to compute the correlation coefficient, which can introduce different rounding errors.


Question:
Index a SciPy sparse matrix with an array of booleans NumPy arrays can be indexed with an array of booleans to select the rows corresponding to True entries:
```

```python
X = np.array([[1,2,3], [4,5,6], [7,8,9]])
```

```python
rows = np.array([True,False,True])
```

```python
X[rows]
#Output
#array([[1, 2, 3],
#       [7, 8, 9]])
```

```python
X[np.logical_not(rows)]
#Output
#array([[4, 5, 6]])
#```
#But this seems not possible with SciPy sparse matrices; the indices are taken as numeric ones, so False select row 0 and True selects row 1. How can I get the NumPy-like behavior?
```

Response:
You can use np.nonzero (or ndarray.nonzero) on your boolean array to get corresponding numerical indices, then use these to access the sparse matrix. Since "fancy indexing" on sparse matrices is quite limited compared to dense ndarrays, you need to unpack the rows tuple returned by nonzero and specify that you want to retrieve all columns using the : slice:
```

```python
rows.nonzero()
#Output
#(array([0, 2]),)
```

```python
indices = rows.nonzero()[0]
```

```python
indices
#Output
#array([0, 2])
```

```python
sparse[indices, :]
#Output
#<2x100 sparse matrix of type '<type 'numpy.float64'>'
#        with 6 stored elements in LInked List format>
#```
```


Question: Multiple scipy.integrate.ode instances I would like to use scipy.integrate.ode (or scipy.integrate.odeint) instances in multiple threads (one for each CPU core) in order to solve multiple IVPs at a time. However the documentation says: "This integrator is not re-entrant. You cannot have two ode instances using the “vode” integrator at the same time." (Also odeint causes internal errors if instantiated multiple times although the documentation does not say so.) Any idea what can be done?

Response:
One option is to use multiprocessing (i.e. use processes instead of threads). Here's an example that uses the map function of the multiprocessing.Pool class. The function solve takes a set of initial conditions and returns a solution generated by odeint. The "serial" version of the code in the main section calls solve repeatedly, once for each set of initial conditions in ics. The "multiprocessing" version uses the map function of a multiprocessing.Pool instance to run several processes simultaneously, each calling solve. The map function takes care of doling out the arguments to solve. My computer has four cores, and as I increase num_processes, the speedup maxes out at about 3.6.
```
from __future__ import division, print_function

import sys
import time
import multiprocessing as mp
import numpy as np
from scipy.integrate import odeint

def lorenz(q, t, sigma, rho, beta):
    x, y, z = q
    return [sigma*(y - x), x*(rho - z) - y, x*y - beta*z]

def solve(ic):
    t = np.linspace(0, 200, 801)
    sigma = 10.0
    rho = 28.0
    beta = 8/3
    sol = odeint(lorenz, ic, t, args=(sigma, rho, beta), rtol=1e-10, atol=1e-12)
    return sol

if __name__ == "__main__":
    ics = np.random.randn(100, 3)

    print("multiprocessing:", end='')
    tstart = time.time()
    num_processes = 5
    p = mp.Pool(num_processes)
    mp_solutions = p.map(solve, ics)
    tend = time.time()
    tmp = tend - tstart
    print(" %8.3f seconds" % tmp)

    print("serial:         ", end='')
    sys.stdout.flush()
    tstart = time.time()
    serial_solutions = [solve(ic) for ic in ics]
    tend = time.time()
    tserial = tend - tstart
    print(" %8.3f seconds" % tserial)

    print("num_processes = %i, speedup = %.2f" % (num_processes, tserial/tmp))

    check = [(sol1 == sol2).all()
             for sol1, sol2 in zip(serial_solutions, mp_solutions)]
    if not all(check):
        print("There was at least one discrepancy in the solutions.")
```
On my computer, the output is:
```
multiprocessing:    6.904 seconds
serial:            24.756 seconds
num_processes = 5, speedup = 3.59
```


Question:
What scipy statistical test do I use to compare sample means? Assuming sample sizes are not equal, what test do I use to compare sample means under the following circumstances (please correct if any of the following are incorrect): Normal Distribution = True and Homogeneity of Variance = True
```
scipy.stats.ttest_ind(sample_1, sample_2)
```
Normal Distribution = True and Homogeneity of Variance = False
```
scipy.stats.ttest_ind(sample_1, sample_2, equal_var = False)
```
Normal Distribution = False and Homogeneity of Variance = True
```
scipy.stats.mannwhitneyu(sample_1, sample_2)
```
Normal Distribution = False and Homogeneity of Variance = False
```
???
```

Response:
Fast answer: Normal Distribution = True and Homogeneity of Variance = False and sample sizes > 30-50
```
scipy.stats.ttest_ind(sample1, sample2, equal_var=False)
```
Good answer: If you check the Central limit theorem, it says (from Wikipedia): "In probability theory, the central limit theorem (CLT) states that, given certain conditions, the arithmetic mean of a sufficiently large number of iterates of independent random variables, each with a well-defined (finite) expected value and finite variance, will be approximately normally distributed, regardless of the underlying distribution" So, although you do not have a normal distributed population, if your sample is big enough (greater than 30 or 50 samples), then the mean of the samples will be normally distributed. So, you can use:
```
scipy.stats.ttest_ind(sample1, sample2, equal_var=False)
```
This is a two-sided test for the null hypothesis that 2 independent samples have identical average (expected) values. With the option equal_var = False it performs a Welch’s t-test, which does not assume equal population variance.


Question:
How to convert python list of points to numpy image array? I have a python list of points (x/y coordinates):
```
[(200, 245), (344, 248), (125, 34), ...]
```
It represents a contour on a 2d plane. I would like to use some numpy/scipy algorithms for smoothing, interpolation etc. They normally require numpy array as input. For example scipy.ndimage.interpolation.zoom. What is the simplest way to get the right numpy array from my list of points? EDIT: I added the word "image" to my question, hope it is clear now, I am really sorry, if it was somehow misleading. Example of what I meant (points to binary image array). Input:
```
[(0, 0), (2, 0), (2, 1)]
```
Output:
```
[[0, 0, 1],
 [1, 0, 1]]
```
Rounding the accepted answer here is the working sample:
```
import numpy as np

coordinates = [(0, 0), (2, 0), (2, 1)]

x, y = [i[0] for i in coordinates], [i[1] for i in coordinates]
max_x, max_y = max(x), max(y)

image = np.zeros((max_y + 1, max_x + 1))

for i in range(len(coordinates)):
    image[max_y - y[i], x[i]] = 1
```

Response:
Ah, better now, so you do have all the points you want to fill... then its very simple:
```
image = np.zeros((max_x, max_y))
image[coordinates] = 1
```
You could create an array first, but its not necessary.


Question:
SciPy/NumPy import guideline Notice: I checked for duplicate and nothing clearly answers my question. I trust you'll let me know if I missed something! In an effort to clean up my code, I have been looking for a standard convention for importing SciPy and NumPy in my programs. I know there is no strict guideline and I can do it the way I want, but from time to time, I still find contradictory instructions. For example, I've read somewhere that NumPy is meant to only implement the array object, while SciPy is there for every other scientific algorithms. So NumPy should be used for array operation and SciPy for everything else... On the other hand, SciPy imports every Numpy functions in its main namespace, such that scipy.array() is the same thing as numpy.array() (see this question), so NumPy should only be used when SciPy is not being used, as they are duplicates... What is the recommended way to work with SciPy and NumPy? Being a scientist, sqrt(-1) should return a complex number, so I'm inclined to go with SciPy only. Right now, my code starts with:
```
import numpy as np
from scipy import *
from matplotlib import pyplot as plt
```
I use scipy for mathematical operation (such as log10()) and numpy for array creation/operations (such as np.zeros()). Would it be fine to go all the way with SciPy and never import NumPy explicitly? Will a future update remove NumPy's array manipulation from SciPy?

Response:
I recommend doing something like
```
import numpy as np
import scipy as sp
```
instead. It is always dangerous to do from ... import * especially with large modules such as numpy and scipy. The following illustrates why:
```

```python
any(['foo'])
#Output
#True
```

```python
from scipy import *
```

```python
any(['foo'])
```

Traceback (most recent call last):
  File "<pyshell#2>", line 1, in <module>
     any(['foo'])
  File "C:\Python27\lib\site-packages\numpy\core\fromnumeric.py", line 1575, in any
    return _wrapit(a, 'any', axis, out)
  File "C:\Python27\lib\site-packages\numpy\core\fromnumeric.py", line 37, in _wrapit
    result = getattr(asarray(obj),method)(*args, **kwds)
TypeError: cannot perform reduce with flexible type
```
What happens here? The standard python builtin function any is replaced by scipy.any which has different behavior. That might break any code that uses the standard any.


Question:
Efficient way to calculate distance matrix given latitude and longitude data in Python I have data for latitude and longitude, and I need to calculate distance matrix between two arrays containing locations. I used this This to get distance between two locations given latitude and longitude. Here is an example of my code:
```
import numpy as np
import math

def get_distances(locs_1, locs_2):
    n_rows_1 = locs_1.shape[0]
    n_rows_2 = locs_2.shape[0]
    dists = np.empty((n_rows_1, n_rows_2))
    # The loops here are inefficient
    for i in xrange(n_rows_1):
        for j in xrange(n_rows_2):
            dists[i, j] = get_distance_from_lat_long(locs_1[i], locs_2[j])
    return dists

def get_distance_from_lat_long(loc_1, loc_2):

    earth_radius = 3958.75

    lat_dif = math.radians(loc_1[0] - loc_2[0])
    long_dif = math.radians(loc_1[1] - loc_2[1])
    sin_d_lat = math.sin(lat_dif / 2)
    sin_d_long = math.sin(long_dif / 2)
    step_1 = (sin_d_lat ** 2) + (sin_d_long ** 2) * math.cos(math.radians(loc_1[0])) * math.cos(math.radians(loc_2[0]))
    step_2 = 2 * math.atan2(math.sqrt(step_1), math.sqrt(1-step_1))
    dist = step_2 * earth_radius

    return dist
```
My expected output is this:
```

```python
locations_1 = np.array([[34, -81], [32, -87], [35, -83]])
```

```python
locations_2 = np.array([[33, -84], [39, -81], [40, -88], [30, -80]])
```

```python
get_distances(locations_1, locations_2)
#Output
#array([[ 186.13522573,  345.46610882,  566.23466349,  282.51056676],
#       [ 187.96657622,  589.43369894,  555.55312473,  436.88855214],
#       [ 149.5853537 ,  297.56950329,  440.81203371,  387.12153747]])
#```
#Performance is important for me, and one thing I could do is use Cython to speed up the loops, but it would be nice if I don't have to go there. Is there a module that can do something like this? Or any other solution?
```

Response:
There's a lot of suboptimal things in the Haversine equations you are using. You can trim some of that and minimize the number of sines, cosines and square roots you need to calculate. The following is the best I have been able to come up with, and on my system runs about 5x faster than Ophion's code (which does mostly the same as far as vectorization goes) on two random arrays of 1000 and 2000 elements:
```
def spherical_dist(pos1, pos2, r=3958.75):
    pos1 = pos1 * np.pi / 180
    pos2 = pos2 * np.pi / 180
    cos_lat1 = np.cos(pos1[..., 0])
    cos_lat2 = np.cos(pos2[..., 0])
    cos_lat_d = np.cos(pos1[..., 0] - pos2[..., 0])
    cos_lon_d = np.cos(pos1[..., 1] - pos2[..., 1])
    return r * np.arccos(cos_lat_d - cos_lat1 * cos_lat2 * (1 - cos_lon_d))
```
If you feed it your two arrays "as is" it will complain, but that's not a bug, it's a feature. Basically, this function computes the distance on a sphere over the last dimension, and broadcasts on the rest. So you can get what you are after as:
```

```python
spherical_dist(locations_1[:, None], locations_2)
#Output
#array([[ 186.13522573,  345.46610882,  566.23466349,  282.51056676],
#       [ 187.96657622,  589.43369894,  555.55312473,  436.88855214],
#       [ 149.5853537 ,  297.56950329,  440.81203371,  387.12153747]])
#```
#But it could also be used to calculate the distances between two lists of points, i.e.:
#```
```

```python
spherical_dist(locations_1, locations_2[:-1])
#Output
#array([ 186.13522573,  589.43369894,  440.81203371])
#```
#Or between two single points:
#```
```

```python
spherical_dist(locations_1[0], locations_2[0])
#Output
#186.1352257300577
#```
#This is inspired on how gufuncs work, and once you get used to it, I have found it to be a wonderful "swiss army knife" coding style, that lets you reuse a single function in lots of different settings.
```


Question:
Python/Numpy: Convert list of bools to unsigned int What is the fastest (or most "Pythonic") way to convert
```
x = [False, False, True, True]
```
into 12? (If there is such a way.) What if x were instead a numpy.array of bools? Is there a special command for that? I have a large m-by-n array of booleans, where each n-element row represents a single low-dimensional hash of a high-dimensional feature vector. (In the example above, n = 4.) I would like to know the answer in order to compress my data as much as possible. Thank you. Edit: Thank you for the responses! Using the following test code,
```
t = 0
for iter in range(500):
    B = scipy.signbit(scipy.randn(1000,20))
    for b in B:
        t0 = time.clock()
        # test code here
        t1 = time.clock()
        t += (t1-t0)
print t
```
...here were the runtimes on my Thinkpad laptop: My answer: 4.26 sec Sven Marnach 1: 7.88 Emil H: 8.51 Sven Marnach 2: 8.72 delnan: 10.14 liori: 53.49 Of course, I welcome any independent tests that may confirm or refute my data! Edit: In my answer below, changing int(j) to simply j still works, but runs six times as slow! Then perhaps the other answers would become faster if the bool was casted using int. But I'm too lazy to test everything again. Edit: liori posted results of independent tests here.

Response:
Taking various ideas from various other answers, here's another way to do it:
```
sum(1<<i for i, b in enumerate(x) if b)
```
It is quite fast in my tests - right up with the numpy method for large number of bits even though it overflows like crazy. I used liori's testing module for testing. Steve's method, with the change I suggested, is just barely faster. However, if a lot of these sorts of conversions need to be done at a time (and with not too many bits), I'm betting that numpy will be faster.


Question:
4D interpolation for irregular (x,y,z) grids by python I have some data that comes in the form (x, y, z, V) where x,y,z are distances, and V is the moisture. I read a lot on StackOverflow about interpolation by python like this and this valuable posts, but all of them were about regular grids of x, y, z. i.e. every value of x contributes equally with every point of y, and every point of z. On the other hand, my points came from 3D finite element grid (as below), where the grid is not regular. The two mentioned posts 1 and 2, defined each of x, y, z as a separate numpy array then they used something like cartcoord = zip(x, y) then scipy.interpolate.LinearNDInterpolator(cartcoord, z) (in a 3D example). I can not do the same as my 3D grid is not regular, thus not each point has a contribution to other points, so if when I repeated these approaches I found many null values, and I got many errors. Here are 10 sample points in the form of [x, y, z, V]
```
data = [[27.827, 18.530, -30.417, 0.205] , [24.002, 17.759, -24.782, 0.197] ,
[22.145, 13.687, -33.282, 0.204] , [17.627, 18.224, -25.197, 0.197] ,
[29.018, 18.841, -38.761, 0.212] , [24.834, 20.538, -33.012, 0.208] ,
[26.232, 22.327, -27.735, 0.204] , [23.017, 23.037, -29.230, 0.205] ,
[28.761, 21.565, -31.586, 0.211] , [26.263, 23.686, -32.766, 0.215]]
```
I want to get the interpolated value V of the point (25, 20, -30) How can I get it?

Response:
I found the answer, and posting it for the benefit of StackOverflow readers. The method is as follows: 1- Imports:
```
import numpy as np
from scipy.interpolate import griddata
from scipy.interpolate import LinearNDInterpolator
```
2- prepare the data as follows:
```
# put the available x,y,z data as a numpy array
points = np.array([
        [ 27.827,  18.53 , -30.417], [ 24.002,  17.759, -24.782],
        [ 22.145,  13.687, -33.282], [ 17.627,  18.224, -25.197],
        [ 29.018,  18.841, -38.761], [ 24.834,  20.538, -33.012],
        [ 26.232,  22.327, -27.735], [ 23.017,  23.037, -29.23 ],
        [ 28.761,  21.565, -31.586], [ 26.263,  23.686, -32.766]])
# and put the moisture corresponding data values in a separate array:
values = np.array([0.205,  0.197,  0.204,  0.197,  0.212,
                   0.208,  0.204,  0.205, 0.211,  0.215])
# Finally, put the desired point/points you want to interpolate over
request = np.array([[25, 20, -30], [27, 20, -32]])
```
3- Write the final line of code to get the interpolated values Method 1, using griddata
```
print griddata(points, values, request)
# OUTPUT: array([ 0.20448536, 0.20782028])
```
Method 2, using LinearNDInterpolator
```
# First, define an interpolator function
linInter= LinearNDInterpolator(points, values)

# Then, apply the function to one or more points
print linInter(np.array([[25, 20, -30]]))
print linInter(request)
# OUTPUT: [0.20448536  0.20782028]
# I think you may use it with python map or pandas.apply as well
```
Hope this benefit every one. Bet regards


Question:
Rotated image coordinates after scipy.ndimage.interpolation.rotate? I have a numpy array for an image that I read in from a FITS file. I rotated it by N degrees using scipy.ndimage.interpolation.rotate. Then I want to figure out where some point (x,y) in the original non-rotated frame ends up in the rotated image -- i.e., what are the rotated frame coordinates (x',y')? This should be a very simple rotation matrix problem but if I do the usual mathematical or programming based rotation equations, the new (x',y') do not end up where they originally were. I suspect this has something to do with needing a translation matrix as well because the scipy rotate function is based on the origin (0,0) rather than the actual center of the image array. Can someone please tell me how to get the rotated frame (x',y')? As an example, you could use
```
from scipy import misc
from scipy.ndimage import rotate
data_orig = misc.face()
data_rot = rotate(data_orig,66) # data array
x0,y0 = 580,300 # left eye; (xrot,yrot) should point there
```
P.S. The following two related questions' answers do not help me: Find new coordinates of a point after rotation New coordinates after image rotation using scipy.ndimage.rotate

Response:
As usual with rotations, one needs to translate to the origin, then rotate, then translate back. Here, we can take the center of the image as origin.
```
import numpy as np
import matplotlib.pyplot as plt
from scipy import misc
from scipy.ndimage import rotate

data_orig = misc.face()
x0,y0 = 580,300 # left eye; (xrot,yrot) should point there

def rot(image, xy, angle):
    im_rot = rotate(image,angle)
    org_center = (np.array(image.shape[:2][::-1])-1)/2.
    rot_center = (np.array(im_rot.shape[:2][::-1])-1)/2.
    org = xy-org_center
    a = np.deg2rad(angle)
    new = np.array([org[0]*np.cos(a) + org[1]*np.sin(a),
            -org[0]*np.sin(a) + org[1]*np.cos(a) ])
    return im_rot, new+rot_center

fig,axes = plt.subplots(2,2)

axes[0,0].imshow(data_orig)
axes[0,0].scatter(x0,y0,c="r" )
axes[0,0].set_title("original")

for i, angle in enumerate([66,-32,90]):
    data_rot, (x1,y1) = rot(data_orig, np.array([x0,y0]), angle)
    axes.flatten()[i+1].imshow(data_rot)
    axes.flatten()[i+1].scatter(x1,y1,c="r" )
    axes.flatten()[i+1].set_title("Rotation: {}deg".format(angle))

plt.show()
```


Question:
2D integrals in SciPy I am trying to integrate a multivariable function in SciPy over a 2D area. What would be the equivalent of the following Mathematica code?
```

```python
= F[x_, y_] := Cos[x] + Cos[y]
```

```python
= Integrate[F[x, y], {x, -\[Pi], \[Pi]}, {y, -\[Pi], \[Pi]}]

Out[2]= 0
```
Looking at the SciPy documentation I could only find support for one-dimensional quadrature. Is there a way to do multidimensional integrals in SciPy?
```

Response:
I think it would work something like this:
```
def func(x,y):
    return cos(x) + cos(y)

def func2(y, a, b):
    return integrate.quad(func, a, b, args=(y,))[0]

print integrate.quad(func2, -pi/2, pi/2, args=(-pi/2, pi/2))[0]
```
Wolfram|Alpha agrees edit: I just discovered dblquad which seems to do exactly what you want:
```
print integrate.dblquad(func, -pi/2, pi/2, lambda x:-pi/2, lambda x:pi/2)[0]
```


Question:
Python: How to get the convolution of two continuous distributions? Let X, Y be 2 random variables, with probability density functions pdf1 and pdf2. Z = X + Y Then the probability density function of Z is given by the convolution of pdf1 and pdf2. Since we can't deal with continuous distributions, we descritize the continuous distributions and deal with them. To find the convolution of uniform distribution and normal distribution, I came up with following code.
```
import matplotlib.pyplot as plt
import numpy as np
import scipy.stats as stats
from scipy import signal

uniform_dist = stats.uniform(loc=2, scale=3)
std = 0.25
normal_dist = stats.norm(loc=0, scale=std)

delta = 1e-4
big_grid = np.arange(-10,10,delta)

pdf1 = uniform_dist.pdf(big_grid)
print("Integral over uniform pdf: "+str(np.trapz(pdf1, big_grid)))

pdf2 = normal_dist.pdf(big_grid)
print("Integral over normal pdf: "+str(np.trapz(pdf2, big_grid)))

conv_pdf = signal.fftconvolve(pdf1,pdf2,'same')
print("Integral over convoluted pdf: "+str(np.trapz(conv_pdf, big_grid)))

plt.plot(big_grid,pdf1, label='Tophat')
plt.plot(big_grid,pdf2, label='Gaussian error')
plt.plot(big_grid,conv_pdf, label='Sum')
plt.legend(loc='best'), plt.suptitle('PDFs')
plt.show()
```
This is the output I get. Integral over uniform pdf: 0.9999999999976696 Integral over normal pdf: 1.0 Integral over convoluted pdf: 10000.0 If the convolution was correct, I should get a value close to 1 for "Integral over convoluted pdf". So what is going wrong here? Is there a better approach to solve this problem? Thanks

Response:
You should descritize your pdf into probability mass function before the convolution.
```
import matplotlib.pyplot as plt
import numpy as np
import scipy.stats as stats
from scipy import signal

uniform_dist = stats.uniform(loc=2, scale=3)
std = 0.25
normal_dist = stats.norm(loc=0, scale=std)

delta = 1e-4
big_grid = np.arange(-10,10,delta)

pmf1 = uniform_dist.pdf(big_grid)*delta
print("Sum of uniform pmf: "+str(sum(pmf1)))

pmf2 = normal_dist.pdf(big_grid)*delta
print("Sum of normal pmf: "+str(sum(pmf2)))

conv_pmf = signal.fftconvolve(pmf1,pmf2,'same')
print("Sum of convoluted pmf: "+str(sum(conv_pmf)))

pdf1 = pmf1/delta
pdf2 = pmf2/delta
conv_pdf = conv_pmf/delta
print("Integration of convoluted pdf: " + str(np.trapz(conv_pdf, big_grid)))

plt.plot(big_grid,pdf1, label='Uniform')
plt.plot(big_grid,pdf2, label='Gaussian')
plt.plot(big_grid,conv_pdf, label='Sum')
plt.legend(loc='best'), plt.suptitle('PDFs')
plt.show()
```


Question:
python's scipy.stats.ranksums vs. R's wilcox.test Both python's scipy.stats.ranksums and R's wilcox.test are supposed to calculate two-sided p-values for a Wilcoxon rank sum test. But when I run both functions on the same data, I get p-values that differ by orders of magnitude: R:
```
> x=c(57.07168,46.95301,31.86423,38.27486,77.89309,76.78879,33.29809,58.61569,18.26473,62.92256,50.46951,19.14473,22.58552,24.14309)
> y=c(8.319966,2.569211,1.306941,8.450002,1.624244,1.887139,1.376355,2.521150,5.940253,1.458392,3.257468,1.574528,2.338976)
> print(wilcox.test(x, y))

        Wilcoxon rank sum test

data:  x and y
W = 182, p-value = 9.971e-08
alternative hypothesis: true location shift is not equal to 0
```
Python:
```

```python
x=[57.07168,46.95301,31.86423,38.27486,77.89309,76.78879,33.29809,58.61569,18.26473,62.92256,50.46951,19.14473,22.58552,24.14309]
```

```python
y=[8.319966,2.569211,1.306941,8.450002,1.624244,1.887139,1.376355,2.521150,5.940253,1.458392,3.257468,1.574528,2.338976]
```

```python
scipy.stats.ranksums(x, y)
#Output
#(4.415880433163923, 1.0059968254463979e-05)
#```
#So R gives me 1e-7 while Python gives me 1e-5. Where does this difference come from and which one is the 'correct' p-value?
```

Response:
It depends on the choice of options (exact vs a normal approximation, with or without continuity correction): R's default: By default (if ‘exact’ is not specified), an exact p-value is computed if the samples contain less than 50 finite values and there are no ties. Otherwise, a normal approximation is used. Default (as shown above):
```
wilcox.test(x, y)

    Wilcoxon rank sum test

data:  x and y
W = 182, p-value = 9.971e-08
alternative hypothesis: true location shift is not equal to 0
```
Normal approximation with continuity correction:
```
> wilcox.test(x, y, exact=FALSE, correct=TRUE)

    Wilcoxon rank sum test with continuity correction

data:  x and y
W = 182, p-value = 1.125e-05
alternative hypothesis: true location shift is not equal to 0
```
Normal approximation without continuity correction:
```
> (w0 <- wilcox.test(x, y, exact=FALSE, correct=FALSE))

    Wilcoxon rank sum test

data:  x and y
W = 182, p-value = 1.006e-05
alternative hypothesis: true location shift is not equal to 0
```
For a little more precision:
```
w0$p.value
[1] 1.005997e-05
```
It looks like the other value Python is giving you (4.415880433163923) is the Z-score:
```
2*pnorm(4.415880433163923,lower.tail=FALSE)
[1] 1.005997e-05
```
I can appreciate wanting to know what's going on, but I would also point out that there is rarely any practical difference between p=1e-7 and p=1e-5 ...


Question: Why does the Gamma distribution in SciPy have three parameters? Usually, the Gamma distribution has two parameters: shape and scale (or alternatively shape and rate). However, it seems that in SciPy the Gamma distribution has three parameters: two shape parameters and a location parameter. Does anyone know the mapping between the SciPy parameters of Gamma, and, e.g., the parameters in the definition given on wikipedia: http://en.wikipedia.org/wiki/Gamma_distribution Thanks!

Response: All the continuous distributions in scipy.stats have location and scale parameters, even those for which the location is not generally used. For the gamma distribution, just leave the location at its default value 0. If you are using the fit method, use the argument floc=0 to ensure that it does not treat the location as a free parameter. The shape and scale parameters in the scipy gamma distribution correspond to k and θ, respectively, in the wikipedia page.


Question:
Wrong spectrogram when using scipy.signal.spectrogram When I use the plt.specgram from matplotlib by using the following code, the spectrogram generated is correct
```
import matplotlib.pyplot as plt
from scipy import signal
from scipy.io import wavfile
import numpy as np

sample_rate, samples = wavfile.read('.\\Wav\\test.wav')

Pxx, freqs, bins, im = plt.specgram(samples[:,1], NFFT=1024, Fs=44100, noverlap=900)
```
However, if I generate the spectrogram by using the example code given by in the scipy page with the following code, I get something like this:
```
import matplotlib.pyplot as plt
from scipy import signal
from scipy.io import wavfile
import numpy as np

sample_rate, samples = wavfile.read('.\\Wav\\test.wav')

frequencies, times, spectrogram = signal.spectrogram(samples[:,1],sample_rate,nfft=1024,noverlap=900, nperseg=1024)

plt.pcolormesh(times, frequencies, spectrogram)
plt.ylabel('Frequency [Hz]')
plt.xlabel('Time [sec]')
```
To debug what's going on, I tried using the Pxx, freqs, bins, generated by the first method, and then use the second method to plot out the data:
```
plt.pcolormesh(bins, freqs, Pxx)
plt.ylabel('Frequency [Hz]')
plt.xlabel('Time [sec]')
```
The graph generated is almost the same as the graph generated by the second method. So, it seems there is no problem with the scipy.signal.spectrogram after all. The problem is the way that we plot the graph. I wonder if plt.pcolormesh is the correct way to plot the spectrogram despite the fact that this method is suggested in the scipy document A similar question has been asked here, but there's no solution to the question yet.

Response:
The default scaling mode for specgram is 'dB' (from the specgram docs) scale : [ ‘default’ | ‘linear’ | ‘dB’ ] The scaling of the values in the spec. ‘linear’ is no scaling. ‘dB’ returns the values in dB scale. When mode is ‘psd’, this is dB power (10 * log10). Otherwise this is dB amplitude (20 * log10). ‘default’ is ‘dB’ if mode is ‘psd’ or ‘magnitude’ and ‘linear’ otherwise. This must be ‘linear’ if mode is ‘angle’ or ‘phase’. mode : [ ‘default’ | ‘psd’ | ‘magnitude’ | ‘angle’ | ‘phase’ ] What sort of spectrum to use. Default is ‘psd’, which takes the power spectral density. ‘complex’ returns the complex-valued frequency spectrum. ‘magnitude’ returns the magnitude spectrum. ‘angle’ returns the phase spectrum without unwrapping. ‘phase’ returns the phase spectrum with unwrapping. To achieve similar results with pcolormesh you will need to scale the data equivalently.
```
plt.pcolormesh(times, frequencies, 10*np.log10(spectrogram))
```
I don't think the pcolormesh example is correct in its scaling. You can clearly see the carrier in the example but the added noise signal is not visible.


Question:
Coefficient of Variation and NumPy I'd like to create a function with two arguments (a, axis=0) that computes the coefficient of variation of each column or row (2-dimensional array) and returns the index of the column or row with the maximum coefficient of variation. I understand that .argmaxreturns the indices of the maximum values along an axis, but I'm unsure of how to proceed after that. I'd like for the code to pass the following tests:
```
print(np.asscalar(arg_cvmax(b)) is 2)
print(np.asscalar(arg_cvmax(b,1)) is 0)
print(np.asscalar(arg_cvmax(b,axis=1)) is 0)
```

Response:
Use scipy to get the coefficent of variation and the use np.argmax to get the max row. You can use the axis argument to the variation function to compute max variation along row or column.
```
from scipy.stats import variation

A = np.random.randn(10, 10)

# max variation along rows of A;
# rows: axis=0, cols: axis=1

var = variation(A, axis=0)
idmax = np.argmax(var)
print(idmax)
```
Since the coefficent of variation is ratio of the standard deviation to the mean, you can avoid importing variation.
```
A = np.random.randn(10, 10)
cv =  lambda x: np.std(x) / np.mean(x)
var = np.apply_along_axis(cv, axis=0, arr=A)
idmax = np.argmax(var)
```


Question:
Improving Performance of Multiplication of Scipy Sparse Matrices Given a Scipy CSC Sparse matrix "sm" with dimensions (170k x 170k) with 440 million non-null points and a sparse CSC vector "v" (170k x 1) with a few non-null points, is there anything that can be done to improve the performance of the operation:
```
resul = sm.dot(v)
```
? Currently it's taking roughly 1 second. Initializing the matrices as CSR increased the time up to 3 seconds, so CSC performed better. SM is a matrix of similarities between products and V is the vector that represents which products the user bought or clicked on. So for every user sm is the same. I'm using Ubuntu 13.04, Intel i3 @3.4GHz, 4 Cores. Researching on SO I read about Ablas package. I typed into the terminal:
```
~$ ldd /usr/lib/python2.7/dist-packages/numpy/core/_dotblas.so
```
Which resulted in:
```
linux-vdso.so.1 =>  (0x00007fff56a88000)
    libblas.so.3 => /usr/lib/libblas.so.3 (0x00007f888137f000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f8880fb7000)
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f8880cb1000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f888183c000)
```
And for what I understood this means that I'm already using a high performance package from Ablas. I'm still not sure though if this package already implements parallel computing but it looks like it doesn't. Could multi-core processing help to boost performance? If so, is there any library that could be helpful in python? I was also considering the idea of implementing this in Cython but I don't know if this would lead to good results. Thanks in advance.

Response:
The sparse matrix multiplication routines are directly coded in C++, and as far as a quick look at the source reveals, there doesn't seem to be any hook to any optimized library. Furthermore, it doesn't seem to be taking advantage of the fact that the second matrix is a vector to minimize calculations. So you can probably speed things up quite a bit by accessing the guts of the sparse matrix, and customizing the multiplication algorithm. The following code does so in pure Python/Numpy, and if the vector really has "a few non-null points" it matches the speed of scipy's C++ code: if you implemented it in Cython, the speed increase should be noticeable:
```
def sparse_col_vec_dot(csc_mat, csc_vec):
    # row numbers of vector non-zero entries
    v_rows = csc_vec.indices
    v_data = csc_vec.data
    # matrix description arrays
    m_dat = csc_mat.data
    m_ind = csc_mat.indices
    m_ptr = csc_mat.indptr
    # output arrays
    sizes = m_ptr.take(v_rows+1) - m_ptr.take(v_rows)
    sizes = np.concatenate(([0], np.cumsum(sizes)))
    data = np.empty((sizes[-1],), dtype=csc_mat.dtype)
    indices = np.empty((sizes[-1],), dtype=np.intp)
    indptr = np.zeros((2,), dtype=np.intp)

    for j in range(len(sizes)-1):
        slice_ = slice(*m_ptr[[v_rows[j] ,v_rows[j]+1]])
        np.multiply(m_dat[slice_], v_data[j], out=data[sizes[j]:sizes[j+1]])
        indices[sizes[j]:sizes[j+1]] = m_ind[slice_]
    indptr[-1] = len(data)
    ret = sps.csc_matrix((data, indices, indptr),
                         shape=csc_vec.shape)
    ret.sum_duplicates()

    return ret
```
A quick explanation of what is going on: a CSC matrix is defined in three linear arrays: data contains the non-zero entries, stored in column major order. indices contains the rows of the non-zero entries. indptr has one entry more than the number of columns of the matrix, and items in column j are found in data[indptr[j]:indptr[j+1]] and are in rows indices[indptr[j]:indptr[j+1]]. So to multiply by a sparse column vector, you can iterate over data and indices of the column vector, and for each (d, r) pair, extract the corresponding column of the matrix and multiply it by d, i.e. data[indptr[r]:indptr[r+1]] * d and indices[indptr[r]:indptr[r+1]].


Question:
Scipy interpolate.splprep error "Invalid Inputs" I am trying to interpolate a curve to a set of (x,y) points using SciPy's interpolate.splprep method, using the procedure followed in this StackOverflow answer. My code (with the data) is given below. Please excuse me for using this large dataset, as the code works perfectly fine on a different dataset. Kindly scroll to the bottom to see the implemetation.
```
#!/usr/bin/env python3

import numpy as np
import matplotlib.pyplot as plt
from scipy import interpolate
# -----------------------------------------------------------------------------
# Data

xp=np.array([ -1.19824526e-01,  -1.19795807e-01,  -1.22298912e-01,
        -1.24784611e-01,  -1.27233423e-01,  -1.27048456e-01,
        -1.29424259e-01,  -1.31781573e-01,  -1.34102825e-01,
        -1.36386619e-01,  -1.41324999e-01,  -1.43569618e-01,
        -1.48471481e-01,  -1.53300646e-01,  -1.55387133e-01,
        -1.57436481e-01,  -1.53938796e-01,  -1.58562951e-01,
        -1.53139517e-01,  -1.50456275e-01,  -1.49637920e-01,
        -1.48774455e-01,  -1.47843528e-01,  -1.44278335e-01,
        -1.43299274e-01,  -1.39716798e-01,  -1.36111285e-01,
        -1.32534352e-01,  -1.28982866e-01,  -1.25433151e-01,
        -1.21912263e-01,  -1.16106245e-01,  -1.12701128e-01,
        -1.09303316e-01,  -1.05947571e-01,  -1.00467194e-01,
        -9.72083398e-02,  -9.39822094e-02,  -9.08033710e-02,
        -8.96420533e-02,  -8.65053261e-02,  -8.34162875e-02,
        -8.03788778e-02,  -7.73929193e-02,  -7.62032638e-02,
        -7.32655732e-02,  -7.03760465e-02,  -6.91826390e-02,
        -6.63378816e-02,  -6.35537275e-02,  -6.08302060e-02,
        -5.96426925e-02,  -5.69864087e-02,  -5.43931715e-02,
        -5.18641746e-02,  -4.93958173e-02,  -4.82415854e-02,
        -4.58486281e-02,  -4.35196817e-02,  -4.01162919e-02,
        -3.79466513e-02,  -3.48161871e-02,  -3.18596693e-02,
        -2.90650417e-02,  -2.64251761e-02,  -2.31429101e-02,
        -1.94312163e-02,  -1.73997964e-02,  -1.55068323e-02,
        -1.43163160e-02,  -1.31800087e-02,  -1.20987991e-02,
        -1.10708190e-02,  -1.05380016e-02,  -9.58116017e-03,
        -9.06399242e-03,  -8.54450012e-03,  -7.67847396e-03,
        -7.17608354e-03,  -6.67181154e-03,  -5.89474349e-03,
        -5.40878144e-03,  -4.92121197e-03,  -4.43202070e-03,
        -3.94148294e-03,  -3.44986011e-03,  -2.82410814e-03,
        -2.35269319e-03,  -1.88058008e-03,  -1.47393691e-03,
        -9.78376399e-04,  -4.82633521e-04,   1.33099164e-05,
         5.09212801e-04,   1.05098855e-03,   1.56929991e-03,
         2.08706303e-03,   2.72055571e-03,   3.26012954e-03,
         3.79870854e-03,   4.33573131e-03,   4.87172652e-03,
         5.40640816e-03,   5.93914581e-03,   6.47004490e-03,
         6.99921852e-03,   7.52610639e-03,   7.70592714e-03,
         8.20559501e-03,   8.70268809e-03,   9.19766855e-03,
         9.68963219e-03,   1.01781695e-02,   1.01960805e-02,
         1.06577199e-02,   1.11156340e-02,   1.15703286e-02,
         1.20215921e-02,   1.24693015e-02,   1.29129042e-02,
         1.33526781e-02,   1.37884367e-02,   1.42204360e-02,
         1.46473802e-02,   1.50699789e-02,   1.54884533e-02,
         1.59020551e-02,   1.63103362e-02,   8.12110387e-02,
         7.80794051e-02,   1.67140103e-02,   8.31537241e-02,
         7.99472912e-02,   7.99472912e-02,   7.67983984e-02,
         1.71128723e-02,   8.50656342e-02,   8.17851028e-02,
         7.85638577e-02,   7.53861405e-02,   1.75061328e-02,
         8.19411806e-02,   7.38391281e-02,   1.78939640e-02,
         8.70866930e-02,   8.36940292e-02,   8.03586974e-02,
         7.70534244e-02,   7.70534244e-02,   7.38013540e-02,
         7.38013540e-02,   7.06147796e-02,   1.82766038e-02,
         8.54279559e-02,   8.20231372e-02,   7.53294330e-02,
         7.20765174e-02,   1.86539411e-02,   8.36524496e-02,
         7.85095832e-02,   7.51592888e-02,   7.18792721e-02,
         1.90250409e-02,   7.82997201e-02,   7.49183992e-02,
         7.49183992e-02,   7.16144248e-02,   7.16144248e-02,
         6.83771846e-02,   1.93904576e-02,   7.46192919e-02,
         7.12865685e-02,   7.12865685e-02,   6.80175748e-02,
         1.97501330e-02,   7.42568965e-02,   7.08996495e-02,
         7.08996495e-02,   6.75887344e-02,   2.01042729e-02,
         7.38173451e-02,   6.70923613e-02,   2.13903228e-02,
         7.50479910e-02,   6.82108239e-02,   5.69753762e-02,
         5.24303656e-02,   5.24303656e-02,   4.52683211e-02,
         4.52683211e-02,   4.25493203e-02,   2.17470907e-02,
         7.45062992e-02,   6.76173090e-02,   6.76173090e-02,
         6.42925100e-02,   6.42925100e-02,   5.94649095e-02,
         5.94649095e-02,   3.92303424e-02,   2.20977481e-02,
         7.21341379e-02,   3.72338037e-02,   2.24415025e-02,
         7.14448972e-02,   3.40025442e-02,   2.27777176e-02,
         7.07064856e-02,   3.57533680e-02,   2.41421550e-02,
         6.81719132e-02,   3.62534788e-02,   2.44798556e-02,
         6.56110398e-02,   3.80586628e-02,   3.29287629e-02,
         2.93070471e-02,   2.48093588e-02,   6.13326924e-02,
         3.85518913e-02,   3.46206958e-02,   2.85091877e-02,
         2.51312268e-02,   5.38330011e-02,   3.76841669e-02,
         3.50540735e-02,   2.77018960e-02,   2.65615352e-02,
         5.28838088e-02,   3.81396763e-02,   3.54777506e-02,
         2.80364970e-02,   2.68822682e-02,   5.03377702e-02,
         3.85814254e-02,   3.58887890e-02,   4.93316503e-02,
         4.04098395e-02,   3.62892096e-02,   4.67615526e-02,
         4.22828625e-02,   3.80435955e-02,   3.84376145e-02,
         4.02332775e-02,   4.06156847e-02,   4.24553741e-02,
         4.43352031e-02,   4.47040511e-02,   4.66233682e-02,
         4.69790035e-02,   4.89341212e-02,   5.09256192e-02,
         5.12584867e-02,   5.32790231e-02,   5.35890744e-02,
         5.38831411e-02,   5.41625645e-02,   5.44267004e-02,
         5.46700348e-02,   5.48984863e-02,   5.51117932e-02,
         5.53082440e-02,   5.54849716e-02,   5.56464539e-02,
         5.57928396e-02,   5.59201893e-02,   5.60294455e-02,
         5.61233441e-02,   5.62020138e-02,   5.62604489e-02,
         5.63017253e-02,   5.63275468e-02,   5.63341408e-02,
         5.63226424e-02,   5.62957310e-02,   5.62533699e-02,
         5.61937444e-02,   5.61140110e-02,   5.60191106e-02,
         5.59087917e-02,   5.57801898e-02,   5.56328560e-02,
         5.54704141e-02,   5.70775198e-02,   5.68728844e-02,
         5.66515897e-02,   5.64149230e-02,   5.61622287e-02,
         5.76630266e-02,   5.73643873e-02,   5.70502787e-02,
         5.67190716e-02,   5.63668473e-02,   5.59997391e-02,
         5.73489998e-02,   5.69355151e-02,   5.65029189e-02,
         5.77751241e-02,   5.72977910e-02,   5.67990710e-02,
         5.79863269e-02,   5.74393835e-02,   5.68773454e-02,
         5.62926261e-02,   5.56922722e-02,   5.50771272e-02,
         5.44454686e-02,   5.37935810e-02,   5.31273003e-02,
         5.24468411e-02,   5.17483760e-02,   5.10330229e-02,
         5.03036776e-02,   4.95607328e-02,   4.87997085e-02,
         4.80238054e-02,   4.72347342e-02,   4.64331616e-02,
         4.56132865e-02,   4.47805574e-02,   4.39358955e-02,
         4.30782240e-02,   4.22044750e-02,   4.01052073e-02,
         3.92354976e-02,   3.83523540e-02,   3.74567873e-02,
         3.65508593e-02,   3.45751478e-02,   3.36740998e-02,
         3.27625023e-02,   3.18417381e-02,   3.09129121e-02,
         2.90665673e-02,   2.81454989e-02,   2.72171846e-02,
         2.62807950e-02,   2.53342284e-02,   2.43816409e-02,
         2.34221736e-02,   2.24541496e-02,   2.08179757e-02,
         1.98678098e-02,   1.89113740e-02,   1.79488243e-02,
         1.69806146e-02,   1.65158032e-02,   1.55075714e-02,
         1.44932106e-02,   1.34746855e-02,   1.24525920e-02,
         1.14268067e-02,   1.03968750e-02,   9.36414487e-03,
         8.58823755e-03,   7.51804527e-03,   6.44485601e-03,
         5.37002690e-03,   4.29398700e-03,   3.31511044e-03,
         2.20302298e-03,   1.09069996e-03,  -2.27320426e-05,
        -1.16892664e-03,  -2.31490869e-03,  -3.46060569e-03,
        -4.74178052e-03,  -5.91852523e-03,  -7.09360822e-03,
        -8.26683115e-03,  -9.43736653e-03,  -1.06042682e-02,
        -1.17686419e-02,  -1.33107457e-02,  -1.45010352e-02,
        -1.56869180e-02,  -1.68693838e-02,  -1.80464175e-02,
        -1.97732638e-02,  -2.09722818e-02,  -2.21650612e-02,
        -2.40185758e-02,  -2.52303300e-02,  -2.71803154e-02,
        -2.84115598e-02,  -3.04489552e-02,  -3.16936647e-02,
        -3.29299358e-02,  -3.50861051e-02,  -3.63332401e-02,
        -3.85745058e-02,  -3.98348648e-02,  -4.21660006e-02,
        -4.34302610e-02,  -4.46836493e-02,  -4.59254575e-02,
        -4.71530952e-02,  -4.96209305e-02,  -4.95594200e-02,
        -5.07435074e-02,  -5.19101301e-02,  -5.16977894e-02,
        -5.14280802e-02,  -5.11057669e-02,  -5.07251169e-02,
        -5.16985297e-02,  -5.12126585e-02,  -5.06852098e-02,
        -5.15589749e-02,  -5.09397027e-02,  -5.17615499e-02,
        -5.10672514e-02,  -5.18313966e-02,  -5.25816754e-02,
        -5.33179227e-02,  -5.40360028e-02,  -5.47358953e-02,
        -5.54213064e-02,  -5.77400978e-02,  -5.84092053e-02,
        -5.90603644e-02,  -6.14284845e-02,  -6.38379284e-02,
        -6.62872262e-02,  -6.69166162e-02,  -6.93865431e-02,
        -7.18947674e-02,  -7.44284962e-02,  -7.69969804e-02,
        -7.96063191e-02,  -8.01834105e-02,  -8.28053535e-02,
        -8.54623715e-02,  -8.59961071e-02,  -8.86660185e-02,
        -8.91520913e-02,  -9.18335218e-02,  -9.45402708e-02,
        -9.49610563e-02,  -9.76401856e-02,  -1.00332460e-01,
        -1.03032191e-01,  -1.03358935e-01,  -1.06040606e-01,
        -1.06322470e-01,  -1.08984284e-01,  -1.09195131e-01,
        -1.11833426e-01,  -1.11994247e-01,  -1.14596404e-01,
        -1.17192554e-01,  -1.17248317e-01])

yp = np.array([ -3.90948536e-05,  -2.12984775e-03,  -4.31095583e-03,
        -6.58019633e-03,  -8.93758156e-03,  -1.11568100e-02,
        -1.36444162e-02,  -1.62222092e-02,  -1.88895170e-02,
        -2.16446498e-02,  -2.49629308e-02,  -2.79508857e-02,
        -3.16029501e-02,  -3.54376380e-02,  -3.87881494e-02,
        -4.22310942e-02,  -4.41873802e-02,  -4.85246067e-02,
        -4.68663315e-02,  -4.60459599e-02,  -4.86676408e-02,
        -5.12750434e-02,  -5.38586293e-02,  -5.54310799e-02,
        -5.79452426e-02,  -5.93547929e-02,  -6.06497762e-02,
        -6.18505946e-02,  -6.29584706e-02,  -6.39609234e-02,
        -6.48713094e-02,  -6.44090476e-02,  -6.51181556e-02,
        -6.57260659e-02,  -6.62541381e-02,  -6.52943568e-02,
        -6.56184758e-02,  -6.58578685e-02,  -6.60229010e-02,
        -6.76012689e-02,  -6.76366183e-02,  -6.76004442e-02,
        -6.74972483e-02,  -6.73282385e-02,  -6.86657097e-02,
        -6.83738036e-02,  -6.80140059e-02,  -6.92366190e-02,
        -6.87491258e-02,  -6.82071471e-02,  -6.76134579e-02,
        -6.86669494e-02,  -6.79695621e-02,  -6.72259327e-02,
        -6.64391135e-02,  -6.56069234e-02,  -6.64563885e-02,
        -6.55361171e-02,  -6.45783892e-02,  -6.18312378e-02,
        -6.07850085e-02,  -5.80009440e-02,  -5.52383021e-02,
        -5.24888121e-02,  -4.97523554e-02,  -4.54714570e-02,
        -3.98863362e-02,  -3.73592876e-02,  -3.48720213e-02,
        -3.37707235e-02,  -3.26655171e-02,  -3.15625118e-02,
        -3.04616664e-02,  -3.06508019e-02,  -2.95344258e-02,
        -2.96968330e-02,  -2.98505905e-02,  -2.87101259e-02,
        -2.88391064e-02,  -2.89597166e-02,  -2.77967360e-02,
        -2.78958771e-02,  -2.79854740e-02,  -2.80670276e-02,
        -2.81405467e-02,  -2.82051366e-02,  -2.69913041e-02,
        -2.70365186e-02,  -2.70739448e-02,  -2.83768113e-02,
        -2.83979671e-02,  -2.84108899e-02,  -2.84155794e-02,
        -2.84104617e-02,  -2.96993141e-02,  -2.96767995e-02,
        -2.96453017e-02,  -3.09305120e-02,  -3.08782748e-02,
        -3.08172540e-02,  -3.07460634e-02,  -3.06652277e-02,
        -3.05756546e-02,  -3.04773301e-02,  -3.03684498e-02,
        -3.02505329e-02,  -3.01240628e-02,  -2.87032761e-02,
        -2.85638294e-02,  -2.84161924e-02,  -2.82602014e-02,
        -2.80957411e-02,  -2.79220043e-02,  -2.65224371e-02,
        -2.63408455e-02,  -2.61506690e-02,  -2.59523304e-02,
        -2.57465736e-02,  -2.55333569e-02,  -2.53114227e-02,
        -2.50819674e-02,  -2.48453976e-02,  -2.46014650e-02,
        -2.43490672e-02,  -2.40896946e-02,  -2.38232320e-02,
        -2.35495727e-02,  -2.32681400e-02,  -1.11708561e-01,
        -1.07398522e-01,  -2.29799277e-02,  -1.10281290e-01,
        -1.06025945e-01,  -1.06025945e-01,  -1.01847844e-01,
        -2.26850806e-02,  -1.08812919e-01,  -1.04614895e-01,
        -1.00492396e-01,  -9.64256156e-02,  -2.23830803e-02,
        -1.01124594e-01,  -9.11212826e-02,  -2.20738630e-02,
        -1.03723227e-01,  -9.96804013e-02,  -9.57062055e-02,
        -9.17682599e-02,  -9.17682599e-02,  -8.78935733e-02,
        -8.78935733e-02,  -8.40962884e-02,  -2.17583603e-02,
        -9.82127298e-02,  -9.42965108e-02,  -8.65980524e-02,
        -8.28570139e-02,  -2.14365508e-02,  -9.28460674e-02,
        -8.71354106e-02,  -8.34157663e-02,  -7.97743543e-02,
        -2.11075333e-02,  -8.39100274e-02,  -8.02849723e-02,
        -8.02849723e-02,  -7.67428202e-02,  -7.67428202e-02,
        -7.32724167e-02,  -2.07721464e-02,  -7.72159766e-02,
        -7.37663681e-02,  -7.37663681e-02,  -7.03828404e-02,
        -2.04308432e-02,  -7.42042591e-02,  -7.08482147e-02,
        -7.08482147e-02,  -6.75385453e-02,  -2.00834820e-02,
        -7.12338454e-02,  -6.47417418e-02,  -2.06352744e-02,
        -6.99333169e-02,  -6.35600774e-02,  -5.30876202e-02,
        -4.88515872e-02,  -4.88515872e-02,  -4.21763073e-02,
        -4.21763073e-02,  -3.96425097e-02,  -2.02588101e-02,
        -6.70368116e-02,  -6.08364913e-02,  -6.08364913e-02,
        -5.78440553e-02,  -5.78440553e-02,  -5.34994049e-02,
        -5.34994049e-02,  -3.52908904e-02,  -1.98763502e-02,
        -6.26583213e-02,  -3.23368135e-02,  -1.94880238e-02,
        -5.99037138e-02,  -2.85040222e-02,  -1.90931928e-02,
        -5.72132575e-02,  -2.89247783e-02,  -1.95297821e-02,
        -5.32198482e-02,  -2.82971986e-02,  -1.91058177e-02,
        -4.94013681e-02,  -2.86515116e-02,  -2.47888430e-02,
        -2.20618305e-02,  -1.86758942e-02,  -4.45232330e-02,
        -2.79827472e-02,  -2.51286391e-02,  -2.06919011e-02,
        -1.82397645e-02,  -3.76607947e-02,  -2.63609122e-02,
        -2.45208701e-02,  -1.93767971e-02,  -1.85788804e-02,
        -3.56379420e-02,  -2.56998805e-02,  -2.39058698e-02,
        -1.88908564e-02,  -1.81130913e-02,  -3.26595065e-02,
        -2.50304222e-02,  -2.32829732e-02,  -3.07966353e-02,
        -2.52257065e-02,  -2.26527986e-02,  -2.80693713e-02,
        -2.53799880e-02,  -2.28350066e-02,  -2.21686432e-02,
        -2.22782703e-02,  -2.15723084e-02,  -2.16081542e-02,
        -2.15998200e-02,  -2.08220272e-02,  -2.07341864e-02,
        -1.99180705e-02,  -1.97463091e-02,  -1.95241512e-02,
        -1.86330762e-02,  -1.83210810e-02,  -1.73881714e-02,
        -1.64501676e-02,  -1.55073488e-02,  -1.45603397e-02,
        -1.36076891e-02,  -1.26514336e-02,  -1.16918550e-02,
        -1.07281971e-02,  -9.76103257e-03,  -8.79150351e-03,
        -7.81935696e-03,  -6.84417527e-03,  -5.86703766e-03,
        -4.88857954e-03,  -3.90851347e-03,  -2.92690669e-03,
        -1.94445885e-03,  -9.62077293e-04,   2.10973681e-05,
         1.00443470e-03,   1.98670872e-03,   2.96920518e-03,
         3.95065293e-03,   4.93054490e-03,   5.90896238e-03,
         6.88594418e-03,   7.86095305e-03,   8.83291761e-03,
         9.80227952e-03,   1.11168744e-02,   1.21109612e-02,
         1.31014370e-02,   1.40884671e-02,   1.50714343e-02,
         1.65579859e-02,   1.75619959e-02,   1.85609524e-02,
         1.95539892e-02,   2.05406220e-02,   2.15208623e-02,
         2.31958067e-02,   2.41936890e-02,   2.51825785e-02,
         2.69676402e-02,   2.79735240e-02,   2.89676199e-02,
         3.08600313e-02,   3.18685118e-02,   3.28673845e-02,
         3.38531747e-02,   3.48305552e-02,   3.57981735e-02,
         3.67545357e-02,   3.76978426e-02,   3.86308181e-02,
         3.95533112e-02,   4.04626970e-02,   4.13583593e-02,
         4.22429533e-02,   4.31163338e-02,   4.39732984e-02,
         4.48174616e-02,   4.56497573e-02,   4.64690781e-02,
         4.72699006e-02,   4.80584575e-02,   4.88339015e-02,
         4.95941309e-02,   5.03364921e-02,   4.95646923e-02,
         5.02584615e-02,   5.09357803e-02,   5.15956682e-02,
         5.22416815e-02,   5.13017754e-02,   5.18954788e-02,
         5.24741267e-02,   5.30389590e-02,   5.35886852e-02,
         5.24828002e-02,   5.29815950e-02,   5.34658214e-02,
         5.39333680e-02,   5.43819816e-02,   5.48154596e-02,
         5.52339801e-02,   5.56342267e-02,   5.42908141e-02,
         5.46458325e-02,   5.49864909e-02,   5.53070690e-02,
         5.56106186e-02,   5.76746395e-02,   5.79561804e-02,
         5.82151857e-02,   5.84584712e-02,   5.86858866e-02,
         5.88950787e-02,   5.90831689e-02,   5.92552324e-02,
         6.12619766e-02,   6.14026109e-02,   6.15224608e-02,
         6.16256880e-02,   6.17123394e-02,   6.36720486e-02,
         6.37186812e-02,   6.37481408e-02,   6.56861133e-02,
         6.56722393e-02,   6.56407664e-02,   6.55917721e-02,
         6.74743714e-02,   6.73786194e-02,   6.72646677e-02,
         6.71325153e-02,   6.69773741e-02,   6.68003322e-02,
         6.66053297e-02,   6.83473413e-02,   6.81027202e-02,
         6.78376164e-02,   6.75542903e-02,   6.72524243e-02,
         6.88634515e-02,   6.85066915e-02,   6.81318613e-02,
         6.96716731e-02,   6.92387957e-02,   7.07292209e-02,
         7.02467261e-02,   7.16584645e-02,   7.11134550e-02,
         7.05499862e-02,   7.18681370e-02,   7.12419450e-02,
         7.24822144e-02,   7.17989089e-02,   7.29694293e-02,
         7.22180480e-02,   7.14476517e-02,   7.06588875e-02,
         6.98486903e-02,   7.08078412e-02,   6.81567317e-02,
         6.72843393e-02,   6.63881936e-02,   6.37899997e-02,
         6.12404950e-02,   5.87433383e-02,   5.62902969e-02,
         5.53950962e-02,   5.29895052e-02,   5.06437557e-02,
         4.97490264e-02,   4.74631181e-02,   4.65678359e-02,
         4.43551128e-02,   4.34554011e-02,   4.25440351e-02,
         4.16213883e-02,   4.06842153e-02,   3.97338457e-02,
         3.87727819e-02,   3.89122376e-02,   3.78978623e-02,
         3.68719281e-02,   3.68766567e-02,   3.68230044e-02,
         3.67095055e-02,   3.55465346e-02,   3.53200609e-02,
         3.50311849e-02,   3.46717730e-02,   3.42461153e-02,
         3.37555022e-02,   3.23610029e-02,   3.17505933e-02,
         3.10701527e-02,   2.95754797e-02,   2.87735213e-02,
         2.72210019e-02,   2.62970023e-02,   2.52956273e-02,
         2.36404433e-02,   2.25053642e-02,   2.12889860e-02,
         1.99902757e-02,   1.81872330e-02,   1.67574555e-02,
         1.49054892e-02,   1.33429656e-02,   1.14391250e-02,
         9.74643800e-03,   7.79267351e-03,   5.96714375e-03,
         4.05355227e-03,   2.00672241e-03])

# -----------------------------------------------------------------------------
# Use scipy to interpolate.
xp = np.r_[xp, xp[0]]
yp = np.r_[yp, yp[0]]
tck, u = interpolate.splprep([xp, yp], s=0, k=1, per=True)
xi, yi = interpolate.splev(np.linspace(0, 1, 1000), tck)

# -----------------------------------------------------------------------------
# Plot result
fig = plt.figure()
ax = plt.subplot(111)
ax.plot(xp, yp, '.', markersize=2)
ax.plot(xi, yi, alpha=0.5)

plt.show()
```
I get the following error on one machine (MacOS),
```
--->      tck, u = interpolate.splprep([xp, yp], s=0, k=1, per=True)
SystemError: <built-in function _parcur> returned NULL without setting an error
```
And this error on another machine (Ubuntu),
```
---->     tck, u = interpolate.splprep([xp, yp], s=0, k=1, per=True)
ValueError: Invalid inputs.
```
interpolate.splprep uses the FORTRAN parcur routine from FITPACK (from the documentation). My questions are - Why does the code work for different datasets? e.g. xp = np.array([0.1, 0.2, 0.3, 0.4]) yp = np.array([-0.1, -0.3, -0.4, 0.2]) and not for this particular one? What does the error mean? How can I get this to work? (Using this method or any other method) i.e. either interpolate a curve or filter the outliers ... Out of curiosity, why is the error machine (and OS) dependent? This is how the data looks when plotted, I think you can guess which curve I'd like to interpolate to (and which outliers I'd like to remove, if possible)

Response:
Fitpack has a fit if it two consecutive inputs are identical. The error happens deep enough that it depends on how the libraries were compiled and linked, hence the assortment of errors. For example, xp[147:149], yp[147:149] (and several others):
```
(array([ 0.07705342,  0.07705342]), array([-0.09176826, -0.09176826]))
```
These are okay:
```
okay = np.where(np.abs(np.diff(xp)) + np.abs(np.diff(yp)) > 0)
xp = np.r_[xp[okay], xp[-1], xp[0]]
yp = np.r_[yp[okay], yp[-1], yp[0]]
#  the rest of your code
```
I add the last point back because the output of diff is always one element shorter, so the last one needs to be included manually. (And then of course, you put the 0th point again for periodicity) Cutting off the weird part This is my attempt to cut off the weird extruding part of the dataset. It uses a Gaussian filter from ndimage. The original points xp, yp are kept this time; the filtered ones are xn, yn.
```
jump = np.sqrt(np.diff(xp)**2 + np.diff(yp)**2)
smooth_jump = ndimage.gaussian_filter1d(jump, 5, mode='wrap')  # window of size 5 is arbitrary
limit = 2*np.median(smooth_jump)    # factor 2 is arbitrary
xn, yn = xp[:-1], yp[:-1]
xn = xn[(jump > 0) & (smooth_jump < limit)]
yn = yn[(jump > 0) & (smooth_jump < limit)]
```
So, we remove not only duplicate points but also the points where the values jump around too much. The rest goes as before, interpolation is built out of xn, yn now. I plot original points for comparison with the new (red) curve):
```
ax.plot(xp, yp, 'o', markersize=2)
ax.plot(xi, yi, 'r', alpha=0.5)
```


Question:
What is probability density function in the context of scipy.stats.norm? This is a very basic question, but I can't seem to find a good answer. What exactly does scipy calculate for
```
scipy.stats.norm(50,10).pdf(45)
```
I understand that the probability of a particular value like 45 in a gaussian with mean 50 and std dev 10 is 0. So what exactly is pdf calculating? Is it the area under the gaussian curve, and if so, what is the range of values on the x axis?

Response:
The probability density function of the normal distribution expressed in Python is
```
from math import pi
from math import exp
from scipy import stats

def normal_pdf(x, mu, sigma):
    return 1.0 / (sigma * (2.0 * pi)**(1/2)) * exp(-1.0 * (x - mu)**2 / (2.0 * (sigma**2)))
```
(compare that to the wikipedia definition). And this is exactly what scipy.stats.norm().pdf() computes: the value of the pdf at point x for a given mu, sigma. Note that this is not a probability (= area under the pdf) but rather the value of the pdf at the point x you pass to pdf(x) (and that value can very well be greater than 1.0!). You can see that, for example, for N(0, 0.1) at x = 0:
```
val = stats.norm(0, 0.1).pdf(0)

print(val)

val = normal_pdf(0, 0, 0.1)

print(val)
```
which gives the output 3.98942280401 3.989422804014327 Not at all a probability = area under the curve! Note that this doesn't contradict the statement that the probability of particular value like x = 0 is 0 because, formally, the area under the pdf for a point (i.e., an interval of length 0) is zero (if f is a continuous function on [a, b] and F is its antiderivative on [a, b], then the definite integral of f over [a, b] = F(a) - F(b). Here, a = b = x hence the value of the integral is F(x) - F(x) = 0).


Question:
Scipy's sparse eigsh() for small eigenvalues I'm trying to write a spectral clustering algorithm using NumPy/SciPy for larger (but still tractable) systems, making use of SciPy's sparse linear algebra library. Unfortunately, I'm running into stability issues with eigsh(). Here's my code:
```
import numpy as np
import scipy.sparse
import scipy.sparse.linalg as SLA
import sklearn.utils.graph as graph

W = self._sparse_rbf_kernel(self.X_, self.datashape)
D = scipy.sparse.csc_matrix(np.diag(np.array(W.sum(axis = 0))[0]))
L = graph.graph_laplacian(W) # D - W
vals, vects = SLA.eigsh(L, k = self.k, M = D, which = 'SM', sigma = 0, maxiter = 1000)
```
The sklearn library refers to the scikit-learn package, specifically this method for calculating a graph laplacian from a sparse SciPy matrix. _sparse_rbf_kernel is a method I wrote to compute pairwise affinities of the data points. It operates by creating a sparse affinity matrix from image data, specifically by only computing pairwise affinities for the 8-neighborhoods around each pixel (instead of pairwise for all pixels with scikit-learn's rbf_kernel method, which for the record doesn't fix this either). Since the laplacian is unnormalized, I'm looking for the smallest eigenvalues and corresponding eigenvectors of the system. I understand that ARPACK is ill-suited for finding small eigenvalues, but I'm trying to use shift-invert to find these values and am still not having much success. With the above arguments (specifically, sigma = 0), I get the following error: RuntimeError: Factor is exactly singular With sigma = 0.001, I get a different error: scipy.sparse.linalg.eigen.arpack.arpack.ArpackNoConvergence: ARPACK error -1: No convergence (1001 iterations, 0/5 eigenvectors converged) I've tried all three different values for mode with the same result. Any suggestions for using the SciPy sparse library for finding small eigenvalues of a large system?

Response: You should use which='LM': in the shift-invert mode, this parameter refers to the transformed eigenvalues. (As explained in the documentation.)


Question: convert list and list of lists to scipy sparse arrays Suppose I have a list or a list of lists (each list with the same size). How do I convert to a sparse vector or sparse matrix, respectively?

Response:
```

```python
scipy.sparse.csr_matrix([[1, 2], [3, 0]])
#Output
#<2x2 sparse matrix of type '<type 'numpy.int64'>'
#        with 3 stored elements in Compressed Sparse Row format>
```

```python
scipy.sparse.csr_matrix([1, 2])
#Output
#<1x2 sparse matrix of type '<type 'numpy.int64'>'
#        with 2 stored elements in Compressed Sparse Row format>
#```
#scipy.sparse.whatever_matrix_type(your_data_structure). It's quite analogous to what you'd do to get a regular array. Note that there is no sparse vector or sparse ndarray class, only sparse matrices.
```


Question:
Getting spline equation from UnivariateSpline object I'm using UnivariateSpline to construct piecewise polynomials for some data that I have. I would then like to use these splines in other programs (either in C or FORTRAN) and so I would like to understand the equation behind the generated spline. Here is my code:
```
import numpy as np
import scipy as sp
from  scipy.interpolate import UnivariateSpline
import matplotlib.pyplot as plt
import bisect

data = np.loadtxt('test_C12H26.dat')
Tmid = 800.0
print "Tmid", Tmid
nmid = bisect.bisect(data[:,0],Tmid)
fig = plt.figure()
plt.plot(data[:,0], data[:,7],ls='',marker='o',markevery=20)
npts = len(data[:,0])
#print "npts", npts
w = np.ones(npts)
w[0] = 100
w[nmid] = 100
w[npts-1] = 100
spline1 = UnivariateSpline(data[:nmid,0],data[:nmid,7],s=1,w=w[:nmid])
coeffs = spline1.get_coeffs()
print coeffs
print spline1.get_knots()
print spline1.get_residual()
print coeffs[0] + coeffs[1] * (data[0,0] - data[0,0]) \
                + coeffs[2] * (data[0,0] - data[0,0])**2 \
                + coeffs[3] * (data[0,0] - data[0,0])**3, \
      data[0,7]
print coeffs[0] + coeffs[1] * (data[nmid,0] - data[0,0]) \
                + coeffs[2] * (data[nmid,0] - data[0,0])**2 \
                + coeffs[3] * (data[nmid,0] - data[0,0])**3, \
      data[nmid,7]

print Tmid,data[-1,0]
spline2 = UnivariateSpline(data[nmid-1:,0],data[nmid-1:,7],s=1,w=w[nmid-1:])
print spline2.get_coeffs()
print spline2.get_knots()
print spline2.get_residual()
plt.plot(data[:,0],spline1(data[:,0]))
plt.plot(data[:,0],spline2(data[:,0]))
plt.savefig('test.png')
```
And here is the resulting plot. I believe I have valid splines for each interval but it looks like my spline equation is not correct... I can't find any reference to what it is supposed to be in the scipy documentation. Anybody knows? Thanks !

Response:
The scipy documentation does not have anything to say about how one can take the coefficients and manually generate the spline curve. However, it is possible to figure out how to do this from the existing literature on B-splines. The following function bspleval shows how to construct the B-spline basis functions (the matrix B in the code), from which one can easily generate the spline curve by multiplying the coefficients with the highest-order basis functions and summing:
```
def bspleval(x, knots, coeffs, order, debug=False):
    '''
    Evaluate a B-spline at a set of points.

    Parameters
    ----------
    x : list or ndarray
        The set of points at which to evaluate the spline.
    knots : list or ndarray
        The set of knots used to define the spline.
    coeffs : list of ndarray
        The set of spline coefficients.
    order : int
        The order of the spline.

    Returns
    -------
    y : ndarray
        The value of the spline at each point in x.
    '''

    k = order
    t = knots
    m = alen(t)
    npts = alen(x)
    B = zeros((m-1,k+1,npts))

    if debug:
        print('k=%i, m=%i, npts=%i' % (k, m, npts))
        print('t=', t)
        print('coeffs=', coeffs)

    ## Create the zero-order B-spline basis functions.
    for i in range(m-1):
        B[i,0,:] = float64(logical_and(x >= t[i], x < t[i+1]))

    if (k == 0):
        B[m-2,0,-1] = 1.0

    ## Next iteratively define the higher-order basis functions, working from lower order to higher.
    for j in range(1,k+1):
        for i in range(m-j-1):
            if (t[i+j] - t[i] == 0.0):
                first_term = 0.0
            else:
                first_term = ((x - t[i]) / (t[i+j] - t[i])) * B[i,j-1,:]

            if (t[i+j+1] - t[i+1] == 0.0):
                second_term = 0.0
            else:
                second_term = ((t[i+j+1] - x) / (t[i+j+1] - t[i+1])) * B[i+1,j-1,:]

            B[i,j,:] = first_term + second_term
        B[m-j-2,j,-1] = 1.0

    if debug:
        plt.figure()
        for i in range(m-1):
            plt.plot(x, B[i,k,:])
        plt.title('B-spline basis functions')

    ## Evaluate the spline by multiplying the coefficients with the highest-order basis functions.
    y = zeros(npts)
    for i in range(m-k-1):
        y += coeffs[i] * B[i,k,:]

    if debug:
        plt.figure()
        plt.plot(x, y)
        plt.title('spline curve')
        plt.show()

    return(y)
```
To give an example of how this can be used with Scipy's existing univariate spline functions, the following is an example script. This takes the input data and uses Scipy's functional and also its object-oriented approach to spline fitting. Taking the coefficients and knot points from either of the two and using these as inputs to our manually-calculated routine bspleval, we reproduce the same curve that they do. Note that the difference between the manually evaluated curve and Scipy's evaluation method is so small that it is almost certainly floating-point noise.
```
x = array([-273.0, -176.4, -79.8, 16.9, 113.5, 210.1, 306.8, 403.4, 500.0])
y = array([2.25927498e-53, 2.56028619e-03, 8.64512988e-01, 6.27456769e+00, 1.73894734e+01,
        3.29052124e+01, 5.14612316e+01, 7.20531200e+01, 9.40718450e+01])

x_nodes = array([-273.0, -263.5, -234.8, -187.1, -120.3, -34.4, 70.6, 194.6, 337.8, 500.0])
y_nodes = array([2.25927498e-53, 3.83520726e-46, 8.46685318e-11, 6.10568083e-04, 1.82380809e-01,
                2.66344008e+00, 1.18164677e+01, 3.01811501e+01, 5.78812583e+01, 9.40718450e+01])

## Now get scipy's spline fit.
k = 3
tck = splrep(x_nodes, y_nodes, k=k, s=0)
knots = tck[0]
coeffs = tck[1]
print('knot points=', knots)
print('coefficients=', coeffs)

## Now try scipy's object-oriented version. The result is exactly the same as "tck": the knots are the
## same and the coeffs are the same, they are just queried in a different way.
uspline = UnivariateSpline(x_nodes, y_nodes, s=0)
uspline_knots = uspline.get_knots()
uspline_coeffs = uspline.get_coeffs()

## Here are scipy's native spline evaluation methods. Again, "ytck" and "y_uspline" are exactly equal.
ytck = splev(x, tck)
y_uspline = uspline(x)
y_knots = uspline(knots)

## Now let's try our manually-calculated evaluation function.
y_eval = bspleval(x, knots, coeffs, k, debug=False)

plt.plot(x, ytck, label='tck')
plt.plot(x, y_uspline, label='uspline')
plt.plot(x, y_eval, label='manual')

## Next plot the knots and nodes.
plt.plot(x_nodes, y_nodes, 'ko', markersize=7, label='input nodes')            ## nodes
plt.plot(knots, y_knots, 'mo', markersize=5, label='tck knots')    ## knots
plt.xlim((-300.0,530.0))
plt.legend(loc='best', prop={'size':14})

plt.figure()
plt.title('difference')
plt.plot(x, ytck-y_uspline, label='tck-uspl')
plt.plot(x, ytck-y_eval, label='tck-manual')
plt.legend(loc='best', prop={'size':14})

plt.show()
```


Question:
Pandas how to split dataframe by column by interval I have a gigantic dataframe with a datetime type column called dt, the data frame is sorted based on dt already. I want to split the dataframe into several dataframes based on dt, each dataframe contains rows within 1 hr range. Split
```
dt                    text
0  20160811 11:05        a
1  20160811 11:35        b
2  20160811 12:03        c
3  20160811 12:36        d
4  20160811 12:52        e
5  20160811 14:32        f
```
into
```
dt                    text
0  20160811 11:05        a
1  20160811 11:35        b
2  20160811 12:03        c

   dt                    text
0  20160811 12:36        d
1  20160811 12:52        e

   dt                    text
0  20160811 14:32        f
```

Response:
You need groupby by difference of first value of column dt converted to hour by astype:
```
S = pd.to_datetime(df.dt)
for i, g in df.groupby([(S - S[0]).astype('timedelta64[h]')]):
        print (g.reset_index(drop=True))

               dt text
0  20160811 11:05    a
1  20160811 11:35    b
2  20160811 12:03    c
               dt text
0  20160811 12:36    d
1  20160811 12:52    e
               dt text
0  20160811 14:32    f
```
List comprehension solution:
```
S = pd.to_datetime(df.dt)

print ((S - S[0]).astype('timedelta64[h]'))
0    0.0
1    0.0
2    0.0
3    1.0
4    1.0
5    3.0
Name: dt, dtype: float64

L = [g.reset_index(drop=True) for i, g in df.groupby([(S - S[0]).astype('timedelta64[h]')])]

print (L[0])
               dt text
0  20160811 11:05    a
1  20160811 11:35    b
2  20160811 12:03    c

print (L[1])
               dt text
0  20160811 12:36    d
1  20160811 12:52    e

print (L[2])
               dt text
0  20160811 14:32    f
```
Old solution, which split by hour: You can use groupby by dt.hour, but first need convert dt to_datetime:
```
for i, g in df.groupby([pd.to_datetime(df.dt).dt.hour]):
    print (g.reset_index(drop=True))

               dt text
0  20160811 11:05    a
1  20160811 11:35    b
               dt text
0  20160811 12:03    c
1  20160811 12:36    d
2  20160811 12:52    e
               dt text
0  20160811 14:32    f
```
List comprehension solution:
```
L = [g.reset_index(drop=True) for i, g in df.groupby([pd.to_datetime(df.dt).dt.hour])]

print (L[0])
               dt text
0  20160811 11:05    a
1  20160811 11:35    b

print (L[1])
               dt text
0  20160811 12:03    c
1  20160811 12:36    d
2  20160811 12:52    e

print (L[2])
               dt text
0  20160811 14:32    f
```
Or use list comprehension with converting column dt to datetime:
```
df.dt = pd.to_datetime(df.dt)
L =[g.reset_index(drop=True) for i, g in df.groupby([df['dt'].dt.hour])]

print (L[1])
                   dt text
0 2016-08-11 12:03:00    c
1 2016-08-11 12:36:00    d
2 2016-08-11 12:52:00    e

print (L[2])
                   dt text
0 2016-08-11 14:32:00    f
```
If need split by dates and hours:
```
#changed dataframe for testing
print (df)
               dt text
0  20160811 11:05    a
1  20160812 11:35    b
2  20160813 12:03    c
3  20160811 12:36    d
4  20160811 12:52    e
5  20160811 14:32    f

serie = pd.to_datetime(df.dt)
for i, g in df.groupby([serie.dt.date, serie.dt.hour]):
    print (g.reset_index(drop=True))
               dt text
0  20160811 11:05    a
               dt text
0  20160811 12:36    d
1  20160811 12:52    e
               dt text
0  20160811 14:32    f
               dt text
0  20160812 11:35    b
               dt text
0  20160813 12:03    c
```


Question:
How can I create a Matlab struct array from scipy.io? Consider the following Matlab code:
```
pmod(1).name{1}  = 'regressor1';
pmod(1).param{1} = [1 2 4 5 6];
pmod(1).poly{1}  = 1;
pmod(2).name{1}  = 'regressor2-1';
pmod(2).param{1} = [1 3 5 7];
pmod(2).poly{1}  = 1;
```
This creates a struct array. Each struct in the array contains three fields of type cell. As such, we have the following hierarchy in pmod:
```
pmod  // struct array
|
*- struct
|  |
|  *- cell  // contains 1 or more strings
|  *- cell  // contains 1 or more arrays
|  *- cell  // contains 1 or more arrays
|
*- struct [...]
```
I'm trying to use scipy.io to produce the above data structures in Python, such that they can be loaded into Matlab (this hierarchy is required by SPM). Creating a struct is straightforward, as scipy.io.savemat saves any dict whose keys are all of type str as a Matlab struct:
```
from scipy.io import savemat

struct = {
    'field1': 1,
    'field2': 2,
}

savemat('/tmp/p.mat', {'a_struct': struct})
```
However, when trying to generalize this to a struct array, I hit the following roadblock:
```
struct_array = [struct, struct]
savemat('/tmp/p.mat', {'s_array': struct_array})
```
This does not behave as expected; when loading p.mat into Matlab, I get a 1x2 cell array, not a struct array. How can I create a struct array using scipy.io? Notes: I have tried savemat('/tmp/p.mat', np.array(struct_array)) and savemat('/tmp/p.mat', np.array(struct_array, dtype=object)), to no avail.

Response:
You can use np.core.records.fromarrays to construct a record array, which is roughly equivalent to a MATLAB struct, and will be converted to a MATLAB struct by scip.io.savemat.
```
from numpy.core.records import fromarrays
from scipy.io import savemat

myrec = fromarrays([[1, 10], [2, 20]], names=['field1', 'field2'])
savemat('p.mat', {'myrec': myrec})
```
When opened in MATLAB, this gives:
```
>> load('p.mat')
>> myrec

myrec =

1x2 struct array with fields:

    field1
    field2
```


Question:
Python - Efficient way to find the largest area of a specific value in a 2D numpy array I have a 2D numpy array where some values are zero, and some are not. I'm trying to find an efficient way to find the biggest clump of zeros in the array (by returning the number of zeros, as well as a rough idea of where the center is) For example in this array, I would like to find the clump of 9, with the center of (3,4):
```
[[ 1, 1, 1, 0, 0 ],
 [ 1, 0, 1, 1, 0 ],
 [ 1, 1, 1, 1, 1 ],
 [ 1, 1, 0, 0, 0 ],
 [ 1, 1, 0, 0, 0 ],
 [ 1, 1, 0, 0, 0 ]]
```
Is there a nice vectorized way to accomplish something like this in numpy or scipy? The clumps will be roughly circular in shape, and have no holes in them. ndimage.label() from scipy does something close to this, but isn't quite what I'm after. I have a feeling numpy.where() and numpy.diff() could be helpful, but not sure how to efficiently use them to solve this problem.

Response:
You're almost there, you just need to combine ndimage.label with numpy.bincount:
```
import numpy as np
from scipy import ndimage

array = np.random.randint(0, 3, size=(200, 200))

label, num_label = ndimage.label(array == 0)
size = np.bincount(label.ravel())
biggest_label = size[1:].argmax() + 1
clump_mask = label == biggest_label
```
Once you have clump_mask you can compute the centroid or use some other method to get the center.


Question:
coefficients of spline interpolation in scipy I want to calculate the coefficients of a spline interpolation by scipy. In MATLAB:
```
x=[0:3];
y=[0,1,4,0];
spl=spline(x,y);
disp(spl.coefs);
```
and it will return:
```
ans =

   -1.5000    5.5000   -3.0000         0
   -1.5000    1.0000    3.5000    1.0000
   -1.5000   -3.5000    1.0000    4.0000
```
But i can't do that by interpolate.splrep in scipy. Can you tell me how to calc it?

Response:
I'm not sure there is any way to get exactly those coefficients from scipy. What scipy.interpolate.splrep gives you is the coefficients for the knots for a b-spline. What Matlab's spline gives you appears to be the partial polynomial coefficients describing the cubic equations connecting the points you pass in, which leads me to believe that the Matlab spline is a control-point based spline such as a Hermite or Catmull-Rom instead of a b-spline. However, scipy.interpolate.interpolate.spltopp does provide a way to get the partial polynomial coefficients of a b-spline. Unfortunately, it doesn't seem to work very well.
```

```python
import scipy.interpolate
```

```python
x = [0, 1, 2, 3]
```

```python
y = [0, 1, 4, 0]
```

```python
tck = scipy.interpolate.splrep(x, y)
```

```python
tck
#Output
#Out:
#    (array([ 0.,  0.,  0.,  0.,  3.,  3.,  3.,  3.]),
#    array([  3.19142761e-16,  -3.00000000e+00,   1.05000000e+01,
#        0.00000000e+00,   0.00000000e+00,   0.00000000e+00,
#        0.00000000e+00,   0.00000000e+00]),
#    3)
```

```python
pp = scipy.interpolate.interpolate.spltopp(tck[0][1:-1], tck[1], tck[2])
```

```python
pp.coeffs.T
#Output
#Out:
#    array([[ -4.54540394e-322,   0.00000000e+000,   0.00000000e+000,
#           0.00000000e+000],
#        [ -4.54540394e-322,   0.00000000e+000,   0.00000000e+000,
#           0.00000000e+000],
#        [ -4.54540394e-322,   0.00000000e+000,   0.00000000e+000,
#           0.00000000e+000],
#        [  0.00000000e+000,   0.00000000e+000,   0.00000000e+000,
#           0.00000000e+000],
#        [  0.00000000e+000,   0.00000000e+000,   0.00000000e+000,
#           0.00000000e+000]])
#```
#Note that there is one set of coefficients per knot, not one for each of the original points passed in. Also, multiplying the coefficients by the b-spline basis matrix doesn't seem to be very helpful.
#```
```

```python
bsbm = array([[-1,  3, -3,  1], [ 3, -6,  3,  0], [-3,  0,  3,  0],
#Output
#                 [ 1,  4,  1,  0]]) * 1.0/6
#Out:
#    array([[-0.16666667,  0.5       , -0.5       ,  0.16666667],
#        [ 0.5       , -1.        ,  0.5       ,  0.        ],
#        [-0.5       ,  0.        ,  0.5       ,  0.        ],
#        [ 0.16666667,  0.66666667,  0.16666667,  0.        ]])
```

```python
dot(pp.coeffs.T, bsbm)
#Output
#Out:
#    array([[  7.41098469e-323,  -2.27270197e-322,   2.27270197e-322,
#           -7.41098469e-323],
#        [  7.41098469e-323,  -2.27270197e-322,   2.27270197e-322,
#           -7.41098469e-323],
#        [  7.41098469e-323,  -2.27270197e-322,   2.27270197e-322,
#           -7.41098469e-323],
#        [  0.00000000e+000,   0.00000000e+000,   0.00000000e+000,
#           0.00000000e+000],
#        [  0.00000000e+000,   0.00000000e+000,   0.00000000e+000,
#           0.00000000e+000]])
#```
#The FORTRAN Piecewise Polynomial Package, PPPack, has a command bsplpp that converts from B-spline to piecewise polynomial form, which may serve your needs. Unfortunately, there isn't a Python wrapper for PPPack at this time.
```


Question:
Save numpy array as image with high precision (16 bits) with scikit-image I am working with 2D floating-point numpy arrays that I would like to save to greyscale .png files with high precision (e.g. 16 bits). I would like to do this using the scikit-image skimage.io package if possible. Here's the main thing I've tried:
```
import numpy as np
from skimage import io, exposure, img_as_uint, img_as_float

im = np.array([[1., 2.], [3., 4.]], dtype='float64')
im = exposure.rescale_intensity(im, out_range='float')
im = img_as_uint(im)
im
```
produces:
```
array([[    0, 21845],
       [43690, 65535]], dtype=uint16)
```
First I tried saving this as an image then reloading using the Python Imaging Library:
```
# try with pil:
io.use_plugin('pil')
io.imsave('test_16bit.png', im)
im2 = io.imread('test_16bit.png')
im2
```
produces:
```
array([[  0,  85],
       [170, 255]], dtype=uint8)
```
So somewhere (in either the write or read) I have lost precision. I then tried with the matplotlib plugin:
```
# try with matplotlib:
io.use_plugin('matplotlib')
io.imsave('test_16bit.png', im)
im3 = io.imread('test_16bit.png')
im3
```
gives me a 32-bit float:
```
array([[ 0.        ,  0.33333334],
       [ 0.66666669,  1.        ]], dtype=float32)
```
but I doubt this is really 32-bits given that I saved a 16-bit uint to the file. It would be great if someone could point me to where I'm going wrong. I would like this to extend to 3D arrays too (i.e. saving 16 bits per colour channel, for 48 bits per image). UPDATE: The problem is with imsave. The images are 8 bits per channel. How can one use io.imsave to output a high bit-depth image?

Response:
You wanna use the freeimage library to do so:
```
import numpy as np
from skimage import io, exposure, img_as_uint, img_as_float

io.use_plugin('freeimage')

im = np.array([[1., 2.], [3., 4.]], dtype='float64')
im = exposure.rescale_intensity(im, out_range='float')
im = img_as_uint(im)

io.imsave('test_16bit.png', im)
im2 = io.imread('test_16bit.png')
```
Result:
```
[[    0 21845]
 [43690 65535]]
```
As for 3D arrays, you need to construct the array properly and then it'll work:
```
# im = np.array([[1, 2.], [3., 4.]], dtype='float64')
im = np.linspace(0, 1., 300).reshape(10, 10, 3)
im = exposure.rescale_intensity(im, out_range='float')
im = img_as_uint(im)

io.imsave('test_16bit.png', im)
im2 = io.imread('test_16bit.png')
```
Note that the read image is flipped, so something like np.fliplr(np.flipud(im2)) will bring it to original shape.


Question:
How to get parameter arguments from a frozen spicy.stats distribution? Frozen Distribution In scipy.stats you can create a frozen distribution that allows the parameterization (shape, location & scale) of the distribution to be permanently set for that instance. For example, you can create an gamma distribution (scipy.stats.gamma) with a,loc and scale parameters and freeze them so they do not have to be passed around every time that distribution is needed.
```
import scipy.stats as stats

# Parameters for this particular gamma distribution
a, loc, scale = 3.14, 5.0, 2.0

# Do something with the general distribution parameterized
print 'gamma stats:', stats.gamma(a, loc=loc, scale=scale).stats()

# Create frozen distribution
rv = stats.gamma(a, loc=loc, scale=scale)

# Do something with the specific, already parameterized, distribution
print 'rv stats   :', rv.stats()
```
```
gamma stats: (array(11.280000000000001), array(12.56))
rv stats   : (array(11.280000000000001), array(12.56))
```
Accessible rv parameters? Since the parameters will most likely not be passed around as a result of this feature, is there a way to get those values back from only the frozen distribution, rv, later on?

Response:
Accessing rv frozen parameters Yes, the parameters used to create a frozen distribution are available within the instance of the distribution. They are stored within the args & kwds attribute. This will be dependent on if the distribution's instance was created with positional arguments or keyword arguments.
```
import scipy.stats as stats

# Parameters for this particular alpha distribution
a, loc, scale = 3.14, 5.0, 2.0

# Create frozen distribution
rv1 = stats.gamma(a, loc, scale)
rv2 = stats.gamma(a, loc=loc, scale=scale)

# Do something with frozen parameters
print 'positional and keyword'
print 'frozen args : {}'.format(rv1.args)
print 'frozen kwds : {}'.format(rv1.kwds)
print
print 'positional only'
print 'frozen args : {}'.format(rv2.args)
print 'frozen kwds : {}'.format(rv2.kwds)
```
```
positional and keyword
frozen args : (3.14, 5.0, 2.0)
frozen kwds : {}

positional only
frozen args : (3.14,)
frozen kwds : {'loc': 5.0, 'scale': 2.0}
```
Bonus: Private method that handles both args and kwds There is an private method, .dist._parse_args(), which handles both cases of parameter passing and will return a consistent result.
```
# Get the original parameters regardless of argument type
shape1, loc1, scale1 = rv1.dist._parse_args(*rv1.args, **rv1.kwds)
shape2, loc2, scale2 = rv2.dist._parse_args(*rv2.args, **rv2.kwds)

print 'positional and keyword'
print 'frozen parameters: shape={}, loc={}, scale={}'.format(shape1, loc1, scale1)
print
print 'positional only'
print 'frozen parameters: shape={}, loc={}, scale={}'.format(shape2, loc2, scale2)
```
```
positional and keyword
frozen parameters: shape=(3.14,), loc=5.0, scale=2.0

positional only
frozen parameters: shape=(3.14,), loc=5.0, scale=2.0
```
Caveat Granted, using private methods is typically bad practice because technically internal APIs can always change, however, sometimes they provide nice features, would be easy to re-implement should things change and nothing is really private in Python :).


Question: Fitting data with integral function When using curve_fit from scipy.optimize to fit a some data in python, one first defines the fitting function (e.g. a 2nd order polynomial) as follows: def f(x, a, b): return a*x**2+b*x And then proceeds with the fitting popt, pcov = curve_fit(f,x,y) But the question is now, how does one go about defining the function in point 1. if the function contains an integral (or a discrete sum), e.g.: The experimental data is still given for x and f(x), so point 2. would be similar I imagine once I can define f(x) in python. By the way I forgot to say that it is assumed that g(t) has a well known form here, and contains the fitting parameters, i.e. parameters like a and b given in the polynomial example. Any help is much appreciated. The question is really supposed to be a generic one, and the functions used in the post are just random examples.

Response:
Here's an example of fitting a curve defined in terms of an integral. The curve is the integral of sin(t*w)/t+p over t from 0 to Pi. Our x data points correspond to w, and we're adjusting the p parameter to to get the data to fit.
```
import math, numpy, scipy.optimize, scipy.integrate

def integrand(t, args):
    w, p = args
    return math.sin(t * w)/t + p

def curve(w, p):
    res = scipy.integrate.quad(integrand, 0.0, math.pi, [w, p])
    return res[0]

vcurve = numpy.vectorize(curve, excluded=set([1]))

truexdata = numpy.asarray([0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0])
trueydata = vcurve(truexdata, 1.0)

xdata = truexdata + 0.1 * numpy.random.randn(8)
ydata = trueydata + 0.1 * numpy.random.randn(8)

popt, pcov = scipy.optimize.curve_fit(vcurve,
                                      xdata, ydata,
                                      p0=[2.0])
print popt
```
That'll print something out fairly close to 1.0, which is what we used as p when we created the trueydata. Note that we use numpy.vectorize on the curve function to produce a vectorized version compatible with scipy.optimize.curve_fit.