Question:
Volume of Voronoi cell (python) I'm using Scipy 0.13.0 in Python 2.7 to calculate a set of Voronoi cells in 3d. I need to get the volume of each cell for (de)weighting output of a proprietary simulation. Is there any simple way of doing this - surely it's a common problem or a common use of Voronoi cells but I can't find anything. The following code runs, and dumps everything that the scipy.spatial.Voronoi manual knows about.
```
from scipy.spatial import Voronoi
x=[0,1,0,1,0,1,0,1,0,1]
y=[0,0,1,1,2,2,3,3.5,4,4.5]
z=[0,0,0,0,0,1,1,1,1,1]
points=zip(x,y,z)
print points
vor=Voronoi(points)
print vor.regions
print vor.vertices
print vor.ridge_points
print vor.ridge_vertices
print vor.points
print vor.point_region
```

Response:
As was mentioned in comments, you can compute ConvexHull of each Voronoi cell. Since Voronoi cells are convex, you will get the proper volumes.
```
def voronoi_volumes(points):
    v = Voronoi(points)
    vol = np.zeros(v.npoints)
    for i, reg_num in enumerate(v.point_region):
        indices = v.regions[reg_num]
        if -1 in indices: # some regions can be opened
            vol[i] = np.inf
        else:
            vol[i] = ConvexHull(v.vertices[indices]).volume
    return vol
```
This method works in any dimensions


Question:
Open / load image as numpy ndarray directly I used to use scipy which would load an image from file straight into an ndarray.
```
from scipy import misc
img = misc.imread('./myimage.jpg')
type(img)

```python
numpy.ndarray
#Output
#```
#But now it gives me a DeprecationWarning and the docs say it will be removed in 1.2.0. and I should use imageio.imread instead. But:
#```
#import imageio
#img = imageio.imread('./myimage.jpg')
#type(img)
```

```python
imageio.core.util.Image
#Output
#```
#I could convert it by doing
#```
#img = numpy.array(img)
#```
#But this seems hacky. Is there any way to load an image straight into a numpy array as I was doing before with scipy's misc.imread (other than using OpenCV)?
```

Response:
The result of imageio.imread is already a NumPy array; imageio.core.util.Image is an ndarray subclass that exists primarily so the array can have a meta attribute holding image metadata. If you want an object of type exactly numpy.ndarray, you can use asarray:
```
array = numpy.asarray(img)
```
Unlike numpy.array(img), this will not copy img's data.


Question: SciPy Create 2D Polygon Mask I need to create a numpy 2D array which represents a binary mask of a polygon, using standard Python packages. input: polygon vertices, image dimensions output: binary mask of polygon (numpy 2D array) (Larger context: I want to get the distance transform of this polygon using scipy.ndimage.morphology.distance_transform_edt.) Can anyone show me how to do this?

Response:
The answer turns out to be quite simple:
```
import numpy
from PIL import Image, ImageDraw

# polygon = [(x1,y1),(x2,y2),...] or [x1,y1,x2,y2,...]
# width = ?
# height = ?

img = Image.new('L', (width, height), 0)
ImageDraw.Draw(img).polygon(polygon, outline=1, fill=1)
mask = numpy.array(img)
```


Question:
In scipy's ConvexHull, what does "area" measure? The value of the "area" attribute in scipy ConvexHull (see http://docs.scipy.org/doc/scipy/reference/generated/scipy.spatial.ConvexHull.html) object does not seem to be (what I understand to be) the area of the convex hull. On the other hand, the value of "volume" does seem to be the area of the convex hull.
```
from scipy.spatial import ConvexHull
import numpy

points = numpy.array([[-1,-1], [1,1], [-1, 1], [1,-1]])
hull = ConvexHull(points)

print("Volume is %2.2f" % hull.volume) # Prints 4.00
print("Area is %2.2f" % hull.area) # Prints 8.00
```
In the above example, I expect the area of the convex hull of the 4 points to be 4.0. That's what the "volume" is. What then does "area" give us?

Response: Volume and area are 3d concepts, but your data is 2d - a 2x2 square. Its area is 4, and perimeter is 8 (the 2d counterparts).


Question:
How to read part of binary file with numpy? I'm converting a matlab script to numpy, but have some problems with reading data from a binary file. Is there an equivelent to fseek when using fromfile to skip the beginning of the file? This is the type of extractions I need to do:
```
fid = fopen(fname);
fseek(fid, 8, 'bof');
second = fread(fid, 1, 'schar');
fseek(fid, 100, 'bof');
total_cycles = fread(fid, 1, 'uint32', 0, 'l');
start_cycle = fread(fid, 1, 'uint32', 0, 'l');
```
Thanks!

Response:
You can use seek with a file object in the normal way, and then use this file object in fromfile. Here's a full example:
```
import numpy as np
import os

data = np.arange(100, dtype=np.int)
data.tofile("temp")  # save the data

f = open("temp", "rb")  # reopen the file
f.seek(256, os.SEEK_SET)  # seek

x = np.fromfile(f, dtype=np.int)  # read the data into numpy
print x
# [64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88
# 89 90 91 92 93 94 95 96 97 98 99]
```


Question:
Efficient method of calculating density of irregularly spaced points I am attempting to generate map overlay images that would assist in identifying hot-spots, that is areas on the map that have high density of data points. None of the approaches that I've tried are fast enough for my needs. Note: I forgot to mention that the algorithm should work well under both low and high zoom scenarios (or low and high data point density). I looked through numpy, pyplot and scipy libraries, and the closest I could find was numpy.histogram2d. As you can see in the image below, the histogram2d output is rather crude. (Each image includes points overlaying the heatmap for better understanding) My second attempt was to iterate over all the data points, and then calculate the hot-spot value as a function of distance. This produced a better looking image, however it is too slow to use in my application. Since it's O(n), it works ok with 100 points, but blows out when I use my actual dataset of 30000 points. My final attempt was to store the data in an KDTree, and use the nearest 5 points to calculate the hot-spot value. This algorithm is O(1), so much faster with large dataset. It's still not fast enough, it takes about 20 seconds to generate a 256x256 bitmap, and I would like this to happen in around 1 second time. Edit The boxsum smoothing solution provided by 6502 works well at all zoom levels and is much faster than my original methods. The gaussian filter solution suggested by Luke and Neil G is the fastest. You can see all four approaches below, using 1000 data points in total, at 3x zoom there are around 60 points visible. Complete code that generates my original 3 attempts, the boxsum smoothing solution provided by 6502 and gaussian filter suggested by Luke (improved to handle edges better and allow zooming in) is here:
```
import matplotlib
import numpy as np
from matplotlib.mlab import griddata
import matplotlib.cm as cm
import matplotlib.pyplot as plt
import math
from scipy.spatial import KDTree
import time
import scipy.ndimage as ndi

def grid_density_kdtree(xl, yl, xi, yi, dfactor):
    zz = np.empty([len(xi),len(yi)], dtype=np.uint8)
    zipped = zip(xl, yl)
    kdtree = KDTree(zipped)
    for xci in range(0, len(xi)):
        xc = xi[xci]
        for yci in range(0, len(yi)):
            yc = yi[yci]
            density = 0.
            retvalset = kdtree.query((xc,yc), k=5)
            for dist in retvalset[0]:
                density = density + math.exp(-dfactor * pow(dist, 2)) / 5
            zz[yci][xci] = min(density, 1.0) * 255
    return zz

def grid_density(xl, yl, xi, yi):
    ximin, ximax = min(xi), max(xi)
    yimin, yimax = min(yi), max(yi)
    xxi,yyi = np.meshgrid(xi,yi)
    #zz = np.empty_like(xxi)
    zz = np.empty([len(xi),len(yi)])
    for xci in range(0, len(xi)):
        xc = xi[xci]
        for yci in range(0, len(yi)):
            yc = yi[yci]
            density = 0.
            for i in range(0,len(xl)):
                xd = math.fabs(xl[i] - xc)
                yd = math.fabs(yl[i] - yc)
                if xd < 1 and yd < 1:
                    dist = math.sqrt(math.pow(xd, 2) + math.pow(yd, 2))
                    density = density + math.exp(-5.0 * pow(dist, 2))
            zz[yci][xci] = density
    return zz

def boxsum(img, w, h, r):
    st = [0] * (w+1) * (h+1)
    for x in xrange(w):
        st[x+1] = st[x] + img[x]
    for y in xrange(h):
        st[(y+1)*(w+1)] = st[y*(w+1)] + img[y*w]
        for x in xrange(w):
            st[(y+1)*(w+1)+(x+1)] = st[(y+1)*(w+1)+x] + st[y*(w+1)+(x+1)] - st[y*(w+1)+x] + img[y*w+x]
    for y in xrange(h):
        y0 = max(0, y - r)
        y1 = min(h, y + r + 1)
        for x in xrange(w):
            x0 = max(0, x - r)
            x1 = min(w, x + r + 1)
            img[y*w+x] = st[y0*(w+1)+x0] + st[y1*(w+1)+x1] - st[y1*(w+1)+x0] - st[y0*(w+1)+x1]

def grid_density_boxsum(x0, y0, x1, y1, w, h, data):
    kx = (w - 1) / (x1 - x0)
    ky = (h - 1) / (y1 - y0)
    r = 15
    border = r * 2
    imgw = (w + 2 * border)
    imgh = (h + 2 * border)
    img = [0] * (imgw * imgh)
    for x, y in data:
        ix = int((x - x0) * kx) + border
        iy = int((y - y0) * ky) + border
        if 0 <= ix < imgw and 0 <= iy < imgh:
            img[iy * imgw + ix] += 1
    for p in xrange(4):
        boxsum(img, imgw, imgh, r)
    a = np.array(img).reshape(imgh,imgw)
    b = a[border:(border+h),border:(border+w)]
    return b

def grid_density_gaussian_filter(x0, y0, x1, y1, w, h, data):
    kx = (w - 1) / (x1 - x0)
    ky = (h - 1) / (y1 - y0)
    r = 20
    border = r
    imgw = (w + 2 * border)
    imgh = (h + 2 * border)
    img = np.zeros((imgh,imgw))
    for x, y in data:
        ix = int((x - x0) * kx) + border
        iy = int((y - y0) * ky) + border
        if 0 <= ix < imgw and 0 <= iy < imgh:
            img[iy][ix] += 1
    return ndi.gaussian_filter(img, (r,r))  ## gaussian convolution

def generate_graph():
    n = 1000
    # data points range
    data_ymin = -2.
    data_ymax = 2.
    data_xmin = -2.
    data_xmax = 2.
    # view area range
    view_ymin = -.5
    view_ymax = .5
    view_xmin = -.5
    view_xmax = .5
    # generate data
    xl = np.random.uniform(data_xmin, data_xmax, n)
    yl = np.random.uniform(data_ymin, data_ymax, n)
    zl = np.random.uniform(0, 1, n)

    # get visible data points
    xlvis = []
    ylvis = []
    for i in range(0,len(xl)):
        if view_xmin < xl[i] < view_xmax and view_ymin < yl[i] < view_ymax:
            xlvis.append(xl[i])
            ylvis.append(yl[i])

    fig = plt.figure()

    # plot histogram
    plt1 = fig.add_subplot(221)
    plt1.set_axis_off()
    t0 = time.clock()
    zd, xe, ye = np.histogram2d(yl, xl, bins=10, range=[[view_ymin, view_ymax],[view_xmin, view_xmax]], normed=True)
    plt.title('numpy.histogram2d - '+str(time.clock()-t0)+"sec")
    plt.imshow(zd, origin='lower', extent=[view_xmin, view_xmax, view_ymin, view_ymax])
    plt.scatter(xlvis, ylvis)

    # plot density calculated with kdtree
    plt2 = fig.add_subplot(222)
    plt2.set_axis_off()
    xi = np.linspace(view_xmin, view_xmax, 256)
    yi = np.linspace(view_ymin, view_ymax, 256)
    t0 = time.clock()
    zd = grid_density_kdtree(xl, yl, xi, yi, 70)
    plt.title('function of 5 nearest using kdtree\n'+str(time.clock()-t0)+"sec")
    cmap=cm.jet
    A = (cmap(zd/256.0)*255).astype(np.uint8)
    #A[:,:,3] = zd
    plt.imshow(A , origin='lower', extent=[view_xmin, view_xmax, view_ymin, view_ymax])
    plt.scatter(xlvis, ylvis)

    # gaussian filter
    plt3 = fig.add_subplot(223)
    plt3.set_axis_off()
    t0 = time.clock()
    zd = grid_density_gaussian_filter(view_xmin, view_ymin, view_xmax, view_ymax, 256, 256, zip(xl, yl))
    plt.title('ndi.gaussian_filter - '+str(time.clock()-t0)+"sec")
    plt.imshow(zd , origin='lower', extent=[view_xmin, view_xmax, view_ymin, view_ymax])
    plt.scatter(xlvis, ylvis)

    # boxsum smoothing
    plt3 = fig.add_subplot(224)
    plt3.set_axis_off()
    t0 = time.clock()
    zd = grid_density_boxsum(view_xmin, view_ymin, view_xmax, view_ymax, 256, 256, zip(xl, yl))
    plt.title('boxsum smoothing - '+str(time.clock()-t0)+"sec")
    plt.imshow(zd, origin='lower', extent=[view_xmin, view_xmax, view_ymin, view_ymax])
    plt.scatter(xlvis, ylvis)

if __name__=='__main__':
    generate_graph()
    plt.show()
```

Response:
This approach is along the lines of some previous answers: increment a pixel for each spot, then smooth the image with a gaussian filter. A 256x256 image runs in about 350ms on my 6-year-old laptop.
```
import numpy as np
import scipy.ndimage as ndi

data = np.random.rand(30000,2)           ## create random dataset
inds = (data * 255).astype('uint')       ## convert to indices

img = np.zeros((256,256))                ## blank image
for i in xrange(data.shape[0]):          ## draw pixels
    img[inds[i,0], inds[i,1]] += 1

img = ndi.gaussian_filter(img, (10,10))
```


Question:
Reason why numpy rollaxis is so confusing? The behavior of the numpy rollaxis function confuses me. The documentation says: Roll the specified axis backwards, until it lies in a given position. And for the start parameter: The axis is rolled until it lies before this position. To me, this is already somehow inconsistent. Ok, straight forward example (from the documentation):
```

```python
a = np.ones((3,4,5,6))
```

```python
np.rollaxis(a, 1, 4).shape
#Output
#(3, 5, 6, 4)
#```
#The axis at index 1 (4) is rolled backward till it lies before index 4. Now, when the start index is smaller than the axis index, we have this behavior:
#```
```

```python
np.rollaxis(a, 3, 1).shape
#Output
#(3, 6, 4, 5)
#```
#Instead of shifting the axis at index 3 before index 1, it ends up at 1. Why is that? Why isn't the axis always rolled to the given start index?
```

Response:
NumPy v1.11 and newer includes a new function, moveaxis, that I recommend using instead of rollaxis (disclaimer: I wrote it!). The source axis always ends up at the destination, without any funny off-by-one issues depending on whether start is greater or less than end:
```
import numpy as np

x = np.zeros((1, 2, 3, 4, 5))
for i in range(5):
    print(np.moveaxis(x, 3, i).shape)
```
Results in:
```
(4, 1, 2, 3, 5)
(1, 4, 2, 3, 5)
(1, 2, 4, 3, 5)
(1, 2, 3, 4, 5)
(1, 2, 3, 5, 4)
```


Question:
Obtaining values used in boxplot, using python and matplotlib I can draw a boxplot from data:
```
import numpy as np
import matplotlib.pyplot as plt

data = np.random.rand(100)
plt.boxplot(data)
```
Then, the box will range from the 25th-percentile to 75th-percentile, and the whisker will range from the smallest value to the largest value between (25th-percentile - 1.5*IQR, 75th-percentile + 1.5*IQR), where the IQR denotes the inter-quartile range. (Of course, the value 1.5 is customizable). Now I want to know the values used in the boxplot, i.e. the median, upper and lower quartile, the upper whisker end point and the lower whisker end point. While the former three are easy to obtain by using np.median() and np.percentile(), the end point of the whiskers will require some verbose coding:
```
median = np.median(data)
upper_quartile = np.percentile(data, 75)
lower_quartile = np.percentile(data, 25)

iqr = upper_quartile - lower_quartile
upper_whisker = data[data<=upper_quartile+1.5*iqr].max()
lower_whisker = data[data>=lower_quartile-1.5*iqr].min()
```
I was wondering, while this is acceptable, would there be a neater way to do this? It seems that the values should be ready to pull-out from the boxplot, as it's already drawn.

Response:
Why do you want to do so? what you are doing is already pretty direct. Yeah, if you want to fetch them for the plot, when the plot is already made, simply use the get_ydata() method.
```
B = plt.boxplot(data)
[item.get_ydata() for item in B['whiskers']]
```
It returns an array of the shape (2,) for each whiskers, the second element is the value we want:
```
[item.get_ydata()[1] for item in B['whiskers']]
```


Question: scipy.stats seed? I am trying to generate scipy.stats.pareto.rvs(b, loc=0, scale=1, size=1) with different seed. In numpy we can seed using numpy.random.seed(seed=233423). Is there any way to seed the random number generated by scipy stats. Note: I am not using numpy pareto because I want to give different values for scale.

Response:
scipy.stats just uses numpy.random to generate its random numbers, so numpy.random.seed() will work here as well. E.g.,
```
import numpy as np
from scipy.stats import pareto
b = 0.9
np.random.seed(seed=233423)
print pareto.rvs(b, loc=0, scale=1, size=5)
np.random.seed(seed=233423)
print pareto.rvs(b, loc=0, scale=1, size=5)
```
will print [ 9.7758784 10.78405752 4.19704602 1.19256849 1.02750628] twice.


Question: Convert ndarray from float64 to integer I've got an ndarray in python with a dtype of float64. I'd like to convert the array to be an array of integers. How should I do this? int() won't work, as it says it can't convert it to a scalar. Changing the dtype field itself obviously doesn't work, as the actual bytes haven't changed. I can't seem to find anything on Google or in the documentation - what's the best way to do this?

Response:
Use .astype.
```

```python
a = numpy.array([1, 2, 3, 4], dtype=numpy.float64)
```

```python
a
#Output
#array([ 1.,  2.,  3.,  4.])
```

```python
a.astype(numpy.int64)
#Output
#array([1, 2, 3, 4])
#```
#See the documentation for more options.
```


Question:
Weighted moving average in python I have data sampled at essentially random intervals. I would like to compute a weighted moving average using numpy (or other python package). I have a crude implementation of a moving average, but I am having trouble finding a good way to do a weighted moving average, so that the values towards the center of the bin are weighted more than values towards the edges. Here I generate some sample data and then take a moving average. How can I most easily implement a weighted moving average? Thanks!
```
import numpy as np
import matplotlib.pyplot as plt

#first generate some datapoint for a randomly sampled noisy sinewave
x = np.random.random(1000)*10
noise = np.random.normal(scale=0.3,size=len(x))
y = np.sin(x) + noise

#plot the data
plt.plot(x,y,'ro',alpha=0.3,ms=4,label='data')
plt.xlabel('Time')
plt.ylabel('Intensity')

#define a moving average function
def moving_average(x,y,step_size=.1,bin_size=1):
    bin_centers  = np.arange(np.min(x),np.max(x)-0.5*step_size,step_size)+0.5*step_size
    bin_avg = np.zeros(len(bin_centers))

    for index in range(0,len(bin_centers)):
        bin_center = bin_centers[index]
        items_in_bin = y[(x>(bin_center-bin_size*0.5) ) & (x<(bin_center+bin_size*0.5))]
        bin_avg[index] = np.mean(items_in_bin)

    return bin_centers,bin_avg

#plot the moving average
bins, average = moving_average(x,y)
plt.plot(bins, average,label='moving average')

plt.show()
```
The output: Using the advice from crs17 to use "weights=" in the np.average function, I came up weighted average function, which uses a Gaussian function to weight the data:
```
def weighted_moving_average(x,y,step_size=0.05,width=1):
    bin_centers  = np.arange(np.min(x),np.max(x)-0.5*step_size,step_size)+0.5*step_size
    bin_avg = np.zeros(len(bin_centers))

    #We're going to weight with a Gaussian function
    def gaussian(x,amp=1,mean=0,sigma=1):
        return amp*np.exp(-(x-mean)**2/(2*sigma**2))

    for index in range(0,len(bin_centers)):
        bin_center = bin_centers[index]
        weights = gaussian(x,mean=bin_center,sigma=width)
        bin_avg[index] = np.average(y,weights=weights)

    return (bin_centers,bin_avg)
```
Results look good:

Response:
You could use numpy.average which allows you to specify weights:
```

```python
bin_avg[index] = np.average(items_in_bin, weights=my_weights)
#Output
#```
#So to calculate the weights you could find the x coordinates of each data point in the bin and calculate their distances to the bin center.
```


Question: Multivariate normal density in Python? Is there any python package that allows the efficient computation of the PDF (probability density function) of a multivariate normal distribution? It doesn't seem to be included in Numpy/Scipy, and surprisingly a Google search didn't turn up any useful thing.

Response:
The multivariate normal is now available on SciPy 0.14.0.dev-16fc0af:
```
from scipy.stats import multivariate_normal
var = multivariate_normal(mean=[0,0], cov=[[1,0],[0,1]])
var.pdf([1,0])
```


Question:
Multiple linear regression in Python I can't seem to find any python libraries that do multiple regression. The only things I find only do simple regression. I need to regress my dependent variable (y) against several independent variables (x1, x2, x3, etc.). For example, with this data:
```
print 'y        x1      x2       x3       x4      x5     x6       x7'
for t in texts:
    print "{:>7.1f}{:>10.2f}{:>9.2f}{:>9.2f}{:>10.2f}{:>7.2f}{:>7.2f}{:>9.2f}" /
   .format(t.y,t.x1,t.x2,t.x3,t.x4,t.x5,t.x6,t.x7)
```
(output for above:)
```
y        x1       x2       x3        x4     x5     x6       x7
   -6.0     -4.95    -5.87    -0.76     14.73   4.02   0.20     0.45
   -5.0     -4.55    -4.52    -0.71     13.74   4.47   0.16     0.50
  -10.0    -10.96   -11.64    -0.98     15.49   4.18   0.19     0.53
   -5.0     -1.08    -3.36     0.75     24.72   4.96   0.16     0.60
   -8.0     -6.52    -7.45    -0.86     16.59   4.29   0.10     0.48
   -3.0     -0.81    -2.36    -0.50     22.44   4.81   0.15     0.53
   -6.0     -7.01    -7.33    -0.33     13.93   4.32   0.21     0.50
   -8.0     -4.46    -7.65    -0.94     11.40   4.43   0.16     0.49
   -8.0    -11.54   -10.03    -1.03     18.18   4.28   0.21     0.55
```
How would I regress these in python, to get the linear regression formula: Y = a1x1 + a2x2 + a3x3 + a4x4 + a5x5 + a6x6 + +a7x7 + c

Response:
sklearn.linear_model.LinearRegression will do it:
```
from sklearn import linear_model
clf = linear_model.LinearRegression()
clf.fit([[getattr(t, 'x%d' % i) for i in range(1, 8)] for t in texts],
        [t.y for t in texts])
```
Then clf.coef_ will have the regression coefficients. sklearn.linear_model also has similar interfaces to do various kinds of regularizations on the regression.


Question: How can I remove a column from a sparse matrix efficiently? If I am using the sparse.lil_matrix format, how can I remove a column from the matrix easily and efficiently?

Response:
Much simpler and faster. You might not even need the conversion to csr, but I just know for sure that it works with csr sparse matrices and converting between shouldn't be an issue.
```
from scipy import sparse

x_new = sparse.lil_matrix(sparse.csr_matrix(x)[:,col_list])
```


Question:
Importing sound files into Python as NumPy arrays (alternatives to audiolab) I've been using Audiolab to import sound files in the past, and it worked quite well. However: It doesn't support some formats, like mp3, because the underlying libsndfile refuses to support them It doesn't work in Python 2.6 under Windows, and the author is not around to fix it -
```

```python
from scikits import audiolab
--------------------------------------------------------------------

ImportError                               Traceback (most recent call last)

C:\Python26\Scripts\<ipython console> in <module>()

C:\Python26\lib\site-packages\scikits\audiolab\__init__.py in <module>()
     23 __version__ = _version
     24
---> 25 from pysndfile import formatinfo, sndfile
     26 from pysndfile import supported_format, supported_endianness, \
     27                       supported_encoding, PyaudioException, \

C:\Python26\lib\site-packages\scikits\audiolab\pysndfile\__init__.py in <module>()
----> 1 from _sndfile import Sndfile, Format, available_file_formats, available_encodings
      2 from compat import formatinfo, sndfile, PyaudioException, PyaudioIOError
      3 from compat import supported_format, supported_endianness, supported_encoding

ImportError: DLL load failed: The specified module could not be found.``
```
So I would like to either: Figure out why it's not working in 2.6 (something wrong with _sndfile.pyd?) and maybe find a way to extend it to work with unsupported formats Find a complete replacement for audiolab
```

Response:
Audiolab is working for me on Ubuntu 9.04 with Python 2.6.2, so it might be a Windows problem. In your link to the forum, the author also suggests that it is a Windows error. In the past, this option has worked for me, too:
```
from scipy.io import wavfile
fs, data = wavfile.read(filename)
```
Just beware that data may have int data type, so it is not scaled within [-1,1). For example, if data is int16, you must divide data by 2**15 to scale within [-1,1).


Question:
Linear regression of arrays containing NAN values I have two arrays, say varx and vary. Both contain NaN values at various positions. However, I would like to do a linear regression on both to show how much the two arrays correlate. This was very helpful so far. However, using the following
```py
slope, intercept, r_value, p_value, std_err = stats.linregress(varx, vary)
```
results in NaNs for every output variable. What is the most convenient way to take only valid values from both arrays as input to the linear regression? I heard about masking arrays, but am not sure how it works exactly.

Response:
You can remove NaNs using a mask:
```
mask = ~np.isnan(varx) & ~np.isnan(vary)
slope, intercept, r_value, p_value, std_err = stats.linregress(varx[mask], vary[mask])
```


Question: plus/minus operator for python ± I am looking for a way to do a plus/minus operation in python 2 or 3. I do not know the command or operator, and I cannot find a command or operator to do this. Am I missing something?

Response:
If you are looking to print the ± symbol, just use:
```
print(u"\u00B1")
```


Question:
Python curve fit library that allows me to assign bounds to parameters I'd like to be able to perform fits that allows me to fit an arbitrary curve function to data, and allows me to set arbitrary bounds on parameters, for example I want to fit function:
```
f(x) = a1(x-a2)^a3\cdot\exp(-\a4*x^a5)
```
and say: a2 is in following range: (-1, 1) a3 and a5 are positive There is nice scipy curve_fit function, but it doesn't allow to specify parameter bounds. There also is nice http://code.google.com/p/pyminuit/ library that does generic minimalization, and it allows to set bounds on parameters, but in my case it did not coverge.

Response:
Note: New in version 0.17 of SciPy Let's suppose you want to fit a model to the data which looks like this:
```
y=a*t**alpha+b
```
and with the constraint on alpha
```
0<alpha<2
```
while other parameters a and b remains free. Then we should use the bounds option of curve_fit in the following fashion:
```
import numpy as np
from scipy.optimize import curve_fit
def func(t, a,alpha,b):
     return a*t**alpha+b
param_bounds=([-np.inf,0,-np.inf],[np.inf,2,np.inf])
popt, pcov = curve_fit(func, xdata, ydata,bounds=param_bounds)
```
Source is here.


Question: How do I read CSV data into a record array in NumPy? Is there a direct way to import the contents of a CSV file into a record array, just like how R's read.table(), read.delim(), and read.csv() import data into R dataframes? Or should I use csv.reader() and then apply numpy.core.records.fromrecords()?

Response:
Use numpy.genfromtxt() by setting the delimiter kwarg to a comma:
```
from numpy import genfromtxt
my_data = genfromtxt('my_file.csv', delimiter=',')
```


Question: how to get derivatives from 1D interpolation Is there a way to get scipy's interp1d (in linear mode) to return the derivative at each interpolated point? I could certainly write my own 1D interpolation routine that does, but presumably scipy's is internally in C and therefore faster, and speed is already a major issue. I am ultimately feeding a munging of the interpolated function into a multi-dimensional minimization routine, so being able to pass analytic derivatives would speed things up a lot rather than having the minimization routine try to calculate them itself. And interp1d must be calculating them internally --- so can I access them?

Response: Use UnivariateSpline instead of interp1d, and use the derivative method to generate the first derivative. The example at the manual page here is pretty self-explanatory.


Question: How to solve a pair of nonlinear equations using Python? What's the (best) way to solve a pair of non linear equations using Python. (Numpy, Scipy or Sympy) eg: x+y^2 = 4 e^x+ xy = 3 A code snippet which solves the above pair will be great

Response:
for numerical solution, you can use fsolve: http://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.fsolve.html#scipy.optimize.fsolve
```
from scipy.optimize import fsolve
import math

def equations(p):
    x, y = p
    return (x+y**2-4, math.exp(x) + x*y - 3)

x, y =  fsolve(equations, (1, 1))

print equations((x, y))
```


Question:
How to calculate the statistics "t-test" with numpy I'm looking to generate some statistics about a model I created in python. I'd like to generate the t-test on it, but was wondering if there was an easy way to do this with numpy/scipy. Are there any good explanations around? For example, I have three related datasets that look like this:
```
[55.0, 55.0, 47.0, 47.0, 55.0, 55.0, 55.0, 63.0]
```
Now, I would like to do the student's t-test on them.

Response:
In a scipy.stats package there are few ttest_... functions. See example from here:
```

```python
print 't-statistic = %6.3f pvalue = %6.4f' %  stats.ttest_1samp(x, m)
#Output
#t-statistic =  0.391 pvalue = 0.6955
#```
```


Question:
Extract blocks or patches from NumPy Array I have a 2-d numpy array as follows:
```
a = np.array([[1,5,9,13],
              [2,6,10,14],
              [3,7,11,15],
              [4,8,12,16]]
```
I want to extract it into patches of 2 by 2 sizes with out repeating the elements. The answer should exactly be the same. This can be 3-d array or list with the same order of elements as below:
```
[[[1,5],
 [2,6]],

 [[3,7],
 [4,8]],

 [[9,13],
 [10,14]],

 [[11,15],
 [12,16]]]
```
How can do it easily? In my real problem the size of a is (36, 72). I can not do it one by one. I want programmatic way of doing it.

Response:
Using scikit-image:
```
import numpy as np
from skimage.util import view_as_blocks

a = np.array([[1,5,9,13],
              [2,6,10,14],
              [3,7,11,15],
              [4,8,12,16]])

print(view_as_blocks(a, (2, 2)))
```


Question:
Minimize function with parameters Currently I have the following code that defines the function f.
```
a = #something
b = #something
c = #something
def f(x):
    """Evaluates some function that depends on parameters a, b, and c"""
    someNumber = #some calculation
    return someNumber
```
Ideally I would do def f(x, a, b, c), BUT I am minimizing f with respect to x and SciPy's optimization toolbox doesn't allow for one to minimize functions with parameters in the arguments. That said I would like to run my minimization code for multiple values of a, b and c. Is there a way I can do this?

Response:
You can specify additional arguments in args
```
from scipy.optimize import minimize
minimize(f, x0, args=(a, b, c))
```


Question:
Sorting arrays in NumPy by column How do I sort a NumPy array by its nth column? For example, given:
```
a = array([[9, 2, 3],
           [4, 5, 6],
           [7, 0, 5]])
```
I want to sort the rows of a by the second column to obtain:
```
array([[7, 0, 5],
       [9, 2, 3],
       [4, 5, 6]])
```

Response:
To sort by the second column of a:
```
a[a[:, 1].argsort()]
```


Question: How to save/restore a model after training? After you train a model in Tensorflow: How do you save the trained model? How do you later restore this saved model?

Response:
In (and after) Tensorflow version 0.11: Save the model:
```
import tensorflow as tf

#Prepare to feed input, i.e. feed_dict and placeholders
w1 = tf.placeholder("float", name="w1")
w2 = tf.placeholder("float", name="w2")
b1= tf.Variable(2.0,name="bias")
feed_dict ={w1:4,w2:8}

#Define a test operation that we will restore
w3 = tf.add(w1,w2)
w4 = tf.multiply(w3,b1,name="op_to_restore")
sess = tf.Session()
sess.run(tf.global_variables_initializer())

#Create a saver object which will save all the variables
saver = tf.train.Saver()

#Run the operation by feeding input
print sess.run(w4,feed_dict)
#Prints 24 which is sum of (w1+w2)*b1

#Now, save the graph
saver.save(sess, 'my_test_model',global_step=1000)
```
Restore the model:
```
import tensorflow as tf

sess=tf.Session()
#First let's load meta graph and restore weights
saver = tf.train.import_meta_graph('my_test_model-1000.meta')
saver.restore(sess,tf.train.latest_checkpoint('./'))

# Access saved Variables directly
print(sess.run('bias:0'))
# This will print 2, which is the value of bias that we saved

# Now, let's access and create placeholders variables and
# create feed-dict to feed new data

graph = tf.get_default_graph()
w1 = graph.get_tensor_by_name("w1:0")
w2 = graph.get_tensor_by_name("w2:0")
feed_dict ={w1:13.0,w2:17.0}

#Now, access the op that you want to run.
op_to_restore = graph.get_tensor_by_name("op_to_restore:0")

print sess.run(op_to_restore,feed_dict)
#This will print 60 which is calculated
```
This and some more advanced use-cases have been explained very well here. A quick complete tutorial to save and restore Tensorflow models


Question: What is the difference between 'SAME' and 'VALID' padding in tf.nn.max_pool of tensorflow? What is the difference between 'SAME' and 'VALID' padding in tf.nn.max_pool of tensorflow? In my opinion, 'VALID' means there will be no zero padding outside the edges when we do max pool. According to A guide to convolution arithmetic for deep learning, it says that there will be no padding in pool operator, i.e. just use 'VALID' of tensorflow. But what is 'SAME' padding of max pool in tensorflow?

Response:
If you like ascii art: "VALID" = without padding:
```
inputs:         1  2  3  4  5  6  7  8  9  10 11 (12 13)
                  |________________|                dropped
                                 |_________________|
```
"SAME" = with zero padding:
```
pad|                                      |pad
   inputs:      0 |1  2  3  4  5  6  7  8  9  10 11 12 13|0  0
               |________________|
                              |_________________|
                                             |________________|
```
In this example: Input width = 13 Filter width = 6 Stride = 5 Notes: "VALID" only ever drops the right-most columns (or bottom-most rows). "SAME" tries to pad evenly left and right, but if the amount of columns to be added is odd, it will add the extra column to the right, as is the case in this example (the same logic applies vertically: there may be an extra row of zeros at the bottom). Edit: About the name: With "SAME" padding, if you use a stride of 1, the layer's outputs will have the same spatial dimensions as its inputs. With "VALID" padding, there's no "made-up" padding inputs. The layer only uses valid input data.


Question: What's the difference of name scope and a variable scope in tensorflow? What's the differences between these functions? tf.variable_op_scope(values, name, default_name, initializer=None) Returns a context manager for defining an op that creates variables. This context manager validates that the given values are from the same graph, ensures that that graph is the default graph, and pushes a name scope and a variable scope. tf.op_scope(values, name, default_name=None) Returns a context manager for use when defining a Python op. This context manager validates that the given values are from the same graph, ensures that that graph is the default graph, and pushes a name scope. tf.name_scope(name) Wrapper for Graph.name_scope() using the default graph. See Graph.name_scope() for more details. tf.variable_scope(name_or_scope, reuse=None, initializer=None) Returns a context for variable scope. Variable scope allows to create new variables and to share already created ones while providing checks to not create or share by accident. For details, see the Variable Scope How To, here we present only a few basic examples.

Response:
Let's begin by a short introduction to variable sharing. It is a mechanism in TensorFlow that allows for sharing variables accessed in different parts of the code without passing references to the variable around. The method tf.get_variable can be used with the name of the variable as the argument to either create a new variable with such name or retrieve the one that was created before. This is different from using the tf.Variable constructor which will create a new variable every time it is called (and potentially add a suffix to the variable name if a variable with such name already exists). It is for the purpose of the variable sharing mechanism that a separate type of scope (variable scope) was introduced. As a result, we end up having two different types of scopes: name scope, created using tf.name_scope variable scope, created using tf.variable_scope Both scopes have the same effect on all operations as well as variables created using tf.Variable, i.e., the scope will be added as a prefix to the operation or variable name. However, name scope is ignored by tf.get_variable. We can see that in the following example:
```python
with tf.name_scope("my_scope"):
    v1 = tf.get_variable("var1", [1], dtype=tf.float32)
    v2 = tf.Variable(1, name="var2", dtype=tf.float32)
    a = tf.add(v1, v2)

print(v1.name)  # var1:0
print(v2.name)  # my_scope/var2:0
print(a.name)   # my_scope/Add:0
```
The only way to place a variable accessed using tf.get_variable in a scope is to use a variable scope, as in the following example:
```python
with tf.variable_scope("my_scope"):
    v1 = tf.get_variable("var1", [1], dtype=tf.float32)
    v2 = tf.Variable(1, name="var2", dtype=tf.float32)
    a = tf.add(v1, v2)

print(v1.name)  # my_scope/var1:0
print(v2.name)  # my_scope/var2:0
print(a.name)   # my_scope/Add:0
```
This allows us to easily share variables across different parts of the program, even within different name scopes:
```python
with tf.name_scope("foo"):
    with tf.variable_scope("var_scope"):
        v = tf.get_variable("var", [1])
with tf.name_scope("bar"):
    with tf.variable_scope("var_scope", reuse=True):
        v1 = tf.get_variable("var", [1])
assert v1 == v
print(v.name)   # var_scope/var:0
print(v1.name)  # var_scope/var:0
```
UPDATE As of version r0.11, op_scope and variable_op_scope are both deprecated and replaced by name_scope and variable_scope.


Question:
Keras, How to get the output of each layer? I have trained a binary classification model with CNN, and here is my code
```
model = Sequential()
model.add(Convolution2D(nb_filters, kernel_size[0], kernel_size[1],
                        border_mode='valid',
                        input_shape=input_shape))
model.add(Activation('relu'))
model.add(Convolution2D(nb_filters, kernel_size[0], kernel_size[1]))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=pool_size))
# (16, 16, 32)
model.add(Convolution2D(nb_filters*2, kernel_size[0], kernel_size[1]))
model.add(Activation('relu'))
model.add(Convolution2D(nb_filters*2, kernel_size[0], kernel_size[1]))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=pool_size))
# (8, 8, 64) = (2048)
model.add(Flatten())
model.add(Dense(1024))
model.add(Activation('relu'))
model.add(Dropout(0.5))
model.add(Dense(2))  # define a binary classification problem
model.add(Activation('softmax'))

model.compile(loss='categorical_crossentropy',
              optimizer='adadelta',
              metrics=['accuracy'])
model.fit(x_train, y_train,
          batch_size=batch_size,
          nb_epoch=nb_epoch,
          verbose=1,
          validation_data=(x_test, y_test))
```
And here, I wanna get the output of each layer just like TensorFlow, how can I do that?

Response:
You can easily get the outputs of any layer by using: model.layers[index].output For all layers use this:
```
from keras import backend as K

inp = model.input                                           # input placeholder
outputs = [layer.output for layer in model.layers]          # all layer outputs
functors = [K.function([inp, K.learning_phase()], [out]) for out in outputs]    # evaluation functions

# Testing
test = np.random.random(input_shape)[np.newaxis,...]
layer_outs = [func([test, 1.]) for func in functors]
print layer_outs
```
Note: To simulate Dropout use learning_phase as 1. in layer_outs otherwise use 0. Edit: (based on comments) K.function creates theano/tensorflow tensor functions which is later used to get the output from the symbolic graph given the input. Now K.learning_phase() is required as an input as many Keras layers like Dropout/Batchnomalization depend on it to change behavior during training and test time. So if you remove the dropout layer in your code you can simply use:
```
from keras import backend as K

inp = model.input                                           # input placeholder
outputs = [layer.output for layer in model.layers]          # all layer outputs
functors = [K.function([inp], [out]) for out in outputs]    # evaluation functions

# Testing
test = np.random.random(input_shape)[np.newaxis,...]
layer_outs = [func([test]) for func in functors]
print layer_outs
```
Edit 2: More optimized I just realized that the previous answer is not that optimized as for each function evaluation the data will be transferred CPU->GPU memory and also the tensor calculations needs to be done for the lower layers over-n-over. Instead this is a much better way as you don't need multiple functions but a single function giving you the list of all outputs:
```
from keras import backend as K

inp = model.input                                           # input placeholder
outputs = [layer.output for layer in model.layers]          # all layer outputs
functor = K.function([inp, K.learning_phase()], outputs )   # evaluation function

# Testing
test = np.random.random(input_shape)[np.newaxis,...]
layer_outs = functor([test, 1.])
print layer_outs
```


Question:
What is the role of "Flatten" in Keras? I am trying to understand the role of the Flatten function in Keras. Below is my code, which is a simple two-layer network. It takes in 2-dimensional data of shape (3, 2), and outputs 1-dimensional data of shape (1, 4):
```
model = Sequential()
model.add(Dense(16, input_shape=(3, 2)))
model.add(Activation('relu'))
model.add(Flatten())
model.add(Dense(4))
model.compile(loss='mean_squared_error', optimizer='SGD')

x = np.array([[[1, 2], [3, 4], [5, 6]]])

y = model.predict(x)

print y.shape
```
This prints out that y has shape (1, 4). However, if I remove the Flatten line, then it prints out that y has shape (1, 3, 4). I don't understand this. From my understanding of neural networks, the model.add(Dense(16, input_shape=(3, 2))) function is creating a hidden fully-connected layer, with 16 nodes. Each of these nodes is connected to each of the 3x2 input elements. Therefore, the 16 nodes at the output of this first layer are already "flat". So, the output shape of the first layer should be (1, 16). Then, the second layer takes this as an input, and outputs data of shape (1, 4). So if the output of the first layer is already "flat" and of shape (1, 16), why do I need to further flatten it?

Response:
If you read the Keras documentation entry for Dense, you will see that this call:
```
Dense(16, input_shape=(5,3))
```
would result in a Dense network with 3 inputs and 16 outputs which would be applied independently for each of 5 steps. So, if D(x) transforms 3 dimensional vector to 16-d vector, what you'll get as output from your layer would be a sequence of vectors: [D(x[0,:]), D(x[1,:]),..., D(x[4,:])] with shape (5, 16). In order to have the behavior you specify you may first Flatten your input to a 15-d vector and then apply Dense:
```
model = Sequential()
model.add(Flatten(input_shape=(3, 2)))
model.add(Dense(16))
model.add(Activation('relu'))
model.add(Dense(4))
model.compile(loss='mean_squared_error', optimizer='SGD')
```
EDIT: As some people struggled to understand - here you have an explaining image:


Question:
Understanding TensorBoard (weight) histograms It is really straightforward to see and understand the scalar values in TensorBoard. However, it's not clear how to understand histogram graphs. For example, they are the histograms of my network weights. (After fixing a bug thanks to sunside) What is the best way to interpret these? Layer 1 weights look mostly flat, what does this mean? I added the network construction code here.
```
X = tf.placeholder(tf.float32, [None, input_size], name="input_x")
x_image = tf.reshape(X, [-1, 6, 10, 1])
tf.summary.image('input', x_image, 4)

# First layer of weights
with tf.name_scope("layer1"):
    W1 = tf.get_variable("W1", shape=[input_size, hidden_layer_neurons],
                         initializer=tf.contrib.layers.xavier_initializer())
    layer1 = tf.matmul(X, W1)
    layer1_act = tf.nn.tanh(layer1)
    tf.summary.histogram("weights", W1)
    tf.summary.histogram("layer", layer1)
    tf.summary.histogram("activations", layer1_act)

# Second layer of weights
with tf.name_scope("layer2"):
    W2 = tf.get_variable("W2", shape=[hidden_layer_neurons, hidden_layer_neurons],
                         initializer=tf.contrib.layers.xavier_initializer())
    layer2 = tf.matmul(layer1_act, W2)
    layer2_act = tf.nn.tanh(layer2)
    tf.summary.histogram("weights", W2)
    tf.summary.histogram("layer", layer2)
    tf.summary.histogram("activations", layer2_act)

# Third layer of weights
with tf.name_scope("layer3"):
    W3 = tf.get_variable("W3", shape=[hidden_layer_neurons, hidden_layer_neurons],
                         initializer=tf.contrib.layers.xavier_initializer())
    layer3 = tf.matmul(layer2_act, W3)
    layer3_act = tf.nn.tanh(layer3)

    tf.summary.histogram("weights", W3)
    tf.summary.histogram("layer", layer3)
    tf.summary.histogram("activations", layer3_act)

# Fourth layer of weights
with tf.name_scope("layer4"):
    W4 = tf.get_variable("W4", shape=[hidden_layer_neurons, output_size],
                         initializer=tf.contrib.layers.xavier_initializer())
    Qpred = tf.nn.softmax(tf.matmul(layer3_act, W4)) # Bug fixed: Qpred = tf.nn.softmax(tf.matmul(layer3, W4))
    tf.summary.histogram("weights", W4)
    tf.summary.histogram("Qpred", Qpred)

# We need to define the parts of the network needed for learning a policy
Y = tf.placeholder(tf.float32, [None, output_size], name="input_y")
advantages = tf.placeholder(tf.float32, name="reward_signal")

# Loss function
# Sum (Ai*logp(yi|xi))
log_lik = -Y * tf.log(Qpred)
loss = tf.reduce_mean(tf.reduce_sum(log_lik * advantages, axis=1))
tf.summary.scalar("Q", tf.reduce_mean(Qpred))
tf.summary.scalar("Y", tf.reduce_mean(Y))
tf.summary.scalar("log_likelihood", tf.reduce_mean(log_lik))
tf.summary.scalar("loss", loss)

# Learning
train = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(loss)
```

Response: It appears that the network hasn't learned anything in the layers one to three. The last layer does change, so that means that there either may be something wrong with the gradients (if you're tampering with them manually), you're constraining learning to the last layer by optimizing only its weights or the last layer really 'eats up' all error. It could also be that only biases are learned. The network appears to learn something though, but it might not be using its full potential. More context would be needed here, but playing around with the learning rate (e.g. using a smaller one) might be worth a shot. In general, histograms display the number of occurrences of a value relative to each other values. Simply speaking, if the possible values are in a range of 0..9 and you see a spike of amount 10 on the value 0, this means that 10 inputs assume the value 0; in contrast, if the histogram shows a plateau of 1 for all values of 0..9, it means that for 10 inputs, each possible value 0..9 occurs exactly once. You can also use histograms to visualize probability distributions when you normalize all histogram values by their total sum; if you do that, you'll intuitively obtain the likelihood with which a certain value (on the x axis) will appear (compared to other inputs). Now for layer1/weights, the plateau means that: most of the weights are in the range of -0.15 to 0.15 it is (mostly) equally likely for a weight to have any of these values, i.e. they are (almost) uniformly distributed Said differently, almost the same number of weights have the values -0.15, 0.0, 0.15 and everything in between. There are some weights having slightly smaller or higher values. So in short, this simply looks like the weights have been initialized using a uniform distribution with zero mean and value range -0.15..0.15 ... give or take. If you do indeed use uniform initialization, then this is typical when the network has not been trained yet. In comparison, layer1/activations forms a bell curve (gaussian)-like shape: The values are centered around a specific value, in this case 0, but they may also be greater or smaller than that (equally likely so, since it's symmetric). Most values appear close around the mean of 0, but values do range from -0.8 to 0.8. I assume that the layer1/activations is taken as the distribution over all layer outputs in a batch. You can see that the values do change over time. The layer 4 histogram doesn't tell me anything specific. From the shape, it's just showing that some weight values around -0.1, 0.05 and 0.25 tend to be occur with a higher probability; a reason could be, that different parts of each neuron there actually pick up the same information and are basically redundant. This can mean that you could actually use a smaller network or that your network has the potential to learn more distinguishing features in order to prevent overfitting. These are just assumptions though. Also, as already stated in the comments below, do add bias units. By leaving them out, you are forcefully constraining your network to a possibly invalid solution.


Question:
What does tf.nn.embedding_lookup function do?
```
tf.nn.embedding_lookup(params, ids, partition_strategy='mod', name=None)
```
I cannot understand the duty of this function. Is it like a lookup table? Which means to return the parameters corresponding to each id (in ids)? For instance, in the skip-gram model if we use tf.nn.embedding_lookup(embeddings, train_inputs), then for each train_input it finds the correspond embedding?

Response:
Yes, this function is hard to understand, until you get the point. In its simplest form, it is similar to tf.gather. It returns the elements of params according to the indexes specified by ids. For example (assuming you are inside tf.InteractiveSession())
```
params = tf.constant([10,20,30,40])
ids = tf.constant([0,1,2,3])
print tf.nn.embedding_lookup(params,ids).eval()
```
would return [10 20 30 40], because the first element (index 0) of params is 10, the second element of params (index 1) is 20, etc. Similarly,
```
params = tf.constant([10,20,30,40])
ids = tf.constant([1,1,3])
print tf.nn.embedding_lookup(params,ids).eval()
```
would return [20 20 40]. But embedding_lookup is more than that. The params argument can be a list of tensors, rather than a single tensor.
```
params1 = tf.constant([1,2])
params2 = tf.constant([10,20])
ids = tf.constant([2,0,2,1,2,3])
result = tf.nn.embedding_lookup([params1, params2], ids)
```
In such a case, the indexes, specified in ids, correspond to elements of tensors according to a partition strategy, where the default partition strategy is 'mod'. In the 'mod' strategy, index 0 corresponds to the first element of the first tensor in the list. Index 1 corresponds to the first element of the second tensor. Index 2 corresponds to the first element of the third tensor, and so on. Simply index i corresponds to the first element of the (i+1)th tensor , for all the indexes 0..(n-1), assuming params is a list of n tensors. Now, index n cannot correspond to tensor n+1, because the list params contains only n tensors. So index n corresponds to the second element of the first tensor. Similarly, index n+1 corresponds to the second element of the second tensor, etc. So, in the code
```
params1 = tf.constant([1,2])
params2 = tf.constant([10,20])
ids = tf.constant([2,0,2,1,2,3])
result = tf.nn.embedding_lookup([params1, params2], ids)
```
index 0 corresponds to the first element of the first tensor: 1 index 1 corresponds to the first element of the second tensor: 10 index 2 corresponds to the second element of the first tensor: 2 index 3 corresponds to the second element of the second tensor: 20 Thus, the result would be:
```
[ 2  1  2 10  2 20]
```


Question:
What does tf.nn.conv2d do in tensorflow? I was looking at the docs of tensorflow about tf.nn.conv2d here. But I can't understand what it does or what it is trying to achieve. It says on the docs, #1 : Flattens the filter to a 2-D matrix with shape [filter_height * filter_width * in_channels, output_channels]. Now what does that do? Is that element-wise multiplication or just plain matrix multiplication? I also could not understand the other two points mentioned in the docs. I have written them below : # 2: Extracts image patches from the the input tensor to form a virtual tensor of shape [batch, out_height, out_width, filter_height * filter_width * in_channels]. # 3: For each patch, right-multiplies the filter matrix and the image patch vector. It would be really helpful if anyone could give an example, a piece of code (extremely helpful) maybe and explain what is going on there and why the operation is like this. I've tried coding a small portion and printing out the shape of the operation. Still, I can't understand. I tried something like this:
```python
op = tf.shape(tf.nn.conv2d(tf.random_normal([1,10,10,10]),
              tf.random_normal([2,10,10,10]),
              strides=[1, 2, 2, 1], padding='SAME'))

with tf.Session() as sess:
    result = sess.run(op)
    print(result)
```
I understand bits and pieces of convolutional neural networks. I studied them here. But the implementation on tensorflow is not what I expected. So it raised the question. EDIT: So, I implemented a much simpler code. But I can't figure out what's going on. I mean how the results are like this. It would be extremely helpful if anyone could tell me what process yields this output.
```python
input = tf.Variable(tf.random_normal([1,2,2,1]))
filter = tf.Variable(tf.random_normal([1,1,1,1]))

op = tf.nn.conv2d(input, filter, strides=[1, 1, 1, 1], padding='SAME')
init = tf.initialize_all_variables()
with tf.Session() as sess:
    sess.run(init)

    print("input")
    print(input.eval())
    print("filter")
    print(filter.eval())
    print("result")
    result = sess.run(op)
    print(result)
```
output
```
input
[[[[ 1.60314465]
   [-0.55022103]]

  [[ 0.00595062]
   [-0.69889867]]]]
filter
[[[[-0.59594476]]]]
result
[[[[-0.95538563]
   [ 0.32790133]]

  [[-0.00354624]
   [ 0.41650501]]]]
```

Response:
Ok I think this is about the simplest way to explain it all. Your example is 1 image, size 2x2, with 1 channel. You have 1 filter, with size 1x1, and 1 channel (size is height x width x channels x number of filters). For this simple case the resulting 2x2, 1 channel image (size 1x2x2x1, number of images x height x width x x channels) is the result of multiplying the filter value by each pixel of the image. Now let's try more channels:
```py
input = tf.Variable(tf.random_normal([1,3,3,5]))
filter = tf.Variable(tf.random_normal([1,1,5,1]))

op = tf.nn.conv2d(input, filter, strides=[1, 1, 1, 1], padding='VALID')
```
Here the 3x3 image and the 1x1 filter each have 5 channels. The resulting image will be 3x3 with 1 channel (size 1x3x3x1), where the value of each pixel is the dot product across channels of the filter with the corresponding pixel in the input image. Now with a 3x3 filter
```py
input = tf.Variable(tf.random_normal([1,3,3,5]))
filter = tf.Variable(tf.random_normal([3,3,5,1]))

op = tf.nn.conv2d(input, filter, strides=[1, 1, 1, 1], padding='VALID')
```
Here we get a 1x1 image, with 1 channel (size 1x1x1x1). The value is the sum of the 9, 5-element dot products. But you could just call this a 45-element dot product. Now with a bigger image
```py
input = tf.Variable(tf.random_normal([1,5,5,5]))
filter = tf.Variable(tf.random_normal([3,3,5,1]))

op = tf.nn.conv2d(input, filter, strides=[1, 1, 1, 1], padding='VALID')
```
The output is a 3x3 1-channel image (size 1x3x3x1). Each of these values is a sum of 9, 5-element dot products. Each output is made by centering the filter on one of the 9 center pixels of the input image, so that none of the filter sticks out. The xs below represent the filter centers for each output pixel.
```
.....
.xxx.
.xxx.
.xxx.
.....
```
Now with "SAME" padding:
```py
input = tf.Variable(tf.random_normal([1,5,5,5]))
filter = tf.Variable(tf.random_normal([3,3,5,1]))

op = tf.nn.conv2d(input, filter, strides=[1, 1, 1, 1], padding='SAME')
```
This gives a 5x5 output image (size 1x5x5x1). This is done by centering the filter at each position on the image. Any of the 5-element dot products where the filter sticks out past the edge of the image get a value of zero. So the corners are only sums of 4, 5-element dot products. Now with multiple filters.
```py
input = tf.Variable(tf.random_normal([1,5,5,5]))
filter = tf.Variable(tf.random_normal([3,3,5,7]))

op = tf.nn.conv2d(input, filter, strides=[1, 1, 1, 1], padding='SAME')
```
This still gives a 5x5 output image, but with 7 channels (size 1x5x5x7). Where each channel is produced by one of the filters in the set. Now with strides 2,2:
```py
input = tf.Variable(tf.random_normal([1,5,5,5]))
filter = tf.Variable(tf.random_normal([3,3,5,7]))

op = tf.nn.conv2d(input, filter, strides=[1, 2, 2, 1], padding='SAME')
```
Now the result still has 7 channels, but is only 3x3 (size 1x3x3x7). This is because instead of centering the filters at every point on the image, the filters are centered at every other point on the image, taking steps (strides) of width 2. The x's below represent the filter center for each output pixel, on the input image.
```
x.x.x
.....
x.x.x
.....
x.x.x
```
And of course the first dimension of the input is the number of images so you can apply it over a batch of 10 images, for example:
```py
input = tf.Variable(tf.random_normal([10,5,5,5]))
filter = tf.Variable(tf.random_normal([3,3,5,7]))

op = tf.nn.conv2d(input, filter, strides=[1, 2, 2, 1], padding='SAME')
```
This performs the same operation, for each image independently, giving a stack of 10 images as the result (size 10x3x3x7)


Question:
How are the new tf.contrib.summary summaries in TensorFlow evaluated? I'm having a bit of trouble understanding the new tf.contrib.summary API. In the old one, it seemed that all one was supposed to do was to run tf.summary.merge_all() and run that as an op. But now we have things like tf.contrib.summary.record_summaries_every_n_global_steps, which can be used like this:
```
import tensorflow.contrib.summary as tfsum

summary_writer = tfsum.create_file_writer(logdir, flush_millis=3000)
summaries = []

# First we create one summary which runs every n global steps
with summary_writer.as_default(), tfsum.record_summaries_every_n_global_steps(30):
    summaries.append(tfsum.scalar("train/loss", loss))

# And then one that runs every single time?
with summary_writer.as_default(), tfsum.always_record_summaries():
    summaries.append(tfsum.scalar("train/accuracy", accuracy))

# Then create an optimizer which uses a global step
step = tf.create_global_step()
train = tf.train.AdamOptimizer().minimize(loss, global_step=step)
```
And now come a few questions: If we just run session.run(summaries) in a loop, I assume that the accuracy summary would get written every single time, while the loss one wouldn't, because it only gets written if the global step is divisible by 30? Assuming the summaries automatically evaluate their dependencies, I never need to run session.run([accuracy, summaries]) but can just run, session.run(summaries) since they have a dependency in the graph, right? If 2) is true, can't I just add a control dependency to the training step so that the summaries are written on every train run? Or is this a bad practice? Is there any downside to using control dependencies in general for things that are going to be evaluated at the same time anyway? Why does tf.contrib.summary.scalar (and others) take in a step parameter? By adding a control dependency in 3) I mean doing this:
```
tf.control_dependencies(summaries):
    train = tf.train.AdamOptimizer().minimize(loss, global_step=step)
```

Response: answer moved from edit to self-answer as requested I just played around with this a little bit, and it seems that if one combines tf.control_dependencies with tf.record_summaries_every_n_global_steps it behaves as expected and the summary only gets recorded every nth step. But if they are run together within a session, such as session.run([train, summs]), the summaries are stored every once in a while, but not exactly every nth step. I tested this with n=2 and with the second approach the summary was often written at odd steps, while with the control dependency approach it was always on an even step.


Question: Should we do learning rate decay for adam optimizer I'm training a network for image localization with Adam optimizer, and someone suggest me to use exponential decay. I don't want to try that because Adam optimizer itself decays learning rate. But that guy insists and he said he did that before. So should I do that and is there any theory behind your suggestion?

Response: It depends. ADAM updates any parameter with an individual learning rate. This means that every parameter in the network has a specific learning rate associated. But the single learning rate for each parameter is computed using lambda (the initial learning rate) as an upper limit. This means that every single learning rate can vary from 0 (no update) to lambda (maximum update). It's true, that the learning rates adapt themselves during training steps, but if you want to be sure that every update step doesn't exceed lambda you can than lower lambda using exponential decay or whatever. It can help to reduce loss during the latest step of training, when the computed loss with the previously associated lambda parameter has stopped to decrease.


Question: Does model.compile() initialize all the weights and biases in Keras (tensorflow backend)? When I start training a model, there is no model saved previously. I can use model.compile() safely. I have now saved the model in a h5 file for further training using checkpoint. Say, I want to train the model further. I am confused at this point: can I use model.compile() here? And should it be placed before or after the model = load_model() statement? If model.compile() reinitializes all the weights and biases, I should place it before model = load_model() statement. After discovering some discussions, it seems to me that model.compile() is only needed when I have no model saved previously. Once I have saved the model, there is no need to use model.compile(). Is it true or false? And when I want to predict using the trained model, should I use model.compile() before predicting?

Response: When to use? If you're using compile, surely it must be after load_model(). After all, you need a model to compile. (PS: load_model automatically compiles the model with the optimizer that was saved along with the model) What does compile do? Compile defines the loss function, the optimizer and the metrics. That's all. It has nothing to do with the weights and you can compile a model as many times as you want without causing any problem to pretrained weights. You need a compiled model to train (because training uses the loss function and the optimizer). But it's not necessary to compile a model for predicting. Do you need to use compile more than once? Only if: You want to change one of these: Loss function Optimizer / Learning rate Metrics The trainable property of some layer You loaded (or created) a model that is not compiled yet. Or your load/save method didn't consider the previous compilation. Consequences of compiling again: If you compile a model again, you will lose the optimizer states. This means that your training will suffer a little at the beginning until it adjusts the learning rate, the momentums, etc. But there is absolutely no damage to the weights (unless, of course, your initial learning rate is so big that the first training step wildly changes the fine tuned weights).


Question: What is the difference between sparse_categorical_crossentropy and categorical_crossentropy? What is the difference between sparse_categorical_crossentropy and categorical_crossentropy? When should one loss be used as opposed to the other? For example, are these losses suitable for linear regression?

Response: Simply: categorical_crossentropy (cce) produces a one-hot array containing the probable match for each category, sparse_categorical_crossentropy (scce) produces a category index of the most likely matching category. Consider a classification problem with 5 categories (or classes). In the case of cce, the one-hot target may be [0, 1, 0, 0, 0] and the model may predict [.2, .5, .1, .1, .1] (probably right) In the case of scce, the target index may be [1] and the model may predict: [.5]. Consider now a classification problem with 3 classes. In the case of cce, the one-hot target might be [0, 0, 1] and the model may predict [.5, .1, .4] (probably inaccurate, given that it gives more probability to the first class) In the case of scce, the target index might be [0], and the model may predict [.5] Many categorical models produce scce output because you save space, but lose A LOT of information (for example, in the 2nd example, index 2 was also very close.) I generally prefer cce output for model reliability. There are a number of situations to use scce, including: when your classes are mutually exclusive, i.e. you don't care at all about other close-enough predictions, the number of categories is large to the prediction output becomes overwhelming. 220405: response to "one-hot encoding" comments: one-hot encoding is used for a category feature INPUT to select a specific category (e.g. male versus female). This encoding allows the model to train more efficiently: training weight is a product of category, which is 0 for all categories except for the given one. cce and scce are a model OUTPUT. cce is a probability array of each category, totally 1.0. scce shows the MOST LIKELY category, totally 1.0. scce is technically a one-hot array, just like a hammer used as a door stop is still a hammer, but its purpose is different. cce is NOT one-hot.


Question:
How to set adaptive learning rate for GradientDescentOptimizer? I am using TensorFlow to train a neural network. This is how I am initializing the GradientDescentOptimizer:
```
init = tf.initialize_all_variables()
sess = tf.Session()
sess.run(init)

mse        = tf.reduce_mean(tf.square(out - out_))
train_step = tf.train.GradientDescentOptimizer(0.3).minimize(mse)
```
The thing here is that I don't know how to set an update rule for the learning rate or a decay value for that. How can I use an adaptive learning rate here?

Response:
First of all, tf.train.GradientDescentOptimizer is designed to use a constant learning rate for all variables in all steps. TensorFlow also provides out-of-the-box adaptive optimizers including the tf.train.AdagradOptimizer and the tf.train.AdamOptimizer, and these can be used as drop-in replacements. However, if you want to control the learning rate with otherwise-vanilla gradient descent, you can take advantage of the fact that the learning_rate argument to the tf.train.GradientDescentOptimizer constructor can be a Tensor object. This allows you to compute a different value for the learning rate in each step, for example:
```
learning_rate = tf.placeholder(tf.float32, shape=[])
# ...
train_step = tf.train.GradientDescentOptimizer(
    learning_rate=learning_rate).minimize(mse)

sess = tf.Session()

# Feed different values for learning rate to each training step.
sess.run(train_step, feed_dict={learning_rate: 0.1})
sess.run(train_step, feed_dict={learning_rate: 0.1})
sess.run(train_step, feed_dict={learning_rate: 0.01})
sess.run(train_step, feed_dict={learning_rate: 0.01})
```
Alternatively, you could create a scalar tf.Variable that holds the learning rate, and assign it each time you want to change the learning rate.


Question:
AttributeError: 'Tensor' object has no attribute 'numpy' I downloaded this code from GitHub.
```
predicted_id = tf.multinomial(tf.exp(predictions), num_samples=1)[0][0].numpy()
```
But I get an error that says:
```
AttributeError: 'Tensor' object has no attribute 'numpy'
```
What is wrong, and how do I fix it?

Response:
Since the accepted answer did not solve the problem for me so I thought it might be helpful for some people who face the problem and that already have tensorflow version >= 2.2.0 and eager execution enabled. The issue seems to be that for certain functions during the fitting model.fit() the @tf.function decorator prohibits the execution of functions like tensor.numpy() for performance reasons. The solution for me was to pass the flag run_eagerly=True to the model.compile() like this:
```
model.compile(..., run_eagerly=True)
```


Question:
What does global_step mean in Tensorflow? In this is tutorial code from TensorFlow website, could anyone help explain what does global_step mean? I found on the Tensorflow website written that global step is used count training steps, but I don't quite get what exactly it means. Also, what does the number 0 mean when setting up global_step?
```py
def training(loss,learning_rate):
        tf.summary.scalar('loss',loss)
        optimizer = tf.train.GradientDescentOptimizer(learning_rate)

        # Why 0 as the first parameter of the global_step tf.Variable?
        global_step = tf.Variable(0, name='global_step',trainable=False)

        train_op = optimizer.minimize(loss, global_step=global_step)

        return train_op
```
According to Tensorflow doc global_step: increment by one after the variables have been updated. Does that mean after one update global_step becomes 1?

Response: global_step refers to the number of batches seen by the graph. Every time a batch is provided, the weights are updated in the direction that minimizes the loss. global_step just keeps track of the number of batches seen so far. When it is passed in the minimize() argument list, the variable is increased by one. Have a look at optimizer.minimize(). You can get the global_step value using tf.train.global_step(). Also handy are the utility methods tf.train.get_global_step or tf.train.get_or_create_global_step. 0 is the initial value of the global step in this context.


Question:
Using a pre-trained word embedding (word2vec or Glove) in TensorFlow I've recently reviewed an interesting implementation for convolutional text classification. However all TensorFlow code I've reviewed uses a random (not pre-trained) embedding vectors like the following:
```
with tf.device('/cpu:0'), tf.name_scope("embedding"):
    W = tf.Variable(
        tf.random_uniform([vocab_size, embedding_size], -1.0, 1.0),
        name="W")
    self.embedded_chars = tf.nn.embedding_lookup(W, self.input_x)
    self.embedded_chars_expanded = tf.expand_dims(self.embedded_chars, -1)
```
Does anybody know how to use the results of Word2vec or a GloVe pre-trained word embedding instead of a random one?

Response:
There are a few ways that you can use a pre-trained embedding in TensorFlow. Let's say that you have the embedding in a NumPy array called embedding, with vocab_size rows and embedding_dim columns and you want to create a tensor W that can be used in a call to tf.nn.embedding_lookup(). Simply create W as a tf.constant() that takes embedding as its value:
```
W = tf.constant(embedding, name="W")
```
This is the easiest approach, but it is not memory efficient because the value of a tf.constant() is stored multiple times in memory. Since embedding can be very large, you should only use this approach for toy examples. Create W as a tf.Variable and initialize it from the NumPy array via a tf.placeholder():
```
W = tf.Variable(tf.constant(0.0, shape=[vocab_size, embedding_dim]),
                trainable=False, name="W")

embedding_placeholder = tf.placeholder(tf.float32, [vocab_size, embedding_dim])
embedding_init = W.assign(embedding_placeholder)

# ...
sess = tf.Session()

sess.run(embedding_init, feed_dict={embedding_placeholder: embedding})
```
This avoid storing a copy of embedding in the graph, but it does require enough memory to keep two copies of the matrix in memory at once (one for the NumPy array, and one for the tf.Variable). Note that I've assumed that you want to hold the embedding matrix constant during training, so W is created with trainable=False. If the embedding was trained as part of another TensorFlow model, you can use a tf.train.Saver to load the value from the other model's checkpoint file. This means that the embedding matrix can bypass Python altogether. Create W as in option 2, then do the following:
```
W = tf.Variable(...)

embedding_saver = tf.train.Saver({"name_of_variable_in_other_model": W})

# ...
sess = tf.Session()
embedding_saver.restore(sess, "checkpoint_filename.ckpt")
```


Question: Can I measure the execution time of individual operations with TensorFlow? I know I can measure the execution time of a call to sess.run(), but is it possible to get a finer granularity and measure the execution time of individual operations?

Response:
I have used the Timeline object to get the time of execution for each node in the graph: you use a classic sess.run() but also specify the optional arguments options and run_metadata you then create a Timeline object with the run_metadata.step_stats data Here is an example program that measures the performance of a matrix multiplication:
```py
import tensorflow as tf
from tensorflow.python.client import timeline

x = tf.random_normal([1000, 1000])
y = tf.random_normal([1000, 1000])
res = tf.matmul(x, y)

# Run the graph with full trace option
with tf.Session() as sess:
    run_options = tf.RunOptions(trace_level=tf.RunOptions.FULL_TRACE)
    run_metadata = tf.RunMetadata()
    sess.run(res, options=run_options, run_metadata=run_metadata)

    # Create the Timeline object, and write it to a json
    tl = timeline.Timeline(run_metadata.step_stats)
    ctf = tl.generate_chrome_trace_format()
    with open('timeline.json', 'w') as f:
        f.write(ctf)
```
You can then open Google Chrome, go to the page chrome://tracing and load the timeline.json file. You should see something like:


Question:
Tensorflow: None of the MLIR optimization passes are enabled (registered 1) I am using a very small model for testing purposes using tensorflow 2.3 and keras. Looking at my terminal, I get the following warning:
```
I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:118] None of the MLIR optimization passes are enabled (registered 1)
```
However, the code works as expected. But what does this message mean? Thanks.

Response: MLIR is being used as another solution to implementing and optimizing Tensorflow logic. This informative message is benign and is saying MLIR was not being used. This is expected as in TF 2.3, the MLIR based implementation is still being developed and proven, so end users are generally not expected to use the MLIR implementation and are instead expected to use the non-MLIR feature complete implementation. Update: still experimental on version 2.9.1. On the docs it is written: DO NOT USE, DEV AND TESTING ONLY AT THE MOMENT.


Question: How to count total number of trainable parameters in a tensorflow model? Is there a function call or another way to count the total number of parameters in a tensorflow model? By parameters I mean: an N dim vector of trainable variables has N parameters, a NxM matrix has N*M parameters, etc. So essentially I'd like to sum the product of the shape dimensions of all the trainable variables in a tensorflow session.

Response:
Loop over the shape of every variable in tf.trainable_variables().
```
total_parameters = 0
for variable in tf.trainable_variables():
    # shape is an array of tf.Dimension
    shape = variable.get_shape()
    print(shape)
    print(len(shape))
    variable_parameters = 1
    for dim in shape:
        print(dim)
        variable_parameters *= dim.value
    print(variable_parameters)
    total_parameters += variable_parameters
print(total_parameters)
```
Update: I wrote an article to clarify the dynamic/static shapes in Tensorflow because of this answer: https://pgaleone.eu/tensorflow/2018/07/28/understanding-tensorflow-tensors-shape-static-dynamic/


Question: How to "reset" tensorboard data after killing tensorflow instance I'm testing different hyperparameters for a cnn model I built, but I'm having a small annoyance when viewing the summaries in Tensorboard. The problem seems to be that the data is just "added" in consecutive runs, so the functions result in a weird superposition unless I see the information as "relative" instead of "by step". See here: I've tried killing tensorboard's process and erasing the log files, but it seems it is not enough. So the question is, how do I reset this information? Thanks!!

Response:
Note: The solution you've posted (erase TensorBoard's log files and kill the process) will work, but it isn't preferred, because it destroys historical information about your training. Instead, you can have each new training job write to a new subdirectory (of your top-level log directory). Then, TensorBoard will consider each job a new "run" and will create a nice comparison view so you can see how the training differed between iterations of your model. In the following an example from https://www.tensorflow.org/tensorboard/get_started:
```
model = create_model()
...
model.compile(...)

log_dir = "logs/fit/" + datetime.datetime.now().strftime("%Y%m%d-%H%M%S")
tensorboard_callback = tf.keras.callbacks.TensorBoard(log_dir=log_dir, histogram_freq=1)

model.fit(..., callbacks=[tensorboard_callback])
```


Question: Tensorflow One Hot Encoder? Does tensorflow have something similar to scikit learn's one hot encoder for processing categorical data? Would using a placeholder of tf.string behave as categorical data? I realize I can manually pre-process the data before sending it to tensorflow, but having it built in is very convenient.

Response:
As of TensorFlow 0.8, there is now a native one-hot op, tf.one_hot that can convert a set of sparse labels to a dense one-hot representation. This is in addition to tf.nn.sparse_softmax_cross_entropy_with_logits, which can in some cases let you compute the cross entropy directly on the sparse labels instead of converting them to one-hot. Previous answer, in case you want to do it the old way: @Salvador's answer is correct - there (used to be) no native op to do it. Instead of doing it in numpy, though, you can do it natively in tensorflow using the sparse-to-dense operators:
```
num_labels = 10

# label_batch is a tensor of numeric labels to process
# 0 <= label < num_labels

sparse_labels = tf.reshape(label_batch, [-1, 1])
derived_size = tf.shape(label_batch)[0]
indices = tf.reshape(tf.range(0, derived_size, 1), [-1, 1])
concated = tf.concat(1, [indices, sparse_labels])
outshape = tf.pack([derived_size, num_labels])
labels = tf.sparse_to_dense(concated, outshape, 1.0, 0.0)
```
The output, labels, is a one-hot matrix of batch_size x num_labels. Note also that as of 2016-02-12 (which I assume will eventually be part of a 0.7 release), TensorFlow also has the tf.nn.sparse_softmax_cross_entropy_with_logits op, which in some cases can let you do training without needing to convert to a one-hot encoding. Edited to add: At the end, you may need to explicitly set the shape of labels. The shape inference doesn't recognize the size of the num_labels component. If you don't need a dynamic batch size with derived_size, this can be simplified. Edited 2016-02-12 to change the assignment of outshape per comment below.


Question:
How to add if condition in a TensorFlow graph? Let's say I have following code:
```
x = tf.placeholder("float32", shape=[None, ins_size**2*3], name = "x_input")
condition = tf.placeholder("int32", shape=[1, 1], name = "condition")
W = tf.Variable(tf.zeros([ins_size**2*3,label_option]), name = "weights")
b = tf.Variable(tf.zeros([label_option]), name = "bias")

if condition > 0:
    y = tf.nn.softmax(tf.matmul(x, W) + b)
else:
    y = tf.nn.softmax(tf.matmul(x, W) - b)
```
Would the if statement work in the calculation (I do not think so)? If not, how can I add an if statement into the TensorFlow calculation graph?

Response:
You're correct that the if statement doesn't work here, because the condition is evaluated at graph construction time, whereas presumably you want the condition to depend on the value fed to the placeholder at runtime. (In fact, it will always take the first branch, because condition > 0 evaluates to a Tensor, which is "truthy" in Python.) To support conditional control flow, TensorFlow provides the tf.cond() operator, which evaluates one of two branches, depending on a boolean condition. To show you how to use it, I'll rewrite your program so that condition is a scalar tf.int32 value for simplicity:
```py
x = tf.placeholder(tf.float32, shape=[None, ins_size**2*3], name="x_input")
condition = tf.placeholder(tf.int32, shape=[], name="condition")
W = tf.Variable(tf.zeros([ins_size**2 * 3, label_option]), name="weights")
b = tf.Variable(tf.zeros([label_option]), name="bias")

y = tf.cond(condition > 0, lambda: tf.matmul(x, W) + b, lambda: tf.matmul(x, W) - b)
```


Question: tf.nn.conv2d vs tf.layers.conv2d Is there any advantage in using tf.nn.* over tf.layers.*? Most of the examples in the doc use tf.nn.conv2d, for instance, but it is not clear why they do so.

Response:
As GBY mentioned, they use the same implementation. There is a slight difference in the parameters. For tf.nn.conv2d:
```
filter: A Tensor. Must have the same type as input. A 4-D tensor of shape [filter_height, filter_width, in_channels, out_channels]
```
For tf.layers.conv2d:
```
filters: Integer, the dimensionality of the output space (i.e. the number of filters in the convolution).
```
I would use tf.nn.conv2d when loading a pretrained model (example code: https://github.com/ry/tensorflow-vgg16), and tf.layers.conv2d for a model trained from scratch.


Question: Tensorflow vs OpenCV [closed] Closed. This question is opinion-based. It is not currently accepting answers. Want to improve this question? Because this question may lead to opinionated discussion, debate, and answers, it has been closed. You may edit the question if you feel you can improve it so that it requires answers that include facts and citations or a detailed explanation of the proposed solution. If edited, the question will be reviewed and might be reopened. Closed 7 years ago. Improve this question I'm new into the AI world, I've start doing some stuff using Python & OpenCV for face detection and so on. I know that with the implementation of some algorithms I can develop AI system using Python & OpenCV. So my question is : What is the position of Tensorflow here? Can I say Tensorflow is an alternative to OpenCV? as I can say Python is an alternative programming language to Java (for example).

Response: The main difference is that TensorFlow is a framework for machine learning, and OpenCV is a library for computer vision. It can be a good start to check the link below to get a grasp for the difference between framework and library: What is the difference between a framework and a library? You can do image recognition with TensorFlow. Though it is suited for more general problems as well, such as: classification, clustering and regression. I guess people downvoted because this question might be more relevant to: https://datascience.stackexchange.com/


Question:
ValueError: Shapes (None, 1) and (None, 2) are incompatible I am training a facial expression (angry vs happy) model. Last dense output layer was previously 1 but when i predict an image it's output was always 1 with 64 % accuracy. So i changed it to 2 for 2 outputs. But now i am getting this error::
```
Epoch 1/15

---------------------------------------------------------------------------

ValueError                                Traceback (most recent call last)

<ipython-input-54-9c7272c38dcb> in <module>()
     11     epochs=epochs,
     12     validation_data = val_data_gen,
---> 13     validation_steps = validation_steps,
     14
     15 )

10 frames

/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/func_graph.py in wrapper(*args, **kwargs)
    966           except Exception as e:  # pylint:disable=broad-except
    967             if hasattr(e, "ag_error_metadata"):
--> 968               raise e.ag_error_metadata.to_exception(e)
    969             else:
    970               raise

ValueError: in user code:

    /usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py:571 train_function  *
        outputs = self.distribute_strategy.run(
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/distribute/distribute_lib.py:951 run  **
        return self._extended.call_for_each_replica(fn, args=args, kwargs=kwargs)
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/distribute/distribute_lib.py:2290 call_for_each_replica
        return self._call_for_each_replica(fn, args, kwargs)
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/distribute/distribute_lib.py:2649 _call_for_each_replica
        return fn(*args, **kwargs)
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py:533 train_step  **
        y, y_pred, sample_weight, regularization_losses=self.losses)
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/compile_utils.py:205 __call__
        loss_value = loss_obj(y_t, y_p, sample_weight=sw)
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/losses.py:143 __call__
        losses = self.call(y_true, y_pred)
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/losses.py:246 call
        return self.fn(y_true, y_pred, **self._fn_kwargs)
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/losses.py:1527 categorical_crossentropy
        return K.categorical_crossentropy(y_true, y_pred, from_logits=from_logits)
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/backend.py:4561 categorical_crossentropy
        target.shape.assert_is_compatible_with(output.shape)
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/tensor_shape.py:1117 assert_is_compatible_with
        raise ValueError("Shapes %s and %s are incompatible" % (self, other))

    ValueError: Shapes (None, 1) and (None, 2) are incompatible
```
The relevant code is :
```
model = Sequential([
    Conv2D(32,3, activation='relu', input_shape=(48,48,1)),
    BatchNormalization(),
    MaxPooling2D(pool_size=(3, 3)),

    Flatten(),
    Dense(512, activation='relu'),
    Dense(2,activation='softmax')
])
model.compile(optimizer='adam',
              loss='categorical_crossentropy',
              metrics=['accuracy'])

model.summary()

Model: "sequential_4"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
conv2d_6 (Conv2D)            (None, 46, 46, 32)        320
_________________________________________________________________
batch_normalization_4 (Batch (None, 46, 46, 32)        128
_________________________________________________________________
max_pooling2d_6 (MaxPooling2 (None, 15, 15, 32)        0
_________________________________________________________________
flatten_4 (Flatten)          (None, 7200)              0
_________________________________________________________________
dense_8 (Dense)              (None, 512)               3686912
_________________________________________________________________
dense_9 (Dense)              (None, 2)                 1026
=================================================================
Total params: 3,688,386
Trainable params: 3,688,322
Non-trainable params: 64
_________________________________________________________________

epochs = 15
steps_per_epoch = train_data_gen.n//train_data_gen.batch_size
validation_steps = val_data_gen.n//val_data_gen.batch_size

history = model.fit(
    x=train_data_gen,
    steps_per_epoch=steps_per_epoch,
    epochs=epochs,
    validation_data = val_data_gen,
    validation_steps = validation_steps,

)
```

Response:
i was facing the same problem my shapes were
```
shape of X (271, 64, 64, 3)
shape of y (271,)
shape of trainX (203, 64, 64, 3)
shape of trainY (203, 1)
shape of testX (68, 64, 64, 3)
shape of testY (68, 1)
```
and
```
loss="categorical_crossentropy"
```
i changed it to
```
loss="sparse_categorical_crossentropy"
```
and it worked like a charm for me


Question:
Negative dimension size caused by subtracting 3 from 1 for 'Conv2D' I'm using Keras with Tensorflow as backend , here is my code:
```
import numpy as np
np.random.seed(1373)
import tensorflow as tf
tf.python.control_flow_ops = tf

import os
from keras.datasets import mnist
from keras.models import Sequential
from keras.layers.core import Dense, Dropout, Activation, Flatten
from keras.layers.convolutional import Convolution2D, MaxPooling2D
from keras.utils import np_utils

batch_size = 128
nb_classes = 10
nb_epoch = 12

img_rows, img_cols = 28, 28

nb_filters = 32

nb_pool = 2

nb_conv = 3

(X_train, y_train), (X_test, y_test) = mnist.load_data()

print(X_train.shape[0])

X_train = X_train.reshape(X_train.shape[0], 1, img_rows, img_cols)
X_test = X_test.reshape(X_test.shape[0], 1, img_rows, img_cols)

X_train = X_train.astype('float32')
X_test = X_test.astype('float32')
X_train /= 255
X_test /= 255

print('X_train shape:', X_train.shape)
print(X_train.shape[0], 'train samples')
print(X_test.shape[0], 'test samples')

Y_train = np_utils.to_categorical(y_train, nb_classes)
Y_test = np_utils.to_categorical(y_test, nb_classes)

model = Sequential()

model.add(Convolution2D(nb_filters, nb_conv, nb_conv,
border_mode='valid',
input_shape=(1, img_rows, img_cols)))
model.add(Activation('relu'))
model.add(Convolution2D(nb_filters, nb_conv, nb_conv))
model.add(Activation('relu'))

model.add(MaxPooling2D(pool_size=(nb_pool, nb_pool)))
model.add(Dropout(0.25))

model.add(Flatten())
model.add(Dense(128))
model.add(Activation('relu'))
model.add(Dropout(0.5))
model.add(Dense(nb_classes))
model.add(Activation('softmax'))

model.compile(loss='categorical_crossentropy', optimizer='adadelta', metrics=["accuracy"])

model.fit(X_train, Y_train, batch_size=batch_size, nb_epoch=nb_epoch,
verbose=1, validation_data=(X_test, Y_test))

score = model.evaluate(X_test, Y_test, verbose=0)

print('Test score:', score[0])
print('Test accuracy:', score[1])
```
and Trackback error:
```
Using TensorFlow backend.
60000
('X_train shape:', (60000, 1, 28, 28))
(60000, 'train samples')
(10000, 'test samples')
Traceback (most recent call last):
  File "mnist.py", line 154, in <module>
    input_shape=(1, img_rows, img_cols)))
  File "/usr/local/lib/python2.7/dist-packages/keras/models.py", line 276, in add
    layer.create_input_layer(batch_input_shape, input_dtype)
  File "/usr/local/lib/python2.7/dist-packages/keras/engine/topology.py", line 370, in create_input_layer
    self(x)
  File "/usr/local/lib/python2.7/dist-packages/keras/engine/topology.py", line 514, in __call__
    self.add_inbound_node(inbound_layers, node_indices, tensor_indices)
  File "/usr/local/lib/python2.7/dist-packages/keras/engine/topology.py", line 572, in add_inbound_node
    Node.create_node(self, inbound_layers, node_indices, tensor_indices)
  File "/usr/local/lib/python2.7/dist-packages/keras/engine/topology.py", line 149, in create_node
    output_tensors = to_list(outbound_layer.call(input_tensors[0], mask=input_masks[0]))
  File "/usr/local/lib/python2.7/dist-packages/keras/layers/convolutional.py", line 466, in call
    filter_shape=self.W_shape)
  File "/usr/local/lib/python2.7/dist-packages/keras/backend/tensorflow_backend.py", line 1579, in conv2d
    x = tf.nn.conv2d(x, kernel, strides, padding=padding)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_nn_ops.py", line 396, in conv2d
    data_format=data_format, name=name)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/op_def_library.py", line 759, in apply_op
    op_def=op_def)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 2242, in create_op
    set_shapes_for_outputs(ret)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1617, in set_shapes_for_outputs
    shapes = shape_func(op)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1568, in call_with_requiring
    return call_cpp_shape_fn(op, require_shape_fn=True)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/common_shapes.py", line 610, in call_cpp_shape_fn
    debug_python_shape_fn, require_shape_fn)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/common_shapes.py", line 675, in _call_cpp_shape_fn_impl
    raise ValueError(err.message)
ValueError: Negative dimension size caused by subtracting 3 from 1 for 'Conv2D' (op: 'Conv2D') with input shapes: [?,1,28,28], [3,3,28,32].
```
First I saw some answers that problem is with Tensorflow version so I upgrade Tensorflow to 0.12.0, but still exist , is that problem with network or I missing something, what should input_shape looks like? Update Here is ./keras/keras.json:
```
{
    "image_dim_ordering": "tf",
    "epsilon": 1e-07,
    "floatx": "float32",
    "backend": "tensorflow"
}
```

Response:
Your issue comes from the image_ordering_dim in keras.json. From Keras Image Processing doc: dim_ordering: One of {"th", "tf"}. "tf" mode means that the images should have shape (samples, height, width, channels), "th" mode means that the images should have shape (samples, channels, height, width). It defaults to the image_dim_ordering value found in your Keras config file at ~/.keras/keras.json. If you never set it, then it will be "tf". Keras maps the convolution operation to the chosen backend (theano or tensorflow). However, both backends have made different choices for the ordering of the dimensions. If your image batch is of N images of HxW size with C channels, theano uses the NCHW ordering while tensorflow uses the NHWC ordering. Keras allows you to choose which ordering you prefer and will do the conversion to map to the backends behind. But if you choose image_ordering_dim="th" it expects Theano-style ordering (NCHW, the one you have in your code) and if image_ordering_dim="tf" it expects tensorflow-style ordering (NHWC). Since your image_ordering_dim is set to "tf", if you reshape your data to the tensorflow style it should work:
```
X_train = X_train.reshape(X_train.shape[0], img_cols, img_rows, 1)
X_test = X_test.reshape(X_test.shape[0], img_cols, img_rows, 1)
```
and
```
input_shape=(img_cols, img_rows, 1)
```