Question:
Efficient method of calculating density of irregularly spaced points I am attempting to generate map overlay images that would assist in identifying hot-spots, that is areas on the map that have high density of data points. None of the approaches that I've tried are fast enough for my needs. Note: I forgot to mention that the algorithm should work well under both low and high zoom scenarios (or low and high data point density). I looked through numpy, pyplot and scipy libraries, and the closest I could find was numpy.histogram2d. As you can see in the image below, the histogram2d output is rather crude. (Each image includes points overlaying the heatmap for better understanding) My second attempt was to iterate over all the data points, and then calculate the hot-spot value as a function of distance. This produced a better looking image, however it is too slow to use in my application. Since it's O(n), it works ok with 100 points, but blows out when I use my actual dataset of 30000 points. My final attempt was to store the data in an KDTree, and use the nearest 5 points to calculate the hot-spot value. This algorithm is O(1), so much faster with large dataset. It's still not fast enough, it takes about 20 seconds to generate a 256x256 bitmap, and I would like this to happen in around 1 second time. Edit The boxsum smoothing solution provided by 6502 works well at all zoom levels and is much faster than my original methods. The gaussian filter solution suggested by Luke and Neil G is the fastest. You can see all four approaches below, using 1000 data points in total, at 3x zoom there are around 60 points visible. Complete code that generates my original 3 attempts, the boxsum smoothing solution provided by 6502 and gaussian filter suggested by Luke (improved to handle edges better and allow zooming in) is here:
```
import matplotlib
import numpy as np
from matplotlib.mlab import griddata
import matplotlib.cm as cm
import matplotlib.pyplot as plt
import math
from scipy.spatial import KDTree
import time
import scipy.ndimage as ndi

def grid_density_kdtree(xl, yl, xi, yi, dfactor):
    zz = np.empty([len(xi),len(yi)], dtype=np.uint8)
    zipped = zip(xl, yl)
    kdtree = KDTree(zipped)
    for xci in range(0, len(xi)):
        xc = xi[xci]
        for yci in range(0, len(yi)):
            yc = yi[yci]
            density = 0.
            retvalset = kdtree.query((xc,yc), k=5)
            for dist in retvalset[0]:
                density = density + math.exp(-dfactor * pow(dist, 2)) / 5
            zz[yci][xci] = min(density, 1.0) * 255
    return zz

def grid_density(xl, yl, xi, yi):
    ximin, ximax = min(xi), max(xi)
    yimin, yimax = min(yi), max(yi)
    xxi,yyi = np.meshgrid(xi,yi)
    #zz = np.empty_like(xxi)
    zz = np.empty([len(xi),len(yi)])
    for xci in range(0, len(xi)):
        xc = xi[xci]
        for yci in range(0, len(yi)):
            yc = yi[yci]
            density = 0.
            for i in range(0,len(xl)):
                xd = math.fabs(xl[i] - xc)
                yd = math.fabs(yl[i] - yc)
                if xd < 1 and yd < 1:
                    dist = math.sqrt(math.pow(xd, 2) + math.pow(yd, 2))
                    density = density + math.exp(-5.0 * pow(dist, 2))
            zz[yci][xci] = density
    return zz

def boxsum(img, w, h, r):
    st = [0] * (w+1) * (h+1)
    for x in xrange(w):
        st[x+1] = st[x] + img[x]
    for y in xrange(h):
        st[(y+1)*(w+1)] = st[y*(w+1)] + img[y*w]
        for x in xrange(w):
            st[(y+1)*(w+1)+(x+1)] = st[(y+1)*(w+1)+x] + st[y*(w+1)+(x+1)] - st[y*(w+1)+x] + img[y*w+x]
    for y in xrange(h):
        y0 = max(0, y - r)
        y1 = min(h, y + r + 1)
        for x in xrange(w):
            x0 = max(0, x - r)
            x1 = min(w, x + r + 1)
            img[y*w+x] = st[y0*(w+1)+x0] + st[y1*(w+1)+x1] - st[y1*(w+1)+x0] - st[y0*(w+1)+x1]

def grid_density_boxsum(x0, y0, x1, y1, w, h, data):
    kx = (w - 1) / (x1 - x0)
    ky = (h - 1) / (y1 - y0)
    r = 15
    border = r * 2
    imgw = (w + 2 * border)
    imgh = (h + 2 * border)
    img = [0] * (imgw * imgh)
    for x, y in data:
        ix = int((x - x0) * kx) + border
        iy = int((y - y0) * ky) + border
        if 0 <= ix < imgw and 0 <= iy < imgh:
            img[iy * imgw + ix] += 1
    for p in xrange(4):
        boxsum(img, imgw, imgh, r)
    a = np.array(img).reshape(imgh,imgw)
    b = a[border:(border+h),border:(border+w)]
    return b

def grid_density_gaussian_filter(x0, y0, x1, y1, w, h, data):
    kx = (w - 1) / (x1 - x0)
    ky = (h - 1) / (y1 - y0)
    r = 20
    border = r
    imgw = (w + 2 * border)
    imgh = (h + 2 * border)
    img = np.zeros((imgh,imgw))
    for x, y in data:
        ix = int((x - x0) * kx) + border
        iy = int((y - y0) * ky) + border
        if 0 <= ix < imgw and 0 <= iy < imgh:
            img[iy][ix] += 1
    return ndi.gaussian_filter(img, (r,r))  ## gaussian convolution

def generate_graph():
    n = 1000
    # data points range
    data_ymin = -2.
    data_ymax = 2.
    data_xmin = -2.
    data_xmax = 2.
    # view area range
    view_ymin = -.5
    view_ymax = .5
    view_xmin = -.5
    view_xmax = .5
    # generate data
    xl = np.random.uniform(data_xmin, data_xmax, n)
    yl = np.random.uniform(data_ymin, data_ymax, n)
    zl = np.random.uniform(0, 1, n)

    # get visible data points
    xlvis = []
    ylvis = []
    for i in range(0,len(xl)):
        if view_xmin < xl[i] < view_xmax and view_ymin < yl[i] < view_ymax:
            xlvis.append(xl[i])
            ylvis.append(yl[i])

    fig = plt.figure()

    # plot histogram
    plt1 = fig.add_subplot(221)
    plt1.set_axis_off()
    t0 = time.clock()
    zd, xe, ye = np.histogram2d(yl, xl, bins=10, range=[[view_ymin, view_ymax],[view_xmin, view_xmax]], normed=True)
    plt.title('numpy.histogram2d - '+str(time.clock()-t0)+"sec")
    plt.imshow(zd, origin='lower', extent=[view_xmin, view_xmax, view_ymin, view_ymax])
    plt.scatter(xlvis, ylvis)

    # plot density calculated with kdtree
    plt2 = fig.add_subplot(222)
    plt2.set_axis_off()
    xi = np.linspace(view_xmin, view_xmax, 256)
    yi = np.linspace(view_ymin, view_ymax, 256)
    t0 = time.clock()
    zd = grid_density_kdtree(xl, yl, xi, yi, 70)
    plt.title('function of 5 nearest using kdtree\n'+str(time.clock()-t0)+"sec")
    cmap=cm.jet
    A = (cmap(zd/256.0)*255).astype(np.uint8)
    #A[:,:,3] = zd
    plt.imshow(A , origin='lower', extent=[view_xmin, view_xmax, view_ymin, view_ymax])
    plt.scatter(xlvis, ylvis)

    # gaussian filter
    plt3 = fig.add_subplot(223)
    plt3.set_axis_off()
    t0 = time.clock()
    zd = grid_density_gaussian_filter(view_xmin, view_ymin, view_xmax, view_ymax, 256, 256, zip(xl, yl))
    plt.title('ndi.gaussian_filter - '+str(time.clock()-t0)+"sec")
    plt.imshow(zd , origin='lower', extent=[view_xmin, view_xmax, view_ymin, view_ymax])
    plt.scatter(xlvis, ylvis)

    # boxsum smoothing
    plt3 = fig.add_subplot(224)
    plt3.set_axis_off()
    t0 = time.clock()
    zd = grid_density_boxsum(view_xmin, view_ymin, view_xmax, view_ymax, 256, 256, zip(xl, yl))
    plt.title('boxsum smoothing - '+str(time.clock()-t0)+"sec")
    plt.imshow(zd, origin='lower', extent=[view_xmin, view_xmax, view_ymin, view_ymax])
    plt.scatter(xlvis, ylvis)

if __name__=='__main__':
    generate_graph()
    plt.show()
```

Response:
This approach is along the lines of some previous answers: increment a pixel for each spot, then smooth the image with a gaussian filter. A 256x256 image runs in about 350ms on my 6-year-old laptop.
```
import numpy as np
import scipy.ndimage as ndi

data = np.random.rand(30000,2)           ## create random dataset
inds = (data * 255).astype('uint')       ## convert to indices

img = np.zeros((256,256))                ## blank image
for i in xrange(data.shape[0]):          ## draw pixels
    img[inds[i,0], inds[i,1]] += 1

img = ndi.gaussian_filter(img, (10,10))
```


Question:
How to get boxplot data for matplotlib boxplots I need to get the statistical data which were generated to draw a box plot in Pandas(using dataframe to create boxplots). i.e. Quartile1,Quartile2,Quartile3, lower whisker value, upper whisker value and outliers. I tried the following query to draw the boxplot.
```
import pandas as pd
df = pd.DataFrame(np.random.rand(100, 5), columns=['A', 'B', 'C', 'D', 'E'])
pd.DataFrame.boxplot(df,return_type = 'both')
```
Is there a way to do it instead of manually calculating the values?

Response:
One option is to use the y data from the plots - probably most useful for the outliers (fliers)
```
_, bp = pd.DataFrame.boxplot(df, return_type='both')

outliers = [flier.get_ydata() for flier in bp["fliers"]]
boxes = [box.get_ydata() for box in bp["boxes"]]
medians = [median.get_ydata() for median in bp["medians"]]
whiskers = [whiskers.get_ydata() for whiskers in bp["whiskers"]]
```
But it's probably more straightforward to get the other values (including IQR) using either
```
quantiles = df.quantile([0.01, 0.25, 0.5, 0.75, 0.99])
```
or, as suggested by WoodChopper
```
stats = df.describe()
```


Question:
Obtaining values used in boxplot, using python and matplotlib I can draw a boxplot from data:
```
import numpy as np
import matplotlib.pyplot as plt

data = np.random.rand(100)
plt.boxplot(data)
```
Then, the box will range from the 25th-percentile to 75th-percentile, and the whisker will range from the smallest value to the largest value between (25th-percentile - 1.5*IQR, 75th-percentile + 1.5*IQR), where the IQR denotes the inter-quartile range. (Of course, the value 1.5 is customizable). Now I want to know the values used in the boxplot, i.e. the median, upper and lower quartile, the upper whisker end point and the lower whisker end point. While the former three are easy to obtain by using np.median() and np.percentile(), the end point of the whiskers will require some verbose coding:
```
median = np.median(data)
upper_quartile = np.percentile(data, 75)
lower_quartile = np.percentile(data, 25)

iqr = upper_quartile - lower_quartile
upper_whisker = data[data<=upper_quartile+1.5*iqr].max()
lower_whisker = data[data>=lower_quartile-1.5*iqr].min()
```
I was wondering, while this is acceptable, would there be a neater way to do this? It seems that the values should be ready to pull-out from the boxplot, as it's already drawn.

Response:
Why do you want to do so? what you are doing is already pretty direct. Yeah, if you want to fetch them for the plot, when the plot is already made, simply use the get_ydata() method.
```
B = plt.boxplot(data)
[item.get_ydata() for item in B['whiskers']]
```
It returns an array of the shape (2,) for each whiskers, the second element is the value we want:
```
[item.get_ydata()[1] for item in B['whiskers']]
```


Question:
Move X-Axis label downwards, but not X-Axis Ticks in matplotlib I'm using Matplotlib to plot a histogram. Using tips from my previous question: Matplotlib - label each bin, I've more or less got the kinks worked out. There's one final issue - previously - the x-axis label ("Time (in milliseconds)") was being rendered underneath the x-axis tickmarks (0.00, 0.04, 0.08, 0.12 etc.) Using the advice from Joe Kingston (see question above), I tried using:
```py
ax.tick_params(axis='x', pad=30)
```
However, this moves both the x-axis tickmarks (0.00, 0.04, 0.08, 0.12 etc.), as well as the x-axis label ("Time (in milliseconds)"): Is there any way to move only the x-axis label to underneath the three rows of figures? NB: You may need to open the PNGs below directly - Right Click on the image, then View Image (in FF), or Open image in new tab (Chrome). The image resize done by SO has rendered them nigh unreadable

Response:
use labelpad parameter:
```
pl.xlabel("...", labelpad=20)
```
or set it after:
```
ax.xaxis.labelpad = 20
```


Question:
Show tick labels when sharing an axis I'm running the following function:
```
def plot_variance_analysis(indices, stat_frames, legend_labels, shape):
    x = np.linspace(1, 5, 500)
    fig, axes = plt.subplots(shape[0], shape[1], sharex=True sharey=True)
    questions_and_axes = zip(indices, axes.ravel())
    frames_and_labels = zip(stat_frames, legend_labels)
    for qa in questions_and_axes:
        q = qa[0]
        ax = qa[1]
        for fl in frames_and_labels:
            frame = fl[0]
            label = fl[1]
            ax.plot(x, stats.norm.pdf(x, frame['mean'][q], frame['std'][q]), label=label)
            ax.set_xlabel(q)
            ax.legend(loc='best')
    plt.xticks([1,2,3,4,5])
    return fig, axes
```
Here's what I get with some of my own sample data: I'm trying to maintain the shared state between axes, but at the same time display the tick labels for the x axis on all subplots (including the top two). I can't find any means to turn this off in the documentation. Any suggestions? Or should I just set the x tick labels axis by axis? I'm running matplotlib 1.4.0, if that's important.

Response:
In Matplotlib 2.2 and above the tick labels can be turned back on using:
```
ax.xaxis.set_tick_params(labelbottom=True)
```


Question: Save matplotlib plot in grayscale I have some color plots that I need to be saved in grayscale. Is there an easy way to do this without changing the plotting formats?

Response:
There is an easy solution:
```
plt.imsave(filename, image, cmap='gray')
```


Question: How can I create a frontend for matplotlib? I feel that I should start this question by mentioning that I am a Chemical Engineer and not a trained programmer (but I am willing to learn). I process a lot of spectra, which are essentially ASCII files with x,y coordinates produced by spectrometer specific software. A standard software used for analysis is Origin from Originlab but I find it is not user friendly, it has a lot more functions than I need and it is not free. So I was hoping that there is a way to develop a frontend for matplotlib that allows to use matplotlib with a GUI. I want to be able to read in and plot data from ASCII files, zoom in and out in the plot, call scripts for things such as peak-detection and fitting functions where function specific parameters can be entered in the GUI. I am simply puzzled where to start, so the question really is: What is the easiest way to develop a frontend/GUI for matplotlib that allows me to program the things that I mentioned above? Also, ultimately, a cross platform application would be nice because many in my group work in Windows.

Response: Check out also good examples from Eli Bendersky's website: matplotlib with PyQt GUIs More PyQt plotting demos matplotlib with wxPython GUIs


Question:
matplotlib axis label format I am having an issue with the format of the tick labels of an axis. I disabled the offset from the y_axis:
```
ax1.ticklabel_format(style = 'sci', useOffset=False)
```
and tried to put it a scientific format but all I get is:
```
0.00355872
```
but I expected something like:
```
3.55872...E-2
```
or similar. what I really want is something like:
```
3.55872... (on the tick label)
x 10^2  (or something similar - on the axis label)
```
I could try to set the labels as static,, but in the end I will have a few tens or hundreds of plots with different values, so it needs to be set dynamically. An alternative would be to place the y_axis offset as the label, but I also have no clue on how to do this.

Response:
You should also specify axis and threshold limits:
```
ax1.ticklabel_format(axis='y', style='sci', scilimits=(-2,2))
```
This would use sci format on y axis when figures are out of the [0.01, 99] bounds.


Question: How to add trendline to a scatter plot How could I add a trendline to a dot graph drawn using matplotlib.scatter?

Response:
as explained here With help from numpy one can calculate for example a linear fitting.
```
# plot the data itself
pylab.plot(x,y,'o')

# calc the trendline
z = numpy.polyfit(x, y, 1)
p = numpy.poly1d(z)
pylab.plot(x,p(x),"r--")
# the line equation:
print "y=%.6fx+(%.6f)"%(z[0],z[1])
```


Question:
Histogram outlined by added edgecolor I have plotted a histogram and was expecting to see the outlines of my bars but this is not the case. I'm using the following code:
```py
import matplotlib.pyplot as plt
from numpy.random import normal
gaussian_numbers = normal(size=1000)
plt.hist(gaussian_numbers)
plt.title("Gaussian Histogram")
plt.xlabel("Value")
plt.ylabel("Frequency")
plt.show()
```
How do I show the outline of the bars?

Response:
It looks like either your linewidth was set to zero or your edgecolor was set to 'none'. Matplotlib changed the defaults for these in 2.0. Try using:
```
plt.hist(gaussian_numbers, edgecolor='black', linewidth=1.2)
```


Question:
how to show values on top of bar plot [duplicate] This question already has answers here: How to add value labels on a bar chart (7 answers) Closed 6 years ago. Python newbie here. I want to show values above each bin in the following graph: This is my code:
```
x=[i for i in range(1,11)]
y=[0.95,
0.95,
0.89,
0.8,
0.74,
0.65,
0.59,
0.51,
0.5,
0.48]

plt.bar(x, height= y)
xlocs, xlabs = plt.xticks()
xlocs=[i+1 for i in range(0,10)]
xlabs=[i/2 for i in range(0,10)]
plt.xlabel('Max Sigma')
plt.ylabel('Test Accuracy')
plt.xticks(xlocs, xlabs)
plt.show()
```
this is the graph that I want:

Response:
Simply add
```
for i, v in enumerate(y):
    plt.text(xlocs[i] - 0.25, v + 0.01, str(v))
```
before plt.show(). You can adjust the centralization or height of the text by changing the (-0.25) and (0.01) values, respectively.


Question: How to highlight specific x-value ranges I'm making a visualization of historical stock data for a project, and I'd like to highlight regions of drops. For instance, when the stock is experiencing significant drawdown, I would like to highlight it with a red region. Can I do this automatically, or will I have to draw a rectangle or something?

Response:
Have a look at axvspan (and axhspan for highlighting a region of the y-axis).
```
import matplotlib.pyplot as plt

plt.plot(range(10))
plt.axvspan(3, 6, color='red', alpha=0.5)
plt.show()
```
If you're using dates, then you'll need to convert your min and max x values to matplotlib dates. Use matplotlib.dates.date2num for datetime objects or matplotlib.dates.datestr2num for various string timestamps.
```
import matplotlib.pyplot as plt
import matplotlib.dates as mdates
import datetime as dt

t = mdates.drange(dt.datetime(2011, 10, 15), dt.datetime(2011, 11, 27),
                  dt.timedelta(hours=2))
y = np.sin(t)

fig, ax = plt.subplots()
ax.plot_date(t, y, 'b-')
ax.axvspan(*mdates.datestr2num(['10/27/2011', '11/2/2011']), color='red', alpha=0.5)
fig.autofmt_xdate()
plt.show()
```


Question:
How to show labels on matplotlib plots When I execute the following code, it doesn't produce a plot with a label.
```py
import matplotlib.pyplot as plt
import numpy as np
x = np.arange(1, 5)
plt.plot(x, x*1.5, label='Normal')
```
Numpy version is '1.6.2' Matplotlib version is '1.3.x' Why is this happening?

Response:
You forgot to display the legend:
```
...
plt.legend(loc='best')
plt.show()
```


Question:
How to highlight one point in a matplotlib plot Suppose, I have the following two lists that correspond to x- and y-coordinates.
```py
x = [1,2,3,4,5,6]
y = [3,4,5,6,7,8]
```
I want the first pair (1,3) to be in a different color or shape. How can this be done using python?

Response:
One of the simplest possible answers.
```
import matplotlib.pyplot as plt

x = [1,2,3,4,5,6]
y = [3,4,5,6,7,8]

plt.plot(x[1:], y[1:], 'ro')
plt.plot(x[0], y[0], 'g*')

plt.show()
```


Question:
Equivalent function for xticks for an AxesSubplot object So I am trying to use Axes objects to control my matlibplot figure. I am not using plt (aka import matlibplot.pyplot as plt) because I am embedding the figure in my tkinter GUI per this. However, I am also using subplots in the figure, so something like:
```py
a = f.add_subplot(121)
a2 = f.add_subplot(122)
a.plot(fn2,mag)
a2.bar(range(0,10), magBin, width)
```
This is all well and good, I can use the axes properties to control things (i.e. a.axesMethod()), but I want string labels for my bar plots, per this, see code. My dilemma is that I cannot use
```py
plt.xticks(ind+width, ('G1', 'G2', 'G3', 'G4', 'G5') )
```
As in the example, because I cannot use plt if I want to embed it into my Tkinter GUI. I am limited to what I can do with Axes objects. I am trying to use a2.set_xticks, but this does not allow for the string as ticks functionality I need for my bar chart.

Response:
you can use instead:
```
axes.set_xticks(ticks, minor=False)
```
and
```
axes.set_xticklabels(labels, fontdict=None, minor=False)
```


Question:
Matplotlib text dimensions Is it possible to determine the dimensions of a matplotlib text object? How can I find the width and height in pixels? Thanks Edit: I think I figured out a way to do this. I've included an example below.
```
import matplotlib as plt

f = plt.figure()
r = f.canvas.get_renderer()
t = plt.text(0.5, 0.5, 'test')

bb = t.get_window_extent(renderer=r)
width = bb.width
height = bb.height
```

Response:
```
from matplotlib import pyplot as plt

f = plt.figure()
r = f.canvas.get_renderer()
t = plt.text(0.5, 0.5, 'test')

bb = t.get_window_extent(renderer=r)
width = bb.width
height = bb.height
```


Question:
Setting matplotlib colorbar range I would like to set the matplotlib colorbar range. Here's what I have so far:
```
import numpy as np
import matplotlib.pyplot as plt
x = np.arange(20)
y = np.arange(20)
data = x[:-1,None]+y[None,:-1]

fig = plt.gcf()
ax = fig.add_subplot(111)

X,Y = np.meshgrid(x,y)
quadmesh = ax.pcolormesh(X,Y,data)
plt.colorbar(quadmesh)

#RuntimeError: You must first define an image, eg with imshow
#plt.clim(vmin=0,vmax=15)

#AttributeError: 'AxesSubplot' object has no attribute 'clim'
#ax.clim(vmin=0,vmax=15)

#AttributeError: 'AxesSubplot' object has no attribute 'set_clim'
#ax.set_clim(vmin=0,vmax=15)

plt.show()
```
How do I set the colorbar limits here?

Response:
Arg. It's always the last thing you try:
```
quadmesh.set_clim(vmin=0, vmax=15)
```
works.


Question:
How to type-hint a matplotlib.axes._subplots.AxesSubplots object in python3 I was wondering how is the "best" way to type-hint the axis-object of matplotlib-subplots. running
```
from matplotlib import pyplot as plt

f, ax = plt.subplots()
print(type(ax))
```
returns
```
<class 'matplotlib.axes._subplots.AxesSubplot'>
```
and running
```
from matplotlib import axes
print(type(axes._subplots))
print(type(axes._subplots.AxesSubplot))
```
yields
```
<class 'module'>
AttributeError: module 'matplotlib.axes._subplots' has no attribute 'AxesSubplots'
```
So far a solution for type-hinting that works is as follows:
```
def multi_rocker(
                 axy: type(plt.subplots()[1]),
                 y_trues: np.ndarray,
                 y_preds: np.ndarray,
                 ):
  """
  One-Vs-All ROC-curve:
  """
  fpr = dict()
  tpr = dict()
  roc_auc = dict()
  n_classes = y_trues.shape[1]
  wanted = list(range(n_classes))
  for i,x in enumerate(wanted):
    fpr[i], tpr[i], _ = roc_curve(y_trues[:, i], y_preds[:, i])
    roc_auc[i] = round(auc(fpr[i], tpr[i]),2)
  extra = 0
  for i in range(n_classes):
    axy.plot(fpr[i], tpr[i],)
  return
```
And the problem with it is that it isn't clear enough for code-sharing

Response:
As described in Type hints for context manager :
```
import matplotlib.pyplot as plt

def plot_func(ax: plt.Axes):
    ...
```


Question: How to draw vertical lines on a given plot Given a plot of a signal in time representation, how can I draw lines marking the corresponding time index? Specifically, given a signal plot with a time index ranging from 0 to 2.6 (seconds), I want to draw vertical red lines indicating the corresponding time index for the list [0.22058956, 0.33088437, 2.20589566]. How can I do it?

Response:
The standard way to add vertical lines that will cover your entire plot window without you having to specify their actual height is plt.axvline
```
import matplotlib.pyplot as plt

plt.axvline(x=0.22058956)
plt.axvline(x=0.33088437)
plt.axvline(x=2.20589566)
```
OR
```
xcoords = [0.22058956, 0.33088437, 2.20589566]
for xc in xcoords:
    plt.axvline(x=xc)
```
You can use many of the keywords available for other plot commands (e.g. color, linestyle, linewidth ...). You can pass in keyword arguments ymin and ymax if you like in axes corrdinates (e.g. ymin=0.25, ymax=0.75 will cover the middle half of the plot). There are corresponding functions for horizontal lines (axhline) and rectangles (axvspan).


Question:
How to plot one single data point? I have the following code to plot a line and a point:
```
df = pd.DataFrame({'x': [1, 2, 3], 'y': [3, 4, 6]})
point = pd.DataFrame({'x': [2], 'y': [5]})
ax = df.plot(x='x', y='y', label='line')
ax = point.plot(x='x', y='y', ax=ax, style='r-', label='point')
```
How do I get the single data point to show up?

Response:
To plot a single point you can do something like this:
```
plt.plot([x], [y], marker='o', markersize=3, color="red")
```


Question: Is it possible to draw a boxplot given the percentile values instead of the original inputs? From what I can see, boxplot() method expects a sequence of raw values (numbers) as input, from which it then computes percentiles to draw the boxplot(s). I would like to have a method by which I could pass in the percentiles and get the corresponding boxplot. For example: Assume that I have run several benchmarks and for each benchmark I've measured latencies ( floating point values ). Now additionally, I have precomputed the percentiles for these values. Hence for each benchmark, I have the 25th, 50th, 75th percentile along with the min and max. Now given these data, I would like to draw the box plots for the benchmarks.

Response:
As of 2020, there is a better method than the one in the accepted answer. The matplotlib.axes.Axes class provides a bxp method, which can be used to draw the boxes and whiskers based on the percentile values. Raw data is only needed for the outliers, and that is optional. Example:
```
import matplotlib.pyplot as plt

fig, ax = plt.subplots()
boxes = [
    {
        'label' : "Male height",
        'whislo': 162.6,    # Bottom whisker position
        'q1'    : 170.2,    # First quartile (25th percentile)
        'med'   : 175.7,    # Median         (50th percentile)
        'q3'    : 180.4,    # Third quartile (75th percentile)
        'whishi': 187.8,    # Top whisker position
        'fliers': []        # Outliers
    }
]
ax.bxp(boxes, showfliers=False)
ax.set_ylabel("cm")
plt.savefig("boxplot.png")
plt.close()
```
This produces the following image:


Question:
matplotlib.pyplot, preserve aspect ratio of the plot Assuming we have a polygon coordinates as polygon = [(x1, y1), (x2, y2), ...], the following code displays the polygon:
```
import matplotlib.pyplot as plt
plt.fill(*zip(*polygon))
plt.show()
```
By default it is trying to adjust the aspect ratio so that the polygon (or whatever other diagram) fits inside the window, and automatically changing it so that it fits even after resizing. Which is great in many cases, except when you are trying to estimate visually if the image is distorted. How to fix the aspect ratio to be strictly 1:1? (Not sure if "aspect ratio" is the right term here, so in case it is not - I need both X and Y axes to have 1:1 scale, so that (0, 1) on both X and Y takes an exact same amount of screen space. And I need to keep it 1:1 no matter how I resize the window.)

Response:
Does it help to use:
```
plt.axis('equal')
```


Question: Set the legend location of a pandas plot I know how to set the legend location of matplotlib plot with plt.legend(loc='lower left'), however, I am plotting with pandas method df.plot() and need to set the legend location to 'lower left'. Does anyone know how to do it? Edited: I am actually looking for a way to do it through pandas' df.plot(), not via plt.legend(loc='lower left')

Response:
Well, Simply chain it.
```
dframe.rank(ascending=False).plot(kind= 'bar').legend(loc='best')
```
Assuming 'dframe' is a DataFrame.


Question: Hide matplotlib descriptions in jupyter notebook [duplicate] This question already has answers here: Disable the output of matplotlib pyplot (4 answers) Closed 8 years ago. I am not sure what is the correct term for this, but here is what I see when I plot something: The plots is actually what I want so see, but jupyter notebook also outputs some text: ,  which I am trying to get rid of. After some searching, the only thing I was able to find is plt.ioff(), which didn't help me. Is there a way to get rid of the text?

Response: You can finish the corresponding (matplotlib) line with a semicolon ;


Question:
Python scatter plot. Size and style of the marker I have a set of data that I want to show as a scatter plot. I want each point to be plotted as a square of size dx.
```
x = [0.5,0.1,0.3]
          y = [0.2,0.7,0.8]
          z = [10.,15.,12.]
          dx = [0.05,0.2,0.1]

          scatter(x,y,c=z,s=dx,marker='s')
```
The problem is that the size s that the scatter function read is in points^2. What I'd like is having each point represented by a square of area dx^2, where this area is in 'real' units, the plot units. I hope you can get this point. I also have another question. The scatter function plots the markers with a black border, how can I drop this option and have no border at all?

Response:
Translate from user data coordinate system to display coordinate system. and use edgecolors='none' to plot faces with no outlines.
```
import numpy as np

fig = figure()
ax = fig.add_subplot(111)
dx_in_points = np.diff(ax.transData.transform(zip([0]*len(dx), dx)))
scatter(x,y,c=z,s=dx_in_points**2,marker='s', edgecolors='none')
```


Question: Fixing color in scatter plots in matplotlib I want to fix the color range on multiple scatter plots and add in a colorbar to each plot (which will be the same in each figure). Essentially, I'm fixing all aspects of the axes and colorspace etc. so that the plots are directly comparable by eye. For the life of me, I can't seem to figure out all the various ways of fixing the color-range. I've tried vmin, vmax, but it doesn't seem to do anything, I've also tried clim(x,y) and that doesn't seem to work either. This must come up here and there, I can't be the only one that wants to compare various subsets of data amongst plots... so, how do you fix the colors so that each data keeps it's color between plots and doesn't get remapped to a different color due to the change in max/min of the subset -v- the whole set?

Response:
Setting vmin and vmax should do this. Here's an example:
```
import matplotlib.pyplot as plt

xyc = range(20)

plt.subplot(121)
plt.scatter(xyc[:13], xyc[:13], c=xyc[:13], s=35, vmin=0, vmax=20)
plt.colorbar()
plt.xlim(0, 20)
plt.ylim(0, 20)

plt.subplot(122)
plt.scatter(xyc[8:20], xyc[8:20], c=xyc[8:20], s=35, vmin=0, vmax=20)
plt.colorbar()
plt.xlim(0, 20)
plt.ylim(0, 20)

plt.show()
```
And the plot this produces:


Question:
What names can be used in plt.cm.get_cmap? I have this code :
```
plt.scatter(data_projected[:,0],data_projected[:,1],c=digits.target
        ,edgecolors='none',alpha=0.5,cmap=plt.cm.get_cmap('nipy_spectral',10));
```
My confusion comes from plt.cm.get_cmap('nipy_spectral',10). Sometimes there will be plt.cm.get_cmap('RdYlBu') instead. Is the 'RdYlBu','nipy_spectral' the name of a color? And is there any other names to use instead? Is there a list of all colors available? I have read the document but it does not seem to help or I do not understand it.

Response:
The first argument to plt.cm.get_cmap('nipy_spectral',10) is the name of a colormap. You can get a list of all available colormaps via
```
import matplotlib.cm
print(matplotlib.cm.cmap_d.keys())
```
or you may have a look at the colormap reference.


Question: How do I get indices of N maximum values in a NumPy array? NumPy proposes a way to get the index of the maximum value of an array via np.argmax. I would like a similar thing, but returning the indexes of the N maximum values. For instance, if I have an array, [1, 3, 2, 4, 5], then nargmax(array, n=3) would return the indices [4, 3, 1] which correspond to the elements [5, 4, 3].

Response:
Newer NumPy versions (1.8 and up) have a function called argpartition for this. To get the indices of the four largest elements, do
```

```python
a = np.array([9, 4, 4, 3, 3, 9, 0, 4, 6, 0])
```

```python
a
#Output
#array([9, 4, 4, 3, 3, 9, 0, 4, 6, 0])
```

```python
ind = np.argpartition(a, -4)[-4:]
```

```python
ind
#Output
#array([1, 5, 8, 0])
```

```python
top4 = a[ind]
```

```python
top4
#Output
#array([4, 9, 6, 9])
#```
#Unlike argsort, this function runs in linear time in the worst case, but the returned indices are not sorted, as can be seen from the result of evaluating a[ind]. If you need that too, sort them afterwards:
#```
```

```python
ind[np.argsort(a[ind])]
#Output
#array([1, 8, 5, 0])
#```
#To get the top-k elements in sorted order in this way takes O(n + k log k) time.
```


Question:
How can the Euclidean distance be calculated with NumPy? I have two points in 3D space:
```none
a = (ax, ay, az)
b = (bx, by, bz)
```
I want to calculate the distance between them:
```none
dist = sqrt((ax-bx)^2 + (ay-by)^2 + (az-bz)^2)
```
How do I do this with NumPy? I have:
```py
import numpy
a = numpy.array((ax, ay, az))
b = numpy.array((bx, by, bz))
```

Response:
Use numpy.linalg.norm:
```
dist = numpy.linalg.norm(a-b)
```
This works because the Euclidean distance is the l2 norm, and the default value of the ord parameter in numpy.linalg.norm is 2. For more theory, see Introduction to Data Mining:


Question:
How do I count the occurrence of a certain item in an ndarray? How do I count the number of 0s and 1s in the following array?
```
y = np.array([0, 0, 0, 1, 0, 1, 1, 0, 0, 0, 0, 1])
```
y.count(0) gives: numpy.ndarray object has no attribute count

Response:
Using numpy.unique:
```
import numpy
a = numpy.array([0, 3, 0, 1, 0, 1, 2, 1, 0, 0, 0, 0, 1, 3, 4])
unique, counts = numpy.unique(a, return_counts=True)

```python
dict(zip(unique, counts))
#Output
#{0: 7, 1: 4, 2: 1, 3: 2, 4: 1}
#```
#Non-numpy method using collections.Counter;
#```
#import collections, numpy
#a = numpy.array([0, 3, 0, 1, 0, 1, 2, 1, 0, 0, 0, 0, 1, 3, 4])
#counter = collections.Counter(a)
```

```python
counter
#Output
#Counter({0: 7, 1: 4, 3: 2, 2: 1, 4: 1})
#```
```


Question:
How do I create a new column where the values are selected based on an existing column? How do I add a color column to the following dataframe so that color='green' if Set == 'Z', and color='red' otherwise?
```
Type  Set
1     A    Z
2     B    Z
3     B    X
4     C    Y
```

Response:
If you only have two choices to select from then use np.where:
```
df['color'] = np.where(df['Set']=='Z', 'green', 'red')
```
For example,
```
import pandas as pd
import numpy as np

df = pd.DataFrame({'Type':list('ABBC'), 'Set':list('ZZXY')})
df['color'] = np.where(df['Set']=='Z', 'green', 'red')
print(df)
```
yields
```
Set Type  color
0   Z    A  green
1   Z    B  green
2   X    B    red
3   Y    C    red
```
If you have more than two conditions then use np.select. For example, if you want color to be yellow when (df['Set'] == 'Z') & (df['Type'] == 'A') otherwise blue when (df['Set'] == 'Z') & (df['Type'] == 'B') otherwise purple when (df['Type'] == 'B') otherwise black, then use
```
df = pd.DataFrame({'Type':list('ABBC'), 'Set':list('ZZXY')})
conditions = [
    (df['Set'] == 'Z') & (df['Type'] == 'A'),
    (df['Set'] == 'Z') & (df['Type'] == 'B'),
    (df['Type'] == 'B')]
choices = ['yellow', 'blue', 'purple']
df['color'] = np.select(conditions, choices, default='black')
print(df)
```
which yields
```
Set Type   color
0   Z    A  yellow
1   Z    B    blue
2   X    B  purple
3   Y    C   black
```


Question:
Pretty-print a NumPy array without scientific notation and with given precision How do I print formatted NumPy arrays in a way similar to this:
```
x = 1.23456
print('%.3f' % x)
```
If I want to print the numpy.ndarray of floats, it prints several decimals, often in 'scientific' format, which is rather hard to read even for low-dimensional arrays. However, numpy.ndarray apparently has to be printed as a string, i.e., with %s. Is there a solution for this?

Response:
Use numpy.set_printoptions to set the precision of the output:
```
import numpy as np
x = np.random.random(10)
print(x)
# [ 0.07837821  0.48002108  0.41274116  0.82993414  0.77610352  0.1023732
#   0.51303098  0.4617183   0.33487207  0.71162095]

np.set_printoptions(precision=3)
print(x)
# [ 0.078  0.48   0.413  0.83   0.776  0.102  0.513  0.462  0.335  0.712]
```
And suppress suppresses the use of scientific notation for small numbers:
```
y = np.array([1.5e-10, 1.5, 1500])
print(y)
# [  1.500e-10   1.500e+00   1.500e+03]

np.set_printoptions(suppress=True)
print(y)
# [    0.      1.5  1500. ]
```
To apply print options locally, using NumPy 1.15.0 or later, you could use the numpy.printoptions context manager. For example, inside the with-suite precision=3 and suppress=True are set:
```
x = np.random.random(10)
with np.printoptions(precision=3, suppress=True):
    print(x)
    # [ 0.073  0.461  0.689  0.754  0.624  0.901  0.049  0.582  0.557  0.348]
```
But outside the with-suite the print options are back to default settings:
```
print(x)
# [ 0.07334334  0.46132615  0.68935231  0.75379645  0.62424021  0.90115836
#   0.04879837  0.58207504  0.55694118  0.34768638]
```
If you are using an earlier version of NumPy, you can create the context manager yourself. For example,
```
import numpy as np
import contextlib

@contextlib.contextmanager
def printoptions(*args, **kwargs):
    original = np.get_printoptions()
    np.set_printoptions(*args, **kwargs)
    try:
        yield
    finally:
        np.set_printoptions(**original)

x = np.random.random(10)
with printoptions(precision=3, suppress=True):
    print(x)
    # [ 0.073  0.461  0.689  0.754  0.624  0.901  0.049  0.582  0.557  0.348]
```
To prevent zeros from being stripped from the end of floats: np.set_printoptions now has a formatter parameter which allows you to specify a format function for each type.
```
np.set_printoptions(formatter={'float': '{: 0.3f}'.format})
print(x)
```
which prints
```
[ 0.078  0.480  0.413  0.830  0.776  0.102  0.513  0.462  0.335  0.712]
```
instead of
```
[ 0.078  0.48   0.413  0.83   0.776  0.102  0.513  0.462  0.335  0.712]
```


Question:
How do I add an extra column to a NumPy array? Given the following 2D array:
```
a = np.array([
    [1, 2, 3],
    [2, 3, 4],
])
```
I want to add a column of zeros along the second axis to get:
```
b = np.array([
    [1, 2, 3, 0],
    [2, 3, 4, 0],
])
```

Response:
np.r_[...] (docs) and np.c_[...] (docs) are useful alternatives to np.vstack and np.hstack. Note that they use square brackets [] instead of parentheses (). Some examples:
```
: import numpy as np
: N = 3
: A = np.eye(N)

: np.c_[ A, np.ones(N) ]              # add a column
array([[ 1.,  0.,  0.,  1.],
       [ 0.,  1.,  0.,  1.],
       [ 0.,  0.,  1.,  1.]])

: np.c_[ np.ones(N), A, np.ones(N) ]  # or two
array([[ 1.,  1.,  0.,  0.,  1.],
       [ 1.,  0.,  1.,  0.,  1.],
       [ 1.,  0.,  0.,  1.,  1.]])

: np.r_[ A, [A[1]] ]              # add a row
array([[ 1.,  0.,  0.],
       [ 0.,  1.,  0.],
       [ 0.,  0.,  1.],
       [ 0.,  1.,  0.]])
: # not np.r_[ A, A[1] ]

: np.r_[ A[0], 1, 2, 3, A[1] ]    # mix vecs and scalars
  array([ 1.,  0.,  0.,  1.,  2.,  3.,  0.,  1.,  0.])

: np.r_[ A[0], [1, 2, 3], A[1] ]  # lists
  array([ 1.,  0.,  0.,  1.,  2.,  3.,  0.,  1.,  0.])

: np.r_[ A[0], (1, 2, 3), A[1] ]  # tuples
  array([ 1.,  0.,  0.,  1.,  2.,  3.,  0.,  1.,  0.])

: np.r_[ A[0], 1:4, A[1] ]        # same, 1:4 == arange(1,4) == 1,2,3
  array([ 1.,  0.,  0.,  1.,  2.,  3.,  0.,  1.,  0.])
```
The reason for square brackets [] instead of round () is that Python converts 1:4 to slice objects in square brackets.


Question:
Comparing two NumPy arrays for equality, element-wise What is the simplest way to compare two NumPy arrays for equality (where equality is defined as: A = B iff for all indices i: A[i] == B[i])? Simply using == gives me a boolean array:
```

```python
numpy.array([1,1,1]) == numpy.array([1,1,1])
```

array([ True,  True,  True], dtype=bool)
```
Do I have to and the elements of this array to determine if the arrays are equal, or is there a simpler way to compare?

Response:
```
(A==B).all()
```
test if all values of array (A==B) are True. Note: maybe you also want to test A and B shape, such as A.shape == B.shape Special cases and alternatives (from dbaupp's answer and yoavram's comment) It should be noted that: this solution can have a strange behavior in a particular case: if either A or B is empty and the other one contains a single element, then it return True. For some reason, the comparison A==B returns an empty array, for which the all operator returns True. Another risk is if A and B don't have the same shape and aren't broadcastable, then this approach will raise an error. In conclusion, if you have a doubt about A and B shape or simply want to be safe: use one of the specialized functions:
```
np.array_equal(A,B)  # test if same shape, same elements values
np.array_equiv(A,B)  # test if broadcastable shape, same elements values
np.allclose(A,B,...) # test if same shape, elements have close enough values
```


Question:
What is the difference between flatten and ravel functions in numpy?
```
import numpy as np
y = np.array(((1,2,3),(4,5,6),(7,8,9)))
OUTPUT:
print(y.flatten())
[1   2   3   4   5   6   7   8   9]
print(y.ravel())
[1   2   3   4   5   6   7   8   9]
```
Both function return the same list. Then what is the need of two different functions performing same job.

Response: The current API is that: flatten always returns a copy. ravel returns a contiguous view of the original array whenever possible. This isn't visible in the printed output, but if you modify the array returned by ravel, it may modify the entries in the original array. If you modify the entries in an array returned from flatten this will never happen. ravel will often be faster since no memory is copied, but you have to be more careful about modifying the array it returns. reshape((-1,)) gets a view whenever the strides of the array allow it even if that means you don't always get a contiguous array.


Question: Convert NumPy array to Python list How do I convert a NumPy array into a Python List?

Response:
Use tolist():
```

```python
import numpy as np
```

```python
np.array([[1,2,3],[4,5,6]]).tolist()
#Output
#[[1, 2, 3], [4, 5, 6]]
#```
#Note that this converts the values from whatever numpy type they may have (e.g. np.int32 or np.float32) to the "nearest compatible Python type" (in a list). If you want to preserve the numpy data types, you could call list() on your array instead, and you'll end up with a list of numpy scalars. (Thanks to Mr_and_Mrs_D for pointing that out in a comment.)
```


Question:
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all() Let x be a NumPy array. The following:
```
(x > 1) and (x < 3)
```
Gives the error message: ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all() How do I fix this?

Response:
If a and b are Boolean NumPy arrays, the & operation returns the elementwise-and of them:
```
a & b
```
That returns a Boolean array. To reduce this to a single Boolean value, use either
```
(a & b).any()
```
or
```
(a & b).all()
```
Note: if a and b are non-Boolean arrays, consider (a - b).any() or (a - b).all() instead. Rationale The NumPy developers felt there was no one commonly understood way to evaluate an array in Boolean context: it could mean True if any element is True, or it could mean True if all elements are True, or True if the array has non-zero length, just to name three possibilities. Since different users might have different needs and different assumptions, the NumPy developers refused to guess and instead decided to raise a ValueError whenever one tries to evaluate an array in Boolean context. Applying and to two numpy arrays causes the two arrays to be evaluated in Boolean context (by calling __bool__ in Python3 or __nonzero__ in Python2).


Question:
NumPy array initialization (fill with identical values) [duplicate] This question already has answers here: Create numpy matrix filled with NaNs (11 answers) Closed 1 year ago. I need to create a NumPy array of length n, each element of which is v. Is there anything better than:
```
a = empty(n)
for i in range(n):
    a[i] = v
```
I know zeros and ones would work for v = 0, 1. I could use v * ones(n), but it won't work when v is None, and also would be much slower.

Response:
NumPy 1.8 introduced np.full(), which is a more direct method than empty() followed by fill() for creating an array filled with a certain value:
```

```python
np.full((3, 5), 7)
#Output
#array([[ 7.,  7.,  7.,  7.,  7.],
#       [ 7.,  7.,  7.,  7.,  7.],
#       [ 7.,  7.,  7.,  7.,  7.]])
```

```python
np.full((3, 5), 7, dtype=int)
#Output
#array([[7, 7, 7, 7, 7],
#       [7, 7, 7, 7, 7],
#       [7, 7, 7, 7, 7]])
#```
#This is arguably the way of creating an array filled with certain values, because it explicitly describes what is being achieved (and it can in principle be very efficient since it performs a very specific task).
```


Question:
Frequency counts for unique values in a NumPy array How do I efficiently obtain the frequency count for each unique value in a NumPy array?
```

```python
x = np.array([1,1,1,2,2,2,5,25,1,1])
```

```python
freq_count(x)
#Output
#[(1, 5), (2, 3), (5, 1), (25, 1)]
#```
```

Response:
Use numpy.unique with return_counts=True (for NumPy 1.9+):
```
import numpy as np

x = np.array([1,1,1,2,2,2,5,25,1,1])
unique, counts = np.unique(x, return_counts=True)

>>> print(np.asarray((unique, counts)).T)
 [[ 1  5]
  [ 2  3]
  [ 5  1]
  [25  1]]
```
In comparison with scipy.stats.itemfreq:
```

```python
x = np.random.random_integers(0,100,1e6)
```

```python
%timeit unique, counts = np.unique(x, return_counts=True)
10 loops, best of 3: 31.5 ms per loop
```

```python
%timeit scipy.stats.itemfreq(x)
10 loops, best of 3: 170 ms per loop
```
```


Question:
Dropping infinite values from dataframes in pandas? How do I drop nan, inf, and -inf values from a DataFrame without resetting mode.use_inf_as_null? Can I tell dropna to include inf in its definition of missing values so that the following works?
```
df.dropna(subset=["col1", "col2"], how="all")
```

Response:
First replace() infs with NaN:
```
df.replace([np.inf, -np.inf], np.nan, inplace=True)
```
and then drop NaNs via dropna():
```
df.dropna(subset=["col1", "col2"], how="all", inplace=True)
```
For example:
```

```python
df = pd.DataFrame({"col1": [1, np.inf, -np.inf], "col2": [2, 3, np.nan]})
```

```python
df
#Output
#   col1  col2
#0   1.0   2.0
#1   inf   3.0
#2  -inf   NaN
```

```python
df.replace([np.inf, -np.inf], np.nan, inplace=True)
```

```python
df
#Output
#   col1  col2
#0   1.0   2.0
#1   NaN   3.0
#2   NaN   NaN
```

```python
df.dropna(subset=["col1", "col2"], how="all", inplace=True)
```

```python
df
#Output
#   col1  col2
#0   1.0   2.0
#1   NaN   3.0
#```
#The same method also works for Series.
```


Question: What is the difference between ndarray and array in NumPy? What is the difference between ndarray and array in NumPy? Where is their implementation in the NumPy source code?

Response: numpy.array is just a convenience function to create an ndarray; it is not a class itself. You can also create an array using numpy.ndarray, but it is not the recommended way. From the docstring of numpy.ndarray: Arrays should be constructed using array, zeros or empty ... The parameters given here refer to a low-level method (ndarray(...)) for instantiating an array. Most of the meat of the implementation is in C code, here in multiarray, but you can start looking at the ndarray interfaces here: https://github.com/numpy/numpy/blob/master/numpy/core/numeric.py


Question:
How do I remove NaN values from a NumPy array? How do I remove NaN values from a NumPy array?
```
[1, 2, NaN, 4, NaN, 8]   ⟶   [1, 2, 4, 8]
```

Response:
To remove NaN values from a NumPy array x:
```
x = x[~numpy.isnan(x)]
```
Explanation The inner function numpy.isnan returns a boolean/logical array which has the value True everywhere that x is not-a-number. Since we want the opposite, we use the logical-not operator ~ to get an array with Trues everywhere that x is a valid number. Lastly, we use this logical array to index into the original array x, in order to retrieve just the non-NaN values.


Question:
Most efficient way to reverse a numpy array Believe it or not, after profiling my current code, the repetitive operation of numpy array reversion ate a giant chunk of the running time. What I have right now is the common view-based method:
```
reversed_arr = arr[::-1]
```
Is there any other way to do it more efficiently, or is it just an illusion from my obsession with unrealistic numpy performance?

Response:
```
reversed_arr = arr[::-1]
```
gives a reversed view into the original array arr. Any changes made to the original array arr will also be immediately visible in reversed_arr. The underlying data buffers for arr and reversed_arr are shared, so creating this view is always instantaneous, and does not require any additional memory allocation or copying for the array contents. See also, this discussion on NumPy views: How do I create a view onto a NumPy array? Possible solutions to performance problems regarding views Are you re-creating the view more often than you need to? You should be able to do something like this:
```
arr = np.array(some_sequence)
reversed_arr = arr[::-1]

do_something(arr)
look_at(reversed_arr)
do_something_else(arr)
look_at(reversed_arr)
```
I'm not a numpy expert, but this seems like it would be the fastest way to do things in numpy. If this is what you are already doing, I don't think you can improve on it.


Question:
How to remove specific elements in a numpy array How can I remove some specific elements from a numpy array? Say I have
```
import numpy as np

a = np.array([1,2,3,4,5,6,7,8,9])
```
I then want to remove 3,4,7 from a. All I know is the index of the values (index=[2,3,6]).

Response:
Use numpy.delete(), which returns a new array with sub-arrays along an axis deleted.
```
numpy.delete(a, index)
```
For your specific question:
```
import numpy as np

a = np.array([1, 2, 3, 4, 5, 6, 7, 8, 9])
index = [2, 3, 6]

new_a = np.delete(a, index)

print(new_a)
# Output: [1, 2, 5, 6, 8, 9]
```
Note that numpy.delete() returns a new array since array scalars are immutable, similar to strings in Python, so each time a change is made to it, a new object is created. I.e., to quote the delete() docs: "A copy of arr with the elements specified by obj removed. Note that delete does not occur in-place..." If the code I post has output, it is the result of running the code.


Question:
Convert array of indices to one-hot encoded array in NumPy Given a 1D array of indices:
```
a = array([1, 0, 3])
```
I want to one-hot encode this as a 2D array:
```
b = array([[0,1,0,0], [1,0,0,0], [0,0,0,1]])
```

Response:
Create a zeroed array b with enough columns, i.e. a.max() + 1. Then, for each row i, set the a[i]th column to 1.
```

```python
a = np.array([1, 0, 3])
```

```python
b = np.zeros((a.size, a.max() + 1))
```

```python
b[np.arange(a.size), a] = 1
```

```python
b
#Output
#array([[ 0.,  1.,  0.,  0.],
#       [ 1.,  0.,  0.,  0.],
#       [ 0.,  0.,  0.,  1.]])
#```
```


Question:
How to take column-slices of dataframe in pandas I load some machine learning data from a CSV file. The first 2 columns are observations and the remaining columns are features. Currently, I do the following:
```
data = pandas.read_csv('mydata.csv')
```
which gives something like:
```
data = pandas.DataFrame(np.random.rand(10,5), columns = list('abcde'))
```
I'd like to slice this dataframe in two dataframes: one containing the columns a and b and one containing the columns c, d and e. It is not possible to write something like
```
observations = data[:'c']
features = data['c':]
```
I'm not sure what the best method is. Do I need a pd.Panel? By the way, I find dataframe indexing pretty inconsistent: data['a'] is permitted, but data[0] is not. On the other side, data['a':] is not permitted but data[0:] is. Is there a practical reason for this? This is really confusing if columns are indexed by Int, given that data[0] != data[0:1]

Response:
2017 Answer - pandas 0.20: .ix is deprecated. Use .loc See the deprecation in the docs .loc uses label based indexing to select both rows and columns. The labels being the values of the index or the columns. Slicing with .loc includes the last element. Let's assume we have a DataFrame with the following columns: foo, bar, quz, ant, cat, sat, dat.
```
# selects all rows and all columns beginning at 'foo' up to and including 'sat'
df.loc[:, 'foo':'sat']
# foo bar quz ant cat sat
```
.loc accepts the same slice notation that Python lists do for both row and columns. Slice notation being start:stop:step
```
# slice from 'foo' to 'cat' by every 2nd column
df.loc[:, 'foo':'cat':2]
# foo quz cat

# slice from the beginning to 'bar'
df.loc[:, :'bar']
# foo bar

# slice from 'quz' to the end by 3
df.loc[:, 'quz'::3]
# quz sat

# attempt from 'sat' to 'bar'
df.loc[:, 'sat':'bar']
# no columns returned

# slice from 'sat' to 'bar'
df.loc[:, 'sat':'bar':-1]
sat cat ant quz bar

# slice notation is syntatic sugar for the slice function
# slice from 'quz' to the end by 2 with slice function
df.loc[:, slice('quz',None, 2)]
# quz cat dat

# select specific columns with a list
# select columns foo, bar and dat
df.loc[:, ['foo','bar','dat']]
# foo bar dat
```
You can slice by rows and columns. For instance, if you have 5 rows with labels v, w, x, y, z
```
# slice from 'w' to 'y' and 'foo' to 'ant' by 3
df.loc['w':'y', 'foo':'ant':3]
#    foo ant
# w
# x
# y
```


Question:
Create numpy matrix filled with NaNs I have the following code:
```
r = numpy.zeros(shape = (width, height, 9))
```
It creates a width x height x 9 matrix filled with zeros. Instead, I'd like to know if there's a function or way to initialize them instead to NaNs in an easy way.

Response:
You rarely need loops for vector operations in numpy. You can create an uninitialized array and assign to all entries at once:
```

```python
a = numpy.empty((3,3,))
```

```python
a[:] = numpy.nan
```

```python
a
#Output
#array([[ NaN,  NaN,  NaN],
#       [ NaN,  NaN,  NaN],
#       [ NaN,  NaN,  NaN]])
#```
#I have timed the alternatives a[:] = numpy.nan here and a.fill(numpy.nan) as posted by Blaenk:
#```
#$ python -mtimeit "import numpy as np; a = np.empty((100,100));" "a.fill(np.nan)"
#10000 loops, best of 3: 54.3 usec per loop
#$ python -mtimeit "import numpy as np; a = np.empty((100,100));" "a[:] = np.nan"
#10000 loops, best of 3: 88.8 usec per loop
#```
#The timings show a preference for ndarray.fill(..) as the faster alternative. OTOH, I like numpy's convenience implementation where you can assign values to whole slices at the time, the code's intention is very clear. Note that ndarray.fill performs its operation in-place, so numpy.empty((3,3,)).fill(numpy.nan) will instead return None.
```


Question: How do I create a numpy array of all True or all False? In Python, how do I create a numpy array of arbitrary shape filled with all True or all False?

Response:
The answer:
```py
numpy.full((2, 2), True)
```
Explanation: numpy creates arrays of all ones or all zeros very easily: e.g. numpy.ones((2, 2)) or numpy.zeros((2, 2)) Since True and False are represented in Python as 1 and 0, respectively, we have only to specify this array should be boolean using the optional dtype parameter and we are done:
```py
numpy.ones((2, 2), dtype=bool)
```
returns:
```
array([[ True,  True],
       [ True,  True]], dtype=bool)
```
UPDATE: 30 October 2013 Since numpy version 1.8, we can use full to achieve the same result with syntax that more clearly shows our intent (as fmonegaglia points out):
```
numpy.full((2, 2), True, dtype=bool)
```
UPDATE: 16 January 2017 Since at least numpy version 1.12, full automatically casts to the dtype of the second parameter, so we can just write:
```
numpy.full((2, 2), True)
```


Question:
Take multiple lists into dataframe How do I take multiple lists and put them as different columns in a python dataframe? I tried this solution but had some trouble. Attempt 1: Have three lists, and zip them together and use that res = zip(lst1,lst2,lst3) Yields just one column Attempt 2:
```
percentile_list = pd.DataFrame({'lst1Tite' : [lst1],
                                'lst2Tite' : [lst2],
                                'lst3Tite' : [lst3] },
                                columns=['lst1Tite','lst1Tite', 'lst1Tite'])
```
yields either one row by 3 columns (the way above) or if I transpose it is 3 rows and 1 column How do I get a 100 row (length of each independent list) by 3 column (three lists) pandas dataframe?

Response:
I think you're almost there, try removing the extra square brackets around the lst's (Also you don't need to specify the column names when you're creating a dataframe from a dict like this):
```
import pandas as pd
lst1 = range(100)
lst2 = range(100)
lst3 = range(100)
percentile_list = pd.DataFrame(
    {'lst1Title': lst1,
     'lst2Title': lst2,
     'lst3Title': lst3
    })

percentile_list
    lst1Title  lst2Title  lst3Title
0          0         0         0
1          1         1         1
2          2         2         2
3          3         3         3
4          4         4         4
5          5         5         5
6          6         6         6
...
```
If you need a more performant solution you can use np.column_stack rather than zip as in your first attempt, this has around a 2x speedup on the example here, however comes at bit of a cost of readability in my opinion:
```
import numpy as np
percentile_list = pd.DataFrame(np.column_stack([lst1, lst2, lst3]),
                               columns=['lst1Title', 'lst2Title', 'lst3Title'])
```


Question:
How to convert a NumPy array to PIL image applying matplotlib colormap I want to take a NumPy 2D array which represents a grayscale image, and convert it to an RGB PIL image while applying some of the matplotlib colormaps. I can get a reasonable PNG output by using the pyplot.figure.figimage command:
```
dpi = 100.0
w, h = myarray.shape[1]/dpi, myarray.shape[0]/dpi
fig = plt.figure(figsize=(w,h), dpi=dpi)
fig.figimage(sub, cmap=cm.gist_earth)
plt.savefig('out.png')
```
Although I could adapt this to get what I want (probably using StringIO do get the PIL image), I wonder if there is not a simpler way to do that, since it seems to be a very natural problem of image visualization. Let's say, something like this:
```
colored_PIL_image = magic_function(array, cmap)
```

Response:
Quite a busy one-liner, but here it is: First ensure your NumPy array, myarray, is normalised with the max value at 1.0. Apply the colormap directly to myarray. Rescale to the 0-255 range. Convert to integers, using np.uint8(). Use Image.fromarray(). And you're done:
```
from PIL import Image
from matplotlib import cm
im = Image.fromarray(np.uint8(cm.gist_earth(myarray)*255))
```
with plt.savefig(): with im.save():


Question:
numpy matrix vector multiplication [duplicate] This question already has answers here: how does multiplication differ for NumPy Matrix vs Array classes? (8 answers) Closed 11 years ago. When I multiply two numpy arrays of sizes (n x n)*(n x 1), I get a matrix of size (n x n). Following normal matrix multiplication rules, an (n x 1) vector is expected, but I simply cannot find any information about how this is done in Python's Numpy module. The thing is that I don't want to implement it manually to preserve the speed of the program. Example code is shown below:
```
a = np.array([[5, 1, 3], [1, 1, 1], [1, 2, 1]])
b = np.array([1, 2, 3])

print a*b
   >>
   [[5 2 9]
   [1 2 3]
   [1 4 3]]
```
What I want is:
```
print a*b
   >>
   [16 6 8]
```

Response:
Simplest solution Use numpy.dot or a.dot(b). See the documentation here.
```

```python
a = np.array([[ 5, 1 ,3],
#Output
#                  [ 1, 1 ,1],
#                  [ 1, 2 ,1]])
```

```python
b = np.array([1, 2, 3])
```

```python
print a.dot(b)
#Output
#array([16, 6, 8])
#```
#This occurs because numpy arrays are not matrices, and the standard operations *, +, -, / work element-wise on arrays. Note that while you can use numpy.matrix (as of early 2021) where * will be treated like standard matrix multiplication, numpy.matrix is deprecated and may be removed in future releases.. See the note in its documentation (reproduced below): It is no longer recommended to use this class, even for linear algebra. Instead use regular arrays. The class may be removed in the future. Thanks @HopeKing. Other Solutions Also know there are other options: As noted below, if using python3.5+ and numpy v1.10+, the @ operator works as you'd expect:
#```
```

```python
print(a @ b)
#Output
#array([16, 6, 8])
#```
#If you want overkill, you can use numpy.einsum. The documentation will give you a flavor for how it works, but honestly, I didn't fully understand how to use it until reading this answer and just playing around with it on my own.
#```
```

```python
np.einsum('ji,i->j', a, b)
#Output
#array([16, 6, 8])
#```
#As of mid 2016 (numpy 1.10.1), you can try the experimental numpy.matmul, which works like numpy.dot with two major exceptions: no scalar multiplication but it works with stacks of matrices.
#```
```

```python
np.matmul(a, b)
#Output
#array([16, 6, 8])
#```
#numpy.inner functions the same way as numpy.dot for matrix-vector multiplication but behaves differently for matrix-matrix and tensor multiplication (see Wikipedia regarding the differences between the inner product and dot product in general or see this SO answer regarding numpy's implementations).
#```
```

```python
np.inner(a, b)
#Output
#array([16, 6, 8])
```

# Beware using for matrix-matrix multiplication though!

```python
b = a.T
```

```python
np.dot(a, b)
#Output
#array([[35,  9, 10],
#       [ 9,  3,  4],
#       [10,  4,  6]])
```

```python
np.inner(a, b)
#Output
#array([[29, 12, 19],
#       [ 7,  4,  5],
#       [ 8,  5,  6]])
#```
#If you have multiple 2D arrays to dot together, you may consider the np.linalg.multi_dot function, which simplifies the syntax of many nested np.dots. Note that this only works with 2D arrays (i.e. not for matrix-vector multiplication).
#```
```

```python
np.dot(np.dot(a, a.T), a).dot(a.T)
#Output
#  array([[1406,  382,  446],
#         [ 382,  106,  126],
#         [ 446,  126,  152]])
#  >>> np.linalg.multi_dot((a, a.T, a, a.T))
#  array([[1406,  382,  446],
#         [ 382,  106,  126],
#         [ 446,  126,  152]])
#```
#Rarer options for edge cases If you have tensors (arrays of dimension greater than or equal to one), you can use numpy.tensordot with the optional argument axes=1:
#```
```

```python
np.tensordot(a, b, axes=1)
#Output
#array([16,  6,  8])
#```
#Don't use numpy.vdot if you have a matrix of complex numbers, as the matrix will be flattened to a 1D array, then it will try to find the complex conjugate dot product between your flattened matrix and vector (which will fail due to a size mismatch n*m vs n).
```


Question:
Transposing a 1D NumPy array I use Python and NumPy and have some problems with "transpose":
```
import numpy as np
a = np.array([5,4])
print(a)
print(a.T)
```
Invoking a.T is not transposing the array. If a is for example [[],[]] then it transposes correctly, but I need the transpose of [...,...,...].

Response:
It's working exactly as it's supposed to. The transpose of a 1D array is still a 1D array! (If you're used to matlab, it fundamentally doesn't have a concept of a 1D array. Matlab's "1D" arrays are 2D.) If you want to turn your 1D vector into a 2D array and then transpose it, just slice it with np.newaxis (or None, they're the same, newaxis is just more readable).
```
import numpy as np
a = np.array([5,4])[np.newaxis]
print(a)
print(a.T)
```
Generally speaking though, you don't ever need to worry about this. Adding the extra dimension is usually not what you want, if you're just doing it out of habit. Numpy will automatically broadcast a 1D array when doing various calculations. There's usually no need to distinguish between a row vector and a column vector (neither of which are vectors. They're both 2D!) when you just want a vector.


Question: Relationship between SciPy and NumPy SciPy appears to provide most (but not all [1]) of NumPy's functions in its own namespace. In other words, if there's a function named numpy.foo, there's almost certainly a scipy.foo. Most of the time, the two appear to be exactly the same, oftentimes even pointing to the same function object. Sometimes, they're different. To give an example that came up recently: numpy.log10 is a ufunc that returns NaNs for negative arguments; scipy.log10 returns complex values for negative arguments and doesn't appear to be a ufunc. The same can be said about log, log2 and logn, but not about log1p [2]. On the other hand, numpy.exp and scipy.exp appear to be different names for the same ufunc. This is also true of scipy.log1p and numpy.log1p. Another example is numpy.linalg.solve vs scipy.linalg.solve. They're similar, but the latter offers some additional features over the former. Why the apparent duplication? If this is meant to be a wholesale import of numpy into the scipy namespace, why the subtle differences in behaviour and the missing functions? Is there some overarching logic that would help clear up the confusion? [1] numpy.min, numpy.max, numpy.abs and a few others have no counterparts in the scipy namespace. [2] Tested using NumPy 1.5.1 and SciPy 0.9.0rc2.

Response:
Last time I checked it, the scipy __init__ method executes a
```
from numpy import *
```
so that the whole numpy namespace is included into scipy when the scipy module is imported. The log10 behavior you are describing is interesting, because both versions are coming from numpy. One is a ufunc, the other is a numpy.lib function. Why scipy is preferring the library function over the ufunc, I don't know off the top of my head. EDIT: In fact, I can answer the log10 question. Looking in the scipy __init__ method I see this:
```
# Import numpy symbols to scipy name space
import numpy as _num
from numpy import oldnumeric
from numpy import *
from numpy.random import rand, randn
from numpy.fft import fft, ifft
from numpy.lib.scimath import *
```
The log10 function you get in scipy comes from numpy.lib.scimath. Looking at that code, it says:
```
"""
Wrapper functions to more user-friendly calling of certain math functions
whose output data-type is different than the input data-type in certain
domains of the input.

For example, for functions like log() with branch cuts, the versions in this
module provide the mathematically valid answers in the complex plane:

```python
import math
```

```python
from numpy.lib import scimath
```

```python
scimath.log(-math.exp(1)) == (1+1j*math.pi)
#Output
#True
```

Similarly, sqrt(), other base logarithms, power() and trig functions are
correctly handled.  See their respective docstrings for specific examples.
"""
```
It seems that module overlays the base numpy ufuncs for sqrt, log, log2, logn, log10, power, arccos, arcsin, and arctanh. That explains the behavior you are seeing. The underlying design reason why it is done like that is probably buried in a mailing list post somewhere.


Question:
Counting unique values in a column in pandas dataframe like in Qlik? If I have a table like this:
```
df = pd.DataFrame({
         'hID': [101, 102, 103, 101, 102, 104, 105, 101],
         'dID': [10, 11, 12, 10, 11, 10, 12, 10],
         'uID': ['James', 'Henry', 'Abe', 'James', 'Henry', 'Brian', 'Claude', 'James'],
         'mID': ['A', 'B', 'A', 'B', 'A', 'A', 'A', 'C']
})
```
I can do count(distinct hID) in Qlik to come up with count of 5 for unique hID. How do I do that in python using a pandas dataframe? Or maybe a numpy array? Similarly, if were to do count(hID) I will get 8 in Qlik. What is the equivalent way to do it in pandas?

Response:
Count distinct values, use nunique:
```
df['hID'].nunique()
5
```
Count only non-null values, use count:
```
df['hID'].count()
8
```
Count total values including null values, use the size attribute:
```
df['hID'].size
8
```
Edit to add condition Use boolean indexing:
```
df.loc[df['mID']=='A','hID'].agg(['nunique','count','size'])
```
OR using query:
```
df.query('mID == "A"')['hID'].agg(['nunique','count','size'])
```
Output:
```
nunique    5
count      5
size       5
Name: hID, dtype: int64
```


Question:
np.mean() vs np.average() in Python NumPy? I notice that
```

```python
np.mean([1, 2, 3])
#Output
#2.0
```

```python
np.average([1, 2, 3])
#Output
#2.0
#```
#However, there should be some differences, since after all they are two different functions. What are the differences between them?
```

Response:
np.average takes an optional weight parameter. If it is not supplied they are equivalent. Take a look at the source code: Mean, Average np.mean:
```
try:
    mean = a.mean
except AttributeError:
    return _wrapit(a, 'mean', axis, dtype, out)
return mean(axis, dtype, out)
```
np.average:
```
...
if weights is None :
    avg = a.mean(axis)
    scl = avg.dtype.type(a.size/avg.size)
else:
    #code that does weighted mean here

if returned: #returned is another optional argument
    scl = np.multiply(avg, 0) + scl
    return avg, scl
else:
    return avg
...
```


Question: How to count the number of true elements in a NumPy bool array I have a NumPy array 'boolarr' of boolean type. I want to count the number of elements whose values are True. Is there a NumPy or Python routine dedicated for this task? Or, do I need to iterate over the elements in my script?

Response:
You have multiple options. Two options are the following.
```
boolarr.sum()
numpy.count_nonzero(boolarr)
```
Here's an example:
```

```python
import numpy as np
```

```python
boolarr = np.array([[0, 0, 1], [1, 0, 1], [1, 0, 1]], dtype=np.bool)
```

```python
boolarr
#Output
#array([[False, False,  True],
#       [ True, False,  True],
#       [ True, False,  True]], dtype=bool)
```

```python
boolarr.sum()
#Output
#5
#```
#Of course, that is a bool-specific answer. More generally, you can use numpy.count_nonzero.
#```
```

```python
np.count_nonzero(boolarr)
#Output
#5
#```
```


Question: How to split data into 3 sets (train, validation and test)? I have a pandas dataframe and I wish to divide it to 3 separate sets. I know that using train_test_split from sklearn.cross_validation, one can divide the data in two sets (train and test). However, I couldn't find any solution about splitting the data into three sets. Preferably, I'd like to have the indices of the original data. I know that a workaround would be to use train_test_split two times and somehow adjust the indices. But is there a more standard / built-in way to split the data into 3 sets instead of 2?

Response:
Numpy solution. We will shuffle the whole dataset first (df.sample(frac=1, random_state=42)) and then split our data set into the following parts: 60% - train set, 20% - validation set, 20% - test set
```

```python
train, validate, test = \
              np.split(df.sample(frac=1, random_state=42),
                       [int(.6*len(df)), int(.8*len(df))])
```

```python
train
#Output
#A         B         C         D         E
#0  0.046919  0.792216  0.206294  0.440346  0.038960
#2  0.301010  0.625697  0.604724  0.936968  0.870064
#1  0.642237  0.690403  0.813658  0.525379  0.396053
#9  0.488484  0.389640  0.599637  0.122919  0.106505
#8  0.842717  0.793315  0.554084  0.100361  0.367465
#7  0.185214  0.603661  0.217677  0.281780  0.938540
```

```python
validate
#Output
#A         B         C         D         E
#5  0.806176  0.008896  0.362878  0.058903  0.026328
#6  0.145777  0.485765  0.589272  0.806329  0.703479
```

```python
test
#Output
#A         B         C         D         E
#4  0.521640  0.332210  0.370177  0.859169  0.401087
#3  0.333348  0.964011  0.083498  0.670386  0.169619
#```
#[int(.6*len(df)), int(.8*len(df))] - is an indices_or_sections array for numpy.split(). Here is a small demo for np.split() usage - let's split 20-elements array into the following parts: 80%, 10%, 10%:
#```
```

```python
a = np.arange(1, 21)
```

```python
a
#Output
#array([ 1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20])
```

```python
np.split(a, [int(.8 * len(a)), int(.9 * len(a))])
#Output
#[array([ 1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16]),
# array([17, 18]),
# array([19, 20])]
#```
```


Question: How to flatten only some dimensions of a numpy array Is there a quick way to "sub-flatten" or flatten only some of the first dimensions in a numpy array? For example, given a numpy array of dimensions (50,100,25), the resultant dimensions would be (5000,25)

Response:
Take a look at numpy.reshape .
```

```python
arr = numpy.zeros((50,100,25))
```

```python
arr.shape
#Output
## (50, 100, 25)
```

```python
new_arr = arr.reshape(5000,25)
```

```python
new_arr.shape
#Output
## (5000, 25)
```

# One shape dimension can be -1.
# In this case, the value is inferred from
# the length of the array and remaining dimensions.

```python
another_arr = arr.reshape(-1, arr.shape[-1])
```

```python
another_arr.shape
#Output
## (5000, 25)
#```
```


Question: What is the inverse function of zip in python? [duplicate] This question already has answers here: Transpose/Unzip Function (inverse of zip)? (14 answers) Closed 2 years ago. I've used the zip function from the Numpy library to sort tuples and now I have a list containing all the tuples. I had since modified that list and now I would like to restore the tuples so I can use my data. How can I do this?

Response:
```
lst1, lst2 = zip(*zipped_list)
```
should give you the unzipped list. *zipped_list unpacks the zipped_list object. it then passes all the tuples from the zipped_list object to zip, which just packs them back up as they were when you passed them in. so if:
```
a = [1,2,3]
b = [4,5,6]
```
then zipped_list = zip(a,b) gives you:
```
[(1,4), (2,5), (3,6)]
```
and *zipped_list gives you back
```
(1,4), (2,5), (3,6)
```
zipping that with zip(*zipped_list) gives you back the two collections:
```
[(1, 2, 3), (4, 5, 6)]
```


Question:
Concatenate a NumPy array to another NumPy array I have a numpy_array. Something like [ a b c ]. And then I want to concatenate it with another NumPy array (just like we create a list of lists). How do we create a NumPy array containing NumPy arrays? I tried to do the following without any luck
```

```python
M = np.array([])
```

```python
M
#Output
#array([], dtype=float64)
```

```python
M.append(a,axis=0)
#Output
#Traceback (most recent call last):
# File "<stdin>", line 1, in <module>
#AttributeError: 'numpy.ndarray' object has no attribute 'append'
```

```python
a
#Output
#array([1, 2, 3])
#```
```

Response:
```

```python
import numpy as np
```

```python
a = np.array([[1, 2, 3], [4, 5, 6]])
```

```python
b = np.array([[9, 8, 7], [6, 5, 4]])
```

```python
np.concatenate((a, b))
#Output
#array([[1, 2, 3],
#       [4, 5, 6],
#       [9, 8, 7],
#       [6, 5, 4]])
#```
#or this:
#```
```

```python
a = np.array([1, 2, 3])
```

```python
b = np.array([4, 5, 6])
```

```python
np.vstack((a, b))
#Output
#array([[1, 2, 3],
#       [4, 5, 6]])
#```
```


Question:
Extracting specific columns in numpy array This is an easy question but say I have an MxN matrix. All I want to do is extract specific columns and store them in another numpy array but I get invalid syntax errors. Here is the code:
```
extractedData = data[[:,1],[:,9]].
```
It seems like the above line should suffice but I guess not. I looked around but couldn't find anything syntax wise regarding this specific scenario.

Response:
I assume you wanted columns 1 and 9? To select multiple columns at once, use
```
X = data[:, [1, 9]]
```
To select one at a time, use
```
x, y = data[:, 1], data[:, 9]
```
With names:
```
data[:, ['Column Name1','Column Name2']]
```
You can get the names from data.dtype.names…


Question:
Numpy first occurrence of value greater than existing value I have a 1D array in numpy and I want to find the position of the index where a value exceeds the value in numpy array. E.g.
```
aa = range(-10,10)
```
Find position in aa where, the value 5 gets exceeded.

Response:
This is a little faster (and looks nicer)
```
np.argmax(aa>5)
```
Since argmax will stop at the first True ("In case of multiple occurrences of the maximum values, the indices corresponding to the first occurrence are returned.") and doesn't save another list.
```

```python
N = 10000
```

```python
aa = np.arange(-N,N)
```

```python
timeit np.argmax(aa>N/2)
100000 loops, best of 3: 52.3 us per loop
```

```python
timeit np.where(aa>N/2)[0][0]
10000 loops, best of 3: 141 us per loop
```

```python
timeit np.nonzero(aa>N/2)[0][0]
10000 loops, best of 3: 142 us per loop
```
```


Question:
How to add a new row to an empty numpy array Using standard Python arrays, I can do the following:
```
arr = []
arr.append([1,2,3])
arr.append([4,5,6])
# arr is now [[1,2,3],[4,5,6]]
```
However, I cannot do the same thing in numpy. For example:
```
arr = np.array([])
arr = np.append(arr, np.array([1,2,3]))
arr = np.append(arr, np.array([4,5,6]))
# arr is now [1,2,3,4,5,6]
```
I also looked into vstack, but when I use vstack on an empty array, I get:
```
ValueError: all the input array dimensions except for the concatenation axis must match exactly
```
So how do I do append a new row to an empty array in numpy?

Response:
The way to "start" the array that you want is:
```
arr = np.empty((0,3), int)
```
Which is an empty array but it has the proper dimensionality.
```
>>> arr
array([], shape=(0, 3), dtype=int64)
```
Then be sure to append along axis 0:
```
arr = np.append(arr, np.array([[1,2,3]]), axis=0)
arr = np.append(arr, np.array([[4,5,6]]), axis=0)
```
But, @jonrsharpe is right. In fact, if you're going to be appending in a loop, it would be much faster to append to a list as in your first example, then convert to a numpy array at the end, since you're really not using numpy as intended during the loop:
```

```python
%%timeit
   .....: l = []
   .....: for i in xrange(1000):
   .....:     l.append([3*i+1,3*i+2,3*i+3])
   .....: l = np.asarray(l)
   .....:
1000 loops, best of 3: 1.18 ms per loop
```

```python
%%timeit
   .....: a = np.empty((0,3), int)
   .....: for i in xrange(1000):
   .....:     a = np.append(a, 3*i+np.array([[1,2,3]]), 0)
   .....:
100 loops, best of 3: 18.5 ms per loop
```

```python
np.allclose(a, l)
#Output
#True
#```
#The numpythonic way to do it depends on your application, but it would be more like:
#```
```

```python
timeit n = np.arange(1,3001).reshape(1000,3)
100000 loops, best of 3: 5.93 µs per loop
```

```python
np.allclose(a, n)
#Output
#True
#```
```


Question:
Numpy where function multiple conditions I have an array of distances called dists. I want to select dists which are within a range.
```
dists[(np.where(dists >= r)) and (np.where(dists <= r + dr))]
```
However, this selects only for the condition
```
(np.where(dists <= r + dr))
```
If I do the commands sequentially by using a temporary variable it works fine. Why does the above code not work, and how do I get it to work?

Response:
The best way in your particular case would just be to change your two criteria to one criterion:
```
dists[abs(dists - r - dr/2.) <= dr/2.]
```
It only creates one boolean array, and in my opinion is easier to read because it says, is dist within a dr or r? (Though I'd redefine r to be the center of your region of interest instead of the beginning, so r = r + dr/2.) But that doesn't answer your question. The answer to your question: You don't actually need where if you're just trying to filter out the elements of dists that don't fit your criteria:
```
dists[(dists >= r) & (dists <= r+dr)]
```
Because the & will give you an elementwise and (the parentheses are necessary). Or, if you do want to use where for some reason, you can do:
```
dists[(np.where((dists >= r) & (dists <= r + dr)))]
```
Why: The reason it doesn't work is because np.where returns a list of indices, not a boolean array. You're trying to get and between two lists of numbers, which of course doesn't have the True/False values that you expect. If a and b are both True values, then a and b returns b. So saying something like [0,1,2] and [2,3,4] will just give you [2,3,4]. Here it is in action:
```

```python
dists = np.arange(0,10,.5)
```

```python
r = 5
```

```python
dr = 1
```

```python
np.where(dists >= r)
#Output
#(array([10, 11, 12, 13, 14, 15, 16, 17, 18, 19]),)
```

```python
np.where(dists <= r+dr)
#Output
#(array([ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12]),)
```

```python
np.where(dists >= r) and np.where(dists <= r+dr)
#Output
#(array([ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12]),)
#```
#What you were expecting to compare was simply the boolean array, for example
#```
```

```python
dists >= r
#Output
#array([False, False, False, False, False, False, False, False, False,
#       False,  True,  True,  True,  True,  True,  True,  True,  True,
#        True,  True], dtype=bool)
```

```python
dists <= r + dr
#Output
#array([ True,  True,  True,  True,  True,  True,  True,  True,  True,
#        True,  True,  True,  True, False, False, False, False, False,
#       False, False], dtype=bool)
```

```python
(dists >= r) & (dists <= r + dr)
#Output
#array([False, False, False, False, False, False, False, False, False,
#       False,  True,  True,  True, False, False, False, False, False,
#       False, False], dtype=bool)
#```
#Now you can call np.where on the combined boolean array:
#```
```

```python
np.where((dists >= r) & (dists <= r + dr))
#Output
#(array([10, 11, 12]),)
```

```python
dists[np.where((dists >= r) & (dists <= r + dr))]
#Output
#array([ 5. ,  5.5,  6. ])
#```
#Or simply index the original array with the boolean array using fancy indexing
#```
```

```python
dists[(dists >= r) & (dists <= r + dr)]
#Output
#array([ 5. ,  5.5,  6. ])
#```
```


Question:
How to normalize a NumPy array to within a certain range? After doing some processing on an audio or image array, it needs to be normalized within a range before it can be written back to a file. This can be done like so:
```
# Normalize audio channels to between -1.0 and +1.0
audio[:,0] = audio[:,0]/abs(audio[:,0]).max()
audio[:,1] = audio[:,1]/abs(audio[:,1]).max()

# Normalize image to between 0 and 255
image = image/(image.max()/255.0)
```
Is there a less verbose, convenience function way to do this? matplotlib.colors.Normalize() doesn't seem to be related.

Response:
```
# Normalize audio channels to between -1.0 and +1.0
audio /= np.max(np.abs(audio),axis=0)
# Normalize image to between 0 and 255
image *= (255.0/image.max())
```
Using /= and *= allows you to eliminate an intermediate temporary array, thus saving some memory. Multiplication is less expensive than division, so
```
image *= 255.0/image.max()    # Uses 1 division and image.size multiplications
```
is marginally faster than
```
image /= image.max()/255.0    # Uses 1+image.size divisions
```
Since we are using basic numpy methods here, I think this is about as efficient a solution in numpy as can be. In-place operations do not change the dtype of the container array. Since the desired normalized values are floats, the audio and image arrays need to have floating-point point dtype before the in-place operations are performed. If they are not already of floating-point dtype, you'll need to convert them using astype. For example,
```
image = image.astype('float64')
```


Question:
From ND to 1D arrays Say I have an array a:
```
a = np.array([[1,2,3], [4,5,6]])

array([[1, 2, 3],
       [4, 5, 6]])
```
I would like to convert it to a 1D array (i.e. a column vector):
```
b = np.reshape(a, (1,np.product(a.shape)))
```
but this returns
```
array([[1, 2, 3, 4, 5, 6]])
```
which is not the same as:
```
array([1, 2, 3, 4, 5, 6])
```
I can take the first element of this array to manually convert it to a 1D array:
```
b = np.reshape(a, (1,np.product(a.shape)))[0]
```
but this requires me to know how many dimensions the original array has (and concatenate [0]'s when working with higher dimensions) Is there a dimensions-independent way of getting a column/row vector from an arbitrary ndarray?

Response:
Use np.ravel (for a 1D view) or np.ndarray.flatten (for a 1D copy) or np.ndarray.flat (for an 1D iterator):
```

```python
a = np.array([[1,2,3], [4,5,6]])
```

```python
b = a.ravel()
```

```python
b
#Output
#array([1, 2, 3, 4, 5, 6])
#```
#Note that ravel() returns a view of a when possible. So modifying b also modifies a. ravel() returns a view when the 1D elements are contiguous in memory, but would return a copy if, for example, a were made from slicing another array using a non-unit step size (e.g. a = x[::2]). If you want a copy rather than a view, use
#```
```

```python
c = a.flatten()
```
If you just want an iterator, use np.ndarray.flat:
```
```

```python
d = a.flat
```

```python
d
#Output
#<numpy.flatiter object at 0x8ec2068>
```

```python
list(d)
#Output
#[1, 2, 3, 4, 5, 6]
#```
```


Question:
Filtering a list based on a list of booleans I have a list of values which I need to filter given the values in a list of booleans:
```
list_a = [1, 2, 4, 6]
filter = [True, False, True, False]
```
I generate a new filtered list with the following line:
```
filtered_list = [i for indx,i in enumerate(list_a) if filter[indx] == True]
```
which results in:
```
print filtered_list
[1,4]
```
The line works but looks (to me) a bit overkill and I was wondering if there was a simpler way to achieve the same. Advices Summary of two good advices given in the answers below: 1- Don't name a list filter like I did because it is a built-in function. 2- Don't compare things to True like I did with if filter[idx]==True.. since it's unnecessary. Just using if filter[idx] is enough.

Response:
You're looking for itertools.compress:
```

```python
from itertools import compress
```

```python
list_a = [1, 2, 4, 6]
```

```python
fil = [True, False, True, False]
```

```python
list(compress(list_a, fil))
#Output
#[1, 4]
#```
#Timing comparisons(py3.x):
#```
```

```python
list_a = [1, 2, 4, 6]
```

```python
fil = [True, False, True, False]
```

```python
%timeit list(compress(list_a, fil))
#Output
#100000 loops, best of 3: 2.58 us per loop
```

```python
%timeit [i for (i, v) in zip(list_a, fil) if v]  #winner
#Output
#100000 loops, best of 3: 1.98 us per loop
```

```python
list_a = [1, 2, 4, 6]*100
```

```python
fil = [True, False, True, False]*100
```

```python
%timeit list(compress(list_a, fil))              #winner
#Output
#10000 loops, best of 3: 24.3 us per loop
```

```python
%timeit [i for (i, v) in zip(list_a, fil) if v]
#Output
#10000 loops, best of 3: 82 us per loop
```

```python
list_a = [1, 2, 4, 6]*10000
```

```python
fil = [True, False, True, False]*10000
```

```python
%timeit list(compress(list_a, fil))              #winner
#Output
#1000 loops, best of 3: 1.66 ms per loop
```

```python
%timeit [i for (i, v) in zip(list_a, fil) if v]
#Output
#100 loops, best of 3: 7.65 ms per loop
#```
#Don't use filter as a variable name, it is a built-in function.
```


Question:
How to copy a 2D array into a 3rd dimension, N times? I'd like to copy a numpy 2D array into a third dimension. For example, given the 2D numpy array:
```
import numpy as np

arr = np.array([[1, 2], [1, 2]])
# arr.shape = (2, 2)
```
convert it into a 3D matrix with N such copies in a new dimension. Acting on arr with N=3, the output should be:
```py
new_arr[:,:,0]
# array([[1, 2], [1, 2]])

new_arr[:,:,1]
# array([[1, 2], [1, 2]])

new_arr[:,:,2]
# array([[1, 2], [1, 2]])

# new_arr.shape = (2, 2, 3)
```

Response:
Probably the cleanest way is to use np.repeat:
```
a = np.array([[1, 2], [1, 2]])
print(a.shape)
# (2,  2)

# indexing with np.newaxis inserts a new 3rd dimension, which we then repeat the
# array along, (you can achieve the same effect by indexing with None, see below)
b = np.repeat(a[:, :, np.newaxis], 3, axis=2)

print(b.shape)
# (2, 2, 3)

print(b[:, :, 0])
# [[1 2]
#  [1 2]]

print(b[:, :, 1])
# [[1 2]
#  [1 2]]

print(b[:, :, 2])
# [[1 2]
#  [1 2]]
```
Having said that, you can often avoid repeating your arrays altogether by using broadcasting. For example, let's say I wanted to add a (3,) vector:
```
c = np.array([1, 2, 3])
```
to a. I could copy the contents of a 3 times in the third dimension, then copy the contents of c twice in both the first and second dimensions, so that both of my arrays were (2, 2, 3), then compute their sum. However, it's much simpler and quicker to do this:
```
d = a[..., None] + c[None, None, :]
```
Here, a[..., None] has shape (2, 2, 1) and c[None, None, :] has shape (1, 1, 3)*. When I compute the sum, the result gets 'broadcast' out along the dimensions of size 1, giving me a result of shape (2, 2, 3):
```
print(d.shape)
# (2,  2, 3)

print(d[..., 0])    # a + c[0]
# [[2 3]
#  [2 3]]

print(d[..., 1])    # a + c[1]
# [[3 4]
#  [3 4]]

print(d[..., 2])    # a + c[2]
# [[4 5]
#  [4 5]]
```
Broadcasting is a very powerful technique because it avoids the additional overhead involved in creating repeated copies of your input arrays in memory. * Although I included them for clarity, the None indices into c aren't actually necessary - you could also do a[..., None] + c, i.e. broadcast a (2, 2, 1) array against a (3,) array. This is because if one of the arrays has fewer dimensions than the other then only the trailing dimensions of the two arrays need to be compatible. To give a more complicated example:
```
a = np.ones((6, 1, 4, 3, 1))  # 6 x 1 x 4 x 3 x 1
b = np.ones((5, 1, 3, 2))     #     5 x 1 x 3 x 2
result = a + b                # 6 x 5 x 4 x 3 x 2
```


Question:
Efficiently sorting a numpy array in descending order? I am surprised this specific question hasn't been asked before, but I really didn't find it on SO nor on the documentation of np.sort. Say I have a random numpy array holding integers, e.g:
```
> temp = np.random.randint(1,10, 10)
> temp
array([2, 4, 7, 4, 2, 2, 7, 6, 4, 4])
```
If I sort it, I get ascending order by default:
```
> np.sort(temp)
array([2, 2, 2, 4, 4, 4, 4, 6, 7, 7])
```
but I want the solution to be sorted in descending order. Now, I know I can always do:
```
reverse_order = np.sort(temp)[::-1]
```
but is this last statement efficient? Doesn't it create a copy in ascending order, and then reverses this copy to get the result in reversed order? If this is indeed the case, is there an efficient alternative? It doesn't look like np.sort accepts parameters to change the sign of the comparisons in the sort operation to get things in reverse order.

Response:
temp[::-1].sort() sorts the array in place, whereas np.sort(temp)[::-1] creates a new array.
```

```python
temp = np.random.randint(1,10, 10)
```

```python
temp
#Output
#array([5, 2, 7, 4, 4, 2, 8, 6, 4, 4])
```

```python
id(temp)
#Output
#139962713524944
```

```python
temp[::-1].sort()
```

```python
temp
#Output
#array([8, 7, 6, 5, 4, 4, 4, 4, 2, 2])
```

```python
id(temp)
#Output
#139962713524944
#```
```


Question:
How to convert list of numpy arrays into single numpy array? Suppose I have ;
```
LIST = [[array([1, 2, 3, 4, 5]), array([1, 2, 3, 4, 5],[1,2,3,4,5])] # inner lists are numpy arrays
```
I try to convert;
```
array([[1, 2, 3, 4, 5],
       [1, 2, 3, 4, 5],
       [1, 2, 3, 4, 5])
```
I am solving it by iteration on vstack right now but it is really slow for especially large LIST What do you suggest for the best efficient way?

Response:
In general you can concatenate a whole sequence of arrays along any axis:
```
numpy.concatenate( LIST, axis=0 )
```
but you do have to worry about the shape and dimensionality of each array in the list (for a 2-dimensional 3x5 output, you need to ensure that they are all 2-dimensional n-by-5 arrays already). If you want to concatenate 1-dimensional arrays as the rows of a 2-dimensional output, you need to expand their dimensionality. As Jorge's answer points out, there is also the function stack, introduced in numpy 1.10:
```
numpy.stack( LIST, axis=0 )
```
This takes the complementary approach: it creates a new view of each input array and adds an extra dimension (in this case, on the left, so each n-element 1D array becomes a 1-by-n 2D array) before concatenating. It will only work if all the input arrays have the same shape. vstack (or equivalently row_stack) is often an easier-to-use solution because it will take a sequence of 1- and/or 2-dimensional arrays and expand the dimensionality automatically where necessary and only where necessary, before concatenating the whole list together. Where a new dimension is required, it is added on the left. Again, you can concatenate a whole list at once without needing to iterate:
```
numpy.vstack( LIST )
```
This flexible behavior is also exhibited by the syntactic shortcut numpy.r_[ array1, ...., arrayN ] (note the square brackets). This is good for concatenating a few explicitly-named arrays but it becomes less readable in your situation because [] subscripting will not accept a list. You would need to convert your sequence to a tuple: numpy.r_[tuple(LIST)]. It's more readable to simply use vstack(). There is also an analogous function column_stack and shortcut c_[...], for horizontal (column-wise) stacking, as well as an almost-analogous function hstack—although for some reason the latter is less flexible (it is stricter about input arrays' dimensionality, and tries to concatenate 1-D arrays end-to-end instead of treating them as columns). Finally, in the specific case of vertical stacking of 1-D arrays, the following also works:
```
numpy.array( LIST )
```
...because arrays can be constructed out of a sequence of other arrays, adding a new dimension to the beginning.


Question: What are the differences between Pandas and NumPy+SciPy in Python? [closed] Closed. This question is opinion-based. It is not currently accepting answers. Want to improve this question? Because this question may lead to opinionated discussion, debate, and answers, it has been closed. You may edit the question if you feel you can improve it so that it requires answers that include facts and citations or a detailed explanation of the proposed solution. If edited, the question will be reviewed and might be reopened. Closed 10 years ago. Improve this question They both seem exceedingly similar and I'm curious as to which package would be more beneficial for financial data analysis.

Response: pandas provides high level data manipulation tools built on top of NumPy. NumPy by itself is a fairly low-level tool, similar to MATLAB. pandas on the other hand provides rich time series functionality, data alignment, NA-friendly statistics, groupby, merge and join methods, and lots of other conveniences. It has become very popular in recent years in financial applications. I will have a chapter dedicated to financial data analysis using pandas in my upcoming book.


Question: Numpy matrix to array I am using numpy. I have a matrix with 1 column and N rows and I want to get an array from with N elements. For example, if i have M = matrix([[1], [2], [3], [4]]), I want to get A = array([1,2,3,4]). To achieve it, I use A = np.array(M.T)[0]. Does anyone know a more elegant way to get the same result? Thanks!

Response:
If you'd like something a bit more readable, you can do this:
```
A = np.squeeze(np.asarray(M))
```
Equivalently, you could also do: A = np.asarray(M).reshape(-1), but that's a bit less easy to read.


Question:
initialize a numpy array [duplicate] This question already has answers here: Create numpy matrix filled with NaNs (11 answers) Closed 1 year ago. Is there way to initialize a numpy array of a shape and add to it? I will explain what I need with a list example. If I want to create a list of objects generated in a loop, I can do:
```
a = []
for i in range(5):
    a.append(i)
```
I want to do something similar with a numpy array. I know about vstack, concatenate etc. However, it seems these require two numpy arrays as inputs. What I need is:
```
big_array # Initially empty. This is where I don't know what to specify
for i in range(5):
    array i of shape = (2,4) created.
    add to big_array
```
The big_array should have a shape (10,4). How to do this? EDIT: I want to add the following clarification. I am aware that I can define big_array = numpy.zeros((10,4)) and then fill it up. However, this requires specifying the size of big_array in advance. I know the size in this case, but what if I do not? When we use the .append function for extending the list in python, we don't need to know its final size in advance. I am wondering if something similar exists for creating a bigger array from smaller arrays, starting with an empty array.

Response: numpy.zeros Return a new array of given shape and type, filled with zeros. or numpy.ones Return a new array of given shape and type, filled with ones. or numpy.empty Return a new array of given shape and type, without initializing entries. However, the mentality in which we construct an array by appending elements to a list is not much used in numpy, because it's less efficient (numpy datatypes are much closer to the underlying C arrays). Instead, you should preallocate the array to the size that you need it to be, and then fill in the rows. You can use numpy.append if you must, though.


Question:
How to create a DataFrame of random integers with Pandas? I know that if I use randn, the following code gives me what I am looking for, but with elements from a normal distribution. But what if I just wanted random integers?
```py
import pandas as pd
import numpy as np
df = pd.DataFrame(np.random.randn(100, 4), columns=list('ABCD'))
```
randint works by providing a range, but not an array like randn. So how do I do this with random integers between some range?

Response:
numpy.random.randint accepts a third argument (size) , in which you can specify the size of the output array. You can use this to create your DataFrame -
```
df = pd.DataFrame(np.random.randint(0,100,size=(100, 4)), columns=list('ABCD'))
```
Here - np.random.randint(0,100,size=(100, 4)) - creates an output array of size (100,4) with random integer elements between [0,100) . Demo -
```python
import numpy as np
import pandas as pd
df = pd.DataFrame(np.random.randint(0,100,size=(100, 4)), columns=list('ABCD'))
```
which produces:
```none
A   B   C   D
0   45  88  44  92
1   62  34   2  86
2   85  65  11  31
3   74  43  42  56
4   90  38  34  93
5    0  94  45  10
6   58  23  23  60
..  ..  ..  ..  ..
```