Question:
Suppress Scientific Notation in Numpy When Creating Array From Nested List I have a nested Python list that looks like the following:
```
my_list = [[3.74, 5162, 13683628846.64, 12783387559.86, 1.81],
 [9.55, 116, 189688622.37, 260332262.0, 1.97],
 [2.2, 768, 6004865.13, 5759960.98, 1.21],
 [3.74, 4062, 3263822121.39, 3066869087.9, 1.93],
 [1.91, 474, 44555062.72, 44555062.72, 0.41],
 [5.8, 5006, 8254968918.1, 7446788272.74, 3.25],
 [4.5, 7887, 30078971595.46, 27814989471.31, 2.18],
 [7.03, 116, 66252511.46, 81109291.0, 1.56],
 [6.52, 116, 47674230.76, 57686991.0, 1.43],
 [1.85, 623, 3002631.96, 2899484.08, 0.64],
 [13.76, 1227, 1737874137.5, 1446511574.32, 4.32],
 [13.76, 1227, 1737874137.5, 1446511574.32, 4.32]]
```
I then import Numpy, and set print options to (suppress=True). When I create an array:
```
my_array = numpy.array(my_list)
```
I can't for the life of me suppress scientific notation:
```
[[  3.74000000e+00   5.16200000e+03   1.36836288e+10   1.27833876e+10
    1.81000000e+00]
 [  9.55000000e+00   1.16000000e+02   1.89688622e+08   2.60332262e+08
    1.97000000e+00]
 [  2.20000000e+00   7.68000000e+02   6.00486513e+06   5.75996098e+06
    1.21000000e+00]
 [  3.74000000e+00   4.06200000e+03   3.26382212e+09   3.06686909e+09
    1.93000000e+00]
 [  1.91000000e+00   4.74000000e+02   4.45550627e+07   4.45550627e+07
    4.10000000e-01]
 [  5.80000000e+00   5.00600000e+03   8.25496892e+09   7.44678827e+09
    3.25000000e+00]
 [  4.50000000e+00   7.88700000e+03   3.00789716e+10   2.78149895e+10
    2.18000000e+00]
 [  7.03000000e+00   1.16000000e+02   6.62525115e+07   8.11092910e+07
    1.56000000e+00]
 [  6.52000000e+00   1.16000000e+02   4.76742308e+07   5.76869910e+07
    1.43000000e+00]
 [  1.85000000e+00   6.23000000e+02   3.00263196e+06   2.89948408e+06
    6.40000000e-01]
 [  1.37600000e+01   1.22700000e+03   1.73787414e+09   1.44651157e+09
    4.32000000e+00]
 [  1.37600000e+01   1.22700000e+03   1.73787414e+09   1.44651157e+09
    4.32000000e+00]]
```
If I create a simple numpy array directly:
```
new_array = numpy.array([1.5, 4.65, 7.845])
```
I have no problem and it prints as follows:
```
[ 1.5    4.65   7.845]
```
Does anyone know what my problem is?

Response:
This is what you need:
```
np.set_printoptions(suppress=True)
```
Here is the documentation which says suppress: bool, optional If True, always print floating point numbers using fixed point notation, in which case numbers equal to zero in the current precision will print as zero. If False, then scientific notation is used when absolute value of the smallest number is  1e3. The default is False. In the original question, the difference between the array created "directly" and the original "big" array is that the big array contains very large numbers (e.g. 1.44651157e+09), so NumPy chooses the scientific notation for it, unless it's suppressed.


Question: concatenate numpy arrays which are elements of a list I have a list containing numpy arrays something like L=[a,b,c] where a, b and c are numpy arrays with sizes N_a in T, N_b in T and N_c in T. I want to row-wise concatenate a, b and c and get a numpy array with shape (N_a+N_b+N_c, T). Clearly one solution is run a for loop and use numpy.concatenate, but is there any pythonic way to do this? Thanks

Response:
Use numpy.vstack.
```
L = (a,b,c)
arr = np.vstack(L)
```


Question:
Convert Python sequence to NumPy array, filling missing values The implicit conversion of a Python sequence of variable-length lists into a NumPy array cause the array to be of type object.
```
v = [[1], [1, 2]]
np.array(v)

```python
array([[1], [1, 2]], dtype=object)
#Output
#```
#Trying to force another type will cause an exception:
#```
#np.array(v, dtype=np.int32)
#ValueError: setting an array element with a sequence.
#```
#What is the most efficient way to get a dense NumPy array of type int32, by filling the "missing" values with a given placeholder? From my sample sequence v, I would like to get something like this, if 0 is the placeholder
#```
#array([[1, 0], [1, 2]], dtype=int32)
#```
```

Response:
You can use itertools.zip_longest:
```
import itertools
np.array(list(itertools.zip_longest(*v, fillvalue=0))).T
Out:
array([[1, 0],
       [1, 2]])
```
Note: For Python 2, it is itertools.izip_longest.


Question:
"isnotnan" functionality in numpy, can this be more pythonic? I need a function that returns non-NaN values from an array. Currently I am doing it this way:
```

```python
a = np.array([np.nan, 1, 2])
```

```python
a
#Output
#array([ NaN,   1.,   2.])
```

```python
np.invert(np.isnan(a))
#Output
#array([False,  True,  True], dtype=bool)
```

```python
a[np.invert(np.isnan(a))]
#Output
#array([ 1.,  2.])
#```
#Python: 2.6.4 numpy: 1.3.0 Please share if you know a better way, Thank you
```

Response:
```
a = a[~np.isnan(a)]
```


Question:
Numpy array dimensions How do I get the dimensions of an array? For instance, this is 2x2:
```py
a = np.array([[1, 2], [3, 4]])
```

Response:
Use .shape to obtain a tuple of array dimensions:
```

```python
a.shape
#Output
#(2, 2)
#```
```


Question:
Get first and second highest values in pandas columns I am using pandas to analyse some election results. I have a DF, Results, which has a row for each constituency and columns representing the votes for the various parties (over 100 of them):
```

```python
Results.columns
#Output
#Index(['Constituency', 'Region', 'Country', 'ID', 'Type', 'Electorate',
#       'Total', 'Unnamed: 9', '30-50', 'Above',
#       ...
#       'WP', 'WRP', 'WVPTFP', 'Yorks', 'Young', 'Zeb', 'Party', 'Votes',
#       'Share', 'Turnout'],
#      dtype='object', length=147)
#```
#So...
#```
```

```python
Results.head()
#Output
#Constituency    Region   Country         ID    Type  \
#PAID
#1                            Aberavon     Wales     Wales  W07000049  County
#2                           Aberconwy     Wales     Wales  W07000058  County
#3                      Aberdeen North  Scotland  Scotland  S14000001   Burgh
#4                      Aberdeen South  Scotland  Scotland  S14000002   Burgh
#5     Aberdeenshire West & Kincardine  Scotland  Scotland  S14000058  County
#
#      Electorate  Total  Unnamed: 9  30-50  Above    ...     WP  WRP  WVPTFP  \
#PAID                                                 ...
#1          49821  31523         NaN    NaN    NaN    ...    NaN  NaN     NaN
#2          45525  30148         NaN    NaN    NaN    ...    NaN  NaN     NaN
#3          67745  43936         NaN    NaN    NaN    ...    NaN  NaN     NaN
#4          68056  48551         NaN    NaN    NaN    ...    NaN  NaN     NaN
#5          73445  55196         NaN    NaN    NaN    ...    NaN  NaN     NaN
#
#      Yorks  Young  Zeb  Party  Votes     Share   Turnout
#PAID
#1       NaN    NaN  NaN    Lab  15416  0.489040  0.632725
#2       NaN    NaN  NaN    Con  12513  0.415052  0.662230
#3       NaN    NaN  NaN    SNP  24793  0.564298  0.648550
#4       NaN    NaN  NaN    SNP  20221  0.416490  0.713398
#5       NaN    NaN  NaN    SNP  22949  0.415773  0.751528
#
#[5 rows x 147 columns]
#```
#The per-constituency results for each party are given in the columns Results.ix[:, 'Unnamed: 9': 'Zeb'] I can find the winning party (i.e. the party which polled highest number of votes) and the number of votes it polled using:
#```
#RawResults = Results.ix[:, 'Unnamed: 9': 'Zeb']
#Results['Party'] = RawResults.idxmax(axis=1)
#Results['Votes'] = RawResults.max(axis=1).astype(int)
#```
#But, I also need to know how many votes the second-place party got (and ideally its index/name). So is there any way in pandas to return the second highest value/index in a set of columns for each row?
```

Response:
To get the highest values of a column you can try nlargest() :
```
df['High'].nlargest(2)
```
The above will give you the 2 highest values of column High. You can also use nsmallest() to get the lowest values.


Question:
Is it possible to vectorize recursive calculation of a NumPy array where each element depends on the previous one?
```
T(i) = Tm(i) + (T(i-1)-Tm(i))**(-tau(i))
```
Tm and tau are NumPy vectors of the same length that have been previously calculated, and the desire is to create a new vector T. The i is included only to indicate the element index for what is desired. Is a for loop necessary for this case?

Response:
You might think this would work:
```
import numpy as np
n = len(Tm)
t = np.empty(n)

t[0] = 0  # or whatever the initial condition is
t[1:] = Tm[1:] + (t[0:n-1] - Tm[1:])**(-tau[1:])
```
but it doesn't: you can't actually do recursion in numpy this way (since numpy calculates the whole RHS and then assigns it to the LHS). So unless you can come up with a non-recursive version of this formula, you're stuck with an explicit loop:
```
tt = np.empty(n)
tt[0] = 0.
for i in range(1,n):
    tt[i] = Tm[i] + (tt[i-1] - Tm[i])**(-tau[i])
```


Question:
Minimize function with parameters Currently I have the following code that defines the function f.
```
a = #something
b = #something
c = #something
def f(x):
    """Evaluates some function that depends on parameters a, b, and c"""
    someNumber = #some calculation
    return someNumber
```
Ideally I would do def f(x, a, b, c), BUT I am minimizing f with respect to x and SciPy's optimization toolbox doesn't allow for one to minimize functions with parameters in the arguments. That said I would like to run my minimization code for multiple values of a, b and c. Is there a way I can do this?

Response:
You can specify additional arguments in args
```
from scipy.optimize import minimize
minimize(f, x0, args=(a, b, c))
```


Question:
Sorting arrays in NumPy by column How do I sort a NumPy array by its nth column? For example, given:
```
a = array([[9, 2, 3],
           [4, 5, 6],
           [7, 0, 5]])
```
I want to sort the rows of a by the second column to obtain:
```
array([[7, 0, 5],
       [9, 2, 3],
       [4, 5, 6]])
```

Response:
To sort by the second column of a:
```
a[a[:, 1].argsort()]
```


Question:
How to call an element in a Numpy array? This looked like a very simple question but I couldn't find the answer. How to call an element in an Numpy array?
```py
import numpy as np

arr = np.array([[1,2,3,4,5],[6,7,8,9,10]])

print arr(0,0)
```
The code above doesn't work.

Response:
Just use square brackets instead:
```
print arr[0,0]
```


Question:
Pandas Split Dataframe into two Dataframes at a specific column I have pandas DataFrame which I have composed from concat. One row consists of 96 values, I would like to split the DataFrame from the value 72. So that the first 72 values of a row are stored in Dataframe1, and the next 24 values of a row in Dataframe2. I create my DF as follows:
```
temps = DataFrame(myData)
datasX = concat(
[temps.shift(72), temps.shift(71), temps.shift(70), temps.shift(69), temps.shift(68), temps.shift(67),
 temps.shift(66), temps.shift(65), temps.shift(64), temps.shift(63), temps.shift(62), temps.shift(61),
 temps.shift(60), temps.shift(59), temps.shift(58), temps.shift(57), temps.shift(56), temps.shift(55),
 temps.shift(54), temps.shift(53), temps.shift(52), temps.shift(51), temps.shift(50), temps.shift(49),
 temps.shift(48), temps.shift(47), temps.shift(46), temps.shift(45), temps.shift(44), temps.shift(43),
 temps.shift(42), temps.shift(41), temps.shift(40), temps.shift(39), temps.shift(38), temps.shift(37),
 temps.shift(36), temps.shift(35), temps.shift(34), temps.shift(33), temps.shift(32), temps.shift(31),
 temps.shift(30), temps.shift(29), temps.shift(28), temps.shift(27), temps.shift(26), temps.shift(25),
 temps.shift(24), temps.shift(23), temps.shift(22), temps.shift(21), temps.shift(20), temps.shift(19),
 temps.shift(18), temps.shift(17), temps.shift(16), temps.shift(15), temps.shift(14), temps.shift(13),
 temps.shift(12), temps.shift(11), temps.shift(10), temps.shift(9), temps.shift(8), temps.shift(7),
 temps.shift(6), temps.shift(5), temps.shift(4), temps.shift(3), temps.shift(2), temps.shift(1), temps,
 temps.shift(-1), temps.shift(-2), temps.shift(-3), temps.shift(-4), temps.shift(-5), temps.shift(-6),
 temps.shift(-7), temps.shift(-8), temps.shift(-9), temps.shift(-10), temps.shift(-11), temps.shift(-12),
 temps.shift(-13), temps.shift(-14), temps.shift(-15), temps.shift(-16), temps.shift(-17), temps.shift(-18),
 temps.shift(-19), temps.shift(-20), temps.shift(-21), temps.shift(-22), temps.shift(-23)], axis=1)
```
Question is: How can split them? :)

Response:
iloc
```
df1 = datasX.iloc[:, :72]
df2 = datasX.iloc[:, 72:]
```
(iloc docs)


Question: Is there a way to get the index of the median in python in one command? Is there something like numpy.argmin(x), but for median?

Response:
a quick approximation:
```
numpy.argsort(data)[len(data)//2]
```


Question:
Opposite of numpy.unwrap In Python numpy, there is an unwrap function that: Unwrap radian phase p by changing absolute jumps greater than discont to their 2*pi complement along the given axis. Now, I'd like to do the opposite function. How can I wrap an array of phases? E.g. how to convert all angles to constraint them between -π and π? The obvious way would be to do something like:
```
for i, a in enumerate(phases):
    while a < pi:
        a += 2 * pi
    while a > pi:
        a -= 2 * pi
    phases[i] = a
```
but is there a simpler / faster way?

Response:
```
phases = (phases + np.pi) % (2 * np.pi) - np.pi
```


Question:
Numpy reshape 1d to 2d array with 1 column In numpy the dimensions of the resulting array vary at run time. There is often confusion between a 1d array and a 2d array with 1 column. In one case I can iterate over the columns, in the other case I cannot. How do you solve elegantly that problem? To avoid littering my code with if statements checking for the dimensionality, I use this function:
```
def reshape_to_vect(ar):
    if len(ar.shape) == 1:
      return ar.reshape(ar.shape[0],1)
    return ar
```
However, this feels inelegant and costly. Is there a better solution?

Response:
The simplest way:
```
ar.reshape(-1, 1)
```


Question:
What does axis = 0 do in Numpy's sum function? I am learning Python, and have encountered numpy.sum. It has an optional parameter axis. This parameter is used to get either column-wise summation or row-wise summation. When axis = 0 we imply to sum it over columns only. For example,
```
a = np.array([[1, 2, 3], [4, 5, 6]])
np.sum(a, axis = 0)
```
This snippet of code produces output: array([5, 7, 9]), fine. But if I do:
```
a = np.array([1, 2, 3])
np.sum(a, axis = 0)
```
I get result: 6, why is that? Shouldn't I get array([1, 2, 3])?

Response: In case someone needs this visual description:


Question:
Renaming column names in Pandas I want to change the column labels of a Pandas DataFrame from
```
['$a', '$b', '$c', '$d', '$e']
```
to
```
['a', 'b', 'c', 'd', 'e']
```

Response:
Rename Specific Columns Use the df.rename() function and refer the columns to be renamed. Not all the columns have to be renamed:
```
df = df.rename(columns={'oldName1': 'newName1', 'oldName2': 'newName2'})

# Or rename the existing DataFrame (rather than creating a copy)
df.rename(columns={'oldName1': 'newName1', 'oldName2': 'newName2'}, inplace=True)
```
Minimal Code Example
```
df = pd.DataFrame('x', index=range(3), columns=list('abcde'))
df

   a  b  c  d  e
0  x  x  x  x  x
1  x  x  x  x  x
2  x  x  x  x  x
```
The following methods all work and produce the same output:
```
df2 = df.rename({'a': 'X', 'b': 'Y'}, axis=1)
df2 = df.rename({'a': 'X', 'b': 'Y'}, axis='columns')
df2 = df.rename(columns={'a': 'X', 'b': 'Y'})

df2

   X  Y  c  d  e
0  x  x  x  x  x
1  x  x  x  x  x
2  x  x  x  x  x
```
Remember to assign the result back, as the modification is not-inplace. Alternatively, specify inplace=True:
```
df.rename({'a': 'X', 'b': 'Y'}, axis=1, inplace=True)
df

   X  Y  c  d  e
0  x  x  x  x  x
1  x  x  x  x  x
2  x  x  x  x  x
```
You can specify errors='raise' to raise errors if an invalid column-to-rename is specified. Reassign Column Headers Use df.set_axis() with axis=1.
```
df2 = df.set_axis(['V', 'W', 'X', 'Y', 'Z'], axis=1)
df2

   V  W  X  Y  Z
0  x  x  x  x  x
1  x  x  x  x  x
2  x  x  x  x  x
```
Headers can be assigned directly:
```
df.columns = ['V', 'W', 'X', 'Y', 'Z']
df

   V  W  X  Y  Z
0  x  x  x  x  x
1  x  x  x  x  x
2  x  x  x  x  x
```


Question:
Delete a column from a Pandas DataFrame To delete a column in a DataFrame, I can successfully use:
```py
del df['column_name']
```
But why can't I use the following?
```py
del df.column_name
```
Since it is possible to access the Series via df.column_name, I expected this to work.

Response:
The best way to do this in Pandas is to use drop:
```
df = df.drop('column_name', axis=1)
```
where 1 is the axis number (0 for rows and 1 for columns.) Or, the drop() method accepts index/columns keywords as an alternative to specifying the axis. So we can now just do:
```
df = df.drop(columns=['column_nameA', 'column_nameB'])
```
This was introduced in v0.21.0 (October 27, 2017) To delete the column without having to reassign df you can do:
```
df.drop('column_name', axis=1, inplace=True)
```
Finally, to drop by column number instead of by column label, try this to delete, e.g. the 1st, 2nd and 4th columns:
```
df = df.drop(df.columns[[0, 1, 3]], axis=1)  # df.columns is zero-based pd.Index
```
Also working with "text" syntax for the columns:
```
df.drop(['column_nameA', 'column_nameB'], axis=1, inplace=True)
```


Question:
Selecting multiple columns in a Pandas dataframe How do I select columns a and b from df, and save them into a new dataframe df1?
```none
index  a   b   c
1      2   3   4
2      3   4   5
```
Unsuccessful attempt:
```py
df1 = df['a':'b']
df1 = df.ix[:, 'a':'b']
```

Response:
The column names (which are strings) cannot be sliced in the manner you tried. Here you have a couple of options. If you know from context which variables you want to slice out, you can just return a view of only those columns by passing a list into the __getitem__ syntax (the []'s).
```
df1 = df[['a', 'b']]
```
Alternatively, if it matters to index them numerically and not by their name (say your code should automatically do this without knowing the names of the first two columns) then you can do this instead:
```
df1 = df.iloc[:, 0:2] # Remember that Python does not slice inclusive of the ending index.
```
Additionally, you should familiarize yourself with the idea of a view into a Pandas object vs. a copy of that object. The first of the above methods will return a new copy in memory of the desired sub-object (the desired slices). Sometimes, however, there are indexing conventions in Pandas that don't do this and instead give you a new variable that just refers to the same chunk of memory as the sub-object or slice in the original object. This will happen with the second way of indexing, so you can modify it with the .copy() method to get a regular copy. When this happens, changing what you think is the sliced object can sometimes alter the original object. Always good to be on the look out for this.
```
df1 = df.iloc[0, 0:2].copy() # To avoid the case where changing df1 also changes df
```
To use iloc, you need to know the column positions (or indices). As the column positions may change, instead of hard-coding indices, you can use iloc along with get_loc function of columns method of dataframe object to obtain column indices.
```
{df.columns.get_loc(c): c for idx, c in enumerate(df.columns)}
```
Now you can use this dictionary to access columns through names and using iloc.


Question:
Change column type in pandas I created a DataFrame from a list of lists:
```py
table = [
    ['a',  '1.2',  '4.2' ],
    ['b',  '70',   '0.03'],
    ['x',  '5',    '0'   ],
]

df = pd.DataFrame(table)
```
How do I convert the columns to specific types? In this case, I want to convert columns 2 and 3 into floats. Is there a way to specify the types while converting the list to DataFrame? Or is it better to create the DataFrame first and then loop through the columns to change the dtype for each column? Ideally I would like to do this in a dynamic way because there can be hundreds of columns, and I don't want to specify exactly which columns are of which type. All I can guarantee is that each column contains values of the same type.

Response:
You have four main options for converting types in pandas: to_numeric() - provides functionality to safely convert non-numeric types (e.g. strings) to a suitable numeric type. (See also to_datetime() and to_timedelta().) astype() - convert (almost) any type to (almost) any other type (even if it's not necessarily sensible to do so). Also allows you to convert to categorial types (very useful). infer_objects() - a utility method to convert object columns holding Python objects to a pandas type if possible. convert_dtypes() - convert DataFrame columns to the "best possible" dtype that supports pd.NA (pandas' object to indicate a missing value). Read on for more detailed explanations and usage of each of these methods. 1. to_numeric() The best way to convert one or more columns of a DataFrame to numeric values is to use pandas.to_numeric(). This function will try to change non-numeric objects (such as strings) into integers or floating-point numbers as appropriate. Basic usage The input to to_numeric() is a Series or a single column of a DataFrame.
```

```python
s = pd.Series(["8", 6, "7.5", 3, "0.9"]) # mixed string and numeric values
```

```python
s
#Output
#0      8
#1      6
#2    7.5
#3      3
#4    0.9
#dtype: object
```

```python
pd.to_numeric(s) # convert everything to float values
#Output
#0    8.0
#1    6.0
#2    7.5
#3    3.0
#4    0.9
#dtype: float64
#```
#As you can see, a new Series is returned. Remember to assign this output to a variable or column name to continue using it:
#```
## convert Series
#my_series = pd.to_numeric(my_series)
```

# convert column "a" of a DataFrame
df["a"] = pd.to_numeric(df["a"])
```
You can also use it to convert multiple columns of a DataFrame via the apply() method:
```
# convert all columns of DataFrame
df = df.apply(pd.to_numeric) # convert all columns of DataFrame

# convert just columns "a" and "b"
df[["a", "b"]] = df[["a", "b"]].apply(pd.to_numeric)
```
As long as your values can all be converted, that's probably all you need. Error handling But what if some values can't be converted to a numeric type? to_numeric() also takes an errors keyword argument that allows you to force non-numeric values to be NaN, or simply ignore columns containing these values. Here's an example using a Series of strings s which has the object dtype:
```

```python
s = pd.Series(['1', '2', '4.7', 'pandas', '10'])
```

```python
s
#Output
#0         1
#1         2
#2       4.7
#3    pandas
#4        10
#dtype: object
#```
#The default behaviour is to raise if it can't convert a value. In this case, it can't cope with the string 'pandas':
#```
```

```python
pd.to_numeric(s) # or pd.to_numeric(s, errors='raise')
#Output
#ValueError: Unable to parse string
#```
#Rather than fail, we might want 'pandas' to be considered a missing/bad numeric value. We can coerce invalid values to NaN as follows using the errors keyword argument:
#```
```

```python
pd.to_numeric(s, errors='coerce')
#Output
#0     1.0
#1     2.0
#2     4.7
#3     NaN
#4    10.0
#dtype: float64
#```
#The third option for errors is just to ignore the operation if an invalid value is encountered:
#```
```

```python
pd.to_numeric(s, errors='ignore')
#Output
## the original Series is returned untouched
#```
#This last option is particularly useful for converting your entire DataFrame, but don't know which of our columns can be converted reliably to a numeric type. In that case, just write:
#```
#df.apply(pd.to_numeric, errors='ignore')
#```
#The function will be applied to each column of the DataFrame. Columns that can be converted to a numeric type will be converted, while columns that cannot (e.g. they contain non-digit strings or dates) will be left alone. Downcasting By default, conversion with to_numeric() will give you either an int64 or float64 dtype (or whatever integer width is native to your platform). That's usually what you want, but what if you wanted to save some memory and use a more compact dtype, like float32, or int8? to_numeric() gives you the option to downcast to either 'integer', 'signed', 'unsigned', 'float'. Here's an example for a simple series s of integer type:
#```
```

```python
s = pd.Series([1, 2, -7])
```

```python
s
#Output
#0    1
#1    2
#2   -7
#dtype: int64
#```
#Downcasting to 'integer' uses the smallest possible integer that can hold the values:
#```
```

```python
pd.to_numeric(s, downcast='integer')
#Output
#0    1
#1    2
#2   -7
#dtype: int8
#```
#Downcasting to 'float' similarly picks a smaller than normal floating type:
#```
```

```python
pd.to_numeric(s, downcast='float')
#Output
#0    1.0
#1    2.0
#2   -7.0
#dtype: float32
#```
#2. astype() The astype() method enables you to be explicit about the dtype you want your DataFrame or Series to have. It's very versatile in that you can try and go from one type to any other. Basic usage Just pick a type: you can use a NumPy dtype (e.g. np.int16), some Python types (e.g. bool), or pandas-specific types (like the categorical dtype). Call the method on the object you want to convert and astype() will try and convert it for you:
#```
## convert all DataFrame columns to the int64 dtype
#df = df.astype(int)
```

# convert column "a" to int64 dtype and "b" to complex type
df = df.astype({"a": int, "b": complex})

# convert Series to float16 type
s = s.astype(np.float16)

# convert Series to Python strings
s = s.astype(str)

# convert Series to categorical type - see docs for more details
s = s.astype('category')
```
Notice I said "try" - if astype() does not know how to convert a value in the Series or DataFrame, it will raise an error. For example, if you have a NaN or inf value you'll get an error trying to convert it to an integer. As of pandas 0.20.0, this error can be suppressed by passing errors='ignore'. Your original object will be returned untouched. Be careful astype() is powerful, but it will sometimes convert values "incorrectly". For example:
```

```python
s = pd.Series([1, 2, -7])
```

```python
s
#Output
#0    1
#1    2
#2   -7
#dtype: int64
#```
#These are small integers, so how about converting to an unsigned 8-bit type to save memory?
#```
```

```python
s.astype(np.uint8)
#Output
#0      1
#1      2
#2    249
#dtype: uint8
#```
#The conversion worked, but the -7 was wrapped round to become 249 (i.e. 28 - 7)! Trying to downcast using pd.to_numeric(s, downcast='unsigned') instead could help prevent this error. 3. infer_objects() Version 0.21.0 of pandas introduced the method infer_objects() for converting columns of a DataFrame that have an object datatype to a more specific type (soft conversions). For example, here's a DataFrame with two columns of object type. One holds actual integers and the other holds strings representing integers:
#```
```

```python
df = pd.DataFrame({'a': [7, 1, 5], 'b': ['3','2','1']}, dtype='object')
```

```python
df.dtypes
#Output
#a    object
#b    object
#dtype: object
#```
#Using infer_objects(), you can change the type of column 'a' to int64:
#```
```

```python
df = df.infer_objects()
```

```python
df.dtypes
#Output
#a     int64
#b    object
#dtype: object
#```
#Column 'b' has been left alone since its values were strings, not integers. If you wanted to force both columns to an integer type, you could use df.astype(int) instead. 4. convert_dtypes() Version 1.0 and above includes a method convert_dtypes() to convert Series and DataFrame columns to the best possible dtype that supports the pd.NA missing value. Here "best possible" means the type most suited to hold the values. For example, this a pandas integer type, if all of the values are integers (or missing values): an object column of Python integer objects are converted to Int64, a column of NumPy int32 values, will become the pandas dtype Int32. With our object DataFrame df, we get the following result:
#```
```

```python
df.convert_dtypes().dtypes
#Output
#a     Int64
#b    string
#dtype: object
#```
#Since column 'a' held integer values, it was converted to the Int64 type (which is capable of holding missing values, unlike int64). Column 'b' contained string objects, so was changed to pandas' string dtype. By default, this method will infer the type from object values in each column. We can change this by passing infer_objects=False:
#```
```

```python
df.convert_dtypes(infer_objects=False).dtypes
#Output
#a    object
#b    string
#dtype: object
#```
#Now column 'a' remained an object column: pandas knows it can be described as an 'integer' column (internally it ran infer_dtype) but didn't infer exactly what dtype of integer it should have so did not convert it. Column 'b' was again converted to 'string' dtype as it was recognised as holding 'string' values.
```


Question:
Use a list of values to select rows from a Pandas dataframe Let’s say I have the following Pandas dataframe:
```
df = DataFrame({'A': [5,6,3,4], 'B': [1,2,3,5]})
df

     A   B
0    5   1
1    6   2
2    3   3
3    4   5
```
I can subset based on a specific value:
```
x = df[df['A'] == 3]
x

     A   B
2    3   3
```
But how can I subset based on a list of values? - something like this:
```
list_of_values = [3, 6]

y = df[df['A'] in list_of_values]
```
To get:
```
A    B
1    6    2
2    3    3
```

Response:
You can use the isin method:
```

```python
df = pd.DataFrame({'A': [5,6,3,4], 'B': [1,2,3,5]})
```

```python
df
#Output
#A  B
#0  5  1
#1  6  2
#2  3  3
#3  4  5
```

```python
df[df['A'].isin([3, 6])]
#Output
#A  B
#1  6  2
#2  3  3
#```
#And to get the opposite use ~:
#```
```

```python
df[~df['A'].isin([3, 6])]
#Output
#A  B
#0  5  1
#3  4  5
#```
```


Question:
How to add a new column to an existing DataFrame I have the following indexed DataFrame with named columns and rows not- continuous numbers:
```none
a         b         c         d
2  0.671399  0.101208 -0.181532  0.241273
3  0.446172 -0.243316  0.051767  1.577318
5  0.614758  0.075793 -0.451460 -0.012493
```
I would like to add a new column, 'e', to the existing data frame and do not want to change anything in the data frame (i.e., the new column always has the same length as the DataFrame).
```none
0   -0.335485
1   -1.166658
2   -0.385571
dtype: float64
```
I tried different versions of join, append, merge, but I did not get the result I wanted, only errors at most. How can I add column e to the above example?

Response:
Edit 2017 As indicated in the comments and by @Alexander, currently the best method to add the values of a Series as a new column of a DataFrame could be using assign:
```
df1 = df1.assign(e=pd.Series(np.random.randn(sLength)).values)
```
Edit 2015 Some reported getting the SettingWithCopyWarning with this code. However, the code still runs perfectly with the current pandas version 0.16.1.
```

```python
sLength = len(df1['a'])
```

```python
df1
#Output
#          a         b         c         d
#6 -0.269221 -0.026476  0.997517  1.294385
#8  0.917438  0.847941  0.034235 -0.448948
```

```python
df1['e'] = pd.Series(np.random.randn(sLength), index=df1.index)
```

```python
df1
#Output
#          a         b         c         d         e
#6 -0.269221 -0.026476  0.997517  1.294385  1.757167
#8  0.917438  0.847941  0.034235 -0.448948  2.228131
```

```python
pd.version.short_version
#Output
#'0.16.1'
#```
#The SettingWithCopyWarning aims to inform of a possibly invalid assignment on a copy of the Dataframe. It doesn't necessarily say you did it wrong (it can trigger false positives) but from 0.13.0 it let you know there are more adequate methods for the same purpose. Then, if you get the warning, just follow its advise: Try using .loc[row_index,col_indexer] = value instead
#```
```

```python
df1.loc[:,'f'] = pd.Series(np.random.randn(sLength), index=df1.index)
```

```python
df1
#Output
#          a         b         c         d         e         f
#6 -0.269221 -0.026476  0.997517  1.294385  1.757167 -0.050927
#8  0.917438  0.847941  0.034235 -0.448948  2.228131  0.006109
#>>>
#```
#In fact, this is currently the more efficient method as described in pandas docs Original answer: Use the original df1 indexes to create the series:
#```
#df1['e'] = pd.Series(np.random.randn(sLength), index=df1.index)
#```
```


Question: Pretty-print an entire Pandas Series / DataFrame I work with Series and DataFrames on the terminal a lot. The default __repr__ for a Series returns a reduced sample, with some head and tail values, but the rest missing. Is there a builtin way to pretty-print the entire Series / DataFrame? Ideally, it would support proper alignment, perhaps borders between columns, and maybe even color-coding for the different columns.

Response:
You can also use the option_context, with one or more options:
```
with pd.option_context('display.max_rows', None, 'display.max_columns', None):  # more options can be specified also
    print(df)
```
This will automatically return the options to their previous values. If you are working on jupyter-notebook, using display(df) instead of print(df) will use jupyter rich display logic (like so).


Question:
How are iloc and loc different? Can someone explain how these two methods of slicing are different? I've seen the docs and I've seen previous similar questions (1, 2), but I still find myself unable to understand how they are different. To me, they seem interchangeable in large part, because they are at the lower levels of slicing. For example, say we want to get the first five rows of a DataFrame. How is it that these two work?
```py
df.loc[:5]
df.iloc[:5]
```
Can someone present cases where the distinction in uses are clearer? Once upon a time, I also wanted to know how these two functions differed from df.ix[:5] but ix has been removed from pandas 1.0, so I don't care anymore.

Response:
Label vs. Location The main distinction between the two methods is: loc gets rows (and/or columns) with particular labels. iloc gets rows (and/or columns) at integer locations. To demonstrate, consider a series s of characters with a non-monotonic integer index:
```

```python
s = pd.Series(list("abcdef"), index=[49, 48, 47, 0, 1, 2])
#Output
#49    a
#48    b
#47    c
#0     d
#1     e
#2     f
```

```python
s.loc[0]    # value at index label 0
#Output
#'d'
```

```python
s.iloc[0]   # value at index location 0
#Output
#'a'
```

```python
s.loc[0:1]  # rows at index labels between 0 and 1 (inclusive)
#Output
#0    d
#1    e
```

```python
s.iloc[0:1] # rows at index location between 0 and 1 (exclusive)
#Output
#49    a
#```
#Here are some of the differences/similarities between s.loc and s.iloc when passed various objects:  description s.loc[] s.iloc[] 0 single item Value at index label 0 (the string 'd') Value at index location 0 (the string 'a') 0:1 slice Two rows (labels 0 and 1) One row (first row at location 0) 1:47 slice with out-of-bounds end Zero rows (empty Series) Five rows (location 1 onwards) 1:47:-1 slice with negative step three rows (labels 1 back to 47) Zero rows (empty Series) [2, 0] integer list Two rows with given labels Two rows with given locations s > 'e' Bool series (indicating which values have the property) One row (containing 'f') NotImplementedError (s>'e').values Bool array One row (containing 'f') Same as loc 999 int object not in index KeyError IndexError (out of bounds) -1 int object not in index KeyError Returns last value in s lambda x: x.index[3] callable applied to series (here returning 3rd item in index) s.loc[s.index[3]] s.iloc[s.index[3]] loc's label-querying capabilities extend well-beyond integer indexes and it's worth highlighting a couple of additional examples. Here's a Series where the index contains string objects:
#```
```

```python
s2 = pd.Series(s.index, index=s.values)
```

```python
s2
#Output
#a    49
#b    48
#c    47
#d     0
#e     1
#f     2
#```
#Since loc is label-based, it can fetch the first value in the Series using s2.loc['a']. It can also slice with non-integer objects:
#```
```

```python
s2.loc['c':'e']  # all rows lying between 'c' and 'e' (inclusive)
#Output
#c    47
#d     0
#e     1
#```
#For DateTime indexes, we don't need to pass the exact date/time to fetch by label. For example:
#```
```

```python
s3 = pd.Series(list('abcde'), pd.date_range('now', periods=5, freq='M'))
```

```python
s3
#Output
#2021-01-31 16:41:31.879768    a
#2021-02-28 16:41:31.879768    b
#2021-03-31 16:41:31.879768    c
#2021-04-30 16:41:31.879768    d
#2021-05-31 16:41:31.879768    e
#```
#Then to fetch the row(s) for March/April 2021 we only need:
#```
```

```python
s3.loc['2021-03':'2021-04']
#Output
#2021-03-31 17:04:30.742316    c
#2021-04-30 17:04:30.742316    d
#```
#Rows and Columns loc and iloc work the same way with DataFrames as they do with Series. It's useful to note that both methods can address columns and rows together. When given a tuple, the first element is used to index the rows and, if it exists, the second element is used to index the columns. Consider the DataFrame defined below:
#```
```

```python
import numpy as np
```

```python
df = pd.DataFrame(np.arange(25).reshape(5, 5),
#Output
#                      index=list('abcde'),
#                      columns=['x','y','z', 8, 9])
```

```python
df
#Output
#    x   y   z   8   9
#a   0   1   2   3   4
#b   5   6   7   8   9
#c  10  11  12  13  14
#d  15  16  17  18  19
#e  20  21  22  23  24
#```
#Then for example:
#```
```

```python
df.loc['c': , :'z']  # rows 'c' and onwards AND columns up to 'z'
#Output
#    x   y   z
#c  10  11  12
#d  15  16  17
#e  20  21  22
```

```python
df.iloc[:, 3]        # all rows, but only the column at index location 3
#Output
#a     3
#b     8
#c    13
#d    18
#e    23
#```
#Sometimes we want to mix label and positional indexing methods for the rows and columns, somehow combining the capabilities of loc and iloc. For example, consider the following DataFrame. How best to slice the rows up to and including 'c' and take the first four columns?
#```
```

```python
import numpy as np
```

```python
df = pd.DataFrame(np.arange(25).reshape(5, 5),
#Output
#                      index=list('abcde'),
#                      columns=['x','y','z', 8, 9])
```

```python
df
#Output
#    x   y   z   8   9
#a   0   1   2   3   4
#b   5   6   7   8   9
#c  10  11  12  13  14
#d  15  16  17  18  19
#e  20  21  22  23  24
#```
#We can achieve this result using iloc and the help of another method:
#```
```

```python
df.iloc[:df.index.get_loc('c') + 1, :4]
#Output
#    x   y   z   8
#a   0   1   2   3
#b   5   6   7   8
#c  10  11  12  13
#```
#get_loc() is an index method meaning "get the position of the label in this index". Note that since slicing with iloc is exclusive of its endpoint, we must add 1 to this value if we want row 'c' as well.
```


Question:
Creating an empty Pandas DataFrame, and then filling it I'm starting from the pandas DataFrame documentation here: Introduction to data structures I'd like to iteratively fill the DataFrame with values in a time series kind of calculation. I'd like to initialize the DataFrame with columns A, B, and timestamp rows, all 0 or all NaN. I'd then add initial values and go over this data calculating the new row from the row before, say row[A][t] = row[A][t-1]+1 or so. I'm currently using the code as below, but I feel it's kind of ugly and there must be a way to do this with a DataFrame directly or just a better way in general.
```py
import pandas as pd
import datetime as dt
import scipy as s
base = dt.datetime.today().date()
dates = [ base - dt.timedelta(days=x) for x in range(9, -1, -1) ]

valdict = {}
symbols = ['A','B', 'C']
for symb in symbols:
    valdict[symb] = pd.Series( s.zeros(len(dates)), dates )

for thedate in dates:
    if thedate > dates[0]:
        for symb in valdict:
            valdict[symb][thedate] = 1 + valdict[symb][thedate - dt.timedelta(days=1)]
```

Response:
NEVER grow a DataFrame row-wise! TLDR: (just read the bold text) Most answers here will tell you how to create an empty DataFrame and fill it out, but no one will tell you that it is a bad thing to do. Here is my advice: Accumulate data in a list, not a DataFrame. Use a list to collect your data, then initialise a DataFrame when you are ready. Either a list-of-lists or list-of-dicts format will work, pd.DataFrame accepts both.
```
data = []
for row in some_function_that_yields_data():
    data.append(row)

df = pd.DataFrame(data)
```
pd.DataFrame converts the list of rows (where each row is a scalar value) into a DataFrame. If your function yields DataFrames instead, call pd.concat. Pros of this approach: It is always cheaper to append to a list and create a DataFrame in one go than it is to create an empty DataFrame (or one of NaNs) and append to it over and over again. Lists also take up less memory and are a much lighter data structure to work with, append, and remove (if needed). dtypes are automatically inferred (rather than assigning object to all of them). A RangeIndex is automatically created for your data, instead of you having to take care to assign the correct index to the row you are appending at each iteration. If you aren't convinced yet, this is also mentioned in the documentation: Iteratively appending rows to a DataFrame can be more computationally intensive than a single concatenate. A better solution is to append those rows to a list and then concatenate the list with the original DataFrame all at once. pandas >= 2.0 update: append has been removed! DataFrame.append was deprecated in version 1.4 and removed from the pandas API entirely in version 2.0. See also this github issue that originally proposed its deprecation. These options are horrible append or concat inside a loop Here is the biggest mistake I've seen from beginners:
```
df = pd.DataFrame(columns=['A', 'B', 'C'])
for a, b, c in some_function_that_yields_data():
    df = df.append({'A': i, 'B': b, 'C': c}, ignore_index=True) # yuck
    # or similarly,
    # df = pd.concat([df, pd.Series({'A': i, 'B': b, 'C': c})], ignore_index=True)
```
Memory is re-allocated for every append or concat operation you have. Couple this with a loop and you have a quadratic complexity operation. The other mistake associated with df.append is that users tend to forget append is not an in-place function, so the result must be assigned back. You also have to worry about the dtypes:
```
df = pd.DataFrame(columns=['A', 'B', 'C'])
df = df.append({'A': 1, 'B': 12.3, 'C': 'xyz'}, ignore_index=True)

df.dtypes
A     object   # yuck!
B    float64
C     object
dtype: object
```
Dealing with object columns is never a good thing, because pandas cannot vectorize operations on those columns. You will need to call the infer_objects() method to fix it:
```
df.infer_objects().dtypes
A      int64
B    float64
C     object
dtype: object
```
loc inside a loop I have also seen loc used to append to a DataFrame that was created empty:
```
df = pd.DataFrame(columns=['A', 'B', 'C'])
for a, b, c in some_function_that_yields_data():
    df.loc[len(df)] = [a, b, c]
```
As before, you have not pre-allocated the amount of memory you need each time, so the memory is re-grown each time you create a new row. It's just as bad as append, and even more ugly. Empty DataFrame of NaNs And then, there's creating a DataFrame of NaNs, and all the caveats associated therewith.
```
df = pd.DataFrame(columns=['A', 'B', 'C'], index=range(5))
df
     A    B    C
0  NaN  NaN  NaN
1  NaN  NaN  NaN
2  NaN  NaN  NaN
3  NaN  NaN  NaN
4  NaN  NaN  NaN
```
It creates a DataFrame of object columns, like the others.
```
df.dtypes
A    object  # you DON'T want this
B    object
C    object
dtype: object
```
Appending still has all the issues as the methods above.
```
for i, (a, b, c) in enumerate(some_function_that_yields_data()):
    df.iloc[i] = [a, b, c]
```
The Proof is in the Pudding Timing these methods is the fastest way to see just how much they differ in terms of their memory and utility. Benchmarking code for reference.


Question:
Shuffle DataFrame rows I have the following DataFrame:
```
Col1  Col2  Col3  Type
0      1     2     3     1
1      4     5     6     1
...
20     7     8     9     2
21    10    11    12     2
...
45    13    14    15     3
46    16    17    18     3
...
```
The DataFrame is read from a CSV file. All rows which have Type 1 are on top, followed by the rows with Type 2, followed by the rows with Type 3, etc. I would like to shuffle the order of the DataFrame's rows so that all Type's are mixed. A possible result could be:
```
Col1  Col2  Col3  Type
0      7     8     9     2
1     13    14    15     3
...
20     1     2     3     1
21    10    11    12     2
...
45     4     5     6     1
46    16    17    18     3
...
```
How can I achieve this?

Response:
The idiomatic way to do this with Pandas is to use the .sample method of your data frame to sample all rows without replacement:
```py
df.sample(frac=1)
```
The frac keyword argument specifies the fraction of rows to return in the random sample, so frac=1 means to return all rows (in random order). Note: If you wish to shuffle your dataframe in-place and reset the index, you could do e.g.
```py
df = df.sample(frac=1).reset_index(drop=True)
```
Here, specifying drop=True prevents .reset_index from creating a column containing the old index entries. Follow-up note: Although it may not look like the above operation is in-place, python/pandas is smart enough not to do another malloc for the shuffled object. That is, even though the reference object has changed (by which I mean id(df_old) is not the same as id(df_new)), the underlying C object is still the same. To show that this is indeed the case, you could run a simple memory profiler:
```
$ python3 -m memory_profiler .\test.py
Filename: .\test.py

Line #    Mem usage    Increment   Line Contents
================================================
     5     68.5 MiB     68.5 MiB   @profile
     6                             def shuffle():
     7    847.8 MiB    779.3 MiB       df = pd.DataFrame(np.random.randn(100, 1000000))
     8    847.9 MiB      0.1 MiB       df = df.sample(frac=1).reset_index(drop=True)
```


Question:
Truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all() I want to filter my dataframe with an or condition to keep rows with a particular column's values that are outside the range [-0.25, 0.25]. I tried:
```
df = df[(df['col'] < -0.25) or (df['col'] > 0.25)]
```
But I get the error: ValueError: The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all().

Response:
The or and and Python statements require truth-values. For pandas, these are considered ambiguous, so you should use "bitwise" | (or) or & (and) operations:
```
df = df[(df['col'] < -0.25) | (df['col'] > 0.25)]
```
These are overloaded for these kinds of data structures to yield the element-wise or or and. Just to add some more explanation to this statement: The exception is thrown when you want to get the bool of a pandas.Series:
```

```python
import pandas as pd
```

```python
x = pd.Series([1])
```

```python
bool(x)
#Output
#ValueError: The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all().
#```
#You hit a place where the operator implicitly converted the operands to bool (you used or but it also happens for and, if and while):
#```
```

```python
x or x
#Output
#ValueError: The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all().
```

```python
x and x
#Output
#ValueError: The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all().
```

```python
if x:
    print('fun')
#Output
#ValueError: The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all().
```

```python
while x:
    print('fun')
#Output
#ValueError: The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all().
#```
#Besides these four statements, there are several Python functions that hide some bool calls (like any, all, filter, ...). These are normally not problematic with pandas.Series, but for completeness I wanted to mention these. In your case, the exception isn't really helpful, because it doesn't mention the right alternatives. For and and or, if you want element-wise comparisons, you can use: numpy.logical_or:
#```
```

```python
import numpy as np
```

```python
np.logical_or(x, y)
#Output
#```
#or simply the | operator:
#```
```

```python
x | y
#Output
#```
#numpy.logical_and:
#```
```

```python
np.logical_and(x, y)
#Output
#```
#or simply the & operator:
#```
```

```python
x & y
#Output
#```
#If you're using the operators, then be sure to set your parentheses correctly because of operator precedence. There are several logical NumPy functions which should work on pandas.Series. The alternatives mentioned in the Exception are more suited if you encountered it when doing if or while. I'll shortly explain each of these: If you want to check if your Series is empty:
#```
```

```python
x = pd.Series([])
```

```python
x.empty
#Output
#True
```

```python
x = pd.Series([1])
```

```python
x.empty
#Output
#False
#```
#Python normally interprets the length of containers (like list, tuple, ...) as truth-value if it has no explicit Boolean interpretation. So if you want the Python-like check, you could do: if x.size or if not x.empty instead of if x. If your Series contains one and only one Boolean value:
#```
```

```python
x = pd.Series([100])
```

```python
(x > 50).bool()
#Output
#True
```

```python
(x < 50).bool()
#Output
#False
#```
#If you want to check the first and only item of your Series (like .bool(), but it works even for non-Boolean contents):
#```
```

```python
x = pd.Series([100])
```

```python
x.item()
#Output
#100
#```
#If you want to check if all or any item is not-zero, not-empty or not-False:
#```
```

```python
x = pd.Series([0, 1, 2])
```

```python
x.all()   # Because one element is zero
#Output
#False
```

```python
x.any()   # because one (or more) elements are non-zero
#Output
#True
#```
```


Question:
How to filter Pandas dataframe using 'in' and 'not in' like in SQL How can I achieve the equivalents of SQL's IN and NOT IN? I have a list with the required values. Here's the scenario:
```py
df = pd.DataFrame({'country': ['US', 'UK', 'Germany', 'China']})
countries_to_keep = ['UK', 'China']

# pseudo-code:
df[df['country'] not in countries_to_keep]
```
My current way of doing this is as follows:
```py
df = pd.DataFrame({'country': ['US', 'UK', 'Germany', 'China']})
df2 = pd.DataFrame({'country': ['UK', 'China'], 'matched': True})

# IN
df.merge(df2, how='inner', on='country')

# NOT IN
not_in = df.merge(df2, how='left', on='country')
not_in = not_in[pd.isnull(not_in['matched'])]
```
But this seems like a horrible kludge. Can anyone improve on it?

Response:
You can use pd.Series.isin. For "IN" use: something.isin(somewhere) Or for "NOT IN": ~something.isin(somewhere) As a worked example:
```

```python
df
#Output
#    country
#0        US
#1        UK
#2   Germany
#3     China
```

```python
countries_to_keep
#Output
#['UK', 'China']
```

```python
df.country.isin(countries_to_keep)
#Output
#0    False
#1     True
#2    False
#3     True
#Name: country, dtype: bool
```

```python
df[df.country.isin(countries_to_keep)]
#Output
#    country
#1        UK
#3     China
```

```python
df[~df.country.isin(countries_to_keep)]
#Output
#    country
#0        US
#2   Germany
#```
```


Question:
Constructing DataFrame from values in variables yields "ValueError: If using all scalar values, you must pass an index" I have two variables as follows.
```py
a = 2
b = 3
```
I want to construct a DataFrame from this:
```py
df2 = pd.DataFrame({'A':a, 'B':b})
```
This generates an error:
```none
ValueError: If using all scalar values, you must pass an index
```
I tried this also:
```py
df2 = (pd.DataFrame({'a':a, 'b':b})).reset_index()
```
This gives the same error message. How do I do what I want?

Response:
The error message says that if you're passing scalar values, you have to pass an index. So you can either not use scalar values for the columns -- e.g. use a list:
```

```python
df = pd.DataFrame({'A': [a], 'B': [b]})
```

```python
df
#Output
#   A  B
#0  2  3
#```
#or use scalar values and pass an index:
#```
```

```python
df = pd.DataFrame({'A': a, 'B': b}, index=[0, 3])
```

```python
df
#Output
#   A  B
#0  2  3
#3  2  3
#```
```


Question:
Get statistics for each group (such as count, mean, etc) using pandas GroupBy? I have a dataframe df and I use several columns from it to groupby:
```py
df['col1','col2','col3','col4'].groupby(['col1','col2']).mean()
```
In the above way, I almost get the table (dataframe) that I need. What is missing is an additional column that contains number of rows in each group. In other words, I have mean but I also would like to know how many were used to get these means. For example in the first group there are 8 values and in the second one 10 and so on. In short: How do I get group-wise statistics for a dataframe?

Response:
Quick Answer: The simplest way to get row counts per group is by calling .size(), which returns a Series:
```
df.groupby(['col1','col2']).size()
```
Usually you want this result as a DataFrame (instead of a Series) so you can do:
```
df.groupby(['col1', 'col2']).size().reset_index(name='counts')
```
If you want to find out how to calculate the row counts and other statistics for each group continue reading below. Detailed example: Consider the following example dataframe:
```

```python
df
#Output
#col1 col2  col3  col4  col5  col6
#0    A    B  0.20 -0.61 -0.49  1.49
#1    A    B -1.53 -1.01 -0.39  1.82
#2    A    B -0.44  0.27  0.72  0.11
#3    A    B  0.28 -1.32  0.38  0.18
#4    C    D  0.12  0.59  0.81  0.66
#5    C    D -0.13 -1.65 -1.64  0.50
#6    C    D -1.42 -0.11 -0.18 -0.44
#7    E    F -0.00  1.42 -0.26  1.17
#8    E    F  0.91 -0.47  1.35 -0.34
#9    G    H  1.48 -0.63 -1.14  0.17
#```
#First let's use .size() to get the row counts:
#```
```

```python
df.groupby(['col1', 'col2']).size()
#Output
#col1  col2
#A     B       4
#C     D       3
#E     F       2
#G     H       1
#dtype: int64
#```
#Then let's use .size().reset_index(name='counts') to get the row counts:
#```
```

```python
df.groupby(['col1', 'col2']).size().reset_index(name='counts')
#Output
#col1 col2  counts
#0    A    B       4
#1    C    D       3
#2    E    F       2
#3    G    H       1
#```
#Including results for more statistics When you want to calculate statistics on grouped data, it usually looks like this:
#```
```

```python
(df
   ...: .groupby(['col1', 'col2'])
   ...: .agg({
   ...:     'col3': ['mean', 'count'],
   ...:     'col4': ['median', 'min', 'count']
   ...: }))
#Output
#col4                  col3
#          median   min count      mean count
#col1 col2
#A    B    -0.810 -1.32     4 -0.372500     4
#C    D    -0.110 -1.65     3 -0.476667     3
#E    F     0.475 -0.47     2  0.455000     2
#G    H    -0.630 -0.63     1  1.480000     1
#```
#The result above is a little annoying to deal with because of the nested column labels, and also because row counts are on a per column basis. To gain more control over the output I usually split the statistics into individual aggregations that I then combine using join. It looks like this:
#```
```

```python
gb = df.groupby(['col1', 'col2'])
   ...: counts = gb.size().to_frame(name='counts')
   ...: (counts
   ...:  .join(gb.agg({'col3': 'mean'}).rename(columns={'col3': 'col3_mean'}))
   ...:  .join(gb.agg({'col4': 'median'}).rename(columns={'col4': 'col4_median'}))
   ...:  .join(gb.agg({'col4': 'min'}).rename(columns={'col4': 'col4_min'}))
   ...:  .reset_index()
   ...: )
   ...:
#Output
#col1 col2  counts  col3_mean  col4_median  col4_min
#0    A    B       4  -0.372500       -0.810     -1.32
#1    C    D       3  -0.476667       -0.110     -1.65
#2    E    F       2   0.455000        0.475     -0.47
#3    G    H       1   1.480000       -0.630     -0.63
#```
#Footnotes The code used to generate the test data is shown below:
#```
```

```python
import numpy as np
   ...: import pandas as pd
   ...:
   ...: keys = np.array([
   ...:         ['A', 'B'],
   ...:         ['A', 'B'],
   ...:         ['A', 'B'],
   ...:         ['A', 'B'],
   ...:         ['C', 'D'],
   ...:         ['C', 'D'],
   ...:         ['C', 'D'],
   ...:         ['E', 'F'],
   ...:         ['E', 'F'],
   ...:         ['G', 'H']
   ...:         ])
   ...:
   ...: df = pd.DataFrame(
   ...:     np.hstack([keys,np.random.randn(10,4).round(2)]),
   ...:     columns = ['col1', 'col2', 'col3', 'col4', 'col5', 'col6']
   ...: )
   ...:
   ...: df[['col3', 'col4', 'col5', 'col6']] = \
   ...:     df[['col3', 'col4', 'col5', 'col6']].astype(float)
   ...:
```
Disclaimer: If some of the columns that you are aggregating have null values, then you really want to be looking at the group row counts as an independent aggregation for each column. Otherwise you may be misled as to how many records are actually being used to calculate things like the mean because pandas will drop NaN entries in the mean calculation without telling you about it.
```


Question:
Set value for particular cell in pandas DataFrame using index I have created a Pandas DataFrame
```py
df = DataFrame(index=['A','B','C'], columns=['x','y'])
```
Now, I would like to assign a value to particular cell, for example to row C and column x. In other words, I would like to perform the following transformation:
```none
x    y             x    y
A  NaN  NaN        A  NaN  NaN
B  NaN  NaN   ⟶   B  NaN  NaN
C  NaN  NaN        C   10  NaN
```
with this code:
```py
df.xs('C')['x'] = 10
```
However, the contents of df has not changed. The dataframe contains yet again only NaNs. How do I what I want?

Response:
RukTech's answer, df.set_value('C', 'x', 10), is far and away faster than the options I've suggested below. However, it has been slated for deprecation. Going forward, the recommended method is .iat/.at. Why df.xs('C')['x']=10 does not work: df.xs('C') by default, returns a new dataframe with a copy of the data, so
```
df.xs('C')['x']=10
```
modifies this new dataframe only. df['x'] returns a view of the df dataframe, so
```
df['x']['C'] = 10
```
modifies df itself. Warning: It is sometimes difficult to predict if an operation returns a copy or a view. For this reason the docs recommend avoiding assignments with "chained indexing". So the recommended alternative is
```
df.at['C', 'x'] = 10
```
which does modify df.
```

```python
%timeit df.set_value('C', 'x', 10)
100000 loops, best of 3: 2.9 µs per loop
```

```python
%timeit df['x']['C'] = 10
100000 loops, best of 3: 6.31 µs per loop
```

```python
%timeit df.at['C', 'x'] = 10
100000 loops, best of 3: 9.2 µs per loop
```
```


Question:
How to delete rows from a pandas DataFrame based on a conditional expression [duplicate] This question already has answers here: Deleting DataFrame row in Pandas based on column value (20 answers) Closed 5 years ago. I have a pandas DataFrame and I want to delete rows from it where the length of the string in a particular column is greater than 2. I expect to be able to do this (per this answer):
```
df[(len(df['column name']) < 2)]
```
but I just get the error:
```
KeyError: u'no item named False'
```
What am I doing wrong? (Note: I know I can use df.dropna() to get rid of rows that contain any NaN, but I didn't see how to remove rows based on a conditional expression.)

Response:
To directly answer this question's original title "How to delete rows from a pandas DataFrame based on a conditional expression" (which I understand is not necessarily the OP's problem but could help other users coming across this question) one way to do this is to use the drop method:
```
df = df.drop(some labels)
df = df.drop(df[<some boolean condition>].index)
```
Example To remove all rows where column 'score' is < 50:
```
df = df.drop(df[df.score < 50].index)
```
In place version (as pointed out in comments)
```
df.drop(df[df.score < 50].index, inplace=True)
```
Multiple conditions (see Boolean Indexing) The operators are: | for or, & for and, and ~ for not. These must be grouped by using parentheses. To remove all rows where column 'score' is  20
```
df = df.drop(df[(df.score < 50) & (df.score > 20)].index)
```


Question:
How to replace NaN values in a dataframe column I have a Pandas Dataframe as below:
```none
itm Date                  Amount
67    420 2012-09-30 00:00:00   65211
68    421 2012-09-09 00:00:00   29424
69    421 2012-09-16 00:00:00   29877
70    421 2012-09-23 00:00:00   30990
71    421 2012-09-30 00:00:00   61303
72    485 2012-09-09 00:00:00   71781
73    485 2012-09-16 00:00:00     NaN
74    485 2012-09-23 00:00:00   11072
75    485 2012-09-30 00:00:00  113702
76    489 2012-09-09 00:00:00   64731
77    489 2012-09-16 00:00:00     NaN
```
When I try to apply a function to the Amount column, I get the following error:
```none
ValueError: cannot convert float NaN to integer
```
I have tried applying a function using math.isnan, pandas' .replace method, .sparse data attribute from pandas 0.9, if NaN == NaN statement in a function; I have also looked at this Q/A; none of them works. How do I do it?

Response:
DataFrame.fillna() or Series.fillna() will do this for you. Example:
```

```python
df
#Output
#0         1
#0       NaN       NaN
#1 -0.494375  0.570994
#2       NaN       NaN
#3  1.876360 -0.229738
#4       NaN       NaN
```

```python
df.fillna(0)
#Output
#0         1
#0  0.000000  0.000000
#1 -0.494375  0.570994
#2  0.000000  0.000000
#3  1.876360 -0.229738
#4  0.000000  0.000000
#```
#To fill the NaNs in only one column, select just that column.
#```
```

```python
df[1] = df[1].fillna(0)
```

```python
df
#Output
#0         1
#0       NaN  0.000000
#1 -0.494375  0.570994
#2       NaN  0.000000
#3  1.876360 -0.229738
#4       NaN  0.000000
#```
#Or you can use the built in column-specific functionality:
#```
#df = df.fillna({1: 0})
#```
```


Question:
Create new column based on values from other columns / apply a function of multiple columns, row-wise in Pandas I want to apply my custom function (it uses an if-else ladder) to these six columns (ERI_Hispanic, ERI_AmerInd_AKNatv, ERI_Asian, ERI_Black_Afr.Amer, ERI_HI_PacIsl, ERI_White) in each row of my dataframe. I've tried different methods from other questions but still can't seem to find the right answer for my problem. The critical piece of this is that if the person is counted as Hispanic they can't be counted as anything else. Even if they have a "1" in another ethnicity column they still are counted as Hispanic not two or more races. Similarly, if the sum of all the ERI columns is greater than 1 they are counted as two or more races and can't be counted as a unique ethnicity(except for Hispanic). It's almost like doing a for loop through each row and if each record meets a criterion they are added to one list and eliminated from the original. From the dataframe below I need to calculate a new column based on the following spec in SQL: CRITERIA
```
IF [ERI_Hispanic] = 1 THEN RETURN “Hispanic”
ELSE IF SUM([ERI_AmerInd_AKNatv] + [ERI_Asian] + [ERI_Black_Afr.Amer] + [ERI_HI_PacIsl] + [ERI_White]) > 1 THEN RETURN “Two or More”
ELSE IF [ERI_AmerInd_AKNatv] = 1 THEN RETURN “A/I AK Native”
ELSE IF [ERI_Asian] = 1 THEN RETURN “Asian”
ELSE IF [ERI_Black_Afr.Amer] = 1 THEN RETURN “Black/AA”
ELSE IF [ERI_HI_PacIsl] = 1 THEN RETURN “Haw/Pac Isl.”
ELSE IF [ERI_White] = 1 THEN RETURN “White”
```
Comment: If the ERI Flag for Hispanic is True (1), the employee is classified as “Hispanic” Comment: If more than 1 non-Hispanic ERI Flag is true, return “Two or More” DATAFRAME
```
lname          fname       rno_cd  eri_afr_amer    eri_asian   eri_hawaiian    eri_hispanic    eri_nat_amer    eri_white   rno_defined
0    MOST           JEFF        E       0               0           0               0               0               1           White
1    CRUISE         TOM         E       0               0           0               1               0               0           White
2    DEPP           JOHNNY              0               0           0               0               0               1           Unknown
3    DICAP          LEO                 0               0           0               0               0               1           Unknown
4    BRANDO         MARLON      E       0               0           0               0               0               0           White
5    HANKS          TOM         0                       0           0               0               0               1           Unknown
6    DENIRO         ROBERT      E       0               1           0               0               0               1           White
7    PACINO         AL          E       0               0           0               0               0               1           White
8    WILLIAMS       ROBIN       E       0               0           1               0               0               0           White
9    EASTWOOD       CLINT       E       0               0           0               0               0               1           White
```

Response:
OK, two steps to this - first is to write a function that does the translation you want - I've put an example together based on your pseudo-code:
```
def label_race(row):
   if row['eri_hispanic'] == 1:
      return 'Hispanic'
   if row['eri_afr_amer'] + row['eri_asian'] + row['eri_hawaiian'] + row['eri_nat_amer'] + row['eri_white'] > 1:
      return 'Two Or More'
   if row['eri_nat_amer'] == 1:
      return 'A/I AK Native'
   if row['eri_asian'] == 1:
      return 'Asian'
   if row['eri_afr_amer'] == 1:
      return 'Black/AA'
   if row['eri_hawaiian'] == 1:
      return 'Haw/Pac Isl.'
   if row['eri_white'] == 1:
      return 'White'
   return 'Other'
```
You may want to go over this, but it seems to do the trick - notice that the parameter going into the function is considered to be a Series object labelled "row". Next, use the apply function in pandas to apply the function - e.g.
```
df.apply(label_race, axis=1)
```
Note the axis=1 specifier, that means that the application is done at a row, rather than a column level. The results are here:
```
0           White
1        Hispanic
2           White
3           White
4           Other
5           White
6     Two Or More
7           White
8    Haw/Pac Isl.
9           White
```
If you're happy with those results, then run it again, saving the results into a new column in your original dataframe.
```
df['race_label'] = df.apply(label_race, axis=1)
```
The resultant dataframe looks like this (scroll to the right to see the new column):
```
lname   fname rno_cd  eri_afr_amer  eri_asian  eri_hawaiian   eri_hispanic  eri_nat_amer  eri_white rno_defined    race_label
0      MOST    JEFF      E             0          0             0              0             0          1       White         White
1    CRUISE     TOM      E             0          0             0              1             0          0       White      Hispanic
2      DEPP  JOHNNY    NaN             0          0             0              0             0          1     Unknown         White
3     DICAP     LEO    NaN             0          0             0              0             0          1     Unknown         White
4    BRANDO  MARLON      E             0          0             0              0             0          0       White         Other
5     HANKS     TOM    NaN             0          0             0              0             0          1     Unknown         White
6    DENIRO  ROBERT      E             0          1             0              0             0          1       White   Two Or More
7    PACINO      AL      E             0          0             0              0             0          1       White         White
8  WILLIAMS   ROBIN      E             0          0             1              0             0          0       White  Haw/Pac Isl.
9  EASTWOOD   CLINT      E             0          0             0              0             0          1       White         White
```


Question:
How to group dataframe rows into list in pandas groupby Given a dataframe, I want to groupby the first column and get second column as lists in rows, so that a dataframe like:
```
a b
A 1
A 2
B 5
B 5
B 4
C 6
```
becomes
```
A [1,2]
B [5,5,4]
C [6]
```
How do I do this?

Response:
You can do this using groupby to group on the column of interest and then apply list to every group:
```

```python
df = pd.DataFrame( {'a':['A','A','B','B','B','C'], 'b':[1,2,5,5,4,6]})
        df
#Output
#a  b
#0  A  1
#1  A  2
#2  B  5
#3  B  5
#4  B  4
#5  C  6
```

```python
df.groupby('a')['b'].apply(list)
#Output
#a
#A       [1, 2]
#B    [5, 5, 4]
#C          [6]
#Name: b, dtype: object
```

```python
df1 = df.groupby('a')['b'].apply(list).reset_index(name='new')
        df1
#Output
#a        new
#0  A     [1, 2]
#1  B  [5, 5, 4]
#2  C        [6]
#```
```


Question:
Get list from pandas dataframe column or row? I have a dataframe df imported from an Excel document like this:
```none
cluster load_date   budget  actual  fixed_price
A   1/1/2014    1000    4000    Y
A   2/1/2014    12000   10000   Y
A   3/1/2014    36000   2000    Y
B   4/1/2014    15000   10000   N
B   4/1/2014    12000   11500   N
B   4/1/2014    90000   11000   N
C   7/1/2014    22000   18000   N
C   8/1/2014    30000   28960   N
C   9/1/2014    53000   51200   N
```
I want to be able to return the contents of column 1 df['cluster'] as a list, so I can run a for-loop over it, and create an Excel worksheet for every cluster. Is it also possible to return the contents of a whole column or row to a list? e.g.
```py
list = [], list[column1] or list[df.ix(row1)]
```

Response:
Pandas DataFrame columns are Pandas Series when you pull them out, which you can then call x.tolist() on to turn them into a Python list. Alternatively you cast it with list(x).
```py
import pandas as pd

data_dict = {'one': pd.Series([1, 2, 3], index=['a', 'b', 'c']),
             'two': pd.Series([1, 2, 3, 4], index=['a', 'b', 'c', 'd'])}

df = pd.DataFrame(data_dict)

print(f"DataFrame:\n{df}\n")
print(f"column types:\n{df.dtypes}")

col_one_list = df['one'].tolist()

col_one_arr = df['one'].to_numpy()

print(f"\ncol_one_list:\n{col_one_list}\ntype:{type(col_one_list)}")
print(f"\ncol_one_arr:\n{col_one_arr}\ntype:{type(col_one_arr)}")
```
Output:
```none
DataFrame:
   one  two
a  1.0    1
b  2.0    2
c  3.0    3
d  NaN    4

column types:
one    float64
two      int64
dtype: object

col_one_list:
[1.0, 2.0, 3.0, nan]
type:<class 'list'>

col_one_arr:
[ 1.  2.  3. nan]
type:<class 'numpy.ndarray'>
```


Question:
Python Pandas: Get index of rows where column matches certain value Given a DataFrame with a column "BoolCol", we want to find the indexes of the DataFrame in which the values for "BoolCol" == True I currently have the iterating way to do it, which works perfectly:
```py
for i in range(100,3000):
    if df.iloc[i]['BoolCol']== True:
         print i,df.iloc[i]['BoolCol']
```
But this is not the correct pandas way to do it. After some research, I am currently using this code:
```py
df[df['BoolCol'] == True].index.tolist()
```
This one gives me a list of indexes, but they don't match, when I check them by doing:
```py
df.iloc[i]['BoolCol']
```
The result is actually False!! Which would be the correct pandas way to do this?

Response:
df.iloc[i] returns the ith row of df. i does not refer to the index label, i is a 0-based index. In contrast, the attribute index returns actual index labels, not numeric row-indices:
```
df.index[df['BoolCol'] == True].tolist()
```
or equivalently,
```
df.index[df['BoolCol']].tolist()
```
You can see the difference quite clearly by playing with a DataFrame with a non-default index that does not equal to the row's numerical position:
```
df = pd.DataFrame({'BoolCol': [True, False, False, True, True]},
       index=[10,20,30,40,50])

```python
df
#Output
#BoolCol
#10    True
#20   False
#30   False
#40    True
#50    True
#
#[5 rows x 1 columns]
```

```python
df.index[df['BoolCol']].tolist()
#Output
#[10, 40, 50]
#```
#If you want to use the index,
#```
```

```python
idx = df.index[df['BoolCol']]
```

```python
idx
#Output
#Int64Index([10, 40, 50], dtype='int64')
#```
#then you can select the rows using loc instead of iloc:
#```
```

```python
df.loc[idx]
#Output
#BoolCol
#10    True
#40    True
#50    True
#
#[3 rows x 1 columns]
#```
#Note that loc can also accept boolean arrays:
#```
```

```python
df.loc[df['BoolCol']]
#Output
#BoolCol
#10    True
#40    True
#50    True
#
#[3 rows x 1 columns]
#```
#If you have a boolean array, mask, and need ordinal index values, you can compute them using np.flatnonzero:
#```
```

```python
np.flatnonzero(df['BoolCol'])
#Output
#array([0, 3, 4])
#```
#Use df.iloc to select rows by ordinal index:
#```
```

```python
df.iloc[np.flatnonzero(df['BoolCol'])]
#Output
#BoolCol
#10    True
#40    True
#50    True
#```
```


Question: How to sort a pandas dataFrame by two or more columns? Suppose I have a dataframe with columns a, b and c. I want to sort the dataframe by column b in ascending order, and by column c in descending order. How do I do this?

Response:
As of the 0.17.0 release, the sort method was deprecated in favor of sort_values. sort was completely removed in the 0.20.0 release. The arguments (and results) remain the same:
```
df.sort_values(['a', 'b'], ascending=[True, False])
```
You can use the ascending argument of sort:
```
df.sort(['a', 'b'], ascending=[True, False])
```
For example:
```

```python
df1 = pd.DataFrame(np.random.randint(1, 5, (10,2)), columns=['a','b'])
```

```python
df1.sort(['a', 'b'], ascending=[True, False])
#Output
#a  b
#2  1  4
#7  1  3
#1  1  2
#3  1  2
#4  3  2
#6  4  4
#0  4  3
#9  4  3
#5  4  1
#8  4  1
#```
#As commented by @renadeen Sort isn't in place by default! So you should assign result of the sort method to a variable or add inplace=True to method call. that is, if you want to reuse df1 as a sorted DataFrame:
#```
#df1 = df1.sort(['a', 'b'], ascending=[True, False])
#```
#or
#```
#df1.sort(['a', 'b'], ascending=[True, False], inplace=True)
#```
```


Question:
Sorting columns in pandas dataframe based on column name I have a dataframe with over 200 columns. The issue is as they were generated the order is
```
['Q1.3','Q6.1','Q1.2','Q1.1',......]
```
I need to sort the columns as follows:
```
['Q1.1','Q1.2','Q1.3',.....'Q6.1',......]
```
Is there some way for me to do this within Python?

Response:
```
df = df.reindex(sorted(df.columns), axis=1)
```
This assumes that sorting the column names will give the order you want. If your column names won't sort lexicographically (e.g., if you want column Q10.3 to appear after Q9.1), you'll need to sort differently, but that has nothing to do with pandas.


Question:
Count the frequency that a value occurs in a dataframe column I have a dataset
```none
category
cat a
cat b
cat a
```
I'd like to return something like the following which shows the unique values and their frequencies
```none
category   freq
cat a       2
cat b       1
```

Response:
Use value_counts() as @DSM commented.
```

```python
df = pd.DataFrame({'a':list('abssbab')})
df['a'].value_counts()
#Output
#b    3
#a    2
#s    2
#dtype: int64
#```
#Also groupby and count. Many ways to skin a cat here.
#```
```

```python
df.groupby('a').count()
#Output
#a
#a
#a  2
#b  3
#s  2
#
#[3 rows x 1 columns]
#```
#See the online docs. If you wanted to add frequency back to the original dataframe use transform to return an aligned index:
#```
```

```python
df['freq'] = df.groupby('a')['a'].transform('count')
df
#Output
#a freq
#0  a    2
#1  b    3
#2  s    2
#3  s    2
#4  b    3
#5  a    2
#6  b    3
#
#[7 rows x 2 columns]
#```
```


Question:
What does `ValueError: cannot reindex from a duplicate axis` mean? I am getting a ValueError: cannot reindex from a duplicate axis when I am trying to set an index to a certain value. I tried to reproduce this with a simple example, but I could not do it. Here is my session inside of ipdb trace. I have a DataFrame with string index, and integer columns, float values. However when I try to create sum index for sum of all columns I am getting ValueError: cannot reindex from a duplicate axis error. I created a small DataFrame with the same characteristics, but was not able to reproduce the problem, what could I be missing? I don't really understand what ValueError: cannot reindex from a duplicate axismeans, what does this error message mean? Maybe this will help me diagnose the problem, and this is most answerable part of my question.
```
ipdb> type(affinity_matrix)
<class 'pandas.core.frame.DataFrame'>
ipdb> affinity_matrix.shape
(333, 10)
ipdb> affinity_matrix.columns
Int64Index([9315684, 9315597, 9316591, 9320520, 9321163, 9320615, 9321187, 9319487, 9319467, 9320484], dtype='int64')
ipdb> affinity_matrix.index
Index([u'001', u'002', u'003', u'004', u'005', u'008', u'009', u'010', u'011', u'014', u'015', u'016', u'018', u'020', u'021', u'022', u'024', u'025', u'026', u'027', u'028', u'029', u'030', u'032', u'033', u'034', u'035', u'036', u'039', u'040', u'041', u'042', u'043', u'044', u'045', u'047', u'047', u'048', u'050', u'053', u'054', u'055', u'056', u'057', u'058', u'059', u'060', u'061', u'062', u'063', u'065', u'067', u'068', u'069', u'070', u'071', u'072', u'073', u'074', u'075', u'076', u'077', u'078', u'080', u'082', u'083', u'084', u'085', u'086', u'089', u'090', u'091', u'092', u'093', u'094', u'095', u'096', u'097', u'098', u'100', u'101', u'103', u'104', u'105', u'106', u'107', u'108', u'109', u'110', u'111', u'112', u'113', u'114', u'115', u'116', u'117', u'118', u'119', u'121', u'122', ...], dtype='object')

ipdb> affinity_matrix.values.dtype
dtype('float64')
ipdb> 'sums' in affinity_matrix.index
False
```
Here is the error:
```
ipdb> affinity_matrix.loc['sums'] = affinity_matrix.sum(axis=0)
*** ValueError: cannot reindex from a duplicate axis
```
I tried to reproduce this with a simple example, but I failed
```

```python
import pandas as pd
```

```python
import numpy as np
```

```python
a = np.arange(35).reshape(5,7)
```

```python
df = pd.DataFrame(a, ['x', 'y', 'u', 'z', 'w'], range(10, 17))
```

```python
df.values.dtype
#Output
#dtype('int64')
```

```python
df.loc['sums'] = df.sum(axis=0)
```

```python
df
#Output
#10  11  12  13  14  15   16
#x      0   1   2   3   4   5    6
#y      7   8   9  10  11  12   13
#u     14  15  16  17  18  19   20
#z     21  22  23  24  25  26   27
#w     28  29  30  31  32  33   34
#sums  70  75  80  85  90  95  100
#```
```

Response: This error usually rises when you join / assign to a column when the index has duplicate values. Since you are assigning to a row, I suspect that there is a duplicate value in affinity_matrix.columns, perhaps not shown in your question.


Question: How can I display full (non-truncated) dataframe information in HTML when converting from Pandas dataframe to HTML? I converted a Pandas dataframe to an HTML output using the DataFrame.to_html function. When I save this to a separate HTML file, the file shows truncated output. For example, in my TEXT column, df.head(1) will show The film was an excellent effort... instead of The film was an excellent effort in deconstructing the complex social sentiments that prevailed during this period. This rendition is fine in the case of a screen-friendly format of a massive Pandas dataframe, but I need an HTML file that will show complete tabular data contained in the dataframe, that is, something that will show the latter text element rather than the former text snippet. How would I be able to show the complete, non-truncated text data for each element in my TEXT column in the HTML version of the information? I would imagine that the HTML table would have to display long cells to show the complete data, but as far as I understand, only column-width parameters can be passed into the DataFrame.to_html function.

Response:
Set the display.max_colwidth option to None (or -1 before version 1.0):
```
pd.set_option('display.max_colwidth', None)
```
set_option documentation For example, in IPython, we see that the information is truncated to 50 characters. Anything in excess is ellipsized: If you set the display.max_colwidth option, the information will be displayed fully:


Question:
How do I create a new column where the values are selected based on an existing column? How do I add a color column to the following dataframe so that color='green' if Set == 'Z', and color='red' otherwise?
```
Type  Set
1     A    Z
2     B    Z
3     B    X
4     C    Y
```

Response:
If you only have two choices to select from then use np.where:
```
df['color'] = np.where(df['Set']=='Z', 'green', 'red')
```
For example,
```
import pandas as pd
import numpy as np

df = pd.DataFrame({'Type':list('ABBC'), 'Set':list('ZZXY')})
df['color'] = np.where(df['Set']=='Z', 'green', 'red')
print(df)
```
yields
```
Set Type  color
0   Z    A  green
1   Z    B  green
2   X    B    red
3   Y    C    red
```
If you have more than two conditions then use np.select. For example, if you want color to be yellow when (df['Set'] == 'Z') & (df['Type'] == 'A') otherwise blue when (df['Set'] == 'Z') & (df['Type'] == 'B') otherwise purple when (df['Type'] == 'B') otherwise black, then use
```
df = pd.DataFrame({'Type':list('ABBC'), 'Set':list('ZZXY')})
conditions = [
    (df['Set'] == 'Z') & (df['Type'] == 'A'),
    (df['Set'] == 'Z') & (df['Type'] == 'B'),
    (df['Type'] == 'B')]
choices = ['yellow', 'blue', 'purple']
df['color'] = np.select(conditions, choices, default='black')
print(df)
```
which yields
```
Set Type   color
0   Z    A  yellow
1   Z    B    blue
2   X    B  purple
3   Y    C   black
```


Question:
Get the row(s) which have the max value in groups using groupby How do I find all rows in a pandas DataFrame which have the max value for count column, after grouping by ['Sp','Mt'] columns? Example 1: the following DataFrame:
```
Sp   Mt Value   count
0  MM1  S1   a     **3**
1  MM1  S1   n       2
2  MM1  S3   cb    **5**
3  MM2  S3   mk    **8**
4  MM2  S4   bg    **10**
5  MM2  S4   dgd     1
6  MM4  S2   rd      2
7  MM4  S2   cb      2
8  MM4  S2   uyi   **7**
```
Expected output is to get the result rows whose count is max in each group, like this:
```
Sp   Mt   Value  count
0  MM1  S1   a      **3**
2  MM1  S3   cb     **5**
3  MM2  S3   mk     **8**
4  MM2  S4   bg     **10**
8  MM4  S2   uyi    **7**
```
Example 2:
```
Sp   Mt   Value  count
4  MM2  S4   bg     10
5  MM2  S4   dgd    1
6  MM4  S2   rd     2
7  MM4  S2   cb     8
8  MM4  S2   uyi    8
```
Expected output:
```
Sp   Mt   Value  count
4  MM2  S4   bg     10
7  MM4  S2   cb     8
8  MM4  S2   uyi    8
```

Response:
Firstly, we can get the max count for each group like this:
```

```python
df
#Output
#Sp  Mt Value  count
#0  MM1  S1     a      3
#1  MM1  S1     n      2
#2  MM1  S3    cb      5
#3  MM2  S3    mk      8
#4  MM2  S4    bg     10
#5  MM2  S4   dgd      1
#6  MM4  S2    rd      2
#7  MM4  S2    cb      2
#8  MM4  S2   uyi      7
```

```python
df.groupby(['Sp', 'Mt'])['count'].max()
#Output
#Sp   Mt
#MM1  S1     3
#     S3     5
#MM2  S3     8
#     S4    10
#MM4  S2     7
#Name: count, dtype: int64
#```
#To get the indices of the original DF you can do:
#```
```

```python
idx = df.groupby(['Sp', 'Mt'])['count'].transform(max) == df['count']
```

```python
df[idx]
#Output
#Sp  Mt Value  count
#0  MM1  S1     a      3
#2  MM1  S3    cb      5
#3  MM2  S3    mk      8
#4  MM2  S4    bg     10
#8  MM4  S2   uyi      7
#```
#Note that if you have multiple max values per group, all will be returned. Update On a Hail Mary chance that this is what the OP is requesting:
#```
```

```python
df['count_max'] = df.groupby(['Sp', 'Mt'])['count'].transform(max)
```

```python
df
#Output
#Sp  Mt Value  count  count_max
#0  MM1  S1     a      3          3
#1  MM1  S1     n      2          3
#2  MM1  S3    cb      5          5
#3  MM2  S3    mk      8          8
#4  MM2  S4    bg     10         10
#5  MM2  S4   dgd      1         10
#6  MM4  S2    rd      2          7
#7  MM4  S2    cb      2          7
#8  MM4  S2   uyi      7          7
#```
```


Question:
Create Pandas DataFrame from a string In order to test some functionality I would like to create a DataFrame from a string. Let's say my test data looks like:
```
TESTDATA="""col1;col2;col3
1;4.4;99
2;4.5;200
3;4.7;65
4;3.2;140
"""
```
What is the simplest way to read that data into a Pandas DataFrame?

Response:
A simple way to do this is to use StringIO.StringIO (python2) or io.StringIO (python3) and pass that to the pandas.read_csv function. E.g:
```
import sys
if sys.version_info[0] < 3:
    from StringIO import StringIO
else:
    from io import StringIO

import pandas as pd

TESTDATA = StringIO("""col1;col2;col3
    1;4.4;99
    2;4.5;200
    3;4.7;65
    4;3.2;140
    """)

df = pd.read_csv(TESTDATA, sep=";")
```


Question: Convert DataFrame column type from string to datetime How can I convert a DataFrame column of strings (in dd/mm/yyyy format) to datetime dtype?

Response:
The easiest way is to use to_datetime:
```
df['col'] = pd.to_datetime(df['col'])
```
It also offers a dayfirst argument for European times (but beware this isn't strict). Here it is in action:
```

```python
pd.to_datetime(pd.Series(['05/23/2005']))
#Output
#0   2005-05-23 00:00:00
#dtype: datetime64[ns]
#```
#You can pass a specific format:
#```
```

```python
pd.to_datetime(pd.Series(['05/23/2005']), format="%m/%d/%Y")
#Output
#0   2005-05-23
#dtype: datetime64[ns]
#```
```


Question:
Convert a Pandas DataFrame to a dictionary I have a DataFrame with four columns. I want to convert this DataFrame to a python dictionary. I want the elements of first column be keys and the elements of other columns in the same row be values. DataFrame:
```py
ID   A   B   C
0   p    1   3   2
1   q    4   3   2
2   r    4   0   9
```
Output should be like this:
```py
{'p': [1,3,2], 'q': [4,3,2], 'r': [4,0,9]}
```

Response:
The to_dict() method sets the column names as dictionary keys so you'll need to reshape your DataFrame slightly. Setting the 'ID' column as the index and then transposing the DataFrame is one way to achieve this. to_dict() also accepts an 'orient' argument which you'll need in order to output a list of values for each column. Otherwise, a dictionary of the form {index: value} will be returned for each column. These steps can be done with the following line:
```

```python
df.set_index('ID').T.to_dict('list')
#Output
#{'p': [1, 3, 2], 'q': [4, 3, 2], 'r': [4, 0, 9]}
#```
#In case a different dictionary format is needed, here are examples of the possible orient arguments. Consider the following simple DataFrame:
#```
```

```python
df = pd.DataFrame({'a': ['red', 'yellow', 'blue'], 'b': [0.5, 0.25, 0.125]})
```

```python
df
#Output
#        a      b
#0     red  0.500
#1  yellow  0.250
#2    blue  0.125
#```
#Then the options are as follows. dict - the default: column names are keys, values are dictionaries of index:data pairs
#```
```

```python
df.to_dict('dict')
#Output
#{'a': {0: 'red', 1: 'yellow', 2: 'blue'},
# 'b': {0: 0.5, 1: 0.25, 2: 0.125}}
#```
#list - keys are column names, values are lists of column data
#```
```

```python
df.to_dict('list')
#Output
#{'a': ['red', 'yellow', 'blue'],
# 'b': [0.5, 0.25, 0.125]}
#```
#series - like 'list', but values are Series
#```
```

```python
df.to_dict('series')
#Output
#{'a': 0       red
#      1    yellow
#      2      blue
#      Name: a, dtype: object,
```

'b': 0    0.500
      1    0.250
      2    0.125
      Name: b, dtype: float64}
```
split - splits columns/data/index as keys with values being column names, data values by row and index labels respectively
```

```python
df.to_dict('split')
#Output
#{'columns': ['a', 'b'],
# 'data': [['red', 0.5], ['yellow', 0.25], ['blue', 0.125]],
# 'index': [0, 1, 2]}
#```
#records - each row becomes a dictionary where key is column name and value is the data in the cell
#```
```

```python
df.to_dict('records')
#Output
#[{'a': 'red', 'b': 0.5},
# {'a': 'yellow', 'b': 0.25},
# {'a': 'blue', 'b': 0.125}]
#```
#index - like 'records', but a dictionary of dictionaries with keys as index labels (rather than a list)
#```
```

```python
df.to_dict('index')
#Output
#{0: {'a': 'red', 'b': 0.5},
# 1: {'a': 'yellow', 'b': 0.25},
# 2: {'a': 'blue', 'b': 0.125}}
#```
```


Question:
pandas: filter rows of DataFrame with operator chaining Most operations in pandas can be accomplished with operator chaining (groupby, aggregate, apply, etc), but the only way I've found to filter rows is via normal bracket indexing
```
df_filtered = df[df['column'] == value]
```
This is unappealing as it requires I assign df to a variable before being able to filter on its values. Is there something more like the following?
```
df_filtered = df.mask(lambda x: x['column'] == value)
```

Response:
I'm not entirely sure what you want, and your last line of code does not help either, but anyway: "Chained" filtering is done by "chaining" the criteria in the boolean index.
```

```python
df
#Output
#A  B  C  D
#a  1  4  9  1
#b  4  5  0  2
#c  5  5  1  0
#d  1  3  9  6
```

```python
df[(df.A == 1) & (df.D == 6)]
#Output
#A  B  C  D
#d  1  3  9  6
#```
#If you want to chain methods, you can add your own mask method and use that one.
#```
```

```python
def mask(df, key, value):
   ....:     return df[df[key] == value]
   ....:
```

```python
pandas.DataFrame.mask = mask
```

```python
df = pandas.DataFrame(np.random.randint(0, 10, (4,4)), index=list('abcd'), columns=list('ABCD'))
```

```python
df.ix['d','A'] = df.ix['a', 'A']
```

```python
df
#Output
#A  B  C  D
#a  1  4  9  1
#b  4  5  0  2
#c  5  5  1  0
#d  1  3  9  6
```

```python
df.mask('A', 1)
#Output
#A  B  C  D
#a  1  4  9  1
#d  1  3  9  6
```

```python
df.mask('A', 1).mask('D', 6)
#Output
#A  B  C  D
#d  1  3  9  6
#```
```


Question:
Apply multiple functions to multiple groupby columns The docs show how to apply multiple functions on a groupby object at a time using a dict with the output column names as the keys:
```

```python
grouped['D'].agg({'result1' : np.sum,
   .....:                   'result2' : np.mean})
   .....:
#Output
#result2   result1
#A
#bar -0.579846 -1.739537
#foo -0.280588 -1.402938
#```
#However, this only works on a Series groupby object. And when a dict is similarly passed to a groupby DataFrame, it expects the keys to be the column names that the function will be applied to. What I want to do is apply multiple functions to several columns (but certain columns will be operated on multiple times). Also, some functions will depend on other columns in the groupby object (like sumif functions). My current solution is to go column by column, and doing something like the code above, using lambdas for functions that depend on other rows. But this is taking a long time, (I think it takes a long time to iterate through a groupby object). I'll have to change it so that I iterate through the whole groupby object in a single run, but I'm wondering if there's a built in way in pandas to do this somewhat cleanly. For example, I've tried something like
#```
#grouped.agg({'C_sum' : lambda x: x['C'].sum(),
#             'C_std': lambda x: x['C'].std(),
#             'D_sum' : lambda x: x['D'].sum()},
#             'D_sumifC3': lambda x: x['D'][x['C'] == 3].sum(), ...)
#```
#but as expected I get a KeyError (since the keys have to be a column if agg is called from a DataFrame). Is there any built in way to do what I'd like to do, or a possibility that this functionality may be added, or will I just need to iterate through the groupby manually?
```

Response:
The second half of the currently accepted answer is outdated and has two deprecations. First and most important, you can no longer pass a dictionary of dictionaries to the agg groupby method. Second, never use .ix. If you desire to work with two separate columns at the same time I would suggest using the apply method which implicitly passes a DataFrame to the applied function. Let's use a similar dataframe as the one from above
```
df = pd.DataFrame(np.random.rand(4,4), columns=list('abcd'))
df['group'] = [0, 0, 1, 1]
df

          a         b         c         d  group
0  0.418500  0.030955  0.874869  0.145641      0
1  0.446069  0.901153  0.095052  0.487040      0
2  0.843026  0.936169  0.926090  0.041722      1
3  0.635846  0.439175  0.828787  0.714123      1
```
A dictionary mapped from column names to aggregation functions is still a perfectly good way to perform an aggregation.
```
df.groupby('group').agg({'a':['sum', 'max'],
                         'b':'mean',
                         'c':'sum',
                         'd': lambda x: x.max() - x.min()})

              a                   b         c         d
            sum       max      mean       sum  <lambda>
group
0      0.864569  0.446069  0.466054  0.969921  0.341399
1      1.478872  0.843026  0.687672  1.754877  0.672401
```
If you don't like that ugly lambda column name, you can use a normal function and supply a custom name to the special __name__ attribute like this:
```
def max_min(x):
    return x.max() - x.min()

max_min.__name__ = 'Max minus Min'

df.groupby('group').agg({'a':['sum', 'max'],
                         'b':'mean',
                         'c':'sum',
                         'd': max_min})

              a                   b         c             d
            sum       max      mean       sum Max minus Min
group
0      0.864569  0.446069  0.466054  0.969921      0.341399
1      1.478872  0.843026  0.687672  1.754877      0.672401
```
Using apply and returning a Series Now, if you had multiple columns that needed to interact together then you cannot use agg, which implicitly passes a Series to the aggregating function. When using apply the entire group as a DataFrame gets passed into the function. I recommend making a single custom function that returns a Series of all the aggregations. Use the Series index as labels for the new columns:
```
def f(x):
    d = {}
    d['a_sum'] = x['a'].sum()
    d['a_max'] = x['a'].max()
    d['b_mean'] = x['b'].mean()
    d['c_d_prodsum'] = (x['c'] * x['d']).sum()
    return pd.Series(d, index=['a_sum', 'a_max', 'b_mean', 'c_d_prodsum'])

df.groupby('group').apply(f)

         a_sum     a_max    b_mean  c_d_prodsum
group
0      0.864569  0.446069  0.466054     0.173711
1      1.478872  0.843026  0.687672     0.630494
```
If you are in love with MultiIndexes, you can still return a Series with one like this:
```
def f_mi(x):
        d = []
        d.append(x['a'].sum())
        d.append(x['a'].max())
        d.append(x['b'].mean())
        d.append((x['c'] * x['d']).sum())
        return pd.Series(d, index=[['a', 'a', 'b', 'c_d'],
                                   ['sum', 'max', 'mean', 'prodsum']])

df.groupby('group').apply(f_mi)

              a                   b       c_d
            sum       max      mean   prodsum
group
0      0.864569  0.446069  0.466054  0.173711
1      1.478872  0.843026  0.687672  0.630494
```


Question:
how do I insert a column at a specific column index in pandas? Can I insert a column at a specific column index in pandas?
```
import pandas as pd
df = pd.DataFrame({'l':['a','b','c','d'], 'v':[1,2,1,2]})
df['n'] = 0
```
This will put column n as the last column of df, but isn't there a way to tell df to put n at the beginning?

Response:
see docs: http://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.insert.html using loc = 0 will insert at the beginning
```
df.insert(loc, column, value)
```
```
df = pd.DataFrame({'B': [1, 2, 3], 'C': [4, 5, 6]})

df
Out:
   B  C
0  1  4
1  2  5
2  3  6

idx = 0
new_col = [7, 8, 9]  # can be a list, a Series, an array or a scalar
df.insert(loc=idx, column='A', value=new_col)

df
Out:
   A  B  C
0  7  1  4
1  8  2  5
2  9  3  6
```


Question: Combining two Series into a DataFrame in pandas I have two Series s1 and s2 with the same (non-consecutive) indices. How do I combine s1 and s2 to being two columns in a DataFrame and keep one of the indices as a third column?

Response:
I think concat is a nice way to do this. If they are present it uses the name attributes of the Series as the columns (otherwise it simply numbers them):
```

```python
s1 = pd.Series([1, 2], index=['A', 'B'], name='s1')
```

```python
s2 = pd.Series([3, 4], index=['A', 'B'], name='s2')
```

```python
pd.concat([s1, s2], axis=1)
#Output
#s1  s2
#A   1   3
#B   2   4
```

```python
pd.concat([s1, s2], axis=1).reset_index()
#Output
#index  s1  s2
#0     A   1   3
#1     B   2   4
#```
#Note: This extends to more than 2 Series.
```


Question:
Get column index from column name in python pandas In R when you need to retrieve a column index based on the name of the column you could do
```
idx <- which(names(my_data)==my_colum_name)
```
Is there a way to do the same with pandas dataframes?

Response:
Sure, you can use .get_loc():
```

```python
df = DataFrame({"pear": [1,2,3], "apple": [2,3,4], "orange": [3,4,5]})
```

```python
df.columns
#Output
#Index([apple, orange, pear], dtype=object)
```

```python
df.columns.get_loc("pear")
#Output
#2
#```
#although to be honest I don't often need this myself. Usually access by name does what I want it to (df["pear"], df[["apple", "orange"]], or maybe df.columns.isin(["orange", "pear"])), although I can definitely see cases where you'd want the index number.
```


Question:
Add column to dataframe with constant value I have an existing dataframe which I need to add an additional column to which will contain the same value for every row. Existing df:
```
Date, Open, High, Low, Close
01-01-2015, 565, 600, 400, 450
```
New df:
```
Name, Date, Open, High, Low, Close
abc, 01-01-2015, 565, 600, 400, 450
```
I know how to append an existing series / dataframe column. But this is a different situation, because all I need is to add the 'Name' column and set every row to the same value, in this case 'abc'.

Response:
df['Name']='abc' will add the new column and set all rows to that value:
```

```python
df
#Output
#Date, Open, High,  Low,  Close
#0  01-01-2015,  565,  600,  400,    450
```

```python
df['Name'] = 'abc'
df
#Output
#Date, Open, High,  Low,  Close Name
#0  01-01-2015,  565,  600,  400,    450  abc
#```
```