Question:
Python Pandas: Is Order Preserved When Using groupby() and agg()? I've frequented used pandas' agg() function to run summary statistics on every column of a data.frame. For example, here's how you would produce the mean and standard deviation:
```
df = pd.DataFrame({'A': ['group1', 'group1', 'group2', 'group2', 'group3', 'group3'],
                   'B': [10, 12, 10, 25, 10, 12],
                   'C': [100, 102, 100, 250, 100, 102]})

```python
df
#Output
#[output]
#        A   B    C
#0  group1  10  100
#1  group1  12  102
#2  group2  10  100
#3  group2  25  250
#4  group3  10  100
#5  group3  12  102
#```
#In both of those cases, the order that individual rows are sent to the agg function does not matter. But consider the following example, which:
#```
#df.groupby('A').agg([np.mean, lambda x: x.iloc[1] ])
```

[output]

        mean  <lambda>  mean  <lambda>
A
group1  11.0        12   101       102
group2  17.5        25   175       250
group3  11.0        12   101       102
```
In this case the lambda functions as intended, outputting the second row in each group. However, I have not been able to find anything in the pandas documentation that implies that this is guaranteed to be true in all cases. I want use agg() along with a weighted average function, so I want to be sure that the rows that come into the function will be in the same order as they appear in the original data frame. Does anyone know, ideally via somewhere in the docs or pandas source code, if this is guaranteed to be the case?

Response:
See this enhancement issue The short answer is yes, the groupby will preserve the orderings as passed in. You can prove this by using your example like this:
```

```python
df.sort_index(ascending=False).groupby('A').agg([np.mean, lambda x: x.iloc[1] ])
#Output
#B             C
#        mean <lambda> mean <lambda>
#A
#group1  11.0       10  101      100
#group2  17.5       10  175      100
#group3  11.0       10  101      100
#```
#This is NOT true for resample however as it requires a monotonic index (it WILL work with a non-monotonic index, but will sort it first). Their is a sort= flag to groupby, but this relates to the sorting of the groups themselves and not the observations within a group. FYI: df.groupby('A').nth(1) is a safe way to get the 2nd value of a group (as your method above will fail if a group has < 2 elements)
```


Question:
pandas apply function that returns multiple values to rows in pandas dataframe I have a dataframe with a timeindex and 3 columns containing the coordinates of a 3D vector:
```
x             y             z
ts
2014-05-15 10:38         0.120117      0.987305      0.116211
2014-05-15 10:39         0.117188      0.984375      0.122070
2014-05-15 10:40         0.119141      0.987305      0.119141
2014-05-15 10:41         0.116211      0.984375      0.120117
2014-05-15 10:42         0.119141      0.983398      0.118164
```
I would like to apply a transformation to each row that also returns a vector
```
def myfunc(a, b, c):
    do something
    return e, f, g
```
but if I do:
```
df.apply(myfunc, axis=1)
```
I end up with a Pandas series whose elements are tuples. This is beacause apply will take the result of myfunc without unpacking it. How can I change myfunc so that I obtain a new df with 3 columns? Edit: All solutions below work. The Series solution does allow for column names, the List solution seem to execute faster.
```
def myfunc1(args):
    e=args[0] + 2*args[1]
    f=args[1]*args[2] +1
    g=args[2] + args[0] * args[1]
    return pd.Series([e,f,g], index=['a', 'b', 'c'])

def myfunc2(args):
    e=args[0] + 2*args[1]
    f=args[1]*args[2] +1
    g=args[2] + args[0] * args[1]
    return [e,f,g]

%timeit df.apply(myfunc1 ,axis=1)

100 loops, best of 3: 4.51 ms per loop

%timeit df.apply(myfunc2 ,axis=1)

100 loops, best of 3: 2.75 ms per loop
```

Response:
Return Series and it will put them in a DataFrame.
```
def myfunc(a, b, c):
    do something
    return pd.Series([e, f, g])
```
This has the bonus that you can give labels to each of the resulting columns. If you return a DataFrame it just inserts multiple rows for the group.


Question:
Pandas Use Value if Not Null, Else Use Value From Next Column Given the following dataframe:
```
import pandas as pd
df = pd.DataFrame({'COL1': ['A', np.nan,'A'],
                   'COL2' : [np.nan,'A','A']})
df
    COL1    COL2
0    A      NaN
1    NaN    A
2    A      A
```
I would like to create a column ('COL3') that uses the value from COL1 per row unless that value is null (or NaN). If the value is null (or NaN), I'd like for it to use the value from COL2. The desired result is:
```
COL1    COL2   COL3
0    A      NaN    A
1    NaN    A      A
2    A      A      A
```
Thanks in advance!

Response:
```

```python
df
#Output
#COL1 COL2
#0    A  NaN
#1  NaN    B
#2    A    B
```

```python
df["COL3"] = df["COL1"].fillna(df["COL2"])
```

```python
df
#Output
#COL1 COL2 COL3
#0    A  NaN    A
#1  NaN    B    B
#2    A    B    A
#```
```


Question:
Divide multiple columns by another column in pandas I need to divide all but the first columns in a DataFrame by the first column. Here's what I'm doing, but I wonder if this isn't the "right" pandas way:
```
df = pd.DataFrame(np.random.rand(10,3), columns=list('ABC'))

df[['B', 'C']] = (df.T.iloc[1:] / df.T.iloc[0]).T
```
Is there a way to do something like df[['B','C']] / df['A']? (That just gives a 10x12 dataframe of nan.) Also, after reading some similar questions on SO, I tried df['A'].div(df[['B', 'C']]) but that gives a broadcast error.

Response: I believe df[['B','C']].div(df.A, axis=0) and df.iloc[:,1:].div(df.A, axis=0) work.


Question: Convert Python list to pandas Series What is the method to convert a Python list of strings to a pd.Series object? (pandas Series objects can be converted to list using tolist() method--but how to do the reverse conversion?)

Response:
To convert the list myList to a Pandas series use:
```
mySeries = pd.Series(myList)
```
This is also one of the basic ways for creating a series from a list in Pandas. Example:
```
myList = ['string1', 'string2', 'string3']
mySeries = pd.Series(myList)
mySeries
# Out:
# 0    string1
# 1    string2
# 2    string3
# dtype: object
```
Note that Pandas will guess the data type of the elements of the list because a series doesn't admit mixed types (contrary to Python lists). In the example above the inferred datatype was object (the Python string) because it's the most general and can accommodate all other data types (see data types). It's possible to specify a data type when creating a series:
```
myList= [1, 2, 3]

# inferred data type is integer
pd.Series(myList).dtype
# Out:
# dtype('int64')

myList= ['1', 2, 3]

# data type is object
pd.Series(myList).dtype
# Out:
# dtype('O')
```
One can specify dtype as integer:
```
myList= ['1', 2.2, '3']
mySeries = pd.Series(myList, dtype='int')
mySeries.dtype
# Out:
# dtype('int64')
```
But this will work only if all elements in the list can be casted to the desired data type.


Question:
Pandas - Case when & default in pandas I have the below case statement in python,
```py
pd_df['difficulty'] = 'Unknown'
pd_df['difficulty'][(pd_df['Time']<30) & (pd_df['Time']>0)] = 'Easy'
pd_df['difficulty'][(pd_df['Time']>=30) & (pd_df['Time']<=60)] = 'Medium'
pd_df['difficulty'][pd_df['Time']>60] = 'Hard'
```
But when I run the code, it throws an error.
```none
A value is trying to be set on a copy of a slice from a DataFrame
```

Response:
Option 1 For performance, use a nested np.where condition. For the condition, you can just use pd.Series.between, and the default value will be inserted accordingly.
```
pd_df['difficulty'] = np.where(
     pd_df['Time'].between(0, 30, inclusive=False),
    'Easy',
     np.where(
        pd_df['Time'].between(0, 30, inclusive=False), 'Medium', 'Unknown'
     )
)
```
Option 2 Similarly, using np.select, this gives more room for adding conditions:
```
pd_df['difficulty'] = np.select(
    [
        pd_df['Time'].between(0, 30, inclusive=False),
        pd_df['Time'].between(30, 60, inclusive=True)
    ],
    [
        'Easy',
        'Medium'
    ],
    default='Unknown'
)
```
Option 3 Another performant solution involves loc:
```
pd_df['difficulty'] = 'Unknown'
pd_df.loc[pd_df['Time'].between(0, 30, inclusive=False), 'difficulty'] = 'Easy'
pd_df.loc[pd_df['Time'].between(30, 60, inclusive=True), 'difficulty'] = 'Medium'
```


Question:
Split pandas dataframe based on values in a column using groupby I want to split the following dataframe based on column ZZ
```
df =
        N0_YLDF  ZZ        MAT
    0  6.286333   2  11.669069
    1  6.317000   6  11.669069
    2  6.324889   6  11.516454
    3  6.320667   5  11.516454
    4  6.325556   5  11.516454
    5  6.359000   6  11.516454
    6  6.359000   6  11.516454
    7  6.361111   7  11.516454
    8  6.360778   7  11.516454
    9  6.361111   6  11.516454
```
As output, I want a new DataFrame with the N0_YLDF column split into 4, one new column for each unique value of ZZ. How do I go about this? I can do groupby, but do not know what to do with the grouped object.

Response:
```
gb = df.groupby('ZZ')
[gb.get_group(x) for x in gb.groups]
```


Question:
Using .loc with a MultiIndex in pandas Does anyone know if it is possible to use the DataFrame.loc method to select from a MultiIndex? I have the following DataFrame and would like to be able to access the values located in the Dwell columns, at the indices of ('at', 1), ('at', 3), ('at', 5), and so on (non-sequential). I'd love to be able to do something like data.loc[['at',[1,3,5]], 'Dwell'], similar to the data.loc[[1,3,5], 'Dwell'] syntax for a regular index (which returns a 3-member series of Dwell values). My purpose is to select an arbitrary subset of the data, perform some analysis only on that subset, and then update the new values with the results of the analysis. I plan on using the same syntax to set new values for these data, so chaining selectors wouldn't really work in this case. Here is a slice of the DataFrame I'm working with:
```
Char    Dwell  Flight  ND_Offset  Offset
QGram
at    0     a      100     120   0.000000       0
      1     t      180       0   0.108363       5
      2     a      100     120   0.000000       0
      3     t      180       0   0.108363       5
      4     a       20     180   0.000000       0
      5     t       80     120   0.108363       5
      6     a       20     180   0.000000       0
      7     t       80     120   0.108363       5
      8     a       20     180   0.000000       0
      9     t       80     120   0.108363       5
      10    a      120     180   0.000000       0
```

Response:
If you are on version 0.14, you can simply pass a tuple to .loc as below:
```
df.loc[('at', [1,3,4]), 'Dwell']
```


Question: What does the term "broadcasting" mean in Pandas documentation? I'm reading through the Pandas documentation, and the term "broadcasting" is used extensively, but never really defined or explained. What does it mean?

Response:
So the term broadcasting comes from numpy, simply put it explains the rules of the output that will result when you perform operations between n-dimensional arrays (could be panels, dataframes, series) or scalar values. Broadcasting using a scalar value So the simplest case is just multiplying by a scalar value:
```

```python
s = pd.Series(np.arange(5))
s
#Output
#0    0
#1    1
#2    2
#3    3
#4    4
#dtype: int32
```

```python
s * 10
#Output
#0     0
#1    10
#2    20
#3    30
#4    40
#dtype: int32
#```
#and we get the same expected results with a dataframe:
#```
```

```python
df = pd.DataFrame({'a':np.random.randn(4), 'b':np.random.randn(4)})
df
#Output
#a         b
#0  0.216920  0.652193
#1  0.968969  0.033369
#2  0.637784  0.856836
#3 -2.303556  0.426238
```

```python
df * 10
#Output
#a         b
#0   2.169204  6.521925
#1   9.689690  0.333695
#2   6.377839  8.568362
#3 -23.035557  4.262381
#```
#So what is technically happening here is that the scalar value has been broadcasted along the same dimensions of the Series and DataFrame above. Broadcasting using a 1-D array Say we have a 2-D dataframe of shape 4 x 3 (4 rows x 3 columns) we can perform an operation along the x-axis by using a 1-D Series that is the same length as the row-length:
#```
```

```python
df = pd.DataFrame({'a':np.random.randn(4), 'b':np.random.randn(4), 'c':np.random.randn(4)})
df
#Output
#a         b         c
#0  0.122073 -1.178127 -1.531254
#1  0.011346 -0.747583 -1.967079
#2 -0.019716 -0.235676  1.419547
#3  0.215847  1.112350  0.659432
```

```python
df.iloc[0]
#Output
#a    0.122073
#b   -1.178127
#c   -1.531254
#Name: 0, dtype: float64
```

```python
df + df.iloc[0]
#Output
#a         b         c
#0  0.244146 -2.356254 -3.062507
#1  0.133419 -1.925710 -3.498333
#2  0.102357 -1.413803 -0.111707
#3  0.337920 -0.065777 -0.871822
#```
#the above looks funny at first until you understand what is happening, I took the first row of values and added this row-wise to the df, it can be visualised using this pic (sourced from scipy): The general rule is this: In order to broadcast, the size of the trailing axes for both arrays in an operation must either be the same size or one of them must be one. So if I tried to add a 1-D array that didn't match in length, say one with 4 elements, unlike numpy which will raise a ValueError, in Pandas you'll get a df full of NaN values:
#```
```

```python
df + pd.Series(np.arange(4))
#Output
#a   b   c   0   1   2   3
#0 NaN NaN NaN NaN NaN NaN NaN
#1 NaN NaN NaN NaN NaN NaN NaN
#2 NaN NaN NaN NaN NaN NaN NaN
#3 NaN NaN NaN NaN NaN NaN NaN
#```
#Now some of the great things about pandas is that it will try to align using existing column names and row labels, this can get in the way of trying to perform a fancier broadcasting like this:
#```
```

```python
df[['a']] + df.iloc[0]
#Output
#a   b   c
#0  0.244146 NaN NaN
#1  0.133419 NaN NaN
#2  0.102357 NaN NaN
#3  0.337920 NaN NaN
#```
#In the above I use double subscripting to force the shape to be (4,1) but we see a problem when trying to broadcast using the first row as the column alignment only aligns on the first column. To get the same form of broadcasting to occur like the diagram above shows we have to decompose to numpy arrays which then become anonymous data:
#```
```

```python
df[['a']].values + df.iloc[0].values
#Output
#array([[ 0.24414608, -1.05605392, -1.4091805 ],
#       [ 0.13341899, -1.166781  , -1.51990758],
#       [ 0.10235701, -1.19784299, -1.55096957],
#       [ 0.33792013, -0.96227987, -1.31540645]])
#```
#It's also possible to broadcast in 3-dimensions but I don't go near that stuff often but the numpy, scipy and pandas book have examples that show how that works. Generally speaking the thing to remember is that aside from scalar values which are simple, for n-D arrays the minor/trailing axes length must match or one of them must be 1. Update it seems that the above now leads to ValueError: Unable to coerce to Series, length must be 1: given 3 in latest version of pandas 0.20.2 so you have to call .values on the df first:
#```
```

```python
df[['a']].values + df.iloc[0].values
#Output
#array([[ 0.244146, -1.056054, -1.409181],
#       [ 0.133419, -1.166781, -1.519908],
#       [ 0.102357, -1.197843, -1.55097 ],
#       [ 0.33792 , -0.96228 , -1.315407]])
#```
#To restore this back to the original df we can construct a df from the np array and pass the original columns in the args to the constructor:
#```
```

```python
pd.DataFrame(df[['a']].values + df.iloc[0].values, columns=df.columns)
#Output
#a         b         c
#0  0.244146 -1.056054 -1.409181
#1  0.133419 -1.166781 -1.519908
#2  0.102357 -1.197843 -1.550970
#3  0.337920 -0.962280 -1.315407
#```
```


Question:
Convert month int to month name in Pandas I want to transform an integer between 1 and 12 into an abbrieviated month name. I have a df which looks like:
```
client Month
1  sss    02
2  yyy    12
3  www    06
```
I want the df to look like this:
```
client Month
1  sss    Feb
2  yyy    Dec
3  www    Jun
```
Most of the info I found was not in python>pandas>dataframe hence the question.

Response:
You can do this efficiently with combining calendar.month_abbr and df[col].apply()
```
import calendar
df['Month'] = df['Month'].apply(lambda x: calendar.month_abbr[x])
```


Question:
How to dynamically update a plot in a loop in IPython notebook (within one cell) Environment: Python 2.7, Matplotlib 1.3, IPython notebook 1.1, Linux, and Chrome. The code is in one single input cell, using --pylab=inline. I want to use IPython notebook and Pandas to consume a stream and dynamically update a plot every five seconds. When I just use a print statement to print the data in text format, it works perfectly fine: the output cell just keeps printing data and adding new rows. But when I try to plot the data (and then update it in a loop), the plot never shows up in the output cell. But if I remove the loop, and just plot it once, it works fine. Then I did some simple test:
```
i = pd.date_range('2013-1-1',periods=100,freq='s')
while True:
    plot(pd.Series(data=np.random.randn(100), index=i))
    #pd.Series(data=np.random.randn(100), index=i).plot() also tried this one
    time.sleep(5)
```
The output will not show anything until I manually interrupt the process (Ctrl + M + I). And after I interrupt it, the plot shows correctly as multiple overlapped lines. But what I really want is a plot that shows up and gets updated every five seconds (or whenever the plot() function gets called, just like what print statement outputs I mentioned above, which works well). Only showing the final chart after the cell is completely done is not what I want. I even tried to explicitly add the draw() function after each plot(), etc. None of them works. How can I dynamically update a plot by a for/while loop within one cell in IPython notebook?

Response:
Use the IPython.display module:
```
%matplotlib inline
import time
import pylab as pl
from IPython import display
for i in range(10):
    pl.plot(pl.randn(100))
    display.clear_output(wait=True)
    display.display(pl.gcf())
    time.sleep(1.0)
```


Question:
Converting a Pandas GroupBy multiindex output from Series back to DataFrame I have a dataframe:
```none
City     Name
0   Seattle    Alice
1   Seattle      Bob
2  Portland  Mallory
3   Seattle  Mallory
4   Seattle      Bob
5  Portland  Mallory
```
I perform the following grouping:
```py
g1 = df1.groupby(["Name", "City"]).count()
```
which when printed looks like:
```none
City  Name
Name    City
Alice   Seattle      1     1
Bob     Seattle      2     2
Mallory Portland     2     2
        Seattle      1     1
```
But what I want eventually is another DataFrame object that contains all the rows in the GroupBy object. In other words I want to get the following result:
```none
City  Name
Name    City
Alice   Seattle      1     1
Bob     Seattle      2     2
Mallory Portland     2     2
Mallory Seattle      1     1
```
How do I do it?

Response:
g1 here is a DataFrame. It has a hierarchical index, though:
```

```python
type(g1)
#Output
#pandas.core.frame.DataFrame
```

```python
g1.index
#Output
#MultiIndex([('Alice', 'Seattle'), ('Bob', 'Seattle'), ('Mallory', 'Portland'),
#       ('Mallory', 'Seattle')], dtype=object)
#```
#Perhaps you want something like this?
#```
```

```python
g1.add_suffix('_Count').reset_index()
#Output
#Name      City  City_Count  Name_Count
#0    Alice   Seattle           1           1
#1      Bob   Seattle           2           2
#2  Mallory  Portland           2           2
#3  Mallory   Seattle           1           1
#```
#Or something like:
#```
```

```python
DataFrame({'count' : df1.groupby( [ "Name", "City"] ).size()}).reset_index()
#Output
#Name      City  count
#0    Alice   Seattle      1
#1      Bob   Seattle      2
#2  Mallory  Portland      2
#3  Mallory   Seattle      1
#```
```


Question:
How to flatten a hierarchical index in columns I have a data frame with a hierarchical index in axis 1 (columns) (from a groupby.agg operation):
```
USAF   WBAN  year  month  day  s_PC  s_CL  s_CD  s_CNT  tempf
                                     sum   sum   sum    sum   amax   amin
0  702730  26451  1993      1    1     1     0    12     13  30.92  24.98
1  702730  26451  1993      1    2     0     0    13     13  32.00  24.98
2  702730  26451  1993      1    3     1    10     2     13  23.00   6.98
3  702730  26451  1993      1    4     1     0    12     13  10.04   3.92
4  702730  26451  1993      1    5     3     0    10     13  19.94  10.94
```
I want to flatten it, so that it looks like this (names aren't critical - I could rename):
```
USAF   WBAN  year  month  day  s_PC  s_CL  s_CD  s_CNT  tempf_amax  tmpf_amin
0  702730  26451  1993      1    1     1     0    12     13  30.92          24.98
1  702730  26451  1993      1    2     0     0    13     13  32.00          24.98
2  702730  26451  1993      1    3     1    10     2     13  23.00          6.98
3  702730  26451  1993      1    4     1     0    12     13  10.04          3.92
4  702730  26451  1993      1    5     3     0    10     13  19.94          10.94
```
How do I do this? (I've tried a lot, to no avail.) Per a suggestion, here is the head in dict form
```
{('USAF', ''): {0: '702730',
  1: '702730',
  2: '702730',
  3: '702730',
  4: '702730'},
 ('WBAN', ''): {0: '26451', 1: '26451', 2: '26451', 3: '26451', 4: '26451'},
 ('day', ''): {0: 1, 1: 2, 2: 3, 3: 4, 4: 5},
 ('month', ''): {0: 1, 1: 1, 2: 1, 3: 1, 4: 1},
 ('s_CD', 'sum'): {0: 12.0, 1: 13.0, 2: 2.0, 3: 12.0, 4: 10.0},
 ('s_CL', 'sum'): {0: 0.0, 1: 0.0, 2: 10.0, 3: 0.0, 4: 0.0},
 ('s_CNT', 'sum'): {0: 13.0, 1: 13.0, 2: 13.0, 3: 13.0, 4: 13.0},
 ('s_PC', 'sum'): {0: 1.0, 1: 0.0, 2: 1.0, 3: 1.0, 4: 3.0},
 ('tempf', 'amax'): {0: 30.920000000000002,
  1: 32.0,
  2: 23.0,
  3: 10.039999999999999,
  4: 19.939999999999998},
 ('tempf', 'amin'): {0: 24.98,
  1: 24.98,
  2: 6.9799999999999969,
  3: 3.9199999999999982,
  4: 10.940000000000001},
 ('year', ''): {0: 1993, 1: 1993, 2: 1993, 3: 1993, 4: 1993}}
```

Response:
I think the easiest way to do this would be to set the columns to the top level:
```
df.columns = df.columns.get_level_values(0)
```
Note: if the to level has a name you can also access it by this, rather than 0. . If you want to combine/join your MultiIndex into one Index (assuming you have just string entries in your columns) you could:
```
df.columns = [' '.join(col).strip() for col in df.columns.values]
```
Note: we must strip the whitespace for when there is no second index.
```

```python
[' '.join(col).strip() for col in df.columns.values]
#Output
#['USAF',
# 'WBAN',
# 'day',
# 'month',
# 's_CD sum',
# 's_CL sum',
# 's_CNT sum',
# 's_PC sum',
# 'tempf amax',
# 'tempf amin',
# 'year']
#```
```


Question:
How to repeat a Pandas DataFrame? This is my DataFrame that should be repeated for 5 times:
```

```python
x = pd.DataFrame({'a':1,'b':2}, index = range(1))
```

```python
x
#Output
#   a  b
#0  1  2
#```
#I want to have the result like this:
#```
```

```python
x.append(x).append(x).append(x)
#Output
#   a  b
#0  1  2
#0  1  2
#0  1  2
#0  1  2
#```
#But there must be a smarter way than appending 4 times. Actually the DataFrame I’m working on should be repeated 50 times. I haven't found anything practical, including those like np.repeat ---- it just doesn't work on a DataFrame. Could anyone help?
```

Response:
You can use the concat function:
```

```python
pd.concat([x]*5)
#Output
#a  b
#0  1  2
#0  1  2
#0  1  2
#0  1  2
#0  1  2
#```
#If you only want to repeat the values and not the index, you can do:
#```
```

```python
pd.concat([x]*5, ignore_index=True)
#Output
#a  b
#0  1  2
#1  1  2
#2  1  2
#3  1  2
#4  1  2
#```
```


Question:
How to shift a column in Pandas DataFrame I would like to shift a column in a Pandas DataFrame, but I haven't been able to find a method to do it from the documentation without rewriting the whole DF. Does anyone know how to do it? DataFrame:
```
##    x1   x2
##0  206  214
##1  226  234
##2  245  253
##3  265  272
##4  283  291
```
Desired output:
```
##    x1   x2
##0  206  nan
##1  226  214
##2  245  234
##3  265  253
##4  283  272
##5  nan  291
```

Response:
```

```python
a
#Output
#x1  x2
#0   0   5
#1   1   6
#2   2   7
#3   3   8
#4   4   9
```

```python
a['x2'] = a.x2.shift(1)
```

```python
a
#Output
#x1  x2
#0   0 NaN
#1   1   5
#2   2   6
#3   3   7
#4   4   8
#```
```


Question:
How to divide two columns element-wise in a pandas dataframe I have two columns in my pandas dataframe. I'd like to divide column A by column B, value by value, and show it as follows:
```py
import pandas as pd

csv1 = pd.read_csv('auto$0$0.csv')
csv2 = pd.read_csv('auto$0$8.csv')

df1 = pd.DataFrame(csv1, columns=['Column A', 'Column B'])
df2 = pd.DataFrame(csv2, columns=['Column A', 'Column B'])

dfnew = pd.concat([df1, df2])
```
The columns:
```none
Column A  Column B
12        2
14        7
16        8
20        5
```
And the expected result:
```none
Result
6
2
2
4
```
How do I do this?

Response:
Just divide the columns:
```

```python
df['Result'] = df['Column A']/df['Column B']
df
#Output
#Column A  Column B  Result
#0        12         2     6.0
#1        14         7     2.0
#2        16         8     2.0
#3        20         5     4.0
#```
```


Question:
Read JSON to pandas dataframe - ValueError: Mixing dicts with non-Series may lead to ambiguous ordering I am trying to read in the JSON structure below into pandas dataframe, but it throws out the error message: ValueError: Mixing dicts with non-Series may lead to ambiguous ordering. Json data:
```
{
    "status": {
        "statuscode": 200,
        "statusmessage": "Everything OK"
    },

    "result": [{
        "id": 22,
        "club_id": 16182
    }, {
        "id": 23,
        "club_id": 16182
    }, {
        "id": 24,
        "club_id": 16182
    }, {
        "id": 25,
        "club_id": 16182
    }, {
        "id": 26,
        "club_id": 16182
    }, {
        "id": 27,
        "club_id": 16182
    }]
}
```
How do I get this right? I have tried the script below...
```
j_df = pd.read_json('json_file.json')
j_df

with open(j_file) as jsonfile:
    data = json.load(jsonfile)
```

Response:
If you just need the result part in a dataframe, then here is the code to help you.
```
import json
import pandas as pd
data = json.load(open('json_file.json'))

df = pd.DataFrame(data["result"])
```


Question:
Python pandas Filtering out nan from a data selection of a column of strings Without using groupby how would I filter out data without NaN? Let say I have a matrix where customers will fill in 'N/A','n/a' or any of its variations and others leave it blank:
```
import pandas as pd
import numpy as np

df = pd.DataFrame({'movie': ['thg', 'thg', 'mol', 'mol', 'lob', 'lob'],
                  'rating': [3., 4., 5., np.nan, np.nan, np.nan],
                  'name': ['John', np.nan, 'N/A', 'Graham', np.nan, np.nan]})

nbs = df['name'].str.extract('^(N/A|NA|na|n/a)')
nms=df[(df['name'] != nbs) ]
```
output:
```

```python
nms
#Output
#  movie    name  rating
#0   thg    John       3
#1   thg     NaN       4
#3   mol  Graham     NaN
#4   lob     NaN     NaN
#5   lob     NaN     NaN
#```
#How would I filter out NaN values so I can get results to work with like this:
#```
#movie    name  rating
#0   thg    John       3
#3   mol  Graham     NaN
#```
#I am guessing I need something like ~np.isnan but the tilda does not work with strings.
```

Response:
Just drop them:
```
nms.dropna(thresh=2)
```
this will drop all rows where there are at least two non-NaN. Then you could then drop where name is NaN:
```

```python
nms
#Output
#movie    name  rating
#0   thg    John       3
#1   thg     NaN       4
#3   mol  Graham     NaN
#4   lob     NaN     NaN
#5   lob     NaN     NaN
#
#[5 rows x 3 columns]
```

```python
nms = nms.dropna(thresh=2)
```

```python
nms[nms.name.notnull()]
#Output
#movie    name  rating
#0   thg    John       3
#3   mol  Graham     NaN
#
#[2 rows x 3 columns]
#```
#EDIT Actually looking at what you originally want you can do just this without the dropna call:
#```
#nms[nms.name.notnull()]
#```
#UPDATE Looking at this question 3 years later, there is a mistake, firstly thresh arg looks for at least n non-NaN values so in fact the output should be:
#```
```

```python
nms.dropna(thresh=2)
#Output
#movie    name  rating
#0   thg    John     3.0
#1   thg     NaN     4.0
#3   mol  Graham     NaN
#```
#It's possible that I was either mistaken 3 years ago or that the version of pandas I was running had a bug, both scenarios are entirely possible.
```


Question:
How to check if a value is in the list in selection from pandas data frame? [duplicate] This question already has answers here: Use a list of values to select rows from a Pandas dataframe (9 answers) Closed 2 years ago. Looks ugly:
```
df_cut = df_new[
             (
             (df_new['l_ext']==31) |
             (df_new['l_ext']==22) |
             (df_new['l_ext']==30) |
             (df_new['l_ext']==25) |
             (df_new['l_ext']==64)
             )
            ]
```
Does not work:
```
df_cut = df_new[(df_new['l_ext'] in [31, 22, 30, 25, 64])]
```
Is there an elegant and working solution of the above "problem"?

Response:
Use isin
```
df_new[df_new['l_ext'].isin([31, 22, 30, 25, 64])]
```


Question:
How to query if a list-type column contains something? I have a dataframe, which contains info about movies. It has a column called genre, which contains a list of genres it belongs to. For example:
```
df['genre']

## returns

0       ['comedy', 'sci-fi']
1       ['action', 'romance', 'comedy']
2       ['documentary']
3       ['crime','horror']
...
```
I want to know how can I query the dataframe, so it returns the movie belonging to a certain genre? For example, something like df['genre'].contains('comedy') returns 0 or 1. I know for a list, I can do things like:
```
'comedy' in  ['comedy', 'sci-fi']
```
However, in pandas, I didn't find something similar, the only thing I know is df['genre'].str.contains(), but it didn't work for the list type.

Response:
You can use apply for create mask and then boolean indexing:
```
mask = df.genre.apply(lambda x: 'comedy' in x)
df1 = df[mask]
print (df1)
                       genre
0           [comedy, sci-fi]
1  [action, romance, comedy]
```


Question:
Strings in a DataFrame, but dtype is object Why does Pandas tell me that I have objects, although every item in the selected column is a string — even after explicit conversion. This is my DataFrame:
```
<class 'pandas.core.frame.DataFrame'>
Int64Index: 56992 entries, 0 to 56991
Data columns (total 7 columns):
id            56992  non-null values
attr1         56992  non-null values
attr2         56992  non-null values
attr3         56992  non-null values
attr4         56992  non-null values
attr5         56992  non-null values
attr6         56992  non-null values
dtypes: int64(2), object(5)
```
Five of them are dtype object. I explicitly convert those objects to strings:
```
for c in df.columns:
    if df[c].dtype == object:
        print "convert ", df[c].name, " to string"
        df[c] = df[c].astype(str)
```
Then, df["attr2"] still has dtype object, although type(df["attr2"].ix[0] reveals str, which is correct. Pandas distinguishes between int64 and float64 and object. What is the logic behind it when there is no dtype str? Why is a str covered by object?

Response: The dtype object comes from NumPy, it describes the type of element in a ndarray. Every element in an ndarray must have the same size in bytes. For int64 and float64, they are 8 bytes. But for strings, the length of the string is not fixed. So instead of saving the bytes of strings in the ndarray directly, Pandas uses an object ndarray, which saves pointers to objects; because of this the dtype of this kind ndarray is object. Here is an example: the int64 array contains 4 int64 value. the object array contains 4 pointers to 3 string objects.


Question: concat series onto dataframe with column name I want to add a Series (s) to a Pandas DataFrame (df) as a new column. The series has more values than there are rows in the dataframe, so I am using the concat method along axis 1. df = pd.concat((df, s), axis=1) This works, but the new column of the dataframe representing the series is given an arbitrary numerical column name, and I would like this column to have a specific name instead. Is there a way to add a series to a dataframe, when the series is longer than the rows of the dataframe, and with a specified column name in the resulting dataframe?

Response:
You can try Series.rename:
```
df = pd.concat((df, s.rename('col')), axis=1)
```


Question:
python pandas dataframe slicing by date conditions I am able to read and slice pandas dataframe using python datetime objects, however I am forced to use only existing dates in index. For example, this works:
```

```python
data
#Output
#<class 'pandas.core.frame.DataFrame'>
#DatetimeIndex: 252 entries, 2010-12-31 00:00:00 to 2010-04-01 00:00:00
#Data columns:
#Adj Close    252  non-null values
#dtypes: float64(1)
```

```python
st = datetime.datetime(2010, 12, 31, 0, 0)
```

```python
en = datetime.datetime(2010, 12, 28, 0, 0)
```

```python
data[st:en]
#Output
#            Adj Close
#Date
#2010-12-31     593.97
#2010-12-30     598.86
#2010-12-29     601.00
#2010-12-28     598.92
#```
#However if I use a start or end date that is not present in the DF, I get python KeyError. My Question : How do I query the dataframe object for a date range; even when the start and end dates are not present in the DataFrame. Does pandas allow for range based slicing? I am using pandas version 0.10.1
```

Response:
Use searchsorted to find the nearest times first, and then use it to slice.
```

```python
df = pd.DataFrame([1, 2, 3], index=[dt.datetime(2013, 1, 1), dt.datetime(2013, 1, 3), dt.datetime(2013, 1, 5)])
```

```python
df
#Output
#0
#2013-01-01  1
#2013-01-03  2
#2013-01-05  3
```

```python
start = df.index.searchsorted(dt.datetime(2013, 1, 2))
```

```python
end = df.index.searchsorted(dt.datetime(2013, 1, 4))
```

```python
df.iloc[start:end]
#Output
#0
#2013-01-03  2
#```
```


Question:
add a row at top in pandas dataframe [duplicate] This question already has answers here: Insert a row to pandas dataframe (20 answers) Closed 7 years ago. Below is my dataframe
```
import pandas as pd
df = pd.DataFrame({'name': ['jon','sam','jane','bob'],
           'age': [30,25,18,26],
           'sex':['male','male','female','male']})

   age  name     sex
0   30   jon    male
1   25   sam    male
2   18  jane  female
3   26   bob    male
```
I want to insert a new row at the first position name: dean, age: 45, sex: male
```
age  name     sex
0   45  dean    male
1   30   jon    male
2   25   sam    male
3   18  jane  female
4   26   bob    male
```
What is the best way to do this in pandas?

Response:
Probably this is not the most efficient way but:
```
df.loc[-1] = ['45', 'Dean', 'male']  # adding a row
df.index = df.index + 1  # shifting index
df.sort_index(inplace=True)
```
Output:
```
age  name     sex
0  45  Dean    male
1  30   jon    male
2  25   sam    male
3  18  jane  female
4  26   bob    male
```


Question:
Append an empty row in dataframe using pandas I am trying to append an empty row at the end of dataframe but unable to do so, even trying to understand how pandas work with append function and still not getting it. Here's the code:
```
import pandas as pd

excel_names = ["ARMANI+EMPORIO+AR0143-book.xlsx"]
excels = [pd.ExcelFile(name) for name in excel_names]
frames = [x.parse(x.sheet_names[0], header=None,index_col=None).dropna(how='all') for x in excels]
for f in frames:
    f.append(0, float('NaN'))
    f.append(2, float('NaN'))
```
There are two columns and random number of row. with "print f" in for loop i Get this:
```
0                 1
0                   Brand Name    Emporio Armani
2                 Model number            AR0143
4                  Part Number            AR0143
6                   Item Shape       Rectangular
8   Dial Window Material Type           Mineral
10               Display Type          Analogue
12                 Clasp Type            Buckle
14               Case Material   Stainless steel
16              Case Diameter    31 millimetres
18               Band Material           Leather
20                 Band Length  Women's Standard
22                 Band Colour             Black
24                 Dial Colour             Black
26            Special Features       second-hand
28                    Movement            Quartz
```

Response:
Add a new pandas.Series using pandas.DataFrame.append(). If you wish to specify the name (AKA the "index") of the new row, use:
```
df.append(pandas.Series(name='NameOfNewRow'))
```
If you don't wish to name the new row, use:
```
df.append(pandas.Series(), ignore_index=True)
```
where df is your pandas.DataFrame.


Question:
How to conditionally update DataFrame column in Pandas With this DataFrame, how can I conditionally set rating to 0 when line_race is equal to zero?
```
line_track  line_race  rating foreign
 25        MTH         10     84    False
 26        MTH          6     88    False
 27        TAM          5     87    False
 28         GP          2     86    False
 29         GP          7     59    False
 30        LCH          0    103     True
 31        LEO          0    125     True
 32        YOR          0    126     True
 33        ASC          0    124     True
```
In other words, what is the proper way on a DataFrame to say if ColumnA = x then ColumnB = y else ColumnB = ColumnB

Response:
```
df.loc[df['line_race'] == 0, 'rating'] = 0
```


Question:
Select only one index of multiindex DataFrame I am trying to create a new DataFrame using only one index from a multi-indexed DataFrame.
```
A         B         C
first second
bar   one     0.895717  0.410835 -1.413681
      two     0.805244  0.813850  1.607920
baz   one    -1.206412  0.132003  1.024180
      two     2.565646 -0.827317  0.569605
foo   one     1.431256 -0.076467  0.875906
      two     1.340309 -1.187678 -2.211372
qux   one    -1.170299  1.130127  0.974466
      two    -0.226169 -1.436737 -2.006747
```
Ideally, I would like something like this:
```
In: df.ix[level="first"]
```
and:
```
Out:

               A         B         C
first
bar        0.895717  0.410835 -1.413681
           0.805244  0.813850  1.607920
baz       -1.206412  0.132003  1.024180
           2.565646 -0.827317  0.569605
foo        1.431256 -0.076467  0.875906
           1.340309 -1.187678 -2.211372
qux       -1.170299  1.130127  0.974466
          -0.226169 -1.436737 -2.006747
`
```
Essentially I want to drop all the other indexes of the multi-index other than level first. Is there an easy way to do this?

Response:
One way could be to simply rebind df.index to the desired level of the MultiIndex. You can do this by specifying the label name you want to keep:
```
df.index = df.index.get_level_values('first')
```
or use the level's integer value:
```
df.index = df.index.get_level_values(0)
```
All other levels of the MultiIndex would disappear here.


Question:
Why is pandas.to_datetime slow for non standard time format such as '2014/12/31' I have a .csv file in such format
```
timestmp, p
2014/12/31 00:31:01:9200, 0.7
2014/12/31 00:31:12:1700, 1.9
...
```
and when read via pd.read_csv and convert the time str to datetime using pd.to_datetime, the performance drops dramatically. Here is a minimal example.
```
import re
import pandas as pd

d = '2014-12-12 01:02:03.0030'
c = re.sub('-', '/', d)

%timeit pd.to_datetime(d)
%timeit pd.to_datetime(c)
%timeit pd.to_datetime(c, format="%Y/%m/%d %H:%M:%S.%f")
```
and the performances are:
```
10000 loops, best of 3: 62.4 µs per loop
10000 loops, best of 3: 181 µs per loop
10000 loops, best of 3: 82.9 µs per loop
```
so, how could I improve the performance of pd.to_datetime when reading date from a csv file?

Response:
This is because pandas falls back to dateutil.parser.parse for parsing the strings when it has a non-default format or when no format string is supplied (this is much more flexible, but also slower). As you have shown above, you can improve the performance by supplying a format string to to_datetime. Or another option is to use infer_datetime_format=True Apparently, the infer_datetime_format cannot infer when there are microseconds. With an example without those, you can see a large speed-up:
```

```python
d = '2014-12-24 01:02:03'
```

```python
c = re.sub('-', '/', d)
```

```python
s_c = pd.Series([c]*10000)
```

```python
%timeit pd.to_datetime(s_c)
1 loops, best of 3: 1.14 s per loop
```

```python
%timeit pd.to_datetime(s_c, infer_datetime_format=True)
10 loops, best of 3: 105 ms per loop
```

```python
%timeit pd.to_datetime(s_c, format="%Y/%m/%d %H:%M:%S")
10 loops, best of 3: 99.5 ms per loop
```
```


Question: Why were pandas merges in python faster than data.table merges in R in 2012? I recently came across the pandas library for python, which according to this benchmark performs very fast in-memory merges. It's even faster than the data.table package in R (my language of choice for analysis). Why is pandas so much faster than data.table? Is it because of an inherent speed advantage python has over R, or is there some tradeoff I'm not aware of? Is there a way to perform inner and outer joins in data.table without resorting to merge(X, Y, all=FALSE) and merge(X, Y, all=TRUE)? Here's the R code and the Python code used to benchmark the various packages.

Response: The reason pandas is faster is because I came up with a better algorithm, which is implemented very carefully using a fast hash table implementation - klib and in C/Cython to avoid the Python interpreter overhead for the non-vectorizable parts. The algorithm is described in some detail in my presentation: A look inside pandas design and development. The comparison with data.table is actually a bit interesting because the whole point of R's data.table is that it contains pre-computed indexes for various columns to accelerate operations like data selection and merges. In this case (database joins) pandas' DataFrame contains no pre-computed information that is being used for the merge, so to speak it's a "cold" merge. If I had stored the factorized versions of the join keys, the join would be significantly faster - as factorizing is the biggest bottleneck for this algorithm. I should also add that the internal design of pandas' DataFrame is much more amenable to these kinds of operations than R's data.frame (which is just a list of arrays internally).


Question:
Selecting/excluding sets of columns in pandas [duplicate] This question already has answers here: Delete a column from a Pandas DataFrame (23 answers) Closed 6 years ago. I would like to create views or dataframes from an existing dataframe based on column selections. For example, I would like to create a dataframe df2 from a dataframe df1 that holds all columns from it except two of them. I tried doing the following, but it didn't work:
```
import numpy as np
import pandas as pd

# Create a dataframe with columns A,B,C and D
df = pd.DataFrame(np.random.randn(100, 4), columns=list('ABCD'))

# Try to create a second dataframe df2 from df with all columns except 'B' and D
my_cols = set(df.columns)
my_cols.remove('B').remove('D')

# This returns an error ("unhashable type: set")
df2 = df[my_cols]
```
What am I doing wrong? Perhaps more generally, what mechanisms does pandas have to support the picking and exclusions of arbitrary sets of columns from a dataframe?

Response:
You can either Drop the columns you do not need OR Select the ones you need
```
# Using DataFrame.drop
df.drop(df.columns[[1, 2]], axis=1, inplace=True)

# drop by Name
df1 = df1.drop(['B', 'C'], axis=1)

# Select the ones you want
df1 = df[['a','d']]
```


Question: Running get_dummies on several DataFrame columns? How can one idiomatically run a function like get_dummies, which expects a single column and returns several, on multiple DataFrame columns?

Response:
With pandas 0.19, you can do that in a single line :
```
pd.get_dummies(data=df, columns=['A', 'B'])
```
Columns specifies where to do the One Hot Encoding.
```

```python
df
#Output
#   A  B  C
#0  a  c  1
#1  b  c  2
#2  a  b  3
```

```python
pd.get_dummies(data=df, columns=['A', 'B'])
#Output
#   C  A_a  A_b  B_b  B_c
#0  1  1.0  0.0  0.0  1.0
#1  2  0.0  1.0  0.0  1.0
#2  3  1.0  0.0  1.0  0.0
#```
```


Question:
Absolute value for column in Python How could I convert the values of column 'count' to absolute value? A summary of my dataframe this:
```none
datetime       count
0   2011-01-20 00:00:00   14.565996
1   2011-01-20 01:00:00   10.204177
2   2011-01-20 02:00:00   -1.261569
3   2011-01-20 03:00:00    1.938322
4   2011-01-20 04:00:00    1.938322
5   2011-01-20 05:00:00   -5.963259
6   2011-01-20 06:00:00   73.711525
```

Response:
Use pandas.DataFrame.abs().
```
import pandas as pd

df = pd.DataFrame(data={'count':[1, -1, 2, -2, 3, -3]})

df['count'] = df['count'].abs()

print(df)
   count
#0      1
#1      1
#2      2
#3      2
#4      3
#5      3
```


Question: Python / Pandas - GUI for viewing a DataFrame or Matrix [closed] Closed. This question is seeking recommendations for software libraries, tutorials, tools, books, or other off-site resources. It does not meet Stack Overflow guidelines. It is not currently accepting answers. We don’t allow questions seeking recommendations for software libraries, tutorials, tools, books, or other off-site resources. You can edit the question so it can be answered with facts and citations. Closed 5 years ago. Improve this question I'm using the Pandas package and it creates a DataFrame object, which is basically a labeled matrix. Often I have columns that have long string fields, or dataframes with many columns, so the simple print command doesn't work well. I've written some text output functions, but they aren't great. What I'd really love is a simple GUI that lets me interact with a dataframe / matrix / table. Just like you would find in a SQL tool. Basically a window that has a read-only spreadsheet like view into the data. I can expand columns, page up and down through long tables, etc. I would suspect something like this exists, but I must be Googling with the wrong terms. It would be great if it is pandas specific, but I would guess I could use any matrix-accepting tool. (BTW - I'm on Windows.) Any pointers? Or, conversely, if someone knows this space well and knows this probably doesn't exist, any suggestions on if there is a simple GUI framework / widget I could use to roll my own? (But since my needs are limited, I'm reluctant to have to learn a big GUI framework and do a bunch of coding for this one piece.)

Response: 2019 update: I'm currently working on a successor tabloo. I wasn't fully satisfied with some other GUIs, so I created my own, which I'm now maintaining on Github. Example: Apart from the basic table + plot functionality, I wanted to have a specific way to filter data: select a column to filter from a combo box write an "underscore expression" to filter on that column using arbitrary Python code. For example: _ > 0 to filter positive values only, or more complex expressions like (_ >= date(2016, 1, 1)) & (_ <= date(2016, 1, 31)) e.g. for datetime columns.


Question:
Pandas: Multilevel column names pandas has support for multi-level column names:
```

```python
 x = pd.DataFrame({'instance':['first','first','first'],'foo':['a','b','c'],'bar':rand(3)})
```

```python
x = x.set_index(['instance','foo']).transpose()
```

```python
x.columns
#Output
#MultiIndex
#[(u'first', u'a'), (u'first', u'b'), (u'first', u'c')]
```

```python
x
#Output
#instance     first
#foo              a         b         c
#bar       0.102885  0.937838  0.907467
#```
#This feature is very useful since it allows multiple versions of the same dataframe to be appended 'horizontally' with the 1st level of the column names (in my example instance) distinguishing the instances. Imagine I already have a dataframe like this:
#```
#a         b         c
#bar       0.102885  0.937838  0.907467
#```
#Is there a nice way to add another level to the column names, similar to this for row index:
#```
#x['instance'] = 'first'
#x.set_level('instance',append=True)
#```
```

Response:
Try this:
```
df=pd.DataFrame({'a':[1,2,3],'b':[4,5,6]})

columns=[('c','a'),('c','b')]

df.columns=pd.MultiIndex.from_tuples(columns)
```


Question: Is it possible to add a string as a legend item I am producing some plots in matplotlib and would like to add explanatory text for some of the data. I want to have a string inside my legend as a separate legend item above the '0-10' item. Does anyone know if there is a possible way to do this? This is the code for my legend: ax.legend(['0-10','10-100','100-500','500+'],loc='best')

Response:
Alternative solution, kind of dirty but pretty quick.
```
import pylab as plt

X = range(50)
Y = range(50)
plt.plot(X, Y, label="Very straight line")

# Create empty plot with blank marker containing the extra label
plt.plot([], [], ' ', label="Extra label on the legend")

plt.legend()
plt.show()
```


Question:
Python Pandas: Convert ".value_counts" output to dataframe Hi I want to get the counts of unique values of the dataframe. count_values implements this however I want to use its output somewhere else. How can I convert .count_values output to a pandas dataframe. here is an example code:
```
import pandas as pd
df = pd.DataFrame({'a':[1, 1, 2, 2, 2]})
value_counts = df['a'].value_counts(dropna=True, sort=True)
print(value_counts)
print(type(value_counts))
```
output is:
```
2    3
1    2
Name: a, dtype: int64
<class 'pandas.core.series.Series'>
```
What I need is a dataframe like this:
```
unique_values  counts
2              3
1              2
```
Thank you.

Response:
Use rename_axis for name of column from index and reset_index:
```
df = df.value_counts().rename_axis('unique_values').reset_index(name='counts')
print (df)
   unique_values  counts
0              2       3
1              1       2
```
Or if need one column DataFrame use Series.to_frame:
```
df = df.value_counts().rename_axis('unique_values').to_frame('counts')
print (df)
               counts
unique_values
2                   3
1                   2
```


Question: Coalesce values from 2 columns into a single column in a pandas dataframe I'm looking for a method that behaves similarly to coalesce in T-SQL. I have 2 columns (column A and B) that are sparsely populated in a pandas dataframe. I'd like to create a new column using the following rules: If the value in column A is not null, use that value for the new column C If the value in column A is null, use the value in column B for the new column C Like I mentioned, this can be accomplished in MS SQL Server via the coalesce function. I haven't found a good pythonic method for this; does one exist?

Response:
use combine_first():
```

```python
df = pd.DataFrame(np.random.randint(0, 10, size=(10, 2)), columns=list('ab'))
```

```python
df.loc[::2, 'a'] = np.nan
```

```python
df
#Output
#a  b
#0  NaN  0
#1  5.0  5
#2  NaN  8
#3  2.0  8
#4  NaN  3
#5  9.0  4
#6  NaN  7
#7  2.0  0
#8  NaN  6
#9  2.0  5
```

```python
df['c'] = df.a.combine_first(df.b)
```

```python
df
#Output
#a  b    c
#0  NaN  0  0.0
#1  5.0  5  5.0
#2  NaN  8  8.0
#3  2.0  8  2.0
#4  NaN  3  3.0
#5  9.0  4  9.0
#6  NaN  7  7.0
#7  2.0  0  2.0
#8  NaN  6  6.0
#9  2.0  5  2.0
#```
```


Question:
Get column name where value is something in pandas dataframe I'm trying to find, at each timestamp, the column name in a dataframe for which the value matches with the one in a timeseries at the same timestamp. Here is my dataframe:
```

```python
df
#Output
#                            col5        col4        col3        col2        col1
#1979-01-01 00:00:00  1181.220328  912.154923  648.848635  390.986156  138.185861
#1979-01-01 06:00:00  1190.724461  920.767974  657.099560  399.395338  147.761352
#1979-01-01 12:00:00  1193.414510  918.121482  648.558837  384.632475  126.254342
#1979-01-01 18:00:00  1171.670276  897.585930  629.201469  366.652033  109.545607
#1979-01-02 00:00:00  1168.892579  900.375126  638.377583  382.584568  132.998706
```

```python
df.to_dict()
#Output
#{'col4': {<Timestamp: 1979-01-01 06:00:00>: 920.76797370744271, <Timestamp: 1979-01-01 00:00:00>: 912.15492332839756, <Timestamp: 1979-01-01 18:00:00>: 897.58592995700656, <Timestamp: 1979-01-01 12:00:00>: 918.1214819496729}, 'col5': {<Timestamp: 1979-01-01 06:00:00>: 1190.7244605667831, <Timestamp: 1979-01-01 00:00:00>: 1181.2203275146587, <Timestamp: 1979-01-01 18:00:00>: 1171.6702763228691, <Timestamp: 1979-01-01 12:00:00>: 1193.4145103184442}, 'col2': {<Timestamp: 1979-01-01 06:00:00>: 399.39533771666561, <Timestamp: 1979-01-01 00:00:00>: 390.98615646597591, <Timestamp: 1979-01-01 18:00:00>: 366.65203285812231, <Timestamp: 1979-01-01 12:00:00>: 384.63247469269874}, 'col3': {<Timestamp: 1979-01-01 06:00:00>: 657.09956023625466, <Timestamp: 1979-01-01 00:00:00>: 648.84863460462293, <Timestamp: 1979-01-01 18:00:00>: 629.20146872682449, <Timestamp: 1979-01-01 12:00:00>: 648.55883747413225}, 'col1': {<Timestamp: 1979-01-01 06:00:00>: 147.7613518219286, <Timestamp: 1979-01-01 00:00:00>: 138.18586102094068, <Timestamp: 1979-01-01 18:00:00>: 109.54560722575859, <Timestamp: 1979-01-01 12:00:00>: 126.25434189361377}}
#```
#And the time series with values I want to match at each timestamp:
#```
```

```python
ts
#Output
#1979-01-01 00:00:00    1181.220328
#1979-01-01 06:00:00    657.099560
#1979-01-01 12:00:00    126.254342
#1979-01-01 18:00:00    109.545607
#Freq: 6H
```

```python
ts.to_dict()
#Output
#{<Timestamp: 1979-01-01 06:00:00>: 657.09956023625466, <Timestamp: 1979-01-01 00:00:00>: 1181.2203275146587, <Timestamp: 1979-01-01 18:00:00>: 109.54560722575859, <Timestamp: 1979-01-01 12:00:00>: 126.25434189361377}
#```
#Then the result would be:
#```
```

```python
df_result
#Output
#                             value  Column
#1979-01-01 00:00:00    1181.220328  col5
#1979-01-01 06:00:00    657.099560   col3
#1979-01-01 12:00:00    126.254342   col1
#1979-01-01 18:00:00    109.545607   col1
#```
#I hope my question is clear enough. Anyone has an idea how to get df_result? Thanks Greg
```

Response:
Just wanted to add that for a situation where multiple columns may have the value and you want all the column names in a list, you can do the following (e.g. get all column names with a value = 'x'):
```
df.apply(lambda row: row[row == 'x'].index, axis=1)
```
The idea is that you turn each row into a series (by adding axis=1) where the column names are now turned into the index of the series. You then filter your series with a condition (e.g. row == 'x'), then take the index values (aka column names!).


Question: Plot a horizontal line on a given plot How do I add a horizontal line to an existing plot?

Response:
Use axhline (a horizontal axis line). For example, this plots a horizontal line at y = 0.5:
```
import matplotlib.pyplot as plt
plt.axhline(y=0.5, color='r', linestyle='-')
plt.show()
```


Question:
iloc giving 'IndexError: single positional indexer is out-of-bounds' I am trying to encode some information to read into a Machine Learning model using the following
```py
import numpy as np
import pandas as pd
import matplotlib.pyplot as py

Dataset = pd.read_csv('filename.csv', sep = ',')

X = Dataset.iloc[:,:-1].values
Y = Dataset.iloc[:,18].values

from sklearn.preprocessing import LabelEncoder, OneHotEncoder
labelencoder_X = LabelEncoder()
X[:, 0] = labelencoder_X.fit_transform(X[:, 0])
onehotencoder = OneHotEncoder(categorical_features = [0])
X = onehotencoder.fit_transform(X).toarray()
```
however I am getting an error that reads
```none
IndexError: single positional indexer is out-of-bounds
```

Response:
This error is caused by:
```
Y = Dataset.iloc[:,18].values
```
Indexing is out of bounds here most probably because there are less than 19 columns in your Dataset, so column 18 does not exist. The following code you provided doesn't use Y at all, so you can just comment out this line for now.


Question:
Python pandas groupby aggregate on multiple columns, then pivot In Python, I have a pandas DataFrame similar to the following:
```
Item | shop1 | shop2 | shop3 | Category
------------------------------------
Shoes| 45    | 50    | 53    | Clothes
TV   | 200   | 300   | 250   | Technology
Book | 20    | 17    | 21    | Books
phone| 300   | 350   | 400   | Technology
```
Where shop1, shop2 and shop3 are the costs of every item in different shops. Now, I need to return a DataFrame, after some data cleaning, like this one:
```
Category (index)| size| sum| mean | std
----------------------------------------
```
where size is the number of items in each Category and sum, mean and std are related to the same functions applied to the 3 shops. How can I do these operations with the split-apply-combine pattern (groupby, aggregate, apply,...) ? Can someone help me out? I'm going crazy with this one...thank you!

Response:
Edited for Pandas 0.22+ considering the deprecation of the use of dictionaries in a group by aggregation. We set up a very similar dictionary where we use the keys of the dictionary to specify our functions and the dictionary itself to rename the columns.
```
rnm_cols = dict(size='Size', sum='Sum', mean='Mean', std='Std')
df.set_index(['Category', 'Item']).stack().groupby('Category') \
  .agg(rnm_cols.keys()).rename(columns=rnm_cols)

            Size   Sum        Mean        Std
Category
Books          3    58   19.333333   2.081666
Clothes        3   148   49.333333   4.041452
Technology     6  1800  300.000000  70.710678
```
option 1 use agg ← link to docs
```
agg_funcs = dict(Size='size', Sum='sum', Mean='mean', Std='std')
df.set_index(['Category', 'Item']).stack().groupby(level=0).agg(agg_funcs)

                  Std   Sum        Mean  Size
Category
Books        2.081666    58   19.333333     3
Clothes      4.041452   148   49.333333     3
Technology  70.710678  1800  300.000000     6
```
option 2 more for less use describe ← link to docs
```
df.set_index(['Category', 'Item']).stack().groupby(level=0).describe().unstack()

            count        mean        std    min    25%    50%    75%    max
Category
Books         3.0   19.333333   2.081666   17.0   18.5   20.0   20.5   21.0
Clothes       3.0   49.333333   4.041452   45.0   47.5   50.0   51.5   53.0
Technology    6.0  300.000000  70.710678  200.0  262.5  300.0  337.5  400.0
```


Question: Pandas every nth row Dataframe.resample() works only with timeseries data. I cannot find a way of getting every nth row from non-timeseries data. What is the best method?

Response:
I'd use iloc, which takes a row/column slice, both based on integer position and following normal python syntax. If you want every 5th row:
```
df.iloc[::5, :]
```


Question:
How to use tqdm with pandas in a jupyter notebook? I'm doing some analysis with pandas in a jupyter notebook and since my apply function takes a long time I would like to see a progress bar. Through this post here I found the tqdm library that provides a simple progress bar for pandas operations. There is also a Jupyter integration that provides a really nice progress bar where the bar itself changes over time. However, I would like to combine the two and don't quite get how to do that. Let's just take the same example as in the documentation
```
import pandas as pd
import numpy as np
from tqdm import tqdm

df = pd.DataFrame(np.random.randint(0, 100, (100000, 6)))

# Register `pandas.progress_apply` and `pandas.Series.map_apply` with `tqdm`
# (can use `tqdm_gui`, `tqdm_notebook`, optional kwargs, etc.)
tqdm.pandas(desc="my bar!")

# Now you can use `progress_apply` instead of `apply`
# and `progress_map` instead of `map`
df.progress_apply(lambda x: x**2)
# can also groupby:
# df.groupby(0).progress_apply(lambda x: x**2)
```
It even says "can use 'tqdm_notebook' " but I don't find a way how. I've tried a few things like
```
tqdm_notebook(tqdm.pandas(desc="my bar!"))
```
or
```
tqdm_notebook.pandas
```
but they don't work. In the definition it looks to me like
```
tqdm.pandas(tqdm_notebook(desc="my bar!"))
```
should work, but the bar doesn't properly show the progress and there is still additional output. Any other ideas?

Response:
My working solution (copied from the documentation):
```
from tqdm.auto import tqdm
tqdm.pandas()
```


Question:
Using pyarrow how do you append to parquet file? How do you append/update to a parquet file with pyarrow?
```
import pandas as pd
import pyarrow as pa
import pyarrow.parquet as pq

 table2 = pd.DataFrame({'one': [-1, np.nan, 2.5], 'two': ['foo', 'bar', 'baz'], 'three': [True, False, True]})
 table3 = pd.DataFrame({'six': [-1, np.nan, 2.5], 'nine': ['foo', 'bar', 'baz'], 'ten': [True, False, True]})

pq.write_table(table2, './dataNew/pqTest2.parquet')
#append pqTest2 here?
```
There is nothing I found in the docs about appending parquet files. And, Can you use pyarrow with multiprocessing to insert/update the data.

Response:
I ran into the same issue and I think I was able to solve it using the following:
```
import pandas as pd
import pyarrow as pa
import pyarrow.parquet as pq

chunksize=10000 # this is the number of lines

pqwriter = None
for i, df in enumerate(pd.read_csv('sample.csv', chunksize=chunksize)):
    table = pa.Table.from_pandas(df)
    # for the first chunk of records
    if i == 0:
        # create a parquet write object giving it an output file
        pqwriter = pq.ParquetWriter('sample.parquet', table.schema)
    pqwriter.write_table(table)

# close the parquet writer
if pqwriter:
    pqwriter.close()
```


Question:
Deleting DataFrame row in Pandas based on column value I have the following DataFrame:
```none
daysago  line_race rating        rw    wrating
 line_date
2007-03-31       62         11     56  1.000000  56.000000
2007-03-10       83         11     67  1.000000  67.000000
2007-02-10      111          9     66  1.000000  66.000000
2007-01-13      139         10     83  0.880678  73.096278
2006-12-23      160         10     88  0.793033  69.786942
2006-11-09      204          9     52  0.636655  33.106077
2006-10-22      222          8     66  0.581946  38.408408
2006-09-29      245          9     70  0.518825  36.317752
2006-09-16      258         11     68  0.486226  33.063381
2006-08-30      275          8     72  0.446667  32.160051
2006-02-11      475          5     65  0.164591  10.698423
2006-01-13      504          0     70  0.142409   9.968634
2006-01-02      515          0     64  0.134800   8.627219
2005-12-06      542          0     70  0.117803   8.246238
2005-11-29      549          0     70  0.113758   7.963072
2005-11-22      556          0     -1  0.109852  -0.109852
2005-11-01      577          0     -1  0.098919  -0.098919
2005-10-20      589          0     -1  0.093168  -0.093168
2005-09-27      612          0     -1  0.083063  -0.083063
2005-09-07      632          0     -1  0.075171  -0.075171
2005-06-12      719          0     69  0.048690   3.359623
2005-05-29      733          0     -1  0.045404  -0.045404
2005-05-02      760          0     -1  0.039679  -0.039679
2005-04-02      790          0     -1  0.034160  -0.034160
2005-03-13      810          0     -1  0.030915  -0.030915
2004-11-09      934          0     -1  0.016647  -0.016647
```
I need to remove the rows where line_race is equal to 0. What's the most efficient way to do this?

Response:
If I'm understanding correctly, it should be as simple as:
```
df = df[df.line_race != 0]
```


Question:
Pandas Plotting with Multi-Index After performing a groupby.sum() on a DataFrame I'm having some trouble trying to create my intended plot.
```py
import pandas as pd
import numpy as np

np.random.seed(365)
rows = 100
data = {'Month': np.random.choice(['2014-01', '2014-02', '2014-03', '2014-04'], size=rows),
        'Code': np.random.choice(['A', 'B', 'C'], size=rows),
        'ColA': np.random.randint(5, 125, size=rows),
        'ColB': np.random.randint(0, 51, size=rows),}
df = pd.DataFrame(data)

     Month Code  ColA  ColB
0  2014-03    C    59    47
1  2014-01    A    24     9
2  2014-02    C    77    50

dfg = df.groupby(['Code', 'Month']).sum()

              ColA  ColB
Code Month
A    2014-01   124   102
     2014-02   398   282
     2014-03   474   198
     2014-04   830   237
B    2014-01   477   300
     2014-02   591   167
     2014-03   522   192
     2014-04   367   169
C    2014-01   412   180
     2014-02   275   205
     2014-03   795   291
     2014-04   901   309
```
How can I create a subplot (kind='bar') for each Code, where the x-axis is the Month and the bars are ColA and ColB?

Response:
I found the unstack(level) method to work perfectly, which has the added benefit of not needing a priori knowledge about how many Codes there are.
```py
ax = dfg.unstack(level=0).plot(kind='bar', subplots=True, rot=0, figsize=(9, 7), layout=(2, 3))
plt.tight_layout()
```


Question:
"Too many indexers" with DataFrame.loc I've read the docs about slicers a million times, but have never got my head round it, so I'm still trying to figure out how to use loc to slice a DataFrame with a MultiIndex. I'll start with the DataFrame from this SO answer:
```
value
first second third fourth
A0    B0     C1    D0          2
                   D1          3
             C2    D0          6
                   D1          7
      B1     C1    D0         10
                   D1         11
             C2    D0         14
                   D1         15
A1    B0     C1    D0         18
                   D1         19
             C2    D0         22
                   D1         23
      B1     C1    D0         26
                   D1         27
             C2    D0         30
                   D1         31
A2    B0     C1    D0         34
                   D1         35
             C2    D0         38
                   D1         39
      B1     C1    D0         42
                   D1         43
             C2    D0         46
                   D1         47
A3    B0     C1    D0         50
                   D1         51
             C2    D0         54
                   D1         55
      B1     C1    D0         58
                   D1         59
             C2    D0         62
                   D1         63
```
To select just A0 and C1 values, I can do:
```

```python
df.loc['A0', :, 'C1', :]
#Output
#value
#first second third fourth
#A0    B0     C1    D0          2
#                   D1          3
#      B1     C1    D0         10
#                   D1         11
#```
#Which also works selecting from three levels, and even with tuples:
#```
```

```python
df.loc['A0', :, ('C1', 'C2'), 'D1']
#Output
#value
#first second third fourth
#A0    B0     C1    D1          3
#             C2    D1          5
#      B1     C1    D1         11
#             C2    D1         13
#```
#So far, intuitive and brilliant. So why can't I select all values from the first index level?
#```
```

```python
df.loc[:, :, 'C1', :]
---------------------------------------------------------------------------
IndexingError                             Traceback (most recent call last)
<ipython-input-30-57b56108d941> in <module>()
----> 1 df.loc[:, :, 'C1', :]

/usr/local/lib/python2.7/dist-packages/pandas/core/indexing.pyc in __getitem__(self, key)
   1176     def __getitem__(self, key):
   1177         if type(key) is tuple:
-> 1178             return self._getitem_tuple(key)
   1179         else:
   1180             return self._getitem_axis(key, axis=0)

/usr/local/lib/python2.7/dist-packages/pandas/core/indexing.pyc in _getitem_tuple(self, tup)
    694
    695         # no multi-index, so validate all of the indexers
--> 696         self._has_valid_tuple(tup)
    697
    698         # ugly hack for GH #836

/usr/local/lib/python2.7/dist-packages/pandas/core/indexing.pyc in _has_valid_tuple(self, key)
    125         for i, k in enumerate(key):
    126             if i >= self.obj.ndim:
--> 127                 raise IndexingError('Too many indexers')
    128             if not self._has_valid_type(k, i):
    129                 raise ValueError("Location based indexing can only have [%s] "

IndexingError: Too many indexers
```
Surely this is not intended behaviour? Note: I know this is possible with df.xs('C1', level='third') but the current .loc behaviour seems inconsistent.
```

Response:
The reason this doesn't work is tied to the need to specify the axis of indexing (mentioned in http://pandas.pydata.org/pandas-docs/stable/advanced.html). An alternative solution to your problem is to simply do this:
```
df.loc(axis=0)[:, :, 'C1', :]
```
Pandas gets confused sometimes when indexes are similar or contain similar values. If you were to have a column named 'C1' or something you would also need to do this under this style of slicing/selecting.


Question:
Convert pandas Series to DataFrame I have a Pandas series sf:
```
email

[email protected]
    [1.0, 0.0, 0.0]

[email protected]
    [2.0, 0.0, 0.0]

[email protected]
    [1.0, 0.0, 0.0]

[email protected]
    [4.0, 0.0, 0.0]

[email protected]
    [1.0, 0.0, 3.0]

[email protected]
    [1.0, 5.0, 0.0]
```
And I would like to transform it to the following DataFrame:
```
index | email             | list
_____________________________________________
0     |
[email protected]
  | [1.0, 0.0, 0.0]
1     |
[email protected]
  | [2.0, 0.0, 0.0]
2     |
[email protected]
  | [1.0, 0.0, 0.0]
3     |
[email protected]
  | [4.0, 0.0, 0.0]
4     |
[email protected]
  | [1.0, 0.0, 3.0]
5     |
[email protected]
  | [1.0, 5.0, 0.0]
```
I found a way to do it, but I doubt it's the more efficient one:
```
df1 = pd.DataFrame(data=sf.index, columns=['email'])
df2 = pd.DataFrame(data=sf.values, columns=['list'])
df = pd.merge(df1, df2, left_index=True, right_index=True)
```

Response:
Rather than create 2 temporary dfs you can just pass these as params within a dict using the DataFrame constructor:
```
pd.DataFrame({'email':sf.index, 'list':sf.values})
```
There are lots of ways to construct a df, see the docs


Question:
How to query MultiIndex index columns values in pandas Code example:
```

```python
A = np.array([1.1, 1.1, 3.3, 3.3, 5.5, 6.6])
```

```python
B = np.array([111, 222, 222, 333, 333, 777])
```

```python
C = randint(10, 99, 6)
```

```python
df = pd.DataFrame(zip(A, B, C), columns=['A', 'B', 'C'])
```

```python
df.set_index(['A', 'B'], inplace=True)
```

```python
df
#Output
#C
#A   B
#1.1 111  20
#    222  31
#3.3 222  24
#    333  65
#5.5 333  22
#6.6 777  74
#```
#Now, I want to retrieve A values: Q1: in range [3.3, 6.6] - expected return value: [3.3, 5.5, 6.6] or [3.3, 3.3, 5.5, 6.6] in case last inclusive, and [3.3, 5.5] or [3.3, 3.3, 5.5] if not. Q2: in range [2.0, 4.0] - expected return value: [3.3] or [3.3, 3.3] Same for any other MultiIndex dimension, for example B values: Q3: in range [111, 500] with repetitions, as number of data rows in range - expected return value: [111, 222, 222, 333, 333] More formal: Let us assume T is a table with columns A, B and C. The table includes n rows. Table cells are numbers, for example A double, B and C integers. Let's create a DataFrame of table T, let us name it DF. Let's set columns A and B indexes of DF (without duplication, i.e. no separate columns A and B as indexes, and separate as data), i.e. A and B in this case MultiIndex. Questions: How to write a query on the index, for example, to query the index A (or B), say in the labels interval [120.0, 540.0]? Labels 120.0 and 540.0 exist. I must clarify that I am interested only in the list of indices as a response to the query! How to the same, but in case of the labels 120.0 and 540.0 do not exist, but there are labels by value lower than 120, higher than 120 and less than 540, or higher than 540? In case the answer for Q1 and Q2 was unique index values, now the same, but with repetitions, as number of data rows in index range. I know the answers to the above questions in the case of columns which are not indexes, but in the indexes case, after a long research in the web and experimentation with the functionality of pandas, I did not succeed. The only method (without additional programming) I see now is to have a duplicate of A and B as data columns in addition to index.
```

Response:
To query the df by the MultiIndex values, for example where (A > 1.7) and (B < 666):
```

```python
result_df = df.loc[(df.index.get_level_values('A') > 1.7) & (df.index.get_level_values('B') < 666)]
```

```python
result_df
#Output
#C
#A   B
#3.3 222  43
#    333  59
#5.5 333  56
#```
#Hence, to get for example the 'A' index values, if still required:
#```
```

```python
result_df.index.get_level_values('A')
#Output
#Index([3.3, 3.3, 5.5], dtype=object)
#```
#The problem is, that in large data frames the performance of by index selection worse by 10% than the sorted regular rows selection. And in repetitive work, looping, the delay accumulated. See example:
#```
```

```python
df = store.select(STORE_EXTENT_BURSTS_DF_KEY)
```

```python
len(df)
#Output
#12857
```

```python
df.sort(inplace=True)
```

```python
df_without_index = df.reset_index()
```

```python
%timeit df.loc[(df.index.get_level_values('END_TIME') > 358200) & (df.index.get_level_values('START_TIME') < 361680)]
1000 loops, best of 3: 562 µs per loop
```

```python
%timeit df_without_index[(df_without_index.END_TIME > 358200) & (df_without_index.START_TIME < 361680)]
1000 loops, best of 3: 507 µs per loop
```
```


Question: How to resolve AttributeError: 'DataFrame' object has no attribute I know that this kind of question was asked before and I've checked all the answers and I have tried several times to find a solution but in vain. In fact I call a Dataframe using Pandas. I've uploaded a csv.file. When I type data.Country and data.Year, I get the 1st Column and the second one displayed. However when I type data.Number, everytime it gives me this error: AttributeError: 'DataFrame' object has no attribute 'Number'.

Response:
Check your DataFrame with data.columns It should print something like this
```
Index([u'regiment', u'company',  u'name',u'postTestScore'], dtype='object')
```
Check for hidden white spaces..Then you can rename with
```
data = data.rename(columns={'Number ': 'Number'})
```


Question:
splitting at underscore in python and storing the first value I have a pandas data frame like df with a column construct_name
```
construct_name
aaaa_t1_2
cccc_t4_10
bbbb_g3_3
```
and so on. I want to first split all the names at the underscore and store the first element (aaaa,cccc, etc.) as another column name. Expected output
```
construct_name  name
aaaa_t1_2       aaaa
cccc_t4_10      bbbb
```
and so on. I tried the following df['construct_name'].map(lambda row:row.split("_")) and it gives me a list like
```
[aaaa,t1,2]
[cccc,t4,10]
```
and so on But when I do df['construct_name'].map(lambda row:row.split("_"))[0] to get the first element of the list I get an error. Can you suggest a fix. Thanks

Response:
Just use the vectorised str method split and use integer indexing on the list to get the first element:
```

```python
df['first'] = df['construct_name'].str.split('_').str[0]
df
#Output
#construct_name first
#0      aaaa_t1_2  aaaa
#1     cccc_t4_10  cccc
#2      bbbb_g3_3  bbbb
#```
```


Question:
Python Pandas - Changing some column types to categories I have fed the following CSV file into iPython Notebook:
```
public = pd.read_csv("categories.csv")
public
```
I've also imported pandas as pd, numpy as np and matplotlib.pyplot as plt. The following data types are present (the below is a summary - there are about 100 columns)
```

```python
public.dtypes
#Output
#parks          object
#           playgrounds    object
#           sports         object
#           roading        object
#           resident       int64
#           children       int64
#```
#I want to change 'parks', 'playgrounds', 'sports' and 'roading' to categories (they have likert scale responses in them - each column has different types of likert responses though (e.g. one has "strongly agree", "agree" etc., another has "very important", "important" etc.), leaving the remainder as int64. I was able to create a separate dataframe - public1 - and change one of the columns to a category type using the following code:
#```
#public1 = {'parks': public.parks}
#public1 = public1['parks'].astype('category')
#```
#However, when I tried to change a number at once using this code, I was unsuccessful:
#```
#public1 = {'parks': public.parks,
#           'playgrounds': public.parks}
#public1 = public1['parks', 'playgrounds'].astype('category')
#```
#Notwithstanding this, I don't want to create a separate dataframe with just the categories columns. I would like them changed in the original dataframe. I tried numerous ways to achieve this, then tried the code here: Change column type in pandas.
#```
#public[['parks', 'playgrounds', 'sports', 'roading']] = public[['parks', 'playgrounds', 'sports', 'roading']].astype('category')
#```
#and got the following error:
#```
#NotImplementedError: > 1 ndim Categorical are not supported at this time
#```
#Is there a way to change 'parks', 'playgrounds', 'sports', 'roading' to categories (so the likert scale responses can then be analysed), leaving 'resident' and 'children' (and the 94 other columns that are string, int + floats) untouched? I am using Python 2.7.
```

Response:
Sometimes, you just have to use a for-loop:
```
for col in ['parks', 'playgrounds', 'sports', 'roading']:
    public[col] = public[col].astype('category')
```


Question:
How to remove levels from a multi-indexed dataframe? For example, I have:
```

```python
df = pd.DataFrame([8, 9],
                          index=pd.MultiIndex.from_tuples([(1, 1, 1),
                                                           (1, 3, 2)]),
                          columns=['A'])

In [2] df
#Output
#A
#1 1 1  8
#  3 2  9
#```
#Is there a better way to remove the last level from the index than this:
#```
```

```python
pd.DataFrame(df.values,
                     index=df.index.droplevel(2),
                     columns=df.columns)
#Output
#A
#1 1  8
#  3  9
#```
```

Response:
```
df.reset_index(level=2, drop=True)
Out[29]:
     A
1 1  8
  3  9
```


Question:
Plotting categorical data with pandas and matplotlib I have a data frame with categorical data:
```
colour  direction
1    red     up
2    blue    up
3    green   down
4    red     left
5    red     right
6    yellow  down
7    blue    down
```
I want to generate some graphs, like pie charts and histograms based on the categories. Is it possible without creating dummy numeric variables? Something like
```
df.plot(kind='hist')
```

Response:
You can simply use value_counts on the series:
```
df['colour'].value_counts().plot(kind='bar')
```


Question:
Turn Pandas Multi-Index into column I have a dataframe with 2 index levels:
```
value
Trial    measurement
    1              0        13
                   1         3
                   2         4
    2              0       NaN
                   1        12
    3              0        34
```
Which I want to turn into this:
```
Trial    measurement       value

    1              0        13
    1              1         3
    1              2         4
    2              0       NaN
    2              1        12
    3              0        34
```
How can I best do this? I need this because I want to aggregate the data as instructed here, but I can't select my columns like that if they are in use as indices.

Response:
The reset_index() is a pandas DataFrame method that will transfer index values into the DataFrame as columns. The default setting for the parameter is drop=False (which will keep the index values as columns). All you have to do call .reset_index() after the name of the DataFrame:
```
df = df.reset_index()
```


Question:
Unpivot Pandas Data I currently have a DataFrame laid out as:
```
Jan Feb Mar Apr ...
2001    1   12  12  19
2002    9   ...
2003    ...
```
and I would like to "unpivot" the data to look like:
```
Date    Value
Jan 2001    1
Feb 2001    1
Mar 2001    12
...
Jan 2002    9
```
What is the best way to accomplish this using pandas/NumPy?

Response:
You just have to do df.unstack() and that will create a MultiIndexed Series with month as a first level and the year as the second level index. If you want them to be columns then just call reset_index() after that.
```py

```python
df
#Output
#      Jan  Feb
#2001    3    4
#2002    2    7
```

```python
df.unstack()
#Output
#Jan  2001    3
#     2002    2
#Feb  2001    4
#     2002    7
```

```python
df = df.unstack().reset_index(name='value')
```

```python
df
#Output
#  level_0  level_1  value
#0     Jan     2001      3
#1     Jan     2002      2
#2     Feb     2001      4
#3     Feb     2002      7
```

```python
df.rename(columns={'level_0': 'month', 'level_1': 'year'}, inplace=True)
```

```python
df
#Output
#  month  year  value
#0   Jan  2001      3
#1   Jan  2002      2
#2   Feb  2001      4
#3   Feb  2002      7
#```
```


Question:
Pandas groupby with categories with redundant nan I am having issues using pandas groupby with categorical data. Theoretically, it should be super efficient: you are grouping and indexing via integers rather than strings. But it insists that, when grouping by multiple categories, every combination of categories must be accounted for. I sometimes use categories even when there's a low density of common strings, simply because those strings are long and it saves memory / improves performance. Sometimes there are thousands of categories in each column. When grouping by 3 columns, pandas forces us to hold results for 1000^3 groups. My question: is there a convenient way to use groupby with categories while avoiding this untoward behaviour? I'm not looking for any of these solutions: Recreating all the functionality via numpy. Continually converting to strings/codes before groupby, reverting to categories later. Making a tuple column from group columns, then group by the tuple column. I'm hoping there's a way to modify just this particular pandas idiosyncrasy. A simple example is below. Instead of 4 categories I want in the output, I end up with 12.
```
import pandas as pd

group_cols = ['Group1', 'Group2', 'Group3']

df = pd.DataFrame([['A', 'B', 'C', 54.34],
                   ['A', 'B', 'D', 61.34],
                   ['B', 'A', 'C', 514.5],
                   ['B', 'A', 'A', 765.4],
                   ['A', 'B', 'D', 765.4]],
                  columns=(group_cols+['Value']))

for col in group_cols:
    df[col] = df[col].astype('category')

df.groupby(group_cols, as_index=False).sum()

Group1  Group2  Group3  Value
#   A   A   A   NaN
#   A   A   C   NaN
#   A   A   D   NaN
#   A   B   A   NaN
#   A   B   C   54.34
#   A   B   D   826.74
#   B   A   A   765.40
#   B   A   C   514.50
#   B   A   D   NaN
#   B   B   A   NaN
#   B   B   C   NaN
#   B   B   D   NaN
```
Bounty update The issue is poorly addressed by pandas development team (cf github.com/pandas-dev/pandas/issues/17594). Therefore, I am looking for responses that address any of the following: Why, with reference to pandas source code, is categorical data treated differently in groupby operations? Why would the current implementation be preferred? I appreciate this is subjective, but I am struggling to find any answer to this question. Current behaviour is prohibitive in many situations without cumbersome, potentially expensive, workarounds. Is there a clean solution to override pandas treatment of categorical data in groupby operations? Note the 3 no-go routes (dropping down to numpy; conversions to/from codes; creating and grouping by tuple columns). I would prefer a solution that is "pandas-compliant" to minimise / avoid loss of other pandas categorical functionality. A response from pandas development team to support and clarify existing treatment. Also, why should considering all category combinations not be configurable as a Boolean parameter? Bounty update #2 To be clear, I'm not expecting answers to all of the above 4 questions. The main question I am asking is whether it's possible, or advisable, to overwrite pandas library methods so that categories are treated in a way that facilitates groupby / set_index operations.

Response:
Since Pandas 0.23.0, the groupby method can now take a parameter observed which fixes this issue if it is set to True (False by default). Below is the exact same code as in the question with just observed=True added :
```
import pandas as pd

group_cols = ['Group1', 'Group2', 'Group3']

df = pd.DataFrame([['A', 'B', 'C', 54.34],
                   ['A', 'B', 'D', 61.34],
                   ['B', 'A', 'C', 514.5],
                   ['B', 'A', 'A', 765.4],
                   ['A', 'B', 'D', 765.4]],
                  columns=(group_cols+['Value']))

for col in group_cols:
    df[col] = df[col].astype('category')

df.groupby(group_cols, as_index=False, observed=True).sum()
```


Question:
Add column in dataframe from list I have a dataframe with some columns like this:
```
A   B   C
0
4
5
6
7
7
6
5
```
The possible range of values in A are only from 0 to 7. Also, I have a list of 8 elements like this:
```
List=[2,5,6,8,12,16,26,32]  //There are only 8 elements in this list
```
If the element in column A is n, I need to insert the n th element from the List in a new column, say 'D'. How can I do this in one go without looping over the whole dataframe? The resulting dataframe would look like this:
```
A   B   C   D
0           2
4           12
5           16
6           26
7           32
7           32
6           26
5           16
```
Note: The dataframe is huge and iteration is the last option option. But I can also arrange the elements in 'List' in any other data structure like dict if necessary.

Response:
Just assign the list directly:
```
df['new_col'] = mylist
```
Alternative Convert the list to a series or array and then assign:
```
se = pd.Series(mylist)
df['new_col'] = se.values
```
or
```
df['new_col'] = np.array(mylist)
```


Question:
How to apply a function on every row on a dataframe? I am new to Python and I am not sure how to solve the following problem. I have a function:
```
def EOQ(D,p,ck,ch):
    Q = math.sqrt((2*D*ck)/(ch*p))
    return Q
```
Say I have the dataframe
```
df = pd.DataFrame({"D": [10,20,30], "p": [20, 30, 10]})

    D   p
0   10  20
1   20  30
2   30  10

ch=0.2
ck=5
```
And ch and ck are float types. Now I want to apply the formula to every row on the dataframe and return it as an extra row 'Q'. An example (that does not work) would be:
```
df['Q']= map(lambda p, D: EOQ(D,p,ck,ch),df['p'], df['D'])
```
(returns only 'map' types) I will need this type of processing more in my project and I hope to find something that works.

Response:
The following should work:
```
def EOQ(D,p,ck,ch):
    Q = math.sqrt((2*D*ck)/(ch*p))
    return Q
ch=0.2
ck=5
df['Q'] = df.apply(lambda row: EOQ(row['D'], row['p'], ck, ch), axis=1)
df
```
If all you're doing is calculating the square root of some result then use the np.sqrt method this is vectorised and will be significantly faster:
```

```python
df['Q'] = np.sqrt((2*df['D']*ck)/(ch*df['p']))

df
#Output
#D   p          Q
#0  10  20   5.000000
#1  20  30   5.773503
#2  30  10  12.247449
#```
#Timings For a 30k row df:
#```
```

```python
import math
ch=0.2
ck=5
def EOQ(D,p,ck,ch):
    Q = math.sqrt((2*D*ck)/(ch*p))
    return Q

%timeit np.sqrt((2*df['D']*ck)/(ch*df['p']))
%timeit df.apply(lambda row: EOQ(row['D'], row['p'], ck, ch), axis=1)
1000 loops, best of 3: 622 µs per loop
1 loops, best of 3: 1.19 s per loop
```
You can see that the np method is ~1900 X faster
```


Question:
How to lowercase a pandas dataframe string column if it has missing values? The following code does not work.
```
import pandas as pd
import numpy as np
df=pd.DataFrame(['ONE','Two', np.nan],columns=['x'])
xLower = df["x"].map(lambda x: x.lower())
```
How should I tweak it to get xLower = ['one','two',np.nan] ? Efficiency is important since the real data frame is huge.

Response:
use pandas vectorized string methods; as in the documentation: these methods exclude missing/NA values automatically .str.lower() is the very first example there;
```

```python
df['x'].str.lower()
#Output
#0    one
#1    two
#2    NaN
#Name: x, dtype: object
#```
```


Question:
Reverse a get_dummies encoding in pandas Column names are: ID,1,2,3,4,5,6,7,8,9. The col values are either 0 or 1 My dataframe looks like this:
```
ID     1    2    3    4    5    6   7   8   9

1002    0    1    0    1    0    0   0   0   0
1003    0    0    0    0    0    0   0   0   0
1004    1    1    0    0    0    0   0   0   0
1005    0    0    0    0    1    0   0   0   0
1006    0    0    0    0    0    1   0   0   0
1007    1    0    1    0    0    0   0   0   0
1000    0    0    0    0    0    0   0   0   0
1009    0    0    1    0    0    0   1   0   0
```
I want the column names in front of the ID where the value in a row is 1. The Dataframe i want should look like this:
```
ID      Col2
1002       2    // has 1 at Col(2) and Col(4)
1002       4
1004       1    // has 1 at col(1) and col(2)
1004       2
1005       5    // has 1 at col(5)
1006       6    // has 1 at col(6)
1007       1    // has 1 at col(1) and col(3)
1007       3
1009       3    // has 1 at col(3) and col(7)
1009       7
```
Please help me in this, Thanks in advance

Response:
Several great answers for the OP post. However, often get_dummies is used for multiple categorical features. Pandas uses a prefix separator prefix_sep to distinguish different values for a column. The following function collapses a "dummified" dataframe while keeping the order of columns:
```py
def undummify(df, prefix_sep="_"):
    cols2collapse = {
        item.split(prefix_sep)[0]: (prefix_sep in item) for item in df.columns
    }
    series_list = []
    for col, needs_to_collapse in cols2collapse.items():
        if needs_to_collapse:
            undummified = (
                df.filter(like=col)
                .idxmax(axis=1)
                .apply(lambda x: x.split(prefix_sep, maxsplit=1)[1])
                .rename(col)
            )
            series_list.append(undummified)
        else:
            series_list.append(df[col])
    undummified_df = pd.concat(series_list, axis=1)
    return undummified_df
```
Example
```

```python
df
#Output
#     a    b    c
#0  A_1  B_1  C_1
#1  A_2  B_2  C_2
```

```python
df2 = pd.get_dummies(df)
```

```python
df2
#Output
#   a_A_1  a_A_2  b_B_1  b_B_2  c_C_1  c_C_2
#0      1      0      1      0      1      0
#1      0      1      0      1      0      1
```

```python
df3 = undummify(df2)
```

```python
df3
#Output
#     a    b    c
#0  A_1  B_1  C_1
#1  A_2  B_2  C_2
#```
```


Question:
Convert pandas series into numpy array [duplicate] This question already has answers here: How do I convert a Pandas series or index to a NumPy array? [duplicate] (8 answers) Closed 6 years ago. I am new to pandas and python. My input data is like
```
category   text
1   hello iam fine. how are you
1   iam good. how are you doing.

inputData= pd.read_csv(Input', sep='\t', names=['category','text'])
X = inputData["text"]
Y = inputData["category"]
```
here Y is the panda series object, which i want to convert into numpy array. so i tried .as_matrix
```
YArray= Y.as_matrix(columns=None)
print YArray
```
But i got the output as [1,1] (which is wrong since i have only one column category and two rows). I want the result as 2x1 matrix.

Response:
To get numpy array, you need
```
Y.values
```


Question:
pandas dataframe str.contains() AND operation I have a df (Pandas Dataframe) with three rows:
```
some_col_name
"apple is delicious"
"banana is delicious"
"apple and banana both are delicious"
```
The function df.col_name.str.contains("apple|banana") will catch all of the rows:
```
"apple is delicious",
"banana is delicious",
"apple and banana both are delicious".
```
How do I apply AND operator to the str.contains() method, so that it only grabs strings that contain BOTH "apple" & "banana"?
```
"apple and banana both are delicious"
```
I'd like to grab strings that contains 10-20 different words (grape, watermelon, berry, orange, ..., etc.)

Response:
You can do that as follows:
```
df[(df['col_name'].str.contains('apple')) & (df['col_name'].str.contains('banana'))]
```


Question:
Pandas - Filtering None Values I'm using Pandas to explore some datasets. I have this dataframe: I want to exclude any row that has a value in column City. So I've tried:
```
new_df = all_df[(all_df["City"] == "None") ]
new_df
```
But then I got an empty dataframe: It works whenever I use any value other than None. Any idea how to filter this dataframe?

Response:
Consider using isnull() to locate missing values
```
all_df[all_df['City'].isnull()]
```


Question: How do I check if a pandas DataFrame is empty? How do I check if a pandas DataFrame is empty? I'd like to print some message in the terminal if the DataFrame is empty.

Response:
You can use the attribute df.empty to check whether it's empty or not:
```
if df.empty:
    print('DataFrame is empty!')
```
Source: Pandas Documentation


Question:
Setting Yaxis in Matplotlib using Pandas Using Pandas to plot in IPython Notebook, I have several plots and because Matplotlib decides the Y axis it is setting them differently and we need to compare that data using the same range. From the Matplotlib doc it seems that I need to set ylim, but can't figure the syntax to do so. I have tried several variants on:
```
df2250.plot(); plt.ylim((100000,500000))
df2260.plot()
df5.plot()
```
I assume I'll need to apply the limits to each plot, but since I can't get one working...

Response:
DataFrame.plot() exposes a ylim parameter that sets the y axis limits:
```
df.plot(ylim=(0, 200))
```
I'm guessing this feature was added after Rutger's answer was accepted in 2013.


Question:
Making a chart bigger in size I'm trying to get a bigger chart. However, the figure method from matplotlib does not seem to be working properly. I get a message, which is not an error:
```none
<matplotlib.figure.Figure at 0xa25f7f0>
```
My code:
```py
import pandas.io.data as web
import pandas as pd
import matplotlib.pyplot as plt

%matplotlib inline
...
plt.figure(figsize=(20,10))
df2['media']= df2['SPY']*.6 + df2['TLT']*.4
df2.plot()
plt.show()
```
What's wrong with my code?

Response:
You can skip the first plt.figure() and just use the argument figsize:
```
df2.plot(figsize=(20,10))
```
See docs.


Question:
float64 with pandas to_csv I'm reading a CSV with float numbers like this:
```
Bob,0.085
Alice,0.005
```
And import into a dataframe, and write this dataframe to a new place
```
df = pd.read_csv(orig)
df.to_csv(pandasfile)
```
Now this pandasfile has:
```
Bob,0.085000000000000006
Alice,0.0050000000000000001
```
What happen? maybe I have to cast to a different type like float32 or something? Im using pandas 0.9.0 and numpy 1.6.2.

Response:
As mentioned in the comments, it is a general floating point problem. However you can use the float_format key word of to_csv to hide it:
```
df.to_csv('pandasfile.csv', float_format='%.3f')
```
or, if you don't want 0.0001 to be rounded to zero:
```
df.to_csv('pandasfile.csv', float_format='%g')
```
will give you:
```
Bob,0.085
Alice,0.005
```
in your output file. For an explanation of %g, see Format Specification Mini-Language.


Question:
Pandas groupby for zero values I have data like this in a csv file
```
Symbol  Action  Year
  AAPL     Buy  2001
  AAPL     Buy  2001
   BAC    Sell  2002
   BAC    Sell  2002
```
I am able to read it and groupby like this
```
df.groupby(['Symbol','Year']).count()
```
I get
```
Action
Symbol Year
AAPL   2001       2
BAC    2002       2
```
I desire this (order does not matter)
```
Action
Symbol Year
AAPL   2001       2
AAPL   2002       0
BAC    2001       0
BAC    2002       2
```
I want to know if its possible to count for zero occurances

Response:
You can use this:
```
df = df.groupby(['Symbol','Year']).count().unstack(fill_value=0).stack()
print (df)
```
Output:
```
Action
Symbol Year
AAPL   2001       2
       2002       0
BAC    2001       0
       2002       2
```


Question:
Apply function to pandas groupby I have a pandas dataframe with a column called my_labels which contains strings: 'A', 'B', 'C', 'D', 'E'. I would like to count the number of occurances of each of these strings then divide the number of counts by the sum of all the counts. I'm trying to do this in Pandas like this:
```
func = lambda x: x.size() / x.sum()
data = frame.groupby('my_labels').apply(func)
```
This code throws an error, 'DataFrame object has no attribute 'size'. How can I apply a function to calculate this in Pandas?

Response:
apply takes a function to apply to each value, not the series, and accepts kwargs. So, the values do not have the .size() method. Perhaps this would work:
```
from pandas import *

d = {"my_label": Series(['A','B','A','C','D','D','E'])}
df = DataFrame(d)

def as_perc(value, total):
    return value/float(total)

def get_count(values):
    return len(values)

grouped_count = df.groupby("my_label").my_label.agg(get_count)
data = grouped_count.apply(as_perc, total=df.my_label.count())
```
The .agg() method here takes a function that is applied to all values of the groupby object.


Question:
Concatenate a list of pandas dataframes together I have a list of Pandas dataframes that I would like to combine into one Pandas dataframe. I am using Python 2.7.10 and Pandas 0.16.2 I created the list of dataframes from:
```
import pandas as pd
dfs = []
sqlall = "select * from mytable"

for chunk in pd.read_sql_query(sqlall , cnxn, chunksize=10000):
    dfs.append(chunk)
```
This returns a list of dataframes
```
type(dfs[0])
Out[6]: pandas.core.frame.DataFrame

type(dfs)
Out[7]: list

len(dfs)
Out[8]: 408
```
Here is some sample data
```
# sample dataframes
d1 = pd.DataFrame({'one' : [1., 2., 3., 4.], 'two' : [4., 3., 2., 1.]})
d2 = pd.DataFrame({'one' : [5., 6., 7., 8.], 'two' : [9., 10., 11., 12.]})
d3 = pd.DataFrame({'one' : [15., 16., 17., 18.], 'two' : [19., 10., 11., 12.]})

# list of dataframes
mydfs = [d1, d2, d3]
```
I would like to combine d1, d2, and d3 into one pandas dataframe. Alternatively, a method of reading a large-ish table directly into a dataframe when using the chunksize option would be very helpful.

Response:
Given that all the dataframes have the same columns, you can simply concat them:
```
import pandas as pd
df = pd.concat(list_of_dataframes)
```