Question:
How to GroupBy a Dataframe in Pandas and keep Columns [duplicate] This question already has answers here: Converting a Pandas GroupBy multiindex output from Series back to DataFrame (13 answers) Closed 1 year ago. Given a dataframe that logs uses of some books like this:
```
Name   Type   ID
Book1  ebook  1
Book2  paper  2
Book3  paper  3
Book1  ebook  1
Book2  paper  2
```
I need to get the count of all the books, keeping the other columns and get this:
```
Name   Type   ID    Count
Book1  ebook  1     2
Book2  paper  2     2
Book3  paper  3     1
```
How can this be done?

Response:
You want the following:
```

```python
df.groupby(['Name','Type','ID']).count().reset_index()
#Output
#Name   Type  ID  Count
#0  Book1  ebook   1      2
#1  Book2  paper   2      2
#2  Book3  paper   3      1
#```
#In your case the 'Name', 'Type' and 'ID' cols match in values so we can groupby on these, call count and then reset_index. An alternative approach would be to add the 'Count' column using transform and then call drop_duplicates:
#```
```

```python
df['Count'] = df.groupby(['Name'])['ID'].transform('count')
df.drop_duplicates()
#Output
#Name   Type  ID  Count
#0  Book1  ebook   1      2
#1  Book2  paper   2      2
#2  Book3  paper   3      1
#```
```


Question:
Find integer index of rows with NaN in pandas dataframe I have a pandas DataFrame like this:
```
a         b
2011-01-01 00:00:00 1.883381  -0.416629
2011-01-01 01:00:00 0.149948  -1.782170
2011-01-01 02:00:00 -0.407604 0.314168
2011-01-01 03:00:00 1.452354  NaN
2011-01-01 04:00:00 -1.224869 -0.947457
2011-01-01 05:00:00 0.498326  0.070416
2011-01-01 06:00:00 0.401665  NaN
2011-01-01 07:00:00 -0.019766 0.533641
2011-01-01 08:00:00 -1.101303 -1.408561
2011-01-01 09:00:00 1.671795  -0.764629
```
Is there an efficient way to find the "integer" index of rows with NaNs? In this case the desired output should be [3, 6].

Response:
Here is a simpler solution: inds = pd.isnull(df).any(1).nonzero()[0]
```

```python
df
#Output
#0         1
#0  0.450319  0.062595
#1 -0.673058  0.156073
#2 -0.871179 -0.118575
#3  0.594188       NaN
#4 -1.017903 -0.484744
#5  0.860375  0.239265
#6 -0.640070       NaN
#7 -0.535802  1.632932
#8  0.876523 -0.153634
#9 -0.686914  0.131185
```

```python
pd.isnull(df).any(1).nonzero()[0]
#Output
#array([3, 6])
#```
```


Question:
pandas read_csv and filter columns with usecols I have a csv file which isn't coming in correctly with pandas.read_csv when I filter the columns with usecols and use multiple indexes.
```
import pandas as pd
csv = r"""dummy,date,loc,x
   bar,20090101,a,1
   bar,20090102,a,3
   bar,20090103,a,5
   bar,20090101,b,1
   bar,20090102,b,3
   bar,20090103,b,5"""

f = open('foo.csv', 'w')
f.write(csv)
f.close()

df1 = pd.read_csv('foo.csv',
        header=0,
        names=["dummy", "date", "loc", "x"],
        index_col=["date", "loc"],
        usecols=["dummy", "date", "loc", "x"],
        parse_dates=["date"])
print df1

# Ignore the dummy columns
df2 = pd.read_csv('foo.csv',
        index_col=["date", "loc"],
        usecols=["date", "loc", "x"], # <----------- Changed
        parse_dates=["date"],
        header=0,
        names=["dummy", "date", "loc", "x"])
print df2
```
I expect that df1 and df2 should be the same except for the missing dummy column, but the columns come in mislabeled. Also the date is getting parsed as a date.
```

```python
%run test.py
               dummy  x
date       loc
2009-01-01 a     bar  1
2009-01-02 a     bar  3
2009-01-03 a     bar  5
2009-01-01 b     bar  1
2009-01-02 b     bar  3
2009-01-03 b     bar  5
              date
date loc
a    1    20090101
     3    20090102
     5    20090103
b    1    20090101
     3    20090102
     5    20090103
```
Using column numbers instead of names give me the same problem. I can workaround the issue by dropping the dummy column after the read_csv step, but I'm trying to understand what is going wrong. I'm using pandas 0.10.1. edit: fixed bad header usage.
```

Response:
The solution lies in understanding these two keyword arguments: names is only necessary when there is no header row in your file and you want to specify other arguments (such as usecols) using column names rather than integer indices. usecols is supposed to provide a filter before reading the whole DataFrame into memory; if used properly, there should never be a need to delete columns after reading. So because you have a header row, passing header=0 is sufficient and additionally passing names appears to be confusing pd.read_csv. Removing names from the second call gives the desired output:
```
import pandas as pd
from StringIO import StringIO

csv = r"""dummy,date,loc,x
bar,20090101,a,1
bar,20090102,a,3
bar,20090103,a,5
bar,20090101,b,1
bar,20090102,b,3
bar,20090103,b,5"""

df = pd.read_csv(StringIO(csv),
        header=0,
        index_col=["date", "loc"],
        usecols=["date", "loc", "x"],
        parse_dates=["date"])
```
Which gives us:
```
x
date       loc
2009-01-01 a    1
2009-01-02 a    3
2009-01-03 a    5
2009-01-01 b    1
2009-01-02 b    3
2009-01-03 b    5
```


Question:
substring of an entire column in pandas dataframe I have a pandas dataframe "df". In this dataframe I have multiple columns, one of which I have to substring. Lets say the column name is "col". I can run a "for" loop like below and substring the column:
```
for i in range(0,len(df)):
  df.iloc[i].col = df.iloc[i].col[:9]
```
But I wanted to know, if there is an option where I don't have to use a "for" loop, and do it directly using an attribute.I have huge amount of data, and if I do this, the data will take a very long time process.

Response:
Use the str accessor with square brackets:
```
df['col'] = df['col'].str[:9]
```
Or str.slice:
```
df['col'] = df['col'].str.slice(0, 9)
```


Question:
Export from pandas to_excel without row names (index)? I'm trying to print out a dataframe from pandas into Excel. Here I am using to_excel() functions. However, I found that the 1st column in Excel is the "index",
```
0   6/6/2021 0:00   8/6/2021 0:00
1   4/10/2024 0:00  6/10/2024 0:00
2   4/14/2024 0:00  6/14/2024 0:00
```
Is there any ways to get rid of the first column?

Response: You need to set index=False in to_excel in order for it to not write the index column out, this semantic is followed in other Pandas IO tools, see http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.to_excel.html and http://pandas.pydata.org/pandas-docs/stable/io.html


Question:
How to convert string to datetime format in pandas? I have a column I_DATE of type string (object) in a dataframe called train as show below.
```none
I_DATE
28-03-2012 2:15:00 PM
28-03-2012 2:17:28 PM
28-03-2012 2:50:50 PM
```
How to convert I_DATE from string to datetime format & specify the format of input string. Also, how to filter rows based on a range of dates in pandas?

Response:
Use to_datetime. There is no need to specify the format in this case since the parser is able to figure it out.
```

```python
pd.to_datetime(df['I_DATE'])
#Output
#0   2012-03-28 14:15:00
#1   2012-03-28 14:17:28
#2   2012-03-28 14:50:50
#Name: I_DATE, dtype: datetime64[ns]
#```
#To access the date/day/time component use the dt accessor:
#```
```

```python
df['I_DATE'].dt.date
#Output
#0    2012-03-28
#1    2012-03-28
#2    2012-03-28
#dtype: object
```

```python
df['I_DATE'].dt.time
#Output
#0    14:15:00
#1    14:17:28
#2    14:50:50
#dtype: object
#```
#You can use strings to filter as an example:
#```
```

```python
df = pd.DataFrame({'date':pd.date_range(start = dt.datetime(2015,1,1), end = dt.datetime.now())})
df[(df['date'] > '2015-02-04') & (df['date'] < '2015-02-10')]
#Output
#date
#35 2015-02-05
#36 2015-02-06
#37 2015-02-07
#38 2015-02-08
#39 2015-02-09
#```
```


Question:
Python: Pandas Dataframe how to multiply entire column with a scalar How do I multiply each element of a given column of my dataframe with a scalar? (I have tried looking on SO, but cannot seem to find the right solution) Doing something like:
```
df['quantity'] *= -1 # trying to multiply each row's quantity column with -1
```
gives me a warning:
```
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead
```
Note: If possible, I do not want to be iterating over the dataframe and do something like this...as I think any standard math operation on an entire column should be possible w/o having to write a loop:
```
for idx, row in df.iterrows():
    df.loc[idx, 'quantity'] *= -1
```
EDIT: I am running 0.16.2 of Pandas full trace:
```
SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
  self.obj[item] = s
```

Response:
Note: for those using pandas 0.20.3 and above, and are looking for an answer, all these options will work:
```
df = pd.DataFrame(np.ones((5,6)),columns=['one','two','three',
                                       'four','five','six'])
df.one *=5
df.two = df.two*5
df.three = df.three.multiply(5)
df['four'] = df['four']*5
df.loc[:, 'five'] *=5
df.iloc[:, 5] = df.iloc[:, 5]*5
```
which results in
```
one  two  three  four  five  six
0  5.0  5.0    5.0   5.0   5.0  5.0
1  5.0  5.0    5.0   5.0   5.0  5.0
2  5.0  5.0    5.0   5.0   5.0  5.0
3  5.0  5.0    5.0   5.0   5.0  5.0
4  5.0  5.0    5.0   5.0   5.0  5.0
```


Question:
Add multiple empty columns to pandas DataFrame How do I add multiple empty columns to a DataFrame from a list? I can do:
```
df["B"] = None
df["C"] = None
df["D"] = None
```
But I can't do:
```
df[["B", "C", "D"]] = None
```
KeyError: "['B' 'C' 'D'] not in index"

Response:
You could use df.reindex to add new columns:
```

```python
df = pd.DataFrame(np.random.randint(10, size=(5,1)), columns=['A'])
```

```python
df
#Output
#A
#0  4
#1  7
#2  0
#3  7
#4  6
```

```python
df.reindex(columns=list('ABCD'))
#Output
#A   B   C   D
#0  4 NaN NaN NaN
#1  7 NaN NaN NaN
#2  0 NaN NaN NaN
#3  7 NaN NaN NaN
#4  6 NaN NaN NaN
#```
#reindex will return a new DataFrame, with columns appearing in the order they are listed:
#```
```

```python
df.reindex(columns=list('DCBA'))
#Output
#D   C   B  A
#0 NaN NaN NaN  4
#1 NaN NaN NaN  7
#2 NaN NaN NaN  0
#3 NaN NaN NaN  7
#4 NaN NaN NaN  6
#```
#The reindex method as a fill_value parameter as well:
#```
```

```python
df.reindex(columns=list('ABCD'), fill_value=0)
#Output
#A  B  C  D
#0  4  0  0  0
#1  7  0  0  0
#2  0  0  0  0
#3  7  0  0  0
#4  6  0  0  0
#```
```


Question:
Pandas dataframe total row I have a dataframe, something like:
```
foo  bar  qux
0    a    1    3.14
1    b    3    2.72
2    c    2    1.62
3    d    9    1.41
4    e    3    0.58
```
and I would like to add a 'total' row to the end of dataframe:
```
foo  bar  qux
0    a    1    3.14
1    b    3    2.72
2    c    2    1.62
3    d    9    1.41
4    e    3    0.58
5  total  18   9.47
```
I've tried to use the sum command but I end up with a Series, which although I can convert back to a Dataframe, doesn't maintain the data types:
```
tot_row = pd.DataFrame(df.sum()).T
tot_row['foo'] = 'tot'
tot_row.dtypes:
     foo    object
     bar    object
     qux    object
```
I would like to maintain the data types from the original data frame as I need to apply other operations to the total row, something like:
```
baz = 2*tot_row['qux'] + 3*tot_row['bar']
```

Response:
Update June 2022 pd.append is now deprecated. You could use pd.concat instead but it's probably easier to use df.loc['Total'] = df.sum(numeric_only=True), as Kevin Zhu commented. Or, better still, don't modify the data frame in place and keep your data separate from your summary statistics! Append a totals row with
```
df.append(df.sum(numeric_only=True), ignore_index=True)
```
The conversion is necessary only if you have a column of strings or objects. It's a bit of a fragile solution so I'd recommend sticking to operations on the dataframe, though. eg.
```
baz = 2*df['qux'].sum() + 3*df['bar'].sum()
```


Question:
How to convert rows in DataFrame in Python to dictionaries For example, I have DataFrame now as
```
id    score1    score2    score3    score4  score5
 1  0.000000  0.108659  0.000000  0.078597       1
 2  0.053238  0.308253  0.286353  0.446433       1
 3  0.000000  0.083979  0.808983  0.233052       1
```
I want to convert it as
```
id scoreDict
1  {'1': 0, '2': 0.1086, ...}
2  {...}
3  {...}
```
Anyway to do that?

Response:
```
import pandas as pd

# your df
# =========================
print(df)

   id  score1  score2  score3  score4  score5
0   1  0.0000  0.1087  0.0000  0.0786       1
1   2  0.0532  0.3083  0.2864  0.4464       1
2   3  0.0000  0.0840  0.8090  0.2331       1

# to_dict
# =========================
df.to_dict(orient='records')

Out[318]:
[{'id': 1.0,
  'score1': 0.0,
  'score2': 0.10865899999999999,
  'score3': 0.0,
  'score4': 0.078597,
  'score5': 1.0},
 {'id': 2.0,
  'score1': 0.053238000000000001,
  'score2': 0.308253,
  'score3': 0.28635300000000002,
  'score4': 0.44643299999999997,
  'score5': 1.0},
 {'id': 3.0,
  'score1': 0.0,
  'score2': 0.083978999999999998,
  'score3': 0.80898300000000001,
  'score4': 0.23305200000000001,
  'score5': 1.0}]
```


Question:
pandas: to_numeric for multiple columns I'm working with the following df:
```
c.sort_values('2005', ascending=False).head(3)
      GeoName ComponentName     IndustryId IndustryClassification Description                                2004 2005  2006  2007  2008  2009 2010 2011 2012 2013 2014
37926 Alabama Real GDP by state 9          213                    Support activities for mining              99   98    117   117   115   87   96   95   103  102  (NA)
37951 Alabama Real GDP by state 34         42                     Wholesale trade                            9898 10613 10952 11034 11075 9722 9765 9703 9600 9884 10199
37932 Alabama Real GDP by state 15         327                    Nonmetallic mineral products manufacturing 980  968   940   1084  861   724  714  701  589  641  (NA)
```
I want to force numeric on all of the years:
```
c['2014'] = pd.to_numeric(c['2014'], errors='coerce')
```
is there an easy way to do this or do I have to type them all out?

Response:
UPDATE: you don't need to convert your values afterwards, you can do it on-the-fly when reading your CSV:
```

```python
df=pd.read_csv(url, index_col=0, na_values=['(NA)']).fillna(0)
```

```python
df.dtypes
#Output
#GeoName                    object
#ComponentName              object
#IndustryId                  int64
#IndustryClassification     object
#Description                object
#2004                        int64
#2005                        int64
#2006                        int64
#2007                        int64
#2008                        int64
#2009                        int64
#2010                        int64
#2011                        int64
#2012                        int64
#2013                        int64
#2014                      float64
#dtype: object
#```
#If you need to convert multiple columns to numeric dtypes - use the following technique: Sample source DF:
#```
```

```python
df
#Output
#id    a  b  c  d  e    f
#0  id_3  AAA  6  3  5  8    1
#1  id_9    3  7  5  7  3  BBB
#2  id_7    4  2  3  5  4    2
#3  id_0    7  3  5  7  9    4
#4  id_0    2  4  6  4  0    2
```

```python
df.dtypes
#Output
#id    object
#a     object
#b      int64
#c      int64
#d      int64
#e      int64
#f     object
#dtype: object
#```
#Converting selected columns to numeric dtypes:
#```
```

```python
cols = df.columns.drop('id')
```

```python
df[cols] = df[cols].apply(pd.to_numeric, errors='coerce')
```

```python
df
#Output
#id    a  b  c  d  e    f
#0  id_3  NaN  6  3  5  8  1.0
#1  id_9  3.0  7  5  7  3  NaN
#2  id_7  4.0  2  3  5  4  2.0
#3  id_0  7.0  3  5  7  9  4.0
#4  id_0  2.0  4  6  4  0  2.0
```

```python
df.dtypes
#Output
#id     object
#a     float64
#b       int64
#c       int64
#d       int64
#e       int64
#f     float64
#dtype: object
#```
#PS if you want to select all string (object) columns use the following simple trick:
#```
#cols = df.columns[df.dtypes.eq('object')]
#```
```


Question:
How do you remove the column name row when exporting a pandas DataFrame? Say I import the following Excel spreadsheet into a dataframe:
```
Val1 Val2 Val3
1     2    3
5     6    7
9     1    2
```
How do I delete the column name row (in this case Val1, Val2, Val3) so that I can export a csv with no column names, just the data? I have tried df.drop() and df.ix[1:] and have not been successful with either.

Response:
You can write to csv without the header using header=False and without the index using index=False. If desired, you also can modify the separator using sep. CSV example with no header row, omitting the header row:
```
df.to_csv('filename.csv', header=False)
```
TSV (tab-separated) example, omitting the index column:
```
df.to_csv('filename.tsv', sep='\t', index=False)
```


Question:
pandas concat columns ignore_index doesn't work I am trying to column-bind dataframes (like R's cbind() does) and having issue with pandas concat, as ignore_index=True doesn't seem to work:
```
df1 = pd.DataFrame({'A': ['A0', 'A1', 'A2', 'A3'],
                    'B': ['B0', 'B1', 'B2', 'B3'],
                    'D': ['D0', 'D1', 'D2', 'D3']},
                   index=[0, 2, 3, 4])

df2 = pd.DataFrame({'A1': ['A4', 'A5', 'A6', 'A7'],
                    'C': ['C4', 'C5', 'C6', 'C7'],
                    'D2': ['D4', 'D5', 'D6', 'D7']},
                   index=[5, 6, 7, 3])
df1
#     A   B   D
# 0  A0  B0  D0
# 2  A1  B1  D1
# 3  A2  B2  D2
# 4  A3  B3  D3

df2
#    A1   C  D2
# 5  A4  C4  D4
# 6  A5  C5  D5
# 7  A6  C6  D6
# 3  A7  C7  D7

dfs = [df1, df2]
df = pd.concat(dfs, axis=1, ignore_index=True)
print df
```
and the result is
```
0    1    2    3    4    5
0   A0   B0   D0  NaN  NaN  NaN
2   A1   B1   D1  NaN  NaN  NaN
3   A2   B2   D2   A7   C7   D7
4   A3   B3   D3  NaN  NaN  NaN
5  NaN  NaN  NaN   A4   C4   D4
6  NaN  NaN  NaN   A5   C5   D5
7  NaN  NaN  NaN   A6   C6   D6
```
Even if I reset index using
```
df1.reset_index()
df2.reset_index()
```
and then try
```
pd.concat([df1, df2], axis=1)
```
it still produces the same result! The expected result is a 6x4 dataframe where the contents of columns A,B,D, A1,C,D2 are horizontally concatenated.

Response:
If I understood you correctly, this is what you would like to do.
```
import pandas as pd

df1 = pd.DataFrame({'A': ['A0', 'A1', 'A2', 'A3'],
                    'B': ['B0', 'B1', 'B2', 'B3'],
                    'D': ['D0', 'D1', 'D2', 'D3']},
                   index=[0, 2, 3, 4])

df2 = pd.DataFrame({'A1': ['A4', 'A5', 'A6', 'A7'],
                    'C': ['C4', 'C5', 'C6', 'C7'],
                    'D2': ['D4', 'D5', 'D6', 'D7']},
                   index=[4, 5, 6 , 7])

df1.reset_index(drop=True, inplace=True)
df2.reset_index(drop=True, inplace=True)

df = pd.concat([df1, df2], axis=1)
```
Which gives:
```
A   B   D   A1  C   D2
0   A0  B0  D0  A4  C4  D4
1   A1  B1  D1  A5  C5  D5
2   A2  B2  D2  A6  C6  D6
3   A3  B3  D3  A7  C7  D7
```
Actually, I would have expected that df = pd.concat(dfs, axis=1, ignore_index=True) gives the same result. This is the excellent explanation from jreback: ignore_index=True ‘ignores’, meaning doesn’t align on the joining axis. it simply pastes them together in the order that they are passed, then reassigns a range for the actual index (e.g. range(len(index))) so the difference between joining on non-overlapping indexes (assume axis=1 in the example), is that with ignore_index=False (the default), you get the concat of the indexes, and with ignore_index=True you get a range.


Question:
Slice Pandas dataframe by index values that are (not) in a list I have a pandas dataframe, df. I want to select all indices in df that are not in a list, blacklist. Now, I use list comprehension to create the desired labels to slice.
```
ix=[i for i in df.index if i not in blacklist]
df_select=df.loc[ix]
```
Works fine, but may be clumsy if I need to do this often. Is there a better way to do this?

Response:
Use isin on the index and invert the boolean index to perform label selection:
```

```python
df = pd.DataFrame({'a':np.random.randn(5)})
df
#Output
#a
#0 -0.548275
#1 -0.411741
#2 -1.187369
#3  1.028967
#4 -2.755030
```

```python
t = [2,4]
df.loc[~df.index.isin(t)]
#Output
#a
#0 -0.548275
#1 -0.411741
#3  1.028967
#```
```


Question:
Pass percentiles to pandas agg function I want to pass the numpy percentile() function through pandas' agg() function as I do below with various other numpy statistics functions. Right now I have a dataframe that looks like this:
```
AGGREGATE   MY_COLUMN
A           10
A           12
B           5
B           9
A           84
B           22
```
And my code looks like this:
```
grouped = dataframe.groupby('AGGREGATE')
column = grouped['MY_COLUMN']
column.agg([np.sum, np.mean, np.std, np.median, np.var, np.min, np.max])
```
The above code works, but I want to do something like
```
column.agg([np.sum, np.mean, np.percentile(50), np.percentile(95)])
```
I.e., specify various percentiles to return from agg(). How should this be done?

Response:
Perhaps not super efficient, but one way would be to create a function yourself:
```
def percentile(n):
    def percentile_(x):
        return x.quantile(n)
    percentile_.__name__ = 'percentile_{:02.0f}'.format(n*100)
    return percentile_
```
Then include this in your agg:
```

```python
column.agg([np.sum, np.mean, np.std, np.median,
                     np.var, np.min, np.max, percentile(50), percentile(95)])
#Output
#sum       mean        std  median          var  amin  amax  percentile_50  percentile_95
#AGGREGATE
#A          106  35.333333  42.158431      12  1777.333333    10    84             12           76.8
#B           36  12.000000   8.888194       9    79.000000     5    22             12           76.8
#```
#Note sure this is how it should be done though...
```


Question:
Add numpy array as column to Pandas data frame I have a Pandas data frame object of shape (X,Y) that looks like this:
```
[[1, 2, 3],
[4, 5, 6],
[7, 8, 9]]
```
and a numpy sparse matrix (CSC) of shape (X,Z) that looks something like this
```
[[0, 1, 0],
[0, 0, 1],
[1, 0, 0]]
```
How can I add the content from the matrix to the data frame in a new named column such that the data frame will end up like this:
```
[[1, 2, 3, [0, 1, 0]],
[4, 5, 6, [0, 0, 1]],
[7, 8, 9, [1, 0, 0]]]
```
Notice the data frame now has shape (X, Y+1) and rows from the matrix are elements in the data frame.

Response:
```
import numpy as np
import pandas as pd
import scipy.sparse as sparse

df = pd.DataFrame(np.arange(1,10).reshape(3,3))
arr = sparse.coo_matrix(([1,1,1], ([0,1,2], [1,2,0])), shape=(3,3))
df['newcol'] = arr.toarray().tolist()
print(df)
```
yields
```
0  1  2     newcol
0  1  2  3  [0, 1, 0]
1  4  5  6  [0, 0, 1]
2  7  8  9  [1, 0, 0]
```


Question: Floor or ceiling of a pandas series in python? I have a pandas series series. If I want to get the element-wise floor or ceiling, is there a built in method or do I have to write the function and use apply? I ask because the data is big so I appreciate efficiency. Also this question has not been asked with respect to the Pandas package.

Response: You can use NumPy's built in methods to do this: np.ceil(series) or np.floor(series). Both return a Series object (not an array) so the index information is preserved.


Question:
Pandas version of rbind In R, you can combine two dataframes by sticking the columns of one onto the bottom of the columns of the other using rbind. In pandas, how do you accomplish the same thing? It seems bizarrely difficult. Using append results in a horrible mess including NaNs and things for reasons I don't understand. I'm just trying to "rbind" two identical frames that look like this: EDIT: I was creating the DataFrames in a stupid way, which was causing issues. Append=rbind to all intents and purposes. See answer below.
```
0         1       2        3          4          5        6                    7
0   ADN.L  20130220   437.4   442.37   436.5000   441.9000  2775364  2013-02-20 18:47:42
1   ADM.L  20130220  1279.0  1300.00  1272.0000  1285.0000   967730  2013-02-20 18:47:42
2   AGK.L  20130220  1717.0  1749.00  1709.0000  1739.0000   834534  2013-02-20 18:47:43
3  AMEC.L  20130220  1030.0  1040.00  1024.0000  1035.0000  1972517  2013-02-20 18:47:43
4   AAL.L  20130220  1998.0  2014.50  1942.4999  1951.0000  3666033  2013-02-20 18:47:44
5  ANTO.L  20130220  1093.0  1097.00  1064.7899  1068.0000  2183931  2013-02-20 18:47:44
6   ARM.L  20130220   941.5   965.10   939.4250   951.5001  2994652  2013-02-20 18:47:45
```
But I'm getting something horrible a la this:
```
0         1        2        3          4         5        6                    7       0         1       2        3          4          5        6                    7
0     NaN       NaN      NaN      NaN        NaN       NaN      NaN                  NaN   ADN.L  20130220   437.4   442.37   436.5000   441.9000  2775364  2013-02-20 18:47:42
1     NaN       NaN      NaN      NaN        NaN       NaN      NaN                  NaN   ADM.L  20130220  1279.0  1300.00  1272.0000  1285.0000   967730  2013-02-20 18:47:42
2     NaN       NaN      NaN      NaN        NaN       NaN      NaN                  NaN   AGK.L  20130220  1717.0  1749.00  1709.0000  1739.0000   834534  2013-02-20 18:47:43
3     NaN       NaN      NaN      NaN        NaN       NaN      NaN                  NaN  AMEC.L  20130220  1030.0  1040.00  1024.0000  1035.0000  1972517  2013-02-20 18:47:43
4     NaN       NaN      NaN      NaN        NaN       NaN      NaN                  NaN   AAL.L  20130220  1998.0  2014.50  1942.4999  1951.0000  3666033  2013-02-20 18:47:44
5     NaN       NaN      NaN      NaN        NaN       NaN      NaN                  NaN  ANTO.L  20130220  1093.0  1097.00  1064.7899  1068.0000  2183931  2013-02-20 18:47:44
6     NaN       NaN      NaN      NaN        NaN       NaN      NaN                  NaN   ARM.L  20130220   941.5   965.10   939.4250   951.5001  2994652  2013-02-20 18:47:45
0     NaN       NaN      NaN      NaN        NaN       NaN      NaN                  NaN   ADN.L  20130220   437.4   442.37   436.5000   441.9000  2775364  2013-02-20 18:47:42
1     NaN       NaN      NaN      NaN        NaN       NaN      NaN                  NaN   ADM.L  20130220  1279.0  1300.00  1272.0000  1285.0000   967730  2013-02-20 18:47:42
2     NaN       NaN      NaN      NaN        NaN       NaN      NaN                  NaN   AGK.L  20130220  1717.0  1749.00  1709.0000  1739.0000   834534  2013-02-20 18:47:43
3     NaN       NaN      NaN      NaN        NaN       NaN      NaN                  NaN
```
And I don't understand why. I'm starting to miss R :(

Response:
pd.concat will serve the purpose of rbind in R.
```
import pandas as pd
df1 = pd.DataFrame({'col1': [1,2], 'col2':[3,4]})
df2 = pd.DataFrame({'col1': [5,6], 'col2':[7,8]})
print(df1)
print(df2)
print(pd.concat([df1, df2]))
```
The outcome will looks like:
```
col1  col2
0     1     3
1     2     4
   col1  col2
0     5     7
1     6     8
   col1  col2
0     1     3
1     2     4
0     5     7
1     6     8
```
If you read the documentation careful enough, it will also explain other operations like cbind, ..etc.


Question:
ValueError: Length of values does not match length of index | Pandas DataFrame.unique() I am trying to get a new dataset, or change the value of the current dataset columns to their unique values. Here is an example of what I am trying to get :
```none
A B
 -----
0| 1 1
1| 2 5
2| 1 5
3| 7 9
4| 7 9
5| 8 9

Wanted Result    Not Wanted Result
       A B              A B
     -----             -----
    0| 1 1           0| 1 1
    1| 2 5           1| 2 5
    2| 7 9           2|
    3| 8             3| 7 9
                     4|
                     5| 8
```
I don't really care about the index but it seems to be the problem. My code so far is pretty simple, I tried 2 approaches, 1 with a new dataFrame and one without.
```py
#With New DataFrame
def UniqueResults(dataframe):
    df = pd.DataFrame()
    for col in dataframe:
        S=pd.Series(dataframe[col].unique())
        df[col]=S.values
    return df

#Without new DataFrame
def UniqueResults(dataframe):
    for col in dataframe:
        dataframe[col]=dataframe[col].unique()
    return dataframe
```
Both times, I get the error:
```none
Length of Values does not match length of index
```

Response:
The error comes up when you are trying to assign a list of numpy array of different length to a data frame, and it can be reproduced as follows: A data frame of four rows:
```
df = pd.DataFrame({'A': [1,2,3,4]})
```
Now trying to assign a list/array of two elements to it:
```
df['B'] = [3,4]   # or df['B'] = np.array([3,4])
```
Both errors out: ValueError: Length of values does not match length of index Because the data frame has four rows but the list and array has only two elements. Work around Solution (use with caution): convert the list/array to a pandas Series, and then when you do assignment, missing index in the Series will be filled with NaN:
```
df['B'] = pd.Series([3,4])

df
#   A     B
#0  1   3.0
#1  2   4.0
#2  3   NaN          # NaN because the value at index 2 and 3 doesn't exist in the Series
#3  4   NaN
```
For your specific problem, if you don't care about the index or the correspondence of values between columns, you can reset index for each column after dropping the duplicates:
```
df.apply(lambda col: col.drop_duplicates().reset_index(drop=True))

#   A     B
#0  1   1.0
#1  2   5.0
#2  7   9.0
#3  8   NaN
```


Question:
Filtering multiple items in a multi-index Pandas dataframe I have the following table:
```none
Area
NSRCODE  PBL_AWI
CM       BONS             44705.492941
         BTNN            253854.591990
         FONG             41625.590370
         FONS             16814.159680
         Lake             57124.819333
         River             1603.906642
         SONS            583958.444751
         STNN             45603.837177
         clearcut        106139.013930
         disturbed       127719.865675
         lowland         118795.578059
         upland         2701289.270193
LBH      BFNN            289207.169650
         BONS           9140084.716743
         BTNI             33713.160390
         BTNN          19748004.789040
         FONG           1687122.469691
         FONS           5169959.591270
         FTNI            317251.976160
         FTNN           6536472.869395
         Lake            258046.508310
         River            44262.807900
         SONS           4379097.677405
         burn regen      744773.210860
         clearcut         54066.756790
         disturbed       597561.471686
         lowland       12591619.141842
         upland        23843453.638117
```
Note: Both NSRCODE and PBL_AWI are indices. How do I search for values in column PBL_AWI? For example I want to keep the values ['Lake', 'River', 'Upland'].

Response:
You can get_level_values in conjunction with Boolean slicing.
```

```python
print df[np.in1d(df.index.get_level_values(1), ['Lake', 'River', 'Upland'])]
                          Area
NSRCODE PBL_AWI
CM      Lake      57124.819333
        River      1603.906642
LBH     Lake     258046.508310
        River     44262.807900
```
The same idea can be expressed in many different ways, such as df[df.index.get_level_values('PBL_AWI').isin(['Lake', 'River', 'Upland'])] Note that you have 'upland' in your data instead of 'Upland'
```


Question:
Pandas: Drop consecutive duplicates What's the most efficient way to drop only consecutive duplicates in pandas? drop_duplicates gives this:
```

```python
a = pandas.Series([1,2,2,3,2], index=[1,2,3,4,5])
```

```python
a.drop_duplicates()
#Output
#1    1
#2    2
#4    3
#dtype: int64
#```
#But I want this:
#```
```

```python
a.something()
#Output
#1    1
#2    2
#4    3
#5    2
#dtype: int64
#```
```

Response:
Use shift:
```
a.loc[a.shift(-1) != a]

Out[3]:

1    1
3    2
4    3
5    2
dtype: int64
```
So the above uses boolean critieria, we compare the dataframe against the dataframe shifted by -1 rows to create the mask Another method is to use diff:
```

```python
a.loc[a.diff() != 0]
#Output
#1    1
#2    2
#4    3
#5    2
#dtype: int64
#```
#But this is slower than the original method if you have a large number of rows. Update Thanks to Bjarke Ebert for pointing out a subtle error, I should actually use shift(1) or just shift() as the default is a period of 1, this returns the first consecutive value:
#```
```

```python
a.loc[a.shift() != a]
#Output
#1    1
#2    2
#4    3
#5    2
#dtype: int64
#```
#Note the difference in index values, thanks @BjarkeEbert!
```


Question:
Groupby and sum only one column So I have a dataframe, df1, that looks like the following:
```
A      B      C
1     foo    12    California
2     foo    22    California
3     bar    8     Rhode Island
4     bar    32    Rhode Island
5     baz    15    Ohio
6     baz    26    Ohio
```
I want to group by column A and then sum column B while keeping the value in column C. Something like this:
```
A       B      C
1    foo     34    California
2    bar     40    Rhode Island
3    baz     41    Ohio
```
The issue is, when I say
```
df.groupby('A').sum()
```
column C gets removed, returning
```
B
A
bar  40
baz  41
foo  34
```
How can I get around this and keep column C when I group and sum?

Response:
The only way to do this would be to include C in your groupby (the groupby function can accept a list). Give this a try:
```
df.groupby(['A','C'])['B'].sum()
```
One other thing to note, if you need to work with df after the aggregation you can also use the as_index=False option to return a dataframe object. This one gave me problems when I was first working with Pandas. Example:
```
df.groupby(['A','C'], as_index=False)['B'].sum()
```


Question:
Concatenate rows of two dataframes in pandas I need to concatenate two dataframes df_a and df_b that have equal number of rows (nRow) horizontally without any consideration of keys. This function is similar to cbind in the R programming language. The number of columns in each dataframe may be different. The resultant dataframe will have the same number of rows nRow and number of columns equal to the sum of number of columns in both the dataframes. In other words, this is a blind columnar concatenation of two dataframes.
```py
import pandas as pd
dict_data = {'Treatment': ['C', 'C', 'C'], 'Biorep': ['A', 'A', 'A'], 'Techrep': [1, 1, 1], 'AAseq': ['ELVISLIVES', 'ELVISLIVES', 'ELVISLIVES'], 'mz':[500.0, 500.5, 501.0]}
df_a = pd.DataFrame(dict_data)
dict_data = {'Treatment1': ['C', 'C', 'C'], 'Biorep1': ['A', 'A', 'A'], 'Techrep1': [1, 1, 1], 'AAseq1': ['ELVISLIVES', 'ELVISLIVES', 'ELVISLIVES'], 'inte1':[1100.0, 1050.0, 1010.0]}
df_b = pd.DataFrame(dict_data)
```

Response:
call concat and pass param axis=1 to concatenate column-wise:
```

```python
pd.concat([df_a,df_b], axis=1)
#Output
#AAseq Biorep  Techrep Treatment     mz      AAseq1 Biorep1  Techrep1  \
#0  ELVISLIVES      A        1         C  500.0  ELVISLIVES       A         1
#1  ELVISLIVES      A        1         C  500.5  ELVISLIVES       A         1
#2  ELVISLIVES      A        1         C  501.0  ELVISLIVES       A         1
#
#  Treatment1  inte1
#0          C   1100
#1          C   1050
#2          C   1010
#```
#There is a useful guide to the various methods of merging, joining and concatenating online. For example, as you have no clashing columns you can merge and use the indices as they have the same number of rows:
#```
```

```python
df_a.merge(df_b, left_index=True, right_index=True)
#Output
#AAseq Biorep  Techrep Treatment     mz      AAseq1 Biorep1  Techrep1  \
#0  ELVISLIVES      A        1         C  500.0  ELVISLIVES       A         1
#1  ELVISLIVES      A        1         C  500.5  ELVISLIVES       A         1
#2  ELVISLIVES      A        1         C  501.0  ELVISLIVES       A         1
#
#  Treatment1  inte1
#0          C   1100
#1          C   1050
#2          C   1010
#```
#And for the same reasons as above a simple join works too:
#```
```

```python
df_a.join(df_b)
#Output
#AAseq Biorep  Techrep Treatment     mz      AAseq1 Biorep1  Techrep1  \
#0  ELVISLIVES      A        1         C  500.0  ELVISLIVES       A         1
#1  ELVISLIVES      A        1         C  500.5  ELVISLIVES       A         1
#2  ELVISLIVES      A        1         C  501.0  ELVISLIVES       A         1
#
#  Treatment1  inte1
#0          C   1100
#1          C   1050
#2          C   1010
#```
```


Question: Pandas rename column by position? [duplicate] This question already has answers here: Changing multiple column names but not all of them - Pandas Python (6 answers) Closed 8 years ago. I was just wondering if I can rename column names by their positions. I know how to rename them by their actual names using: df.rename(columns = {}) How do I do it if I do not know the column names and know only their positions?

Response:
try this
```
df.rename(columns={ df.columns[1]: "your value" }, inplace = True)
```


Question:
Python Pandas replicate rows in dataframe If the dataframe looks like:
```
Store,Dept,Date,Weekly_Sales,IsHoliday
1,1,2010-02-05,24924.5,FALSE
1,1,2010-02-12,46039.49,TRUE
1,1,2010-02-19,41595.55,FALSE
1,1,2010-02-26,19403.54,FALSE
1,1,2010-03-05,21827.9,FALSE
1,1,2010-03-12,21043.39,FALSE
1,1,2010-03-19,22136.64,FALSE
1,1,2010-03-26,26229.21,FALSE
1,1,2010-04-02,57258.43,FALSE
```
And I wanna duplicate rows with IsHoliday equal to TRUE, I can do:
```
is_hol = df['IsHoliday'] == True
df_try = df[is_hol]
df=df.append(df_try*10)
```
But is there a better way to do this as I need to duplicate holiday rows 5 times, and I have to append 5 times if using the above way.

Response:
You can put df_try inside a list and then do what you have in mind:
```

```python
df.append([df_try]*5,ignore_index=True)
```

Store  Dept       Date  Weekly_Sales IsHoliday
0       1     1 2010-02-05      24924.50     False
1       1     1 2010-02-12      46039.49      True
2       1     1 2010-02-19      41595.55     False
3       1     1 2010-02-26      19403.54     False
4       1     1 2010-03-05      21827.90     False
5       1     1 2010-03-12      21043.39     False
6       1     1 2010-03-19      22136.64     False
7       1     1 2010-03-26      26229.21     False
8       1     1 2010-04-02      57258.43     False
9       1     1 2010-02-12      46039.49      True
10      1     1 2010-02-12      46039.49      True
11      1     1 2010-02-12      46039.49      True
12      1     1 2010-02-12      46039.49      True
13      1     1 2010-02-12      46039.49      True
```


Question:
Elegant way to create empty pandas DataFrame with NaN of type float I want to create a Pandas DataFrame filled with NaNs. During my research I found an answer:
```
import pandas as pd

df = pd.DataFrame(index=range(0,4),columns=['A'])
```
This code results in a DataFrame filled with NaNs of type "object". So they cannot be used later on for example with the interpolate() method. Therefore, I created the DataFrame with this complicated code (inspired by this answer):
```
import pandas as pd
import numpy as np

dummyarray = np.empty((4,1))
dummyarray[:] = np.nan

df = pd.DataFrame(dummyarray)
```
This results in a DataFrame filled with NaN of type "float", so it can be used later on with interpolate(). Is there a more elegant way to create the same result?

Response:
Simply pass the desired value as first argument, like 0, math.inf or, here, np.nan. The constructor then initializes and fills the value array to the size specified by arguments index and columns:
```

```python
import numpy as np
```

```python
import pandas as pd
```

```python
df = pd.DataFrame(np.nan, index=[0, 1, 2, 3], columns=['A', 'B'])
```

```python
df
#Output
#    A   B
#0 NaN NaN
#1 NaN NaN
#2 NaN NaN
#3 NaN NaN
```

```python
df.dtypes
#Output
#A    float64
#B    float64
#dtype: object
#```
```


Question:
How do I convert a Pandas dataframe to a PyTorch tensor? How do I train a simple neural network with PyTorch on a pandas dataframe df? The column df["Target"] is the target (e.g. labels) of the network. This doesn't work:
```
import pandas as pd
import torch.utils.data as data_utils

target = pd.DataFrame(df['Target'])
train = data_utils.TensorDataset(df, target)
train_loader = data_utils.DataLoader(train, batch_size=10, shuffle=True)
```

Response:
I'm referring to the question in the title as you haven't really specified anything else in the text, so just converting the DataFrame into a PyTorch tensor. Without information about your data, I'm just taking float values as example targets here. Convert Pandas dataframe to PyTorch tensor?
```py
import pandas as pd
import torch
import random

# creating dummy targets (float values)
targets_data = [random.random() for i in range(10)]

# creating DataFrame from targets_data
targets_df = pd.DataFrame(data=targets_data)
targets_df.columns = ['targets']

# creating tensor from targets_df
torch_tensor = torch.tensor(targets_df['targets'].values)

# printing out result
print(torch_tensor)
```
Output:
```
tensor([ 0.5827,  0.5881,  0.1543,  0.6815,  0.9400,  0.8683,  0.4289,
         0.5940,  0.6438,  0.7514], dtype=torch.float64)
```
Tested with Pytorch 0.4.0. I hope this helps, if you have any further questions - just ask. :)


Question:
Python Pandas: Convert Rows as Column headers [duplicate] This question already has answers here: How can I pivot a dataframe? [closed] (5 answers) Closed 6 years ago. I have the following dataframe:
```
Year    Country          medal    no of medals
1896    Afghanistan      Gold        5
1896    Afghanistan      Silver      4
1896    Afghanistan      Bronze      3
1896    Algeria          Gold        1
1896    Algeria          Silver      2
1896    Algeria          Bronze      3
```
I want it this way.
```
Year    Country      Gold   Silver   Bronze
1896    Afghanistan    5      4         3
1896    Algeria        1      2         3
```
Stack/Unstack dont seem to work.

Response:
You're looking for pivot_table:
```

```python
medals = df.pivot_table('no of medals', ['Year', 'Country'], 'medal')
```

```python
medals
#Output
#medal             Bronze  Gold  Silver
#Year Country
#1896 Afghanistan       3     5       4
#     Algeria           3     1       2
#```
#and if you want to reorder the columns:
#```
```

```python
medals.reindex_axis(['Gold', 'Silver', 'Bronze'], axis=1)
#Output
#medal             Gold  Silver  Bronze
#Year Country
#1896 Afghanistan     5       4       3
#     Algeria         1       2       3
#```
```


Question:
How to group pandas DataFrame entries by date in a non-unique column A Pandas DataFrame contains column named "date" that contains non-unique datetime values. I can group the lines in this frame using:
```
data.groupby(data['date'])
```
However, this splits the data by the datetime values. I would like to group these data by the year stored in the "date" column. This page shows how to group by year in cases where the time stamp is used as an index, which is not true in my case. How do I achieve this grouping?

Response:
I'm using pandas 0.16.2. This has better performance on my large dataset:
```
data.groupby(data.date.dt.year)
```
Using the dt option and playing around with weekofyear, dayofweek etc. becomes far easier.


Question:
Is it possible to insert a row at an arbitrary position in a dataframe using pandas? I have a DataFrame object similar to this one:
```
onset    length
1      2.215    1.3
2     23.107    1.3
3     41.815    1.3
4     61.606    1.3
...
```
What I would like to do is insert a row at a position specified by some index value and update the following indices accordingly. E.g.:
```
onset    length
1      2.215    1.3
2     23.107    1.3
3     30.000    1.3  # new row
4     41.815    1.3
5     61.606    1.3
...
```
What would be the best way to do this?

Response:
You could slice and use concat to get what you want.
```
from pandas import DataFrame, concat
line = DataFrame({"onset": 30.0, "length": 1.3}, index=[3])
df2 = concat([df.iloc[:2], line, df.iloc[2:]]).reset_index(drop=True)
```
This will produce the dataframe in your example output. As far as I'm aware, concat is the best method to achieve an insert type operation in pandas, but admittedly I'm by no means a pandas expert.


Question:
Remove duplicates from dataframe, based on two columns A,B, keeping row with max value in another column C I have a pandas dataframe which contains duplicates values according to two columns (A and B):
```
A B C
1 2 1
1 2 4
2 7 1
3 4 0
3 4 8
```
I want to remove duplicates keeping the row with max value in column C. This would lead to:
```
A B C
1 2 4
2 7 1
3 4 8
```
I cannot figure out how to do that. Should I use drop_duplicates(), something else?

Response:
You can do it using group by:
```
c_maxes = df.groupby(['A', 'B']).C.transform(max)
df = df.loc[df.C == c_maxes]
```
c_maxes is a Series of the maximum values of C in each group but which is of the same length and with the same index as df. If you haven't used .transform then printing c_maxes might be a good idea to see how it works. Another approach using drop_duplicates would be
```
df.sort('C').drop_duplicates(subset=['A', 'B'], take_last=True)
```
Not sure which is more efficient but I guess the first approach as it doesn't involve sorting. EDIT: From pandas 0.18 up the second solution would be
```
df.sort_values('C').drop_duplicates(subset=['A', 'B'], keep='last')
```
or, alternatively,
```
df.sort_values('C', ascending=False).drop_duplicates(subset=['A', 'B'])
```
In any case, the groupby solution seems to be significantly more performing:
```
%timeit -n 10 df.loc[df.groupby(['A', 'B']).C.max == df.C]
10 loops, best of 3: 25.7 ms per loop

%timeit -n 10 df.sort_values('C').drop_duplicates(subset=['A', 'B'], keep='last')
10 loops, best of 3: 101 ms per loop
```


Question:
Create a day-of-week column Create a day-of-week column in a Pandas dataframe using Python I’d like to read a csv file into a pandas dataframe, parse a column of dates from string format to a date object, and then generate a new column that indicates the day of the week. This is what I’m trying: What I’d like to do is something like:
```
import pandas as pd

import csv

df = pd.read_csv('data.csv', parse_dates=['date']))

df['day-of-week'] = df['date'].weekday()

AttributeError: 'Series' object has no attribute 'weekday'
```

Response:
Pandas 0.23+ Use pandas.Series.dt.day_name(), since pandas.Timestamp.weekday_name has been deprecated:
```
import pandas as pd

df = pd.DataFrame({'my_dates':['2015-01-01','2015-01-02','2015-01-03'],'myvals':[1,2,3]})
df['my_dates'] = pd.to_datetime(df['my_dates'])

df['day_of_week'] = df['my_dates'].dt.day_name()
```
Output:
```
my_dates  myvals day_of_week
0 2015-01-01       1    Thursday
1 2015-01-02       2      Friday
2 2015-01-03       3    Saturday
```
Pandas 0.18.1+ As user jezrael points out below, dt.weekday_name was added in version 0.18.1 Pandas Docs
```
import pandas as pd

df = pd.DataFrame({'my_dates':['2015-01-01','2015-01-02','2015-01-03'],'myvals':[1,2,3]})
df['my_dates'] = pd.to_datetime(df['my_dates'])
df['day_of_week'] = df['my_dates'].dt.weekday_name
```
Output:
```
my_dates  myvals day_of_week
0 2015-01-01       1    Thursday
1 2015-01-02       2      Friday
2 2015-01-03       3    Saturday
```
Original Answer: Use this: http://pandas.pydata.org/pandas-docs/stable/generated/pandas.Series.dt.dayofweek.html See this: Get weekday/day-of-week for Datetime column of DataFrame If you want a string instead of an integer do something like this:
```
import pandas as pd

df = pd.DataFrame({'my_dates':['2015-01-01','2015-01-02','2015-01-03'],'myvals':[1,2,3]})
df['my_dates'] = pd.to_datetime(df['my_dates'])
df['day_of_week'] = df['my_dates'].dt.dayofweek

days = {0:'Mon',1:'Tues',2:'Weds',3:'Thurs',4:'Fri',5:'Sat',6:'Sun'}

df['day_of_week'] = df['day_of_week'].apply(lambda x: days[x])
```
Output:
```
my_dates  myvals day_of_week
0 2015-01-01       1       Thurs
1 2015-01-02       2         Fri
2 2015-01-01       3       Thurs
```


Question:
How do I fill a column with one value in Pandas? I have a column with consecutive digits in a Pandas DataFrame.
```
A
1
2
3
4
```
I would like to change all those values to a simple string, say "foo", resulting in
```
A
foo
foo
foo
foo
```

Response:
Just select the column and assign like normal:
```

```python
df['A'] = 'foo'
df
#Output
#A
#0  foo
#1  foo
#2  foo
#3  foo
#```
#Assigning a scalar value will set all the rows to the same scalar value
```


Question:
get dataframe row count based on conditions I want to get the count of dataframe rows based on conditional selection. I tried the following code.
```
print df[(df.IP == head.idxmax()) & (df.Method == 'HEAD') & (df.Referrer == '"-"')].count()
```
output:
```
IP          57
Time        57
Method      57
Resource    57
Status      57
Bytes       57
Referrer    57
Agent       57
dtype: int64
```
The output shows the count for each an every column in the dataframe. Instead I need to get a single count where all of the above conditions satisfied? How to do this? If you need more explanation about my dataframe please let me know.

Response:
You are asking for the condition where all the conditions are true, so len of the frame is the answer, unless I misunderstand what you are asking
```

```python
df = DataFrame(randn(20,4),columns=list('ABCD'))
```

```python
df[(df['A']>0) & (df['B']>0) & (df['C']>0)]
#Output
#A         B         C         D
#12  0.491683  0.137766  0.859753 -1.041487
#13  0.376200  0.575667  1.534179  1.247358
#14  0.428739  1.539973  1.057848 -1.254489
```

```python
df[(df['A']>0) & (df['B']>0) & (df['C']>0)].count()
#Output
#A    3
#B    3
#C    3
#D    3
#dtype: int64
```

```python
len(df[(df['A']>0) & (df['B']>0) & (df['C']>0)])
#Output
#3
#```
```


Question:
how to merge two data frames based on particular column in pandas python? [duplicate] This question already has answers here: Pandas Merging 101 (8 answers) join two pandas dataframe using a specific column (2 answers) Closed 7 months ago. I have to merge two dataframes: df1
```
company,standard
tata,A1
cts,A2
dell,A3
```
df2
```
company,return
tata,71
dell,78
cts,27
hcl,23
```
I have to unify both dataframes to one dataframe. I need output like:
```
company,standard,return
tata,A1,71
cts,A2,27
dell,A3,78
```

Response:
Use merge:
```
print (pd.merge(df1, df2, on='company'))
```
Sample:
```
print (df1)
  company standard
0    tata       A1
1     cts       A2
2    dell       A3

print (df2)
  company  return
0    tata      71
1    dell      78
2     cts      27
3     hcl      23

print (pd.merge(df1, df2, on='company'))
  company standard  return
0    tata       A1      71
1     cts       A2      27
2    dell       A3      78
```


Question:
pandas select from Dataframe using startswith This works (using Pandas 12 dev)
```
table2=table[table['SUBDIVISION'] =='INVERNESS']
```
Then I realized I needed to select the field using "starts with" Since I was missing a bunch. So per the Pandas doc as near as I could follow I tried
```
criteria = table['SUBDIVISION'].map(lambda x: x.startswith('INVERNESS'))
table2 = table[criteria]
```
And got AttributeError: 'float' object has no attribute 'startswith' So I tried an alternate syntax with the same result
```
table[[x.startswith('INVERNESS') for x in table['SUBDIVISION']]]
```
Reference http://pandas.pydata.org/pandas-docs/stable/indexing.html#boolean-indexing Section 4: List comprehensions and map method of Series can also be used to produce more complex criteria: What am I missing?

Response:
You can use the str.startswith DataFrame method to give more consistent results:
```

```python
s = pd.Series(['a', 'ab', 'c', 11, np.nan])
```

```python
s
#Output
#0      a
#1     ab
#2      c
#3     11
#4    NaN
#dtype: object
```

```python
s.str.startswith('a', na=False)
#Output
#0     True
#1     True
#2    False
#3    False
#4    False
#dtype: bool
#```
#and the boolean indexing will work just fine (I prefer to use loc, but it works just the same without):
#```
```

```python
s.loc[s.str.startswith('a', na=False)]
#Output
#0     a
#1    ab
#dtype: object
#```
#. It looks least one of your elements in the Series/column is a float, which doesn't have a startswith method hence the AttributeError, the list comprehension should raise the same error...
```


Question:
Python Pandas User Warning: Sorting because non-concatenation axis is not aligned I'm doing some code practice and applying merging of data frames while doing this getting user warning /usr/lib64/python2.7/site-packages/pandas/core/frame.py:6201: FutureWarning: Sorting because non-concatenation axis is not aligned. A future version of pandas will change to not sort by default. To accept the future behavior, pass 'sort=True'. To retain the current behavior and silence the warning, pass sort=False On these lines of code: Can you please help to get the solution of this warning.
```
placement_video = [self.read_sql_vdx_summary, self.read_sql_video_km]
placement_video_summary = reduce(lambda left, right: pd.merge(left, right, on='PLACEMENT', sort=False), placement_video)

placement_by_video = placement_video_summary.loc[:, ["PLACEMENT", "PLACEMENT_NAME", "COST_TYPE", "PRODUCT",
                                                     "VIDEONAME", "VIEW0", "VIEW25", "VIEW50", "VIEW75",
                                                     "VIEW100",
                                                     "ENG0", "ENG25", "ENG50", "ENG75", "ENG100", "DPE0",
                                                     "DPE25",
                                                     "DPE50", "DPE75", "DPE100"]]

# print (placement_by_video)

placement_by_video["Placement# Name"] = placement_by_video[["PLACEMENT",
                                                            "PLACEMENT_NAME"]].apply(lambda x: ".".join(x),
                                                                                axis=1)

placement_by_video_new = placement_by_video.loc[:,
                         ["PLACEMENT", "Placement# Name", "COST_TYPE", "PRODUCT", "VIDEONAME",
                          "VIEW0", "VIEW25", "VIEW50", "VIEW75", "VIEW100",
                          "ENG0", "ENG25", "ENG50", "ENG75", "ENG100", "DPE0", "DPE25",
                          "DPE50", "DPE75", "DPE100"]]

placement_by_km_video = [placement_by_video_new, self.read_sql_km_for_video]
placement_by_km_video_summary = reduce(lambda left, right: pd.merge(left, right, on=['PLACEMENT', 'PRODUCT'], sort=False),
                                       placement_by_km_video)

#print (list(placement_by_km_video_summary))
#print(placement_by_km_video_summary)
#exit()
# print(placement_by_video_new)
"""Conditions for 25%view"""
mask17 = placement_by_km_video_summary["PRODUCT"].isin(['Display', 'Mobile'])
mask18 = placement_by_km_video_summary["COST_TYPE"].isin(["CPE", "CPM", "CPCV"])
mask19 = placement_by_km_video_summary["PRODUCT"].isin(["InStream"])
mask20 = placement_by_km_video_summary["COST_TYPE"].isin(["CPE", "CPM", "CPE+", "CPCV"])
mask_video_video_completions = placement_by_km_video_summary["COST_TYPE"].isin(["CPCV"])
mask21 = placement_by_km_video_summary["COST_TYPE"].isin(["CPE+"])
mask22 = placement_by_km_video_summary["COST_TYPE"].isin(["CPE", "CPM"])
mask23 = placement_by_km_video_summary["PRODUCT"].isin(['Display', 'Mobile', 'InStream'])
mask24 = placement_by_km_video_summary["COST_TYPE"].isin(["CPE", "CPM", "CPE+"])

choice25video_eng = placement_by_km_video_summary["ENG25"]
choice25video_vwr = placement_by_km_video_summary["VIEW25"]
choice25video_deep = placement_by_km_video_summary["DPE25"]

placement_by_km_video_summary["25_pc_video"] = np.select([mask17 & mask18, mask19 & mask20, mask17 & mask21],
                                                  [choice25video_eng, choice25video_vwr, choice25video_deep])

"""Conditions for 50%view"""
choice50video_eng = placement_by_km_video_summary["ENG50"]
choice50video_vwr = placement_by_km_video_summary["VIEW50"]
choice50video_deep = placement_by_km_video_summary["DPE50"]

placement_by_km_video_summary["50_pc_video"] = np.select([mask17 & mask18, mask19 & mask20, mask17 & mask21],
                                                  [choice50video_eng,
                                                   choice50video_vwr, choice50video_deep])

"""Conditions for 75%view"""

choice75video_eng = placement_by_km_video_summary["ENG75"]
choice75video_vwr = placement_by_km_video_summary["VIEW75"]
choice75video_deep = placement_by_km_video_summary["DPE75"]

placement_by_km_video_summary["75_pc_video"] = np.select([mask17 & mask18, mask19 & mask20, mask17 & mask21],
                                                  [choice75video_eng,
                                                   choice75video_vwr,
                                                   choice75video_deep])

"""Conditions for 100%view"""

choice100video_eng = placement_by_km_video_summary["ENG100"]
choice100video_vwr = placement_by_km_video_summary["VIEW100"]
choice100video_deep = placement_by_km_video_summary["DPE100"]
choicecompletions = placement_by_km_video_summary['COMPLETIONS']

placement_by_km_video_summary["100_pc_video"] = np.select([mask17 & mask22, mask19 & mask24, mask17 & mask21, mask23 & mask_video_video_completions],
                                                          [choice100video_eng, choice100video_vwr, choice100video_deep, choicecompletions])

"""conditions for 0%view"""

choice0video_eng = placement_by_km_video_summary["ENG0"]
choice0video_vwr = placement_by_km_video_summary["VIEW0"]
choice0video_deep = placement_by_km_video_summary["DPE0"]

placement_by_km_video_summary["Views"] = np.select([mask17 & mask18, mask19 & mask20, mask17 & mask21],
                                                   [choice0video_eng,
                                                    choice0video_vwr,
                                                    choice0video_deep])

#print (placement_by_km_video_summary)
#exit()

#final Table

placement_by_video_summary = placement_by_km_video_summary.loc[:,
                             ["PLACEMENT", "Placement# Name", "PRODUCT", "VIDEONAME", "COST_TYPE",
                              "Views", "25_pc_video", "50_pc_video", "75_pc_video","100_pc_video",
                              "ENGAGEMENTS","IMPRESSIONS", "DPEENGAMENTS"]]

#placement_by_km_video = [placement_by_video_summary, self.read_sql_km_for_video]
#placement_by_km_video_summary = reduce(lambda left, right: pd.merge(left, right, on=['PLACEMENT', 'PRODUCT']),
                                       #placement_by_km_video)

#print(placement_by_video_summary)
#exit()
# dup_col =["IMPRESSIONS","ENGAGEMENTS","DPEENGAMENTS"]

# placement_by_video_summary.loc[placement_by_video_summary.duplicated(dup_col),dup_col] = np.nan

# print ("Dhar",placement_by_video_summary)

'''adding views based on conditions'''
#filter maximum value from videos

placement_by_video_summary_new = placement_by_km_video_summary.loc[
    placement_by_km_video_summary.reset_index().groupby(['PLACEMENT', 'PRODUCT'])['Views'].idxmax()]
#print (placement_by_video_summary_new)
#exit()
# print (placement_by_video_summary_new)
# mask22 = (placement_by_video_summary_new.PRODUCT.str.upper ()=='DISPLAY') & (placement_by_video_summary_new.COST_TYPE=='CPE')

placement_by_video_summary_new.loc[mask17 & mask18, 'Views'] = placement_by_video_summary_new['ENGAGEMENTS']
placement_by_video_summary_new.loc[mask19 & mask20, 'Views'] = placement_by_video_summary_new['IMPRESSIONS']
placement_by_video_summary_new.loc[mask17 & mask21, 'Views'] = placement_by_video_summary_new['DPEENGAMENTS']

#print (placement_by_video_summary_new)
#exit()
placement_by_video_summary = placement_by_video_summary.drop(placement_by_video_summary_new.index).append(
    placement_by_video_summary_new).sort_index()

placement_by_video_summary["Video Completion Rate"] = placement_by_video_summary["100_pc_video"] / \
                                                      placement_by_video_summary["Views"]

placement_by_video_final = placement_by_video_summary.loc[:,
                           ["Placement# Name", "PRODUCT", "VIDEONAME", "Views",
                            "25_pc_video", "50_pc_video", "75_pc_video", "100_pc_video",
                            "Video Completion Rate"]]
```

Response:
tl;dr: concat and append currently sort the non-concatenation index (e.g. columns if you're adding rows) if the columns don't match. In pandas 0.23 this started generating a warning; pass the parameter sort=True to silence it. In the future the default will change to not sort, so it's best to specify either sort=True or False now, or better yet ensure that your non-concatenation indices match. The warning is new in pandas 0.23.0: In a future version of pandas pandas.concat() and DataFrame.append() will no longer sort the non-concatenation axis when it is not already aligned. The current behavior is the same as the previous (sorting), but now a warning is issued when sort is not specified and the non-concatenation axis is not aligned, link. More information from linked very old github issue, comment by smcinerney : When concat'ing DataFrames, the column names get alphanumerically sorted if there are any differences between them. If they're identical across DataFrames, they don't get sorted. This sort is undocumented and unwanted. Certainly the default behavior should be no-sort. After some time the parameter sort was implemented in pandas.concat and DataFrame.append: sort : boolean, default None Sort non-concatenation axis if it is not already aligned when join is 'outer'. The current default of sorting is deprecated and will change to not-sorting in a future version of pandas. Explicitly pass sort=True to silence the warning and sort. Explicitly pass sort=False to silence the warning and not sort. This has no effect when join='inner', which already preserves the order of the non-concatenation axis. So if both DataFrames have the same columns in the same order, there is no warning and no sorting:
```
df1 = pd.DataFrame({"a": [1, 2], "b": [0, 8]}, columns=['a', 'b'])
df2 = pd.DataFrame({"a": [4, 5], "b": [7, 3]}, columns=['a', 'b'])

print (pd.concat([df1, df2]))
   a  b
0  1  0
1  2  8
0  4  7
1  5  3

df1 = pd.DataFrame({"a": [1, 2], "b": [0, 8]}, columns=['b', 'a'])
df2 = pd.DataFrame({"a": [4, 5], "b": [7, 3]}, columns=['b', 'a'])

print (pd.concat([df1, df2]))
   b  a
0  0  1
1  8  2
0  7  4
1  3  5
```
But if the DataFrames have different columns, or the same columns in a different order, pandas returns a warning if no parameter sort is explicitly set (sort=None is the default value):
```
df1 = pd.DataFrame({"a": [1, 2], "b": [0, 8]}, columns=['b', 'a'])
df2 = pd.DataFrame({"a": [4, 5], "b": [7, 3]}, columns=['a', 'b'])

print (pd.concat([df1, df2]))
```
FutureWarning: Sorting because non-concatenation axis is not aligned.
```
a  b
0  1  0
1  2  8
0  4  7
1  5  3

print (pd.concat([df1, df2], sort=True))
   a  b
0  1  0
1  2  8
0  4  7
1  5  3

print (pd.concat([df1, df2], sort=False))
   b  a
0  0  1
1  8  2
0  7  4
1  3  5
```
If the DataFrames have different columns, but the first columns are aligned - they will be correctly assigned to each other (columns a and b from df1 with a and b from df2 in the example below) because they exist in both. For other columns that exist in one but not both DataFrames, missing values are created. Lastly, if you pass sort=True, columns are sorted alphanumerically. If sort=False and the second DafaFrame has columns that are not in the first, they are appended to the end with no sorting:
```
df1 = pd.DataFrame({"a": [1, 2], "b": [0, 8], 'e':[5, 0]},
                    columns=['b', 'a','e'])
df2 = pd.DataFrame({"a": [4, 5], "b": [7, 3], 'c':[2, 8], 'd':[7, 0]},
                    columns=['c','b','a','d'])

print (pd.concat([df1, df2]))
```
FutureWarning: Sorting because non-concatenation axis is not aligned.
```
a  b    c    d    e
0  1  0  NaN  NaN  5.0
1  2  8  NaN  NaN  0.0
0  4  7  2.0  7.0  NaN
1  5  3  8.0  0.0  NaN

print (pd.concat([df1, df2], sort=True))
   a  b    c    d    e
0  1  0  NaN  NaN  5.0
1  2  8  NaN  NaN  0.0
0  4  7  2.0  7.0  NaN
1  5  3  8.0  0.0  NaN

print (pd.concat([df1, df2], sort=False))

   b  a    e    c    d
0  0  1  5.0  NaN  NaN
1  8  2  0.0  NaN  NaN
0  7  4  NaN  2.0  7.0
1  3  5  NaN  8.0  0.0
```
In your code:
```
placement_by_video_summary = placement_by_video_summary.drop(placement_by_video_summary_new.index)
                                                       .append(placement_by_video_summary_new, sort=True)
                                                       .sort_index()
```


Question:
group by week in pandas I'm having this data frame:
```
Name   Date    Quantity
Apple  07/11/17  20
orange 07/14/17  20
Apple  07/14/17  70
Orange 07/25/17  40
Apple  07/20/17  30
```
I want to aggregate this by Name and Date to get sum of quantities Details: Date: Group, the result should be at the beginning of the week (or just on Monday) Quantity: Sum, if two or more records have same Name and Date (if falls on same interval) The desired output is given below:
```
Name   Date    Quantity
Apple  07/10/17  90
orange 07/10/17  20
Apple  07/17/17  30
orange 07/24/17  40
```

Response:
First, convert column date to_datetime and subtract one week as we want the sum for the week ahead of the date and not the week before that date. Then use groupby with Grouper by W-MON and aggregate sum:
```py
df['Date'] = pd.to_datetime(df['Date']) - pd.to_timedelta(7, unit='d')
df = df.groupby(['Name', pd.Grouper(key='Date', freq='W-MON')])['Quantity']
    .sum()
    .reset_index()
    .sort_values('Date')
print (df)
```
```
Name       Date  Quantity
0   Apple 2017-07-10        90
3  orange 2017-07-10        20
1   Apple 2017-07-17        30
2  Orange 2017-07-24        40
```


Question:
Nested dictionary to multiindex dataframe where dictionary keys are column labels Say I have a dictionary that looks like this:
```
dictionary = {'A' : {'a': [1,2,3,4,5],
                     'b': [6,7,8,9,1]},

              'B' : {'a': [2,3,4,5,6],
                     'b': [7,8,9,1,2]}}
```
and I want a dataframe that looks something like this:
```
A   B
     a b a b
  0  1 6 2 7
  1  2 7 3 8
  2  3 8 4 9
  3  4 9 5 1
  4  5 1 6 2
```
Is there a convenient way to do this? If I try:
```

```python
DataFrame(dictionary)
#Output
#A               B
#a   [1, 2, 3, 4, 5] [2, 3, 4, 5, 6]
#b   [6, 7, 8, 9, 1] [7, 8, 9, 1, 2]
#```
#I get a dataframe where each element is a list. What I need is a multiindex where each level corresponds to the keys in the nested dict and the rows corresponding to each element in the list as shown above. I think I can work a very crude solution but I'm hoping there might be something a bit simpler.
```

Response:
Pandas wants the MultiIndex values as tuples, not nested dicts. The simplest thing is to convert your dictionary to the right format before trying to pass it to DataFrame:
```

```python
reform = {(outerKey, innerKey): values for outerKey, innerDict in dictionary.items() for innerKey, values in innerDict.items()}
```

```python
reform
#Output
#{('A', 'a'): [1, 2, 3, 4, 5],
# ('A', 'b'): [6, 7, 8, 9, 1],
# ('B', 'a'): [2, 3, 4, 5, 6],
# ('B', 'b'): [7, 8, 9, 1, 2]}
```

```python
pandas.DataFrame(reform)
#Output
#   A     B
#   a  b  a  b
#0  1  6  2  7
#1  2  7  3  8
#2  3  8  4  9
#3  4  9  5  1
#4  5  1  6  2
```

[5 rows x 4 columns]
```


Question:
How do I properly set the Datetimeindex for a Pandas datetime object in a dataframe? I have a pandas dataframe:
```
lat         lng         alt days              date        time
0   40.003834   116.321462  211 39745.175405      2008-10-24  04:12:35
1   40.003783   116.321431  201 39745.175463  2008-10-24      04:12:40
2   40.003690   116.321429  203 39745.175521      2008-10-24      04:12:45
3   40.003589   116.321427  194 39745.175579      2008-10-24      04:12:50
4   40.003522   116.321412  190 39745.175637      2008-10-24      04:12:55
5   40.003509   116.321484  188 39745.175694      2008-10-24      04:13:00
```
For which I am trying to convert the df['date'] and df['time'] columns into a datetime. I can do:
```
df['Datetime'] = pd.to_datetime(df['date']+df['time'])
df = df.set_index(['Datetime'])
del df['date']
del df['time']
```
And I get:
```
lat         lng         alt days
Datetime
2008-10-2404:12:35  40.003834   116.321462  211 39745.175405
2008-10-2404:12:40  40.003783   116.321431  201 39745.175463
2008-10-2404:12:45  40.003690   116.321429  203 39745.175521
2008-10-2404:12:50  40.003589   116.321427  194 39745.175579
2008-10-2404:12:55  40.003522   116.321412  190 39745.175637
```
But then if I try:
```
df.between_time(time(1),time(22,59,59))['lng'].std()
```
I get an error - 'TypeError: Index must be DatetimeIndex' So, I've also tried setting the DatetimeIndex:
```
df['Datetime'] = pd.to_datetime(df['date']+df['time'])
#df = df.set_index(['Datetime'])
df = df.set_index(pd.DatetimeIndex(df['Datetime']))
del df['date']
del df['time']
```
And this throws an error also - 'DateParseError: unknown string format' How do I create the datetime column and DatetimeIndex correctly so that df.between_time() works right?

Response:
To simplify Kirubaharan's answer a bit:
```
df['Datetime'] = pd.to_datetime(df['date'] + ' ' + df['time'])
df = df.set_index('Datetime')
```
And to get rid of unwanted columns (as OP did but did not specify per se in the question):
```
df = df.drop(['date','time'], axis=1)
```


Question:
pandas get the row-wise minimum value of two or more columns How can I reference the minimum value of two dataframes as part of a pandas dataframe equation? I tried using the python min() function which did not work. I am looking for something along the lines of this:
```py
data['eff'] = pd.DataFrame([data['flow_h'], data['flow_c']]).min() *Cp* (data[' Thi'] - data[' Tci'])
```
I also tried to use pandas min() function, which is also not working.
```py
min_flow = pd.DataFrame([data['flow_h'], data['flow_c']]).min()
```
```none
InvalidIndexError: Reindexing only valid with uniquely valued Index objects
```
I was confused by this error. The data columns are just numbers and a name, I wasn't sure where the index comes into play.
```py
import pandas as pd
import numpy as np

np.random.seed(365)
rows = 10
flow = {'flow_c': [np.random.randint(100) for _ in range(rows)],
        'flow_d': [np.random.randint(100) for _ in range(rows)],
        'flow_h': [np.random.randint(100) for _ in range(rows)]}
data = pd.DataFrame(flow)

# display(data)
   flow_c  flow_d  flow_h
0      82      36      43
1      52      48      12
2      33      28      77
3      91      99      11
4      44      95      27
5       5      94      64
6      98       3      88
7      73      39      92
8      26      39      62
9      56      74      50
```

Response:
If you are trying to get the row-wise mininum of two or more columns, use pandas.DataFrame.min. Note that by default axis=0; specifying axis=1 is necessary.
```py
data['min_c_h'] = data[['flow_h','flow_c']].min(axis=1)

# display(data)
   flow_c  flow_d  flow_h  min_c_h
0      82      36      43       43
1      52      48      12       12
2      33      28      77       33
3      91      99      11       11
4      44      95      27       27
5       5      94      64        5
6      98       3      88       88
7      73      39      92       73
8      26      39      62       26
9      56      74      50       50
```


Question:
Find empty or NaN entry in Pandas Dataframe I am trying to search through a Pandas Dataframe to find where it has a missing entry or a NaN entry. Here is a dataframe that I am working with:
```
cl_id       a           c         d         e        A1              A2             A3
    0       1   -0.419279  0.843832 -0.530827    text76        1.537177      -0.271042
    1       2    0.581566  2.257544  0.440485    dafN_6        0.144228       2.362259
    2       3   -1.259333  1.074986  1.834653    system                       1.100353
    3       4   -1.279785  0.272977  0.197011     Fifty       -0.031721       1.434273
    4       5    0.578348  0.595515  0.553483   channel        0.640708       0.649132
    5       6   -1.549588 -0.198588  0.373476     audio       -0.508501
    6       7    0.172863  1.874987  1.405923    Twenty             NaN            NaN
    7       8   -0.149630 -0.502117  0.315323  file_max             NaN            NaN
```
NOTE: The blank entries are empty strings - this is because there was no alphanumeric content in the file that the dataframe came from. If I have this dataframe, how can I find a list with the indexes where the NaN or blank entry occurs?

Response:
np.where(pd.isnull(df)) returns the row and column indices where the value is NaN:
```

```python
import numpy as np
```

```python
import pandas as pd
```

```python
np.where(pd.isnull(df))
#Output
#(array([2, 5, 6, 6, 7, 7]), array([7, 7, 6, 7, 6, 7]))
```

```python
df.iloc[2,7]
#Output
#nan
```

```python
[df.iloc[i,j] for i,j in zip(*np.where(pd.isnull(df)))]
#Output
#[nan, nan, nan, nan, nan, nan]
#```
#Finding values which are empty strings could be done with applymap:
#```
```

```python
np.where(df.applymap(lambda x: x == ''))
#Output
#(array([5]), array([7]))
#```
#Note that using applymap requires calling a Python function once for each cell of the DataFrame. That could be slow for a large DataFrame, so it would be better if you could arrange for all the blank cells to contain NaN instead so you could use pd.isnull.
```


Question:
Pandas: Convert Timestamp to datetime.date I have a pandas column of Timestamp data
```

```python
train["Original_Quote_Date"][6]
#Output
#Timestamp('2013-12-25 00:00:00')
#```
#How can check equivalence of these objects to datetime.date objects of the type
#```
#datetime.date(2013, 12, 25)
#```
```

Response:
Use the .date method:
```

```python
t = pd.Timestamp('2013-12-25 00:00:00')
```

```python
t.date()
#Output
#datetime.date(2013, 12, 25)
```

```python
t.date() == datetime.date(2013, 12, 25)
#Output
#True
#```
#To compare against a DatetimeIndex (i.e. an array of Timestamps), you'll want to do it the other way around:
#```
```

```python
pd.Timestamp(datetime.date(2013, 12, 25))
#Output
#Timestamp('2013-12-25 00:00:00')
```

```python
ts = pd.DatetimeIndex([t])
```

```python
ts == pd.Timestamp(datetime.date(2013, 12, 25))
#Output
#array([ True], dtype=bool)
#```
```


Question: Element-wise logical OR in Pandas I am aware that AND corresponds to & and NOT, ~. What is the element-wise logical OR operator? I know "or" itself is not what I am looking for.

Response:
The corresponding operator is |:
```
df[(df < 3) | (df == 5)]
```
would elementwise check if value is less than 3 or equal to 5. If you need a function to do this, we have np.logical_or. For two conditions, you can use
```
df[np.logical_or(df<3, df==5)]
```
Or, for multiple conditions use the logical_or.reduce,
```
df[np.logical_or.reduce([df<3, df==5])]
```
Since the conditions are specified as individual arguments, parentheses grouping is not needed. More information on logical operations with pandas can be found here.


Question:
How do I change a single index value in pandas dataframe?
```
energy.loc['Republic of Korea']
```
I want to change the value of index from 'Republic of Korea' to 'South Korea'. But the dataframe is too large and it is not possible to change every index value. How do I change only this single value?

Response:
@EdChum's solution looks good. Here's one using rename, which would replace all these values in the index.
```
energy.rename(index={'Republic of Korea':'South Korea'},inplace=True)
```
Here's an example
```

```python
example = pd.DataFrame({'key1' : ['a','a','a','b','a','b'],
#Output
#           'data1' : [1,2,2,3,nan,4],
#           'data2' : list('abcdef')})
```

```python
example.set_index('key1',inplace=True)
```

```python
example
#Output
#      data1 data2
#key1
#a       1.0     a
#a       2.0     b
#a       2.0     c
#b       3.0     d
#a       NaN     e
#b       4.0     f
```

```python
example.rename(index={'a':'c'}) # can also use inplace=True
#Output
#      data1 data2
#key1
#c       1.0     a
#c       2.0     b
#c       2.0     c
#b       3.0     d
#c       NaN     e
#b       4.0     f
#```
```


Question:
How to convert columns into one datetime column in pandas? I have a dataframe where the first 3 columns are 'MONTH', 'DAY', 'YEAR' In each column there is an integer. Is there a Pythonic way to convert all three columns into datetimes while there are in the dataframe? From:
```
M    D    Y    Apples   Oranges
5    6  1990      12        3
5    7  1990      14        4
5    8  1990      15       34
5    9  1990      23       21
```
into:
```
Datetimes    Apples   Oranges
1990-6-5        12        3
1990-7-5        14        4
1990-8-5        15       34
1990-9-5        23       21
```

Response:
In version 0.18.1 you can use to_datetime, but: The names of the columns have to be year, month, day, hour, minute and second: Minimal columns are year, month and day Sample:
```
import pandas as pd

df = pd.DataFrame({'year': [2015, 2016],
                   'month': [2, 3],
                    'day': [4, 5],
                    'hour': [2, 3],
                    'minute': [10, 30],
                    'second': [21,25]})

print df
   day  hour  minute  month  second  year
0    4     2      10      2      21  2015
1    5     3      30      3      25  2016

print pd.to_datetime(df[['year', 'month', 'day']])
0   2015-02-04
1   2016-03-05
dtype: datetime64[ns]

print pd.to_datetime(df[['year', 'month', 'day', 'hour']])
0   2015-02-04 02:00:00
1   2016-03-05 03:00:00
dtype: datetime64[ns]

print pd.to_datetime(df[['year', 'month', 'day', 'hour', 'minute']])
0   2015-02-04 02:10:00
1   2016-03-05 03:30:00
dtype: datetime64[ns]

print pd.to_datetime(df)
0   2015-02-04 02:10:21
1   2016-03-05 03:30:25
dtype: datetime64[ns]
```
Another solution is convert to dictionary:
```
print df
   M  D     Y  Apples  Oranges
0  5  6  1990      12        3
1  5  7  1990      14        4
2  5  8  1990      15       34
3  5  9  1990      23       21

print pd.to_datetime(dict(year=df.Y, month=df.M, day=df.D))
0   1990-05-06
1   1990-05-07
2   1990-05-08
3   1990-05-09
dtype: datetime64[ns]
```


Question:
Pandas: Get unique MultiIndex level values by label Say you have this MultiIndex-ed DataFrame:
```
df = pd.DataFrame({'country':['DE','DE','FR','FR'],
                   'biome':['Lake','Forest','Lake','Forest'],
                   'area':[10,20,30,40],
                   'count':[7,5,2,3]})
df = df.set_index(['country','biome'])
```
Which looks like this:
```
area  count
country biome
DE      Lake      10      7
        Forest    20      5
FR      Lake      30      2
        Forest    40      3
```
I would like to retrieve the unique values per index level. This can be accomplished using
```

```python
df.index.levels[0]
#Output
#['DE', 'FR']
```

```python
df.index.levels[1]
#Output
#['Lake', 'Forest']
#```
#What I would really like to do, is to retrieve these lists by addressing the levels by their name, i.e. 'country' and 'biome'. The shortest two ways I could find looks like this:
#```
```

```python
list(set(df.index.get_level_values('country')))
#Output
#['DE', 'FR']
```

```python
df.index.levels[df.index.names.index('country')]
#Output
#['DE', 'FR']
#```
#But non of them are very elegant. Is there a shorter and/or more performant way?
```

Response:
Pandas 0.23.0 finally introduced a much cleaner solution to this problem: the level argument to Index.unique():
```

```python
df.index.unique(level='country')
#Output
#Index(['DE', 'FR'], dtype='object', name='country')
#```
#This is now the recommended solution. It is far more efficient because it avoids creating a complete representation of the level values in memory, and re-scanning it.
```


Question:
What causes "indexing past lexsort depth" warning in Pandas? I'm indexing a large multi-index Pandas df using df.loc[(key1, key2)]. Sometimes I get a series back (as expected), but other times I get a dataframe. I'm trying to isolate the cases which cause the latter, but so far all I can see is that it's correlated with getting a PerformanceWarning: indexing past lexsort depth may impact performance warning. I'd like to reproduce it to post here, but I can't generate another case that gives me the same warning. Here's my attempt:
```
def random_dates(start, end, n=10):
    start_u = start.value//10**9
    end_u = end.value//10**9
    return pd.to_datetime(np.random.randint(start_u, end_u, n), unit='s')

np.random.seed(0)
df = pd.DataFrame(np.random.random(3255000).reshape(465000,7))  # same shape as my data
df['date'] = random_dates(pd.to_datetime('1990-01-01'), pd.to_datetime('2018-01-01'), 465000)
df = df.set_index([0, 'date'])
df = df.sort_values(by=[3])  # unsort indices, just in case
df.index.lexsort_depth
> 0
df.index.is_monotonic
> False
df.loc[(0.9987185534991936, pd.to_datetime('2012-04-16 07:04:34'))]
# no warning
```
So my question is: what causes this warning? How do I artificially induce it?

Response:
TL;DR: your index is unsorted and this severely impacts performance. Sort your DataFrame's index using df.sort_index() to address the warning and improve performance. I've actually written about this in detail in my writeup: Select rows in pandas MultiIndex DataFrame (under "Question 3"). To reproduce,
```
mux = pd.MultiIndex.from_arrays([
    list('aaaabbbbbccddddd'),
    list('tuvwtuvwtuvwtuvw')
], names=['one', 'two'])

df = pd.DataFrame({'col': np.arange(len(mux))}, mux)

         col
one two
a   t      0
    u      1
    v      2
    w      3
b   t      4
    u      5
    v      6
    w      7
    t      8
c   u      9
    v     10
d   w     11
    t     12
    u     13
    v     14
    w     15
```
You'll notice that the second level is not properly sorted. Now, try to index a specific cross section:
```
df.loc[pd.IndexSlice[('c', 'u')]]
PerformanceWarning: indexing past lexsort depth may impact performance.
  # encoding: utf-8

         col
one two
c   u      9
```
You'll see the same behaviour with xs:
```
df.xs(('c', 'u'), axis=0)
PerformanceWarning: indexing past lexsort depth may impact performance.
  self.interact()

         col
one two
c   u      9
```
The docs, backed by this timing test I once did seem to suggest that handling un-sorted indexes imposes a slowdown—Indexing is O(N) time when it could/should be O(1). If you sort the index before slicing, you'll notice the difference:
```
df2 = df.sort_index()
df2.loc[pd.IndexSlice[('c', 'u')]]

         col
one two
c   u      9

%timeit df.loc[pd.IndexSlice[('c', 'u')]]
%timeit df2.loc[pd.IndexSlice[('c', 'u')]]

802 µs ± 12.1 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
648 µs ± 20.3 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
```
Finally, if you want to know whether the index is sorted or not, check with MultiIndex.is_lexsorted.
```
df.index.is_lexsorted()
# False

df2.index.is_lexsorted()
# True
```
As for your question on how to induce this behaviour, simply permuting the indices should suffice. This works if your index is unique:
```
df2 = df.loc[pd.MultiIndex.from_tuples(np.random.permutation(df2.index))]
```
If your index is not unique, add a cumcounted level first,
```
df.set_index(
    df.groupby(level=list(range(len(df.index.levels)))).cumcount(), append=True)
df2 = df.loc[pd.MultiIndex.from_tuples(np.random.permutation(df2.index))]
df2 = df2.reset_index(level=-1, drop=True)
```


Question:
Getting min and max Dates from a pandas dataframe How do I get the min and max Dates from a dataframe's major axis?
```
value
Date
2014-03-13  10000.000
2014-03-21   2000.000
2014-03-27   2000.000
2014-03-17    200.000
2014-03-17      5.000
2014-03-17     70.000
2014-03-21    200.000
2014-03-27      5.000
2014-03-27     25.000
2014-03-31      0.020
2014-03-31     12.000
2014-03-31      0.022
```
Essentially I want a way to get the min and max dates, i.e. 2014-03-13 and 2014-03-31. I tried using numpy.min or df.min(axis=0), I'm able to get the min or max value but that's not what I want

Response:
'Date' is your index so you want to do,
```
print (df.index.min())
print (df.index.max())

2014-03-13 00:00:00
2014-03-31 00:00:00
```


Question: Is there a parameter in matplotlib/pandas to have the Y axis of a histogram as percentage? I would like to compare two histograms by having the Y axis show the percentage of each column from the overall dataset size instead of an absolute value. Is that possible? I am using Pandas and matplotlib. Thanks

Response:
The density=True (normed=True for matplotlib < 2.2.0) returns a histogram for which np.sum(pdf * np.diff(bins)) equals 1. If you want the sum of the histogram to be 1 you can use Numpy's histogram() and normalize the results yourself.
```
x = np.random.randn(30)

fig, ax = plt.subplots(1,2, figsize=(10,4))

ax[0].hist(x, density=True, color='grey')

hist, bins = np.histogram(x)
ax[1].bar(bins[:-1], hist.astype(np.float32) / hist.sum(), width=(bins[1]-bins[0]), color='grey')

ax[0].set_title('normed=True')
ax[1].set_title('hist = hist / hist.sum()')
```
Btw: Strange plotting glitch at the first bin of the left plot.


Question:
Plotting histograms from grouped data in a pandas DataFrame How do I plot a block of histograms from a group of data in a dataframe? For example, given:
```
from pandas import DataFrame
import numpy as np
x = ['A']*300 + ['B']*400 + ['C']*300
y = np.random.randn(1000)
df = DataFrame({'Letter': x, 'N': y})
```
I tried:
```
df.groupby('Letter').hist()
```
...which failed with the error message: TypeError: cannot concatenate 'str' and 'float' objects

Response:
I'm on a roll, just found an even simpler way to do it using the by keyword in the hist method:
```
df.hist('N', by='Letter')
```
That's a very handy little shortcut for quickly scanning your grouped data! For future visitors, the product of this call is the following chart: In answer to questions below, here's an example of specific tailoring of the histogram plots:
```
# import libraries
import pandas as pd
import numpy as np

# Create test dataframe
x = ['A']*300 + ['B']*400 + ['C']*300
y = np.random.randn(1000)
z = np.random.randn(1000)
df = pd.DataFrame({'Letter':x, 'N1':y, 'N2':z})

# Plot histograms
axes = df.hist(['N1','N2'], by='Letter',bins=10, layout=(2,2),
               legend=True, yrot=90,sharex=True,sharey=True,
               log=True, figsize=(6,6))
for ax in axes.flatten():
    ax.set_xlabel('N')
    ax.set_ylabel('Count')
    ax.set_ylim(bottom=1,top=100)
```


Question:
datetime to string with series in pandas How should I transform from datetime to string? My attempt:
```
dates = p.to_datetime(p.Series(['20010101', '20010331']), format = '%Y%m%d')
dates.str
```

Response:
There is no .str accessor for datetimes and you can't do .astype(str) either. Instead, use .dt.strftime:
```

```python
series = pd.Series(['20010101', '20010331'])
```

```python
dates = pd.to_datetime(series, format='%Y%m%d')
```

```python
dates.dt.strftime('%Y-%m-%d')
#Output
#0    2001-01-01
#1    2001-03-31
#dtype: object
#```
#See the docs on customizing date string formats here: strftime() and strptime() Behavior. For old pandas versions <0.17.0, one can instead can call .apply with the Python standard library's datetime.strftime:
#```
```

```python
dates.apply(lambda x: x.strftime('%Y-%m-%d'))
#Output
#0    2001-01-01
#1    2001-03-31
#dtype: object
#```
```


Question: Finding the intersection between two series in Pandas I have two series s1 and s2 in pandas and want to compute the intersection i.e. where all of the values of the series are common. How would I use the concat function to do this? I have been trying to work it out but have been unable to (I don't want to compute the intersection on the indices of s1 and s2, but on the values).

Response:
Place both series in Python's set container then use the set intersection method:
```
s1.intersection(s2)
```
and then transform back to list if needed. Just noticed pandas in the tag. Can translate back to that:
```
pd.Series(list(set(s1).intersection(set(s2))))
```
From comments I have changed this to a more Pythonic expression, which is shorter and easier to read:
```
Series(list(set(s1) & set(s2)))
```
should do the trick, except if the index data is also important to you. Have added the list(...) to translate the set before going to pd.Series as pandas does not accept a set as direct input for a Series.


Question:
How to slice a pandas DataFrame by position? I have a Pandas Data Frame object that has 1000 rows and 10 columns. I would simply like to slice the Data Frame and take the first 10 rows. How can I do this? I've been trying to use this:
```

```python
df.shape
#Output
#(1000,10)
```

```python
my_slice = df.ix[10,:]
```

```python
my_slice.shape
#Output
#(10,)
#```
#Shouldn't my_slice be the first ten rows, ie. a 10 x 10 Data Frame? How can I get the first ten rows, such that my_slice is a 10x10 Data Frame object? Thanks.
```

Response:
http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.head.html?highlight=head#pandas.DataFrame.head
```
df2 = df.head(10)
```
should do the trick


Question:
Python generating a list of dates between two dates I want to generate a list of dates between two dates and store them in a list in string format. This list is useful to compare with other dates I have. My code is given below:
```
from datetime import date, timedelta

sdate = date(2019,3,22)   # start date
edate = date(2019,4,9)   # end date

def dates_bwn_twodates(start_date, end_date):
    for n in range(int ((end_date - start_date).days)):
        yield start_date + timedelta(n)
print(dates_bwn_twodates(sdate,edate))
```
My present output:
```
<generator object dates_bwn_twodates at 0x000002A8E7929410>
```
My expected output:
```
['2019-03-22',.....,'2019-04-08']
```
Something wrong in my code.

Response:
You can use pandas.date_range() for this:
```py
import pandas
pandas.date_range(sdate,edate-timedelta(days=1),freq='d')
```
```
DatetimeIndex(['2019-03-22', '2019-03-23', '2019-03-24', '2019-03-25',
           '2019-03-26', '2019-03-27', '2019-03-28', '2019-03-29',
           '2019-03-30', '2019-03-31', '2019-04-01', '2019-04-02',
           '2019-04-03', '2019-04-04', '2019-04-05', '2019-04-06',
           '2019-04-07', '2019-04-08'],
          dtype='datetime64[ns]', freq='D')
```


Question:
Correlation heatmap I want to represent correlation matrix using a heatmap. There is something called correlogram in R, but I don't think there's such a thing in Python. How can I do this? The values go from -1 to 1, for example:
```
[[ 1.          0.00279981  0.95173379  0.02486161 -0.00324926 -0.00432099]
 [ 0.00279981  1.          0.17728303  0.64425774  0.30735071  0.37379443]
 [ 0.95173379  0.17728303  1.          0.27072266  0.02549031  0.03324756]
 [ 0.02486161  0.64425774  0.27072266  1.          0.18336236  0.18913512]
 [-0.00324926  0.30735071  0.02549031  0.18336236  1.          0.77678274]
 [-0.00432099  0.37379443  0.03324756  0.18913512  0.77678274  1.        ]]
```
I was able to produce the following heatmap based on another question, but the problem is that my values get 'cut' at 0, so I would like to have a map which goes from blue(-1) to red(1), or something like that, but here values below 0 are not presented in an adequate way. Here's the code for that:
```
plt.imshow(correlation_matrix,cmap='hot',interpolation='nearest')
```

Response:
Another alternative is to use the heatmap function in seaborn to plot the covariance. This example uses the 'mpg' data set from seaborn.
```
import seaborn as sns
%matplotlib inline

# load the Auto dataset
auto_df = sns.load_dataset('mpg')

# calculate the correlation matrix on the numeric columns
corr = auto_df.select_dtypes('number').corr()

# plot the heatmap
sns.heatmap(corr)
```
If you wanted to be even more fancy, you can use Pandas Style, for example:
```
cmap = sns.diverging_palette(5, 250, as_cmap=True)

def magnify():
    return [dict(selector="th",
                 props=[("font-size", "7pt")]),
            dict(selector="td",
                 props=[('padding', "0em 0em")]),
            dict(selector="th:hover",
                 props=[("font-size", "12pt")]),
            dict(selector="tr:hover td:hover",
                 props=[('max-width', '200px'),
                        ('font-size', '12pt')])
]

corr.style.background_gradient(cmap, axis=1)\
    .format(precision=3)\
    .set_properties(**{'max-width': '80px', 'font-size': '10pt'})\
    .set_caption("Hover to magify")\
    .set_table_styles(magnify())
```


Question:
Pandas - check if ALL values are NaN in Series I have a data series which looks like this:
```
print mydf

id_L1
2       NaN
3       NaN
4       NaN
5       NaN
6       NaN
7       NaN
8       NaN
```
I would like to check if all the values are NaN. My attempt:
```
pd.isnull(mydf).all()
```
Output:
```
True
```
Is this the correct way to do it?

Response:
Yes, that's correct, but I think a more idiomatic way would be:
```
mys.isnull().all()
```


Question:
Pandas column values to columns? [duplicate] This question already has answers here: How can I pivot a dataframe? [closed] (5 answers) Closed 3 years ago. I've seen a few variations on the theme of exploding a column/series into multiple columns of a Pandas dataframe, but I've been trying to do something and not really succeeding with the existing approaches. Given a DataFrame like so:
```
key       val
id
2   foo   oranges
2   bar   bananas
2   baz    apples
3   foo    grapes
3   bar     kiwis
```
I want to convert the items in the key series into columns, with the val values serving as the values, like so:
```
foo        bar        baz
id
2   oranges    bananas     apples
3    grapes      kiwis        NaN
```
I feel like this is something that should be relatively straightforward, but I've been bashing my head against this for a few hours now with increasing levels of convolution, and no success.

Response:
There are a few ways: using .pivot_table:
```

```python
df.pivot_table(values='val', index=df.index, columns='key', aggfunc='first')
#Output
#key      bar     baz      foo
#id
#2    bananas  apples  oranges
#3      kiwis     NaN   grapes
#```
#using .pivot:
#```
```

```python
df.pivot(index=df.index, columns='key')['val']
#Output
#key      bar     baz      foo
#id
#2    bananas  apples  oranges
#3      kiwis     NaN   grapes
#```
#using .groupby followed by .unstack:
#```
```

```python
df.reset_index().groupby(['id', 'key'])['val'].aggregate('first').unstack()
#Output
#key      bar     baz      foo
#id
#2    bananas  apples  oranges
#3      kiwis     NaN   grapes
#```
```


Question:
Merge multiple column values into one column in python pandas I have a pandas data frame like this:
```
Column1  Column2  Column3  Column4  Column5
 0    a        1        2        3        4
 1    a        3        4        5
 2    b        6        7        8
 3    c        7        7
```
What I want to do now is getting a new dataframe containing Column1 and a new columnA. This columnA should contain all values from columns 2 -(to) n (where n is the number of columns from Column2 to the end of the row) like this:
```
Column1  ColumnA
0   a      1,2,3,4
1   a      3,4,5
2   b      6,7,8
3   c      7,7
```
How could I best approach this issue?

Response:
You can call apply pass axis=1 to apply row-wise, then convert the dtype to str and join:
```

```python
df['ColumnA'] = df[df.columns[1:]].apply(
    lambda x: ','.join(x.dropna().astype(str)),
    axis=1
)
df
#Output
#Column1  Column2  Column3  Column4  Column5  ColumnA
#0       a        1        2        3        4  1,2,3,4
#1       a        3        4        5      NaN    3,4,5
#2       b        6        7        8      NaN    6,7,8
#3       c        7        7      NaN      NaN      7,7
#```
#Here I call dropna to get rid of the NaN, however we need to cast again to int so we don't end up with floats as str.
```


Question:
Find the end of the month of a Pandas DataFrame Series I have a series within a DataFrame that I read in initially as an object, and then need to convert it to a date in the form of yyyy-mm-dd where dd is the end of the month. As an example, I have DataFrame df with a column Date as an object:
```
...      Date    ...
...     200104   ...
...     200508   ...
```
What I want when this is all said and done is a date object:
```
...      Date    ...
...  2001-04-30  ...
...  2005-08-31  ...
```
such that df['Date'].item() returns
```
datetime.date(2001, 04, 30)
```
I've used the following code to get almost there, but all my dates are at the beginning of the month, not the end. Please advise.
```
df['Date'] = pd.to_datetime(df['Date'], format="%Y%m").dt.date
```
Note: I've already imported Pandas (pd), and datetime (dt)

Response:
You can use pandas.tseries.offsets.MonthEnd:
```
from pandas.tseries.offsets import MonthEnd

df['Date'] = pd.to_datetime(df['Date'], format="%Y%m") + MonthEnd(0)
```
The 0 in MonthEnd just specifies to roll forward to the end of the given month. Note that if we'd used MonthEnd(1), then we'd have got the next date which is at the end of the month. If you wanted the last day of the next month, you'd then add an extra MonthEnd(1), etc. This should work for any month, so you don't need to know the number days in the month, or anything like that. More offset information can be found in the documentation. Example usage and output:
```
df = pd.DataFrame({'Date': [200104, 200508, 201002, 201602, 199912, 200611]})
df['EndOfMonth'] = pd.to_datetime(df['Date'], format="%Y%m") + MonthEnd(1)

     Date EndOfMonth
0  200104 2001-04-30
1  200508 2005-08-31
2  201002 2010-02-28
3  201602 2016-02-29
4  199912 1999-12-31
5  200611 2006-11-30
```