Question:
How to get rid of multilevel index after using pivot table pandas? I had following data frame (the real data frame is much more larger than this one ) :
```
sale_user_id    sale_product_id count
1                 1              1
1                 8              1
1                 52             1
1                 312            5
1                 315            1
```
Then reshaped it to move the values in sale_product_id as column headers using the following code:
```
reshaped_df=id_product_count.pivot(index='sale_user_id',columns='sale_product_id',values='count')
```
and the resulting data frame is:
```
sale_product_id -1057   1   2   3   4   5   6   8   9   10  ... 98  980 981 982 983 984 985 986 987 99
sale_user_id
1                NaN    1.0 NaN NaN NaN NaN NaN 1.0 NaN NaN ... NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
3                NaN    1.0 NaN NaN NaN NaN NaN NaN NaN NaN ... NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
4                NaN    NaN 1.0 NaN NaN NaN NaN NaN NaN NaN ... NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
```
as you can see we have a multililevel index , what i need is to have sale_user_is in the first column without multilevel indexing: i take the following approach :
```
reshaped_df.reset_index()
```
the the result would be like this i still have the sale_product_id column , but i do not need it anymore:
```
sale_product_id sale_user_id    -1057   1   2   3   4   5   6   8   9   ... 98  980 981 982 983 984 985 986 987 99
0                          1    NaN 1.0 NaN NaN NaN NaN NaN 1.0 NaN ... NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
1                          3    NaN 1.0 NaN NaN NaN NaN NaN NaN NaN ... NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
2                          4    NaN NaN 1.0 NaN NaN NaN NaN NaN NaN ... NaN NaN NaN NaN NaN NaN NaN NaN NaN
```
i can subset this data frame to get rid of sale_product_id but i don't think it would be efficient.I am looking for an efficient way to get rid of multilevel indexing while reshaping the original data frame

Response:
You need remove only index name, use rename_axis (new in pandas 0.18.0):
```
print (reshaped_df)
sale_product_id  1    8    52   312  315
sale_user_id
1                  1    1    1    5    1

print (reshaped_df.index.name)
sale_user_id

print (reshaped_df.rename_axis(None))
sale_product_id  1    8    52   312  315
1                  1    1    1    5    1
```
Another solution working in pandas below 0.18.0:
```
reshaped_df.index.name = None
print (reshaped_df)

sale_product_id  1    8    52   312  315
1                  1    1    1    5    1
```
If need remove columns name also:
```
print (reshaped_df.columns.name)
sale_product_id

print (reshaped_df.rename_axis(None).rename_axis(None, axis=1))
   1    8    52   312  315
1    1    1    1    5    1
```
Another solution:
```
reshaped_df.columns.name = None
reshaped_df.index.name = None
print (reshaped_df)
   1    8    52   312  315
1    1    1    1    5    1
```
EDIT by comment: You need reset_index with parameter drop=True:
```
reshaped_df = reshaped_df.reset_index(drop=True)
print (reshaped_df)
sale_product_id  1    8    52   312  315
0                  1    1    1    5    1

#if need reset index nad remove column name
reshaped_df = reshaped_df.reset_index(drop=True).rename_axis(None, axis=1)
print (reshaped_df)
   1    8    52   312  315
0    1    1    1    5    1
```
Of if need remove only column name:
```
reshaped_df = reshaped_df.rename_axis(None, axis=1)
print (reshaped_df)
              1    8    52   312  315
sale_user_id
1               1    1    1    5    1
```
Edit1: So if need create new column from index and remove columns names:
```
reshaped_df =  reshaped_df.rename_axis(None, axis=1).reset_index()
print (reshaped_df)
   sale_user_id  1  8  52  312  315
0             1  1  1   1    5    1
```


Question: Get business days between start and end date using pandas I'm using pandas and I'm wondering what's the easiest way to get the business days between a start and end date using pandas? There are a lot of posts out there regarding doing this in Python (for example), but I would be interested to use directly pandas as I think that pandas can probably handle this quite easy.

Response:
You can also use date_range for this purpose.
```

```python
pd.date_range('2011-01-05', '2011-01-09', freq=BDay())
#Output
#DatetimeIndex(['2011-01-05', '2011-01-06', '2011-01-07'], dtype='datetime64[ns]', freq='B', tz=None)
#```
#EDIT Or even simpler
#```
```

```python
pd.bdate_range('2011-01-05', '2011-01-09')
#Output
#DatetimeIndex(['2011-01-05', '2011-01-06', '2011-01-07'], dtype='datetime64[ns]', freq='B', tz=None)
#```
#Note that both start and end dates are inclusive. Source: http://pandas.pydata.org/pandas-docs/stable/generated/pandas.bdate_range.html
```


Question:
pandas dataframe remove constant column I have a dataframe that may or may not have columns that are the same value. For example
```
row    A    B
    1      9    0
    2      7    0
    3      5    0
    4      2    0
```
I'd like to return just
```
row    A
   1      9
   2      7
   3      5
   4      2
```
Is there a simple way to identify if any of these columns exist and then remove them?

Response:
I believe this option will be faster than the other answers here as it will traverse the data frame only once for the comparison and short-circuit if a non-unique value is found.
```

```python
df
```

0  1  2
0  1  9  0
1  2  7  0
2  3  7  0

```python
df.loc[:, (df != df.iloc[0]).any()]
```

0  1
0  1  9
1  2  7
2  3  7
```


Question:
How to write Pandas dataframe to sqlite with Index I have a list of stockmarket data pulled from Yahoo in a pandas DataFrame (see format below). The date is serving as the index in the DataFrame. I want to write the data (including the index) out to a SQLite database.
```
AAPL     GE
Date
2009-01-02  89.95  14.76
2009-01-05  93.75  14.38
2009-01-06  92.20  14.58
2009-01-07  90.21  13.93
2009-01-08  91.88  13.95
```
Based on my reading of the write_frame code for Pandas, it does not currently support writing the index. I've attempted to use to_records instead, but ran into the issue with Numpy 1.6.2 and datetimes. Now I'm trying to write tuples using .itertuples, but SQLite throws an error that the data type isn't supported (see code and result below). I'm relatively new to Python, Pandas and Numpy, so it is entirely possible I'm missing something obvious. I think I'm running into a problem trying to write a datetime to SQLite, but I think I might be overcomplicating this. I think I may be able to fix the issue by upgrading to Numpy 1.7 or the development version of Pandas, which has a fix posted on GitHub. I'd prefer to develop using release versions of software - I'm new to this and I don't want stability issues confusing matters further. Is there a way to accomplish this using Python 2.7.2, Pandas 0.10.0, and Numpy 1.6.2? Perhaps cleaning the datetimes somehow? I'm in a bit over my head, any help would be appreciated. Code:
```
import numpy as np
import pandas as pd
from pandas import DataFrame, Series
import sqlite3 as db

# download data from yahoo
all_data = {}

for ticker in ['AAPL', 'GE']:
    all_data[ticker] = pd.io.data.get_data_yahoo(ticker, '1/1/2009','12/31/2012')

# create a data frame
price = DataFrame({tic: data['Adj Close'] for tic, data in all_data.iteritems()})

# get output ready for database export
output = price.itertuples()
data = tuple(output)

# connect to a test DB with one three-column table titled "Demo"
con = db.connect('c:/Python27/test.db')
wildcards = ','.join(['?'] * 3)
insert_sql = 'INSERT INTO Demo VALUES (%s)' % wildcards
con.executemany(insert_sql, data)
```
Result:
```
---------------------------------------------------------------------------
InterfaceError                            Traceback (most recent call last)
<ipython-input-15-680cc9889c56> in <module>()
----> 1 con.executemany(insert_sql, data)

InterfaceError: Error binding parameter 0 - probably unsupported type.
```

Response:
In recent pandas the index will be saved in the database (you used to have to reset_index first). Following the docs (setting a SQLite connection in memory):
```
import sqlite3
# Create your connection.
cnx = sqlite3.connect(':memory:')
```
Note: You can also pass a SQLAlchemy engine here (see end of answer). We can save price2 to cnx:
```
price2.to_sql(name='price2', con=cnx)
```
We can retrieve via read_sql:
```
p2 = pd.read_sql('select * from price2', cnx)
```
However, when stored (and retrieved) dates are unicode rather than Timestamp. To convert back to what we started with we can use pd.to_datetime:
```
p2.Date = pd.to_datetime(p2.Date)
p = p2.set_index('Date')
```
We get back the same DataFrame as prices:
```

```python
p2
#Output
#<class 'pandas.core.frame.DataFrame'>
#DatetimeIndex: 1006 entries, 2009-01-02 00:00:00 to 2012-12-31 00:00:00
#Data columns:
#AAPL    1006  non-null values
#GE      1006  non-null values
#dtypes: float64(2)
#```
#You can also use a SQLAlchemy engine:
#```
#from sqlalchemy import create_engine
#e = create_engine('sqlite://')  # pass your db url
#
#price2.to_sql(name='price2', con=cnx)
#```
#This allows you to use read_sql_table (which can only be used with SQLAlchemy):
#```
#pd.read_sql_table(table_name='price2', con=e)
##         Date   AAPL     GE
## 0 2009-01-02  89.95  14.76
## 1 2009-01-05  93.75  14.38
## 2 2009-01-06  92.20  14.58
## 3 2009-01-07  90.21  13.93
## 4 2009-01-08  91.88  13.95
#```
```


Question:
Pandas DataFrame merge summing column I'm trying to merge two DataFrames summing columns value.
```

```python
print(df1)
#Output
#   id name  weight
#0   1    A       0
#1   2    B      10
#2   3    C      10
```

```python
print(df2)
#Output
#   id name  weight
#0   2    B      15
#1   3    C      10
#```
#I need to sum weight values during merging for similar values in the common column.
#```
#merge = pd.merge(df1, df2, how='inner')
#```
#So the output will be something like following.
#```
#id name  weight
#1   2    B      25
#2   3    C      20
#```
```

Response:
This solution works also if you want to sum more than one column. Assume data frames
```

```python
df1
#Output
#   id name  weight  height
#0   1    A       0       5
#1   2    B      10      10
#2   3    C      10      15
```

```python
df2
#Output
#   id name  weight  height
#0   2    B      25      20
#1   3    C      20      30
#```
#You can concatenate them and group by index columns.
#```
```

```python
pd.concat([df1, df2]).groupby(['id', 'name']).sum().reset_index()
#Output
#   id name  weight  height
#0   1    A       0       5
#1   2    B      35      30
#2   3    C      30      45
#```
```


Question:
Convert a column of datetimes to epoch in Python I'm currently having an issue with Python. I have a Pandas DataFrame and one of the columns is a string with a date. The format is : "%Y-%m-%d %H:%m:00.000". For example : "2011-04-24 01:30:00.000" I need to convert the entire column to integers. I tried to run this code, but it is extremely slow and I have a few million rows.
```
for i in range(calls.shape[0]):
    calls['dateint'][i] = int(time.mktime(time.strptime(calls.DATE[i], "%Y-%m-%d %H:%M:00.000")))
```
Do you guys know how to convert the whole column to epoch time?

Response:
convert the string to a datetime using to_datetime and then subtract datetime 1970-1-1 and call dt.total_seconds():
```

```python
import pandas as pd
import datetime as dt
df = pd.DataFrame({'date':['2011-04-24 01:30:00.000']})
df
#Output
#date
#0  2011-04-24 01:30:00.000
```

```python
df['date'] = pd.to_datetime(df['date'])
df
#Output
#date
#0 2011-04-24 01:30:00
```

```python
(df['date'] - dt.datetime(1970,1,1)).dt.total_seconds()
#Output
#0    1303608600
#Name: date, dtype: float64
#```
#You can see that converting this value back yields the same time:
#```
```

```python
pd.to_datetime(1303608600, unit='s')
#Output
#Timestamp('2011-04-24 01:30:00')
#```
#So you can either add a new column or overwrite:
#```
```

```python
df['epoch'] = (df['date'] - dt.datetime(1970,1,1)).dt.total_seconds()
df
#Output
#date       epoch
#0 2011-04-24 01:30:00  1303608600
#```
#EDIT better method as suggested by @Jeff:
#```
```

```python
df['date'].astype('int64')//1e9
#Output
#0    1303608600
#Name: date, dtype: float64
```

```python
%timeit (df['date'] - dt.datetime(1970,1,1)).dt.total_seconds()
%timeit df['date'].astype('int64')//1e9

100 loops, best of 3: 1.72 ms per loop
1000 loops, best of 3: 275 µs per loop
```
You can also see that it is significantly faster
```


Question:
Seaborn multiple barplots I have a pandas dataframe that looks like this:
```
class       men       woman   children
0   first   0.91468    0.667971   0.660562
1   second  0.30012    0.329380   0.882608
2   third   0.11899    0.189747   0.121259
```
How would I create a plot using seaborn that looks like this? Do I have to rearrange my data in some way? (source: mwaskom at stanford.edu)

Response:
Tested in python 3.12.0, pandas 2.1.1, matplotlib 3.8.0, seaborn 0.13.0 Reshape the DataFrame with pandas.DataFrame.melt or pandas.melt:
```
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt

# convert the dataframe to a long format
dfm = pd.melt(df, id_vars="class", var_name="sex", value_name="survival rate")
dfm
Out:
    class       sex  survival rate
0   first       men       0.914680
1  second       men       0.300120
2   third       men       0.118990
3   first     woman       0.667971
4  second     woman       0.329380
5   third     woman       0.189747
6   first  children       0.660562
7  second  children       0.882608
8   third  children       0.121259
```
Consolidate the plot by creating a single facet with grouped bars, instead of multiple facets with single bars. Plot with the figure-level method sns.catplot
```
g = sns.catplot(x='class', y='survival rate', hue='sex', data=dfm, kind='bar', height=5, aspect=1)
```
Plot with the axes-level method sns.barplot
```
# the following code matches the plot produced by catplot
plt.figure(figsize=(5, 5))
ax = sns.barplot(x='class', y='survival rate', hue='sex', data=dfm)
ax.spines[['top', 'right']].set_visible(False)
sns.move_legend(ax, bbox_to_anchor=(1, 0.5), loc='center left', frameon=False)
```
Deprecated factorplot (v0.8.1 or earlier):
```
sns.factorplot(x='class', y='survival rate', hue='sex', data=df, kind='bar')
```


Question:
How to draw horizontal grid only (using pandas plot + pyplot) I would like to get only horizontal grid using pandas plot. The integrated parameter of pandas only has grid=True or grid=False, so I tried with matplotlib pyplot, changing the axes parameters, specifically with this code:
```py
import pandas as pd
import matplotlib.pyplot as plt
fig = plt.figure()
ax2 = plt.subplot()
ax2.grid(axis='x')
df.plot(kind='bar',ax=ax2, fontsize=10, sort_columns=True)
plt.show(fig)
```
But I get no grid, neither horizontal nor vertical. Is Pandas overwriting the axes? Or am I doing something wrong?

Response:
Try setting the grid after plotting the DataFrame. Also, to get the horizontal grid, you need to use ax2.grid(axis='y'). Below is an answer using a sample DataFrame. I have restructured how you define ax2 by making use of subplots.
```
import pandas as pd
import matplotlib.pyplot as plt

df = pd.DataFrame({'lab':['A', 'B', 'C'], 'val':[10, 30, 20]})

fig, ax2 = plt.subplots()

df.plot(kind='bar',ax=ax2, fontsize=10, sort_columns=True)
ax2.grid(axis='y')
plt.show()
```
Alternatively, you can also do the following: Use the axis object returned from the DataFrame plot directly to turn on the horizontal grid
```
fig = plt.figure()

ax2 = df.plot(kind='bar', fontsize=10, sort_columns=True)
ax2.grid(axis='y')
```
Third option as suggested by @ayorgo in the comments is to chain the two commands as
```
df.plot(kind='bar',ax=ax2, fontsize=10, sort_columns=True).grid(axis='y')
```


Question:
Removing header column from pandas dataframe I have the foll. dataframe: df
```
A   B
0  23  12
1  21  44
2  98  21
```
How do I remove the column names A and B from this dataframe? One way might be to write it into a csv file and then read it in specifying header=None. is there a way to do that without writing out to csv and re-reading?

Response:
I think you cant remove column names, only reset them by range with shape:
```
print df.shape[1]
2

print range(df.shape[1])
[0, 1]

df.columns = range(df.shape[1])
print df
    0   1
0  23  12
1  21  44
2  98  21
```
This is same as using to_csv and read_csv:
```
print df.to_csv(header=None,index=False)
23,12
21,44
98,21

print pd.read_csv(io.StringIO(u""+df.to_csv(header=None,index=False)), header=None)
    0   1
0  23  12
1  21  44
2  98  21
```
Next solution with skiprows:
```
print df.to_csv(index=False)
A,B
23,12
21,44
98,21

print pd.read_csv(io.StringIO(u""+df.to_csv(index=False)), header=None, skiprows=1)
    0   1
0  23  12
1  21  44
2  98  21
```


Question:
Rotate xtick labels in seaborn boxplot? I have a question that is basically the same as a question back from 2014 (see here). However, my script still throws an error. Here is what I do: I have a pandas dataframe with a few columns. I plot a simple boxplot comparison.
```
g = sns.boxplot(x='categories', y='oxygen', hue='target', data=df)
g.set_xticklabels(rotation=30)
```
The graph looks like this: I'd like to rotate the x-labels by 30 degrees. Hence I use g.set_xticklabels(rotation=30). However, I get the following error: set_xticklabels() missing 1 required positional argument: 'labels' I don't know how to pass the matplotlib labels argument to seaborns sns.boxplot. Any ideas?

Response:
The question you link to uses a factorplot. A factorplot returns its own class which has a method called set_xticklabels(rotation). This is different from the set_xticklabels method of the matplotlib Axes. In the linked question's answers there are also other options which you may use
```
ax = sns.boxplot(x='categories', y='oxygen', hue='target', data=df)
ax.set_xticklabels(ax.get_xticklabels(),rotation=30)
```
or
```
ax = sns.boxplot(x='categories', y='oxygen', hue='target', data=df)
plt.setp(ax.get_xticklabels(), rotation=45)
```


Question:
How can I extract the nth row of a pandas data frame as a pandas data frame? Suppose a Pandas dataframe looks like:
```none
BoxRatio  Thrust  Velocity  OnBalRun  vwapGain
5     -0.163  -0.817     0.741     1.702     0.218
8      0.000   0.000     0.732     1.798     0.307
11     0.417  -0.298     2.036     4.107     1.793
13     0.054  -0.574     1.323     2.553     1.185
```
How can I extract the third row (as row3) as a pandas dataframe? In other words, row3.shape should be (1,5) and row3.head() should be:
```none
0.417  -0.298     2.036     4.107     1.793
```

Response:
Use .iloc with double brackets to extract a DataFrame, or single brackets to pull out a Series.
```

```python
import pandas as pd
```

```python
df = pd.DataFrame({'col1': [1, 2], 'col2': [3, 4]})
```

```python
df
#Output
#   col1  col2
#0     1     3
#1     2     4
```

```python
df.iloc[[1]]  # DataFrame result
#Output
#   col1  col2
#1     2     4
```

```python
df.iloc[1]  # Series result
#Output
#col1    2
#col2    4
#Name: 1, dtype: int64
#```
#This extends to other forms of DataFrame indexing as well, namely .loc and .__getitem__():
#```
```

```python
df.loc[:, ['col2']]
#Output
#   col2
#0     3
#1     4
```

```python
df[['col2']]
#Output
#   col2
#0     3
#1     4
#```
```


Question: What is as_index in groupby in pandas? What exactly is the function of as_index in groupby in Pandas?

Response:
print() is your friend when you don't understand a thing. It clears out doubts many times. Take a look:
```
import pandas as pd

df = pd.DataFrame(data={'books':['bk1','bk1','bk1','bk2','bk2','bk3'], 'price': [12,12,12,15,15,17]})

print(df)

print(df.groupby('books', as_index=True).sum())

print(df.groupby('books', as_index=False).sum())
```
Output:
```
books  price
0   bk1     12
1   bk1     12
2   bk1     12
3   bk2     15
4   bk2     15
5   bk3     17

       price
books
bk1       36
bk2       30
bk3       17

  books  price
0   bk1     36
1   bk2     30
2   bk3     17
```
When as_index=True the key(s) you use in groupby() will become an index in the new dataframe. The benefits you get when you set the column as index are: Speed. When you filter values based on the index column eg. df.loc['bk1'], it would be faster because of hashing of index column. It doesn't have to traverse the entire books column to find 'bk1'. It will just calculate the hash value of 'bk1' and find it in 1 go. Ease. When as_index=True you can use this syntax df.loc['bk1'] which is shorter and faster as opposed to df.loc[df.books=='bk1'] which is longer and slower.


Question:
ValueError: cannot insert ID, already exists I have this data:
```
ID   TIME
1    2
1    4
1    2
2    3
```
I want to group the data by ID and calculate the mean time and the size of each group.
```
ID   MEAN_TIME COUNT
1    2.67      3
2    3.00      1
```
If I run this code, then I get an error "ValueError: cannot insert ID, already exists":
```
result = df.groupby(['ID']).agg({'TIME': 'mean', 'ID': 'count'}).reset_index()
```

Response:
Use parameter drop=True which not create new column with index but remove it:
```
result = df.groupby(['ID']).agg({'TIME': 'mean', 'ID': 'count'}).reset_index(drop=True)
print (result)
   ID      TIME
0   3  2.666667
1   1  3.000000
```
But if need new column from index need rename old column names first:
```
result = df.groupby(['ID']).agg({'TIME': 'mean', 'ID': 'count'})
           .rename(columns={'ID':'COUNT','TIME':'MEAN_TIME'})
           .reset_index()
print (result)
   ID  COUNT  MEAN_TIME
0   1      3   2.666667
1   2      1   3.000000
```
Solution if need aggreagate by multiple columns:
```
result = df.groupby(['ID']).agg({'TIME':{'MEAN_TIME': 'mean'}, 'ID': {'COUNT': 'count'}})
result.columns = result.columns.droplevel(0)
print (result.reset_index())
   ID  COUNT  MEAN_TIME
0   1      3   2.666667
1   2      1   3.000000
```


Question: Retrieve name of column from its Index in Pandas I have a pandas dataframe and a numpy array of values of that dataframe. I have the index of a specific column and I already have the row index of an important value. Now I need to get the column name of that particular value from my dataframe. After searching through the documentations, I found out that I can do the opposite but not what I want.

Response:
I think you need index columns names by position (python counts from 0, so for fourth column need 3):
```
colname = df.columns[pos]
```
Sample:
```
df = pd.DataFrame({'A':[1,2,3],
                   'B':[4,5,6],
                   'C':[7,8,9],
                   'D':[1,3,5],
                   'E':[5,3,6],
                   'F':[7,4,3]})

print (df)
   A  B  C  D  E  F
0  1  4  7  1  5  7
1  2  5  8  3  3  4
2  3  6  9  5  6  3

pos = 3
colname = df.columns[pos]
print (colname)
D
```
```
pos = [3,5]
colname = df.columns[pos]
print (colname)
Index(['D', 'F'], dtype='object')
```


Question:
StringIO and pandas read_csv I'm trying to mix StringIO and BytesIO with pandas and struggling with some basic stuff. For example, I can't get "output" below to work, whereas "output2" below does work. But "output" is closer to the real world example I'm trying to do. The way in "output2" is from an old pandas example but not really a useful way for me to do it.
```
import io   # note for python 3 only
            # in python2 need to import StringIO

output = io.StringIO()
output.write('x,y\n')
output.write('1,2\n')

output2 = io.StringIO("""x,y
1,2
""")
```
They seem to be the same in terms of type and contents:
```
type(output) == type(output2)
Out[159]: True

output.getvalue() == output2.getvalue()
Out[160]: True
```
But no, not the same:
```
output == output2
Out[161]: False
```
More to the point of the problem I'm trying to solve:
```
pd.read_csv(output)   # ValueError: No columns to parse from file
pd.read_csv(output2)  # works fine, same as reading from a file
```

Response:
io.StringIO here is behaving just like a file -- you wrote to it, and now the file pointer is pointing at the end. When you try to read from it after that, there's nothing after the point you wrote, so: no columns to parse. Instead, just like you would with an ordinary file, seek to the start, and then read:
```

```python
output = io.StringIO()
```

```python
output.write('x,y\n')
#Output
#4
```

```python
output.write('1,2\n')
#Output
#4
```

```python
output.seek(0)
#Output
#0
```

```python
pd.read_csv(output)
#Output
#   x  y
#0  1  2
#```
```


Question:
Python: ufunc 'add' did not contain a loop with signature matching types dtype('S21') dtype('S21') dtype('S21') I have two dataframes, which both have an Order ID and a date. I wanted to add a flag into the first dataframe df1: if a record with the same order id and date is in dataframe df2, then add a Y:
```
[ df1['R'] = np.where(orders['key'].isin(df2['key']), 'Y', 0)]
```
To accomplish that, I was going to create a key, which would be the concatenation of the order_id and date, but when I try the following code:
```
df1['key']=df1['Order_ID']+'_'+df1['Date']
```
I get this error
```
ufunc 'add' did not contain a loop with signature matching types dtype('S21') dtype('S21') dtype('S21')
```
df1 looks like this:
```
Date | Order_ID | other data points ...
201751 4395674  ...
201762 3487535  ...
```
These are the datatypes:
```
df1.info()
RangeIndex: 157443 entries, 0 to 157442
Data columns (total 6 columns):
Order_ID                                 157429 non-null object
Date                                     157443 non-null int64
...
dtypes: float64(2), int64(2), object(2)
memory usage: 7.2+ MB

df1['Order_ID'].values
array(['782833030', '782834969', '782836416', ..., '783678018',
       '783679806', '783679874'], dtype=object)
```

Response:
The problem is that you can't add an object array (containing strings) to a number array, that's just ambiguous:
```

```python
import pandas as pd
```

```python
pd.Series(['abc', 'def']) + pd.Series([1, 2])
#Output
#TypeError: ufunc 'add' did not contain a loop with signature matching types dtype('<U21') dtype('<U21') dtype('<U21')
#```
#You need to explicitly convert your Dates to str. I don't know how to do that efficiently in pandas but you can use:
#```
#df1['key'] = df1['Order_ID'] + '_' + df1['Date'].apply(str)  # .apply(str) is new
#```
```


Question:
Read excel sheet with multiple header using Pandas I have an excel sheet with multiple header like:
```none
_________________________________________________________________________
    ____|_____|        Header1    |        Header2     |        Header3      |
    ColX|ColY |ColA|ColB|ColC|ColD||ColD|ColE|ColF|ColG||ColH|ColI|ColJ|ColDK|
    1   | ds  | 5  | 6  |9   |10  | .......................................
    2   | dh  |  ..........................................................
    3   | ge  |  ..........................................................
    4   | ew  |  ..........................................................
    5   | er  |  ..........................................................
```
Now here you can see that first two columns do not have headers they are blank but other columns have headers like Header1, Header2 and Header3. So I want to read this sheet and merge it with other sheet with similar structure. I want to merge it on first column 'ColX'. Right now I am doing this:
```py
import pandas as pd

totalMergedSheet = pd.DataFrame([1,2,3,4,5], columns=['ColX'])
file = pd.ExcelFile('ExcelFile.xlsx')
for i in range (1, len(file.sheet_names)):
    df1 = file.parse(file.sheet_names[i-1])
    df2 = file.parse(file.sheet_names[i])
    newMergedSheet = pd.merge(df1, df2, on='ColX')
    totalMergedSheet = pd.merge(totalMergedSheet, newMergedSheet, on='ColX')
```
But I don't know its neither reading columns correctly and I think will not return the results in the way I want. So, I want the resulting frame should be like:
```none
________________________________________________________________________________________________________
    ____|_____|        Header1    |        Header2     |        Header3      |        Header4     |        Header5      |
    ColX|ColY |ColA|ColB|ColC|ColD||ColD|ColE|ColF|ColG||ColH|ColI|ColJ|ColK| ColL|ColM|ColN|ColO||ColP|ColQ|ColR|ColS|
    1   | ds  | 5  | 6  |9   |10  | ..................................................................................
    2   | dh  |  ...................................................................................
    3   | ge  |  ....................................................................................
    4   | ew  |  ...................................................................................
    5   | er  |  ......................................................................................
```

Response:
[See comments for updates and corrections] Pandas already has a function that will read in an entire Excel spreadsheet for you, so you don't need to manually parse/merge each sheet. Take a look pandas.read_excel(). It not only lets you read in an Excel file in a single line, it also provides options to help solve the problem you're having. Since you have subcolumns, what you're looking for is MultiIndexing. By default, pandas will read in the top row as the sole header row. You can pass a header argument into pandas.read_excel() that indicates how many rows are to be used as headers. In your particular case, you'd want header=[0, 1], indicating the first two rows. You might also have multiple sheets, so you can pass sheetname=None as well (this tells it to go through all sheets). The command would be:
```
df_dict = pandas.read_excel('ExcelFile.xlsx', header=[0, 1], sheetname=None)
```
This returns a dictionary where the keys are the sheet names, and the values are the DataFrames for each sheet. If you want to collapse it all into one DataFrame, you can simply use pandas.concat:
```
df = pandas.concat(df_dict.values(), axis=0)
```


Question:
Reversing 'one-hot' encoding in Pandas I want to go from this data frame which is basically one hot encoded.
```

```python
pd.DataFrame({"monkey":[0,1,0],"rabbit":[1,0,0],"fox":[0,0,1]})
#Output
#fox  monkey  rabbit
#    0    0       0       1
#    1    0       1       0
#    2    1       0       0
#    3    0       0       0
#    4    0       0       0
#```
#To this one which is 'reverse' one-hot encoded.
#```
```

```python
pd.DataFrame({"animal":["monkey","rabbit","fox"]})
#Output
#animal
#    0  monkey
#    1  rabbit
#    2     fox
#```
#I imagine there's some sort of clever use of apply or zip to do thins but I'm not sure how... Can anyone help? I've not had much success using indexing etc to try to solve this problem.
```

Response:
UPDATE: as Henry Ecker has already mentioned in his answer, as of Pandas 1.5.0 there is a native Pandas method for doing this - pandas.from_dummies() Demo:
```

```python
s = pd.Series(['dog', 'cat', 'dog', 'bird', 'fox', 'dog'])
```

```python
dummies = pd.get_dummies(s)
```

```python
dummies
#Output
#bird  cat  dog  fox
#0     0    0    1    0
#1     0    1    0    0
#2     0    0    1    0
#3     1    0    0    0
#4     0    0    0    1
#5     0    0    1    0
```

```python
pd.from_dummies(dummies)
#Output
#0   dog
#1   cat
#2   dog
#3  bird
#4   fox
#5   dog
#```
#NOTE: the pd.from_dummies() might work improperly, if dummies have been created with drop_first=True parameter, like: pd.get_dummies(data, drop_first=True) Demo:
#```
```

```python
dummies = pd.get_dummies(s, drop_first=True)
```

```python
dummies
#Output
#cat  dog  fox
#0    0    1    0
#1    1    0    0
#2    0    1    0
#3    0    0    0
#4    0    0    1
#5    0    1    0
```

```python
pd.from_dummies(dummies)
...
ValueError: Dummy DataFrame contains unassigned value(s); First instance in row: 3
```
OLD ANSWER: i think ayhan is right and it should be:
```
df.idxmax(axis=1)
```
This chooses a column label for each row, where the label has the maximum value. Since the data are 1s and 0s, it will pick the positions of 1s. Demo:
```
```

```python
s = pd.Series(['dog', 'cat', 'dog', 'bird', 'fox', 'dog'])
```

```python
s
#Output
#0     dog
#1     cat
#2     dog
#3    bird
#4     fox
#5     dog
#dtype: object
```

```python
pd.get_dummies(s)
#Output
#bird  cat  dog  fox
#0   0.0  0.0  1.0  0.0
#1   0.0  1.0  0.0  0.0
#2   0.0  0.0  1.0  0.0
#3   1.0  0.0  0.0  0.0
#4   0.0  0.0  0.0  1.0
#5   0.0  0.0  1.0  0.0
```

```python
pd.get_dummies(s).idxmax(1)
#Output
#0     dog
#1     cat
#2     dog
#3    bird
#4     fox
#5     dog
#dtype: object
#```
```


Question:
Pandas split column into multiple columns by comma I am trying to split a column into multiple columns based on comma/space separation. My dataframe currently looks like
```
KEYS                                                  1
0   FIT-4270                                          4000.0439
1   FIT-4269                                          4000.0420, 4000.0471
2   FIT-4268                                          4000.0419
3   FIT-4266                                          4000.0499
4   FIT-4265                                          4000.0490, 4000.0499, 4000.0500, 4000.0504,
```
I would like
```
KEYS                                                  1           2            3        4
0   FIT-4270                                          4000.0439
1   FIT-4269                                          4000.0420  4000.0471
2   FIT-4268                                          4000.0419
3   FIT-4266                                          4000.0499
4   FIT-4265                                          4000.0490  4000.0499  4000.0500  4000.0504
```
My code currently removes The KEYS column and I'm not sure why. Could anyone improve or help fix the issue?
```
v = dfcleancsv[1]

#splits the columns by spaces into new columns but removes KEYS?

dfcleancsv = dfcleancsv[1].str.split(' ').apply(Series, 1)
```

Response:
In case someone else wants to split a single column (deliminated by a value) into multiple columns - try this:
```
series.str.split(',', expand=True)
```
This answered the question I came here looking for. Credit to EdChum's code that includes adding the split columns back to the dataframe.
```
pd.concat([df[[0]], df[1].str.split(', ', expand=True)], axis=1)
```
Note: The first argument df[[0]] is DataFrame. The second argument df[1].str.split is the series that you want to split. split Documentation concat Documentation


Question:
Convert Pandas series containing string to boolean I have a DataFrame named df as
```
Order Number       Status
1         1668  Undelivered
2        19771  Undelivered
3    100032108  Undelivered
4         2229    Delivered
5        00056  Undelivered
```
I would like to convert the Status column to boolean (True when Status is Delivered and False when Status is Undelivered) but if Status is neither 'Undelivered' neither 'Delivered' it should be considered as NotANumber or something like that. I would like to use a dict
```
d = {
  'Delivered': True,
  'Undelivered': False
}
```
so I could easily add other string which could be either considered as True or False.

Response:
You can just use map:
```

```python
df = pd.DataFrame({'Status':['Delivered', 'Delivered', 'Undelivered',
                                     'SomethingElse']})
```

```python
df
#Output
#Status
#0      Delivered
#1      Delivered
#2    Undelivered
#3  SomethingElse
```

```python
d = {'Delivered': True, 'Undelivered': False}
```

```python
df['Status'].map(d)
#Output
#0     True
#1     True
#2    False
#3      NaN
#Name: Status, dtype: object
#```
```


Question:
plotting value_counts() in seaborn barplot I'm having trouble getting a barplot in seaborn. Here's my reproducible data:
```
people = ['Hannah', 'Bethany', 'Kris', 'Alex', 'Earl', 'Lori']
reputation = ['awesome', 'cool', 'brilliant', 'meh', 'awesome', 'cool']
dictionary = dict(zip(people, reputation))
df = pd.DataFrame(dictionary.values(), dictionary.keys())
df = df.rename(columns={0:'reputation'})
```
Then I want to get a bar plot showing the value counts of different reputation. I've tried:
```
sns.barplot(x = 'reputation', y = df['reputation'].value_counts(), data = df, ci = None)
```
and
```
sns.barplot(x = 'reputation', y = df['reputation'].value_counts().values, data = df, ci = None)
```
but both return blank plots. Any idea what I can do to get this?

Response:
In the latest seaborn, you can use the countplot function:
```
seaborn.countplot(x='reputation', data=df)
```
To do it with barplot you'd need something like this:
```
seaborn.barplot(x=df.reputation.value_counts().index, y=df.reputation.value_counts())
```
You can't pass 'reputation' as a column name to x while also passing the counts in y. Passing 'reputation' for x will use the values of df.reputation (all of them, not just the unique ones) as the x values, and seaborn has no way to align these with the counts. So you need to pass the unique values as x and the counts as y. But you need to call value_counts twice (or do some other sorting on both the unique values and the counts) to ensure they match up right.


Question:
How to replace a value in pandas, with NaN? I am new to pandas , I am trying to load the csv in Dataframe. My data has missing values represented as ? , and I am trying to replace it with standard Missing values - NaN Kindly help me with this . I have tried reading through Pandas docs, but I am not able to follow.
```py
def readData(filename):
    DataLabels =["age", "workclass", "fnlwgt", "education", "education-num", "marital-status",
               "occupation", "relationship", "race", "sex", "capital-gain",
               "capital-loss", "hours-per-week", "native-country", "class"]

    # ==== trying to replace ? with Nan using na_values
    rawfile = pd.read_csv(filename, header=None, names=DataLabels, na_values=["?"])
    age = rawfile["age"]
    print(age)
    print(rawfile[25:40])

    #========trying to replace ?
    rawfile.replace("?", "NaN")
    print(rawfile[25:40])
    return rawfile
```
```py
age   workclass  fnlwgt      education  education-num       marital-status        occupation    relationship                 race    sex  capital-gain  capital-loss  hours-per-week  native-country   class
25   56   Local-gov  216851      Bachelors             13   Married-civ-spouse      Tech-support         Husband                White   Male             0             0              40   United-States    >50K
26   19     Private  168294        HS-grad              9        Never-married      Craft-repair       Own-child                White   Male             0             0              40   United-States   <=50K
27   54           ?  180211   Some-college             10   Married-civ-spouse                 ?         Husband   Asian-Pac-Islander   Male             0             0              60           South    >50K
28   39     Private  367260        HS-grad              9             Divorced   Exec-managerial   Not-in-family                White   Male             0             0              80   United-States   <=50K
29   49     Private  193366        HS-grad              9   Married-civ-spouse      Craft-repair         Husband                White   Male             0             0              40   United-States   <=50K
```
Data adult.data
```
39, State-gov, 77516, Bachelors, 13, Never-married, Adm-clerical, Not-in-family, White, Male, 2174, 0, 40, United-States, <=50K
50, Self-emp-not-inc, 83311, Bachelors, 13, Married-civ-spouse, Exec-managerial, Husband, White, Male, 0, 0, 13, United-States, <=50K
38, Private, 215646, HS-grad, 9, Divorced, Handlers-cleaners, Not-in-family, White, Male, 0, 0, 40, United-States, <=50K
53, Private, 234721, 11th, 7, Married-civ-spouse, Handlers-cleaners, Husband, Black, Male, 0, 0, 40, United-States, <=50K
28, Private, 338409, Bachelors, 13, Married-civ-spouse, Prof-specialty, Wife, Black, Female, 0, 0, 40, Cuba, <=50K
37, Private, 284582, Masters, 14, Married-civ-spouse, Exec-managerial, Wife, White, Female, 0, 0, 40, United-States, <=50K
49, Private, 160187, 9th, 5, Married-spouse-absent, Other-service, Not-in-family, Black, Female, 0, 0, 16, Jamaica, <=50K
52, Self-emp-not-inc, 209642, HS-grad, 9, Married-civ-spouse, Exec-managerial, Husband, White, Male, 0, 0, 45, United-States, >50K
31, Private, 45781, Masters, 14, Never-married, Prof-specialty, Not-in-family, White, Female, 14084, 0, 50, United-States, >50K
42, Private, 159449, Bachelors, 13, Married-civ-spouse, Exec-managerial, Husband, White, Male, 5178, 0, 40, United-States, >50K
37, Private, 280464, Some-college, 10, Married-civ-spouse, Exec-managerial, Husband, Black, Male, 0, 0, 80, United-States, >50K
30, State-gov, 141297, Bachelors, 13, Married-civ-spouse, Prof-specialty, Husband, Asian-Pac-Islander, Male, 0, 0, 40, India, >50K
23, Private, 122272, Bachelors, 13, Never-married, Adm-clerical, Own-child, White, Female, 0, 0, 30, United-States, <=50K
32, Private, 205019, Assoc-acdm, 12, Never-married, Sales, Not-in-family, Black, Male, 0, 0, 50, United-States, <=50K
40, Private, 121772, Assoc-voc, 11, Married-civ-spouse, Craft-repair, Husband, Asian-Pac-Islander, Male, 0, 0, 40, ?, >50K
34, Private, 245487, 7th-8th, 4, Married-civ-spouse, Transport-moving, Husband, Amer-Indian-Eskimo, Male, 0, 0, 45, Mexico, <=50K
25, Self-emp-not-inc, 176756, HS-grad, 9, Never-married, Farming-fishing, Own-child, White, Male, 0, 0, 35, United-States, <=50K
32, Private, 186824, HS-grad, 9, Never-married, Machine-op-inspct, Unmarried, White, Male, 0, 0, 40, United-States, <=50K
38, Private, 28887, 11th, 7, Married-civ-spouse, Sales, Husband, White, Male, 0, 0, 50, United-States, <=50K
43, Self-emp-not-inc, 292175, Masters, 14, Divorced, Exec-managerial, Unmarried, White, Female, 0, 0, 45, United-States, >50K
40, Private, 193524, Doctorate, 16, Married-civ-spouse, Prof-specialty, Husband, White, Male, 0, 0, 60, United-States, >50K
54, Private, 302146, HS-grad, 9, Separated, Other-service, Unmarried, Black, Female, 0, 0, 20, United-States, <=50K
35, Federal-gov, 76845, 9th, 5, Married-civ-spouse, Farming-fishing, Husband, Black, Male, 0, 0, 40, United-States, <=50K
43, Private, 117037, 11th, 7, Married-civ-spouse, Transport-moving, Husband, White, Male, 0, 2042, 40, United-States, <=50K
59, Private, 109015, HS-grad, 9, Divorced, Tech-support, Unmarried, White, Female, 0, 0, 40, United-States, <=50K
56, Local-gov, 216851, Bachelors, 13, Married-civ-spouse, Tech-support, Husband, White, Male, 0, 0, 40, United-States, >50K
19, Private, 168294, HS-grad, 9, Never-married, Craft-repair, Own-child, White, Male, 0, 0, 40, United-States, <=50K
54, ?, 180211, Some-college, 10, Married-civ-spouse, ?, Husband, Asian-Pac-Islander, Male, 0, 0, 60, South, >50K
39, Private, 367260, HS-grad, 9, Divorced, Exec-managerial, Not-in-family, White, Male, 0, 0, 80, United-States, <=50K
49, Private, 193366, HS-grad, 9, Married-civ-spouse, Craft-repair, Husband, White, Male, 0, 0, 40, United-States, <=50K
23, Local-gov, 190709, Assoc-acdm, 12, Never-married, Protective-serv, Not-in-family, White, Male, 0, 0, 52, United-States, <=50K
20, Private, 266015, Some-college, 10, Never-married, Sales, Own-child, Black, Male, 0, 0, 44, United-States, <=50K
45, Private, 386940, Bachelors, 13, Divorced, Exec-managerial, Own-child, White, Male, 0, 1408, 40, United-States, <=50K
30, Federal-gov, 59951, Some-college, 10, Married-civ-spouse, Adm-clerical, Own-child, White, Male, 0, 0, 40, United-States, <=50K
22, State-gov, 311512, Some-college, 10, Married-civ-spouse, Other-service, Husband, Black, Male, 0, 0, 15, United-States, <=50K
48, Private, 242406, 11th, 7, Never-married, Machine-op-inspct, Unmarried, White, Male, 0, 0, 40, Puerto-Rico, <=50K
21, Private, 197200, Some-college, 10, Never-married, Machine-op-inspct, Own-child, White, Male, 0, 0, 40, United-States, <=50K
19, Private, 544091, HS-grad, 9, Married-AF-spouse, Adm-clerical, Wife, White, Female, 0, 0, 25, United-States, <=50K
31, Private, 84154, Some-college, 10, Married-civ-spouse, Sales, Husband, White, Male, 0, 0, 38, ?, >50K
48, Self-emp-not-inc, 265477, Assoc-acdm, 12, Married-civ-spouse, Prof-specialty, Husband, White, Male, 0, 0, 40, United-States, <=50K
31, Private, 507875, 9th, 5, Married-civ-spouse, Machine-op-inspct, Husband, White, Male, 0, 0, 43, United-States, <=50K
53, Self-emp-not-inc, 88506, Bachelors, 13, Married-civ-spouse, Prof-specialty, Husband, White, Male, 0, 0, 40, United-States, <=50K
24, Private, 172987, Bachelors, 13, Married-civ-spouse, Tech-support, Husband, White, Male, 0, 0, 50, United-States, <=50K
49, Private, 94638, HS-grad, 9, Separated, Adm-clerical, Unmarried, White, Female, 0, 0, 40, United-States, <=50K
25, Private, 289980, HS-grad, 9, Never-married, Handlers-cleaners, Not-in-family, White, Male, 0, 0, 35, United-States, <=50K
57, Federal-gov, 337895, Bachelors, 13, Married-civ-spouse, Prof-specialty, Husband, Black, Male, 0, 0, 40, United-States, >50K
53, Private, 144361, HS-grad, 9, Married-civ-spouse, Machine-op-inspct, Husband, White, Male, 0, 0, 38, United-States, <=50K
44, Private, 128354, Masters, 14, Divorced, Exec-managerial, Unmarried, White, Female, 0, 0, 40, United-States, <=50K
41, State-gov, 101603, Assoc-voc, 11, Married-civ-spouse, Craft-repair, Husband, White, Male, 0, 0, 40, United-States, <=50K
29, Private, 271466, Assoc-voc, 11, Never-married, Prof-specialty, Not-in-family, White, Male, 0, 0, 43, United-States, <=50K
25, Private, 32275, Some-college, 10, Married-civ-spouse, Exec-managerial, Wife, Other, Female, 0, 0, 40, United-States, <=50K
18, Private, 226956, HS-grad, 9, Never-married, Other-service, Own-child, White, Female, 0, 0, 30, ?, <=50K
47, Private, 51835, Prof-school, 15, Married-civ-spouse, Prof-specialty, Wife, White, Female, 0, 1902, 60, Honduras, >50K
50, Federal-gov, 251585, Bachelors, 13, Divorced, Exec-managerial, Not-in-family, White, Male, 0, 0, 55, United-States, >50K
47, Self-emp-inc, 109832, HS-grad, 9, Divorced, Exec-managerial, Not-in-family, White, Male, 0, 0, 60, United-States, <=50K
43, Private, 237993, Some-college, 10, Married-civ-spouse, Tech-support, Husband, White, Male, 0, 0, 40, United-States, >50K
46, Private, 216666, 5th-6th, 3, Married-civ-spouse, Machine-op-inspct, Husband, White, Male, 0, 0, 40, Mexico, <=50K
35, Private, 56352, Assoc-voc, 11, Married-civ-spouse, Other-service, Husband, White, Male, 0, 0, 40, Puerto-Rico, <=50K
41, Private, 147372, HS-grad, 9, Married-civ-spouse, Adm-clerical, Husband, White, Male, 0, 0, 48, United-States, <=50K
30, Private, 188146, HS-grad, 9, Married-civ-spouse, Machine-op-inspct, Husband, White, Male, 5013, 0, 40, United-States, <=50K
30, Private, 59496, Bachelors, 13, Married-civ-spouse, Sales, Husband, White, Male, 2407, 0, 40, United-States, <=50K
32, ?, 293936, 7th-8th, 4, Married-spouse-absent, ?, Not-in-family, White, Male, 0, 0, 40, ?, <=50K
48, Private, 149640, HS-grad, 9, Married-civ-spouse, Transport-moving, Husband, White, Male, 0, 0, 40, United-States, <=50K
42, Private, 116632, Doctorate, 16, Married-civ-spouse, Prof-specialty, Husband, White, Male, 0, 0, 45, United-States, >50K
29, Private, 105598, Some-college, 10, Divorced, Tech-support, Not-in-family, White, Male, 0, 0, 58, United-States, <=50K
36, Private, 155537, HS-grad, 9, Married-civ-spouse, Craft-repair, Husband, White, Male, 0, 0, 40, United-States, <=50K
28, Private, 183175, Some-college, 10, Divorced, Adm-clerical, Not-in-family, White, Female, 0, 0, 40, United-States, <=50K
53, Private, 169846, HS-grad, 9, Married-civ-spouse, Adm-clerical, Wife, White, Female, 0, 0, 40, United-States, >50K
49, Self-emp-inc, 191681, Some-college, 10, Married-civ-spouse, Exec-managerial, Husband, White, Male, 0, 0, 50, United-States, >50K
25, ?, 200681, Some-college, 10, Never-married, ?, Own-child, White, Male, 0, 0, 40, United-States, <=50K
19, Private, 101509, Some-college, 10, Never-married, Prof-specialty, Own-child, White, Male, 0, 0, 32, United-States, <=50K
31, Private, 309974, Bachelors, 13, Separated, Sales, Own-child, Black, Female, 0, 0, 40, United-States, <=50K
29, Self-emp-not-inc, 162298, Bachelors, 13, Married-civ-spouse, Sales, Husband, White, Male, 0, 0, 70, United-States, >50K
23, Private, 211678, Some-college, 10, Never-married, Machine-op-inspct, Not-in-family, White, Male, 0, 0, 40, United-States, <=50K
79, Private, 124744, Some-college, 10, Married-civ-spouse, Prof-specialty, Other-relative, White, Male, 0, 0, 20, United-States, <=50K
```

Response:
You can replace this just for that column using replace:
```
df['workclass'].replace('?', np.NaN)
```
or for the whole df:
```
df.replace('?', np.NaN)
```
UPDATE OK I figured out your problem, by default if you don't pass a separator character then read_csv will use commas ',' as the separator. Your data and in particular one example where you have a problematic line:
```
54, ?, 180211, Some-college, 10, Married-civ-spouse, ?, Husband, Asian-Pac-Islander, Male, 0, 0, 60, South, >50K
```
has in fact a comma and a space as the separator so when you passed the na_value=['?'] this didn't match because all your values have a space character in front of them all which you can't observe. if you change your line to this:
```
rawfile = pd.read_csv(filename, header=None, names=DataLabels, sep=',\s', na_values=["?"])
```
then you should find that it all works:
```
27      54               NaN  180211  Some-college             10
```


Question:
How to extract the regression coefficient from statsmodels.api?
```
result = sm.OLS(gold_lookback, silver_lookback ).fit()
```
After I get the result, how can I get the coefficient and the constant? In other words, if y = ax + c how to get the values a and c?

Response:
You can use the params property of a fitted model to get the coefficients. For example, the following code:
```
import statsmodels.api as sm
import numpy as np
np.random.seed(1)
X = sm.add_constant(np.arange(100))
y = np.dot(X, [1,2]) + np.random.normal(size=100)
result = sm.OLS(y, X).fit()
print(result.params)
```
will print you a numpy array [ 0.89516052 2.00334187] - estimates of intercept and slope respectively. If you want more information, you can use the object result.summary() that contains 3 detailed tables with model description.


Question:
Dataframe set_index not setting I have a dataframe and am trying to set the index to the column 'Timestamp'. Currently the index is just a row number. An example of Timestamp's format is: 2015-09-03 16:35:00 I've tried to set the index:
```
df.set_index('Timestamp')
```
I don't get an error, but when I print the dataframe, the index is still the row number. How can I use Timestamp as the index?

Response:
You need to either specify inplace=True, or assign the result to a variable. Try:
```
df.set_index('Timestamp', inplace=True, drop=True)
```
Basically, there are two things that you might want to do when you set the index. One is new_df = old_df.set_index('Timestamp', inplace=False). I.e. You want a new DataFrame that has the new index, but still want a copy of the original DataFrame. The other is df.set_index('Timestamp', inplace=True). Which is for when you want to modify the existing object.


Question:
How to remove decimal points in pandas I have a pandas data frame, df, which looks like this:
```
Cut-off             <=35   >35
Calcium              0.0   1.0
Copper               1.0   0.0
Helium               0.0   8.0
Hydrogen             0.0   1.0
```
How can I remove the decimal point so that the data frame looks like this:
```
Cut-off             <= 35  > 35
Calcium              0     1
Copper               1     0
Helium               0     8
Hydrogen             0     1
```
I have tried df.round(0) without success.

Response:
You have a few options... 1) convert everything to integers.
```
df.astype(int)
          <=35  >35
Cut-off
Calcium      0    1
Copper       1    0
Helium       0    8
Hydrogen     0    1
```
2) Use round:
```

```python
df.round()
#Output
#          <=35  >35
#Cut-off
#Calcium      0    1
#Copper       1    0
#Helium       0    8
#Hydrogen     0    1
#```
#but not always great...
#```
```

```python
(df - .2).round()
#Output
#          <=35  >35
#Cut-off
#Calcium     -0    1
#Copper       1   -0
#Helium      -0    8
#Hydrogen    -0    1
#```
#3) Change your display precision option in Pandas.
#```
#pd.set_option('precision', 0)
```

```python
df
#Output
#          <=35  >35
#Cut-off
#Calcium      0    1
#Copper       1    0
#Helium       0    8
#Hydrogen     0    1
#```
```


Question: pandas pd.options.display.max_rows not working as expected I’m using pandas 0.25.1 in Jupyter Lab and the maximum number of rows I can display is 10, regardless of what pd.options.display.max_rows is set to. However, if pd.options.display.max_rows is set to less than 10 it takes effect and if pd.options.display.max_rows = None then all rows show. Any idea how I can get a pd.options.display.max_rows of more than 10 to take effect?

Response:
Final answer as of pandas v0.25.1 To display all the rows, set max_rows greater than the number of rows in the DataFrame. To display more than 10 rows when the dataframe is truncated, set min_rows greater than 10. With more than 200 rows of data, if max_rows is 200 and min_rows is 20, 10 from the head and 10 from the tail will be displayed. With more than 200 rows of data, if max_rows is 200 and min_rows is None, 100 from the head and 100 from the tail will be displayed. Discovery & Notes: Both pd.set_option('display.max_rows', x) and pd.options.display.max_rows = x (where x is some number), should work. FYI: the default is 10 rows, as per pandas.set_option Options and settings This is also in Jupyter Lab Other useful pandas options:
```py
pd.set_option('display.max_columns', 200)
pd.set_option('display.max_rows', 100)
pd.set_option('display.min_rows', 100)
pd.set_option('display.expand_frame_repr', True)
```
get.options to return the current value:
```py
pd.get_option("display.max_rows")
```
Update: After some discovery testing, the setting only displays more rows, if the setting is greater than the number of rows in the DataFrame. If pd.set_option('display.max_rows', 100), but the DataFrame has 200 rows, only 10 will show up. If pd.set_option('display.max_rows', 200), but the DataFrame has 100 rows, all 100 will display. Per the pandas docs: display.max_rows: default=60 This sets the maximum number of rows pandas should output when printing out various output. For example, this value determines whether the repr() for a dataframe prints out fully or just a truncated or summary repr. ‘None’ value means unlimited. display.min_rows: default=10 The numbers of rows to show in a truncated repr (when max_rows is exceeded). Ignored when max_rows is set to None or 0. When set to None, follows the value of max_rows. display.large_repr: default=truncate For DataFrames exceeding max_rows/max_cols, the repr (and HTML repr) can show a truncated table (the default), or switch to the view from df.info() (the behaviour in earlier versions of pandas). allowable settings, [‘truncate’, ‘info’]


Question:
Writing pandas DataFrame to JSON in unicode I'm trying to write a pandas DataFrame containing unicode to json, but the built in .to_json function escapes the non-ascii characters. How do I fix this? Example:
```
import pandas as pd

df = pd.DataFrame([["τ", "a", 1], ["π", "b", 2]])
df.to_json("df.json")
```
This gives:
```
{"0":{"0":"\u03c4","1":"\u03c0"},"1":{"0":"a","1":"b"},"2":{"0":1,"1":2}}
```
Which differs from the desired result:
```
{"0":{"0":"τ","1":"π"},"1":{"0":"a","1":"b"},"2":{"0":1,"1":2}}
```
I have tried adding the force_ascii=False argument:
```
import pandas as pd

df = pd.DataFrame([["τ", "a", 1], ["π", "b", 2]])
df.to_json("df.json", force_ascii=False)
```
But this gives the following error:
```
UnicodeEncodeError: 'charmap' codec can't encode character '\u03c4' in position 11: character maps to <undefined>
```
This occurs on pandas versions 0.18 to 2.2+, on python 3.4 to 3.12+

Response:
Opening a file with the encoding set to utf-8, and then passing that file to the .to_json function fixes the problem:
```
with open('df.json', 'w', encoding='utf-8') as file:
    df.to_json(file, force_ascii=False)
```
gives the correct:
```
{"0":{"0":"τ","1":"π"},"1":{"0":"a","1":"b"},"2":{"0":1,"1":2}}
```
Note: it does still require the force_ascii=False argument.


Question: Delete rows if there are null values in a specific column in Pandas dataframe [duplicate] This question already has answers here: How to drop a row whose particular column is empty/NaN? (2 answers) How do I select rows from a DataFrame based on column values? (20 answers) Closed 7 years ago. I'm new to python pandas. Need some help with deleting a few rows where there are null values. In the screenshot, I need to delete rows where charge_per_line == "-" using python pandas.

Response:
If the relevant entries in Charge_Per_Line are empty (NaN) when you read into pandas, you can use df.dropna:
```
df = df.dropna(axis=0, subset=['Charge_Per_Line'])
```
If the values are genuinely -, then you can replace them with np.nan and then use df.dropna:
```
import numpy as np

df['Charge_Per_Line'] = df['Charge_Per_Line'].replace('-', np.nan)
df = df.dropna(axis=0, subset=['Charge_Per_Line'])
```


Question: Find unique values in a Pandas dataframe, irrespective of row or column location I have a Pandas dataframe and I want to find all the unique values in that dataframe...irrespective of row/columns. If I have a 10 x 10 dataframe, and suppose they have 84 unique values, I need to find them - Not the count. I can create a set and add the values of each rows by iterating over the rows of the dataframe. But, I feel that it may be inefficient (cannot justify that). Is there an efficient way to find it? Is there a predefined function?

Response:
```

```python
df = DataFrame(np.random.randint(0,10,size=100).reshape(10,10))
```

```python
df
#Output
#0  1  2  3  4  5  6  7  8  9
#0  2  2  3  2  6  1  9  9  3  3
#1  1  2  5  8  5  2  5  0  6  3
#2  0  7  0  7  5  5  9  1  0  3
#3  5  3  2  3  7  6  8  3  8  4
#4  8  0  2  2  3  9  7  1  2  7
#5  3  2  8  5  6  4  3  7  0  8
#6  4  2  6  5  3  3  4  5  3  2
#7  7  6  0  6  6  7  1  7  5  1
#8  7  4  3  1  0  6  9  7  7  3
#9  5  3  4  5  2  0  8  6  4  7
```

```python
Series(df.values.ravel()).unique()
#Output
#array([9, 1, 4, 6, 0, 7, 5, 8, 3, 2])
#```
#Numpy unique sorts, so its faster to do it this way (and then sort if you need to)
#```
```

```python
df = DataFrame(np.random.randint(0,10,size=10000).reshape(100,100))
```

```python
%timeit Series(df.values.ravel()).unique()
10000 loops, best of 3: 137 ﾵs per loop
```

```python
%timeit np.unique(df.values.ravel())
1000 loops, best of 3: 270 ﾵs per loop
```
```


Question:
Count distinct words from a Pandas Data Frame I've a Pandas data frame, where one column contains text. I'd like to get a list of unique words appearing across the entire column (space being the only split).
```
import pandas as pd

r1=['My nickname is ft.jgt','Someone is going to my place']

df=pd.DataFrame(r1,columns=['text'])
```
The output should look like this:
```
['my','nickname','is','ft.jgt','someone','going','to','place']
```
It wouldn't hurt to get a count as well, but it is not required.

Response:
Use a set to create the sequence of unique elements. Do some clean-up on df to get the strings in lower case and split:
```
df['text'].str.lower().str.split()
Out[43]:
0             [my, nickname, is, ft.jgt]
1    [someone, is, going, to, my, place]
```
Each list in this column can be passed to set.update function to get unique values. Use apply to do so:
```
results = set()
df['text'].str.lower().str.split().apply(results.update)
print(results)

set(['someone', 'ft.jgt', 'my', 'is', 'to', 'going', 'place', 'nickname'])
```
Or use with Counter() from comments:
```
from collections import Counter
results = Counter()
df['text'].str.lower().str.split().apply(results.update)
print(results)
```


Question:
Python pandas: remove everything after a delimiter in a string I have data frames which contain e.g.:
```
"vendor a::ProductA"
"vendor b::ProductA"
"vendor a::Productb"
```
I need to remove everything (and including) the two :: so that I end up with:
```
"vendor a"
"vendor b"
"vendor a"
```
I tried str.trim (which seems to not exist) and str.split without success. what would be the easiest way to accomplish this?

Response:
You can use pandas.Series.str.split just like you would use split normally. Just split on the string '::', and index the list that's created from the split method:
```

```python
df = pd.DataFrame({'text': ["vendor a::ProductA", "vendor b::ProductA", "vendor a::Productb"]})
```

```python
df
#Output
#                 text
#0  vendor a::ProductA
#1  vendor b::ProductA
#2  vendor a::Productb
```

```python
df['text_new'] = df['text'].str.split('::').str[0]
```

```python
df
#Output
#                 text  text_new
#0  vendor a::ProductA  vendor a
#1  vendor b::ProductA  vendor b
#2  vendor a::Productb  vendor a
#```
#Here's a non-pandas solution:
#```
```

```python
df['text_new1'] = [x.split('::')[0] for x in df['text']]
```

```python
df
#Output
#                 text  text_new text_new1
#0  vendor a::ProductA  vendor a  vendor a
#1  vendor b::ProductA  vendor b  vendor b
#2  vendor a::Productb  vendor a  vendor a
#```
#Edit: Here's the step-by-step explanation of what's happening in pandas above:
#```
## Select the pandas.Series object you want
```

```python
df['text']
#Output
#0    vendor a::ProductA
#1    vendor b::ProductA
#2    vendor a::Productb
#Name: text, dtype: object
```

# using pandas.Series.str allows us to implement "normal" string methods
# (like split) on a Series

```python
df['text'].str
#Output
#<pandas.core.strings.StringMethods object at 0x110af4e48>
```

# Now we can use the split method to split on our '::' string. You'll see that
# a Series of lists is returned (just like what you'd see outside of pandas)

```python
df['text'].str.split('::')
#Output
#0    [vendor a, ProductA]
#1    [vendor b, ProductA]
#2    [vendor a, Productb]
#Name: text, dtype: object
```

# using the pandas.Series.str method, again, we will be able to index through
# the lists returned in the previous step

```python
df['text'].str.split('::').str
#Output
#<pandas.core.strings.StringMethods object at 0x110b254a8>
```

# now we can grab the first item in each list above for our desired output

```python
df['text'].str.split('::').str[0]
#Output
#0    vendor a
#1    vendor b
#2    vendor a
#Name: text, dtype: object
#```
#I would suggest checking out the pandas.Series.str docs, or, better yet, Working with Text Data in pandas.
```


Question:
How to save the Pandas dataframe/series data as a figure? It sounds somewhat weird， but I need to save the Pandas console output string to png pics. For example:
```

```python
df
#Output
#                   sales  net_pft     ROE    ROIC
#STK_ID RPT_Date
#600809 20120331  22.1401   4.9253  0.1651  0.6656
#       20120630  38.1565   7.8684  0.2567  1.0385
#       20120930  52.5098  12.4338  0.3587  1.2867
#       20121231  64.7876  13.2731  0.3736  1.2205
#       20130331  27.9517   7.5182  0.1745  0.3723
#       20130630  40.6460   9.8572  0.2560  0.4290
#       20130930  53.0501  11.8605  0.2927  0.4369
#```
#Is there any way like df.output_as_png(filename='df_data.png') to generate a pic file which just display above content inside?
```

Response:
Option-1: use matplotlib table functionality, with some additional styling:
```
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

df = pd.DataFrame()
df['date'] = ['2016-04-01', '2016-04-02', '2016-04-03']
df['calories'] = [2200, 2100, 1500]
df['sleep hours'] = [8, 7.5, 8.2]
df['gym'] = [True, False, False]

def render_mpl_table(data, col_width=3.0, row_height=0.625, font_size=14,
                     header_color='#40466e', row_colors=['#f1f1f2', 'w'], edge_color='w',
                     bbox=[0, 0, 1, 1], header_columns=0,
                     ax=None, **kwargs):
    if ax is None:
        size = (np.array(data.shape[::-1]) + np.array([0, 1])) * np.array([col_width, row_height])
        fig, ax = plt.subplots(figsize=size)
        ax.axis('off')
    mpl_table = ax.table(cellText=data.values, bbox=bbox, colLabels=data.columns, **kwargs)
    mpl_table.auto_set_font_size(False)
    mpl_table.set_fontsize(font_size)

    for k, cell in mpl_table._cells.items():
        cell.set_edgecolor(edge_color)
        if k[0] == 0 or k[1] < header_columns:
            cell.set_text_props(weight='bold', color='w')
            cell.set_facecolor(header_color)
        else:
            cell.set_facecolor(row_colors[k[0]%len(row_colors) ])
    return ax.get_figure(), ax

fig,ax = render_mpl_table(df, header_columns=0, col_width=2.0)
fig.savefig("table_mpl.png")
```
Options-2 Use Plotly + kaleido
```
import plotly.figure_factory as ff
import pandas as pd

df = pd.DataFrame()
df['date'] = ['2016-04-01', '2016-04-02', '2016-04-03']
df['calories'] = [2200, 2100, 1500]
df['sleep hours'] = [8, 7.5, 8.2]
df['gym'] = [True, False, False]

fig =  ff.create_table(df)
fig.update_layout(
    autosize=False,
    width=500,
    height=200,
)
fig.write_image("table_plotly.png", scale=2)
fig.show()
```
For the above, the font size can be changed using the font attribute:
```
fig.update_layout(
    autosize=False,
    width=500,
    height=200,
    font={'size':8}
)
```


Question:
How to unpack a Series of tuples in Pandas? Sometimes I end up with a series of tuples/lists when using Pandas. This is common when, for example, doing a group-by and passing a function that has multiple return values:
```
import numpy as np
from scipy import stats
df = pd.DataFrame(dict(x=np.random.randn(100),
                       y=np.repeat(list("abcd"), 25)))
out = df.groupby("y").x.apply(stats.ttest_1samp, 0)
print out

y
a       (1.3066417476, 0.203717485506)
b    (0.0801133382517, 0.936811414675)
c      (1.55784329113, 0.132360504653)
d     (0.267999459642, 0.790989680709)
dtype: object
```
What is the correct way to "unpack" this structure so that I get a DataFrame with two columns? A related question is how I can unpack either this structure or the resulting dataframe into two Series/array objects. This almost works:
```
t, p = zip(*out)
```
but it t is
```
(array(1.3066417475999257),
 array(0.08011333825171714),
 array(1.557843291126335),
 array(0.267999459641651))
```
and one needs to take the extra step of squeezing it.

Response:
Maybe this is most strightforward (most pythonic i guess):
```
out.apply(pd.Series)
```
If you wanted to rename the columns to something more meaningful, than:
```
out.columns=['Kstats','Pvalue']
```
If you do not want the default name for the index:
```
out.index.name=None
```


Question:
Creating dummy variables in pandas for python I'm trying to create a series of dummy variables from a categorical variable using pandas in python. I've come across the get_dummies function, but whenever I try to call it I receive an error that the name is not defined. Any thoughts or other ways to create the dummy variables would be appreciated. EDIT: Since others seem to be coming across this, the get_dummies function in pandas now works perfectly fine. This means the following should work:
```
import pandas as pd

dummies = pd.get_dummies(df['Category'])
```
See https://web.archive.org/web/20130510062951/http://blog.yhathq.com/posts/logistic-regression-and-python.html for further information.

Response:
When I think of dummy variables I think of using them in the context of OLS regression, and I would do something like this:
```
import numpy as np
import pandas as pd
import statsmodels.api as sm

my_data = np.array([[5, 'a', 1],
                    [3, 'b', 3],
                    [1, 'b', 2],
                    [3, 'a', 1],
                    [4, 'b', 2],
                    [7, 'c', 1],
                    [7, 'c', 1]])

df = pd.DataFrame(data=my_data, columns=['y', 'dummy', 'x'])
just_dummies = pd.get_dummies(df['dummy'])

step_1 = pd.concat([df, just_dummies], axis=1)
step_1.drop(['dummy', 'c'], inplace=True, axis=1)
# to run the regression we want to get rid of the strings 'a', 'b', 'c' (obviously)
# and we want to get rid of one dummy variable to avoid the dummy variable trap
# arbitrarily chose "c", coefficients on "a" an "b" would show effect of "a" and "b"
# relative to "c"
step_1 = step_1.applymap(np.int)

result = sm.OLS(step_1['y'], sm.add_constant(step_1[['x', 'a', 'b']])).fit()
print result.summary()
```


Question:
Select certain rows by index of another DataFrame I have a DataFrame and I would select only rows that contain index value into df1.index. for Example:
```

```python
df
#Output
#A  B  C  D
#1  1  4  9  1
#2  4  5  0  2
#3  5  5  1  0
#22 1  3  9  6
#```
#and these indexes
#```
```

```python
df1.index
#Output
#Int64Index([  1,   3,   4,   5,   6,   7,  22,  28,  29,  32,], dtype='int64', length=253)
#```
#I would like this output:
#```
```

```python
df
#Output
#A  B  C  D
#1  1  4  9  1
#3  5  5  1  0
#22 1  3  9  6
#```
```

Response:
Use isin:
```
df = df[df.index.isin(df1.index)]
```
Or get all intersectioned indices and select by loc:
```
df = df.loc[df.index & df1.index]
df = df.loc[np.intersect1d(df.index, df1.index)]
df = df.loc[df.index.intersection(df1.index)]
```
```
print (df)
    A  B  C  D
1   1  4  9  1
3   5  5  1  0
22  1  3  9  6
```
EDIT: I tried solution: df = df.loc[df1.index]. Do you think that this solution is correct? Solution is incorrect:
```
df = df.loc[df1.index]
print (df)

      A    B    C    D
1   1.0  4.0  9.0  1.0
3   5.0  5.0  1.0  0.0
4   NaN  NaN  NaN  NaN
5   NaN  NaN  NaN  NaN
6   NaN  NaN  NaN  NaN
7   NaN  NaN  NaN  NaN
22  1.0  3.0  9.0  6.0
28  NaN  NaN  NaN  NaN
29  NaN  NaN  NaN  NaN
32  NaN  NaN  NaN  NaN
C:/Dropbox/work-joy/so/_t/t.py:23: FutureWarning:
Passing list-likes to .loc or [] with any missing label will raise
KeyError in the future, you can use .reindex() as an alternative.

See the documentation here:
http://pandas.pydata.org/pandas-docs/stable/indexing.html#deprecate-loc-reindex-listlike
  print (df)
```


Question:
How to translate "bytes" objects into literal strings in pandas Dataframe, Python3.x? I have a Python3.x pandas DataFrame whereby certain columns are strings which as expressed as bytes (like in Python2.x)
```
import pandas as pd
df = pd.DataFrame(...)
df
       COLUMN1         ....
0      b'abcde'        ....
1      b'dog'          ....
2      b'cat1'         ....
3      b'bird1'        ....
4      b'elephant1'    ....
```
When I access by column with df.COLUMN1, I see Name: COLUMN1, dtype: object However, if I access by element, it is a "bytes" object
```
df.COLUMN1.ix[0].dtype
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'bytes' object has no attribute 'dtype'
```
How do I convert these into "regular" strings? That is, how can I get rid of this b'' prefix?

Response:
You can use vectorised str.decode to decode byte strings into ordinary strings:
```
df['COLUMN1'].str.decode("utf-8")
```
To do this for multiple columns you can select just the str columns:
```
str_df = df.select_dtypes([np.object])
```
convert all of them:
```
str_df = str_df.stack().str.decode('utf-8').unstack()
```
You can then swap out converted cols with the original df cols:
```
for col in str_df:
    df[col] = str_df[col]
```


Question:
In Pandas how do I convert a string of date strings to datetime objects and put them in a DataFrame?
```
import pandas as pd
date_stngs = ('2008-12-20','2008-12-21','2008-12-22','2008-12-23')

a = pd.Series(range(4),index = (range(4)))

for idx, date in enumerate(date_stngs):
    a[idx]= pd.to_datetime(date)
```
This code bit produces error: TypeError:" 'int' object is not iterable" Can anyone tell me how to get this series of date time strings into a DataFrame as DateTime objects?

Response:
```

```python
import pandas as pd
```

```python
date_stngs = ('2008-12-20','2008-12-21','2008-12-22','2008-12-23')
```

```python
a = pd.Series([pd.to_datetime(date) for date in date_stngs])
```

```python
a
#Output
#0    2008-12-20 00:00:00
#1    2008-12-21 00:00:00
#2    2008-12-22 00:00:00
#3    2008-12-23 00:00:00
#```
#UPDATE Use pandas.to_datetime(pd.Series(..)). It's concise and much faster than above code.
#```
```

```python
pd.to_datetime(pd.Series(date_stngs))
#Output
#0   2008-12-20 00:00:00
#1   2008-12-21 00:00:00
#2   2008-12-22 00:00:00
#3   2008-12-23 00:00:00
#```
```


Question:
Pandas Correlation Groupby Assuming I have a dataframe similar to the below, how would I get the correlation between 2 specific columns and then group by the 'ID' column? I believe the Pandas 'corr' method finds the correlation between all columns. If possible I would also like to know how I could find the 'groupby' correlation using the .agg function (i.e. np.correlate). What I have:
```
ID  Val1    Val2    OtherData   OtherData
A   5       4       x           x
A   4       5       x           x
A   6       6       x           x
B   4       1       x           x
B   8       2       x           x
B   7       9       x           x
C   4       8       x           x
C   5       5       x           x
C   2       1       x           x
```
What I need:
```
ID  Correlation_Val1_Val2
A   0.12
B   0.22
C   0.05
```

Response:
You pretty much figured out all the pieces, just need to combine them:
```

```python
df.groupby('ID')[['Val1','Val2']].corr()
```

Val1      Val2
ID
A  Val1  1.000000  0.500000
   Val2  0.500000  1.000000
B  Val1  1.000000  0.385727
   Val2  0.385727  1.000000
```
In your case, printing out a 2x2 for each ID is excessively verbose. I don't see an option to print a scalar correlation instead of the whole matrix, but you can do something simple like this if you only have two variables:
```

```python
df.groupby('ID')[['Val1','Val2']].corr().iloc[0::2,-1]
```

ID
A   Val1    0.500000
B   Val1    0.385727
```
For the more general case of 3+ variables For 3 or more variables, it is not straightforward to create concise output but you could do something like this:
```
groups = list('Val1', 'Val2', 'Val3', 'Val4')
df2 = pd.DataFrame()
for i in range( len(groups)-1):
    df2 = df2.append( df.groupby('ID')[groups].corr().stack()
                        .loc[:,groups[i],groups[i+1]:].reset_index() )

df2.columns = ['ID', 'v1', 'v2', 'corr']
df2.set_index(['ID','v1','v2']).sort_index()
```
Note that if we didn't have the groupby element, it would be straightforward to use an upper or lower triangle function from numpy. But since that element is present, it is not so easy to produce concise output in a more elegant manner as far as I can tell.


Question:
index of non "NaN" values in Pandas From Pandas data frame, how to get index of non "NaN" values? My data frame is
```
A    b     c
0   1    q1    1
1   2    NaN   3
2   3    q2    3
3   4    q1    NaN
4   5    q2    7
```
And I want the index of the rows in which column b is not NaN. (there can be NaN values in other column e.g. c ) non_nana_index = [0,2,3,4] Using this non "NaN" index list I want to create new data frame which column b do not have "Nan" df2=
```
A    b     c
0   1    q1    1
1   3    q2    3
2   4    q1    NaN
3   5    q2    7
```

Response:
Just filter them
```

```python
df['b'].notnull()
#Output
#0     True
#1    False
#2     True
#3     True
#4     True
#Name: b, dtype: bool
```

```python
df[df['b'].notnull()]
#Output
#A   b   c
#0  1  q1   1
#2  3  q2   3
#3  4  q1 NaN
#4  5  q2   7
#```
```


Question:
Python pandas apply function if a column value is not NULL I have a dataframe (in Python 2.7, pandas 0.15.0):
```
df=
       A    B               C
0    NaN   11             NaN
1    two  NaN  ['foo', 'bar']
2  three   33             NaN
```
I want to apply a simple function for rows that does not contain NULL values in a specific column. My function is as simple as possible:
```
def my_func(row):
    print row
```
And my apply code is the following:
```
df[['A','B']].apply(lambda x: my_func(x) if(pd.notnull(x[0])) else x, axis = 1)
```
It works perfectly. If I want to check column 'B' for NULL values the pd.notnull() works perfectly as well. But if I select column 'C' that contains list objects:
```
df[['A','C']].apply(lambda x: my_func(x) if(pd.notnull(x[1])) else x, axis = 1)
```
then I get the following error message: ValueError: ('The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()', u'occurred at index 1') Does anybody know why pd.notnull() works only for integer and string columns but not for 'list columns'? And is there a nicer way to check for NULL values in column 'C' instead of this:
```
df[['A','C']].apply(lambda x: my_func(x) if(str(x[1]) != 'nan') else x, axis = 1)
```
Thank you!

Response:
The problem is that pd.notnull(['foo', 'bar']) operates elementwise and returns array([ True, True], dtype=bool). Your if condition trys to convert that to a boolean, and that's when you get the exception. To fix it, you could simply wrap the isnull statement with np.all:
```
df[['A','C']].apply(lambda x: my_func(x) if(np.all(pd.notnull(x[1]))) else x, axis = 1)
```
Now you'll see that np.all(pd.notnull(['foo', 'bar'])) is indeed True.


Question:
Converting a geopandas geodataframe into a pandas dataframe What is the most efficient way to convert a geopandas geodataframe into a pandas dataframe? Below is the method I use, is there another method which is more efficient or better in general at not generating errors?
```
import geopandas as gpd
import pandas as pd

# assuming I have a shapefile named shp1.shp
gdf1 = gpd.read_file('shp1.shp')

# then for the conversion, I drop the last column (geometry) and specify the column names for the new df
df1 = pd.DataFrame(gdf1.iloc[:,:-1].values, columns = list(gdf1.columns.values)[:-1] )
```

Response:
You don't need to convert the GeoDataFrame to an array of values, you can pass it directly to the DataFrame constructor:
```
df1 = pd.DataFrame(gdf)
```
The above will keep the 'geometry' column, which is no problem for having it as a normal DataFrame. But if you actually want to drop that column, you can do (assuming the column is called 'geometry'):
```
df1 = pd.DataFrame(gdf.drop(columns='geometry'))
# for older versions of pandas (< 0.21), the drop part: gdf.drop('geometry', axis=1)
```
Two notes: It is often not needed to convert a GeoDataFrame to a normal DataFrame, because most methods that you know from a DataFrame will just work as well. Of course, there are a few cases where it is indeed needed (e.g. to plot the data without the geometries), and then the above method is the best way. The first way (df1 = pd.DataFrame(gdf)) will not take a copy of the data in the GeoDataFrame. This will often be good from an efficiency point of view, but depending on what you want to do with the DataFrame, you might want an actual copy: df1 = pd.DataFrame(gdf, copy=True)


Question:
How to do group by on a multiindex in pandas? Below is my dataframe. I made some transformations to create the category column and dropped the original column it was derived from. Now I need to do a group-by to remove the dups e.g. Love and Fashion can be rolled up via a groupby sum.
```
df.colunms = array([category, clicks, revenue, date, impressions, size], dtype=object)
df.values=
[[Love 0 0.36823 2013-11-04 380 300x250]
 [Love 183 474.81522 2013-11-04 374242 300x250]
 [Fashion 0 0.19434 2013-11-04 197 300x250]
 [Fashion 9 18.26422 2013-11-04 13363 300x250]]
```
Here is the index that is created when I created the dataframe
```
print df.index
array([ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16,
       17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33,
       34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48])
```
I assume I want to drop the index, and create date, and category as a multiindex then do a groupby sum of the metrics. How do I do this in pandas dataframe?
```
df.head(15).to_dict()= {'category': {0: 'Love', 1: 'Love', 2: 'Fashion', 3: 'Fashion', 4: 'Hair', 5: 'Movies', 6: 'Movies', 7: 'Health', 8: 'Health', 9: 'Celebs', 10: 'Celebs', 11: 'Travel', 12: 'Weightloss', 13: 'Diet', 14: 'Bags'}, 'impressions': {0: 380, 1: 374242, 2: 197, 3: 13363, 4: 4, 5: 189, 6: 60632, 7: 269, 8: 40189, 9: 138, 10: 66590, 11: 2227, 12: 22668, 13: 21707, 14: 229}, 'date': {0: '2013-11-04', 1: '2013-11-04', 2: '2013-11-04', 3: '2013-11-04', 4: '2013-11-04', 5: '2013-11-04', 6: '2013-11-04', 7: '2013-11-04', 8: '2013-11-04', 9: '2013-11-04', 10: '2013-11-04', 11: '2013-11-04', 12: '2013-11-04', 13: '2013-11-04', 14: '2013-11-04'}, 'cpc_cpm_revenue': {0: 0.36823, 1: 474.81522000000001, 2: 0.19434000000000001, 3: 18.264220000000002, 4: 0.00080000000000000004, 5: 0.23613000000000001, 6: 81.391139999999993, 7: 0.27171000000000001, 8: 51.258200000000002, 9: 0.11536, 10: 83.966859999999997, 11: 3.43248, 12: 31.695889999999999, 13: 28.459320000000002, 14: 0.43524000000000002}, 'clicks': {0: 0, 1: 183, 2: 0, 3: 9, 4: 0, 5: 1, 6: 20, 7: 0, 8: 21, 9: 0, 10: 32, 11: 1, 12: 12, 13: 9, 14: 2}, 'size': {0: '300x250', 1: '300x250', 2: '300x250', 3: '300x250', 4: '300x250', 5: '300x250', 6: '300x250', 7: '300x250', 8: '300x250', 9: '300x250', 10: '300x250', 11: '300x250', 12: '300x250', 13: '300x250', 14: '300x250'}}
```
Python is 2.7 and pandas is 0.7.0 on ubuntu 12.04. Below is the error I get if I run the below
```
import pandas
print pandas.__version__
df = pandas.DataFrame.from_dict(
    {
     'category': {0: 'Love', 1: 'Love', 2: 'Fashion', 3: 'Fashion', 4: 'Hair', 5: 'Movies', 6: 'Movies', 7: 'Health', 8: 'Health', 9: 'Celebs', 10: 'Celebs', 11: 'Travel', 12: 'Weightloss', 13: 'Diet', 14: 'Bags'},
     'impressions': {0: 380, 1: 374242, 2: 197, 3: 13363, 4: 4, 5: 189, 6: 60632, 7: 269, 8: 40189, 9: 138, 10: 66590, 11: 2227, 12: 22668, 13: 21707, 14: 229},
     'date': {0: '2013-11-04', 1: '2013-11-04', 2: '2013-11-04', 3: '2013-11-04', 4: '2013-11-04', 5: '2013-11-04', 6: '2013-11-04', 7: '2013-11-04', 8: '2013-11-04', 9: '2013-11-04', 10: '2013-11-04', 11: '2013-11-04', 12: '2013-11-04', 13: '2013-11-04', 14: '2013-11-04'}, 'cpc_cpm_revenue': {0: 0.36823, 1: 474.81522000000001, 2: 0.19434000000000001, 3: 18.264220000000002, 4: 0.00080000000000000004, 5: 0.23613000000000001, 6: 81.391139999999993, 7: 0.27171000000000001, 8: 51.258200000000002, 9: 0.11536, 10: 83.966859999999997, 11: 3.43248, 12: 31.695889999999999, 13: 28.459320000000002, 14: 0.43524000000000002}, 'clicks': {0: 0, 1: 183, 2: 0, 3: 9, 4: 0, 5: 1, 6: 20, 7: 0, 8: 21, 9: 0, 10: 32, 11: 1, 12: 12, 13: 9, 14: 2}, 'size': {0: '300x250', 1: '300x250', 2: '300x250', 3: '300x250', 4: '300x250', 5: '300x250', 6: '300x250', 7: '300x250', 8: '300x250', 9: '300x250', 10: '300x250', 11: '300x250', 12: '300x250', 13: '300x250', 14: '300x250'}
    }
)
df.set_index(['date', 'category'], inplace=True)
df.groupby(level=[0,1]).sum()

Traceback (most recent call last):
  File "/home/ubuntu/workspace/devops/reports/groupby_sub.py", line 9, in <module>
    df.set_index(['date', 'category'], inplace=True)
  File "/usr/lib/pymodules/python2.7/pandas/core/frame.py", line 1927, in set_index
    raise Exception('Index has duplicate keys: %s' % duplicates)
Exception: Index has duplicate keys: [('2013-11-04', 'Celebs'), ('2013-11-04', 'Fashion'), ('2013-11-04', 'Health'), ('2013-11-04', 'Love'), ('2013-11-04', 'Movies')]
```

Response:
You can create the index on the existing dataframe. With the subset of data provided, this works for me:
```
import pandas
df = pandas.DataFrame.from_dict(
    {
     'category': {0: 'Love', 1: 'Love', 2: 'Fashion', 3: 'Fashion', 4: 'Hair', 5: 'Movies', 6: 'Movies', 7: 'Health', 8: 'Health', 9: 'Celebs', 10: 'Celebs', 11: 'Travel', 12: 'Weightloss', 13: 'Diet', 14: 'Bags'},
     'impressions': {0: 380, 1: 374242, 2: 197, 3: 13363, 4: 4, 5: 189, 6: 60632, 7: 269, 8: 40189, 9: 138, 10: 66590, 11: 2227, 12: 22668, 13: 21707, 14: 229},
     'date': {0: '2013-11-04', 1: '2013-11-04', 2: '2013-11-04', 3: '2013-11-04', 4: '2013-11-04', 5: '2013-11-04', 6: '2013-11-04', 7: '2013-11-04', 8: '2013-11-04', 9: '2013-11-04', 10: '2013-11-04', 11: '2013-11-04', 12: '2013-11-04', 13: '2013-11-04', 14: '2013-11-04'}, 'cpc_cpm_revenue': {0: 0.36823, 1: 474.81522000000001, 2: 0.19434000000000001, 3: 18.264220000000002, 4: 0.00080000000000000004, 5: 0.23613000000000001, 6: 81.391139999999993, 7: 0.27171000000000001, 8: 51.258200000000002, 9: 0.11536, 10: 83.966859999999997, 11: 3.43248, 12: 31.695889999999999, 13: 28.459320000000002, 14: 0.43524000000000002}, 'clicks': {0: 0, 1: 183, 2: 0, 3: 9, 4: 0, 5: 1, 6: 20, 7: 0, 8: 21, 9: 0, 10: 32, 11: 1, 12: 12, 13: 9, 14: 2}, 'size': {0: '300x250', 1: '300x250', 2: '300x250', 3: '300x250', 4: '300x250', 5: '300x250', 6: '300x250', 7: '300x250', 8: '300x250', 9: '300x250', 10: '300x250', 11: '300x250', 12: '300x250', 13: '300x250', 14: '300x250'}
    }
)
df.set_index(['date', 'category'], inplace=True)
df.groupby(level=[0,1]).sum()
```
If you're having duplicate index issues with the full dataset, you'll need to clean up the data a bit. Remove the duplicate rows if that's amenable. If the duplicate rows are valid, then what sets them apart from each other? If you can add that to the dataframe and include it in the index, that's ideal. If not, just create a dummy column that defaults to 1, but can be 2 or 3 or ... N in the case of N duplicates -- and then include that field in the index as well. Alternatively, I'm pretty sure you can skip the index creation and directly groupby with columns: df.groupby(by=['date', 'category']).sum() Again, that works on the subset of data that you posted.


Question:
Count number of elements in each column less than x I have a DataFrame which looks like below. I am trying to count the number of elements less than 2.0 in each column, then I will visualize the result in a bar plot. I did it using lists and loops, but I wonder if there is a "Pandas way" to do this quickly.
```py
x = []
for i in range(6):
    x.append(df[df.ix[:,i]<2.0].count()[i])
```
then I can get a bar plot using list x.
```none
A          B          C          D          E          F
0       2.142      1.929      1.674      1.547      3.395      2.382
1       2.077      1.871      1.614      1.491      3.110      2.288
2       2.098      1.889      1.610      1.487      3.020      2.262
3       1.990      1.760      1.479      1.366      2.496      2.128
4       1.935      1.765      1.656      1.530      2.786      2.433
```

Response:
```

```python
df = pd.DataFrame({'a':randn(10), 'b':randn(10), 'c':randn(10)})
df
#Output
#a         b         c
#0 -0.849903  0.944912  1.285790
#1 -1.038706  1.445381  0.251002
#2  0.683135 -0.539052 -0.622439
#3 -1.224699 -0.358541  1.361618
#4 -0.087021  0.041524  0.151286
#5 -0.114031 -0.201018 -0.030050
#6  0.001891  1.601687 -0.040442
#7  0.024954 -1.839793  0.917328
#8 -1.480281  0.079342 -0.405370
#9  0.167295 -1.723555 -0.033937
#
#[10 rows x 3 columns]
```

```python
df[df > 1.0].count()
#Output
#a    0
#b    2
#c    2
#dtype: int64
#```
#So in your case:
#```
#df[df < 2.0 ].count()
#```
#should work EDIT some timings
#```
```

```python
%timeit df[df < 1.0 ].count()
%timeit (df < 1.0).sum()
%timeit (df < 1.0).apply(np.count_nonzero)
1000 loops, best of 3: 1.47 ms per loop
1000 loops, best of 3: 560 us per loop
1000 loops, best of 3: 529 us per loop
```
So @DSM's suggestions are correct and much faster than my suggestion
```


Question:
quickest way to swap index with values consider the pd.Series s
```
s = pd.Series(list('abcdefghij'), list('ABCDEFGHIJ'))
s

A    a
B    b
C    c
D    d
E    e
F    f
G    g
H    h
I    i
J    j
dtype: object
```
What is the quickest way to swap index and values and get the following
```
a    A
b    B
c    C
d    D
e    E
f    F
g    G
h    H
i    I
j    J
dtype: object
```

Response:
One posible solution is swap keys and values by:
```
s1 = pd.Series(dict((v,k) for k,v in s.iteritems()))
print (s1)
a    A
b    B
c    C
d    D
e    E
f    F
g    G
h    H
i    I
j    J
dtype: object
```
Another the fastest:
```
print (pd.Series(s.index.values, index=s ))
a    A
b    B
c    C
d    D
e    E
f    F
g    G
h    H
i    I
j    J
dtype: object
```
Timings:
```

```python
%timeit pd.Series(dict((v,k) for k,v in s.iteritems()))
The slowest run took 6.55 times longer than the fastest. This could mean that an intermediate result is being cached.
10000 loops, best of 3: 146 µs per loop
```

```python
%timeit (pd.Series(s.index.values, index=s ))
The slowest run took 7.42 times longer than the fastest. This could mean that an intermediate result is being cached.
10000 loops, best of 3: 102 µs per loop
```
If length of Series is 1M:
```
s = pd.Series(list('abcdefghij'), list('ABCDEFGHIJ'))
s = pd.concat([s]*1000000).reset_index(drop=True)
print (s)
```

```python
%timeit (pd.Series(s.index, index=s ))
10000 loops, best of 3: 106 µs per loop
```

```python
%timeit pd.Series(dict((v,k) for k,v in s.iteritems()))
1 loop, best of 3: 1.77 s per loop
```

```python
%timeit (pd.Series(s.index, index=s ))
10 loops, best of 3: 130 ms per loop
```

```python
%timeit (pd.Series(s.index.values, index=s ))
10 loops, best of 3: 26.5 ms per loop
```
```


Question:
Pandas add column with value based on condition based on other columns I have the following pandas dataframe:
```py
import pandas as pd
import numpy as np

d = {'age' : [21, 45, 45, 5],
     'salary' : [20, 40, 10, 100]}

df = pd.DataFrame(d)
```
and would like to add an extra column called "is_rich" which captures if a person is rich depending on his/her salary. I found multiple ways to accomplish this:
```py
# method 1
df['is_rich_method1'] = np.where(df['salary']>=50, 'yes', 'no')

# method 2
df['is_rich_method2'] = ['yes' if x >= 50 else 'no' for x in df['salary']]

# method 3
df['is_rich_method3'] = 'no'
df.loc[df['salary'] > 50,'is_rich_method3'] = 'yes'
```
resulting in: However I don't understand what the preferred way is. Are all methods equally good depending on your application?

Response:
Use the timeits, Luke! Conclusion List comprehensions perform the best on smaller amounts of data because they incur very little overhead, even though they are not vectorized. OTOH, on larger data, loc and numpy.where perform better - vectorisation wins the day. Keep in mind that the applicability of a method depends on your data, the number of conditions, and the data type of your columns. My suggestion is to test various methods on your data before settling on an option. One sure take away from here, however, is that list comprehensions are pretty competitive—they're implemented in C and are highly optimised for performance. Benchmarking code, for reference. Here are the functions being timed:
```
def numpy_where(df):
  return df.assign(is_rich=np.where(df['salary'] >= 50, 'yes', 'no'))

def list_comp(df):
  return df.assign(is_rich=['yes' if x >= 50 else 'no' for x in df['salary']])

def loc(df):
  df = df.assign(is_rich='no')
  df.loc[df['salary'] > 50, 'is_rich'] = 'yes'
  return df
```