Question:
Pandas DataFrame: SettingWithCopyWarning: A value is trying to be set on a copy of a slice from a DataFrame [duplicate] This question already has answers here: How to deal with SettingWithCopyWarning in Pandas (27 answers) Closed 5 years ago. I know there are tons of posts about this warning, but I couldn't find a solution to my situation. Here's my code:
```
df.loc[:, 'my_col'] = df.loc[:, 'my_col'].astype(int)
#df.loc[:, 'my_col'] = df.loc[:, 'my_col'].astype(int).copy()
#df.loc[:, 'my_col'] = df['my_col'].astype(int)
```
It produces the warning: SettingWithCopyWarning: A value is trying to be set on a copy of a slice from a DataFrame. Try using .loc[row_indexer,col_indexer] = value instead Even though I changed the code as suggested, I still get this warning? All I need to do is to convert the data type of one column. **Remark: ** Originally the column is of type float having one decimal (example: 4711.0). Therefore I change it to integer (4711) and then to string ('4711') - just to remove the decimal. Appreciate your help! Update: The warning was a side effect on a filtering of the original data that was done just before. I was missing the DataFrame.copy(). Using the copy instead, solved the problem!
```
df = df[df['my_col'].notnull()].copy()
df.loc[:, 'my_col'] = df['my_col'].astype(int).astype(str)
#df['my_col'] = df['my_col'].astype(int).astype(str) # works too!
```

Response:
I think need copy and omit loc for select columns:
```
df = df[df['my_col'].notnull()].copy()
df['my_col'] = df['my_col'].astype(int).astype(str)
```
Explanation: If you modify values in df later you will find that the modifications do not propagate back to the original data (df), and that Pandas does warning.


Question:
Pandas Resampling error: Only valid with DatetimeIndex, TimedeltaIndex or PeriodIndex When using pandas' resample function on a DataFrame in order to convert tick data to OHLCV, a resampling error is encountered. How should we solve the error?
```py
# Resample data into 30min bins
bars = data.Price.resample('30min', how='ohlc')
volumes = data.Volume.resample('30min', how='sum')
```
This gives the error:
```none
TypeError: Only valid with DatetimeIndex, TimedeltaIndex or PeriodIndex, but got an instance of 'Int64Index'
```

Response:
Convert the integer timestamps in the index to a DatetimeIndex:
```
data.index = pd.to_datetime(data.index, unit='s')
```
This interprets the integers as seconds since the Epoch. For example, given
```
data = pd.DataFrame(
    {'Timestamp':[1313331280, 1313334917, 1313334917, 1313340309, 1313340309],
     'Price': [10.4]*3 + [10.5]*2, 'Volume': [0.779, 0.101, 0.316, 0.150, 1.8]})
data = data.set_index(['Timestamp'])
#             Price  Volume
# Timestamp
# 1313331280   10.4   0.779
# 1313334917   10.4   0.101
# 1313334917   10.4   0.316
# 1313340309   10.5   0.150
# 1313340309   10.5   1.800

data.index = pd.to_datetime(data.index, unit='s')
```
yields
```
Price  Volume
2011-08-14 14:14:40   10.4   0.779
2011-08-14 15:15:17   10.4   0.101
2011-08-14 15:15:17   10.4   0.316
2011-08-14 16:45:09   10.5   0.150
2011-08-14 16:45:09   10.5   1.800
```
Then
```
ticks = data.ix[:, ['Price', 'Volume']]
bars = ticks.Price.resample('30min').ohlc()
volumes = ticks.Volume.resample('30min').sum()
```
can be computed:
```

```python
bars
#Output
#open  high   low  close
#2011-08-14 14:00:00  10.4  10.4  10.4   10.4
#2011-08-14 14:30:00   NaN   NaN   NaN    NaN
#2011-08-14 15:00:00  10.4  10.4  10.4   10.4
#2011-08-14 15:30:00   NaN   NaN   NaN    NaN
#2011-08-14 16:00:00   NaN   NaN   NaN    NaN
#2011-08-14 16:30:00  10.5  10.5  10.5   10.5
```

```python
volumes
#Output
#2011-08-14 14:00:00    0.779
#2011-08-14 14:30:00      NaN
#2011-08-14 15:00:00    0.417
#2011-08-14 15:30:00      NaN
#2011-08-14 16:00:00      NaN
#2011-08-14 16:30:00    1.950
#Freq: 30T, Name: Volume, dtype: float64
#```
```


Question: Check if rows in one dataframe exist in another dataframe I have a data frame A dfA like this: And another data frame B dfB which looks like this: I want to add a column 'Exist' to dfA so that if User and Movie both exist in dfB then 'Exist' is True, otherwise it is False. So dfA should become like this:

Response:
You can use merge with parameter indicator, then remove column Rating and use numpy.where:
```
df = pd.merge(df1, df2, on=['User','Movie'], how='left', indicator='Exist')
df.drop('Rating', inplace=True, axis=1)
df['Exist'] = np.where(df.Exist == 'both', True, False)
print (df)
   User  Movie  Exist
0     1    333  False
1     1   1193   True
2     1      3  False
3     2    433  False
4     3     54   True
5     3    343  False
6     3     76   True
```


Question: How does one insert statistical annotations (stars or p-values) This seems like a trivial question, but I've been searching for a while and can't seem to find an answer. It also seems like something that should be a standard part of these packages. Does anyone know if there is a standard way to include statistical annotation between distribution plots in seaborn? For example, between two box or swarmplots?

Response:
A brace / bracket can be plotted direct with matplotlib.pyplot.plot or matplotlib.axes.Axes.plot, and annotations can be added with matplotlib.pyplot.text or matplotlib.axes.Axes.text. seaborn categorical plots are 0 indexed, whereas box plots, by default, with matplotlib and pandas, start at range(1, N+1), which can be adjusted with the positions parameter. seaborn is a high-level API for matplotlib, and pandas.DataFrame.plot uses matplotlib as the default backend. Imports and DataFrame
```py
import seaborn as sns
import matplotlib.pyplot as plt

# dataframe in long form for seaborn
tips = sns.load_dataset("tips")

# dataframe in wide form for plotting with pandas.DataFrame.plot
df = tips.pivot(columns='day', values='total_bill')

# data as a list of lists for plotting directly with matplotlib (no nan values allowed)
data = [df[c].dropna().tolist() for c in df.columns]
```
seaborn
```py
sns.boxplot(x="day", y="total_bill", data=tips, palette="PRGn")

# statistical annotation
x1, x2 = 2, 3   # columns 'Sat' and 'Sun' (first column: 0, see plt.xticks())
y, h, col = tips['total_bill'].max() + 2, 2, 'k'

plt.plot([x1, x1, x2, x2], [y, y+h, y+h, y], lw=1.5, c=col)
plt.text((x1+x2)*.5, y+h, "ns", ha='center', va='bottom', color=col)

plt.show()
```
pandas.DataFrame.plot
```py
ax = df.plot(kind='box', positions=range(len(df.columns)))

x1, x2 = 2, 3
y, h, col = df.max().max() + 2, 2, 'k'

ax.plot([x1, x1, x2, x2], [y, y+h, y+h, y], lw=1.5, c=col)
ax.text((x1+x2)*.5, y+h, "ns", ha='center', va='bottom', color=col)
```
matplotlib
```py
plt.boxplot(data, positions=range(len(data)))

x1, x2 = 2, 3

y, h, col = max(map(max, data)) + 2, 2, 'k'

plt.plot([x1, x1, x2, x2], [y, y+h, y+h, y], lw=1.5, c=col)
plt.text((x1+x2)*.5, y+h, "ns", ha='center', va='bottom', color=col)
```
tips.head()
```none
total_bill   tip     sex smoker  day    time  size
0       16.99  1.01  Female     No  Sun  Dinner     2
1       10.34  1.66    Male     No  Sun  Dinner     3
2       21.01  3.50    Male     No  Sun  Dinner     3
3       23.68  3.31    Male     No  Sun  Dinner     2
4       24.59  3.61  Female     No  Sun  Dinner     4
```
df.head()
```none
day  Thur  Fri  Sat    Sun
0     NaN  NaN  NaN  16.99
1     NaN  NaN  NaN  10.34
2     NaN  NaN  NaN  21.01
3     NaN  NaN  NaN  23.68
4     NaN  NaN  NaN  24.59
```
data
```none
[[27.2, 22.76, 17.29, ..., 20.53, 16.47, 18.78],
 [28.97, 22.49, 5.75, ..., 13.42, 16.27, 10.09],
 [20.65, 17.92, 20.29, ..., 29.03, 27.18, 22.67, 17.82],
 [16.99, 10.34, 21.01, ..., 18.15, 23.1, 15.69]]
```


Question:
Calculate new column as the mean of other columns in pandas [duplicate] This question already has answers here: Row-wise average for a subset of columns with missing values (4 answers) Closed 7 years ago. I have a this data frame and I would like to calculate a new column as the mean of salary_1, salary_2 and salary_3:
```
df = pd.DataFrame({
    'salary_1': [230, 345, 222],
    'salary_2': [235, 375, 292],
    'salary_3': [210, 385, 260]
})
```
```none
salary_1     salary_2    salary_3
0        230           235        210
1        345           375        385
2        222           292        260
```
How can I do it in pandas in the most efficient way? Actually I have many more columns and I don't want to write this one by one. Something like this:
```none
salary_1     salary_2    salary_3     salary_mean
0        230           235        210     (230+235+210)/3
1        345           375        385       ...
2        222           292        260       ...
```

Response:
Use .mean. By specifying the axis you can take the average across the row or the column.
```
df['average'] = df.mean(axis=1)
df
```
returns
```
salary_1  salary_2  salary_3     average
0       230       235       210  225.000000
1       345       375       385  368.333333
2       222       292       260  258.000000
```
If you only want the mean of a few you can select only those columns. E.g.
```
df['average_1_3'] = df[['salary_1', 'salary_3']].mean(axis=1)
df
```
returns
```
salary_1  salary_2  salary_3  average_1_3
0       230       235       210        220.0
1       345       375       385        365.0
2       222       292       260        241.0
```


Question:
iterate over pandas dataframe using itertuples I am iterating over a pandas dataframe using itertuples. I also want to capture the row number while iterating:
```
for row in df.itertuples():
    print row['name']
```
Expected output :
```
1 larry
2 barry
3 michael
```
1, 2, 3 are row numbers. I want to avoid using a counter and getting the row number. Is there an easy way to achieve this using pandas?

Response:
When using itertuples you get a named tuple for every row. By default, you can access the index value for that row with row.Index. If the index value isn't what you were looking for then you can use enumerate
```
for i, row in enumerate(df.itertuples(), 1):
    print(i, row.name)
```
enumerate takes the place of an ugly counter construct


Question:
Creating an empty MultiIndex I would like to create an empty DataFrame with a MultiIndex before assigning rows to it. I already found that empty DataFrames don't like to be assigned MultiIndexes on the fly, so I'm setting the MultiIndex names during creation. However, I don't want to assign levels, as this will be done later. This is the best code I got to so far:
```
def empty_multiindex(names):
    """
    Creates empty MultiIndex from a list of level names.
    """
    return MultiIndex.from_tuples(tuples=[(None,) * len(names)], names=names)
```
Which gives me
```

```python
empty_multiindex(['one','two', 'three'])
#Output
#MultiIndex(levels=[[], [], []],
#           labels=[[-1, -1, -1], [-1, -1, -1], [-1, -1, -1]],
#           names=[u'one', u'two', u'three'])
#```
#and
#```
```

```python
DataFrame(index=empty_multiindex(['one','two', 'three']))
#Output
#one two three
#NaN NaN NaN
#```
#Well, I have no use for these NaNs. I can easily drop them later, but this is obviously a hackish solution. Anyone has a better one?
```

Response:
The solution is to leave out the labels. This works fine for me:
```

```python
import pandas as pd
```

```python
my_index = pd.MultiIndex(levels=[[],[],[]],
                         codes=[[],[],[]],
                         names=[u'one', u'two', u'three'])
```

```python
my_index
#Output
#MultiIndex([], names=['one', 'two', 'three'])
```

```python
my_columns = [u'alpha', u'beta']
```

```python
df = pd.DataFrame(index=my_index, columns=my_columns)
```

```python
df
#Output
#Empty DataFrame
#Columns: [alpha, beta]
#Index: []
```

```python
df.loc[('apple','banana','cherry'),:] = [0.1, 0.2]
```

```python
df
#Output
#                    alpha beta
#one   two    three
#apple banana cherry   0.1  0.2
#```
#For Pandas Version < 0.25.1: The keyword labels can be used in place of codes
```


Question:
How to remove or hide x-axis labels from a plot I have a boxplot and need to remove the x-axis ('user_type' and 'member_gender') label. How do I do this given the below format?
```
sb.boxplot(x="user_type", y="Seconds", data=df, color = default_color, ax = ax[0,0], sym='').set_title('User-Type (0=Non-Subscriber, 1=Subscriber)')
sb.boxplot(x="member_gender", y="Seconds", data=df, color = default_color, ax = ax[1,0], sym='').set_title('Gender (0=Male, 1=Female, 2=Other)')
```

Response:
After creating the boxplot, use .set(). .set(xticklabels=[]) should remove tick labels. This doesn't work if you use .set_title(), but you can use .set(title=''). Do not use sns.boxplot(...).set(xticklabels=[]) because, while this works, the object type is changed from matplotlib.axes._axes.Axes for sns.boxplot(...), to list. .set(xlabel=None) should remove the axis label. .tick_params(bottom=False) will remove the ticks. Similarly, for the y-axis: How to remove or hide y-axis ticklabels from a plot Tested in python 3.11, pandas 1.5.2, matplotlib 3.6.2, seaborn 0.12.1 From the OP: No sample data
```py
fig, ax = plt.subplots(2, 1)

g1 = sb.boxplot(x="user_type", y="Seconds", data=df, color = default_color, ax = ax[0], sym='')
g1.set(xticklabels=[])
g1.set(title='User-Type (0=Non-Subscriber, 1=Subscriber)')
g1.set(xlabel=None)

g2 = sb.boxplot(x="member_gender", y="Seconds", data=df, color = default_color, ax = ax[1], sym='')
g2.set(xticklabels=[])
g2.set(title='Gender (0=Male, 1=Female, 2=Other)')
g2.set(xlabel=None)
```
Example 1 With xticks and xlabel
```py
import seaborn as sns
import matplotlib.pyplot as plt

# load data
exercise = sns.load_dataset('exercise')
pen = sns.load_dataset('penguins')

# create figures
fig, ax = plt.subplots(2, 1, figsize=(8, 8))

# plot data
g1 = sns.boxplot(x='time', y='pulse', hue='kind', data=exercise, ax=ax[0])

g2 = sns.boxplot(x='species', y='body_mass_g', hue='sex', data=pen, ax=ax[1])

plt.show()
```
Without xticks and xlabel
```py
fig, ax = plt.subplots(2, 1, figsize=(8, 8))

g1 = sns.boxplot(x='time', y='pulse', hue='kind', data=exercise, ax=ax[0])

g1.set(xticklabels=[])  # remove the tick labels
g1.set(title='Exercise: Pulse by Time for Exercise Type')  # add a title
g1.set(xlabel=None)  # remove the axis label

g2 = sns.boxplot(x='species', y='body_mass_g', hue='sex', data=pen, ax=ax[1])

g2.set(xticklabels=[])
g2.set(title='Penguins: Body Mass by Species for Gender')
g2.set(xlabel=None)
g2.tick_params(bottom=False)  # remove the ticks

plt.show()
```
Example 2
```py
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd

# sinusoidal sample data
sample_length = range(1, 1+1) # number of columns of frequencies
rads = np.arange(0, 2*np.pi, 0.01)
data = np.array([(np.cos(t*rads)*10**67) + 3*10**67 for t in sample_length])
df = pd.DataFrame(data.T, index=pd.Series(rads.tolist(), name='radians'), columns=[f'freq: {i}x' for i in sample_length])
df.reset_index(inplace=True)

# plot
fig, ax = plt.subplots(figsize=(8, 8))
ax.plot('radians', 'freq: 1x', data=df)

# or skip the previous two lines and plot df directly
# ax = df.plot(x='radians', y='freq: 1x', figsize=(8, 8), legend=False)
```
Remove Labels
```py
# plot
fig, ax = plt.subplots(figsize=(8, 8))
ax.plot('radians', 'freq: 1x', data=df)

# or skip the previous two lines and plot df directly
# ax = df.plot(x='radians', y='freq: 1x', figsize=(8, 8), legend=False)

ax.set(xticklabels=[])  # remove the tick labels
ax.tick_params(bottom=False)  # remove the ticks
```


Question:
Get all keys from GroupBy object in Pandas I'm looking for a way to get a list of all the keys in a GroupBy object, but I can't seem to find one via the docs nor through Google. There is definitely a way to access the groups through their keys, like so:
```
df_gb = df.groupby(['EmployeeNumber'])
df_gb.get_group(key)
```
...so I figure there's a way to access a list (or the like) of the keys in a GroupBy object. I'm looking for something like this:
```
df_gb.keys
Out: [1234, 2356, 6894, 9492]
```
I figure I could just loop through the GroupBy object and get the keys that way, but I think there's got to be a better way.

Response:
You can access this via attribute .groups on the groupby object, this returns a dict, the keys of the dict gives you the groups:
```

```python
df = pd.DataFrame({'group':[0,1,1,1,2,2,3,3,3], 'val':np.arange(9)})
gp = df.groupby('group')
gp.groups.keys()
#Output
#dict_keys([0, 1, 2, 3])
#```
#here is the output from groups:
#```
```

```python
gp.groups
#Output
#{0: Int64Index([0], dtype='int64'),
# 1: Int64Index([1, 2, 3], dtype='int64'),
# 2: Int64Index([4, 5], dtype='int64'),
# 3: Int64Index([6, 7, 8], dtype='int64')}
#```
#Update it looks like that because the type of groups is a dict then the group order isn't maintained when you call keys:
#```
```

```python
df = pd.DataFrame({'group':list('bgaaabxeb'), 'val':np.arange(9)})
gp = df.groupby('group')
gp.groups.keys()
#Output
#dict_keys(['b', 'e', 'g', 'a', 'x'])
#```
#if you call groups you can see the order is maintained:
#```
```

```python
gp.groups
#Output
#{'a': Int64Index([2, 3, 4], dtype='int64'),
# 'b': Int64Index([0, 5, 8], dtype='int64'),
# 'e': Int64Index([7], dtype='int64'),
# 'g': Int64Index([1], dtype='int64'),
# 'x': Int64Index([6], dtype='int64')}
#```
#then the key order is maintained, a hack around this is to access the .name attribute of each group:
#```
```

```python
gp.apply(lambda x: x.name)
#Output
#group
#a    a
#b    b
#e    e
#g    g
#x    x
#dtype: object
#```
#which isn't great as this isn't vectorised, however if you already have an aggregated object then you can just get the index values:
#```
```

```python
agg = gp.sum()
agg
#Output
#val
#group
#a        9
#b       13
#e        7
#g        1
#x        6
```

```python
agg.index.get_level_values(0)
#Output
#Index(['a', 'b', 'e', 'g', 'x'], dtype='object', name='group')
#```
```


Question:
How to do/workaround a conditional join in python Pandas? I am trying to calculate time-based aggregations in Pandas based on date values stored in a separate tables. The top of the first table table_a looks like this:
```
COMPANY_ID  DATE            MEASURE
    1   2010-01-01 00:00:00     10
    1   2010-01-02 00:00:00     10
    1   2010-01-03 00:00:00     10
    1   2010-01-04 00:00:00     10
    1   2010-01-05 00:00:00     10
```
Here is the code to create the table:
```
table_a = pd.concat(\
    [pd.DataFrame({'DATE': pd.date_range("01/01/2010", "12/31/2010", freq="D"),\
    'COMPANY_ID': 1 , 'MEASURE': 10}),\
    pd.DataFrame({'DATE': pd.date_range("01/01/2010", "12/31/2010", freq="D"),\
    'COMPANY_ID': 2 , 'MEASURE': 10})])
```
The second table, table_b, looks like this:
```
COMPANY     END_DATE
        1   2010-03-01 00:00:00
        1   2010-06-02 00:00:00
        2   2010-03-01 00:00:00
        2   2010-06-02 00:00:00
```
and the code to create it is:
```
table_b = pd.DataFrame({'END_DATE':pd.to_datetime(['03/01/2010','06/02/2010','03/01/2010','06/02/2010']),\
                    'COMPANY':(1,1,2,2)})
```
I want to be able to get the sum of the 'measure' column for each 'COMPANY_ID' for each 30-day period prior to the 'END_DATE' in table_b. This is (I think) the SQL equivalent:
```
select
       b.COMPANY_ID,
       b.DATE
       sum(a.MEASURE) AS MEASURE_TO_END_DATE
 from table_a a, table_b b
 where a.COMPANY = b.COMPANY and
       a.DATE < b.DATE and
       a.DATE > b.DATE - 30
 group by b.COMPANY;
```

Response:
Well, I can think of a few ways: essentially blow up the dataframe by just merging on the exact field (company)... then filter on the 30-day windows after the merge. should be fast but could use lots of memory Move the merging and filtering on the 30-day window into a groupby(). results in a merge for each group, so slower but should use less memory Option #1 Suppose your data looks like the following (I expanded your sample data):
```
print df

    company       date  measure
0         0 2010-01-01       10
1         0 2010-01-15       10
2         0 2010-02-01       10
3         0 2010-02-15       10
4         0 2010-03-01       10
5         0 2010-03-15       10
6         0 2010-04-01       10
7         1 2010-03-01        5
8         1 2010-03-15        5
9         1 2010-04-01        5
10        1 2010-04-15        5
11        1 2010-05-01        5
12        1 2010-05-15        5

print windows

   company   end_date
0        0 2010-02-01
1        0 2010-03-15
2        1 2010-04-01
3        1 2010-05-15
```
Create a beginning date for the 30 day windows:
```
windows['beg_date'] = (windows['end_date'].values.astype('datetime64[D]') -
                       np.timedelta64(30,'D'))
print windows

   company   end_date   beg_date
0        0 2010-02-01 2010-01-02
1        0 2010-03-15 2010-02-13
2        1 2010-04-01 2010-03-02
3        1 2010-05-15 2010-04-15
```
Now do a merge and then select based on if date falls within beg_date and end_date:
```
df = df.merge(windows,on='company',how='left')
df = df[(df.date >= df.beg_date) & (df.date <= df.end_date)]
print df

    company       date  measure   end_date   beg_date
2         0 2010-01-15       10 2010-02-01 2010-01-02
4         0 2010-02-01       10 2010-02-01 2010-01-02
7         0 2010-02-15       10 2010-03-15 2010-02-13
9         0 2010-03-01       10 2010-03-15 2010-02-13
11        0 2010-03-15       10 2010-03-15 2010-02-13
16        1 2010-03-15        5 2010-04-01 2010-03-02
18        1 2010-04-01        5 2010-04-01 2010-03-02
21        1 2010-04-15        5 2010-05-15 2010-04-15
23        1 2010-05-01        5 2010-05-15 2010-04-15
25        1 2010-05-15        5 2010-05-15 2010-04-15
```
You can compute the 30 day window sums by grouping on company and end_date:
```
print df.groupby(['company','end_date']).sum()

                    measure
company end_date
0       2010-02-01       20
        2010-03-15       30
1       2010-04-01       10
        2010-05-15       15
```
Option #2 Move all merging into a groupby. This should be better on memory but I would think much slower:
```
windows['beg_date'] = (windows['end_date'].values.astype('datetime64[D]') -
                       np.timedelta64(30,'D'))

def cond_merge(g,windows):
    g = g.merge(windows,on='company',how='left')
    g = g[(g.date >= g.beg_date) & (g.date <= g.end_date)]
    return g.groupby('end_date')['measure'].sum()

print df.groupby('company').apply(cond_merge,windows)

company  end_date
0        2010-02-01    20
         2010-03-15    30
1        2010-04-01    10
         2010-05-15    15
```
Another option Now if your windows never overlap (like in the example data), you could do something like the following as an alternative that doesn't blow up a dataframe but is pretty fast:
```
windows['date'] = windows['end_date']

df = df.merge(windows,on=['company','date'],how='outer')
print df

    company       date  measure   end_date
0         0 2010-01-01       10        NaT
1         0 2010-01-15       10        NaT
2         0 2010-02-01       10 2010-02-01
3         0 2010-02-15       10        NaT
4         0 2010-03-01       10        NaT
5         0 2010-03-15       10 2010-03-15
6         0 2010-04-01       10        NaT
7         1 2010-03-01        5        NaT
8         1 2010-03-15        5        NaT
9         1 2010-04-01        5 2010-04-01
10        1 2010-04-15        5        NaT
11        1 2010-05-01        5        NaT
12        1 2010-05-15        5 2010-05-15
```
This merge essentially inserts your window end dates into the dataframe and then backfilling the end dates (by group) will give you a structure to easily create you summation windows:
```
df['end_date'] = df.groupby('company')['end_date'].apply(lambda x: x.bfill())

print df

    company       date  measure   end_date
0         0 2010-01-01       10 2010-02-01
1         0 2010-01-15       10 2010-02-01
2         0 2010-02-01       10 2010-02-01
3         0 2010-02-15       10 2010-03-15
4         0 2010-03-01       10 2010-03-15
5         0 2010-03-15       10 2010-03-15
6         0 2010-04-01       10        NaT
7         1 2010-03-01        5 2010-04-01
8         1 2010-03-15        5 2010-04-01
9         1 2010-04-01        5 2010-04-01
10        1 2010-04-15        5 2010-05-15
11        1 2010-05-01        5 2010-05-15
12        1 2010-05-15        5 2010-05-15

df = df[df.end_date.notnull()]
df['beg_date'] = (df['end_date'].values.astype('datetime64[D]') -
                   np.timedelta64(30,'D'))

print df

   company       date  measure   end_date   beg_date
0         0 2010-01-01       10 2010-02-01 2010-01-02
1         0 2010-01-15       10 2010-02-01 2010-01-02
2         0 2010-02-01       10 2010-02-01 2010-01-02
3         0 2010-02-15       10 2010-03-15 2010-02-13
4         0 2010-03-01       10 2010-03-15 2010-02-13
5         0 2010-03-15       10 2010-03-15 2010-02-13
7         1 2010-03-01        5 2010-04-01 2010-03-02
8         1 2010-03-15        5 2010-04-01 2010-03-02
9         1 2010-04-01        5 2010-04-01 2010-03-02
10        1 2010-04-15        5 2010-05-15 2010-04-15
11        1 2010-05-01        5 2010-05-15 2010-04-15
12        1 2010-05-15        5 2010-05-15 2010-04-15

df = df[(df.date >= df.beg_date) & (df.date <= df.end_date)]
print df.groupby(['company','end_date']).sum()

                    measure
company end_date
0       2010-02-01       20
        2010-03-15       30
1       2010-04-01       10
        2010-05-15       15
```
Another alternative is to resample your first dataframe to daily data and then compute rolling_sums with a 30 day window; and select the dates at the end that you are interested in. This could be quite memory intensive too.


Question:
Replace string/value in entire DataFrame I have a very large dataset were I want to replace strings with numbers. I would like to operate on the dataset without typing a mapping function for each key (column) in the dataset. (similar to the fillna method, but replace specific string with assosiated value). Is there anyway to do this? Here is an example of my dataset
```
data
   resp          A          B          C
0     1       poor       poor       good
1     2       good       poor       good
2     3  very good  very good  very good
3     4       bad        poor       bad
4     5   very bad   very bad   very bad
5     6       poor       good   very bad
6     7       good       good       good
7     8  very good  very good  very good
8     9       bad        bad    very bad
9    10   very bad   very bad   very bad
```
The desired result:
```
data
   resp  A  B  C
0      1  3  3  4
1     2  4  3  4
2     3  5  5  5
3     4  2  3  2
4     5  1  1  1
5     6  3  4  1
6     7  4  4  4
7     8  5  5  5
8     9  2  2  1
9    10  1  1  1
```
very bad=1, bad=2, poor=3, good=4, very good=5 //Jonas

Response:
Use replace
```

```python
df.replace(['very bad', 'bad', 'poor', 'good', 'very good'],
                     [1, 2, 3, 4, 5])
#Output
#resp  A  B  C
#   0     1  3  3  4
#   1     2  4  3  4
#   2     3  5  5  5
#   3     4  2  3  2
#   4     5  1  1  1
#   5     6  3  4  1
#   6     7  4  4  4
#   7     8  5  5  5
#   8     9  2  2  1
#   9    10  1  1  1
#```
```


Question:
How to fix IndexError: invalid index to scalar variable This code generates error:
```
IndexError: invalid index to scalar variable.
```
at the line: results.append(RMSPE(np.expm1(y_train[testcv]), [y[1] for y in y_test])) How to fix it?
```
import pandas as pd
import numpy as np
from sklearn import ensemble
from sklearn import cross_validation

def ToWeight(y):
    w = np.zeros(y.shape, dtype=float)
    ind = y != 0
    w[ind] = 1./(y[ind]**2)
    return w

def RMSPE(y, yhat):
    w = ToWeight(y)
    rmspe = np.sqrt(np.mean( w * (y - yhat)**2 ))
    return rmspe

forest = ensemble.RandomForestRegressor(n_estimators=10, min_samples_split=2, n_jobs=-1)

print ("Cross validations")
cv = cross_validation.KFold(len(train), n_folds=5)

results = []
for traincv, testcv in cv:
    y_test = np.expm1(forest.fit(X_train[traincv], y_train[traincv]).predict(X_train[testcv]))
    results.append(RMSPE(np.expm1(y_train[testcv]), [y[1] for y in y_test]))
```
testcv is:
```
[False False False ...,  True  True  True]
```

Response:
You are trying to index into a scalar (non-iterable) value:
```
[y[1] for y in y_test]
#  ^ this is the problem
```
When you call [y for y in test] you are iterating over the values already, so you get a single value in y. Your code is the same as trying to do the following:
```
y_test = [1, 2, 3]
y = y_test[0] # y = 1
print(y[0]) # this line will fail
```
I'm not sure what you're trying to get into your results array, but you need to get rid of [y[1] for y in y_test]. If you want to append each y in y_test to results, you'll need to expand your list comprehension out further to something like this:
```
[results.append(..., y) for y in y_test]
```
Or just use a for loop:
```
for y in y_test:
    results.append(..., y)
```


Question: removing newlines from messy strings in pandas dataframe cells? I've used multiple ways of splitting and stripping the strings in my pandas dataframe to remove all the '\n'characters, but for some reason it simply doesn't want to delete the characters that are attached to other words, even though I split them. I have a pandas dataframe with a column that captures text from web pages using Beautifulsoup. The text has been cleaned a bit already by beautifulsoup, but it failed in removing the newlines attached to other characters. My strings look a bit like this: "hands-on\ndevelopment of games. We will study a variety of software technologies\nrelevant to games including programming languages, scripting\nlanguages, operating systems, file systems, networks, simulation\nengines, and multi-media design systems. We will also study some of\nthe underlying scientific concepts from computer science and related\nfields including" Is there an easy python way to remove these "\n" characters?

Response:
EDIT: the correct answer to this is:
```
df = df.replace(r'\n',' ', regex=True)
```
I think you need replace:
```
df = df.replace('\n','', regex=True)
```
Or:
```
df = df.replace('\n',' ', regex=True)
```
Or:
```
df = df.replace(r'\\n',' ', regex=True)
```
Sample:
```
text = '''hands-on\ndev nologies\nrelevant scripting\nlang
'''
df = pd.DataFrame({'A':[text]})
print (df)
                                                   A
0  hands-on\ndev nologies\nrelevant scripting\nla...

df = df.replace('\n',' ', regex=True)
print (df)
                                                A
0  hands-on dev nologies relevant scripting lang
```


Question:
Pandas sum two columns, skipping NaN If I add two columns to create a third, any columns containing NaN (representing missing data in my world) cause the resulting output column to be NaN as well. Is there a way to skip NaNs without explicitly setting the values to 0 (which would lose the notion that those values are "missing")?
```

```python
frame = pd.DataFrame({'a': [1, 2, np.nan], 'b': [3, np.nan, 4]})
```

```python
frame['c'] = frame['a'] + frame['b']
```

```python
frame
#Output
#a   b   c
#0   1   3   4
#1   2 NaN NaN
#2 NaN   4 NaN
#```
#In the above, I would like column c to be [4, 2, 4]. Thanks...
```

Response:
with fillna()
```
frame['c'] = frame.fillna(0)['a'] + frame.fillna(0)['b']
```
or as suggested :
```
frame['c'] = frame.a.fillna(0) + frame.b.fillna(0)
```
giving :
```
a   b  c
0   1   3  4
1   2 NaN  2
2 NaN   4  4
```


Question:
How to extract multiple JSON objects from one file? I am very new to Json files. If I have a json file with multiple json objects such as following:
```json
{"ID":"12345","Timestamp":"20140101", "Usefulness":"Yes",
 "Code":[{"event1":"A","result":"1"},…]}
{"ID":"1A35B","Timestamp":"20140102", "Usefulness":"No",
 "Code":[{"event1":"B","result":"1"},…]}
{"ID":"AA356","Timestamp":"20140103", "Usefulness":"No",
 "Code":[{"event1":"B","result":"0"},…]}
…
```
I want to extract all "Timestamp" and "Usefulness" into a data frames:
```none
Timestamp    Usefulness
 0   20140101      Yes
 1   20140102      No
 2   20140103      No
 …
```
Does anyone know a general way to deal with such problems?

Response:
Update: I wrote a solution that does not require reading the entire file in one go. It is too big for a stackoverflow answer, but can be found here jsonstream. You can use json.JSONDecoder.raw_decode to decode arbitarily big strings of "stacked" JSON (so long as they can fit in memory). raw_decode stops once it has a valid object and returns the last position where was not part of the parsed object. It is poorly documented [1] (see footer), but you can pass this position back to raw_decode and it start parsing again from that position. Unfortunately, the Python json module doesn ot accept strings that have prefixing whitespace. So we need to search to find the first non-whitespace part of your document.
```
from json import JSONDecoder, JSONDecodeError
import re

NOT_WHITESPACE = re.compile(r'\S')

def decode_stacked(document, idx=0, decoder=JSONDecoder()):
    while True:
        match = NOT_WHITESPACE.search(document, idx)
        if not match:
            return
        idx = match.start()

        try:
            obj, idx = decoder.raw_decode(document, idx)
        except JSONDecodeError:
            # do something sensible if there's some error
            raise
        yield obj

s = """

{"a": 1}

   [
1
,
2
]

"""

for obj in decode_stacked(s):
    print(obj)
```
prints:
```
{'a': 1}
[1, 2]
```
Note About Missing Documentation The current signature of raw_decode() dates from 2009, when simplejson was ported into the standard library. The documentation for raw_decode() in simplejson mentions an optional idx argument that can be used to start parsing at an offset. Given that the signature of raw_decode() has not changed since 2009, I think it is fair to assume the API is fairly stable. Especially as decode() uses the idx argument of raw_decode() to ignore prefixing whitespace when parsing a string. And this is exactly what this answer is using the idx argument for too. The documentation of raw_decode() in simplejson is: raw_decode(s[, idx=0]) Decode a JSON document from s (a str or unicode beginning with a JSON document) starting from the index idx and return a 2-tuple of the Python representation and the index in s where the document ended. This can be used to decode a JSON document from a string that may have extraneous data at the end, or to decode a string that has a series of JSON objects. JSONDecodeError will be raised if the given JSON document is not valid.


Question:
Remove prefix (or suffix) substring from column headers in pandas I'm trying to remove the sub string _x that is located in the end of part of my df column names. Sample df code:
```
import pandas as pd

d = {'W_x': ['abcde','abcde','abcde']}
df = pd.DataFrame(data=d)

df['First_x']=[0,0,0]
df['Last_x']=[1,2,3]
df['Slice']=['abFC=0.01#%sdadf','12fdak*4%FC=-0.035faf,dd43','FC=0.5fasff']
```
output:
```
W_x  First_x Last_x                 Slice
0  abcde      0     1                   abFC=0.01
1  abcde      0     2  12fdak*4%FC=-0.035faf,dd43
2  abcde      0     3                 FC=0.5fasff
```
Desired output:
```
W  First  Last                       Slice
0  abcde      0     1                   abFC=0.01
1  abcde      0     2  12fdak*4%FC=-0.035faf,dd43
2  abcde      0     3                 FC=0.5fasff
```

Response:
python < 3.9, pandas < 1.4 Use str.strip/rstrip:
```
# df.columns = df.columns.str.strip('_x')
# Or,
df.columns = df.columns.str.rstrip('_x')  # strip suffix at the right end only.

df.columns
# Index(['W', 'First', 'Last', 'Slice'], dtype='object')
```
To avoid the issue highlighted in the comments: Beware of strip() if any column name starts or ends with either _ or x beyond the suffix. You could use str.replace,
```
df.columns = df.columns.str.replace(r'_x$', '')

df.columns
# Index(['W', 'First', 'Last', 'Slice'], dtype='object')
```
Update: python >= 3.9, pandas >= 1.4 From version 1.4, you will soon be able to use str.removeprefix/str.removesuffix. Examples:
```
s = pd.Series(["str_foo", "str_bar", "no_prefix"])
s
0    str_foo
1    str_bar
2    no_prefix
dtype: object

s.str.removeprefix("str_")
0    foo
1    bar
2    no_prefix
dtype: object
```
```
s = pd.Series(["foo_str", "bar_str", "no_suffix"])
s
0    foo_str
1    bar_str
2    no_suffix
dtype: object

s.str.removesuffix("_str")
0    foo
1    bar
2    no_suffix
dtype: object
```


Question:
Pandas: Why are double brackets needed to select column after boolean indexing For a df table like below,
```
A B C D
0  0 1 1 1
1  2 3 5 7
3  3 1 2 8
```
why are the double brackets needed for selecting specific columns after boolean indexing?
```
the [['A','C']] part of

df[df['A'] < 3][['A','C']]
```

Response: For pandas objects (Series, DataFrame), the indexing operator [] only accepts colname or list of colnames to select column(s) slicing or Boolean array to select row(s), i.e. it only refers to one dimension of the dataframe. For df[[colname(s)]], the interior brackets are for list, and the outside brackets are indexing operator, i.e. you must use double brackets if you select two or more columns. With one column name, single pair of brackets returns a Series, while double brackets return a dataframe. Also, df.ix[df['A'] < 3,['A','C']] or df.loc[df['A'] < 3,['A','C']] is better than the chained selection for avoiding returning a copy versus a view of the dataframe. Please refer pandas documentation for details


Question:
Check if value from one dataframe exists in another dataframe I have 2 dataframes.
```
Df1 = pd.DataFrame({'name': ['Marc', 'Jake', 'Sam', 'Brad']
Df2 = pd.DataFrame({'IDs': ['Jake', 'John', 'Marc', 'Tony', 'Bob']
```
I want to loop over every row in Df1['name'] and check if each name is somewhere in Df2['IDs']. The result should return 1 if the name is in there, 0 if it is not like so:
```
Marc  1
Jake  1
Sam   0
Brad  0
```
Thank you.

Response:
Use isin
```
Df1.name.isin(Df2.IDs).astype(int)

0    1
1    1
2    0
3    0
Name: name, dtype: int32
```
Show result in data frame
```
Df1.assign(InDf2=Df1.name.isin(Df2.IDs).astype(int))

   name  InDf2
0  Marc      1
1  Jake      1
2   Sam      0
3  Brad      0
```
In a Series object
```
pd.Series(Df1.name.isin(Df2.IDs).values.astype(int), Df1.name.values)

Marc    1
Jake    1
Sam     0
Brad    0
dtype: int32
```


Question:
applying regex to a pandas dataframe I'm having trouble applying a regex function a column in a python dataframe. Here is the head of my dataframe:
```
Name   Season          School   G    MP  FGA  3P  3PA    3P%
 74       Joe Dumars  1982-83   McNeese State  29   NaN  487   5    8  0.625
 84      Sam Vincent  1982-83  Michigan State  30  1066  401   5   11  0.455
 176  Gerald Wilkins  1982-83     Chattanooga  30   820  350   0    2  0.000
 177  Gerald Wilkins  1983-84     Chattanooga  23   737  297   3   10  0.300
 243    Delaney Rudd  1982-83     Wake Forest  32  1004  324  13   29  0.448
```
I thought I had a pretty good grasp of applying functions to Dataframes, so maybe my Regex skills are lacking. Here is what I put together:
```
import re

def split_it(year):
    return re.findall('(\d\d\d\d)', year)

 df['Season2'] = df['Season'].apply(split_it(x))

TypeError: expected string or buffer
```
Output would be a column called Season2 that contains the year before the hyphen. I'm sure theres an easier way to do it without regex, but more importantly, i'm trying to figure out what I did wrong Thanks for any help in advance.

Response:
When I try (a variant of) your code I get NameError: name 'x' is not defined-- which it isn't. You could use either
```
df['Season2'] = df['Season'].apply(split_it)
```
or
```
df['Season2'] = df['Season'].apply(lambda x: split_it(x))
```
but the second one is just a longer and slower way to write the first one, so there's not much point (unless you have other arguments to handle, which we don't here.) Your function will return a list, though:
```

```python
df["Season"].apply(split_it)
#Output
#74     [1982]
#84     [1982]
#176    [1982]
#177    [1983]
#243    [1982]
#Name: Season, dtype: object
#```
#although you could easily change that. FWIW, I'd use vectorized string operations and do something like
#```
```

```python
df["Season"].str[:4].astype(int)
#Output
#74     1982
#84     1982
#176    1982
#177    1983
#243    1982
#Name: Season, dtype: int64
#```
#or
#```
```

```python
df["Season"].str.split("-").str[0].astype(int)
#Output
#74     1982
#84     1982
#176    1982
#177    1983
#243    1982
#Name: Season, dtype: int64
#```
```


Question:
Dictionary of lists to dataframe I have a dictionary with each key holding a list of float values. These lists are not of same size. I'd like to convert this dictionary to a pandas dataframe so that I can perform some analysis functions on the data easily such as (min, max, average, standard deviation, more). My dictionary looks like this:
```
{
    'key1': [10, 100.1, 0.98, 1.2],
    'key2': [72.5],
    'key3': [1, 5.2, 71.2, 9, 10.11, 12.21, 65, 7]
}
```
What is the best way to get this into a dataframe so that I can utilize basic functions like sum, mean, describe, std? The examples I find (like the link above), all assume each of the keys have the same number of values in the list.

Response:
```
d={
    'key1': [10, 100.1, 0.98, 1.2],
    'key2': [72.5],
    'key3': [1, 5.2, 71.2, 9, 10.11, 12.21, 65, 7]
}

df=pd.DataFrame.from_dict(d,orient='index').transpose()
```
Then df is
```
key3    key2    key1
0   1.00    72.5    10.00
1   5.20    NaN     100.10
2   71.20   NaN     0.98
3   9.00    NaN     1.20
4   10.11   NaN     NaN
```
Note that numpy has some built in functions that can do calculations ignoring NaN values, which may be relevant here. For example, if you want to find the mean of 'key1' column, you can do it as follows:
```
import numpy as np
np.nanmean(df[['key1']])
28.07
```
Other useful functions include numpy.nanstd, numpy.nanvar, numpy.nanmedian, numpy.nansum. EDIT: Note that the functions from your basic functions link can also handle nan values. However, their estimators may be different from those of numpy. For example, they calculate the unbiased estimator of sample variance, while the numpy version calculates the "usual" estimator of sample variance.


Question:
Mapping columns from one dataframe to another to create a new column [duplicate] This question already has answers here: Pandas Merging 101 (8 answers) Closed 6 years ago. i have a dataframe
```
id  store    address
1    100        xyz
2    200        qwe
3    300        asd
4    400        zxc
5    500        bnm
```
i have another dataframe df2
```
serialNo    store_code  warehouse
    1          300         Land
    2          500         Sea
    3          100         Land
    4          200         Sea
    5          400         Land
```
I want my final dataframe to look like:
```
id  store    address  warehouse
1    100        xyz     Land
2    200        qwe     Sea
3    300        asd     Land
4    400        zxc     Land
5    500        bnm     Sea
```
i.e map from one dataframe onto another creating new column

Response:
df.merge
```
out = (df1.merge(df2, left_on='store', right_on='store_code')
          .reindex(columns=['id', 'store', 'address', 'warehouse']))
print(out)

   id  store address warehouse
0   1    100     xyz      Land
1   2    200     qwe       Sea
2   3    300     asd      Land
3   4    400     zxc      Land
4   5    500     bnm       Sea
```
pd.concat + df.sort_values
```
u = df1.sort_values('store')
v = df2.sort_values('store_code')[['warehouse']].reset_index(drop=1)
out = pd.concat([u, v], 1)

print(out)

   id  store address warehouse
0   1    100     xyz      Land
1   2    200     qwe       Sea
2   3    300     asd      Land
3   4    400     zxc      Land
4   5    500     bnm       Sea
```
The first sort call is redundant assuming your dataframe is already sorted on store, in which case you may remove it. df.replace/df.map
```
s = df1.store.replace(df2.set_index('store_code')['warehouse'])
print(s)
0    Land
1     Sea
2    Land
3    Land
4     Sea

df1['warehouse'] = s
print(df1)

   id  store address warehouse
0   1    100     xyz      Land
1   2    200     qwe       Sea
2   3    300     asd      Land
3   4    400     zxc      Land
4   5    500     bnm       Sea
```
Alternatively, create a mapping explicitly. This works if you want to use it later.
```
mapping = dict(df2[['store_code', 'warehouse']].values)
df1['warehouse'] = df1.store.map(mapping)
print(df1)

   id  store address warehouse
0   1    100     xyz      Land
1   2    200     qwe       Sea
2   3    300     asd      Land
3   4    400     zxc      Land
4   5    500     bnm       Sea
```


Question:
What is the process to create pdf reports with charts from a DB? [closed] Closed. This question is seeking recommendations for software libraries, tutorials, tools, books, or other off-site resources. It does not meet Stack Overflow guidelines. It is not currently accepting answers. We don’t allow questions seeking recommendations for software libraries, tutorials, tools, books, or other off-site resources. You can edit the question so it can be answered with facts and citations. Closed 6 months ago. The community reviewed whether to reopen this question 6 months ago and left it closed: Original close reason(s) were not resolved Improve this question I have a database generated by a survey to evaluate university professors. What I want is a python script that takes the information from that database, generates a graphing table for each user, creates graphs for each user, and then renders it in a template to export it to a pdf. What does the database look like?
```
User    Professor_evaluated  Category       Question    Answer
_________________________________________________________________
Mike    Professor Criss       respect           1         3
Mike    Professor Criss       respect           2         4
Mike    Professor Criss       wisdom            3         5
Mike    Professor Criss       wisdom            4         3
Charles Professor Criss       respect           1         3
Charles Professor Criss       respect           2         4
Charles Professor Criss       wisdom            3         5
Charles Professor Criss       wisdom            4         3
```
Each teacher has several categories assigned to be evaluated (respect, wisdom, etc.) and in turn each category has associated questions. In other words, a category has several questions. Each row of the DB is the answer to a question from a student evaluating a teacher What do I need? I need to create a script for automatically generate pdf reports that summarizes this information through charts, for example a chart with the overall score of each teacher, another chart with the score of each teacher by category, another chart with the average of each student, etc..Finally, every teacher would have a report.I want a report like this What is my question? my question is about which python packages and modules I would need to do this task. And what would be the general process of doing so. I don't need the code, because I know the answer is very general, but the knowledge of how I could do it. For example: you would first need to process the information with pandas, to create a table that summarizes the information you want to graph, then plot it, then create a template of your report with XYZ module and then export it to pdf with XYZ module.

Response:
There are a lot of options for creating a pdf in python. Some of these options are ReportLab, pydf2, pdfdocument and FPDF. The FPDF library is fairly stragihtforward to use and is what I've used in this example. FPDF Documentation can be found here. It's perhaps also good to think about what python modules you might want to use to create graphs and tables. In my example, I use matplotlib (link to docs) and I also use Pandas to create a dataframe using pandas.dataframe(). I've posted a rather lengthy but fully reproducible example below, using pandas, matplotlib and fpdf. The data are a subset of what the OP provided in the question. I loop through the dataframe in my example to create the table, but there are alternative and perhaps more efficient ways to do this.
```
import pandas as pd
import matplotlib
from pylab import title, figure, xlabel, ylabel, xticks, bar, legend, axis, savefig
from fpdf import FPDF

df = pd.DataFrame()
df['Question'] = ["Q1", "Q2", "Q3", "Q4"]
df['Charles'] = [3, 4, 5, 3]
df['Mike'] = [3, 3, 4, 4]

title("Professor Criss's Ratings by Users")
xlabel('Question Number')
ylabel('Score')

c = [2.0, 4.0, 6.0, 8.0]
m = [x - 0.5 for x in c]

xticks(c, df['Question'])

bar(m, df['Mike'], width=0.5, color="#91eb87", label="Mike")
bar(c, df['Charles'], width=0.5, color="#eb879c", label="Charles")

legend()
axis([0, 10, 0, 8])
savefig('barchart.png')

pdf = FPDF()
pdf.add_page()
pdf.set_xy(0, 0)
pdf.set_font('arial', 'B', 12)
pdf.cell(60)
pdf.cell(75, 10, "A Tabular and Graphical Report of Professor Criss's Ratings by Users Charles and Mike", 0, 2, 'C')
pdf.cell(90, 10, " ", 0, 2, 'C')
pdf.cell(-40)
pdf.cell(50, 10, 'Question', 1, 0, 'C')
pdf.cell(40, 10, 'Charles', 1, 0, 'C')
pdf.cell(40, 10, 'Mike', 1, 2, 'C')
pdf.cell(-90)
pdf.set_font('arial', '', 12)
for i in range(0, len(df)):
    pdf.cell(50, 10, '%s' % (df['Question'].iloc[i]), 1, 0, 'C')
    pdf.cell(40, 10, '%s' % (str(df.Mike.iloc[i])), 1, 0, 'C')
    pdf.cell(40, 10, '%s' % (str(df.Charles.iloc[i])), 1, 2, 'C')
    pdf.cell(-90)
pdf.cell(90, 10, " ", 0, 2, 'C')
pdf.cell(-30)
pdf.image('barchart.png', x = None, y = None, w = 0, h = 0, type = '', link = '')
pdf.output('test.pdf', 'F')
```
Expected test.pdf: Update (April 2020): I made an edit to the original answer in April 2020 to replace use of pandas.DataFrame.ix() since this is deprecated. In my example I was able to replace it's use with pandas.DataFrame.iloc and the output is the same as before.


Question:
How do I change order/grouping/level of pandas MultiIndex columns? I'm trying to reorder/swaplevel/pivot/something columns in a pandas dataframe. The columns are a MultiIndex, but I can't find the sauce to do what I want. The fastest varying column in my multiIndex is month, but I would like it to be the slowest varying column. I've got a nbviewer notebook if you would like to try it out yourself: http://nbviewer.ipython.org/gist/flamingbear/4cfac24c80fe34a67474 What I have:
```
+-------------------------------------------------------------------+
|+-----+------+------+-----+------+-----+-----+------+-----+-----+  |
||     |weight             |extent            |rank                ||
|+-----+------+------+-----+------+-----+-----+------+-----+-----+  |
||month|'1Jan'|'Feb' |'Mar'|'1Jan'|'Feb'|'Mar'|'1Jan'|'Feb'|'Mar'|  |
|+-----+------+------+-----+------+-----+-----+------+-----+-----+  |
||year |      |      |     |      |     |     |      |     |     |  |
|+-----+------+------+-----+------+-----+-----+------+-----+-----+  |
||2000 |45.1  |46.1  |25.1 |13.442|14.94|15.02|13    |17   |14   |  |
|+-----+------+------+-----+------+-----+-----+------+-----+-----+  |
||2001 |85.0  |16.0  |49.0 |13.380|14.81|15.14|12    |15   |17   |  |
|+-----+------+------+-----+------+-----+-----+------+-----+-----+  |
||2002 |90.0  |33.0  |82.0 |13.590|15.13|14.88|15    |22   |10   |  |
|+-----+------+------+-----+------+-----+-----+------+-----+-----+  |
||2003 |47.0  |34.0  |78.0 |13.640|14.83|15.27|17    |16   |22   |  |
|+-----+------+------+-----+------+-----+-----+------+-----+-----+  |
+-------------------------------------------------------------------+
```
What I want
```
+------------------------------------------------------------------+
|+-----+------+------+----+------+------+-----+------+------+----+ |
||month|1Jan              |Feb                |Mar                ||
|+-----+------+------+----+------+------+-----+------+------+----+ |
||     |weight|extent|rank|weight|extent|rank |weight|extent|rank| |
|+-----+------+------+----+------+------+-----+------+------+----+ |
||year |      |      |    |      |      |     |      |      |    | |
|+-----+------+------+----+------+------+-----+------+------+----+ |
||2000 |45.1  |13.442|13  |46.1  |14.94 |17   | 25.1 |15.02 |14  | |
|+-----+------+------+----+------+------+-----+------+------+----+ |
||2001 |85.0  |13.380|12  |16.0  |14.81 |15   | 49.0 |15.14 |17  | |
|+-----+------+------+----+------+------+-----+------+------+----+ |
||2002 |90.0  |13.590|15  |33.0  |15.13 |22   | 82.0 |14.88 |10  | |
|+-----+------+------+----+------+------+-----+------+------+----+ |
||2003 |47.0  |13.640|17  |34.0  |14.83 |16   | 78.0 |15.27 |22  | |
|+-----+------+------+-----------+------+-----+------+------+----+ |
+------------------------------------------------------------------+
```
Any help would be appreciated. I can work with my original DataFrame, but writing to a CSV with the desired ordering would be fantastic. Thanks in advance, Matt

Response:
Your columns are a MultiIndex. You need to reassign the DataFrame's columns with a new MultiIndex created from swapping levels of the existing one:
```
df.columns = df.columns.swaplevel(0, 1)
df.sort_index(axis=1, level=0, inplace=True)

```python
df
```

month   '1Jan'                 'Feb'                 'Mar'
        weight  extent  rank  weight  extent  rank  weight  extent  rank
year
2000      45.1  13.442    13    46.1   14.94    17    25.1   15.02    14
2001      85.0  13.380    12    16.0   14.81    15    49.0   15.14    17
2002      90.0  13.590    15    33.0   15.13    22    82.0   14.88    10
2003      47.0  13.640    17    34.0   14.83    16    78.0   15.27    22
```
You can then export to csv:
```
df.to_csv(filename)
```


Question:
DataFrame constructor not properly called! error I am new to Python and I am facing problem in creating the Dataframe in the format of key and value i.e.
```
data = [{'key':'\[GlobalProgramSizeInThousands\]','value':'1000'},]
```
Here is my code:
```
columnsss = ['key','value'];
query = "select * from bparst_tags where tag_type = 1 ";
result = database.cursor(db.cursors.DictCursor);
result.execute(query);
result_set = result.fetchall();
data = "[";
for row in result_set:
`row["tag_expression"]`)
    data +=  "{'value': %s , 'key': %s }," % ( `row["tag_expression"]`, `row["tag_name"]` )
data += "]" ;
df = DataFrame(data , columns=columnsss);
```
But when I pass the data in DataFrame it shows me pandas.core.common.PandasError: DataFrame constructor not properly called! while if I print the data and assign the same value to data variable then it works.

Response:
You are providing a string representation of a dict to the DataFrame constructor, and not a dict itself. So this is the reason you get that error. So if you want to use your code, you could do:
```
df = DataFrame(eval(data))
```
But better would be to not create the string in the first place, but directly putting it in a dict. Something roughly like:
```
data = []
for row in result_set:
    data.append({'value': row["tag_expression"], 'key': row["tag_name"]})
```
But probably even this is not needed, as depending on what is exactly in your result_set you could probably: provide this directly to a DataFrame: DataFrame(result_set) or use the pandas read_sql_query function to do this for you (see docs on this)


Question:
'module' object has no attribute 'DataFrame' [closed] Closed. This question needs debugging details. It is not currently accepting answers. Edit the question to include desired behavior, a specific problem or error, and the shortest code necessary to reproduce the problem. This will help others answer the question. Closed 6 years ago. Improve this question For the following code:
```
import pandas as pd
df = pd.DataFrame(np.random.rand(12,2), columns=['Apples', 'Oranges'] )
df['Categories'] = pd.Series(list('AAAABBBBCCCC'))
pd.options.display.mpl_style = 'default'
df.boxplot(by='Categories')
```
I get the error:
```
'pandas' object has no attribute 'DataFrame'.
```
Any ideas on what is happening and how to fix this problem?

Response: The code presented here doesn't show this discrepancy, but sometimes I get stuck when invoking dataframe in all lower case. Switching to camel-case (pd.DataFrame()) cleans up the problem.


Question: Pandas dataframe hide index functionality? Is it possible to hide the index when displaying pandas DataFrames, so that only the column names appear at the top of the table? This would need to work for both the html representation in ipython notebook and to_latex() function (which I'm using with nbconvert).

Response:
Starting from v. 0.17.1 it is possible to hide the index via styling, see hiding the index or colums: if df is your Data Frame just do
```py
df.style.hide(axis="index")
```
```py
# Legacy Pandas versions
df.style.hide_index()
```
Please note that styling works only in the notebook, and not within the LaTeX conversion.


Question: How to delete all rows in a dataframe? I want to delete all the rows in a dataframe. The reason I want to do this is so that I can reconstruct the dataframe with an iterative loop. I want to start with a completely empty dataframe. Alternatively, I could create an empty df from just the column / type information if that is possible

Response:
Here's another method if you have an existing DataFrame that you'd like to empty without recreating the column information:
```
df_empty = df[0:0]
```
df_empty is a DataFrame with zero rows but with the same column structure as df


Question:
Group by and find top n value_counts pandas I have a dataframe of taxi data with two columns that looks like this:
```
Neighborhood    Borough        Time
Midtown         Manhattan      X
Melrose         Bronx          Y
Grant City      Staten Island  Z
Midtown         Manhattan      A
Lincoln Square  Manhattan      B
```
Basically, each row represents a taxi pickup in that neighborhood in that borough. Now, I want to find the top 5 neighborhoods in each borough with the most number of pickups. I tried this:
```
df['Neighborhood'].groupby(df['Borough']).value_counts()
```
Which gives me something like this:
```
borough
Bronx          High  Bridge          3424
               Mott Haven            2515
               Concourse Village     1443
               Port Morris           1153
               Melrose                492
               North Riverdale        463
               Eastchester            434
               Concourse              395
               Fordham                252
               Wakefield              214
               Kingsbridge            212
               Mount Hope             200
               Parkchester            191
......

Staten Island  Castleton Corners        4
               Dongan Hills             4
               Eltingville              4
               Graniteville             4
               Great Kills              4
               Castleton                3
               Woodrow                  1
```
How do I filter it so that I get only the top 5 from each? I know there are a few questions with a similar title but they weren't helpful to my case.

Response:
I think you can use nlargest - you can change 1 to 5:
```
s = df['Neighborhood'].groupby(df['Borough']).value_counts()
print s
Borough
Bronx          Melrose            7
Manhattan      Midtown           12
               Lincoln Square     2
Staten Island  Grant City        11
dtype: int64

print s.groupby(level=[0,1]).nlargest(1)
Bronx          Bronx          Melrose        7
Manhattan      Manhattan      Midtown       12
Staten Island  Staten Island  Grant City    11
dtype: int64
```
additional columns were getting created, specified level info


Question:
Set column name for apply result over groupby This is a fairly trivial problem, but its triggering my OCD and I haven't been able to find a suitable solution for the past half hour. For background, I'm looking to calculate a value (let's call it F) for each group in a DataFrame derived from different aggregated measures of columns in the existing DataFrame. Here's a toy example of what I'm trying to do:
```
import pandas as pd
import numpy as np

df = pd.DataFrame({'A': ['X', 'Y', 'X', 'Y', 'Y', 'Y', 'Y', 'X', 'Y', 'X'],
                'B': ['N', 'N', 'N', 'M', 'N', 'M', 'M', 'N', 'M', 'N'],
                'C': [69, 83, 28, 25, 11, 31, 14, 37, 14,  0],
                'D': [ 0.3,  0.1,  0.1,  0.8,  0.8,  0. ,  0.8,  0.8,  0.1,  0.8],
                'E': [11, 11, 12, 11, 11, 12, 12, 11, 12, 12]
                })

df_grp = df.groupby(['A','B'])
df_grp.apply(lambda x: x['C'].sum() * x['D'].mean() / x['E'].max())
```
What I'd like to do is assign a name to the result of apply (or lambda). Is there anyway to do this without moving lambda to a named function or renaming the column after running the last line?

Response:
You could convert your series to a dataframe using reset_index() and provide name='yout_col_name' -- The name of the column corresponding to the Series values
```
(df_grp.apply(lambda x: x['C'].sum() * x['D'].mean() / x['E'].max())
      .reset_index(name='your_col_name'))

   A  B  your_col_name
0  X  N   5.583333
1  Y  M   2.975000
2  Y  N   3.845455
```


Question:
pandas crashes on repeated DataFrame.reset_index() Very weird bug here: I'm using pandas to merge several dataframes. As part of the merge, I have to call reset_index several times. But when I do, it crashes unexpectedly on the second or third use of reset_index. Here's minimal code to reproduce the error:
```
import pandas
A = pandas.DataFrame({
    'val' :  ['aaaaa', 'acaca', 'ddddd', 'zzzzz'],
    'extra' : range(10,14),
})
A = A.reset_index()
A = A.reset_index()
A = A.reset_index()
```
Here's the relevant part of the traceback:
```
....
    A = A.reset_index()
  File "/usr/local/lib/python2.7/dist-packages/pandas/core/frame.py", line 2393, in reset_index
    new_obj.insert(0, name, _maybe_cast(self.index.values))
  File "/usr/local/lib/python2.7/dist-packages/pandas/core/frame.py", line 1787, in insert
    self._data.insert(loc, column, value)
  File "/usr/local/lib/python2.7/dist-packages/pandas/core/internals.py", line 893, in insert
    raise Exception('cannot insert %s, already exists' % item)
Exception: cannot insert level_0, already exists
```
Any idea what's going wrong here? How do I work around it?

Response:
Inspecting frame.py, it looks like pandas tries to insert a column 'index' or 'level_0'. If either/both(??) of them are already taken, then it throws the error. Fortunately, there's a "drop" option. AFAICT, this drops an existing index with the same name and replaces it with the new, reset index. This might get you in trouble if you have a column named "index," but I think otherwise you're okay. "Fixed" code:
```
import pandas
A = pandas.DataFrame({
    'val' :  ['aaaaa', 'acaca', 'ddddd', 'zzzzz'],
    'extra' : range(10,14),
})
A = A.reset_index(drop=True)
A = A.reset_index(drop=True)
A = A.reset_index(drop=True)
```


Question: Reconstruct a categorical variable from dummies in pandas pd.get_dummies allows to convert a categorical variable into dummy variables. Besides the fact that it's trivial to reconstruct the categorical variable, is there a preferred/quick way to do it?

Response:
It's been a few years, so this may well not have been in the pandas toolkit back when this question was originally asked, but this approach seems a little easier to me. idxmax will return the index corresponding to the largest element (i.e. the one with a 1). We do axis=1 because we want the column name where the 1 occurs. EDIT: I didn't bother making it categorical instead of just a string, but you can do that the same way as @Jeff did by wrapping it with pd.Categorical (and pd.Series, if desired).
```

```python
import pandas as pd
```

```python
s = pd.Series(['a', 'b', 'a', 'c'])
```

```python
s
#Output
#0    a
#1    b
#2    a
#3    c
#dtype: object
```

```python
dummies = pd.get_dummies(s)
```

```python
dummies
#Output
#a  b  c
#0  1  0  0
#1  0  1  0
#2  1  0  0
#3  0  0  1
```

```python
s2 = dummies.idxmax(axis=1)
```

```python
s2
#Output
#0    a
#1    b
#2    a
#3    c
#dtype: object
```

```python
(s2 == s).all()
#Output
#True
#```
#EDIT in response to @piRSquared's comment: This solution does indeed assume there's one 1 per row. I think this is usually the format one has. pd.get_dummies can return rows that are all 0 if you have drop_first=True or if there are NaN values and dummy_na=False (default) (any cases I'm missing?). A row of all zeros will be treated as if it was an instance of the variable named in the first column (e.g. a in the example above). If drop_first=True, you have no way to know from the dummies dataframe alone what the name of the "first" variable was, so that operation isn't invertible unless you keep extra information around; I'd recommend leaving drop_first=False (default). Since dummy_na=False is the default, this could certainly cause problems. Please set dummy_na=True when you call pd.get_dummies if you want to use this solution to invert the "dummification" and your data contains any NaNs. Setting dummy_na=True will always add a "nan" column, even if that column is all 0s, so you probably don't want to set this unless you actually have NaNs. A nice approach might be to set dummies = pd.get_dummies(series, dummy_na=series.isnull().any()). What's also nice is that idxmax solution will correctly regenerate your NaNs (not just a string that says "nan"). It's also worth mentioning that setting drop_first=True and dummy_na=False means that NaNs become indistinguishable from an instance of the first variable, so this should be strongly discouraged if your dataset may contain any NaN values.
```


Question:
How to create a DataFrame while preserving order of the columns? How can I create a DataFrame from multiple numpy arrays, Pandas Series, or Pandas DataFrame's while preserving the order of the columns? For example, I have these two numpy arrays and I want to combine them as a Pandas DataFrame.
```
foo = np.array( [ 1, 2, 3 ] )
bar = np.array( [ 4, 5, 6 ] )
```
If I do this, the bar column would come first because dict doesn't preserve order.
```
pd.DataFrame( { 'foo': pd.Series(foo), 'bar': pd.Series(bar) } )

    bar foo
0   4   1
1   5   2
2   6   3
```
I can do this, but it gets tedious when I need to combine many variables.
```
pd.DataFrame( { 'foo': pd.Series(foo), 'bar': pd.Series(bar) }, columns = [ 'foo', 'bar' ] )
```
EDIT: Is there a way to specify the variables to be joined and to organize the column order in one operation? That is, I don't mind using multiple lines to complete the entire operation, but I'd rather not having to specify the variables to be joined multiple times (since I will be changing the code a lot and this is pretty error prone). EDIT2: One more point. If I want to add or remove one of the variables to be joined, I only want to add/remove in one place.

Response:
Original Solution: Incorrect Usage of collections.OrderedDict In my original solution, I proposed to use OrderedDict from the collections package in python's standard library.
```

```python
import numpy as np
```

```python
import pandas as pd
```

```python
from collections import OrderedDict
#Output
#>>>
```

```python
foo = np.array( [ 1, 2, 3 ] )
```

```python
bar = np.array( [ 4, 5, 6 ] )
#Output
#>>>
```

```python
pd.DataFrame( OrderedDict( { 'foo': pd.Series(foo), 'bar': pd.Series(bar) } ) )
```

foo  bar
0    1    4
1    2    5
2    3    6
```
Right Solution: Passing Key-Value Tuple Pairs for Order Preservation However, as noted, if a normal dictionary is passed to OrderedDict, the order may still not be preserved since the order is randomized when constructing the dictionary. However, a work around is to convert a list of key-value tuple pairs into an OrderedDict, as suggested from this SO post:
```

```python
import numpy as np
```

```python
import pandas as pd
```

```python
from collections import OrderedDict
#Output
#>>>
```

```python
a = np.array( [ 1, 2, 3 ] )
```

```python
b = np.array( [ 4, 5, 6 ] )
```

```python
c = np.array( [ 7, 8, 9 ] )
#Output
#>>>
```

```python
pd.DataFrame( OrderedDict( { 'a': pd.Series(a), 'b': pd.Series(b), 'c': pd.Series(c) } ) )
```

a  c  b
0  1  7  4
1  2  8  5
2  3  9  6

```python
pd.DataFrame( OrderedDict( (('a', pd.Series(a)), ('b', pd.Series(b)), ('c', pd.Series(c))) ) )
```

a  b  c
0  1  4  7
1  2  5  8
2  3  6  9
```


Question:
How do I get a summary count of missing/NaN data by column in 'pandas'? In R I can quickly see a count of missing data using the summary command, but the equivalent pandas DataFrame method, describe does not report these values. I gather I can do something like
```
len(mydata.index) - mydata.count()
```
to compute the number of missing values for each column, but I wonder if there's a better idiom (or if my approach is even right).

Response:
Both describe and info report the count of non-missing values.
```

```python
df = DataFrame(np.random.randn(10,2))
```

```python
df.iloc[3:6,0] = np.nan
```

```python
df
#Output
#0         1
#0 -0.560342  1.862640
#1 -1.237742  0.596384
#2  0.603539 -1.561594
#3       NaN  3.018954
#4       NaN -0.046759
#5       NaN  0.480158
#6  0.113200 -0.911159
#7  0.990895  0.612990
#8  0.668534 -0.701769
#9 -0.607247 -0.489427
#
#[10 rows x 2 columns]
```

```python
df.describe()
#Output
#0          1
#count  7.000000  10.000000
#mean  -0.004166   0.286042
#std    0.818586   1.363422
#min   -1.237742  -1.561594
#25%   -0.583795  -0.648684
#50%    0.113200   0.216699
#75%    0.636036   0.608839
#max    0.990895   3.018954
#
#[8 rows x 2 columns]
```

```python
df.info()
<class 'pandas.core.frame.DataFrame'>
Int64Index: 10 entries, 0 to 9
Data columns (total 2 columns):
0    7 non-null float64
1    10 non-null float64
dtypes: float64(2)
```
To get a count of missing, your soln is correct
```
```

```python
len(df.index)-df.count()
#Output
#0    3
#1    0
#dtype: int64
#```
#You could do this too
#```
```

```python
df.isnull().sum()
#Output
#0    3
#1    0
#dtype: int64
#```
```


Question:
How to do left outer join exclusion in pandas I have two dataframes, A and B, and I want to get those in A but not in B, just like the one right below the top left corner. Dataframe A has columns ['a','b' + others] and B has columns ['a','b' + others]. There are no NaN values. I tried the following: 1.
```
dfm = dfA.merge(dfB, on=['a','b'])
dfe = dfA[(~dfA['a'].isin(dfm['a']) | (~dfA['b'].isin(dfm['b'])
```
2.
```
dfm = dfA.merge(dfB, on=['a','b'])
dfe = dfA[(~dfA['a'].isin(dfm['a']) & (~dfA['b'].isin(dfm['b'])
```
3.
```
dfe = dfA[(~dfA['a'].isin(dfB['a']) | (~dfA['b'].isin(dfB['b'])
```
4.
```
dfe = dfA[(~dfA['a'].isin(dfB['a']) & (~dfA['b'].isin(dfB['b'])
```
but when I get len(dfm) and len(dfe), they don't sum up to dfA (it's off by a few numbers). I've tried doing this on dummy cases and #1 works, so maybe my dataset may have some peculiarities I am unable to reproduce. What's the right way to do this?

Response:
Check out this link
```
df = pd.merge(dfA, dfB, on=['a','b'], how="outer", indicator=True)
df = df[df['_merge'] == 'left_only']
```
One liner :
```
df = pd.merge(dfA, dfB, on=['a','b'], how="outer", indicator=True
              ).query('_merge=="left_only"')
```


Question:
Passing list-likes to .loc or [] with any missing labels is no longer supported I want to create a modified dataframe with the specified columns. I tried the following but throws the error "Passing list-likes to .loc or [] with any missing labels is no longer supported"
```
# columns to keep
filtered_columns = ['text', 'agreeCount', 'disagreeCount', 'id', 'user.firstName', 'user.lastName', 'user.gender', 'user.id']
tips_filtered = tips_df.loc[:, filtered_columns]

# display tips
tips_filtered
```
Thank you

Response: It looks like Pandas has deprecated this method of indexing. According to their docs: This behavior is deprecated and will show a warning message pointing to this section. The recommended alternative is to use .reindex() Using the new recommended method, you can filter your columns using: tips_filtered = tips_df.reindex(columns = filtered_columns). NB: To reindex rows, you would use reindex(index = ...) (More information here).


Question:
Create single row python pandas dataframe I want to create a python pandas DataFrame with a single row, to use further pandas functionality like dumping to *.csv. I have seen code like the following being used, but I only end up with the column structure, but empty data
```
import pandas as pd

df = pd.DataFrame()
df['A'] = 1
df['B'] = 1.23
df['C'] = "Hello"
df.columns = [['A','B','C']]

print df

Empty DataFrame
Columns: [A, B, C]
Index: []
```
While I know there are other ways to do it (like from a dictionary), I want to understand why this piece of code is not working for me!? Is this a version issue? (using pandas==0.19.2)

Response:
```

```python
df = pd.DataFrame(columns=list('ABC'))
```

```python
df.loc[0] = [1,1.23,'Hello']
```

```python
df
#Output
#A     B      C
#0  1  1.23  Hello
#```
#or:
#```
```

```python
df = pd.DataFrame([[1,1.23,'Hello']], columns=list('ABC'))
```

```python
df
#Output
#A     B      C
#0  1  1.23  Hello
#```
```


Question:
python Pandas DataFrame copy(deep=False) vs copy(deep=True) vs '=' Could somebody explain to me a difference between
```
df2 = df1

df2 = df1.copy()

df3 = df1.copy(deep=False)
```
I have tried all options and did as follows:
```
df1 = pd.DataFrame([1,2,3,4,5])
df2 = df1
df3 = df1.copy()
df4 = df1.copy(deep=False)
df1 = pd.DataFrame([9,9,9])
```
and returned as follows:
```
df1: [9,9,9]
df2: [1,2,3,4,5]
df3: [1,2,3,4,5]
df4: [1,2,3,4,5]
```
So, I observe no difference in the output between .copy() and .copy(deep=False). Why? I would expect one of the options '=', copy(), copy(deep=False) to return [9,9,9] What am I missing please?

Response:
If you see the object IDs of the various DataFrames you create, you can clearly see what is happening. When you write df2 = df1, you are creating a variable named df2, and binding it with an object with id 4541269200. When you write df1 = pd.DataFrame([9,9,9]), you are creating a new object with id 4541271120 and binding it to variable df1, but the object with id 4541269200 which was previously bound to df1 continues to live. If there were no variables bound to that object, it will get garbage collected by Python.
```

```python
import pandas as pd
```

```python
df1 = pd.DataFrame([1,2,3,4,5])
```

```python
id(df1)
#Output
#4541269200
```

```python
df2 = df1
```

```python
id(df2)
#Output
#4541269200  # Same id as df1
```

```python
df3 = df1.copy()
```

```python
id(df3)
#Output
#4541269584  # New object, new id.
```

```python
df4 = df1.copy(deep=False)
```

```python
id(df4)
#Output
#4541269072  # New object, new id.
```

```python
df1 = pd.DataFrame([9, 9, 9])
```

```python
id(df1)
#Output
#4541271120  # New object created and bound to name 'df1'.
```

```python
id(df2)
#Output
#4541269200  # Old object's id not impacted.
#```
#Edit: Added on 7/30/2018 Deep copying doesn't work in pandas and the devs consider putting mutable objects inside a DataFrame as an antipattern. Consider the following:
#```
```

```python
arr1 = [1, 2, 3]
```

```python
arr2 = [1, 2, 3, 4]
```

```python
df1 = pd.DataFrame([[arr1], [arr2]], columns=['A'])
```

```python
df1.applymap(id)
#Output
#A
#0  4515714832
#1  4515734952
```

```python
df2 = df1.copy(deep=True)
```

```python
df2.applymap(id)
#Output
#A
#0  4515714832
#1  4515734952
```

```python
df2.loc[0, 'A'].append(55)
```

```python
df2
#Output
#A
#0  [1, 2, 3, 55]
#1   [1, 2, 3, 4]
```

```python
df1
#Output
#A
#0  [1, 2, 3, 55]
#1   [1, 2, 3, 4]
#```
#df2, if it was a true deep copy should have had new ids for the lists contained within it. As a result, when you modify a list inside df2, it affects the list inside df1 as well, because they are the same objects.
```


Question:
Create multiple dataframes in loop I have a list, with each entry being a company name
```
companies = ['AA', 'AAPL', 'BA', ....., 'YHOO']
```
I want to create a new dataframe for each entry in the list. Something like (pseudocode)
```
for c in companies:
     c = pd.DataFrame()
```
I have searched for a way to do this but can't find it. Any ideas?

Response:
Just to underline my comment to @maxymoo's answer, it's almost invariably a bad idea ("code smell") to add names dynamically to a Python namespace. There are a number of reasons, the most salient being: Created names might easily conflict with variables already used by your logic. Since the names are dynamically created, you typically also end up using dynamic techniques to retrieve the data. This is why dicts were included in the language. The correct way to proceed is:
```
d = {}
for name in companies:
    d[name] = pd.DataFrame()
```
Nowadays you can write a single dict comprehension expression to do the same thing, but some people find it less readable:
```
d = {name: pd.DataFrame() for name in companies}
```
Once d is created the DataFrame for company x can be retrieved as d[x], so you can look up a specific company quite easily. To operate on all companies you would typically use a loop like:
```
for name, df in d.items():
    # operate on DataFrame 'df' for company 'name'
```
In Python 2 you were better writing
```
for name, df in d.iteritems():
```
because this avoids instantiating the list of (name, df) tuples that .items() creates in the older version. That's now largely of historical interest, though there will of course be Python 2 applications still extant and requiring (hopefully occasional) maintenance.


Question:
Convert numpy type to python I have a list of dicts in the following form that I generate from pandas. I want to convert it to a JSON format.
```
list_val = [{1.0: 685}, {2.0: 8}]
output = json.dumps(list_val)
```
However, json.dumps throws an error: TypeError: 685 is not JSON serializable I am guessing it's a type conversion issue from numpy to python(?). However, when I convert the values v of each dict in the array using np.int32(v) it still throws the error. Here's the full code:
```py
new = df[df[label] == label_new]
ks_dict = json.loads(content)
ks_list = ks_dict['variables']
freq_counts = []

for ks_var in ks_list:

    freq_var = dict()
    freq_var["name"] = ks_var["name"]
    ks_series = new[ks_var["name"]]
    temp_df = ks_series.value_counts().to_dict()
    freq_var["new"] = [{u: np.int32(v)} for (u, v) in temp_df.iteritems()]
    freq_counts.append(freq_var)

out = json.dumps(freq_counts)
```

Response:
It looks like you're correct:
```

```python
import numpy
```

```python
import json
```

```python
json.dumps(numpy.int32(685))
#Output
#Traceback (most recent call last):
#  File "<stdin>", line 1, in <module>
#  File "/usr/lib/python2.7/json/__init__.py", line 243, in dumps
#    return _default_encoder.encode(obj)
#  File "/usr/lib/python2.7/json/encoder.py", line 207, in encode
#    chunks = self.iterencode(o, _one_shot=True)
#  File "/usr/lib/python2.7/json/encoder.py", line 270, in iterencode
#    return _iterencode(o, 0)
#  File "/usr/lib/python2.7/json/encoder.py", line 184, in default
#    raise TypeError(repr(o) + " is not JSON serializable")
#TypeError: 685 is not JSON serializable
#```
#The unfortunate thing here is that numpy numbers' __repr__ doesn't give you any hint about what type they are. They're running around masquerading as ints when they aren't (gasp). Ultimately, it looks like json is telling you that an int isn't serializable, but really, it's telling you that this particular np.int32 (or whatever type you actually have) isn't serializable. (No real surprise there -- No np.int32 is serializable). This is also why the dict that you inevitably printed before passing it to json.dumps looks like it just has integers in it as well. The easiest workaround here is probably to write your own serializer1:
#```
#class MyEncoder(json.JSONEncoder):
#    def default(self, obj):
#        if isinstance(obj, numpy.integer):
#            return int(obj)
#        elif isinstance(obj, numpy.floating):
#            return float(obj)
#        elif isinstance(obj, numpy.ndarray):
#            return obj.tolist()
#        else:
#            return super(MyEncoder, self).default(obj)
#```
#You use it like this:
#```
#json.dumps(numpy.float32(1.2), cls=MyEncoder)
#json.dumps(numpy.arange(12), cls=MyEncoder)
#json.dumps({'a': numpy.int32(42)}, cls=MyEncoder)
#```
#etc. 1Or you could just write the default function and pass that as the defaut keyword argument to json.dumps. In this scenario, you'd replace the last line with raise TypeError, but ... meh. The class is more extensible :-)
```


Question:
Count and Sort with Pandas I have a dataframe for values form a file by which I have grouped by two columns, which return a count of the aggregation. Now I want to sort by the max count value, however I get the following error: KeyError: 'count' Looks the group by agg count column is some sort of index so not sure how to do this, I'm a beginner to Python and Panda. Here's the actual code, please let me know if you need more detail:
```
def answer_five():
    df = census_df#.set_index(['STNAME'])
    df = df[df['SUMLEV'] == 50]
    df = df[['STNAME','CTYNAME']].groupby(['STNAME']).agg(['count']).sort(['count'])
    #df.set_index(['count'])
    print(df.index)
    # get sorted count max item
    return df.head(5)
```

Response:
I think you need add reset_index, then parameter ascending=False to sort_values because sort return: FutureWarning: sort(columns=....) is deprecated, use sort_values(by=.....) .sort_values(['count'], ascending=False)
```
df = df[['STNAME','CTYNAME']].groupby(['STNAME'])['CTYNAME'] \
                             .count() \
                             .reset_index(name='count') \
                             .sort_values(['count'], ascending=False) \
                             .head(5)
```
Sample:
```
df = pd.DataFrame({'STNAME':list('abscscbcdbcsscae'),
                   'CTYNAME':[4,5,6,5,6,2,3,4,5,6,4,5,4,3,6,5]})

print (df)
    CTYNAME STNAME
0         4      a
1         5      b
2         6      s
3         5      c
4         6      s
5         2      c
6         3      b
7         4      c
8         5      d
9         6      b
10        4      c
11        5      s
12        4      s
13        3      c
14        6      a
15        5      e

df = df[['STNAME','CTYNAME']].groupby(['STNAME'])['CTYNAME'] \
                             .count() \
                             .reset_index(name='count') \
                             .sort_values(['count'], ascending=False) \
                             .head(5)

print (df)
  STNAME  count
2      c      5
5      s      4
1      b      3
0      a      2
3      d      1
```
But it seems you need Series.nlargest:
```
df = df[['STNAME','CTYNAME']].groupby(['STNAME'])['CTYNAME'].count().nlargest(5)
```
or:
```
df = df[['STNAME','CTYNAME']].groupby(['STNAME'])['CTYNAME'].size().nlargest(5)
```
The difference between size and count is: size counts NaN values, count does not. Sample:
```
df = pd.DataFrame({'STNAME':list('abscscbcdbcsscae'),
                   'CTYNAME':[4,5,6,5,6,2,3,4,5,6,4,5,4,3,6,5]})

print (df)
    CTYNAME STNAME
0         4      a
1         5      b
2         6      s
3         5      c
4         6      s
5         2      c
6         3      b
7         4      c
8         5      d
9         6      b
10        4      c
11        5      s
12        4      s
13        3      c
14        6      a
15        5      e

df = df[['STNAME','CTYNAME']].groupby(['STNAME'])['CTYNAME']
                             .size()
                             .nlargest(5)
                             .reset_index(name='top5')
print (df)
  STNAME  top5
0      c     5
1      s     4
2      b     3
3      a     2
4      d     1
```


Question:
Pandas converting String object to lower case and checking for string I have the below code
```
import pandas as pd
private = pd.read_excel("file.xlsx","Pri")
public = pd.read_excel("file.xlsx","Pub")
private["ISH"] = private.HolidayName.str.lower().contains("holiday|recess")
public["ISH"] = public.HolidayName.str.lower().contains("holiday|recess")
```
I get the following error:
```
AttributeError: 'Series' object has no attribute 'contains'
```
Is there anyway to convert the 'HolidayName' column to lower case and then check the regular expression ("Holiday|Recess")using .contains in one step?

Response:
```
private["ISH"] = private.HolidayName.str.contains("(?i)holiday|recess")
```
The (?i) in the regex pattern tells the re module to ignore case. The reason why you were getting an error is because the Series object does not have the contains method; instead the Series.str attribute has the contains method. So you could avoid the error with:
```
private["ISH"] = private.HolidayName.str.lower().str.contains("holiday|recess")
```


Question:
Keep other columns when using sum() with groupby I have a pandas dataframe below:
```
df

    name    value1    value2  otherstuff1 otherstuff2
0   Jack       1         1       1.19        2.39
1   Jack       1         2       1.19        2.39
2   Luke       0         1       1.08        1.08
3   Mark       0         1       3.45        3.45
4   Luke       1         0       1.08        1.08
```
Same name will have the same value for otherstuff1 and otherstuff2. I'm trying to group by column name and sum both columns value1 and value2. (Not sum value1 with value2!!! But sum them individually in each column.) Expecting to get result below:
```
newdf

    name    value1    value2  otherstuff1 otherstuff2
0   Jack       2         3       1.19        2.39
1   Luke       1         1       1.08        1.08
2   Mark       0         1       3.45        3.45
```
I've tried
```
newdf = df.groupby(['name'], as_index=False).sum()
```
which groups by name and sums up both value1 and value2 columns correctly, but ends up dropping columns otherstuff1 and otherstuff2.

Response:
You should specify what pandas must do with the other columns. In your case, I think you want to keep one row, regardless of its position within the group. This could be done with agg on a group. agg accepts a parameter that specifies what operation should be performed for each column.
```
df.groupby(['name'], as_index=False).agg({'value1': 'sum', 'value2': 'sum', 'otherstuff1': 'first', 'otherstuff2': 'first'})
```


Question:
Get column by number in Pandas What's the difference between:
```
Maand['P_Sanyo_Gesloten']
Out[119]:
Time
2012-08-01 00:00:11    0
2012-08-01 00:05:10    0
2012-08-01 00:10:11    0
2012-08-01 00:20:10    0
2012-08-01 00:25:10    0
2012-08-01 00:30:09    0
2012-08-01 00:40:10    0
2012-08-01 00:50:09    0
2012-08-01 01:05:10    0
2012-08-01 01:10:10    0
2012-08-01 01:15:10    0
2012-08-01 01:25:10    0
2012-08-01 01:30:10    0
2012-08-01 01:35:09    0
2012-08-01 01:40:10    0
...
2012-08-30 22:35:09    0
2012-08-30 22:45:10    0
2012-08-30 22:50:09    0
2012-08-30 22:55:10    0
2012-08-30 23:00:09    0
2012-08-30 23:05:10    0
2012-08-30 23:10:09    0
2012-08-30 23:15:10    0
2012-08-30 23:20:09    0
2012-08-30 23:25:10    0
2012-08-30 23:35:09    0
2012-08-30 23:40:10    0
2012-08-30 23:45:09    0
2012-08-30 23:50:10    0
2012-08-30 23:55:11    0
Name: P_Sanyo_Gesloten, Length: 7413, dtype: int64
```
And
```
Maand[[1]]
Out[120]:
&ltclass 'pandas.core.frame.DataFrame'&gt
DatetimeIndex: 7413 entries, 2012-08-01 00:00:11 to 2012-08-30 23:55:11
Data columns (total 1 columns):
P_Sanyo_Gesloten    7413  non-null values
dtypes: int64(1)
```
How can I get column by its index number? And not by an index string?

Response:
One is a column (aka Series), while the other is a DataFrame:
```

```python
df = pd.DataFrame([[1,2], [3,4]], columns=['a', 'b'])
```

```python
df
#Output
#a  b
#0  1  2
#1  3  4
#```
#The column 'b' (aka Series):
#```
```

```python
df['b']
#Output
#0    2
#1    4
#Name: b, dtype: int64
#```
#The subdataframe with columns (position) in [1]:
#```
```

```python
df[[1]]
#Output
#b
#0  2
#1  4
#```
#Note: it's preferable (and less ambiguous) to specify whether you're talking about the column name e.g. ['b'] or the integer location, since sometimes you can have columns named as integers:
#```
```

```python
df.iloc[:, [1]]
#Output
#b
#0  2
#1  4
```

```python
df.loc[:, ['b']]
#Output
#b
#0  2
#1  4
```

```python
df.loc[:, 'b']
#Output
#0    2
#1    4
#Name: b, dtype: int64
#```
```


Question:
Adding dummy columns to the original dataframe I have a dataframe looks like this:
```none
EXEC_FULLNAME   YEAR BECAMECEO
CO_PER_ROL
5622        Ira A. Eichner   1992  19550101
5622        Ira A. Eichner   1993  19550101
5622        Ira A. Eichner   1994  19550101
5623       David P. Storch   1994  19961009
5623       David P. Storch   1995  19961009
5623       David P. Storch   1996  19961009
```
For the YEAR column, I want to add year columns (1993, 1994..., 2009) to the original dataframe. For example, if a YEAR value for a row is 1992, then the value in the 1992 column should be 1 otherwise 0 for that row. I used a for loop, but it seems to run forever as I have a large dataset.

Response:
```

```python
df = pd.concat([df, pd.get_dummies(df['YEAR'])], axis=1); df
#Output
#JOINED_CO GENDER    EXEC_FULLNAME  GVKEY  YEAR    CONAME  BECAMECEO  \
#5622        NaN   MALE   Ira A. Eichner   1004  1992  AAR CORP   19550101
#5622        NaN   MALE   Ira A. Eichner   1004  1993  AAR CORP   19550101
#5622        NaN   MALE   Ira A. Eichner   1004  1994  AAR CORP   19550101
#5622        NaN   MALE   Ira A. Eichner   1004  1995  AAR CORP   19550101
#5622        NaN   MALE   Ira A. Eichner   1004  1996  AAR CORP   19550101
#5622        NaN   MALE   Ira A. Eichner   1004  1997  AAR CORP   19550101
#5622        NaN   MALE   Ira A. Eichner   1004  1998  AAR CORP   19550101
#5623        NaN   MALE  David P. Storch   1004  1992  AAR CORP   19961009
#5623        NaN   MALE  David P. Storch   1004  1993  AAR CORP   19961009
#5623        NaN   MALE  David P. Storch   1004  1994  AAR CORP   19961009
#5623        NaN   MALE  David P. Storch   1004  1995  AAR CORP   19961009
#5623        NaN   MALE  David P. Storch   1004  1996  AAR CORP   19961009
#
#      REJOIN   LEFTOFC    LEFTCO  RELEFT    REASON  PAGE  1992  1993  1994  \
#5622     NaN  19961001  19990531     NaN  RESIGNED    79     1     0     0
#5622     NaN  19961001  19990531     NaN  RESIGNED    79     0     1     0
#5622     NaN  19961001  19990531     NaN  RESIGNED    79     0     0     1
#5622     NaN  19961001  19990531     NaN  RESIGNED    79     0     0     0
#5622     NaN  19961001  19990531     NaN  RESIGNED    79     0     0     0
#5622     NaN  19961001  19990531     NaN  RESIGNED    79     0     0     0
#5622     NaN  19961001  19990531     NaN  RESIGNED    79     0     0     0
#5623     NaN       NaN       NaN     NaN       NaN    57     1     0     0
#5623     NaN       NaN       NaN     NaN       NaN    57     0     1     0
#5623     NaN       NaN       NaN     NaN       NaN    57     0     0     1
#5623     NaN       NaN       NaN     NaN       NaN    57     0     0     0
#5623     NaN       NaN       NaN     NaN       NaN    57     0     0     0
#
#      1995  1996  1997  1998
#5622     0     0     0     0
#5622     0     0     0     0
#5622     0     0     0     0
#5622     1     0     0     0
#5622     0     1     0     0
#5622     0     0     1     0
#5622     0     0     0     1
#5623     0     0     0     0
#5623     0     0     0     0
#5623     0     0     0     0
#5623     1     0     0     0
#5623     0     1     0     0
#```
#If you'd like to delete the YEAR column, then you could follow this up with del df['YEAR']. Or, drop the YEAR column from df before calling concat:
#```
#df = pd.concat([df.drop('YEAR', axis=1), pd.get_dummies(df['YEAR'])], axis=1)
#```
```


Question:
Alternative to .concat() of empty dataframe, now that it is being deprecated? I have two dataframes that can both be empty, and I want to concat them. Before I could just do :
```
output_df= pd.concat([df1, df2])
```
But now I run into FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation. An easy fix would be:
```
if not df1.empty and not df2.empty:
    result_df = pd.concat([df1, df2], axis=0)
elif not df1.empty:
    result_df = df1.copy()
elif not df2.empty:
    result_df = df2.copy()
else:
    result_df = pd.DataFrame()
```
But that seems pretty ugly. Does anyone have a better solution ? FYI: this appeared after pandas released v2.1.0

Response:
To be precise, concat is not deprecated (and won't be IMHO) but I can trigger this FutureWarning in 2.1.1 with the following example, while df2 being an empty DataFrame with a different dtypes than df1 :
```
df1 = pd.DataFrame({"A": [.1, .2, .3]})
df2 = pd.DataFrame(columns=["A"], dtype="object")

out = pd.concat([df1, df2])
print(out)

     A
0  0.1
1  0.2
2  0.3
```
As a solution in your case, you can try something like you did :
```
out = (df1.copy() if df2.empty else df2.copy() if df1.empty
       else pd.concat([df1, df2]) # if both DataFrames non empty
      )
```
Or maybe even this one? :
```
out = pd.concat([df1.astype(df2.dtypes), df2.astype(df1.dtypes)])
```


Question:
What is the Spark DataFrame method `toPandas` actually doing? I'm a beginner of Spark-DataFrame API. I use this code to load csv tab-separated into Spark Dataframe
```
lines = sc.textFile('tail5.csv')
parts = lines.map(lambda l : l.strip().split('\t'))
fnames = *some name list*
schemaData = StructType([StructField(fname, StringType(), True) for fname in fnames])
ddf = sqlContext.createDataFrame(parts,schemaData)
```
Suppose I create DataFrame with Spark from new files, and convert it to pandas using built-in method toPandas(), Does it store the Pandas object to local memory? Does Pandas low-level computation handled all by Spark? Does it exposed all pandas dataframe functionality?(I guess yes) Can I convert it toPandas and just be done with it, without so much touching DataFrame API?

Response: Using spark to read in a CSV file to pandas is quite a roundabout method for achieving the end goal of reading a CSV file into memory. It seems like you might be misunderstanding the use cases of the technologies in play here. Spark is for distributed computing (though it can be used locally). It's generally far too heavyweight to be used for simply reading in a CSV file. In your example, the sc.textFile method will simply give you a spark RDD that is effectively a list of text lines. This likely isn't what you want. No type inference will be performed, so if you want to sum a column of numbers in your CSV file, you won't be able to because they are still strings as far as Spark is concerned. Just use pandas.read_csv and read the whole CSV into memory. Pandas will automatically infer the type of each column. Spark doesn't do this. Now to answer your questions: Does it store the Pandas object to local memory: Yes. toPandas() will convert the Spark DataFrame into a Pandas DataFrame, which is of course in memory. Does Pandas low-level computation handled all by Spark No. Pandas runs its own computations, there's no interplay between spark and pandas, there's simply some API compatibility. Does it exposed all pandas dataframe functionality? No, if it is a Spark dataframe. For example, Series objects have an interpolate method which isn't available in PySpark Column objects. There are many many methods and functions that are in the pandas API that are not in the PySpark API. Yes, if you run toPandas() because you are literally converting it to a pandas dataframe. Can I convert it toPandas and just be done with it, without so much touching DataFrame API? Absolutely. In fact, you probably shouldn't even use Spark at all in this case. pandas.read_csv will likely handle your use case unless you're working with a huge amount of data. Try to solve your problem with simple, low-tech, easy-to-understand libraries, and only go to something more complicated as you need it. Many times, you won't need the more complex technology.