Question:
How to properly apply a lambda function into a pandas data frame column I have a pandas data frame, sample, with one of the columns called PR to which am applying a lambda function as follows:
```
sample['PR'] = sample['PR'].apply(lambda x: NaN if x < 90)
```
I then get the following syntax error message:
```
sample['PR'] = sample['PR'].apply(lambda x: NaN if x < 90)
                                                         ^
SyntaxError: invalid syntax
```
What am I doing wrong?

Response:
You need mask:
```
sample['PR'] = sample['PR'].mask(sample['PR'] < 90, np.nan)
```
Another solution with loc and boolean indexing:
```
sample.loc[sample['PR'] < 90, 'PR'] = np.nan
```
Sample:
```
import pandas as pd
import numpy as np

sample = pd.DataFrame({'PR':[10,100,40] })
print (sample)
    PR
0   10
1  100
2   40

sample['PR'] = sample['PR'].mask(sample['PR'] < 90, np.nan)
print (sample)
      PR
0    NaN
1  100.0
2    NaN
```
```
sample.loc[sample['PR'] < 90, 'PR'] = np.nan
print (sample)
      PR
0    NaN
1  100.0
2    NaN
```
EDIT: Solution with apply:
```
sample['PR'] = sample['PR'].apply(lambda x: np.nan if x < 90 else x)
```
Timings len(df)=300k:
```
sample = pd.concat([sample]*100000).reset_index(drop=True)

```python
%timeit sample['PR'].apply(lambda x: np.nan if x < 90 else x)
10 loops, best of 3: 102 ms per loop
```

```python
%timeit sample['PR'].mask(sample['PR'] < 90, np.nan)
The slowest run took 4.28 times longer than the fastest. This could mean that an intermediate result is being cached.
100 loops, best of 3: 3.71 ms per loop
```
```


Question:
reading excel sheet as multiindex dataframe through pd.read_excel() I'm struggle to read a excel sheet with pd.read_excel(). My excel table looks like this in it's raw form: I expected the dataframe to look like this:
```
bar                 baz                 foo
                one       two       one       two       one       two
                A         B         C         D         E         F
baz one         0.085930 -0.848468  0.911572 -0.705026 -1.284458 -0.602760
    two         0.385054  2.539314  0.589164  0.765126  0.210199 -0.481789
    three      -0.352475 -0.975200 -0.403591  0.975707  0.533924 -0.195430
```
is this even possible? My failed attempt:
```
xls_file = pd.read_excel(data_file, header=[0,1,2], index_col=None)
```
Link to the raw excel file:
```
https://www.dropbox.com/s/ek646ab4yb1fvdq/ipsos_excel_tables_type_2_trimed_nosig.xlsx?dl=0
```

Response:
You can add parameter index_col=[0,1] to read_excel, because index is Multindex too: EDIT: You need also change header from header=[0,1,2] to header=[0,1], and remove empty rows - row 5 and 7. You can add parameter sheetname:
```
import pandas as pd

df = pd.read_excel('test/ipsos_excel_tables_type_2_trimed_nosig.xlsx',
                   header=[0,1],
                   index_col=[0,1],
                   sheetname="0001")
print df
```
```
T  \
                                                                   Total
Q1. Do you have a social media account (such as... Unweighted base  2019
                                                   Weighted base    2019
                                                   Yes              1519
                                                   Yes                75
                                                   No                494
                                                   No                 24
                                                   Don’t know          5

                                                                   Gender  \
                                                                     Male
Q1. Do you have a social media account (such as... Unweighted base   1011
                                                   Weighted base     1000
                                                   Yes                705
                                                   Yes                 70
                                                   No                 291
                                                   No                  29
                                                   Don’t know           4

                                                                           \
                                                                   Female
Q1. Do you have a social media account (such as... Unweighted base   1008
                                                   Weighted base     1019
                                                   Yes                814
                                                   Yes                 80
                                                   No                 204
                                                   No                  20
                                                   Don’t know           1

                                                                     Age  \
                                                                   16-24
Q1. Do you have a social media account (such as... Unweighted base   321
                                                   Weighted base     323
                                                   Yes               293
                                                   Yes                91
                                                   No                 28
                                                   No                  9
                                                   Don’t know          1

                                                                          \
                                                                   25-34
Q1. Do you have a social media account (such as... Unweighted base   361
                                                   Weighted base     361
                                                   Yes               318
                                                   Yes                88
                                                   No                 41
                                                   No                 11
                                                   Don’t know          2

                                                                          \
                                                                   35-44
Q1. Do you have a social media account (such as... Unweighted base   372
                                                   Weighted base     370
                                                   Yes               289
                                                   Yes                78
                                                   No                 81
                                                   No                 22
                                                   Don’t know          -

                                                                          \
                                                                   45-54
Q1. Do you have a social media account (such as... Unweighted base   376
                                                   Weighted base     376
                                                   Yes               258
                                                   Yes                69
                                                   No                118
                                                   No                 31
                                                   Don’t know          -

                                                                          \
                                                                   55-75
Q1. Do you have a social media account (such as... Unweighted base   589
                                                   Weighted base     589
                                                   Yes               361
                                                   Yes                61
                                                   No                227
                                                   No                 38
                                                   Don’t know          2

                                                                   Social grade  \
                                                                             AB
Q1. Do you have a social media account (such as... Unweighted base          593
                                                   Weighted base            533
                                                   Yes                      416
                                                   Yes                       78
                                                   No                       116
                                                   No                        22
                                                   Don’t know                 1

                                                                         \
                                                                     C1
Q1. Do you have a social media account (such as... Unweighted base  588
                                                   Weighted base    563
                                                   Yes              417
                                                   Yes               74
                                                   No               147
                                                   No                26
                                                   Don’t know         -

                                                                         ...        \
                                                                         ...
Q1. Do you have a social media account (such as... Unweighted base       ...
                                                   Weighted base         ...
                                                   Yes                   ...
                                                   Yes                   ...
                                                   No                    ...
                                                   No                    ...
                                                   Don’t know            ...

                                                                   Region (4 code scale)  \
                                                                                East
Q1. Do you have a social media account (such as... Unweighted base                   341
                                                   Weighted base                     342
                                                   Yes                               259
                                                   Yes                                76
                                                   No                                 83
                                                   No                                 24
                                                   Don’t know                          -

                                                                            Education  \
                                                                   GCSE/O Level/NVQ12
Q1. Do you have a social media account (such as... Unweighted base                503
                                                   Weighted base                  520
                                                   Yes                            370
                                                   Yes                             71
                                                   No                             147
                                                   No                              28
                                                   Don’t know                       4

                                                                                \
                                                                   A Level or equivalent
Q1. Do you have a social media account (such as... Unweighted base                   454
                                                   Weighted base                     461
                                                   Yes                               359
                                                   Yes                                78
                                                   No                                101
                                                   No                                 22
                                                   Don’t know                          1

                                                                                \
                                                                   Degree/Masters/PhD
Q1. Do you have a social media account (such as... Unweighted base                914
                                                   Weighted base                  886
                                                   Yes                            697
                                                   Yes                             79
                                                   No                             189
                                                   No                              21
                                                   Don’t know                       -

                                                                                \
                                                                   No formal qualifications
Q1. Do you have a social media account (such as... Unweighted base                      148
                                                   Weighted base                        152
                                                   Yes                                   93
                                                   Yes                                   61
                                                   No                                    58
                                                   No                                    38
                                                   Don’t know                             1

                                                                   Employment status  \
                                                                           Full-time
Q1. Do you have a social media account (such as... Unweighted base               774
                                                   Weighted base                 763
                                                   Yes                           598
                                                   Yes                            78
                                                   No                            162
                                                   No                             21
                                                   Don’t know                      3

                                                                              \
                                                                   Part-time
Q1. Do you have a social media account (such as... Unweighted base       272
                                                   Weighted base         274
                                                   Yes                   195
                                                   Yes                    71
                                                   No                     79
                                                   No                     29
                                                   Don’t know              -

                                                                                \
                                                                   Self-Employed
Q1. Do you have a social media account (such as... Unweighted base           166
                                                   Weighted base             162
                                                   Yes                       108
                                                   Yes                        67
                                                   No                         54
                                                   No                         33
                                                   Don’t know                  -

                                                                                \
                                                                   ANY WORKING
Q1. Do you have a social media account (such as... Unweighted base        1212
                                                   Weighted base          1200
                                                   Yes                     901
                                                   Yes                      75
                                                   No                      295
                                                   No                       25
                                                   Don’t know                3

                                                                   ANY NOT WORKING
Q1. Do you have a social media account (such as... Unweighted base             625
                                                   Weighted base               645
                                                   Yes                         460
                                                   Yes                          71
                                                   No                          183
                                                   No                           28
                                                   Don’t know                    2

[7 rows x 25 columns]
```


Question: pandas DataFrame output end of csv I wonder how to add new DataFrame data onto the end of an existing csv file? The to_csv doesn't mention such functionality.

Response:
You can append using to_csv by passing a file which is open in append mode:
```
with open(file_name, 'a') as f:
    df.to_csv(f, header=False)
```
Use header=None, so as not to append the column names. In fact, pandas has a wrapper to do this in to_csv using the mode argument (see Joe's answer):
```
df.to_csv(f, mode='a', header=False)
```


Question: Matplotlib, horizontal bar chart (barh) is upside-down TL'DR, the vertical bar charts are shown in a conventional way -- things line up from left to right. However, when it is converted to horizontal bar chart (from bar to barh), everything is upside-down. I.e., for a grouped bar chart, not only the order of the grouped bar is wrong, the order of the each group is wrong as well. For e.g., the graph from http://dwheelerau.com/2014/05/28/pandas-data-analysis-new-zealanders-and-their-sheep/ If you look closely, you will find that the the bar and legend are in reverse order -- Beef shows on top in legend but on bottom in the graph. As the simplest demo, I changed kind='bar', to kind='barh', from this graph https://plot.ly/pandas/bar-charts/#pandas-grouped-bar-chart and the result looks like this: https://plot.ly/7/~xpt/ I.e., the bars in the horizontal grouped bar chart is ordered upside-down. How to fix it? EDIT: @Ajean, it is actually not only the order of the grouped bar is wrong, the order of the each group is wrong as well. The graph from Simple customization of matplotlib/pandas bar chart (labels, ticks, etc.) shows it clearly: We can see that the order is unconventional too, because people would expect the graph to be top-down, with "AAA" at the top, not the bottom. If you search for "Excel upside-down", you will find people are complaining about this in Excel all over the places. The Microsoft Excel has a fix for it, do Matplotlib/Panda/Searborn/Ploty/etc has a fix for it?

Response:
I believe the joint wrong order of groups and subgroups boils down to a single feature: that the y axis increases upwards, as in a usual plot. Try reversing the y axis of your axes as in this pandas-less example:
```py
import numpy as np
import matplotlib.pyplot as plt

x = range(5)
y = np.random.randn(5)

# plot 1: bar
plt.figure()
plt.bar(x, y)

# plot 2: barh, wrong order
plt.figure()
plt.barh(x, y)

# plot 3: barh with correct order: top-down y axis
plt.figure()
plt.barh(x, y)
plt.gca().invert_yaxis()

plt.show()
```
Specifically for pandas, pandas.DataFrame.plot and its various plotting submethods return a matplotlib axes object, so you can invert its y axis directly:
```py
ax = df.plot.barh()  # or df.plot(), or similar
ax.invert_yaxis()
```


Question:
Dropping Multiple Columns from a dataframe I know how to drop columns from a data frame using Python. But for my problem the data set is vast, the columns I want to drop are grouped together or are basically singularly spread out across the column heading axis. Is there a shorter way to slice or drop all the columns with fewer lines of code rather than to write it out like how I have done. The way I have done it here works but I would like a more summarized way. The flight_data_copy_final is the variable in which it should be stored. Here's my code:
```
from IPython.display import display

flight_data_copy_version1 = flight_data_copy.drop(flight_data_copy.ix[:,"Year": "FlightDate"].columns, axis=1)
flight_data_copy_version2 = flight_data_copy_version1.drop("TailNum", axis=1)
flight_data_copy_version3 = flight_data_copy_version2.drop("OriginStateFips", axis=1)
flight_data_copy_version4 = flight_data_copy_version3.drop("DestStateFips", axis=1)
flight_data_copy_version5 = flight_data_copy_version4.drop("Diverted", axis=1)
flight_data_copy_version6 = flight_data_copy_version5.drop("Flights", axis=1)
flight_data_copy_final = flight_data_copy.drop(flight_data_copy_version6.ix[:,"FirstDepTime":].columns, axis=1)

print (display (flight_data_copy_final))
```

Response:
To delete multiple columns at the same time in pandas, you could specify the column names as shown below. The option inplace=True is needed if one wants the change affected column in the same dataframe. Otherwise remove it.
```
flight_data_copy.drop(['TailNum', 'OriginStateFips',
                'DestStateFips', 'Diverted'], axis=1, inplace=True)
```
Source: Python Pandas - Deleting multiple series from a data frame in one command


Question:
binning a dataframe in pandas in Python [duplicate] This question already has answers here: Binning a column with pandas (4 answers) Closed 5 years ago. Given the following dataframe in pandas:
```
import numpy as np
df = pandas.DataFrame({"a": np.random.random(100), "b": np.random.random(100), "id": np.arange(100)})
```
where id is an id for each point consisting of an a and b value, how can I bin a and b into a specified set of bins (so that I can then take the median/average value of a and b in each bin)? df might have NaN values for a or b (or both) for any given row in df. Here's a better example using Joe Kington's solution with a more realistic df. The thing I'm unsure about is how to access the df.b elements for each df.a group below:
```
a = np.random.random(20)
df = pandas.DataFrame({"a": a, "b": a + 10})
# bins for df.a
bins = np.linspace(0, 1, 10)
# bin df according to a
groups = df.groupby(np.digitize(df.a,bins))
# Get the mean of a in each group
print groups.mean()
## But how to get the mean of b for each group of a?
# ...
```

Response:
There may be a more efficient way (I have a feeling pandas.crosstab would be useful here), but here's how I'd do it:
```
import numpy as np
import pandas

df = pandas.DataFrame({"a": np.random.random(100),
                       "b": np.random.random(100),
                       "id": np.arange(100)})

# Bin the data frame by "a" with 10 bins...
bins = np.linspace(df.a.min(), df.a.max(), 10)
groups = df.groupby(np.digitize(df.a, bins))

# Get the mean of each bin:
print groups.mean() # Also could do "groups.aggregate(np.mean)"

# Similarly, the median:
print groups.median()

# Apply some arbitrary function to aggregate binned data
print groups.aggregate(lambda x: np.mean(x[x > 0.5]))
```
Edit: As the OP was asking specifically for just the means of b binned by the values in a, just do
```
groups.mean().b
```
Also if you wanted the index to look nicer (e.g. display intervals as the index), as they do in @bdiamante's example, use pandas.cut instead of numpy.digitize. (Kudos to bidamante. I didn't realize pandas.cut existed.)
```
import numpy as np
import pandas

df = pandas.DataFrame({"a": np.random.random(100),
                       "b": np.random.random(100) + 10})

# Bin the data frame by "a" with 10 bins...
bins = np.linspace(df.a.min(), df.a.max(), 10)
groups = df.groupby(pandas.cut(df.a, bins))

# Get the mean of b, binned by the values in a
print groups.mean().b
```
This results in:
```
a
(0.00186, 0.111]    10.421839
(0.111, 0.22]       10.427540
(0.22, 0.33]        10.538932
(0.33, 0.439]       10.445085
(0.439, 0.548]      10.313612
(0.548, 0.658]      10.319387
(0.658, 0.767]      10.367444
(0.767, 0.876]      10.469655
(0.876, 0.986]      10.571008
Name: b
```


Question:
Add Multiple Columns to Pandas Dataframe from Function I have a pandas data frame mydf that has two columns,and both columns are datetime datatypes: mydate and mytime. I want to add three more columns: hour, weekday, and weeknum.
```
def getH(t): #gives the hour
    return t.hour
def getW(d): #gives the week number
    return d.isocalendar()[1]
def getD(d): #gives the weekday
    return d.weekday() # 0 for Monday, 6 for Sunday

mydf["hour"] = mydf.apply(lambda row:getH(row["mytime"]), axis=1)
mydf["weekday"] = mydf.apply(lambda row:getD(row["mydate"]), axis=1)
mydf["weeknum"] = mydf.apply(lambda row:getW(row["mydate"]), axis=1)
```
The snippet works, but it's not computationally efficient as it loops through the data frame at least three times. I would just like to know if there's a faster and/or more optimal way to do this. For example, using zip or merge? If, for example, I just create one function that returns three elements, how should I implement this? To illustrate, the function would be:
```
def getHWd(d,t):
    return t.hour, d.isocalendar()[1], d.weekday()
```

Response:
Here's on approach to do it using one apply Say, df is like
```

```python
df
#Output
#mydate     mytime
#0  2011-01-01 2011-11-14
#1  2011-01-02 2011-11-15
#2  2011-01-03 2011-11-16
#3  2011-01-04 2011-11-17
#4  2011-01-05 2011-11-18
#5  2011-01-06 2011-11-19
#6  2011-01-07 2011-11-20
#7  2011-01-08 2011-11-21
#8  2011-01-09 2011-11-22
#9  2011-01-10 2011-11-23
#10 2011-01-11 2011-11-24
#11 2011-01-12 2011-11-25
#```
#We'll take the lambda function out to separate line for readability and define it like
#```
```

```python
lambdafunc = lambda x: pd.Series([x['mytime'].hour,
                                           x['mydate'].isocalendar()[1],
                                           x['mydate'].weekday()])
```
And, apply and store the result to df[['hour', 'weekday', 'weeknum']]
```
```

```python
df[['hour', 'weekday', 'weeknum']] = df.apply(lambdafunc, axis=1)
```
And, the output is like
```
```

```python
df
#Output
#mydate     mytime  hour  weekday  weeknum
#0  2011-01-01 2011-11-14     0       52        5
#1  2011-01-02 2011-11-15     0       52        6
#2  2011-01-03 2011-11-16     0        1        0
#3  2011-01-04 2011-11-17     0        1        1
#4  2011-01-05 2011-11-18     0        1        2
#5  2011-01-06 2011-11-19     0        1        3
#6  2011-01-07 2011-11-20     0        1        4
#7  2011-01-08 2011-11-21     0        1        5
#8  2011-01-09 2011-11-22     0        1        6
#9  2011-01-10 2011-11-23     0        2        0
#10 2011-01-11 2011-11-24     0        2        1
#11 2011-01-12 2011-11-25     0        2        2
#```
```


Question:
Exception Handling in Pandas .apply() function If I have a DataFrame:
```
myDF = DataFrame(data=[[11,11],[22,'2A'],[33,33]], columns = ['A','B'])
```
Gives the following dataframe (Starting out on stackoverflow and don't have enough reputation for an image of the DataFrame)
```
| A  | B  |

0  | 11 | 11 |

1  | 22 | 2A |

2  | 33 | 33 |
```
If i want to convert column B to int values and drop values that can't be converted I have to do:
```
def convertToInt(cell):
    try:
        return int(cell)
    except:
        return None
myDF['B'] = myDF['B'].apply(convertToInt)
```
If I only do: myDF['B'].apply(int) the error obviously is: C:\WinPython-32bit-2.7.5.3\python-2.7.5\lib\site-packages\pandas\lib.pyd in pandas.lib.map_infer (pandas\lib.c:42840)() ValueError: invalid literal for int() with base 10: '2A' Is there a way to add exception handling to myDF['B'].apply() Thank you in advance!

Response:
I had the same question, but for a more general case where it was hard to tell if the function would generate an exception (i.e. you couldn't explicitly check this condition with something as straightforward as isdigit). After thinking about it for a while, I came up with the solution of embedding the try/except syntax in a separate function. I'm posting a toy example in case it helps anyone.
```
import pandas as pd
import numpy as np

x=pd.DataFrame(np.array([['a','a'], [1,2]]))

def augment(x):
    try:
        return int(x)+1
    except:
        return 'error:' + str(x)

x[0].apply(lambda x: augment(x))
```


Question:
pandas combine two strings ignore nan values I have two columns with strings. I would like to combine them and ignore nan values. Such that:
```
ColA, Colb, ColA+ColB
str   str    strstr
str   nan    str
nan   str    str
```
I tried df['ColA+ColB'] = df['ColA'] + df['ColB'] but that creates a nan value if either column is nan. I've also thought about using concat. I suppose I could just go with that, and then use some df.ColA+ColB[df[ColA] = nan] = df[ColA] but that seems like quite the workaround.

Response:
Call fillna and pass an empty str as the fill value and then sum with param axis=1:
```

```python
df = pd.DataFrame({'a':['asd',np.NaN,'asdsa'], 'b':['asdas','asdas',np.NaN]})
df
#Output
#a      b
#0    asd  asdas
#1    NaN  asdas
#2  asdsa    NaN
```

```python
df['a+b'] = df.fillna('').sum(axis=1)
df
#Output
#a      b       a+b
#0    asd  asdas  asdasdas
#1    NaN  asdas     asdas
#2  asdsa    NaN     asdsa
#```
```


Question:
Add a new row to a Pandas DataFrame with specific index name I'm trying to add a new row to the DataFrame with a specific index name 'e'.
```
number   variable       values
a    NaN       bank          true
b    3.0       shop          false
c    0.5       market        true
d    NaN       government    true
```
I have tried the following but it's creating a new column instead of a new row.
```
new_row = [1.0, 'hotel', 'true']
df = df.append(new_row)
```
Still don't understand how to insert the row with a specific index. Will be grateful for any suggestions.

Response:
You can use df.loc[_not_yet_existing_index_label_] = new_row. Demo:
```

```python
df.loc['e'] = [1.0, 'hotel', 'true']
```

```python
df
#Output
#number    variable values
#a     NaN        bank   True
#b     3.0        shop  False
#c     0.5      market   True
#d     NaN  government   True
#e     1.0       hotel   true
#```
#PS using this method you can't add a row with already existing (duplicate) index value (label) - a row with this index label will be updated in this case. UPDATE: This might not work in recent Pandas/Python3 if the index is a DateTimeIndex and the new row's index doesn't exist. it'll work if we specify correct index value(s). Demo (using pandas: 0.23.4):
#```
```

```python
ix = pd.date_range('2018-11-10 00:00:00', periods=4, freq='30min')
```

```python
df = pd.DataFrame(np.random.randint(100, size=(4,3)), columns=list('abc'), index=ix)
```

```python
df
#Output
#a   b   c
#2018-11-10 00:00:00  77  64  90
#2018-11-10 00:30:00   9  39  26
#2018-11-10 01:00:00  63  93  72
#2018-11-10 01:30:00  59  75  37
```

```python
df.loc[pd.to_datetime('2018-11-10 02:00:00')] = [100,100,100]
```

```python
df
#Output
#a    b    c
#2018-11-10 00:00:00   77   64   90
#2018-11-10 00:30:00    9   39   26
#2018-11-10 01:00:00   63   93   72
#2018-11-10 01:30:00   59   75   37
#2018-11-10 02:00:00  100  100  100
```

```python
df.index
#Output
#DatetimeIndex(['2018-11-10 00:00:00', '2018-11-10 00:30:00', '2018-11-10 01:00:00', '2018-11-10 01:30:00', '2018-11-10 02:00:00'], dtype='da
#tetime64[ns]', freq=None)
#```
```


Question:
Python Pandas dataframe reading exact specified range in an excel sheet I have a lot of different table (and other unstructured data in an excel sheet) .. I need to create a dataframe out of range 'A3:D20' from 'Sheet2' of Excel sheet 'data'. All examples that I come across drilldown up to sheet level, but not how to pick it from an exact range.
```
import openpyxl
import pandas as pd

wb = openpyxl.load_workbook('data.xlsx')
sheet = wb.get_sheet_by_name('Sheet2')
range = ['A3':'D20']   #<-- how to specify this?
spots = pd.DataFrame(sheet.range) #what should be the exact syntax for this?

print (spots)
```
Once I get this, I plan to look up data in column A and find its corresponding value in column B. Edit 1: I realised that openpyxl takes too long, and so have changed that to pandas.read_excel('data.xlsx','Sheet2') instead, and it is much faster at that stage at least. Edit 2: For the time being, I have put my data in just one sheet and: removed all other info added column names, applied index_col on my leftmost column then used wb.loc[]

Response:
Use the following arguments from pandas read_excel documentation: skiprows : list-like Rows to skip at the beginning (0-indexed) nrows: int, default None Number of rows to parse. parse_cols : int or list, default None If None then parse all columns, If int then indicates last column to be parsed If list of ints then indicates list of column numbers to be parsed If string then indicates comma separated list of column names and column ranges (e.g. “A:E” or “A,C,E:F”) I imagine the call will look like:
```
df = read_excel(filename, 'Sheet2', skiprows = 2, nrows=18,  parse_cols = 'A:D')
```
EDIT: in later version of pandas parse_cols has been renamed to usecols so the above call should be rewritten as:
```
df = read_excel(filename, 'Sheet2', skiprows = 2, nrows=18,  usecols= 'A:D')
```


Question:
Pandas: Filtering multiple conditions I'm trying to do boolean indexing with a couple conditions using Pandas. My original DataFrame is called df. If I perform the below, I get the expected result:
```
temp = df[df["bin"] == 3]
temp = temp[(~temp["Def"])]
temp = temp[temp["days since"] > 7]
temp.head()
```
However, if I do this (which I think should be equivalent), I get no rows back:
```
temp2 = df[df["bin"] == 3]
temp2 = temp2[~temp2["Def"] & temp2["days since"] > 7]
temp2.head()
```
Any idea what accounts for the difference?

Response:
Use () because operator precedence:
```
temp2 = df[~df["Def"] & (df["days since"] > 7) & (df["bin"] == 3)]
```
Alternatively, create conditions on separate rows:
```
cond1 = df["bin"] == 3
cond2 = df["days since"] > 7
cond3 = ~df["Def"]

temp2 = df[cond1 & cond2 & cond3]
```
Sample:
```
df = pd.DataFrame({'Def':[True] *2 + [False]*4,
                   'days since':[7,8,9,14,2,13],
                   'bin':[1,3,5,3,3,3]})

print (df)
     Def  bin  days since
0   True    1           7
1   True    3           8
2  False    5           9
3  False    3          14
4  False    3           2
5  False    3          13

temp2 = df[~df["Def"] & (df["days since"] > 7) & (df["bin"] == 3)]
print (temp2)
     Def  days since  bin
3  False          14    3
5  False          13    3
```
Another idea is use functions like Series.lt Series.gt Series.le Series.ge Series.ne:
```
temp2 = df[~df["Def"] & df["days since"].gt(7) & df["bin"].eq(3)]
print (temp2)
     Def  days since  bin
3  False          14    3
5  False          13    3
```


Question:
vlookup in Pandas using join I have the following 2 dataframes
```
Example1
sku loc flag
122  61 True
123  61 True
113  62 True
122  62 True
123  62 False
122  63 False
301  63 True

Example2
sku dept
113 a
122 b
123 b
301 c
```
I want to perform a merge, or join opertation using Pandas (or whichever Python operator is best) to produce the below data frame.
```
Example3
sku loc flag   dept
122  61 True   b
123  61 True   b
113  62 True   a
122  62 True   b
123  62 False  b
122  63 False  b
301  63 True   c

Both
df_Example1.join(df_Example2,lsuffix='_ProdHier')
df_Example1.join(df_Example2,how='outer',lsuffix='_ProdHier')
```
Aren't working. What am I doing wrong?

Response:
Perform a left merge, this will use sku column as the column to join on:
```

```python
df.merge(df1, on='sku', how='left')
#Output
#sku  loc   flag dept
#0  122   61   True    b
#1  122   62   True    b
#2  122   63  False    b
#3  123   61   True    b
#4  123   62  False    b
#5  113   62   True    a
#6  301   63   True    c
#```
#If sku is in fact your index then do this:
#```
```

```python
df.merge(df1, left_index=True, right_index=True, how='left')
#Output
#loc   flag dept
#sku
#113   62   True    a
#122   61   True    b
#122   62   True    b
#122   63  False    b
#123   61   True    b
#123   62  False    b
#301   63   True    c
#```
#Another method is to use map, if you set sku as the index on your second df, so in effect it becomes a Series then the code simplifies to this:
#```
```

```python
df['dept']=df.sku.map(df1.dept)
df
#Output
#sku  loc   flag dept
#0  122   61   True    b
#1  123   61   True    b
#2  113   62   True    a
#3  122   62   True    b
#4  123   62  False    b
#5  122   63  False    b
#6  301   63   True    c
#```
```


Question:
pandas DataFrame, how to apply function to a specific column? I have read the docs of DataFrame.apply DataFrame.apply(func, axis=0, broadcast=False, raw=False, reduce=None, args=(), **kwds)¶ Applies function along input axis of DataFrame. So, How can I apply a function to a specific column?
```

```python
import pandas as pd
```

```python
data = {'A': [1, 2, 3], 'B': [4, 5, 6], 'C': [7, 8, 9]}
```

```python
df = pd.DataFrame(data)
```

```python
df
#Output
#A  B  C
#0  1  4  7
#1  2  5  8
#2  3  6  9
```

```python
def addOne(v):
...:        v += 1
...:        return v
...:
```

```python
df.apply(addOne, axis=1)
#Output
#A  B   C
#0  2  5   8
#1  3  6   9
#2  4  7  10
#```
#I want to addOne to every value in df['A'], not all columns. How can I do that with DataFrame.apply. Thanks for help!
```

Response:
The answer is,
```
df['A'] = df['A'].map(addOne)
```
and maybe you would be better to know about the difference of map, applymap, apply. but if you insist to use apply, you could try like below.
```
def addOne(v):
    v['A'] += 1
    return v

df.apply(addOne, axis=1)
```


Question:
Adding calculated column in Pandas I have a dataframe with 10 columns. I want to add a new column 'age_bmi' which should be a calculated column multiplying 'age' * 'bmi'. age is an INT, bmi is a FLOAT. That then creates the new dataframe with 11 columns. Something I am doing isn't quite right. I think it's a syntax issue. Any ideas? Thanks
```
df2['age_bmi'] = df(['age'] * ['bmi'])
print(df2)
```

Response: try df2['age_bmi'] = df.age * df.bmi. You're trying to call the dataframe as a function, when you need to get the values of the columns, which you can access by key like a dictionary or by property if it's a lowercase name with no spaces that doesn't match a built-in DataFrame method. Someone linked this in a comment the other day and it's pretty awesome. I recommend giving it a watch, even if you don't do the exercises: https://www.youtube.com/watch?v=5JnMutdy6Fw


Question:
Plot a bar plot from a Pandas DataFrame Assuming I have a DataFrame that looks like this:
```none
Hour  V1  V2  A1  A2
   0  15  13  25  37
   1  26  52  21  45
   2  18  45  45  25
   3  65  38  98  14
```
I'm trying to create a bar plot to compare columns V1 and V2 by the Hour. When I do:
```py
import matplotlib.pyplot as plt
ax = df.plot(kind='bar', title ="V comp",figsize=(15,10),legend=True, fontsize=12)
ax.set_xlabel("Hour",fontsize=12)
ax.set_ylabel("V",fontsize=12)
```
I get a plot and a legend with all the columns' values and names. How can I modify my code so the plot and legend only displays the columns V1 and V2?

Response:
To plot just a selection of your columns you can select the columns of interest by passing a list to the subscript operator:
```
ax = df[['V1','V2']].plot(kind='bar', title ="V comp", figsize=(15, 10), legend=True, fontsize=12)
```
What you tried was df['V1','V2'] this will raise a KeyError as correctly no column exists with that label, although it looks funny at first you have to consider that your are passing a list hence the double square brackets [[]].
```
import matplotlib.pyplot as plt
ax = df[['V1','V2']].plot(kind='bar', title ="V comp", figsize=(15, 10), legend=True, fontsize=12)
ax.set_xlabel("Hour", fontsize=12)
ax.set_ylabel("V", fontsize=12)
plt.show()
```


Question:
Create new columns and fill with calculated values from same dataframe Here is a simplified example of my df:
```py
ds = pd.DataFrame(np.abs(randn(3, 4)), index=[1,2,3], columns=['A','B','C','D'])
ds['sum'] = ds.sum(axis=1)
```
which looks like
```none
A         B         C         D       sum
1  0.095389  0.556978  1.646888  1.959295  4.258550
2  1.076190  2.668270  0.825116  1.477040  6.046616
3  0.245034  1.066285  0.967124  0.791606  3.070049
```
I would like to create 4 new columns and calculate the percentage value from the total (sum) in every row. So first value in the first new column should be (0.095389/4.258550), first value in the second new column (0.556978/4.258550)...and so on.

Response:
You can do this easily manually for each column like this:
```
df['A_perc'] = df['A']/df['sum']
```
If you want to do this in one step for all columns, you can use the div method (http://pandas.pydata.org/pandas-docs/stable/basics.html#matching-broadcasting-behavior):
```
ds.div(ds['sum'], axis=0)
```
And if you want this in one step added to the same dataframe:
```

```python
ds.join(ds.div(ds['sum'], axis=0), rsuffix='_perc')
#Output
#          A         B         C         D       sum    A_perc    B_perc  \
#1  0.151722  0.935917  1.033526  0.941962  3.063127  0.049532  0.305543
#2  0.033761  1.087302  1.110695  1.401260  3.633017  0.009293  0.299283
#3  0.761368  0.484268  0.026837  1.276130  2.548603  0.298739  0.190013
```

C_perc    D_perc  sum_perc
1  0.337409  0.307517         1
2  0.305722  0.385701         1
3  0.010530  0.500718         1
```


Question:
Python error : TypeError: Object of type 'Timestamp' is not JSON serializable' I have a Dataframe that has a time stamp column of type 'datetime64[ns]'. When I try to insert it to Salesforce platform get an error 'TypeError: Object of type 'Timestamp' is not JSON serializable'. How could I change this timestamp column to have it updated properly. Given below is the view of the Dataframe.
```
Id,Name,Date,Type
1,ProdA,2018-05-18 04:45:08,S
1,ProdB,2018-05-18 02:15:00,S
1,ProdC,2018-05-16 10:20:00,S
```
Datatype for each of these 4 columns:
```
Id                                     object
Name                                   object
Date                           datetime64[ns]
Type                                   object
dtype: object
```
Could anyone assist on this. Thanks.

Response:
You can try convert datetime to string:
```
df['Date'] = df['Date'].astype(str)
```
Or:
```
df['Date'] = df['Date'].dt.strftime('%Y-%m-%d %H:%M:%S')

print (df.dtypes)
Id      object
Name    object
Date    object
Type    object
dtype: object
```


Question:
How to implode(reverse of pandas explode) based on a column I have a dataframe df like below
```
NETWORK       config_id       APPLICABLE_DAYS  Case    Delivery
0   Grocery     5399            SUN               10       1
1   Grocery     5399            MON               20       2
2   Grocery     5399            TUE               30       3
3   Grocery     5399            WED               40       4
```
I want to implode( combine Applicable_days from multiple rows into single row like below) and get the average case and delivery per config_id
```
NETWORK       config_id       APPLICABLE_DAYS      Avg_Cases    Avg_Delivery
0   Grocery     5399            SUN,MON,TUE,WED         90           10
```
using the groupby on network,config_id i can get the avg_cases and avg_delivery like below.
```
df.groupby(['network','config_id']).agg({'case':'mean','delivery':'mean'})
```
But How do i be able to join APPLICABLE_DAYS while performing this aggregation?

Response:
If you want the "opposite" of explode, then that means bringing it into a list in Solution #1. You can also join as a string in Solution #2: Use lambda x: x.tolist() for the 'APPLICABLE_DAYS' column within your .agg groupby function:
```
df = (df.groupby(['NETWORK','config_id'])
      .agg({'APPLICABLE_DAYS': lambda x: x.tolist(),'Case':'mean','Delivery':'mean'})
      .rename({'Case' : 'Avg_Cases','Delivery' : 'Avg_Delivery'},axis=1)
      .reset_index())
df
Out[1]:
   NETWORK  config_id       APPLICABLE_DAYS  Avg_Cases  Avg_Delivery
0  Grocery       5399  [SUN, MON, TUE, WED]         25           2.5
```
Use lambda x: ",".join(x) for the 'APPLICABLE_DAYS' column within your .agg groupby function:
```
df = (df.groupby(['NETWORK','config_id'])
      .agg({'APPLICABLE_DAYS': lambda x: ",".join(x),'Case':'mean','Delivery':'mean'})
      .rename({'Case' : 'Avg_Cases','Delivery' : 'Avg_Delivery'},axis=1)
      .reset_index())
df
Out[1]:
   NETWORK  config_id       APPLICABLE_DAYS  Avg_Cases  Avg_Delivery
0  Grocery       5399       SUN,MON,TUE,WED         25           2.5
```
If you are looking for the sum, then you can just change mean to sum for the Cases and Delivery columns.


Question:
Use Pandas index in Plotly Express Plotly Express allows me to easily plot a pandas dataframe, as explained in their examples. Rather than using a named column for x and a named column for y, I would like to use the dataframe's index for x and a named column for y. Example using named columns
```
import plotly.express as px
iris = px.data.iris()
fig = px.scatter(iris, x="sepal_width", y="sepal_length")
fig.show()
```
What i want (bogus example)
```
import plotly.express as px
iris = px.data.iris()
fig = px.scatter(iris, x="index", y="sepal_length")
fig.show()
```
This obviously throws: ValueError: Value of 'x' is not the name of a column in 'data_frame'. Expected one of ['sepal_length', 'sepal_width', 'petal_length', 'petal_width', 'species', 'species_id'] but received: index Ugly fix
```
import plotly.express as px
iris = px.data.iris().reset_index()
fig = px.scatter(iris, x="index", y="sepal_length")
fig.show()
```

Response:
Reference: https://plot.ly/python/px-arguments/#using-the-index-of-a-dataframe You can pass the index as reference explicitly. So in your case, this would be:
```
import plotly.express as px
iris = px.data.iris()
fig = px.scatter(iris, x=iris.index, y="sepal_length")
fig.show()
```
-- BONUS QUESTION: what if iris has a pd.MultiIndex? Use pd.MultiIndex.get_level_values.
```
import plotly.express as px

# dummy example for multiindex
iris = px.data.iris().set_index(['species', 'species_id', iris.index])

fig = px.scatter(
   iris,
   x=iris.index.get_level_values(2),
   y="sepal_length"
)

fig.show()
```


Question:
How do i find the iloc of a row in pandas dataframe? I have an indexed pandas dataframe. By searching through its index, I find a row of interest. How do I find out the iloc of this row? Example:
```
dates = pd.date_range('1/1/2000', periods=8)
df = pd.DataFrame(np.random.randn(8, 4), index=dates, columns=['A', 'B', 'C', 'D'])
df
                   A         B         C         D
2000-01-01 -0.077564  0.310565  1.112333  1.023472
2000-01-02 -0.377221 -0.303613 -1.593735  1.354357
2000-01-03  1.023574 -0.139773  0.736999  1.417595
2000-01-04 -0.191934  0.319612  0.606402  0.392500
2000-01-05 -0.281087 -0.273864  0.154266  0.374022
2000-01-06 -1.953963  1.429507  1.730493  0.109981
2000-01-07  0.894756 -0.315175 -0.028260 -1.232693
2000-01-08 -0.032872 -0.237807  0.705088  0.978011

window_stop_row = df[df.index < '2000-01-04'].iloc[-1]
window_stop_row
Timestamp('2000-01-08 00:00:00', offset='D')
#which is the iloc of window_stop_row?
```

Response:
Generally speaking, pass the named index value to index.get_loc:
```
df.index.get_loc(row_of_interest_named_index)
```
Since you’re dealing with dates it may be more convenient to retrieve the index value with .name:
```

```python
dates = pd.date_range('1/1/2000', periods=8)
df = pd.DataFrame(np.random.randn(8, 4), index=dates, columns=['A', 'B', 'C', 'D'])
df
#Output
#A         B         C         D
#2000-01-01  0.095234 -1.000863  0.899732 -1.742152
#2000-01-02 -0.517544 -1.274137  1.734024 -1.369487
#2000-01-03  0.134112  1.964386 -0.120282  0.573676
#2000-01-04 -0.737499 -0.581444  0.528500 -0.737697
#2000-01-05 -1.777800  0.795093  0.120681  0.524045
#2000-01-06 -0.048432 -0.751365 -0.760417 -0.181658
#2000-01-07 -0.570800  0.248608 -1.428998 -0.662014
#2000-01-08 -0.147326  0.717392  3.138620  1.208639
```

```python
window_stop_row = df[df.index < '2000-01-04'].iloc[-1]
window_stop_row.name
#Output
#Timestamp('2000-01-03 00:00:00', offset='D')
```

```python
df.index.get_loc(window_stop_row.name)
#Output
#2
#```
#get_loc returns the ordinal position of the label in your index which is what you want:
#```
```

```python
df.iloc[df.index.get_loc(window_stop_row.name)]
#Output
#A    0.134112
#B    1.964386
#C   -0.120282
#D    0.573676
#Name: 2000-01-03 00:00:00, dtype: float64
#```
#if you just want to search the index then so long as it is sorted then you can use searchsorted:
#```
```

```python
df.index.searchsorted('2000-01-04') - 1
#Output
#2
#```
```


Question:
Pandas: IndexingError: Unalignable boolean Series provided as indexer I'm trying to run what I think is simple code to eliminate any columns with all NaNs, but can't get this to work (axis = 1 works just fine when eliminating rows):
```
import pandas as pd
import numpy as np

df = pd.DataFrame({'a':[1,2,np.nan,np.nan], 'b':[4,np.nan,6,np.nan], 'c':[np.nan, 8,9,np.nan], 'd':[np.nan,np.nan,np.nan,np.nan]})

df = df[df.notnull().any(axis = 0)]

print df
```
Full error: raise IndexingError('Unalignable boolean Series provided as 'pandas.core.indexing.IndexingError: Unalignable boolean Series provided as indexer (index of the boolean Series and of the indexed object do not match Expected output:
```
a    b    c
0  1.0  4.0  NaN
1  2.0  NaN  8.0
2  NaN  6.0  9.0
3  NaN  NaN  NaN
```

Response:
You need loc, because filter by columns:
```
print (df.notnull().any(axis = 0))
a     True
b     True
c     True
d    False
dtype: bool

df = df.loc[:, df.notnull().any(axis = 0)]
print (df)

     a    b    c
0  1.0  4.0  NaN
1  2.0  NaN  8.0
2  NaN  6.0  9.0
3  NaN  NaN  NaN
```
Or filter columns and then select by []:
```
print (df.columns[df.notnull().any(axis = 0)])
Index(['a', 'b', 'c'], dtype='object')

df = df[df.columns[df.notnull().any(axis = 0)]]
print (df)

     a    b    c
0  1.0  4.0  NaN
1  2.0  NaN  8.0
2  NaN  6.0  9.0
3  NaN  NaN  NaN
```
Or dropna with parameter how='all' for remove all columns filled by NaNs only:
```
print (df.dropna(axis=1, how='all'))
     a    b    c
0  1.0  4.0  NaN
1  2.0  NaN  8.0
2  NaN  6.0  9.0
3  NaN  NaN  NaN
```


Question:
Remove rows with empty lists from pandas data frame I have a data frame with some columns with empty lists and others with lists of strings:
```
donation_orgs                              donation_context
0            []                                           []
1   [the research of Dr. ...]   [In lieu of flowers , memorial donations ...]
```
I'm trying to return a data set without any of the rows where there are empty lists. I've tried just checking for null values:
```
dfnotnull = df[df.donation_orgs != []]
dfnotnull
```
and
```
dfnotnull = df[df.notnull().any(axis=1)]
pd.options.display.max_rows=500
dfnotnull
```
And I've tried looping through and checking for values that exist, but I think the lists aren't returning Null or None like I thought they would:
```
dfnotnull = pd.DataFrame(columns=('donation_orgs', 'donation_context'))
for i in range(0,len(df)):
    if df['donation_orgs'].iloc(i):
        dfnotnull.loc[i] = df.iloc[i]
```
All three of the above methods simply return every row in the original data frame.=

Response:
To avoid converting to str and actually use the lists, you can do this:
```
df[df['donation_orgs'].map(len) > 0]
```
It maps the donation_orgs column to the length of the lists of each row and keeps only the ones that have at least one element, filtering out empty lists. It returns
```
Out[1]:
                            donation_context          donation_orgs
1  [In lieu of flowers , memorial donations]  [the research of Dr.]
```
as expected.


Question:
reading excel to a python data frame starting from row 5 and including headers I have an excel workbook that runs some vba on opening which refreshes a pivot table and does some other stuff. Then I wish to import the results of the pivot table refresh into a dataframe in python for further analysis.
```py
import xlrd

wb = xlrd.open_workbook('C:\Users\cb\Machine_Learning\cMap_Joins.xlsm')
```
The refreshing and opening of the file works fine. But how do I select the data from the first sheet from say row 5 including header down to last record n.

Response:
You can use pandas' ExcelFile parse method to read Excel sheets, see io docs:
```
xls = pd.ExcelFile('C:\Users\cb\Machine_Learning\cMap_Joins.xlsm')

df = xls.parse('Sheet1', skiprows=4, index_col=None, na_values=['NA'])
```
skiprows will ignore the first 4 rows (i.e. start at row index 4), and several other options.


Question:
Pivot Tables of Counts in Pandas DataFrame I have a pandas dataframe:
```none
Col X    Col Y
class 1    cat 1
class 2    cat 1
class 3    cat 2
class 2    cat 3
```
that I want to transform into:
```none
cat 1  cat 2  cat 3
class 1      1      0      0
class 2      1      0      1
class 3      0      1      0
```
where the values are value counts. How do I do it?

Response:
Here are couple of ways to reshape your data df
```

```python
df
#Output
#Col X  Col Y
#0  class 1  cat 1
#1  class 2  cat 1
#2  class 3  cat 2
#3  class 2  cat 3
#```
#1) Using pd.crosstab()
#```
```

```python
pd.crosstab(df['Col X'], df['Col Y'])
#Output
#Col Y    cat 1  cat 2  cat 3
#Col X
#class 1      1      0      0
#class 2      1      0      1
#class 3      0      1      0
#```
#2) Or, use groupby on 'Col X','Col Y' with unstack over Col Y, then fill NaNs with zeros.
#```
```

```python
df.groupby(['Col X','Col Y']).size().unstack('Col Y', fill_value=0)
#Output
#Col Y    cat 1  cat 2  cat 3
#Col X
#class 1      1      0      0
#class 2      1      0      1
#class 3      0      1      0
#```
#3) Or, use pd.pivot_table() with index=Col X, columns=Col Y
#```
```

```python
pd.pivot_table(df, index=['Col X'], columns=['Col Y'], aggfunc=len, fill_value=0)
#Output
#Col Y    cat 1  cat 2  cat 3
#Col X
#class 1      1      0      0
#class 2      1      0      1
#class 3      0      1      0
#```
#4) Or, use set_index with unstack
#```
```

```python
df.assign(v=1).set_index(['Col X', 'Col Y'])['v'].unstack(fill_value=0)
#Output
#Col Y    cat 1  cat 2  cat 3
#Col X
#class 1      1      0      0
#class 2      1      0      1
#class 3      0      1      0
#```
```


Question:
How to extract first 8 characters from a string in pandas I have column in a dataframe and i am trying to extract 8 digits from a string. How can I do it
```
Input
 Shipment ID
20180504-S-20000
20180514-S-20537
20180514-S-20541
20180514-S-20644
20180514-S-20644
20180516-S-20009
20180516-S-20009
20180516-S-20009
20180516-S-20009
```
Expected Output
```
Order_Date
20180504
20180514
20180514
20180514
20180514
20180516
20180516
20180516
20180516
```
I tried below code and it didnt work.
```
data['Order_Date'] = data['Shipment ID'][:8]
```

Response:
You are close, need indexing with str which is apply for each value of Series:
```
data['Order_Date'] = data['Shipment ID'].str[:8]
```
For better performance if no NaNs values:
```
data['Order_Date'] = [x[:8] for x in data['Shipment ID']]
```
```
print (data)
        Shipment ID Order_Date
0  20180504-S-20000   20180504
1  20180514-S-20537   20180514
2  20180514-S-20541   20180514
3  20180514-S-20644   20180514
4  20180514-S-20644   20180514
5  20180516-S-20009   20180516
6  20180516-S-20009   20180516
7  20180516-S-20009   20180516
8  20180516-S-20009   20180516
```
If omit str code filter column by position, first N values like:
```
print (data['Shipment ID'][:2])
0    20180504-S-20000
1    20180514-S-20537
Name: Shipment ID, dtype: object
```


Question:
Python & Pandas - Group by day and count for each day I am new on pandas and for now i don't get how to arrange my time serie, take a look at it :
```
date & time of connection
19/06/2017 12:39
19/06/2017 12:40
19/06/2017 13:11
20/06/2017 12:02
20/06/2017 12:04
21/06/2017 09:32
21/06/2017 18:23
21/06/2017 18:51
21/06/2017 19:08
21/06/2017 19:50
22/06/2017 13:22
22/06/2017 13:41
22/06/2017 18:01
23/06/2017 16:18
23/06/2017 17:00
23/06/2017 19:25
23/06/2017 20:58
23/06/2017 21:03
23/06/2017 21:05
```
This is a sample of a dataset of 130 k raws,I tried : df.groupby('date & time of connection')['date & time of connection'].apply(list) Not enough i guess I think i should : Create a dictionnary with index from dd/mm/yyyy to dd/mm/yyyy Convert "date & time of connection" type dateTime to Date Group and count Date of "date & time of connection" Put the numbers i count inside the dictionary ? What do you think about my logic ? Do you know some tutos ? Thank you very much

Response:
You can use dt.floor for convert to dates and then value_counts or groupby with size:
```
df = (pd.to_datetime(df['date & time of connection'])
       .dt.floor('d')
       .value_counts()
       .rename_axis('date')
       .reset_index(name='count'))
print (df)
        date  count
0 2017-06-23      6
1 2017-06-21      5
2 2017-06-19      3
3 2017-06-22      3
4 2017-06-20      2
```
Or:
```
s = pd.to_datetime(df['date & time of connection'])
df = s.groupby(s.dt.floor('d')).size().reset_index(name='count')
print (df)
  date & time of connection  count
0                2017-06-19      3
1                2017-06-20      2
2                2017-06-21      5
3                2017-06-22      3
4                2017-06-23      6
```
Timings:
```
np.random.seed(1542)

N = 220000
a = np.unique(np.random.randint(N, size=int(N/2)))
df = pd.DataFrame(pd.date_range('2000-01-01', freq='37T', periods=N)).drop(a)
df.columns = ['date & time of connection']
df['date & time of connection'] = df['date & time of connection'].dt.strftime('%d/%m/%Y %H:%M:%S')
print (df.head())

```python
%%timeit
     ...: df['date & time of connection']=pd.to_datetime(df['date & time of connection'])
     ...: df1 = df.groupby(by=df['date & time of connection'].dt.date).count()
     ...:
539 ms ± 45.9 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
```

```python
%%timeit
     ...: df1 = (pd.to_datetime(df['date & time of connection'])
     ...:        .dt.floor('d')
     ...:        .value_counts()
     ...:        .rename_axis('date')
     ...:        .reset_index(name='count'))
     ...:
12.4 ms ± 350 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
```

```python
%%timeit
     ...: s = pd.to_datetime(df['date & time of connection'])
     ...: df2 = s.groupby(s.dt.floor('d')).size().reset_index(name='count')
     ...:
17.7 ms ± 140 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
```
```


Question: Pandas text matching like SQL's LIKE? Is there a way to do something similar to SQL's LIKE syntax on a pandas text DataFrame column, such that it returns a list of indices, or a list of booleans that can be used for indexing the dataframe? For example, I would like to be able to match all rows where the column starts with 'prefix_', similar to WHERE  LIKE prefix_% in SQL.

Response:
You can use the Series method str.startswith (which takes a regex):
```

```python
s = pd.Series(['aa', 'ab', 'ca', np.nan])
```

```python
s.str.startswith('a', na=False)
#Output
#0     True
#1     True
#2    False
#3    False
#dtype: bool
#```
#You can also do the same with str.contains (using a regex):
#```
```

```python
s.str.contains('^a', na=False)
#Output
#0     True
#1     True
#2    False
#3    False
#dtype: bool
#```
#So you can do df[col].str.startswith... See also the SQL comparison section of the docs. Note: (as pointed out by OP) by default NaNs will propagate (and hence cause an indexing error if you want to use the result as a boolean mask), we use this flag to say that NaN should map to False.
#```
```

```python
s.str.startswith('a')  # can't use as boolean mask
#Output
#0     True
#1     True
#2    False
#3      NaN
#dtype: object
#```
```


Question:
Set index name of pandas DataFrame I have a pandas dataframe like this:
```
''     count
sugar      420
milk       108
vanilla    450
...
```
The first column has no header and I would like to give it the name: 'ingredient'. I created the dataframe from a csv file:
```
df = pd.read_csv('./data/file_name.csv', index_col=False, encoding="ISO-8859-1")
df = df['ingredient_group']  #selecting column
df = df.value_counts()       #calculating string occurance which return series obj
df = pd.DataFrame(df)        #creating dataframe from series obj
```
How do I assign the name 'ingredient' to the first column which has currently no name? I already tried:
```
df_count.rename(columns={'': 'ingredient'}, inplace=True)

df = pd.DataFrame(df, columns = ['ingredient','count']
```
How do I prevent this from happening?
```
''        count
ingredient  ''
sugar      420
milk       108
vanilla    450
...
```

Response:
if ingredients is the name of the index then you can set it by
```
df.index.name='ingredient'
```
With the current solutions you have 'ingredient' as name of the index, which is printed in different row to that of column names. This cannot be changed as is. Try the modified solution below, here the index is copied on to a new column with column name and the index replaced with sequence of numbers.
```
df['ingredient']=df.index
df = df.reset_index(drop=True)
```


Question: pandas join DataFrame force suffix? How can I force a suffix on a merge or join. I understand it's possible to provide one if there is a collision but in my case I'm merging df1 with df2 which doesn't cause any collision but then merging again on df2 which uses the suffixes but I would prefer for each merge to have a suffix because it gets confusing if I do different combinations as you could imagine.

Response:
You could force a suffix on the actual DataFrame:
```

```python
df_a = pd.DataFrame([[1], [2]], columns=['A'])
```

```python
df_b = pd.DataFrame([[3], [4]], columns=['B'])
```

```python
df_a.join(df_b)
#Output
#A  B
#0  1  3
#1  2  4
#```
#By appending to its column's names:
#```
```

```python
df_a.columns = df_a.columns.map(lambda x: str(x) + '_a')
```

```python
df_a
#Output
#A_a
#0    1
#1    2
#```
#Now joins won't need the suffix correction, whether they collide or not:
#```
```

```python
df_b.columns = df_b.columns.map(lambda x: str(x) + '_b')
```

```python
df_a.join(df_b)
#Output
#A_a  B_b
#0    1    3
#1    2    4
#```
```


Question:
Stack two pandas data frames How do I stack the following 2 dataframes:
```
df1
    hzdept_r    hzdepb_r    sandtotal_r
0   0           114         0
1   114         152         92.1

df2
    hzdept_r    hzdepb_r    sandtotal_r
0   0           23          83.5
1   23          152         45
```
to give the following result:
```
hzdept_r    hzdepb_r    sandtotal_r
0   0           114         0
1   114         152         92.1
2   0           23          83.5
3   23          152         45
```
Using the pandas merge operations does not work since it just lines the dataframes horizontally (and not vertically, which is what I want)

Response:
```

```python
a = pd.DataFrame(data=np.random.randint(0,100,(2,5)),columns=list('ABCDE'))
```

```python
b = pd.DataFrame(data=np.random.randint(0,100,(2,5)),columns=list('ABCDE'))
```

```python
c = pd.concat([a,b],ignore_index=True)
```

```python
c
#Output
#A   B   C   D   E
#0  12  56  62  35  20
#1  10  71  63   0  70
#2  61  72  29  10  71
#3  88  82  39  73  94
#```
```


Question: Closest equivalent of a factor variable in Python Pandas What is the closest equivalent to an R Factor variable in Python pandas?

Response:
This question seems to be from a year back but since it is still open here's an update. pandas has introduced a categorical dtype and it operates very similar to factors in R. Please see this link for more information: http://pandas-docs.github.io/pandas-docs-travis/categorical.html Reproducing a snippet from the link above showing how to create a "factor" variable in pandas.
```

```python
s = Series(["a","b","c","a"], dtype="category")
```

```python
s
#Output
#0    a
#1    b
#2    c
#3    a
#dtype: category
#Categories (3, object): [a < b < c]
#```
```


Question:
Specifying dtype float32 with pandas.read_csv on pandas 0.10.1 I'm attempting to read a simple space-separated file with pandas read_csv method. However, pandas doesn't seem to be obeying my dtype argument. Maybe I'm incorrectly specifying it? I've distilled down my somewhat complicated call to read_csv to this simple test case. I'm actually using the converters argument in my 'real' scenario but I removed this for simplicity. Below is my ipython session:
```

```python
cat test.out
#Output
#a b
#0.76398 0.81394
#0.32136 0.91063
```

```python
import pandas
```

```python
import numpy
```

```python
x = pandas.read_csv('test.out', dtype={'a': numpy.float32}, delim_whitespace=True)
```

```python
x
#Output
#         a        b
#0  0.76398  0.81394
#1  0.32136  0.91063
```

```python
x.a.dtype
#Output
#dtype('float64')
#```
#I've also tried this using this with a dtype of numpy.int32 or numpy.int64. These choices result in an exception:
#```
#AttributeError: 'NoneType' object has no attribute 'dtype'
#```
#I'm assuming the AttributeError is because pandas will not automatically try to convert/truncate the float values into an integer? I'm running on a 32-bit machine with a 32-bit version of Python.
#```
```

```python
!uname -a
#Output
#Linux ubuntu 3.0.0-13-generic #22-Ubuntu SMP Wed Nov 2 13:25:36 UTC 2011 i686 i686 i386 GNU/Linux
```

```python
import platform
```

```python
platform.architecture()
#Output
#('32bit', 'ELF')
```

```python
pandas.__version__
#Output
#'0.10.1'
#```
```

Response:
0.10.1 doesn't really support float32 very much see this http://pandas.pydata.org/pandas-docs/dev/whatsnew.html#dtype-specification you can do this in 0.11 like this:
```
# dont' use dtype converters explicity for the columns you care about
# they will be converted to float64 if possible, or object if they cannot
df = pd.read_csv('test.csv'.....)

#### this is optional and related to the issue you posted ####
# force anything that is not a numeric to nan
# columns are the list of columns that you are interesetd in
df[columns] = df[columns].convert_objects(convert_numeric=True)

    # astype
    df[columns] = df[columns].astype('float32')

see http://pandas.pydata.org/pandas-docs/dev/basics.html#object-conversion

Its not as efficient as doing it directly in read_csv (but that requires
 some low-level changes)
```
I have confirmed that with 0.11-dev, this DOES work (on 32-bit and 64-bit, results are the same)
```

```python
x = pd.read_csv(StringIO.StringIO(data), dtype={'a': np.float32}, delim_whitespace=True)
```

```python
x
#Output
#a        b
#0  0.76398  0.81394
#1  0.32136  0.91063
```

```python
x.dtypes
#Output
#a    float32
#b    float64
#dtype: object
```

```python
pd.__version__
#Output
#'0.11.0.dev-385ff82'
```

```python
quit()
vagrant@precise32:~/pandas$ uname -a
Linux precise32 3.2.0-23-generic-pae #36-Ubuntu SMP Tue Apr 10 22:19:09 UTC 2012 i686 i686 i386 GNU/Linux
```
```


Question:
Extract value from single row of pandas DataFrame I have a dataset in a relational database format (linked by ID's over various .csv files). I know that each data frame contains only one value of an ID, and I'd like to know the simplest way to extract values from that row. What I'm doing now:
```
# the group has only one element
purchase_group = purchase_groups.get_group(user_id)
price = list(purchase_group['Column_name'])[0]
```
The third row is bothering me as it seems ugly, however I'm not sure what is the workaround. The grouping (I guess) assumes that there might be multiple values and returns a  object, while I'd like just a row returned.

Response:
If you want just the value and not a df/series then call values and index the first element [0] so just:
```
price = purchase_group['Column_name'].values[0]
```
will work.


Question:
Python: reduce precision pandas timestamp dataframe Hello I have the following dataframe
```
df =

       Record_ID       Time
        94704   2014-03-10 07:19:19.647342
        94705   2014-03-10 07:21:44.479363
        94706   2014-03-10 07:21:45.479581
        94707   2014-03-10 07:21:54.481588
        94708   2014-03-10 07:21:55.481804
```
Is it possible to the have following?
```
df1 =

       Record_ID       Time
        94704   2014-03-10 07:19:19
        94705   2014-03-10 07:21:44
        94706   2014-03-10 07:21:45
        94707   2014-03-10 07:21:54
        94708   2014-03-10 07:21:55
```

Response:
You could convert the underlying datetime64[ns] values to datetime64[s] values using astype:
```

```python
df['Time'] = df['Time'].astype('datetime64[s]')
```

```python
df
#Output
#Record_ID                Time
#0      94704 2014-03-10 07:19:19
#1      94705 2014-03-10 07:21:44
#2      94706 2014-03-10 07:21:45
#3      94707 2014-03-10 07:21:54
#4      94708 2014-03-10 07:21:55
#```
#Note that since Pandas Series and DataFrames store all datetime values as datetime64[ns] these datetime64[s] values are automatically converted back to datetime64[ns], so the end result is still stored as datetime64[ns] values, but the call to astype causes the fractional part of the seconds to be removed. If you wish to have a NumPy array of datetime64[s] values, you could use df['Time'].values.astype('datetime64[s]').
```


Question:
How to apply custom function to pandas data frame for each row [duplicate] This question already has answers here: How to apply a function to two columns of Pandas dataframe (16 answers) Closed 4 years ago. I want to apply a custom function and create a derived column called population2050 that is based on two columns already present in my data frame.
```
import pandas as pd
import sqlite3
conn = sqlite3.connect('factbook.db')
query = "select * from facts where area_land =0;"
facts = pd.read_sql_query(query,conn)
print(list(facts.columns.values))

def final_pop(initial_pop,growth_rate):
    final = initial_pop*math.e**(growth_rate*35)
    return(final)

facts['pop2050'] = facts['population','population_growth'].apply(final_pop,axis=1)
```
When I run the above code, I get an error. Am I not using the 'apply' function correctly?

Response:
You were almost there:
```
facts['pop2050'] = facts.apply(lambda row: final_pop(row['population'],row['population_growth']),axis=1)
```
Using lambda allows you to keep the specific (interesting) parameters listed in your function, rather than bundling them in a 'row'.


Question: Seaborn heatmap not displaying all xticks and yticks I have a pandas dataframe of shape (39, 67). When I plot it's seaborn heatmap, I don't get as many labels on the X and Y axes. .get_xticklabels() method also returns only 23 labels. matplotlib doesn't show any labels (only numbers) as well. Both these heatmaps are for the same dataframe (39, 67).

Response:
To ensure the labels are visible, you have to set the parameters xticklabels, yticklabels to True, like so.
```
import seaborn as sns
sns.heatmap(dataframe, xticklabels=True, yticklabels=True)
```
Here's the documentation for the heatmap function.


Question:
vectorize conditional assignment in pandas dataframe If I have a dataframe df with column x and want to create column y based on values of x using this in pseudo code:
```
if df['x'] < -2 then df['y'] = 1
else if df['x'] > 2 then df['y'] = -1
else df['y'] = 0
```
How would I achieve this? I assume np.where is the best way to do this but not sure how to code it correctly.

Response:
One simple method would be to assign the default value first and then perform 2 loc calls:
```

```python
df = pd.DataFrame({'x':[0,-3,5,-1,1]})
df
#Output
#x
#0  0
#1 -3
#2  5
#3 -1
#4  1
```

```python
df['y'] = 0
df.loc[df['x'] < -2, 'y'] = 1
df.loc[df['x'] > 2, 'y'] = -1
df
#Output
#x  y
#0  0  0
#1 -3  1
#2  5 -1
#3 -1  0
#4  1  0
#```
#If you wanted to use np.where then you could do it with a nested np.where:
#```
```

```python
df['y'] = np.where(df['x'] < -2 , 1, np.where(df['x'] > 2, -1, 0))
df
#Output
#x  y
#0  0  0
#1 -3  1
#2  5 -1
#3 -1  0
#4  1  0
#```
#So here we define the first condition as where x is less than -2, return 1, then we have another np.where which tests the other condition where x is greater than 2 and returns -1, otherwise return 0 timings
#```
```

```python
%timeit df['y'] = np.where(df['x'] < -2 , 1, np.where(df['x'] > 2, -1, 0))

1000 loops, best of 3: 1.79 ms per loop
```

```python
%%timeit
df['y'] = 0
df.loc[df['x'] < -2, 'y'] = 1
df.loc[df['x'] > 2, 'y'] = -1

100 loops, best of 3: 3.27 ms per loop
```
So for this sample dataset the np.where method is twice as fast
```


Question:
How to save image created with 'pandas.DataFrame.plot'? [duplicate] This question already has answers here: Saving plots (AxesSubPlot) generated from python pandas with matplotlib's savefig (6 answers) save a pandas.Series histogram plot to file (4 answers) Closed 5 years ago. When trying to save plot image created with 'pandas.DataFrame.plot' from ' pandas.core.series.Series' object :
```
%matplotlib inline
type(class_counts) # pandas.core.series.Series
class_counts.plot(kind='bar',  figsize=(20, 16), fontsize=26)
```
Like this:
```
import matplotlib.pyplot as plt
plt.savefig('figure_1.pdf', dpi=300)
```
results in empty pdf file. How to save image created with 'pandas.DataFrame.plot'?

Response:
Try this :
```
fig = class_counts.plot(kind='bar',
        figsize=(20, 16), fontsize=26).get_figure()

fig.savefig('test.pdf')
```


Question:
Group by two columns and count the occurrences of each combination in Pandas I have the following data frame:
```
data = pd.DataFrame({'user_id' : ['a1', 'a1', 'a1', 'a2','a2','a2','a3','a3','a3'], 'product_id' : ['p1','p1','p2','p1','p1','p1','p2','p2','p3']})

product_id  user_id
    p1       a1
    p1       a1
    p2       a1
    p1       a2
    p1       a2
    p1       a2
    p2       a3
    p2       a3
    p3       a3
```
in real case there might be some other columns as well, but what i need to do is to group by data frame by product_id and user_id columns and count number of each combination and add it as a new column in a new dat frame output should be something like this:
```
user_id product_id  count
a1       p1            2
a1       p2            1
a2       p1            3
a3       p2            2
a3       p3            1
```
I have tried the following code:
```
grouped=data.groupby(['user_id','product_id']).count()
```
but the result is:
```
user_id product_id
 a1       p1
          p2
 a2       p1
 a3       p2
          p3
```
actually the most important thing for me is to have a column names count that has the number of occurrences , i need to use the column later.

Response:
Maybe this is what you want?
```

```python
data = pd.DataFrame({'user_id' : ['a1', 'a1', 'a1', 'a2','a2','a2','a3','a3','a3'], 'product_id' : ['p1','p1','p2','p1','p1','p1','p2','p2','p3']})
```

```python
count_series = data.groupby(['user_id', 'product_id']).size()
```

```python
count_series
#Output
#user_id  product_id
#a1       p1            2
#         p2            1
#a2       p1            3
#a3       p2            2
#         p3            1
#dtype: int64
```

```python
new_df = count_series.to_frame(name = 'size').reset_index()
```

```python
new_df
#Output
#  user_id product_id  size
#0      a1         p1     2
#1      a1         p2     1
#2      a2         p1     3
#3      a3         p2     2
#4      a3         p3     1
```

```python
new_df['size']
#Output
#0    2
#1    1
#2    3
#3    2
#4    1
#Name: size, dtype: int64
#```
```


Question: Opening a pdf and reading in tables with python pandas Is it possible to open PDFs and read it in using python pandas or do I have to use the pandas clipboard for this function?

Response:
you can use tabula https://blog.chezo.uno/tabula-py-extract-table-from-pdf-into-python-dataframe-6c7acfa5f302
```
from tabula import read_pdf
df = read_pdf('data.pdf')
```
I can see more in the link!


Question:
Pandas selecting by label sometimes return Series, sometimes returns DataFrame In Pandas, when I select a label that only has one entry in the index I get back a Series, but when I select an entry that has more then one entry I get back a data frame. Why is that? Is there a way to ensure I always get back a data frame?
```

```python
import pandas as pd
```

```python
df = pd.DataFrame(data=range(5), index=[1, 2, 3, 3, 3])
```

```python
type(df.loc[3])
#Output
#pandas.core.frame.DataFrame
```

```python
type(df.loc[1])
#Output
#pandas.core.series.Series
#```
```

Response:
Granted that the behavior is inconsistent, but I think it's easy to imagine cases where this is convenient. Anyway, to get a DataFrame every time, just pass a list to loc. There are other ways, but in my opinion this is the cleanest.
```

```python
type(df.loc[[3]])
#Output
#pandas.core.frame.DataFrame
```

```python
type(df.loc[[1]])
#Output
#pandas.core.frame.DataFrame
#```
```


Question:
Extracting just Month and Year separately from Pandas Datetime column I have a Dataframe, df, with the following column:
```none
ArrivalDate
936   2012-12-31
938   2012-12-29
965   2012-12-31
966   2012-12-31
967   2012-12-31
968   2012-12-31
969   2012-12-31
970   2012-12-29
971   2012-12-31
972   2012-12-29
973   2012-12-29
```
The elements of the column are pandas.tslib.Timestamp type. I want to extract the year and month. Here's what I've tried:
```py
df['ArrivalDate'].resample('M', how = 'mean')
```
which throws the following error:
```none
Only valid with DatetimeIndex or PeriodIndex
```
Then I tried:
```py
df['ArrivalDate'].apply(lambda(x):x[:-2])
```
which throws the following error:
```none
'Timestamp' object has no attribute '__getitem__'
```
My current solution is
```py
df.index = df['ArrivalDate']
```
Then, I can resample another column using the index. But I'd still like a method for reconfiguring the entire column. Any ideas?

Response:
If you want new columns showing year and month separately you can do this:
```
df['year'] = pd.DatetimeIndex(df['ArrivalDate']).year
df['month'] = pd.DatetimeIndex(df['ArrivalDate']).month
```
or...
```
df['year'] = df['ArrivalDate'].dt.year
df['month'] = df['ArrivalDate'].dt.month
```
Then you can combine them or work with them just as they are.


Question:
Add missing dates to pandas dataframe My data can have multiple events on a given date or NO events on a date. I take these events, get a count by date and plot them. However, when I plot them, my two series don't always match.
```
idx = pd.date_range(df['simpleDate'].min(), df['simpleDate'].max())
s = df.groupby(['simpleDate']).size()
```
In the above code idx becomes a range of say 30 dates. 09-01-2013 to 09-30-2013 However S may only have 25 or 26 days because no events happened for a given date. I then get an AssertionError as the sizes dont match when I try to plot:
```
fig, ax = plt.subplots()
ax.bar(idx.to_pydatetime(), s, color='green')
```
What's the proper way to tackle this? Do I want to remove dates with no values from IDX or (which I'd rather do) is add to the series the missing date with a count of 0. I'd rather have a full graph of 30 days with 0 values. If this approach is right, any suggestions on how to get started? Do I need some sort of dynamic reindex function? Here's a snippet of S ( df.groupby(['simpleDate']).size() ), notice no entries for 04 and 05.
```
09-02-2013     2
09-03-2013    10
09-06-2013     5
09-07-2013     1
```

Response:
You could use Series.reindex:
```
import pandas as pd

idx = pd.date_range('09-01-2013', '09-30-2013')

s = pd.Series({'09-02-2013': 2,
               '09-03-2013': 10,
               '09-06-2013': 5,
               '09-07-2013': 1})
s.index = pd.DatetimeIndex(s.index)

s = s.reindex(idx, fill_value=0)
print(s)
```
yields
```
2013-09-01     0
2013-09-02     2
2013-09-03    10
2013-09-04     0
2013-09-05     0
2013-09-06     5
2013-09-07     1
2013-09-08     0
...
```


Question:
Python: Scaling numbers column by column with pandas I have a Pandas data frame 'df' in which I'd like to perform some scalings column by column. In column 'a', I need the maximum number to be 1, the minimum number to be 0, and all other to be spread accordingly. In column 'b', however, I need the minimum number to be 1, the maximum number to be 0, and all other to be spread accordingly. Is there a Pandas function to perform these two operations? If not, numpy would certainly do.
```
a    b
A   14   103
B   90   107
C   90   110
D   96   114
E   91   114
```

Response:
This is how you can do it using sklearn and the preprocessing module. Sci-Kit Learn has many pre-processing functions for scaling and centering data.
```

```python
from sklearn.preprocessing import MinMaxScaler
```

```python
df = pd.DataFrame({'A':[14,90,90,96,91],
                           'B':[103,107,110,114,114]}).astype(float)
```

```python
df
#Output
#A    B
#0  14  103
#1  90  107
#2  90  110
#3  96  114
#4  91  114
```

```python
scaler = MinMaxScaler()
```

```python
df_scaled = pd.DataFrame(scaler.fit_transform(df), columns=df.columns)
```

```python
df_scaled
#Output
#A         B
#0  0.000000  0.000000
#1  0.926829  0.363636
#2  0.926829  0.636364
#3  1.000000  1.000000
#4  0.939024  1.000000
#```
```


Question:
sorting by a custom list in pandas After reading through: http://pandas.pydata.org/pandas-docs/version/0.13.1/generated/pandas.DataFrame.sort.html I still can't seem to figure out how to sort a column by a custom list. Obviously, the default sort is alphabetical. I'll give an example. Here is my (very abridged) dataframe:
```
Player      Year   Age   Tm     G
2967     Cedric Hunter   1991    27  CHH     6
5335     Maurice Baker   2004    25  VAN     7
13950    Ratko Varda     2001    22  TOT     60
6141     Ryan Bowen      2009    34  OKC     52
6169     Adrian Caldwell 1997    31  DAL     81
```
I want to be able to sort by Player, Year and then Tm. The default sort by Player and Year is fine for me, in normal order. However, I do not want Team sorted alphabetically b/c I want TOT always at the top. Here is the list I created:
```
sorter = ['TOT', 'ATL', 'BOS', 'BRK', 'CHA', 'CHH', 'CHI', 'CLE', 'DAL', 'DEN',
   'DET', 'GSW', 'HOU', 'IND', 'LAC', 'LAL', 'MEM', 'MIA', 'MIL',
   'MIN', 'NJN', 'NOH', 'NOK', 'NOP', 'NYK', 'OKC', 'ORL', 'PHI',
   'PHO', 'POR', 'SAC', 'SAS', 'SEA', 'TOR', 'UTA', 'VAN',
   'WAS', 'WSB']
```
After reading through the link above, I thought this would work but it didn't:
```
df.sort(['Player', 'Year', 'Tm'], ascending = [True, True, sorter])
```
It still has ATL at the top, meaning that it sorted alphabetically and not according to my custom list. Any help would really be greatly appreciated, I just can't figure this out.

Response:
The below answer is an old answer. It still works. Anyhow, another very elegant solution has been posted below , using the key argument. I just discovered that with pandas 15.1 it is possible to use categorical series (https://pandas.pydata.org/docs/user_guide/categorical.html) As for your example, lets define the same data-frame and sorter:
```
import pandas as pd

data = {
    'id': [2967, 5335, 13950, 6141, 6169],
    'Player': ['Cedric Hunter', 'Maurice Baker',
               'Ratko Varda' ,'Ryan Bowen' ,'Adrian Caldwell'],
    'Year': [1991, 2004, 2001, 2009, 1997],
    'Age': [27, 25, 22, 34, 31],
    'Tm': ['CHH', 'VAN', 'TOT', 'OKC', 'DAL'],
    'G': [6, 7, 60, 52, 81]
}

# Create DataFrame
df = pd.DataFrame(data)

# Define the sorter
sorter = ['TOT', 'ATL', 'BOS', 'BRK', 'CHA', 'CHH', 'CHI', 'CLE', 'DAL', 'DEN',
          'DET', 'GSW', 'HOU', 'IND', 'LAC', 'LAL', 'MEM', 'MIA', 'MIL',
          'MIN', 'NJN', 'NOH', 'NOK', 'NOP', 'NYK', 'OKC', 'ORL', 'PHI',
          'PHO', 'POR', 'SAC', 'SAS', 'SEA', 'TOR', 'UTA', 'VAN', 'WAS', 'WSB']
```
With the data-frame and sorter, which is a category-order, we can do the following in pandas 15.1:
```
# Convert Tm-column to category and in set the sorter as categories hierarchy
# You could also do both lines in one just appending the cat.set_categories()
df.Tm = df.Tm.astype("category")
df.Tm = df.Tm.cat.set_categories(sorter)

print(df.Tm)
Out[48]:
0    CHH
1    VAN
2    TOT
3    OKC
4    DAL
Name: Tm, dtype: category
Categories (38, object): [TOT < ATL < BOS < BRK ... UTA < VAN < WAS < WSB]

df.sort_values(["Tm"])  ## 'sort' changed to 'sort_values'
Out[49]:
   Age   G           Player   Tm  Year     id
2   22  60      Ratko Varda  TOT  2001  13950
0   27   6    Cedric Hunter  CHH  1991   2967
4   31  81  Adrian Caldwell  DAL  1997   6169
3   34  52       Ryan Bowen  OKC  2009   6141
1   25   7    Maurice Baker  VAN  2004   5335
```


Question:
Pandas - Replace values based on index If I create a dataframe like so:
```
import pandas as pd, numpy as np

df = pd.DataFrame(np.random.randint(0,100,size=(100, 2)), columns=list('AB'))
```
How would I change the entry in column A to be the number 16 from row 0 -15, for example? In other words, how do I replace cells based purely on index?

Response:
Use loc:
```
df.loc[0:15,'A'] = 16
print (df)
     A   B
0   16  45
1   16   5
2   16  97
3   16  58
4   16  26
5   16  87
6   16  51
7   16  17
8   16  39
9   16  73
10  16  94
11  16  69
12  16  57
13  16  24
14  16  43
15  16  77
16  41   0
17   3  21
18   0  98
19  45  39
20  66  62
21   8  53
22  69  47
23  48  53
```
Solution with ix is deprecated.


Question:
Python Pandas add column for row-wise max value of selected columns [duplicate] This question already has answers here: Find the max of two or more columns with pandas (4 answers) Closed 6 years ago.
```
data = {'name' : ['bill', 'joe', 'steve'],
    'test1' : [85, 75, 85],
    'test2' : [35, 45, 83],
     'test3' : [51, 61, 45]}
frame = pd.DataFrame(data)
```
I would like to add a new column that shows the max value for each row. desired output:
```
name test1 test2 test3 HighScore
 bill  75    75    85    85
 joe   35    45    83    83
 steve  51   61    45    61
```
Sometimes
```
frame['HighScore'] = max(data['test1'], data['test2'], data['test3'])
```
works but most of the time gives this error: ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all() Why does it only work sometimes? Is there another way of doing it?

Response:
```

```python
frame['HighScore'] = frame[['test1','test2','test3']].max(axis=1)
```

```python
frame
#Output
#    name  test1  test2  test3  HighScore
#0   bill     85     35     51         85
#1    joe     75     45     61         75
#2  steve     85     83     45         85
#```
```


Question:
How to add a column with values 1 to len(df) to a dataframe The index that I have in the dataframe (with 30 rows) is of the form:
```
Int64Index([171, 174, 173, 172, 199, …, 175, 200])
```
The index is not strictly increasing because the data frame is the output of a sort(). I want to add a column which is the series:
```
[1, 2, 3, 4, 5, …, 30]
```
How should I go about doing that?

Response:
How about:
```
df['new_col'] = range(1, len(df) + 1)
```
Alternatively if you want the index to be the ranks and store the original index as a column:
```
df = df.reset_index()
```


Question:
How do I assign values based on multiple conditions for existing columns? I would like to create a new column with a numerical value based on the following conditions: a. if gender is male & pet1==pet2, points = 5 b. if gender is female & (pet1 is 'cat' or pet1 is 'dog'), points = 5 c. all other combinations, points = 0
```none
gender    pet1      pet2
0   male      dog       dog
1   male      cat       cat
2   male      dog       cat
3   female    cat       squirrel
4   female    dog       dog
5   female    squirrel  cat
6   squirrel  dog       cat
```
I would like the end result to be as follows:
```none
gender    pet1      pet2      points
0   male      dog       dog       5
1   male      cat       cat       5
2   male      dog       cat       0
3   female    cat       squirrel  5
4   female    dog       dog       5
5   female    squirrel  cat       0
6   squirrel  dog       cat       0
```
How do I accomplish this?

Response:
numpy.select This is a perfect case for np.select where we can create a column based on multiple conditions and it's a readable method when there are more conditions:
```
conditions = [
    df['gender'].eq('male') & df['pet1'].eq(df['pet2']),
    df['gender'].eq('female') & df['pet1'].isin(['cat', 'dog'])
]

choices = [5,5]

df['points'] = np.select(conditions, choices, default=0)

print(df)
     gender      pet1      pet2  points
0      male       dog       dog       5
1      male       cat       cat       5
2      male       dog       cat       0
3    female       cat  squirrel       5
4    female       dog       dog       5
5    female  squirrel       cat       0
6  squirrel       dog       cat       0
```


Question:
pyspark show dataframe as table with horizontal scroll in ipython notebook a pyspark.sql.DataFrame displays messy with DataFrame.show() - lines wrap instead of a scroll. but displays with pandas.DataFrame.head I tried these options
```
import IPython
IPython.auto_scroll_threshold = 9999

from IPython.core.interactiveshell import InteractiveShell
InteractiveShell.ast_node_interactivity = "all"
from IPython.display import display
```
but no luck. Although the scroll works when used within Atom editor with jupyter plugin:

Response:
Just add (and execute)
```
from IPython.core.display import HTML
display(HTML("<style>pre { white-space: pre !important; }</style>"))
```
And you'll get the df.show() with the scrollbar


Question:
pandas assign with new column name as string I recently discovered pandas "assign" method which I find very elegant. My issue is that the name of the new column is assigned as keyword, so it cannot have spaces or dashes in it.
```
df = DataFrame({'A': range(1, 11), 'B': np.random.randn(10)})
df.assign(ln_A=lambda x: np.log(x.A))
        A         B      ln_A
0   1  0.426905  0.000000
1   2 -0.780949  0.693147
2   3 -0.418711  1.098612
3   4 -0.269708  1.386294
4   5 -0.274002  1.609438
5   6 -0.500792  1.791759
6   7  1.649697  1.945910
7   8 -1.495604  2.079442
8   9  0.549296  2.197225
9  10 -0.758542  2.302585
```
but what if I want to name the new column "ln(A)" for example? E.g.
```
df.assign(ln(A) = lambda x: np.log(x.A))
df.assign("ln(A)" = lambda x: np.log(x.A))

File "<ipython-input-7-de0da86dce68>", line 1
df.assign(ln(A) = lambda x: np.log(x.A))
SyntaxError: keyword can't be an expression
```
I know I could rename the column right after the .assign call, but I want to understand more about this method and its syntax.

Response:
You can pass the keyword arguments to assign as a dictionary, like so:
```
kwargs = {"ln(A)" : lambda x: np.log(x.A)}
df.assign(**kwargs)

    A         B     ln(A)
0   1  0.500033  0.000000
1   2 -0.392229  0.693147
2   3  0.385512  1.098612
3   4 -0.029816  1.386294
4   5 -2.386748  1.609438
5   6 -1.828487  1.791759
6   7  0.096117  1.945910
7   8 -2.867469  2.079442
8   9 -0.731787  2.197225
9  10 -0.686110  2.302585
```


Question:
Check if string is in a pandas dataframe I would like to see if a particular string exists in a particular column within my dataframe. I'm getting the error ValueError: The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all().
```
import pandas as pd

BabyDataSet = [('Bob', 968), ('Jessica', 155), ('Mary', 77), ('John', 578), ('Mel', 973)]

a = pd.DataFrame(data=BabyDataSet, columns=['Names', 'Births'])

if a['Names'].str.contains('Mel'):
    print ("Mel is there")
```

Response:
a['Names'].str.contains('Mel') will return an indicator vector of boolean values of size len(BabyDataSet) Therefore, you can use
```
mel_count=a['Names'].str.contains('Mel').sum()
if mel_count>0:
    print ("There are {m} Mels".format(m=mel_count))
```
Or any(), if you don't care how many records match your query
```
if a['Names'].str.contains('Mel').any():
    print ("Mel is there")
```


Question:
Calculating difference between two rows in Python / Pandas In python, how can I reference previous row and calculate something against it? Specifically, I am working with dataframes in pandas - I have a data frame full of stock price information that looks like this:
```
Date   Close  Adj Close
251  2011-01-03  147.48     143.25
250  2011-01-04  147.64     143.41
249  2011-01-05  147.05     142.83
248  2011-01-06  148.66     144.40
247  2011-01-07  147.93     143.69
```
Here is how I created this dataframe:
```
import pandas

url = 'http://ichart.finance.yahoo.com/table.csv?s=IBM&a=00&b=1&c=2011&d=11&e=31&f=2011&g=d&ignore=.csv'
data = data = pandas.read_csv(url)

## now I sorted the data frame ascending by date
data = data.sort(columns='Date')
```
Starting with row number 2, or in this case, I guess it's 250 (PS - is that the index?), I want to calculate the difference between 2011-01-03 and 2011-01-04, for every entry in this dataframe. I believe the appropriate way is to write a function that takes the current row, then figures out the previous row, and calculates the difference between them, the use the pandas apply function to update the dataframe with the value. Is that the right approach? If so, should I be using the index to determine the difference? (note - I'm still in python beginner mode, so index may not be the right term, nor even the correct way to implement this)

Response:
I think you want to do something like this:
```

```python
data
#Output
#Date   Close  Adj Close
#251  2011-01-03  147.48     143.25
#250  2011-01-04  147.64     143.41
#249  2011-01-05  147.05     142.83
#248  2011-01-06  148.66     144.40
#247  2011-01-07  147.93     143.69
```

```python
data.set_index('Date').diff()
#Output
#Close  Adj Close
#Date
#2011-01-03    NaN        NaN
#2011-01-04   0.16       0.16
#2011-01-05  -0.59      -0.58
#2011-01-06   1.61       1.57
#2011-01-07  -0.73      -0.71
#```
```


Question:
Convert Pandas dataframe to csv string Here is an example of what I am trying to get: I have:
```
import pandas as pd
df = pd.DataFrame({'A' : [0, 1], 'B' : [1, 6]})
```
My goal is:
```
',A,B\n0,0,1\n1,1,6\n'
```
I can achieve this with lazy and horrible:
```
df.to_csv('temp.csv') # create unnecessary file
body = open('temp.csv').read()
```
Also to_string() methods looks very promising; however, the best I can come up with is this:
```
body = df.to_string()[1:].replace('  ', ',') + '\n'
```
This does not create an unnecessary file, but seems sloppy and perhaps not very reliable. Am I missing a simpler solution?

Response:
The simplest way is just to not input any filename, in this case a string is returned:
```

```python
df = pd.DataFrame({'A' : [0, 1], 'B' : [1, 6]})
```

```python
df.to_csv()
#Output
#',A,B\n0,0,1\n1,1,6\n'
#```
```


Question:
Pandas dataframe to json without index I'm trying to take a dataframe and transform it into a particular json format. Here's my dataframe example:
```
DataFrame name: Stops
id    location
0     [50, 50]
1     [60, 60]
2     [70, 70]
3     [80, 80]
```
Here's the json format I'd like to transform into:
```
"stops":
[
{
    "id": 1,
    "location": [50, 50]
},
{
    "id": 2,
    "location": [60, 60]
},
... (and so on)
]
```
Notice it's a list of dicts. I have it nearly there with the following code: df.reset_index().to_json(orient='index) However, that line also includes the index like this:
```
"stops":
{
"0":
    {
        "id": 0,
        "location": [50, 50]
    },
"1":
    {
        "id": 1,
        "location": [60, 60]
    },
... (and so on)
}
```
Notice this is a dict of dicts and also includes the index twice (in the first dict and as the "id" in the second dict! Any help would be appreciated.

Response:
You can use orient='records'
```
print df.reset_index().to_json(orient='records')

[
     {"id":0,"location":"[50, 50]"},
     {"id":1,"location":"[60, 60]"},
     {"id":2,"location":"[70, 70]"},
     {"id":3,"location":"[80, 80]"}
]
```


Question:
Use Pandas groupby() + apply() with arguments I would like to use df.groupby() in combination with apply() to apply a function to each row per group. I normally use the following code, which usually works (note, that this is without groupby()):
```
df.apply(myFunction, args=(arg1,))
```
With the groupby() I tried the following:
```
df.groupby('columnName').apply(myFunction, args=(arg1,))
```
However, I get the following error:
```none
TypeError: myFunction() got an unexpected keyword argument 'args'
```
Hence, my question is: How can I use groupby() and apply() with a function that needs arguments?

Response:
pandas.core.groupby.GroupBy.apply does NOT have named parameter args, but pandas.DataFrame.apply does have it. So try this:
```
df.groupby('columnName').apply(lambda x: myFunction(x, arg1))
```
or as suggested by @Zero:
```
df.groupby('columnName').apply(myFunction, ('arg1'))
```
Demo:
```

```python
df = pd.DataFrame(np.random.randint(5,size=(5,3)), columns=list('abc'))
```

```python
df
#Output
#a  b  c
#0  0  3  1
#1  0  3  4
#2  3  0  4
#3  4  2  3
#4  3  4  1
```

```python
def f(ser, n):
    ...:     return ser.max() * n
    ...:
```

```python
df.apply(f, args=(10,))
#Output
#a    40
#b    40
#c    40
#dtype: int64
#```
#when using GroupBy.apply you can pass either a named arguments:
#```
```

```python
df.groupby('a').apply(f, n=10)
#Output
#a   b   c
#a
#0   0  30  40
#3  30  40  40
#4  40  20  30
#```
#a tuple of arguments:
#```
```

```python
df.groupby('a').apply(f, (10))
#Output
#a   b   c
#a
#0   0  30  40
#3  30  40  40
#4  40  20  30
#```
```


Question:
Add x and y labels to a pandas plot Suppose I have the following code that plots something very simple using pandas:
```
import pandas as pd
values = [[1, 2], [2, 5]]
df2 = pd.DataFrame(values, columns=['Type A', 'Type B'],
                   index=['Index 1', 'Index 2'])
df2.plot(lw=2, colormap='jet', marker='.', markersize=10,
         title='Video streaming dropout by category')
```
How do I easily set x and y-labels while preserving my ability to use specific colormaps? I noticed that the plot() wrapper for pandas DataFrames doesn't take any parameters specific for that.

Response:
The df.plot() function returns a matplotlib.axes.AxesSubplot object. You can set the labels on that object.
```
ax = df2.plot(lw=2, colormap='jet', marker='.', markersize=10, title='Video streaming dropout by category')
ax.set_xlabel("x label")
ax.set_ylabel("y label")
```
Or, more succinctly: ax.set(xlabel="x label", ylabel="y label"). Alternatively, the index x-axis label is automatically set to the Index name, if it has one. so df2.index.name = 'x label' would work too.