Question: Pandas: Combining Two DataFrames Horizontally [duplicate] This question already has answers here: Concatenate rows of two dataframes in pandas (3 answers) Closed 8 years ago. I have two Pandas DataFrames, each with different columns. I want to basically glue them together horizontally (they each have the same number of rows so this shouldn't be an issue). There must be a simple way of doing this but I've gone through the docs and concat isn't what I'm looking for (I don't think). Any ideas? Thanks!

Response:
concat is indeed what you're looking for, you just have to pass it a different value for the "axis" argument than the default. Code sample below:
```
import pandas as pd

df1 = pd.DataFrame({
    'A': [1,2,3,4,5],
    'B': [1,2,3,4,5]
})

df2 = pd.DataFrame({
    'C': [1,2,3,4,5],
    'D': [1,2,3,4,5]
})

df_concat = pd.concat([df1, df2], axis=1)

print(df_concat)
```
With the result being:
```
A  B  C  D
0  1  1  1  1
1  2  2  2  2
2  3  3  3  3
3  4  4  4  4
4  5  5  5  5
```


Question:
Filter Pandas DataFrame by time index I have a pandas DataFrame from 6:36 AM to 5:31 PM. I want to remove all observations where the time is less than 8:00:00 AM. Here is my attempt:
```
df = df[df.index < '2013-10-16 08:00:00']
```
This does nothing, please help.

Response: You want df.loc[df.index < '2013-10-16 08:00:00'] since you're selecting by label (index) and not by value. selecting by label


Question:
transform scipy sparse csr to pandas? I have used the
```
sklearn.preprocessing.OneHotEncoder
```
to transform some data the output is scipy.sparse.csr.csr_matrix how can I merge it back into my original dataframe along with the other columns? I tried to use pd.concat but I get
```
TypeError: cannot concatenate a non-NDFrame object
```
Thanks

Response:
If A is csr_matrix, you can use .toarray() (there's also .todense() that produces a numpy matrix, which is also works for the DataFrame constructor):
```
df = pd.DataFrame(A.toarray())
```
You can then use this with pd.concat().
```
A = csr_matrix([[1, 0, 2], [0, 3, 0]])

  (0, 0)    1
  (0, 2)    2
  (1, 1)    3

<class 'scipy.sparse.csr.csr_matrix'>

pd.DataFrame(A.todense())

   0  1  2
0  1  0  2
1  0  3  0

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 2 entries, 0 to 1
Data columns (total 3 columns):
0    2 non-null int64
1    2 non-null int64
2    2 non-null int64
```
In version 0.20, pandas introduced sparse data structures, including the SparseDataFrame. In pandas 1.0, SparseDataFrame was removed: In older versions of pandas, the SparseSeries and SparseDataFrame classes were the preferred way to work with sparse data. With the advent of extension arrays, these subclasses are no longer needed. Their purpose is better served by using a regular Series or DataFrame with sparse values instead. The migration guide shows how to use these new data structures. For instance, to create a DataFrame from a sparse matrix:
```
from scipy.sparse import csr_matrix

A = csr_matrix([[1, 0, 2], [0, 3, 0]])

df = pd.DataFrame.sparse.from_spmatrix(A, columns=['A', 'B', 'C'])

df

   A  B  C
0  1  0  2
1  0  3  0

df.dtypes
A    Sparse[float64, 0]
B    Sparse[float64, 0]
C    Sparse[float64, 0]
dtype: object
```
Alternatively, you can pass sparse matrices to sklearn to avoid running out of memory when converting back to pandas. Just convert your other data to sparse format by passing a numpy array to the scipy.sparse.csr_matrix constructor and use scipy.sparse.hstack to combine (see docs).


Question:
Pandas: Converting to numeric, creating NaNs when necessary Say I have a column in a dataframe that has some numbers and some non-numbers
```
>> df['foo']
0       0.0
1     103.8
2     751.1
3       0.0
4       0.0
5         -
6         -
7       0.0
8         -
9       0.0
Name: foo, Length: 9, dtype: object
```
How can I convert this column to np.float, and have everything else that is not float convert it to NaN? When I try:
```
>> df['foo'].astype(np.float)
```
or
```
>> df['foo'].apply(np.float)
```
I get ValueError: could not convert string to float: -

Response:
In pandas 0.17.0 convert_objects raises a warning: FutureWarning: convert_objects is deprecated. Use the data-type specific converters pd.to_datetime, pd.to_timedelta and pd.to_numeric. You could use pd.to_numeric method and apply it for the dataframe with arg coerce.
```
df1 = df.apply(pd.to_numeric, args=('coerce',))
```
or maybe more appropriately:
```
df1 = df.apply(pd.to_numeric, errors='coerce')
```
EDIT The above method is only valid for pandas version >= 0.17.0, from docs what's new in pandas 0.17.0: pd.to_numeric is a new function to coerce strings to numbers (possibly with coercion) (GH11133)


Question:
How to select a range of values in a pandas dataframe column?
```
import pandas as pd
import numpy as np
data = 'filename.csv'
df = pd.DataFrame(data)
df

        one       two     three  four   five
a  0.469112 -0.282863 -1.509059  bar   True
b  0.932424  1.224234  7.823421  bar  False
c -1.135632  1.212112 -0.173215  bar  False
d  0.232424  2.342112  0.982342  unbar True
e  0.119209 -1.044236 -0.861849  bar   True
f -2.104569 -0.494929  1.071804  bar  False
```
I would like to select a range for a certain column, let's say column two. I would like to select all values between -0.5 and +0.5. How does one do this? I expected to use
```
-0.5 < df["two"] < 0.5
```
But this (naturally) gives a ValueError:
```
ValueError: The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all().
```
I tried
```
-0.5 (< df["two"] < 0.5)
```
But this outputs all True. The correct output should be
```
0    True
1    False
2    False
3    False
4    False
5    True
```
What is the correct way to find a range of values in a pandas dataframe column? EDIT: Question Using .between() with
```
df['two'].between(-0.5, 0.5, inclusive=False)
```
would would be the difference between
```
-0.5 < df['two'] < 0.5
```
and inequalities like
```
-0.5 =< df['two'] < 0.5
```
?

Response:
Use between with inclusive=False for strict inequalities:
```
df['two'].between(-0.5, 0.5, inclusive=False)
```
The inclusive parameter determines if the endpoints are included or not (True: <=, False: <). This applies to both signs. If you want mixed inequalities, you'll need to code them explicitly:
```
(df['two'] >= -0.5) & (df['two'] < 0.5)
```


Question:
Add column of empty lists to DataFrame Similar to this question How to add an empty column to a dataframe?, I am interested in knowing the best way to add a column of empty lists to a DataFrame. What I am trying to do is basically initialize a column and as I iterate over the rows to process some of them, then add a filled list in this new column to replace the initialized value. For example, if below is my initial DataFrame:
```
df = pd.DataFrame(d = {'a': [1,2,3], 'b': [5,6,7]}) # Sample DataFrame

```python
df
#Output
#   a  b
#0  1  5
#1  2  6
#2  3  7
#```
#Then I want to ultimately end up with something like this, where each row has been processed separately (sample results shown):
#```
```

```python
df
#Output
#   a  b          c
#0  1  5     [5, 6]
#1  2  6     [9, 0]
#2  3  7  [1, 2, 3]
#```
#Of course, if I try to initialize like df['e'] = [] as I would with any other constant, it thinks I am trying to add a sequence of items with length 0, and hence fails. If I try initializing a new column as None or NaN, I run in to the following issues when trying to assign a list to a location.
#```
#df['d'] = None
```

```python
df
#Output
#   a  b     d
#0  1  5  None
#1  2  6  None
#2  3  7  None
#```
#Issue 1 (it would be perfect if I can get this approach to work! Maybe something trivial I am missing):
#```
```

```python
df.loc[0,'d'] = [1,3]
```

...
ValueError: Must have equal len keys and value when setting with an iterable
```
Issue 2 (this one works, but not without a warning because it is not guaranteed to work as intended):
```

```python
df['d'][0] = [1,3]
```

C:\Python27\Scripts\ipython:1: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame
```
Hence I resort to initializing with empty lists and extending them as needed. There are a couple of methods I can think of to initialize this way, but is there a more straightforward way? Method 1:
```
df['empty_lists1'] = [list() for x in range(len(df.index))]

```python
df
#Output
#   a  b   empty_lists1
#0  1  5             []
#1  2  6             []
#2  3  7             []
#```
#Method 2:
#```
#df['empty_lists2'] = df.apply(lambda x: [], axis=1)
```

```python
df
#Output
#   a  b   empty_lists1   empty_lists2
#0  1  5             []             []
#1  2  6             []             []
#2  3  7             []             []
#```
#Summary of questions: Is there any minor syntax change that can be addressed in Issue 1 that can allow a list to be assigned to a None/NaN initialized field? If not, then what is the best way to initialize a new column with empty lists?
```

Response:
One more way is to use np.empty:
```
df['empty_list'] = np.empty((len(df), 0)).tolist()
```
You could also knock off .index in your "Method 1" when trying to find len of df.
```
df['empty_list'] = [[] for _ in range(len(df))]
```
Turns out, np.empty is faster...
```

```python
import pandas as pd
```

```python
df = pd.DataFrame(pd.np.random.rand(1000000, 5))
```

```python
timeit df['empty1'] = pd.np.empty((len(df), 0)).tolist()
10 loops, best of 3: 127 ms per loop
```

```python
timeit df['empty2'] = [[] for _ in range(len(df))]
10 loops, best of 3: 193 ms per loop
```

```python
timeit df['empty3'] = df.apply(lambda x: [], axis=1)
1 loops, best of 3: 5.89 s per loop
```
```


Question:
Convert Pandas dataframe to Sparse Numpy Matrix directly I am creating a matrix from a Pandas dataframe as follows:
```
dense_matrix = np.array(df.as_matrix(columns = None), dtype=bool).astype(np.int)
```
And then into a sparse matrix with:
```
sparse_matrix = scipy.sparse.csr_matrix(dense_matrix)
```
Is there any way to go from a df straight to a sparse matrix? Thanks in advance.

Response:
df.values is a numpy array, and accessing values that way is always faster than np.array.
```
scipy.sparse.csr_matrix(df.values)
```
You might need to take the transpose first, like df.values.T. In DataFrames, the columns are axis 0.


Question: Python - Turn all items in a Dataframe to strings I followed the following procedure: In Python, how do I convert all of the items in a list to floats? because each column of my Dataframe is list, but instead of floats I chose to change all the values to strings. df = [str(i) for i in df] But this failed. It simply erased all the data except for the first row of column names. Then, trying df = [str(i) for i in df.values] resulted in changing the entire Dataframe into one big list, but that messes up the data too much to be able to meet the goal of my script which is to export the Dataframe to my Oracle table. Is there a way to convert all the items that are in my Dataframe that are NOT strings into strings?

Response:
You can use this:
```
df = df.astype(str)
```
out of curiosity I decided to see if there is any difference in efficiency between the accepted solution and mine. The results are below: example df:
```
df = pd.DataFrame([list(range(1000))], index=[0])
```
test df.astype:
```
%timeit df.astype(str)
>> 100 loops, best of 3: 2.18 ms per loop
```
test df.applymap:
```
%timeit df.applymap(str)
1 loops, best of 3: 245 ms per loop
```
It seems df.astype is quite a lot faster :)


Question: Convert percent string to float in pandas read_csv Is there a way to convert values like '34%' directly to int or float when using read_csv() command in pandas? I want '34%' to be directly read as 0.34 Using this in read_csv did not work: read_csv(..., dtype={'col':np.float}) After loading the csv as 'df' this also did not work with the error "invalid literal for float(): 34%" df['col'] = df['col'].astype(float) I ended up using this which works but is long winded: df['col'] = df['col'].apply(lambda x: np.nan if x in ['-'] else x[:-1]).astype(float)/100

Response:
You were very close with your df attempt. Try changing:
```
df['col'] = df['col'].astype(float)
```
to:
```
df['col'] = df['col'].str.rstrip('%').astype('float') / 100.0
#                     ^ use str funcs to elim '%'     ^ divide by 100
# could also be:     .str[:-1].astype(...
```
Pandas supports Python's string processing functions on string columns. Just precede the string function you want with .str and see if it does what you need. (This includes string slicing, too, of course.) Above we utilize .str.rstrip() to get rid of the trailing percent sign, then we divide the array in its entirety by 100.0 to convert from percentage to actual value. For example, 45% is equivalent to 0.45. Although .str.rstrip('%') could also just be .str[:-1], I prefer to explicitly remove the '%' rather than blindly removing the last char, just in case...


Question:
change multiple columns in pandas dataframe to datetime I have a dataframe of 13 columns and 55,000 rows I am trying to convert 5 of those rows to datetime, right now they are returning the type 'object' and I need to transform this data for machine learning I know that if I do
```
data['birth_date'] = pd.to_datetime(data[birth_date], errors ='coerce')
```
it will return a datetime column but I want to do it for 4 other columns as well, is there one line that I can write to call all of them? I dont think I can index like
```
data[:,7:12]
```
thanks!

Response:
You can use apply to iterate through each column using pd.to_datetime
```
data.iloc[:, 7:12] = data.iloc[:, 7:12].apply(pd.to_datetime, errors='coerce')
```
As part of the changes in pandas 1.3.0, iloc/loc will no longer update the column dtype on assignment. Use column labels directly instead:
```
cols = data.columns[7:12]
data[cols] = data[cols].apply(pd.to_datetime, errors='coerce')
```


Question:
Truncate `TimeStamp` column to hour precision in pandas `DataFrame` I have a pandas.DataFrame called df which has an automatically generated index, with a column dt:
```
df['dt'].dtype, df['dt'][0]
# (dtype('<M8[ns]'), Timestamp('2014-10-01 10:02:45'))
```
What I'd like to do is create a new column truncated to hour precision. I'm currently using:
```
df['dt2'] = df['dt'].apply(lambda L: datetime(L.year, L.month, L.day, L.hour))
```
This works, so that's fine. However, I've an inkling there's some nice way using pandas.tseries.offsets or creating a DatetimeIndex or similar. So if possible, is there some pandas wizardry to do this?

Response:
In pandas 0.18.0 and later, there are datetime floor, ceil and round methods to round timestamps to a given fixed precision/frequency. To round down to hour precision, you can use:
```

```python
df['dt2'] = df['dt'].dt.floor('h')
```

```python
df
#Output
#                      dt                     dt2
#0    2014-10-01 10:02:45     2014-10-01 10:00:00
#1    2014-10-01 13:08:17     2014-10-01 13:00:00
#2    2014-10-01 17:39:24     2014-10-01 17:00:00
#```
#Here's another alternative to truncate the timestamps. Unlike floor, it supports truncating to a precision such as year or month. You can temporarily adjust the precision unit of the underlying NumPy datetime64 datatype, changing it from [ns] to [h]:
#```
#df['dt'].values.astype('<M8[h]')
#```
#This truncates everything to hour precision. For example:
#```
```

```python
df
#Output
#                       dt
#0     2014-10-01 10:02:45
#1     2014-10-01 13:08:17
#2     2014-10-01 17:39:24
```

```python
df['dt2'] = df['dt'].values.astype('<M8[h]')
```

```python
df
#Output
#                      dt                     dt2
#0    2014-10-01 10:02:45     2014-10-01 10:00:00
#1    2014-10-01 13:08:17     2014-10-01 13:00:00
#2    2014-10-01 17:39:24     2014-10-01 17:00:00
```

```python
df.dtypes
#Output
#dt     datetime64[ns]
#dt2    datetime64[ns]
#```
#The same method should work for any other unit: months 'M', minutes 'm', and so on: Keep up to year: '<M8[Y]' Keep up to month: '<M8[M]' Keep up to day: '<M8[D]' Keep up to minute: '<M8[m]' Keep up to second: '<M8[s]'
```


Question:
Reset a column's MultiIndex levels Is there a shorter way of dropping a column MultiIndex level (in my case, basic_amt) except transposing it twice?
```

```python
test
#Output
#basic_amt
#Faculty          NSW  QLD  VIC  All
#All                1    1    2    4
#Full Time          0    1    0    1
#Part Time          1    0    2    3
```

```python
test.reset_index(level=0, drop=True)
#Output
#basic_amt
#Faculty        NSW  QLD  VIC  All
#0                1    1    2    4
#1                0    1    0    1
#2                1    0    2    3
```

```python
test.transpose().reset_index(level=0, drop=True).transpose()
#Output
#Faculty    NSW  QLD  VIC  All
#All          1    1    2    4
#Full Time    0    1    0    1
#Part Time    1    0    2    3
#```
```

Response:
Another solution is to use MultiIndex.droplevel with rename_axis (new in pandas 0.18.0):
```
import pandas as pd

cols = pd.MultiIndex.from_arrays([['basic_amt']*4,
                                     ['NSW','QLD','VIC','All']],
                                     names = [None, 'Faculty'])
idx = pd.Index(['All', 'Full Time', 'Part Time'])

df = pd.DataFrame([(1,1,2,4),
                   (0,1,0,1),
                   (1,0,2,3)], index = idx, columns=cols)

print (df)
          basic_amt
Faculty         NSW QLD VIC All
All               1   1   2   4
Full Time         0   1   0   1
Part Time         1   0   2   3

df.columns = df.columns.droplevel(0)
#pandas 0.18.0 and higher
df = df.rename_axis(None, axis=1)
#pandas bellow 0.18.0
#df.columns.name = None

print (df)
           NSW  QLD  VIC  All
All          1    1    2    4
Full Time    0    1    0    1
Part Time    1    0    2    3

print (df.columns)
Index(['NSW', 'QLD', 'VIC', 'All'], dtype='object')
```
If you need both column names, use list comprehension:
```
df.columns = ['_'.join(col) for col in df.columns]
print (df)
           basic_amt_NSW  basic_amt_QLD  basic_amt_VIC  basic_amt_All
All                    1              1              2              4
Full Time              0              1              0              1
Part Time              1              0              2              3

print (df.columns)
Index(['basic_amt_NSW', 'basic_amt_QLD', 'basic_amt_VIC', 'basic_amt_All'], dtype='object')
```


Question:
How can I use cumsum within a group in Pandas? I have
```py
df = pd.DataFrame.from_dict({'id': ['A', 'B', 'A', 'C', 'D', 'B', 'C'], 'val': [1,2,-3,1,5,6,-2], 'stuff':['12','23232','13','1234','3235','3236','732323']})

  id   stuff  val
0  A      12    1
1  B   23232    2
2  A      13   -3
3  C    1234    1
4  D    3235    5
5  B    3236    6
6  C  732323   -2
```
I'd like to get a running sum of val for each id, so the desired output looks like this:
```none
id   stuff  val  cumsum
0  A      12    1   1
1  B   23232    2   2
2  A      13   -3   -2
3  C    1234    1   1
4  D    3235    5   5
5  B    3236    6   8
6  C  732323   -2  -1
```
This is what I tried:
```py
df['cumsum'] = df.groupby('id').cumsum(['val'])
```
This is the error I get:
```none
ValueError: Wrong number of items passed 0, placement implies 1
```

Response:
You can call transform and pass the cumsum function to add that column to your df:
```

```python
df['cumsum'] = df.groupby('id')['val'].transform(pd.Series.cumsum)
df
#Output
#id   stuff  val  cumsum
#0  A      12    1       1
#1  B   23232    2       2
#2  A      13   -3      -2
#3  C    1234    1       1
#4  D    3235    5       5
#5  B    3236    6       8
#6  C  732323   -2      -1
#```
#With respect to your error, you can't call cumsum on a Series groupby object, secondly you're passing the name of the column as a list which is meaningless. So this works:
#```
```

```python
df.groupby('id')['val'].cumsum()
#Output
#0    1
#1    2
#2   -2
#3    1
#4    5
#5    8
#6   -1
#dtype: int64
#```
```


Question:
Pandas sort by group aggregate and column Given the following dataframe
```

```python
rand = np.random.RandomState(1)
         df = pd.DataFrame({'A': ['foo', 'bar', 'baz'] * 2,
                            'B': rand.randn(6),
                            'C': rand.rand(6) > .5})
```

```python
df
#Output
#A         B      C
#         0  foo  1.624345  False
#         1  bar -0.611756   True
#         2  baz -0.528172  False
#         3  foo -1.072969   True
#         4  bar  0.865408  False
#         5  baz -2.301539   True
#```
#I would like to sort it in groups (A) by the aggregated sum of B, and then by the value in C (not aggregated). So basically get the order of the A groups with
#```
```

```python
df.groupby('A').sum().sort('B')
#Output
#B  C
#         A
#         baz -2.829710  1
#         bar  0.253651  1
#         foo  0.551377  1
#```
#And then by True/False, so that it ultimately looks like this:
#```
```

```python
df.ix[[5, 2, 1, 4, 3, 0]]
#Output
#A         B      C
#    5  baz -2.301539   True
#    2  baz -0.528172  False
#    1  bar -0.611756   True
#    4  bar  0.865408  False
#    3  foo -1.072969   True
#    0  foo  1.624345  False
#```
#How can this be done?
```

Response:
Groupby A:
```

```python
grp = df.groupby('A')
```
Within each group, sum over B and broadcast the values using transform. Then sort by B:
```
```

```python
grp[['B']].transform(sum).sort('B')
#Output
#B
#2 -2.829710
#5 -2.829710
#1  0.253651
#4  0.253651
#0  0.551377
#3  0.551377
#```
#Index the original df by passing the index from above. This will re-order the A values by the aggregate sum of the B values:
#```
```

```python
sort1 = df.ix[grp[['B']].transform(sum).sort('B').index]
```

```python
sort1
#Output
#A         B      C
#2  baz -0.528172  False
#5  baz -2.301539   True
#1  bar -0.611756   True
#4  bar  0.865408  False
#0  foo  1.624345  False
#3  foo -1.072969   True
#```
#Finally, sort the 'C' values within groups of 'A' using the sort=False option to preserve the A sort order from step 1:
#```
```

```python
f = lambda x: x.sort('C', ascending=False)
```

```python
sort2 = sort1.groupby('A', sort=False).apply(f)
```

```python
sort2
#Output
#A         B      C
#A
#baz 5  baz -2.301539   True
#    2  baz -0.528172  False
#bar 1  bar -0.611756   True
#    4  bar  0.865408  False
#foo 3  foo -1.072969   True
#    0  foo  1.624345  False
#```
#Clean up the df index by using reset_index with drop=True:
#```
```

```python
sort2.reset_index(0, drop=True)
#Output
#A         B      C
#5  baz -2.301539   True
#2  baz -0.528172  False
#1  bar -0.611756   True
#4  bar  0.865408  False
#3  foo -1.072969   True
#0  foo  1.624345  False
#```
```


Question:
Melt the Upper Triangular Matrix of a Pandas Dataframe Given a square pandas DataFrame of the following form:
```
a  b  c
a  1 .5 .3
b .5  1 .4
c .3 .4  1
```
How can the upper triangle be melted to get a matrix of the following form
```
Row     Column    Value
  a        a       1
  a        b       .5
  a        c       .3
  b        b       1
  b        c       .4
  c        c       1

#Note the combination a,b is only listed once.  There is no b,a listing
```
I'm more interested in an idiomatic pandas solution, a custom indexer would be easy enough to write by hand... Thank you in advance for your consideration and response.

Response:
First I convert lower values of df to NaN by where and numpy.triu and then stack, reset_index and set column names:
```
import numpy as np

print df
     a    b    c
a  1.0  0.5  0.3
b  0.5  1.0  0.4
c  0.3  0.4  1.0

print np.triu(np.ones(df.shape)).astype(np.bool)
[[ True  True  True]
 [False  True  True]
 [False False  True]]

df = df.where(np.triu(np.ones(df.shape)).astype(np.bool))
print df
    a    b    c
a   1  0.5  0.3
b NaN  1.0  0.4
c NaN  NaN  1.0

df = df.stack().reset_index()
df.columns = ['Row','Column','Value']
print df

  Row Column  Value
0   a      a    1.0
1   a      b    0.5
2   a      c    0.3
3   b      b    1.0
4   b      c    0.4
5   c      c    1.0
```


Question:
Pandas : compute mean or std (standard deviation) over entire dataframe Here is my problem, I have a dataframe like this :
```
Depr_1  Depr_2  Depr_3
S3  0   5   9
S2  4   11  8
S1  6   11  12
S5  0   4   11
S4  4   8   8
```
and I just want to calculate the mean over the full dataframe, as the following doesn't work :
```
df.mean()
```
Then I came up with :
```
df.mean().mean()
```
But this trick won't work for computing the standard deviation. My final attempts were :
```
df.get_values().mean()
df.get_values().std()
```
Except that in the latter case, it uses mean() and std() function from numpy. It's not a problem for the mean, but it is for std, as the pandas function uses by default ddof=1, unlike the numpy one where ddof=0.

Response:
You could convert the dataframe to be a single column with stack (this changes the shape from 5x3 to 15x1) and then take the standard deviation:
```
df.stack().std()         # pandas default degrees of freedom is one
```
Alternatively, you can use values to convert from a pandas dataframe to a numpy array before taking the standard deviation:
```
df.values.std(ddof=1)    # numpy default degrees of freedom is zero
```
Unlike pandas, numpy will give the standard deviation of the entire array by default, so there is no need to reshape before taking the standard deviation. A couple of additional notes: The numpy approach here is a bit faster than the pandas one, which is generally true when you have the option to accomplish the same thing with either numpy or pandas. The speed difference will depend on the size of your data, but numpy was roughly 10x faster when I tested a few different sized dataframes on my laptop (numpy version 1.15.4 and pandas version 0.23.4). The numpy and pandas approaches here will not give exactly the same answers, but will be extremely close (identical at several digits of precision). The discrepancy is due to slight differences in implementation behind the scenes that affect how the floating point values get rounded.


Question:
Check if string in one column is contained in string of another column in the same row I have a dataframe like this:
```none
RecID| A  |B
----------------
1    |a   | abc
2    |b   | cba
3    |c   | bca
4    |d   | bac
5    |e   | abc
```
I want to create another column, C, out of A and B such that for the same row, if the string in column A is contained in the string of column B, then C = True and if not then C = False. The example output I am looking for is this:
```none
RecID| A  |B    |C
--------------------
1    |a   | abc |True
2    |b   | cba |True
3    |c   | bca |True
4    |d   | bac |False
5    |e   | abc |False
```
Is there a way to do this in pandas quickly and without using a loop?

Response:
You need apply with in:
```
df['C'] = df.apply(lambda row: row.A in row.B, axis=1)
print(df)

   RecID  A    B      C
0      1  a  abc   True
1      2  b  cba   True
2      3  c  bca   True
3      4  d  bac  False
4      5  e  abc  False
```
Another solution with list comprehension is faster, but there has to be no NaNs:
```
df['C'] = [row[0] in row[1] for row in zip(df['A'], df['B'])]
print(df)

   RecID  A    B      C
0      1  a  abc   True
1      2  b  cba   True
2      3  c  bca   True
3      4  d  bac  False
4      5  e  abc  False
```


Question:
How to find out where a Python Warning is from I'm still kinda new with Python, using Pandas, and I've got some issues debugging my Python script. I've got the following warning message :
```
[...]\pandas\core\index.py:756: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal
return self._engine.get_loc(key)
```
And can't find where it's from. After some research, I tried to do that in the Pandas lib file (index.py):
```
try:
    return self._engine.get_loc(key)
except UnicodeWarning:
    warnings.warn('Oh Non', stacklevel=2)
```
But that didn't change anything about the warning message.

Response:
You can filter the warnings to raise which will enable you to debug (e.g. using pdb):
```
import warnings
warnings.filterwarnings('error')
```
*The warnings filter can be managed more finely (which is probably more appropriate) e.g.:
```
warnings.filterwarnings('error', category=UnicodeWarning)
warnings.filterwarnings('error', message='*equal comparison failed*')
```
Multiple filters will be looked up sequentially. ("Entries closer to the front of the list override entries later in the list, if both match a particular warning.")


Question:
Solution for SpecificationError: nested renamer is not supported while agg() along with groupby()
```
def stack_plot(data, xtick, col2='project_is_approved', col3='total'):
    ind = np.arange(data.shape[0])

    plt.figure(figsize=(20,5))
    p1 = plt.bar(ind, data[col3].values)
    p2 = plt.bar(ind, data[col2].values)

    plt.ylabel('Projects')
    plt.title('Number of projects aproved vs rejected')
    plt.xticks(ind, list(data[xtick].values))
    plt.legend((p1[0], p2[0]), ('total', 'accepted'))
    plt.show()

def univariate_barplots(data, col1, col2='project_is_approved', top=False):
    # Count number of zeros in dataframe python: https://stackoverflow.com/a/51540521/4084039
    temp = pd.DataFrame(project_data.groupby(col1)[col2].agg(lambda x: x.eq(1).sum())).reset_index()

    # Pandas dataframe grouby count: https://stackoverflow.com/a/19385591/4084039
    temp['total'] = pd.DataFrame(project_data.groupby(col1)[col2].agg({'total':'count'})).reset_index()['total']

    temp['Avg'] = pd.DataFrame(project_data.groupby(col1)[col2].agg({'Avg':'mean'})).reset_index()['Avg']

    temp.sort_values(by=['total'],inplace=True, ascending=False)

    if top:
        temp = temp[0:top]

    stack_plot(temp, xtick=col1, col2=col2, col3='total')
    print(temp.head(5))
    print("="*50)
    print(temp.tail(5))

univariate_barplots(project_data, 'school_state', 'project_is_approved', False)
```
Error:
```
SpecificationError                        Traceback (most recent call last)
<ipython-input-21-2cace8f16608> in <module>()
----> 1 univariate_barplots(project_data, 'school_state', 'project_is_approved', False)

<ipython-input-20-856fcc83737b> in univariate_barplots(data, col1, col2, top)
      4
      5     # Pandas dataframe grouby count: https://stackoverflow.com/a/19385591/4084039
----> 6     temp['total'] = pd.DataFrame(project_data.groupby(col1)[col2].agg({'total':'count'})).reset_index()['total']
      7     print (temp['total'].head(2))
      8     temp['Avg'] = pd.DataFrame(project_data.groupby(col1)[col2].agg({'Avg':'mean'})).reset_index()['Avg']

~\AppData\Roaming\Python\Python36\site-packages\pandas\core\groupby\generic.py in aggregate(self, func, *args, **kwargs)
    251             # but not the class list / tuple itself.
    252             func = _maybe_mangle_lambdas(func)
--> 253             ret = self._aggregate_multiple_funcs(func)
    254             if relabeling:
    255                 ret.columns = columns

~\AppData\Roaming\Python\Python36\site-packages\pandas\core\groupby\generic.py in _aggregate_multiple_funcs(self, arg)
    292             # GH 15931
    293             if isinstance(self._selected_obj, Series):
--> 294                 raise SpecificationError("nested renamer is not supported")
    295
    296             columns = list(arg.keys())

SpecificationError: **nested renamer is not supported**
```

Response:
In this specific case you can change
```py
temp['total'] = pd.DataFrame(project_data.groupby(col1)[col2].agg({'total':'count'})).reset_index()['total']

temp['Avg'] = pd.DataFrame(project_data.groupby(col1)[col2].agg({'Avg':'mean'})).reset_index()['Avg']
```
to the new syntax
```py
temp['total'] = pd.DataFrame(project_data.groupby(col1)[col2].agg(total='count')).reset_index()['total']
temp['Avg'] = pd.DataFrame(project_data.groupby(col1)[col2].agg(Avg='mean')).reset_index()['Avg']
```
New syntax:
```py
df.groupby(
    [columns]
).agg(
    new_column_name=("column", aggregation_function)
)
```
The reason for this is that the new pandas version named aggregation is the recommended replacement for the deprecated dict-of-dicts approach to naming the output of column-specific aggregations (Deprecate groupby.agg() with a dictionary when renaming). source: https://pandas.pydata.org/pandas-docs/stable/whatsnew/v0.25.0.html


Question:
Pandas DataFrame How to query the closest datetime index? How do i query for the closest index from a Pandas DataFrame? The index is DatetimeIndex
```
2016-11-13 20:00:10.617989120   7.0 132.0
2016-11-13 22:00:00.022737152   1.0 128.0
2016-11-13 22:00:28.417561344   1.0 132.0
```
I tried this:
```
df.index.get_loc(df.index[0], method='nearest')
```
but it give me InvalidIndexError: Reindexing only valid with uniquely valued Index objects Same error if I tried this:
```
dt = datetime.datetime.strptime("2016-11-13 22:01:25", "%Y-%m-%d %H:%M:%S")
df.index.get_loc(dt, method='nearest')
```
But if I remove method='nearest' it works, but that is not I want, I want to find the closest index from my query datetime

Response:
It seems you need first get position by get_loc and then select by []:
```
dt = pd.to_datetime("2016-11-13 22:01:25.450")
print (dt)
2016-11-13 22:01:25.450000

print (df.index.get_loc(dt, method='nearest'))
2

idx = df.index[df.index.get_loc(dt, method='nearest')]
print (idx)
2016-11-13 22:00:28.417561344
```
```
#if need select row to Series use iloc
s = df.iloc[df.index.get_loc(dt, method='nearest')]
print (s)
b      1.0
c    132.0
Name: 2016-11-13 22:00:28.417561344, dtype: float64
```


Question:
Comparing previous row values in Pandas DataFrame
```
import pandas as pd
data={'col1':[1,3,3,1,2,3,2,2]}
df=pd.DataFrame(data,columns=['col1'])
print df

         col1
    0     1
    1     3
    2     3
    3     1
    4     2
    5     3
    6     2
    7     2
```
I have the following Pandas DataFrame and I want to create another column that compares the previous row of col1 to see if they are equal. What would be the best way to do this? It would be like the following DataFrame. Thanks
```
col1  match
0     1   False
1     3   False
2     3   True
3     1   False
4     2   False
5     3   False
6     2   False
7     2   True
```

Response:
You need eq with shift:
```
df['match'] = df.col1.eq(df.col1.shift())
print (df)
   col1  match
0     1  False
1     3  False
2     3   True
3     1  False
4     2  False
5     3  False
6     2  False
7     2   True
```
Or instead eq use ==, but it is a bit slowier in large DataFrame:
```
df['match'] = df.col1 == df.col1.shift()
print (df)
   col1  match
0     1  False
1     3  False
2     3   True
3     1  False
4     2  False
5     3  False
6     2  False
7     2   True
```
Timings:
```
import pandas as pd
data={'col1':[1,3,3,1,2,3,2,2]}
df=pd.DataFrame(data,columns=['col1'])
print (df)
#[80000 rows x 1 columns]
df = pd.concat([df]*10000).reset_index(drop=True)

df['match'] = df.col1 == df.col1.shift()
df['match1'] = df.col1.eq(df.col1.shift())
print (df)

```python
%timeit df.col1.eq(df.col1.shift())
The slowest run took 4.83 times longer than the fastest. This could mean that an intermediate result is being cached.
1000 loops, best of 3: 933 µs per loop
```

```python
%timeit df.col1 == df.col1.shift()
1000 loops, best of 3: 1 ms per loop
```
```


Question:
What is the pandas equivalent of dplyr summarize/aggregate by multiple functions? I want to convert my R code using dplyr package into pandas where I group-by and perform multiple summarizations. Here is my current code:
```
import pandas as pd
data = pd.DataFrame(
    {'col1':[1,1,1,1,1,2,2,2,2,2],
    'col2':[1,2,3,4,5,6,7,8,9,0],
     'col3':[-1,-2,-3,-4,-5,-6,-7,-8,-9,0]
    }
)
result = []
for k,v in data.groupby('col1'):
    result.append([k, max(v['col2']), min(v['col3'])])
print pd.DataFrame(result, columns=['col1', 'col2_agg', 'col3_agg'])
```
Issues: too verbose probably can be optimized and efficient. (I rewrote a for-loop groupby implementation into groupby.agg and the performance enhancement was huge). In R the equivalent code would be:
```
data %>% groupby(col1) %>% summarize(col2_agg=max(col2), col3_agg=min(col3))
```
Also, what is the equivalent of groupby().summarize(newcolumn=max(col2 * col3)) i.e. an aggregation/summarization where the function is a compound function of 2+ columns?

Response:
The equivalent of
```
df %>% groupby(col1) %>% summarize(col2_agg=max(col2), col3_agg=min(col3))
```
is
```
df.groupby('col1').agg({'col2': 'max', 'col3': 'min'})
```
which returns
```
col2  col3
col1
1        5    -5
2        9    -9
```
The returning object is a pandas.DataFrame with an index called col1 and columns named col2 and col3. By default, when you group your data pandas sets the grouping column(s) as index for efficient access and modification. However, if you don't want that, there are two alternatives to set col1 as a column. Pass as_index=False:
```
df.groupby('col1', as_index=False).agg({'col2': 'max', 'col3': 'min'})
```
Call reset_index:
```
df.groupby('col1').agg({'col2': 'max', 'col3': 'min'}).reset_index()
```
both yield
```
col1  col2  col3
   1     5    -5
   2     9    -9
```
You can also pass multiple functions to groupby.agg.
```
agg_df = df.groupby('col1').agg({'col2': ['max', 'min', 'std'],
                                 'col3': ['size', 'std', 'mean', 'max']})
```
This also returns a DataFrame but now it has a MultiIndex for columns.
```
col2               col3
      max min       std size       std mean max
col1
1       5   1  1.581139    5  1.581139   -3  -1
2       9   0  3.535534    5  3.535534   -6   0
```
MultiIndex is very handy for selection and grouping. Here are some examples:
```
agg_df['col2']  # select the second column
      max  min       std
col1
1       5    1  1.581139
2       9    0  3.535534

agg_df[('col2', 'max')]  # select the maximum of the second column
Out:
col1
1    5
2    9
Name: (col2, max), dtype: int64

agg_df.xs('max', axis=1, level=1)  # select the maximum of all columns
Out:
      col2  col3
col1
1        5    -1
2        9     0
```
Earlier (before version 0.20.0) it was possible to use dictionaries for renaming the columns in the agg call. For example
```
df.groupby('col1')['col2'].agg({'max_col2': 'max'})
```
would return the maximum of the second column as max_col2:
```
max_col2
col1
1            5
2            9
```
However, it was deprecated in favor of the rename method:
```
df.groupby('col1')['col2'].agg(['max']).rename(columns={'max': 'col2_max'})

      col2_max
col1
1            5
2            9
```
It can get verbose for a DataFrame like agg_df defined above. You can use a renaming function to flatten those levels in that case:
```
agg_df.columns = ['_'.join(col) for col in agg_df.columns]

      col2_max  col2_min  col2_std  col3_size  col3_std  col3_mean  col3_max
col1
1            5         1  1.581139          5  1.581139         -3        -1
2            9         0  3.535534          5  3.535534         -6         0
```
For operations like groupby().summarize(newcolumn=max(col2 * col3)), you can still use agg by first adding a new column with assign.
```
df.assign(new_col=df.eval('col2 * col3')).groupby('col1').agg('max')

      col2  col3  new_col
col1
1        5    -1       -1
2        9     0        0
```
This returns maximum for old and new columns but as always you can slice that.
```
df.assign(new_col=df.eval('col2 * col3')).groupby('col1')['new_col'].agg('max')

col1
1   -1
2    0
Name: new_col, dtype: int64
```
With groupby.apply this would be shorter:
```
df.groupby('col1').apply(lambda x: (x.col2 * x.col3).max())

col1
1   -1
2    0
dtype: int64
```
However, groupby.apply treats this as a custom function so it is not vectorized. Up to now, the functions we passed to agg ('min', 'max', 'min', 'size' etc.) are vectorized and these are aliases for those optimized functions. You can replace df.groupby('col1').agg('min') with df.groupby('col1').agg(min), df.groupby('col1').agg(np.min) or df.groupby('col1').min() and they will all execute the same function. You will not see the same efficiency when you use custom functions. Lastly, as of version 0.20, agg can be used on DataFrames directly, without having to group first. See examples here.


Question:
return default if pandas dataframe.loc location doesn't exist I find myself often having to check whether a column or row exists in a dataframe before trying to reference it. For example I end up adding a lot of code like:
```
if 'mycol' in df.columns and 'myindex' in df.index:
    x = df.loc[myindex, mycol]
else:
    x = mydefault
```
Is there any way to do this more nicely? For example on an arbitrary object I can do x = getattr(anobject, 'id', default) - is there anything similar to this in pandas? Really any way to achieve what I'm doing more gracefully?

Response:
There is a method for Series: So you could do:
```
df.mycol.get(myIndex, NaN)
```
Example:
```

```python
df = pd.DataFrame({'mycol':arange(5), 'dummy':arange(5)})
df
#Output
#dummy  mycol
#0      0      0
#1      1      1
#2      2      2
#3      3      3
#4      4      4
#
#[5 rows x 2 columns]
```

```python
print(df.mycol.get(2, NaN))
print(df.mycol.get(5, NaN))
2
nan
```
```


Question:
How to get scalar value on a cell using conditional indexing I have the dataframe shown below. I need to get the scalar value of column B, dependent on the value of A (which is a variable in my script). I'm trying the loc() function but it returns a Series instead of a scalar value. How do I get the scalar value()?
```

```python
x = pd.DataFrame({'A' : [0,1,2], 'B' : [4,5,6]})
```

```python
x
#Output
#   A  B
#0  0  4
#1  1  5
#2  2  6
```

```python
x.loc[x['A'] == 2]['B']
#Output
#2    6
#Name: B, dtype: int64
```

```python
type(x.loc[x['A'] == 2]['B'])
#Output
#<class 'pandas.core.series.Series'>
#```
```

Response:
First of all, you're better off accessing both the row and column indices from the .loc:
```
x.loc[x['A'] == 2, 'B']
```
Second, you can always get at the underlying numpy matrix using .values on a series or dataframe:
```
In : x.loc[x['A'] == 2, 'B'].values[0]
Out: 6
```
Finally, if you're not interested in the original question's "conditional indexing", there are also specific accessors designed to get a single scalar value from a DataFrame: dataframe.at[index, column] or dataframe.iat[i, j] (these are similar to .loc[] and .iloc[] but designed for quick access to a single value).


Question:
python pandas replacing strings in dataframe with numbers Is there any way to use the mapping function or something better to replace values in an entire dataframe? I only know how to perform the mapping on series. I would like to replace the strings in the 'tesst' and 'set' column with a number for example set = 1, test =2 Here is a example of my dataset: (Original dataset is very large)
```
ds_r
  respondent  brand engine  country  aware  aware_2  aware_3  age tesst   set
0          a  volvo      p      swe      1        0        1   23   set   set
1          b  volvo   None      swe      0        0        1   45   set   set
2          c    bmw      p       us      0        0        1   56  test  test
3          d    bmw      p       us      0        1        1   43  test  test
4          e    bmw      d  germany      1        0        1   34   set   set
5          f   audi      d  germany      1        0        1   59   set   set
6          g  volvo      d      swe      1        0        0   65  test   set
7          h   audi      d      swe      1        0        0   78  test   set
8          i  volvo      d       us      1        1        1   32   set   set
```
Final result should be
```
ds_r
  respondent  brand engine  country  aware  aware_2  aware_3  age  tesst  set
0          a  volvo      p      swe      1        0        1   23      1    1
1          b  volvo   None      swe      0        0        1   45      1    1
2          c    bmw      p       us      0        0        1   56      2    2
3          d    bmw      p       us      0        1        1   43      2    2
4          e    bmw      d  germany      1        0        1   34      1    1
5          f   audi      d  germany      1        0        1   59      1    1
6          g  volvo      d      swe      1        0        0   65      2    1
7          h   audi      d      swe      1        0        0   78      2    1
8          i  volvo      d       us      1        1        1   32      1    1
```

Response:
What about DataFrame.replace?
```

```python
mapping = {'set': 1, 'test': 2}
```

```python
df.replace({'set': mapping, 'tesst': mapping})
#Output
#Unnamed: 0 respondent  brand engine  country  aware  aware_2  aware_3  age  \
#0           0          a  volvo      p      swe      1        0        1   23
#1           1          b  volvo   None      swe      0        0        1   45
#2           2          c    bmw      p       us      0        0        1   56
#3           3          d    bmw      p       us      0        1        1   43
#4           4          e    bmw      d  germany      1        0        1   34
#5           5          f   audi      d  germany      1        0        1   59
#6           6          g  volvo      d      swe      1        0        0   65
#7           7          h   audi      d      swe      1        0        0   78
#8           8          i  volvo      d       us      1        1        1   32
#
#  tesst set
#0     2   1
#1     1   2
#2     2   1
#3     1   2
#4     2   1
#5     1   2
#6     2   1
#7     1   2
#8     2   1
#```
#As @Jeff pointed out in the comments, in pandas versions < 0.11.1, manually tack .convert_objects() onto the end to properly convert tesst and set to int64 columns, in case that matters in subsequent operations.
```


Question:
Split cell into multiple rows in pandas dataframe I have a dataframe contains orders data, each order has multiple packages stored as comma separated string [package & package_code] columns I want to split the packages data and create a row for each package including its order details Here is a sample input dataframe:
```
import pandas as pd
df = pd.DataFrame({"order_id":[1,3,7],"order_date":["20/5/2018","22/5/2018","23/5/2018"], "package":["p1,p2,p3","p4","p5,p6"],"package_code":["#111,#222,#333","#444","#555,#666"]})
```
And this is what I am trying to achieve as output: How can I do that with pandas?

Response:
pandas >= 0.25 Assuming all splittable columns have the same number of comma separated items, you can split on comma and then use Series.explode on each column:
```
(df.set_index(['order_id', 'order_date'])
   .apply(lambda x: x.str.split(',').explode())
   .reset_index())

   order_id order_date package package_code
0         1  20/5/2018      p1         #111
1         1  20/5/2018      p2         #222
2         1  20/5/2018      p3         #333
3         3  22/5/2018      p4         #444
4         7  23/5/2018      p5         #555
5         7  23/5/2018      p6         #666
```
Details Set the columns not to be touched as the index,
```
df.set_index(['order_id', 'order_date'])

                      package    package_code
order_id order_date
1        20/5/2018   p1,p2,p3  #111,#222,#333
3        22/5/2018         p4            #444
7        23/5/2018      p5,p6       #555,#666
```
The next step is a 2-step process: Split on comma to get a column of lists, then call explode to explode the list values into their own rows.
```
_.apply(lambda x: x.str.split(',').explode())

                    package package_code
order_id order_date
1        20/5/2018       p1         #111
         20/5/2018       p2         #222
         20/5/2018       p3         #333
3        22/5/2018       p4         #444
7        23/5/2018       p5         #555
         23/5/2018       p6         #666
```
Finally, reset the index.
```
_.reset_index()

   order_id order_date package package_code
0         1  20/5/2018      p1         #111
1         1  20/5/2018      p2         #222
2         1  20/5/2018      p3         #333
3         3  22/5/2018      p4         #444
4         7  23/5/2018      p5         #555
5         7  23/5/2018      p6         #666
```
pandas <= 0.24 This should work for any number of columns like this. The essence is a little stack-unstacking magic with str.split.
```
(df.set_index(['order_date', 'order_id'])
   .stack()
   .str.split(',', expand=True)
   .stack()
   .unstack(-2)
   .reset_index(-1, drop=True)
   .reset_index()
)

  order_date  order_id package package_code
0  20/5/2018         1      p1         #111
1  20/5/2018         1      p2         #222
2  20/5/2018         1      p3         #333
3  22/5/2018         3      p4         #444
4  23/5/2018         7      p5         #555
5  23/5/2018         7      p6         #666
```
There is another performant alternative involving chain, but you'd need to explicitly chain and repeat every column (a bit of a problem with a lot of columns). Choose whatever fits the description of your problem best, as there's no single answer. Details First, set the columns that are not to be touched as the index.
```
df.set_index(['order_date', 'order_id'])

                      package    package_code
order_date order_id
20/5/2018  1         p1,p2,p3  #111,#222,#333
22/5/2018  3               p4            #444
23/5/2018  7            p5,p6       #555,#666
```
Next, stack the rows.
```
_.stack()

order_date  order_id
20/5/2018   1         package               p1,p2,p3
                      package_code    #111,#222,#333
22/5/2018   3         package                     p4
                      package_code              #444
23/5/2018   7         package                  p5,p6
                      package_code         #555,#666
dtype: object
```
We have a series now. So call str.split on comma.
```
_.str.split(',', expand=True)

                                     0     1     2
order_date order_id
20/5/2018  1        package         p1    p2    p3
                    package_code  #111  #222  #333
22/5/2018  3        package         p4  None  None
                    package_code  #444  None  None
23/5/2018  7        package         p5    p6  None
                    package_code  #555  #666  None
```
We need to get rid of NULL values, so call stack again.
```
_.stack()

order_date  order_id
20/5/2018   1         package       0      p1
                                    1      p2
                                    2      p3
                      package_code  0    #111
                                    1    #222
                                    2    #333
22/5/2018   3         package       0      p4
                      package_code  0    #444
23/5/2018   7         package       0      p5
                                    1      p6
                      package_code  0    #555
                                    1    #666
dtype: object
```
We're almost there. Now we want the second last level of the index to become our columns, so unstack using unstack(-2) (unstack on the second last level)
```
_.unstack(-2)

                      package package_code
order_date order_id
20/5/2018  1        0      p1         #111
                    1      p2         #222
                    2      p3         #333
22/5/2018  3        0      p4         #444
23/5/2018  7        0      p5         #555
                    1      p6         #666
```
Get rid of the superfluous last level using reset_index:
```
_.reset_index(-1, drop=True)

                    package package_code
order_date order_id
20/5/2018  1             p1         #111
           1             p2         #222
           1             p3         #333
22/5/2018  3             p4         #444
23/5/2018  7             p5         #555
           7             p6         #666
```
And finally,
```
_.reset_index()

  order_date  order_id package package_code
0  20/5/2018         1      p1         #111
1  20/5/2018         1      p2         #222
2  20/5/2018         1      p3         #333
3  22/5/2018         3      p4         #444
4  23/5/2018         7      p5         #555
5  23/5/2018         7      p6         #666
```


Question:
Normalize rows of pandas data frame by their sums [duplicate] This question already has answers here: Normalizing pandas DataFrame rows by their sums (2 answers) Closed 4 years ago. I have a pandas dataframe containing spectral data and metadata. The columns are labeled with a multiindex so that df['wvl'] gives the spectra and df['meta'] gives the metadata. Within df['wvl'] the column labels are the wavelength values for the spectrometer channels. What I want to do is normalize each row of df['wvl'] by the sum of that row so that adding up the values in the row gives a total of 1.0. Here's what one row of the dataframe looks like:
```
df['wvl'].iloc[0]
246.050003     128.533035
246.102005     102.756321
246.156006      99.930775
...
848.697205     121.313347
848.896423     127.011662
849.095703     123.234168
Name: 0, dtype: float64
```
But when I do something like:
```
df['wvl'].iloc[0]=df['wvl'].iloc[0]/df['wvl'].iloc[0].sum()
```
Nothing happens! I get the exact same values:
```
df['wvl'].iloc[0]
246.050003     128.533035
246.102005     102.756321
246.156006      99.930775
...
848.697205     121.313347
848.896423     127.011662
849.095703     123.234168
Name: 0, dtype: float64
```
If I create a temporary variable to hold the row, I can do the normalization just fine:
```
temp=df['wvl'].iloc[0]

temp=temp/temp.sum()

temp
246.050003    0.000027
246.102005    0.000022
246.156006    0.000021
                ...
848.697205    0.000026
848.896423    0.000027
849.095703    0.000026
Name: 0, dtype: float64
```
But if I try to replace the dataframe row with the normalized temporary variable, nothing happens:
```
df['wvl'].iloc[0]=temp

df['wvl'].iloc[0]
246.050003     128.533035
246.102005     102.756321
246.156006      99.930775
                 ...
848.697205     121.313347
848.896423     127.011662
849.095703     123.234168
Name: 0, dtype: float64
```
I'm obviously missing something here, but I can't figure out what and it's driving me insane. Help? Thanks in advance!

Response:
You can use
```
df.div(df.sum(axis=1), axis=0)
```
df.sum(axis=1) sums up each row; df.div(..., axis=0) then divides. Example:
```
import pandas as pd

df = pd.DataFrame({'a': [1, 2], 'b': [3, 4]})

```python
df.div(df.sum(axis=1), axis=0)
#Output
#    a   b
#0   0.250000    0.750000
#1   0.333333    0.666667
#```
```


Question:
Calculate summary statistics of columns in dataframe I have a dataframe of the following form (for example)
```
shopper_num,is_martian,number_of_items,count_pineapples,birth_country,tranpsortation_method
1,FALSE,0,0,MX,
2,FALSE,1,0,MX,
3,FALSE,0,0,MX,
4,FALSE,22,0,MX,
5,FALSE,0,0,MX,
6,FALSE,0,0,MX,
7,FALSE,5,0,MX,
8,FALSE,0,0,MX,
9,FALSE,4,0,MX,
10,FALSE,2,0,MX,
11,FALSE,0,0,MX,
12,FALSE,13,0,MX,
13,FALSE,0,0,CA,
14,FALSE,0,0,US,
```
How can I use Pandas to calculate summary statistics of each column (column data types are variable, some columns have no information And then return the a dataframe of the form:
```
columnname, max, min, median,

is_martian, NA, NA, FALSE
```
So on and so on

Response:
describe may give you everything you want otherwise you can perform aggregations using groupby and pass a list of agg functions: http://pandas.pydata.org/pandas-docs/stable/groupby.html#applying-multiple-functions-at-once
```

```python
df.describe()
#Output
#shopper_num is_martian  number_of_items  count_pineapples
#count      14.0000         14        14.000000                14
#mean        7.5000          0         3.357143                 0
#std         4.1833          0         6.452276                 0
#min         1.0000      False         0.000000                 0
#25%         4.2500          0         0.000000                 0
#50%         7.5000          0         0.000000                 0
#75%        10.7500          0         3.500000                 0
#max        14.0000      False        22.000000                 0
#
#[8 rows x 4 columns]
#```
#Note that some columns cannot be summarised as there is no logical way to summarise them, for instance columns containing string data As you prefer you can transpose the result if you prefer:
#```
```

```python
df.describe().transpose()
#Output
#count      mean       std    min   25%  50%    75%    max
#shopper_num         14       7.5    4.1833      1  4.25  7.5  10.75     14
#is_martian          14         0         0  False     0    0      0  False
#number_of_items     14  3.357143  6.452276      0     0    0    3.5     22
#count_pineapples    14         0         0      0     0    0      0      0
#
#[4 rows x 8 columns]
#```
```


Question:
Set values on the diagonal of pandas.DataFrame I have a pandas dataframe I would like to se the diagonal to 0
```
import numpy
import pandas

df = pandas.DataFrame(numpy.random.rand(5,5))
df

Out[6]:
     0           1           2           3               4
0    0.536596    0.674319    0.032815    0.908086    0.215334
1    0.735022    0.954506    0.889162    0.711610    0.415118
2    0.119985    0.979056    0.901891    0.687829    0.947549
3    0.186921    0.899178    0.296294    0.521104    0.638924
4    0.354053    0.060022    0.275224    0.635054    0.075738
5 rows × 5 columns
```
now I want to set the diagonal to 0:
```
for i in range(len(df.index)):
    for j in range(len(df.columns)):
        if i==j:
            df.loc[i,j] = 0
df
Out[9]:
     0           1           2           3           4
0    0.000000    0.674319    0.032815    0.908086    0.215334
1    0.735022    0.000000    0.889162    0.711610    0.415118
2    0.119985    0.979056    0.000000    0.687829    0.947549
3    0.186921    0.899178    0.296294    0.000000    0.638924
4    0.354053    0.060022    0.275224    0.635054    0.000000
5 rows × 5 columns
```
but there must be a more pythonic way than that!?

Response:
```

```python
df.values[[np.arange(df.shape[0])]*2] = 0
```

```python
df
#Output
#0         1         2         3         4
#0  0.000000  0.931374  0.604412  0.863842  0.280339
#1  0.531528  0.000000  0.641094  0.204686  0.997020
#2  0.137725  0.037867  0.000000  0.983432  0.458053
#3  0.594542  0.943542  0.826738  0.000000  0.753240
#4  0.357736  0.689262  0.014773  0.446046  0.000000
#```
#Note that this will only work if df has the same number of rows as columns. Another way which will work for arbitrary shapes is to use np.fill_diagonal:
#```
```

```python
np.fill_diagonal(df.values, 0)
```
```


Question:
how to do a left,right and mid of a string in a pandas dataframe in a pandas dataframe how can I apply a sort of excel left('state',2) to only take the first two letters. Ideally I want to learn how to use left,right and mid in a dataframe too. So need an equivalent and not a "trick" for this specific example.
```
data = {'state': ['Auckland', 'Otago', 'Wellington', 'Dunedin', 'Hamilton'],
'year': [2000, 2001, 2002, 2001, 2002],
'pop': [1.5, 1.7, 3.6, 2.4, 2.9]}
df = pd.DataFrame(data)

print df

     pop       state  year
 0  1.5    Auckland  2000
 1  1.7       Otago  2001
 2  3.6  Wellington  2002
 3  2.4     Dunedin  2001
 4  2.9    Hamilton  2002
```
I want to get this:
```
pop       state     year  StateInitial
 0  1.5       Auckland    2000     Au
 1  1.7       Otago       2001     Ot
 2  3.6       Wellington  2002     We
 3  2.4       Dunedin     2001     Du
 4  2.9       Hamilton    2002     Ha
```

Response:
First two letters for each value in a column:
```

```python
df['StateInitial'] = df['state'].str[:2]
```

```python
df
#Output
#   pop       state  year StateInitial
#0  1.5    Auckland  2000           Au
#1  1.7       Otago  2001           Ot
#2  3.6  Wellington  2002           We
#3  2.4     Dunedin  2001           Du
#4  2.9    Hamilton  2002           Ha
#```
#For last two that would be df['state'].str[-2:]. Don't know what exactly you want for middle, but you can apply arbitrary function to a column with apply method:
#```
```

```python
df['state'].apply(lambda x: x[len(x)/2-1:len(x)/2+1])
#Output
#0    kl
#1    ta
#2    in
#3    ne
#4    il
#```
```


Question:
how do I remove rows with duplicate values of columns in pandas data frame? I have a pandas data frame which looks like this.
```
Column1  Column2 Column3
0     cat        1       C
1     dog        1       A
2     cat        1       B
```
I want to identify that cat and bat are same values which have been repeated and hence want to remove one record and preserve only the first record. The resulting data frame should only have.
```
Column1  Column2 Column3
0     cat        1       C
1     dog        1       A
```

Response:
Using drop_duplicates with subset with list of columns to check for duplicates on and keep='first' to keep first of duplicates. If dataframe is:
```
df = pd.DataFrame({'Column1': ["'cat'", "'toy'", "'cat'"],
                   'Column2': ["'bat'", "'flower'", "'bat'"],
                   'Column3': ["'xyz'", "'abc'", "'lmn'"]})
print(df)
```
Result:
```
Column1   Column2 Column3
0   'cat'     'bat'   'xyz'
1   'toy'  'flower'   'abc'
2   'cat'     'bat'   'lmn'
```
Then:
```
result_df = df.drop_duplicates(subset=['Column1', 'Column2'], keep='first')
print(result_df)
```
Result:
```
Column1   Column2 Column3
0   'cat'     'bat'   'xyz'
1   'toy'  'flower'   'abc'
```


Question:
How to show a pandas dataframe into a existing flask html table? This may sound a noob question, but I'm stuck with it as Python is not one of my best languages. I have a html page with a table inside it, and I would like to show a pandas dataframe in it. What is the best way to do it? Use pandasdataframe.to_html? py
```
from flask import Flask;
import pandas as pd;
from pandas import DataFrame, read_csv;

file = r'C:\Users\myuser\Desktop\Test.csv'
df = pd.read_csv(file)
df.to_html(header="true", table_id="table")
```
html
```
<div class="table_entrances" style="overflow-x: auto;">

  <table id="table">

    <thead></thead>
    <tr></tr>

  </table>

</div>
```

Response:
working example: python code:
```
from flask import Flask, request, render_template, session, redirect
import numpy as np
import pandas as pd

app = Flask(__name__)

df = pd.DataFrame({'A': [0, 1, 2, 3, 4],
                   'B': [5, 6, 7, 8, 9],
                   'C': ['a', 'b', 'c--', 'd', 'e']})

@app.route('/', methods=("POST", "GET"))
def html_table():

    return render_template('simple.html',  tables=[df.to_html(classes='data')], titles=df.columns.values)

if __name__ == '__main__':
    app.run(host='0.0.0.0')
```
html:
```
<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <title>Title</title>
</head>
<body>

{% for table in tables %}
            {{titles[loop.index]}}
            {{ table|safe }}
{% endfor %}
</body>
</html>
```
or else use
```
return render_template('simple.html',  tables=[df.to_html(classes='data', header="true")])
```
and remove {{titles[loop.index]}} line from html if you inspect element on html
```html
<html lang="en"><head>
    <meta charset="UTF-8">
    <title>Title</title>
</head>
<body style="">

            <table border="1" class="dataframe data">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>A</th>
      <th>B</th>
      <th>C</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>0</td>
      <td>5</td>
      <td>a</td>
    </tr>
    <tr>
      <th>1</th>
      <td>1</td>
      <td>6</td>
      <td>b</td>
    </tr>
    <tr>
      <th>2</th>
      <td>2</td>
      <td>7</td>
      <td>c--</td>
    </tr>
    <tr>
      <th>3</th>
      <td>3</td>
      <td>8</td>
      <td>d</td>
    </tr>
    <tr>
      <th>4</th>
      <td>4</td>
      <td>9</td>
      <td>e</td>
    </tr>
  </tbody>
</table>

</body></html>
```
as you can see it has tbody and thead with in table html. so you can easily apply css.


Question:
After groupby, how to flatten column headers? I'm trying to left join multiple pandas dataframes on a single Id column, but when I attempt the merge I get warning: KeyError: 'Id'. I think it might be because my dataframes have offset columns resulting from a groupby statement, but I could very well be wrong. Either way I can't figure out how to "unstack" my dataframe column headers. None of the answers at this question seem to work. My groupby code:
```
step1 = pd.DataFrame(step3.groupby(['Id', 'interestingtabsplittest2__grp'])['applications'].sum())
step1.sort('applications', ascending=False).head(3)
```
Returns: How to get those offset headers into the top level?

Response:
You're looking for .reset_index().
```

```python
df = pd.DataFrame([[2, 3], [5, 6]], pd.Index([1, 4], name="A"), columns=["B", "C"])
```

```python
df
#Output
#B  C
#A
#1  2  3
#4  5  6
```

```python
df.reset_index()
#Output
#A  B  C
#0  1  2  3
#1  4  5  6
#```
#Note: That you can avoid this step by using as_index=False when doing the groupby.
#```
#step1 = step3.groupby(['Id', 'interestingtabsplittest2__grp'], as_index=False)['applications'].sum()
#```
```


Question:
obtaining last value of dataframe column without index Suppose I have a DataFrame such as:
```
df = pd.DataFrame(np.random.randn(10, 5), columns=['a','b','c','d','e'])
```
and I would like to retrieve the last value in column e. I could do:
```
df['e'].tail(1)
```
but this would return a series which has index 9 with it. Ideally, I just want to obtain the value as a number that I can work with directly. I could also do:
```
np.array(df['e'].tail(1))
```
but this would then require me to access/call the 0th element of it before I can really work with it. Is there a more direct/easy way to do this?

Response:
You could try iloc method of dataframe:
```

```python
df
#Output
#a         b         c         d         e
#0 -1.079547 -0.722903  0.457495 -0.687271 -0.787058
#1  1.326133  1.359255 -0.964076 -1.280502  1.460792
#2  0.479599 -1.465210 -0.058247 -0.984733 -0.348068
#3 -0.608238 -1.238068 -0.126889  0.572662 -1.489641
#4 -1.533707 -0.218298 -0.877619  0.679370  0.485987
#5 -0.864651 -0.180165 -0.528939  0.270885  1.313946
#6  0.747612 -1.206509  0.616815 -1.758354 -0.158203
#7 -2.309582 -0.739730 -0.004303  0.125640 -0.973230
#8  1.735822 -0.750698  1.225104  0.431583 -1.483274
#9 -0.374557 -1.132354  0.875028  0.032615 -1.131971
```

```python
df['e'].iloc[-1]
#Output
#-1.1319705662711321
#```
#Or if you want just scalar you could use iat which is faster. From docs: If you only want to access a scalar value, the fastest way is to use the at and iat methods, which are implemented on all of the data structures
#```
```

```python
df.e.iat[-1]
#Output
#-1.1319705662711321
#```
#Benchmarking:
#```
```

```python
%timeit df.e.iat[-1]
100000 loops, best of 3: 18 µs per loop
```

```python
%timeit df.e.iloc[-1]
10000 loops, best of 3: 24 µs per loop
```
```


Question:
Sorting columns and selecting top n rows in each group pandas dataframe I have a dataframe like this:
```none
mainid  pidx    pidy   score
  1      a        b      2
  1      a        c      5
  1      c        a      7
  1      c        b      2
  1      a        e      8
  2      x        y      1
  2      y        z      3
  2      z        y      5
  2      x        w      12
  2      x        v      1
  2      y        x      6
```
I want to groupby on column 'pidx' and then sort score in descending order in each group i.e for each pidx and then select head(2) i.e top 2 from each group. The result I am looking for is like this:
```none
mainid   pidx    pidy    score
  1        a      e        8
  1        a      c        5
  1        c      a        7
  1        c      b        2
  2        x      w        12
  2        x      y        1
  2        y      x        6
  2        y      z        3
  2        z      y        5
```
What I tried was:
```py
df.sort(['pidx','score'],ascending = False).groupby('pidx').head(2)
```
and this seems to work, but I dont know if it's the right approach if working on a huge dataset. What other best method can I use to get such result?

Response:
There are 2 solutions: 1.sort_values and aggregate head:
```
df1 = df.sort_values('score',ascending = False).groupby('pidx').head(2)
print (df1)

    mainid pidx pidy  score
8        2    x    w     12
4        1    a    e      8
2        1    c    a      7
10       2    y    x      6
1        1    a    c      5
7        2    z    y      5
6        2    y    z      3
3        1    c    b      2
5        2    x    y      1
```
2.set_index and aggregate nlargest:
```
df = df.set_index(['mainid','pidy']).groupby('pidx')['score'].nlargest(2).reset_index()
print (df)
  pidx  mainid pidy  score
0    a       1    e      8
1    a       1    c      5
2    c       1    a      7
3    c       1    b      2
4    x       2    w     12
5    x       2    y      1
6    y       2    x      6
7    y       2    z      3
8    z       2    y      5
```
Timings:
```
np.random.seed(123)
N = 1000000

L1 = list('abcdefghijklmnopqrstu')
L2 = list('efghijklmnopqrstuvwxyz')
df = pd.DataFrame({'mainid':np.random.randint(1000, size=N),
                   'pidx': np.random.randint(10000, size=N),
                   'pidy': np.random.choice(L2, N),
                   'score':np.random.randint(1000, size=N)})
#print (df)

def epat(df):
    grouped = df.groupby('pidx')
    new_df = pd.DataFrame([], columns = df.columns)
    for key, values in grouped:
        new_df = pd.concat([new_df, grouped.get_group(key).sort_values('score', ascending=True)[:2]], 0)
    return (new_df)

print (epat(df))

```python
%timeit (df.sort_values('score',ascending = False).groupby('pidx').head(2))
1 loop, best of 3: 309 ms per loop
```

```python
%timeit (df.set_index(['mainid','pidy']).groupby('pidx')['score'].nlargest(2).reset_index())
1 loop, best of 3: 7.11 s per loop
```

```python
%timeit (epat(df))
1 loop, best of 3: 22 s per loop
```
```


Question:
Writing to MySQL database with pandas using SQLAlchemy, to_sql trying to write pandas dataframe to MySQL table using to_sql. Previously been using flavor='mysql', however it will be depreciated in the future and wanted to start the transition to using SQLAlchemy engine. sample code:
```
import pandas as pd
import mysql.connector
from sqlalchemy import create_engine

engine = create_engine('mysql+mysqlconnector://[user]:[pass]@[host]:[port]/[schema]', echo=False)
cnx = engine.raw_connection()
data = pd.read_sql('SELECT * FROM sample_table', cnx)
data.to_sql(name='sample_table2', con=cnx, if_exists = 'append', index=False)
```
The read works fine but the to_sql has an error: DatabaseError: Execution failed on sql 'SELECT name FROM sqlite_master WHERE type='table' AND name=?;': Wrong number of arguments during string formatting Why does it look like it is trying to use sqlite? What is the correct use of a sqlalchemy connection with mysql and specifically mysql.connector? I also tried passing the engine in as the connection as well, and that gave me an error referencing no cursor object.
```
data.to_sql(name='sample_table2', con=engine, if_exists = 'append', index=False)
>>AttributeError: 'Engine' object has no attribute 'cursor'
```

Response:
Using the engine in place of the raw_connection() worked:
```
import pandas as pd
import mysql.connector
from sqlalchemy import create_engine

engine = create_engine('mysql+mysqlconnector://[user]:[pass]@[host]:[port]/[schema]', echo=False)
data.to_sql(name='sample_table2', con=engine, if_exists = 'append', index=False)
```
Not clear on why when I tried this yesterday it gave me the earlier error.


Question:
Python - Drop row if two columns are NaN This is an extension to this question, where OP wanted to know how to drop rows where the values in a single column are NaN. I'm wondering how I can drop rows where the values in 2 (or more) columns are both NaN. Using the second answer's created Data Frame:
```

```python
df = pd.DataFrame(np.random.randn(10,3))
```

```python
df.ix[::2,0] = np.nan; df.ix[::4,1] = np.nan; df.ix[::3,2] = np.nan;
```

```python
df
#Output
#0         1         2
#0       NaN       NaN       NaN
#1  2.677677 -1.466923 -0.750366
#2       NaN  0.798002 -0.906038
#3  0.672201  0.964789       NaN
#4       NaN       NaN  0.050742
#5 -1.250970  0.030561 -2.678622
#6       NaN  1.036043       NaN
#7  0.049896 -0.308003  0.823295
#8       NaN       NaN  0.637482
#9 -0.310130  0.078891       NaN
#```
#If I use the drop.na() command, specifically the drop.na(subset=[1,2]), then it completes an "or" type drop and leaves:
#```
```

```python
df.dropna(subset=[1,2])
#Output
#0         1         2
#1  2.677677 -1.466923 -0.750366
#2       NaN  0.798002 -0.906038
#5 -1.250970  0.030561 -2.678622
#7  0.049896 -0.308003  0.823295
#```
#What I want is an "and" type drop, where it drops rows where there is an NaN in column index 1 and 2. This would leave:
#```
#0         1         2
#1  2.677677 -1.466923 -0.750366
#2       NaN  0.798002 -0.906038
#3  0.672201  0.964789       NaN
#4       NaN       NaN  0.050742
#5 -1.250970  0.030561 -2.678622
#6       NaN  1.036043       NaN
#7  0.049896 -0.308003  0.823295
#8       NaN       NaN  0.637482
#9 -0.310130  0.078891       NaN
#```
#where only the first row is dropped. Any ideas? EDIT: changed data frame values for consistency
```

Response:
Any one of the following two:
```
df.dropna(subset=[1, 2], how='all')
```
or
```
df.dropna(subset=[1, 2], thresh=1)
```


Question:
When to apply(pd.to_numeric) and when to astype(np.float64) in python? I have a pandas DataFrame object named xiv which has a column of int64 Volume measurements.
```

```python
xiv['Volume'].head(5)
#Output
#0    252000
#1    484000
#2     62000
#3    168000
#4    232000
#Name: Volume, dtype: int64
#```
#I have read other posts (like this and this) that suggest the following solutions. But when I use either approach, it doesn't appear to change the dtype of the underlying data:
#```
```

```python
xiv['Volume'] = pd.to_numeric(xiv['Volume'])
```

```python
xiv['Volume'].dtypes
#Output
#dtype('int64')
#```
#Or...
#```
```

```python
xiv['Volume'] = pd.to_numeric(xiv['Volume'])
#Output
####omitted for brevity###
```

```python
xiv['Volume'].dtypes
#Output
#dtype('int64')
```

```python
xiv['Volume'] = xiv['Volume'].apply(pd.to_numeric)
```

```python
xiv['Volume'].dtypes
#Output
#dtype('int64')
#```
#I've also tried making a separate pandas Series and using the methods listed above on that Series and reassigning to the x['Volume'] obect, which is a pandas.core.series.Series object. I have, however, found a solution to this problem using the numpy package's float64 type - this works but I don't know why it's different.
#```
```

```python
xiv['Volume'] = xiv['Volume'].astype(np.float64)
```

```python
xiv['Volume'].dtypes
#Output
#dtype('float64')
#```
#Can someone explain how to accomplish with the pandas library what the numpy library seems to do easily with its float64 class; that is, convert the column in the xiv DataFrame to a float64 in place.
```

Response:
If you already have numeric dtypes (int8|16|32|64,float64,boolean) you can convert it to another "numeric" dtype using Pandas .astype() method. Demo:
```

```python
df = pd.DataFrame(np.random.randint(10**5,10**7,(5,3)),columns=list('abc'), dtype=np.int64)
```

```python
df
#Output
#a        b        c
#0  9059440  9590567  2076918
#1  5861102  4566089  1947323
#2  6636568   162770  2487991
#3  6794572  5236903  5628779
#4   470121  4044395  4546794
```

```python
df.dtypes
#Output
#a    int64
#b    int64
#c    int64
#dtype: object
```

```python
df['a'] = df['a'].astype(float)
```

```python
df.dtypes
#Output
#a    float64
#b      int64
#c      int64
#dtype: object
#```
#It won't work for object (string) dtypes, that can't be converted to numbers:
#```
```

```python
df.loc[1, 'b'] = 'XXXXXX'
```

```python
df
#Output
#a        b        c
#0  9059440.0  9590567  2076918
#1  5861102.0   XXXXXX  1947323
#2  6636568.0   162770  2487991
#3  6794572.0  5236903  5628779
#4   470121.0  4044395  4546794
```

```python
df.dtypes
#Output
#a    float64
#b     object
#c      int64
#dtype: object
```

```python
df['b'].astype(float)
...
skipped
...
ValueError: could not convert string to float: 'XXXXXX'
```
So here we want to use pd.to_numeric() method:
```
```

```python
df['b'] = pd.to_numeric(df['b'], errors='coerce')
```

```python
df
#Output
#a          b        c
#0  9059440.0  9590567.0  2076918
#1  5861102.0        NaN  1947323
#2  6636568.0   162770.0  2487991
#3  6794572.0  5236903.0  5628779
#4   470121.0  4044395.0  4546794
```

```python
df.dtypes
#Output
#a    float64
#b    float64
#c      int64
#dtype: object
#```
```


Question:
Replace column values based on another dataframe python pandas - better way? Note:for simplicity's sake, i'm using a toy example, because copy/pasting dataframes is difficult in stack overflow (please let me know if there's an easy way to do this). Is there a way to merge the values from one dataframe onto another without getting the _X, _Y columns? I'd like the values on one column to replace all zero values of another column.
```
df1:

Name   Nonprofit    Business    Education

X      1             1           0
Y      0             1           0   <- Y and Z have zero values for Nonprofit and Educ
Z      0             0           0
Y      0             1           0

df2:

Name   Nonprofit    Education
Y       1            1     <- this df has the correct values.
Z       1            1

pd.merge(df1, df2, on='Name', how='outer')

Name   Nonprofit_X    Business    Education_X     Nonprofit_Y     Education_Y
Y       1                1          1                1               1
Y      1                 1          1                1               1
X      1                 1          0               nan             nan
Z      1                 1          1                1               1
```
In a previous post, I tried combine_First and dropna(), but these don't do the job. I want to replace zeros in df1 with the values in df2. Furthermore, I want all rows with the same Names to be changed according to df2.
```
Name    Nonprofit     Business    Education
Y        1             1           1
Y        1             1           1
X        1             1           0
Z        1             0           1
```
(need to clarify: The value in 'Business' column where name = Z should 0.) My existing solution does the following: I subset based on the names that exist in df2, and then replace those values with the correct value. However, I'd like a less hacky way to do this.
```
pubunis_df = df2
sdf = df1

regex = str_to_regex(', '.join(pubunis_df.ORGS))

pubunis = searchnamesre(sdf, 'ORGS', regex)

sdf.ix[pubunis.index, ['Education', 'Public']] = 1
searchnamesre(sdf, 'ORGS', regex)
```

Response:
Attention: In latest version of pandas, both answers above doesn't work anymore: KSD's answer will raise error:
```
df1 = pd.DataFrame([["X",1,1,0],
              ["Y",0,1,0],
              ["Z",0,0,0],
              ["Y",0,0,0]],columns=["Name","Nonprofit","Business", "Education"])

df2 = pd.DataFrame([["Y",1,1],
              ["Z",1,1]],columns=["Name","Nonprofit", "Education"])

df1.loc[df1.Name.isin(df2.Name), ['Nonprofit', 'Education']] = df2.loc[df2.Name.isin(df1.Name),['Nonprofit', 'Education']].values

df1.loc[df1.Name.isin(df2.Name), ['Nonprofit', 'Education']] = df2[['Nonprofit', 'Education']].values

Out[851]:
ValueError: shape mismatch: value array of shape (2,) could not be broadcast to indexing result of shape (3,)
```
and EdChum's answer will give us the wrong result:
```
df1.loc[df1.Name.isin(df2.Name), ['Nonprofit', 'Education']] = df2[['Nonprofit', 'Education']]

df1
Out[852]:
  Name  Nonprofit  Business  Education
0    X        1.0         1        0.0
1    Y        1.0         1        1.0
2    Z        NaN         0        NaN
3    Y        NaN         1        NaN
```
Well, it will work safely only if values in column 'Name' are unique and are sorted in both data frames. Here is my answer: Way 1:
```
df1 = df1.merge(df2,on='Name',how="left")
df1['Nonprofit_y'] = df1['Nonprofit_y'].fillna(df1['Nonprofit_x'])
df1['Business_y'] = df1['Business_y'].fillna(df1['Business_x'])
df1.drop(["Business_x","Nonprofit_x"],inplace=True,axis=1)
df1.rename(columns={'Business_y':'Business','Nonprofit_y':'Nonprofit'},inplace=True)
```
Way 2:
```
df1 = df1.set_index('Name')
df2 = df2.set_index('Name')
df1.update(df2)
df1.reset_index(inplace=True)
```
More guide about update.. The columns names of both data frames need to set index are not necessary same before 'update'. You could try 'Name1' and 'Name2'. Also, it works even if other unnecessary row in df2, which won't update df1. In other words, df2 doesn't need to be the super set of df1. Example:
```
df1 = pd.DataFrame([["X",1,1,0],
              ["Y",0,1,0],
              ["Z",0,0,0],
              ["Y",0,1,0]],columns=["Name1","Nonprofit","Business", "Education"])

df2 = pd.DataFrame([["Y",1,1],
              ["Z",1,1],
              ['U',1,3]],columns=["Name2","Nonprofit", "Education"])

df1 = df1.set_index('Name1')
df2 = df2.set_index('Name2')

df1.update(df2)
```
result:
```
Nonprofit  Business  Education
Name1
X           1.0         1        0.0
Y           1.0         1        1.0
Z           1.0         0        1.0
Y           1.0         1        1.0
```


Question:
How to remove numbers from string terms in a pandas dataframe I have a data frame similar to the one below:
```
Name    Volume  Value
May21   23      21321
James   12      12311
Adi22   11      4435
Hello   34      32454
Girl90  56      654654
```
I want the output to be in the format:
```
Name    Volume  Value
May     23      21321
James   12      12311
Adi     11      4435
Hello   34      32454
Girl    56      654654
```
Want to remove all the numbers from the Name column. Closest I have come is doing it at a cell level with the following code:
```
result = ''.join([i for i in df['Name'][1] if not i.isdigit()])
```
Any idea how to do it in a better way at the series/dataframe level.

Response:
You can apply str.replace to the Name column in combination with regular expressions:
```
import pandas as pd

# Example DataFrame
df = pd.DataFrame.from_dict({'Name'  : ['May21', 'James', 'Adi22', 'Hello', 'Girl90'],
                             'Volume': [23, 12, 11, 34, 56],
                             'Value' : [21321, 12311, 4435, 32454, 654654]})

df['Name'] = df['Name'].str.replace('\d+', '')

print(df)
```
Output:
```
Name   Value  Volume
0    May   21321      23
1  James   12311      12
2    Adi    4435      11
3  Hello   32454      34
4   Girl  654654      56
```
In the regular expression \d stands for "any digit" and + stands for "one or more". Thus, str.replace('\d+', '') means: "Replace all occurring digits in the strings with nothing".


Question:
Annotate data points while plotting from Pandas DataFrame I would like to annotate the data points with their values next to the points on the plot. The examples I found only deal with x and y as vectors. However, I would like to do this for a pandas DataFrame that contains multiple columns.
```
ax = plt.figure().add_subplot(1, 1, 1)
df.plot(ax = ax)
plt.show()
```
What is the best way to annotate all the points for a multi-column DataFrame?

Response:
Here's a (very) slightly slicker version of Dan Allan's answer:
```
import matplotlib.pyplot as plt
import pandas as pd
import numpy as np
import string

df = pd.DataFrame({'x':np.random.rand(10), 'y':np.random.rand(10)},
                  index=list(string.ascii_lowercase[:10]))
```
Which gives:
```
x         y
a  0.541974  0.042185
b  0.036188  0.775425
c  0.950099  0.888305
d  0.739367  0.638368
e  0.739910  0.596037
f  0.974529  0.111819
g  0.640637  0.161805
h  0.554600  0.172221
i  0.718941  0.192932
j  0.447242  0.172469
```
And then:
```
fig, ax = plt.subplots()
df.plot('x', 'y', kind='scatter', ax=ax)

for k, v in df.iterrows():
    ax.annotate(k, v)
```
Finally, if you're in interactive mode you might need to refresh the plot:
```
fig.canvas.draw()
```
Which produces: Or, since that looks incredibly ugly, you can beautify things a bit pretty easily:
```
from matplotlib import cm
cmap = cm.get_cmap('Spectral')
df.plot('x', 'y', kind='scatter', ax=ax, s=120, linewidth=0,
        c=range(len(df)), colormap=cmap)

for k, v in df.iterrows():
    ax.annotate(k, v,
                xytext=(10,-5), textcoords='offset points',
                family='sans-serif', fontsize=18, color='darkslategrey')
```
Which looks a lot nicer:


Question:
Converting a datetime column to a string column I'm trying to convert a datetime column back to a string in Pandas dataframe. the syntax I have so far is:
```
all_data['Order Day new'] = dt.date.strftime(all_data['Order Day new'], '%d/%m/%Y')
```
but this returns the error: descriptor 'strftime' requires a 'datetime.date' object but received a 'Series'. Can anyone tell me where I'm going wrong.

Response:
If you're using version 0.17.0 or higher then you can call this using .dt.strftime which is vectorised:
```
all_data['Order Day new'] = all_data['Order Day new'].dt.strftime('%Y-%m-%d')
```
** If your pandas version is older than 0.17.0 then you have to call apply and pass the data to strftime:
```

```python
all_data = pd.DataFrame({'Order Day new':[dt.datetime(2014,5,9), dt.datetime(2012,6,19)]})
print(all_data)
all_data.info()
  Order Day new
0    2014-05-09
1    2012-06-19
<class 'pandas.core.frame.DataFrame'>
Int64Index: 2 entries, 0 to 1
Data columns (total 1 columns):
Order Day new    2 non-null datetime64[ns]
dtypes: datetime64[ns](1)
memory usage: 32.0 bytes
```

```python
all_data['Order Day new'] = all_data['Order Day new'].apply(lambda x: dt.datetime.strftime(x, '%Y-%m-%d'))
all_data
#Output
#Order Day new
#0    2014-05-09
#1    2012-06-19
```

```python
all_data.info()
<class 'pandas.core.frame.DataFrame'>
Int64Index: 2 entries, 0 to 1
Data columns (total 1 columns):
Order Day new    2 non-null object
dtypes: object(1)
memory usage: 32.0+ bytes
```
You can't call strftime on the column as it doesn't understand Series as a param hence the error
```


Question:
Subtract a year from a datetime column in pandas I have a datetime column as below -
```

```python
df['ACC_DATE'].head(2)
#Output
#538   2006-04-07
#550   2006-04-12
#Name: ACC_DATE, dtype: datetime64[ns]
#```
#Now, I want to subtract an year from each row of this column. How can I achieve the same & which library can I use? The expected field -
#```
#ACC_DATE    NEW_DATE
#538   2006-04-07  2005-04-07
#549   2006-04-12  2005-04-12
#```
```

Response:
You can use DateOffset to achieve this:
```

```python
df['NEW_DATE'] = df['ACC_DATE'] - pd.DateOffset(years=1)
df
#Output
#ACC_DATE   NEW_DATE
#index
#538   2006-04-07 2005-04-07
#550   2006-04-12 2005-04-12
#```
```


Question:
Pandas how to use pd.cut() Here is the snippet:
```
test = pd.DataFrame({'days': [0,31,45]})
test['range'] = pd.cut(test.days, [0,30,60])
```
Output:
```
days    range
0   0       NaN
1   31      (30, 60]
2   45      (30, 60]
```
I am surprised that 0 is not in (0, 30], what should I do to categorize 0 as (0, 30]?

Response:
pandas.cut
```
test['range'] = pd.cut(test.days, [0,30,60], include_lowest=True)
print (test)
   days           range
0     0  (-0.001, 30.0]
1    31    (30.0, 60.0]
2    45    (30.0, 60.0]
```
See difference:
```
test = pd.DataFrame({'days': [0,20,30,31,45,60]})

test['range1'] = pd.cut(test.days, [0,30,60], include_lowest=True)
#30 value is in [30, 60) group
test['range2'] = pd.cut(test.days, [0,30,60], right=False)
#30 value is in (0, 30] group
test['range3'] = pd.cut(test.days, [0,30,60])
print (test)
   days          range1    range2    range3
0     0  (-0.001, 30.0]   [0, 30)       NaN
1    20  (-0.001, 30.0]   [0, 30)   (0, 30]
2    30  (-0.001, 30.0]  [30, 60)   (0, 30]
3    31    (30.0, 60.0]  [30, 60)  (30, 60]
4    45    (30.0, 60.0]  [30, 60)  (30, 60]
5    60    (30.0, 60.0]       NaN  (30, 60]
```
Or use numpy.searchsorted, but values of days has to be sorted:
```
arr = np.array([0,30,60])
test['range1'] = arr.searchsorted(test.days)
test['range2'] = arr.searchsorted(test.days, side='right') - 1
print (test)
   days  range1  range2
0     0       0       0
1    20       1       0
2    30       1       1
3    31       2       1
4    45       2       1
5    60       2       2
```


Question:
Pandas Timedelta in months How can I calculate the elapsed months using pandas? I have write the following, but this code is not elegant. Could you tell me a better way?
```python
import pandas as pd

df = pd.DataFrame([pd.Timestamp('20161011'),
                   pd.Timestamp('20161101') ], columns=['date'])
df['today'] = pd.Timestamp('20161202')

df = df.assign(
    elapsed_months=(12 *
                    (df["today"].map(lambda x: x.year) -
                     df["date"].map(lambda x: x.year)) +
                    (df["today"].map(lambda x: x.month) -
                     df["date"].map(lambda x: x.month))))
# Out[34]:
#         date      today  elapsed_months
# 0 2016-10-11 2016-12-02               2
# 1 2016-11-01 2016-12-02               1
```

Response:
Update for pandas 0.24.0: Since 0.24.0 has changed the api to return MonthEnd object from period subtraction, you could do some manual calculation as follows to get the whole month difference:
```
12 * (df.today.dt.year - df.date.dt.year) + (df.today.dt.month - df.date.dt.month)

# 0    2
# 1    1
# dtype: int64
```
Wrap in a function:
```
def month_diff(a, b):
    return 12 * (a.dt.year - b.dt.year) + (a.dt.month - b.dt.month)

month_diff(df.today, df.date)
# 0    2
# 1    1
# dtype: int64
```
Prior to pandas 0.24.0. You can round the date to Month with to_period() and then subtract the result:
```
df['elapased_months'] = df.today.dt.to_period('M') - df.date.dt.to_period('M')

df
#         date       today  elapased_months
#0  2016-10-11  2016-12-02                2
#1  2016-11-01  2016-12-02                1
```


Question: Pandas dataframe first x columns [duplicate] This question already has answers here: Is there a pandas function to display the first/last n columns, as in .head() & .tail()? (5 answers) Closed 10 years ago. I have a dataframe with about 500 columns and that's why I am wondering if there is anyway that I could use head() function but want to see the first 50 columns for example. Thanks

Response:
I wasn't sure if you meant rows or columns. If it's rows, then
```
df.head(50)
```
will do the trick. If it's columns, then
```
df.iloc[:, : 50]
```
will work. Of course, you can combine them. You can see this stuff at Indexing and Selecting Data.


Question: Plot multiple columns of pandas DataFrame using Seaborn suppose I have DataFrame with columns ['X_Axis','col_2','col_3',...,'col_n',] I need to plot the first column on X-Axis and rest on Y-Axis. FYI : all the values have been grouped according to X-Axis, the X-Axis values range from 0-25 and all other column values have been normalized to the scale of 0 - 1. I want it on same graph plot, not subplots. Preferred : FactorPlot , normal line graph.

Response:
Some seaborn plots will accept a wide dataframe, sns.pointplot(data=df, x='X_Axis', y='col_2'), but not sns.pointplot(data=df, x='X_Axis', y=['col_2', 'col_3']), so it's better to reshape the DataFrame. Reshape the DataFrame from wide to long with pandas.DataFrame.melt. Converting the dataframe from a wide to long form is standard for all seaborn plots, not just the examples shown. Tested in python 3.8.12, pandas 1.3.4, matplotlib 3.4.3, seaborn 0.11.2 Sample DataFrame
```py
import pandas as pd
import seaborn as sns

df = pd.DataFrame({'X_Axis':[1,3,5,7,10,20],
                   'col_2':[.4,.5,.4,.5,.5,.4],
                   'col_3':[.7,.8,.9,.4,.2,.3],
                   'col_4':[.1,.3,.5,.7,.1,.0],
                   'col_5':[.5,.3,.6,.9,.2,.4]})

# display(df)
   X_Axis  col_2  col_3  col_4  col_5
0       1    0.4    0.7    0.1    0.5
1       3    0.5    0.8    0.3    0.3
2       5    0.4    0.9    0.5    0.6
3       7    0.5    0.4    0.7    0.9
4      10    0.5    0.2    0.1    0.2
5      20    0.4    0.3    0.0    0.4

# convert to long (tidy) form
dfm = df.melt('X_Axis', var_name='cols', value_name='vals')

# display(dfm.head())
   X_Axis   cols  vals
0       1  col_2   0.4
1       3  col_2   0.5
2       5  col_2   0.4
3       7  col_2   0.5
4      10  col_2   0.5
```
Current Plot Methods catplot: figure-level Use seaborn.catplot with kind= (e.g. kind='point' to reproduce the FactorPlot default):
```py
g = sns.catplot(x="X_Axis", y="vals", hue='cols', data=dfm, kind='point')
```
pointplot: axes-level
```py
sns.pointplot(x="X_Axis", y="vals", hue='cols', data=dfm)
```
Original factorplot: was renamed to catplot v0.9.0 (July 2018) New versions of seaborn get warning: The factorplot function has been renamed to catplot. The original name will be removed in a future release. Please update your code. Note that the default kind in factorplot ('point') has changed 'strip' in catplot.
```py
g = sns.factorplot(x="X_Axis", y="vals", hue='cols', data=dfm)

# using pd.melt instead of pd.DataFrame.melt for pandas < 0.20.0
# dfm = pd.melt(df, 'X_Axis', var_name='cols',  value_name='vals')
# g = sns.factorplot(x="X_Axis", y="vals", hue='cols', data=dfm)
```


Question:
Sort dataframe by string length I want to sort by name length. There doesn't appear to be a key parameter for sort_values so I'm not sure how to accomplish this. Here is a test df:
```
import pandas as pd
df = pd.DataFrame({'name': ['Steve', 'Al', 'Markus', 'Greg'], 'score': [2, 4, 2, 3]})
```

Response:
You can use reindex of index of Series created by len with sort_values:
```
print (df.name.str.len())
0    5
1    2
2    6
3    4
Name: name, dtype: int64

print (df.name.str.len().sort_values())
1    2
3    4
0    5
2    6
Name: name, dtype: int64

s = df.name.str.len().sort_values().index
print (s)
Int64Index([1, 3, 0, 2], dtype='int64')

print (df.reindex(s))
     name  score
1      Al      4
3    Greg      3
0   Steve      2
2  Markus      2
```
```
df1 = df.reindex(s)
df1 = df1.reset_index(drop=True)
print (df1)
     name  score
0      Al      4
1    Greg      3
2   Steve      2
3  Markus      2
```


Question:
Pandas Extract Number from String Given the following data frame:
```
import pandas as pd
import numpy as np
df = pd.DataFrame({'A':['1a',np.nan,'10a','100b','0b'],
                   })
df

    A
0   1a
1   NaN
2   10a
3   100b
4   0b
```
I'd like to extract the numbers from each cell (where they exist). The desired result is:
```
A
0   1
1   NaN
2   10
3   100
4   0
```
I know it can be done with str.extract, but I'm not sure how.

Response:
Give it a regex capture group:
```
df.A.str.extract('(\d+)')
```
Gives you:
```
0      1
1    NaN
2     10
3    100
4      0
Name: A, dtype: object
```
(\d+) is a regex capturing group, and \d+ specifies a regex pattern that matches only digits. Note that this will only work for whole numbers and not floats.


Question: When to use pandas series, numpy ndarrays or simply python dictionaries? I am new to learning Python, and some of its libraries (numpy, pandas). I have found a lot of documentation on how numpy ndarrays, pandas series and python dictionaries work. But owing to my inexperience with Python, I have had a really hard time determining when to use each one of them. And I haven't found any best-practices that will help me understand and decide when it is better to use each type of data structure. As a general matter, are there any best practices for deciding which, if any, of these three data structures a specific data set should be loaded into?

Response: The rule of thumb that I usually apply: use the simplest data structure that still satisfies your needs. If we rank the data structures from most simple to least simple, it usually ends up like this: Dictionaries / lists Numpy arrays Pandas series / dataframes So first consider dictionaries / lists. If these allow you to do all data operations that you need, then all is fine. If not, start considering numpy arrays. Some typical reasons for moving to numpy arrays are: Your data is 2-dimensional (or higher). Although nested dictionaries/lists can be used to represent multi-dimensional data, in most situations numpy arrays will be more efficient. You have to perform a bunch of numerical calculations. As already pointed out by zhqiat, numpy will give a significant speed-up in this case. Furthermore numpy arrays come bundled with a large amount of mathematical functions. Then there are also some typical reasons for going beyond numpy arrays and to the more-complex but also more-powerful pandas series/dataframes: You have to merge multiple data sets with each other, or do reshaping/reordering of your data. This diagram gives a nice overview of all the 'data wrangling' operations that pandas allows you to do. You have to import data from or export data to a specific file format like Excel, HDF5 or SQL. Pandas comes with convenient import/export functions for this.


Question:
how to convert pandas series to tuple of index and value I'm looking for an efficient way to convert a series to a tuple of its index with its values.
```
s = pd.Series([1, 2, 3], ['a', 'b', 'c'])
```
I want an array, list, series, some iterable:
```
[(1, 'a'), (2, 'b'), (3, 'c')]
```

Response:
Well it seems simply zip(s,s.index) works too! For Python-3.x, we need to wrap it with list -
```
list(zip(s,s.index))
```
To get a tuple of tuples, use tuple() : tuple(zip(s,s.index)). Sample run -
```

```python
s
#Output
#a    1
#b    2
#c    3
#dtype: int64
```

```python
list(zip(s,s.index))
#Output
#[(1, 'a'), (2, 'b'), (3, 'c')]
```

```python
tuple(zip(s,s.index))
#Output
#((1, 'a'), (2, 'b'), (3, 'c'))
#```
```


Question:
JOIN two dataframes on common column in pandas I have a dataframe df:
```
id   name   count
1    a       10
2    b       20
3    c       30
4    d       40
5    e       50
```
Here I have another dataframe df2:
```
id1  price   rating
 1     100     1.0
 2     200     2.0
 3     300     3.0
 5     500     5.0
```
I want to join these two dataframes on column id and id1. Here is an example of df3:
```
id   name   count   price   rating
1    a       10      100      1.0
2    b       20      200      2.0
3    c       30      300      3.0
4    d       40      Nan      Nan
5    e       50      500      5.0
```
Should I use df.merge or pd.concat?

Response:
Use merge:
```
print (pd.merge(df1, df2, left_on='id', right_on='id1', how='left').drop('id1', axis=1))
   id name  count  price  rating
0   1    a     10  100.0     1.0
1   2    b     20  200.0     2.0
2   3    c     30  300.0     3.0
3   4    d     40    NaN     NaN
4   5    e     50  500.0     5.0
```
Another solution is simple rename column:
```
print (pd.merge(df1, df2.rename(columns={'id1':'id'}), on='id',  how='left'))
   id name  count  price  rating
0   1    a     10  100.0     1.0
1   2    b     20  200.0     2.0
2   3    c     30  300.0     3.0
3   4    d     40    NaN     NaN
4   5    e     50  500.0     5.0
```
If need only column price the simpliest is map:
```
df1['price'] = df1.id.map(df2.set_index('id1')['price'])
print (df1)
   id name  count  price
0   1    a     10  100.0
1   2    b     20  200.0
2   3    c     30  300.0
3   4    d     40    NaN
4   5    e     50  500.0
```
Another 2 solutions:
```
print (pd.merge(df1, df2, left_on='id', right_on='id1', how='left')
         .drop(['id1', 'rating'], axis=1))
   id name  count  price
0   1    a     10  100.0
1   2    b     20  200.0
2   3    c     30  300.0
3   4    d     40    NaN
4   5    e     50  500.0
```
```
print (pd.merge(df1, df2[['id1','price']], left_on='id', right_on='id1', how='left')
         .drop('id1', axis=1))
   id name  count  price
0   1    a     10  100.0
1   2    b     20  200.0
2   3    c     30  300.0
3   4    d     40    NaN
4   5    e     50  500.0
```