Question:
Reshape wide to long in pandas Let's assume that I have the following dataframe in pandas:
```none
AA  BB  CC
 date
05/03     1   2   3
06/03     4   5   6
07/03     7   8   9
08/03     5   7   1
```
and I want to transform it to the following:
```none
AA  05/03  1
AA  06/03  4
AA  07/03  7
AA  08/03  5
BB  05/03  2
BB  06/03  5
BB  07/03  8
BB  08/03  7
CC  05/03  3
CC  06/03  6
CC  07/03  9
CC  08/03  1
```
How can I do it? The reason of the transformation from wide to long is that, in the next stage, I would like to merge this dataframe with another one, based on dates and the initial column names (AA, BB, CC).

Response:
Use pandas.melt or pandas.DataFrame.melt to transform from wide to long:
```
df = pd.DataFrame({
    'date' : ['05/03', '06/03', '07/03', '08/03'],
    'AA' : [1, 4, 7, 5],
    'BB' : [2, 5, 8, 7],
    'CC' : [3, 6, 9, 1]
}).set_index('date')
df

        AA  BB  CC
date
05/03   1   2   3
06/03   4   5   6
07/03   7   8   9
08/03   5   7   1
```
To convert, we just need to reset the index and then melt:
```
df = df.reset_index()
pd.melt(df, id_vars='date', value_vars=['AA', 'BB', 'CC'])
```
Using .reset_index after .melt, removes the need to specify value_vars.
```py
dfm = df.melt(ignore_index=False).reset_index()
```
Final Result - both options
```none
date variable  value
0   05/03       AA      1
1   06/03       AA      4
2   07/03       AA      7
3   08/03       AA      5
4   05/03       BB      2
5   06/03       BB      5
6   07/03       BB      8
7   08/03       BB      7
8   05/03       CC      3
9   06/03       CC      6
10  07/03       CC      9
11  08/03       CC      1
```


Question:
Drop rows with all zeros in pandas data frame I can use pandas dropna() functionality to remove rows with some or all columns set as NA's. Is there an equivalent function for dropping rows with all columns having value 0?
```
P   kt  b   tt  mky depth
1   0   0   0   0   0
2   0   0   0   0   0
3   0   0   0   0   0
4   0   0   0   0   0
5   1.1 3   4.5 2.3 9.0
```
In this example, we would like to drop the first 4 rows from the data frame. thanks!

Response:
One-liner. No transpose needed:
```
df.loc[~(df==0).all(axis=1)]
```
And for those who like symmetry, this also works...
```
df.loc[(df!=0).any(axis=1)]
```


Question: Appending a list or series to a pandas DataFrame as a row? So I have initialized an empty pandas DataFrame and I would like to iteratively append lists (or Series) as rows in this DataFrame. What is the best way of doing this?

Response:
Sometimes it's easier to do all the appending outside of pandas, then, just create the DataFrame in one shot.
```

```python
import pandas as pd
```

```python
simple_list=[['a','b']]
```

```python
simple_list.append(['e','f'])
```

```python
df=pd.DataFrame(simple_list,columns=['col1','col2'])
#Output
#   col1 col2
#0    a    b
#1    e    f
#```
```


Question:
Merging dataframes on index with pandas [duplicate] This question already has answers here: Pandas Merging 101 (8 answers) Closed 7 months ago. I have two dataframes and each one has two index columns. I would like to merge them. For example, the first dataframe is the following:
```none
V1
A      1/1/2012    12
       2/1/2012    14
B      1/1/2012    15
       2/1/2012     8
C      1/1/2012    17
       2/1/2012     9
```
The second dataframe is the following:
```none
V2
A      1/1/2012    15
       3/1/2012    21
B      1/1/2012    24
       2/1/2012     9
D      1/1/2012     7
       2/1/2012    16
```
and as result I would like to get the following:
```none
V1   V2
A      1/1/2012    12   15
       2/1/2012    14  N/A
       3/1/2012   N/A   21
B      1/1/2012    15   24
       2/1/2012     8    9
C      1/1/2012    17  N/A
       2/1/2012     9  N/A
D      1/1/2012   N/A    7
       2/1/2012   N/A   16
```
I have tried a few versions using the pd.merge and .join methods, but nothing seems to work. Do you have any suggestions?

Response:
You should be able to use join, which joins on the index as default. Given your desired result, you must use outer as the join type.
```

```python
df1.join(df2, how='outer')
#Output
#            V1  V2
#A 1/1/2012  12  15
#  2/1/2012  14 NaN
#  3/1/2012 NaN  21
#B 1/1/2012  15  24
#  2/1/2012   8   9
#C 1/1/2012  17 NaN
#  2/1/2012   9 NaN
#D 1/1/2012 NaN   7
#  2/1/2012 NaN  16
#```
#Signature: _.join(other, on=None, how='left', lsuffix='', rsuffix='', sort=False) Docstring: Join columns with other DataFrame either on index or on a key column. Efficiently Join multiple DataFrame objects by index at once by passing a list.
```


Question:
Find length of longest string in Pandas dataframe column Is there a faster way to find the length of the longest string in a Pandas DataFrame than what's shown in the example below?
```
import numpy as np
import pandas as pd

x = ['ab', 'bcd', 'dfe', 'efghik']
x = np.repeat(x, 1e7)
df = pd.DataFrame(x, columns=['col1'])

print df.col1.map(lambda x: len(x)).max()
# result --> 6
```
It takes about 10 seconds to run df.col1.map(lambda x: len(x)).max() when timing it with IPython's %timeit.

Response:
DSM's suggestion seems to be about the best you're going to get without doing some manual microoptimization:
```
%timeit -n 100 df.col1.str.len().max()
100 loops, best of 3: 11.7 ms per loop

%timeit -n 100 df.col1.map(lambda x: len(x)).max()
100 loops, best of 3: 16.4 ms per loop

%timeit -n 100 df.col1.map(len).max()
100 loops, best of 3: 10.1 ms per loop
```
Note that explicitly using the str.len() method doesn't seem to be much of an improvement. If you're not familiar with IPython, which is where that very convenient %timeit syntax comes from, I'd definitely suggest giving it a shot for quick testing of things like this. Update Added screenshot:


Question:
pandas combine two columns with null values I have a df with two columns and I want to combine both columns ignoring the NaN values. The catch is that sometimes both columns have NaN values in which case I want the new column to also have NaN. Here's the example:
```
df = pd.DataFrame({'foodstuff':['apple-martini', 'apple-pie', None, None, None], 'type':[None, None, 'strawberry-tart', 'dessert', None]})

df
Out[10]:
foodstuff   type
0   apple-martini   None
1   apple-pie   None
2   None    strawberry-tart
3   None    dessert
4   None    None
```
I tried to use fillna and solve this :
```
df['foodstuff'].fillna('') + df['type'].fillna('')
```
and I got :
```
0      apple-martini
1          apple-pie
2    strawberry-tart
3            dessert
4
dtype: object
```
The row 4 has become a blank value. What I want in this situation is a NaN value since both the combining columns are NaNs.
```
0      apple-martini
1          apple-pie
2    strawberry-tart
3            dessert
4            None
dtype: object
```

Response:
Use fillna on one column with the fill values being the other column:
```
df['foodstuff'].fillna(df['type'])
```
The resulting output:
```
0      apple-martini
1          apple-pie
2    strawberry-tart
3            dessert
4               None
```


Question:
Creating a Pandas DataFrame from a Numpy array: How do I specify the index column and column headers? I have a Numpy array consisting of a list of lists, representing a two-dimensional array with row labels and column names as shown below:
```py
data = np.array([['','Col1','Col2'],['Row1',1,2],['Row2',3,4]])
```
I'd like the resulting DataFrame to have Row1 and Row2 as index values, and Col1, Col2 as header values. I can specify the index as follows:
```py
df = pd.DataFrame(data, index=data[:,0])
```
However, I am unsure how to best assign column headers.

Response:
Specify data, index and columns to the DataFrame constructor, as follows:
```

```python
pd.DataFrame(data=data[1:,1:],    # values
             index=data[1:,0],    # 1st column as index
             columns=data[0,1:])  # 1st row as the column names
#Output
#```
#As @joris mentions, you may need to change above to np.int_(data[1:,1:]) to have the correct data type.
```


Question: Keep selected column as DataFrame instead of Series When selecting a single column from a pandas DataFrame(say df.iloc[:, 0], df['A'], or df.A, etc), the resulting vector is automatically converted to a Series instead of a single-column DataFrame. However, I am writing some functions that takes a DataFrame as an input argument. Therefore, I prefer to deal with single-column DataFrame instead of Series so that the function can assume say df.columns is accessible. Right now I have to explicitly convert the Series into a DataFrame by using something like pd.DataFrame(df.iloc[:, 0]). This doesn't seem like the most clean method. Is there a more elegant way to index from a DataFrame directly so that the result is a single-column DataFrame instead of Series?

Response:
As @Jeff mentions there are a few ways to do this, but I recommend using loc/iloc to be more explicit (and raise errors early if you're trying something ambiguous):
```

```python
df = pd.DataFrame([[1, 2], [3, 4]], columns=['A', 'B'])
```

```python
df
#Output
#A  B
#0  1  2
#1  3  4
```

```python
df[['A']]
```

```python
df[[0]]
```

```python
df.loc[:, ['A']]
```

```python
df.iloc[:, [0]]

Out[12-15]:  # they all return the same thing:
   A
0  1
1  3
```
The latter two choices remove ambiguity in the case of integer column names (precisely why loc/iloc were created). For example:
```
```

```python
df = pd.DataFrame([[1, 2], [3, 4]], columns=['A', 0])
```

```python
df
#Output
#A  0
#0  1  2
#1  3  4
```

```python
df[[0]]  # ambiguous
#Output
#A
#0  1
#1  3
#```
```


Question:
What does axis in pandas mean? Here is my code to generate a dataframe:
```
import pandas as pd
import numpy as np

dff = pd.DataFrame(np.random.randn(1, 2), columns=list('AB'))
```
then I got the dataframe:
```
A        B
0  0.626386  1.52325
```
When I type the command dff.mean(axis=1), I get:
```
0    1.074821
dtype: float64
```
According to the reference of pandas, axis=1 stands for columns and I expect the result of the command to be
```
A    0.626386
B    1.523255
dtype: float64
```
So what does axis in pandas mean?

Response:
It specifies the axis along which the means are computed. By default axis=0. This is consistent with the numpy.mean usage when axis is specified explicitly (in numpy.mean, axis==None by default, which computes the mean value over the flattened array) , in which axis=0 along the rows (namely, index in pandas), and axis=1 along the columns. For added clarity, one may choose to specify axis='index' (instead of axis=0) or axis='columns' (instead of axis=1).
```
A        B
0  0.626386  1.52325  → → axis=1 → →
          ↓        ↓
          ↓ axis=0 ↓
          ↓        ↓
```


Question:
How to convert index of a pandas dataframe into a column How to convert an index of a dataframe into a column? For example:
```none
gi       ptt_loc
 0  384444683      593
 1  384444684      594
 2  384444686      596
```
to
```none
index1    gi       ptt_loc
 0  0     384444683      593
 1  1     384444684      594
 2  2     384444686      596
```

Response:
either:
```
df['index1'] = df.index
```
or .reset_index:
```
df = df.reset_index()
```
If you have a multi-index frame with 3 levels of index, like:
```

```python
df
#Output
#                       val
#tick       tag obs
#2016-02-26 C   2    0.0139
#2016-02-27 A   2    0.5577
#2016-02-28 C   6    0.0303
#```
#and you want to convert the 1st (tick) and 3rd (obs) levels in the index into columns, you could do:
#```
```

```python
df.reset_index(level=['tick', 'obs'])
#Output
#          tick  obs     val
#tag
#C   2016-02-26    2  0.0139
#A   2016-02-27    2  0.5577
#C   2016-02-28    6  0.0303
#```
```


Question: pandas three-way joining multiple dataframes on columns I have 3 CSV files. Each has the first column as the (string) names of people, while all the other columns in each dataframe are attributes of that person. How can I "join" together all three CSV documents to create a single CSV with each row having all the attributes for each unique value of the person's string name? The join() function in pandas specifies that I need a multiindex, but I'm confused about what a hierarchical indexing scheme has to do with making a join based on a single index.

Response:
Zero's answer is basically a reduce operation. If I had more than a handful of dataframes, I'd put them in a list like this (generated via list comprehensions or loops or whatnot):
```
dfs = [df0, df1, df2, ..., dfN]
```
Assuming they have a common column, like name in your example, I'd do the following:
```
import functools as ft
df_final = ft.reduce(lambda left, right: pd.merge(left, right, on='name'), dfs)
```
That way, your code should work with whatever number of dataframes you want to merge.


Question:
Splitting a pandas dataframe column by delimiter i have a small sample data:
```
import pandas as pd

df = {'ID': [3009, 129, 119, 120, 121, 122, 130, 3014, 266, 849, 174, 844],
  'V': ['IGHV7-B*01', 'IGHV7-B*01', 'IGHV6-A*01', 'GHV6-A*01', 'IGHV6-A*01',
        'IGHV6-A*01', 'IGHV4-L*03', 'IGHV4-L*03', 'IGHV5-A*01', 'IGHV5-A*04',
        'IGHV6-A*02','IGHV6-A*02'],
  'Prob': [1, 1, 0.8, 0.8056, 0.9, 0.805, 1, 1, 0.997, 0.401, 1, 1]}

df = pd.DataFrame(df)
```
looks like
```
df

Out[25]:
      ID    Prob           V
0    3009  1.0000  IGHV7-B*01
1     129  1.0000  IGHV7-B*01
2     119  0.8000  IGHV6-A*01
3     120  0.8056  IGHV6-A*01
4     121  0.9000  IGHV6-A*01
5     122  0.8050  IGHV6-A*01
6     130  1.0000  IGHV4-L*03
7    3014  1.0000  IGHV4-L*03
8     266  0.9970  IGHV5-A*01
9     849  0.4010  IGHV5-A*04
10    174  1.0000  IGHV6-A*02
11    844  1.0000  IGHV6-A*02
```
I want to split the column 'V' by the '-' delimiter and move it to another column named 'allele'
```
Out[25]:
      ID    Prob      V    allele
0    3009  1.0000  IGHV7    B*01
1     129  1.0000  IGHV7    B*01
2     119  0.8000  IGHV6    A*01
3     120  0.8056  IGHV6    A*01
4     121  0.9000  IGHV6    A*01
5     122  0.8050  IGHV6    A*01
6     130  1.0000  IGHV4    L*03
7    3014  1.0000  IGHV4    L*03
8     266  0.9970  IGHV5    A*01
9     849  0.4010  IGHV5    A*04
10    174  1.0000  IGHV6    A*02
11    844  1.0000  IGHV6    A*02
```
the code i have tried so far is incomplete and didn't work:
```
df1 = pd.DataFrame()
df1[['V']] = pd.DataFrame([ x.split('-') for x in df['V'].tolist() ])
```
or
```
df.add(Series, axis='columns', level = None, fill_value = None)
newdata = df.DataFrame({'V':df['V'].iloc[::2].values,
                        'Allele': df['V'].iloc[1::2].values})
```

Response:
Use vectoried str.split with expand=True:
```

```python
df[['V','allele']] = df['V'].str.split('-',expand=True)
df
#Output
#ID    Prob      V allele
#0   3009  1.0000  IGHV7   B*01
#1    129  1.0000  IGHV7   B*01
#2    119  0.8000  IGHV6   A*01
#3    120  0.8056   GHV6   A*01
#4    121  0.9000  IGHV6   A*01
#5    122  0.8050  IGHV6   A*01
#6    130  1.0000  IGHV4   L*03
#7   3014  1.0000  IGHV4   L*03
#8    266  0.9970  IGHV5   A*01
#9    849  0.4010  IGHV5   A*04
#10   174  1.0000  IGHV6   A*02
#11   844  1.0000  IGHV6   A*02
#```
```


Question:
Drop non-numeric columns from a pandas DataFrame [duplicate] This question already has answers here: How do I find numeric columns in Pandas? (15 answers) Closed 6 years ago. In my application I load text files that are structured as follows: First non numeric column (ID) A number of non-numeric columns (strings) A number of numeric columns (floats) The number of the non-numeric columns is variable. Currently I load the data into a DataFrame like this:
```
source = pandas.read_table(inputfile, index_col=0)
```
I would like to drop all non-numeric columns in one fell swoop, without knowing their names or indices, since this could be doable reading their dtype. Is this possible with pandas or do I have to cook up something on my own?

Response: To avoid using a private method you can also use select_dtypes, where you can either include or exclude the dtypes you want. Ran into it on this post on the exact same thing. Or in your case, specifically: source.select_dtypes(['number']) or source.select_dtypes([np.number]


Question:
How can I obtain the element-wise logical NOT of a pandas Series? I have a pandas Series object containing boolean values. How can I get a series containing the logical NOT of each value? For example, consider a series containing:
```
True
True
True
False
```
The series I'd like to get would contain:
```
False
False
False
True
```
This seems like it should be reasonably simple, but apparently I've misplaced my mojo =(

Response:
To invert a boolean Series, use ~s:
```

```python
s = pd.Series([True, True, False, True])
```

```python
~s
#Output
#0    False
#1    False
#2     True
#3    False
#dtype: bool
#```
#Using Python2.7, NumPy 1.8.0, Pandas 0.13.1:
#```
```

```python
s = pd.Series([True, True, False, True]*10000)
```

```python
%timeit np.invert(s)
10000 loops, best of 3: 91.8 µs per loop
```

```python
%timeit ~s
10000 loops, best of 3: 73.5 µs per loop
```

```python
%timeit (-s)
10000 loops, best of 3: 73.5 µs per loop
```
As of Pandas 0.13.0, Series are no longer subclasses of numpy.ndarray; they are now subclasses of pd.NDFrame. This might have something to do with why np.invert(s) is no longer as fast as ~s or -s. Caveat: timeit results may vary depending on many factors including hardware, compiler, OS, Python, NumPy and Pandas versions.
```


Question: Convert floats to ints in Pandas? I've been working with data imported from a CSV. Pandas changed some columns to float, so now the numbers in these columns get displayed as floating points! However, I need them to be displayed as integers or without comma. Is there a way to convert them to integers or not display the comma?

Response:
Use the pandas.DataFrame.astype() function to manipulate column dtypes.
```

```python
df = pd.DataFrame(np.random.rand(3,4), columns=list("ABCD"))
```

```python
df
#Output
#          A         B         C         D
#0  0.542447  0.949988  0.669239  0.879887
#1  0.068542  0.757775  0.891903  0.384542
#2  0.021274  0.587504  0.180426  0.574300
```

```python
df[list("ABCD")] = df[list("ABCD")].astype(int)
```

```python
df
#Output
#   A  B  C  D
#0  0  0  0  0
#1  0  0  0  0
#2  0  0  0  0
#```
#EDIT: To handle missing values:
#```
```

```python
df
#Output
#          A         B     C         D
#0  0.475103  0.355453  0.66  0.869336
#1  0.260395  0.200287   NaN  0.617024
#2  0.517692  0.735613  0.18  0.657106
```

```python
df[list("ABCD")] = df[list("ABCD")].fillna(0.0).astype(int)
```

```python
df
#Output
#   A  B  C  D
#0  0  0  0  0
#1  0  0  0  0
#2  0  0  0  0
#```
```


Question:
Python Pandas update a dataframe value from another dataframe I have two dataframes in python. I want to update rows in first dataframe using matching values from another dataframe. Second dataframe serves as an override. Here is an example with same data and code: DataFrame 1 : DataFrame 2: I want to update update dataframe 1 based on matching code and name. In this example Dataframe 1 should be updated as below: Note : Row with Code =2 and Name= Company2 is updated with value 1000 (coming from Dataframe 2)
```
import pandas as pd

data1 = {
         'Code': [1, 2, 3],
         'Name': ['Company1', 'Company2', 'Company3'],
         'Value': [200, 300, 400],

    }
df1 = pd.DataFrame(data1, columns= ['Code','Name','Value'])

data2 = {
         'Code': [2],
         'Name': ['Company2'],
         'Value': [1000],
    }

df2 = pd.DataFrame(data2, columns= ['Code','Name','Value'])
```
Any pointers or hints?

Response:
Using DataFrame.update, which aligns on indices (https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.update.html):
```

```python
df1.set_index('Code', inplace=True)
```

```python
df1.update(df2.set_index('Code'))
```

```python
df1.reset_index()  # to recover the initial structure
```

Code      Name   Value
0     1  Company1   200.0
1     2  Company2  1000.0
2     3  Company3   400.0
```


Question:
How to "select distinct" across multiple data frame columns in pandas? I'm looking for a way to do the equivalent to the SQL
```
SELECT DISTINCT col1, col2 FROM dataframe_table
```
The pandas sql comparison doesn't have anything about distinct. .unique() only works for a single column, so I suppose I could concat the columns, or put them in a list/tuple and compare that way, but this seems like something pandas should do in a more native way. Am I missing something obvious, or is there no way to do this?

Response:
You can use the drop_duplicates method to get the unique rows in a DataFrame:
```

```python
df = pd.DataFrame({'a':[1,2,1,2], 'b':[3,4,3,5]})
```

```python
df
#Output
#a  b
#0  1  3
#1  2  4
#2  1  3
#3  2  5
```

```python
df.drop_duplicates()
#Output
#a  b
#0  1  3
#1  2  4
#3  2  5
#```
#You can also provide the subset keyword argument if you only want to use certain columns to determine uniqueness. See the docstring.
```


Question:
How to sort a Pandas DataFrame by index? When there is a DataFrame like the following:
```py
import pandas as pd
df = pd.DataFrame(1, index=[100, 29, 234, 1, 150], columns=['A'])
```
How can I sort this dataframe by index with each combination of index and column value intact?

Response:
Dataframes have a sort_index method which returns a copy by default. Pass inplace=True to operate in place.
```
import pandas as pd
df = pd.DataFrame([1, 2, 3, 4, 5], index=[100, 29, 234, 1, 150], columns=['A'])
df.sort_index(inplace=True)
print(df.to_string())
```
Gives me:
```
A
1    4
29   2
100  1
150  5
234  3
```


Question:
How to convert dataframe to dictionary in pandas WITHOUT index I have a dataframe df as follows:
```
| name  | coverage |
|-------|----------|
| Jason | 25.1     |
```
I want to convert it to a dictionary. I used the following command in pandas :
```
dict=df.to_dict()
```
The output of dict gave me the following:
```
{'coverage': {0: 25.1}, 'name': {0: 'Jason'}}
```
I do not want the 0 in my output. I believe this is captured because of the column index in my dataframe df. What can I do to eliminate 0 in my output ( I do not want index to be captured.) expected output :
```
{'coverage': 25.1, 'name': 'Jason'}
```

Response:
When I see your dataset with 2 columns I see a series and not a dataframe. Try this: d = df.set_index('name')['coverage'].to_dict() which will convert your dataframe to a series and output that. However, if your intent is to have more columns and not a common key you could store them in an array instead using 'records'. d = df.to_dict('r'). ` Runnable code:
```
import pandas as pd

df = pd.DataFrame({
    'name': ['Jason'],
    'coverage': [25.1]
})

print(df.to_dict())
print(df.set_index('name')['coverage'].to_dict())
print(df.to_dict('records'))
```
Returns:
```
{'name': {0: 'Jason'}, 'coverage': {0: 25.1}}
{'Jason': 25.1}
[{'name': 'Jason', 'coverage': 25.1}]
```
And one more thing, try to avoid to use variable name dict as it is reserved. Updated 2013-11-01. Now use 'records' instead of 'r' thanks to comment.


Question:
Binning a column with pandas I have a data frame column with numeric values:
```
df['percentage'].head()
46.5
44.2
100.0
42.12
```
I want to see the column as bin counts:
```
bins = [0, 1, 5, 10, 25, 50, 100]
```
How can I get the result as bins with their value counts?
```
[0, 1] bin amount
[1, 5] etc
[5, 10] etc
...
```

Response:
You can use pandas.cut:
```
bins = [0, 1, 5, 10, 25, 50, 100]
df['binned'] = pd.cut(df['percentage'], bins)
print (df)
   percentage     binned
0       46.50   (25, 50]
1       44.20   (25, 50]
2      100.00  (50, 100]
3       42.12   (25, 50]
```
```
bins = [0, 1, 5, 10, 25, 50, 100]
labels = [1,2,3,4,5,6]
df['binned'] = pd.cut(df['percentage'], bins=bins, labels=labels)
print (df)
   percentage binned
0       46.50      5
1       44.20      5
2      100.00      6
3       42.12      5
```
Or numpy.searchsorted:
```
bins = [0, 1, 5, 10, 25, 50, 100]
df['binned'] = np.searchsorted(bins, df['percentage'].values)
print (df)
   percentage  binned
0       46.50       5
1       44.20       5
2      100.00       6
3       42.12       5
```
...and then value_counts or groupby and aggregate size:
```
s = pd.cut(df['percentage'], bins=bins).value_counts()
print (s)
(25, 50]     3
(50, 100]    1
(10, 25]     0
(5, 10]      0
(1, 5]       0
(0, 1]       0
Name: percentage, dtype: int64
```
```
s = df.groupby(pd.cut(df['percentage'], bins=bins)).size()
print (s)
percentage
(0, 1]       0
(1, 5]       0
(5, 10]      0
(10, 25]     0
(25, 50]     3
(50, 100]    1
dtype: int64
```
By default cut returns categorical. Series methods like Series.value_counts() will use all categories, even if some categories are not present in the data, operations in categorical.


Question:
How to change the starting index of iterrows()? We can use the following to iterate rows of a data frame.
```
for index, row in df.iterrows():
```
What if I want to begin from a different row index? (not from first row)?

Response:
i know this has an answer, but why not just do:
```
for i, r in df.iloc[1:].iterrows():
```


Question:
How to simply add a column level to a pandas dataframe let's say I have a dataframe that looks like this:
```
df = pd.DataFrame({'A': range(5), 'B': range(5)}, index=list('abcde'))
 df
Out[92]:
   A  B
a  0  0
b  1  1
c  2  2
d  3  3
e  4  4
```
Asumming that this dataframe already exist, how can I simply add a level 'C' to the column index so I get this:
```
df
Out[92]:
   A  B
   C  C
a  0  0
b  1  1
c  2  2
d  3  3
e  4  4
```
I saw SO anwser like this python/pandas: how to combine two dataframes into one with hierarchical column index? but this concat different dataframe instead of adding a column level to an already existing dataframe.

Response:
As suggested by @StevenG himself, a better answer:
```
df.columns = pd.MultiIndex.from_product(df.columns.levels + [['C']])

print(df)
#    A  B
#    C  C
# a  0  0
# b  1  1
# c  2  2
# d  3  3
# e  4  4
```


Question:
How to replace NaNs by preceding or next values in pandas DataFrame? Suppose I have a DataFrame with some NaNs:
```

```python
import pandas as pd
```

```python
df = pd.DataFrame([[1, 2, 3], [4, None, None], [None, None, 9]])
```

```python
df
#Output
#    0   1   2
#0   1   2   3
#1   4 NaN NaN
#2 NaN NaN   9
#```
#What I need to do is replace every NaN with the first non-NaN value in the same column above it. It is assumed that the first row will never contain a NaN. So for the previous example the result would be
#```
#0  1  2
#0  1  2  3
#1  4  2  3
#2  4  2  9
#```
#I can just loop through the whole DataFrame column-by-column, element-by-element and set the values directly, but is there an easy (optimally a loop-free) way of achieving this?
```

Response:
You could use the fillna method on the DataFrame and specify the method as ffill (forward fill):
```

```python
df = pd.DataFrame([[1, 2, 3], [4, None, None], [None, None, 9]])
```

```python
df.fillna(method='ffill')
#Output
#   0  1  2
#0  1  2  3
#1  4  2  3
#2  4  2  9
#```
#This method... propagate[s] last valid observation forward to next valid To go the opposite way, there's also a bfill method. This method doesn't modify the DataFrame inplace - you'll need to rebind the returned DataFrame to a variable or else specify inplace=True:
#```
#df.fillna(method='ffill', inplace=True)
#```
```


Question:
How can I get a value from a cell of a dataframe? I have constructed a condition that extracts exactly one row from my dataframe:
```py
d2 = df[(df['l_ext']==l_ext) & (df['item']==item) & (df['wn']==wn) & (df['wd']==1)]
```
Now I would like to take a value from a particular column:
```py
val = d2['col_name']
```
But as a result, I get a dataframe that contains one row and one column (i.e., one cell). It is not what I need. I need one value (one float number). How can I do it in pandas?

Response:
If you have a DataFrame with only one row, then access the first (only) row as a Series using iloc, and then the value using the column name:
```

```python
sub_df
#Output
#A         B
#2 -0.133653 -0.030854
```

```python
sub_df.iloc[0]
#Output
#A   -0.133653
#B   -0.030854
#Name: 2, dtype: float64
```

```python
sub_df.iloc[0]['A']
#Output
#-0.13365288513107493
#```
```


Question:
How to get the number of the most frequent value in a column? I have a data frame and I would like to know how many times a given column has the most frequent value. I try to do it in the following way:
```
items_counts = df['item'].value_counts()
max_item = items_counts.max()
```
As a result I get:
```
ValueError: cannot convert float NaN to integer
```
As far as I understand, with the first line I get series in which the values from a column are used as key and frequency of these values are used as values. So, I just need to find the largest value in the series and, because of some reason, it does not work. Does anybody know how this problem can be solved?

Response: It looks like you may have some nulls in the column. You can drop them with df = df.dropna(subset=['item']). Then df['item'].value_counts().max() should give you the max counts, and df['item'].value_counts().idxmax() should give you the most frequent value.


Question:
Convert Pandas DataFrame to JSON format I have a Pandas DataFrame with two columns – one with the filename and one with the hour in which it was generated:
```
File       Hour
  F1         1
  F1         2
  F2         1
  F3         1
```
I am trying to convert it to a JSON file with the following format:
```
{"File":"F1","Hour":"1"}
{"File":"F1","Hour":"2"}
{"File":"F2","Hour":"1"}
{"File":"F3","Hour":"1"}
```
When I use the command DataFrame.to_json(orient = "records"), I get the records in the below format:
```
[{"File":"F1","Hour":"1"},
 {"File":"F1","Hour":"2"},
 {"File":"F2","Hour":"1"},
 {"File":"F3","Hour":"1"}]
```
I'm just wondering whether there is an option to get the JSON file in the desired format. Any help would be appreciated.

Response:
In newer versions of pandas (0.20.0+, I believe), this can be done directly:
```
df.to_json('temp.json', orient='records', lines=True)
```
Direct compression is also possible:
```
df.to_json('temp.json.gz', orient='records', lines=True, compression='gzip')
```


Question:
Filter out rows based on list of strings in Pandas I have a large time series data frame (called df), and the first 5 records look like this:
```
df

         stn     years_of_data  total_minutes avg_daily TOA_daily   K_daily
date
1900-01-14  AlberniElementary      4    5745    34.100  114.600 0.298
1900-01-14  AlberniWeather         6    7129    29.500  114.600 0.257
1900-01-14  Arbutus                8    11174   30.500  114.600 0.266
1900-01-14  Arrowview              7    10080   27.600  114.600 0.241
1900-01-14  Bayside                7    9745    33.800  114.600 0.295
```
Goal: I am trying to remove rows where any of the strings in a list are present in the 'stn' column. So,I am basically trying to filter this dataset to not include rows containing any of the strings in following list. Attempt:
```
remove_list = ['Arbutus','Bayside']

cleaned = df[df['stn'].str.contains('remove_list')]
```
Returns: Out[78]:
```
stn years_of_data   total_minutes   avg_daily   TOA_daily   K_daily
date
```
Nothing! I have tried a few combinations of quotes, brackets, and even a lambda function; though I am fairly new, so probably not using syntax properly..

Response:
Use isin:
```
cleaned = df[~df['stn'].isin(remove_list)]

```python
remove_list = ['Arbutus','Bayside']
df[~df['stn'].isin(remove_list)]
#Output
#stn  years_of_data  total_minutes  avg_daily  \
#date
#1900-01-14  AlberniElementary              4           5745       34.1
#1900-01-14     AlberniWeather              6           7129       29.5
#1900-01-14          Arrowview              7          10080       27.6
#
#            TOA_daily  K_daily
#date
#1900-01-14      114.6    0.298
#1900-01-14      114.6    0.257
#1900-01-14      114.6    0.241
#```
```


Question:
Remove duplicates by columns A, keeping the row with the highest value in column B I have a dataframe with repeat values in column A. I want to drop duplicates, keeping the row with the highest value in column B. So this:
```
A B
1 10
1 20
2 30
2 40
3 10
```
Should turn into this:
```
A B
1 20
2 40
3 10
```
I'm guessing there's probably an easy way to do this—maybe as easy as sorting the DataFrame before dropping duplicates—but I don't know groupby's internal logic well enough to figure it out. Any suggestions?

Response:
This takes the last. Not the maximum though:
```

```python
df.drop_duplicates(subset='A', keep="last")
#Output
#A   B
#1  1  20
#3  2  40
#4  3  10
#```
#You can do also something like:
#```
```

```python
df.groupby('A', group_keys=False).apply(lambda x: x.loc[x.B.idxmax()])
#Output
#A   B
#A
#1  1  20
#2  2  40
#3  3  10
#```
```


Question:
Redefining the Index in a Pandas DataFrame object I am trying to re-index a pandas DataFrame object, like so,
```
From:
            a   b   c
        0   1   2   3
        1  10  11  12
        2  20  21  22

To :
           b   c
       1   2   3
      10  11  12
      20  21  22
```
I am going about this as shown below and am getting the wrong answer. Any clues on how to do this?
```

```python
col = ['a','b','c']
```

```python
data = DataFrame([[1,2,3],[10,11,12],[20,21,22]],columns=col)
```

```python
data
#Output
#    a   b   c
#0   1   2   3
#1  10  11  12
#2  20  21  22
```

```python
idx2 = data.a.values
```

```python
idx2
#Output
#array([ 1, 10, 20], dtype=int64)
```

```python
data2 = DataFrame(data,index=idx2,columns=col[1:])
```

```python
data2
#Output
#     b   c
#1   11  12
#10 NaN NaN
#20 NaN NaN
#```
#Any idea why this is happening?
```

Response:
Why don't you simply use set_index method?
```
In : col = ['a','b','c']

In : data = DataFrame([[1,2,3],[10,11,12],[20,21,22]],columns=col)

In : data
Out:
    a   b   c
0   1   2   3
1  10  11  12
2  20  21  22

In : data2 = data.set_index('a')

In : data2
Out:
     b   c
a
1    2   3
10  11  12
20  21  22
```


Question: Conditionally fill column values based on another columns value in pandas I have a DataFrame with a few columns. One columns contains a symbol for which currency is being used, for instance a euro or a dollar sign. Another column contains a budget value. So for instance in one row it could mean a budget of 5000 in euro and in the next row it could say a budget of 2000 in dollar. In pandas I would like to add an extra column to my DataFrame, normalizing the budgets in euro. So basically, for each row the value in the new column should be the value from the budget column * 1 if the symbol in the currency column is a euro sign, and the value in the new column should be the value of the budget column * 0.78125 if the symbol in the currency column is a dollar sign. I know how to add a column, fill it with values, copy values from another column etc. but not how to fill the new column conditionally based on the value of another column. Any suggestions?

Response:
You probably want to do
```
df['Normalized'] = np.where(df['Currency'] == '$', df['Budget'] * 0.78125, df['Budget'])
```


Question:
How to determine the length of lists in a pandas dataframe column How can the length of the lists in the column be determine without iteration? I have a dataframe like this:
```none
CreationDate
2013-12-22 15:25:02                  [ubuntu, mac-osx, syslinux]
2009-12-14 14:29:32  [ubuntu, mod-rewrite, laconica, apache-2.2]
2013-12-22 15:42:00               [ubuntu, nat, squid, mikrotik]
```
I am calculating the length of lists in the CreationDate column and making a new Length column like this:
```py
df['Length'] = df.CreationDate.apply(lambda x: len(x))
```
Which gives me this:
```none
CreationDate  Length
2013-12-22 15:25:02                  [ubuntu, mac-osx, syslinux]       3
2009-12-14 14:29:32  [ubuntu, mod-rewrite, laconica, apache-2.2]       4
2013-12-22 15:42:00               [ubuntu, nat, squid, mikrotik]       4
```
Is there a more pythonic way to do this?

Response:
You can use the str accessor for some list operations as well. In this example,
```
df['CreationDate'].str.len()
```
returns the length of each list. See the docs for str.len.
```
df['Length'] = df['CreationDate'].str.len()
df
Out:
                                                    CreationDate  Length
2013-12-22 15:25:02                  [ubuntu, mac-osx, syslinux]       3
2009-12-14 14:29:32  [ubuntu, mod-rewrite, laconica, apache-2.2]       4
2013-12-22 15:42:00               [ubuntu, nat, squid, mikrotik]       4
```
For these operations, vanilla Python is generally faster. pandas handles NaNs though. Here are timings:
```
ser = pd.Series([random.sample(string.ascii_letters,
                               random.randint(1, 20)) for _ in range(10**6)])

%timeit ser.apply(lambda x: len(x))
1 loop, best of 3: 425 ms per loop

%timeit ser.str.len()
1 loop, best of 3: 248 ms per loop

%timeit [len(x) for x in ser]
10 loops, best of 3: 84 ms per loop

%timeit pd.Series([len(x) for x in ser], index=ser.index)
1 loop, best of 3: 236 ms per loop
```


Question:
Python Pandas How to assign groupby operation results back to columns in parent dataframe? I have the following data frame in IPython, where each row is a single stock:
```

```python
bdata
#Output
#<class 'pandas.core.frame.DataFrame'>
#Int64Index: 21210 entries, 0 to 21209
#Data columns:
#BloombergTicker      21206  non-null values
#Company              21210  non-null values
#Country              21210  non-null values
#MarketCap            21210  non-null values
#PriceReturn          21210  non-null values
#SEDOL                21210  non-null values
#yearmonth            21210  non-null values
#dtypes: float64(2), int64(1), object(4)
#```
#I want to apply a groupby operation that computes cap-weighted average return across everything, per each date in the "yearmonth" column. This works as expected:
#```
```

```python
bdata.groupby("yearmonth").apply(lambda x: (x["PriceReturn"]*x["MarketCap"]/x["MarketCap"].sum()).sum())
#Output
#yearmonth
#201204      -0.109444
#201205      -0.290546
#```
#But then I want to sort of "broadcast" these values back to the indices in the original data frame, and save them as constant columns where the dates match.
#```
```

```python
dateGrps = bdata.groupby("yearmonth")
```

```python
dateGrps["MarketReturn"] = dateGrps.apply(lambda x: (x["PriceReturn"]*x["MarketCap"]/x["MarketCap"].sum()).sum())
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
/mnt/bos-devrnd04/usr6/home/espears/ws/Research/Projects/python-util/src/util/<ipython-input-264-4a68c8782426> in <module>()
----> 1 dateGrps["MarketReturn"] = dateGrps.apply(lambda x: (x["PriceReturn"]*x["MarketCap"]/x["MarketCap"].sum()).sum())

TypeError: 'DataFrameGroupBy' object does not support item assignment
```
I realize this naive assignment should not work. But what is the "right" Pandas idiom for assigning the result of a groupby operation into a new column on the parent dataframe? In the end, I want a column called "MarketReturn" than will be a repeated constant value for all indices that have matching date with the output of the groupby operation. One hack to achieve this would be the following:
```
marketRetsByDate  = dateGrps.apply(lambda x: (x["PriceReturn"]*x["MarketCap"]/x["MarketCap"].sum()).sum())

bdata["MarketReturn"] = np.repeat(np.NaN, len(bdata))

for elem in marketRetsByDate.index.values:
    bdata["MarketReturn"][bdata["yearmonth"]==elem] = marketRetsByDate.ix[elem]
```
But this is slow, bad, and unPythonic.
```

Response:
```

```python
df = pandas.DataFrame({'month': np.random.randint(0,11, 100), 'A': np.random.randn(100), 'B': np.random.randn(100)})
```

```python
df.join(df.groupby('month')['A'].sum(), on='month', rsuffix='_r')
#Output
#A         B  month       A_r
#0  -0.040710  0.182269      0 -0.331816
#1  -0.004867  0.642243      1  2.448232
#2  -0.162191  0.442338      4  2.045909
#3  -0.979875  1.367018      5 -2.736399
#4  -1.126198  0.338946      5 -2.736399
#5  -0.992209 -1.343258      1  2.448232
#6  -1.450310  0.021290      0 -0.331816
#7  -0.675345 -1.359915      9  2.722156
#```
```


Question: Difference between asfreq and resample Can some please explain the difference between the asfreq and resample methods in pandas? When should one use what?

Response:
resample is more general than asfreq. For example, using resample I can pass an arbitrary function to perform binning over a Series or DataFrame object in bins of arbitrary size. asfreq is a concise way of changing the frequency of a DatetimeIndex object. It also provides padding functionality. As the pandas documentation says, asfreq is a thin wrapper around a call to date_range + a call to reindex. See here for an example. An example of resample that I use in my daily work is computing the number of spikes of a neuron in 1 second bins by resampling a large boolean array where True means "spike" and False means "no spike". I can do that as easy as large_bool.resample('S', how='sum'). Kind of neat! asfreq can be used when you want to change a DatetimeIndex to have a different frequency while retaining the same values at the current index. Here's an example where they are equivalent:
```

```python
dr = date_range('1/1/2010', periods=3, freq=3 * datetools.bday)
```

```python
raw = randn(3)
```

```python
ts = Series(raw, index=dr)
```

```python
ts
#Output
#2010-01-01   -1.948
#2010-01-06    0.112
#2010-01-11   -0.117
#Freq: 3B, dtype: float64
```

```python
ts.asfreq(datetools.BDay())
#Output
#2010-01-01   -1.948
#2010-01-04      NaN
#2010-01-05      NaN
#2010-01-06    0.112
#2010-01-07      NaN
#2010-01-08      NaN
#2010-01-11   -0.117
#Freq: B, dtype: float64
```

```python
ts.resample(datetools.BDay())
#Output
#2010-01-01   -1.948
#2010-01-04      NaN
#2010-01-05      NaN
#2010-01-06    0.112
#2010-01-07      NaN
#2010-01-08      NaN
#2010-01-11   -0.117
#Freq: B, dtype: float64
#```
#As far as when to use either: it depends on the problem you have in mind...care to share?
```


Question:
How to access the last element in a Pandas series Let us consider the following data frame:
```
import pandas as pd

d = {'col1': [1, 2, 3], 'col2': [3, 4, 5]}
df=pd.DataFrame(data=d)
```
If I want to access the first element in pandas series df['col1'], I can simply go df['col1'][0]. But how can I access the last element in this series? I have tried df['col1'][-1] which returns the following error: KeyError: -1L I know that I could go for something like df['col1'][len(df)-1] but why is reverse indexing impossible here?

Response:
For select last value need Series.iloc or Series.iat, because df['col1'] return Series:
```
print (df['col1'].iloc[-1])
3
print (df['col1'].iat[-1])
3
```
Or convert Series to numpy array and select last:
```
print (df['col1'].values[-1])
3
```
Or use DataFrame.iloc or DataFrame.iat - but is necessary position of column by Index.get_loc:
```
print (df.iloc[-1, df.columns.get_loc('col1')])
3
print (df.iat[-1, df.columns.get_loc('col1')])
3
```
Or is possible use last value of index (necessary not duplicated) and select by DataFrame.loc:
```
print (df.loc[df.index[-1], 'col1'])
3
```


Question:
What's the fastest way in Python to calculate cosine similarity given sparse matrix data? Given a sparse matrix listing, what's the best way to calculate the cosine similarity between each of the columns (or rows) in the matrix? I would rather not iterate n-choose-two times. Say the input matrix is:
```
A=
[0 1 0 0 1
 0 0 1 1 1
 1 1 0 1 0]
```
The sparse representation is:
```
A =
0, 1
0, 4
1, 2
1, 3
1, 4
2, 0
2, 1
2, 3
```
In Python, it's straightforward to work with the matrix-input format:
```
import numpy as np
from sklearn.metrics import pairwise_distances
from scipy.spatial.distance import cosine

A = np.array(
[[0, 1, 0, 0, 1],
[0, 0, 1, 1, 1],
[1, 1, 0, 1, 0]])

dist_out = 1-pairwise_distances(A, metric="cosine")
dist_out
```
Gives:
```
array([[ 1.        ,  0.40824829,  0.40824829],
       [ 0.40824829,  1.        ,  0.33333333],
       [ 0.40824829,  0.33333333,  1.        ]])
```
That's fine for a full-matrix input, but I really want to start with the sparse representation (due to the size and sparsity of my matrix). Any ideas about how this could best be accomplished?

Response:
You can compute pairwise cosine similarity on the rows of a sparse matrix directly using sklearn. As of version 0.17 it also supports sparse output:
```
from sklearn.metrics.pairwise import cosine_similarity
from scipy import sparse

A =  np.array([[0, 1, 0, 0, 1], [0, 0, 1, 1, 1],[1, 1, 0, 1, 0]])
A_sparse = sparse.csr_matrix(A)

similarities = cosine_similarity(A_sparse)
print('pairwise dense output:\n {}\n'.format(similarities))

#also can output sparse matrices
similarities_sparse = cosine_similarity(A_sparse,dense_output=False)
print('pairwise sparse output:\n {}\n'.format(similarities_sparse))
```
Results:
```
pairwise dense output:
[[ 1.          0.40824829  0.40824829]
[ 0.40824829  1.          0.33333333]
[ 0.40824829  0.33333333  1.        ]]

pairwise sparse output:
(0, 1)  0.408248290464
(0, 2)  0.408248290464
(0, 0)  1.0
(1, 0)  0.408248290464
(1, 2)  0.333333333333
(1, 1)  1.0
(2, 1)  0.333333333333
(2, 0)  0.408248290464
(2, 2)  1.0
```
If you want column-wise cosine similarities simply transpose your input matrix beforehand:
```
A_sparse.transpose()
```


Question:
Access index of last element in data frame I've looking around for this but I can't seem to find it (though it must be extremely trivial). The problem that I have is that I would like to retrieve the value of a column for the first and last entries of a data frame. But if I do:
```
df.ix[0]['date']
```
I get:
```
datetime.datetime(2011, 1, 10, 16, 0)
```
but if I do:
```
df[-1:]['date']
```
I get:
```
myIndex
13         2011-12-20 16:00:00
Name: mydate
```
with a different format. Ideally, I would like to be able to access the value of the last index of the data frame, but I can't find how. I even tried to create a column (IndexCopy) with the values of the index and try:
```
df.ix[df.tail(1)['IndexCopy']]['mydate']
```
but this also yields a different format (since df.tail(1)['IndexCopy'] does not output a simple integer). Any ideas?

Response:
The former answer is now superseded by .iloc:
```

```python
df = pd.DataFrame({"date": range(10, 64, 8)})
```

```python
df.index += 17
```

```python
df
#Output
#    date
#17    10
#18    18
#19    26
#20    34
#21    42
#22    50
#23    58
```

```python
df["date"].iloc[0]
#Output
#10
```

```python
df["date"].iloc[-1]
#Output
#58
#```
#The shortest way I can think of uses .iget():
#```
```

```python
df = pd.DataFrame({"date": range(10, 64, 8)})
```

```python
df.index += 17
```

```python
df
#Output
#    date
#17    10
#18    18
#19    26
#20    34
#21    42
#22    50
#23    58
```

```python
df['date'].iget(0)
#Output
#10
```

```python
df['date'].iget(-1)
#Output
#58
#```
#Alternatively:
#```
```

```python
df['date'][df.index[0]]
#Output
#10
```

```python
df['date'][df.index[-1]]
#Output
#58
#```
#There's also .first_valid_index() and .last_valid_index(), but depending on whether or not you want to rule out NaNs they might not be what you want. Remember that df.ix[0] doesn't give you the first, but the one indexed by 0. For example, in the above case, df.ix[0] would produce
#```
```

```python
df.ix[0]
#Output
#Traceback (most recent call last):
#  File "<ipython-input-489-494245247e87>", line 1, in <module>
#    df.ix[0]
#[...]
#KeyError: 0
#```
```


Question:
Python 'map' function inserting NaN, possible to return original values instead? I am passing a dictionary to the map function to recode values in the column of a Pandas dataframe. However, I noticed that if there is a value in the original series that is not explicitly in the dictionary, it gets recoded to NaN. Here is a simple example: Typing...
```
s = pd.Series(['one','two','three','four'])
```
...creates the series
```
0      one
1      two
2    three
3     four
dtype: object
```
But applying the map...
```
recodes = {'one':'A', 'two':'B', 'three':'C'}
s.map(recodes)
```
...returns the series
```
0      A
1      B
2      C
3    NaN
dtype: object
```
I would prefer that if any element in series s is not in the recodes dictionary, it remains unchanged. That is, I would prefer to return the series below (with the original four instead of NaN).
```
0      A
1      B
2      C
3   four
dtype: object
```
Is there an easy way to do this, for example an option to pass to the map function? The challenge I am having is that I can't always anticipate all possible values that will be in the series I'm recoding - the data will be updated in the future and new values could appear. Thanks!

Response:
Use replace instead of map:
```

```python
s = pd.Series(['one','two','three','four'])
```

```python
recodes = {'one':'A', 'two':'B', 'three':'C'}
```

```python
s.map(recodes)
#Output
#0      A
#1      B
#2      C
#3    NaN
#dtype: object
```

```python
s.replace(recodes)
#Output
#0       A
#1       B
#2       C
#3    four
#dtype: object
#```
```


Question:
Getting the integer index of a Pandas DataFrame row fulfilling a condition? I have the following DataFrame:
```
a  b  c
b
2  1  2  3
5  4  5  6
```
As you can see, column b is used as an index. I want to get the ordinal number of the row fulfilling ('b' == 5), which in this case would be 1. The column being tested can be either an index column (as with b in this case) or a regular column, e.g. I may want to find the index of the row fulfilling ('c' == 6).

Response:
Use Index.get_loc instead. Reusing @unutbu's set up code, you'll achieve the same results.
```

```python
import pandas as pd
```

```python
import numpy as np
```

```python
df = pd.DataFrame(np.arange(1,7).reshape(2,3),
#Output
#                  columns = list('abc'),
#                  index=pd.Series([2,5], name='b'))
```

```python
df
#Output
#   a  b  c
#b
#2  1  2  3
#5  4  5  6
```

```python
df.index.get_loc(5)
#Output
#1
#```
```


Question: How do I round datetime column to nearest quarter hour I have loaded a data file into a Python pandas dataframe. I has a datetime column of the format 2015-07-18 13:53:33.280. What I need to do is create a new column that rounds this out to its nearest quarter hour. So, the date above will be rounded to 2015-07-18 13:45:00.000. How do I do this in pandas? I tried using the solution from here, but get an 'Series' object has no attribute 'year' error.

Response:
You can use round(freq). There is also a shortcut column.dt for datetime functions access (as @laurens-koppenol suggests). Here's one-liner:
```
df['old column'].dt.round('15min')
```
String aliases for valid frequencies can be found here. Full working example:
```

```python
import pandas as pd
```

```python
df = pd.DataFrame([pd.Timestamp('2015-07-18 13:53:33.280'),
                           pd.Timestamp('2015-07-18 13:33:33.330')],
                         columns=['old column'])
```

```python
df['new column']=df['old column'].dt.round('15min')
```

```python
df
#Output
#old column          new column
#0 2015-07-18 13:53:33.280 2015-07-18 14:00:00
#1 2015-07-18 13:33:33.330 2015-07-18 13:30:00
#```
```


Question:
Applying function with multiple arguments to create a new pandas column I want to create a new column in a pandas data frame by applying a function to two existing columns. Following this answer I've been able to create a new column when I only need one column as an argument:
```
import pandas as pd
df = pd.DataFrame({"A": [10,20,30], "B": [20, 30, 10]})

def fx(x):
    return x * x

print(df)
df['newcolumn'] = df.A.apply(fx)
print(df)
```
However, I cannot figure out how to do the same thing when the function requires multiple arguments. For example, how do I create a new column by passing column A and column B to the function below?
```
def fxy(x, y):
    return x * y
```

Response:
You can go with @greenAfrican example, if it's possible for you to rewrite your function. But if you don't want to rewrite your function, you can wrap it into anonymous function inside apply, like this:
```

```python
def fxy(x, y):
    return x * y
```

```python
df['newcolumn'] = df.apply(lambda x: fxy(x['A'], x['B']), axis=1)
```

```python
df
#Output
#    A   B  newcolumn
#0  10  20        200
#1  20  30        600
#2  30  10        300
#```
```


Question:
Normalize columns of a dataframe I have a dataframe in pandas where each column has different value range. For example: df:
```
A     B   C
1000  10  0.5
765   5   0.35
800   7   0.09
```
Any idea how I can normalize the columns of this dataframe where each value is between 0 and 1? My desired output is:
```
A     B    C
1     1    1
0.765 0.5  0.7
0.8   0.7  0.18(which is 0.09/0.5)
```

Response:
one easy way by using Pandas: (here I want to use mean normalization)
```
normalized_df=(df-df.mean())/df.std()
```
to use min-max normalization:
```
normalized_df=(df-df.min())/(df.max()-df.min())
```
Edit: To address some concerns, need to say that Pandas automatically applies colomn-wise function in the code above.


Question:
pandas to_csv output quoting issue I'm having trouble getting the pandas dataframe.to_csv(...) output quoting strings right.
```
import pandas as pd

text = 'this is "out text"'
df = pd.DataFrame(index=['1'],columns=['1','2'])
df.loc['1','1']=123
df.loc['1','2']=text
df.to_csv('foo.txt',index=False,header=False)
```
The output is: 123,"this is ""out text""" But I would like: 123,this is "out text" Does anyone know how to get this right?

Response:
You could pass quoting=csv.QUOTE_NONE, for example:
```

```python
df.to_csv('foo.txt',index=False,header=False)
```

```python
!cat foo.txt
#Output
#123,"this is ""out text"""
```

```python
import csv
```

```python
df.to_csv('foo.txt',index=False,header=False, quoting=csv.QUOTE_NONE)
```

```python
!cat foo.txt
#Output
#123,this is "out text"
#```
#but in my experience it's better to quote more, rather than less.
```


Question:
Replacing few values in a pandas dataframe column with another value [duplicate] This question already has answers here: Remap values in pandas column with a dict, preserve NaNs (12 answers) Closed 1 year ago. I have a pandas dataframe df as illustrated below:
```
BrandName Specialty
A          H
B          I
ABC        J
D          K
AB         L
```
I want to replace 'ABC' and 'AB' in column BrandName by 'A'. Can someone help with this?

Response:
The easiest way is to use the replace method on the column. The arguments are a list of the things you want to replace (here ['ABC', 'AB']) and what you want to replace them with (the string 'A' in this case):
```

```python
df['BrandName'].replace(['ABC', 'AB'], 'A')
#Output
#0    A
#1    B
#2    A
#3    D
#4    A
#```
#This creates a new Series of values so you need to assign this new column to the correct column name:
#```
#df['BrandName'] = df['BrandName'].replace(['ABC', 'AB'], 'A')
#```
```


Question:
Pandas DataFrame to List of Dictionaries I have the following DataFrame:
```
customer    item1      item2    item3
1           apple      milk     tomato
2           water      orange   potato
3           juice      mango    chips
```
which I want to translate it to list of dictionaries per row
```
rows = [
    {
        'customer': 1,
        'item1': 'apple',
        'item2': 'milk',
        'item3': 'tomato'
    }, {
        'customer': 2,
        'item1':
        'water',
        'item2': 'orange',
        'item3': 'potato'
    }, {
        'customer': 3,
        'item1': 'juice',
        'item2': 'mango',
        'item3': 'chips'
    }
]
```

Response:
Use df.to_dict('records') -- gives the output without having to transpose externally.
```

```python
df.to_dict('records')
#Output
#[{'customer': 1L, 'item1': 'apple', 'item2': 'milk', 'item3': 'tomato'},
# {'customer': 2L, 'item1': 'water', 'item2': 'orange', 'item3': 'potato'},
# {'customer': 3L, 'item1': 'juice', 'item2': 'mango', 'item3': 'chips'}]
#```
```


Question:
Creating a pandas DataFrame from columns of other DataFrames with similar indexes I have 2 DataFrames df1 and df2 with the same column names ['a','b','c'] and indexed by dates. The date index can have similar values. I would like to create a DataFrame df3 with only the data from columns ['c'] renamed respectively 'df1' and 'df2' and with the correct date index. My problem is that I cannot get how to merge the index properly.
```
df1 = pd.DataFrame(np.random.randn(5,3), index=pd.date_range('01/02/2014',periods=5,freq='D'), columns=['a','b','c'] )
df2 = pd.DataFrame(np.random.randn(8,3), index=pd.date_range('01/01/2014',periods=8,freq='D'), columns=['a','b','c'] )
df1
                 a        b            c
2014-01-02   0.580550    0.480814    1.135899
2014-01-03  -1.961033    0.546013    1.093204
2014-01-04   2.063441   -0.627297    2.035373
2014-01-05   0.319570    0.058588    0.350060
2014-01-06   1.318068   -0.802209   -0.939962

df2
                 a        b            c
2014-01-01   0.772482    0.899337    0.808630
2014-01-02   0.518431   -1.582113    0.323425
2014-01-03   0.112109    1.056705   -1.355067
2014-01-04   0.767257   -2.311014    0.340701
2014-01-05   0.794281   -1.954858    0.200922
2014-01-06   0.156088    0.718658   -1.030077
2014-01-07   1.621059    0.106656   -0.472080
2014-01-08  -2.061138   -2.023157    0.257151
```
The df3 DataFrame should have the following form :
```
df3
                 df1        df2
2014-01-01   NaN        0.808630
2014-01-02   1.135899   0.323425
2014-01-03   1.093204   -1.355067
2014-01-04   2.035373   0.340701
2014-01-05   0.350060   0.200922
2014-01-06   -0.939962  -1.030077
2014-01-07   NaN        -0.472080
2014-01-08   NaN        0.257151
```
But with NaN in the df1 column as the date index of df2 is wider. (In this example, I would get NaN for the ollowing dates : 2014-01-01, 2014-01-07 and 2014-01-08) Thanks for your help.

Response:
You can use concat:
```

```python
pd.concat([df1['c'], df2['c']], axis=1, keys=['df1', 'df2'])
#Output
#df1       df2
#2014-01-01       NaN -0.978535
#2014-01-02 -0.106510 -0.519239
#2014-01-03 -0.846100 -0.313153
#2014-01-04 -0.014253 -1.040702
#2014-01-05  0.315156 -0.329967
#2014-01-06 -0.510577 -0.940901
#2014-01-07       NaN -0.024608
#2014-01-08       NaN -1.791899
#
#[8 rows x 2 columns]
#```
#The axis argument determines the way the DataFrames are stacked:
#```
#df1 = pd.DataFrame([1, 2, 3])
#df2 = pd.DataFrame(['a', 'b', 'c'])
#
#pd.concat([df1, df2], axis=0)
#   0
#0  1
#1  2
#2  3
#0  a
#1  b
#2  c
#
#pd.concat([df1, df2], axis=1)
#
#   0  0
#0  1  a
#1  2  b
#2  3  c
#```
```


Question:
How to plot multiple bars grouped How to plot multiple bars in matplotlib, when I tried to call the bar function multiple times, they overlap and as seen the below figure the highest value red can be seen only. How can I plot the multiple bars with dates on the x-axes? So far, I tried this:
```py
import matplotlib.pyplot as plt
import datetime

x = [
    datetime.datetime(2011, 1, 4, 0, 0),
    datetime.datetime(2011, 1, 5, 0, 0),
    datetime.datetime(2011, 1, 6, 0, 0)
]
y = [4, 9, 2]
z = [1, 2, 3]
k = [11, 12, 13]

ax = plt.subplot(111)
ax.bar(x, y, width=0.5, color='b', align='center')
ax.bar(x, z, width=0.5, color='g', align='center')
ax.bar(x, k, width=0.5, color='r', align='center')
ax.xaxis_date()

plt.show()
```
I got this: The results should be something like, but with the dates are on the x-axes and bars are next to each other:

Response:
```
import matplotlib.pyplot as plt
from matplotlib.dates import date2num
import datetime

x = [
    datetime.datetime(2011, 1, 4, 0, 0),
    datetime.datetime(2011, 1, 5, 0, 0),
    datetime.datetime(2011, 1, 6, 0, 0)
]
x = date2num(x)

y = [4, 9, 2]
z = [1, 2, 3]
k = [11, 12, 13]

ax = plt.subplot(111)
ax.bar(x-0.2, y, width=0.2, color='b', align='center')
ax.bar(x, z, width=0.2, color='g', align='center')
ax.bar(x+0.2, k, width=0.2, color='r', align='center')
ax.xaxis_date()

plt.show()
```
I don't know what's the "y values are also overlapping" means, does the following code solve your problem?
```
ax = plt.subplot(111)
w = 0.3
ax.bar(x-w, y, width=w, color='b', align='center')
ax.bar(x, z, width=w, color='g', align='center')
ax.bar(x+w, k, width=w, color='r', align='center')
ax.xaxis_date()
ax.autoscale(tight=True)

plt.show()
```


Question:
How to create dummies for certain columns with pandas.get_dummies()
```
df = pd.DataFrame({'A': ['x', 'y', 'x'], 'B': ['z', 'u', 'z'],
                  'C': ['1', '2', '3'],
                  'D':['j', 'l', 'j']})
```
I just want Column A and D to get dummies not for Column B. If I used pd.get_dummies(df), all columns turned into dummies. I want the final result containing all of columns , which means column C and column B exit,like 'A_x','A_y','B','C','D_j','D_l'.

Response:
It can be done without concatenation, using get_dummies() with required parameters
```

```python
pd.get_dummies(df, prefix=['A', 'D'], columns=['A', 'D'])
#Output
#B  C  A_x  A_y  D_j  D_l
#0  z  1  1.0  0.0  1.0  0.0
#1  u  2  0.0  1.0  0.0  1.0
#2  z  3  1.0  0.0  1.0  0.0
#```
```


Question:
Pandas: rolling mean by time interval I've got a bunch of polling data; I want to compute a Pandas rolling mean to get an estimate for each day based on a three-day window. According to this question, the rolling_* functions compute the window based on a specified number of values, and not a specific datetime range. How do I implement this functionality? Sample input data:
```
polls_subset.tail(20)
Out[185]:
            favorable  unfavorable  other

enddate
2012-10-25       0.48         0.49   0.03
2012-10-25       0.51         0.48   0.02
2012-10-27       0.51         0.47   0.02
2012-10-26       0.56         0.40   0.04
2012-10-28       0.48         0.49   0.04
2012-10-28       0.46         0.46   0.09
2012-10-28       0.48         0.49   0.03
2012-10-28       0.49         0.48   0.03
2012-10-30       0.53         0.45   0.02
2012-11-01       0.49         0.49   0.03
2012-11-01       0.47         0.47   0.05
2012-11-01       0.51         0.45   0.04
2012-11-03       0.49         0.45   0.06
2012-11-04       0.53         0.39   0.00
2012-11-04       0.47         0.44   0.08
2012-11-04       0.49         0.48   0.03
2012-11-04       0.52         0.46   0.01
2012-11-04       0.50         0.47   0.03
2012-11-05       0.51         0.46   0.02
2012-11-07       0.51         0.41   0.00
```
Output would have only one row for each date.

Response:
In the meantime, a time-window capability was added. See this link.
```

```python
df = DataFrame({'B': range(5)})
```

```python
df.index = [Timestamp('20130101 09:00:00'),
   ...:             Timestamp('20130101 09:00:02'),
   ...:             Timestamp('20130101 09:00:03'),
   ...:             Timestamp('20130101 09:00:05'),
   ...:             Timestamp('20130101 09:00:06')]
```

```python
df
#Output
#B
#2013-01-01 09:00:00  0
#2013-01-01 09:00:02  1
#2013-01-01 09:00:03  2
#2013-01-01 09:00:05  3
#2013-01-01 09:00:06  4
```

```python
df.rolling(2, min_periods=1).sum()
#Output
#B
#2013-01-01 09:00:00  0.0
#2013-01-01 09:00:02  1.0
#2013-01-01 09:00:03  3.0
#2013-01-01 09:00:05  5.0
#2013-01-01 09:00:06  7.0
```

```python
df.rolling('2s', min_periods=1).sum()
#Output
#B
#2013-01-01 09:00:00  0.0
#2013-01-01 09:00:02  1.0
#2013-01-01 09:00:03  3.0
#2013-01-01 09:00:05  3.0
#2013-01-01 09:00:06  7.0
#```
```


Question:
how to reset index pandas dataframe after dropna() pandas dataframe I'm not sure how to reset index after dropna(). I have
```py
df_all = df_all.dropna()
df_all.reset_index(drop=True)
```
but after running my code, row index skips steps. For example, it becomes 0,1,2,4,...

Response:
The code you've posted already does what you want, but does not do it "in place." Try adding inplace=True to reset_index() or else reassigning the result to df_all. Note that you can also use inplace=True with dropna(), so:
```
df_all.dropna(inplace=True)
df_all.reset_index(drop=True, inplace=True)
```
Does it all in place. Or,
```
df_all = df_all.dropna()
df_all = df_all.reset_index(drop=True)
```
to reassign df_all.


Question:
Find out the percentage of missing values in each column in the given dataset
```
import pandas as pd
df = pd.read_csv('https://query.data.world/s/Hfu_PsEuD1Z_yJHmGaxWTxvkz7W_b0')
percent= 100*(len(df.loc[:,df.isnull().sum(axis=0)>=1 ].index) / len(df.index))
print(round(percent,2))
```
input is https://query.data.world/s/Hfu_PsEuD1Z_yJHmGaxWTxvkz7W_b0 and the output should be
```
Ord_id                 0.00
Prod_id                0.00
Ship_id                0.00
Cust_id                0.00
Sales                  0.24
Discount               0.65
Order_Quantity         0.65
Profit                 0.65
Shipping_Cost          0.65
Product_Base_Margin    1.30
dtype: float64
```

Response:
How about this? I think I actually found something similar on here once before, but I'm not seeing it now...
```
percent_missing = df.isnull().sum() * 100 / len(df)
missing_value_df = pd.DataFrame({'column_name': df.columns,
                                 'percent_missing': percent_missing})
```
And if you want the missing percentages sorted, follow the above with:
```
missing_value_df.sort_values('percent_missing', inplace=True)
```
As mentioned in the comments, you may also be able to get by with just the first line in my code above, i.e.:
```
percent_missing = df.isnull().sum() * 100 / len(df)
```


Question:
Pandas How to filter a Series I have a Series like this after doing groupby('name') and used mean() function on other column
```
name
383      3.000000
663      1.000000
726      1.000000
737      9.000000
833      8.166667
```
Could anyone please show me how to filter out the rows with 1.000000 mean values? Thank you and I greatly appreciate your help.

Response:
```

```python
import pandas as pd

test = {
383:    3.000000,
663:    1.000000,
726:    1.000000,
737:    9.000000,
833:    8.166667
}

s = pd.Series(test)
s = s[s != 1]
s
#Output
#383    3.000000
#737    9.000000
#833    8.166667
#dtype: float64
#```
```


Question:
How to calculate 1st and 3rd quartiles? I have DataFrame:
```
time_diff   avg_trips
0   0.450000    1.0
1   0.483333    1.0
2   0.500000    1.0
3   0.516667    1.0
4   0.533333    2.0
```
I want to get 1st quartile, 3rd quartile and median for the column time_diff. To obtain median, I use np.median(df["time_diff"].values). How can I calculate quartiles?

Response:
You can use np.percentile to calculate quartiles (including the median):
```

```python
np.percentile(df.time_diff, 25)  # Q1
#Output
#0.48333300000000001
```

```python
np.percentile(df.time_diff, 50)  # median
#Output
#0.5
```

```python
np.percentile(df.time_diff, 75)  # Q3
#Output
#0.51666699999999999
#```
#Or all at once:
#```
```

```python
np.percentile(df.time_diff, [25, 50, 75])
#Output
#array([ 0.483333,  0.5     ,  0.516667])
#```
```


Question:
python, sort descending dataframe with pandas I'm trying to sort a dataframe by descending. I put 'False' in the ascending argument, but my order is still ascending. My code is:
```
from pandas import DataFrame
import pandas as pd

d = {'one':[2,3,1,4,5],
     'two':[5,4,3,2,1],
     'letter':['a','a','b','b','c']}

df = DataFrame(d)

test = df.sort(['one'], ascending=[False])
```
but the output is
```
letter  one  two
2      b    1    3
0      a    2    5
1      a    3    4
3      b    4    2
4      c    5    1
```

Response:
New syntax (either):
```
test = df.sort_values(['one'], ascending=[False])
 test = df.sort_values(['one'], ascending=[0])
```


Question:
Pandas populate new dataframe column based on matching columns in another dataframe I have a df which contains my main data which has one million rows. My main data also has 30 columns. Now I want to add another column to my df called category. The category is a column in df2 which contains around 700 rows and two other columns that will match with two columns in df. I begin with setting an index in df2 and df that will match between the frames, however some of the index in df2 doesn't exist in df. The remaining columns in df2 are called AUTHOR_NAME and CATEGORY. The relevant column in df is called AUTHOR_NAME. Some of the AUTHOR_NAME in df doesn't exist in df2 and vice versa. The instruction I want is: when index in df matches with index in df2 and title in df matches with title in df2, add category to df, else add NaN in category. Example data:
```
df2
           AUTHOR_NAME              CATEGORY
Index
Pub1        author1                 main
Pub2        author1                 main
Pub3        author1                 main
Pub1        author2                 sub
Pub3        author2                 sub
Pub2        author4                 sub

df
            AUTHOR_NAME     ...n amount of other columns
Index
Pub1        author1
Pub2        author1
Pub1        author2
Pub1        author3
Pub2        author4

expected_result
            AUTHOR_NAME             CATEGORY   ...n amount of other columns
Index
Pub1        author1                 main
Pub2        author1                 main
Pub1        author2                 sub
Pub1        author3                 NaN
Pub2        author4                 sub
```
If I use df2.merge(df,left_index=True,right_index=True,how='left', on=['AUTHOR_NAME']) my df becomes three times bigger than it is supposed to be. So I thought maybe merging was the wrong way to go about this. What I am really trying to do is use df2 as a lookup table and then return type values to df depending on if certain conditions are met.
```
def calculate_category(df2, d):
    category_row = df2[(df2["Index"] == d["Index"]) & (df2["AUTHOR_NAME"] == d["AUTHOR_NAME"])]
    return str(category_row['CATEGORY'].iat[0])

df.apply(lambda d: calculate_category(df2, d), axis=1)
```
However, this throws me an error:
```
IndexError: ('index out of bounds', u'occurred at index 7614')
```

Response:
Consider the following dataframes df and df2
```
df = pd.DataFrame(dict(
        AUTHOR_NAME=list('AAABBCCCCDEEFGG'),
        title=      list('zyxwvutsrqponml')
    ))

df2 = pd.DataFrame(dict(
        AUTHOR_NAME=list('AABCCEGG'),
        title      =list('zwvtrpml'),
        CATEGORY   =list('11223344')
    ))
```
option 1 merge
```
df.merge(df2, how='left')
```
option 2 join
```
cols = ['AUTHOR_NAME', 'title']
df.join(df2.set_index(cols), on=cols)
```
both options yield


Question:
Pandas column access w/column names containing spaces If I import or create a pandas column that contains no spaces, I can access it as such:
```
from pandas import DataFrame

df1 = DataFrame({'key': ['b', 'b', 'a', 'c', 'a', 'a', 'b'],
                 'data1': range(7)})

df1.data1
```
which would return that series for me. If, however, that column has a space in its name, it isn't accessible via that method:
```
from pandas import DataFrame

df2 = DataFrame({'key': ['a','b','d'],
                 'data 2': range(3)})

df2.data 2      # <--- not the droid I'm looking for.
```
I know I can access it using .xs():
```
df2.xs('data 2', axis=1)
```
There's got to be another way. I've googled it like mad and can't think of any other way to google it. I've read all 96 entries here on SO that contain "column" and "string" and "pandas" and could find no previous answer. Is this the only way, or is there something better?

Response:
Old post, but may be interesting: an idea (which is destructive, but does the job if you want it quick and dirty) is to rename columns using underscores:
```
df1.columns = [c.replace(' ', '_') for c in df1.columns]
```


Question: How can I compute a histogram (frequency table) for a single Series? How can I generate a frequency table (or histogram) for a single Series? For example, if I have my_series = pandas.Series([1,2,2,3,3,3]), how can I get a result like {1: 1, 2: 2, 3: 3} - that is, a count of how many times each value appears in the Series?

Response:
Maybe .value_counts()?
```

```python
import pandas
```

```python
my_series = pandas.Series([1,2,2,3,3,3, "fred", 1.8, 1.8])
```

```python
my_series
#Output
#0       1
#1       2
#2       2
#3       3
#4       3
#5       3
#6    fred
#7     1.8
#8     1.8
```

```python
counts = my_series.value_counts()
```

```python
counts
#Output
#3       3
#2       2
#1.8     2
#fred    1
#1       1
```

```python
len(counts)
#Output
#5
```

```python
sum(counts)
#Output
#9
```

```python
counts["fred"]
#Output
#1
```

```python
dict(counts)
#Output
#{1.8: 2, 2: 2, 3: 3, 1: 1, 'fred': 1}
#```
```


Question:
How to get unique values from multiple columns in a pandas groupby Starting from this dataframe df:
```
df = pd.DataFrame({'c':[1,1,1,2,2,2],'l1':['a','a','b','c','c','b'],'l2':['b','d','d','f','e','f']})

   c l1 l2
0  1  a  b
1  1  a  d
2  1  b  d
3  2  c  f
4  2  c  e
5  2  b  f
```
I would like to perform a groupby over the c column to get unique values of the l1 and l2 columns. For one columns I can do:
```
g = df.groupby('c')['l1'].unique()
```
that correctly returns:
```
c
1    [a, b]
2    [c, b]
Name: l1, dtype: object
```
but using:
```
g = df.groupby('c')['l1','l2'].unique()
```
returns:
```
AttributeError: 'DataFrameGroupBy' object has no attribute 'unique'
```
I know I can get the unique values for the two columns with (among others):
```

```python
np.unique(df[['l1','l2']])
#Output
#array(['a', 'b', 'c', 'd', 'e', 'f'], dtype=object)
#```
#Is there a way to apply this method to the groupby in order to get something like:
#```
#c
#1    [a, b, d]
#2    [c, b, e, f]
#Name: l1, dtype: object
#```
```

Response:
Alternatively, you can use agg:
```
g = df.groupby('c')['l1','l2'].agg(['unique'])
```


Question:
how to multiply multiple columns by a column in Pandas I would like to have:
```
df[['income_1', 'income_2']] * df['mtaz_proportion']
```
return those columns multiplied by df['mtaz_proportion'] so that I can set
```
df[['mtaz_income_1', 'mtaz_income_2']] =
df[['income_1', 'income_2']] * df['mtaz_proportion']
```
but instead I get:
```
income_1    income_2    0   1   2   3   4   5   6   7   8   9   10  11  12  13  14  15  16  17
0   NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN ...
1   NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN ...
2   NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN ...
```
ect... what simple thing am I missing? Thank you!

Response:
use multiply method and set axis="index":
```
df[["A", "B"]].multiply(df["C"], axis="index")
```


Question:
Pandas recalculate index after a concatenation I have a problem where I produce a pandas dataframe by concatenating along the row axis (stacking vertically). Each of the constituent dataframes has an autogenerated index (ascending numbers). After concatenation, my index is screwed up: it counts up to n (where n is the shape[0] of the corresponding dataframe), and restarts at zero at the next dataframe. I am trying to "re-calculate the index, given the current order", or "re-index" (or so I thought). Turns out that isn't exactly what DataFrame.reindex seems to be doing. Here is what I tried to do:
```
train_df = pd.concat(train_class_df_list)
train_df = train_df.reindex(index=[i for i in range(train_df.shape[0])])
```
It failed with "cannot reindex from a duplicate axis." I don't want to change the order of my data... just need to delete the old index and set up a new one, with the order of rows preserved.

Response:
If your index is autogenerated and you don't want to keep it, you can use the ignore_index option. `
```
train_df = pd.concat(train_class_df_list, ignore_index=True)
```
This will autogenerate a new index for you, and my guess is that this is exactly what you are after.


Question: Check if single cell value is NaN in Pandas I just want to check if a single cell in Pandas series is null or not i.e. to check if a value is NaN. All other answers are for series and arrays, but not for single value. I have tried pandas.notnull, pandas.isnull, numpy.isnan. Is there a solution for a single value only?

Response:
Try this:
```
import pandas as pd
import numpy as np
from pandas import *

```python
L = [4, nan ,6]
```

```python
df = Series(L)
```

```python
df
#Output
#0     4
#1   NaN
#2     6
```

```python
if(pd.isnull(df[1])):
#Output
#        print "Found"
```

Found

```python
if(np.isnan(df[1])):
#Output
#        print "Found"
```

Found
```


Question:
How to add/subtract time (hours, minutes, etc.) from a Pandas DataFrame.Index whos objects are of type datetime.time? I have a DataFrame whose index values are of type datetime.time. There is no method in DataFrame.Index to shift the time. datetime.time has replace but that will only work on individual items. Here's an example:
```

```python
dfa.index[:5]
#Output
#Index([21:12:19, 21:12:20, 21:12:21, 21:12:21, 21:12:22], dtype='object')
```

```python
type(dfa.index[0])
#Output
#datetime.time
#```
```

Response:
Liam's link looks great, but also check out pandas.Timedelta - looks like it plays nicely with NumPy's and Python's time deltas. https://pandas.pydata.org/pandas-docs/stable/timedeltas.html
```
pd.date_range('2014-01-01', periods=10) + pd.Timedelta(days=1)
```


Question:
changing sort in value_counts If I do
```
mt = mobile.PattLen.value_counts()   # sort True by default
```
I get
```
4    2831
3    2555
5    1561
[...]
```
If I do
```
mt = mobile.PattLen.value_counts(sort=False)
```
I get
```
8    225
9    120
2   1234
[...]
```
What I am trying to do is get the output in 2, 3, 4 ascending order (the left numeric column). Can I change value_counts somehow or do I need to use a different function.

Response:
I think you need sort_index, because the left column is called index. The full command would be mt = mobile.PattLen.value_counts().sort_index(). For example:
```
mobile = pd.DataFrame({'PattLen':[1,1,2,6,6,7,7,7,7,8]})
print (mobile)
   PattLen
0        1
1        1
2        2
3        6
4        6
5        7
6        7
7        7
8        7
9        8

print (mobile.PattLen.value_counts())
7    4
6    2
1    2
8    1
2    1
Name: PattLen, dtype: int64

mt = mobile.PattLen.value_counts().sort_index()
print (mt)
1    2
2    1
6    2
7    4
8    1
Name: PattLen, dtype: int64
```


Question:
python pandas pivot_table count frequency in one column I am still new to Python pandas' pivot_table and would like to ask a way to count frequencies of values in one column, which is also linked to another column of ID. The DataFrame looks like the following.
```
import pandas as pd
df = pd.DataFrame({'Account_number':[1,1,2,2,2,3,3],
                   'Product':['A', 'A', 'A', 'B', 'B','A', 'B']
                  })
```
For the output, I'd like to get something like the following:
```
Product
                A      B
Account_number
      1         2      0
      2         1      2
      3         1      1
```
So far, I tried this code:
```
df.pivot_table(rows = 'Account_number', cols= 'Product', aggfunc='count')
```
This code gives me the two same things. What is the problems with the code above? A part of the reason why I am asking this question is that this DataFrame is just an example. The real data that I am working on has tens of thousands of account_numbers.

Response:
Solution: Use aggfunc='size' Using aggfunc=len or aggfunc='count' like all the other answers on this page will not work for DataFrames with more than three columns. By default, pandas will apply this aggfunc to all the columns not found in index or columns parameters. For instance, if we had two more columns in our original DataFrame defined like this:
```
df = pd.DataFrame({'Account_number':[1, 1, 2 ,2 ,2 ,3 ,3],
                   'Product':['A', 'A', 'A', 'B', 'B','A', 'B'],
                   'Price': [10] * 7,
                   'Quantity': [100] * 7})
```
Output:
```
Account_number Product  Price  Quantity
0               1       A     10       100
1               1       A     10       100
2               2       A     10       100
3               2       B     10       100
4               2       B     10       100
5               3       A     10       100
6               3       B     10       100
```
If you apply the current solutions to this DataFrame, you would get the following:
```
df.pivot_table(index='Account_number',
               columns='Product',
               aggfunc=len,
               fill_value=0)
```
Output:
```
Price    Quantity
Product            A  B        A  B
Account_number
1                  2  0        2  0
2                  1  2        1  2
3                  1  1        1  1
```
Solution Instead, use aggfunc='size'. Since size always returns the same number for each column, pandas does not call it on every single column and just does it once.
```
df.pivot_table(index='Account_number',
               columns='Product',
               aggfunc='size',
               fill_value=0)
```
Output:
```
Product         A  B
Account_number
1               2  0
2               1  2
3               1  1
```


Question:
Pandas Pivot tables row subtotals I'm using Pandas 0.10.1 Considering this Dataframe:
```
Date       State   City    SalesToday  SalesMTD  SalesYTD
20130320     stA    ctA            20       400      1000
20130320     stA    ctB            30       500      1100
20130320     stB    ctC            10       500       900
20130320     stB    ctD            40       200      1300
20130320     stC    ctF            30       300       800
```
How can i group subtotals per state?
```
State   City  SalesToday  SalesMTD  SalesYTD
  stA    ALL          50       900      2100
  stA    ctA          20       400      1000
  stA    ctB          30       500      1100
```
I tried with a pivot table but i only can have subtotals in columns
```
table = pivot_table(df, values=['SalesToday', 'SalesMTD','SalesYTD'],\
                     rows=['State','City'], aggfunc=np.sum, margins=True)
```
I can achieve this on excel, with a pivot table.

Response:
If you put State and City not both in the rows, you'll get separate margins. Reshape and you get the table you're after:
```

```python
table = pivot_table(df, values=['SalesToday', 'SalesMTD','SalesYTD'],\
                     rows=['State'], cols=['City'], aggfunc=np.sum, margins=True)
```

```python
table.stack('City')
#Output
#SalesMTD  SalesToday  SalesYTD
#State City
#stA   All        900          50      2100
#      ctA        400          20      1000
#      ctB        500          30      1100
#stB   All        700          50      2200
#      ctC        500          10       900
#      ctD        200          40      1300
#stC   All        300          30       800
#      ctF        300          30       800
#All   All       1900         130      5100
#      ctA        400          20      1000
#      ctB        500          30      1100
#      ctC        500          10       900
#      ctD        200          40      1300
#      ctF        300          30       800
#```
#I admit this isn't totally obvious.
```


Question:
Anti-Join Pandas I have two tables and I would like to append them so that only all the data in table A is retained and data from table B is only added if its key is unique (Key values are unique in table A and B however in some cases a Key will occur in both table A and B). I think the way to do this will involve some sort of filtering join (anti-join) to get values in table B that do not occur in table A then append the two tables. I am familiar with R and this is the code I would use to do this in R.
```
library("dplyr")

## Filtering join to remove values already in "TableA" from "TableB"
FilteredTableB <- anti_join(TableB,TableA, by = "Key")

## Append "FilteredTableB" to "TableA"
CombinedTable <- bind_rows(TableA,FilteredTableB)
```
How would I achieve this in python?

Response:
indicator = True in merge command will tell you which join was applied by creating new column _merge with three possible values: left_only right_only both Keep right_only and left_only. That is it.
```
outer_join = TableA.merge(TableB, how = 'outer', indicator = True)

anti_join = outer_join[~(outer_join._merge == 'both')].drop('_merge', axis = 1)
```
easy! Here is a comparison with a solution from piRSquared: 1) When run on this example matching based on one column, piRSquared's solution is faster. 2) But it only works for matching on one column. If you want to match on several columns - my solution works just as fine as with one column. So it's up for you to decide.


Question:
Create a Pandas Dataframe by appending one row at a time [duplicate] This question already has answers here: Creating an empty Pandas DataFrame, and then filling it (8 answers) Closed 1 year ago. How do I create an empty DataFrame, then add rows, one by one? I created an empty DataFrame:
```
df = pd.DataFrame(columns=('lib', 'qty1', 'qty2'))
```
Then I can add a new row at the end and fill a single field with:
```
df = df._set_value(index=len(df), col='qty1', value=10.0)
```
It works for only one field at a time. What is a better way to add new row to df?

Response:
You can use df.loc[i], where the row with index i will be what you specify it to be in the dataframe.
```

```python
import pandas as pd
```

```python
from numpy.random import randint
```

```python
df = pd.DataFrame(columns=['lib', 'qty1', 'qty2'])
```

```python
for i in range(5):
```

```python
    df.loc[i] = ['name' + str(i)] + list(randint(10, size=2))
```

```python
df
#Output
#     lib qty1 qty2
#0  name0    3    3
#1  name1    2    4
#2  name2    2    8
#3  name3    2    1
#4  name4    9    6
#```
```


Question:
Groupby Pandas DataFrame and calculate mean and stdev of one column I have a Pandas DataFrame as below:
```none
a      b      c      d
0  Apple  3      5      7
1  Banana 4      4      8
2  Cherry 7      1      3
3  Apple  3      4      7
```
I would like to group the rows by column 'a' while replacing values in column 'c' by the mean of values in grouped rows and add another column with std deviation of the values in column 'c' whose mean has been calculated. The values in column 'b' or 'd' are constant for all rows being grouped. So, the desired output would be:
```none
a      b      c      d      e
0  Apple  3      4.5    7      0.707107
1  Banana 4      4      8      0
2  Cherry 7      1      3      0
```
What is the best way to achieve this?

Response:
You could use a groupby-agg operation:
```

```python
result = df.groupby(['a'], as_index=False).agg(
                      {'c':['mean','std'],'b':'first', 'd':'first'})
```
and then rename and reorder the columns:
```
```

```python
result.columns = ['a','c','e','b','d']
```

```python
result.reindex(columns=sorted(result.columns))
#Output
#a  b    c  d         e
#0   Apple  3  4.5  7  0.707107
#1  Banana  4  4.0  8       NaN
#2  Cherry  7  1.0  3       NaN
#```
#Pandas computes the sample std by default. To compute the population std:
#```
#def pop_std(x):
#    return x.std(ddof=0)
#
#result = df.groupby(['a'], as_index=False).agg({'c':['mean',pop_std],'b':'first', 'd':'first'})
#
#result.columns = ['a','c','e','b','d']
#result.reindex(columns=sorted(result.columns))
#```
#yields
#```
#a  b    c  d    e
#0   Apple  3  4.5  7  0.5
#1  Banana  4  4.0  8  0.0
#2  Cherry  7  1.0  3  0.0
#```
```


Question:
How to convert column with list of values into rows in Pandas DataFrame Hi I have a dataframe like this:
```
A             B
0:  some value    [[L1, L2]]
```
I want to change it into:
```
A             B
0:  some value    L1
1:  some value    L2
```
How can I do that?

Response:
Pandas >= 0.25
```
df1 = pd.DataFrame({'A':['a','b'],
               'B':[[['1', '2']],[['3', '4', '5']]]})
print(df1)
```
```
A   B
0   a   [[1, 2]]
1   b   [[3, 4, 5]]
```
```
df1 = df1.explode('B')
df1.explode('B')
```
```
A   B
0   a   1
0   a   2
1   b   3
1   b   4
1   b   5
```
I don't know how good this approach is but it works when you have a list of items.


Question:
pandas Series to Dataframe using Series indexes as columns I have a Series, like this:
```
series = pd.Series({'a': 1, 'b': 2, 'c': 3})
```
I want to convert it to a dataframe like this:
```
a   b   c
0   1   2   3
```
pd.Series.to_frame() doesn't work, it got result like,
```
0
a   1
b   2
c   3
```
How can I construct a DataFrame from Series, with index of Series as columns?

Response:
You can also try this :
```
df = DataFrame(series).transpose()
```
Using the transpose() function you can interchange the indices and the columns. The output looks like this :
```
a   b   c
0   1   2   3
```


Question: Difference between map, applymap and apply methods in Pandas Can you tell me when to use these vectorization methods with basic examples? I see that map is a Series method whereas the rest are DataFrame methods. I got confused about apply and applymap methods though. Why do we have two methods for applying a function to a DataFrame? Again, simple examples which illustrate the usage would be great!

Response:
apply works on a row / column basis of a DataFrame applymap works element-wise on a DataFrame map works element-wise on a Series Straight from Wes McKinney's Python for Data Analysis book, pg. 132 (I highly recommended this book): Another frequent operation is applying a function on 1D arrays to each column or row. DataFrame’s apply method does exactly this:
```

```python
frame = DataFrame(np.random.randn(4, 3), columns=list('bde'), index=['Utah', 'Ohio', 'Texas', 'Oregon'])
```

```python
frame
#Output
#b         d         e
#Utah   -0.029638  1.081563  1.280300
#Ohio    0.647747  0.831136 -1.549481
#Texas   0.513416 -0.884417  0.195343
#Oregon -0.485454 -0.477388 -0.309548
```

```python
f = lambda x: x.max() - x.min()
```

```python
frame.apply(f)
#Output
#b    1.133201
#d    1.965980
#e    2.829781
#dtype: float64
#```
#Many of the most common array statistics (like sum and mean) are DataFrame methods, so using apply is not necessary. Element-wise Python functions can be used, too. Suppose you wanted to compute a formatted string from each floating point value in frame. You can do this with applymap:
#```
```

```python
format = lambda x: '%.2f' % x
```

```python
frame.applymap(format)
#Output
#b      d      e
#Utah    -0.03   1.08   1.28
#Ohio     0.65   0.83  -1.55
#Texas    0.51  -0.88   0.20
#Oregon  -0.49  -0.48  -0.31
#```
#The reason for the name applymap is that Series has a map method for applying an element-wise function:
#```
```

```python
frame['e'].map(format)
#Output
#Utah       1.28
#Ohio      -1.55
#Texas      0.20
#Oregon    -0.31
#Name: e, dtype: object
#```
```


Question: In Pandas, how to delete rows from a Data Frame based on another Data Frame? I have 2 Data Frames, one named USERS and another named EXCLUDE. Both of them have a field named "email". Basically, I want to remove every row in USERS that has an email contained in EXCLUDE. How can I do it?

Response:
You can use boolean indexing and condition with isin, inverting boolean Series is by ~:
```
import pandas as pd

USERS = pd.DataFrame({'email':['
[email protected]
','
[email protected]
','
[email protected]
','
[email protected]
','
[email protected]
']})
print (USERS)
     email
0
[email protected]

1
[email protected]

2
[email protected]

3
[email protected]

4
[email protected]

EXCLUDE = pd.DataFrame({'email':['
[email protected]
','
[email protected]
']})
print (EXCLUDE)
     email
0
[email protected]

1
[email protected]
```
```
print (USERS.email.isin(EXCLUDE.email))
0     True
1    False
2    False
3    False
4     True
Name: email, dtype: bool

print (~USERS.email.isin(EXCLUDE.email))
0    False
1     True
2     True
3     True
4    False
Name: email, dtype: bool

print (USERS[~USERS.email.isin(EXCLUDE.email)])
     email
1
[email protected]

2
[email protected]

3
[email protected]
```
Another solution with merge:
```
df = pd.merge(USERS, EXCLUDE, how='outer', indicator=True)
print (df)
     email     _merge
0
[email protected]
       both
1
[email protected]
  left_only
2
[email protected]
  left_only
3
[email protected]
  left_only
4
[email protected]
       both

print (df.loc[df._merge == 'left_only', ['email']])
     email
1
[email protected]

2
[email protected]

3
[email protected]
```


Question: Merge a list of pandas dataframes I have a list of data frames and I need to merge them together using a unique column (date). Field names are different so concat is out. I can manually use df[0].merge(df[1],on='Date').merge(df[3],on='Date) etc. to merge each df one by one, but the issue is that the number of data frames in the list differs with user input. Is there any way to merge that just combines all data frames in a list in one go? Or perhaps some for loop that does that?

Response:
You can use reduce function where dfList is your list of data frames:
```
import pandas as pd
from functools import reduce
reduce(lambda x, y: pd.merge(x, y, on = 'Date'), dfList)
```
As a demo:
```
df = pd.DataFrame({'Date': [1,2,3,4], 'Value': [2,3,3,4]})
dfList = [df, df, df]
dfList

# [   Date  Value
#  0     1      2
#  1     2      3
#  2     3      3
#  3     4      4,    Date  Value
#  0     1      2
#  1     2      3
#  2     3      3
#  3     4      4,    Date  Value
#  0     1      2
#  1     2      3
#  2     3      3
#  3     4      4]

reduce(lambda x, y: pd.merge(x, y, on = 'Date'), dfList)
#   Date  Value_x  Value_y  Value
# 0    1        2        2      2
# 1    2        3        3      3
# 2    3        3        3      3
# 3    4        4        4      4
```


Question: How do I create test and train samples from one dataframe with pandas? I have a fairly large dataset in the form of a dataframe and I was wondering how I would be able to split the dataframe into two random samples (80% and 20%) for training and testing. Thanks!

Response:
Scikit Learn's train_test_split is a good one. It will split both numpy arrays and dataframes.
```
from sklearn.model_selection import train_test_split

train, test = train_test_split(df, test_size=0.2)
```


Question:
Convert timedelta64[ns] column to seconds in Python Pandas DataFrame A pandas DataFrame column duration contains timedelta64[ns] as shown. How can you convert them to seconds?
```
0   00:20:32
1   00:23:10
2   00:24:55
3   00:13:17
4   00:18:52
Name: duration, dtype: timedelta64[ns]
```
I tried the following
```
print df[:5]['duration'] / np.timedelta64(1, 's')
```
but got the error
```
Traceback (most recent call last):
  File "test.py", line 16, in <module>
    print df[0:5]['duration'] / np.timedelta64(1, 's')
  File "C:\Python27\lib\site-packages\pandas\core\series.py", line 130, in wrapper
    "addition and subtraction, but the operator [%s] was passed" % name)
TypeError: can only operate on a timedeltas for addition and subtraction, but the operator [__div__] was passed
```
Also tried
```
print df[:5]['duration'].astype('timedelta64[s]')
```
but received the error
```
Traceback (most recent call last):
  File "test.py", line 17, in <module>
    print df[:5]['duration'].astype('timedelta64[s]')
  File "C:\Python27\lib\site-packages\pandas\core\series.py", line 934, in astype
    values = com._astype_nansafe(self.values, dtype)
  File "C:\Python27\lib\site-packages\pandas\core\common.py", line 1653, in _astype_nansafe
    raise TypeError("cannot astype a timedelta from [%s] to [%s]" % (arr.dtype,dtype))
TypeError: cannot astype a timedelta from [timedelta64[ns]] to [timedelta64[s]]
```

Response:
Use the Series dt accessor to get access to the methods and attributes of a datetime (timedelta) series.
```

```python
s
#Output
#0   -1 days +23:45:14.304000
#1   -1 days +23:46:57.132000
#2   -1 days +23:49:25.913000
#3   -1 days +23:59:48.913000
#4            00:00:00.820000
#dtype: timedelta64[ns]
#>>>
```

```python
s.dt.total_seconds()
#Output
#0   -885.696
#1   -782.868
#2   -634.087
#3    -11.087
#4      0.820
#dtype: float64
#```
#There are other Pandas Series Accessors for String, Categorical, and Sparse data types.
```


Question:
How to split text in a column into multiple rows I'm working with a large csv file and the next to last column has a string of text that I want to split by a specific delimiter. I was wondering if there is a simple way to do this using pandas or python?
```
CustNum  CustomerName     ItemQty  Item   Seatblocks                 ItemExt
32363    McCartney, Paul      3     F04    2:218:10:4,6                   60
31316    Lennon, John        25     F01    1:13:36:1,12 1:13:37:1,13     300
```
I want to split by the space (' ') and then the colon (':') in the Seatblocks column, but each cell would result in a different number of columns. I have a function to rearrange the columns so the Seatblocks column is at the end of the sheet, but I'm not sure what to do from there. I can do it in excel with the built in text-to-columns function and a quick macro, but my dataset has too many records for excel to handle. Ultimately, I want to take records such John Lennon's and create multiple lines, with the info from each set of seats on a separate line.

Response:
This splits the Seatblocks by space and gives each its own row.
```

```python
df
#Output
#CustNum     CustomerName  ItemQty Item                 Seatblocks  ItemExt
#0    32363  McCartney, Paul        3  F04               2:218:10:4,6       60
#1    31316     Lennon, John       25  F01  1:13:36:1,12 1:13:37:1,13      300
```

```python
s = df['Seatblocks'].str.split(' ').apply(Series, 1).stack()
```

```python
s.index = s.index.droplevel(-1) # to line up with df's index
```

```python
s.name = 'Seatblocks' # needs a name to join
```

```python
s
#Output
#0    2:218:10:4,6
#1    1:13:36:1,12
#1    1:13:37:1,13
#Name: Seatblocks, dtype: object
```

```python
del df['Seatblocks']
```

```python
df.join(s)
#Output
#CustNum     CustomerName  ItemQty Item  ItemExt    Seatblocks
#0    32363  McCartney, Paul        3  F04       60  2:218:10:4,6
#1    31316     Lennon, John       25  F01      300  1:13:36:1,12
#1    31316     Lennon, John       25  F01      300  1:13:37:1,13
#```
#Or, to give each colon-separated string in its own column:
#```
```

```python
df.join(s.apply(lambda x: Series(x.split(':'))))
#Output
#CustNum     CustomerName  ItemQty Item  ItemExt  0    1   2     3
#0    32363  McCartney, Paul        3  F04       60  2  218  10   4,6
#1    31316     Lennon, John       25  F01      300  1   13  36  1,12
#1    31316     Lennon, John       25  F01      300  1   13  37  1,13
#```
#This is a little ugly, but maybe someone will chime in with a prettier solution.
```