Question:
How to split a dataframe string column into two columns? I have a data frame with one (string) column and I'd like to split it into two (string) columns, with one column header as 'fips' and the other 'row' My dataframe df looks like this:
```
row
0    00000 UNITED STATES
1    01000 ALABAMA
2    01001 Autauga County, AL
3    01003 Baldwin County, AL
4    01005 Barbour County, AL
```
I do not know how to use df.row.str[:] to achieve my goal of splitting the row cell. I can use df['fips'] = hello to add a new column and populate it with hello. Any ideas?
```
fips       row
0    00000 UNITED STATES
1    01000 ALABAMA
2    01001 Autauga County, AL
3    01003 Baldwin County, AL
4    01005 Barbour County, AL
```

Response:
TL;DR version: For the simple case of: I have a text column with a delimiter and I want two columns The simplest solution is:
```
df[['A', 'B']] = df['AB'].str.split(' ', n=1, expand=True)
```
You must use expand=True if your strings have a non-uniform number of splits and you want None to replace the missing values. Notice how, in either case, the .tolist() method is not necessary. Neither is zip(). In detail: Andy Hayden's solution is most excellent in demonstrating the power of the str.extract() method. But for a simple split over a known separator (like, splitting by dashes, or splitting by whitespace), the .str.split() method is enough1. It operates on a column (Series) of strings, and returns a column (Series) of lists:
```

```python
import pandas as pd
```

```python
df = pd.DataFrame({'AB': ['A1-B1', 'A2-B2']})
```

```python
df
```

AB
0  A1-B1
1  A2-B2

```python
df['AB_split'] = df['AB'].str.split('-')
```

```python
df
```

AB  AB_split
0  A1-B1  [A1, B1]
1  A2-B2  [A2, B2]
```
1: If you're unsure what the first two parameters of .str.split() do, I recommend the docs for the plain Python version of the method. But how do you go from: a column containing two-element lists to: two columns, each containing the respective element of the lists? Well, we need to take a closer look at the .str attribute of a column. It's a magical object that is used to collect methods that treat each element in a column as a string, and then apply the respective method in each element as efficient as possible:
```

```python
upper_lower_df = pd.DataFrame({"U": ["A", "B", "C"]})
```

```python
upper_lower_df
```

U
0  A
1  B
2  C

```python
upper_lower_df["L"] = upper_lower_df["U"].str.lower()
```

```python
upper_lower_df
```

U  L
0  A  a
1  B  b
2  C  c
```
But it also has an "indexing" interface for getting each element of a string by its index:
```

```python
df['AB'].str[0]
```

0    A
1    A
Name: AB, dtype: object

```python
df['AB'].str[1]
```

0    1
1    2
Name: AB, dtype: object
```
Of course, this indexing interface of .str doesn't really care if each element it's indexing is actually a string, as long as it can be indexed, so:
```

```python
df['AB'].str.split('-', 1).str[0]
```

0    A1
1    A2
Name: AB, dtype: object

```python
df['AB'].str.split('-', 1).str[1]
```

0    B1
1    B2
Name: AB, dtype: object
```
Then, it's a simple matter of taking advantage of the Python tuple unpacking of iterables to do
```

```python
df['A'], df['B'] = df['AB'].str.split('-', n=1).str
```

```python
df
```

AB  AB_split   A   B
0  A1-B1  [A1, B1]  A1  B1
1  A2-B2  [A2, B2]  A2  B2
```
Of course, getting a DataFrame out of splitting a column of strings is so useful that the .str.split() method can do it for you with the expand=True parameter:
```

```python
df['AB'].str.split('-', n=1, expand=True)
```

0   1
0  A1  B1
1  A2  B2
```
So, another way of accomplishing what we wanted is to do:
```

```python
df = df[['AB']]
```

```python
df
```

AB
0  A1-B1
1  A2-B2

```python
df.join(df['AB'].str.split('-', n=1, expand=True).rename(columns={0:'A', 1:'B'}))
```

AB   A   B
0  A1-B1  A1  B1
1  A2-B2  A2  B2
```
The expand=True version, although longer, has a distinct advantage over the tuple unpacking method. Tuple unpacking doesn't deal well with splits of different lengths:
```

```python
df = pd.DataFrame({'AB': ['A1-B1', 'A2-B2', 'A3-B3-C3']})
```

```python
df
#Output
#         AB
#0     A1-B1
#1     A2-B2
#2  A3-B3-C3
```

```python
df['A'], df['B'], df['C'] = df['AB'].str.split('-')
#Output
#Traceback (most recent call last):
#  [...]
#ValueError: Length of values does not match length of index
#>>>
#```
#But expand=True handles it nicely by placing None in the columns for which there aren't enough "splits":
#```
```

```python
df.join(
    df['AB'].str.split('-', expand=True).rename(
        columns={0:'A', 1:'B', 2:'C'}
    )
)
#Output
#         AB   A   B     C
#0     A1-B1  A1  B1  None
#1     A2-B2  A2  B2  None
#2  A3-B3-C3  A3  B3    C3
#```
```


Question:
Dropping infinite values from dataframes in pandas? How do I drop nan, inf, and -inf values from a DataFrame without resetting mode.use_inf_as_null? Can I tell dropna to include inf in its definition of missing values so that the following works?
```
df.dropna(subset=["col1", "col2"], how="all")
```

Response:
First replace() infs with NaN:
```
df.replace([np.inf, -np.inf], np.nan, inplace=True)
```
and then drop NaNs via dropna():
```
df.dropna(subset=["col1", "col2"], how="all", inplace=True)
```
For example:
```

```python
df = pd.DataFrame({"col1": [1, np.inf, -np.inf], "col2": [2, 3, np.nan]})
```

```python
df
#Output
#   col1  col2
#0   1.0   2.0
#1   inf   3.0
#2  -inf   NaN
```

```python
df.replace([np.inf, -np.inf], np.nan, inplace=True)
```

```python
df
#Output
#   col1  col2
#0   1.0   2.0
#1   NaN   3.0
#2   NaN   NaN
```

```python
df.dropna(subset=["col1", "col2"], how="all", inplace=True)
```

```python
df
#Output
#   col1  col2
#0   1.0   2.0
#1   NaN   3.0
#```
#The same method also works for Series.
```


Question:
How do I use Pandas group-by to get the sum? I am using this dataframe:
```
Fruit   Date      Name  Number
Apples  10/6/2016 Bob    7
Apples  10/6/2016 Bob    8
Apples  10/6/2016 Mike   9
Apples  10/7/2016 Steve 10
Apples  10/7/2016 Bob    1
Oranges 10/7/2016 Bob    2
Oranges 10/6/2016 Tom   15
Oranges 10/6/2016 Mike  57
Oranges 10/6/2016 Bob   65
Oranges 10/7/2016 Tony   1
Grapes  10/7/2016 Bob    1
Grapes  10/7/2016 Tom   87
Grapes  10/7/2016 Bob   22
Grapes  10/7/2016 Bob   12
Grapes  10/7/2016 Tony  15
```
I would like to aggregate this by Name and then by Fruit to get a total number of Fruit per Name. For example:
```
Bob,Apples,16
```
I tried grouping by Name and Fruit but how do I get the total number of Fruit?

Response:
Use GroupBy.sum:
```
df.groupby(['Fruit','Name']).sum()

Out[31]:
               Number
Fruit   Name
Apples  Bob        16
        Mike        9
        Steve      10
Grapes  Bob        35
        Tom        87
        Tony       15
Oranges Bob        67
        Mike       57
        Tom        15
        Tony        1
```
To specify the column to sum, use this: df.groupby(['Name', 'Fruit'])['Number'].sum()


Question:
How to show all columns' names on a large pandas dataframe? I have a dataframe that consist of hundreds of columns, and I need to see all column names. What I did:
```

```python
data_all2.columns
```
The output is:
```
#Output
#Index(['customer_id', 'incoming', 'outgoing', 'awan', 'bank', 'family', 'food',
#       'government', 'internet', 'isipulsa',
#       ...
#       'overdue_3months_feature78', 'overdue_3months_feature79',
#       'overdue_3months_feature80', 'overdue_3months_feature81',
#       'overdue_3months_feature82', 'overdue_3months_feature83',
#       'overdue_3months_feature84', 'overdue_3months_feature85',
#       'overdue_3months_feature86', 'loan_overdue_3months_total_y'],
#      dtype='object', length=102)
#```
#How do I show all columns, instead of a truncated list?
```

Response:
You can globally set printing options. I think this should work: Method 1:
```
pd.set_option('display.max_columns', None)
pd.set_option('display.max_rows', None)
```
Method 2:
```
pd.options.display.max_columns = None
pd.options.display.max_rows = None
```
This will allow you to see all column names & rows when you are doing .head(). None of the column name will be truncated. If you just want to see the column names you can do:
```
print(df.columns.tolist())
```


Question:
Convert Pandas column containing NaNs to dtype `int` I read data from a .csv file to a Pandas dataframe as below. For one of the columns, namely id, I want to specify the column type as int. The problem is the id series has missing/empty values. When I try to cast the id column to integer while reading the .csv, I get:
```
df= pd.read_csv("data.csv", dtype={'id': int})
error: Integer column has NA values
```
Alternatively, I tried to convert the column type after reading as below, but this time I get:
```
df= pd.read_csv("data.csv")
df[['id']] = df[['id']].astype(int)
error: Cannot convert NA to integer
```
How can I tackle this?

Response:
In version 0.24.+ pandas has gained the ability to hold integer dtypes with missing values. Nullable Integer Data Type. Pandas can represent integer data with possibly missing values using arrays.IntegerArray. This is an extension types implemented within pandas. It is not the default dtype for integers, and will not be inferred; you must explicitly pass the dtype into array() or Series:
```
arr = pd.array([1, 2, np.nan], dtype=pd.Int64Dtype())
pd.Series(arr)

0      1
1      2
2    NaN
dtype: Int64
```
For convert column to nullable integers use:
```
df['myCol'] = df['myCol'].astype('Int64')
```


Question:
Split a Pandas column of lists into multiple columns I have a Pandas DataFrame with one column:
```
import pandas as pd

df = pd.DataFrame({"teams": [["SF", "NYG"] for _ in range(7)]})

       teams
0  [SF, NYG]
1  [SF, NYG]
2  [SF, NYG]
3  [SF, NYG]
4  [SF, NYG]
5  [SF, NYG]
6  [SF, NYG]
```
How can split this column of lists into two columns? Desired result:
```
team1 team2
0    SF   NYG
1    SF   NYG
2    SF   NYG
3    SF   NYG
4    SF   NYG
5    SF   NYG
6    SF   NYG
```

Response:
You can use the DataFrame constructor with lists created by to_list:
```
import pandas as pd

d1 = {'teams': [['SF', 'NYG'],['SF', 'NYG'],['SF', 'NYG'],
                ['SF', 'NYG'],['SF', 'NYG'],['SF', 'NYG'],['SF', 'NYG']]}
df2 = pd.DataFrame(d1)
print (df2)
       teams
0  [SF, NYG]
1  [SF, NYG]
2  [SF, NYG]
3  [SF, NYG]
4  [SF, NYG]
5  [SF, NYG]
6  [SF, NYG]
```
```
df2[['team1','team2']] = pd.DataFrame(df2.teams.tolist(), index= df2.index)
print (df2)
       teams team1 team2
0  [SF, NYG]    SF   NYG
1  [SF, NYG]    SF   NYG
2  [SF, NYG]    SF   NYG
3  [SF, NYG]    SF   NYG
4  [SF, NYG]    SF   NYG
5  [SF, NYG]    SF   NYG
6  [SF, NYG]    SF   NYG
```
And for a new DataFrame:
```
df3 = pd.DataFrame(df2['teams'].to_list(), columns=['team1','team2'])
print (df3)
  team1 team2
0    SF   NYG
1    SF   NYG
2    SF   NYG
3    SF   NYG
4    SF   NYG
5    SF   NYG
6    SF   NYG
```
A solution with apply(pd.Series) is very slow:
```
#7k rows
df2 = pd.concat([df2]*1000).reset_index(drop=True)

```python
%timeit df2['teams'].apply(pd.Series)
1.79 s ± 52.5 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
```

```python
%timeit pd.DataFrame(df2['teams'].to_list(), columns=['team1','team2'])
1.63 ms ± 54.3 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
```
```


Question: Apply pandas function to column to create multiple new columns? How to do this in pandas: I have a function extract_text_features on a single text column, returning multiple output columns. Specifically, the function returns 6 values. The function works, however there doesn't seem to be any proper return type (pandas DataFrame/ numpy array/ Python list) such that the output can get correctly assigned df.ix[: ,10:16] = df.textcol.map(extract_text_features) So I think I need to drop back to iterating with df.iterrows(), as per this? UPDATE: Iterating with df.iterrows() is at least 20x slower, so I surrendered and split out the function into six distinct .map(lambda ...) calls. UPDATE 2: this question was asked back around v0.11.0, before the useability of df.apply was improved or df.assign() was added in v0.16. Hence much of the question and answers are not too relevant since then.

Response:
I usually do this using zip:
```

```python
df = pd.DataFrame([[i] for i in range(10)], columns=['num'])
```

```python
df
#Output
#    num
#0    0
#1    1
#2    2
#3    3
#4    4
#5    5
#6    6
#7    7
#8    8
#9    9
```

```python
def powers(x):
```

```python
    return x, x**2, x**3, x**4, x**5, x**6
```

```python
df['p1'], df['p2'], df['p3'], df['p4'], df['p5'], df['p6'] = \
```

```python
    zip(*df['num'].map(powers))
```

```python
df
#Output
#        num     p1      p2      p3      p4      p5      p6
#0       0       0       0       0       0       0       0
#1       1       1       1       1       1       1       1
#2       2       2       4       8       16      32      64
#3       3       3       9       27      81      243     729
#4       4       4       16      64      256     1024    4096
#5       5       5       25      125     625     3125    15625
#6       6       6       36      216     1296    7776    46656
#7       7       7       49      343     2401    16807   117649
#8       8       8       64      512     4096    32768   262144
#9       9       9       81      729     6561    59049   531441
#```
```


Question:
Pandas: Setting no. of max rows I have a problem viewing the following DataFrame:
```
n = 100
foo = DataFrame(index=range(n))
foo['floats'] = np.random.randn(n)
foo
```
The problem is that it does not print all rows per default in ipython notebook, but I have to slice to view the resulting rows. Even the following option does not change the output:
```
pd.set_option('display.max_rows', 500)
```
Does anyone know how to display the whole array?

Response:
Set display.max_rows:
```
pd.set_option('display.max_rows', 500)
```
For older versions of pandas (<=0.11.0) you need to change both display.height and display.max_rows.
```
pd.set_option('display.height', 500)
pd.set_option('display.max_rows', 500)
```
See also pd.describe_option('display'). You can set an option only temporarily for this one time like this:
```
from IPython.display import display
with pd.option_context('display.max_rows', 100, 'display.max_columns', 10):
    display(df) #need display to show the dataframe when using with in jupyter
    #some pandas stuff
```
You can also reset an option back to its default value like this: pd.reset_option('display.max_rows') And reset all of them back: pd.reset_option('all')


Question:
Extracting specific selected columns to new DataFrame as a copy I have a pandas DataFrame with 4 columns and I want to create a new DataFrame that only has three of the columns. This question is similar to: Extracting specific columns from a data frame but for pandas not R. The following code does not work, raises an error, and is certainly not the pandas way to do it.
```py
import pandas as pd
old = pd.DataFrame({'A' : [4,5], 'B' : [10,20], 'C' : [100,50], 'D' : [-30,-50]})
new = pd.DataFrame(zip(old.A, old.C, old.D))
# raises TypeError: data argument can't be an iterator
```
What is the pandas way to do it?

Response:
There is a way of doing this and it actually looks similar to R
```
new = old[['A', 'C', 'D']].copy()
```
Here you are just selecting the columns you want from the original data frame and creating a variable for those. If you want to modify the new dataframe at all you'll probably want to use .copy() to avoid a SettingWithCopyWarning. An alternative method is to use filter which will create a copy by default:
```
new = old.filter(['A','B','D'], axis=1)
```
Finally, depending on the number of columns in your original dataframe, it might be more succinct to express this using a drop (this will also create a copy by default):
```
new = old.drop('B', axis=1)
```


Question:
Pandas create empty DataFrame with only column names I have a dynamic DataFrame which works fine, but when there are no data to be added into the DataFrame I get an error. And therefore I need a solution to create an empty DataFrame with only the column names. For now I have something like this:
```py
df = pd.DataFrame(columns=COLUMN_NAMES) # Note that there is no row data inserted.
```
PS: It is important that the column names would still appear in a DataFrame. But when I use it like this I get something like that as a result:
```none
Index([], dtype='object')
Empty DataFrame
```
The "Empty DataFrame" part is good! But instead of the Index thing I need to still display the columns. An important thing that I found out: I am converting this DataFrame to a PDF using Jinja2, so therefore I'm calling out a method to first output it to HTML like that:
```py
df.to_html()
```
This is where the columns get lost I think. In general, I followed this example: http://pbpython.com/pdf-reports.html. The css is also from the link. That's what I do to send the dataframe to the PDF:
```py
env = Environment(loader=FileSystemLoader('.'))
template = env.get_template("pdf_report_template.html")
template_vars = {"my_dataframe": df.to_html()}

html_out = template.render(template_vars)
HTML(string=html_out).write_pdf("my_pdf.pdf", stylesheets=["pdf_report_style.css"])
```

Response:
You can create an empty DataFrame with either column names or an Index:
```

```python
import pandas as pd
```

```python
df = pd.DataFrame(columns=['A','B','C','D','E','F','G'])
```

```python
df
#Output
#Empty DataFrame
#Columns: [A, B, C, D, E, F, G]
#Index: []
#```
#Or
#```
```

```python
df = pd.DataFrame(index=range(1,10))
```

```python
df
#Output
#Empty DataFrame
#Columns: []
#Index: [1, 2, 3, 4, 5, 6, 7, 8, 9]
#```
#Edit: Even after your amendment with the .to_html, I can't reproduce. This:
#```
#df = pd.DataFrame(columns=['A','B','C','D','E','F','G'])
#df.to_html('test.html')
#```
#Produces:
#```
#

#

#

#
#      A
#      B
#      C
#      D
#      E
#      F
#      G
#

#

#

#

#

#```
```


Question:
Merge two dataframes by index [duplicate] This question already has answers here: Pandas Merging 101 (8 answers) What are the 'levels', 'keys', and names arguments for in Pandas' concat function? (1 answer) Closed 7 months ago. I have the following dataframes:
```
> df1
  id  begin conditional confidence discoveryTechnique
0 278    56       false        0.0                  1
1 421    18       false        0.0                  1

> df2
   concept
0  A
1  B
```
How do I merge on the indices to get:
```
id  begin conditional confidence discoveryTechnique concept
0 278    56       false        0.0                  1       A
1 421    18       false        0.0                  1       B
```
I ask because it is my understanding that merge() i.e. df1.merge(df2) uses columns to do the matching. In fact, doing this I get:
```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/dist-packages/pandas/core/frame.py", line 4618, in merge
    copy=copy, indicator=indicator)
  File "/usr/local/lib/python2.7/dist-packages/pandas/tools/merge.py", line 58, in merge
    copy=copy, indicator=indicator)
  File "/usr/local/lib/python2.7/dist-packages/pandas/tools/merge.py", line 491, in __init__
    self._validate_specification()
  File "/usr/local/lib/python2.7/dist-packages/pandas/tools/merge.py", line 812, in _validate_specification
    raise MergeError('No common columns to perform merge on')
pandas.tools.merge.MergeError: No common columns to perform merge on
```
Is it bad practice to merge on index? Is it impossible? If so, how can I shift the index into a new column called "index"?

Response:
Use merge, which is an inner join by default:
```
pd.merge(df1, df2, left_index=True, right_index=True)
```
Or join, which is a left join by default:
```
df1.join(df2)
```
Or concat, which is an outer join by default:
```
pd.concat([df1, df2], axis=1)
```
Samples:
```
df1 = pd.DataFrame({'a':range(6),
                    'b':[5,3,6,9,2,4]}, index=list('abcdef'))

print (df1)
   a  b
a  0  5
b  1  3
c  2  6
d  3  9
e  4  2
f  5  4

df2 = pd.DataFrame({'c':range(4),
                    'd':[10,20,30, 40]}, index=list('abhi'))

print (df2)
   c   d
a  0  10
b  1  20
h  2  30
i  3  40
```
```
# Default inner join
df3 = pd.merge(df1, df2, left_index=True, right_index=True)
print (df3)
   a  b  c   d
a  0  5  0  10
b  1  3  1  20

# Default left join
df4 = df1.join(df2)
print (df4)
   a  b    c     d
a  0  5  0.0  10.0
b  1  3  1.0  20.0
c  2  6  NaN   NaN
d  3  9  NaN   NaN
e  4  2  NaN   NaN
f  5  4  NaN   NaN

# Default outer join
df5 = pd.concat([df1, df2], axis=1)
print (df5)
     a    b    c     d
a  0.0  5.0  0.0  10.0
b  1.0  3.0  1.0  20.0
c  2.0  6.0  NaN   NaN
d  3.0  9.0  NaN   NaN
e  4.0  2.0  NaN   NaN
f  5.0  4.0  NaN   NaN
h  NaN  NaN  2.0  30.0
i  NaN  NaN  3.0  40.0
```


Question: How to determine whether a Pandas Column contains a particular value I am trying to determine whether there is an entry in a Pandas column that has a particular value. I tried to do this with if x in df['id']. I thought this was working, except when I fed it a value that I knew was not in the column 43 in df['id'] it still returned True. When I subset to a data frame only containing entries matching the missing id df[df['id'] == 43] there are, obviously, no entries in it. How to I determine if a column in a Pandas data frame contains a particular value and why doesn't my current method work? (FYI, I have the same problem when I use the implementation in this answer to a similar question).

Response:
in of a Series checks whether the value is in the index:
```

```python
s = pd.Series(list('abc'))
```

```python
s
#Output
#0    a
#1    b
#2    c
#dtype: object
```

```python
1 in s
#Output
#True
```

```python
'a' in s
#Output
#False
#```
#One option is to see if it's in unique values:
#```
```

```python
s.unique()
#Output
#array(['a', 'b', 'c'], dtype=object)
```

```python
'a' in s.unique()
#Output
#True
#```
#or a python set:
#```
```

```python
set(s)
#Output
#{'a', 'b', 'c'}
```

```python
'a' in set(s)
#Output
#True
#```
#As pointed out by @DSM, it may be more efficient (especially if you're just doing this for one value) to just use in directly on the values:
#```
```

```python
s.values
#Output
#array(['a', 'b', 'c'], dtype=object)
```

```python
'a' in s.values
#Output
#True
#```
```


Question:
Appending to an empty DataFrame in Pandas? Is it possible to append to an empty data frame that doesn't contain any indices or columns? I have tried to do this, but keep getting an empty dataframe at the end. e.g.
```
import pandas as pd

df = pd.DataFrame()
data = ['some kind of data here' --> I have checked the type already, and it is a dataframe]
df.append(data)
```
The result looks like this:
```
Empty DataFrame
Columns: []
Index: []
```

Response:
This should work:
```

```python
df = pd.DataFrame()
```

```python
data = pd.DataFrame({"A": range(3)})
```

```python
df = df.append(data)
```

```python
df
```

A
0  0
1  1
2  2
```
Since the append doesn't happen in-place, so you'll have to store the output if you want it:
```

```python
df = pd.DataFrame()
```

```python
data = pd.DataFrame({"A": range(3)})
```

```python
df.append(data)  # without storing
```

```python
df
#Output
#Empty DataFrame
#Columns: []
#Index: []
```

```python
df = df.append(data)
```

```python
df
#Output
#   A
#0  0
#1  1
#2  2
#```
```


Question:
How to take column-slices of dataframe in pandas I load some machine learning data from a CSV file. The first 2 columns are observations and the remaining columns are features. Currently, I do the following:
```
data = pandas.read_csv('mydata.csv')
```
which gives something like:
```
data = pandas.DataFrame(np.random.rand(10,5), columns = list('abcde'))
```
I'd like to slice this dataframe in two dataframes: one containing the columns a and b and one containing the columns c, d and e. It is not possible to write something like
```
observations = data[:'c']
features = data['c':]
```
I'm not sure what the best method is. Do I need a pd.Panel? By the way, I find dataframe indexing pretty inconsistent: data['a'] is permitted, but data[0] is not. On the other side, data['a':] is not permitted but data[0:] is. Is there a practical reason for this? This is really confusing if columns are indexed by Int, given that data[0] != data[0:1]

Response:
2017 Answer - pandas 0.20: .ix is deprecated. Use .loc See the deprecation in the docs .loc uses label based indexing to select both rows and columns. The labels being the values of the index or the columns. Slicing with .loc includes the last element. Let's assume we have a DataFrame with the following columns: foo, bar, quz, ant, cat, sat, dat.
```
# selects all rows and all columns beginning at 'foo' up to and including 'sat'
df.loc[:, 'foo':'sat']
# foo bar quz ant cat sat
```
.loc accepts the same slice notation that Python lists do for both row and columns. Slice notation being start:stop:step
```
# slice from 'foo' to 'cat' by every 2nd column
df.loc[:, 'foo':'cat':2]
# foo quz cat

# slice from the beginning to 'bar'
df.loc[:, :'bar']
# foo bar

# slice from 'quz' to the end by 3
df.loc[:, 'quz'::3]
# quz sat

# attempt from 'sat' to 'bar'
df.loc[:, 'sat':'bar']
# no columns returned

# slice from 'sat' to 'bar'
df.loc[:, 'sat':'bar':-1]
sat cat ant quz bar

# slice notation is syntatic sugar for the slice function
# slice from 'quz' to the end by 2 with slice function
df.loc[:, slice('quz',None, 2)]
# quz cat dat

# select specific columns with a list
# select columns foo, bar and dat
df.loc[:, ['foo','bar','dat']]
# foo bar dat
```
You can slice by rows and columns. For instance, if you have 5 rows with labels v, w, x, y, z
```
# slice from 'w' to 'y' and 'foo' to 'ant' by 3
df.loc['w':'y', 'foo':'ant':3]
#    foo ant
# w
# x
# y
```


Question:
How do I get a list of all the duplicate items using pandas in python? I have a list of items that likely has some export issues. I would like to get a list of the duplicate items so I can manually compare them. When I try to use pandas duplicated method, it only returns the first duplicate. Is there a a way to get all of the duplicates and not just the first one? A small subsection of my dataset looks like this:
```
ID,ENROLLMENT_DATE,TRAINER_MANAGING,TRAINER_OPERATOR,FIRST_VISIT_DATE
1536D,12-Feb-12,"06DA1B3-Lebanon NH",,15-Feb-12
F15D,18-May-12,"06405B2-Lebanon NH",,25-Jul-12
8096,8-Aug-12,"0643D38-Hanover NH","0643D38-Hanover NH",25-Jun-12
A036,1-Apr-12,"06CB8CF-Hanover NH","06CB8CF-Hanover NH",9-Aug-12
8944,19-Feb-12,"06D26AD-Hanover NH",,4-Feb-12
1004E,8-Jun-12,"06388B2-Lebanon NH",,24-Dec-11
11795,3-Jul-12,"0649597-White River VT","0649597-White River VT",30-Mar-12
30D7,11-Nov-12,"06D95A3-Hanover NH","06D95A3-Hanover NH",30-Nov-11
3AE2,21-Feb-12,"06405B2-Lebanon NH",,26-Oct-12
B0FE,17-Feb-12,"06D1B9D-Hartland VT",,16-Feb-12
127A1,11-Dec-11,"064456E-Hanover NH","064456E-Hanover NH",11-Nov-12
161FF,20-Feb-12,"0643D38-Hanover NH","0643D38-Hanover NH",3-Jul-12
A036,30-Nov-11,"063B208-Randolph VT","063B208-Randolph VT",
475B,25-Sep-12,"06D26AD-Hanover NH",,5-Nov-12
151A3,7-Mar-12,"06388B2-Lebanon NH",,16-Nov-12
CA62,3-Jan-12,,,
D31B,18-Dec-11,"06405B2-Lebanon NH",,9-Jan-12
20F5,8-Jul-12,"0669C50-Randolph VT",,3-Feb-12
8096,19-Dec-11,"0649597-White River VT","0649597-White River VT",9-Apr-12
14E48,1-Aug-12,"06D3206-Hanover NH",,
177F8,20-Aug-12,"063B208-Randolph VT","063B208-Randolph VT",5-May-12
553E,11-Oct-12,"06D95A3-Hanover NH","06D95A3-Hanover NH",8-Mar-12
12D5F,18-Jul-12,"0649597-White River VT","0649597-White River VT",2-Nov-12
C6DC,13-Apr-12,"06388B2-Lebanon NH",,
11795,27-Feb-12,"0643D38-Hanover NH","0643D38-Hanover NH",19-Jun-12
17B43,11-Aug-12,,,22-Oct-12
A036,11-Aug-12,"06D3206-Hanover NH",,19-Jun-12
```
My code looks like this currently:
```
df_bigdata_duplicates = df_bigdata[df_bigdata.duplicated(cols='ID')]
```
There area a couple duplicate items. But, when I use the above code, I only get the first item. In the API reference, I see how I can get the last item, but I would like to have all of them so I can visually inspect them to see why I am getting the discrepancy. So, in this example I would like to get all three A036 entries and both 11795 entries and any other duplicated entries, instead of the just first one. Any help is most appreciated.

Response:
Method #1: print all rows where the ID is one of the IDs in duplicated:
```

```python
import pandas as pd
```

```python
df = pd.read_csv("dup.csv")
```

```python
ids = df["ID"]
```

```python
df[ids.isin(ids[ids.duplicated()])].sort_values("ID")
#Output
#       ID ENROLLMENT_DATE        TRAINER_MANAGING        TRAINER_OPERATOR FIRST_VISIT_DATE
#24  11795       27-Feb-12      0643D38-Hanover NH      0643D38-Hanover NH        19-Jun-12
#6   11795        3-Jul-12  0649597-White River VT  0649597-White River VT        30-Mar-12
#18   8096       19-Dec-11  0649597-White River VT  0649597-White River VT         9-Apr-12
#2    8096        8-Aug-12      0643D38-Hanover NH      0643D38-Hanover NH        25-Jun-12
#12   A036       30-Nov-11     063B208-Randolph VT     063B208-Randolph VT              NaN
#3    A036        1-Apr-12      06CB8CF-Hanover NH      06CB8CF-Hanover NH         9-Aug-12
#26   A036       11-Aug-12      06D3206-Hanover NH                     NaN        19-Jun-12
#```
#but I couldn't think of a nice way to prevent repeating ids so many times. I prefer method #2: groupby on the ID.
#```
```

```python
pd.concat(g for _, g in df.groupby("ID") if len(g) > 1)
#Output
#       ID ENROLLMENT_DATE        TRAINER_MANAGING        TRAINER_OPERATOR FIRST_VISIT_DATE
#6   11795        3-Jul-12  0649597-White River VT  0649597-White River VT        30-Mar-12
#24  11795       27-Feb-12      0643D38-Hanover NH      0643D38-Hanover NH        19-Jun-12
#2    8096        8-Aug-12      0643D38-Hanover NH      0643D38-Hanover NH        25-Jun-12
#18   8096       19-Dec-11  0649597-White River VT  0649597-White River VT         9-Apr-12
#3    A036        1-Apr-12      06CB8CF-Hanover NH      06CB8CF-Hanover NH         9-Aug-12
#12   A036       30-Nov-11     063B208-Randolph VT     063B208-Randolph VT              NaN
#26   A036       11-Aug-12      06D3206-Hanover NH                     NaN        19-Jun-12
#```
```


Question:
How to iterate over columns of a pandas dataframe I have this code using Pandas in Python:
```
all_data = {}
for ticker in ['FIUIX', 'FSAIX', 'FSAVX', 'FSTMX']:
    all_data[ticker] = web.get_data_yahoo(ticker, '1/1/2010', '1/1/2015')

prices = DataFrame({tic: data['Adj Close'] for tic, data in all_data.iteritems()})
returns = prices.pct_change()
```
I know I can run a regression like this:
```
regs = sm.OLS(returns.FIUIX,returns.FSTMX).fit()
```
but how can I do this for each column in the dataframe? Specifically, how can I iterate over columns, in order to run the regression on each? Specifically, I want to regress each other ticker symbol (FIUIX, FSAIX and FSAVX) on FSTMX, and store the residuals for each regression. I've tried various versions of the following, but nothing I've tried gives the desired result:
```
resids = {}
for k in returns.keys():
    reg = sm.OLS(returns[k],returns.FSTMX).fit()
    resids[k] = reg.resid
```
Is there something wrong with the returns[k] part of the code? How can I use the k value to access a column? Or else is there a simpler approach?

Response:
Old answer:
```
for column in df:
    print(df[column])
```
The previous answer still works, but was added around the time of pandas 0.16.0. Better versions are available. Now you can do:
```
for series_name, series in df.items():
    print(series_name)
    print(series)
```


Question:
Pandas DataFrame Groupby two columns and get counts I have a pandas dataframe in the following format:
```py
df = pd.DataFrame([
    [1.1, 1.1, 1.1, 2.6, 2.5, 3.4,2.6,2.6,3.4,3.4,2.6,1.1,1.1,3.3],
    list('AAABBBBABCBDDD'),
    [1.1, 1.7, 2.5, 2.6, 3.3, 3.8,4.0,4.2,4.3,4.5,4.6,4.7,4.7,4.8],
    ['x/y/z','x/y','x/y/z/n','x/u','x','x/u/v','x/y/z','x','x/u/v/b','-','x/y','x/y/z','x','x/u/v/w'],
    ['1','3','3','2','4','2','5','3','6','3','5','1','1','1']
]).T
df.columns = ['col1','col2','col3','col4','col5']
```
df:
```none
col1 col2 col3     col4 col5
0   1.1    A  1.1    x/y/z    1
1   1.1    A  1.7      x/y    3
2   1.1    A  2.5  x/y/z/n    3
3   2.6    B  2.6      x/u    2
4   2.5    B  3.3        x    4
5   3.4    B  3.8    x/u/v    2
6   2.6    B    4    x/y/z    5
7   2.6    A  4.2        x    3
8   3.4    B  4.3  x/u/v/b    6
9   3.4    C  4.5        -    3
10  2.6    B  4.6      x/y    5
11  1.1    D  4.7    x/y/z    1
12  1.1    D  4.7        x    1
13  3.3    D  4.8  x/u/v/w    1
```
I want to get the count by each row like following. Expected Output:
```none
col5 col2 count
1    A      1
     D      3
2    B      2
etc...
```
How to get my expected output? And I want to find largest count for each 'col2' value?

Response:
You are looking for size:
```

```python
df.groupby(['col5', 'col2']).size()
#Output
#col5  col2
#1     A       1
#      D       3
#2     B       2
#3     A       3
#      C       1
#4     B       1
#5     B       2
#6     B       1
#dtype: int64
#```
#To get the same answer as waitingkuo (the "second question"), but slightly cleaner, is to groupby the level:
#```
```

```python
df.groupby(['col5', 'col2']).size().groupby(level=1).max()
#Output
#col2
#A       3
#B       2
#C       1
#D       3
#dtype: int64
#```
```


Question: pandas DataFrame: replace nan values with average of columns I've got a pandas DataFrame filled mostly with real numbers, but there is a few nan values in it as well. How can I replace the nans with averages of columns where they are? This question is very similar to this one: numpy array: replace nan values with average of columns but, unfortunately, the solution given there doesn't work for a pandas DataFrame.

Response:
You can simply use DataFrame.fillna to fill the nan's directly:
```

```python
df
#Output
#A         B         C
#0 -0.166919  0.979728 -0.632955
#1 -0.297953 -0.912674 -1.365463
#2 -0.120211 -0.540679 -0.680481
#3       NaN -2.027325  1.533582
#4       NaN       NaN  0.461821
#5 -0.788073       NaN       NaN
#6 -0.916080 -0.612343       NaN
#7 -0.887858  1.033826       NaN
#8  1.948430  1.025011 -2.982224
#9  0.019698 -0.795876 -0.046431
```

```python
df.mean()
#Output
#A   -0.151121
#B   -0.231291
#C   -0.530307
#dtype: float64
```

```python
df.fillna(df.mean())
#Output
#A         B         C
#0 -0.166919  0.979728 -0.632955
#1 -0.297953 -0.912674 -1.365463
#2 -0.120211 -0.540679 -0.680481
#3 -0.151121 -2.027325  1.533582
#4 -0.151121 -0.231291  0.461821
#5 -0.788073 -0.231291 -0.530307
#6 -0.916080 -0.612343 -0.530307
#7 -0.887858  1.033826 -0.530307
#8  1.948430  1.025011 -2.982224
#9  0.019698 -0.795876 -0.046431
#```
#The docstring of fillna says that value should be a scalar or a dict, however, it seems to work with a Series as well. If you want to pass a dict, you could use df.mean().to_dict().
```


Question:
How to add multiple columns to pandas dataframe in one assignment I'm trying to figure out how to add multiple columns to pandas simultaneously with Pandas. I would like to do this in one step rather than multiple repeated steps.
```
import pandas as pd

data = {'col_1': [0, 1, 2, 3],
        'col_2': [4, 5, 6, 7]}
df = pd.DataFrame(data)
```
I thought this would work here...
```
df[['column_new_1', 'column_new_2', 'column_new_3']] = [np.nan, 'dogs', 3]
```

Response:
I would have expected your syntax to work too. The problem arises because when you create new columns with the column-list syntax (df[[new1, new2]] = ...), pandas requires that the right hand side be a DataFrame (note that it doesn't actually matter if the columns of the DataFrame have the same names as the columns you are creating). Your syntax works fine for assigning scalar values to existing columns, and pandas is also happy to assign scalar values to a new column using the single-column syntax (df[new1] = ...). So the solution is either to convert this into several single-column assignments, or create a suitable DataFrame for the right-hand side. Here are several approaches that will work:
```
import pandas as pd
import numpy as np

df = pd.DataFrame({
    'col_1': [0, 1, 2, 3],
    'col_2': [4, 5, 6, 7]
})
```
Then one of the following: 1) Three assignments in one, using iterable unpacking
```
df['column_new_1'], df['column_new_2'], df['column_new_3'] = np.nan, 'dogs', 3
```
2) Use DataFrame() to expand a single row to match the index
```
df[['column_new_1', 'column_new_2', 'column_new_3']] = pd.DataFrame([[np.nan, 'dogs', 3]], index=df.index)
```
3) Combine with a temporary DataFrame using pd.concat
```
df = pd.concat(
    [
        df,
        pd.DataFrame(
            [[np.nan, 'dogs', 3]],
            index=df.index,
            columns=['column_new_1', 'column_new_2', 'column_new_3']
        )
    ], axis=1
)
```
4) Combine with a temporary DataFrame using .join This is similar to 3, but may be less efficient.
```
df = df.join(pd.DataFrame(
    [[np.nan, 'dogs', 3]],
    index=df.index,
    columns=['column_new_1', 'column_new_2', 'column_new_3']
))
```
5) Use a dictionary instead of the lists used in 3 and 4 This is a more "natural" way to create the temporary DataFrame than the previous two. Note that in Python 3.5 or earlier, the new columns will be sorted alphabetically.
```
df = df.join(pd.DataFrame(
    {
        'column_new_1': np.nan,
        'column_new_2': 'dogs',
        'column_new_3': 3
    }, index=df.index
))
```
6) Use .assign() with multiple column arguments This may be the winner in Python 3.6+. But like the previous one, the new columns will be sorted alphabetically in earlier versions of Python.
```
df = df.assign(column_new_1=np.nan, column_new_2='dogs', column_new_3=3)
```
7) Create new columns, then assign all values at once Based on this answer. This is interesting, but I don't know when it would be worth the trouble.
```
new_cols = ['column_new_1', 'column_new_2', 'column_new_3']
new_vals = [np.nan, 'dogs', 3]
df = df.reindex(columns=df.columns.tolist() + new_cols)   # add empty cols
df[new_cols] = new_vals  # multi-column assignment works for existing cols
```
8) Three separate assignments In the end, it's hard to beat this.
```
df['column_new_1'] = np.nan
df['column_new_2'] = 'dogs'
df['column_new_3'] = 3
```
Note: many of these options have already been covered in other questions: Add multiple columns to DataFrame and set them equal to an existing column Is it possible to add several columns at once to a pandas DataFrame? Add multiple empty columns to pandas DataFrame


Question: How do I convert a Pandas series or index to a NumPy array? [duplicate] This question already has answers here: Convert Pandas dataframe to NumPy array (17 answers) Closed 5 years ago. The community reviewed whether to reopen this question 3 years ago and left it closed: Original close reason(s) were not resolved How can I get the index or column of a DataFrame as a NumPy array or Python list?

Response:
To get a NumPy array, you should use the values attribute:
```

```python
df = pd.DataFrame({'A': [1, 2, 3], 'B': [4, 5, 6]}, index=['a', 'b', 'c']); df
   A  B
a  1  4
b  2  5
c  3  6
```

```python
df.index.values
#Output
#array(['a', 'b', 'c'], dtype=object)
#```
#This accesses how the data is already stored, so there isn't any need for a conversion. Note: This attribute is also available for many other pandas objects.
#```
```

```python
df['A'].values
#Output
#Out[16]: array([1, 2, 3])
#```
#To get the index as a list, call tolist:
#```
```

```python
df.index.tolist()
#Output
#['a', 'b', 'c']
#```
#And similarly, for columns.
```


Question:
Take multiple lists into dataframe How do I take multiple lists and put them as different columns in a python dataframe? I tried this solution but had some trouble. Attempt 1: Have three lists, and zip them together and use that res = zip(lst1,lst2,lst3) Yields just one column Attempt 2:
```
percentile_list = pd.DataFrame({'lst1Tite' : [lst1],
                                'lst2Tite' : [lst2],
                                'lst3Tite' : [lst3] },
                                columns=['lst1Tite','lst1Tite', 'lst1Tite'])
```
yields either one row by 3 columns (the way above) or if I transpose it is 3 rows and 1 column How do I get a 100 row (length of each independent list) by 3 column (three lists) pandas dataframe?

Response:
I think you're almost there, try removing the extra square brackets around the lst's (Also you don't need to specify the column names when you're creating a dataframe from a dict like this):
```
import pandas as pd
lst1 = range(100)
lst2 = range(100)
lst3 = range(100)
percentile_list = pd.DataFrame(
    {'lst1Title': lst1,
     'lst2Title': lst2,
     'lst3Title': lst3
    })

percentile_list
    lst1Title  lst2Title  lst3Title
0          0         0         0
1          1         1         1
2          2         2         2
3          3         3         3
4          4         4         4
5          5         5         5
6          6         6         6
...
```
If you need a more performant solution you can use np.column_stack rather than zip as in your first attempt, this has around a 2x speedup on the example here, however comes at bit of a cost of readability in my opinion:
```
import numpy as np
percentile_list = pd.DataFrame(np.column_stack([lst1, lst2, lst3]),
                               columns=['lst1Title', 'lst2Title', 'lst3Title'])
```


Question:
Load data from txt with pandas I am loading a txt file containig a mix of float and string data. I want to store them in an array where I can access each element. Now I am just doing
```py
import pandas as pd

data = pd.read_csv('output_list.txt', header = None)
print data
```
Each line in the input file looks like the following:
```none
1 0 2000.0 70.2836942112 1347.28369421 /file_address.txt
```
Now the data are imported as a unique column. How can I divide it, so to store different elements separately (so I can call data[i,j])? And how can I define a header?

Response:
You can use:
```
data = pd.read_csv('output_list.txt', sep=" ", header=None)
data.columns = ["a", "b", "c", "etc."]
```
Add sep=" " in your code, leaving a blank space between the quotes. So pandas can detect spaces between values and sort in columns. Data columns is for naming your columns.


Question:
Random row selection in Pandas dataframe Is there a way to select random rows from a DataFrame in Pandas. In R, using the car package, there is a useful function some(x, n) which is similar to head but selects, in this example, 10 rows at random from x. I have also looked at the slicing documentation and there seems to be nothing equivalent. Update Now using version 20. There is a sample method.
```
df.sample(n)
```

Response:
With pandas version 0.16.1 and up, there is now a DataFrame.sample method built-in:
```
import pandas

df = pandas.DataFrame(pandas.np.random.random(100))

# Randomly sample 70% of your dataframe
df_percent = df.sample(frac=0.7)

# Randomly sample 7 elements from your dataframe
df_elements = df.sample(n=7)
```
For either approach above, you can get the rest of the rows by doing:
```
df_rest = df.loc[~df.index.isin(df_percent.index)]
```
Per Pedram's comment, if you would like to get reproducible samples, pass the random_state parameter.
```
df_percent = df.sample(frac=0.7, random_state=42)
```


Question:
Pandas DataFrame: replace all values in a column, based on condition I have a simple DataFrame like the following: Team First Season Total Games 0 Dallas Cowboys 1960 894 1 Chicago Bears 1920 1357 2 Green Bay Packers 1921 1339 3 Miami Dolphins 1966 792 4 Baltimore Ravens 1996 326 5 San Francisco 49ers 1950 1003 I want to select all values from the First Season column and replace those that are over 1990 by 1. In this example, only Baltimore Ravens would have the 1996 replaced by 1 (keeping the rest of the data intact). I have used the following:
```
df.loc[(df['First Season'] > 1990)] = 1
```
But, it replaces all the values in that row by 1, not just the values in the 'First Season' column. How can I replace just the values from that column?

Response:
You need to select that column:
```

```python
df.loc[df['First Season'] > 1990, 'First Season'] = 1
df
#Output
#Team  First Season  Total Games
#0      Dallas Cowboys          1960          894
#1       Chicago Bears          1920         1357
#2   Green Bay Packers          1921         1339
#3      Miami Dolphins          1966          792
#4    Baltimore Ravens             1          326
#5  San Franciso 49ers          1950         1003
#```
#So the syntax here is:
#```
#df.loc[<mask>(here mask is generating the labels to index) , <optional column(s)> ]
#```
#You can check the docs and also the 10 minutes to pandas which shows the semantics EDIT If you want to generate a boolean indicator then you can just use the boolean condition to generate a boolean Series and cast the dtype to int this will convert True and False to 1 and 0 respectively:
#```
```

```python
df['First Season'] = (df['First Season'] > 1990).astype(int)
df
#Output
#Team  First Season  Total Games
#0      Dallas Cowboys             0          894
#1       Chicago Bears             0         1357
#2   Green Bay Packers             0         1339
#3      Miami Dolphins             0          792
#4    Baltimore Ravens             1          326
#5  San Franciso 49ers             0         1003
#```
```


Question:
pandas get column average/mean I can't get the average or mean of a column in pandas. A have a dataframe. Neither of things I tried below gives me the average of the column weight
```

```python
allDF
#Output
#         ID           birthyear  weight
#0        619040       1962       0.1231231
#1        600161       1963       0.981742
#2      25602033       1963       1.3123124
#3        624870       1987       0.94212
#```
#The following returns several values, not one:
#```
#allDF[['weight']].mean(axis=1)
#```
#So does this:
#```
#allDF.groupby('weight').mean()
#```
```

Response:
If you only want the mean of the weight column, select the column (which is a Series) and call .mean():
```

```python
df
#Output
#ID  birthyear    weight
#0    619040       1962  0.123123
#1    600161       1963  0.981742
#2  25602033       1963  1.312312
#3    624870       1987  0.942120
```

```python
df.loc[:, 'weight'].mean()
#Output
#0.83982437500000007
#```
```


Question:
How to display pandas DataFrame of floats using a format string for columns? I would like to display a pandas dataframe with a given format using print() and the IPython display(). For example:
```
df = pd.DataFrame([123.4567, 234.5678, 345.6789, 456.7890],
                  index=['foo','bar','baz','quux'],
                  columns=['cost'])
print df

         cost
foo   123.4567
bar   234.5678
baz   345.6789
quux  456.7890
```
I would like to somehow coerce this into printing
```
cost
foo   $123.46
bar   $234.57
baz   $345.68
quux  $456.79
```
without having to modify the data itself or create a copy, just change the way it is displayed. How can I do this?

Response:
```
import pandas as pd
pd.options.display.float_format = '${:,.2f}'.format
df = pd.DataFrame([123.4567, 234.5678, 345.6789, 456.7890],
                  index=['foo','bar','baz','quux'],
                  columns=['cost'])
print(df)
```
yields
```
cost
foo  $123.46
bar  $234.57
baz  $345.68
quux $456.79
```
but this only works if you want every float to be formatted with a dollar sign. Otherwise, if you want dollar formatting for some floats only, then I think you'll have to pre-modify the dataframe (converting those floats to strings):
```
import pandas as pd
df = pd.DataFrame([123.4567, 234.5678, 345.6789, 456.7890],
                  index=['foo','bar','baz','quux'],
                  columns=['cost'])
df['foo'] = df['cost']
df['cost'] = df['cost'].map('${:,.2f}'.format)
print(df)
```
yields
```
cost       foo
foo   $123.46  123.4567
bar   $234.57  234.5678
baz   $345.68  345.6789
quux  $456.79  456.7890
```


Question:
Search for "does-not-contain" on a DataFrame in pandas I've done some searching and can't figure out how to filter a dataframe by
```
df["col"].str.contains(word)
```
however I'm wondering if there is a way to do the reverse: filter a dataframe by that set's compliment. eg: to the effect of
```
!(df["col"].str.contains(word))
```
Can this be done through a DataFrame method?

Response:
You can use the invert (~) operator (which acts like a not for boolean data):
```
new_df = df[~df["col"].str.contains(word)]
```
where new_df is the copy returned by RHS. contains also accepts a regular expression... If the above throws a ValueError or TypeError, the reason is likely because you have mixed datatypes, so use na=False:
```
new_df = df[~df["col"].str.contains(word, na=False)]
```
Or,
```
new_df = df[df["col"].str.contains(word) == False]
```


Question:
Format / Suppress Scientific Notation from Pandas Aggregation Results How can one modify the format for the output from a groupby operation in pandas that produces scientific notation for very large numbers? I know how to do string formatting in python but I'm at a loss when it comes to applying it here.
```
df1.groupby('dept')['data1'].sum()

dept
value1       1.192433e+08
value2       1.293066e+08
value3       1.077142e+08
```
This suppresses the scientific notation if I convert to string but now I'm just wondering how to string format and add decimals.
```
sum_sales_dept.astype(str)
```

Response:
Granted, the answer I linked in the comments is not very helpful. You can specify your own string converter like so.
```

```python
pd.set_option('display.float_format', lambda x: '%.3f' % x)
```

```python
Series(np.random.randn(3))*1000000000
#Output
#0    -757322420.605
#1   -1436160588.997
#2   -1235116117.064
#dtype: float64
#```
#I'm not sure if that's the preferred way to do this, but it works. Converting numbers to strings purely for aesthetic purposes seems like a bad idea, but if you have a good reason, this is one way:
#```
```

```python
Series(np.random.randn(3)).apply(lambda x: '%.3f' % x)
#Output
#0     0.026
#1    -0.482
#2    -0.694
#dtype: object
#```
```


Question: Find column whose name contains a specific string I have a dataframe with column names, and I want to find the one that contains a certain string, but does not exactly match it. I'm searching for 'spike' in column names like 'spike-2', 'hey spike', 'spiked-in' (the 'spike' part is always continuous). I want the column name to be returned as a string or a variable, so I access the column later with df['name'] or df[name] as normal. I've tried to find ways to do this, to no avail. Any tips?

Response:
Just iterate over DataFrame.columns, now this is an example in which you will end up with a list of column names that match:
```
import pandas as pd

data = {'spike-2': [1,2,3], 'hey spke': [4,5,6], 'spiked-in': [7,8,9], 'no': [10,11,12]}
df = pd.DataFrame(data)

spike_cols = [col for col in df.columns if 'spike' in col]
print(list(df.columns))
print(spike_cols)
```
Output:
```
['hey spke', 'no', 'spike-2', 'spiked-in']
['spike-2', 'spiked-in']
```
Explanation: df.columns returns a list of column names [col for col in df.columns if 'spike' in col] iterates over the list df.columns with the variable col and adds it to the resulting list if col contains 'spike'. This syntax is list comprehension. If you only want the resulting data set with the columns that match you can do this:
```
df2 = df.filter(regex='spike')
print(df2)
```
Output:
```
spike-2  spiked-in
0        1          7
1        2          8
2        3          9
```


Question:
Replacing blank values (white space) with NaN in pandas I want to find all values in a Pandas dataframe that contain whitespace (any arbitrary amount) and replace those values with NaNs. Any ideas how this can be improved? Basically I want to turn this:
```none
A    B    C
2000-01-01 -0.532681  foo    0
2000-01-02  1.490752  bar    1
2000-01-03 -1.387326  foo    2
2000-01-04  0.814772  baz
2000-01-05 -0.222552         4
2000-01-06 -1.176781  qux
```
Into this:
```none
A     B     C
2000-01-01 -0.532681   foo     0
2000-01-02  1.490752   bar     1
2000-01-03 -1.387326   foo     2
2000-01-04  0.814772   baz   NaN
2000-01-05 -0.222552   NaN     4
2000-01-06 -1.176781   qux   NaN
```
I've managed to do it with the code below, but man is it ugly. It's not Pythonic and I'm sure it's not the most efficient use of pandas either. I loop through each column and do boolean replacement against a column mask generated by applying a function that does a regex search of each value, matching on whitespace.
```
for i in df.columns:
    df[i][df[i].apply(lambda i: True if re.search('^\s*$', str(i)) else False)]=None
```
It could be optimized a bit by only iterating through fields that could contain empty strings:
```
if df[i].dtype == np.dtype('object')
```
But that's not much of an improvement And finally, this code sets the target strings to None, which works with Pandas' functions like fillna(), but it would be nice for completeness if I could actually insert a NaN directly instead of None.

Response:
I think df.replace() does the job, since pandas 0.13:
```
df = pd.DataFrame([
    [-0.532681, 'foo', 0],
    [1.490752, 'bar', 1],
    [-1.387326, 'foo', 2],
    [0.814772, 'baz', ' '],
    [-0.222552, '   ', 4],
    [-1.176781,  'qux', '  '],
], columns='A B C'.split(), index=pd.date_range('2000-01-01','2000-01-06'))

# replace field that's entirely space (or empty) with NaN
print(df.replace(r'^\s*$', np.nan, regex=True))
```
Produces:
```none
A    B   C
2000-01-01 -0.532681  foo   0
2000-01-02  1.490752  bar   1
2000-01-03 -1.387326  foo   2
2000-01-04  0.814772  baz NaN
2000-01-05 -0.222552  NaN   4
2000-01-06 -1.176781  qux NaN
```
As Temak pointed it out, use df.replace(r'^\s+$', np.nan, regex=True) in case your valid data contains white spaces.


Question: Find difference between two data frames I have two data frames df1 and df2, where df2 is a subset of df1. How do I get a new data frame (df3) which is the difference between the two data frames? In other word, a data frame that has all the rows/columns in df1 that are not in df2?

Response:
By using drop_duplicates
```
pd.concat([df1,df2]).drop_duplicates(keep=False)
```
Update : The above method only works for those data frames that don't already have duplicates themselves. For example:
```
df1=pd.DataFrame({'A':[1,2,3,3],'B':[2,3,4,4]})
df2=pd.DataFrame({'A':[1],'B':[2]})
```
It will output like below , which is wrong Wrong Output :
```
pd.concat([df1, df2]).drop_duplicates(keep=False)
Out[655]:
   A  B
1  2  3
```
Correct Output
```
Out[656]:
   A  B
1  2  3
2  3  4
3  3  4
```
How to achieve that? Method 1: Using isin with tuple
```
df1[~df1.apply(tuple,1).isin(df2.apply(tuple,1))]
Out[657]:
   A  B
1  2  3
2  3  4
3  3  4
```
Method 2: merge with indicator
```
df1.merge(df2,indicator = True, how='left').loc[lambda x : x['_merge']!='both']
Out[421]:
   A  B     _merge
1  2  3  left_only
2  3  4  left_only
3  3  4  left_only
```


Question: How to find which columns contain any NaN value in Pandas dataframe Given a pandas dataframe containing possible NaN values scattered here and there: Question: How do I determine which columns contain NaN values? In particular, can I get a list of the column names containing NaNs?

Response:
UPDATE: using Pandas 0.22.0 Newer Pandas versions have new methods 'DataFrame.isna()' and 'DataFrame.notna()'
```

```python
df
#Output
#a    b  c
#0  NaN  7.0  0
#1  0.0  NaN  4
#2  2.0  NaN  4
#3  1.0  7.0  0
#4  1.0  3.0  9
#5  7.0  4.0  9
#6  2.0  6.0  9
#7  9.0  6.0  4
#8  3.0  0.0  9
#9  9.0  0.0  1
```

```python
df.isna().any()
#Output
#a     True
#b     True
#c    False
#dtype: bool
#```
#as list of columns:
#```
```

```python
df.columns[df.isna().any()].tolist()
#Output
#['a', 'b']
#```
#to select those columns (containing at least one NaN value):
#```
```

```python
df.loc[:, df.isna().any()]
#Output
#a    b
#0  NaN  7.0
#1  0.0  NaN
#2  2.0  NaN
#3  1.0  7.0
#4  1.0  3.0
#5  7.0  4.0
#6  2.0  6.0
#7  9.0  6.0
#8  3.0  0.0
#9  9.0  0.0
#```
#OLD answer: Try to use isnull():
#```
```

```python
df
#Output
#a    b  c
#0  NaN  7.0  0
#1  0.0  NaN  4
#2  2.0  NaN  4
#3  1.0  7.0  0
#4  1.0  3.0  9
#5  7.0  4.0  9
#6  2.0  6.0  9
#7  9.0  6.0  4
#8  3.0  0.0  9
#9  9.0  0.0  1
```

```python
pd.isnull(df).sum() > 0
#Output
#a     True
#b     True
#c    False
#dtype: bool
#```
#or as @root proposed clearer version:
#```
```

```python
df.isnull().any()
#Output
#a     True
#b     True
#c    False
#dtype: bool
```

```python
df.columns[df.isnull().any()].tolist()
#Output
#['a', 'b']
#```
#to select a subset - all columns containing at least one NaN value:
#```
```

```python
df.loc[:, df.isnull().any()]
#Output
#a    b
#0  NaN  7.0
#1  0.0  NaN
#2  2.0  NaN
#3  1.0  7.0
#4  1.0  3.0
#5  7.0  4.0
#6  2.0  6.0
#7  9.0  6.0
#8  3.0  0.0
#9  9.0  0.0
#```
```


Question:
Convert columns into rows with Pandas So my dataset has some information by location for n dates. The problem is each date is actually a different column header. For example the CSV looks like
```
location    name    Jan-2010    Feb-2010    March-2010
A           "test"  12          20          30
B           "foo"   18          20          25
```
What I would like is for it to look like
```
location    name    Date        Value
A           "test"  Jan-2010    12
A           "test"  Feb-2010    20
A           "test"  March-2010  30
B           "foo"   Jan-2010    18
B           "foo"   Feb-2010    20
B           "foo"   March-2010  25
```
My problem is I don't know how many dates are in the column (though I know they will always start after name)

Response:
Use .melt:
```
df.melt(id_vars=["location", "name"],
        var_name="Date",
        value_name="Value")

  location    name        Date  Value
0        A  "test"    Jan-2010     12
1        B   "foo"    Jan-2010     18
2        A  "test"    Feb-2010     20
3        B   "foo"    Feb-2010     20
4        A  "test"  March-2010     30
5        B   "foo"  March-2010     25
```
Old(er) versions: <0.20 You can use pd.melt to get most of the way there, and then sort:
```

```python
df
#Output
#  location  name  Jan-2010  Feb-2010  March-2010
#0        A  test        12        20          30
#1        B   foo        18        20          25
```

```python
df2 = pd.melt(df,
#Output
#                  id_vars=["location", "name"],
#                  var_name="Date",
#                  value_name="Value")
```

```python
df2
#Output
#  location  name        Date  Value
#0        A  test    Jan-2010     12
#1        B   foo    Jan-2010     18
#2        A  test    Feb-2010     20
#3        B   foo    Feb-2010     20
#4        A  test  March-2010     30
#5        B   foo  March-2010     25
```

```python
df2 = df2.sort(["location", "name"])
```

```python
df2
#Output
#  location  name        Date  Value
#0        A  test    Jan-2010     12
#2        A  test    Feb-2010     20
#4        A  test  March-2010     30
#1        B   foo    Jan-2010     18
#3        B   foo    Feb-2010     20
#5        B   foo  March-2010     25
#```
#(Might want to throw in a .reset_index(drop=True), just to keep the output clean.) Note: pd.DataFrame.sort has been deprecated in favour of pd.DataFrame.sort_values.
```


Question:
Counting unique values in a column in pandas dataframe like in Qlik? If I have a table like this:
```
df = pd.DataFrame({
         'hID': [101, 102, 103, 101, 102, 104, 105, 101],
         'dID': [10, 11, 12, 10, 11, 10, 12, 10],
         'uID': ['James', 'Henry', 'Abe', 'James', 'Henry', 'Brian', 'Claude', 'James'],
         'mID': ['A', 'B', 'A', 'B', 'A', 'A', 'A', 'C']
})
```
I can do count(distinct hID) in Qlik to come up with count of 5 for unique hID. How do I do that in python using a pandas dataframe? Or maybe a numpy array? Similarly, if were to do count(hID) I will get 8 in Qlik. What is the equivalent way to do it in pandas?

Response:
Count distinct values, use nunique:
```
df['hID'].nunique()
5
```
Count only non-null values, use count:
```
df['hID'].count()
8
```
Count total values including null values, use the size attribute:
```
df['hID'].size
8
```
Edit to add condition Use boolean indexing:
```
df.loc[df['mID']=='A','hID'].agg(['nunique','count','size'])
```
OR using query:
```
df.query('mID == "A"')['hID'].agg(['nunique','count','size'])
```
Output:
```
nunique    5
count      5
size       5
Name: hID, dtype: int64
```


Question:
Pandas convert dataframe to array of tuples I have manipulated some data using pandas and now I want to carry out a batch save back to the database. This requires me to convert the dataframe into an array of tuples, with each tuple corresponding to a "row" of the dataframe. My DataFrame looks something like:
```

```python
data_set
#Output
#index data_date   data_1  data_2
#0  14303 2012-02-17  24.75   25.03
#1  12009 2012-02-16  25.00   25.07
#2  11830 2012-02-15  24.99   25.15
#3  6274  2012-02-14  24.68   25.05
#4  2302  2012-02-13  24.62   24.77
#5  14085 2012-02-10  24.38   24.61
#```
#I want to convert it to an array of tuples like:
#```
#[(datetime.date(2012,2,17),24.75,25.03),
#(datetime.date(2012,2,16),25.00,25.07),
#...etc. ]
#```
#Any suggestion on how I can efficiently do this?
```

Response:
```
list(data_set.itertuples(index=False))
```
As of 17.1, the above will return a list of namedtuples. If you want a list of ordinary tuples, pass name=None as an argument:
```
list(data_set.itertuples(index=False, name=None))
```


Question:
python pandas: apply a function with arguments to a series I want to apply a function with arguments to a series in python pandas:
```
x = my_series.apply(my_function, more_arguments_1)
y = my_series.apply(my_function, more_arguments_2)
...
```
The documentation describes support for an apply method, but it doesn't accept any arguments. Is there a different method that accepts arguments? Alternatively, am I missing a simple workaround? Update (October 2017): Note that since this question was originally asked that pandas apply() has been updated to handle positional and keyword arguments and the documentation link above now reflects that and shows how to include either type of argument.

Response:
Newer versions of pandas do allow you to pass extra arguments (see the new documentation). So now you can do:
```
my_series.apply(your_function, args=(2,3,4), extra_kw=1)
```
The positional arguments are added after the element of the series. For older version of pandas: The documentation explains this clearly. The apply method accepts a python function which should have a single parameter. If you want to pass more parameters you should use functools.partial as suggested by Joel Cornett in his comment. An example:
```

```python
import functools
```

```python
import operator
```

```python
add_3 = functools.partial(operator.add,3)
```

```python
add_3(2)
#Output
#5
```

```python
add_3(7)
#Output
#10
#```
#You can also pass keyword arguments using partial. Another way would be to create a lambda:
#```
#my_series.apply((lambda x: your_func(a,b,c,d,...,x)))
#```
#But I think using partial is better.
```


Question:
How to plot in multiple subplots I am a little confused about how this code works:
```
fig, axes = plt.subplots(nrows=2, ncols=2)
plt.show()
```
How does the fig, axes work in this case? What does it do? Also why wouldn't this work to do the same thing:
```
fig = plt.figure()
axes = fig.subplots(nrows=2, ncols=2)
```

Response:
There are several ways to do it. The subplots method creates the figure along with the subplots that are then stored in the ax array. For example:
```
import matplotlib.pyplot as plt

x = range(10)
y = range(10)

fig, ax = plt.subplots(nrows=2, ncols=2)

for row in ax:
    for col in row:
        col.plot(x, y)

plt.show()
```
However, something like this will also work, it's not so "clean" though since you are creating a figure with subplots and then add on top of them:
```
fig = plt.figure()

plt.subplot(2, 2, 1)
plt.plot(x, y)

plt.subplot(2, 2, 2)
plt.plot(x, y)

plt.subplot(2, 2, 3)
plt.plot(x, y)

plt.subplot(2, 2, 4)
plt.plot(x, y)

plt.show()
```


Question:
Right way to reverse a pandas DataFrame? Here is my code:
```
import pandas as pd

data = pd.DataFrame({'Odd':[1,3,5,6,7,9], 'Even':[0,2,4,6,8,10]})

for i in reversed(data):
    print(data['Odd'], data['Even'])
```
When I run this code, i get the following error:
```
Traceback (most recent call last):
  File "C:\Python33\lib\site-packages\pandas\core\generic.py", line 665, in _get_item_cache
    return cache[item]
KeyError: 5

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\*****\Documents\******\********\****.py", line 5, in <module>
    for i in reversed(data):
  File "C:\Python33\lib\site-packages\pandas\core\frame.py", line 2003, in __getitem__
    return self._get_item_cache(key)
  File "C:\Python33\lib\site-packages\pandas\core\generic.py", line 667, in _get_item_cache
    values = self._data.get(item)
  File "C:\Python33\lib\site-packages\pandas\core\internals.py", line 1656, in get
    _, block = self._find_block(item)
  File "C:\Python33\lib\site-packages\pandas\core\internals.py", line 1936, in _find_block
    self._check_have(item)
  File "C:\Python33\lib\site-packages\pandas\core\internals.py", line 1943, in _check_have
    raise KeyError('no item named %s' % com.pprint_thing(item))
KeyError: 'no item named 5'
```
Why am I getting this error? How can I fix that? What is the right way to reverse pandas.DataFrame?

Response:
```
data.reindex(index=data.index[::-1])
```
or simply:
```
data.iloc[::-1]
```
will reverse your data frame, if you want to have a for loop which goes from down to up you may do:
```
for idx in reversed(data.index):
    print(idx, data.loc[idx, 'Even'], data.loc[idx, 'Odd'])
```
or
```
for idx in reversed(data.index):
    print(idx, data.Even[idx], data.Odd[idx])
```
You are getting an error because reversed first calls data.__len__() which returns 6. Then it tries to call data[j - 1] for j in range(6, 0, -1), and the first call would be data[5]; but in pandas dataframe data[5] means column 5, and there is no column 5 so it will throw an exception. ( see docs )


Question:
How to replace text in a string column of a Pandas dataframe? I have a column in my dataframe like this:
```
range
"(2,30)"
"(50,290)"
"(400,1000)"
...
```
and I want to replace the , comma with - dash. I'm currently using this method but nothing is changed.
```
org_info_exc['range'].replace(',', '-', inplace=True)
```
Can anybody help?

Response:
Use the vectorised str method replace:
```
df['range'] = df['range'].str.replace(',','-')

df
      range
0    (2-30)
1  (50-290)
```
EDIT: so if we look at what you tried and why it didn't work:
```
df['range'].replace(',','-',inplace=True)
```
from the docs we see this description: str or regex: str: string exactly matching to_replace will be replaced with value So because the str values do not match, no replacement occurs, compare with the following:
```
df = pd.DataFrame({'range':['(2,30)',',']})
df['range'].replace(',','-', inplace=True)

df['range']

0    (2,30)
1         -
Name: range, dtype: object
```
here we get an exact match on the second row and the replacement occurs.


Question:
How do I retrieve the number of columns in a Pandas data frame? How do you programmatically retrieve the number of columns in a pandas dataframe? I was hoping for something like:
```
df.num_columns
```

Response:
Like so:
```
import pandas as pd
df = pd.DataFrame({"pear": [1,2,3], "apple": [2,3,4], "orange": [3,4,5]})

len(df.columns)
3
```


Question:
getting the index of a row in a pandas apply function I am trying to access the index of a row in a function applied across an entire DataFrame in Pandas. I have something like this:
```
df = pandas.DataFrame([[1,2,3],[4,5,6]], columns=['a','b','c'])

```python
df
#Output
#   a  b  c
#0  1  2  3
#1  4  5  6
#```
#and I'll define a function that access elements with a given row
#```
#def rowFunc(row):
#    return row['a'] + row['b'] * row['c']
#```
#I can apply it like so:
#```
#df['d'] = df.apply(rowFunc, axis=1)
```

```python
df
#Output
#   a  b  c   d
#0  1  2  3   7
#1  4  5  6  34
#```
#Awesome! Now what if I want to incorporate the index into my function? The index of any given row in this DataFrame before adding d would be Index([u'a', u'b', u'c', u'd'], dtype='object'), but I want the 0 and 1. So I can't just access row.index. I know I could create a temporary column in the table where I store the index, but I'm wondering if it is stored in the row object somewhere.
```

Response:
To access the index in this case you access the name attribute:
```

```python
df = pd.DataFrame([[1,2,3],[4,5,6]], columns=['a','b','c'])
def rowFunc(row):
    return row['a'] + row['b'] * row['c']

def rowIndex(row):
    return row.name
df['d'] = df.apply(rowFunc, axis=1)
df['rowIndex'] = df.apply(rowIndex, axis=1)
df
#Output
#a  b  c   d  rowIndex
#0  1  2  3   7         0
#1  4  5  6  34         1
#```
#Note that if this is really what you are trying to do that the following works and is much faster:
#```
```

```python
df['d'] = df['a'] + df['b'] * df['c']
df
#Output
#a  b  c   d
#0  1  2  3   7
#1  4  5  6  34
```

```python
%timeit df['a'] + df['b'] * df['c']
%timeit df.apply(rowIndex, axis=1)
10000 loops, best of 3: 163 µs per loop
1000 loops, best of 3: 286 µs per loop
```
EDIT Looking at this question 3+ years later, you could just do:
```
```

```python
df['d'],df['rowIndex'] = df['a'] + df['b'] * df['c'], df.index
df
#Output
#a  b  c   d  rowIndex
#0  1  2  3   7         0
#1  4  5  6  34         1
#```
#but assuming it isn't as trivial as this, whatever your rowFunc is really doing, you should look to use the vectorised functions, and then use them against the df index:
#```
```

```python
df['newCol'] = df['a'] + df['b'] + df['c'] + df.index
df
#Output
#a  b  c   d  rowIndex  newCol
#0  1  2  3   7         0       6
#1  4  5  6  34         1      16
#```
```


Question:
Whether to use apply vs transform on a group object, to subtract two columns and get mean Consider the following dataframe:
```
columns = ['A', 'B', 'C', 'D']
records = [
    ['foo', 'one', 0.162003, 0.087469],
    ['bar', 'one', -1.156319, -1.5262719999999999],
    ['foo', 'two', 0.833892, -1.666304],
    ['bar', 'three', -2.026673, -0.32205700000000004],
    ['foo', 'two', 0.41145200000000004, -0.9543709999999999],
    ['bar', 'two', 0.765878, -0.095968],
    ['foo', 'one', -0.65489, 0.678091],
    ['foo', 'three', -1.789842, -1.130922]
]
df = pd.DataFrame.from_records(records, columns=columns)

"""
     A      B         C         D
0  foo    one  0.162003  0.087469
1  bar    one -1.156319 -1.526272
2  foo    two  0.833892 -1.666304
3  bar  three -2.026673 -0.322057
4  foo    two  0.411452 -0.954371
5  bar    two  0.765878 -0.095968
6  foo    one -0.654890  0.678091
7  foo  three -1.789842 -1.130922
"""
```
The following commands work:
```
df.groupby('A').apply(lambda x: (x['C'] - x['D']))
df.groupby('A').apply(lambda x: (x['C'] - x['D']).mean())
```
but none of the following work:
```
df.groupby('A').transform(lambda x: (x['C'] - x['D']))
# KeyError or ValueError: could not broadcast input array from shape (5) into shape (5,3)

df.groupby('A').transform(lambda x: (x['C'] - x['D']).mean())
# KeyError or TypeError: cannot concatenate a non-NDFrame object
```
Why? The example on the documentation seems to suggest that calling transform on a group allows one to do row-wise operation processing:
```
# Note that the following suggests row-wise operation (x.mean is the column mean)
zscore = lambda x: (x - x.mean()) / x.std()
transformed = ts.groupby(key).transform(zscore)
```
In other words, I thought that transform is essentially a specific type of apply (the one that does not aggregate). Where am I wrong? For reference, below is the construction of the original dataframe above:
```
df = pd.DataFrame({'A' : ['foo', 'bar', 'foo', 'bar',
                          'foo', 'bar', 'foo', 'foo'],
                   'B' : ['one', 'one', 'two', 'three',
                         'two', 'two', 'one', 'three'],
                   'C' : randn(8), 'D' : randn(8)})
```

Response:
Two major differences between apply and transform There are two major differences between the transform and apply groupby methods. Input: apply implicitly passes all the columns for each group as a DataFrame to the custom function. while transform passes each column for each group individually as a Series to the custom function. Output: The custom function passed to apply can return a scalar, or a Series or DataFrame (or numpy array or even list). The custom function passed to transform must return a sequence (a one dimensional Series, array or list) the same length as the group. So, transform works on just one Series at a time and apply works on the entire DataFrame at once. Inspecting the custom function It can help quite a bit to inspect the input to your custom function passed to apply or transform. Examples Let's create some sample data and inspect the groups so that you can see what I am talking about:
```
import pandas as pd
import numpy as np
df = pd.DataFrame({'State':['Texas', 'Texas', 'Florida', 'Florida'],
                   'a':[4,5,1,3], 'b':[6,10,3,11]})

     State  a   b
0    Texas  4   6
1    Texas  5  10
2  Florida  1   3
3  Florida  3  11
```
Let's create a simple custom function that prints out the type of the implicitly passed object and then raises an exception so that execution can be stopped.
```
def inspect(x):
    print(type(x))
    raise
```
Now let's pass this function to both the groupby apply and transform methods to see what object is passed to it:
```
df.groupby('State').apply(inspect)

<class 'pandas.core.frame.DataFrame'>
<class 'pandas.core.frame.DataFrame'>
RuntimeError
```
As you can see, a DataFrame is passed into the inspect function. You might be wondering why the type, DataFrame, got printed out twice. Pandas runs the first group twice. It does this to determine if there is a fast way to complete the computation or not. This is a minor detail that you shouldn't worry about. Now, let's do the same thing with transform
```
df.groupby('State').transform(inspect)
<class 'pandas.core.series.Series'>
<class 'pandas.core.series.Series'>
RuntimeError
```
It is passed a Series - a totally different Pandas object. So, transform is only allowed to work with a single Series at a time. It is impossible for it to act on two columns at the same time. So, if we try and subtract column a from b inside of our custom function we would get an error with transform. See below:
```
def subtract_two(x):
    return x['a'] - x['b']

df.groupby('State').transform(subtract_two)
KeyError: ('a', 'occurred at index a')
```
We get a KeyError as pandas is attempting to find the Series index a which does not exist. You can complete this operation with apply as it has the entire DataFrame:
```
df.groupby('State').apply(subtract_two)

State
Florida  2   -2
         3   -8
Texas    0   -2
         1   -5
dtype: int64
```
The output is a Series and a little confusing as the original index is kept, but we have access to all columns. Displaying the passed pandas object It can help even more to display the entire pandas object within the custom function, so you can see exactly what you are operating with. You can use print statements by I like to use the display function from the IPython.display module so that the DataFrames get nicely outputted in HTML in a jupyter notebook:
```
from IPython.display import display
def subtract_two(x):
    display(x)
    return x['a'] - x['b']
```
Screenshot: Transform must return a single dimensional sequence the same size as the group The other difference is that transform must return a single dimensional sequence the same size as the group. In this particular instance, each group has two rows, so transform must return a sequence of two rows. If it does not then an error is raised:
```
def return_three(x):
    return np.array([1, 2, 3])

df.groupby('State').transform(return_three)
ValueError: transform must return a scalar value for each group
```
The error message is not really descriptive of the problem. You must return a sequence the same length as the group. So, a function like this would work:
```
def rand_group_len(x):
    return np.random.rand(len(x))

df.groupby('State').transform(rand_group_len)

          a         b
0  0.962070  0.151440
1  0.440956  0.782176
2  0.642218  0.483257
3  0.056047  0.238208
```
Returning a single scalar object also works for transform If you return just a single scalar from your custom function, then transform will use it for each of the rows in the group:
```
def group_sum(x):
    return x.sum()

df.groupby('State').transform(group_sum)

   a   b
0  9  16
1  9  16
2  4  14
3  4  14
```


Question:
pandas: multiple conditions while indexing data frame - unexpected behavior I am filtering rows in a dataframe by values in two columns. For some reason the OR operator behaves like I would expect AND operator to behave and vice versa. My test code:
```py
df = pd.DataFrame({'a': range(5), 'b': range(5) })

# let's insert some -1 values
df['a'][1] = -1
df['b'][1] = -1
df['a'][3] = -1
df['b'][4] = -1

df1 = df[(df.a != -1) & (df.b != -1)]
df2 = df[(df.a != -1) | (df.b != -1)]

print(pd.concat([df, df1, df2], axis=1,
                keys = [ 'original df', 'using AND (&)', 'using OR (|)',]))
```
And the result:
```none
original df      using AND (&)      using OR (|)
             a  b              a   b             a   b
0            0  0              0   0             0   0
1           -1 -1            NaN NaN           NaN NaN
2            2  2              2   2             2   2
3           -1  3            NaN NaN            -1   3
4            4 -1            NaN NaN             4  -1

[5 rows x 6 columns]
```
As you can see, the AND operator drops every row in which at least one value equals -1. On the other hand, the OR operator requires both values to be equal to -1 to drop them. I would expect exactly the opposite result. Could anyone explain this behavior? I am using pandas 0.13.1.

Response:
As you can see, the AND operator drops every row in which at least one value equals -1. On the other hand, the OR operator requires both values to be equal to -1 to drop them. That's right. Remember that you're writing the condition in terms of what you want to keep, not in terms of what you want to drop. For df1:
```
df1 = df[(df.a != -1) & (df.b != -1)]
```
You're saying "keep the rows in which df.a isn't -1 and df.b isn't -1", which is the same as dropping every row in which at least one value is -1. For df2:
```
df2 = df[(df.a != -1) | (df.b != -1)]
```
You're saying "keep the rows in which either df.a or df.b is not -1", which is the same as dropping rows where both values are -1. PS: chained access like df['a'][1] = -1 can get you into trouble. It's better to get into the habit of using .loc and .iloc.


Question:
How to keep index when using pandas merge I would like to merge two DataFrames, and keep the index from the first frame as the index on the merged dataset. However, when I do the merge, the resulting DataFrame has integer index. How can I specify that I want to keep the index from the left data frame?
```

```python
a = pd.DataFrame({'col1': {'a': 1, 'b': 2, 'c': 3},
                          'to_merge_on': {'a': 1, 'b': 3, 'c': 4}})
```

```python
b = pd.DataFrame({'col2': {0: 1, 1: 2, 2: 3},
                          'to_merge_on': {0: 1, 1: 3, 2: 5}})
```

```python
a
#Output
#col1  to_merge_on
#a     1            1
#b     2            3
#c     3            4
```

```python
b
#Output
#col2  to_merge_on
#0     1            1
#1     2            3
#2     3            5
```

```python
a.merge(b, how='left')
#Output
#col1  to_merge_on  col2
#0     1            1   1.0
#1     2            3   2.0
#2     3            4   NaN
```

```python
_.index
#Output
#Int64Index([0, 1, 2], dtype='int64')
#```
#EDIT: Switched to example code that can be easily reproduced
```

Response:
```

```python
a.reset_index().merge(b, how="left").set_index('index')
#Output
#col1  to_merge_on  col2
#index
#a         1            1     1
#b         2            3     2
#c         3            4   NaN
#```
#Note that for some left merge operations, you may end up with more rows than in a when there are multiple matches between a and b. In this case, you may need to drop duplicates.
```


Question:
Changing a specific column name in pandas DataFrame [duplicate] This question already has answers here: Renaming column names in Pandas (33 answers) Closed 4 years ago. I was looking for an elegant way to change a specified column name in a DataFrame. play data ...
```
import pandas as pd
d = {
         'one': [1, 2, 3, 4, 5],
         'two': [9, 8, 7, 6, 5],
         'three': ['a', 'b', 'c', 'd', 'e']
    }
df = pd.DataFrame(d)
```
The most elegant solution I have found so far ...
```
names = df.columns.tolist()
names[names.index('two')] = 'new_name'
df.columns = names
```
I was hoping for a simple one-liner ... this attempt failed ...
```
df.columns[df.columns.tolist().index('one')] = 'another_name'
```
Any hints gratefully received.

Response:
A one liner does exist:
```

```python
df=df.rename(columns = {'two':'new_name'})
```

```python
df
#Output
#one three  new_name
#0    1     a         9
#1    2     b         8
#2    3     c         7
#3    4     d         6
#4    5     e         5
#```
#Following is the docstring for the rename method.
#```
#Definition: df.rename(self, index=None, columns=None, copy=True, inplace=False)
#Docstring:
#Alter index and / or columns using input function or
#functions. Function / dict values must be unique (1-to-1). Labels not
#contained in a dict / Series will be left as-is.
#
#Parameters
#----------
#index : dict-like or function, optional
#    Transformation to apply to index values
#columns : dict-like or function, optional
#    Transformation to apply to column values
#copy : boolean, default True
#    Also copy underlying data
#inplace : boolean, default False
#    Whether to return a new DataFrame. If True then value of copy is
#    ignored.
#
#See also
#--------
#Series.rename
#
#Returns
#-------
#renamed : DataFrame (new object)
#```
```


Question:
Concatenate strings from several rows using Pandas groupby I want to apply some sort of concatenation of the strings in a column using groupby. This is my code so far:
```
import pandas as pd
from io import StringIO

data = StringIO("""
"name1","hej","2014-11-01"
"name1","du","2014-11-02"
"name1","aj","2014-12-01"
"name1","oj","2014-12-02"
"name2","fin","2014-11-01"
"name2","katt","2014-11-02"
"name2","mycket","2014-12-01"
"name2","lite","2014-12-01"
""")

# load string as stream into dataframe
df = pd.read_csv(data,header=0, names=["name","text","date"],parse_dates=[2])

# add column with month
df["month"] = df["date"].apply(lambda x: x.month)
```
I want the end result to look like this:

Response:
You can groupby the 'name' and 'month' columns, then call transform which will return data aligned to the original df and apply a lambda where we join the text entries:
```

```python
df['text'] = df[['name','text','month']].groupby(['name','month'])['text'].transform(lambda x: ','.join(x))
df[['name','text','month']].drop_duplicates()
#Output
#name         text  month
#0  name1       hej,du     11
#2  name1        aj,oj     12
#4  name2     fin,katt     11
#6  name2  mycket,lite     12
#```
#I sub the original df by passing a list of the columns of interest df[['name','text','month']] here and then call drop_duplicates EDIT actually I can just call apply and then reset_index:
#```
```

```python
df.groupby(['name','month'])['text'].apply(lambda x: ','.join(x)).reset_index()
#Output
#name  month         text
#0  name1     11       hej,du
#1  name1     12        aj,oj
#2  name2     11     fin,katt
#3  name2     12  mycket,lite
#```
#update the lambda is unnecessary here:
#```
```

```python
df.groupby(['name','month'])['text'].apply(','.join).reset_index()
#Output
#name  month         text
#0  name1     11           du
#1  name1     12        aj,oj
#2  name2     11     fin,katt
#3  name2     12  mycket,lite
#```
```


Question:
check if variable is dataframe when my function f is called with a variable I want to check if var is a pandas dataframe:
```
def f(var):
    if var == pd.DataFrame():
        print "do stuff"
```
I guess the solution might be quite simple but even with
```
def f(var):
    if var.values != None:
        print "do stuff"
```
I can't get it to work like expected.

Response:
Use isinstance, nothing else:
```
if isinstance(x, pd.DataFrame):
    ... # do something
```
PEP8 says explicitly that isinstance is the preferred way to check types
```
No:  type(x) is pd.DataFrame
No:  type(x) == pd.DataFrame
Yes: isinstance(x, pd.DataFrame)
```
And don't even think about
```
if obj.__class__.__name__ = 'DataFrame':
    expect_problems_some_day()
```
isinstance handles inheritance (see What are the differences between type() and isinstance()?). For example, it will tell you if a variable is a string (either str or unicode), because they derive from basestring)
```
if isinstance(obj, basestring):
    i_am_string(obj)
```
Specifically for pandas DataFrame objects:
```
import pandas as pd
isinstance(var, pd.DataFrame)
```


Question:
How do I combine two dataframes? I have a initial dataframe D. I extract two data frames from it like this:
```
A = D[D.label == k]
B = D[D.label != k]
```
I want to combine A and B into one DataFrame. The order of the data is not important. However, when we sample A and B from D, they retain their indexes from D.

Response:
DEPRECATED: DataFrame.append and Series.append were deprecated in v1.4.0. Use append:
```
df_merged = df1.append(df2, ignore_index=True)
```
And to keep their indexes, set ignore_index=False.


Question: How to split data into 3 sets (train, validation and test)? I have a pandas dataframe and I wish to divide it to 3 separate sets. I know that using train_test_split from sklearn.cross_validation, one can divide the data in two sets (train and test). However, I couldn't find any solution about splitting the data into three sets. Preferably, I'd like to have the indices of the original data. I know that a workaround would be to use train_test_split two times and somehow adjust the indices. But is there a more standard / built-in way to split the data into 3 sets instead of 2?

Response:
Numpy solution. We will shuffle the whole dataset first (df.sample(frac=1, random_state=42)) and then split our data set into the following parts: 60% - train set, 20% - validation set, 20% - test set
```

```python
train, validate, test = \
              np.split(df.sample(frac=1, random_state=42),
                       [int(.6*len(df)), int(.8*len(df))])
```

```python
train
#Output
#A         B         C         D         E
#0  0.046919  0.792216  0.206294  0.440346  0.038960
#2  0.301010  0.625697  0.604724  0.936968  0.870064
#1  0.642237  0.690403  0.813658  0.525379  0.396053
#9  0.488484  0.389640  0.599637  0.122919  0.106505
#8  0.842717  0.793315  0.554084  0.100361  0.367465
#7  0.185214  0.603661  0.217677  0.281780  0.938540
```

```python
validate
#Output
#A         B         C         D         E
#5  0.806176  0.008896  0.362878  0.058903  0.026328
#6  0.145777  0.485765  0.589272  0.806329  0.703479
```

```python
test
#Output
#A         B         C         D         E
#4  0.521640  0.332210  0.370177  0.859169  0.401087
#3  0.333348  0.964011  0.083498  0.670386  0.169619
#```
#[int(.6*len(df)), int(.8*len(df))] - is an indices_or_sections array for numpy.split(). Here is a small demo for np.split() usage - let's split 20-elements array into the following parts: 80%, 10%, 10%:
#```
```

```python
a = np.arange(1, 21)
```

```python
a
#Output
#array([ 1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20])
```

```python
np.split(a, [int(.8 * len(a)), int(.9 * len(a))])
#Output
#[array([ 1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16]),
# array([17, 18]),
# array([19, 20])]
#```
```


Question:
Format y axis as percent I have an existing plot that was created with pandas like this:
```
df['myvar'].plot(kind='bar')
```
The y axis is format as float and I want to change the y axis to percentages. All of the solutions I found use ax.xyz syntax and I can only place code below the line above that creates the plot (I cannot add ax=ax to the line above.) How can I format the y axis as percentages without changing the line above? Here is the solution I found but requires that I redefine the plot:
```
import matplotlib.pyplot as plt
import numpy as np
import matplotlib.ticker as mtick

data = [8,12,15,17,18,18.5]
perc = np.linspace(0,100,len(data))

fig = plt.figure(1, (7,4))
ax = fig.add_subplot(1,1,1)

ax.plot(perc, data)

fmt = '%.0f%%' # Format you want the ticks, e.g. '40%'
xticks = mtick.FormatStrFormatter(fmt)
ax.xaxis.set_major_formatter(xticks)

plt.show()
```
Link to the above solution: Pyplot: using percentage on x axis

Response:
This is a few months late, but I have created PR#6251 with matplotlib to add a new PercentFormatter class. With this class you just need one line to reformat your axis (two if you count the import of matplotlib.ticker):
```
import ...
import matplotlib.ticker as mtick

ax = df['myvar'].plot(kind='bar')
ax.yaxis.set_major_formatter(mtick.PercentFormatter())
```
PercentFormatter() accepts three arguments, xmax, decimals, symbol. xmax allows you to set the value that corresponds to 100% on the axis. This is nice if you have data from 0.0 to 1.0 and you want to display it from 0% to 100%. Just do PercentFormatter(1.0). The other two parameters allow you to set the number of digits after the decimal point and the symbol. They default to None and '%', respectively. decimals=None will automatically set the number of decimal points based on how much of the axes you are showing. Update PercentFormatter was introduced into Matplotlib proper in version 2.1.0.


Question:
pandas dataframe columns scaling with sklearn I have a pandas dataframe with mixed type columns, and I'd like to apply sklearn's min_max_scaler to some of the columns. Ideally, I'd like to do these transformations in place, but haven't figured out a way to do that yet. I've written the following code that works:
```
import pandas as pd
import numpy as np
from sklearn import preprocessing

scaler = preprocessing.MinMaxScaler()

dfTest = pd.DataFrame({'A':[14.00,90.20,90.95,96.27,91.21],'B':[103.02,107.26,110.35,114.23,114.68], 'C':['big','small','big','small','small']})
min_max_scaler = preprocessing.MinMaxScaler()

def scaleColumns(df, cols_to_scale):
    for col in cols_to_scale:
        df[col] = pd.DataFrame(min_max_scaler.fit_transform(pd.DataFrame(dfTest[col])),columns=[col])
    return df

dfTest

    A   B   C
0    14.00   103.02  big
1    90.20   107.26  small
2    90.95   110.35  big
3    96.27   114.23  small
4    91.21   114.68  small

scaled_df = scaleColumns(dfTest,['A','B'])
scaled_df

A   B   C
0    0.000000    0.000000    big
1    0.926219    0.363636    small
2    0.935335    0.628645    big
3    1.000000    0.961407    small
4    0.938495    1.000000    small
```
I'm curious if this is the preferred/most efficient way to do this transformation. Is there a way I could use df.apply that would be better? I'm also surprised I can't get the following code to work:
```
bad_output = min_max_scaler.fit_transform(dfTest['A'])
```
If I pass an entire dataframe to the scaler it works:
```
dfTest2 = dfTest.drop('C', axis = 1)
good_output = min_max_scaler.fit_transform(dfTest2)
good_output
```
I'm confused why passing a series to the scaler fails. In my full working code above I had hoped to just pass a series to the scaler then set the dataframe column = to the scaled series.

Response:
I am not sure if previous versions of pandas prevented this but now the following snippet works perfectly for me and produces exactly what you want without having to use apply
```

```python
import pandas as pd
```

```python
from sklearn.preprocessing import MinMaxScaler
```

```python
scaler = MinMaxScaler()
```

```python
dfTest = pd.DataFrame({'A':[14.00,90.20,90.95,96.27,91.21],
#Output
#                           'B':[103.02,107.26,110.35,114.23,114.68],
#                           'C':['big','small','big','small','small']})
```

```python
dfTest[['A', 'B']] = scaler.fit_transform(dfTest[['A', 'B']])
```

```python
dfTest
#Output
#          A         B      C
#0  0.000000  0.000000    big
#1  0.926219  0.363636  small
#2  0.935335  0.628645    big
#3  1.000000  0.961407  small
#4  0.938495  1.000000  small
#```
```


Question:
Can pandas automatically read dates from a CSV file? Today I was positively surprised by the fact that while reading data from a data file (for example) pandas is able to recognize types of values:
```
df = pandas.read_csv('test.dat', delimiter=r"\s+", names=['col1','col2','col3'])
```
For example it can be checked in this way:
```
for i, r in df.iterrows():
    print type(r['col1']), type(r['col2']), type(r['col3'])
```
In particular integer, floats and strings were recognized correctly. However, I have a column that has dates in the following format: 2013-6-4. These dates were recognized as strings (not as python date-objects). Is there a way to "learn" pandas to recognized dates?

Response:
You should add parse_dates=True, or parse_dates=['column name'] when reading, thats usually enough to magically parse it. But there are always weird formats which need to be defined manually. In such a case you can also add a date parser function, which is the most flexible way possible. Suppose you have a column 'datetime' with your string, then:
```
from datetime import datetime
dateparse = lambda x: datetime.strptime(x, '%Y-%m-%d %H:%M:%S')

df = pd.read_csv(infile, parse_dates=['datetime'], date_parser=dateparse)
```
This way you can even combine multiple columns into a single datetime column, this merges a 'date' and a 'time' column into a single 'datetime' column:
```
dateparse = lambda x: datetime.strptime(x, '%Y-%m-%d %H:%M:%S')

df = pd.read_csv(infile, parse_dates={'datetime': ['date', 'time']}, date_parser=dateparse)
```
You can find directives (i.e. the letters to be used for different formats) for strptime and strftime in this page.


Question:
Ignoring NaNs with str.contains I want to find rows that contain a string, like so:
```
DF[DF.col.str.contains("foo")]
```
However, this fails because some elements are NaN: ValueError: cannot index with vector containing NA / NaN values So I resort to the obfuscated
```
DF[DF.col.notnull()][DF.col.dropna().str.contains("foo")]
```
Is there a better way?

Response:
There's a flag for that:
```

```python
df = pd.DataFrame([["foo1"], ["foo2"], ["bar"], [np.nan]], columns=['a'])
```

```python
df.a.str.contains("foo")
#Output
#0     True
#1     True
#2    False
#3      NaN
#Name: a, dtype: object
```

```python
df.a.str.contains("foo", na=False)
#Output
#0     True
#1     True
#2    False
#3    False
#Name: a, dtype: bool
#```
#See the str.replace docs: na : default NaN, fill value for missing values. So you can do the following:
#```
```

```python
df.loc[df.a.str.contains("foo", na=False)]
#Output
#a
#0  foo1
#1  foo2
#```
```


Question:
Extract column value based on another column in Pandas I am kind of getting stuck on extracting value of one variable conditioning on another variable. For example, the following dataframe:
```
A  B
p1 1
p1 2
p3 3
p2 4
```
How can I get the value of A when B=3? Every time when I extracted the value of A, I got an object, not a string.

Response:
You could use loc to get series which satisfying your condition and then iloc to get first element:
```

```python
df
#Output
#A  B
#0  p1  1
#1  p1  2
#2  p3  3
#3  p2  4
```

```python
df.loc[df['B'] == 3, 'A']
#Output
#2    p3
#Name: A, dtype: object
```

```python
df.loc[df['B'] == 3, 'A'].iloc[0]
#Output
#'p3'
#```
```


Question: How to plot multiple dataframes in subplots I have a few Pandas DataFrames sharing the same value scale, but having different columns and indices. When invoking df.plot(), I get separate plot images. what I really want is to have them all in the same plot as subplots, but I'm unfortunately failing to come up with a solution to how and would highly appreciate some help.

Response:
You can manually create the subplots with matplotlib, and then plot the dataframes on a specific subplot using the ax keyword. For example for 4 subplots (2x2):
```
import matplotlib.pyplot as plt

fig, axes = plt.subplots(nrows=2, ncols=2)

df1.plot(ax=axes[0,0])
df2.plot(ax=axes[0,1])
...
```
Here axes is an array which holds the different subplot axes, and you can access one just by indexing axes. If you want a shared x-axis, then you can provide sharex=True to plt.subplots.


Question:
How to load a tsv file into a Pandas DataFrame? I'm trying to get a TAB-delimited (tsv) file loaded into a pandas DataFrame. This is what I'm trying and the error I'm getting:
```

```python
df1 = DataFrame(csv.reader(open('c:/~/trainSetRel3.txt'), delimiter='\t'))
```

Traceback (most recent call last):
  File "<pyshell#28>", line 1, in <module>
    df1 = DataFrame(csv.reader(open('c:/~/trainSetRel3.txt'), delimiter='\t'))
  File "C:\Python27\lib\site-packages\pandas\core\frame.py", line 318, in __init__
    raise PandasError('DataFrame constructor not properly called!')
PandasError: DataFrame constructor not properly called!
```

Response:
The .read_csv function does what you want:
```
pd.read_csv('c:/~/trainSetRel3.txt', sep='\t')
```
If you have a header, you can pass header=0.
```
pd.read_csv('c:/~/trainSetRel3.txt', sep='\t', header=0)
```
Note: Prior 17.0, pd.DataFrame.from_csv was used (it is now deprecated and the .from_csv documentation link redirects to the page for pd.read_csv).